Live backend

Load and chaos report

The services behind the playground's Live API mode — FastAPI, Django, RabbitMQ, Kafka, workers and a WebSocket hub — under load, and with parts of them broken on purpose. Every number on this page comes from a real run of scripts that are in the repository.

Measured on a developer machine, not in production: the services run in Docker Desktop on Windows with the CPU and memory limits from compose, and the load comes from the same machine. A server will give other absolute numbers; what this shows is the shape — what stays fast, what queues, what breaks and how it comes back.

Send a live request in the playground

Load

Simulated playground visitors, each with a sandbox of their own, doing what the Live API mode does: reads, orders, background jobs, payments and webhooks. Measured on 6 November 2026.

20 visitors

37.4 requests/s · p95 150 ms · 0 failed of 2242

20 visitors
Request	req/s	p50	p95	p99
`GET /api/orders/42`Reads · cache	7.5	5	8	9
`GET /api/orders/42/checkout`Payments API · Django	2.1	7	12	30
`GET /api/products/7`Reads · cache	7.6	4	6	8
`GET /api/traces/[id]`Traces	8.3	3	6	7
`POST /api/orders`Orders · Kafka	4.2	8	12	13
`POST /api/orders (422)`Reads · cache	2.1	4	6	9
`POST /api/orders/42/pay`Jobs · RabbitMQ	2	15	24	44
`POST /api/reports`Jobs · RabbitMQ	1.9	14	23	29
`POST /webhooks/payments`Payments API · Django	1.6	8	11	19

Queued work, end to end

From the request's start to the end of its last background step, taken from the trace: measured by the services themselves, not by the load generator.

Queued work, end to end
Request	Done	p50	p95	p99
order → Kafka → 3 groups	251	150	260	270
payment job → worker → provider	120	19	34	55
report job → worker	114	18	28	32

Busiest containers

Busiest containers
Service	CPU peak (cores)	Memory peak
rabbitmq	1.65 CPU	145 MB of 512 MB
kafka	0.47 CPU	396 MB of 768 MB
provider-stub	0.19 of 0.25 CPU	54 MB of 128 MB
ops-bot	0.15 of 0.25 CPU	96 MB of 128 MB
orders-api	0.09 of 1 CPU	76 MB of 256 MB
realtime-hub	0.05 of 0.25 CPU	54 MB of 128 MB

60 visitors

112.3 requests/s · p95 140 ms · 0 failed of 6737

60 visitors
Request	req/s	p50	p95	p99
`GET /api/orders/42`Reads · cache	23	5	10	13
`GET /api/orders/42/checkout`Payments API · Django	6.2	8	13	43
`GET /api/products/7`Reads · cache	22.9	4	8	10
`GET /api/traces/[id]`Traces	24.9	4	7	10
`POST /api/orders`Orders · Kafka	11.8	8	13	16
`POST /api/orders (422)`Reads · cache	6.1	4	7	8
`POST /api/orders/42/pay`Jobs · RabbitMQ	6.2	16	25	32
`POST /api/reports`Jobs · RabbitMQ	5.9	15	24	28
`POST /webhooks/payments`Payments API · Django	5.3	7	13	38

Queued work, end to end

From the request's start to the end of its last background step, taken from the trace: measured by the services themselves, not by the load generator.

Queued work, end to end
Request	Done	p50	p95	p99
order → Kafka → 3 groups	708	140	260	270
payment job → worker → provider	369	21	29	44
report job → worker	358	19	28	34

Busiest containers

Busiest containers
Service	CPU peak (cores)	Memory peak
kafka	1.56 CPU	460 MB of 768 MB
orders-api	0.39 of 1 CPU	79 MB of 256 MB
rabbitmq	0.36 CPU	105 MB of 512 MB
ops-bot	0.17 of 0.25 CPU	96 MB of 128 MB
postgres	0.16 CPU	100 MB
order-worker	0.07 of 0.5 CPU	67 MB of 192 MB

Chaos

Each experiment states what should hold, breaks one thing for real — kills a container, stops a broker — measures what happens and whether the system comes back by itself. Measured on 6 November 2026.

A worker dies in the middle of payments

Hypothesis held

What was broken: 100 payments whose first provider call fails; while both order-worker replicas work through them, one is killed (SIGKILL)
Hypothesis: No payment is lost and none is charged twice: RabbitMQ redelivers the unacknowledged jobs to the other replica, and the provider gets the task id as its idempotency key.

What was measured

Payments finished: 100 of 100 succeeded
Charges at the provider: 100 charge ids for 100 payments: none charged twice
Redelivered to the other worker: 1 job(s) the killed worker held unacknowledged
All done after the kill: 1.1 s

Kafka is down while orders come in

Hypothesis held

What was broken: Kafka stopped; 10 orders created; Kafka started again
Hypothesis: Orders are still accepted at full speed: each event waits in the outbox, in the same transaction as its order, and every consumer group catches up once Kafka is back.

What was measured

Orders accepted while Kafka was down: 10 of 10 (201), median 8 ms
Events waiting in the outbox: 10
Every consumer group caught up after Kafka was back: 16.0 s: 10 reserved, counted and notified

RabbitMQ is down

Hypothesis held

What was broken: RabbitMQ stopped, then started again
Hypothesis: Only queued work is refused, with 503 and Retry-After; reads and orders keep working; once RabbitMQ is back the API and the workers reconnect by themselves.

What was measured

A new job while RabbitMQ is down: 503, Retry-After 5 s
Everything that needs no queue: reads 200, orders 201
Jobs accepted again, without a restart: after 9.1 s
That job done by a reconnected worker: 0.0 s later

Redis is down

Hypothesis held

What was broken: Redis stopped, then started again
Hypothesis: The API fails fast and honestly (503 with Retry-After, readiness down) instead of erroring or hanging, and serves again as soon as Redis is back.

What was measured

A request while Redis is down: 503, Retry-After 5, in 545 ms
Readiness: 503: redis unavailable: TimeoutError
Serving again, without a restart: after 2.4 s
A sandbox from before the restart: 401: gone with Redis, as designed (nothing personal is kept)
Realtime hub subscribed again: after 0.9 s
An order right after the restart: its trace completed in 0.5 s: every service reconnected

Found and fixed. The first run disproved this hypothesis: without Redis the Orders API answered 500 after five seconds — nothing handled a lost Redis, and the trace of the failure was written into that same Redis. Now a lost Redis gives a 503 with Retry-After within half a second (the client's timeout), a trace is never written into a Redis that just failed, and a regression test keeps it that way. The load run right after found one more: the first call on a connection opened before the restart failed, so one order event's trace never completed (the order itself was handled). Every Redis client now repeats a call on a closed connection, and this experiment now places an order right after the restart and waits for its trace.

Where it was measured

Machine: AMD Ryzen 5 7500F 6-Core Processor, 12 logical CPUs, 31.6 GB RAM, Windows-11
Docker: Docker Desktop 29.2.1, 12 logical CPUs, 15.4 GB RAM
Load generator: Locust 2.47.0 on the same machine

Container limits (compose)

Container limits (compose)
Service	Replicas	CPU	Memory
edge	1	0.5	128 MB
kafka	1	—	768 MB
ops-bot	1	0.25	128 MB
ops-bot-run	1	0.25	128 MB
order-worker	2	0.5	192 MB
orders-api	1	1	256 MB
outbox-relay	1	0.25	128 MB
payments	1	0.5	256 MB
payments-sweeper	1	0.1	128 MB
postgres	1	—	—
provider-stub	1	0.25	128 MB
rabbitmq	1	—	512 MB
realtime-hub	2	0.25	128 MB
redis	1	—	—

Method

Load: Locust. Each level ramps up, settles, then is measured over a steady window; a visitor waits 0.2–1 s between actions.
The per-visitor rate limits are lifted for the run (compose.load.yaml): one machine plays every visitor from one IP, and the point is the services, not the limiter. The limiter has a scenario of its own in the playground.
Latencies are percentiles over the measured window; Locust rounds times above 100 ms to the nearest 10 ms.
Chaos: the experiments run one after another; after each, whatever was broken is started again before the next one begins.

Mix of actions per visitor

GET /api/orders/42 × 4
GET /api/products/7 × 4
POST /api/orders (422) × 1
POST /api/orders × 2
POST /api/reports × 1
POST /api/orders/42/pay × 1
GET /api/orders/42/checkout × 1
POST /webhooks/payments × 1

Run it yourself

cd backend
docker compose up -d --build
uv run --group load python scripts/run_load.py
uv run python scripts/run_chaos.py