Your gateway is fine, until everyone hits it at once
A hundred developers on Claude Code average about one tool call every twenty seconds. Every gateway can serve that. Then a CI run, a multi-agent workflow, and the morning rush land together, and the gateways stop looking alike. We put five of them through the same burst on real AWS hardware. A call that comes back in a quarter of a second through Highflame takes thirty-seven seconds through LiteLLM.
MEASURED AWS t3.medium and t3.xlarge, us-east-1, median of three runs each, matching the instance sizes in the published Bifrost benchmark. Load generator colocated with each gateway; full method below.
Same burst, same hardware, five gateways
Response time under a momentary crush of 500 concurrent calls, and throughput each gateway can actually hold. Switch between them.
A hundred developers make about one tool call every twenty seconds. Average load never stresses a gateway.
Bursts do. The real question is not whether your gateway can do 5,000 requests a second. It is what happens in the few seconds a day your team asks for a few hundred at once.
Who actually sustains 5,000 a second
Aimed at a steady 5,000 requests a second for a minute on a t3.xlarge. This is the cleanest comparison here, because the gateways that keep up are nowhere near straining.
| Gateway | Sustained rps | p50 | p99 | Success |
|---|---|---|---|---|
| Highflame | 4,999.9 | 1.14 ms | 13.29 ms | 100.00% |
| Bifrost | 4,998.4 | 6.98 ms | 30.07 ms | 99.99% |
| Kong | 1,546.3 | 28.37 ms | 65.34 ms | 99.95% |
| Portkey | 306.0 | 155.06 ms | 271.22 ms | 99.73% |
| LiteLLM | 95.5 | 490.54 ms | 978.97 ms | 99.13% |
Only Highflame and Bifrost held the full target at essentially 100% success. Between them, Highflame ran about six times lower at the median and a little over twice as low at p99, at the same throughput. Kong, Portkey, and LiteLLM topped out well short and let requests queue behind them.
So we kept pushing, to find the real ceiling
Maxim's test stops at 5,000. We aimed 10,000 and then 20,000 at the same box to see where Highflame and Bifrost actually top out.
| Achieved rps | 5,000 target | 10,000 target | 20,000 target | Real ceiling |
|---|---|---|---|---|
| Highflame | 5,000 | 8,263 | 8,135 | ~8,200 |
| Bifrost | 4,998 | 5,822 | 5,724 | ~5,770 |
Both flatten well before 10,000 and hold flat to 20,000, the signature of a real ceiling rather than an under-powered test. Highflame tops out near 8,200 a second, about 42% more headroom than Bifrost on identical hardware. What matters more than the ceiling is the behavior at it: pushed two to four times past their limit, neither collapses. Highflame's p99 moves from 11.8ms to 12.3ms across the whole range, Bifrost's from 26.8ms to 27.3ms. They cap and hold, the opposite of what LiteLLM and Portkey do under a burst.
It comes down to the runtime
| Highflame | Bifrost | Kong | Portkey | LiteLLM | |
|---|---|---|---|---|---|
| Language | Rust | Go | Lua on nginx | Node / TS | Python |
| Concurrency | Tokio async | Goroutines | nginx workers | single event loop | uvicorn workers |
| Under a 500-call burst (p99) | 0.23 s | 0.25 s | 0.52 s | 6.8 s | 37.5 s |
The two natively concurrent compiled gateways stay flat under load. The interpreted and single-threaded runtimes fall into whole seconds when a burst lands, because one slow moment stalls everything behind it. We have a theory about the exact mechanism in Portkey's case and have not confirmed it with a profiler, so the table shows the measured number, not the diagnosis.
How we ran this
Honesty about test conditions is a feature, not a footnote.
- Real hardware, matched to theirs
- The 500-concurrent burst test ran on a t3.medium and the 5,000-rps stress test on a t3.xlarge, us-east-1, the instance sizes in the Bifrost benchmark we reproduced.
- Median of three runs
- Every published figure is the median of three repeats on dedicated hardware, so a single noisy run cannot set the story.
- A mock that holds a set delay
- We built a configurable Go mock so we could run every gateway at a 0ms and a 60ms backend, and always state which delay produced a number.
- Five gateways, one harness
- Highflame, Bifrost, Kong, Portkey, and LiteLLM, each through the same load generator and mock, one at a time with no contention between them.
- Burst response is not overhead
- The 500-concurrent p99 figures are what a user feels under a brief overload, not a clean gateway-overhead number. The clean comparison is the 5,000-rps table.
- One caveat we are still checking
- The load generator ran on the same box as each gateway, the same way for all five, so the comparison between them is fair. Whether that matches every detail of the original benchmark's rig, we cannot yet say.
Reading the numbers
Are these numbers real or a laptop?
Real AWS hardware. The 500-concurrent burst test ran on a t3.medium and the 5,000-rps stress test on a t3.xlarge, us-east-1, each the median of three runs, matching the instance sizes in the Bifrost benchmark we set out to reproduce.
Why did you measure at a burst instead of average load?
Because average load never stresses a gateway. A hundred developers doing organic tool calls average about one request every twenty seconds. The pressure comes from bursts, CI fan-out, multi-agent workflows, everyone starting the day at once, and that is where the gateways stop behaving alike.
Why is LiteLLM so far behind?
Under 500 concurrent connections a single Python worker process cannot keep up, so requests queue for seconds and some time out. It is a runtime limit, not a configuration mistake.
Is Highflame just a faster proxy, then?
No. Highflame inspects every LLM and MCP call inline against your policy before forwarding it. Speed is what makes running that inspection on the hot path viable instead of a tax teams switch off.
Can I reproduce this?
Yes. The harness, the configurable mock, and the Kong and Portkey adapters are in the firehog performance suite. Get in touch and we will point you at it.
The interesting question is what runs on the hot path
Highflame stays at the front of that tail-latency chart so it can do the thing a plain proxy does not: inspect every LLM and MCP call inline, against your policy, before it forwards. See what that looks like.