Your gateway is fine, until everyone hits it at once
A hundred developers on Claude Code average about one tool call every twenty seconds. Every gateway can serve that. Then a CI run, a multi-agent workflow, and the morning rush land together, and the gateways stop looking alike. We put four of them through the same crush on a two-host AWS rig. A call that comes back in under a second through Highflame drags out to seventeen seconds through LiteLLM, and most of LiteLLM's never come back at all.
MEASURED AWS t2.xlarge, us-east-1, five-minute sustained runs, with the load generator on a separate host so every request crosses a real network. Full method below.
Same rush, same hardware, four gateways
Response time when 5,000 connections land at once, and the throughput each gateway holds for five minutes. Switch between them.
A hundred developers make about one tool call every twenty seconds. Average load never stresses a gateway.
Bursts do. The real question is not whether your gateway can do fourteen thousand requests a second. It is what happens in the few seconds a day your team asks for a few hundred at once.
Who actually holds the rate
A constant, heavy load held for five minutes on a t2.xlarge. The backend answers instantly, so the number is each gateway's own throughput, with nowhere to hide.
| Gateway | Sustained rps | p99 drift | Peak memory | Failed requests |
|---|---|---|---|---|
| Highflame | 14,331 | +0.4% | 47 MB | 0 |
| Bifrost | 6,799 | +3.2% | 320 MB | 0 |
| Portkey | 432 | −1.4% | 320 MB | 0 |
| LiteLLM | 243 | −6.5% | 1751 MB | 0 |
Highflame held 14,331 requests a second, its p99 drifting under half a percent across the five minutes and memory flat at 47 MB, a little over twice Bifrost's throughput on the same box. Portkey and LiteLLM never cleared a few hundred a second, because each does its real work inside one interpreted process. Nobody dropped a request at this load; the gap here is raw headroom.
So we pushed past the ceiling
We ramped each gateway to its limit, then deliberately offered about 50% more, to see whether it queues or falls over.
| Gateway | Throughput ceiling | Delivered when overloaded | HTTP success |
|---|---|---|---|
| Highflame | ~14,700 rps | 13,595 rps | 100% |
| Bifrost | ~6,900 rps | 6,980 rps | 100% |
| Portkey | ~430 rps | 419 rps | 100% |
| LiteLLM | ~245 rps | 246 rps | 100% |
Every gateway kept returning 200s; none errored. Pushed half again past its ceiling, Highflame queued the excess and still delivered 13,595 a second at 100% success, its p99 pinned near 12 milliseconds. Bifrost sat at its own ceiling and held. Highflame's ceiling is a little over twice Bifrost's on identical hardware, and neither compiled gateway collapses at the edge.
It comes down to the runtime
| Highflame | Bifrost | Portkey | LiteLLM | |
|---|---|---|---|---|
| Language | Rust | Go | Node / TS | Python |
| Concurrency | Tokio async | Goroutines | single event loop | uvicorn workers |
| Holding 5,000 connections (p99) | 0.83 s | 0.99 s | 7.3 s | 17.2 s |
The two natively concurrent compiled gateways stay flat under load. The interpreted, largely single-threaded runtimes fall into whole seconds when a rush lands, because one slow moment stalls everything queued behind it. We have a theory about the exact mechanism in Portkey's case and have not confirmed it with a profiler, so the table shows the measured number, not the diagnosis.
How we ran this
Honesty about test conditions is a feature, not a footnote.
- Real hardware, load on its own host
- The gateway and its backend sit on a t2.xlarge (4 vCPU, 16 GiB, us-east-1). The load generator runs on a separate 8-vCPU EC2 box, so every request crosses a real network hop rather than loopback.
- Five-minute sustained runs
- Every headline figure is a continuous five-minute run held at load, not a sixty-second burst. A one-second sampler records CPU, memory, and TCP state on both hosts throughout.
- An instant backend, on purpose
- The backend is an nginx mock that answers in roughly zero milliseconds, so each number is the gateway's own cost, with the backend-direct figure recorded alongside as the floor.
- Four gateways, one harness
- Highflame, Bifrost, Portkey, and LiteLLM, each through the same load generator and mock, one at a time with no contention between them.
- Rush response is not overhead
- The p99 while holding 5,000 connections is what a user feels under a brief overload, not a clean gateway-overhead number. The clean comparison is the sustained table.
- The caveat we are still closing
- Responses are small fixed-size bodies here, so work a gateway spends parsing and re-serializing large responses is undercounted. The next run standardizes larger responses to close that gap.
Reading the numbers
Are these numbers real or a laptop?
Real AWS hardware. The gateway and its backend run on a t2.xlarge in us-east-1, with the load generator on a separate EC2 box so every request crosses a real network. Each figure is a five-minute sustained run.
Why did you measure at a rush instead of average load?
Because average load never stresses a gateway. A hundred developers doing organic tool calls average about one request every twenty seconds. The pressure comes from bursts, CI fan-out, multi-agent workflows, everyone starting the day at once, and that is where the gateways stop behaving alike.
Why is LiteLLM so far behind?
Under thousands of concurrent connections a single Python worker process cannot keep up, so requests queue for seconds and some time out. It is a runtime limit, not a configuration mistake.
Is Highflame just a faster proxy, then?
No. Highflame inspects every LLM and MCP call inline against your policy before forwarding it. Speed is what makes running that inspection on the hot path viable instead of a tax teams switch off.
Can I reproduce this?
Yes. The harness, the configurable mock, and the gateway adapters are in the firehog performance suite. Get in touch and we will point you at it.
The interesting question is what runs on the hot path
Highflame stays at the front of that tail-latency chart so it can do the thing a plain proxy does not: inspect every LLM and MCP call inline, against your policy, before it forwards. See what that looks like.