The morning your whole team opens Claude Code at once, a tool call that should come back in under a second can take seventeen. The model is fine; the wait lives in the gateway in front of it.
Load testing an AI gateway means driving concurrent traffic through it and measuring what happens at the peak, not the average. A gateway that is invisible at one request every twenty seconds can add seventeen seconds of latency the morning a hundred developers land at once.
Most of the day that gateway is invisible. A hundred developers doing normal work average about one request every twenty seconds, and any gateway ever written can serve that. The one you choose only starts to matter in the few seconds when everyone lands together: the morning rush, a CI run fanning out, three agent workflows firing at once.
So we measured exactly those seconds. Four gateways, the same AWS hardware, the same rush.
Forget the average
Real developer traffic is spiky. Everyone starts their morning inside the same half hour. A CI run fans out a dozen tool calls at once. An agentic workflow chains six calls back to back inside a single session, and three of those workflows fire together. For a few seconds at a time, “one request every twenty seconds” turns into hundreds of requests in flight, and then it falls back to nothing.
Those spikes are the only moments your gateway is under real pressure, and they are exactly the moments a developer is watching a tool call, waiting for it to come back.
How we ran it
We rebuilt the benchmark Maxim publishes for their Bifrost gateway, then made it harder to argue with. The load generator runs on its own EC2 box, separate from the gateway, so every request crosses a real network hop instead of a loopback shortcut. And every number here comes from a five-minute run held at a constant load, not a sixty-second burst, because a gateway that looks composed for a minute can still drift once its pools warm up and its garbage collector kicks in.
The gateway under test and its backend sit on a t2.xlarge in us-east-1. The backend is an nginx mock that answers in about zero milliseconds, so whatever latency or throughput you read is the gateway’s own cost, not the model’s. Four gateways went through it: Highflame (Rust), Bifrost (Go), Portkey (Node), and LiteLLM (Python).
Under steady load, Highflame barely moves
Before the rush, the steady state. Hold a heavy, constant load on each gateway for five minutes and watch for drift.
| Gateway | Sustained rps (5-min hold) | p99 drift | Peak memory | Failed requests |
|---|---|---|---|---|
| Highflame | 14,331 | +0.4% | 47 MB | 0 |
| Bifrost | 6,799 | +3.2% | 320 MB | 0 |
| Portkey | 432 | −1.4% | 320 MB | 0 |
| LiteLLM | 243 | −6.5% | 1751 MB | 0 |
Highflame held 14,331 requests a second for the full five minutes: its p99 drifted four tenths of a percent, memory stayed flat at 47 MB, and not one request failed across roughly 4.3 million. Bifrost held 6,799, a little under half that. Portkey and LiteLLM never cleared a few hundred a second, each capped by its single interpreted process. Nobody dropped a request here, so the gap is not reliability, it is raw headroom.
Headroom for when you scale
A rate you can sustain is only half the answer. The other half is what happens when a spike asks for more than the gateway can serve. We ramped each one to its ceiling, then offered about 50% more.
Highflame’s ceiling is a little over twice Bifrost’s, and pushed half again past it, it still holds 100% success with its p99 pinned around 12 milliseconds. It queues the overflow rather than shedding it, so it caps and holds instead of falling over. Your team will never sit at fourteen thousand requests a second, but it will spike toward the ceiling now and then, and that is exactly when you want the gateway to flatten out.
Now everyone hits at once
Here is the version that matches the morning-standup moment: thousands of requests in flight at once, each holding a connection open while it waits. We held 5,000 open at once and kept them there for five minutes. This is the response time a developer actually feels, at p99.
Highflame answers the rush in 0.83 seconds, the quickest of the four, with Bifrost just behind at 0.99. Then the floor drops out. The same tool call, on the same hardware, takes about seven seconds through Portkey and seventeen through LiteLLM, and both start dropping requests: Portkey finished 38% of them, LiteLLM 67%. The rest never came back, which is a developer watching a spinner, giving up, and re-running the command.
The pattern comes down to what actually runs the request.
The two natively concurrent compiled gateways, Highflame on Rust and Bifrost on Go, serve thousands of connections at once and stay under a second even under the crush. The two built on an interpreted, largely single-threaded runtime fall off a cliff, into whole seconds, because one slow moment stalls everything queued behind it. We have a theory about why Portkey’s tail explodes the way it does, but we have not put a profiler on it, so we will let the number stand and save the diagnosis for another day.
What this buys your team
Put Highflame on the hot path and the rush your team makes every morning never turns into a support ticket. Tool calls come back in about the time it takes to notice they left, at the rush and at the ceiling both, so nobody sits on a spinner or re-runs a command to check whether the agent hung. And because it holds latency flat as load climbs, the gateway becomes one thing you do not have to revisit the next time headcount doubles.
Caveats
The load generator ran on a separate box from each gateway, the same way for all four, so the comparison between them is fair; whether that matches every detail of Maxim’s own rig, we cannot say, so treat our numbers against theirs with that in mind. And the response-time figures under the rush are what a person feels when a system is briefly overloaded, not a clean gateway-overhead number; the clean comparison is the sustained table up top. The next step is to run the same gateways under real-life conditions rather than a benchmark’s, and the full four-gateway breakdown lives on our benchmarks page.
Why this is the gateway to run
Speed is only half of it. Highflame does more than move bytes between a developer and a model: it inspects every LLM and MCP call inline, checks it against your policy, and only then forwards it. That inspection costs about 4 milliseconds on the median request, a fraction of a percent of a real one-to-two-second model turn. Inspection that costs a developer real time is inspection a team quietly switches off. Leading that tail-latency chart is what lets Highflame run that detection on the hot path without ever being the reason your developers wait.
The speed is the easy part to measure. What Highflame looks for in each of those calls is where it gets interesting. See what runs on the hot path.
Frequently asked questions
What is AI gateway load testing?
Driving many concurrent requests through the gateway and measuring latency and error rate at the peak, not the average. It tells you what users feel during the morning rush, not during a quiet afternoon.
Why does an AI gateway only slow down under concurrency?
Most of the day the traffic is light enough that any gateway keeps up. The differences show only in the seconds when everyone lands together, when a gateway that queues or blocks starts adding seconds of latency.
How many concurrent connections should an AI gateway handle?
Enough to cover your worst simultaneous burst, not your average load. In these tests we held 5,000 connections open at once; Highflame answered them in under a second while a Python-based gateway took 17 seconds and dropped a third of the calls.
What makes a gateway drop requests under load?
Usually a runtime that can’t keep enough work in flight: a global lock, a thread-per-request model, or unbounded queuing that eventually times out. The gateways that held up here kept latency flat as load climbed instead of collapsing.
