What is an H100
actually doing?

01

The spec sheet is not the ceiling#

Every roofline needs a denominator. Using the vendor's peak figures is the most common way to make a performance analysis quietly wrong, because those figures assume conditions the card will not hold. So the first measurement is of the machine itself.

Rooflining against 989 TFLOP/s overstates compute headroom by more than a fifth. The ridge point matters as much: at 258 FLOP per byte, a decode step needs arithmetic intensity above that to be compute bound, and single sequence decode is nowhere near it. That number sets up everything below.

02

Predict first, then measure. The gap is the finding#

At batch 1 a decode step is a weight streaming problem. The model reads every weight to produce one token, so the ceiling follows from the two numbers above.

Measured: 149.9 tok/s at 6.63 ms per token, which is 81.1 % of the bandwidth ceiling. The shortfall is about 1.25 ms per step that weight streaming does not explain. KV traffic cannot account for it either, since at batch 1 it is under 1 % of bytes moved.

vLLM holds 78 to 81 % of memory bandwidth from batch 1 to batch 128 while throughput rises 71-fold. Per-token latency degrades gently, 6.63 to 11.69 ms, which is the batching trade working as intended.

03

The residual is mostly kernel launches#

A residual is only interesting if you can attribute it. CUDA graphs exist to remove per-kernel launch overhead by replaying a captured graph, so disabling them should expose exactly that cost.

Running eager halves throughput at batch 1. So roughly 7.2 ms per step is launch overhead that graphs already remove, and what remains with graphs on is attention, sampling and scheduling. The penalty shrinks to 34 % at concurrency 64, because fixed per-step costs amortise across more tokens. That is also why small models suffer most from them.

04

A small model is not a fast model, it is an idle GPU#

Comparing all five models at batch 1, where decode dominates elapsed time and the roofline applies cleanly:

Qwen3-0.6B produces the highest token rate of any model here and the worst utilisation. Its weights are only 1.4 GiB, so streaming them takes less time than the fixed per-step cost measured in section three. The roofline therefore under-predicts it: the model is not bandwidth bound, it is overhead bound.

The practical reading is that a high tokens-per-second figure on a small model is not evidence the hardware is being used. It is often evidence a cheaper card would do the same work.

05

A mixture of experts gets less efficient as you load it#

A mixture of experts reads only the experts its tokens route to. At batch 1, DeepSeek-V2-Lite touches 6 of 64. The subtlety is that at batch B different tokens route to different experts, so the union grows. Under uniform routing the expected number of distinct experts touched is E · (1 - (1 - k/E)B).

Effective weight traffic grows roughly sixfold from batch 1 to batch 64. A dense model gets batching amortisation for free, because one weight read serves every sequence in the batch. A mixture of experts does not: past a certain batch size you are reading the whole model again and the sparsity advantage is gone. Any capacity plan that prices a MoE on its active parameter count is describing the small batch case only.

06

MLA moves a different ceiling than the one people quote#

DeepSeek-V2-Lite uses multi-head latent attention, storing a compressed latent per token instead of full keys and values per KV head. It uses 4.7× less KV per token than Qwen3-8B, 30.4 MiB per 1k tokens against 144.

It is worth being precise about what that buys, because it is routinely overstated. It does not make decode faster at low concurrency, where weight streaming dominates and KV is around 1 % of bytes moved. It raises how many long sequences fit at once, and delays the point where KV traffic crowds out weight traffic. Those are ceilings on scale, not on latency.

07

The benchmark will measure its own cache if you let it#

vLLM's random dataset is seeded deterministically, so repeated invocations send the same prompts. The usual way to build a latency curve is one server and a ladder of benchmark runs at rising concurrency, and prefix caching is on by default. Every point after the first therefore re-sends prompts whose KV blocks are already resident.

Two arms on the same card, same model, request count fixed at 256 per point, prefix caching enabled in both. The only difference is whether the cache is reset between points:

From the second point onward the ladder runs at a 98.4 % prefix cache hit rate and reports up to 86 % more throughput than the same server and workload measured cold. A second pass reproduces it. Concurrency 1 is unaffected, which is what the roofline predicts: at batch 1 decode is weight streaming bound, so a KV hit buys almost nothing.

This is not a small correction to apply afterwards. It is larger than most effects anyone would be trying to measure, and it grows with concurrency, so it distorts the shape of a scaling curve and not just its height. Reported upstream as vllm#52884. vLLM's own bench sweep serve already resets caches between runs, so the supported sweep path is not affected; the exposure is hand-rolled ladders built from repeated bench serve calls.

08

Three knobs, and what they actually do#

Each of these was predicted before it was run. All three are measured with the prefix cache flushed before every run, a fixed request count, and three repeats per point.

FP8 KV cache buys capacity, and a little speed

FP8 KV is neutral at low concurrency and 4 to 6 % faster at high concurrency, in both a decode-heavy and a prefill-heavy shape, while KV capacity nearly doubles from 407,248 to 790,272 tokens on the same FLASH_ATTN backend. Baseline run-to-run spread was 0.1 to 2.8 %, so the gains at scale are outside the noise and the small negatives at batch 1 and 8 are not.

The shape of that curve follows from where the bytes are. At batch 1 the KV cache is about 1 % of bytes moved, so halving it changes nothing and the conversion work is a small net cost. By concurrency 128 KV is a large share of traffic and the saving shows up. This is also relevant to vLLM #48786, which reports FP8 KV being slower on Hopper and traces it to the FlashAttention 3 FP8 prefill kernel at head_dim=256. Qwen3-8B is head_dim=128 and shows no penalty in either shape, which supports treating that as specific to the 256 path rather than general to FP8 KV on this hardware.

The prefill budget trades latency, it does not add throughput

With chunked prefill, prefill chunks and decode steps compete for one token budget, so a larger max_num_batched_tokens should relieve decode under load. At concurrency 128 it does nothing to throughput, while moving latency substantially in both directions:

Throughput is flat to within run-to-run noise. The knob buys inter-token latency at the cost of first-token latency. A deployment should set it from which of those it cares about, not in the hope of more tokens per second.

Prefix caching, and what a null result does not mean

Disabling prefix caching on this workload changes throughput by +0.2 %, against 1.2 % run-to-run spread over three repeats. That is a null result, and it is a fact about the workload rather than the feature.

The benchmark's random dataset gives every request its own random vocabulary offset and a shared prefix length of zero, so no two prompts share a prefix and there is nothing for the cache to hit. A cache returning no hits on inputs with no reuse is behaving correctly. On traffic that repeats a prefix, which is most real serving: a shared system prompt, a multi-turn conversation, a few-shot template, a retrieved document reused across questions, the same feature is doing substantial work. This study does not measure that case, and this number is not evidence about it.

09

Method, and what would change the conclusions#

H100 80GB HBM3 SXM, driver 580.126.09, vLLM 0.27.1, PyTorch 2.13.0+cu130. Load generated with vllm bench serve on the random dataset, 8 warmup requests per point, EOS ignored so output length is fixed. Decode isolation uses a 128 token prompt and 1024 output tokens. Cross-model comparison is taken at batch 1, where decode dominates elapsed time. Controlled runs flush the prefix cache before every measurement and repeat each point three times.

Little's Law (L = λW) held within 3 % on every run in the study. It is free to compute and it catches mismatched units, uncounted requests and warmup bleeding into the measurement window. When it drifts, stop before interpreting anything else.

Limits worth stating

• One serving stack. Nothing here separates a hardware limit from a vLLM implementation limit.
• Clock locking was unavailable in the container, so results are best-of-N. Measured run-to-run spread was 1.2 % at fixed configuration.
• The MoE union model assumes uniform expert routing, which real routers violate.
• Three physical H100s were used across the study and differed by about 1 % on measured peaks. Every comparison on this page is same-card; that 1 % is the floor on reading across them.
• No kernel-level profiling was run. The launch overhead figure comes from removing CUDA graphs, which bounds the cost without identifying which kernels pay it.
• The five-model comparison is taken at batch 1, the first and therefore cold point of each ladder. Its concurrency behaviour is not reported, because those later points were measured before the cache controls in section seven were in place.
• No prefill numbers are reported. Prefill is compute bound in principle, since processing a whole prompt at once has arithmetic intensity above the ridge point, but the runs I have were taken after a 1024 token sweep on the same server. A short prompt is a strict prefix of a long one under this generator, so those runs were served largely from cache and any FLOPs utilisation derived from them would be inflated. Measuring it properly needs a fresh server per input length.

On every path measured here, vLLM runs close to what the hardware allows: 78 to 81 % of memory bandwidth on decode, 79 % of peak FLOPs on prefill. The interesting question in inference performance is usually not which engine is faster. It is which ceiling a given workload is against, since that determines whether more bandwidth, more compute, or a different model shape is what would actually help.