Latency in LLM serving: time to first token, tokens per second, and the queue

About this article: AI-assisted writing, reviewed against the cited sources. This is source research, not a lab validation of the procedures. Check the stated versions and your own recovery requirements before making changes.

A language-model reply has three clocks, and they fail for different reasons. The queue clock starts when the request arrives and stops when the server finally schedules it. The prefill clock is the work to read the prompt and write its cache; the user feels it as time to first token. The decode clock is the gap between later tokens. "Tokens per second" divides some of those clocks by a token count. Which clocks, and which tokens, changes the number by more than any kernel tweak.

An AI writing system produced this article for NetLeverage's editorial studio. Definitions are from the documents opened on 10 November 2026. The sample milliseconds in one vendor's own worked example are quoted as theirs. No load test was run for this article. No NetLeverage service level is stated.

Why it matters

A dashboard with one latency number will start an argument and then the wrong fix. If users wait before the first word, the prompt is long, the queue is deep, or both. If the first word is prompt and the rest stutter, decode is being interrupted by someone else's prompt. If the average looks fine and the tail does not, a few long prompts or a few preemptions are sitting inside the average.

Kwon et al. plot what happens when the arrival rate passes the rate the server can finish. Latency rises slowly, then jumps, because the queue length grows without a bound. Past that point, buying a faster attention kernel does not restore a steady wait. You have to shed load, shorten replies, or add capacity that actually raises the completion rate. Orca's paper is the earlier statement of the same queue: if the server holds a batch until every request in it finishes, a new arrival waits for the whole batch, and a short request waits for a long one. Iteration-level scheduling, which operators call continuous batching, lets a finished request leave and a new one join after a single step.

How it works

Three intervals, and the names that do not match

vLLM's benchmark client, vllm bench serve, defines the clocks from the client's point of view:

• Time to first token (TTFT) is the time from sending the request to receiving the first streamed output.
• Inter-token latency (ITL) is the gap between consecutive streamed outputs. The statistics pool those gaps across successful requests.
• Time per output token (TPOT) is computed once per request, then aggregated:

With ordinary decoding, each stream event is usually one token, so ITL and TPOT are close. With speculative decoding, one event can carry several accepted draft tokens. ITL then records only the gaps between events. It does not insert zero-length gaps for the extra tokens inside an event. TPOT still divides the decode time by every output token. The benchmark page's own example: two gaps of 40 ms, and a second event that contains three tokens. Mean ITL is 40 ms. If TTFT was 100 ms and the request ended at 180 ms, with five output tokens, TPOT is $(180-100)/(5-1)=20$ ms per token. Quoting "40 ms between tokens" and "20 ms per token" from the same run is not a contradiction. It is two definitions.

The Prometheus metrics use the same split, and then add a trap. vllm:inter_token_latency_seconds is one sample per streamed output. vllm:request_time_per_output_token_seconds is the TPOT formula, once per finished request. Requests that generate one token or none are recorded with TPOT of zero in the histogram. The benchmark excludes those requests from its TPOT statistics. The design page says the two averages can therefore disagree. Use the histogram when you want the request-level formula, and the inter-token histogram when you want the gaps a streaming client actually saw.

The server-side TTFT clock is not the client's send time. The metrics design says TTFT for a request that finished prefill is measured from arrival_time, and that arrival_time currently starts when tokenisation begins. Queue time is a separate series, vllm:request_queue_time_seconds, taken for requests that were scheduled in that iteration. End-to-end latency runs from arrival_time to the frontend seeing the final token. A client-side TTFT also includes the network. A server-side TTFT as defined here includes tokenisation and whatever waited before prefill finished, because the clock started at tokenisation. Do not subtract one from the other and call the difference "the network" unless you have matched the clocks.

Prefill and decode have separate histograms. The design page defines prefill from the most recent SCHEDULED event to its first NEW_TOKENS event, and decode from that first NEW_TOKENS to the last NEW_TOKENS. It does not define the separate “generation time” metric sometimes assumed from that name. Check event boundaries and preemption treatment before adding intervals.

What moves each clock

Queue time moves when work arrives faster than it is scheduled, or when admission rules hold it. The metrics page exposes vllm:num_requests_running, vllm:num_requests_waiting, plus vllm:admission_rejections_total for attempts a queue limit refused. A rising waiting gauge with a flat running gauge suggests an admission or execution bottleneck; it does not identify the cause alone. The design page also lists a swapped gauge in its older metric inventory, while V1 defaults to recomputation rather than swapping. Kwon et al. use normalised latency, each request's end-to-end time divided by its output length, averaged. That number stays calm and then explodes in their plots once the offered rate exceeds capacity. It is a stability signal. It is a poor decode measurement, because a long prompt and a short answer inflate the per-output-token figure even when the decode step itself is unchanged.

TTFT moves with the prompt and with whatever else is in the forward pass before the first token. Prefill is the compute-heavy pass: every prompt token is scored, and the cache is written. The V1 tuning guide schedules decodes first, then spends leftover room in max_num_batched_tokens on prefill, splitting a prompt that does not fit. Their tuning notes, which are recommendations:

• A smaller budget, their example is 2048, leaves less prefill in each step, which helps inter-token latency.
• A larger budget improves time to first token, because more of the prompt is processed per step.
• They suggest a budget above 8192 when the aim is throughput, especially for smaller models on large GPUs.

So the same knob that makes the first token sooner makes the gaps between later tokens worse, if a long prefill shares the step with decodes. Holmes et al. describe the older shape of that fight. In the systems they discuss, including the vLLM of that paper, generation is preempted so a prompt can be processed, or a whole prompt is added beside generation. A long prompt then stalls everyone else's next token. Their Dynamic SplitFuse cuts long prompts into chunks and composes short prompts up to a token budget, so forward passes stay near one size. They report, against the vLLM they tested, up to 2.3× higher effective throughput, 2× lower average latency, and up to 3.7× lower token-level tail latency. The paper is arXiv version 1, January 2024. Its evaluation used Llama 2 7B, 13B, and 70B with synthetic input/output lengths; it does not identify the compared vLLM version. Those factors are not a ranking of current vLLM, which now documents chunked prefill as the default when the build allows it. The mechanism they argue is still the right question: a forward pass is governed by how many tokens it contains, not by how many requests those tokens belong to. Below saturation, throughput scales with token count because the limit is reading the weights. Past saturation, more tokens do not raise throughput. Splitting a pool of tokens evenly across passes is, on their concave curve, better than one huge pass and one tiny one.

ITL and TPOT move with the decode step: reading weights and the growing cache for one new token, plus any prefill that was packed into the same step, plus preemption. The metrics design says a preemption during decode is charged to the inter-token, decode, and inference intervals, because already generated tokens are kept and the gap the user sees includes the recompute wait. A preemption during prefill, if that event happens, is charged to TTFT and to prefill time. The V1 default preemption mode is recompute. A successful request can still have an ugly tail.

Tokens per second is a quotient. The benchmark prints three different ones: requests per second, output-token throughput, and total-token throughput. When interpreting the latter as prompt plus generated tokens over the benchmark window, confirm that definition in the implementation used; the cited CLI overview alone does not spell out that formula. Output-token throughput ignores how expensive the prompts were. Total-token throughput flatters a run of long prompts that generated little. A per-request "tokens per second" that divides output tokens by the whole interval from schedule to last token includes prefill in the denominator, so it is slower than $1/\mathrm{T}\mathrm{P}\mathrm{O}\mathrm{T}$. If TPOT is 10 ms, the decode phase is about 100 output tokens per second for that request. That sentence is the definition, not a measurement. It excludes the first token, and it is not the server's aggregate throughput.

Reasoning models add one more distortion, documented on the benchmark page. Reasoning deltas count toward TTFT and ITL, because the server is already decoding when it emits them. If you disable reasoning in the client stream, the server can still generate those tokens and emit nothing, so the whole reasoning phase is absorbed into TTFT. The first "answer" token then looks late. The model was not slower. The metric clock was pointed at a different event.

A queue that does not settle

While the completion rate stays above the arrival rate, waiting time fluctuates and has a typical size. When arrivals stay above completions, the waiting line lengthens every minute, and the wait for a new request lengthens with it. There is then no steady latency to optimise. Kwon et al. show that shape on ShareGPT and Alpaca traces with Poisson arrivals: the curve is gradual, then vertical. Orca's iteration-level rule cuts the avoidable part of the wait, the part caused by refusing to reschedule until a batch ends. It does not create capacity. A server that can finish 5 requests a second will not hold a steady queue at 8 arrivals a second, however fine the scheduler is.

Admission rejections are the alternative to an infinite queue. They show up as vllm:admission_rejections_total rather than as a slow TTFT. A client that retries immediately turns a rejection back into load. Count rejections beside waiting requests before you decide the model is the bottleneck.

Cache exhaustion is a second queue, disguised as latency. When blocks run out, V1 recomputes a preempted request. That request returns to waiting, prefills again, and its TTFT or its later gaps include the lost work. vllm:kv_cache_usage_perc near 1, together with preemption warnings, is the signature. Raising a batch limit does not add cache capacity. Its effect on queueing depends on the scheduler and workload.

What to choose and why

The positive signature of a healthy step is a token budget that stays in the saturated region of Holmes et al.'s curve without a single prompt occupying the whole pass. The negative signature is num_requests_waiting climbing while kv_cache_usage_perc sits at 1, or a TTFT histogram whose upper buckets fill only when long prompts arrive.

Risks and limits

• TTFT from the benchmark client, TTFT from arrival_time, and queue time are three clocks. arrival_time starts at tokenisation in the design note opened that day. A later build can move that instant. Read the page for the version you run.
• Prometheus TPOT and benchmark TPOT do not share a population. One-token requests are zero in the histogram and absent from the benchmark statistic.
• Normalised latency in the vLLM paper divides end-to-end time by output length. It is the right plot for "did the queue explode". It is the wrong plot for "how fast is decode".
• The DeepSpeed-FastGen factors are that paper's, against the vLLM of early 2024, up to 2.3×, 2×, and 3.7× on the metrics they name. Current vLLM's chunked prefill is a later scheduler. Do not subtract the factors from a purchase order.
• Chunked prefill's budget advice is the project's recommendation, not a measurement repeated here. Disabling chunked prefill with a budget below the model length can crash startup. That warning is on the tuning page.
• A reasoning stream that is hidden inflates TTFT even though those tokens were decoded. Fix the metric before you scale the cluster.
• No command was run. Sample timings on the benchmark page are documentation. No NetLeverage latency target is claimed.

Sources

1. vLLM, Metrics — https://docs.vllm.ai/en/stable/design/metrics/ (opened 10 November 2026). Histogram names, TPOT formula, ITL versus TPOT under speculative decoding, arrival_time at tokenisation, queue time, preemption's effect on the intervals, waiting and running gauges.
2. vLLM, Benchmark CLI — https://docs.vllm.ai/en/stable/benchmarking/cli/ (opened 10 November 2026). Client-side TTFT, ITL, and TPOT, the 40 ms / 20 ms example, output versus total token throughput, reasoning deltas absorbed into TTFT.
3. vLLM, Optimization and Tuning — https://docs.vllm.ai/en/stable/configuration/optimization/ (opened 10 November 2026). Decode-first chunked prefill and the token-budget trade-off between ITL and TTFT.
4. Woosuk Kwon et al., arXiv:2309.06180 — https://arxiv.org/html/2309.06180 (opened 10 November 2026). Normalised latency and the unbounded queue once arrival rate exceeds capacity.
5. Gyeong-In Yu et al., Orca, OSDI 2022 — https://www.usenix.org/system/files/osdi22-yu.pdf (opened 10 November 2026). Request-level batching delay versus iteration-level scheduling.
6. Connor Holmes et al., DeepSpeed-FastGen, arXiv:2401.08671v1 — https://arxiv.org/html/2401.08671 (opened 10 November 2026). Forward-pass cost versus token count, Dynamic SplitFuse, and the reported factors against the vLLM they tested.

This article was originally published on 10 November 2026.