llama.cpp, vLLM, and Text Generation Inference: architecture, and when each fits

About this article: AI-assisted writing, reviewed against the cited sources. This is source research, not a lab validation of the procedures. Check the stated versions and your own recovery requirements before making changes.

These three projects all serve a language model over an HTTP API. They do not solve the same problem. llama.cpp is a C/C++ runtime that will run a quantised model on a CPU, a GPU, or a mix, including a model that does not fit in video memory. vLLM is a Python serving library built to keep many requests in flight on accelerators by paging the key-value cache. Text Generation Inference (TGI) is a Rust router in front of a Python model server, and its own repository now says it is in maintenance mode.

An AI writing system helped prepare this article for NetLeverage. Nothing was timed on a GPU. Where a project claims a speed-up, that claim is quoted as theirs. No NetLeverage product is described.

Why it matters

The expensive mistake is not picking a "slower" engine. It is picking an engine whose assumptions do not match the machine.

Our engineering inference is that a deployment whose weights exceed its configured memory budget needs a deliberate placement or offload plan; the failure mode depends on the engine and configuration. llama.cpp's stated purpose is hybrid CPU and GPU inference for that case. If you have several data-centre GPUs and many concurrent users, the limit is how the key-value cache is packed and how requests join a batch. That is the problem vLLM's paper and its current README both centre on. If you already run TGI, the operational question is no longer "is TGI fast". The README and the documentation index say the project accepts only minor fixes, and they name vLLM, SGLang, llama.cpp, and MLX as the engines they recommend instead.

A second mistake is copying a default from the wrong component. TGI's architecture page prints bare-router help with input and total limits of 1024 and 2048 tokens. Its launcher reference instead derives defaults from the model and available allocation: min(max_allocatable, max_position_embeddings) - 1 for input and min(max_allocatable, max_position_embeddings) for total tokens. Check the effective values reported by your deployment.

How it works

llama.cpp: one GGUF file, many backends, optional CPU

The project README states the goal as LLM and vision-language inference in plain C/C++ with minimal setup, on a wide range of hardware, locally and in the cloud. It is built on the ggml library. The same README lists integer quantisation at 1.5, 2, 3, 4, 5, 6, and 8 bits, custom CUDA kernels, AMD via HIP, Apple via NEON, Accelerate, and Metal, plus Vulkan, SYCL, and a CPU-plus-GPU path "to partially accelerate models larger than the total VRAM capacity". The backend table also names Huawei CANN, Intel OpenVINO (marked in progress), IBM zDNN, and others. Treat that table as a menu of builds, not as a promise that every model is fast on every row.

Weights are stored in GGUF. The ggml specification describes GGUF as the successor to GGML, GGMF, and GGJT: one file, metadata as key-value pairs rather than a fixed list of untyped fields, extensible without breaking older readers, and suitable for mmap. A model trained in another framework is converted into GGUF. The filename convention is human-oriented, not a strict parser. The spec's own examples include a size label, a version, and an encoding such as Q4_0. Shards use 00001-of-00009 style names. An mtp- prefix marks a multi-token-prediction sidecar for speculative decoding. The server README's Hugging Face flag defaults the quant to Q4_K_M when you pass a repository and do not name one, and falls back to the first file if that quant is absent. That default is a convenience, not a quality judgement published in the spec.

Loading modes on the server include mmap (the automatic choice unless a device cannot use it), mlock to pin RAM, and direct I/O. mlock is the switch that stops the operating system paging the weights out. It does not create GPU memory.

Multi-GPU behaviour is documented separately, and the split mode matters more than the GPU count:

--n-gpu-layers defaults to auto. all pushes as many layers as possible onto the GPUs; the rest stay on the CPU. --tensor-split 3,1 puts three quarters on the first device. Tensor mode does not auto-fit: you set context size yourself. If two GPUs are slower than one, the page says to suspect the link, try layer, and look for NVIDIA Collective Communications Library (NCCL) is unavailable, multi GPU performance will be suboptimal. Peer-to-peer copies stay off unless you set GGML_CUDA_P2P. The page warns that enabling them can crash or corrupt outputs, including when the IOMMU is on.

The HTTP server README describes continuous batching on by default, parallel slots (--parallel, default -1 for automatic), and a context size that defaults to 0, meaning the model's own context. Logical batch size defaults to 2048 and the physical micro-batch to 512. The listen address defaults to 127.0.0.1 port 9931. Routes include OpenAI-compatible chat, responses, and embeddings, and an Anthropic Messages shape. Speculative decoding exists and defaults to off. Built-in tools that can read files or run a shell are off, and the help text says not to enable them on an untrusted port.

The repository README's quick start says llama cli and llama serve. The multi-GPU page says llama-cli and llama-server. Check --help on the build you installed.

vLLM: a serving library whose first problem is the cache

The vLLM README describes a Python library for inference and serving, originally from UC Berkeley's Sky Computing Lab. The features it lists are the ones that matter for a shared GPU service: paged attention for key-value memory, continuous batching, chunked prefill, prefix caching, CUDA and HIP graphs, and parallelism of several kinds — tensor, pipeline, data, expert, and context. Quantisation on that list includes FP8, MXFP8/MXFP4, NVFP4, INT8, INT4, GPTQ, AWQ, GGUF, compressed-tensors, ModelOpt, and TorchAO. Attention backends named there include FlashAttention, FlashInfer, TRTLLM-GEN, FlashMLA, and Triton. The API surface it names is an OpenAI-compatible server, an Anthropic Messages API, and gRPC. It also claims support for more than 200 Hugging Face architectures, including mixture-of-experts and hybrid state-space models. That count is the project's, on the day the README was opened. It is not a certification that every architecture has the same kernels.

The 2023 paper, already the basis of the paging article in this series, is still the citation the README asks researchers to use. The idea that has not gone stale is the split of duties: iteration-level scheduling decides who runs this step, and paging decides where their cache blocks live so the server does not reserve a maximum length for every request. Current V1 documentation, opened for that other article, adds chunked prefill by default and recompute as the default preemption mode. Those statements describe the documented vLLM configuration; they do not establish which equivalent features another engine has.

Install, as the README states it, is uv pip install vllm or pip, with an optional vllm download-kernels step for precompiled kernels. The library expects Hugging Face-style checkpoints more naturally than a hand-converted GGUF, though GGUF is on the quantisation list. Hardware named in the README includes NVIDIA, AMD, Intel, and CPUs (x86, ARM, PowerPC), plus plugins for TPUs and other accelerators. CPU support being listed is not the same claim as llama.cpp's hybrid offload. vLLM's Engine Arguments reference documents --cpu-offload-gb, the GiB per GPU to keep in host memory. It says part of the model is loaded from CPU memory to GPU memory during each forward pass and requires a fast CPU-GPU interconnect. That differs from llama.cpp running leftover layers on the CPU. Compare the actual offload mechanism rather than concluding that vLLM has no host-memory option.

TGI: a Rust router, a Python server, and a maintenance banner

TGI's README calls it a Rust, Python, and gRPC server, used at Hugging Face for Hugging Chat, the Inference API, and Inference Endpoints. The same file, and the documentation index, open with a caution: the project is in maintenance mode; pull requests are for minor bug fixes, documentation, and light maintenance; new work should go to vLLM, SGLang, llama.cpp, or MLX. Read that before you adopt it for a new platform.

The architecture page splits the process into three parts:

• Router (text-generation-router), a Rust web server. It accepts a custom HTTP API and an OpenAI-compatible chat API, queues requests, and builds batches.
• Launcher, which starts the router and one or more model servers.
• Model server, Python, loading a model and serving gRPC. Shards on several GPUs synchronise with NCCL or the equivalent. The router and the model server may be different machines.

The router README defines continuous batching as running several queries in one forward step and removing a query when it finishes, rather than waiting for a client to assemble a static batch. Prefill builds the cache for the prompt. Decode then adds one token. A new request cannot join the decode group until its prefill has run, so the router has to insert prefill work without stalling every decode. The README illustrates the trade-off as "a 2x latency slowdown for 10x more users". That sentence is an illustration in the router document, not a measurement on a named model.

The table below reproduces defaults printed in the architecture page's bare-router help, not the effective defaults of every launcher deployment:

The launcher reference defines --waiting-served-ratio as the waiting-to-running query ratio at which it considers admitting waiting queries into a batch. Its example uses 1.2: twelve waiting queries against ten running. It lists a default of 0.3, whereas the architecture page's router help prints 1.2. The launcher also derives the input and total limits from model capacity, and defaults the prefill budget to max_input_tokens + 50. These are different component descriptions; inspect the values your installed release actually starts with.

gRPC between router and server has schemas v2 and v3. The architecture page says they are almost the same except that v3 adds input chunks (text and image) and paged attention. The model server's serve quantisation enum on that page is bitsandbytes, bitsandbytes-nf4, bitsandbytes-fp4, gptq, awq, eetq, exl2, and fp8. The README also names Marlin. --speculate takes an integer; the README claims "speculation ~2x latency". That is their claim. It is not reproduced here. The same page spells the quantise subcommand's package as qptq while the option list says gptq. Treat that spelling split as a documentation defect to check in the tree you build, not as two different algorithms.

The README's Docker example is ghcr.io/huggingface/text-generation-inference:3.3.5, with --shm-size 1g, because NCCL may fall back to host shared memory when peer-to-peer is unavailable. In Kubernetes the equivalent is an emptyDir of medium Memory mounted on /dev/shm. NCCL_SHM_DISABLE=1 turns that path off and, the README says, hurts performance. The container listens on port 80 in that example, mapped to 8080 on the host. That is the launcher image, not the router binary's default of port 3000. NVIDIA drivers with CUDA 12.2 or higher are what the README recommends. It says CPU is not the intended platform. AMD support called out there is Instinct MI210 and MI250, via the -rocm image tag. Other backends (ROCm details, Intel GPU, Gaudi on a fork, Neuron, TPU via Optimum) do not all expose the same features. The page says so.

Safetensors is the weight loader the README names. Flash Attention and Paged Attention are the kernels it names, the latter by reference to vLLM. TGI's high-throughput path is, by its own description, a consumer of that paging idea behind a different router.

What to choose and why

SGLang and MLX appear only as names on TGI's banner. Their internals were not opened for this article, so they are not compared.

No tokens-per-second ranking is given. The pages opened here do not share one benchmark, and a number from an unrelated blog would not survive a change of model, quantisation, or batch size.

Risks and limits

• Binary names in llama.cpp moved between documents. llama serve and llama-server both appear in official pages opened on 10 November 2026. Copy the help text from the binary, not from memory.
• --split-mode tensor is experimental. It refuses quantised key-value cache, refuses flash attention off, disables auto-fit, and refuses a published list of architectures. GGML_CUDA_P2P=1 can crash or corrupt output. The warning text is in the multi-GPU page.
• Continuous batching is on by default in llama.cpp's server and is the point of TGI's router and of vLLM. The inference to test is that disabling it changes how requests share iterations. Compare latency and throughput under the actual workload before changing a busy deployment.
• TGI's architecture-page router help and launcher reference give different defaults. The former prints 1024 and 2048; the latter derives limits from model capacity. Verify the effective value before attributing a rejected request to the model. The Docker example tag 3.3.5 is the tag in the README that day, not a promise about later tags.
• TGI's "~2x" speculation line and the router's "2x latency for 10x users" line are project illustrations. They are not results from this draft.
• Shared memory of 1 GB in the TGI container is a NCCL requirement in their note, not a model-weight cache. Disabling it changes performance. It does not change the token cap.
• vLLM's "200+ architectures" and "state-of-the-art throughput" are README claims. Throughput still depends on the cache budget, the parallel mode, and the model. Paging does not make a model that does not fit suddenly fit, unless you also change quantisation or the parallel split.
• llama.cpp tools that execute shell commands are documented as unsafe in untrusted environments. Leave them off on any port that is not strictly local.
• No command in this article was run. Docker lines, install lines, and flags are quotations of project documentation.
• No NetLeverage product behaviour is claimed.

Sources

1. ggml-org llama.cpp README — https://github.com/ggml-org/llama.cpp/blob/master/README.md (raw file opened 10 November 2026). Goals, quantisation widths, backends, hybrid CPU/GPU, llama cli / llama serve.
2. llama.cpp server README — https://github.com/ggml-org/llama.cpp/blob/master/tools/server/README.md (raw file opened 10 November 2026). Continuous batching default, slots, context, batch sizes, bind address, port 9931, OpenAI and Anthropic routes.
3. llama.cpp multi-GPU guide — https://github.com/ggml-org/llama.cpp/blob/master/docs/multi-gpu.md (raw file opened 10 November 2026). Split modes, NCCL warning, P2P warning, tensor-mode architecture exclusions.
4. ggml GGUF specification — https://github.com/ggml-org/ggml/blob/master/docs/gguf.md (raw file opened 10 November 2026). Single-file layout, metadata keys, mmap, naming, shards.
5. vLLM README — https://github.com/vllm-project/vllm/blob/main/README.md (raw file opened 10 November 2026). Feature list, parallelism, quantisation, APIs. Paging mechanics remain Kwon et al., arXiv:2309.06180, opened for the companion article.
6. Hugging Face, Text Generation Inference README — https://github.com/huggingface/text-generation-inference/blob/main/README.md (raw file opened 10 November 2026). Maintenance-mode caution, Docker tag 3.3.5, shared memory, quantisation list, CUDA 12.2 note.
7. Hugging Face, TGI architecture — https://huggingface.co/docs/text-generation-inference/architecture (opened 10 November 2026). Router, launcher, model server, gRPC v2/v3, published router defaults.

8. TGI router README — https://github.com/huggingface/text-generation-inference/blob/main/router/README.md (raw file opened 10 November 2026). Continuous batching definition and the latency illustration.

9. Hugging Face, TGI launcher reference, reopened 10 November 2026. Model-derived token caps, prefill budget and waiting/served ratio.

10. vLLM, Engine Arguments, reopened 10 November 2026. CPU offloading mechanism and interconnect requirement.

This article was originally published on 10 November 2026.