About this article: AI-assisted writing, reviewed against the cited sources. This is source research, not a lab validation of the procedures. Check the stated versions and your own recovery requirements before making changes.
A GPU does not sit in a flat pool of cores. It is a PCI device on one path to one set of CPUs and one set of memory. Host memory allocated on the other socket, and a network card on the other socket, cross the link between sockets on every transfer that the GPU itself does not keep on NVLink. Pinning is the way to stop the kernel moving the process away from the memory it already faulted in. Affinity or a new task memory policy does not itself move pages faulted earlier, although automatic NUMA balancing or explicit migration can, and a cgroup that hides the GPU's CPUs makes the affinity column useless.
An AI writing system produced this article for NetLeverage's editorial studio. The commands below are quoted from the manuals opened on 10 November 2026. None was run on a GPU server. No NetLeverage platform is described. No bandwidth figure below is a measurement from a card.
Why it matters
Loading weights and staging inputs or network data can move bytes between host memory and a GPU. KV-cache transfers depend on whether the serving system actually offloads or exchanges its cache; they are not a universal host-copy cost of inference. Those copies use the path the platform actually wired. Two GPUs that only meet by crossing the inter-socket link are a different machine from two GPUs behind one PCIe switch, even when both pairs show up as "GPU 0" and "GPU 1". A later pin can leave existing buffers on a remote node: a new task policy governs later allocations. Automatic NUMA balancing, if enabled and applicable, or explicit migration can change placement later.
The failure is quiet. The model loads. Tokens arrive. Throughput is lower than the same binary on a different socket, and nothing in the serving log names the socket.
How it works
What Linux calls a node
The kernel's NUMA note, started in 1999 and still the current "What is NUMA?" page, splits the hardware view from the software view. Hardware is cells: CPUs, local memory, and I/O buses, joined by a crossbar or a point-to-point link. Access time and bandwidth depend on how far the CPU or the I/O bus is from the memory. Linux maps those cells onto software nodes. On x86, a cell with no memory is hidden, and its CPUs are reassigned to a node that does have memory. Those CPUs then do not all see the same local latency. Default allocation is local: the kernel tries the node of the CPU that is running the allocation, and falls back along a distance-ordered list if that node cannot satisfy it. Local stays local only while the task stays on that node. The scheduler tries not to migrate a task into a distant domain, but it does not account for the task's NUMA footprint directly, so imbalance can still move it.
Memory policy is a separate mechanism from that default, documented on the kernel's NUMA memory-policy page. The system default, once the machine is up, is local allocation. A task policy covers later allocations made by or for that thread unless a more specific policy applies and is inherited across fork and exec, which is why a launcher can set policy for a program that knows nothing about NUMA. It does not cover sibling threads that already existed. Installing the task policy does not itself migrate pages already faulted. A VMA policy, set with mbind, can migrate pages; the same page says that support dates from Linux 2.6.16. Cpusets win: when a cpuset and a memory policy both apply, the cpuset's allowed nodes take priority.
Modes that matter for a GPU process:
| Mode | What the kernel does | What fails |
|---|---|---|
| Default / local | Allocate on the node of the CPU doing the fault, then fall back by distance | A thread that starts on the wrong CPU faults the buffers there |
Bind (MPOL_BIND, numactl --membind) |
Memory must come from the listed nodes, choosing the closest of those with free memory | The allocation fails when those nodes are short, even if another node is empty |
Preferred (--preferred) |
Try one node, then other nodes in distance order | The job runs, on remote memory, with no error |
| Interleave | Spread pages across the listed nodes | A thread on one node then reads remote pages on purpose |
numactl sets the policy and then runs the command. Children inherit it. --cpunodebind limits execution to the CPUs of a node. --physcpubind limits it to CPU numbers as listed in the processor fields of /proc/cpuinfo. --localalloc tries the process's current node and falls back. --preferred is the soft one-node preference. --membind is the strict one. A node can be named as pci:[seg:]bus:dev[:func] or netdev:DEV, so the bind can follow the GPU's PCI address or the network device instead of a hard-coded node number. --show reports the policy of that numactl process, not of some other already-running server. The manual's page, as rendered, was taken from the numactl git tree on 4 August 2026.
What the GPU tool actually prints
NVIDIA's nvidia-smi manual says topology output shows every GPU the management library can see, but CPU and NUMA affinity only for Kepler and newer. The same page warns that nvidia-smi text output is not guaranteed to stay compatible across drivers. The legend below is the one on the page opened on 10 November 2026.
nvidia-smi topo -m prints a matrix of GPUs and NICs, plus CPU and memory affinity. On some machines a NIC is only a bridge for NVLink switches; the command tries to filter those ports, and on some kernels it needs root to read the sysfs entries it uses. The legend:
| Code | The path |
|---|---|
| X | The device to itself |
| PIX | A single PCIe switch |
| PXB | More than one PCIe switch, without the PCIe host bridge |
| PHB | PCIe and a PCIe host bridge, typically the CPU |
| NODE | PCIe and the link between PCIe host bridges inside one NUMA node |
| SYS | PCIe and the SMP interconnect between NUMA nodes (the manual's examples are QPI and UPI) |
| NV# | A bonded set of that many NVLinks |
NV2 means two NVLinks in that bonded set, not "twice as fast as PCIe". The page does not give a gigabyte-per-second figure for any NV# code. nvidia-smi topo -mp drops NVLink and shows the PCI path only. nvidia-smi topo -p2p is a different matrix: read, write, NVLink, atomics, or PCIe peer-to-peer status, not the path class. nvidia-smi topo -C is the NUMA id of the nearest CPU. nvidia-smi topo -M is the NUMA id of the nearest memory. nvidia-smi topo -gnid is the GPU's own NUMA id when the platform exposes GPU memory as a node, and N/A when it does not. Those three are not the same number on every machine, which matches the kernel's warning that x86 may hide a memoryless cell and park its CPUs elsewhere.
nvidia-smi topo -cpu adds columns for memory affinity, GPU NUMA id, and CPU affinity. If the current cgroup's CPUs do not overlap the GPU's affinity, the column is N/A. A container that was given a random slice of cores will print that, and a pin inside the container cannot see the cores the GPU is nearest to.
The kernel's PCI sysfs page lists local_cpus on the device directory as the nearby CPU mask. That is the kernel's own hint for the same idea, expressed as a cpumask rather than as PIX or SYS.
NVLink has its own controls, and they are not the topology code. Setting the NVLink bandwidth mode to OFF means peer-to-peer uses the PCIe bus. Both the set and get bandwidth-mode interfaces are deprecated for Blackwell and later, not just selected mode values. The data-rate query accounts for forward-error-correction overhead. This article does not turn those modes into a throughput claim.
What a PCIe generation number is, and is not
A generation identifies signalling technology; negotiated width identifies how many lanes are active. A link can train narrower than the connector suggests, and practical transfer rates include encoding, protocol and software costs. The PCI-SIG FAQ could not be independently reopened during this review, so its numerical rate table is not reproduced here.
The kernel's pcie_link_bandwidth_mbps uses the negotiated lane count and, in the kernel's words, does not account for encoding overhead. A negotiated width of zero yields zero. At the same per-lane generation rate, a slot marketed as x16 that trains as x8 has half the nominal lane aggregate. That arithmetic does not establish measured application bandwidth. This article did not read a negotiated width on a machine. A generation label without negotiated lane count and encoding/payload accounting is not an observed bandwidth.
What to choose and why
Read the matrix before choosing a bind. The useful pair is nvidia-smi topo -m for the path class between GPUs and NICs, and nvidia-smi topo -M plus nvidia-smi topo -cpu for the memory node and the CPU list. If -gnid is N/A, GPU memory is not a Linux node on that platform. Bind to the nearest memory node, not to an imagined GPU node.
| What you see | Placement |
|---|---|
| One process, one GPU, cgroup overlaps the affinity | Start the process with --cpunodebind and --membind on the node from topo -M, or name the GPU with pci:. Do this at start, not after the buffers exist |
-cpu prints N/A |
The cgroup does not contain the GPU's CPUs. Fix the cgroup before pinning inside it. Cpuset restrictions override a memory policy |
Two GPUs marked NV# |
Their peer path is NVLink, not the CPU. Host staging for each process still belongs on that GPU's memory node |
Two GPUs marked SYS |
The path crosses the inter-socket link. Do not treat them as neighbours for a job that stages through host memory |
GPU and the request NIC marked SYS |
The receive path and the GPU are on opposite sides of the interconnect. netdev: binds a process to the NIC's node; that is the documented selector, not a measured saving |
| Node can run out of memory | --membind fails the allocation. --preferred runs on remote memory instead. Prefer the failure when a silent remote copy is worse than a refused start |
Pinning still matters when the platform exposes more than one node, because local allocation follows the CPU that faults the page. A second node can exist on one processor. The kernel's heterogeneous-memory note was not used to invent a bandwidth for a GPU's own memory.
The check, without a benchmark, is local. After the process starts under the policy, nvidia-smi topo -cpu from a shell in the same cgroup shows a real CPU list rather than N/A, and the memory policy was set before the first large allocation. A later numactl --show in a different shell does not describe the server. Nothing in this article was checked that way on hardware.
Risks and limits
- A pin does not relocate pages already faulted. The memory-policy document says a task policy applies to allocations after it is installed. Starting the server, then binding it, leaves the weights and the staging buffers where the first touch put them, unless another mechanism moves them.
mbindcan request migration, and automatic NUMA balancing may migrate eligible pages toward nodes that access them frequently. Themigratepagesmanual was not opened, so this article does not give its flags. --membindcan fail a start that--preferredwould have slowed down. That is the documented difference. Pick it on purpose.- x86 may split "nearest CPU" from "nearest memory". Use
-Cand-Mseparately. Do not assume one node number covers both. NV#is a link count, not a benchmark. NVLink OFF means peer traffic falls back to PCIe. No NVLink gigabyte rate was taken from the manual.- Raw PCIe rates are not payload rates. The kernel helper explicitly excludes encoding overhead; application throughput has additional protocol and software costs.
nvidia-smitext can change between drivers. The legend is the page as opened, not a promise about an older driver on a host.- No command was run. No NetLeverage server, BIOS setting, or IRQ affinity was changed. Interrupt affinity was not opened as a separate document and is not prescribed here.
Sources
- NVIDIA, nvidia-smi manual — https://docs.nvidia.com/deploy/nvidia-smi/ (opened 10 November 2026). Topology legend,
-m,-mp,-C,-M,-gnid,-cpuand the cgroupN/Arule, NVLink bandwidth mode OFF, Blackwell deprecation, data-rate query and FEC. Page date unavailable. - Linux kernel, What is NUMA? — https://docs.kernel.org/mm/numa.html (opened 10 November 2026). Cells, x86 memoryless nodes, local allocation, scheduler limit, taskset and numactl as the affinity tools.
- Linux kernel, NUMA Memory Policy — https://docs.kernel.org/admin-guide/mm/numa_memory_policy.html (opened 10 November 2026). Scopes, bind, preferred, interleave, cpuset priority, policy applies only to later faults,
mbindmigration since 2.6.16. numactl(8), man7.org, page obtained 4 August 2026 from the upstream tree — https://man7.org/linux/man-pages/man8/numactl.8.html. Inheritance,--membind,--preferred,--cpunodebind,--physcpubind,pci:andnetdev:node names.- Linux kernel, Accessing PCI device resources through sysfs — https://docs.kernel.org/PCI/sysfs-pci.html (opened 10 November 2026).
local_cpusis the nearby CPU mask. - Linux kernel, PCI support library — https://docs.kernel.org/driver-api/pci/pci.html (opened 10 November 2026).
pcie_link_bandwidth_mbpsuses negotiated lanes and does not account for encoding overhead. - Linux kernel, kernel sysctl documentation,
numa_balancing— https://docs.kernel.org/admin-guide/sysctl/kernel.html#numa-balancing (opened 10 November 2026). Automatic page-fault-based migration and its overhead.
This article was originally published on .