About this article: AI-assisted writing, reviewed against the cited sources. This is source research, not a lab validation of the procedures. Check the stated versions and your own recovery requirements before making changes.
A GPU training pod has two separate problems. The node has to admit a real device, and every member of the job has to be admitted together. Kubernetes answers the first with a device plugin or with dynamic resource allocation (DRA). With default feature settings it does not answer the second; native gang scheduling is now available as an opt-in Beta feature. A job that needs eight pods can bind three, hold three GPUs, and leave the rest pending forever.
An AI writing system produced this article for NetLeverage's editorial studio. "The scheduler waits for the group" means a plugin counted pods. It is not a claim about experience.
Nothing below was applied to a cluster. The scheduler snippet in the coscheduling README replaces queue sorting and sets leaderElect: false. That is a demonstration configuration. Loading it onto a production control plane changes how every pod is ordered.
Why it matters
Extended resources, which is how a device plugin publishes a GPU, are integers. The Kubernetes device-plugin page says they cannot be overcommitted, and that devices cannot be shared between containers. A request for nvidia.com/gpu: 1 is one advertised device entry; it may represent a MIG instance under the operator's single strategy, rather than a whole physical card. You do not get a fraction, and a second container in the same pod does not inherit it.
That model schedules one pod at a time. A distributed training job is not one pod. If three of eight pods bind, those three GPUs are busy and the other five stay Pending, because the remaining nodes no longer have a free device. The job makes no progress, and the GPUs it holds are not free for anyone else. Gang scheduling is the rule that the group starts together or not at all. The out-of-tree Coscheduling plugin is one implementation. Kubernetes v1.37 also has native gang scheduling, Beta and disabled by default, under the GenericWorkload feature gate. These are distinct APIs, not interchangeable labels.
DRA changes the device request. It does not, in the documentation opened here, add that group rule. It also cannot preempt. A higher-priority pod that needs a DRA device waits until the pod holding it exits.
Device plugins
The device plugin framework has been a stable Kubernetes feature since v1.26. Vendors ship a plugin, usually as a DaemonSet, instead of patching Kubernetes. The page lists GPUs, high-performance NICs, FPGAs and InfiniBand adapters as the sort of device this is for. The same page, lower down, says the device plugin API is not stable, and that kubelet upgrades are not guaranteed to be non-breaking for plugins. Both sentences are in the document dated 27 July 2026. Treat the framework as supported, and still check the plugin's API versions before you upgrade the node.
Registration is a gRPC call to the kubelet. The plugin must already be serving before it registers. The kubelet listens on the Unix socket /var/lib/kubelet/device-plugins/kubelet.sock. That directory is hardcoded. It does not follow the kubelet --root-dir. The plugin sends its own socket name, the Device Plugin API version it was built against, and a resource name of the form vendor-domain/resourcetype. The documentation's example for an NVIDIA GPU is nvidia.com/gpu.
After registration the plugin streams devices with ListAndWatch. The kubelet advertises healthy devices on the node status. A pod then requests the resource in a container limit, the same way it requests CPU, with those two differences: integer only, and no sharing between containers. Allocate runs at container creation. The plugin may reset or clean the device, then returns mounts, device nodes, environment variables, annotations, or CDI device names. CDI names in this path need the DevicePluginCDIDevices feature gate, which the page says went to GA in v1.31.
The plugin has to run privileged, because that socket directory requires it. A DaemonSet must mount /var/lib/kubelet/device-plugins into the pod. When the kubelet restarts it deletes the sockets in that directory. The plugin is expected to notice the deletion and register again. A plugin failing to re-register is one documented explanation for missing resources after restart; logs and plugin state are needed to distinguish it from other failures.
Health is a different counter from capacity. If the plugin marks a device unhealthy, the kubelet reduces allocatable for that resource. Capacity stays the same. Pods already assigned to the failed device stay assigned. If their restart policy is not Always they can go Failed; otherwise they crash-loop. Since v1.36, with ResourceHealthStatus beta and on by default, each container status can carry allocatedResourcesStatus, including a message such as an over-temperature report. Before v1.31 the page says you used the PodResources API to see which pod sat on the failed device.
NVIDIA's GPU Operator documentation, opened 10 November 2026, describes Multi-Instance GPU as a partition of Ampere-and-later GPUs, with the A100 given as the example, into separate GPU instances for CUDA. Those instances are what a plugin can advertise as more than one device. They are not the sharing the device-plugin page says Kubernetes refuses between containers. Partitioning the card and sharing one device entry are different mechanisms.
Dynamic resource allocation
DRA has been stable since Kubernetes v1.35, and was available from v1.30. The feature gate DynamicResourceAllocation is locked. Setting it is ignored, and Kubernetes does not report an error. The page was last modified on 27 July 2026.
The model is the one used for storage. A device class is the category, the way a StorageClass is a category. A ResourceClaim or a ResourceClaimTemplate is the request, the way a PersistentVolumeClaim is a request. Drivers publish ResourceSlices that tell the cluster which nodes have which devices, and they update those slices when capacity changes. The scheduler places the pod on a node that can reach the allocated device. The driver configures the device and attaches it with the Container Device Interface.
The DRA page lists what this adds, and it names the device-plugin limits in the same paragraph:
- CEL expressions can filter on device attributes, instead of only a resource count.
- Several pods or containers can reference one claim and share the device.
- Vendor configuration hangs off the claim, per workload, rather than only a per-node setting.
- The pod does not have to put a device quantity in its resource requests. It references the claim.
Three roles appear in the docs. The device owner ships the driver, the slices, and optionally the classes. The cluster admin attaches hardware, installs the driver, and may define classes. The workload operator creates claims and points pods at them.
The limitation that matters for a busy cluster is preemption. The scheduler does not preempt a running pod to free a DRA device. A higher-priority pod stays Pending until the holder terminates or is deleted. Priority will not clear a GPU that was claimed this way. Plan capacity for the claim; stopping or deleting an owner is a separate workload-impact decision. Do not expect the preemption you get for CPU.
Gang scheduling
With default feature gates, kube-scheduler places pods individually. Kubernetes v1.37 adds native gang scheduling as an opt-in Beta feature through GenericWorkload, the GangScheduling plugin and scheduling.k8s.io/v1beta1 PodGroup objects. Its minimum group size is minCount. The similarly named out-of-tree PodGroup below belongs to scheduling.x-k8s.io/v1alpha1 and uses minMember; do not mix these fields or assume either controller handles the other API. Kueue and Volcano are outside this article's scope. The coscheduling plugin in kubernetes-sigs/scheduler-plugins is the implementation that was opened for the group rule. Its README, on the master branch as fetched 10 November 2026, marks maturity Beta (the Stable box is unchecked). The API it shows is scheduling.x-k8s.io/v1alpha1, kind PodGroup.
A PodGroup has minMember and scheduleTimeoutSeconds. Pods join the group with the label scheduling.x-k8s.io/pod-group set to the PodGroup's name. The README says the scheduler adds Running pods to Waiting pods (assumed, not yet bound). If that sum is at least minMember, the waiting pods proceed. Pods in one out-of-tree group should share one priority. Mixed priorities are called out as unintended behaviour.
The README's own demo is the failure mode. A cluster that can run three nginx pods, a ReplicaSet of six, and minMember: 3, ends with three Running and three Pending. Set minMember to 4 on that same scarce cluster and every pod stays Pending, because the group never reaches the minimum. For a training job, minMember should be the number of pods that must start together, and the cluster must actually have that many devices free. Setting the minimum above what you have does not queue politely in the demo. It schedules nothing.
Two groups are ordered as follows, in the same README. A higher-priority group precedes a lower one. If priorities match, the group created first precedes the other. The opened text does not define what scheduleTimeoutSeconds does when it fires. The example value is 10. Do not invent a delete or a retry from that field until you read the controller that consumes it.
The plugin is not on by default. The sample KubeSchedulerConfiguration enables Coscheduling and, under queueSort, enables it while disabling every other queue-sort plugin (disabled: name: "*"). The README says queueSort, permit and unreserve must be enabled. preFilter is optional: it rejects a pod early when the group has fewer pods than minMember, and the README suggests turning it on when minMember is large (their example of "small" is under 5). The sample also sets leaderElect: false and a placeholder kubeconfig. That file is a tutorial, not a drop-in for a highly available control plane. Changing the scheduler is a cluster-wide outage risk. This source-only article supplies no procedure for applying that sample or changing the control plane.
What to choose and why
| Need | Use | Why |
|---|---|---|
| One container, one whole GPU, counted in a limit | Device plugin resource, for NVIDIA the documented name nvidia.com/gpu |
Integer extended resource, no sharing, advertised by ListAndWatch. |
| Select a GPU by attributes, or share one device across pods by reference | DRA: a DeviceClass and a ResourceClaim | The DRA page states those as the gaps in device plugins. Sharing is a claim reference, not a fractional limit. |
| All pods of a training job, or none | Coscheduling PodGroup with minMember equal to the group size, one priority |
With default feature gates, the scheduler can bind a subset; native opt-in gang scheduling is another route. The plugin's demo shows the subset, and shows a too-high minimum binding nothing. |
| A card split into instances | MIG, then advertise the instances | NVIDIA's operator page describes MIG as separate instances on Ampere and later. That is not container sharing of one plugin device. |
Our recommendation, not a Kubernetes guarantee: pick the allocation API first, then add gang scheduling if the job is a group. DRA does not replace the PodGroup. A PodGroup does not advertise the GPU. You need both when the job is both "these devices" and "all of us".
The safe order when GPUs look free and the job is pending
These are reads. Do them before you change the scheduler or delete a pod.
1. Compare capacity and allocatable on the node for the resource name. Allocatable lower than capacity, with the plugin running, matches the documented unhealthy path: the device is still installed, and it is not offered to new pods.
2. Read the pod status. On a cluster where ResourceHealthStatus is on (beta, default since v1.36), allocatedResourcesStatus on the container is the documented place for a device health message. A healthy device status does not prove an application fault or exclude driver, runtime or reporting failures. A failed device status is evidence about an assigned device; correlate its timestamp and the pod events before attributing the crash.
3. Identify the scheduling implementation before interpreting a partial job. Check whether native GenericWorkload gang scheduling is enabled and whether the core PodGroup exists with the expected minCount. For the separate out-of-tree route: If some pods are Running and the rest Pending, and they do not share scheduling.x-k8s.io/pod-group, the default scheduler is doing what it does: placing whoever fits. If they share the label, check that a PodGroup exists, that minMember is not above the free devices, and that the running scheduler actually loaded Coscheduling. A label without the plugin does nothing.
4. For DRA, do not wait for preemption. A Pending pod that references a claim, while a lower-priority pod holds that class of device, matches the documented limitation. Record the allocation and hand it to the workload owner. Any stop or deletion needs its own recovery and approval plan; increasing priority alone does not free the device in the documented DRA path.
5. Hand over the collected evidence. A plugin restart or scheduler change is separate operational work requiring version-specific review, workload protection and a tested recovery plan. Missing resources alone do not authorise restarting every privileged plugin pod.
Risks and limits
- The framework is stable since v1.26 and the API is still described as not stable on the same page. Upgrade the plugin for the new API before you upgrade the nodes, which is the order that page recommends.
- The socket path is hardcoded, the plugin must be privileged, and it must re-register after the kubelet deletes sockets.
- Unhealthy devices stay assigned to the pods that had them. Allocatable drops. Capacity does not.
- DRA will not preempt. The feature is stable since v1.35 and the gate is locked. Sharing and CEL filters are real; priority inversion on a claimed GPU is also real.
- Coscheduling in the opened README is Beta,
v1alpha1, and not the default scheduler. The sample disables other queue-sort plugins and turns leader election off. Pods in that out-of-tree group should share a priority. The timeout field is present; its effect is not specified in that README. - MIG, as described on the NVIDIA operator page opened here, partitions a supported GPU into instances. It is not a fractional Kubernetes resource quantity. Under the NVIDIA
singlestrategy, each MIG instance is advertised asnvidia.com/gpu, so the name alone does not establish a whole physical card. - No NetLeverage cluster, scheduler or GPU service is described.
Sources
- Kubernetes, Device Plugins — https://kubernetes.io/docs/concepts/extend-kubernetes/compute-storage-net/device-plugins/ (opened 10 November 2026; footer last modified 27 July 2026). Stable since v1.26; registration;
nvidia.com/gpu; no sharing; unhealthy allocatable versus capacity;allocatedResourcesStatussince v1.36. - Kubernetes, Dynamic Resource Allocation — https://kubernetes.io/docs/concepts/scheduling-eviction/dynamic-resource-allocation/ (opened 10 November 2026; last modified 27 July 2026). Stable since v1.35, available since v1.30, gate locked; claims, classes, slices; no DRA preemption.
- kubernetes-sigs/scheduler-plugins,
pkg/coscheduling/README.mdon master — https://github.com/kubernetes-sigs/scheduler-plugins/blob/master/pkg/coscheduling/README.md (raw file opened 10 November 2026). Beta; PodGroupv1alpha1;minMember; the three-versus-four demo; scheduler plugin config. -
NVIDIA, GPU Operator with MIG — https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/gpu-operator-mig.html (opened 10 November 2026). MIG partitions Ampere-and-later GPUs, A100 given as the example, into separate CUDA instances.
-
Kubernetes, Gang Scheduling — https://kubernetes.io/docs/concepts/scheduling-eviction/gang-scheduling/. Native Beta since v1.37, disabled by default.
- Kubernetes, Workload API — https://kubernetes.io/docs/concepts/workloads/workload-api/. Core API distinguished from out-of-tree Coscheduling.
This article was originally published on .