What the GPU Device Plugin Does: Architecture and Where the Security Model Breaks Down
For executives
The NVIDIA Kubernetes Device Plugin (k8s-device-plugin) is the Kubernetes component responsible for GPU resource allocation and enforcement. It runs on every GPU node in a Kubernetes cluster, tells the scheduler how many GPUs are available, and allocates specific GPUs to specific pods. CVE-2024-0135 demonstrated that the device plugin's enforcement was bypassable. This article explains how the plugin works at a level of detail sufficient to understand the bypass.
The device plugin's architecture
The device plugin runs as a container in a DaemonSet pod on each GPU node. It communicates via two interfaces:
To the kubelet: via a gRPC socket (/var/lib/kubelet/device-plugins/nvidia.sock), the plugin registers itself, advertises GPU resources, and receives allocation requests.
To the Container Toolkit: the device plugin's allocation response specifies which GPU device files to mount into the pod and which environment variables to set.
GPU resource advertisement
When the device plugin starts, it discovers GPU devices on the node via NVML (the NVIDIA Management Library) and registers nvidia.com/gpu as an extended resource. It then streams a DevicesUpdate to the kubelet reporting the UUIDs of all available GPUs and their health status.
GPU allocation
When a pod is scheduled requesting GPUs, the kubelet calls the device plugin's Allocate gRPC endpoint. The plugin selects specific GPUs by UUID, marks them as allocated, and returns an allocation response specifying: container devices (the /dev/nvidia* device files to mount), container mounts (directories to mount), and container environment variables (NVIDIA_VISIBLE_DEVICES set to the allocated GPU UUIDs).
The enforcement gap that CVE-2024-0135 exploited
The Container Toolkit's code at the time of CVE-2024-0135: read NVIDIA_VISIBLE_DEVICES from the container environment and grant access to those devices. Any container environment variable set to NVIDIA_VISIBLE_DEVICES=all would receive access to all GPUs, regardless of whether the device plugin had allocated them.
A pod spec that manually set NVIDIA_VISIBLE_DEVICES=all — without requesting any nvidia.com/gpu resources — would pass the scheduler (no GPU resources requested means no device plugin involvement) and then receive all GPU access through the Container Toolkit's environment variable processing.
After the patch: the authorised device list
Post-patch, the Container Toolkit accepts only the device files explicitly provided by the device plugin as volume mounts. Environment variables alone are no longer sufficient to grant GPU access to an unprivileged container.
The persistent supply chain concern
As documented in the attack class library, the k8s-device-plugin container image carries a persistent inventory of unpatched OS package CVEs. The plugin's elevated permissions — direct kubelet API access, node object modification rights — make its security posture directly relevant.