Skip to main content

How Kubernetes GPU Scheduling Works: Device Plugins, Resource Requests, and Where It Breaks

For executives​

Kubernetes manages GPU resources differently from CPU and memory. GPUs are extended resources managed through the device plugin framework — a plugin that runs on each GPU node, advertises GPU availability to the scheduler, and allocates GPU device files to pods that request them. Understanding this framework is the prerequisite for understanding how CVE-2024-0135 (the Device Plugin API bypass) works, why GPU resource isolation has different guarantees from CPU resource isolation, and where the security model breaks down.

Why GPUs are not natively supported​

Kubernetes was designed to schedule CPU and memory. GPUs are hardware accelerators with device-specific access requirements that the generic Kubernetes scheduler cannot manage natively. The device plugin framework, introduced in Kubernetes 1.8, allows hardware vendors to write plugins that extend Kubernetes scheduling to cover their devices.

The device plugin framework​

A device plugin is a gRPC server running as a DaemonSet on each node. It performs three functions:

Discovery: the plugin detects available GPU devices on the node and reports them to the kubelet as available resources (e.g., nvidia.com/gpu: 8 for a node with 8 NVIDIA GPUs).

Allocation: when the kubelet schedules a pod that requests a GPU, it calls the device plugin's Allocate RPC. The plugin selects a specific GPU device and returns the device configuration to the kubelet.

Health checking: the plugin monitors GPU health and updates the kubelet on device availability.

The GPU allocation flow​

  • A pod is submitted with resource requests including nvidia.com/gpu: 1.
  • The Kubernetes scheduler selects a node with available GPU capacity.
  • The kubelet on the selected node calls the device plugin's Allocate RPC.
  • The device plugin selects GPU device 0 and returns device mounts — /dev/nvidia0, /dev/nvidiactl, /dev/nvidia-uvm — and environment variables including NVIDIA_VISIBLE_DEVICES=GPU-<UUID>.
  • The kubelet creates the pod with those device files mounted and the environment variables set.
  • The NVIDIA Container Toolkit sees NVIDIA_VISIBLE_DEVICES and grants the container access to the specified GPU.

Where the security model breaks down​

CVE-2024-0135 exposed a gap in step 6. The Container Toolkit's pre-patch behaviour was: read NVIDIA_VISIBLE_DEVICES from the container's environment and grant access to those devices, regardless of whether it was set by the device plugin or set directly in the pod specification by an operator or user.

Setting NVIDIA_VISIBLE_DEVICES=all in a pod spec, without requesting any GPU resources, gave the container access to all GPUs on the node. The device plugin controls allocation, but the Container Toolkit did not verify that NVIDIA_VISIBLE_DEVICES originated from the device plugin's allocation.