What a CUDA Context Is: And Why Tracking It Per Container Matters for Security
For executives
A CUDA context is the fundamental unit of GPU resource ownership. Every process that uses a GPU creates a CUDA context. The context has its own GPU address space, its own loaded kernels, and its own resource allocation. Tracking which containers create GPU contexts, when those contexts are created, and what code is loaded into them provides the first layer of GPU security telemetry that is useful for detection. This article explains the context in enough depth to understand why tracking it is meaningful.
The CUDA context lifecycle
A CUDA context is created lazily: when a process calls any CUDA runtime function for the first time, the runtime creates a context on the currently selected GPU device. The context persists until either the process explicitly destroys it or the process terminates.
From the GPU's perspective, a context is an isolated execution environment with its own:
GPU virtual address space: memory allocations made within the context have addresses that are private to that context. Other contexts cannot access those addresses via normal CUDA API calls.
Loaded modules: compiled CUDA kernels (GPU programs) are loaded into a context before they can be executed. The set of modules loaded into a context defines what GPU code that context can run.
Command queue: kernel launch requests and memory operations are queued within the context's streams and executed in order.
Why one container should typically have one context
A GPU container running a single AI workload — a training job, an inference server, a data processing pipeline — typically creates one CUDA context per GPU device it uses. The AI framework (PyTorch, TensorFlow, JAX) creates the context when the workload starts and uses it for the duration.
A container that creates multiple CUDA contexts, or a container whose context creation timing is unexpected (context created much later than workload startup, or multiple contexts created in rapid succession), may be running additional code beyond its declared workload.
A container running legitimate training creates one context. A container running training plus a background cryptomining process creates two contexts (one for training, one for mining) — unless the mining code is embedded in the same process, in which case both are in the same context but the module set is abnormal.
GPU module hash verification
When GPU code is loaded into a context, the compiled GPU kernel binary is transferred to the GPU and installed for execution. That binary has a deterministic hash: the same source code, compiled with the same toolchain, produces the same binary.
Known-good AI framework CUDA kernels have known hashes. PyTorch's matrix multiplication kernel for a specific GPU architecture has a specific hash that does not change between runs (for a given framework version). An unexpected kernel hash in a context running a declared AI training workload means something other than the expected framework kernels are loaded.
This is the detection mechanism: maintain a baseline of expected GPU module hashes for each workload type. Alert when a CUDA context loads modules that are not in the expected set for its container's declared workload.
Per-container context tracking
Tracking CUDA contexts per container — not per host — requires mapping each CUDA context to the container it was created from. A single GPU node may have dozens of running containers, each with their own GPU contexts. Context tracking at the container level identifies which container's workload is responsible for each context.
This mapping requires correlating the PID that created the context with the container namespace that PID belongs to. That correlation is available from the host's cgroup and namespace infrastructure — the same infrastructure the Container Toolkit uses to manage container GPU access.