How the CUDA Execution Model Works: Kernels, Contexts, and What Runs Where
For executives
The CUDA execution model is the mechanism through which every AI workload uses a GPU. It determines how code gets to the GPU, how memory is accessed, how results come back, and how multiple workloads share GPU hardware. Understanding it at a conceptual level is necessary for understanding why GPU security monitoring requires different approaches from CPU security monitoring.
From Python to GPU hardware: the execution path
A PyTorch model training step in Python ultimately causes GPU code to execute. The path from Python to GPU hardware:
- PyTorch receives the Python call and determines which CUDA operations are needed.
- PyTorch calls cuDNN or cuBLAS for standard operations, or its own compiled CUDA kernels for others.
- The CUDA runtime receives kernel launch requests and queues them to the GPU.
- The CUDA driver translates these into ioctl calls to the NVIDIA kernel module.
- The NVIDIA kernel module communicates with the GPU hardware to schedule and launch the kernels.
- The GPU executes the kernels across thousands of parallel threads.
- Results are written to VRAM.
- The CPU is notified via synchronisation that results are available.
Steps 6-7 are entirely invisible to CPU-side monitoring. The GPU is computing autonomously.
CUDA kernels
A CUDA kernel is a function written in CUDA C/C++ that executes on the GPU. When a kernel is launched, the CUDA runtime creates a grid of thread blocks, each containing a specified number of threads. All threads execute the same kernel function simultaneously on different data.
Kernels are compiled into PTX (Parallel Thread Execution) intermediate representation and then to device-specific binary code. The compiled code is loaded into the GPU's instruction cache before execution.
CUDA contexts
A CUDA context is the primary resource container for GPU computation. When a process first calls any CUDA runtime function, the runtime creates a context on the selected GPU. The context contains: a virtual address space for that context's GPU memory allocations, a set of loaded GPU modules, and a command queue for kernel launches and memory operations.
From a security monitoring perspective, the CUDA context is the unit of analysis. Tracking context creation, module loading, and memory operations provides the GPU execution telemetry layer.
The CUDA stream and asynchrony
CUDA operations are submitted to streams — ordered queues of operations that execute sequentially within the stream but potentially in parallel with other streams. The CPU submits a kernel launch to a stream and returns immediately. The GPU executes the kernel asynchronously.
A CPU-side trace of ioctl calls reflects submission, not execution. The GPU execution timeline diverges from the CPU timeline.
What kernel hash verification means
Every compiled CUDA kernel has a deterministic binary representation that can be hashed. A legitimate PyTorch installation's CUDA kernels have known hashes. A cryptomining kernel has a different hash. A kernel modified to include malicious code has a different hash from the original.
When a GPU module is loaded into a CUDA context, the module's hash can be computed and compared against a known-good baseline. Unexpected kernel hashes indicate that something other than the declared workload is being executed on the GPU.