What Full-Stack GPU Telemetry Means: From CUDA Kernel to Network Egress
For executives
Full-stack GPU telemetry means a single, correlated event timeline that spans from the lowest GPU execution level (CUDA kernel launches, UVM memory events) through the container runtime layer (hook process behaviour, process creation) to the host network layer (outbound connections, data transfers). No individual existing tool covers this full stack. The value of a unified timeline is that attacks which are invisible at any single layer become visible when events across layers are correlated. This article explains what each layer contributes to the timeline and what correlation reveals.
Layer 1: UVM and memory events
The NVIDIA Unified Virtual Memory system generates events for allocation, eviction, and migration of memory between CPU and GPU. These events are the lowest-level GPU telemetry available: they occur at the driver level, before any user-space process is aware of them.
UVM events record: allocation size and type (4KB, 64KB, 2MB page sizes), the physical address of the allocated or evicted page, the process ID that triggered the allocation, the timestamp.
Security-relevant patterns in UVM events:
Abnormal small-page allocation bursts (the Phase 1 GPUBreach signature documented by Stealthium). Rapid alternating allocation-eviction cycles (the Phase 2 GPUBreach page table massaging signature). Large cudaMemcpy DeviceToHost operations from inference processes.
Layer 2: CUDA context and module events
At the CUDA context layer: context creation (which process, which GPU, when), module loads (which compiled kernel binary, what hash), and context destruction.
Security-relevant patterns:
Unexpected context creation from a container that should not be using the GPU. Module hash mismatches against the declared workload baseline. Multiple contexts in a container that should have only one. Context creation timing anomalies (context created long after workload startup).
Layer 3: Container process and hook events
At the container runtime layer: OCI hook process behaviour (what libraries the nvidia-ctk process loads, what environment variables it reads), container process creation, process executions within the container.
Security-relevant patterns:
LD_PRELOAD set in a container's hook environment (the NVIDIAScape detection signal). Symlink resolution outside the container boundary during hook execution. Unexpected process creation inside a container that should not be spawning children.
Layer 4: Host process and network events
At the host layer: processes that appear after a suspected container escape (running in the host's PID namespace rather than the container's), outbound network connections from processes that should not be making them, large data transfers.
Security-relevant patterns:
A process appearing in the host's PID namespace that originated from a GPU container's hook context. Outbound connections to unusual destinations from the GPU host. Large HTTPS transfers from the host that are not attributable to declared workload egress.
The timeline correlation value
An attack sequence that spans all four layers:
- UVM events: abnormal allocation pattern begins. (Layer 1)
- CUDA context: module with unexpected hash loads. (Layer 2)
- Hook process: no anomaly (attack is not via hook). (Layer 3)
- Network: large outbound transfer from GPU host process. (Layer 4)
At Layer 1 alone: possibly normal memory pressure. At Layer 2 alone: possible legitimate framework update. At Layer 4 alone: possibly normal cloud storage operation. Correlated: an unexpected module executing on the GPU, producing activity in GPU memory, followed by a large outbound transfer — model exfiltration with high confidence.
The correlation that elevates each layer's ambiguous signal into a high-confidence detection requires a unified timeline where events from all layers are timestamped, attributed to the same process and container, and evaluated together.