What GPU Telemetry Exists Today: Utilisation, Memory, and Why None of It Is a Security Signal
For executives
GPU monitoring is mature for operations and performance. Every cloud platform, every Kubernetes monitoring stack, and every GPU cluster management tool provides GPU utilisation percentages, memory usage totals, temperature readings, and performance counters. Security teams looking at GPU infrastructure see dashboards full of metrics. None of those metrics are security signals. This article explains what existing GPU telemetry measures, why it is not useful for security monitoring, and what security-relevant GPU telemetry looks like.
What nvidia-smi and NVML report
The standard GPU monitoring interface is NVML (NVIDIA Management Library) and its command-line tool nvidia-smi. These report:
GPU utilisation: percentage of time the GPU's streaming multiprocessors were executing a kernel in the last sampling interval. A value of 95% means the GPU was busy 95% of the time. It says nothing about what it was doing.
Memory utilisation: percentage of VRAM in use. Combined with the total VRAM capacity, this gives free and used memory. It does not say which process is using which memory, what those memory regions contain, or whether memory was properly cleared between workloads.
Temperature: GPU die temperature. Relevant for thermal management, not security.
Power draw: GPU power consumption in watts. Can vary between different types of computation but is too coarse for meaningful security analysis.
Clock speeds: GPU and memory clock frequencies. Useful for performance debugging.
SM utilisation by context (via NVML): with additional APIs, NVML can report per-process GPU utilisation. This tells you which PID is using GPU compute, which is a step up from aggregate utilisation but still does not reveal what that process is computing.
Why these metrics are not security signals
Cryptomining on an H100 generates 95-100% GPU utilisation. Legitimate AI training on an H100 generates 95-100% GPU utilisation. The utilisation metric is identical. There is no threshold, no ratio, no pattern in the aggregate utilisation metric that distinguishes mining from training.
Model weight exfiltration involves a CUDA memcpy from device to host memory, followed by a network transfer. The memcpy shows up as a brief spike in memory bandwidth utilisation — indistinguishable from any legitimate GPU-to-CPU data transfer that happens continuously in inference workloads. The network transfer might show up in network monitoring if it is large enough, but the source of the transfer is the CPU process making the network call, not the GPU.
A Rowhammer VRAM attack manifests as extremely high VRAM utilisation (a legitimate indicator of memory pressure) and high UVM eviction rates (also a legitimate indicator of memory pressure in workloads that exceed VRAM capacity). The GPUBreach research identified that the attack has specific small-page allocation patterns and specific eviction timing signatures — but those signatures are not in the summary metrics. They require event-level UVM telemetry.
What security-relevant GPU telemetry looks like
Security signals come from event-level data, not summary metrics:
CUDA context events: which processes created CUDA contexts, when, and on which GPU device. Unexpected context creation from a process that should not be using the GPU is a signal.
GPU module load events: which compiled CUDA kernel modules were loaded into each context. Kernel hash verification against known-good baselines for declared workload types is a signal for unexpected code on the GPU.
UVM events: the allocation size, eviction timing, and page migration patterns documented in the GPUBreach detection section. Abnormal small-page allocation bursts, specific eviction-reallocate patterns at the right timing, are detection signals for Rowhammer-style attacks.
CUDA memory operation patterns: large cudaMemcpy DeviceToHost transfers from inference processes that do not normally transfer data to the CPU are anomalous.
Hook process behaviour: what the nvidia-ctk hook process does when a container starts — which libraries it loads, which environment variables it reads, which files it opens.