GPU Memory Snooping: Residual Data in VRAM Between Workloads in Shared Environments
For executives
When a GPU workload finishes, the GPU memory it used is returned to the allocator. In most GPU configurations, that memory is not zeroed before it is reallocated to the next workload. The previous workload's data — model weights, inference inputs, intermediate computation results, user data — persists in VRAM until it is overwritten by the next operation. In shared GPU environments where multiple users or workloads run sequentially on the same hardware, this residual data is accessible to subsequent allocations.
How GPU memory allocation works
GPU memory allocation via cudaMalloc() requests a block of VRAM from the GPU's memory allocator. The allocator returns a pointer to a free block. That block is not zeroed — it contains whatever was last written there.
On a CPU, this is mitigated by OS-level memory isolation: the OS ensures freshly allocated virtual memory pages are presented to processes with zero content. On the GPU, no equivalent OS-level guarantee exists. The VRAM block returned by cudaMalloc() may contain data from a previous allocation by the same process, a previous process, or in shared GPU environments, a previous tenant.
What can be in residual VRAM
Model weights: the primary data the inference process works with. For a deployed LLM, these are the billion-scale parameters defining the model's behaviour.
Inference inputs: user queries passed to the model are copied to VRAM for processing. These may include user-supplied text, images, audio, or proprietary business data.
Key-Value (KV) cache: transformer-based language models maintain a KV cache of intermediate attention results. For a deployed chatbot, the KV cache may contain the conversation history of multiple users' sessions.
Mitigations
NVIDIA's Confidential Computing mode (H100 and later) enables hardware encryption and isolation of GPU memory. Driver-level memory zeroing configures the NVIDIA driver to zero memory on deallocation before returning it to the allocator. MIG hardware partitioning provides hardware isolation between partitions at the memory controller level.