Skip to main content

Cross-Tenant Data Leakage in AI Cloud: What Happens When Accelerator Memory Is Not Properly Cleared

For executives​

When a cloud GPU workload finishes and the GPU is returned to the pool for the next customer, what happens to the data in GPU memory? In configurations that do not explicitly enforce memory clearing, it persists until overwritten. A customer who gets a GPU that previously processed another customer's AI workload may be able to read residual data from that workload — model weights, inference inputs, intermediate activations, or user data that was in GPU memory at the time the previous customer's session ended.

The GPU memory lifecycle in cloud environments​

Customer A's workload loads model weights into VRAM, processes inference requests, and their session ends. The GPU is assigned to Customer B. Between these two events, the GPU's VRAM contains Customer A's residual data. Unless the cloud provider explicitly zeros VRAM between customer sessions — which has a performance cost and is not universally done — Customer B can allocate GPU memory that contains Customer A's data.

If Customer B's workload includes code that reads uninitialized GPU memory before writing to it (which malicious code can do deliberately), they can read Customer A's residual data.

What data is at risk​

Model weights: if Customer A's session involved a proprietary model, its weights were resident in VRAM until the session ended. Inference inputs: user queries processed by Customer A's model were in VRAM during inference. KV cache: a transformer-based chatbot accumulates conversation history in the KV cache. The full conversation history may be in VRAM at session end.

The compliance dimension​

For regulated industries, cross-tenant GPU memory leakage is a compliance risk. Healthcare organisations subject to HIPAA deploying AI inference on shared GPU infrastructure need to know whether patient data processed through their inference endpoints persists in VRAM after their session ends. The answer depends entirely on whether the cloud provider zeros VRAM between sessions.

Existing and emerging mitigations​

Memory zeroing between sessions: the cloud provider or the customer can explicitly zero GPU memory allocations. The performance cost is proportional to how much memory is being cleared and how frequently sessions change. Confidential Computing: NVIDIA's H100 TEE encrypts GPU memory with per-VM keys. Dedicated hardware: eliminating multi-tenancy entirely.