Cross-Request KV Cache Leak in Cloudflare Workers AI: When an LLM Response Contained Other Users' Data
By Arash Latifi
A technical breakdown of the GLM-5.3 Workers AI incident — 155 affected responses, the SGLang bug, and lessons for anyone self-hosting inference.
TL;DR — Cloudflare disclosed yesterday that GLM-5.3 on Workers AI returned responses containing other customers' data, caused by a config mismatch plus an SGLang bug. If you self-host LLM inference, read this.
What happened
Yesterday (October 26, 2026), Cloudflare published a full post-mortem of a security incident, disclosed in coordination with the researcher and the SGLang team. The highlights:
- From August 27 to September 9, 2026, a small number of customers using
@cf/zai-org/glm-5.3on Workers AI received responses containing data from other customers' requests. - In total, 155 responses were affected — all to prompts longer than 681,280 tokens.
- It surfaced via a HackerOne report by security researcher Gadi Evron (Knostic) on September 8. Cloudflare reproduced it the next morning, took the model offline within 40 minutes of declaring the incident, and restored it the same day with a corrected configuration.
- No evidence of deliberate exploitation was found; request patterns matched ordinary long-context use.
Root cause: two bugs overlapping
1) Misconfiguration. GLM 5.3 was deployed with a 1,048,576-token context window, but the KV cache pool on the prefill GPUs only had room for 681,280 tokens. Cloudflare knew about the mismatch before launch — their tests only validated HTTP status codes and streaming body shape, not model output coherence.
2) The SGLang bug in disaggregated mode. When a prompt didn't fit the pool, SGLang's scheduler rejected it and swapped the prompt for a one-token placeholder. The problem: admission queues on prefill and decode checked whether a request was finished(), not whether it was rejected. A rejected request reported itself as live, so:
- prefill ran a trivial forward pass on one token,
- transferred one KV page and signaled "transfer complete,"
- decode attended over all reserved pages — page P0 written, but pages P1–P7 still held earlier requests' data (other users' prompts and partial generations),
- the client got HTTP 200 instead of HTTP 400, with a coherent, suspiciously fast reply containing other users' content.
Every request over the threshold behaved this way. Notably, the bug had been reported to SGLang as issue #30233 back on July 6, 2026.
Incident timeline
| Date (2026) | Event | |---|---| | Aug 27 | GLM 5.3 launched with a KV pool smaller than its context window | | Sep 8, 23:11 | HackerOne report received | | Sep 9, 13:22 | Incident declared; reproduced with two independent accounts in 25 minutes | | Sep 9, 14:00 | Model set to private; requests return HTTP 400 | | Sep 9, 19:40 | Model restored with corrected config + engine patch | | Oct 2 | SGLang v0.5.21 released | | Oct 19 | SGLang v0.5.22 released (includes all fixes) | | Oct 26 | Coordinated publication with the researcher and RadixArk |
The fixes
Config side: the pool now holds about 2.04 million tokens — roughly twice the 1,048,576 context length, so valid requests never reach the rejection path.
Engine side: the rejection path is now safe — a rejected request is retired with its original error before any bootstrap, allocation, or KV transfer. Core fixes landed in #36988 (v0.5.20) and #38935; decode now also verifies that tokens transferred match tokens reserved. Cloudflare recommends all operators upgrade to SGLang v0.5.22 or later.
Cloudflare's recommendations for LLM operators
If you run SGLang in disaggregated prefill/decode mode (or any self-hosted inference):
- Measure your KV pool and compare it to your context length — on every node, from the engine's startup log, after every change to weights, quantization, or parallelism. If the pool is smaller, lower the context length. Don't rely on a proxy.
- Upgrade to v0.5.22+ so over-length requests are rejected before handoff regardless of pool size.
- Run with deferred KV release on both roles and bound your Mooncake transfer batches.
- Enable corruption detection where supported:
--disaggregation-enable-kv-checksumand--kv-canary(not yet covering every model architecture). - Don't let your proxy generate unnecessary aborts — allow requests to proceed up to the first token.
Lessons beyond this incident
- Shared tenancy is a real attack surface. One top HN commenter (233 points) put it well: many people assume shared hosts magically isolate their tokens, but shared caches carry whole classes of attacks. Highly sensitive data should sit on non-shared hosts.
- HTTP status codes are not a test. Cloudflare's own gap was validating response codes and performance but not output coherence. Inference tests must validate model output.
- Treat odd LLM responses seriously. A commenter noted this incident might explain times they received completely unrelated responses from a certain provider's API.
- If you run apps on Cloudflare Workers and worry about isolation boundaries, see our guide on self-hosting AI agents on Cloudflare's free tier, and our analysis of formal verification of LLMs at Anthropic for another angle on inference-safety guarantees.
Q: Was my data affected?
Only if you used @cf/zai-org/glm-5.3 on Workers AI between August 27 and September 9, 2026, with prompts longer than 681,280 tokens. Cloudflare identified 155 affected responses and says no other model or shorter contexts were affected; no deliberate exploitation was found.
Q: What is this bug, in plain terms?
The model's KV cache was shared incorrectly between requests across nodes. When a prompt didn't fit in memory, the system attended over stale cache pages (another user's data) instead of erroring out.
Q: How do I make sure my LLM service is safe?
For managed services, follow the provider's security advisories. For self-hosting: upgrade SGLang to v0.5.22+, compare pool size against context length on every change, and validate model output in tests — not just HTTP codes.