ShadowKV (ByteDance)
https://arxiv.org/abs/2410.21465
During prefill, value caches are offloaded to CPU and only low-rank pre-RoPE keys plus compressed landmarks and outliers are kept on GPU. During decoding, landmarks select chunk indices for key reconstruction and value fetching from CPU. Effectively unlimited context without GPU memory blowup. ICML 2025 Spotlight.
At a glance
- Type
- Low-rank CPU-offloaded KV cache + sparse reconstruction
- Tier
- T2
- Created
- 2024-10
- Latest release
- no releases
- License
- Apache-2.0
- GitHub
- 297★ Python
- Pricing
- not applicable — research paper
- Funding
- not applicable — not commercial
Taxonomy
- storage
- kv-cache
- retrieval
- cache-lookup
- persistence
- session
- update
- evict-oldest
- unit
- kv-token
- governance
- opaque
- conflict
- n/a
When to use
Optimised for: not applicable - research paper
Anti-fit: not applicable - research paper
Pros & cons
Pros
Validated on six diverse model families with no accuracy degradation; ICML 2025 Spotlight; open-source.
Cons
Requires CPU-GPU data transfer during decoding (PCIe-bandwidth-sensitive latency); low-rank key approximation may miss precise token-level distinctions in adversarial retrieval.
Claims & capabilities
Up to 6× larger batch sizes and 3.04× throughput on A100 without accuracy loss; validated on RULER, LongBench, NIAH across Llama-3.1, GLM-4, Yi, Phi-3, Qwen2.
Technical surface
- API surface
- searched not found
- Backend storage
- searched not found
- Deployment
- not applicable — research paper
- Embedding model
- searched not found
- Multi-tenancy
- searched not found
- MCP
- not applicable — research paper, no deployed product
- A2A
- not applicable — research paper, no deployed product
- OpenTelemetry
- not applicable — research paper, no deployed product
Similar systems
Other recent method papers — theorized, no distinct product in the catalog, ranked by inbound references.
- Compressive Transformer T3
Maintains recent states in full resolution while compressing older memories with learned compression functions. DeepMind.
- MemGPT v2 / agent-tools T3
Already in catalog as the foundational MemGPT paper. Note: Letta is the productionised successor (cross-listed).
- Transformer-XL T3
Extends context through segment-level recurrence + caching of hidden states from prior segments. Foundational long-context architecture.
- Generative Agents T3
Park et al. — landmark agent-simulation paper. Reflection + memory stream + retrieval enable believable agent behavior.
- MemoryBank T3
Enhances LLMs with long-term memory. Early influential paper.
- Reflexion T3
Language agents with verbal reinforcement learning.