#kvcache

Google TurboQuant (ICLR 2026): 6x KV cache compression, 8x H100 speedup, zero accuracy loss — no retraining needed. What this means for LLM inference costs.

A new paper shows a frozen small language model's accuracy on a hard math test can jump from 80.0% to 93.3% simply by grafting a byte-exact cache of verified solutions into it, with no weight changes at all.

Two new techniques treat a language model's long-context memory like an operating system's memory hierarchy - keeping coarse summaries on the GPU and paging compressed detail out to the CPU - with one, SeKV, cutting GPU memory use by 53% at 128,000 t

Originally published on andrew.ooo — visit the original for any updates, code snippets that aged...

Why the “Context Tier” Is Quietly Hijacking AI Inference Performance The AI research...

Many people have an intuition when using LLMs: longer conversations mean more expensive tokens, so...

When people estimate token costs, they usually watch TTFT, TPOT, and throughput. What actually makes bills differ by 10× is whether the KV cache hits. The model, server, and user layers all have to line up.

SpecKV (arXiv:2605.02888) shows fixed γ=4 costs 56% throughput. Adaptive gamma, KV cache compression effects, and vLLM production tuning guide.

The Problem Nobody Warns You About Most LLM inference guides talk about model quantization...

Why Memory Fragmentation Kills LLM Serving Throughput Here's a number that should make you...

The model loaded. The first prompt worked. Then longer prompts or multiple users showed up, and...

The O(N2) Killer: How KV Cache Supercharges LLM Inference? ⁉️Introduction — What is...