
Running a language model in-house is one thing; keeping it responsive when dozens or hundreds of users hit it simultaneously is another challenge entirely. This episode of LLM.co digs into the specific mechanics of KV cache pressure — why it fills GPU memory faster than almost any other factor, and what engineering teams can actually do about it. The discussion is grounded in the KV cache optimization deep-dive for self-hosted LLMs that underpins the episode. Here's what the episode covers: What the KV cache is and why it scales so aggressively — key-value tensors grow with layer count, head count, context length, and batch size, and naive allocators waste 60–80% of reserved memory to fragmentation before a single request completes. PagedAttention — borrowed from OS virtual memory design, it maps fixed-size cache blocks to physical GPU pages on demand rather than pre-reserving contiguous chunks, routinely doubling concurrent request capacity at a given context length. Prefix caching and RadixAttention — shared system prompts, retrieved documents, and chain-of-thought scaffolding can be computed once and reused across many requests; SGLang's RadixAttention approach has shown up to 5× throughput improvements on structured generation workloads, making it especially relevant for secure enterprise agentic deployments that reuse large prompt fragments. KV cache quantization — dropping tensors from BF16 to FP8 halves memory with negligible quality loss on most models, but gains are entirely engine-dependent; TensorRT-LLM showed up to 45% higher throughput while vLLM showed almost none, because the inference engine must be able to consume the quantized cache. Architectural choices that reduce KV footprint by design — Grouped-Query Attention (as in Llama 3 70B's 8 KV heads vs. 64 query heads) shrinks cache size 8×; DeepSeek's Multi-Head Latent Attention compresses further still, reporting a 93% KV reduction and a 5.76× throughput gain; sliding-window attention caps cache at a fixed context window for chat and short-RAG workloads. These gains matter particularly for teams deploying on-premises custom AI chatbots at scale. CPU offload and speculative decoding — spilling cold KV blocks to host DRAM and using a small draft model to propose tokens in parallel are two tactics for pushing throughput higher when memory is still constrained after everything else has been optimized. The episode closes with a practical order of operations: architectural choices first, PagedAttention and prefix caching second, quantization third (with engine compatibility verified), and offload or speculative decoding last. For more on the hardware side of self-hosted inference, check out the companion episode Which NVIDIA GPU Do You Actually Need for On-Prem LLMs? LLM.co hardware.co
Podzilla Summary coming soon
Sign up to get notified when the full AI-powered summary is ready.
Free forever for up to 3 podcasts. No credit card required.

Which NVIDIA GPU Do You Actually Need for On-Prem LLMs?

Pinecone, Qdrant, Weaviate, and pgvector for Regulated Enterprise RAG

The Numbers Behind On-Prem AI Hardware in 2026

How to Audit and Fix Your Private AI Stack Before It Breaks in Production
Free AI-powered recaps of LLM.co and your other favorite podcasts, delivered to your inbox.
Free forever for up to 3 podcasts. No credit card required.