LLM.co

Which NVIDIA GPU Do You Actually Need for On-Prem LLMs?

November 15, 2026·6 min
Episode Description from the Publisher

GPU selection for on-premises AI is widely treated as a hardware preference question, when it's really a memory budget problem with a straightforward formula. This episode of LLM.co walks through the GPU requirements guide for open-weight LLMs published by hardware.co, turning the model-by-model reference into an internalized framework any team can apply before they ever read a spec sheet. The episode covers the full sizing methodology and then applies it to the models teams are actually running in production today: The three-number formula: Weight memory at your chosen quantization, KV cache cost multiplied by your concurrent user count, and a 10–15% overhead buffer — add them up and your shortlist nearly writes itself. Quantization trade-offs in practice: FP8 is the most robust option across task types; 70B-scale models typically hold strong accuracy at four bits; smaller models can degrade noticeably at INT4. KV cache as the underweighted variable: A dense 70B model can consume 5 GB per request at 8K context or 80 GB at 128K — meaning KV cache, not weight size, often sets the real memory floor for multi-user deployments. Tier-by-tier GPU mapping: 7B–32B models on one or two L40S or RTX 6000 Ada cards; 70B class (e.g., Llama 3.3 70B) on two to four H100s for department-scale concurrency; flagship MoE models like DeepSeek V3/R1 and Qwen3-235B on 8× H100 or H200 nodes. MoE sizing traps: Only a fraction of parameters activate per token, but every weight must still be resident in VRAM — teams must plan for total parameter count, not the active slice. Notable outlier — gpt-oss-120b: OpenAI's Apache 2.0 MoE model, optimized for native MXFP4 on a single 80 GB GPU, is seeing fast on-prem uptake precisely because it breaks the usual multi-card requirement at 120B parameters. Teams unsure where their workload falls on this spectrum can use the AI Hardware Estimator to calculate VRAM requirements across models, quantizations, and user counts — and compare configurations against hardware.co's rack and cluster offerings. If the right hardware tier turns out to be a multi-card server build, the AI server configuration comparison page maps out four size options side by side. For more on the retrieval infrastructure that often sits alongside these models in regulated environments, the episode Pinecone, Qdrant, Weaviate, and pgvector for Regulated Enterprise RAG is a natural companion listen. hardware.co

Podzilla Summary coming soon

Sign up to get notified when the full AI-powered summary is ready.

Get Free Summaries →

Free forever for up to 3 podcasts. No credit card required.

Get summaries like this every morning.

Free AI-powered recaps of LLM.co and your other favorite podcasts, delivered to your inbox.

Get Free Summaries →

Free forever for up to 3 podcasts. No credit card required.