
Choosing the right GPU server for an open-weight LLM isn't as simple as headcount or budget — it comes down to model size, interconnect bandwidth, KV cache math, and concurrency requirements that only become visible once hardware is on a purchase order. This episode walks through the detailed guidance in the GPU server configuration guide for open-weight LLMs published on hardware.co, translating the key trade-offs into practical decision-making across four distinct build tiers. Here's what the episode covers: Start with the model, not the chassis. Four parameter bands — 7–14B, 24–34B, 70B dense, and large MoE — each carry specific VRAM, RAM, and interconnect requirements that should drive the server spec, not the other way around. VRAM and KV cache are the two numbers that matter most. Weights run ~2 bytes per parameter at BF16 and drop to ~0.5 bytes at INT4; KV cache scales with context length and concurrency and can easily exceed model weight storage at long-context serving loads. Single-GPU workstations are a legitimate production tier. An 80 GB card can serve a 70B model at 4-bit, a 32B at 8-bit, or any 7–14B at full precision — making it the right landing spot for most pilots and small-team deployments. Teams planning this kind of build can estimate their exact VRAM needs with the AI Hardware Estimator. Two-GPU builds live or die by their interconnect. Whether to run dual independent replicas, tensor-parallel a larger model, or dedicate one card to embeddings and reranking depends almost entirely on whether the cards share NVLink or are limited to PCIe peer-to-peer — a gap that can represent an order-of-magnitude difference in cross-GPU bandwidth. Four GPUs is the honest tier for 70B-class dense models and medium MoE. Mixtral's MoE architecture means you pay for ~46.7B parameters in VRAM but compute at roughly 13B-dense speeds — a useful planning target that explains why four-GPU configs suit MoE workloads so well. Eight-GPU nodes unlock frontier MoE and serious fine-tuning — but DeepSeek-V3's 671B total parameters require multi-node InfiniBand or RoCE setups, a parallel filesystem, and 8–12 kW of power capacity even before accounting for KV cache. For teams scaling to this level, a purpose-built GPU cluster for large models and high concurrency is worth evaluating. The episode makes clear that GPU count is the headline spec, but interconnect, system RAM, and NVMe storage are often where deployments succeed or fail. For more on the cost and security trade-offs of running models on your own hardware versus a managed service, check out the related episode Private LLM vs ChatGPT Enterprise: Cost, Security, and Performance. hardware.co
Podzilla Summary coming soon
Sign up to get notified when the full AI-powered summary is ready.
Free forever for up to 3 podcasts. No credit card required.

KV Cache Optimization for Self-Hosted LLMs Under Load

Which NVIDIA GPU Do You Actually Need for On-Prem LLMs?

Pinecone, Qdrant, Weaviate, and pgvector for Regulated Enterprise RAG

The Numbers Behind On-Prem AI Hardware in 2026
Free AI-powered recaps of LLM.co and your other favorite podcasts, delivered to your inbox.
Free forever for up to 3 podcasts. No credit card required.