Kafka in Production: Failure Scenarios for Staff-Level Interviews

A single misconfigured setting in Kafka doesn't just slow a page down. It can silently vaporize financial transactions. In this deep dive we walk through real production failure scenarios and the exact reasoning interviewers look for in staff-level system design interviews.You'll learn:- Why acks=1 loses data, and the four configuration layers for zero data loss- Consistency vs availability: why a healthy cluster refuses writes on purpose- Pushing from 400,000 to 2 million events/s with batching, linger.ms and compression- How retries reorder a debit and a credit, and how idempotent producers fix it- Rebalance storms, cooperative sticky assignment and static membership- Hot partitions and the irreducible trade-off between ordering and scale- How deleting and recreating a topic silently skips hours of data- A 3 AM role-play: 85 under-replicated partitions, a disk at 96%, and why you never restart the broker- Exactly-once with Kafka transactions, and exactly-once into PostgreSQL- Tiered retry topics and dead letter queues- Split brain, KRaft quorums and vanishing tombstones- System design at scale: multi-tenant topics, quotas, event-driven sagas and change data capture with DebeziumIdeal for backend and platform engineers preparing for senior and staff-level interviews.Chapters00:00 Distributed systems have no X-ray01:08 Why Kafka interviews matter02:19 acks=1 and the 200 missing payments03:27 Four layers of zero data loss05:21 The CAP theorem trade-off06:31 Throughput: taxis vs buses07:42 batch.size, linger.ms and compression09:38 When retries reorder messages11:38 Idempotent producers12:48 Message keys and partition ordering13:12 Flash sale: buffer exhausted14:22 Decoupling with a local buffer and circuit breaker16:18 Rebalance storms18:13 Eager vs cooperative sticky rebalancing19:25 Static group membership20:12 Hot partitions and the hot key problem23:18 Silent data loss after recreating a topic25:20 auto.offset.reset and topic versioning26:08 3 AM: 85 under-replicated partitions28:54 Why you never restart the broker29:42 Disk at 96%: the 15-minute response31:36 Throttled reassignment, Cruise Control, tiered storage33:36 Duplicates in consume-transform-produce34:47 Kafka transactions37:11 Exactly-once into PostgreSQL39:56 Head-of-line blocking from retries41:08 Tiered retry topics and DLQs42:42 Split brain44:38 KRaft and quorum math45:27 Log compaction tombstones48:33 Multi-tenant topic design50:07 Tenant tiers and client quotas51:40 Event-driven sagas53:15 Compensating transactions54:51 Change data capture with Debezium57:13 Recap58:46 Will offsets still matter in five years?#Kafka #SystemDesign #DistributedSystems #SoftwareEngineering #InterviewPrep #BackendEngineeringInterview Prep Podcast