Why Is Kafka Fast? Part 2: Partitions, Replication, and Consumer Groups

Kafka on a single machine is already fast—sequential writes, the page cache, zero-copy. If you caught part one, you know one box can push a serious amount of throughput. But here's the thing: single-node speed isn't really Kafka's superpower. The superpower is that when you add machines, throughput scales up almost linearly. And the whole trick behind that is one idea—partitions.

So let's start there, because partitions are the unit of performance in Kafka. The core principle is this: every partition is completely independent. A partition is just an append-only log—messages get tacked onto the end, in order. And because partitions don't talk to each other, reads and writes on different partitions never step on each other's toes. They can run in parallel, on different brokers, at the same time.

This is a real departure from how something like RabbitMQ worked, at least before version 3.8. Over there, a single queue was single-threaded on dispatch—didn't matter how many consumers you attached, that one queue was the bottleneck. Kafka just doesn't have that ceiling. You want more parallelism? Add more partitions.

Picture a topic called "orders" split into three partitions, spread across three brokers. Partition zero lives on broker one as its leader, with a copy on broker two. Partition one leads on broker two, backed up on broker three. Partition two leads on broker three, backed up on broker one. Three brokers, three partitions, load spread evenly. And if you bolt on a fourth broker, Kafka rebalances the partitions automatically to make use of it.

Now, you might be tempted to think more partitions is always better. It's not—there's a tradeoff. On the upside: more partitions means more consumers running in parallel, near-linear scaling, and since each partition is smaller, leader elections happen faster when something goes wrong. But there's a cost. Every partition eats file handles and OS threads on the broker. Every partition has its own log segment on disk, so pile up too many and you're drowning the broker in file management. And here's the subtle one—more partitions doesn't automatically mean lower end-to-end latency. That depends entirely on where your actual bottleneck is. A decent rule of thumb to start with: keep it under a hundred partitions per broker. This is not a "crank it to the max" situation.

Okay, so partitions give you the parallelism on the storage side. The thing that actually cashes in on that parallelism when reading is the consumer group. A consumer group is just a set of consumers that split the work of reading a topic, and the rule is each partition gets owned by exactly one consumer in the group.

Go back to our orders topic with three partitions. If you've got one consumer in the group, it handles all three partitions by itself. Add a second and third consumer, and now each one owns a single partition—your throughput roughly triples. But—and this is the catch people trip on—if you add a fourth consumer to that group, it just sits there idle. There's no fourth partition for it to own. Your partition count is the hard ceiling on parallelism. More consumers than partitions buys you nothing.

Here's the part that's genuinely elegant, though: multiple consumer groups can each read the same topic completely independently. Each group tracks its own position—its own offsets. So one stream of events can feed a bunch of different downstream services, and they all consume at their own pace without blocking each other. That's Kafka's broadcasting ability in a nutshell. Your analytics service and your billing service can both chew through the same order events, one racing ahead, one lagging behind, and neither one cares what the other's doing.

Now let's talk durability, because that's where replication comes in—and durability always costs something. Each partition has one leader and some number of followers. Producers only ever talk to the leader. The leader writes the message, and the followers asynchronously pull that data over and replicate it.

The knob that controls how careful you're being is a setting called "acks." Think of it as: how many confirmations do you wait for before you consider a write done? With acks set to zero, you don't wait for any confirmation—fire and forget. Lowest latency you can get, but if the broker crashes, that message is just gone. Set acks to one, and you wait for the leader to confirm it wrote the message. That's the middle ground—decent latency, but you can still lose data if the leader dies before the followers copied it over. And then acks equals "all," sometimes written as negative one, means you wait for all the in-sync replicas to confirm. Highest latency, lowest risk of losing anything.

In production, the common setup is acks equals all combined with a setting called min-insync-replicas set to two. What that combo buys you is: any single broker failure won't lose your data. The price you pay is that every write has to wait for a follower to confirm before it's acknowledged, so latency goes up. This synchronization is really the central tuning dial in Kafka performance—you're trading speed for safety, and you get to decide where on that dial you sit.

Which brings us to ISR—in-sync replicas—because that acks-equals-all guarantee is only as good as the ISR behind it. ISR is the set of replicas that are actually keeping up with the leader. And Kafka defines "keeping up" with a time window—there's a setting, replica-dot-lag-dot-time-dot-max-dot-ms. If a follower hasn't fetched from the leader within that window, it gets kicked out of the ISR. It's fallen behind, so it no longer counts.

Why does this matter? When a leader fails, the component called the controller elects a new leader—and it only ever picks from the ISR. That's the whole point. Because every replica in the ISR is caught up, whichever one gets promoted is guaranteed to have all the committed messages. No data loss on failover.

And here's a design decision I really like. Suppose your ISR shrinks down until only the leader is left—everybody else fell behind. But you told Kafka min-insync-replicas is two. Now an acks-equals-all write comes in. Kafka doesn't quietly relax its promise. It refuses the write. It would rather reject your data than silently lower the consistency guarantee it made you. That's an explicit, deliberate choice—fail loudly instead of lying about durability.

Let me quickly place Kafka against its neighbors, RabbitMQ and Pulsar, because they're not interchangeable. On storage, Kafka is an append-only log—messages stick around based on time or size, and consumers can re-read them. RabbitMQ is a queue: once a message is consumed, it's deleted. Pulsar separates storage out into a system called BookKeeper. On scaling, Kafka grows linearly by adding partitions; RabbitMQ leans on federation and shovel; Pulsar shards topics. On the consumer model, Kafka is pull-based—consumers ask for data. RabbitMQ pushes data to consumers. Pulsar does both. And on ordering, Kafka guarantees order within a partition, RabbitMQ within a queue, Pulsar within a partition. The one-liner version: Kafka's home turf is high-throughput event streams, RabbitMQ shines at complex routing and task queues, and Pulsar's story is multi-tenant, cloud-native workloads.

So when should you actually reach for Kafka? When you're ingesting logs or metrics at millions of events per second. When you're doing event sourcing. When multiple downstream services all need to consume the same event stream. Or when you need historical replay—the ability for a consumer to rewind and re-read from any point.

And when should you not? If it's a task queue where each task should be done by exactly one worker, something like SQS or RabbitMQ fits better. If you need complex routing or filtering, RabbitMQ's exchange-and-binding model is far more flexible. And if it's low-traffic, low-latency, point-to-point messaging, Kafka's overhead just isn't worth it at small volumes.

So let me leave you with the three things that really matter here. First: the partition is the atom of everything. It's the unit of parallelism, the unit of scaling, and the thing that caps how many consumers can work at once—choose your partition count thoughtfully, because it's your ceiling. Second: replication and the acks setting are a dial, not a switch. You're consciously trading latency for durability, and acks-equals-all with min-insync-replicas of two is the sweet spot most production systems land on. And third: the ISR is what makes the durability promise real—by only ever electing a caught-up replica as the new leader, and by refusing writes rather than quietly weakening its guarantees, Kafka keeps its word even when brokers fall over. Get those three ideas, and you understand why Kafka scales the way it does.