Consumers, Consumer Groups, Rebalancing, and Offset Management
Understand how Kafka consumers scale, share work, track progress, and recover after failures or restarts.
Inside this chapter
- What a Consumer Does
- Consumer Groups
- Rebalancing
- Offset Commit Strategies
- At-Least-Once Thinking
- Real Usage Example
Series navigation
Study the chapters in order for the clearest path from Kafka basics and local setup to stream processing, platform operations, cloud usage, and advanced event-driven architecture thinking. Use the navigation at the bottom to move smoothly through the full tutorial series.
What a Consumer Does
A consumer reads records from Kafka topics. Consumers may process events for business workflows, trigger notifications, write to databases, feed analytics pipelines, or update caches and search indexes.
Consumer Groups
A consumer group is a set of consumers working together to process a topic. Each partition is assigned to one consumer within the group at a time. This enables parallel processing while avoiding duplicate work inside the same group.
Rebalancing
When consumers join or leave a group, Kafka may rebalance partition assignments. Rebalancing is normal but can briefly pause processing. Students should understand that scale and elasticity come with coordination cost.
Offset Commit Strategies
- Automatic offset commit
- Manual synchronous commit
- Manual asynchronous commit
Offset strategy matters because it shapes duplicate-processing risk, failure recovery behavior, and operational confidence.
At-Least-Once Thinking
Many Kafka consumer flows are designed for at-least-once processing, meaning a record may be processed again after failure or restart. Consumers therefore often need idempotent logic or deduplication safeguards.
Real Usage Example
A shipment consumer group may process order events and schedule delivery jobs. If the system fails after creating the job but before offset commit, reprocessing may happen. Good design plans for that reality.