Kafka fundamentals
Apache Kafka stores streams of records across server processes called brokers. MailGen uses it for durable background work. This page explains the terms needed for project architecture.
Producers, consumers, and brokers
A producer publishes records. A consumer reads them. A broker stores and serves records as part of the Kafka cluster.
Topics, partitions, and offsets
A topic names a logical stream. A partition is an ordered part of a topic. An offset identifies a record's position within one partition.
Kafka preserves order within a partition. It does not establish global order across topics or partitions. Choose the business ordering boundary, such as one mailbox, before choosing routing.
Keys and order
A key can route related records to one partition. MailGen mailbox work also uses an explicit persistent partition assignment. The producer supplies key = String(userId) and the assigned partition.
A shared key alone does not establish order between different topics. Partition-count changes can also affect default key hashing. Keep the explicit mailbox mapping and broker partition layout consistent.
Consumer groups
A consumer group divides a topic's partitions among its members. Within a group, one active consumer owns a partition at a time. More consumers than partitions do not create more parallel partition ownership.
Different groups maintain separate positions and can read the same stream independently. This is fan-out. Their effects on a shared application database still depend on the application's own business rules.
Delivery semantics
| Model | Behavior |
|---|---|
| At-most-once | Committing before processing can lose work after failure |
| At-least-once | Committing after processing can repeat work after failure |
| Exactly-once | Requires an appropriate atomic transaction/state contract, not merely a consumer flag |
The shared MailGen JSON consumer commits after its handler. Database deduplication protects the transactionally recorded effects for the same group/topic/partition/offset. It does not make arbitrary external API calls globally exactly-once. See project patterns.