Kafka patterns in this project

The reusable Kafka layer lives under backend src/lib/kafka. These patterns separate database work, publication, and consumption. Use fundamentals for the terminology.

Outbox

An outbox stores an intended event in the database before a dispatcher publishes it. This decouples a domain write from Kafka availability.

The normal sequence is:

1. Domain code creates an outbox row.
2. The dispatcher atomically claims PENDING -> DISPATCHING.
3. Successful publication completes DISPATCHING -> PUBLISHED.
4. Consumption enters UNDER_PROCESSING.
5. Processing ends at SUCCESS or PUSHED_FAILURE.

PREQUEUED holds work until release to PENDING. FAILURE represents a publication failure path. SUCCESS, PUSHED_FAILURE, and FAILURE are terminal in the ordinary transition helper.

A consumer can advance before the publisher finishes its database acknowledgment. The transition contract therefore also permits processing from DISPATCHING. markPublished uses compare-and-set and accepts already-advanced processing/terminal success without moving it backward.

Stale dispatch recovery probes the topic/partition before deciding whether to mark an existing publication or requeue. A timeout alone does not prove that nothing reached Kafka. Unknown-partition failures have a separate feature-controlled recovery path. Avoid manual status edits.

Source map

Module/function	Responsibility
src/lib/kafka/patterns/outbox/outbox-queue.ts	Create durable queue rows
outbox-publisher.ts: claimPendingEventForDispatch	Atomic pending claim
outbox-publisher.ts: markPublished, markPublishFailed	Conditional publication completion
outbox-publisher.ts: requeueStaleDispatchingEvents	Probe/recover stale dispatch work
outbox-tracing.ts: transitionOutboxEventStatus, getOutboxEventStatus	Shared trace/status access
outbox-status.ts	Allowed transitions and terminal-state checks
src/services/KafkaTester/modules/mailbox/processing	KafkaTester consumption and result state
src/services/KafkaTester/modules/runtime/kafka-tester.service.ts	All-in-one service orchestration

Outbox filenames above share the src/lib/kafka/patterns/outbox/ directory.

Idempotency

src/lib/kafka/patterns/idempotency.ts implements processKafkaMessageOnce. Its unique identity is consumerGroup + topic + partition + offset.

The function inserts ProcessedKafkaMessage and calls work(tx) in one database transaction. A failed transaction rolls back both. A unique conflict returns already_processed. Other failures propagate. The consumer skips terminal outbox work. For an already-processed result with incomplete success state, it reconciles the missing success transition without repeating projection work.

This protects the recorded transactional effects for that offset identity. A different group or republished record has a different identity. External effects need their own idempotency contract.

Sequential partition assignment

src/lib/kafka/patterns/sequential-partition.ts reserves a partition when creating a user. It uses MAX_USERS_PER_QueProcessor_PARTITION, creates a new assignment when capacity requires it, and synchronizes broker partition count through the admin client.

Key and explicit partition

Publication includes both key = String(userId) and the user's assigned partition. This keeps that user's ordered mailbox sequence on its persistent route. Preserve the database mapping and broker layout together.

Normal mailbox commands use the canonical mailbox user identity, not the browser login actor. Do not substitute a display email or Better Auth ID. See identity concepts.