Kafka architecture and source map
The backend separates reusable transport/pattern code from KafkaTester-specific orchestration. Read fundamentals before using this source map.
Reusable core
Under src/lib/kafka/ | Responsibility |
|---|---|
env.ts | Environment loading and Zod validation |
config.ts | Application/admin client configuration |
producer.ts | JSON send with optional explicit partition |
consumer.ts | JSON decoding, handler execution, and manual offset commit |
partition-admin.ts | Topic creation and partition-count checks/expansion |
topics.ts | Topic registry |
core-service.ts | Shared service lifecycle |
patterns/ | Outbox, idempotency, and sequential partition assignment |
Shared Kafka types reside under src/lib/types/kafka/. Database models include MailboxUser, SequentialTopicPartition, OutboxEvent, and ProcessedKafkaMessage in Prisma's kafka schema.
KafkaTester
Under src/services/KafkaTester/ | Responsibility |
|---|---|
runKT.ts | Fastify entrypoint and request handling |
events/mailbox-event.ts | Mailbox event contract |
modules/runtime/kafka-tester.service.ts | Consumer/dispatcher orchestration |
modules/mailbox/ | Users, projection, processing, and workers |
scripts/producer-daemon.ts | Interactive producer |
Use current module paths rather than the older flat services/ layout.
Runtime roles
The producer queues or explicitly publishes requested work. The outbox dispatcher claims/publishes pending records. The mailbox consumer handles the event and records processing results. dev:kt combines consumer, dispatcher, and HTTP health endpoint.
Choose all-in-one or split CLI runtime. Do not accidentally run both against one exercise database/stream.
State ownership
The shared patterns/outbox/outbox-publisher.ts owns dispatch claims and publication completion. KafkaTester mailbox processing owns its consumption result. Shared tracing/status helpers enforce transitions. See exact pattern behavior.
Kafka producer credentials use the application role. Topic/partition operations use the separate administrator role. Clients disable automatic topic creation. The producer enables idempotence and acks=all. The consumer disables automatic commits and commits after the handler.
Mailbox routing includes a canonical user key and persistent explicit partition. Kafka ordering is partition-local. Database offset deduplication does not establish exactly-once external provider writes. The application must still reconcile accepted operations with provider state.
For settings, startup, and operational checks, use the developer guide.