GraphOneTimeUserOnboarder
GraphOneTimeUserOnboarder is the production Microsoft mailbox import worker. It consumes authorized Kafka commands and prepares the mailbox workspace. Start with Microsoft Graph setup.
Start a mailbox import
Use the numbered Connect and provision one mailbox procedure. It requires an available Microsoft connector backed by a prepared tenant connection and actor permission mail.onboarding. Select your owned connected account before choosing Start onboarding or Set up mailbox.
The start handler does not require a pre-existing mailbox row or mailboxId. The worker verifies the Graph user against the authorized Entra subject and creates or updates the local mailbox. This is distinct from creating the deployment's first tenant connection before the connector button becomes available.
Responsibilities
The worker resolves provider identity, creates or resumes checkpoints, reads folders and reference data, and imports message metadata. It writes body.html objects and updates newMails metadata. It exports raw.fts through Graph exportItems, parses attachments with FTS_Parser_Lib, and stores their files and metadata.
Production import requires Graph Application permission MailboxItem.Export.All and tenant administrator consent. Mail.Read or Mail.ReadWrite alone does not authorize exportItems. Export and FastTransfer attachment extraction are mandatory completion prerequisites. See Graph export permission setup.
After storage and indexing barriers pass, it assigns stable Kafka partition ownership. It releases held work, emits lifecycle states, and queues mailbox.onboarding.initializingDone. See the full sequence.
| Dependency | Use |
|---|---|
| Microsoft Graph | Folder/message delta and mailbox export |
| Kafka | Commands, lifecycle events, failures, final initialization operation |
| PostgreSQL | Metadata, checkpoints, retries, and export state |
| S3 | Bodies, raw export objects, attachment files |
FTS_Parser_Lib | Attachment extraction from FastTransfer bytes |
DB_Lib | Shared database context and models |
Runtime selection
The unified runtime selects this worker. Use the supported development lifecycle for that environment. Do not start a second standalone worker beside it.
For a separately prepared standalone development environment, the legacy launcher runs from mailGen/ECS_MovePST_To_TempSQL:
bash scripts/run-onetimedev.sh
The root resetDevServers.sh erases the selected complete instance, not just this worker. Use it only for an intentional full data reset with required backups. Do not run it beside another active complete stack.
Configuration reference
GraphOneTimeUserOnboarder/.env.example lists standalone settings, but its Graph credential names are stale. The actual worker requires the BACKEND_* names below, including standalone execution. Root Compose passes these names through unchanged. Keep one credential source for that runtime.
| Variable | Meaning and format | Required when |
|---|---|---|
POSTGRES_CONNECTION_STRING | Npgsql string with host, port, database, username, and password. Use the database selected for this instance | Every run |
KAFKA_CLIENT_ID | Kafka client identity | Kafka consumption/publication |
KAFKA_NETWORK | Selected lan or tailscale network | Kafka consumption/publication |
KAFKA_LAN_BOOTSTRAP_SERVERS, KAFKA_TAILSCALE_BOOTSTRAP_SERVERS | Comma-separated host:port broker list for the selected network | Kafka consumption/publication. All advertised brokers must be reachable |
KAFKA_SECURITY_PROTOCOL, KAFKA_SASL_MECHANISM | Protocol and authentication mechanism | Kafka connection |
KAFKA_ADMIN_USERNAME, KAFKA_ADMIN_PASSWORD, KAFKA_APP_USERNAME, KAFKA_APP_PASSWORD | Separate cluster admin and app credentials | Authenticated Kafka connection |
BACKEND_TENANT_ID, BACKEND_CLIENT_ID, BACKEND_CLIENT_SECRET | Entra tenant, app ID, and application credential | Microsoft import |
GRAPH_WEBHOOK_URL, GRAPH_LIFECYCLE_WEBHOOK_URL | Public HTTPS WebhookCatcher callbacks | Subscription creation. Use this deployment's hostname |
GRAPH_WEBHOOK_CLIENT_STATE | Stable random validation secret | Graph subscriptions |
S3_ENDPOINT, S3_REGION, S3_ACCESS_KEY, S3_SECRET_KEY, S3_BUCKET_EMAIL_BODIES, S3_FORCE_PATH_STYLE | Object-store endpoint, region, credentials, bucket, and addressing mode. Local examples: http://127.0.0.1:8333, us-east-1, true | Body, raw-export, and attachment writes |
OPENSEARCH_URL, OPENSEARCH_INDEX, OPENSEARCH_USERNAME, OPENSEARCH_PASSWORD | Endpoint, target index, and credentials. Credentials can be absent when authentication is disabled | Indexing |
ONBOARDING_RETRY_* | Per-step attempts, delays, and jitter in milliseconds | Optional tuning. Consult the committed example for individual settings |
ONBOARDING_REPLAY_RUN_ID, ONBOARDING_REPLAY_STEP | Explicit run and checkpoint for authorized replay | Planned replay only. Leave unset otherwise |
Implementation locations
Paths are relative to the motherrepo:
| Path | Contract |
|---|---|
mailGen/ECS_MovePST_To_TempSQL/GraphOneTimeUserOnboarder/Configuration/OnboardingRuntimeOptions.cs | Required runtime variable names |
mailGen/ECS_MovePST_To_TempSQL/GraphOneTimeUserOnboarder/Processing/OnboardingPipeline.cs | Checkpoints, export barrier, lifecycle publication, final queue command |
mailGen/ECS_MovePST_To_TempSQL/GraphOneTimeUserOnboarder/Services/GraphMailboxDumpService.cs | Graph metadata and export calls |
mailGen/msmicroservices-architexture/src/services/GraphPullPoller/modules/graph-sync/graph-sync-scheduler.ts | Later Graph synchronization scheduling |
mailGen/msmicroservices-architexture/src/services/QueProcessor/modules/sync/apply-handler.ts | Ordered application of fetched changes |
Storage contract
users/{providerMailboxUserId}/emails/{emailId}/body.html
users/{providerMailboxUserId}/emails/{emailId}/raw.fts
users/{providerMailboxUserId}/emails/{emailId}/attachments/{emailId}_|_{filename}
Current imports use the users/ layout. See storage reference for metadata and compatibility reads. The worker does not adopt destructive startup cleanup from GraphExportApiTester. Use import recovery for incomplete work.