Use knowledge libraries and document retrieval

A knowledge library stores documents for search and authorized AI use.

Sign in and select a mailbox, then open Knowledge. Creating, managing, searching, sharing, and automatic-email use require their corresponding knowledge.* permissions. Processing also needs configured storage, scanning, extraction, and embedding services. Choose the deployment path in environment configuration, then check the environment reference.

An administrator must configure the active native OpenAI embedding provider/model and your group model access in the shared AI catalog. Knowledge permissions alone do not grant model access.

Create a library, choose its folder, and upload a supported file or ZIP. Watch each file until it reaches ready. Only then can retrieval use its indexed content. An archive reaching ready does not mean every imported child is already searchable.

Ownership and access

A new library is private to its creating mailbox. All folders and files inherit the library access control list (ACL). There are no folder/file overrides. Tenant group shares can grant five independent rights: read, manage, search, share, and useInAutomaticEmail. Each also needs the matching feature permission. Knowing a UUID never grants access across tenants.

Read access does not authorize automatic outgoing email. Automatic-email retrieval checks search and outbound-use rights together. The backend rereads group membership and library state. The model supplies queries and selected library/folder IDs, never tenant, actor, or mailbox ownership.

A sharer cannot delegate rights they do not hold. Only the owner mailbox can delete the complete library. File or folder deletion immediately excludes content from retrieval. Later Qdrant/S3 cleanup does not control authorization.

Upload and processing

The browser uploads sequential 512 KiB parts. The backend reserves declared storage at upload start. An owner-bound database lock prevents concurrent uploads from exceeding mailbox quota. A repeated part is idempotent only with the same SHA-256 hash. The backend rejects different content or part indices.

Default limits are 50 MiB per file and 250 MiB per ZIP. They also allow 1 GiB expanded content, 2,000 archive entries, and 2 GiB per mailbox. Tenant policy can change these limits. A single file cannot exceed 256 MiB. The ZIP ceiling is 1 GiB and the entry ceiling is 100,000. The interface returns effective limits.

Supported extensions are txt, md, csv, htm, html, json, doc, docx, xls, xlsx, ppt, pptx, pdf, png, jpg, jpeg, gif, webp, tif, tiff, bmp, and zip.

Originals remain as checked parts in S3. Reads check size and hash again. ClamAV scans the complete stream before extraction. Scanner failure blocks extraction and embedding.

ZIP import reads entries one at a time and preserves relative folders. Name collisions create numbered copies. Traversal or symlinks stop import. Encrypted entries, nested ZIPs, unsupported formats, and oversized entries produce warnings. Entry and expanded-size limits apply. Durable entry identity prevents duplicates after restart.

The isolated Apache Tika 4 endpoint is PUT /tika/json/html. Extraction uses tk:content, not the old X-TIKA:content key. OCR uses hun+eng. Document, page, sheet, and heading citations come from extracted structure, not model invention. Partial or empty extraction is an error.

IUDG processes durable jobs one at a time without a separate Node service. Leases and heartbeats protect concurrent processes. The maximum is three attempts. Retry delays are 30 and then 60 seconds. Errors remain visible and support an explicit retry.

Embedding and retrieval

Embedding converts text into numeric vectors for similarity search. The configured model is text-embedding-3-small with 1536 dimensions. It needs an active tenant OpenAI catalog model/provider and an applicable group model grant for the user. The model must use apiType: embedding, 1536 dimensions, and reasoning default. The provider must resolve to native api.openai.com. An arbitrary OpenAI-compatible endpoint does not satisfy this embedding path.

Credentials come from encrypted provider configuration.

Embedding uses normal model admission, context limits, checked HTTPS transport, and the usage ledger with source knowledge.embedding. Qdrant stores identifiers, not document text. PostgreSQL supplies result text and citations after fresh access and state checks. Old vectors cannot expose revoked or deleted content. searchKnowledge(input, "automatic-email") is the stricter outbound entry point.

API reference

Endpoint	Contract
GET /knowledge?libraryId=…&cursor=…	Accessible libraries, folders, 100 files per page, nextCursor, shares, selectable tenant groups, and limits.
POST /knowledge	library.save/delete, folder.save/rename/delete, share.save/delete, upload.start/part/finish, file.rename/delete/retry, and search.
GET /knowledge/file?id=…	Rechecked original download for a ready file, with attachment disposition and no-store. It does not serve inline HTML.
AdminService GET/PUT knowledge-limits	Reads/manages tenant knowledge.limits with admin.ai.models.read/manage.

Strict action contracts are in backend src/lib/knowledge/contracts.ts and the associated type modules. Search returns {query, results}. Each result has chunkId, fileId, libraryId, path, text, score, and citation.

Operations and failure states

In the local Compose deployment, service qdrant uses the private knowledge network. knowledge-extractor uses the isolated extraction network. Neither publishes a host port. IUDG accesses them. Browsers use only application APIs.

The extractor has a read-only filesystem, bounded /tmp tmpfs, CPU/PID/memory limits, and no capabilities. Its configuration is docker/knowledge/tika-config.json. Its image definition is docker/images/knowledge-extractor.Dockerfile.

IUDG uses QDRANT_URL, QDRANT_API_KEY, and KNOWLEDGE_EXTRACTOR_URL for the configured services. Existing object-storage configuration supplies S3 access. Scanner transport has two supported paths:

• TCP: CLAMAV_HOST and CLAMAV_PORT select the scanner. Defaults are clamav and port 3310.
• Authenticated HTTPS: configure both CLAMAV_SCAN_URL and CLAMAV_SCAN_TOKEN. The URL must target /scan.

Defining either HTTPS variable selects that transport. An incomplete pair, connection failure, or invalid verdict fails scanning without TCP fallback. Only an exact clean verdict permits extraction. Choose the applicable transport through the environment reference. Do not put scanner tokens in public examples.

The Qdrant key never goes to the browser. The first indexing creates collection ecas_knowledge_small_1536. Changing model or dimensions needs reindexing.

The pipeline is uploading → queued → scanning → extracting → indexing → ready. ZIP children receive separate jobs. warnings records skipped entries. skipped identifies encrypted documents. failed identifies stopped processing. Public errors contain neither credentials nor complete original content.

The worker checks every 5 seconds, writes heartbeats every 15 seconds, and treats a 180-second lease as abandoned. Claiming increments attempts, including process failures. file.retry restarts a fully uploaded failed file with fresh actor context and can incur embedding cost.

An uploading file inactive for 24 hours becomes deleted with cleanupPending. This releases reserved mailbox quota immediately. Cleanup removes original S3 parts and part metadata. Each uploaded part refreshes activity. This expiry does not affect queued, processing, or ready files. Retrieval does not reindex automatically.

Symptom	Check
Stays uploading	All numbered parts and upload.finish must arrive. receivedBytes must equal size.
Scan failure	ClamAV availability, signatures, and INSTREAM limit. Bypassing scanning is unsupported.
Extraction failure	Extractor logs, OCR packs, timeouts, tmpfs, and output limits. Encrypted or partial documents cannot index.
Embedding failure	Active OpenAI provider, catalog model, model grant, context limit, and usage ledger.
No results	File is ready, selected scope includes it, and current feature permissions and ACL permit search.
Deferred cleanup	Qdrant/S3 availability. Deleted metadata already blocks reading and retrieval.

Backups need PostgreSQL metadata, S3 original parts, and Qdrant storage together. Qdrant alone has no document text. Library deletion stores a tombstone before cleanup removes vectors, originals, and chunks. Failed cleanup retries on a later sweep. Do not remove the tombstone before cleanup completes.

Extractor resource limits

One parser fork uses a maximum 768 MiB Java heap and two logical CPUs. The server heap limit is 256 MiB. The container has 2 GiB memory and 512 MiB /tmp tmpfs. Document spooling and parser scratch files share that tmpfs.

The per-file policy ceiling and HTTP request ceiling are both 256 MiB. The default stays 50 MiB. This does not guarantee extraction of every compressed 256 MiB document. Excessive scratch, OCR, or response size produces a visible failure. The archive worker applies the per-file limit to each child.

Readiness sends ten-byte ecas-ready text to /tika/text with Bash TCP support. It expects HTTP 200 and identical output. This checks the parser path, not only an open port. No extra curl dependency is required. Healthcheck configuration takes effect when the container is recreated through the authorized deployment workflow.