inbuxa-server/docs/spec/features/journaling.md

Feature spec: journaling

Status: approved 2026-09-28, with the answers under Settled. Not a rebuild of an upstream feature, so it has no line in SPEC.md §4's table. Rule IDs: JR-.

Provenance

Written for the record SPEC.md §3 rule 3 asks for. Sources, and nothing else:

Source	License	Used for
This repository at 94a3a76 (2026-09-28): crates/smtp/src/inbound/data.rs, inbound/rcpt.rs, queue/spool.rs, outbound/delivery.rs, crates/common/src/network/mta.rs, crates/features/src/{hold,audit,mailflow,undelete}, crates/store/src/write/{mod,blob}.rs, crates/jmap/src/inbuxa/hold_export.rs	AGPL-3.0-only	Where every message passes, what the envelope holds, how holds keep blobs, how the audit chain and hold export work
inbuxa-drafts/queue/journaling.md	Own	What John asked for, and the gaps to settle
The DLP and mail flow rules spec, the audit-hold-lock spec, the personal-data catalog spec	Own	Conditions, the audit log, legal holds, roles, the catalog check
RFC 5321, RFC 3461 (DSN, ORCPT), RFC 2046 (message/rfc822), RFC 5322	Public	The envelope, the original recipient of an expanded list, the report's shape

No Enterprise-only file or snippet was used, and no third-party journaling product or report format was consulted: the journal report below is our own layout of the SMTP envelope around the untouched message.

What it is

A journal is a copy of each message the server handles, captured in transit with its envelope (the real sender and every recipient, including Bcc and the members of lists), kept where nobody can change or remove it until its retention ends, or sent to an outside archive. It sits beside two things that exist:

• Legal hold keeps what's in chosen mailboxes, including what their owners delete. It starts when a hold is placed and can't see Bcc or what was sent from a mailbox that no longer exists.
• The audit log records what people and the server did, never the mail.

A journal answers the question neither can: what went through, to whom, from the day it was turned on.

Out of scope: journaling mail stored before it's turned on, files, calendar and contacts, IMAP APPEND (a mail app saving to its own Sent folder sends nothing), and mail a mail app sends through another server.

Nothing in code, docs, UI text or output claims the product meets a legal or regulatory standard. The pages say what's captured, where it's kept and for how long.

1. What exists today

Checked by reading the code at 94a3a76:

Need	Today
One place all mail passes	MessageWrapper::queue (queue/spool.rs ~L375). SMTP, JMAP submission (jmap/src/submission/set.rs builds a local session and runs queue_message), inbound mail, Sieve redirects and vacation replies, and DSNs all queue through it. Local and remote delivery both start from the queue.
The envelope	At queue time: mail_from, every rcpt_to (Bcc included), the authenticated account with its groups and tenant, the queue id. Lists are already expanded at RCPT (rcpt_resolve → RcptResolution::Expand, inbound/rcpt.rs); the list address survives as each member's ORCPT (dsn_info).
A copy out	Sieve at DATA, milters and MTA hooks can send one, but all run before DLP and transport rules, so they miss recipients the rules add, and a Sieve copy carries no envelope.
Keeping a blob nobody can delete	No "undeletable" flag. Blobs are content-addressed (can't be edited); a BlobLink::Temporary { until } keeps one until until. Legal hold uses until = year 9999.
A record nobody can quietly change	The audit log's per-node SHA-256 chain (features/src/audit/log.rs): each entry carries prev, the head is asserted on append, purge leaves a floor hash, verify walks it.
Export	Hold export (LH-12): a ZIP of .eml files, manifest.csv with a SHA-256 per file, manifest.sha256, capped at 2 GiB.
Conditions by sender, recipient, group, tenant	The mail flow engine (features/src/mailflow/engine.rs), at DATA.

2. Design

2.1 Where the copy is taken (JR-1, JR-2)

JR-1. The journal is taken in MessageWrapper::queue, after the message is spooled, behind one // inbuxa: marked block. That's after DLP and transport rules, so the envelope is the one the message actually leaves or arrives with, and it covers every path that queues mail.

JR-2. What isn't journaled: journal reports themselves (they carry a queue flag, so a report to an outside archive can't journal itself), and the server's own DMARC and TLS reports. DSNs and Sieve redirects and vacation replies are journaled (question 4). A message refused at DATA (DLP block, a transport rule's refusal) was never accepted and isn't journaled; the audit log already records it. A message held for DLP review is journaled when it's queued, which is when it's held, with the hold noted in the entry.

2.2 The journal report (JR-3, JR-4)

JR-3. Each copy is a journal report: a new message whose first part is text/plain, one field a line:

Sender: [email protected]
Signed in as: [email protected]
Subject: Q3 figures
Message-ID: <…>
Queue ID: 1a2b3c…
Received: 2026-09-28T14:03:11Z
Direction: outgoing
To: [email protected]
Cc: [email protected]
Bcc: [email protected]
Expanded: [email protected] -> [email protected], [email protected]
Held for review: yes

and whose second part is the message as queued, byte for byte, as message/rfc822. Bcc: lists envelope recipients that aren't in the message's To or Cc headers. Expanded: groups the members of a list under the list address, from their ORCPT. Recipients a transport rule added say so (Added by rule: <name>). The field names are fixed English (they're a record, not interface text), so a script can read them.

JR-4. One report per queued message, with the whole envelope, whatever the scope matched on (§2.4). A message to 40 recipients is one report, not 40.

2.3 Where reports go (JR-5 to JR-8)

Each journal has a destination (question 1):

JR-5. The built-in journal. Records under a new prefix J in SUBSPACE_INBUXA: queue id, received time, direction, sender, recipients, the tenant(s), which journal matched, the report's blob hash and size, its SHA-256, and the time it may be purged. The report blob is kept by a BlobLink::Temporary { until } set to the end of its retention. There is no JMAP set or destroy for entries: nothing in the product changes or removes one before its time.

JR-6. The chain. Each entry carries the SHA-256 of the entry before it, one chain per node, the same construction as the audit log (and its code, generalized rather than copied). The console's Check the journal walks it, and every blob's hash against its entry, and says what it found. Someone with the server's disks can still remove data, and the chain is how that shows; the docs say exactly that, and don't say it can't happen.

JR-7. An outside archive. The report is queued to an address (the archive's journal mailbox) like any mail, with the queue's retries. A report the archive refuses permanently, or can't take within the queue's limit, goes into the built-in journal instead and raises a warning on the Overview (question 6). Delivery is by the ordinary queue, so TLS and routing settings apply; a queue route can be chosen for it.

JR-8. Both: the built-in journal and an outside archive.

2.4 Which mail: journals and their scope (JR-9 to JR-11)

JR-9. A journal is a named object (inbuxa:Journal): on or off, a destination, a retention, and a scope. The scope is who: everyone, or senders and recipients in chosen accounts, groups, domains or tenants, and which direction: outgoing, incoming, internal, any. A message is journaled once per journal whose scope any sender or recipient is in; two journals with the same destination never write the same message twice.

JR-10. By what's in it: a new mail flow rule action, Journal it, names a journal. The rule's conditions (detectors, words, attachments, headers) decide; the copy is still taken at queue time (the rule only marks the message). This is the rule-based journaling the queue note called premium; here it's one more action, not a separate tier (question 2).

JR-11. Scope is evaluated from the envelope and directory membership at queue time (Server::member_of), no message parsing, so journaling everything costs a lookup per recipient and one blob write per message.

2.5 Retention and legal hold (JR-12 to JR-14)

JR-12. Each journal has a retention in days (question 3). An entry keeps the retention it was written with: shortening a journal's retention applies to new entries only, so nobody can empty the journal by editing a number. Lengthening it applies to new entries too, and the console says so.

JR-13. Purge runs in the daily maintenance, removes entries past their time and drops their blob link, and leaves a floor hash so the chain still verifies, as the audit log does. An entry whose sender or any recipient is under a legal hold isn't purged while the hold lasts (holds_on, read uncached, as holds are everywhere).

JR-14. Deleting an account doesn't remove its journal entries; they end with their retention (question 7). The privacy catalog says so.

2.6 Search, reading, export (JR-15 to JR-17)

JR-15. Management › Compliance › Journal: search by sender, recipient, date range, direction, subject words (from the report's header fields, not the body: no full-text index of the journal in this version). Results list the envelope; Read… opens the report.

JR-16. Export a search as a ZIP in the hold export's shape: the reports as .eml, manifest.csv with the envelope columns and a SHA-256 per file, manifest.sha256, the same 2 GiB cap. Export runs as a task and the result is a blob owned by the person who asked for it.

JR-17. Every search, read and export is in the audit log, with who and the search terms; so is every change to a journal.

2.7 Permissions (JR-18)

JR-18. New permissions after the DLP set (680 onward): sysJournalGet / sysJournalUpdate (see and change journals), sysJournalSearch (search and read entries), sysJournalExport. Superuser only, by default. The officer grant audience adds Get, Search and Export to the Compliance Officer; administrators configure journals but don't read them unless granted Search (question 5). Journals are server-level, with a tenant scope, as DLP rules are; nobody in a tenant reaches them.

2.8 Privacy catalog

New objects get catalog entries (resources/privacy/catalog.toml): inbuxa:Journal (none), inbuxa:JournalEntry (mail content and envelope, kept for the journal's retention, access audited, not erased with the account). privacy-check.py enforces it.

2.9 Mixed versions, clusters, rollback

• Entries and journals live in the shared data store; report blobs in the blob store. A node-local blob store (FileSystem, or RocksDB/SQLite as the blob store) on a cluster means a node's journal lives on that node; the console warns when journaling is on and the blob store isn't shared.
• During a rolling upgrade a node on the old version doesn't journal. The console says so when nodes report different versions; the release notes say to turn journals on after every node is upgraded.
• Rollback: the new prefix and the queue flag are ignored by an older version; nothing in the queue's archived format changes (the "journal report" flag rides in the existing message flags if one is free, else in a side key by queue id; checked in phase 2 before writing code).

2.10 Cost

One extra blob per journaled message (the report wraps the original, so it doesn't share its hash), plus one small record. The console shows the journal's size and growth per day on the journal page, from the entries.

3. Console

• Management › Compliance › Journaling: journals (name, scope, destination, retention, on/off), described in words like DLP rules ("Journal all mail to and from Finance into the built-in journal, kept 7 years"); Check the journal.
• Management › Compliance › Journal: search, read, export.
• The mail flow rule editor gains Journal it.
• The Overview warns about undelivered outside reports (JR-7) and a node-local blob store (§2.9).

4. Webmail

Nothing. People aren't told a message was journaled, as they aren't told about legal hold; the docs say journaling exists and what it captures.

5. Tests

Unit: the report's fields (Bcc computed from headers, list expansion from ORCPT, rule-added recipients), scope matching, retention arithmetic, the chain. Integration (tests/src/system/): SMTP and JMAP sends, inbound mail, internal mail, a list and a Bcc recipient, a DLP-held message, a Sieve redirect; the report equals the queued bytes; no set/destroy; shortening retention doesn't touch existing entries; a hold stops purge; account deletion leaves entries; an outside archive that refuses falls back to the built-in journal; export manifest hashes; audit records for search, read, export.

6. Phases

1. This spec, approved.
2. Capture at the queue, the report, the built-in journal with its chain, retention and purge, holds; inbuxa:Journal and inbuxa:JournalEntry; catalog entries; tests.
3. Outside archive and the fallback; Journal it in mail flow rules.
4. Search, read and export (a task), audit records.
5. Console pages; docs; a row in inbuxa-drafts/divergence-log.md.

Each phase is its own PR with tests; releases as John decides. Like DLP, it stays out of production until John says.

As built

Phase 2 (feature/journal-capture), where it differs from the design or fills in what it left open:

• The chain is the journal's own (crates/features/src/journal/ entries.rs), not the audit log's code shared. Entries expire out of chain order (each keeps its journal's retention, and holds keep some longer), so a link names its entry by SHA-256 instead of holding it: purging removes the entry, its indexes and its report's blob link, and writes a purge marker; the link stays. An entry missing without a marker is a broken chain. Purged links at a chain's start are cleared and a floor recorded, as the audit log does.
• If the copy can't be taken, the message isn't queued: the sender gets a temporary failure and tries again. Nothing leaves unjournaled.
• The report says Authenticated: yes|no instead of the signed-in account (the queue doesn't keep which account it was). Added by rule comes with Journal it in phase 3. A recipient given with an ORCPT that names another address counts as expanded from that address.
• Journal reports the server queues carry message flag bit 48 (FROM_JOURNAL); an older version ignores the bit.
• Permissions 680–683: administrators get sysJournalGet/Update; the Compliance Officer gets Get, Search and Export. So that an administrator can still appoint an officer (and grant reading as settled answer 5 describes), whoever holds sysJournalUpdate may grant Search and Export without holding them; the role change is in the audit log.
• inbuxa:JournalEntry (get, query) and Check the journal over JMAP come in phase 4 with search, so every read is audited from the first version that allows one. Phase 2 has inbuxa:Journal only.
• Outside archives (a journal's destination) come in phase 3; every journal writes to the built-in journal until then.

Known gaps

• A message a person saves to Sent over IMAP, or sends through another server, never reaches the queue.
• Mail stored before journaling is on isn't journaled (legal hold covers mailboxes).
• Search reads envelope and header fields, not bodies.
• Group accounts (GroupAccount) resolve as one account, not members; their mail is journaled under the group's address.

Settled

John, 2026-09-28, all seven as recommended:

1. Destinations: the built-in journal, an outside archive by address, or both, per journal (JR-5, JR-7, JR-8).
2. Scope: everyone, or chosen accounts, groups, domains and tenants by direction, plus a Journal it rule action; no standard/premium split (JR-9, JR-10).
3. Retention: no default; 30 days to 10 years, picked when a journal is turned on; existing entries keep theirs (JR-12).
4. Which mail: everything queued, including DSNs, Sieve redirects and vacation replies, except DMARC/TLS reports and journal reports (JR-2).
5. Who reads it: administrators configure; Compliance Officers search, read and export; administrators read only if granted Search (JR-18).
6. An outside archive that won't take a report: kept in the built-in journal, with a warning (JR-7).
7. Deleted accounts: journal entries stay until their retention ends, and the catalog says so (JR-14).