Journaling: capture at the queue, the built-in journal, retention #115

Merged

jcoffey-dev merged 1 commits from feature/journal-capture into main

2026-09-29 04:11:40 +00:00

jcoffey-dev commented

2026-09-29 03:46:09 +00:00

Owner

Phase 2 of the journaling spec (#113).

Copy taken in MessageWrapper::queue, after DLP and transport rules; a failure to journal means the message isn't queued (temporary failure).
Journal report: envelope fields (Bcc from the envelope, list members from ORCPT, direction, held for review) around the queued message byte for byte.
Built-in journal with a per-node chain whose links name entries by hash, so entries expire out of chain order; purge marks, verify catches changes, early removal and mismatched reports.
Retention per journal (30-3650 days), kept per entry; daily purge skips anyone a legal hold covers (deleted accounts included) and audits the counts.
inbuxa:Journal get/set, audited. Permissions 680-683: admins see/change; Compliance Officer sees/searches/exports.

One call to review: administrators don't hold search/export (settled answer 5), and the server refuses granting what you don't hold, which broke appointing an officer. Holders of sysJournalUpdate may now grant search and export without holding them (can_grant_permissions); the role change is audited.

Deferred per the as-built notes: inbuxa:JournalEntry and Check the journal over JMAP (phase 4, with audited reads), outside archives and Journal it (phase 3).

Tests: journal_tests (new), mail_rules_tests, compliance_tests, legal_hold_tests, audit_log_tests; common/features/smtp unit tests; the SMTP suite (dkim2_all_disclosed timed out once under the full run, passes alone twice). Stays out of production like DLP.

Phase 2 of the journaling spec (#113). - Copy taken in `MessageWrapper::queue`, after DLP and transport rules; a failure to journal means the message isn't queued (temporary failure). - Journal report: envelope fields (Bcc from the envelope, list members from ORCPT, direction, held for review) around the queued message byte for byte. - Built-in journal with a per-node chain whose links name entries by hash, so entries expire out of chain order; purge marks, verify catches changes, early removal and mismatched reports. - Retention per journal (30-3650 days), kept per entry; daily purge skips anyone a legal hold covers (deleted accounts included) and audits the counts. - `inbuxa:Journal` get/set, audited. Permissions 680-683: admins see/change; Compliance Officer sees/searches/exports. **One call to review:** administrators don't hold search/export (settled answer 5), and the server refuses granting what you don't hold, which broke appointing an officer. Holders of sysJournalUpdate may now grant search and export without holding them (`can_grant_permissions`); the role change is audited. Deferred per the as-built notes: `inbuxa:JournalEntry` and Check the journal over JMAP (phase 4, with audited reads), outside archives and Journal it (phase 3). Tests: journal_tests (new), mail_rules_tests, compliance_tests, legal_hold_tests, audit_log_tests; common/features/smtp unit tests; the SMTP suite (dkim2_all_disclosed timed out once under the full run, passes alone twice). Stays out of production like DLP.

jcoffey-dev added 1 commit 2026-09-29 03:46:09 +00:00

Journaling: capture at the queue, the built-in journal, retention

ci / fork-checks (pull_request) Successful in 41s

Details

ci / build (pull_request) Successful in 8m2s

Details

441ad0b18e

Phase 2 of the journaling spec.

- A copy of each message is taken in MessageWrapper::queue, after DLP and
  transport rules, for every enabled journal that takes it (direction and
  scope: everyone, or accounts, groups, domains, tenants). If the copy
  can't be taken the message isn't queued (temporary failure).
- The journal report: the envelope one field a line (sender, To, Cc, Bcc
  from the envelope, list members from their ORCPT, direction, held for
  review), then the queued message byte for byte as message/rfc822.
- The built-in journal under J in the inbuxa subspace: one chain per node
  whose links name each entry by SHA-256, so entries can expire out of
  chain order; purge leaves a marker, and verify catches an entry changed
  or removed early and a report that doesn't match.
- Retention per journal (30 to 3650 days); an entry keeps what it was
  written with. The daily maintenance purges what's due, keeping entries
  whose people a legal hold covers (deleted accounts a hold keeps too),
  and records the counts in the audit log.
- inbuxa:Journal get/set, audited by the request layer. Permissions
  680-683: administrators see and change journals; the Compliance Officer
  sees, searches and exports. Whoever changes journals may grant search and
  export without holding them, so officers can still be appointed.
- Catalog entries (inbuxa:Journal, source "journal"); spec as-built notes.

tests/src/system/journal.rs: validation, internal mail with a Bcc,
outgoing into two journals, incoming over LMTP, the report and its
original, tamper and early removal caught, hold-aware purge, retention
changes leave entries alone, disabled and removed journals take nothing.

jcoffey-dev merged commit 4c5583e725 into main

2026-09-29 04:11:40 +00:00

jcoffey-dev deleted branch feature/journal-capture

2026-09-29 04:11:40 +00:00

jcoffey-dev referenced this pull request

2026-09-29 04:25:12 +00:00

Security to-do list: accepted items, kept on the server #114

jcoffey-dev referenced this pull request

2026-09-29 05:39:08 +00:00

Release 2026.9.29 #121