jcoffey-dev is traveling from Thursday 1 October through Sunday 4 October. Issues and pull requests are welcome, and will get an answer after that. Thanks for your patience.
Phase 4 of the journaling spec.
- inbuxa:JournalEntry/query and /get (sysJournalSearch): filter by time,
sender, recipient, either, direction, subject words, Message-ID and
journal, newest first; the whole report only when asked for.
- inbuxa:JournalExport/set (sysJournalExport): a reason is required; a
ZIP of the matching reports with manifest.csv, exceptions.csv and
manifest.sha256, up to 10,000 reports and 1 GB.
- inbuxa:JournalVerification/set (sysJournalGet): chains and reports
rechecked.
- Every search, listing, read, export and check is written to the audit
log before anything is returned, with existing actions only.
- Catalog entries for the three objects; spec as-built notes.
journal_tests: administrators can't search; a Compliance Officer searches,
lists, reads a report, exports (reason required) and checks the chain;
the officer can't change journals; each of those is in the audit log.
Phase 3 of the journaling spec.
- A journal's destination: builtIn (true for journals stored before) and
archiveAddress, at least one. Reports to an archive are queued from the
empty sender, one per address, flagged so they're never journaled.
- A pending record per report. When the queue lets go of one without
delivering it (refused, expired, deleted), it becomes its own entry in
the built-in journal under the sending journals' retention, the
journal's archiveFailures (count, last time, reason) goes up, and the
audit log records it; if that can't be written it stays queued.
- Journal it: a rule action naming a journal, on mail flow rules and
beside a DLP rule's block, warn or hold. A journal whose scope chooses
nobody takes only what rules send it.
- The report lists recipients a rule added or redirected to under
"Added by rule", by rule name.
- A rule's route is cleared between messages in one SMTP session, with the
new journal marks; a second message used to keep the first one's route.
tests/src/system/journal.rs: destination validation, a rule-only journal
fed by a rule that also adds a recipient, an unreachable archive's report
kept in the built-in journal with the failure counted, a report delivered
to an archive here and not journaled itself.
Phase 2 of the journaling spec.
- A copy of each message is taken in MessageWrapper::queue, after DLP and
transport rules, for every enabled journal that takes it (direction and
scope: everyone, or accounts, groups, domains, tenants). If the copy
can't be taken the message isn't queued (temporary failure).
- The journal report: the envelope one field a line (sender, To, Cc, Bcc
from the envelope, list members from their ORCPT, direction, held for
review), then the queued message byte for byte as message/rfc822.
- The built-in journal under J in the inbuxa subspace: one chain per node
whose links name each entry by SHA-256, so entries can expire out of
chain order; purge leaves a marker, and verify catches an entry changed
or removed early and a report that doesn't match.
- Retention per journal (30 to 3650 days); an entry keeps what it was
written with. The daily maintenance purges what's due, keeping entries
whose people a legal hold covers (deleted accounts a hold keeps too),
and records the counts in the audit log.
- inbuxa:Journal get/set, audited by the request layer. Permissions
680-683: administrators see and change journals; the Compliance Officer
sees, searches and exports. Whoever changes journals may grant search and
export without holding them, so officers can still be appointed.
- Catalog entries (inbuxa:Journal, source "journal"); spec as-built notes.
tests/src/system/journal.rs: validation, internal mail with a Bcc,
outgoing into two journals, incoming over LMTP, the report and its
original, tamper and early removal caught, hold-aware purge, retention
changes leave entries alone, disabled and removed journals take nothing.
senderGroup, senderTenant and recipientGroup conditions now read and write group and tenant ids as JMAP ids ("b", "c"…), like legal hold scopes and the rest of the API, so the console can use its object pickers; plain numbers are still read. Held as numbers for matching. Unit test for both forms and a bad id; mail_rules_tests round-trips a tenant condition over JMAP.
inbuxa:DlpSettings (singleton, urn:inbuxa:jmap): keepHeldDays, 1 to 90,
7 by default (settled answer 5 made it a setting). sysDlpPolicyGet reads
it, sysDlpPolicyUpdate changes it, server-level, audited by the request
layer. Each held message keeps the days it was given, and the sender's
notices say that number. Privacy catalog entry; spec §2.6 updated.
mail_rules_tests: 7 by default, 0 refused, 3 set and a message held
afterwards expires 3 days after it was held, the expiry notice says 3.
An EmailSubmission create's response carries inbuxa:held (dlp-and-mail-flow-rules spec, §2.6, §4): true when the message is held for review, false otherwise, so the webmail can say so at once. A sender can't read the review queue, and a held message's sendAt is its real send time, not the century-off release, so this is how the sender learns. mail_rules_tests checks both values.
Schema layout entries for the console pages of the DLP and mail flow rules spec (§3): Held Mail and Data Loss Prevention under Management > Compliance after Legal Holds, and Mail Flow Rules beside the server Sieve scripts (the console's nine-group Settings bar places it under Mail flow). An older console shows these as unknown pages, so this ships with the console that has them.
The hold action now holds (dlp-and-mail-flow-rules spec, §2.6), where
until now it blocked.
- At DATA a hold decision queues the message with its release a century
off (the queue's future-release mechanism, so the stored format is
unchanged and an older node just never sends it), transport rules
still applied, and replies 250 Held for review. A review record under
R/h + queue id keeps the sender, recipients, subject, size, rules and
detector counts. The sender is told when the rule asks.
- smtp/queue/held.rs: release (each recipient due now, its next notice
as far off as it was, its lifetime counted from the release), reject
(removed from the queue, the sender told, with the reviewer's note),
and expiry: the daily clean-up rejects what nobody reviewed in 7 days,
recorded as the server's doing.
- inbuxa:HeldMessage get/set: the review queue, sysDlpReviewGet to list
and read (preview, 64 KB of text, only when asked for and recorded as
blobAccess), sysDlpReviewUpdate to release or reject, a reason
required and audited by the request layer; no create or destroy;
server-level only.
- Guards: Emails > Queue refuses to change or delete held mail; the
sender can't unsend it.
- Privacy catalog entry for inbuxa:HeldMessage; spec §2.6 as built.
Tests: mail_rules_tests gains the whole flow (held and listed with
counts, sender notified and nothing delivered, queue and unsend
refused, preview recorded, reject needs a reason and tells the sender
the note, release delivers, expiry returns it, decisions audited with
reasons). smtp inbound, system_tests (after one BlobNotFound in
antispam, the known flake, then clean), features and common unit tests.
Phase 2g of the DLP and mail flow rules spec: transport rules now act,
on outgoing and incoming mail.
- features/mailflow/rewrite.rs: add or remove a header, prefix or set the
subject (an RFC 2047 word when not ASCII), add a disclaimer. A
disclaimer edits the message's main text and HTML bodies only, each
decoded, changed and written back as UTF-8 quoted-printable with its
other headers kept, top or bottom (after <body> or before </body> in
HTML); attachments and attached messages are left alone, and a
disclaimer already present isn't added again.
- smtp/inbound/mailflow.rs: the check runs for incoming mail too
(transport rules only; DLP stays outgoing). After DLP passes, each
matched transport rule's actions run in order: message edits,
add-recipient and redirect (envelope changes DATA applies), route (a
per-message queue ahead of the queue strategy), refuse (550 5.7.1
with the rule's text). The override tag is stripped with the same
subject writer, so a non-ASCII subject stays valid.
- Audit: refusals and changes to where mail goes are recorded (sender,
or system:mail-flow for incoming mail); wording and header changes
aren't, or a banner rule would record every message (spec §2.7).
Tests: rewrite unit tests (headers, encoded subjects, disclaimers on a
single part and on multipart/alternative with an attachment, once
only); mail_rules_tests gains the actions end to end: disclaimer,
header and subject prefix on a delivered message, a redirect, a
refusal, a banner on incoming LMTP mail that outgoing rules leave
alone, and which of those are audited.
Phase 2f of the DLP and mail flow rules spec: the rules now run on mail
an authenticated sender submits, after the DATA system script and
before headers and DKIM signing (§2.1).
- smtp/inbound/mailflow.rs: builds what the rules look at from the
message (subject, the text version of each body, one level of attached
messages, attachment text via the extractor, 10 MB of text at most)
and the envelope (sender's groups and tenant; each recipient local or
not, and its groups). Skipped entirely when no enabled rule applies to
outgoing mail. Rules that can't be loaded refuse with a 451: nothing
unchecked leaves.
- Block: 550 5.7.1 with the rule's notice. Warn: 550 5.7.1 with the
notice and how to override: "[override: reason]" at the start of the
subject, taken out before the message goes on (settled answer 1).
Until phase 3, a hold rule blocks rather than let mail through.
- JMAP: EmailSubmission takes inbuxa:dlpOverride {reason}; a refusal
comes back as inbuxa:dlpWarning or inbuxa:dlpBlocked with each rule's
name and notice (description too, for older clients).
- Audit: one record per DLP match, the sender as actor, action create,
target a message: the recipient domains, each rule with its detectors'
counts, the outcome, an override's reason. Never the matched text. No
new audit action: an older node that meets one fails its daily
clean-up, which would make rolling back unsafe (spec §2.7 updated).
Tests: mail_rules_tests gains the DLP flow over JMAP (no rules, warning
with rule and notice, local recipient not warned, override with a
reason, block that no reason passes, the subject tag stripped from the
delivered message, audit records with no card or key text). smtp
inbound tests pass; system_tests passed twice after one timeout in the
email delivery tests that didn't recur.
Phase 2e of the DLP and mail flow rules spec, the API half.
- inbuxa:MailRule/get and /set under urn:inbuxa:jmap. Rules convert
through serde, so what a client sends is the stored format. A create
or change is validated whole (Rule::validate) and refused with the
property at fault; id, createdBy, createdAt and updatedAt are the
server's. Every change goes through the request layer's audit record.
- Six permissions, ids 674-679 (enum and schema labels): mail flow rules
(sysMailRuleGet/Update), DLP rules (sysDlpPolicyGet/Update) and held
mail (sysDlpReviewGet/Update, for phase 3). Either kind's permission
gets through the gate; the handler shows and changes each rule only
with its own kind's. All server-level: a tenant is refused (settled
answer 3).
- Administrators get all six; the server-level Compliance Officer gets
DLP rules to see and held mail to review (settled answer 4), added
once to an existing server's officer role by the grant mechanism,
which gains an officer audience.
- Privacy catalog entry for inbuxa:MailRule.
tests/src/system/mail_rules.rs: create, list in order, validation,
server-set properties refused, update, kind-separated permissions for
an officer, destroy, audit records.
Phase 2e of the DLP and mail flow rules spec, in the features crate.
- rules.rs: a rule (§2.2) with its conditions (§2.3) and actions (§2.4),
as JSON under R/r in the fork's subspace. validate() enforces the
spec's shape: DLP rules check outgoing mail and have exactly one of
block, warn or hold; transport rules have neither those nor
detectors; lists, header names, header values (one line), addresses,
texts, word lists, patterns and detector ids are checked.
- engine.rs: rules compiled once (word lists to automata, patterns to
size-limited regexes) and run in priority order with exceptions and
stop processing. Each detector runs at most once per message and
only when a rule asks for it. The outcome lists what matched with
each detector's count, and decides DLP strictest first: block, hold,
warn; an override answers warnings only (§2.5).
- cache.rs: each node's compiled copy, refreshed after 30 seconds or at
once when this node changes a rule.
Nothing calls this yet: the JMAP object and the check at DATA follow.
55 unit tests in mailflow.
Phase 2b of the DLP and mail flow rules spec: every identifier in the
§2.3 catalog, each implemented from its issuer's published rules and
tested against published examples.
US (SSN, ITIN, EIN, ABA routing, driver's licenses, MBI, NPI, DEA), UK
(NI number, NHS number, UTR), Canada (SIN), Australia (TFN, Medicare),
the EU (Germany's tax ID and ID card, France's NIR, Spain's DNI/NIE,
Italy's codice fiscale, the Dutch BSN, Belgium's national number,
Poland's PESEL, Sweden's personnummer, Denmark's CPR, Finland's HETU,
Ireland's PPS, Portugal's NIF, Austria's SVNR), Norway, Switzerland,
India (Aadhaar, PAN), China, Japan, Singapore, South Korea, Brazil (CPF,
CNPJ), Mexico (CURP) and South Africa. 49 detectors in all, plus seven
templates named for what they find.
An identifier that is only digits and whose check about one random
number in ten passes counts alone only in its written form
(536-22-1234, 943 476 5919) and as bare digits only beside a word; ABA
routing numbers and NPIs always need one. Spec §2.3 records this.
A test runs every detector over an ordinary business email (order,
invoice and tracking numbers, dates, amounts, an address) and requires
nothing to fire but the contact detectors. 47 unit tests.
Phase 2a of the DLP and mail flow rules spec: pure functions in
crates/features/src/mailflow, nothing wired into the mail path yet.
- Detectors report distinct values found, each either checked by its
published check digit or counted only beside a corroborating word
within 50 characters. This PR adds the region-free ones: payment
cards (issuer prefixes, Luhn), IBAN (registry lengths, mod 97),
SWIFT/BIC, email addresses and phone numbers in bulk, dates of birth,
passport numbers, private keys and published service-token formats.
Regional identifiers follow, a region per PR.
- Word lists (Aho-Corasick, whole words, any case) and patterns (regex
with a compiled-size limit) count occurrences.
- Attachment text: text files with or without a UTF-16 mark, HTML,
DOCX/XLSX/PPTX, ODT/ODS/ODP and ZIP archives one level deep, read
with the zip and quick-xml crates the workspace already has.
Encrypted files, PDF, legacy binary Office files, nested archives
and anything past the limits come back as not inspectable, with why.
21 unit tests, against the networks' test card numbers and the IBAN
registry's own examples among others.
All six settled as recommended. Answer 6 ("and any other recognized and protected PII") becomes a catalog of identifiers with published formats and checks, grouped by region, each either checked by its check digit or counted only beside a corroborating word, plus templates named for what they find. Data with no number to find is covered by word lists and not claimed as detection. Office documents are read; PDF counts as can't be inspected.
Phase 1: one native rule engine at DATA, after the system Sieve script,
for both DLP policies and transport rules. DLP checks outgoing mail
with counted detectors (payment cards, IBAN, US SSN, word lists,
patterns) and blocks, warns with an audited override, or holds for
review. Held mail stays in the queue unscheduled, with its own review
record, so the queue's stored format is unchanged. Matches go to the
audit log without the matched text. Six questions for John at the end.
jcoffey-dev