jcoffey-dev is traveling from Thursday 1 October through Sunday 4 October. Issues and pull requests are welcome, and will get an answer after that. Thanks for your patience.
Documents under C-5 why oAuthClientOverride counts only in bootstrap
and recovery mode, and adds tests/e2e/client_override.py: the
recovery administrator keeps the override in both modes; after setup,
an administrator gets no code for an unregistered client or a
redirect URI its client didn't register, and a device code approved
for an unregistered client can't be exchanged. The script fails
against a build without the change (3 of 8) and passes with it.
The recovery administrator signs in before any OAuth client is
registered, so it needs to skip the registration check. Outside
bootstrap and recovery mode, every account now signs in through a
registered client and one of its redirect URIs.
Anyone could host a copy of a front end on a server of their own,
collect a person's password there, and replay it as HTTP Basic against
JMAP or the API. Cross-origin rules don't stop that, since a server
isn't a browser, and neither does client registration, since Basic
never goes through OAuth (contract C-23).
JMAP (session, API, upload, download, event source, WebSocket), /api,
/auth/introspect, /auth/userinfo and authenticated /auth/register now
refuse an Authorization: Basic header before looking at the password,
with a 401 whose only challenge is Bearer. A wrong password gets the
same answer as the right one. CalDAV and CardDAV keep Basic, and their
401s still offer it. The sign-in page's /api/auth takes the password in
its body and is unaffected, as is the token endpoint's client
authentication.
Bootstrap and recovery mode accept Basic everywhere, as they keep
permissive CORS. INBUXA_HTTP_BASIC_AUTH=all puts it back everywhere;
dav is the default, and any other value logs a warning and keeps it.
Test builds accept Basic everywhere, since the integration suites sign
in with passwords, and legacy_protocols.py sets the variable.
Tested: unit tests for the paths, and tests/e2e/http_basic_auth.py
against the debug build, 26 checks, including both front ends' sign-in
path and a refused unregistered redirect.
Phase 4 of the journaling spec.
- inbuxa:JournalEntry/query and /get (sysJournalSearch): filter by time,
sender, recipient, either, direction, subject words, Message-ID and
journal, newest first; the whole report only when asked for.
- inbuxa:JournalExport/set (sysJournalExport): a reason is required; a
ZIP of the matching reports with manifest.csv, exceptions.csv and
manifest.sha256, up to 10,000 reports and 1 GB.
- inbuxa:JournalVerification/set (sysJournalGet): chains and reports
rechecked.
- Every search, listing, read, export and check is written to the audit
log before anything is returned, with existing actions only.
- Catalog entries for the three objects; spec as-built notes.
journal_tests: administrators can't search; a Compliance Officer searches,
lists, reads a report, exports (reason required) and checks the chain;
the officer can't change journals; each of those is in the audit log.
Phase 3 of the journaling spec.
- A journal's destination: builtIn (true for journals stored before) and
archiveAddress, at least one. Reports to an archive are queued from the
empty sender, one per address, flagged so they're never journaled.
- A pending record per report. When the queue lets go of one without
delivering it (refused, expired, deleted), it becomes its own entry in
the built-in journal under the sending journals' retention, the
journal's archiveFailures (count, last time, reason) goes up, and the
audit log records it; if that can't be written it stays queued.
- Journal it: a rule action naming a journal, on mail flow rules and
beside a DLP rule's block, warn or hold. A journal whose scope chooses
nobody takes only what rules send it.
- The report lists recipients a rule added or redirected to under
"Added by rule", by rule name.
- A rule's route is cleared between messages in one SMTP session, with the
new journal marks; a second message used to keep the first one's route.
tests/src/system/journal.rs: destination validation, a rule-only journal
fed by a rule that also adds a recipient, an unreachable archive's report
kept in the built-in journal with the failure counted, a report delivered
to an archive here and not journaled itself.
Phase 2 of the journaling spec.
- A copy of each message is taken in MessageWrapper::queue, after DLP and
transport rules, for every enabled journal that takes it (direction and
scope: everyone, or accounts, groups, domains, tenants). If the copy
can't be taken the message isn't queued (temporary failure).
- The journal report: the envelope one field a line (sender, To, Cc, Bcc
from the envelope, list members from their ORCPT, direction, held for
review), then the queued message byte for byte as message/rfc822.
- The built-in journal under J in the inbuxa subspace: one chain per node
whose links name each entry by SHA-256, so entries can expire out of
chain order; purge leaves a marker, and verify catches an entry changed
or removed early and a report that doesn't match.
- Retention per journal (30 to 3650 days); an entry keeps what it was
written with. The daily maintenance purges what's due, keeping entries
whose people a legal hold covers (deleted accounts a hold keeps too),
and records the counts in the audit log.
- inbuxa:Journal get/set, audited by the request layer. Permissions
680-683: administrators see and change journals; the Compliance Officer
sees, searches and exports. Whoever changes journals may grant search and
export without holding them, so officers can still be appointed.
- Catalog entries (inbuxa:Journal, source "journal"); spec as-built notes.
tests/src/system/journal.rs: validation, internal mail with a Bcc,
outgoing into two journals, incoming over LMTP, the report and its
original, tamper and early removal caught, hold-aware purge, retention
changes leave entries alone, disabled and removed journals take nothing.
senderGroup, senderTenant and recipientGroup conditions now read and write group and tenant ids as JMAP ids ("b", "c"…), like legal hold scopes and the rest of the API, so the console can use its object pickers; plain numbers are still read. Held as numbers for matching. Unit test for both forms and a bad id; mail_rules_tests round-trips a tenant condition over JMAP.
inbuxa:DlpSettings (singleton, urn:inbuxa:jmap): keepHeldDays, 1 to 90,
7 by default (settled answer 5 made it a setting). sysDlpPolicyGet reads
it, sysDlpPolicyUpdate changes it, server-level, audited by the request
layer. Each held message keeps the days it was given, and the sender's
notices say that number. Privacy catalog entry; spec §2.6 updated.
mail_rules_tests: 7 by default, 0 refused, 3 set and a message held
afterwards expires 3 days after it was held, the expiry notice says 3.
An EmailSubmission create's response carries inbuxa:held (dlp-and-mail-flow-rules spec, §2.6, §4): true when the message is held for review, false otherwise, so the webmail can say so at once. A sender can't read the review queue, and a held message's sendAt is its real send time, not the century-off release, so this is how the sender learns. mail_rules_tests checks both values.
Schema layout entries for the console pages of the DLP and mail flow rules spec (§3): Held Mail and Data Loss Prevention under Management > Compliance after Legal Holds, and Mail Flow Rules beside the server Sieve scripts (the console's nine-group Settings bar places it under Mail flow). An older console shows these as unknown pages, so this ships with the console that has them.
The hold action now holds (dlp-and-mail-flow-rules spec, §2.6), where
until now it blocked.
- At DATA a hold decision queues the message with its release a century
off (the queue's future-release mechanism, so the stored format is
unchanged and an older node just never sends it), transport rules
still applied, and replies 250 Held for review. A review record under
R/h + queue id keeps the sender, recipients, subject, size, rules and
detector counts. The sender is told when the rule asks.
- smtp/queue/held.rs: release (each recipient due now, its next notice
as far off as it was, its lifetime counted from the release), reject
(removed from the queue, the sender told, with the reviewer's note),
and expiry: the daily clean-up rejects what nobody reviewed in 7 days,
recorded as the server's doing.
- inbuxa:HeldMessage get/set: the review queue, sysDlpReviewGet to list
and read (preview, 64 KB of text, only when asked for and recorded as
blobAccess), sysDlpReviewUpdate to release or reject, a reason
required and audited by the request layer; no create or destroy;
server-level only.
- Guards: Emails > Queue refuses to change or delete held mail; the
sender can't unsend it.
- Privacy catalog entry for inbuxa:HeldMessage; spec §2.6 as built.
Tests: mail_rules_tests gains the whole flow (held and listed with
counts, sender notified and nothing delivered, queue and unsend
refused, preview recorded, reject needs a reason and tells the sender
the note, release delivers, expiry returns it, decisions audited with
reasons). smtp inbound, system_tests (after one BlobNotFound in
antispam, the known flake, then clean), features and common unit tests.
Phase 2g of the DLP and mail flow rules spec: transport rules now act,
on outgoing and incoming mail.
- features/mailflow/rewrite.rs: add or remove a header, prefix or set the
subject (an RFC 2047 word when not ASCII), add a disclaimer. A
disclaimer edits the message's main text and HTML bodies only, each
decoded, changed and written back as UTF-8 quoted-printable with its
other headers kept, top or bottom (after <body> or before </body> in
HTML); attachments and attached messages are left alone, and a
disclaimer already present isn't added again.
- smtp/inbound/mailflow.rs: the check runs for incoming mail too
(transport rules only; DLP stays outgoing). After DLP passes, each
matched transport rule's actions run in order: message edits,
add-recipient and redirect (envelope changes DATA applies), route (a
per-message queue ahead of the queue strategy), refuse (550 5.7.1
with the rule's text). The override tag is stripped with the same
subject writer, so a non-ASCII subject stays valid.
- Audit: refusals and changes to where mail goes are recorded (sender,
or system:mail-flow for incoming mail); wording and header changes
aren't, or a banner rule would record every message (spec §2.7).
Tests: rewrite unit tests (headers, encoded subjects, disclaimers on a
single part and on multipart/alternative with an attachment, once
only); mail_rules_tests gains the actions end to end: disclaimer,
header and subject prefix on a delivered message, a redirect, a
refusal, a banner on incoming LMTP mail that outgoing rules leave
alone, and which of those are audited.
Phase 2f of the DLP and mail flow rules spec: the rules now run on mail
an authenticated sender submits, after the DATA system script and
before headers and DKIM signing (§2.1).
- smtp/inbound/mailflow.rs: builds what the rules look at from the
message (subject, the text version of each body, one level of attached
messages, attachment text via the extractor, 10 MB of text at most)
and the envelope (sender's groups and tenant; each recipient local or
not, and its groups). Skipped entirely when no enabled rule applies to
outgoing mail. Rules that can't be loaded refuse with a 451: nothing
unchecked leaves.
- Block: 550 5.7.1 with the rule's notice. Warn: 550 5.7.1 with the
notice and how to override: "[override: reason]" at the start of the
subject, taken out before the message goes on (settled answer 1).
Until phase 3, a hold rule blocks rather than let mail through.
- JMAP: EmailSubmission takes inbuxa:dlpOverride {reason}; a refusal
comes back as inbuxa:dlpWarning or inbuxa:dlpBlocked with each rule's
name and notice (description too, for older clients).
- Audit: one record per DLP match, the sender as actor, action create,
target a message: the recipient domains, each rule with its detectors'
counts, the outcome, an override's reason. Never the matched text. No
new audit action: an older node that meets one fails its daily
clean-up, which would make rolling back unsafe (spec §2.7 updated).
Tests: mail_rules_tests gains the DLP flow over JMAP (no rules, warning
with rule and notice, local recipient not warned, override with a
reason, block that no reason passes, the subject tag stripped from the
delivered message, audit records with no card or key text). smtp
inbound tests pass; system_tests passed twice after one timeout in the
email delivery tests that didn't recur.
Phase 2e of the DLP and mail flow rules spec, the API half.
- inbuxa:MailRule/get and /set under urn:inbuxa:jmap. Rules convert
through serde, so what a client sends is the stored format. A create
or change is validated whole (Rule::validate) and refused with the
property at fault; id, createdBy, createdAt and updatedAt are the
server's. Every change goes through the request layer's audit record.
- Six permissions, ids 674-679 (enum and schema labels): mail flow rules
(sysMailRuleGet/Update), DLP rules (sysDlpPolicyGet/Update) and held
mail (sysDlpReviewGet/Update, for phase 3). Either kind's permission
gets through the gate; the handler shows and changes each rule only
with its own kind's. All server-level: a tenant is refused (settled
answer 3).
- Administrators get all six; the server-level Compliance Officer gets
DLP rules to see and held mail to review (settled answer 4), added
once to an existing server's officer role by the grant mechanism,
which gains an officer audience.
- Privacy catalog entry for inbuxa:MailRule.
tests/src/system/mail_rules.rs: create, list in order, validation,
server-set properties refused, update, kind-separated permissions for
an officer, destroy, audit records.
Phase 2e of the DLP and mail flow rules spec, in the features crate.
- rules.rs: a rule (§2.2) with its conditions (§2.3) and actions (§2.4),
as JSON under R/r in the fork's subspace. validate() enforces the
spec's shape: DLP rules check outgoing mail and have exactly one of
block, warn or hold; transport rules have neither those nor
detectors; lists, header names, header values (one line), addresses,
texts, word lists, patterns and detector ids are checked.
- engine.rs: rules compiled once (word lists to automata, patterns to
size-limited regexes) and run in priority order with exceptions and
stop processing. Each detector runs at most once per message and
only when a rule asks for it. The outcome lists what matched with
each detector's count, and decides DLP strictest first: block, hold,
warn; an override answers warnings only (§2.5).
- cache.rs: each node's compiled copy, refreshed after 30 seconds or at
once when this node changes a rule.
Nothing calls this yet: the JMAP object and the check at DATA follow.
55 unit tests in mailflow.
Phase 2b of the DLP and mail flow rules spec: every identifier in the
§2.3 catalog, each implemented from its issuer's published rules and
tested against published examples.
US (SSN, ITIN, EIN, ABA routing, driver's licenses, MBI, NPI, DEA), UK
(NI number, NHS number, UTR), Canada (SIN), Australia (TFN, Medicare),
the EU (Germany's tax ID and ID card, France's NIR, Spain's DNI/NIE,
Italy's codice fiscale, the Dutch BSN, Belgium's national number,
Poland's PESEL, Sweden's personnummer, Denmark's CPR, Finland's HETU,
Ireland's PPS, Portugal's NIF, Austria's SVNR), Norway, Switzerland,
India (Aadhaar, PAN), China, Japan, Singapore, South Korea, Brazil (CPF,
CNPJ), Mexico (CURP) and South Africa. 49 detectors in all, plus seven
templates named for what they find.
An identifier that is only digits and whose check about one random
number in ten passes counts alone only in its written form
(536-22-1234, 943 476 5919) and as bare digits only beside a word; ABA
routing numbers and NPIs always need one. Spec §2.3 records this.
A test runs every detector over an ordinary business email (order,
invoice and tracking numbers, dates, amounts, an address) and requires
nothing to fire but the contact detectors. 47 unit tests.
jcoffey-dev