jcoffey-dev is traveling from Thursday 1 October through Sunday 4 October. Issues and pull requests are welcome, and will get an answer after that. Thanks for your patience.
Phase 2g of the DLP and mail flow rules spec: transport rules now act,
on outgoing and incoming mail.
- features/mailflow/rewrite.rs: add or remove a header, prefix or set the
subject (an RFC 2047 word when not ASCII), add a disclaimer. A
disclaimer edits the message's main text and HTML bodies only, each
decoded, changed and written back as UTF-8 quoted-printable with its
other headers kept, top or bottom (after <body> or before </body> in
HTML); attachments and attached messages are left alone, and a
disclaimer already present isn't added again.
- smtp/inbound/mailflow.rs: the check runs for incoming mail too
(transport rules only; DLP stays outgoing). After DLP passes, each
matched transport rule's actions run in order: message edits,
add-recipient and redirect (envelope changes DATA applies), route (a
per-message queue ahead of the queue strategy), refuse (550 5.7.1
with the rule's text). The override tag is stripped with the same
subject writer, so a non-ASCII subject stays valid.
- Audit: refusals and changes to where mail goes are recorded (sender,
or system:mail-flow for incoming mail); wording and header changes
aren't, or a banner rule would record every message (spec §2.7).
Tests: rewrite unit tests (headers, encoded subjects, disclaimers on a
single part and on multipart/alternative with an attachment, once
only); mail_rules_tests gains the actions end to end: disclaimer,
header and subject prefix on a delivered message, a redirect, a
refusal, a banner on incoming LMTP mail that outgoing rules leave
alone, and which of those are audited.
Phase 2f of the DLP and mail flow rules spec: the rules now run on mail
an authenticated sender submits, after the DATA system script and
before headers and DKIM signing (§2.1).
- smtp/inbound/mailflow.rs: builds what the rules look at from the
message (subject, the text version of each body, one level of attached
messages, attachment text via the extractor, 10 MB of text at most)
and the envelope (sender's groups and tenant; each recipient local or
not, and its groups). Skipped entirely when no enabled rule applies to
outgoing mail. Rules that can't be loaded refuse with a 451: nothing
unchecked leaves.
- Block: 550 5.7.1 with the rule's notice. Warn: 550 5.7.1 with the
notice and how to override: "[override: reason]" at the start of the
subject, taken out before the message goes on (settled answer 1).
Until phase 3, a hold rule blocks rather than let mail through.
- JMAP: EmailSubmission takes inbuxa:dlpOverride {reason}; a refusal
comes back as inbuxa:dlpWarning or inbuxa:dlpBlocked with each rule's
name and notice (description too, for older clients).
- Audit: one record per DLP match, the sender as actor, action create,
target a message: the recipient domains, each rule with its detectors'
counts, the outcome, an override's reason. Never the matched text. No
new audit action: an older node that meets one fails its daily
clean-up, which would make rolling back unsafe (spec §2.7 updated).
Tests: mail_rules_tests gains the DLP flow over JMAP (no rules, warning
with rule and notice, local recipient not warned, override with a
reason, block that no reason passes, the subject tag stripped from the
delivered message, audit records with no card or key text). smtp
inbound tests pass; system_tests passed twice after one timeout in the
email delivery tests that didn't recur.
Phase 2e of the DLP and mail flow rules spec, the API half.
- inbuxa:MailRule/get and /set under urn:inbuxa:jmap. Rules convert
through serde, so what a client sends is the stored format. A create
or change is validated whole (Rule::validate) and refused with the
property at fault; id, createdBy, createdAt and updatedAt are the
server's. Every change goes through the request layer's audit record.
- Six permissions, ids 674-679 (enum and schema labels): mail flow rules
(sysMailRuleGet/Update), DLP rules (sysDlpPolicyGet/Update) and held
mail (sysDlpReviewGet/Update, for phase 3). Either kind's permission
gets through the gate; the handler shows and changes each rule only
with its own kind's. All server-level: a tenant is refused (settled
answer 3).
- Administrators get all six; the server-level Compliance Officer gets
DLP rules to see and held mail to review (settled answer 4), added
once to an existing server's officer role by the grant mechanism,
which gains an officer audience.
- Privacy catalog entry for inbuxa:MailRule.
tests/src/system/mail_rules.rs: create, list in order, validation,
server-set properties refused, update, kind-separated permissions for
an officer, destroy, audit records.
Phase 2e of the DLP and mail flow rules spec, in the features crate.
- rules.rs: a rule (§2.2) with its conditions (§2.3) and actions (§2.4),
as JSON under R/r in the fork's subspace. validate() enforces the
spec's shape: DLP rules check outgoing mail and have exactly one of
block, warn or hold; transport rules have neither those nor
detectors; lists, header names, header values (one line), addresses,
texts, word lists, patterns and detector ids are checked.
- engine.rs: rules compiled once (word lists to automata, patterns to
size-limited regexes) and run in priority order with exceptions and
stop processing. Each detector runs at most once per message and
only when a rule asks for it. The outcome lists what matched with
each detector's count, and decides DLP strictest first: block, hold,
warn; an override answers warnings only (§2.5).
- cache.rs: each node's compiled copy, refreshed after 30 seconds or at
once when this node changes a rule.
Nothing calls this yet: the JMAP object and the check at DATA follow.
55 unit tests in mailflow.
Phase 2b of the DLP and mail flow rules spec: every identifier in the
§2.3 catalog, each implemented from its issuer's published rules and
tested against published examples.
US (SSN, ITIN, EIN, ABA routing, driver's licenses, MBI, NPI, DEA), UK
(NI number, NHS number, UTR), Canada (SIN), Australia (TFN, Medicare),
the EU (Germany's tax ID and ID card, France's NIR, Spain's DNI/NIE,
Italy's codice fiscale, the Dutch BSN, Belgium's national number,
Poland's PESEL, Sweden's personnummer, Denmark's CPR, Finland's HETU,
Ireland's PPS, Portugal's NIF, Austria's SVNR), Norway, Switzerland,
India (Aadhaar, PAN), China, Japan, Singapore, South Korea, Brazil (CPF,
CNPJ), Mexico (CURP) and South Africa. 49 detectors in all, plus seven
templates named for what they find.
An identifier that is only digits and whose check about one random
number in ten passes counts alone only in its written form
(536-22-1234, 943 476 5919) and as bare digits only beside a word; ABA
routing numbers and NPIs always need one. Spec §2.3 records this.
A test runs every detector over an ordinary business email (order,
invoice and tracking numbers, dates, amounts, an address) and requires
nothing to fire but the contact detectors. 47 unit tests.
Phase 2a of the DLP and mail flow rules spec: pure functions in
crates/features/src/mailflow, nothing wired into the mail path yet.
- Detectors report distinct values found, each either checked by its
published check digit or counted only beside a corroborating word
within 50 characters. This PR adds the region-free ones: payment
cards (issuer prefixes, Luhn), IBAN (registry lengths, mod 97),
SWIFT/BIC, email addresses and phone numbers in bulk, dates of birth,
passport numbers, private keys and published service-token formats.
Regional identifiers follow, a region per PR.
- Word lists (Aho-Corasick, whole words, any case) and patterns (regex
with a compiled-size limit) count occurrences.
- Attachment text: text files with or without a UTF-16 mark, HTML,
DOCX/XLSX/PPTX, ODT/ODS/ODP and ZIP archives one level deep, read
with the zip and quick-xml crates the workspace already has.
Encrypted files, PDF, legacy binary Office files, nested archives
and anything past the limits come back as not inspectable, with why.
21 unit tests, against the networks' test card numbers and the IBAN
registry's own examples among others.
All six settled as recommended. Answer 6 ("and any other recognized and protected PII") becomes a catalog of identifiers with published formats and checks, grouped by region, each either checked by its check digit or counted only beside a corroborating word, plus templates named for what they find. Data with no number to find is covered by word lists and not claimed as detection. Office documents are read; PDF counts as can't be inspected.
Phase 1: one native rule engine at DATA, after the system Sieve script,
for both DLP policies and transport rules. DLP checks outgoing mail
with counted detectors (payment cards, IBAN, US SSN, word lists,
patterns) and blocks, warns with an audited override, or holds for
review. Held mail stays in the queue unscheduled, with its own review
record, so the queue's stored format is unchanged. Matches go to the
audit log without the matched text. Six questions for John at the end.
The automation suite's DNS test compared the published zone with one
copied from upstream v0.16.22. Its _ua-auto-config record carries a
SHA-256 of the account-configuration (PACC) document, and that document
names the provider as the brand, which the rebrand changed. The digest
the server publishes is right; the expected zone still held upstream's.
With the new digest the whole automation suite passes: ACME (including
the not-due reschedule check), DKIM, DNS and RFC 2136. It had been
failing at this point on main since the rebrand.
POST /api/directory/test takes a saved directory's id, an address and
optionally a password, and answers whether the directory opened, what a
recipient lookup of the address finds (account or group, with its
aliases, groups and name), and whether the password signs in. A wrong
password is told apart from a directory that can't be reached or is set
up wrong.
It calls the directory itself, below the sign-in path: a test never
creates or updates an account, never counts toward the sign-in ban and
doesn't depend on which domains use the directory. A password hash a
directory returns is never sent back. OIDC directories report their
discovered issuer; they take no passwords.
For server-level administrators with directory update permission. The
console's guided directory setup uses it to test a real person before
any domain is switched over.
The registry knows, for every expression field, the constants it may
evaluate to and the variables its conditions may read, and enforces both.
The schema served to INBUXA Admin described every one as a bare
x:Expression, so the console could offer nothing better than free text.
tools/fork/expr-schema.py reads those contexts from the generated registry
code and writes them onto each field's type as
expression: {constants, variables}. All 124 expression fields are covered.
CI runs it with --check so the schema can't drift from the registry.
The personal-data catalog and what it feeds: the data inventory and its
snapshots (#83, #89), the Compliance Officer roles (#88), the Compliance
Overview and Data Inventory menu entries (#92). Log file retention
(#87), privacy defaults for new installs (#85, #90), webhooks that send
only the events they name (#82), and upstream v0.16.24 (#84), whose
spam rules updates keep what an admin edited.
Explain: 12 settings asked about (upstream's new createdAt fields, the
certificate dates and the webhook events policy), with the release's
recommended model built locally; 705 answers carry over.
When a valid certificate already covered a domain's names (one stored
by hand before the domain was switched to automatic, for instance), the
renewal task ended with NotDue, which the task manager treats as a
permanent failure. Nothing rescheduled it, so the certificate expired
unrenewed. The renewal now returns a new AcmeRenewal task due when the
certificate falls due, the same way a successful renewal does, and logs
it as a backoff.
The ACME integration suite checks that renewing again right after
issuance hands back one AcmeRenewal for that domain, due at the
certificate's renewal point.
Seen in the console's first Overview. x:DmarcTroubleshoot and
x:SpamClassify are one-off actions whose results come back in the
response, not kept: object-life, not unbounded. Tasks go when done
(only a failed one's status may stay, still unconfirmed): object-life.
x:Log reads the log files, so it follows inbuxa:LogSettings.keepForDays.
x:TracerLog, x:WebHook and the OpenTelemetry tracers are configuration:
their credential fields stay classified, but they are no longer listed
in the inventory, where they counted as always sent off the server
even with none configured; the log-file, webhooks and otel-tracer
sources carry what they send.
Personal-data catalog spec, §8: the Compliance section in the console
reads Overview, Data Inventory, Legal Holds, Audit Log, Locked
Accounts. The two new entries are hand-built console pages
(CustomComponent/ComplianceOverview, CustomComponent/DataInventory),
shown to people with sysComplianceGet.
A console from before these pages shows the links and answers
"Unknown component", so the console that has them should be deployed
with the server release that carries this. Schema edited as the fork's
earlier Compliance entries were, hash updated.
Personal-data catalog spec, default D5 (settled 2026-09-28; built after
the v0.16.24 import's spam-rules loader landed). msbl.org's EBL is sent
a SHA-1 of every email address it's asked about. A new install's first
boot now leaves a note, and the rules update, once the bundled rules
are in, switches STWT_MSBL_EBL_EMAIL off and forgets the note, so it
happens once; the loader keeps that switch through later updates. An
existing server has no note and keeps every blocklist as it is.
Also fixes the data inventory's DNSBL endpoints: a zone is an
expression (`ip_reverse + '.zen.spamhaus.org'`, conditional branches,
`hash(email, 'sha1') + '.ebl.msbl.org'`), and the zone names are now
the quoted literals that start with a dot, from every branch, rather
than the expression's text.
Tested: unit test for the zone rule; the compliance system test (no
note, no change; the inventory lists ebl.msbl.org, not a hash; with the
note the blocklist goes off; the note works once); the system suite;
fork checks.
Personal-data catalog spec, §6 (Phase 3c).
inbuxa:DataInventory/get evaluates the catalog against the server's
live settings and says what this server holds: for each source and each
object that can hold personal data, its categories and whose data it
is, whether it is collected here at all, what bounds its retention (the
live value of the setting that does, or unbounded), whether it leaves
the host and to which endpoints, and a summary. Every host that
receives something is listed once as a candidate processor with what it
receives. Inside a tenant it answers with the tenant's slice and none
of the server's processors. Read-only, with sysComplianceGet.
inbuxa:InventorySnapshot/get is the history: a dated copy of the
evaluated inventory, recorded when it changes -- after a registry write
to an object the inventory reads, after inbuxa's log, audit or AI
settings change, and on the daily clean-up -- and kept as long as the
audit log's records. ids: null lists every snapshot, newest first; the
full inventory only when asked for.
The catalog is embedded and parsed at start (new dependency: toml,
MIT/Apache); the evaluation is a pure function of it and the live
facts, so each configuration is tested without a server. Loopback
endpoints stay on the host; any other configured endpoint leaves it.
Tested: unit tests for the evaluation (a new install's defaults, an
external blob store, a hosted AI endpoint, telemetry off, a tenant's
slice, hosts from URLs, loopback), snapshots, and the fact gathering's
store and duration rules; the compliance system test, extended (the
officer reads the inventory, a plain user is refused, a tenant's
officer sees its slice and no processors, a webhook to another host
becomes a processor and a snapshot names x:WebHook, a retention change
reads through); the system, audit, legal hold and account lock suites;
fork checks. The system suite failed once of three runs with an email
import's blob not found, in antispam.rs; the same happened once in
purge.rs on the previous branch. Nothing here touches uploads; noted
for a separate look.
The schema conflicted as a binary file: taken from main and the one
edit here re-applied (sysComplianceGet after sysLegalHoldExport). The
import kept the permission count at 673, so the new id stays 673.
Retested on the merged tree in its own target directory: the
compliance and system suites pass. One earlier system run failed in
purge.rs (an imported blob not found) and didn't recur.
Personal-data catalog spec, §7 (settled 2026-09-28).
sysComplianceGet (673) sees the data inventory and compliance
overview: superusers and, for their tenant's slice, tenant
administrators, by default and through the one-time grants on servers
that already have their roles stored.
A Compliance Officer role at server level holds it with reading and
exporting the audit log, placing, widening, releasing and exporting
legal holds, seeing account locks, and reading accounts, lists,
domains, tenants and roles. It changes no server setting, creates or
deletes no account, and can't shorten audit retention.
A tenant's accounts can hold only roles of their own tenant (MT-3), so
the tenant role is one "Compliance Officer" role per tenant, without
holds (LH-13): made once for every tenant a server has, and whenever a
tenant is created. While nobody holds it, it is removed with its tenant
so it doesn't block the delete, and put back if the delete is refused
for another reason. Both roles carry a user's own permissions too,
since roles given to a person replace the default user role, which a
tenant's accounts can't hold anyway.
Every server makes these once, new or existing -- the built-in roles
are only made on a server with none -- and records each under P c, so a
role an administrator deletes stays deleted.
Tested: unit tests (neither role changes a setting beyond a user's
own; holds for the server's officer only; per-place records); a new
compliance system test (one server-level role; an officer reads the
audit log, places and releases a hold, and is refused a setting, an
account and audit retention; a tenant gets its role, whose holder reads
the tenant's audit log and no holds; a tenant with an unused role is
deleted and the role goes with it); the system, audit, legal hold,
account lock and SCIM suites; fork checks. The directory suite needs
its LDAP container and wasn't run here.
Personal-data catalog spec, default D1 (settled 2026-09-28): log files
were never deleted. inbuxa:LogSettings.keepForDays says how many days
rotated log files are kept; unset (null) keeps every file, as before,
and a new install sets 30 days.
It is a fork-owned setting, stored under T + l as audit retention is,
not a field on x:TracerLog: that object is also stored inside
x:Bootstrap with a field after it, so a new field would change
x:Bootstrap's stored format. Server-level, with the tracers'
permissions (sysTracerGet, sysTracerUpdate); changes are in the audit
log, before and after.
Log files are local, so every node deletes its own: hourly, and at once
when the setting changes on that node. Only regular files named
<prefix>.<something> in each enabled log tracer's directory, last
changed more than the limit ago, are removed; the file being written is
never that old, and nothing else in the directory is touched. Minimum
one day. The catalog classifies inbuxa:LogSettings and points the log
file's retention at it.
Tested: unit tests for the file rule (only this log's old files; the
current file, other files and directories stay) and a purge on disk;
the system suite, which reads, sets, refuses zero, restores null and
checks the audit records; fork checks.
The schema, which both sides changed, merged as JSON with no conflicts.
The personal-data catalog (#83) gains upstream's new x:DnsServerPowerDns:
nothing personal but its API key, like the other DNS providers.
D1 becomes a fork-owned setting, as audit retention is, because a field
on x:TracerLog would change x:Bootstrap's stored format. D5 waits for
the v0.16.24 import's reworked spam-rules loader.
Personal-data catalog spec, defaults D2, D3, D4, D6 and D7 (settled
2026-09-28, new installs only):
- D2: automatic IP bans expire after 30 days instead of never; D3:
spam training samples, whole messages, are kept 90 days instead of
180; D4: Pyzor, which sends a digest of each message's text to a
public server, is off; D6: delivery history is kept 14 days instead
of 30. Written on the first boot of a new install only -- one with no
roles yet, the same test the built-in roles use -- by reading each
singleton, setting these fields and writing it back whole. A server
with roles keeps its settings, saved or default.
- D7: a webhook created from now on starts with the include policy and
no events, so it sends nothing until events are chosen (Rust default
and schema default, marked). The registry stores every field, so
existing webhooks keep their policy.
- Expired bans are also removed by the daily data clean-up. They
already stopped blocking and were deleted when settings next loaded;
a server that seldom reloads kept them.
D1 (log retention) and D5 (the hashed-address blocklist off) are held,
and the spec says why: x:TracerLog is stored inside x:Bootstrap with a
field after it, so adding one changes that object's stored format; and
the spam-rules loader D5 touches is being reworked by the v0.16.24
import. The spec also corrects finding 3: expired bans were deleted on
settings load; bans were permanent only because no period is set.
Tested: unit tests for the new-install values and that everything else
in each singleton stays; the system suite, whose security test now
purges an expired ban and checks its record is gone; the telemetry
test; common's unit tests; fork checks.
Phase 2 of the personal-data catalog spec.
resources/privacy/catalog.toml classifies every object in the schema
(316) and inbuxa's own JMAP objects (12): each property that can hold
personal data, with its categories, and for objects that hold any,
whose data it is, where it lives, its scope and what bounds its
retention (a named setting where there is one). Twenty sources that
are no object -- the log file, exporters, webhooks, spam lookups, the
Explain cache, relays and hooks, push, legacy-use records -- carry the
same facts plus the settings that turn them on, whether the data
leaves the host, and the code that writes it. Classifications of
objects that hold data about people are from the spec's source map;
the rest are typed from the schema alone (address, IP, secret).
tools/fork/privacy-check.py fails CI when an object or inbuxa object
has no entry, when a property the schema types as an address, IP or
secret is left to its object's default, when an entry names an
object, property, setting or code path that is gone, or when it uses
a word outside the catalog's vocabulary. --unlisted prints starting
entries. strip.py's report gains "Unclassified in the privacy
catalog": objects and fields new in an import and not classified,
informational like the Enterprise flags.
Tested: 13 unit tests (tools/fork/tests): the check passes on this
tree; fails on an unclassified object, an address hidden behind a
default, a secret in a set or object reference, stale properties,
objects, settings and code paths, an unlisted inbuxa object and a
word outside the vocabulary; --unlisted's entries; and the strip
report on a synthetic import. The check and the tests run in the
fork-checks job.
The per-protocol switches (#79) added legacyAllowed to the account's
urn:inbuxa:jmap capability, but this expectation wasn't updated, so
jmap_tests stopped here and the suites after it never ran.
A webhook has a level (info by default) that nothing read: its events
were chosen by its list and policy alone. With the default policy,
exclude, and nothing listed, that meant every event type, including
smtp.raw-input (the raw SMTP bytes, DATA included) and the model's
reply to the spam classifier. The docs suggest a webhook to pass the
audit log to a SIEM; set up that way it would have received whole
messages. Found by the personal-data catalog investigation (finding 1).
Now an include list is sent as named, whatever each event's level:
naming an event is the choice. Otherwise a webhook gets only events at
or above its level, as a tracer does, and never a protocol's raw input
or output (IMAP, SMTP, POP3, ManageSieve, delivery, milter), which
carries whole messages and credentials; those go out only when named.
Tested: unit tests for the rule (level, raw I/O only when named, a
named event below the level, custom event levels, a webhook's own
errors); the telemetry system test, whose webhook names debug-level
connection events and still receives them.
Eight conflicted files resolved, plus the lock file and the schema:
- crates/services/src/task_manager/spam_classifier.rs: upstream's rules
update now replaces existing rules, DNSBL servers, lookups and file
extensions, keeping only whether each is on. Taken, with one difference:
an object an admin edited is kept as it is. Every object an update writes
is fingerprinted (content without `enable`, SHA-256, stored under
SUBSPACE_INBUXA "Sf"), and only one that still matches is replaced.
Scores are never replaced, as upstream has it. The AU-1.10 summary record
now names what was added, replaced and kept, and the bundled rules are
marked applied only when the update fully succeeded, so a failure runs
again on the next start. The marker becomes "3.0.2+2", which runs the
update once on upgrade to fingerprint every rule still as bundled.
- crates/common/src/network/autoconfig/autodiscover.rs: upstream's rewrite
(implicit TLS first, labeled SSL), with the per-protocol switches (LP-7,
LP-14a) passed in as a filter.
- crates/store/src/backend/mysql/{search,write}.rs: upstream's chunked
deletes (no unbounded first DELETE, stop on a short chunk, halve the
chunk on the new chunk-too-large errors) inside the fork's query timeout.
- crates/smtp/src/lib.rs: the fork's queue spawn kept. It already fixed the
stall upstream fixes here (a node without outboundMta stops accepting
mail at about 1024 queued messages), and follows role changes live.
- crates/jmap/src/registry/mapping/bootstrap.rs: the log path stays
/var/log/inbuxa/; upstream's PowerDNS mapping taken.
- crates/main/Cargo.toml: the AGPL-only license kept, version 0.16.24.
- tests/src/jmap/principal/get.rs: the fork's capabilities kept.
- resources/schema/schema.json.gz: merged as JSON; upstream relabeled the
vendor Sieve extensions "(Stalwart)", kept as "(vnd.inbuxa)".
- Cargo.lock: upstream's, with the fork's crates added by Cargo.
Also:
- tests/src/smtp/inbound/spam_rules_kept.rs: an edited rule survives an
update, an unedited one is updated, rules from before fingerprints are
handled, and the audit summary says so. Upstream's own spam_rules test
passes unchanged.
- tests/src/smtp/reporting/reschedule.rs moves to port 19058; upstream's
new spam_rules test took 19057.
- tools/fork/renames.py renames the "(Stalwart)" labels and the default
log path, so neither conflicts again.
- tools/fork/notice-check.py compares against the newest snapshot in the
checked-out history instead of the upstream branch head, so moving the
branch no longer fails other open pull requests.
- tests/src/directory/issuer.rs (since v0.16.23) stays out, and is on the
build check's known list: it tests issuer-based directory routing, which
the fork doesn't have (DIR-2).
- Strip report: docs/fork/strip-reports/v0.16.24.{md,json}.
jcoffey-dev