Merge upstream v0.16.24 #84

Merged

jcoffey-dev merged 4 commits from merge/upstream-v0.16.24 into main

2026-09-28 15:55:17 +00:00

jcoffey-dev commented

2026-09-28 13:56:03 +00:00

Owner

Brings in upstream v0.16.24, stripped: the same Enterprise footprint as v0.16.23 (63 files, 118 snippets, verification clean). The strip report is in docs/fork/strip-reports/v0.16.24.md.

The snapshot commit f59b084 is this PR's second parent. After this merges, fast-forward the upstream branch to f59b084. Doing it before would fail notice-check on other open PRs; this PR changes the check so that can't happen again.

Spam rules updates keep admin edits

Upstream now replaces existing rules on update, keeping only whether each is switched on. This branch takes that, except that anything an admin edited is left alone (decided 2026-09-28):

Every object an update writes is fingerprinted: its content without enable, as SHA-256.
An existing object is replaced only while it still matches its fingerprint. Scores are never replaced, as upstream has it.
Switching a rule on or off isn't an edit.
The bundled-rules marker becomes 3.0.2+2, so the first start after upgrade runs the update once. That fingerprints every rule still exactly as bundled, and the next rules release can update them.
The AU-1.10 audit summary names what was added, replaced and kept.
The bundled rules are marked applied only when the update fully succeeded, so a failed update runs again on the next start.
A deleted bundled rule comes back with the next rules release, as before. To get rid of one, switch it off.

Conflicts

Autodiscover: upstream's rewrite (implicit TLS first, Outlook's SSL/TLS labels), with the per-protocol switches passed in as a filter.
MySQL: upstream's chunked deletes, inside the fork's query timeout.
Queue spawn: ours kept. It already fixed the ~1024-queued-messages stall on nodes without outboundMta, and it follows role changes live.
Log path, license, capabilities test: ours kept. Upstream's PowerDNS mapping is taken.
Schema: merged as JSON. Upstream relabeled the vendor Sieve extensions "(Stalwart)"; they stay "(vnd.inbuxa)".

Also in this PR

renames.py now renames those labels and the default log path, so neither conflicts again.
Our reschedule test moves to port 19058, because upstream's new spam_rules test took 19057.
tests/src/directory/issuer.rs (upstream, since v0.16.23) stays out and is on the build check's known list: the fork has no issuer-based directory routing (DIR-2).

Also fixed: the principal get test (#79)

#79 added legacyAllowed to the account capability without updating tests/src/jmap/principal/get.rs. That made jmap_tests stop partway on main too, so the suites after it never ran. The second commit updates the expectation.

Tested locally

Unit tests: common (spam_rules, autodiscover), store, services: pass.
Spam rules: smtp::inbound::spam_rules (upstream's test, unchanged) and spam_rules_kept (new): pass.
SMTP group, serially: 78 pass. The three failures are #[ignore]d tests that need the live internet or an outside milter (asn, dane_live_smtp_hosts, milter_client_test). Every milter, DATA and Sieve test passes.
imap_tests and jmap_tests: pass.
system::: the audit log, legal hold, account lock, masked email, branding, monitoring, AI and settings reload suites pass. The six #[ignore]d *_compat/AI-model tests need recordings or a local model.
MySQL: store_tests (chunked deletes, SQL statement timeouts against MariaDB) and search_tests pass. The first store_tests run failed because the harness started a stopped MariaDB container mid-test; the rerun passed.
Fork checks: name-check and notice-check are clean.

Brings in upstream v0.16.24, stripped: the same Enterprise footprint as v0.16.23 (63 files, 118 snippets, verification clean). The strip report is in `docs/fork/strip-reports/v0.16.24.md`. The snapshot commit `f59b084` is this PR's second parent. **After this merges, fast-forward the `upstream` branch to `f59b084`.** Doing it before would fail `notice-check` on other open PRs; this PR changes the check so that can't happen again. ## Spam rules updates keep admin edits Upstream now replaces existing rules on update, keeping only whether each is switched on. This branch takes that, except that anything an admin edited is left alone (decided 2026-09-28): - Every object an update writes is fingerprinted: its content without `enable`, as SHA-256. - An existing object is replaced only while it still matches its fingerprint. Scores are never replaced, as upstream has it. - Switching a rule on or off isn't an edit. - The bundled-rules marker becomes `3.0.2+2`, so the first start after upgrade runs the update once. That fingerprints every rule still exactly as bundled, and the next rules release can update them. - The AU-1.10 audit summary names what was added, replaced and kept. - The bundled rules are marked applied only when the update fully succeeded, so a failed update runs again on the next start. - A deleted bundled rule comes back with the next rules release, as before. To get rid of one, switch it off. ## Conflicts - **Autodiscover:** upstream's rewrite (implicit TLS first, Outlook's SSL/TLS labels), with the per-protocol switches passed in as a filter. - **MySQL:** upstream's chunked deletes, inside the fork's query timeout. - **Queue spawn:** ours kept. It already fixed the ~1024-queued-messages stall on nodes without `outboundMta`, and it follows role changes live. - **Log path, license, capabilities test:** ours kept. Upstream's PowerDNS mapping is taken. - **Schema:** merged as JSON. Upstream relabeled the vendor Sieve extensions "(Stalwart)"; they stay "(vnd.inbuxa)". ## Also in this PR - `renames.py` now renames those labels and the default log path, so neither conflicts again. - Our `reschedule` test moves to port 19058, because upstream's new `spam_rules` test took 19057. - `tests/src/directory/issuer.rs` (upstream, since v0.16.23) stays out and is on the build check's known list: the fork has no issuer-based directory routing (DIR-2). ## Also fixed: the principal get test (#79) #79 added `legacyAllowed` to the account capability without updating `tests/src/jmap/principal/get.rs`. That made `jmap_tests` stop partway on `main` too, so the suites after it never ran. The second commit updates the expectation. ## Tested locally - **Unit tests:** `common` (spam_rules, autodiscover), `store`, `services`: pass. - **Spam rules:** `smtp::inbound::spam_rules` (upstream's test, unchanged) and `spam_rules_kept` (new): pass. - **SMTP group, serially:** 78 pass. The three failures are `#[ignore]`d tests that need the live internet or an outside milter (`asn`, `dane_live_smtp_hosts`, `milter_client_test`). Every milter, DATA and Sieve test passes. - **`imap_tests` and `jmap_tests`:** pass. - **`system::`:** the audit log, legal hold, account lock, masked email, branding, monitoring, AI and settings reload suites pass. The six `#[ignore]`d `*_compat`/AI-model tests need recordings or a local model. - **MySQL:** `store_tests` (chunked deletes, SQL statement timeouts against MariaDB) and `search_tests` pass. The first `store_tests` run failed because the harness started a stopped MariaDB container mid-test; the rerun passed. - **Fork checks:** `name-check` and `notice-check` are clean.

jcoffey-dev added 3 commits 2026-09-28 13:56:03 +00:00

Import upstream v0.16.24, stripped f59b084ce5

Upstream commit: af37a234981722493b74623a983581691d2b70b6
Enterprise-only files removed or emptied: 63
Enterprise-only snippets removed: 118 in 50 files
Dangling module declarations removed: 5
Edits turning enterprise off: 25
Third-party code: 14 files, 0 not in THIRD-PARTY.md
Renamed identifiers: 62 in 18 files
Verification: clean

The same Enterprise footprint as v0.16.23. The build check fails only on
tests/src/directory/issuer.rs, unchanged since v0.16.23: it calls a helper
from upstream's Enterprise-only OIDC test, and tests issuer-based directory
routing, an Enterprise feature. main has never carried it.

Merge upstream v0.16.24 b2453d066b

Eight conflicted files resolved, plus the lock file and the schema:

- crates/services/src/task_manager/spam_classifier.rs: upstream's rules
  update now replaces existing rules, DNSBL servers, lookups and file
  extensions, keeping only whether each is on. Taken, with one difference:
  an object an admin edited is kept as it is. Every object an update writes
  is fingerprinted (content without `enable`, SHA-256, stored under
  SUBSPACE_INBUXA "Sf"), and only one that still matches is replaced.
  Scores are never replaced, as upstream has it. The AU-1.10 summary record
  now names what was added, replaced and kept, and the bundled rules are
  marked applied only when the update fully succeeded, so a failure runs
  again on the next start. The marker becomes "3.0.2+2", which runs the
  update once on upgrade to fingerprint every rule still as bundled.
- crates/common/src/network/autoconfig/autodiscover.rs: upstream's rewrite
  (implicit TLS first, labeled SSL), with the per-protocol switches (LP-7,
  LP-14a) passed in as a filter.
- crates/store/src/backend/mysql/{search,write}.rs: upstream's chunked
  deletes (no unbounded first DELETE, stop on a short chunk, halve the
  chunk on the new chunk-too-large errors) inside the fork's query timeout.
- crates/smtp/src/lib.rs: the fork's queue spawn kept. It already fixed the
  stall upstream fixes here (a node without outboundMta stops accepting
  mail at about 1024 queued messages), and follows role changes live.
- crates/jmap/src/registry/mapping/bootstrap.rs: the log path stays
  /var/log/inbuxa/; upstream's PowerDNS mapping taken.
- crates/main/Cargo.toml: the AGPL-only license kept, version 0.16.24.
- tests/src/jmap/principal/get.rs: the fork's capabilities kept.
- resources/schema/schema.json.gz: merged as JSON; upstream relabeled the
  vendor Sieve extensions "(Stalwart)", kept as "(vnd.inbuxa)".
- Cargo.lock: upstream's, with the fork's crates added by Cargo.

Also:

- tests/src/smtp/inbound/spam_rules_kept.rs: an edited rule survives an
  update, an unedited one is updated, rules from before fingerprints are
  handled, and the audit summary says so. Upstream's own spam_rules test
  passes unchanged.
- tests/src/smtp/reporting/reschedule.rs moves to port 19058; upstream's
  new spam_rules test took 19057.
- tools/fork/renames.py renames the "(Stalwart)" labels and the default
  log path, so neither conflicts again.
- tools/fork/notice-check.py compares against the newest snapshot in the
  checked-out history instead of the upstream branch head, so moving the
  branch no longer fails other open pull requests.
- tests/src/directory/issuer.rs (since v0.16.23) stays out, and is on the
  build check's known list: it tests issuer-based directory routing, which
  the fork doesn't have (DIR-2).
- Strip report: docs/fork/strip-reports/v0.16.24.{md,json}.

Principal get test: expect legacyAllowed

ci / fork-checks (pull_request) Successful in 12s

Details

ci / build (pull_request) Successful in 1h18m28s

Details

eac3db34e9

The per-protocol switches (#79) added legacyAllowed to the account's
urn:inbuxa:jmap capability, but this expectation wasn't updated, so
jmap_tests stopped here and the suites after it never ran.

jcoffey-dev added 1 commit 2026-09-28 15:19:28 +00:00

Merge main into merge/upstream-v0.16.24

ci / fork-checks (pull_request) Successful in 21s

Details

ci / build (pull_request) Successful in 35m18s

Details

80d6c09c59

The schema, which both sides changed, merged as JSON with no conflicts.
The personal-data catalog (#83) gains upstream's new x:DnsServerPowerDns:
nothing personal but its API key, like the other DNS providers.

jcoffey-dev merged commit 9893452ca2 into main

2026-09-28 15:55:17 +00:00

jcoffey-dev deleted branch merge/upstream-v0.16.24

2026-09-28 15:55:17 +00:00

jcoffey-dev referenced this issue from a commit

2026-09-28 19:22:38 +00:00

Release 2026.9.28.4

jcoffey-dev referenced this pull request

2026-09-28 19:22:39 +00:00

Release 2026.9.28.4 #94