Merge upstream v1.0.0 #215

Closed

jcoffey-dev wants to merge 5 commits from merge/upstream-v1.0.0 into next

pull from: merge/upstream-v1.0.0

merge into: inbuxa:next

inbuxa:main

inbuxa:release/2026

inbuxa:upstream

inbuxa:next

Conversation 1 Commits 5 Files Changed 2142

+448746 -110122

jcoffey-dev commented 2026-10-27 21:40:24 +00:00

Owner

Copy link

Copy source

Merges upstream v1.0.0 (stripped, upstream d1e48ba) into next, a new long-lived branch made from main. main stays the 0.16 line and keeps its weekly 2026.x releases; the 27.1 release candidates will come from next. This is the last upstream import; from here the fork diverges fully and moves to its own series (27.1).

What 1.0 changed under us, and what this merge does about it

• Store layout. 1.0 can't open a 0.16 store and refuses the other edition's. The server now opens a 1.0 store of either edition and stamps it with our own store version (System key 0x80). A store written by a newer inbuxa line is refused. Upgrading from 0.16 goes through the migration tool into a new, empty store (wrapper to follow).
• Features 1.0 dropped from its AGPL code, restored as ours: multi-tenancy plumbing, metrics and tracing stores, the in-memory store kept in the data store, the _ subspace (audit log, holds, journal, deliverability, AI limits).
• Registry. Generated by our own generator (tools/registry-gen) from the merged schema, full edition included. Our permissions follow 1.0's last id (643–672), and our trc events moved to 655–669.
• SQL backends. Our 0.16 work was re-ported onto 1.0's rewritten PostgreSQL/MySQL code: query time limits (stalled connections detached, not returned to the pool), MySQL pool wait timeout, PostgreSQL search tokenizing, prepare for unindex, MySQL stopwords.
• Security re-expressions. Anonymous OAuth client registration stays off unless an operator turns it on (1.0 opened it). Tenant scoping is kept on principal and report reads.
• Compatibility kept for 0.16 users:
  • counter_incr / counter_get in expressions (1.0 dropped them);
  • vnd.stalwart.while / vnd.stalwart.expressions Sieve capabilities;
  • spam-filter blobs move from the pre-rename keys at startup;
  • stalwart-app.json app manifests.
• Tests. 1.0 moved its test suites private and deleted the unit tests in crates/. Our tests/ suite was ported to 1.0 behaviour. Test helpers sit behind --cfg inbuxa_tests, renamed from upstream's cfg, and never set in release builds. Upstream's calls into its unpublished test instrumentation were removed.
• Privacy catalog updated for 1.0, including a new cache-swap source for 1.0's account-cache snapshots.
• THIRD-PARTY.md covers the third-party files new in 1.0. The strip report is in docs/fork/strip-reports/v1.0.0.*.

Tested

Locally, with RUSTFLAGS="--cfg inbuxa_tests":

• Integration suite on RocksDB: 102 of 104 passed in the full run. cluster::broadcast::cluster_tests needs a coordinator service (COORDINATOR=) this machine doesn't run. system::system_tests failed in the purge step (see the quota fix below) and passes with the fix.
• PostgreSQL: store::search_tests, store::store_tests, store::search_gin and the pool-timeout tests pass.
• MySQL: store::store_tests and the pool-timeout tests pass. store::search_tests fails on the phrase query 'rustic bridge', exactly as it does on main today, so it is not from this merge. InnoDB's full-text index loses the phrase for some rows written by concurrent index transactions (the words match; rebuilding the table fixes it). Follow-up below.
• Fork checks, cargo build -p inbuxa --locked, the workspace test build, and a release build.

One test-isolation fix that also applies to main: the quota test left a one-second upload lifetime behind, so the purge test that follows could lose its uploads on a loaded machine. It now restores the upload settings.

Licensing and safety review

• Every changed file's license header was checked. Upstream files are AGPL-3.0-only OR LicenseRef-SEL, ours are AGPL-3.0-only, and the permissive third-party files are listed in THIRD-PARTY.md. No SEL-only file is included, and upstream's enterprise/ code is absent; the only file under an enterprise/ path is ours.
• THIRD-PARTY.md now also credits the PDF tables in crates/text-extract, which match Adobe's cmap-resources, Adobe Glyph List and Core 14 font metrics. It records the four crates 1.0 moved in-tree and relabeled (calcard, jmap-tools, smtp-proto, and mail-auth's report module as smtp-report), all previously published as Apache-2.0 OR MIT.
• New dependencies: 14 from crates.io, all permissive except evalexpr 13 (AGPL-3.0-only), plus three in-tree path crates. No new git sources.
• The added lines contain no private keys, tokens, live hostnames or personal data. Hidden-character and injected-instruction scans are clean.
• Test hooks exist only under --cfg inbuxa_tests, which no release or image build sets. The release paths were checked: upload quota, DNSSEC detection, OIDC opaque tokens, DNS updater and MTA-STS policy.
• Anonymous OAuth client registration defaults to off and otherwise requires the oAuthClientRegistration permission. The 39 new unsafe lines are all upstream's, unchanged.

Follow-ups (not in this PR)

• MySQL phrase search after concurrent indexing (above).
• Vendor the opentelemetry crates: they come from a git fork on upstream's GitHub account (pinned commit; already the case on main).
• Audit the Sieve runtime's unsafe arena code (new in 1.0; it runs user scripts).
• Principal/get account objects (new in 1.0) don't carry the urn:inbuxa:jmap or masked-email capabilities that the session object does.
• Port back the unit tests 1.0 deleted (0.16.27's, where the code still exists).
• Upgrade wrapper around the migration tool, a new upgrade test plan with rehearsals on a copy of production that check the result independently of the tool, operator docs, and installer support (permission and event id remap, _ copy, deleted items, in-memory tables, a new empty database, full stop and start).
• 27.1 versioning (Cargo versions, brand_version!, release workflow, image tags).
• Wire renames for the console, webmail, installer and docs (wire-renames-0.16-to-1.0.txt).

Merges upstream v1.0.0 (stripped, `upstream` d1e48ba) into `next`, a new long-lived branch made from main. main stays the 0.16 line and keeps its weekly 2026.x releases; the 27.1 release candidates will come from `next`. This is the last upstream import; from here the fork diverges fully and moves to its own series (27.1). ## What 1.0 changed under us, and what this merge does about it - **Store layout.** 1.0 can't open a 0.16 store and refuses the other edition's. The server now opens a 1.0 store of either edition and stamps it with our own store version (`System` key `0x80`). A store written by a newer inbuxa line is refused. Upgrading from 0.16 goes through the migration tool into a new, empty store (wrapper to follow). - **Features 1.0 dropped from its AGPL code**, restored as ours: multi-tenancy plumbing, metrics and tracing stores, the in-memory store kept in the data store, the `_` subspace (audit log, holds, journal, deliverability, AI limits). - **Registry.** Generated by our own generator (`tools/registry-gen`) from the merged schema, full edition included. Our permissions follow 1.0's last id (643–672), and our trc events moved to 655–669. - **SQL backends.** Our 0.16 work was re-ported onto 1.0's rewritten PostgreSQL/MySQL code: query time limits (stalled connections detached, not returned to the pool), MySQL pool wait timeout, PostgreSQL search tokenizing, `prepare` for unindex, MySQL stopwords. - **Security re-expressions.** Anonymous OAuth client registration stays off unless an operator turns it on (1.0 opened it). Tenant scoping is kept on principal and report reads. - **Compatibility kept for 0.16 users:** - `counter_incr` / `counter_get` in expressions (1.0 dropped them); - `vnd.stalwart.while` / `vnd.stalwart.expressions` Sieve capabilities; - spam-filter blobs move from the pre-rename keys at startup; - `stalwart-app.json` app manifests. - **Tests.** 1.0 moved its test suites private and deleted the unit tests in `crates/`. Our `tests/` suite was ported to 1.0 behaviour. Test helpers sit behind `--cfg inbuxa_tests`, renamed from upstream's cfg, and never set in release builds. Upstream's calls into its unpublished test instrumentation were removed. - **Privacy catalog** updated for 1.0, including a new `cache-swap` source for 1.0's account-cache snapshots. - **THIRD-PARTY.md** covers the third-party files new in 1.0. The strip report is in `docs/fork/strip-reports/v1.0.0.*`. ## Tested Locally, with `RUSTFLAGS="--cfg inbuxa_tests"`: - Integration suite on RocksDB: 102 of 104 passed in the full run. `cluster::broadcast::cluster_tests` needs a coordinator service (`COORDINATOR=`) this machine doesn't run. `system::system_tests` failed in the purge step (see the quota fix below) and passes with the fix. - PostgreSQL: `store::search_tests`, `store::store_tests`, `store::search_gin` and the pool-timeout tests pass. - MySQL: `store::store_tests` and the pool-timeout tests pass. `store::search_tests` fails on the phrase query `'rustic bridge'`, exactly as it does on main today, so it is not from this merge. InnoDB's full-text index loses the phrase for some rows written by concurrent index transactions (the words match; rebuilding the table fixes it). Follow-up below. - Fork checks, `cargo build -p inbuxa --locked`, the workspace test build, and a release build. One test-isolation fix that also applies to main: the quota test left a one-second upload lifetime behind, so the purge test that follows could lose its uploads on a loaded machine. It now restores the upload settings. ## Licensing and safety review - Every changed file's license header was checked. Upstream files are AGPL-3.0-only OR LicenseRef-SEL, ours are AGPL-3.0-only, and the permissive third-party files are listed in THIRD-PARTY.md. No SEL-only file is included, and upstream's `enterprise/` code is absent; the only file under an `enterprise/` path is ours. - THIRD-PARTY.md now also credits the PDF tables in `crates/text-extract`, which match Adobe's cmap-resources, Adobe Glyph List and Core 14 font metrics. It records the four crates 1.0 moved in-tree and relabeled (calcard, jmap-tools, smtp-proto, and mail-auth's report module as smtp-report), all previously published as Apache-2.0 OR MIT. - New dependencies: 14 from crates.io, all permissive except evalexpr 13 (AGPL-3.0-only), plus three in-tree path crates. No new git sources. - The added lines contain no private keys, tokens, live hostnames or personal data. Hidden-character and injected-instruction scans are clean. - Test hooks exist only under `--cfg inbuxa_tests`, which no release or image build sets. The release paths were checked: upload quota, DNSSEC detection, OIDC opaque tokens, DNS updater and MTA-STS policy. - Anonymous OAuth client registration defaults to off and otherwise requires the `oAuthClientRegistration` permission. The 39 new `unsafe` lines are all upstream's, unchanged. ## Follow-ups (not in this PR) - MySQL phrase search after concurrent indexing (above). - Vendor the opentelemetry crates: they come from a git fork on upstream's GitHub account (pinned commit; already the case on main). - Audit the Sieve runtime's `unsafe` arena code (new in 1.0; it runs user scripts). - `Principal/get` account objects (new in 1.0) don't carry the `urn:inbuxa:jmap` or masked-email capabilities that the session object does. - Port back the unit tests 1.0 deleted (0.16.27's, where the code still exists). - Upgrade wrapper around the migration tool, a new upgrade test plan with rehearsals on a copy of production that check the result independently of the tool, operator docs, and installer support (permission and event id remap, `_` copy, deleted items, in-memory tables, a new empty database, full stop and start). - 27.1 versioning (Cargo versions, `brand_version!`, release workflow, image tags). - Wire renames for the console, webmail, installer and docs (`wire-renames-0.16-to-1.0.txt`).

jcoffey-dev added 2 commits 2026-10-27 21:40:24 +00:00

Import upstream v1.0.0, stripped ... d1e48baf90

Upstream commit: 47df4c0baeb8
Enterprise-only files removed or emptied: 68
Enterprise-only snippets removed: 2080 in 147 files
Dangling module declarations removed: 533
Edits turning enterprise off: 12
Third-party code: 25 files, 14 not in THIRD-PARTY.md
Renamed identifiers: 51 in 11 files
Verification: clean

The last upstream import. 1.0 has a new store layout that can't open a
0.16 store, renumbers the settings schema, ties the store's schema
version to the edition, and moves calcard, smtp-proto, jmap-tools and
sieve in-tree (AGPL-3.0-only OR LicenseRef-SEL; taken under the AGPL).
The tests/ directory is gone upstream; main keeps its own. The build
check is clean, and three files on the known list now compile:
tests/src/directory/issuer.rs, tests/src/directory/mod.rs and
tests/src/smtp/inbound/antispam.rs. The 14 new third-party files are
the in-tree calcard, jmap-tools, smtp-proto and smtp-report code.

Merge upstream v1.0.0 16af21c154

jcoffey-dev added 1 commit 2026-10-27 21:51:11 +00:00

THIRD-PARTY: credit the Adobe font data and the crates 1.0 moved in-tree 0a71a123e5

jcoffey-dev added 2 commits 2026-10-27 22:28:54 +00:00

Vendor upstream's opentelemetry-rust fork at the pinned commit 1cb9af1ca4

Check UTF-8 in trusted metadata reads; harden the Sieve script arena 697eb9de70

jcoffey-dev commented 2026-10-27 22:29:36 +00:00

Author

Owner

Copy link

Copy source

Parked, not merged. inbuxa stays on its current base, and v0.16.27 is the last upstream import. 27.1 ships from main.

This branch, next and upstream (v1.0.0 snapshot) stay as the reference for a v1.0 store converter in inbuxa-migrate and for porting individual fixes to main.

Parked, not merged. inbuxa stays on its current base, and v0.16.27 is the last upstream import. 27.1 ships from main. This branch, `next` and `upstream` (v1.0.0 snapshot) stay as the reference for a v1.0 store converter in inbuxa-migrate and for porting individual fixes to main.

jcoffey-dev closed this pull request 2026-10-27 22:29:37 +00:00

Pull request closed

Please reopen this pull request to perform a merge.

Sign in to join this conversation.

Reviewers

No Reviewers

Labels

Clear labels

bug

Something isn't working

dependencies

Dependency updates

documentation

Docs, README or site copy

duplicate

Already tracked elsewhere

enhancement

New feature or improvement

good first issue

Good for newcomers

help wanted

Contributions welcome

invalid

Not a valid issue

needs info

Waiting on the reporter for details

question

Needs an answer, not a code change

security

Security issue or hardening

translation

New language or translation fixes

wontfix

Will not be worked on

priority

p1

High priority

priority

p2

Medium priority

priority

p3

Low priority

No labels

Milestone

No items

No Milestone

Projects

Clear projects

No projects

Assignees

Clear assignees

jcoffey-dev (John Coffey)

No Assignees

1 Participants

Notifications

Subscribe

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: inbuxa/inbuxa-server#215