Cutover: rehearsed, and the Enterprise build can't read the fork's store

The runbook said nothing in it had been rehearsed. Now the sequence has
been, on data made up for the purpose: upstream 0.16.22 in a container as
the install running today, the fork beside it, both unprivileged with
CAP_NET_BIND_SERVICE. It rehearses the sequence, not the data, which is
what the compat tests are for. 27 of 27 checks passed and the rollback
took 1.5 seconds.

The question §"Open" asked about carrying a store back is answered, and
the answer is no. Upstream refuses to start on a store the fork has
opened: "Column families not opened: _". The fork adds one RocksDB column
family for masked email (SUBSPACE_INBUXA = b'_') and opens with
create_missing_column_families, so it creates it on first open; upstream
has no descriptor for it and RocksDB will not open a database holding one
it was not told about.

That makes step 3's "a copy, not a move" load-bearing in a way the step
did not say. One open by the fork is enough: pointing it at the original
even once, to check something, leaves the Enterprise install unable to
start, and there is no rollback after that. It fails loudly and before
reading anything, which is the good version of this failure, but it is
not recoverable.

Two things the rehearsal found that would have wasted time on the day:
memberTenantId does not come down from the domain and is refused on
create, so a tenant "admin" set up the obvious way is a server
administrator and the check passes while proving nothing; and IMAP's
INBOX is not JMAP's account, because mail from an unauthenticated sender
is filed as spam, so the two counts differ before and after alike.

What the rehearsal does not cover is in its README and in §"Open":
systemd and `systemctl disable stalwart` above all, ACME renewal, load,
the front ends, and INBUXA's own data.
This commit is contained in:
2026-09-19 22:23:10 -07:00
parent f338ddf57d
commit 67619f64e9
6 changed files with 809 additions and 6 deletions
+80 -6
View File
@@ -1,6 +1,8 @@
# Cutting INBUXA over to the fork
Status: draft, 2026-09-19. Nothing here has been rehearsed yet.
Status: draft, 2026-09-19. The sequence has been rehearsed once, on
synthetic data ("The rehearsal", below). INBUXA's own data has not been
through it.
SPEC.md §7 step 4. This run is also the first run of the migration tool
INBUXA will ship (`migration.md`): what is done by hand here is what that
@@ -177,12 +179,84 @@ cutover.
- **The logs**, for the `STALWART_*` fallback warnings, which name settings
worth renaming while nobody is under pressure.
## The rehearsal, 2026-09-19
`tools/fork/cutover-rehearsal/` runs this document against data it makes up:
upstream 0.16.22 in a container as the install running today, the fork beside
it, both unprivileged with `CAP_NET_BIND_SERVICE`. It rehearses the sequence,
not the data — the data is `compat-tests.md`. **27 of 27 checks passed, and
the rollback took 1.5 seconds.**
What it turned from assumption into observation:
- The fork opens and serves a store the previous build wrote. Every account
signed in with the password it had, over JMAP and over IMAP; mail read
back as the same set; an alias still delivered.
- A tenant administrator saw exactly its own accounts and domains and still
could not read listeners — across the copy, unchanged. That is the shape
of the check `tenant_compat` could not make, `tenantAdmins` being empty on
the real run. It is not INBUXA's tenant, but the mechanism holds.
- Ports 25, 465 and 993 were bound by an unprivileged process.
- The `STALWART_*` fallback warnings of SPEC.md §2.5 fire, naming each
setting to rename. A third warning is worth knowing about in advance:
with the server configured and not in recovery mode,
`INBUXA_RECOVERY_ADMIN` is **ignored**, and the log says to remove it.
- The rollback restored the exact pre-cutover state and left behind only
what the fork had accepted — the documented cost, measured rather than
asserted.
Two things to carry into the day, neither of them a fault:
- **Set the fixture up right or the tenant check is vacuous.**
`memberTenantId` does not come down from the domain, and is refused on
create (`invalidForeignKey`); it has to be set afterwards. An account
without one is server-wide, so a tenant "admin" without one is a server
administrator and sees everything.
- **IMAP's INBOX is not JMAP's account.** Mail from an unauthenticated
sender is filed as spam, so INBOX counts and message counts differ, before
and after alike. Compare each against itself on the day, or a faithful
move will look like a loss.
## Answered: the Enterprise build cannot read the fork's store
This was open. It is now settled, and the answer is no.
Pointed at a store the fork had opened, upstream 0.16.22 refuses to start:
```
⚠️ Startup failed: Failed to open database:
Error { message: "Invalid argument: Column families not opened: _" }
```
The fork adds one RocksDB column family for masked email
(`SUBSPACE_INBUXA: u8 = b'_'`, `crates/store/src/lib.rs`) and opens the
database with `create_missing_column_families(true)`, so it creates `_` on
first open. Upstream has no descriptor for it, and RocksDB will not open a
database holding a column family it was not told about.
Three consequences:
- The failure is a hard one, at startup, exit code 1, **before any data is
read**. That is the good version: it is loud and immediate, not a slow
corruption.
- **"A copy, not a move" is load-bearing, and more so than step 3 says.**
One open by the fork is enough: the store gains `_` and upstream can never
open it again. Pointing the fork at the original even once — to "just
check" — destroys the rollback path. The copy is the only thing that keeps
the Enterprise install able to start.
- The side-by-side plan is not merely preferable, it is the only shape with
a rollback at all. An in-place swap would have no way back.
## Open
- Whether the Enterprise build can read a store the fork has written. With
the side-by-side plan this only matters if the fork's store has to be
carried back, since the rollback path is the old store, untouched.
- Whether ACME renewal works on the host, which the test suite has not been
able to settle (`container-tests.md`).
able to settle (`container-tests.md`). Untouched by the rehearsal, which
runs with `requestTlsCertificate: false`.
- Whether the front ends need anything at cutover, or follow separately
(SPEC.md §5).
(SPEC.md §5). The rehearsal does not start them.
- What the sequence does on the host rather than in containers: systemd,
`AmbientCapabilities`, and above all `systemctl disable stalwart`, which
has no analogue in the rehearsal and is the one step guarding against two
servers on one set of ports.
- Whether INBUXA's own data survives the sequence, as opposed to opening
under it. That needs a snapshot and a repeat of "Before the day" step 2.