Cutover: the old install is a reference, kept until someone says otherwise

John, 2026-09-19, on both counts. The old install is kept, shut down, not
removed: its unit installed and disabled, its store read-only, started
again if a rollback is ever wanted. When it stops being worth the disk the
tool asks — keep or delete — rather than deciding, because it does not
remove the thing its own rollback depends on.

And the new stack depends on nothing in it. That is the shape's purpose:
/opt/stalwart is a reference, everything needed is copied to new paths, and
when it goes nothing notices. Nothing in the fork works against that —
inbuxa.service substitutes its own prefix, no path names the old tree, and
certificates and ACME keys are in the registry inside the store — so a
dependency, if one appears, was made by hand during the move.

Which is worth proving rather than asserting, and reversibly: nothing open
under the old tree, then rename it and leave it a day under real traffic.
Deleting proves the same thing and cannot be undone.

Two corrections this forces. The rollback has to make the original store
writable again first: the guard of step 3 blocks the Enterprise build
exactly as it blocks the fork, and finding that out during a rollback is
the worst time. And the copy has to be chowned — rsync -a preserves
ownership, so it arrives owned by the old service user while the unit runs
as User=inbuxa.

Also drops the stale "untested" wording about carrying the data back. It
was tested; it is impossible.

docs/spec/cutover.md

@@ -109,6 +109,11 @@ unprivileged process hold port 25.
    Time both passes during the rehearsal. The second one is the number that
    matters: it is the window.
 
+   **Then chown it.** `rsync -a` preserves ownership, so the copy arrives
+   owned by the old service user while `inbuxa.service` runs as `User=inbuxa`
+   — the fork will not be able to write its own store. `chown -R
+   inbuxa:inbuxa` the copied tree before starting anything.
+
    **Then make the original read-only, before the fork exists on this host**
    (`chmod -R a-w`, or `chattr +i` on the directory, or keep it on a
    read-only bind mount). Until this moment the running server's own RocksDB
@@ -165,8 +170,15 @@ before the day, not during it.
 ## Rollback
 
 The Enterprise install and its data are untouched, so rolling back is: stop
-the fork, re-enable and start `stalwart.service`, bring the old ihasmail
+the fork, **make the original store writable again**, re-enable and start
-container back. Minutes, and no restore.
+`stalwart.service`, bring the old ihasmail container back. Minutes, and no
+restore.
+
+The writable step is easy to forget and the guard of step 3 is what makes it
+necessary: the same read-only flag that stops the fork opening the original
+by mistake also stops the Enterprise build starting on it. Undo it first, or
+the rollback fails at the worst possible moment with what looks like a
+permissions problem and isn't.
 
 What it costs is everything the fork accepted while it served: the two
 stores diverge from the moment the fork starts. Mail delivered to the fork's
@@ -175,47 +187,67 @@ the decision to roll back is worth making early and deliberately, and the
 window between "started" and "letting mail flow" is where a rollback is
 free.
 
-**Rolling the data back is a different thing, and untested.** Pointing the
+**Rolling the data back is a different thing, and impossible.** Pointing the
-Enterprise build at the store the fork has been writing to would ask it to
+Enterprise build at the store the fork has been writing to does not work and
-read objects the fork's own features own. Whether it tolerates them has
+cannot be made to: it refuses to start on it at all (below). The fork's
-never been checked, and checking it is worth an hour beforehand rather than
+store is forward-only. The rollback is the old store or nothing.
-an argument at 2am. Until it is checked, treat the fork's store as
-forward-only.
 
 ## Retiring the old install
 
-`/opt/stalwart` goes away once the migration is confirmed (John,
+**The new stack depends on nothing in the old one.** That is the point of
-2026-09-19). It holds the old binary, its config and its data, and nothing
+the shape, not a side effect of it: `/opt/stalwart` is a reference, and
-else the fork needs: ACME account keys and issued certificates live in the
+everything the fork needs — the store, the config, anything else found along
-registry, inside the store, so they came across with the copy
+the way — is copied to new paths under its own tree. When the old directory
-(`crates/common/src/network/acme/`). Removing the directory strands nothing.
+goes, nothing should notice.
 
-What it does end is the rollback, permanently, so it is worth being exact
+Nothing in the fork works against that. `inbuxa.service` substitutes its own
-about when.
+prefix and no path in the tree names `/opt/stalwart`; certificates and ACME
+account keys live in the registry inside the store. The dependency, if one
+appears, will have been created by hand during the move — a store path left
+where it was, a config that still names the old tree, a webmail container
+with a volume under it.
 
-**The rollback stops being a rollback long before this.** From the moment
+So prove it rather than assert it, and prove it reversibly:
-mail flows, going back to the old store means losing everything that
-arrived since. After a day, `/opt/stalwart` is not something anyone would
-actually switch to — it is a cold copy of the pre-cutover state. The
-decision is therefore not "when do we stop being able to roll back", which
-answers itself within hours, but "when do we stop keeping a copy of what
-was there before".
 
-**The gate.** Keep it until the fork has renewed a certificate. That is the
+- Nothing open: `lsof +D /opt/stalwart` against the running fork returns
-one thing in "The first week" that fails quietly and late, and the only
+  nothing, and neither does its config or unit mention the path.
-check whose failure would send anyone back. Waiting for it naturally means
+- **Then rename it.** `mv /opt/stalwart /opt/stalwart.retired` and leave it
-waiting most of 90 days; forcing a renewal instead closes the gate in a
+  a day under real traffic. If anything depended on it, that is when it
-day, and is worth doing for that reason alone.
+  says so, and the fix is one `mv` back. Deleting proves the same thing and
+  cannot be undone; renaming is the same test with a way out.
 
-**Archive before removing.** A compressed copy of the pre-cutover store,
-kept off the host, costs little and answers "what did this look like
-before" for as long as anyone asks. Deleting the directory is fine; being
-left with no copy of the pre-migration state is a different thing, and not
-what is being asked for here.
 
-Only then: remove the unit file, `/opt/stalwart`, and the old ihasmail
+The old install is **kept, shut down**, not removed (John, 2026-09-19). Its
-container and image. Until then the unit stays installed and disabled, and
+unit stays installed and disabled and its store stays read-only, so it costs
-the store stays read-only (step 3).
+a stopped service and the disk its data sits on, and nothing else. If a
+rollback is ever wanted it is started again (see Rollback).
+
+When it stops being worth keeping, that is the operator's call and the tool
+**asks** rather than decides: keep it, or delete it. The tool never removes
+the thing its own rollback depends on without being told to
+(`migration.md`).
+
+**Budget for two full copies indefinitely, not temporarily.** Step 3's
+second copy is not reclaimed on a schedule; it is reclaimed when somebody
+answers the prompt, which may be never.
+
+Deleting it, when that is the answer, strands nothing else the fork needs:
+ACME account keys and issued certificates live in the registry, inside the
+store, so they came across with the copy
+(`crates/common/src/network/acme/`). `/opt/stalwart` holds the old binary,
+its config and its data, and that is all.
+
+**When to ask.** Not on a timer. The signal worth waiting for is the fork
+renewing a certificate, which is the one item in "The first week" that fails
+quietly and late and the only failure that would genuinely send anyone back.
+Waiting for it to happen naturally means most of 90 days; forcing a renewal
+closes the question in a day, and is worth doing for that reason alone.
+
+**Archive before deleting**, if deleting is the answer. A compressed copy of
+the pre-cutover store, kept off the host, costs little and answers "what did
+this look like before" for as long as anyone asks. Losing the last copy of
+the pre-migration state is a different decision from reclaiming a directory,
+and worth making separately.
 
 ## After: stock ihasmail against the fork
 

docs/spec/migration.md

@@ -111,10 +111,18 @@ A first-class path, not an appendix.
   has flowed leaves that mail in the fork's store. The tool says how many
   messages that is rather than letting the operator find out.
 - **What it never does**: point the old server at the store the fork has
-  been writing. That has never been tested and is not a recovery path
+  been writing. That is not a recovery path and cannot be made into one —
+  the old build refuses to open a store the fork has touched at all
   (`cutover.md`).
+- **What it undoes first**: the read-only flag on the original store. The
+  guard that stops the fork opening it by mistake also stops the old build
+  starting on it.
 - The old install is kept until the operator retires it deliberately. The
-  tool does not delete the thing its rollback depends on.
+  tool does not delete the thing its rollback depends on, and when the
+  operator is done with it the tool **asks** — keep, or delete — rather
+  than deciding. Nothing the fork runs on may be inside it by then, which
+  `verify` checks: no open file under the old tree, and no path in the
+  fork's config or unit naming it.
 
 ## Rehearsal