Replace the sandbox dry run with a read-only rehearsal
`run --dry-run` cloned the data directory into a sandbox, migrated the copy,
booted it, and compared content before and after. Running that design
against a real 0.15.5 instance and a real production settings corpus
retired it:
* The mechanics were never the risk. Backup, dump, convert and the
recovery-mode store migration all worked essentially first time.
* Its final comparison cannot work at all. It needs the migrated sandbox
to answer an API, and server.listener is not among the settings
migrate_v016.py carries - so a migrated instance has no listeners and
answers on nothing. That is the true post-migration state, not a
sandbox artifact to engineer around.
* The expensive half bought the least: against a 3.6 GB production store
it copies the data twice, reading a live mail store, to prove RocksDB
files copy and recovery mode can open them.
Meanwhile the cheap half found every problem that would have derailed a
real migration - an empty defaultHostname v0.16 rejects, passwords v0.16
refuses to create, and a 12,182-key reconstruction worklist - and needs no
data copy at all.
So `stalwart-migrate rehearse`: preflight, dump, convert, report. It copies
nothing, starts no server, and never writes to the store, so it is safe to
run against production repeatedly without a maintenance window. It needs no
target binary either, since convert is pure Python.
The scratch directory is cleaned up as before, with the rehearsal's two
conclusions lifted out first and recorded as artifacts: export.json (what
will carry over) and unmigrated.txt (what will not). Recording an artifact
whose path was about to be deleted was a bug in the first cut of this;
both now resolve.
`run` keeps its refusal and explains where rehearse went. `--dry-run` is
kept as a flag purely to say what replaced it.
Verified against the smoke VM end to end: rehearsal completes read-only in
seconds and reports 3505 unmigrated settings on a default install,
listeners included.
This commit is contained in:
+161
-113
@@ -90,7 +90,7 @@ smoke test, not a full migration.
|
||||
┌─────────────┐ ┌───────────┐ ┌────────────┐ ┌───────────────┐ ┌────────────┐ ┌────────────┐
|
||||
│ PREFLIGHT │──▶│ BACKUP │──▶│ STAGE NEW │──▶│ RECOVERY-MODE │──▶│ CUTOVER │──▶│ VALIDATE │
|
||||
│ (checks, │ │ (defense │ │ BINARY + │ │ MIGRATE │ │ (swap, up, │ │ (functional│
|
||||
│ dry-run) │ │ in depth) │ │ config │ │ (apply plan) │ │ smoke) │ │ + counts) │
|
||||
│ rehearse) │ │ in depth) │ │ config │ │ (apply plan) │ │ smoke) │ │ + counts) │
|
||||
└─────────────┘ └───────────┘ └────────────┘ └───────────────┘ └────────────┘ └────────────┘
|
||||
│ │ │ │ │ │
|
||||
└─────────────────┴────────────────┴─── on failure ──┴─────────────────┴──▶ STOP + REPORT
|
||||
@@ -197,9 +197,17 @@ pre-migration instance is still up, rather than after it isn't.
|
||||
rules, auth backend config — by diffing the old effective config against
|
||||
the new schema and emitting a best-effort JMAP object set for
|
||||
`stalwart-cli apply`. This is flagged clearly as best-effort and included
|
||||
in the final report for manual review; it's the one part of the
|
||||
documented procedure that's explicitly manual today, and silently getting
|
||||
it wrong (rather than flagging it) would be worse than not attempting it.
|
||||
in the final report for manual review; silently getting it wrong (rather
|
||||
than flagging it) would be worse than not attempting it.
|
||||
|
||||
**This is no longer an optional enhancement.** Measured against a real
|
||||
production instance, `migrate_v016.py` migrated 219 of 12,401 settings —
|
||||
1.8% — leaving 12,182 for the operator to recreate by hand, including
|
||||
`server.listener`. A migrated instance therefore serves nothing until
|
||||
somebody rebuilds its listeners, whatever else went right. Something has
|
||||
to generate that plan; the only question is whether it is this tool,
|
||||
reviewably, or a human under time pressure during a cutover window.
|
||||
`unmigrated.txt` (§4.9) is the input it should be built from.
|
||||
- Stage new systemd unit / Compose file changes without activating them.
|
||||
|
||||
### 4.4 Recovery-mode migration
|
||||
@@ -295,6 +303,21 @@ code path, so it doesn't rot independently.
|
||||
|
||||
### 4.7 Post-migration validation
|
||||
|
||||
**What this suite can assert depends on the boundary being crossed, and on
|
||||
the 0.15/0.16 boundary it is less than this section originally claimed.**
|
||||
Stalwart 0.15.x reports no per-mailbox message counts at any endpoint, and
|
||||
the impersonation login 0.16 offers returns 401 there, so there are no
|
||||
"before" counts to compare against — the before/after message-count
|
||||
comparison is simply unavailable for the migration this tool exists to
|
||||
perform. Both versions report per-account used quota, which is captured on
|
||||
both sides, but §4.5 notes the migration resets quotas to zero pending
|
||||
recalculation, so it is recorded rather than asserted on. What remains
|
||||
checkable across the boundary is that every account and every domain
|
||||
survived, and the reports say so in those words rather than implying a
|
||||
no-data-loss guarantee that was not measured. See
|
||||
`internal/validate/content_integrity.go`.
|
||||
|
||||
|
||||
Runs automatically after cutover; failure here stops the run, reports
|
||||
loudly, and exits non-zero, leaving the operator to decide what to restore
|
||||
(§4.8).
|
||||
@@ -389,66 +412,78 @@ the run and reports, and a human decides what to restore. Both are
|
||||
deliberate trades for not shipping a recovery path that has never been
|
||||
tested against a real server.
|
||||
|
||||
### 4.9 Dry run
|
||||
### 4.9 Rehearsal (was: dry run)
|
||||
|
||||
`stalwart-migrate run --dry-run` runs the real migration mechanics against a
|
||||
disposable sandbox clone of the data, so an operator can get genuine
|
||||
confidence *before* committing to a real cutover — not a simulation that
|
||||
skips the fragile parts, the actual recovery-mode migration (§4.4) and a
|
||||
post-migration boot check, just pointed somewhere disposable:
|
||||
**This section was rewritten after running the previous design against a
|
||||
real 0.15.5 instance and a real production settings corpus. What it found
|
||||
inverted the design's assumptions, so the reasoning is recorded here rather
|
||||
than quietly replaced.**
|
||||
|
||||
1. **Preflight** (§4.1) runs for real, read-only, against the live instance.
|
||||
2. **Backup** (§4.2) runs for real too, with one exception:
|
||||
`SkipBinaryPreservation` is set, so the production binary at the real
|
||||
install path is never moved aside. Taking a *consistent* filesystem
|
||||
snapshot of an embedded store still means the live service should be
|
||||
stopped first (the same requirement Stalwart's own export tooling has) —
|
||||
this tool doesn't automate that stop/start today (no systemd/Docker
|
||||
control exists yet), so a dry-run without a manual stop first is a
|
||||
best-effort snapshot of a live, in-use store, and the CLI says so.
|
||||
3. **Convert**: `migrate_v016.py convert` turns the settings/principals dump
|
||||
into `config.json` + `export.json`, using the script's own documented
|
||||
`--patch-paths <old>=<new>` flag to point the generated config at the
|
||||
sandbox data directory instead of the real one. This is the officially
|
||||
documented mechanism for exactly this kind of path redirection — the tool
|
||||
deliberately does not try to rewrite `config.json`'s contents itself,
|
||||
since depending on its exact schema (which has already changed once,
|
||||
0.15 → 0.16) is a correctness risk this tool avoids wherever an official
|
||||
alternative exists.
|
||||
4. The verified backup copy is cloned again into the sandbox directory
|
||||
(never reusing the same directory recovery mode is about to mutate as the
|
||||
one a manual restore would use).
|
||||
5. **Recovery-mode migration** (§4.4) runs for real against the sandbox:
|
||||
the actual target binary, actual `STALWART_RECOVERY_MODE=1` boot, actual
|
||||
`stalwart-cli apply`.
|
||||
6. **Boot check + content integrity**: the migrated sandbox is started once
|
||||
more as an ordinary boot (no recovery-mode env vars) and polled until its
|
||||
HTTP listener answers, confirming the migrated store doesn't just accept
|
||||
a settings apply but actually comes up cleanly afterward. If preflight
|
||||
captured a pre-migration snapshot (§4.1, requires `--admin-url`), the
|
||||
same boot is then used to capture a fresh post-migration snapshot and
|
||||
compare the two — this is the actual no-data-loss guarantee, not just
|
||||
"the mechanics ran": every account and mailbox from before must still be
|
||||
found afterward (matching by exact name, falling back to the part before
|
||||
`@` since v0.16's own migration rewrites bare usernames to full email
|
||||
addresses) with an identical message count. A mismatch or a missing
|
||||
account fails the check. This covers the message-count half of §4.7's
|
||||
full suite; DKIM/TLS fingerprint checks and a live mail-flow SMTP→IMAP
|
||||
smoke test are still open.
|
||||
7. Every byte written by steps 2–6 (the fs-backup copy, settings/principals
|
||||
dumps, downloaded `migrate_v016.py`, sandbox clone, and generated
|
||||
`config.json`/`export.json`) lives under one per-run directory
|
||||
(`work-dir/<run-id>`), which is removed on *every* exit path - success,
|
||||
a failed check partway through, or an early refusal - via a deferred
|
||||
cleanup, not just the happy path. The only thing left behind afterward
|
||||
is the checkpoint's `state.json` under `--state-dir`: a small structured
|
||||
success/failure log (which check failed and why), not bulk data.
|
||||
`--keep-artifacts` opts out for inspecting a failure. Nothing at the real
|
||||
binary path, the real service, or the real data directory's *contents*
|
||||
is ever mutated by steps 2–6 in the first place.
|
||||
The original dry run existed to answer *"will the migration mechanics
|
||||
work?"* — it cloned the data, ran the real recovery-mode migration against
|
||||
the clone, booted the result, and compared content before and after. Three
|
||||
findings retire that design:
|
||||
|
||||
A same-boundary patch bump (§4.6) has no recovery phase to simulate — dry
|
||||
run for that plan is just preflight + backup.
|
||||
1. **The mechanics were never the risk.** Backup, settings dump, convert
|
||||
and the recovery-mode store migration all worked essentially first time
|
||||
against real software. The failures were everywhere else.
|
||||
2. **The final comparison cannot work, at all.** It needs the migrated
|
||||
sandbox to answer an API. `server.listener` is not among the settings
|
||||
`migrate_v016.py` migrates, so a migrated instance has no listeners and
|
||||
answers on nothing. That is not a sandbox artifact to engineer around —
|
||||
it is the true post-migration state.
|
||||
3. **The expensive half buys the least.** Against a 3.6 GB production store
|
||||
the old flow copies the data twice (backup + sandbox clone, ~11 GB and a
|
||||
long wait) while reading a live mail store, to prove that RocksDB files
|
||||
copy correctly and that recovery mode can open them. Real, but modest.
|
||||
|
||||
Meanwhile the cheap half — dump, convert, and report what did *not* convert
|
||||
— is what caught every problem that would have derailed a real migration:
|
||||
an empty `defaultHostname` that v0.16 rejects, accounts whose passwords
|
||||
v0.16 refuses to create, and a reconstruction worklist of 12,182 settings.
|
||||
It needs no data copy at all.
|
||||
|
||||
So the phase reduces to the half that earns its cost:
|
||||
|
||||
**`stalwart-migrate rehearse`.** Run preflight (§4.1, read-only), dump
|
||||
settings and principals from the live instance, run `migrate_v016.py
|
||||
convert`, and report:
|
||||
|
||||
- the generated `export.json` plan (what *will* carry over), and
|
||||
- `unmigrated.txt` (what will *not*, grouped and counted — see §4.3).
|
||||
|
||||
That is the whole phase. It copies no data, clones nothing, starts no
|
||||
server, and never writes to the store — so it is safe to run against
|
||||
production repeatedly, early and often, without a maintenance window. It
|
||||
answers the question that actually decides a migration plan: *what will I
|
||||
have to rebuild by hand, and does my configuration convert at all?*
|
||||
|
||||
The sandbox is gone. Cloning the store to run a migration against the copy
|
||||
proved only that the store migrates and opens — which is worth something,
|
||||
but not the disk and the wait, and not the risk of reading a live store to
|
||||
get it. Where that assurance is wanted, rehearse the whole thing on a
|
||||
throwaway VM restored from a backup, which is what the smoke environment
|
||||
already does and does better.
|
||||
|
||||
Consequences worth stating, since they make this phase much cheaper than
|
||||
its predecessor:
|
||||
|
||||
- **No target binary is needed.** `convert` is pure Python; nothing in
|
||||
this phase executes a Stalwart binary of either version.
|
||||
- **No disk headroom is needed.** Nothing is copied. Preflight's
|
||||
free-space check still runs, but it is anticipating the backup a real
|
||||
`run` will take, not anything rehearse does — and it says so.
|
||||
- **Rehearsal performs no content-integrity comparison.** §4.7 explains
|
||||
why that is unavailable on this boundary regardless of how it is staged.
|
||||
|
||||
Artifacts live under `work-dir/<run-id>` and are removed on every exit path
|
||||
unless `--keep-artifacts` is passed — with one deliberate exception.
|
||||
`unmigrated.txt` is the operator's reconstruction worklist and is preserved
|
||||
and checksummed even on a clean run, because deleting it would throw away
|
||||
the most useful output of the whole exercise.
|
||||
|
||||
A same-boundary patch bump (§4.6) needs no settings conversion at all, so
|
||||
`rehearse` for that plan reports that there is nothing to rehearse.
|
||||
|
||||
## 5. State machine / checkpointing
|
||||
|
||||
@@ -471,21 +506,22 @@ is the same problem.
|
||||
|
||||
```
|
||||
stalwart-migrate preflight [--config PATH] ... # read-only, prints the report
|
||||
stalwart-migrate run --dry-run [--target-binary PATH] ... # implemented — see §4.9
|
||||
[--keep-artifacts]
|
||||
(without --dry-run: refused today — see §8)
|
||||
stalwart-migrate rehearse [--keep-artifacts] ... # read-only; see §4.9
|
||||
stalwart-migrate run (refused today — see §8)
|
||||
stalwart-migrate status [run-id] # implemented
|
||||
stalwart-migrate report <run-id> [--json] # not yet implemented
|
||||
```
|
||||
|
||||
`run` is the only command that mutates anything, and it always starts with
|
||||
preflight. Nothing in this tool restores a failed migration (§4.8), so there
|
||||
`run` is the only command that mutates anything: `rehearse` reads the live
|
||||
instance and writes only inside its own work directory. `run` always starts
|
||||
with preflight. Nothing in this tool restores a failed migration (§4.8), so there
|
||||
is no `rollback` command, and no `confirm` step to close a rollback window
|
||||
that no longer exists.
|
||||
|
||||
The migration-time artifacts a run leaves behind — the preserved old binary,
|
||||
the settings and principals dumps, the preserved service definition, and
|
||||
(for the dry-run path) the filesystem copy — are never pruned automatically.
|
||||
and rehearsal's converted plan and unmigrated worklist — are never pruned
|
||||
automatically.
|
||||
They're small next to the data directory, they're what a manual restore
|
||||
reaches for first, and deleting them on a schedule to reclaim disk would be
|
||||
the tool making a call that isn't its to make. Flags shown here are the
|
||||
@@ -512,10 +548,10 @@ stalwart-migrator/
|
||||
|
||||
There's no separate `internal/stage` package: the `convert` half of
|
||||
`migrate_v016.py` lives in `internal/backup` next to `dump` (same script,
|
||||
same invocation pattern), and the dry-run sandbox-cloning logic that stands
|
||||
in for the rest of §4.3 currently lives directly in `cmd/run.go` rather than
|
||||
its own package. Now that `internal/cutover` exists, that's the code a real
|
||||
staging phase would be generalized out of.
|
||||
same invocation pattern). The sandbox-cloning logic that used to stand in
|
||||
for the rest of §4.3 lived directly in `cmd/run.go` and goes away with the
|
||||
sandbox (§4.9); what §4.3 still needs is the apply-plan generator, which
|
||||
has no code yet at all.
|
||||
|
||||
There's no `internal/rollback` either, and that's a deliberate removal
|
||||
rather than a gap — see §4.8.
|
||||
@@ -541,12 +577,13 @@ happens to need them. `preflight.DeploymentKind` is a type alias for
|
||||
script is Stalwart's, not ours — need a policy for what happens when it
|
||||
changes upstream (re-vendor + re-test before bumping the pin, never
|
||||
silently float to `main`).
|
||||
- **Best-effort settings apply-plan (§4.3)**: needs real-world testing
|
||||
against a variety of existing SMTP/routing/spam configs before it's
|
||||
trusted un-reviewed; v1 should probably always require operator sign-off
|
||||
on that specific generated plan even with `--yes` set for everything else.
|
||||
Not started — dry-run currently only replays what `migrate_v016.py`
|
||||
itself converts.
|
||||
- **Settings apply-plan (§4.3): now the critical path, not an enhancement.**
|
||||
Measured against production, `migrate_v016.py` carries 1.8% of the
|
||||
settings; `server.listener` is not among them, so a migrated instance
|
||||
answers on no ports until the rest is rebuilt. Building this from
|
||||
`unmigrated.txt` is what would make both a meaningful rehearsal and a
|
||||
working cutover possible. It should still require explicit operator
|
||||
sign-off even with `--yes` set for everything else.
|
||||
- **Account/mailbox enumeration** (`stalwartapi.Client.AccountSnapshot`):
|
||||
**implemented**, including per-mailbox message counts. Account count and
|
||||
domains come from `x:Account/query` + `x:Account/get` against Stalwart's
|
||||
@@ -579,40 +616,43 @@ happens to need them. `preflight.DeploymentKind` is a type alias for
|
||||
the current released version actually exposes — a reminder that "read the
|
||||
source" and "read what's actually shipped" can disagree, and it's worth
|
||||
checking both before changing already-working code on the strength of one.
|
||||
Preflight now populates `RunState.PreflightSnapshot.MailboxCounts` when
|
||||
`--admin-url` is set, and `validate.BootCheck` now compares it against a
|
||||
fresh post-migration snapshot as part of the same boot (§4.9 step 6) —
|
||||
proven end-to-end with a live smoke test that deliberately made the
|
||||
"after" instance report fewer messages than the "before" snapshot and
|
||||
confirmed the dry run failed loudly with the exact before/after counts,
|
||||
rather than just trusting that. **Still open**: preflight/validate always
|
||||
attempt every account serially with no sampling/threshold, which could be
|
||||
slow on a large install — `--full-validation`'s sampling idea from §4.7
|
||||
hasn't been built yet for this; and DKIM/TLS fingerprint checks plus a
|
||||
live mail-flow SMTP→IMAP smoke test (the rest of §4.7's suite) aren't
|
||||
implemented. With this done, recovery, backup, dry-run, and account/
|
||||
mailbox snapshotting all work end-to-end, and dry-run's comparison is now
|
||||
the closest thing to §4.7's actual no-data-loss guarantee this tool has —
|
||||
the remaining major gap is §4.3 staging and the production pipeline
|
||||
(below).
|
||||
**Superseded in part.** That mailbox-count comparison was verified only
|
||||
against fabricated fixtures, and against a real 0.15.5 source it does not
|
||||
work at all: 0.15.x reports no per-mailbox counts and refuses the
|
||||
impersonation login, so the "before" side is always empty, and the
|
||||
comparison used to iterate that empty map and report "all message counts
|
||||
match" — a vacuous pass on the strongest claim this tool makes. It now
|
||||
states plainly when counts were not compared (§4.7). Account and domain
|
||||
enumeration against 0.15.x works via its REST principal API.
|
||||
**Still open**: preflight/validate attempt every account serially with no
|
||||
sampling, which could be slow on a large install; and DKIM/TLS
|
||||
fingerprint checks plus a live mail-flow SMTP→IMAP smoke test (the rest
|
||||
of §4.7's suite) aren't implemented.
|
||||
- **Cutover is built; nothing wires it into a production run yet.**
|
||||
`internal/cutover` (§4.5) and `internal/service` are implemented and
|
||||
tested. `run` without `--dry-run` still refuses, for one remaining
|
||||
reason: **§4.3 stage doesn't exist**, and neither does the production
|
||||
pipeline that would run preflight → backup → stage → recovery-mode →
|
||||
cutover → validate against real paths instead of a sandbox. What stage
|
||||
still needs: downloading and verifying the target binary into a staging
|
||||
path (`preflight.ResolveRelease` and `backup.DownloadFile` between them
|
||||
already have the pieces), running the convert step against real paths
|
||||
rather than the dry-run's patched sandbox ones, and the best-effort
|
||||
settings apply-plan, which is its own open question below.
|
||||
- **Nothing has ever run against a real Stalwart.** Every test in this
|
||||
repo drives fake `systemctl`, `psql` and `stalwart` binaries and
|
||||
httptest servers. That's sound for logic and ordering and is not
|
||||
evidence about production. One smoke test on a throwaway VM - real
|
||||
0.15.5, real systemd unit, a few accounts with mail - would settle the
|
||||
quota wire format, systemd drop-in handling, and cutover's unit rewrite
|
||||
at once. It should happen before §4.3 is wired, not after.
|
||||
tested against fakes. `run` still refuses, for one remaining reason:
|
||||
**§4.3 stage doesn't exist**, and neither does the production pipeline
|
||||
that would run preflight → backup → stage → recovery-mode → cutover →
|
||||
validate against real paths. What stage still needs: downloading and
|
||||
verifying the target binary into a staging path
|
||||
(`preflight.ResolveRelease` and `backup.DownloadFile` between them
|
||||
already have the pieces), running convert against real paths, and the
|
||||
settings apply-plan — which, per the measurement above, is what decides
|
||||
whether the migrated server serves anything at all.
|
||||
- **What has and hasn't been proven against real software.** A smoke VM
|
||||
(Debian 13, real Stalwart 0.15.5 under a real systemd unit, RocksDB,
|
||||
seeded accounts and mail) has now exercised preflight, backup, the
|
||||
settings dump, `migrate_v016.py` convert, and the recovery-mode store
|
||||
migration end to end - and a scrubbed copy of a production settings
|
||||
corpus has been through the converter. Everything in §4.9's rewrite and
|
||||
most of §8's newer entries came from that, not from reading code.
|
||||
|
||||
Still unproven against real software: **cutover** (never executed - its
|
||||
unit rewrite, service control and quota recalculation are tested only
|
||||
against fakes), the **`x:Task` quota wire format**, **systemd drop-in**
|
||||
handling, and anything on a **non-RocksDB backend** or a **Docker**
|
||||
deployment. Cutover is the gap that matters most, since it is the phase
|
||||
that mutates production.
|
||||
- **Quota recalculation is grounded but unproven.** The `x:Task` wire
|
||||
format comes from Stalwart's schema reference rather than a live server;
|
||||
§4.5 lists exactly which two details are inferred. A smoke test against a
|
||||
@@ -630,10 +670,18 @@ happens to need them. `preflight.DeploymentKind` is a type alias for
|
||||
them and refusing.
|
||||
- **Nothing prevents concurrent runs.** Two invocations against the same
|
||||
run-id would both proceed; there's no lock file or equivalent.
|
||||
- **Dry-run's un-stopped backup snapshot** (§4.9 step 2): a dry-run still
|
||||
backs up a live, in-use store unless the operator stops it manually first.
|
||||
`internal/service` now makes doing this properly possible - dry-run just
|
||||
hasn't been wired to offer it yet.
|
||||
- **`rehearse` (§4.9) is designed but not built.** The command is still
|
||||
`run --dry-run` with the old sandbox-cloning shape. Building it is mostly
|
||||
deletion: the dump, convert and report pieces already exist and work
|
||||
against real instances; what goes away is the backup clone, the sandbox,
|
||||
the recovery-mode cycle and the boot check. `cmd/run.go`'s sandbox logic
|
||||
disappears with it, which also removes the reason §7 gives for there
|
||||
being no `internal/stage` package.
|
||||
- **Post-migration validation has no reachable instance to validate**
|
||||
(§4.7/§4.9). Until the apply-plan reconstructs listeners, nothing that
|
||||
boots from a converted store can answer an API, so "did the migration
|
||||
preserve the data" cannot be asked of the migrated instance at all. This
|
||||
is the strongest argument for building the apply-plan first.
|
||||
|
||||
## Sources
|
||||
|
||||
|
||||
@@ -19,17 +19,20 @@ them into a production run yet, so `run` still refuses.
|
||||
| Command | State |
|
||||
|---|---|
|
||||
| `stalwart-migrate preflight` | **Works** — read-only checks and a migration plan |
|
||||
| `stalwart-migrate run --dry-run` | **Works** — preflight, real backup, sandboxed trial conversion |
|
||||
| `stalwart-migrate rehearse` | **Works** — read-only; converts your settings and reports what won't carry over |
|
||||
| `stalwart-migrate run` | **Refuses on purpose** — see below |
|
||||
| `stalwart-migrate status <id>` | **Works** |
|
||||
| `stalwart-migrate report <id>` | Not implemented |
|
||||
|
||||
**`run` without `--dry-run` deliberately refuses to proceed.** Cutover
|
||||
(ARCHITECTURE.md §4.5) is implemented and tested, but nothing calls it: the
|
||||
staging phase (§4.3) and the production pipeline that would run preflight →
|
||||
backup → stage → recovery-mode → cutover → validate against real paths don't
|
||||
exist yet. `run` stops rather than going partway. That refusal is the correct
|
||||
behaviour today, not a bug.
|
||||
**`run` deliberately refuses to proceed.** Cutover (ARCHITECTURE.md §4.5) is
|
||||
implemented, but nothing calls it: the staging phase (§4.3) and the pipeline
|
||||
that would run preflight → backup → stage → recovery-mode → cutover →
|
||||
validate against real paths don't exist yet. `run` stops rather than going
|
||||
partway. That refusal is the correct behaviour today, not a bug.
|
||||
|
||||
**Start with `rehearse` instead.** It is read-only, needs no maintenance
|
||||
window, and answers the question that actually shapes a migration plan —
|
||||
see below.
|
||||
|
||||
Package state:
|
||||
|
||||
@@ -91,11 +94,36 @@ pre-created writable `/var/lib/stalwart-migrator`. (`run` takes `--work-dir`
|
||||
for its scratch space, but that is a different directory and does not move
|
||||
the checkpoint store.)
|
||||
|
||||
`run --dry-run` performs a **real backup**, which touches the live data
|
||||
directory — read the caveat the command prints before using it on anything
|
||||
you care about. Where the plan crosses the 0.15/0.16 boundary it clones that
|
||||
verified backup into a disposable sandbox and converts the copy, leaving the
|
||||
original untouched.
|
||||
`rehearse` is the next step, and unlike everything else here it is worth
|
||||
running today — see the next section.
|
||||
|
||||
## Rehearse before you migrate
|
||||
|
||||
```sh
|
||||
stalwart-migrate rehearse --admin-url https://mail.example.com \
|
||||
--admin-user admin --target 0.16.14
|
||||
```
|
||||
|
||||
It runs preflight, dumps your settings and principals, converts them with
|
||||
Stalwart's own `migrate_v016.py`, and reports **both halves** of the result:
|
||||
the apply plan of what will carry over, and the worklist of what will not.
|
||||
|
||||
It copies no data, clones nothing, starts no server, and never writes to the
|
||||
store, so it is safe to run against production repeatedly and without a
|
||||
maintenance window.
|
||||
|
||||
Expect the worklist to be long. Measured against a real production instance,
|
||||
`migrate_v016.py` carried **219 of 12,401 settings — 1.8%**. The rest,
|
||||
including `server.listener`, has to be rebuilt by hand; until it is, a
|
||||
migrated instance answers on no ports at all. Both outputs are preserved
|
||||
under `<state-dir>/runs/<run-id>/` (`export.json` and `unmigrated.txt`) even
|
||||
though the rest of the scratch directory is cleaned up, because they are the
|
||||
conclusions.
|
||||
|
||||
This replaced an earlier `run --dry-run` that cloned the data directory into
|
||||
a sandbox and migrated the copy. That proved the store opens, at the cost of
|
||||
copying it twice — while the half that found every real problem needed no
|
||||
copy at all. ARCHITECTURE.md §4.9 has the reasoning.
|
||||
|
||||
## Recovery is your job
|
||||
|
||||
|
||||
@@ -23,6 +23,8 @@ func main() {
|
||||
switch os.Args[1] {
|
||||
case "preflight":
|
||||
err = runPreflight(os.Args[2:])
|
||||
case "rehearse":
|
||||
err = runRehearse(os.Args[2:])
|
||||
case "run":
|
||||
err = runRun(os.Args[2:])
|
||||
case "status":
|
||||
@@ -45,7 +47,8 @@ func usage() {
|
||||
|
||||
commands:
|
||||
preflight run read-only checks and print the migration plan
|
||||
run --dry-run: simulate and validate against a sandbox (real cutover isn't implemented yet)
|
||||
rehearse convert this instance's settings and report what will NOT carry over (read-only)
|
||||
run perform the migration (not implemented yet - refuses)
|
||||
status show the state of an in-progress or completed run
|
||||
report print the validation report for a run`)
|
||||
}
|
||||
|
||||
@@ -0,0 +1,246 @@
|
||||
// SPDX-FileCopyrightText: 2026 LINUXexpert-org
|
||||
// SPDX-License-Identifier: GPL-3.0-or-later
|
||||
|
||||
package main
|
||||
|
||||
import (
|
||||
"context"
|
||||
"flag"
|
||||
"fmt"
|
||||
"io"
|
||||
"net/http"
|
||||
"os"
|
||||
"path/filepath"
|
||||
|
||||
"github.com/LINUXexpert-org/stalwart-migrator/internal/backup"
|
||||
"github.com/LINUXexpert-org/stalwart-migrator/internal/checkpoint"
|
||||
"github.com/LINUXexpert-org/stalwart-migrator/internal/plan"
|
||||
"github.com/LINUXexpert-org/stalwart-migrator/internal/preflight"
|
||||
)
|
||||
|
||||
// runRehearse implements `stalwart-migrate rehearse` (ARCHITECTURE.md §4.9):
|
||||
// preflight, dump this instance's settings and principals, convert them with
|
||||
// migrate_v016.py, and report both halves of the result - what will carry
|
||||
// over, and what will not.
|
||||
//
|
||||
// It copies no data, clones nothing, starts no server, and never writes to
|
||||
// the store, so it is safe to run against production repeatedly and without
|
||||
// a maintenance window. That is a deliberate narrowing from the sandbox-
|
||||
// cloning dry run this replaces: cloning the store proved only that the
|
||||
// store migrates and opens, at the cost of copying the data twice and
|
||||
// reading a live store to do it, while the half that found every real
|
||||
// problem - an empty defaultHostname v0.16 rejects, passwords v0.16 refuses,
|
||||
// and a 12,182-key reconstruction worklist - needs no copy at all.
|
||||
//
|
||||
// The worklist is the point. Measured against a real production instance,
|
||||
// migrate_v016.py carried 219 of 12,401 settings; server.listener was not
|
||||
// among them, so a migrated instance answers on no ports until an operator
|
||||
// rebuilds them. Anything that reported such a migration as a success
|
||||
// without saying so would be actively misleading.
|
||||
func runRehearse(args []string) (err error) {
|
||||
fs := flag.NewFlagSet("rehearse", flag.ExitOnError)
|
||||
binaryPath := fs.String("binary", "/usr/local/bin/stalwart", "path to the currently-installed stalwart binary")
|
||||
configPath := fs.String("config", "/etc/stalwart/config.toml", "path to stalwart's current config file")
|
||||
dataDir := fs.String("data-dir", "/var/lib/stalwart", "stalwart data directory (read-only here; used by preflight's checks)")
|
||||
containerName := fs.String("container", "stalwart", "docker container name, if applicable")
|
||||
adminURL := fs.String("admin-url", "", "base URL for the live instance's admin/JMAP API (required)")
|
||||
adminUser := fs.String("admin-user", "", "admin username")
|
||||
adminPassword := fs.String("admin-password", os.Getenv("STALWART_MIGRATE_ADMIN_PASSWORD"),
|
||||
"admin password (or set STALWART_MIGRATE_ADMIN_PASSWORD)")
|
||||
targetVersion := fs.String("target", "latest", `target Stalwart version, or "latest"`)
|
||||
stateDir := fs.String("state-dir", checkpoint.DefaultBaseDir, "directory to store run checkpoints in")
|
||||
workDir := fs.String("work-dir", "/var/lib/stalwart-migrator/work", "scratch directory for the dumps and converted plan (cleaned up afterward - see --keep-artifacts)")
|
||||
pythonPath := fs.String("python", "python3", "path to python3")
|
||||
migrationScriptSHA256 := fs.String("migration-script-sha256", "", "pinned sha256 of migrate_v016.py (recommended; the first run prints the hash to pin)")
|
||||
minFree := fs.Float64("min-free-multiple", 2.0, "free-space multiple preflight checks for; rehearsal itself copies nothing")
|
||||
keepArtifacts := fs.Bool("keep-artifacts", false, "don't delete work-dir/<run-id> afterward")
|
||||
if err := fs.Parse(args); err != nil {
|
||||
return err
|
||||
}
|
||||
if *adminURL == "" {
|
||||
return fmt.Errorf("--admin-url is required: rehearsal converts the settings this instance actually has, which means reading them from it")
|
||||
}
|
||||
|
||||
ctx := context.Background()
|
||||
httpClient := &http.Client{}
|
||||
|
||||
if err := os.MkdirAll(*workDir, 0o750); err != nil {
|
||||
return fmt.Errorf("create work dir %s: %w", *workDir, err)
|
||||
}
|
||||
|
||||
store := checkpoint.NewStore(*stateDir)
|
||||
rs, err := store.Create("", *targetVersion)
|
||||
if err != nil {
|
||||
return fmt.Errorf("create run: %w", err)
|
||||
}
|
||||
fmt.Printf("run id: %s\n\n", rs.RunID)
|
||||
|
||||
runWorkDir := filepath.Join(*workDir, rs.RunID)
|
||||
runStateDir := filepath.Join(*stateDir, rs.RunID)
|
||||
// The rehearsal's two conclusions survive cleanup: the worklist of what
|
||||
// won't carry over, and the plan of what will. Everything else in the
|
||||
// work directory is scratch. Recording an artifact that points into a
|
||||
// directory about to be deleted would leave the checkpoint referring to
|
||||
// files that aren't there.
|
||||
keptWorklist := filepath.Join(runStateDir, "unmigrated.txt")
|
||||
keptPlan := filepath.Join(runStateDir, "export.json")
|
||||
defer func() {
|
||||
if _, statErr := os.Stat(runWorkDir); os.IsNotExist(statErr) {
|
||||
return
|
||||
}
|
||||
if *keepArtifacts {
|
||||
fmt.Printf("\nartifacts kept at %s (--keep-artifacts)\n", runWorkDir)
|
||||
return
|
||||
}
|
||||
if rmErr := os.RemoveAll(runWorkDir); rmErr != nil {
|
||||
fmt.Fprintf(os.Stderr, "\nwarning: failed to clean up %s: %v (remove it manually)\n", runWorkDir, rmErr)
|
||||
return
|
||||
}
|
||||
fmt.Printf("\ncleaned up %s; the run log is at %s\n", runWorkDir, filepath.Join(runStateDir, "state.json"))
|
||||
}()
|
||||
|
||||
fmt.Println("--- preflight ---")
|
||||
checker := preflight.New(preflight.Options{
|
||||
BinaryPath: *binaryPath, ConfigPath: *configPath, DataDir: *dataDir, ContainerName: *containerName,
|
||||
AdminURL: *adminURL, AdminUser: *adminUser, AdminPassword: *adminPassword,
|
||||
TargetVersion: *targetVersion, MinFreeMultiple: *minFree, HTTPClient: httpClient,
|
||||
})
|
||||
pfReport, err := checker.Run(ctx, store, rs)
|
||||
fmt.Print(pfReport.String())
|
||||
if err != nil {
|
||||
return fmt.Errorf("preflight failed to complete: %w", err)
|
||||
}
|
||||
if pfReport.Blocking() {
|
||||
return fmt.Errorf("preflight found blocking issues - see FAIL lines above")
|
||||
}
|
||||
|
||||
p, err := plan.Decide(rs.SourceVersion, rs.TargetVersion)
|
||||
if err != nil {
|
||||
return fmt.Errorf("plan: %w", err)
|
||||
}
|
||||
fmt.Printf("\nplan: %s\n", p.Reason)
|
||||
if !p.CrossesMajorBoundary {
|
||||
fmt.Println("\nthis is a same-boundary patch upgrade: its settings don't need converting, " +
|
||||
"so there is nothing to rehearse. A real run would be a binary swap and restart.")
|
||||
return nil
|
||||
}
|
||||
|
||||
scriptDest := filepath.Join(runWorkDir, "migrate_v016.py")
|
||||
settingsPath := filepath.Join(runWorkDir, "settings.json")
|
||||
principalsPath := filepath.Join(runWorkDir, "principals.json")
|
||||
convertedConfig := filepath.Join(runWorkDir, "config.json")
|
||||
convertedExport := filepath.Join(runWorkDir, "export.json")
|
||||
unmigratedPath := filepath.Join(runWorkDir, "unmigrated.txt")
|
||||
|
||||
fmt.Println("\n--- dump (read-only) ---")
|
||||
if _, err := store.RunStep(rs, checkpoint.PhaseBackup, "settings-dump", func() (checkpoint.StepOutcome, error) {
|
||||
if err := os.MkdirAll(runWorkDir, 0o750); err != nil {
|
||||
return checkpoint.StepOutcome{}, err
|
||||
}
|
||||
sum, err := backup.DownloadFile(ctx, httpClient, backup.DefaultMigrationScriptURL, scriptDest, *migrationScriptSHA256)
|
||||
if err != nil {
|
||||
return checkpoint.StepOutcome{}, err
|
||||
}
|
||||
pinNote := ""
|
||||
if *migrationScriptSHA256 == "" {
|
||||
pinNote = fmt.Sprintf(" (no pin configured - record sha256 %s as --migration-script-sha256 to pin it)", sum)
|
||||
}
|
||||
if err := backup.RunSettingsDump(ctx, backup.SettingsDumpOptions{
|
||||
PythonPath: *pythonPath, ScriptPath: scriptDest, URL: *adminURL,
|
||||
Username: *adminUser, Password: *adminPassword,
|
||||
SettingsPath: settingsPath, PrincipalsPath: principalsPath,
|
||||
}); err != nil {
|
||||
return checkpoint.StepOutcome{}, err
|
||||
}
|
||||
_, settingsSize, err := backup.HashFile(settingsPath)
|
||||
if err != nil {
|
||||
return checkpoint.StepOutcome{}, err
|
||||
}
|
||||
_, principalsSize, err := backup.HashFile(principalsPath)
|
||||
if err != nil {
|
||||
return checkpoint.StepOutcome{}, err
|
||||
}
|
||||
return checkpoint.StepOutcome{
|
||||
Detail: fmt.Sprintf("dumped settings (%d bytes) and principals (%d bytes) from %s%s",
|
||||
settingsSize, principalsSize, *adminURL, pinNote),
|
||||
}, nil
|
||||
}); err != nil {
|
||||
return fmt.Errorf("settings dump: %w", err)
|
||||
}
|
||||
fmt.Println(rs.Outcome(checkpoint.PhaseBackup, "settings-dump").Detail)
|
||||
|
||||
fmt.Println("\n--- convert ---")
|
||||
if _, err := store.RunStep(rs, checkpoint.PhaseStage, "convert-settings", func() (checkpoint.StepOutcome, error) {
|
||||
if err := backup.RunSettingsConvert(ctx, backup.SettingsConvertOptions{
|
||||
PythonPath: *pythonPath, ScriptPath: scriptDest,
|
||||
SettingsPath: settingsPath, PrincipalsPath: principalsPath,
|
||||
ConfigPath: convertedConfig, OutputPath: convertedExport,
|
||||
// migrate_v016.py writes unmigrated.txt into its working
|
||||
// directory; without this it lands wherever the operator
|
||||
// happened to be, or fails the convert if that isn't writable.
|
||||
WorkDir: runWorkDir,
|
||||
}); err != nil {
|
||||
return checkpoint.StepOutcome{}, err
|
||||
}
|
||||
detail := "converted this instance's settings into a v0.16 apply plan"
|
||||
if report, readErr := backup.ReadUnmigratedReport(unmigratedPath); readErr == nil && report != nil {
|
||||
detail += fmt.Sprintf("; %d setting(s) will NOT carry over", report.TotalKeys)
|
||||
}
|
||||
return checkpoint.StepOutcome{Detail: detail}, nil
|
||||
}); err != nil {
|
||||
return fmt.Errorf("convert settings: %w", err)
|
||||
}
|
||||
|
||||
if err := copyFile(convertedExport, keptPlan); err != nil {
|
||||
fmt.Fprintf(os.Stderr, "warning: couldn't preserve the apply plan: %v\n", err)
|
||||
} else if sum, size, hashErr := backup.HashFile(keptPlan); hashErr == nil {
|
||||
rs.RecordArtifact("converted-export", checkpoint.Artifact{Path: keptPlan, SHA256: sum, SizeBytes: size})
|
||||
fmt.Printf("apply plan: %s (%d bytes) - what WILL carry over\n", keptPlan, size)
|
||||
}
|
||||
|
||||
unmigrated, err := backup.ReadUnmigratedReport(unmigratedPath)
|
||||
if err != nil {
|
||||
fmt.Fprintf(os.Stderr, "warning: couldn't read the unmigrated-settings report: %v\n", err)
|
||||
} else if unmigrated != nil && unmigrated.TotalKeys > 0 {
|
||||
if err := copyFile(unmigratedPath, keptWorklist); err != nil {
|
||||
fmt.Fprintf(os.Stderr, "warning: couldn't preserve the worklist: %v\n", err)
|
||||
} else {
|
||||
unmigrated.Path = keptWorklist
|
||||
if sum, size, hashErr := backup.HashFile(keptWorklist); hashErr == nil {
|
||||
rs.RecordArtifact("unmigrated-settings", checkpoint.Artifact{Path: keptWorklist, SHA256: sum, SizeBytes: size})
|
||||
}
|
||||
}
|
||||
fmt.Printf("\n !! %s\n", unmigrated.Summary(10))
|
||||
fmt.Println(" These do not carry over. Rebuild them on the migrated instance before it serves mail -")
|
||||
fmt.Println(" note that server.listener is typically among them, so until you do, it answers on nothing.")
|
||||
}
|
||||
if err := store.Save(rs); err != nil {
|
||||
return fmt.Errorf("save run state: %w", err)
|
||||
}
|
||||
|
||||
fmt.Printf("\nREHEARSAL COMPLETE for run %s. Nothing was modified: no data was copied, no server was started,\n"+
|
||||
"and the store was never written to.\n", rs.RunID)
|
||||
return nil
|
||||
}
|
||||
|
||||
// copyFile duplicates src to dst, used to lift the worklist out of the
|
||||
// scratch directory before it's cleaned up.
|
||||
func copyFile(src, dst string) error {
|
||||
if err := os.MkdirAll(filepath.Dir(dst), 0o750); err != nil {
|
||||
return err
|
||||
}
|
||||
in, err := os.Open(src)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
defer in.Close()
|
||||
out, err := os.OpenFile(dst, os.O_WRONLY|os.O_CREATE|os.O_TRUNC, 0o640)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
defer out.Close()
|
||||
if _, err := io.Copy(out, in); err != nil {
|
||||
return err
|
||||
}
|
||||
return out.Sync()
|
||||
}
|
||||
+36
-255
@@ -4,275 +4,56 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"context"
|
||||
"flag"
|
||||
"fmt"
|
||||
"net/http"
|
||||
"os"
|
||||
"path/filepath"
|
||||
|
||||
"github.com/LINUXexpert-org/stalwart-migrator/internal/backup"
|
||||
"github.com/LINUXexpert-org/stalwart-migrator/internal/checkpoint"
|
||||
"github.com/LINUXexpert-org/stalwart-migrator/internal/plan"
|
||||
"github.com/LINUXexpert-org/stalwart-migrator/internal/preflight"
|
||||
"github.com/LINUXexpert-org/stalwart-migrator/internal/recovery"
|
||||
"github.com/LINUXexpert-org/stalwart-migrator/internal/validate"
|
||||
)
|
||||
|
||||
// runRun implements `stalwart-migrate run`. Only --dry-run is available
|
||||
// today. internal/cutover exists, so the gap is no longer a phase but the
|
||||
// pipeline around it: §4.3 staging, and the wiring that would drive
|
||||
// runRun implements `stalwart-migrate run`, which refuses.
|
||||
//
|
||||
// It refuses because §4.3 staging and the pipeline that would drive
|
||||
// preflight -> backup -> stage -> recovery-mode -> cutover -> validate
|
||||
// against real paths rather than a sandbox. See ARCHITECTURE.md §8. `run`
|
||||
// without --dry-run refuses rather than doing a migration partway.
|
||||
// against real paths don't exist. The phases themselves mostly do:
|
||||
// internal/preflight, internal/backup, internal/recovery and
|
||||
// internal/cutover are all implemented, and preflight, backup, the settings
|
||||
// dump, convert and the recovery-mode store migration have been exercised
|
||||
// against a real Stalwart 0.15.5. Cutover has not - it has never run
|
||||
// outside its own tests, and it is the phase that mutates production.
|
||||
//
|
||||
// --dry-run runs preflight and a real backup (see the caveat printed below
|
||||
// about why backup still touches the live data directory), then - if the
|
||||
// plan crosses the 0.15/0.16 boundary - clones the verified backup into a
|
||||
// disposable sandbox, converts the settings snapshot to point at that
|
||||
// sandbox (via migrate_v016.py's own documented --patch-paths mechanism,
|
||||
// not by this tool guessing at config.json's schema), runs the real
|
||||
// recovery-mode migration against the sandbox, and boots the result
|
||||
// normally to confirm it comes up. Nothing at the real binary path or the
|
||||
// real service is ever touched.
|
||||
//
|
||||
// Every byte a dry run writes - the fs-backup copy, the settings/principals
|
||||
// dumps, the downloaded migrate_v016.py, the sandbox clone and its
|
||||
// config/export files - lives under one per-run directory
|
||||
// (work-dir/<run-id>) that a deferred cleanup at the bottom of this
|
||||
// function removes on every exit path: success, a failed check partway
|
||||
// through, or an early refusal. The only thing left behind afterward is the
|
||||
// checkpoint's state.json under --state-dir, which is exactly the
|
||||
// success/failure log a rerun's `status <run-id>` reads - not bulk data.
|
||||
// --keep-artifacts opts out, for when a failure needs inspecting.
|
||||
func runRun(args []string) (err error) {
|
||||
// What used to live here was `--dry-run`, which cloned the store into a
|
||||
// sandbox and migrated the copy. That is now `stalwart-migrate rehearse`,
|
||||
// minus the cloning: see ARCHITECTURE.md §4.9 for why the expensive half
|
||||
// was dropped rather than fixed.
|
||||
func runRun(args []string) error {
|
||||
fs := flag.NewFlagSet("run", flag.ExitOnError)
|
||||
binaryPath := fs.String("binary", "/usr/local/bin/stalwart", "path to the currently-installed stalwart binary")
|
||||
targetBinaryPath := fs.String("target-binary", "", "path to an already-downloaded target-version stalwart binary (required to simulate a major-boundary migration)")
|
||||
configPath := fs.String("config", "/etc/stalwart/config.toml", "path to stalwart's current config file")
|
||||
dataDir := fs.String("data-dir", "/var/lib/stalwart", "stalwart data directory")
|
||||
containerName := fs.String("container", "stalwart", "docker container name, if applicable")
|
||||
adminURL := fs.String("admin-url", "", "base URL for the live instance's admin/JMAP API (required)")
|
||||
adminUser := fs.String("admin-user", "", "admin username")
|
||||
adminPassword := fs.String("admin-password", os.Getenv("STALWART_MIGRATE_ADMIN_PASSWORD"),
|
||||
"admin password (or set STALWART_MIGRATE_ADMIN_PASSWORD)")
|
||||
targetVersion := fs.String("target", "latest", `target Stalwart version, or "latest"`)
|
||||
stateDir := fs.String("state-dir", checkpoint.DefaultBaseDir, "directory to store run checkpoints in")
|
||||
workDir := fs.String("work-dir", "/var/lib/stalwart-migrator/work", "scratch directory for backups, dumps, and the dry-run sandbox (cleaned up afterward - see --keep-artifacts)")
|
||||
stalwartCLI := fs.String("stalwart-cli", "stalwart-cli", "path to the stalwart-cli binary")
|
||||
pythonPath := fs.String("python", "python3", "path to python3")
|
||||
migrationScriptSHA256 := fs.String("migration-script-sha256", "", "pinned sha256 of migrate_v016.py (recommended; see preflight/backup output for the hash to pin after a first unpinned run)")
|
||||
recoveryPort := fs.Int("recovery-port", 8080, "port recovery mode's HTTP listener binds, per UPGRADING/v0_16.md's own examples")
|
||||
minFree := fs.Float64("min-free-multiple", 2.0, "required free disk space as a multiple of the data directory size")
|
||||
dryRun := fs.Bool("dry-run", false, "simulate and validate the migration against a disposable sandbox, without touching production")
|
||||
keepArtifacts := fs.Bool("keep-artifacts", false, "don't delete work-dir/<run-id> afterward (the fs-backup copy, dumps, and sandbox) - useful for inspecting a failure")
|
||||
fs.String("binary", "/usr/local/bin/stalwart", "path to the currently-installed stalwart binary")
|
||||
fs.String("config", "/etc/stalwart/config.toml", "path to stalwart's current config file")
|
||||
fs.String("data-dir", "/var/lib/stalwart", "stalwart data directory")
|
||||
fs.String("target", "latest", `target Stalwart version, or "latest"`)
|
||||
fs.String("state-dir", checkpoint.DefaultBaseDir, "directory to store run checkpoints in")
|
||||
dryRun := fs.Bool("dry-run", false, "removed - see `stalwart-migrate rehearse`")
|
||||
if err := fs.Parse(args); err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
if !*dryRun {
|
||||
return fmt.Errorf(
|
||||
"real (non-dry-run) migrations aren't available yet: cutover is implemented (ARCHITECTURE.md §4.5), but nothing wires it " +
|
||||
"into a production run - the staging phase (§4.3) and the real pipeline don't exist yet, so this command has no path " +
|
||||
"that touches production. Run with --dry-run to validate the migration mechanics against a disposable sandbox copy " +
|
||||
"of your data. Note that when a real run does land, recovery from a failed migration will be your own snapshot or " +
|
||||
"backup - this tool does not undo a migration (§4.8)",
|
||||
)
|
||||
}
|
||||
if *adminURL == "" {
|
||||
return fmt.Errorf("--admin-url is required")
|
||||
if *dryRun {
|
||||
return fmt.Errorf("--dry-run has been replaced by `stalwart-migrate rehearse`, which converts this " +
|
||||
"instance's settings and reports what will and won't carry over. It no longer clones the data " +
|
||||
"directory: that only proved the store opens, and cost a full copy of it to find out " +
|
||||
"(ARCHITECTURE.md §4.9)")
|
||||
}
|
||||
|
||||
ctx := context.Background()
|
||||
httpClient := &http.Client{}
|
||||
|
||||
if err := os.MkdirAll(*workDir, 0o750); err != nil {
|
||||
return fmt.Errorf("create work dir %s: %w", *workDir, err)
|
||||
}
|
||||
|
||||
store := checkpoint.NewStore(*stateDir)
|
||||
rs, err := store.Create("", *targetVersion)
|
||||
if err != nil {
|
||||
return fmt.Errorf("create run: %w", err)
|
||||
}
|
||||
fmt.Printf("run id: %s\n\n", rs.RunID)
|
||||
|
||||
runWorkDir := filepath.Join(*workDir, rs.RunID)
|
||||
logPath := filepath.Join(*stateDir, rs.RunID, "state.json")
|
||||
defer func() {
|
||||
if _, statErr := os.Stat(runWorkDir); os.IsNotExist(statErr) {
|
||||
return // nothing was ever written (e.g. refused before backup ran)
|
||||
}
|
||||
if *keepArtifacts {
|
||||
fmt.Printf("\nartifacts kept at %s (--keep-artifacts) - remove manually when done inspecting\n", runWorkDir)
|
||||
return
|
||||
}
|
||||
outcome := "succeeded"
|
||||
if err != nil {
|
||||
outcome = "failed"
|
||||
}
|
||||
if rmErr := os.RemoveAll(runWorkDir); rmErr != nil {
|
||||
fmt.Fprintf(os.Stderr, "\nwarning: dry run %s, but failed to clean up %s: %v (remove it manually)\n", outcome, runWorkDir, rmErr)
|
||||
return
|
||||
}
|
||||
fmt.Printf("\ndry run %s - cleaned up %s; the run log is at %s\n", outcome, runWorkDir, logPath)
|
||||
}()
|
||||
|
||||
fmt.Println("--- preflight ---")
|
||||
checker := preflight.New(preflight.Options{
|
||||
BinaryPath: *binaryPath, ConfigPath: *configPath, DataDir: *dataDir, ContainerName: *containerName,
|
||||
AdminURL: *adminURL, AdminUser: *adminUser, AdminPassword: *adminPassword,
|
||||
TargetVersion: *targetVersion, MinFreeMultiple: *minFree, HTTPClient: httpClient,
|
||||
})
|
||||
pfReport, err := checker.Run(ctx, store, rs)
|
||||
fmt.Print(pfReport.String())
|
||||
if err != nil {
|
||||
return fmt.Errorf("preflight failed to complete: %w", err)
|
||||
}
|
||||
if pfReport.Blocking() {
|
||||
return fmt.Errorf("preflight found blocking issues - see FAIL lines above")
|
||||
}
|
||||
|
||||
p, err := plan.Decide(rs.SourceVersion, rs.TargetVersion)
|
||||
if err != nil {
|
||||
return fmt.Errorf("plan: %w", err)
|
||||
}
|
||||
fmt.Printf("\nplan: %s\n", p.Reason)
|
||||
|
||||
fmt.Println("\n--- backup ---")
|
||||
fmt.Println("(dry-run does not stop the live Stalwart service itself - internal/service can now do that, but dry-run " +
|
||||
"isn't wired to offer it. For a guaranteed-consistent snapshot, stop stalwart before running this; otherwise the " +
|
||||
"filesystem copy may reflect a live, in-use store. This is unrelated to whether production gets touched - it never does.)")
|
||||
|
||||
backupDir := filepath.Join(runWorkDir, "backup")
|
||||
scriptDest := filepath.Join(runWorkDir, "migrate_v016.py")
|
||||
settingsPath := filepath.Join(runWorkDir, "settings.json")
|
||||
principalsPath := filepath.Join(runWorkDir, "principals.json")
|
||||
|
||||
backupOpts := backup.Options{
|
||||
BinaryPath: *binaryPath,
|
||||
SkipBinaryPreservation: true, // dry-run: never touch the production binary
|
||||
DataDir: *dataDir,
|
||||
BackupDir: backupDir,
|
||||
MigrationScriptSHA256: *migrationScriptSHA256,
|
||||
ScriptDestPath: scriptDest,
|
||||
AdminURL: *adminURL,
|
||||
AdminUser: *adminUser,
|
||||
AdminPassword: *adminPassword,
|
||||
SettingsDumpPath: settingsPath,
|
||||
PrincipalsDumpPath: principalsPath,
|
||||
PythonPath: *pythonPath,
|
||||
HTTPClient: httpClient,
|
||||
}
|
||||
bkReport, err := backup.Run(ctx, store, rs, backupOpts)
|
||||
fmt.Print(bkReport.String())
|
||||
if err != nil {
|
||||
return fmt.Errorf("backup failed: %w", err)
|
||||
}
|
||||
|
||||
if !p.CrossesMajorBoundary {
|
||||
fmt.Println("\nthis is a same-boundary patch upgrade: there's no recovery-mode phase to simulate. " +
|
||||
"preflight and backup above are as far as a dry-run goes for this path - a real run would be a binary swap and restart.")
|
||||
return nil
|
||||
}
|
||||
|
||||
if *targetBinaryPath == "" {
|
||||
return fmt.Errorf("--target-binary is required to simulate a major-boundary migration (0.15 -> 0.16 crosses one here)")
|
||||
}
|
||||
|
||||
fmt.Println("\n--- convert (settings -> sandbox config) ---")
|
||||
sandboxDataDir := filepath.Join(runWorkDir, "sandbox-data")
|
||||
sandboxConfigPath := filepath.Join(runWorkDir, "sandbox-config.json")
|
||||
sandboxExportPath := filepath.Join(runWorkDir, "sandbox-export.json")
|
||||
|
||||
if _, err := store.RunStep(rs, checkpoint.PhaseStage, "clone-sandbox-data", func() (checkpoint.StepOutcome, error) {
|
||||
manifest, err := backup.CopyDataDir(backupDir, sandboxDataDir)
|
||||
if err != nil {
|
||||
return checkpoint.StepOutcome{}, err
|
||||
}
|
||||
return checkpoint.StepOutcome{Detail: fmt.Sprintf("cloned the verified backup (%d files) into the sandbox at %s", len(manifest.Files), sandboxDataDir)}, nil
|
||||
}); err != nil {
|
||||
return fmt.Errorf("clone sandbox data: %w", err)
|
||||
}
|
||||
fmt.Printf("cloned verified backup into sandbox: %s\n", sandboxDataDir)
|
||||
|
||||
unmigratedPath := filepath.Join(runWorkDir, "unmigrated.txt")
|
||||
if _, err := store.RunStep(rs, checkpoint.PhaseStage, "convert-settings", func() (checkpoint.StepOutcome, error) {
|
||||
if err := backup.RunSettingsConvert(ctx, backup.SettingsConvertOptions{
|
||||
PythonPath: *pythonPath, ScriptPath: scriptDest,
|
||||
SettingsPath: settingsPath, PrincipalsPath: principalsPath,
|
||||
ConfigPath: sandboxConfigPath, OutputPath: sandboxExportPath,
|
||||
PatchPaths: map[string]string{*dataDir: sandboxDataDir},
|
||||
// Without this the script writes unmigrated.txt into whatever
|
||||
// directory this command was launched from - or fails outright
|
||||
// if that isn't writable.
|
||||
WorkDir: runWorkDir,
|
||||
}); err != nil {
|
||||
return checkpoint.StepOutcome{}, err
|
||||
}
|
||||
detail := fmt.Sprintf("generated %s and %s, patched to point at the sandbox", sandboxConfigPath, sandboxExportPath)
|
||||
if report, err := backup.ReadUnmigratedReport(unmigratedPath); err == nil && report != nil && report.TotalKeys > 0 {
|
||||
detail += fmt.Sprintf("; %d setting(s) were NOT migrated", report.TotalKeys)
|
||||
}
|
||||
return checkpoint.StepOutcome{Detail: detail, Extra: unmigratedPath}, nil
|
||||
}); err != nil {
|
||||
return fmt.Errorf("convert settings: %w", err)
|
||||
}
|
||||
fmt.Println("generated sandbox config.json and export.json")
|
||||
|
||||
// What the converter could NOT carry over matters more than what it
|
||||
// could: against a real instance this is the overwhelming majority of
|
||||
// the configuration, including the listeners, and an operator who
|
||||
// doesn't read it will bring up a server that answers on no ports.
|
||||
unmigrated, err := backup.ReadUnmigratedReport(unmigratedPath)
|
||||
if err != nil {
|
||||
fmt.Fprintf(os.Stderr, "warning: couldn't read the unmigrated-settings report: %v\n", err)
|
||||
} else if unmigrated != nil && unmigrated.TotalKeys > 0 {
|
||||
if sum, size, hashErr := backup.HashFile(unmigratedPath); hashErr == nil {
|
||||
rs.RecordArtifact("unmigrated-settings", checkpoint.Artifact{Path: unmigratedPath, SHA256: sum, SizeBytes: size})
|
||||
if saveErr := store.Save(rs); saveErr != nil {
|
||||
fmt.Fprintf(os.Stderr, "warning: couldn't record the unmigrated-settings artifact: %v\n", saveErr)
|
||||
}
|
||||
}
|
||||
fmt.Printf("\n !! %s\n", unmigrated.Summary(10))
|
||||
fmt.Println(" These do not carry over. Recreate them on the migrated instance before it serves mail.")
|
||||
}
|
||||
|
||||
fmt.Println("\n--- recovery-mode migration (against the sandbox) ---")
|
||||
listenURL := fmt.Sprintf("http://127.0.0.1:%d/", *recoveryPort)
|
||||
recReport, err := recovery.Run(ctx, store, rs, recovery.Options{
|
||||
BinaryPath: *targetBinaryPath, ConfigPath: sandboxConfigPath, ListenURL: listenURL,
|
||||
AdminUser: "admin", ApplyFiles: []string{sandboxExportPath}, CLIBinaryPath: *stalwartCLI,
|
||||
HTTPClient: httpClient,
|
||||
})
|
||||
fmt.Print(recReport.String())
|
||||
if err != nil {
|
||||
return fmt.Errorf("recovery-mode migration against the sandbox failed: %w", err)
|
||||
}
|
||||
|
||||
fmt.Println("\n--- boot check (normal boot of the migrated sandbox) ---")
|
||||
if rs.PreflightSnapshot != nil {
|
||||
fmt.Println("(comparing against the pre-migration account/mailbox snapshot preflight captured - " +
|
||||
"this is the actual no-data-loss check, not just a reachability probe)")
|
||||
} else {
|
||||
fmt.Println("(no pre-migration snapshot to compare against - preflight couldn't capture one, most likely " +
|
||||
"because --admin-url wasn't set; only reachability is checked)")
|
||||
}
|
||||
valReport, err := validate.Run(ctx, store, rs, validate.BootCheckOptions{
|
||||
BinaryPath: *targetBinaryPath, ConfigPath: sandboxConfigPath, ListenURL: listenURL, HTTPClient: httpClient,
|
||||
ContentIntegrityBefore: rs.PreflightSnapshot,
|
||||
AdminUser: *adminUser,
|
||||
AdminPassword: *adminPassword,
|
||||
})
|
||||
fmt.Print(valReport.String())
|
||||
if err != nil {
|
||||
return fmt.Errorf("post-migration validation failed: %w", err)
|
||||
}
|
||||
|
||||
verified := "the migration mechanics succeeded"
|
||||
if rs.PreflightSnapshot != nil {
|
||||
verified = "the migration mechanics succeeded AND every account/mailbox message count matched before vs. after"
|
||||
}
|
||||
fmt.Printf("\nDRY RUN COMPLETE for run %s: %s, against a disposable sandbox copy of your data. Nothing in production was touched.\n", rs.RunID, verified)
|
||||
return nil
|
||||
fmt.Fprintln(os.Stderr,
|
||||
"real migrations aren't available yet: the staging phase (ARCHITECTURE.md §4.3) and the pipeline that\n"+
|
||||
"would drive preflight -> backup -> stage -> recovery-mode -> cutover -> validate don't exist, so this\n"+
|
||||
"command has no path that touches production.\n\n"+
|
||||
"Two things worth knowing while you wait:\n"+
|
||||
" * `stalwart-migrate rehearse` converts your settings and reports what will NOT carry over. Measured\n"+
|
||||
" against a production instance that was 98% of them, listeners included - so it decides your\n"+
|
||||
" migration plan, and it's safe to run now.\n"+
|
||||
" * Recovery from a failed migration is your own snapshot or backup. This tool does not undo a\n"+
|
||||
" migration (§4.8).")
|
||||
return fmt.Errorf("`run` is not implemented")
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user