Preflight refuses a container, and until now that was all it said. The two
things that actually decide whether one could be migrated at all were never
looked at, so an operator was refused without being told what to fix or what
a manual migration would involve.
It now inspects the container and reports three things. What image it is
running, tag and digest kept apart because a moved tag makes them disagree
and only one of them says what is really there. Whether compose manages it,
which matters beyond this tool's current refusal: recreating a
compose-managed container out from under compose leaves the container and
the compose file disagreeing about what is deployed, and the next
`compose up` reverts the migration. And whether the data is on a volume at
all -- a container keeping it in its own writable layer loses it when the
container is replaced, and replacing the container is what migrating it
means, so that one is fatal in a way no later phase could recover from.
All three are advisory under rehearse, on the same reasoning that already
made the deployment check advisory there: rehearse never stops or recreates
anything, and an operator doing this by hand needs these facts more than an
automated run does.
Preflight stays read-only. Preserving the container definition as an
artifact belongs with the phase that replaces it, so it goes with cutover
rather than here.
One limitation this surfaced and does not fix: --data-dir naming a path
inside the container breaks the host-side disk-space check, which stats it
locally. Path translation is the next piece of work; the data-volume check
says so when the path matches no mount.
Cutover already refused a container -- recreating one from a new image is
not swapping a binary and rewriting a unit, and this tool does not automate
it -- but it refused from cutover.Run, which run.go calls at line 324. The
service is stopped at line 243. So the sequence on a container was: stop
Stalwart, convert the settings, then discover the deployment cannot be cut
over, return the error, and exit with mail still down. Reported as #1.
The deployment kind is known in preflight, before anything has been touched,
and that is now where it is acted on: docker is a blocking check. rehearse
keeps working -- it never stops the service or cuts over, and telling an
operator what the migration involves is most useful precisely when the tool
cannot do it for them -- so it sets DeploymentCheckAdvisory, alongside the
ToolCheckAdvisory it already set for the same reason.
The second half is not docker's alone. Every return between the stop and the
end of cutover returned with the service down; a failed settings conversion
would have done the same to a systemd host. run now registers a restart on
the way out, after the stop rather than before, so it only ever starts
something this tool stopped. It does not claim to have recovered the
migration -- a part-migrated store still needs --resume or the operator's
recovery point -- it removes the narrower failure of exiting on a
foreseeable error while the server it stopped stays stopped.
A dress rehearsal on a clone of production migrated cleanly and then failed
all three post-cutover steps with 401: the health check, the quota rebuild,
and the content comparison. The account was `admin` - a v0.15
[authentication.fallback-admin], not a directory account. v0.16 keeps its
configuration in the store, so the block defining it does not survive, and
the credential stops working the instant the migration completes.
Every check passed beforehand, because on v0.15 that account authenticates
perfectly well. The README has said to use a named account for a while; the
tool now says it too, while nothing has been touched and changing it costs
one flag.
The migration itself is unaffected - it succeeds either way. What is lost is
the ability to verify it afterwards, which is the part worth having.
preflight asked the GitHub release API which version it was upgrading to,
so a host with no internet failed there even with the target binary already
on disk - and `run` never passed its own --target-binary down to preflight,
so supplying one did not help. Read the version from the binary instead
when one is given, and fail if it is not the version the run targets.
Found by the first preflight against the production clone, which is
deliberately cut off from the network.
A live migration on 2026-08-24 stopped a production mail server and then
discovered the host's stalwart-cli was 0.13.4 - present, but from when the
CLI shipped with the server, with no `apply` command. The migration needs
v1.0.2+ from the separately-versioned stalwartlabs/cli repository.
Recovery was closed in both directions. v0.16's recovery-mode boot had
already bumped the store schema to v6, so the 0.15.5 binary refused to
reopen it ("expected 5 or below, found 6"). Going forward needed
export.json, which this tool's own failure path had deleted - and
regenerating it required a settings dump from a live v0.15 instance that
could no longer start. The operator restored a day-old snapshot and lost a
day of mail across nine domains.
Three fixes:
1. preflight.CheckExternalTools verifies stalwart-cli exists and is v1.0.2
or later, and that python3 runs - before anything is touched. Every fact
needed to prevent this was available in under a second from a stopped
state. Skipped for a patch upgrade, which invokes neither tool.
2. A failed run no longer deletes its work directory. Cleaning up on every
exit path was right for a sandboxed rehearsal and catastrophic here:
once the service is stopped the settings dump cannot be regenerated, so
deleting it removes the only way forward. The failure now prints the
resume command instead.
3. `run --resume <id>` continues an interrupted run. The checkpoint
machinery existed but never engaged, because run created a new run every
invocation - so a retry re-ran preflight against a binary already moved
aside, and failed. Completed steps are skipped from the checkpoint.
Proven against a VM built to match the failure: stalwart-cli 0.15.5,
accounts and mail seeded.
* preflight refused, service still active, mail still accepted
* a stub CLI passing --version and failing apply left the run stopped
with all eight inputs intact and the resume command printed
* --resume carried it to a clean finish: five seconds of downtime,
listeners regenerated, admin role restored, quotas rebuilt
That failure-path test is the one that should have run before production.
Every earlier test had stalwart-cli installed from the start, and the one
failure I did exercise happened to leave its artifacts behind.
x:Account.domainId is an internal id on v0.16 ("b"), not a domain name. A
pre-migration snapshot taken from a v0.15 instance records names
("smoke.test"), so the post-migration directory comparison compared ids
against names and would have reported every domain as having vanished -
a false alarm on the check whose whole job is proving nothing was lost.
The client now resolves them with x:Domain/query + x:Domain/get in a single
request, using a JMAP back-reference (RFC 8620 3.7). Confirmed against a
live 0.16.14 before being written:
["x:Domain/get", {"list":[{"name":"smoke.test","id":"b"}]}, "g"]
An id that can't be resolved is kept as-is - a domain that can't be named is
still a domain that exists - but a failure of the resolution call itself is
an error rather than a silent fallback, since quietly comparing ids against
names is precisely the bug being fixed.
Verified against the live migrated instance: the snapshot that reported
domains=[b] now reports domains=[smoke.test], matching what the
pre-migration snapshot recorded.
Ran a complete 0.15.5 -> 0.16.14 migration of the smoke VM, driving the
phases in the order the real pipeline will. It worked - all mail intact and
readable afterwards, all ten listeners up, cutover executed for the first
time ever and checkpoint resume exercised - and it exposed three defects.
1. The converted config was installed root-owned while the service runs as
its own user. Stalwart crash-looped 28 times on "Failed to read data
store settings: Permission denied", minutes after the mistake and
nowhere near it. This is the same ownership trap that retired the
rollback implementation, in a new place: writing files as root is the
natural thing for a tool running as root to do, and it is wrong every
time the service is not root.
Cutover now installs the config itself, copying ownership and mode from
the config being replaced.
2. v0.16.14 does not serve /api - the endpoint stalwartapi assumed.
Confirmed against a fully migrated, fully configured, serving instance
rather than a sandbox: /api, /api/principal and /jmap/ all 404. The JMAP
endpoint is the one the session document advertises, which is what RFC
8620 discovery is for.
The client now discovers it, re-basing the advertised path onto the
operator's host: a real instance advertises its canonical public URL
("https://mail.smoke.test/jmap/") which frequently isn't reachable from
where this tool runs. The session is authoritative about the path; the
operator is authoritative about the host.
3. Dispatching on the urn:stalwart:jmap capability was wrong, because
NEITHER version advertises it - not 0.15.5, and not a fully migrated
0.16.14. That sent 0.16 instances down the 0.15 REST path where every
call 404s. The client probes what the instance actually serves instead.
Less elegant than a declared capability, with the advantage of being
true.
Also: a JMAP "forbidden" now explains itself. An account holding the admin
role before the migration was refused x:Account/query afterwards, and a
bare "forbidden" gives an operator nowhere to start. Whether the role
failed to carry or v0.16 wants different permissions was not isolated, and
that question is recorded as open - it gates quota recalculation and any
post-migration validation.
Verified against both live instances: the 0.15.5 reports 3 accounts and its
domain over REST, and the migrated 0.16.14 routes to JMAP, finds the right
endpoint, and returns the explained refusal.
Found by running preflight against a real Stalwart 0.15.5 in a VM. Two
defects, the second worse than the first.
1. AccountSnapshot could not read the version this tool migrates FROM.
0.15.5 advertises no urn:stalwart:jmap capability and POST /api returns
404 - the JMAP management API and x:Account are 0.16 features. 0.15.x
exposes a REST API at GET /api/principal instead. So preflight's
account-snapshot check warned and moved on, and every run against a real
source instance had no "before" data at all.
AccountSnapshot now dispatches on the capability the session document
advertises - a positive signal, not an inference from a failed call -
and internal/stalwartapi/principal.go implements the 0.15.x REST path,
including its 1-based page/limit pagination so an install larger than
one page isn't silently truncated.
2. With no "before" counts, the content-integrity comparison iterated an
empty map, checked nothing, and reported "all message counts match".
That is the strongest claim this tool makes - ARCHITECTURE 4.7 calls it
the actual no-data-loss guarantee - made vacuously, and it would have
passed on a migration that lost every message.
The comparison now derives its account set from whatever the source
could report, verifies every account and domain survived either way, and
carries MessageCountsCompared so the report says plainly "MESSAGE COUNTS
NOT COMPARED ... no-data-loss is NOT verified here" rather than implying
otherwise.
What can and cannot be checked across the 0.15/0.16 boundary, now that a
real server has answered: 0.15.x has no per-mailbox message count at any
endpoint, and the impersonation login 0.16 offers returns 401 there, so
before/after message counts are impossible for the boundary migration this
tool exists for. Both versions do report per-account used quota (usedQuota
in 0.15's REST list, usedDiskQuota on 0.16's x:Account), so that is
captured on both sides. It is recorded and reported, not asserted on:
4.5 notes the 0.16 migration resets quotas to zero pending recalculation,
so comparing those bytes across the boundary would be a false alarm
generator.
Test servers across preflight, validate and stalwartapi now advertise
urn:stalwart:jmap, since they stand in for 0.16 instances and that
capability is what says so.
Verified end to end against the smoke VM: all nine preflight checks pass,
and the checkpoint records 2 accounts, 1 domain and per-account used quota
where it previously recorded nothing.
GPLv3's "How to Apply These Terms" asks for a notice in each source file;
this is the modern two-line SPDX form of it rather than the full paragraph.
82 files, including tests.
The blank line after the header is load-bearing. In Go a comment block
immediately preceding `package X` becomes the package doc comment, so
without the separator the SPDX lines would be absorbed into the doc for the
eleven packages whose doc.go (or main.go) opens with one, and `go doc` would
print them. Verified it doesn't.
Rename module github.com/johnellis/stalwart-migrator ->
github.com/LINUXexpert-org/stalwart-migrator to match the repository
location, so the module resolves under `go get`.
The initial commit's .gitignore listed the compiled binary as a bare
`stalwart-migrate` pattern, which Git matches at any depth -- so it also
excluded the cmd/stalwart-migrate/ source directory, and the initial
commit shipped without the CLI entrypoint. Anchor the pattern to the
repo root as /stalwart-migrate and add the four missing files.
go build, go vet, and go test ./... all pass.
In-place upgrade tool for Stalwart Mail Server (0.15.5 -> latest) with
checkpointed rollback and post-migration validation. Design stage; see
ARCHITECTURE.md.
jcoffey-dev