Everything the container migration needs has landed a piece at a time and
nothing called any of it. `run` now does: stage pulls and verifies an image
instead of downloading a binary, the recovery cycle launches a throwaway
container against the live container's own mounts, and cutover recreates it.
Preflight's blanket refusal of docker goes with it -- what still refuses is
specific to a container rather than to containers, which is compose and
data that is not on a volume.
It refuses without --container-path-unproven, and that flag is the honest
part of this change. Every test drives a fake docker. That proves the right
commands are assembled and proves nothing about whether a real image reads
the config it is handed -- which is the exact limit ARCHITECTURE.md section
4.8 records about the rollback code that was deleted for being tested only
against fakes. A doc note seemed too quiet for a tool that stops a mail
server, so it is a flag nobody reaches without being told.
The converted config reaches the container through the data volume. It is
written under the host side of whichever mount covers --data-dir and named
on the container side, because cutover recreates a container with the mounts
it had and cannot invent a new one for a config file. --data-dir therefore
names the path inside the container, which preflight already says when it
matches no mount.
PatchPaths stays unused, deliberately. Its documented purpose is pointing a
rehearsal at a sandbox; a real container's dumped settings already carry
container-side paths, because they come from the live server rather than
from a file on this host.
The preflight test that asserted docker was refused outright now asserts
the replacement rather than being deleted -- "docker is allowed through
here" is the thing that would be wrong to regress. Its fixture had to make
--data-dir both a real host directory and one the fake container mounts,
since disk-space stats it and container-data-volume wants it covered.
README gains the container section and, at the top, the note that this is
ihasmail's companion.
Preflight refuses a container, and until now that was all it said. The two
things that actually decide whether one could be migrated at all were never
looked at, so an operator was refused without being told what to fix or what
a manual migration would involve.
It now inspects the container and reports three things. What image it is
running, tag and digest kept apart because a moved tag makes them disagree
and only one of them says what is really there. Whether compose manages it,
which matters beyond this tool's current refusal: recreating a
compose-managed container out from under compose leaves the container and
the compose file disagreeing about what is deployed, and the next
`compose up` reverts the migration. And whether the data is on a volume at
all -- a container keeping it in its own writable layer loses it when the
container is replaced, and replacing the container is what migrating it
means, so that one is fatal in a way no later phase could recover from.
All three are advisory under rehearse, on the same reasoning that already
made the deployment check advisory there: rehearse never stops or recreates
anything, and an operator doing this by hand needs these facts more than an
automated run does.
Preflight stays read-only. Preserving the container definition as an
artifact belongs with the phase that replaces it, so it goes with cutover
rather than here.
One limitation this surfaced and does not fix: --data-dir naming a path
inside the container breaks the host-side disk-space check, which stats it
locally. Path translation is the next piece of work; the data-volume check
says so when the path matches no mount.
Cutover already refused a container -- recreating one from a new image is
not swapping a binary and rewriting a unit, and this tool does not automate
it -- but it refused from cutover.Run, which run.go calls at line 324. The
service is stopped at line 243. So the sequence on a container was: stop
Stalwart, convert the settings, then discover the deployment cannot be cut
over, return the error, and exit with mail still down. Reported as #1.
The deployment kind is known in preflight, before anything has been touched,
and that is now where it is acted on: docker is a blocking check. rehearse
keeps working -- it never stops the service or cuts over, and telling an
operator what the migration involves is most useful precisely when the tool
cannot do it for them -- so it sets DeploymentCheckAdvisory, alongside the
ToolCheckAdvisory it already set for the same reason.
The second half is not docker's alone. Every return between the stop and the
end of cutover returned with the service down; a failed settings conversion
would have done the same to a systemd host. run now registers a restart on
the way out, after the stop rather than before, so it only ever starts
something this tool stopped. It does not claim to have recovered the
migration -- a part-migrated store still needs --resume or the operator's
recovery point -- it removes the narrower failure of exiting on a
foreseeable error while the server it stopped stays stopped.
A dress rehearsal on a clone of production migrated cleanly and then failed
all three post-cutover steps with 401: the health check, the quota rebuild,
and the content comparison. The account was `admin` - a v0.15
[authentication.fallback-admin], not a directory account. v0.16 keeps its
configuration in the store, so the block defining it does not survive, and
the credential stops working the instant the migration completes.
Every check passed beforehand, because on v0.15 that account authenticates
perfectly well. The README has said to use a named account for a while; the
tool now says it too, while nothing has been touched and changing it costs
one flag.
The migration itself is unaffected - it succeeds either way. What is lost is
the ability to verify it afterwards, which is the part worth having.
preflight asked the GitHub release API which version it was upgrading to,
so a host with no internet failed there even with the target binary already
on disk - and `run` never passed its own --target-binary down to preflight,
so supplying one did not help. Read the version from the binary instead
when one is given, and fail if it is not the version the run targets.
Found by the first preflight against the production clone, which is
deliberately cut off from the network.
The second production attempt failed during recovery-mode apply, with the
mail server already stopped and the store already at schema v6:
create Account restore-13: invalidForeignKey | Object id: Domain#d
v0.16 requires a tenant-scoped Account to sit on a Domain owned by that
same tenant, for its primary domain and for every alias. v0.15 imposed no
such rule, and migrate_v016.py carries the two facts over independently:
_build_domains sets a domain's memberTenantId only for domains declared as
their own `domain` principal with a `tenant`, while _build_user sets the
account's from the account's own record. A domain that exists only inside
an email address is inferred, gets no tenant, and every tenant-scoped
account using it is then rejected.
Established by reproduction rather than inference: a synthetic v0.15
principal dump, run through the unpatched upstream converter and applied to
a real 0.16.14 in recovery mode, reproduces the error character for
character - the `#d` is the server's own object id for the offending
domain, not a plan client-id. The same harness establishes which directions
are constrained: a tenant-scoped account on a tenant-less domain or on
another tenant's domain is rejected; a global account on a tenant-owned
domain is accepted.
- applyplan.ReconcileDomainTenants repairs the plan between convert and
apply. Where a tenant-less domain is used only by accounts of one
tenant, the domain adopts that tenant - the sole assignment that both
applies and keeps every account. Where accounts genuinely disagree it
changes nothing and reports why, because forcing such a plan through
would mean dropping mailboxes.
- stalwartapi.FetchTenantLayout maps tenant membership over the 0.15 REST
API and predicts the outcome with the same rule the server enforces, so
preflight either warns about the domains that will adopt a tenant or
fails - while the service is still running.
- The plan is parsed generically rather than through the typed Operation.
A real export.json mixes shapes: `create` maps a client-id to an object,
`update` carries a flat one. The typed form failed on the first `update`
line, found by running against actual converter output. Numbers decode
as json.Number so a 10 GiB quota is not rewritten as 1.073741824e+10.
Corrects the record: the previous commit claimed the converter emits every
Account with `tenantId: null` and made preflight refuse every multi-tenant
install on that basis. The field is memberTenantId, the converter does
populate it, and the export had been inspected for a key no version of the
script ever writes. The refusal is now narrowed to what v0.16 genuinely
cannot represent.
The same fix has been prepared for migrate_v016.py upstream. The tool
downloads that script rather than vendoring it, so the repair stays here
until a released version carries it, and is a no-op on a consistent plan.
A second live attempt failed in the same shape as the first: preflight
clean, settings dumped and converted, then a failure during recovery-mode
migration with the mail server already stopped.
created Tenant (1)
created Domain (9)
create Account restore-13: invalidForeignKey | Object id: Domain#d
migrate_v016.py carries the Tenant and the Domains but emits every Account
with a null tenantId, so the account references a tenant-owned domain while
belonging to no tenant and the foreign key is rejected. That is Stalwart's
converter and there is nothing this tool can do about it: a multi-tenant
install has to be migrated by hand until the converter handles tenants.
What this tool got wrong was the timing. Tenant principals are one API call
away and were readable the entire time the server was running. Preflight now
queries them and fails before anything is touched, with an explanation of
exactly what would otherwise fail and when.
This is the same lesson as the stalwart-cli check: knowable in advance,
discovered after a production mail server had been stopped, twice. Any
dependency of the conversion belongs in preflight, not in the phase that
consumes it.
Also makes the external-tool checks advisory during `rehearse`, which never
invokes stalwart-cli - refusing to run read-only reconnaissance because the
operator lacks a tool that reconnaissance would tell them to get was
backwards.
A live migration on 2026-08-24 stopped a production mail server and then
discovered the host's stalwart-cli was 0.13.4 - present, but from when the
CLI shipped with the server, with no `apply` command. The migration needs
v1.0.2+ from the separately-versioned stalwartlabs/cli repository.
Recovery was closed in both directions. v0.16's recovery-mode boot had
already bumped the store schema to v6, so the 0.15.5 binary refused to
reopen it ("expected 5 or below, found 6"). Going forward needed
export.json, which this tool's own failure path had deleted - and
regenerating it required a settings dump from a live v0.15 instance that
could no longer start. The operator restored a day-old snapshot and lost a
day of mail across nine domains.
Three fixes:
1. preflight.CheckExternalTools verifies stalwart-cli exists and is v1.0.2
or later, and that python3 runs - before anything is touched. Every fact
needed to prevent this was available in under a second from a stopped
state. Skipped for a patch upgrade, which invokes neither tool.
2. A failed run no longer deletes its work directory. Cleaning up on every
exit path was right for a sandboxed rehearsal and catastrophic here:
once the service is stopped the settings dump cannot be regenerated, so
deleting it removes the only way forward. The failure now prints the
resume command instead.
3. `run --resume <id>` continues an interrupted run. The checkpoint
machinery existed but never engaged, because run created a new run every
invocation - so a retry re-ran preflight against a binary already moved
aside, and failed. Completed steps are skipped from the checkpoint.
Proven against a VM built to match the failure: stalwart-cli 0.15.5,
accounts and mail seeded.
* preflight refused, service still active, mail still accepted
* a stub CLI passing --version and failing apply left the run stopped
with all eight inputs intact and the resume command printed
* --resume carried it to a clean finish: five seconds of downtime,
listeners regenerated, admin role restored, quotas rebuilt
That failure-path test is the one that should have run before production.
Every earlier test had stalwart-cli installed from the start, and the one
failure I did exercise happened to leave its artifacts behind.
Ran the rehearsal read-only against a live production instance. It
completed, and two preflight checks were wrong in ways a test instance
could not have shown.
1. Store-backend detection missed the config entirely. A config generated
by `stalwart --init` declares `type = "rocksdb"` inside a
`[store.rocksdb]` section, which is what every test fixture here used.
The production config has no section headers at all and declares
`store.rocksdb.type = "rocksdb"` flat. Detection only matched a bare
`type` key, so it reported "no known store backend type found in config"
and left topology.store_backend empty.
That mattered more than a warning suggests: backup.Run treated an
unrecognized backend as a *skip*, so a real run would have continued
with no filesystem or database backup at all - the single artifact that
phase exists to produce, quietly absent. Flat dotted keys are now
detected, and an unrecognized backend is a hard failure rather than a
skip.
2. The cluster warning didn't say where it matched. It fires on any
occurrence of "cluster" anywhere in the config, which is the correct
bias - a missed cluster corrupts a shared store - but on the production
config the only match was inside the value of an unrelated setting,
leaving a whole config to search to establish that. It now names the
location and distinguishes a match in the setting name from one in its
value.
Both verified against the real config: store-backend now reports
"rocksdb (store.rocksdb)", and the cluster warning names the setting,
making a false positive dismissible at a glance.
No production data is in this commit: the fixtures use example.com and the
flat-key shape only. Coverage numbers from that run matched the earlier
scrubbed-corpus measurement exactly.
Found by running preflight against a real Stalwart 0.15.5 in a VM. Two
defects, the second worse than the first.
1. AccountSnapshot could not read the version this tool migrates FROM.
0.15.5 advertises no urn:stalwart:jmap capability and POST /api returns
404 - the JMAP management API and x:Account are 0.16 features. 0.15.x
exposes a REST API at GET /api/principal instead. So preflight's
account-snapshot check warned and moved on, and every run against a real
source instance had no "before" data at all.
AccountSnapshot now dispatches on the capability the session document
advertises - a positive signal, not an inference from a failed call -
and internal/stalwartapi/principal.go implements the 0.15.x REST path,
including its 1-based page/limit pagination so an install larger than
one page isn't silently truncated.
2. With no "before" counts, the content-integrity comparison iterated an
empty map, checked nothing, and reported "all message counts match".
That is the strongest claim this tool makes - ARCHITECTURE 4.7 calls it
the actual no-data-loss guarantee - made vacuously, and it would have
passed on a migration that lost every message.
The comparison now derives its account set from whatever the source
could report, verifies every account and domain survived either way, and
carries MessageCountsCompared so the report says plainly "MESSAGE COUNTS
NOT COMPARED ... no-data-loss is NOT verified here" rather than implying
otherwise.
What can and cannot be checked across the 0.15/0.16 boundary, now that a
real server has answered: 0.15.x has no per-mailbox message count at any
endpoint, and the impersonation login 0.16 offers returns 401 there, so
before/after message counts are impossible for the boundary migration this
tool exists for. Both versions do report per-account used quota (usedQuota
in 0.15's REST list, usedDiskQuota on 0.16's x:Account), so that is
captured on both sides. It is recorded and reported, not asserted on:
4.5 notes the 0.16 migration resets quotas to zero pending recalculation,
so comparing those bytes across the boundary would be a false alarm
generator.
Test servers across preflight, validate and stalwartapi now advertise
urn:stalwart:jmap, since they stand in for 0.16 instances and that
capability is what says so.
Verified end to end against the smoke VM: all nine preflight checks pass,
and the checkpoint records 2 accounts, 1 domain and per-account used quota
where it previously recorded nothing.
GPLv3's "How to Apply These Terms" asks for a notice in each source file;
this is the modern two-line SPDX form of it rather than the full paragraph.
82 files, including tests.
The blank line after the header is load-bearing. In Go a comment block
immediately preceding `package X` becomes the package doc comment, so
without the separator the SPDX lines would be absorbed into the doc for the
eleven packages whose doc.go (or main.go) opens with one, and `go doc` would
print them. Verified it doesn't.
Rename module github.com/johnellis/stalwart-migrator ->
github.com/LINUXexpert-org/stalwart-migrator to match the repository
location, so the module resolves under `go get`.
The initial commit's .gitignore listed the compiled binary as a bare
`stalwart-migrate` pattern, which Git matches at any depth -- so it also
excluded the cmd/stalwart-migrate/ source directory, and the initial
commit shipped without the CLI entrypoint. Anchor the pattern to the
repo root as /stalwart-migrate and add the four missing files.
go build, go vet, and go test ./... all pass.
In-place upgrade tool for Stalwart Mail Server (0.15.5 -> latest) with
checkpointed rollback and post-migration validation. Design stage; see
ARCHITECTURE.md.
jcoffey-dev