8ba5d58f762fab302430672649e68981734e2d42
59
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 jcoffey-dev
|
8ba5d58f76 | ci: add Gitea Actions workflow ported from .gitlab-ci.yml | ||
|
jcoffey-dev
|
5760eed824 |
Merge branch 'ci/release-permalink' into 'main'
Give releases a stable latest-download URL See merge request coffey-labs/stalwart-migrator!2 |
||
|
jcoffey-dev
|
c4e496d3e1 |
Give releases a stable latest-download URL
The install guide tells people to curl .../releases/latest/download/<file> which is a GitHub URL shape. GitLab's equivalent is /-/releases/permalink/latest/downloads/<path> but it only resolves for assets that declare direct_asset_path, and the release job was creating plain links to the package registry. Those carry the tag in the URL, so they can never be a "latest" link. Each asset now also declares /binaries/<file>, which is what the docs will point at. The path is load-bearing: changing it breaks a documented install command. |
||
|
jcoffey-dev
|
a83dc51158 |
Merge branch 'ci/gitlab-pipeline' into 'main'
Run CI on the self-hosted GitLab See merge request coffey-labs/stalwart-migrator!1 |
||
|
jcoffey-dev
|
7b07bd6bbb |
Run releases on the self-hosted GitLab
Ports .github/workflows/release.yml after the GitHub account was suspended. Same shape: tag-driven, amd64 and arm64, SOURCE_DATE_EPOCH for reproducible tarballs, and the same refusal to release a tag that is not an ancestor of the default branch. The publishing half had to change. There is no `gh release`, so the tarballs go to the project's generic package registry and the Release is created with release-cli pointing at them. The install guide sends people straight at release asset URLs, so uploading before creating the Release is deliberate: a Release whose assets 404 is worse than no Release. The Actions workflow stays in the tree as the reference. |
||
|
jcoffey-dev
|
0543931616 |
Release binaries for linux amd64 and arm64
Pushing a v* tag runs .github/workflows/release.yml: vet, test, govulncheck, then scripts/build-release.sh builds reproducible archives for linux/amd64 and linux/arm64 with a SHA256SUMS file, and attaches them to the release. workflow_dispatch takes a tag for a run that never started. Same shape as ihasmail-oneshot's releases. Adds a version subcommand, set at build time. go.mod moves to 1.26.8: the workflow builds with the go.mod version, and govulncheck finds four standard-library vulnerabilities the tool reaches in 1.26.5 (GO-2026-6218, GO-2026-6090, GO-2026-5972, GO-2026-5026), all fixed in 1.26.6. README installs from the latest release, with building from source as the alternative; CONTRIBUTING describes how releases are cut. |
||
|
jcoffey-dev
|
1b4258113e | Name the documentation rows the way ihasmail-oneshot's table does | ||
|
jcoffey-dev
|
bb2f56bfa5 | Add license and docs badges, as ihasmail-oneshot has | ||
|
jcoffey-dev
|
bd117e6fca | Put the documentation links first, with the guide marked as the place to start | ||
|
jcoffey-dev
|
6608f3a97f |
Shorten the README; move the technical detail into docs/
The README keeps what the tool is, how to install it and the first commands, and points to the guide on docs.ihasmail.org. Everything else moves, whole, into docs/ and CONTRIBUTING.md, where it is organized for readers who want the detail. Where the old README disagreed with the code, the code wins. |
||
|
jcoffey-dev
|
3126330eed |
Point the Sponsor button at the current GitHub username
The account behind it was renamed from LINUXexpert-org to jcoffey-dev, and GitHub does not redirect the old name: github.com/sponsors/ LINUXexpert-org answers 404 while the new one answers 200. So the Sponsor button on this repository has been leading nowhere. Worth fixing rather than leaving to redirect, because a released username can be registered by anyone -- a stale link stops being a dead end and starts being someone else's page. |
||
|
jcoffey-dev
|
12ec0c3fd4 |
Rename the module to the Coffey-Labs organisation
The repositories moved off LINUXexpert-org. Here that is not a documentation change: the old organisation was the module path, so it is declared in go.mod and repeated in every internal import. Leaving it would have been worse than a stale link. GitHub redirects the repository, but a go.mod whose module line disagrees with the path it was fetched from is an error rather than a redirect, so `go get` on the new address would have failed against the old declaration. go.mod, 34 files of imports, and the repository links in README and ARCHITECTURE. go mod tidy leaves go.sum untouched -- no dependency moved, only our own path. |
||
|
jcoffey-dev
|
77e80cb9f2 |
Change the copyright holder to Coffey Labs
119 SPDX-FileCopyrightText headers and the README's licence line. The distinction that matters here: LINUXexpert-org appears in this repository in two completely different roles. As a copyright holder in the SPDX headers, which is what changes, and as the GitHub organisation in the module path and 64 import statements, which does not -- the repository still lives at github.com/LINUXexpert-org/stalwart-migrator, and rewriting that would not be a licence change, it would break the build. Both replacements are anchored to their copyright forms, so an import path cannot match either. Import count is 64 before and after, and go.mod is untouched. LICENSE untouched: the FSF's copyright on the GPL text and the "<name of author>" placeholders are not ours to edit. go vet, go build and go test all clean. |
||
|
jcoffey-dev
|
56152ed7de |
Ask the container what it is running
Preflight's first check ran `--version` on --binary. A container-only host has no such file, so the check failed, and because it is first, nothing downstream ever ran — including every container check that exists to decide whether that container can be migrated at all. The container path was unreachable on exactly the hosts it is for. A host that happens to have a binary is the worse case, not the better one: a stray /usr/local/bin/stalwart from an older install answers confidently with a version nothing is running, and the whole migration plan is derived from that number. The source version now comes from running the image the container is on, by ID rather than by the tag it was started from, using the same command and the same fallback stage already uses for the target image — the two have to agree about what a Stalwart image reports or the source and target could be read by different rules. Verified against a real stalwartlabs/stalwart image, not only the fake. Deployment kind is detected once and shared, rather than asked again by the check that reports it. Two answers for one run is not a thing this should be able to produce. The same reasoning retires preserve-binary on a container: there is nothing on this host to move aside, and the equivalent is already guaranteed, since cutover renames the old container and never prunes the old image. Renaming a stray binary would have preserved something nothing was running. |
||
|
jcoffey-dev
|
d0d2954ad1 |
Say what the field report changed and what it left open
Three real 0.15.5 -> 0.16.19 migrations by @kaya-eu, at 221 GB and on arm64, are worth recording — with the qualifications that make them readable. They ran a commit predating the automated Docker cutover and did that step by hand, so what those runs exercise is preflight, the dumps, the conversion and the recovery-mode store migration. Three new sections for what they found: The store migration may need one more recovery boot. This is open. Two failures, never both on one run, each cured by one extra recovery-mode boot, and Stalwart's own "multi-step process" panic says the store migration is not finished when the single boot exits. Documented as the manual step rather than left for the next operator to rediscover, and documented as unfixed rather than implied away. A certificate that serves HTTPS may not serve the mail ports. IMAPS/SMTPS/POP3S keep a self-signed certificate until SystemSettings.defaultCertificateId is set. Written up beside the ACME section, which has the same shape: something the converter does not carry and nothing warns about. This tool should generate it and does not yet. Do not boot recovery mode again afterwards already landed with the artifacts it argues for; this links the production paragraph to it. Two status claims were also stale. §4.5 still said the container path was not reachable from the CLI, which #9 changed, and both it and §7 said no real Stalwart image had been met — one has now been inspected, which is how #11's two bugs were found. What remains untested is everything after the recreate, and the Docker section now says what cutover carries and how the config reaches the container. |
||
|
jcoffey-dev
|
f324d66c5c |
Keep the four files a run cannot produce again
The settings and principals dumps, the apply plan and its supplement lived only in --work-dir, which a successful run deletes. All four are irreplaceable once the store has been migrated: the dumps can only be taken from a live pre-migration instance, and the plan is what was actually replayed. `rehearse` already kept the plan and the supplement, so the read-only command preserved more of its conclusions than the destructive one did. They are now copied into the run's state directory before the store is touched, recorded as artifacts with checksums, and kept whether or not the run succeeded and whether or not --keep-artifacts was passed. README claimed the dumps stayed on disk; now they do. What made this concrete: an operator who booted recovery mode again after a completed migration, for an unrelated reason, and found Domain and Account queries coming back empty on the next start — twice, on two different servers, and verified as genuinely gone rather than a stale read. Re-applying that run's export.json and supplement.json against a fresh recovery boot is what got the server back both times, and they had those files only because they had thought to pass --keep-artifacts. Nobody should have to guess that in advance. The README now says not to boot recovery mode after a migration. That is Stalwart's behaviour rather than this tool's, but this tool is where an operator learns the technique, and it said nothing about it being a one-time step. Reported by @kaya-eu in #1. |
||
|
jcoffey-dev
|
e96e72bf79 |
Tell what a container inherits from what it overrides
Checked against a real stalwartlabs/stalwart image, `docker inspect` on an ordinary container reports User "stalwart", Entrypoint ["/usr/local/bin/stalwart"] and Cmd ["--config", "/etc/stalwart/config.json"] — all three inherited, none of them given. Two things followed from reading those as the operator's. A container user was listed as configuration a recreate would drop, so every container off the official image was refused as unrecreatable. That refusal lived in cutover, downstream of the stop, the settings conversion and the store migration: it arrived with mail down and data already moved, which is the failure issue #1 was filed for. Each of the three is now compared against `docker image inspect` of the image the container is on. Inherited values are left to the new image, whose own defaults are the ones that go with it. Overrides are carried: --user, --entrypoint, and the rest of an entrypoint as leading argv. Cmd and Entrypoint were not being read at all, so an overridden one was silently dropped — the exact loss the unsupported list exists to prevent. The recreatability question also moved into preflight, while the server is still running. Cutover asks it again, since the two are separated by the whole migration, but only one of them can refuse without cost. The other half: the recreated container is now started with `--config` pointing at the migrated config in the data volume. Left to the image's default command it came up on /etc/stalwart/config.json — a different volume, holding whatever the old version left there — so cutover would have produced a running server with nothing to do with the migration that preceded it. An overridden command and that --config are the same argv and cannot be merged honestly, so a container with one is refused and told why. The config is also chowned to whatever owns the data directory, before the recovery cycle opens it. The image runs as uid 2000 and this tool writes as root; §4.8 is the standing reminder that byte-perfect and unreadable is a way to report success. Found while checking @kaya-eu's field report in #1 against a real image. Their three manual migrations are where the config step comes from. |
||
|
jcoffey-dev
|
ee38c38861 |
Stage the build for the machine it will run on
The release asset was a constant naming the x86_64 Linux server build, so an arm64 host downloaded an x86_64 binary and met "exec format error" the first time anything executed it — which is stage's own version check, so it failed early and before the service stopped, but with nothing in the message to say the download had been for the wrong machine. The asset now follows runtime.GOARCH. Only amd64 and arm64 are selected for: those are the two with an unambiguous plain gnu server build, while GOARCH=arm does not say whether a host wants the arm or the armv7 archive. Anything else is refused by name and pointed at --target-binary rather than falling back to x86_64, which is the bug being fixed. Matching stays exact — stalwart-foundationdb-aarch64-unknown-linux-gnu is a substring away from the right answer. Reported by @kaya-eu, who hit this on an arm64 home server and worked around it by fetching the aarch64 archive by hand. |
||
|
jcoffey-dev
|
272439cf2a |
Wire the container path up, behind a flag that says what it is
Everything the container migration needs has landed a piece at a time and nothing called any of it. `run` now does: stage pulls and verifies an image instead of downloading a binary, the recovery cycle launches a throwaway container against the live container's own mounts, and cutover recreates it. Preflight's blanket refusal of docker goes with it -- what still refuses is specific to a container rather than to containers, which is compose and data that is not on a volume. It refuses without --container-path-unproven, and that flag is the honest part of this change. Every test drives a fake docker. That proves the right commands are assembled and proves nothing about whether a real image reads the config it is handed -- which is the exact limit ARCHITECTURE.md section 4.8 records about the rollback code that was deleted for being tested only against fakes. A doc note seemed too quiet for a tool that stops a mail server, so it is a flag nobody reaches without being told. The converted config reaches the container through the data volume. It is written under the host side of whichever mount covers --data-dir and named on the container side, because cutover recreates a container with the mounts it had and cannot invent a new one for a config file. --data-dir therefore names the path inside the container, which preflight already says when it matches no mount. PatchPaths stays unused, deliberately. Its documented purpose is pointing a rehearsal at a sandbox; a real container's dumped settings already carry container-side paths, because they come from the live server rather than from a file on this host. The preflight test that asserted docker was refused outright now asserts the replacement rather than being deleted -- "docker is allowed through here" is the thing that would be wrong to regress. Its fixture had to make --data-dir both a real host directory and one the fake container mounts, since disk-space stats it and container-data-volume wants it covered. README gains the container section and, at the top, the note that this is ihasmail's companion. |
||
|
jcoffey-dev
|
ad7c2d5135 |
Cut over a container, or refuse to for a reason
A container cannot be edited in place the way a unit file can, so cutting one over means rebuilding it. That makes silent loss the default failure: a container recreated without its capabilities, its custom network or its device mappings starts cleanly and is quietly not the server it was. Section 4.5 already answers this for a unit file -- it rewrites in place rather than regenerating, because a generated unit would drop hardening options this tool has no business having an opinion about, and it refuses to edit a line it only partly understands. The same rule applies here, where the whole definition has to be rebuilt: the parts this understands are carried across, and a container using anything else is refused by name rather than rebuilt without it. The list of what it looks for is conservative and not exhaustive, which is the safe direction: docker's HostConfig has far more fields, and one this does not know about is a reason not to be recreating that container at all. The old container is renamed, not removed, and nothing here prunes the old image. Together they are the container's manual restore path -- one command starts the previous container again -- which is as close to section 4.2's preserved binary as a container gets. The inspect output is preserved as an artifact before anything is replaced, for the reason the unit file is: an operator putting a machine back by hand should not also be reconstructing the definition from memory. Recovery-mode variables are stripped from the recreated container's environment. Leaving STALWART_RECOVERY_MODE set would recovery-boot on every restart, which is the same footgun the unit rewrite exists to prevent. Run now branches, with the health check and quota recalculation shared: those ask the same question whatever started the server. The binary path moved inside an else and is otherwise untouched -- no existing test needed editing, which is the evidence for that. The container path is opt-in through Options.Container, so a Docker deployment without it is still refused exactly as before. Nothing calls it yet; wiring `run` up and lifting preflight's refusal is what remains of #3, and ARCHITECTURE.md says so in both places it previously said Docker was refused outright. |
||
|
jcoffey-dev
|
0a6b3ad173 |
Stage a container image the way a binary is staged
The container deployment's answer to downloading a release: pull the image the operator named, then ask it what it is. That last part is the point of the phase, exactly as it is for a binary - the tag, the registry and the repository name are all assumptions about someone else's publishing process, and the image's own answer is the only thing that settles what arrived. The image is never derived from the running container by swapping its tag. That derivation is wrong for a digest-pinned image, wrong for a mirror and wrong for a fork, and being wrong here means pulling the wrong software into a mail server. It is named in full or the phase refuses. What comes back is the image's ID rather than the tag it arrived under. A tag can move between staging and cutover -- that is the whole reason latest is a hazard -- and running the tag later would run something other than what was verified here. VersionFromOutput is exported from preflight so both paths parse a version identically. Two copies of that regex could disagree about what they staged, which is a difference nobody would look for. One caveat recorded rather than hidden: asking an image its version means running it with --version, which assumes its entrypoint is the server and passes flags through. That has not been confirmed against a published Stalwart image, there being none to hand. If the assumption is wrong this fails loudly with the image's own output rather than staging something unverified, and the fallback tries the binary by name before giving up. SkipPull is for a host that loaded the image from a tarball, where a pull cannot work and its failure would say nothing useful. |
||
|
jcoffey-dev
|
0698235d4c |
Run the recovery cycle in a container
The second implementation of the launcher seam: a throwaway container from the staged image, against the live container's own mounts, with recovery mode's two environment variables. Everything the cycle does afterwards - the settings apply, the health check, the boot check - is unchanged, which was the point of putting the seam there. Process gained an unexported start(cmd, what) so a container shares its supervision rather than reimplementing it. For a container the child process is the `docker run` client, so the captured output is the server's own: a recovery boot that dies on a bind conflict says so here, where a detached container would have left it in `docker logs` for nobody. Stopping is the part that needed care, because the client and the container are two things and conflating them loses a store. Signals do reach the container through an attached client, so the ordinary path is fine - but Process.Stop escalates to SIGKILL when the grace period expires, and killing the client does not kill the container. It would be left running, holding the store open, while the run moved on to the next phase believing it had stopped. So the container is stopped by name with docker's own timeout, and its absence confirmed afterwards; a container still running after being told to stop is an error loud enough to halt the run, because nothing may touch that data until it is gone. Two refusals rather than defaults. No image, and no mounts - the second because a recovery container with no mounts would migrate an empty directory and report success, which is the worst outcome available here. Eight tests drive a fake docker: the assembled run command, recovery variables present in recovery mode and absent in an ordinary boot, both refusals, the stop reaching the container by name, the survived-container error, an already-removed container being the normal --rm path rather than a failure, and output capture. Two races in those tests are worth recording, because both were real and neither was in the code. Launch returns once the OS has started the client, which is before the shell it started has run anything, so reading the invocation log or the output immediately raced them; both now poll. And a forked `sleep` in the fake outlived its shell, held the output pipe open and made Wait block for the sleep's full duration - `exec` in the fake fixes it, and the same shape would affect any child that forks. |
||
|
jcoffey-dev
|
5a7a126960 |
Make how the target version is started a seam
Both phases that bring the target version up against a not-yet-migrated store -- the recovery cycle, and the ordinary boot validate does after it -- constructed a child process from a path on this host directly. That is the one thing about them packaging changes: a container runs an image against the data volume instead. Everything either of them is started *for* is identical afterwards. So starting it is now a Launcher, returning a Supervised the callers stop and read output from. BinaryLauncher is today's behaviour and the default when Options.Launcher is nil, so every existing caller is unchanged -- no test needed editing, which is the evidence for that rather than a claim about it. Deliberately narrower than the interface sketched in issue #3. Stage and cutover also differ by packaging, but designing their interfaces now would be designing against a guess: there is no second implementation yet to shape them, and the shape a container needs is what PR 3 and PR 4 find out. This seam is different because it already had two callers doing the same thing for the same reason, so extracting it describes the code rather than predicting it. outputSuffix now takes Supervised. It only ever needed Output(), and the diagnosis it exists to preserve -- the server's own words about a bind conflict or a rejected config value, which a bare timeout loses -- matters whatever started the process. |
||
|
jcoffey-dev
|
6e7a05e478 |
Tell a container operator what stands in the way
Preflight refuses a container, and until now that was all it said. The two things that actually decide whether one could be migrated at all were never looked at, so an operator was refused without being told what to fix or what a manual migration would involve. It now inspects the container and reports three things. What image it is running, tag and digest kept apart because a moved tag makes them disagree and only one of them says what is really there. Whether compose manages it, which matters beyond this tool's current refusal: recreating a compose-managed container out from under compose leaves the container and the compose file disagreeing about what is deployed, and the next `compose up` reverts the migration. And whether the data is on a volume at all -- a container keeping it in its own writable layer loses it when the container is replaced, and replacing the container is what migrating it means, so that one is fatal in a way no later phase could recover from. All three are advisory under rehearse, on the same reasoning that already made the deployment check advisory there: rehearse never stops or recreates anything, and an operator doing this by hand needs these facts more than an automated run does. Preflight stays read-only. Preserving the container definition as an artifact belongs with the phase that replaces it, so it goes with cutover rather than here. One limitation this surfaced and does not fix: --data-dir naming a path inside the container breaks the host-side disk-space check, which stats it locally. Path translation is the next piece of work; the data-volume check says so when the path matches no mount. |
||
|
jcoffey-dev
|
edd8279743 |
Refuse a Docker deployment before stopping anything
Cutover already refused a container -- recreating one from a new image is not swapping a binary and rewriting a unit, and this tool does not automate it -- but it refused from cutover.Run, which run.go calls at line 324. The service is stopped at line 243. So the sequence on a container was: stop Stalwart, convert the settings, then discover the deployment cannot be cut over, return the error, and exit with mail still down. Reported as #1. The deployment kind is known in preflight, before anything has been touched, and that is now where it is acted on: docker is a blocking check. rehearse keeps working -- it never stops the service or cuts over, and telling an operator what the migration involves is most useful precisely when the tool cannot do it for them -- so it sets DeploymentCheckAdvisory, alongside the ToolCheckAdvisory it already set for the same reason. The second half is not docker's alone. Every return between the stop and the end of cutover returned with the service down; a failed settings conversion would have done the same to a systemd host. run now registers a restart on the way out, after the stop rather than before, so it only ever starts something this tool stopped. It does not claim to have recovered the migration -- a part-migrated store still needs --resume or the operator's recovery point -- it removes the narrower failure of exiting on a foreseeable error while the server it stopped stays stopped. |
||
|
jcoffey-dev
|
80a76fbb4a |
Lead with the production result and the two preconditions
Someone deciding whether to trust this needs the first fact immediately: it has migrated a real mail server, and what that cost in downtime. It was buried in a Status section under a line count. The two things an operator has to fix before starting - multi-tenancy, and not migrating as the built-in admin - now sit above everything else too. Both are changes to the directory rather than flags, so finding them after booking a maintenance window is the wrong order, and that is exactly how both were found on the production run. Status keeps the line counts and the command table and no longer repeats the paragraph above it. |
||
|
jcoffey-dev
|
7ec1e70109 |
Say up front what an operator has to fix before migrating
Both blockers found on the production run were changes to the directory, not flags, and both were discovered by reading a preflight failure rather than the documentation. They now lead the README. Multi-tenancy had no section at all: v0.16 requires a tenant-scoped account to sit on a domain owned by that same tenant, v0.15 did not, and an install that is valid today can be unrepresentable tomorrow. Where one tenant's accounts use a tenant-less domain the conversion repairs it; where two tenants share a domain nothing can, and it has to be resolved in v0.15. The admin section predated the preflight check that now refuses a fallback-admin, and it stopped one step short: the account has to keep its rights *after* the migration, not merely exist. An account whose admin came only from `tenant-admin` authenticates afterwards and is still refused every management operation, and preflight cannot predict that - it cannot know which roles the converter carries across. |
||
|
jcoffey-dev
|
3342662b33 |
Record that this migrated a production server, and what it took
On 2026-08-25 the tool migrated a live mail server - nine domains, six accounts, a 2.4 GB RocksDB store - from 0.15.5 to 0.16.19 with 8 seconds of downtime and every phase green, validation included. The practice worth copying is the dress rehearsal on a clone that preceded it. `rehearse` is read-only and stops before applying, which is where a real migration fails; a clone closes that gap for about six seconds of production downtime. It caught four faults that would each have reached production, and three of them only appear against a real instance. The README now describes how to build one, including the parts that are easy to get wrong: a consistent copy of a single-writer store, and a guest with no route out so a copy of a live mail server cannot renew real certificates or deliver real queued mail. It also documents what Stalwart's own converter drops without saying so. migrate_v016.py consumes every acme.* setting, emits nothing, and does not report them as unmigrated - so a migration reports success while removing certificate renewal, and nothing looks wrong until the certificate expires about ninety days later. The AcmeProvider that had to be rebuilt by hand is recorded in ARCHITECTURE.md, shape confirmed against a live 0.16.19, as the specification for generating it in the supplemental plan. |
||
|
jcoffey-dev
|
019696bce7 |
Do not call it data loss when the instance would not show us the accounts
The dress rehearsal reported MISSING ACCOUNT [email protected] after a migration that lost nothing: SMTP on the migrated instance accepts mail for it, while an address that does not exist is refused, so the account is there. Enumeration is permission-scoped. The "before" snapshot was read by an account with full rights; the "after" snapshot by one whose admin role the migration had not carried across - proven one step earlier, where the quota rebuild was refused as unauthorised. It was shown a fraction of the accounts and the comparison called the remainder lost. "Missing" and "not permitted to see" are different findings and only one of them is about the data. When the migrated instance shows fewer accounts than existed, the result now says COULD NOT VERIFY and names the reader, rather than asserting a loss the evidence does not support. It still fails the run: an unverified migration is not a verified one, and the fix is to re-check with an admin account, which --resume makes cheap. Counting accounts is not a strong enough signal to tell the two cases apart, so it does not try to - it reports the ambiguity instead of guessing. Also fixes the test fake, which ignored ?types= and returned domains as individuals, making every account count in these tests wrong. |
||
|
jcoffey-dev
|
ce0e237051 |
Refuse a fallback-admin at preflight, not after the migration finishes
A dress rehearsal on a clone of production migrated cleanly and then failed all three post-cutover steps with 401: the health check, the quota rebuild, and the content comparison. The account was `admin` - a v0.15 [authentication.fallback-admin], not a directory account. v0.16 keeps its configuration in the store, so the block defining it does not survive, and the credential stops working the instant the migration completes. Every check passed beforehand, because on v0.15 that account authenticates perfectly well. The README has said to use a named account for a while; the tool now says it too, while nothing has been touched and changing it costs one flag. The migration itself is unaffected - it succeeds either way. What is lost is the ability to verify it afterwards, which is the part worth having. |
||
|
jcoffey-dev
|
0332f40b25 |
Add tenants: show who owns what before deciding how to unblock a migration
preflight refuses a layout v0.16 cannot represent, but it reports the problem rather than the shape of the instance around it. Choosing between giving a tenant its own domains, moving accounts, or collapsing everything into one tenant needs the whole picture: which domains have an owner, which accounts reach across to somebody else's, and which of those the converter would have repaired by itself. Read-only, and it names the offending domain against each account that touches it rather than making the reader join the two lists by eye. |
||
|
jcoffey-dev
|
b786e89b42 |
Take the target version from the binary when there is no route out
preflight asked the GitHub release API which version it was upgrading to, so a host with no internet failed there even with the target binary already on disk - and `run` never passed its own --target-binary down to preflight, so supplying one did not help. Read the version from the binary instead when one is given, and fail if it is not the version the run targets. Found by the first preflight against the production clone, which is deliberately cut off from the network. |
||
|
jcoffey-dev
|
0739866c14 |
Let a host with no internet supply the migration script itself
`run` and `rehearse` always fetched migrate_v016.py from GitHub, so a mail server with no route out could not be migrated at all - an air-gapped host, or a clone deliberately cut off so it cannot renew certificates or deliver queued mail for the domains it was copied from. --migration-script takes a local copy instead, still checked against --migration-script-sha256 when one is pinned. Found while staging a production clone for a dress rehearsal: the clone has no route out on purpose, and that is exactly the property that stops a copy of a live mail server doing something in the real world. |
||
|
jcoffey-dev
|
3adaee3bc6 |
Stop a clean migration reporting domains it never lost
The post-migration comparison had the two versions counting domains differently, and yesterday's wiring turned that into a gate: `run` would have failed a migration that lost nothing. The 0.15 side added every domain appearing in any account's address on top of the domain principals - the fallback's own comment says "if the instance has no explicit domain principals", but the loop ran unconditionally. The 0.16 side did the reverse, listing only domains some account calls its primary, discarding the full Domain list it had already fetched. An instance with three declared domains and accounts aliased across nine reported nine before and three after. INBUXA is exactly that shape, and this was the account/domain over-count noted as undiagnosed. Both sides now mean "the domains this server holds". A domain that still goes missing is reported as a warning rather than failing the run: what the two versions call a domain differs across this boundary in ways we have now been caught by once, and a missing account - which is compared with a local-part fallback and is what actually matters - still fails. Narrowing OK() also made String() return before printing the domain lines, so the new warning would have been silent. Caught by its own test. |
||
|
jcoffey-dev
|
28128633ef |
Check the migration actually kept everything, and let report say so
internal/validate was written and tested and then never called: `run` ended at cutover, so the tool performed a migration and never confirmed it had carried the data across, and `report` was an error message pointing at the package that would have answered. `run` now compares the migrated instance against the snapshot preflight took and fails if an account or a domain that existed before is missing from it. The comparison runs against the service cutover has just started, which is the instance people will actually use - its real config, its real ports, under its real service manager - and costs no extra downtime; booting a second copy inside the maintenance window would. BootCheck stays as the equivalent for an instance the tool boots itself. The service is left running on a failure. By that point the store has been migrated in place, so stopping it undoes nothing, and only the operator can weigh the finding against their recovery point. A check that could not run is reported as skipped, never as a pass. Preflight only captures the "before" when it has an admin URL, and a run without one has to say it compared nothing rather than imply everything survived - which is the exact failure ARCHITECTURE.md §4.7 warns about. `report <run-id>` re-reads the recorded verdict rather than re-checking: run again next week and you would be asking how the instance looks now, not how it looked when it was migrated. §4.7 said validation ran after cutover while the only implementation booted its own copy, and listed a suite far larger than what exists. It now says which of the two happens, and which checks are real. |
||
|
jcoffey-dev
|
558af1005f |
Stop the docs saying the migration does not work
Two places told an operator that `run` is unimplemented and refuses: the usage text, and the first line of the README's status. Both predate `run` being wired up, and the README contradicted itself three lines later, where the command table says it works. Anyone reading either would conclude the tool cannot migrate. The rest of that section had drifted too: staging exists as a package, and the package table was missing it along with applyplan, while the line counts had aged. Recounted, and the column now says what it is measuring. `report` really is unimplemented, so the usage text now says so where it is advertised. |
||
|
jcoffey-dev
|
e955a41d58 |
Fix the domain/tenant mismatch that failed the second live migration
The second production attempt failed during recovery-mode apply, with the
mail server already stopped and the store already at schema v6:
create Account restore-13: invalidForeignKey | Object id: Domain#d
v0.16 requires a tenant-scoped Account to sit on a Domain owned by that
same tenant, for its primary domain and for every alias. v0.15 imposed no
such rule, and migrate_v016.py carries the two facts over independently:
_build_domains sets a domain's memberTenantId only for domains declared as
their own `domain` principal with a `tenant`, while _build_user sets the
account's from the account's own record. A domain that exists only inside
an email address is inferred, gets no tenant, and every tenant-scoped
account using it is then rejected.
Established by reproduction rather than inference: a synthetic v0.15
principal dump, run through the unpatched upstream converter and applied to
a real 0.16.14 in recovery mode, reproduces the error character for
character - the `#d` is the server's own object id for the offending
domain, not a plan client-id. The same harness establishes which directions
are constrained: a tenant-scoped account on a tenant-less domain or on
another tenant's domain is rejected; a global account on a tenant-owned
domain is accepted.
- applyplan.ReconcileDomainTenants repairs the plan between convert and
apply. Where a tenant-less domain is used only by accounts of one
tenant, the domain adopts that tenant - the sole assignment that both
applies and keeps every account. Where accounts genuinely disagree it
changes nothing and reports why, because forcing such a plan through
would mean dropping mailboxes.
- stalwartapi.FetchTenantLayout maps tenant membership over the 0.15 REST
API and predicts the outcome with the same rule the server enforces, so
preflight either warns about the domains that will adopt a tenant or
fails - while the service is still running.
- The plan is parsed generically rather than through the typed Operation.
A real export.json mixes shapes: `create` maps a client-id to an object,
`update` carries a flat one. The typed form failed on the first `update`
line, found by running against actual converter output. Numbers decode
as json.Number so a 10 GiB quota is not rewritten as 1.073741824e+10.
Corrects the record: the previous commit claimed the converter emits every
Account with `tenantId: null` and made preflight refuse every multi-tenant
install on that basis. The field is memberTenantId, the converter does
populate it, and the export had been inspected for a key no version of the
script ever writes. The refusal is now narrowed to what v0.16 genuinely
cannot represent.
The same fix has been prepared for migrate_v016.py upstream. The tool
downloads that script rather than vendoring it, so the repair stays here
until a released version carries it, and is a no-op on a consistent plan.
|
||
|
jcoffey-dev
|
c29140b6b3 |
Refuse multi-tenant installs in preflight, not after the service is stopped
A second live attempt failed in the same shape as the first: preflight
clean, settings dumped and converted, then a failure during recovery-mode
migration with the mail server already stopped.
created Tenant (1)
created Domain (9)
create Account restore-13: invalidForeignKey | Object id: Domain#d
migrate_v016.py carries the Tenant and the Domains but emits every Account
with a null tenantId, so the account references a tenant-owned domain while
belonging to no tenant and the foreign key is rejected. That is Stalwart's
converter and there is nothing this tool can do about it: a multi-tenant
install has to be migrated by hand until the converter handles tenants.
What this tool got wrong was the timing. Tenant principals are one API call
away and were readable the entire time the server was running. Preflight now
queries them and fails before anything is touched, with an explanation of
exactly what would otherwise fail and when.
This is the same lesson as the stalwart-cli check: knowable in advance,
discovered after a production mail server had been stopped, twice. Any
dependency of the conversion belongs in preflight, not in the phase that
consumes it.
Also makes the external-tool checks advisory during `rehearse`, which never
invokes stalwart-cli - refusing to run read-only reconnaissance because the
operator lacks a tool that reconnaissance would tell them to get was
backwards.
|
||
|
jcoffey-dev
|
9faa21f4f1 |
Fix the three defects that cost a production restore
A live migration on 2026-08-24 stopped a production mail server and then
discovered the host's stalwart-cli was 0.13.4 - present, but from when the
CLI shipped with the server, with no `apply` command. The migration needs
v1.0.2+ from the separately-versioned stalwartlabs/cli repository.
Recovery was closed in both directions. v0.16's recovery-mode boot had
already bumped the store schema to v6, so the 0.15.5 binary refused to
reopen it ("expected 5 or below, found 6"). Going forward needed
export.json, which this tool's own failure path had deleted - and
regenerating it required a settings dump from a live v0.15 instance that
could no longer start. The operator restored a day-old snapshot and lost a
day of mail across nine domains.
Three fixes:
1. preflight.CheckExternalTools verifies stalwart-cli exists and is v1.0.2
or later, and that python3 runs - before anything is touched. Every fact
needed to prevent this was available in under a second from a stopped
state. Skipped for a patch upgrade, which invokes neither tool.
2. A failed run no longer deletes its work directory. Cleaning up on every
exit path was right for a sandboxed rehearsal and catastrophic here:
once the service is stopped the settings dump cannot be regenerated, so
deleting it removes the only way forward. The failure now prints the
resume command instead.
3. `run --resume <id>` continues an interrupted run. The checkpoint
machinery existed but never engaged, because run created a new run every
invocation - so a retry re-ran preflight against a binary already moved
aside, and failed. Completed steps are skipped from the checkpoint.
Proven against a VM built to match the failure: stalwart-cli 0.15.5,
accounts and mail seeded.
* preflight refused, service still active, mail still accepted
* a stub CLI passing --version and failing apply left the run stopped
with all eight inputs intact and the resume command printed
* --resume carried it to a clean finish: five seconds of downtime,
listeners regenerated, admin role restored, quotas rebuilt
That failure-path test is the one that should have run before production.
Every earlier test had stalwart-cli installed from the start, and the one
failure I did exercise happened to leave its artifacts behind.
|
||
|
jcoffey-dev
|
5a4c175042 |
Implement run: the migration pipeline, end to end
The phases have all existed for a while; nothing chained them. The order
here is the one arrived at by performing this migration by hand against a
clone of production before writing it down:
preflight -> stage -> dump -> preserve binary -> STOP ->
convert -> supplement -> recovery-mode migration -> cutover -> START
The dump runs before the stop because it reads settings over the admin API,
and a stopped server has no admin API. Everything from the stop to the end
of cutover is downtime.
internal/stage fills the last missing phase (4.3): resolve the release,
take the x86_64 linux-gnu server build and refuse to substitute another,
verify a pinned checksum if one was given, extract the binary - refusing
any archive entry that isn't a regular file, since a tarball is untrusted
input - and confirm the result reports the version its tag claimed.
Everything upstream of that last check is an assumption about someone
else's release process.
Two gates, separate on purpose. --yes is about intent. --recovery-point-
confirmed is a claim about the world: this tool cannot undo a migration
(4.8) and cannot check whether a snapshot exists, so a run that proceeded
without the operator asserting one would be proceeding on a hope.
Verified end to end against a real Stalwart 0.15.5 with email-style account
names, a named admin account, and seeded mail:
MIGRATION COMPLETE. Mail was down for 6s.
Every cutover step green, including recalculate-quotas ("rebuilt disk
quotas for 2 account(s)") - the first time the x:Task wire format inferred
from Stalwart's schema reference has actually been exercised. It works,
now that endpoint discovery and role restoration make it reachable. After
the migration the named admin still administers, alice logs in with
unchanged credentials to the same four messages, and new SMTP delivery is
accepted.
Both refusal gates were tested, as was the failure path: an apply that
fails leaves the run stopped with the store part-migrated, and the error
says to restore the recovery point rather than restart the old version
against it.
|
||
|
jcoffey-dev
|
479e6d563e |
README: require a named admin account, and say why
A config-file fallback admin does not survive the migration, and an operator whose only administrator is one comes out of a successful migration unable to administer the server. Three verified reasons, all found on real migrations: v0.16 keeps its config in the store so the old [authentication.fallback-admin] block stops being read at all; migrate_v016.py assigns every migrated account the User role regardless of what it held, so even a directory admin needs its role restored; and the restoration is refused when an account's local part is ambiguous across domains, since granting administrator rights to the wrong account is worse than granting them to none. |
||
|
jcoffey-dev
|
2ca9522f9a |
Fix two defects a production-clone dress rehearsal exposed
Streamed a clone of a production store into the smoke VM - 3.6 GB, 12,361 settings, 6 accounts across 9 domains - and migrated it 0.15.5 -> 0.16.14 with the tool's own phases. The migration succeeded. Two defects surfaced that no smaller instance could have shown, plus one finding worth recording. 1. Account roles broke on production-shaped names. v0.16 stores an account as a local part plus a domain reference: a v0.15 account named "[email protected]" becomes name "john" with a domainId. The generator passed the full address and the server rejected it outright ("Invalid email local part"), failing the apply. The smoke instance used bare usernames - alice, bob - and never exercised this. Fixed to use the local part. And because local parts are unique only within a domain - [email protected] and [email protected] both become "postmaster" - an ambiguous one is now refused with a warning rather than risking an upsert that grants Admin to the wrong account. Verified on the clone: the one admin came out with roles {"@type": "Admin"} and the other five accounts untouched. 2. Cutover's health check conflated liveness with credentials. A config fallback-admin does not survive the migration - v0.16's config is a store pointer, so the old [authentication.fallback-admin] block simply ceases to exist - so the credentials supplied for the pre-migration instance came back 401 on the migrated one, and the check reported the service as never having answered. It had answered; it was up and serving on all ten ports. Liveness and credentials are now separate: any response proves the service is up, and credentials that stopped working are a warning that names this cause. Also recorded: a failed apply leaves the store in bootstrap mode, where only Bootstrap objects are accessible. A half-applied plan is not a partially configured server but an unusable one. Timing, which is the other reason to rehearse: the recovery-mode conversion of that 3.6 GB store took 2 seconds. A migration window is dominated by waiting and verification, not data volume. No production data in this commit; fixtures use example.net and the shapes involved. |
||
|
jcoffey-dev
|
0d83283caa |
Classify the unmigrated settings instead of just counting them
A production rehearsal reported 12,182 settings not carried over by
migrate_v016.py. As a bare number that reads as an impossible amount of
manual reconstruction, and it is misleading. Snapshotting a v0.16.14 store
migrated from a real v0.15.5 showed what those settings actually are:
8547 regenerates server.blocked-ip
3337 shipped with v0.16 lookup.url-redirectors, lookup.trusted-domains,
spam-filter.list, spam-filter.rule,
spam-filter.dnsbl, lookup.surbl-hashbl
224 already carried server.listener, signature.* (DKIM)
293 NEED YOUR REVIEW queue.schedule, config.local-keys,
server.auto-ban, spam-filter.llm, queue.tls, ...
server.blocked-ip is auto-ban state that repopulates from live traffic. The
stock groups are data v0.16 provides itself - 2,084 MemoryLookupKey, 66
SpamRule and 18 SpamDnsblServer objects were already present in the migrated
store. DKIM came across as DkimSignature objects with private keys intact,
verified on that instance. So the real worklist is ~293 keys, not 12,182.
backup.UnmigratedReport.Classify encodes this and the rehearsal now reports
the categorised view. Rules match longest-prefix-first, because
server.blocked-ip is runtime state while server.auto-ban beside it is
configuration, and an unrecognized prefix defaults to "needs review" -
assuming an unknown setting is safe to ignore is the wrong default.
This also retired the lookup and spam-filter generators that were the
planned next step. v0.15's rules are stwt_rbl_senderscore_ip; v0.16's are
STWT_RBL_SENDERSCORE_IP - the same stock set, already installed. Generating
them from v0.15 would duplicate every rule and revert upstream updates, so
they were deliberately not written. The targets worth generating are the
small site-specific groups instead: queue.schedule, queue.tls,
session.auth, server.auto-ban.
No production data in this commit: the test fixture uses the real group
names and counts with example.com standing in for customer domains.
|
||
|
jcoffey-dev
|
a69f0bbff1 |
Fix two preflight defects a live production rehearsal found
Ran the rehearsal read-only against a live production instance. It completed, and two preflight checks were wrong in ways a test instance could not have shown. 1. Store-backend detection missed the config entirely. A config generated by `stalwart --init` declares `type = "rocksdb"` inside a `[store.rocksdb]` section, which is what every test fixture here used. The production config has no section headers at all and declares `store.rocksdb.type = "rocksdb"` flat. Detection only matched a bare `type` key, so it reported "no known store backend type found in config" and left topology.store_backend empty. That mattered more than a warning suggests: backup.Run treated an unrecognized backend as a *skip*, so a real run would have continued with no filesystem or database backup at all - the single artifact that phase exists to produce, quietly absent. Flat dotted keys are now detected, and an unrecognized backend is a hard failure rather than a skip. 2. The cluster warning didn't say where it matched. It fires on any occurrence of "cluster" anywhere in the config, which is the correct bias - a missed cluster corrupts a shared store - but on the production config the only match was inside the value of an unrelated setting, leaving a whole config to search to establish that. It now names the location and distinguishes a match in the setting name from one in its value. Both verified against the real config: store-backend now reports "rocksdb (store.rocksdb)", and the cluster warning names the setting, making a false positive dismissible at a glance. No production data is in this commit: the fixtures use example.com and the flat-key shape only. Coverage numbers from that run matched the earlier scrubbed-corpus measurement exactly. |
||
|
jcoffey-dev
|
1684c88877 |
Resolve v0.16 domain ids to names before comparing directories
x:Account.domainId is an internal id on v0.16 ("b"), not a domain name. A
pre-migration snapshot taken from a v0.15 instance records names
("smoke.test"), so the post-migration directory comparison compared ids
against names and would have reported every domain as having vanished -
a false alarm on the check whose whole job is proving nothing was lost.
The client now resolves them with x:Domain/query + x:Domain/get in a single
request, using a JMAP back-reference (RFC 8620 3.7). Confirmed against a
live 0.16.14 before being written:
["x:Domain/get", {"list":[{"name":"smoke.test","id":"b"}]}, "g"]
An id that can't be resolved is kept as-is - a domain that can't be named is
still a domain that exists - but a failure of the resolution call itself is
an error rather than a silent fallback, since quietly comparing ids against
names is precisely the bug being fixed.
Verified against the live migrated instance: the snapshot that reported
domains=[b] now reports domains=[smoke.test], matching what the
pre-migration snapshot recorded.
|
||
|
jcoffey-dev
|
28c0fa57cb |
Restore administrator roles that migrate_v016.py drops
Chased down why a migrated instance had no working administrator. The
account authenticated fine and was refused every management call, and the
cause is that migrate_v016.py assigns every migrated account the User role
regardless of what it held before: an account that was `roles: ["admin"]`
in v0.15 comes out the far side as `roles: {"@type": "User"}`.
Ordinary users were never affected - User is what they had and what they
get - and their credentials, mail and mailboxes survive untouched. It is
specifically administrators who lose their privileges, which is a bad thing
to discover after cutting over.
The v0.16 shape came from the server's own schema document rather than the
published reference: GET /api/schema defines x:UserRoles as a multi-variant
type with variants User, Admin and Custom. Account is itself multi-variant,
so an upsert needs its own "@type" too - without it the server rejects the
operation outright ("upsert entry is missing `@type`").
applyplan.AccountRoleOperations restores roles from the principals dump,
emitting operations only for accounts whose role actually changes.
Rewriting every account would be a much larger blast radius for no benefit.
Where v0.15 listed several roles, admin wins - under-privileging an
administrator locks them out, which is the failure being fixed - and the
collapse is reported rather than done silently, as are roles with no known
v0.16 equivalent.
Verified end to end on the smoke VM: rehearse against the real 0.15.5 put
the role operation in the supplement, applying that supplement to a
migrated 0.16.14 whose admin was broken restored management access
(accounts=3), and alice and bob logged in over IMAPS with unchanged
credentials, read their mail, and accepted new SMTP delivery.
Also recorded: x:Account.domainId returns an internal id on v0.16, not a
domain name, so the post-migration directory comparison would read every
domain as missing. Resolving that needs an x:Domain/get call not yet
confirmed against the binary.
|
||
|
jcoffey-dev
|
3e155fa42c |
Fix three defects a full VM migration exposed
Ran a complete 0.15.5 -> 0.16.14 migration of the smoke VM, driving the
phases in the order the real pipeline will. It worked - all mail intact and
readable afterwards, all ten listeners up, cutover executed for the first
time ever and checkpoint resume exercised - and it exposed three defects.
1. The converted config was installed root-owned while the service runs as
its own user. Stalwart crash-looped 28 times on "Failed to read data
store settings: Permission denied", minutes after the mistake and
nowhere near it. This is the same ownership trap that retired the
rollback implementation, in a new place: writing files as root is the
natural thing for a tool running as root to do, and it is wrong every
time the service is not root.
Cutover now installs the config itself, copying ownership and mode from
the config being replaced.
2. v0.16.14 does not serve /api - the endpoint stalwartapi assumed.
Confirmed against a fully migrated, fully configured, serving instance
rather than a sandbox: /api, /api/principal and /jmap/ all 404. The JMAP
endpoint is the one the session document advertises, which is what RFC
8620 discovery is for.
The client now discovers it, re-basing the advertised path onto the
operator's host: a real instance advertises its canonical public URL
("https://mail.smoke.test/jmap/") which frequently isn't reachable from
where this tool runs. The session is authoritative about the path; the
operator is authoritative about the host.
3. Dispatching on the urn:stalwart:jmap capability was wrong, because
NEITHER version advertises it - not 0.15.5, and not a fully migrated
0.16.14. That sent 0.16 instances down the 0.15 REST path where every
call 404s. The client probes what the instance actually serves instead.
Less elegant than a declared capability, with the advantage of being
true.
Also: a JMAP "forbidden" now explains itself. An account holding the admin
role before the migration was refused x:Account/query afterwards, and a
bare "forbidden" gives an operator nowhere to start. Whether the role
failed to carry or v0.16 wants different permissions was not isolated, and
that question is recorded as open - it gates quota recalculation and any
post-migration validation.
Verified against both live instances: the 0.15.5 reports 3 accounts and its
domain over REST, and the migrated 0.16.14 routes to JMAP, finds the right
endpoint, and returns the explained refusal.
|
||
|
jcoffey-dev
|
3bd694114f |
Generate a v0.16 apply plan for the listeners migrate_v016.py leaves behind
First piece of ARCHITECTURE.md 4.3's apply-plan, and the piece that decides
whether a migrated server answers at all: server.listener is not among the
settings the official converter carries, so a freshly migrated instance
binds nothing. Every other unmigrated setting degrades the server; this one
stops it being a server.
internal/applyplan maps server.listener.* onto x:NetworkListener objects and
reports its own coverage. Against the smoke instance that is 24 of 3,505
unmigrated keys - 0.7% - and the output says 0.7%, listing the largest
groups it did not touch. A plan covering a fraction while implying
completeness would be worse than no plan.
The wire format was confirmed against the binary, not the documentation.
The published schema reference gives NetworkListener.bind as a JSON array;
0.16.14 rejects that outright ("Invalid value for object property.
Properties: bind"). The encoding it accepts is a value-keyed set,
{"[::]:25": true}, found by applying a plan to a live recovery-mode 0.16.14
and reading it back with `stalwart-cli snapshot`. Only mappings confirmed
that way are in DefaultGenerators; managesieve -> manageSieve is the one
protocol whose spelling changes, and an unrecognized protocol is reported
and skipped rather than passed through to fail at apply time.
Operations are upserts matched on name, so a plan can be re-run - an
operator will run it more than once - and the supplement is applied after
export.json rather than merged into it, so a generated mapping can never
override one the official script got right.
Verified end to end: rehearse against a real 0.15.5 generated ten
listeners, `stalwart-cli apply` created all ten on a real 0.16.14 with zero
failures, a snapshot read them back with correct protocols, binds and TLS
flags, and re-applying reported 10 updated / 0 created / 0 failed.
|
||
|
jcoffey-dev
|
a0f846a31b |
Replace the sandbox dry run with a read-only rehearsal
`run --dry-run` cloned the data directory into a sandbox, migrated the copy,
booted it, and compared content before and after. Running that design
against a real 0.15.5 instance and a real production settings corpus
retired it:
* The mechanics were never the risk. Backup, dump, convert and the
recovery-mode store migration all worked essentially first time.
* Its final comparison cannot work at all. It needs the migrated sandbox
to answer an API, and server.listener is not among the settings
migrate_v016.py carries - so a migrated instance has no listeners and
answers on nothing. That is the true post-migration state, not a
sandbox artifact to engineer around.
* The expensive half bought the least: against a 3.6 GB production store
it copies the data twice, reading a live mail store, to prove RocksDB
files copy and recovery mode can open them.
Meanwhile the cheap half found every problem that would have derailed a
real migration - an empty defaultHostname v0.16 rejects, passwords v0.16
refuses to create, and a 12,182-key reconstruction worklist - and needs no
data copy at all.
So `stalwart-migrate rehearse`: preflight, dump, convert, report. It copies
nothing, starts no server, and never writes to the store, so it is safe to
run against production repeatedly without a maintenance window. It needs no
target binary either, since convert is pure Python.
The scratch directory is cleaned up as before, with the rehearsal's two
conclusions lifted out first and recorded as artifacts: export.json (what
will carry over) and unmigrated.txt (what will not). Recording an artifact
whose path was about to be deleted was a bug in the first cut of this;
both now resolve.
`run` keeps its refusal and explains where rehearse went. `--dry-run` is
kept as a flag purely to say what replaced it.
Verified against the smoke VM end to end: rehearsal completes read-only in
seconds and reports 3505 unmigrated settings on a default install,
listeners included.
|
||
|
jcoffey-dev
|
b88724632c |
Capture and surface the settings migrate_v016.py does NOT migrate
Ran the converter against a real production settings corpus pulled from the
instance this tool is being built to migrate (secrets scrubbed on that host;
nothing sensitive transited). The result reframes what "a successful
migration" means:
total settings: 12401
NOT migrated: 12182 (98.2%)
actually migrated: 219 (1.8%)
Stalwart's own script reports this in an unmigrated.txt it writes beside its
output - and this tool was throwing that file away. Worse, RunSettingsConvert
never set cmd.Dir, so the script wrote unmigrated.txt into whatever directory
the operator happened to launch from, or failed the whole convert when that
directory wasn't writable. Both reproduced.
The largest groups left behind on production are spam-filter rules, DNSBLs,
trusted-domain and URL-redirector lookups, queue scheduling and TLS
settings - and server.listener. That last one explains something that had
been puzzling from an earlier smoke run: a freshly migrated 0.16 instance
answered on none of the ports the old one did, and served nothing but
/admin. Its listeners never migrated.
So:
- SettingsConvertOptions gains WorkDir, and the convert runs there. The
report lands somewhere known and an unwritable cwd can't fail the step.
- ReadUnmigratedReport parses it; UnmigratedReport.Summary renders the
largest groups first.
- The dry run records it as an "unmigrated-settings" artifact with a
checksum, notes the count in the checkpoint step, and prints it as a
standing warning rather than a footnote.
ARCHITECTURE 4.3 anticipated a "best-effort apply-plan" gap here. The gap is
not a few stragglers needing review; it is effectively the entire
configuration, and the tool has to say so where nobody can miss it.
Verified against the smoke VM: even a default `stalwart --init` instance
reports 3505 of ~3514 settings unmigrated, listeners included.
|
||
|
jcoffey-dev
|
680e554f23 |
Capture and report the supervised Stalwart process's output
A dry run against a real 0.15.5 instance failed with:
recovery mode did not come up: http://127.0.0.1:8081/ did not become
reachable within 1m0s: connect: connection refused
Stalwart had explained itself immediately - "Failed to bind to [::]:8080:
Address already in use" - into a pipe nothing was reading. Diagnosing a
one-line problem took several rounds because the tool threw away the only
evidence. Anything that reports a supervised process failing has to be able
to say why.
Process now captures the child's combined stdout and stderr into a bounded
buffer (64 KiB, keeping the most recent output, with truncation marked
rather than silent - a dead server's reason is at the end of its log), and
exposes it via Output(). recovery.Run appends it to both the startup-timeout
and settings-apply failures, and validate.BootCheck to its boot failure.
Fixes a second bug found while testing the first: Stop returned early when
Signal reported the process had already exited, so cmd.Wait was never
called. Wait is what reaps the child AND waits for the goroutines copying
its output - so the output was discarded in exactly the case where it
matters most, the server dying on its own. os.ErrProcessDone is now treated
as "already gone, still reap it".
The test reproduces the original failure shape: hold the port, start the
helper, let the health check time out, and assert the child's own bind
error survived. Confirmed against the smoke VM too - the same run now ends
with Stalwart's "Address already in use (os error 98)" printed inside the
tool's error.
|
||
|
jcoffey-dev
|
4568f9abbf |
Capture the pre-migration snapshot from 0.15.x, and stop claiming counts match when none were compared
Found by running preflight against a real Stalwart 0.15.5 in a VM. Two defects, the second worse than the first. 1. AccountSnapshot could not read the version this tool migrates FROM. 0.15.5 advertises no urn:stalwart:jmap capability and POST /api returns 404 - the JMAP management API and x:Account are 0.16 features. 0.15.x exposes a REST API at GET /api/principal instead. So preflight's account-snapshot check warned and moved on, and every run against a real source instance had no "before" data at all. AccountSnapshot now dispatches on the capability the session document advertises - a positive signal, not an inference from a failed call - and internal/stalwartapi/principal.go implements the 0.15.x REST path, including its 1-based page/limit pagination so an install larger than one page isn't silently truncated. 2. With no "before" counts, the content-integrity comparison iterated an empty map, checked nothing, and reported "all message counts match". That is the strongest claim this tool makes - ARCHITECTURE 4.7 calls it the actual no-data-loss guarantee - made vacuously, and it would have passed on a migration that lost every message. The comparison now derives its account set from whatever the source could report, verifies every account and domain survived either way, and carries MessageCountsCompared so the report says plainly "MESSAGE COUNTS NOT COMPARED ... no-data-loss is NOT verified here" rather than implying otherwise. What can and cannot be checked across the 0.15/0.16 boundary, now that a real server has answered: 0.15.x has no per-mailbox message count at any endpoint, and the impersonation login 0.16 offers returns 401 there, so before/after message counts are impossible for the boundary migration this tool exists for. Both versions do report per-account used quota (usedQuota in 0.15's REST list, usedDiskQuota on 0.16's x:Account), so that is captured on both sides. It is recorded and reported, not asserted on: 4.5 notes the 0.16 migration resets quotas to zero pending recalculation, so comparing those bytes across the boundary would be a false alarm generator. Test servers across preflight, validate and stalwartapi now advertise urn:stalwart:jmap, since they stand in for 0.16 instances and that capability is what says so. Verified end to end against the smoke VM: all nine preflight checks pass, and the checkpoint records 2 accounts, 1 domain and per-account used quota where it previously recorded nothing. |
||
|
jcoffey-dev
|
4b0bec8956 |
Add SPDX headers to every Go file
GPLv3's "How to Apply These Terms" asks for a notice in each source file; this is the modern two-line SPDX form of it rather than the full paragraph. 82 files, including tests. The blank line after the header is load-bearing. In Go a comment block immediately preceding `package X` becomes the package doc comment, so without the separator the SPDX lines would be absorbed into the doc for the eleven packages whose doc.go (or main.go) opens with one, and `go doc` would print them. Verified it doesn't. |
||
|
jcoffey-dev
|
6c838e8b70 |
Add GPL-3.0 license
Copyright LINUXexpert-org. LICENSE is the canonical FSF text from gnu.org/licenses/gpl-3.0.txt verbatim, not a reflowed or paraphrased copy, and README gains the standard notice. GPLv3 rather than the AGPLv3 used by cairnobs: this is a command-line tool, so AGPL's network clause - the whole reason to reach for AGPL over GPL - has nothing to attach to here. No per-file SPDX headers, matching this workspace's existing convention. Nothing is vendored: the tool is standard library only, and migrate_v016.py is fetched from Stalwart at runtime rather than redistributed. |
||
|
jcoffey-dev
|
7e04351b0f |
Add cutover; drop rollback in favour of operator-provided recovery
Two changes that arrived together: the cutover phase (ARCHITECTURE.md 4.5) is implemented, and the rollback phase is deleted. Recovery from a failed migration is now explicitly the operator's own snapshot or backup, and out of scope for this tool. internal/cutover implements 4.5 as seven checkpointed steps: verify the staged binary's version, install it, preserve and rewrite the service definition, reload, start, wait for a healthy JMAP session, recalculate quotas. The unit is rewritten in place rather than generated from a template. An operator's unit carries hardening options, limits and dependencies this tool has no business having an opinion about, and regenerating it would silently drop them. It repoints ExecStart (preserving systemd's -@:+! prefix characters and every argument after the executable), updates --config, and strips recovery-mode Environment lines - leaving STALWART_RECOVERY_MODE=1 set would recovery-boot the service on every restart, forever. It refuses on a unit with no ExecStart, and on an Environment line mixing a recovery variable with others: a line it only partly understands is one it must not edit. Quota recalculation is the one step allowed to fail without failing the phase. Its wire format is grounded in Stalwart's x:Task schema reference - Task/set creating one AccountMaintenance per account with maintenanceType recalculateQuota - but the upgrade guide only documents the WebUI path, so two details remain inferred and are called out in stalwartapi/task.go: whether the schema's "read-only" annotation on accountId/maintenanceType means "immutable after creation", and whether a finished task simply leaves the queue (TaskStatus documents Pending/Retry/Failed with no success state). Warning rather than failing is the honest response to that uncertainty, and stale counters are an accounting problem next to calling for a restore of a machine that is otherwise migrated and serving mail. Docker deployments are refused outright: cutting a container over means pulling an image and recreating it, not swapping a binary. On removing rollback. The implementation worked and was tested, and it was removed because restoring bytes correctly is not the hard part. It copied file contents and permissions and verified every restored file against a manifest - and did not preserve ownership. Run as root, as this tool requires, it would have produced a byte-perfect, checksum-verified, root-owned data directory that Stalwart, running as its own user, could not open, and it would have reported success. The PostgreSQL path was worse: pg_dump without --clean emits CREATE TABLE + COPY, which fails replaying into a database whose tables still exist, and the ON_ERROR_STOP=1 added so a half-applied restore couldn't be reported as success turned that into a hard failure. None of it had ever run against a real server. A filesystem snapshot has none of these failure modes, because it never lost the metadata to begin with. So cutover's gate is no longer rollback.CanRollBack but an explicit RecoveryPointConfirmed acknowledgement. That is an assertion, not a check - this tool cannot verify someone else's snapshot - and its only value is that nobody migrates a production mail server having never been asked the question. Two consequences are accepted deliberately: restoring any pre-migration recovery point discards mail delivered since, and a failed migration now stops and reports rather than undoing itself. What the tool still does to make a manual restore easier: the old binary is preserved and never deleted, the original service definition is preserved before the rewrite, the settings and principals dumps stay on disk, and every artifact path and checksum stays in the checkpoint where `status <run-id>` can print it. Also removed: the `confirm` command stub and RollbackWindowClosed, whose only purpose was closing a rollback window that no longer exists, and checkpoint.PhaseRollback. Old state.json files still load - JSON ignores the now-unknown field. Still open, and recorded in 8: cutover ignores systemd drop-ins, so an ExecStart or Environment override in stalwart.service.d/*.conf is invisible to the rewrite - including the recovery variable it exists to strip; nothing prevents concurrent runs on the same run-id; and nothing in this repo has ever run against a real Stalwart, real systemd, or a real store. |
||
|
jcoffey-dev
|
ade9906275 |
Implement internal/rollback and the service control it needs
Rollback was the one phase gating everything else: `run` without --dry-run refused because this tool could not undo a cutover it had committed to. That reason is now gone, and the refusal has narrowed to the fact that there is no real cutover to undo yet. internal/rollback implements ARCHITECTURE.md 4.8 as eight checkpointed steps under PhaseRollback: verify-backup, stop-service, preserve-failed-state, restore-data, restore-binary, restore-service-config, start-service, verify-rollback. Three things depart from what 4.8 specified, each for a reason: - The backup is re-verified against its manifest *before* the service is stopped, which the design didn't call out. Finding a corrupt backup is survivable while the failed instance is still up, and unsurvivable once its data directory has been moved aside. - BuildPlan is separate from Run, so every reason to refuse (closed rollback window, FoundationDB, no recorded backup, unknown deployment kind, missing database credentials) is found before anything is touched. The CLI prints that resolved plan and acts only with --yes. - The restore is re-verified against the same manifest after writing. A restore that put back truncated bytes and reported success would be worse than one that failed outright. Nothing from the failed attempt is deleted: the half-migrated data directory and the displaced binary are moved to .failed-<run-id> names, so a retry after the underlying issue is fixed still has both the evidence and the artifacts. Afterwards a reduced validation suite runs against the *restored* instance (version, reachability, directory counts) rather than assuming the restore worked. internal/service is a new package holding the systemd/Docker control this needs. It's separate rather than living inside internal/rollback because cutover will need the identical operations, and because the commands that can take mail delivery down belong in one auditable place - the same reasoning that makes stalwartapi the only thing speaking JMAP. preflight.DeploymentKind is now a type alias for service.Kind so detection and control can't drift apart. Its Active() reads `systemctl is-active`'s output rather than its exit status: systemctl exits non-zero for every non-active state, so exit-status logic would make "inactive" - the answer a rollback most needs - look like a failure to read the state at all. Also fixes a pre-existing bug in `status`: Go's flag package stops parsing at the first positional argument, so `status <run-id> --state-dir X` looked the run up in the default directory and reported it missing. `rollback` would have inherited the same footgun on a command whose flags decide what gets overwritten. Still open: `confirm` cannot set RollbackWindowClosed. Rollback honours the flag and refuses when it's set, but closing the window is the point of no return for the backups this restores from, so it should land with the retention policy 6 describes rather than before it. Verified end to end against a fake systemd deployment: half-migrated data restored to its original contents, failed state preserved, old binary reinstalled and reporting 0.15.5, unit restarted, and a re-run of the completed rollback inert. |
||
|
jcoffey-dev
|
56f465d8a9 |
docs: rewrite the README around what actually works
The old README was wrong in both directions. It called the project "architected, not yet implemented" while ~6,000 lines of tested code exist across backup, preflight, checkpointing, validation and recovery; and it listed six subcommands as a flat menu when three of them return "not implemented yet" and a fourth refuses unless given --dry-run. Replaced with a per-command and per-package status table, each entry checked against the binary rather than the design doc: rollback, confirm and report were run to confirm they error out, and the package line counts come from the tree. Records why `run` refuses without --dry-run -- internal/rollback is a doc.go and nothing else, and committing to a migration with no working rollback would break the one guarantee the tool exists to make. That refusal is correct behaviour today, not a defect to be filed. Corrects a claim in my own first draft: preflight is not purely read-only. Its checks against the Stalwart install are, but it records the run as a checkpoint first, so it fails without write access to /var/lib/stalwart-migrator -- a compile-time constant with no flag or env override. Found by running it. |
||
|
jcoffey-dev
|
5566eed1c8 |
Fix module path and restore cmd/ omitted by gitignore
Rename module github.com/johnellis/stalwart-migrator -> github.com/LINUXexpert-org/stalwart-migrator to match the repository location, so the module resolves under `go get`. The initial commit's .gitignore listed the compiled binary as a bare `stalwart-migrate` pattern, which Git matches at any depth -- so it also excluded the cmd/stalwart-migrate/ source directory, and the initial commit shipped without the CLI entrypoint. Anchor the pattern to the repo root as /stalwart-migrate and add the four missing files. go build, go vet, and go test ./... all pass. |
||
|
jcoffey-dev
|
719a945d64 |
Initial commit: stalwart-migrator design and scaffolding
In-place upgrade tool for Stalwart Mail Server (0.15.5 -> latest) with checkpointed rollback and post-migration validation. Design stage; see ARCHITECTURE.md. |