d0d2954ad15a6019a5ffd40ed098921ded4950a7
21
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 jcoffey-dev
|
e96e72bf79 |
Tell what a container inherits from what it overrides
Checked against a real stalwartlabs/stalwart image, `docker inspect` on an ordinary container reports User "stalwart", Entrypoint ["/usr/local/bin/stalwart"] and Cmd ["--config", "/etc/stalwart/config.json"] — all three inherited, none of them given. Two things followed from reading those as the operator's. A container user was listed as configuration a recreate would drop, so every container off the official image was refused as unrecreatable. That refusal lived in cutover, downstream of the stop, the settings conversion and the store migration: it arrived with mail down and data already moved, which is the failure issue #1 was filed for. Each of the three is now compared against `docker image inspect` of the image the container is on. Inherited values are left to the new image, whose own defaults are the ones that go with it. Overrides are carried: --user, --entrypoint, and the rest of an entrypoint as leading argv. Cmd and Entrypoint were not being read at all, so an overridden one was silently dropped — the exact loss the unsupported list exists to prevent. The recreatability question also moved into preflight, while the server is still running. Cutover asks it again, since the two are separated by the whole migration, but only one of them can refuse without cost. The other half: the recreated container is now started with `--config` pointing at the migrated config in the data volume. Left to the image's default command it came up on /etc/stalwart/config.json — a different volume, holding whatever the old version left there — so cutover would have produced a running server with nothing to do with the migration that preceded it. An overridden command and that --config are the same argv and cannot be merged honestly, so a container with one is refused and told why. The config is also chowned to whatever owns the data directory, before the recovery cycle opens it. The image runs as uid 2000 and this tool writes as root; §4.8 is the standing reminder that byte-perfect and unreadable is a way to report success. Found while checking @kaya-eu's field report in #1 against a real image. Their three manual migrations are where the config step comes from. |
||
|
jcoffey-dev
|
272439cf2a |
Wire the container path up, behind a flag that says what it is
Everything the container migration needs has landed a piece at a time and nothing called any of it. `run` now does: stage pulls and verifies an image instead of downloading a binary, the recovery cycle launches a throwaway container against the live container's own mounts, and cutover recreates it. Preflight's blanket refusal of docker goes with it -- what still refuses is specific to a container rather than to containers, which is compose and data that is not on a volume. It refuses without --container-path-unproven, and that flag is the honest part of this change. Every test drives a fake docker. That proves the right commands are assembled and proves nothing about whether a real image reads the config it is handed -- which is the exact limit ARCHITECTURE.md section 4.8 records about the rollback code that was deleted for being tested only against fakes. A doc note seemed too quiet for a tool that stops a mail server, so it is a flag nobody reaches without being told. The converted config reaches the container through the data volume. It is written under the host side of whichever mount covers --data-dir and named on the container side, because cutover recreates a container with the mounts it had and cannot invent a new one for a config file. --data-dir therefore names the path inside the container, which preflight already says when it matches no mount. PatchPaths stays unused, deliberately. Its documented purpose is pointing a rehearsal at a sandbox; a real container's dumped settings already carry container-side paths, because they come from the live server rather than from a file on this host. The preflight test that asserted docker was refused outright now asserts the replacement rather than being deleted -- "docker is allowed through here" is the thing that would be wrong to regress. Its fixture had to make --data-dir both a real host directory and one the fake container mounts, since disk-space stats it and container-data-volume wants it covered. README gains the container section and, at the top, the note that this is ihasmail's companion. |
||
|
jcoffey-dev
|
ad7c2d5135 |
Cut over a container, or refuse to for a reason
A container cannot be edited in place the way a unit file can, so cutting one over means rebuilding it. That makes silent loss the default failure: a container recreated without its capabilities, its custom network or its device mappings starts cleanly and is quietly not the server it was. Section 4.5 already answers this for a unit file -- it rewrites in place rather than regenerating, because a generated unit would drop hardening options this tool has no business having an opinion about, and it refuses to edit a line it only partly understands. The same rule applies here, where the whole definition has to be rebuilt: the parts this understands are carried across, and a container using anything else is refused by name rather than rebuilt without it. The list of what it looks for is conservative and not exhaustive, which is the safe direction: docker's HostConfig has far more fields, and one this does not know about is a reason not to be recreating that container at all. The old container is renamed, not removed, and nothing here prunes the old image. Together they are the container's manual restore path -- one command starts the previous container again -- which is as close to section 4.2's preserved binary as a container gets. The inspect output is preserved as an artifact before anything is replaced, for the reason the unit file is: an operator putting a machine back by hand should not also be reconstructing the definition from memory. Recovery-mode variables are stripped from the recreated container's environment. Leaving STALWART_RECOVERY_MODE set would recovery-boot on every restart, which is the same footgun the unit rewrite exists to prevent. Run now branches, with the health check and quota recalculation shared: those ask the same question whatever started the server. The binary path moved inside an else and is otherwise untouched -- no existing test needed editing, which is the evidence for that. The container path is opt-in through Options.Container, so a Docker deployment without it is still refused exactly as before. Nothing calls it yet; wiring `run` up and lifting preflight's refusal is what remains of #3, and ARCHITECTURE.md says so in both places it previously said Docker was refused outright. |
||
|
jcoffey-dev
|
0a6b3ad173 |
Stage a container image the way a binary is staged
The container deployment's answer to downloading a release: pull the image the operator named, then ask it what it is. That last part is the point of the phase, exactly as it is for a binary - the tag, the registry and the repository name are all assumptions about someone else's publishing process, and the image's own answer is the only thing that settles what arrived. The image is never derived from the running container by swapping its tag. That derivation is wrong for a digest-pinned image, wrong for a mirror and wrong for a fork, and being wrong here means pulling the wrong software into a mail server. It is named in full or the phase refuses. What comes back is the image's ID rather than the tag it arrived under. A tag can move between staging and cutover -- that is the whole reason latest is a hazard -- and running the tag later would run something other than what was verified here. VersionFromOutput is exported from preflight so both paths parse a version identically. Two copies of that regex could disagree about what they staged, which is a difference nobody would look for. One caveat recorded rather than hidden: asking an image its version means running it with --version, which assumes its entrypoint is the server and passes flags through. That has not been confirmed against a published Stalwart image, there being none to hand. If the assumption is wrong this fails loudly with the image's own output rather than staging something unverified, and the fallback tries the binary by name before giving up. SkipPull is for a host that loaded the image from a tarball, where a pull cannot work and its failure would say nothing useful. |
||
|
jcoffey-dev
|
6e7a05e478 |
Tell a container operator what stands in the way
Preflight refuses a container, and until now that was all it said. The two things that actually decide whether one could be migrated at all were never looked at, so an operator was refused without being told what to fix or what a manual migration would involve. It now inspects the container and reports three things. What image it is running, tag and digest kept apart because a moved tag makes them disagree and only one of them says what is really there. Whether compose manages it, which matters beyond this tool's current refusal: recreating a compose-managed container out from under compose leaves the container and the compose file disagreeing about what is deployed, and the next `compose up` reverts the migration. And whether the data is on a volume at all -- a container keeping it in its own writable layer loses it when the container is replaced, and replacing the container is what migrating it means, so that one is fatal in a way no later phase could recover from. All three are advisory under rehearse, on the same reasoning that already made the deployment check advisory there: rehearse never stops or recreates anything, and an operator doing this by hand needs these facts more than an automated run does. Preflight stays read-only. Preserving the container definition as an artifact belongs with the phase that replaces it, so it goes with cutover rather than here. One limitation this surfaced and does not fix: --data-dir naming a path inside the container breaks the host-side disk-space check, which stats it locally. Path translation is the next piece of work; the data-volume check says so when the path matches no mount. |
||
|
jcoffey-dev
|
edd8279743 |
Refuse a Docker deployment before stopping anything
Cutover already refused a container -- recreating one from a new image is not swapping a binary and rewriting a unit, and this tool does not automate it -- but it refused from cutover.Run, which run.go calls at line 324. The service is stopped at line 243. So the sequence on a container was: stop Stalwart, convert the settings, then discover the deployment cannot be cut over, return the error, and exit with mail still down. Reported as #1. The deployment kind is known in preflight, before anything has been touched, and that is now where it is acted on: docker is a blocking check. rehearse keeps working -- it never stops the service or cuts over, and telling an operator what the migration involves is most useful precisely when the tool cannot do it for them -- so it sets DeploymentCheckAdvisory, alongside the ToolCheckAdvisory it already set for the same reason. The second half is not docker's alone. Every return between the stop and the end of cutover returned with the service down; a failed settings conversion would have done the same to a systemd host. run now registers a restart on the way out, after the stop rather than before, so it only ever starts something this tool stopped. It does not claim to have recovered the migration -- a part-migrated store still needs --resume or the operator's recovery point -- it removes the narrower failure of exiting on a foreseeable error while the server it stopped stays stopped. |
||
|
jcoffey-dev
|
ce0e237051 |
Refuse a fallback-admin at preflight, not after the migration finishes
A dress rehearsal on a clone of production migrated cleanly and then failed all three post-cutover steps with 401: the health check, the quota rebuild, and the content comparison. The account was `admin` - a v0.15 [authentication.fallback-admin], not a directory account. v0.16 keeps its configuration in the store, so the block defining it does not survive, and the credential stops working the instant the migration completes. Every check passed beforehand, because on v0.15 that account authenticates perfectly well. The README has said to use a named account for a while; the tool now says it too, while nothing has been touched and changing it costs one flag. The migration itself is unaffected - it succeeds either way. What is lost is the ability to verify it afterwards, which is the part worth having. |
||
|
jcoffey-dev
|
b786e89b42 |
Take the target version from the binary when there is no route out
preflight asked the GitHub release API which version it was upgrading to, so a host with no internet failed there even with the target binary already on disk - and `run` never passed its own --target-binary down to preflight, so supplying one did not help. Read the version from the binary instead when one is given, and fail if it is not the version the run targets. Found by the first preflight against the production clone, which is deliberately cut off from the network. |
||
|
jcoffey-dev
|
e955a41d58 |
Fix the domain/tenant mismatch that failed the second live migration
The second production attempt failed during recovery-mode apply, with the
mail server already stopped and the store already at schema v6:
create Account restore-13: invalidForeignKey | Object id: Domain#d
v0.16 requires a tenant-scoped Account to sit on a Domain owned by that
same tenant, for its primary domain and for every alias. v0.15 imposed no
such rule, and migrate_v016.py carries the two facts over independently:
_build_domains sets a domain's memberTenantId only for domains declared as
their own `domain` principal with a `tenant`, while _build_user sets the
account's from the account's own record. A domain that exists only inside
an email address is inferred, gets no tenant, and every tenant-scoped
account using it is then rejected.
Established by reproduction rather than inference: a synthetic v0.15
principal dump, run through the unpatched upstream converter and applied to
a real 0.16.14 in recovery mode, reproduces the error character for
character - the `#d` is the server's own object id for the offending
domain, not a plan client-id. The same harness establishes which directions
are constrained: a tenant-scoped account on a tenant-less domain or on
another tenant's domain is rejected; a global account on a tenant-owned
domain is accepted.
- applyplan.ReconcileDomainTenants repairs the plan between convert and
apply. Where a tenant-less domain is used only by accounts of one
tenant, the domain adopts that tenant - the sole assignment that both
applies and keeps every account. Where accounts genuinely disagree it
changes nothing and reports why, because forcing such a plan through
would mean dropping mailboxes.
- stalwartapi.FetchTenantLayout maps tenant membership over the 0.15 REST
API and predicts the outcome with the same rule the server enforces, so
preflight either warns about the domains that will adopt a tenant or
fails - while the service is still running.
- The plan is parsed generically rather than through the typed Operation.
A real export.json mixes shapes: `create` maps a client-id to an object,
`update` carries a flat one. The typed form failed on the first `update`
line, found by running against actual converter output. Numbers decode
as json.Number so a 10 GiB quota is not rewritten as 1.073741824e+10.
Corrects the record: the previous commit claimed the converter emits every
Account with `tenantId: null` and made preflight refuse every multi-tenant
install on that basis. The field is memberTenantId, the converter does
populate it, and the export had been inspected for a key no version of the
script ever writes. The refusal is now narrowed to what v0.16 genuinely
cannot represent.
The same fix has been prepared for migrate_v016.py upstream. The tool
downloads that script rather than vendoring it, so the repair stays here
until a released version carries it, and is a no-op on a consistent plan.
|
||
|
jcoffey-dev
|
c29140b6b3 |
Refuse multi-tenant installs in preflight, not after the service is stopped
A second live attempt failed in the same shape as the first: preflight
clean, settings dumped and converted, then a failure during recovery-mode
migration with the mail server already stopped.
created Tenant (1)
created Domain (9)
create Account restore-13: invalidForeignKey | Object id: Domain#d
migrate_v016.py carries the Tenant and the Domains but emits every Account
with a null tenantId, so the account references a tenant-owned domain while
belonging to no tenant and the foreign key is rejected. That is Stalwart's
converter and there is nothing this tool can do about it: a multi-tenant
install has to be migrated by hand until the converter handles tenants.
What this tool got wrong was the timing. Tenant principals are one API call
away and were readable the entire time the server was running. Preflight now
queries them and fails before anything is touched, with an explanation of
exactly what would otherwise fail and when.
This is the same lesson as the stalwart-cli check: knowable in advance,
discovered after a production mail server had been stopped, twice. Any
dependency of the conversion belongs in preflight, not in the phase that
consumes it.
Also makes the external-tool checks advisory during `rehearse`, which never
invokes stalwart-cli - refusing to run read-only reconnaissance because the
operator lacks a tool that reconnaissance would tell them to get was
backwards.
|
||
|
jcoffey-dev
|
9faa21f4f1 |
Fix the three defects that cost a production restore
A live migration on 2026-08-24 stopped a production mail server and then
discovered the host's stalwart-cli was 0.13.4 - present, but from when the
CLI shipped with the server, with no `apply` command. The migration needs
v1.0.2+ from the separately-versioned stalwartlabs/cli repository.
Recovery was closed in both directions. v0.16's recovery-mode boot had
already bumped the store schema to v6, so the 0.15.5 binary refused to
reopen it ("expected 5 or below, found 6"). Going forward needed
export.json, which this tool's own failure path had deleted - and
regenerating it required a settings dump from a live v0.15 instance that
could no longer start. The operator restored a day-old snapshot and lost a
day of mail across nine domains.
Three fixes:
1. preflight.CheckExternalTools verifies stalwart-cli exists and is v1.0.2
or later, and that python3 runs - before anything is touched. Every fact
needed to prevent this was available in under a second from a stopped
state. Skipped for a patch upgrade, which invokes neither tool.
2. A failed run no longer deletes its work directory. Cleaning up on every
exit path was right for a sandboxed rehearsal and catastrophic here:
once the service is stopped the settings dump cannot be regenerated, so
deleting it removes the only way forward. The failure now prints the
resume command instead.
3. `run --resume <id>` continues an interrupted run. The checkpoint
machinery existed but never engaged, because run created a new run every
invocation - so a retry re-ran preflight against a binary already moved
aside, and failed. Completed steps are skipped from the checkpoint.
Proven against a VM built to match the failure: stalwart-cli 0.15.5,
accounts and mail seeded.
* preflight refused, service still active, mail still accepted
* a stub CLI passing --version and failing apply left the run stopped
with all eight inputs intact and the resume command printed
* --resume carried it to a clean finish: five seconds of downtime,
listeners regenerated, admin role restored, quotas rebuilt
That failure-path test is the one that should have run before production.
Every earlier test had stalwart-cli installed from the start, and the one
failure I did exercise happened to leave its artifacts behind.
|
||
|
jcoffey-dev
|
5a4c175042 |
Implement run: the migration pipeline, end to end
The phases have all existed for a while; nothing chained them. The order
here is the one arrived at by performing this migration by hand against a
clone of production before writing it down:
preflight -> stage -> dump -> preserve binary -> STOP ->
convert -> supplement -> recovery-mode migration -> cutover -> START
The dump runs before the stop because it reads settings over the admin API,
and a stopped server has no admin API. Everything from the stop to the end
of cutover is downtime.
internal/stage fills the last missing phase (4.3): resolve the release,
take the x86_64 linux-gnu server build and refuse to substitute another,
verify a pinned checksum if one was given, extract the binary - refusing
any archive entry that isn't a regular file, since a tarball is untrusted
input - and confirm the result reports the version its tag claimed.
Everything upstream of that last check is an assumption about someone
else's release process.
Two gates, separate on purpose. --yes is about intent. --recovery-point-
confirmed is a claim about the world: this tool cannot undo a migration
(4.8) and cannot check whether a snapshot exists, so a run that proceeded
without the operator asserting one would be proceeding on a hope.
Verified end to end against a real Stalwart 0.15.5 with email-style account
names, a named admin account, and seeded mail:
MIGRATION COMPLETE. Mail was down for 6s.
Every cutover step green, including recalculate-quotas ("rebuilt disk
quotas for 2 account(s)") - the first time the x:Task wire format inferred
from Stalwart's schema reference has actually been exercised. It works,
now that endpoint discovery and role restoration make it reachable. After
the migration the named admin still administers, alice logs in with
unchanged credentials to the same four messages, and new SMTP delivery is
accepted.
Both refusal gates were tested, as was the failure path: an apply that
fails leaves the run stopped with the store part-migrated, and the error
says to restore the recovery point rather than restart the old version
against it.
|
||
|
jcoffey-dev
|
a69f0bbff1 |
Fix two preflight defects a live production rehearsal found
Ran the rehearsal read-only against a live production instance. It completed, and two preflight checks were wrong in ways a test instance could not have shown. 1. Store-backend detection missed the config entirely. A config generated by `stalwart --init` declares `type = "rocksdb"` inside a `[store.rocksdb]` section, which is what every test fixture here used. The production config has no section headers at all and declares `store.rocksdb.type = "rocksdb"` flat. Detection only matched a bare `type` key, so it reported "no known store backend type found in config" and left topology.store_backend empty. That mattered more than a warning suggests: backup.Run treated an unrecognized backend as a *skip*, so a real run would have continued with no filesystem or database backup at all - the single artifact that phase exists to produce, quietly absent. Flat dotted keys are now detected, and an unrecognized backend is a hard failure rather than a skip. 2. The cluster warning didn't say where it matched. It fires on any occurrence of "cluster" anywhere in the config, which is the correct bias - a missed cluster corrupts a shared store - but on the production config the only match was inside the value of an unrelated setting, leaving a whole config to search to establish that. It now names the location and distinguishes a match in the setting name from one in its value. Both verified against the real config: store-backend now reports "rocksdb (store.rocksdb)", and the cluster warning names the setting, making a false positive dismissible at a glance. No production data is in this commit: the fixtures use example.com and the flat-key shape only. Coverage numbers from that run matched the earlier scrubbed-corpus measurement exactly. |
||
|
jcoffey-dev
|
1684c88877 |
Resolve v0.16 domain ids to names before comparing directories
x:Account.domainId is an internal id on v0.16 ("b"), not a domain name. A
pre-migration snapshot taken from a v0.15 instance records names
("smoke.test"), so the post-migration directory comparison compared ids
against names and would have reported every domain as having vanished -
a false alarm on the check whose whole job is proving nothing was lost.
The client now resolves them with x:Domain/query + x:Domain/get in a single
request, using a JMAP back-reference (RFC 8620 3.7). Confirmed against a
live 0.16.14 before being written:
["x:Domain/get", {"list":[{"name":"smoke.test","id":"b"}]}, "g"]
An id that can't be resolved is kept as-is - a domain that can't be named is
still a domain that exists - but a failure of the resolution call itself is
an error rather than a silent fallback, since quietly comparing ids against
names is precisely the bug being fixed.
Verified against the live migrated instance: the snapshot that reported
domains=[b] now reports domains=[smoke.test], matching what the
pre-migration snapshot recorded.
|
||
|
jcoffey-dev
|
3e155fa42c |
Fix three defects a full VM migration exposed
Ran a complete 0.15.5 -> 0.16.14 migration of the smoke VM, driving the
phases in the order the real pipeline will. It worked - all mail intact and
readable afterwards, all ten listeners up, cutover executed for the first
time ever and checkpoint resume exercised - and it exposed three defects.
1. The converted config was installed root-owned while the service runs as
its own user. Stalwart crash-looped 28 times on "Failed to read data
store settings: Permission denied", minutes after the mistake and
nowhere near it. This is the same ownership trap that retired the
rollback implementation, in a new place: writing files as root is the
natural thing for a tool running as root to do, and it is wrong every
time the service is not root.
Cutover now installs the config itself, copying ownership and mode from
the config being replaced.
2. v0.16.14 does not serve /api - the endpoint stalwartapi assumed.
Confirmed against a fully migrated, fully configured, serving instance
rather than a sandbox: /api, /api/principal and /jmap/ all 404. The JMAP
endpoint is the one the session document advertises, which is what RFC
8620 discovery is for.
The client now discovers it, re-basing the advertised path onto the
operator's host: a real instance advertises its canonical public URL
("https://mail.smoke.test/jmap/") which frequently isn't reachable from
where this tool runs. The session is authoritative about the path; the
operator is authoritative about the host.
3. Dispatching on the urn:stalwart:jmap capability was wrong, because
NEITHER version advertises it - not 0.15.5, and not a fully migrated
0.16.14. That sent 0.16 instances down the 0.15 REST path where every
call 404s. The client probes what the instance actually serves instead.
Less elegant than a declared capability, with the advantage of being
true.
Also: a JMAP "forbidden" now explains itself. An account holding the admin
role before the migration was refused x:Account/query afterwards, and a
bare "forbidden" gives an operator nowhere to start. Whether the role
failed to carry or v0.16 wants different permissions was not isolated, and
that question is recorded as open - it gates quota recalculation and any
post-migration validation.
Verified against both live instances: the 0.15.5 reports 3 accounts and its
domain over REST, and the migrated 0.16.14 routes to JMAP, finds the right
endpoint, and returns the explained refusal.
|
||
|
jcoffey-dev
|
4568f9abbf |
Capture the pre-migration snapshot from 0.15.x, and stop claiming counts match when none were compared
Found by running preflight against a real Stalwart 0.15.5 in a VM. Two defects, the second worse than the first. 1. AccountSnapshot could not read the version this tool migrates FROM. 0.15.5 advertises no urn:stalwart:jmap capability and POST /api returns 404 - the JMAP management API and x:Account are 0.16 features. 0.15.x exposes a REST API at GET /api/principal instead. So preflight's account-snapshot check warned and moved on, and every run against a real source instance had no "before" data at all. AccountSnapshot now dispatches on the capability the session document advertises - a positive signal, not an inference from a failed call - and internal/stalwartapi/principal.go implements the 0.15.x REST path, including its 1-based page/limit pagination so an install larger than one page isn't silently truncated. 2. With no "before" counts, the content-integrity comparison iterated an empty map, checked nothing, and reported "all message counts match". That is the strongest claim this tool makes - ARCHITECTURE 4.7 calls it the actual no-data-loss guarantee - made vacuously, and it would have passed on a migration that lost every message. The comparison now derives its account set from whatever the source could report, verifies every account and domain survived either way, and carries MessageCountsCompared so the report says plainly "MESSAGE COUNTS NOT COMPARED ... no-data-loss is NOT verified here" rather than implying otherwise. What can and cannot be checked across the 0.15/0.16 boundary, now that a real server has answered: 0.15.x has no per-mailbox message count at any endpoint, and the impersonation login 0.16 offers returns 401 there, so before/after message counts are impossible for the boundary migration this tool exists for. Both versions do report per-account used quota (usedQuota in 0.15's REST list, usedDiskQuota on 0.16's x:Account), so that is captured on both sides. It is recorded and reported, not asserted on: 4.5 notes the 0.16 migration resets quotas to zero pending recalculation, so comparing those bytes across the boundary would be a false alarm generator. Test servers across preflight, validate and stalwartapi now advertise urn:stalwart:jmap, since they stand in for 0.16 instances and that capability is what says so. Verified end to end against the smoke VM: all nine preflight checks pass, and the checkpoint records 2 accounts, 1 domain and per-account used quota where it previously recorded nothing. |
||
|
jcoffey-dev
|
4b0bec8956 |
Add SPDX headers to every Go file
GPLv3's "How to Apply These Terms" asks for a notice in each source file; this is the modern two-line SPDX form of it rather than the full paragraph. 82 files, including tests. The blank line after the header is load-bearing. In Go a comment block immediately preceding `package X` becomes the package doc comment, so without the separator the SPDX lines would be absorbed into the doc for the eleven packages whose doc.go (or main.go) opens with one, and `go doc` would print them. Verified it doesn't. |
||
|
jcoffey-dev
|
7e04351b0f |
Add cutover; drop rollback in favour of operator-provided recovery
Two changes that arrived together: the cutover phase (ARCHITECTURE.md 4.5) is implemented, and the rollback phase is deleted. Recovery from a failed migration is now explicitly the operator's own snapshot or backup, and out of scope for this tool. internal/cutover implements 4.5 as seven checkpointed steps: verify the staged binary's version, install it, preserve and rewrite the service definition, reload, start, wait for a healthy JMAP session, recalculate quotas. The unit is rewritten in place rather than generated from a template. An operator's unit carries hardening options, limits and dependencies this tool has no business having an opinion about, and regenerating it would silently drop them. It repoints ExecStart (preserving systemd's -@:+! prefix characters and every argument after the executable), updates --config, and strips recovery-mode Environment lines - leaving STALWART_RECOVERY_MODE=1 set would recovery-boot the service on every restart, forever. It refuses on a unit with no ExecStart, and on an Environment line mixing a recovery variable with others: a line it only partly understands is one it must not edit. Quota recalculation is the one step allowed to fail without failing the phase. Its wire format is grounded in Stalwart's x:Task schema reference - Task/set creating one AccountMaintenance per account with maintenanceType recalculateQuota - but the upgrade guide only documents the WebUI path, so two details remain inferred and are called out in stalwartapi/task.go: whether the schema's "read-only" annotation on accountId/maintenanceType means "immutable after creation", and whether a finished task simply leaves the queue (TaskStatus documents Pending/Retry/Failed with no success state). Warning rather than failing is the honest response to that uncertainty, and stale counters are an accounting problem next to calling for a restore of a machine that is otherwise migrated and serving mail. Docker deployments are refused outright: cutting a container over means pulling an image and recreating it, not swapping a binary. On removing rollback. The implementation worked and was tested, and it was removed because restoring bytes correctly is not the hard part. It copied file contents and permissions and verified every restored file against a manifest - and did not preserve ownership. Run as root, as this tool requires, it would have produced a byte-perfect, checksum-verified, root-owned data directory that Stalwart, running as its own user, could not open, and it would have reported success. The PostgreSQL path was worse: pg_dump without --clean emits CREATE TABLE + COPY, which fails replaying into a database whose tables still exist, and the ON_ERROR_STOP=1 added so a half-applied restore couldn't be reported as success turned that into a hard failure. None of it had ever run against a real server. A filesystem snapshot has none of these failure modes, because it never lost the metadata to begin with. So cutover's gate is no longer rollback.CanRollBack but an explicit RecoveryPointConfirmed acknowledgement. That is an assertion, not a check - this tool cannot verify someone else's snapshot - and its only value is that nobody migrates a production mail server having never been asked the question. Two consequences are accepted deliberately: restoring any pre-migration recovery point discards mail delivered since, and a failed migration now stops and reports rather than undoing itself. What the tool still does to make a manual restore easier: the old binary is preserved and never deleted, the original service definition is preserved before the rewrite, the settings and principals dumps stay on disk, and every artifact path and checksum stays in the checkpoint where `status <run-id>` can print it. Also removed: the `confirm` command stub and RollbackWindowClosed, whose only purpose was closing a rollback window that no longer exists, and checkpoint.PhaseRollback. Old state.json files still load - JSON ignores the now-unknown field. Still open, and recorded in 8: cutover ignores systemd drop-ins, so an ExecStart or Environment override in stalwart.service.d/*.conf is invisible to the rewrite - including the recovery variable it exists to strip; nothing prevents concurrent runs on the same run-id; and nothing in this repo has ever run against a real Stalwart, real systemd, or a real store. |
||
|
jcoffey-dev
|
ade9906275 |
Implement internal/rollback and the service control it needs
Rollback was the one phase gating everything else: `run` without --dry-run refused because this tool could not undo a cutover it had committed to. That reason is now gone, and the refusal has narrowed to the fact that there is no real cutover to undo yet. internal/rollback implements ARCHITECTURE.md 4.8 as eight checkpointed steps under PhaseRollback: verify-backup, stop-service, preserve-failed-state, restore-data, restore-binary, restore-service-config, start-service, verify-rollback. Three things depart from what 4.8 specified, each for a reason: - The backup is re-verified against its manifest *before* the service is stopped, which the design didn't call out. Finding a corrupt backup is survivable while the failed instance is still up, and unsurvivable once its data directory has been moved aside. - BuildPlan is separate from Run, so every reason to refuse (closed rollback window, FoundationDB, no recorded backup, unknown deployment kind, missing database credentials) is found before anything is touched. The CLI prints that resolved plan and acts only with --yes. - The restore is re-verified against the same manifest after writing. A restore that put back truncated bytes and reported success would be worse than one that failed outright. Nothing from the failed attempt is deleted: the half-migrated data directory and the displaced binary are moved to .failed-<run-id> names, so a retry after the underlying issue is fixed still has both the evidence and the artifacts. Afterwards a reduced validation suite runs against the *restored* instance (version, reachability, directory counts) rather than assuming the restore worked. internal/service is a new package holding the systemd/Docker control this needs. It's separate rather than living inside internal/rollback because cutover will need the identical operations, and because the commands that can take mail delivery down belong in one auditable place - the same reasoning that makes stalwartapi the only thing speaking JMAP. preflight.DeploymentKind is now a type alias for service.Kind so detection and control can't drift apart. Its Active() reads `systemctl is-active`'s output rather than its exit status: systemctl exits non-zero for every non-active state, so exit-status logic would make "inactive" - the answer a rollback most needs - look like a failure to read the state at all. Also fixes a pre-existing bug in `status`: Go's flag package stops parsing at the first positional argument, so `status <run-id> --state-dir X` looked the run up in the default directory and reported it missing. `rollback` would have inherited the same footgun on a command whose flags decide what gets overwritten. Still open: `confirm` cannot set RollbackWindowClosed. Rollback honours the flag and refuses when it's set, but closing the window is the point of no return for the backups this restores from, so it should land with the retention policy 6 describes rather than before it. Verified end to end against a fake systemd deployment: half-migrated data restored to its original contents, failed state preserved, old binary reinstalled and reporting 0.15.5, unit restarted, and a re-run of the completed rollback inert. |
||
|
jcoffey-dev
|
5566eed1c8 |
Fix module path and restore cmd/ omitted by gitignore
Rename module github.com/johnellis/stalwart-migrator -> github.com/LINUXexpert-org/stalwart-migrator to match the repository location, so the module resolves under `go get`. The initial commit's .gitignore listed the compiled binary as a bare `stalwart-migrate` pattern, which Git matches at any depth -- so it also excluded the cmd/stalwart-migrate/ source directory, and the initial commit shipped without the CLI entrypoint. Anchor the pattern to the repo root as /stalwart-migrate and add the four missing files. go build, go vet, and go test ./... all pass. |
||
|
jcoffey-dev
|
719a945d64 |
Initial commit: stalwart-migrator design and scaffolding
In-place upgrade tool for Stalwart Mail Server (0.15.5 -> latest) with checkpointed rollback and post-migration validation. Design stage; see ARCHITECTURE.md. |