The repositories moved off LINUXexpert-org. Here that is not a
documentation change: the old organisation was the module path, so it is
declared in go.mod and repeated in every internal import.
Leaving it would have been worse than a stale link. GitHub redirects the
repository, but a go.mod whose module line disagrees with the path it was
fetched from is an error rather than a redirect, so `go get` on the new
address would have failed against the old declaration.
go.mod, 34 files of imports, and the repository links in README and
ARCHITECTURE. go mod tidy leaves go.sum untouched -- no dependency moved,
only our own path.
119 SPDX-FileCopyrightText headers and the README's licence line.
The distinction that matters here: LINUXexpert-org appears in this repository
in two completely different roles. As a copyright holder in the SPDX headers,
which is what changes, and as the GitHub organisation in the module path and 64
import statements, which does not -- the repository still lives at
github.com/LINUXexpert-org/stalwart-migrator, and rewriting that would not be a
licence change, it would break the build.
Both replacements are anchored to their copyright forms, so an import path
cannot match either. Import count is 64 before and after, and go.mod is
untouched.
LICENSE untouched: the FSF's copyright on the GPL text and the "<name of
author>" placeholders are not ours to edit.
go vet, go build and go test all clean.
`run` and `rehearse` always fetched migrate_v016.py from GitHub, so a mail
server with no route out could not be migrated at all - an air-gapped host,
or a clone deliberately cut off so it cannot renew certificates or deliver
queued mail for the domains it was copied from. --migration-script takes a
local copy instead, still checked against --migration-script-sha256 when one
is pinned.
Found while staging a production clone for a dress rehearsal: the clone has
no route out on purpose, and that is exactly the property that stops a copy
of a live mail server doing something in the real world.
A production rehearsal reported 12,182 settings not carried over by
migrate_v016.py. As a bare number that reads as an impossible amount of
manual reconstruction, and it is misleading. Snapshotting a v0.16.14 store
migrated from a real v0.15.5 showed what those settings actually are:
8547 regenerates server.blocked-ip
3337 shipped with v0.16 lookup.url-redirectors, lookup.trusted-domains,
spam-filter.list, spam-filter.rule,
spam-filter.dnsbl, lookup.surbl-hashbl
224 already carried server.listener, signature.* (DKIM)
293 NEED YOUR REVIEW queue.schedule, config.local-keys,
server.auto-ban, spam-filter.llm, queue.tls, ...
server.blocked-ip is auto-ban state that repopulates from live traffic. The
stock groups are data v0.16 provides itself - 2,084 MemoryLookupKey, 66
SpamRule and 18 SpamDnsblServer objects were already present in the migrated
store. DKIM came across as DkimSignature objects with private keys intact,
verified on that instance. So the real worklist is ~293 keys, not 12,182.
backup.UnmigratedReport.Classify encodes this and the rehearsal now reports
the categorised view. Rules match longest-prefix-first, because
server.blocked-ip is runtime state while server.auto-ban beside it is
configuration, and an unrecognized prefix defaults to "needs review" -
assuming an unknown setting is safe to ignore is the wrong default.
This also retired the lookup and spam-filter generators that were the
planned next step. v0.15's rules are stwt_rbl_senderscore_ip; v0.16's are
STWT_RBL_SENDERSCORE_IP - the same stock set, already installed. Generating
them from v0.15 would duplicate every rule and revert upstream updates, so
they were deliberately not written. The targets worth generating are the
small site-specific groups instead: queue.schedule, queue.tls,
session.auth, server.auto-ban.
No production data in this commit: the test fixture uses the real group
names and counts with example.com standing in for customer domains.
Ran the rehearsal read-only against a live production instance. It
completed, and two preflight checks were wrong in ways a test instance
could not have shown.
1. Store-backend detection missed the config entirely. A config generated
by `stalwart --init` declares `type = "rocksdb"` inside a
`[store.rocksdb]` section, which is what every test fixture here used.
The production config has no section headers at all and declares
`store.rocksdb.type = "rocksdb"` flat. Detection only matched a bare
`type` key, so it reported "no known store backend type found in config"
and left topology.store_backend empty.
That mattered more than a warning suggests: backup.Run treated an
unrecognized backend as a *skip*, so a real run would have continued
with no filesystem or database backup at all - the single artifact that
phase exists to produce, quietly absent. Flat dotted keys are now
detected, and an unrecognized backend is a hard failure rather than a
skip.
2. The cluster warning didn't say where it matched. It fires on any
occurrence of "cluster" anywhere in the config, which is the correct
bias - a missed cluster corrupts a shared store - but on the production
config the only match was inside the value of an unrelated setting,
leaving a whole config to search to establish that. It now names the
location and distinguishes a match in the setting name from one in its
value.
Both verified against the real config: store-backend now reports
"rocksdb (store.rocksdb)", and the cluster warning names the setting,
making a false positive dismissible at a glance.
No production data is in this commit: the fixtures use example.com and the
flat-key shape only. Coverage numbers from that run matched the earlier
scrubbed-corpus measurement exactly.
Chased down why a migrated instance had no working administrator. The
account authenticated fine and was refused every management call, and the
cause is that migrate_v016.py assigns every migrated account the User role
regardless of what it held before: an account that was `roles: ["admin"]`
in v0.15 comes out the far side as `roles: {"@type": "User"}`.
Ordinary users were never affected - User is what they had and what they
get - and their credentials, mail and mailboxes survive untouched. It is
specifically administrators who lose their privileges, which is a bad thing
to discover after cutting over.
The v0.16 shape came from the server's own schema document rather than the
published reference: GET /api/schema defines x:UserRoles as a multi-variant
type with variants User, Admin and Custom. Account is itself multi-variant,
so an upsert needs its own "@type" too - without it the server rejects the
operation outright ("upsert entry is missing `@type`").
applyplan.AccountRoleOperations restores roles from the principals dump,
emitting operations only for accounts whose role actually changes.
Rewriting every account would be a much larger blast radius for no benefit.
Where v0.15 listed several roles, admin wins - under-privileging an
administrator locks them out, which is the failure being fixed - and the
collapse is reported rather than done silently, as are roles with no known
v0.16 equivalent.
Verified end to end on the smoke VM: rehearse against the real 0.15.5 put
the role operation in the supplement, applying that supplement to a
migrated 0.16.14 whose admin was broken restored management access
(accounts=3), and alice and bob logged in over IMAPS with unchanged
credentials, read their mail, and accepted new SMTP delivery.
Also recorded: x:Account.domainId returns an internal id on v0.16, not a
domain name, so the post-migration directory comparison would read every
domain as missing. Resolving that needs an x:Domain/get call not yet
confirmed against the binary.
First piece of ARCHITECTURE.md 4.3's apply-plan, and the piece that decides
whether a migrated server answers at all: server.listener is not among the
settings the official converter carries, so a freshly migrated instance
binds nothing. Every other unmigrated setting degrades the server; this one
stops it being a server.
internal/applyplan maps server.listener.* onto x:NetworkListener objects and
reports its own coverage. Against the smoke instance that is 24 of 3,505
unmigrated keys - 0.7% - and the output says 0.7%, listing the largest
groups it did not touch. A plan covering a fraction while implying
completeness would be worse than no plan.
The wire format was confirmed against the binary, not the documentation.
The published schema reference gives NetworkListener.bind as a JSON array;
0.16.14 rejects that outright ("Invalid value for object property.
Properties: bind"). The encoding it accepts is a value-keyed set,
{"[::]:25": true}, found by applying a plan to a live recovery-mode 0.16.14
and reading it back with `stalwart-cli snapshot`. Only mappings confirmed
that way are in DefaultGenerators; managesieve -> manageSieve is the one
protocol whose spelling changes, and an unrecognized protocol is reported
and skipped rather than passed through to fail at apply time.
Operations are upserts matched on name, so a plan can be re-run - an
operator will run it more than once - and the supplement is applied after
export.json rather than merged into it, so a generated mapping can never
override one the official script got right.
Verified end to end: rehearse against a real 0.15.5 generated ten
listeners, `stalwart-cli apply` created all ten on a real 0.16.14 with zero
failures, a snapshot read them back with correct protocols, binds and TLS
flags, and re-applying reported 10 updated / 0 created / 0 failed.
Ran the converter against a real production settings corpus pulled from the
instance this tool is being built to migrate (secrets scrubbed on that host;
nothing sensitive transited). The result reframes what "a successful
migration" means:
total settings: 12401
NOT migrated: 12182 (98.2%)
actually migrated: 219 (1.8%)
Stalwart's own script reports this in an unmigrated.txt it writes beside its
output - and this tool was throwing that file away. Worse, RunSettingsConvert
never set cmd.Dir, so the script wrote unmigrated.txt into whatever directory
the operator happened to launch from, or failed the whole convert when that
directory wasn't writable. Both reproduced.
The largest groups left behind on production are spam-filter rules, DNSBLs,
trusted-domain and URL-redirector lookups, queue scheduling and TLS
settings - and server.listener. That last one explains something that had
been puzzling from an earlier smoke run: a freshly migrated 0.16 instance
answered on none of the ports the old one did, and served nothing but
/admin. Its listeners never migrated.
So:
- SettingsConvertOptions gains WorkDir, and the convert runs there. The
report lands somewhere known and an unwritable cwd can't fail the step.
- ReadUnmigratedReport parses it; UnmigratedReport.Summary renders the
largest groups first.
- The dry run records it as an "unmigrated-settings" artifact with a
checksum, notes the count in the checkpoint step, and prints it as a
standing warning rather than a footnote.
ARCHITECTURE 4.3 anticipated a "best-effort apply-plan" gap here. The gap is
not a few stragglers needing review; it is effectively the entire
configuration, and the tool has to say so where nobody can miss it.
Verified against the smoke VM: even a default `stalwart --init` instance
reports 3505 of ~3514 settings unmigrated, listeners included.
GPLv3's "How to Apply These Terms" asks for a notice in each source file;
this is the modern two-line SPDX form of it rather than the full paragraph.
82 files, including tests.
The blank line after the header is load-bearing. In Go a comment block
immediately preceding `package X` becomes the package doc comment, so
without the separator the SPDX lines would be absorbed into the doc for the
eleven packages whose doc.go (or main.go) opens with one, and `go doc` would
print them. Verified it doesn't.
Two changes that arrived together: the cutover phase (ARCHITECTURE.md 4.5)
is implemented, and the rollback phase is deleted. Recovery from a failed
migration is now explicitly the operator's own snapshot or backup, and out
of scope for this tool.
internal/cutover implements 4.5 as seven checkpointed steps: verify the
staged binary's version, install it, preserve and rewrite the service
definition, reload, start, wait for a healthy JMAP session, recalculate
quotas.
The unit is rewritten in place rather than generated from a template. An
operator's unit carries hardening options, limits and dependencies this
tool has no business having an opinion about, and regenerating it would
silently drop them. It repoints ExecStart (preserving systemd's -@:+!
prefix characters and every argument after the executable), updates
--config, and strips recovery-mode Environment lines - leaving
STALWART_RECOVERY_MODE=1 set would recovery-boot the service on every
restart, forever. It refuses on a unit with no ExecStart, and on an
Environment line mixing a recovery variable with others: a line it only
partly understands is one it must not edit.
Quota recalculation is the one step allowed to fail without failing the
phase. Its wire format is grounded in Stalwart's x:Task schema reference -
Task/set creating one AccountMaintenance per account with maintenanceType
recalculateQuota - but the upgrade guide only documents the WebUI path, so
two details remain inferred and are called out in stalwartapi/task.go:
whether the schema's "read-only" annotation on accountId/maintenanceType
means "immutable after creation", and whether a finished task simply leaves
the queue (TaskStatus documents Pending/Retry/Failed with no success
state). Warning rather than failing is the honest response to that
uncertainty, and stale counters are an accounting problem next to calling
for a restore of a machine that is otherwise migrated and serving mail.
Docker deployments are refused outright: cutting a container over means
pulling an image and recreating it, not swapping a binary.
On removing rollback. The implementation worked and was tested, and it was
removed because restoring bytes correctly is not the hard part. It copied
file contents and permissions and verified every restored file against a
manifest - and did not preserve ownership. Run as root, as this tool
requires, it would have produced a byte-perfect, checksum-verified,
root-owned data directory that Stalwart, running as its own user, could not
open, and it would have reported success. The PostgreSQL path was worse:
pg_dump without --clean emits CREATE TABLE + COPY, which fails replaying
into a database whose tables still exist, and the ON_ERROR_STOP=1 added so
a half-applied restore couldn't be reported as success turned that into a
hard failure. None of it had ever run against a real server. A filesystem
snapshot has none of these failure modes, because it never lost the
metadata to begin with.
So cutover's gate is no longer rollback.CanRollBack but an explicit
RecoveryPointConfirmed acknowledgement. That is an assertion, not a check -
this tool cannot verify someone else's snapshot - and its only value is
that nobody migrates a production mail server having never been asked the
question. Two consequences are accepted deliberately: restoring any
pre-migration recovery point discards mail delivered since, and a failed
migration now stops and reports rather than undoing itself.
What the tool still does to make a manual restore easier: the old binary is
preserved and never deleted, the original service definition is preserved
before the rewrite, the settings and principals dumps stay on disk, and
every artifact path and checksum stays in the checkpoint where `status
<run-id>` can print it.
Also removed: the `confirm` command stub and RollbackWindowClosed, whose
only purpose was closing a rollback window that no longer exists, and
checkpoint.PhaseRollback. Old state.json files still load - JSON ignores
the now-unknown field.
Still open, and recorded in 8: cutover ignores systemd drop-ins, so an
ExecStart or Environment override in stalwart.service.d/*.conf is invisible
to the rewrite - including the recovery variable it exists to strip;
nothing prevents concurrent runs on the same run-id; and nothing in this
repo has ever run against a real Stalwart, real systemd, or a real store.
Rename module github.com/johnellis/stalwart-migrator ->
github.com/LINUXexpert-org/stalwart-migrator to match the repository
location, so the module resolves under `go get`.
The initial commit's .gitignore listed the compiled binary as a bare
`stalwart-migrate` pattern, which Git matches at any depth -- so it also
excluded the cmd/stalwart-migrate/ source directory, and the initial
commit shipped without the CLI entrypoint. Anchor the pattern to the
repo root as /stalwart-migrate and add the four missing files.
go build, go vet, and go test ./... all pass.
In-place upgrade tool for Stalwart Mail Server (0.15.5 -> latest) with
checkpointed rollback and post-migration validation. Design stage; see
ARCHITECTURE.md.
jcoffey-dev