Fix the three defects that cost a production restore

A live migration on 2026-08-24 stopped a production mail server and then
discovered the host's stalwart-cli was 0.13.4 - present, but from when the
CLI shipped with the server, with no `apply` command. The migration needs
v1.0.2+ from the separately-versioned stalwartlabs/cli repository.

Recovery was closed in both directions. v0.16's recovery-mode boot had
already bumped the store schema to v6, so the 0.15.5 binary refused to
reopen it ("expected 5 or below, found 6"). Going forward needed
export.json, which this tool's own failure path had deleted - and
regenerating it required a settings dump from a live v0.15 instance that
could no longer start. The operator restored a day-old snapshot and lost a
day of mail across nine domains.

Three fixes:

1. preflight.CheckExternalTools verifies stalwart-cli exists and is v1.0.2
   or later, and that python3 runs - before anything is touched. Every fact
   needed to prevent this was available in under a second from a stopped
   state. Skipped for a patch upgrade, which invokes neither tool.

2. A failed run no longer deletes its work directory. Cleaning up on every
   exit path was right for a sandboxed rehearsal and catastrophic here:
   once the service is stopped the settings dump cannot be regenerated, so
   deleting it removes the only way forward. The failure now prints the
   resume command instead.

3. `run --resume <id>` continues an interrupted run. The checkpoint
   machinery existed but never engaged, because run created a new run every
   invocation - so a retry re-ran preflight against a binary already moved
   aside, and failed. Completed steps are skipped from the checkpoint.

Proven against a VM built to match the failure: stalwart-cli 0.15.5,
accounts and mail seeded.

  * preflight refused, service still active, mail still accepted
  * a stub CLI passing --version and failing apply left the run stopped
    with all eight inputs intact and the resume command printed
  * --resume carried it to a clean finish: five seconds of downtime,
    listeners regenerated, admin role restored, quotas rebuilt

That failure-path test is the one that should have run before production.
Every earlier test had stalwart-cli installed from the start, and the one
failure I did exercise happened to leave its artifacts behind.
This commit is contained in:
2026-08-23 23:20:47 -07:00
parent 5a4c175042
commit 9faa21f4f1
10 changed files with 370 additions and 11 deletions
+12 -2
View File
@@ -97,10 +97,20 @@ func rewriteExecStart(line, binaryPath, configPath string) (string, error) {
if configPath != "" {
replaced := false
for i := 0; i < len(fields)-1; i++ {
if fields[i] == "--config" || fields[i] == "-c" {
for i := 0; i < len(fields); i++ {
switch {
// Both spellings are real, and getting this wrong is not
// cosmetic: a production unit uses the equals form, and only
// matching the separated one appended a second --config, so
// the service started with two and used the wrong one - the
// v0.15 config, which v0.16 cannot read as a store descriptor.
case strings.HasPrefix(fields[i], "--config=") || strings.HasPrefix(fields[i], "-c="):
fields[i] = "--config=" + configPath
replaced = true
case (fields[i] == "--config" || fields[i] == "-c") && i+1 < len(fields):
fields[i+1] = configPath
replaced = true
i++
}
}
if !replaced {
+37
View File
@@ -128,3 +128,40 @@ func TestRewriteUnitHandlesMultipleExecStartLines(t *testing.T) {
t.Fatalf("an empty ExecStart= names no executable and should be refused, got:\n%s", got)
}
}
// A real production unit writes ExecStart=... --config=/path. Only matching
// the separated "--config /path" form appended a second flag, leaving the
// service started with two configs and using the v0.15 one - which v0.16
// cannot read as a store descriptor. Found while preparing a live
// migration, before it ran.
func TestRewriteUnitReplacesTheEqualsFormConfig(t *testing.T) {
unit := "[Service]\nExecStart=/opt/stalwart/bin/stalwart --config=/opt/stalwart/etc/config.toml\n"
got, err := RewriteUnit(unit, "/opt/stalwart/bin/stalwart", "/opt/stalwart/etc/config.json")
if err != nil {
t.Fatal(err)
}
if strings.Count(got, "--config") != 1 {
t.Errorf("expected exactly one --config argument, got:\n%s", got)
}
if !strings.Contains(got, "--config=/opt/stalwart/etc/config.json") {
t.Errorf("equals-form config not replaced:\n%s", got)
}
if strings.Contains(got, "config.toml") {
t.Errorf("the old config path survived:\n%s", got)
}
}
// The separated form must keep working; both spellings are real.
func TestRewriteUnitReplacesTheSeparatedFormConfig(t *testing.T) {
unit := "[Service]\nExecStart=/usr/local/bin/stalwart --config /etc/stalwart/config.toml\n"
got, err := RewriteUnit(unit, "/usr/local/bin/stalwart", "/etc/stalwart/config.json")
if err != nil {
t.Fatal(err)
}
if strings.Count(got, "--config") != 1 {
t.Errorf("expected exactly one --config argument, got:\n%s", got)
}
if !strings.Contains(got, "--config /etc/stalwart/config.json") {
t.Errorf("separated-form config not replaced:\n%s", got)
}
}