Add cutover; drop rollback in favour of operator-provided recovery
Two changes that arrived together: the cutover phase (ARCHITECTURE.md 4.5) is implemented, and the rollback phase is deleted. Recovery from a failed migration is now explicitly the operator's own snapshot or backup, and out of scope for this tool. internal/cutover implements 4.5 as seven checkpointed steps: verify the staged binary's version, install it, preserve and rewrite the service definition, reload, start, wait for a healthy JMAP session, recalculate quotas. The unit is rewritten in place rather than generated from a template. An operator's unit carries hardening options, limits and dependencies this tool has no business having an opinion about, and regenerating it would silently drop them. It repoints ExecStart (preserving systemd's -@:+! prefix characters and every argument after the executable), updates --config, and strips recovery-mode Environment lines - leaving STALWART_RECOVERY_MODE=1 set would recovery-boot the service on every restart, forever. It refuses on a unit with no ExecStart, and on an Environment line mixing a recovery variable with others: a line it only partly understands is one it must not edit. Quota recalculation is the one step allowed to fail without failing the phase. Its wire format is grounded in Stalwart's x:Task schema reference - Task/set creating one AccountMaintenance per account with maintenanceType recalculateQuota - but the upgrade guide only documents the WebUI path, so two details remain inferred and are called out in stalwartapi/task.go: whether the schema's "read-only" annotation on accountId/maintenanceType means "immutable after creation", and whether a finished task simply leaves the queue (TaskStatus documents Pending/Retry/Failed with no success state). Warning rather than failing is the honest response to that uncertainty, and stale counters are an accounting problem next to calling for a restore of a machine that is otherwise migrated and serving mail. Docker deployments are refused outright: cutting a container over means pulling an image and recreating it, not swapping a binary. On removing rollback. The implementation worked and was tested, and it was removed because restoring bytes correctly is not the hard part. It copied file contents and permissions and verified every restored file against a manifest - and did not preserve ownership. Run as root, as this tool requires, it would have produced a byte-perfect, checksum-verified, root-owned data directory that Stalwart, running as its own user, could not open, and it would have reported success. The PostgreSQL path was worse: pg_dump without --clean emits CREATE TABLE + COPY, which fails replaying into a database whose tables still exist, and the ON_ERROR_STOP=1 added so a half-applied restore couldn't be reported as success turned that into a hard failure. None of it had ever run against a real server. A filesystem snapshot has none of these failure modes, because it never lost the metadata to begin with. So cutover's gate is no longer rollback.CanRollBack but an explicit RecoveryPointConfirmed acknowledgement. That is an assertion, not a check - this tool cannot verify someone else's snapshot - and its only value is that nobody migrates a production mail server having never been asked the question. Two consequences are accepted deliberately: restoring any pre-migration recovery point discards mail delivered since, and a failed migration now stops and reports rather than undoing itself. What the tool still does to make a manual restore easier: the old binary is preserved and never deleted, the original service definition is preserved before the rewrite, the settings and principals dumps stay on disk, and every artifact path and checksum stays in the checkpoint where `status <run-id>` can print it. Also removed: the `confirm` command stub and RollbackWindowClosed, whose only purpose was closing a rollback window that no longer exists, and checkpoint.PhaseRollback. Old state.json files still load - JSON ignores the now-unknown field. Still open, and recorded in 8: cutover ignores systemd drop-ins, so an ExecStart or Environment override in stalwart.service.d/*.conf is invisible to the rewrite - including the recovery variable it exists to strip; nothing prevents concurrent runs on the same run-id; and nothing in this repo has ever run against a real Stalwart, real systemd, or a real store.
This commit is contained in:
@@ -1,7 +1,8 @@
|
||||
// Command stalwart-migrate drives an in-place Stalwart Mail Server upgrade
|
||||
// (0.15.5 -> latest) through preflight checks, a defense-in-depth backup,
|
||||
// a checkpointed migration, and post-migration validation, with rollback
|
||||
// available at every step. See ARCHITECTURE.md for the full design.
|
||||
// a checkpointed migration, and post-migration validation. Recovery from a
|
||||
// failed migration is the operator's own snapshot or backup and is out of
|
||||
// scope for this tool - see ARCHITECTURE.md §4.8.
|
||||
package main
|
||||
|
||||
import (
|
||||
@@ -23,10 +24,6 @@ func main() {
|
||||
err = runRun(os.Args[2:])
|
||||
case "status":
|
||||
err = runStatus(os.Args[2:])
|
||||
case "rollback":
|
||||
err = runRollback(os.Args[2:])
|
||||
case "confirm":
|
||||
err = fmt.Errorf("not implemented yet: see internal/checkpoint")
|
||||
case "report":
|
||||
err = fmt.Errorf("not implemented yet: see internal/validate")
|
||||
default:
|
||||
@@ -47,7 +44,5 @@ commands:
|
||||
preflight run read-only checks and print the migration plan
|
||||
run --dry-run: simulate and validate against a sandbox (real cutover isn't implemented yet)
|
||||
status show the state of an in-progress or completed run
|
||||
rollback restore the pre-migration backup for a given run (prints the plan; --yes to act)
|
||||
confirm close the rollback window for a completed run
|
||||
report print the validation report for a run`)
|
||||
}
|
||||
|
||||
@@ -1,159 +0,0 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"context"
|
||||
"flag"
|
||||
"fmt"
|
||||
"net/http"
|
||||
"os"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"github.com/LINUXexpert-org/stalwart-migrator/internal/backup"
|
||||
"github.com/LINUXexpert-org/stalwart-migrator/internal/checkpoint"
|
||||
"github.com/LINUXexpert-org/stalwart-migrator/internal/rollback"
|
||||
"github.com/LINUXexpert-org/stalwart-migrator/internal/service"
|
||||
)
|
||||
|
||||
// splitRunID pulls the run-id out of args wherever it appears, so
|
||||
// `rollback <run-id> --data-dir X` works as naturally as
|
||||
// `rollback --data-dir X <run-id>`. Go's flag package stops parsing at the
|
||||
// first positional argument, which would otherwise make the obvious
|
||||
// invocation order silently drop every flag after the run-id - on a command
|
||||
// whose flags decide what gets overwritten, that is not a failure mode
|
||||
// worth living with. Tokens consumed as a flag's value are skipped by
|
||||
// asking the FlagSet itself which flags take one, rather than by
|
||||
// maintaining a second list of them here.
|
||||
func splitRunID(fs *flag.FlagSet, args []string) (runID string, rest []string) {
|
||||
rest = make([]string, 0, len(args))
|
||||
for i := 0; i < len(args); i++ {
|
||||
arg := args[i]
|
||||
if arg == "--" {
|
||||
rest = append(rest, args[i:]...)
|
||||
break
|
||||
}
|
||||
if strings.HasPrefix(arg, "-") {
|
||||
rest = append(rest, arg)
|
||||
name := strings.TrimLeft(arg, "-")
|
||||
if !strings.Contains(arg, "=") && takesValue(fs, name) && i+1 < len(args) {
|
||||
i++
|
||||
rest = append(rest, args[i])
|
||||
}
|
||||
continue
|
||||
}
|
||||
if runID == "" {
|
||||
runID = arg
|
||||
continue
|
||||
}
|
||||
rest = append(rest, arg) // a second positional: let Parse report it
|
||||
}
|
||||
return runID, rest
|
||||
}
|
||||
|
||||
func takesValue(fs *flag.FlagSet, name string) bool {
|
||||
f := fs.Lookup(name)
|
||||
if f == nil {
|
||||
return false
|
||||
}
|
||||
boolFlag, ok := f.Value.(interface{ IsBoolFlag() bool })
|
||||
return !ok || !boolFlag.IsBoolFlag()
|
||||
}
|
||||
|
||||
// runRollback implements `stalwart-migrate rollback <run-id>`: put the
|
||||
// instance back the way it was before the named run touched it
|
||||
// (ARCHITECTURE.md §4.8).
|
||||
//
|
||||
// This is the only command that stops a running mail server and overwrites
|
||||
// a live data directory, so it always prints the resolved plan first and
|
||||
// refuses to act without --yes. Everything the plan needs comes from the
|
||||
// run's own checkpoint - which backup, which manifest, which preserved
|
||||
// binary - so an operator rolling back days after the fact doesn't have to
|
||||
// remember any of it. The flags below exist to supply what the checkpoint
|
||||
// deliberately doesn't store (database credentials) or to override a
|
||||
// detection that was wrong.
|
||||
func runRollback(args []string) error {
|
||||
fs := flag.NewFlagSet("rollback", flag.ExitOnError)
|
||||
stateDir := fs.String("state-dir", checkpoint.DefaultBaseDir, "directory runs are checkpointed in")
|
||||
dataDir := fs.String("data-dir", "/var/lib/stalwart", "stalwart data directory to restore into (embedded backends)")
|
||||
binaryPath := fs.String("binary", "/usr/local/bin/stalwart", "path the preserved old binary is reinstalled to")
|
||||
serviceUnitPath := fs.String("service-unit", "", "path a preserved systemd unit or Compose file is restored to (only used if the run recorded one)")
|
||||
|
||||
deployment := fs.String("deployment", "", `override how the service is controlled: "systemd" or "docker" (default: whatever preflight detected for this run)`)
|
||||
unitName := fs.String("unit", "stalwart", "systemd unit name")
|
||||
containerName := fs.String("container", "stalwart", "docker container name")
|
||||
stopTimeout := fs.Duration("stop-timeout", 60*time.Second, "how long to wait for the service to actually stop")
|
||||
startTimeout := fs.Duration("start-timeout", 60*time.Second, "how long to wait for the restored service to come back")
|
||||
|
||||
adminURL := fs.String("admin-url", "", "base URL for the restored instance's admin/JMAP API, for post-rollback verification")
|
||||
adminUser := fs.String("admin-user", "", "admin username")
|
||||
adminPassword := fs.String("admin-password", os.Getenv("STALWART_MIGRATE_ADMIN_PASSWORD"),
|
||||
"admin password (or set STALWART_MIGRATE_ADMIN_PASSWORD)")
|
||||
verifyTimeout := fs.Duration("verify-timeout", 60*time.Second, "how long to wait for the restored instance to answer")
|
||||
|
||||
dbHost := fs.String("db-host", "", "external database host (postgresql/mysql backends)")
|
||||
dbPort := fs.String("db-port", "", "external database port")
|
||||
dbName := fs.String("db-name", "", "external database name")
|
||||
dbUser := fs.String("db-user", "", "external database user")
|
||||
dbPassword := fs.String("db-password", os.Getenv("STALWART_MIGRATE_DB_PASSWORD"),
|
||||
"external database password (or set STALWART_MIGRATE_DB_PASSWORD)")
|
||||
sqlDump := fs.String("sql-dump", "", "override the dump file to replay (default: the one this run recorded)")
|
||||
|
||||
yes := fs.Bool("yes", false, "actually perform the rollback; without it, the plan is printed and nothing is touched")
|
||||
|
||||
runID, rest := splitRunID(fs, args)
|
||||
if err := fs.Parse(rest); err != nil {
|
||||
return err
|
||||
}
|
||||
if runID == "" || fs.NArg() != 0 {
|
||||
return fmt.Errorf("usage: stalwart-migrate rollback <run-id> [flags]")
|
||||
}
|
||||
|
||||
store := checkpoint.NewStore(*stateDir)
|
||||
rs, err := store.Load(runID)
|
||||
if err != nil {
|
||||
return fmt.Errorf("load run %s: %w", runID, err)
|
||||
}
|
||||
|
||||
opts := rollback.Options{
|
||||
Deployment: service.Options{
|
||||
Kind: service.Kind(*deployment), UnitName: *unitName, ContainerName: *containerName,
|
||||
},
|
||||
StopTimeout: *stopTimeout, StartTimeout: *startTimeout,
|
||||
DataDir: *dataDir,
|
||||
BinaryPath: *binaryPath,
|
||||
ServiceUnitPath: *serviceUnitPath,
|
||||
SQL: backup.SQLOptions{
|
||||
Host: *dbHost, Port: *dbPort, Database: *dbName, User: *dbUser, Password: *dbPassword, OutPath: *sqlDump,
|
||||
},
|
||||
AdminURL: *adminURL, AdminUser: *adminUser, AdminPassword: *adminPassword,
|
||||
HTTPClient: &http.Client{}, VerifyTimeout: *verifyTimeout,
|
||||
}
|
||||
|
||||
plan, err := rollback.BuildPlan(rs, opts)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
fmt.Print(plan.String())
|
||||
|
||||
if !*yes {
|
||||
fmt.Println("\nnothing has been touched. Re-run with --yes to perform this rollback.")
|
||||
return nil
|
||||
}
|
||||
if *adminURL == "" {
|
||||
fmt.Println("\nnote: without --admin-url, the reachability and directory-count checks after the restart are skipped - " +
|
||||
"the rollback will report success on the restore mechanics alone.")
|
||||
}
|
||||
|
||||
fmt.Println("\n--- rollback ---")
|
||||
report, err := rollback.Run(context.Background(), store, rs, opts)
|
||||
fmt.Print(report.String())
|
||||
if err != nil {
|
||||
return fmt.Errorf("rollback did not complete: %w", err)
|
||||
}
|
||||
|
||||
fmt.Printf("\nROLLBACK COMPLETE for run %s: the instance is back on %s. "+
|
||||
"The failed attempt's data and binary are preserved under .failed-%s names, and this run's backups are untouched, "+
|
||||
"so a retry after the underlying issue is fixed doesn't have to re-capture anything.\n",
|
||||
rs.RunID, rs.SourceVersion, rs.RunID)
|
||||
return nil
|
||||
}
|
||||
@@ -17,10 +17,11 @@ import (
|
||||
)
|
||||
|
||||
// runRun implements `stalwart-migrate run`. Only --dry-run is available
|
||||
// today: the cutover phase itself (§4.5) isn't built. internal/rollback and
|
||||
// internal/service now exist, so the missing piece is no longer "this can't
|
||||
// be undone" but "there's nothing here to undo" - see ARCHITECTURE.md §8.
|
||||
// `run` without --dry-run refuses rather than doing a migration partway.
|
||||
// today. internal/cutover exists, so the gap is no longer a phase but the
|
||||
// pipeline around it: §4.3 staging, and the wiring that would drive
|
||||
// preflight -> backup -> stage -> recovery-mode -> cutover -> validate
|
||||
// against real paths rather than a sandbox. See ARCHITECTURE.md §8. `run`
|
||||
// without --dry-run refuses rather than doing a migration partway.
|
||||
//
|
||||
// --dry-run runs preflight and a real backup (see the caveat printed below
|
||||
// about why backup still touches the live data directory), then - if the
|
||||
@@ -68,10 +69,11 @@ func runRun(args []string) (err error) {
|
||||
|
||||
if !*dryRun {
|
||||
return fmt.Errorf(
|
||||
"real (non-dry-run) migrations aren't available yet: the cutover phase - installing the new binary, rewriting the " +
|
||||
"service definition, and starting the migrated instance for real (ARCHITECTURE.md §4.5) - isn't implemented. " +
|
||||
"Rollback is, so a future cutover will be undoable; there just isn't one to undo yet. Run with --dry-run to " +
|
||||
"validate the migration mechanics against a disposable sandbox copy of your data - nothing in production is touched",
|
||||
"real (non-dry-run) migrations aren't available yet: cutover is implemented (ARCHITECTURE.md §4.5), but nothing wires it " +
|
||||
"into a production run - the staging phase (§4.3) and the real pipeline don't exist yet, so this command has no path " +
|
||||
"that touches production. Run with --dry-run to validate the migration mechanics against a disposable sandbox copy " +
|
||||
"of your data. Note that when a real run does land, recovery from a failed migration will be your own snapshot or " +
|
||||
"backup - this tool does not undo a migration (§4.8)",
|
||||
)
|
||||
}
|
||||
if *adminURL == "" {
|
||||
|
||||
@@ -3,6 +3,7 @@ package main
|
||||
import (
|
||||
"flag"
|
||||
"fmt"
|
||||
"strings"
|
||||
|
||||
"github.com/LINUXexpert-org/stalwart-migrator/internal/checkpoint"
|
||||
)
|
||||
@@ -11,9 +12,9 @@ func runStatus(args []string) error {
|
||||
fs := flag.NewFlagSet("status", flag.ExitOnError)
|
||||
stateDir := fs.String("state-dir", checkpoint.DefaultBaseDir, "directory runs are checkpointed in")
|
||||
|
||||
// Same treatment as `rollback`: without this, `status <run-id>
|
||||
// --state-dir X` would look up the run in the default directory and
|
||||
// report it missing, since flag parsing stops at the run-id.
|
||||
// Go's flag package stops parsing at the first positional argument, so
|
||||
// without this `status <run-id> --state-dir X` would look the run up in
|
||||
// the default directory and report it missing.
|
||||
runID, rest := splitRunID(fs, args)
|
||||
if err := fs.Parse(rest); err != nil {
|
||||
return err
|
||||
@@ -49,7 +50,6 @@ func runStatus(args []string) error {
|
||||
fmt.Printf("source: %s\n", rs.SourceVersion)
|
||||
fmt.Printf("target: %s\n", rs.TargetVersion)
|
||||
fmt.Printf("topology: deployment=%s store=%s\n", rs.Topology.DeploymentKind, rs.Topology.StoreBackend)
|
||||
fmt.Printf("rollback window closed: %v\n", rs.RollbackWindowClosed)
|
||||
fmt.Println("steps:")
|
||||
for _, step := range rs.Steps {
|
||||
tag := string(step.Status)
|
||||
@@ -67,3 +67,45 @@ func runStatus(args []string) error {
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// splitRunID pulls the run-id out of args wherever it appears, so
|
||||
// `status <run-id> --state-dir X` works as naturally as
|
||||
// `status --state-dir X <run-id>`. Go's flag package stops parsing at the
|
||||
// first positional argument, which would otherwise make the obvious
|
||||
// invocation order silently drop every flag after the run-id. Tokens
|
||||
// consumed as a flag's value are skipped by asking the FlagSet itself which
|
||||
// flags take one, rather than by maintaining a second list of them here.
|
||||
func splitRunID(fs *flag.FlagSet, args []string) (runID string, rest []string) {
|
||||
rest = make([]string, 0, len(args))
|
||||
for i := 0; i < len(args); i++ {
|
||||
arg := args[i]
|
||||
if arg == "--" {
|
||||
rest = append(rest, args[i:]...)
|
||||
break
|
||||
}
|
||||
if strings.HasPrefix(arg, "-") {
|
||||
rest = append(rest, arg)
|
||||
name := strings.TrimLeft(arg, "-")
|
||||
if !strings.Contains(arg, "=") && takesValue(fs, name) && i+1 < len(args) {
|
||||
i++
|
||||
rest = append(rest, args[i])
|
||||
}
|
||||
continue
|
||||
}
|
||||
if runID == "" {
|
||||
runID = arg
|
||||
continue
|
||||
}
|
||||
rest = append(rest, arg) // a second positional: let Parse report it
|
||||
}
|
||||
return runID, rest
|
||||
}
|
||||
|
||||
func takesValue(fs *flag.FlagSet, name string) bool {
|
||||
f := fs.Lookup(name)
|
||||
if f == nil {
|
||||
return false
|
||||
}
|
||||
boolFlag, ok := f.Value.(interface{ IsBoolFlag() bool })
|
||||
return !ok || !boolFlag.IsBoolFlag()
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user