Merge pull request #9 from LINUXexpert-org/docker-wire-run

Wire the container path up, behind a flag that says what it is

ARCHITECTURE.md

@@ -834,16 +834,26 @@ happens to need them. `preflight.DeploymentKind` is a type alias for
   §4.5 lists exactly which two details are inferred. A smoke test against a
   real 0.16 instance would settle both, and would let this step be promoted
   from "warns on failure" to a hard check.
-- **Docker is implemented but not yet wired to the CLI.** Preflight
+- **Docker is wired end to end and has never met a real Stalwart image.**
-  inspects a container and reports what stands in the way; stage pulls and
+  Preflight inspects a container and blocks on what stands in the way;
-  verifies an image; the recovery cycle runs in a throwaway container
+  stage pulls and verifies an image; the recovery cycle runs in a throwaway
-  against the live data; and cutover recreates the container, refusing one
+  container against the live data; cutover recreates the container,
-  whose definition it would not carry across intact. What is missing is
+  refusing one whose definition it would not carry across intact. Every
-  `run` passing those options and preflight lifting its refusal for the
+  test drives a fake `docker`, which proves the right commands are
-  containers now handled. Compose stays refused deliberately: recreating a
+  assembled and proves nothing about whether the image reads the config it
-  compose-managed container out from under compose leaves the container and
+  is handed - the same limit §4.8 records about the deleted rollback code,
-  the compose file disagreeing, and the next `compose up` reverts the
+  and the reason `run` refuses a container without
-  migration.
+  `--container-path-unproven`. A rehearsal on a clone, then a real
+  migration, is what would retire that flag.
+  Compose stays refused deliberately: recreating a compose-managed
+  container out from under compose leaves the container and the compose
+  file disagreeing, and the next `compose up` reverts the migration.
+- **The converted config reaches a container through the data volume.**
+  `run` writes it under the host side of whichever mount covers
+  `--data-dir` and names it on the container side, because cutover recreates
+  a container with the mounts it had and cannot invent a new one for a
+  config file. That is an inference from how the mounts must line up rather
+  than something a real deployment has confirmed.
 - **Cutover ignores systemd drop-ins.** It rewrites only the main unit
   file, so an `ExecStart` or `Environment` override in
   `/etc/systemd/system/stalwart.service.d/*.conf` is invisible to it -

README.md

@@ -10,6 +10,11 @@ job](#recovery-is-your-job) before using it on anything you care about.
 
 Go, standard library only — no external dependencies.
 
+A companion to [**ihasmail**](https://github.com/LINUXexpert-org/ihasmail),
+a JMAP-first webmail client for Stalwart. That one is what you read your
+mail in; this one is what gets the server underneath it onto a version that
+speaks the protocol it needs.
+
 **This has been used to migrate a production mail server.** On 2026-08-25 it
 took a live server — nine domains, six accounts, a 2.4 GB RocksDB store —
 from 0.15.5 to 0.16.19 with **8 seconds** of downtime, every phase green
@@ -55,7 +60,7 @@ described above.
 |---|---|
 | `stalwart-migrate preflight` | **Works** — read-only checks and a migration plan |
 | `stalwart-migrate rehearse` | **Works** — read-only; converts your settings and reports what won't carry over |
-| `stalwart-migrate run` | **Works** — performs the migration; `--recovery-point-confirmed --yes` |
+| `stalwart-migrate run` | **Works** — performs the migration; `--recovery-point-confirmed --yes`. Container deployments additionally need `--container-path-unproven` (see [Docker deployments](#docker-deployments)) |
 | `stalwart-migrate tenants` | **Works** — read-only; who owns which domain, and what would block a migration |
 | `stalwart-migrate status <id>` | **Works** |
 | `stalwart-migrate report <id>` | **Works** — prints what validation found for a run |
@@ -70,6 +75,49 @@ tool cannot undo a migration and will not start without it).
 **Start with `rehearse` first.** It is read-only, needs no maintenance
 window, and tells you what `run` will and won't carry over.
 
+### Docker deployments
+
+A containerised Stalwart can be migrated, with two things to know first.
+
+**The container path has never been run against a real Stalwart image.**
+Its logic is tested and its refusals are real, but a fake `docker` proves
+only that the right commands are assembled — not that the image reads the
+config it is handed. `run` refuses a container deployment unless you pass
+`--container-path-unproven`, which is there so nobody reaches it without
+being told. Rehearse on a clone first; that advice goes double here.
+
+**Two flags and one convention:**
+
+- `--target-image` names the image in full, e.g.
+  `stalwartlabs/stalwart:v0.16.14`. It is never derived from the running
+  container by swapping the tag — that is wrong for a digest-pinned image,
+  a mirror or a fork, and being wrong means pulling the wrong software into
+  a mail server.
+- `--container` names the container (default `stalwart`).
+- `--data-dir` must name the path **inside** the container, since that is
+  where its data actually lives. `preflight` says so if it matches none of
+  the container's mounts.
+
+**What it refuses, and why.** A container cannot be edited in place the way
+a unit file can, so cutting one over means rebuilding it — and a container
+rebuilt without its capabilities, its custom network or its device mappings
+starts cleanly and is quietly not the server it was. So cutover carries
+across what it understands (mounts, ports, environment, restart policy,
+labels) and refuses outright when it finds anything else, naming what it
+found. It also refuses a container whose data is not on a volume — an
+upgrade replaces the container, and the writable layer goes with it — and
+one managed by Docker Compose, because recreating it out from under compose
+leaves the container and the compose file disagreeing about what is
+deployed, and the next `compose up` reverts the migration. Compose
+deployments are migrated by editing the image tag in the compose file and
+running `compose up -d`.
+
+**What it keeps.** The old container is renamed rather than removed, the old
+image is never pruned, and the container's `docker inspect` is preserved as
+an artifact before anything is replaced. Together those are the manual
+restore path — see [Recovery is your job](#recovery-is-your-job), which
+applies here exactly as it does to a binary install.
+
 Measured on a full migration: the store converts in seconds, and the service
 was down for **6 seconds** end to end. Plan the window around verification,
 not data volume.

cmd/stalwart-migrate/run.go

@@ -9,6 +9,7 @@ import (
 	"fmt"
 	"net/http"
 	"os"
+	"path"
 	"path/filepath"
 	"time"
 
@@ -61,6 +62,9 @@ func runRun(args []string) (err error) {
 		"admin password (or set STALWART_MIGRATE_ADMIN_PASSWORD)")
 	targetVersion := fs.String("target", "latest", `target Stalwart version, or "latest"`)
 	targetBinary := fs.String("target-binary", "", "use an already-downloaded target binary instead of fetching one")
+	targetImage := fs.String("target-image", "", "for a container deployment: the target image, named in full "+
+		"(e.g. stalwartlabs/stalwart:v0.16.14). Never guessed from the running container - a derived tag is wrong for a "+
+		"digest-pinned image, a mirror or a fork")
 	stateDir := fs.String("state-dir", checkpoint.DefaultBaseDir, "directory to store run checkpoints in")
 	workDir := fs.String("work-dir", "/var/lib/stalwart-migrator/work", "scratch directory")
 	pythonPath := fs.String("python", "python3", "path to python3")
@@ -73,6 +77,10 @@ func runRun(args []string) (err error) {
 	keepArtifacts := fs.Bool("keep-artifacts", false, "don't delete work-dir/<run-id> afterward")
 	resume := fs.String("resume", "", "resume an interrupted run by id instead of starting a new one (see `status` for ids)")
 	yes := fs.Bool("yes", false, "actually perform the migration")
+	containerUnproven := fs.Bool("container-path-unproven", false,
+		"acknowledge that the container migration path has never been run against a real Stalwart image. Its logic is "+
+			"tested and its refusals are real, but a fake docker proves only that the right commands are assembled - not that "+
+			"the image reads the config it is handed. Required for a container deployment")
 	recoveryConfirmed := fs.Bool("recovery-point-confirmed", false,
 		"confirm you have a snapshot or backup you have verified you can restore - this tool cannot undo a migration")
 	if err := fs.Parse(args); err != nil {
@@ -174,22 +182,79 @@ func runRun(args []string) (err error) {
 	}
 	fmt.Printf("\nplan: %s\n", p.Reason)
 
+	// Preflight recorded how this deployment is run, and from here the two
+	// differ in three places: what gets staged, what the recovery cycle
+	// launches, and what cutover replaces. Everything between them is the
+	// same migration.
+	isContainer := service.Kind(rs.Topology.DeploymentKind) == service.Docker
+	var containerFacts preflight.ContainerFacts
+	if isContainer {
+		if !*containerUnproven {
+			return fmt.Errorf(
+				"refusing to start: this is a container deployment, and that path has never been run against a real Stalwart " +
+					"image. Its logic is tested and its refusals are real, but a fake docker proves only that the right commands " +
+					"are assembled, not that the image reads the config it is handed. Pass --container-path-unproven if you " +
+					"accept that, ideally against a clone of production first")
+		}
+		if *targetImage == "" {
+			return fmt.Errorf("refusing to start: a container deployment needs --target-image; it is never guessed from the running container")
+		}
+		if containerFacts, err = preflight.InspectContainer(ctx, *containerName); err != nil {
+			return err
+		}
+	}
+
 	fmt.Println("\n--- stage ---")
 	staged := *targetBinary
-	if staged == "" {
+	stagedImage := ""
+	switch {
+	case isContainer:
+		img, err := stage.RunImage(ctx, store, rs, stage.ImageOptions{
+			Image: *targetImage, TargetVersion: rs.TargetVersion,
+		})
+		if err != nil {
+			return fmt.Errorf("stage: %w", err)
+		}
+		stagedImage = img.Ref
+		fmt.Println(rs.Outcome(checkpoint.PhaseStage, "stage-image").Detail)
+	case staged == "":
 		staged = filepath.Join(runWorkDir, "stalwart-"+rs.TargetVersion)
 		if staged, err = stage.Run(ctx, store, rs, stage.Options{
 			TargetVersion: *targetVersion, DestPath: staged, SHA256: *binarySHA, HTTPClient: httpClient,
 		}); err != nil {
 			return fmt.Errorf("stage: %w", err)
 		}
-	}
 		fmt.Println(rs.Outcome(checkpoint.PhaseStage, "stage-binary").Detail)
+	default:
+		fmt.Println("using the already-staged binary at", staged)
+	}
 
 	script := filepath.Join(runWorkDir, "migrate_v016.py")
 	settingsPath := filepath.Join(runWorkDir, "settings.json")
 	principalsPath := filepath.Join(runWorkDir, "principals.json")
+	// The converted config has to be readable by whatever boots next. For a
+	// binary that is any path on this host; for a container it has to be
+	// somewhere the container already mounts, because cutover recreates it
+	// with the mounts it had and cannot invent a new one. So it goes inside
+	// the data volume, written on the host side and named on the container
+	// side.
 	convertedConfig := filepath.Join(runWorkDir, "config.json")
+	containerConfigPath := ""
+	if isContainer {
+		mount, ok := containerFacts.MountFor(*dataDir)
+		if !ok {
+			return fmt.Errorf(
+				"refusing to start: --data-dir %s is not covered by any of %s's mounts (%s). For a container it must name the "+
+					"path *inside* the container, since that is where its data actually lives",
+				*dataDir, *containerName, preflight.DescribeMounts(containerFacts.Mounts))
+		}
+		hostDir := filepath.Join(mount.Source, "stalwart-migrate")
+		if err := os.MkdirAll(hostDir, 0o750); err != nil {
+			return fmt.Errorf("create %s (the host side of %s): %w", hostDir, mount.Destination, err)
+		}
+		convertedConfig = filepath.Join(hostDir, "config.json")
+		containerConfigPath = path.Join(mount.Destination, "stalwart-migrate", "config.json")
+	}
 	convertedExport := filepath.Join(runWorkDir, "export.json")
 	unmigratedPath := filepath.Join(runWorkDir, "unmigrated.txt")
 	supplementPath := filepath.Join(runWorkDir, "supplement.json")
@@ -338,12 +403,32 @@ func runRun(args []string) (err error) {
 		}
 
 		fmt.Println("\n--- recovery-mode migration (the store is migrated IN PLACE) ---")
-		recReport, err := recovery.Run(ctx, store, rs, recovery.Options{
+		recOpts := recovery.Options{
 			BinaryPath: staged, ConfigPath: convertedConfig,
 			ListenURL: "http://127.0.0.1:8080/", AdminUser: "admin",
 			ApplyFiles: applyFiles, CLIBinaryPath: *stalwartCLI,
 			StartupTimeout: 20 * time.Minute, HTTPClient: httpClient,
-		})
+		}
+		if isContainer {
+			// The live container is stopped, so the recovery one takes its
+			// mounts and publishes recovery mode's listener where the
+			// health check on this side can reach it.
+			mounts := make([]recovery.ContainerMount, 0, len(containerFacts.Mounts))
+			for _, m := range containerFacts.Mounts {
+				src := m.Name
+				if src == "" {
+					src = m.Source
+				}
+				mounts = append(mounts, recovery.ContainerMount{Source: src, Destination: m.Destination, ReadOnly: !m.RW})
+			}
+			recOpts.ConfigPath = containerConfigPath
+			recOpts.Launcher = recovery.ContainerLauncher{
+				Image: stagedImage, Mounts: mounts,
+				Name:    *containerName + "-migrate-recovery",
+				Publish: []string{"127.0.0.1:8080:8080"},
+			}
+		}
+		recReport, err := recovery.Run(ctx, store, rs, recOpts)
 		fmt.Print(recReport.String())
 		if err != nil {
 			return fmt.Errorf("recovery-mode migration failed - the store may be part-migrated and the service is still "+
@@ -361,6 +446,7 @@ func runRun(args []string) (err error) {
 		ServiceUnitPath: *serviceUnitPath, ConfigPath: *newConfigPath,
 		ConfigSource: configSource, ConfigOwnerReference: *configPath,
 		Deployment:             service.Options{Kind: service.Kind(rs.Topology.DeploymentKind), UnitName: *unitName, ContainerName: *containerName},
+		Container:              containerCutover(isContainer, *containerName, stagedImage, runStateDir),
 		RecoveryPointConfirmed: *recoveryConfirmed,
 		AdminURL:               *adminURL, AdminUser: *adminUser, AdminPassword: *adminPassword,
 		HTTPClient: httpClient, RecalculateQuotas: *recalcQuotas && p.CrossesMajorBoundary,
@@ -434,3 +520,13 @@ func buildSupplement(settingsPath, principalsPath, unmigratedPath, outPath strin
 	}
 	return nil
 }
+
+// containerCutover is the cutover options for a container deployment, or
+// nil for a binary one. Nil is what keeps cutover refusing a container it
+// was given no image to recreate from.
+func containerCutover(isContainer bool, name, image, preserveDir string) *cutover.ContainerOptions {
+	if !isContainer {
+		return nil
+	}
+	return &cutover.ContainerOptions{ContainerName: name, StagedImage: image, PreserveDir: preserveDir}
+}

internal/preflight/checks.go

@@ -200,19 +200,19 @@ func (c *Checker) Run(ctx context.Context, store *checkpoint.Store, rs *checkpoi
 		// automate it - but cutover runs after the service has been
 		// stopped. Refusing there means refusing with mail already down,
 		// which is how a migration attempt turned into an outage.
-		if kind == DeploymentDocker && !c.opts.DeploymentCheckAdvisory {
-			return CheckResult{
-				Status: StatusFail,
-				Detail: "detected deployment kind: docker - this tool cannot cut over a container. " +
-					"Migrating one means pulling the new image and recreating the container, which has to be done by hand; " +
-					"`rehearse` still works and will tell you what the migration involves",
-			}, string(kind)
-		}
 		status := StatusOK
-		if kind == DeploymentUnknown {
+		detail := fmt.Sprintf("detected deployment kind: %s", kind)
+		switch kind {
+		case DeploymentUnknown:
 			status = StatusWarn
+		case DeploymentDocker:
+			// No longer a refusal on its own: a container can be migrated
+			// now. What still refuses is specific and checked below -
+			// compose, and data that is not on a volume - because those are
+			// properties of this container rather than of containers.
+			detail += " - the container checks below decide whether this one can be migrated"
 		}
-		return CheckResult{Status: status, Detail: fmt.Sprintf("detected deployment kind: %s", kind)}, string(kind)
+		return CheckResult{Status: status, Detail: detail}, string(kind)
 	})
 	if err != nil {
 		return report, err

internal/preflight/checks_test.go

@@ -475,7 +475,16 @@ func withFakeDocker(t *testing.T) {
 // install with a fake docker on PATH.
 func dockerPreflight(t *testing.T, advisory bool) Report {
 	t.Helper()
-	withFakeDocker(t)
+	// disk-space stats DataDir on this host, and container-data-volume
+	// wants it covered by a mount, so it has to be both: a real directory,
+	// mounted by the fake container.
+	dataDir := t.TempDir()
+	for _, p := range systemdUnitPaths {
+		if _, err := os.Stat(p); err == nil {
+			t.Skipf("host has %s, which detection prefers over docker", p)
+		}
+	}
+	fakeInspect(t, inspectDoc(t, nil, []Mount{dataVolume(dataDir)}))
 
 	counterPath := filepath.Join(t.TempDir(), "invocations")
 	binaryPath := writeFakeBinary(t, "0.15.5", counterPath)
@@ -495,7 +504,7 @@ func dockerPreflight(t *testing.T, advisory bool) Report {
 	// only thing that varies: whether stalwart-cli happens to be installed
 	// on the machine running the tests is not what this is testing.
 	report, err := New(Options{
-		BinaryPath: binaryPath, ConfigPath: configPath, DataDir: t.TempDir(),
+		BinaryPath: binaryPath, ConfigPath: configPath, DataDir: dataDir,
 		TargetVersion: "latest", ToolCheckAdvisory: true, DeploymentCheckAdvisory: advisory,
 	}).Run(context.Background(), store, rs)
 	if err != nil {
@@ -504,21 +513,25 @@ func dockerPreflight(t *testing.T, advisory bool) Report {
 	return report
 }
 
-// A container has to be refused here, in preflight, and not later. Cutover
+// Being a container is no longer a refusal on its own - cutover can
-// already refuses it -- but cutover runs after the service has been stopped,
+// recreate one now. What refuses is specific to *this* container: compose
-// so refusing there refuses with mail down, which turned an attempted
+// management, and data that is not on a volume. Those live in
-// migration into an outage.
+// container_test.go, and both still block.
-func TestPreflightBlocksADockerDeployment(t *testing.T) {
+//
+// This previously asserted the blanket refusal. It asserts the replacement
+// rather than being deleted, because "docker is allowed through here" is
+// the thing that would be wrong to regress.
+func TestPreflightAllowsAPlainContainerThroughTheKindCheck(t *testing.T) {
 	report := dockerPreflight(t, false)
-	if !report.Blocking() {
+	if report.Blocking() {
-		t.Fatalf("expected a blocking report for a docker deployment, got:\n%s", report.String())
+		t.Fatalf("a plain container on a volume should not be blocked:\n%s", report.String())
 	}
 	var found bool
 	for _, res := range report.Results {
 		if res.Name == "deployment-kind" {
 			found = true
-			if res.Status != StatusFail {
+			if res.Status != StatusOK {
-				t.Errorf("deployment-kind status = %q, want %q", res.Status, StatusFail)
+				t.Errorf("deployment-kind status = %q, want %q", res.Status, StatusOK)
 			}
 			if !strings.Contains(res.Detail, "docker") {
 				t.Errorf("deployment-kind detail does not mention docker: %q", res.Detail)

internal/preflight/container.go

@@ -300,9 +300,9 @@ func (c *Checker) runContainerChecks(ctx context.Context, runCheck checkFunc) er
 			return CheckResult{Status: status, Detail: fmt.Sprintf(
 				"data dir %s is not covered by any of the container's mounts (%s), so it lives in the writable layer and would "+
 					"not survive the container being replaced. Check whether --data-dir names the path inside the container",
-				c.opts.DataDir, describeMounts(facts.Mounts))}, ""
+				c.opts.DataDir, DescribeMounts(facts.Mounts))}, ""
 		}
-		return CheckResult{Status: StatusOK, Detail: "container has writable mounts: " + describeMounts(writable)}, ""
+		return CheckResult{Status: StatusOK, Detail: "container has writable mounts: " + DescribeMounts(writable)}, ""
 	})
 	return err
 }
@@ -322,7 +322,8 @@ func mountSource(m Mount) string {
 	return m.Source
 }
 
-func describeMounts(mounts []Mount) string {
+// DescribeMounts renders mounts for an operator-facing message.
+func DescribeMounts(mounts []Mount) string {
 	if len(mounts) == 0 {
 		return "none"
 	}