Cut over a container, or refuse to for a reason
A container cannot be edited in place the way a unit file can, so cutting one over means rebuilding it. That makes silent loss the default failure: a container recreated without its capabilities, its custom network or its device mappings starts cleanly and is quietly not the server it was. Section 4.5 already answers this for a unit file -- it rewrites in place rather than regenerating, because a generated unit would drop hardening options this tool has no business having an opinion about, and it refuses to edit a line it only partly understands. The same rule applies here, where the whole definition has to be rebuilt: the parts this understands are carried across, and a container using anything else is refused by name rather than rebuilt without it. The list of what it looks for is conservative and not exhaustive, which is the safe direction: docker's HostConfig has far more fields, and one this does not know about is a reason not to be recreating that container at all. The old container is renamed, not removed, and nothing here prunes the old image. Together they are the container's manual restore path -- one command starts the previous container again -- which is as close to section 4.2's preserved binary as a container gets. The inspect output is preserved as an artifact before anything is replaced, for the reason the unit file is: an operator putting a machine back by hand should not also be reconstructing the definition from memory. Recovery-mode variables are stripped from the recreated container's environment. Leaving STALWART_RECOVERY_MODE set would recovery-boot on every restart, which is the same footgun the unit rewrite exists to prevent. Run now branches, with the health check and quota recalculation shared: those ask the same question whatever started the server. The binary path moved inside an else and is otherwise untouched -- no existing test needed editing, which is the evidence for that. The container path is opt-in through Options.Container, so a Docker deployment without it is still refused exactly as before. Nothing calls it yet; wiring `run` up and lifting preflight's refusal is what remains of #3, and ARCHITECTURE.md says so in both places it previously said Docker was refused outright.
This commit is contained in:
+32
-6
@@ -290,9 +290,28 @@ against an already-migrated store.
|
|||||||
- The original unit is preserved and recorded as the `service-unit`
|
- The original unit is preserved and recorded as the `service-unit`
|
||||||
artifact *before* the rewrite, so an operator restoring by hand isn't
|
artifact *before* the rewrite, so an operator restoring by hand isn't
|
||||||
reconstructing a unit file from memory.
|
reconstructing a unit file from memory.
|
||||||
- Docker deployments are refused: cutting over a container means pulling a
|
- Docker deployments recreate rather than rewrite, because a container
|
||||||
new image and recreating the container, not swapping a binary and
|
cannot be edited in place the way a unit file can. That makes silent loss
|
||||||
rewriting a unit.
|
the default failure: a container rebuilt without its capabilities, its
|
||||||
|
custom network or its device mappings starts cleanly and is quietly not
|
||||||
|
the server it was. So the same rule the unit rewrite follows applies
|
||||||
|
here - a definition this only partly understands is one it must not
|
||||||
|
rebuild - and cutover refuses a container using anything outside the set
|
||||||
|
it carries across, naming what it found. The list is conservative and
|
||||||
|
deliberately not exhaustive; docker's HostConfig has far more fields than
|
||||||
|
it checks, and one it does not know about is a reason not to be
|
||||||
|
recreating that container at all.
|
||||||
|
- The old container is renamed rather than removed, and the old image is
|
||||||
|
never pruned. Together they are the container's manual restore path, the
|
||||||
|
nearest equivalent to the preserved binary of §4.2: one command starts
|
||||||
|
the previous container again. The `docker inspect` of the container as it
|
||||||
|
was is preserved as the `container-definition` artifact before anything
|
||||||
|
is replaced, for the same reason the unit file is.
|
||||||
|
- **Status: the container path is implemented and not yet reachable from
|
||||||
|
the CLI** - `run` does not pass container options, so a container is
|
||||||
|
still refused there. Wiring it up, and lifting preflight's refusal for
|
||||||
|
the containers it can now handle, is the remaining work in
|
||||||
|
[#3](https://github.com/LINUXexpert-org/stalwart-migrator/issues/3).
|
||||||
- Quota recalculation is the one step allowed to fail without failing the
|
- Quota recalculation is the one step allowed to fail without failing the
|
||||||
phase. Stale counters are an accounting problem; a failed cutover is one
|
phase. Stale counters are an accounting problem; a failed cutover is one
|
||||||
an operator has to respond to by restoring a machine that is otherwise
|
an operator has to respond to by restoring a machine that is otherwise
|
||||||
@@ -815,9 +834,16 @@ happens to need them. `preflight.DeploymentKind` is a type alias for
|
|||||||
§4.5 lists exactly which two details are inferred. A smoke test against a
|
§4.5 lists exactly which two details are inferred. A smoke test against a
|
||||||
real 0.16 instance would settle both, and would let this step be promoted
|
real 0.16 instance would settle both, and would let this step be promoted
|
||||||
from "warns on failure" to a hard check.
|
from "warns on failure" to a hard check.
|
||||||
- **Cutover doesn't handle Docker.** It refuses container deployments
|
- **Docker is implemented but not yet wired to the CLI.** Preflight
|
||||||
outright, since cutting one over means pulling an image and recreating
|
inspects a container and reports what stands in the way; stage pulls and
|
||||||
the container rather than swapping a binary and rewriting a unit.
|
verifies an image; the recovery cycle runs in a throwaway container
|
||||||
|
against the live data; and cutover recreates the container, refusing one
|
||||||
|
whose definition it would not carry across intact. What is missing is
|
||||||
|
`run` passing those options and preflight lifting its refusal for the
|
||||||
|
containers now handled. Compose stays refused deliberately: recreating a
|
||||||
|
compose-managed container out from under compose leaves the container and
|
||||||
|
the compose file disagreeing, and the next `compose up` reverts the
|
||||||
|
migration.
|
||||||
- **Cutover ignores systemd drop-ins.** It rewrites only the main unit
|
- **Cutover ignores systemd drop-ins.** It rewrites only the main unit
|
||||||
file, so an `ExecStart` or `Environment` override in
|
file, so an `ExecStart` or `Environment` override in
|
||||||
`/etc/systemd/system/stalwart.service.d/*.conf` is invisible to it -
|
`/etc/systemd/system/stalwart.service.d/*.conf` is invisible to it -
|
||||||
|
|||||||
@@ -0,0 +1,187 @@
|
|||||||
|
// SPDX-FileCopyrightText: 2026 LINUXexpert-org
|
||||||
|
// SPDX-License-Identifier: GPL-3.0-or-later
|
||||||
|
|
||||||
|
package cutover
|
||||||
|
|
||||||
|
import (
|
||||||
|
"bytes"
|
||||||
|
"context"
|
||||||
|
"fmt"
|
||||||
|
"os"
|
||||||
|
"os/exec"
|
||||||
|
"path/filepath"
|
||||||
|
"strings"
|
||||||
|
|
||||||
|
"github.com/LINUXexpert-org/stalwart-migrator/internal/checkpoint"
|
||||||
|
"github.com/LINUXexpert-org/stalwart-migrator/internal/preflight"
|
||||||
|
)
|
||||||
|
|
||||||
|
// ArtifactContainerDefinition is the preserved `docker inspect` of the
|
||||||
|
// container as it was before cutover replaced it - the container's
|
||||||
|
// equivalent of ArtifactServiceUnit, and for the same reason. Recovery is
|
||||||
|
// out of scope (ARCHITECTURE.md §4.8), so what this tool owes an operator
|
||||||
|
// putting a machine back by hand is the definition they would otherwise be
|
||||||
|
// reconstructing from memory.
|
||||||
|
const ArtifactContainerDefinition = "container-definition"
|
||||||
|
|
||||||
|
// ContainerOptions configures cutting over a container deployment.
|
||||||
|
type ContainerOptions struct {
|
||||||
|
// ContainerName is the live container, which must already be stopped.
|
||||||
|
ContainerName string
|
||||||
|
|
||||||
|
// StagedImage is the image ID stage verified - an ID rather than a tag
|
||||||
|
// deliberately, so what runs is what was checked.
|
||||||
|
StagedImage string
|
||||||
|
|
||||||
|
// PreserveDir is where the inspected definition is written.
|
||||||
|
PreserveDir string
|
||||||
|
|
||||||
|
DockerBinary string
|
||||||
|
}
|
||||||
|
|
||||||
|
func (o ContainerOptions) docker() string {
|
||||||
|
if o.DockerBinary == "" {
|
||||||
|
return "docker"
|
||||||
|
}
|
||||||
|
return o.DockerBinary
|
||||||
|
}
|
||||||
|
|
||||||
|
// runContainerCutover replaces the container with one running the staged
|
||||||
|
// image, carrying across the parts of its definition this tool understands
|
||||||
|
// and refusing outright when it finds parts it does not.
|
||||||
|
//
|
||||||
|
// The old container is renamed rather than removed, and the old image is
|
||||||
|
// never pruned. Together they are the manual restore path: an operator can
|
||||||
|
// start the previous container again with one command, which is as close to
|
||||||
|
// the preserved-binary guarantee (§4.2) as a container gets.
|
||||||
|
func runContainerCutover(ctx context.Context, rs *checkpoint.RunState, step stepFunc, opts ContainerOptions) error {
|
||||||
|
if opts.ContainerName == "" {
|
||||||
|
return fmt.Errorf("cutover: no container name")
|
||||||
|
}
|
||||||
|
if opts.StagedImage == "" {
|
||||||
|
return fmt.Errorf("cutover: no staged image - stage the target image before cutting over to it")
|
||||||
|
}
|
||||||
|
|
||||||
|
var facts preflight.ContainerFacts
|
||||||
|
|
||||||
|
if err := step("preserve-container-definition", func() (checkpoint.StepOutcome, error) {
|
||||||
|
raw, err := dockerOut(ctx, opts.docker(), "inspect", opts.ContainerName)
|
||||||
|
if err != nil {
|
||||||
|
return checkpoint.StepOutcome{}, fmt.Errorf("inspect %s: %w (%s)", opts.ContainerName, err, raw)
|
||||||
|
}
|
||||||
|
if opts.PreserveDir == "" {
|
||||||
|
return checkpoint.StepOutcome{}, fmt.Errorf("no directory to preserve the container definition in")
|
||||||
|
}
|
||||||
|
if err := os.MkdirAll(opts.PreserveDir, 0o750); err != nil {
|
||||||
|
return checkpoint.StepOutcome{}, err
|
||||||
|
}
|
||||||
|
dest := filepath.Join(opts.PreserveDir, opts.ContainerName+".inspect.json")
|
||||||
|
if err := os.WriteFile(dest, []byte(raw), 0o640); err != nil {
|
||||||
|
return checkpoint.StepOutcome{}, err
|
||||||
|
}
|
||||||
|
sum, size, err := hashFile(dest)
|
||||||
|
if err != nil {
|
||||||
|
return checkpoint.StepOutcome{}, err
|
||||||
|
}
|
||||||
|
// Recorded before anything is replaced, so a crash between
|
||||||
|
// preserving and recreating still leaves the original findable.
|
||||||
|
rs.RecordArtifact(ArtifactContainerDefinition, checkpoint.Artifact{Path: dest, SHA256: sum, SizeBytes: size})
|
||||||
|
|
||||||
|
facts, err = preflight.InspectContainer(ctx, opts.ContainerName)
|
||||||
|
if err != nil {
|
||||||
|
return checkpoint.StepOutcome{}, err
|
||||||
|
}
|
||||||
|
return checkpoint.StepOutcome{Detail: "preserved the container definition at " + dest}, nil
|
||||||
|
}); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
|
||||||
|
if err := step("container-is-recreatable", func() (checkpoint.StepOutcome, error) {
|
||||||
|
if len(facts.Unsupported) > 0 {
|
||||||
|
return checkpoint.StepOutcome{}, fmt.Errorf(
|
||||||
|
"this container uses configuration cutting over would not carry across: %s. Recreating it without those would "+
|
||||||
|
"start cleanly and quietly not be the server it was, so this tool will not do it. Migrate this one by hand: "+
|
||||||
|
"the definition is preserved as %s, and the staged image is %s",
|
||||||
|
strings.Join(facts.Unsupported, "; "), rs.Artifacts[ArtifactContainerDefinition].Path, opts.StagedImage)
|
||||||
|
}
|
||||||
|
return checkpoint.StepOutcome{Detail: "the container's definition is entirely within what a recreate carries across"}, nil
|
||||||
|
}); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
|
||||||
|
retired := opts.ContainerName + "-premigration"
|
||||||
|
if rs.SourceVersion != "" {
|
||||||
|
retired += "-" + rs.SourceVersion
|
||||||
|
}
|
||||||
|
|
||||||
|
if err := step("retire-old-container", func() (checkpoint.StepOutcome, error) {
|
||||||
|
// Renamed, not removed. The old container plus the old image - which
|
||||||
|
// nothing here prunes - is what an operator restores by hand.
|
||||||
|
if out, err := dockerOut(ctx, opts.docker(), "rename", opts.ContainerName, retired); err != nil {
|
||||||
|
return checkpoint.StepOutcome{}, fmt.Errorf("rename %s: %w (%s)", opts.ContainerName, err, out)
|
||||||
|
}
|
||||||
|
return checkpoint.StepOutcome{Detail: fmt.Sprintf("kept the previous container as %s, still on image %s", retired, facts.Image)}, nil
|
||||||
|
}); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
|
||||||
|
return step("create-container", func() (checkpoint.StepOutcome, error) {
|
||||||
|
args := []string{"run", "-d", "--name", opts.ContainerName}
|
||||||
|
if facts.RestartPolicy != "" && facts.RestartPolicy != "no" {
|
||||||
|
args = append(args, "--restart", facts.RestartPolicy)
|
||||||
|
}
|
||||||
|
for _, e := range facts.Env {
|
||||||
|
// Recovery-mode variables must never survive into a normal
|
||||||
|
// start: leaving STALWART_RECOVERY_MODE set would recovery-boot
|
||||||
|
// on every restart, the same footgun §4.5 strips from a unit.
|
||||||
|
if strings.HasPrefix(e, "STALWART_RECOVERY_") {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
args = append(args, "-e", e)
|
||||||
|
}
|
||||||
|
for _, m := range facts.Mounts {
|
||||||
|
src := m.Name
|
||||||
|
if src == "" {
|
||||||
|
src = m.Source
|
||||||
|
}
|
||||||
|
spec := src + ":" + m.Destination
|
||||||
|
if !m.RW {
|
||||||
|
spec += ":ro"
|
||||||
|
}
|
||||||
|
args = append(args, "-v", spec)
|
||||||
|
}
|
||||||
|
for port, bindings := range facts.Ports {
|
||||||
|
for _, b := range bindings {
|
||||||
|
spec := b.HostPort + ":" + strings.SplitN(port, "/", 2)[0]
|
||||||
|
if b.HostIP != "" {
|
||||||
|
spec = b.HostIP + ":" + spec
|
||||||
|
}
|
||||||
|
args = append(args, "-p", spec)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
for k, v := range facts.Labels {
|
||||||
|
args = append(args, "--label", k+"="+v)
|
||||||
|
}
|
||||||
|
args = append(args, opts.StagedImage)
|
||||||
|
|
||||||
|
if out, err := dockerOut(ctx, opts.docker(), args...); err != nil {
|
||||||
|
return checkpoint.StepOutcome{}, fmt.Errorf(
|
||||||
|
"create %s from %s: %w (%s). The previous container is still here as %s",
|
||||||
|
opts.ContainerName, opts.StagedImage, err, out, retired)
|
||||||
|
}
|
||||||
|
return checkpoint.StepOutcome{Detail: fmt.Sprintf("recreated %s on image %s", opts.ContainerName, opts.StagedImage)}, nil
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
// stepFunc is cutover.Run's checkpointed step runner, passed in so the
|
||||||
|
// container path reports through the same Report the systemd path does.
|
||||||
|
type stepFunc func(name string, fn func() (checkpoint.StepOutcome, error)) error
|
||||||
|
|
||||||
|
func dockerOut(ctx context.Context, name string, args ...string) (string, error) {
|
||||||
|
cmd := exec.CommandContext(ctx, name, args...)
|
||||||
|
var out bytes.Buffer
|
||||||
|
cmd.Stdout = &out
|
||||||
|
cmd.Stderr = &out
|
||||||
|
err := cmd.Run()
|
||||||
|
return strings.TrimSpace(out.String()), err
|
||||||
|
}
|
||||||
@@ -0,0 +1,254 @@
|
|||||||
|
// SPDX-FileCopyrightText: 2026 LINUXexpert-org
|
||||||
|
// SPDX-License-Identifier: GPL-3.0-or-later
|
||||||
|
|
||||||
|
package cutover
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"encoding/json"
|
||||||
|
"fmt"
|
||||||
|
"os"
|
||||||
|
"path/filepath"
|
||||||
|
"strings"
|
||||||
|
"testing"
|
||||||
|
|
||||||
|
"github.com/LINUXexpert-org/stalwart-migrator/internal/checkpoint"
|
||||||
|
)
|
||||||
|
|
||||||
|
// inspectJSON builds a `docker inspect` document. extra is merged into
|
||||||
|
// HostConfig so a test can add the configuration a recreate would drop.
|
||||||
|
func inspectJSON(t *testing.T, extraHost map[string]any, networks map[string]any) string {
|
||||||
|
t.Helper()
|
||||||
|
host := map[string]any{
|
||||||
|
"PortBindings": map[string]any{"143/tcp": []map[string]string{{"HostIp": "0.0.0.0", "HostPort": "143"}}},
|
||||||
|
"RestartPolicy": map[string]any{"Name": "unless-stopped"},
|
||||||
|
"NetworkMode": "bridge",
|
||||||
|
"LogConfig": map[string]any{"Type": "json-file"},
|
||||||
|
}
|
||||||
|
for k, v := range extraHost {
|
||||||
|
host[k] = v
|
||||||
|
}
|
||||||
|
if networks == nil {
|
||||||
|
networks = map[string]any{"bridge": map[string]any{}}
|
||||||
|
}
|
||||||
|
doc := []map[string]any{{
|
||||||
|
"Name": "/stalwart",
|
||||||
|
"Image": "sha256:old",
|
||||||
|
"Config": map[string]any{
|
||||||
|
"Image": "stalwartlabs/stalwart:v0.15.5",
|
||||||
|
"Env": []string{"TZ=UTC", "STALWART_RECOVERY_MODE=1"},
|
||||||
|
},
|
||||||
|
"State": map[string]any{"Running": false},
|
||||||
|
"Mounts": []map[string]any{{"Type": "volume", "Name": "stalwart-data", "Destination": "/opt/stalwart", "RW": true}},
|
||||||
|
"HostConfig": host,
|
||||||
|
"NetworkSettings": map[string]any{"Networks": networks},
|
||||||
|
}}
|
||||||
|
b, err := json.Marshal(doc)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
return string(b)
|
||||||
|
}
|
||||||
|
|
||||||
|
// fakeDockerCutover installs a docker that records arguments and serves the
|
||||||
|
// given inspect document.
|
||||||
|
func fakeDockerCutover(t *testing.T, doc string) (log string) {
|
||||||
|
t.Helper()
|
||||||
|
dir := t.TempDir()
|
||||||
|
log = filepath.Join(dir, "args.log")
|
||||||
|
inspectFile := filepath.Join(dir, "inspect.json")
|
||||||
|
if err := os.WriteFile(inspectFile, []byte(doc), 0o644); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
script := fmt.Sprintf(`#!/bin/sh
|
||||||
|
echo "$@" >> %q
|
||||||
|
case "$1" in
|
||||||
|
inspect) cat %q ;;
|
||||||
|
rename) exit 0 ;;
|
||||||
|
run) echo newcontainerid ;;
|
||||||
|
esac
|
||||||
|
`, log, inspectFile)
|
||||||
|
if err := os.WriteFile(filepath.Join(dir, "docker"), []byte(script), 0o755); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
t.Setenv("PATH", dir+string(os.PathListSeparator)+os.Getenv("PATH"))
|
||||||
|
return log
|
||||||
|
}
|
||||||
|
|
||||||
|
func runContainerFor(t *testing.T, doc string) (*checkpoint.RunState, Report, error) {
|
||||||
|
t.Helper()
|
||||||
|
fakeDockerCutover(t, doc)
|
||||||
|
store := checkpoint.NewStore(t.TempDir())
|
||||||
|
rs, err := store.Create("0.15.5", "0.16.14")
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
var report Report
|
||||||
|
step := func(name string, fn func() (checkpoint.StepOutcome, error)) error {
|
||||||
|
outcome, err := store.RunStep(rs, checkpoint.PhaseCutover, name, fn)
|
||||||
|
if err != nil {
|
||||||
|
report.Results = append(report.Results, CheckResult{Name: name, Status: StatusFail, Detail: err.Error()})
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
report.Results = append(report.Results, CheckResult{Name: name, Status: StatusOK, Detail: outcome.Detail})
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
err = runContainerCutover(context.Background(), rs, step, ContainerOptions{
|
||||||
|
ContainerName: "stalwart", StagedImage: "sha256:new", PreserveDir: t.TempDir(),
|
||||||
|
})
|
||||||
|
return rs, report, err
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestContainerCutoverPreservesTheDefinitionFirst(t *testing.T) {
|
||||||
|
rs, _, err := runContainerFor(t, inspectJSON(t, nil, nil))
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("runContainerCutover: %v", err)
|
||||||
|
}
|
||||||
|
art, ok := rs.Artifacts[ArtifactContainerDefinition]
|
||||||
|
if !ok {
|
||||||
|
t.Fatal("the container definition was not recorded as an artifact")
|
||||||
|
}
|
||||||
|
if art.SHA256 == "" || art.SizeBytes == 0 {
|
||||||
|
t.Errorf("artifact recorded without a checksum or size: %+v", art)
|
||||||
|
}
|
||||||
|
body, err := os.ReadFile(art.Path)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("preserved definition unreadable: %v", err)
|
||||||
|
}
|
||||||
|
if !strings.Contains(string(body), "stalwart-data") {
|
||||||
|
t.Error("preserved definition does not contain the container's mounts")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// The old container is kept, not removed: with the old image unpruned it is
|
||||||
|
// the manual restore path (ARCHITECTURE.md §4.8).
|
||||||
|
func TestContainerCutoverRetiresRatherThanRemoves(t *testing.T) {
|
||||||
|
log := fakeDockerCutover(t, inspectJSON(t, nil, nil))
|
||||||
|
store := checkpoint.NewStore(t.TempDir())
|
||||||
|
rs, _ := store.Create("0.15.5", "0.16.14")
|
||||||
|
step := func(name string, fn func() (checkpoint.StepOutcome, error)) error {
|
||||||
|
_, err := store.RunStep(rs, checkpoint.PhaseCutover, name, fn)
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
if err := runContainerCutover(context.Background(), rs, step, ContainerOptions{
|
||||||
|
ContainerName: "stalwart", StagedImage: "sha256:new", PreserveDir: t.TempDir(),
|
||||||
|
}); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
args := readLog(t, log)
|
||||||
|
if !strings.Contains(args, "rename stalwart stalwart-premigration-0.15.5") {
|
||||||
|
t.Errorf("old container was not retired by rename:\n%s", args)
|
||||||
|
}
|
||||||
|
if strings.Contains(args, "rm stalwart") || strings.Contains(args, "image rm") || strings.Contains(args, "prune") {
|
||||||
|
t.Errorf("cutover removed something it should have kept:\n%s", args)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestContainerCutoverRecreatesWithTheCarriedSettings(t *testing.T) {
|
||||||
|
log := fakeDockerCutover(t, inspectJSON(t, nil, nil))
|
||||||
|
store := checkpoint.NewStore(t.TempDir())
|
||||||
|
rs, _ := store.Create("0.15.5", "0.16.14")
|
||||||
|
step := func(name string, fn func() (checkpoint.StepOutcome, error)) error {
|
||||||
|
_, err := store.RunStep(rs, checkpoint.PhaseCutover, name, fn)
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
if err := runContainerCutover(context.Background(), rs, step, ContainerOptions{
|
||||||
|
ContainerName: "stalwart", StagedImage: "sha256:new", PreserveDir: t.TempDir(),
|
||||||
|
}); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
args := readLog(t, log)
|
||||||
|
for _, want := range []string{
|
||||||
|
"run -d --name stalwart",
|
||||||
|
"--restart unless-stopped",
|
||||||
|
"-e TZ=UTC",
|
||||||
|
"-v stalwart-data:/opt/stalwart",
|
||||||
|
"-p 0.0.0.0:143:143",
|
||||||
|
"sha256:new",
|
||||||
|
} {
|
||||||
|
if !strings.Contains(args, want) {
|
||||||
|
t.Errorf("recreate missing %q\ngot: %s", want, args)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
// Leaving recovery mode set would recovery-boot on every restart - the
|
||||||
|
// footgun §4.5 strips from a unit.
|
||||||
|
if strings.Contains(args, "STALWART_RECOVERY_MODE") {
|
||||||
|
t.Errorf("recovery-mode env survived into the recreated container:\n%s", args)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// The hazard this design exists for: a container carrying configuration a
|
||||||
|
// recreate would drop must be refused, not quietly rebuilt without it.
|
||||||
|
func TestContainerCutoverRefusesConfigurationItWouldDrop(t *testing.T) {
|
||||||
|
for _, tc := range []struct {
|
||||||
|
name string
|
||||||
|
host map[string]any
|
||||||
|
nets map[string]any
|
||||||
|
wants string
|
||||||
|
}{
|
||||||
|
{"capabilities", map[string]any{"CapAdd": []string{"NET_ADMIN"}}, nil, "capabilities"},
|
||||||
|
{"devices", map[string]any{"Devices": []any{map[string]any{}}}, nil, "device mappings"},
|
||||||
|
{"sysctls", map[string]any{"Sysctls": map[string]string{"net.core.somaxconn": "1024"}}, nil, "sysctls"},
|
||||||
|
{"privileged", map[string]any{"Privileged": true}, nil, "privileged"},
|
||||||
|
{"log driver", map[string]any{"LogConfig": map[string]any{"Type": "syslog"}}, nil, "log driver"},
|
||||||
|
{"user network", nil, map[string]any{"mailnet": map[string]any{}}, "user-defined network"},
|
||||||
|
} {
|
||||||
|
t.Run(tc.name, func(t *testing.T) {
|
||||||
|
_, _, err := runContainerFor(t, inspectJSON(t, tc.host, tc.nets))
|
||||||
|
if err == nil {
|
||||||
|
t.Fatalf("expected a refusal for a container with %s", tc.name)
|
||||||
|
}
|
||||||
|
if !strings.Contains(err.Error(), tc.wants) {
|
||||||
|
t.Errorf("refusal should name %q, got: %v", tc.wants, err)
|
||||||
|
}
|
||||||
|
if !strings.Contains(err.Error(), "by hand") {
|
||||||
|
t.Errorf("refusal should tell the operator what to do instead, got: %v", err)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// A refusal must happen before anything is touched.
|
||||||
|
func TestContainerCutoverRefusesBeforeRetiringAnything(t *testing.T) {
|
||||||
|
log := fakeDockerCutover(t, inspectJSON(t, map[string]any{"Privileged": true}, nil))
|
||||||
|
store := checkpoint.NewStore(t.TempDir())
|
||||||
|
rs, _ := store.Create("0.15.5", "0.16.14")
|
||||||
|
step := func(name string, fn func() (checkpoint.StepOutcome, error)) error {
|
||||||
|
_, err := store.RunStep(rs, checkpoint.PhaseCutover, name, fn)
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
if err := runContainerCutover(context.Background(), rs, step, ContainerOptions{
|
||||||
|
ContainerName: "stalwart", StagedImage: "sha256:new", PreserveDir: t.TempDir(),
|
||||||
|
}); err == nil {
|
||||||
|
t.Fatal("expected a refusal")
|
||||||
|
}
|
||||||
|
if args := readLog(t, log); strings.Contains(args, "rename") || strings.Contains(args, "run -d") {
|
||||||
|
t.Errorf("refusal came after the container was already changed:\n%s", args)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestContainerCutoverNeedsAStagedImage(t *testing.T) {
|
||||||
|
store := checkpoint.NewStore(t.TempDir())
|
||||||
|
rs, _ := store.Create("0.15.5", "0.16.14")
|
||||||
|
step := func(name string, fn func() (checkpoint.StepOutcome, error)) error {
|
||||||
|
_, err := store.RunStep(rs, checkpoint.PhaseCutover, name, fn)
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
if err := runContainerCutover(context.Background(), rs, step, ContainerOptions{
|
||||||
|
ContainerName: "stalwart", PreserveDir: t.TempDir(),
|
||||||
|
}); err == nil {
|
||||||
|
t.Fatal("expected a refusal with no staged image")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func readLog(t *testing.T, path string) string {
|
||||||
|
t.Helper()
|
||||||
|
b, err := os.ReadFile(path)
|
||||||
|
if os.IsNotExist(err) {
|
||||||
|
return ""
|
||||||
|
}
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
return string(b)
|
||||||
|
}
|
||||||
@@ -43,6 +43,12 @@ type Options struct {
|
|||||||
// moved the old binary aside, so this path is normally empty by now.
|
// moved the old binary aside, so this path is normally empty by now.
|
||||||
BinaryPath string
|
BinaryPath string
|
||||||
|
|
||||||
|
// Container, when set, cuts over a container deployment instead of a
|
||||||
|
// binary and a unit file. Opt-in: a Docker deployment without it is
|
||||||
|
// still refused, so a caller that has not been taught to stage an image
|
||||||
|
// cannot reach a half-built container path by accident.
|
||||||
|
Container *ContainerOptions
|
||||||
|
|
||||||
// ServiceUnitPath is the systemd unit to rewrite. ConfigPath, if set,
|
// ServiceUnitPath is the systemd unit to rewrite. ConfigPath, if set,
|
||||||
// becomes the unit's --config argument.
|
// becomes the unit's --config argument.
|
||||||
ServiceUnitPath string
|
ServiceUnitPath string
|
||||||
@@ -147,9 +153,19 @@ func BuildPlan(rs *checkpoint.RunState, opts Options) (Plan, error) {
|
|||||||
kind = service.Kind(rs.Topology.DeploymentKind)
|
kind = service.Kind(rs.Topology.DeploymentKind)
|
||||||
}
|
}
|
||||||
if kind == service.Docker {
|
if kind == service.Docker {
|
||||||
|
if opts.Container == nil {
|
||||||
return p, fmt.Errorf(
|
return p, fmt.Errorf(
|
||||||
"cutover: this run's deployment is a Docker container, where cutting over means pulling a new image and recreating the " +
|
"cutover: this run's deployment is a Docker container, where cutting over means pulling a new image and recreating the " +
|
||||||
"container rather than swapping a binary and rewriting a unit. This tool doesn't automate that - do it by hand")
|
"container rather than swapping a binary and rewriting a unit. No container options were given, so there is nothing " +
|
||||||
|
"to recreate it from - stage the target image and pass them, or do it by hand")
|
||||||
|
}
|
||||||
|
if opts.Container.StagedImage == "" {
|
||||||
|
return p, fmt.Errorf("cutover: no staged image to cut over to - stage the target image first")
|
||||||
|
}
|
||||||
|
// The binary checks below are about a file and a unit, neither of
|
||||||
|
// which a container has.
|
||||||
|
p.Target = "docker container " + opts.Container.ContainerName
|
||||||
|
return p, nil
|
||||||
}
|
}
|
||||||
deployment := opts.Deployment
|
deployment := opts.Deployment
|
||||||
deployment.Kind = kind
|
deployment.Kind = kind
|
||||||
@@ -229,6 +245,23 @@ func Run(ctx context.Context, store *checkpoint.Store, rs *checkpoint.RunState,
|
|||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// A container is replaced rather than edited, so none of the binary
|
||||||
|
// path applies: there is no file to install and no unit to rewrite.
|
||||||
|
// What follows either branch - confirming it answers, and the quota
|
||||||
|
// recalculation - is the same question whatever started it.
|
||||||
|
if opts.Container != nil {
|
||||||
|
if err := runContainerCutover(ctx, rs, step, *opts.Container); err != nil {
|
||||||
|
return report, err
|
||||||
|
}
|
||||||
|
if err := step("wait-running", func() (checkpoint.StepOutcome, error) {
|
||||||
|
if err := service.WaitFor(ctx, controller, true, startTimeoutOr(opts)); err != nil {
|
||||||
|
return checkpoint.StepOutcome{}, err
|
||||||
|
}
|
||||||
|
return checkpoint.StepOutcome{Detail: fmt.Sprintf("%s is running the migrated instance", controller.Target())}, nil
|
||||||
|
}); err != nil {
|
||||||
|
return report, err
|
||||||
|
}
|
||||||
|
} else {
|
||||||
if err := step("verify-staged-binary", func() (checkpoint.StepOutcome, error) {
|
if err := step("verify-staged-binary", func() (checkpoint.StepOutcome, error) {
|
||||||
got, err := preflight.DetectVersion(ctx, plan.StagedBinaryPath)
|
got, err := preflight.DetectVersion(ctx, plan.StagedBinaryPath)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
@@ -312,10 +345,7 @@ func Run(ctx context.Context, store *checkpoint.Store, rs *checkpoint.RunState,
|
|||||||
return report, err
|
return report, err
|
||||||
}
|
}
|
||||||
|
|
||||||
startTimeout := opts.StartTimeout
|
startTimeout := startTimeoutOr(opts)
|
||||||
if startTimeout <= 0 {
|
|
||||||
startTimeout = 60 * time.Second
|
|
||||||
}
|
|
||||||
if err := step("start-service", func() (checkpoint.StepOutcome, error) {
|
if err := step("start-service", func() (checkpoint.StepOutcome, error) {
|
||||||
if err := controller.Start(ctx); err != nil {
|
if err := controller.Start(ctx); err != nil {
|
||||||
return checkpoint.StepOutcome{}, err
|
return checkpoint.StepOutcome{}, err
|
||||||
@@ -327,6 +357,7 @@ func Run(ctx context.Context, store *checkpoint.Store, rs *checkpoint.RunState,
|
|||||||
}); err != nil {
|
}); err != nil {
|
||||||
return report, err
|
return report, err
|
||||||
}
|
}
|
||||||
|
}
|
||||||
|
|
||||||
healthTimeout := opts.HealthTimeout
|
healthTimeout := opts.HealthTimeout
|
||||||
if healthTimeout <= 0 {
|
if healthTimeout <= 0 {
|
||||||
@@ -587,3 +618,12 @@ func installConfig(src, dst, reference string) (ownership string, err error) {
|
|||||||
}
|
}
|
||||||
return fmt.Sprintf("mode %v, ownership unchanged (no reference file to copy it from)", perm), nil
|
return fmt.Sprintf("mode %v, ownership unchanged (no reference file to copy it from)", perm), nil
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// startTimeoutOr is the wait for the service to come up, defaulted. Shared
|
||||||
|
// because both branches wait for the same thing.
|
||||||
|
func startTimeoutOr(opts Options) time.Duration {
|
||||||
|
if opts.StartTimeout > 0 {
|
||||||
|
return opts.StartTimeout
|
||||||
|
}
|
||||||
|
return 60 * time.Second
|
||||||
|
}
|
||||||
|
|||||||
@@ -45,6 +45,23 @@ type ContainerFacts struct {
|
|||||||
Labels map[string]string
|
Labels map[string]string
|
||||||
Mounts []Mount
|
Mounts []Mount
|
||||||
Running bool
|
Running bool
|
||||||
|
|
||||||
|
// The rest is what cutover would have to carry across when it recreates
|
||||||
|
// the container. Recreating is the container equivalent of rewriting a
|
||||||
|
// unit file, except that a unit can be edited in place and a container
|
||||||
|
// cannot - so anything not carried here is silently dropped, which is
|
||||||
|
// the failure UnsupportedForRecreate exists to prevent.
|
||||||
|
Env []string
|
||||||
|
Ports map[string][]PortBinding
|
||||||
|
RestartPolicy string
|
||||||
|
NetworkMode string
|
||||||
|
Unsupported []string // populated by UnsupportedForRecreate
|
||||||
|
}
|
||||||
|
|
||||||
|
// PortBinding is one published port.
|
||||||
|
type PortBinding struct {
|
||||||
|
HostIP string `json:"HostIp"`
|
||||||
|
HostPort string `json:"HostPort"`
|
||||||
}
|
}
|
||||||
|
|
||||||
// ComposeProject returns the compose project managing this container, or
|
// ComposeProject returns the compose project managing this container, or
|
||||||
@@ -95,11 +112,37 @@ type inspectOutput struct {
|
|||||||
Config struct {
|
Config struct {
|
||||||
Image string `json:"Image"`
|
Image string `json:"Image"`
|
||||||
Labels map[string]string `json:"Labels"`
|
Labels map[string]string `json:"Labels"`
|
||||||
|
Env []string `json:"Env"`
|
||||||
|
User string `json:"User"`
|
||||||
} `json:"Config"`
|
} `json:"Config"`
|
||||||
State struct {
|
State struct {
|
||||||
Running bool `json:"Running"`
|
Running bool `json:"Running"`
|
||||||
} `json:"State"`
|
} `json:"State"`
|
||||||
Mounts []Mount `json:"Mounts"`
|
Mounts []Mount `json:"Mounts"`
|
||||||
|
HostConfig struct {
|
||||||
|
PortBindings map[string][]PortBinding `json:"PortBindings"`
|
||||||
|
RestartPolicy struct {
|
||||||
|
Name string `json:"Name"`
|
||||||
|
} `json:"RestartPolicy"`
|
||||||
|
NetworkMode string `json:"NetworkMode"`
|
||||||
|
CapAdd []string `json:"CapAdd"`
|
||||||
|
CapDrop []string `json:"CapDrop"`
|
||||||
|
Devices []any `json:"Devices"`
|
||||||
|
Sysctls map[string]string `json:"Sysctls"`
|
||||||
|
Ulimits []any `json:"Ulimits"`
|
||||||
|
Privileged bool `json:"Privileged"`
|
||||||
|
ExtraHosts []string `json:"ExtraHosts"`
|
||||||
|
DNS []string `json:"Dns"`
|
||||||
|
GroupAdd []string `json:"GroupAdd"`
|
||||||
|
SecurityOpt []string `json:"SecurityOpt"`
|
||||||
|
Tmpfs map[string]string `json:"Tmpfs"`
|
||||||
|
LogConfig struct {
|
||||||
|
Type string `json:"Type"`
|
||||||
|
} `json:"LogConfig"`
|
||||||
|
} `json:"HostConfig"`
|
||||||
|
NetworkSettings struct {
|
||||||
|
Networks map[string]any `json:"Networks"`
|
||||||
|
} `json:"NetworkSettings"`
|
||||||
}
|
}
|
||||||
|
|
||||||
// InspectContainer reads the facts about containerName. An error here is
|
// InspectContainer reads the facts about containerName. An error here is
|
||||||
@@ -123,14 +166,70 @@ func InspectContainer(ctx context.Context, containerName string) (ContainerFacts
|
|||||||
return ContainerFacts{}, fmt.Errorf("preflight: docker inspect %s returned no container", containerName)
|
return ContainerFacts{}, fmt.Errorf("preflight: docker inspect %s returned no container", containerName)
|
||||||
}
|
}
|
||||||
c := got[0]
|
c := got[0]
|
||||||
return ContainerFacts{
|
f := ContainerFacts{
|
||||||
Name: strings.TrimPrefix(c.Name, "/"),
|
Name: strings.TrimPrefix(c.Name, "/"),
|
||||||
Image: c.Config.Image,
|
Image: c.Config.Image,
|
||||||
ImageID: c.Image,
|
ImageID: c.Image,
|
||||||
Labels: c.Config.Labels,
|
Labels: c.Config.Labels,
|
||||||
Mounts: c.Mounts,
|
Mounts: c.Mounts,
|
||||||
Running: c.State.Running,
|
Running: c.State.Running,
|
||||||
}, nil
|
Env: c.Config.Env,
|
||||||
|
Ports: c.HostConfig.PortBindings,
|
||||||
|
RestartPolicy: c.HostConfig.RestartPolicy.Name,
|
||||||
|
NetworkMode: c.HostConfig.NetworkMode,
|
||||||
|
}
|
||||||
|
f.Unsupported = unsupportedForRecreate(c)
|
||||||
|
return f, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// unsupportedForRecreate names every piece of this container's
|
||||||
|
// configuration that recreating it would not carry across.
|
||||||
|
//
|
||||||
|
// Cutover recreates rather than edits, because a container cannot be edited
|
||||||
|
// in place the way a unit file can. That makes silent loss the default
|
||||||
|
// failure: a container recreated without its capabilities, its custom
|
||||||
|
// network or its device mappings starts cleanly and is quietly not the
|
||||||
|
// server it was. §4.5 already refuses to edit a unit line it only partly
|
||||||
|
// understands; this is the same rule, applied where the whole definition
|
||||||
|
// has to be rebuilt.
|
||||||
|
//
|
||||||
|
// The list is deliberately conservative and deliberately not exhaustive -
|
||||||
|
// docker's HostConfig has far more fields than these. It names the ones a
|
||||||
|
// mail server plausibly uses, and anything it does not know about is a
|
||||||
|
// reason this tool should not be recreating that container at all.
|
||||||
|
func unsupportedForRecreate(c inspectOutput) []string {
|
||||||
|
var out []string
|
||||||
|
add := func(cond bool, what string) {
|
||||||
|
if cond {
|
||||||
|
out = append(out, what)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
h := c.HostConfig
|
||||||
|
add(len(h.CapAdd) > 0, "added capabilities (--cap-add)")
|
||||||
|
add(len(h.CapDrop) > 0, "dropped capabilities (--cap-drop)")
|
||||||
|
add(len(h.Devices) > 0, "device mappings (--device)")
|
||||||
|
add(len(h.Sysctls) > 0, "sysctls (--sysctl)")
|
||||||
|
add(len(h.Ulimits) > 0, "ulimits (--ulimit)")
|
||||||
|
add(h.Privileged, "privileged mode (--privileged)")
|
||||||
|
add(len(h.ExtraHosts) > 0, "extra hosts (--add-host)")
|
||||||
|
add(len(h.DNS) > 0, "custom DNS (--dns)")
|
||||||
|
add(len(h.GroupAdd) > 0, "supplementary groups (--group-add)")
|
||||||
|
add(len(h.SecurityOpt) > 0, "security options (--security-opt)")
|
||||||
|
add(len(h.Tmpfs) > 0, "tmpfs mounts (--tmpfs)")
|
||||||
|
add(h.LogConfig.Type != "" && h.LogConfig.Type != "json-file", "a non-default log driver (--log-driver "+h.LogConfig.Type+")")
|
||||||
|
add(c.Config.User != "", "a container user (--user "+c.Config.User+")")
|
||||||
|
|
||||||
|
// A user-defined network is a name in NetworkSettings.Networks that is
|
||||||
|
// not one of docker's built-ins. Recreating without it puts the server
|
||||||
|
// somewhere nothing else can reach it.
|
||||||
|
for name := range c.NetworkSettings.Networks {
|
||||||
|
switch name {
|
||||||
|
case "bridge", "host", "none":
|
||||||
|
default:
|
||||||
|
out = append(out, "a user-defined network ("+name+")")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return out
|
||||||
}
|
}
|
||||||
|
|
||||||
// runContainerChecks adds the checks that only apply to a container. They
|
// runContainerChecks adds the checks that only apply to a container. They
|
||||||
|
|||||||
Reference in New Issue
Block a user