coffey-labs/stalwart-migrator
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 1 Wiki Activity

Tell what a container inherits from what it overrides

Browse Source 
Checked against a real stalwartlabs/stalwart image, `docker inspect` on an
ordinary container reports User "stalwart", Entrypoint
["/usr/local/bin/stalwart"] and Cmd ["--config",
"/etc/stalwart/config.json"] — all three inherited, none of them given.

Two things followed from reading those as the operator's.

A container user was listed as configuration a recreate would drop, so
every container off the official image was refused as unrecreatable. That
refusal lived in cutover, downstream of the stop, the settings conversion
and the store migration: it arrived with mail down and data already
moved, which is the failure issue #1 was filed for. Each of the three is
now compared against `docker image inspect` of the image the container is
on. Inherited values are left to the new image, whose own defaults are
the ones that go with it. Overrides are carried: --user, --entrypoint,
and the rest of an entrypoint as leading argv. Cmd and Entrypoint were
not being read at all, so an overridden one was silently dropped — the
exact loss the unsupported list exists to prevent.

The recreatability question also moved into preflight, while the server
is still running. Cutover asks it again, since the two are separated by
the whole migration, but only one of them can refuse without cost.

The other half: the recreated container is now started with `--config`
pointing at the migrated config in the data volume. Left to the image's
default command it came up on /etc/stalwart/config.json — a different
volume, holding whatever the old version left there — so cutover would
have produced a running server with nothing to do with the migration that
preceded it. An overridden command and that --config are the same argv
and cannot be merged honestly, so a container with one is refused and
told why.

The config is also chowned to whatever owns the data directory, before
the recovery cycle opens it. The image runs as uid 2000 and this tool
writes as root; §4.8 is the standing reminder that byte-perfect and
unreadable is a way to report success.

Found while checking @kaya-eu's field report in #1 against a real image.
Their three manual migrations are where the config step comes from.
This commit is contained in:
jcoffey-dev
2026-08-29 17:45:23 -07:00
parent 964b61bb47
commit e96e72bf79
8 changed files with 654 additions and 25 deletions
Download Patch File Download Diff File Expand all files Collapse all files
internal/preflight/checks_test.go
+55 -1
View File
@@ -474,6 +474,14 @@ func withFakeDocker(t *testing.T) {
// dockerPreflight runs a minimal but real preflight against a fake 0.15.5
// install with a fake docker on PATH.
func dockerPreflight(t *testing.T, advisory bool) Report {
t.Helper()
return dockerPreflightOn(t, advisory, nil)
}
// dockerPreflightOn runs preflight against a container whose inspect
// document has been rewritten by edit, for the cases that need it to be
// something other than an ordinary one.
func dockerPreflightOn(t *testing.T, advisory bool, edit func(string) string) Report {
t.Helper()
// disk-space stats DataDir on this host, and container-data-volume
// wants it covered by a mount, so it has to be both: a real directory,
@@ -484,7 +492,11 @@ func dockerPreflight(t *testing.T, advisory bool) Report {
t.Skipf("host has %s, which detection prefers over docker", p)
}
}
fakeInspect(t, inspectDoc(t, nil, []Mount{dataVolume(dataDir)}))
doc := inspectDoc(t, nil, []Mount{dataVolume(dataDir)})
if edit != nil {
doc = edit(doc)
}
fakeInspect(t, doc)
counterPath := filepath.Join(t.TempDir(), "invocations")
binaryPath := writeFakeBinary(t, "0.15.5", counterPath)
@@ -553,3 +565,45 @@ func TestRehearseStillRunsAgainstADockerDeployment(t *testing.T) {
t.Fatalf("advisory mode should not block on docker, got:\n%s", report.String())
}
}
// Cutover already refused a container it could not recreate, but cutover
// is downstream of the stop, the settings conversion and the store
// migration - so that refusal arrived with the mail down and the data
// already moved, which is the shape of failure issue #1 was filed for.
// The answer never changes between the two points, so it is asked here,
// while the server is still running.
func TestPreflightRefusesAContainerItCouldNotRecreate(t *testing.T) {
report := dockerPreflightOn(t, false, func(doc string) string {
return strings.Replace(doc, `"State":`, `"HostConfig":{"Privileged":true},"State":`, 1)
})
if !report.Blocking() {
t.Fatalf("a container preflight cannot recreate should block before anything stops:\n%s", report.String())
}
var found bool
for _, res := range report.Results {
if res.Name == "container-recreatable" {
found = true
if res.Status != StatusFail {
t.Errorf("container-recreatable status = %q, want %q", res.Status, StatusFail)
}
if !strings.Contains(res.Detail, "privileged") {
t.Errorf("detail should name what would be dropped, got %q", res.Detail)
}
}
}
if !found {
t.Fatalf("no container-recreatable result in report:\n%s", report.String())
}
}
// rehearse never stops or recreates anything, so the same finding is a
// warning there: an operator migrating by hand needs to know it more than
// an automated run does.
func TestRehearseWarnsRatherThanBlocksOnAnUnrecreatableContainer(t *testing.T) {
report := dockerPreflightOn(t, true, func(doc string) string {
return strings.Replace(doc, `"State":`, `"HostConfig":{"Privileged":true},"State":`, 1)
})
if report.Blocking() {
t.Fatalf("advisory mode should not block, got:\n%s", report.String())
}
}
internal/preflight/container.go
+140 -11
View File
@@ -55,7 +55,23 @@ type ContainerFacts struct {
Ports map[string][]PortBinding
RestartPolicy string
NetworkMode string
Unsupported []string // populated by UnsupportedForRecreate
Unsupported []string // populated by unsupportedForRecreate
// User, Entrypoint and Cmd are set only when the container overrides
// what its image already says.
//
// The distinction is the whole point. `docker inspect` reports these
// three whether the operator set them or the image did - a container
// off the official image reports User "stalwart" and Cmd
// ["--config", "/etc/stalwart/config.json"] having been given
// neither. Treating an inherited value as the operator's would either
// refuse every ordinary container or pin the new image to the old
// image's defaults, and the new image's defaults are the ones that go
// with the new image. Only a genuine override is the operator's
// decision, and only that has to survive a recreate.
User string
Entrypoint []string
Cmd []string
}
// PortBinding is one published port.
@@ -107,15 +123,10 @@ func (f ContainerFacts) MountFor(path string) (Mount, bool) {
// separately from ContainerFacts because docker's shape is docker's to
// change, and the rest of this package should not have to know it.
type inspectOutput struct {
Name string `json:"Name"`
Image string `json:"Image"`
Config struct {
Image string `json:"Image"`
Labels map[string]string `json:"Labels"`
Env []string `json:"Env"`
User string `json:"User"`
} `json:"Config"`
State struct {
Name string `json:"Name"`
Image string `json:"Image"`
Config containerConfig `json:"Config"`
State struct {
Running bool `json:"Running"`
} `json:"State"`
Mounts []Mount `json:"Mounts"`
@@ -145,6 +156,24 @@ type inspectOutput struct {
} `json:"NetworkSettings"`
}
// containerConfig is the part of a container's or an image's Config this
// reasons about. Both docker objects carry the same shape here, which is
// what makes comparing them possible.
type containerConfig struct {
Image string `json:"Image"`
Labels map[string]string `json:"Labels"`
Env []string `json:"Env"`
User string `json:"User"`
Entrypoint []string `json:"Entrypoint"`
Cmd []string `json:"Cmd"`
}
// imageInspectOutput is `docker image inspect`, which reports the defaults
// a container inherits when it was given none of its own.
type imageInspectOutput struct {
Config containerConfig `json:"Config"`
}
// InspectContainer reads the facts about containerName. An error here is
// an error, not an absent container: callers reach this only after
// DetectDeploymentKind has already established that a container answers to
@@ -178,10 +207,66 @@ func InspectContainer(ctx context.Context, containerName string) (ContainerFacts
RestartPolicy: c.HostConfig.RestartPolicy.Name,
NetworkMode: c.HostConfig.NetworkMode,
}
// The image the container is actually on, by ID rather than by the tag
// it was started from: a tag can have moved since, and then this would
// be comparing the container against something it never inherited
// from.
base, err := inspectImage(ctx, c.Image)
if err != nil {
return ContainerFacts{}, err
}
if c.Config.User != base.User {
f.User = c.Config.User
}
if !sameArgs(c.Config.Entrypoint, base.Entrypoint) {
f.Entrypoint = c.Config.Entrypoint
}
if !sameArgs(c.Config.Cmd, base.Cmd) {
f.Cmd = c.Config.Cmd
}
f.Unsupported = unsupportedForRecreate(c)
return f, nil
}
// inspectImage reads the defaults an image gives the containers made from
// it. A failure here is an error for the same reason a failed container
// inspect is: without it there is no way to tell an operator's --user from
// the image's own USER, and the difference decides what a recreate has to
// carry.
func inspectImage(ctx context.Context, imageID string) (containerConfig, error) {
if imageID == "" {
return containerConfig{}, fmt.Errorf("preflight: container reports no image to compare its configuration against")
}
out, err := exec.CommandContext(ctx, "docker", "image", "inspect", imageID).Output()
if err != nil {
return containerConfig{}, fmt.Errorf("preflight: docker image inspect %s: %w", imageID, err)
}
var got []imageInspectOutput
if err := json.Unmarshal(out, &got); err != nil {
return containerConfig{}, fmt.Errorf("preflight: parsing docker image inspect %s: %w", imageID, err)
}
if len(got) == 0 {
return containerConfig{}, fmt.Errorf("preflight: docker image inspect %s returned no image", imageID)
}
return got[0].Config, nil
}
// sameArgs compares two argv slices, treating nil and empty as the same
// thing - docker reports an absent Cmd either way depending on version.
func sameArgs(a, b []string) bool {
if len(a) != len(b) {
return false
}
for i := range a {
if a[i] != b[i] {
return false
}
}
return true
}
// unsupportedForRecreate names every piece of this container's
// configuration that recreating it would not carry across.
//
@@ -217,7 +302,6 @@ func unsupportedForRecreate(c inspectOutput) []string {
add(len(h.SecurityOpt) > 0, "security options (--security-opt)")
add(len(h.Tmpfs) > 0, "tmpfs mounts (--tmpfs)")
add(h.LogConfig.Type != "" && h.LogConfig.Type != "json-file", "a non-default log driver (--log-driver "+h.LogConfig.Type+")")
add(c.Config.User != "", "a container user (--user "+c.Config.User+")")
// A user-defined network is a name in NetworkSettings.Networks that is
// not one of docker's built-ins. Recreating without it puts the server
@@ -274,6 +358,31 @@ func (c *Checker) runContainerChecks(ctx context.Context, runCheck checkFunc) er
return err
}
if _, err := runCheck("container-recreatable", func() (CheckResult, string) {
// Asked here, while the server is still running, rather than at
// cutover where the answer was first needed. Cutover is downstream
// of the stop, the settings conversion and the store migration, so
// a refusal there is a refusal with the mail already down and the
// data already moved - the shape of failure issue #1 was filed for.
// Nothing about this answer changes between the two points.
if len(facts.Unsupported) > 0 {
status := StatusFail
if c.opts.DeploymentCheckAdvisory {
status = StatusWarn
}
return CheckResult{Status: status, Detail: fmt.Sprintf(
"this container uses configuration that recreating it would not carry across: %s. A container is replaced "+
"rather than edited, so those would be silently dropped and the result would start cleanly without being "+
"the server it was. Migrate this one by hand",
strings.Join(facts.Unsupported, "; "))}, strings.Join(facts.Unsupported, "; ")
}
carried := describeOverrides(facts)
return CheckResult{Status: StatusOK, Detail: "the container's definition is entirely within what a recreate " +
"carries across" + carried}, ""
}); err != nil {
return err
}
_, err := runCheck("container-data-volume", func() (CheckResult, string) {
writable := facts.WritableMounts()
if len(writable) == 0 {
@@ -307,6 +416,26 @@ func (c *Checker) runContainerChecks(ctx context.Context, runCheck checkFunc) er
return err
}
// describeOverrides names the settings a container holds that its image
// does not, so an operator reading a green check can see what a recreate
// is being trusted to carry rather than taking "entirely within" on faith.
func describeOverrides(f ContainerFacts) string {
var parts []string
if f.User != "" {
parts = append(parts, "--user "+f.User)
}
if len(f.Entrypoint) > 0 {
parts = append(parts, "--entrypoint "+strings.Join(f.Entrypoint, " "))
}
if len(f.Cmd) > 0 {
parts = append(parts, "a command ("+strings.Join(f.Cmd, " ")+")")
}
if len(parts) == 0 {
return ""
}
return ", including what it overrides on its image: " + strings.Join(parts, ", ")
}
func shortID(id string) string {
id = strings.TrimPrefix(id, "sha256:")
if len(id) > 12 {
internal/preflight/container_test.go
+110 -5
View File
@@ -10,33 +10,72 @@ import (
"net/http"
"os"
"path/filepath"
"strings"
"testing"
"github.com/LINUXexpert-org/stalwart-migrator/internal/checkpoint"
)
// The defaults the official Stalwart image gives every container made from
// it. A container reporting exactly these has overridden nothing, which is
// the case the image comparison exists to recognise - `docker inspect`
// reports all three either way.
var (
imageUser = "stalwart"
imageEntrypoint = []string{"/usr/local/bin/stalwart"}
imageCmd = []string{"--config", "/etc/stalwart/config.json"}
)
// fakeInspect writes a `docker` that answers `inspect` with the given JSON
// document, so the container checks can be exercised without a container.
// `image inspect` answers with the official image's own defaults.
func fakeInspect(t *testing.T, doc string) {
t.Helper()
fakeInspectOn(t, doc, imageDoc(t, imageUser, imageEntrypoint, imageCmd))
}
// fakeInspectOn is fakeInspect with the image's defaults named, for the
// tests that need the container and its image to disagree.
func fakeInspectOn(t *testing.T, containerDoc, imgDoc string) {
t.Helper()
dir := t.TempDir()
out := filepath.Join(dir, "inspect.json")
if err := os.WriteFile(out, []byte(doc), 0o644); err != nil {
if err := os.WriteFile(out, []byte(containerDoc), 0o644); err != nil {
t.Fatal(err)
}
script := fmt.Sprintf("#!/bin/sh\ncase \"$1\" in inspect) cat %q ;; *) exit 1 ;; esac\n", out)
img := filepath.Join(dir, "image.json")
if err := os.WriteFile(img, []byte(imgDoc), 0o644); err != nil {
t.Fatal(err)
}
script := fmt.Sprintf("#!/bin/sh\n"+
"case \"$1 $2\" in \"image inspect\") cat %q ; exit 0 ;; esac\n"+
"case \"$1\" in inspect) cat %q ;; *) exit 1 ;; esac\n", img, out)
if err := os.WriteFile(filepath.Join(dir, "docker"), []byte(script), 0o755); err != nil {
t.Fatal(err)
}
t.Setenv("PATH", dir+string(os.PathListSeparator)+os.Getenv("PATH"))
}
func imageDoc(t *testing.T, user string, entrypoint, cmd []string) string {
t.Helper()
b, err := json.Marshal([]map[string]any{{
"Config": map[string]any{"User": user, "Entrypoint": entrypoint, "Cmd": cmd},
}})
if err != nil {
t.Fatal(err)
}
return string(b)
}
func inspectDoc(t *testing.T, labels map[string]string, mounts []Mount) string {
t.Helper()
doc := []map[string]any{{
"Name": "/stalwart",
"Image": "sha256:0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef",
"Config": map[string]any{"Image": "stalwartlabs/stalwart:v0.15.5", "Labels": labels},
"Name": "/stalwart",
"Image": "sha256:0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef",
"Config": map[string]any{
"Image": "stalwartlabs/stalwart:v0.15.5", "Labels": labels,
"User": imageUser, "Entrypoint": imageEntrypoint, "Cmd": imageCmd,
},
"State": map[string]any{"Running": true},
"Mounts": mounts,
}}
@@ -218,3 +257,69 @@ func TestRehearseReportsContainerProblemsWithoutBlocking(t *testing.T) {
}
}
}
// docker reports Config.User, Cmd and Entrypoint whether the operator set
// them or the image did. A container off the official image reports user
// "stalwart" having been given no --user, and reading that as an operator
// override made this tool refuse to recreate every ordinary Stalwart
// container - at cutover, with the mail already down. Found while checking
// @kaya-eu's field report against a real image.
func TestInspectContainerIgnoresWhatItInheritedFromItsImage(t *testing.T) {
fakeInspect(t, inspectDoc(t, nil, []Mount{dataVolume("/var/lib/stalwart")}))
facts, err := InspectContainer(context.Background(), "stalwart")
if err != nil {
t.Fatalf("InspectContainer: %v", err)
}
if facts.User != "" {
t.Errorf("User = %q, want empty: it is the image's own USER, not an override", facts.User)
}
if len(facts.Cmd) != 0 {
t.Errorf("Cmd = %v, want none: it is the image's own CMD", facts.Cmd)
}
if len(facts.Entrypoint) != 0 {
t.Errorf("Entrypoint = %v, want none: it is the image's own ENTRYPOINT", facts.Entrypoint)
}
if len(facts.Unsupported) != 0 {
t.Errorf("Unsupported = %v, want none for a plain container off the official image", facts.Unsupported)
}
}
// The other half of the same distinction: what the operator really did
// override has to be visible, because a recreate that drops it starts
// cleanly as a different server.
func TestInspectContainerReportsWhatTheOperatorOverrode(t *testing.T) {
doc := inspectDoc(t, nil, []Mount{dataVolume("/var/lib/stalwart")})
doc = strings.Replace(doc, `"User":"stalwart"`, `"User":"1500:1500"`, 1)
doc = strings.Replace(doc, `"Cmd":["--config","/etc/stalwart/config.json"]`, `"Cmd":["--config","/srv/mine.toml"]`, 1)
if strings.Contains(doc, `"User":"stalwart"`) || strings.Contains(doc, "/etc/stalwart/config.json") {
t.Fatal("the fixture did not take the overrides; the inspect document shape changed")
}
fakeInspect(t, doc)
facts, err := InspectContainer(context.Background(), "stalwart")
if err != nil {
t.Fatalf("InspectContainer: %v", err)
}
if facts.User != "1500:1500" {
t.Errorf("User = %q, want the overridden 1500:1500", facts.User)
}
if strings.Join(facts.Cmd, " ") != "--config /srv/mine.toml" {
t.Errorf("Cmd = %v, want the overridden command", facts.Cmd)
}
// An entrypoint it did not override still reads as inherited.
if len(facts.Entrypoint) != 0 {
t.Errorf("Entrypoint = %v, want none", facts.Entrypoint)
}
}
// Without the image's defaults there is no way to tell an override from an
// inheritance, and guessing decides what a recreate carries. Same rule as
// a failed container inspect: an error, not an assumption.
func TestInspectContainerRefusesWhenTheImageCannotBeRead(t *testing.T) {
fakeInspectOn(t, inspectDoc(t, nil, []Mount{dataVolume("/var/lib/stalwart")}), "")
if _, err := InspectContainer(context.Background(), "stalwart"); err == nil {
t.Fatal("want an error when the image's defaults cannot be read")
}
}