coffey-labs/stalwart-migrator
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 1 Wiki Activity

Tell what a container inherits from what it overrides

Browse Source 
Checked against a real stalwartlabs/stalwart image, `docker inspect` on an
ordinary container reports User "stalwart", Entrypoint
["/usr/local/bin/stalwart"] and Cmd ["--config",
"/etc/stalwart/config.json"] — all three inherited, none of them given.

Two things followed from reading those as the operator's.

A container user was listed as configuration a recreate would drop, so
every container off the official image was refused as unrecreatable. That
refusal lived in cutover, downstream of the stop, the settings conversion
and the store migration: it arrived with mail down and data already
moved, which is the failure issue #1 was filed for. Each of the three is
now compared against `docker image inspect` of the image the container is
on. Inherited values are left to the new image, whose own defaults are
the ones that go with it. Overrides are carried: --user, --entrypoint,
and the rest of an entrypoint as leading argv. Cmd and Entrypoint were
not being read at all, so an overridden one was silently dropped — the
exact loss the unsupported list exists to prevent.

The recreatability question also moved into preflight, while the server
is still running. Cutover asks it again, since the two are separated by
the whole migration, but only one of them can refuse without cost.

The other half: the recreated container is now started with `--config`
pointing at the migrated config in the data volume. Left to the image's
default command it came up on /etc/stalwart/config.json — a different
volume, holding whatever the old version left there — so cutover would
have produced a running server with nothing to do with the migration that
preceded it. An overridden command and that --config are the same argv
and cannot be merged honestly, so a container with one is refused and
told why.

The config is also chowned to whatever owns the data directory, before
the recovery cycle opens it. The image runs as uid 2000 and this tool
writes as root; §4.8 is the standing reminder that byte-perfect and
unreadable is a way to report success.

Found while checking @kaya-eu's field report in #1 against a real image.
Their three manual migrations are where the config step comes from.
This commit is contained in:
jcoffey-dev
2026-08-29 17:45:23 -07:00
parent 964b61bb47
commit e96e72bf79
8 changed files with 654 additions and 25 deletions
Download Patch File Download Diff File Expand all files Collapse all files
internal/cutover/container.go
+59 -1
View File
@@ -36,6 +36,18 @@ type ContainerOptions struct {
// PreserveDir is where the inspected definition is written.
PreserveDir string
// ConfigPath is the migrated v0.16 config, named as the *container*
// sees it - the path inside a mount the container already has, since a
// recreate carries the mounts it had and cannot invent a new one.
//
// Without this the recreated container falls back to its image's own
// default command, which on the official image is
// `--config /etc/stalwart/config.json`. That path is a different
// volume from the data directory and holds whatever the old version
// left there, so the new container would come up on a config that has
// nothing to do with the migration that just happened.
ConfigPath string
DockerBinary string
}
@@ -96,6 +108,11 @@ func runContainerCutover(ctx context.Context, rs *checkpoint.RunState, step step
return err
}
// Preflight asked this too, before anything stopped (§4.1). It is
// asked again here because the two are separated by the whole
// migration, and a container can be reconfigured in between - but by
// the time this refuses, the answer has cost an outage, which is why
// preflight is where it is meant to be caught.
if err := step("container-is-recreatable", func() (checkpoint.StepOutcome, error) {
if len(facts.Unsupported) > 0 {
return checkpoint.StepOutcome{}, fmt.Errorf(
@@ -104,6 +121,19 @@ func runContainerCutover(ctx context.Context, rs *checkpoint.RunState, step step
"the definition is preserved as %s, and the staged image is %s",
strings.Join(facts.Unsupported, "; "), rs.Artifacts[ArtifactContainerDefinition].Path, opts.StagedImage)
}
// A command of the operator's own and a config this tool has to
// hand over are the same argv, and there is no honest way to merge
// them: their command may point at another config, or at something
// that is not the server at all. Refusing names both rather than
// picking one and being quietly wrong about which server came up.
if opts.ConfigPath != "" && len(facts.Cmd) > 0 {
return checkpoint.StepOutcome{}, fmt.Errorf(
"this container overrides its image's command (%s), and cutting over has to start the new one with "+
"`--config %s` - the migrated configuration. Both are the container's argv and this tool will not guess at "+
"a merge. Recreate it by hand from the preserved definition at %s, on image %s, with your command adjusted "+
"to that config",
strings.Join(facts.Cmd, " "), opts.ConfigPath, rs.Artifacts[ArtifactContainerDefinition].Path, opts.StagedImage)
}
return checkpoint.StepOutcome{Detail: "the container's definition is entirely within what a recreate carries across"}, nil
}); err != nil {
return err
@@ -130,6 +160,17 @@ func runContainerCutover(ctx context.Context, rs *checkpoint.RunState, step step
if facts.RestartPolicy != "" && facts.RestartPolicy != "no" {
args = append(args, "--restart", facts.RestartPolicy)
}
// Only what the container overrode on its old image is carried.
// An inherited value belongs to the image, and the new image's own
// default is the one that goes with the new image - pinning the
// old image's USER or ENTRYPOINT onto it would be carrying across
// a decision nobody made.
if facts.User != "" {
args = append(args, "--user", facts.User)
}
if len(facts.Entrypoint) > 0 {
args = append(args, "--entrypoint", facts.Entrypoint[0])
}
for _, e := range facts.Env {
// Recovery-mode variables must never survive into a normal
// start: leaving STALWART_RECOVERY_MODE set would recovery-boot
@@ -164,12 +205,29 @@ func runContainerCutover(ctx context.Context, rs *checkpoint.RunState, step step
}
args = append(args, opts.StagedImage)
// Everything after the image is the container's argv. `docker run`
// takes only the first word of an entrypoint as --entrypoint, so
// the rest of it leads here.
if len(facts.Entrypoint) > 1 {
args = append(args, facts.Entrypoint[1:]...)
}
switch {
case opts.ConfigPath != "":
args = append(args, "--config", opts.ConfigPath)
case len(facts.Cmd) > 0:
args = append(args, facts.Cmd...)
}
if out, err := dockerOut(ctx, opts.docker(), args...); err != nil {
return checkpoint.StepOutcome{}, fmt.Errorf(
"create %s from %s: %w (%s). The previous container is still here as %s",
opts.ContainerName, opts.StagedImage, err, out, retired)
}
return checkpoint.StepOutcome{Detail: fmt.Sprintf("recreated %s on image %s", opts.ContainerName, opts.StagedImage)}, nil
detail := fmt.Sprintf("recreated %s on image %s", opts.ContainerName, opts.StagedImage)
if opts.ConfigPath != "" {
detail += ", started with --config " + opts.ConfigPath
}
return checkpoint.StepOutcome{Detail: detail}, nil
})
}
internal/cutover/container_test.go
+178 -1
View File
@@ -37,6 +37,12 @@ func inspectJSON(t *testing.T, extraHost map[string]any, networks map[string]any
"Config": map[string]any{
"Image": "stalwartlabs/stalwart:v0.15.5",
"Env": []string{"TZ=UTC", "STALWART_RECOVERY_MODE=1"},
// Matching imageJSON's defaults, so nothing here reads as an
// override. A container off the official image reports all
// three having been given none of them.
"User": imageUser,
"Entrypoint": imageEntrypoint,
"Cmd": imageCmd,
},
"State": map[string]any{"Running": false},
"Mounts": []map[string]any{{"Type": "volume", "Name": "stalwart-data", "Destination": "/opt/stalwart", "RW": true}},
@@ -50,6 +56,28 @@ func inspectJSON(t *testing.T, extraHost map[string]any, networks map[string]any
return string(b)
}
// The defaults the official Stalwart image gives every container made
// from it. They are here rather than inline because the whole point of
// the image comparison is that a container reporting exactly these has
// overridden nothing.
var (
imageUser = "stalwart"
imageEntrypoint = []string{"/usr/local/bin/stalwart"}
imageCmd = []string{"--config", "/etc/stalwart/config.json"}
)
// imageJSON is `docker image inspect` for the image a container is on.
func imageJSON(t *testing.T) string {
t.Helper()
b, err := json.Marshal([]map[string]any{{
"Config": map[string]any{"User": imageUser, "Entrypoint": imageEntrypoint, "Cmd": imageCmd},
}})
if err != nil {
t.Fatal(err)
}
return string(b)
}
// fakeDockerCutover installs a docker that records arguments and serves the
// given inspect document.
func fakeDockerCutover(t *testing.T, doc string) (log string) {
@@ -60,14 +88,21 @@ func fakeDockerCutover(t *testing.T, doc string) (log string) {
if err := os.WriteFile(inspectFile, []byte(doc), 0o644); err != nil {
t.Fatal(err)
}
imageFile := filepath.Join(dir, "image.json")
if err := os.WriteFile(imageFile, []byte(imageJSON(t)), 0o644); err != nil {
t.Fatal(err)
}
script := fmt.Sprintf(`#!/bin/sh
echo "$@" >> %q
case "$1 $2" in
"image inspect") cat %q ; exit 0 ;;
esac
case "$1" in
inspect) cat %q ;;
rename) exit 0 ;;
run) echo newcontainerid ;;
esac
`, log, inspectFile)
`, log, imageFile, inspectFile)
if err := os.WriteFile(filepath.Join(dir, "docker"), []byte(script), 0o755); err != nil {
t.Fatal(err)
}
@@ -95,10 +130,15 @@ func runContainerFor(t *testing.T, doc string) (*checkpoint.RunState, Report, er
}
err = runContainerCutover(context.Background(), rs, step, ContainerOptions{
ContainerName: "stalwart", StagedImage: "sha256:new", PreserveDir: t.TempDir(),
ConfigPath: containerConfig,
})
return rs, report, err
}
// containerConfig is the container-side path to the migrated config that
// run passes to cutover, mirroring run.go's <data mount>/stalwart-migrate.
const containerConfig = "/opt/stalwart/stalwart-migrate/config.json"
func TestContainerCutoverPreservesTheDefinitionFirst(t *testing.T) {
rs, _, err := runContainerFor(t, inspectJSON(t, nil, nil))
if err != nil {
@@ -252,3 +292,140 @@ func readLog(t *testing.T, path string) string {
}
return string(b)
}
// The recreated container has to be started on the config the migration
// just produced. Left to the image's own default command it would come up
// on /etc/stalwart/config.json - a different volume from the data
// directory, holding whatever the old version left there - and be a
// server with nothing to do with the migration that preceded it.
// @kaya-eu did this step by hand on three real migrations.
func TestContainerCutoverStartsOnTheMigratedConfig(t *testing.T) {
log := fakeDockerCutover(t, inspectJSON(t, nil, nil))
store := checkpoint.NewStore(t.TempDir())
rs, err := store.Create("0.15.5", "0.16.14")
if err != nil {
t.Fatal(err)
}
if err := runContainerCutover(context.Background(), rs, noopStep(store, rs), ContainerOptions{
ContainerName: "stalwart", StagedImage: "sha256:new", PreserveDir: t.TempDir(),
ConfigPath: containerConfig,
}); err != nil {
t.Fatalf("runContainerCutover: %v", err)
}
run := runLine(t, log)
if !strings.Contains(run, "--config "+containerConfig) {
t.Errorf("run should start the container on the migrated config, got: %s", run)
}
// After the image, not before: everything past it is the argv.
if strings.Index(run, "sha256:new") > strings.Index(run, "--config "+containerConfig) {
t.Errorf("--config must come after the image, got: %s", run)
}
}
// An inherited USER belongs to the image. Passing --user stalwart to the
// new image would be carrying across a decision nobody made, and would
// break outright on an image that named its user differently.
func TestContainerCutoverDoesNotCarryInheritedDefaults(t *testing.T) {
log := fakeDockerCutover(t, inspectJSON(t, nil, nil))
store := checkpoint.NewStore(t.TempDir())
rs, _ := store.Create("0.15.5", "0.16.14")
if err := runContainerCutover(context.Background(), rs, noopStep(store, rs), ContainerOptions{
ContainerName: "stalwart", StagedImage: "sha256:new", PreserveDir: t.TempDir(),
ConfigPath: containerConfig,
}); err != nil {
t.Fatalf("runContainerCutover: %v", err)
}
run := runLine(t, log)
for _, unwanted := range []string{"--user", "--entrypoint"} {
if strings.Contains(run, unwanted) {
t.Errorf("run carried %s across from the old image's defaults: %s", unwanted, run)
}
}
}
// What the operator did override is theirs, and a recreate that drops it
// starts cleanly as a different server - the failure Unsupported exists to
// prevent, for two settings that were not being read at all.
func TestContainerCutoverCarriesTheOperatorsOverrides(t *testing.T) {
doc := inspectJSON(t, nil, nil)
doc = strings.Replace(doc, `"User":"stalwart"`, `"User":"1500:1500"`, 1)
doc = strings.Replace(doc, `"Entrypoint":["/usr/local/bin/stalwart"]`,
`"Entrypoint":["/usr/local/bin/wrapper","--trace"]`, 1)
log := fakeDockerCutover(t, doc)
store := checkpoint.NewStore(t.TempDir())
rs, _ := store.Create("0.15.5", "0.16.14")
if err := runContainerCutover(context.Background(), rs, noopStep(store, rs), ContainerOptions{
ContainerName: "stalwart", StagedImage: "sha256:new", PreserveDir: t.TempDir(),
ConfigPath: containerConfig,
}); err != nil {
t.Fatalf("runContainerCutover: %v", err)
}
run := runLine(t, log)
if !strings.Contains(run, "--user 1500:1500") {
t.Errorf("run should carry the overridden user: %s", run)
}
// docker run takes one word as --entrypoint; the rest is argv.
if !strings.Contains(run, "--entrypoint /usr/local/bin/wrapper") {
t.Errorf("run should carry the overridden entrypoint: %s", run)
}
if !strings.Contains(run, "sha256:new --trace") {
t.Errorf("the rest of the entrypoint should lead the argv: %s", run)
}
}
// A command of the operator's own and the config this tool has to hand
// over are the same argv. Their command may point at another config, or
// at something that is not the server - so this refuses rather than
// merging and being quietly wrong about which server came up.
func TestContainerCutoverRefusesToMergeACommandWithTheConfig(t *testing.T) {
doc := strings.Replace(inspectJSON(t, nil, nil),
`"Cmd":["--config","/etc/stalwart/config.json"]`, `"Cmd":["--config","/srv/mine.toml"]`, 1)
_, _, err := runContainerFor(t, doc)
if err == nil {
t.Fatal("want a refusal when an overridden command collides with the migrated config")
}
for _, want := range []string{"/srv/mine.toml", containerConfig, "by hand"} {
if !strings.Contains(err.Error(), want) {
t.Errorf("refusal should name %q, got: %v", want, err)
}
}
}
// A patch bump converts nothing, so there is no config to point at and
// the container keeps the command its image gives it.
func TestContainerCutoverKeepsTheImageCommandWithoutAConfig(t *testing.T) {
log := fakeDockerCutover(t, inspectJSON(t, nil, nil))
store := checkpoint.NewStore(t.TempDir())
rs, _ := store.Create("0.16.14", "0.16.19")
if err := runContainerCutover(context.Background(), rs, noopStep(store, rs), ContainerOptions{
ContainerName: "stalwart", StagedImage: "sha256:new", PreserveDir: t.TempDir(),
}); err != nil {
t.Fatalf("runContainerCutover: %v", err)
}
if run := runLine(t, log); !strings.HasSuffix(strings.TrimSpace(run), "sha256:new") {
t.Errorf("run should end at the image, with no argv of its own: %s", run)
}
}
func noopStep(store *checkpoint.Store, rs *checkpoint.RunState) stepFunc {
return func(name string, fn func() (checkpoint.StepOutcome, error)) error {
_, err := store.RunStep(rs, checkpoint.PhaseCutover, name, fn)
return err
}
}
// runLine is the `docker run` the fake recorded.
func runLine(t *testing.T, log string) string {
t.Helper()
data, err := os.ReadFile(log)
if err != nil {
t.Fatal(err)
}
for _, line := range strings.Split(string(data), "\n") {
if strings.HasPrefix(line, "run ") {
return line
}
}
t.Fatalf("no `docker run` in the recorded arguments:\n%s", data)
return ""
}