The second implementation of the launcher seam: a throwaway container from
the staged image, against the live container's own mounts, with recovery
mode's two environment variables. Everything the cycle does afterwards -
the settings apply, the health check, the boot check - is unchanged, which
was the point of putting the seam there.
Process gained an unexported start(cmd, what) so a container shares its
supervision rather than reimplementing it. For a container the child
process is the `docker run` client, so the captured output is the server's
own: a recovery boot that dies on a bind conflict says so here, where a
detached container would have left it in `docker logs` for nobody.
Stopping is the part that needed care, because the client and the container
are two things and conflating them loses a store. Signals do reach the
container through an attached client, so the ordinary path is fine - but
Process.Stop escalates to SIGKILL when the grace period expires, and
killing the client does not kill the container. It would be left running,
holding the store open, while the run moved on to the next phase believing
it had stopped. So the container is stopped by name with docker's own
timeout, and its absence confirmed afterwards; a container still running
after being told to stop is an error loud enough to halt the run, because
nothing may touch that data until it is gone.
Two refusals rather than defaults. No image, and no mounts - the second
because a recovery container with no mounts would migrate an empty
directory and report success, which is the worst outcome available here.
Eight tests drive a fake docker: the assembled run command, recovery
variables present in recovery mode and absent in an ordinary boot, both
refusals, the stop reaching the container by name, the survived-container
error, an already-removed container being the normal --rm path rather than
a failure, and output capture.
Two races in those tests are worth recording, because both were real and
neither was in the code. Launch returns once the OS has started the client,
which is before the shell it started has run anything, so reading the
invocation log or the output immediately raced them; both now poll. And a
forked `sleep` in the fake outlived its shell, held the output pipe open
and made Wait block for the sleep's full duration - `exec` in the fake
fixes it, and the same shape would affect any child that forks.
A dry run against a real 0.15.5 instance failed with:
recovery mode did not come up: http://127.0.0.1:8081/ did not become
reachable within 1m0s: connect: connection refused
Stalwart had explained itself immediately - "Failed to bind to [::]:8080:
Address already in use" - into a pipe nothing was reading. Diagnosing a
one-line problem took several rounds because the tool threw away the only
evidence. Anything that reports a supervised process failing has to be able
to say why.
Process now captures the child's combined stdout and stderr into a bounded
buffer (64 KiB, keeping the most recent output, with truncation marked
rather than silent - a dead server's reason is at the end of its log), and
exposes it via Output(). recovery.Run appends it to both the startup-timeout
and settings-apply failures, and validate.BootCheck to its boot failure.
Fixes a second bug found while testing the first: Stop returned early when
Signal reported the process had already exited, so cmd.Wait was never
called. Wait is what reaps the child AND waits for the goroutines copying
its output - so the output was discarded in exactly the case where it
matters most, the server dying on its own. os.ErrProcessDone is now treated
as "already gone, still reap it".
The test reproduces the original failure shape: hold the port, start the
helper, let the health check time out, and assert the child's own bind
error survived. Confirmed against the smoke VM too - the same run now ends
with Stalwart's "Address already in use (os error 98)" printed inside the
tool's error.
GPLv3's "How to Apply These Terms" asks for a notice in each source file;
this is the modern two-line SPDX form of it rather than the full paragraph.
82 files, including tests.
The blank line after the header is load-bearing. In Go a comment block
immediately preceding `package X` becomes the package doc comment, so
without the separator the SPDX lines would be absorbed into the doc for the
eleven packages whose doc.go (or main.go) opens with one, and `go doc` would
print them. Verified it doesn't.
In-place upgrade tool for Stalwart Mail Server (0.15.5 -> latest) with
checkpointed rollback and post-migration validation. Design stage; see
ARCHITECTURE.md.
jcoffey-dev