Run the recovery cycle in a container
The second implementation of the launcher seam: a throwaway container from the staged image, against the live container's own mounts, with recovery mode's two environment variables. Everything the cycle does afterwards - the settings apply, the health check, the boot check - is unchanged, which was the point of putting the seam there. Process gained an unexported start(cmd, what) so a container shares its supervision rather than reimplementing it. For a container the child process is the `docker run` client, so the captured output is the server's own: a recovery boot that dies on a bind conflict says so here, where a detached container would have left it in `docker logs` for nobody. Stopping is the part that needed care, because the client and the container are two things and conflating them loses a store. Signals do reach the container through an attached client, so the ordinary path is fine - but Process.Stop escalates to SIGKILL when the grace period expires, and killing the client does not kill the container. It would be left running, holding the store open, while the run moved on to the next phase believing it had stopped. So the container is stopped by name with docker's own timeout, and its absence confirmed afterwards; a container still running after being told to stop is an error loud enough to halt the run, because nothing may touch that data until it is gone. Two refusals rather than defaults. No image, and no mounts - the second because a recovery container with no mounts would migrate an empty directory and report success, which is the worst outcome available here. Eight tests drive a fake docker: the assembled run command, recovery variables present in recovery mode and absent in an ordinary boot, both refusals, the stop reaching the container by name, the survived-container error, an already-removed container being the normal --rm path rather than a failure, and output capture. Two races in those tests are worth recording, because both were real and neither was in the code. Launch returns once the OS has started the client, which is before the shell it started has run anything, so reading the invocation log or the output immediately raced them; both now poll. And a forked `sleep` in the fake outlived its shell, held the output pipe open and made Wait block for the sleep's full duration - `exec` in the fake fixes it, and the same shape would affect any child that forks.
This commit is contained in:
@@ -114,11 +114,21 @@ func (p *Process) Start(ctx context.Context, o ProcessOptions) error {
|
||||
|
||||
cmd := exec.CommandContext(ctx, o.BinaryPath, "--config", o.ConfigPath)
|
||||
cmd.Env = append(os.Environ(), env...)
|
||||
return p.start(cmd, o.BinaryPath)
|
||||
}
|
||||
|
||||
// start attaches the output buffer and launches cmd. Split out of Start so
|
||||
// a deployment that runs the target version some other way - a container,
|
||||
// where the child is `docker run` rather than the server itself - gets the
|
||||
// same supervision: the same captured output, and the same Stop.
|
||||
//
|
||||
// what names the thing being started, for the error if it will not.
|
||||
func (p *Process) start(cmd *exec.Cmd, what string) error {
|
||||
p.output = &outputBuffer{}
|
||||
cmd.Stdout = p.output
|
||||
cmd.Stderr = p.output
|
||||
if err := cmd.Start(); err != nil {
|
||||
return fmt.Errorf("recovery: start %s: %w", o.BinaryPath, err)
|
||||
return fmt.Errorf("recovery: start %s: %w", what, err)
|
||||
}
|
||||
p.cmd = cmd
|
||||
return nil
|
||||
|
||||
Reference in New Issue
Block a user