Implement internal/rollback and the service control it needs
Rollback was the one phase gating everything else: `run` without --dry-run refused because this tool could not undo a cutover it had committed to. That reason is now gone, and the refusal has narrowed to the fact that there is no real cutover to undo yet. internal/rollback implements ARCHITECTURE.md 4.8 as eight checkpointed steps under PhaseRollback: verify-backup, stop-service, preserve-failed-state, restore-data, restore-binary, restore-service-config, start-service, verify-rollback. Three things depart from what 4.8 specified, each for a reason: - The backup is re-verified against its manifest *before* the service is stopped, which the design didn't call out. Finding a corrupt backup is survivable while the failed instance is still up, and unsurvivable once its data directory has been moved aside. - BuildPlan is separate from Run, so every reason to refuse (closed rollback window, FoundationDB, no recorded backup, unknown deployment kind, missing database credentials) is found before anything is touched. The CLI prints that resolved plan and acts only with --yes. - The restore is re-verified against the same manifest after writing. A restore that put back truncated bytes and reported success would be worse than one that failed outright. Nothing from the failed attempt is deleted: the half-migrated data directory and the displaced binary are moved to .failed-<run-id> names, so a retry after the underlying issue is fixed still has both the evidence and the artifacts. Afterwards a reduced validation suite runs against the *restored* instance (version, reachability, directory counts) rather than assuming the restore worked. internal/service is a new package holding the systemd/Docker control this needs. It's separate rather than living inside internal/rollback because cutover will need the identical operations, and because the commands that can take mail delivery down belong in one auditable place - the same reasoning that makes stalwartapi the only thing speaking JMAP. preflight.DeploymentKind is now a type alias for service.Kind so detection and control can't drift apart. Its Active() reads `systemctl is-active`'s output rather than its exit status: systemctl exits non-zero for every non-active state, so exit-status logic would make "inactive" - the answer a rollback most needs - look like a failure to read the state at all. Also fixes a pre-existing bug in `status`: Go's flag package stops parsing at the first positional argument, so `status <run-id> --state-dir X` looked the run up in the default directory and reported it missing. `rollback` would have inherited the same footgun on a command whose flags decide what gets overwritten. Still open: `confirm` cannot set RollbackWindowClosed. Rollback honours the flag and refuses when it's set, but closing the window is the point of no return for the backups this restores from, so it should land with the retention policy 6 describes rather than before it. Verified end to end against a fake systemd deployment: half-migrated data restored to its original contents, failed state preserved, old binary reinstalled and reporting 0.15.5, unit restarted, and a re-run of the completed rollback inert.
This commit is contained in:
@@ -0,0 +1,192 @@
|
||||
package service
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"os/exec"
|
||||
"strings"
|
||||
"time"
|
||||
)
|
||||
|
||||
// Kind is how a Stalwart instance is run, and therefore how it has to be
|
||||
// stopped and started. Preflight detects it (preflight.DetectDeploymentKind
|
||||
// is an alias for this type's detector-side constants) and records it in
|
||||
// the checkpoint's Topology, so a rollback days later controls the same
|
||||
// thing the original run observed rather than re-guessing.
|
||||
type Kind string
|
||||
|
||||
const (
|
||||
Systemd Kind = "systemd"
|
||||
Docker Kind = "docker"
|
||||
Unknown Kind = "unknown"
|
||||
)
|
||||
|
||||
// Options names the thing to control. Only the field matching Kind is used.
|
||||
type Options struct {
|
||||
Kind Kind
|
||||
UnitName string // systemd; defaults to "stalwart"
|
||||
ContainerName string // docker; defaults to "stalwart"
|
||||
}
|
||||
|
||||
// Controller stops and starts one Stalwart deployment. It is deliberately
|
||||
// the only thing in this tool that shells out to systemctl or docker, for
|
||||
// the same reason stalwartapi is the only thing that speaks JMAP: the
|
||||
// commands that can take mail delivery down belong in one auditable place,
|
||||
// not scattered across the phases that happen to need them.
|
||||
type Controller interface {
|
||||
// Stop stops the service and returns once the command reports success.
|
||||
// It does not wait for the process to actually be gone - use WaitFor
|
||||
// for that, since both systemd and docker can report success while the
|
||||
// unit is still shutting down.
|
||||
Stop(ctx context.Context) error
|
||||
// Start starts the service.
|
||||
Start(ctx context.Context) error
|
||||
// Active reports whether the service is currently running (or still
|
||||
// transitioning into or out of running). An error means the state
|
||||
// couldn't be determined at all - which is different from, and must
|
||||
// never be silently collapsed into, "not running".
|
||||
Active(ctx context.Context) (bool, error)
|
||||
// ReloadConfig re-reads unit/service definitions after one has been
|
||||
// rewritten on disk. It's a no-op for deployments that don't have such
|
||||
// a step, so callers never need to branch on Kind.
|
||||
ReloadConfig(ctx context.Context) error
|
||||
// Target describes what this controller acts on, for operator-facing
|
||||
// messages ("stopped systemd unit stalwart").
|
||||
Target() string
|
||||
}
|
||||
|
||||
// New returns a Controller for the given deployment. It refuses an Unknown
|
||||
// (or unrecognized) kind rather than guessing: picking the wrong mechanism
|
||||
// here means a rollback that reports "service stopped" while the old
|
||||
// instance is still running and holding the data directory open, which is
|
||||
// exactly the kind of quiet wrongness this tool exists to avoid.
|
||||
func New(o Options) (Controller, error) {
|
||||
switch o.Kind {
|
||||
case Systemd:
|
||||
unit := o.UnitName
|
||||
if unit == "" {
|
||||
unit = "stalwart"
|
||||
}
|
||||
return &systemdController{unit: unit}, nil
|
||||
case Docker:
|
||||
name := o.ContainerName
|
||||
if name == "" {
|
||||
name = "stalwart"
|
||||
}
|
||||
return &dockerController{container: name}, nil
|
||||
case Unknown, "":
|
||||
return nil, fmt.Errorf("service: deployment kind is unknown - this tool won't guess how to stop Stalwart; re-run preflight, or name the systemd unit or docker container explicitly")
|
||||
default:
|
||||
return nil, fmt.Errorf("service: unsupported deployment kind %q", o.Kind)
|
||||
}
|
||||
}
|
||||
|
||||
// WaitFor polls c.Active until it reports want, or timeout elapses. Both
|
||||
// systemctl and docker return as soon as the *request* to stop succeeded,
|
||||
// so without this a caller would move on to overwriting the data directory
|
||||
// while the old process still had it open.
|
||||
func WaitFor(ctx context.Context, c Controller, want bool, timeout time.Duration) error {
|
||||
deadline := time.Now().Add(timeout)
|
||||
var lastErr error
|
||||
for {
|
||||
active, err := c.Active(ctx)
|
||||
if err != nil {
|
||||
lastErr = err
|
||||
} else if active == want {
|
||||
return nil
|
||||
}
|
||||
if !time.Now().Before(deadline) {
|
||||
break
|
||||
}
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
return ctx.Err()
|
||||
case <-time.After(250 * time.Millisecond):
|
||||
}
|
||||
}
|
||||
state := "stopped"
|
||||
if want {
|
||||
state = "running"
|
||||
}
|
||||
if lastErr != nil {
|
||||
return fmt.Errorf("service: %s was still not %s after %s, and its state couldn't be read: %w", c.Target(), state, timeout, lastErr)
|
||||
}
|
||||
return fmt.Errorf("service: %s was still not %s after %s", c.Target(), state, timeout)
|
||||
}
|
||||
|
||||
type systemdController struct{ unit string }
|
||||
|
||||
func (s *systemdController) Target() string { return "systemd unit " + s.unit }
|
||||
|
||||
func (s *systemdController) Stop(ctx context.Context) error { return s.run(ctx, "stop") }
|
||||
func (s *systemdController) Start(ctx context.Context) error { return s.run(ctx, "start") }
|
||||
|
||||
func (s *systemdController) ReloadConfig(ctx context.Context) error {
|
||||
return runCommand(ctx, "systemctl", "daemon-reload")
|
||||
}
|
||||
|
||||
func (s *systemdController) run(ctx context.Context, verb string) error {
|
||||
return runCommand(ctx, "systemctl", verb, s.unit)
|
||||
}
|
||||
|
||||
// Active maps `systemctl is-active` output rather than its exit status:
|
||||
// the command exits non-zero for every not-active state, so treating a
|
||||
// non-zero exit as a failure to read the state would make "inactive" -
|
||||
// the answer we most want - look like an error.
|
||||
func (s *systemdController) Active(ctx context.Context) (bool, error) {
|
||||
out, err := exec.CommandContext(ctx, "systemctl", "is-active", s.unit).Output()
|
||||
state := strings.TrimSpace(string(out))
|
||||
switch state {
|
||||
case "active", "activating", "reloading", "deactivating":
|
||||
// deactivating counts as active on purpose: it means the old
|
||||
// process is still there, which is precisely what a caller waiting
|
||||
// for a clean stop must not mistake for "gone".
|
||||
return true, nil
|
||||
case "inactive", "failed":
|
||||
return false, nil
|
||||
}
|
||||
if err != nil {
|
||||
return false, fmt.Errorf("service: systemctl is-active %s: %w (output: %q)", s.unit, err, state)
|
||||
}
|
||||
return false, fmt.Errorf("service: systemctl is-active %s returned unrecognized state %q", s.unit, state)
|
||||
}
|
||||
|
||||
type dockerController struct{ container string }
|
||||
|
||||
func (d *dockerController) Target() string { return "docker container " + d.container }
|
||||
|
||||
func (d *dockerController) Stop(ctx context.Context) error {
|
||||
return runCommand(ctx, "docker", "stop", d.container)
|
||||
}
|
||||
|
||||
func (d *dockerController) Start(ctx context.Context) error {
|
||||
return runCommand(ctx, "docker", "start", d.container)
|
||||
}
|
||||
|
||||
// ReloadConfig is a no-op: a container has no equivalent of daemon-reload -
|
||||
// a changed Compose file takes effect when the container is recreated, and
|
||||
// recreating containers is beyond what this controller does.
|
||||
func (d *dockerController) ReloadConfig(context.Context) error { return nil }
|
||||
|
||||
func (d *dockerController) Active(ctx context.Context) (bool, error) {
|
||||
out, err := exec.CommandContext(ctx, "docker", "inspect", "-f", "{{.State.Running}}", d.container).Output()
|
||||
state := strings.TrimSpace(string(out))
|
||||
switch state {
|
||||
case "true":
|
||||
return true, nil
|
||||
case "false":
|
||||
return false, nil
|
||||
}
|
||||
if err != nil {
|
||||
return false, fmt.Errorf("service: docker inspect %s: %w (output: %q)", d.container, err, state)
|
||||
}
|
||||
return false, fmt.Errorf("service: docker inspect %s returned unrecognized state %q", d.container, state)
|
||||
}
|
||||
|
||||
func runCommand(ctx context.Context, name string, args ...string) error {
|
||||
out, err := exec.CommandContext(ctx, name, args...).CombinedOutput()
|
||||
if err != nil {
|
||||
return fmt.Errorf("service: %s %s: %w (output: %s)", name, strings.Join(args, " "), err, strings.TrimSpace(string(out)))
|
||||
}
|
||||
return nil
|
||||
}
|
||||
Reference in New Issue
Block a user