Add cutover; drop rollback in favour of operator-provided recovery
Two changes that arrived together: the cutover phase (ARCHITECTURE.md 4.5) is implemented, and the rollback phase is deleted. Recovery from a failed migration is now explicitly the operator's own snapshot or backup, and out of scope for this tool. internal/cutover implements 4.5 as seven checkpointed steps: verify the staged binary's version, install it, preserve and rewrite the service definition, reload, start, wait for a healthy JMAP session, recalculate quotas. The unit is rewritten in place rather than generated from a template. An operator's unit carries hardening options, limits and dependencies this tool has no business having an opinion about, and regenerating it would silently drop them. It repoints ExecStart (preserving systemd's -@:+! prefix characters and every argument after the executable), updates --config, and strips recovery-mode Environment lines - leaving STALWART_RECOVERY_MODE=1 set would recovery-boot the service on every restart, forever. It refuses on a unit with no ExecStart, and on an Environment line mixing a recovery variable with others: a line it only partly understands is one it must not edit. Quota recalculation is the one step allowed to fail without failing the phase. Its wire format is grounded in Stalwart's x:Task schema reference - Task/set creating one AccountMaintenance per account with maintenanceType recalculateQuota - but the upgrade guide only documents the WebUI path, so two details remain inferred and are called out in stalwartapi/task.go: whether the schema's "read-only" annotation on accountId/maintenanceType means "immutable after creation", and whether a finished task simply leaves the queue (TaskStatus documents Pending/Retry/Failed with no success state). Warning rather than failing is the honest response to that uncertainty, and stale counters are an accounting problem next to calling for a restore of a machine that is otherwise migrated and serving mail. Docker deployments are refused outright: cutting a container over means pulling an image and recreating it, not swapping a binary. On removing rollback. The implementation worked and was tested, and it was removed because restoring bytes correctly is not the hard part. It copied file contents and permissions and verified every restored file against a manifest - and did not preserve ownership. Run as root, as this tool requires, it would have produced a byte-perfect, checksum-verified, root-owned data directory that Stalwart, running as its own user, could not open, and it would have reported success. The PostgreSQL path was worse: pg_dump without --clean emits CREATE TABLE + COPY, which fails replaying into a database whose tables still exist, and the ON_ERROR_STOP=1 added so a half-applied restore couldn't be reported as success turned that into a hard failure. None of it had ever run against a real server. A filesystem snapshot has none of these failure modes, because it never lost the metadata to begin with. So cutover's gate is no longer rollback.CanRollBack but an explicit RecoveryPointConfirmed acknowledgement. That is an assertion, not a check - this tool cannot verify someone else's snapshot - and its only value is that nobody migrates a production mail server having never been asked the question. Two consequences are accepted deliberately: restoring any pre-migration recovery point discards mail delivered since, and a failed migration now stops and reports rather than undoing itself. What the tool still does to make a manual restore easier: the old binary is preserved and never deleted, the original service definition is preserved before the rewrite, the settings and principals dumps stay on disk, and every artifact path and checksum stays in the checkpoint where `status <run-id>` can print it. Also removed: the `confirm` command stub and RollbackWindowClosed, whose only purpose was closing a rollback window that no longer exists, and checkpoint.PhaseRollback. Old state.json files still load - JSON ignores the now-unknown field. Still open, and recorded in 8: cutover ignores systemd drop-ins, so an ExecStart or Environment override in stalwart.service.d/*.conf is invisible to the rewrite - including the recovery variable it exists to strip; nothing prevents concurrent runs on the same run-id; and nothing in this repo has ever run against a real Stalwart, real systemd, or a real store.
This commit is contained in:
@@ -0,0 +1,497 @@
|
||||
package cutover
|
||||
|
||||
import (
|
||||
"context"
|
||||
"crypto/sha256"
|
||||
"encoding/hex"
|
||||
"fmt"
|
||||
"io"
|
||||
"net/http"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"github.com/LINUXexpert-org/stalwart-migrator/internal/checkpoint"
|
||||
"github.com/LINUXexpert-org/stalwart-migrator/internal/preflight"
|
||||
"github.com/LINUXexpert-org/stalwart-migrator/internal/service"
|
||||
"github.com/LINUXexpert-org/stalwart-migrator/internal/stalwartapi"
|
||||
)
|
||||
|
||||
// Artifact names this phase records. ArtifactServiceUnit is the preserved
|
||||
// copy of the service definition as it was before this phase rewrote it -
|
||||
// recovery is out of scope for this tool (see ARCHITECTURE.md §4.8), but
|
||||
// the operator putting a machine back by hand should not also have to
|
||||
// reconstruct their unit file from memory.
|
||||
const (
|
||||
ArtifactNewBinary = "new-binary"
|
||||
ArtifactServiceUnit = "service-unit"
|
||||
)
|
||||
|
||||
// Options configures the cutover phase - ARCHITECTURE.md §4.5.
|
||||
type Options struct {
|
||||
// StagedBinaryPath is the already-downloaded target-version binary.
|
||||
// Cutover verifies its version before installing it and never
|
||||
// downloads anything itself.
|
||||
StagedBinaryPath string
|
||||
// BinaryPath is where the staged binary is installed - the path the
|
||||
// service definition runs. backup's preserve-binary step has already
|
||||
// moved the old binary aside, so this path is normally empty by now.
|
||||
BinaryPath string
|
||||
|
||||
// ServiceUnitPath is the systemd unit to rewrite. ConfigPath, if set,
|
||||
// becomes the unit's --config argument.
|
||||
ServiceUnitPath string
|
||||
ConfigPath string
|
||||
|
||||
// RecoveryPointConfirmed is the operator asserting that a recovery
|
||||
// point exists for this machine. This tool does not take one, verify
|
||||
// one, or restore from one - see ARCHITECTURE.md §4.8 - so this is an
|
||||
// acknowledgement, not a check, and BuildPlan refuses without it. An
|
||||
// unverifiable assertion is weaker than a guarantee; making it explicit
|
||||
// at least means nobody migrates a production mail server having never
|
||||
// been asked the question.
|
||||
RecoveryPointConfirmed bool
|
||||
|
||||
Deployment service.Options
|
||||
Controller service.Controller
|
||||
|
||||
StartTimeout time.Duration // waiting for the service to report running; default 60s
|
||||
HealthTimeout time.Duration // waiting for it to answer JMAP; default 120s
|
||||
|
||||
AdminURL string
|
||||
AdminUser string
|
||||
AdminPassword string
|
||||
HTTPClient *http.Client
|
||||
|
||||
// RecalculateQuotas schedules the post-migration quota rebuild
|
||||
// (ARCHITECTURE.md §4.5's last step). It's needed when crossing the
|
||||
// 0.15/0.16 boundary, where Stalwart's own upgrade guide says quotas
|
||||
// were reset to zero and have to be rebuilt; a patch bump doesn't
|
||||
// touch them. TenantIDs additionally rebuilds tenant-level counters,
|
||||
// which only multi-tenant installs have.
|
||||
RecalculateQuotas bool
|
||||
TenantIDs []string
|
||||
QuotaTimeout time.Duration // default 30m; large installs legitimately take a while
|
||||
}
|
||||
|
||||
// Plan is what a cutover would do, resolved before anything is touched.
|
||||
type Plan struct {
|
||||
RunID string
|
||||
TargetVersion string
|
||||
Target string // what the service controller acts on
|
||||
|
||||
StagedBinaryPath string
|
||||
BinaryPath string
|
||||
ServiceUnitPath string
|
||||
ConfigPath string
|
||||
RecalculateQuotas bool
|
||||
}
|
||||
|
||||
func (p Plan) String() string {
|
||||
var b strings.Builder
|
||||
fmt.Fprintf(&b, "cutover plan for run %s:\n", p.RunID)
|
||||
fmt.Fprintf(&b, " 1. confirm %s really is %s, then install it as %s\n", p.StagedBinaryPath, p.TargetVersion, p.BinaryPath)
|
||||
fmt.Fprintf(&b, " 2. preserve %s, then point its ExecStart at the new binary and strip any recovery-mode env vars\n", p.ServiceUnitPath)
|
||||
fmt.Fprintf(&b, " 3. reload the service definition and start %s\n", p.Target)
|
||||
fmt.Fprint(&b, " 4. wait for it to answer an authenticated JMAP session request\n")
|
||||
if p.RecalculateQuotas {
|
||||
fmt.Fprint(&b, " 5. schedule per-account quota recalculation and wait for the task queue to drain\n")
|
||||
} else {
|
||||
fmt.Fprint(&b, " 5. skip quota recalculation - not needed for this upgrade path\n")
|
||||
}
|
||||
return b.String()
|
||||
}
|
||||
|
||||
// BuildPlan resolves the cutover and refuses up front for anything it
|
||||
// shouldn't attempt. The most important refusal is the first one: a cutover
|
||||
// is only allowed to proceed if this run could still be rolled back, which
|
||||
// is what makes "never commit to a change you can't undo" a property of the
|
||||
// code rather than a claim in a design document.
|
||||
func BuildPlan(rs *checkpoint.RunState, opts Options) (Plan, error) {
|
||||
p := Plan{
|
||||
RunID: rs.RunID, TargetVersion: rs.TargetVersion,
|
||||
StagedBinaryPath: opts.StagedBinaryPath, BinaryPath: opts.BinaryPath,
|
||||
ServiceUnitPath: opts.ServiceUnitPath, ConfigPath: opts.ConfigPath,
|
||||
RecalculateQuotas: opts.RecalculateQuotas,
|
||||
}
|
||||
|
||||
if !opts.RecoveryPointConfirmed {
|
||||
return p, fmt.Errorf(
|
||||
"cutover: no recovery point has been confirmed for this run. This phase migrates a live mail server in place and this " +
|
||||
"tool has no way to undo that - recovery is the operator's own snapshot or backup (ARCHITECTURE.md §4.8). Take one, " +
|
||||
"verify you can actually restore from it, and re-run confirming that you have")
|
||||
}
|
||||
|
||||
kind := opts.Deployment.Kind
|
||||
if kind == "" {
|
||||
kind = service.Kind(rs.Topology.DeploymentKind)
|
||||
}
|
||||
if kind == service.Docker {
|
||||
return p, fmt.Errorf(
|
||||
"cutover: this run's deployment is a Docker container, where cutting over means pulling a new image and recreating the " +
|
||||
"container rather than swapping a binary and rewriting a unit. This tool doesn't automate that - do it by hand")
|
||||
}
|
||||
deployment := opts.Deployment
|
||||
deployment.Kind = kind
|
||||
controller := opts.Controller
|
||||
if controller == nil {
|
||||
var err error
|
||||
controller, err = service.New(deployment)
|
||||
if err != nil {
|
||||
return p, err
|
||||
}
|
||||
}
|
||||
p.Target = controller.Target()
|
||||
|
||||
if opts.StagedBinaryPath == "" {
|
||||
return p, fmt.Errorf("cutover: no staged target binary was given - cutover installs an already-downloaded binary, it doesn't fetch one")
|
||||
}
|
||||
if _, err := os.Stat(opts.StagedBinaryPath); err != nil {
|
||||
return p, fmt.Errorf("cutover: staged binary %s: %w", opts.StagedBinaryPath, err)
|
||||
}
|
||||
if opts.BinaryPath == "" {
|
||||
return p, fmt.Errorf("cutover: no path to install the new binary to was given")
|
||||
}
|
||||
if opts.ServiceUnitPath == "" {
|
||||
return p, fmt.Errorf("cutover: no service definition path was given - without one this phase can't point the service at the new binary")
|
||||
}
|
||||
if _, err := os.Stat(opts.ServiceUnitPath); err != nil {
|
||||
return p, fmt.Errorf("cutover: service definition %s: %w", opts.ServiceUnitPath, err)
|
||||
}
|
||||
return p, nil
|
||||
}
|
||||
|
||||
// Run executes ARCHITECTURE.md §4.5. Every step is checkpointed, and the
|
||||
// order is chosen so that the irreversible-looking parts happen only after
|
||||
// the reversible checks pass: the staged binary's version is confirmed
|
||||
// before it's installed, and the service definition is preserved before
|
||||
// it's rewritten.
|
||||
//
|
||||
// Quota recalculation is the one step allowed to fail without failing the
|
||||
// cutover, and that's deliberate. Stale quota counters are an accounting
|
||||
// problem, while a failed cutover is one an operator has to respond to by
|
||||
// restoring a machine that is otherwise migrated and serving mail
|
||||
// correctly. Calling for that over a counter would be the worse outcome, so
|
||||
// this step warns loudly and tells the operator how to finish it by hand.
|
||||
func Run(ctx context.Context, store *checkpoint.Store, rs *checkpoint.RunState, opts Options) (Report, error) {
|
||||
var report Report
|
||||
|
||||
plan, err := BuildPlan(rs, opts)
|
||||
if err != nil {
|
||||
report.Results = append(report.Results, CheckResult{Name: "plan", Status: StatusFail, Detail: err.Error()})
|
||||
return report, err
|
||||
}
|
||||
|
||||
controller := opts.Controller
|
||||
if controller == nil {
|
||||
deployment := opts.Deployment
|
||||
if deployment.Kind == "" {
|
||||
deployment.Kind = service.Kind(rs.Topology.DeploymentKind)
|
||||
}
|
||||
controller, err = service.New(deployment)
|
||||
if err != nil {
|
||||
report.Results = append(report.Results, CheckResult{Name: "plan", Status: StatusFail, Detail: err.Error()})
|
||||
return report, err
|
||||
}
|
||||
}
|
||||
|
||||
step := func(name string, fn func() (checkpoint.StepOutcome, error)) error {
|
||||
outcome, err := store.RunStep(rs, checkpoint.PhaseCutover, name, fn)
|
||||
if err != nil {
|
||||
report.Results = append(report.Results, CheckResult{Name: name, Status: StatusFail, Detail: err.Error()})
|
||||
return err
|
||||
}
|
||||
status := Status(outcome.Verdict)
|
||||
if status == "" {
|
||||
status = StatusOK
|
||||
}
|
||||
report.Results = append(report.Results, CheckResult{Name: name, Status: status, Detail: outcome.Detail})
|
||||
return nil
|
||||
}
|
||||
|
||||
if err := step("verify-staged-binary", func() (checkpoint.StepOutcome, error) {
|
||||
got, err := preflight.DetectVersion(ctx, plan.StagedBinaryPath)
|
||||
if err != nil {
|
||||
return checkpoint.StepOutcome{}, fmt.Errorf("couldn't read the staged binary's version: %w", err)
|
||||
}
|
||||
if rs.TargetVersion != "" && got != rs.TargetVersion {
|
||||
return checkpoint.StepOutcome{}, fmt.Errorf(
|
||||
"staged binary %s reports version %s, but this run targets %s - installing it would migrate to a version nobody planned for",
|
||||
plan.StagedBinaryPath, got, rs.TargetVersion)
|
||||
}
|
||||
return checkpoint.StepOutcome{Detail: fmt.Sprintf("staged binary %s reports %s, matching this run's target", plan.StagedBinaryPath, got), Extra: got}, nil
|
||||
}); err != nil {
|
||||
return report, err
|
||||
}
|
||||
|
||||
if err := step("install-binary", func() (checkpoint.StepOutcome, error) {
|
||||
sum, size, err := installBinary(plan.StagedBinaryPath, plan.BinaryPath)
|
||||
if err != nil {
|
||||
return checkpoint.StepOutcome{}, err
|
||||
}
|
||||
rs.RecordArtifact(ArtifactNewBinary, checkpoint.Artifact{Path: plan.BinaryPath, SHA256: sum, SizeBytes: size})
|
||||
return checkpoint.StepOutcome{Detail: fmt.Sprintf("installed %s as %s (%d bytes)", plan.StagedBinaryPath, plan.BinaryPath, size)}, nil
|
||||
}); err != nil {
|
||||
return report, err
|
||||
}
|
||||
|
||||
if err := step("update-service-definition", func() (checkpoint.StepOutcome, error) {
|
||||
preserved, err := preserveUnit(plan.ServiceUnitPath, rs.RunID)
|
||||
if err != nil {
|
||||
return checkpoint.StepOutcome{}, err
|
||||
}
|
||||
sum, size, err := hashFile(preserved)
|
||||
if err != nil {
|
||||
return checkpoint.StepOutcome{}, err
|
||||
}
|
||||
// Recorded before the rewrite, so a crash between preserving and
|
||||
// rewriting still leaves the original findable.
|
||||
rs.RecordArtifact(ArtifactServiceUnit, checkpoint.Artifact{Path: preserved, SHA256: sum, SizeBytes: size})
|
||||
|
||||
original, err := os.ReadFile(preserved)
|
||||
if err != nil {
|
||||
return checkpoint.StepOutcome{}, fmt.Errorf("cutover: read preserved unit %s: %w", preserved, err)
|
||||
}
|
||||
rewritten, err := RewriteUnit(string(original), plan.BinaryPath, plan.ConfigPath)
|
||||
if err != nil {
|
||||
return checkpoint.StepOutcome{}, err
|
||||
}
|
||||
if err := writeFileAtomic(plan.ServiceUnitPath, []byte(rewritten), 0o644); err != nil {
|
||||
return checkpoint.StepOutcome{}, err
|
||||
}
|
||||
return checkpoint.StepOutcome{
|
||||
Detail: fmt.Sprintf("pointed %s at %s; the original is preserved at %s", plan.ServiceUnitPath, plan.BinaryPath, preserved),
|
||||
Extra: preserved,
|
||||
}, nil
|
||||
}); err != nil {
|
||||
return report, err
|
||||
}
|
||||
|
||||
if err := step("reload-service-definition", func() (checkpoint.StepOutcome, error) {
|
||||
if err := controller.ReloadConfig(ctx); err != nil {
|
||||
return checkpoint.StepOutcome{}, err
|
||||
}
|
||||
return checkpoint.StepOutcome{Detail: "service manager re-read the updated definition"}, nil
|
||||
}); err != nil {
|
||||
return report, err
|
||||
}
|
||||
|
||||
startTimeout := opts.StartTimeout
|
||||
if startTimeout <= 0 {
|
||||
startTimeout = 60 * time.Second
|
||||
}
|
||||
if err := step("start-service", func() (checkpoint.StepOutcome, error) {
|
||||
if err := controller.Start(ctx); err != nil {
|
||||
return checkpoint.StepOutcome{}, err
|
||||
}
|
||||
if err := service.WaitFor(ctx, controller, true, startTimeout); err != nil {
|
||||
return checkpoint.StepOutcome{}, err
|
||||
}
|
||||
return checkpoint.StepOutcome{Detail: fmt.Sprintf("%s is running the migrated instance", controller.Target())}, nil
|
||||
}); err != nil {
|
||||
return report, err
|
||||
}
|
||||
|
||||
healthTimeout := opts.HealthTimeout
|
||||
if healthTimeout <= 0 {
|
||||
healthTimeout = 120 * time.Second
|
||||
}
|
||||
if err := step("wait-healthy", func() (checkpoint.StepOutcome, error) {
|
||||
if opts.AdminURL == "" {
|
||||
return checkpoint.StepOutcome{
|
||||
Verdict: string(StatusSkipped),
|
||||
Detail: "no admin URL configured - the service was started, but nothing confirmed it actually answers",
|
||||
}, nil
|
||||
}
|
||||
client := newClient(opts)
|
||||
if err := client.WaitForPing(ctx, healthTimeout); err != nil {
|
||||
return checkpoint.StepOutcome{}, fmt.Errorf("the migrated service started but never answered at %s within %s: %w", opts.AdminURL, healthTimeout, err)
|
||||
}
|
||||
return checkpoint.StepOutcome{Detail: fmt.Sprintf("migrated instance answered an authenticated JMAP session request at %s", opts.AdminURL)}, nil
|
||||
}); err != nil {
|
||||
return report, err
|
||||
}
|
||||
|
||||
// From here on, failures are warnings: the service is up and serving
|
||||
// mail, and rolling that back over a quota counter would be worse than
|
||||
// leaving the counter stale.
|
||||
quotaOutcome, quotaErr := store.RunStep(rs, checkpoint.PhaseCutover, "recalculate-quotas", func() (checkpoint.StepOutcome, error) {
|
||||
return recalculateQuotas(ctx, opts)
|
||||
})
|
||||
switch {
|
||||
case quotaErr != nil:
|
||||
report.Results = append(report.Results, CheckResult{
|
||||
Name: "recalculate-quotas", Status: StatusWarn,
|
||||
Detail: fmt.Sprintf("%v - the migration is complete and serving mail; finish this from the WebUI's Tasks panel "+
|
||||
"(\"Recalculate disk quotas\"), and note that until it runs, per-account usage counters read low", quotaErr),
|
||||
})
|
||||
default:
|
||||
status := Status(quotaOutcome.Verdict)
|
||||
if status == "" {
|
||||
status = StatusOK
|
||||
}
|
||||
report.Results = append(report.Results, CheckResult{Name: "recalculate-quotas", Status: status, Detail: quotaOutcome.Detail})
|
||||
}
|
||||
|
||||
return report, nil
|
||||
}
|
||||
|
||||
func newClient(opts Options) *stalwartapi.Client {
|
||||
return &stalwartapi.Client{
|
||||
BaseURL: opts.AdminURL, Username: opts.AdminUser, Password: opts.AdminPassword, HTTPClient: opts.HTTPClient,
|
||||
}
|
||||
}
|
||||
|
||||
func recalculateQuotas(ctx context.Context, opts Options) (checkpoint.StepOutcome, error) {
|
||||
if !opts.RecalculateQuotas {
|
||||
return checkpoint.StepOutcome{
|
||||
Verdict: string(StatusSkipped),
|
||||
Detail: "not needed for this upgrade path - quotas are only reset by the 0.15/0.16 schema migration",
|
||||
}, nil
|
||||
}
|
||||
if opts.AdminURL == "" {
|
||||
return checkpoint.StepOutcome{
|
||||
Verdict: string(StatusSkipped),
|
||||
Detail: "no admin URL configured - quota recalculation has to be triggered from the WebUI's Tasks panel by hand",
|
||||
}, nil
|
||||
}
|
||||
|
||||
timeout := opts.QuotaTimeout
|
||||
if timeout <= 0 {
|
||||
timeout = 30 * time.Minute
|
||||
}
|
||||
client := newClient(opts)
|
||||
|
||||
accountIDs, err := client.AccountIDs(ctx)
|
||||
if err != nil {
|
||||
return checkpoint.StepOutcome{}, fmt.Errorf("couldn't enumerate accounts to recalculate: %w", err)
|
||||
}
|
||||
if len(accountIDs) == 0 {
|
||||
return checkpoint.StepOutcome{Verdict: string(StatusSkipped), Detail: "the instance reports no accounts, so there are no quotas to rebuild"}, nil
|
||||
}
|
||||
|
||||
taskIDs, err := client.CreateQuotaRecalculationTasks(ctx, accountIDs)
|
||||
if err != nil {
|
||||
return checkpoint.StepOutcome{}, err
|
||||
}
|
||||
failures, err := client.WaitForTasks(ctx, taskIDs, timeout)
|
||||
if err != nil {
|
||||
return checkpoint.StepOutcome{}, err
|
||||
}
|
||||
if len(failures) > 0 {
|
||||
return checkpoint.StepOutcome{}, fmt.Errorf("%d of %d account quota task(s) failed: %v", len(failures), len(taskIDs), failures)
|
||||
}
|
||||
detail := fmt.Sprintf("rebuilt disk quotas for %d account(s)", len(accountIDs))
|
||||
|
||||
// Tenant totals aggregate the per-account numbers, so they can only run
|
||||
// once every account task above has finished - which is why this is
|
||||
// sequenced after the wait rather than scheduled alongside it.
|
||||
if len(opts.TenantIDs) > 0 {
|
||||
tenantTasks, err := client.CreateTenantQuotaRecalculationTasks(ctx, opts.TenantIDs)
|
||||
if err != nil {
|
||||
return checkpoint.StepOutcome{}, fmt.Errorf("%s, but scheduling tenant recalculation failed: %w", detail, err)
|
||||
}
|
||||
tenantFailures, err := client.WaitForTasks(ctx, tenantTasks, timeout)
|
||||
if err != nil {
|
||||
return checkpoint.StepOutcome{}, fmt.Errorf("%s, but waiting on tenant recalculation failed: %w", detail, err)
|
||||
}
|
||||
if len(tenantFailures) > 0 {
|
||||
return checkpoint.StepOutcome{}, fmt.Errorf("%s, but %d tenant quota task(s) failed: %v", detail, len(tenantFailures), tenantFailures)
|
||||
}
|
||||
detail += fmt.Sprintf(" and %d tenant(s)", len(opts.TenantIDs))
|
||||
}
|
||||
return checkpoint.StepOutcome{Detail: detail}, nil
|
||||
}
|
||||
|
||||
// installBinary copies the staged binary into place through a temp file in
|
||||
// the same directory, so the service definition never points at a
|
||||
// half-written executable. It's idempotent: a retry that finds the right
|
||||
// bytes already installed reports them rather than copying again.
|
||||
func installBinary(stagedPath, binaryPath string) (sha256Hex string, size int64, err error) {
|
||||
stagedSum, stagedSize, err := hashFile(stagedPath)
|
||||
if err != nil {
|
||||
return "", 0, err
|
||||
}
|
||||
if existingSum, existingSize, err := hashFile(binaryPath); err == nil && existingSum == stagedSum {
|
||||
return existingSum, existingSize, nil
|
||||
}
|
||||
|
||||
data, err := os.ReadFile(stagedPath)
|
||||
if err != nil {
|
||||
return "", 0, fmt.Errorf("cutover: read staged binary %s: %w", stagedPath, err)
|
||||
}
|
||||
if err := writeFileAtomic(binaryPath, data, 0o755); err != nil {
|
||||
return "", 0, err
|
||||
}
|
||||
return stagedSum, stagedSize, nil
|
||||
}
|
||||
|
||||
// preserveUnit copies the current service definition to
|
||||
// "<path>.pre-<run-id>" so an operator restoring by hand has the original,
|
||||
// and returns that path.
|
||||
// Idempotent: a retry finds the copy already there and keeps it, since the
|
||||
// file at path may by then be this phase's own rewrite.
|
||||
func preserveUnit(unitPath, runID string) (preservedPath string, err error) {
|
||||
preservedPath = fmt.Sprintf("%s.pre-%s", unitPath, runID)
|
||||
if _, err := os.Stat(preservedPath); err == nil {
|
||||
return preservedPath, nil
|
||||
} else if !os.IsNotExist(err) {
|
||||
return "", fmt.Errorf("cutover: stat %s: %w", preservedPath, err)
|
||||
}
|
||||
data, err := os.ReadFile(unitPath)
|
||||
if err != nil {
|
||||
return "", fmt.Errorf("cutover: read service definition %s: %w", unitPath, err)
|
||||
}
|
||||
perm := os.FileMode(0o644)
|
||||
if info, err := os.Stat(unitPath); err == nil {
|
||||
perm = info.Mode().Perm()
|
||||
}
|
||||
if err := writeFileAtomic(preservedPath, data, perm); err != nil {
|
||||
return "", err
|
||||
}
|
||||
return preservedPath, nil
|
||||
}
|
||||
|
||||
func writeFileAtomic(path string, data []byte, perm os.FileMode) error {
|
||||
tmp, err := os.CreateTemp(filepath.Dir(path), filepath.Base(path)+".tmp-*")
|
||||
if err != nil {
|
||||
return fmt.Errorf("cutover: create temp file next to %s: %w", path, err)
|
||||
}
|
||||
tmpPath := tmp.Name()
|
||||
defer os.Remove(tmpPath) // no-op once the rename succeeds
|
||||
|
||||
if _, err := tmp.Write(data); err != nil {
|
||||
tmp.Close()
|
||||
return fmt.Errorf("cutover: write %s: %w", tmpPath, err)
|
||||
}
|
||||
if err := tmp.Chmod(perm); err != nil {
|
||||
tmp.Close()
|
||||
return fmt.Errorf("cutover: chmod %s: %w", tmpPath, err)
|
||||
}
|
||||
if err := tmp.Sync(); err != nil {
|
||||
tmp.Close()
|
||||
return fmt.Errorf("cutover: sync %s: %w", tmpPath, err)
|
||||
}
|
||||
if err := tmp.Close(); err != nil {
|
||||
return fmt.Errorf("cutover: close %s: %w", tmpPath, err)
|
||||
}
|
||||
if err := os.Rename(tmpPath, path); err != nil {
|
||||
return fmt.Errorf("cutover: move %s into place at %s: %w", tmpPath, path, err)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func hashFile(path string) (sha256Hex string, size int64, err error) {
|
||||
f, err := os.Open(path)
|
||||
if err != nil {
|
||||
return "", 0, fmt.Errorf("cutover: hash %s: %w", path, err)
|
||||
}
|
||||
defer f.Close()
|
||||
h := sha256.New()
|
||||
n, err := io.Copy(h, f)
|
||||
if err != nil {
|
||||
return "", 0, fmt.Errorf("cutover: hash %s: %w", path, err)
|
||||
}
|
||||
return hex.EncodeToString(h.Sum(nil)), n, nil
|
||||
}
|
||||
@@ -0,0 +1,387 @@
|
||||
package cutover
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/LINUXexpert-org/stalwart-migrator/internal/checkpoint"
|
||||
"github.com/LINUXexpert-org/stalwart-migrator/internal/service"
|
||||
)
|
||||
|
||||
// fakeController stands in for systemd, recording call order.
|
||||
type fakeController struct {
|
||||
calls []string
|
||||
active bool
|
||||
startErr error
|
||||
}
|
||||
|
||||
func (f *fakeController) Stop(context.Context) error {
|
||||
f.calls = append(f.calls, "stop")
|
||||
f.active = false
|
||||
return nil
|
||||
}
|
||||
|
||||
func (f *fakeController) Start(context.Context) error {
|
||||
f.calls = append(f.calls, "start")
|
||||
if f.startErr != nil {
|
||||
return f.startErr
|
||||
}
|
||||
f.active = true
|
||||
return nil
|
||||
}
|
||||
|
||||
func (f *fakeController) Active(context.Context) (bool, error) { return f.active, nil }
|
||||
|
||||
func (f *fakeController) ReloadConfig(context.Context) error {
|
||||
f.calls = append(f.calls, "reload")
|
||||
return nil
|
||||
}
|
||||
|
||||
func (f *fakeController) Target() string { return "test service" }
|
||||
|
||||
func readFile(t *testing.T, path string) string {
|
||||
t.Helper()
|
||||
data, err := os.ReadFile(path)
|
||||
if os.IsNotExist(err) {
|
||||
return ""
|
||||
}
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
return string(data)
|
||||
}
|
||||
|
||||
// migratedRun builds a checkpoint for a run that backed up successfully and
|
||||
// is ready to cut over: the state this phase is actually invoked against.
|
||||
func migratedRun(t *testing.T) (store *checkpoint.Store, rs *checkpoint.RunState, opts Options) {
|
||||
t.Helper()
|
||||
root := t.TempDir()
|
||||
|
||||
store = checkpoint.NewStore(filepath.Join(root, "runs"))
|
||||
rs, err := store.Create("0.15.5", "0.16.14")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
rs.Topology = checkpoint.Topology{DeploymentKind: "systemd", StoreBackend: "rocksdb"}
|
||||
|
||||
staged := filepath.Join(root, "staged-stalwart")
|
||||
if err := os.WriteFile(staged, []byte("#!/bin/sh\necho 'stalwart 0.16.14'\n"), 0o755); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
unitPath := filepath.Join(root, "stalwart.service")
|
||||
if err := os.WriteFile(unitPath, []byte(realisticUnit), 0o644); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
return store, rs, Options{
|
||||
StagedBinaryPath: staged,
|
||||
BinaryPath: filepath.Join(root, "bin-stalwart"),
|
||||
ServiceUnitPath: unitPath,
|
||||
RecoveryPointConfirmed: true,
|
||||
Controller: &fakeController{},
|
||||
}
|
||||
}
|
||||
|
||||
// This tool can't undo a cutover, so the least it can do is refuse to
|
||||
// perform one without the operator having been asked the question.
|
||||
func TestBuildPlanRefusesWithoutAConfirmedRecoveryPoint(t *testing.T) {
|
||||
_, rs, opts := migratedRun(t)
|
||||
opts.RecoveryPointConfirmed = false
|
||||
|
||||
_, err := BuildPlan(rs, opts)
|
||||
if err == nil {
|
||||
t.Fatal("BuildPlan: want refusal when no recovery point has been confirmed, got nil")
|
||||
}
|
||||
if !strings.Contains(err.Error(), "no way to undo") {
|
||||
t.Errorf("error %q should be plain that this is irreversible for the tool", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestBuildPlanRefusesDockerDeployments(t *testing.T) {
|
||||
_, rs, opts := migratedRun(t)
|
||||
rs.Topology.DeploymentKind = string(service.Docker)
|
||||
opts.Controller = nil
|
||||
|
||||
_, err := BuildPlan(rs, opts)
|
||||
if err == nil {
|
||||
t.Fatal("BuildPlan: want refusal for a container deployment, got nil")
|
||||
}
|
||||
if !strings.Contains(err.Error(), "recreating the container") {
|
||||
t.Errorf("error %q should explain what cutting over a container would actually involve", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestBuildPlanRefusesAMissingStagedBinaryOrUnit(t *testing.T) {
|
||||
for _, tc := range []struct{ name, field string }{{"staged binary", "staged"}, {"service unit", "unit"}} {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
_, rs, opts := migratedRun(t)
|
||||
if tc.field == "staged" {
|
||||
opts.StagedBinaryPath = filepath.Join(t.TempDir(), "absent")
|
||||
} else {
|
||||
opts.ServiceUnitPath = filepath.Join(t.TempDir(), "absent.service")
|
||||
}
|
||||
if _, err := BuildPlan(rs, opts); err == nil {
|
||||
t.Fatalf("BuildPlan: want refusal for a missing %s, got nil", tc.name)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestRunInstallsRewritesAndStarts(t *testing.T) {
|
||||
store, rs, opts := migratedRun(t)
|
||||
ctl := opts.Controller.(*fakeController)
|
||||
|
||||
report, err := Run(context.Background(), store, rs, opts)
|
||||
if err != nil {
|
||||
t.Fatalf("Run: %v\n%s", err, report)
|
||||
}
|
||||
|
||||
if got := readFile(t, opts.BinaryPath); !strings.Contains(got, "0.16.14") {
|
||||
t.Errorf("installed binary = %q, want the staged one", got)
|
||||
}
|
||||
if info, err := os.Stat(opts.BinaryPath); err != nil || info.Mode().Perm() != 0o755 {
|
||||
t.Errorf("installed binary mode = %v (err %v), want 0755 - a non-executable binary won't start", info.Mode().Perm(), err)
|
||||
}
|
||||
unit := readFile(t, opts.ServiceUnitPath)
|
||||
if !strings.Contains(unit, "ExecStart="+opts.BinaryPath) {
|
||||
t.Errorf("unit not repointed at the new binary:\n%s", unit)
|
||||
}
|
||||
if got, want := strings.Join(ctl.calls, ","), "reload,start"; got != want {
|
||||
t.Errorf("controller calls = %q, want %q - the definition must be reloaded before the start", got, want)
|
||||
}
|
||||
if report.Blocking() {
|
||||
t.Errorf("report should be clean:\n%s", report)
|
||||
}
|
||||
}
|
||||
|
||||
// Recovery is the operator's own snapshot, but a snapshot revert doesn't
|
||||
// help someone who only wants their unit file back - so this phase has to
|
||||
// leave the original where they can find it.
|
||||
func TestRunPreservesTheOriginalServiceDefinition(t *testing.T) {
|
||||
store, rs, opts := migratedRun(t)
|
||||
if _, err := Run(context.Background(), store, rs, opts); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
art, found := rs.Artifacts[ArtifactServiceUnit]
|
||||
if !found {
|
||||
t.Fatalf("no %q artifact recorded; an operator restoring by hand would have to reconstruct the unit from memory", ArtifactServiceUnit)
|
||||
}
|
||||
preserved := readFile(t, art.Path)
|
||||
if !strings.Contains(preserved, "ExecStart=/usr/local/bin/stalwart --config /etc/stalwart/config.toml") {
|
||||
t.Errorf("preserved unit = %q, want the definition as it was before the rewrite", preserved)
|
||||
}
|
||||
if art.SHA256 == "" {
|
||||
t.Error("preserved unit artifact has no checksum")
|
||||
}
|
||||
}
|
||||
|
||||
// Installing a binary that isn't the version the run planned for would
|
||||
// migrate to a version nobody chose.
|
||||
func TestRunRefusesAStagedBinaryOfTheWrongVersion(t *testing.T) {
|
||||
store, rs, opts := migratedRun(t)
|
||||
if err := os.WriteFile(opts.StagedBinaryPath, []byte("#!/bin/sh\necho 'stalwart 0.16.9'\n"), 0o755); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
report, err := Run(context.Background(), store, rs, opts)
|
||||
if err == nil {
|
||||
t.Fatal("Run: want failure for a staged binary of the wrong version, got nil")
|
||||
}
|
||||
if _, statErr := os.Stat(opts.BinaryPath); !os.IsNotExist(statErr) {
|
||||
t.Error("the wrong-version binary was installed anyway")
|
||||
}
|
||||
if got := readFile(t, opts.ServiceUnitPath); !strings.Contains(got, "/usr/local/bin/stalwart") {
|
||||
t.Error("the service definition was rewritten despite the refusal")
|
||||
}
|
||||
if !report.Blocking() {
|
||||
t.Error("report should be blocking")
|
||||
}
|
||||
}
|
||||
|
||||
func TestRunResumesWithoutRedoingCompletedSteps(t *testing.T) {
|
||||
store, rs, opts := migratedRun(t)
|
||||
if _, err := Run(context.Background(), store, rs, opts); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
second := &fakeController{active: true}
|
||||
opts.Controller = second
|
||||
reloaded, err := store.Load(rs.RunID)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if _, err := Run(context.Background(), store, reloaded, opts); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if len(second.calls) != 0 {
|
||||
t.Errorf("second invocation called the controller %v, want nothing - every step was already done", second.calls)
|
||||
}
|
||||
}
|
||||
|
||||
func TestRunFailsWhenTheMigratedServiceNeverAnswers(t *testing.T) {
|
||||
store, rs, opts := migratedRun(t)
|
||||
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
w.WriteHeader(http.StatusBadGateway)
|
||||
}))
|
||||
defer srv.Close()
|
||||
opts.AdminURL = srv.URL
|
||||
opts.HealthTimeout = 300 * time.Millisecond
|
||||
|
||||
report, err := Run(context.Background(), store, rs, opts)
|
||||
if err == nil {
|
||||
t.Fatal("Run: want failure when the started service never answers, got nil")
|
||||
}
|
||||
if !strings.Contains(err.Error(), "never answered") {
|
||||
t.Errorf("error %q should distinguish 'started but not answering' from 'failed to start'", err)
|
||||
}
|
||||
if !report.Blocking() {
|
||||
t.Error("report should be blocking")
|
||||
}
|
||||
}
|
||||
|
||||
// Rolling back a migration that completed successfully, because a counter
|
||||
// didn't get rebuilt, would be worse than a stale counter.
|
||||
func TestRunWarnsRatherThanFailsWhenQuotaRecalculationFails(t *testing.T) {
|
||||
store, rs, opts := migratedRun(t)
|
||||
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
if r.Method == http.MethodGet {
|
||||
json.NewEncoder(w).Encode(map[string]any{"apiUrl": "/api"})
|
||||
return
|
||||
}
|
||||
w.WriteHeader(http.StatusInternalServerError) // the management API is unhappy
|
||||
}))
|
||||
defer srv.Close()
|
||||
opts.AdminURL = srv.URL
|
||||
opts.RecalculateQuotas = true
|
||||
|
||||
report, err := Run(context.Background(), store, rs, opts)
|
||||
if err != nil {
|
||||
t.Fatalf("a failed quota rebuild must not fail the cutover: %v\n%s", err, report)
|
||||
}
|
||||
if report.Blocking() {
|
||||
t.Errorf("report should not be blocking:\n%s", report)
|
||||
}
|
||||
var warned bool
|
||||
for _, res := range report.Results {
|
||||
if res.Name == "recalculate-quotas" {
|
||||
warned = res.Status == StatusWarn
|
||||
if !strings.Contains(res.Detail, "Tasks panel") {
|
||||
t.Errorf("warning %q should tell the operator how to finish it by hand", res.Detail)
|
||||
}
|
||||
}
|
||||
}
|
||||
if !warned {
|
||||
t.Errorf("quota failure should be a warning, not silence:\n%s", report)
|
||||
}
|
||||
}
|
||||
|
||||
func TestRunSkipsQuotaRecalculationOnThePatchFastPath(t *testing.T) {
|
||||
store, rs, opts := migratedRun(t)
|
||||
opts.RecalculateQuotas = false
|
||||
|
||||
report, err := Run(context.Background(), store, rs, opts)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
for _, res := range report.Results {
|
||||
if res.Name == "recalculate-quotas" {
|
||||
if res.Status != StatusSkipped {
|
||||
t.Errorf("recalculate-quotas = %s, want skip", res.Status)
|
||||
}
|
||||
if !strings.Contains(res.Detail, "0.15/0.16") {
|
||||
t.Errorf("skip detail %q should say why it isn't needed", res.Detail)
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// quotaServer answers the three calls quota recalculation makes: enumerate
|
||||
// accounts, schedule one task per account, then poll until the queue
|
||||
// drains. Tasks are removed once fetched, modelling a queue whose entries
|
||||
// are consumed when they run.
|
||||
func quotaServer(t *testing.T, accountIDs []string) (*httptest.Server, *[]map[string]any) {
|
||||
t.Helper()
|
||||
var scheduled []map[string]any
|
||||
queued := map[string]bool{}
|
||||
|
||||
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
if r.Method == http.MethodGet {
|
||||
json.NewEncoder(w).Encode(map[string]any{"apiUrl": "/api"})
|
||||
return
|
||||
}
|
||||
var body map[string]any
|
||||
json.NewDecoder(r.Body).Decode(&body)
|
||||
call := body["methodCalls"].([]any)[0].([]any)
|
||||
name := call[0].(string)
|
||||
args := call[1].(map[string]any)
|
||||
|
||||
switch name {
|
||||
case "x:Account/query":
|
||||
json.NewEncoder(w).Encode(map[string]any{"methodResponses": []any{
|
||||
[]any{"x:Account/query", map[string]any{"ids": accountIDs}, "q"},
|
||||
}})
|
||||
case "x:Task/set":
|
||||
created := map[string]any{}
|
||||
for creationID, obj := range args["create"].(map[string]any) {
|
||||
scheduled = append(scheduled, obj.(map[string]any))
|
||||
id := "task-" + creationID
|
||||
queued[id] = true
|
||||
created[creationID] = map[string]any{"id": id}
|
||||
}
|
||||
json.NewEncoder(w).Encode(map[string]any{"methodResponses": []any{
|
||||
[]any{"x:Task/set", map[string]any{"created": created}, "s"},
|
||||
}})
|
||||
case "x:Task/get":
|
||||
// Report every task as gone: it ran and left the queue.
|
||||
for _, raw := range args["ids"].([]any) {
|
||||
delete(queued, raw.(string))
|
||||
}
|
||||
json.NewEncoder(w).Encode(map[string]any{"methodResponses": []any{
|
||||
[]any{"x:Task/get", map[string]any{"list": []any{}}, "g"},
|
||||
}})
|
||||
default:
|
||||
t.Errorf("unexpected method call %s", name)
|
||||
}
|
||||
}))
|
||||
t.Cleanup(srv.Close)
|
||||
return srv, &scheduled
|
||||
}
|
||||
|
||||
func TestRunSchedulesOneQuotaTaskPerAccountAndWaits(t *testing.T) {
|
||||
store, rs, opts := migratedRun(t)
|
||||
srv, scheduled := quotaServer(t, []string{"a1", "a2", "a3"})
|
||||
opts.AdminURL = srv.URL
|
||||
opts.AdminUser = "admin"
|
||||
opts.RecalculateQuotas = true
|
||||
|
||||
report, err := Run(context.Background(), store, rs, opts)
|
||||
if err != nil {
|
||||
t.Fatalf("Run: %v\n%s", err, report)
|
||||
}
|
||||
if len(*scheduled) != 3 {
|
||||
t.Fatalf("scheduled %d task(s), want one per account", len(*scheduled))
|
||||
}
|
||||
for _, obj := range *scheduled {
|
||||
if obj["maintenanceType"] != "recalculateQuota" {
|
||||
t.Errorf("maintenanceType = %v, want recalculateQuota", obj["maintenanceType"])
|
||||
}
|
||||
}
|
||||
for _, res := range report.Results {
|
||||
if res.Name == "recalculate-quotas" {
|
||||
if res.Status != StatusOK {
|
||||
t.Errorf("recalculate-quotas = %s: %s", res.Status, res.Detail)
|
||||
}
|
||||
if !strings.Contains(res.Detail, "3 account(s)") {
|
||||
t.Errorf("detail %q should say how many accounts were rebuilt", res.Detail)
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,3 @@
|
||||
// Package cutover implements switching the live service onto the migrated instance: binary swap, service definition, restart, quota recalculation.
|
||||
// See ARCHITECTURE.md §4.5 for the design.
|
||||
package cutover
|
||||
@@ -0,0 +1,45 @@
|
||||
package cutover
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"strings"
|
||||
)
|
||||
|
||||
type Status string
|
||||
|
||||
const (
|
||||
StatusOK Status = "ok"
|
||||
StatusWarn Status = "warn"
|
||||
StatusSkipped Status = "skip"
|
||||
StatusFail Status = "fail"
|
||||
)
|
||||
|
||||
type CheckResult struct {
|
||||
Name string
|
||||
Status Status
|
||||
Detail string
|
||||
}
|
||||
|
||||
type Report struct {
|
||||
Results []CheckResult
|
||||
}
|
||||
|
||||
// Blocking reports whether anything failed outright. A warning does not
|
||||
// block: the one step allowed to warn is quota recalculation, which leaves
|
||||
// counters stale rather than mail unreachable (see Run).
|
||||
func (r Report) Blocking() bool {
|
||||
for _, res := range r.Results {
|
||||
if res.Status == StatusFail {
|
||||
return true
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
func (r Report) String() string {
|
||||
var b strings.Builder
|
||||
for _, res := range r.Results {
|
||||
fmt.Fprintf(&b, "[%-4s] %-24s %s\n", strings.ToUpper(string(res.Status)), res.Name, res.Detail)
|
||||
}
|
||||
return b.String()
|
||||
}
|
||||
@@ -0,0 +1,128 @@
|
||||
package cutover
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"strings"
|
||||
)
|
||||
|
||||
// recoveryEnvVars are the two variables that must never survive into the
|
||||
// live service definition. ARCHITECTURE.md §4.5 calls leaving
|
||||
// STALWART_RECOVERY_MODE=1 set a documented footgun, and it is: the service
|
||||
// would recovery-boot on every restart from then on, quietly, forever.
|
||||
//
|
||||
// This tool never puts them in a unit itself - internal/recovery runs
|
||||
// recovery mode as a supervised child process, not through systemd - so
|
||||
// finding them here means an operator followed the manual upgrade guide by
|
||||
// hand at some point. That's exactly the case worth catching.
|
||||
var recoveryEnvVars = []string{"STALWART_RECOVERY_MODE", "STALWART_RECOVERY_ADMIN"}
|
||||
|
||||
// RewriteUnit points a systemd unit's ExecStart at a new binary (and, if
|
||||
// configPath is non-empty, a new --config path) and strips any recovery-mode
|
||||
// environment lines, returning the rewritten file.
|
||||
//
|
||||
// It rewrites in place rather than generating a unit from a template: the
|
||||
// operator's unit is theirs, and it may carry hardening options, resource
|
||||
// limits, dependencies and overrides this tool has no business having an
|
||||
// opinion about. Replacing it with something generated would silently drop
|
||||
// all of that.
|
||||
//
|
||||
// It refuses rather than guesses in two cases: a unit with no ExecStart at
|
||||
// all, and an Environment line that mixes a recovery variable with other
|
||||
// variables. Both mean the file isn't shaped the way this rewrite assumes,
|
||||
// and editing it anyway risks producing a unit that starts something other
|
||||
// than what the operator intended.
|
||||
func RewriteUnit(unit, binaryPath, configPath string) (string, error) {
|
||||
lines := strings.Split(unit, "\n")
|
||||
out := make([]string, 0, len(lines))
|
||||
execStarts := 0
|
||||
|
||||
for _, line := range lines {
|
||||
trimmed := strings.TrimSpace(line)
|
||||
|
||||
if strings.HasPrefix(trimmed, "ExecStart=") {
|
||||
rewritten, err := rewriteExecStart(line, binaryPath, configPath)
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
execStarts++
|
||||
out = append(out, rewritten)
|
||||
continue
|
||||
}
|
||||
|
||||
if strings.HasPrefix(trimmed, "Environment=") || strings.HasPrefix(trimmed, "Environment ") {
|
||||
mentions, only := classifyEnvironmentLine(trimmed)
|
||||
if mentions && !only {
|
||||
return "", fmt.Errorf(
|
||||
"cutover: the unit's %q line sets a recovery-mode variable alongside others, and this tool won't edit a line it "+
|
||||
"only partly understands - remove the STALWART_RECOVERY_* assignment by hand and re-run", trimmed)
|
||||
}
|
||||
if mentions {
|
||||
continue // the whole line is recovery-only: drop it
|
||||
}
|
||||
}
|
||||
|
||||
out = append(out, line)
|
||||
}
|
||||
|
||||
if execStarts == 0 {
|
||||
return "", fmt.Errorf("cutover: the service definition has no ExecStart= line, so there's nothing to point at the new binary - is this the right unit file?")
|
||||
}
|
||||
return strings.Join(out, "\n"), nil
|
||||
}
|
||||
|
||||
// rewriteExecStart replaces the executable in an ExecStart line, preserving
|
||||
// every argument after it (and any leading whitespace or systemd prefix
|
||||
// characters like "-" or "@"), then updates --config if asked to.
|
||||
func rewriteExecStart(line, binaryPath, configPath string) (string, error) {
|
||||
indent := line[:len(line)-len(strings.TrimLeft(line, " \t"))]
|
||||
value := strings.TrimSpace(line)[len("ExecStart="):]
|
||||
|
||||
// systemd allows prefix characters on the executable ("-", "@", ":",
|
||||
// "+", "!"). Preserve whatever is there rather than dropping semantics
|
||||
// the operator chose deliberately.
|
||||
prefix := ""
|
||||
for len(value) > 0 && strings.ContainsRune("-@:+!", rune(value[0])) {
|
||||
prefix += string(value[0])
|
||||
value = value[1:]
|
||||
}
|
||||
|
||||
fields := strings.Fields(value)
|
||||
if len(fields) == 0 {
|
||||
return "", fmt.Errorf("cutover: the unit's ExecStart= line names no executable")
|
||||
}
|
||||
fields[0] = binaryPath
|
||||
|
||||
if configPath != "" {
|
||||
replaced := false
|
||||
for i := 0; i < len(fields)-1; i++ {
|
||||
if fields[i] == "--config" || fields[i] == "-c" {
|
||||
fields[i+1] = configPath
|
||||
replaced = true
|
||||
}
|
||||
}
|
||||
if !replaced {
|
||||
fields = append(fields, "--config", configPath)
|
||||
}
|
||||
}
|
||||
return indent + "ExecStart=" + prefix + strings.Join(fields, " "), nil
|
||||
}
|
||||
|
||||
// classifyEnvironmentLine reports whether an Environment= line mentions a
|
||||
// recovery variable at all, and whether that's all it sets.
|
||||
func classifyEnvironmentLine(trimmed string) (mentions, only bool) {
|
||||
value := trimmed[strings.Index(trimmed, "=")+1:]
|
||||
assignments := strings.Fields(value)
|
||||
if len(assignments) == 0 {
|
||||
return false, false
|
||||
}
|
||||
recoveryCount := 0
|
||||
for _, a := range assignments {
|
||||
a = strings.Trim(a, `"'`)
|
||||
for _, name := range recoveryEnvVars {
|
||||
if strings.HasPrefix(a, name+"=") {
|
||||
recoveryCount++
|
||||
}
|
||||
}
|
||||
}
|
||||
return recoveryCount > 0, recoveryCount == len(assignments)
|
||||
}
|
||||
@@ -0,0 +1,127 @@
|
||||
package cutover
|
||||
|
||||
import (
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
const realisticUnit = `[Unit]
|
||||
Description=Stalwart Mail Server
|
||||
After=network.target
|
||||
|
||||
[Service]
|
||||
Type=simple
|
||||
User=stalwart
|
||||
ExecStart=/usr/local/bin/stalwart --config /etc/stalwart/config.toml
|
||||
Restart=on-failure
|
||||
LimitNOFILE=65536
|
||||
ProtectSystem=strict
|
||||
ReadWritePaths=/var/lib/stalwart
|
||||
|
||||
[Install]
|
||||
WantedBy=multi-user.target
|
||||
`
|
||||
|
||||
func TestRewriteUnitRepointsExecStart(t *testing.T) {
|
||||
got, err := RewriteUnit(realisticUnit, "/usr/local/bin/stalwart", "/etc/stalwart/config.json")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if !strings.Contains(got, "ExecStart=/usr/local/bin/stalwart --config /etc/stalwart/config.json") {
|
||||
t.Errorf("ExecStart not repointed:\n%s", got)
|
||||
}
|
||||
}
|
||||
|
||||
// The operator's unit is theirs: hardening options, limits and paths this
|
||||
// tool has no opinion about must survive untouched.
|
||||
func TestRewriteUnitPreservesEverythingElse(t *testing.T) {
|
||||
got, err := RewriteUnit(realisticUnit, "/opt/stalwart/bin/stalwart", "")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
for _, want := range []string{
|
||||
"Description=Stalwart Mail Server", "User=stalwart", "Restart=on-failure",
|
||||
"LimitNOFILE=65536", "ProtectSystem=strict", "ReadWritePaths=/var/lib/stalwart",
|
||||
"WantedBy=multi-user.target",
|
||||
} {
|
||||
if !strings.Contains(got, want) {
|
||||
t.Errorf("rewrite dropped %q:\n%s", want, got)
|
||||
}
|
||||
}
|
||||
if !strings.Contains(got, "ExecStart=/opt/stalwart/bin/stalwart --config /etc/stalwart/config.toml") {
|
||||
t.Errorf("existing --config should be preserved when no new one is given:\n%s", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestRewriteUnitAddsConfigWhenTheUnitHasNone(t *testing.T) {
|
||||
got, err := RewriteUnit("[Service]\nExecStart=/usr/local/bin/stalwart\n", "/usr/local/bin/stalwart", "/etc/stalwart/config.json")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if !strings.Contains(got, "ExecStart=/usr/local/bin/stalwart --config /etc/stalwart/config.json") {
|
||||
t.Errorf("--config not added:\n%s", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestRewriteUnitKeepsSystemdExecPrefixes(t *testing.T) {
|
||||
got, err := RewriteUnit("[Service]\nExecStart=-@/old/stalwart --config /c\n", "/new/stalwart", "")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if !strings.Contains(got, "ExecStart=-@/new/stalwart --config /c") {
|
||||
t.Errorf("systemd exec prefix characters were dropped, changing what the unit means:\n%s", got)
|
||||
}
|
||||
}
|
||||
|
||||
// Leaving STALWART_RECOVERY_MODE=1 in the unit is the documented footgun
|
||||
// from §4.5: the service would recovery-boot on every restart, forever.
|
||||
func TestRewriteUnitStripsRecoveryEnvironmentLines(t *testing.T) {
|
||||
unit := `[Service]
|
||||
Environment=STALWART_RECOVERY_MODE=1
|
||||
Environment="STALWART_RECOVERY_ADMIN=admin:hunter2"
|
||||
Environment=RUST_LOG=info
|
||||
ExecStart=/usr/local/bin/stalwart
|
||||
`
|
||||
got, err := RewriteUnit(unit, "/usr/local/bin/stalwart", "")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
for _, gone := range []string{"STALWART_RECOVERY_MODE", "STALWART_RECOVERY_ADMIN"} {
|
||||
if strings.Contains(got, gone) {
|
||||
t.Errorf("%s survived the rewrite - the service would recovery-boot on every restart:\n%s", gone, got)
|
||||
}
|
||||
}
|
||||
if !strings.Contains(got, "Environment=RUST_LOG=info") {
|
||||
t.Errorf("unrelated Environment line was dropped:\n%s", got)
|
||||
}
|
||||
}
|
||||
|
||||
// A line this tool only partly understands is one it must not edit.
|
||||
func TestRewriteUnitRefusesAMixedEnvironmentLine(t *testing.T) {
|
||||
unit := "[Service]\nEnvironment=RUST_LOG=info STALWART_RECOVERY_MODE=1\nExecStart=/usr/local/bin/stalwart\n"
|
||||
_, err := RewriteUnit(unit, "/usr/local/bin/stalwart", "")
|
||||
if err == nil {
|
||||
t.Fatal("want refusal for an Environment line mixing recovery and other variables, got nil")
|
||||
}
|
||||
if !strings.Contains(err.Error(), "by hand") {
|
||||
t.Errorf("error %q should tell the operator what to do about it", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestRewriteUnitRefusesAUnitWithNoExecStart(t *testing.T) {
|
||||
_, err := RewriteUnit("[Unit]\nDescription=Something else entirely\n", "/usr/local/bin/stalwart", "")
|
||||
if err == nil {
|
||||
t.Fatal("want refusal for a unit with no ExecStart, got nil")
|
||||
}
|
||||
if !strings.Contains(err.Error(), "right unit file") {
|
||||
t.Errorf("error %q should question whether this is the right file", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestRewriteUnitHandlesMultipleExecStartLines(t *testing.T) {
|
||||
unit := "[Service]\nExecStart=\nExecStart=/old/stalwart --config /c\n"
|
||||
got, err := RewriteUnit(unit, "/new/stalwart", "")
|
||||
if err == nil {
|
||||
t.Fatalf("an empty ExecStart= names no executable and should be refused, got:\n%s", got)
|
||||
}
|
||||
}
|
||||
Reference in New Issue
Block a user