migrate: --repeat, keep syncing until the cutover #73

Merged
jcoffey-dev merged 1 commits from feat/migrate-repeat into main 2026-11-12 19:52:02 +00:00
6 changed files with 259 additions and 28 deletions

No files matched your search

+3 -1
View File
@@ -62,7 +62,9 @@ list of domains, accounts, groups, lists and aliases to set up again
elsewhere, without passwords, and `import-directory` sets them up on an
inbuxa server, with a new password for each user. `migrate` moves many
accounts with one plan file: the same steps for each account, several at
once, resuming where it stopped, with one report at the end. `guide`
once, resuming where it stopped, with one report at the end. With
`--repeat` it keeps syncing, each pass moving only what arrived since, so
the new server stays close behind the old one until the cutover. `guide`
walks through the same move in the browser: it asks where the mail is,
which accounts and which inbuxa server, previews, and runs the plan it
writes. The full
+16 -2
View File
@@ -659,6 +659,7 @@ server and counts what it would create, sends nothing and writes no file;
```
inbuxa-migrate migrate --plan <FILE> [--parallel <N>] [--again]
[--repeat <DURATION> [--max-passes <N>] [--until-quiet]]
[--account <EMAIL>]... [--dry-run] [-q] [--progress json] <DIR>
```
@@ -725,6 +726,17 @@ account, the state a later run resumes from, and the report.
every account step once more, done or not. Use it for the final pass
once the old server stops taking mail: the steps converge, so it only
moves what changed. `--account` limits a run to the accounts named.
- **Keep syncing until cutover.** `--repeat <DURATION>` runs the accounts
again and again, waiting that long between passes (`30s`, `5m`, `1h`),
each pass moving only what has arrived at the source since the last. The
setup runs once, before the first pass. Leave it running while the old
server still takes mail, so the new one stays close behind; then change
the MX and stop it, and the last pass is a short catch-up instead of the
whole mailbox. Stop it by interrupting it (Ctrl-C): a pass in flight is
resumable, so a later run picks up where it left off. `--max-passes <N>`
stops after N passes, and `--until-quiet` stops on the first pass that
moves nothing new -- the source has gone quiet and the two servers are in
step. Each `--repeat` pass re-runs every step, like `--again`.
- **The report.** `<DIR>/report.csv` has one row per step per account:
how it ended (`done`, `partial`, `failed`, `not run`, or
`done earlier`), the counts summed over the step's types, and, for one
@@ -735,8 +747,10 @@ account, the state a later run resumes from, and the report.
`start` (`accounts`, `parallel`, `dryRun`), `setup` (a setup step ended),
`step` (an account's step began, or ended with `result`, `counts` and
`detail`), `account` (an account ended: `email`, `n` of `total`,
`result`) and `end` (`done`, `unfinished`, `report`). Messages still go
to stderr.
`result`) and `end` (`pass`, `done`, `unfinished`, `report`). Under
`--repeat`, `end` closes each pass and carries its `pass` number, and a
`waiting` event (`seconds`) comes between passes. Messages still go to
stderr.
The work directory is made readable by its owner only, and so are the
logs. With `--dry-run`, each step runs with `--dry-run`, nothing is
+75
View File
@@ -779,6 +779,28 @@ struct MigrateArgs {
)]
again: bool,
#[arg(
long,
value_name = "DURATION",
help = "Keep syncing: after each pass wait this long and run again, each pass catching what has arrived since, until interrupted. The new server stays close to the old one, so the cutover moves only the last few minutes. E.g. 30s, 5m, 1h."
)]
repeat: Option<String>,
#[arg(
long,
value_name = "N",
requires = "repeat",
help = "With --repeat, stop after this many passes instead of running until interrupted"
)]
max_passes: Option<u32>,
#[arg(
long,
requires = "repeat",
help = "With --repeat, stop once a pass moves nothing new: the source has gone quiet and the servers are in step"
)]
until_quiet: bool,
#[arg(
long = "account",
value_name = "EMAIL",
@@ -1641,6 +1663,13 @@ fn resolve_import_directory(args: ImportDirectoryArgs) -> Result<Action, Error>
}
fn resolve_migrate(args: MigrateArgs) -> Result<Action, Error> {
let repeat = match args.repeat.as_deref() {
Some(s) => Some(parse_duration(s)?),
None => None,
};
if args.max_passes == Some(0) {
return Err(Error::Usage("--max-passes must be at least 1".to_owned()));
}
Ok(Action::Migrate(MigrateConfig {
plan: args.plan,
work: args.work,
@@ -1651,9 +1680,40 @@ fn resolve_migrate(args: MigrateArgs) -> Result<Action, Error> {
program: std::env::current_exe()?,
logger: Logger::from_flags(args.quiet, 0),
progress_json: args.progress.is_some(),
repeat,
max_passes: args.max_passes,
until_quiet: args.until_quiet,
}))
}
/// Parses a plain duration like `30s`, `5m`, `1h`, or a bare number of
/// seconds, for `--repeat`. Zero is rejected: a pass must not chase its own
/// tail with no pause.
fn parse_duration(s: &str) -> Result<std::time::Duration, Error> {
let s = s.trim();
let (num, unit_secs) = match s.strip_suffix(['s', 'm', 'h']) {
Some(rest) => {
let mult = match s.chars().last() {
Some('s') => 1,
Some('m') => 60,
Some('h') => 3600,
_ => unreachable!(),
};
(rest.trim(), mult)
}
None => (s, 1),
};
let n: u64 = num.parse().map_err(|_| {
Error::Usage(format!(
"--repeat {s:?}: want a duration like 30s, 5m or 1h"
))
})?;
if n == 0 {
return Err(Error::Usage("--repeat must be more than zero".to_owned()));
}
Ok(std::time::Duration::from_secs(n * unit_secs))
}
fn resolve_inspect(args: InspectArgs) -> Result<Action, Error> {
let target = match args.target.as_deref() {
Some(token) => Some(ObjectType::parse(token)?),
@@ -2174,6 +2234,21 @@ mod tests {
use super::*;
use std::env;
#[test]
fn parse_duration_reads_units_and_rejects_nonsense() {
use std::time::Duration;
assert_eq!(parse_duration("30s").unwrap(), Duration::from_secs(30));
assert_eq!(parse_duration("5m").unwrap(), Duration::from_secs(300));
assert_eq!(parse_duration("1h").unwrap(), Duration::from_secs(3600));
assert_eq!(parse_duration("45").unwrap(), Duration::from_secs(45));
assert_eq!(parse_duration(" 2m ").unwrap(), Duration::from_secs(120));
assert!(parse_duration("0s").is_err());
assert!(parse_duration("0").is_err());
assert!(parse_duration("").is_err());
assert!(parse_duration("5x").is_err());
assert!(parse_duration("m").is_err());
}
fn lock() -> std::sync::MutexGuard<'static, ()> {
static M: std::sync::Mutex<()> = std::sync::Mutex::new(());
M.lock().unwrap_or_else(|e| e.into_inner())
+6 -1
View File
@@ -141,8 +141,13 @@ fn run() -> i32 {
return match migrate::run(config) {
Ok(outcome) => {
if !json {
let passes = if outcome.passes > 1 {
format!(" after {} passes", outcome.passes)
} else {
String::new()
};
println!(
"accounts: {} done, {} not finished; report: {}",
"accounts: {} done, {} not finished{passes}; report: {}",
outcome.done,
outcome.unfinished,
outcome.report.display()
+98 -24
View File
@@ -19,6 +19,7 @@ use std::path::{Path, PathBuf};
use std::process::{Command, Stdio};
use std::sync::Mutex;
use std::sync::atomic::{AtomicUsize, Ordering};
use std::time::Duration;
use serde::{Deserialize, Serialize};
use serde_json::{Value, json};
@@ -43,6 +44,16 @@ pub struct MigrateConfig {
pub logger: Logger,
/// Progress as JSON lines on stdout, for a program that runs this one.
pub progress_json: bool,
/// Keep syncing: after each pass wait this long and run the accounts
/// again, so the new server stays close to the old one until the cutover.
/// `None` is a single pass. Setup runs once, before the first pass.
pub repeat: Option<Duration>,
/// With `repeat`, stop after this many passes instead of until the run is
/// interrupted. `None` is unbounded.
pub max_passes: Option<u32>,
/// With `repeat`, stop once a pass moves nothing new -- the source has
/// gone quiet, so the servers are in step and it is safe to cut over.
pub until_quiet: bool,
}
/// Writes one progress event, a JSON object on a line of its own, when
@@ -68,6 +79,8 @@ pub struct Outcome {
pub done: usize,
pub unfinished: usize,
pub report: PathBuf,
/// How many passes ran: 1 for a single run, more under `--repeat`.
pub passes: usize,
}
const STATE_FORMAT: &str = "inbuxa-migrate-state";
@@ -171,6 +184,86 @@ pub fn run(config: MigrateConfig) -> Result<Outcome, Error> {
}
}
let mut pass = 0usize;
loop {
pass += 1;
// A single run keeps its resume behavior (done steps at the front are
// skipped); every pass of a --repeat run re-runs the steps, so each
// one catches what has arrived at the source since the last.
let again = config.again || config.repeat.is_some();
let reports = run_accounts_pass(&config, &plan, &work, &accounts, &state, &save, again)?;
let unfinished = reports
.iter()
.filter(|(_, steps)| {
steps
.iter()
.any(|s| s.result != "done" && s.result != "done earlier")
})
.count();
let changed = reports
.iter()
.flat_map(|(_, steps)| steps)
.any(|s| s.counts.created + s.counts.updated + s.counts.deleted > 0);
let report = write_report(&work, &config, &setup_reports, &reports)?;
emit(
&config,
"end",
json!({"pass": pass, "done": reports.len() - unfinished,
"unfinished": unfinished, "report": report.display().to_string()}),
);
let Some(interval) = config.repeat else {
return done_outcome(reports.len() - unfinished, unfinished, report, pass);
};
if config.max_passes.is_some_and(|m| pass as u32 >= m) {
return done_outcome(reports.len() - unfinished, unfinished, report, pass);
}
if config.until_quiet && !changed {
if config.logger.enabled(crate::logging::LEVEL_DEFAULT) {
eprintln!("nothing new this pass: the source is quiet, stopping");
}
return done_outcome(reports.len() - unfinished, unfinished, report, pass);
}
emit(&config, "waiting", json!({"seconds": interval.as_secs()}));
if config.logger.enabled(crate::logging::LEVEL_DEFAULT) {
eprintln!(
"pass {pass} done; next in {}s (interrupt to stop)",
interval.as_secs()
);
}
std::thread::sleep(interval);
}
}
fn done_outcome(
done: usize,
unfinished: usize,
report: PathBuf,
passes: usize,
) -> Result<Outcome, Error> {
Ok(Outcome {
done,
unfinished,
report,
passes,
})
}
/// One pass over every account: each account's steps run on a worker, in
/// parallel up to `parallel`, their reports gathered and sorted into plan
/// order. A state-save failure stops the pass and surfaces as an error, so
/// nothing runs unrecorded.
#[allow(clippy::too_many_arguments)]
fn run_accounts_pass(
config: &MigrateConfig,
plan: &Plan,
work: &Path,
accounts: &[String],
state: &Mutex<State>,
save: &(dyn Fn(&State) -> Result<(), Error> + Sync),
again: bool,
) -> Result<Vec<(String, Vec<StepReport>)>, Error> {
let next = AtomicUsize::new(0);
let finished = AtomicUsize::new(0);
let reports: Mutex<Vec<(String, Vec<StepReport>)>> = Mutex::new(Vec::new());
@@ -182,11 +275,11 @@ pub fn run(config: MigrateConfig) -> Result<Outcome, Error> {
loop {
let i = next.fetch_add(1, Ordering::SeqCst);
let Some(email) = accounts.get(i) else { break };
match migrate_account(&config, &plan, &work, email, &state, &save) {
match migrate_account(config, plan, work, email, state, save, again) {
Ok(steps) => {
let n = finished.fetch_add(1, Ordering::SeqCst) + 1;
emit(
&config,
config,
"account",
json!({"email": email, "n": n, "total": total,
"result": result_of(&steps)}),
@@ -210,29 +303,9 @@ pub fn run(config: MigrateConfig) -> Result<Outcome, Error> {
if let Some(e) = lock(&failure).take() {
return Err(e);
}
let mut reports = reports.into_inner().unwrap_or_else(|e| e.into_inner());
reports.sort_by_key(|(email, _)| accounts.iter().position(|a| a == email));
let unfinished = reports
.iter()
.filter(|(_, steps)| {
steps
.iter()
.any(|s| s.result != "done" && s.result != "done earlier")
})
.count();
let report = write_report(&work, &config, &setup_reports, &reports)?;
emit(
&config,
"end",
json!({"done": reports.len() - unfinished, "unfinished": unfinished,
"report": report.display().to_string()}),
);
Ok(Outcome {
done: reports.len() - unfinished,
unfinished,
report,
})
Ok(reports)
}
fn lock<T>(m: &Mutex<T>) -> std::sync::MutexGuard<'_, T> {
@@ -312,12 +385,13 @@ fn migrate_account(
email: &str,
state: &Mutex<State>,
save: &(dyn Fn(&State) -> Result<(), Error> + Sync),
again: bool,
) -> Result<Vec<StepReport>, Error> {
let dir = work.join(email);
private_dir(&dir)?;
let archive = dir.join("archive.sqlite");
let mut reports = Vec::new();
let mut running = config.again;
let mut running = again;
let mut stopped = false;
for step in &plan.steps {
if stopped {
+61
View File
@@ -44,6 +44,9 @@ fn config(plan: &Path, work: &Path) -> MigrateConfig {
program: PathBuf::from(env!("CARGO_BIN_EXE_inbuxa-migrate")),
logger: Logger::from_flags(true, 0),
progress_json: false,
repeat: None,
max_passes: None,
until_quiet: false,
}
}
@@ -150,6 +153,64 @@ fn accounts_run_their_steps_resume_and_run_again() {
assert_eq!(emails(&work.join("[email protected]/archive.sqlite")), 2);
}
#[test]
fn repeat_runs_several_passes_and_stops_when_the_source_is_quiet() {
let dir = TempDir::new().unwrap();
let root = dir.path();
message(&root.join("mail/ann"), "m1", "First");
let plan = root.join("plan.json");
write_plan(
&plan,
json!([{"name": "mail", "args": ["import", "maildir", "mail/{local}", "{archive}"]}]),
);
let work = root.join("work");
// --repeat with a bound: every pass re-runs the import. The first moves
// the one message; later passes find nothing new, so with --until-quiet
// the run stops on the first quiet pass -- here the second -- without
// reaching the three-pass ceiling.
let mut cfg = config(&plan, &work);
cfg.only = vec!["[email protected]".into()];
cfg.repeat = Some(std::time::Duration::from_millis(5));
cfg.max_passes = Some(3);
cfg.until_quiet = true;
let out = migrate::run(cfg).unwrap();
assert_eq!(
out.passes, 2,
"pass 1 moves mail, pass 2 is quiet and stops"
);
assert_eq!((out.done, out.unfinished), (1, 0));
// The message landed once, not once per pass.
assert_eq!(emails(&work.join("[email protected]/archive.sqlite")), 1);
// The final report is the quiet pass: the re-run found nothing new.
assert_eq!(report(&out), ["[email protected],mail,done,0,0"]);
}
#[test]
fn repeat_with_a_ceiling_runs_exactly_that_many_passes() {
let dir = TempDir::new().unwrap();
let root = dir.path();
message(&root.join("mail/ann"), "m1", "First");
let plan = root.join("plan.json");
write_plan(
&plan,
json!([{"name": "mail", "args": ["import", "maildir", "mail/{local}", "{archive}"]}]),
);
let work = root.join("work");
// Without --until-quiet the run keeps going to the ceiling even though
// later passes are quiet, and the message is still imported only once.
let mut cfg = config(&plan, &work);
cfg.only = vec!["[email protected]".into()];
cfg.repeat = Some(std::time::Duration::from_millis(5));
cfg.max_passes = Some(3);
let out = migrate::run(cfg).unwrap();
assert_eq!(out.passes, 3);
assert_eq!(emails(&work.join("[email protected]/archive.sqlite")), 1);
}
#[test]
fn a_step_sees_only_the_secrets_it_names() {
let dir = TempDir::new().unwrap();