Add migrate --plan: many accounts moved by one plan file #38

Merged
jcoffey-dev merged 1 commits from feat/migrate-plan into main 2026-10-30 16:26:04 +00:00
9 changed files with 1327 additions and 4 deletions

No files matched your search

+8
View File
@@ -14,6 +14,14 @@ candidates before a series ships (`27.1-rc.1`). The tag is the version with a
is left as it is, so running it again creates nothing. Each user created
gets a generated password, written to a new file readable by its owner
only.
- `migrate --plan` moves many accounts with one command. The plan names
the steps, which are the commands in the docs with `{email}` and
`{archive}` filled in per account, and the accounts, listed or taken
from `directory.json`. Several accounts run at once, each with its own
archive and logs. A second run resumes where each stopped, `--again`
runs the final pass, and `report.csv` says what moved and why anything
didn't. Secrets are named by environment variable, never written in the
plan.
### Fixed
- `inspect` opens the archive read-only. It used to create an empty archive
+5 -2
View File
@@ -24,6 +24,7 @@ contacts, identities, Sieve scripts and files.
inbuxa-migrate export-carddav … alice.sqlite its contacts into any CardDAV server
inbuxa-migrate export-directory … out a server's accounts, no secrets
inbuxa-migrate import-directory … out those accounts, set up on inbuxa
inbuxa-migrate migrate --plan plan.json work many accounts, by a plan
Between the two sides sits an archive: one SQLite file holding one account.
`import` fills it from the old server; `export` writes it into the new one.
@@ -49,8 +50,10 @@ servers and mail, calendar and contacts apps open, with no network and no
store format in the way. For a whole server, `export-directory` writes the
list of domains, accounts, groups, lists and aliases to set up again
elsewhere, without passwords, and `import-directory` sets them up on an
inbuxa server, with a new password for each user. The full command
reference is in [docs/usage.md](docs/usage.md).
inbuxa server, with a new password for each user. `migrate` moves many
accounts with one plan file: the same steps for each account, several at
once, resuming where it stopped, with one report at the end. The full
command reference is in [docs/usage.md](docs/usage.md).
## Sources
+80 -1
View File
@@ -13,7 +13,7 @@ and how to install it.
## Commands and global flags
```
inbuxa-migrate <import|export|export-imap|export-managesieve|export-caldav|export-carddav|export-files|export-directory|import-directory|inspect> [args...]
inbuxa-migrate <import|export|export-imap|export-managesieve|export-caldav|export-carddav|export-files|export-directory|import-directory|migrate|inspect> [args...]
```
Every command takes a few global flags -- verbosity, worker pool size, retry
@@ -551,6 +551,85 @@ point at it, so a user finds its domain, its groups and its roles.
server and counts what it would create, sends nothing and writes no file;
`--passwords` isn't needed then.
## Migrate many accounts
```
inbuxa-migrate migrate --plan <FILE> [--parallel <N>] [--again]
[--account <EMAIL>]... [--dry-run] [-q] <DIR>
```
Moves a list of accounts with one command. The plan file names the steps
and the accounts. Each step is one of the commands on this page, written
the same way, and `migrate` runs it for every account, several accounts at
once. `<DIR>` is the work directory. It holds an archive and a log per
account, the state a later run resumes from, and the report.
```json
{
"format": "inbuxa-migrate-plan",
"formatVersion": 1,
"setup": [
{"name": "directory",
"args": ["import-directory", "--url", "https://mail.example.org",
"--auth-bearer", "--passwords", "passwords.csv", "directory"],
"secrets": {"token": "INBUXA_ADMIN_TOKEN"}}
],
"accounts": {"directory": "directory/directory.json"},
"steps": [
{"name": "mail",
"args": ["import", "imap", "--url", "imaps://old.example.org",
"--auth-basic", "{email}*master", "{archive}"],
"secrets": {"password": "OLD_MASTER_PASSWORD"}},
{"name": "inbuxa",
"args": ["export", "--url", "https://mail.example.org", "--auth-bearer",
"--account-name", "{email}", "{archive}"],
"secrets": {"token": "INBUXA_ADMIN_TOKEN"}}
]
}
```
- **Steps.** `setup` steps run once, before any account:
`import-directory` or `export-directory`. The `steps` run for each
account, in order, and each has to name `{archive}`. `{email}`,
`{local}` and `{domain}` are the account's address and its two halves.
Steps run in the plan's directory, so a relative path means the same as
it does in the plan.
- **Accounts.** A list of addresses, or `{"directory": "…/directory.json"}`
for the users that `export-directory` wrote.
- **Secrets stay out of the plan.** A step names the environment variable
that holds each secret it needs: `password`, `token`, `clientSecret`
(EWS) or `graphToken`. A step sees only the secrets it names. A plan that
writes a password or token into `args` is refused. A step that needs a
secret it doesn't name fails at once instead of waiting for a prompt.
- **Signing in for each account.** Each step signs in the way its command
does, so the plan decides how. To avoid collecting every user's password,
use a login that can act for each account. The example uses Dovecot's
master user, `{email}*master`. Use the source's own form where it has
one: Exchange impersonation through `--mailbox {email}`, or an
administrator on a JMAP source with `--account-name {email}`.
- **One account at a time, several at once.** `--parallel` (default 4)
sets how many accounts run at the same time. Each step runs as its own
process. Its output goes to `<DIR>/<email>/<step>.log`, and one account
failing doesn't stop the others. A step that fails stops its account.
A step that leaves some objects behind (exit code 5) doesn't. A failed
setup step stops the run.
- **Resuming.** A second run with the same plan and work directory skips
every account that finished. For the rest, it skips the steps that were
done and runs everything from the first one that wasn't. `--again` runs
every account step once more, done or not. Use it for the final pass
once the old server stops taking mail: the steps converge, so it only
moves what changed. `--account` limits a run to the accounts named.
- **The report.** `<DIR>/report.csv` has one row per step per account:
how it ended (`done`, `partial`, `failed`, `not run`, or
`done earlier`), the counts summed over the step's types, and, for one
that didn't finish, its last error. Exit code 5 means some account
didn't finish.
The work directory is made readable by its owner only, and so are the
logs. With `--dry-run`, each step runs with `--dry-run`, nothing is
recorded, and the report goes to `dry-run-report.csv`. An export step
reads the archive as it is, so on a first dry run it has nothing to show.
## Inspect
```
+64
View File
@@ -21,6 +21,7 @@ use crate::inspect::InspectConfig;
use crate::jmap::account::AccountSelector;
use crate::jmap::http::Auth;
use crate::logging::Logger;
use crate::migrate::MigrateConfig;
use crate::secret;
use crate::sync::export_dav::ExportDavConfig;
use crate::sync::export_imap::ExportImapConfig;
@@ -90,6 +91,10 @@ enum Command {
name = "import-directory"
)]
ImportDirectory(Box<ImportDirectoryArgs>),
#[command(
about = "Move many accounts by a plan file: the same steps for each, several at once, resumable"
)]
Migrate(Box<MigrateArgs>),
#[command(about = "Show the contents of a local archive (read-only)")]
Inspect(Box<InspectArgs>),
}
@@ -684,6 +689,50 @@ struct ImportDirectoryArgs {
dir: PathBuf,
}
#[derive(Args)]
struct MigrateArgs {
#[arg(
long,
value_name = "FILE",
help = "Plan file (JSON): the setup steps, the steps for each account, and the accounts"
)]
plan: PathBuf,
#[arg(
long,
value_name = "N",
default_value_t = 4,
help = "Accounts migrated at once"
)]
parallel: usize,
#[arg(
long,
help = "Run every account step again, done or not: the final pass once the old server stops taking mail"
)]
again: bool,
#[arg(
long = "account",
value_name = "EMAIL",
action = ArgAction::Append,
help = "Only this account (repeatable; default: every account in the plan)"
)]
accounts: Vec<String>,
#[arg(long, help = "Run each step with --dry-run; record nothing")]
dry_run: bool,
#[arg(short = 'q', long, help = "Quiet: no line per account")]
quiet: bool,
#[arg(
value_name = "DIR",
help = "Work directory: an archive and logs per account, the state and the report"
)]
work: PathBuf,
}
#[derive(Args)]
struct InspectArgs {
#[arg(value_name = "ARCHIVE", help = "Local SQLite archive")]
@@ -724,6 +773,7 @@ pub enum Action {
ExportFiles(ExportFilesConfig),
ExportDirectory(ExportDirectoryConfig),
ImportDirectory(ImportDirectoryConfig),
Migrate(MigrateConfig),
ExportImap(CommonConfig, ExportImapConfig),
ExportManageSieve(CommonConfig, ExportManageSieveConfig),
ExportDav(CommonConfig, ExportDavConfig),
@@ -855,6 +905,7 @@ impl Cli {
Command::ExportFiles(args) => resolve_export_files(*args),
Command::ExportDirectory(args) => resolve_export_directory(*args),
Command::ImportDirectory(args) => resolve_import_directory(*args),
Command::Migrate(args) => resolve_migrate(*args),
Command::ExportImap(args) => resolve_export_imap(*args),
Command::ExportManageSieve(args) => resolve_export_managesieve(*args),
Command::ExportCaldav(args) => resolve_export_dav(*args, DavKindArg::Caldav),
@@ -1339,6 +1390,19 @@ fn resolve_import_directory(args: ImportDirectoryArgs) -> Result<Action, Error>
}))
}
fn resolve_migrate(args: MigrateArgs) -> Result<Action, Error> {
Ok(Action::Migrate(MigrateConfig {
plan: args.plan,
work: args.work,
parallel: args.parallel,
again: args.again,
only: args.accounts,
dry_run: args.dry_run,
program: std::env::current_exe()?,
logger: Logger::from_flags(args.quiet, 0),
}))
}
fn resolve_inspect(args: InspectArgs) -> Result<Action, Error> {
let target = match args.target.as_deref() {
Some(token) => Some(ObjectType::parse(token)?),
+1
View File
@@ -20,6 +20,7 @@ pub mod inspect;
pub mod jmap;
pub mod logging;
pub mod managesieve;
pub mod migrate;
pub mod net;
pub mod secret;
pub mod sync;
+23 -1
View File
@@ -10,7 +10,7 @@ use clap::Parser;
use inbuxa_migrate::cli::{Action, Cli};
use inbuxa_migrate::error::Error;
use inbuxa_migrate::sync::{self, RunOutcome, Summary};
use inbuxa_migrate::{export_directory, export_files, import_directory, inspect};
use inbuxa_migrate::{export_directory, export_files, import_directory, inspect, migrate};
fn main() {
let code = run();
@@ -125,6 +125,28 @@ fn run() -> i32 {
logger,
)
}
Action::Migrate(config) => {
return match migrate::run(config) {
Ok(outcome) => {
println!(
"accounts: {} done, {} not finished; report: {}",
outcome.done,
outcome.unfinished,
outcome.report.display()
);
if outcome.unfinished > 0 {
eprintln!(
"error: some accounts did not finish; the report says why. Fix that \
and run the same command again: it resumes where each stopped"
);
5
} else {
0
}
}
Err(err) => fail(&err),
};
}
Action::Inspect(config) => {
return match inspect::run(config) {
Ok(()) => 0,
+536
View File
@@ -0,0 +1,536 @@
/*
* SPDX-FileCopyrightText: 2026 Coffey Labs LLC <[email protected]>
*
* SPDX-License-Identifier: Apache-2.0 OR MIT
*/
//! `migrate`: many accounts moved by one plan. Each step is a run of this
//! same binary, so it behaves exactly as the command the docs describe, its
//! output goes to that account's log, and one account failing leaves the
//! rest running. The work directory holds an archive and logs per account,
//! the state a second run resumes from, and the report.
pub mod plan;
use std::collections::BTreeMap;
use std::fs::{self, File, OpenOptions};
use std::io::{Read, Seek, SeekFrom, Write};
use std::path::{Path, PathBuf};
use std::process::{Command, Stdio};
use std::sync::Mutex;
use std::sync::atomic::{AtomicUsize, Ordering};
use serde::{Deserialize, Serialize};
use crate::error::Error;
use crate::logging::Logger;
use plan::{Plan, SECRET_VARS, Step};
pub struct MigrateConfig {
pub plan: PathBuf,
pub work: PathBuf,
/// Accounts migrated at once.
pub parallel: usize,
/// Run every account step again, done or not.
pub again: bool,
/// Only these accounts; empty for all of them.
pub only: Vec<String>,
pub dry_run: bool,
/// The binary each step runs: this one.
pub program: PathBuf,
pub logger: Logger,
}
pub struct Outcome {
pub done: usize,
pub unfinished: usize,
pub report: PathBuf,
}
const STATE_FORMAT: &str = "inbuxa-migrate-state";
#[derive(Serialize, Deserialize, Default)]
#[serde(rename_all = "camelCase")]
struct State {
format: String,
format_version: u32,
setup: BTreeMap<String, Ended>,
accounts: BTreeMap<String, BTreeMap<String, Ended>>,
}
/// How a step ended. Only `Done` is skipped by the next run.
#[derive(Serialize, Deserialize, Clone, Copy, PartialEq, Eq, Debug)]
#[serde(rename_all = "camelCase")]
enum Ended {
Done,
Partial,
Failed,
}
/// What one step did this run, for the report.
struct StepReport {
step: String,
result: &'static str,
counts: Counts,
detail: String,
}
#[derive(Default, Clone, Copy)]
struct Counts {
fetched: u64,
created: u64,
updated: u64,
deleted: u64,
skipped: u64,
failed: u64,
}
struct Ran {
result: Ended,
counts: Counts,
detail: String,
}
pub fn run(config: MigrateConfig) -> Result<Outcome, Error> {
let plan = plan::load(&config.plan)?;
let accounts = chosen(&plan, &config.only)?;
if config.parallel == 0 {
return Err(Error::Usage("--parallel must be at least 1".to_owned()));
}
let work = std::path::absolute(&config.work)?;
private_dir(&work)?;
let state_path = work.join("state.json");
let state = Mutex::new(load_state(&state_path)?);
let save = |state: &State| -> Result<(), Error> {
if config.dry_run {
return Ok(());
}
let tmp = work.join("state.json.tmp");
fs::write(&tmp, serde_json::to_vec_pretty(state).map_err(io_err)?)?;
fs::rename(&tmp, &state_path)?;
Ok(())
};
// Setup runs once, before any account, and a failure stops the run:
// the accounts need what it makes.
let mut setup_reports = Vec::new();
if !plan.setup.is_empty() {
private_dir(&work.join("setup"))?;
}
for step in &plan.setup {
if lock(&state).setup.get(&step.name) == Some(&Ended::Done) {
setup_reports.push(earlier(&step.name));
continue;
}
let log = work.join("setup").join(format!("{}.log", step.name));
let ran = run_step(&config, &plan, step, None, &log);
setup_reports.push(report_of(&step.name, &ran));
if !config.dry_run {
lock(&state).setup.insert(step.name.clone(), ran.result);
}
save(&lock(&state))?;
if ran.result == Ended::Failed {
write_report(&work, &config, &setup_reports, &[])?;
return Err(Error::Partial(format!(
"setup step {} failed: {}; see {}",
step.name,
ran.detail,
log.display()
)));
}
}
let next = AtomicUsize::new(0);
let finished = AtomicUsize::new(0);
let reports: Mutex<Vec<(String, Vec<StepReport>)>> = Mutex::new(Vec::new());
let failure: Mutex<Option<Error>> = Mutex::new(None);
let total = accounts.len();
std::thread::scope(|scope| {
for _ in 0..config.parallel.min(total) {
scope.spawn(|| {
loop {
let i = next.fetch_add(1, Ordering::SeqCst);
let Some(email) = accounts.get(i) else { break };
match migrate_account(&config, &plan, &work, email, &state, &save) {
Ok(steps) => {
let n = finished.fetch_add(1, Ordering::SeqCst) + 1;
if config.logger.enabled(crate::logging::LEVEL_DEFAULT) {
eprintln!("[{n}/{total}] {email}: {}", line_of(&steps));
}
lock(&reports).push((email.clone(), steps));
}
Err(e) => {
// The state can't be saved: stop handing out
// accounts, so nothing runs unrecorded.
next.store(total, Ordering::SeqCst);
lock(&failure).get_or_insert(e);
}
}
}
});
}
});
if let Some(e) = lock(&failure).take() {
return Err(e);
}
let mut reports = reports.into_inner().unwrap_or_else(|e| e.into_inner());
reports.sort_by_key(|(email, _)| accounts.iter().position(|a| a == email));
let unfinished = reports
.iter()
.filter(|(_, steps)| {
steps
.iter()
.any(|s| s.result != "done" && s.result != "done earlier")
})
.count();
let report = write_report(&work, &config, &setup_reports, &reports)?;
Ok(Outcome {
done: reports.len() - unfinished,
unfinished,
report,
})
}
fn lock<T>(m: &Mutex<T>) -> std::sync::MutexGuard<'_, T> {
m.lock().unwrap_or_else(|e| e.into_inner())
}
fn io_err(e: serde_json::Error) -> Error {
Error::Io(std::io::Error::other(e))
}
fn chosen(plan: &Plan, only: &[String]) -> Result<Vec<String>, Error> {
if only.is_empty() {
return Ok(plan.accounts.clone());
}
let mut out = Vec::new();
for email in only {
let email = email.trim().to_ascii_lowercase();
if !plan.accounts.contains(&email) {
return Err(Error::Usage(format!(
"--account {email} is not in the plan"
)));
}
if !out.contains(&email) {
out.push(email);
}
}
Ok(out)
}
/// The work directory holds whole mailboxes: readable by its owner only.
fn private_dir(path: &Path) -> Result<(), Error> {
if path.is_dir() {
return Ok(());
}
let mut builder = fs::DirBuilder::new();
builder.recursive(true);
#[cfg(unix)]
{
use std::os::unix::fs::DirBuilderExt;
builder.mode(0o700);
}
builder.create(path)?;
Ok(())
}
fn load_state(path: &Path) -> Result<State, Error> {
let text = match fs::read_to_string(path) {
Ok(text) => text,
Err(e) if e.kind() == std::io::ErrorKind::NotFound => {
return Ok(State {
format: STATE_FORMAT.to_owned(),
format_version: 1,
..State::default()
});
}
Err(e) => return Err(e.into()),
};
let state: State = serde_json::from_str(&text)
.map_err(|e| Error::Usage(format!("{}: {e}", path.display())))?;
if state.format != STATE_FORMAT || state.format_version != 1 {
return Err(Error::Usage(format!(
"{} is not a migrate state file this release reads",
path.display()
)));
}
Ok(state)
}
/// One account's steps. Done steps at the front are skipped; from the first
/// that isn't done, every step runs, so an import that picked up more is
/// always followed by the export that delivers it. A step that fails stops
/// the account; a partial one doesn't.
fn migrate_account(
config: &MigrateConfig,
plan: &Plan,
work: &Path,
email: &str,
state: &Mutex<State>,
save: &(dyn Fn(&State) -> Result<(), Error> + Sync),
) -> Result<Vec<StepReport>, Error> {
let dir = work.join(email);
private_dir(&dir)?;
let archive = dir.join("archive.sqlite");
let mut reports = Vec::new();
let mut running = config.again;
let mut stopped = false;
for step in &plan.steps {
if stopped {
reports.push(StepReport {
step: step.name.clone(),
result: "not run",
counts: Counts::default(),
detail: "an earlier step failed".to_owned(),
});
continue;
}
let was = lock(state)
.accounts
.get(email)
.and_then(|steps| steps.get(&step.name))
.copied();
if !running && was == Some(Ended::Done) {
reports.push(earlier(&step.name));
continue;
}
running = true;
let log = dir.join(format!("{}.log", step.name));
let ran = run_step(config, plan, step, Some((email, &archive)), &log);
reports.push(report_of(&step.name, &ran));
if !config.dry_run {
let mut s = lock(state);
s.accounts
.entry(email.to_owned())
.or_default()
.insert(step.name.clone(), ran.result);
save(&s)?;
}
stopped = ran.result == Ended::Failed;
}
Ok(reports)
}
fn earlier(step: &str) -> StepReport {
StepReport {
step: step.to_owned(),
result: "done earlier",
counts: Counts::default(),
detail: String::new(),
}
}
fn report_of(step: &str, ran: &Ran) -> StepReport {
StepReport {
step: step.to_owned(),
result: match ran.result {
Ended::Done => "done",
Ended::Partial => "partial",
Ended::Failed => "failed",
},
counts: ran.counts,
detail: ran.detail.clone(),
}
}
fn line_of(steps: &[StepReport]) -> String {
match steps
.iter()
.find(|s| s.result == "failed" || s.result == "partial")
{
Some(s) if s.detail.is_empty() => format!("{} at {}", s.result, s.step),
Some(s) => format!("{} at {}: {}", s.result, s.step, s.detail),
None => "done".to_owned(),
}
}
/// Runs one step as a child: no terminal, so a missing secret fails
/// rather than waits; only the secrets the step names; stderr to the log,
/// appended so a resumed run keeps the history.
fn run_step(
config: &MigrateConfig,
plan: &Plan,
step: &Step,
account: Option<(&str, &Path)>,
log: &Path,
) -> Ran {
let failed = |detail: String| Ran {
result: Ended::Failed,
counts: Counts::default(),
detail,
};
let mut args = step.args_for(account);
if config.dry_run {
args.push("--dry-run".to_owned());
}
let (mut file, start) = match open_log(log) {
Ok(opened) => opened,
Err(e) => return failed(format!("can't write {}: {e}", log.display())),
};
// The args hold no secrets; the plan refuses them.
let _ = writeln!(file, "== inbuxa-migrate {}", args.join(" "));
let stderr = match file.try_clone() {
Ok(f) => f,
Err(e) => return failed(format!("can't write {}: {e}", log.display())),
};
let mut command = Command::new(&config.program);
command
.args(&args)
.current_dir(&plan.base)
.stdin(Stdio::null())
.stdout(Stdio::piped())
.stderr(Stdio::from(stderr));
for var in SECRET_VARS {
command.env_remove(var);
}
for (var, value) in &step.env {
command.env(var, value);
}
let output = match command.output() {
Ok(output) => output,
Err(e) => return failed(format!("can't run {}: {e}", config.program.display())),
};
let stdout = String::from_utf8_lossy(&output.stdout);
let _ = file.write_all(stdout.as_bytes());
let counts = counts_of(&stdout);
let result = match output.status.code() {
Some(0) => Ended::Done,
Some(5) => Ended::Partial,
_ => Ended::Failed,
};
let detail = match result {
Ended::Done => String::new(),
_ => last_error(log, start).unwrap_or_else(|| match output.status.code() {
Some(code) => format!("exit code {code}"),
None => "stopped by a signal".to_owned(),
}),
};
Ran {
result,
counts,
detail,
}
}
fn open_log(path: &Path) -> std::io::Result<(File, u64)> {
let mut options = OpenOptions::new();
options.create(true).append(true);
#[cfg(unix)]
{
use std::os::unix::fs::OpenOptionsExt;
options.mode(0o600);
}
let file = options.open(path)?;
let start = file.metadata()?.len();
Ok((file, start))
}
/// The last `error:` line this run wrote to the log.
fn last_error(log: &Path, start: u64) -> Option<String> {
let mut file = File::open(log).ok()?;
file.seek(SeekFrom::Start(start)).ok()?;
let mut text = String::new();
file.read_to_string(&mut text).ok()?;
text.lines()
.rev()
.find_map(|l| l.strip_prefix("error: "))
.map(str::to_owned)
}
/// The per-type lines a step prints, summed:
/// `email: created=1 fetched=2 updated=0 deleted=0 skipped=0 failed=0`.
fn counts_of(stdout: &str) -> Counts {
let mut c = Counts::default();
for line in stdout.lines() {
let Some((_, rest)) = line.split_once(": ") else {
continue;
};
for field in rest.split_whitespace() {
let Some((key, n)) = field.split_once('=') else {
continue;
};
let Ok(n) = n.parse::<u64>() else { continue };
match key {
"fetched" => c.fetched += n,
"created" => c.created += n,
"updated" => c.updated += n,
"deleted" => c.deleted += n,
"skipped" => c.skipped += n,
"failed" => c.failed += n,
_ => {}
}
}
}
c
}
/// `report.csv` in the work directory (`dry-run-report.csv` for a dry
/// run): one row per step per account, setup first.
fn write_report(
work: &Path,
config: &MigrateConfig,
setup: &[StepReport],
accounts: &[(String, Vec<StepReport>)],
) -> Result<PathBuf, Error> {
let path = work.join(if config.dry_run {
"dry-run-report.csv"
} else {
"report.csv"
});
let mut out =
String::from("account,step,result,fetched,created,updated,deleted,skipped,failed,detail\n");
let rows = setup.iter().map(|s| ("(setup)", s)).chain(
accounts
.iter()
.flat_map(|(email, steps)| steps.iter().map(move |s| (email.as_str(), s))),
);
for (account, s) in rows {
let c = s.counts;
out.push_str(&format!(
"{account},{},{},{},{},{},{},{},{},{}\n",
s.step,
s.result,
c.fetched,
c.created,
c.updated,
c.deleted,
c.skipped,
c.failed,
csv_field(&s.detail)
));
}
fs::write(&path, out)?;
Ok(path)
}
fn csv_field(s: &str) -> String {
if s.contains([',', '"', '\n', '\r']) {
format!("\"{}\"", s.replace('"', "\"\""))
} else {
s.to_owned()
}
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn counts_are_summed_over_the_types() {
let c = counts_of(
"email: created=3 fetched=4 updated=1 deleted=0 skipped=2 failed=1\n\
mailbox: created=2 fetched=2 updated=0 deleted=0 skipped=0 failed=0\n",
);
assert_eq!((c.fetched, c.created, c.updated), (6, 5, 1));
assert_eq!((c.skipped, c.failed), (2, 1));
}
#[test]
fn details_are_quoted_when_they_need_it() {
assert_eq!(csv_field("plain"), "plain");
assert_eq!(csv_field("a, \"b\""), "\"a, \"\"b\"\"\"");
}
}
+430
View File
@@ -0,0 +1,430 @@
/*
* SPDX-FileCopyrightText: 2026 Coffey Labs LLC <[email protected]>
*
* SPDX-License-Identifier: Apache-2.0 OR MIT
*/
//! The plan file: the steps to run once, the steps to run for each account,
//! and the accounts. A step is one inbuxa-migrate command line, written as
//! the docs write it, with `{email}`, `{local}`, `{domain}` and `{archive}`
//! filled in per account. Secrets are never in the plan: a step names the
//! environment variable that holds each one.
use std::collections::HashSet;
use std::path::{Path, PathBuf};
use serde::Deserialize;
use serde_json::Value;
use crate::error::Error;
pub const FORMAT: &str = "inbuxa-migrate-plan";
pub const FORMAT_VERSION: u32 = 1;
/// The variables a child reads its secrets from. All are cleared before a
/// step runs, so a step only ever sees the secrets it names.
pub const SECRET_VARS: &[&str] = &[
"INBUXA_MIGRATE_PASSWORD",
"INBUXA_MIGRATE_API_KEY",
"INBUXA_MIGRATE_TOKEN",
"INBUXA_MIGRATE_EWS_CLIENT_SECRET",
"INBUXA_MIGRATE_GRAPH_TOKEN",
];
const ACCOUNT_COMMANDS: &[&str] = &[
"import",
"export",
"export-imap",
"export-managesieve",
"export-caldav",
"export-carddav",
"export-files",
];
const SETUP_COMMANDS: &[&str] = &["import-directory", "export-directory"];
const PLACEHOLDERS: &[&str] = &["{email}", "{local}", "{domain}", "{archive}"];
#[derive(Deserialize)]
#[serde(rename_all = "camelCase", deny_unknown_fields)]
struct RawPlan {
format: String,
format_version: u32,
#[serde(default)]
setup: Vec<RawStep>,
accounts: RawAccounts,
steps: Vec<RawStep>,
}
#[derive(Deserialize)]
#[serde(untagged)]
enum RawAccounts {
List(Vec<String>),
Directory(FromDirectory),
}
#[derive(Deserialize)]
#[serde(deny_unknown_fields)]
struct FromDirectory {
directory: PathBuf,
}
#[derive(Deserialize)]
#[serde(deny_unknown_fields)]
struct RawStep {
name: String,
args: Vec<String>,
#[serde(default)]
secrets: Secrets,
}
/// Each secret by the variable that holds it.
#[derive(Deserialize, Default)]
#[serde(rename_all = "camelCase", deny_unknown_fields)]
struct Secrets {
password: Option<String>,
token: Option<String>,
client_secret: Option<String>,
graph_token: Option<String>,
}
pub struct Plan {
/// The plan's own directory: steps run there, so relative paths in a
/// step mean what they say in the plan.
pub base: PathBuf,
pub setup: Vec<Step>,
pub accounts: Vec<String>,
pub steps: Vec<Step>,
}
/// No `Debug`: `env` holds secrets.
pub struct Step {
pub name: String,
pub args: Vec<String>,
/// The child's secret variables, with their values.
pub env: Vec<(&'static str, String)>,
}
impl Step {
/// The command line for one account, or for setup.
pub fn args_for(&self, account: Option<(&str, &Path)>) -> Vec<String> {
let Some((email, archive)) = account else {
return self.args.clone();
};
let (local, domain) = email.rsplit_once('@').unwrap_or((email, ""));
let archive = archive.to_string_lossy();
self.args
.iter()
.map(|a| {
a.replace("{email}", email)
.replace("{local}", local)
.replace("{domain}", domain)
.replace("{archive}", &archive)
})
.collect()
}
}
pub fn load(path: &Path) -> Result<Plan, Error> {
let text = std::fs::read_to_string(path)
.map_err(|e| Error::Usage(format!("can't read plan {}: {e}", path.display())))?;
let raw: RawPlan = serde_json::from_str(&text)
.map_err(|e| Error::Usage(format!("plan {}: {e}", path.display())))?;
if raw.format != FORMAT {
return Err(Error::Usage(format!(
"plan {}: format is \"{}\", not \"{FORMAT}\"",
path.display(),
raw.format
)));
}
if raw.format_version != FORMAT_VERSION {
return Err(Error::Usage(format!(
"plan {}: formatVersion {} is not one this release reads ({FORMAT_VERSION})",
path.display(),
raw.format_version
)));
}
let base = std::path::absolute(path)?
.parent()
.map(Path::to_path_buf)
.unwrap_or_default();
if raw.steps.is_empty() {
return Err(Error::Usage("the plan has no steps".to_owned()));
}
let mut names = HashSet::new();
let mut setup = Vec::new();
for step in raw.setup {
setup.push(step_of(step, false, &mut names)?);
}
let mut steps = Vec::new();
for step in raw.steps {
steps.push(step_of(step, true, &mut names)?);
}
let listed = match raw.accounts {
RawAccounts::List(list) => list,
RawAccounts::Directory(from) => directory_users(&base.join(from.directory))?,
};
let mut accounts = Vec::new();
let mut seen = HashSet::new();
for email in listed {
let email = account_email(&email)?;
if !seen.insert(email.clone()) {
return Err(Error::Usage(format!("account {email} is listed twice")));
}
accounts.push(email);
}
if accounts.is_empty() {
return Err(Error::Usage("the plan lists no accounts".to_owned()));
}
Ok(Plan {
base,
setup,
accounts,
steps,
})
}
fn step_of(raw: RawStep, per_account: bool, names: &mut HashSet<String>) -> Result<Step, Error> {
let name = raw.name;
if name.is_empty()
|| !name
.chars()
.all(|c| c.is_ascii_alphanumeric() || c == '-' || c == '_')
{
return Err(Error::Usage(format!(
"step name \"{name}\": use letters, digits, - and _"
)));
}
if !names.insert(name.clone()) {
return Err(Error::Usage(format!("step name \"{name}\" is used twice")));
}
check_args(&name, &raw.args, per_account)?;
let mut env = Vec::new();
let secrets = raw.secrets;
for (child_var, from) in [
("INBUXA_MIGRATE_PASSWORD", secrets.password),
("INBUXA_MIGRATE_API_KEY", secrets.token),
("INBUXA_MIGRATE_EWS_CLIENT_SECRET", secrets.client_secret),
("INBUXA_MIGRATE_GRAPH_TOKEN", secrets.graph_token),
] {
let Some(from) = from else { continue };
match std::env::var(&from) {
Ok(value) if !value.is_empty() => env.push((child_var, value)),
_ => {
return Err(Error::Usage(format!(
"step {name}: ${from} is not set; it holds a secret the step names"
)));
}
}
}
Ok(Step {
name,
args: raw.args,
env,
})
}
fn check_args(name: &str, args: &[String], per_account: bool) -> Result<(), Error> {
let usage = |m: String| Err(Error::Usage(format!("step {name}: {m}")));
let (allowed, kind) = if per_account {
(ACCOUNT_COMMANDS, "an account step")
} else {
(SETUP_COMMANDS, "a setup step")
};
match args.first() {
Some(command) if allowed.contains(&command.as_str()) => {}
Some(command) => {
return usage(format!(
"{kind} runs one of {}, not \"{command}\"",
allowed.join(", ")
));
}
None => return usage("args is empty".to_owned()),
}
for (i, arg) in args.iter().enumerate() {
let flag = arg.split_once('=').map_or(arg.as_str(), |(f, _)| f);
match flag {
"--auth-password" | "--ews-client-secret" => {
return usage(format!(
"{flag} would put a secret in the plan; name its variable in \"secrets\""
));
}
"--auth-bearer" | "--access-token" => {
let next = args.get(i + 1);
if arg.contains('=') || next.is_some_and(|n| !n.starts_with('-')) {
return usage(format!(
"{flag} takes its token from \"secrets\"; write it with no value, \
before another flag"
));
}
}
"--dry-run" => {
return usage("leave out --dry-run; migrate --dry-run adds it".to_owned());
}
_ => {}
}
}
let placeholders = args
.iter()
.any(|a| PLACEHOLDERS.iter().any(|p| a.contains(p)));
if per_account && !args.iter().any(|a| a.contains("{archive}")) {
return usage("an account step has to name {archive}".to_owned());
}
if !per_account && placeholders {
return usage(
"a setup step runs once, so it can't use {email}, {archive} and the rest".to_owned(),
);
}
Ok(())
}
/// The users in a `directory.json` that `export-directory` wrote.
fn directory_users(path: &Path) -> Result<Vec<String>, Error> {
let text = std::fs::read_to_string(path)
.map_err(|e| Error::Usage(format!("can't read {}: {e}", path.display())))?;
let doc: Value = serde_json::from_str(&text)
.map_err(|e| Error::Usage(format!("{}: {e}", path.display())))?;
if doc.get("format").and_then(Value::as_str) != Some("inbuxa-migrate-directory") {
return Err(Error::Usage(format!(
"{} is not a directory.json from export-directory",
path.display()
)));
}
Ok(doc
.get("users")
.and_then(Value::as_array)
.map(|users| {
users
.iter()
.filter_map(|u| u.get("email").and_then(Value::as_str))
.map(str::to_owned)
.collect()
})
.unwrap_or_default())
}
/// An address that is safe as a directory name, lowercased.
fn account_email(email: &str) -> Result<String, Error> {
let email = email.trim().to_ascii_lowercase();
let ok = match email.split_once('@') {
Some((local, domain)) => {
!local.is_empty()
&& !domain.is_empty()
&& !domain.contains('@')
&& !email.starts_with('.')
&& !email
.chars()
.any(|c| c.is_whitespace() || c.is_control() || matches!(c, '/' | '\\'))
}
None => false,
};
if !ok {
return Err(Error::Usage(format!(
"account \"{email}\" is not an address the plan can use"
)));
}
Ok(email)
}
#[cfg(test)]
mod tests {
use super::*;
fn args(list: &[&str]) -> Vec<String> {
list.iter().map(|s| (*s).to_owned()).collect()
}
fn err(list: &[&str], per_account: bool) -> String {
check_args("s", &args(list), per_account)
.unwrap_err()
.to_string()
}
#[test]
fn secrets_and_dry_run_stay_out_of_the_args() {
let e = err(
&["import", "imap", "--auth-password", "x", "{archive}"],
true,
);
assert!(e.contains("secrets"), "{e}");
let e = err(&["import", "imap", "--auth-password=x", "{archive}"], true);
assert!(e.contains("secrets"), "{e}");
// A bare --auth-bearer before the archive would take it as the token.
let e = err(&["export", "--auth-bearer", "{archive}"], true);
assert!(e.contains("before another flag"), "{e}");
let e = err(&["export", "--dry-run", "{archive}"], true);
assert!(e.contains("--dry-run"), "{e}");
check_args(
"s",
&args(&[
"export",
"--auth-bearer",
"--account-name",
"{email}",
"{archive}",
]),
true,
)
.unwrap();
}
#[test]
fn steps_run_the_commands_they_are_for() {
let e = err(&["import-directory", "dir"], true);
assert!(e.contains("an account step runs one of"), "{e}");
let e = err(&["import", "imap", "--url", "x"], true);
assert!(e.contains("{archive}"), "{e}");
let e = err(&["import-directory", "--url", "{domain}", "dir"], false);
assert!(e.contains("runs once"), "{e}");
check_args("s", &args(&["import-directory", "dir"]), false).unwrap();
}
#[test]
fn placeholders_are_filled_per_account() {
let step = Step {
name: "mail".into(),
args: args(&[
"import",
"imap",
"--auth-basic",
"{local}*master@{domain}",
"{archive}",
"^A\\d{4}$",
]),
env: Vec::new(),
};
assert_eq!(
step.args_for(Some((
"[email protected]",
Path::new("/w/ann/archive.sqlite")
))),
args(&[
"import",
"imap",
"--auth-basic",
"ann*[email protected]",
"/w/ann/archive.sqlite",
"^A\\d{4}$"
])
);
}
#[test]
fn accounts_are_plain_addresses() {
assert_eq!(
account_email(" [email protected] ").unwrap(),
"[email protected]"
);
for bad in [
"ann",
"@example.org",
"ann@",
"a/[email protected]",
"../x@y",
"a b@c",
] {
assert!(account_email(bad).is_err(), "{bad}");
}
}
}
+180
View File
@@ -0,0 +1,180 @@
/*
* SPDX-FileCopyrightText: 2026 Coffey Labs LLC <[email protected]>
*
* SPDX-License-Identifier: Apache-2.0 OR MIT
*/
//! `migrate` runs a plan's steps for each account as runs of the real
//! binary: Maildir in, files out, offline. One account fails and the rest
//! go on; a second run resumes only what didn't finish; `--again` runs it
//! all once more. Counts in the report are summed over the types: a
//! mailbox and an email are two.
use std::fs;
use std::path::{Path, PathBuf};
use inbuxa_migrate::logging::Logger;
use inbuxa_migrate::migrate::{self, MigrateConfig, Outcome};
use rusqlite::Connection;
use serde_json::json;
use tempfile::TempDir;
fn message(dir: &Path, name: &str, subject: &str) {
for sub in ["cur", "new", "tmp"] {
fs::create_dir_all(dir.join(sub)).unwrap();
}
fs::write(
dir.join("cur").join(format!("{name}:2,S")),
format!(
"From: [email protected]\r\nTo: [email protected]\r\nSubject: {subject}\r\n\
Message-ID: <{name}@example.org>\r\nDate: Mon, 4 Mar 2024 09:00:00 +0000\r\n\r\nHi\r\n"
),
)
.unwrap();
}
fn config(plan: &Path, work: &Path) -> MigrateConfig {
MigrateConfig {
plan: plan.to_path_buf(),
work: work.to_path_buf(),
parallel: 2,
again: false,
only: Vec::new(),
dry_run: false,
program: PathBuf::from(env!("CARGO_BIN_EXE_inbuxa-migrate")),
logger: Logger::from_flags(true, 0),
}
}
fn emails(archive: &Path) -> i64 {
Connection::open(archive)
.unwrap()
.query_row("SELECT COUNT(*) FROM emails", [], |r| r.get(0))
.unwrap()
}
fn report(outcome: &Outcome) -> Vec<String> {
fs::read_to_string(&outcome.report)
.unwrap()
.lines()
.skip(1)
.map(|l| l.split(',').take(5).collect::<Vec<_>>().join(","))
.collect()
}
fn write_plan(path: &Path, steps: serde_json::Value) {
let plan = json!({
"format": "inbuxa-migrate-plan",
"formatVersion": 1,
"accounts": ["[email protected]", "[email protected]"],
"steps": steps,
});
fs::write(path, serde_json::to_vec_pretty(&plan).unwrap()).unwrap();
}
#[test]
fn accounts_run_their_steps_resume_and_run_again() {
let dir = TempDir::new().unwrap();
let root = dir.path();
message(&root.join("mail/ann"), "m1", "First");
let plan = root.join("plan.json");
// Relative paths are the plan's own: the steps run in its directory.
write_plan(
&plan,
json!([
{"name": "mail", "args": ["import", "maildir", "mail/{local}", "{archive}"]},
{"name": "files", "args": ["export-files", "{archive}", "out/{email}"]},
]),
);
let work = root.join("work");
// Bob has no Maildir yet: his import fails, his export never runs.
let first = migrate::run(config(&plan, &work)).unwrap();
assert_eq!((first.done, first.unfinished), (1, 1));
assert_eq!(
report(&first),
[
"[email protected],mail,done,2,2",
"[email protected],files,done,0,2",
"[email protected],mail,failed,0,0",
"[email protected],files,not run,0,0",
]
);
let detail = fs::read_to_string(&first.report).unwrap();
assert!(
detail.contains("mail/bob"),
"the reason is reported: {detail}"
);
assert_eq!(emails(&work.join("[email protected]/archive.sqlite")), 1);
assert!(root.join("out/[email protected]/manifest.json").exists());
let log = fs::read_to_string(work.join("[email protected]/mail.log")).unwrap();
assert!(
log.starts_with("== inbuxa-migrate import maildir mail/ann "),
"{log}"
);
#[cfg(unix)]
{
use std::os::unix::fs::PermissionsExt;
let mode = |p: &Path| fs::metadata(p).unwrap().permissions().mode() & 0o777;
assert_eq!(mode(&work), 0o700);
assert_eq!(mode(&work.join("[email protected]/mail.log")), 0o600);
}
// The second run leaves Ann alone (her export-files would refuse a
// directory that isn't empty) and finishes Bob.
message(&root.join("mail/bob"), "m2", "Second");
let second = migrate::run(config(&plan, &work)).unwrap();
assert_eq!((second.done, second.unfinished), (2, 0));
assert_eq!(
report(&second),
[
"[email protected],mail,done earlier,0,0",
"[email protected],files,done earlier,0,0",
"[email protected],mail,done,2,2",
"[email protected],files,done,0,2",
]
);
// --again: every step runs once more and picks up what's new.
write_plan(
&plan,
json!([{"name": "mail", "args": ["import", "maildir", "mail/{local}", "{archive}"]}]),
);
message(&root.join("mail/ann"), "m3", "Third");
let mut again = config(&plan, &work);
again.again = true;
again.only = vec!["[email protected]".into()];
let third = migrate::run(again).unwrap();
assert_eq!(report(&third), ["[email protected],mail,done,1,1"]);
assert_eq!(emails(&work.join("[email protected]/archive.sqlite")), 2);
}
#[test]
fn a_step_sees_only_the_secrets_it_names() {
let dir = TempDir::new().unwrap();
let plan = dir.path().join("plan.json");
let step = |secrets: serde_json::Value| {
json!([{
"name": "mail",
"args": ["import", "imap", "--url", "imap://127.0.0.1:9", "--allow-cleartext",
"--auth-basic", "{email}", "{archive}"],
"secrets": secrets,
}])
};
write_plan(&plan, step(json!({"password": "MIGRATE_TEST_UNSET_VAR"})));
let err = migrate::run(config(&plan, &dir.path().join("w1")))
.err()
.unwrap()
.to_string();
assert!(err.contains("$MIGRATE_TEST_UNSET_VAR is not set"), "{err}");
// No secret named: the password this test run may have in its own
// environment is not passed on, and with no terminal the step fails
// at once rather than waiting for one.
write_plan(&plan, step(json!({})));
let out = migrate::run(config(&plan, &dir.path().join("w2"))).unwrap();
assert_eq!(out.unfinished, 2);
let text = fs::read_to_string(&out.report).unwrap();
assert!(text.contains("no password provided"), "{text}");
}