export: a dry run that predicts failures and exits as the real run would

jcoffey-dev commented

2026-09-30 19:46:58 +00:00

Owner

Roadmap item 10: a dry run that predicts failures. Stacked on #10 (feat/export-progress-batching); merge #10 first, and this PR's own diff is the last two commits.

export --dry-run now checks, before anything is written:

Messages larger than the target's maxSizeUpload.
Objects too large for one request under maxSizeRequest: a contact or event with a photo inlined, for instance.
Sieve scripts the target would reject, via SieveScript/validate, run on the renamed bytes (vnd.stalwart.* → vnd.inbuxa.*) so what's checked is what would be uploaded. The method needs the script as a blob, so a dry run uploads Sieve scripts, and nothing else; the server discards an unused blob. A target without the method gets one warning, and its scripts aren't checked.

The counts are kept rather than dropped, so a dry run where anything would fail exits 5, like the real run (main skips its count lines for a dry run, since the plan replaces them). The report is now a plan in plain words: per type, what would be created, updated, left unchanged, deleted (--prune) and would fail, then each predicted failure with its reason.

Also fixed, as its own commit: a dry run no longer claims the default address book. It used to send AddressBook/set with onSuccessSetIsDefault, a write, naming the dry run's made-up id.

Tests: an oversize message and an oversize contact each predicted to fail with exit-5 semantics and no writes; a rejected Sieve script counted as a failure; a valid one validated on its renamed bytes with nothing written; a target without SieveScript/validate still gets a plan; plan text shape and the --prune column. 1,394 tests pass; fmt and clippy clean. Documented in docs/usage.md.

Limit: predicted failures from the size and Sieve checks are listed with reasons; any other failure a dry run counts (rare) is summed as "N more; the warnings above say why".

Roadmap item 10: a dry run that predicts failures. **Stacked on #10** (`feat/export-progress-batching`); merge #10 first, and this PR's own diff is the last two commits. `export --dry-run` now checks, before anything is written: - **Messages** larger than the target's `maxSizeUpload`. - **Objects too large for one request** under `maxSizeRequest`: a contact or event with a photo inlined, for instance. - **Sieve scripts** the target would reject, via `SieveScript/validate`, run on the renamed bytes (`vnd.stalwart.*` → `vnd.inbuxa.*`) so what's checked is what would be uploaded. The method needs the script as a blob, so a dry run uploads Sieve scripts, and nothing else; the server discards an unused blob. A target without the method gets one warning, and its scripts aren't checked. The counts are kept rather than dropped, so a dry run where anything would fail **exits 5, like the real run** (`main` skips its count lines for a dry run, since the plan replaces them). The report is now a plan in plain words: per type, what would be created, updated, left unchanged, deleted (`--prune`) and would fail, then each predicted failure with its reason. Also fixed, as its own commit: **a dry run no longer claims the default address book.** It used to send `AddressBook/set` with `onSuccessSetIsDefault`, a write, naming the dry run's made-up id. Tests: an oversize message and an oversize contact each predicted to fail with exit-5 semantics and no writes; a rejected Sieve script counted as a failure; a valid one validated on its renamed bytes with nothing written; a target without `SieveScript/validate` still gets a plan; plan text shape and the `--prune` column. 1,394 tests pass; fmt and clippy clean. Documented in `docs/usage.md`. Limit: predicted failures from the size and Sieve checks are listed with reasons; any other failure a dry run counts (rare) is summed as "N more; the warnings above say why".

jcoffey-dev added 4 commits 2026-09-30 19:46:58 +00:00

http: a way to send a write that must not be applied twice 4ce772d6e5

A POST that failed in transport -- including a timeout while waiting for the
answer -- is resent today, and so is one that got a 502 or 504. For a write
such as Email/import the server may already have applied the first copy,
so the resend can create a duplicate.

post_json_once and Request::send_once send it once: a transport failure or
a gateway error comes back to the caller, which can check the target before
trying again. 429 and 503, which mean the request was not processed, are
still retried.

export: batch Email/import, upload in parallel, and show progress

ci / test (pull_request) Skipped

Details

github/ci (branch) GitHub Actions

Details

ci / github (pull_request) Successful in 3m11s

Details

ci / announce (pull_request) Skipped

Details

3f832e17b2

Export wrote one message per request and said nothing while it did, so a
large mailbox took hours of silence.

Messages now go in batches of up to the target's maxObjectsInSet (at most
50), their blobs uploaded several at once, up to maxConcurrentUpload and no
more than --threads; each upload thread reads the archive through its own
read-only connection. Every message is still counted on its own: one the
target rejects fails alone, a request too large is split, and a method
error on the whole call is retried a message at a time.

A batch is sent once. If it ends without a clear answer -- a dropped
connection, a gateway timeout, a partial failure -- the target is read
again, the messages that arrived count as created, and only the rest are
imported again, so none is doubled.

A progress line (count, rate, time left) is printed every few seconds for
mail, contacts and events, and each type ends with a line of what was
created, updated, left unchanged and failed.

export: a dry run no longer claims the default address book 1d63dfe25e

When export created a default address book, it went on to send
AddressBook/set with onSuccessSetIsDefault -- in a dry run too, naming the
dry run's made-up id. A dry run writes nothing; the claim is now only
counted as planned.

export: a dry run that predicts failures and exits as the real run would

ci / test (pull_request) Skipped

Details

github/ci (branch) GitHub Actions

Details

ci / github (pull_request) Successful in 2m30s

Details

ci / announce (pull_request) Skipped

Details

e47c074d43

export --dry-run counted what it would create and printed a table, but it
missed the failures it could see coming and always exited 0.

It now checks, before anything is written:
- a message larger than the target's maxSizeUpload;
- an object too large for one request under maxSizeRequest, as a contact
  with its photo inlined can be;
- a Sieve script the target would reject, with SieveScript/validate, after
  the vnd.stalwart names are renamed, so what is checked is what would be
  uploaded. Sieve scripts are uploaded as blobs for that check, and nothing
  else is; a target without the method gets one warning.

The counts are kept instead of being dropped, so a dry run in which
anything would fail exits 5, like the real run. The report is now a plan in
plain words -- per type, what would be created, updated, left unchanged,
deleted and would fail -- followed by each predicted failure and why.

jcoffey-dev merged commit d3ce33b8e2 into main

2026-09-30 19:49:53 +00:00

jcoffey-dev deleted branch feat/predictive-dry-run

2026-09-30 19:49:53 +00:00

export: a dry run that predicts failures and exits as the real run would #11