export: batch Email/import, upload in parallel, and show progress

jcoffey-dev commented

2026-09-30 19:40:19 +00:00

Owner

Roadmap item 8: export shows progress and runs in batches.

Batched Email/import: up to the target's maxObjectsInSet, capped at 50. Each message is still counted on its own: one the target rejects fails alone, a request too large is split in two, and a method error that rejects the whole call is retried one message at a time.
Parallel uploads: blobs are uploaded several at once, up to maxConcurrentUpload and no more than --threads. Each upload thread reads the archive through its own read-only connection, so at most one blob per thread is in memory.
No doubled messages: a batch is sent once (Request::send_once): a transport failure or a 502/504 is not resent. If a batch ends without a clear answer, the target is read again, what arrived counts as created, and only the rest is imported again, one at a time. Today's single imports could already be resent after a read timeout; this is safer than before, not only than a naive batch.
Unchanged: matching, folding copies across folders (#4) and updates (#7) happen before any of this and are untouched.
Progress: a line every 5 seconds for mail, contacts and events (count, rate, time left), and a "done" line per type with created, updated, unchanged and failed.

Tests: batches honor maxObjectsInSet; a rejected message in a batch fails alone; an unclear batch is settled against the target and only the missing message is resent; the upload pool never runs more than its cap; send-once doesn't resend after a transport failure or a 504, and still retries a 503; progress line shapes. The upstream test that pinned one message per call is replaced by these. 1,387 tests pass; fmt and clippy clean.

Limit: contacts and events are still created one per request; only mail is batched here.

Roadmap item 8: export shows progress and runs in batches. - **Batched `Email/import`:** up to the target's `maxObjectsInSet`, capped at 50. Each message is still counted on its own: one the target rejects fails alone, a request too large is split in two, and a method error that rejects the whole call is retried one message at a time. - **Parallel uploads:** blobs are uploaded several at once, up to `maxConcurrentUpload` and no more than `--threads`. Each upload thread reads the archive through its own read-only connection, so at most one blob per thread is in memory. - **No doubled messages:** a batch is sent once (`Request::send_once`): a transport failure or a 502/504 is not resent. If a batch ends without a clear answer, the target is read again, what arrived counts as created, and only the rest is imported again, one at a time. Today's single imports could already be resent after a read timeout; this is safer than before, not only than a naive batch. - **Unchanged:** matching, folding copies across folders (#4) and updates (#7) happen before any of this and are untouched. - **Progress:** a line every 5 seconds for mail, contacts and events (count, rate, time left), and a "done" line per type with created, updated, unchanged and failed. Tests: batches honor `maxObjectsInSet`; a rejected message in a batch fails alone; an unclear batch is settled against the target and only the missing message is resent; the upload pool never runs more than its cap; send-once doesn't resend after a transport failure or a 504, and still retries a 503; progress line shapes. The upstream test that pinned one message per call is replaced by these. 1,387 tests pass; fmt and clippy clean. Limit: contacts and events are still created one per request; only mail is batched here.

jcoffey-dev added 2 commits 2026-09-30 19:40:19 +00:00

http: a way to send a write that must not be applied twice 4ce772d6e5

A POST that failed in transport -- including a timeout while waiting for the
answer -- is resent today, and so is one that got a 502 or 504. For a write
such as Email/import the server may already have applied the first copy,
so the resend can create a duplicate.

post_json_once and Request::send_once send it once: a transport failure or
a gateway error comes back to the caller, which can check the target before
trying again. 429 and 503, which mean the request was not processed, are
still retried.

export: batch Email/import, upload in parallel, and show progress

ci / test (pull_request) Skipped

Details

github/ci (branch) GitHub Actions

Details

ci / github (pull_request) Successful in 3m11s

Details

ci / announce (pull_request) Skipped

Details

3f832e17b2

Export wrote one message per request and said nothing while it did, so a
large mailbox took hours of silence.

Messages now go in batches of up to the target's maxObjectsInSet (at most
50), their blobs uploaded several at once, up to maxConcurrentUpload and no
more than --threads; each upload thread reads the archive through its own
read-only connection. Every message is still counted on its own: one the
target rejects fails alone, a request too large is split, and a method
error on the whole call is retried a message at a time.

A batch is sent once. If it ends without a clear answer -- a dropped
connection, a gateway timeout, a partial failure -- the target is read
again, the messages that arrived count as created, and only the rest are
imported again, so none is doubled.

A progress line (count, rate, time left) is printed every few seconds for
mail, contacts and events, and each type ends with a line of what was
created, updated, left unchanged and failed.

jcoffey-dev referenced this pull request

2026-09-30 19:46:58 +00:00

export: a dry run that predicts failures and exits as the real run would #11

jcoffey-dev merged commit fde1f0f68b into main

2026-09-30 19:47:47 +00:00

jcoffey-dev deleted branch feat/export-progress-batching

2026-09-30 19:47:47 +00:00

export: batch Email/import, upload in parallel, and show progress #10