export: batch Email/import, upload in parallel, and show progress
ci / test (pull_request) Skipped
github/ci (branch) GitHub Actions
ci / github (pull_request) Successful in 3m11s
ci / announce (pull_request) Skipped

Export wrote one message per request and said nothing while it did, so a
large mailbox took hours of silence.

Messages now go in batches of up to the target's maxObjectsInSet (at most
50), their blobs uploaded several at once, up to maxConcurrentUpload and no
more than --threads; each upload thread reads the archive through its own
read-only connection. Every message is still counted on its own: one the
target rejects fails alone, a request too large is split, and a method
error on the whole call is retried a message at a time.

A batch is sent once. If it ends without a clear answer -- a dropped
connection, a gateway timeout, a partial failure -- the target is read
again, the messages that arrived count as created, and only the rest are
imported again, so none is doubled.

A progress line (count, rate, time left) is printed every few seconds for
mail, contacts and events, and each type ends with a line of what was
created, updated, left unchanged and failed.
This commit is contained in:
2026-09-30 12:39:57 -07:00
parent 4ce772d6e5
commit 3f832e17b2
8 changed files with 1084 additions and 204 deletions
+14
View File
@@ -229,6 +229,20 @@ what changed at the source in between.
- **Sieve scripts** are matched by name, and the target's is replaced when
its content differs from the archive's.
Messages go in batches, as many to an `Email/import` as the target's
`maxObjectsInSet` allows, up to 50, with their blobs uploaded several at a
time -- the target's `maxConcurrentUpload`, and no more than `--threads`.
Each message is still counted on its own: one the target rejects fails
alone, and the rest of its batch lands. A batch that ends without a clear
answer -- a dropped connection, a gateway timeout -- is never sent again as
it was. The target is read first, the messages that arrived are counted as
created, and only the rest are imported again, so none is ever doubled.
While it runs, export prints a line every few seconds for mail, contacts and
events: how many of how many, how fast, and about how long is left. Each
type ends with a line of what was created, updated, left unchanged and
failed.
`--prune` also deletes what is on the target and not in the archive. It asks
first; `--yes` answers for it, for scripts. Export speaks JMAP only.