jcoffey-dev is traveling from Thursday 1 October through Sunday 4 October. Issues and pull requests are welcome, and will get an answer after that. Thanks for your patience.
Two things let a large mailbox fill memory. IMAP workers handed every
fetched message to the archive writer through an unbounded queue, so fast
workers could hold whole folders' worth of bodies while the single writer
caught up. And fetch batches were sized by count only: 20 items per EWS
GetItem, 8 connections at a time, is a few megabytes of ordinary mail and
several gigabytes of large attachments.
- The IMAP event queue is bounded at two events per worker, so a worker
waits for the writer instead of running ahead of it.
- Fetch batches are bounded by bytes as well as by count, with a shared
helper, `sync::batch::by_count_and_bytes`: 32 MiB by default, and a
single larger message goes alone.
- IMAP learns each new message's RFC822.SIZE on the control connection,
in 1000-UID metadata fetches, before the body fetch. A server that won't
say leaves the chunks sized by count. `--fetch-batch-mib` sets the cap.
- EWS asks FindItem for `item:Size` and packs GetItem batches by it,
fetched `--ews-connections` batches at a time. `--ews-getitem-batch-mib`
sets the cap. Items from an incremental SyncFolderItems run carry no
size and stay batched by count.
- docs/usage.md describes both options.
IMAP import wrote a whole folder in one transaction and stopped the folder
at the first message it could not import. A crash near the end of a large
INBOX kept nothing, and one bad INTERNALDATE lost the rest of the folder.
Worse, when a folder stopped early, fetches still in flight for it could be
filed into the next folder's mailbox.
- The transaction is committed after every fetch chunk. Each message is
written in its own savepoint, so what is committed is always whole, and a
rerun fetches only the UIDs still missing.
- A message that cannot be imported is rolled back on its own, logged with
its folder and UID, and counted as failed; the folder carries on, and the
message stays out of the UID map so the next run tries it again. Archive
and I/O errors still stop the run.
- Fetch jobs and events carry a folder generation. Moving to a new folder
cancels queued work for older ones, and any event from an older
generation is dropped, never filed. Shutdown drains in-flight events
before joining the workers, so it cannot hang on a blocked worker.
- INTERNALDATE month names are matched in any case.
- Maildir import gets the same per-message savepoint, and commits every 500
new messages instead of once per folder.
The crate, binary, library path and every name the tool writes or reads
become inbuxa-migrate: the CLI name and about text, the HTTP user agent,
the credential variables (INBUXA_MIGRATE_PASSWORD, _TOKEN,
_EWS_CLIENT_SECRET and _GRAPH_TOKEN, with no fallback to the old names),
thread names, synthesized UIDs and calendar addresses
(urn:x-inbuxa-migrate:attendee:), and the Maildir file suffix used by the
export script. This is a new tool, so archives written by Vandelay are not
read back.
The version moves to the date scheme the release tags use (2026.9.30), and
the package metadata points at the inbuxa-migrate repository. The cargo-dist
and MSI metadata goes with the release tooling it served.
Modified files carry an added copyright line; the original notices stay.
jcoffey-dev
Maurus Decimus