import: bound the memory an IMAP or EWS import holds
Two things let a large mailbox fill memory. IMAP workers handed every
fetched message to the archive writer through an unbounded queue, so fast
workers could hold whole folders' worth of bodies while the single writer
caught up. And fetch batches were sized by count only: 20 items per EWS
GetItem, 8 connections at a time, is a few megabytes of ordinary mail and
several gigabytes of large attachments.
- The IMAP event queue is bounded at two events per worker, so a worker
waits for the writer instead of running ahead of it.
- Fetch batches are bounded by bytes as well as by count, with a shared
helper, `sync::batch::by_count_and_bytes`: 32 MiB by default, and a
single larger message goes alone.
- IMAP learns each new message's RFC822.SIZE on the control connection,
in 1000-UID metadata fetches, before the body fetch. A server that won't
say leaves the chunks sized by count. `--fetch-batch-mib` sets the cap.
- EWS asks FindItem for `item:Size` and packs GetItem batches by it,
fetched `--ews-connections` batches at a time. `--ews-getitem-batch-mib`
sets the cap. Items from an incremental SyncFolderItems run carry no
size and stay batched by count.
- docs/usage.md describes both options.
This commit is contained in:
+17
-2
@@ -83,7 +83,7 @@ inbuxa-migrate import imap \
|
||||
[--include <REGEX>...] [--exclude <REGEX>...] [--exclude-special <ROLE>...] \
|
||||
[--folder <NAME>...] [--subscribed-only] [--noautomap] \
|
||||
[--include-deleted] [--allow-cleartext] [--compress] \
|
||||
[--fetch-batch <N>] [--imap-connections <1..8>] \
|
||||
[--fetch-batch <N>] [--fetch-batch-mib <MIB>] [--imap-connections <1..8>] \
|
||||
<ARCHIVE>
|
||||
```
|
||||
|
||||
@@ -91,6 +91,14 @@ Imports mail, and only mail, from any IMAP server. Folders are chosen with
|
||||
`--include` and `--exclude` patterns, or by exact name with `--folder`, but
|
||||
not both. `--exclude-special` drops folders by SPECIAL-USE role.
|
||||
|
||||
Messages are fetched in chunks of at most `--fetch-batch` messages and
|
||||
`--fetch-batch-mib` MiB (32 by default), so a folder of large attachments is
|
||||
fetched a little at a time, like any other. A single message larger than the
|
||||
cap is fetched on its own. Each chunk is written to the archive as it
|
||||
arrives: an interrupted import keeps what it fetched, and the next run picks
|
||||
up from there. A message that can't be imported is reported with its folder
|
||||
and UID, and the rest of the folder carries on.
|
||||
|
||||
### CalDAV
|
||||
|
||||
```
|
||||
@@ -170,7 +178,8 @@ inbuxa-migrate import exchange-ews \
|
||||
(--auth-basic <USER> [--auth-password <PASS>] \
|
||||
| --auth-bearer [TOKEN] [--ews-tenant <T> --ews-client-id <ID> \
|
||||
(--ews-device-code | --ews-client-secret <SECRET>)]) \
|
||||
[--ews-connections <1..8>] [--ews-getitem-batch <N>] [--ews-attachment-batch <N>] \
|
||||
[--ews-connections <1..8>] [--ews-getitem-batch <N>] [--ews-getitem-batch-mib <MIB>] \
|
||||
[--ews-attachment-batch <N>] \
|
||||
[--ews-no-syncfolderitems] \
|
||||
<ARCHIVE>
|
||||
```
|
||||
@@ -180,6 +189,12 @@ Imports a mailbox from an on-premises Exchange Server through EWS. Without
|
||||
Basic, with a bearer token acquired beforehand, with OAuth's interactive
|
||||
device-code flow, or with app-only client credentials.
|
||||
|
||||
Items are fetched in GetItem batches of at most `--ews-getitem-batch` items
|
||||
and `--ews-getitem-batch-mib` MiB (32 by default), `--ews-connections` at a
|
||||
time. The byte cap applies where Exchange reports each item's size, which it
|
||||
does on a full listing; items found through an incremental sync are batched
|
||||
by count.
|
||||
|
||||
For Exchange Online, use `exchange-graph` instead. Microsoft is retiring EWS in Exchange Online: from October 1, 2026 it is blocked unless a tenant administrator sets `EwsEnabled` to `True` and adds the client id to `EwsAllowedAppIDs`, and on April 1, 2027 it is switched off for every tenant. On-premises Exchange Server is not affected.
|
||||
|
||||
### Microsoft Exchange (Graph)
|
||||
|
||||
Reference in New Issue
Block a user