jcoffey-dev is traveling from Thursday 1 October through Sunday 4 October. Issues and pull requests are welcome, and will get an answer after that. Thanks for your patience.
export --dry-run counted what it would create and printed a table, but it
missed the failures it could see coming and always exited 0.
It now checks, before anything is written:
- a message larger than the target's maxSizeUpload;
- an object too large for one request under maxSizeRequest, as a contact
with its photo inlined can be;
- a Sieve script the target would reject, with SieveScript/validate, after
the vnd.stalwart names are renamed, so what is checked is what would be
uploaded. Sieve scripts are uploaded as blobs for that check, and nothing
else is; a target without the method gets one warning.
The counts are kept instead of being dropped, so a dry run in which
anything would fail exits 5, like the real run. The report is now a plan in
plain words -- per type, what would be created, updated, left unchanged,
deleted and would fail -- followed by each predicted failure and why.
Export wrote one message per request and said nothing while it did, so a
large mailbox took hours of silence.
Messages now go in batches of up to the target's maxObjectsInSet (at most
50), their blobs uploaded several at once, up to maxConcurrentUpload and no
more than --threads; each upload thread reads the archive through its own
read-only connection. Every message is still counted on its own: one the
target rejects fails alone, a request too large is split, and a method
error on the whole call is retried a message at a time.
A batch is sent once. If it ends without a clear answer -- a dropped
connection, a gateway timeout, a partial failure -- the target is read
again, the messages that arrived count as created, and only the rest are
imported again, so none is doubled.
A progress line (count, rate, time left) is printed every few seconds for
mail, contacts and events, and each type ends with a line of what was
created, updated, left unchanged and failed.
Export matched each item against the target and then skipped it, so a second
run -- the usual final pass of a cutover -- never carried anything that had
changed at the source since the first: read and flagged state, moves between
folders, edited contacts, events and Sieve scripts. It reported them as
skipped and exited 0, while the usage guide said matched items were updated.
Matched items are now updated, with one batched /set per type:
- Email: keywords are set to the archive's, added and removed, compared
case-insensitively. Memberships of folders this run migrated are added and
removed to match; folders that exist only on the target are left alone,
and a message is never left in no folder. Properties the server did not
report are not touched.
- Contacts and events: when both copies carry `updated`, the archive's is
written only if it is newer, compared as instants so an offset or a
fraction of a second is not taken for a change; otherwise each property the
archive writes is compared, and those that differ are sent whole.
- Sieve scripts: the target's copy is downloaded and compared byte for byte
with what export would write -- after renaming Stalwart's vendor names for
an inbuxa target -- and replaced when it differs, so a renamed script is
not re-uploaded on every run.
Updated items are counted as `updated`; unchanged ones stay `skipped`. The
usage guide now describes this.
inbuxa accepts vnd.inbuxa.while and vnd.inbuxa.expressions, and names its
environment items vnd.inbuxa.*, with no alias for Stalwart's vnd.stalwart.*
names. A script carried over unchanged failed to compile on the target, and
the account was left with no filtering behind a single warning.
When the target lists vnd.inbuxa extensions in its sieveExtensions, export
now renames Stalwart's names in the three places they are names: the strings
of a require list, the item name of an environment test, and
${env.vnd.stalwart.*} references inside strings. A small tokenizer skips
comments and reads quoted strings and text: blocks whole, so other text that
happens to contain the name is copied unchanged. Each rename is printed.
Against a target without the inbuxa names nothing changes.
The activation call is now checked. A method error, or an active script
that could not be created, is logged as an error naming the script and
counted as a failure, so export exits non-zero instead of leaving
filtering off with one warning line.
jcoffey-dev
Maurus Decimus