jcoffey-dev is traveling from Thursday 1 October through Sunday 4 October. Issues and pull requests are welcome, and will get an answer after that. Thanks for your patience.
An item the hold covers whose stored record or content can't be read
goes in exceptions.csv with the path it would have had and the reason,
rather than being left out silently. The file is always in the ZIP, so a
header-only one shows nothing was missed, and manifest.sha256 carries
its hash beside the manifest's.
inbuxa:HoldExport/set takes a hold, optionally some of the accounts it
covers, and a reason; the collection runs in the background and get
says when it's ready. The ZIP has, per account, mail as .eml under its
folders, calendars as .ics, contacts as .vcf, files as stored, and the
archived items the hold keeps under archived/; a manifest.csv gives each
entry's account, kind, folder, date, whether it was archived, size and
SHA-256, and manifest.sha256 hashes the manifest. Accounts the hold
doesn't cover are left out, and items outside its date range are too:
live mail by arrival, events by start, and archived items the same way,
so an export doesn't carry deleted items that only another hold keeps.
The finished file is a blob of whoever started the export, so only they
download it, and it lasts as long as any upload (uploadTtl). Exports
are records under the hold (SUBSPACE_INBUXA H/e): never changed or
destroyed, each with its status, counts, size and checksum. Starting
one needs sysLegalHoldExport, an active hold and a reason, and is
recorded in the audit log like the audit log's own export.
The build is in memory and capped at 2 GB; bigger holds fail with a
message saying so, and are split by picking accounts.
Tested: unit tests for safe ZIP names and the manifest and its hash;
the legal_hold system test, on RocksDB, PostgreSQL and MySQL, exports a
hold end to end (live and archived mail, the manifest's hash, an asked-
for account the hold doesn't cover left out) and checks the refusals
(no reason, a user without the permission, a released hold) and the
audit record; and by hand from the console on a local server. Not
covered by a test: the archived-item date range with two holds of
different ranges over one account.
An account's or mailing list's name is only its local part, so the log
said "Account ken.gosling" where two domains could each have one; it
now says [email protected]. A change to a legal hold was
recorded under its id; the hold's current state is now read first, so
the record carries its case name and each change reads before/after.
inbuxa:LegalHold/get answers accountsCovered, itemsHeld and sizeHeld
when asked: the accounts a hold reaches now (deleted ones it keeps
included) and the archived items it keeps, with their size. Worked out
in one pass over accounts and archive, only for requests that name them.
Held items stay out of the user's quota, as all archived copies do
(LH-9).
Destroying a held account removes the login, as offboarding needs, but
keeps its data as a deleted account with no expiry, whether or not
undelete keeps accounts; its addresses stay reserved and its holds name
it from then on. Destroy-now refuses it, and its DestroyAccount task
defers itself while it's held or its time hasn't come. Holds placed or
released later freeze or free kept accounts in the same settle pass,
with 30 days' grace after the last release (LH-8, LH-10).
Placing or widening a hold freezes what's already archived in its scope
and range, its old deadline noted; releasing one gives each item no other
hold covers that deadline back, or release plus 30 days if later. One
pass over the archive does both and changes nothing twice (LH-6, LH-10,
LH-11). A held archived item can't be destroyed; restoring still can,
and the hold is named only to callers who may see holds (LH-7). Audit
records about a held account survive the purge (AU-7).
Fixes the daily clean-up of expired archived items (UD-13), which never
found any: the registry's unfiltered query reads an all-ids index that
archived items aren't in. Items are now walked account by account, kept
deleted accounts included. Expired items were still removed whenever
their account's archive was read.
Every way of deleting mail (JMAP, IMAP EXPUNGE, POP3, mailbox removal,
Trash emptying) and Sieve scripts, events, contacts and files now asks
how the account's deletions are kept: a hold keeps them with no expiry
(archivedUntil 9999-12-31), even with undelete off; otherwise undelete's
period applies as before (LH-4).
A hold's date range decides by the item's own date (LH-3). Mail is noted
as held at deletion and settled when it's archived, once its received
date is known; outside the range it gets undelete's deadline or isn't
kept. Events go by their start, with a day's slack for time zones;
recurring events, contacts, files and scripts are held whole.
A groupware item's note now stays until its archive succeeds, and a
failure retries the task instead of being logged and lost (LH-5).
A hold reaches an account by name, through any of its addresses'
domains, its groups or its tenant, as they are now, so an account added
to a held domain later is held too. An account that leaves a held
domain, group or tenant stays held: the registry write hook adds it to
the hold by name on every account change, whoever makes it (LH-2).
Server::holds_on answers for the deletion paths, from the store each
time so a hold binds every node at once.
inbuxa:LegalHold get/set places a hold on accounts, groups, domains,
tenants or the whole server, with an optional date range. A hold's range
and scope can only widen, a released hold is read-only, and none is ever
deleted. Placing, changing and releasing each need a reason and are
audited (LH-1, LH-3, LH-10, AU-12).
Permissions 669-672 (see, place, widen or release, export held data)
go to server administrators only; the tenant ceiling always strips them,
as it does Impersonate (LH-13). Schema: Compliance > Legal Holds.
What a hold keeps comes next, through the undelete hooks.
Also moves the lock expiry helpers below the lock module's imports.
jcoffey-dev