Files
jcoffey-dev 6b65a50d0a
ci / fork-checks (pull_request) Skipped
ci / build (pull_request) Skipped
github/ci (branch) GitHub Actions
ci / github (pull_request) Successful in 14m7s
Count what's kept after deletion, and check what's really stored (#500)
Part of the inbuxa/inbuxa-admin#257 remediation (Refs inbuxa/inbuxa-admin#257).

Some stored data wasn't counted toward an account's or its tenant's disk
quota, or was checked against less than was stored.

- Undelete archive (UD-14 changed): an archived item now counts toward
  its account's and its tenant's usage while it's kept, and comes off
  when it goes: destroyed, expired, or restored (the restored item counts
  as itself from then on). Each counted item has a `q` + account + item
  key with its bytes, so it comes off exactly once (asserted) and items
  archived before this aren't taken off. Deleting never fails because of
  this; it just doesn't free space until the item goes. A restore is
  checked less what the kept copy already counts (UD-10), so restoring a
  counted item into a full account works; email ingest leaves a
  restore's check to the restore task. The per-account recount adds the
  counted archive. A deleted account's archive stays counted with the
  account and goes with it (records::discard leaves the counters).
- Held spam-training samples (UD-14a): a sample's hold on a message's
  blob now ends with the account's own copy, when the message's data is
  removed with nothing archiving it or when its archived copy goes other
  than by restore. Holds are noted as `s` + account + hash + until, and
  ended notes are cleared by the scheduled clean-up. Counting held samples
  instead would count every trained message twice while it's still in
  the mailbox (the hold and the message share one blob). The classifier
  already drops a sample whose blob is gone from its replay pool, so the
  cost is a replay pool of mail that's still kept; what it learned stays.
- Encrypted mail: ingest checks the stored size (the encrypted message
  plus the headers added on delivery) when it's more than was checked.
- iTIP: a new invitation is checked against the event and the
  notification it stores (twice the message), an update against the
  event's growth plus the notification.
- VacationResponse/set checks a bigger script against quota, like
  SieveScript/set, and answers overQuota.

Left alone: parallel writes can each pass the check before any of them
saves. A reservation would have to be released on every write path's
success and failure, and a cheaper timed reservation would refuse mail
for a moment after each write; refused SMTP deliveries become bounces.
The overshoot is one burst of (concurrent writes) x (largest item),
after which every check fails, and upstream behaves the same.

Tests: undelete_tests (counted while kept, comes off on destroy, a
counted item restores into a full account, a refused restore needs the
account over its limit), vacation_response (a script past the quota is
refused), samples key layout unit test.
2026-11-20 10:13:11 -07:00
..