Count what's kept after deletion, and check what's really stored (#500) #502

Merged
jcoffey-dev merged 1 commits from fix/tenant-uncounted-storage into main 2026-11-20 17:29:06 +00:00
Owner

Part of the inbuxa/inbuxa-admin#257 remediation (Refs inbuxa/inbuxa-admin#257).

Some stored data wasn't counted toward an account's or its tenant's disk
quota, or was checked against less than was stored.

  • Undelete archive (UD-14 changed): an archived item now counts toward
    its account's and its tenant's usage while it's kept, and comes off
    when it goes: destroyed, expired, or restored (the restored item counts
    as itself from then on). Each counted item has a q + account + item
    key with its bytes, so it comes off exactly once (asserted) and items
    archived before this aren't taken off. Deleting never fails because of
    this; it just doesn't free space until the item goes. A restore is
    checked less what the kept copy already counts (UD-10), so restoring a
    counted item into a full account works; email ingest leaves a
    restore's check to the restore task. The per-account recount adds the
    counted archive. A deleted account's archive stays counted with the
    account and goes with it (records::discard leaves the counters).
  • Held spam-training samples (UD-14a): a sample's hold on a message's
    blob now ends with the account's own copy, when the message's data is
    removed with nothing archiving it or when its archived copy goes other
    than by restore. Holds are noted as s + account + hash + until, and
    ended notes are cleared by the scheduled clean-up. Counting held samples
    instead would count every trained message twice while it's still in
    the mailbox (the hold and the message share one blob). The classifier
    already drops a sample whose blob is gone from its replay pool, so the
    cost is a replay pool of mail that's still kept; what it learned stays.
  • Encrypted mail: ingest checks the stored size (the encrypted message
    plus the headers added on delivery) when it's more than was checked.
  • iTIP: a new invitation is checked against the event and the
    notification it stores (twice the message), an update against the
    event's growth plus the notification.
  • VacationResponse/set checks a bigger script against quota, like
    SieveScript/set, and answers overQuota.

Left alone: parallel writes can each pass the check before any of them
saves. A reservation would have to be released on every write path's
success and failure, and a cheaper timed reservation would refuse mail
for a moment after each write; refused SMTP deliveries become bounces.
The overshoot is one burst of (concurrent writes) x (largest item),
after which every check fails, and upstream behaves the same.

Tests: undelete_tests (counted while kept, comes off on destroy, a
counted item restores into a full account, a refused restore needs the
account over its limit), vacation_response (a script past the quota is
refused), samples key layout unit test.

Refs #500.

Part of the inbuxa/inbuxa-admin#257 remediation (Refs inbuxa/inbuxa-admin#257). Some stored data wasn't counted toward an account's or its tenant's disk quota, or was checked against less than was stored. - Undelete archive (UD-14 changed): an archived item now counts toward its account's and its tenant's usage while it's kept, and comes off when it goes: destroyed, expired, or restored (the restored item counts as itself from then on). Each counted item has a `q` + account + item key with its bytes, so it comes off exactly once (asserted) and items archived before this aren't taken off. Deleting never fails because of this; it just doesn't free space until the item goes. A restore is checked less what the kept copy already counts (UD-10), so restoring a counted item into a full account works; email ingest leaves a restore's check to the restore task. The per-account recount adds the counted archive. A deleted account's archive stays counted with the account and goes with it (records::discard leaves the counters). - Held spam-training samples (UD-14a): a sample's hold on a message's blob now ends with the account's own copy, when the message's data is removed with nothing archiving it or when its archived copy goes other than by restore. Holds are noted as `s` + account + hash + until, and ended notes are cleared by the scheduled clean-up. Counting held samples instead would count every trained message twice while it's still in the mailbox (the hold and the message share one blob). The classifier already drops a sample whose blob is gone from its replay pool, so the cost is a replay pool of mail that's still kept; what it learned stays. - Encrypted mail: ingest checks the stored size (the encrypted message plus the headers added on delivery) when it's more than was checked. - iTIP: a new invitation is checked against the event and the notification it stores (twice the message), an update against the event's growth plus the notification. - VacationResponse/set checks a bigger script against quota, like SieveScript/set, and answers overQuota. Left alone: parallel writes can each pass the check before any of them saves. A reservation would have to be released on every write path's success and failure, and a cheaper timed reservation would refuse mail for a moment after each write; refused SMTP deliveries become bounces. The overshoot is one burst of (concurrent writes) x (largest item), after which every check fails, and upstream behaves the same. Tests: undelete_tests (counted while kept, comes off on destroy, a counted item restores into a full account, a refused restore needs the account over its limit), vacation_response (a script past the quota is refused), samples key layout unit test. Refs #500.
jcoffey-dev added this to the 27.1 milestone 2026-11-20 16:57:05 +00:00
jcoffey-dev added 1 commit 2026-11-20 17:14:04 +00:00
Count what's kept after deletion, and check what's really stored (#500)
ci / fork-checks (pull_request) Skipped
ci / build (pull_request) Skipped
github/ci (branch) GitHub Actions
ci / github (pull_request) Successful in 14m7s
6b65a50d0a
Part of the inbuxa/inbuxa-admin#257 remediation (Refs inbuxa/inbuxa-admin#257).

Some stored data wasn't counted toward an account's or its tenant's disk
quota, or was checked against less than was stored.

- Undelete archive (UD-14 changed): an archived item now counts toward
  its account's and its tenant's usage while it's kept, and comes off
  when it goes: destroyed, expired, or restored (the restored item counts
  as itself from then on). Each counted item has a `q` + account + item
  key with its bytes, so it comes off exactly once (asserted) and items
  archived before this aren't taken off. Deleting never fails because of
  this; it just doesn't free space until the item goes. A restore is
  checked less what the kept copy already counts (UD-10), so restoring a
  counted item into a full account works; email ingest leaves a
  restore's check to the restore task. The per-account recount adds the
  counted archive. A deleted account's archive stays counted with the
  account and goes with it (records::discard leaves the counters).
- Held spam-training samples (UD-14a): a sample's hold on a message's
  blob now ends with the account's own copy, when the message's data is
  removed with nothing archiving it or when its archived copy goes other
  than by restore. Holds are noted as `s` + account + hash + until, and
  ended notes are cleared by the scheduled clean-up. Counting held samples
  instead would count every trained message twice while it's still in
  the mailbox (the hold and the message share one blob). The classifier
  already drops a sample whose blob is gone from its replay pool, so the
  cost is a replay pool of mail that's still kept; what it learned stays.
- Encrypted mail: ingest checks the stored size (the encrypted message
  plus the headers added on delivery) when it's more than was checked.
- iTIP: a new invitation is checked against the event and the
  notification it stores (twice the message), an update against the
  event's growth plus the notification.
- VacationResponse/set checks a bigger script against quota, like
  SieveScript/set, and answers overQuota.

Left alone: parallel writes can each pass the check before any of them
saves. A reservation would have to be released on every write path's
success and failure, and a cheaper timed reservation would refuse mail
for a moment after each write; refused SMTP deliveries become bounces.
The overshoot is one burst of (concurrent writes) x (largest item),
after which every check fails, and upstream behaves the same.

Tests: undelete_tests (counted while kept, comes off on destroy, a
counted item restores into a full account, a refused restore needs the
account over its limit), vacation_response (a script past the quota is
refused), samples key layout unit test.
jcoffey-dev force-pushed fix/tenant-uncounted-storage from fac6c6ec13 to 6b65a50d0a 2026-11-20 17:14:04 +00:00 Compare
jcoffey-dev merged commit 26eaf1869f into main 2026-11-20 17:29:06 +00:00
jcoffey-dev deleted branch fix/tenant-uncounted-storage 2026-11-20 17:29:07 +00:00
Sign in to join this conversation.