Store check: find blobs that are missing, damaged or unlinked (#322) #324

Merged
jcoffey-dev merged 1 commits from feat/322-verify-store into main 2026-11-09 02:10:35 +00:00
Owner

Fixes #322.

An administrator can now ask the server whether its blobs are all there and intact. The check reports what it finds and changes nothing.

  • quick: every blob the data links to is in the blob store.
  • full: each one also reads back whole, and its BLAKE3 hash matches its id. That catches a blob whose content is wrong.
  • Mail is also checked from the message side. A message whose metadata names a blob that nothing links to any more is reported as unlinked (the case in forum topic 238), along with whether the blob is still there.

It's an inbuxa:StoreCheck under urn:inbuxa:jmap:

  • Creating one starts it in the background on the node you reached, against the stores every node shares.
  • It's throttled: 1,000 blobs/s and 20 MB/s by default, 0 for no limit.
  • It saves its progress every 10 seconds.
  • One check runs at a time. A check that goes quiet for 5 minutes, or whose node restarts, reads as interrupted.
  • The first 1,000 problems are kept, each with its account, object, blob id and what was wrong. Every problem is also logged as store.data-corruption.
  • action: "stop" stops a check. A finished one can be destroyed, and the latest 20 are kept.

Permission: the new sysStoreCheck (695) is in the superuser and admin grants. A tenant administrator is refused, because a check reads the whole server. Starting and stopping a check are audited.

Spec: docs/spec/features/store-check.md, plus a "Not a rebuild" line in SPEC.md.

Tests: system::store_check runs against a real server:

  • a clean store reads clean;
  • then a missing blob, a blob shared by two messages, an overwritten blob (full mode only) and a removed link;
  • plus the one-at-a-time refusal, stop, the destroy rules, bad input, the audit events, a user's refusal and a tenant administrator's.

There are also unit tests for the model and the throttle.

Not in this PR:

  • the console page;
  • an inbuxa-cli command;
  • a docs page;
  • metadata checks for files and Sieve scripts;
  • #323, the online snapshot.

Local checks: fmt, cargo build -p inbuxa, cargo test --workspace --no-run, store with s3,azure, and the fork checks.

Fixes #322. An administrator can now ask the server whether its blobs are all there and intact. The check reports what it finds and changes nothing. - **quick**: every blob the data links to is in the blob store. - **full**: each one also reads back whole, and its BLAKE3 hash matches its id. That catches a blob whose content is wrong. - Mail is also checked from the message side. A message whose metadata names a blob that nothing links to any more is reported as `unlinked` (the case in forum topic 238), along with whether the blob is still there. It's an `inbuxa:StoreCheck` under `urn:inbuxa:jmap`: - Creating one starts it in the background on the node you reached, against the stores every node shares. - It's throttled: 1,000 blobs/s and 20 MB/s by default, 0 for no limit. - It saves its progress every 10 seconds. - One check runs at a time. A check that goes quiet for 5 minutes, or whose node restarts, reads as `interrupted`. - The first 1,000 problems are kept, each with its account, object, blob id and what was wrong. Every problem is also logged as `store.data-corruption`. - `action: "stop"` stops a check. A finished one can be destroyed, and the latest 20 are kept. **Permission:** the new `sysStoreCheck` (695) is in the superuser and admin grants. A tenant administrator is refused, because a check reads the whole server. Starting and stopping a check are audited. **Spec:** `docs/spec/features/store-check.md`, plus a "Not a rebuild" line in SPEC.md. **Tests:** `system::store_check` runs against a real server: - a clean store reads clean; - then a missing blob, a blob shared by two messages, an overwritten blob (full mode only) and a removed link; - plus the one-at-a-time refusal, stop, the destroy rules, bad input, the audit events, a user's refusal and a tenant administrator's. There are also unit tests for the model and the throttle. **Not in this PR:** - the console page; - an `inbuxa-cli` command; - a docs page; - metadata checks for files and Sieve scripts; - #323, the online snapshot. **Local checks:** fmt, `cargo build -p inbuxa`, `cargo test --workspace --no-run`, store with `s3,azure`, and the fork checks.
jcoffey-dev added 1 commit 2026-11-09 02:01:56 +00:00
Store check: find blobs that are missing, damaged or unlinked (#322)
ci / fork-checks (pull_request) Skipped
ci / build (pull_request) Skipped
github/ci (branch) GitHub Actions
ci / github (pull_request) Successful in 8m5s
dfdc47ae9e
An administrator can now ask the server to check its blobs. A quick
check confirms that every blob the data holds a link to is in the blob
store. A full check also reads each one back and compares its BLAKE3
hash with its id. Mail is checked from the message side too, so a
message whose metadata names a blob that nothing links to any more is
reported. That is the case from forum topic 238. The check changes
nothing.

It is an inbuxa:StoreCheck under urn:inbuxa:jmap. Creating one starts
it in the background on the node that was reached. It runs one slice
of the blob index at a time, throttled (1,000 blobs/s and 20 MB/s by
default, 0 for no limit), and saves its progress every 10 seconds.
Only one check runs at a time across the server. A check that goes
quiet for five minutes, or whose node restarts, reads as interrupted.
The first 1,000 problems are kept with the account, object, blob id
and what was wrong, and each problem is also logged as
store.data-corruption. An update with action "stop" stops a check. A
finished check can be destroyed. The latest 20 checks are kept.

The new permission sysStoreCheck (695) is in the superuser and admin
grants. A tenant administrator is refused, because a check reads the
whole server. Starting and stopping a check are audited.

Spec: docs/spec/features/store-check.md. Test: system::store_check.
jcoffey-dev merged commit f2bd0cf4b5 into main 2026-11-09 02:10:35 +00:00
jcoffey-dev deleted branch feat/322-verify-store 2026-11-09 02:10:35 +00:00
Sign in to join this conversation.