verify-store: check that every stored blob is there and intact #322

Closed
opened 2026-11-08 23:06:05 +00:00 by jcoffey-dev · 2 comments
Owner

From the forum: https://community.coffeylabs.org/t/238 (Stefan). They found a mail whose metadata was present but whose blob was missing from the store. The server ran normally, health checks were green, and backups copied the broken state.

Wanted: a check that reports, and never repairs, the logical consistency between the data store and the blob store.

  • Quick: every blob a document links to (BlobLink::Document, crates/store/src/write/blob.rs) exists in the blob store.
  • Full: every such blob also reads back and hashes to its id. BlobHash is BLAKE3 of the content (crates/types/src/blob_hash.rs), so this proves the content intact, not only present.
  • Report: for each problem, the account, the object (mail, file, sieve script, and so on), its id and the blob id, with missing and damaged told apart.

Shape: an admin task that runs inside the server, started from the console or inbuxa-cli, not an offline command. RocksDB lets only one process open the store, so an offline check means stopping the server, and a cluster on Postgres plus a separate blob store can't be checked from one machine. It should run throttled, so a full check on a big store doesn't hurt mail. The result should be readable afterwards: an object or report, plus an event per problem.

Before or alongside this: find out how the blob in that report went missing. Questions are on the forum thread (version, blob store, how the message arrived, whether the store was restored from a live copy). If it is a purge race or an import gap, that is a bug of its own.

Spec first: a section in docs/spec, with test ids.

From the forum: https://community.coffeylabs.org/t/238 (Stefan). They found a mail whose metadata was present but whose blob was missing from the store. The server ran normally, health checks were green, and backups copied the broken state. Wanted: a check that reports, and never repairs, the logical consistency between the data store and the blob store. - **Quick:** every blob a document links to (`BlobLink::Document`, `crates/store/src/write/blob.rs`) exists in the blob store. - **Full:** every such blob also reads back and hashes to its id. `BlobHash` is BLAKE3 of the content (`crates/types/src/blob_hash.rs`), so this proves the content intact, not only present. - **Report:** for each problem, the account, the object (mail, file, sieve script, and so on), its id and the blob id, with missing and damaged told apart. Shape: an admin task that runs inside the server, started from the console or inbuxa-cli, not an offline command. RocksDB lets only one process open the store, so an offline check means stopping the server, and a cluster on Postgres plus a separate blob store can't be checked from one machine. It should run throttled, so a full check on a big store doesn't hurt mail. The result should be readable afterwards: an object or report, plus an event per problem. Before or alongside this: find out how the blob in that report went missing. Questions are on the forum thread (version, blob store, how the message arrived, whether the store was restored from a live copy). If it is a purge race or an import gap, that is a bug of its own. Spec first: a section in docs/spec, with test ids.
jcoffey-dev added the enhancement
priority
p2
labels 2026-11-08 23:06:05 +00:00
jcoffey-dev self-assigned this 2026-11-08 23:06:14 +00:00
Author
Owner

Fixed in #324. The server can now check its blobs, as an inbuxa:StoreCheck. In quick mode it checks that every linked blob is in the blob store. In full mode it also reads each one back and checks it against its hash.

Mail is also checked from the message side. A message whose metadata names a blob that nothing links to any more is reported as unlinked, along with whether the blob is still there. That is the case from forum topic 238. The check is read-only and throttled, and only one runs at a time. It needs the new sysStoreCheck permission.

Spec: docs/spec/features/store-check.md. It ships in the next release candidate. The console page, an inbuxa-cli command and a docs page come next; #323 is separate.

Fixed in #324. The server can now check its blobs, as an `inbuxa:StoreCheck`. In quick mode it checks that every linked blob is in the blob store. In full mode it also reads each one back and checks it against its hash. Mail is also checked from the message side. A message whose metadata names a blob that nothing links to any more is reported as `unlinked`, along with whether the blob is still there. That is the case from forum topic 238. The check is read-only and throttled, and only one runs at a time. It needs the new `sysStoreCheck` permission. Spec: `docs/spec/features/store-check.md`. It ships in the next release candidate. The console page, an `inbuxa-cli` command and a docs page come next; #323 is separate.
Author
Owner

Backported to the 2026 series in #340 (merged into release/2026 as 7396c23f). It ships in the next 2026.x release after v2026.11.9. On 27.1 it is in v27.1-rc.16.

Backported to the 2026 series in #340 (merged into release/2026 as 7396c23f). It ships in the next 2026.x release after v2026.11.9. On 27.1 it is in v27.1-rc.16.
Sign in to join this conversation.
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: inbuxa/inbuxa-server#322