Changing x:BlobStore strands existing blobs without a warning #333

Closed
opened 2026-11-09 13:48:31 +00:00 by jcoffey-dev · 2 comments
Owner

Reported on the forum (t/238). A server on RocksDB with x:BlobStore = Default was switched to a FileSystem blob store for a test, then switched back to Default a minute later. One message arrived by SMTP in between. Its metadata went to RocksDB as always, but its blob went to the filesystem store. After the rollback the server only reads blobs from RocksDB, so that message's body returned 404. Nothing deleted the blob; it was still in the filesystem store's directory.

Changing the blob store today doesn't move existing blobs, check for them, or warn that it won't. Any switch strands whatever was written to the old store, in either direction.

What I'd like:

  • Warn on change. When x:BlobStore changes on a server whose data store already holds blobs, the console and the JMAP set response say that existing blobs stay in the old store and won't be readable from the new one.
  • Copy task. An admin task that copies blobs from one blob store to another (source given as a store config), skips ones already present, verifies each copy by its content hash, and reports counts and failures. Run before switching, and again after a rollback to pick up anything written in between.
  • Docs. The blob store page explains the above and the order to do it in.

The store check from #322 already reports these blobs as missing, so it can confirm a switch went cleanly.

Reported on the forum (t/238). A server on RocksDB with `x:BlobStore` = `Default` was switched to a `FileSystem` blob store for a test, then switched back to `Default` a minute later. One message arrived by SMTP in between. Its metadata went to RocksDB as always, but its blob went to the filesystem store. After the rollback the server only reads blobs from RocksDB, so that message's body returned 404. Nothing deleted the blob; it was still in the filesystem store's directory. Changing the blob store today doesn't move existing blobs, check for them, or warn that it won't. Any switch strands whatever was written to the old store, in either direction. What I'd like: - **Warn on change.** When `x:BlobStore` changes on a server whose data store already holds blobs, the console and the JMAP set response say that existing blobs stay in the old store and won't be readable from the new one. - **Copy task.** An admin task that copies blobs from one blob store to another (source given as a store config), skips ones already present, verifies each copy by its content hash, and reports counts and failures. Run before switching, and again after a rollback to pick up anything written in between. - **Docs.** The blob store page explains the above and the order to do it in. The store check from #322 already reports these blobs as missing, so it can confirm a switch went cleanly.
jcoffey-dev added the enhancement
priority
p2
labels 2026-11-09 13:48:31 +00:00
jcoffey-dev self-assigned this 2026-11-09 13:48:57 +00:00
Author
Owner

Fixed in #335 (5a0034ee), which ships in the next server rc.

When a server starts with x:BlobStore pointing somewhere new, it now records an inbuxa:BlobStoreChange and keeps the old store readable. Until the change is resolved, a blob the new store lacks is read from the old one and kept. An administrator copies the rest with {"action": "copy"}: content-named blobs are checked by hash, and anything missing or damaged is reported. Destroying a change dismisses it. Snapshots are refused while a change is unresolved. Spec: docs/spec/features/blob-store-change.md.

Still to come: a console page and a note on the blob store form, a CLI command, and docs.

Fixed in #335 (5a0034ee), which ships in the next server rc. When a server starts with `x:BlobStore` pointing somewhere new, it now records an `inbuxa:BlobStoreChange` and keeps the old store readable. Until the change is resolved, a blob the new store lacks is read from the old one and kept. An administrator copies the rest with `{"action": "copy"}`: content-named blobs are checked by hash, and anything missing or damaged is reported. Destroying a change dismisses it. Snapshots are refused while a change is unresolved. Spec: `docs/spec/features/blob-store-change.md`. Still to come: a console page and a note on the blob store form, a CLI command, and docs.
Author
Owner

Backported to the 2026 series in #340 (merged into release/2026 as 7396c23f). It ships in the next 2026.x release after v2026.11.9. On 27.1 it is in v27.1-rc.16.

Backported to the 2026 series in #340 (merged into release/2026 as 7396c23f). It ships in the next 2026.x release after v2026.11.9. On 27.1 it is in v27.1-rc.16.
Sign in to join this conversation.
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: inbuxa/inbuxa-server#333