Changing the blob store: read through and copy what it left behind (#333) #335

Merged
jcoffey-dev merged 1 commits from feat/333-blob-store-change into main 2026-11-09 14:41:38 +00:00
Owner

Fixes #333. Spec: docs/spec/features/blob-store-change.md.

Changing x:BlobStore moved nothing: every blob written under the old setting stayed behind, and a message whose blob was left there failed to read with nothing to say why (forum t/238). Now:

  • Noticed at start-up (BM-1, BM-2). Each node compares where its blob store is (kind and location, never credentials or tuning) with where the store last said it was. When they differ, the first node records an inbuxa:BlobStoreChange (from, to, when, which node) and keeps the old store's settings, and the start-up log warns, naming both places. Going back to an earlier store (A to B, then back to A) marks the A-to-B change reverted (BM-4).
  • Read-through (BM-3). While a change is unresolved, a blob the current store lacks is looked for in the old store. If it's there, it's returned and written to the current store under the same key, and store.blob-read-through (a new event) is logged. Writes and deletes go only to the current store, and nothing is ever deleted from the old one. Stefan's message would have kept working with no repair.
  • Copy (BM-5, BM-6). {"action": "copy"} walks the blob index (256 slices, throttled like the store check) and copies every committed blob something holds, plus the spam classifier's two fixed-name files, from the old store as stored. Blobs held by a document, a queued message or a kept deleted item are checked against their hash first, and a damaged one isn't copied. The copy ends copied (resolved: the old store's settings are dropped and every node stops reading through within a minute) or incomplete, with the first 1,000 problems kept. Stop, interrupted and failed work as in the store check.
  • Dismiss (BM-7). Destroy forgets the change and the settings it kept. Nothing is deleted from either store.
  • A store snapshot is refused while a change is unresolved, since it would miss blobs only the old store has (BM-8).
  • New permission sysBlobStoreChange (697), for administrators only and granted once to existing admin roles. Copy, stop and dismiss are audited, and get never returns the old store's settings (BM-9).

Design choices to look at

  • Read-through changes the blob read path. It only does anything on a miss, and only while a change is unresolved. With no unresolved change, the server uses its blob store unwrapped, as before.
  • The old store's settings, credentials included, are kept in the fork subspace (W) until the change is resolved or dismissed. They're catalogued as the blob-store-change-settings source.
  • A sharded set counts as one place: the set moves blobs between its members itself (ST-16 to ST-22).
  • The copy runs after the change, not before. Reading through covers the time in between, so no copy-ahead mode was needed (listed under "Not yet").

Tests

  • store: read-through and keep, absent in both, writes and deletes touch only the new store, nothing read through once resolved; where a store is (BM-1).
  • inbuxa-features: a move and a move back, recorded once across nodes, the revert, one copy at a time, a quiet copy interrupted, the resolved state.
  • system::blob_store_change, against a real server: the t/238 case end to end, covering read-through, refusals, a slow copy stopped (and not dismissable while it runs), an incomplete copy reporting the damaged and the missing blob with their message, a second copy resolving the change, and dismissal.

Fork checks run locally: name, notice, context, privacy, expr-schema --check, and the tools tests.

Not yet

  • A console page, and a note on the blob store form that existing blobs stay where they are (inbuxa-admin).
  • An inbuxa-cli command, and a docs page.
  • Copying ahead of a change, and cleaning up an old store once its change is resolved.
Fixes #333. Spec: `docs/spec/features/blob-store-change.md`. Changing `x:BlobStore` moved nothing: every blob written under the old setting stayed behind, and a message whose blob was left there failed to read with nothing to say why (forum t/238). Now: - **Noticed at start-up (BM-1, BM-2).** Each node compares where its blob store is (kind and location, never credentials or tuning) with where the store last said it was. When they differ, the first node records an `inbuxa:BlobStoreChange` (from, to, when, which node) and keeps the old store's settings, and the start-up log warns, naming both places. Going back to an earlier store (A to B, then back to A) marks the A-to-B change `reverted` (BM-4). - **Read-through (BM-3).** While a change is unresolved, a blob the current store lacks is looked for in the old store. If it's there, it's returned and written to the current store under the same key, and `store.blob-read-through` (a new event) is logged. Writes and deletes go only to the current store, and nothing is ever deleted from the old one. Stefan's message would have kept working with no repair. - **Copy (BM-5, BM-6).** `{"action": "copy"}` walks the blob index (256 slices, throttled like the store check) and copies every committed blob something holds, plus the spam classifier's two fixed-name files, from the old store as stored. Blobs held by a document, a queued message or a kept deleted item are checked against their hash first, and a damaged one isn't copied. The copy ends `copied` (resolved: the old store's settings are dropped and every node stops reading through within a minute) or `incomplete`, with the first 1,000 problems kept. Stop, interrupted and failed work as in the store check. - **Dismiss (BM-7).** Destroy forgets the change and the settings it kept. Nothing is deleted from either store. - A store snapshot is refused while a change is unresolved, since it would miss blobs only the old store has (BM-8). - New permission `sysBlobStoreChange` (697), for administrators only and granted once to existing admin roles. Copy, stop and dismiss are audited, and `get` never returns the old store's settings (BM-9). ### Design choices to look at - **Read-through changes the blob read path.** It only does anything on a miss, and only while a change is unresolved. With no unresolved change, the server uses its blob store unwrapped, as before. - The old store's settings, credentials included, are kept in the fork subspace (`W`) until the change is resolved or dismissed. They're catalogued as the `blob-store-change-settings` source. - A sharded set counts as one place: the set moves blobs between its members itself (ST-16 to ST-22). - The copy runs after the change, not before. Reading through covers the time in between, so no copy-ahead mode was needed (listed under "Not yet"). ### Tests - `store`: read-through and keep, absent in both, writes and deletes touch only the new store, nothing read through once resolved; where a store is (BM-1). - `inbuxa-features`: a move and a move back, recorded once across nodes, the revert, one copy at a time, a quiet copy interrupted, the resolved state. - `system::blob_store_change`, against a real server: the t/238 case end to end, covering read-through, refusals, a slow copy stopped (and not dismissable while it runs), an incomplete copy reporting the damaged and the missing blob with their message, a second copy resolving the change, and dismissal. Fork checks run locally: name, notice, context, privacy, `expr-schema --check`, and the tools tests. ### Not yet - A console page, and a note on the blob store form that existing blobs stay where they are (inbuxa-admin). - An `inbuxa-cli` command, and a docs page. - Copying ahead of a change, and cleaning up an old store once its change is resolved.
jcoffey-dev added 1 commit 2026-11-09 14:33:47 +00:00
Changing the blob store: read through and copy what it left behind (#333)
ci / fork-checks (pull_request) Skipped
ci / build (pull_request) Skipped
github/ci (branch) GitHub Actions
ci / github (pull_request) Successful in 7m46s
6b382ca65f
A node that starts with x:BlobStore somewhere else records an
inbuxa:BlobStoreChange and keeps the old store's settings. Until the
change is resolved, a blob the new store lacks is read from the old one
and kept; an administrator has one node copy the rest, checked by hash,
with what couldn't be copied reported. Snapshots are refused meanwhile.
New permission sysBlobStoreChange and event store.blob-read-through.

Spec: docs/spec/features/blob-store-change.md
jcoffey-dev merged commit 5a0034eee7 into main 2026-11-09 14:41:38 +00:00
jcoffey-dev deleted branch feat/333-blob-store-change 2026-11-09 14:41:39 +00:00
Sign in to join this conversation.