Spec: cleaning up the old store after a blob store change #357

Merged
jcoffey-dev merged 2 commits from spec/old-store-cleanup into main 2026-11-10 17:06:59 +00:00
3 changed files with 213 additions and 1 deletions

No files matched your search

+1 -1
View File
@@ -390,7 +390,7 @@ Not a rebuild: the **store check** is INBUXA's own design (`features/store-check
Not a rebuild: the **store snapshot** is INBUXA's own design (`features/store-snapshot.md`, inbuxa-server #323, building). An administrator has the server take a consistent copy of its RocksDB store, and of a filesystem blob store's files, into `snapshots/` in the store's directory while mail flows, for a backup tool to copy; the latest two are kept: `inbuxa:StoreSnapshot` (`crates/jmap/src/inbuxa/store_snapshot.rs`, `crates/services/src/inbuxa_store_snapshot.rs`, `crates/store/src/write/snapshot.rs`, `crates/store/src/backend/rocksdb/checkpoint.rs`), and the `sysStoreSnapshot` permission.
Not a rebuild: **changing the blob store** is INBUXA's own design (`features/blob-store-change.md`, inbuxa-server #333, building). When a node starts with the blob store setting pointing somewhere else, it records the change and keeps the old store's settings; until the change is resolved, a blob the new store lacks is read from the old one and kept, and an administrator has one node copy the rest over, checked by hash, with what couldn't be copied reported: `inbuxa:BlobStoreChange` (`crates/jmap/src/inbuxa/blob_store_change.rs`, `crates/features/src/blob_move/`, `crates/services/src/inbuxa_blob_move.rs`, `crates/common/src/manager/blob_moves.rs`, `crates/store/src/write/blob_move.rs`), the `store.blob-read-through` event, and the `sysBlobStoreChange` permission. Planning a change and copying ahead of it (`features/blob-store-copy-ahead.md`, inbuxa-server #341, building): `inbuxa:BlobStoreChange/set` creates a planned change with the new store's settings, copied ahead and applied to `x:BlobStore`; deletes follow to it until the switch.
Not a rebuild: **changing the blob store** is INBUXA's own design (`features/blob-store-change.md`, inbuxa-server #333, building). When a node starts with the blob store setting pointing somewhere else, it records the change and keeps the old store's settings; until the change is resolved, a blob the new store lacks is read from the old one and kept, and an administrator has one node copy the rest over, checked by hash, with what couldn't be copied reported: `inbuxa:BlobStoreChange` (`crates/jmap/src/inbuxa/blob_store_change.rs`, `crates/features/src/blob_move/`, `crates/services/src/inbuxa_blob_move.rs`, `crates/common/src/manager/blob_moves.rs`, `crates/store/src/write/blob_move.rs`), the `store.blob-read-through` event, and the `sysBlobStoreChange` permission. Planning a change and copying ahead of it (`features/blob-store-copy-ahead.md`, inbuxa-server #341, building): `inbuxa:BlobStoreChange/set` creates a planned change with the new store's settings, copied ahead and applied to `x:BlobStore`; deletes follow to it until the switch. Cleaning up the old store once a change is resolved (`features/blob-store-cleanup.md`, inbuxa-server #342, approved): `action: "cleanUp"` deletes from the old store each blob checked in the new one, after a preview and a confirmation.
Not a rebuild: **the store layout, changed in steps** is INBUXA's own design (`features/store-layout.md`, approved 2026-11-08). A store carries its layout version (`Vs` in `_`); at boot one node, holding a lease, runs each step from the stored version up and marks the store after each. Step 1 → 2 gives SQL rate-limit counters their own expiry column; step 2 → 3 moves the audit log, journal, holds and undelete notes into subspaces of their own. A step stops rollback, so it ships at a series.
+2
View File
@@ -160,3 +160,5 @@ or `fixed`, for the first thing found holding the blob.
- An `inbuxa-cli` command.
- Copying ahead of a change (the old store to a new one before the
setting changes): `features/blob-store-copy-ahead.md` (approved, #341).
- Deleting what a resolved change left in the old store:
`features/blob-store-cleanup.md` (approved, #342).
+210
View File
@@ -0,0 +1,210 @@
# Feature spec: cleaning up the old store after a blob store change
Status: approved, 2026-11-10 (inbuxa-server #342, PR #357), with the
answers under [Settled](#settled). Not a rebuild (SPEC.md
§4): inbuxa's own design. It extends `features/blob-store-change.md`
(BM-1 to BM-10) and `features/blob-store-copy-ahead.md` (CA-1 to CA-9),
which it doesn't change except where it says so.
## Provenance
| Source | Used for |
|---|---|
| inbuxa-server #342, after #333 and forum topic 238 | The need: once a change is resolved, every blob is in the new store, but the old one keeps all of them for good (BM-7) |
| The blob store change (`features/blob-store-change.md`, `crates/features/src/blob_move/`, `crates/services/src/inbuxa_blob_move.rs`) | The change record, the old store's settings and where they're dropped today (at `copied`), the copy's walk, hash check, throttle and one-run-at-a-time lock |
| The store snapshot (`features/store-snapshot.md`) | What a snapshot holds, by hard link, and so what space a cleanup can't free while it exists |
| This server's blob dispatch (`crates/store/src/dispatch/blob.rs`) | Deleting a blob's data from a store without touching the blob index |
Upstream's open code has nothing like it, and nothing here comes from
Enterprise code.
## What it is
After a change is resolved, the old store still holds a copy of every
blob that was in it. For a directory or a bucket of its own, the
administrator can delete it. For the data store (`Default`), or a bucket
prefix shared with other things, they can't do that by hand: the blobs
sit among things that must stay.
With this, an administrator has the server delete from the old store the
blobs it has checked are in the new one. It's a separate, explicit step,
taken when the administrator is ready, never as part of the copy. It can
be previewed first, and it can be run again.
The hard part isn't the deleting, it's what still depends on the old
store after the change is resolved: backups taken before the change, a
node that hasn't restarted, a snapshot. The requirements below are mostly
about those.
## Requirements
- **OC-1.** **The old store is kept.** At `copied` (BM-6), a change keeps
its old store's settings instead of dropping them. Reading through
still stops then (BM-3, BM-10); the settings are kept only for a
cleanup. They go when the cleanup ends `done` (OC-6) or the change is
dismissed (BM-7). Changes resolved before this spec have no settings
left, and their cleanup is refused (OC-3).
- **OC-2.** **Each node says where its blob store is.** When a node reads
the changes (BM-10, every minute), it also records where its own blob
store is (BM-1) and the time, under its node id. A node not heard from
for ten minutes is left out of OC-3's check.
- **OC-3.** **Starting a cleanup.** An update with `action: "cleanUp"` on
a change starts a cleanup of its old store on the node that takes it.
It's refused when:
- the change isn't `copied`. A `reverted` change has no old store to
clean: it's the current one again (BM-4);
- its old store's settings aren't kept (OC-1);
- its `to` isn't where the current blob store is, or its `from` is.
After A to B, then B to A, cleaning A would delete the blob store
itself;
- a node recorded under OC-2 has a blob store anywhere but `to`: it may
still write to or read from the old store (BM-10);
- the two stores overlap: one is a sharded set with the other as a
member, or a probe written under a reserved key in the new store can
be read from the old one. Deleting from the old store would then
delete from the new;
- a planned change's `to` is the old store's place (CA-1): its copy
ahead would write what the cleanup deletes;
- any copy, copy ahead or cleanup runs: one at a time across the
server, as BM-5;
- it isn't confirmed (OC-8).
The old store must open, as it must for a copy; an error opening it is
the refusal.
- **OC-4.** **What it deletes.** It goes through the blob index as a copy
does (BM-5): the same 256 slices, the same blobs (documents, queued
messages, kept deleted items, unexpired links, the spam classifier's
two files). For each blob:
- not in the old store: counted as `notThere`;
- not in the new store: kept in the old one, and a `missing` problem;
- a blob named by the hash of its content whose copy in the new store
doesn't hash to its name: kept, and a `damaged` problem;
- any other blob whose copy in the new store isn't byte for byte the
old one's: kept, and a `different` problem;
- an error from either store: kept, and an `unreadable` problem;
- otherwise deleted from the old store: `deleted`, with its size added
to `bytesFreed`.
Only the blob's data is deleted from the old store, through that
store's own delete. The blob index and everything that links to the
blob are the data store's and stay as they are. Each problem is
logged as `store.data-corruption`, since it's a blob the new store
should have and doesn't. Throttled as BM-5, with `blobsPerSecond` and
`bytesPerSecond` alongside the action.
- **OC-5.** **A preview.** `dryRun: true` alongside the action runs the
same walk and checks and deletes nothing. It counts what would be
deleted, the bytes, and the problems, and its result is kept as the
cleanup's (with `dryRun` set) until the next run. A preview needs no
confirmation (OC-8).
- **OC-6.** **How it ends**, as a copy does (BM-6):
- `done`: no problems. Everything the server holds was checked in the
new store and deleted from the old. The old store's settings go
(OC-1), and the change can't be cleaned again;
- `incomplete`: some blobs were kept (OC-4). The first 1,000 problems
are kept. It can be run again; blobs already deleted count as
`notThere`;
- `stopped`, by `action: "stop"`; `failed`, with the error;
`interrupted`, when its node restarted or went quiet for five
minutes. Each can be run again.
A preview ends the same way and changes nothing about the change.
- **OC-7.** **What it doesn't delete.** Only blobs the blob index names
are looked for. Anything else in the old store stays: blobs nothing
held when the change was made, a filesystem store's empty directories,
other things sharing a bucket or prefix. `done` means every blob the
server knows about is out of the old store, not that the old store is
empty. For a directory or bucket of its own, the docs say the
administrator can delete what's left once the cleanup is `done`.
- **OC-8.** **Backups taken before the change.** A backup of the data
store from before the change, restored, has the old blob store setting
and reads its blobs from the old store, which the cleanup empties.
The server can't see a backup tool's backups, so a cleanup that
deletes needs `confirm` set to the change's id alongside the action,
and the refusal without it says: backups of the data store taken
before the change was resolved need the old store's blobs to restore
in full; keep the old store until those backups have aged out. A
restored server whose blobs are gone can be pointed at the new store
instead: a change from the old store to the new one is recorded
(BM-2), and every blob that was still held when the cleanup ran is
there. Blobs deleted from the new store since then aren't.
- **OC-9.** **Snapshots.** A cleanup and a snapshot (store-snapshot spec)
can run at once: the snapshot takes the current store, and the cleanup
deletes only from the old one. A snapshot taken before the change
holds the old store's blobs by hard link (a filesystem store) or in its
checkpoint (the data store), so their space comes back only when that
snapshot is destroyed. The cleanup's result says how many snapshots
older than the change are on disk.
- **OC-10.** **Freed space** depends on the store: RocksDB frees it as it
compacts; PostgreSQL and MySQL after a vacuum or optimize; a versioned
bucket keeps old versions until its own lifecycle rules remove them.
The docs say so; the server doesn't vacuum or compact for it.
- **OC-11.** **Forgetting.** A change waiting on a cleanup counts among
the 20 kept (`MAX_CHANGES`), and forgetting it forgets its old store's
settings. Changes whose settings are gone are forgotten first. A change
whose cleanup runs isn't forgotten, and can't be dismissed (as BM-7).
- **OC-12.** Permissions as BM-9: `sysBlobStoreChange`. Starting,
previewing and stopping a cleanup are audited, and so is each run's
end with its counts.
## The object
`inbuxa:BlobStoreChange` gains:
| Property | Type | |
|---|---|---|
| `oldStoreKept` | Boolean | Whether the old store's settings are kept for a cleanup (OC-1) |
| `cleanup` | Object | The latest cleanup or preview, null before the first |
| `action` | String | Write-only: also `cleanUp` |
| `dryRun` | Boolean | Write-only, with `cleanUp` (OC-5) |
| `confirm` | Id | Write-only, with `cleanUp`: the change's own id (OC-8) |
`cleanup` is `{status, dryRun, startedAt, startedBy, finishedAt, node,
blobsPerSecond, bytesPerSecond, progress, blobsChecked, deleted,
bytesFreed, notThere, missing, damaged, different, unreadable, problems,
snapshotsBefore, error}`, with `status` one of `running`, `done`,
`incomplete`, `stopped`, `failed`, `interrupted`. A problem is as BM-5's,
with `different` as a kind.
## Tests
- Model (`crates/features/src/blob_move/`): settings kept at `copied`
and dropped at a cleanup's `done` or a dismiss; each refusal in OC-3;
forgetting order (OC-11); a preview leaving the change as it was.
- Store level, with two filesystem stores: a delete from the old one
leaves the new one and the blob index untouched; the overlap probe
finds two stores on the same directory and depth through different
settings.
- `system::blob_store_change`, against a real server: messages
delivered on a filesystem store, the setting changed to the data
store, copied and resolved. A preview counts every blob and deletes
none; `cleanUp` without `confirm` is refused; with it, the old
directory loses those blobs and every message still reads. A blob
removed from the new store by hand, and one altered there, are kept
in the old store and reported, and the run ends `incomplete`; put
back, a second run ends `done` and the old settings are gone. A node
recorded on the old store refuses the cleanup. Over JMAP: stop,
destroy while running, a tenant administrator's refusal.
## Settled
Approved 2026-11-10 with every recommendation:
1. **Guarding backups:** the confirmation naming the change (OC-8), and
no wait after resolution. The refusal and the console say plainly
what deleting costs; the preview and a store check let the
administrator look first.
2. **Checking the new store:** every blob is read back and checked
before its old copy is deleted (OC-4), at the cost of a read per blob,
which on S3 or Azure is a request and egress each.
3. **Each node's place** is recorded with the minute poll (OC-2).
4. **A dismissed plan's target** isn't cleaned by this spec (CA-8 is
unchanged).
## Not yet
- The console: **Clean up the old store…** on a resolved change, with
the preview's numbers, the backup warning and the confirmation
(inbuxa-admin), and `inbuxa-cli store` commands to preview, clean up
and stop.
- Docs: a "Cleaning up the old store" section in docs.inbuxa.org's
`run/store.md`, with OC-7, OC-8 and OC-10.