Spec: copying blobs ahead of a blob store change #354
No files matched your search
+1
-1
@@ -390,7 +390,7 @@ Not a rebuild: the **store check** is INBUXA's own design (`features/store-check
|
||||
|
||||
Not a rebuild: the **store snapshot** is INBUXA's own design (`features/store-snapshot.md`, inbuxa-server #323, building). An administrator has the server take a consistent copy of its RocksDB store, and of a filesystem blob store's files, into `snapshots/` in the store's directory while mail flows, for a backup tool to copy; the latest two are kept: `inbuxa:StoreSnapshot` (`crates/jmap/src/inbuxa/store_snapshot.rs`, `crates/services/src/inbuxa_store_snapshot.rs`, `crates/store/src/write/snapshot.rs`, `crates/store/src/backend/rocksdb/checkpoint.rs`), and the `sysStoreSnapshot` permission.
|
||||
|
||||
Not a rebuild: **changing the blob store** is INBUXA's own design (`features/blob-store-change.md`, inbuxa-server #333, building). When a node starts with the blob store setting pointing somewhere else, it records the change and keeps the old store's settings; until the change is resolved, a blob the new store lacks is read from the old one and kept, and an administrator has one node copy the rest over, checked by hash, with what couldn't be copied reported: `inbuxa:BlobStoreChange` (`crates/jmap/src/inbuxa/blob_store_change.rs`, `crates/features/src/blob_move/`, `crates/services/src/inbuxa_blob_move.rs`, `crates/common/src/manager/blob_moves.rs`, `crates/store/src/write/blob_move.rs`), the `store.blob-read-through` event, and the `sysBlobStoreChange` permission.
|
||||
Not a rebuild: **changing the blob store** is INBUXA's own design (`features/blob-store-change.md`, inbuxa-server #333, building). When a node starts with the blob store setting pointing somewhere else, it records the change and keeps the old store's settings; until the change is resolved, a blob the new store lacks is read from the old one and kept, and an administrator has one node copy the rest over, checked by hash, with what couldn't be copied reported: `inbuxa:BlobStoreChange` (`crates/jmap/src/inbuxa/blob_store_change.rs`, `crates/features/src/blob_move/`, `crates/services/src/inbuxa_blob_move.rs`, `crates/common/src/manager/blob_moves.rs`, `crates/store/src/write/blob_move.rs`), the `store.blob-read-through` event, and the `sysBlobStoreChange` permission. Planning a change and copying ahead of it (`features/blob-store-copy-ahead.md`, inbuxa-server #341, approved 2026-11-10).
|
||||
|
||||
Not a rebuild: **the store layout, changed in steps** is INBUXA's own design (`features/store-layout.md`, approved 2026-11-08). A store carries its layout version (`Vs` in `_`); at boot one node, holding a lease, runs each step from the stored version up and marks the store after each. Step 1 → 2 gives SQL rate-limit counters their own expiry column; step 2 → 3 moves the audit log, journal, holds and undelete notes into subspaces of their own. A step stops rollback, so it ships at a series.
|
||||
|
||||
|
||||
@@ -159,6 +159,4 @@ or `fixed`, for the first thing found holding the blob.
|
||||
that existing blobs stay where they are.
|
||||
- An `inbuxa-cli` command.
|
||||
- Copying ahead of a change (the old store to a new one before the
|
||||
setting changes). Reading through makes it unnecessary for the
|
||||
server's own reads, but a long copy after the change means more
|
||||
reads go to the old store meanwhile.
|
||||
setting changes): `features/blob-store-copy-ahead.md` (approved, #341).
|
||||
@@ -0,0 +1,152 @@
|
||||
# Feature spec: copying blobs ahead of a blob store change
|
||||
|
||||
Status: **approved 2026-11-10** (inbuxa-server #341, PR #354), with the
|
||||
answers under [Settled](#settled). Not a rebuild (SPEC.md
|
||||
§4): inbuxa's own design. It extends `features/blob-store-change.md`
|
||||
(BM-1 to BM-10), which it doesn't change except where it says so.
|
||||
|
||||
## Provenance
|
||||
|
||||
| Source | Used for |
|
||||
|---|---|
|
||||
| inbuxa-server #341, after forum topic 238 | The need: on a large store, the copy after a change takes long, and until it ends many reads go to the old store, each first read slower |
|
||||
| The blob store change (`features/blob-store-change.md`, `crates/features/src/blob_move/`, `crates/store/src/write/blob_move.rs`, `crates/common/src/manager/blob_moves.rs`) | The object, where a store is (BM-1), how the old store's settings are kept (`Location.settings`, an `x:BlobStore` value), the copy and its throttle (BM-5) |
|
||||
|
||||
Upstream's open code has nothing like it, and nothing here comes from
|
||||
Enterprise code.
|
||||
|
||||
## What it is
|
||||
|
||||
Today a blob store change starts when a node starts with the new
|
||||
setting. Everything is then read through from the old store until a copy
|
||||
has moved it all (BM-3, BM-5). For a store of millions of blobs, that
|
||||
copy runs for hours, and every first read of an old blob meanwhile goes
|
||||
to the old store.
|
||||
|
||||
With this, an administrator **plans** the change first: gives the new
|
||||
store's settings, and has the server copy everything from the current
|
||||
store into it while the current store is still the blob store. Nothing
|
||||
reads from or writes to the new store yet, apart from the copy. When the
|
||||
setting then changes to that store, the planned change becomes the
|
||||
change (BM-2), and its copy is a last pass that picks up only what was
|
||||
written in between.
|
||||
|
||||
## Requirements
|
||||
|
||||
- **CA-1.** **Planning.** A create of `inbuxa:BlobStoreChange` with a
|
||||
`target` (an `x:BlobStore` value, as the blob store setting takes it)
|
||||
records a **planned** change: `stage` is `planned`, `from` is where the
|
||||
current blob store is and `to` is where the target is (BM-1). The
|
||||
server keeps the target's settings in the change, as it keeps an old
|
||||
store's (BM-2). `get` never returns them, only `to` (BM-9). The create
|
||||
is refused:
|
||||
- when `to` is where the current blob store is;
|
||||
- when another change is planned: one at a time;
|
||||
- when a change is unresolved (BM-6): copying ahead reads the current
|
||||
store only, so it would miss the blobs only an older store has. The
|
||||
refusal says to copy or dismiss that change first;
|
||||
- when the target can't be opened, or a probe written under a
|
||||
reserved key can't be read back and deleted. The refusal carries the
|
||||
store's error.
|
||||
- **CA-2.** **Copying ahead.** `action: "copy"` on a planned change copies
|
||||
from the current store to the target, exactly as BM-5 copies from an
|
||||
old store to the current one: the same walk, the same blobs, the same
|
||||
hash check, the same throttle and the same one-copy-at-a-time rule
|
||||
across the server (a copy ahead and a copy after a change can't run at
|
||||
once). It ends as BM-6 says, with one difference: `copied` doesn't
|
||||
resolve anything, since nothing is reading through. `stop` stops it,
|
||||
and copying again skips what the target has.
|
||||
- **CA-3.** **Nothing else changes while it's planned.** Writes and reads
|
||||
go to the current store only. A planned change isn't read through
|
||||
(BM-3) and doesn't refuse a snapshot (BM-8). On a cluster, every node
|
||||
reads planned changes with the unresolved ones (BM-10), for CA-4.
|
||||
- **CA-4.** **Deletes follow.** While a change is planned, a blob the
|
||||
server deletes from the current store (the blob cleanup, an expired
|
||||
upload) is also deleted from the target, so the target doesn't keep
|
||||
blobs nothing holds any more. An error from the target is logged with
|
||||
the change's id and doesn't fail the delete. A blob deleted while a
|
||||
copy is reading it can still reach the target; it's a leftover that
|
||||
costs space and nothing else.
|
||||
- **CA-5.** **The switch.** When a node starts with a blob store whose
|
||||
place (BM-1) is a planned change's `to`, and the last recorded place is
|
||||
its `from`, that planned change becomes the change instead of a new one
|
||||
being recorded (BM-2): `stage` becomes `changed`, `changedAt` and
|
||||
`node` are set, and `status` starts again at `pending`. The copy
|
||||
ahead's numbers are kept in `copiedAhead` (below). From then on it's an
|
||||
ordinary change: read through, refusing snapshots, copied and
|
||||
resolved as BM-3 to BM-10 say. A start with any other blob store
|
||||
records a new change as before, and the planned one stays planned.
|
||||
- **CA-6.** **The last pass** is the copy after the switch (BM-5). It
|
||||
still checks every blob, but most are already in the new store
|
||||
(`alreadyThere`), so it transfers only what was written between the
|
||||
copy ahead and the switch. Checking a blob in the new store costs one
|
||||
lookup, so on S3 or Azure the last pass is a request per blob, at the
|
||||
same throttle.
|
||||
- **CA-7.** **Making it the setting.** `action: "apply"` on a planned
|
||||
change whose latest copy ahead ended `copied` writes its target's
|
||||
settings to the blob store setting (`x:BlobStore`), as an update to the
|
||||
setting would, with its secrets, so nobody types the credentials twice
|
||||
and the setting can't name a different place by mistake. It takes
|
||||
effect when the nodes restart, as the blob store setting always does.
|
||||
It's refused for any other status, and once applied, a second `apply`
|
||||
is refused.
|
||||
- **CA-8.** **Dismissing.** Destroying a planned change forgets it and
|
||||
its target's settings. Nothing is deleted from the target; what was
|
||||
copied there stays, and removing it is the administrator's job (as in
|
||||
BM-7). A planned change whose copy runs can't be destroyed; stop the
|
||||
copy first.
|
||||
- **CA-9.** Permissions as BM-9: `sysBlobStoreChange`. Planning, copying
|
||||
ahead, applying and dismissing are audited.
|
||||
|
||||
## The object
|
||||
|
||||
`inbuxa:BlobStoreChange` gains:
|
||||
|
||||
| Property | Type | |
|
||||
|---|---|---|
|
||||
| `stage` | String | `planned` or `changed`. Changes recorded before this spec are `changed` |
|
||||
| `target` | Object | Write-only, on create: the new store's settings (CA-1). Never returned |
|
||||
| `copiedAhead` | Object | `{startedAt, finishedAt, status, copied, alreadyThere, bytesCopied, problems}` of the last copy ahead, kept at the switch (CA-5); null for a change that wasn't planned |
|
||||
| `appliedAt`, `appliedBy` | UTCDate, String | When and by whom `apply` wrote the setting (CA-7) |
|
||||
| `action` | String | Write-only: `copy`, `stop` or, on a planned change, `apply` |
|
||||
|
||||
While `stage` is `planned`, `status` and the copy properties describe the
|
||||
copy ahead, and `changedAt` and `node` are null.
|
||||
|
||||
## Tests
|
||||
|
||||
- Model (`crates/features/src/blob_move/`): planning and its refusals;
|
||||
the switch turning a planned change into the change, keeping
|
||||
`copiedAhead`; a start with another store leaving it planned; `apply`
|
||||
only after `copied`, and once.
|
||||
- Store level (`crates/store/src/write/blob_move.rs`): with two
|
||||
filesystem stores, a delete from the current store also removes the
|
||||
key from a planned target; an error from the target doesn't fail the
|
||||
delete.
|
||||
- `system::blob_store_change`, against a real server: messages
|
||||
delivered, a filesystem target planned and copied ahead, more mail
|
||||
delivered, `apply`, the server rebuilt on the new setting. The
|
||||
planned change is now the change; reading through returns the late
|
||||
messages; the last pass copies only those (the rest `alreadyThere`)
|
||||
and resolves it. Over JMAP: each refusal in CA-1 and CA-7, destroy
|
||||
while copying, a tenant administrator's refusal.
|
||||
|
||||
## Settled
|
||||
|
||||
Approved 2026-11-10 with every recommendation:
|
||||
|
||||
1. **Deletes follow** the current store to the planned target (CA-4).
|
||||
2. **Writes don't follow.** The target stays off the mail path; the last
|
||||
pass and reading through cover the gap.
|
||||
3. **`apply`** writes the target's settings to the blob store setting
|
||||
(CA-7).
|
||||
4. **Planning is refused** while another change is unresolved (CA-1).
|
||||
|
||||
## Not yet
|
||||
|
||||
- The console: a **Plan a change** form on the Blob Store Changes page
|
||||
with the blob store form's fields, **Copy ahead** and **Make it the
|
||||
blob store** on a planned change (inbuxa-admin), and the matching
|
||||
`inbuxa-cli store` commands.
|
||||
- Docs: a "Planning a change" section in docs.inbuxa.org's
|
||||
`run/store.md`.
|
||||
Reference in new issue
Block a user