An administrator can have the server take a snapshot of its store while mail keeps flowing. A backup tool copies the snapshot instead of stopping the server for a cold copy.
What's in it
A RocksDB checkpoint of every column family, taken at one point in time. Blobs kept in the data store (the default) are inside it.
With a separate filesystem blob store, the blob files are copied after the checkpoint. The blob purge is held during the copy (an in-process lock; a purge task that comes due is put off five minutes). So every blob the checkpoint links to is still there to copy.
Hard links are used where possible, so a snapshot is quick and takes almost no space at first. Where links can't be made, files are copied.
Other data stores (SQL, FoundationDB) and other blob stores (S3, Azure, sharded) are refused, and the refusal says why.
Where it goes
Snapshots go under snapshots/ in the RocksDB directory, which is on the data volume. Each is named for its UTC time, e.g. 20261109T013000Z/, and holds data/, blobs/ and snapshot.json.
A snapshot is written as .<name>.partial/ and renamed once complete. latest links to the newest one.
The server keeps the latest two.
The location is fixed; a request can't name a path.
The object is inbuxa:StoreSnapshot under urn:inbuxa:jmap:
create takes a snapshot in the background;
get lists the snapshots on disk, plus the one being taken;
destroy deletes a snapshot's directory;
only one is taken at a time, and a snapshot can't be changed.
A partial snapshot left by a stop is removed at the next start.
Permission: the new sysStoreSnapshot (696) is in the superuser and admin grants. Tenant administrators are refused. Taking and destroying a snapshot are audited.
Also changed: the filesystem blob store now writes each blob to a temporary file and renames it into place. It used to rewrite files in place, which would change a snapshot's hard-linked copy, and a stop mid-write left half a blob under its id.
Spec:docs/spec/features/store-snapshot.md, plus a "Not a rebuild" line in SPEC.md. Restore steps are in the spec and in each snapshot.json.
Tests
system::store_snapshot was run twice: with blobs in RocksDB, and with BLOB_STORE=FileSystem.
A RocksDB store opened from the snapshot has the messages and their blobs, and not a message that arrived after the snapshot.
Also covered: the layout and latest, the manifest, keeping the latest two, destroy, one at a time, no destroy while one is being taken, the audit events, and refusals for a user and for a tenant administrator.
Unit tests cover the atomic blob write, the tree copy, the purge gate, snapshot names and the manifest.
The existing blob store tests pass with a filesystem store and with the default store, and store_check_tests still passes.
Local checks: fmt, cargo build -p inbuxa, cargo test --workspace --no-run, store with s3,azure, clippy on the changed files, and the fork checks.
Not in this PR
the console page;
an inbuxa-cli command a backup tool can call and wait on;
a docs page;
SQLite (VACUUM INTO would do it).
Fixes #323.
An administrator can have the server take a snapshot of its store while mail keeps flowing. A backup tool copies the snapshot instead of stopping the server for a cold copy.
**What's in it**
- A RocksDB checkpoint of every column family, taken at one point in time. Blobs kept in the data store (the default) are inside it.
- With a separate filesystem blob store, the blob files are copied after the checkpoint. The blob purge is held during the copy (an in-process lock; a purge task that comes due is put off five minutes). So every blob the checkpoint links to is still there to copy.
- Hard links are used where possible, so a snapshot is quick and takes almost no space at first. Where links can't be made, files are copied.
- Other data stores (SQL, FoundationDB) and other blob stores (S3, Azure, sharded) are refused, and the refusal says why.
**Where it goes**
- Snapshots go under `snapshots/` in the RocksDB directory, which is on the data volume. Each is named for its UTC time, e.g. `20261109T013000Z/`, and holds `data/`, `blobs/` and `snapshot.json`.
- A snapshot is written as `.<name>.partial/` and renamed once complete. `latest` links to the newest one.
- The server keeps the latest two.
- The location is fixed; a request can't name a path.
**The object** is `inbuxa:StoreSnapshot` under `urn:inbuxa:jmap`:
- create takes a snapshot in the background;
- get lists the snapshots on disk, plus the one being taken;
- destroy deletes a snapshot's directory;
- only one is taken at a time, and a snapshot can't be changed.
A partial snapshot left by a stop is removed at the next start.
**Permission:** the new `sysStoreSnapshot` (696) is in the superuser and admin grants. Tenant administrators are refused. Taking and destroying a snapshot are audited.
**Also changed:** the filesystem blob store now writes each blob to a temporary file and renames it into place. It used to rewrite files in place, which would change a snapshot's hard-linked copy, and a stop mid-write left half a blob under its id.
**Spec:** `docs/spec/features/store-snapshot.md`, plus a "Not a rebuild" line in SPEC.md. Restore steps are in the spec and in each `snapshot.json`.
**Tests**
- `system::store_snapshot` was run twice: with blobs in RocksDB, and with `BLOB_STORE=FileSystem`.
- A RocksDB store opened from the snapshot has the messages and their blobs, and not a message that arrived after the snapshot.
- Also covered: the layout and `latest`, the manifest, keeping the latest two, destroy, one at a time, no destroy while one is being taken, the audit events, and refusals for a user and for a tenant administrator.
- Unit tests cover the atomic blob write, the tree copy, the purge gate, snapshot names and the manifest.
- The existing blob store tests pass with a filesystem store and with the default store, and `store_check_tests` still passes.
**Local checks:** fmt, `cargo build -p inbuxa`, `cargo test --workspace --no-run`, store with `s3,azure`, clippy on the changed files, and the fork checks.
**Not in this PR**
- the console page;
- an `inbuxa-cli` command a backup tool can call and wait on;
- a docs page;
- SQLite (`VACUUM INTO` would do it).
An administrator can now have the server take a snapshot of its store
while mail keeps flowing. A backup tool can then copy the snapshot
instead of stopping the server for a cold copy.
The snapshot is a RocksDB checkpoint of every column family, taken at
one point in time. Blobs kept in the data store are inside it. With a
separate filesystem blob store, the blob files are copied after the
checkpoint. The blob purge is held during the copy, so every blob the
checkpoint links to is still there to copy. Both the checkpoint and the
blob copy use hard links where they can and copy where they can't.
Other data stores (SQL, FoundationDB) and other blob stores (S3, Azure,
sharded) are refused, with the reason.
Snapshots go under snapshots/ in the RocksDB directory, named for their
UTC time. A snapshot is written as a .partial directory and renamed
once it is complete. A latest link points at the newest one, and each
snapshot has a snapshot.json that says what it holds and how to restore
it. The server keeps the latest two. The location is fixed, so a
request can't name a path.
It is an inbuxa:StoreSnapshot under urn:inbuxa:jmap. Creating one takes
a snapshot in the background. get lists the snapshots on disk plus the
one being taken, and destroy deletes a snapshot's directory. Only one
snapshot is taken at a time, and a snapshot can't be changed. A partial
snapshot left by a stop is removed at the next start. The new
permission sysStoreSnapshot (696) is in the superuser and admin grants.
Tenant administrators are refused. Taking and destroying a snapshot are
audited.
The filesystem blob store now writes each blob to a temporary file and
renames it into place. Before, it rewrote files in place. That would
have changed a snapshot's hard-linked copy, and a stop mid-write left
half a blob under its id.
Spec: docs/spec/features/store-snapshot.md. Test: system::store_snapshot,
run with blobs in RocksDB and with BLOB_STORE=FileSystem.
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Fixes #323.
An administrator can have the server take a snapshot of its store while mail keeps flowing. A backup tool copies the snapshot instead of stopping the server for a cold copy.
What's in it
Where it goes
snapshots/in the RocksDB directory, which is on the data volume. Each is named for its UTC time, e.g.20261109T013000Z/, and holdsdata/,blobs/andsnapshot.json..<name>.partial/and renamed once complete.latestlinks to the newest one.The object is
inbuxa:StoreSnapshotunderurn:inbuxa:jmap:A partial snapshot left by a stop is removed at the next start.
Permission: the new
sysStoreSnapshot(696) is in the superuser and admin grants. Tenant administrators are refused. Taking and destroying a snapshot are audited.Also changed: the filesystem blob store now writes each blob to a temporary file and renames it into place. It used to rewrite files in place, which would change a snapshot's hard-linked copy, and a stop mid-write left half a blob under its id.
Spec:
docs/spec/features/store-snapshot.md, plus a "Not a rebuild" line in SPEC.md. Restore steps are in the spec and in eachsnapshot.json.Tests
system::store_snapshotwas run twice: with blobs in RocksDB, and withBLOB_STORE=FileSystem.latest, the manifest, keeping the latest two, destroy, one at a time, no destroy while one is being taken, the audit events, and refusals for a user and for a tenant administrator.store_check_testsstill passes.Local checks: fmt,
cargo build -p inbuxa,cargo test --workspace --no-run, store withs3,azure, clippy on the changed files, and the fork checks.Not in this PR
inbuxa-clicommand a backup tool can call and wait on;VACUUM INTOwould do it).