Online snapshot of the store, without stopping the server #323

Closed
opened 2026-11-08 23:06:05 +00:00 by jcoffey-dev · 2 comments
Owner

From the forum: https://community.coffeylabs.org/t/238 (Stefan), the second, less urgent idea. Their backup stops inbuxa, copies the store cold and starts it again. That works, but it's a short outage each time.

Wanted: inbuxa makes a consistent local snapshot of its own store, and nothing more. Encryption, off-site copies and retention stay with the backup tool.

  • RocksDB: a checkpoint (rocksdb::checkpoint::Checkpoint, in the rocksdb 0.25 crate we use) is consistent and nearly instant, using hard links on the same filesystem.
  • A separate blob store (fs, S3, ...): snapshot the data store first, then copy blobs, and hold blob purging (purge_blobs, crates/store/src/write/blob.rs) until the copy is done. Blobs are written before anything links to them, so every blob the snapshot points at then exists.
  • SQL and other data stores: probably out of scope; their own tools do this. Say so in the docs.

Shape: an admin action (console or inbuxa-cli) that writes to a target directory on the server's machine, with the snapshot's time and what it holds. Restoring is a cold copy back while the server is stopped, and the docs should say how.

The existing --export needs the server stopped too, so it doesn't cover this.

Lower priority than the verify-store issue filed alongside this one.

From the forum: https://community.coffeylabs.org/t/238 (Stefan), the second, less urgent idea. Their backup stops inbuxa, copies the store cold and starts it again. That works, but it's a short outage each time. Wanted: inbuxa makes a consistent local snapshot of its own store, and nothing more. Encryption, off-site copies and retention stay with the backup tool. - **RocksDB:** a checkpoint (`rocksdb::checkpoint::Checkpoint`, in the rocksdb 0.25 crate we use) is consistent and nearly instant, using hard links on the same filesystem. - **A separate blob store (fs, S3, ...):** snapshot the data store first, then copy blobs, and hold blob purging (`purge_blobs`, `crates/store/src/write/blob.rs`) until the copy is done. Blobs are written before anything links to them, so every blob the snapshot points at then exists. - **SQL and other data stores:** probably out of scope; their own tools do this. Say so in the docs. Shape: an admin action (console or inbuxa-cli) that writes to a target directory on the server's machine, with the snapshot's time and what it holds. Restoring is a cold copy back while the server is stopped, and the docs should say how. The existing `--export` needs the server stopped too, so it doesn't cover this. Lower priority than the verify-store issue filed alongside this one.
jcoffey-dev added the
priority
p3
enhancement
labels 2026-11-08 23:06:05 +00:00
jcoffey-dev self-assigned this 2026-11-08 23:06:14 +00:00
Author
Owner

Fixed in #328. The server can now take a snapshot of its store without stopping, as an inbuxa:StoreSnapshot. It is a RocksDB checkpoint, plus a copy of a filesystem blob store's files with the blob purge held during the copy.

Snapshots go under snapshots/ in the RocksDB directory, named for their UTC time, with a latest link and a snapshot.json that says how to restore the snapshot. The latest two are kept. Other data stores (SQL, FoundationDB) and other blob stores (S3, Azure, sharded) are refused, with the reason. It needs the new sysStoreSnapshot permission.

The filesystem blob store also now writes each blob to a temporary file and renames it into place, instead of rewriting it in place.

Spec: docs/spec/features/store-snapshot.md. It ships in the next release candidate. The console page, an inbuxa-cli command for backup tools and a docs page come next.

Fixed in #328. The server can now take a snapshot of its store without stopping, as an `inbuxa:StoreSnapshot`. It is a RocksDB checkpoint, plus a copy of a filesystem blob store's files with the blob purge held during the copy. Snapshots go under `snapshots/` in the RocksDB directory, named for their UTC time, with a `latest` link and a `snapshot.json` that says how to restore the snapshot. The latest two are kept. Other data stores (SQL, FoundationDB) and other blob stores (S3, Azure, sharded) are refused, with the reason. It needs the new `sysStoreSnapshot` permission. The filesystem blob store also now writes each blob to a temporary file and renames it into place, instead of rewriting it in place. Spec: `docs/spec/features/store-snapshot.md`. It ships in the next release candidate. The console page, an `inbuxa-cli` command for backup tools and a docs page come next.
Author
Owner

Backported to the 2026 series in #340 (merged into release/2026 as 7396c23f). It ships in the next 2026.x release after v2026.11.9. On 27.1 it is in v27.1-rc.16.

Backported to the 2026 series in #340 (merged into release/2026 as 7396c23f). It ships in the next 2026.x release after v2026.11.9. On 27.1 it is in v27.1-rc.16.
Sign in to join this conversation.
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: inbuxa/inbuxa-server#323