Store snapshot: a consistent copy of the store without stopping the server (#323) #328

Merged
jcoffey-dev merged 2 commits from feat/323-store-snapshot into main 2026-11-09 06:13:57 +00:00
2 Commits
Author SHA1 Message Date
jcoffey-dev 7dd577ed71 Merge remote-tracking branch 'origin/main' into feat/323-store-snapshot
ci / fork-checks (pull_request) Skipped
ci / build (pull_request) Skipped
github/ci (branch) GitHub Actions
ci / github (pull_request) Successful in 8m25s
# Conflicts:
#	docs/spec/SPEC.md
2026-11-08 23:05:19 -07:00
jcoffey-dev 359ae39396 Store snapshot: a consistent copy of the store without stopping the server (#323)
ci / fork-checks (pull_request) Skipped
ci / build (pull_request) Skipped
github/ci (branch) GitHub Actions
ci / github (pull_request) Successful in 7m28s
An administrator can now have the server take a snapshot of its store
while mail keeps flowing. A backup tool can then copy the snapshot
instead of stopping the server for a cold copy.

The snapshot is a RocksDB checkpoint of every column family, taken at
one point in time. Blobs kept in the data store are inside it. With a
separate filesystem blob store, the blob files are copied after the
checkpoint. The blob purge is held during the copy, so every blob the
checkpoint links to is still there to copy. Both the checkpoint and the
blob copy use hard links where they can and copy where they can't.
Other data stores (SQL, FoundationDB) and other blob stores (S3, Azure,
sharded) are refused, with the reason.

Snapshots go under snapshots/ in the RocksDB directory, named for their
UTC time. A snapshot is written as a .partial directory and renamed
once it is complete. A latest link points at the newest one, and each
snapshot has a snapshot.json that says what it holds and how to restore
it. The server keeps the latest two. The location is fixed, so a
request can't name a path.

It is an inbuxa:StoreSnapshot under urn:inbuxa:jmap. Creating one takes
a snapshot in the background. get lists the snapshots on disk plus the
one being taken, and destroy deletes a snapshot's directory. Only one
snapshot is taken at a time, and a snapshot can't be changed. A partial
snapshot left by a stop is removed at the next start. The new
permission sysStoreSnapshot (696) is in the superuser and admin grants.
Tenant administrators are refused. Taking and destroying a snapshot are
audited.

The filesystem blob store now writes each blob to a temporary file and
renames it into place. Before, it rewrote files in place. That would
have changed a snapshot's hard-linked copy, and a stop mid-write left
half a blob under its id.

Spec: docs/spec/features/store-snapshot.md. Test: system::store_snapshot,
run with blobs in RocksDB and with BLOB_STORE=FileSystem.
2026-11-08 22:55:45 -07:00