Backups for Kubernetes: export to and import from S3, store checks as a command (K-C11) #491

Merged
jcoffey-dev merged 1 commits from feat/k8s-backups into main 2026-11-19 21:04:02 +00:00
Owner

For the chart's scheduled backup and store checks (inbuxa/inbuxa-charts#11), and three import bugs the lab's restore test found.

Added

  • --export s3://bucket/prefix: streams the export into multipart uploads, in a new folder named for the time, with export.json written last (an export without it didn't finish). Nothing the size of the store touches local disk; one file can be nearly 3 TiB.
  • --import s3://bucket/prefix/<name>, or s3://bucket/prefix for the newest finished export.
  • The bucket from INBUXA_EXPORT_S3_ENDPOINT (S3-compatible, path-style; AWS without it), INBUXA_EXPORT_S3_REGION, and keys from INBUXA_EXPORT_S3_ACCESS_KEY_FILE / _SECRET_KEY_FILE (AWS defaults without them).
  • --store-check quick|full: a store check to its end, then exit 0 (nothing wrong), 3 (problems), 2 (another running), 1 (failed). Saved with the console's checks.

Fixed

  • A live export could name blobs it didn't have: areas went out in a set's order, so blobs could go first. Blobs go last now.
  • An import into a fresh store refused to run, from a directory too. Every start writes the fork's subspace (blob store location, permission grants, log files' setting), the import's own start included, and the import took that for data. Since #333: 27.1-rc.16 and 2026.11.9.1. The store test now imports over a start-up record, and fails without the fix.
  • An import put the blobs into the data store even when the imported settings name S3, so the restored server found none of them. Blobs are imported last, into the blob store the imported settings name.
  • The S3 export and import could hang for good on a pod limited to one CPU (one runtime worker). A wait hands its worker over first.

Tested: inbuxa-k8s-lab backup-test.sh on kind (Postgres, Garage, NATS, five pods): mail, the chart's backup CronJob, the database dropped and blob bucket emptied, the import, every mailbox's count and message hashes the same after, quick and full store checks clean. Store tests on SQLite and RocksDB; unit tests; notice and name checks.

For the chart's scheduled backup and store checks (inbuxa/inbuxa-charts#11), and three import bugs the lab's restore test found. **Added** - `--export s3://bucket/prefix`: streams the export into multipart uploads, in a new folder named for the time, with `export.json` written last (an export without it didn't finish). Nothing the size of the store touches local disk; one file can be nearly 3 TiB. - `--import s3://bucket/prefix/<name>`, or `s3://bucket/prefix` for the newest finished export. - The bucket from `INBUXA_EXPORT_S3_ENDPOINT` (S3-compatible, path-style; AWS without it), `INBUXA_EXPORT_S3_REGION`, and keys from `INBUXA_EXPORT_S3_ACCESS_KEY_FILE` / `_SECRET_KEY_FILE` (AWS defaults without them). - `--store-check quick|full`: a store check to its end, then exit 0 (nothing wrong), 3 (problems), 2 (another running), 1 (failed). Saved with the console's checks. **Fixed** - A live export could name blobs it didn't have: areas went out in a set's order, so blobs could go first. Blobs go last now. - **An import into a fresh store refused to run, from a directory too.** Every start writes the fork's subspace (blob store location, permission grants, log files' setting), the import's own start included, and the import took that for data. Since #333: 27.1-rc.16 and 2026.11.9.1. The store test now imports over a start-up record, and fails without the fix. - **An import put the blobs into the data store even when the imported settings name S3**, so the restored server found none of them. Blobs are imported last, into the blob store the imported settings name. - The S3 export and import could hang for good on a pod limited to one CPU (one runtime worker). A wait hands its worker over first. **Tested**: inbuxa-k8s-lab `backup-test.sh` on kind (Postgres, Garage, NATS, five pods): mail, the chart's backup CronJob, the database dropped and blob bucket emptied, the import, every mailbox's count and message hashes the same after, quick and full store checks clean. Store tests on SQLite and RocksDB; unit tests; notice and name checks.
jcoffey-dev added 1 commit 2026-11-19 20:49:20 +00:00
Backups for Kubernetes: export to and import from S3, store checks as a command (K-C11)
ci / fork-checks (pull_request) Skipped
ci / build (pull_request) Skipped
github/ci (branch) GitHub Actions
ci / github (pull_request) Successful in 14m26s
4f2187b805
For the chart's scheduled backup and store checks (inbuxa-charts#11), and
three bugs the lab's restore test found on the way.

Added
- `--export s3://bucket/prefix` streams the export into multipart
  uploads, in a new folder named for the time (20261119T204500Z), and
  writes export.json last, naming the files: an export without it didn't
  finish. Nothing the size of the store lands on local disk. Parts start
  at 16 MiB and double every thousand up to 512 MiB, so one file can be
  nearly 3 TiB; one part is held at a time.
- `--import s3://bucket/prefix/<name>` reads one back, streamed; from
  `s3://bucket/prefix`, the newest finished one.
- The bucket from INBUXA_EXPORT_S3_ENDPOINT (S3-compatible, path-style;
  AWS without it), INBUXA_EXPORT_S3_REGION, and the keys from
  INBUXA_EXPORT_S3_ACCESS_KEY_FILE and INBUXA_EXPORT_S3_SECRET_KEY_FILE
  (the AWS defaults without them).
- `--store-check quick|full` runs a store check to its end and exits: 0
  nothing wrong, 3 problems found, 2 another check running, 1 failed. It's
  saved with the console's other checks, under a node id no server holds.

Fixed
- A live export could name blobs it didn't have: the store's areas went
  out in a set's order, so the blobs could go first and a message stored
  after them pointed at a blob the export never got. Blobs go last now.
- An import into a fresh store refused to run: every start writes the
  fork's subspace (where the blob store is, permission grants, the log
  files' setting), the import's own start too, and the import took that
  for data. Since the blob store change (#333): 27.1-rc.16 and
  2026.11.9.1. The fork's subspace is no longer counted; its records are
  keyed values an import sets over. The store test now imports over a
  start-up record, and fails without this.
- An import wrote the blobs into the data store, the empty store's blob
  store, even when the imported settings name another (S3, say): the
  restored server then found none of them. Blobs are imported last, into
  the blob store the imported x:BlobStore names.
- The S3 export and import could stop for good on a pod limited to one CPU
  (one runtime worker): a reader or sender waiting on the transfer held
  the worker the transfer needed. A wait hands its worker over first.

Tested: inbuxa-k8s-lab backup-test.sh, on the kind lab (Postgres, Garage,
NATS, five server pods): mail for an account, the chart's backup CronJob,
the database dropped and the blob bucket emptied, the import, and every
mailbox's count and message hashes the same after; quick and full store
checks clean. store tests pass on SQLite and RocksDB; unit tests pass.
jcoffey-dev merged commit f5fb942093 into main 2026-11-19 21:04:02 +00:00
jcoffey-dev deleted branch feat/k8s-backups 2026-11-19 21:04:02 +00:00
jcoffey-dev referenced this issue from a commit 2026-11-20 01:59:38 +00:00
Sign in to join this conversation.