--upgrade-store: upgrade the store layout, then exit #448

Merged
jcoffey-dev merged 1 commits from feat/upgrade-store into main 2026-11-18 15:56:02 +00:00
Owner

Closes #426 (K-S3).

27.1's layout step 2 → 3 can't roll (lab spike S5): a node starting on it waits for every other node to stop, so a rolling update deadlocks. The chart will scale to 0, run the upgrade as a Job, then roll out. This is that Job's command.

inbuxa --config <PATH> --upgrade-store (or -u, or with CONFIG_PATH)

  • Runs what a start runs on the store before it serves (try_migrate's chain), then exits. No listeners, no services, no spam model or ASN fetch, no safe defaults written.
  • Exit 0: Upgraded the store layout from version 2 to 3. or The store layout is at version 3: nothing to upgrade.
  • Exit 1: Failed to upgrade the store layout: …
  • Exit 2: Refused to upgrade the store layout: …
    • Another node holds the lease. It refuses at once (a starting node would wait) and leaves that lease alone.
    • Or a step needs every other node stopped and some still show as running after 6 minutes. That wait is there because a node counts as gone 3 minutes after its last heartbeat, so pods scaled down just before the Job don't fail it.
  • Refuses bootstrap and recovery mode (nothing to upgrade / meant for a person).

Argument parsing (boot.rs), needed for this and fixed on the way:

  • Arguments are read even with CONFIG_PATH set; before, they were ignored entirely, so a Job using CONFIG_PATH couldn't pass the flag.
  • An option that takes no value (--console, --upgrade-store) no longer swallows the next argument: --upgrade-store --config x used to fail with "Missing value for argument".
  • A store command without a configuration exits 1, not 0 (an upgrade Job must not pass without doing anything). Same for --export/--import.

Tested

  • Integration (store_version::store_version, RocksDB): a current store is left alone twice over; a v1 store under node 900's live lease is refused within 5 s, mark and lease untouched; once the lease runs out it's upgraded from 1, the lease released, and a second run is a no-op. Both store_version tests pass.
    • The test's lease_held helper read the 10-byte lease as a u64. It had only ever run with no lease present, so the bug never showed until now; it reads the raw value now.
  • The binary, by hand: no config → exit 1; --upgrade-store --config and --config … --upgrade-store and CONFIG_PATH … -u on a new store → 0, "nothing to upgrade", twice; --upgrade-store=yes → 1; a normal start still binds all seven listeners.
  • fmt, workspace --no-run, fork checks: ok; no new clippy warnings.

Not in it: the Job leaves its node-id slot to expire like a stopped node does. It never heartbeats, so it shows as a node for a few minutes after it exits. Harmless: the mark is current by then, so nothing waits on it.

Docs PR on inbuxa/inbuxa.org (merge after the release).

Closes #426 (K-S3). 27.1's layout step 2 → 3 can't roll (lab spike S5): a node starting on it waits for every other node to stop, so a rolling update deadlocks. The chart will scale to 0, run the upgrade as a Job, then roll out. This is that Job's command. **`inbuxa --config <PATH> --upgrade-store`** (or `-u`, or with `CONFIG_PATH`) - Runs what a start runs on the store before it serves (`try_migrate`'s chain), then exits. No listeners, no services, no spam model or ASN fetch, no safe defaults written. - Exit **0**: `Upgraded the store layout from version 2 to 3.` or `The store layout is at version 3: nothing to upgrade.` - Exit **1**: `Failed to upgrade the store layout: …` - Exit **2**: `Refused to upgrade the store layout: …` - Another node holds the lease. It refuses at once (a starting node would wait) and leaves that lease alone. - Or a step needs every other node stopped and some still show as running after 6 minutes. That wait is there because a node counts as gone 3 minutes after its last heartbeat, so pods scaled down just before the Job don't fail it. - Refuses bootstrap and recovery mode (nothing to upgrade / meant for a person). **Argument parsing** (`boot.rs`), needed for this and fixed on the way: - Arguments are read even with `CONFIG_PATH` set; before, they were ignored entirely, so a Job using `CONFIG_PATH` couldn't pass the flag. - An option that takes no value (`--console`, `--upgrade-store`) no longer swallows the next argument: `--upgrade-store --config x` used to fail with "Missing value for argument". - A store command without a configuration exits 1, not 0 (an upgrade Job must not pass without doing anything). Same for `--export`/`--import`. **Tested** - Integration (`store_version::store_version`, RocksDB): a current store is left alone twice over; a v1 store under node 900's live lease is refused within 5 s, mark and lease untouched; once the lease runs out it's upgraded from 1, the lease released, and a second run is a no-op. Both store_version tests pass. - The test's `lease_held` helper read the 10-byte lease as a u64. It had only ever run with no lease present, so the bug never showed until now; it reads the raw value now. - The binary, by hand: no config → exit 1; `--upgrade-store --config` and `--config … --upgrade-store` and `CONFIG_PATH … -u` on a new store → 0, "nothing to upgrade", twice; `--upgrade-store=yes` → 1; a normal start still binds all seven listeners. - fmt, workspace `--no-run`, fork checks: ok; no new clippy warnings. Not in it: the Job leaves its node-id slot to expire like a stopped node does. It never heartbeats, so it shows as a node for a few minutes after it exits. Harmless: the mark is current by then, so nothing waits on it. Docs PR on inbuxa/inbuxa.org (merge after the release).
jcoffey-dev added this to the 27.1 milestone 2026-11-18 15:43:04 +00:00
jcoffey-dev added 1 commit 2026-11-18 15:43:05 +00:00
--upgrade-store: upgrade the store layout, then exit
ci / fork-checks (pull_request) Skipped
ci / build (pull_request) Skipped
github/ci (branch) GitHub Actions
ci / github (pull_request) Successful in 9m47s
122b7f7d2d
For an orchestrator that upgrades the store as a job of its own before it
rolls a release out (27.1's step to layout 3 can't roll; lab spike S5).
It runs what a start runs on the store before serving, opens no ports and
starts no services, and exits 0 when the store is at this build's layout
(upgraded or found so), 1 when the upgrade failed, 2 when it refused.

Unlike a starting node it doesn't wait out another node's lease: it
refuses, and leaves that lease alone. A step that needs every other node
stopped waits up to six minutes for them to count as gone, so pods
stopped just before it don't fail the job, and is refused after that.

The arguments are now read with CONFIG_PATH set too, an option that takes
no value no longer swallows the next argument (so --upgrade-store
--config <PATH> works in that order), and a store command without a
configuration exits 1 instead of 0.

Closes #426
jcoffey-dev merged commit ccf44f31c6 into main 2026-11-18 15:56:02 +00:00
jcoffey-dev deleted branch feat/upgrade-store 2026-11-18 15:56:02 +00:00
Sign in to join this conversation.