Defaults: one node at a time on a fresh store #482

Merged
jcoffey-dev merged 1 commits from fix/436-defaults-lease into main 2026-11-19 17:08:05 +00:00
Owner

Fixes #436.

Cause. Every normal start adds what the store is missing, as "none of these yet, so insert them". Two nodes starting together on a fresh store both did that. The second found 6 of the 7 default listeners already written (http is last in DEFAULT_LISTENERS), took the set as complete and came up without http, matching the lab report. The default roles and the Authentication record could be stored twice the same way.

Fix. The normal-mode defaults run under a short store lease (_ key Bl, under the first-boot-defaults letter in fork_subspaces::OWNERS), taken with a compare-and-swap like the store layout lease: 60 s, renewed every 15 s, released when done. A node that finds it held waits for release or expiry, then checks the store itself. Bootstrap and recovery mode unchanged.

Tests. tests/src/store/defaults_lease.rs: four tasks on one store never hold the lease together; a live foreign lease blocks; an expired or released one is taken at once. Both pass locally; fork checks and cargo fmt --check clean.

Not in this PR: the issue's other "done when", that a listener that can't bind should be retried or stop the server. In this case http was never tried, so the race fix covers it. Making a bind failure fatal changes behaviour for every install, so it's left for a decision. The two-node lab test is for the Kubernetes lab.

Fixes #436. **Cause.** Every normal start adds what the store is missing, as "none of these yet, so insert them". Two nodes starting together on a fresh store both did that. The second found 6 of the 7 default listeners already written (`http` is last in `DEFAULT_LISTENERS`), took the set as complete and came up without `http`, matching the lab report. The default roles and the Authentication record could be stored twice the same way. **Fix.** The normal-mode defaults run under a short store lease (`_` key `Bl`, under the first-boot-defaults letter in `fork_subspaces::OWNERS`), taken with a compare-and-swap like the store layout lease: 60 s, renewed every 15 s, released when done. A node that finds it held waits for release or expiry, then checks the store itself. Bootstrap and recovery mode unchanged. **Tests.** `tests/src/store/defaults_lease.rs`: four tasks on one store never hold the lease together; a live foreign lease blocks; an expired or released one is taken at once. Both pass locally; fork checks and `cargo fmt --check` clean. **Not in this PR:** the issue's other "done when", that a listener that can't bind should be retried or stop the server. In this case `http` was never tried, so the race fix covers it. Making a bind failure fatal changes behaviour for every install, so it's left for a decision. The two-node lab test is for the Kubernetes lab.
jcoffey-dev added 1 commit 2026-11-19 16:29:58 +00:00
Defaults: one node at a time on a fresh store (#436)
ci / fork-checks (pull_request) Skipped
ci / build (pull_request) Skipped
github/ci (branch) GitHub Actions
ci / github (pull_request) Successful in 11m49s
3e6300f802
Every normal start adds what the store is missing, each as "none of
these yet, so insert them". Two nodes starting together on a freshly
bootstrapped store both did that at once. The second could find some of
the seven default listeners already written by the first, take the set
as complete and come up without the rest: in the Kubernetes lab, every
listener but http, the last one, so the node answered nothing on 8080.
The default roles and the Authentication record could be stored twice
the same way.

The normal-mode defaults now run under a short lease in the store
(`_` key "Bl", under the first boot defaults' letter), taken with a
compare-and-swap like the store layout lease: 60 seconds, renewed every
15 while held, released when done. A node that finds it held waits until
it's released, or runs out because its holder stopped, then checks the
store itself, which by then has everything. Bootstrap and recovery mode
are unchanged.

Test: tests/src/store/defaults_lease.rs: four tasks sharing one store
never hold the lease together; a live lease another node holds blocks;
one that ran out, or was released, is taken at once.
jcoffey-dev merged commit 31df4724c4 into main 2026-11-19 17:08:05 +00:00
jcoffey-dev deleted branch fix/436-defaults-lease 2026-11-19 17:08:05 +00:00
Sign in to join this conversation.