Cluster: one node at a time does the singleton roles' work (#437) #524

Open
jcoffey-dev wants to merge 1 commits from feat/singleton-role-lease into main
pull from: feat/singleton-role-lease
Owner

Fixes #437.

Every node with taskScheduler queued its own copy of each scheduled maintenance task, and every node with metricsCalculate evaluated alerts and sent their emails (inbuxa-k8s-lab#4). So each of those roles had to be on exactly one node, and losing it paused that work until it came back.

What changes

  • One lease per role in the store: Xs (task scheduler) and Xm (metrics and alerts) in _, taken with a compare-and-swap like the store layout and defaults leases. Every node with the role tries for it each minute (spawn_singleton_leases); the holder renews it and does the work: queuing the scheduled tasks and the daily domain-task check, or the cluster-wide gauges, alert evaluation and alert emails.
  • Handover: a lease lasts 3 minutes, and its holder stops acting 15 seconds before it runs out. When a holder dies, another node with the role takes over within about 4 minutes, inside the issue's "two lease periods". A clean stop gives the leases up, so a restart or a roll hands over at the next round, with no gap. Ownership is a per-process token, so a reassigned node id can't confuse it.
  • No coordinator, no lease: a node without a cluster coordinator decides by its role alone, as today. A single node behaves exactly as before.
  • Alerts across a handover:
    • The alerts firing at the last evaluation are stored (Xa), and a node taking over the metrics role loads them, so an alert still firing isn't sent again.
    • The new holder sets the cluster-wide gauges before its first evaluation, so no alert reads a gauge it never set.
    • The daily account and domain counts are taken at once by a new holder, not up to a day later.
  • Events: cluster.singleton-lease-taken and cluster.singleton-lease-lost (665, 666, level info), also in the served schema so the console can list them.
  • Docs: X is registered in the _ owners table (fork_subspaces.rs). In store-layout.md, the table gains X plus the N (passkeys) and O (listener status) rows it was missing; the code already had them. monitoring.md MON-25/26 now say who evaluates, and a new "Singleton roles" item covers the lease.

Upgrading

  • No store layout step. The keys are new; an older node never reads them.
  • Rolling-safe for today's setups, where each role is on one node: that node takes the lease and carries on. Old and new nodes mixed during a roll don't double up, since only one node has the role.
  • Prod: host1's scheduler and alerts carry on; on its restart in a roll, the lease is given up and taken back at once.

Checked

  • New unit tests for the lease bytes, the holder's margin and the firing set round trip.
  • New store tests in tests/src/store/singleton_lease.rs, with nodes played by tokens on one store:
    • 8 nodes at once get exactly one holder;
    • the holder renews and nobody else gets it;
    • a lease that ran out, or was given up, passes on at once;
    • giving up another node's lease does nothing;
    • an unreadable value counts as no lease.
  • These pass, with the existing defaults-lease tests: cargo test -p tests -- store::singleton_lease store::defaults_lease.
  • Also clean locally: cargo fmt --check, cargo build -p inbuxa --locked, clippy on the changed crates, the trc tests (event count), the migration tests, cargo test --workspace --no-run, expr-schema.py --check, and the privacy, name, notice and context checks.
  • Not tested here: two live nodes end to end. The k8s session will add that once this is in a signed RC: a lab test with two core replicas, killing the holder, plus the chart change letting core run two replicas.

Milestone: #437 is in After 27.1. Merging to main puts it in the next 27.1 RC, so that's John's call.

Fixes #437. Every node with `taskScheduler` queued its own copy of each scheduled maintenance task, and every node with `metricsCalculate` evaluated alerts and sent their emails (inbuxa-k8s-lab#4). So each of those roles had to be on exactly one node, and losing it paused that work until it came back. **What changes** - **One lease per role** in the store: `Xs` (task scheduler) and `Xm` (metrics and alerts) in `_`, taken with a compare-and-swap like the store layout and defaults leases. Every node with the role tries for it each minute (`spawn_singleton_leases`); the holder renews it and does the work: queuing the scheduled tasks and the daily domain-task check, or the cluster-wide gauges, alert evaluation and alert emails. - **Handover:** a lease lasts 3 minutes, and its holder stops acting 15 seconds before it runs out. When a holder dies, another node with the role takes over within about 4 minutes, inside the issue's "two lease periods". A clean stop gives the leases up, so a restart or a roll hands over at the next round, with no gap. Ownership is a per-process token, so a reassigned node id can't confuse it. - **No coordinator, no lease:** a node without a cluster coordinator decides by its role alone, as today. A single node behaves exactly as before. - **Alerts across a handover:** - The alerts firing at the last evaluation are stored (`Xa`), and a node taking over the metrics role loads them, so an alert still firing isn't sent again. - The new holder sets the cluster-wide gauges before its first evaluation, so no alert reads a gauge it never set. - The daily account and domain counts are taken at once by a new holder, not up to a day later. - **Events:** `cluster.singleton-lease-taken` and `cluster.singleton-lease-lost` (665, 666, level info), also in the served schema so the console can list them. - **Docs:** `X` is registered in the `_` owners table (`fork_subspaces.rs`). In `store-layout.md`, the table gains `X` plus the `N` (passkeys) and `O` (listener status) rows it was missing; the code already had them. `monitoring.md` MON-25/26 now say who evaluates, and a new "Singleton roles" item covers the lease. **Upgrading** - No store layout step. The keys are new; an older node never reads them. - Rolling-safe for today's setups, where each role is on one node: that node takes the lease and carries on. Old and new nodes mixed during a roll don't double up, since only one node has the role. - Prod: host1's scheduler and alerts carry on; on its restart in a roll, the lease is given up and taken back at once. **Checked** - New unit tests for the lease bytes, the holder's margin and the firing set round trip. - New store tests in `tests/src/store/singleton_lease.rs`, with nodes played by tokens on one store: - 8 nodes at once get exactly one holder; - the holder renews and nobody else gets it; - a lease that ran out, or was given up, passes on at once; - giving up another node's lease does nothing; - an unreadable value counts as no lease. - These pass, with the existing defaults-lease tests: `cargo test -p tests -- store::singleton_lease store::defaults_lease`. - Also clean locally: `cargo fmt --check`, `cargo build -p inbuxa --locked`, clippy on the changed crates, the `trc` tests (event count), the `migration` tests, `cargo test --workspace --no-run`, `expr-schema.py --check`, and the privacy, name, notice and context checks. - Not tested here: two live nodes end to end. The k8s session will add that once this is in a signed RC: a lab test with two `core` replicas, killing the holder, plus the chart change letting `core` run two replicas. **Milestone:** #437 is in **After 27.1**. Merging to main puts it in the next 27.1 RC, so that's John's call.
jcoffey-dev added 1 commit 2026-11-21 08:07:35 +00:00
Cluster: one node at a time does the singleton roles' work (#437)
ci / fork-checks (pull_request) Skipped
ci / build (pull_request) Skipped
github/ci (branch) GitHub Actions
ci / github (pull_request) Successful in 9m5s
2358f4814c
Every node with taskScheduler queued its own copy of each scheduled
maintenance task, and every node with metricsCalculate evaluated alerts
and sent their emails, so each role had to be on exactly one node, and
losing that node paused the work until it came back.

Each role now has a lease in the store (Xs, Xm in `_`), taken with a
compare-and-swap like the store layout lease. Every node with the role
tries for it each minute; the holder renews it and does the work. A
lease lasts 3 minutes and its holder stops acting 15 seconds before it
runs out, so another node with the role takes over within about 4
minutes of the holder stopping. A clean stop gives the leases up, so a
restart or a roll hands over at the next round. A node without a
cluster coordinator needs no lease: the role alone decides, as before.

The firing alerts are stored too (Xa), so a node that takes over alerts
doesn't send the ones still firing again, and it sets the cluster-wide
gauges before its first evaluation. The daily account and domain counts
no longer wait a day on a node that just took the role over.

New events cluster.singleton-lease-taken and -lost (665, 666), also in
the served schema. `X` is registered in the `_` owners table; the spec's
copy of that table gains the N (passkeys) and O (listener status) rows
it was missing.
All checks were successful
ci / fork-checks (pull_request) Skipped
ci / build (pull_request) Skipped
github/ci (branch) GitHub Actions
ci / github (pull_request) Successful in 9m5s
You are not authorized to merge this pull request.
This pull request can be merged automatically.
View command line instructions

Checkout

From your project repository, check out a new branch and test the changes.
git fetch -u origin feat/singleton-role-lease:feat/singleton-role-lease
git checkout feat/singleton-role-lease
Sign in to join this conversation.