Cluster: honour the push role on updates; step down when a node id lease is taken #166

Merged

jcoffey-dev merged 1 commits from cluster/push-role-lease-stepdown into main

2026-10-08 06:07:55 +00:00

jcoffey-dev commented

2026-10-08 05:17:46 +00:00

Owner

Two cluster fixes from the survey of what upstream's single-node design misses on three nodes.

Web push once per node. Subscription updates are broadcast to every node and the push manager only checked the node's role when loading subscriptions at startup. A node whose role turns push off still registered each updated subscription and POSTed its own copy. Event::Update now checks the role as well as the shard. (hosts-baseline gets the matching role change: pushNotifications off on peer/peer-mta.)

Colliding snowflake ids after a lost lease. A node silent for over an hour can have its id taken by another host. On its return the renewal only logged held by another host and the node kept minting ids as that node. It now claims a fresh id, re-points the id generator and logs cluster.node-id-reassigned (event 649). RegistryStoreInner.node_id is atomic for that, and the claim loop is shared by startup and the step-down.

Checks: cargo check + clippy on store/services (no new warnings), cargo test -p store --lib build::cluster (4 pass, one new), fork checks clean.

Two cluster fixes from the survey of what upstream's single-node design misses on three nodes. **Web push once per node.** Subscription updates are broadcast to every node and the push manager only checked the node's role when loading subscriptions at startup. A node whose role turns push off still registered each updated subscription and POSTed its own copy. `Event::Update` now checks the role as well as the shard. (hosts-baseline gets the matching role change: `pushNotifications` off on `peer`/`peer-mta`.) **Colliding snowflake ids after a lost lease.** A node silent for over an hour can have its id taken by another host. On its return the renewal only logged `held by another host` and the node kept minting ids as that node. It now claims a fresh id, re-points the id generator and logs `cluster.node-id-reassigned` (event 649). `RegistryStoreInner.node_id` is atomic for that, and the claim loop is shared by startup and the step-down. Checks: `cargo check` + `clippy` on store/services (no new warnings), `cargo test -p store --lib build::cluster` (4 pass, one new), fork checks clean.

jcoffey-dev added 1 commit 2026-10-08 05:17:47 +00:00

Cluster: honour the push role on updates; step down when a node id lease is taken

ci / fork-checks (pull_request) Skipped

Details

ci / build (pull_request) Skipped

Details

github/ci (branch) GitHub Actions

Details

ci / github (pull_request) Successful in 7m47s

Details

d7234aa1ca

Push subscription updates are broadcast to every node, and the push manager
only checked the role when loading subscriptions at startup. A node whose
role turns push off still registered each updated subscription and sent
its own copy, so a client was pushed once per node. Event::Update now
checks the role as well as the shard.

A node that loses its id lease (silent for over an hour, another host took
the id) only logged the failed renewal and carried on minting snowflake ids
as that node, which can collide with the new holder's. The node now claims
a fresh id, re-points the id generator and logs cluster.node-id-reassigned.
The registry's node id is atomic for that; the claim loop is shared by
startup and the step-down.

jcoffey-dev merged commit bb2ce08232 into main

2026-10-08 06:07:55 +00:00

jcoffey-dev deleted branch cluster/push-role-lease-stepdown

2026-10-08 06:07:55 +00:00

jcoffey-dev referenced this pull request

2026-10-08 06:32:11 +00:00

Release 2026.10.8 #168

jcoffey-dev referenced this pull request

2026-10-08 17:57:46 +00:00

Release 2026.10.8.1: size the collector tables for event 649 #169