Report a node unhealthy after three minutes of silence #54

Merged

jcoffey-dev merged 1 commits from feature/node-heartbeat into main

2026-09-25 08:15:45 +00:00

jcoffey-dev commented

2026-09-25 08:06:02 +00:00

Owner

Every node now renews its lease once a minute instead of every 30 minutes, so the lease works as a heartbeat.

x:ClusterNode reports a node Stale once it has gone 3 minutes without renewing. Before, that took an hour. Inactive still means a day of silence.
Taking over a lease still needs a full hour of silence, so a node that is slow rather than gone never loses its id to another host, and snowflake ids stay unique.
Cost: one small store write per node per minute.

The admin dashboard's new Cluster Health card (inbuxa-admin, feature/cluster-health) counts these statuses.

Tests: cargo test -p store --lib build::cluster (3 new unit tests on the thresholds).

Every node now renews its lease once a minute instead of every 30 minutes, so the lease works as a heartbeat. - `x:ClusterNode` reports a node **Stale** once it has gone 3 minutes without renewing. Before, that took an hour. **Inactive** still means a day of silence. - Taking over a lease still needs a full hour of silence, so a node that is slow rather than gone never loses its id to another host, and snowflake ids stay unique. - Cost: one small store write per node per minute. The admin dashboard's new Cluster Health card (inbuxa-admin, feature/cluster-health) counts these statuses. Tests: `cargo test -p store --lib build::cluster` (3 new unit tests on the thresholds).

jcoffey-dev added 1 commit 2026-09-25 08:06:03 +00:00

Report a node unhealthy after three minutes of silence

ci / fork-checks (pull_request) Successful in 38s

Details

ci / build (pull_request) Successful in 7m24s

Details

5245abd08d

Every node renews its lease once a minute instead of every 30 minutes,
so the lease works as a heartbeat. x:ClusterNode reports a node Stale
once it has gone three minutes without renewing (it used to take an
hour), and Inactive after a day, as before.

Taking over a lease still needs a full hour of silence. A node that is
slow rather than gone never loses its id to another host, so snowflake
ids stay unique.

The admin dashboard's Cluster Health card counts these statuses.

jcoffey-dev merged commit 5e2ddf644f into main

2026-09-25 08:15:45 +00:00

jcoffey-dev deleted branch feature/node-heartbeat

2026-09-25 08:15:45 +00:00

jcoffey-dev referenced this pull request

2026-09-25 08:16:11 +00:00

Release 2026.9.25.1 #55

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: inbuxa/inbuxa-server#54