Cluster-wide lease for singleton roles (scheduler, metrics, alerts) #437

Open
opened 2026-11-18 07:07:37 +00:00 by jcoffey-dev · 0 comments
Owner

Part of inbuxa for Kubernetes, after 27.1 (tracking: inbuxa-charts#14).

Today every node with the taskScheduler role queues its own copy of each scheduled maintenance task, and every node with metricsCalculate evaluates alerts, so these roles must be on exactly one node (shown in inbuxa-k8s-lab#4). On Kubernetes that makes the core group a single replica: losing it pauses scheduled jobs and alerts until it is rescheduled.

Take a lease in the store for each singleton role (as the layout runner does for upgrades: renewed every minute, taken over when it lapses), so several nodes can carry the role and only the lease holder acts.

Done when

  • Two nodes with every role queue each scheduled task once and send each alert once
  • Killing the lease holder hands the work over within two lease periods
  • The chart lets core run two replicas
Part of inbuxa for Kubernetes, after 27.1 (tracking: inbuxa-charts#14). Today every node with the `taskScheduler` role queues its own copy of each scheduled maintenance task, and every node with `metricsCalculate` evaluates alerts, so these roles must be on exactly one node (shown in inbuxa-k8s-lab#4). On Kubernetes that makes the `core` group a single replica: losing it pauses scheduled jobs and alerts until it is rescheduled. Take a lease in the store for each singleton role (as the layout runner does for upgrades: renewed every minute, taken over when it lapses), so several nodes can carry the role and only the lease holder acts. **Done when** - Two nodes with every role queue each scheduled task once and send each alert once - Killing the lease holder hands the work over within two lease periods - The chart lets `core` run two replicas
jcoffey-dev added this to the 27.7 milestone 2026-11-18 07:07:37 +00:00
jcoffey-dev added the enhancement label 2026-11-18 07:07:37 +00:00
jcoffey-dev self-assigned this 2026-11-18 07:16:14 +00:00
jcoffey-dev added the
priority
p3
label 2026-11-18 07:16:14 +00:00
Sign in to join this conversation.
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: inbuxa/inbuxa-server#437