Cluster Health: readable on Kubernetes, no pile-up after rollouts (K-A3) #258

Merged
jcoffey-dev merged 1 commits from feat/cluster-health-pods into main 2026-11-20 07:33:25 +00:00
Owner

For K-A3 (#235).

Before: on Kubernetes a node's hostname is its pod name, and pod names are never reused. Every pod a rollout replaced, and every finished Job pod (the chart's upgrade hook joins the cluster for its run), left a lease that went Silent and stayed counted for 24 hours. The lab after today's restarts: 5 of 10 nodes up, in red, five healthy pods running, plus an attention item for nodes not responding.

Now:

  • podName() reads a pod name: Deployment <name>-<hash>-<5>, Job or DaemonSet <name>-<5>, using Kubernetes' vowel-free alphabet for the generated parts. FQDNs, plain hostnames and StatefulSet names (inbuxa-0) don't match.
  • A silent lease with a pod name is a pod that's gone. It isn't counted and goes on one line (5 pods replaced or finished, not counted: inbuxa-upgrade, inbuxa-front ×2, inbuxa-mta ×2) instead of a red card.
  • A Deployment with no pod renewing still counts as down, once, shown by its latest pod. Silent servers that aren't pods count exactly as before.
  • The roster groups pods under what made them (inbuxa-mta · 2 of 2 pods up). Each card is titled with the pod's own part (674c7dffc5-c9bp7), with the full name on hover.
  • Attention, the command center gauge and the header chip use the same counts.

Checked: the lab's real x:ClusterNode list (5 live pods, 4 replaced, 1 upgrade Job), rendered in a harness: 5/5 up, three groups, one line for the gone pods. With every mta pod removed: 3/4 up, crit, inbuxa-mta 0 of 1 pod up with its Silent card. Unit tests cover both cases plus a silent non-pod server. Typecheck, lint, i18n:check, all 1156 tests, the release/changelog/IPv6 scripts and the build pass locally.

For K-A3 (#235). **Before:** on Kubernetes a node's hostname is its pod name, and pod names are never reused. Every pod a rollout replaced, and every finished Job pod (the chart's upgrade hook joins the cluster for its run), left a lease that went Silent and stayed counted for 24 hours. The lab after today's restarts: **5 of 10 nodes up, in red**, five healthy pods running, plus an attention item for nodes not responding. **Now:** - `podName()` reads a pod name: Deployment `<name>-<hash>-<5>`, Job or DaemonSet `<name>-<5>`, using Kubernetes' vowel-free alphabet for the generated parts. FQDNs, plain hostnames and StatefulSet names (`inbuxa-0`) don't match. - A silent lease with a pod name is a pod that's gone. It isn't counted and goes on one line (`5 pods replaced or finished, not counted: inbuxa-upgrade, inbuxa-front ×2, inbuxa-mta ×2`) instead of a red card. - A Deployment with **no** pod renewing still counts as down, once, shown by its latest pod. Silent servers that aren't pods count exactly as before. - The roster groups pods under what made them (`inbuxa-mta · 2 of 2 pods up`). Each card is titled with the pod's own part (`674c7dffc5-c9bp7`), with the full name on hover. - Attention, the command center gauge and the header chip use the same counts. **Checked:** the lab's real `x:ClusterNode` list (5 live pods, 4 replaced, 1 upgrade Job), rendered in a harness: 5/5 up, three groups, one line for the gone pods. With every mta pod removed: 3/4 up, crit, `inbuxa-mta 0 of 1 pod up` with its Silent card. Unit tests cover both cases plus a silent non-pod server. Typecheck, lint, i18n:check, all 1156 tests, the release/changelog/IPv6 scripts and the build pass locally.
jcoffey-dev added 1 commit 2026-11-20 07:30:47 +00:00
Cluster Health: readable on Kubernetes, no pile-up after rollouts (K-A3)
ci / build (pull_request) Skipped
ci / publish (pull_request) Skipped
ci / notices (pull_request) Successful in 56s
github/ci (branch) GitHub Actions
ci / github (pull_request) Successful in 2m14s
1c06da14d9
On Kubernetes a node's hostname is its pod's name, and a pod's name is
never used again. Every pod a rollout replaced (and every finished Job
pod, like the chart's upgrade hook) kept a lease that went Silent and
stayed counted for a day: the lab after a few restarts showed 5 of 10
nodes up, in red, with five healthy pods running.

clusterHealth.ts reads a pod's name (Deployment <name>-<hash>-<5>, Job
<name>-<5>, Kubernetes' vowel-free alphabet). A silent lease with a pod's
name is a pod that's gone: left out of the count and listed on one line
instead of a red card. Only a Deployment with no pod renewing at all
counts as down, once. Silent servers that aren't pods count as before.

The roster groups pods under what made them (inbuxa-mta: 2 of 2 pods
up), each card titled with the pod's own part, full name on hover.
Attention and the command center use the same counts.
jcoffey-dev merged commit 9f9fb088c3 into main 2026-11-20 07:33:25 +00:00
jcoffey-dev deleted branch feat/cluster-health-pods 2026-11-20 07:33:25 +00:00
Sign in to join this conversation.