Metrics for Kubernetes: scraping and OTLP from the environment, alert gauges, four metric fixes #488

Merged
jcoffey-dev merged 6 commits from feat/k8s-metrics into main 2026-11-19 18:35:13 +00:00
Owner

For the Helm chart's metrics (K-C10, inbuxa-charts#10), and four metric bugs the lab found on the way.

Metrics from the deployment, so a chart can turn them on for every pod:

  • INBUXA_METRICS_PROMETHEUS_USERNAME and INBUXA_METRICS_PROMETHEUS_SECRET_FILE open /metrics/prometheus with that username and the password in the file. Both or neither: never opened without a password this way.
  • When the store has no OpenTelemetry metrics set, OTEL_EXPORTER_OTLP_ENDPOINT (or _METRICS_ENDPOINT) turns OTLP on from the standard OTEL_ variables (protocol, timeout, interval, resource attributes). INBUXA_OTLP_HEADERS_FILE reads the headers (a collector token) from a file. Every pod pushes its own, whatever its role: each process counts only its own traffic.

Three gauges for alerts, each a Unix time:

  • queue.oldest-message: when the oldest queued message arrived, from the first queue id (a snowflake), with no message read.
  • cluster.lease-renewed: when this node last renewed its cluster lease. (A count of Stale nodes wouldn't do on Kubernetes: a pod that rolls away never gives its lease back.)
  • tls.earliest-expiry: when the first name served runs out of certificate.

Fixes the lab found

  • /healthz/live went through the anonymous request limit, kept in the in-memory store. With the data store down the probe timed out and Kubernetes restarted every pod at once. Liveness answers before the limit now.
  • Counters still at zero were left out of the scrape, so one first seen at 3 never read as having risen and rate()/increase() alerts missed it.
  • Gauges could wrap below zero: an mta pod reported 18446744073709552000 queued messages.
  • In a cluster, each node moved queue.count by its own queued and delivered messages, so the counting node read low (three waiting showed as none). With a coordinator the gauge is only the count from the store.

Tested on the kind lab (inbuxa-k8s-lab, metrics-test.sh and alerts-test.sh): scraping with and without the password on all five pods, OTLP from every pod with the token, and every alert in the chart's rules quiet on a healthy install and firing from real breakage. trc and common unit tests pass; notice and name checks clean.

For the Helm chart's metrics (K-C10, inbuxa-charts#10), and four metric bugs the lab found on the way. **Metrics from the deployment**, so a chart can turn them on for every pod: - `INBUXA_METRICS_PROMETHEUS_USERNAME` and `INBUXA_METRICS_PROMETHEUS_SECRET_FILE` open `/metrics/prometheus` with that username and the password in the file. Both or neither: never opened without a password this way. - When the store has no OpenTelemetry metrics set, `OTEL_EXPORTER_OTLP_ENDPOINT` (or `_METRICS_ENDPOINT`) turns OTLP on from the standard `OTEL_` variables (protocol, timeout, interval, resource attributes). `INBUXA_OTLP_HEADERS_FILE` reads the headers (a collector token) from a file. Every pod pushes its own, whatever its role: each process counts only its own traffic. **Three gauges for alerts**, each a Unix time: - `queue.oldest-message`: when the oldest queued message arrived, from the first queue id (a snowflake), with no message read. - `cluster.lease-renewed`: when this node last renewed its cluster lease. (A count of Stale nodes wouldn't do on Kubernetes: a pod that rolls away never gives its lease back.) - `tls.earliest-expiry`: when the first name served runs out of certificate. **Fixes the lab found** - `/healthz/live` went through the anonymous request limit, kept in the in-memory store. With the data store down the probe timed out and Kubernetes restarted every pod at once. Liveness answers before the limit now. - Counters still at zero were left out of the scrape, so one first seen at 3 never read as having risen and `rate()`/`increase()` alerts missed it. - Gauges could wrap below zero: an mta pod reported 18446744073709552000 queued messages. - In a cluster, each node moved `queue.count` by its own queued and delivered messages, so the counting node read low (three waiting showed as none). With a coordinator the gauge is only the count from the store. **Tested** on the kind lab (inbuxa-k8s-lab, `metrics-test.sh` and `alerts-test.sh`): scraping with and without the password on all five pods, OTLP from every pod with the token, and every alert in the chart's rules quiet on a healthy install and firing from real breakage. trc and common unit tests pass; notice and name checks clean.
jcoffey-dev added 6 commits 2026-11-19 18:30:58 +00:00
A pod's metrics settings can come from its deployment instead of the
store, so a Helm chart can turn them on for every pod:

- INBUXA_METRICS_PROMETHEUS_USERNAME and INBUXA_METRICS_PROMETHEUS_SECRET_FILE
  open /metrics/prometheus with that username and the password in the file.
  Both or neither; the endpoint is never opened without a password this way.
- When the store has no OpenTelemetry metrics set, OTEL_EXPORTER_OTLP_ENDPOINT
  (or _METRICS_ENDPOINT) turns the exporter on from the standard OTEL_
  variables: protocol, headers, timeout, OTEL_METRIC_EXPORT_INTERVAL and
  OTEL_RESOURCE_ATTRIBUTES.

Three gauges for alerts, each a Unix time:

- queue.oldest-message: when the oldest queued message arrived, read from
  the first queue id (a snowflake) when the queue is counted, without
  reading any message. 0 when the queue is empty.
- cluster.lease-renewed: when this node last renewed its cluster lease. A
  pod that runs on but can't reach the store stops moving it. (A count of
  Stale nodes wouldn't do on Kubernetes: a pod that rolls away never gives
  its lease back, so it reads Stale for a day after every rollout.)
- tls.earliest-expiry: when the first name served runs out of certificate.
A collector's token belongs in a mounted file, not the environment, as for
the other secrets a pod reads. key=value pairs, by commas or lines.
Each process counts only its own traffic, so an exporter that pushes from
the metricsPush nodes alone leaves the rest out. One set from the
environment names its pod in the resource, so every pod pushes, whatever
its role, and the collector adds them up. The store's exporter still
follows metricsPush.
/healthz/live went through the limit for anonymous requests, which lives
in the in-memory store. Where that is the data store, a data store outage
made the check wait on it until the kubelet's probe timed out, and every
pod was restarted at once, though each process was fine. Found on the
Kubernetes lab, cutting one pod off from Postgres: it was killed within a
minute. The liveness answer is a fixed 200 and costs nothing to serve, so
it no longer waits for the limit; readiness still does.
- The queue count on a node that delivers what another queued went below
  zero and wrapped: an mta pod reported 18446744073709552000 messages.
  Gauges now stop at zero.
- /metrics/prometheus left out counters still at zero, so one first seen
  at 3 never read as having risen, and rate() and increase() alerts
  missed it. Every collected counter is in the scrape now, zeros too.
Queue gauge in a cluster: only the count from the store
github/ci (branch) GitHub Actions
ci / build (pull_request) Skipped
ci / fork-checks (pull_request) Skipped
ci / github (pull_request) Successful in 5s
4b8644710a
Each node moved queue.count by what it queued and delivered itself, so in
a cluster the node that counts the queue subtracted deliveries of
messages other nodes had queued, and read low until the next count: three
messages waiting showed as none. With a coordinator, the gauge changes
only when the queue is counted (every five minutes, by the nodes that
calculate metrics). A single server keeps its live count.
jcoffey-dev merged commit 33e493b1ee into main 2026-11-19 18:35:13 +00:00
jcoffey-dev deleted branch feat/k8s-metrics 2026-11-19 18:35:13 +00:00
jcoffey-dev referenced this issue from a commit 2026-11-20 01:59:38 +00:00
Sign in to join this conversation.