Metrics for Kubernetes: scraping and OTLP from the environment, alert gauges, four metric fixes #488

Merged
jcoffey-dev merged 6 commits from feat/k8s-metrics into main 2026-11-19 18:35:13 +00:00
6 Commits
Author SHA1 Message Date
jcoffey-dev 4b8644710a Queue gauge in a cluster: only the count from the store
github/ci (branch) GitHub Actions
ci / build (pull_request) Skipped
ci / fork-checks (pull_request) Skipped
ci / github (pull_request) Successful in 5s
Each node moved queue.count by what it queued and delivered itself, so in
a cluster the node that counts the queue subtracted deliveries of
messages other nodes had queued, and read low until the next count: three
messages waiting showed as none. With a coordinator, the gauge changes
only when the queue is counted (every five minutes, by the nodes that
calculate metrics). A single server keeps its live count.
2026-11-19 10:53:45 -07:00
jcoffey-dev baf7d7ba93 Metrics: a gauge never wraps below zero; counters scraped from zero
github/ci (branch) GitHub Actions
- The queue count on a node that delivers what another queued went below
  zero and wrapped: an mta pod reported 18446744073709552000 messages.
  Gauges now stop at zero.
- /metrics/prometheus left out counters still at zero, so one first seen
  at 3 never read as having risen, and rate() and increase() alerts
  missed it. Every collected counter is in the scrape now, zeros too.
2026-11-19 10:22:11 -07:00
jcoffey-dev 09ac5d9fef Liveness answers before the anonymous request limit
github/ci (branch) GitHub Actions
/healthz/live went through the limit for anonymous requests, which lives
in the in-memory store. Where that is the data store, a data store outage
made the check wait on it until the kubelet's probe timed out, and every
pod was restarted at once, though each process was fine. Found on the
Kubernetes lab, cutting one pod off from Postgres: it was killed within a
minute. The liveness answer is a fixed 200 and costs nothing to serve, so
it no longer waits for the limit; readiness still does.
2026-11-19 09:48:40 -07:00
jcoffey-dev 5527d7fb39 OTLP from the environment: every pod pushes
github/ci (branch) GitHub Actions
Each process counts only its own traffic, so an exporter that pushes from
the metricsPush nodes alone leaves the rest out. One set from the
environment names its pod in the resource, so every pod pushes, whatever
its role, and the collector adds them up. The store's exporter still
follows metricsPush.
2026-11-19 09:01:56 -07:00
jcoffey-dev 4472616ae8 OTLP headers from a file: INBUXA_OTLP_HEADERS_FILE
github/ci (branch) GitHub Actions
A collector's token belongs in a mounted file, not the environment, as for
the other secrets a pod reads. key=value pairs, by commas or lines.
2026-11-19 08:37:42 -07:00
jcoffey-dev 9cffec3a6b Metrics for Kubernetes: scraping and OTLP from the environment, three gauges
github/ci (branch) GitHub Actions
A pod's metrics settings can come from its deployment instead of the
store, so a Helm chart can turn them on for every pod:

- INBUXA_METRICS_PROMETHEUS_USERNAME and INBUXA_METRICS_PROMETHEUS_SECRET_FILE
  open /metrics/prometheus with that username and the password in the file.
  Both or neither; the endpoint is never opened without a password this way.
- When the store has no OpenTelemetry metrics set, OTEL_EXPORTER_OTLP_ENDPOINT
  (or _METRICS_ENDPOINT) turns the exporter on from the standard OTEL_
  variables: protocol, headers, timeout, OTEL_METRIC_EXPORT_INTERVAL and
  OTEL_RESOURCE_ATTRIBUTES.

Three gauges for alerts, each a Unix time:

- queue.oldest-message: when the oldest queued message arrived, read from
  the first queue id (a snowflake) when the queue is counted, without
  reading any message. 0 when the queue is empty.
- cluster.lease-renewed: when this node last renewed its cluster lease. A
  pod that runs on but can't reach the store stops moving it. (A count of
  Stale nodes wouldn't do on Kubernetes: a pod that rolls away never gives
  its lease back, so it reads Stale for a day after every rollout.)
- tls.earliest-expiry: when the first name served runs out of certificate.
2026-11-19 08:32:12 -07:00