For the Helm chart's metrics (K-C10, inbuxa-charts#10), and four metric bugs the lab found on the way.
Metrics from the deployment, so a chart can turn them on for every pod:
INBUXA_METRICS_PROMETHEUS_USERNAME and INBUXA_METRICS_PROMETHEUS_SECRET_FILE open /metrics/prometheus with that username and the password in the file. Both or neither: never opened without a password this way.
When the store has no OpenTelemetry metrics set, OTEL_EXPORTER_OTLP_ENDPOINT (or _METRICS_ENDPOINT) turns OTLP on from the standard OTEL_ variables (protocol, timeout, interval, resource attributes). INBUXA_OTLP_HEADERS_FILE reads the headers (a collector token) from a file. Every pod pushes its own, whatever its role: each process counts only its own traffic.
Three gauges for alerts, each a Unix time:
queue.oldest-message: when the oldest queued message arrived, from the first queue id (a snowflake), with no message read.
cluster.lease-renewed: when this node last renewed its cluster lease. (A count of Stale nodes wouldn't do on Kubernetes: a pod that rolls away never gives its lease back.)
tls.earliest-expiry: when the first name served runs out of certificate.
Fixes the lab found
/healthz/live went through the anonymous request limit, kept in the in-memory store. With the data store down the probe timed out and Kubernetes restarted every pod at once. Liveness answers before the limit now.
Counters still at zero were left out of the scrape, so one first seen at 3 never read as having risen and rate()/increase() alerts missed it.
Gauges could wrap below zero: an mta pod reported 18446744073709552000 queued messages.
In a cluster, each node moved queue.count by its own queued and delivered messages, so the counting node read low (three waiting showed as none). With a coordinator the gauge is only the count from the store.
Tested on the kind lab (inbuxa-k8s-lab, metrics-test.sh and alerts-test.sh): scraping with and without the password on all five pods, OTLP from every pod with the token, and every alert in the chart's rules quiet on a healthy install and firing from real breakage. trc and common unit tests pass; notice and name checks clean.
For the Helm chart's metrics (K-C10, inbuxa-charts#10), and four metric bugs the lab found on the way.
**Metrics from the deployment**, so a chart can turn them on for every pod:
- `INBUXA_METRICS_PROMETHEUS_USERNAME` and `INBUXA_METRICS_PROMETHEUS_SECRET_FILE` open `/metrics/prometheus` with that username and the password in the file. Both or neither: never opened without a password this way.
- When the store has no OpenTelemetry metrics set, `OTEL_EXPORTER_OTLP_ENDPOINT` (or `_METRICS_ENDPOINT`) turns OTLP on from the standard `OTEL_` variables (protocol, timeout, interval, resource attributes). `INBUXA_OTLP_HEADERS_FILE` reads the headers (a collector token) from a file. Every pod pushes its own, whatever its role: each process counts only its own traffic.
**Three gauges for alerts**, each a Unix time:
- `queue.oldest-message`: when the oldest queued message arrived, from the first queue id (a snowflake), with no message read.
- `cluster.lease-renewed`: when this node last renewed its cluster lease. (A count of Stale nodes wouldn't do on Kubernetes: a pod that rolls away never gives its lease back.)
- `tls.earliest-expiry`: when the first name served runs out of certificate.
**Fixes the lab found**
- `/healthz/live` went through the anonymous request limit, kept in the in-memory store. With the data store down the probe timed out and Kubernetes restarted every pod at once. Liveness answers before the limit now.
- Counters still at zero were left out of the scrape, so one first seen at 3 never read as having risen and `rate()`/`increase()` alerts missed it.
- Gauges could wrap below zero: an mta pod reported 18446744073709552000 queued messages.
- In a cluster, each node moved `queue.count` by its own queued and delivered messages, so the counting node read low (three waiting showed as none). With a coordinator the gauge is only the count from the store.
**Tested** on the kind lab (inbuxa-k8s-lab, `metrics-test.sh` and `alerts-test.sh`): scraping with and without the password on all five pods, OTLP from every pod with the token, and every alert in the chart's rules quiet on a healthy install and firing from real breakage. trc and common unit tests pass; notice and name checks clean.
A pod's metrics settings can come from its deployment instead of the
store, so a Helm chart can turn them on for every pod:
- INBUXA_METRICS_PROMETHEUS_USERNAME and INBUXA_METRICS_PROMETHEUS_SECRET_FILE
open /metrics/prometheus with that username and the password in the file.
Both or neither; the endpoint is never opened without a password this way.
- When the store has no OpenTelemetry metrics set, OTEL_EXPORTER_OTLP_ENDPOINT
(or _METRICS_ENDPOINT) turns the exporter on from the standard OTEL_
variables: protocol, headers, timeout, OTEL_METRIC_EXPORT_INTERVAL and
OTEL_RESOURCE_ATTRIBUTES.
Three gauges for alerts, each a Unix time:
- queue.oldest-message: when the oldest queued message arrived, read from
the first queue id (a snowflake) when the queue is counted, without
reading any message. 0 when the queue is empty.
- cluster.lease-renewed: when this node last renewed its cluster lease. A
pod that runs on but can't reach the store stops moving it. (A count of
Stale nodes wouldn't do on Kubernetes: a pod that rolls away never gives
its lease back, so it reads Stale for a day after every rollout.)
- tls.earliest-expiry: when the first name served runs out of certificate.
Each process counts only its own traffic, so an exporter that pushes from
the metricsPush nodes alone leaves the rest out. One set from the
environment names its pod in the resource, so every pod pushes, whatever
its role, and the collector adds them up. The store's exporter still
follows metricsPush.
/healthz/live went through the limit for anonymous requests, which lives
in the in-memory store. Where that is the data store, a data store outage
made the check wait on it until the kubelet's probe timed out, and every
pod was restarted at once, though each process was fine. Found on the
Kubernetes lab, cutting one pod off from Postgres: it was killed within a
minute. The liveness answer is a fixed 200 and costs nothing to serve, so
it no longer waits for the limit; readiness still does.
- The queue count on a node that delivers what another queued went below
zero and wrapped: an mta pod reported 18446744073709552000 messages.
Gauges now stop at zero.
- /metrics/prometheus left out counters still at zero, so one first seen
at 3 never read as having risen, and rate() and increase() alerts
missed it. Every collected counter is in the scrape now, zeros too.
Each node moved queue.count by what it queued and delivered itself, so in
a cluster the node that counts the queue subtracted deliveries of
messages other nodes had queued, and read low until the next count: three
messages waiting showed as none. With a coordinator, the gauge changes
only when the queue is counted (every five minutes, by the nodes that
calculate metrics). A single server keeps its live count.
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
For the Helm chart's metrics (K-C10, inbuxa-charts#10), and four metric bugs the lab found on the way.
Metrics from the deployment, so a chart can turn them on for every pod:
INBUXA_METRICS_PROMETHEUS_USERNAMEandINBUXA_METRICS_PROMETHEUS_SECRET_FILEopen/metrics/prometheuswith that username and the password in the file. Both or neither: never opened without a password this way.OTEL_EXPORTER_OTLP_ENDPOINT(or_METRICS_ENDPOINT) turns OTLP on from the standardOTEL_variables (protocol, timeout, interval, resource attributes).INBUXA_OTLP_HEADERS_FILEreads the headers (a collector token) from a file. Every pod pushes its own, whatever its role: each process counts only its own traffic.Three gauges for alerts, each a Unix time:
queue.oldest-message: when the oldest queued message arrived, from the first queue id (a snowflake), with no message read.cluster.lease-renewed: when this node last renewed its cluster lease. (A count of Stale nodes wouldn't do on Kubernetes: a pod that rolls away never gives its lease back.)tls.earliest-expiry: when the first name served runs out of certificate.Fixes the lab found
/healthz/livewent through the anonymous request limit, kept in the in-memory store. With the data store down the probe timed out and Kubernetes restarted every pod at once. Liveness answers before the limit now.rate()/increase()alerts missed it.queue.countby its own queued and delivered messages, so the counting node read low (three waiting showed as none). With a coordinator the gauge is only the count from the store.Tested on the kind lab (inbuxa-k8s-lab,
metrics-test.shandalerts-test.sh): scraping with and without the password on all five pods, OTLP from every pod with the token, and every alert in the chart's rules quiet on a healthy install and firing from real breakage. trc and common unit tests pass; notice and name checks clean.