Metrics: scraping, alert rules, OTLP and a Grafana dashboard (K-C10) #27

Merged
jcoffey-dev merged 4 commits from feat/observability into main 2026-11-19 21:06:52 +00:00
Owner

K-C10, closes #10. Needs server 27.1-rc.34 (inbuxa/inbuxa-server#488): hold until it ships.

server.metrics, off by default:

  • prometheus: every server pod serves /metrics/prometheus behind a username and a password the pods read as a file (a Secret the chart makes and keeps, or existingSecret). A headless Service, <release>-metrics, lists every pod. serviceMonitor scrapes each one and labels its group.
  • prometheus.rules: PrometheusRules: pod down, cluster lease not renewed, queue large, a message waiting too long (both read from the core pod), deliveries bouncing (deliveryFailureRatio once deliveryFailureMinimum), certificate about to expire, slow data store, and restarts and flapping readiness from kube-state-metrics (kubeStateMetrics).
  • otlp: push to a collector; the token from headersSecret, read as a file.
  • grafanaDashboard: a ConfigMap for Grafana's sidecar.

networkPolicy.metricsFrom lets Prometheus reach the http port when httpFrom narrows it.

Tested: tools/check.sh (helm lint 4 and 3, kubeconform 1.35 to 1.37 with the CRD schemas, kube-linter, 14 new unit tests). On kind (inbuxa-k8s-lab metrics-test.sh, alerts-test.sh): Prometheus v3.5.0 with these rules was quiet on a healthy install, and each alert fired from real breakage. The two kube-state-metrics alerts weren't run: the lab has no kube-state-metrics.

K-C10, closes #10. Needs server 27.1-rc.34 (inbuxa/inbuxa-server#488): hold until it ships. `server.metrics`, off by default: - **prometheus**: every server pod serves `/metrics/prometheus` behind a username and a password the pods read as a file (a Secret the chart makes and keeps, or `existingSecret`). A headless Service, `<release>-metrics`, lists every pod. `serviceMonitor` scrapes each one and labels its group. - **prometheus.rules**: PrometheusRules: pod down, cluster lease not renewed, queue large, a message waiting too long (both read from the core pod), deliveries bouncing (`deliveryFailureRatio` once `deliveryFailureMinimum`), certificate about to expire, slow data store, and restarts and flapping readiness from kube-state-metrics (`kubeStateMetrics`). - **otlp**: push to a collector; the token from `headersSecret`, read as a file. - **grafanaDashboard**: a ConfigMap for Grafana's sidecar. `networkPolicy.metricsFrom` lets Prometheus reach the http port when `httpFrom` narrows it. **Tested**: `tools/check.sh` (helm lint 4 and 3, kubeconform 1.35 to 1.37 with the CRD schemas, kube-linter, 14 new unit tests). On kind (inbuxa-k8s-lab `metrics-test.sh`, `alerts-test.sh`): Prometheus v3.5.0 with these rules was quiet on a healthy install, and each alert fired from real breakage. The two kube-state-metrics alerts weren't run: the lab has no kube-state-metrics.
jcoffey-dev added 4 commits 2026-11-19 21:05:48 +00:00
server.metrics, off by default:
- prometheus: every server pod serves /metrics/prometheus behind a
  username and a password the pods read as a file (a Secret the chart
  makes, or existingSecret). A headless Service, <release>-metrics, lists
  every pod; a ServiceMonitor scrapes each one and labels its group.
- prometheus.rules: PrometheusRules for a pod down, a cluster lease not
  renewed, a large or stuck queue (read from the core pod, which counts the
  queue itself), deliveries bouncing, a certificate about to expire, a slow
  data store, and restarts and flapping readiness from kube-state-metrics.
- otlp: push to a collector from the standard OTEL_ variables, with each
  pod's name in the resource; a token in headersSecret, read as a file.
- grafanaDashboard: a ConfigMap for Grafana's dashboard sidecar.

networkPolicy.metricsFrom lets Prometheus reach the http port when httpFrom
narrows it. Needs server 27.1-rc.33.
Metrics need server 27.1-rc.34
ci / chart (pull_request) Successful in 44s
7a2a14fa42
jcoffey-dev force-pushed feat/observability from baa950329e to 7a2a14fa42 2026-11-19 21:05:48 +00:00 Compare
jcoffey-dev merged commit 9414b617a7 into main 2026-11-19 21:06:52 +00:00
jcoffey-dev deleted branch feat/observability 2026-11-19 21:06:52 +00:00
Sign in to join this conversation.