Metrics: scraping, alert rules, OTLP and a Grafana dashboard (K-C10) #27

Merged
jcoffey-dev merged 4 commits from feat/observability into main 2026-11-19 21:06:52 +00:00
4 Commits
Author SHA1 Message Date
jcoffey-dev 7a2a14fa42 Metrics need server 27.1-rc.34
ci / chart (pull_request) Successful in 44s
2026-11-19 14:05:18 -07:00
jcoffey-dev 45d9bda6e4 rules: deliveryFailureMinimum, the bounces before the bounce alert 2026-11-19 14:05:18 -07:00
jcoffey-dev 2c687e7369 values: every pod pushes OTLP 2026-11-19 14:05:18 -07:00
jcoffey-dev 7fe3e8ea7b Metrics: scraping, alert rules, OTLP and a Grafana dashboard (K-C10)
server.metrics, off by default:
- prometheus: every server pod serves /metrics/prometheus behind a
  username and a password the pods read as a file (a Secret the chart
  makes, or existingSecret). A headless Service, <release>-metrics, lists
  every pod; a ServiceMonitor scrapes each one and labels its group.
- prometheus.rules: PrometheusRules for a pod down, a cluster lease not
  renewed, a large or stuck queue (read from the core pod, which counts the
  queue itself), deliveries bouncing, a certificate about to expire, a slow
  data store, and restarts and flapping readiness from kube-state-metrics.
- otlp: push to a collector from the standard OTEL_ variables, with each
  pod's name in the resource; a token in headersSecret, read as a file.
- grafanaDashboard: a ConfigMap for Grafana's dashboard sidecar.

networkPolicy.metricsFrom lets Prometheus reach the http port when httpFrom
narrows it. Needs server 27.1-rc.33.
2026-11-19 14:05:18 -07:00