metrics-test and alerts-test: scraping, OTLP and the chart's alerts (K-C10) #28

Merged
jcoffey-dev merged 1 commits from feat/metrics-test into main 2026-11-19 18:35:10 +00:00
1 Commits
Author SHA1 Message Date
jcoffey-dev f2f13c3367 metrics-test and alerts-test: scraping, OTLP and the chart's alerts (K-C10)
metrics-test.sh: every server pod refuses /metrics/prometheus without the
password and with a wrong one, answers the chart's with the alert gauges,
renewed its cluster lease in the last 150 s, is listed by the headless
metrics Service, and pushes OTLP (HAProxy on the kind network stands in
for the collector) with the token from the headers Secret.

alerts-test.sh: Prometheus v3.5.0 on its own (no operator), scraping as
the chart's ServiceMonitor does, with the chart's rules and every for cut
to 0s. Quiet on a healthy install; then each alert from real breakage: a
stopped server process, a pod cut off from Postgres, mail to a host that
never answers (wait.lab.test, added to CoreDNS for the test), mail to
example.com's null MX, a certificate that runs out in two days. The store
latency alert with a threshold any read crosses. Puts everything back.

admin-jmap.py and queue-clean.py: JMAP as the lab admin, and removing
the mail a test queued.
2026-11-19 11:30:05 -07:00