metrics-test.sh: every server pod refuses /metrics/prometheus without the password and with a wrong one, answers the chart's with the alert gauges, renewed its cluster lease in the last 150 s, is listed by the headless metrics Service, and pushes OTLP (HAProxy on the kind network stands in for the collector) with the token from the headers Secret. alerts-test.sh: Prometheus v3.5.0 on its own (no operator), scraping as the chart's ServiceMonitor does, with the chart's rules and every for cut to 0s. Quiet on a healthy install; then each alert from real breakage: a stopped server process, a pod cut off from Postgres, mail to a host that never answers (wait.lab.test, added to CoreDNS for the test), mail to example.com's null MX, a certificate that runs out in two days. The store latency alert with a threshold any read crosses. Puts everything back. admin-jmap.py and queue-clean.py: JMAP as the lab admin, and removing the mail a test queued.
24 lines
1.1 KiB
Python
Executable File
24 lines
1.1 KiB
Python
Executable File
#!/usr/bin/env python3
|
|
"""Removes the queued messages a test sent: from a sender, since a time.
|
|
|
|
queue-clean.py <server> <admin password file> <sender> <since, ISO 8601 UTC>
|
|
|
|
Prints how many it removed.
|
|
"""
|
|
import json, subprocess, sys
|
|
server, pwfile, sender, since = sys.argv[1:]
|
|
here = __file__.rsplit("/", 1)[0]
|
|
def jmap(calls):
|
|
out = subprocess.run([sys.executable, f"{here}/admin-jmap.py", server, pwfile], input=json.dumps(calls),
|
|
capture_output=True, text=True, check=True).stdout
|
|
return json.loads(out)
|
|
got = jmap([["x:QueuedMessage/query", {}, "q"],
|
|
["x:QueuedMessage/get", {"#ids": {"resultOf": "q", "name": "x:QueuedMessage/query", "path": "/ids"},
|
|
"properties": ["returnPath", "createdAt"]}, "g"]])
|
|
ids = [m["id"] for m in got[1][1]["list"] if m.get("returnPath") == sender and m.get("createdAt", "") >= since]
|
|
if ids:
|
|
res = jmap([["x:QueuedMessage/set", {"destroy": ids}, "d"]])[0][1]
|
|
if res.get("notDestroyed"):
|
|
sys.exit(f"not removed: {json.dumps(res['notDestroyed'])[:300]}")
|
|
print(len(ids))
|