The soak harness (K-L7) and report.sh for RC testers (K-L9) #37

Merged
jcoffey-dev merged 3 commits from feat/soak-and-report into main 2026-11-20 09:20:59 +00:00
3 Commits
Author SHA1 Message Date
jcoffey-dev 07add6a662 Soak: stay under the server's sending limit; a failed chaos scenario fails the run
ci / check (pull_request) Successful in 19s
The trial (two 30-minute days) found both. Outbound mail went to one
address, the same one mailcheck.py uses, and ran into the default limit
of 25 an hour from one sender domain to one recipient; the drain day's
check afterwards was refused for it. Outbound now goes to soak-0 ...
[email protected] in turn. And the run read PASSED with that day's chaos
FAILED: a day is clean, and a run passes, only when every scenario
recovered.
2026-11-20 02:07:24 -07:00
jcoffey-dev 25243e84fb report.sh for release-candidate testers (K-L9)
Reads the cluster and the release and prints a block to paste into a
report: Kubernetes version and distribution, nodes' OS, kernel, runtime
and architecture, network plugin, load balancer, Pod Security level and
refusals, chart and image versions, each component's pods, warning events
and any test results (--results). Never changes anything.

No names by default: nodes, registries, addresses and hostnames are left
out, and the block is scrubbed of anything shaped like an IPv4 or IPv6
address, a MAC or a hostname (times, versions and file names survive).
--include-names keeps them. README covers the soak and the report.
2026-11-20 01:07:31 -07:00
jcoffey-dev 315e84e242 The soak: a week of steady load and daily chaos (K-L7)
lab/soak/soak.sh start runs SOAK_DAYS (7) days on an installed chart as a
systemd user unit: soak-load.py from this host through the load balancer
(mail in every 10 s, out to the sink every minute, two IMAP clients in
IDLE), three webmail tabs in a pod (chart/webmail-tabs.py, now with a
summary every minute and signing in again when its session runs out),
chart/chaos-test.sh with one scenario a day, and soak-watch.py: the sink's
log, the tabs' summaries and every pod in the namespaces each minute,
day-N.md at each day's end and report.md for the run.

It passes with no message lost or doubled and no restart outside the
chaos windows. Every count survives the load generator restarting: what
was sent is saved before the next send, and an arrival is saved, with its
IMAP UID, before it's deleted.
2026-11-20 01:07:27 -07:00