The soak harness (K-L7) and report.sh for RC testers (K-L9) #37

Merged
jcoffey-dev merged 3 commits from feat/soak-and-report into main 2026-11-20 09:20:59 +00:00
Owner

K-L7 (#14): lab/soak/soak.sh start. SOAK_DAYS days (7) of steady load on an installed chart, run as a systemd user unit (status, stop; start resumes):

  • soak-load.py, from the host through the load balancer: mail in every 10 s (a sender domain each), out every minute over submission to [email protected], and two IMAP clients in IDLE.
  • Three webmail tabs in a pod (chart/webmail-tabs.py, now with a summary a minute and signing in again when the 12 h session runs out).
  • One chart/chaos-test.sh scenario a day, two hours in.
  • soak-watch.py collects every minute: the sink's log, the tabs, and every pod in inbuxa, data, lab-sink and lab-soak. It writes day-N.md at each day's end and report.md for the run.

A day is clean, and the run passes, with nothing lost or doubled, no restart or new pod outside a chaos window, and every chaos scenario recovered. Every count survives the load generator restarting: sends are saved before the next one, and arrivals are saved with their IMAP UID before they're deleted.

Trial (two 30-minute days, pod-kill then drain), which found and fixed three bugs:

  • A lock deadlock in the load generator.
  • Restarts resending a number and faking duplicates.
  • Outbound mail running into the server's 25-an-hour limit, which also failed the drain day's client check, while the run still read PASSED.

The trial's numbers: 362 in, 0 lost or doubled; 26 out, all at the sink; 1,114 IDLE notices; 803 webmail requests, 0 failed; 6 restarts, all during chaos. A 7-day rehearsal on the kind lab started 11-20. The January run on k3s VMs needs the k3s download OK'd.

K-L9 (#16): lab/report.sh. Prints a block to paste into an RC report: Kubernetes version and distribution, nodes' OS, kernel, runtime and architecture, network plugin, load balancer, Pod Security level and refusals, chart and image versions, each component's pods, warning events, and test results (--results). It only reads. By default there are no names (nodes, registries, release and namespace) and the block is scrubbed of IPv4, IPv6, MACs and hostnames, while times, versions and file names survive. --include-names keeps them. Reviewed on the lab, and the scrub was tested against a results file of addresses and hostnames.

**K-L7 (#14): `lab/soak/soak.sh start`.** SOAK_DAYS days (7) of steady load on an installed chart, run as a systemd user unit (`status`, `stop`; `start` resumes): - `soak-load.py`, from the host through the load balancer: mail in every 10 s (a sender domain each), out every minute over submission to `[email protected]`, and two IMAP clients in IDLE. - Three webmail tabs in a pod (`chart/webmail-tabs.py`, now with a summary a minute and signing in again when the 12 h session runs out). - One `chart/chaos-test.sh` scenario a day, two hours in. - `soak-watch.py` collects every minute: the sink's log, the tabs, and every pod in inbuxa, data, lab-sink and lab-soak. It writes `day-N.md` at each day's end and `report.md` for the run. A day is clean, and the run passes, with nothing lost or doubled, no restart or new pod outside a chaos window, and every chaos scenario recovered. Every count survives the load generator restarting: sends are saved before the next one, and arrivals are saved with their IMAP UID before they're deleted. **Trial** (two 30-minute days, pod-kill then drain), which found and fixed three bugs: - A lock deadlock in the load generator. - Restarts resending a number and faking duplicates. - Outbound mail running into the server's 25-an-hour limit, which also failed the drain day's client check, while the run still read PASSED. The trial's numbers: 362 in, 0 lost or doubled; 26 out, all at the sink; 1,114 IDLE notices; 803 webmail requests, 0 failed; 6 restarts, all during chaos. **A 7-day rehearsal on the kind lab started 11-20**. The January run on k3s VMs needs the k3s download OK'd. **K-L9 (#16): `lab/report.sh`.** Prints a block to paste into an RC report: Kubernetes version and distribution, nodes' OS, kernel, runtime and architecture, network plugin, load balancer, Pod Security level and refusals, chart and image versions, each component's pods, warning events, and test results (`--results`). It only reads. By default there are no names (nodes, registries, release and namespace) and the block is scrubbed of IPv4, IPv6, MACs and hostnames, while times, versions and file names survive. `--include-names` keeps them. Reviewed on the lab, and the scrub was tested against a results file of addresses and hostnames.
jcoffey-dev added 3 commits 2026-11-20 09:07:56 +00:00
lab/soak/soak.sh start runs SOAK_DAYS (7) days on an installed chart as a
systemd user unit: soak-load.py from this host through the load balancer
(mail in every 10 s, out to the sink every minute, two IMAP clients in
IDLE), three webmail tabs in a pod (chart/webmail-tabs.py, now with a
summary every minute and signing in again when its session runs out),
chart/chaos-test.sh with one scenario a day, and soak-watch.py: the sink's
log, the tabs' summaries and every pod in the namespaces each minute,
day-N.md at each day's end and report.md for the run.

It passes with no message lost or doubled and no restart outside the
chaos windows. Every count survives the load generator restarting: what
was sent is saved before the next send, and an arrival is saved, with its
IMAP UID, before it's deleted.
Reads the cluster and the release and prints a block to paste into a
report: Kubernetes version and distribution, nodes' OS, kernel, runtime
and architecture, network plugin, load balancer, Pod Security level and
refusals, chart and image versions, each component's pods, warning events
and any test results (--results). Never changes anything.

No names by default: nodes, registries, addresses and hostnames are left
out, and the block is scrubbed of anything shaped like an IPv4 or IPv6
address, a MAC or a hostname (times, versions and file names survive).
--include-names keeps them. README covers the soak and the report.
The trial (two 30-minute days) found both. Outbound mail went to one
address, the same one mailcheck.py uses, and ran into the default limit
of 25 an hour from one sender domain to one recipient; the drain day's
check afterwards was refused for it. Outbound now goes to soak-0 ...
[email protected] in turn. And the run read PASSED with that day's chaos
FAILED: a day is clean, and a run passes, only when every scenario
recovered.
jcoffey-dev merged commit 7b5ea5d81d into main 2026-11-20 09:20:59 +00:00
jcoffey-dev deleted branch feat/soak-and-report 2026-11-20 09:20:59 +00:00
Sign in to join this conversation.