Chaos tests: five faults with mail flowing, nothing lost or doubled (K-L4) #35

Merged
jcoffey-dev merged 1 commits from feat/chaos into main 2026-11-20 07:18:47 +00:00
1 Commits
Author SHA1 Message Date
jcoffey-dev 7392b60316 Chaos tests: five faults with mail flowing, nothing lost or doubled (K-L4)
ci / check (pull_request) Successful in 42s
lab/chart/chaos-test.sh, with mailflow.py sending a message a second:
an mta and a front pod killed outright, a worker drained (not the one
with the PostgreSQL primary or Garage) and uncordoned, the PostgreSQL
primary deleted for CloudNativePG to fail over, NATS restarted, and
Garage frozen for 60 s with SIGSTOP. Every accepted message has to
arrive exactly once; mailcheck.sh runs after the lot.

mailflow.py now waits for replies as an MTA does (RFC 5321 4.5.3.2):
20 s to connect, 5 minutes for a reply. With 20 s for everything it gave
up on the end-of-DATA reply while S3 was frozen and sent again, which
made a duplicate; waiting, the server answers once.

CI: a chaos job in e2e.yml, nightly, by hand and on the lab's pushes,
newest minor, amd64 and arm64 (CHAOS=1 lab/e2e.sh). Not on chart
branches, whose check stays quick.
2026-11-20 00:03:14 -07:00