Chaos tests: five faults with mail flowing, nothing lost or doubled (K-L4) #35

Merged
jcoffey-dev merged 1 commits from feat/chaos into main 2026-11-20 07:18:47 +00:00
Owner

Closes K-L4 (#11). lab/chart/chaos-test.sh: with mailflow.py sending a message a second (retrying like an MTA when refused), each fault happens, the lab recovers, and every message the server accepted has to arrive exactly once. mailcheck.sh runs after the lot.

Scenario How Bench
pod-kill an mta and a front pod deleted with no grace 34 accepted, 34 arrived once
drain the worker with most server pods drained (never the PostgreSQL primary's or Garage's, whose data is an emptyDir in the lab), then uncordoned 49 / 49
pg-failover the primary deleted; CloudNativePG promotes the replica 200 / 200
nats-restart rollout restart of the StatefulSet 92 / 92
s3-outage Garage frozen 60 s with SIGSTOP (requests hang, like a stuck S3), then SIGCONT 31 / 31, twice

A cordoned node and a frozen Garage are put back on any exit.

One finding, in the test client, not the server: the first S3 run had one message twice. mailflow.py gave up on the reply to the end of DATA after 20 s while S3 was frozen and sent again: the duplicate SMTP allows when a sender stops waiting, which is why RFC 5321 §4.5.3.2 gives senders 10 minutes for that reply. It now waits like an MTA (20 s to connect, 5 minutes for a reply); two more runs had no retries and no duplicates, the server holding its answer through the freeze. The rolling-upgrade test uses the same client, so it's more realistic too.

CI: a chaos job in e2e.yml (CHAOS=1 lab/e2e.sh): nightly, by hand and on the lab's own pushes, newest minor, amd64 and arm64. Not on chart branches, which stay at ~4 minutes.

Closes K-L4 (#11). `lab/chart/chaos-test.sh`: with `mailflow.py` sending a message a second (retrying like an MTA when refused), each fault happens, the lab recovers, and every message the server accepted has to arrive **exactly once**. `mailcheck.sh` runs after the lot. | Scenario | How | Bench | |---|---|---| | pod-kill | an mta and a front pod deleted with no grace | 34 accepted, 34 arrived once | | drain | the worker with most server pods drained (never the PostgreSQL primary's or Garage's, whose data is an emptyDir in the lab), then uncordoned | 49 / 49 | | pg-failover | the primary deleted; CloudNativePG promotes the replica | 200 / 200 | | nats-restart | `rollout restart` of the StatefulSet | 92 / 92 | | s3-outage | Garage frozen 60 s with SIGSTOP (requests hang, like a stuck S3), then SIGCONT | 31 / 31, twice | A cordoned node and a frozen Garage are put back on any exit. **One finding, in the test client, not the server:** the first S3 run had one message twice. `mailflow.py` gave up on the reply to the end of DATA after 20 s while S3 was frozen and sent again: the duplicate SMTP allows when a sender stops waiting, which is why RFC 5321 §4.5.3.2 gives senders 10 minutes for that reply. It now waits like an MTA (20 s to connect, 5 minutes for a reply); two more runs had no retries and no duplicates, the server holding its answer through the freeze. The rolling-upgrade test uses the same client, so it's more realistic too. **CI:** a `chaos` job in `e2e.yml` (`CHAOS=1 lab/e2e.sh`): nightly, by hand and on the lab's own pushes, newest minor, amd64 and arm64. Not on chart branches, which stay at ~4 minutes.
jcoffey-dev added 1 commit 2026-11-20 07:11:53 +00:00
lab/chart/chaos-test.sh, with mailflow.py sending a message a second:
an mta and a front pod killed outright, a worker drained (not the one
with the PostgreSQL primary or Garage) and uncordoned, the PostgreSQL
primary deleted for CloudNativePG to fail over, NATS restarted, and
Garage frozen for 60 s with SIGSTOP. Every accepted message has to
arrive exactly once; mailcheck.sh runs after the lot.

mailflow.py now waits for replies as an MTA does (RFC 5321 4.5.3.2):
20 s to connect, 5 minutes for a reply. With 20 s for everything it gave
up on the end-of-DATA reply while S3 was frozen and sent again, which
made a duplicate; waiting, the server answers once.

CI: a chaos job in e2e.yml, nightly, by hand and on the lab's pushes,
newest minor, amd64 and arm64 (CHAOS=1 lab/e2e.sh). Not on chart
branches, whose check stays quick.
jcoffey-dev merged commit 9dafbeeee1 into main 2026-11-20 07:18:47 +00:00
jcoffey-dev deleted branch feat/chaos 2026-11-20 07:18:47 +00:00
Sign in to join this conversation.