Closes K-L4 (#11). lab/chart/chaos-test.sh: with mailflow.py sending a message a second (retrying like an MTA when refused), each fault happens, the lab recovers, and every message the server accepted has to arrive exactly once. mailcheck.sh runs after the lot.
Scenario
How
Bench
pod-kill
an mta and a front pod deleted with no grace
34 accepted, 34 arrived once
drain
the worker with most server pods drained (never the PostgreSQL primary's or Garage's, whose data is an emptyDir in the lab), then uncordoned
49 / 49
pg-failover
the primary deleted; CloudNativePG promotes the replica
200 / 200
nats-restart
rollout restart of the StatefulSet
92 / 92
s3-outage
Garage frozen 60 s with SIGSTOP (requests hang, like a stuck S3), then SIGCONT
31 / 31, twice
A cordoned node and a frozen Garage are put back on any exit.
One finding, in the test client, not the server: the first S3 run had one message twice. mailflow.py gave up on the reply to the end of DATA after 20 s while S3 was frozen and sent again: the duplicate SMTP allows when a sender stops waiting, which is why RFC 5321 §4.5.3.2 gives senders 10 minutes for that reply. It now waits like an MTA (20 s to connect, 5 minutes for a reply); two more runs had no retries and no duplicates, the server holding its answer through the freeze. The rolling-upgrade test uses the same client, so it's more realistic too.
CI: a chaos job in e2e.yml (CHAOS=1 lab/e2e.sh): nightly, by hand and on the lab's own pushes, newest minor, amd64 and arm64. Not on chart branches, which stay at ~4 minutes.
Closes K-L4 (#11). `lab/chart/chaos-test.sh`: with `mailflow.py` sending a message a second (retrying like an MTA when refused), each fault happens, the lab recovers, and every message the server accepted has to arrive **exactly once**. `mailcheck.sh` runs after the lot.
| Scenario | How | Bench |
|---|---|---|
| pod-kill | an mta and a front pod deleted with no grace | 34 accepted, 34 arrived once |
| drain | the worker with most server pods drained (never the PostgreSQL primary's or Garage's, whose data is an emptyDir in the lab), then uncordoned | 49 / 49 |
| pg-failover | the primary deleted; CloudNativePG promotes the replica | 200 / 200 |
| nats-restart | `rollout restart` of the StatefulSet | 92 / 92 |
| s3-outage | Garage frozen 60 s with SIGSTOP (requests hang, like a stuck S3), then SIGCONT | 31 / 31, twice |
A cordoned node and a frozen Garage are put back on any exit.
**One finding, in the test client, not the server:** the first S3 run had one message twice. `mailflow.py` gave up on the reply to the end of DATA after 20 s while S3 was frozen and sent again: the duplicate SMTP allows when a sender stops waiting, which is why RFC 5321 §4.5.3.2 gives senders 10 minutes for that reply. It now waits like an MTA (20 s to connect, 5 minutes for a reply); two more runs had no retries and no duplicates, the server holding its answer through the freeze. The rolling-upgrade test uses the same client, so it's more realistic too.
**CI:** a `chaos` job in `e2e.yml` (`CHAOS=1 lab/e2e.sh`): nightly, by hand and on the lab's own pushes, newest minor, amd64 and arm64. Not on chart branches, which stay at ~4 minutes.
lab/chart/chaos-test.sh, with mailflow.py sending a message a second:
an mta and a front pod killed outright, a worker drained (not the one
with the PostgreSQL primary or Garage) and uncordoned, the PostgreSQL
primary deleted for CloudNativePG to fail over, NATS restarted, and
Garage frozen for 60 s with SIGSTOP. Every accepted message has to
arrive exactly once; mailcheck.sh runs after the lot.
mailflow.py now waits for replies as an MTA does (RFC 5321 4.5.3.2):
20 s to connect, 5 minutes for a reply. With 20 s for everything it gave
up on the end-of-DATA reply while S3 was frozen and sent again, which
made a duplicate; waiting, the server answers once.
CI: a chaos job in e2e.yml, nightly, by hand and on the lab's pushes,
newest minor, amd64 and arm64 (CHAOS=1 lab/e2e.sh). Not on chart
branches, whose check stays quick.
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Closes K-L4 (#11).
lab/chart/chaos-test.sh: withmailflow.pysending a message a second (retrying like an MTA when refused), each fault happens, the lab recovers, and every message the server accepted has to arrive exactly once.mailcheck.shruns after the lot.rollout restartof the StatefulSetA cordoned node and a frozen Garage are put back on any exit.
One finding, in the test client, not the server: the first S3 run had one message twice.
mailflow.pygave up on the reply to the end of DATA after 20 s while S3 was frozen and sent again: the duplicate SMTP allows when a sender stops waiting, which is why RFC 5321 §4.5.3.2 gives senders 10 minutes for that reply. It now waits like an MTA (20 s to connect, 5 minutes for a reply); two more runs had no retries and no duplicates, the server holding its answer through the freeze. The rolling-upgrade test uses the same client, so it's more realistic too.CI: a
chaosjob ine2e.yml(CHAOS=1 lab/e2e.sh): nightly, by hand and on the lab's own pushes, newest minor, amd64 and arm64. Not on chart branches, which stay at ~4 minutes.