Upgrade tests with the published images (K-L5) #30

Merged
jcoffey-dev merged 1 commits from feat/upgrade-real-images into main 2026-11-20 04:10:09 +00:00
Owner

K-L5: the upgrade tests on the published images.

Rolling (rolling-upgrade-test.sh, FROM=27.1-rc.33 TO=27.1-rc.34): both from ghcr.io by digest, mail flowing through the roll (mailflow.py: a message a second, each from its own sender domain, retried like an MTA), then every accepted message checked to have arrived exactly once.
Result: rolled in 19 s; 43 accepted, 43 arrived, 0 missing, 0 twice; one connection refused while a pod went away, retried.

Crossing (crossing-test.sh, OLD=2026.11.18.3 TO=27.1-rc.34): a store made by 2026.11.18.3 (layout 1), 15 messages in through the chart's own mail Service, mailbox digests taken; the chart's upgrade refuses without acknowledgeLayout=3 (store untouched), then crosses in 194 s (steps 1→2→3), and the digests match.

Fixes found on the way:

  • fresh-store.sh's pods and seed-job.yaml now carry the chart's labels: the Kyverno policy (K-L3) refused them, since no deny-all covered them.
  • fresh-store.sh / seed-store.sh take PULL for published images.
  • mailtest.py / mailbox-digest.py use a sender domain of their own per run/message: the server's default throttle allows 25 an hour per sender domain → recipient, and repeated test runs hit it. mailbox-digest.py sends one message per session, paced under 5/s (2026.x caps messages per session).

Lab left seeded, on rc.34.

K-L5: the upgrade tests on the published images. **Rolling** (`rolling-upgrade-test.sh`, `FROM=27.1-rc.33 TO=27.1-rc.34`): both from ghcr.io by digest, mail flowing through the roll (`mailflow.py`: a message a second, each from its own sender domain, retried like an MTA), then every accepted message checked to have arrived exactly once. Result: rolled in 19 s; 43 accepted, 43 arrived, 0 missing, 0 twice; one connection refused while a pod went away, retried. **Crossing** (`crossing-test.sh`, `OLD=2026.11.18.3 TO=27.1-rc.34`): a store made by 2026.11.18.3 (layout 1), 15 messages in through the chart's own mail Service, mailbox digests taken; the chart's upgrade refuses without `acknowledgeLayout=3` (store untouched), then crosses in 194 s (steps 1→2→3), and the digests match. Fixes found on the way: - `fresh-store.sh`'s pods and `seed-job.yaml` now carry the chart's labels: the Kyverno policy (K-L3) refused them, since no deny-all covered them. - `fresh-store.sh` / `seed-store.sh` take `PULL` for published images. - `mailtest.py` / `mailbox-digest.py` use a sender domain of their own per run/message: the server's default throttle allows 25 an hour per sender domain → recipient, and repeated test runs hit it. `mailbox-digest.py` sends one message per session, paced under 5/s (2026.x caps messages per session). Lab left seeded, on rc.34.
jcoffey-dev added 1 commit 2026-11-20 04:07:48 +00:00
rolling-upgrade-test.sh: release N-1 to N from ghcr.io, pinned by digest,
with mail flowing the whole way (mailflow.py, a message a second, retried
as an MTA would) and every accepted message checked to arrive exactly
once. 27.1-rc.33 to rc.34: rolled in 19s, 43 accepted, 43 arrived, one
connection refused while a pod went away and retried.

crossing-test.sh: a store made and filled by 2026.11.18.3 (layout 1),
upgraded by the chart to 27.1-rc.34: refused until the layout change is
acknowledged, then crossed in 194s (1 to 2 to 3), mailbox digests equal.

Fixes on the way: fresh-store.sh's pods and seed-job.yaml carry the
chart's labels, so its NetworkPolicies cover them as the Kyverno policy
requires; fresh-store.sh and seed-store.sh take PULL for published images;
mailtest.py and mailbox-digest.py send from a sender domain of their own
(the default throttle allows 25 an hour per sender domain and recipient),
and mailbox-digest.py one message per session, paced.
jcoffey-dev merged commit cdc0b94e68 into main 2026-11-20 04:10:09 +00:00
jcoffey-dev deleted branch feat/upgrade-real-images 2026-11-20 04:10:09 +00:00
Sign in to join this conversation.