server.upgrade: the store's upgrade before the rollout (K-C9) #17

Merged
jcoffey-dev merged 1 commits from feat/upgrade-hooks into main 2026-11-18 20:47:16 +00:00
Owner

K-C9 (tracker #14). Needs server #457 (--upgrade-plan) in the image, so 27.1-rc.32 or later.

Why: spike S5 found that the 2026.x → 27.1 crossing includes a layout step that can't run with older pods up. A rolling update stalls: the new pod waits for the old ones to stop, and they're never removed because the new one never gets ready.

What it does: a pre-upgrade,pre-rollback Job on the new image, running files/upgrade.sh:

  1. inbuxa --upgrade-plan reports the layout steps the upgrade would run.
  2. With no steps, it exits at once and Helm does a normal rolling update.
  3. A layout change is refused until server.upgrade.acknowledgeLayout names the target version (3 for the 27.1 crossing). The refusal says so: going back means a backup taken first. This is the crossing guard, and it covers any later layout change too.
  4. When a step needs every node stopped, it scales this release's server Deployments to 0 and waits for their pods to go. Mail is down until the rollout. With stopServers: false it skips that, and the server waits for them, failing if they don't stop.
  5. inbuxa --upgrade-store, then Helm rolls out the new pods at their usual sizes.
  6. On a rollback, the plan refuses a store the older release can't read.

Rights: a ServiceAccount and Role that exist only as hooks, limited to deployments/scale on this release's three Deployments plus pods list. They're deleted when the hook succeeds. After a failure they stay until the next upgrade, like the failed Job's log.

Only sh, sed and curl: the image has nothing more, so there's no kubectl image to pin. One finding: the API server pretty-prints JSON for curl ("items": []), so the first version's wait never saw the pods gone. Fixed, and the pattern is checked against both forms.

Tested on kind (inbuxa-k8s-lab lab/chart/upgrade-test.sh, image from #457):

  1. No layout change: upgraded in 7 s, mail works.
  2. Layout 2 → 3, not acknowledged: refused with the plan and the --set to use. The store stays at 2 and the same pods stay up.
  3. Acknowledged: servers scaled to 0, store upgraded to 3, and the Deployments came back at 2/2/1. 189 s in all, mostly the server waiting about 3 min for the old heartbeats to expire (as S5 said). Mail works.

tools/check.sh: 37/37 unit tests, lint clean.

For the docs (K-D6): helm upgrade needs --timeout 20m or so. Helm's 5 m default fails a layout step that waits for heartbeats.

K-C9 (tracker #14). Needs server #457 (`--upgrade-plan`) in the image, so 27.1-rc.32 or later. **Why:** spike S5 found that the 2026.x → 27.1 crossing includes a layout step that can't run with older pods up. A rolling update stalls: the new pod waits for the old ones to stop, and they're never removed because the new one never gets ready. **What it does:** a `pre-upgrade,pre-rollback` Job on the new image, running `files/upgrade.sh`: 1. `inbuxa --upgrade-plan` reports the layout steps the upgrade would run. 2. With no steps, it exits at once and Helm does a normal rolling update. 3. A layout change is refused until `server.upgrade.acknowledgeLayout` names the target version (3 for the 27.1 crossing). The refusal says so: going back means a backup taken first. This is the crossing guard, and it covers any later layout change too. 4. When a step needs every node stopped, it scales this release's server Deployments to 0 and waits for their pods to go. Mail is down until the rollout. With `stopServers: false` it skips that, and the server waits for them, failing if they don't stop. 5. `inbuxa --upgrade-store`, then Helm rolls out the new pods at their usual sizes. 6. On a rollback, the plan refuses a store the older release can't read. **Rights:** a ServiceAccount and Role that exist only as hooks, limited to `deployments/scale` on this release's three Deployments plus `pods` list. They're deleted when the hook succeeds. After a failure they stay until the next upgrade, like the failed Job's log. **Only `sh`, `sed` and `curl`:** the image has nothing more, so there's no kubectl image to pin. One finding: the API server pretty-prints JSON for curl (`"items": []`), so the first version's wait never saw the pods gone. Fixed, and the pattern is checked against both forms. **Tested on kind** (inbuxa-k8s-lab `lab/chart/upgrade-test.sh`, image from #457): 1. No layout change: upgraded in 7 s, mail works. 2. Layout 2 → 3, not acknowledged: refused with the plan and the `--set` to use. The store stays at 2 and the same pods stay up. 3. Acknowledged: servers scaled to 0, store upgraded to 3, and the Deployments came back at 2/2/1. 189 s in all, mostly the server waiting about 3 min for the old heartbeats to expire (as S5 said). Mail works. `tools/check.sh`: 37/37 unit tests, lint clean. **For the docs (K-D6):** `helm upgrade` needs `--timeout 20m` or so. Helm's 5 m default fails a layout step that waits for heartbeats.
jcoffey-dev added 1 commit 2026-11-18 20:44:05 +00:00
Before an upgrade or rollback rolls out, a Job on the new image asks the
server what it does to the store (--upgrade-plan). A layout change waits
for upgrade.acknowledgeLayout to name its version: the 27.1 crossing and
any later one. When a step can't run with older pods up, the Job scales
the server Deployments to 0 and waits for their pods, which a rolling
update can't do; then --upgrade-store upgrades the store and Helm rolls
the new pods out at their usual sizes.

The scaling rights belong to a ServiceAccount and Role that exist only
while the hook runs, limited to this release's server Deployments. The
script (files/upgrade.sh) uses only sh, sed and curl, which the image
has.
jcoffey-dev merged commit 61d25b59c5 into main 2026-11-18 20:47:16 +00:00
jcoffey-dev deleted branch feat/upgrade-hooks 2026-11-18 20:47:16 +00:00
Sign in to join this conversation.