Spike: can older nodes run against the 27.1 store layout? #5

Closed
opened 2026-11-18 05:14:37 +00:00 by jcoffey-dev · 2 comments
Owner

S5 · size S · part of inbuxa for Kubernetes (tracking issue in inbuxa-charts)

Decides whether the 2026.x to 27.1 store upgrade can be rolling or needs every server pod stopped first.

Done when

  • Answer recorded with evidence, feeding K-S3 and K-C9
**S5** · size S · part of inbuxa for Kubernetes (tracking issue in inbuxa-charts) Decides whether the 2026.x to 27.1 store upgrade can be rolling or needs every server pod stopped first. **Done when** - Answer recorded with evidence, feeding K-S3 and K-C9
jcoffey-dev added this to the 27.1 milestone 2026-11-18 05:14:37 +00:00
jcoffey-dev added the
priority
p1
enhancement
labels 2026-11-18 05:14:37 +00:00
jcoffey-dev self-assigned this 2026-11-18 06:28:54 +00:00
Author
Owner

Answer (from the code, 2026-11-17): no. The 2026.x to 27.1 crossing can't be rolling.

  • 2026.x is store layout 1; 27.1 is layout 3 (crates/migration/src/layout.rs).
  • Step 1→2 is rolling-safe. Step 2→3 is not (SL-18: old nodes would keep writing the moved data, and an old binary can't open a RocksDB store with column families it doesn't know).
  • 27.1 handles this itself: one node takes the layout lease (SL-3); a step that isn't rolling-safe waits until no other node shows as Active (SL-6), and a stopped node stops showing as Active 3 minutes after its last heartbeat. 2026.x refuses a store with a newer mark (SL-4).

What this means on Kubernetes: a normal rolling update stalls. The new pod waits forever for the old pods to stop, and the old pods aren't removed because the new one never becomes ready. So for the crossing the chart must:

  1. Scale every server Deployment to 0 (pre-upgrade hook).
  2. Run one Job on the new image that does the layout upgrade and exits (K-S3). It will wait up to ~3 min for the old heartbeats to expire.
  3. Then roll out the new Deployments.

Mail is down for roughly 3 minutes plus the step's run time. Later 27.1.x upgrades are plain rolling updates unless a future step is marked not rolling-safe; the hook should read that from the server rather than hard-code it (worth a field in K-S3's output).

**Answer (from the code, 2026-11-17): no. The 2026.x to 27.1 crossing can't be rolling.** - 2026.x is store layout 1; 27.1 is layout 3 (`crates/migration/src/layout.rs`). - Step 1→2 is rolling-safe. Step 2→3 is not (SL-18: old nodes would keep writing the moved data, and an old binary can't open a RocksDB store with column families it doesn't know). - 27.1 handles this itself: one node takes the layout lease (SL-3); a step that isn't rolling-safe waits until no other node shows as Active (SL-6), and a stopped node stops showing as Active 3 minutes after its last heartbeat. 2026.x refuses a store with a newer mark (SL-4). **What this means on Kubernetes:** a normal rolling update stalls. The new pod waits forever for the old pods to stop, and the old pods aren't removed because the new one never becomes ready. So for the crossing the chart must: 1. Scale every server Deployment to 0 (pre-upgrade hook). 2. Run one Job on the new image that does the layout upgrade and exits (K-S3). It will wait up to ~3 min for the old heartbeats to expire. 3. Then roll out the new Deployments. Mail is down for roughly 3 minutes plus the step's run time. Later 27.1.x upgrades are plain rolling updates unless a future step is marked not rolling-safe; the hook should read that from the server rather than hard-code it (worth a field in K-S3's output).
Author
Owner

Folded into the checklist on inbuxa/inbuxa-charts#14 to keep the issue list short. The details stay here; reopen this when work on it starts.

Folded into the checklist on inbuxa/inbuxa-charts#14 to keep the issue list short. The details stay here; reopen this when work on it starts.
Sign in to join this conversation.