Files
inbuxa-server/docs/spec/features/store-layout.md
jcoffey-dev 2358f4814c
ci / fork-checks (pull_request) Skipped
ci / build (pull_request) Skipped
github/ci (branch) GitHub Actions
ci / github (pull_request) Successful in 9m5s
Cluster: one node at a time does the singleton roles' work (#437)
Every node with taskScheduler queued its own copy of each scheduled
maintenance task, and every node with metricsCalculate evaluated alerts
and sent their emails, so each role had to be on exactly one node, and
losing that node paused the work until it came back.

Each role now has a lease in the store (Xs, Xm in `_`), taken with a
compare-and-swap like the store layout lease. Every node with the role
tries for it each minute; the holder renews it and does the work. A
lease lasts 3 minutes and its holder stops acting 15 seconds before it
runs out, so another node with the role takes over within about 4
minutes of the holder stopping. A clean stop gives the leases up, so a
restart or a roll hands over at the next round. A node without a
cluster coordinator needs no lease: the role alone decides, as before.

The firing alerts are stored too (Xa), so a node that takes over alerts
doesn't send the ones still firing again, and it sets the cluster-wide
gauges before its first evaluation. The daily account and domain counts
no longer wait a day on a node that just took the role over.

New events cluster.singleton-lease-taken and -lost (665, 666), also in
the served schema. `X` is registered in the `_` owners table; the spec's
copy of that table gains the N (passkeys) and O (listener status) rows
it was missing.
2026-11-21 01:07:14 -07:00

11 KiB

Feature spec: inbuxa's own store layout, changed in steps

Status: approved direction, 2026-11-08 (John: "follow your recommendations"). Not a rebuild (SPEC.md §4): inbuxa's own design.

Provenance

Source Used for
The store-independence decision (2026-10-29) and docs/spec/import-0.16.md inbuxa owns its layout from 27.1. The Stalwart 0.16.27 layout is a frozen import format, not a promise
This server's store mark (INBUXA_STORE_VERSION, key Vs in _, crates/migration) Where a store's layout version lives, and the refusal of a newer one
This server's in-memory store on SQL (crates/store/src/dispatch/lookup.rs) and the fork's subspace (SUBSPACE_INBUXA) The two changes this spec makes

Nothing here comes from Enterprise code or from upstream's 1.0 layout.

What it is

Today a store is at layout version 1, which is the 0.16 layout plus the fork's _ subspace. This spec adds the machinery to move a store from one layout version to the next in place, at boot. It also defines the first two steps:

  • 1 → 2: rate-limit counters on PostgreSQL, MySQL and SQLite keep their expiry in the counter's own row.
  • 2 → 3: the fork's large data (the audit log, the journal, holds and undelete notes) moves out of _ into subspaces of its own.

A layout step stops rollback to any build that reads only an older version. So a step is a breaking change and ships at a series: 27.1 if it is ready before January, else 27.7. Release candidates may carry it before then.

Requirements

The runner

  • SL-1. crates/migration holds one function per step, keyed by the version it starts from. At boot the runner reads the mark, treats a missing mark on an existing store as 1, and runs every step from there up to INBUXA_STORE_VERSION. It writes the mark right after each step's last write; SL-2 makes a step that stopped before the mark safe to run again. A new install is marked at INBUXA_STORE_VERSION straight away. The steps are in crates/migration/src/layout.rs.
  • SL-2. Every step can run twice safely. A step interrupted half-way runs again from the start on the next boot and comes to the same result.
  • SL-3. One node runs the steps. Before the first step a node takes a lease: an AssertValue compare-and-set on key Vm in _, holding the node id and an expiry 10 minutes ahead, renewed every minute while steps run. A node that can't take the lease waits, re-reads the mark every 10 seconds, and goes on booting once the mark reaches its own version. A lease left by a node that died expires and the next node takes it over. A node also takes over a lease held under its own node id, which a run of its own that stopped left behind. A node whose renewal finds the lease taken stops before writing the mark.
  • SL-4. A newer mark than the build's own is refused before anything is written, as now.
  • SL-5. A running node re-reads the mark every 60 seconds. When it finds a version above its own, it logs why, stops taking connections and exits non-zero. Then a node left on the old build during an upgrade can't keep writing the old layout. This ships in the release before the first step that isn't rolling-safe (SL-6). It ships with the runner.
  • SL-6. Each step says whether it is rolling-safe: whether nodes on the build before it can keep running, unharmed, against the new layout. A step that isn't refuses to start while another node shows as active (cluster_node_list, status Active). A stopped node stops showing as active three minutes after its last heartbeat (UNRESPONSIVE_NODE_TIMEOUT). The step waits, re-checking every 10 seconds, and logs every minute that the other nodes must be stopped first. Only other node ids on other hosts count, so a single-node install never waits, even on an id of its own from before a restart.
  • SL-7. Each step logs when it starts and when it ends, with counts of what it changed. The startup log states the store's layout version.
  • SL-8. Before a step runs, the server logs that the upgrade can't be rolled back without a backup, and where the docs are. The upgrade docs tell operators to take a database dump right before the upgrade, because the nightly dump isn't enough.

Reading what a newer build wrote

  • SL-9. Stored lists and map keys of enum values (permissions, events, and others) skip a value the build doesn't know instead of failing the whole object. This is needed before the first step, and on release/2026, so that a rollback still loads its roles. It shipped as its own change (registry Pickle::unpickle_item).

Step 1 → 2: counters keep their own expiry (SQL backends)

  • SL-10. On PostgreSQL, MySQL and SQLite, the counter table y gets a nullable expiry column e. A counter with an expiry is written as one row, (k, v, e), instead of a counter row in y plus an expiry marker row in m. RocksDB and FoundationDB keep today's form, and the step does nothing there except write the mark.
  • SL-11. The column is added at boot by every build that has it, before any step runs, so a node is never without it. The step fills e from each counter's marker (an upsert adding 0 and keeping the later expiry), then deletes those markers. A counter whose marker has already run out is deleted with it. Locks, greylist entries, Sieve duplicate ids and other keys in m stay where they are.
  • SL-12. The purge deletes expired counters with one statement per table (DELETE FROM y WHERE e <= now). Rows with a NULL e, written by a node on the old build during a roll, are purged the old way.
  • SL-13. Rolling-safe. An old node's counter upsert still works with the column there.
  • SL-14. Also, on every backend: write the expiry marker only when the counter's bucket is new, not on every increment. That half of the gain needs no step and may land first. Each process remembers the markers it has written; an increment leaves the marker out only once that write is in and its expiry is at least the one asked for. A counter that comes back equal to the amount added was just created, so if its marker was left out (another node reset or purged the bucket) it is written straight after: a counter is never left without a marker, which would keep it for good.

Step 2 → 3: the fork's data gets its own subspaces

  • SL-15. Subspaces get names, not only bytes. Each one has an id byte and a name, and the SQL table and RocksDB column family come from the name. Upstream's subspaces keep their one-letter names, so nothing changes for them. New ones take id bytes from 0x80 up, which are never letters, and names starting ib_. Every place that builds a table or column family name from the byte uses the name instead.
  • SL-16. The audit log, the journal, holds and undelete notes each move to a subspace of their own, dropping the feature prefix byte from each key. The small single keys (the store mark, the lease, settings marks) stay in _. The step's design first lists every key prefix in _ and its owner. A prefix nobody claims stops the step.
  • SL-17. For each moved prefix the step copies the rows, compares counts, and for the audit log also compares the chain head, before it deletes the originals. Any mismatch aborts the step with nothing deleted. On PostgreSQL the copy is done server-side (INSERT … SELECT).
  • SL-18. Not rolling-safe (SL-6). On RocksDB an old binary can't open a store with column families it doesn't know.
  • SL-19. The moved data travels in exports in a family of its own (compliance), included by default, so a configuration export can leave it out. Importing an export taken at version 1 or 2 runs the steps after the import.
  • SL-20. The legacy v and c column families, which nothing writes, are dropped in the same step.

The keys in _ and their owners (SL-16)

Every key in _ starts with a byte naming its feature. The step lists the two-byte prefixes in _, reading one key per prefix, and stops on any it can't place, before it writes anything. The table is OWNERS in crates/migration/src/fork_subspaces.rs.

First byte Owner At step 2 → 3
A AI limits stays
B first boot defaults stays
C account snapshots stays
D deliverability reports stays
E inbuxa relay stays
F front ends stays
G scheduled reports stays
H legal holds (h, o, e) moves to ib_holds (0x82)
I migrations stays
J journal: entries (e, c, p, t, x, h, f), archive (o, w) moves to ib_journal (0x81); the journals themselves (j) are settings and stay
K account locks stays
L audit log (e, t, o, h, f) moves to ib_audit (0x80); its settings (s) stay
M masked email stays
N passkeys stays
O listener status stays
P permissions and protocol policies stays
Q acceptance records stays
R mail flow rules stays
S spam rules, scale-out storage, scheduled reports from 2026.10.6.2 stays
T log files stays
U undelete notes (n, x, b, r, c, k, a, g) moves to ib_undelete (0x83)
V store checks, the layout mark (Vs), lease (Vm) and step progress (Vt) stays
W blob store changes, the sharing policy stays
X singleton role leases (Xs, Xm) and the alerts firing (Xa), #437 stays

A key moves without its first byte, so Le + node + seq becomes e + node

  • seq in ib_audit. A feature's settings stay in _, where a configuration export (registry) carries them. On prod on 2026-11-11, _ held 9,629 rows (3.2 MB), of which 6,813 were the audit log and none were journal, hold or undelete rows.

The step copies each moving prefix (on SQL with one INSERT … SELECT), then checks every copy: the row count, and for the audit log's and journal's chain heads and floors the rows themselves. Only when every check passes does it write Vt, then delete the originals and drop v and c. A run that stopped while deleting copies what's left again and deletes it without checking again; a run that stopped before Vt starts over.

Imports and checks

  • SL-21. A Stalwart 0.16 store has no mark, so it goes through every step. tools/import-check/check.sh runs on each RC that adds a step.
  • SL-22. Tests for each step: on every backend it changes, the step on a seeded store at the version before (data kept, mark written, a second run changes nothing); two nodes starting together on PostgreSQL (one runs the step); a build that reads only the older version refusing the store; and tests/src/store_version.rs extended to the new version.

Order

  1. SL-9 (skip unknown enum values), on main and release/2026.
  2. SL-14 (marker only for a new bucket), no step.
  3. The runner (SL-1 to SL-8) with step 1 → 2 (SL-10 to SL-13). It is rolling-safe, so prod can roll it node by node.
  4. Step 2 → 3 (SL-15 to SL-20), after step 1 → 2 has run on prod. SL-5 has to be on every node before it, and prod stops all three nodes for it.

Measure first

On the PostgreSQL leader, before and after each step: pg_stat_user_tables for m, y and _ (live rows, inserts, updates); WAL bytes over an hour; and SELECT get_byte(k,0), count(*), sum(length(v)) FROM "_" GROUP BY 1 for the size of step 2 → 3.