Every node with taskScheduler queued its own copy of each scheduled maintenance task, and every node with metricsCalculate evaluated alerts and sent their emails, so each role had to be on exactly one node, and losing that node paused the work until it came back. Each role now has a lease in the store (Xs, Xm in `_`), taken with a compare-and-swap like the store layout lease. Every node with the role tries for it each minute; the holder renews it and does the work. A lease lasts 3 minutes and its holder stops acting 15 seconds before it runs out, so another node with the role takes over within about 4 minutes of the holder stopping. A clean stop gives the leases up, so a restart or a roll hands over at the next round. A node without a cluster coordinator needs no lease: the role alone decides, as before. The firing alerts are stored too (Xa), so a node that takes over alerts doesn't send the ones still firing again, and it sets the cluster-wide gauges before its first evaluation. The daily account and domain counts no longer wait a day on a node that just took the role over. New events cluster.singleton-lease-taken and -lost (665, 666), also in the served schema. `X` is registered in the `_` owners table; the spec's copy of that table gains the N (passkeys) and O (listener status) rows it was missing.
11 KiB
Feature spec: inbuxa's own store layout, changed in steps
Status: approved direction, 2026-11-08 (John: "follow your recommendations"). Not a rebuild (SPEC.md §4): inbuxa's own design.
Provenance
| Source | Used for |
|---|---|
The store-independence decision (2026-10-29) and docs/spec/import-0.16.md |
inbuxa owns its layout from 27.1. The Stalwart 0.16.27 layout is a frozen import format, not a promise |
This server's store mark (INBUXA_STORE_VERSION, key Vs in _, crates/migration) |
Where a store's layout version lives, and the refusal of a newer one |
This server's in-memory store on SQL (crates/store/src/dispatch/lookup.rs) and the fork's subspace (SUBSPACE_INBUXA) |
The two changes this spec makes |
Nothing here comes from Enterprise code or from upstream's 1.0 layout.
What it is
Today a store is at layout version 1, which is the 0.16 layout plus the fork's
_ subspace. This spec adds the machinery to move a store from one layout
version to the next in place, at boot. It also defines the first two steps:
- 1 → 2: rate-limit counters on PostgreSQL, MySQL and SQLite keep their expiry in the counter's own row.
- 2 → 3: the fork's large data (the audit log, the journal, holds and
undelete notes) moves out of
_into subspaces of its own.
A layout step stops rollback to any build that reads only an older version. So a step is a breaking change and ships at a series: 27.1 if it is ready before January, else 27.7. Release candidates may carry it before then.
Requirements
The runner
- SL-1.
crates/migrationholds one function per step, keyed by the version it starts from. At boot the runner reads the mark, treats a missing mark on an existing store as 1, and runs every step from there up toINBUXA_STORE_VERSION. It writes the mark right after each step's last write; SL-2 makes a step that stopped before the mark safe to run again. A new install is marked atINBUXA_STORE_VERSIONstraight away. The steps are incrates/migration/src/layout.rs. - SL-2. Every step can run twice safely. A step interrupted half-way runs again from the start on the next boot and comes to the same result.
- SL-3. One node runs the steps. Before the first step a node takes a
lease: an
AssertValuecompare-and-set on keyVmin_, holding the node id and an expiry 10 minutes ahead, renewed every minute while steps run. A node that can't take the lease waits, re-reads the mark every 10 seconds, and goes on booting once the mark reaches its own version. A lease left by a node that died expires and the next node takes it over. A node also takes over a lease held under its own node id, which a run of its own that stopped left behind. A node whose renewal finds the lease taken stops before writing the mark. - SL-4. A newer mark than the build's own is refused before anything is written, as now.
- SL-5. A running node re-reads the mark every 60 seconds. When it finds a version above its own, it logs why, stops taking connections and exits non-zero. Then a node left on the old build during an upgrade can't keep writing the old layout. This ships in the release before the first step that isn't rolling-safe (SL-6). It ships with the runner.
- SL-6. Each step says whether it is rolling-safe: whether nodes on the
build before it can keep running, unharmed, against the new layout. A
step that isn't refuses to start while another node shows as active
(
cluster_node_list, status Active). A stopped node stops showing as active three minutes after its last heartbeat (UNRESPONSIVE_NODE_TIMEOUT). The step waits, re-checking every 10 seconds, and logs every minute that the other nodes must be stopped first. Only other node ids on other hosts count, so a single-node install never waits, even on an id of its own from before a restart. - SL-7. Each step logs when it starts and when it ends, with counts of what it changed. The startup log states the store's layout version.
- SL-8. Before a step runs, the server logs that the upgrade can't be rolled back without a backup, and where the docs are. The upgrade docs tell operators to take a database dump right before the upgrade, because the nightly dump isn't enough.
Reading what a newer build wrote
- SL-9. Stored lists and map keys of enum values (permissions, events,
and others) skip a value the build doesn't know instead of failing the
whole object. This is needed before the first step, and on release/2026,
so that a rollback still loads its roles. It shipped as its own change
(registry
Pickle::unpickle_item).
Step 1 → 2: counters keep their own expiry (SQL backends)
- SL-10. On PostgreSQL, MySQL and SQLite, the counter table
ygets a nullable expiry columne. A counter with an expiry is written as one row,(k, v, e), instead of a counter row inyplus an expiry marker row inm. RocksDB and FoundationDB keep today's form, and the step does nothing there except write the mark. - SL-11. The column is added at boot by every build that has it, before
any step runs, so a node is never without it. The step fills
efrom each counter's marker (an upsert adding 0 and keeping the later expiry), then deletes those markers. A counter whose marker has already run out is deleted with it. Locks, greylist entries, Sieve duplicate ids and other keys inmstay where they are. - SL-12. The purge deletes expired counters with one statement per table
(
DELETE FROM y WHERE e <= now). Rows with a NULLe, written by a node on the old build during a roll, are purged the old way. - SL-13. Rolling-safe. An old node's counter upsert still works with the column there.
- SL-14. Also, on every backend: write the expiry marker only when the counter's bucket is new, not on every increment. That half of the gain needs no step and may land first. Each process remembers the markers it has written; an increment leaves the marker out only once that write is in and its expiry is at least the one asked for. A counter that comes back equal to the amount added was just created, so if its marker was left out (another node reset or purged the bucket) it is written straight after: a counter is never left without a marker, which would keep it for good.
Step 2 → 3: the fork's data gets its own subspaces
- SL-15. Subspaces get names, not only bytes. Each one has an id byte and
a name, and the SQL table and RocksDB column family come from the name.
Upstream's subspaces keep their one-letter names, so nothing changes for
them. New ones take id bytes from 0x80 up, which are never letters, and
names starting
ib_. Every place that builds a table or column family name from the byte uses the name instead. - SL-16. The audit log, the journal, holds and undelete notes each move
to a subspace of their own, dropping the feature prefix byte from each key.
The small single keys (the store mark, the lease, settings marks) stay in
_. The step's design first lists every key prefix in_and its owner. A prefix nobody claims stops the step. - SL-17. For each moved prefix the step copies the rows, compares counts,
and for the audit log also compares the chain head, before it deletes the
originals. Any mismatch aborts the step with nothing deleted. On
PostgreSQL the copy is done server-side (
INSERT … SELECT). - SL-18. Not rolling-safe (SL-6). On RocksDB an old binary can't open a store with column families it doesn't know.
- SL-19. The moved data travels in exports in a family of its own
(
compliance), included by default, so a configuration export can leave it out. Importing an export taken at version 1 or 2 runs the steps after the import. - SL-20. The legacy
vandccolumn families, which nothing writes, are dropped in the same step.
The keys in _ and their owners (SL-16)
Every key in _ starts with a byte naming its feature. The step lists the
two-byte prefixes in _, reading one key per prefix, and stops on any it
can't place, before it writes anything. The table is OWNERS in
crates/migration/src/fork_subspaces.rs.
| First byte | Owner | At step 2 → 3 |
|---|---|---|
A |
AI limits | stays |
B |
first boot defaults | stays |
C |
account snapshots | stays |
D |
deliverability reports | stays |
E |
inbuxa relay | stays |
F |
front ends | stays |
G |
scheduled reports | stays |
H |
legal holds (h, o, e) |
moves to ib_holds (0x82) |
I |
migrations | stays |
J |
journal: entries (e, c, p, t, x, h, f), archive (o, w) |
moves to ib_journal (0x81); the journals themselves (j) are settings and stay |
K |
account locks | stays |
L |
audit log (e, t, o, h, f) |
moves to ib_audit (0x80); its settings (s) stay |
M |
masked email | stays |
N |
passkeys | stays |
O |
listener status | stays |
P |
permissions and protocol policies | stays |
Q |
acceptance records | stays |
R |
mail flow rules | stays |
S |
spam rules, scale-out storage, scheduled reports from 2026.10.6.2 | stays |
T |
log files | stays |
U |
undelete notes (n, x, b, r, c, k, a, g) |
moves to ib_undelete (0x83) |
V |
store checks, the layout mark (Vs), lease (Vm) and step progress (Vt) |
stays |
W |
blob store changes, the sharing policy | stays |
X |
singleton role leases (Xs, Xm) and the alerts firing (Xa), #437 |
stays |
A key moves without its first byte, so Le + node + seq becomes e + node
- seq in
ib_audit. A feature's settings stay in_, where a configuration export (registry) carries them. On prod on 2026-11-11,_held 9,629 rows (3.2 MB), of which 6,813 were the audit log and none were journal, hold or undelete rows.
The step copies each moving prefix (on SQL with one INSERT … SELECT), then
checks every copy: the row count, and for the audit log's and journal's
chain heads and floors the rows themselves. Only when every check passes
does it write Vt, then delete the originals and drop v and c. A run
that stopped while deleting copies what's left again and deletes it
without checking again; a run that stopped before Vt starts over.
Imports and checks
- SL-21. A Stalwart 0.16 store has no mark, so it goes through every step.
tools/import-check/check.shruns on each RC that adds a step. - SL-22. Tests for each step: on every backend it changes, the step on a
seeded store at the version before (data kept, mark written, a second run
changes nothing); two nodes starting together on PostgreSQL (one runs the
step); a build that reads only the older version refusing the store; and
tests/src/store_version.rsextended to the new version.
Order
- SL-9 (skip unknown enum values), on main and release/2026.
- SL-14 (marker only for a new bucket), no step.
- The runner (SL-1 to SL-8) with step 1 → 2 (SL-10 to SL-13). It is rolling-safe, so prod can roll it node by node.
- Step 2 → 3 (SL-15 to SL-20), after step 1 → 2 has run on prod. SL-5 has to be on every node before it, and prod stops all three nodes for it.
Measure first
On the PostgreSQL leader, before and after each step:
pg_stat_user_tables for m, y and _ (live rows, inserts, updates);
WAL bytes over an hour; and
SELECT get_byte(k,0), count(*), sum(length(v)) FROM "_" GROUP BY 1
for the size of step 2 → 3.