inbuxa-server/docs/spec/features/scale-out-storage.md

Feature spec: scale-out storage

Status: draft, 2026-09-18. Feature 8 in SPEC.md §4. Lowest priority of the eight.

Provenance

Written for the clean room (SPEC.md §3). Sources, and nothing else:

Source	License	Used for
Stalwart's registry schema: resources/schema/schema.json.gz and crates/registry/src/schema/*.rs in this repository (v0.16.22)	AGPL-3.0-only OR LicenseRef-SEL, taken under the AGPL	The config objects and fields, what upstream flags as Enterprise, which variants may be shard members
This repository's AGPL code after the strip: crates/store (lib.rs, build/, dispatch/, backend/{postgres,mysql}/main.rs, write/blob.rs, query/log.rs), crates/services/src/task_manager/maintenance.rs, crates/types (blob_hash.rs, type_state.rs), crates/trc event names, tests/src/store	AGPL-3.0-only (Enterprise code was stripped before commit)	The seams, what's missing, how blobs and change ids are keyed today
The strip report, docs/fork/strip-reports/v0.16.22.json	Ours	Which files and how many snippets were removed. It records counts and paths only, no removed text
Stalwart documentation (website repo): docs/storage/backends/composite/{index,sql-replica,sharded-blob,sharded-in-memory}.md, the same four under docs/0.15/, the "Read replicas" sections of docs/storage/backends/{postgresql,mysql}.md, and docs/ref/object/{data-store,blob-store,in-memory-store,search-store}.md	Unlicensed public documentation: facts used, prose not copied	What each composite store does, which variants it takes, the "keep the shard list stable" rule, where readReplicas applies
PostgreSQL documentation: "Hot Standby" and "System Administration Functions" (current)	PostgreSQL License	Standby visibility, recovery conflicts, pg_is_in_recovery(), pg_last_wal_replay_lsn(), pg_current_wal_lsn()
MySQL 8.4 Reference Manual: SHOW REPLICA STATUS, replica options	Public documentation: facts used	Seconds_Behind_Source and when it's NULL, the REPLICATION CLIENT privilege, replica_preserve_commit_order

No Enterprise-only file or snippet was used. The drafting session is a fresh one that never saw Enterprise code, upstream's or anyone else's. The removed files are known here only by their paths in the strip report. Nothing in this spec describes how upstream implements any of it. Where a behavior couldn't be settled from these sources, it's marked Decision or listed under "Open questions / to observe".

Disclosure: the drafting session saved its own listing of the documentation repository's file tree under /tmp/claude-1000/, a path the rules bar because upstream checkouts live there. It grepped that one file, its own gh api output, then deleted it, and read nothing else under that path.

There is no "Observed" section. INBUXA runs a single store, so there is no live Enterprise deployment of these features to probe.

What it is

Three ways to spread storage over more than one backend, for installs too large for one database or one bucket:

• SQL read replicas. A PostgreSQL or MySQL store gets a list of replicas. Writes go to the primary. Some reads go to the replicas, which takes load off the primary.
• Sharded blob store. Message bodies, attachments and other blobs are spread over two or more blob backends. Each blob lives on one of them, chosen from its key.
• Sharded in-memory store. Short-lived data (rate-limit counters, locks, temporary tokens) is spread over two or more Redis-type backends the same way. The same object serves a sharded lookup store.

Upstream ships these only in its Enterprise Edition. inbuxa-server ships them to everybody, with no edition check.

Priority. INBUXA runs one store and doesn't need any of this. It's the last feature in SPEC.md §4, and nothing else waits on it. It matters for one reason beyond scale: configs written for upstream Enterprise that use these fields must open in inbuxa-server and do what they say, rather than being silently ignored (see "What the fork has today").

What the fork has today

The strip removed four whole files under crates/store/src/backend/composite/ (mod.rs, read_replica.rs, sharded_blob.rs, sharded_lookup.rs) and snippets from these files: lib.rs (6), backend/mod.rs (1), backend/mysql/main.rs (1), backend/postgres/main.rs (1), build/blob.rs (2), build/lookup.rs (1), build/memory.rs (2), dispatch/blob.rs (6), dispatch/lookup.rs (10), dispatch/mod.rs (1), dispatch/search.rs (6), dispatch/store.rs (8). This spec doesn't guess what they held. What remains:

• The config is complete. The registry schema and its generated Rust types still carry every field: readReplicas on x:PostgreSqlStore and x:MySqlStore, the Sharded variant of x:BlobStore, x:InMemoryStore and x:LookupStore, and x:ShardedBlobStore / x:ShardedInMemoryStore. The server accepts and stores them.
• Replicas are connected to, then dropped. PostgresStore::open and MysqlStore::open build a connection pool for each readReplicas entry and put them in a local replicas list. Nothing uses the list, and open returns the primary alone. Replica pools inherit everything from the primary's config except host, port, database, user, secret (and options on PostgreSQL). Both drivers connect lazily, so today a configured replica costs nothing and does nothing. The one exception is a bad replica secret, which fails open. The replicas are ignored without a warning.
• Sharded stores fail to build. BlobStore::build, InMemoryStore::build and LookupStores::parse_stores have no arm for Sharded. It falls to the catch-all, which reports "Binary was not compiled with the selected … backend". That message is wrong: no build option provides it.
• The store enums have no composite variant. Store, BlobStore and InMemoryStore in lib.rs list only single backends. Every method in dispatch/{store,blob,lookup,search}.rs is a match over those variants. Those matches are the seams where composite variants go.
• Leftovers from the strip, to clean up when this lands:
  • lib.rs, Store::is_same, still has an arm for Store::SQLReadReplica under #[cfg(all(feature = "enterprise", …))]. The variant doesn't exist. It compiles only because the feature is off.
  • crates/store/Cargo.toml still declares enterprise = []. SPEC.md §2.3 says the feature is gone.
  • build/lookup.rs gets an unused-import warning for LookupStore. It's one of the six in SPEC.md §2.2b. The inbuxa binary's default features are ["rocks"] only. In that build every remaining arm that names LookupStore is behind a backend feature (postgres, mysql, sqlite, redis), so the import is unused. A Sharded arm needs no backend feature, so it uses the import again. Leave the warning until then. Don't silence it.
  • Also in build/lookup.rs, LookupStore::RedisSentinel has no arm either, even with redis on. Whether that's upstream's own gap isn't known. It isn't part of this feature. Noted for triage.
• Related, and unaffected: blob purging already works in 256 slices by the first byte of the blob hash (purge_blobs, and shardIndex on x:TaskStoreMaintenance). That's slicing of the purge task, not store sharding, and it isn't flagged Enterprise.

Where the code goes. SPEC.md §2.3 puts rebuilt features in a fork-owned crate. That doesn't work here. The composite variants must be variants of store's own enums, and a separate crate that depends on store can't also be depended on by it. Decision: the rebuild lives in crates/store in new fork-owned files (AGPL-3.0-only headers) under crates/store/src/backend/scaleout/, with one variant added to each enum and one arm to each dispatch match. It isn't put back under backend/composite/, because that path belongs to upstream's stripped files and would collide on every sync.

Data model

Unchanged from upstream, so existing configs open as they are. No field is added. Upstream flags only two fields as Enterprise, and in inbuxa-server they're ordinary: x:MySqlStore.readReplicas and x:PostgreSqlStore.readReplicas. The sharded variants and objects aren't flagged in the schema at all, though upstream's documentation calls them Enterprise.

Read replicas

readReplicas is a list on x:PostgreSqlStore (entries x:PostgreSqlSettings) and x:MySqlStore (entries x:MySqlSettings). Each entry has host, port, database, authUsername, authSecret, and on PostgreSQL options. Defaults match the store's: user and database stalwart, port 5432 or 3306.

A PostgreSQL or MySQL store object can appear as the x:DataStore, x:BlobStore, x:SearchStore, a x:StoreLookup store, a member of a sharded blob store, and the x:TracingStore or x:MetricsStore. The field is the same in every place.

Upstream's 0.15 documentation describes an older standalone sql-read-replica store type naming a primary and replica store ids. The 0.16 registry has no such type. It's out of scope. Configs from before 0.16 go through upstream's own migration.

Sharded stores

Object	Field	Type	Members may be
x:ShardedBlobStore, as x:BlobStore @type: "Sharded"	stores	list of x:BlobStoreBase, at least 2	S3, Azure, FileSystem, FoundationDb, PostgreSql, MySql
x:ShardedInMemoryStore, as x:InMemoryStore @type: "Sharded"	stores	list of x:InMemoryStoreBase, at least 2	Redis, RedisCluster, RedisSentinel
x:ShardedInMemoryStore, as a x:StoreLookup's store, @type: "Sharded"	stores	as above	as above

The base types leave out Default and Sharded, so a shard can't be the data store by reference and shards can't nest. A member can still be a PostgreSQL or MySQL store with its own readReplicas.

Permissions are the existing sysBlobStore*, sysInMemoryStore*, sysDataStore*, sysSearchStore* ones. Nothing new.

Required behavior

Each requirement has an ID, and tests name the IDs they check.

General

• ST-1. Off unless configured. An install with no readReplicas entries and no Sharded variant behaves exactly as it does today: the same connections, queries, events and code paths, and no extra work per request. An empty readReplicas list is the same as none.
• ST-2. Existing configs open unchanged, with no edition check. A config with readReplicas or Sharded either works as this spec says, or the server reports why at startup (ST-15, ST-23, ST-29). Silently ignoring a configured replica or shard, as the fork does today, isn't allowed. Decision (2026-09-19): built in two steps. Sharded stores (ST-16 to ST-29) come first; read-replica routing (ST-5 to ST-15) follows after per-domain directories (feature 9). Until then each configured replica is reported at startup with an error event naming it, saying replicas aren't used yet, and every operation goes to the primary.
• ST-3. Composite stores are transparent. Every check on what kind of backend a store is answers as its primary does (for replicas) or as its members do (for shards): Store::id(), is_sql(), is_pg_or_mysql(), SearchStore::is_postgres() / is_mysql(), InMemoryStore::is_redis(). So a SQL directory on a data store with replicas is still a SQL directory, and PostgreSQL full-text search still uses PostgreSQL's syntax.
• ST-4. No new Cargo feature. Decision: composite stores are always compiled. A composite is usable whenever its member backends are compiled in. A member whose backend isn't compiled gets the existing "not compiled" build error, naming the member's position in the list.

Read replicas: routing

• ST-5. The primary is the default. Every operation goes to the primary unless ST-6 names it. That includes:

  • every write: write batches, delete_range, delete_documents, purge_store, create_tables, account destruction;
  • every read that feeds a write: value assertions, counters read back from a write (add_and_get), document-id assignment, try_lock's read of the current lock, quota checks;
  • the in-memory store when it's the data store (InMemoryStore::Store): rate limits, locks, tokens;
  • the registry and settings, bootstrap, recovery mode, cluster membership and the task queue;
  • maintenance, purges, migrations, reindexing, backup and export;
  • sql_query. Decision: the statement is operator-written (SQL directories, lookup stores, Sieve and expression queries), so the server can't tell a read from a write. It always goes to the primary.

• ST-6. Replica-eligible reads are opted into at the call site, never inferred by the store. Decision: the store keeps its current API, which always means the primary, and adds an explicit read handle for a given account (see "Interfaces"). The first call sites to use it:

  • JMAP /get, /query, /changes and /queryChanges on account data, and blob download;
  • IMAP FETCH, SEARCH, STATUS and LIST, and POP3 RETR and TOP;
  • WebDAV, CalDAV and CardDAV GET, PROPFIND and REPORT;
  • full-text search queries (SearchStore::query_account, query_global) on a PostgreSQL or MySQL search store;
  • blob reads from a PostgreSQL or MySQL blob store (ST-9).

  A read in a request that also writes goes to the primary: every method in a JMAP request after its first /set, /copy or /import, and every read inside an IMAP command that writes (STORE, COPY, MOVE, APPEND, EXPUNGE).

• ST-7. Read-your-writes. After a write for an account returns to the client, every later read for that account sees it, on any node. The mechanism, a Decision:

  1. Every write for an account produces a change id (the per-account counter already returned by the write, AssignedIds). Each node keeps, per account, the highest change id it has written or heard of (its high-water mark). It learns of other nodes' writes from the StateChange broadcasts it already receives, which carry the account and change id.
  2. When the server runs more than one node, a write also records the account's high-water mark in the in-memory store, with an expiry of twice the lag limit (ST-11). This happens before the write's result goes back to the client, so no node can answer from a replica before it can see the mark. A single node keeps the mark in memory only.
  3. Before an account's read goes to a replica, the server compares the replica's latest change id for that account with the mark: the local mark, and the shared one when there is one. If the replica is behind, the read goes to the primary. The replica's figure may be cached per request.
  4. A state the client presents raises the mark for that read: JMAP sinceState and ifInState, EventSource and push resumption, IMAP CONDSTORE mod-sequences and QRESYNC.

  So after Email/set creates a message, the next Email/get finds it, Email/changes from the old state lists it, and a renamed mailbox reads back with its new name, whichever node answers.

• ST-8. A miss on a replica isn't final. An id that a replica reports missing is looked up on the primary before the server answers notFound, or its equivalent in other protocols. This backs ST-7 up at the cost of one primary read per miss.

• ST-9. Blobs on a SQL store with replicas. Reads may go to a replica (ST-6). A blob the replica doesn't have is read from the primary. Writes and deletes go to the primary. Most blob keys are content hashes and never change. The few named blobs that are overwritten (the spam classifier model and training data, installed app resources) may read stale for up to the lag limit. That's accepted.

Read replicas: lag, failure and validation

• ST-10. Lag is measured, not assumed. Each node samples every replica once a second (Decision):

  • PostgreSQL: the replica must answer pg_is_in_recovery() true. The node samples the primary's pg_current_wal_lsn() with a timestamp, and the replica's pg_last_wal_replay_lsn(). The lag is the age of the oldest primary sample the replica hasn't yet replayed. This stays correct when the primary is idle, which a replay timestamp doesn't.
  • MySQL with GTIDs: the same scheme, comparing the primary's @@global.gtid_executed samples with the replica's using GTID_SUBSET().
  • MySQL without GTIDs: Seconds_Behind_Source from SHOW REPLICA STATUS, which needs the REPLICATION CLIENT privilege. NULL means replication is stopped, and the replica is unhealthy.

• ST-11. Lag limit. A replica more than 5 seconds behind gets no reads. It's admitted again once it's under 2.5 seconds. Decision: fixed values in the first version, because configuring them needs a new field, and new fields wait for the fork's namespace (SPEC.md §8). A replica whose lag can't be measured (missing privilege, unsupported server) gets no reads, and the server logs why once at startup.

• ST-12. Replica failure never fails a request. A connection error, a timeout, or a query cancelled by the standby (PostgreSQL cancels queries that conflict with replay, subject to max_standby_streaming_delay) retries that read once on the primary. The replica is marked down, gets no reads, and is probed again every 10 seconds (Decision). Down and up are logged as events.

• ST-13. Choosing a replica. Reads go round-robin across the replicas that are up and under the lag limit. Each replica has its own pool, sized by the primary's pool settings, as today.

• ST-14. Primary failure. Writes fail as they do today with one database. The server doesn't fail over or promote a replica: that's the database's job, behind the host name the primary entry points at. While the primary is down, replica-eligible reads whose check in ST-7 passes keep working. Everything else fails as it does today.

• ST-15. Validation at startup, per replica. A replica is left out, with an error event naming it, and the server runs on the rest (primary alone if need be), when:

  • it's the primary itself: the same host, port and database;
  • it isn't read-only: PostgreSQL pg_is_in_recovery() is false, or MySQL has neither read_only nor super_read_only on;
  • it isn't a copy of this primary. Decision: at startup the node writes a random marker value to the primary, and the replica must show it within the lag limit. This catches a replica of some other database;
  • MySQL applies in parallel without preserving commit order (parallel workers above 0 and replica_preserve_commit_order off). Out-of-order commits would break the change-id check in ST-7.

  A replica that fails later checks is handled by ST-12, not left out for good. The primary's checks are unchanged.

Sharded blob store

• ST-16. Placement. A blob's home is xxh3_64(key) mod N: the 64-bit XXH3 hash, seed 0, over the full key bytes, where N is the number of entries in stores, in the order listed. The hash crate is already a dependency. Decision, fixed forever once shipped. Upstream's documentation says only "hash and modulus", so this doesn't claim to match upstream's placement. Keys are usually 32-byte content hashes, but some are names (the spam classifier's keys), which is why the whole key is hashed rather than its first bytes.
• ST-17. Reads. A read goes to the home shard. If the home shard doesn't have the blob, the other shards are tried in list order. If one has it, the blob is returned and a misplaced-blob event is logged with the key and the shard it was found on. Nothing is moved automatically. If no shard has it, the result is "not found", as today. A blob is only read after the data store says it exists, so this probing happens only for blobs placed under an earlier shard list, or ones that really are lost.
• ST-18. Writes and deletes. A write goes to the home shard only. A delete goes to the home shard. If the home shard reports that it didn't have the blob, the other shards are tried until one deletes it. Blob purging (purge_blobs) needs no change: it deletes through the same call.
• ST-19. Changing the shard list. Appending a shard, or reordering, keeps every existing blob readable, through ST-17, at the cost of extra lookups for blobs whose home moved. A named blob overwritten after its home moved leaves its old copy behind on the old shard. That's accepted: the new home is read first. Moving blobs to their new home (resharding, rebalancing) is out of scope.
• ST-20. Layout record. The first time a sharded blob store opens, the server stores a description of the list in the data store: each member's kind and location (bucket, container, path, host and database), never its secrets. On every later start:
  • the same list: nothing to do;
  • members added or reordered: a warning event naming what changed, and the record is updated;
  • a recorded member missing from the list: the blob store refuses to open, naming the missing member, because blobs on it would be unreachable. Decision. How an operator confirms that a removal is intended is an open question.
• ST-21. One shard failing. Operations whose home is the failing shard fail with that backend's usual error, and the rest carry on. A read whose home shard errors (rather than reporting a miss) returns the error without probing the others. A write whose home shard is down fails, so local delivery answers with a temporary failure and the message stays queued, as it would with a single blob store down. Decision: writes aren't redirected to another shard. A redirected named blob would leave an older copy on its recovered home, and that copy would be read first.
• ST-22. Validation. At least two members (the schema already requires it). Two members pointing at the same place (the same bucket and prefix, the same directory, the same database) are refused. Every member must open at startup, or the blob store fails to build with an error naming the member's position.

Sharded in-memory and lookup stores

• ST-23. Placement. Every single-key operation goes to the key's home, chosen as in ST-16 over the full key, prefix byte included: key_set, key_get, key_exists, key_delete, counter_incr, counter_get, counter_delete, rate limits, try_lock and remove_lock. A lock and its release therefore always meet on the same member.
• ST-24. Operations over many keys go to every member: key_delete_prefix and purge_in_memory_store. They succeed only if every member does, and a failure names the member.
• ST-25. No fallback reads. ST-17 doesn't apply here. Probing other members would break locks and counters. Changing the member list moves keys, and data on the old home is abandoned until it expires: rate-limit windows start again, and temporary tokens and greylist entries may be lost. That's accepted for short-lived data. Locks are the risk: two nodes with different lists could both take the same lock. So:
• ST-26. Every node must use the same member list. As in ST-20, the list is recorded in the data store. A node starting with a different list logs an error event naming the difference, then runs with its own list. Decision: it doesn't refuse, because the data is short-lived and refusing would block a planned change. Operators change the list by restarting every node together.
• ST-27. One member failing. Operations on its keys fail as they would with that single Redis down today. Other keys are unaffected. What callers do with that error (rate limits, locks, greylisting) is today's behavior, unchanged (see "Open questions").
• ST-28. A sharded lookup store (x:StoreLookup with a Sharded store) behaves as ST-23 to ST-27. The existing rule that each namespace appears once still applies. into_store() returns none, as for Redis, so SQL queries against it stay unsupported.
• ST-29. Validation as ST-22: at least two members, no duplicates, and every member opens.

Observability

• ST-30. Decision: no new event names in the first version, because new names belong in the fork's namespace (SPEC.md §8). Replica and shard problems are logged with the existing events (store.postgresql-error, store.mysql-error, store.redis-error, store.s3-error, store.azure-error, store.filesystem-error, store.pool-error), with details naming the replica host or the shard's position. Replica down and up, lag over the limit, and misplaced blobs are logged the same way.

Interfaces

• Config, unchanged: the fields and variants under "Data model", through x:DataStore, x:BlobStore, x:InMemoryStore, x:SearchStore and x:StoreLookup, with their existing permissions.
• Inside the server, new: a read handle for replica-eligible reads, for example store.replica_read(account_id). It carries the account's high-water mark (ST-7). On a store with no replicas it's the store itself, so call sites don't branch (ST-1). Every existing Store method keeps meaning "primary".
• Inside the server, new: the per-account high-water mark, fed by write results and StateChange broadcasts, shared through the in-memory store when there's more than one node.
• Data store keys, new: the shard layout records (ST-20, ST-26), under a key in the fork's namespace.
• JMAP, IMAP, SMTP and the rest: no change visible to clients.
• ihasmail: nothing required. The storage settings forms come from the schema, which doesn't change. No new ihasmail strings, so no translation work.

Acceptance tests

Every test runs against inbuxa-server built with no Enterprise code, with the backend features the test needs (postgres, mysql, redis, s3). The default inbuxa build compiles only RocksDB.

Needs nothing extra:

1. A single-store install (RocksDB, and PostgreSQL with no replicas) opens no extra connections, and the existing store suites in tests/src/store pass unchanged (ST-1).
2. Sharded blob store over three FileSystem members: each blob lands on exactly one member, on its ST-16 home. The existing blob suite (tests/src/store/blob.rs) passes against it (ST-16, ST-18).
3. Append a fourth FileSystem member: every existing message still downloads over JMAP and IMAP. New blobs land by the new mapping. Reads of moved blobs log the misplaced-blob event (ST-17, ST-19, ST-20).
4. Remove a member: the blob store refuses to open and names it (ST-20).
5. Make one member's directory unreadable: blobs on the others still read. Delivery of a message whose home is that member fails temporarily and stays queued (ST-21).
6. Purge after deleting messages: unlinked blobs are removed from whichever member holds them, including misplaced ones (ST-18).
7. Two members naming the same directory: refused (ST-22).
8. A config with Sharded no longer reports "not compiled" (ST-2).

Needs a PostgreSQL primary with a streaming replica:

9. An upstream-written config with readReplicas opens. JMAP Email/get and IMAP FETCH are served by the replica, and the replica's statement log shows no writes, locks or rate-limit traffic (ST-2, ST-5, ST-6).
10. With replay paused (pg_wal_replay_pause()): Email/set creates a message, and a separate Email/get finds it. Mailbox/set renames a mailbox, and Mailbox/get shows the new name. Email/changes from the old state lists the new message (ST-7, ST-8).
11. Two nodes, replay paused: /set through node A, then /get through node B straight away, sees the change (ST-7).
12. Replay paused for 10 seconds: the replica gets no reads, then gets them again after replay resumes and catches up (ST-10, ST-11).
13. Replica stopped: requests keep succeeding, and the down and up events are logged. Replica restarted: back in use within about 10 seconds (ST-12).
14. Replica entry pointing at a writable, unrelated database: left out at startup with an error, and the server runs on the primary (ST-15).
15. A SQL directory on the data store with replicas still authenticates, and PostgreSQL full-text search works with the search store on a replica (ST-3, ST-6).
16. Primary stopped: eligible reads still answer, and writes fail as with a single database (ST-14).

Needs a MySQL source with a replica:

17. Tests 9, 10 and 12 with GTIDs on (ST-10).
18. With GTIDs off, with and without REPLICATION CLIENT: lag comes from Seconds_Behind_Source in the first case, and the replica is left out with a logged reason in the second (ST-10, ST-11).
19. Parallel replica with replica_preserve_commit_order off: left out at startup (ST-15).

Needs two or more Redis servers:

20. Sharded in-memory store over two Redis servers: rate limits, locks and the resetRateLimiters and removeLock* maintenance types behave as with one Redis. A prefix delete clears keys on both (ST-23, ST-24).
21. One Redis stopped: keys homed on the other still work (ST-27).
22. Two nodes started with different member lists: the second logs the difference (ST-26).
23. A sharded lookup store serves reads and writes to its namespace (ST-28).

Needs S3 or Azure (optional): test 2 with mixed members (S3, FileSystem and PostgreSQL), to show members of different kinds work together (ST-16).

Open questions / to observe

• Placement compatibility with upstream. Upstream's hash isn't public, so ST-16 is the fork's own. An install coming from upstream Enterprise with a sharded blob store would read every blob through ST-17's probing, which works but is slow. Is a one-off relocation tool worth building? There's no such install to observe: INBUXA doesn't shard.
• Configurable limits. The lag limit, probe interval and re-probe interval (ST-10 to ST-12) are fixed until the fork's namespace (SPEC.md §8) allows new fields.
• New event names for replica down and up, lag and misplaced blobs (ST-30). Same dependency on the namespace.
• Cost of the shared high-water mark (ST-7, step 2): one in-memory write per account write and one read per request, on multi-node installs only. Measure it against sticky sessions at the load balancer, which would make it unnecessary.
• SQL directory queries on replicas. Sign-in lookups are a large share of read load, but ST-5 keeps sql_query on the primary. A per-directory opt-in would need a new field.
• MariaDB, Galera, AlloyDB. Upstream's documentation lists them. MariaDB's GTIDs differ from MySQL's, so ST-10 falls back to Seconds_Behind_Master there. None of them is tested.
• Confirming a removed shard (ST-20): how an operator says "yes, drop it" without a new field. Perhaps a recovery-mode command (SPEC.md §6.3).
• Callers' behavior on in-memory errors (ST-27): whether a rate-limit check, lock or greylist lookup that errors lets the request through or refuses it. To observe on a stock build with Redis stopped, before any code changes.
• Replicas on the tracing and metrics stores. They're feature 6's objects. ST-4 to ST-15 apply as written, but nothing reads them on a request path yet.
• LookupStore::RedisSentinel has no build arm in build/lookup.rs. It's outside this feature. Check a stock build to see whether a sentinel lookup store works at all.

Implementation status

Built 2026-09-19 from this spec, clean-room, under the multi-tenancy hand-off brief's rules, in crates/store/src/backend/scaleout/ (as decided above), with one variant added to Store, BlobStore and InMemoryStore and one arm to each dispatch and build match, marked inbuxa:. Sharded stores came first; read replicas followed per-domain directories (the ST-2 Decision).

• ST-1 to ST-30: built, with the limits below.
• Read replicas. A data store with readReplicas becomes Store::Replicated. The read handle of "Interfaces" is a read scope that the call sites ST-6 names open around a request (replica_read), and that spawned work carries along. Only account data (properties, indexes, change logs, counters, ACLs, blobs, the search index) is read from a replica; the registry, in-memory values, the task queue and everything else go to the primary (ST-5).
• Tests. store::scaleout::scaleout_blob_tests covers tests 2 to 7 on three and four FileSystem members, opening the store directly against the data store. The existing blob suite passes against a sharded store with BLOB_STORE=Sharded (three FileSystem members), which is test 8 and the second half of test 2. scaleout_memory_tests, built with redis, covers tests 20, 22 and 23 over two databases of one Redis server, which the store treats as two members. store::replica::replica_tests, built with postgres and run with STORE=PostgreSqlReplicated, runs a primary and a streaming hot standby in containers and covers tests 9, 10, 12, 13, 14 and 15. replica_cluster_tests (with redis) covers test 11, and replica_mysql covers tests 17 to 19 against two MySQL pairs, one replicating with GTIDs and one by binary log position. Test 1 is the existing store, blob and protocol suites passing unchanged.
• Not exercised: test 3's downloads over JMAP and IMAP after a restart (the same blob reads are checked at the store), test 5's queued delivery (the failing write is checked), test 21 (one of two Redis servers stopped), the resetRateLimiters and removeLock* maintenance types (the store operations they use are checked), and test 16 (the primary stopped). Test 9 checks that the replica served the reads, not the replica's statement log, and test 15 checks full-text search, not a SQL directory. Test 11 uses one server and a second replicated store with its own marks, rather than two whole nodes.
• Settled from the code, not a change of intent:
  • The FileSystem backend reports any unreadable file as missing, so a FileSystem member that can't be read looks like a miss (ST-17's search of the other members), not ST-21's error. Its blobs still aren't returned, and writes homed on it fail.
  • Member lists are recorded under the fork's own keys (_ subspace, Sb for the blob store, Sm for the in-memory store, Sl and the namespace for a lookup store). An in-memory list that differs is recorded as the new one after the error is logged, so a planned change made by restarting every node settles.
  • A misplaced blob is logged as store.unexpected-error, naming the member it was found on and its home.
  • Any write made inside a read scope (a first mailbox read creating the default mailboxes, an IMAP FETCH setting \Seen) sends the rest of the scope to the primary, as ST-6 asks of requests that write.
  • A replica that hasn't shown the startup marker within the lag limit is tried again at each 10-second probe, and left out after six misses: at startup a replica can still be replaying a burst of writes.
  • Shared high-water marks (ST-7, step 2) go through the in-memory store only when it's Redis; a cluster whose in-memory store is the data store relies on the state-change broadcasts and ST-8.
• Known limits, not requirements of this spec:
  • Placement is the fork's own (ST-16), so an install coming from a sharded upstream deployment reads through ST-17's search (open question).
  • ST-7's step 4 raises the mark from a JMAP sinceState and an IMAP FETCH ... CHANGEDSINCE; a JMAP queryChanges state, QRESYNC on SELECT and push resumption don't yet.
  • The store's enterprise Cargo feature stays: other crates' feature lists name it.

Observed

Settled on 2026-09-18 against INBUXA's live Enterprise server (Stalwart 0.16.22), read-only, as a server-level administrator and the throwaway test account. No upstream code was read.

INBUXA runs a single RocksDB data store, with the default blob, in-memory and search stores. No replicas, and no sharded stores. Nothing to carry over, and nothing here affects its cutover.