inbuxa-server/docs/spec/features/scale-out-storage.md

# Feature spec: scale-out storage

Status: draft, 2026-09-18. Feature 8 in SPEC.md §4. Lowest priority of the
eight.

## Provenance

Written for the clean room (SPEC.md §3). Sources, and nothing else:

| Source | License | Used for |
|---|---|---|
| Stalwart's registry schema: `resources/schema/schema.json.gz` and `crates/registry/src/schema/*.rs` in this repository (v0.16.22) | AGPL-3.0-only OR LicenseRef-SEL, taken under the AGPL | The config objects and fields, what upstream flags as Enterprise, which variants may be shard members |
| This repository's AGPL code after the strip: `crates/store` (`lib.rs`, `build/`, `dispatch/`, `backend/{postgres,mysql}/main.rs`, `write/blob.rs`, `query/log.rs`), `crates/services/src/task_manager/maintenance.rs`, `crates/types` (`blob_hash.rs`, `type_state.rs`), `crates/trc` event names, `tests/src/store` | AGPL-3.0-only (Enterprise code was stripped before commit) | The seams, what's missing, how blobs and change ids are keyed today |
| The strip report, `docs/fork/strip-reports/v0.16.22.json` | Ours | Which files and how many snippets were removed. It records counts and paths only, no removed text |
| Stalwart documentation (`website` repo): `docs/storage/backends/composite/{index,sql-replica,sharded-blob,sharded-in-memory}.md`, the same four under `docs/0.15/`, the "Read replicas" sections of `docs/storage/backends/{postgresql,mysql}.md`, and `docs/ref/object/{data-store,blob-store,in-memory-store,search-store}.md` | Unlicensed public documentation: facts used, prose not copied | What each composite store does, which variants it takes, the "keep the shard list stable" rule, where `readReplicas` applies |
| PostgreSQL documentation: "Hot Standby" and "System Administration Functions" (current) | PostgreSQL License | Standby visibility, recovery conflicts, `pg_is_in_recovery()`, `pg_last_wal_replay_lsn()`, `pg_current_wal_lsn()` |
| MySQL 8.4 Reference Manual: `SHOW REPLICA STATUS`, replica options | Public documentation: facts used | `Seconds_Behind_Source` and when it's NULL, the `REPLICATION CLIENT` privilege, `replica_preserve_commit_order` |

No Enterprise-only file or snippet was used. The drafting session is a fresh
one that never saw Enterprise code, upstream's or anyone else's. The removed
files are known here only by their paths in the strip report. Nothing in this
spec describes how upstream implements any of it. Where a behavior couldn't be
settled from these sources, it's marked **Decision** or listed under "Open
questions / to observe".

Disclosure: the drafting session saved its own listing of the documentation
repository's file tree under `/tmp/claude-1000/`, a path the rules bar because
upstream checkouts live there. It grepped that one file, its own `gh api`
output, then deleted it, and read nothing else under that path.

There is no "Observed" section. INBUXA runs a single store, so there is no
live Enterprise deployment of these features to probe.

## What it is

Three ways to spread storage over more than one backend, for installs too
large for one database or one bucket:

- **SQL read replicas.** A PostgreSQL or MySQL store gets a list of replicas.
  Writes go to the primary. Some reads go to the replicas, which takes load
  off the primary.
- **Sharded blob store.** Message bodies, attachments and other blobs are
  spread over two or more blob backends. Each blob lives on one of them,
  chosen from its key.
- **Sharded in-memory store.** Short-lived data (rate-limit counters, locks,
  temporary tokens) is spread over two or more Redis-type backends the same
  way. The same object serves a sharded lookup store.

Upstream ships these only in its Enterprise Edition. inbuxa-server ships them
to everybody, with no edition check.

**Priority.** INBUXA runs one store and doesn't need any of this. It's the
last feature in SPEC.md §4, and nothing else waits on it. It matters for one
reason beyond scale: configs written for upstream Enterprise that use these
fields must open in inbuxa-server and do what they say, rather than being
silently ignored (see "What the fork has today").

## What the fork has today

The strip removed four whole files under `crates/store/src/backend/composite/`
(`mod.rs`, `read_replica.rs`, `sharded_blob.rs`, `sharded_lookup.rs`) and
snippets from these files: `lib.rs` (6), `backend/mod.rs` (1),
`backend/mysql/main.rs` (1), `backend/postgres/main.rs` (1), `build/blob.rs`
(2), `build/lookup.rs` (1), `build/memory.rs` (2), `dispatch/blob.rs` (6),
`dispatch/lookup.rs` (10), `dispatch/mod.rs` (1), `dispatch/search.rs` (6),
`dispatch/store.rs` (8). This spec doesn't guess what they held. What remains:

- **The config is complete.** The registry schema and its generated Rust
  types still carry every field: `readReplicas` on `x:PostgreSqlStore` and
  `x:MySqlStore`, the `Sharded` variant of `x:BlobStore`, `x:InMemoryStore` and
  `x:LookupStore`, and `x:ShardedBlobStore` / `x:ShardedInMemoryStore`. The
  server accepts and stores them.
- **Replicas are connected to, then dropped.** `PostgresStore::open` and
  `MysqlStore::open` build a connection pool for each `readReplicas` entry
  and put them in a local `replicas` list. Nothing uses the list, and `open`
  returns the primary alone. Replica pools inherit everything from the primary's
  config except host, port, database, user, secret (and `options` on
  PostgreSQL). Both drivers connect lazily, so today a configured replica
  costs nothing and does nothing. The one exception is a bad replica secret,
  which fails `open`. **The replicas are ignored without a warning.**
- **Sharded stores fail to build.** `BlobStore::build`,
  `InMemoryStore::build` and `LookupStores::parse_stores` have no arm for
  `Sharded`. It falls to the catch-all, which reports "Binary was not compiled
  with the selected … backend". That message is wrong: no build option
  provides it.
- **The store enums have no composite variant.** `Store`, `BlobStore` and
  `InMemoryStore` in `lib.rs` list only single backends. Every method in
  `dispatch/{store,blob,lookup,search}.rs` is a `match` over those variants.
  Those matches are the seams where composite variants go.
- **Leftovers from the strip, to clean up when this lands:**
  - `lib.rs`, `Store::is_same`, still has an arm for
    `Store::SQLReadReplica` under `#[cfg(all(feature = "enterprise", …))]`.
    The variant doesn't exist. It compiles only because the feature is off.
  - `crates/store/Cargo.toml` still declares `enterprise = []`. SPEC.md §2.3
    says the feature is gone.
  - `build/lookup.rs` gets an unused-import warning for `LookupStore`. It's one
    of the six in SPEC.md §2.2b. The `inbuxa` binary's default features are
    `["rocks"]` only. In that build every remaining arm that names
    `LookupStore` is behind a backend feature (`postgres`, `mysql`, `sqlite`,
    `redis`), so the import is unused. A `Sharded` arm needs no backend
    feature, so it uses the import again. Leave the warning until then. Don't
    silence it.
  - Also in `build/lookup.rs`, `LookupStore::RedisSentinel` has no arm either,
    even with `redis` on. Whether that's upstream's own gap isn't known. It
    isn't part of this feature. Noted for triage.
- **Related, and unaffected:** blob purging already works in 256 slices by
  the first byte of the blob hash (`purge_blobs`, and `shardIndex` on
  `x:TaskStoreMaintenance`). That's slicing of the purge task, not store
  sharding, and it isn't flagged Enterprise.

**Where the code goes.** SPEC.md §2.3 puts rebuilt features in a fork-owned
crate. That doesn't work here. The composite variants must be variants of
`store`'s own enums, and a separate crate that depends on `store` can't also
be depended on by it. **Decision:** the rebuild lives in `crates/store` in new
fork-owned files (AGPL-3.0-only headers) under
`crates/store/src/backend/scaleout/`, with one variant added to each enum and
one arm to each dispatch `match`. It isn't put back under `backend/composite/`,
because that path belongs to upstream's stripped files and would collide on
every sync.

## Data model

Unchanged from upstream, so existing configs open as they are. No field is
added. Upstream flags only two fields as Enterprise, and in inbuxa-server
they're ordinary: `x:MySqlStore.readReplicas` and
`x:PostgreSqlStore.readReplicas`. The sharded variants and objects aren't
flagged in the schema at all, though upstream's documentation calls them
Enterprise.

### Read replicas

`readReplicas` is a list on `x:PostgreSqlStore` (entries `x:PostgreSqlSettings`)
and `x:MySqlStore` (entries `x:MySqlSettings`). Each entry has `host`, `port`,
`database`, `authUsername`, `authSecret`, and on PostgreSQL `options`. Defaults
match the store's: user and database `stalwart`, port 5432 or 3306.

A PostgreSQL or MySQL store object can appear as the `x:DataStore`,
`x:BlobStore`, `x:SearchStore`, a `x:StoreLookup` store, a member of a sharded
blob store, and the `x:TracingStore` or `x:MetricsStore`. The field is the same
in every place.

Upstream's 0.15 documentation describes an older standalone
`sql-read-replica` store type naming a primary and replica store ids. The 0.16
registry has no such type. It's out of scope. Configs from before 0.16 go
through upstream's own migration.

### Sharded stores

| Object | Field | Type | Members may be |
|---|---|---|---|
| `x:ShardedBlobStore`, as `x:BlobStore` `@type: "Sharded"` | `stores` | list of `x:BlobStoreBase`, at least 2 | `S3`, `Azure`, `FileSystem`, `FoundationDb`, `PostgreSql`, `MySql` |
| `x:ShardedInMemoryStore`, as `x:InMemoryStore` `@type: "Sharded"` | `stores` | list of `x:InMemoryStoreBase`, at least 2 | `Redis`, `RedisCluster`, `RedisSentinel` |
| `x:ShardedInMemoryStore`, as a `x:StoreLookup`'s `store`, `@type: "Sharded"` | `stores` | as above | as above |

The base types leave out `Default` and `Sharded`, so a shard can't be the data
store by reference and shards can't nest. A member can still be a PostgreSQL or
MySQL store with its own `readReplicas`.

Permissions are the existing `sysBlobStore*`, `sysInMemoryStore*`,
`sysDataStore*`, `sysSearchStore*` ones. Nothing new.

## Required behavior

Each requirement has an ID, and tests name the IDs they check.

### General

- **ST-1.** Off unless configured. An install with no `readReplicas` entries
  and no `Sharded` variant behaves exactly as it does today: the same
  connections, queries, events and code paths, and no extra work per request.
  An empty `readReplicas` list is the same as none.
- **ST-2.** Existing configs open unchanged, with no edition check. A config
  with `readReplicas` or `Sharded` either works as this spec says, or the
  server reports why at startup (ST-15, ST-23, ST-29). Silently ignoring a
  configured replica or shard, as the fork does today, isn't allowed.
  **Decision** (2026-09-19): built in two steps. Sharded stores (ST-16 to
  ST-29) come first; read-replica routing (ST-5 to ST-15) follows after
  per-domain directories (feature 9). Until then each configured replica is
  reported at startup with an error event naming it, saying replicas aren't
  used yet, and every operation goes to the primary.
- **ST-3.** Composite stores are transparent. Every check on what kind of
  backend a store is answers as its primary does (for replicas) or as its
  members do (for shards): `Store::id()`, `is_sql()`, `is_pg_or_mysql()`,
  `SearchStore::is_postgres()` / `is_mysql()`, `InMemoryStore::is_redis()`.
  So a SQL directory on a data store with replicas is still a SQL directory,
  and PostgreSQL full-text search still uses PostgreSQL's syntax.
- **ST-4.** No new Cargo feature. **Decision:** composite stores are always
  compiled. A composite is usable whenever its member backends are compiled
  in. A member whose backend isn't compiled gets the existing "not compiled"
  build error, naming the member's position in the list.

### Read replicas: routing

- **ST-5.** The primary is the default. Every operation goes to the primary
  unless ST-6 names it. That includes:
  - every write: `write` batches, `delete_range`, `delete_documents`,
    `purge_store`, `create_tables`, account destruction;
  - every read that feeds a write: value assertions, counters read back from
    a write (`add_and_get`), document-id assignment, `try_lock`'s read of the
    current lock, quota checks;
  - the in-memory store when it's the data store (`InMemoryStore::Store`):
    rate limits, locks, tokens;
  - the registry and settings, bootstrap, recovery mode, cluster membership
    and the task queue;
  - maintenance, purges, migrations, reindexing, backup and export;
  - `sql_query`. **Decision:** the statement is operator-written (SQL
    directories, lookup stores, Sieve and expression queries), so the server
    can't tell a read from a write. It always goes to the primary.
- **ST-6.** Replica-eligible reads are opted into at the call site, never
  inferred by the store. **Decision:** the store keeps its current API, which
  always means the primary, and adds an explicit read handle for a given
  account (see "Interfaces"). The first call sites to use it:
  - JMAP `/get`, `/query`, `/changes` and `/queryChanges` on account data,
    and blob download;
  - IMAP `FETCH`, `SEARCH`, `STATUS` and `LIST`, and POP3 `RETR` and `TOP`;
  - WebDAV, CalDAV and CardDAV `GET`, `PROPFIND` and `REPORT`;
  - full-text search queries (`SearchStore::query_account`, `query_global`)
    on a PostgreSQL or MySQL search store;
  - blob reads from a PostgreSQL or MySQL blob store (ST-9).

  A read in a request that also writes goes to the primary: every method in a
  JMAP request after its first `/set`, `/copy` or `/import`, and every read
  inside an IMAP command that writes (`STORE`, `COPY`, `MOVE`, `APPEND`,
  `EXPUNGE`).
- **ST-7.** Read-your-writes. After a write for an account returns to the
  client, every later read for that account sees it, on any node. The
  mechanism, a **Decision**:
  1. Every write for an account produces a change id (the per-account
     counter already returned by the write, `AssignedIds`). Each node keeps,
     per account, the highest change id it has written or heard of (its
     *high-water mark*). It learns of other nodes' writes from the
     `StateChange` broadcasts it already receives, which carry the account and
     change id.
  2. When the server runs more than one node, a write also records the
     account's high-water mark in the in-memory store, with an expiry of twice
     the lag limit (ST-11). This happens before the write's result goes back
     to the client, so no node can answer from a replica before it can see the
     mark. A single node keeps the mark in memory only.
  3. Before an account's read goes to a replica, the server compares the
     replica's latest change id for that account with the mark: the local
     mark, and the shared one when there is one. If the replica is behind, the
     read goes to the primary. The replica's figure may be cached per request.
  4. A state the client presents raises the mark for that read: JMAP
     `sinceState` and `ifInState`, EventSource and push resumption, IMAP
     `CONDSTORE` mod-sequences and `QRESYNC`.

  So after `Email/set` creates a message, the next `Email/get` finds it,
  `Email/changes` from the old state lists it, and a renamed mailbox reads
  back with its new name, whichever node answers.
- **ST-8.** A miss on a replica isn't final. An id that a replica reports
  missing is looked up on the primary before the server answers `notFound`,
  or its equivalent in other protocols. This backs ST-7 up at the cost of one
  primary read per miss.
- **ST-9.** Blobs on a SQL store with replicas. Reads may go to a replica
  (ST-6). A blob the replica doesn't have is read from the primary. Writes and
  deletes go to the primary. Most blob keys are content hashes and never
  change. The few named blobs that are overwritten (the spam classifier model
  and training data, installed app resources) may read stale for up to the lag
  limit. That's accepted.

### Read replicas: lag, failure and validation

- **ST-10.** Lag is measured, not assumed. Each node samples every replica
  once a second (**Decision**):
  - **PostgreSQL:** the replica must answer `pg_is_in_recovery()` true. The
    node samples the primary's `pg_current_wal_lsn()` with a timestamp, and the
    replica's `pg_last_wal_replay_lsn()`. The lag is the age of the oldest
    primary sample the replica hasn't yet replayed. This stays correct when
    the primary is idle, which a replay timestamp doesn't.
  - **MySQL with GTIDs:** the same scheme, comparing the primary's
    `@@global.gtid_executed` samples with the replica's using `GTID_SUBSET()`.
  - **MySQL without GTIDs:** `Seconds_Behind_Source` from
    `SHOW REPLICA STATUS`, which needs the `REPLICATION CLIENT` privilege.
    NULL means replication is stopped, and the replica is unhealthy.
- **ST-11.** Lag limit. A replica more than 5 seconds behind gets no reads. It's
  admitted again once it's under 2.5 seconds. **Decision:** fixed values in the
  first version, because configuring them needs a new field, and new fields
  wait for the fork's namespace (SPEC.md §8). A replica whose lag can't be
  measured (missing privilege, unsupported server) gets no reads, and the
  server logs why once at startup.
- **ST-12.** Replica failure never fails a request. A connection error, a
  timeout, or a query cancelled by the standby (PostgreSQL cancels queries
  that conflict with replay, subject to `max_standby_streaming_delay`) retries
  that read once on the primary. The replica is marked down, gets no reads, and
  is probed again every 10 seconds (**Decision**). Down and up are logged as
  events.
- **ST-13.** Choosing a replica. Reads go round-robin across the replicas
  that are up and under the lag limit. Each replica has its own pool, sized by
  the primary's pool settings, as today.
- **ST-14.** Primary failure. Writes fail as they do today with one database.
  The server doesn't fail over or promote a replica: that's the database's job,
  behind the host name the primary entry points at. While the primary is down,
  replica-eligible reads whose check in ST-7 passes keep working. Everything
  else fails as it does today.
- **ST-15.** Validation at startup, per replica. A replica is left out, with
  an error event naming it, and the server runs on the rest (primary alone if
  need be), when:
  - it's the primary itself: the same host, port and database;
  - it isn't read-only: PostgreSQL `pg_is_in_recovery()` is false, or MySQL
    has neither `read_only` nor `super_read_only` on;
  - it isn't a copy of this primary. **Decision:** at startup the node writes
    a random marker value to the primary, and the replica must show it within
    the lag limit. This catches a replica of some other database;
  - MySQL applies in parallel without preserving commit order (parallel
    workers above 0 and `replica_preserve_commit_order` off). Out-of-order
    commits would break the change-id check in ST-7.

  A replica that fails later checks is handled by ST-12, not left out for good.
  The primary's checks are unchanged.

### Sharded blob store

- **ST-16.** Placement. A blob's *home* is `xxh3_64(key) mod N`: the 64-bit
  XXH3 hash, seed 0, over the full key bytes, where N is the number of entries
  in `stores`, in the order listed. The hash crate is already a dependency.
  **Decision**, fixed forever once shipped. Upstream's documentation says only
  "hash and modulus", so this doesn't claim to match upstream's placement. Keys
  are usually 32-byte content hashes, but some are names (the spam classifier's
  keys), which is why the whole key is hashed rather than its first bytes.
- **ST-17.** Reads. A read goes to the home shard. If the home shard doesn't
  have the blob, the other shards are tried in list order. If one has it, the
  blob is returned and a misplaced-blob event is logged with the key and the
  shard it was found on. Nothing is moved automatically. If no shard has it, the
  result is "not found", as today. A blob is only read after the data store
  says it exists, so this probing happens only for blobs placed under an
  earlier shard list, or ones that really are lost.
- **ST-18.** Writes and deletes. A write goes to the home shard only. A
  delete goes to the home shard. If the home shard reports that it didn't have
  the blob, the other shards are tried until one deletes it. Blob purging
  (`purge_blobs`) needs no change: it deletes through the same call.
- **ST-19.** Changing the shard list. Appending a shard, or reordering, keeps
  every existing blob readable, through ST-17, at the cost of extra lookups for
  blobs whose home moved. A named blob overwritten after its home moved leaves
  its old copy behind on the old shard. That's accepted: the new home is read
  first. **Moving blobs to their new home (resharding, rebalancing) is out of
  scope.**
- **ST-20.** Layout record. The first time a sharded blob store opens, the
  server stores a description of the list in the data store: each member's
  kind and location (bucket, container, path, host and database), never its
  secrets. On every later start:
  - the same list: nothing to do;
  - members added or reordered: a warning event naming what changed, and the
    record is updated;
  - a recorded member missing from the list: the blob store refuses to open,
    naming the missing member, because blobs on it would be unreachable.
    **Decision.** How an operator confirms that a removal is intended is an
    open question.
- **ST-21.** One shard failing. Operations whose home is the failing shard
  fail with that backend's usual error, and the rest carry on. A read whose home
  shard errors (rather than reporting a miss) returns the error without probing
  the others. A write whose home shard is down fails, so local delivery answers
  with a temporary failure and the message stays queued, as it would with a
  single blob store down. **Decision:** writes aren't redirected to another
  shard. A redirected named blob would leave an older copy on its recovered
  home, and that copy would be read first.
- **ST-22.** Validation. At least two members (the schema already requires
  it). Two members pointing at the same place (the same bucket and prefix, the
  same directory, the same database) are refused. Every member must open at
  startup, or the blob store fails to build with an error naming the member's
  position.

### Sharded in-memory and lookup stores

- **ST-23.** Placement. Every single-key operation goes to the key's home,
  chosen as in ST-16 over the full key, prefix byte included: `key_set`,
  `key_get`, `key_exists`, `key_delete`, `counter_incr`, `counter_get`,
  `counter_delete`, rate limits, `try_lock` and `remove_lock`. A lock and its
  release therefore always meet on the same member.
- **ST-24.** Operations over many keys go to every member:
  `key_delete_prefix` and `purge_in_memory_store`. They succeed only if every
  member does, and a failure names the member.
- **ST-25.** No fallback reads. ST-17 doesn't apply here. Probing other members
  would break locks and counters. Changing the member list moves keys, and data
  on the old home is abandoned until it expires: rate-limit windows start
  again, and temporary tokens and greylist entries may be lost. That's accepted
  for short-lived data. Locks are the risk: two nodes with different lists
  could both take the same lock. So:
- **ST-26.** Every node must use the same member list. As in ST-20, the list
  is recorded in the data store. A node starting with a different list logs an
  error event naming the difference, then runs with its own list.
  **Decision:** it doesn't refuse, because the data is short-lived and
  refusing would block a planned change. Operators change the list by
  restarting every node together.
- **ST-27.** One member failing. Operations on its keys fail as they would
  with that single Redis down today. Other keys are unaffected. What callers do
  with that error (rate limits, locks, greylisting) is today's behavior,
  unchanged (see "Open questions").
- **ST-28.** A sharded lookup store (`x:StoreLookup` with a `Sharded` store)
  behaves as ST-23 to ST-27. The existing rule that each namespace appears once
  still applies. `into_store()` returns none, as for Redis, so SQL queries
  against it stay unsupported.
- **ST-29.** Validation as ST-22: at least two members, no duplicates, and
  every member opens.

### Observability

- **ST-30.** **Decision:** no new event names in the first version, because
  new names belong in the fork's namespace (SPEC.md §8). Replica and shard
  problems are logged with the existing events (`store.postgresql-error`,
  `store.mysql-error`, `store.redis-error`, `store.s3-error`,
  `store.azure-error`, `store.filesystem-error`, `store.pool-error`), with
  details naming the replica host or the shard's position. Replica down and up,
  lag over the limit, and misplaced blobs are logged the same way.

## Interfaces

- **Config, unchanged:** the fields and variants under "Data model", through
  `x:DataStore`, `x:BlobStore`, `x:InMemoryStore`, `x:SearchStore` and
  `x:StoreLookup`, with their existing permissions.
- **Inside the server, new:** a read handle for replica-eligible reads, for
  example `store.replica_read(account_id)`. It carries the account's
  high-water mark (ST-7). On a store with no replicas it's the store itself, so
  call sites don't branch (ST-1). Every existing `Store` method keeps meaning
  "primary".
- **Inside the server, new:** the per-account high-water mark, fed by write
  results and `StateChange` broadcasts, shared through the in-memory store when
  there's more than one node.
- **Data store keys, new:** the shard layout records (ST-20, ST-26), under a
  key in the fork's namespace.
- **JMAP, IMAP, SMTP and the rest:** no change visible to clients.
- **ihasmail:** nothing required. The storage settings forms come from the
  schema, which doesn't change. No new ihasmail strings, so no translation
  work.

## Acceptance tests

Every test runs against inbuxa-server built with no Enterprise code, with the
backend features the test needs (`postgres`, `mysql`, `redis`, `s3`). The
default `inbuxa` build compiles only RocksDB.

**Needs nothing extra:**

1. A single-store install (RocksDB, and PostgreSQL with no replicas) opens no
   extra connections, and the existing store suites in `tests/src/store` pass
   unchanged (ST-1).
2. Sharded blob store over three `FileSystem` members: each blob lands on
   exactly one member, on its ST-16 home. The existing blob suite
   (`tests/src/store/blob.rs`) passes against it (ST-16, ST-18).
3. Append a fourth `FileSystem` member: every existing message still
   downloads over JMAP and IMAP. New blobs land by the new mapping. Reads of
   moved blobs log the misplaced-blob event (ST-17, ST-19, ST-20).
4. Remove a member: the blob store refuses to open and names it (ST-20).
5. Make one member's directory unreadable: blobs on the others still read.
   Delivery of a message whose home is that member fails temporarily and stays
   queued (ST-21).
6. Purge after deleting messages: unlinked blobs are removed from whichever
   member holds them, including misplaced ones (ST-18).
7. Two members naming the same directory: refused (ST-22).
8. A config with `Sharded` no longer reports "not compiled" (ST-2).

**Needs a PostgreSQL primary with a streaming replica:**

9. An upstream-written config with `readReplicas` opens. JMAP `Email/get`
   and IMAP `FETCH` are served by the replica, and the replica's statement log
   shows no writes, locks or rate-limit traffic (ST-2, ST-5, ST-6).
10. With replay paused (`pg_wal_replay_pause()`): `Email/set` creates a
    message, and a separate `Email/get` finds it. `Mailbox/set` renames a
    mailbox, and `Mailbox/get` shows the new name. `Email/changes` from the
    old state lists the new message (ST-7, ST-8).
11. Two nodes, replay paused: `/set` through node A, then `/get` through node
    B straight away, sees the change (ST-7).
12. Replay paused for 10 seconds: the replica gets no reads, then gets them
    again after replay resumes and catches up (ST-10, ST-11).
13. Replica stopped: requests keep succeeding, and the down and up events are
    logged. Replica restarted: back in use within about 10 seconds (ST-12).
14. Replica entry pointing at a writable, unrelated database: left out at
    startup with an error, and the server runs on the primary (ST-15).
15. A SQL directory on the data store with replicas still authenticates, and
    PostgreSQL full-text search works with the search store on a replica
    (ST-3, ST-6).
16. Primary stopped: eligible reads still answer, and writes fail as with a
    single database (ST-14).

**Needs a MySQL source with a replica:**

17. Tests 9, 10 and 12 with GTIDs on (ST-10).
18. With GTIDs off, with and without `REPLICATION CLIENT`: lag comes from
    `Seconds_Behind_Source` in the first case, and the replica is left out with
    a logged reason in the second (ST-10, ST-11).
19. Parallel replica with `replica_preserve_commit_order` off: left out at
    startup (ST-15).

**Needs two or more Redis servers:**

20. Sharded in-memory store over two Redis servers: rate limits, locks and
    the `resetRateLimiters` and `removeLock*` maintenance types behave as with
    one Redis. A prefix delete clears keys on both (ST-23, ST-24).
21. One Redis stopped: keys homed on the other still work (ST-27).
22. Two nodes started with different member lists: the second logs the
    difference (ST-26).
23. A sharded lookup store serves reads and writes to its namespace (ST-28).

**Needs S3 or Azure (optional):** test 2 with mixed members (S3, FileSystem
and PostgreSQL), to show members of different kinds work together (ST-16).

## Open questions / to observe

- **Placement compatibility with upstream.** Upstream's hash isn't public,
  so ST-16 is the fork's own. An install coming from upstream Enterprise with
  a sharded blob store would read every blob through ST-17's probing, which
  works but is slow. Is a one-off relocation tool worth building? There's no
  such install to observe: INBUXA doesn't shard.
- **Configurable limits.** The lag limit, probe interval and re-probe interval
  (ST-10 to ST-12) are fixed until the fork's namespace (SPEC.md §8) allows new
  fields.
- **New event names** for replica down and up, lag and misplaced blobs
  (ST-30). Same dependency on the namespace.
- **Cost of the shared high-water mark** (ST-7, step 2): one in-memory write
  per account write and one read per request, on multi-node installs only.
  Measure it against sticky sessions at the load balancer, which would make it
  unnecessary.
- **SQL directory queries on replicas.** Sign-in lookups are a large share of
  read load, but ST-5 keeps `sql_query` on the primary. A per-directory
  opt-in would need a new field.
- **MariaDB, Galera, AlloyDB.** Upstream's documentation lists them. MariaDB's
  GTIDs differ from MySQL's, so ST-10 falls back to `Seconds_Behind_Master`
  there. None of them is tested.
- **Confirming a removed shard** (ST-20): how an operator says "yes, drop it"
  without a new field. Perhaps a recovery-mode command (SPEC.md §6.3).
- **Callers' behavior on in-memory errors** (ST-27): whether a rate-limit
  check, lock or greylist lookup that errors lets the request through or
  refuses it. To observe on a stock build with Redis stopped, before any code
  changes.
- **Replicas on the tracing and metrics stores.** They're feature 6's objects.
  ST-4 to ST-15 apply as written, but nothing reads them on a request path yet.
- **`LookupStore::RedisSentinel` has no build arm** in `build/lookup.rs`. It's
  outside this feature. Check a stock build to see whether a sentinel lookup
  store works at all.

## Implementation status

Built 2026-09-19 from this spec, clean-room, under the multi-tenancy hand-off
brief's rules, in `crates/store/src/backend/scaleout/` (as decided above),
with one variant added to `Store`, `BlobStore` and `InMemoryStore` and one
arm to each dispatch and build `match`, marked `inbuxa:`. Sharded stores
came first; read replicas followed per-domain directories (the ST-2
Decision).

- **ST-1 to ST-30:** built, with the limits below.
- **Read replicas.** A data store with `readReplicas` becomes
  `Store::Replicated`. The read handle of "Interfaces" is a read scope that
  the call sites ST-6 names open around a request (`replica_read`), and that
  spawned work carries along. Only account data (properties, indexes,
  change logs, counters, ACLs, blobs, the search index) is read from a
  replica; the registry, in-memory values, the task queue and everything
  else go to the primary (ST-5).
- **Tests.** `store::scaleout::scaleout_blob_tests` covers tests 2 to 7 on
  three and four FileSystem members, opening the store directly against the
  data store. The existing blob suite passes against a sharded store with
  `BLOB_STORE=Sharded` (three FileSystem members), which is test 8 and the
  second half of test 2. `scaleout_memory_tests`, built with `redis`, covers
  tests 20, 22 and 23 over two databases of one Redis server, which the store
  treats as two members. `store::replica::replica_tests`, built with
  `postgres` and run with `STORE=PostgreSqlReplicated`, runs a primary and a
  streaming hot standby in containers and covers tests 9, 10, 12, 13, 14
  and 15. `replica_cluster_tests` (with `redis`) covers test 11, and
  `replica_mysql` covers tests 17 to 19 against two MySQL pairs, one
  replicating with GTIDs and one by binary log position. Test 1 is the
  existing store, blob and protocol suites passing unchanged.
- **Not exercised:** test 3's downloads over JMAP and IMAP after a restart
  (the same blob reads are checked at the store), test 5's queued delivery
  (the failing write is checked), test 21 (one of two Redis servers stopped),
  the `resetRateLimiters` and `removeLock*` maintenance types (the store
  operations they use are checked), and test 16 (the primary stopped).
  Test 9 checks that the replica served the reads, not the replica's
  statement log, and test 15 checks full-text search, not a SQL directory.
  Test 11 uses one server and a second replicated store with its own
  marks, rather than two whole nodes.
- **Settled from the code, not a change of intent:**
  - The FileSystem backend reports any unreadable file as missing, so a
    FileSystem member that can't be read looks like a miss (ST-17's search
    of the other members), not ST-21's error. Its blobs still aren't
    returned, and writes homed on it fail.
  - Member lists are recorded under the fork's own keys (`_` subspace,
    `Sb` for the blob store, `Sm` for the in-memory store, `Sl` and the
    namespace for a lookup store). An in-memory list that differs is
    recorded as the new one after the error is logged, so a planned change
    made by restarting every node settles.
  - A misplaced blob is logged as `store.unexpected-error`, naming the
    member it was found on and its home.
  - Any write made inside a read scope (a first mailbox read creating the
    default mailboxes, an IMAP `FETCH` setting `\Seen`) sends the rest of
    the scope to the primary, as ST-6 asks of requests that write.
  - A replica that hasn't shown the startup marker within the lag limit is
    tried again at each 10-second probe, and left out after six misses: at
    startup a replica can still be replaying a burst of writes.
  - Shared high-water marks (ST-7, step 2) go through the in-memory store
    only when it's Redis; a cluster whose in-memory store is the data store
    relies on the state-change broadcasts and ST-8.
- **Known limits, not requirements of this spec:**
  - Placement is the fork's own (ST-16), so an install coming from a
    sharded upstream deployment reads through ST-17's search (open
    question).
  - ST-7's step 4 raises the mark from a JMAP `sinceState` and an IMAP
    `FETCH ... CHANGEDSINCE`; a JMAP `queryChanges` state, `QRESYNC` on
    SELECT and push resumption don't yet.
  - The store's `enterprise` Cargo feature stays: other crates' feature
    lists name it.

## Observed

Settled on 2026-09-18 against INBUXA's live Enterprise server (Stalwart
0.16.22), read-only, as a server-level administrator and the throwaway test
account. No upstream code was read.

INBUXA runs a single RocksDB data store, with the default blob, in-memory
and search stores. No replicas, and no sharded stores. Nothing to carry
over, and nothing here affects its cutover.