Scale-out storage: implementation status of read replicas (ST-5 to ST-15)

Where the routing lives and what a read scope covers, which acceptance
tests the container suite runs, what isn't exercised (two nodes, the
primary stopped, and the MySQL paths, which are built but unrun), what
was settled from the code, and the known limits.
This commit is contained in:
2026-09-19 14:21:54 -07:00
parent 1518c69033
commit 216bb5b711
+37 -15
View File
@@ -529,31 +529,42 @@ and PostgreSQL), to show members of different kinds work together (ST-16).
## Implementation status ## Implementation status
Sharded stores built 2026-09-19 from this spec, clean-room, under the Built 2026-09-19 from this spec, clean-room, under the multi-tenancy hand-off
multi-tenancy hand-off brief's rules, in `crates/store/src/backend/scaleout/` brief's rules, in `crates/store/src/backend/scaleout/` (as decided above),
(as decided above), with one variant added to `BlobStore` and with one variant added to `Store`, `BlobStore` and `InMemoryStore` and one
`InMemoryStore` and one arm to each dispatch and build `match`, marked arm to each dispatch and build `match`, marked `inbuxa:`. Sharded stores
`inbuxa:`. Read-replica routing waits for per-domain directories (the ST-2 came first; read replicas followed per-domain directories (the ST-2
Decision). Decision).
- **ST-1 to ST-4, ST-16 to ST-30:** built, with the limits below. - **ST-1 to ST-30:** built, with the limits below.
- **ST-5 to ST-15:** not built yet. Each configured replica is reported at - **Read replicas.** A data store with `readReplicas` becomes
startup with `store.postgresql-error` or `store.mysql-error` naming it, and `Store::Replicated`. The read handle of "Interfaces" is a read scope that
nothing connects to it (the pools the stripped code opened and dropped are the call sites ST-6 names open around a request (`replica_read`), and that
gone). spawned work carries along. Only account data (properties, indexes,
change logs, counters, ACLs, blobs, the search index) is read from a
replica; the registry, in-memory values, the task queue and everything
else go to the primary (ST-5).
- **Tests.** `store::scaleout::scaleout_blob_tests` covers tests 2 to 7 on - **Tests.** `store::scaleout::scaleout_blob_tests` covers tests 2 to 7 on
three and four FileSystem members, opening the store directly against the three and four FileSystem members, opening the store directly against the
data store. The existing blob suite passes against a sharded store with data store. The existing blob suite passes against a sharded store with
`BLOB_STORE=Sharded` (three FileSystem members), which is test 8 and the `BLOB_STORE=Sharded` (three FileSystem members), which is test 8 and the
second half of test 2. `scaleout_memory_tests`, built with `redis`, covers second half of test 2. `scaleout_memory_tests`, built with `redis`, covers
tests 20, 22 and 23 over two databases of one Redis server, which the store tests 20, 22 and 23 over two databases of one Redis server, which the store
treats as two members. Test 1 is the existing store and blob suites treats as two members. `store::replica::replica_tests`, built with
passing unchanged. `postgres` and run with `STORE=PostgreSqlReplicated`, runs a primary and a
streaming hot standby in containers and covers tests 9, 10, 12, 13, 14
and 15. Test 1 is the existing store, blob and protocol suites passing
unchanged.
- **Not exercised:** test 3's downloads over JMAP and IMAP after a restart - **Not exercised:** test 3's downloads over JMAP and IMAP after a restart
(the same blob reads are checked at the store), test 5's queued delivery (the same blob reads are checked at the store), test 5's queued delivery
(the failing write is checked), test 21 (one of two Redis servers stopped), (the failing write is checked), test 21 (one of two Redis servers stopped),
the `resetRateLimiters` and `removeLock*` maintenance types (the store the `resetRateLimiters` and `removeLock*` maintenance types (the store
operations they use are checked), and tests 9 to 19 (replicas). operations they use are checked), test 11 (two nodes), test 16 (the
primary stopped), and tests 17 to 19: the MySQL code (GTID and
`Seconds_Behind_Source` lag, the read-only and commit-order checks) is
built but hasn't run against a MySQL replica. Test 9 checks that the
replica served the reads, not the replica's statement log, and test 15
checks full-text search, not a SQL directory.
- **Settled from the code, not a change of intent:** - **Settled from the code, not a change of intent:**
- The FileSystem backend reports any unreadable file as missing, so a - The FileSystem backend reports any unreadable file as missing, so a
FileSystem member that can't be read looks like a miss (ST-17's search FileSystem member that can't be read looks like a miss (ST-17's search
@@ -566,12 +577,23 @@ Decision).
made by restarting every node settles. made by restarting every node settles.
- A misplaced blob is logged as `store.unexpected-error`, naming the - A misplaced blob is logged as `store.unexpected-error`, naming the
member it was found on and its home. member it was found on and its home.
- Any write made inside a read scope (a first mailbox read creating the
default mailboxes, an IMAP `FETCH` setting `\Seen`) sends the rest of
the scope to the primary, as ST-6 asks of requests that write.
- A replica that hasn't shown the startup marker within the lag limit is
tried again at each 10-second probe, and left out after six misses: at
startup a replica can still be replaying a burst of writes.
- Shared high-water marks (ST-7, step 2) go through the in-memory store
only when it's Redis; a cluster whose in-memory store is the data store
relies on the state-change broadcasts and ST-8.
- **Known limits, not requirements of this spec:** - **Known limits, not requirements of this spec:**
- Placement is the fork's own (ST-16), so an install coming from a - Placement is the fork's own (ST-16), so an install coming from a
sharded upstream deployment reads through ST-17's search (open sharded upstream deployment reads through ST-17's search (open
question). question).
- The leftover `Store::SQLReadReplica` arm and the store's `enterprise` - ST-7's step 4 raises the mark from a JMAP `sinceState` only; IMAP
feature stay until replica routing lands. `CONDSTORE` and `QRESYNC` values and push resumption don't yet.
- The store's `enterprise` Cargo feature stays: other crates' feature
lists name it.
## Observed ## Observed