Scale-out storage: implementation status of the sharded stores (ST-1 to ST-4, ST-16 to ST-30)

Where the code lives, which suite covers each acceptance test, what isn't
exercised, what was settled from the code, and that read-replica routing
(ST-5 to ST-15) waits for per-domain directories.
This commit is contained in:
2026-09-19 10:21:30 -07:00
parent fdfa2bc259
commit b080fa0736
+46
View File
@@ -527,6 +527,52 @@ and PostgreSQL), to show members of different kinds work together (ST-16).
outside this feature. Check a stock build to see whether a sentinel lookup outside this feature. Check a stock build to see whether a sentinel lookup
store works at all. store works at all.
## Implementation status
Sharded stores built 2026-09-19 from this spec, clean-room, under the
multi-tenancy hand-off brief's rules, in `crates/store/src/backend/scaleout/`
(as decided above), with one variant added to `BlobStore` and
`InMemoryStore` and one arm to each dispatch and build `match`, marked
`inbuxa:`. Read-replica routing waits for per-domain directories (the ST-2
Decision).
- **ST-1 to ST-4, ST-16 to ST-30:** built, with the limits below.
- **ST-5 to ST-15:** not built yet. Each configured replica is reported at
startup with `store.postgresql-error` or `store.mysql-error` naming it, and
nothing connects to it (the pools the stripped code opened and dropped are
gone).
- **Tests.** `store::scaleout::scaleout_blob_tests` covers tests 2 to 7 on
three and four FileSystem members, opening the store directly against the
data store. The existing blob suite passes against a sharded store with
`BLOB_STORE=Sharded` (three FileSystem members), which is test 8 and the
second half of test 2. `scaleout_memory_tests`, built with `redis`, covers
tests 20, 22 and 23 over two databases of one Redis server, which the store
treats as two members. Test 1 is the existing store and blob suites
passing unchanged.
- **Not exercised:** test 3's downloads over JMAP and IMAP after a restart
(the same blob reads are checked at the store), test 5's queued delivery
(the failing write is checked), test 21 (one of two Redis servers stopped),
the `resetRateLimiters` and `removeLock*` maintenance types (the store
operations they use are checked), and tests 9 to 19 (replicas).
- **Settled from the code, not a change of intent:**
- The FileSystem backend reports any unreadable file as missing, so a
FileSystem member that can't be read looks like a miss (ST-17's search
of the other members), not ST-21's error. Its blobs still aren't
returned, and writes homed on it fail.
- Member lists are recorded under the fork's own keys (`_` subspace,
`Sb` for the blob store, `Sm` for the in-memory store, `Sl` and the
namespace for a lookup store). An in-memory list that differs is
recorded as the new one after the error is logged, so a planned change
made by restarting every node settles.
- A misplaced blob is logged as `store.unexpected-error`, naming the
member it was found on and its home.
- **Known limits, not requirements of this spec:**
- Placement is the fork's own (ST-16), so an install coming from a
sharded upstream deployment reads through ST-17's search (open
question).
- The leftover `Store::SQLReadReplica` arm and the store's `enterprise`
feature stay until replica routing lands.
## Observed ## Observed
Settled on 2026-09-18 against INBUXA's live Enterprise server (Stalwart Settled on 2026-09-18 against INBUXA's live Enterprise server (Stalwart