Scale-out storage: PostgreSQL and MySQL read replicas (ST-5 to ST-15)

A data store with readReplicas becomes a replicated store. Writes,
operator-written SQL and everything outside a read scope go to the
primary. JMAP reads before a request's first write, IMAP LIST, STATUS,
SEARCH, SORT and FETCH, POP3 RETR and TOP, DAV GET, PROPFIND and REPORT,
and blob downloads run in a read scope. Only account data (properties,
indexes, change logs, counters, ACLs, blobs, the search index) is read
from a replica; the registry, in-memory values, the task queue and the
rest stay on the primary.

In a scope, the first read picks a replica round-robin among those up
and under the lag limit, and only if it has every change this node has
written or heard of for the scope's accounts: marks come from write
results, the cluster's state-change broadcasts, a sinceState the client
presents, and, with more than one node, Redis. A write inside the scope
sends the rest of it to the primary. A miss on a replica is looked up on
the primary, and a replica error retries the read there and marks the
replica down.

Each node samples lag every second (WAL positions on PostgreSQL; GTID
sets or Seconds_Behind_Source on MySQL), stops reading from a replica
over 5 s and starts again under 2.5 s, and probes a down replica every
10 s. At startup a replica is left out if it's the primary, isn't
read-only, applies out of commit order, or doesn't show a marker written
to the primary within six tries.

replica_tests (postgres, STORE=PostgreSqlReplicated) runs a primary and a
streaming hot standby in containers: tests 9, 10, 12, 13, 14 and 15 pass.

crates/store/src/lib.rs

@@ -159,6 +159,8 @@ pub enum Store {
     #[cfg(feature = "rocks")]
     RocksDb(Arc<backend::rocksdb::RocksDbStore>),
     Ephemeral(Arc<EphemeralStore>),
+    // inbuxa: ST-5 to ST-15: a PostgreSQL or MySQL primary with read replicas
+    Replicated(Arc<backend::scaleout::replica::ReplicatedStore>),
     #[default]
     None,
 }
@@ -654,8 +656,8 @@ impl Store {
             #[cfg(feature = "rocks")]
             (Store::RocksDb(a), Store::RocksDb(b)) => Arc::ptr_eq(a, b),
             (Store::Ephemeral(a), Store::Ephemeral(b)) => Arc::ptr_eq(a, b),
-            #[cfg(all(feature = "enterprise", any(feature = "postgres", feature = "mysql")))]
-            (Store::SQLReadReplica(a), Store::SQLReadReplica(b)) => Arc::ptr_eq(a, b),
+            // inbuxa: ST-3
+            (Store::Replicated(a), Store::Replicated(b)) => Arc::ptr_eq(a, b),
             (Store::None, Store::None) => true,
             _ => false,
         }
@@ -664,6 +666,8 @@ impl Store {
     #[inline(always)]
     pub fn is_sql(&self) -> bool {
         match self {
+            // inbuxa: ST-3: as its primary
+            Store::Replicated(store) => store.primary.is_sql(),
             #[cfg(feature = "sqlite")]
             Store::SQLite(_) => true,
             #[cfg(feature = "postgres")]
@@ -677,6 +681,8 @@ impl Store {
     #[inline(always)]
     pub fn is_pg_or_mysql(&self) -> bool {
         match self {
+            // inbuxa: ST-3: as its primary
+            Store::Replicated(store) => store.primary.is_pg_or_mysql(),
             #[cfg(feature = "mysql")]
             Store::MySQL(_) => true,
             #[cfg(feature = "postgres")]
@@ -715,6 +721,7 @@ impl std::fmt::Debug for Store {
             #[cfg(feature = "rocks")]
             Self::RocksDb(_) => f.debug_tuple("RocksDb").finish(),
             Self::Ephemeral(_) => f.debug_tuple("Ephemeral").finish(),
+            Self::Replicated(store) => f.debug_tuple("Replicated").field(&store.primary).finish(),
 
             Self::None => f.debug_tuple("None").finish(),
         }