Scale-out storage: PostgreSQL and MySQL read replicas (ST-5 to ST-15)

A data store with readReplicas becomes a replicated store. Writes,
operator-written SQL and everything outside a read scope go to the
primary. JMAP reads before a request's first write, IMAP LIST, STATUS,
SEARCH, SORT and FETCH, POP3 RETR and TOP, DAV GET, PROPFIND and REPORT,
and blob downloads run in a read scope. Only account data (properties,
indexes, change logs, counters, ACLs, blobs, the search index) is read
from a replica; the registry, in-memory values, the task queue and the
rest stay on the primary.

In a scope, the first read picks a replica round-robin among those up
and under the lag limit, and only if it has every change this node has
written or heard of for the scope's accounts: marks come from write
results, the cluster's state-change broadcasts, a sinceState the client
presents, and, with more than one node, Redis. A write inside the scope
sends the rest of it to the primary. A miss on a replica is looked up on
the primary, and a replica error retries the read there and marks the
replica down.

Each node samples lag every second (WAL positions on PostgreSQL; GTID
sets or Seconds_Behind_Source on MySQL), stops reading from a replica
over 5 s and starts again under 2.5 s, and probes a down replica every
10 s. At startup a replica is left out if it's the primary, isn't
read-only, applies out of commit order, or doesn't show a marker written
to the primary within six tries.

replica_tests (postgres, STORE=PostgreSqlReplicated) runs a primary and a
streaming hot standby in containers: tests 9, 10, 12, 13, 14 and 15 pass.

crates/store/src/dispatch/blob.rs

@@ -145,6 +145,33 @@ impl BlobStore {
                 #[cfg(feature = "rocks")]
                 Store::RocksDb(store) => store.get_blob(key, 0..usize::MAX).await,
                 Store::Ephemeral(store) => store.get_blob(key, 0..usize::MAX).await,
+                // inbuxa: ST-9: a replica, then the primary for what it lacks
+                Store::Replicated(store) => match store.read_target(crate::SUBSPACE_BLOBS).await {
+                    Some(index) => match crate::sql_backend!(
+                        &store.replicas[index].store,
+                        db => db.get_blob(key, 0..usize::MAX).await
+                    ) {
+                        Ok(Some(data)) => {
+                            store.served(index);
+                            Ok(Some(data))
+                        }
+                        Ok(None) => crate::sql_backend!(
+                            &store.primary,
+                            db => db.get_blob(key, 0..usize::MAX).await
+                        ),
+                        Err(err) => {
+                            store.failed(index, err);
+                            crate::sql_backend!(
+                                &store.primary,
+                                db => db.get_blob(key, 0..usize::MAX).await
+                            )
+                        }
+                    },
+                    None => crate::sql_backend!(
+                        &store.primary,
+                        db => db.get_blob(key, 0..usize::MAX).await
+                    ),
+                },
                 Store::None => Err(trc::StoreEvent::NotConfigured.into()),
             },
             BlobStore::Fs(store) => store.get_blob(key, 0..usize::MAX).await,
@@ -171,6 +198,9 @@ impl BlobStore {
                 #[cfg(feature = "rocks")]
                 Store::RocksDb(store) => store.put_blob(key, data).await,
                 Store::Ephemeral(store) => store.put_blob(key, data).await,
+                Store::Replicated(store) => {
+                    crate::sql_backend!(&store.primary, db => db.put_blob(key, data).await)
+                }
                 Store::None => Err(trc::StoreEvent::NotConfigured.into()),
             },
             BlobStore::Fs(store) => store.put_blob(key, data).await,
@@ -197,6 +227,9 @@ impl BlobStore {
                 #[cfg(feature = "rocks")]
                 Store::RocksDb(store) => store.delete_blob(key).await,
                 Store::Ephemeral(store) => store.delete_blob(key).await,
+                Store::Replicated(store) => {
+                    crate::sql_backend!(&store.primary, db => db.delete_blob(key).await)
+                }
                 Store::None => Err(trc::StoreEvent::NotConfigured.into()),
             },
             BlobStore::Fs(store) => store.delete_blob(key).await,