Scale-out storage: PostgreSQL and MySQL read replicas (ST-5 to ST-15)

A data store with readReplicas becomes a replicated store. Writes,
operator-written SQL and everything outside a read scope go to the
primary. JMAP reads before a request's first write, IMAP LIST, STATUS,
SEARCH, SORT and FETCH, POP3 RETR and TOP, DAV GET, PROPFIND and REPORT,
and blob downloads run in a read scope. Only account data (properties,
indexes, change logs, counters, ACLs, blobs, the search index) is read
from a replica; the registry, in-memory values, the task queue and the
rest stay on the primary.

In a scope, the first read picks a replica round-robin among those up
and under the lag limit, and only if it has every change this node has
written or heard of for the scope's accounts: marks come from write
results, the cluster's state-change broadcasts, a sinceState the client
presents, and, with more than one node, Redis. A write inside the scope
sends the rest of it to the primary. A miss on a replica is looked up on
the primary, and a replica error retries the read there and marks the
replica down.

Each node samples lag every second (WAL positions on PostgreSQL; GTID
sets or Seconds_Behind_Source on MySQL), stops reading from a replica
over 5 s and starts again under 2.5 s, and probes a down replica every
10 s. At startup a replica is left out if it's the primary, isn't
read-only, applies out of commit order, or doesn't show a marker written
to the primary within six tries.

replica_tests (postgres, STORE=PostgreSqlReplicated) runs a primary and a
streaming hot standby in containers: tests 9, 10, 12, 13, 14 and 15 pass.

crates/dav/src/request.rs

@@ -74,9 +74,19 @@ pub(crate) trait DavRequestDispatcher: Sync + Send {
         method: DavMethod,
         body: Vec<u8>,
     ) -> impl Future<Output = crate::Result<HttpResponse>> + Send;
+
+    fn dispatch_dav_inner(
+        &self,
+        headers: &RequestHeaders<'_>,
+        access_token: AccessToken,
+        resource: DavResourceName,
+        method: DavMethod,
+        body: Vec<u8>,
+    ) -> impl Future<Output = crate::Result<HttpResponse>> + Send;
 }
 
 impl DavRequestDispatcher for Server {
+    // inbuxa: ST-6: GET, PROPFIND and REPORT may be served by a read replica
     async fn dispatch_dav_request(
         &self,
         headers: &RequestHeaders<'_>,
@@ -84,6 +94,33 @@ impl DavRequestDispatcher for Server {
         resource: DavResourceName,
         method: DavMethod,
         body: Vec<u8>,
+    ) -> crate::Result<HttpResponse> {
+        if matches!(
+            method,
+            DavMethod::GET | DavMethod::HEAD | DavMethod::PROPFIND | DavMethod::REPORT
+        ) {
+            let accounts = access_token
+                .all_ids()
+                .map(|account_id| (account_id, 0))
+                .collect::<Vec<_>>();
+            store::backend::scaleout::replica::replica_read(
+                accounts,
+                self.dispatch_dav_inner(headers, access_token, resource, method, body),
+            )
+            .await
+        } else {
+            self.dispatch_dav_inner(headers, access_token, resource, method, body)
+                .await
+        }
+    }
+
+    async fn dispatch_dav_inner(
+        &self,
+        headers: &RequestHeaders<'_>,
+        access_token: AccessToken,
+        resource: DavResourceName,
+        method: DavMethod,
+        body: Vec<u8>,
     ) -> crate::Result<HttpResponse> {
         // Dispatch
         match method {