Scale-out storage: PostgreSQL and MySQL read replicas (ST-5 to ST-15)

A data store with readReplicas becomes a replicated store. Writes,
operator-written SQL and everything outside a read scope go to the
primary. JMAP reads before a request's first write, IMAP LIST, STATUS,
SEARCH, SORT and FETCH, POP3 RETR and TOP, DAV GET, PROPFIND and REPORT,
and blob downloads run in a read scope. Only account data (properties,
indexes, change logs, counters, ACLs, blobs, the search index) is read
from a replica; the registry, in-memory values, the task queue and the
rest stay on the primary.

In a scope, the first read picks a replica round-robin among those up
and under the lag limit, and only if it has every change this node has
written or heard of for the scope's accounts: marks come from write
results, the cluster's state-change broadcasts, a sinceState the client
presents, and, with more than one node, Redis. A write inside the scope
sends the rest of it to the primary. A miss on a replica is looked up on
the primary, and a replica error retries the read there and marks the
replica down.

Each node samples lag every second (WAL positions on PostgreSQL; GTID
sets or Seconds_Behind_Source on MySQL), stops reading from a replica
over 5 s and starts again under 2.5 s, and probes a down replica every
10 s. At startup a replica is left out if it's the primary, isn't
read-only, applies out of commit order, or doesn't show a marker written
to the primary within six tries.

replica_tests (postgres, STORE=PostgreSqlReplicated) runs a primary and a
streaming hot standby in containers: tests 9, 10, 12, 13, 14 and 15 pass.

crates/store/src/backend/mysql/main.rs

@@ -20,6 +20,8 @@ use mysql_async::{
 
 impl MysqlStore {
     pub async fn open(config: structs::MySqlStore) -> Result<Store, String> {
+        // inbuxa: ST-15: where the primary is, to tell a replica from it
+        let primary_location = (config.host.clone(), config.port as u16, config.database.clone());
         let mut opts = OptsBuilder::default()
             .ip_or_hostname(config.host)
             .user(config.auth_username)
@@ -51,24 +53,43 @@ impl MysqlStore {
             PoolOpts::default().with_constraints(PoolConstraints::new(pool_min, pool_max).unwrap()),
         );
 
-        // inbuxa: ST-2: replicas aren't used yet (the scale-out decision), so
-        // each one is reported rather than silently ignored
+        // inbuxa: ST-5 to ST-15: each replica inherits the primary's settings
+        // except where it is and how to sign in
+        let mut replicas = vec![];
         for replica in config.read_replicas {
-            trc::event!(
-                Store(trc::StoreEvent::MysqlError),
-                Details = format!(
-                    "Read replica {}:{} {} isn't used yet: every operation goes to the primary",
-                    replica.host, replica.port, replica.database
-                ),
-            );
+            replicas.push(crate::backend::scaleout::replica::Replica::new(
+                Store::MySQL(Arc::new(MysqlStore {
+                    conn_pool: Pool::new(
+                        opts.clone()
+                            .ip_or_hostname(replica.host.clone())
+                            .user(replica.auth_username)
+                            .pass(replica.auth_secret.secret().await?.map(|v| v.into_owned()))
+                            .db_name(Some(replica.database.clone()))
+                            .tcp_port(replica.port as u16),
+                    ),
+                })),
+                replica.host,
+                replica.port as u16,
+                replica.database,
+            ));
         }
 
         let primary = Store::MySQL(Arc::new(MysqlStore {
             conn_pool: Pool::new(opts),
         }));
 
-
-        Ok(primary)
+        // ST-1: no replicas, no change
+        if replicas.is_empty() {
+            return Ok(primary);
+        }
+        Ok(Store::Replicated(
+            crate::backend::scaleout::replica::ReplicatedStore::new(
+                primary,
+                primary_location,
+                replicas,
+                crate::backend::scaleout::replica::ReplicaKind::MySql,
+            ),
+        ))
     }
 
     pub(crate) async fn create_storage_tables(&self) -> trc::Result<()> {

crates/store/src/backend/mysql/mod.rs

@@ -26,7 +26,7 @@ pub struct MysqlStore {
 }
 
 #[inline(always)]
-fn into_error(err: impl Display) -> trc::Error {
+pub(crate) fn into_error(err: impl Display) -> trc::Error {
     trc::StoreEvent::MysqlError.reason(err)
 }