Scale-out storage: PostgreSQL and MySQL read replicas (ST-5 to ST-15)

A data store with readReplicas becomes a replicated store. Writes,
operator-written SQL and everything outside a read scope go to the
primary. JMAP reads before a request's first write, IMAP LIST, STATUS,
SEARCH, SORT and FETCH, POP3 RETR and TOP, DAV GET, PROPFIND and REPORT,
and blob downloads run in a read scope. Only account data (properties,
indexes, change logs, counters, ACLs, blobs, the search index) is read
from a replica; the registry, in-memory values, the task queue and the
rest stay on the primary.

In a scope, the first read picks a replica round-robin among those up
and under the lag limit, and only if it has every change this node has
written or heard of for the scope's accounts: marks come from write
results, the cluster's state-change broadcasts, a sinceState the client
presents, and, with more than one node, Redis. A write inside the scope
sends the rest of it to the primary. A miss on a replica is looked up on
the primary, and a replica error retries the read there and marks the
replica down.

Each node samples lag every second (WAL positions on PostgreSQL; GTID
sets or Seconds_Behind_Source on MySQL), stops reading from a replica
over 5 s and starts again under 2.5 s, and probes a down replica every
10 s. At startup a replica is left out if it's the primary, isn't
read-only, applies out of commit order, or doesn't show a marker written
to the primary within six tries.

replica_tests (postgres, STORE=PostgreSqlReplicated) runs a primary and a
streaming hot standby in containers: tests 9, 10, 12, 13, 14 and 15 pass.

crates/store/src/backend/postgres/main.rs

@@ -27,6 +27,8 @@ use utils::tls::rustls_client_config;
 
 impl PostgresStore {
     pub async fn open(config: structs::PostgreSqlStore) -> Result<Store, String> {
+        // inbuxa: ST-15: where the primary is, to tell a replica from it
+        let primary_location = (config.host.clone(), config.port as u16, config.database.clone());
         let mut cfg = Config::new();
         cfg.dbname = config.database.into();
         cfg.host = config.host.into();
@@ -57,16 +59,35 @@ impl PostgresStore {
         .map_err(|e| format!("Failed to create connection pool: {e}"))?;
         let ts_configs = discover_ts_configs(&primary_pool).await;
 
-        // inbuxa: ST-2: replicas aren't used yet (the scale-out decision), so
-        // each one is reported rather than silently ignored
+        // inbuxa: ST-5 to ST-15: each replica inherits the primary's settings
+        // except where it is and how to sign in
+        let mut replicas = vec![];
         for replica in config.read_replicas {
-            trc::event!(
-                Store(trc::StoreEvent::PostgresqlError),
-                Details = format!(
-                    "Read replica {}:{} {} isn't used yet: every operation goes to the primary",
-                    replica.host, replica.port, replica.database
-                ),
-            );
+            let mut cfg = cfg.clone();
+            cfg.dbname = replica.database.clone().into();
+            cfg.host = replica.host.clone().into();
+            cfg.user = replica.auth_username;
+            cfg.password = replica.auth_secret.secret().await?.map(|v| v.into_owned());
+            cfg.port = (replica.port as u16).into();
+            cfg.options = replica.options;
+            let pool = if config.use_tls {
+                cfg.create_pool(
+                    Some(Runtime::Tokio1),
+                    MakeRustlsConnect::new(rustls_client_config(config.allow_invalid_certs)?),
+                )
+            } else {
+                cfg.create_pool(Some(Runtime::Tokio1), NoTls)
+            }
+            .map_err(|e| format!("Failed to create connection pool: {e}"))?;
+            replicas.push(crate::backend::scaleout::replica::Replica::new(
+                Store::PostgreSQL(Arc::new(PostgresStore {
+                    conn_pool: pool,
+                    ts_configs: ts_configs.clone(),
+                })),
+                replica.host,
+                replica.port as u16,
+                replica.database,
+            ));
         }
 
         let primary = Store::PostgreSQL(Arc::new(PostgresStore {
@@ -74,8 +95,18 @@ impl PostgresStore {
             ts_configs,
         }));
 
-
-        Ok(primary)
+        // ST-1: no replicas, no change
+        if replicas.is_empty() {
+            return Ok(primary);
+        }
+        Ok(Store::Replicated(
+            crate::backend::scaleout::replica::ReplicatedStore::new(
+                primary,
+                primary_location,
+                replicas,
+                crate::backend::scaleout::replica::ReplicaKind::PostgreSql,
+            ),
+        ))
     }
 
     pub(crate) async fn create_storage_tables(&self) -> trc::Result<()> {

crates/store/src/backend/postgres/mod.rs

@@ -29,7 +29,7 @@ pub struct PostgresStore {
 }
 
 #[inline(always)]
-fn into_error(err: tokio_postgres::error::Error) -> trc::Error {
+pub(crate) fn into_error(err: tokio_postgres::error::Error) -> trc::Error {
     let mut local_err = trc::StoreEvent::PostgresqlError.reason(error_chain(&err));
     if let Some(db_err) = err.as_db_error() {
         local_err = local_err.code(db_err.code().code().to_string());
@@ -71,7 +71,7 @@ pub(crate) fn is_timeout_error(err: &tokio_postgres::Error) -> bool {
 }
 
 #[inline(always)]
-fn into_pool_error(err: deadpool_postgres::PoolError) -> trc::Error {
+pub(crate) fn into_pool_error(err: deadpool_postgres::PoolError) -> trc::Error {
     match err {
         deadpool_postgres::PoolError::Backend(err) => into_error(err),
         err => trc::StoreEvent::PostgresqlError.reason(error_chain(&err)),