Scale-out storage: PostgreSQL and MySQL read replicas (ST-5 to ST-15)

A data store with readReplicas becomes a replicated store. Writes,
operator-written SQL and everything outside a read scope go to the
primary. JMAP reads before a request's first write, IMAP LIST, STATUS,
SEARCH, SORT and FETCH, POP3 RETR and TOP, DAV GET, PROPFIND and REPORT,
and blob downloads run in a read scope. Only account data (properties,
indexes, change logs, counters, ACLs, blobs, the search index) is read
from a replica; the registry, in-memory values, the task queue and the
rest stay on the primary.

In a scope, the first read picks a replica round-robin among those up
and under the lag limit, and only if it has every change this node has
written or heard of for the scope's accounts: marks come from write
results, the cluster's state-change broadcasts, a sinceState the client
presents, and, with more than one node, Redis. A write inside the scope
sends the rest of it to the primary. A miss on a replica is looked up on
the primary, and a replica error retries the read there and marks the
replica down.

Each node samples lag every second (WAL positions on PostgreSQL; GTID
sets or Seconds_Behind_Source on MySQL), stops reading from a replica
over 5 s and starts again under 2.5 s, and probes a down replica every
10 s. At startup a replica is left out if it's the primary, isn't
read-only, applies out of commit order, or doesn't show a marker written
to the primary within six tries.

replica_tests (postgres, STORE=PostgreSqlReplicated) runs a primary and a
streaming hot standby in containers: tests 9, 10, 12, 13, 14 and 15 pass.

tests/src/utils/storage.rs

@@ -62,6 +62,35 @@ impl RegistryEnvStores for RegistryStore {
 }
 
 pub async fn build_data_store(typ: &str, path: &str) -> DataStore {
+    // inbuxa: scale-out storage: a primary with a streaming read replica
+    if typ == "PostgreSqlReplicated" {
+        crate::utils::containers::ensure_postgres_replicated().await;
+        let secret = || {
+            SecretKeyOptional::Value(SecretKeyValue {
+                secret: "stalwart".into(),
+            })
+        };
+        return DataStore::PostgreSql(PostgreSqlStore {
+            host: "localhost".into(),
+            port: crate::utils::containers::PG_PRIMARY_PORT as u64,
+            auth_username: "stalwart".to_string().into(),
+            auth_secret: secret(),
+            database: "stalwart".into(),
+            use_tls: false,
+            allow_invalid_certs: true,
+            read_replicas: registry::types::list::List::from_iter([
+                registry::schema::structs::PostgreSqlSettings {
+                    host: "localhost".into(),
+                    port: crate::utils::containers::PG_REPLICA_PORT as u64,
+                    database: "stalwart".into(),
+                    auth_username: "stalwart".to_string().into(),
+                    auth_secret: secret(),
+                    options: None,
+                },
+            ]),
+            ..Default::default()
+        });
+    }
     if typ == "MariaDb" {
         crate::utils::containers::ensure_mariadb().await;
         return DataStore::MySql(MySqlStore {