inbuxa/inbuxa-server
1
0
Fork 0
Code Issues Pull Requests Actions 2 Packages Projects Releases 2 Wiki Activity

Scale-out storage: PostgreSQL and MySQL read replicas (ST-5 to ST-15)

Browse Source 
A data store with readReplicas becomes a replicated store. Writes,
operator-written SQL and everything outside a read scope go to the
primary. JMAP reads before a request's first write, IMAP LIST, STATUS,
SEARCH, SORT and FETCH, POP3 RETR and TOP, DAV GET, PROPFIND and REPORT,
and blob downloads run in a read scope. Only account data (properties,
indexes, change logs, counters, ACLs, blobs, the search index) is read
from a replica; the registry, in-memory values, the task queue and the
rest stay on the primary.

In a scope, the first read picks a replica round-robin among those up
and under the lag limit, and only if it has every change this node has
written or heard of for the scope's accounts: marks come from write
results, the cluster's state-change broadcasts, a sinceState the client
presents, and, with more than one node, Redis. A write inside the scope
sends the rest of it to the primary. A miss on a replica is looked up on
the primary, and a replica error retries the read there and marks the
replica down.

Each node samples lag every second (WAL positions on PostgreSQL; GTID
sets or Seconds_Behind_Source on MySQL), stops reading from a replica
over 5 s and starts again under 2.5 s, and probes a down replica every
10 s. At startup a replica is left out if it's the primary, isn't
read-only, applies out of commit order, or doesn't show a marker written
to the primary within six tries.

replica_tests (postgres, STORE=PostgreSqlReplicated) runs a primary and a
streaming hot standby in containers: tests 9, 10, 12, 13, 14 and 15 pass.
This commit is contained in:
jcoffey-dev
2026-09-19 14:05:59 -07:00
parent a635b490ec
commit 1518c69033
24 changed files with 1722 additions and 45 deletions
Download Patch File Download Diff File Expand all files Collapse all files
tests/src/utils/containers.rs
+96
View File
@@ -32,6 +32,14 @@ static CHALLTESTSRV: OnceCell<ContainerAsync<GenericImage>> = OnceCell::const_ne
static PEBBLE: OnceCell<ContainerAsync<GenericImage>> = OnceCell::const_new();
static POWERDNS: OnceCell<ContainerAsync<GenericImage>> = OnceCell::const_new();
static SCIM_TESTER: OnceCell<ContainerAsync<GenericImage>> = OnceCell::const_new();
// inbuxa: scale-out storage (ST-5 to ST-15): a primary and a streaming replica
static PG_PRIMARY: OnceCell<ContainerAsync<GenericImage>> = OnceCell::const_new();
static PG_REPLICA: OnceCell<ContainerAsync<GenericImage>> = OnceCell::const_new();
const PG_REPLICATION_NETWORK: &str = "inbuxa-test-pg-replication";
pub const PG_PRIMARY_PORT: u16 = 5442;
pub const PG_REPLICA_PORT: u16 = 5443;
pub const PG_REPLICA_CONTAINER: &str = "inbuxa-test-pg-replica";
pub const PG_PRIMARY_CONTAINER: &str = "inbuxa-test-pg-primary";
const OPENLDAP_LDAPI_URL: &str = "ldapi://%2Fvar%2Frun%2Fslapd%2Fldapi/";
@@ -541,3 +549,91 @@ async fn wait_for_http(url: &str) {
tokio::time::sleep(Duration::from_millis(500)).await;
}
}
/// inbuxa: a PostgreSQL primary on port 5442 with a hot-standby streaming
/// replica on 5443, for the read-replica tests (ST-5 to ST-15).
pub async fn ensure_postgres_replicated() {
PG_PRIMARY
.get_or_init(|| async {
GenericImage::new("postgres", "16-alpine")
.with_wait_for(WaitFor::message_on_stderr(
"database system is ready to accept connections",
))
.with_env_var("POSTGRES_USER", "stalwart")
.with_env_var("POSTGRES_PASSWORD", "stalwart")
.with_env_var("POSTGRES_DB", "stalwart")
.with_copy_to(
"/docker-entrypoint-initdb.d/replication.sh",
b"#!/bin/sh\necho 'host replication all all scram-sha-256' >> \"$PGDATA/pg_hba.conf\"\n"
.to_vec(),
)
.with_cmd([
"postgres",
"-c",
"wal_level=replica",
"-c",
"max_wal_senders=10",
"-c",
"hot_standby=on",
])
.with_network(PG_REPLICATION_NETWORK)
.with_mapped_port(PG_PRIMARY_PORT, 5432.tcp())
.with_startup_timeout(READY_TIMEOUT)
.with_container_name(PG_PRIMARY_CONTAINER)
.with_reuse(ReuseDirective::Always)
.start()
.await
.expect("Failed to start the PostgreSQL primary")
})
.await;
wait_for_tcp(PG_PRIMARY_PORT).await;
PG_REPLICA
.get_or_init(|| async {
GenericImage::new("postgres", "16-alpine")
.with_wait_for(WaitFor::message_on_stderr(
"database system is ready to accept read-only connections",
))
.with_env_var("PGPASSWORD", "stalwart")
.with_cmd([
"sh",
"-c",
concat!(
"if [ ! -s /var/lib/postgresql/replica/PG_VERSION ]; then ",
"until pg_basebackup -h inbuxa-test-pg-primary -U stalwart ",
"-D /var/lib/postgresql/replica -R -X stream; do sleep 1; done; fi; ",
"chown -R postgres:postgres /var/lib/postgresql/replica; ",
"chmod 700 /var/lib/postgresql/replica; ",
"exec su-exec postgres postgres -D /var/lib/postgresql/replica ",
"-c hot_standby=on"
),
])
.with_network(PG_REPLICATION_NETWORK)
.with_mapped_port(PG_REPLICA_PORT, 5432.tcp())
.with_startup_timeout(READY_TIMEOUT)
.with_container_name(PG_REPLICA_CONTAINER)
.with_reuse(ReuseDirective::Always)
.start()
.await
.expect("Failed to start the PostgreSQL replica")
})
.await;
wait_for_tcp(PG_REPLICA_PORT).await;
}
/// inbuxa: runs SQL on a test container, through `psql`.
pub fn psql(container: &str, sql: &str) -> String {
let output = std::process::Command::new("docker")
.args(["exec", container, "psql", "-U", "stalwart", "-d", "stalwart", "-tAc", sql])
.output()
.expect("docker exec");
String::from_utf8_lossy(&output.stdout).trim().to_string()
}
/// inbuxa: stops or starts a test container.
pub fn docker(action: &str, container: &str) {
let status = std::process::Command::new("docker")
.args([action, container])
.status()
.expect("docker");
assert!(status.success(), "docker {action} {container}");
}