inbuxa-server/docs/spec/container-tests.md

Running the suites the plain regression doesn't reach

Status: 2026-09-19.

cargo test -p tests runs everything that needs nothing but a store on disk. The fork's own feature suites are #[ignore]d on top of that, because each one needs something the plain run hasn't got: a container, a STORE the harness only builds on request, or, for the eight *_compat tests, a copy of INBUXA's data. A green regression therefore says nothing about them, and they have to be run by name.

The compat eight have their own runbook, compat-tests.md. This one covers the rest.

Before any of them

• Docker, for every suite whose "Needs" column names a container. The suites start and reuse their own containers through the ensure_* helpers in tests/src/utils/containers.rs; nothing has to be started by hand.
• RUST_MIN_STACK=8388608, or smtp::inbound::data::data and some suites overflow the 2 MiB test-thread stack.
• CARGO_TARGET_DIR=target TMPDIR=$PWD/target/tmp, a fresh TMPDIR per run.
• One at a time. Each suite starts a server on the same fixed ports, and the machine is CPU-bound: a second suite, or a build running alongside, makes the timing checks flake.

The suites

Suite	Feature	Needs	STORE
scim::scim_tests	7, SCIM	Nothing (SCIM_CONFORMANCE=1 adds the third-party clients, in a container)	RocksDb
scim::scim_oidc_tests	7 and 9 (test 5)	Keycloak	RocksDb
directory::per_domain::per_domain_directory_tests	9, per-domain directories	Nothing; needs the sqlite feature	RocksDb
store::scaleout::scaleout_blob_tests	8, scale-out storage (tests 2 to 8)	Nothing: the members are directories	RocksDb
store::scaleout::scaleout_memory_tests	8 (tests 20, 22, 23)	Redis; needs the redis feature	RocksDb
store::replica::replica_tests	8 (tests 9, 10, 12 to 15)	PostgreSQL primary and hot standby	PostgreSqlReplicated
store::replica_cluster::replica_cluster_tests	8 (test 11)	The same pair, and Redis	PostgreSqlReplicated
store::replica_mysql::mysql_replica_tests	8 (test 17)	MySQL source and replica, GTIDs	MySqlReplicated
store::replica_mysql::mysql_replica_position_tests	8 (tests 18, 19)	MySQL source and replica, by binary log position	MySqlReplicatedPosition

system::{ai,branding,masked_email,monitoring,tenant,undelete}_tests are also #[ignore]d, but only as a way to run one suite alone: the same code runs inside system_tests, which the regression does run.

Running one

CARGO_TARGET_DIR=target TMPDIR=$PWD/target/tmp RUST_MIN_STACK=8388608 \
STORE=<store> cargo test -p tests --features <backends> <suite> -- --ignored

<suite> is the test's name, such as scim_tests or replica_tests, and --exact with its full path if the name matches more than one. STORE has no default: without it the harness panics with "Missing or invalid store type" before the suite starts, so the table names one for every suite. The three replicated values are the harness's own (tests/src/utils/storage.rs); they bring up the pair they name and point the server at the primary.

For example:

STORE=PostgreSqlReplicated cargo test -p tests --features postgres,redis \
    replica_cluster_tests -- --ignored

LOG=<level> turns on the test server's own logging, which is the only way to see why a task failed rather than that it failed: LOG=error is what found the ACME fault below.

The suites the regression runs, but not with its own settings

Three of the suites a plain cargo test -p tests runs can't pass on the settings it uses. Run them by name, with these:

# Two nodes, so a shared store and a coordinator, never RocksDb
STORE=PostgreSql COORDINATOR=Redis cargo test -p tests --features postgres,redis \
    -- --exact cluster::broadcast::cluster_tests

# The spam rules the expectations were recorded against
STORE=RocksDb SPAM_RULES_URL=file:///path/to/spam-filter-rules.json.gz \
    cargo test -p tests -- --exact smtp::inbound::antispam::antispam

# The ACME pair, which keeps its containers between runs
STORE=RocksDb cargo test -p tests -- --exact automation::automation_tests

What a plain regression leaves failing

STORE=RocksDb cargo test -p tests -- --test-threads=1 was 87 passed, 3 failed, 23 ignored in 17m 26s on 2026-09-19, the run after the ACME fix. That is every one of the 113 tests a default build holds; the five the postgres, mysql and redis features add are all in the table above and all #[ignore]d. None of the three failures is a fork regression: each is the invocation or the environment rather than the code.

An earlier run the same day was recorded here as 86 passed, 4 failed, 27 ignored. That totals 117, which no feature set of this tree produces — 113 by default, 118 with all three backends — and no commit since has added or removed a test. So the two aren't the same build and shouldn't be read as a trend; the numbers above are the reference, with the command that produced them.

• smtp::inbound::antispam::antispam fails every time. Its rules URL defaults to file:///Users/me/code/spam-filter/spam-filter-rules.json.gz, a path from the upstream import that exists on no machine here, so no rules load and every tag scores 0.00 against an expectation full of weights. Give it SPAM_RULES_URL.

• cluster::broadcast::cluster_tests needs COORDINATOR=Redis (or Nats) and a store the nodes can share: it passes on STORE=PostgreSql, and can't work on RocksDb, where each node gets its own.

• smtp::outbound::lmtp::lmtp_delivery once counted three DSNs where it wanted four, and passed in this run: queue timing under a loaded sequential run, not a fault to chase.

• automation::automation_tests had failed against pebble with 400 malformed: "Cannot update challenge with status processing, only status pending", because the client re-posted the challenge on every poll. That was a real renewal bug and is fixed; this run logged no 400 at all, and the client polls as RFC 8555 section 7.5.1 says to.

  The suite still doesn't pass here: pebble never validates the TLS-ALPN challenge, so the authorizations stay pending until the client gives up and no certificate is issued — the failure is an Option::unwrap() on the certificate that never arrived (tests/src/automation/acme.rs:223). Validation needs pebble, in its container, to reach the test server's 0.0.0.0:8899 across the docker bridge, and ufw is active on this machine. That wasn't proved — standing up a listener to test it needs a permission this session didn't have — so before reading anything into an ACME failure, check that path first.

The last sweep

All nine were re-run on 2026-09-19, one at a time, after docker rm -f on every inbuxa-test- and stalwart-test- container, so each suite built its own from scratch. All nine pass:

Suite	Time
scim_tests	21s
scim_oidc_tests	23s
per_domain_directory_tests	4s
scaleout_blob_tests	1s
scaleout_memory_tests	1s
replica_tests	15s
replica_cluster_tests	12s
mysql_replica_tests	36s
mysql_replica_position_tests	27s, failed; 7s on a second run

mysql_replica_position_tests was the one failure, on test 18's first assertion: the replica's lag has to be under five seconds, and Seconds_Behind_Source is still above that while a pair seeded half a minute earlier catches up. It failed before the point where the suite changes the replica's settings, so nothing was left to restore, and it passed on a second run against the same, now warm, pair.

When one fails

The containers are reused between runs, so a suite that fails in a way its own assertions don't explain is worth re-running once against fresh ones: docker rm -f the containers it names (they are all prefixed inbuxa-test-) and run it again. A replica pair left paused or stopped by an interrupted run is the usual cause — replica_tests pauses WAL replay, and mysql_replica_position_tests changes the replica's parallel-apply settings, both of which they restore only if they finish.

A pair built fresh has the opposite problem: it is still catching up, and a lag assertion can fail on a replica that is working perfectly well. Give the suite a second run before reading anything into it.