inbuxa-server/docs/spec/container-tests.md

# Running the suites the plain regression doesn't reach

Status: 2026-09-19.

`cargo test -p tests` runs everything that needs nothing but a store on
disk. The fork's own feature suites are `#[ignore]`d on top of that,
because each one needs something the plain run hasn't got: a container, a
`STORE` the harness only builds on request, or, for the eight `*_compat`
tests, a copy of INBUXA's data. A green regression therefore says nothing
about them, and they have to be run by name.

The compat eight have their own runbook, `compat-tests.md`. This one
covers the rest.

## Before any of them

- Docker, for every suite whose "Needs" column names a container. The
  suites start and reuse their own containers through the `ensure_*`
  helpers in `tests/src/utils/containers.rs`; nothing has to be started by
  hand.
- `RUST_MIN_STACK=8388608`, or `smtp::inbound::data::data` and some suites
  overflow the 2 MiB test-thread stack.
- `CARGO_TARGET_DIR=target TMPDIR=$PWD/target/tmp`, a fresh `TMPDIR` per
  run.
- **One at a time.** Each suite starts a server on the same fixed ports,
  and the machine is CPU-bound: a second suite, or a build running
  alongside, makes the timing checks flake.

## The suites

| Suite | Feature | Needs | `STORE` |
|---|---|---|---|
| `scim::scim_tests` | 7, SCIM | Nothing (`SCIM_CONFORMANCE=1` adds the third-party clients, in a container) | `RocksDb` |
| `scim::scim_oidc_tests` | 7 and 9 (test 5) | Keycloak | `RocksDb` |
| `directory::per_domain::per_domain_directory_tests` | 9, per-domain directories | Nothing; needs the `sqlite` feature | `RocksDb` |
| `store::scaleout::scaleout_blob_tests` | 8, scale-out storage (tests 2 to 8) | Nothing: the members are directories | `RocksDb` |
| `store::scaleout::scaleout_memory_tests` | 8 (tests 20, 22, 23) | Redis; needs the `redis` feature | `RocksDb` |
| `store::replica::replica_tests` | 8 (tests 9, 10, 12 to 15) | PostgreSQL primary and hot standby | `PostgreSqlReplicated` |
| `store::replica_cluster::replica_cluster_tests` | 8 (test 11) | The same pair, and Redis | `PostgreSqlReplicated` |
| `store::replica_mysql::mysql_replica_tests` | 8 (test 17) | MySQL source and replica, GTIDs | `MySqlReplicated` |
| `store::replica_mysql::mysql_replica_position_tests` | 8 (tests 18, 19) | MySQL source and replica, by binary log position | `MySqlReplicatedPosition` |

`system::{ai,branding,masked_email,monitoring,tenant,undelete}_tests` are
also `#[ignore]`d, but only as a way to run one suite alone: the same code
runs inside `system_tests`, which the regression does run.

## Running one

```
CARGO_TARGET_DIR=target TMPDIR=$PWD/target/tmp RUST_MIN_STACK=8388608 \
STORE=<store> cargo test -p tests --features <backends> <suite> -- --ignored
```

`<suite>` is the test's name, such as `scim_tests` or `replica_tests`, and
`--exact` with its full path if the name matches more than one. `STORE` has
no default: without it the harness panics with "Missing or invalid store
type" before the suite starts, so the table names one for every suite. The
three replicated values are the harness's own
(`tests/src/utils/storage.rs`); they bring up the pair they name and point
the server at the primary.

For example:

```
STORE=PostgreSqlReplicated cargo test -p tests --features postgres,redis \
    replica_cluster_tests -- --ignored
```

`LOG=<level>` turns on the test server's own logging, which is the only way
to see why a task failed rather than that it failed: `LOG=error` is what
found the ACME fault below.

## The suites the regression runs, but not with its own settings

Three of the suites a plain `cargo test -p tests` runs can't pass on the
settings it uses. Run them by name, with these:

```
# Two nodes, so a shared store and a coordinator, never RocksDb
STORE=PostgreSql COORDINATOR=Redis cargo test -p tests --features postgres,redis \
    -- --exact cluster::broadcast::cluster_tests

# The spam rules the expectations were recorded against
STORE=RocksDb SPAM_RULES_URL=file:///path/to/spam-filter-rules.json.gz \
    cargo test -p tests -- --exact smtp::inbound::antispam::antispam

# The ACME pair, which keeps its containers between runs
STORE=RocksDb cargo test -p tests -- --exact automation::automation_tests
```

## What a plain regression leaves failing

`STORE=RocksDb cargo test -p tests -- --test-threads=1` was 87 passed, 3
failed, 23 ignored in 17m 26s on 2026-09-19, the run after the ACME fix.
That is every one of the 113 tests a default build holds; the five the
`postgres`, `mysql` and `redis` features add are all in the table above and
all `#[ignore]`d. None of the three failures is a fork regression: each is
the invocation or the environment rather than the code.

An earlier run the same day was recorded here as 86 passed, 4 failed, 27
ignored. That totals 117, which no feature set of this tree produces — 113
by default, 118 with all three backends — and no commit since has added or
removed a test. So the two aren't the same build and shouldn't be read as a
trend; the numbers above are the reference, with the command that produced
them.

- `smtp::inbound::antispam::antispam` fails every time. Its rules URL
  defaults to `file:///Users/me/code/spam-filter/spam-filter-rules.json.gz`,
  a path from the upstream import that exists on no machine here, so no
  rules load and every tag scores 0.00 against an expectation full of
  weights. Give it `SPAM_RULES_URL`.
- `cluster::broadcast::cluster_tests` needs `COORDINATOR=Redis` (or `Nats`)
  and a store the nodes can share: it passes on `STORE=PostgreSql`, and
  can't work on RocksDb, where each node gets its own.
- `smtp::outbound::lmtp::lmtp_delivery` once counted three DSNs where it
  wanted four, and passed in this run: queue timing under a loaded
  sequential run, not a fault to chase.
- `automation::automation_tests` had failed against pebble with
  `400 malformed: "Cannot update challenge with status processing, only
  status pending"`, because the client re-posted the challenge on every
  poll. That was a real renewal bug and is fixed; this run logged no 400 at
  all, and the client polls as RFC 8555 section 7.5.1 says to.

  The suite still doesn't pass here. Half of the reason first recorded on
  this page is wrong: pebble does validate, and the authorizations do not
  stay pending. Whether `ufw` is in the way is still open. Checked on
  2026-09-19:

  - **TCP reaches the host.** From containers on `stalwart-test-acme`, on
    two different images, `nc` to the host's gateway addresses connects on
    port 22 and is refused *immediately* on every closed port tried (8899,
    9, 14000). A `ufw` DROP would hang until the timeout instead. No
    firewall rule was read or changed to establish this — `ufw status`
    needs a privilege this session doesn't have.

    **But the network question is not settled, and the probes disagree.**
    `openssl s_client` against the same closed 8899 hung for its full
    timeout instead of reporting the refusal `nc` had just seen. Worse,
    during a live run — when the suite certainly is listening on 8899, it
    takes `with_default_listeners()` — twelve probes from a container
    completed no handshake at all, with or without `-alpn acme-tls/1`,
    while the same `openssl` connects to pebble's own TLS port from the
    same container and prints its certificate.

    A listener that is up but unreachable from a container is what a `ufw`
    DROP looks like; an instant refusal when nothing listens is not. Both
    were observed, minutes apart. Settling it needs `sudo ufw status
    verbose`, and a listener bound by hand on `0.0.0.0:8899` to probe
    against — neither of which this session could do. Until then the
    original `ufw` suspicion is neither confirmed nor dismissed.
  - **Pebble does validate.** Its log shows 20 validation attempts in the
    regression run, five for each of `autoconfig`, `autodiscover`,
    `mta-sts` and `ua-auto-config.tls.org`, and it then sets each
    authorization `INVALID by completed challenge` and the order `INVALID`.
    The challenges are answered and refused, not left pending.
  - So no certificate is issued and the test ends on an `Option::unwrap()`
    of the certificate that never arrived
    (`tests/src/automation/acme.rs:223`). A second run with `LOG=error`
    reproduced it exactly, in 84s.

  What that leaves is the TLS-ALPN handshake itself. The responder is
  upstream's and intact (`ACME_TLS_ALPN_NAME` in
  `crates/common/src/network/acme/resolver.rs`), and `listen.rs` decides
  per accepted connection whether to offer it, from
  `has_acme_tls_providers()` — which reads `has_acme_tls_challenge`,
  computed when the network config is parsed. **Unproved hypothesis:** the
  test adds its TLS-ALPN provider after the server is up, so if nothing
  recomputes that flag, the listener never offers `acme-tls/1` and every
  challenge fails exactly as observed. Worth testing before anything else.
  An `openssl s_client -alpn acme-tls/1` against `:8899` during a renewal
  was tried and proved nothing: the same probe hangs against a port with
  nothing behind it, so its silence says nothing about ALPN. A probe that
  first shows a handshake against a known-good endpoint is needed.

## The last sweep

All nine were re-run on 2026-09-19, one at a time, after `docker rm -f` on
every `inbuxa-test-` and `stalwart-test-` container, so each suite built
its own from scratch. All nine pass:

| Suite | Time |
|---|---|
| `scim_tests` | 21s |
| `scim_oidc_tests` | 23s |
| `per_domain_directory_tests` | 4s |
| `scaleout_blob_tests` | 1s |
| `scaleout_memory_tests` | 1s |
| `replica_tests` | 15s |
| `replica_cluster_tests` | 12s |
| `mysql_replica_tests` | 36s |
| `mysql_replica_position_tests` | 27s, failed; 7s on a second run |

`mysql_replica_position_tests` was the one failure, on test 18's first
assertion: the replica's lag has to be under five seconds, and
`Seconds_Behind_Source` is still above that while a pair seeded half a
minute earlier catches up. It failed before the point where the suite
changes the replica's settings, so nothing was left to restore, and it
passed on a second run against the same, now warm, pair.

## When one fails

The containers are reused between runs, so a suite that fails in a way its
own assertions don't explain is worth re-running once against fresh ones:
`docker rm -f` the containers it names (they are all prefixed
`inbuxa-test-`) and run it again. A replica pair left paused or stopped by
an interrupted run is the usual cause — `replica_tests` pauses WAL replay,
and `mysql_replica_position_tests` changes the replica's parallel-apply
settings, both of which they restore only if they finish.

A pair built fresh has the opposite problem: it is still catching up, and a
lag assertion can fail on a replica that is working perfectly well. Give
the suite a second run before reading anything into it.