PostgreSQL: pipeline the statements of a write batch

jcoffey-dev commented

2026-10-08 05:56:11 +00:00

Owner

First of the cluster-shape optimizations (backlog item D).

Upstream awaits each statement of a write batch before building the next. On a single node that costs nothing; on the cluster every await is a network round trip over the mesh, and a batch of a few hundred operations (an ingest with its index and log entries) holds the account's change-id row locked for all of them.

Statements that need no answer (sets, index inserts/deletes, log entries, atomic adds, clears) are now queued on a FuturesOrdered, each polled once on push so it goes out immediately and in operation order; their replies are read at the next flush. PostgreSQL still runs them in the order sent. Operations that need a reply to continue (AssertValue, MergeFnc, AddAndGet) drain the queue first, so the first error reported is the real one and not current transaction is aborted. The change-id bumps for the accounts in a batch go out together as well. Set/SetFnc keep the zero-rows-means-assert-failed check, inside the queued future.

batch.ops is iterated immutably now (nothing in the loop mutated it), which lets Set values be borrowed into the queue rather than copied.

Verified: cargo test -p tests --lib --features postgres -- store::store_tests with STORE=PostgreSql passes (24 s). Fork checks clean. Production before-numbers, message-ingest.ham elapsed over the last three days of logs: host1 p50 48 ms / p90 127 ms (n=49), host2 p50 42 ms / p90 127 ms (n=110). After the roll I'll post the same numbers here.

First of the cluster-shape optimizations (backlog item D). Upstream awaits each statement of a write batch before building the next. On a single node that costs nothing; on the cluster every await is a network round trip over the mesh, and a batch of a few hundred operations (an ingest with its index and log entries) holds the account's change-id row locked for all of them. Statements that need no answer (sets, index inserts/deletes, log entries, atomic adds, clears) are now queued on a `FuturesOrdered`, each polled once on push so it goes out immediately and in operation order; their replies are read at the next flush. PostgreSQL still runs them in the order sent. Operations that need a reply to continue (`AssertValue`, `MergeFnc`, `AddAndGet`) drain the queue first, so the first error reported is the real one and not `current transaction is aborted`. The change-id bumps for the accounts in a batch go out together as well. Set/SetFnc keep the zero-rows-means-assert-failed check, inside the queued future. `batch.ops` is iterated immutably now (nothing in the loop mutated it), which lets `Set` values be borrowed into the queue rather than copied. **Verified:** `cargo test -p tests --lib --features postgres -- store::store_tests` with `STORE=PostgreSql` passes (24 s). Fork checks clean. Production before-numbers, `message-ingest.ham elapsed` over the last three days of logs: host1 p50 48 ms / p90 127 ms (n=49), host2 p50 42 ms / p90 127 ms (n=110). After the roll I'll post the same numbers here.

jcoffey-dev added 1 commit 2026-10-08 05:56:11 +00:00

PostgreSQL: pipeline the statements of a write batch

ci / fork-checks (pull_request) Skipped

Details

ci / build (pull_request) Skipped

Details

github/ci (branch) GitHub Actions

Details

ci / github (pull_request) Successful in 7m45s

Details

6e53461075

Upstream awaits each statement of a write batch before building the next,
which costs nothing on a single node and one network round trip each on
the cluster; a batch of a few hundred operations held the account's
change-id row locked for all of them. Statements that need no answer are
now sent as soon as they are built and their replies read later, in
order; PostgreSQL still runs them in the order sent. Operations that need
a reply to continue (asserts, merges, AddAndGet) drain the queue first,
so the first error reported is the real one. The change-id bumps for the
accounts in a batch go out together too.

Verified with the tests crate's store suite against PostgreSQL.

jcoffey-dev merged commit 32b125ea96 into main

2026-10-08 06:31:22 +00:00

jcoffey-dev deleted branch perf/pg-write-pipeline

2026-10-08 06:31:22 +00:00

jcoffey-dev commented

2026-10-08 06:31:25 +00:00

Author

Owner

Suites against PostgreSQL (STORE=PostgreSql, local container): store 12 s pass, IMAP 43 s pass, JMAP one compliance failure, email/filter-header-name-value ("special-headers should match"). The same suite with main's write.rs on the same PostgreSQL fails on exactly that one test, so it's pre-existing on the PostgreSQL FTS path (the name-only header filter passes), not from this change. Everything else in the JMAP suite passes on both.

Suites against PostgreSQL (`STORE=PostgreSql`, local container): store 12 s pass, IMAP 43 s pass, JMAP one compliance failure, `email/filter-header-name-value` ("special-headers should match"). The same suite with main's `write.rs` on the same PostgreSQL fails on exactly that one test, so it's pre-existing on the PostgreSQL FTS path (the name-only header filter passes), not from this change. Everything else in the JMAP suite passes on both.

jcoffey-dev referenced this pull request

2026-10-08 06:32:11 +00:00

Release 2026.10.8 #168

jcoffey-dev referenced this pull request

2026-10-08 18:13:23 +00:00

Read message archives in batches #170

jcoffey-dev commented

2026-10-09 23:53:14 +00:00

Author

Owner

A day after the roll: ham ingest p50 is 37 ms on host1 and 34 ms on host2 (was 48 and 42), p90 99 and 108 ms (was 127). host3 only took two messages, so nothing to read there. Smaller sample than the before numbers, and the batched reads from 2026.10.9 are in there too, but it's moving the right way.

PostgreSQL: pipeline the statements of a write batch #167