inbuxa/inbuxa-server

Fork 0

SQL queries time out; readiness follows the data store #45

Merged

jcoffey-dev merged 1 commits from fix/query-timeouts into main

2026-09-25 00:45:10 +00:00

jcoffey-dev commented

2026-09-24 23:46:22 +00:00

Owner

Problem

In cluster rehearsal 3, PostgreSQL was paused with docker pause. Its kernel kept answering TCP keepalives, so the connections stayed up. Requests on connections that were already checked out hung until PostgreSQL came back. /healthz/ready stayed 200 for the whole outage.

#41 bounded getting a connection, not using one.

Change

Client-side query limits (`store::backend::query_timeout`)

Every operation on a PostgreSQL or MySQL connection now runs under a time limit.

Why client-side. A server-side limit can't handle this case: the server that would enforce it is the one that stopped answering. That covers PostgreSQL's statement_timeout and MySQL's MAX_EXECUTION_TIME, which only covers SELECTs anyway.

What happens on timeout. The connection is closed rather than returned to the pool. A query may still be in flight on it, or a transaction left open.

PostgreSQL: deadpool's Object::take.
MySQL: Conn::disconnect. It marks the connection closed before it sends anything, so the pool discards it even when the server never answers.

The limits:

Limit	Applies to	Why this value
query, 2 min	reads; writes (the whole transaction, retries included); blobs; SQL lookups; search queries; indexing	These take milliseconds. Two minutes still leaves room for a large blob over a slow link, and ends a hang.
maintenance, 30 min	range deletes (account removal, purges), unindexing, `purge_store`, creating tables and indexes at startup	These can legitimately run long in a single statement. Their existing chunked fallback for server-side statement timeouts is unchanged.
per wait in `iterate`, 2 min	exports, reindexing, maintenance scans	A scan can run for hours. So the limit bounds each wait for the database (preparing, the query starting, the next row), not the whole scan.

Configuration. The limits are fixed, like #41's pool timeouts, because the DataStore schema has no field for them. Tests set them with Store::with_query_timeouts, which exists in test_mode only. Making them configurable needs a schema change; say if you want that.

Two things the limits don't change:

Returned MySQL connections. mysql_async resets a connection when it goes back to the pool. A connection returned during an outage waits in that reset, and taking one from the pool is still bounded by #41's 30 s wait.
Read-replica health probes. They still run without these limits.

Readiness

/healthz/ready used to answer 200 whenever a data store was configured. Now:

It reads one key from the data store with a 2 s limit.
It reuses the answer for 2 s, so probes can't load the database. While one probe is running, other probes get the last answer.
The first failed probe of an outage is logged.
With read replicas, the probe read may be served by a replica.

/healthz/live stays 200. Restarting a node doesn't bring its database back, and an orchestrator that restarts on failed liveness would restart every node at once. The container HEALTHCHECK already uses /healthz/live.

Tests

All in store::pool_timeout. A proxy that stops forwarding while keeping its connections open plays the paused database.

postgres_query_timeout and mysql_query_timeout (new): pass.

Setup: four pooled connections open, test limit 2 s.
During the pause, a read, a scan and a write each fail with Query timed out 2.0 s after the pause.
Once the proxy forwards again, the store answers.
With the limits set to an hour (upstream's behavior), the read was still waiting at the test's 20 s cutoff.

postgres_readiness (new, STORE=PostgreSql): pass. A running node's data store is pointed through the proxy:

/healthz/ready answers 200.
About 4 s after the pause it answers 503, while /healthz/live stays 200.
About 2 s after the pause ends it answers 200 again.

postgres_pool_timeout and mysql_pool_timeout: pass, as before.

Other suites:

store::store_tests on PostgreSql and on MySql: pass. This includes the MariaDB statement-timeout step, which exercises the chunked fallbacks under the new wrappers.
store::task_locks on PostgreSql: pass.
store::search_tests on PostgreSql: fails at the same ordering assertion (query.rs:684) as on main.

Command:

STORE=PostgreSql cargo test -p tests --features postgres,mysql store::pool_timeout -- --test-threads=1

## Problem In cluster rehearsal 3, PostgreSQL was paused with `docker pause`. Its kernel kept answering TCP keepalives, so the connections stayed up. Requests on connections that were already checked out hung until PostgreSQL came back. `/healthz/ready` stayed 200 for the whole outage. #41 bounded *getting* a connection, not *using* one. ## Change ### Client-side query limits (`store::backend::query_timeout`) Every operation on a PostgreSQL or MySQL connection now runs under a time limit. **Why client-side.** A server-side limit can't handle this case: the server that would enforce it is the one that stopped answering. That covers PostgreSQL's `statement_timeout` and MySQL's `MAX_EXECUTION_TIME`, which only covers SELECTs anyway. **What happens on timeout.** The connection is closed rather than returned to the pool. A query may still be in flight on it, or a transaction left open. - PostgreSQL: deadpool's `Object::take`. - MySQL: `Conn::disconnect`. It marks the connection closed before it sends anything, so the pool discards it even when the server never answers. **The limits:** | Limit | Applies to | Why this value | |---|---|---| | **query**, 2 min | reads; writes (the whole transaction, retries included); blobs; SQL lookups; search queries; indexing | These take milliseconds. Two minutes still leaves room for a large blob over a slow link, and ends a hang. | | **maintenance**, 30 min | range deletes (account removal, purges), unindexing, `purge_store`, creating tables and indexes at startup | These can legitimately run long in a single statement. Their existing chunked fallback for server-side statement timeouts is unchanged. | | **per wait** in `iterate`, 2 min | exports, reindexing, maintenance scans | A scan can run for hours. So the limit bounds each wait for the database (preparing, the query starting, the next row), not the whole scan. | **Configuration.** The limits are fixed, like #41's pool timeouts, because the DataStore schema has no field for them. Tests set them with `Store::with_query_timeouts`, which exists in `test_mode` only. Making them configurable needs a schema change; say if you want that. **Two things the limits don't change:** - **Returned MySQL connections.** mysql_async resets a connection when it goes back to the pool. A connection returned during an outage waits in that reset, and taking one from the pool is still bounded by #41's 30 s wait. - **Read-replica health probes.** They still run without these limits. ### Readiness `/healthz/ready` used to answer 200 whenever a data store was configured. Now: - It reads one key from the data store with a 2 s limit. - It reuses the answer for 2 s, so probes can't load the database. While one probe is running, other probes get the last answer. - The first failed probe of an outage is logged. - With read replicas, the probe read may be served by a replica. `/healthz/live` stays 200. Restarting a node doesn't bring its database back, and an orchestrator that restarts on failed liveness would restart every node at once. The container `HEALTHCHECK` already uses `/healthz/live`. ## Tests All in `store::pool_timeout`. A proxy that stops forwarding while keeping its connections open plays the paused database. **`postgres_query_timeout` and `mysql_query_timeout` (new): pass.** - Setup: four pooled connections open, test limit 2 s. - During the pause, a read, a scan and a write each fail with `Query timed out` 2.0 s after the pause. - Once the proxy forwards again, the store answers. - With the limits set to an hour (upstream's behavior), the read was still waiting at the test's 20 s cutoff. **`postgres_readiness` (new, `STORE=PostgreSql`): pass.** A running node's data store is pointed through the proxy: - `/healthz/ready` answers 200. - About 4 s after the pause it answers 503, while `/healthz/live` stays 200. - About 2 s after the pause ends it answers 200 again. **`postgres_pool_timeout` and `mysql_pool_timeout`: pass**, as before. **Other suites:** - `store::store_tests` on PostgreSql and on MySql: pass. This includes the MariaDB statement-timeout step, which exercises the chunked fallbacks under the new wrappers. - `store::task_locks` on PostgreSql: pass. - `store::search_tests` on PostgreSql: fails at the same ordering assertion (`query.rs:684`) as on `main`. Command: ``` STORE=PostgreSql cargo test -p tests --features postgres,mysql store::pool_timeout -- --test-threads=1 ```

jcoffey-dev added 1 commit 2026-09-24 23:46:23 +00:00

SQL queries time out; readiness follows the data store

ci / fork-checks (pull_request) Successful in 17s

Details

ci / build (pull_request) Successful in 7m13s

Details

08f29926d4

Cluster rehearsal 3: with PostgreSQL paused (docker pause, so its
kernel still answered TCP keepalives), requests on connections already
checked out hung until it came back, and /healthz/ready stayed 200
through the outage. #41 bounded getting a connection, not using one.

Client-side query limits (store::backend::query_timeout). Every
operation on a PostgreSQL or MySQL connection now runs under a time
limit. A server-side statement_timeout (or MySQL's MAX_EXECUTION_TIME,
which covers SELECTs only) can't do this: the server that would enforce
it is the one not answering. When an operation runs out, its connection
is closed instead of pooled, since a query may still be in flight on it
or a transaction open: deadpool's Object::take on PostgreSQL;
Conn::disconnect on MySQL, which marks the connection closed before it
sends anything, so the pool discards it even when the server never
answers.

- query, 2 minutes: reads, writes (the whole transaction with its
  retries), blobs, SQL lookups, search queries and indexing. These take
  milliseconds; two minutes leaves room for a large blob over a slow
  link and still ends a hang.
- maintenance, 30 minutes: range deletes (account removal, purges),
  unindexing, purge_store, and creating tables and indexes at startup,
  which can legitimately run long in one statement. Their existing
  chunked fallback for server-side statement timeouts is unchanged.
- iterate (exports, reindexing, maintenance scans) can run for hours,
  so the query limit bounds each wait for the database (preparing, the
  query starting, the next row) rather than the whole scan.

The limits are fixed, like the pool timeouts; the DataStore schema has
no field for them. Tests set them with Store::with_query_timeouts
(test_mode only).

Readiness. /healthz/ready answered 200 whenever a data store was
configured. It now reads one key from the data store with a 2 s limit
and reuses the answer for 2 s, so probes can't load the database;
while one probe runs, others get the last answer. The first failed
probe of an outage is logged. /healthz/live stays 200: restarting a
node doesn't bring its database back, and an orchestrator restarting on
failed liveness would restart every node at once. The container
HEALTHCHECK already uses /healthz/live.

Tests, store::pool_timeout (a proxy that stops forwarding while
keeping connections open plays the paused database):
- postgres_query_timeout, mysql_query_timeout (new): with four pooled
  connections open, a read, a scan and a write each fail with "Query
  timed out" 2.0 s after the pause (2 s test limit); once the proxy
  forwards again the store answers. With the limits set to an hour
  (upstream's behavior), the read was still waiting at the test's 20 s
  limit.
- postgres_readiness (new, STORE=PostgreSql): a node's data store
  goes through the proxy; /healthz/ready is 200, 503 about 4 s after
  the pause while /healthz/live stays 200, and 200 again about 2 s
  after it ends.
- postgres_pool_timeout, mysql_pool_timeout: pass as before.
store::store_tests (PostgreSql, MySql, including the MariaDB statement
timeout step) and store::task_locks (PostgreSql) pass;
store::search_tests (PostgreSql) fails at the same ordering assertion
(query.rs:684) as on main.

jcoffey-dev merged commit ad58c35f39 into main

2026-09-25 00:45:10 +00:00

jcoffey-dev deleted branch fix/query-timeouts

2026-09-25 00:45:10 +00:00

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: inbuxa/inbuxa-server#45