SQL pools time out; task locks are a renewed five-minute lease #41

Merged

jcoffey-dev merged 1 commits from fix/pool-timeouts into main 2026-09-24 20:18:57 +00:00

Conversation 0 Commits 1 Files Changed 16

+395 -24

jcoffey-dev commented 2026-09-24 20:13:48 +00:00

Owner

Copy Link

Copy Source

(a) Pool timeouts

PostgreSQL (crates/store/src/backend/postgres/main.rs:49-50): the deadpool pool was built with PoolConfig::new and no timeouts. Without poolMaxConnections it had no pool config at all. A request waited as long as it took for a free slot, a new connection or a recycled one. Against a server that stopped answering, that is forever. Now:

Timeout	Value	Notes
create	the store's timeout, else 15 s	Covers the whole handshake. tokio-postgres's connect_timeout only covers the TCP connect, so a server that accepts and then says nothing hung here.
wait	30 s, or create if that is longer
recycle	10 s

The pool config is now always set. Read replicas inherit it.

MySQL: mysql_async's pool has no wait timeout at all. Every connection now goes through MysqlStore::conn(), which gives up after 30 s (16 call sites).

Both: TCP keepalive starts after 60 s idle. A server that vanished without closing the connection is noticed in minutes, not after the two-hour system default. That matters for a query already sent on a live connection, which no pool timeout covers.

The DataStore schema has no pool-timeout settings. These are fixed defaults, apart from the store's existing timeout. Making them configurable needs a schema change; say if you want that.

(b) Task locks as a renewed lease

A task lock lasted 1 h (TaskLocks::DEFAULT_EXPIRY). After a hard crash, the dead node's tasks waited up to an hour plus the 5-minute recheck. The change is contained, so I made it:

• Lease length. The lock is now a 5-minute lease.
• Renewal. While a node runs a task, a heartbeat in the task manager renews its lease every third of the lifetime. It ticks every second, so a changed lifetime takes effect at once.
• InMemoryStore::renew_lock. On the store backends it's a compare-and-set that only extends a lock that is still live. On Redis it's SET … XX EX. Sharded stores route it to the lock's member. A lease that has already expired is left alone (another node may have it), and the event is logged.
• Scan. The claim scan no longer hands a task this node holds to a worker again.
• Crash recovery. A killed node's tasks run elsewhere within about 5 minutes plus the claim recheck (25 s), instead of an hour.
• Graceful stop. Unchanged from #35: locks are handed back at once.

Trade-off: a task only keeps its claim while its node keeps renewing. If a node's runtime stalls for more than 5 minutes, it can lose a claim while the task is still running. Before, a task that ran longer than an hour lost its claim the same way.

Tests

• store::pool_timeout (new; --features postgres,mysql, no database needed). A local listener that accepts and never answers plays a hung server:
  • PostgreSQL store with a 2 s timeout: error ("Timeout occurred while creating a new object") after 4 s.
  • MySQL store: error after 30 s.
  • On main, PostgreSQL was still waiting at the test's 20 s limit, and MySQL at its 60 s limit.
• store::task_locks::task_lock_tests, new step 3: a task held for 1.5 lock lifetimes keeps its lease, and its lock is free the moment the task ends.
  • RocksDb and PostgreSql: pass.
  • On main: fails ("lease lapsed while the task ran").
• store::store_tests, which exercises every backend path through the new connection helpers:
  • PostgreSql: pass.
  • MySql: registry, import/export and ops all passed. The run then stopped in the statement-timeout step, which needs a MariaDB container on port 3308 that isn't available here.
• system::system_tests (RocksDb): pass.
• After rebasing onto #40: task_lock_tests, postgres_pool_timeout and cluster::task_roles::task_role_tests all pass on PostgreSql.
• The Redis renew_lock path compiles (--features redis), but I didn't run it: there's no Redis here.

## (a) Pool timeouts **PostgreSQL** (`crates/store/src/backend/postgres/main.rs:49-50`): the deadpool pool was built with `PoolConfig::new` and no timeouts. Without `poolMaxConnections` it had no pool config at all. A request waited as long as it took for a free slot, a new connection or a recycled one. Against a server that stopped answering, that is forever. Now: | Timeout | Value | Notes | |---|---|---| | create | the store's `timeout`, else 15 s | Covers the whole handshake. tokio-postgres's `connect_timeout` only covers the TCP connect, so a server that accepts and then says nothing hung here. | | wait | 30 s, or `create` if that is longer | | | recycle | 10 s | | The pool config is now always set. Read replicas inherit it. **MySQL**: mysql_async's pool has no wait timeout at all. Every connection now goes through `MysqlStore::conn()`, which gives up after 30 s (16 call sites). **Both**: TCP keepalive starts after 60 s idle. A server that vanished without closing the connection is noticed in minutes, not after the two-hour system default. That matters for a query already sent on a live connection, which no pool timeout covers. The DataStore schema has no pool-timeout settings. These are fixed defaults, apart from the store's existing `timeout`. Making them configurable needs a schema change; say if you want that. ## (b) Task locks as a renewed lease A task lock lasted 1 h (`TaskLocks::DEFAULT_EXPIRY`). After a hard crash, the dead node's tasks waited up to an hour plus the 5-minute recheck. The change is contained, so I made it: - **Lease length.** The lock is now a **5-minute lease**. - **Renewal.** While a node runs a task, a heartbeat in the task manager renews its lease every third of the lifetime. It ticks every second, so a changed lifetime takes effect at once. - **`InMemoryStore::renew_lock`.** On the store backends it's a compare-and-set that only extends a lock that is still live. On Redis it's `SET … XX EX`. Sharded stores route it to the lock's member. A lease that has already expired is left alone (another node may have it), and the event is logged. - **Scan.** The claim scan no longer hands a task this node holds to a worker again. - **Crash recovery.** A killed node's tasks run elsewhere within about 5 minutes plus the claim recheck (25 s), instead of an hour. - **Graceful stop.** Unchanged from #35: locks are handed back at once. Trade-off: a task only keeps its claim while its node keeps renewing. If a node's runtime stalls for more than 5 minutes, it can lose a claim while the task is still running. Before, a task that ran longer than an hour lost its claim the same way. ## Tests - **`store::pool_timeout`** (new; `--features postgres,mysql`, no database needed). A local listener that accepts and never answers plays a hung server: - PostgreSQL store with a 2 s timeout: error ("Timeout occurred while creating a new object") after 4 s. - MySQL store: error after 30 s. - On `main`, PostgreSQL was still waiting at the test's 20 s limit, and MySQL at its 60 s limit. - **`store::task_locks::task_lock_tests`**, new step 3: a task held for 1.5 lock lifetimes keeps its lease, and its lock is free the moment the task ends. - RocksDb and PostgreSql: pass. - On `main`: fails ("lease lapsed while the task ran"). - **`store::store_tests`**, which exercises every backend path through the new connection helpers: - PostgreSql: pass. - MySql: registry, import/export and ops all passed. The run then stopped in the statement-timeout step, which needs a MariaDB container on port 3308 that isn't available here. - **`system::system_tests`** (RocksDb): pass. - **After rebasing onto #40:** `task_lock_tests`, `postgres_pool_timeout` and `cluster::task_roles::task_role_tests` all pass on PostgreSql. - The Redis `renew_lock` path compiles (`--features redis`), but I didn't run it: there's no Redis here.

jcoffey-dev added 1 commit 2026-09-24 20:13:48 +00:00

SQL pools time out; task locks are a renewed five-minute lease ... 6e50ba25a9

A 3-node rehearsal (PostgreSQL + NATS + Garage) found two ways a crash
leaves work stuck:

Pool hangs. The PostgreSQL pool (deadpool) was built with no timeouts,
so a request waited for a free connection, and for one to be opened or
recycled, for as long as it took: forever when the server stopped
answering. MySQL's pool (mysql_async) has no wait timeout at all.

- PostgreSQL: wait 30 s (or the store's timeout if longer), create the
  store's timeout or 15 s (it bounds the whole handshake, where
  tokio-postgres's connect_timeout covers only the TCP connect), recycle
  10 s. The pool config is now always set, not only with
  poolMaxConnections.
- MySQL: every connection is taken through MysqlStore::conn(), which
  gives up after 30 s.
- Both: TCP keepalive after 60 s idle, so a server that vanished
  without closing the connection is noticed in minutes rather than the
  two-hour system default.

The DataStore schema has no pool timeout settings, so these are fixed
defaults; the store's own timeout bounds connecting on PostgreSQL.

Task locks. A task lock lasted an hour, so after a hard crash the dead
node's tasks waited up to an hour and five minutes. The lock is now a
five-minute lease: while this node runs a task, the task manager renews
its lock every third of the lifetime (InMemoryStore::renew_lock, a
compare-and-set on the store backends and SET XX EX on Redis, which
leaves a lock that already expired alone). A killed node's tasks run
elsewhere within about five minutes plus the claim recheck. A task this
node holds isn't handed to a worker again by the scan.

store::pool_timeout (new): a local listener that accepts connections
and never answers plays a hung server; a PostgreSQL store with a 2 s
timeout returns an error in about 4 s, and a MySQL store in 30 s.
Without the timeouts both wait for good. store::task_locks gains a
task held for 1.5 lock lifetimes: its lease is still held, and released
when the task ends.

jcoffey-dev merged commit 89860aa5cc into main 2026-09-24 20:18:57 +00:00

jcoffey-dev deleted branch fix/pool-timeouts 2026-09-24 20:18:57 +00:00

Sign in to join this conversation.

Reviewers

No Reviewers

Labels

No items

No labels

Milestone

No items

No Milestone

Projects

Clear projects

No projects

Assignees

Clear assignees

jcoffey-dev (John Coffey)

No Assignees

1 Participants

Notifications

Subscribe

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: inbuxa/inbuxa-server#41