SQL pools time out; task locks are a renewed five-minute lease

A 3-node rehearsal (PostgreSQL + NATS + Garage) found two ways a crash
leaves work stuck:

Pool hangs. The PostgreSQL pool (deadpool) was built with no timeouts,
so a request waited for a free connection, and for one to be opened or
recycled, for as long as it took: forever when the server stopped
answering. MySQL's pool (mysql_async) has no wait timeout at all.

- PostgreSQL: wait 30 s (or the store's timeout if longer), create the
  store's timeout or 15 s (it bounds the whole handshake, where
  tokio-postgres's connect_timeout covers only the TCP connect), recycle
  10 s. The pool config is now always set, not only with
  poolMaxConnections.
- MySQL: every connection is taken through MysqlStore::conn(), which
  gives up after 30 s.
- Both: TCP keepalive after 60 s idle, so a server that vanished
  without closing the connection is noticed in minutes rather than the
  two-hour system default.

The DataStore schema has no pool timeout settings, so these are fixed
defaults; the store's own timeout bounds connecting on PostgreSQL.

Task locks. A task lock lasted an hour, so after a hard crash the dead
node's tasks waited up to an hour and five minutes. The lock is now a
five-minute lease: while this node runs a task, the task manager renews
its lock every third of the lifetime (InMemoryStore::renew_lock, a
compare-and-set on the store backends and SET XX EX on Redis, which
leaves a lock that already expired alone). A killed node's tasks run
elsewhere within about five minutes plus the claim recheck. A task this
node holds isn't handed to a worker again by the scan.

store::pool_timeout (new): a local listener that accepts connections
and never answers plays a hung server; a PostgreSQL store with a 2 s
timeout returns an error in about 4 s, and a MySQL store in 30 s.
Without the timeouts both wait for good. store::task_locks gains a
task held for 1.5 lock lifetimes: its lease is still held, and released
when the task ends.

crates/store/src/backend/redis/lookup.rs

@@ -2,6 +2,8 @@
  * SPDX-FileCopyrightText: 2020 Stalwart Labs LLC <[email protected]>
  *
  * SPDX-License-Identifier: AGPL-3.0-only OR LicenseRef-SEL
+ *
+ * Modified by Coffey Labs in 2026 for INBUXA.
  */
 
 use super::{RedisPool, RedisStore, into_error};
@@ -79,6 +81,30 @@ impl RedisStore {
         }
     }
 
+    // inbuxa: see InMemoryStore::renew_lock
+    pub async fn renew_lock(&self, key: &[u8], expires: u64) -> trc::Result<bool> {
+        match &self.pool {
+            RedisPool::Single(pool) => {
+                with_conn(pool, async |conn| {
+                    Self::renew_lock_(conn, key, expires).await
+                })
+                .await
+            }
+            RedisPool::Cluster(pool) => {
+                with_conn(pool, async |conn| {
+                    Self::renew_lock_(conn, key, expires).await
+                })
+                .await
+            }
+            RedisPool::Sentinel(pool) => {
+                with_conn(pool, async |conn| {
+                    Self::renew_lock_(conn, key, expires).await
+                })
+                .await
+            }
+        }
+    }
+
     pub async fn key_delete(&self, key: &[u8]) -> trc::Result<()> {
         match &self.pool {
             RedisPool::Single(pool) => {
@@ -226,6 +252,22 @@ impl RedisStore {
             .map(|reply| reply.is_some())
     }
 
+    async fn renew_lock_(
+        conn: &mut impl AsyncCommands,
+        key: &[u8],
+        expires: u64,
+    ) -> RedisResult<bool> {
+        redis::cmd("SET")
+            .arg(key)
+            .arg(now() + expires)
+            .arg("XX")
+            .arg("EX")
+            .arg(expires as i64)
+            .query_async::<Option<String>>(conn)
+            .await
+            .map(|reply| reply.is_some())
+    }
+
     async fn key_delete_(conn: &mut impl AsyncCommands, key: &[u8]) -> RedisResult<()> {
         conn.del(key).await
     }