SQL pools time out; task locks are a renewed five-minute lease

A 3-node rehearsal (PostgreSQL + NATS + Garage) found two ways a crash
leaves work stuck:

Pool hangs. The PostgreSQL pool (deadpool) was built with no timeouts,
so a request waited for a free connection, and for one to be opened or
recycled, for as long as it took: forever when the server stopped
answering. MySQL's pool (mysql_async) has no wait timeout at all.

- PostgreSQL: wait 30 s (or the store's timeout if longer), create the
  store's timeout or 15 s (it bounds the whole handshake, where
  tokio-postgres's connect_timeout covers only the TCP connect), recycle
  10 s. The pool config is now always set, not only with
  poolMaxConnections.
- MySQL: every connection is taken through MysqlStore::conn(), which
  gives up after 30 s.
- Both: TCP keepalive after 60 s idle, so a server that vanished
  without closing the connection is noticed in minutes rather than the
  two-hour system default.

The DataStore schema has no pool timeout settings, so these are fixed
defaults; the store's own timeout bounds connecting on PostgreSQL.

Task locks. A task lock lasted an hour, so after a hard crash the dead
node's tasks waited up to an hour and five minutes. The lock is now a
five-minute lease: while this node runs a task, the task manager renews
its lock every third of the lifetime (InMemoryStore::renew_lock, a
compare-and-set on the store backends and SET XX EX on Redis, which
leaves a lock that already expired alone). A killed node's tasks run
elsewhere within about five minutes plus the claim recheck. A task this
node holds isn't handed to a worker again by the scan.

store::pool_timeout (new): a local listener that accepts connections
and never answers plays a hung server; a PostgreSQL store with a 2 s
timeout returns an error in about 4 s, and a MySQL store in 30 s.
Without the timeouts both wait for good. store::task_locks gains a
task held for 1.5 lock lifetimes: its lease is still held, and released
when the task ends.

crates/services/src/task_manager/lock.rs

@@ -82,3 +82,42 @@ pub async fn release_task_locks(server: &Server) -> usize {
     }
     ids.len()
 }
+
+/// inbuxa: renews the lease on every task this node is running, so it stays
+/// claimed for as long as it runs while a node that dies loses its claims
+/// within one lock lifetime. Returns how many leases were renewed and how
+/// many were found lost (expired, perhaps taken by another node).
+pub async fn renew_task_locks(server: &Server) -> (usize, usize) {
+    let locks = &server.inner.ipc.task_locks;
+    let expiry = locks.expiry();
+    let (mut renewed, mut lost) = (0, 0);
+    for id in locks.held_ids() {
+        match server
+            .in_memory_store()
+            .renew_lock(KV_LOCK_TASK, &id.to_be_bytes(), expiry)
+            .await
+        {
+            Ok(true) => renewed += 1,
+            Ok(false) => {
+                // Still held here as far as this node knows; the task
+                // finishes and its lock is removed as usual
+                if locks.is_held(id) {
+                    lost += 1;
+                    trc::event!(
+                        TaskManager(TaskManagerEvent::TaskLocked),
+                        Id = id,
+                        Details = "Task lock expired while the task was running",
+                    );
+                }
+            }
+            Err(err) => {
+                trc::error!(
+                    err.details("Failed to renew task lock")
+                        .ctx(trc::Key::Id, id)
+                        .caused_by(trc::location!())
+                );
+            }
+        }
+    }
+    (renewed, lost)
+}