SQL pools time out; task locks are a renewed five-minute lease

A 3-node rehearsal (PostgreSQL + NATS + Garage) found two ways a crash
leaves work stuck:

Pool hangs. The PostgreSQL pool (deadpool) was built with no timeouts,
so a request waited for a free connection, and for one to be opened or
recycled, for as long as it took: forever when the server stopped
answering. MySQL's pool (mysql_async) has no wait timeout at all.

- PostgreSQL: wait 30 s (or the store's timeout if longer), create the
  store's timeout or 15 s (it bounds the whole handshake, where
  tokio-postgres's connect_timeout covers only the TCP connect), recycle
  10 s. The pool config is now always set, not only with
  poolMaxConnections.
- MySQL: every connection is taken through MysqlStore::conn(), which
  gives up after 30 s.
- Both: TCP keepalive after 60 s idle, so a server that vanished
  without closing the connection is noticed in minutes rather than the
  two-hour system default.

The DataStore schema has no pool timeout settings, so these are fixed
defaults; the store's own timeout bounds connecting on PostgreSQL.

Task locks. A task lock lasted an hour, so after a hard crash the dead
node's tasks waited up to an hour and five minutes. The lock is now a
five-minute lease: while this node runs a task, the task manager renews
its lock every third of the lifetime (InMemoryStore::renew_lock, a
compare-and-set on the store backends and SET XX EX on Redis, which
leaves a lock that already expired alone). A killed node's tasks run
elsewhere within about five minutes plus the claim recheck. A task this
node holds isn't handed to a worker again by the scan.

store::pool_timeout (new): a local listener that accepts connections
and never answers plays a hung server; a PostgreSQL store with a 2 s
timeout returns an error in about 4 s, and a MySQL store in 30 s.
Without the timeouts both wait for good. store::task_locks gains a
task held for 1.5 lock lifetimes: its lease is still held, and released
when the task ends.

crates/services/src/task_manager/manager.rs

@@ -13,7 +13,7 @@ use crate::task_manager::dkim::DkimManagementTask;
 use crate::task_manager::dns::DnsManagementTask;
 use crate::task_manager::imip::SendImipTask;
 use crate::task_manager::index::SearchIndexTask;
-use crate::task_manager::lock::TaskLockManager;
+use crate::task_manager::lock::{TaskLockManager, renew_task_locks};
 use crate::task_manager::maintenance::MaintenanceTask;
 use crate::task_manager::merge_threads::MergeThreadsTask;
 use crate::task_manager::report::{self, SubmitReportTask};
@@ -75,6 +75,28 @@ pub fn spawn_task_manager(inner: Arc<Inner>) {
 
     trc::event!(TaskManager(TaskManagerEvent::ManagerStarted));
 
+    // inbuxa: keep the leases of running tasks alive, every third of a lock
+    // lifetime, until the node stops
+    {
+        let inner = inner.clone();
+        tokio::spawn(async move {
+            let mut renewed_at = Instant::now();
+            loop {
+                tokio::time::sleep(Duration::from_secs(1)).await;
+                let locks = &inner.ipc.task_locks;
+                if locks.is_stopping() {
+                    break;
+                }
+                if renewed_at.elapsed() >= Duration::from_secs((locks.expiry() / 3).max(1)) {
+                    renewed_at = Instant::now();
+                    if locks.held() > 0 {
+                        renew_task_locks(&inner.build_server()).await;
+                    }
+                }
+            }
+        });
+    }
+
     // Create dummy server instance for alarms
     let server_instance = Arc::new(ServerInstance {
         id: "_local".to_string(),
@@ -292,6 +314,12 @@ impl TaskQueueManager for Server {
                                     .caused_by(trc::location!())
                                     .ctx(trc::Key::Value, value)
                             })?;
+                            // inbuxa: running here under a lease this node
+                            // renews; don't hand it to a worker again
+                            if task_locks.is_held(task_id) {
+                                return Ok(true);
+                            }
+
                             let enabled = task_enabled(roles, task_type);
 
                             if !enabled {