jcoffey-dev
6e50ba25a9
A 3-node rehearsal (PostgreSQL + NATS + Garage) found two ways a crash leaves work stuck: Pool hangs. The PostgreSQL pool (deadpool) was built with no timeouts, so a request waited for a free connection, and for one to be opened or recycled, for as long as it took: forever when the server stopped answering. MySQL's pool (mysql_async) has no wait timeout at all. - PostgreSQL: wait 30 s (or the store's timeout if longer), create the store's timeout or 15 s (it bounds the whole handshake, where tokio-postgres's connect_timeout covers only the TCP connect), recycle 10 s. The pool config is now always set, not only with poolMaxConnections. - MySQL: every connection is taken through MysqlStore::conn(), which gives up after 30 s. - Both: TCP keepalive after 60 s idle, so a server that vanished without closing the connection is noticed in minutes rather than the two-hour system default. The DataStore schema has no pool timeout settings, so these are fixed defaults; the store's own timeout bounds connecting on PostgreSQL. Task locks. A task lock lasted an hour, so after a hard crash the dead node's tasks waited up to an hour and five minutes. The lock is now a five-minute lease: while this node runs a task, the task manager renews its lock every third of the lifetime (InMemoryStore::renew_lock, a compare-and-set on the store backends and SET XX EX on Redis, which leaves a lock that already expired alone). A killed node's tasks run elsewhere within about five minutes plus the claim recheck. A task this node holds isn't handed to a worker again by the scan. store::pool_timeout (new): a local listener that accepts connections and never answers plays a hung server; a PostgreSQL store with a 2 s timeout returns an error in about 4 s, and a MySQL store in 30 s. Without the timeouts both wait for good. store::task_locks gains a task held for 1.5 lock lifetimes: its lease is still held, and released when the task ends.
124 lines
4.0 KiB
Rust
124 lines
4.0 KiB
Rust
/*
|
|
* SPDX-FileCopyrightText: 2020 Stalwart Labs LLC <[email protected]>
|
|
*
|
|
* SPDX-License-Identifier: AGPL-3.0-only OR LicenseRef-SEL
|
|
*
|
|
* Modified by Coffey Labs in 2026 for INBUXA.
|
|
*/
|
|
|
|
use crate::task_manager::*;
|
|
|
|
pub trait TaskLockManager: Sync + Send {
|
|
fn try_lock_task(&self, task: u64) -> impl Future<Output = bool> + Send;
|
|
fn remove_index_lock(&self, id: u64) -> impl Future<Output = ()> + Send;
|
|
}
|
|
|
|
impl TaskLockManager for Server {
|
|
async fn try_lock_task(&self, id: u64) -> bool {
|
|
// inbuxa: a node that is stopping claims nothing new
|
|
let locks = &self.inner.ipc.task_locks;
|
|
if locks.is_stopping() {
|
|
return false;
|
|
}
|
|
|
|
match self
|
|
.in_memory_store()
|
|
.try_lock(KV_LOCK_TASK, &id.to_be_bytes(), locks.expiry())
|
|
.await
|
|
{
|
|
Ok(result) => {
|
|
if result {
|
|
locks.insert(id);
|
|
} else {
|
|
trc::event!(
|
|
TaskManager(TaskManagerEvent::TaskLocked),
|
|
Id = id,
|
|
Details = "Task details not available",
|
|
);
|
|
}
|
|
result
|
|
}
|
|
Err(err) => {
|
|
trc::error!(err.id(id).details("Failed to lock task"));
|
|
|
|
false
|
|
}
|
|
}
|
|
}
|
|
|
|
async fn remove_index_lock(&self, id: u64) {
|
|
if let Err(err) = self
|
|
.in_memory_store()
|
|
.remove_lock(KV_LOCK_TASK, &id.to_be_bytes())
|
|
.await
|
|
{
|
|
trc::error!(
|
|
err.details("Failed to unlock task")
|
|
.ctx(trc::Key::Id, id)
|
|
.caused_by(trc::location!())
|
|
);
|
|
}
|
|
self.inner.ipc.task_locks.remove(id);
|
|
}
|
|
}
|
|
|
|
/// inbuxa: on a graceful stop, stops claiming tasks and releases every task
|
|
/// lock this node holds, so the rest of the cluster can pick the tasks up at
|
|
/// once instead of after the lock expires. Returns how many were released.
|
|
pub async fn release_task_locks(server: &Server) -> usize {
|
|
let ids = server.inner.ipc.task_locks.stop();
|
|
for id in &ids {
|
|
if let Err(err) = server
|
|
.in_memory_store()
|
|
.remove_lock(KV_LOCK_TASK, &id.to_be_bytes())
|
|
.await
|
|
{
|
|
trc::error!(
|
|
err.details("Failed to release task lock on shutdown")
|
|
.ctx(trc::Key::Id, *id)
|
|
.caused_by(trc::location!())
|
|
);
|
|
}
|
|
}
|
|
ids.len()
|
|
}
|
|
|
|
/// inbuxa: renews the lease on every task this node is running, so it stays
|
|
/// claimed for as long as it runs while a node that dies loses its claims
|
|
/// within one lock lifetime. Returns how many leases were renewed and how
|
|
/// many were found lost (expired, perhaps taken by another node).
|
|
pub async fn renew_task_locks(server: &Server) -> (usize, usize) {
|
|
let locks = &server.inner.ipc.task_locks;
|
|
let expiry = locks.expiry();
|
|
let (mut renewed, mut lost) = (0, 0);
|
|
for id in locks.held_ids() {
|
|
match server
|
|
.in_memory_store()
|
|
.renew_lock(KV_LOCK_TASK, &id.to_be_bytes(), expiry)
|
|
.await
|
|
{
|
|
Ok(true) => renewed += 1,
|
|
Ok(false) => {
|
|
// Still held here as far as this node knows; the task
|
|
// finishes and its lock is removed as usual
|
|
if locks.is_held(id) {
|
|
lost += 1;
|
|
trc::event!(
|
|
TaskManager(TaskManagerEvent::TaskLocked),
|
|
Id = id,
|
|
Details = "Task lock expired while the task was running",
|
|
);
|
|
}
|
|
}
|
|
Err(err) => {
|
|
trc::error!(
|
|
err.details("Failed to renew task lock")
|
|
.ctx(trc::Key::Id, id)
|
|
.caused_by(trc::location!())
|
|
);
|
|
}
|
|
}
|
|
}
|
|
(renewed, lost)
|
|
}
|