jcoffey-dev
7c80a12d75
A cluster rehearsal (PostgreSQL + NATS) left index tasks pending well past the one-hour task lock after the node that claimed them was stopped or killed. The exact cause there isn't confirmed; this closes every path found in the task manager that stretches a takeover past the lock, or keeps a task claimed without running it: - A graceful stop never released the locks it held, so every task the node had claimed stayed blocked for an hour. The server now tracks the locks it holds (common::ipc::TaskLocks) and, once the shutdown signal arrives, stops claiming and releases them before exiting. - A node that failed to claim a task (another node held it) set its own local hold for a full lock lifetime from that scan. If the holder claimed it just after the scan began, or ran on a clock ahead, that hold ran out a moment before the lock did and was set for another hour: two hours in all. Such claims are now tried again every five minutes (a twelfth of the lock lifetime), and the task manager wakes up for them: before, a node without a coordinator could sleep up to five minutes past the recheck, or until something else woke it. - A worker that panicked took its task type down on that node for good, while the scan kept claiming that type's tasks and failing to hand them over, re-taking each lock as it expired and so starving every other node of them. Each batch now runs on a task of its own; a panic is logged, the batch's locks are released and the worker carries on. A failed hand-over releases the lock too. - A claimed task the worker couldn't read, or found gone, kept its lock for the hour. It is released. - An IndexDocument task for a file (not indexed) returned no result, which shifted every later result in the batch onto the wrong task in update_tasks. It returns Ignored. Nothing queues such a task today. The lock lifetime stays one hour; it now lives per server so the tests can shorten it. store::task_locks::task_lock_tests plays a second node by writing its locks straight into the in-memory store: tasks it claimed and abandoned run here once its locks expire, including locks that outlive this node's view of them, and a graceful stop hands this node's locks back at once and claims nothing more. It passes on RocksDB, SQLite and PostgreSQL. With the old recheck it fails.
85 lines
2.5 KiB
Rust
85 lines
2.5 KiB
Rust
/*
|
|
* SPDX-FileCopyrightText: 2020 Stalwart Labs LLC <[email protected]>
|
|
*
|
|
* SPDX-License-Identifier: AGPL-3.0-only OR LicenseRef-SEL
|
|
*
|
|
* Modified by Coffey Labs in 2026 for INBUXA.
|
|
*/
|
|
|
|
use crate::task_manager::*;
|
|
|
|
pub trait TaskLockManager: Sync + Send {
|
|
fn try_lock_task(&self, task: u64) -> impl Future<Output = bool> + Send;
|
|
fn remove_index_lock(&self, id: u64) -> impl Future<Output = ()> + Send;
|
|
}
|
|
|
|
impl TaskLockManager for Server {
|
|
async fn try_lock_task(&self, id: u64) -> bool {
|
|
// inbuxa: a node that is stopping claims nothing new
|
|
let locks = &self.inner.ipc.task_locks;
|
|
if locks.is_stopping() {
|
|
return false;
|
|
}
|
|
|
|
match self
|
|
.in_memory_store()
|
|
.try_lock(KV_LOCK_TASK, &id.to_be_bytes(), locks.expiry())
|
|
.await
|
|
{
|
|
Ok(result) => {
|
|
if result {
|
|
locks.insert(id);
|
|
} else {
|
|
trc::event!(
|
|
TaskManager(TaskManagerEvent::TaskLocked),
|
|
Id = id,
|
|
Details = "Task details not available",
|
|
);
|
|
}
|
|
result
|
|
}
|
|
Err(err) => {
|
|
trc::error!(err.id(id).details("Failed to lock task"));
|
|
|
|
false
|
|
}
|
|
}
|
|
}
|
|
|
|
async fn remove_index_lock(&self, id: u64) {
|
|
if let Err(err) = self
|
|
.in_memory_store()
|
|
.remove_lock(KV_LOCK_TASK, &id.to_be_bytes())
|
|
.await
|
|
{
|
|
trc::error!(
|
|
err.details("Failed to unlock task")
|
|
.ctx(trc::Key::Id, id)
|
|
.caused_by(trc::location!())
|
|
);
|
|
}
|
|
self.inner.ipc.task_locks.remove(id);
|
|
}
|
|
}
|
|
|
|
/// inbuxa: on a graceful stop, stops claiming tasks and releases every task
|
|
/// lock this node holds, so the rest of the cluster can pick the tasks up at
|
|
/// once instead of after the lock expires. Returns how many were released.
|
|
pub async fn release_task_locks(server: &Server) -> usize {
|
|
let ids = server.inner.ipc.task_locks.stop();
|
|
for id in &ids {
|
|
if let Err(err) = server
|
|
.in_memory_store()
|
|
.remove_lock(KV_LOCK_TASK, &id.to_be_bytes())
|
|
.await
|
|
{
|
|
trc::error!(
|
|
err.details("Failed to release task lock on shutdown")
|
|
.ctx(trc::Key::Id, *id)
|
|
.caused_by(trc::location!())
|
|
);
|
|
}
|
|
}
|
|
ids.len()
|
|
}
|