jcoffey-dev
7c80a12d75
A cluster rehearsal (PostgreSQL + NATS) left index tasks pending well past the one-hour task lock after the node that claimed them was stopped or killed. The exact cause there isn't confirmed; this closes every path found in the task manager that stretches a takeover past the lock, or keeps a task claimed without running it: - A graceful stop never released the locks it held, so every task the node had claimed stayed blocked for an hour. The server now tracks the locks it holds (common::ipc::TaskLocks) and, once the shutdown signal arrives, stops claiming and releases them before exiting. - A node that failed to claim a task (another node held it) set its own local hold for a full lock lifetime from that scan. If the holder claimed it just after the scan began, or ran on a clock ahead, that hold ran out a moment before the lock did and was set for another hour: two hours in all. Such claims are now tried again every five minutes (a twelfth of the lock lifetime), and the task manager wakes up for them: before, a node without a coordinator could sleep up to five minutes past the recheck, or until something else woke it. - A worker that panicked took its task type down on that node for good, while the scan kept claiming that type's tasks and failing to hand them over, re-taking each lock as it expired and so starving every other node of them. Each batch now runs on a task of its own; a panic is logged, the batch's locks are released and the worker carries on. A failed hand-over releases the lock too. - A claimed task the worker couldn't read, or found gone, kept its lock for the hour. It is released. - An IndexDocument task for a file (not indexed) returned no result, which shifted every later result in the batch onto the wrong task in update_tasks. It returns Ignored. Nothing queues such a task today. The lock lifetime stays one hour; it now lives per server so the tests can shorten it. store::task_locks::task_lock_tests plays a second node by writing its locks straight into the in-memory store: tasks it claimed and abandoned run here once its locks expire, including locks that outlive this node's view of them, and a graceful stop hands this node's locks back at once and claims nothing more. It passes on RocksDB, SQLite and PostgreSQL. With the old recheck it fails.
159 lines
4.2 KiB
Rust
159 lines
4.2 KiB
Rust
/*
|
|
* SPDX-FileCopyrightText: 2020 Stalwart Labs LLC <[email protected]>
|
|
*
|
|
* SPDX-License-Identifier: AGPL-3.0-only OR LicenseRef-SEL
|
|
*
|
|
* Modified by Coffey Labs in 2026 for INBUXA.
|
|
*/
|
|
|
|
use common::{KV_LOCK_TASK, Server};
|
|
use registry::schema::enums::TaskType;
|
|
use registry::schema::structs::Task;
|
|
use registry::types::EnumImpl;
|
|
use std::future::Future;
|
|
use std::time::Instant;
|
|
use store::ahash::AHashMap;
|
|
use store::write::Operation;
|
|
use tokio::sync::mpsc;
|
|
use trc::TaskManagerEvent;
|
|
|
|
pub mod acme;
|
|
pub mod alarm;
|
|
pub mod destroy_account;
|
|
pub mod dkim;
|
|
pub mod dns;
|
|
pub mod imip;
|
|
pub mod inbuxa_restore; // inbuxa: undelete
|
|
pub mod index;
|
|
pub mod lock;
|
|
pub mod maintenance;
|
|
pub mod manager;
|
|
pub mod merge_threads;
|
|
pub mod report;
|
|
pub mod restore_item;
|
|
pub mod scheduler;
|
|
pub mod spam_classifier;
|
|
|
|
const QUEUE_REFRESH_INTERVAL: u64 = 60 * 5; // 5 minutes
|
|
// inbuxa: the lock lifetime (one hour) lives in common::ipc::TaskLocks, per
|
|
// server, so a graceful stop can release the locks and the tests can shorten it
|
|
const CLAIM_RECHECK_INTERVAL: u64 = 60 * 5; // 5 minutes
|
|
|
|
pub(crate) struct TaskManagerIpc {
|
|
txs: [mpsc::Sender<TaskJob>; TaskType::COUNT],
|
|
locked: AHashMap<u64, Locked>,
|
|
revision: u64,
|
|
}
|
|
|
|
#[derive(Debug)]
|
|
pub(crate) struct Locked {
|
|
expires: Instant,
|
|
due: u64,
|
|
revision: u64,
|
|
}
|
|
|
|
#[derive(Debug)]
|
|
pub(crate) struct TaskDetails {
|
|
task: Task,
|
|
info: TaskJob,
|
|
}
|
|
|
|
#[derive(Debug)]
|
|
pub(crate) struct TaskJob {
|
|
id: u64,
|
|
due: u64,
|
|
typ: TaskType,
|
|
}
|
|
|
|
#[derive(Debug, PartialEq, Eq)]
|
|
pub(crate) enum TaskResult {
|
|
Success(Vec<Task>),
|
|
Update([Operation; 2]),
|
|
Failure {
|
|
typ: TaskFailureType,
|
|
message: String,
|
|
max_attempts: Option<u64>,
|
|
},
|
|
Ignored,
|
|
}
|
|
|
|
#[derive(Debug, Clone, PartialEq, Eq)]
|
|
#[allow(dead_code)]
|
|
pub(crate) enum TaskFailureType {
|
|
Retry(u64),
|
|
Temporary,
|
|
Perpetual,
|
|
Permanent,
|
|
}
|
|
|
|
pub(crate) trait TaskInfo {
|
|
fn name(&self) -> &'static str;
|
|
}
|
|
|
|
impl TaskInfo for Task {
|
|
fn name(&self) -> &'static str {
|
|
match self {
|
|
Task::IndexDocument(_) => "IndexDocument",
|
|
Task::UnindexDocument(_) => "UnindexDocument",
|
|
Task::IndexTrace(_) => "IndexTrace",
|
|
Task::CalendarAlarmEmail(_) => "CalendarAlarmEmail",
|
|
Task::CalendarAlarmNotification(_) => "CalendarAlarmNotification",
|
|
Task::CalendarItipMessage(_) => "CalendarItipMessage",
|
|
Task::MergeThreads(_) => "MergeThreads",
|
|
Task::DmarcReport(_) => "DmarcReport",
|
|
Task::TlsReport(_) => "TlsReport",
|
|
Task::RestoreArchivedItem(_) => "RestoreArchivedItem",
|
|
Task::DestroyAccount(_) => "DestroyAccount",
|
|
Task::AccountMaintenance(_) => "AccountMaintenance",
|
|
Task::StoreMaintenance(_) => "StoreMaintenance",
|
|
Task::SpamFilterMaintenance(_) => "SpamFilterMaintenance",
|
|
Task::AcmeRenewal(_) => "AcmeRenewal",
|
|
Task::DkimManagement(_) => "DkimManagement",
|
|
Task::DnsManagement(_) => "DnsManagement",
|
|
Task::TenantMaintenance(_) => "TenantMaintenance",
|
|
}
|
|
}
|
|
}
|
|
|
|
impl TaskResult {
|
|
pub fn permanent(message: impl Into<String>) -> Self {
|
|
TaskResult::Failure {
|
|
typ: TaskFailureType::Permanent,
|
|
message: message.into(),
|
|
max_attempts: None,
|
|
}
|
|
}
|
|
|
|
pub fn temporary(message: impl Into<String>) -> Self {
|
|
TaskResult::Failure {
|
|
typ: TaskFailureType::Temporary,
|
|
message: message.into(),
|
|
max_attempts: None,
|
|
}
|
|
}
|
|
|
|
pub fn perpetual(message: impl Into<String>) -> Self {
|
|
TaskResult::Failure {
|
|
typ: TaskFailureType::Perpetual,
|
|
message: message.into(),
|
|
max_attempts: None,
|
|
}
|
|
}
|
|
|
|
pub fn deferred(retry_at: Option<u64>, message: impl Into<String>) -> Self {
|
|
match retry_at {
|
|
Some(retry_at) => TaskResult::Failure {
|
|
typ: TaskFailureType::Retry(retry_at),
|
|
message: message.into(),
|
|
max_attempts: None,
|
|
},
|
|
None => TaskResult::temporary(message),
|
|
}
|
|
}
|
|
}
|
|
|
|
pub(crate) fn deferred_retry_time(err: &trc::Error) -> Option<u64> {
|
|
err.value(trc::Key::NextRetry)
|
|
.and_then(|value| value.to_uint())
|
|
}
|