Files
inbuxa-server/crates/main/src/main.rs
T
jcoffey-dev 7c80a12d75
ci / fork-checks (pull_request) Successful in 18s
ci / build (pull_request) Successful in 17m2s
Task manager: release task locks on stop, recheck claims held elsewhere
A cluster rehearsal (PostgreSQL + NATS) left index tasks pending well
past the one-hour task lock after the node that claimed them was stopped
or killed. The exact cause there isn't confirmed; this closes every path
found in the task manager that stretches a takeover past the lock, or
keeps a task claimed without running it:

- A graceful stop never released the locks it held, so every task the
  node had claimed stayed blocked for an hour. The server now tracks the
  locks it holds (common::ipc::TaskLocks) and, once the shutdown signal
  arrives, stops claiming and releases them before exiting.
- A node that failed to claim a task (another node held it) set its own
  local hold for a full lock lifetime from that scan. If the holder
  claimed it just after the scan began, or ran on a clock ahead, that
  hold ran out a moment before the lock did and was set for another
  hour: two hours in all. Such claims are now tried again every five
  minutes (a twelfth of the lock lifetime), and the task manager wakes
  up for them: before, a node without a coordinator could sleep up to
  five minutes past the recheck, or until something else woke it.
- A worker that panicked took its task type down on that node for good,
  while the scan kept claiming that type's tasks and failing to hand them
  over, re-taking each lock as it expired and so starving every other
  node of them. Each batch now runs on a task of its own; a panic is
  logged, the batch's locks are released and the worker carries on. A
  failed hand-over releases the lock too.
- A claimed task the worker couldn't read, or found gone, kept its lock
  for the hour. It is released.
- An IndexDocument task for a file (not indexed) returned no result,
  which shifted every later result in the batch onto the wrong task in
  update_tasks. It returns Ignored. Nothing queues such a task today.

The lock lifetime stays one hour; it now lives per server so the tests
can shorten it.

store::task_locks::task_lock_tests plays a second node by writing its
locks straight into the in-memory store: tasks it claimed and abandoned
run here once its locks expire, including locks that outlive this node's
view of them, and a graceful stop hands this node's locks back at once
and claims nothing more. It passes on RocksDB, SQLite and PostgreSQL.
With the old recheck it fails.
2026-09-24 08:19:05 -07:00

173 lines
5.5 KiB
Rust

/*
* SPDX-FileCopyrightText: 2020 Stalwart Labs LLC <[email protected]>
*
* SPDX-License-Identifier: AGPL-3.0-only OR LicenseRef-SEL
*
* Modified by Coffey Labs in 2026 for INBUXA.
*/
#![warn(clippy::large_futures)]
#![warn(clippy::cast_possible_truncation)]
#![warn(clippy::cast_possible_wrap)]
#![warn(clippy::cast_sign_loss)]
use common::{
BuildServer, Inner,
config::server::{Listener, ServerProtocol},
manager::boot::BootManager,
network::TcpAcceptor,
};
use http::HttpSessionManager;
use imap::core::ImapSessionManager;
use managesieve::core::ManageSieveSessionManager;
use pop3::Pop3SessionManager;
use services::{StartServices, broadcast::subscriber::spawn_broadcast_subscriber};
use smtp::{StartQueueManager, core::SmtpSessionManager};
use std::sync::Arc;
use std::time::Duration;
use tokio::sync::watch;
use trc::Collector;
use utils::wait_for_shutdown;
#[cfg(feature = "dev_mode")]
pub mod test_data;
#[cfg(not(any(target_env = "msvc", target_os = "freebsd")))]
use tikv_jemallocator::Jemalloc;
#[cfg(not(any(target_env = "msvc", target_os = "freebsd")))]
#[global_allocator]
static GLOBAL: Jemalloc = Jemalloc;
#[tokio::main]
async fn main() -> std::io::Result<()> {
// Install AWS-LC-RS as the default Rustls crypto provider
rustls::crypto::aws_lc_rs::default_provider()
.install_default()
.expect("failed to install aws-lc-rs as the default rustls crypto provider");
// Build the shared outbound TLS configurations
utils::http::init_shared_tls_configs();
// Load config and apply macros
let mut init = Box::pin(BootManager::init()).await;
// Migrate database
if let Err(err) = migration::try_migrate(&init.inner.build_server()).await {
trc::event!(
Server(trc::ServerEvent::StartupError),
Details = "Failed to migrate database, aborting startup.",
Reason = err,
);
return Ok(());
}
// Init services
init.start_services().await;
init.start_queue_manager();
// Log configuration errors
init.bootstrap.log_errors();
init.bootstrap.log_warnings();
#[cfg(feature = "dev_mode")]
if std::env::var("INSERT_TEST_DATA").is_ok() {
let server = init.inner.build_server();
test_data::insert_test_data(&server).await;
server.insert_test_metrics().await;
}
// Spawn servers
// Each listener gets its own shutdown channel, registered under its id, so
// the legacy-protocols switch can close one protocol's ports and leave the
// rest accepting (legacy-protocols LP-2). The registry lives in `Data` and
// so outlives the listeners, which it must: it owns the sending ends.
let listener_control = &init.inner.data.listener_control;
let spawn_inner = init.inner.clone();
let (shutdown_tx, shutdown_rx) =
init.servers
.spawn_with_control(listener_control, |server, acceptor, shutdown_rx| {
spawn_listener(&spawn_inner, server, acceptor, shutdown_rx);
});
// Leave behind how to spawn a listener, so putting one back opens its port
// without a restart (LP-5). Only this file knows the session manager for a
// protocol, so only this file can say.
let spawn_inner = init.inner.clone();
init.inner
.data
.listener_control
.set_spawner(Box::new(move |server, acceptor, shutdown_rx| {
spawn_listener(&spawn_inner, server, acceptor, shutdown_rx);
}));
// Start broadcast subscriber
let inner = init.inner.clone();
spawn_broadcast_subscriber(init.inner, shutdown_rx);
// Wait for shutdown signal
wait_for_shutdown().await;
// inbuxa: hand back the task locks this node holds, so other nodes can
// run those tasks now rather than when the locks expire
services::task_manager::lock::release_task_locks(&inner.build_server()).await;
// Shutdown collector
Collector::shutdown();
// Stop services, then the listeners: the shutdown sender no longer reaches
// them, since each holds its own channel (LP-2).
let _ = shutdown_tx.send(true);
inner.data.listener_control.stop_all();
// Wait for services to finish
tokio::time::sleep(Duration::from_secs(1)).await;
Ok(())
}
/// Starts one listener under the session manager its protocol calls for.
///
/// Used twice: once for every listener at startup, and again whenever the
/// legacy-protocols switch puts a listener back (LP-5).
fn spawn_listener(
inner: &Arc<Inner>,
server: Listener,
acceptor: TcpAcceptor,
shutdown_rx: watch::Receiver<bool>,
) {
match &server.protocol {
ServerProtocol::Smtp | ServerProtocol::Lmtp => server.spawn(
SmtpSessionManager::new(inner.clone()),
inner.clone(),
acceptor,
shutdown_rx,
),
ServerProtocol::Http => server.spawn(
HttpSessionManager::new(inner.clone()),
inner.clone(),
acceptor,
shutdown_rx,
),
ServerProtocol::Imap => server.spawn(
ImapSessionManager::new(inner.clone()),
inner.clone(),
acceptor,
shutdown_rx,
),
ServerProtocol::Pop3 => server.spawn(
Pop3SessionManager::new(inner.clone()),
inner.clone(),
acceptor,
shutdown_rx,
),
ServerProtocol::ManageSieve => server.spawn(
ManageSieveSessionManager::new(inner.clone()),
inner.clone(),
acceptor,
shutdown_rx,
),
}
}