Cluster role changes apply to delivery and tasks without a restart

In cluster rehearsal 3, turning outboundMta off on node1's role was
reported applied (x:settingsReload applied: true), yet node1 kept
delivering mail, a report message included, until it was restarted.
The queue and report managers were started at boot only when the
node's role included outboundMta (crates/smtp/src/lib.rs), and the task
manager only when the role had some task type (spawn_task_manager).
After that nothing looked at the role again: a queue manager that was
running kept claiming and delivering, and one that wasn't never
started.

They now start on every node (outside recovery mode) and follow the
role live:

- Queue manager: before each scan it reads the role from the running
  settings. Without outboundMta it claims nothing new; deliveries
  already running finish and report back as usual, which releases
  their locks. When the role comes back (a reload wakes the manager
  with ReloadSettings, and it looks again every 30 s regardless) it
  logs queue.started and scans the whole queue at once.
- Report scheduler: DMARC and TLS report events are handled only while
  the role has outboundMta, as at boot; events arriving without it are
  dropped, as they were on a node started without the role.
- Task manager: task_enabled already read the current role on every
  scan. It now also runs on nodes whose role has no task type (the
  scan returns at once until one is added), a job claimed before a
  role change is handed back at once rather than run or held until
  its lease lapses, and a settings reload wakes the manager so a role
  that gained task types starts claiming them straight away.

Starting the queue manager on every node also drains the queue channel
on nodes without outboundMta. Upstream left that channel unread, so
each message queued there parked a refresh in it, and by the code,
queueing would block once 1024 had piled up (not reproduced here).

A role object edit reaches the nodes that name that role in
INBUXA_ROLE. Moving a node to another role still means changing its
environment, and so a restart. Listener changes in a role still need a
restart too (listeners bind at boot); this change is about tasks and
delivery.

cluster::live_roles::live_role_tests (new; PostgreSQL, two nodes over
one store):
1. A node started with outboundMta delivers and runs a TLS report
   task; after its role loses outboundMta and the settings reload, a
   new message isn't attempted and a new report task stays pending;
   with the role back, both are taken up.
2. A node started with no task type at all gains outboundMta: a
   waiting message is attempted and a report task runs.
On main the test fails at step 1 ("delivery attempted without
outboundMta"); with step 1 bypassed, step 2 fails (nothing picked the
message up in 20 s).

crates/services/src/task_manager/manager.rs

@@ -55,23 +55,12 @@ const PERPETUAL_RETRY_MIN_DELAY: u64 = 3600;
 const PERPETUAL_RETRY_MAX_DELAY: u64 = 21600;
 
 pub fn spawn_task_manager(inner: Arc<Inner>) {
-    let is_clustered = {
-        let server = inner.build_server();
-        let roles = &server.core.network.roles;
-
-        // inbuxa: outbound_mta too, which now governs report tasks
-        if !roles.account_maintenance
-            && !roles.store_maintenance
-            && !roles.search_indexing
-            && !roles.spam_training
-            && !roles.outbound_mta
-            && !roles.task_manager
-        {
-            return;
-        }
-
-        server.core.storage.coordinator.is_enabled()
-    };
+    // inbuxa: upstream didn't start the task manager on a node whose role
+    // had no task types at boot, so adding one later did nothing until a
+    // restart. It now always runs and reads the role on every scan and
+    // before every job (task_enabled), so a role change applies at the next
+    // settings reload.
+    let is_clustered = inner.build_server().core.storage.coordinator.is_enabled();
 
     trc::event!(TaskManager(TaskManagerEvent::ManagerStarted));
 
@@ -151,20 +140,23 @@ pub fn spawn_task_manager(inner: Arc<Inner>) {
                     let server = inner.build_server();
                     let batch_size = server.core.email.index_batch_size;
                     let mut batch = Vec::with_capacity(batch_size);
-                    if let Some(task) = fetch_task(&server, job).await {
+                    if let Some(task) = fetch_enabled_task(&server, job).await {
                         batch.push(task);
                     }
 
                     while batch.len() < batch_size {
                         match rx.try_recv() {
                             Ok(job) => {
-                                if let Some(task) = fetch_task(&server, job).await {
+                                if let Some(task) = fetch_enabled_task(&server, job).await {
                                     batch.push(task);
                                 }
                             }
                             Err(_) => break,
                         }
                     }
+                    if batch.is_empty() {
+                        continue;
+                    }
 
                     // Dispatch. inbuxa: on a task of its own, so a panic
                     // releases the batch's locks and leaves this worker
@@ -205,7 +197,8 @@ pub fn spawn_task_manager(inner: Arc<Inner>) {
                     let server = inner.build_server();
                     let mut refresh_queue = false;
 
-                    if let Some(TaskDetails { task, info }) = fetch_task(&server, job).await {
+                    if let Some(TaskDetails { task, info }) = fetch_enabled_task(&server, job).await
+                    {
                         // inbuxa: on a task of its own, as above
                         let run = {
                             let server = server.clone();
@@ -274,6 +267,17 @@ impl TaskQueueManager for Server {
         if task_locks.is_stopping() {
             return Duration::from_secs(QUEUE_REFRESH_INTERVAL);
         }
+        // inbuxa: with no task type enabled by this node's role there is
+        // nothing to claim; a settings reload wakes the manager when that
+        // changes
+        let roles = &self.core.network.roles;
+        if !(0..TaskType::COUNT as u16)
+            .filter_map(TaskType::from_id)
+            .any(|task_type| task_enabled(roles, task_type))
+        {
+            ipc.locked.clear();
+            return Duration::from_secs(QUEUE_REFRESH_INTERVAL);
+        }
         let lock_expiry = task_locks.expiry();
         let now_timestamp = now();
         let from_key = ValueKey::<ValueClass> {
@@ -296,7 +300,6 @@ impl TaskQueueManager for Server {
         let mut tasks = Vec::new();
         let now = Instant::now();
         let mut next_event = None;
-        let roles = &self.core.network.roles;
         ipc.revision += 1;
         let _ = self
             .store()
@@ -544,6 +547,25 @@ async fn run_task(
     }
 }
 
+/// inbuxa: reads a claimed task when this node's role still allows its type.
+/// The role may have changed since the task was claimed (a settings reload in
+/// between); the claim is then handed back at once for a node that may run
+/// it, rather than held until the lease runs out.
+async fn fetch_enabled_task(server: &Server, job: TaskJob) -> Option<TaskDetails> {
+    if task_enabled(&server.core.network.roles, job.typ) {
+        fetch_task(server, job).await
+    } else {
+        trc::event!(
+            TaskManager(TaskManagerEvent::TaskIgnored),
+            Id = job.id,
+            Details = job.typ.as_str(),
+            Reason = "Task type was disabled by cluster roles after it was claimed.",
+        );
+        server.remove_index_lock(job.id).await;
+        None
+    }
+}
+
 /// Reads a claimed task. When it is gone or can't be read, the claim is
 /// released: inbuxa: holding it would block the task, everywhere, until
 /// the lock expired.