inbuxa/inbuxa-server
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 3 Wiki Activity

Task manager: every task type follows the node's cluster role #40

Merged
jcoffey-dev merged 1 commits from fix/task-role-filtering into main 2026-09-24 20:06:55 +00:00
Conversation 0 Commits 1 Files Changed 3
+265 -20
jcoffey-dev commented 2026-09-24 19:47:00 +00:00
Owner
Copy Link
Copy Source

Problem

The taskQueueProcessing cluster role didn't filter any tasks. roles.task_manager only decided whether the task manager started (crates/services/src/task_manager/manager.rs:61-66).

In the claim scan, report, ACME, DKIM, DNS, calendar, thread-merge and restore tasks matched => true (manager.rs:301-310), so they ran on any node that had a task manager. A node whose role turned taskQueueProcessing off still ran them if it also did search indexing or maintenance.

Change

Each task type now answers to exactly one ClusterTaskType, in task_enabled() in manager.rs:

Task types Role
IndexDocument, UnindexDocument, IndexTrace searchIndexing (unchanged)
AccountMaintenance, TenantMaintenance, DestroyAccount accountMaintenance (unchanged)
StoreMaintenance storeMaintenance (unchanged)
SpamFilterMaintenance spamClassifierTraining (unchanged)
DmarcReport, TlsReport outboundMta
CalendarAlarmEmail, CalendarAlarmNotification, CalendarItipMessage, MergeThreads, RestoreArchivedItem, AcmeRenewal, DkimManagement, DnsManagement taskQueueProcessing

Why these:

  • Reports → outboundMta. Report tasks build reports and send them to other domains. TLS reports may go straight to an HTTPS endpoint. That is the outbound MTA's job, so a node kept off outbound mail doesn't send them.
  • Everything else → taskQueueProcessing. ACME, DKIM and DNS automation, calendar alarms and iTIP, thread merges and restores have no dedicated role. taskQueueProcessing is the catch-all its name promises.

How claiming works now:

  • A node that may not run a task skips it before claiming. It takes no lock, so the task stays free for a node that may run it.
  • The task manager now also starts when outboundMta is a node's only task role, so reports still run there.

If no node's role allows a task type, those tasks wait. That was already true for the dedicated roles.

Tests

cluster::task_roles::task_role_tests is new. It runs two task managers over one PostgreSQL store (STORE=PostgreSql; it skips itself on RocksDb and SQLite, whose files two nodes can't share).

  1. Node A has taskQueueProcessing only. Three tasks are scheduled: a DNS task, an unindex task (searchIndexing) and a TLS report (outboundMta). A runs the DNS task. The other two are still pending about 5 s later.
  2. Node B comes up with searchIndexing and outboundMta. It runs the unindex task and the TLS report.
  3. A new DNS task stays pending on B, then runs on A.

Results:

  • PostgreSql: pass.
  • On main: fails at step 1 (TLS report ran on node A).
  • store::task_locks::task_lock_tests (RocksDb): pass.
  • system::system_tests (RocksDb): its quota step failed here with and without this change (main's manager.rs swapped in, same worktree), and passed on another worktree with main's code. It looks timing-dependent on this shared machine, not related to this PR.
## Problem The `taskQueueProcessing` cluster role didn't filter any tasks. `roles.task_manager` only decided whether the task manager started (`crates/services/src/task_manager/manager.rs:61-66`). In the claim scan, report, ACME, DKIM, DNS, calendar, thread-merge and restore tasks matched `=> true` (`manager.rs:301-310`), so they ran on any node that had a task manager. A node whose role turned `taskQueueProcessing` off still ran them if it also did search indexing or maintenance. ## Change Each task type now answers to exactly one `ClusterTaskType`, in `task_enabled()` in `manager.rs`: | Task types | Role | |---|---| | IndexDocument, UnindexDocument, IndexTrace | searchIndexing *(unchanged)* | | AccountMaintenance, TenantMaintenance, DestroyAccount | accountMaintenance *(unchanged)* | | StoreMaintenance | storeMaintenance *(unchanged)* | | SpamFilterMaintenance | spamClassifierTraining *(unchanged)* | | DmarcReport, TlsReport | **outboundMta** | | CalendarAlarmEmail, CalendarAlarmNotification, CalendarItipMessage, MergeThreads, RestoreArchivedItem, AcmeRenewal, DkimManagement, DnsManagement | **taskQueueProcessing** | Why these: - **Reports → outboundMta.** Report tasks build reports and send them to other domains. TLS reports may go straight to an HTTPS endpoint. That is the outbound MTA's job, so a node kept off outbound mail doesn't send them. - **Everything else → taskQueueProcessing.** ACME, DKIM and DNS automation, calendar alarms and iTIP, thread merges and restores have no dedicated role. `taskQueueProcessing` is the catch-all its name promises. How claiming works now: - A node that may not run a task skips it before claiming. It takes no lock, so the task stays free for a node that may run it. - The task manager now also starts when `outboundMta` is a node's only task role, so reports still run there. **If no node's role allows a task type, those tasks wait.** That was already true for the dedicated roles. ## Tests `cluster::task_roles::task_role_tests` is new. It runs two task managers over one PostgreSQL store (`STORE=PostgreSql`; it skips itself on RocksDb and SQLite, whose files two nodes can't share). 1. Node A has `taskQueueProcessing` only. Three tasks are scheduled: a DNS task, an unindex task (`searchIndexing`) and a TLS report (`outboundMta`). A runs the DNS task. The other two are still pending about 5 s later. 2. Node B comes up with `searchIndexing` and `outboundMta`. It runs the unindex task and the TLS report. 3. A new DNS task stays pending on B, then runs on A. Results: - PostgreSql: pass. - On `main`: fails at step 1 (`TLS report ran on node A`). - `store::task_locks::task_lock_tests` (RocksDb): pass. - `system::system_tests` (RocksDb): its quota step failed here with and without this change (`main`'s `manager.rs` swapped in, same worktree), and passed on another worktree with `main`'s code. It looks timing-dependent on this shared machine, not related to this PR.
jcoffey-dev added 1 commit 2026-09-24 19:47:01 +00:00
Task manager: every task type follows the node's cluster role
ci / fork-checks (pull_request) Successful in 14s
Details
ci / build (pull_request) Successful in 3m17s
Details
1543ea5a9e 
A 3-node rehearsal found taskQueueProcessing didn't filter anything:
roles.task_manager only decided whether the task manager started, and
report, ACME, DKIM, DNS, calendar, thread-merge and restore tasks ran on
any node with a task manager (manager.rs returned true for them). A node
whose role left taskQueueProcessing off still ran them if it indexed or
did maintenance.

Every task type now answers to one ClusterTaskType (task_enabled):

- IndexDocument, UnindexDocument, IndexTrace: searchIndexing
- AccountMaintenance, TenantMaintenance, DestroyAccount:
  accountMaintenance
- StoreMaintenance: storeMaintenance
- SpamFilterMaintenance: spamClassifierTraining
- DmarcReport, TlsReport: outboundMta. They build and send reports to
  other domains (TLS reports can go straight to an HTTPS endpoint),
  which is the outbound MTA's business.
- CalendarAlarmEmail, CalendarAlarmNotification, CalendarItipMessage,
  MergeThreads, RestoreArchivedItem, AcmeRenewal, DkimManagement,
  DnsManagement: taskQueueProcessing, the role for queue tasks with no
  role of their own.

A node that may not run a task leaves it unclaimed (no lock), so a node
that may picks it up. The task manager also starts on a node whose only
task role is outboundMta, so reports still run there.

cluster::task_roles::task_role_tests (new, two task managers over one
PostgreSQL store): node A (taskQueueProcessing only) runs a DNS task and
leaves an unindex task and a TLS report pending; node B (searchIndexing
and outboundMta) comes up and runs those two; a DNS task scheduled next
stays pending on B and runs on A. On main node A runs the TLS report.
jcoffey-dev merged commit 127ef5701d into main 2026-09-24 20:06:55 +00:00
Sign in to join this conversation.
Reviewers
No Reviewers
Labels
No items
No labels
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
jcoffey-dev (John Coffey)
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: inbuxa/inbuxa-server#40