Cluster role changes apply to delivery and tasks without a restart #44

Merged

jcoffey-dev merged 1 commits from fix/live-role-changes into main

2026-09-25 00:56:31 +00:00

jcoffey-dev commented

2026-09-24 23:15:41 +00:00

Owner

Problem

In cluster rehearsal 3, turning outboundMta off in node1's role came back x:settingsReload: {"applied": true}. Node1 kept delivering mail anyway, including a message in the report queue, until it was restarted.

The role was read only once, at boot:

Queue and report managers (crates/smtp/src/lib.rs:47): started only if the node's role included outboundMta at boot.
Task manager (crates/services/src/task_manager/manager.rs:62-70): started only if the role had some task type at boot.

After boot nothing looked at the role again. A running queue manager kept claiming and delivering, and a manager that wasn't started never started.

Choice: follow the role live

I made the services follow the role instead of adding a restartRequired flag. The change turned out to be small: each loop already has a natural point to check the role, and the running settings are reloaded on every role write (#39). A restart flag would have left the rehearsal's behavior in place and only reported it.

They now start on every node (outside recovery mode):

Queue manager. Before each scan it reads the role from the running settings.
- Without outboundMta it claims nothing new.
- Deliveries already running finish and report back as usual, which releases their locks.
- When the role comes back it logs queue.started and scans the whole queue at once. A reload wakes it through the existing ReloadSettings queue event, and it also rechecks every 30 s.
Report scheduler. It handles DMARC/TLS report events only while the role has outboundMta, as at boot. Events that arrive without the role are dropped, as they were on a node started without it.
Task manager.
- task_enabled already read the current role on every scan.
- It now also runs on nodes whose role has no task type. The scan returns at once until the role gains one.
- A job claimed before a role change is handed back at once. Before, it was run anyway or held until its lease lapsed.
- A settings reload now wakes the task manager, so newly enabled task types are claimed straight away.

Side effect. Starting the queue manager everywhere also drains the queue channel on nodes without outboundMta. Upstream never read that channel, so every message queued on such a node parked a refresh in it. Reading the code, queueing would block once 1,024 had piled up. I haven't reproduced that.

What still needs a restart

Moving a node to a different role. A role object edit reaches the nodes whose INBUXA_ROLE names that role. Moving a node to another role means changing its environment, so it restarts anyway.
Listener changes in a role. Listeners bind at boot. This PR only covers tasks and delivery.
Tracer settings (the optional item): not done. A reload applies a tracer's new event interests but not its other settings. Examples: a log tracer's path, a webhook URL, an OTel endpoint. Telemetry::update refreshes interests only. Fixing it means respawning any tracer whose settings changed, for every tracer type. That isn't trivial, so I left it out.

Tests

cluster::live_roles::live_role_tests is new. It runs on PostgreSQL with two nodes over one store. Each node queues a message to an unreachable domain and schedules a TLS report task, then:

Rehearsal case. A node that starts with outboundMta delivers and runs the report task. Its role then loses outboundMta and the settings reload:
- A new message isn't attempted and a new report task stays pending.
- When the role gets outboundMta back, both are picked up.
Role gained after boot. A node that starts with no task type at all gains outboundMta. The waiting message is attempted and the report task runs.

Results:

Run	Result
This branch	pass (23.5 s)
`main`	fails at step 1: `delivery attempted without outboundMta`
`main`, earlier version of the test with only step 2	fails: nothing picked the message up in 20 s

Regressions:

cluster::task_roles::task_role_tests (PostgreSql): pass.
smtp:: (RocksDb), 75 tests. The suite is flaky under parallel load on main too:
- main: 1 of 2 runs failed (smtp::inbound::dmarc::dmarc).
- This branch: 2 of 3 runs failed (smtp::inbound::dkim2::dkim2_all_disclosed in both, plus smtp::outbound::smtp::smtp_delivery in the first). The third run passed all 75.
- Rerun on their own, smtp_delivery and the six smtp::inbound::dkim2 tests pass.
- The SMTP tests run with every role on, where the new code does nothing different.

Command: STORE=PostgreSql cargo test -p tests --features postgres cluster::live_roles

## Problem In cluster rehearsal 3, turning outboundMta off in node1's role came back `x:settingsReload: {"applied": true}`. Node1 kept delivering mail anyway, including a message in the `report` queue, until it was restarted. The role was read only once, at boot: - **Queue and report managers** (`crates/smtp/src/lib.rs:47`): started only if the node's role included outboundMta at boot. - **Task manager** (`crates/services/src/task_manager/manager.rs:62-70`): started only if the role had some task type at boot. After boot nothing looked at the role again. A running queue manager kept claiming and delivering, and a manager that wasn't started never started. ## Choice: follow the role live I made the services follow the role instead of adding a `restartRequired` flag. The change turned out to be small: each loop already has a natural point to check the role, and the running settings are reloaded on every role write (#39). A restart flag would have left the rehearsal's behavior in place and only reported it. They now start on every node (outside recovery mode): - **Queue manager.** Before each scan it reads the role from the running settings. - Without outboundMta it claims nothing new. - Deliveries already running finish and report back as usual, which releases their locks. - When the role comes back it logs `queue.started` and scans the whole queue at once. A reload wakes it through the existing `ReloadSettings` queue event, and it also rechecks every 30 s. - **Report scheduler.** It handles DMARC/TLS report events only while the role has outboundMta, as at boot. Events that arrive without the role are dropped, as they were on a node started without it. - **Task manager.** - `task_enabled` already read the current role on every scan. - It now also runs on nodes whose role has no task type. The scan returns at once until the role gains one. - A job claimed before a role change is handed back at once. Before, it was run anyway or held until its lease lapsed. - A settings reload now wakes the task manager, so newly enabled task types are claimed straight away. **Side effect.** Starting the queue manager everywhere also drains the queue channel on nodes without outboundMta. Upstream never read that channel, so every message queued on such a node parked a refresh in it. Reading the code, queueing would block once 1,024 had piled up. I haven't reproduced that. ## What still needs a restart - **Moving a node to a different role.** A role object edit reaches the nodes whose `INBUXA_ROLE` names that role. Moving a node to another role means changing its environment, so it restarts anyway. - **Listener changes in a role.** Listeners bind at boot. This PR only covers tasks and delivery. - **Tracer settings (the optional item): not done.** A reload applies a tracer's new event interests but not its other settings. Examples: a log tracer's path, a webhook URL, an OTel endpoint. `Telemetry::update` refreshes interests only. Fixing it means respawning any tracer whose settings changed, for every tracer type. That isn't trivial, so I left it out. ## Tests `cluster::live_roles::live_role_tests` is new. It runs on PostgreSQL with two nodes over one store. Each node queues a message to an unreachable domain and schedules a TLS report task, then: 1. **Rehearsal case.** A node that starts with outboundMta delivers and runs the report task. Its role then loses outboundMta and the settings reload: - A new message isn't attempted and a new report task stays pending. - When the role gets outboundMta back, both are picked up. 2. **Role gained after boot.** A node that starts with no task type at all gains outboundMta. The waiting message is attempted and the report task runs. Results: | Run | Result | |---|---| | This branch | pass (23.5 s) | | `main` | fails at step 1: `delivery attempted without outboundMta` | | `main`, earlier version of the test with only step 2 | fails: nothing picked the message up in 20 s | Regressions: - `cluster::task_roles::task_role_tests` (PostgreSql): pass. - `smtp::` (RocksDb), 75 tests. The suite is flaky under parallel load on `main` too: - `main`: 1 of 2 runs failed (`smtp::inbound::dmarc::dmarc`). - This branch: 2 of 3 runs failed (`smtp::inbound::dkim2::dkim2_all_disclosed` in both, plus `smtp::outbound::smtp::smtp_delivery` in the first). The third run passed all 75. - Rerun on their own, `smtp_delivery` and the six `smtp::inbound::dkim2` tests pass. - The SMTP tests run with every role on, where the new code does nothing different. Command: `STORE=PostgreSql cargo test -p tests --features postgres cluster::live_roles`

jcoffey-dev added 1 commit 2026-09-25 00:48:56 +00:00

Cluster role changes apply to delivery and tasks without a restart

ci / fork-checks (pull_request) Successful in 17s

Details

ci / build (pull_request) Successful in 7m11s

Details

e00978c0b4

In cluster rehearsal 3, turning outboundMta off on node1's role was
reported applied (x:settingsReload applied: true), yet node1 kept
delivering mail, a report message included, until it was restarted.
The queue and report managers were started at boot only when the
node's role included outboundMta (crates/smtp/src/lib.rs), and the task
manager only when the role had some task type (spawn_task_manager).
After that nothing looked at the role again: a queue manager that was
running kept claiming and delivering, and one that wasn't never
started.

They now start on every node (outside recovery mode) and follow the
role live:

- Queue manager: before each scan it reads the role from the running
  settings. Without outboundMta it claims nothing new; deliveries
  already running finish and report back as usual, which releases
  their locks. When the role comes back (a reload wakes the manager
  with ReloadSettings, and it looks again every 30 s regardless) it
  logs queue.started and scans the whole queue at once.
- Report scheduler: DMARC and TLS report events are handled only while
  the role has outboundMta, as at boot; events arriving without it are
  dropped, as they were on a node started without the role.
- Task manager: task_enabled already read the current role on every
  scan. It now also runs on nodes whose role has no task type (the
  scan returns at once until one is added), a job claimed before a
  role change is handed back at once rather than run or held until
  its lease lapses, and a settings reload wakes the manager so a role
  that gained task types starts claiming them straight away.

Starting the queue manager on every node also drains the queue channel
on nodes without outboundMta. Upstream left that channel unread, so
each message queued there parked a refresh in it, and by the code,
queueing would block once 1024 had piled up (not reproduced here).

A role object edit reaches the nodes that name that role in
INBUXA_ROLE. Moving a node to another role still means changing its
environment, and so a restart. Listener changes in a role still need a
restart too (listeners bind at boot); this change is about tasks and
delivery.

cluster::live_roles::live_role_tests (new; PostgreSQL, two nodes over
one store):
1. A node started with outboundMta delivers and runs a TLS report
   task; after its role loses outboundMta and the settings reload, a
   new message isn't attempted and a new report task stays pending;
   with the role back, both are taken up.
2. A node started with no task type at all gains outboundMta: a
   waiting message is attempted and a report task runs.
On main the test fails at step 1 ("delivery attempted without
outboundMta"); with step 1 bypassed, step 2 fails (nothing picked the
message up in 20 s).

jcoffey-dev force-pushed fix/live-role-changes from 6a80aaea08 to e00978c0b4

2026-09-25 00:48:56 +00:00

Compare

jcoffey-dev merged commit b90a7f173e into main

2026-09-25 00:56:31 +00:00

jcoffey-dev deleted branch fix/live-role-changes

2026-09-25 00:56:31 +00:00

jcoffey-dev referenced this pull request

2026-09-25 01:28:32 +00:00

Every node records DMARC and TLS results for the aggregate reports #47