5dde9793eb0df8df372cfc01da29bbda4f36416c
5
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 jcoffey-dev
|
5dde9793eb |
Every node records DMARC and TLS results for the aggregate reports
The report scheduler dropped DMARC and TLS events on a node whose role lacks outboundMta (upstream never started it there, so they sat in a channel nobody read). Mail received on a front node therefore never reached an aggregate report, which is meant to cover all of a domain's inbound mail, whichever node received it. In rehearsal, five messages received on port 25 on a front node were missing from every report. - The report scheduler records on every node. Recording is a store write the nodes already share, so it needs nothing from the outbound MTA. Building and sending a report (the DmarcReport and TlsReport tasks) stay with outboundMta nodes, as the task manager already enforces. - More nodes now append to one report at once. Appends already guard the report's versioned primary key; a write that loses now retries up to ten times after a short random pause, not three times at once. - The node sending a report deletes it only if it is unchanged since it was read, and reads it again otherwise, so a record another node appends meanwhile goes out with the report instead of being deleted unsent. Test: cluster::front_reports (PostgreSQL and MySQL). A front node's results appear in the report the MTA node sends, alongside eight appended at once from both nodes, and the front node never runs the report task. It fails on main: the front node's results are never recorded. |
||
|
jcoffey-dev
|
e00978c0b4 |
Cluster role changes apply to delivery and tasks without a restart
In cluster rehearsal 3, turning outboundMta off on node1's role was
reported applied (x:settingsReload applied: true), yet node1 kept
delivering mail, a report message included, until it was restarted.
The queue and report managers were started at boot only when the
node's role included outboundMta (crates/smtp/src/lib.rs), and the task
manager only when the role had some task type (spawn_task_manager).
After that nothing looked at the role again: a queue manager that was
running kept claiming and delivering, and one that wasn't never
started.
They now start on every node (outside recovery mode) and follow the
role live:
- Queue manager: before each scan it reads the role from the running
settings. Without outboundMta it claims nothing new; deliveries
already running finish and report back as usual, which releases
their locks. When the role comes back (a reload wakes the manager
with ReloadSettings, and it looks again every 30 s regardless) it
logs queue.started and scans the whole queue at once.
- Report scheduler: DMARC and TLS report events are handled only while
the role has outboundMta, as at boot; events arriving without it are
dropped, as they were on a node started without the role.
- Task manager: task_enabled already read the current role on every
scan. It now also runs on nodes whose role has no task type (the
scan returns at once until one is added), a job claimed before a
role change is handed back at once rather than run or held until
its lease lapses, and a settings reload wakes the manager so a role
that gained task types starts claiming them straight away.
Starting the queue manager on every node also drains the queue channel
on nodes without outboundMta. Upstream left that channel unread, so
each message queued there parked a refresh in it, and by the code,
queueing would block once 1024 had piled up (not reproduced here).
A role object edit reaches the nodes that name that role in
INBUXA_ROLE. Moving a node to another role still means changing its
environment, and so a restart. Listener changes in a role still need a
restart too (listeners bind at boot); this change is about tasks and
delivery.
cluster::live_roles::live_role_tests (new; PostgreSQL, two nodes over
one store):
1. A node started with outboundMta delivers and runs a TLS report
task; after its role loses outboundMta and the settings reload, a
new message isn't attempted and a new report task stays pending;
with the role back, both are taken up.
2. A node started with no task type at all gains outboundMta: a
waiting message is attempted and a report task runs.
On main the test fails at step 1 ("delivery attempted without
outboundMta"); with step 1 bypassed, step 2 fails (nothing picked the
message up in 20 s).
|
||
|
jcoffey-dev
|
1543ea5a9e |
Task manager: every task type follows the node's cluster role
A 3-node rehearsal found taskQueueProcessing didn't filter anything: roles.task_manager only decided whether the task manager started, and report, ACME, DKIM, DNS, calendar, thread-merge and restore tasks ran on any node with a task manager (manager.rs returned true for them). A node whose role left taskQueueProcessing off still ran them if it indexed or did maintenance. Every task type now answers to one ClusterTaskType (task_enabled): - IndexDocument, UnindexDocument, IndexTrace: searchIndexing - AccountMaintenance, TenantMaintenance, DestroyAccount: accountMaintenance - StoreMaintenance: storeMaintenance - SpamFilterMaintenance: spamClassifierTraining - DmarcReport, TlsReport: outboundMta. They build and send reports to other domains (TLS reports can go straight to an HTTPS endpoint), which is the outbound MTA's business. - CalendarAlarmEmail, CalendarAlarmNotification, CalendarItipMessage, MergeThreads, RestoreArchivedItem, AcmeRenewal, DkimManagement, DnsManagement: taskQueueProcessing, the role for queue tasks with no role of their own. A node that may not run a task leaves it unclaimed (no lock), so a node that may picks it up. The task manager also starts on a node whose only task role is outboundMta, so reports still run there. cluster::task_roles::task_role_tests (new, two task managers over one PostgreSQL store): node A (taskQueueProcessing only) runs a DNS task and leaves an unindex task and a TLS report pending; node B (searchIndexing and outboundMta) comes up and runs those two; a DNS task scheduled next stays pending on B and runs on A. On main node A runs the TLS report. |
||
|
jcoffey-dev
|
95f0445d83 |
Coordinator: join the cluster when NATS comes up, report the connection
A node that started while NATS was down never got a coordinator. The
connect failed at boot, bootstrap recorded a build error and the node ran
with Coordinator::None until restarted. It had no broadcast subscriber
or publisher, so cross-node push and cache invalidation to it stayed
broken, and its healthcheck said nothing about it. Losing NATS after
startup was silent too.
- The NATS client now connects in the background
(retry_on_initial_connect): startup never waits on NATS or fails over
it, the node gets its coordinator, subscriber and publisher at once,
and the client keeps trying (async-nats's backoff, at most 4 s apart)
until NATS answers. Subscriptions made meanwhile start delivering when
it does. A configured maxReconnects still ends the attempts.
- Three new events report the connection: cluster.coordinator-connected
(info), cluster.coordinator-disconnected (warn: lost, closed, gave up,
or not connected within the connection timeout at startup) and
cluster.coordinator-error (warn: a failed attempt, reported once per
outage rather than every retry, and server errors, slow consumers and
lame duck mode). They are in the packaged schema, ids 644 to 646.
- GET /healthz/cluster reports the coordinator: 200
{"coordinator":"connected"}, 503 {"coordinator":"disconnected"}, or
200 with "none" (no coordinator) or "unknown" (a backend that doesn't
track its connection). /healthz/live and /healthz/ready are unchanged
on purpose: a node without its coordinator still serves mail, and
failing those would have orchestrators restart, or pull out of
service, every node at once whenever NATS is down.
Only NATS connects lazily; the other coordinator backends still fail at
boot as before.
cluster::coordinator::coordinator_reconnect_tests starts a node against a
NATS port with nothing behind it, checks it boots with a coordinator and
reports it disconnected, subscribes, then starts NATS on that port: the
node connects on its own and the subscription receives a message from a
second client. Stopping and restarting NATS shows disconnected, then
connected, and the same subscription keeps working.
|
||
|
jcoffey-dev
|
7dae9b29fd |
Import upstream v0.16.22, stripped
Upstream commit: 474dd0229cb20cf513036619781ed97bd8073c3f Enterprise-only files removed or emptied: 63 Enterprise-only snippets removed: 117 in 50 files Dangling module declarations removed: 5 Cargo edits turning enterprise off: 14 Verification: clean Enterprise feature gates left for rebuilt features: 19 in 18 files Produced by tools/fork/strip.py. The full report is in docs/fork/strip-reports/ on main. |