jcoffey-dev is traveling from Thursday 1 October through Sunday 4 October. Issues and pull requests are welcome, and will get an answer after that. Thanks for your patience.
What administrators and the server itself do to the control plane is now
recorded, from inbuxa-drafts/specs/audit-hold-lock.md (AU-1 to AU-12):
settings, accounts, domains, roles and every other registry change, with
each field's before and after (secrets only as "changed"); the fork's own
settings objects; administrator sign-ins (and failed ones to administrator
accounts), master-user and recovery-admin sign-ins, once an hour per
account, method and address; access to another account's data through
impersonation or FetchAnyBlob, once an hour; exports and tamper checks;
and registry writes the server makes on its own, named by subsystem
(system:AcmeRenewal, system:auto-ban, system:directory-sync, ...), with a
spam rules update as one summary record.
No change without its record (AU-3): before a set method changes anything,
a pending record per requested create, update and destroy is written; if
that fails, the method is refused with serverFail. Its outcome follows as
a later entry. A change interrupted by a crash stays "unfinished".
Records live in the fork's subspace under L, as one SHA-256 hash chain per
node. The chain's head is stored, never cached, and every append asserts
it, so two writers can't take the same place. Nothing can edit or delete
a record; the daily purge removes the oldest past the retention (default
730 days, minimum 90) and records where the chain now starts, so
verification still passes. security.audit-recorded (647) copies each
record to webhooks, OpenTelemetry and the log; security.audit-write-failed
(648) reports a failed write.
New JMAP objects under urn:inbuxa:jmap: inbuxa:AuditEvent/get and /query
(filters: time, actor, action, target, account, tenant, outcome, address,
text), inbuxa:AuditSettings, inbuxa:AuditExport (CSV or JSON Lines built
on the server, each line with its chain hash, ending in a manifest; the
created object names the blob and its SHA-256) and
inbuxa:AuditVerification. New permissions sysAuditGet, sysAuditExport and
sysAuditSettingsUpdate: the Administrator role gets all three, the Tenant
Administrator role gets read and export, once, on existing installs too.
A tenant administrator sees records whose actor or target is in its
tenant, including a server administrator's changes there.
Sign-in method on the session: access tokens now remember how they signed
in (password, app password, API key, OAuth client, directory, master user,
recovery admin), including across the HTTP credential cache. New OAuth
access tokens carry their client id in the sealed claims; older ones show
as client "unknown" until they expire.
The schema gains the permissions, the two events and a Management >
Compliance > Audit Log link.
Stack: the request layer boxes every inner future where it's made. Without
that, a debug build overflowed the default 2 MB worker stack on a registry
set; measured with the same request, the branch and main now overflow at
the same stack size (between 1856 and 1920 KiB, debug), so the layer adds
nothing measurable.
Tests: unit tests in inbuxa-features and jmap; system::audit::audit_log_tests
(run with --ignored) passes on RocksDB, SQLite, PostgreSQL, PostgreSQL with a
read replica, MySQL, MySQL with a replica and FoundationDB. The system, JMAP
and SCIM suites pass. authorization.rs skipped fork permissions that guard
no registry object; the audit suite checks a plain user is refused instead.
inbuxa's own mark (#62): the kitten over a server with a bay for each
piece of the suite, on the built-in sign-in and RSVP pages, the web
logo and the email logo.
Prepared Explain answers relabeled for this release; no setting changed
since 2026.9.26.1, so all 706 carry over.
inbuxa's mark was ihasmail's cat-and-envelope reused unchanged. The new
one keeps the family's face, paws and colors, over a server with a bay
for each piece of the suite: the letter (webmail), a prompt (console),
status lights (server).
- The built-in sign-in and calendar RSVP pages, and the web logo, drew
the old cat as an embedded PNG. They now draw the mark as vector in
the same slot, keeping class="symbol"; each page is about 31 KB
lighter. The .min copies are updated the same way and the .min.gz
regenerated with gzip -9 -n, as minify_html.sh does.
- resources/branding: email-logo.png (the compact lockup at 380x80 on
white, as before) with its .b64 regenerated byte-for-byte in the old
76-column form, and favicon-64.png.
- img/brand: the logo bundle, now pure vector, with its README.
announce.yml runs coffey-labs/actions discourse-release on every published
release, posting it to this project's Announcements category on
community.coffeylabs.org. The release workflow also announces
from its own job, since a release made with the job token fires no
'on: release' workflow in Gitea.
Prepared Explain answers relabeled for this release; 11 settings whose
default is the time of creation drop out, since their answers could never
match.
ai-explain spec, amendment 1 (EX-22 to EX-28):
- answers are three or four sentences, max_tokens 160, cut at 700 chars;
- POST /api/explain streams the answer as server-sent events;
- each node remembers answers in memory (1,000, 24 h), keyed by the facts,
prompt version and model, shared by server-level administrators;
- resources/explain/settings.json.gz ships answers for settings at their
defaults, generated with prepare_setting_explanations (717 for 2026.9.27);
- the system prompt no longer carries the per-request marker, so a model
server can reuse it;
- inbuxa:Explanation gains source, answeredAt and preparedFor.
When a listener couldn't bind its address (a port below 1024 without
root, a port already in use, or the legacy-protocols switch putting a
listener back after privileges were dropped), the bind error was
reported but the socket was still passed to listen(). The kernel then
bound it itself, to a random port on every interface, and the server
logged the listener as started on the port it was configured with.
listen() now refuses a socket that isn't bound, so the listener is
reported with a listen error and skipped, and nothing opens anywhere
unexpected.
A singleton such as x:SpamSettings has no stored object until someone
saves it; /get shows its defaults instead. Explain looked only for the
stored object, so every setting still at its defaults answered "No
such x:SpamSettings." It now falls back to the defaults the same way.
A new method, inbuxa:Explanation/set, asks the node's local model for a
short plain-words reading of one thing an administrator is looking at:
a failed recipient in the queue, a Classify verdict, a log line or trace
event, or one setting with its saved value. The server builds the prompt
itself from stored data and the registry schema, never from text the
console sends, and grounds SMTP replies in RFC 3463 and RFC 5321.
What the model is never shown: secrets (including ones nested inside a
setting, like an AI model's HTTP auth), raw protocol events, and the
contents of any other event. A tag name that doesn't have a tag's shape is
refused before a model is asked.
Calls share the AI gate with spam classification, but mail always keeps
its slot, and Explain has its own hourly count per account and its own
on/off switch in inbuxa:AiLimits. The permission is sysAiExplain,
superuser only; tenant administrators can't use it. The session carries
an aiExplain flag so a console knows when to offer the button.
An install whose roles were stored before the permission existed gets it
added once, at start-up, to the roles that are administrators' alone,
not the User role their defaults share with every account. An operator
who removes it later isn't overruled.
Tests: unit tests in inbuxa-features and jmap, and ai_explain_tests
(run with --ignored) covering the acceptance tests and the upgrade.
Every node renews its lease once a minute instead of every 30 minutes,
so the lease works as a heartbeat. x:ClusterNode reports a node Stale
once it has gone three minutes without renewing (it used to take an
hour), and Inactive after a day, as before.
Taking over a lease still needs a full hour of silence. A node that is
slow rather than gone never loses its id to another host, so snowflake
ids stay unique.
The admin dashboard's Cluster Health card counts these statuses.
After #37, address fields on PostgreSQL are split into words as the
built-in index splits them, but language text (subject, body,
attachments) still goes straight to PostgreSQL's parser, which keeps a
URL, host, path or file name as tokens of its own:
"https://x.example/shipping-support/" becomes a url, a host and a
url_path, "invoice-2024.pdf" a file. So TEXT/BODY "shipping" missed
messages where the word appears only inside a link, while RocksDB and
the other built-in backends found them: 8 messages across a handful
of searches in the rehearsal.
On insert, language text is now indexed as it was, followed by the
word parts of each token that holds a URL separator (/ . @ : ? = & # _
% + ~ \), split with SpaceTokenizer as keyword_terms() splits addresses.
The parts go through the same text search configuration as the rest of
the text, so they are stemmed like the words around them. Plain words,
words that only carry punctuation ("end.", "(see") and hyphenated words
(the parser already splits those) add nothing, so text without links
is indexed exactly as before. Each part is added once per document.
On sample mail, the text vector of a short order notice with three
links grows from 546 to 716 bytes, a newsletter with 25 tracking links
from 5586 to 6430, and a plain letter not at all.
On search, a query word written as a URL, host, file or hyphenated word
also matches as its word parts, ORed with the query as written, so
"shipping-support" or "invoice-2024.pdf" match the new parts and
documents indexed before this change still match as they did.
Existing messages keep their old vectors until they are reindexed (the
reindexAccounts task); new and reindexed messages match at once.
store::search_tests gains test_url_word_search: five bodies, 19 body
searches for words found only in a URL path, query string, host or
file name, the tokens as written, plain words and non-matches, with the
same expected ids on every backend. It passes on RocksDB, SQLite,
MySQL and PostgreSQL; on main PostgreSQL fails at the first ("shipping"
finds [3], not [0, 3]). On PostgreSQL the suite then stops at the
account sort assertion (query.rs:689) exactly as it does on main.
A cluster rehearsal sent ten x:<Object>/set requests at once and got
ten full reloads on every node. #39's coalescing only joined writes
that queued behind a running reload, but the requests reached the
server about 33 ms apart and a reload takes tens of milliseconds, so
none overlapped one.
A full reload after a registry write now waits for writes to settle:
75 ms after the last one, and at most 250 ms after the first it
covers, so a steady stream still reloads at least four times a
second. 75 ms is a little over twice the gap the rehearsal saw between
requests. A single write pays it once: in the tests a settings write
takes about 140 ms instead of 60. The reload runs in a task of its
own, so a request that goes away doesn't cancel it for the others.
Each write takes the result of the first reload that started after it
was stored (the gate keeps the last 64 results), so applied true or
false still describes the reload that covered that write.
The 33 ms gap was a queue on the server, not password hashing: Basic
credentials are cached per Authorization header, so they are checked
once. Every authenticated HTTP request counted itself against the
account's rate limit by incrementing one counter per account in the
in-memory store, so parallel requests from one account queued on that
key: a row lock on PostgreSQL (a few round trips to the database
each) and conflict retries with a 50-300 ms backoff on RocksDB. An
account with the unlimitedRequests permission (administrators, by
default) passes the rate and concurrency limits anyway, so its
requests are no longer counted. Ten parallel Core/echo calls as the
admin now finish in 1-4 ms; before, they finished one after another
over 20 ms on a local PostgreSQL and 300-450 ms on RocksDB. Other
accounts still count every request.
system::auto_reload::settings_reload_tests: ten concurrent writes now
take one reload (the gate counts them; at most two allowed), all are
applied: true and in the running settings, and a single write takes
exactly one reload. RocksDB and PostgreSQL, 1 reload in 141-196 ms.
With the old behavior (no wait, requests counted) the same writes
took 5 reloads; without the wait but with the rate fix, 2.
cluster::broadcast (3 nodes, PostgreSQL + NATS) and system::reload
still pass.
A cluster rehearsal moved a Log tracer to another directory: the write
was reported x:settingsReload applied:true, but the tracer kept writing
to the old file until a restart. Telemetry::update only refreshed each
running tracer's events, level and lossiness; a tracer's own settings
(path, prefix, rotation, format, endpoint, headers, ...) stayed as built.
Each tracer now carries a hash of the registry object it was built
from, less the fields that change in place. The reload compares it with
the running tracer's: unchanged ones are updated in place as before,
changed ones are started over, new ones started and removed ones
stopped. Only tracers this server started are removed; upstream removed
every subscriber not in the settings, which also cut off live-tracing
streams on each reload.
Starting over is a swap in the collector, so no event is lost or
written twice: a subscriber registered under a running one's id
replaces it between two collection passes. The old one's batch is sent
first (what its full channel can't take moves to the new one), and
dropping it closes its channel, so its task writes what is queued and
ends. Per tracer kind:
- Log: a tracer started over on the same files (rotation or format
changed) waits for the old one to finish, so lines don't interleave.
- Webhook: the task held a sender of its own channel for retries, so
it never ended; retries now use a weak sender, and pending events are
posted when the channel closes.
- OpenTelemetry: pending logs and spans are exported when the channel
closes instead of dropped, and a span that was open across the swap
is exported by the new tracer with the events it saw.
- Console and journal: nothing kept between batches.
- Trace history: built from the tracing store, which takes a restart,
so it is never started over.
No kind needs a restart, so x:settingsReload doesn't gain one.
system::tracer_reload::tracer_reload_tests (new): a Log tracer created
over JMAP writes to its directory; its path is changed over JMAP while
2000 numbered events are emitted; after the reload, events land in the
new file and not the old one, each numbered event is in exactly one of
the two files, and a destroyed tracer writes nothing. On main the new
file never appears.
The publish workflow only built a tag whose commit is on main. That keeps
every image tied to reviewed code, but it means production can only get a
fix together with everything that has landed on main since its release.
A tag on a release/* branch is now accepted too. A hotfix branch starts at
an earlier release tag, takes fixes through pull requests into it (so the
code is still reviewed and CI-tested before it is tagged), bumps
brand_version! and is tagged there. The tag must still equal
v<brand_version!>, and the step prints which branch it was found on.
A tag runs the workflow file from its own commit, so a hotfix branch that
starts before this change needs this commit cherry-picked onto it before
its tag is pushed.
The report scheduler dropped DMARC and TLS events on a node whose role
lacks outboundMta (upstream never started it there, so they sat in a
channel nobody read). Mail received on a front node therefore never
reached an aggregate report, which is meant to cover all of a domain's
inbound mail, whichever node received it. In rehearsal, five messages
received on port 25 on a front node were missing from every report.
- The report scheduler records on every node. Recording is a store write
the nodes already share, so it needs nothing from the outbound MTA.
Building and sending a report (the DmarcReport and TlsReport tasks) stay
with outboundMta nodes, as the task manager already enforces.
- More nodes now append to one report at once. Appends already guard the
report's versioned primary key; a write that loses now retries up to ten
times after a short random pause, not three times at once.
- The node sending a report deletes it only if it is unchanged since it
was read, and reads it again otherwise, so a record another node appends
meanwhile goes out with the report instead of being deleted unsent.
Test: cluster::front_reports (PostgreSQL and MySQL). A front node's
results appear in the report the MTA node sends, alongside eight appended
at once from both nodes, and the front node never runs the report task.
It fails on main: the front node's results are never recorded.
Setting deliverAt on an internal DMARC or TLS report wrote the new task
queue row with the report's object type (0x21, 0x6e) instead of the task
type (7, 8), and left the task row at its old due. The task manager's scan
failed on that row with store.data-corruption ("Failed to iterate over task
queue"), and because the error ended the whole scan, every task due after
the row stopped running on every node.
- reschedule_ops writes the new queue row through schedule_task_with_id, so
it carries the task type and the task row gets the new due. It removes
the row the task is actually queued under (the task's due, which differs
from deliverAt once the task has been retried) and any row an earlier
reschedule left at deliverAt.
- x:DmarcInternalReport/set and x:TlsInternalReport/set lock the report's
task while they move it, as x:Task/set does, refuse while the report is
being sent, release the locks however the request ends, and wake the task
manager.
- The task manager logs a queue row it can't read (id, due, key, value) and
skips it instead of ending the scan. It then repairs the row from its task:
the row is rewritten with the task's type, and a row with no task behind
it is removed. A row holding a report's object type for a report task is
what the old reschedule wrote: the task is moved to that row's time, as
the reschedule intended, and its old queue row is removed. Stores that
already hold such a row recover on their own once it comes due.
- x:Task/query with a type filter skips an unreadable row instead of
failing.
Test: smtp::reporting::reschedule (RocksDB and PostgreSQL). It fails on
main: x:Task/get shows the old due, and with that check removed, neither
report nor a later task ever runs.
In cluster rehearsal 3, turning outboundMta off on node1's role was
reported applied (x:settingsReload applied: true), yet node1 kept
delivering mail, a report message included, until it was restarted.
The queue and report managers were started at boot only when the
node's role included outboundMta (crates/smtp/src/lib.rs), and the task
manager only when the role had some task type (spawn_task_manager).
After that nothing looked at the role again: a queue manager that was
running kept claiming and delivering, and one that wasn't never
started.
They now start on every node (outside recovery mode) and follow the
role live:
- Queue manager: before each scan it reads the role from the running
settings. Without outboundMta it claims nothing new; deliveries
already running finish and report back as usual, which releases
their locks. When the role comes back (a reload wakes the manager
with ReloadSettings, and it looks again every 30 s regardless) it
logs queue.started and scans the whole queue at once.
- Report scheduler: DMARC and TLS report events are handled only while
the role has outboundMta, as at boot; events arriving without it are
dropped, as they were on a node started without the role.
- Task manager: task_enabled already read the current role on every
scan. It now also runs on nodes whose role has no task type (the
scan returns at once until one is added), a job claimed before a
role change is handed back at once rather than run or held until
its lease lapses, and a settings reload wakes the manager so a role
that gained task types starts claiming them straight away.
Starting the queue manager on every node also drains the queue channel
on nodes without outboundMta. Upstream left that channel unread, so
each message queued there parked a refresh in it, and by the code,
queueing would block once 1024 had piled up (not reproduced here).
A role object edit reaches the nodes that name that role in
INBUXA_ROLE. Moving a node to another role still means changing its
environment, and so a restart. Listener changes in a role still need a
restart too (listeners bind at boot); this change is about tasks and
delivery.
cluster::live_roles::live_role_tests (new; PostgreSQL, two nodes over
one store):
1. A node started with outboundMta delivers and runs a TLS report
task; after its role loses outboundMta and the settings reload, a
new message isn't attempted and a new report task stays pending;
with the role back, both are taken up.
2. A node started with no task type at all gains outboundMta: a
waiting message is attempted and a report task runs.
On main the test fails at step 1 ("delivery attempted without
outboundMta"); with step 1 bypassed, step 2 fails (nothing picked the
message up in 20 s).
Cluster rehearsal 3: with PostgreSQL paused (docker pause, so its
kernel still answered TCP keepalives), requests on connections already
checked out hung until it came back, and /healthz/ready stayed 200
through the outage. #41 bounded getting a connection, not using one.
Client-side query limits (store::backend::query_timeout). Every
operation on a PostgreSQL or MySQL connection now runs under a time
limit. A server-side statement_timeout (or MySQL's MAX_EXECUTION_TIME,
which covers SELECTs only) can't do this: the server that would enforce
it is the one not answering. When an operation runs out, its connection
is closed instead of pooled, since a query may still be in flight on it
or a transaction open: deadpool's Object::take on PostgreSQL;
Conn::disconnect on MySQL, which marks the connection closed before it
sends anything, so the pool discards it even when the server never
answers.
- query, 2 minutes: reads, writes (the whole transaction with its
retries), blobs, SQL lookups, search queries and indexing. These take
milliseconds; two minutes leaves room for a large blob over a slow
link and still ends a hang.
- maintenance, 30 minutes: range deletes (account removal, purges),
unindexing, purge_store, and creating tables and indexes at startup,
which can legitimately run long in one statement. Their existing
chunked fallback for server-side statement timeouts is unchanged.
- iterate (exports, reindexing, maintenance scans) can run for hours,
so the query limit bounds each wait for the database (preparing, the
query starting, the next row) rather than the whole scan.
The limits are fixed, like the pool timeouts; the DataStore schema has
no field for them. Tests set them with Store::with_query_timeouts
(test_mode only).
Readiness. /healthz/ready answered 200 whenever a data store was
configured. It now reads one key from the data store with a 2 s limit
and reuses the answer for 2 s, so probes can't load the database;
while one probe runs, others get the last answer. The first failed
probe of an outage is logged. /healthz/live stays 200: restarting a
node doesn't bring its database back, and an orchestrator restarting on
failed liveness would restart every node at once. The container
HEALTHCHECK already uses /healthz/live.
Tests, store::pool_timeout (a proxy that stops forwarding while
keeping connections open plays the paused database):
- postgres_query_timeout, mysql_query_timeout (new): with four pooled
connections open, a read, a scan and a write each fail with "Query
timed out" 2.0 s after the pause (2 s test limit); once the proxy
forwards again the store answers. With the limits set to an hour
(upstream's behavior), the read was still waiting at the test's 20 s
limit.
- postgres_readiness (new, STORE=PostgreSql): a node's data store
goes through the proxy; /healthz/ready is 200, 503 about 4 s after
the pause while /healthz/live stays 200, and 200 again about 2 s
after it ends.
- postgres_pool_timeout, mysql_pool_timeout: pass as before.
store::store_tests (PostgreSql, MySql, including the MariaDB statement
timeout step) and store::task_locks (PostgreSql) pass;
store::search_tests (PostgreSql) fails at the same ordering assertion
(query.rs:684) as on main.
A three-node rehearsal on PostgreSQL saw searches take about 185 ms
with 80 to 260 pages in the full-text indexes' pending lists, 2 to 6 ms
right after gin_clean_pending_list() or VACUUM, then creep back up as
mail came in. The search tables' GIN indexes were created with the
default fastupdate=on: new entries wait in an unindexed pending list
that every search scans in full until VACUUM (or 4 MB of backlog)
merges it, and autovacuum only visits an insert-only table after
thousands of inserts.
The search GIN indexes are now created WITH (fastupdate = off), so an
insert pays its index update at once. The schema step runs at every
startup (create_search_tables, via SearchStore::create_indexes), so
indexes made before this change are switched there: when an index's
reloptions don't already turn fastupdate off, ALTER INDEX ... SET
(fastupdate = off) and one gin_clean_pending_list() merge its backlog.
The ALTER takes a SHARE UPDATE EXCLUSIVE lock, which blocks neither
reads nor writes; after the first startup the step is one catalog read
per index. A failure is logged and startup goes on (search still
works, only slower).
Per-table autovacuum settings for the search tables are left alone.
The pending list was the only reason the insert threshold mattered for
search; dead tuples and freezing are served by the defaults, and table
settings would override whatever tuning the DBA has done.
MySQL is unaffected: InnoDB FULLTEXT keeps new entries in an in-memory
cache that queries read directly, with no setting like fastupdate.
store::search_gin::postgres_gin_fastupdate (new, PostgreSQL) builds
the search schema in a schema of its own and checks pg_class.reloptions:
fastupdate=off on every GIN index of a fresh schema; then, with the
option reset to the default and 500 rows pending, one startup turns it
off everywhere and leaves no pending tuples (pgstatginindex); a second
startup changes nothing. On main it fails at the first check.
write_reload_target sent AllowedIp writes to the blocked-IP reload, but
that reload rebuilds only BlockedIps. Allowed IPs are parsed into the
core's security settings (Security::parse), which only a full reload
rebuilds, so an AllowedIp write reported x:settingsReload applied: true
while the change wasn't live until the next full reload.
AllowedIp now maps to the full reload, like the other settings objects;
BlockedIp keeps its targeted reload.
system::auto_reload::settings_reload_tests now creates an allowed IP
over JMAP and checks that is_ip_allowed sees it with no ReloadSettings,
and that destroying it takes it out again. On main it fails ("allowed
IP not in the running settings").
A 3-node rehearsal (PostgreSQL + NATS + Garage) found two ways a crash
leaves work stuck:
Pool hangs. The PostgreSQL pool (deadpool) was built with no timeouts,
so a request waited for a free connection, and for one to be opened or
recycled, for as long as it took: forever when the server stopped
answering. MySQL's pool (mysql_async) has no wait timeout at all.
- PostgreSQL: wait 30 s (or the store's timeout if longer), create the
store's timeout or 15 s (it bounds the whole handshake, where
tokio-postgres's connect_timeout covers only the TCP connect), recycle
10 s. The pool config is now always set, not only with
poolMaxConnections.
- MySQL: every connection is taken through MysqlStore::conn(), which
gives up after 30 s.
- Both: TCP keepalive after 60 s idle, so a server that vanished
without closing the connection is noticed in minutes rather than the
two-hour system default.
The DataStore schema has no pool timeout settings, so these are fixed
defaults; the store's own timeout bounds connecting on PostgreSQL.
Task locks. A task lock lasted an hour, so after a hard crash the dead
node's tasks waited up to an hour and five minutes. The lock is now a
five-minute lease: while this node runs a task, the task manager renews
its lock every third of the lifetime (InMemoryStore::renew_lock, a
compare-and-set on the store backends and SET XX EX on Redis, which
leaves a lock that already expired alone). A killed node's tasks run
elsewhere within about five minutes plus the claim recheck. A task this
node holds isn't handed to a worker again by the scan.
store::pool_timeout (new): a local listener that accepts connections
and never answers plays a hung server; a PostgreSQL store with a 2 s
timeout returns an error in about 4 s, and a MySQL store in 30 s.
Without the timeouts both wait for good. store::task_locks gains a
task held for 1.5 lock lifetimes: its lease is still held, and released
when the task ends.
A 3-node rehearsal found taskQueueProcessing didn't filter anything:
roles.task_manager only decided whether the task manager started, and
report, ACME, DKIM, DNS, calendar, thread-merge and restore tasks ran on
any node with a task manager (manager.rs returned true for them). A node
whose role left taskQueueProcessing off still ran them if it indexed or
did maintenance.
Every task type now answers to one ClusterTaskType (task_enabled):
- IndexDocument, UnindexDocument, IndexTrace: searchIndexing
- AccountMaintenance, TenantMaintenance, DestroyAccount:
accountMaintenance
- StoreMaintenance: storeMaintenance
- SpamFilterMaintenance: spamClassifierTraining
- DmarcReport, TlsReport: outboundMta. They build and send reports to
other domains (TLS reports can go straight to an HTTPS endpoint),
which is the outbound MTA's business.
- CalendarAlarmEmail, CalendarAlarmNotification, CalendarItipMessage,
MergeThreads, RestoreArchivedItem, AcmeRenewal, DkimManagement,
DnsManagement: taskQueueProcessing, the role for queue tasks with no
role of their own.
A node that may not run a task leaves it unclaimed (no lock), so a node
that may picks it up. The task manager also starts on a node whose only
task role is outboundMta, so reports still run there.
cluster::task_roles::task_role_tests (new, two task managers over one
PostgreSQL store): node A (taskQueueProcessing only) runs a DNS task and
leaves an unindex task and a TLS report pending; node B (searchIndexing
and outboundMta) comes up and runs those two; a DNS task scheduled next
stays pending on B and runs on A. On main node A runs the TLS report.
jcoffey-dev