After #37, address fields on PostgreSQL are split into words as the
built-in index splits them, but language text (subject, body,
attachments) still goes straight to PostgreSQL's parser, which keeps a
URL, host, path or file name as tokens of its own:
"https://x.example/shipping-support/" becomes a url, a host and a
url_path, "invoice-2024.pdf" a file. So TEXT/BODY "shipping" missed
messages where the word appears only inside a link, while RocksDB and
the other built-in backends found them: 8 messages across a handful
of searches in the rehearsal.
On insert, language text is now indexed as it was, followed by the
word parts of each token that holds a URL separator (/ . @ : ? = & # _
% + ~ \), split with SpaceTokenizer as keyword_terms() splits addresses.
The parts go through the same text search configuration as the rest of
the text, so they are stemmed like the words around them. Plain words,
words that only carry punctuation ("end.", "(see") and hyphenated words
(the parser already splits those) add nothing, so text without links
is indexed exactly as before. Each part is added once per document.
On sample mail, the text vector of a short order notice with three
links grows from 546 to 716 bytes, a newsletter with 25 tracking links
from 5586 to 6430, and a plain letter not at all.
On search, a query word written as a URL, host, file or hyphenated word
also matches as its word parts, ORed with the query as written, so
"shipping-support" or "invoice-2024.pdf" match the new parts and
documents indexed before this change still match as they did.
Existing messages keep their old vectors until they are reindexed (the
reindexAccounts task); new and reindexed messages match at once.
store::search_tests gains test_url_word_search: five bodies, 19 body
searches for words found only in a URL path, query string, host or
file name, the tokens as written, plain words and non-matches, with the
same expected ids on every backend. It passes on RocksDB, SQLite,
MySQL and PostgreSQL; on main PostgreSQL fails at the first ("shipping"
finds [3], not [0, 3]). On PostgreSQL the suite then stops at the
account sort assertion (query.rs:689) exactly as it does on main.
A cluster rehearsal sent ten x:<Object>/set requests at once and got
ten full reloads on every node. #39's coalescing only joined writes
that queued behind a running reload, but the requests reached the
server about 33 ms apart and a reload takes tens of milliseconds, so
none overlapped one.
A full reload after a registry write now waits for writes to settle:
75 ms after the last one, and at most 250 ms after the first it
covers, so a steady stream still reloads at least four times a
second. 75 ms is a little over twice the gap the rehearsal saw between
requests. A single write pays it once: in the tests a settings write
takes about 140 ms instead of 60. The reload runs in a task of its
own, so a request that goes away doesn't cancel it for the others.
Each write takes the result of the first reload that started after it
was stored (the gate keeps the last 64 results), so applied true or
false still describes the reload that covered that write.
The 33 ms gap was a queue on the server, not password hashing: Basic
credentials are cached per Authorization header, so they are checked
once. Every authenticated HTTP request counted itself against the
account's rate limit by incrementing one counter per account in the
in-memory store, so parallel requests from one account queued on that
key: a row lock on PostgreSQL (a few round trips to the database
each) and conflict retries with a 50-300 ms backoff on RocksDB. An
account with the unlimitedRequests permission (administrators, by
default) passes the rate and concurrency limits anyway, so its
requests are no longer counted. Ten parallel Core/echo calls as the
admin now finish in 1-4 ms; before, they finished one after another
over 20 ms on a local PostgreSQL and 300-450 ms on RocksDB. Other
accounts still count every request.
system::auto_reload::settings_reload_tests: ten concurrent writes now
take one reload (the gate counts them; at most two allowed), all are
applied: true and in the running settings, and a single write takes
exactly one reload. RocksDB and PostgreSQL, 1 reload in 141-196 ms.
With the old behavior (no wait, requests counted) the same writes
took 5 reloads; without the wait but with the rate fix, 2.
cluster::broadcast (3 nodes, PostgreSQL + NATS) and system::reload
still pass.
A cluster rehearsal moved a Log tracer to another directory: the write
was reported x:settingsReload applied:true, but the tracer kept writing
to the old file until a restart. Telemetry::update only refreshed each
running tracer's events, level and lossiness; a tracer's own settings
(path, prefix, rotation, format, endpoint, headers, ...) stayed as built.
Each tracer now carries a hash of the registry object it was built
from, less the fields that change in place. The reload compares it with
the running tracer's: unchanged ones are updated in place as before,
changed ones are started over, new ones started and removed ones
stopped. Only tracers this server started are removed; upstream removed
every subscriber not in the settings, which also cut off live-tracing
streams on each reload.
Starting over is a swap in the collector, so no event is lost or
written twice: a subscriber registered under a running one's id
replaces it between two collection passes. The old one's batch is sent
first (what its full channel can't take moves to the new one), and
dropping it closes its channel, so its task writes what is queued and
ends. Per tracer kind:
- Log: a tracer started over on the same files (rotation or format
changed) waits for the old one to finish, so lines don't interleave.
- Webhook: the task held a sender of its own channel for retries, so
it never ended; retries now use a weak sender, and pending events are
posted when the channel closes.
- OpenTelemetry: pending logs and spans are exported when the channel
closes instead of dropped, and a span that was open across the swap
is exported by the new tracer with the events it saw.
- Console and journal: nothing kept between batches.
- Trace history: built from the tracing store, which takes a restart,
so it is never started over.
No kind needs a restart, so x:settingsReload doesn't gain one.
system::tracer_reload::tracer_reload_tests (new): a Log tracer created
over JMAP writes to its directory; its path is changed over JMAP while
2000 numbered events are emitted; after the reload, events land in the
new file and not the old one, each numbered event is in exactly one of
the two files, and a destroyed tracer writes nothing. On main the new
file never appears.
The report scheduler dropped DMARC and TLS events on a node whose role
lacks outboundMta (upstream never started it there, so they sat in a
channel nobody read). Mail received on a front node therefore never
reached an aggregate report, which is meant to cover all of a domain's
inbound mail, whichever node received it. In rehearsal, five messages
received on port 25 on a front node were missing from every report.
- The report scheduler records on every node. Recording is a store write
the nodes already share, so it needs nothing from the outbound MTA.
Building and sending a report (the DmarcReport and TlsReport tasks) stay
with outboundMta nodes, as the task manager already enforces.
- More nodes now append to one report at once. Appends already guard the
report's versioned primary key; a write that loses now retries up to ten
times after a short random pause, not three times at once.
- The node sending a report deletes it only if it is unchanged since it
was read, and reads it again otherwise, so a record another node appends
meanwhile goes out with the report instead of being deleted unsent.
Test: cluster::front_reports (PostgreSQL and MySQL). A front node's
results appear in the report the MTA node sends, alongside eight appended
at once from both nodes, and the front node never runs the report task.
It fails on main: the front node's results are never recorded.
Setting deliverAt on an internal DMARC or TLS report wrote the new task
queue row with the report's object type (0x21, 0x6e) instead of the task
type (7, 8), and left the task row at its old due. The task manager's scan
failed on that row with store.data-corruption ("Failed to iterate over task
queue"), and because the error ended the whole scan, every task due after
the row stopped running on every node.
- reschedule_ops writes the new queue row through schedule_task_with_id, so
it carries the task type and the task row gets the new due. It removes
the row the task is actually queued under (the task's due, which differs
from deliverAt once the task has been retried) and any row an earlier
reschedule left at deliverAt.
- x:DmarcInternalReport/set and x:TlsInternalReport/set lock the report's
task while they move it, as x:Task/set does, refuse while the report is
being sent, release the locks however the request ends, and wake the task
manager.
- The task manager logs a queue row it can't read (id, due, key, value) and
skips it instead of ending the scan. It then repairs the row from its task:
the row is rewritten with the task's type, and a row with no task behind
it is removed. A row holding a report's object type for a report task is
what the old reschedule wrote: the task is moved to that row's time, as
the reschedule intended, and its old queue row is removed. Stores that
already hold such a row recover on their own once it comes due.
- x:Task/query with a type filter skips an unreadable row instead of
failing.
Test: smtp::reporting::reschedule (RocksDB and PostgreSQL). It fails on
main: x:Task/get shows the old due, and with that check removed, neither
report nor a later task ever runs.
In cluster rehearsal 3, turning outboundMta off on node1's role was
reported applied (x:settingsReload applied: true), yet node1 kept
delivering mail, a report message included, until it was restarted.
The queue and report managers were started at boot only when the
node's role included outboundMta (crates/smtp/src/lib.rs), and the task
manager only when the role had some task type (spawn_task_manager).
After that nothing looked at the role again: a queue manager that was
running kept claiming and delivering, and one that wasn't never
started.
They now start on every node (outside recovery mode) and follow the
role live:
- Queue manager: before each scan it reads the role from the running
settings. Without outboundMta it claims nothing new; deliveries
already running finish and report back as usual, which releases
their locks. When the role comes back (a reload wakes the manager
with ReloadSettings, and it looks again every 30 s regardless) it
logs queue.started and scans the whole queue at once.
- Report scheduler: DMARC and TLS report events are handled only while
the role has outboundMta, as at boot; events arriving without it are
dropped, as they were on a node started without the role.
- Task manager: task_enabled already read the current role on every
scan. It now also runs on nodes whose role has no task type (the
scan returns at once until one is added), a job claimed before a
role change is handed back at once rather than run or held until
its lease lapses, and a settings reload wakes the manager so a role
that gained task types starts claiming them straight away.
Starting the queue manager on every node also drains the queue channel
on nodes without outboundMta. Upstream left that channel unread, so
each message queued there parked a refresh in it, and by the code,
queueing would block once 1024 had piled up (not reproduced here).
A role object edit reaches the nodes that name that role in
INBUXA_ROLE. Moving a node to another role still means changing its
environment, and so a restart. Listener changes in a role still need a
restart too (listeners bind at boot); this change is about tasks and
delivery.
cluster::live_roles::live_role_tests (new; PostgreSQL, two nodes over
one store):
1. A node started with outboundMta delivers and runs a TLS report
task; after its role loses outboundMta and the settings reload, a
new message isn't attempted and a new report task stays pending;
with the role back, both are taken up.
2. A node started with no task type at all gains outboundMta: a
waiting message is attempted and a report task runs.
On main the test fails at step 1 ("delivery attempted without
outboundMta"); with step 1 bypassed, step 2 fails (nothing picked the
message up in 20 s).
Cluster rehearsal 3: with PostgreSQL paused (docker pause, so its
kernel still answered TCP keepalives), requests on connections already
checked out hung until it came back, and /healthz/ready stayed 200
through the outage. #41 bounded getting a connection, not using one.
Client-side query limits (store::backend::query_timeout). Every
operation on a PostgreSQL or MySQL connection now runs under a time
limit. A server-side statement_timeout (or MySQL's MAX_EXECUTION_TIME,
which covers SELECTs only) can't do this: the server that would enforce
it is the one not answering. When an operation runs out, its connection
is closed instead of pooled, since a query may still be in flight on it
or a transaction open: deadpool's Object::take on PostgreSQL;
Conn::disconnect on MySQL, which marks the connection closed before it
sends anything, so the pool discards it even when the server never
answers.
- query, 2 minutes: reads, writes (the whole transaction with its
retries), blobs, SQL lookups, search queries and indexing. These take
milliseconds; two minutes leaves room for a large blob over a slow
link and still ends a hang.
- maintenance, 30 minutes: range deletes (account removal, purges),
unindexing, purge_store, and creating tables and indexes at startup,
which can legitimately run long in one statement. Their existing
chunked fallback for server-side statement timeouts is unchanged.
- iterate (exports, reindexing, maintenance scans) can run for hours,
so the query limit bounds each wait for the database (preparing, the
query starting, the next row) rather than the whole scan.
The limits are fixed, like the pool timeouts; the DataStore schema has
no field for them. Tests set them with Store::with_query_timeouts
(test_mode only).
Readiness. /healthz/ready answered 200 whenever a data store was
configured. It now reads one key from the data store with a 2 s limit
and reuses the answer for 2 s, so probes can't load the database;
while one probe runs, others get the last answer. The first failed
probe of an outage is logged. /healthz/live stays 200: restarting a
node doesn't bring its database back, and an orchestrator restarting on
failed liveness would restart every node at once. The container
HEALTHCHECK already uses /healthz/live.
Tests, store::pool_timeout (a proxy that stops forwarding while
keeping connections open plays the paused database):
- postgres_query_timeout, mysql_query_timeout (new): with four pooled
connections open, a read, a scan and a write each fail with "Query
timed out" 2.0 s after the pause (2 s test limit); once the proxy
forwards again the store answers. With the limits set to an hour
(upstream's behavior), the read was still waiting at the test's 20 s
limit.
- postgres_readiness (new, STORE=PostgreSql): a node's data store
goes through the proxy; /healthz/ready is 200, 503 about 4 s after
the pause while /healthz/live stays 200, and 200 again about 2 s
after it ends.
- postgres_pool_timeout, mysql_pool_timeout: pass as before.
store::store_tests (PostgreSql, MySql, including the MariaDB statement
timeout step) and store::task_locks (PostgreSql) pass;
store::search_tests (PostgreSql) fails at the same ordering assertion
(query.rs:684) as on main.
A three-node rehearsal on PostgreSQL saw searches take about 185 ms
with 80 to 260 pages in the full-text indexes' pending lists, 2 to 6 ms
right after gin_clean_pending_list() or VACUUM, then creep back up as
mail came in. The search tables' GIN indexes were created with the
default fastupdate=on: new entries wait in an unindexed pending list
that every search scans in full until VACUUM (or 4 MB of backlog)
merges it, and autovacuum only visits an insert-only table after
thousands of inserts.
The search GIN indexes are now created WITH (fastupdate = off), so an
insert pays its index update at once. The schema step runs at every
startup (create_search_tables, via SearchStore::create_indexes), so
indexes made before this change are switched there: when an index's
reloptions don't already turn fastupdate off, ALTER INDEX ... SET
(fastupdate = off) and one gin_clean_pending_list() merge its backlog.
The ALTER takes a SHARE UPDATE EXCLUSIVE lock, which blocks neither
reads nor writes; after the first startup the step is one catalog read
per index. A failure is logged and startup goes on (search still
works, only slower).
Per-table autovacuum settings for the search tables are left alone.
The pending list was the only reason the insert threshold mattered for
search; dead tuples and freezing are served by the defaults, and table
settings would override whatever tuning the DBA has done.
MySQL is unaffected: InnoDB FULLTEXT keeps new entries in an in-memory
cache that queries read directly, with no setting like fastupdate.
store::search_gin::postgres_gin_fastupdate (new, PostgreSQL) builds
the search schema in a schema of its own and checks pg_class.reloptions:
fastupdate=off on every GIN index of a fresh schema; then, with the
option reset to the default and 500 rows pending, one startup turns it
off everywhere and leaves no pending tuples (pgstatginindex); a second
startup changes nothing. On main it fails at the first check.
write_reload_target sent AllowedIp writes to the blocked-IP reload, but
that reload rebuilds only BlockedIps. Allowed IPs are parsed into the
core's security settings (Security::parse), which only a full reload
rebuilds, so an AllowedIp write reported x:settingsReload applied: true
while the change wasn't live until the next full reload.
AllowedIp now maps to the full reload, like the other settings objects;
BlockedIp keeps its targeted reload.
system::auto_reload::settings_reload_tests now creates an allowed IP
over JMAP and checks that is_ip_allowed sees it with no ReloadSettings,
and that destroying it takes it out again. On main it fails ("allowed
IP not in the running settings").
A 3-node rehearsal (PostgreSQL + NATS + Garage) found two ways a crash
leaves work stuck:
Pool hangs. The PostgreSQL pool (deadpool) was built with no timeouts,
so a request waited for a free connection, and for one to be opened or
recycled, for as long as it took: forever when the server stopped
answering. MySQL's pool (mysql_async) has no wait timeout at all.
- PostgreSQL: wait 30 s (or the store's timeout if longer), create the
store's timeout or 15 s (it bounds the whole handshake, where
tokio-postgres's connect_timeout covers only the TCP connect), recycle
10 s. The pool config is now always set, not only with
poolMaxConnections.
- MySQL: every connection is taken through MysqlStore::conn(), which
gives up after 30 s.
- Both: TCP keepalive after 60 s idle, so a server that vanished
without closing the connection is noticed in minutes rather than the
two-hour system default.
The DataStore schema has no pool timeout settings, so these are fixed
defaults; the store's own timeout bounds connecting on PostgreSQL.
Task locks. A task lock lasted an hour, so after a hard crash the dead
node's tasks waited up to an hour and five minutes. The lock is now a
five-minute lease: while this node runs a task, the task manager renews
its lock every third of the lifetime (InMemoryStore::renew_lock, a
compare-and-set on the store backends and SET XX EX on Redis, which
leaves a lock that already expired alone). A killed node's tasks run
elsewhere within about five minutes plus the claim recheck. A task this
node holds isn't handed to a worker again by the scan.
store::pool_timeout (new): a local listener that accepts connections
and never answers plays a hung server; a PostgreSQL store with a 2 s
timeout returns an error in about 4 s, and a MySQL store in 30 s.
Without the timeouts both wait for good. store::task_locks gains a
task held for 1.5 lock lifetimes: its lease is still held, and released
when the task ends.
A 3-node rehearsal found taskQueueProcessing didn't filter anything:
roles.task_manager only decided whether the task manager started, and
report, ACME, DKIM, DNS, calendar, thread-merge and restore tasks ran on
any node with a task manager (manager.rs returned true for them). A node
whose role left taskQueueProcessing off still ran them if it indexed or
did maintenance.
Every task type now answers to one ClusterTaskType (task_enabled):
- IndexDocument, UnindexDocument, IndexTrace: searchIndexing
- AccountMaintenance, TenantMaintenance, DestroyAccount:
accountMaintenance
- StoreMaintenance: storeMaintenance
- SpamFilterMaintenance: spamClassifierTraining
- DmarcReport, TlsReport: outboundMta. They build and send reports to
other domains (TLS reports can go straight to an HTTPS endpoint),
which is the outbound MTA's business.
- CalendarAlarmEmail, CalendarAlarmNotification, CalendarItipMessage,
MergeThreads, RestoreArchivedItem, AcmeRenewal, DkimManagement,
DnsManagement: taskQueueProcessing, the role for queue tasks with no
role of their own.
A node that may not run a task leaves it unclaimed (no lock), so a node
that may picks it up. The task manager also starts on a node whose only
task role is outboundMta, so reports still run there.
cluster::task_roles::task_role_tests (new, two task managers over one
PostgreSQL store): node A (taskQueueProcessing only) runs a DNS task and
leaves an unindex task and a TLS report pending; node B (searchIndexing
and outboundMta) comes up and runs those two; a DNS task scheduled next
stays pending on B and runs on A. On main node A runs the TLS report.
A 3-node rehearsal found that saving an MtaDeliverySchedule left it
unknown to the queue ("Queue strategy not found") until someone ran
x:Action ReloadSettings; only Directory and Authentication writes
reloaded (DIR-17). The admin UI has to remember a separate reload after
every save, and a script or API client that doesn't gets a server
running stale settings.
x:<Object>/set now reloads the running settings when it created,
updated or destroyed an object they are built from, and broadcasts the
same RegistryChange::Reload over the coordinator as ReloadSettings, so
every node applies it:
- Settings objects (MTA, spam filter, listeners, tracers, Sieve system
scripts, cluster roles, directories, ...: the object types the core,
telemetry, listener and directory builders read) get a full reload.
- Certificates, lookup stores and blocked/allowed IPs get their own
targeted reloads.
- Accounts, domains, roles and other data read as needed, stores (they
take a restart) and applications (their own reload action) get none.
Full reloads are coalesced: a write waits for a reload that started
after it was stored and joins one if it can, so a burst of writes, or
a request with many objects, costs one or two reloads, not one each.
The write itself is never undone. When the reload is refused (build
errors in objects that were working, the rule from the previous
commit), the set response says so in a new x:settingsReload field,
{"applied": false, "description": "Saved, but the running settings
were not reloaded. <object>: <error>"}; {"applied": true} otherwise.
The field is absent when the write needs no reload. The description
helper is shared with ReloadSettings' refusal.
Each reload sends the queue a ReloadSettings event, so the SMTP test
harness's read_event, try_read_event and assert_no_events now pass over
those; expect_reload_settings still waits for one.
system::auto_reload::settings_reload_tests (new): an MtaVirtualQueue
and an MtaDeliverySchedule created over JMAP are in the running
settings with no ReloadSettings, and gone once destroyed; eight
concurrent creates all land; a write whose reload fails is stored and
reported applied: false with the error; a domain write carries no
x:settingsReload. On main the new schedule is missing. The cluster
broadcast test (three nodes, PostgreSQL + NATS) now checks that every
node has a schedule created on node 0 without a reload.
A 3-node rehearsal found every settings reload refused, cluster-wide,
because one node couldn't resolve the Pyzor server:
- PyzorConfig::parse resolved the host while building the settings and
made a failed lookup a build error. It now keeps the host and port and
resolves when a message is checked (an IP address is used as is, a
name is reused for five minutes, the lookup counts against the Pyzor
timeout). A failure there is a Pyzor error for that message.
- A milter's hostname was resolved the same way, with a blocking
to_socket_addrs in async code. An IP address is kept; a name is now
resolved on each connection.
Other build-time I/O is already non-fatal: directories that can't
connect become unavailable with a warning (DIR-21), and the AI model
locality check only warns.
reload_registry swapped the core only when the whole build was free of
errors, while boot runs with whatever built. One failing object thus
refused every later reload, and the running settings went stale. Now a
reload is refused only for errors in objects that built when the
running settings were built (at boot or by the last applied reload):
applying it would lose those. Objects that already failed then are
missing from the running settings anyway, as at boot, so their errors
are logged and returned as known_errors but don't hold the reload back.
Refusing on new errors keeps a bad edit from taking a working object
out of service; the admin gets the error instead.
ReloadSettings now says "Settings were not reloaded." and names the
object and its error ("Tracer with id ...: Only one console tracer is
allowed"), with a count of any further errors. A refused reload after a
directory change logs its errors too.
system::reload::reload_tests (new): with Pyzor enabled on an
unresolvable host, ReloadSettings succeeds (on main it fails with
"Invalid address: failed to lookup address information"); an IP host
needs no lookup; a new build error refuses the reload, names the object
and leaves the running settings unchanged; the same error, once known
from the running settings' build, no longer blocks; once fixed, a new
error there blocks again. smtp::inbound::milter's session test now
names its milter "localhost", so the connect-time lookup is exercised.
A 3-node PostgreSQL rehearsal found IMAP SEARCH FROM "noreply" matched
0-2 messages where RocksDB matched 23 of 930. The message indexer hands
each address and display name of From/To/Cc/Bcc to the search store as
keyword text (Language::None). The built-in index splits keyword text
into lowercase runs of alphanumerics, so an address is found by its full
form, its local part, its domain or a display-name word. The SQL
backends didn't:
- PostgreSQL's text parser keeps "[email protected]" as one email
token (host names and URLs likewise), so neither "noreply" nor
"amazon.com" ever matched it. Keyword text is now split the same way
as the built-in index (SpaceTokenizer) before to_tsvector on insert
and before plainto_tsquery/phraseto_tsquery on search, still under
the 'simple' configuration, so the GIN index keeps serving the query.
The sort columns keep the raw text.
- MySQL's FULLTEXT parser already splits on punctuation, but InnoDB
never indexes its stopwords ("com", "de", "www", ...) or words under
innodb_ft_min_token_size (3), and a required +word it hasn't indexed
matches no row. So "amazon.com", "[email protected]" or "jane doe" found
nothing. Those words are now matched with a word-boundary REGEXP on
the rows the indexed words select. In language text (bodies,
subjects) they are dropped when other words remain, and only checked
when nothing else is left, so "the invoice" no longer finds nothing
either.
Existing PostgreSQL search indexes hold the old single-token vectors and
need a reindex (the reindexAccounts task) before address searches find
old messages. MySQL needs none: only the query changed.
store::search_tests gains test_address_search: five messages, 28
FROM/TO/CC/BCC searches by full address, local part, domain, domain
labels, display name and hyphenated local part, plus a TEXT-style OR,
with the same expected ids on every backend. It passes on RocksDB,
SQLite, PostgreSQL and MySQL; on main it fails on PostgreSQL (From
"noreply") and MySQL (From "[email protected]").
A node that started while NATS was down never got a coordinator. The
connect failed at boot, bootstrap recorded a build error and the node ran
with Coordinator::None until restarted. It had no broadcast subscriber
or publisher, so cross-node push and cache invalidation to it stayed
broken, and its healthcheck said nothing about it. Losing NATS after
startup was silent too.
- The NATS client now connects in the background
(retry_on_initial_connect): startup never waits on NATS or fails over
it, the node gets its coordinator, subscriber and publisher at once,
and the client keeps trying (async-nats's backoff, at most 4 s apart)
until NATS answers. Subscriptions made meanwhile start delivering when
it does. A configured maxReconnects still ends the attempts.
- Three new events report the connection: cluster.coordinator-connected
(info), cluster.coordinator-disconnected (warn: lost, closed, gave up,
or not connected within the connection timeout at startup) and
cluster.coordinator-error (warn: a failed attempt, reported once per
outage rather than every retry, and server errors, slow consumers and
lame duck mode). They are in the packaged schema, ids 644 to 646.
- GET /healthz/cluster reports the coordinator: 200
{"coordinator":"connected"}, 503 {"coordinator":"disconnected"}, or
200 with "none" (no coordinator) or "unknown" (a backend that doesn't
track its connection). /healthz/live and /healthz/ready are unchanged
on purpose: a node without its coordinator still serves mail, and
failing those would have orchestrators restart, or pull out of
service, every node at once whenever NATS is down.
Only NATS connects lazily; the other coordinator backends still fail at
boot as before.
cluster::coordinator::coordinator_reconnect_tests starts a node against a
NATS port with nothing behind it, checks it boots with a coordinator and
reports it disconnected, subscribes, then starts NATS on that port: the
node connects on its own and the subscription receives a message from a
second client. Stopping and restarting NATS shows disconnected, then
connected, and the same subscription keeps working.
A cluster rehearsal (PostgreSQL + NATS) left index tasks pending well
past the one-hour task lock after the node that claimed them was stopped
or killed. The exact cause there isn't confirmed; this closes every path
found in the task manager that stretches a takeover past the lock, or
keeps a task claimed without running it:
- A graceful stop never released the locks it held, so every task the
node had claimed stayed blocked for an hour. The server now tracks the
locks it holds (common::ipc::TaskLocks) and, once the shutdown signal
arrives, stops claiming and releases them before exiting.
- A node that failed to claim a task (another node held it) set its own
local hold for a full lock lifetime from that scan. If the holder
claimed it just after the scan began, or ran on a clock ahead, that
hold ran out a moment before the lock did and was set for another
hour: two hours in all. Such claims are now tried again every five
minutes (a twelfth of the lock lifetime), and the task manager wakes
up for them: before, a node without a coordinator could sleep up to
five minutes past the recheck, or until something else woke it.
- A worker that panicked took its task type down on that node for good,
while the scan kept claiming that type's tasks and failing to hand them
over, re-taking each lock as it expired and so starving every other
node of them. Each batch now runs on a task of its own; a panic is
logged, the batch's locks are released and the worker carries on. A
failed hand-over releases the lock too.
- A claimed task the worker couldn't read, or found gone, kept its lock
for the hour. It is released.
- An IndexDocument task for a file (not indexed) returned no result,
which shifted every later result in the batch onto the wrong task in
update_tasks. It returns Ignored. Nothing queues such a task today.
The lock lifetime stays one hour; it now lives per server so the tests
can shorten it.
store::task_locks::task_lock_tests plays a second node by writing its
locks straight into the in-memory store: tasks it claimed and abandoned
run here once its locks expire, including locks that outlive this node's
view of them, and a graceful stop hands this node's locks back at once
and claims nothing more. It passes on RocksDB, SQLite and PostgreSQL.
With the old recheck it fails.
The trace index task wrote the event type (its name) and the queue id as
text, but the tracing search index types both as integers on every
backend: BIGINT on PostgreSQL and MySQL, long on Elasticsearch. On
PostgreSQL every batch holding a trace document failed with "cannot
convert between the Rust type String and the Postgres type int8", and
since a batch writes trace and email documents together, email indexing
stalled behind it.
The document is now built by trace_search_document(), which writes:
- the event type as the opening event's numeric id, the event
x:Trace/query's event filter already matches on;
- the queue id as an integer, the first one the trace names;
- every queue id into the keywords as well, since the column holds one
value and an SMTP session can queue several messages.
index_keyword() replaced the field on every call, so before this only the
last event type and queue id survived anyway.
x:Trace/query's queueId filter parses the id (a string, or now a number)
and matches the column or the keywords, so a session is found by any of
its queue ids on every backend. The monitoring spec says what is indexed.
Traces indexed before this on the built-in index keep their text values;
the reindexTelemetry maintenance task rebuilds them.
Tests: the search store suite builds trace documents with the index
task's code, indexes them and finds them by queue id, event type and
keyword (Sqlite, PostgreSQL, MySQL); the monitoring suite finds a real
trace by queueId through x:Trace/query.
--export skipped three things, so a move from one database to another
(RocksDB to PostgreSQL, say) lost them without a word:
- archived items (subspace j), the records behind undelete;
- spam training samples (subspace w);
- the trained spam classifier and its trainer state, blobs stored under
fixed names that no blob link points at, so the walk over links never
reached them.
j and w now travel with the registry family, where their indexes and id
counters already were, so EXPORT_TYPES=registry keeps them consistent.
The two named blobs travel with the blob family. The file format is
unchanged and import reads any subspace it is given, so an export made
by an older binary still imports.
The full-text index (subspace z) stays out, on purpose. It belongs to one
search backend: PostgreSQL and MySQL index into their own tables and have
no z table at all, and external engines keep the index themselves. So
--import now returns the subspaces it wrote, and boot queues the
reindexAccounts and reindexTelemetry store maintenance tasks, the same
ones an administrator can queue by hand, to rebuild the index for
whichever search store the server runs with once it starts.
The round trip also turned up a loss in import itself: the SQL stores
add a negative amount with an UPDATE, which does nothing to a row that
isn't there yet, so every negative counter or quota vanished on import
into PostgreSQL, MySQL or SQLite. Import now creates the row first.
The in-memory subspaces (m, y) stay out: rate limits, locks, greylisting,
ACME challenge tokens and OAuth codes, all short-lived. Issued
certificates are registry objects and travel.
The store test now writes archived items, spam samples, directory
entries, the fork's own subspace and the named blobs, checks they come
back in place, then imports the same export into a fresh store of the
other local backend (RocksDB to SQLite, or SQLite to RocksDB), compares
it key for key and counter for counter, and checks the queued reindex.
It fails on the old export code ("Subspace j was not exported").
--help now says what an export holds.
The server fetched upstream's latest published rules from GitHub at run
time: a version nobody here tested, code-like expressions from an account
we don't control, and the upstream name as a default in the admin form.
The published rules of spam-filter v3.0.2 are now embedded
(resources/spam-filter/, MIT, in THIRD-PARTY.md) and used whenever no other
source is configured. An empty setting and upstream's old default both mean
the bundled rules, so existing installs switch without a settings change;
the URL stays an operator override (https:// or file://). The schema default
is dropped and its description says what empty means, and the strip's
rename pass does the same to each import.
Rules load on first boot as before, and again whenever the bundled version
differs from the last one loaded, which only adds missing rules and tags.
That brings the AI classifier's LLM_* scores to installs that predate them:
production has none today.
upstream-watch now also opens an issue when spam-filter publishes a newer
release; resources/spam-filter/README.md says how to take it.
The antispam test now runs on the bundled rules, the path production
takes; SPAM_RULES_URL tests another set. Unit tests cover the URL handling
and that the bundled rules parse and score the AI tags as the AI spec says.
The server sets a deferred recipient's next retry from its clock when the
attempt defers, in whole seconds. The test subtracted its own clock taken
when the loop next saw the message, after saving and reporting, so
whenever that lag crossed a second boundary the 2 s retry measured 1 s
and the test failed. Under load, after the other SMTP tests, that was
most runs.
It now measures from when the test started the attempt, which the
server's deferral can only follow, by under a second: each retry is its
interval or one more. Each position is still checked against its own
interval, so a wrong schedule still fails.
It listens on HTTP 19048, as the dkim2 DSN and report tests do. Those are
marked serial, this wasn't, so when the SMTP tests ran together (as
upstream's CI runs them) it could start beside them and requests reached
whichever server had the port: missing JMAP creates here, and 'You must
authenticate first' in dkim2_dsn_is_signed.
It failed everywhere but upstream's machines, for two reasons:
- The spam rules, which carry every score, came from a path on an
upstream developer's own disk. Without SPAM_RULES_URL none loaded, every
score was 0.00 and the combined case came out ham instead of spam at
13.70. The published rules of spam-filter v3.0.2 are now pinned beside
the test cases (Apache-2.0 or MIT, taken as MIT; in THIRD-PARTY.md).
SPAM_RULES_URL still overrides.
- The first combined case expects a Pyzor hit, and its digest (that of an
empty body) wasn't among the three the test mode answers, so it went to
a public Pyzor server: it failed offline and would drift with that
server's counts. Test mode now answers every digest from a fixed table,
with the empty body's added, and never reaches the network.
The test passes online and offline, alone and with the rest of the SMTP
tests. queue_retry, unrelated, still fails when it runs after the others
in one process, though it passes alone every time.
The name is inbuxa, lowercase, like the wordmark; INBUXA reads as an
acronym. The admin and webmail already changed. Here that's everything
the server shows people: the brand macro behind the protocol greetings,
the HTTP and SCIM realms, the startup banner and the calendar and contact
PRODID; the first-party OAuth client descriptions; the legacy-protocol
refusals; the default calendar and address book names and the SMTP
greeting default, in the code and the schema served to the admin
(checksum regenerated); startup and shutdown events; the User-Agent;
the sign-in and RSVP pages; the service units; the OpenAPI realm; the
crate descriptions and the README, where it's set in bold.
Identifiers that are uppercase for their own reasons stay: INBUXA_*
settings, SUBSPACE_INBUXA. So do code comments and the AGPL 5(a) notice
lines.
Tests follow: the IMAP ID name, the default collection names, the PRODID
in the iTIP fixtures and the CalDAV free-busy expectations, and the e2e
legacy-protocol refusals. The webdav, imap and jmap suites pass, so do
the unit tests of every crate touched, and 73 of 75 SMTP tests; of the
other two, antispam fails on main too, and queue_retry is a timing flake
that passes on its own.
Everything clients, users and operators meet now carries the fork's name,
with no aliases (SPEC.md §2.4, changed here from "protocol identifiers
stay"):
- JMAP: upstream's registry capability is urn:inbuxa:jmap:registry, beside
the fork's own urn:inbuxa:jmap.
- WebDAV lock and sync tokens are urn:inbuxa:dav*; clients resync once.
- Sieve: vnd.inbuxa.while and vnd.inbuxa.expressions. sieve-rs spells these
into its compiler, so it's vendored (vendor/sieve-rs, 0.7.3) and patched in;
a unit test fails if Cargo.lock ever moves past the vendored copy. The
trusted runtime now names itself too, rather than answering sieve-rs's
default.
- The web interface's OAuth client is inbuxa-webui. On every start the old
stalwart-webui client is removed and any application naming it is moved
over.
- The spam filter's blobs are INBUXA_SPAM_*; every start moves any left
under the old keys, so a trained model survives.
- SQL stores and log files default to inbuxa, in the code and in the
schema served to the admin (checksum regenerated).
- Settings are INBUXA_* only. A STALWART_* variable that's set where its
INBUXA_* one isn't stops the server at startup, naming it.
- The version-upgrade messages link docs.inbuxa.org's migration page, and
the OpenAPI description, smtp crate metadata and web-push test fixtures
lose the name.
Kept on purpose, allowlisted with reasons: the OAuth key-derivation
contexts (renaming them would end every session and invalidate every
sealed client id) and the hashed application prefix.
Also fixes a latent start-up failure: ensure_client updated an existing
first-party client with a revision of 0, which the registry's assertion
never matches, so adding a redirect URI or changing the webmail secret
failed start-up. And the principal session test now expects
legacyProtocols (C-1, added 2026-09-21), which it had missed.
Tested: the server builds without warnings; common's 106 unit tests,
including the vendoring check; a new integration test for the two
start-up migrations; and the webdav, jmap, imap and SMTP Sieve suites.
tests/src/directory/issuer.rs, new in v0.16.23, tests routing a bearer token
to a directory by its issuer. That routing is Enterprise-only upstream (the
body of get_directory_for_issuer), and the fork doesn't build it: a token
naming no address gets the server default (DIR-2). The test also calls a
helper from upstream's Enterprise-only OIDC test, so it can't compile here.
mta.rs imported types::id::Id for code inside an Enterprise snippet; the
stripped tree leaves it unused, upstream's as well as ours.
Five conflicts, resolved:
- crates/common/src/auth/authentication.rs: upstream's get_directory_for_token
and JwtClaims replace extract_jwt_domain; the per-domain directory code
(DIR-1, DIR-5 to DIR-7) is kept, and the token lookup routes through it.
The release's one new Enterprise snippet was the body of
get_directory_for_issuer, which stays returning None: a token naming no
address gets the server default, as DIR-2 specifies and as v0.16.22 did.
- crates/common/src/manager/application.rs: upstream's rewrite of the tests,
with the temp directory names renamed again, and the 5(a) notice the
name-purge change should have added.
- crates/common/src/network/mta.rs: both sides' imports.
- crates/main/Cargo.toml: the AGPL-only license kept, version 0.16.23.
- Cargo.lock: upstream's, with the fork's crates added by Cargo.
The AGPL asks a modified version to carry prominent notices saying it was
modified, and giving a date. Publishing the source is the conveyance that
asks for it, so it wants doing before the repository is public rather than
at the release.
Every upstream file the fork changed now says so in its header, beneath the
notice it came with: 164 files, found by diffing against the upstream
snapshot branch rather than by guessing, so the list is what actually
differs. Files the fork wrote itself already carry their own copyright and
need nothing. Upstream's notices are untouched, which its licence requires
and which was already true.
The README says the same thing in prose, since the obligation is on the
work as a whole and not only its Rust files.
Builds unchanged: the server and the test binary both compile.
The run against INBUXA's store hung printing "Waiting for pending task
AcmeRenewal(...)": the copy carries that server's task queue, and a renewal
due in 2026-11 will not come due while a test watches it.
Under NO_INSERT the wait now skips tasks that aren't due and ones that have
permanently failed, which leaves the tasks the test itself caused — a
restore in undelete_compat comes due at once — and gives up after a minute
with the offending task printed. A test that was really waiting on its own
work now fails on its assertion, which says more than a spinner.
Ordinary runs are untouched: system_tests, which waits on tasks throughout,
still passes in 135s.
The first run against INBUXA's store failed all eight tests identically,
before reading a single record: the copy carries that server's listeners on
25, 443, 465, 587, 110, 143, 993 and 995, and nothing in a test run is
root, so each one failed with "Permission denied (os error 13)".
The builder now remembers the listeners it adds, and under NO_INSERT drops
build errors for any it didn't. Every other error still stands, including a
bind failing on one of its own, so this can't hide the case where the
harness's own port is taken.
The copy isn't edited for this: its listeners are simply not what a compat
run needs, and it reaches the server over the compat- ones instead. Checked
that a NO_INSERT run still boots and that scim_tests, which takes the
ordinary path, still passes.
Rehearsed the run against a real RocksDB store, and it died at startup
before checking anything: the harness inserts listeners of its own, the
registry keys them by name, and a real server already has a "jmap" and an
"imap". The message was "Primary key conflict on property name with
existing object NetworkListener", which says nothing about what to do.
Under NO_INSERT the harness now calls its listeners compat-jmap and so on,
and the same run gets through to the test's own checks.
run-compat.sh copies the store for each test and removes the copy after,
because several of these write to what they open: monitoring_compat purges
the history it reads and undelete_compat restores what it finds. The source
stays untouched, which matters when it is the only copy of a production
store anyone took that day.
INBUXA runs RocksDB, so a copy is a directory copy. The SQL backends would
need more than this: the harness builds its own container and connects to
fixed local credentials, so it cannot open a dump in place.
None of the eight had ever executed, so all eight ran against an empty store
with synthetic inputs. The plumbing works: the documented JSON shapes parse,
and NO_INSERT stops each one before the harness touches the store, which a
sentinel file in each store directory confirmed — it survived every run,
including the one launched without NO_INSERT.
Two things the runbook got wrong, both of which would have cost a day on the
day the copy exists:
- TMPDIR is the copy's parent, not the copy. The harness opens
$TMPDIR/<test name>, so a TMPDIR pointing at the copy gets an empty store
created beside it and the test calls INBUXA's data missing.
- masked_email_compat and undelete_compat need INBUXA_COMPAT_MASKS and
INBUXA_COMPAT_ARCHIVED, which only the tests' doc comments mentioned.
Every run ended on a 401 raised as "Missing list in response", which reads
as INBUXA's data being wrong when the login is what's wrong. Each test now
authenticates once first and names the variable that failed.
`allowScimProvisioning` is cached with the domain as DOMAIN_FLAG_SCIM, but
it wasn't among the fields whose change drops the cached entry, so flipping
the flag changed nothing until something else evicted the domain. SCIM-60
says the change takes effect without a restart.
Found by the two acceptance checks that were written but never called from
the driver, so neither had ever run: `authority` (SCIM-58 to SCIM-60,
through `synchronize_account` itself) and `rate_limits` (SCIM-14). Both are
wired in now, and `scim_tests` passes with them.
Two long-standing flakes in system_tests, both timing:
- security: the ban period was set to one second, but the test makes a
hundred more requests before it checks that a valid password from the
banned address is refused, so under load the ban had already expired.
Five seconds, and the expiry check sleeps six.
- task: a task scheduled one second out was queried for straight away,
and under load the query landed after the task manager had run and
removed it. Three seconds.
Seven consecutive system_tests runs, neither recurred.
A token the OIDC directory rejects is an authentication failure, so it
counts toward the ban; a network, provider or configuration fault stays
an error and doesn't. Before, a rejected token was an error too, so bad
tokens never led to a ban.
The Keycloak container now imports a second realm, so test 10 checks
/api/discover and the PACC record answer with each domain's own provider.
Test 18 checks that eight sign-ins during an outage don't ban the client,
while bad tokens do.
A FETCH with CHANGEDSINCE presents its mod-sequence to the read scope, so
a replica must have that change before it answers, as a JMAP sinceState
already did. replica_cluster_tests covers test 11: with the replica's
replay paused, a write on one node is read back through a second store
with its own marks, sharing through Redis.
Composite stores nest store futures deeply enough to pass rustc's default
query depth once both postgres and redis are compiled in, so the server
crates raise their recursion limit.
Two source-and-replica pairs run in containers: one replicating with
GTIDs, one by binary log position. mysql_replica_tests covers test 17
(tests 9, 10 and 12 with GTIDs) and mysql_replica_position_tests covers
test 18 (lag from Seconds_Behind_Source, and a replica whose account
lacks REPLICATION CLIENT getting no reads) and test 19 (a parallel
replica without replica_preserve_commit_order left out at startup).
The lag reader now takes Seconds_Behind_Source whatever numeric type the
server returns, accepts the older column name, and says in the log why it
gave up measuring.
A data store with readReplicas becomes a replicated store. Writes,
operator-written SQL and everything outside a read scope go to the
primary. JMAP reads before a request's first write, IMAP LIST, STATUS,
SEARCH, SORT and FETCH, POP3 RETR and TOP, DAV GET, PROPFIND and REPORT,
and blob downloads run in a read scope. Only account data (properties,
indexes, change logs, counters, ACLs, blobs, the search index) is read
from a replica; the registry, in-memory values, the task queue and the
rest stay on the primary.
In a scope, the first read picks a replica round-robin among those up
and under the lag limit, and only if it has every change this node has
written or heard of for the scope's accounts: marks come from write
results, the cluster's state-change broadcasts, a sinceState the client
presents, and, with more than one node, Redis. A write inside the scope
sends the rest of it to the primary. A miss on a replica is looked up on
the primary, and a replica error retries the read there and marks the
replica down.
Each node samples lag every second (WAL positions on PostgreSQL; GTID
sets or Seconds_Behind_Source on MySQL), stops reading from a replica
over 5 s and starts again under 2.5 s, and probes a down replica every
10 s. At startup a replica is left out if it's the primary, isn't
read-only, applies out of commit order, or doesn't show a marker written
to the primary within six tries.
replica_tests (postgres, STORE=PostgreSqlReplicated) runs a primary and a
streaming hot standby in containers: tests 9, 10, 12, 13, 14 and 15 pass.
per_domain_directory_compat, ignored, checks a copy of INBUXA's data has
no directory, no server default and no domain with its own directory, as
observed. The status names where the rules live, which suite covers each
acceptance test and how far, what was settled from the code, and the
known limits. SCIM's status notes its test 5 now passes.
directory_tests now runs a new oidc module in place of the removed one,
with Keycloak as example.org's own directory: first sign-in creates the
account with its name and group, an existing account is reused, a token
named as another user is refused, password sign-in is refused, forged
JWTs (HS256, unknown kid, another issuer, expired) are refused, an OIDC
address is a recipient only once an administrator creates it, and sync
can't pass a tenant's account limit. scim_oidc_tests, deferred until
this feature, passes.
A write to x:Directory or Authentication reloads the directories at once,
here and across the cluster. A directory that fails to open is logged as
a warning against its id and becomes unavailable; before, it was a build
error, and any build error stopped every later reload from applying.
per_domain_directory_tests covers acceptance tests 1, 3, 4, 6, 7, 9, 11,
15 and 19 over SQL directories on SQLite files: each domain against its
own directory and the default, no fallback to internal passwords, a
directory answering for another directory's domain, aliases and groups
dropped, recipients through the directory and a 4xx while it's down, app
passwords while it's down, password changes refused and then allowed
after a move to the internal directory, linked directories, tenant
foreign keys, and changes taking effect without a reload.
A Sharded blob store places each blob on xxh3(key) mod N, over the whole
key; reads fall back to the other members, so blobs placed under an
earlier member list stay readable, and deletes find them wherever they
are. The member list is recorded in the data store (secrets left out):
added or reordered members are a warning, a missing one refuses to open.
Blobs are compressed and marked before they reach a member. A Sharded
in-memory or lookup store sends each key to its home Redis member, and
prefix deletes and purges to all; a node whose member list differs from
the recorded one logs an error and runs on. Members are checked for
duplicates and must all open.
Until read-replica routing is built, each configured replica is reported
at startup instead of being silently ignored, and nothing connects to it.
The new scaleout_blob_tests covers tests 2 to 7, and the existing blob
suite passes against three FileSystem members (BLOB_STORE=Sharded).
A principal's key without unlimitedRequests gets 429 with Retry-After
over the limit, while its unlimited key still works. scim_compat, ignored,
checks a copy of INBUXA's data reads back as observed: no domain open to
SCIM and no externalId.
scim2-client builds its models from /Schemas, so meta is described there
as the mapping tables give it, without lastModified. The tester container
now shares the host's network, so a host firewall that drops the Docker
bridge doesn't block the test server. With SCIM_CONFORMANCE=1, the
scim2-client lifecycle (12 steps), the eight replayed Okta, Keycloak and
Entra payloads, and scim2-tester (errors only for its generated
non-address userName) all pass.
jcoffey-dev