eac3db34e91e1fb628c6a8935767a1eb2d538f11
3
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 jcoffey-dev
|
71ce11c57d |
Settings writes: wait for a burst to settle before reloading
A cluster rehearsal sent ten x:<Object>/set requests at once and got ten full reloads on every node. #39's coalescing only joined writes that queued behind a running reload, but the requests reached the server about 33 ms apart and a reload takes tens of milliseconds, so none overlapped one. A full reload after a registry write now waits for writes to settle: 75 ms after the last one, and at most 250 ms after the first it covers, so a steady stream still reloads at least four times a second. 75 ms is a little over twice the gap the rehearsal saw between requests. A single write pays it once: in the tests a settings write takes about 140 ms instead of 60. The reload runs in a task of its own, so a request that goes away doesn't cancel it for the others. Each write takes the result of the first reload that started after it was stored (the gate keeps the last 64 results), so applied true or false still describes the reload that covered that write. The 33 ms gap was a queue on the server, not password hashing: Basic credentials are cached per Authorization header, so they are checked once. Every authenticated HTTP request counted itself against the account's rate limit by incrementing one counter per account in the in-memory store, so parallel requests from one account queued on that key: a row lock on PostgreSQL (a few round trips to the database each) and conflict retries with a 50-300 ms backoff on RocksDB. An account with the unlimitedRequests permission (administrators, by default) passes the rate and concurrency limits anyway, so its requests are no longer counted. Ten parallel Core/echo calls as the admin now finish in 1-4 ms; before, they finished one after another over 20 ms on a local PostgreSQL and 300-450 ms on RocksDB. Other accounts still count every request. system::auto_reload::settings_reload_tests: ten concurrent writes now take one reload (the gate counts them; at most two allowed), all are applied: true and in the running settings, and a single write takes exactly one reload. RocksDB and PostgreSQL, 1 reload in 141-196 ms. With the old behavior (no wait, requests counted) the same writes took 5 reloads; without the wait but with the rate fix, 2. cluster::broadcast (3 nodes, PostgreSQL + NATS) and system::reload still pass. |
||
|
jcoffey-dev
|
fcef4b1c3f |
Allowed IPs take the full settings reload after a write
write_reload_target sent AllowedIp writes to the blocked-IP reload, but
that reload rebuilds only BlockedIps. Allowed IPs are parsed into the
core's security settings (Security::parse), which only a full reload
rebuilds, so an AllowedIp write reported x:settingsReload applied: true
while the change wasn't live until the next full reload.
AllowedIp now maps to the full reload, like the other settings objects;
BlockedIp keeps its targeted reload.
system::auto_reload::settings_reload_tests now creates an allowed IP
over JMAP and checks that is_ip_allowed sees it with no ReloadSettings,
and that destroying it takes it out again. On main it fails ("allowed
IP not in the running settings").
|
||
|
jcoffey-dev
|
2c684be5c9 |
Registry writes apply to the running settings without ReloadSettings
A 3-node rehearsal found that saving an MtaDeliverySchedule left it
unknown to the queue ("Queue strategy not found") until someone ran
x:Action ReloadSettings; only Directory and Authentication writes
reloaded (DIR-17). The admin UI has to remember a separate reload after
every save, and a script or API client that doesn't gets a server
running stale settings.
x:<Object>/set now reloads the running settings when it created,
updated or destroyed an object they are built from, and broadcasts the
same RegistryChange::Reload over the coordinator as ReloadSettings, so
every node applies it:
- Settings objects (MTA, spam filter, listeners, tracers, Sieve system
scripts, cluster roles, directories, ...: the object types the core,
telemetry, listener and directory builders read) get a full reload.
- Certificates, lookup stores and blocked/allowed IPs get their own
targeted reloads.
- Accounts, domains, roles and other data read as needed, stores (they
take a restart) and applications (their own reload action) get none.
Full reloads are coalesced: a write waits for a reload that started
after it was stored and joins one if it can, so a burst of writes, or
a request with many objects, costs one or two reloads, not one each.
The write itself is never undone. When the reload is refused (build
errors in objects that were working, the rule from the previous
commit), the set response says so in a new x:settingsReload field,
{"applied": false, "description": "Saved, but the running settings
were not reloaded. <object>: <error>"}; {"applied": true} otherwise.
The field is absent when the write needs no reload. The description
helper is shared with ReloadSettings' refusal.
Each reload sends the queue a ReloadSettings event, so the SMTP test
harness's read_event, try_read_event and assert_no_events now pass over
those; expect_reload_settings still waits for one.
system::auto_reload::settings_reload_tests (new): an MtaVirtualQueue
and an MtaDeliverySchedule created over JMAP are in the running
settings with no ReloadSettings, and gone once destroyed; eight
concurrent creates all land; a write whose reload fails is stored and
reported applied: false with the error; a domain write carries no
x:settingsReload. On main the new schedule is missing. The cluster
broadcast test (three nodes, PostgreSQL + NATS) now checks that every
node has a schedule created on node 0 without a reload.
|