Settings writes: wait for a burst to settle before reloading
ci / fork-checks (pull_request) Successful in 56s
ci / build (pull_request) Successful in 12m43s

A cluster rehearsal sent ten x:<Object>/set requests at once and got
ten full reloads on every node. #39's coalescing only joined writes
that queued behind a running reload, but the requests reached the
server about 33 ms apart and a reload takes tens of milliseconds, so
none overlapped one.

A full reload after a registry write now waits for writes to settle:
75 ms after the last one, and at most 250 ms after the first it
covers, so a steady stream still reloads at least four times a
second. 75 ms is a little over twice the gap the rehearsal saw between
requests. A single write pays it once: in the tests a settings write
takes about 140 ms instead of 60. The reload runs in a task of its
own, so a request that goes away doesn't cancel it for the others.
Each write takes the result of the first reload that started after it
was stored (the gate keeps the last 64 results), so applied true or
false still describes the reload that covered that write.

The 33 ms gap was a queue on the server, not password hashing: Basic
credentials are cached per Authorization header, so they are checked
once. Every authenticated HTTP request counted itself against the
account's rate limit by incrementing one counter per account in the
in-memory store, so parallel requests from one account queued on that
key: a row lock on PostgreSQL (a few round trips to the database
each) and conflict retries with a 50-300 ms backoff on RocksDB. An
account with the unlimitedRequests permission (administrators, by
default) passes the rate and concurrency limits anyway, so its
requests are no longer counted. Ten parallel Core/echo calls as the
admin now finish in 1-4 ms; before, they finished one after another
over 20 ms on a local PostgreSQL and 300-450 ms on RocksDB. Other
accounts still count every request.

system::auto_reload::settings_reload_tests: ten concurrent writes now
take one reload (the gate counts them; at most two allowed), all are
applied: true and in the running settings, and a single write takes
exactly one reload. RocksDB and PostgreSQL, 1 reload in 141-196 ms.
With the old behavior (no wait, requests counted) the same writes
took 5 reloads; without the wait but with the rate fix, 2.
cluster::broadcast (3 nodes, PostgreSQL + NATS) and system::reload
still pass.
This commit is contained in:
2026-09-24 21:15:44 -07:00
parent 51b159a1a2
commit 71ce11c57d
3 changed files with 198 additions and 18 deletions
+12
View File
@@ -2,6 +2,8 @@
* SPDX-FileCopyrightText: 2020 Stalwart Labs LLC <[email protected]>
*
* SPDX-License-Identifier: AGPL-3.0-only OR LicenseRef-SEL
*
* Modified by Coffey Labs in 2026 for INBUXA.
*/
use crate::auth::AccessToken;
@@ -18,6 +20,16 @@ impl Server {
access_token: &AccessToken,
addr: IpAddr,
) -> trc::Result<Option<InFlight>> {
// inbuxa: an account with unlimited requests passes both limits
// below anyway, so don't count its requests. The count is a write to
// one counter per account in the in-memory store, and concurrent
// requests from one account queue on that key (a row lock on SQL,
// conflict retries on RocksDB): in a cluster rehearsal ten parallel
// admin writes were accepted one after another, about 33 ms apart.
if access_token.has_permission(Permission::UnlimitedRequests) {
return Ok(None);
}
let rate_reset = if let Some(rate) = &self.core.network.http.rate_authenticated {
if self.is_ip_allowed(addr) {
None