Drop local caches and reload settings when the coordinator comes back

jcoffey-dev commented

2026-10-08 21:34:46 +00:00

Owner

Backlog Tier 2, correctness. Core NATS delivers at most once. While a node is disconnected the async-nats client reconnects and re-subscribes on its own, but every broadcast published meanwhile is gone. The account, access-token, role, domain, email-address, DKIM and DAV caches have no TTL and only change on those broadcasts, so after a NATS blip a node could serve stale principals or permissions until restarted; a missed registry change left its running settings stale the same way.

The NATS backend counts reconnections (a Connected event after a first connection) on a tokio watch; the Coordinator exposes it (None for other backends).
The broadcast subscriber selects on that watch next to the message stream. On a reconnection, and after subscribing again when its stream ended, it drops every local cache (invalidate_all_local_caches, negative caches included) and reloads the settings, certificates and blocked addresses through reload_registry, then logs cluster.coordinator-connected with the reason. The message caches check the change log on each use and need nothing.

Unit test: only a connection that comes back counts as a reconnection. No new event ids.

Backlog Tier 2, correctness. Core NATS delivers at most once. While a node is disconnected the async-nats client reconnects and re-subscribes on its own, but every broadcast published meanwhile is gone. The account, access-token, role, domain, email-address, DKIM and DAV caches have no TTL and only change on those broadcasts, so after a NATS blip a node could serve stale principals or permissions until restarted; a missed registry change left its running settings stale the same way. - The NATS backend counts reconnections (a Connected event after a first connection) on a tokio watch; the Coordinator exposes it (None for other backends). - The broadcast subscriber selects on that watch next to the message stream. On a reconnection, and after subscribing again when its stream ended, it drops every local cache (invalidate_all_local_caches, negative caches included) and reloads the settings, certificates and blocked addresses through reload_registry, then logs cluster.coordinator-connected with the reason. The message caches check the change log on each use and need nothing. Unit test: only a connection that comes back counts as a reconnection. No new event ids.

jcoffey-dev added 1 commit 2026-10-08 21:34:47 +00:00

Drop local caches and reload settings when the coordinator comes back

ci / fork-checks (pull_request) Skipped

Details

ci / build (pull_request) Skipped

Details

github/ci (branch) GitHub Actions

Details

ci / github (pull_request) Successful in 8m26s

Details

5344615845

Core NATS delivers at most once. While a node's connection is down the
client reconnects by itself and subscribes again, but whatever the
cluster published meanwhile never arrives, and the account, token,
role, domain and DAV caches only ever change on those events, so they
stayed stale until a restart; a missed registry change left the running
settings stale the same way.

The NATS backend now counts reconnections on a watch. The broadcast
subscriber drops every local cache and reloads the settings,
certificates and blocked addresses on each one, and after subscribing
again when its stream ended. The message caches check the change log
on every use and need nothing.

jcoffey-dev merged commit 9b5e111538 into main

2026-10-08 22:31:17 +00:00

jcoffey-dev deleted branch fix/caches-on-reconnect

2026-10-08 22:31:17 +00:00

jcoffey-dev referenced this pull request

2026-10-08 22:42:53 +00:00

Release 2026.10.10 #179

Drop local caches and reload settings when the coordinator comes back #176