Coordinator: join the cluster when NATS comes up, report the connection #36

Merged

jcoffey-dev merged 1 commits from fix/coordinator-retry-and-health into main

2026-09-24 15:51:07 +00:00

jcoffey-dev commented

2026-09-24 15:29:48 +00:00

Owner

Fixes B2 and B7 from the 3-node PostgreSQL + NATS cluster rehearsal.

B2: a node that boots while NATS is down never gets a coordinator

NatsPubSub::open connected eagerly. When NATS was unreachable, Coordinator::build (crates/coordinator/src/bootstrap.rs) recorded a build error and returned None, and StorageConfig fell back to Coordinator::None for the life of the process. Everything that depends on a coordinator is decided once at startup:

the IPC broadcast channel (build_ipc(has_pubsub));
spawn_broadcast_subscriber, which returns immediately when there is no coordinator;
the publisher;
the task manager's clustered refresh;
the node-id lease renewal.

So cross-node push to that node stayed broken until a restart. The rehearsal saw it still broken 4+ minutes after NATS came back.

Fix. Cargo.lock has async-nats 0.50, which supports ConnectOptions::retry_on_initial_connect(). The client is created at once and connects in the background with the crate's reconnect backoff: immediately, then doubling, at most 4 s apart, and unlimited unless maxReconnects is set. So the node always boots with Coordinator::Nats. Subscriber, publisher and the rest start as normal, and subscriptions made before the connection exists start delivering once it does. Startup never waits on NATS or fails because of it. A config reload no longer blocks on it either.

I chose this over a background rebuild-and-swap of the coordinator because of the startup decisions above. With a swap, every one of them would also have to be redone when the coordinator appears. The lazy client needs none of that.

Not covered: Redis, Kafka and Zenoh coordinators still fail at boot the old way. The rehearsal used NATS.

B7: losing the coordinator was silent

New events (in trc and the packaged schema, ids 644 to 646):

Event	Level	When
`cluster.coordinator-connected`	info	first connection and every reconnection
`cluster.coordinator-disconnected`	warn	connection lost; closed; gave up (`maxReconnects`); or not connected within `timeoutConnection` at startup (the B2 case)
`cluster.coordinator-error`	warn	a failed connection attempt, only the first after each connect/disconnect, since the client retries every few seconds; also server errors, slow consumers, lame duck mode

Health. GET /healthz/cluster returns:

200 {"coordinator":"connected"} when connected;
503 {"coordinator":"disconnected"} when a NATS coordinator is configured but not connected;
200 with "none" when there is no coordinator;
200 with "unknown" for backends that don't track their connection (Redis, Kafka, Zenoh).

/healthz/live (the container HEALTHCHECK) and /healthz/ready are deliberately unchanged. A node without its coordinator still accepts, stores and serves mail; only cross-node push and cache invalidation lag. If either endpoint failed while NATS was down, an orchestrator would restart every node, or pull every node out of service, at once, turning a coordinator outage into a mail outage. /healthz/cluster is there for monitoring and alerting, or for a readiness gate where that trade-off is wanted.

The docs site should list /healthz/cluster next to live and ready.

Tests

New: cluster::coordinator::coordinator_reconnect_tests (needs Docker and the nats feature; STORE=RocksDb). Passes.
1. Start a node against a NATS port with nothing behind it. It boots without build errors, has a coordinator, reports 503 disconnected, and subscribes.
2. Start a NATS container on that port. The node reports 200 connected, and its early subscription receives a message published by a second client.
3. Stop NATS: 503 disconnected. Start it again: 200 connected, and the same subscription keeps working.
4. With LOG=info, the events above appear in that order, with one coordinator-error per outage per client.
The existing cluster::broadcast::cluster_tests (3 nodes, STORE=PostgreSql COORDINATOR=Nats) passes.
tools/fork/*-check.py: clean.

Fixes B2 and B7 from the 3-node PostgreSQL + NATS cluster rehearsal. ## B2: a node that boots while NATS is down never gets a coordinator `NatsPubSub::open` connected eagerly. When NATS was unreachable, `Coordinator::build` (`crates/coordinator/src/bootstrap.rs`) recorded a build error and returned `None`, and `StorageConfig` fell back to `Coordinator::None` for the life of the process. Everything that depends on a coordinator is decided once at startup: - the IPC broadcast channel (`build_ipc(has_pubsub)`); - `spawn_broadcast_subscriber`, which returns immediately when there is no coordinator; - the publisher; - the task manager's clustered refresh; - the node-id lease renewal. So cross-node push to that node stayed broken until a restart. The rehearsal saw it still broken 4+ minutes after NATS came back. **Fix.** Cargo.lock has async-nats 0.50, which supports `ConnectOptions::retry_on_initial_connect()`. The client is created at once and connects in the background with the crate's reconnect backoff: immediately, then doubling, at most 4 s apart, and unlimited unless `maxReconnects` is set. So the node always boots with `Coordinator::Nats`. Subscriber, publisher and the rest start as normal, and subscriptions made before the connection exists start delivering once it does. Startup never waits on NATS or fails because of it. A config reload no longer blocks on it either. I chose this over a background rebuild-and-swap of the coordinator because of the startup decisions above. With a swap, every one of them would also have to be redone when the coordinator appears. The lazy client needs none of that. **Not covered:** Redis, Kafka and Zenoh coordinators still fail at boot the old way. The rehearsal used NATS. ## B7: losing the coordinator was silent **New events** (in `trc` and the packaged schema, ids 644 to 646): | Event | Level | When | |---|---|---| | `cluster.coordinator-connected` | info | first connection and every reconnection | | `cluster.coordinator-disconnected` | warn | connection lost; closed; gave up (`maxReconnects`); or not connected within `timeoutConnection` at startup (the B2 case) | | `cluster.coordinator-error` | warn | a failed connection attempt, only the first after each connect/disconnect, since the client retries every few seconds; also server errors, slow consumers, lame duck mode | **Health.** `GET /healthz/cluster` returns: - `200 {"coordinator":"connected"}` when connected; - `503 {"coordinator":"disconnected"}` when a NATS coordinator is configured but not connected; - `200` with `"none"` when there is no coordinator; - `200` with `"unknown"` for backends that don't track their connection (Redis, Kafka, Zenoh). `/healthz/live` (the container `HEALTHCHECK`) and `/healthz/ready` are deliberately unchanged. A node without its coordinator still accepts, stores and serves mail; only cross-node push and cache invalidation lag. If either endpoint failed while NATS was down, an orchestrator would restart every node, or pull every node out of service, at once, turning a coordinator outage into a mail outage. `/healthz/cluster` is there for monitoring and alerting, or for a readiness gate where that trade-off is wanted. The docs site should list `/healthz/cluster` next to live and ready. ## Tests - New: `cluster::coordinator::coordinator_reconnect_tests` (needs Docker and the `nats` feature; `STORE=RocksDb`). Passes. 1. Start a node against a NATS port with nothing behind it. It boots without build errors, has a coordinator, reports `503 disconnected`, and subscribes. 2. Start a NATS container on that port. The node reports `200 connected`, and its early subscription receives a message published by a second client. 3. Stop NATS: `503 disconnected`. Start it again: `200 connected`, and the same subscription keeps working. 4. With `LOG=info`, the events above appear in that order, with one `coordinator-error` per outage per client. - The existing `cluster::broadcast::cluster_tests` (3 nodes, `STORE=PostgreSql COORDINATOR=Nats`) passes. - `tools/fork/*-check.py`: clean.

jcoffey-dev added 1 commit 2026-09-24 15:39:54 +00:00

Coordinator: join the cluster when NATS comes up, report the connection

ci / fork-checks (pull_request) Successful in 46s

Details

ci / build (pull_request) Successful in 11m8s

Details

95f0445d83

A node that started while NATS was down never got a coordinator. The
connect failed at boot, bootstrap recorded a build error and the node ran
with Coordinator::None until restarted. It had no broadcast subscriber
or publisher, so cross-node push and cache invalidation to it stayed
broken, and its healthcheck said nothing about it. Losing NATS after
startup was silent too.

- The NATS client now connects in the background
  (retry_on_initial_connect): startup never waits on NATS or fails over
  it, the node gets its coordinator, subscriber and publisher at once,
  and the client keeps trying (async-nats's backoff, at most 4 s apart)
  until NATS answers. Subscriptions made meanwhile start delivering when
  it does. A configured maxReconnects still ends the attempts.
- Three new events report the connection: cluster.coordinator-connected
  (info), cluster.coordinator-disconnected (warn: lost, closed, gave up,
  or not connected within the connection timeout at startup) and
  cluster.coordinator-error (warn: a failed attempt, reported once per
  outage rather than every retry, and server errors, slow consumers and
  lame duck mode). They are in the packaged schema, ids 644 to 646.
- GET /healthz/cluster reports the coordinator: 200
  {"coordinator":"connected"}, 503 {"coordinator":"disconnected"}, or
  200 with "none" (no coordinator) or "unknown" (a backend that doesn't
  track its connection). /healthz/live and /healthz/ready are unchanged
  on purpose: a node without its coordinator still serves mail, and
  failing those would have orchestrators restart, or pull out of
  service, every node at once whenever NATS is down.

Only NATS connects lazily; the other coordinator backends still fail at
boot as before.

cluster::coordinator::coordinator_reconnect_tests starts a node against a
NATS port with nothing behind it, checks it boots with a coordinator and
reports it disconnected, subscribes, then starts NATS on that port: the
node connects on its own and the subscription receives a message from a
second client. Stopping and restarting NATS shows disconnected, then
connected, and the same subscription keeps working.

jcoffey-dev force-pushed fix/coordinator-retry-and-health from 8a8f48c944 to 95f0445d83

2026-09-24 15:39:54 +00:00

Compare

jcoffey-dev merged commit 9311c1a38b into main

2026-09-24 15:51:07 +00:00

jcoffey-dev deleted branch fix/coordinator-retry-and-health

2026-09-24 15:51:07 +00:00

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: inbuxa/inbuxa-server#36