Coordinator: join the cluster when NATS comes up, report the connection
A node that started while NATS was down never got a coordinator. The
connect failed at boot, bootstrap recorded a build error and the node ran
with Coordinator::None until restarted. It had no broadcast subscriber
or publisher, so cross-node push and cache invalidation to it stayed
broken, and its healthcheck said nothing about it. Losing NATS after
startup was silent too.
- The NATS client now connects in the background
(retry_on_initial_connect): startup never waits on NATS or fails over
it, the node gets its coordinator, subscriber and publisher at once,
and the client keeps trying (async-nats's backoff, at most 4 s apart)
until NATS answers. Subscriptions made meanwhile start delivering when
it does. A configured maxReconnects still ends the attempts.
- Three new events report the connection: cluster.coordinator-connected
(info), cluster.coordinator-disconnected (warn: lost, closed, gave up,
or not connected within the connection timeout at startup) and
cluster.coordinator-error (warn: a failed attempt, reported once per
outage rather than every retry, and server errors, slow consumers and
lame duck mode). They are in the packaged schema, ids 644 to 646.
- GET /healthz/cluster reports the coordinator: 200
{"coordinator":"connected"}, 503 {"coordinator":"disconnected"}, or
200 with "none" (no coordinator) or "unknown" (a backend that doesn't
track its connection). /healthz/live and /healthz/ready are unchanged
on purpose: a node without its coordinator still serves mail, and
failing those would have orchestrators restart, or pull out of
service, every node at once whenever NATS is down.
Only NATS connects lazily; the other coordinator backends still fail at
boot as before.
cluster::coordinator::coordinator_reconnect_tests starts a node against a
NATS port with nothing behind it, checks it boots with a coordinator and
reports it disconnected, subscribes, then starts NATS on that port: the
node connects on its own and the subscription receives a message from a
second client. Stopping and restarting NATS shows disconnected, then
connected, and the same subscription keeps working.
This commit is contained in:
@@ -10,8 +10,9 @@
|
||||
|
||||
// inbuxa: 637 to 641 are the fork's SCIM events (SCIM-54); 642 is
|
||||
// auth.legacy-protocol-refused (legacy-protocols LP-6); 643 is
|
||||
// security.legacy-protocols-changed (LP-8)
|
||||
pub const TOTAL_EVENT_COUNT: usize = 644;
|
||||
// security.legacy-protocols-changed (LP-8); 644 to 646 are the cluster
|
||||
// coordinator's connection events
|
||||
pub const TOTAL_EVENT_COUNT: usize = 647;
|
||||
pub const TOTAL_METRIC_COUNT: usize = 369;
|
||||
|
||||
#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)]
|
||||
@@ -150,6 +151,10 @@ pub enum ClusterEvent {
|
||||
MessageSkipped = 47,
|
||||
MessageInvalid = 49,
|
||||
NodeIdRenewed = 275,
|
||||
// inbuxa: the coordinator's connection
|
||||
CoordinatorConnected = 644,
|
||||
CoordinatorDisconnected = 645,
|
||||
CoordinatorError = 646,
|
||||
}
|
||||
|
||||
#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)]
|
||||
|
||||
@@ -81,6 +81,10 @@ impl EventType {
|
||||
b"cluster.message-skipped" => EventType::Cluster(ClusterEvent::MessageSkipped),
|
||||
b"cluster.message-invalid" => EventType::Cluster(ClusterEvent::MessageInvalid),
|
||||
b"cluster.node-id-renewed" => EventType::Cluster(ClusterEvent::NodeIdRenewed),
|
||||
// inbuxa: coordinator connection
|
||||
b"cluster.coordinator-connected" => EventType::Cluster(ClusterEvent::CoordinatorConnected),
|
||||
b"cluster.coordinator-disconnected" => EventType::Cluster(ClusterEvent::CoordinatorDisconnected),
|
||||
b"cluster.coordinator-error" => EventType::Cluster(ClusterEvent::CoordinatorError),
|
||||
b"dane.authentication-success" => EventType::Dane(DaneEvent::AuthenticationSuccess),
|
||||
b"dane.authentication-failure" => EventType::Dane(DaneEvent::AuthenticationFailure),
|
||||
b"dane.no-certificates-found" => EventType::Dane(DaneEvent::NoCertificatesFound),
|
||||
@@ -742,6 +746,14 @@ impl EventType {
|
||||
EventType::Cluster(ClusterEvent::MessageSkipped) => "cluster.message-skipped",
|
||||
EventType::Cluster(ClusterEvent::MessageInvalid) => "cluster.message-invalid",
|
||||
EventType::Cluster(ClusterEvent::NodeIdRenewed) => "cluster.node-id-renewed",
|
||||
// inbuxa: coordinator connection
|
||||
EventType::Cluster(ClusterEvent::CoordinatorConnected) => {
|
||||
"cluster.coordinator-connected"
|
||||
}
|
||||
EventType::Cluster(ClusterEvent::CoordinatorDisconnected) => {
|
||||
"cluster.coordinator-disconnected"
|
||||
}
|
||||
EventType::Cluster(ClusterEvent::CoordinatorError) => "cluster.coordinator-error",
|
||||
EventType::Dane(DaneEvent::AuthenticationSuccess) => "dane.authentication-success",
|
||||
EventType::Dane(DaneEvent::AuthenticationFailure) => "dane.authentication-failure",
|
||||
EventType::Dane(DaneEvent::NoCertificatesFound) => "dane.no-certificates-found",
|
||||
@@ -1524,6 +1536,10 @@ impl EventType {
|
||||
EventType::Cluster(ClusterEvent::MessageSkipped) => 47,
|
||||
EventType::Cluster(ClusterEvent::MessageInvalid) => 49,
|
||||
EventType::Cluster(ClusterEvent::NodeIdRenewed) => 275,
|
||||
// inbuxa: coordinator connection
|
||||
EventType::Cluster(ClusterEvent::CoordinatorConnected) => 644,
|
||||
EventType::Cluster(ClusterEvent::CoordinatorDisconnected) => 645,
|
||||
EventType::Cluster(ClusterEvent::CoordinatorError) => 646,
|
||||
EventType::Dane(DaneEvent::AuthenticationSuccess) => 67,
|
||||
EventType::Dane(DaneEvent::AuthenticationFailure) => 66,
|
||||
EventType::Dane(DaneEvent::NoCertificatesFound) => 69,
|
||||
@@ -2176,6 +2192,10 @@ impl EventType {
|
||||
47 => Some(EventType::Cluster(ClusterEvent::MessageSkipped)),
|
||||
49 => Some(EventType::Cluster(ClusterEvent::MessageInvalid)),
|
||||
275 => Some(EventType::Cluster(ClusterEvent::NodeIdRenewed)),
|
||||
// inbuxa: coordinator connection
|
||||
644 => Some(EventType::Cluster(ClusterEvent::CoordinatorConnected)),
|
||||
645 => Some(EventType::Cluster(ClusterEvent::CoordinatorDisconnected)),
|
||||
646 => Some(EventType::Cluster(ClusterEvent::CoordinatorError)),
|
||||
67 => Some(EventType::Dane(DaneEvent::AuthenticationSuccess)),
|
||||
66 => Some(EventType::Dane(DaneEvent::AuthenticationFailure)),
|
||||
69 => Some(EventType::Dane(DaneEvent::NoCertificatesFound)),
|
||||
@@ -3114,6 +3134,10 @@ impl EventType {
|
||||
EventType::Auth(AuthEvent::TooManyAttempts) => Level::Warn,
|
||||
EventType::Calendar(CalendarEvent::AlarmFailed) => Level::Warn,
|
||||
EventType::Cluster(ClusterEvent::SubscriberDisconnected) => Level::Warn,
|
||||
// inbuxa: coordinator connection
|
||||
EventType::Cluster(ClusterEvent::CoordinatorConnected) => Level::Info,
|
||||
EventType::Cluster(ClusterEvent::CoordinatorDisconnected) => Level::Warn,
|
||||
EventType::Cluster(ClusterEvent::CoordinatorError) => Level::Warn,
|
||||
EventType::Delivery(DeliveryEvent::MissingOutboundHostname) => Level::Warn,
|
||||
EventType::Delivery(DeliveryEvent::ConcurrencyLimitExceeded) => Level::Warn,
|
||||
EventType::Delivery(DeliveryEvent::RateLimitExceeded) => Level::Warn,
|
||||
@@ -3244,6 +3268,10 @@ impl EventType {
|
||||
EventType::Cluster(ClusterEvent::MessageSkipped) => "PubSub message skipped",
|
||||
EventType::Cluster(ClusterEvent::MessageInvalid) => "Invalid PubSub message",
|
||||
EventType::Cluster(ClusterEvent::NodeIdRenewed) => "Node ID renewed",
|
||||
// inbuxa: coordinator connection
|
||||
EventType::Cluster(ClusterEvent::CoordinatorConnected) => "Coordinator connected",
|
||||
EventType::Cluster(ClusterEvent::CoordinatorDisconnected) => "Coordinator unavailable",
|
||||
EventType::Cluster(ClusterEvent::CoordinatorError) => "Coordinator error",
|
||||
EventType::Dane(DaneEvent::AuthenticationSuccess) => "DANE authentication successful",
|
||||
EventType::Dane(DaneEvent::AuthenticationFailure) => "DANE authentication failed",
|
||||
EventType::Dane(DaneEvent::NoCertificatesFound) => "No certificates found for DANE",
|
||||
@@ -4322,6 +4350,10 @@ impl EventType {
|
||||
EventType::Cluster(ClusterEvent::MessageSkipped),
|
||||
EventType::Cluster(ClusterEvent::MessageInvalid),
|
||||
EventType::Cluster(ClusterEvent::NodeIdRenewed),
|
||||
// inbuxa: coordinator connection
|
||||
EventType::Cluster(ClusterEvent::CoordinatorConnected),
|
||||
EventType::Cluster(ClusterEvent::CoordinatorDisconnected),
|
||||
EventType::Cluster(ClusterEvent::CoordinatorError),
|
||||
EventType::Dane(DaneEvent::AuthenticationSuccess),
|
||||
EventType::Dane(DaneEvent::AuthenticationFailure),
|
||||
EventType::Dane(DaneEvent::NoCertificatesFound),
|
||||
|
||||
Reference in New Issue
Block a user