Listeners that can't open: retry, then report through readiness #487

Merged
jcoffey-dev merged 1 commits from feat/485-listener-bind-failure into main 2026-11-19 17:52:18 +00:00
Owner

Closes #485 (follow-up to #436). As decided: retry, then report, don't exit by default.

  • Retry. Each listener opens in its own task. A port in use is retried with backoff (0.5 s doubling to 5 s) for 30 s. Any other bind error fails at once.
  • Readiness. While a listener isn't open (and while it's being retried), /healthz/ready answers 503 with {"unboundListeners": ["id address: reason"]}. Liveness is unchanged. Stopping a listener on purpose (legacy-protocols switch) clears it.
  • Console. Each node writes its list to the store (_ key Ol + node id; owner O "listener status" added to fork_subspaces::OWNERS). x:ClusterNode gets a read-only unboundListeners (property 947, served schema updated), read from there, so the console sees every node's list. A node clears its record at startup.
  • Optional exit. INBUXA_LISTENER_FAILURE=exit ends the process with code 1 when a listener configured at startup is still shut after the window. report is the default. Listeners put back later (LP-5) only report.

Pairs with inbuxa/inbuxa-admin#252 (dashboard item) and inbuxa/inbuxa.org#178 (docs).

Tests: bind_health (failure list, backoff, which errors retry, the setting); listen.rs against real sockets (a port in use is waited for, one held too long fails, other errors aren't retried). Fork checks, cargo fmt --check, cargo check -p inbuxa clean.

Live check (debug build, port 12993 held by another process):

  • Held 8 s: ready 503 naming imaps [::]:12993 ... Address already in use; retrying, same on x:ClusterNode/get. The port opened on its own once freed; ready 200; record cleared.
  • Held 50 s, default: after 30 s the reason reads (still in use after 30 s), ready stays 503, process stays up.
  • Held 45 s, INBUXA_LISTENER_FAILURE=exit: process exited code 1 at 30 s, naming the listener and reason.
Closes #485 (follow-up to #436). As decided: retry, then report, don't exit by default. - **Retry.** Each listener opens in its own task. A port in use is retried with backoff (0.5 s doubling to 5 s) for 30 s. Any other bind error fails at once. - **Readiness.** While a listener isn't open (and while it's being retried), `/healthz/ready` answers 503 with `{"unboundListeners": ["id address: reason"]}`. Liveness is unchanged. Stopping a listener on purpose (legacy-protocols switch) clears it. - **Console.** Each node writes its list to the store (`_` key `Ol` + node id; owner `O` "listener status" added to `fork_subspaces::OWNERS`). `x:ClusterNode` gets a read-only `unboundListeners` (property 947, served schema updated), read from there, so the console sees every node's list. A node clears its record at startup. - **Optional exit.** `INBUXA_LISTENER_FAILURE=exit` ends the process with code 1 when a listener configured at startup is still shut after the window. `report` is the default. Listeners put back later (LP-5) only report. Pairs with inbuxa/inbuxa-admin#252 (dashboard item) and inbuxa/inbuxa.org#178 (docs). **Tests:** bind_health (failure list, backoff, which errors retry, the setting); listen.rs against real sockets (a port in use is waited for, one held too long fails, other errors aren't retried). Fork checks, `cargo fmt --check`, `cargo check -p inbuxa` clean. **Live check** (debug build, port 12993 held by another process): - Held 8 s: ready 503 naming `imaps [::]:12993 ... Address already in use; retrying`, same on `x:ClusterNode/get`. The port opened on its own once freed; ready 200; record cleared. - Held 50 s, default: after 30 s the reason reads `(still in use after 30 s)`, ready stays 503, process stays up. - Held 45 s, `INBUXA_LISTENER_FAILURE=exit`: process exited **code 1** at 30 s, naming the listener and reason.
jcoffey-dev added 1 commit 2026-11-19 17:25:32 +00:00
Listeners that can't open: retry, then report through readiness (#485)
ci / fork-checks (pull_request) Skipped
ci / build (pull_request) Skipped
github/ci (branch) GitHub Actions
ci / github (pull_request) Successful in 13m50s
ea6c706125
A listener that couldn't bind used to be logged once and forgotten: the
node looked alive to anything checking the process, and a port the old
process still held for a moment after a restart was never tried again.

- Each listener now opens in its own task. A port in use is retried
  with backoff (0.5 s doubling to 5 s) for 30 s; any other bind error
  fails at once.
- A listener that isn't open is kept in ListenerControl's failures,
  "id address: reason" (also while it's being retried). /healthz/ready
  answers 503 with {"unboundListeners": [...]} while any is listed;
  liveness is unchanged. Stopping a listener on purpose (the
  legacy-protocols switch) clears it.
- Each node writes its list to the store (`_` key "Ol" + node id, owner
  'O' "listener status" in fork_subspaces), and x:ClusterNode gets a
  read-only unboundListeners (property 947) read from there, so the
  console can show it whichever node it talks to. A node clears its
  record at startup.
- INBUXA_LISTENER_FAILURE=exit ends the process with exit code 1 when a
  listener configured at startup can't open, after the retry window.
  "report" is the default. Listeners put back later (LP-5) only report.

Test: bind_health (failure list, backoff, which errors retry, the
setting) and listen.rs (a port in use is waited for; one held too long
fails after the window; other bind errors aren't retried).
jcoffey-dev merged commit 9be5cd4fdd into main 2026-11-19 17:52:18 +00:00
jcoffey-dev deleted branch feat/485-listener-bind-failure 2026-11-19 17:52:19 +00:00
jcoffey-dev referenced this issue from a commit 2026-11-20 01:59:38 +00:00
Sign in to join this conversation.