Node can come up without its HTTP listener after a concurrent first start #436

Closed
opened 2026-11-18 07:03:26 +00:00 by jcoffey-dev · 1 comment
Owner

Found in the Kubernetes lab (inbuxa-k8s-lab, spike S4), 27.1-rc.27, PostgreSQL + NATS + S3.

What happened. Straight after first boot (x:Bootstrap/set against an empty PostgreSQL database), two nodes were started at the same moment. Both reached Running. One started every listener except http (8080): its log has network.listen-start for smtp, submissions, imaps, pop3s, sieve and https, none for http, and nothing on 8080 in /proc/net/tcp6. That node answered nothing on /healthz/live or the API, while the process stayed up. The same node also logged, at the same second:

ERROR PostgreSQL error (store.postgresql-error) reason = "db error: ERROR: current transaction is aborted, commands ignored until end of transaction block", code = "25P02", details = "Failed to perform spam filter maintenance task"

Restarting that pod brought http up. Three more simultaneous two-node starts on the now-initialized store did not reproduce it, so it looks tied to the first start on a fresh store.

Why it matters. A listener that fails to start should either be retried or stop the process. Today the node looks alive to anything that checks the process, and only an HTTP probe notices. Under Kubernetes the liveness probe would restart it, but on systemd or plain Docker it would sit there half up.

Done when

  • A listener that can't start is logged as an error and either retried or makes the server exit non-zero.
  • Two nodes starting together on a freshly bootstrapped store both come up complete (lab test).

Workaround in the chart meanwhile: after first boot, start the first server node alone and wait for readiness; HTTP liveness probe on all server pods.

Found in the Kubernetes lab (inbuxa-k8s-lab, spike S4), 27.1-rc.27, PostgreSQL + NATS + S3. **What happened.** Straight after first boot (`x:Bootstrap/set` against an empty PostgreSQL database), two nodes were started at the same moment. Both reached Running. One started every listener except `http` (8080): its log has `network.listen-start` for smtp, submissions, imaps, pop3s, sieve and https, none for `http`, and nothing on 8080 in `/proc/net/tcp6`. That node answered nothing on `/healthz/live` or the API, while the process stayed up. The same node also logged, at the same second: ``` ERROR PostgreSQL error (store.postgresql-error) reason = "db error: ERROR: current transaction is aborted, commands ignored until end of transaction block", code = "25P02", details = "Failed to perform spam filter maintenance task" ``` Restarting that pod brought `http` up. Three more simultaneous two-node starts on the now-initialized store did not reproduce it, so it looks tied to the first start on a fresh store. **Why it matters.** A listener that fails to start should either be retried or stop the process. Today the node looks alive to anything that checks the process, and only an HTTP probe notices. Under Kubernetes the liveness probe would restart it, but on systemd or plain Docker it would sit there half up. **Done when** - A listener that can't start is logged as an error and either retried or makes the server exit non-zero. - Two nodes starting together on a freshly bootstrapped store both come up complete (lab test). Workaround in the chart meanwhile: after first boot, start the first server node alone and wait for readiness; HTTP liveness probe on all server pods.
jcoffey-dev added this to the 27.1 milestone 2026-11-18 07:03:26 +00:00
jcoffey-dev added the bug label 2026-11-18 07:03:26 +00:00
jcoffey-dev self-assigned this 2026-11-18 07:16:14 +00:00
jcoffey-dev added the
priority
p2
label 2026-11-18 07:16:14 +00:00
Author
Owner

Fixed in #482. Cause: a race on first start. Each start adds any default listeners not yet stored; the second node found 6 of 7 already written (http is last) and took the set as complete. The default roles could be stored twice the same way. Now one node at a time adds the defaults, under a short lease in the store; the other waits, then reads the complete set. Ships in the next RC.

Not covered yet: what happens when a listener that is tried fails to bind (retry, exit, or report). That is being decided separately. The two-node lab test is for the Kubernetes lab.

Fixed in #482. Cause: a race on first start. Each start adds any default listeners not yet stored; the second node found 6 of 7 already written (`http` is last) and took the set as complete. The default roles could be stored twice the same way. Now one node at a time adds the defaults, under a short lease in the store; the other waits, then reads the complete set. Ships in the next RC. Not covered yet: what happens when a listener that *is* tried fails to bind (retry, exit, or report). That is being decided separately. The two-node lab test is for the Kubernetes lab.
Sign in to join this conversation.
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: inbuxa/inbuxa-server#436