Found in the Kubernetes lab (inbuxa-k8s-lab, spike S4), 27.1-rc.27, PostgreSQL + NATS + S3.
What happened. Straight after first boot (x:Bootstrap/set against an empty PostgreSQL database), two nodes were started at the same moment. Both reached Running. One started every listener except http (8080): its log has network.listen-start for smtp, submissions, imaps, pop3s, sieve and https, none for http, and nothing on 8080 in /proc/net/tcp6. That node answered nothing on /healthz/live or the API, while the process stayed up. The same node also logged, at the same second:
ERROR PostgreSQL error (store.postgresql-error) reason = "db error: ERROR: current transaction is aborted, commands ignored until end of transaction block", code = "25P02", details = "Failed to perform spam filter maintenance task"
Restarting that pod brought http up. Three more simultaneous two-node starts on the now-initialized store did not reproduce it, so it looks tied to the first start on a fresh store.
Why it matters. A listener that fails to start should either be retried or stop the process. Today the node looks alive to anything that checks the process, and only an HTTP probe notices. Under Kubernetes the liveness probe would restart it, but on systemd or plain Docker it would sit there half up.
Done when
A listener that can't start is logged as an error and either retried or makes the server exit non-zero.
Two nodes starting together on a freshly bootstrapped store both come up complete (lab test).
Workaround in the chart meanwhile: after first boot, start the first server node alone and wait for readiness; HTTP liveness probe on all server pods.
Found in the Kubernetes lab (inbuxa-k8s-lab, spike S4), 27.1-rc.27, PostgreSQL + NATS + S3.
**What happened.** Straight after first boot (`x:Bootstrap/set` against an empty PostgreSQL database), two nodes were started at the same moment. Both reached Running. One started every listener except `http` (8080): its log has `network.listen-start` for smtp, submissions, imaps, pop3s, sieve and https, none for `http`, and nothing on 8080 in `/proc/net/tcp6`. That node answered nothing on `/healthz/live` or the API, while the process stayed up. The same node also logged, at the same second:
```
ERROR PostgreSQL error (store.postgresql-error) reason = "db error: ERROR: current transaction is aborted, commands ignored until end of transaction block", code = "25P02", details = "Failed to perform spam filter maintenance task"
```
Restarting that pod brought `http` up. Three more simultaneous two-node starts on the now-initialized store did not reproduce it, so it looks tied to the first start on a fresh store.
**Why it matters.** A listener that fails to start should either be retried or stop the process. Today the node looks alive to anything that checks the process, and only an HTTP probe notices. Under Kubernetes the liveness probe would restart it, but on systemd or plain Docker it would sit there half up.
**Done when**
- A listener that can't start is logged as an error and either retried or makes the server exit non-zero.
- Two nodes starting together on a freshly bootstrapped store both come up complete (lab test).
Workaround in the chart meanwhile: after first boot, start the first server node alone and wait for readiness; HTTP liveness probe on all server pods.
jcoffey-dev
added this to the 27.1 milestone 2026-11-18 07:03:26 +00:00
jcoffey-dev
added the bug label 2026-11-18 07:03:26 +00:00
jcoffey-dev
self-assigned this 2026-11-18 07:16:14 +00:00
Fixed in #482. Cause: a race on first start. Each start adds any default listeners not yet stored; the second node found 6 of 7 already written (http is last) and took the set as complete. The default roles could be stored twice the same way. Now one node at a time adds the defaults, under a short lease in the store; the other waits, then reads the complete set. Ships in the next RC.
Not covered yet: what happens when a listener that is tried fails to bind (retry, exit, or report). That is being decided separately. The two-node lab test is for the Kubernetes lab.
Fixed in #482. Cause: a race on first start. Each start adds any default listeners not yet stored; the second node found 6 of 7 already written (`http` is last) and took the set as complete. The default roles could be stored twice the same way. Now one node at a time adds the defaults, under a short lease in the store; the other waits, then reads the complete set. Ships in the next RC.
Not covered yet: what happens when a listener that *is* tried fails to bind (retry, exit, or report). That is being decided separately. The two-node lab test is for the Kubernetes lab.
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Found in the Kubernetes lab (inbuxa-k8s-lab, spike S4), 27.1-rc.27, PostgreSQL + NATS + S3.
What happened. Straight after first boot (
x:Bootstrap/setagainst an empty PostgreSQL database), two nodes were started at the same moment. Both reached Running. One started every listener excepthttp(8080): its log hasnetwork.listen-startfor smtp, submissions, imaps, pop3s, sieve and https, none forhttp, and nothing on 8080 in/proc/net/tcp6. That node answered nothing on/healthz/liveor the API, while the process stayed up. The same node also logged, at the same second:Restarting that pod brought
httpup. Three more simultaneous two-node starts on the now-initialized store did not reproduce it, so it looks tied to the first start on a fresh store.Why it matters. A listener that fails to start should either be retried or stop the process. Today the node looks alive to anything that checks the process, and only an HTTP probe notices. Under Kubernetes the liveness probe would restart it, but on systemd or plain Docker it would sit there half up.
Done when
Workaround in the chart meanwhile: after first boot, start the first server node alone and wait for readiness; HTTP liveness probe on all server pods.
Fixed in #482. Cause: a race on first start. Each start adds any default listeners not yet stored; the second node found 6 of 7 already written (
httpis last) and took the set as complete. The default roles could be stored twice the same way. Now one node at a time adds the defaults, under a short lease in the store; the other waits, then reads the complete set. Ships in the next RC.Not covered yet: what happens when a listener that is tried fails to bind (retry, exit, or report). That is being decided separately. The two-node lab test is for the Kubernetes lab.