How do I troubleshoot "Live updates reconnecting"? #71

Closed
opened 2026-11-14 19:46:19 +00:00 by oli_is_cool · 10 comments

Hello, I'm running Stalwart v1.0.3 with ihasmail 2026.11.6-g2e7fd9b. Ever since I upgraded Stalwart to 1.x, the "push-status" is perpetually yellow/connecting with a tooltip of "Live updates reconnecting...".

I checked the logs of both containers and the only entry that seems relevant is this one:

[ihasmail] push: no verification for [email protected] within 180s; relay in use

(I redacted my domain)

I found https://docs.ihasmail.org/install/?h=push#push-by-subscription and my configuration in indeed includes PUSH_URL=https://mail.example.com and turning that back off does not improve the situation.

Is there something I have to turn on/off on the Stalwart side?

I checked the Stalwart documentation on the topic and I found this:

A subscription is accepted only for a public https URL. A URL with another scheme, a URL with credentials, and a URL whose host is a private, loopback or otherwise non-public address or name are refused with invalidProperties on url, with a description such as URL must use the https scheme or Destination is a local or reserved IP address.

...but that seems to be for "clients that are not running". Maybe that's what ihasmail was using, and it got stricter in Stalwart 1.x? (I'm running both on the same host, with a private address)

Thanks!

Hello, I'm running Stalwart `v1.0.3` with ihasmail `2026.11.6-g2e7fd9b`. Ever since I upgraded Stalwart to 1.x, the "push-status" is perpetually yellow/connecting with a tooltip of "Live updates reconnecting...". I checked the logs of both containers and the only entry that seems relevant is this one: ```text [ihasmail] push: no verification for [email protected] within 180s; relay in use ``` (I redacted my domain) I found https://docs.ihasmail.org/install/?h=push#push-by-subscription and my configuration in indeed includes `PUSH_URL=https://mail.example.com` and turning that back off does not improve the situation. Is there something I have to turn on/off on the Stalwart side? I checked the Stalwart documentation on the topic and I found this: > A subscription is accepted only for a public `https` URL. A URL with another scheme, a URL with credentials, and a URL whose host is a private, loopback or otherwise non-public address or name are refused with `invalidProperties` on `url`, with a description such as `URL must use the https scheme` or `Destination is a local or reserved IP address`. ...but that seems to be for "clients that are not running". Maybe that's what ihasmail was using, and it got stricter in Stalwart 1.x? (I'm running both on the same host, with a private address) Thanks!
Author

I cranked the Stalwart logging level to DEBUG and, what do you know?

2026-11-14T19:48:06Z DEBUG Push subscription error (push-subscription.error) details = "HTTP POST failed", url = "https://mail.example.com/api/push/G2O6l807imNRE1hE83PRoNXBhH7uIzhEkzp06xootSM", reason = "error sending request for url (https://mail.example.com/api/push/G2O6l807imNRE1hE83PRoNXBhH7uIzhEkzp06xootSM): client error (Connect): dns error: error resolving DNS: Destination is a local or reserved IP address"

Another thing I noticed is if I try to visit https://mail.example.com/api/events?types=*&closeafter=no&ping=30 with my browser, I get:

{"error":"cross_site_request"}

Thanks!

I cranked the Stalwart logging level to `DEBUG` and, what do you know? `2026-11-14T19:48:06Z DEBUG Push subscription error (push-subscription.error) details = "HTTP POST failed", url = "https://mail.example.com/api/push/G2O6l807imNRE1hE83PRoNXBhH7uIzhEkzp06xootSM", reason = "error sending request for url (https://mail.example.com/api/push/G2O6l807imNRE1hE83PRoNXBhH7uIzhEkzp06xootSM): client error (Connect): dns error: error resolving DNS: Destination is a local or reserved IP address"` Another thing I noticed is if I try to visit https://mail.example.com/api/events?types=*&closeafter=no&ping=30 with my browser, I get: ```json {"error":"cross_site_request"} ``` Thanks!
Owner

Thanks @oli_is_cool, that debug line helps a lot.

I can't reproduce the stuck dot here. Against a fresh Stalwart 1.0.3, the event stream connects normally. Your log does point at the likely cause, though. Stalwart 1.x won't deliver push to a private or reserved address, so with PUSH_URL resolving to your private IP the subscription never verifies, and ihasmail falls back to relaying Stalwart's event stream to each tab (that's the "relay in use" line). On that path Stalwart sends nothing, not even headers, until its first ping 30 seconds in, and ihasmail waits for that before answering the browser. So the dot stays yellow for at least 30 seconds on every connect. If something between your browser and ihasmail (a reverse proxy, a tunnel) gives up on a response that slow, the browser retries and never gets to green.

The cross_site_request reply is expected and unrelated. ihasmail refuses API requests that the browser marks as coming from another site, which is what happens when you follow a link to /api/events from somewhere else, like this issue. The app's own requests are same-origin and don't hit that check.

Could you send a bit more?

  • what sits in front of ihasmail (nginx, Caddy, Traefik, Cloudflare, ...) and any read or response timeouts it has
  • what /api/events shows in the browser's dev tools Network tab: the status, and whether it stays pending or fails after a set time
  • whether the dot ever turns green, even briefly, with PUSH_URL unset

On a private address you lose nothing by leaving PUSH_URL unset, since Stalwart 1.x can't reach it anyway.

Thanks @oli_is_cool, that debug line helps a lot. I can't reproduce the stuck dot here. Against a fresh Stalwart 1.0.3, the event stream connects normally. Your log does point at the likely cause, though. Stalwart 1.x won't deliver push to a private or reserved address, so with PUSH_URL resolving to your private IP the subscription never verifies, and ihasmail falls back to relaying Stalwart's event stream to each tab (that's the "relay in use" line). On that path Stalwart sends nothing, not even headers, until its first ping 30 seconds in, and ihasmail waits for that before answering the browser. So the dot stays yellow for at least 30 seconds on every connect. If something between your browser and ihasmail (a reverse proxy, a tunnel) gives up on a response that slow, the browser retries and never gets to green. The `cross_site_request` reply is expected and unrelated. ihasmail refuses API requests that the browser marks as coming from another site, which is what happens when you follow a link to /api/events from somewhere else, like this issue. The app's own requests are same-origin and don't hit that check. Could you send a bit more? - what sits in front of ihasmail (nginx, Caddy, Traefik, Cloudflare, ...) and any read or response timeouts it has - what /api/events shows in the browser's dev tools Network tab: the status, and whether it stays pending or fails after a set time - whether the dot ever turns green, even briefly, with PUSH_URL unset On a private address you lose nothing by leaving PUSH_URL unset, since Stalwart 1.x can't reach it anyway.
Author

I'm kind of answering my own question here, but I posted An admin setting or allow-list for internal push targets - Feature Ideas - Stalwart Support to allow me to restore notifications in ihasmail when co-located on an internal-only server.

I'm kind of answering my own question here, but I posted [An admin setting or allow-list for internal push targets - Feature Ideas - Stalwart Support](https://support.stalw.art/t/an-admin-setting-or-allow-list-for-internal-push-targets/2368) to allow me to restore notifications in ihasmail when co-located on an internal-only server.
Author

Sorry I missed your reply earlier, I'm not getting notifications from your Gitea instance very often.

Here's the relevant portion of my nginx configuration:

server {
    server_name hasmail.example.com ihasmail.example.com mail.example.com;
    include /etc/nginx/ssl.conf;

    # https://docs.ihasmail.org/install/#putting-tls-in-front
    client_max_body_size 60m;

    location / {
        proxy_pass http://127.0.0.1:3080;
        proxy_http_version 1.1;
        proxy_set_header Host $host;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        # Keep live updates flowing
        proxy_buffering off;
        proxy_read_timeout 3600s;
    }
}

If there's something else you're looking for in my nginx configuration, please let me know.

As for /api/events, from what I can tell, it just sits there until I hard-refresh my browser, at which point it's aborted. I haven't seen a response come back.

I'm pretty sure I haven't seen the dot go green since the update, either, regardless of whether we're in subscribe or relay mode.

I can fiddle with more settings, if you'd like. Just tell me which!

Thanks!

Sorry I missed your reply earlier, I'm not getting notifications from your Gitea instance very often. Here's the relevant portion of my `nginx` configuration: ```text server { server_name hasmail.example.com ihasmail.example.com mail.example.com; include /etc/nginx/ssl.conf; # https://docs.ihasmail.org/install/#putting-tls-in-front client_max_body_size 60m; location / { proxy_pass http://127.0.0.1:3080; proxy_http_version 1.1; proxy_set_header Host $host; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_set_header X-Forwarded-Proto $scheme; # Keep live updates flowing proxy_buffering off; proxy_read_timeout 3600s; } } ``` If there's something else you're looking for in my `nginx` configuration, please let me know. As for `/api/events`, from what I can tell, it just sits there until I hard-refresh my browser, at which point it's aborted. I haven't seen a response come back. I'm pretty sure I haven't seen the dot go green since the update, either, regardless of whether we're in `subscribe` or `relay` mode. I can fiddle with more settings, if you'd like. Just tell me which! Thanks!
Author

I also tried sending myself an e-mail and ihasmail didn't "notice" until I gave its browser tab focus again (after letting it sit fora while), at which point it looks like it triggered a few requests to its server and quickly updated the unread count and message list.

I also tried sending myself an e-mail and ihasmail didn't "notice" until I gave its browser tab focus again (after letting it sit fora while), at which point it looks like it triggered a few requests to its server and quickly updated the unread count and message list.
Owner

Thanks, that helps. Your nginx config looks right, so the proxy in front of ihasmail isn't the problem.

If /api/events never gets a response at all, ihasmail is waiting on Stalwart's event stream and getting nothing back. Stalwart should send a ping within 30 seconds. Could you try this from the machine ihasmail runs on (or inside its container, if STALWART_URL uses a container name), with the same address ihasmail has in STALWART_URL?

curl -N -u '[email protected]:password' "$STALWART_URL/jmap/eventsource/?types=*&closeafter=no&ping=30"

You should see event: ping within about 30 seconds. If it just hangs, whatever sits between ihasmail and Stalwart is holding the stream, and that's where to look next. What is STALWART_URL set to, and is there a proxy in front of Stalwart as well?

The new mail only showing up on focus fits that: with no live stream, ihasmail only catches up when the tab gets focus again.

Thanks, that helps. Your nginx config looks right, so the proxy in front of ihasmail isn't the problem. If /api/events never gets a response at all, ihasmail is waiting on Stalwart's event stream and getting nothing back. Stalwart should send a ping within 30 seconds. Could you try this from the machine ihasmail runs on (or inside its container, if STALWART_URL uses a container name), with the same address ihasmail has in STALWART_URL? ``` curl -N -u '[email protected]:password' "$STALWART_URL/jmap/eventsource/?types=*&closeafter=no&ping=30" ``` You should see `event: ping` within about 30 seconds. If it just hangs, whatever sits between ihasmail and Stalwart is holding the stream, and that's where to look next. What is STALWART_URL set to, and is there a proxy in front of Stalwart as well? The new mail only showing up on focus fits that: with no live stream, ihasmail only catches up when the tab gets focus again.
Owner

v2026.11.15 is out with two small fixes from this issue. ihasmail no longer makes the browser wait on Stalwart before answering /api/events, and the dot only turns green once something actually arrives on the stream.

That won't fix your setup by itself, though: if Stalwart's event stream never answers ihasmail, the dot will now just stay yellow, which is at least honest. The curl test above is still the thing that will tell us where the stream is getting stuck.

v2026.11.15 is out with two small fixes from this issue. ihasmail no longer makes the browser wait on Stalwart before answering /api/events, and the dot only turns green once something actually arrives on the stream. That won't fix your setup by itself, though: if Stalwart's event stream never answers ihasmail, the dot will now just stay yellow, which is at least honest. The curl test above is still the thing that will tell us where the stream is getting stuck.
Author

Thank you for your help on this! The Stalwart /jmap/eventsource endpoint was appearing stuck because nginx was apparently misconfigured to buffer the output! Once I applied the same configuration ihasmail has:

        # Keep live updates flowing
        proxy_buffering off;
        proxy_read_timeout 3600s;

...after 30 seconds, the dot turned green!

I also tried upgrading ihasmail and now I see in my browser's devtools that the /api/events endpoint receives : relay soon after connecting and the dot turned green as soon as an event came in.

Thanks again for your help!

Thank you for your help on this! The Stalwart `/jmap/eventsource` endpoint was appearing stuck because `nginx` was apparently misconfigured to buffer the output! Once I applied the same configuration ihasmail has: ``` # Keep live updates flowing proxy_buffering off; proxy_read_timeout 3600s; ``` ...after 30 seconds, the dot turned green! I also tried upgrading ihasmail and now I see in my browser's devtools that the `/api/events` endpoint receives `: relay` soon after connecting and the dot turned green as soon as an event came in. Thanks again for your help!
Author

Oh and for the record, the ihasmail container does NOT include curl. I was able to use wget but the user experience wasn't as good as curl's.

Oh and for the record, the ihasmail container does NOT include `curl`. I was able to use `wget` but the user experience wasn't as good as curl's.
Owner

Sorry, I forgot when I suggested it: we strip most tools out of the images for security, so there's no curl inside (only busybox wget, which the healthcheck uses). For a test like this, a throwaway curl container on the same Docker network does the same job:

docker run --rm -it --network <ihasmail's network> curlimages/curl \
  -N -u '[email protected]:password' "$STALWART_URL/jmap/eventsource/?types=*&closeafter=no&ping=30"
Sorry, I forgot when I suggested it: we strip most tools out of the images for security, so there's no curl inside (only busybox wget, which the healthcheck uses). For a test like this, a throwaway curl container on the same Docker network does the same job: ``` docker run --rm -it --network <ihasmail's network> curlimages/curl \ -N -u '[email protected]:password' "$STALWART_URL/jmap/eventsource/?types=*&closeafter=no&ping=30" ```
Sign in to join this conversation.
2 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: coffey-labs/ihasmail#71