Broadcast subscriber: fix the inverted subscribe retry backoff #32

Merged

jcoffey-dev merged 1 commits from fix/subscriber-backoff into main

2026-09-24 14:26:53 +00:00

jcoffey-dev commented

2026-09-24 14:02:13 +00:00

Owner

Problem

crates/services/src/broadcast/subscriber.rs computed the wait between failed subscribe attempts as 1 << retry_count.max(6) seconds. max(6) acts as a floor rather than a cap, so:

the first retry waited 64 s instead of 1 s;
each later retry doubled with no upper bound (128 s, 256 s, ...);
after enough failures the shift overflows.

A 3-node cluster rehearsal (PostgreSQL + NATS) hit this when NATS dropped out.

Change

New subscribe_retry_delay(retry_count): 1 s, 2 s, 4 s ... capped at 64 s (1u64 << retry_count.min(6)).
retry_count is a u32 and uses saturating_add.

Tests

cargo test -p services --lib subscribe_retry: new unit test subscribe_retry_backoff_grows_then_caps checks the schedule [1, 2, 4, 8, 16, 32, 64, 64, 64, 64] and u32::MAX → 64 s. Passes.

## Problem `crates/services/src/broadcast/subscriber.rs` computed the wait between failed subscribe attempts as `1 << retry_count.max(6)` seconds. `max(6)` acts as a floor rather than a cap, so: - the first retry waited 64 s instead of 1 s; - each later retry doubled with no upper bound (128 s, 256 s, ...); - after enough failures the shift overflows. A 3-node cluster rehearsal (PostgreSQL + NATS) hit this when NATS dropped out. ## Change - New `subscribe_retry_delay(retry_count)`: 1 s, 2 s, 4 s ... capped at 64 s (`1u64 << retry_count.min(6)`). - `retry_count` is a `u32` and uses `saturating_add`. ## Tests - `cargo test -p services --lib subscribe_retry`: new unit test `subscribe_retry_backoff_grows_then_caps` checks the schedule `[1, 2, 4, 8, 16, 32, 64, 64, 64, 64]` and `u32::MAX` → 64 s. Passes.

jcoffey-dev added 1 commit 2026-09-24 14:02:14 +00:00

Broadcast subscriber: fix the inverted subscribe retry backoff

ci / fork-checks (pull_request) Successful in 56s

Details

ci / build (pull_request) Successful in 5m14s

Details

ca3abf40f0

The broadcast subscriber waited 1 << retry_count.max(6) seconds between
failed subscribe attempts. max(6) turns the cap into a floor: the first
retry waited 64 s instead of 1 s, and each later one doubled without a
bound (and would overflow the shift after enough failures).

The delay now comes from subscribe_retry_delay(), 1 s, 2 s, 4 s ... capped
at 64 s, and the retry counter saturates. A unit test pins the schedule
and the top of the range.

jcoffey-dev merged commit 57d1c5b074 into main

2026-09-24 14:26:53 +00:00

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: inbuxa/inbuxa-server#32