Retry object store transport errors, from a tenth of a second

jcoffey-dev commented

2026-10-08 21:43:48 +00:00

Owner

Backlog Tier 3. In crates/store/src/backend/s3/mod.rs upstream retries only a 5xx response. A transport error from the client (connection reset, timeout, a Garage node mid-restart) returned straight away through map_err, so the blob read or write failed on the first hiccup; and the first retry slept a full second (1 << attempt seconds, capped at 64 s), which on a user-facing read is already a long wait.

A transport error on GET, PUT, the HEAD after PUT, and DELETE now retries like a 5xx, within the same max_retries budget.
429 is retried too.
The waits run 100 ms, 200 ms, 400 ms ... capped at 5 s (retry_wait, unit-tested). With the default retry count the total wait stays bounded in seconds instead of minutes.

No behaviour change on a 2xx or 404. The S3 store test needs an object store container, so this is covered by the unit test for the schedule and by compile/clippy; production is the real test (Garage rf 3, three nodes).

Backlog Tier 3. In crates/store/src/backend/s3/mod.rs upstream retries only a 5xx response. A transport error from the client (connection reset, timeout, a Garage node mid-restart) returned straight away through map_err, so the blob read or write failed on the first hiccup; and the first retry slept a full second (1 << attempt seconds, capped at 64 s), which on a user-facing read is already a long wait. - A transport error on GET, PUT, the HEAD after PUT, and DELETE now retries like a 5xx, within the same max_retries budget. - 429 is retried too. - The waits run 100 ms, 200 ms, 400 ms ... capped at 5 s (retry_wait, unit-tested). With the default retry count the total wait stays bounded in seconds instead of minutes. No behaviour change on a 2xx or 404. The S3 store test needs an object store container, so this is covered by the unit test for the schedule and by compile/clippy; production is the real test (Garage rf 3, three nodes).

jcoffey-dev added 1 commit 2026-10-08 21:43:49 +00:00

Retry object store transport errors, from a tenth of a second

ci / fork-checks (pull_request) Skipped

Details

ci / build (pull_request) Skipped

Details

github/ci (branch) GitHub Actions

Details

ci / github (pull_request) Failing after 9m25s

Details

e864ac2458

The S3 backend retried only a 5xx answer: a dropped connection or a
timeout on the way to the object store failed the read or write at
once, and the first retry waited a whole second. A transport error and
a 429 now retry the same way, and the waits run 100 ms, 200 ms, 400 ms
... capped at 5 s, so a Garage node restarting costs a request tenths
of a second rather than failing it.

jcoffey-dev added 1 commit 2026-10-08 22:31:01 +00:00

Mark the S3 backend as modified

ci / fork-checks (pull_request) Skipped

Details

ci / build (pull_request) Skipped

Details

github/ci (branch) GitHub Actions

Details

ci / github (pull_request) Successful in 10m4s

Details

e61e855ac5

jcoffey-dev merged commit accb61d0d3 into main

2026-10-08 22:42:39 +00:00

jcoffey-dev deleted branch fix/s3-retries

2026-10-08 22:42:39 +00:00

jcoffey-dev referenced this pull request

2026-10-08 22:42:53 +00:00

Release 2026.10.10 #179

Retry object store transport errors, from a tenth of a second #178