Cache message blobs on each node

jcoffey-dev commented

2026-10-08 21:22:12 +00:00

Owner

Backlog item G from the 2026-10-07 cluster survey. Upstream has no blob cache: a message's blob is fetched from the blob store on every read. On the cluster that is a round trip to Garage for the spool blob straight back at delivery, for the same message when an IMAP client asks for BODYSTRUCTURE and then BODY[], for every outbound retry, and for the index task right after ingest.

What this adds

common/src/storage/blob_cache.rs: a per-node BlobCache on utils::cache::Cache (quick_cache, weighted), keyed by the 32-byte content hash. INBUXA_BLOB_CACHE sets the capacity (64M default, 0 turns it off, 512K/1G/bytes accepted). A blob above a sixteenth of the capacity is not cached, so one large attachment cannot evict everything.
Server::get_message_blob(hash, range): cache hit, or one whole-blob fetch that fills the cache and serves the range (the store never read a range anyway). Server::cache_message_blob fills it on write.
Fed on write at the three upload paths (JMAP blob upload, temporary blobs, the SMTP spool) and on read.
Readers switched: IMAP FETCH, POP3 RETR/TOP, JMAP Email/get and blob sections, the index task, outbound delivery, DSNs, email push, holds, the journal.

What stays out of it: anything read by a named key that is rewritten in place (the spam classifier model and trainer data, application bundles). Those keys still go through blob_store().get_blob(); only get_message_blob callers use the cache, and it refuses keys that are not 32 bytes. Content hashes never change meaning, so there is no invalidation: a deleted blob is simply never asked for again.

Tests: three unit tests in the module (size parsing, the off switch, hash-only keys and the per-item cap). IMAP and JMAP suites on PostgreSQL: SUITES

No registry setting yet; an environment variable matches how the role is set. Cluster numbers follow after a roll.

Backlog item G from the 2026-10-07 cluster survey. Upstream has no blob cache: a message's blob is fetched from the blob store on every read. On the cluster that is a round trip to Garage for the spool blob straight back at delivery, for the same message when an IMAP client asks for BODYSTRUCTURE and then BODY[], for every outbound retry, and for the index task right after ingest. **What this adds** - `common/src/storage/blob_cache.rs`: a per-node `BlobCache` on `utils::cache::Cache` (quick_cache, weighted), keyed by the 32-byte content hash. `INBUXA_BLOB_CACHE` sets the capacity (`64M` default, `0` turns it off, `512K`/`1G`/bytes accepted). A blob above a sixteenth of the capacity is not cached, so one large attachment cannot evict everything. - `Server::get_message_blob(hash, range)`: cache hit, or one whole-blob fetch that fills the cache and serves the range (the store never read a range anyway). `Server::cache_message_blob` fills it on write. - Fed on write at the three upload paths (JMAP blob upload, temporary blobs, the SMTP spool) and on read. - Readers switched: IMAP FETCH, POP3 RETR/TOP, JMAP Email/get and blob sections, the index task, outbound delivery, DSNs, email push, holds, the journal. **What stays out of it**: anything read by a named key that is rewritten in place (the spam classifier model and trainer data, application bundles). Those keys still go through `blob_store().get_blob()`; only `get_message_blob` callers use the cache, and it refuses keys that are not 32 bytes. Content hashes never change meaning, so there is no invalidation: a deleted blob is simply never asked for again. **Tests**: three unit tests in the module (size parsing, the off switch, hash-only keys and the per-item cap). IMAP and JMAP suites on PostgreSQL: SUITES No registry setting yet; an environment variable matches how the role is set. Cluster numbers follow after a roll.

jcoffey-dev added 1 commit 2026-10-08 21:22:12 +00:00

Cache message blobs on each node

ci / fork-checks (pull_request) Skipped

Details

ci / build (pull_request) Skipped

Details

github/ci (branch) GitHub Actions

Details

ci / github (pull_request) Successful in 8m26s

Details

b9d908bda2

A per-node cache of message blobs keyed by content hash, 64 MiB by
default (INBUXA_BLOB_CACHE sets it; 0 turns it off), fed on write and
on read. Upstream fetched a message's blob from the blob store every
time: the spool blob straight back for delivery, the same message for
BODYSTRUCTURE and then BODY[], every outbound retry; on the cluster
each is a round trip to the object store.

Server::get_message_blob serves the message read paths (IMAP FETCH,
POP3, JMAP Email/get and blob sections, the index task, outbound
delivery, DSNs, push, holds and the journal). Named keys that are
rewritten in place keep using the blob store directly. A blob above a
sixteenth of the capacity is not cached.

jcoffey-dev merged commit ff706c193b into main

2026-10-08 21:32:38 +00:00

jcoffey-dev deleted branch perf/blob-cache

2026-10-08 21:32:38 +00:00

jcoffey-dev commented

2026-10-08 21:34:42 +00:00

Author

Owner

Suites on PostgreSQL for b9d908b: IMAP passes (44 s); JMAP shows only the pre-existing email/filter-header-name-value compliance failure, same as main.

Suites on PostgreSQL for b9d908b: IMAP passes (44 s); JMAP shows only the pre-existing email/filter-header-name-value compliance failure, same as main.

jcoffey-dev referenced this pull request

2026-10-08 22:42:53 +00:00

Release 2026.10.10 #179

jcoffey-dev commented

2026-10-09 23:53:14 +00:00

Author

Owner

Garage counters a day after the roll, as deltas since the pre-roll snapshot: host1 did 102 gets for 154 puts (0.7 per put, was 1.2), host3 16 for 8 (2, was 9.3), host2 499 for 20 (25, was 27). So the cache is cutting re-fetches on host1 and host3, and host2's reads look like people actually reading mail.

One odd thing: host2's HeadObject count went up about 31k since yesterday, so that isn't just an old burst. Looking into it separately.

Garage counters a day after the roll, as deltas since the pre-roll snapshot: host1 did 102 gets for 154 puts (0.7 per put, was 1.2), host3 16 for 8 (2, was 9.3), host2 499 for 20 (25, was 27). So the cache is cutting re-fetches on host1 and host3, and host2's reads look like people actually reading mail. One odd thing: host2's HeadObject count went up about 31k since yesterday, so that isn't just an old burst. Looking into it separately.

Cache message blobs on each node #175