Sort and filter by received date on the message cache

jcoffey-dev commented

2026-10-08 19:53:59 +00:00

Owner

Backlog item F from the 2026-10-07 cluster survey. On the cluster every Email/query that sorts by receivedAt (the webmail's mailbox listing, and its queryChanges) ran SELECT docid FROM s_email WHERE accid = $1 AND docid BETWEEN min AND max ORDER BY rcvd DESC with no limit: every message id of the account, ordered, over the wire, on every call. Production's largest account has 29k messages, so that was 29k rows per listing (plan on the leader: 30,830 buffer hits, quicksort 1.7 MB). IMAP SORT by arrival, Thread/get's ordering, and the date filters (before/after, IMAP SINCE/BEFORE/ON/OLDER/YOUNGER) went the same way.

The message cache carries the received date

MessageCache.received_at: u64 (seconds; the IMAP INTERNALDATE, the same value the index stores) and MessagesCache.has_received_at.
A full cache build reads the account's dates from the search index in one query (SearchStore::unsigned_values, PostgreSQL and MySQL, replica-aware) and the metadata of the messages the index task has not reached yet, a batch at a time (Store::get_values from #170).
A refresh keeps the dates of the messages it already had (a received date never changes) and reads the metadata of the new ones. Usually that is one message.
With a search store that is not SQL the dates stay unknown, has_received_at is false, and every caller keeps the upstream path. So single-node SQLite/RocksDB/FoundationDB installs see no change.

What now runs on the cache instead of the store

JMAP Email/query sort by receivedAt (a SearchComparator::SortedSet of ranks built from the cache, so dates past 2106 do not tie), and the before/after filters.
Thread/get's email ordering.
IMAP SORT ARRIVAL, and SEARCH BEFORE/SINCE/ON/OLDER/YOUNGER.

Sorts by other fields (sentAt, from, to, subject, size) still go to the store. For those the PostgreSQL search tables get an (accid, column) index next to each existing single-column index, so the query walks an ordered index for the account instead of sorting its rows. The indexes are created at startup with IF NOT EXISTS; on production's 31k-row s_email that is a few seconds on the first node to restart.

Also fixed on the way: a query with no filter beyond the account id (Thread/get's) fell into the local filter, which skips operators and answered with nothing; it now starts from the mask. Upstream never hit it because that query always carried a store-side comparator.

Not in this PR: pushing the candidate id set into the query (docid = ANY) for the remaining store-side sorts, and the index task building one document per task.

Tests

IMAP suite on PostgreSQL: passes (44 s).
JMAP suite on PostgreSQL: only the pre-existing email/filter-header-name-value failure, same as main.
IMAP suite on SQLite (the None path): passes (44 s).

Cluster numbers follow after a roll.

Backlog item F from the 2026-10-07 cluster survey. On the cluster every Email/query that sorts by `receivedAt` (the webmail's mailbox listing, and its queryChanges) ran `SELECT docid FROM s_email WHERE accid = $1 AND docid BETWEEN min AND max ORDER BY rcvd DESC` with no limit: every message id of the account, ordered, over the wire, on every call. Production's largest account has 29k messages, so that was 29k rows per listing (plan on the leader: 30,830 buffer hits, quicksort 1.7 MB). IMAP SORT by arrival, Thread/get's ordering, and the date filters (`before`/`after`, IMAP `SINCE`/`BEFORE`/`ON`/`OLDER`/`YOUNGER`) went the same way. **The message cache carries the received date** - `MessageCache.received_at: u64` (seconds; the IMAP INTERNALDATE, the same value the index stores) and `MessagesCache.has_received_at`. - A full cache build reads the account's dates from the search index in one query (`SearchStore::unsigned_values`, PostgreSQL and MySQL, replica-aware) and the metadata of the messages the index task has not reached yet, a batch at a time (`Store::get_values` from #170). - A refresh keeps the dates of the messages it already had (a received date never changes) and reads the metadata of the new ones. Usually that is one message. - With a search store that is not SQL the dates stay unknown, `has_received_at` is false, and every caller keeps the upstream path. So single-node SQLite/RocksDB/FoundationDB installs see no change. **What now runs on the cache instead of the store** - JMAP Email/query sort by `receivedAt` (a `SearchComparator::SortedSet` of ranks built from the cache, so dates past 2106 do not tie), and the `before`/`after` filters. - Thread/get's email ordering. - IMAP SORT `ARRIVAL`, and SEARCH `BEFORE`/`SINCE`/`ON`/`OLDER`/`YOUNGER`. Sorts by other fields (sentAt, from, to, subject, size) still go to the store. For those the PostgreSQL search tables get an `(accid, column)` index next to each existing single-column index, so the query walks an ordered index for the account instead of sorting its rows. The indexes are created at startup with `IF NOT EXISTS`; on production's 31k-row `s_email` that is a few seconds on the first node to restart. Also fixed on the way: a query with no filter beyond the account id (Thread/get's) fell into the local filter, which skips operators and answered with nothing; it now starts from the mask. Upstream never hit it because that query always carried a store-side comparator. Not in this PR: pushing the candidate id set into the query (`docid = ANY`) for the remaining store-side sorts, and the index task building one document per task. **Tests** - IMAP suite on PostgreSQL: passes (44 s). - JMAP suite on PostgreSQL: only the pre-existing `email/filter-header-name-value` failure, same as main. - IMAP suite on SQLite (the `None` path): passes (44 s). Cluster numbers follow after a roll.

jcoffey-dev added 2 commits 2026-10-08 19:53:59 +00:00

Sort and filter by received date on the message cache 37da91c0a6

The message cache now carries each message's received date. On an SQL
search store a full cache build reads the account's dates from the
search index in one query and the metadata of the messages the index
task has not reached yet; a refresh keeps the dates of the messages it
had and reads the metadata of the new ones, a batch at a time. With any
other search store the dates stay unknown and nothing changes.

With the dates in the cache, JMAP Email/query and IMAP SORT by received
date, Thread/get's ordering, and the before/after/since/on/older/younger
filters run on the cache. Upstream answered each of those with a query
that pulled every message id of the account from the search store in
order, with no limit, on every call.

The PostgreSQL search tables also get an (account, column) index for
each sorted column, so the sorts that still go to the database walk an
ordered index instead of sorting the account's rows.

Keep the received date whole and answer a mask-only query

ci / fork-checks (pull_request) Skipped

Details

ci / build (pull_request) Skipped

Details

github/ci (branch) GitHub Actions

Details

ci / github (pull_request) Successful in 7m28s

Details

68a67be170

The cache holds the date as u64 and the local sort ranks the messages,
so dates past 2106 no longer tie. A query with no filter beyond the
account used to fall into the local filter, which skips operators and
answered with nothing; it now starts from the mask.

jcoffey-dev merged commit 372a47b560 into main

2026-10-08 20:02:22 +00:00

jcoffey-dev deleted branch perf/local-received-at

2026-10-08 20:02:22 +00:00

jcoffey-dev referenced this pull request

2026-10-08 20:02:43 +00:00

Release 2026.10.9 #173

jcoffey-dev referenced this pull request

2026-10-08 21:12:30 +00:00

Skip the account-led search index on tables without an account column #174

jcoffey-dev referenced this pull request

2026-10-08 21:41:17 +00:00

Ask the DNS blocklists together #177

jcoffey-dev commented

2026-10-09 23:53:14 +00:00

Author

Owner

No after numbers for this one yet. The logs don't record Email/query or SORT timings at the current event levels, so there's nothing to compare. I'd rather time a few queries against a test account with a big mailbox than turn logging up on the cluster.

Sort and filter by received date on the message cache #172