Report reschedules keep the task queue readable #46
Assignees
jcoffey-dev (John Coffey)
Clear assignees
jcoffey-dev (John Coffey)
No Assignees
Notifications
Due Date
No due date set.
Dependencies
No dependencies set.
Reference: inbuxa/inbuxa-server#46
Reference in New Issue
Block a user
Setting
deliverAton an internal DMARC or TLS report (x:DmarcInternalReport/set,x:TlsInternalReport/set) wedged the task queue on every node. This is in production (2026.9.24.3).Cause
InternalReportIndex::reschedule_ops(crates/smtp/src/reporting/index.rs) wrote the new task queue row withobject_id.serialize(): the report's object type (0x21 DmarcInternalReport, 0x6e TlsInternalReport), not the task typeschedule_task_with_idwrites (7 DmarcReport, 8 TlsReport). It also left the task row at its old due, sox:Task/getshowed a stale due.process_tasks(crates/services/src/task_manager/manager.rs) turned that row intostore.data-corruptionwith a?inside the iterate closure, which ended the whole scan: "Failed to iterate over task queue", and every task due after the row stayed stuck, cluster-wide.Change
index.rs,jmap/.../mapping/report.rs): the new row goes throughschedule_task_with_id, so it carries the task type and the task row gets the new due. The old row is removed under the key the task is actually queued at (the task's own due, which differs fromdeliverAtonce the task has been retried), plus any row an earlier reschedule left atdeliverAt. The set locks the report's task while it moves it, asx:Task/setdoes, refuses while the report is being sent, releases its locks however the request ends (a held lock is renewed, so a leaked one would block the task for good), and wakes the task manager.manager.rs): a row whose type can't be read is logged (store.data-corruption, "Unreadable task queue row skipped", with id, due, key and value) and skipped; the scan carries on. The row is then repaired from its task row: rewritten with the task's type, or removed if no task is behind it. A report task's row that holds the report's object type is exactly what the old reschedule wrote, so the task is moved to that row's time, as the admin asked, and its old row is removed. The task manager looks again once a row is repaired; a repair that fails is logged and tried on the next scan, without a busy loop.x:Task/queryby type (jmap/.../mapping/task.rs) skips an unreadable row instead of failing.Existing broken rows: production may already hold one if anyone rescheduled a report. Nothing needs doing by hand. When the row comes due, the scanner logs it, rewrites it as the DmarcReport/TlsReport task at the rescheduled time, and the report goes out. Tasks due after it run either way. Until the row comes due,
x:Task/getstill shows the old due; x:Task/query no longer fails on it.Tests
tests/src/smtp/reporting/reschedule.rs(new), single node, task manager running:x:Task/getshows the new due, and the queue row holds the task type. Both reports go out at the new time, and a task due after them still runs.x:Task/queryfiltered by type succeeds with an unreadable row in the queue.On main it fails:
x:Task/getshows the old due. With that assertion removed, main logs "Failed to iterate over task queue" and neither report nor the later task ever runs ("tasks [...] never ran").Also run:
smtp::on RocksDB (75 pass;lmtp_deliveryfailed once in the full run and passed twice alone; it's timing and doesn't touch this code), andcluster::live_roles,cluster::task_roles,cluster::stress,store::task_lockson PostgreSQL.cluster::broadcastneedsCOORDINATORand wasn't run.Release
The fix is self-contained. Cherry-picked onto 2026.9.24.3 (
e223f7d), onlymanager.rsconflicts, in one hunk: the release has the older inline rolematchwhere main callstask_enabled. Keep the release'smatchand take the new type decoding above it. Every API the change calls exists ate223f7d.publish.ymlonly publishes tags that are on main, so a release still has to be cut from main (by tag, after this merges).Setting deliverAt on an internal DMARC or TLS report wrote the new task queue row with the report's object type (0x21, 0x6e) instead of the task type (7, 8), and left the task row at its old due. The task manager's scan failed on that row with store.data-corruption ("Failed to iterate over task queue"), and because the error ended the whole scan, every task due after the row stopped running on every node. - reschedule_ops writes the new queue row through schedule_task_with_id, so it carries the task type and the task row gets the new due. It removes the row the task is actually queued under (the task's due, which differs from deliverAt once the task has been retried) and any row an earlier reschedule left at deliverAt. - x:DmarcInternalReport/set and x:TlsInternalReport/set lock the report's task while they move it, as x:Task/set does, refuse while the report is being sent, release the locks however the request ends, and wake the task manager. - The task manager logs a queue row it can't read (id, due, key, value) and skips it instead of ending the scan. It then repairs the row from its task: the row is rewritten with the task's type, and a row with no task behind it is removed. A row holding a report's object type for a report task is what the old reschedule wrote: the task is moved to that row's time, as the reschedule intended, and its old queue row is removed. Stores that already hold such a row recover on their own once it comes due. - x:Task/query with a type filter skips an unreadable row instead of failing. Test: smtp::reporting::reschedule (RocksDB and PostgreSQL). It fails on main: x:Task/get shows the old due, and with that check removed, neither report nor a later task ever runs.