Report reschedules keep the task queue readable

Setting deliverAt on an internal DMARC or TLS report wrote the new task
queue row with the report's object type (0x21, 0x6e) instead of the task
type (7, 8), and left the task row at its old due. The task manager's scan
failed on that row with store.data-corruption ("Failed to iterate over task
queue"), and because the error ended the whole scan, every task due after
the row stopped running on every node.

- reschedule_ops writes the new queue row through schedule_task_with_id, so
  it carries the task type and the task row gets the new due. It removes
  the row the task is actually queued under (the task's due, which differs
  from deliverAt once the task has been retried) and any row an earlier
  reschedule left at deliverAt.
- x:DmarcInternalReport/set and x:TlsInternalReport/set lock the report's
  task while they move it, as x:Task/set does, refuse while the report is
  being sent, release the locks however the request ends, and wake the task
  manager.
- The task manager logs a queue row it can't read (id, due, key, value) and
  skips it instead of ending the scan. It then repairs the row from its task:
  the row is rewritten with the task's type, and a row with no task behind
  it is removed. A row holding a report's object type for a report task is
  what the old reschedule wrote: the task is moved to that row's time, as
  the reschedule intended, and its old queue row is removed. Stores that
  already hold such a row recover on their own once it comes due.
- x:Task/query with a type filter skips an unreadable row instead of
  failing.

Test: smtp::reporting::reschedule (RocksDB and PostgreSQL). It fails on
main: x:Task/get shows the old due, and with that check removed, neither
report nor a later task ever runs.

(cherry picked from commit 1a7859a8cc)
This commit is contained in:
2026-09-24 18:35:50 -07:00
parent 4e6c8b916e
commit 0b4aa9c084
6 changed files with 602 additions and 33 deletions
+29 -13
View File
@@ -2,6 +2,8 @@
* SPDX-FileCopyrightText: 2020 Stalwart Labs LLC <[email protected]>
*
* SPDX-License-Identifier: AGPL-3.0-only OR LicenseRef-SEL
*
* Modified by Coffey Labs in 2026 for INBUXA.
*/
use registry::{
@@ -40,35 +42,49 @@ pub trait InternalReportIndex: ObjectImpl {
fn primary_key(&self) -> ValueClass;
/// Moves the report's delivery, and its queued task, to `at`.
///
/// inbuxa: the new queue row carries the task's type, as
/// `schedule_task_with_id` writes it, and the task row gets the new due
/// too. `queued` is the task as stored: its due, not the report's
/// `deliverAt`, is the queue row that exists (they differ once the task
/// has been retried).
fn reschedule_ops(
&mut self,
batch: &mut BatchBuilder,
item_id: u64,
revision: u64,
at: UTCDateTime,
queued: Option<&Task>,
) {
let current_deliver_at = self.deliver_at();
let current_due = current_deliver_at.timestamp() as u64;
let queued_due = queued.map_or(current_due, |task| task.due_timestamp());
let new_due = at.timestamp() as u64;
if current_deliver_at != at {
if current_deliver_at != at || queued_due != new_due {
let object = Self::OBJECT;
let object_id = object.to_id();
let key = ValueClass::Registry(RegistryClass::Item { object_id, item_id });
self.set_deliver_at(at);
batch
.assert_value(key.clone(), AssertValue::Hash(revision))
.clear(ValueClass::TaskQueue(TaskQueueClass::Due {
batch.assert_value(key.clone(), AssertValue::Hash(revision));
if queued_due != new_due {
batch.clear(ValueClass::TaskQueue(TaskQueueClass::Due {
id: item_id,
due: current_deliver_at.timestamp() as u64,
}))
.set(
ValueClass::TaskQueue(TaskQueueClass::Due {
id: item_id,
due: at.timestamp() as u64,
}),
object_id.serialize(),
)
due: queued_due,
}));
}
// A row an earlier reschedule left at the report's deliverAt
if current_due != new_due && current_due != queued_due {
batch.clear(ValueClass::TaskQueue(TaskQueueClass::Due {
id: item_id,
due: current_due,
}));
}
batch
.schedule_task_with_id(item_id, self.task(item_id))
.set(key, self.to_pickled_vec());
}
}