jcoffey-dev
5dde9793eb
The report scheduler dropped DMARC and TLS events on a node whose role lacks outboundMta (upstream never started it there, so they sat in a channel nobody read). Mail received on a front node therefore never reached an aggregate report, which is meant to cover all of a domain's inbound mail, whichever node received it. In rehearsal, five messages received on port 25 on a front node were missing from every report. - The report scheduler records on every node. Recording is a store write the nodes already share, so it needs nothing from the outbound MTA. Building and sending a report (the DmarcReport and TlsReport tasks) stay with outboundMta nodes, as the task manager already enforces. - More nodes now append to one report at once. Appends already guard the report's versioned primary key; a write that loses now retries up to ten times after a short random pause, not three times at once. - The node sending a report deletes it only if it is unchanged since it was read, and reads it again otherwise, so a record another node appends meanwhile goes out with the report instead of being deleted unsent. Test: cluster::front_reports (PostgreSQL and MySQL). A front node's results appear in the report the MTA node sends, alongside eight appended at once from both nodes, and the front node never runs the report task. It fails on main: the front node's results are never recorded.
46 lines
1.6 KiB
Rust
46 lines
1.6 KiB
Rust
/*
|
|
* SPDX-FileCopyrightText: 2026 Coffey Labs
|
|
*
|
|
* SPDX-License-Identifier: AGPL-3.0-only
|
|
*/
|
|
|
|
//! inbuxa: internal DMARC and TLS reports are shared by every node. Any node
|
|
//! that receives mail appends to them, so several nodes can write one report
|
|
//! at once, and the node that sends it may do so while another is appending.
|
|
//! Appends already guard the report's versioned primary key and retry when
|
|
//! another writer got there first; these helpers give those retries room and
|
|
//! let the sender delete exactly the report it read.
|
|
|
|
use rand::RngExt;
|
|
use std::time::Duration;
|
|
use store::{Deserialize, xxhash_rust::xxh3::xxh3_64};
|
|
|
|
/// How many times a report write that lost to another writer is retried.
|
|
/// Upstream retried three times, when only outbound MTA nodes wrote.
|
|
pub(crate) const MAX_WRITE_RETRIES: u32 = 10;
|
|
|
|
/// A short random pause, longer on each attempt, before retrying a report
|
|
/// write that lost to another node, so the writers spread out instead of
|
|
/// colliding again.
|
|
pub(crate) async fn write_retry_pause(attempt: u32) {
|
|
let ms = rand::rng().random_range(5..=25u64) * u64::from(attempt.max(1));
|
|
tokio::time::sleep(Duration::from_millis(ms)).await;
|
|
}
|
|
|
|
/// A stored value with the hash of the bytes it was read from, for
|
|
/// `AssertValue::Hash`: a write asserting it fails if anyone changed the
|
|
/// value since.
|
|
pub(crate) struct Revisioned<T> {
|
|
pub revision: u64,
|
|
pub value: T,
|
|
}
|
|
|
|
impl<T: Deserialize> Deserialize for Revisioned<T> {
|
|
fn deserialize(bytes: &[u8]) -> trc::Result<Self> {
|
|
Ok(Revisioned {
|
|
revision: xxh3_64(bytes),
|
|
value: T::deserialize(bytes)?,
|
|
})
|
|
}
|
|
}
|