Deliverability check: each node asks what the internet sees of it
ci / fork-checks (pull_request) Skipped
ci / build (pull_request) Skipped
github/ci (branch) GitHub Actions
ci / github (pull_request) Successful in 7m6s

Deliverability spec (inbuxa-drafts specs/deliverability.md), the server
side. Every node that sends mail checks itself once a day, at its own
minute in the first hour (UTC), and when an administrator asks:

- its outgoing addresses (the connection strategy's, or what its EHLO
  name resolves to), their reverse DNS and whether it resolves back,
  and nine blocklists, read by each list's own codes so a refused
  query is never taken for a listing (DL-1 to DL-6);
- for every domain: SPF for each address, each DKIM key (by signing a
  message that's never sent and verifying it as a receiver would),
  DMARC, the MTA-STS policy against the MX, TLS reporting, and the
  domain blocklists (DL-7 to DL-12);
- whether it holds a certificate for its EHLO and MX names (DL-13).

It keeps one report per node, facts only; the console grades them.

- inbuxa:DeliverabilityReport: /get, and a create that asks every node
  to check now, broadcast as DeliverabilityCheck (DL-15). A tenant
  administrator gets their own domains only (DL-20).
- inbuxa:DeliverabilitySettings: which built-in lists are left out, and
  the lists themselves (DL-6).
- sysDeliverabilityGet, sysDeliverabilityUpdate, sysDeliverabilityCheck;
  a tenant ceiling always turns the last two off.
This commit is contained in:
jcoffey-dev committed 2026-10-05 16:17:33 -07:00
1 parent f791c78d17
commit a24ed3b60a
34 files changed
+2576 -5

No files matched your search

+282
View File
@@ -0,0 +1,282 @@
/*
* SPDX-FileCopyrightText: 2026 Coffey Labs
*
* SPDX-License-Identifier: AGPL-3.0-only
*/
//! The blocklists a node asks about itself (deliverability spec, DL-6), and
//! how to read each one's answer.
//!
//! A list answers with an address in 127.0.0.0/8. Each list says which of
//! those mean "listed" and which mean "I won't answer you": Spamhaus, for
//! one, answers `127.255.255.254` to a query that came through a public
//! resolver. A refusal is never read as a listing (DL-4).
use std::net::{IpAddr, Ipv4Addr};
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum Scope {
/// Looked up by the reversed address: `2.0.0.127.zen.spamhaus.org`.
Ip,
/// Looked up by name: `example.org.dbl.spamhaus.org`.
Domain,
}
#[derive(Debug, Clone, Copy)]
pub struct BlockList {
/// What the page and the settings call it.
pub name: &'static str,
pub zone: &'static str,
pub scope: Scope,
/// Where an administrator looks the address up and asks for removal.
pub lookup: &'static str,
/// Something the page says beside the list.
pub note: Option<&'static str>,
read: fn(Ipv4Addr) -> Answer,
}
/// What a list's answer means.
#[derive(Debug, Clone, PartialEq, Eq)]
pub enum Answer {
Listed(&'static str),
/// The list won't answer this resolver, or not now.
Refused(&'static str),
/// A code the list doesn't define: neither listed nor clean.
Unknown,
}
impl BlockList {
pub fn read(&self, answer: Ipv4Addr) -> Answer {
(self.read)(answer)
}
/// The name to look up for `subject`, or None when the subject doesn't
/// suit the list (a domain on an IP list, or an IPv6 address: none of
/// these lists publish IPv6 zones worth asking).
pub fn query(&self, subject: &Subject<'_>) -> Option<String> {
match (self.scope, subject) {
(Scope::Ip, Subject::Ip(IpAddr::V4(ip))) => {
let [a, b, c, d] = ip.octets();
Some(format!("{d}.{c}.{b}.{a}.{}.", self.zone))
}
(Scope::Domain, Subject::Domain(domain)) => {
Some(format!("{}.{}.", domain.trim_end_matches('.'), self.zone))
}
_ => None,
}
}
}
pub enum Subject<'x> {
Ip(IpAddr),
Domain(&'x str),
}
/// Spamhaus' error codes, the same on every Spamhaus zone.
fn spamhaus_refusal(ip: Ipv4Addr) -> Option<Answer> {
match ip.octets() {
[127, 255, 255, 252] => Some(Answer::Refused("The query was malformed")),
[127, 255, 255, 254] => Some(Answer::Refused(
"Spamhaus doesn't answer public resolvers; use the server's own",
)),
[127, 255, 255, 255] => Some(Answer::Refused("Too many queries from this resolver")),
_ => None,
}
}
fn zen(ip: Ipv4Addr) -> Answer {
if let Some(refused) = spamhaus_refusal(ip) {
return refused;
}
match ip.octets() {
[127, 0, 0, 2] => Answer::Listed("SBL: a known spam source"),
[127, 0, 0, 3] => Answer::Listed("CSS: sent spam recently"),
[127, 0, 0, 4..=7] => Answer::Listed("XBL: a compromised or infected host"),
[127, 0, 0, 9] => Answer::Listed("DROP: a hijacked or criminal network"),
[127, 0, 0, 10 | 11] => {
Answer::Listed("PBL: an address that isn't meant to send mail directly")
}
_ => Answer::Unknown,
}
}
fn dbl(ip: Ipv4Addr) -> Answer {
if let Some(refused) = spamhaus_refusal(ip) {
return refused;
}
match ip.octets() {
[127, 0, 1, 2] => Answer::Listed("A spam domain"),
[127, 0, 1, 4] => Answer::Listed("A phishing domain"),
[127, 0, 1, 5] => Answer::Listed("A malware domain"),
[127, 0, 1, 6] => Answer::Listed("A botnet controller"),
[127, 0, 1, 102..=106] => Answer::Listed("A legitimate domain being abused"),
[127, 0, 1, 255] => Answer::Refused("The query was malformed"),
_ => Answer::Unknown,
}
}
/// Most lists answer 127.0.0.2 for "listed" and define nothing else.
fn just_two(ip: Ipv4Addr) -> Answer {
match ip.octets() {
[127, 0, 0, 2] => Answer::Listed("Listed"),
_ => Answer::Unknown,
}
}
fn surbl(ip: Ipv4Addr) -> Answer {
match ip.octets() {
[127, 0, 0, 1] => Answer::Refused("SURBL doesn't answer this resolver"),
[127, 0, 0, bits] if bits & (8 | 16 | 64 | 128) != 0 => {
Answer::Listed("Seen in phishing, malware, abuse or cracked sites")
}
_ => Answer::Unknown,
}
}
fn uribl(ip: Ipv4Addr) -> Answer {
match ip.octets() {
[127, 0, 0, 1] => Answer::Refused("URIBL doesn't answer public resolvers"),
[127, 0, 0, bits] if bits & (2 | 8) != 0 => Answer::Listed("Seen in spam"),
[127, 0, 0, bits] if bits & 4 != 0 => {
Answer::Listed("Grey: seen in bulk mail some people don't want")
}
_ => Answer::Unknown,
}
}
pub const LISTS: &[BlockList] = &[
BlockList {
name: "Spamhaus ZEN",
zone: "zen.spamhaus.org",
scope: Scope::Ip,
lookup: "https://check.spamhaus.org/",
note: None,
read: zen,
},
BlockList {
name: "SpamCop",
zone: "bl.spamcop.net",
scope: Scope::Ip,
lookup: "https://www.spamcop.net/bl.shtml",
note: None,
read: just_two,
},
BlockList {
name: "Barracuda",
zone: "b.barracudacentral.org",
scope: Scope::Ip,
lookup: "https://www.barracudacentral.org/lookups",
note: Some(
"Barracuda answers only resolvers whose address is registered with it (free, at barracudacentral.org/rbl). Until then its lookups can't be checked.",
),
read: just_two,
},
BlockList {
name: "UCEPROTECT level 1",
zone: "dnsbl-1.uceprotect.net",
scope: Scope::Ip,
lookup: "https://www.uceprotect.net/en/rblcheck.php",
note: None,
read: just_two,
},
BlockList {
name: "Mailspike",
zone: "bl.mailspike.net",
scope: Scope::Ip,
lookup: "https://mailspike.org/iplookup.html",
note: None,
read: just_two,
},
BlockList {
name: "PSBL",
zone: "psbl.surriel.com",
scope: Scope::Ip,
lookup: "https://psbl.org/",
note: None,
read: just_two,
},
BlockList {
name: "Spamhaus DBL",
zone: "dbl.spamhaus.org",
scope: Scope::Domain,
lookup: "https://check.spamhaus.org/",
note: None,
read: dbl,
},
BlockList {
name: "SURBL",
zone: "multi.surbl.org",
scope: Scope::Domain,
lookup: "https://surbl.org/surbl-analysis",
note: None,
read: surbl,
},
BlockList {
name: "URIBL",
zone: "multi.uribl.com",
scope: Scope::Domain,
lookup: "https://admin.uribl.com/",
note: None,
read: uribl,
},
];
pub fn by_name(name: &str) -> Option<&'static BlockList> {
LISTS.iter().find(|list| list.name == name)
}
#[cfg(test)]
mod tests {
use super::*;
fn ip(s: &str) -> Ipv4Addr {
s.parse().unwrap()
}
#[test]
fn a_refusal_is_not_a_listing() {
let zen = by_name("Spamhaus ZEN").unwrap();
assert!(matches!(
zen.read(ip("127.255.255.254")),
Answer::Refused(_)
));
assert!(matches!(zen.read(ip("127.0.0.2")), Answer::Listed(_)));
assert!(matches!(zen.read(ip("127.0.0.10")), Answer::Listed(_)));
assert_eq!(zen.read(ip("127.0.0.200")), Answer::Unknown);
let uribl = by_name("URIBL").unwrap();
assert!(matches!(uribl.read(ip("127.0.0.1")), Answer::Refused(_)));
assert!(matches!(uribl.read(ip("127.0.0.2")), Answer::Listed(_)));
}
#[test]
fn queries_are_built_per_scope() {
let zen = by_name("Spamhaus ZEN").unwrap();
let dbl = by_name("Spamhaus DBL").unwrap();
let v4 = Subject::Ip("192.0.2.10".parse().unwrap());
let v6 = Subject::Ip("2001:db8::1".parse().unwrap());
let domain = Subject::Domain("example.org");
assert_eq!(
zen.query(&v4).as_deref(),
Some("10.2.0.192.zen.spamhaus.org.")
);
assert_eq!(zen.query(&v6), None);
assert_eq!(zen.query(&domain), None);
assert_eq!(
dbl.query(&domain).as_deref(),
Some("example.org.dbl.spamhaus.org.")
);
assert_eq!(dbl.query(&v4), None);
}
#[test]
fn names_are_unique() {
for (i, a) in LISTS.iter().enumerate() {
assert!(
LISTS[i + 1..].iter().all(|b| b.name != a.name),
"{}",
a.name
);
}
}
}
+410
View File
@@ -0,0 +1,410 @@
/*
* SPDX-FileCopyrightText: 2026 Coffey Labs
*
* SPDX-License-Identifier: AGPL-3.0-only
*/
//! The deliverability check (deliverability spec): what other mail servers
//! see when this one sends. Not a rebuild of anything upstream ships.
//!
//! Every node that sends mail checks itself, because only it knows which
//! address it leaves from, and keeps one report. The report holds facts: an
//! address's reverse DNS, what each blocklist answered, what SPF said for
//! each address, whether a DKIM key in DNS matches the one signing. The
//! console grades them, so its wording can change without a server release.
//!
//! Kept in the fork's subspace (`store::SUBSPACE_INBUXA`). Every key starts
//! with `D`, then one byte for the kind:
//!
//! - `r` + node id (u64): that node's last report, as JSON.
//! - `s`: the settings, as JSON.
//!
//! Numbers are big-endian.
pub mod lists;
use serde::{Deserialize as SerdeDeserialize, Serialize as SerdeSerialize};
use store::{
Deserialize, IterateParams, SUBSPACE_INBUXA, Serialize, Store, ValueKey,
write::{AnyClass, BatchBuilder, ValueClass},
};
use trc::AddContext;
const FEATURE: u8 = b'D';
const KIND_REPORT: u8 = b'r';
const KIND_SETTINGS: u8 = b's';
/// DL-15: **Check now** runs a node again only this long after its last run.
pub const MIN_INTERVAL_SECS: u64 = 600;
#[derive(Debug, Clone, Default, PartialEq, SerdeSerialize, SerdeDeserialize)]
#[serde(rename_all = "camelCase", default)]
pub struct Report {
/// The node's cluster id, as metric samples carry it.
pub node_id: u64,
pub hostname: String,
/// Seconds since the epoch.
pub checked_at: u64,
pub addresses: Vec<Address>,
pub domains: Vec<DomainReport>,
pub certificates: Vec<Certificate>,
}
#[derive(Debug, Clone, Default, PartialEq, SerdeSerialize, SerdeDeserialize)]
#[serde(rename_all = "camelCase", default)]
pub struct Address {
pub ip: String,
/// DL-2: how the node came by the address.
pub source: AddressSource,
/// The connection strategy that sends from it.
pub strategy: String,
/// The name the node greets with from this address.
pub ehlo: String,
/// The PTR names, empty when there's none.
pub ptr: Vec<String>,
/// Some PTR name resolves back to the address.
pub forward_confirmed: bool,
/// The forward-confirmed name is the EHLO name.
pub ehlo_matches: bool,
/// Set when the reverse lookup itself failed, rather than found nothing.
pub ptr_error: Option<String>,
pub listings: Vec<Listing>,
}
#[derive(Debug, Clone, Copy, Default, PartialEq, Eq, SerdeSerialize, SerdeDeserialize)]
#[serde(rename_all = "camelCase")]
pub enum AddressSource {
/// Set in the connection strategy's source addresses.
#[default]
Configured,
/// What the EHLO name resolves to.
Ehlo,
}
#[derive(Debug, Clone, Default, PartialEq, SerdeSerialize, SerdeDeserialize)]
#[serde(rename_all = "camelCase", default)]
pub struct Listing {
/// The list's name, as in [`lists::LISTS`].
pub list: String,
pub state: ListingState,
/// The address the list answered, when it answered one.
pub code: Option<String>,
/// What the list says the answer means.
pub meaning: Option<String>,
}
#[derive(Debug, Clone, Copy, Default, PartialEq, Eq, SerdeSerialize, SerdeDeserialize)]
#[serde(rename_all = "camelCase")]
pub enum ListingState {
#[default]
Clean,
Listed,
/// The list wouldn't answer, or the lookup failed: neither listed nor clean.
Refused,
Error,
/// Switched off in the settings, so not asked.
Off,
}
#[derive(Debug, Clone, Default, PartialEq, SerdeSerialize, SerdeDeserialize)]
#[serde(rename_all = "camelCase", default)]
pub struct DomainReport {
pub domain: String,
/// DL-20: a tenant administrator sees only their tenant's domains.
pub tenant_id: Option<u32>,
/// DL-7: what SPF says for each of the node's addresses.
pub spf: Vec<SpfResult>,
/// DL-8: each DKIM key the domain signs with.
pub dkim: Vec<DkimKey>,
/// DL-9: the DMARC record, if there's one.
pub dmarc: Option<Dmarc>,
/// DL-10.
pub mta_sts: MtaSts,
/// DL-11: there's a `_smtp._tls` record.
pub tls_rpt: bool,
/// DL-12.
pub listings: Vec<Listing>,
}
#[derive(Debug, Clone, Default, PartialEq, SerdeSerialize, SerdeDeserialize)]
#[serde(rename_all = "camelCase", default)]
pub struct SpfResult {
pub ip: String,
/// `pass`, `fail`, `softFail`, `neutral`, `none`, `tempError` or `permError`.
pub result: String,
}
#[derive(Debug, Clone, Default, PartialEq, SerdeSerialize, SerdeDeserialize)]
#[serde(rename_all = "camelCase", default)]
pub struct DkimKey {
pub selector: String,
pub state: DkimState,
}
#[derive(Debug, Clone, Copy, Default, PartialEq, Eq, SerdeSerialize, SerdeDeserialize)]
#[serde(rename_all = "camelCase")]
pub enum DkimState {
#[default]
Matches,
/// Nothing published at `<selector>._domainkey.<domain>`.
Missing,
/// Published, but a different key.
Different,
/// The lookup failed.
Error,
}
#[derive(Debug, Clone, Default, PartialEq, SerdeSerialize, SerdeDeserialize)]
#[serde(rename_all = "camelCase", default)]
pub struct Dmarc {
/// `none`, `quarantine` or `reject`.
pub policy: String,
/// DKIM alignment: `relaxed` or `strict`.
pub adkim: String,
/// SPF alignment: `relaxed` or `strict`.
pub aspf: String,
}
#[derive(Debug, Clone, Default, PartialEq, SerdeSerialize, SerdeDeserialize)]
#[serde(rename_all = "camelCase", default)]
pub struct MtaSts {
/// The `_mta-sts` record's id; None when there's no record.
pub record_id: Option<String>,
/// The policy was fetched and parsed. False with a record means the
/// fetch or the parse failed, and `error` says why.
pub fetched: bool,
pub error: Option<String>,
/// `enforce`, `testing` or `none`.
pub mode: Option<String>,
pub max_age: Option<u64>,
/// The domain's MX names no `mx:` line matches.
pub mx_not_covered: Vec<String>,
}
#[derive(Debug, Clone, Default, PartialEq, SerdeSerialize, SerdeDeserialize)]
#[serde(rename_all = "camelCase", default)]
pub struct Certificate {
/// The EHLO name, or an MX name that points at this node.
pub name: String,
/// The node holds a certificate for the name.
pub covered: bool,
}
#[derive(Debug, Clone, Default, PartialEq, SerdeSerialize, SerdeDeserialize)]
#[serde(rename_all = "camelCase", default)]
pub struct Settings {
/// DL-6: lists not to ask, by name.
pub disabled_lists: Vec<String>,
}
impl Settings {
pub fn is_off(&self, list: &str) -> bool {
self.disabled_lists.iter().any(|name| name == list)
}
/// Only the built-in lists' names, once each.
pub fn validate(&self) -> Result<(), String> {
for (i, name) in self.disabled_lists.iter().enumerate() {
if lists::by_name(name).is_none() {
return Err(format!("There's no list called {name:?}."));
}
if self.disabled_lists[..i].contains(name) {
return Err(format!("{name:?} is named twice."));
}
}
Ok(())
}
}
impl Report {
/// DL-20: what a tenant administrator may see: their tenant's domains
/// and nothing about the node's addresses or certificates.
pub fn for_tenant(&self, tenant_id: u32) -> Report {
Report {
node_id: self.node_id,
hostname: self.hostname.clone(),
checked_at: self.checked_at,
addresses: Vec::new(),
domains: self
.domains
.iter()
.filter(|d| d.tenant_id == Some(tenant_id))
.cloned()
.collect(),
certificates: Vec::new(),
}
}
}
// --- Storage --------------------------------------------------------------
struct Json<T>(T);
impl<T: SerdeSerialize> Serialize for Json<T> {
fn serialize(&self) -> trc::Result<Vec<u8>> {
serde_json::to_vec(&self.0).map_err(|err| {
trc::StoreEvent::UnexpectedError
.into_err()
.details("Failed to serialize deliverability data")
.reason(err)
})
}
}
impl<T: for<'de> SerdeDeserialize<'de> + Send + Sync> Deserialize for Json<T> {
fn deserialize(bytes: &[u8]) -> trc::Result<Self> {
serde_json::from_slice(bytes).map(Json).map_err(|err| {
trc::StoreEvent::DataCorruption
.into_err()
.details("Invalid deliverability data")
.reason(err)
})
}
}
fn class(kind: u8, node_id: Option<u64>) -> ValueClass {
let mut key = Vec::with_capacity(10);
key.push(FEATURE);
key.push(kind);
if let Some(node_id) = node_id {
key.extend_from_slice(&node_id.to_be_bytes());
}
ValueClass::Any(AnyClass {
subspace: SUBSPACE_INBUXA,
key,
})
}
pub async fn report(data: &Store, node_id: u64) -> trc::Result<Option<Report>> {
Ok(data
.get_value::<Json<Report>>(ValueKey::from(class(KIND_REPORT, Some(node_id))))
.await
.caused_by(trc::location!())?
.map(|Json(report)| report))
}
/// Every node's report, by node id.
pub async fn reports(data: &Store) -> trc::Result<Vec<Report>> {
let mut out = Vec::new();
data.iterate(
IterateParams::new(
ValueKey::from(class(KIND_REPORT, Some(0))),
ValueKey::from(class(KIND_REPORT, Some(u64::MAX))),
),
|_, value| {
if let Ok(Json(report)) = Json::<Report>::deserialize(value) {
out.push(report);
}
Ok(true)
},
)
.await
.caused_by(trc::location!())?;
out.sort_by_key(|r| r.node_id);
Ok(out)
}
/// Replaces the node's report.
pub async fn put_report(data: &Store, report: &Report) -> trc::Result<()> {
let mut batch = BatchBuilder::new();
batch.set(
class(KIND_REPORT, Some(report.node_id)),
Json(report).serialize()?,
);
data.write(batch.build_all())
.await
.caused_by(trc::location!())?;
Ok(())
}
pub async fn settings(data: &Store) -> trc::Result<Settings> {
Ok(data
.get_value::<Json<Settings>>(ValueKey::from(class(KIND_SETTINGS, None)))
.await
.caused_by(trc::location!())?
.map(|Json(settings)| settings)
.unwrap_or_default())
}
pub async fn put_settings(data: &Store, settings: &Settings) -> trc::Result<()> {
let mut batch = BatchBuilder::new();
batch.set(class(KIND_SETTINGS, None), Json(settings).serialize()?);
data.write(batch.build_all())
.await
.caused_by(trc::location!())?;
Ok(())
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn settings_name_only_built_in_lists_once() {
let ok = Settings {
disabled_lists: vec!["Barracuda".into(), "URIBL".into()],
};
assert!(ok.validate().is_ok());
assert!(ok.is_off("Barracuda"));
assert!(!ok.is_off("SpamCop"));
let unknown = Settings {
disabled_lists: vec!["My list".into()],
};
assert!(unknown.validate().is_err());
let twice = Settings {
disabled_lists: vec!["URIBL".into(), "URIBL".into()],
};
assert!(twice.validate().is_err());
}
#[test]
fn a_tenant_sees_only_its_domains() {
let report = Report {
node_id: 2,
hostname: "mx2.example.org".into(),
checked_at: 1,
addresses: vec![Address {
ip: "192.0.2.10".into(),
..Default::default()
}],
domains: vec![
DomainReport {
domain: "a.example".into(),
tenant_id: Some(7),
..Default::default()
},
DomainReport {
domain: "b.example".into(),
tenant_id: Some(8),
..Default::default()
},
DomainReport {
domain: "server.example".into(),
tenant_id: None,
..Default::default()
},
],
certificates: vec![Certificate {
name: "mx2.example.org".into(),
covered: true,
}],
};
let seen = report.for_tenant(7);
assert!(seen.addresses.is_empty());
assert!(seen.certificates.is_empty());
assert_eq!(
seen.domains
.iter()
.map(|d| d.domain.as_str())
.collect::<Vec<_>>(),
["a.example"]
);
}
#[test]
fn a_report_reads_back_with_missing_fields() {
let report: Report = serde_json::from_str(r#"{"nodeId": 3}"#).unwrap();
assert_eq!(report.node_id, 3);
assert!(report.domains.is_empty());
}
}
+1
View File
@@ -21,6 +21,7 @@
pub mod ai;
pub mod audit;
pub mod branding;
pub mod deliverability; // inbuxa: the deliverability check (not a rebuild)
pub mod hold;
pub mod journal;
pub mod lock;