inbuxa/inbuxa-server
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 23 Wiki Activity

Merge pull request 'Merge upstream v0.16.24' (#84) from merge/upstream-v0.16.24 into main
ci / fork-checks (push) Successful in 54s
Details
ci / build (push) Canceled after 39m0s
Details

Browse Source
This commit was merged in pull request #84.
This commit is contained in:
jcoffey-dev
2026-09-28 15:55:16 +00:00
parent d107c1b2bb 80d6c09c59
commit 9893452ca2
96 changed files with 4212 additions and 1066 deletions
Download Patch File Download Diff File Expand all files Collapse all files
CHANGELOG.md
+33
View File
@@ -2,6 +2,39 @@
All notable changes to this project will be documented in this file. This project adheres to [Semantic Versioning](http://semver.org/).
## [0.16.24] - 2026-09-27
If you are upgrading from v0.16.x, replace the binary (or run `docker pull`). If you are upgrading from v0.15.x and below, please read the [upgrading documentation](https://github.com/stalwartlabs/stalwart/blob/main/UPGRADING/v0_16.md) for more information on how to upgrade from previous versions.
## Added
- DNS: PowerDNS Authoritative provider for automatic DNS record management.
## Changed
## Fixed
- Troubleshoot tool: `TLSA` records are looked up for every MX host, including hosts whose zone is not DNSSEC signed.
- Spam filter:
- OpenPhish and PhishTank entries containing uppercase characters never match, since message URLs are lowercased while HTTP lookup entries keep their original case. HTTP lookups now match keys case-insensitively.
- URL shortener links are followed using the lowercased URL, so case-sensitive short links resolve to the wrong destination or not at all.
- Incremental training never advances its position past the first run, so every retained sample added since then is trained again, and counted again in the reservoir, on each run until it expires.
- Updating the rules only adds new objects, so upstream changes to existing rules, DNSBL servers, HTTP lookups, lookup keys and file extensions never reach an existing installation.
- Updating the rules reports success when objects fail to import, or when a configuration error stops the updated settings from being activated.
- JMAP:
- A `PushSubscription` created within the verification rate limit window of another one on the same account never receives its `PushVerification`, since the blocked verification is dropped instead of being sent once the window expires.
- A push notification retried after a failed delivery can report an older state than a change queued during the failed attempt, since the older state changes are merged last and overwrite the newer ones.
- Changes made while a push request is in flight are not delivered until the next change reaches the same subscription, since a successful delivery cancels the pending retry.
- The VAPID `aud` claim is derived from a hand-written parse of the push URL, so a crafted push URL can make the server sign a token for a push service other than the one the request is sent to.
- `Email/import` rejects a `blobId` that refers to a `Blob/upload` creation id in the same request (`"#u0"`) with `Invalid blob id.`.
- `Email/set` with a full `mailboxIds` object identical to the current mailboxes, together with a keyword change, stores the message with IMAP UID 0, so IMAP clients stop seeing it.
- MTA:
- A node without the `outboundMta` role stops replying to `DATA` and to JMAP submissions once about 1024 messages have been queued on it.
- MX records are resolved through the DNSSEC-validating resolver even when DANE is disabled.
- A `DATA` stage Sieve script does not see headers added by milters or MTA hooks, and discards every milter and MTA hook change when it edits the message.
- MySQL: Range deletions and search index removals start with a single unbounded `DELETE` and switch to chunks only after a timeout.
- IMAP: `COPY` and `MOVE` fail with `NO [CONTACTADMIN]` when another session changes the same message at the same time.
- Autodiscover: Implicit TLS ports (993, 995, 465) are advertised with `<Encryption>TLS</Encryption>`, which Outlook reads as STARTTLS.
- HTTP: Idle keep-alive connections are never closed.
## [0.16.23] - 2026-09-21
If you are upgrading from v0.16.x, replace the binary (or run `docker pull`). If you are upgrading from v0.15.x and below, please read the [upgrading documentation](https://github.com/stalwartlabs/stalwart/blob/main/UPGRADING/v0_16.md) for more information on how to upgrade from previous versions.
Cargo.lock
Generated
+144 -177
View File
File diff suppressed because it is too large Load Diff
crates/common/Cargo.toml
+1 -1
View File
@@ -1,6 +1,6 @@
[package]
name = "common"
version = "0.16.23"
version = "0.16.24"
edition = "2024"
build = "build.rs"
crates/common/src/manager/defaults.rs
+3 -3
View File
@@ -570,8 +570,8 @@ async fn insert_safe_defaults(bp: &mut Bootstrap) -> trc::Result<()> {
// inbuxa: rules are always to hand, since a copy ships with the server
// (spam_rules). They load on first boot, and again when the bundled
// version differs from the one last loaded, which only adds what's
// missing: new tags and rules, never a changed score.
// rules differ from the ones last loaded: new tags and rules, fixes to
// rules nobody edited, never a changed score or an admin's edit.
let rules_url = super::spam_rules::rules_url(
bp.registry
.object::<SpamSettings>(Id::singleton())
@@ -582,7 +582,7 @@ async fn insert_safe_defaults(bp: &mut Bootstrap) -> trc::Result<()> {
&& super::spam_rules::applied_version(&bp.data_store)
.await?
.as_deref()
!= Some(super::spam_rules::BUNDLED_SPAM_RULES_VERSION);
!= Some(super::spam_rules::BUNDLED_SPAM_RULES_APPLIED);
if bp.registry.count_object(ObjectType::SpamRule).await? == 0 || bundled_is_new {
let mut batch = BatchBuilder::new();
batch.schedule_task(Task::SpamFilterMaintenance(TaskSpamFilterMaintenance {
crates/common/src/manager/spam_rules.rs
+61 -8
View File
@@ -12,11 +12,16 @@
//! and license) and uses it whenever no other source is configured. The rules
//! URL remains an operator override (`https://` or `file://`).
//!
//! Loading rules only ever adds what's missing, never changes an existing rule
//! or score. They load on first boot, and again whenever the bundled version
//! differs from the one last applied, so an upgrade brings new tags (the AI
//! classifier's `LLM_*` scores, say) to an install that already had rules.
//! Loading rules adds what's missing and brings an existing rule up to date,
//! but never touches one an admin edited: every object an update writes is
//! fingerprinted, and one that no longer matches its fingerprint is kept as
//! it is. Tags (scores) are never replaced. Switching a rule on or off isn't
//! an edit, and is kept either way. They load on first boot, and again
//! whenever the bundled rules differ from the ones last applied, so an
//! upgrade brings new tags (the AI classifier's `LLM_*` scores, say) and
//! fixed rules to an install that already had rules.
use registry::{schema::prelude::ObjectType, types::EnumImpl};
use std::io::Read;
use store::{
SUBSPACE_INBUXA, Store, ValueKey,
@@ -27,13 +32,17 @@ use trc::AddContext;
/// The version of spam-filter the embedded rules come from.
pub const BUNDLED_SPAM_RULES_VERSION: &str = "3.0.2";
/// What's recorded once the bundled rules are loaded: their version, then the
/// fork's own generation of the update, so a change to how an update applies
/// runs it once more. Generation 2 fingerprints (upstream v0.16.24).
pub const BUNDLED_SPAM_RULES_APPLIED: &str = "3.0.2+2";
static BUNDLED_SPAM_RULES: &[u8] =
include_bytes!("../../../../resources/spam-filter/spam-filter-rules.json.gz");
/// Upstream's default rules source, the value every install created before
/// the rules were bundled has saved. Read only to treat it as unset.
const LEGACY_DEFAULT_URL: &str =
"https://github.com/stalwartlabs/spam-filter/releases/latest/download/spam-filter-rules.json.gz";
const LEGACY_DEFAULT_URL: &str = "https://github.com/stalwartlabs/spam-filter/releases/latest/download/spam-filter-rules.json.gz";
/// The URL to fetch rules from, or `None` for the bundled rules. An empty
/// setting and upstream's old default both mean the bundled rules.
@@ -57,14 +66,49 @@ fn applied_key() -> ValueClass {
})
}
/// The bundled version last loaded into the registry, if any.
fn fingerprint_key(object: ObjectType, id: u64) -> ValueClass {
let mut key = b"Sf".to_vec();
key.extend_from_slice(object.as_str().as_bytes());
key.push(0);
key.extend_from_slice(&id.to_be_bytes());
ValueClass::Any(AnyClass {
subspace: SUBSPACE_INBUXA,
key,
})
}
/// The fingerprint of what a rules update last wrote to this object, if one
/// did.
pub async fn fingerprint(data: &Store, object: ObjectType, id: u64) -> trc::Result<Option<String>> {
data.get_value::<String>(ValueKey::from(fingerprint_key(object, id)))
.await
.caused_by(trc::location!())
}
/// Records the fingerprint of what a rules update wrote to this object.
pub async fn set_fingerprint(
data: &Store,
object: ObjectType,
id: u64,
fingerprint: &str,
) -> trc::Result<()> {
let mut batch = BatchBuilder::new();
batch.set(fingerprint_key(object, id), fingerprint.as_bytes().to_vec());
data.write(batch.build_all())
.await
.caused_by(trc::location!())
.map(|_| ())
}
/// The bundled rules last loaded into the registry, if any
/// ([`BUNDLED_SPAM_RULES_APPLIED`]'s form).
pub async fn applied_version(data: &Store) -> trc::Result<Option<String>> {
data.get_value::<String>(ValueKey::from(applied_key()))
.await
.caused_by(trc::location!())
}
/// Records that the bundled rules of this version have been loaded.
/// Records that the bundled rules have been loaded.
pub async fn set_applied_version(data: &Store, version: &str) -> trc::Result<()> {
let mut batch = BatchBuilder::new();
batch.set(applied_key(), version.as_bytes().to_vec());
@@ -90,6 +134,15 @@ mod tests {
);
}
#[test]
fn applied_marker_names_the_bundled_version() {
assert!(
BUNDLED_SPAM_RULES_APPLIED
.strip_prefix(BUNDLED_SPAM_RULES_VERSION)
.is_some_and(|generation| generation.starts_with('+'))
);
}
#[test]
fn bundled_rules_parse_and_score_the_ai_tags() {
let rules: serde_json::Value = serde_json::from_slice(&bundled_rules().unwrap()).unwrap();
crates/common/src/network/autoconfig/autodiscover.rs
+162 -72
View File
@@ -10,8 +10,9 @@ use crate::{Server, manager::application::Resource};
use quick_xml::Reader;
use quick_xml::XmlVersion;
use quick_xml::events::Event;
use registry::schema::enums::ServiceProtocol;
use registry::schema::{enums::ServiceProtocol, structs::Service};
use std::fmt::Write;
use utils::map::vec_map::VecMap;
impl Server {
pub async fn handle_autodiscover_request(
@@ -26,89 +27,103 @@ impl Server {
.details("Failed to parse autodiscover request")
.ctx(trc::Key::Reason, err)
})?;
let default_host = &self.core.network.server_name;
// Build XML response
let mut config = String::with_capacity(1024);
let _ = writeln!(&mut config, "<?xml version=\"1.0\" encoding=\"UTF-8\"?>");
let _ = writeln!(
&mut config,
"<Autodiscover xmlns=\"http://schemas.microsoft.com/exchange/autodiscover/responseschema/2006\">"
);
let _ = writeln!(
&mut config,
"\t<Response xmlns=\"http://schemas.microsoft.com/exchange/autodiscover/outlook/responseschema/2006a\">"
);
let _ = writeln!(&mut config, "\t\t<User>");
let _ = writeln!(
&mut config,
"\t\t\t<DisplayName>{emailaddress}</DisplayName>"
);
let _ = writeln!(
&mut config,
"\t\t\t<AutoDiscoverSMTPAddress>{emailaddress}</AutoDiscoverSMTPAddress>"
);
// DeploymentId is a required field of User but we are not a MS Exchange server so use a random value
let _ = writeln!(
&mut config,
"\t\t\t<DeploymentId>644560b8-a1ce-429c-8ace-23395843f701</DeploymentId>"
);
let _ = writeln!(&mut config, "\t\t</User>");
let _ = writeln!(&mut config, "\t\t<Account>");
let _ = writeln!(&mut config, "\t\t\t<AccountType>email</AccountType>");
let _ = writeln!(&mut config, "\t\t\t<Action>settings</Action>");
// inbuxa: legacy-protocols LP-7, LP-14a
let legacy_off = match emailaddress.rsplit_once('@') {
Some((_, domain)) => self.legacy_off_for(domain).await?,
None => self.legacy_off_for("").await?,
};
for (protocol, service) in &self.core.network.info.services {
if legacy_off.service(protocol) {
continue;
}
let (protocol, ports) = match protocol {
ServiceProtocol::Imap => ("IMAP", [143, 993]),
ServiceProtocol::Pop3 => ("POP3", [110, 995]),
ServiceProtocol::Smtp => ("SMTP", [587, 465]),
_ => continue,
};
for (is_tls, port) in ports.into_iter().enumerate() {
if is_tls == 1 || service.cleartext {
let server_name = service.hostname.as_deref().unwrap_or(default_host);
let _ = writeln!(&mut config, "\t\t\t<Protocol>");
let _ = writeln!(&mut config, "\t\t\t\t<Type>{protocol}</Type>",);
let _ = writeln!(&mut config, "\t\t\t\t<Server>{server_name}</Server>");
let _ = writeln!(&mut config, "\t\t\t\t<Port>{port}</Port>");
let _ = writeln!(&mut config, "\t\t\t\t<LoginName>{emailaddress}</LoginName>");
let _ = writeln!(&mut config, "\t\t\t\t<AuthRequired>on</AuthRequired>");
let _ = writeln!(&mut config, "\t\t\t\t<DirectoryPort>0</DirectoryPort>");
let _ = writeln!(&mut config, "\t\t\t\t<ReferralPort>0</ReferralPort>");
let _ = writeln!(
&mut config,
"\t\t\t\t<SSL>{}</SSL>",
if is_tls == 1 { "on" } else { "off" }
);
if is_tls == 1 {
let _ = writeln!(&mut config, "\t\t\t\t<Encryption>TLS</Encryption>");
}
let _ = writeln!(&mut config, "\t\t\t\t<SPA>off</SPA>");
let _ = writeln!(&mut config, "\t\t\t</Protocol>");
}
}
}
let _ = writeln!(&mut config, "\t\t</Account>");
let _ = writeln!(&mut config, "\t</Response>");
let _ = writeln!(&mut config, "</Autodiscover>");
Ok(Resource::new(
"application/xml; charset=utf-8",
config.into_bytes(),
build_autodiscover_response(
&emailaddress,
&self.core.network.server_name,
&self.core.network.info.services,
|protocol| legacy_off.service(protocol),
)
.into_bytes(),
))
}
}
fn build_autodiscover_response(
emailaddress: &str,
default_host: &str,
services: &VecMap<ServiceProtocol, Service>,
switched_off: impl Fn(&ServiceProtocol) -> bool,
) -> String {
// Build XML response
let mut config = String::with_capacity(1024);
let _ = writeln!(&mut config, "<?xml version=\"1.0\" encoding=\"UTF-8\"?>");
let _ = writeln!(
&mut config,
"<Autodiscover xmlns=\"http://schemas.microsoft.com/exchange/autodiscover/responseschema/2006\">"
);
let _ = writeln!(
&mut config,
"\t<Response xmlns=\"http://schemas.microsoft.com/exchange/autodiscover/outlook/responseschema/2006a\">"
);
let _ = writeln!(&mut config, "\t\t<User>");
let _ = writeln!(
&mut config,
"\t\t\t<DisplayName>{emailaddress}</DisplayName>"
);
let _ = writeln!(
&mut config,
"\t\t\t<AutoDiscoverSMTPAddress>{emailaddress}</AutoDiscoverSMTPAddress>"
);
// DeploymentId is a required field of User but we are not a MS Exchange server so use a random value
let _ = writeln!(
&mut config,
"\t\t\t<DeploymentId>644560b8-a1ce-429c-8ace-23395843f701</DeploymentId>"
);
let _ = writeln!(&mut config, "\t\t</User>");
let _ = writeln!(&mut config, "\t\t<Account>");
let _ = writeln!(&mut config, "\t\t\t<AccountType>email</AccountType>");
let _ = writeln!(&mut config, "\t\t\t<Action>settings</Action>");
for (protocol, service) in services {
if switched_off(protocol) {
continue;
}
let (protocol, ports) = match protocol {
ServiceProtocol::Imap => ("IMAP", [(993, true), (143, false)]),
ServiceProtocol::Pop3 => ("POP3", [(995, true), (110, false)]),
ServiceProtocol::Smtp => ("SMTP", [(465, true), (587, false)]),
_ => continue,
};
// Implicit TLS is listed first so that it is preferred (RFC 8314)
for (port, is_tls) in ports {
if is_tls || service.cleartext {
let server_name = service.hostname.as_deref().unwrap_or(default_host);
let _ = writeln!(&mut config, "\t\t\t<Protocol>");
let _ = writeln!(&mut config, "\t\t\t\t<Type>{protocol}</Type>",);
let _ = writeln!(&mut config, "\t\t\t\t<Server>{server_name}</Server>");
let _ = writeln!(&mut config, "\t\t\t\t<Port>{port}</Port>");
let _ = writeln!(&mut config, "\t\t\t\t<LoginName>{emailaddress}</LoginName>");
let _ = writeln!(&mut config, "\t\t\t\t<AuthRequired>on</AuthRequired>");
let _ = writeln!(&mut config, "\t\t\t\t<DirectoryPort>0</DirectoryPort>");
let _ = writeln!(&mut config, "\t\t\t\t<ReferralPort>0</ReferralPort>");
let (ssl, encryption) = if is_tls {
("on", "SSL")
} else {
("off", "TLS")
};
let _ = writeln!(&mut config, "\t\t\t\t<SSL>{ssl}</SSL>");
let _ = writeln!(&mut config, "\t\t\t\t<Encryption>{encryption}</Encryption>");
let _ = writeln!(&mut config, "\t\t\t\t<SPA>off</SPA>");
let _ = writeln!(&mut config, "\t\t\t</Protocol>");
}
}
}
let _ = writeln!(&mut config, "\t\t</Account>");
let _ = writeln!(&mut config, "\t</Response>");
let _ = writeln!(&mut config, "</Autodiscover>");
config
}
fn parse_autodiscover_request(bytes: &[u8]) -> Result<String, String> {
if bytes.is_empty() {
return Err("Empty request body".to_string());
@@ -211,4 +226,79 @@ mod tests {
"[email protected]"
);
}
#[test]
fn autodiscover_encryption() {
use registry::schema::{enums::ServiceProtocol, structs::Service};
use utils::map::vec_map::VecMap;
fn tag<'x>(block: &'x str, name: &str) -> &'x str {
block
.split_once(&format!("<{name}>"))
.and_then(|(_, rest)| rest.split_once(&format!("</{name}>")))
.map(|(value, _)| value)
.unwrap()
}
for (cleartext, expected) in [
(
false,
vec![
("IMAP", "993", "on", "SSL"),
("POP3", "995", "on", "SSL"),
("SMTP", "465", "on", "SSL"),
],
),
(
true,
vec![
("IMAP", "993", "on", "SSL"),
("IMAP", "143", "off", "TLS"),
("POP3", "995", "on", "SSL"),
("POP3", "110", "off", "TLS"),
("SMTP", "465", "on", "SSL"),
("SMTP", "587", "off", "TLS"),
],
),
] {
let services: VecMap<ServiceProtocol, Service> = [
ServiceProtocol::Imap,
ServiceProtocol::Pop3,
ServiceProtocol::Smtp,
ServiceProtocol::Jmap,
]
.into_iter()
.map(|protocol| {
(
protocol,
Service {
hostname: None,
cleartext,
},
)
})
.collect();
let response = super::build_autodiscover_response(
"[email protected]",
"mail.example.com",
&services,
|_| false,
);
assert_eq!(
response
.split("<Protocol>")
.skip(1)
.map(|block| (
tag(block, "Type"),
tag(block, "Port"),
tag(block, "SSL"),
tag(block, "Encryption"),
))
.collect::<Vec<_>>(),
expected,
"cleartext: {cleartext}"
);
}
}
}
crates/common/src/network/dns/update.rs
+14
View File
@@ -961,6 +961,20 @@ impl DnsUpdater {
)
.map_err(|err| format!("Failed to build DNS updater: {}", err))?,
}),
DnsServer::PowerDns(server) => Ok(DnsUpdater {
polling_interval: server.polling_interval.into_inner(),
propagation_timeout: server.propagation_timeout.into_inner(),
propagation_delay: server.propagation_delay.map(|d| d.into_inner()),
ttl: server.ttl.into_inner(),
core,
updater: dns_update::DnsUpdater::new_pdns(
server.api_key.secret().await?,
server.endpoint,
server.server_id,
server.timeout.into_inner().into(),
)
.map_err(|err| format!("Failed to build DNS updater: {}", err))?,
}),
DnsServer::Safedns(server) => Ok(DnsUpdater {
polling_interval: server.polling_interval.into_inner(),
propagation_timeout: server.propagation_timeout.into_inner(),
crates/common/src/network/webpush.rs
+102 -36
View File
@@ -6,33 +6,79 @@
* Modified by Coffey Labs in 2026 for INBUXA.
*/
use ahash::AHashMap;
use base64::{Engine, engine::general_purpose::URL_SAFE_NO_PAD};
use p256::{
SecretKey,
ecdsa::{Signature, SigningKey, signature::Signer},
pkcs8::{DecodePrivateKey, PrivateKeyInfo, der::SecretDocument},
};
use parking_lot::Mutex;
use reqwest::{Url, header::HeaderValue};
use std::sync::Arc;
const VAPID_TOKEN_TTL: u64 = 12 * 60 * 60;
const VAPID_TOKEN_REFRESH: u64 = VAPID_TOKEN_TTL / 2;
#[derive(Clone)]
pub struct Vapid {
key: VapidKey,
contact: Option<String>,
tokens: Arc<Mutex<AHashMap<String, VapidToken>>>,
}
struct VapidToken {
authorization: HeaderValue,
issued_at: u64,
}
impl Vapid {
pub fn new(key: VapidKey, contact: Option<String>) -> Self {
Self { key, contact }
Self {
key,
contact,
tokens: Arc::default(),
}
}
pub fn public_key(&self) -> &str {
self.key.public_key()
}
pub fn authorization(&self, endpoint: &str, now: u64) -> Option<String> {
self.key
.authorization(endpoint, self.contact.as_deref(), now)
pub fn authorization(&self, endpoint: &str, now: u64) -> Option<HeaderValue> {
let prefix = endpoint_prefix(endpoint)?;
if let Some(token) = self
.tokens
.lock()
.get(prefix)
.filter(|token| token.is_fresh(now))
{
return Some(token.authorization.clone());
}
let authorization = HeaderValue::try_from(self.key.authorization(
endpoint,
self.contact.as_deref(),
now,
)?)
.ok()?;
let mut tokens = self.tokens.lock();
tokens.retain(|_, token| token.is_fresh(now));
tokens.insert(
prefix.to_string(),
VapidToken {
authorization: authorization.clone(),
issued_at: now,
},
);
Some(authorization)
}
}
impl VapidToken {
fn is_fresh(&self, now: u64) -> bool {
now.checked_sub(self.issued_at)
.is_some_and(|age| age < VAPID_TOKEN_REFRESH)
}
}
@@ -105,41 +151,15 @@ impl VapidKey {
}
}
fn endpoint_origin(url: &str) -> Option<String> {
fn endpoint_prefix(url: &str) -> Option<&str> {
let (scheme, rest) = url.split_once("://")?;
let scheme = scheme.to_ascii_lowercase();
let authority = rest.split(['/', '?', '#']).next()?;
let authority = authority
.rsplit_once('@')
.map(|(_, host)| host)
.unwrap_or(authority);
if authority.is_empty() {
return None;
}
url.get(..scheme.len() + "://".len() + authority.len())
}
let (host, port) = if let Some(rest) = authority.strip_prefix('[') {
let (addr, tail) = rest.split_once(']')?;
(
format!("[{}]", addr.to_ascii_lowercase()),
tail.strip_prefix(':').filter(|port| !port.is_empty()),
)
} else if let Some((host, port)) = authority.rsplit_once(':') {
(
host.to_ascii_lowercase(),
Some(port).filter(|p| !p.is_empty()),
)
} else {
(authority.to_ascii_lowercase(), None)
};
match port {
Some(port)
if !((scheme == "https" && port == "443") || (scheme == "http" && port == "80")) =>
{
Some(format!("{scheme}://{host}:{port}"))
}
_ => Some(format!("{scheme}://{host}")),
}
fn endpoint_origin(url: &str) -> Option<String> {
let origin = Url::parse(url).ok()?.origin();
origin.is_tuple().then(|| origin.ascii_serialization())
}
pub fn normalize_contact(contact: &str) -> Option<String> {
@@ -206,7 +226,12 @@ mod tests {
endpoint_origin("http://[2001:DB8::1]:80/p").unwrap(),
"http://[2001:db8::1]"
);
assert_eq!(
endpoint_origin("https://attacker.example\\@fcm.googleapis.com/fcm/send/x").unwrap(),
"https://attacker.example"
);
assert!(endpoint_origin("not-a-url").is_none());
assert!(endpoint_origin("mailto:[email protected]").is_none());
}
#[test]
@@ -336,6 +361,47 @@ B4yDfR2rGOd2H6Kv3fQNHPj9Nu5Tks8QYMLzrX8ONCNoFnNUQl9S0r0QS6phVqD0
}
}
#[test]
fn authorization_is_reused_per_endpoint_prefix() {
let vapid = Vapid::new(test_key(), None);
let now = 1_700_000_000;
let token = vapid
.authorization("https://push.example.com/push/a", now)
.unwrap();
assert_eq!(
vapid
.authorization("https://push.example.com/push/b?x=1", now + 60)
.unwrap(),
token
);
assert_ne!(
vapid
.authorization("https://other.example.com/push/a", now)
.unwrap(),
token
);
assert_ne!(
vapid
.authorization("https://push.example.com/push/a", now - 1)
.unwrap(),
token
);
let refreshed = vapid
.authorization("https://push.example.com/push/a", now + VAPID_TOKEN_REFRESH)
.unwrap();
assert_ne!(refreshed, token);
assert_eq!(
vapid
.authorization(
"https://push.example.com/push/c",
now + VAPID_TOKEN_REFRESH + 1
)
.unwrap(),
refreshed
);
}
#[test]
fn authorization_omits_subject_when_no_contact() {
let key = test_key();
crates/coordinator/Cargo.toml
+1 -1
View File
@@ -1,6 +1,6 @@
[package]
name = "coordinator"
version = "0.16.23"
version = "0.16.24"
edition = "2024"
[dependencies]
crates/dav-proto/Cargo.toml
+1 -1
View File
@@ -1,6 +1,6 @@
[package]
name = "dav-proto"
version = "0.16.23"
version = "0.16.24"
edition = "2024"
[dependencies]
crates/dav/Cargo.toml
+1 -1
View File
@@ -1,6 +1,6 @@
[package]
name = "dav"
version = "0.16.23"
version = "0.16.24"
edition = "2024"
[dependencies]
crates/directory/Cargo.toml
+1 -1
View File
@@ -1,6 +1,6 @@
[package]
name = "directory"
version = "0.16.23"
version = "0.16.24"
edition = "2024"
[dependencies]
crates/email/Cargo.toml
+1 -1
View File
@@ -1,6 +1,6 @@
[package]
name = "email"
version = "0.16.23"
version = "0.16.24"
edition = "2024"
[dependencies]
crates/groupware/Cargo.toml
+1 -1
View File
@@ -1,6 +1,6 @@
[package]
name = "groupware"
version = "0.16.23"
version = "0.16.24"
edition = "2024"
[dependencies]
crates/http-proto/Cargo.toml
+1 -1
View File
@@ -1,6 +1,6 @@
[package]
name = "http_proto"
version = "0.16.23"
version = "0.16.24"
edition = "2024"
[dependencies]
crates/http/Cargo.toml
+1 -1
View File
@@ -1,6 +1,6 @@
[package]
name = "http"
version = "0.16.23"
version = "0.16.24"
edition = "2024"
[dependencies]
crates/http/src/api/diagnose.rs
+103 -70
View File
@@ -12,7 +12,7 @@ use common::{
},
};
use hyper::body::{Bytes, Frame};
use mail_auth::{IpLookupStrategy, mta_sts::TlsRpt};
use mail_auth::{DnssecStatus, IpLookupStrategy, mta_sts::TlsRpt};
use serde::{Deserialize, Serialize};
use smtp::outbound::{
client::{SmtpClient, StartTlsResult},
@@ -382,81 +382,25 @@ async fn delivery_diagnose(
}
}
// Fetch TLSA record
tx.send(DeliveryStage::TlsaLookupStart).await?;
let now = Instant::now();
let dane_policy = match server.tlsa_lookup(format!("_25._tcp.{hostname}.")).await {
Ok(TlsaResult::Secure(tlsa)) if tlsa.has_end_entities => {
tx.send(DeliveryStage::TlsaLookupSuccess {
record: tlsa.as_ref().clone(),
elapsed: now.elapsed_ms(),
})
.await?;
Some(tlsa)
}
Ok(TlsaResult::Secure(_)) => {
tx.send(DeliveryStage::TlsaLookupError {
elapsed: now.elapsed_ms(),
reason: "TLSA record does not have end entities".to_string(),
})
.await?;
None
}
Ok(TlsaResult::Bogus) => {
tx.send(DeliveryStage::TlsaLookupError {
elapsed: now.elapsed_ms(),
reason: "Bogus TLSA record".to_string(),
})
.await?;
continue 'outer;
}
Ok(TlsaResult::Missing) => {
tx.send(DeliveryStage::TlsaNotFound {
elapsed: now.elapsed_ms(),
reason: "No TLSA DNSSEC records found".to_string(),
})
.await?;
None
}
Err(err) => {
if matches!(
&err,
mail_auth::Error::Dns(mail_auth::DnsError::RecordNotFound(_))
) {
tx.send(DeliveryStage::TlsaNotFound {
elapsed: now.elapsed_ms(),
reason: "No TLSA records found for MX".to_string(),
})
.await?;
None
} else {
tx.send(DeliveryStage::TlsaLookupError {
elapsed: now.elapsed_ms(),
reason: err.to_string(),
})
.await?;
continue 'outer;
}
}
};
tx.send(DeliveryStage::IpLookupStart).await?;
let now = Instant::now();
let remote_ips = match host.fqdn_hostname() {
let validate_addresses = server.core.smtp.resolvers.dnssec_available
&& host.dnssec_status() == DnssecStatus::Secure;
let (remote_ips, addresses_dnssec_status) = match host.fqdn_hostname() {
HostOrIp::Host(hostname) => {
match server
.ip_lookup(&hostname, IpLookupStrategy::Ipv4thenIpv6, usize::MAX, false)
.ip_lookup(
&hostname,
IpLookupStrategy::Ipv4thenIpv6,
usize::MAX,
validate_addresses,
)
.await
{
Ok((remote_ips, _)) if !remote_ips.is_empty() => remote_ips,
Ok((remote_ips, dnssec_status)) if !remote_ips.is_empty() => {
(remote_ips, dnssec_status)
}
Ok(_) => {
tx.send(DeliveryStage::IpLookupError {
reason: "No IP addresses found for host".to_string(),
@@ -475,7 +419,7 @@ async fn delivery_diagnose(
}
}
}
HostOrIp::Ip(ip) => vec![ip],
HostOrIp::Ip(ip) => (vec![ip], DnssecStatus::Indeterminate),
};
tx.send(DeliveryStage::IpLookupSuccess {
@@ -484,6 +428,95 @@ async fn delivery_diagnose(
})
.await?;
// Fetch TLSA record
tx.send(DeliveryStage::TlsaLookupStart).await?;
let now = Instant::now();
let dane_policy = match host.dane_status(addresses_dnssec_status) {
(DnssecStatus::Secure, _) => {
match server.tlsa_lookup(format!("_25._tcp.{hostname}.")).await {
Ok(TlsaResult::Secure(tlsa)) if tlsa.has_end_entities => {
tx.send(DeliveryStage::TlsaLookupSuccess {
record: tlsa.as_ref().clone(),
elapsed: now.elapsed_ms(),
})
.await?;
Some(tlsa)
}
Ok(TlsaResult::Secure(_)) => {
tx.send(DeliveryStage::TlsaLookupError {
elapsed: now.elapsed_ms(),
reason: "TLSA record does not have end entities".to_string(),
})
.await?;
None
}
Ok(TlsaResult::Bogus) => {
tx.send(DeliveryStage::TlsaLookupError {
elapsed: now.elapsed_ms(),
reason: "Bogus TLSA record".to_string(),
})
.await?;
continue 'outer;
}
Ok(TlsaResult::Missing) => {
tx.send(DeliveryStage::TlsaNotFound {
elapsed: now.elapsed_ms(),
reason: "No TLSA DNSSEC records found".to_string(),
})
.await?;
None
}
Err(err) => {
if matches!(
&err,
mail_auth::Error::Dns(mail_auth::DnsError::RecordNotFound(_))
) {
tx.send(DeliveryStage::TlsaNotFound {
elapsed: now.elapsed_ms(),
reason: "No TLSA records found for MX".to_string(),
})
.await?;
None
} else {
tx.send(DeliveryStage::TlsaLookupError {
elapsed: now.elapsed_ms(),
reason: err.to_string(),
})
.await?;
continue 'outer;
}
}
}
}
(DnssecStatus::Bogus, dnssec_entity) => {
tx.send(DeliveryStage::TlsaLookupError {
elapsed: now.elapsed_ms(),
reason: format!("Bogus {dnssec_entity} records were found"),
})
.await?;
continue 'outer;
}
(_, dnssec_entity) => {
tx.send(DeliveryStage::TlsaNotFound {
elapsed: now.elapsed_ms(),
reason: format!(
"{dnssec_entity} records are not DNSSEC signed, DANE does not apply"
),
})
.await?;
None
}
};
for remote_ip in remote_ips {
// Start connection
tx.send(DeliveryStage::ConnectionStart { remote_ip })
crates/http/src/request.rs
+3 -1
View File
@@ -36,7 +36,7 @@ use hyper::{
server::conn::http1,
service::service_fn,
};
use hyper_util::rt::TokioIo;
use hyper_util::rt::{TokioIo, TokioTimer};
use jmap::{
api::{
ToJmapHttpResponse, event_source::EventSourceHandler, request::RequestHandler,
@@ -690,6 +690,7 @@ async fn handle_session<T: SessionStream>(inner: Arc<Inner>, session: SessionDat
let is_tls = session.stream.is_tls();
if let Err(http_err) = http1::Builder::new()
.timer(TokioTimer::new())
.keep_alive(true)
.serve_connection(
TokioIo::new(session.stream),
@@ -875,6 +876,7 @@ async fn handle_session<T: SessionStream>(inner: Arc<Inner>, session: SessionDat
)
.with_upgrades()
.await
&& !http_err.is_timeout()
{
if http_err.is_parse() {
let server = inner.build_server();
crates/imap-proto/Cargo.toml
+1 -1
View File
@@ -1,6 +1,6 @@
[package]
name = "imap_proto"
version = "0.16.23"
version = "0.16.24"
edition = "2024"
[dependencies]
crates/imap/Cargo.toml
+1 -1
View File
@@ -1,6 +1,6 @@
[package]
name = "imap"
version = "0.16.23"
version = "0.16.24"
edition = "2024"
[dependencies]
crates/imap/src/op/copy_move.rs
+249 -158
View File
@@ -9,6 +9,7 @@ use crate::{
core::{MailboxId, SelectedMailbox, Session, SessionData},
spawn_op,
};
use ahash::AHashMap;
use common::{ipc::PushNotification, network::SessionStream, storage::index::ObjectIndexBuilder};
use email::{
cache::{MessageCacheFetch, email::MessageCacheAccess},
@@ -22,8 +23,13 @@ use email::{
use imap_proto::{
Command, ResponseCode, StatusResponse, protocol::copy_move::Arguments, receiver::Request,
};
use rand::RngExt;
use registry::schema::enums::Permission;
use std::{sync::Arc, time::Instant};
use std::{
ops::RangeInclusive,
sync::Arc,
time::{Duration, Instant},
};
use store::{
ValueKey,
roaring::RoaringBitmap,
@@ -36,6 +42,9 @@ use types::{
type_state::{DataType, StateChange},
};
const MAX_MOVE_RETRIES: u32 = 3;
const MOVE_RETRY_BACKOFF_MS: RangeInclusive<u64> = 1..=15;
impl<T: SessionStream> Session<T> {
pub async fn handle_copy_move(
&mut self,
@@ -236,148 +245,180 @@ impl<T: SessionStream> SessionData<T> {
// Mailboxes are in the same account
let account_id = src_mailbox.id.account_id;
let dest_mailbox_id = UidMailbox::new_unassigned(dest_mailbox_id);
let mut batch = BatchBuilder::new();
let mut written_uids = AHashMap::with_capacity(ids.len());
let mut retries = 0;
for (id, imap_id) in ids {
// Obtain mailbox tags
let data_ = if let Some(result) = self
.get_message_data(account_id, id)
.await
.imap_ctx(&arguments.tag, trc::location!())?
{
result
} else {
continue;
};
loop {
let mut batch = BatchBuilder::new();
copied_ids.clear();
did_move = false;
// Deserialize
let data = data_
.to_unarchived::<MessageData>()
.imap_ctx(&arguments.tag, trc::location!())?;
for (&id, imap_id) in &ids {
// Obtain mailbox tags
let data_ = if let Some(result) = self
.get_message_data(account_id, id)
.await
.imap_ctx(&arguments.tag, trc::location!())?
{
result
} else {
continue;
};
// Make sure the message still belongs to this mailbox
if !data
.inner
.mailboxes
.iter()
.any(|mailbox| mailbox.mailbox_id == src_mailbox.id.mailbox_id)
{
continue;
}
// Deserialize
let data = data_
.to_unarchived::<MessageData>()
.imap_ctx(&arguments.tag, trc::location!())?;
// If the message is already in the destination mailbox, skip it.
if let Some(mailbox) = data
.inner
.mailboxes
.iter()
.find(|mailbox| mailbox.mailbox_id == dest_mailbox_id.mailbox_id)
{
copied_ids.push((imap_id.uid, mailbox.uid.to_native()));
if is_move {
let mut new_data = data.inner.to_builder();
new_data.remove_mailbox(src_mailbox.id.mailbox_id);
batch
.with_account_id(account_id)
.with_collection(Collection::Email)
.with_document(id)
.custom(
ObjectIndexBuilder::new()
.with_current(data)
.with_changes(new_data.seal()),
)
.imap_ctx(&arguments.tag, trc::location!())?
.log_vanished_item(
VanishedCollection::Email,
(src_mailbox.id.mailbox_id, imap_id.uid),
)
.commit_point();
did_move = true;
// Make sure the message still belongs to this mailbox
if !data
.inner
.mailboxes
.iter()
.any(|mailbox| mailbox.mailbox_id == src_mailbox.id.mailbox_id)
{
// Moved by a chunk of a previous attempt
if let Some(&uid) = written_uids.get(&id)
&& data.inner.message_uid(dest_mailbox_id.mailbox_id) == Some(uid)
{
copied_ids.push((imap_id.uid, uid));
did_move = true;
}
continue;
}
continue;
}
// If the message is already in the destination mailbox, skip it.
if let Some(mailbox) = data
.inner
.mailboxes
.iter()
.find(|mailbox| mailbox.mailbox_id == dest_mailbox_id.mailbox_id)
{
let uid = mailbox.uid.to_native();
copied_ids.push((imap_id.uid, uid));
// Prepare changes
let mut new_data = data.inner.to_builder();
if is_move {
let mut new_data = data.inner.to_builder();
new_data.remove_mailbox(src_mailbox.id.mailbox_id);
batch
.with_account_id(account_id)
.with_collection(Collection::Email)
.with_document(id)
.custom(
ObjectIndexBuilder::new()
.with_current(data)
.with_changes(new_data.seal()),
)
.imap_ctx(&arguments.tag, trc::location!())?
.log_vanished_item(
VanishedCollection::Email,
(src_mailbox.id.mailbox_id, imap_id.uid),
)
.commit_point();
written_uids.insert(id, uid);
did_move = true;
}
// Add destination folder
new_data.add_mailbox(dest_mailbox_id);
if is_move {
new_data.remove_mailbox(src_mailbox.id.mailbox_id);
}
continue;
}
// Assign IMAP UIDs
let ids = self
.server
.assign_email_ids(
account_id,
new_data
.mailboxes
.iter()
.filter(|m| m.uid == 0)
.map(|m| m.mailbox_id),
false,
)
.await
.caused_by(trc::location!())?;
// Prepare changes
let mut new_data = data.inner.to_builder();
for (uid_mailbox, uid) in new_data
.mailboxes
.iter_mut()
.filter(|m| m.uid == 0)
.zip(ids)
{
copied_ids.push((imap_id.uid, uid));
uid_mailbox.uid = uid;
}
// Add destination folder
new_data.add_mailbox(dest_mailbox_id);
if is_move {
new_data.remove_mailbox(src_mailbox.id.mailbox_id);
}
// Prepare write batch
batch
.with_account_id(account_id)
.with_collection(Collection::Email)
.with_document(id)
.custom(
ObjectIndexBuilder::new()
.with_current(data)
.with_changes(new_data.seal()),
)
.imap_ctx(&arguments.tag, trc::location!())?;
if is_move {
batch.log_vanished_item(
VanishedCollection::Email,
(src_mailbox.id.mailbox_id, imap_id.uid),
);
}
// Add message to training queue
if dest_mailbox_id.mailbox_id == JUNK_ID {
self.server
.add_account_spam_sample(&mut batch, account_id, id, true, self.session_id)
.await
.imap_ctx(&arguments.tag, trc::location!())?;
} else if src_mailbox.id.mailbox_id == JUNK_ID
&& dest_mailbox_id.mailbox_id != TRASH_ID
{
self.server
.add_account_spam_sample(&mut batch, account_id, id, false, self.session_id)
// Assign IMAP UIDs
let ids = self
.server
.assign_email_ids(
account_id,
new_data
.mailboxes
.iter()
.filter(|m| m.uid == 0)
.map(|m| m.mailbox_id),
false,
)
.await
.caused_by(trc::location!())?;
for (uid_mailbox, uid) in new_data
.mailboxes
.iter_mut()
.filter(|m| m.uid == 0)
.zip(ids)
{
copied_ids.push((imap_id.uid, uid));
written_uids.insert(id, uid);
uid_mailbox.uid = uid;
}
// Prepare write batch
batch
.with_account_id(account_id)
.with_collection(Collection::Email)
.with_document(id)
.custom(
ObjectIndexBuilder::new()
.with_current(data)
.with_changes(new_data.seal()),
)
.imap_ctx(&arguments.tag, trc::location!())?;
if is_move {
batch.log_vanished_item(
VanishedCollection::Email,
(src_mailbox.id.mailbox_id, imap_id.uid),
);
}
// Add message to training queue
if dest_mailbox_id.mailbox_id == JUNK_ID {
self.server
.add_account_spam_sample(
&mut batch,
account_id,
id,
true,
self.session_id,
)
.await
.imap_ctx(&arguments.tag, trc::location!())?;
} else if src_mailbox.id.mailbox_id == JUNK_ID
&& dest_mailbox_id.mailbox_id != TRASH_ID
{
self.server
.add_account_spam_sample(
&mut batch,
account_id,
id,
false,
self.session_id,
)
.await
.imap_ctx(&arguments.tag, trc::location!())?;
}
batch.commit_point();
// Update changelog
if is_move {
did_move = true;
}
}
batch.commit_point();
// Update changelog
if is_move {
did_move = true;
// Write changes
match self.server.commit_batch(batch).await {
Ok(_) => break,
Err(err) => {
retry_after_conflict(err, &mut retries, &arguments.tag, trc::location!())
.await?
}
}
}
// Write changes
self.server
.commit_batch(batch)
.await
.imap_ctx(&arguments.tag, trc::location!())?;
} else {
// Obtain quota for target account
let src_account_id = src_mailbox.id.account_id;
@@ -400,7 +441,7 @@ impl<T: SessionStream> SessionData<T> {
let mut train_batch = BatchBuilder::new();
let mut did_train = false;
train_batch.with_account_id(src_account_id);
for (id, imap_id) in ids {
'next_message: for (id, imap_id) in ids {
match self
.server
.copy_message(
@@ -448,22 +489,27 @@ impl<T: SessionStream> SessionData<T> {
{
copied_ids.push((imap_id.uid, uid));
} else {
let data_ = if let Some(data_) = self
.get_message_data(dest_account_id, existing_id)
.await
.imap_ctx(&arguments.tag, trc::location!())?
{
data_
} else {
continue;
};
let data = data_
.to_unarchived::<MessageData>()
.imap_ctx(&arguments.tag, trc::location!())?;
let mut retries = 0;
loop {
let data_ = if let Some(data_) = self
.get_message_data(dest_account_id, existing_id)
.await
.imap_ctx(&arguments.tag, trc::location!())?
{
data_
} else {
continue 'next_message;
};
let data = data_
.to_unarchived::<MessageData>()
.imap_ctx(&arguments.tag, trc::location!())?;
if let Some(uid) = data.inner.message_uid(dest_mailbox_id) {
copied_ids.push((imap_id.uid, uid));
break;
}
if let Some(uid) = data.inner.message_uid(dest_mailbox_id) {
copied_ids.push((imap_id.uid, uid));
} else {
let mut new_data = data.inner.to_builder();
new_data.add_mailbox(UidMailbox::new_unassigned(dest_mailbox_id));
@@ -504,15 +550,27 @@ impl<T: SessionStream> SessionData<T> {
)
.imap_ctx(&arguments.tag, trc::location!())?;
dest_change_id = self
match self
.server
.commit_batch(batch)
.await
.and_then(|ids| ids.last_change_id(dest_account_id))
.imap_ctx(&arguments.tag, trc::location!())?
.into();
copied_ids.push((imap_id.uid, assigned_uid));
{
Ok(change_id) => {
dest_change_id = change_id.into();
copied_ids.push((imap_id.uid, assigned_uid));
break;
}
Err(err) => {
retry_after_conflict(
err,
&mut retries,
&arguments.tag,
trc::location!(),
)
.await?
}
}
}
}
}
@@ -554,21 +612,33 @@ impl<T: SessionStream> SessionData<T> {
// Untag or delete emails
if !destroy_ids.is_empty() {
let mut batch = BatchBuilder::new();
self.email_untag_or_delete(
src_account_id,
src_mailbox.id.mailbox_id,
&destroy_ids,
&mut batch,
)
.await
.imap_ctx(&arguments.tag, trc::location!())?;
let mut retries = 0;
self.server
.commit_batch(batch)
loop {
let mut batch = BatchBuilder::new();
self.email_untag_or_delete(
src_account_id,
src_mailbox.id.mailbox_id,
&destroy_ids,
&mut batch,
)
.await
.imap_ctx(&arguments.tag, trc::location!())?;
match self.server.commit_batch(batch).await {
Ok(_) => break,
Err(err) => {
retry_after_conflict(
err,
&mut retries,
&arguments.tag,
trc::location!(),
)
.await?
}
}
}
did_move = true;
}
@@ -731,3 +801,24 @@ impl<T: SessionStream> SessionData<T> {
}
}
}
async fn retry_after_conflict(
err: trc::Error,
retries: &mut u32,
tag: &str,
location: &'static str,
) -> trc::Result<()> {
if !err.is_assertion_failure() {
Err(err).imap_ctx(tag, location)
} else if *retries < MAX_MOVE_RETRIES {
*retries += 1;
let backoff = rand::rng().random_range(MOVE_RETRY_BACKOFF_MS);
tokio::time::sleep(Duration::from_millis(backoff)).await;
Ok(())
} else {
Err(trc::ImapEvent::Error
.into_err()
.details("Some messages were modified by another process.")
.id(tag.to_string()))
}
}
crates/jmap-proto/Cargo.toml
+1 -1
View File
@@ -1,6 +1,6 @@
[package]
name = "jmap_proto"
version = "0.16.23"
version = "0.16.24"
edition = "2024"
[dependencies]
crates/jmap-proto/src/method/import.rs
+1 -2
View File
@@ -12,7 +12,6 @@ use crate::{
email::{EmailProperty, EmailValue},
},
request::{
MaybeInvalid,
deserialize::{DeserializeArguments, deserialize_request},
reference::{MaybeIdReference, MaybeResultReference, ResultReference},
},
@@ -33,7 +32,7 @@ pub struct ImportEmailRequest {
#[derive(Debug, Clone, Default)]
pub struct ImportEmail {
pub blob_id: MaybeInvalid<BlobId>,
pub blob_id: MaybeIdReference<BlobId>,
pub mailbox_ids: MaybeResultReference<Vec<MaybeIdReference<Id>>>,
pub keywords: Vec<Keyword>,
pub received_at: Option<UTCDate>,
crates/jmap-proto/src/references/resolve.rs
+4
View File
@@ -392,6 +392,10 @@ impl ResolveReference for ImportEmailRequest {
fn resolve_references(&mut self, response: &Response<'_>) -> trc::Result<()> {
// Resolve email mailbox references
for email in self.emails.values_mut() {
if let MaybeIdReference::Reference(ir) = &email.blob_id {
email.blob_id = MaybeIdReference::Id(response.eval_blob_id_reference(ir)?);
}
match &mut email.mailbox_ids {
MaybeResultReference::Reference(reference) => {
email.mailbox_ids = MaybeResultReference::Value(
crates/jmap-proto/src/request/reference.rs
+6
View File
@@ -105,6 +105,12 @@ impl<V: Default> Default for MaybeResultReference<V> {
}
}
impl<V: FromStr> Default for MaybeIdReference<V> {
fn default() -> Self {
MaybeIdReference::Invalid(String::new())
}
}
impl<T: Default> MaybeResultReference<T> {
pub fn unwrap(self) -> T {
match self {
crates/jmap/Cargo.toml
+1 -1
View File
@@ -1,6 +1,6 @@
[package]
name = "jmap"
version = "0.16.23"
version = "0.16.24"
edition = "2024"
[dependencies]
crates/jmap/src/email/import.rs
+2 -2
View File
@@ -18,7 +18,7 @@ use jmap_proto::{
error::set::{SetError, SetErrorType},
method::import::{ImportEmailRequest, ImportEmailResponse},
object::email::EmailProperty,
request::MaybeInvalid,
request::reference::MaybeIdReference,
types::state::State,
};
use mail_parser::{HeaderName, MessageParser};
@@ -128,7 +128,7 @@ impl EmailImport for Server {
}
}
let MaybeInvalid::Value(blob_id) = email.blob_id else {
let MaybeIdReference::Id(blob_id) = email.blob_id else {
response.not_created.append(
id,
SetError::invalid_properties()
crates/jmap/src/email/set.rs
+5 -2
View File
@@ -854,8 +854,11 @@ impl EmailSet for Server {
new_data.set_mailboxes(
ids.into_expanded_boolean_set()
.filter_map(|id| {
UidMailbox::new_unassigned(
id.try_into_property()?.try_into_id()?.document_id(),
let mailbox_id =
id.try_into_property()?.try_into_id()?.document_id();
UidMailbox::new(
mailbox_id,
data.inner.message_uid(mailbox_id).unwrap_or(0),
)
.into()
})
crates/jmap/src/registry/mapping/bootstrap.rs
+1
View File
@@ -641,6 +641,7 @@ fn map_dns_server(dns_server: &DnsServerBootstrap) -> Option<registry::schema::s
DnsServerBootstrap::Ns1(inner) => DnsServer::Ns1(inner.clone()).into(),
DnsServerBootstrap::OracleCloud(inner) => DnsServer::OracleCloud(inner.clone()).into(),
DnsServerBootstrap::Plesk(inner) => DnsServer::Plesk(inner.clone()).into(),
DnsServerBootstrap::PowerDns(inner) => DnsServer::PowerDns(inner.clone()).into(),
DnsServerBootstrap::Safedns(inner) => DnsServer::Safedns(inner.clone()).into(),
DnsServerBootstrap::Scaleway(inner) => DnsServer::Scaleway(inner.clone()).into(),
DnsServerBootstrap::TencentCloud(inner) => DnsServer::TencentCloud(inner.clone()).into(),
crates/main/Cargo.toml
+1 -1
View File
@@ -7,7 +7,7 @@ keywords = ["imap", "jmap", "smtp", "email", "mail", "webdav", "server"]
categories = ["email"]
# Upstream offers AGPL-3.0-only OR LicenseRef-SEL; inbuxa takes the AGPL only.
license = "AGPL-3.0-only"
version = "0.16.23"
version = "0.16.24"
edition = "2024"
[[bin]]
crates/managesieve/Cargo.toml
+1 -1
View File
@@ -1,6 +1,6 @@
[package]
name = "managesieve"
version = "0.16.23"
version = "0.16.24"
edition = "2024"
[dependencies]
crates/migration/Cargo.toml
+1 -1
View File
@@ -1,6 +1,6 @@
[package]
name = "migration"
version = "0.16.23"
version = "0.16.24"
edition = "2024"
[dependencies]
crates/nlp/Cargo.toml
+1 -1
View File
@@ -1,6 +1,6 @@
[package]
name = "nlp"
version = "0.16.23"
version = "0.16.24"
edition = "2024"
[dependencies]
crates/pop3/Cargo.toml
+1 -1
View File
@@ -1,6 +1,6 @@
[package]
name = "pop3"
version = "0.16.23"
version = "0.16.24"
edition = "2024"
[dependencies]
crates/registry/Cargo.toml
+1 -1
View File
@@ -1,6 +1,6 @@
[package]
name = "registry"
version = "0.16.23"
version = "0.16.24"
edition = "2024"
[dependencies]
crates/registry/src/schema/enums.rs
+2
View File
@@ -620,6 +620,7 @@ pub enum DnsServerBootstrapType {
Vultr = 68,
WebSupport = 69,
YandexCloud = 70,
PowerDns = 71,
}
#[derive(Debug, Clone, Copy, Default, PartialEq, Eq, Hash)]
@@ -696,6 +697,7 @@ pub enum DnsServerType {
Vultr = 67,
WebSupport = 68,
YandexCloud = 69,
PowerDns = 70,
}
#[derive(Debug, Clone, Copy, Default, PartialEq, Eq, Hash)]
crates/registry/src/schema/enums_impl.rs
+8 -2
View File
@@ -3023,6 +3023,7 @@ impl EnumImpl for DnsServerBootstrapType {
b"Vultr" => DnsServerBootstrapType::Vultr,
b"WebSupport" => DnsServerBootstrapType::WebSupport,
b"YandexCloud" => DnsServerBootstrapType::YandexCloud,
b"PowerDns" => DnsServerBootstrapType::PowerDns,
}
.copied()
}
@@ -3100,6 +3101,7 @@ impl EnumImpl for DnsServerBootstrapType {
DnsServerBootstrapType::Vultr => "Vultr",
DnsServerBootstrapType::WebSupport => "WebSupport",
DnsServerBootstrapType::YandexCloud => "YandexCloud",
DnsServerBootstrapType::PowerDns => "PowerDns",
}
}
@@ -3180,11 +3182,12 @@ impl EnumImpl for DnsServerBootstrapType {
68 => Some(DnsServerBootstrapType::Vultr),
69 => Some(DnsServerBootstrapType::WebSupport),
70 => Some(DnsServerBootstrapType::YandexCloud),
71 => Some(DnsServerBootstrapType::PowerDns),
_ => None,
}
}
const COUNT: usize = 71;
const COUNT: usize = 72;
}
impl serde::Serialize for DnsServerBootstrapType {
@@ -3281,6 +3284,7 @@ impl EnumImpl for DnsServerType {
b"Vultr" => DnsServerType::Vultr,
b"WebSupport" => DnsServerType::WebSupport,
b"YandexCloud" => DnsServerType::YandexCloud,
b"PowerDns" => DnsServerType::PowerDns,
}
.copied()
}
@@ -3357,6 +3361,7 @@ impl EnumImpl for DnsServerType {
DnsServerType::Vultr => "Vultr",
DnsServerType::WebSupport => "WebSupport",
DnsServerType::YandexCloud => "YandexCloud",
DnsServerType::PowerDns => "PowerDns",
}
}
@@ -3436,11 +3441,12 @@ impl EnumImpl for DnsServerType {
67 => Some(DnsServerType::Vultr),
68 => Some(DnsServerType::WebSupport),
69 => Some(DnsServerType::YandexCloud),
70 => Some(DnsServerType::PowerDns),
_ => None,
}
}
const COUNT: usize = 70;
const COUNT: usize = 71;
}
impl serde::Serialize for DnsServerType {
crates/registry/src/schema/properties.rs
+1
View File
@@ -1042,6 +1042,7 @@ pub enum Property {
SentinelUsername = 914,
Separator = 97,
ServerHostname = 121,
ServerId = 934,
Servers = 308,
ServiceAccountJson = 316,
ServiceName = 913,
crates/registry/src/schema/properties_impl.rs
+6 -1
View File
@@ -1195,6 +1195,7 @@ impl EnumImpl for Property {
b"sentinelUsername" => Property::SentinelUsername,
b"separator" => Property::Separator,
b"serverHostname" => Property::ServerHostname,
b"serverId" => Property::ServerId,
b"servers" => Property::Servers,
b"serviceAccountJson" => Property::ServiceAccountJson,
b"serviceName" => Property::ServiceName,
@@ -2134,6 +2135,7 @@ impl EnumImpl for Property {
Property::SentinelUsername => "sentinelUsername",
Property::Separator => "separator",
Property::ServerHostname => "serverHostname",
Property::ServerId => "serverId",
Property::Servers => "servers",
Property::ServiceAccountJson => "serviceAccountJson",
Property::ServiceName => "serviceName",
@@ -3077,6 +3079,7 @@ impl EnumImpl for Property {
914 => Some(Property::SentinelUsername),
97 => Some(Property::Separator),
121 => Some(Property::ServerHostname),
934 => Some(Property::ServerId),
308 => Some(Property::Servers),
316 => Some(Property::ServiceAccountJson),
913 => Some(Property::ServiceName),
@@ -3227,7 +3230,7 @@ impl EnumImpl for Property {
}
}
const COUNT: usize = 934;
const COUNT: usize = 935;
}
impl serde::Serialize for Property {
@@ -4493,6 +4496,7 @@ impl ObjectInner {
ObjectInner::DnsServer(DnsServer::Vultr(obj)) => obj.member_tenant_id,
ObjectInner::DnsServer(DnsServer::WebSupport(obj)) => obj.member_tenant_id,
ObjectInner::DnsServer(DnsServer::YandexCloud(obj)) => obj.member_tenant_id,
ObjectInner::DnsServer(DnsServer::PowerDns(obj)) => obj.member_tenant_id,
ObjectInner::Domain(obj) => obj.member_tenant_id,
ObjectInner::MailingList(obj) => obj.member_tenant_id,
ObjectInner::OAuthClient(obj) => obj.member_tenant_id,
@@ -4595,6 +4599,7 @@ impl ObjectInner {
ObjectInner::DnsServer(DnsServer::Vultr(obj)) => obj.member_tenant_id = Some(id),
ObjectInner::DnsServer(DnsServer::WebSupport(obj)) => obj.member_tenant_id = Some(id),
ObjectInner::DnsServer(DnsServer::YandexCloud(obj)) => obj.member_tenant_id = Some(id),
ObjectInner::DnsServer(DnsServer::PowerDns(obj)) => obj.member_tenant_id = Some(id),
ObjectInner::Domain(obj) => obj.member_tenant_id = Some(id),
ObjectInner::MailingList(obj) => obj.member_tenant_id = Some(id),
ObjectInner::OAuthClient(obj) => obj.member_tenant_id = Some(id),
crates/registry/src/schema/structs.rs
+27
View File
@@ -1457,6 +1457,7 @@ pub enum DnsServer {
Vultr(DnsServerCloud),
WebSupport(DnsServerWebSupport),
YandexCloud(DnsServerYandexCloud),
PowerDns(DnsServerPowerDns),
}
#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
@@ -1672,6 +1673,7 @@ pub enum DnsServerBootstrap {
Vultr(DnsServerCloud),
WebSupport(DnsServerWebSupport),
YandexCloud(DnsServerYandexCloud),
PowerDns(DnsServerPowerDns),
}
#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
@@ -2435,6 +2437,31 @@ pub struct DnsServerPorkbun {
pub propagation_delay: Option<Duration>,
}
#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
#[serde(default)]
pub struct DnsServerPowerDns {
#[serde(rename = "apiKey")]
pub api_key: SecretKey,
#[serde(rename = "endpoint")]
pub endpoint: Option<String>,
#[serde(rename = "serverId")]
pub server_id: Option<String>,
#[serde(rename = "description")]
pub description: String,
#[serde(rename = "memberTenantId")]
pub member_tenant_id: Option<Id>,
#[serde(rename = "timeout")]
pub timeout: Duration,
#[serde(rename = "ttl")]
pub ttl: Duration,
#[serde(rename = "pollingInterval")]
pub polling_interval: Duration,
#[serde(rename = "propagationTimeout")]
pub propagation_timeout: Duration,
#[serde(rename = "propagationDelay")]
pub propagation_delay: Option<Duration>,
}
#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
#[serde(default)]
pub struct DnsServerRoute53 {
crates/registry/src/schema/structs_impl.rs
+179
View File
@@ -10559,6 +10559,7 @@ impl ObjectImpl for DnsServer {
DnsServer::Vultr(inner) => inner.validate(errors),
DnsServer::WebSupport(inner) => inner.validate(errors),
DnsServer::YandexCloud(inner) => inner.validate(errors),
DnsServer::PowerDns(inner) => inner.validate(errors),
}
}
@@ -10772,6 +10773,9 @@ impl ObjectImpl for DnsServer {
DnsServer::YandexCloud(object) => {
object.index(i);
}
DnsServer::PowerDns(object) => {
object.index(i);
}
}
}
}
@@ -11064,6 +11068,10 @@ impl Pickle for DnsServer {
69u16.pickle(out);
inner.pickle(out);
}
DnsServer::PowerDns(inner) => {
70u16.pickle(out);
inner.pickle(out);
}
}
}
@@ -11139,6 +11147,7 @@ impl Pickle for DnsServer {
67 => Pickle::unpickle(stream).map(DnsServer::Vultr),
68 => Pickle::unpickle(stream).map(DnsServer::WebSupport),
69 => Pickle::unpickle(stream).map(DnsServer::YandexCloud),
70 => Pickle::unpickle(stream).map(DnsServer::PowerDns),
_ => None,
}
}
@@ -11635,6 +11644,13 @@ impl IntoValue for DnsServer {
.insert_unchecked(Property::Type, JmapValue::Str("YandexCloud".into()));
obj
}
DnsServer::PowerDns(obj) => {
let mut obj = obj.into_value();
obj.as_object_mut()
.unwrap()
.insert_unchecked(Property::Type, JmapValue::Str("PowerDns".into()));
obj
}
}
}
}
@@ -11719,6 +11735,7 @@ impl RegistryJsonPatch for DnsServer {
DnsServerType::Vultr => *self = DnsServer::Vultr(Default::default()),
DnsServerType::WebSupport => *self = DnsServer::WebSupport(Default::default()),
DnsServerType::YandexCloud => *self = DnsServer::YandexCloud(Default::default()),
DnsServerType::PowerDns => *self = DnsServer::PowerDns(Default::default()),
}
}
match self {
@@ -11792,6 +11809,7 @@ impl RegistryJsonPatch for DnsServer {
DnsServer::Vultr(inner) => inner.patch(pointer, value),
DnsServer::WebSupport(inner) => inner.patch(pointer, value),
DnsServer::YandexCloud(inner) => inner.patch(pointer, value),
DnsServer::PowerDns(inner) => inner.patch(pointer, value),
}
}
}
@@ -11869,6 +11887,7 @@ impl DnsServer {
DnsServer::Vultr(_) => DnsServerType::Vultr,
DnsServer::WebSupport(_) => DnsServerType::WebSupport,
DnsServer::YandexCloud(_) => DnsServerType::YandexCloud,
DnsServer::PowerDns(_) => DnsServerType::PowerDns,
}
}
}
@@ -12704,6 +12723,7 @@ impl DnsServerBootstrap {
DnsServerBootstrap::Vultr(inner) => inner.validate(errors),
DnsServerBootstrap::WebSupport(inner) => inner.validate(errors),
DnsServerBootstrap::YandexCloud(inner) => inner.validate(errors),
DnsServerBootstrap::PowerDns(inner) => inner.validate(errors),
}
}
}
@@ -12999,6 +13019,10 @@ impl Pickle for DnsServerBootstrap {
70u16.pickle(out);
inner.pickle(out);
}
DnsServerBootstrap::PowerDns(inner) => {
71u16.pickle(out);
inner.pickle(out);
}
}
}
@@ -13075,6 +13099,7 @@ impl Pickle for DnsServerBootstrap {
68 => Pickle::unpickle(stream).map(DnsServerBootstrap::Vultr),
69 => Pickle::unpickle(stream).map(DnsServerBootstrap::WebSupport),
70 => Pickle::unpickle(stream).map(DnsServerBootstrap::YandexCloud),
71 => Pickle::unpickle(stream).map(DnsServerBootstrap::PowerDns),
_ => None,
}
}
@@ -13576,6 +13601,13 @@ impl IntoValue for DnsServerBootstrap {
.insert_unchecked(Property::Type, JmapValue::Str("YandexCloud".into()));
obj
}
DnsServerBootstrap::PowerDns(obj) => {
let mut obj = obj.into_value();
obj.as_object_mut()
.unwrap()
.insert_unchecked(Property::Type, JmapValue::Str("PowerDns".into()));
obj
}
}
}
}
@@ -13793,6 +13825,9 @@ impl RegistryJsonPatch for DnsServerBootstrap {
DnsServerBootstrapType::YandexCloud => {
*self = DnsServerBootstrap::YandexCloud(Default::default())
}
DnsServerBootstrapType::PowerDns => {
*self = DnsServerBootstrap::PowerDns(Default::default())
}
}
}
match self {
@@ -13867,6 +13902,7 @@ impl RegistryJsonPatch for DnsServerBootstrap {
DnsServerBootstrap::Vultr(inner) => inner.patch(pointer, value),
DnsServerBootstrap::WebSupport(inner) => inner.patch(pointer, value),
DnsServerBootstrap::YandexCloud(inner) => inner.patch(pointer, value),
DnsServerBootstrap::PowerDns(inner) => inner.patch(pointer, value),
}
}
}
@@ -13945,6 +13981,7 @@ impl DnsServerBootstrap {
DnsServerBootstrap::Vultr(_) => DnsServerBootstrapType::Vultr,
DnsServerBootstrap::WebSupport(_) => DnsServerBootstrapType::WebSupport,
DnsServerBootstrap::YandexCloud(_) => DnsServerBootstrapType::YandexCloud,
DnsServerBootstrap::PowerDns(_) => DnsServerBootstrapType::PowerDns,
}
}
}
@@ -18228,6 +18265,148 @@ impl RegistryJsonPropertyPatch for DnsServerPorkbun {
}
}
impl DnsServerPowerDns {
fn validate(&self, errors: &mut Vec<ValidationError>) -> bool {
let neb = errors.len();
let value = &self.api_key;
value.validate(errors);
if let Some(value) = &self.endpoint {
if value.is_empty() {
errors.push(ValidationError::required(Property::Endpoint));
}
}
if let Some(value) = &self.server_id {
if value.is_empty() {
errors.push(ValidationError::required(Property::ServerId));
}
}
let value = &self.description;
if value.is_empty() {
errors.push(ValidationError::required(Property::Description));
}
if let Some(value) = &self.member_tenant_id {
if !value.is_valid() {
errors.push(ValidationError::required(Property::MemberTenantId));
}
}
errors.len() == neb
}
fn index<'x>(&'x self, i: &mut IndexBuilder<'x>) {
i.foreign_key(ObjectType::Tenant, self.member_tenant_id, None);
if let Some(value) = &self.member_tenant_id {
i.search(Property::MemberTenantId, value);
}
}
}
impl Pickle for DnsServerPowerDns {
fn pickle(&self, out: &mut Vec<u8>) {
self.api_key.pickle(out);
self.endpoint.pickle(out);
self.server_id.pickle(out);
self.description.pickle(out);
self.member_tenant_id.pickle(out);
self.timeout.pickle(out);
self.ttl.pickle(out);
self.polling_interval.pickle(out);
self.propagation_timeout.pickle(out);
self.propagation_delay.pickle(out);
}
fn unpickle(stream: &mut crate::pickle::PickledStream<'_>) -> Option<Self> {
let mut this = Self::default();
this.api_key = Pickle::unpickle(stream)?;
this.endpoint = Pickle::unpickle(stream)?;
this.server_id = Pickle::unpickle(stream)?;
this.description = Pickle::unpickle(stream)?;
this.member_tenant_id = Pickle::unpickle(stream)?;
this.timeout = Pickle::unpickle(stream)?;
this.ttl = Pickle::unpickle(stream)?;
this.polling_interval = Pickle::unpickle(stream)?;
this.propagation_timeout = Pickle::unpickle(stream)?;
this.propagation_delay = Pickle::unpickle(stream)?;
Some(this)
}
}
impl Default for DnsServerPowerDns {
fn default() -> Self {
Self {
api_key: Default::default(),
endpoint: Default::default(),
server_id: Default::default(),
description: Default::default(),
member_tenant_id: Default::default(),
timeout: Duration::from_millis(30000),
ttl: Duration::from_millis(300000),
polling_interval: Duration::from_millis(15000),
propagation_timeout: Duration::from_millis(60000),
propagation_delay: Default::default(),
}
}
}
impl IntoValue for DnsServerPowerDns {
fn into_value(self) -> JmapValue<'static> {
let mut map = jmap_tools::Map::with_capacity(12);
map.insert_unchecked(Property::ApiKey, self.api_key.into_value());
map.insert_unchecked(Property::Endpoint, self.endpoint.into_value());
map.insert_unchecked(Property::ServerId, self.server_id.into_value());
map.insert_unchecked(Property::Description, self.description.into_value());
map.insert_unchecked(Property::MemberTenantId, self.member_tenant_id.into_value());
map.insert_unchecked(Property::Timeout, self.timeout.into_value());
map.insert_unchecked(Property::Ttl, self.ttl.into_value());
map.insert_unchecked(
Property::PollingInterval,
self.polling_interval.into_value(),
);
map.insert_unchecked(
Property::PropagationTimeout,
self.propagation_timeout.into_value(),
);
map.insert_unchecked(
Property::PropagationDelay,
self.propagation_delay.into_value(),
);
JmapValue::Object(map)
}
}
impl RegistryJsonPropertyPatch for DnsServerPowerDns {
fn patch_property<'x>(
&mut self,
mut pointer: JsonPointerPatch<'_>,
value: JmapValue<'x>,
) -> PatchResult<'x> {
match pointer.next_property() {
Some(Property::ApiKey) => self.api_key.patch(pointer, value),
Some(Property::Endpoint) => self
.endpoint
.patch(pointer.with_validators(&[StringValidator::Trim]), value),
Some(Property::ServerId) => self
.server_id
.patch(pointer.with_validators(&[StringValidator::Trim]), value),
Some(Property::Description) => self
.description
.patch(pointer.with_validators(&[StringValidator::Trim]), value),
Some(Property::MemberTenantId) => self
.member_tenant_id
.patch(pointer.assert_can_set_tenant()?, value),
Some(Property::Timeout) => self.timeout.patch(pointer, value),
Some(Property::Ttl) => self.ttl.patch(pointer, value),
Some(Property::PollingInterval) => self.polling_interval.patch(pointer, value),
Some(Property::PropagationTimeout) => self.propagation_timeout.patch(pointer, value),
Some(Property::PropagationDelay) => self.propagation_delay.patch(pointer, value),
Some(Property::Type) => Ok(MaybeUnpatched::Unpatched {
property: Property::Type,
value,
}),
_ => Err(PatchError::new(pointer, "Invalid property")),
}
}
}
impl DnsServerRoute53 {
fn validate(&self, errors: &mut Vec<ValidationError>) -> bool {
let neb = errors.len();
crates/registry/src/utils/mod.rs
+1
View File
@@ -14,6 +14,7 @@ pub mod dkim;
pub mod http;
pub mod report;
pub mod secret;
pub mod spam;
pub mod task;
impl Roles {
crates/registry/src/utils/spam.rs
+59
View File
@@ -0,0 +1,59 @@
/*
* SPDX-FileCopyrightText: 2020 Stalwart Labs LLC <hello@stalw.art>
*
* SPDX-License-Identifier: AGPL-3.0-only OR LicenseRef-SEL
*/
use crate::schema::prelude::{SpamDnsblServer, SpamRule};
impl SpamRule {
pub fn enable(&self) -> bool {
match self {
SpamRule::Any(rule) => rule.enable,
SpamRule::Url(rule) => rule.enable,
SpamRule::Domain(rule) => rule.enable,
SpamRule::Email(rule) => rule.enable,
SpamRule::Ip(rule) => rule.enable,
SpamRule::Header(rule) => rule.enable,
SpamRule::Body(rule) => rule.enable,
}
}
pub fn set_enable(&mut self, enable: bool) {
match self {
SpamRule::Any(rule) => rule.enable = enable,
SpamRule::Url(rule) => rule.enable = enable,
SpamRule::Domain(rule) => rule.enable = enable,
SpamRule::Email(rule) => rule.enable = enable,
SpamRule::Ip(rule) => rule.enable = enable,
SpamRule::Header(rule) => rule.enable = enable,
SpamRule::Body(rule) => rule.enable = enable,
}
}
}
impl SpamDnsblServer {
pub fn enable(&self) -> bool {
match self {
SpamDnsblServer::Any(server) => server.enable,
SpamDnsblServer::Url(server) => server.enable,
SpamDnsblServer::Domain(server) => server.enable,
SpamDnsblServer::Email(server) => server.enable,
SpamDnsblServer::Ip(server) => server.enable,
SpamDnsblServer::Header(server) => server.enable,
SpamDnsblServer::Body(server) => server.enable,
}
}
pub fn set_enable(&mut self, enable: bool) {
match self {
SpamDnsblServer::Any(server) => server.enable = enable,
SpamDnsblServer::Url(server) => server.enable = enable,
SpamDnsblServer::Domain(server) => server.enable = enable,
SpamDnsblServer::Email(server) => server.enable = enable,
SpamDnsblServer::Ip(server) => server.enable = enable,
SpamDnsblServer::Header(server) => server.enable = enable,
SpamDnsblServer::Body(server) => server.enable = enable,
}
}
}
crates/scim-proto/Cargo.toml
+1 -1
View File
@@ -1,6 +1,6 @@
[package]
name = "scim-proto"
version = "0.16.23"
version = "0.16.24"
edition = "2024"
[dependencies]
crates/scim/Cargo.toml
+1 -1
View File
@@ -1,6 +1,6 @@
[package]
name = "scim"
version = "0.16.23"
version = "0.16.24"
edition = "2024"
[dependencies]
crates/services/Cargo.toml
+1 -1
View File
@@ -1,6 +1,6 @@
[package]
name = "services"
version = "0.16.23"
version = "0.16.24"
edition = "2024"
[dependencies]
crates/services/src/state_manager/http.rs
+71 -57
View File
@@ -9,7 +9,7 @@ use super::{
ece::{ECE_WEBPUSH_MAX_PLAINTEXT_SIZE, WEBPUSH_MAX_BODY_SIZE, ece_encrypt},
email_push::build_email_push_object,
};
use crate::state_manager::PushRegistration;
use crate::state_manager::{PushBatch, PushRegistration};
use calcard::jscalendar::JSCalendarDateTime;
use common::{Server, ipc::PushNotification, network::webpush::Vapid};
use email::push::{PushSubscription, Urgency};
@@ -29,7 +29,7 @@ use std::time::{Duration, Instant};
use store::write::now;
use tokio::sync::mpsc;
use trc::PushSubscriptionEvent;
use types::{id::Id, type_state::DataType};
use types::id::Id;
use utils::map::vec_map::VecMap;
const MAX_ERROR_RESPONSE_LEN: usize = 1024;
@@ -48,34 +48,29 @@ impl PushRegistration {
pub fn send(
&mut self,
id: Id,
push_client: &Client,
push_tx: mpsc::Sender<Event>,
push_timeout: Duration,
server: Server,
) {
let subscription = self.server.clone();
let push_client = self.client.clone();
let notifications = std::mem::take(&mut self.notifications);
let push_client = push_client.clone();
let batch = std::mem::take(&mut self.pending);
self.in_flight = true;
self.last_request = Instant::now();
tokio::spawn(async move {
let mut changed: VecMap<Id, VecMap<DataType, State>> = VecMap::new();
let vapid = server.core.jmap.vapid.as_ref();
let mut email_pushes: VecMap<Id, EmailPushObject> = VecMap::new();
let mut failed_state_change = false;
let mut failed = PushBatch::default();
let mut failed_email_pushes = Vec::new();
let mut failed_calendar_alerts = Vec::new();
for notification in &notifications {
for notification in &batch.notifications {
match notification {
PushNotification::StateChange(state_change) => {
for type_state in state_change.types {
changed
.get_mut_or_insert(state_change.account_id.into())
.set(type_state, State::Exact(state_change.change_id));
}
}
PushNotification::StateChange(_) => {}
PushNotification::CalendarAlert(calendar_alert) => {
let payload = PushObject::CalendarAlert {
account_id: calendar_alert.account_id.into(),
@@ -86,12 +81,12 @@ impl PushRegistration {
}),
alert_id: calendar_alert.alert_id.clone(),
};
if !http_request(
if !post_object(
&push_client,
&subscription,
serde_json::to_string(&payload).unwrap().into_bytes(),
&payload,
push_timeout,
server.core.jmap.vapid.as_ref(),
vapid,
Urgency::Normal,
)
.await
@@ -155,18 +150,23 @@ impl PushRegistration {
}
}
if !changed.is_empty() {
failed_state_change = !http_request(
if !batch.state_changes.is_empty() {
let payload = PushObject::StateChange {
changed: batch.state_changes,
};
if !post_object(
&push_client,
&subscription,
serde_json::to_string(&PushObject::StateChange { changed })
.unwrap()
.into_bytes(),
&payload,
push_timeout,
server.core.jmap.vapid.as_ref(),
vapid,
Urgency::Normal,
)
.await;
.await
&& let PushObject::StateChange { changed } = payload
{
failed.state_changes = changed;
}
}
for (account_id, email_push) in email_pushes {
@@ -180,12 +180,12 @@ impl PushRegistration {
state: email_push.change_id.map(State::Exact),
};
if !http_request(
if !post_object(
&push_client,
&subscription,
serde_json::to_string(&payload).unwrap().into_bytes(),
&payload,
push_timeout,
server.core.jmap.vapid.as_ref(),
vapid,
email_push.urgency,
)
.await
@@ -194,44 +194,26 @@ impl PushRegistration {
}
}
let result = if !failed_state_change
let result = if failed.state_changes.is_empty()
&& failed_email_pushes.is_empty()
&& failed_calendar_alerts.is_empty()
{
Event::DeliverySuccess { id }
} else {
let mut failed_notifications = Vec::with_capacity(
failed_state_change as usize
+ failed_email_pushes.len()
+ failed_calendar_alerts.len(),
);
for notification in notifications {
match &notification {
PushNotification::StateChange(_) => {
if failed_state_change {
failed_notifications.push(notification);
}
}
failed.notifications = batch
.notifications
.into_iter()
.filter(|notification| match notification {
PushNotification::StateChange(_) => false,
PushNotification::EmailPush(email_push) => {
if failed_email_pushes.contains(&email_push.account_id) {
failed_notifications.push(notification);
}
failed_email_pushes.contains(&email_push.account_id)
}
PushNotification::CalendarAlert(calendar_alert) => {
if failed_calendar_alerts
.contains(&(calendar_alert.account_id, calendar_alert.event_id))
{
failed_notifications.push(notification);
}
}
}
}
PushNotification::CalendarAlert(calendar_alert) => failed_calendar_alerts
.contains(&(calendar_alert.account_id, calendar_alert.event_id)),
})
.collect();
Event::DeliveryFailure {
id,
notifications: failed_notifications,
}
Event::DeliveryFailure { id, failed }
};
push_tx.send(result).await.ok();
@@ -239,6 +221,38 @@ impl PushRegistration {
}
}
async fn post_object(
push_client: &Client,
subscription: &PushSubscription,
object: &PushObject,
push_timeout: Duration,
vapid: Option<&Vapid>,
urgency: Urgency,
) -> bool {
match serde_json::to_vec(object) {
Ok(body) => {
http_request(
push_client,
subscription,
body,
push_timeout,
vapid,
urgency,
)
.await
}
Err(err) => {
trc::event!(
PushSubscription(PushSubscriptionEvent::Error),
Details = "Failed to serialize push object",
Url = subscription.url.to_string(),
Reason = err.to_string()
);
true
}
}
}
pub(crate) fn build_push_client() -> Client {
utils::http::http_client_builder(cfg!(feature = "test_mode"))
.redirect(Policy::custom(|attempt| match attempt.previous().last() {
crates/services/src/state_manager/manager.rs
+26 -22
View File
@@ -14,7 +14,7 @@ use common::{
};
use std::{sync::Arc, time::Instant};
use store::ahash::AHashMap;
use tokio::sync::mpsc;
use tokio::sync::mpsc::{self, error::TrySendError};
use trc::ServerEvent;
#[derive(Default)]
@@ -99,7 +99,7 @@ pub fn spawn_push_router(inner: Arc<Inner>, mut change_rx: mpsc::Receiver<PushEv
} => {
// Publish event to cluster
if broadcast
&& let Some(broadcast_tx) = &inner.ipc.broadcast_tx.clone()
&& let Some(broadcast_tx) = inner.ipc.broadcast_tx.as_ref()
&& broadcast_tx
.send(BroadcastEvent::PushNotification(notification.clone()))
.await
@@ -117,26 +117,30 @@ pub fn spawn_push_router(inner: Arc<Inner>, mut change_rx: mpsc::Receiver<PushEv
for subscriber in &subscribers.ipc {
if let Some(notification) = notification.filter_types(&subscriber.types)
{
if subscriber.is_valid() {
let subscriber_tx = subscriber.tx.clone();
match subscriber.tx.try_send(notification) {
Ok(()) => {}
Err(TrySendError::Full(notification)) => {
let subscriber_tx = subscriber.tx.clone();
tokio::spawn(async move {
// Timeout after 500ms in case there is a blocked client
if subscriber_tx
.send_timeout(notification, SEND_TIMEOUT)
.await
.is_err()
{
trc::event!(
Server(ServerEvent::ThreadError),
Details =
"Error sending state change to subscriber.",
CausedBy = trc::location!()
);
}
});
} else {
purge_needed = true;
tokio::spawn(async move {
// Timeout after 500ms in case there is a blocked client
if subscriber_tx
.send_timeout(notification, SEND_TIMEOUT)
.await
.is_err()
{
trc::event!(
Server(ServerEvent::ThreadError),
Details =
"Error sending state change to subscriber.",
CausedBy = trc::location!()
);
}
});
}
Err(TrySendError::Closed(_)) => {
purge_needed = true;
}
}
}
}
@@ -159,7 +163,7 @@ pub fn spawn_push_router(inner: Arc<Inner>, mut change_rx: mpsc::Receiver<PushEv
} => {
// Publish event to cluster
if broadcast
&& let Some(broadcast_tx) = &inner.ipc.broadcast_tx.clone()
&& let Some(broadcast_tx) = inner.ipc.broadcast_tx.as_ref()
&& broadcast_tx
.send(BroadcastEvent::PushServerUpdate(account_id))
.await
crates/services/src/state_manager/mod.rs
+140 -17
View File
@@ -14,14 +14,14 @@ pub mod push;
use common::ipc::PushNotification;
use email::push::PushSubscription;
use reqwest::Client;
use jmap_proto::types::state::State;
use std::{
sync::Arc,
time::{Duration, Instant},
};
use tokio::sync::mpsc;
use types::{id::Id, type_state::DataType};
use utils::map::bitmap::Bitmap;
use utils::map::{bitmap::Bitmap, vec_map::VecMap};
const PURGE_EVERY: Duration = Duration::from_secs(3600);
const SEND_TIMEOUT: Duration = Duration::from_millis(500);
@@ -40,26 +40,22 @@ pub struct PushRegistration {
member_account_ids: Vec<u32>,
num_attempts: u32,
last_request: Instant,
notifications: Vec<PushNotification>,
pending: PushBatch,
in_flight: bool,
client: Client,
}
#[derive(Debug, Default)]
pub struct PushBatch {
state_changes: VecMap<Id, VecMap<DataType, State>>,
notifications: Vec<PushNotification>,
}
#[derive(Debug)]
pub enum Event {
Push {
notification: PushNotification,
},
Update {
account_id: u32,
},
DeliverySuccess {
id: Id,
},
DeliveryFailure {
id: Id,
notifications: Vec<PushNotification>,
},
Push { notification: PushNotification },
Update { account_id: u32 },
DeliverySuccess { id: Id },
DeliveryFailure { id: Id, failed: PushBatch },
Reset,
}
@@ -68,3 +64,130 @@ impl IpcSubscriber {
!self.tx.is_closed()
}
}
impl PushBatch {
pub fn push(&mut self, notification: PushNotification) {
match notification {
PushNotification::StateChange(state_change) => {
if !state_change.types.is_empty() {
let states = self
.state_changes
.get_mut_or_insert(Id::from(state_change.account_id));
for data_type in state_change.types {
merge_state(states, data_type, state_change.change_id);
}
}
}
notification => self.notifications.push(notification),
}
}
pub fn merge_failed(&mut self, failed: PushBatch) {
for (account_id, failed_states) in failed.state_changes {
let states = self.state_changes.get_mut_or_insert(account_id);
for (data_type, state) in failed_states {
if let State::Exact(change_id) = state {
merge_state(states, data_type, change_id);
}
}
}
if !failed.notifications.is_empty() {
let mut notifications = failed.notifications;
notifications.append(&mut self.notifications);
self.notifications = notifications;
}
}
pub fn is_empty(&self) -> bool {
self.state_changes.is_empty() && self.notifications.is_empty()
}
pub fn clear(&mut self) {
self.state_changes.clear();
self.notifications.clear();
}
}
fn merge_state(states: &mut VecMap<DataType, State>, data_type: DataType, change_id: u64) {
match states.get_mut(&data_type) {
Some(State::Exact(current)) if *current >= change_id => {}
Some(state) => *state = State::Exact(change_id),
None => states.append(data_type, State::Exact(change_id)),
}
}
#[cfg(test)]
mod tests {
use super::PushBatch;
use common::ipc::{EmailPush, PushNotification};
use jmap_proto::types::state::State;
use types::{
id::Id,
type_state::{DataType, StateChange},
};
use utils::map::bitmap::Bitmap;
fn state_change<const N: usize>(change_id: u64, types: [DataType; N]) -> PushNotification {
PushNotification::StateChange(StateChange {
account_id: 1,
change_id,
types: Bitmap::from_iter(types),
})
}
fn email_push(email_id: u32) -> PushNotification {
PushNotification::EmailPush(EmailPush {
account_id: 1,
email_id,
change_id: email_id.into(),
})
}
#[test]
fn batch_keeps_newest_state_per_type() {
let mut batch = PushBatch::default();
batch.push(state_change(5, [DataType::Email]));
batch.push(state_change(7, [DataType::Email, DataType::Mailbox]));
batch.push(state_change(6, [DataType::Mailbox]));
let mut failed = PushBatch::default();
failed.push(state_change(
4,
[DataType::Email, DataType::Mailbox, DataType::Thread],
));
batch.merge_failed(failed);
let states = batch.state_changes.get(&Id::from(1u32)).unwrap();
assert_eq!(states.len(), 3);
assert_eq!(states.get(&DataType::Email), Some(&State::Exact(7)));
assert_eq!(states.get(&DataType::Mailbox), Some(&State::Exact(7)));
assert_eq!(states.get(&DataType::Thread), Some(&State::Exact(4)));
assert!(!batch.is_empty());
batch.clear();
assert!(batch.is_empty());
}
#[test]
fn failed_notifications_are_retried_first() {
let mut batch = PushBatch::default();
batch.push(email_push(3));
let mut failed = PushBatch::default();
failed.push(email_push(1));
failed.push(email_push(2));
batch.merge_failed(failed);
let email_ids = batch
.notifications
.iter()
.filter_map(|notification| match notification {
PushNotification::EmailPush(email_push) => Some(email_push.email_id),
_ => None,
})
.collect::<Vec<_>>();
assert_eq!(email_ids, [1, 2, 3]);
assert!(batch.state_changes.is_empty());
}
}
crates/services/src/state_manager/push.rs
+157 -89
View File
@@ -8,13 +8,14 @@ use super::{
Event,
http::{build_push_client, http_request},
};
use crate::state_manager::PushRegistration;
use crate::state_manager::{PushBatch, PushRegistration};
use common::{
BuildServer, IPC_CHANNEL_BUFFER, Inner, LONG_1Y_SLUMBER, Server,
auth::BuildAccessToken,
ipc::{PushEvent, PushNotification},
};
use email::push::{PushSubscription, PushSubscriptions, Urgency};
use reqwest::Client;
use std::{
collections::hash_map::Entry,
sync::Arc,
@@ -36,7 +37,10 @@ pub fn spawn_push_manager(inner: Arc<Inner>) -> mpsc::Sender<Event> {
tokio::spawn(async move {
let mut push_servers: AHashMap<Id, PushRegistration> = AHashMap::default();
let mut account_push_ids: AHashMap<u32, AHashSet<Id>> = AHashMap::default();
let mut last_verify: AHashMap<u32, Instant> = AHashMap::default();
let mut last_verify: AHashMap<u32, (Instant, u32)> = AHashMap::default();
let mut pending_verify: AHashMap<u32, (Instant, Arc<PushSubscription>)> =
AHashMap::default();
let mut next_verify: Option<Instant> = None;
let mut last_retry = Instant::now();
let mut retry_timeout = LONG_1Y_SLUMBER;
let mut retry_ids = AHashSet::default();
@@ -90,10 +94,9 @@ pub fn spawn_push_manager(inner: Arc<Inner>) -> mpsc::Sender<Event> {
last_request: Instant::now()
- (server.core.jmap.push_throttle
+ Duration::from_millis(1)),
notifications: Vec::new(),
pending: PushBatch::default(),
server: subscription.clone(),
in_flight: false,
client: push_client.clone(),
},
);
}
@@ -129,8 +132,39 @@ pub fn spawn_push_manager(inner: Arc<Inner>) -> mpsc::Sender<Event> {
}
loop {
if let Some(verify_due) = next_verify {
let current_instant = Instant::now();
if verify_due <= current_instant {
let server = inner.build_server();
let push_timeout = server.core.jmap.push_timeout;
let current_time = now();
next_verify = None;
pending_verify.retain(|account_id, (verify_due, subscription)| {
if *verify_due > current_instant {
next_verify =
Some(next_verify.map_or(*verify_due, |next| next.min(*verify_due)));
true
} else {
if subscription.expires > current_time {
last_verify.insert(*account_id, (current_instant, subscription.id));
send_verification(
&push_client,
subscription.clone(),
&server,
push_timeout,
);
}
false
}
});
}
}
// Wait for the next event or timeout
let event_or_timeout = tokio::time::timeout(retry_timeout, push_rx.recv()).await;
let wait_timeout = next_verify.map_or(retry_timeout, |verify_due| {
retry_timeout.min(verify_due.saturating_duration_since(Instant::now()))
});
let event_or_timeout = tokio::time::timeout(wait_timeout, push_rx.recv()).await;
// Load settings
let server = inner.build_server();
@@ -193,10 +227,9 @@ pub fn spawn_push_manager(inner: Arc<Inner>) -> mpsc::Sender<Event> {
num_attempts: 0,
last_request: Instant::now()
- (push_throttle + Duration::from_millis(1)),
notifications: Vec::new(),
pending: PushBatch::default(),
server: subscription.clone(),
in_flight: false,
client: push_client.clone(),
});
}
}
@@ -213,52 +246,56 @@ pub fn spawn_push_manager(inner: Arc<Inner>) -> mpsc::Sender<Event> {
#[cfg(feature = "test_mode")]
if subscription.url.contains("skip_checks") {
last_verify.insert(
account_id,
current_time - (push_verify_timeout + Duration::from_millis(1)),
);
last_verify.remove(&account_id);
}
if last_verify
match last_verify
.get(&account_id)
.map(|last_verify| {
current_time - *last_verify > push_verify_timeout
.map(|(verified_at, verified_id)| {
(*verified_at + push_verify_timeout, *verified_id)
})
.unwrap_or(true)
.filter(|(verify_due, _)| *verify_due >= current_time)
{
let core = server.core.clone();
let push_client = push_client.clone();
tokio::spawn(async move {
http_request(
None => {
last_verify.retain(|_, (verified_at, _)| {
current_time.duration_since(*verified_at)
<= push_verify_timeout
});
last_verify.insert(account_id, (current_time, subscription.id));
pending_verify.remove(&account_id);
send_verification(
&push_client,
&subscription,
format!(
concat!(
"{{\"@type\":\"PushVerification\",",
"\"pushSubscriptionId\":\"{}\",",
"\"verificationCode\":\"{}\"}}"
),
Id::from(subscription.id),
subscription.verification_code
)
.into_bytes(),
subscription,
&server,
push_timeout,
core.jmap.vapid.as_ref(),
Urgency::Normal,
)
.await;
});
last_verify.insert(account_id, current_time);
} else {
trc::event!(
PushSubscription(PushSubscriptionEvent::Error),
Details = "Failed to verify push subscription",
Url = subscription.url.clone(),
AccountId = account_id,
Reason = "Too many requests"
);
);
}
Some((_, verified_id)) if verified_id == subscription.id => {
trc::event!(
PushSubscription(PushSubscriptionEvent::Error),
Details = "Failed to verify push subscription",
Url = subscription.url.clone(),
AccountId = account_id,
Reason = "Too many requests"
);
pending_verify.remove(&account_id);
}
Some((verify_due, _)) => {
trc::event!(
PushSubscription(PushSubscriptionEvent::Error),
Details = "Push subscription verification deferred",
Url = subscription.url.clone(),
AccountId = account_id,
Reason = "Too many requests"
);
next_verify = Some(
next_verify.map_or(verify_due, |next| next.min(verify_due)),
);
pending_verify.insert(account_id, (verify_due, subscription));
}
}
} else {
pending_verify.remove(&account_id);
}
// Update subscriptions
@@ -342,7 +379,7 @@ pub fn spawn_push_manager(inner: Arc<Inner>) -> mpsc::Sender<Event> {
);
}
subscription.notifications.push(notification);
subscription.pending.push(notification);
let last_request = subscription.last_request.elapsed();
if !subscription.in_flight
@@ -354,6 +391,7 @@ pub fn spawn_push_manager(inner: Arc<Inner>) -> mpsc::Sender<Event> {
{
subscription.send(
*id,
&push_client,
push_tx.clone(),
push_timeout,
server.clone(),
@@ -402,19 +440,25 @@ pub fn spawn_push_manager(inner: Arc<Inner>) -> mpsc::Sender<Event> {
Event::Reset => {
push_servers.clear();
account_push_ids.clear();
pending_verify.clear();
next_verify = None;
}
Event::DeliverySuccess { id } => {
if let Some(subscription) = push_servers.get_mut(&id) {
subscription.num_attempts = 0;
subscription.in_flight = false;
retry_ids.remove(&id);
if subscription.pending.is_empty() {
retry_ids.remove(&id);
} else {
retry_ids.insert(id);
}
}
}
Event::DeliveryFailure { id, notifications } => {
Event::DeliveryFailure { id, failed } => {
if let Some(subscription) = push_servers.get_mut(&id) {
subscription.last_request = Instant::now();
subscription.num_attempts += 1;
subscription.notifications.extend(notifications);
subscription.pending.merge_failed(failed);
subscription.in_flight = false;
retry_ids.insert(id);
}
@@ -430,52 +474,46 @@ pub fn spawn_push_manager(inner: Arc<Inner>) -> mpsc::Sender<Event> {
let last_retry_elapsed = last_retry.elapsed();
if last_retry_elapsed >= push_retry_interval {
let mut remove_ids = Vec::with_capacity(retry_ids.len());
retry_ids.retain(|retry_id| {
let Some(subscription) = push_servers.get_mut(retry_id) else {
return false;
};
let last_request = subscription.last_request.elapsed();
let is_due = !subscription.in_flight
&& ((subscription.num_attempts == 0 && last_request >= push_throttle)
|| (subscription.num_attempts > 0
&& last_request >= push_attempt_interval));
if !is_due {
return true;
}
for retry_id in &retry_ids {
if let Some(subscription) = push_servers.get_mut(retry_id) {
let last_request = subscription.last_request.elapsed();
if !subscription.in_flight
&& ((subscription.num_attempts == 0
&& last_request >= push_throttle)
|| (subscription.num_attempts > 0
&& last_request >= push_attempt_interval))
{
if subscription.num_attempts < push_attempts_max {
subscription.send(
*retry_id,
push_tx.clone(),
push_timeout,
server.clone(),
);
} else {
trc::event!(
PushSubscription(PushSubscriptionEvent::Error),
Details = "Failed to deliver push subscription",
Url = subscription.server.url.clone(),
Reason = "Too many failed attempts"
);
subscription.notifications.clear();
subscription.num_attempts = 0;
}
remove_ids.push(*retry_id);
}
if subscription.num_attempts < push_attempts_max {
subscription.send(
*retry_id,
&push_client,
push_tx.clone(),
push_timeout,
server.clone(),
);
} else {
remove_ids.push(*retry_id);
}
}
trc::event!(
PushSubscription(PushSubscriptionEvent::Error),
Details = "Failed to deliver push subscription",
Url = subscription.server.url.clone(),
Reason = "Too many failed attempts"
);
if remove_ids.len() < retry_ids.len() {
for remove_id in remove_ids {
retry_ids.remove(&remove_id);
subscription.pending.clear();
subscription.num_attempts = 0;
}
false
});
if retry_ids.is_empty() {
LONG_1Y_SLUMBER
} else {
last_retry = Instant::now();
push_retry_interval
} else {
retry_ids.clear();
LONG_1Y_SLUMBER
}
} else {
push_retry_interval - last_retry_elapsed
@@ -489,6 +527,36 @@ pub fn spawn_push_manager(inner: Arc<Inner>) -> mpsc::Sender<Event> {
push_tx_
}
fn send_verification(
push_client: &Client,
subscription: Arc<PushSubscription>,
server: &Server,
push_timeout: Duration,
) {
let core = server.core.clone();
let push_client = push_client.clone();
tokio::spawn(async move {
http_request(
&push_client,
&subscription,
format!(
concat!(
"{{\"@type\":\"PushVerification\",",
"\"pushSubscriptionId\":\"{}\",",
"\"verificationCode\":\"{}\"}}"
),
Id::from(subscription.id),
subscription.verification_code
)
.into_bytes(),
push_timeout,
core.jmap.vapid.as_ref(),
Urgency::Normal,
)
.await;
});
}
async fn load_push_subscriptions(
server: &Server,
account_id: u32,
crates/services/src/task_manager/spam_classifier.rs
+302 -184
View File
@@ -15,13 +15,13 @@ use common::{
use registry::{
schema::{
enums::TaskSpamFilterMaintenanceType,
prelude::ObjectType,
prelude::{Object, ObjectInner, ObjectType},
structs::{
HttpLookup, MemoryLookupKey, SpamDnsblServer, SpamFileExtension, SpamRule, SpamTag,
TaskSpamFilterMaintenance,
},
},
types::EnumImpl,
types::{EnumImpl, ObjectImpl},
};
use spam_filter::modules::classifier::SpamClassifier;
use std::time::{Duration, Instant};
@@ -100,13 +100,90 @@ struct Rules {
file_exts: Vec<SpamFileExtension>,
}
#[derive(Default)]
trait UpstreamObject: ObjectImpl + PartialEq + From<Object> + Into<ObjectInner> {
fn replacement_for(self, _local: &Self) -> Option<Self> {
Some(self)
}
// inbuxa: the object as its fingerprint sees it. Switching a rule on or
// off isn't an edit, so `enable` is left out.
fn without_enable(self) -> Self {
self
}
}
impl UpstreamObject for SpamRule {
fn replacement_for(mut self, local: &Self) -> Option<Self> {
self.set_enable(local.enable());
Some(self)
}
fn without_enable(mut self) -> Self {
self.set_enable(true);
self
}
}
impl UpstreamObject for SpamDnsblServer {
fn replacement_for(mut self, local: &Self) -> Option<Self> {
self.set_enable(local.enable());
Some(self)
}
fn without_enable(mut self) -> Self {
self.set_enable(true);
self
}
}
impl UpstreamObject for HttpLookup {
fn replacement_for(mut self, local: &Self) -> Option<Self> {
self.enable = local.enable;
Some(self)
}
fn without_enable(mut self) -> Self {
self.enable = true;
self
}
}
impl UpstreamObject for SpamTag {
fn replacement_for(self, _local: &Self) -> Option<Self> {
None
}
}
impl UpstreamObject for MemoryLookupKey {}
impl UpstreamObject for SpamFileExtension {}
struct RuleUpdateResult {
success: usize,
already_exists: usize,
object_type: ObjectType,
added: usize,
updated: usize,
unchanged: usize,
kept: usize,
failed: usize,
}
impl RuleUpdateResult {
fn new(object_type: ObjectType) -> Self {
RuleUpdateResult {
object_type,
added: 0,
updated: 0,
unchanged: 0,
kept: 0,
failed: 0,
}
}
fn has_changes(&self) -> bool {
self.added + self.updated > 0
}
}
async fn update_spam_rules(server: &Server) -> trc::Result<TaskResult> {
let started = Instant::now();
let bundled = server.core.spam.spam_rules_url.is_none();
@@ -121,171 +198,67 @@ async fn update_spam_rules(server: &Server) -> trc::Result<TaskResult> {
}
};
let registry = server.registry();
let mut stats: AHashMap<ObjectType, RuleUpdateResult> = AHashMap::new();
let settings = [
apply_upstream(server, rules.rules).await?,
apply_upstream(server, rules.dnsbls).await?,
apply_upstream(server, rules.tags).await?,
apply_upstream(server, rules.file_exts).await?,
];
let lookups = [
apply_upstream(server, rules.http_lookups).await?,
apply_upstream(server, rules.key_lookups).await?,
];
let mut reload_settings = false;
let mut reload_lookups = false;
for rule in rules.rules {
match registry.write(RegistryWrite::insert(&rule.into())).await? {
RegistryWriteResult::Success(_) => {
stats.entry(ObjectType::SpamRule).or_default().success += 1;
reload_settings = true;
}
RegistryWriteResult::PrimaryKeyConflict { .. } => {
stats
.entry(ObjectType::SpamRule)
.or_default()
.already_exists += 1;
}
_ => {
stats.entry(ObjectType::SpamRule).or_default().failed += 1;
}
let mut reload_errors = Vec::new();
for object in [
settings
.iter()
.any(RuleUpdateResult::has_changes)
.then_some(ObjectType::SpamRule),
lookups
.iter()
.any(RuleUpdateResult::has_changes)
.then_some(ObjectType::MemoryLookupKey),
]
.into_iter()
.flatten()
{
if let Err(reason) = reload_and_broadcast(server, object).await {
reload_errors.push(reason);
}
}
for dnsbl in rules.dnsbls {
match registry.write(RegistryWrite::insert(&dnsbl.into())).await? {
RegistryWriteResult::Success(_) => {
stats
.entry(ObjectType::SpamDnsblServer)
.or_default()
.success += 1;
reload_settings = true;
}
RegistryWriteResult::PrimaryKeyConflict { .. } => {
stats
.entry(ObjectType::SpamDnsblServer)
.or_default()
.already_exists += 1;
}
_ => {
stats.entry(ObjectType::SpamDnsblServer).or_default().failed += 1;
}
}
}
for tag in rules.tags {
match registry.write(RegistryWrite::insert(&tag.into())).await? {
RegistryWriteResult::Success(_) => {
stats.entry(ObjectType::SpamTag).or_default().success += 1;
reload_settings = true;
}
RegistryWriteResult::PrimaryKeyConflict { .. } => {
stats.entry(ObjectType::SpamTag).or_default().already_exists += 1;
}
_ => {
stats.entry(ObjectType::SpamTag).or_default().failed += 1;
}
}
}
for lookup in rules.http_lookups {
match registry
.write(RegistryWrite::insert(&lookup.into()))
.await?
{
RegistryWriteResult::Success(_) => {
stats.entry(ObjectType::HttpLookup).or_default().success += 1;
reload_lookups = true;
}
RegistryWriteResult::PrimaryKeyConflict { .. } => {
stats
.entry(ObjectType::HttpLookup)
.or_default()
.already_exists += 1;
}
_ => {
stats.entry(ObjectType::HttpLookup).or_default().failed += 1;
}
}
}
for key_lookup in rules.key_lookups {
match registry
.write(RegistryWrite::insert(&key_lookup.into()))
.await?
{
RegistryWriteResult::Success(_) => {
stats
.entry(ObjectType::MemoryLookupKey)
.or_default()
.success += 1;
reload_lookups = true;
}
RegistryWriteResult::PrimaryKeyConflict { .. } => {
stats
.entry(ObjectType::MemoryLookupKey)
.or_default()
.already_exists += 1;
}
_ => {
stats.entry(ObjectType::MemoryLookupKey).or_default().failed += 1;
}
}
}
for ext in rules.file_exts {
match registry.write(RegistryWrite::insert(&ext.into())).await? {
RegistryWriteResult::Success(_) => {
stats
.entry(ObjectType::SpamFileExtension)
.or_default()
.success += 1;
reload_settings = true;
}
RegistryWriteResult::PrimaryKeyConflict { .. } => {
stats
.entry(ObjectType::SpamFileExtension)
.or_default()
.already_exists += 1;
}
_ => {
stats
.entry(ObjectType::SpamFileExtension)
.or_default()
.failed += 1;
}
}
}
if reload_settings {
if let Err(err) =
Box::pin(server.reload_registry(RegistryChange::Reload(ObjectType::SpamRule))).await
{
trc::error!(err.details("Failed to reload registry after updating spam rules"));
}
server
.cluster_broadcast(BroadcastEvent::RegistryChange(RegistryChange::Reload(
ObjectType::SpamRule,
)))
.await;
}
if reload_lookups {
if let Err(err) =
Box::pin(server.reload_registry(RegistryChange::Reload(ObjectType::MemoryLookupKey)))
.await
{
trc::error!(err.details("Failed to reload registry after updating spam rules"));
}
server
.cluster_broadcast(BroadcastEvent::RegistryChange(RegistryChange::Reload(
ObjectType::MemoryLookupKey,
)))
.await;
}
// inbuxa: AU-1.10: what the update added, as one audit record
let added = stats
let failed: usize = settings
.iter()
.filter(|(_, result)| result.success > 0)
.map(|(object_type, result)| format!("{} {}", result.success, object_type.as_str()))
.collect::<Vec<_>>();
if !added.is_empty() {
let mut added = added;
added.sort();
.chain(&lookups)
.map(|result| result.failed)
.sum();
// inbuxa: AU-1.10: what the update changed, as one audit record
let summary = |count: fn(&RuleUpdateResult) -> usize| {
let mut parts = settings
.iter()
.chain(&lookups)
.filter(|result| count(result) > 0)
.map(|result| format!("{} {}", count(result), result.object_type.as_str()))
.collect::<Vec<_>>();
parts.sort();
parts.join(", ")
};
let details = [
("added", summary(|result| result.added)),
("replaced", summary(|result| result.updated)),
("kept as edited locally", summary(|result| result.kept)),
("failed", summary(|result| result.failed)),
]
.into_iter()
.filter(|(_, part)| !part.is_empty())
.map(|(what, part)| format!("{what} {part}"))
.collect::<Vec<_>>();
if settings
.iter()
.chain(&lookups)
.any(RuleUpdateResult::has_changes)
{
server
.audit_note(inbuxa_features::audit::Record {
at: std::time::SystemTime::now()
@@ -301,7 +274,7 @@ async fn update_spam_rules(server: &Server) -> trc::Result<TaskResult> {
..Default::default()
},
changes: vec![],
details: Some(format!("Rules update added {}", added.join(", "))),
details: Some(format!("Rules update {}", details.join("; "))),
reason: None,
outcome: inbuxa_features::audit::Outcome::success(),
})
@@ -310,27 +283,173 @@ async fn update_spam_rules(server: &Server) -> trc::Result<TaskResult> {
trc::event!(
Spam(SpamEvent::RulesUpdated),
Details = stats
Details = settings
.into_iter()
.map(|(object_type, result)| {
.chain(lookups)
.map(|result| {
Value::Array(vec![
Value::String(object_type.as_str().into()),
Value::from(result.success),
Value::from(result.already_exists),
Value::String(result.object_type.as_str().into()),
Value::from(result.added),
Value::from(result.updated),
Value::from(result.unchanged),
Value::from(result.failed),
Value::from(result.kept),
])
})
.collect::<Vec<_>>(),
Elapsed = started.elapsed(),
);
// inbuxa: so the next start knows these bundled rules are in
if bundled {
spam_rules::set_applied_version(server.store(), spam_rules::BUNDLED_SPAM_RULES_VERSION)
.await?;
if !reload_errors.is_empty() {
Ok(TaskResult::permanent(format!(
"Spam rules were stored but not activated ({}); fix the logged errors and run Reload settings",
reload_errors.join("; ")
)))
} else if failed > 0 {
Ok(TaskResult::permanent(format!(
"{failed} spam filter objects failed to import or update"
)))
} else {
// inbuxa: so the next start knows these bundled rules are in. Only
// once they all are: a failed update runs again on the next start.
if bundled {
spam_rules::set_applied_version(server.store(), spam_rules::BUNDLED_SPAM_RULES_APPLIED)
.await?;
}
Ok(TaskResult::Success(vec![]))
}
}
async fn reload_and_broadcast(server: &Server, object: ObjectType) -> Result<(), String> {
match Box::pin(server.reload_registry(RegistryChange::Reload(object))).await {
Ok(result) => {
result.log();
if result.has_errors() {
return Err(format!("{} configuration errors", result.errors.len()));
}
server
.cluster_broadcast(BroadcastEvent::RegistryChange(RegistryChange::Reload(
object,
)))
.await;
Ok(())
}
Err(err) => {
let reason = err.to_string();
trc::error!(err.details("Failed to reload registry after updating spam rules"));
Err(reason)
}
}
}
async fn apply_upstream<T: UpstreamObject>(
server: &Server,
objects: Vec<T>,
) -> trc::Result<RuleUpdateResult> {
let registry = server.registry();
let mut result = RuleUpdateResult::new(T::OBJECT);
for upstream in objects {
// inbuxa: every object the update writes is fingerprinted, and an
// existing object is replaced only while it still matches: one an
// admin edited is kept as it is.
let upstream_print = fingerprint(&upstream);
let written = Object::from(upstream.clone());
let existing_id = match registry.write(RegistryWrite::insert(&written)).await? {
RegistryWriteResult::Success(id) => {
spam_rules::set_fingerprint(server.store(), T::OBJECT, id.id(), &upstream_print)
.await?;
result.added += 1;
continue;
}
RegistryWriteResult::PrimaryKeyConflict { existing_id, .. }
if existing_id.object() == T::OBJECT =>
{
existing_id
}
RegistryWriteResult::PrimaryKeyConflict { .. } => {
result.unchanged += 1;
continue;
}
_ => {
result.failed += 1;
continue;
}
};
let Some(local) = registry.get(existing_id).await? else {
result.failed += 1;
continue;
};
let revision = local.revision;
let local = T::from(local);
let local_print = fingerprint(&local);
let written_print =
spam_rules::fingerprint(server.store(), T::OBJECT, existing_id.id().id()).await?;
if local_print == upstream_print {
// The same as upstream's. An install from before fingerprints
// gets one here, so the next release can replace it.
if written_print.as_deref() != Some(upstream_print.as_str()) {
spam_rules::set_fingerprint(
server.store(),
T::OBJECT,
existing_id.id().id(),
&upstream_print,
)
.await?;
}
result.unchanged += 1;
continue;
}
if written_print.as_deref() != Some(local_print.as_str()) {
// Changed since the update wrote it, or never written by one.
result.kept += 1;
continue;
}
let Some(replacement) = upstream
.replacement_for(&local)
.filter(|replacement| replacement != &local)
else {
result.unchanged += 1;
continue;
};
let replacement = Object::from(replacement);
let local = Object::with_revision(local.into(), revision);
match registry
.write(RegistryWrite::update(
existing_id.id(),
&replacement,
&local,
))
.await?
{
RegistryWriteResult::Success(_) => {
spam_rules::set_fingerprint(
server.store(),
T::OBJECT,
existing_id.id().id(),
&upstream_print,
)
.await?;
result.updated += 1
}
_ => result.failed += 1,
}
}
Ok(TaskResult::Success(vec![]))
Ok(result)
}
/// inbuxa: a digest of an object's content, `enable` aside, in hex.
fn fingerprint<T: UpstreamObject>(object: &T) -> String {
use sha2::Digest;
sha2::Sha256::digest(serde_json::to_vec(&object.clone().without_enable()).unwrap_or_default())
.iter()
.map(|byte| format!("{byte:02x}"))
.collect()
}
async fn fetch_spam_rules(server: &Server) -> Result<Rules, RuleUpdateError> {
@@ -347,13 +466,12 @@ async fn fetch_spam_rules(server: &Server) -> Result<Rules, RuleUpdateError> {
reason,
}),
};
let rules_json: AHashMap<String, Vec<serde_json::Value>> =
bytes.and_then(|bytes| {
serde_json::from_slice(&bytes).map_err(|err| RuleUpdateError {
typ: TaskFailureType::Permanent,
reason: format!("Failed to parse spam rules JSON: {err}"),
})
})?;
let rules_json: AHashMap<String, Vec<serde_json::Value>> = bytes.and_then(|bytes| {
serde_json::from_slice(&bytes).map_err(|err| RuleUpdateError {
typ: TaskFailureType::Permanent,
reason: format!("Failed to parse spam rules JSON: {err}"),
})
})?;
let mut rules = Rules::default();
for (object_type, values) in rules_json {
crates/smtp/Cargo.toml
+1 -1
View File
@@ -6,7 +6,7 @@ homepage = "https://inbuxa.org"
keywords = ["smtp", "email", "mail", "server"]
categories = ["email"]
license = "AGPL-3.0-only OR LicenseRef-SEL"
version = "0.16.23"
version = "0.16.24"
edition = "2024"
[dependencies]
crates/smtp/src/inbound/data.rs
+6 -1
View File
@@ -696,7 +696,12 @@ impl<T: SessionStream> Session<T> {
.map(|a| a.as_str())
.unwrap_or_default(),
)
.with_message(parsed_message);
.with_message(
edited_message
.as_deref()
.and_then(|message| MessageParser::new().parse(message))
.unwrap_or(parsed_message),
);
let modifications = match self.run_script(script_id, script.clone(), params).await {
ScriptResult::Accept { modifications } => modifications,
crates/smtp/src/inbound/hooks/message.rs
+1 -1
View File
@@ -132,7 +132,7 @@ impl<T: SessionStream> Session<T> {
name: "X-Quarantine".into(),
value: "true".into(),
});
FilterResponse::accept()
continue;
}
};
crates/smtp/src/outbound/delivery.rs
+42 -10
View File
@@ -531,11 +531,23 @@ impl QueuedMessage {
};
// Obtain remote hosts list
let mx_unvalidated = mx_config.is_some() && !tls_strategy.try_dane();
let mx_list;
if let Some(mx_config) = mx_config {
// Lookup MX
let time = Instant::now();
mx_list = match server.mx_lookup(domain).await {
let mx_lookup = if mx_unvalidated {
server
.core
.smtp
.resolvers
.dns
.mx_lookup(domain, Some(&server.inner.cache.dns_mx))
.await
} else {
server.mx_lookup(domain).await
};
mx_list = match mx_lookup {
Ok(mx) => mx,
Err(mail_auth::Error::Dns(mail_auth::DnsError::RecordNotFound(_))) => {
trc::event!(
@@ -674,6 +686,33 @@ impl QueuedMessage {
message.span_id,
);
let validated_host;
let remote_host = if mx_unvalidated && tls_strategy.try_dane() {
let time = Instant::now();
let dnssec_status = match server.mx_lookup(domain).await {
Ok(mx) => mx.dnssec_status,
Err(mail_auth::Error::Dns(mail_auth::DnsError::RecordNotFound(_))) => {
DnssecStatus::Indeterminate
}
Err(err) => {
trc::event!(
Delivery(DeliveryEvent::MxLookupFailed),
SpanId = message.span_id,
Domain = domain.to_string(),
CausedBy = trc::Error::from(err.clone()),
Elapsed = time.elapsed(),
);
last_status = Status::from_mail_auth_error(domain, err);
continue 'next_host;
}
};
validated_host = remote_host.with_dnssec_status(dnssec_status);
&validated_host
} else {
remote_host
};
// Obtain source and remote IPs
let time = Instant::now();
let validate_addresses = server.core.smtp.resolvers.dnssec_available
@@ -722,15 +761,8 @@ impl QueuedMessage {
let time = Instant::now();
let strict = tls_strategy.is_dane_required();
let (dnssec_status, dnssec_entity) = match remote_host.dnssec_status() {
DnssecStatus::Secure => match addresses_dnssec_status {
status @ (DnssecStatus::Insecure | DnssecStatus::Bogus) => {
(status, "A/AAAA")
}
_ => (DnssecStatus::Secure, "MX"),
},
status => (status, "MX"),
};
let (dnssec_status, dnssec_entity) =
remote_host.dane_status(addresses_dnssec_status);
match dnssec_status {
DnssecStatus::Secure => {
crates/smtp/src/outbound/mod.rs
+28 -1
View File
@@ -350,12 +350,39 @@ impl NextHop<'_> {
}
}
fn dnssec_status(&self) -> DnssecStatus {
pub fn dnssec_status(&self) -> DnssecStatus {
match self {
NextHop::MX { dnssec_status, .. } => *dnssec_status,
NextHop::Relay(_) => DnssecStatus::Indeterminate,
}
}
fn with_dnssec_status(&self, dnssec_status: DnssecStatus) -> Self {
match self {
NextHop::MX {
is_implicit,
host,
config,
..
} => NextHop::MX {
is_implicit: *is_implicit,
host,
config,
dnssec_status,
},
NextHop::Relay(relay) => NextHop::Relay(relay),
}
}
pub fn dane_status(&self, addresses: DnssecStatus) -> (DnssecStatus, &'static str) {
match self.dnssec_status() {
DnssecStatus::Secure => match addresses {
status @ (DnssecStatus::Insecure | DnssecStatus::Bogus) => (status, "A/AAAA"),
_ => (DnssecStatus::Secure, "MX"),
},
status => (status, "MX"),
}
}
}
impl DeliveryResult {
crates/smtp/src/queue/manager.rs
+5
View File
@@ -67,6 +67,10 @@ impl SpawnQueue for mpsc::Receiver<QueueEvent> {
Queue::new(core, self).start().await;
});
}
fn discard(mut self) {
tokio::spawn(async move { while self.recv().await.is_some() {} });
}
}
const BACK_PRESSURE_WARN_INTERVAL: Duration = Duration::from_secs(60);
@@ -510,6 +514,7 @@ impl Recipient {
pub trait SpawnQueue {
fn spawn(self, core: Arc<Inner>);
fn discard(self);
}
impl QueueStats {
crates/spam-filter/Cargo.toml
+1 -1
View File
@@ -1,6 +1,6 @@
[package]
name = "spam-filter"
version = "0.16.23"
version = "0.16.24"
edition = "2024"
[dependencies]
crates/spam-filter/src/analysis/url.rs
+28 -2
View File
@@ -203,7 +203,7 @@ impl SpamFilterAnalyzeUrl for Server {
if !ctx.result.has_tag("URL_REDIRECTOR_NESTED") {
let mut redirect_count = 1;
let mut url_redirect = Cow::Borrowed(url.element.url.as_str());
let mut url_redirect = url.element.request_url();
while redirect_count <= 3 {
match http_get_header(
@@ -224,7 +224,8 @@ impl SpamFilterAnalyzeUrl for Server {
)
.await
{
url_redirect = Cow::Owned(location.url);
url_redirect =
Cow::Owned(location.request_url().into_owned());
redirect_count += 1;
continue;
} else {
@@ -487,6 +488,15 @@ impl<'x> UrlParts<'x> {
.is_some_and(|url| url.host.fqdn.starts_with("www."))
}
pub fn request_url(&self) -> Cow<'_, str> {
let url = self.url_original.trim();
if self.has_scheme {
Cow::Borrowed(url)
} else {
Cow::Owned([HTTPS_SCHEME, url].concat())
}
}
fn parse(url: &str) -> Option<UrlParsed> {
url.parse::<Uri>().ok().and_then(|parts| {
parts
@@ -505,3 +515,19 @@ impl<'x> UrlParts<'x> {
}
}
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn request_url_keeps_case() {
let url = UrlParts::new(" https://Bit.ly/3AbCdEf ");
assert_eq!(url.url, "https://bit.ly/3abcdef");
assert_eq!(url.request_url(), "https://Bit.ly/3AbCdEf");
let url = UrlParts::no_scheme("Bit.ly/3AbCdEf");
assert_eq!(url.url, "https://bit.ly/3abcdef");
assert_eq!(url.request_url(), "https://Bit.ly/3AbCdEf");
}
}
crates/spam-filter/src/modules/classifier.rs
+1 -3
View File
@@ -259,9 +259,7 @@ impl SpamClassifier for Server {
remove_entries = true;
}
if trainer.last_id == 0 {
trainer.last_id = id;
}
trainer.last_id = trainer.last_id.max(id);
Ok(true)
},
crates/store/Cargo.toml
+1 -1
View File
@@ -1,6 +1,6 @@
[package]
name = "store"
version = "0.16.23"
version = "0.16.24"
edition = "2024"
[dependencies]
crates/store/src/backend/http/lookup.rs
+142 -25
View File
@@ -4,21 +4,19 @@
* SPDX-License-Identifier: AGPL-3.0-only OR LicenseRef-SEL
*/
use super::{HttpStore, HttpStoreConfig};
use crate::{Value, backend::http::HttpStoreFormat, write::now};
use ahash::AHashMap;
use compact_str::ToCompactString;
use rand::seq::IndexedRandom;
use std::{
borrow::Cow,
io::{BufRead, BufReader},
sync::{Arc, atomic::Ordering},
time::Instant,
};
use ahash::AHashMap;
use compact_str::ToCompactString;
use rand::seq::IndexedRandom;
use utils::HttpLimitResponse;
use crate::{Value, backend::http::HttpStoreFormat, write::now};
use super::HttpStore;
const BROWSER_USER_AGENTS: [&str; 5] = [
"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36",
"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Edge/120.0.0.0 Safari/537.36",
@@ -36,7 +34,7 @@ pub(crate) trait HttpStoreGet {
impl HttpStoreGet for Arc<HttpStore> {
fn get(&self, key: &str) -> Option<Value<'static>> {
self.refresh();
self.entries.load().get(key).cloned()
self.entries.load().get(lookup_key(key).as_ref()).cloned()
}
fn contains(&self, key: &str) -> bool {
@@ -59,7 +57,7 @@ impl HttpStoreGet for Arc<HttpStore> {
}
self.refresh();
self.entries.load().contains_key(key)
self.entries.load().contains_key(lookup_key(key).as_ref())
}
fn refresh(&self) {
@@ -142,9 +140,10 @@ impl HttpStore {
Box::new(&bytes[..])
};
let mut entries = AHashMap::new();
for (pos, line) in BufReader::new(reader).lines().enumerate() {
let line_ = line.map_err(|err| {
let entries = self
.config
.parse_entries(BufReader::new(reader))
.map_err(|err| {
trc::StoreEvent::HttpStoreError
.into_err()
.reason(err)
@@ -153,11 +152,31 @@ impl HttpStore {
.details("Failed to read line")
})?;
match &self.config.format {
trc::event!(
Store(trc::StoreEvent::HttpStoreFetch),
Url = self.config.url.to_compact_string(),
Total = entries.len(),
Elapsed = time.elapsed(),
);
Ok(entries)
}
}
impl HttpStoreConfig {
fn parse_entries(
&self,
reader: impl BufRead,
) -> std::io::Result<AHashMap<String, Value<'static>>> {
let mut entries = AHashMap::new();
for (pos, line) in reader.lines().enumerate() {
let line_ = line?;
match &self.format {
HttpStoreFormat::List => {
let line = line_.trim();
if !line.is_empty() {
entries.insert(line.to_string(), Value::Integer(1));
entries.insert(lookup_key(line).into_owned(), Value::Integer(1));
}
}
HttpStoreFormat::Csv {
@@ -188,12 +207,12 @@ impl HttpStore {
}
} else if col_num == *index_key {
entry_key.push(ch);
if entry_key.len() > self.config.max_entry_size {
if entry_key.len() > self.max_entry_size {
break;
}
} else if index_value.is_some_and(|v| col_num == v) {
entry_value.push(ch);
if entry_value.len() > self.config.max_entry_size {
if entry_value.len() > self.max_entry_size {
break;
}
}
@@ -209,24 +228,122 @@ impl HttpStore {
} else {
Value::Integer(1)
};
let entry_key = match lookup_key(&entry_key) {
Cow::Owned(key) => key,
Cow::Borrowed(_) => entry_key,
};
entries.insert(entry_key, entry_value);
}
}
_ => (),
}
if entries.len() == self.config.max_entries {
if entries.len() == self.max_entries {
break;
}
}
trc::event!(
Store(trc::StoreEvent::HttpStoreFetch),
Url = self.config.url.to_compact_string(),
Total = entries.len(),
Elapsed = time.elapsed(),
);
Ok(entries)
}
}
fn lookup_key(key: &str) -> Cow<'_, str> {
if key.bytes().any(|b| !b.is_ascii() || b.is_ascii_uppercase()) {
Cow::Owned(key.to_lowercase())
} else {
Cow::Borrowed(key)
}
}
#[cfg(test)]
mod tests {
use super::*;
use arc_swap::ArcSwap;
use reqwest::Client;
use std::{
sync::atomic::{AtomicBool, AtomicU64},
time::Duration,
};
fn http_store(format: HttpStoreFormat, feed: &str) -> Arc<HttpStore> {
let config = HttpStoreConfig {
id: "test".into(),
url: "https://lists.example.org/feed".into(),
retry: 0,
refresh: 0,
timeout: Duration::from_secs(1),
gzipped: false,
max_size: 1024 * 1024,
max_entries: 100,
max_entry_size: 512,
format,
};
let entries = config
.parse_entries(feed.as_bytes())
.expect("feed is readable");
Arc::new(HttpStore {
entries: ArcSwap::from_pointee(entries),
expires: AtomicU64::new(u64::MAX),
in_flight: AtomicBool::new(false),
config,
client: Client::new(),
})
}
#[test]
fn list_keys_ignore_case() {
let store = http_store(
HttpStoreFormat::List,
"https://phish.example.org/Account/Verify?Token=AbC123\n\
https://PHISH.example.net/lower\n",
);
assert!(store.contains("https://phish.example.org/account/verify?token=abc123"));
assert!(store.contains("https://phish.example.org/Account/Verify?Token=AbC123"));
assert!(store.contains("https://phish.example.net/lower"));
assert!(store.contains("HTTPS://PHISH.EXAMPLE.NET/LOWER"));
assert!(!store.contains("https://phish.example.org/account/verify"));
}
#[test]
fn csv_keys_ignore_case() {
let store = http_store(
HttpStoreFormat::Csv {
index_key: 1,
index_value: None,
separator: ',',
skip_first: true,
},
"phish_id,url,phish_detail_url\n\
1,\"https://phish.example.org/Login.PHP?Id=Xy\",https://phishtank.example/1\n",
);
assert!(store.contains("https://phish.example.org/login.php?id=xy"));
assert!(store.contains("https://phish.example.org/Login.PHP?Id=Xy"));
assert!(!store.contains("phish_id"));
assert!(!store.contains("url"));
}
#[test]
fn csv_values_keep_case() {
let store = http_store(
HttpStoreFormat::Csv {
index_key: 0,
index_value: Some(1),
separator: ',',
skip_first: false,
},
"Example.ORG,Some Value\n",
);
assert_eq!(
store.get("example.org"),
Some(Value::Text("Some Value".into()))
);
assert_eq!(
store.get("EXAMPLE.org"),
Some(Value::Text("Some Value".into()))
);
}
}
crates/store/src/backend/mysql/mod.rs
+14
View File
@@ -99,7 +99,10 @@ pub(crate) fn into_error(err: impl Display) -> trc::Error {
trc::StoreEvent::MysqlError.reason(err)
}
const ER_UNKNOWN_ERROR: u16 = 1105;
const ER_TRANS_CACHE_FULL: u16 = 1197;
const ER_LOCK_WAIT_TIMEOUT: u16 = 1205;
const ER_LOCK_TABLE_FULL: u16 = 1206;
const ER_STATEMENT_TIMEOUT: u16 = 1969;
const ER_QUERY_TIMEOUT: u16 = 3024;
@@ -116,6 +119,17 @@ pub(crate) fn is_timeout_error(err: &mysql_async::Error) -> bool {
)
}
#[inline(always)]
pub(crate) fn is_chunk_too_large_error(err: &mysql_async::Error) -> bool {
is_timeout_error(err)
|| matches!(err, mysql_async::Error::Server(err)
if matches!(
err.code,
ER_UNKNOWN_ERROR | ER_TRANS_CACHE_FULL | ER_LOCK_TABLE_FULL
)
)
}
impl SearchIndex {
pub fn mysql_table(&self) -> &'static str {
match self {
crates/store/src/backend/mysql/search.rs
+5 -12
View File
@@ -11,7 +11,7 @@ use crate::{
MAX_TOKEN_LENGTH,
mysql::{
DELETE_CHUNK_SIZE, MIN_DELETE_CHUNK_SIZE, MysqlSearchField, MysqlStore, bounded,
into_error, is_timeout_error,
into_error, is_chunk_too_large_error,
},
},
search::{
@@ -123,14 +123,6 @@ impl MysqlStore {
let mut conn = self.conn().await?;
let limit = self.timeouts.maintenance;
let result = tokio::time::timeout(limit, async {
let s = conn.prep(&query).await.map_err(into_error)?;
match conn.exec_drop(s, params.clone()).await {
Ok(_) => return Ok(conn.affected_rows()),
Err(err) if is_timeout_error(&err) => (),
Err(err) => return Err(into_error(err)),
}
let mut chunk_size = DELETE_CHUNK_SIZE;
let mut deleted = 0;
@@ -144,13 +136,14 @@ impl MysqlStore {
match conn.exec_drop(&s, params.clone()).await {
Ok(_) => {
let affected = conn.affected_rows();
if affected == 0 {
deleted += affected;
if affected < chunk_size as u64 {
return Ok(deleted);
}
deleted += affected;
}
Err(err)
if is_timeout_error(&err) && chunk_size > MIN_DELETE_CHUNK_SIZE =>
if is_chunk_too_large_error(&err)
&& chunk_size > MIN_DELETE_CHUNK_SIZE =>
{
chunk_size = (chunk_size / 2).max(MIN_DELETE_CHUNK_SIZE);
break;
crates/store/src/backend/mysql/write.rs
+17 -12
View File
@@ -7,7 +7,8 @@
*/
use super::{
DELETE_CHUNK_SIZE, MIN_DELETE_CHUNK_SIZE, MysqlStore, bounded, into_error, is_timeout_error,
DELETE_CHUNK_SIZE, MIN_DELETE_CHUNK_SIZE, MysqlStore, bounded, into_error,
is_chunk_too_large_error,
};
use crate::{
IndexKey, Key, LogKey, SUBSPACE_COUNTER, SUBSPACE_IN_MEMORY_COUNTER, SUBSPACE_QUOTA,
@@ -416,12 +417,6 @@ impl MysqlStore {
.await
.map_err(into_error)?;
match conn.exec_drop(&delete, (&from, &to)).await {
Ok(_) => return Ok(()),
Err(err) if is_timeout_error(&err) => (),
Err(err) => return Err(into_error(err)),
}
let mut chunk_size = DELETE_CHUNK_SIZE;
loop {
@@ -438,7 +433,10 @@ impl MysqlStore {
.await
{
Ok(next) => next,
Err(err) if is_timeout_error(&err) && chunk_size > MIN_DELETE_CHUNK_SIZE => {
Err(err)
if is_chunk_too_large_error(&err)
&& chunk_size > MIN_DELETE_CHUNK_SIZE =>
{
chunk_size = (chunk_size / 2).max(MIN_DELETE_CHUNK_SIZE);
break;
}
@@ -450,7 +448,10 @@ impl MysqlStore {
.await
{
Ok(_) => (),
Err(err) if is_timeout_error(&err) && chunk_size > MIN_DELETE_CHUNK_SIZE => {
Err(err)
if is_chunk_too_large_error(&err)
&& chunk_size > MIN_DELETE_CHUNK_SIZE =>
{
chunk_size = (chunk_size / 2).max(MIN_DELETE_CHUNK_SIZE);
break;
}
@@ -477,7 +478,7 @@ async fn purge_table(conn: &mut Conn, table: char) -> trc::Result<()> {
match conn.exec_drop(&s, ()).await {
Ok(_) => return Ok(()),
Err(err) if is_timeout_error(&err) => (),
Err(err) if is_chunk_too_large_error(&err) => (),
Err(err) => return Err(into_error(err)),
}
@@ -505,7 +506,9 @@ async fn purge_table(conn: &mut Conn, table: char) -> trc::Result<()> {
loop {
let next = match conn.exec_first::<Vec<u8>, _, _>(&boundary, (&from,)).await {
Ok(next) => next,
Err(err) if is_timeout_error(&err) && chunk_size > MIN_DELETE_CHUNK_SIZE => {
Err(err)
if is_chunk_too_large_error(&err) && chunk_size > MIN_DELETE_CHUNK_SIZE =>
{
chunk_size = (chunk_size / 2).max(MIN_DELETE_CHUNK_SIZE);
break;
}
@@ -519,7 +522,9 @@ async fn purge_table(conn: &mut Conn, table: char) -> trc::Result<()> {
match result {
Ok(_) => (),
Err(err) if is_timeout_error(&err) && chunk_size > MIN_DELETE_CHUNK_SIZE => {
Err(err)
if is_chunk_too_large_error(&err) && chunk_size > MIN_DELETE_CHUNK_SIZE =>
{
chunk_size = (chunk_size / 2).max(MIN_DELETE_CHUNK_SIZE);
break;
}
crates/store/src/backend/redis/lookup.rs
+6 -14
View File
@@ -9,16 +9,7 @@
use super::{RedisPool, RedisStore, into_error};
use crate::{Deserialize, write::now};
use deadpool::managed::{Manager, Object, Pool};
use redis::{AsyncCommands, RedisError, RedisResult, RetryMethod, Script};
use std::sync::LazyLock;
static INCR_EXPIRE: LazyLock<Script> = LazyLock::new(|| {
Script::new(
"redis.call('INCRBY', KEYS[1], ARGV[1])
redis.call('EXPIRE', KEYS[1], ARGV[2])
return redis.call('GET', KEYS[1])",
)
});
use redis::{AsyncCommands, RedisError, RedisResult, RetryMethod};
impl RedisStore {
pub async fn key_set(&self, key: &[u8], value: &[u8], expires: Option<u64>) -> trc::Result<()> {
@@ -48,19 +39,19 @@ impl RedisStore {
match &self.pool {
RedisPool::Single(pool) => {
with_conn(pool, async |conn| {
Self::key_incr_(conn, key, value, expires).await
self.key_incr_(conn, key, value, expires).await
})
.await
}
RedisPool::Cluster(pool) => {
with_conn(pool, async |conn| {
Self::key_incr_(conn, key, value, expires).await
self.key_incr_(conn, key, value, expires).await
})
.await
}
RedisPool::Sentinel(pool) => {
with_conn(pool, async |conn| {
Self::key_incr_(conn, key, value, expires).await
self.key_incr_(conn, key, value, expires).await
})
.await
}
@@ -219,13 +210,14 @@ impl RedisStore {
}
async fn key_incr_(
&self,
conn: &mut impl AsyncCommands,
key: &[u8],
value: i64,
expires: Option<u64>,
) -> RedisResult<i64> {
if let Some(expires) = expires {
INCR_EXPIRE
self.incr_expire
.key(key)
.arg(value)
.arg(expires as i64)
crates/store/src/backend/redis/mod.rs
+22 -10
View File
@@ -10,7 +10,7 @@ use deadpool::{
managed::{Manager, Pool},
};
use redis::{
Client, ConnectionAddr, IntoConnectionInfo, ProtocolVersion, TlsMode,
Client, ConnectionAddr, IntoConnectionInfo, ProtocolVersion, Script, TlsMode,
cluster::{ClusterClient, ClusterClientBuilder},
cluster_read_routing::RandomReplicaStrategy,
sentinel::{SentinelClient, SentinelClientBuilder, SentinelServerType},
@@ -27,6 +27,7 @@ pub mod pool;
#[derive(Debug)]
pub struct RedisStore {
pub pool: RedisPool,
incr_expire: Script,
}
pub struct RedisConnectionManager {
@@ -51,9 +52,20 @@ pub enum RedisPool {
}
impl RedisStore {
fn new(pool: RedisPool) -> Self {
RedisStore {
pool,
incr_expire: Script::new(
"redis.call('INCRBY', KEYS[1], ARGV[1])
redis.call('EXPIRE', KEYS[1], ARGV[2])
return redis.call('GET', KEYS[1])",
),
}
}
pub async fn open_single(config: structs::RedisStore) -> Result<InMemoryStore, String> {
Ok(InMemoryStore::Redis(Arc::new(RedisStore {
pool: RedisPool::Single(build_pool(
Ok(InMemoryStore::Redis(Arc::new(RedisStore::new(
RedisPool::Single(build_pool(
RedisConnectionManager {
client: Client::open(config.url)
.map_err(|err| format!("Failed to open Redis client: {err:?}"))?,
@@ -64,7 +76,7 @@ impl RedisStore {
config.pool_timeout_wait,
config.pool_timeout_recycle,
)?),
})))
))))
}
pub async fn open_cluster(config: structs::RedisClusterStore) -> Result<InMemoryStore, String> {
@@ -95,8 +107,8 @@ impl RedisStore {
.build()
.map_err(|err| format!("Failed to open Redis client: {err:?}"))?;
Ok(InMemoryStore::Redis(Arc::new(RedisStore {
pool: RedisPool::Cluster(build_pool(
Ok(InMemoryStore::Redis(Arc::new(RedisStore::new(
RedisPool::Cluster(build_pool(
RedisClusterConnectionManager {
client,
timeout: config.timeout.into_inner(),
@@ -106,7 +118,7 @@ impl RedisStore {
config.pool_timeout_wait,
config.pool_timeout_recycle,
)?),
})))
))))
}
pub async fn open_sentinel(
@@ -167,8 +179,8 @@ impl RedisStore {
.build()
.map_err(|err| format!("Failed to open Redis Sentinel client: {err:?}"))?;
Ok(InMemoryStore::Redis(Arc::new(RedisStore {
pool: RedisPool::Sentinel(build_pool(
Ok(InMemoryStore::Redis(Arc::new(RedisStore::new(
RedisPool::Sentinel(build_pool(
RedisSentinelConnectionManager {
client: tokio::sync::Mutex::new(client),
timeout: config.timeout.into_inner(),
@@ -178,7 +190,7 @@ impl RedisStore {
config.pool_timeout_wait,
config.pool_timeout_recycle,
)?),
})))
))))
}
}
crates/trc/Cargo.toml
+1 -1
View File
@@ -1,6 +1,6 @@
[package]
name = "trc"
version = "0.16.23"
version = "0.16.24"
edition = "2024"
[dependencies]
crates/trc/event-macro/Cargo.toml
+1 -1
View File
@@ -1,6 +1,6 @@
[package]
name = "event_macro"
version = "0.16.23"
version = "0.16.24"
edition = "2024"
[lib]
crates/types/Cargo.toml
+1 -1
View File
@@ -1,6 +1,6 @@
[package]
name = "types"
version = "0.16.23"
version = "0.16.24"
edition = "2024"
[dependencies]
crates/utils/Cargo.toml
+1 -1
View File
@@ -1,6 +1,6 @@
[package]
name = "utils"
version = "0.16.23"
version = "0.16.24"
edition = "2024"
[dependencies]
crates/utils/proc-macros/Cargo.toml
+1 -1
View File
@@ -1,6 +1,6 @@
[package]
name = "proc_macros"
version = "0.16.23"
version = "0.16.24"
edition = "2024"
[lib]
docs/fork/strip-reports/v0.16.24.json
+724
View File
@@ -0,0 +1,724 @@
{
"ref": "v0.16.24",
"commit": "af37a234981722493b74623a983581691d2b70b6",
"removed_files": [
"crates/common/src/enterprise/alerts.rs",
"crates/common/src/enterprise/config.rs",
"crates/common/src/enterprise/license.rs",
"crates/common/src/enterprise/llm.rs",
"crates/common/src/enterprise/masked.rs",
"crates/common/src/enterprise/mod.rs",
"crates/common/src/telemetry/metrics/store.rs",
"crates/common/src/telemetry/metrics/test_data.rs",
"crates/common/src/telemetry/tracers/store.rs",
"crates/http/src/api/telemetry.rs",
"crates/jmap/src/registry/mapping/archived_item.rs",
"crates/jmap/src/registry/mapping/masked_email.rs",
"crates/jmap/src/registry/mapping/telemetry.rs",
"crates/scim-proto/src/attributes.rs",
"crates/scim-proto/src/etag.rs",
"crates/scim-proto/src/filter.rs",
"crates/scim-proto/src/json.rs",
"crates/scim-proto/src/lib.rs",
"crates/scim-proto/src/message/bulk.rs",
"crates/scim-proto/src/message/error.rs",
"crates/scim-proto/src/message/list.rs",
"crates/scim-proto/src/message/mod.rs",
"crates/scim-proto/src/message/patch.rs",
"crates/scim-proto/src/message/search.rs",
"crates/scim-proto/src/path.rs",
"crates/scim-proto/src/schema/group.rs",
"crates/scim-proto/src/schema/mod.rs",
"crates/scim-proto/src/schema/spc.rs",
"crates/scim-proto/src/schema/user.rs",
"crates/scim/src/auth.rs",
"crates/scim/src/bulk.rs",
"crates/scim/src/context.rs",
"crates/scim/src/discovery.rs",
"crates/scim/src/error.rs",
"crates/scim/src/groups/get.rs",
"crates/scim/src/groups/mod.rs",
"crates/scim/src/groups/patch.rs",
"crates/scim/src/groups/set.rs",
"crates/scim/src/lib.rs",
"crates/scim/src/query/cursor.rs",
"crates/scim/src/query/mod.rs",
"crates/scim/src/request.rs",
"crates/scim/src/users/get.rs",
"crates/scim/src/users/mod.rs",
"crates/scim/src/users/patch.rs",
"crates/scim/src/users/set.rs",
"crates/spam-filter/src/analysis/llm.rs",
"crates/store/src/backend/composite/mod.rs",
"crates/store/src/backend/composite/read_replica.rs",
"crates/store/src/backend/composite/sharded_blob.rs",
"crates/store/src/backend/composite/sharded_lookup.rs",
"crates/trc/src/serializers/binary.rs",
"tests/src/directory/oidc.rs",
"tests/src/scim/auth.rs",
"tests/src/scim/bulk.rs",
"tests/src/scim/discovery.rs",
"tests/src/scim/groups.rs",
"tests/src/scim/limits.rs",
"tests/src/scim/mod.rs",
"tests/src/scim/query.rs",
"tests/src/scim/users.rs",
"tests/src/system/archiving.rs",
"tests/src/system/tenant.rs"
],
"removed_snippets": {
"crates/common/src/auth/authentication.rs": 3,
"crates/common/src/auth/mod.rs": 2,
"crates/common/src/auth/permissions.rs": 1,
"crates/common/src/cache/directory.rs": 6,
"crates/common/src/cache/invalidate.rs": 1,
"crates/common/src/cache/principals.rs": 2,
"crates/common/src/cache/reload.rs": 1,
"crates/common/src/config/mod.rs": 2,
"crates/common/src/config/telemetry.rs": 4,
"crates/common/src/lib.rs": 2,
"crates/common/src/manager/boot.rs": 1,
"crates/common/src/network/mta.rs": 1,
"crates/common/src/scripts/plugins/llm_prompt.rs": 1,
"crates/common/src/storage/quota.rs": 2,
"crates/common/src/telemetry/metrics/mod.rs": 1,
"crates/common/src/telemetry/metrics/prometheus.rs": 1,
"crates/common/src/telemetry/mod.rs": 1,
"crates/common/src/telemetry/tracers/mod.rs": 1,
"crates/http/src/api/mod.rs": 4,
"crates/http/src/auth/permissions.rs": 1,
"crates/http/src/request.rs": 4,
"crates/jmap/src/registry/get.rs": 1,
"crates/jmap/src/registry/mapping/mod.rs": 1,
"crates/jmap/src/registry/mapping/principal.rs": 3,
"crates/jmap/src/registry/mapping/queued_message.rs": 1,
"crates/jmap/src/registry/mapping/task.rs": 1,
"crates/jmap/src/registry/mod.rs": 1,
"crates/jmap/src/registry/query.rs": 1,
"crates/jmap/src/registry/set.rs": 2,
"crates/main/src/main.rs": 1,
"crates/services/src/task_manager/alarm.rs": 1,
"crates/services/src/task_manager/imip.rs": 1,
"crates/services/src/task_manager/index.rs": 2,
"crates/services/src/task_manager/maintenance.rs": 3,
"crates/services/src/task_manager/scheduler.rs": 8,
"crates/spam-filter/src/analysis/mod.rs": 1,
"crates/spam-filter/src/analysis/score.rs": 2,
"crates/store/src/backend/mod.rs": 1,
"crates/store/src/backend/mysql/main.rs": 1,
"crates/store/src/backend/postgres/main.rs": 1,
"crates/store/src/build/blob.rs": 2,
"crates/store/src/build/lookup.rs": 1,
"crates/store/src/build/memory.rs": 2,
"crates/store/src/dispatch/blob.rs": 6,
"crates/store/src/dispatch/lookup.rs": 10,
"crates/store/src/dispatch/mod.rs": 1,
"crates/store/src/dispatch/search.rs": 6,
"crates/store/src/dispatch/store.rs": 8,
"crates/store/src/lib.rs": 6,
"crates/trc/src/serializers/mod.rs": 1
},
"cargo_edits": [
{
"file": "crates/main/Cargo.toml",
"line": 50,
"before": "default = [\"rocks\", \"enterprise\"]",
"after": "default = [\"rocks\"]"
},
{
"file": "tests/Cargo.toml",
"line": 22,
"before": "store = { path = \"../crates/store\", features = [\"test_mode\", \"enterprise\"] }",
"after": "store = { path = \"../crates/store\", features = [\"test_mode\"] }"
},
{
"file": "tests/Cargo.toml",
"line": 24,
"before": "directory = { path = \"../crates/directory\", features = [\"test_mode\", \"enterprise\"] }",
"after": "directory = { path = \"../crates/directory\", features = [\"test_mode\"] }"
},
{
"file": "tests/Cargo.toml",
"line": 25,
"before": "coordinator = { path = \"../crates/coordinator\", features = [\"test_mode\", \"enterprise\"] }",
"after": "coordinator = { path = \"../crates/coordinator\", features = [\"test_mode\"] }"
},
{
"file": "tests/Cargo.toml",
"line": 26,
"before": "jmap = { path = \"../crates/jmap\", features = [\"test_mode\", \"enterprise\"] }",
"after": "jmap = { path = \"../crates/jmap\", features = [\"test_mode\"] }"
},
{
"file": "tests/Cargo.toml",
"line": 35,
"before": "http = { path = \"../crates/http\", features = [\"test_mode\", \"enterprise\"] }",
"after": "http = { path = \"../crates/http\", features = [\"test_mode\"] }"
},
{
"file": "tests/Cargo.toml",
"line": 37,
"before": "scim = { path = \"../crates/scim\", features = [\"test_mode\", \"enterprise\"] }",
"after": "scim = { path = \"../crates/scim\", features = [\"test_mode\"] }"
},
{
"file": "tests/Cargo.toml",
"line": 39,
"before": "services = { path = \"../crates/services\", features = [\"test_mode\", \"enterprise\"] }",
"after": "services = { path = \"../crates/services\", features = [\"test_mode\"] }"
},
{
"file": "tests/Cargo.toml",
"line": 41,
"before": "smtp = { path = \"../crates/smtp\", features = [\"test_mode\", \"enterprise\"] }",
"after": "smtp = { path = \"../crates/smtp\", features = [\"test_mode\"] }"
},
{
"file": "tests/Cargo.toml",
"line": 42,
"before": "common = { path = \"../crates/common\", features = [\"test_mode\", \"enterprise\"] }",
"after": "common = { path = \"../crates/common\", features = [\"test_mode\"] }"
},
{
"file": "tests/Cargo.toml",
"line": 44,
"before": "email = { path = \"../crates/email\", features = [\"test_mode\", \"enterprise\"] }",
"after": "email = { path = \"../crates/email\", features = [\"test_mode\"] }"
},
{
"file": "tests/Cargo.toml",
"line": 45,
"before": "spam-filter = { path = \"../crates/spam-filter\", features = [\"test_mode\", \"enterprise\"] }",
"after": "spam-filter = { path = \"../crates/spam-filter\", features = [\"test_mode\"] }"
},
{
"file": "tests/Cargo.toml",
"line": 46,
"before": "trc = { path = \"../crates/trc\", features = [\"enterprise\"] }",
"after": "trc = { path = \"../crates/trc\", features = [] }"
},
{
"file": "tests/Cargo.toml",
"line": 47,
"before": "managesieve = { path = \"../crates/managesieve\", features = [\"test_mode\", \"enterprise\"] }",
"after": "managesieve = { path = \"../crates/managesieve\", features = [\"test_mode\"] }"
},
{
"file": ".github/workflows/ci.yml",
"line": 341,
"before": "cargo build --release --target ${{matrix.target}} -p stalwart --no-default-features --features \"sqlite postgres mysql rocks s3 redis azure nats enterprise\"",
"after": "cargo build --release --target ${{matrix.target}} -p stalwart --no-default-features --features \"sqlite postgres mysql rocks s3 redis azure nats\""
},
{
"file": ".github/workflows/ci.yml",
"line": 379,
"before": "# cargo build --release --target ${{matrix.target}} -p stalwart --no-default-features --features \"foundationdb s3 redis nats enterprise\"",
"after": "# cargo build --release --target ${{matrix.target}} -p stalwart --no-default-features --features \"foundationdb s3 redis nats\""
},
{
"file": ".github/workflows/ci.yml",
"line": 386,
"before": "cargo build --release --target ${{matrix.target}} -p stalwart --no-default-features --features \"sqlite postgres mysql rocks s3 redis azure nats enterprise\"",
"after": "cargo build --release --target ${{matrix.target}} -p stalwart --no-default-features --features \"sqlite postgres mysql rocks s3 redis azure nats\""
},
{
"file": ".github/workflows/ci.yml",
"line": 442,
"before": "cargo build --release -p stalwart --no-default-features --features \"sqlite postgres mysql rocks s3 redis azure nats enterprise\"",
"after": "cargo build --release -p stalwart --no-default-features --features \"sqlite postgres mysql rocks s3 redis azure nats\""
},
{
"file": "Dockerfile",
"line": 22,
"before": "RUN RUSTFLAGS=\"$(cat /flags.txt)\" cargo chef cook --target \"$(cat /target.txt)\" --release --no-default-features --features \"sqlite postgres mysql rocks s3 redis azure nats enterprise\" --recipe-path /recipe.json",
"after": "RUN RUSTFLAGS=\"$(cat /flags.txt)\" cargo chef cook --target \"$(cat /target.txt)\" --release --no-default-features --features \"sqlite postgres mysql rocks s3 redis azure nats\" --recipe-path /recipe.json"
},
{
"file": "Dockerfile",
"line": 24,
"before": "RUN RUSTFLAGS=\"$(cat /flags.txt)\" cargo build --target \"$(cat /target.txt)\" --release -p stalwart --no-default-features --features \"sqlite postgres mysql rocks s3 redis azure nats enterprise\"",
"after": "RUN RUSTFLAGS=\"$(cat /flags.txt)\" cargo build --target \"$(cat /target.txt)\" --release -p stalwart --no-default-features --features \"sqlite postgres mysql rocks s3 redis azure nats\""
},
{
"file": "Dockerfile.build",
"line": 111,
"before": "RUSTFLAGS=\"-L /usr/lib\" cargo chef cook --recipe-path recipe.json --zigbuild --release --target ${TARGET} -p stalwart --no-default-features --features \"foundationdb s3 redis nats enterprise\"; \\",
"after": "RUSTFLAGS=\"-L /usr/lib\" cargo chef cook --recipe-path recipe.json --zigbuild --release --target ${TARGET} -p stalwart --no-default-features --features \"foundationdb s3 redis nats\"; \\"
},
{
"file": "Dockerfile.build",
"line": 119,
"before": "cargo chef cook --recipe-path recipe.json --zigbuild --release --target ${TARGET} -p stalwart --no-default-features --features \"sqlite postgres mysql rocks s3 redis azure nats enterprise\"",
"after": "cargo chef cook --recipe-path recipe.json --zigbuild --release --target ${TARGET} -p stalwart --no-default-features --features \"sqlite postgres mysql rocks s3 redis azure nats\""
},
{
"file": "Dockerfile.build",
"line": 132,
"before": "RUSTFLAGS=\"-L /usr/lib\" cargo zigbuild --release --target ${TARGET} -p stalwart --no-default-features --features \"foundationdb s3 redis nats enterprise\" && \\",
"after": "RUSTFLAGS=\"-L /usr/lib\" cargo zigbuild --release --target ${TARGET} -p stalwart --no-default-features --features \"foundationdb s3 redis nats\" && \\"
},
{
"file": "Dockerfile.build",
"line": 142,
"before": "cargo zigbuild --release --target ${TARGET} -p stalwart --no-default-features --features \"sqlite postgres mysql rocks s3 redis azure nats enterprise\" && \\",
"after": "cargo zigbuild --release --target ${TARGET} -p stalwart --no-default-features --features \"sqlite postgres mysql rocks s3 redis azure nats\" && \\"
},
{
"file": "Dockerfile.fdb",
"line": 56,
"before": "RUN cargo build -p stalwart --no-default-features --features \"foundationdb s3 redis azure nats enterprise\" --release",
"after": "RUN cargo build -p stalwart --no-default-features --features \"foundationdb s3 redis azure nats\" --release"
}
],
"dangling_mods": [
{
"file": "crates/common/src/telemetry/metrics/mod.rs",
"line": 12,
"module": "test_data",
"lines": [
"#[cfg(any(feature = \"dev_mode\", feature = \"test_mode\"))]",
"pub mod test_data;"
]
},
{
"file": "tests/src/directory/mod.rs",
"line": 11,
"module": "oidc",
"lines": [
"pub mod oidc;"
]
},
{
"file": "tests/src/lib.rs",
"line": 27,
"module": "scim",
"lines": [
"#[cfg(test)]",
"pub mod scim;"
]
},
{
"file": "tests/src/system/mod.rs",
"line": 8,
"module": "archiving",
"lines": [
"pub mod archiving;"
]
},
{
"file": "tests/src/system/mod.rs",
"line": 19,
"module": "tenant",
"lines": [
"pub mod tenant;"
]
}
],
"problems": [
"tests/src/directory/issuer.rs:8: does not compile: unresolved import `crate::directory::oidc`",
"tests/src/directory/issuer.rs:67: does not compile: the size for values of type `str` cannot be known at compilation time",
"tests/src/directory/issuer.rs:74: does not compile: the size for values of type `str` cannot be known at compilation time"
],
"feature_gates": {
"crates/common/src/cache/reload.rs": 1,
"crates/common/src/manager/boot.rs": 1,
"crates/common/src/storage/mod.rs": 1,
"crates/common/src/storage/quota.rs": 1,
"crates/common/src/telemetry/metrics/prometheus.rs": 1,
"crates/http/src/api/mod.rs": 1,
"crates/http/src/auth/permissions.rs": 1,
"crates/jmap/src/registry/get.rs": 1,
"crates/jmap/src/registry/mapping/principal.rs": 2,
"crates/jmap/src/registry/mapping/queued_message.rs": 1,
"crates/jmap/src/registry/mapping/task.rs": 1,
"crates/jmap/src/registry/set.rs": 1,
"crates/services/src/task_manager/alarm.rs": 1,
"crates/services/src/task_manager/imip.rs": 1,
"crates/services/src/task_manager/index.rs": 1,
"crates/services/src/task_manager/maintenance.rs": 1,
"crates/services/src/task_manager/scheduler.rs": 1,
"crates/store/src/lib.rs": 1
},
"edition_checks": {},
"schema": {
"objects": [
"x:AiModel",
"x:Alert",
"x:ArchivedItem",
"x:MaskedEmail",
"x:MetricsStore",
"x:SpamLlm",
"x:Tenant",
"x:Trace",
"x:TracingStore"
],
"fields": [
"x:AcmeProvider.memberTenantId",
"x:ArfExternalReport.memberTenantId",
"x:Authentication.defaultTenantRoleIds",
"x:CalendarAlarm.template",
"x:CalendarScheduling.emailTemplate",
"x:CalendarScheduling.httpRsvpTemplate",
"x:DataRetention.archiveDeletedAccountsFor",
"x:DataRetention.archiveDeletedItemsFor",
"x:DataRetention.holdMetricsFor",
"x:DataRetention.holdTracesFor",
"x:DataRetention.metricsCollectionInterval",
"x:Dkim1Signature.memberTenantId",
"x:Dkim2Signature.memberTenantId",
"x:DmarcExternalReport.memberTenantId",
"x:Domain.allowScimProvisioning",
"x:Domain.directoryId",
"x:Domain.logo",
"x:Domain.memberTenantId",
"x:Email.maxMaskedAddresses",
"x:Enterprise.logoUrl",
"x:GroupAccount.externalId",
"x:LdapDirectory.memberTenantId",
"x:MySqlStore.readReplicas",
"x:OidcDirectory.memberTenantId",
"x:PostgreSqlStore.readReplicas",
"x:Search.indexTelemetry",
"x:Search.indexTracingFields",
"x:SqlDirectory.memberTenantId",
"x:TlsExternalReport.memberTenantId",
"x:UserAccount.externalId"
]
},
"third_party": {
"crates/common/src/network/acme/directory.rs": [
{
"line": 7,
"text": "Adapted from rustls-acme (https://github.com/FlorianUekermann/rustls-acme), licensed under MIT/Apache-2.0."
}
],
"crates/common/src/network/acme/jose.rs": [
{
"line": 7,
"text": "Adapted from rustls-acme (https://github.com/FlorianUekermann/rustls-acme), licensed under MIT/Apache-2.0."
}
],
"crates/common/src/network/acme/order.rs": [
{
"line": 7,
"text": "Adapted from rustls-acme (https://github.com/FlorianUekermann/rustls-acme), licensed under MIT/Apache-2.0."
}
],
"crates/common/src/scripts/functions/text.rs": [
{
"line": 239,
"text": "MIT licensed."
},
{
"line": 241,
"text": "Copyright (c) 2016 Titus Wormer <[email protected]>"
}
],
"crates/common/src/telemetry/tracers/journald.rs": [
{
"line": 70,
"text": "SPDX-FileCopyrightText: 2018 Benjamin Saunders <[email protected]>"
},
{
"line": 71,
"text": "SPDX-License-Identifier: MIT"
}
],
"crates/imap-proto/src/utf7.rs": [
{
"line": 7,
"text": "Ported from https://github.com/jstedfast/MailKit/blob/master/MailKit/Net/Imap/ImapEncoding.cs"
}
],
"crates/jmap/src/registry/mapping/log.rs": [
{
"line": 341,
"text": "SPDX-FileCopyrightText: 2017 Michael Coyne <[email protected]>"
},
{
"line": 343,
"text": "SPDX-License-Identifier: MIT"
},
{
"line": 346,
"text": "Adapted from https://github.com/mjc-gh/rev_lines/blob/main/src/lib.rs"
}
],
"crates/jmap-proto/src/types/date.rs": [
{
"line": 121,
"text": "Ported from http://howardhinnant.github.io/date_algorithms.html#civil_from_days"
},
{
"line": 158,
"text": "Ported from https://github.com/protocolbuffers/upb/blob/22182e6e/upb/json_decode.c#L982-L992"
}
],
"crates/nlp/src/tokenizers/japanese.rs": [
{
"line": 73,
"text": "Ported from https://github.com/woxtu/rust-tinysegmenter, MIT license"
}
],
"crates/nlp/src/tokenizers/types.rs": [
{
"line": 738,
"text": "Credits: test suite from linkify crate"
}
],
"crates/registry/src/types/datetime.rs": [
{
"line": 150,
"text": "Ported from http://howardhinnant.github.io/date_algorithms.html#civil_from_days"
},
{
"line": 188,
"text": "Ported from https://github.com/protocolbuffers/upb/blob/22182e6e/upb/json_decode.c#L982-L992"
}
],
"crates/store/src/backend/postgres/tls.rs": [
{
"line": 7,
"text": "Credits: https://github.com/jbg/tokio-postgres-rustls"
}
],
"crates/types/src/id.rs": [
{
"line": 69,
"text": "From https://github.com/archer884/crockford by J/A <[email protected]>"
},
{
"line": 70,
"text": "License: MIT/Apache 2.0"
}
],
"crates/utils/src/glob.rs": [
{
"line": 107,
"text": "Credits: Algorithm ported from https://research.swtch.com/glob"
}
]
},
"third_party_unlisted": [],
"renames": {
"\"URL to download spam filter rules from\" \u2192 \"URL to download spam filter rules from. Empty uses the rules bundled with the server.\"": {
"resources/schema/schema.json.gz": 1
},
"\"spamFilterRulesUrl\":\"https://github.com/stalwartlabs/spam-filter/releases/latest/download/spam-filter-rules.json.gz\", \u2192 ": {
"resources/schema/schema.json.gz": 1
},
"\"stalwart\" \u2192 \"inbuxa\"": {
"resources/schema/schema.json.gz": 15
},
"\"stalwart\".to_string() \u2192 \"inbuxa\".to_string()": {
"crates/jmap/src/registry/mapping/bootstrap.rs": 1,
"crates/registry/src/schema/structs_impl.rs": 12
},
"STALWART_SPAM_ \u2192 INBUXA_SPAM_": {
"crates/common/src/manager/mod.rs": 2
},
"spam_filter_rules_url: Some(\"https://github.com/stalwartlabs/spam-filter/releases/latest/download/spam-filter-rules.json.gz\".to_string()), \u2192 spam_filter_rules_url: None,": {
"crates/registry/src/schema/structs_impl.rs": 1
},
"stalwart-webui \u2192 inbuxa-webui": {
"crates/common/src/manager/application.rs": 3
},
"urn:stalwart: \u2192 urn:inbuxa:": {
"crates/dav/src/common/lock.rs": 1,
"crates/dav/src/common/uri.rs": 5,
"tests/src/webdav/lock.rs": 2
},
"urn:stalwart:jmap \u2192 urn:inbuxa:jmap:registry": {
"crates/jmap-proto/src/request/capability.rs": 3,
"tests/resources/scripts/stress_test_prepare.py": 1,
"tests/src/jmap/principal/get.rs": 2,
"tests/src/utils/jmap.rs": 1
},
"vnd.stalwart \u2192 vnd.inbuxa": {
"resources/schema/schema.json.gz": 2
},
"vnd.stalwart. \u2192 vnd.inbuxa.": {
"crates/registry/src/schema/enums_impl.rs": 4,
"tests/resources/jmap/sieve/test_mailbox.sieve": 1,
"tests/resources/smtp/sieve/awl.sieve": 1,
"tests/resources/smtp/sieve/awl_include.sieve": 1,
"tests/resources/smtp/sieve/stage_mail.sieve": 1,
"tests/resources/smtp/sieve/stage_rcpt.sieve": 1
}
},
"build": {
"errors": [
{
"file": "tests/src/directory/issuer.rs",
"line": 8,
"message": "unresolved import `crate::directory::oidc`"
},
{
"file": "tests/src/directory/issuer.rs",
"line": 67,
"message": "the size for values of type `str` cannot be known at compilation time"
},
{
"file": "tests/src/directory/issuer.rs",
"line": 74,
"message": "the size for values of type `str` cannot be known at compilation time"
}
],
"expected": [
{
"file": "tests/src/directory/mod.rs",
"line": 19,
"message": "cannot find module or crate `oidc` in this scope"
},
{
"file": "tests/src/smtp/inbound/antispam.rs",
"line": 17,
"message": "cannot find `enterprise` in `common`"
},
{
"file": "tests/src/smtp/inbound/antispam.rs",
"line": 49,
"message": "unresolved import `spam_filter::analysis::llm`"
},
{
"file": "tests/src/smtp/inbound/antispam.rs",
"line": 661,
"message": "no method named `spam_filter_analyze_llm` found for reference `&common::Server` in the current scope"
},
{
"file": "tests/src/system/mod.rs",
"line": 63,
"message": "cannot find module or crate `tenant` in this scope"
},
{
"file": "tests/src/system/mod.rs",
"line": 70,
"message": "cannot find module or crate `archiving` in this scope"
},
{
"file": "tests/src/telemetry/alerts.rs",
"line": 89,
"message": "no method named `process_alerts` found for struct `common::Server` in the current scope"
},
{
"file": "tests/src/telemetry/metrics.rs",
"line": 8,
"message": "unresolved import `common::telemetry::metrics::store`"
},
{
"file": "tests/src/telemetry/metrics.rs",
"line": 33,
"message": "no method named `insert_test_metrics` found for struct `common::Server` in the current scope"
},
{
"file": "tests/src/telemetry/metrics.rs",
"line": 190,
"message": "no method named `purge_metrics` found for reference `&store::Store` in the current scope"
},
{
"file": "tests/src/telemetry/tracing.rs",
"line": 8,
"message": "unresolved import `common::telemetry::tracers::store`"
},
{
"file": "tests/src/telemetry/tracing.rs",
"line": 35,
"message": "no method named `purge_spans` found for reference `&store::Store` in the current scope"
},
{
"file": "tests/src/telemetry/tracing.rs",
"line": 89,
"message": "no method named `purge_spans` found for reference `&store::Store` in the current scope"
},
{
"file": "tests/src/telemetry/tracing.rs",
"line": 152,
"message": "no method named `purge_spans` found for reference `&store::Store` in the current scope"
},
{
"file": "tests/src/telemetry/webhooks.rs",
"line": 11,
"message": "unresolved import `common::telemetry::tracers::store`"
},
{
"file": "tests/src/telemetry/webhooks.rs",
"line": 123,
"message": "no method named `purge_spans` found for reference `&store::Store` in the current scope"
}
],
"unused": [
{
"file": "crates/common/src/cache/invalidate.rs",
"line": 20,
"message": "unused imports: `registry::RegistryQuery` and `roaring::RoaringBitmap`"
},
{
"file": "crates/common/src/cache/principals.rs",
"line": 31,
"message": "unused import: `MaskedEmail`"
},
{
"file": "crates/common/src/cache/principals.rs",
"line": 41,
"message": "unused import: `now`"
},
{
"file": "crates/common/src/network/mta.rs",
"line": 27,
"message": "unused import: `structs::MaskedEmail`"
},
{
"file": "crates/common/src/network/mta.rs",
"line": 36,
"message": "unused import: `now`"
},
{
"file": "crates/common/src/network/mta.rs",
"line": 39,
"message": "unused import: `types::id::Id`"
},
{
"file": "crates/common/src/scripts/plugins/llm_prompt.rs",
"line": 7,
"message": "unused import: `compiler::Number`"
},
{
"file": "crates/common/src/scripts/plugins/llm_prompt.rs",
"line": 8,
"message": "unused import: `std::time::Instant`"
},
{
"file": "crates/common/src/scripts/plugins/llm_prompt.rs",
"line": 9,
"message": "unused imports: `AiEvent` and `SecurityEvent`"
},
{
"file": "crates/jmap/src/registry/mapping/principal.rs",
"line": 24,
"message": "unused imports: `RegistryObjectCounter` and `RegistryQuery`"
},
{
"file": "crates/jmap/src/registry/mapping/queued_message.rs",
"line": 25,
"message": "unused import: `ObjectType`"
},
{
"file": "crates/jmap/src/registry/mapping/queued_message.rs",
"line": 43,
"message": "unused import: `RegistryQuery`"
},
{
"file": "crates/services/src/task_manager/index.rs",
"line": 14,
"message": "unused imports: `ObjectType` and `Property`"
},
{
"file": "crates/services/src/task_manager/index.rs",
"line": 32,
"message": "unused import: `blob_hash::BlobHash`"
}
],
"known_clean": []
},
"ossify_log": "$ ossify.py crates\nProcessing Rust files in: ../inbuxa-import-work/strip-v0.16.24/tree/crates\n\nFound 958 Rust files\n\n\nSummary:\n- 50 files were completely removed\n- 2 crate roots were emptied\n- 118 proprietary snippets were removed from 50 files\n\n$ ossify.py tests\nProcessing Rust files in: ../inbuxa-import-work/strip-v0.16.24/tree/tests\n\nFound 212 Rust files\n\n\nSummary:\n- 11 files were completely removed\n- 0 crate roots were emptied\n- 0 proprietary snippets were removed from 0 files\n"
}
docs/fork/strip-reports/v0.16.24.md
+274
View File
@@ -0,0 +1,274 @@
# Strip report: upstream v0.16.24 (af37a2349817)
- Enterprise-only files removed or emptied: **63**
- Enterprise-only snippets removed: **118** in 50 files
- Cargo edits turning `enterprise` off: **25**
- Dangling module declarations removed: **5**
- Verification: **3 problems**
- Left for the rebuilt features to replace: 19 `enterprise` feature gates in 18 files; 0 `is_enterprise_edition()` checks in 0 files
- Third-party code: 14 files, **0** not in THIRD-PARTY.md
- Renamed identifiers: 62 in 18 files
- Build check: **3 errors**, 16 expected errors in 7 rebuilt-feature tests, 14 imports left unused
- Upstream schema flags 9 objects and 30 fields as Enterprise
## Removed files
- `crates/common/src/enterprise/alerts.rs`
- `crates/common/src/enterprise/config.rs`
- `crates/common/src/enterprise/license.rs`
- `crates/common/src/enterprise/llm.rs`
- `crates/common/src/enterprise/masked.rs`
- `crates/common/src/enterprise/mod.rs`
- `crates/common/src/telemetry/metrics/store.rs`
- `crates/common/src/telemetry/metrics/test_data.rs`
- `crates/common/src/telemetry/tracers/store.rs`
- `crates/http/src/api/telemetry.rs`
- `crates/jmap/src/registry/mapping/archived_item.rs`
- `crates/jmap/src/registry/mapping/masked_email.rs`
- `crates/jmap/src/registry/mapping/telemetry.rs`
- `crates/scim-proto/src/attributes.rs`
- `crates/scim-proto/src/etag.rs`
- `crates/scim-proto/src/filter.rs`
- `crates/scim-proto/src/json.rs`
- `crates/scim-proto/src/lib.rs`
- `crates/scim-proto/src/message/bulk.rs`
- `crates/scim-proto/src/message/error.rs`
- `crates/scim-proto/src/message/list.rs`
- `crates/scim-proto/src/message/mod.rs`
- `crates/scim-proto/src/message/patch.rs`
- `crates/scim-proto/src/message/search.rs`
- `crates/scim-proto/src/path.rs`
- `crates/scim-proto/src/schema/group.rs`
- `crates/scim-proto/src/schema/mod.rs`
- `crates/scim-proto/src/schema/spc.rs`
- `crates/scim-proto/src/schema/user.rs`
- `crates/scim/src/auth.rs`
- `crates/scim/src/bulk.rs`
- `crates/scim/src/context.rs`
- `crates/scim/src/discovery.rs`
- `crates/scim/src/error.rs`
- `crates/scim/src/groups/get.rs`
- `crates/scim/src/groups/mod.rs`
- `crates/scim/src/groups/patch.rs`
- `crates/scim/src/groups/set.rs`
- `crates/scim/src/lib.rs`
- `crates/scim/src/query/cursor.rs`
- `crates/scim/src/query/mod.rs`
- `crates/scim/src/request.rs`
- `crates/scim/src/users/get.rs`
- `crates/scim/src/users/mod.rs`
- `crates/scim/src/users/patch.rs`
- `crates/scim/src/users/set.rs`
- `crates/spam-filter/src/analysis/llm.rs`
- `crates/store/src/backend/composite/mod.rs`
- `crates/store/src/backend/composite/read_replica.rs`
- `crates/store/src/backend/composite/sharded_blob.rs`
- `crates/store/src/backend/composite/sharded_lookup.rs`
- `crates/trc/src/serializers/binary.rs`
- `tests/src/directory/oidc.rs`
- `tests/src/scim/auth.rs`
- `tests/src/scim/bulk.rs`
- `tests/src/scim/discovery.rs`
- `tests/src/scim/groups.rs`
- `tests/src/scim/limits.rs`
- `tests/src/scim/mod.rs`
- `tests/src/scim/query.rs`
- `tests/src/scim/users.rs`
- `tests/src/system/archiving.rs`
- `tests/src/system/tenant.rs`
## Removed snippets
- `crates/common/src/auth/authentication.rs`: 3
- `crates/common/src/auth/mod.rs`: 2
- `crates/common/src/auth/permissions.rs`: 1
- `crates/common/src/cache/directory.rs`: 6
- `crates/common/src/cache/invalidate.rs`: 1
- `crates/common/src/cache/principals.rs`: 2
- `crates/common/src/cache/reload.rs`: 1
- `crates/common/src/config/mod.rs`: 2
- `crates/common/src/config/telemetry.rs`: 4
- `crates/common/src/lib.rs`: 2
- `crates/common/src/manager/boot.rs`: 1
- `crates/common/src/network/mta.rs`: 1
- `crates/common/src/scripts/plugins/llm_prompt.rs`: 1
- `crates/common/src/storage/quota.rs`: 2
- `crates/common/src/telemetry/metrics/mod.rs`: 1
- `crates/common/src/telemetry/metrics/prometheus.rs`: 1
- `crates/common/src/telemetry/mod.rs`: 1
- `crates/common/src/telemetry/tracers/mod.rs`: 1
- `crates/http/src/api/mod.rs`: 4
- `crates/http/src/auth/permissions.rs`: 1
- `crates/http/src/request.rs`: 4
- `crates/jmap/src/registry/get.rs`: 1
- `crates/jmap/src/registry/mapping/mod.rs`: 1
- `crates/jmap/src/registry/mapping/principal.rs`: 3
- `crates/jmap/src/registry/mapping/queued_message.rs`: 1
- `crates/jmap/src/registry/mapping/task.rs`: 1
- `crates/jmap/src/registry/mod.rs`: 1
- `crates/jmap/src/registry/query.rs`: 1
- `crates/jmap/src/registry/set.rs`: 2
- `crates/main/src/main.rs`: 1
- `crates/services/src/task_manager/alarm.rs`: 1
- `crates/services/src/task_manager/imip.rs`: 1
- `crates/services/src/task_manager/index.rs`: 2
- `crates/services/src/task_manager/maintenance.rs`: 3
- `crates/services/src/task_manager/scheduler.rs`: 8
- `crates/spam-filter/src/analysis/mod.rs`: 1
- `crates/spam-filter/src/analysis/score.rs`: 2
- `crates/store/src/backend/mod.rs`: 1
- `crates/store/src/backend/mysql/main.rs`: 1
- `crates/store/src/backend/postgres/main.rs`: 1
- `crates/store/src/build/blob.rs`: 2
- `crates/store/src/build/lookup.rs`: 1
- `crates/store/src/build/memory.rs`: 2
- `crates/store/src/dispatch/blob.rs`: 6
- `crates/store/src/dispatch/lookup.rs`: 10
- `crates/store/src/dispatch/mod.rs`: 1
- `crates/store/src/dispatch/search.rs`: 6
- `crates/store/src/dispatch/store.rs`: 8
- `crates/store/src/lib.rs`: 6
- `crates/trc/src/serializers/mod.rs`: 1
## Dangling module declarations removed
- `crates/common/src/telemetry/metrics/mod.rs:12`: `mod test_data` (#[cfg(any(feature = "dev_mode", feature = "test_mode"))] / pub mod test_data;)
- `tests/src/directory/mod.rs:11`: `mod oidc` (pub mod oidc;)
- `tests/src/lib.rs:27`: `mod scim` (#[cfg(test)] / pub mod scim;)
- `tests/src/system/mod.rs:8`: `mod archiving` (pub mod archiving;)
- `tests/src/system/mod.rs:19`: `mod tenant` (pub mod tenant;)
## Cargo edits
- `crates/main/Cargo.toml:50`: `default = ["rocks", "enterprise"]` → `default = ["rocks"]`
- `tests/Cargo.toml:22`: `store = { path = "../crates/store", features = ["test_mode", "enterprise"] }` → `store = { path = "../crates/store", features = ["test_mode"] }`
- `tests/Cargo.toml:24`: `directory = { path = "../crates/directory", features = ["test_mode", "enterprise"] }` → `directory = { path = "../crates/directory", features = ["test_mode"] }`
- `tests/Cargo.toml:25`: `coordinator = { path = "../crates/coordinator", features = ["test_mode", "enterprise"] }` → `coordinator = { path = "../crates/coordinator", features = ["test_mode"] }`
- `tests/Cargo.toml:26`: `jmap = { path = "../crates/jmap", features = ["test_mode", "enterprise"] }` → `jmap = { path = "../crates/jmap", features = ["test_mode"] }`
- `tests/Cargo.toml:35`: `http = { path = "../crates/http", features = ["test_mode", "enterprise"] }` → `http = { path = "../crates/http", features = ["test_mode"] }`
- `tests/Cargo.toml:37`: `scim = { path = "../crates/scim", features = ["test_mode", "enterprise"] }` → `scim = { path = "../crates/scim", features = ["test_mode"] }`
- `tests/Cargo.toml:39`: `services = { path = "../crates/services", features = ["test_mode", "enterprise"] }` → `services = { path = "../crates/services", features = ["test_mode"] }`
- `tests/Cargo.toml:41`: `smtp = { path = "../crates/smtp", features = ["test_mode", "enterprise"] }` → `smtp = { path = "../crates/smtp", features = ["test_mode"] }`
- `tests/Cargo.toml:42`: `common = { path = "../crates/common", features = ["test_mode", "enterprise"] }` → `common = { path = "../crates/common", features = ["test_mode"] }`
- `tests/Cargo.toml:44`: `email = { path = "../crates/email", features = ["test_mode", "enterprise"] }` → `email = { path = "../crates/email", features = ["test_mode"] }`
- `tests/Cargo.toml:45`: `spam-filter = { path = "../crates/spam-filter", features = ["test_mode", "enterprise"] }` → `spam-filter = { path = "../crates/spam-filter", features = ["test_mode"] }`
- `tests/Cargo.toml:46`: `trc = { path = "../crates/trc", features = ["enterprise"] }` → `trc = { path = "../crates/trc", features = [] }`
- `tests/Cargo.toml:47`: `managesieve = { path = "../crates/managesieve", features = ["test_mode", "enterprise"] }` → `managesieve = { path = "../crates/managesieve", features = ["test_mode"] }`
- `.github/workflows/ci.yml:341`: `cargo build --release --target ${{matrix.target}} -p stalwart --no-default-features --features "sqlite postgres mysql rocks s3 redis azure nats enterprise"` → `cargo build --release --target ${{matrix.target}} -p stalwart --no-default-features --features "sqlite postgres mysql rocks s3 redis azure nats"`
- `.github/workflows/ci.yml:379`: `# cargo build --release --target ${{matrix.target}} -p stalwart --no-default-features --features "foundationdb s3 redis nats enterprise"` → `# cargo build --release --target ${{matrix.target}} -p stalwart --no-default-features --features "foundationdb s3 redis nats"`
- `.github/workflows/ci.yml:386`: `cargo build --release --target ${{matrix.target}} -p stalwart --no-default-features --features "sqlite postgres mysql rocks s3 redis azure nats enterprise"` → `cargo build --release --target ${{matrix.target}} -p stalwart --no-default-features --features "sqlite postgres mysql rocks s3 redis azure nats"`
- `.github/workflows/ci.yml:442`: `cargo build --release -p stalwart --no-default-features --features "sqlite postgres mysql rocks s3 redis azure nats enterprise"` → `cargo build --release -p stalwart --no-default-features --features "sqlite postgres mysql rocks s3 redis azure nats"`
- `Dockerfile:22`: `RUN RUSTFLAGS="$(cat /flags.txt)" cargo chef cook --target "$(cat /target.txt)" --release --no-default-features --features "sqlite postgres mysql rocks s3 redis azure nats enterprise" --recipe-path /recipe.json` → `RUN RUSTFLAGS="$(cat /flags.txt)" cargo chef cook --target "$(cat /target.txt)" --release --no-default-features --features "sqlite postgres mysql rocks s3 redis azure nats" --recipe-path /recipe.json`
- `Dockerfile:24`: `RUN RUSTFLAGS="$(cat /flags.txt)" cargo build --target "$(cat /target.txt)" --release -p stalwart --no-default-features --features "sqlite postgres mysql rocks s3 redis azure nats enterprise"` → `RUN RUSTFLAGS="$(cat /flags.txt)" cargo build --target "$(cat /target.txt)" --release -p stalwart --no-default-features --features "sqlite postgres mysql rocks s3 redis azure nats"`
- `Dockerfile.build:111`: `RUSTFLAGS="-L /usr/lib" cargo chef cook --recipe-path recipe.json --zigbuild --release --target ${TARGET} -p stalwart --no-default-features --features "foundationdb s3 redis nats enterprise"; \` → `RUSTFLAGS="-L /usr/lib" cargo chef cook --recipe-path recipe.json --zigbuild --release --target ${TARGET} -p stalwart --no-default-features --features "foundationdb s3 redis nats"; \`
- `Dockerfile.build:119`: `cargo chef cook --recipe-path recipe.json --zigbuild --release --target ${TARGET} -p stalwart --no-default-features --features "sqlite postgres mysql rocks s3 redis azure nats enterprise"` → `cargo chef cook --recipe-path recipe.json --zigbuild --release --target ${TARGET} -p stalwart --no-default-features --features "sqlite postgres mysql rocks s3 redis azure nats"`
- `Dockerfile.build:132`: `RUSTFLAGS="-L /usr/lib" cargo zigbuild --release --target ${TARGET} -p stalwart --no-default-features --features "foundationdb s3 redis nats enterprise" && \` → `RUSTFLAGS="-L /usr/lib" cargo zigbuild --release --target ${TARGET} -p stalwart --no-default-features --features "foundationdb s3 redis nats" && \`
- `Dockerfile.build:142`: `cargo zigbuild --release --target ${TARGET} -p stalwart --no-default-features --features "sqlite postgres mysql rocks s3 redis azure nats enterprise" && \` → `cargo zigbuild --release --target ${TARGET} -p stalwart --no-default-features --features "sqlite postgres mysql rocks s3 redis azure nats" && \`
- `Dockerfile.fdb:56`: `RUN cargo build -p stalwart --no-default-features --features "foundationdb s3 redis azure nats enterprise" --release` → `RUN cargo build -p stalwart --no-default-features --features "foundationdb s3 redis azure nats" --release`
## Flagged Enterprise in upstream's schema
**Objects:** `x:AiModel`, `x:Alert`, `x:ArchivedItem`, `x:MaskedEmail`, `x:MetricsStore`, `x:SpamLlm`, `x:Tenant`, `x:Trace`, `x:TracingStore`
**Fields:** `x:AcmeProvider.memberTenantId`, `x:ArfExternalReport.memberTenantId`, `x:Authentication.defaultTenantRoleIds`, `x:CalendarAlarm.template`, `x:CalendarScheduling.emailTemplate`, `x:CalendarScheduling.httpRsvpTemplate`, `x:DataRetention.archiveDeletedAccountsFor`, `x:DataRetention.archiveDeletedItemsFor`, `x:DataRetention.holdMetricsFor`, `x:DataRetention.holdTracesFor`, `x:DataRetention.metricsCollectionInterval`, `x:Dkim1Signature.memberTenantId`, `x:Dkim2Signature.memberTenantId`, `x:DmarcExternalReport.memberTenantId`, `x:Domain.allowScimProvisioning`, `x:Domain.directoryId`, `x:Domain.logo`, `x:Domain.memberTenantId`, `x:Email.maxMaskedAddresses`, `x:Enterprise.logoUrl`, `x:GroupAccount.externalId`, `x:LdapDirectory.memberTenantId`, `x:MySqlStore.readReplicas`, `x:OidcDirectory.memberTenantId`, `x:PostgreSqlStore.readReplicas`, `x:Search.indexTelemetry`, `x:Search.indexTracingFields`, `x:SqlDirectory.memberTenantId`, `x:TlsExternalReport.memberTenantId`, `x:UserAccount.externalId`
## Third-party code
Comments in the stripped tree that name another copyright holder, another license, or a source the code came from. Files marked **new** aren't in THIRD-PARTY.md yet.
- `crates/common/src/network/acme/directory.rs`
- 7: Adapted from rustls-acme (https://github.com/FlorianUekermann/rustls-acme), licensed under MIT/Apache-2.0.
- `crates/common/src/network/acme/jose.rs`
- 7: Adapted from rustls-acme (https://github.com/FlorianUekermann/rustls-acme), licensed under MIT/Apache-2.0.
- `crates/common/src/network/acme/order.rs`
- 7: Adapted from rustls-acme (https://github.com/FlorianUekermann/rustls-acme), licensed under MIT/Apache-2.0.
- `crates/common/src/scripts/functions/text.rs`
- 239: MIT licensed.
- 241: Copyright (c) 2016 Titus Wormer <tituswormer@gmail.com>
- `crates/common/src/telemetry/tracers/journald.rs`
- 70: SPDX-FileCopyrightText: 2018 Benjamin Saunders <ben.e.saunders@gmail.com>
- 71: SPDX-License-Identifier: MIT
- `crates/imap-proto/src/utf7.rs`
- 7: Ported from https://github.com/jstedfast/MailKit/blob/master/MailKit/Net/Imap/ImapEncoding.cs
- `crates/jmap/src/registry/mapping/log.rs`
- 341: SPDX-FileCopyrightText: 2017 Michael Coyne <mjc@hey.com>
- 343: SPDX-License-Identifier: MIT
- 346: Adapted from https://github.com/mjc-gh/rev_lines/blob/main/src/lib.rs
- `crates/jmap-proto/src/types/date.rs`
- 121: Ported from http://howardhinnant.github.io/date_algorithms.html#civil_from_days
- 158: Ported from https://github.com/protocolbuffers/upb/blob/22182e6e/upb/json_decode.c#L982-L992
- `crates/nlp/src/tokenizers/japanese.rs`
- 73: Ported from https://github.com/woxtu/rust-tinysegmenter, MIT license
- `crates/nlp/src/tokenizers/types.rs`
- 738: Credits: test suite from linkify crate
- `crates/registry/src/types/datetime.rs`
- 150: Ported from http://howardhinnant.github.io/date_algorithms.html#civil_from_days
- 188: Ported from https://github.com/protocolbuffers/upb/blob/22182e6e/upb/json_decode.c#L982-L992
- `crates/store/src/backend/postgres/tls.rs`
- 7: Credits: https://github.com/jbg/tokio-postgres-rustls
- `crates/types/src/id.rs`
- 69: From https://github.com/archer884/crockford by J/A <archer884@gmail.com>
- 70: License: MIT/Apache 2.0
- `crates/utils/src/glob.rs`
- 107: Credits: Algorithm ported from https://research.swtch.com/glob
## Renamed identifiers
- `"URL to download spam filter rules from" → "URL to download spam filter rules from. Empty uses the rules bundled with the server."`: `resources/schema/schema.json.gz` (1)
- `"spamFilterRulesUrl":"https://github.com/stalwartlabs/spam-filter/releases/latest/download/spam-filter-rules.json.gz", → `: `resources/schema/schema.json.gz` (1)
- `"stalwart" → "inbuxa"`: `resources/schema/schema.json.gz` (15)
- `"stalwart".to_string() → "inbuxa".to_string()`: `crates/jmap/src/registry/mapping/bootstrap.rs` (1), `crates/registry/src/schema/structs_impl.rs` (12)
- `STALWART_SPAM_ → INBUXA_SPAM_`: `crates/common/src/manager/mod.rs` (2)
- `spam_filter_rules_url: Some("https://github.com/stalwartlabs/spam-filter/releases/latest/download/spam-filter-rules.json.gz".to_string()), → spam_filter_rules_url: None,`: `crates/registry/src/schema/structs_impl.rs` (1)
- `stalwart-webui → inbuxa-webui`: `crates/common/src/manager/application.rs` (3)
- `urn:stalwart: → urn:inbuxa:`: `crates/dav/src/common/lock.rs` (1), `crates/dav/src/common/uri.rs` (5), `tests/src/webdav/lock.rs` (2)
- `urn:stalwart:jmap → urn:inbuxa:jmap:registry`: `crates/jmap-proto/src/request/capability.rs` (3), `tests/resources/scripts/stress_test_prepare.py` (1), `tests/src/jmap/principal/get.rs` (2), `tests/src/utils/jmap.rs` (1)
- `vnd.stalwart → vnd.inbuxa`: `resources/schema/schema.json.gz` (2)
- `vnd.stalwart. → vnd.inbuxa.`: `crates/registry/src/schema/enums_impl.rs` (4), `tests/resources/jmap/sieve/test_mailbox.sieve` (1), `tests/resources/smtp/sieve/awl.sieve` (1), `tests/resources/smtp/sieve/awl_include.sieve` (1), `tests/resources/smtp/sieve/stage_mail.sieve` (1), `tests/resources/smtp/sieve/stage_rcpt.sieve` (1)
## Build check
Errors mean shared code calls something the strip removed: usually a dual-licensed file that only serves an Enterprise feature. Drop or rework it in the merge into `main`, never on `upstream`.
- error `tests/src/directory/issuer.rs:8`: unresolved import `crate::directory::oidc`
- error `tests/src/directory/issuer.rs:67`: the size for values of type `str` cannot be known at compilation time
- error `tests/src/directory/issuer.rs:74`: the size for values of type `str` cannot be known at compilation time
- unused `crates/common/src/cache/invalidate.rs:20`: unused imports: `registry::RegistryQuery` and `roaring::RoaringBitmap`
- unused `crates/common/src/cache/principals.rs:31`: unused import: `MaskedEmail`
- unused `crates/common/src/cache/principals.rs:41`: unused import: `now`
- unused `crates/common/src/network/mta.rs:27`: unused import: `structs::MaskedEmail`
- unused `crates/common/src/network/mta.rs:36`: unused import: `now`
- unused `crates/common/src/network/mta.rs:39`: unused import: `types::id::Id`
- unused `crates/common/src/scripts/plugins/llm_prompt.rs:7`: unused import: `compiler::Number`
- unused `crates/common/src/scripts/plugins/llm_prompt.rs:8`: unused import: `std::time::Instant`
- unused `crates/common/src/scripts/plugins/llm_prompt.rs:9`: unused imports: `AiEvent` and `SecurityEvent`
- unused `crates/jmap/src/registry/mapping/principal.rs:24`: unused imports: `RegistryObjectCounter` and `RegistryQuery`
- unused `crates/jmap/src/registry/mapping/queued_message.rs:25`: unused import: `ObjectType`
- unused `crates/jmap/src/registry/mapping/queued_message.rs:43`: unused import: `RegistryQuery`
- unused `crates/services/src/task_manager/index.rs:14`: unused imports: `ObjectType` and `Property`
- unused `crates/services/src/task_manager/index.rs:32`: unused import: `blob_hash::BlobHash`
Expected: upstream's tests of features the fork rebuilt on `main` (tools/fork/build-check-known.txt).
- `tests/src/directory/mod.rs:19`: cannot find module or crate `oidc` in this scope
- `tests/src/smtp/inbound/antispam.rs:17`: cannot find `enterprise` in `common`
- `tests/src/smtp/inbound/antispam.rs:49`: unresolved import `spam_filter::analysis::llm`
- `tests/src/smtp/inbound/antispam.rs:661`: no method named `spam_filter_analyze_llm` found for reference `&common::Server` in the current scope
- `tests/src/system/mod.rs:63`: cannot find module or crate `tenant` in this scope
- `tests/src/system/mod.rs:70`: cannot find module or crate `archiving` in this scope
- `tests/src/telemetry/alerts.rs:89`: no method named `process_alerts` found for struct `common::Server` in the current scope
- `tests/src/telemetry/metrics.rs:8`: unresolved import `common::telemetry::metrics::store`
- `tests/src/telemetry/metrics.rs:33`: no method named `insert_test_metrics` found for struct `common::Server` in the current scope
- `tests/src/telemetry/metrics.rs:190`: no method named `purge_metrics` found for reference `&store::Store` in the current scope
- `tests/src/telemetry/tracing.rs:8`: unresolved import `common::telemetry::tracers::store`
- `tests/src/telemetry/tracing.rs:35`: no method named `purge_spans` found for reference `&store::Store` in the current scope
- `tests/src/telemetry/tracing.rs:89`: no method named `purge_spans` found for reference `&store::Store` in the current scope
- `tests/src/telemetry/tracing.rs:152`: no method named `purge_spans` found for reference `&store::Store` in the current scope
- `tests/src/telemetry/webhooks.rs:11`: unresolved import `common::telemetry::tracers::store`
- `tests/src/telemetry/webhooks.rs:123`: no method named `purge_spans` found for reference `&store::Store` in the current scope
## Problems
- tests/src/directory/issuer.rs:8: does not compile: unresolved import `crate::directory::oidc`
- tests/src/directory/issuer.rs:67: does not compile: the size for values of type `str` cannot be known at compilation time
- tests/src/directory/issuer.rs:74: does not compile: the size for values of type `str` cannot be known at compilation time
resources/privacy/catalog.toml
+5
View File
@@ -1050,6 +1050,11 @@ default = "none"
secret = ["credential"]
secretApiKey = ["credential"]
[object."x:DnsServerPowerDns"]
default = "none"
[object."x:DnsServerPowerDns".properties]
apiKey = ["credential"]
[object."x:DnsServerRoute53"]
default = "none"
[object."x:DnsServerRoute53".properties]
resources/schema/schema.json.gz
BIN
View File
Binary file not shown.
resources/schema/schema.json.sha256
+1 -1
View File
@@ -1 +1 @@
MiWRzsz0AHdRshG_8JimktRqBO_pHGAUBHFcfV5jwI4
ZjiCQWnwHbujcF7hNolPZQa-VtrNTNfc_HbSuZZ_TiY
tests/Cargo.toml
+1 -1
View File
@@ -1,6 +1,6 @@
[package]
name = "tests"
version = "0.16.23"
version = "0.16.24"
edition = "2024"
[features]
tests/src/directory/mod.rs
+3
View File
@@ -16,6 +16,9 @@ pub mod per_domain; // inbuxa: per-domain directories
pub mod sql;
pub mod synchronization;
pub mod unavailable;
// inbuxa: upstream's issuer.rs (since v0.16.23) is left out. It tests routing a
// token that names no address by its issuer, which upstream ships in
// Enterprise; here such a token gets the server's default directory (DIR-2).
#[tokio::test(flavor = "multi_thread")]
pub async fn directory_tests() {
tests/src/imap/antispam.rs
+15
View File
@@ -129,6 +129,21 @@ pub async fn test(test: &TestServer) {
assert_eq!(samples.iter().filter(|x| !x.1.is_spam).count(), 11);
assert_eq!(samples.iter().filter(|x| x.1.is_spam).count(), 11);
let last_id = samples.iter().map(|(id, _)| id.id()).max().unwrap();
for _ in 0..2 {
admin
.registry_create_object(Task::SpamFilterMaintenance(TaskSpamFilterMaintenance {
maintenance_type: TaskSpamFilterMaintenanceType::Train,
status: TaskStatus::now(),
}))
.await;
test.wait_for_tasks().await;
let model = spam_classifier_model(&test.server).await;
assert_eq!(model.reservoir.ham.total_seen, 11);
assert_eq!(model.reservoir.spam.total_seen, 11);
assert_eq!(model.last_id, last_id);
}
// Global spam samples should not appear in the account
let samples = account.spam_training_samples().await;
assert_eq!(samples.iter().filter(|x| !x.1.is_spam).count(), 11);
tests/src/imap/copy_move.rs
+49
View File
@@ -174,4 +174,53 @@ pub async fn test(imap: &mut ImapConnection, imap_check: &mut ImapConnection) {
imap.assert_read(Type::Tagged, ResponseType::Ok)
.await
.assert_contains("COPYUID");
move_store_race(imap, imap_check).await;
}
async fn move_store_race(imap: &mut ImapConnection, imap_check: &mut ImapConnection) {
const RACE_MESSAGES: usize = 10;
const RACE_ROUNDS: usize = 20;
// A MOVE that loses a race with a STORE on the same messages in another
// session is retried instead of failing with CONTACTADMIN
imap.send_ok("CREATE \"Race Left\"").await;
imap.send_ok("CREATE \"Race Right\"").await;
for i in 0..RACE_MESSAGES {
imap.append(
"Race Left",
&format!("From: [email protected]\r\nSubject: Move race {i}\r\n\r\nrace\r\n"),
)
.await;
}
for round in 0..RACE_ROUNDS {
let (src, dest, store) = if round % 2 == 0 {
("Race Left", "Race Right", "UID STORE 1:* +FLAGS (\\Seen)")
} else {
("Race Right", "Race Left", "UID STORE 1:* -FLAGS (\\Seen)")
};
imap.send_ok(&format!("SELECT \"{src}\"")).await;
imap_check.send_ok(&format!("SELECT \"{src}\"")).await;
// The STORE may lose the race instead, so only the MOVE is checked
imap.send(&format!("UID MOVE 1:* \"{dest}\"")).await;
imap_check.send(store).await;
let (moved, _) = tokio::join!(
imap.assert_read(Type::Tagged, ResponseType::Ok),
imap_check.read(Type::Tagged)
);
moved.assert_contains("COPYUID");
imap.send(&format!("STATUS \"{dest}\" (MESSAGES)")).await;
imap.assert_read(Type::Tagged, ResponseType::Ok)
.await
.assert_contains(&format!("(MESSAGES {RACE_MESSAGES})"));
}
// Restore the state the following tests expect
imap.send_ok("SELECT \"Burrata al Tartufo\"").await;
imap_check.send_ok("SELECT \"Burrata al Tartufo\"").await;
imap.send_ok("DELETE \"Race Left\"").await;
imap.send_ok("DELETE \"Race Right\"").await;
}
tests/src/jmap/core/blob.rs
+76
View File
@@ -367,6 +367,82 @@ pub async fn test(test: &TestServer) {
}
test.blob_expire_all().await;
let inbox_id = Id::from(INBOX_ID).to_string();
let response = account
.jmap_method_calls(json!([
[
"Blob/upload",
{
"accountId": account.id_string(),
"create": {
"m0": {
"data": [
{
"data:asText": concat!(
"From: [email protected]\r\n",
"To: [email protected]\r\n",
"Subject: Blob reference import\r\n",
"\r\n",
"Imported through a Blob/upload creation id."
)
}
],
"type": "message/rfc822"
}
}
},
"U0"
],
[
"Email/import",
{
"accountId": account.id_string(),
"emails": {
"i0": {
"blobId": "#m0",
"mailboxIds": { (inbox_id.as_str()): true }
}
}
},
"I0"
],
[
"Email/import",
{
"accountId": account.id_string(),
"emails": {
"i1": {
"blobId": "#missing",
"mailboxIds": { (inbox_id.as_str()): true }
}
}
},
"I1"
]
]))
.await;
assert_eq!(response.name_at(1), "Email/import", "{response:?}");
assert!(
response
.pointer("/methodResponses/1/1/created/i0/id")
.and_then(|v| v.as_str())
.is_some(),
"{response:?}"
);
assert!(
response
.pointer("/methodResponses/1/1/notCreated")
.is_none(),
"{response:?}"
);
assert_eq!(
response.error_type_at(2),
Some("invalidResultReference"),
"{response:?}"
);
test.blob_expire_all().await;
// Blob/lookup
let client = account.jmap_client().await;
let blob_id = client
tests/src/jmap/core/push_subscription.rs
+35 -4
View File
@@ -188,13 +188,37 @@ pub async fn test(test: &TestServer) {
client.push_subscription_destroy(&push_id).await.unwrap();
// Only one verification per minute is allowed
let push_id = client
.push_subscription_create("invalid", "https://127.0.0.1:19000/push", None)
let first_id = client
.push_subscription_create(
"first",
"https://127.0.0.1:19000/push?skip_checks=true",
None,
)
.await
.unwrap()
.take_id();
let verification = expect_push(&mut event_rx).await.unwrap_verification();
assert_eq!(verification.push_subscription_id, first_id);
let deferred_id = client
.push_subscription_create("deferred", "https://127.0.0.1:19000/push", None)
.await
.unwrap()
.take_id();
expect_nothing(&mut event_rx).await;
client.push_subscription_destroy(&push_id).await.unwrap();
let verification = expect_push_within(&mut event_rx, Duration::from_secs(8))
.await
.unwrap_verification();
assert_eq!(verification.push_subscription_id, deferred_id);
account
.jmap_request(
&["urn:ietf:params:jmap:core"],
json!([[
"PushSubscription/set",
{ "destroy": [first_id, deferred_id] },
"0"
]]),
)
.await;
// Register push notification (with encryption)
let push_id = client
@@ -731,7 +755,14 @@ fn assert_vapid_authorization(header: &str, expected_key: &str, expected_origin:
}
async fn expect_push(event_rx: &mut mpsc::Receiver<PushMessage>) -> PushMessage {
match tokio::time::timeout(Duration::from_millis(1500), event_rx.recv()).await {
expect_push_within(event_rx, Duration::from_millis(1500)).await
}
async fn expect_push_within(
event_rx: &mut mpsc::Receiver<PushMessage>,
wait: Duration,
) -> PushMessage {
match tokio::time::timeout(wait, event_rx.recv()).await {
Ok(Some(push)) => {
//println!("Push received: {:?}", push);
push
tests/src/jmap/mail/set.rs
+96 -3
View File
@@ -8,7 +8,7 @@ use crate::{
jmap::{find_values, replace_blob_ids, replace_boundaries, replace_values},
utils::server::TestServer,
};
use ::email::mailbox::INBOX_ID;
use ::email::{mailbox::INBOX_ID, message::metadata::MessageData};
use ahash::AHashSet;
use jmap_client::{
Error, Set,
@@ -18,8 +18,12 @@ use jmap_client::{
mailbox::Role,
};
use registry::schema::prelude::ObjectType;
use std::{fs, path::PathBuf};
use types::id::Id;
use std::{fs, path::PathBuf, str::FromStr};
use store::{
ValueKey,
write::{AlignedBytes, Archive},
};
use types::{collection::Collection, id::Id};
pub async fn test(test: &TestServer) {
println!("Running Email Set tests...");
@@ -29,6 +33,7 @@ pub async fn test(test: &TestServer) {
create(&client, &mailbox_id).await;
update(&client, &mailbox_id).await;
update_preserves_uids(test, &client, account.id().document_id(), &mailbox_id).await;
test.destroy_all_mailboxes(account).await;
test.account("[email protected]")
@@ -296,6 +301,94 @@ async fn update(client: &Client, root_mailbox_id: &str) {
.unwrap();
}
async fn update_preserves_uids(
test: &TestServer,
client: &Client,
account_id: u32,
root_mailbox_id: &str,
) {
let email_id = client
.email_query(
email::query::Filter::in_mailbox(root_mailbox_id).into(),
None::<Vec<_>>,
)
.await
.unwrap()
.take_ids()
.pop()
.unwrap();
let document_id = Id::from_str(&email_id).unwrap().document_id();
let test_mailbox_id = client
.mailbox_create("UID Test", None::<String>, Role::None)
.await
.unwrap()
.take_id();
let test_mailbox_document_id = Id::from_str(&test_mailbox_id).unwrap().document_id();
let uids = message_uids(test, account_id, document_id).await;
let inbox_uid = uids[&INBOX_ID];
assert_ne!(inbox_uid, 0);
// Full mailboxIds identical to the current ones plus a keyword change must keep the UID
let mut request = client.build();
request
.set_email()
.update(&email_id)
.mailbox_ids([root_mailbox_id])
.keywords(["uid-test"]);
request
.send_set_email()
.await
.unwrap()
.updated(&email_id)
.unwrap();
assert_eq!(
message_uids(test, account_id, document_id).await,
[(INBOX_ID, inbox_uid)].into_iter().collect()
);
// Full mailboxIds that keeps a mailbox and adds another must only assign a UID to the new one
let mut request = client.build();
request
.set_email()
.update(&email_id)
.mailbox_ids([root_mailbox_id, test_mailbox_id.as_str()]);
request
.send_set_email()
.await
.unwrap()
.updated(&email_id)
.unwrap();
let uids = message_uids(test, account_id, document_id).await;
assert_eq!(uids.len(), 2);
assert_eq!(uids[&INBOX_ID], inbox_uid);
assert_ne!(uids[&test_mailbox_document_id], 0);
client.mailbox_destroy(&test_mailbox_id, true).await.unwrap();
}
async fn message_uids(
test: &TestServer,
account_id: u32,
document_id: u32,
) -> std::collections::BTreeMap<u32, u32> {
test.server
.store()
.get_value::<Archive<AlignedBytes>>(ValueKey::archive(
account_id,
Collection::Email,
document_id,
))
.await
.unwrap()
.unwrap()
.deserialize::<MessageData>()
.unwrap()
.mailboxes
.iter()
.map(|m| (m.mailbox_id, m.uid))
.collect()
}
pub async fn assert_email_properties(
client: &Client,
message_id: &str,
tests/src/jmap/mod.rs
+2
View File
@@ -99,6 +99,7 @@ pub async fn jmap_tests() {
push_throttle: 500u64.into(),
websocket_throttle: 500u64.into(),
push_attempt_wait: 500u64.into(),
push_verify_timeout: 5000u64.into(),
..Default::default()
},
&[
@@ -108,6 +109,7 @@ pub async fn jmap_tests() {
Property::PushThrottle,
Property::WebsocketThrottle,
Property::PushAttemptWait,
Property::PushVerifyTimeout,
],
)
.await;
tests/src/jmap/principal/get.rs
+1 -1
View File
@@ -254,7 +254,7 @@ pub async fn test(test: &TestServer) {
// inbuxa: MT-22, the logo that applies to the account, and
// LP-19, whether the legacy protocols are open to it, and
// ai-explain EX-1, whether Explain can be offered
"urn:inbuxa:jmap": { "logo": null, "legacyProtocols": "enabled", "aiExplain": false },
"urn:inbuxa:jmap": { "logo": null, "legacyProtocols": "enabled", "legacyAllowed": ["imap", "pop3", "manageSieve", "submission"], "aiExplain": false },
"https://www.fastmail.com/dev/maskedemail": {}
}
}
tests/src/smtp/inbound/milter.rs
+73 -2
View File
@@ -28,9 +28,12 @@ use registry::{
schema::{
enums::{self, MtaStage},
prelude::{ObjectType, Property},
structs::{Expression, MtaHook, MtaMilter, MtaStageRcpt},
structs::{
Expression, ExpressionMatch, MtaHook, MtaMilter, MtaStageData, MtaStageRcpt,
SieveSystemScript,
},
},
types::map::Map,
types::{list::List, map::Map},
};
use serde::Deserialize;
use smtp::{
@@ -238,6 +241,34 @@ async fn mta_hook_session() {
..Default::default()
})
.await;
admin
.registry_create_object(MtaStageData {
script: Expression {
match_: List::from_iter([ExpressionMatch {
if_: "sender = '[email protected]'".into(),
then: "'tag_quarantine'".into(),
}]),
else_: "false".into(),
},
..Default::default()
})
.await;
admin
.registry_create_object(SieveSystemScript {
contents: concat!(
"require [\"editheader\"];\n",
"addheader \"X-Sieve\" \"Seen\";\n",
"if exists \"X-Quarantine\" {\n",
" deleteheader \"Subject\";\n",
" addheader \"Subject\" \"INFECTED\";\n",
"}\n"
)
.into(),
description: None,
is_active: true,
name: "tag_quarantine".into(),
})
.await;
admin.reload_settings().await;
test.reload_core();
test.expect_reload_settings().await;
@@ -355,6 +386,41 @@ async fn mta_hook_session() {
.await
.assert_contains("X-Spam: Yes")
.assert_contains("123456");
// Test quarantine
session
.send_message(
"[email protected]",
&["[email protected]"],
"test:no_dkim",
"250 2.0.0",
)
.await;
test.expect_message()
.await
.read_lines(&test)
.await
.assert_contains("X-Quarantine: true")
.assert_contains("Subject: Is dinner ready?")
.assert_contains("Are you hungry yet?");
// Test that a DATA stage Sieve script sees and preserves hook modifications
session
.send_message(
"[email protected]",
&["[email protected]"],
"test:no_dkim",
"250 2.0.0",
)
.await;
test.expect_message()
.await
.read_lines(&test)
.await
.assert_contains("X-Quarantine: true")
.assert_contains("X-Sieve: Seen")
.assert_contains("Subject: INFECTED")
.assert_contains("Are you hungry yet?");
}
#[test]
@@ -865,6 +931,11 @@ fn handle_mta_hook(request: Request, tests: Arc<Vec<HeaderTest>>) -> hooks::Resp
response: None,
modifications: vec![],
},
"quarantine" | "quarantine_only" => hooks::Response {
action: hooks::Action::Quarantine,
response: None,
modifications: vec![],
},
"temp_fail" => hooks::Response {
action: hooks::Action::Reject,
response: SmtpResponse {
tests/src/smtp/inbound/mod.rs
+2
View File
@@ -36,6 +36,8 @@ pub mod rcpt;
pub mod rewrite;
pub mod scripts;
pub mod sign;
pub mod spam_rules;
pub mod spam_rules_kept; // inbuxa: spam rules updates keep admin edits
pub mod throttle;
pub mod vrfy;
tests/src/smtp/inbound/spam_rules.rs
+364
View File
@@ -0,0 +1,364 @@
/*
* SPDX-FileCopyrightText: 2020 Stalwart Labs LLC <hello@stalw.art>
*
* SPDX-License-Identifier: AGPL-3.0-only OR LicenseRef-SEL
*/
use crate::utils::server::{TestServer, TestServerBuilder};
use registry::{
schema::{
enums::TaskSpamFilterMaintenanceType,
prelude::{Object, ObjectType},
structs::{
HttpLookup, MemoryLookupKey, MemoryLookupKeyValue, SpamDnsblServer, SpamFileExtension,
SpamRule, SpamSettings, SpamTag, SpamTagScore, Task, TaskSpamFilterMaintenance,
TaskStatus,
},
},
types::{ObjectImpl, float::Float},
};
use serde_json::{Value, json};
use std::fs;
use store::registry::RegistryObject;
#[tokio::test(flavor = "multi_thread")]
async fn spam_rules_update() {
let test = TestServerBuilder::new("spam_rules_update_test")
.await
.with_http_listener(19057)
.await
.build()
.await;
let admin = test.account("admin");
let rules_path = test.temp_dir.path.join("spam-filter-rules.json");
admin
.registry_create_object(SpamSettings {
spam_filter_rules_url: format!("file://{}", rules_path.display()).into(),
..Default::default()
})
.await;
admin
.registry_create_object(MemoryLookupKeyValue {
namespace: "test-keys".into(),
key: "conflict.org".into(),
value: "local".into(),
is_glob_pattern: false,
})
.await;
admin.reload_settings().await;
update_rules(&test, &release_1()).await;
for name in ["STWT_TEST_RULE", "STWT_TEST_DISABLED"] {
assert!(
stored::<SpamRule>(&test, "name", name)
.await
.unwrap()
.object
.enable()
);
}
assert!(
stored::<SpamDnsblServer>(&test, "name", "STWT_TEST_DNSBL")
.await
.unwrap()
.object
.enable()
);
assert!(
stored::<MemoryLookupKey>(&test, "key", "conflict.org")
.await
.is_none()
);
let disabled_rule = stored::<SpamRule>(&test, "name", "STWT_TEST_DISABLED")
.await
.unwrap();
admin
.registry_update_object(
ObjectType::SpamRule,
disabled_rule.id.id(),
json!({"enable": false}),
)
.await;
let local_tag = stored::<SpamTag>(&test, "tag", "TEST_TAG_LOCAL")
.await
.unwrap();
admin
.registry_update_object(
ObjectType::SpamTag,
local_tag.id.id(),
json!({"score": 2.0}),
)
.await;
let feed = stored::<HttpLookup>(&test, "namespace", "stwt_test_feed")
.await
.unwrap();
admin
.registry_update_object(
ObjectType::HttpLookup,
feed.id.id(),
json!({"enable": false}),
)
.await;
let release_2 = release_2();
for _ in 0..2 {
update_rules(&test, &release_2).await;
let rule = stored::<SpamRule>(&test, "name", "STWT_TEST_RULE")
.await
.unwrap()
.object;
assert!(rule.enable());
assert_eq!(object_json(&rule)["priority"], 400);
let rule = stored::<SpamRule>(&test, "name", "STWT_TEST_DISABLED")
.await
.unwrap()
.object;
assert!(!rule.enable());
assert_eq!(object_json(&rule)["priority"], 450);
assert!(
stored::<SpamRule>(&test, "name", "STWT_TEST_NEW")
.await
.is_some()
);
for upstream in release_2["SpamRule"].as_array().unwrap() {
let mut upstream: SpamRule = serde_json::from_value(upstream.clone()).unwrap();
let local = stored::<SpamRule>(
&test,
"name",
object_json(&upstream)["name"].as_str().unwrap(),
)
.await
.unwrap()
.object;
upstream.set_enable(local.enable());
assert_eq!(local, upstream);
}
for upstream in release_2["SpamDnsblServer"].as_array().unwrap() {
let upstream: SpamDnsblServer = serde_json::from_value(upstream.clone()).unwrap();
let local = stored::<SpamDnsblServer>(
&test,
"name",
object_json(&upstream)["name"].as_str().unwrap(),
)
.await
.unwrap()
.object;
assert_eq!(local, upstream);
}
for (tag, score) in [("TEST_TAG", 6.5), ("TEST_TAG_LOCAL", 2.0)] {
assert_eq!(
stored::<SpamTag>(&test, "tag", tag).await.unwrap().object,
SpamTag::Score(SpamTagScore {
tag: tag.into(),
score: Float::new(score),
})
);
}
let feed = stored::<HttpLookup>(&test, "namespace", "stwt_test_feed")
.await
.unwrap()
.object;
assert!(!feed.enable);
assert_eq!(feed.max_entries, 2000);
assert!(
stored::<MemoryLookupKey>(&test, "key", "example.org")
.await
.unwrap()
.object
.is_glob_pattern
);
assert!(
stored::<MemoryLookupKey>(&test, "key", "conflict.org")
.await
.is_none()
);
assert_eq!(
stored::<MemoryLookupKeyValue>(&test, "key", "conflict.org")
.await
.unwrap()
.object
.value,
"local"
);
assert!(
stored::<SpamFileExtension>(&test, "extension", "tst")
.await
.unwrap()
.object
.is_bad
);
}
}
async fn update_rules(test: &TestServer, rules: &Value) {
fs::write(
test.temp_dir.path.join("spam-filter-rules.json"),
rules.to_string(),
)
.unwrap();
test.account("admin")
.registry_create_object(Task::SpamFilterMaintenance(TaskSpamFilterMaintenance {
maintenance_type: TaskSpamFilterMaintenanceType::UpdateRules,
status: TaskStatus::now(),
}))
.await;
test.wait_for_tasks().await;
}
async fn stored<T: ObjectImpl + From<Object>>(
test: &TestServer,
property: &str,
value: &str,
) -> Option<RegistryObject<T>> {
test.server
.registry()
.list::<T>()
.await
.unwrap()
.into_iter()
.find(|item| object_json(&item.object)[property] == value)
}
fn object_json<T: ObjectImpl>(object: &T) -> Value {
serde_json::to_value(object).unwrap()
}
fn release_1() -> Value {
json!({
"SpamRule": [
{
"@type": "Any",
"name": "STWT_TEST_RULE",
"enable": true,
"priority": 500,
"condition": {"else": "false", "match": {"0": {"if": "$MISSING_ESSENTIAL_HEADERS && $SINGLE_SHORT_PART", "then": "'SHORT_PART_BAD_HEADERS'"}}}
},
{
"@type": "Any",
"name": "STWT_TEST_DISABLED",
"enable": true,
"priority": 500,
"condition": {"else": "false", "match": {"0": {"if": "$MISSING_ESSENTIAL_HEADERS && $SINGLE_SHORT_PART", "then": "'SHORT_PART_BAD_HEADERS'"}}}
}
],
"SpamDnsblServer": [
{
"@type": "Domain",
"name": "STWT_TEST_DNSBL",
"enable": true,
"tag": {"else": "false", "match": {
"0": {"if": "octets[3] == 1", "then": "'URIBL_BLOCKED'"},
"1": {"if": "octets[3] == 2", "then": "'URIBL_BLACK'"},
"2": {"if": "octets[3] == 4", "then": "'URIBL_GREY'"},
"3": {"if": "octets[3] == 8", "then": "'URIBL_RED'"}
}},
"zone": {"else": "value + '.multi.uribl.com'", "match": {}}
}
],
"SpamTag": [
{"@type": "Score", "tag": "TEST_TAG", "score": 6.5},
{"@type": "Score", "tag": "TEST_TAG_LOCAL", "score": 1.0}
],
"HttpLookup": [
{
"namespace": "stwt_test_feed",
"url": "http://127.0.0.1:1/feed.txt",
"format": {"@type": "List"},
"enable": true,
"isGzipped": false,
"maxSize": 1048576,
"maxEntries": 1000,
"maxEntrySize": 512,
"refresh": 43200000,
"retry": 3600000,
"timeout": 30000
}
],
"MemoryLookupKey": [
{"namespace": "test-keys", "key": "example.org", "isGlobPattern": false},
{"namespace": "test-keys", "key": "conflict.org", "isGlobPattern": false}
],
"SpamFileExtension": [
{"extension": "tst", "contentTypes": {}, "isArchive": false, "isBad": false, "isNz": false}
]
})
}
fn release_2() -> Value {
json!({
"SpamRule": [
{
"@type": "Any",
"name": "STWT_TEST_RULE",
"enable": true,
"priority": 400,
"condition": {"else": "false", "match": {"0": {"if": "$MISSING_ESSENTIAL_HEADERS && $SINGLE_SHORT_PART", "then": "'SHORT_PART_BAD_HEADERS'"}}}
},
{
"@type": "Any",
"name": "STWT_TEST_DISABLED",
"enable": true,
"priority": 450,
"condition": {"else": "false", "match": {"0": {"if": "$MISSING_ESSENTIAL_HEADERS && $SINGLE_SHORT_PART", "then": "'SHORT_PART_BAD_HEADERS'"}}}
},
{
"@type": "Any",
"name": "STWT_TEST_NEW",
"enable": true,
"priority": 500,
"condition": {"else": "false", "match": {"0": {"if": "$MISSING_ESSENTIAL_HEADERS && $SINGLE_SHORT_PART", "then": "'SHORT_PART_BAD_HEADERS'"}}}
}
],
"SpamDnsblServer": [
{
"@type": "Domain",
"name": "STWT_TEST_DNSBL",
"enable": true,
"tag": {"else": "false", "match": {
"0": {"if": "octets[0] != 127", "then": "false"},
"1": {"if": "octets[3] == 1", "then": "'URIBL_BLOCKED'"},
"2": {"if": "bit_and(octets[3], 2) != 0", "then": "'URIBL_BLACK'"},
"3": {"if": "bit_and(octets[3], 4) != 0", "then": "'URIBL_GREY'"},
"4": {"if": "bit_and(octets[3], 8) != 0", "then": "'URIBL_RED'"}
}},
"zone": {"else": "value + '.multi.uribl.com'", "match": {}}
}
],
"SpamTag": [
{"@type": "Score", "tag": "TEST_TAG", "score": 4.5},
{"@type": "Score", "tag": "TEST_TAG_LOCAL", "score": 1.0}
],
"HttpLookup": [
{
"namespace": "stwt_test_feed",
"url": "http://127.0.0.1:1/feed.txt",
"format": {"@type": "List"},
"enable": true,
"isGzipped": false,
"maxSize": 1048576,
"maxEntries": 2000,
"maxEntrySize": 512,
"refresh": 43200000,
"retry": 3600000,
"timeout": 30000
}
],
"MemoryLookupKey": [
{"namespace": "test-keys", "key": "example.org", "isGlobPattern": true},
{"namespace": "test-keys", "key": "conflict.org", "isGlobPattern": false}
],
"SpamFileExtension": [
{"extension": "tst", "contentTypes": {}, "isArchive": false, "isBad": true, "isNz": false}
]
})
}
tests/src/smtp/inbound/spam_rules_kept.rs
+204
View File
@@ -0,0 +1,204 @@
/*
* SPDX-FileCopyrightText: 2026 Coffey Labs
*
* SPDX-License-Identifier: AGPL-3.0-only
*/
//! A spam rules update brings unedited rules up to date and leaves an admin's
//! edits alone (common::manager::spam_rules), including on an install whose
//! rules were loaded before updates fingerprinted what they wrote. Each
//! update records what it replaced and what it kept in one audit record
//! (AU-1.10).
use crate::utils::server::{TestServer, TestServerBuilder};
use registry::{
schema::{
enums::TaskSpamFilterMaintenanceType,
prelude::ObjectType,
structs::{SpamRule, SpamSettings, Task, TaskSpamFilterMaintenance, TaskStatus},
},
types::ObjectImpl,
};
use serde_json::{Value, json};
use std::fs;
const USING: &[&str] = &["urn:ietf:params:jmap:core", "urn:inbuxa:jmap"];
#[tokio::test(flavor = "multi_thread")]
async fn spam_rules_keep_admin_edits() {
let test = TestServerBuilder::new("spam_rules_kept_test")
.await
.with_http_listener(19059)
.await
.build()
.await;
let admin = test.account("admin");
let rules_path = test.temp_dir.path.join("spam-filter-rules.json");
admin
.registry_create_object(SpamSettings {
spam_filter_rules_url: format!("file://{}", rules_path.display()).into(),
..Default::default()
})
.await;
// Two rules an admin made before any update ran: one exactly as the
// release has it, as an install from before fingerprints would, and one
// different from it, as an edited one would be.
admin.registry_create_object(rule("INBX_PRESET", 500)).await;
admin
.registry_create_object(rule("INBX_PRESET_EDITED", 300))
.await;
admin.reload_settings().await;
update_rules(&test, &release(500)).await;
assert_eq!(priority(&test, "INBX_UNTOUCHED").await, 500);
assert_eq!(priority(&test, "INBX_PRESET_EDITED").await, 300);
// An admin edits one rule the update wrote, and switches another off.
let edited = id_of(&test, "INBX_EDITED").await;
admin
.registry_update_object(ObjectType::SpamRule, edited, json!({"priority": 300}))
.await;
let switched_off = id_of(&test, "INBX_SWITCHED_OFF").await;
admin
.registry_update_object(ObjectType::SpamRule, switched_off, json!({"enable": false}))
.await;
for run in 0..2 {
update_rules(&test, &release(400)).await;
for (name, expected) in [
("INBX_UNTOUCHED", 400),
("INBX_SWITCHED_OFF", 400),
("INBX_PRESET", 400),
("INBX_EDITED", 300),
("INBX_PRESET_EDITED", 300),
] {
assert_eq!(priority(&test, name).await, expected, "{name}, run {run}");
}
assert!(
!stored(&test, "INBX_SWITCHED_OFF").await.enable(),
"run {run}: switching a rule off was undone"
);
}
// Two updates changed something: the first and the first of release 400.
// The second run of 400 changed nothing, so it isn't recorded.
let records = audit(&test, json!({"targetKind": "x:SpamRule"})).await;
let details = records
.iter()
.filter_map(|record| record["details"].as_str())
.filter(|details| details.starts_with("Rules update"))
.collect::<Vec<_>>();
assert_eq!(details.len(), 2, "{details:?}");
assert!(
details[0].contains("replaced 3 SpamRule")
&& details[0].contains("kept as edited locally 2 SpamRule"),
"{}",
details[0]
);
assert!(details[1].contains("added 3 SpamRule"), "{}", details[1]);
}
fn rule(name: &str, priority: i64) -> SpamRule {
serde_json::from_value(rule_json(name, priority)).unwrap()
}
fn rule_json(name: &str, priority: i64) -> Value {
json!({
"@type": "Any",
"name": name,
"enable": true,
"priority": priority,
"condition": {"else": "false", "match": {"0": {"if": "$MISSING_ESSENTIAL_HEADERS && $SINGLE_SHORT_PART", "then": "'SHORT_PART_BAD_HEADERS'"}}}
})
}
fn release(priority: i64) -> Value {
let rules = [
"INBX_UNTOUCHED",
"INBX_EDITED",
"INBX_SWITCHED_OFF",
"INBX_PRESET",
"INBX_PRESET_EDITED",
]
.into_iter()
.map(|name| rule_json(name, priority))
.collect::<Vec<_>>();
json!({ "SpamRule": rules })
}
async fn update_rules(test: &TestServer, rules: &Value) {
fs::write(
test.temp_dir.path.join("spam-filter-rules.json"),
rules.to_string(),
)
.unwrap();
test.account("admin")
.registry_create_object(Task::SpamFilterMaintenance(TaskSpamFilterMaintenance {
maintenance_type: TaskSpamFilterMaintenanceType::UpdateRules,
status: TaskStatus::now(),
}))
.await;
test.wait_for_tasks().await;
}
async fn stored(test: &TestServer, name: &str) -> SpamRule {
test.server
.registry()
.list::<SpamRule>()
.await
.unwrap()
.into_iter()
.find(|item| object_json(&item.object)["name"] == name)
.unwrap_or_else(|| panic!("no rule {name}"))
.object
}
async fn id_of(test: &TestServer, name: &str) -> types::id::Id {
test.server
.registry()
.list::<SpamRule>()
.await
.unwrap()
.into_iter()
.find(|item| object_json(&item.object)["name"] == name)
.unwrap_or_else(|| panic!("no rule {name}"))
.id
.id()
}
async fn priority(test: &TestServer, name: &str) -> i64 {
object_json(&stored(test, name).await)["priority"]
.as_i64()
.unwrap()
}
fn object_json<T: ObjectImpl>(object: &T) -> Value {
serde_json::to_value(object).unwrap()
}
/// Audit records matching `filter`, newest first.
async fn audit(test: &TestServer, filter: Value) -> Vec<Value> {
let admin = test.account("admin");
let response = admin
.jmap_request(
USING,
json!([
["inbuxa:AuditEvent/query", {
"accountId": admin.id_string(), "filter": filter, "limit": 100
}, "q"],
["inbuxa:AuditEvent/get", {
"accountId": admin.id_string(),
"#ids": {"resultOf": "q", "name": "inbuxa:AuditEvent/query", "path": "/ids"}
}, "g"]
]),
)
.await;
response
.0
.pointer("/methodResponses/1/1/list")
.and_then(Value::as_array)
.cloned()
.unwrap_or_else(|| panic!("audit query failed: {}", response.0))
}
tests/src/smtp/reporting/reschedule.rs
+1 -1
View File
@@ -51,7 +51,7 @@ use utils::snowflake::SnowflakeIdGenerator;
async fn report_reschedule() {
let mut test = TestServerBuilder::new("smtp_report_reschedule")
.await
.with_http_listener(19057)
.with_http_listener(19058)
.await
.capture_queue()
.build()
tests/src/system/task.rs
+43 -19
View File
@@ -6,7 +6,7 @@
* Modified by Coffey Labs in 2026 for INBUXA.
*/
use crate::utils::{account::Account, server::TestServer};
use crate::utils::{account::Account, jmap::JmapUtils, server::TestServer};
use registry::{
schema::{
enums::TaskStoreMaintenanceType,
@@ -301,14 +301,29 @@ impl Account {
.await
}
pub async fn tasks(&self) -> Vec<TaskId> {
pub async fn tasks(&self) -> Option<Vec<TaskId>> {
let ids = self.task_ids().await;
let mut results = Vec::with_capacity(ids.len());
for id in ids {
let sample = self.registry_get::<Task>(id).await;
results.push(TaskId { id, task: sample });
if ids.is_empty() {
return Some(Vec::new());
}
results
let response = self.registry_get_many(ObjectType::Task, &ids).await;
if response.not_found().next().is_some() {
return None;
}
Some(
response
.list()
.iter()
.map(|item| TaskId {
id: item.object_id(),
task: serde_json::from_str(&item.to_string()).unwrap_or_else(|err| {
panic!("Failed to deserialize {item}: {err}");
}),
})
.collect(),
)
}
async fn assert_no_tasks(&self) {
@@ -344,19 +359,28 @@ impl Account {
) -> Vec<TaskId> {
let mut attempt = 0;
loop {
let tasks = self.tasks().await;
if tasks.len() == count && tasks.iter().all(&is_expected) {
return tasks;
match self.tasks().await {
Some(tasks) if tasks.len() == count && tasks.iter().all(&is_expected) => {
return tasks;
}
Some(tasks) => {
attempt += 1;
assert!(
attempt < TASK_WAIT_ATTEMPTS,
"Expected {} tasks, found {}: {:?}",
count,
tasks.len(),
tasks
);
}
None => {
attempt += 1;
assert!(
attempt < TASK_WAIT_ATTEMPTS,
"Task/query keeps returning ids that Task/get reports as not found"
);
}
}
attempt += 1;
assert!(
attempt < TASK_WAIT_ATTEMPTS,
"Expected {} tasks, found {}: {:?}",
count,
tasks.len(),
tasks
);
tokio::time::sleep(TASK_WAIT_INTERVAL).await;
}
}
tools/fork/README.md
+6 -2
View File
@@ -57,8 +57,12 @@ under the reason it stays.
Fails when an upstream file the fork changed doesn't carry the AGPL 5(a)
notice, `Modified by Coffey Labs in <year> for INBUXA.`, under upstream's
license line. "Changed" means it differs from the `upstream` branch, so the
list comes from the diff, not from memory. CI runs it beside the name check.
license line. "Changed" means it differs from the newest `upstream` snapshot
in the checked-out history (found by its "Import upstream v…" subject, so CI
needs a full clone), so the list comes from the diff, not from memory. A
branch merging a new release is checked against that release, and other
branches aren't affected when the `upstream` branch moves. CI runs it beside
the name check.
```bash
tools/fork/notice-check.py # exit 1 on a missing notice
tools/fork/build-check-known.txt
+3
View File
@@ -7,6 +7,9 @@
# OIDC directories: `main` has its own tests/src/directory/oidc.rs.
tests/src/directory/mod.rs
# Issuer-based directory routing (since v0.16.23), which `main` doesn't have
# and doesn't carry the test for (DIR-2).
tests/src/directory/issuer.rs
# Tenants and archiving.
tests/src/system/mod.rs
# The LLM spam-filter classifier.
tools/fork/notice-check.py
+12 -8
View File
@@ -13,9 +13,13 @@ beneath upstream's own notice:
* Modified by Coffey Labs in 2026 for INBUXA.
"Changed" is measured against the `upstream` branch, which holds the stripped
upstream release `main` was last merged with (docs/spec/SPEC.md §2.2a), so
the list is what actually differs rather than a guess. A file counts as
"Changed" is measured against the newest stripped snapshot this tree has
merged, a commit on the `upstream` branch (docs/spec/SPEC.md §2.2a), so the
list is what actually differs rather than a guess. It's found in the tree's
own history, by the strip's commit subject, not taken from the branch head:
a branch that merges a new release is checked against it, and every other
branch against the release it's built on, whenever the `upstream` branch
moves. A file counts as
upstream's when its header names Stalwart Labs as a copyright holder; files
the fork wrote carry their own copyright and need nothing. Files with no
comment header at all (README, manifests) are covered by the README's prose.
@@ -37,11 +41,11 @@ def git(*args):
def snapshot_ref():
for ref in ('origin/upstream', 'upstream'):
if subprocess.run(['git', 'rev-parse', '--verify', '--quiet', f'{ref}^{{commit}}'],
capture_output=True).returncode == 0:
return ref
sys.exit('notice-check: no upstream snapshot branch (origin/upstream or upstream); fetch it first')
ref = git('log', '-1', '--format=%h', '--grep=^Import upstream v',
'--grep=^Re-import upstream v', 'HEAD').strip()
if not ref:
sys.exit('notice-check: no upstream snapshot in this history; fetch it in full first')
return ref
def changed_upstream_files(ref):
tools/fork/renames.py
+3
View File
@@ -46,6 +46,7 @@ TEXT_RENAMES = [
# that must match their containers and identity provider (database users,
# passwords, an OIDC audience), and name their databases explicitly.
('"stalwart".to_string()', '"inbuxa".to_string()', ('crates',)),
('"/var/log/stalwart', '"/var/log/inbuxa', ('crates',)),
# The spam filter rules ship with the server (common::manager::spam_rules);
# upstream's default of fetching its latest from GitHub becomes unset.
('spam_filter_rules_url: Some("https://github.com/stalwartlabs/spam-filter/releases/latest/download/spam-filter-rules.json.gz".to_string()),',
@@ -62,6 +63,8 @@ SCHEMA_HASH = Path('resources/schema/schema.json.sha256')
SCHEMA_RENAMES = [
('"stalwart"', '"inbuxa"'),
('vnd.stalwart', 'vnd.inbuxa'),
# Sieve extension labels, "(vnd.stalwart)" until upstream v0.16.24.
('(Stalwart)"', '(vnd.inbuxa)"'),
# The bundled spam rules: no default URL, and say what empty means.
('"spamFilterRulesUrl":"https://github.com/stalwartlabs/spam-filter/releases/latest/download/spam-filter-rules.json.gz",', ''),
('"URL to download spam filter rules from"',