jcoffey-dev
17426f6d60
The server fetched upstream's latest published rules from GitHub at run time: a version nobody here tested, code-like expressions from an account we don't control, and the upstream name as a default in the admin form. The published rules of spam-filter v3.0.2 are now embedded (resources/spam-filter/, MIT, in THIRD-PARTY.md) and used whenever no other source is configured. An empty setting and upstream's old default both mean the bundled rules, so existing installs switch without a settings change; the URL stays an operator override (https:// or file://). The schema default is dropped and its description says what empty means, and the strip's rename pass does the same to each import. Rules load on first boot as before, and again whenever the bundled version differs from the last one loaded, which only adds missing rules and tags. That brings the AI classifier's LLM_* scores to installs that predate them: production has none today. upstream-watch now also opens an issue when spam-filter publishes a newer release; resources/spam-filter/README.md says how to take it. The antispam test now runs on the bundled rules, the path production takes; SPAM_RULES_URL tests another set. Unit tests cover the URL handling and that the bundled rules parse and score the AI tags as the AI spec says.
87 lines
2.6 KiB
Rust
87 lines
2.6 KiB
Rust
/*
|
|
* SPDX-FileCopyrightText: 2020 Stalwart Labs LLC <[email protected]>
|
|
*
|
|
* SPDX-License-Identifier: AGPL-3.0-only OR LicenseRef-SEL
|
|
*
|
|
* Modified by Coffey Labs in 2026 for INBUXA.
|
|
*/
|
|
|
|
use crate::USER_AGENT;
|
|
use hyper::HeaderMap;
|
|
use mail_auth::flate2;
|
|
use std::{
|
|
io::{BufReader, Read},
|
|
time::Duration,
|
|
};
|
|
use utils::HttpLimitResponse;
|
|
|
|
pub mod application;
|
|
pub mod backup;
|
|
pub mod boot;
|
|
pub mod console;
|
|
pub mod defaults;
|
|
pub mod first_party;
|
|
pub mod restore;
|
|
pub mod spam_rules; // inbuxa: rules bundled with the server
|
|
|
|
pub const SPAM_TRAINER_KEY: &[u8] = "INBUXA_SPAM_TRAIN_DATA.lz4".as_bytes();
|
|
pub const SPAM_CLASSIFIER_KEY: &[u8] = "INBUXA_SPAM_CLASSIFIER_MODEL.lz4".as_bytes();
|
|
|
|
pub async fn fetch_resource(
|
|
url: &str,
|
|
headers: Option<HeaderMap>,
|
|
timeout: Duration,
|
|
max_size: usize,
|
|
) -> Result<Vec<u8>, String> {
|
|
if let Some(path) = url.strip_prefix("file://") {
|
|
tokio::fs::read(path)
|
|
.await
|
|
.map_err(|err| format!("Failed to read {path}: {err}"))
|
|
} else {
|
|
let response = utils::http::http_client_builder(is_localhost_url(url))
|
|
.timeout(timeout)
|
|
.user_agent(USER_AGENT)
|
|
.build()
|
|
.unwrap_or_default()
|
|
.get(url)
|
|
.headers(headers.unwrap_or_default())
|
|
.send()
|
|
.await
|
|
.map_err(|err| format!("Failed to fetch {url}: {err}"))?;
|
|
|
|
if response.status().is_success() {
|
|
response
|
|
.bytes_with_limit(max_size)
|
|
.await
|
|
.map_err(|err| format!("Failed to fetch {url}: {err}"))
|
|
.and_then(|bytes| bytes.ok_or_else(|| format!("Resource too large: {url}")))
|
|
} else {
|
|
let code = response.status().canonical_reason().unwrap_or_default();
|
|
let reason = response.text().await.unwrap_or_default();
|
|
|
|
Err(format!(
|
|
"Failed to fetch {url}: Code: {code}, Details: {reason}",
|
|
))
|
|
}
|
|
}
|
|
.and_then(|bytes| {
|
|
if url.ends_with(".gz") || url.ends_with(".gzip") {
|
|
BufReader::new(flate2::read::GzDecoder::new(&bytes[..]))
|
|
.bytes()
|
|
.collect::<Result<Vec<u8>, _>>()
|
|
.map_err(|err| format!("Failed to decompress {url}: {err}"))
|
|
} else {
|
|
Ok(bytes)
|
|
}
|
|
})
|
|
}
|
|
|
|
pub fn is_localhost_url(url: &str) -> bool {
|
|
url.split_once("://")
|
|
.map(|(_, url)| url.split_once('/').map_or(url, |(host, _)| host))
|
|
.is_some_and(|host| {
|
|
let host = host.rsplit_once(':').map_or(host, |(host, _)| host);
|
|
host == "localhost" || host == "127.0.0.1" || host == "[::1]"
|
|
})
|
|
}
|