PostgreSQL search: find words inside URLs and file names in body text

After #37, address fields on PostgreSQL are split into words as the built-in index splits them, but language text (subject, body, attachments) still goes straight to PostgreSQL's parser, which keeps a URL, host, path or file name as tokens of its own: "https://x.example/shipping-support/" becomes a url, a host and a url_path, "invoice-2024.pdf" a file. So TEXT/BODY "shipping" missed messages where the word appears only inside a link, while RocksDB and the other built-in backends found them: 8 messages across a handful of searches in the rehearsal. On insert, language text is now indexed as it was, followed by the word parts of each token that holds a URL separator (/ . @ : ? = & # _ % + ~ \), split with SpaceTokenizer as keyword_terms() splits addresses. The parts go through the same text search configuration as the rest of the text, so they are stemmed like the words around them. Plain words, words that only carry punctuation ("end.", "(see") and hyphenated words (the parser already splits those) add nothing, so text without links is indexed exactly as before. Each part is added once per document. On sample mail, the text vector of a short order notice with three links grows from 546 to 716 bytes, a newsletter with 25 tracking links from 5586 to 6430, and a plain letter not at all. On search, a query word written as a URL, host, file or hyphenated word also matches as its word parts, ORed with the query as written, so "shipping-support" or "invoice-2024.pdf" match the new parts and documents indexed before this change still match as they did. Existing messages keep their old vectors until they are reindexed (the reindexAccounts task); new and reindexed messages match at once. store::search_tests gains test_url_word_search: five bodies, 19 body searches for words found only in a URL path, query string, host or file name, the tokens as written, plain words and non-matches, with the same expected ids on every backend. It passes on RocksDB, SQLite, MySQL and PostgreSQL; on main PostgreSQL fails at the first ("shipping" finds [3], not [0, 3]). On PostgreSQL the suite then stops at the account sort assertion (query.rs:689) exactly as it does on main.
2026-09-24 21:33:17 -07:00
5 changed files with 23 additions and 256 deletions
@@ -2,8 +2,6 @@
 * SPDX-FileCopyrightText: 2020 Stalwart Labs LLC <[email protected]>
 *
 * SPDX-License-Identifier: AGPL-3.0-only OR LicenseRef-SEL
 *
 * Modified by Coffey Labs in 2026 for INBUXA.
 */
 use crate::auth::AccessToken;
@@ -20,16 +18,6 @@ impl Server {
        access_token: &AccessToken,
        addr: IpAddr,
    ) -> trc::Result<Option<InFlight>> {
        // inbuxa: an account with unlimited requests passes both limits
        // below anyway, so don't count its requests. The count is a write to
        // one counter per account in the in-memory store, and concurrent
        // requests from one account queue on that key (a row lock on SQL,
        // conflict retries on RocksDB): in a cluster rehearsal ten parallel
        // admin writes were accepted one after another, about 33 ms apart.
        if access_token.has_permission(Permission::UnlimitedRequests) {
            return Ok(None);
        }
        let rate_reset = if let Some(rate) = &self.core.network.http.rate_authenticated {
            if self.is_ip_allowed(addr) {
                None
@@ -7,7 +7,7 @@
 */
 use crate::{
-    BuildServer, Core, Server,
+    Core, Server,
    config::{
        server::{Listeners, tls::parse_certificates},
        storage::Storage,
@@ -245,73 +245,21 @@ fn error_object(error: &Error) -> Option<ObjectId> {
 // until someone reloaded. Writes to objects the settings are built from now
 // reload them, here and across the cluster, as ReloadSettings does.
-/// Coalesces the full reloads that registry writes trigger. A write waits
+/// Coalesces the full reloads that registry writes trigger: a write waits for
-/// for more writes before a reload starts (see [`WRITE_QUIET`]), then
+/// a reload that started after it was stored, and joins one if it can, so a
-/// takes the result of the first reload that started after it was stored,
+/// burst of writes costs a reload or two rather than one each.
-/// so a burst of writes, or a request with many objects, costs one reload
+#[derive(Default)]
 /// or two rather than one each.
 pub struct SettingsReloadGate {
    requested: std::sync::atomic::AtomicU64,
-    reloads: std::sync::atomic::AtomicU64,
+    state: tokio::sync::Mutex<SettingsReloadState>,
    state: parking_lot::Mutex<SettingsReloadState>,
    completed: tokio::sync::watch::Sender<u64>,
 }
 #[derive(Default)]
 struct SettingsReloadState {
-    /// A reload is waiting for writes to settle, or running.
+    completed: u64,
-    scheduled: bool,
+    refused: Option<String>,
    /// When the oldest write not yet covered by a reload was stored, and
    /// the newest.
    first_write: Option<std::time::Instant>,
    last_write: Option<std::time::Instant>,
    /// Recent reloads, oldest first: the last write each covered, and why
    /// it was refused, if it was.
    results: std::collections::VecDeque<(u64, Option<String>)>,
 }
 impl Default for SettingsReloadGate {
    fn default() -> Self {
        Self {
            requested: Default::default(),
            reloads: Default::default(),
            state: Default::default(),
            completed: tokio::sync::watch::Sender::new(0),
        }
    }
 }
 impl SettingsReloadGate {
    /// How many full reloads registry writes have run.
    pub fn reloads(&self) -> u64 {
        self.reloads.load(std::sync::atomic::Ordering::Relaxed)
    }
 }
 impl SettingsReloadState {
    /// The result of the reload that covered write `ticket`, once it ran.
    fn result_for(&self, ticket: u64) -> Option<Result<(), String>> {
        self.results
            .iter()
            .find(|(covers, _)| *covers >= ticket)
            .map(|(_, refused)| refused.clone().map_or(Ok(()), Err))
    }
 }
 /// How long a full reload waits after the last registry write for another.
 /// Parallel requests reach the server tens of milliseconds apart (in a
 /// cluster rehearsal, ten x:<Object>/set requests sent at once arrived about
 /// 33 ms apart and each got a reload of its own), so the window is a little
 /// over twice that. A single write pays it once, on top of the reload.
 pub const WRITE_QUIET: std::time::Duration = std::time::Duration::from_millis(75);
 /// The longest a full reload waits after the first write it covers, so a
 /// steady stream of writes still reloads at least this often.
 pub const WRITE_MAX_WAIT: std::time::Duration = std::time::Duration::from_millis(250);
 /// How many past reload results a waiting write can look up.
 const RELOAD_RESULTS: usize = 64;
 /// The reload a write to `object` calls for: the object to reload, or None
 /// when the running settings don't hold that object (accounts, domains and
 /// other data read as needed, stores, which take a restart, and objects with
@@ -422,85 +370,21 @@ impl Server {
            return Some(result);
        }
        // inbuxa: #39 joined only writes that queued behind a running
        // reload; requests that arrive tens of milliseconds apart never
        // overlapped one, so each got a reload of its own. The reload now
        // waits until writes settle (WRITE_QUIET after the last one, at
        // most WRITE_MAX_WAIT after the first) and covers them all. It runs
        // in a task of its own, so a request that goes away doesn't take
        // it with it; each write then takes the result of the reload that
        // started after it was stored.
        let gate = &self.inner.data.settings_reload;
        let ticket = gate
            .requested
            .fetch_add(1, std::sync::atomic::Ordering::SeqCst)
            + 1;
-        let now = std::time::Instant::now();
+        let mut state = gate.state.lock().await;
-        {
+        if state.completed >= ticket {
-            let mut state = gate.state.lock();
+            // A reload that started after this write was stored has run
-            state.first_write.get_or_insert(now);
+            return Some(state.refused.clone().map_or(Ok(()), Err));
            state.last_write = Some(now);
        }
-
+        let covers = gate.requested.load(std::sync::atomic::Ordering::SeqCst);
-        loop {
+        let result = self.reload_and_broadcast(change).await;
-            let mut completed = {
+        state.completed = covers;
-                let mut state = gate.state.lock();
+        state.refused = result.clone().err();
-                if let Some(result) = state.result_for(ticket) {
+        Some(result)
                    return Some(result);
                }
                if !state.scheduled {
                    state.scheduled = true;
                    let server = self.clone();
                    tokio::spawn(async move {
                        server.run_write_reload(change).await;
                    });
                }
                gate.completed.subscribe()
            };
            if completed.changed().await.is_err() {
                return Some(Err("The settings reload was interrupted".to_string()));
            }
        }
    }
    /// Waits for registry writes to settle, then reloads the settings once
    /// for all the writes stored so far.
    async fn run_write_reload(&self, change: RegistryChange) {
        let gate = &self.inner.data.settings_reload;
        loop {
            let deadline = {
                let state = gate.state.lock();
                let now = std::time::Instant::now();
                let first = state.first_write.unwrap_or(now);
                let last = state.last_write.unwrap_or(now);
                (last + WRITE_QUIET).min(first + WRITE_MAX_WAIT)
            };
            if deadline <= std::time::Instant::now() {
                break;
            }
            tokio::time::sleep_until(deadline.into()).await;
        }
        // Writes stored from here on wait for the next reload
        let covers = {
            let mut state = gate.state.lock();
            state.first_write = None;
            state.last_write = None;
            gate.requested.load(std::sync::atomic::Ordering::SeqCst)
        };
        gate.reloads
            .fetch_add(1, std::sync::atomic::Ordering::Relaxed);
        let result = self.inner.build_server().reload_and_broadcast(change).await;
        {
            let mut state = gate.state.lock();
            if state.results.len() == RELOAD_RESULTS {
                state.results.pop_front();
            }
            state.results.push_back((covers, result.err()));
            state.scheduled = false;
        }
        gate.completed.send_replace(covers);
    }
    async fn reload_and_broadcast(&self, change: RegistryChange) -> Result<(), String> {
@@ -2,8 +2,6 @@
 * SPDX-FileCopyrightText: 2020 Stalwart Labs LLC <[email protected]>
 *
 * SPDX-License-Identifier: AGPL-3.0-only OR LicenseRef-SEL
 *
 * Modified by Coffey Labs in 2026 for INBUXA.
 */
 use crate::{
@@ -25,14 +23,6 @@ use utils::snowflake::MAX_NODE_ID;
 const STALE_NODE_TIMEOUT: u64 = 60 * 60; // 1 hour
 const DEAD_NODE_TIMEOUT: u64 = 60 * 60 * 24; // 24 hours
 // INBUXA: every node renews its lease once a minute, so the lease doubles as
 // a heartbeat. A node not heard from in three minutes is reported Stale, which
 // is what Cluster Health on the dashboard counts. Taking over a lease still
 // needs the full hour of silence, so a node that is slow rather than gone
 // never loses its id to another host.
 const HEARTBEAT_INTERVAL: u64 = 60; // 1 minute
 const UNRESPONSIVE_NODE_TIMEOUT: u64 = 3 * HEARTBEAT_INTERVAL;
 const MAX_LEASE_RETRIES: u32 = 5;
 struct NodeSlot {
@@ -106,7 +96,7 @@ impl RegistryStore {
    }
    pub fn refresh_node_id_interval(&self) -> Duration {
-        Duration::from_secs(HEARTBEAT_INTERVAL)
+        Duration::from_secs(STALE_NODE_TIMEOUT / 2)
    }
    pub async fn cluster_node_list(&self) -> trc::Result<Vec<ClusterNode>> {
@@ -299,10 +289,6 @@ impl NodeSlot {
        self.elapsed > DEAD_NODE_TIMEOUT
    }
    fn is_responsive(&self) -> bool {
        self.elapsed <= UNRESPONSIVE_NODE_TIMEOUT
    }
    fn is_assignable(&self) -> bool {
        self.node_id <= MAX_NODE_ID
    }
@@ -310,10 +296,10 @@ impl NodeSlot {
    fn status(&self) -> ClusterNodeStatus {
        if self.is_dead() {
            ClusterNodeStatus::Inactive
-        } else if self.is_responsive() {
+        } else if self.is_stale() {
            ClusterNodeStatus::Active
        } else {
            ClusterNodeStatus::Stale
        } else {
            ClusterNodeStatus::Active
        }
    }
 }
@@ -328,42 +314,3 @@ impl From<NodeSlot> for ClusterNode {
        }
    }
 }
 #[cfg(test)]
 mod tests {
    use super::*;
    fn slot(elapsed: u64) -> NodeSlot {
        NodeSlot {
            node_id: 1,
            hostname: "mx2.example.org".into(),
            last_renewal: 0,
            elapsed,
            hash: 0,
        }
    }
    #[test]
    fn status_follows_the_heartbeat() {
        assert_eq!(slot(0).status(), ClusterNodeStatus::Active);
        assert_eq!(slot(UNRESPONSIVE_NODE_TIMEOUT).status(), ClusterNodeStatus::Active);
        assert_eq!(slot(UNRESPONSIVE_NODE_TIMEOUT + 1).status(), ClusterNodeStatus::Stale);
        assert_eq!(slot(DEAD_NODE_TIMEOUT).status(), ClusterNodeStatus::Stale);
        assert_eq!(slot(DEAD_NODE_TIMEOUT + 1).status(), ClusterNodeStatus::Inactive);
    }
    #[test]
    fn a_silent_node_keeps_its_id_for_an_hour() {
        // Reported Stale after three minutes, but not free to take over.
        let quiet = slot(UNRESPONSIVE_NODE_TIMEOUT + 1);
        assert_eq!(quiet.status(), ClusterNodeStatus::Stale);
        assert!(!quiet.is_stale());
        assert!(slot(STALE_NODE_TIMEOUT + 1).is_stale());
    }
    #[test]
    fn several_renewals_fit_before_a_node_looks_unresponsive() {
        assert!(UNRESPONSIVE_NODE_TIMEOUT >= 3 * HEARTBEAT_INTERVAL);
        assert!(HEARTBEAT_INTERVAL * 2 < STALE_NODE_TIMEOUT);
    }
 }
@@ -81,7 +81,7 @@ fn legacy_setting(name: &str, is_set: impl Fn(&str) -> bool) -> Option<String> {
 #[macro_export]
 macro_rules! brand_version {
    () => {
-        "2026.9.25.1"
+        "2026.9.24.3"
    };
 }
@@ -116,64 +116,12 @@ async fn test_write_applies(test: &TestServer) {
    for name in &names {
        assert!(has_schedule(test, name), "{name} missing");
    }
    // A burst of separate requests shares a reload or two: each arrives
    // tens of milliseconds after the last, so none overlaps a running
    // reload, and the reload waits for writes to settle instead
    let reloads = test.server.inner.data.settings_reload.reloads();
    let started = std::time::Instant::now();
    let burst = (0..10)
        .map(|i| format!("autoreload-burst-{i}"))
        .collect::<Vec<_>>();
    let mut writes = Vec::new();
    for name in &burst {
        writes.push(admin.registry_create([MtaDeliverySchedule {
            name: name.clone(),
            queue_id,
            ..Default::default()
        }]));
    }
    for response in futures::future::join_all(writes).await {
        assert_applied(&response);
        schedule_ids.push(response.created_id(0));
    }
    let burst_reloads = test.server.inner.data.settings_reload.reloads() - reloads;
    println!(
        "10 concurrent writes: {burst_reloads} reload(s), {} ms",
        started.elapsed().as_millis()
    );
    assert!(
        (1..=2).contains(&burst_reloads),
        "{burst_reloads} reloads for 10 concurrent writes"
    );
    for name in &burst {
        assert!(has_schedule(test, name), "{name} missing");
    }
    // A single write still reloads promptly
    let reloads = test.server.inner.data.settings_reload.reloads();
    let started = std::time::Instant::now();
    let response = admin
        .registry_create([MtaDeliverySchedule {
            name: "autoreload-single".into(),
            queue_id,
            ..Default::default()
        }])
        .await;
    assert_applied(&response);
    schedule_ids.push(response.created_id(0));
    println!("1 write: {} ms", started.elapsed().as_millis());
    assert_eq!(
        test.server.inner.data.settings_reload.reloads() - reloads,
        1
    );
    assert!(has_schedule(test, "autoreload-single"));
    // Several objects in one request: one reload
    let response = admin
        .registry_destroy(ObjectType::MtaDeliverySchedule, schedule_ids.iter())
        .await;
    assert_applied(&response);
-    for name in names.iter().chain(&burst) {
+    for name in &names {
        assert!(!has_schedule(test, name), "{name} still present");
    }