inbuxa/inbuxa-server
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 5 Wiki Activity

Search: find addresses by local part, domain or name on PostgreSQL and MySQL
ci / fork-checks (pull_request) Successful in 43s
Details
ci / build (pull_request) Successful in 4m20s
Details

Browse Source 
A 3-node PostgreSQL rehearsal found IMAP SEARCH FROM "noreply" matched
0-2 messages where RocksDB matched 23 of 930. The message indexer hands
each address and display name of From/To/Cc/Bcc to the search store as
keyword text (Language::None). The built-in index splits keyword text
into lowercase runs of alphanumerics, so an address is found by its full
form, its local part, its domain or a display-name word. The SQL
backends didn't:

- PostgreSQL's text parser keeps "[email protected]" as one email
  token (host names and URLs likewise), so neither "noreply" nor
  "amazon.com" ever matched it. Keyword text is now split the same way
  as the built-in index (SpaceTokenizer) before to_tsvector on insert
  and before plainto_tsquery/phraseto_tsquery on search, still under
  the 'simple' configuration, so the GIN index keeps serving the query.
  The sort columns keep the raw text.
- MySQL's FULLTEXT parser already splits on punctuation, but InnoDB
  never indexes its stopwords ("com", "de", "www", ...) or words under
  innodb_ft_min_token_size (3), and a required +word it hasn't indexed
  matches no row. So "amazon.com", "[email protected]" or "jane doe" found
  nothing. Those words are now matched with a word-boundary REGEXP on
  the rows the indexed words select. In language text (bodies,
  subjects) they are dropped when other words remain, and only checked
  when nothing else is left, so "the invoice" no longer finds nothing
  either.

Existing PostgreSQL search indexes hold the old single-token vectors and
need a reindex (the reindexAccounts task) before address searches find
old messages. MySQL needs none: only the query changed.

store::search_tests gains test_address_search: five messages, 28
FROM/TO/CC/BCC searches by full address, local part, domain, domain
labels, display name and hyphenated local part, plus a TEXT-style OR,
with the same expected ids on every backend. It passes on RocksDB,
SQLite, PostgreSQL and MySQL; on main it fails on PostgreSQL (From
"noreply") and MySQL (From "[email protected]").
This commit is contained in:
jcoffey-dev
2026-09-24 11:25:25 -07:00
parent 9311c1a38b
commit 639a415a4f
3 changed files with 312 additions and 23 deletions
Download Patch File Download Diff File Expand all files Collapse all files
crates/store/src/backend/mysql/search.rs
+76 -13
View File
@@ -2,6 +2,8 @@
* SPDX-FileCopyrightText: 2020 Stalwart Labs LLC <[email protected]>
*
* SPDX-License-Identifier: AGPL-3.0-only OR LicenseRef-SEL
*
* Modified by Coffey Labs in 2026 for INBUXA.
*/
use crate::{
@@ -19,7 +21,7 @@ use crate::{
write::SearchIndex,
};
use mysql_async::{IsolationLevel, TxOpts, Value, prelude::Queryable};
use nlp::tokenizers::word::WordTokenizer;
use nlp::{language::Language, tokenizers::word::WordTokenizer};
use std::fmt::Write;
impl MysqlStore {
@@ -146,6 +148,20 @@ impl MysqlStore {
}
}
// inbuxa: InnoDB's default full-text stopword list
// (INFORMATION_SCHEMA.INNODB_FT_DEFAULT_STOPWORD) and innodb_ft_min_token_size
// default; words outside these are not in a FULLTEXT index.
const FT_STOPWORDS: &[&str] = &[
"a", "about", "an", "are", "as", "at", "be", "by", "com", "de", "en", "for", "from", "how",
"i", "in", "is", "it", "la", "of", "on", "or", "that", "the", "this", "to", "was", "what",
"when", "where", "who", "will", "with", "und", "www",
];
const FT_MIN_TOKEN_SIZE: usize = 3;
fn is_ft_indexed(word: &str) -> bool {
word.chars().count() >= FT_MIN_TOKEN_SIZE && !FT_STOPWORDS.contains(&word)
}
fn build_filter(query: &mut String, filters: &[SearchFilter]) -> Vec<Value> {
if filters.is_empty() {
return Vec::new();
@@ -171,30 +187,77 @@ fn build_filter(query: &mut String, filters: &[SearchFilter]) -> Vec<Value> {
if field.is_text() && matches!(op, SearchOperator::Equal | SearchOperator::Contains)
{
let (value, mode) = match (value, op) {
(SearchValue::Text { value, .. }, SearchOperator::Equal) => {
(Value::Bytes(format!("{value:?}").into_bytes()), "BOOLEAN")
}
(SearchValue::Text { value, .. }, ..) => {
let (value, mode, unindexed) = match (value, op) {
(SearchValue::Text { value, .. }, SearchOperator::Equal) => (
Value::Bytes(format!("{value:?}").into_bytes()),
"BOOLEAN",
Vec::new(),
),
(SearchValue::Text { value, language }, ..) => {
let mut text_query = String::with_capacity(value.len() + 1);
let mut unindexed = Vec::new();
for item in WordTokenizer::new(value, MAX_TOKEN_LENGTH) {
if !text_query.is_empty() {
text_query.push(' ');
// inbuxa: InnoDB never indexes stopwords ("com",
// "de", "www", ...) or words under
// innodb_ft_min_token_size, and a required
// (+word) term it has not indexed matches no row,
// so "example.com" or "[email protected]" found
// nothing. Such words are matched with a
// word-boundary REGEXP instead.
if is_ft_indexed(&item.word) {
if !text_query.is_empty() {
text_query.push(' ');
}
text_query.push('+');
text_query.push_str(&item.word);
} else {
unindexed.push(item.word);
}
text_query.push('+');
text_query.push_str(&item.word);
}
(Value::Bytes(text_query.into_bytes()), "BOOLEAN")
// For language text (bodies, subjects) the unindexed
// words are noise words and only checked when nothing
// else is left to match; keyword text (addresses,
// contact fields) checks every word, as the other
// backends do.
if !text_query.is_empty() && !matches!(language, Language::None) {
unindexed.clear();
}
(Value::Bytes(text_query.into_bytes()), "BOOLEAN", unindexed)
}
_ => {
debug_assert!(false, "Invalid search value for text field");
continue;
}
};
let _ = write!(query, "MATCH({}) AGAINST(? IN {mode} MODE)", field.column());
values.push(value);
if unindexed.is_empty() {
let _ =
write!(query, "MATCH({}) AGAINST(? IN {mode} MODE)", field.column());
values.push(value);
} else {
query.push('(');
let is_empty = matches!(&value, Value::Bytes(v) if v.is_empty());
if !is_empty {
let _ = write!(
query,
"MATCH({}) AGAINST(? IN {mode} MODE) AND ",
field.column()
);
values.push(value);
}
for (i, word) in unindexed.iter().enumerate() {
if i > 0 {
query.push_str(" AND ");
}
let _ = write!(query, "{} REGEXP ?", field.column());
values.push(Value::Bytes(
format!("(^|[^[:alnum:]]){word}([^[:alnum:]]|$)").into_bytes(),
));
}
query.push(')');
}
} else if let SearchValue::KeyValues(kv) = value {
let (key, value) = kv.iter().next().unwrap();