jcoffey-dev
fde43774b4
A three-node rehearsal on PostgreSQL saw searches take about 185 ms with 80 to 260 pages in the full-text indexes' pending lists, 2 to 6 ms right after gin_clean_pending_list() or VACUUM, then creep back up as mail came in. The search tables' GIN indexes were created with the default fastupdate=on: new entries wait in an unindexed pending list that every search scans in full until VACUUM (or 4 MB of backlog) merges it, and autovacuum only visits an insert-only table after thousands of inserts. The search GIN indexes are now created WITH (fastupdate = off), so an insert pays its index update at once. The schema step runs at every startup (create_search_tables, via SearchStore::create_indexes), so indexes made before this change are switched there: when an index's reloptions don't already turn fastupdate off, ALTER INDEX ... SET (fastupdate = off) and one gin_clean_pending_list() merge its backlog. The ALTER takes a SHARE UPDATE EXCLUSIVE lock, which blocks neither reads nor writes; after the first startup the step is one catalog read per index. A failure is logged and startup goes on (search still works, only slower). Per-table autovacuum settings for the search tables are left alone. The pending list was the only reason the insert threshold mattered for search; dead tuples and freezing are served by the defaults, and table settings would override whatever tuning the DBA has done. MySQL is unaffected: InnoDB FULLTEXT keeps new entries in an in-memory cache that queries read directly, with no setting like fastupdate. store::search_gin::postgres_gin_fastupdate (new, PostgreSQL) builds the search schema in a schema of its own and checks pg_class.reloptions: fastupdate=off on every GIN index of a fresh schema; then, with the option reset to the default and 500 rows pending, one startup turns it off everywhere and leaves no pending tuples (pgstatginindex); a second startup changes nothing. On main it fails at the first check.
83 lines
2.3 KiB
Rust
83 lines
2.3 KiB
Rust
/*
|
|
* SPDX-FileCopyrightText: 2020 Stalwart Labs LLC <[email protected]>
|
|
*
|
|
* SPDX-License-Identifier: AGPL-3.0-only OR LicenseRef-SEL
|
|
*
|
|
* Modified by Coffey Labs in 2026 for INBUXA.
|
|
*/
|
|
|
|
pub mod blob;
|
|
pub mod import_export;
|
|
pub mod lookup;
|
|
pub mod ops;
|
|
#[cfg(any(feature = "postgres", feature = "mysql"))]
|
|
pub mod pool_timeout; // inbuxa: SQL pools give up instead of hanging
|
|
pub mod query;
|
|
pub mod registry;
|
|
#[cfg(feature = "postgres")]
|
|
pub mod replica; // inbuxa: read replicas
|
|
#[cfg(feature = "mysql")]
|
|
pub mod replica_mysql; // inbuxa: read replicas on MySQL
|
|
#[cfg(all(feature = "postgres", feature = "redis"))]
|
|
pub mod replica_cluster; // inbuxa: read replicas across nodes
|
|
pub mod scaleout; // inbuxa: scale-out storage
|
|
#[cfg(feature = "postgres")]
|
|
pub mod search_gin; // inbuxa: GIN indexes without a pending list
|
|
#[cfg(any(feature = "postgres", feature = "mysql"))]
|
|
pub mod sql_timeout;
|
|
pub mod task_locks; // inbuxa: task locks across nodes
|
|
|
|
use crate::utils::server::TestServerBuilder;
|
|
use std::io::Read;
|
|
|
|
#[tokio::test(flavor = "multi_thread")]
|
|
pub async fn store_tests() {
|
|
let test = TestServerBuilder::new("store_tests").await.build().await;
|
|
|
|
println!("Testing store {}...", std::env::var("STORE").unwrap());
|
|
|
|
test.destroy_store().await;
|
|
|
|
registry::test(&test).await;
|
|
import_export::test(&test).await;
|
|
ops::test(&test).await;
|
|
#[cfg(any(feature = "postgres", feature = "mysql"))]
|
|
sql_timeout::test(&test).await;
|
|
|
|
if test.is_reset() {
|
|
test.temp_dir.delete();
|
|
}
|
|
}
|
|
|
|
#[tokio::test(flavor = "multi_thread")]
|
|
pub async fn search_tests() {
|
|
let test = TestServerBuilder::new("search_store_tests")
|
|
.await
|
|
.build()
|
|
.await;
|
|
|
|
println!(
|
|
"Testing search store {}...",
|
|
std::env::var("SEARCH_STORE").unwrap_or("default".to_string())
|
|
);
|
|
|
|
query::test(&test).await;
|
|
|
|
if test.is_reset() {
|
|
test.temp_dir.delete();
|
|
}
|
|
}
|
|
|
|
pub fn deflate_test_resource(name: &str) -> Vec<u8> {
|
|
let mut csv_path = std::path::PathBuf::from(env!("CARGO_MANIFEST_DIR"));
|
|
csv_path.push("resources");
|
|
csv_path.push(name);
|
|
|
|
let mut decoder = flate2::bufread::GzDecoder::new(std::io::BufReader::new(
|
|
std::fs::File::open(csv_path).unwrap(),
|
|
));
|
|
let mut result = Vec::new();
|
|
decoder.read_to_end(&mut result).unwrap();
|
|
result
|
|
}
|