inbuxa/inbuxa-server
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 6 Wiki Activity

SQL queries time out; readiness follows the data store
ci / fork-checks (pull_request) Successful in 17s
Details
ci / build (pull_request) Successful in 7m13s
Details

Browse Source 
Cluster rehearsal 3: with PostgreSQL paused (docker pause, so its
kernel still answered TCP keepalives), requests on connections already
checked out hung until it came back, and /healthz/ready stayed 200
through the outage. #41 bounded getting a connection, not using one.

Client-side query limits (store::backend::query_timeout). Every
operation on a PostgreSQL or MySQL connection now runs under a time
limit. A server-side statement_timeout (or MySQL's MAX_EXECUTION_TIME,
which covers SELECTs only) can't do this: the server that would enforce
it is the one not answering. When an operation runs out, its connection
is closed instead of pooled, since a query may still be in flight on it
or a transaction open: deadpool's Object::take on PostgreSQL;
Conn::disconnect on MySQL, which marks the connection closed before it
sends anything, so the pool discards it even when the server never
answers.

- query, 2 minutes: reads, writes (the whole transaction with its
  retries), blobs, SQL lookups, search queries and indexing. These take
  milliseconds; two minutes leaves room for a large blob over a slow
  link and still ends a hang.
- maintenance, 30 minutes: range deletes (account removal, purges),
  unindexing, purge_store, and creating tables and indexes at startup,
  which can legitimately run long in one statement. Their existing
  chunked fallback for server-side statement timeouts is unchanged.
- iterate (exports, reindexing, maintenance scans) can run for hours,
  so the query limit bounds each wait for the database (preparing, the
  query starting, the next row) rather than the whole scan.

The limits are fixed, like the pool timeouts; the DataStore schema has
no field for them. Tests set them with Store::with_query_timeouts
(test_mode only).

Readiness. /healthz/ready answered 200 whenever a data store was
configured. It now reads one key from the data store with a 2 s limit
and reuses the answer for 2 s, so probes can't load the database;
while one probe runs, others get the last answer. The first failed
probe of an outage is logged. /healthz/live stays 200: restarting a
node doesn't bring its database back, and an orchestrator restarting on
failed liveness would restart every node at once. The container
HEALTHCHECK already uses /healthz/live.

Tests, store::pool_timeout (a proxy that stops forwarding while
keeping connections open plays the paused database):
- postgres_query_timeout, mysql_query_timeout (new): with four pooled
  connections open, a read, a scan and a write each fail with "Query
  timed out" 2.0 s after the pause (2 s test limit); once the proxy
  forwards again the store answers. With the limits set to an hour
  (upstream's behavior), the read was still waiting at the test's 20 s
  limit.
- postgres_readiness (new, STORE=PostgreSql): a node's data store
  goes through the proxy; /healthz/ready is 200, 503 about 4 s after
  the pause while /healthz/live stays 200, and 200 again about 2 s
  after it ends.
- postgres_pool_timeout, mysql_pool_timeout: pass as before.
store::store_tests (PostgreSql, MySql, including the MariaDB statement
timeout step) and store::task_locks (PostgreSql) pass;
store::search_tests (PostgreSql) fails at the same ordering assertion
(query.rs:684) as on main.
This commit is contained in:
jcoffey-dev
2026-09-24 16:45:54 -07:00
parent d86e7639ac
commit 08f29926d4
22 changed files with 1525 additions and 787 deletions
Download Patch File Download Diff File Expand all files Collapse all files
crates/store/src/backend/mysql/blob.rs
+50 -35
View File
@@ -10,7 +10,7 @@ use std::ops::Range;
use mysql_async::prelude::Queryable;
use super::{MysqlStore, into_error};
use super::{MysqlStore, bounded, into_error};
impl MysqlStore {
pub(crate) async fn get_blob(
@@ -19,48 +19,63 @@ impl MysqlStore {
range: Range<usize>,
) -> trc::Result<Option<Vec<u8>>> {
let mut conn = self.conn().await?;
let s = conn
.prep("SELECT v FROM t WHERE k = ?")
.await
.map_err(into_error)?;
conn.exec_first::<Vec<u8>, _, _>(&s, (key,))
.await
.map(|bytes| {
if range.start == 0 && range.end == usize::MAX {
bytes
} else {
bytes.map(|bytes| {
let limit = self.timeouts.query;
let result = tokio::time::timeout(limit, async {
let s = conn
.prep("SELECT v FROM t WHERE k = ?")
.await
.map_err(into_error)?;
conn.exec_first::<Vec<u8>, _, _>(&s, (key,))
.await
.map(|bytes| {
if range.start == 0 && range.end == usize::MAX {
bytes
.get(range.start..std::cmp::min(bytes.len(), range.end))
.unwrap_or_default()
.to_vec()
})
}
})
.map_err(into_error)
} else {
bytes.map(|bytes| {
bytes
.get(range.start..std::cmp::min(bytes.len(), range.end))
.unwrap_or_default()
.to_vec()
})
}
})
.map_err(into_error)
})
.await;
bounded(conn, result, limit)
}
pub(crate) async fn put_blob(&self, key: &[u8], data: &[u8]) -> trc::Result<()> {
let mut conn = self.conn().await?;
let s = conn
.prep("INSERT INTO t (k, v) VALUES (?, ?) ON DUPLICATE KEY UPDATE v = VALUES(v)")
.await
.map_err(into_error)?;
conn.exec_drop(&s, (key, data))
.await
.map_err(into_error)
.map(|_| ())
let limit = self.timeouts.query;
let result = tokio::time::timeout(limit, async {
let s = conn
.prep("INSERT INTO t (k, v) VALUES (?, ?) ON DUPLICATE KEY UPDATE v = VALUES(v)")
.await
.map_err(into_error)?;
conn.exec_drop(&s, (key, data))
.await
.map_err(into_error)
.map(|_| ())
})
.await;
bounded(conn, result, limit)
}
pub(crate) async fn delete_blob(&self, key: &[u8]) -> trc::Result<bool> {
let mut conn = self.conn().await?;
let s = conn
.prep("DELETE FROM t WHERE k = ?")
.await
.map_err(into_error)?;
conn.exec_iter(&s, (key,))
.await
.map_err(into_error)
.map(|hits| hits.affected_rows() > 0)
let limit = self.timeouts.query;
let result = tokio::time::timeout(limit, async {
let s = conn
.prep("DELETE FROM t WHERE k = ?")
.await
.map_err(into_error)?;
conn.exec_iter(&s, (key,))
.await
.map_err(into_error)
.map(|hits| hits.affected_rows() > 0)
})
.await;
bounded(conn, result, limit)
}
}
crates/store/src/backend/mysql/lookup.rs
+26 -21
View File
@@ -10,7 +10,7 @@ use mysql_async::{Params, Row, prelude::Queryable};
use crate::{IntoRows, QueryResult, QueryType, Value};
use super::{MysqlStore, into_error};
use super::{MysqlStore, bounded, into_error};
impl MysqlStore {
pub(crate) async fn sql_query<T: QueryResult>(
@@ -19,27 +19,32 @@ impl MysqlStore {
params: &[Value<'_>],
) -> trc::Result<T> {
let mut conn = self.conn().await?;
let s = conn.prep(query).await.map_err(into_error)?;
let params = Params::Positional(params.iter().map(Into::into).collect());
let limit = self.timeouts.query;
let result = tokio::time::timeout(limit, async {
let s = conn.prep(query).await.map_err(into_error)?;
let params = Params::Positional(params.iter().map(Into::into).collect());
match T::query_type() {
QueryType::Execute => conn.exec_drop(s, params).await.map_or_else(
|e| Err(into_error(e)),
|_| Ok(T::from_exec(conn.affected_rows() as usize)),
),
QueryType::Exists => conn
.exec_first::<Row, _, _>(s, params)
.await
.map_or_else(|e| Err(into_error(e)), |r| Ok(T::from_exists(r.is_some()))),
QueryType::QueryOne => conn
.exec_first::<Row, _, _>(s, params)
.await
.map_or_else(|e| Err(into_error(e)), |r| Ok(T::from_query_one(r))),
QueryType::QueryAll => conn
.exec::<Row, _, _>(s, params)
.await
.map_or_else(|e| Err(into_error(e)), |r| Ok(T::from_query_all(r))),
}
match T::query_type() {
QueryType::Execute => conn.exec_drop(s, params).await.map_or_else(
|e| Err(into_error(e)),
|_| Ok(T::from_exec(conn.affected_rows() as usize)),
),
QueryType::Exists => conn
.exec_first::<Row, _, _>(s, params)
.await
.map_or_else(|e| Err(into_error(e)), |r| Ok(T::from_exists(r.is_some()))),
QueryType::QueryOne => conn
.exec_first::<Row, _, _>(s, params)
.await
.map_or_else(|e| Err(into_error(e)), |r| Ok(T::from_query_one(r))),
QueryType::QueryAll => conn
.exec::<Row, _, _>(s, params)
.await
.map_or_else(|e| Err(into_error(e)), |r| Ok(T::from_query_all(r))),
}
})
.await;
bounded(conn, result, limit)
}
}
crates/store/src/backend/mysql/main.rs
+81 -71
View File
@@ -6,7 +6,7 @@
* Modified by Coffey Labs in 2026 for INBUXA.
*/
use super::{MysqlStore, into_error};
use super::{MysqlStore, bounded, into_error};
use crate::{
backend::mysql::MysqlSearchField,
search::{
@@ -72,6 +72,7 @@ impl MysqlStore {
.db_name(Some(replica.database.clone()))
.tcp_port(replica.port as u16),
),
timeouts: Default::default(),
})),
replica.host,
replica.port as u16,
@@ -81,6 +82,7 @@ impl MysqlStore {
let primary = Store::MySQL(Arc::new(MysqlStore {
conn_pool: Pool::new(opts),
timeouts: Default::default(),
}));
// ST-1: no replicas, no change
@@ -99,88 +101,96 @@ impl MysqlStore {
pub(crate) async fn create_storage_tables(&self) -> trc::Result<()> {
let mut conn = self.conn().await?;
let limit = self.timeouts.maintenance;
let result = tokio::time::timeout(limit, async {
for table in [
SUBSPACE_ACL,
SUBSPACE_TASK_QUEUE,
SUBSPACE_DELETED_ITEMS,
SUBSPACE_SPAM_SAMPLES,
crate::SUBSPACE_INBUXA, // inbuxa: masked email
SUBSPACE_BLOB_LINK,
SUBSPACE_IN_MEMORY_VALUE,
SUBSPACE_PROPERTY,
SUBSPACE_REGISTRY,
SUBSPACE_REGISTRY_PK,
SUBSPACE_DIRECTORY,
SUBSPACE_QUEUE_MESSAGE,
SUBSPACE_QUEUE_EVENT,
SUBSPACE_REPORT_OUT,
SUBSPACE_REPORT_IN,
SUBSPACE_LOGS,
SUBSPACE_TELEMETRY_SPAN,
SUBSPACE_TELEMETRY_METRIC,
] {
let table = char::from(table);
conn.query_drop(format!(
"CREATE TABLE IF NOT EXISTS {table} (
k VARBINARY(255) NOT NULL,
v MEDIUMBLOB NOT NULL,
PRIMARY KEY (k)
) ENGINE=InnoDB"
))
.await
.map_err(into_error)?;
}
for table in [
SUBSPACE_ACL,
SUBSPACE_TASK_QUEUE,
SUBSPACE_DELETED_ITEMS,
SUBSPACE_SPAM_SAMPLES,
crate::SUBSPACE_INBUXA, // inbuxa: masked email
SUBSPACE_BLOB_LINK,
SUBSPACE_IN_MEMORY_VALUE,
SUBSPACE_PROPERTY,
SUBSPACE_REGISTRY,
SUBSPACE_REGISTRY_PK,
SUBSPACE_DIRECTORY,
SUBSPACE_QUEUE_MESSAGE,
SUBSPACE_QUEUE_EVENT,
SUBSPACE_REPORT_OUT,
SUBSPACE_REPORT_IN,
SUBSPACE_LOGS,
SUBSPACE_TELEMETRY_SPAN,
SUBSPACE_TELEMETRY_METRIC,
] {
let table = char::from(table);
conn.query_drop(format!(
"CREATE TABLE IF NOT EXISTS {table} (
k VARBINARY(255) NOT NULL,
v MEDIUMBLOB NOT NULL,
PRIMARY KEY (k)
) ENGINE=InnoDB"
))
.await
.map_err(into_error)?;
}
conn.query_drop(format!(
"CREATE TABLE IF NOT EXISTS {} (
k VARBINARY(255) NOT NULL,
v LONGBLOB NOT NULL,
PRIMARY KEY (k)
) ENGINE=InnoDB",
char::from(SUBSPACE_BLOBS),
))
.await
.map_err(into_error)?;
for table in [SUBSPACE_INDEXES, SUBSPACE_REGISTRY_IDX] {
let table = char::from(table);
conn.query_drop(format!(
"CREATE TABLE IF NOT EXISTS {table} (
k BLOB,
PRIMARY KEY (k(400))
) ENGINE=InnoDB"
))
.await
.map_err(into_error)?;
}
for table in [SUBSPACE_COUNTER, SUBSPACE_QUOTA, SUBSPACE_IN_MEMORY_COUNTER] {
conn.query_drop(format!(
"CREATE TABLE IF NOT EXISTS {} (
k VARBINARY(255) NOT NULL,
v BIGINT NOT NULL DEFAULT 0,
PRIMARY KEY (k)
) ENGINE=InnoDB",
char::from(table)
k VARBINARY(255) NOT NULL,
v LONGBLOB NOT NULL,
PRIMARY KEY (k)
) ENGINE=InnoDB",
char::from(SUBSPACE_BLOBS),
))
.await
.map_err(into_error)?;
}
Ok(())
for table in [SUBSPACE_INDEXES, SUBSPACE_REGISTRY_IDX] {
let table = char::from(table);
conn.query_drop(format!(
"CREATE TABLE IF NOT EXISTS {table} (
k BLOB,
PRIMARY KEY (k(400))
) ENGINE=InnoDB"
))
.await
.map_err(into_error)?;
}
for table in [SUBSPACE_COUNTER, SUBSPACE_QUOTA, SUBSPACE_IN_MEMORY_COUNTER] {
conn.query_drop(format!(
"CREATE TABLE IF NOT EXISTS {} (
k VARBINARY(255) NOT NULL,
v BIGINT NOT NULL DEFAULT 0,
PRIMARY KEY (k)
) ENGINE=InnoDB",
char::from(table)
))
.await
.map_err(into_error)?;
}
Ok(())
})
.await;
bounded(conn, result, limit)
}
pub(crate) async fn create_search_tables(&self) -> trc::Result<()> {
let mut conn = self.conn().await?;
let limit = self.timeouts.maintenance;
let result = tokio::time::timeout(limit, async {
create_search_tables::<EmailSearchField>(&mut conn).await?;
create_search_tables::<CalendarSearchField>(&mut conn).await?;
create_search_tables::<ContactSearchField>(&mut conn).await?;
//create_search_tables::<FileSearchField>(&mut conn).await?;
create_search_tables::<TracingSearchField>(&mut conn).await?;
create_search_tables::<EmailSearchField>(&mut conn).await?;
create_search_tables::<CalendarSearchField>(&mut conn).await?;
create_search_tables::<ContactSearchField>(&mut conn).await?;
//create_search_tables::<FileSearchField>(&mut conn).await?;
create_search_tables::<TracingSearchField>(&mut conn).await?;
Ok(())
Ok(())
})
.await;
bounded(conn, result, limit)
}
}
crates/store/src/backend/mysql/mod.rs
+41 -1
View File
@@ -6,6 +6,7 @@
* Modified by Coffey Labs in 2026 for INBUXA.
*/
use crate::backend::query_timeout::QueryTimeouts;
use crate::{
search::{
CalendarSearchField, ContactSearchField, EmailSearchField, FileSearchField, SearchField,
@@ -14,7 +15,7 @@ use crate::{
write::SearchIndex,
};
use mysql_async::Pool;
use std::fmt::Display;
use std::{fmt::Display, time::Duration};
pub mod blob;
pub mod lookup;
@@ -25,6 +26,8 @@ pub mod write;
pub struct MysqlStore {
pub(crate) conn_pool: Pool,
/// inbuxa: client-side query limits (see backend::query_timeout)
pub(crate) timeouts: QueryTimeouts,
}
/// inbuxa: how long a request waits for a pooled connection (including
@@ -54,6 +57,43 @@ pub(crate) async fn pool_conn(
}
}
/// inbuxa: the error for an operation that ran past its time limit.
pub(crate) fn query_timeout_error(limit: Duration) -> trc::Error {
trc::StoreEvent::MysqlError
.reason("Query timed out")
.details(format!(
"No answer from the database within {} s",
limit.as_secs()
))
}
/// inbuxa: ends an operation run on `conn` under `limit`. When it ran out,
/// the connection is closed rather than returned to the pool: a query may
/// still be in flight on it, or a transaction open. Conn::disconnect marks
/// the connection closed before it sends anything, so even when the server
/// doesn't answer and the attempt is dropped, the pool discards it instead
/// of waiting to clean it up.
pub(crate) fn bounded<T>(
conn: mysql_async::Conn,
result: Result<trc::Result<T>, tokio::time::error::Elapsed>,
limit: Duration,
) -> trc::Result<T> {
match result {
Ok(result) => result,
Err(_) => {
discard(conn);
Err(query_timeout_error(limit))
}
}
}
/// inbuxa: closes a connection whose state is unknown (see bounded).
pub(crate) fn discard(conn: mysql_async::Conn) {
tokio::spawn(async move {
let _ = tokio::time::timeout(Duration::from_secs(1), conn.disconnect()).await;
});
}
#[inline(always)]
pub(crate) fn into_error(err: impl Display) -> trc::Error {
trc::StoreEvent::MysqlError.reason(err)
crates/store/src/backend/mysql/read.rs
+109 -66
View File
@@ -6,7 +6,7 @@
* Modified by Coffey Labs in 2026 for INBUXA.
*/
use super::{MysqlStore, into_error, is_timeout_error};
use super::{MysqlStore, bounded, discard, into_error, is_timeout_error, query_timeout_error};
use crate::{Deserialize, IterateParams, Key, ValueKey, write::ValueClass};
use futures::TryStreamExt;
use mysql_async::{Row, prelude::Queryable};
@@ -17,40 +17,50 @@ impl MysqlStore {
U: Deserialize + 'static,
{
let mut conn = self.conn().await?;
let s = conn
.prep(format!(
"SELECT v FROM {} WHERE k = ?",
char::from(key.subspace())
))
.await
.map_err(into_error)?;
let key = key.serialize(0);
conn.exec_first::<Vec<u8>, _, _>(&s, (&key,))
.await
.map_err(into_error)
.and_then(|r| {
if let Some(r) = r {
Ok(Some(U::deserialize_owned_with_key(&key, r)?))
} else {
Ok(None)
}
})
let limit = self.timeouts.query;
let result = tokio::time::timeout(limit, async {
let s = conn
.prep(format!(
"SELECT v FROM {} WHERE k = ?",
char::from(key.subspace())
))
.await
.map_err(into_error)?;
let key = key.serialize(0);
conn.exec_first::<Vec<u8>, _, _>(&s, (&key,))
.await
.map_err(into_error)
.and_then(|r| {
if let Some(r) = r {
Ok(Some(U::deserialize_owned_with_key(&key, r)?))
} else {
Ok(None)
}
})
})
.await;
bounded(conn, result, limit)
}
pub(crate) async fn key_exists(&self, key: impl Key) -> trc::Result<bool> {
let mut conn = self.conn().await?;
let s = conn
.prep(format!(
"SELECT 1 FROM {} WHERE k = ?",
char::from(key.subspace())
))
.await
.map_err(into_error)?;
let key = key.serialize(0);
conn.exec_first::<u8, _, _>(&s, (&key,))
.await
.map_err(into_error)
.map(|r| r.is_some())
let limit = self.timeouts.query;
let result = tokio::time::timeout(limit, async {
let s = conn
.prep(format!(
"SELECT 1 FROM {} WHERE k = ?",
char::from(key.subspace())
))
.await
.map_err(into_error)?;
let key = key.serialize(0);
conn.exec_first::<u8, _, _>(&s, (&key,))
.await
.map_err(into_error)
.map(|r| r.is_some())
})
.await;
bounded(conn, result, limit)
}
pub(crate) async fn iterate<T: Key>(
@@ -64,28 +74,36 @@ impl MysqlStore {
let end = params.end.serialize(0);
let keys = if params.values { "k, v" } else { "k" };
let s = conn
.prep(&match (params.first, params.ascending) {
(true, true) => {
format!(
"SELECT {keys} FROM {table} WHERE k >= ? AND k <= ? ORDER BY k ASC LIMIT 1"
)
}
(true, false) => {
format!(
"SELECT {keys} FROM {table} WHERE k >= ? AND k <= ? ORDER BY k DESC LIMIT 1"
)
}
(false, true) => {
format!("SELECT {keys} FROM {table} WHERE k >= ? AND k <= ? ORDER BY k ASC")
}
(false, false) => {
format!("SELECT {keys} FROM {table} WHERE k >= ? AND k <= ? ORDER BY k DESC")
}
})
.await
.map_err(into_error)?;
// inbuxa: a scan may run for hours, so the query limit bounds each
// wait for the database (preparing, the query starting, the next
// row) rather than the scan. A wait that runs out closes the
// connection.
let limit = self.timeouts.query;
let query = match (params.first, params.ascending) {
(true, true) => {
format!("SELECT {keys} FROM {table} WHERE k >= ? AND k <= ? ORDER BY k ASC LIMIT 1")
}
(true, false) => {
format!(
"SELECT {keys} FROM {table} WHERE k >= ? AND k <= ? ORDER BY k DESC LIMIT 1"
)
}
(false, true) => {
format!("SELECT {keys} FROM {table} WHERE k >= ? AND k <= ? ORDER BY k ASC")
}
(false, false) => {
format!("SELECT {keys} FROM {table} WHERE k >= ? AND k <= ? ORDER BY k DESC")
}
};
let s = match tokio::time::timeout(limit, conn.prep(&query)).await {
Ok(s) => s.map_err(into_error)?,
Err(_) => {
discard(conn);
return Err(query_timeout_error(limit));
}
};
let mut from = begin;
let mut stalled = false;
let mut to = end;
let mut resume_key = None;
@@ -94,13 +112,26 @@ impl MysqlStore {
let mut timed_out = false;
{
let mut rows = conn
.exec_stream::<Row, _, _>(&s, (from.clone(), to.clone()))
.await
.map_err(into_error)?;
let mut rows = match tokio::time::timeout(
limit,
conn.exec_stream::<Row, _, _>(&s, (from.clone(), to.clone())),
)
.await
{
Ok(rows) => rows.map_err(into_error)?,
// Leaves the scan loop for the timeout below
Err(_) => break,
};
loop {
match rows.try_next().await {
let next = match tokio::time::timeout(limit, rows.try_next()).await {
Ok(next) => next,
Err(_) => {
stalled = true;
break;
}
};
match next {
Ok(Some(mut row)) => {
let value = if params.values {
row.take_opt::<Vec<u8>, _>(1)
@@ -136,6 +167,10 @@ impl MysqlStore {
}
}
if stalled {
break;
}
match last_key {
Some(last_key) if timed_out => {
if params.ascending {
@@ -148,6 +183,9 @@ impl MysqlStore {
_ => return Ok(()),
}
}
discard(conn);
Err(query_timeout_error(limit))
}
pub(crate) async fn get_counter(
@@ -158,14 +196,19 @@ impl MysqlStore {
let table = char::from(key.subspace());
let key = key.serialize(0);
let mut conn = self.conn().await?;
let s = conn
.prep(format!("SELECT v FROM {table} WHERE k = ?"))
.await
.map_err(into_error)?;
match conn.exec_first::<i64, _, _>(&s, (key,)).await {
Ok(Some(num)) => Ok(num),
Ok(None) => Ok(0),
Err(e) => Err(into_error(e)),
}
let limit = self.timeouts.query;
let result = tokio::time::timeout(limit, async {
let s = conn
.prep(format!("SELECT v FROM {table} WHERE k = ?"))
.await
.map_err(into_error)?;
match conn.exec_first::<i64, _, _>(&s, (key,)).await {
Ok(Some(num)) => Ok(num),
Ok(None) => Ok(0),
Err(e) => Err(into_error(e)),
}
})
.await;
bounded(conn, result, limit)
}
}
crates/store/src/backend/mysql/search.rs
+96 -79
View File
@@ -10,8 +10,8 @@ use crate::{
backend::{
MAX_TOKEN_LENGTH,
mysql::{
DELETE_CHUNK_SIZE, MIN_DELETE_CHUNK_SIZE, MysqlSearchField, MysqlStore, into_error,
is_timeout_error,
DELETE_CHUNK_SIZE, MIN_DELETE_CHUNK_SIZE, MysqlSearchField, MysqlStore, bounded,
into_error, is_timeout_error,
},
},
search::{
@@ -27,57 +27,62 @@ use std::fmt::Write;
impl MysqlStore {
pub async fn index(&self, documents: Vec<IndexDocument>) -> trc::Result<()> {
let mut conn = self.conn().await?;
let mut tx_opts = TxOpts::default();
tx_opts
.with_consistent_snapshot(false)
.with_isolation_level(IsolationLevel::ReadCommitted);
let mut trx = conn.start_transaction(tx_opts).await.map_err(into_error)?;
let limit = self.timeouts.query;
let result = tokio::time::timeout(limit, async {
let mut tx_opts = TxOpts::default();
tx_opts
.with_consistent_snapshot(false)
.with_isolation_level(IsolationLevel::ReadCommitted);
let mut trx = conn.start_transaction(tx_opts).await.map_err(into_error)?;
for document in documents {
let index = document.index;
let primary_keys = index.primary_keys();
let all_fields = index.all_fields();
let mut fields = document.fields;
let mut values = Vec::with_capacity(fields.len() + 2);
let mut query = format!("INSERT INTO {} (", index.mysql_table());
for document in documents {
let index = document.index;
let primary_keys = index.primary_keys();
let all_fields = index.all_fields();
let mut fields = document.fields;
let mut values = Vec::with_capacity(fields.len() + 2);
let mut query = format!("INSERT INTO {} (", index.mysql_table());
for (i, field) in primary_keys.iter().chain(all_fields).enumerate() {
if i > 0 {
query.push(',');
for (i, field) in primary_keys.iter().chain(all_fields).enumerate() {
if i > 0 {
query.push(',');
}
query.push_str(field.column());
}
query.push_str(field.column());
query.push_str(") VALUES (");
for (i, field) in primary_keys.iter().chain(all_fields).enumerate() {
if i > 0 {
query.push(',');
}
if let Some(value) = fields.remove(field) {
query.push('?');
values.push(value);
} else {
query.push_str("NULL");
}
}
query.push_str(") ON DUPLICATE KEY UPDATE ");
for (i, field) in all_fields.iter().enumerate() {
if i > 0 {
query.push(',');
}
let column = field.column();
let _ = write!(&mut query, "{column} = VALUES({column})");
}
let s = trx.prep(&query).await.map_err(into_error)?;
trx.exec_drop(&s, values).await.map_err(into_error)?;
}
query.push_str(") VALUES (");
for (i, field) in primary_keys.iter().chain(all_fields).enumerate() {
if i > 0 {
query.push(',');
}
if let Some(value) = fields.remove(field) {
query.push('?');
values.push(value);
} else {
query.push_str("NULL");
}
}
query.push_str(") ON DUPLICATE KEY UPDATE ");
for (i, field) in all_fields.iter().enumerate() {
if i > 0 {
query.push(',');
}
let column = field.column();
let _ = write!(&mut query, "{column} = VALUES({column})");
}
let s = trx.prep(&query).await.map_err(into_error)?;
trx.exec_drop(&s, values).await.map_err(into_error)?;
}
trx.commit().await.map_err(into_error)
trx.commit().await.map_err(into_error)
})
.await;
bounded(conn, result, limit)
}
pub async fn query<R: SearchDocumentId>(
@@ -97,12 +102,17 @@ impl MysqlStore {
}
let mut conn = self.conn().await?;
let s = conn.prep(query).await.map_err(into_error)?;
let limit = self.timeouts.query;
let result = tokio::time::timeout(limit, async {
let s = conn.prep(query).await.map_err(into_error)?;
conn.exec::<i64, _, _>(s, params)
.await
.map(|r| r.into_iter().map(|r| R::from_u64(r as u64)).collect())
.map_err(into_error)
conn.exec::<i64, _, _>(s, params)
.await
.map(|r| r.into_iter().map(|r| R::from_u64(r as u64)).collect())
.map_err(into_error)
})
.await;
bounded(conn, result, limit)
}
pub async fn unindex(&self, filter: SearchQuery) -> trc::Result<u64> {
@@ -111,40 +121,47 @@ impl MysqlStore {
let params = build_filter(&mut query, &filter.filters);
let mut conn = self.conn().await?;
let s = conn.prep(&query).await.map_err(into_error)?;
let limit = self.timeouts.maintenance;
let result = tokio::time::timeout(limit, async {
let s = conn.prep(&query).await.map_err(into_error)?;
match conn.exec_drop(s, params.clone()).await {
Ok(_) => return Ok(conn.affected_rows()),
Err(err) if is_timeout_error(&err) => (),
Err(err) => return Err(into_error(err)),
}
match conn.exec_drop(s, params.clone()).await {
Ok(_) => return Ok(conn.affected_rows()),
Err(err) if is_timeout_error(&err) => (),
Err(err) => return Err(into_error(err)),
}
let mut chunk_size = DELETE_CHUNK_SIZE;
let mut deleted = 0;
loop {
let s = conn
.prep(format!("{query} LIMIT {chunk_size}"))
.await
.map_err(into_error)?;
let mut chunk_size = DELETE_CHUNK_SIZE;
let mut deleted = 0;
loop {
match conn.exec_drop(&s, params.clone()).await {
Ok(_) => {
let affected = conn.affected_rows();
if affected == 0 {
return Ok(deleted);
let s = conn
.prep(format!("{query} LIMIT {chunk_size}"))
.await
.map_err(into_error)?;
loop {
match conn.exec_drop(&s, params.clone()).await {
Ok(_) => {
let affected = conn.affected_rows();
if affected == 0 {
return Ok(deleted);
}
deleted += affected;
}
deleted += affected;
Err(err)
if is_timeout_error(&err) && chunk_size > MIN_DELETE_CHUNK_SIZE =>
{
chunk_size = (chunk_size / 2).max(MIN_DELETE_CHUNK_SIZE);
break;
}
Err(err) => return Err(into_error(err)),
}
Err(err) if is_timeout_error(&err) && chunk_size > MIN_DELETE_CHUNK_SIZE => {
chunk_size = (chunk_size / 2).max(MIN_DELETE_CHUNK_SIZE);
break;
}
Err(err) => return Err(into_error(err)),
}
}
}
})
.await;
bounded(conn, result, limit)
}
}
crates/store/src/backend/mysql/write.rs
+100 -84
View File
@@ -6,7 +6,9 @@
* Modified by Coffey Labs in 2026 for INBUXA.
*/
use super::{DELETE_CHUNK_SIZE, MIN_DELETE_CHUNK_SIZE, MysqlStore, into_error, is_timeout_error};
use super::{
DELETE_CHUNK_SIZE, MIN_DELETE_CHUNK_SIZE, MysqlStore, bounded, into_error, is_timeout_error,
};
use crate::{
IndexKey, Key, LogKey, SUBSPACE_COUNTER, SUBSPACE_IN_MEMORY_COUNTER, SUBSPACE_QUOTA,
SUBSPACE_REGISTRY_IDX,
@@ -32,41 +34,45 @@ impl MysqlStore {
let start = Instant::now();
let mut retry_count = 0;
let mut conn = self.conn().await?;
loop {
let err = match self.write_trx(&mut conn, &mut batch).await {
Ok(result) => {
return Ok(result);
}
Err(err) => err,
};
let _ = conn.query_drop("ROLLBACK;").await;
match err {
CommitError::Mysql(Error::Server(err))
if [1062, 1213].contains(&err.code)
&& retry_count < MAX_COMMIT_ATTEMPTS
&& start.elapsed() < MAX_COMMIT_TIME => {}
/*CommitError::Retry => {
if retry_count > MAX_COMMIT_ATTEMPTS || start.elapsed() > MAX_COMMIT_TIME {
return Err(trc::StoreEvent::AssertValueFailed
.into_err()
.caused_by(trc::location!()));
let limit = self.timeouts.query;
let result = tokio::time::timeout(limit, async {
loop {
let err = match self.write_trx(&mut conn, &mut batch).await {
Ok(result) => {
return Ok(result);
}
}*/
CommitError::Mysql(err) => {
return Err(into_error(err));
}
CommitError::Internal(err) => {
return Err(err);
}
}
Err(err) => err,
};
let backoff = rand::rng().random_range(50..=300);
tokio::time::sleep(Duration::from_millis(backoff)).await;
retry_count += 1;
}
let _ = conn.query_drop("ROLLBACK;").await;
match err {
CommitError::Mysql(Error::Server(err))
if [1062, 1213].contains(&err.code)
&& retry_count < MAX_COMMIT_ATTEMPTS
&& start.elapsed() < MAX_COMMIT_TIME => {}
/*CommitError::Retry => {
if retry_count > MAX_COMMIT_ATTEMPTS || start.elapsed() > MAX_COMMIT_TIME {
return Err(trc::StoreEvent::AssertValueFailed
.into_err()
.caused_by(trc::location!()));
}
}*/
CommitError::Mysql(err) => {
return Err(into_error(err));
}
CommitError::Internal(err) => {
return Err(err);
}
}
let backoff = rand::rng().random_range(50..=300);
tokio::time::sleep(Duration::from_millis(backoff)).await;
retry_count += 1;
}
})
.await;
bounded(conn, result, limit)
}
async fn write_trx(
@@ -385,71 +391,81 @@ impl MysqlStore {
pub(crate) async fn purge_store(&self) -> trc::Result<()> {
let mut conn = self.conn().await?;
for subspace in [SUBSPACE_QUOTA, SUBSPACE_COUNTER, SUBSPACE_IN_MEMORY_COUNTER] {
purge_table(&mut conn, char::from(subspace)).await?;
}
let limit = self.timeouts.maintenance;
let result = tokio::time::timeout(limit, async {
for subspace in [SUBSPACE_QUOTA, SUBSPACE_COUNTER, SUBSPACE_IN_MEMORY_COUNTER] {
purge_table(&mut conn, char::from(subspace)).await?;
}
Ok(())
Ok(())
})
.await;
bounded(conn, result, limit)
}
pub(crate) async fn delete_range(&self, from: impl Key, to: impl Key) -> trc::Result<()> {
let mut conn = self.conn().await?;
let table = char::from(from.subspace());
let mut from = from.serialize(0);
let to = to.serialize(0);
let limit = self.timeouts.maintenance;
let result = tokio::time::timeout(limit, async {
let table = char::from(from.subspace());
let mut from = from.serialize(0);
let to = to.serialize(0);
let delete = conn
.prep(format!("DELETE FROM {table} WHERE k >= ? AND k < ?"))
.await
.map_err(into_error)?;
match conn.exec_drop(&delete, (&from, &to)).await {
Ok(_) => return Ok(()),
Err(err) if is_timeout_error(&err) => (),
Err(err) => return Err(into_error(err)),
}
let mut chunk_size = DELETE_CHUNK_SIZE;
loop {
let boundary = conn
.prep(format!(
"SELECT k FROM {table} WHERE k >= ? AND k < ? ORDER BY k ASC LIMIT 1 OFFSET {chunk_size}"
))
let delete = conn
.prep(format!("DELETE FROM {table} WHERE k >= ? AND k < ?"))
.await
.map_err(into_error)?;
match conn.exec_drop(&delete, (&from, &to)).await {
Ok(_) => return Ok(()),
Err(err) if is_timeout_error(&err) => (),
Err(err) => return Err(into_error(err)),
}
let mut chunk_size = DELETE_CHUNK_SIZE;
loop {
let next = match conn
.exec_first::<Vec<u8>, _, _>(&boundary, (&from, &to))
let boundary = conn
.prep(format!(
"SELECT k FROM {table} WHERE k >= ? AND k < ? ORDER BY k ASC LIMIT 1 OFFSET {chunk_size}"
))
.await
{
Ok(next) => next,
Err(err) if is_timeout_error(&err) && chunk_size > MIN_DELETE_CHUNK_SIZE => {
chunk_size = (chunk_size / 2).max(MIN_DELETE_CHUNK_SIZE);
break;
}
Err(err) => return Err(into_error(err)),
};
.map_err(into_error)?;
match conn
.exec_drop(&delete, (&from, next.as_ref().unwrap_or(&to)))
.await
{
Ok(_) => (),
Err(err) if is_timeout_error(&err) && chunk_size > MIN_DELETE_CHUNK_SIZE => {
chunk_size = (chunk_size / 2).max(MIN_DELETE_CHUNK_SIZE);
break;
}
Err(err) => return Err(into_error(err)),
}
loop {
let next = match conn
.exec_first::<Vec<u8>, _, _>(&boundary, (&from, &to))
.await
{
Ok(next) => next,
Err(err) if is_timeout_error(&err) && chunk_size > MIN_DELETE_CHUNK_SIZE => {
chunk_size = (chunk_size / 2).max(MIN_DELETE_CHUNK_SIZE);
break;
}
Err(err) => return Err(into_error(err)),
};
match next {
Some(next) => from = next,
None => return Ok(()),
match conn
.exec_drop(&delete, (&from, next.as_ref().unwrap_or(&to)))
.await
{
Ok(_) => (),
Err(err) if is_timeout_error(&err) && chunk_size > MIN_DELETE_CHUNK_SIZE => {
chunk_size = (chunk_size / 2).max(MIN_DELETE_CHUNK_SIZE);
break;
}
Err(err) => return Err(into_error(err)),
}
match next {
Some(next) => from = next,
None => return Ok(()),
}
}
}
}
})
.await;
bounded(conn, result, limit)
}
}