SQL pools time out; task locks are a renewed five-minute lease
A 3-node rehearsal (PostgreSQL + NATS + Garage) found two ways a crash leaves work stuck: Pool hangs. The PostgreSQL pool (deadpool) was built with no timeouts, so a request waited for a free connection, and for one to be opened or recycled, for as long as it took: forever when the server stopped answering. MySQL's pool (mysql_async) has no wait timeout at all. - PostgreSQL: wait 30 s (or the store's timeout if longer), create the store's timeout or 15 s (it bounds the whole handshake, where tokio-postgres's connect_timeout covers only the TCP connect), recycle 10 s. The pool config is now always set, not only with poolMaxConnections. - MySQL: every connection is taken through MysqlStore::conn(), which gives up after 30 s. - Both: TCP keepalive after 60 s idle, so a server that vanished without closing the connection is noticed in minutes rather than the two-hour system default. The DataStore schema has no pool timeout settings, so these are fixed defaults; the store's own timeout bounds connecting on PostgreSQL. Task locks. A task lock lasted an hour, so after a hard crash the dead node's tasks waited up to an hour and five minutes. The lock is now a five-minute lease: while this node runs a task, the task manager renews its lock every third of the lifetime (InMemoryStore::renew_lock, a compare-and-set on the store backends and SET XX EX on Redis, which leaves a lock that already expired alone). A killed node's tasks run elsewhere within about five minutes plus the claim recheck. A task this node holds isn't handed to a worker again by the scan. store::pool_timeout (new): a local listener that accepts connections and never answers plays a hung server; a PostgreSQL store with a 2 s timeout returns an error in about 4 s, and a MySQL store in 30 s. Without the timeouts both wait for good. store::task_locks gains a task held for 1.5 lock lifetimes: its lease is still held, and released when the task ends.
This commit is contained in:
@@ -26,7 +26,7 @@ use std::fmt::Write;
|
||||
|
||||
impl MysqlStore {
|
||||
pub async fn index(&self, documents: Vec<IndexDocument>) -> trc::Result<()> {
|
||||
let mut conn = self.conn_pool.get_conn().await.map_err(into_error)?;
|
||||
let mut conn = self.conn().await?;
|
||||
let mut tx_opts = TxOpts::default();
|
||||
tx_opts
|
||||
.with_consistent_snapshot(false)
|
||||
@@ -96,7 +96,7 @@ impl MysqlStore {
|
||||
build_sort(&mut query, sort);
|
||||
}
|
||||
|
||||
let mut conn = self.conn_pool.get_conn().await.map_err(into_error)?;
|
||||
let mut conn = self.conn().await?;
|
||||
let s = conn.prep(query).await.map_err(into_error)?;
|
||||
|
||||
conn.exec::<i64, _, _>(s, params)
|
||||
@@ -110,7 +110,7 @@ impl MysqlStore {
|
||||
let mut query = format!("DELETE FROM {table} ");
|
||||
let params = build_filter(&mut query, &filter.filters);
|
||||
|
||||
let mut conn = self.conn_pool.get_conn().await.map_err(into_error)?;
|
||||
let mut conn = self.conn().await?;
|
||||
let s = conn.prep(&query).await.map_err(into_error)?;
|
||||
|
||||
match conn.exec_drop(s, params.clone()).await {
|
||||
|
||||
Reference in New Issue
Block a user