Add agent heartbeat monitoring and fix a query-language lexer bug

Agents now send an independent "still alive" record on a configurable
schedule (seconds/minutes/hours, [heartbeat] in agent.toml), separate
from real log traffic and tagged with a sentry.heartbeat attribute.
No new wire protocol -- it's an ordinary record through the same
PushBatch RPC/mTLS identity every log line already uses. Unavailability
alerting reuses the existing absence-condition alert rule type
unchanged; no new alerting code was needed. See
/docs/agent-heartbeat-monitoring.md for the design and how to build the
alert rule.

While verifying the alert rule live, found that the query language's
lexer never treated '-' as part of an identifier, so any unquoted
hyphenated filter value -- including the reference doc's own canonical
example, `host!=host-03` -- failed to parse at all. Fixed in
api/internal/querylang/lexer/lexer.go with regression tests; a leading
'-' still lexes as its own token so earliest=-1h/sort -count are
unaffected.
This commit is contained in:
2026-08-16 18:08:05 -07:00
parent 7d316f92db
commit 4df6931869
8 changed files with 367 additions and 3 deletions
+46
View File
@@ -97,8 +97,20 @@ pub async fn run_agent(config_path: Option<PathBuf>) -> Result<()> {
let mut batcher = Batcher::new(cfg.batch.max_size, flush_interval);
let mut ticker = tokio::time::interval(flush_interval.max(Duration::from_millis(50)));
// Heartbeat's own ticker, independent of the batch flush ticker above
// -- it always fires on cfg.heartbeat.interval regardless of
// cfg.batch's settings or whether any real log traffic is flowing.
// Built unconditionally even when disabled (tokio::time::interval
// doesn't fail on construction); the `if cfg.heartbeat.enabled`
// select! guard is what actually turns it off, so a disabled
// heartbeat costs nothing beyond one idle timer.
let mut heartbeat_ticker = tokio::time::interval(cfg.heartbeat.interval.max(Duration::from_millis(50)));
loop {
tokio::select! {
_ = heartbeat_ticker.tick(), if cfg.heartbeat.enabled => {
send_heartbeat(&mut client, &host, &service).await;
}
maybe_line = rx.recv() => {
let Some(raw) = maybe_line else {
tracing::warn!("source exited, flushing remaining batch and shutting down");
@@ -191,6 +203,40 @@ async fn spawn_source(source: config::SourceConfig, tx: source::LineSender) {
}
}
/// Sends a single synthetic record through the same `PushBatch` RPC and
/// mTLS identity as real log data -- no new proto message, no new ingest
/// code, no new ClickHouse schema. Bypasses `Batcher` (see the heartbeat
/// ticker's own comment above): a heartbeat that got queued behind
/// `batch.max_size` or `batch.flush_interval_ms` would defeat the point
/// of a punctual "still alive" signal. Distinguished from a real log
/// record purely by the `sentry.heartbeat` attribute -- `service` stays
/// the agent's real configured service so it doesn't pollute
/// service-based dashboards/faceting with a fake value. See
/// /docs/agent-heartbeat-monitoring.md for how an absence alert rule
/// turns a run of missed heartbeats into a notification.
async fn send_heartbeat(client: &mut LogIngestClient<Channel>, host: &str, service: &str) {
let record = LogRecord {
timestamp_unix_nano: now_unix_nanos(),
host: host.to_string(),
service: service.to_string(),
severity: Severity::Info as i32,
message: "agent heartbeat".to_string(),
attributes: std::collections::HashMap::from([("sentry.heartbeat".to_string(), "true".to_string())]),
record_id: String::new(),
};
match grpc::send_batch(client, format!("heartbeat-{}", batch_id()), vec![record]).await {
Ok(_) => tracing::debug!(host, "heartbeat sent"),
Err(e) => tracing::warn!(error = %e, host, "heartbeat send failed"),
}
}
fn now_unix_nanos() -> i64 {
std::time::SystemTime::now()
.duration_since(std::time::UNIX_EPOCH)
.map(|d| d.as_nanos() as i64)
.unwrap_or(0)
}
async fn flush(client: &mut LogIngestClient<Channel>, batch: Vec<LogRecord>) {
let n = batch.len();
let batch_id = batch_id();