coffey-labs/cairnobs
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Add agent inventory, management, and remote config

Browse Source 
Extends the heartbeat mechanism with a second gRPC service on the same
mTLS channel (AgentControl.CheckIn, agent-initiated on the existing
heartbeat ticker -- still push-only, no inbound port on any agent) so
an agent reports its running config and can pick up an operator-set
override. A new web UI section (/agents) lists every agent that's
checked in, shows its reported config, and lets an operator edit a
narrow, deliberately-scoped subset remotely: batch/heartbeat tuning,
and (journald sources only) the unit filter.

TLS material and the ingest endpoint are never reportable or remotely
editable, by proto shape rather than a validation rule -- a bad or
malicious edit there could permanently strand an agent or redirect
where its logs go, unlike every other editable field, which only
degrades behavior.

An override lives only in the agent's memory (agent.toml is never
rewritten) and re-syncs on the agent's own schedule; changing the
journald filter aborts and respawns the source task since there's no
other way to change what's being tailed. Building the hot-reload path
surfaced a real, independent, pre-existing bug: shutdown was using
poll_timeout(), which only drains once flush_interval has elapsed,
silently dropping anything buffered more recently on every graceful
shutdown that landed between flushes -- fixed with a new unconditional
Batcher::flush_all(), now used at both shutdown and hot-reload.

Verified live end-to-end against a real stack: an edited heartbeat
interval changed a running agent's actual send cadence within one
check-in cycle (confirmed by the real timestamps landing in
ClickHouse), and an edited journald filter triggered a real source
restart, both reflected back in the next reported-config snapshot.

See /docs/agent-management-design.md.
This commit is contained in:
jcoffey-dev
2026-08-16 18:08:51 -07:00
parent 4df6931869
commit 4f0da1ae5e
29 changed files with 2618 additions and 53 deletions
Download Patch File Download Diff File Expand all files Collapse all files
agent/sentry-agent/build.rs
+4 -1
View File
@@ -2,7 +2,10 @@ fn main() -> Result<(), Box<dyn std::error::Error>> {
tonic_build::configure()
.build_server(false)
.compile_protos(
&["../../proto/sentry/logs/v1/logs.proto"],
&[
"../../proto/sentry/logs/v1/logs.proto",
"../../proto/sentry/agent/v1/agent_control.proto",
],
&["../../proto"],
)?;
Ok(())
agent/sentry-agent/src/batch.rs
+29
View File
@@ -45,6 +45,20 @@ impl Batcher {
}
}
/// Unconditionally drains whatever is buffered, ignoring both
/// `max_size` and `flush_interval` -- for shutdown and for a
/// config hot-reload replacing this `Batcher` outright (Phase:
/// agent management's remote config editing), neither of which
/// should silently drop records just because the timeout hadn't
/// elapsed yet.
pub fn flush_all(&mut self) -> Option<Vec<LogRecord>> {
if self.buf.is_empty() {
None
} else {
Some(self.drain())
}
}
fn drain(&mut self) -> Vec<LogRecord> {
self.last_flush = Instant::now();
std::mem::replace(&mut self.buf, Vec::with_capacity(self.max_size))
@@ -106,4 +120,19 @@ mod tests {
b.push(rec("a"));
assert!(b.poll_timeout().is_none());
}
// Regression test: shutdown used to call poll_timeout(), which
// silently drops anything buffered before flush_interval elapses --
// real data loss on a graceful shutdown that happened to land
// between flushes. flush_all() is what shutdown (and hot-reload)
// must use instead.
#[test]
fn flush_all_drains_regardless_of_timeout() {
let mut b = Batcher::new(10, Duration::from_secs(999));
b.push(rec("a"));
assert!(b.poll_timeout().is_none(), "sanity: timeout hasn't elapsed");
let batch = b.flush_all().expect("flush_all should drain unconditionally");
assert_eq!(batch.len(), 1);
assert!(b.flush_all().is_none(), "buffer should be empty after draining");
}
}
agent/sentry-agent/src/grpc.rs
+20 -8
View File
@@ -1,12 +1,18 @@
use crate::config::{IngestConfig, TlsConfig};
use crate::pb::agent::v1::{agent_control_client::AgentControlClient, CheckInRequest, CheckInResponse};
use crate::pb::{log_ingest_client::LogIngestClient, LogRecord, PushBatchRequest};
use anyhow::{Context, Result};
use tonic::transport::{Certificate, Channel, ClientTlsConfig, Identity};
/// Establishes an mTLS gRPC channel to the ingest service. Agents never
/// talk to Redpanda directly — this is the only network egress the agent
/// has, by design (see /docs/architecture.md).
pub async fn connect(ingest: &IngestConfig, tls: &TlsConfig) -> Result<LogIngestClient<Channel>> {
/// Establishes the one mTLS gRPC channel an agent has to ingest —
/// agents never talk to Redpanda directly, and never accept an inbound
/// connection either (see /docs/architecture.md and
/// /docs/agent-management-design.md). Returns the bare `Channel` rather
/// than a client wrapper so callers can build both `LogIngestClient`
/// (data plane) and `AgentControlClient` (control plane) from the same
/// connection — `Channel` is a cheap-to-clone handle, not the socket
/// itself, so there's no cost to sharing it across two client stubs.
pub async fn connect(ingest: &IngestConfig, tls: &TlsConfig) -> Result<Channel> {
let ca = tokio::fs::read(&tls.ca_cert)
.await
.with_context(|| format!("reading CA cert at {}", tls.ca_cert.display()))?;
@@ -21,15 +27,13 @@ pub async fn connect(ingest: &IngestConfig, tls: &TlsConfig) -> Result<LogIngest
.ca_certificate(Certificate::from_pem(ca))
.identity(Identity::from_pem(cert, key));
let channel = Channel::from_shared(ingest.endpoint.clone())
Channel::from_shared(ingest.endpoint.clone())
.context("invalid ingest endpoint URL")?
.tls_config(tls_config)
.context("configuring mTLS")?
.connect()
.await
.context("connecting to ingest service")?;
Ok(LogIngestClient::new(channel))
.context("connecting to ingest service")
}
pub async fn send_batch(
@@ -43,3 +47,11 @@ pub async fn send_batch(
.context("PushBatch RPC failed")?;
Ok(resp.into_inner().accepted)
}
pub async fn check_in(
client: &mut AgentControlClient<Channel>,
req: CheckInRequest,
) -> Result<CheckInResponse> {
let resp = client.check_in(req).await.context("CheckIn RPC failed")?;
Ok(resp.into_inner())
}
agent/sentry-agent/src/main.rs
+183 -15
View File
@@ -8,12 +8,19 @@ mod service;
pub mod pb {
tonic::include_proto!("sentry.logs.v1");
pub mod agent {
pub mod v1 {
tonic::include_proto!("sentry.agent.v1");
}
}
}
use anyhow::{Context, Result};
use batch::Batcher;
use clap::Parser;
use config::Config;
use pb::agent::v1::{agent_control_client::AgentControlClient, CheckInRequest, DesiredOverride, ReportedConfig};
use pb::{log_ingest_client::LogIngestClient, LogRecord, Severity};
use std::path::PathBuf;
use std::time::Duration;
@@ -85,31 +92,92 @@ pub async fn run_agent(config_path: Option<PathBuf>) -> Result<()> {
let host = cfg.agent.host.clone().unwrap_or_else(default_hostname);
let service = cfg.agent.service.clone();
let (tx, mut rx) = mpsc::channel(1024);
let source_handle = tokio::spawn(spawn_source(cfg.source.clone(), tx));
// source_cfg is mutable: a remote DesiredOverride's journald_unit
// field (see apply_override below) can change it at runtime, which
// means aborting and respawning the source task with the new
// filter -- source_handle/rx are mutable for the same reason.
let mut source_cfg = cfg.source.clone();
let (mut source_handle, mut rx) = spawn_source_task(source_cfg.clone());
let mut client = grpc::connect(&cfg.ingest, &cfg.tls)
let channel = grpc::connect(&cfg.ingest, &cfg.tls)
.await
.context("connecting to ingest service")?;
let mut client = LogIngestClient::new(channel.clone());
let mut control_client = AgentControlClient::new(channel);
tracing::info!(endpoint = %cfg.ingest.endpoint, "connected to ingest service");
let flush_interval = Duration::from_millis(cfg.batch.flush_interval_ms);
let mut batcher = Batcher::new(cfg.batch.max_size, flush_interval);
// Effective runtime settings, seeded from local config -- every one
// of these is mutable because a remote DesiredOverride can change
// it (see apply_override). The local agent.toml is never rewritten;
// an override lives only in memory here and reverts to agent.toml's
// own values on restart, re-syncing on the next successful CheckIn
// (see /docs/agent-management-design.md's merge-semantics section).
let mut batch_max_size = cfg.batch.max_size;
let mut flush_interval = Duration::from_millis(cfg.batch.flush_interval_ms);
let mut heartbeat_enabled = cfg.heartbeat.enabled;
let mut heartbeat_interval = cfg.heartbeat.interval;
// Empty until the first override is ever applied -- echoed back on
// every CheckIn as-is so the server can tell "pending" (an edit
// exists this agent hasn't picked up) from "applied."
let mut applied_override_version = String::new();
let mut batcher = Batcher::new(batch_max_size, flush_interval);
let mut ticker = tokio::time::interval(flush_interval.max(Duration::from_millis(50)));
// Heartbeat's own ticker, independent of the batch flush ticker above
// -- it always fires on cfg.heartbeat.interval regardless of
// cfg.batch's settings or whether any real log traffic is flowing.
// Built unconditionally even when disabled (tokio::time::interval
// doesn't fail on construction); the `if cfg.heartbeat.enabled`
// select! guard is what actually turns it off, so a disabled
// heartbeat costs nothing beyond one idle timer.
let mut heartbeat_ticker = tokio::time::interval(cfg.heartbeat.interval.max(Duration::from_millis(50)));
// -- it always fires on heartbeat_interval regardless of the batch
// settings or whether any real log traffic is flowing. Also drives
// CheckIn (see the arm below) unconditionally -- CheckIn keeps
// running even when heartbeat_enabled is false, since that's an
// agent's only path to ever receive a remote override that
// re-enables it; only the heartbeat log record itself is gated on
// heartbeat_enabled.
let mut heartbeat_ticker = tokio::time::interval(heartbeat_interval.max(Duration::from_millis(50)));
loop {
tokio::select! {
_ = heartbeat_ticker.tick(), if cfg.heartbeat.enabled => {
send_heartbeat(&mut client, &host, &service).await;
_ = heartbeat_ticker.tick() => {
if heartbeat_enabled {
send_heartbeat(&mut client, &host, &service).await;
}
let reported = ReportedConfig {
agent_version: env!("CARGO_PKG_VERSION").to_string(),
source_kind: source_kind_name(&source_cfg),
source_detail: source_detail_summary(&source_cfg),
batch_max_size: batch_max_size as u64,
batch_flush_interval_ms: flush_interval.as_millis() as u64,
heartbeat_enabled,
heartbeat_interval_ms: heartbeat_interval.as_millis() as u64,
};
match grpc::check_in(&mut control_client, CheckInRequest {
host: host.clone(),
service: service.clone(),
current_config: Some(reported),
applied_override_version: applied_override_version.clone(),
}).await {
Ok(resp) => {
if let Some(ov) = resp.has_override.then_some(resp.r#override).flatten() {
if ov.version != applied_override_version {
apply_override(
&ov,
&mut batch_max_size, &mut flush_interval,
&mut heartbeat_enabled, &mut heartbeat_interval,
&mut batcher, &mut ticker, &mut heartbeat_ticker,
&mut source_cfg, &mut source_handle, &mut rx,
&mut client,
).await;
applied_override_version = ov.version.clone();
tracing::info!(version = %applied_override_version, "applied remote config override");
}
}
}
// A failed check-in is not fatal -- same graceful-
// degradation posture as a failed heartbeat/batch
// flush: an agent management feature being
// unreachable must never stop log collection.
Err(e) => tracing::debug!(error = %e, "check-in failed"),
}
}
maybe_line = rx.recv() => {
let Some(raw) = maybe_line else {
@@ -148,13 +216,113 @@ pub async fn run_agent(config_path: Option<PathBuf>) -> Result<()> {
}
}
if let Some(batch) = batcher.poll_timeout() {
// flush_all(), not poll_timeout(): shutdown must send whatever's
// buffered unconditionally -- poll_timeout() only drains once
// flush_interval has elapsed, so anything buffered more recently
// than that would otherwise be silently dropped on every graceful
// shutdown that happens to land between flushes. Same reasoning
// applies to a config hot-reload replacing this batcher outright
// (see apply_override).
if let Some(batch) = batcher.flush_all() {
flush(&mut client, batch).await;
}
source_handle.abort();
Ok(())
}
/// Applies a newly-received DesiredOverride to the agent's in-memory
/// runtime state -- see run_agent's CheckIn arm, and
/// /docs/agent-management-design.md's merge-semantics section for why
/// this never touches the local agent.toml file. Every field is
/// independently optional (unset = keep the current value); batch/
/// heartbeat settings always get their batcher/ticker rebuilt together
/// when *any* override arrives, for simplicity, rather than tracking
/// which specific field changed -- this only runs when a human edits an
/// agent's config from the web UI, not on a hot path, so the extra
/// timer/allocation churn doesn't matter.
#[allow(clippy::too_many_arguments)]
async fn apply_override(
ov: &DesiredOverride,
batch_max_size: &mut usize,
flush_interval: &mut Duration,
heartbeat_enabled: &mut bool,
heartbeat_interval: &mut Duration,
batcher: &mut Batcher,
ticker: &mut tokio::time::Interval,
heartbeat_ticker: &mut tokio::time::Interval,
source_cfg: &mut config::SourceConfig,
source_handle: &mut tokio::task::JoinHandle<()>,
rx: &mut mpsc::Receiver<source::RawLine>,
client: &mut LogIngestClient<Channel>,
) {
if let Some(v) = ov.batch_max_size {
*batch_max_size = v as usize;
}
if let Some(v) = ov.batch_flush_interval_ms {
*flush_interval = Duration::from_millis(v);
}
// Flush whatever the old batcher was holding before replacing it --
// a hot-reload must never silently drop buffered-but-not-yet-due
// records, same reasoning as shutdown's flush_all() above.
if let Some(old) = batcher.flush_all() {
flush(client, old).await;
}
*batcher = Batcher::new(*batch_max_size, *flush_interval);
*ticker = tokio::time::interval((*flush_interval).max(Duration::from_millis(50)));
if let Some(v) = ov.heartbeat_enabled {
*heartbeat_enabled = v;
}
if let Some(v) = ov.heartbeat_interval_ms {
*heartbeat_interval = Duration::from_millis(v);
}
*heartbeat_ticker = tokio::time::interval((*heartbeat_interval).max(Duration::from_millis(50)));
// Only meaningful (and only ever sent by the server) when this
// agent's local source is journald -- ignored otherwise, per
// agent_control.proto's DesiredOverride.journald_unit comment.
// Changing it means aborting and respawning the source task: unlike
// batch/heartbeat, there's no way to change what journald::run is
// tailing without restarting that task.
if let Some(unit) = &ov.journald_unit {
if let config::SourceConfig::Journald { unit: current_unit } = source_cfg {
let new_unit = if unit.is_empty() { None } else { Some(unit.clone()) };
if *current_unit != new_unit {
*source_cfg = config::SourceConfig::Journald { unit: new_unit };
source_handle.abort();
let (new_handle, new_rx) = spawn_source_task(source_cfg.clone());
*source_handle = new_handle;
*rx = new_rx;
tracing::info!(unit = ?unit, "applied remote journald unit override, restarted source");
}
}
}
}
fn spawn_source_task(source_cfg: config::SourceConfig) -> (tokio::task::JoinHandle<()>, mpsc::Receiver<source::RawLine>) {
let (tx, rx) = mpsc::channel(1024);
let handle = tokio::spawn(spawn_source(source_cfg, tx));
(handle, rx)
}
fn source_kind_name(cfg: &config::SourceConfig) -> String {
match cfg {
config::SourceConfig::Journald { .. } => "journald".to_string(),
config::SourceConfig::File { .. } => "file".to_string(),
config::SourceConfig::EventLog { .. } => "eventlog".to_string(),
config::SourceConfig::Etw { .. } => "etw".to_string(),
}
}
fn source_detail_summary(cfg: &config::SourceConfig) -> String {
match cfg {
config::SourceConfig::Journald { unit } => unit.clone().unwrap_or_else(|| "(whole journal)".to_string()),
config::SourceConfig::File { path, .. } => path.display().to_string(),
config::SourceConfig::EventLog { channels } => channels.join(","),
config::SourceConfig::Etw { providers } => providers.join(","),
}
}
// `tx` genuinely goes unused in one rare-but-valid combination: Windows
// features enabled while targeting a non-Windows platform (e.g. sanity-
// checking the Windows source arms compile shape from Linux, which is