Add agent inventory, management, and remote config

Extends the heartbeat mechanism with a second gRPC service on the same
mTLS channel (AgentControl.CheckIn, agent-initiated on the existing
heartbeat ticker -- still push-only, no inbound port on any agent) so
an agent reports its running config and can pick up an operator-set
override. A new web UI section (/agents) lists every agent that's
checked in, shows its reported config, and lets an operator edit a
narrow, deliberately-scoped subset remotely: batch/heartbeat tuning,
and (journald sources only) the unit filter.

TLS material and the ingest endpoint are never reportable or remotely
editable, by proto shape rather than a validation rule -- a bad or
malicious edit there could permanently strand an agent or redirect
where its logs go, unlike every other editable field, which only
degrades behavior.

An override lives only in the agent's memory (agent.toml is never
rewritten) and re-syncs on the agent's own schedule; changing the
journald filter aborts and respawns the source task since there's no
other way to change what's being tailed. Building the hot-reload path
surfaced a real, independent, pre-existing bug: shutdown was using
poll_timeout(), which only drains once flush_interval has elapsed,
silently dropping anything buffered more recently on every graceful
shutdown that landed between flushes -- fixed with a new unconditional
Batcher::flush_all(), now used at both shutdown and hot-reload.

Verified live end-to-end against a real stack: an edited heartbeat
interval changed a running agent's actual send cadence within one
check-in cycle (confirmed by the real timestamps landing in
ClickHouse), and an edited journald filter triggered a real source
restart, both reflected back in the next reported-config snapshot.

See /docs/agent-management-design.md.
This commit is contained in:
2026-08-16 18:08:51 -07:00
parent 4df6931869
commit 4f0da1ae5e
29 changed files with 2618 additions and 53 deletions
+20 -8
View File
@@ -1,12 +1,18 @@
use crate::config::{IngestConfig, TlsConfig};
use crate::pb::agent::v1::{agent_control_client::AgentControlClient, CheckInRequest, CheckInResponse};
use crate::pb::{log_ingest_client::LogIngestClient, LogRecord, PushBatchRequest};
use anyhow::{Context, Result};
use tonic::transport::{Certificate, Channel, ClientTlsConfig, Identity};
/// Establishes an mTLS gRPC channel to the ingest service. Agents never
/// talk to Redpanda directly — this is the only network egress the agent
/// has, by design (see /docs/architecture.md).
pub async fn connect(ingest: &IngestConfig, tls: &TlsConfig) -> Result<LogIngestClient<Channel>> {
/// Establishes the one mTLS gRPC channel an agent has to ingest —
/// agents never talk to Redpanda directly, and never accept an inbound
/// connection either (see /docs/architecture.md and
/// /docs/agent-management-design.md). Returns the bare `Channel` rather
/// than a client wrapper so callers can build both `LogIngestClient`
/// (data plane) and `AgentControlClient` (control plane) from the same
/// connection — `Channel` is a cheap-to-clone handle, not the socket
/// itself, so there's no cost to sharing it across two client stubs.
pub async fn connect(ingest: &IngestConfig, tls: &TlsConfig) -> Result<Channel> {
let ca = tokio::fs::read(&tls.ca_cert)
.await
.with_context(|| format!("reading CA cert at {}", tls.ca_cert.display()))?;
@@ -21,15 +27,13 @@ pub async fn connect(ingest: &IngestConfig, tls: &TlsConfig) -> Result<LogIngest
.ca_certificate(Certificate::from_pem(ca))
.identity(Identity::from_pem(cert, key));
let channel = Channel::from_shared(ingest.endpoint.clone())
Channel::from_shared(ingest.endpoint.clone())
.context("invalid ingest endpoint URL")?
.tls_config(tls_config)
.context("configuring mTLS")?
.connect()
.await
.context("connecting to ingest service")?;
Ok(LogIngestClient::new(channel))
.context("connecting to ingest service")
}
pub async fn send_batch(
@@ -43,3 +47,11 @@ pub async fn send_batch(
.context("PushBatch RPC failed")?;
Ok(resp.into_inner().accepted)
}
pub async fn check_in(
client: &mut AgentControlClient<Channel>,
req: CheckInRequest,
) -> Result<CheckInResponse> {
let resp = client.check_in(req).await.context("CheckIn RPC failed")?;
Ok(resp.into_inner())
}