jcoffey-dev
cd8aa290ca
Extends the agent, ingest, storage, api, and web with Windows Event Log/ETW sourcing and Tantivy-backed free-text search, per the approved Phase 1 plan. - CLAUDE.md: materialized on disk (never existed as a file before) with a new Phase 1 "done looks like" section. - agent: Windows Event Log (EvtSubscribe) and ETW sources, Windows service wrapper (install/uninstall/run-service), both feature- and target_os-gated so Linux builds/tests/clippy stay unaffected. Also fixed two pre-existing Phase 0 clippy gaps (dead-code on default-features-only builds, a type-inference edge case) found while testing every feature combination properly for the first time. UNVERIFIED on real Windows -- no Windows toolchain existed anywhere in the build environment; flagged prominently in three places. - proto/ingest: new record_id field, assigned once server-side in ingest's gRPC front end so ClickHouse and Tantivy agree on the same ID for the same record. - storage: record_id column + bloom filter index, verified against a live ClickHouse. - search: new service, Tantivy index, rskafka consumer as an independent second consumer group on the same Redpanda topic ingest already reads. - api/web: new /search endpoint and page, sharing the query page's result-table shape and component. - hack/windows-fixture: sends realistic Windows-shaped data straight to ingest, so the pipeline's handling of it is verifiable without a Windows host. Verified end-to-end on the live docker-compose stack: the same record_id comes back from both /query and /search for the same log line, including for windows-fixture's synthetic Windows Event Log data. Real bugs found and fixed along the way: api/Dockerfile missing proto/ in its build context, search's logs being completely silent (RUST_LOG gap), and search/target/ missing from .gitignore/.dockerignore.
75 lines
2.4 KiB
Rust
75 lines
2.4 KiB
Rust
use super::{LineSender, RawLine};
|
|
use anyhow::{Context, Result};
|
|
use std::io::SeekFrom;
|
|
use std::path::Path;
|
|
use std::time::{Duration, SystemTime, UNIX_EPOCH};
|
|
use tokio::fs::File;
|
|
use tokio::io::{AsyncBufReadExt, AsyncSeekExt, BufReader};
|
|
|
|
const POLL_INTERVAL: Duration = Duration::from_millis(500);
|
|
|
|
/// Polling-based file tailer: no inotify/`notify` crate dependency. Good
|
|
/// enough for Phase 0 (journald is the primary source). Handles basic
|
|
/// truncation (e.g. logrotate `copytruncate`) by detecting the file shrank
|
|
/// and reopening from the start. Does not follow rename-based rotation
|
|
/// (logrotate `create`) — that's deferred until file-tail is more than a
|
|
/// fallback path.
|
|
pub async fn run(path: &Path, from_beginning: bool, tx: LineSender) -> Result<()> {
|
|
let file = File::open(path)
|
|
.await
|
|
.with_context(|| format!("opening {}", path.display()))?;
|
|
|
|
let mut pos = if from_beginning { 0 } else { file.metadata().await?.len() };
|
|
|
|
let mut reader = BufReader::new(file);
|
|
reader.seek(SeekFrom::Start(pos)).await?;
|
|
let mut buf = String::new();
|
|
|
|
loop {
|
|
buf.clear();
|
|
let n = reader
|
|
.read_line(&mut buf)
|
|
.await
|
|
.context("reading line from file")?;
|
|
if n == 0 {
|
|
let metadata = tokio::fs::metadata(path).await.context("stat-ing file")?;
|
|
if metadata.len() < pos {
|
|
tracing::warn!(path = %path.display(), "file shrank, assuming truncation and reopening from start");
|
|
let f = File::open(path)
|
|
.await
|
|
.context("reopening file after truncation")?;
|
|
reader = BufReader::new(f);
|
|
pos = 0;
|
|
}
|
|
tokio::time::sleep(POLL_INTERVAL).await;
|
|
continue;
|
|
}
|
|
pos += n as u64;
|
|
|
|
let line = buf.trim_end_matches(['\n', '\r']).to_string();
|
|
if line.is_empty() {
|
|
continue;
|
|
}
|
|
|
|
let timestamp_unix_nano = SystemTime::now()
|
|
.duration_since(UNIX_EPOCH)
|
|
.map(|d| d.as_nanos() as i64)
|
|
.unwrap_or(0);
|
|
|
|
if tx
|
|
.send(RawLine {
|
|
line,
|
|
timestamp_unix_nano,
|
|
severity_hint: None,
|
|
extra_attributes: Default::default(),
|
|
})
|
|
.await
|
|
.is_err()
|
|
{
|
|
break;
|
|
}
|
|
}
|
|
|
|
Ok(())
|
|
}
|