Phase 1: Windows log collection + full-text search
Extends the agent, ingest, storage, api, and web with Windows Event Log/ETW sourcing and Tantivy-backed free-text search, per the approved Phase 1 plan. - CLAUDE.md: materialized on disk (never existed as a file before) with a new Phase 1 "done looks like" section. - agent: Windows Event Log (EvtSubscribe) and ETW sources, Windows service wrapper (install/uninstall/run-service), both feature- and target_os-gated so Linux builds/tests/clippy stay unaffected. Also fixed two pre-existing Phase 0 clippy gaps (dead-code on default-features-only builds, a type-inference edge case) found while testing every feature combination properly for the first time. UNVERIFIED on real Windows -- no Windows toolchain existed anywhere in the build environment; flagged prominently in three places. - proto/ingest: new record_id field, assigned once server-side in ingest's gRPC front end so ClickHouse and Tantivy agree on the same ID for the same record. - storage: record_id column + bloom filter index, verified against a live ClickHouse. - search: new service, Tantivy index, rskafka consumer as an independent second consumer group on the same Redpanda topic ingest already reads. - api/web: new /search endpoint and page, sharing the query page's result-table shape and component. - hack/windows-fixture: sends realistic Windows-shaped data straight to ingest, so the pipeline's handling of it is verifiable without a Windows host. Verified end-to-end on the live docker-compose stack: the same record_id comes back from both /query and /search for the same log line, including for windows-fixture's synthetic Windows Event Log data. Real bugs found and fixed along the way: api/Dockerfile missing proto/ in its build context, search's logs being completely silent (RUST_LOG gap), and search/target/ missing from .gitignore/.dockerignore.
This commit is contained in:
+17
-11
@@ -5,13 +5,16 @@ ingest, and ClickHouse, to a browser table. This is the actual
|
||||
"done" criterion for Phase 0 — if this doesn't work, Phase 0 isn't done,
|
||||
regardless of what any individual component's tests say.
|
||||
|
||||
**This sequence has not been run end-to-end** in the environment that
|
||||
built it (no Docker available there — see the caveats each component's
|
||||
summary already flagged). Individual pieces are unit-tested and built
|
||||
successfully in isolation; this document is the logical sequence to run
|
||||
for real, not a report that it's been run. Expect to debug something on
|
||||
first attempt, and treat the "Troubleshooting" section at the bottom as a
|
||||
starting point, not an exhaustive list.
|
||||
**Update:** this sequence has since been run for real, more than once,
|
||||
against a live Docker install — not just written and trusted. Two real
|
||||
bugs turned up doing that (ClickHouse's official image silently disabling
|
||||
network access without a password set; `rpk`'s exact flag syntax) and got
|
||||
fixed; see the git history around the "Fix two bugs found by actually
|
||||
running the Phase 0 pipeline end-to-end" commit if you want the details.
|
||||
The steps below reflect what was actually run, not just planned. The
|
||||
"Troubleshooting" section below is still worth reading first if something
|
||||
doesn't work — it's not an exhaustive list, but it does reflect real
|
||||
failures encountered, not hypothetical ones.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
@@ -113,12 +116,15 @@ Reading the system journal generally needs root (or membership in the
|
||||
by distro, root is the reliable path for this runbook):
|
||||
|
||||
```sh
|
||||
sudo ./target/release/sentry-agent
|
||||
sudo RUST_LOG=info ./target/release/sentry-agent
|
||||
```
|
||||
|
||||
Leave it running in this terminal — you should see a `connected to ingest
|
||||
service` log line. If you see a TLS or connection error instead, stop
|
||||
here and check the Troubleshooting section before continuing.
|
||||
`RUST_LOG=info` matters: `tracing_subscriber`'s default filter is
|
||||
otherwise strict enough to suppress even the startup log line, and the
|
||||
agent will look like it's silently doing nothing. Leave it running in
|
||||
this terminal — you should see a `connected to ingest service` log line.
|
||||
If you see a TLS or connection error instead, stop here and check the
|
||||
Troubleshooting section before continuing.
|
||||
|
||||
## 6. Generate a test log line
|
||||
|
||||
|
||||
@@ -0,0 +1,243 @@
|
||||
# Phase 1 runbook
|
||||
|
||||
Extends `/docs/phase-0-runbook.md` with Windows log collection and
|
||||
full-text search. Read that one first — this assumes the Phase 0 stack
|
||||
(dev certs, `docker compose up`, backend sanity check) already works;
|
||||
Phase 1 layers on top of it, doesn't replace it.
|
||||
|
||||
## What's actually been verified vs. what needs real Windows
|
||||
|
||||
Unlike Phase 0's original draft, most of this runbook reflects steps
|
||||
actually run in this session against a live stack, not just planned:
|
||||
|
||||
- **Verified for real:** the full Linux pipeline through search (agent →
|
||||
ingest's `record_id` assignment → Redpanda → both consumers →
|
||||
ClickHouse *and* Tantivy → both `/query` and `/search` → the same
|
||||
`record_id` back from both). The `windows-fixture` generator sending
|
||||
Windows-*shaped* data through the same pipeline and being correctly
|
||||
queryable both ways, including `winevt.*` attributes and severity
|
||||
mapping.
|
||||
- **Not verified, and can't be from the environment this was built in:**
|
||||
the actual Windows agent binary — `EvtSubscribe`, ETW session creation,
|
||||
Windows service registration. No Windows toolchain was available
|
||||
anywhere (confirmed: only the Linux target's std library installed, no
|
||||
rustup, no way to even `cargo check --target x86_64-pc-windows-*`).
|
||||
Part C below is the logical sequence to run on a real or virtualized
|
||||
Windows host, not a report that it's been run.
|
||||
|
||||
## Prerequisites (beyond Phase 0's)
|
||||
|
||||
- A Windows host or VM (Windows 10/11 or Windows Server) for Part C.
|
||||
- `mingw-w64` if cross-compiling the Windows build from Linux (optional —
|
||||
building natively on Windows with `rustup target add
|
||||
x86_64-pc-windows-msvc` works too and needs no extra setup on the Linux
|
||||
side).
|
||||
- Administrator access on the Windows host, for service registration and
|
||||
(if you enable it) ETW.
|
||||
|
||||
## Part A: full-text search (Linux-only, no Windows needed)
|
||||
|
||||
Only needs what Phase 0's runbook already set up.
|
||||
|
||||
### A1. Bring the stack up (if not already)
|
||||
|
||||
```sh
|
||||
docker compose up -d --build
|
||||
```
|
||||
|
||||
Same as Phase 0, now also builds and starts `search` (Tantivy full-text
|
||||
indexing). Confirm it's actually logging — same `RUST_LOG` gap the agent
|
||||
has by default:
|
||||
|
||||
```sh
|
||||
docker compose logs search
|
||||
```
|
||||
|
||||
You should see "search gRPC server listening" and rskafka connecting to
|
||||
all of `sentry.logs.raw`'s partitions. If you see nothing at all, check
|
||||
`RUST_LOG=info` is set on the `search` service in `docker-compose.yml`.
|
||||
|
||||
### A2. Generate a log line and confirm both query paths agree
|
||||
|
||||
Follow Phase 0's runbook to get the agent running and generate a test
|
||||
line (steps 4–6 there — mTLS certs, build, run, `logger`). Then, instead
|
||||
of just checking `/query`, check both:
|
||||
|
||||
```sh
|
||||
curl -X POST http://localhost:8080/query -H 'Content-Type: application/json' \
|
||||
-d '{"sql": "SELECT record_id, message FROM logs ORDER BY timestamp DESC LIMIT 1"}'
|
||||
|
||||
curl -X POST http://localhost:8080/search -H 'Content-Type: application/json' \
|
||||
-d '{"query": "<a distinctive word from your test log line>"}'
|
||||
```
|
||||
|
||||
**The `record_id` in both responses should match.** That's the actual
|
||||
Phase 1 exit criterion (`/CLAUDE.md`) for the Linux half: the same
|
||||
record, reachable both ways. If `/search` returns nothing yet, give it a
|
||||
few more seconds — Tantivy commits on a timer (`COMMIT_INTERVAL_MS`,
|
||||
default 2s), so there's a small window where a record is in ClickHouse
|
||||
but not yet searchable.
|
||||
|
||||
### A3. Confirm from the web UI
|
||||
|
||||
Open `http://localhost:3000` — there are now two pages, linked via the
|
||||
top nav: **SQL Query** (unchanged from Phase 0) and **Full-Text Search**
|
||||
(new). Run the same free-text term on the search page and confirm you
|
||||
see the row.
|
||||
|
||||
## Part B: Windows-shaped data without a Windows host
|
||||
|
||||
Still no Windows needed — this tests the pipeline's handling of
|
||||
Windows-*shaped* data, not the real Windows integration (see
|
||||
`/hack/windows-fixture/README.md` for the exact distinction).
|
||||
|
||||
```sh
|
||||
cd hack/windows-fixture
|
||||
go run . --count 5
|
||||
```
|
||||
|
||||
Then repeat A2's pattern: query for one of the synthetic events by
|
||||
`attributes['winevt.event_id']` via `/query`, and by a distinctive word
|
||||
from its message via `/search`. Both should return it, and `attributes`
|
||||
should carry `winevt.event_id`/`winevt.provider`/`winevt.channel`/
|
||||
`winevt.computer`.
|
||||
|
||||
## Part C: the real Windows agent (needs actual Windows)
|
||||
|
||||
### C1. Build
|
||||
|
||||
On the Windows host itself (simplest — avoids cross-compilation
|
||||
entirely):
|
||||
|
||||
```powershell
|
||||
rustup target add x86_64-pc-windows-msvc
|
||||
cd agent
|
||||
cargo build --release --target x86_64-pc-windows-msvc --no-default-features --features windows-eventlog,etw
|
||||
```
|
||||
|
||||
Or cross-compile from Linux, then copy the binary over:
|
||||
|
||||
```sh
|
||||
rustup target add x86_64-pc-windows-gnu
|
||||
cargo build --release --target x86_64-pc-windows-gnu --no-default-features --features windows-eventlog,etw
|
||||
```
|
||||
|
||||
`protoc` needs to be on `PATH` either way (used by `tonic-build` at
|
||||
compile time), same requirement as the Linux build.
|
||||
|
||||
### C2. Get mTLS certs onto the Windows host
|
||||
|
||||
Copy `hack/dev-certs/out/{ca,client,client-key}.pem` from wherever you
|
||||
ran `generate.sh` to `C:\ProgramData\SentryAgent\` on the Windows host
|
||||
(create the directory first). Same dev-only certs Phase 0's Linux agent
|
||||
uses — the CA doesn't care what platform the client is on, only that the
|
||||
client cert was signed by it.
|
||||
|
||||
### C3. Config
|
||||
|
||||
Create `C:\ProgramData\SentryAgent\agent.toml`:
|
||||
|
||||
```toml
|
||||
[source]
|
||||
kind = "eventlog"
|
||||
channels = ["Application", "System", "Security"]
|
||||
|
||||
[ingest]
|
||||
endpoint = "https://<host-running-docker-compose>:4317"
|
||||
```
|
||||
|
||||
If Docker Compose runs on a different machine than the Windows host,
|
||||
`ingest`'s server cert SAN needs to cover that hostname/IP too — see
|
||||
`hack/dev-certs/generate.sh` and regenerate with an updated SAN if
|
||||
needed (same note as Phase 0's runbook's troubleshooting section).
|
||||
|
||||
### C4. Run it directly first, before installing as a service
|
||||
|
||||
```powershell
|
||||
$env:RUST_LOG="info"
|
||||
.\sentry-agent.exe --config C:\ProgramData\SentryAgent\agent.toml
|
||||
```
|
||||
|
||||
Confirms the Event Log source and mTLS connection work before adding the
|
||||
Windows service layer on top — if something's wrong, it's much easier to
|
||||
diagnose here than after wrapping it in a service.
|
||||
|
||||
### C5. Generate a Windows Event Log entry and confirm it flows through
|
||||
|
||||
From another PowerShell window (or Event Viewer):
|
||||
|
||||
```powershell
|
||||
eventcreate /T INFORMATION /ID 1 /L APPLICATION /SO "SentryTest" /D "phase1 windows verification line"
|
||||
```
|
||||
|
||||
Then check both query paths, same pattern as A2.
|
||||
|
||||
### C6. Install as a service
|
||||
|
||||
```powershell
|
||||
.\sentry-agent.exe install
|
||||
sc.exe start SentryAgent
|
||||
```
|
||||
|
||||
Verify it's running (`sc.exe query SentryAgent`) and generate another
|
||||
test event to confirm it's still flowing through while running as a
|
||||
service, not just in the foreground. **Known gap:** no console under the
|
||||
SCM means `tracing`'s log output currently has nowhere to go — see
|
||||
`/agent/README.md`'s "Running as a Windows service" section. If
|
||||
something goes wrong here, you're debugging blind until that's
|
||||
addressed; C4's foreground run is where to diagnose real problems.
|
||||
|
||||
```powershell
|
||||
sc.exe stop SentryAgent
|
||||
.\sentry-agent.exe uninstall
|
||||
```
|
||||
|
||||
### C7 (optional). ETW
|
||||
|
||||
Only if you actually want it running — **read the privilege section in
|
||||
`/agent/README.md` first.** ETW needs elevated privileges (an
|
||||
administrator token or `SeSystemProfilePrivilege`), a real consideration
|
||||
for a log-shipping agent, not a formality. Providers are configured by
|
||||
GUID (`logman query providers "<Name>"` to look one up):
|
||||
|
||||
```toml
|
||||
[source]
|
||||
kind = "etw"
|
||||
providers = ["{22FB2CD6-0E7B-422B-A0C7-2FAD1FD0E716}"]
|
||||
```
|
||||
|
||||
### C8 (informational). WEF
|
||||
|
||||
No new steps — see `/agent/README.md`'s WEF section. The supported
|
||||
pattern is running this same agent (Event Log source) on a Windows
|
||||
Server already acting as a native Windows Event Collector, pointed at
|
||||
the `ForwardedEvents` channel instead of the usual three. A true
|
||||
agentless WS-Management receiver is explicitly not built in Phase 1.
|
||||
|
||||
## Troubleshooting (Phase 1-specific)
|
||||
|
||||
**`/search` returns nothing but `/query` finds the record.**
|
||||
Check `COMMIT_INTERVAL_MS` hasn't elapsed yet (default 2s) — Tantivy
|
||||
batches commits, same reasoning as ClickHouse batching inserts. If it's
|
||||
been well over that and still nothing: `docker compose logs search` —
|
||||
look for "skipping record with empty record_id" (would mean something
|
||||
upstream isn't assigning IDs — shouldn't happen) or connection errors to
|
||||
Redpanda.
|
||||
|
||||
**Windows agent connects but Event Log entries never show up.**
|
||||
Check the channel name is exactly right (`Application`/`System`/
|
||||
`Security`, case matters to the Windows API) and that the account
|
||||
running the agent has read access to that log — `Security` specifically
|
||||
often needs elevated rights beyond what `Application`/`System` need.
|
||||
|
||||
**Windows build fails to find `protoc`.**
|
||||
Same requirement as the Linux build — install `protoc` and ensure it's
|
||||
on `PATH` before `cargo build`. On Windows, the official protoc release
|
||||
zip plus adding its `bin/` to `PATH` is the simplest route.
|
||||
|
||||
**Nothing in this section covers the problem.**
|
||||
Genuinely possible — this is the least-tested part of the whole Phase 1
|
||||
build (see the caveat at the top). Check `/agent/README.md`'s Windows
|
||||
sections for the specific module involved (`source/windows_eventlog.rs`,
|
||||
`source/etw.rs`, `service.rs`) and their own "UNVERIFIED" comments for
|
||||
what's most likely to need a real fix.
|
||||
Reference in New Issue
Block a user