Files

T

jcoffey-dev cd8aa290ca Phase 1: Windows log collection + full-text search

Extends the agent, ingest, storage, api, and web with Windows Event
Log/ETW sourcing and Tantivy-backed free-text search, per the approved
Phase 1 plan.

- CLAUDE.md: materialized on disk (never existed as a file before) with
  a new Phase 1 "done looks like" section.
- agent: Windows Event Log (EvtSubscribe) and ETW sources, Windows
  service wrapper (install/uninstall/run-service), both feature- and
  target_os-gated so Linux builds/tests/clippy stay unaffected. Also
  fixed two pre-existing Phase 0 clippy gaps (dead-code on
  default-features-only builds, a type-inference edge case) found while
  testing every feature combination properly for the first time.
  UNVERIFIED on real Windows -- no Windows toolchain existed anywhere in
  the build environment; flagged prominently in three places.
- proto/ingest: new record_id field, assigned once server-side in
  ingest's gRPC front end so ClickHouse and Tantivy agree on the same ID
  for the same record.
- storage: record_id column + bloom filter index, verified against a
  live ClickHouse.
- search: new service, Tantivy index, rskafka consumer as an independent
  second consumer group on the same Redpanda topic ingest already reads.
- api/web: new /search endpoint and page, sharing the query page's
  result-table shape and component.
- hack/windows-fixture: sends realistic Windows-shaped data straight to
  ingest, so the pipeline's handling of it is verifiable without a
  Windows host.

Verified end-to-end on the live docker-compose stack: the same record_id
comes back from both /query and /search for the same log line, including
for windows-fixture's synthetic Windows Event Log data. Real bugs found
and fixed along the way: api/Dockerfile missing proto/ in its build
context, search's logs being completely silent (RUST_LOG gap), and
search/target/ missing from .gitignore/.dockerignore.

2026-08-13 11:27:35 -07:00

3.7 KiB

Raw Blame History

api

Sentry's query API: two intentionally crude endpoints — raw SQL and free-text search — that Phase 2's real query layer replaces outright.

Why plain REST, not gRPC + REST gateway

CLAUDE.md pins the control plane to "Go, gRPC + REST gateway." This service is plain net/http instead — a deliberate simplification, not a change to the pinned stack. Wiring up .proto services, google.api.http annotations, and protoc-gen-grpc-gateway codegen for two endpoints that Phase 2 replaces outright with a real SPL-like query layer would be exactly the kind of premature machinery this project's conventions warn against. api does speak gRPC internally though — to /search (see below) — this simplification is specifically about the public-facing surface, not a blanket avoidance of gRPC.

Endpoints

POST /query — body {"sql": "SELECT ..."}, response {"columns": [...], "rows": [[...], ...]} or {"error": "..."}. SELECT-only, single-statement, basic keyword-based injection guarding (see internal/queryapi/validate.go for exactly what that does and doesn't catch — it's not a SQL parser).
POST /search — body {"query": "...", "limit": 100}, same response shape as /query. Calls /search's SearchService.Search gRPC RPC to resolve the free-text query into matching record_ids, then joins those back against ClickHouse (SELECT * FROM logs WHERE record_id IN (...)) to return full rows — so both endpoints return the same {columns, rows} shape and /web can reuse one table component for both. Every record_id is validated as a real UUID before being embedded in the generated SQL (defense in depth: record_ids come from an internal, trusted service, not raw user input, but a value that fails to parse as a UUID can't contain SQL-breaking characters either way).
GET /healthz — for docker-compose/k8s liveness checks.

No auth. Not scoped yet — don't expose this beyond a trusted dev/homelab network.

Configuration

Environment variables (see internal/config/config.go):

Var	Default	Purpose
`HTTP_LISTEN_ADDR`	`:8080`
`CLICKHOUSE_ADDR`	`localhost:9000`	Native protocol port
`CLICKHOUSE_DATABASE` / `_USERNAME` / `_PASSWORD`	`sentry` / `default` / ``
`SEARCH_GRPC_ADDR`	`localhost:50052`	Must match `/search`'s `GRPC_LISTEN_ADDR`
`QUERY_TIMEOUT_SECONDS`	`30`	Per-request timeout, both endpoints
`CORS_ALLOWED_ORIGIN`	`*`	Wide open by default since there's no auth yet; tighten together

searchclient.Dial connects to /search over plain TCP, no TLS — same trust boundary as api's existing plain-TCP connection to ClickHouse. mTLS in this project is specifically the agent↔ingest edge boundary, not every internal hop.

Building & testing

go build ./...
go vet ./...
go test ./...

# from the repo root, not api/
docker build -f api/Dockerfile -t sentry-api .

Testing notes

internal/queryapi's HTTP handlers depend on ClickHouse and /search only through narrow interfaces (queryExecutor, searchClient), so routing, validation, JSON encoding, error-status mapping, and the record_id-to-SQL query building are all unit-tested against fakes — no live ClickHouse or /search instance needed. Executor itself (the reflection-based row scanning against driver.Rows) and internal/searchclient's actual gRPC dial are not unit-tested — the former because faking ClickHouse's driver.Rows interface fully would be significant test-only scaffolding the driver's own docs say isn't meant to be implemented by adopters; the latter because it's a thin wrapper with nothing but wiring to test. Both are exercised end-to-end via the docker-compose flow in /docs/phase-1-runbook.md instead.

3.7 KiB Raw Blame History

api