Phase 1: Windows log collection + full-text search
Extends the agent, ingest, storage, api, and web with Windows Event Log/ETW sourcing and Tantivy-backed free-text search, per the approved Phase 1 plan. - CLAUDE.md: materialized on disk (never existed as a file before) with a new Phase 1 "done looks like" section. - agent: Windows Event Log (EvtSubscribe) and ETW sources, Windows service wrapper (install/uninstall/run-service), both feature- and target_os-gated so Linux builds/tests/clippy stay unaffected. Also fixed two pre-existing Phase 0 clippy gaps (dead-code on default-features-only builds, a type-inference edge case) found while testing every feature combination properly for the first time. UNVERIFIED on real Windows -- no Windows toolchain existed anywhere in the build environment; flagged prominently in three places. - proto/ingest: new record_id field, assigned once server-side in ingest's gRPC front end so ClickHouse and Tantivy agree on the same ID for the same record. - storage: record_id column + bloom filter index, verified against a live ClickHouse. - search: new service, Tantivy index, rskafka consumer as an independent second consumer group on the same Redpanda topic ingest already reads. - api/web: new /search endpoint and page, sharing the query page's result-table shape and component. - hack/windows-fixture: sends realistic Windows-shaped data straight to ingest, so the pipeline's handling of it is verifiable without a Windows host. Verified end-to-end on the live docker-compose stack: the same record_id comes back from both /query and /search for the same log line, including for windows-fixture's synthetic Windows Event Log data. Real bugs found and fixed along the way: api/Dockerfile missing proto/ in its build context, search's logs being completely silent (RUST_LOG gap), and search/target/ missing from .gitignore/.dockerignore.
This commit is contained in:
+41
-19
@@ -1,17 +1,19 @@
|
||||
# api
|
||||
|
||||
Sentry's Phase 0 query API: one crude, intentionally placeholder endpoint.
|
||||
Sentry's query API: two intentionally crude endpoints — raw SQL and
|
||||
free-text search — that Phase 2's real query layer replaces outright.
|
||||
|
||||
## Why plain REST, not gRPC + REST gateway
|
||||
|
||||
CLAUDE.md pins the control plane to "Go, gRPC + REST gateway." This
|
||||
service is plain `net/http` instead — a deliberate Phase 0 simplification,
|
||||
not a change to the pinned stack. Wiring up a `.proto` service,
|
||||
`google.api.http` annotations, and `protoc-gen-grpc-gateway` codegen for a
|
||||
single endpoint that Phase 2 replaces outright with a real SPL-like query
|
||||
service is plain `net/http` instead — a deliberate simplification, not a
|
||||
change to the pinned stack. Wiring up `.proto` services,
|
||||
`google.api.http` annotations, and `protoc-gen-grpc-gateway` codegen for
|
||||
two endpoints that Phase 2 replaces outright with a real SPL-like query
|
||||
layer would be exactly the kind of premature machinery this project's
|
||||
conventions warn against. Adopt the gRPC+gateway pattern once `/api` grows
|
||||
a second real, durable endpoint.
|
||||
conventions warn against. `api` *does* speak gRPC internally though — to
|
||||
`/search` (see below) — this simplification is specifically about the
|
||||
public-facing surface, not a blanket avoidance of gRPC.
|
||||
|
||||
## Endpoints
|
||||
|
||||
@@ -20,10 +22,21 @@ a second real, durable endpoint.
|
||||
SELECT-only, single-statement, basic keyword-based injection guarding
|
||||
(see `internal/queryapi/validate.go` for exactly what that does and
|
||||
doesn't catch — it's not a SQL parser).
|
||||
- `POST /search` — body `{"query": "...", "limit": 100}`, same response
|
||||
shape as `/query`. Calls `/search`'s `SearchService.Search` gRPC RPC to
|
||||
resolve the free-text query into matching `record_id`s, then joins
|
||||
those back against ClickHouse (`SELECT * FROM logs WHERE record_id IN
|
||||
(...)`) to return full rows — so both endpoints return the same
|
||||
`{columns, rows}` shape and `/web` can reuse one table component for
|
||||
both. Every `record_id` is validated as a real UUID before being
|
||||
embedded in the generated SQL (defense in depth: `record_id`s come from
|
||||
an internal, trusted service, not raw user input, but a value that
|
||||
fails to parse as a UUID can't contain SQL-breaking characters either
|
||||
way).
|
||||
- `GET /healthz` — for docker-compose/k8s liveness checks.
|
||||
|
||||
No auth. Not scoped for Phase 0 — don't expose this beyond a trusted
|
||||
dev/homelab network.
|
||||
No auth. Not scoped yet — don't expose this beyond a trusted dev/homelab
|
||||
network.
|
||||
|
||||
## Configuration
|
||||
|
||||
@@ -34,9 +47,15 @@ Environment variables (see `internal/config/config.go`):
|
||||
| `HTTP_LISTEN_ADDR` | `:8080` | |
|
||||
| `CLICKHOUSE_ADDR` | `localhost:9000` | Native protocol port |
|
||||
| `CLICKHOUSE_DATABASE` / `_USERNAME` / `_PASSWORD` | `sentry` / `default` / `` | |
|
||||
| `QUERY_TIMEOUT_SECONDS` | `30` | Per-request ClickHouse query timeout |
|
||||
| `SEARCH_GRPC_ADDR` | `localhost:50052` | Must match `/search`'s `GRPC_LISTEN_ADDR` |
|
||||
| `QUERY_TIMEOUT_SECONDS` | `30` | Per-request timeout, both endpoints |
|
||||
| `CORS_ALLOWED_ORIGIN` | `*` | Wide open by default since there's no auth yet; tighten together |
|
||||
|
||||
`searchclient.Dial` connects to `/search` over plain TCP, no TLS — same
|
||||
trust boundary as `api`'s existing plain-TCP connection to ClickHouse.
|
||||
mTLS in this project is specifically the agent↔ingest edge boundary, not
|
||||
every internal hop.
|
||||
|
||||
## Building & testing
|
||||
|
||||
```sh
|
||||
@@ -52,12 +71,15 @@ docker build -f api/Dockerfile -t sentry-api .
|
||||
|
||||
## Testing notes
|
||||
|
||||
`internal/queryapi`'s HTTP handler depends on ClickHouse only through a
|
||||
one-method `queryExecutor` interface, so routing, validation, JSON
|
||||
encoding, and error-status mapping are all unit-tested against a fake —
|
||||
no live ClickHouse needed. `Executor` itself (the reflection-based row
|
||||
scanning against `driver.Rows`) is not unit-tested — faking ClickHouse's
|
||||
`driver.Rows` interface fully would be significant test-only scaffolding
|
||||
for a Phase 0 placeholder, and the driver package's own docs note it isn't
|
||||
meant to be implemented by adopters. It's exercised end-to-end via the
|
||||
docker-compose flow in `/docs/phase-0-runbook.md` instead.
|
||||
`internal/queryapi`'s HTTP handlers depend on ClickHouse and `/search`
|
||||
only through narrow interfaces (`queryExecutor`, `searchClient`), so
|
||||
routing, validation, JSON encoding, error-status mapping, and the
|
||||
record_id-to-SQL query building are all unit-tested against fakes — no
|
||||
live ClickHouse or `/search` instance needed. `Executor` itself (the
|
||||
reflection-based row scanning against `driver.Rows`) and
|
||||
`internal/searchclient`'s actual gRPC dial are not unit-tested — the
|
||||
former because faking ClickHouse's `driver.Rows` interface fully would be
|
||||
significant test-only scaffolding the driver's own docs say isn't meant
|
||||
to be implemented by adopters; the latter because it's a thin wrapper
|
||||
with nothing but wiring to test. Both are exercised end-to-end via the
|
||||
docker-compose flow in `/docs/phase-1-runbook.md` instead.
|
||||
|
||||
Reference in New Issue
Block a user