jcoffey-dev
cd8aa290ca
Extends the agent, ingest, storage, api, and web with Windows Event Log/ETW sourcing and Tantivy-backed free-text search, per the approved Phase 1 plan. - CLAUDE.md: materialized on disk (never existed as a file before) with a new Phase 1 "done looks like" section. - agent: Windows Event Log (EvtSubscribe) and ETW sources, Windows service wrapper (install/uninstall/run-service), both feature- and target_os-gated so Linux builds/tests/clippy stay unaffected. Also fixed two pre-existing Phase 0 clippy gaps (dead-code on default-features-only builds, a type-inference edge case) found while testing every feature combination properly for the first time. UNVERIFIED on real Windows -- no Windows toolchain existed anywhere in the build environment; flagged prominently in three places. - proto/ingest: new record_id field, assigned once server-side in ingest's gRPC front end so ClickHouse and Tantivy agree on the same ID for the same record. - storage: record_id column + bloom filter index, verified against a live ClickHouse. - search: new service, Tantivy index, rskafka consumer as an independent second consumer group on the same Redpanda topic ingest already reads. - api/web: new /search endpoint and page, sharing the query page's result-table shape and component. - hack/windows-fixture: sends realistic Windows-shaped data straight to ingest, so the pipeline's handling of it is verifiable without a Windows host. Verified end-to-end on the live docker-compose stack: the same record_id comes back from both /query and /search for the same log line, including for windows-fixture's synthetic Windows Event Log data. Real bugs found and fixed along the way: api/Dockerfile missing proto/ in its build context, search's logs being completely silent (RUST_LOG gap), and search/target/ missing from .gitignore/.dockerignore.
86 lines
3.7 KiB
Markdown
86 lines
3.7 KiB
Markdown
# api
|
|
|
|
Sentry's query API: two intentionally crude endpoints — raw SQL and
|
|
free-text search — that Phase 2's real query layer replaces outright.
|
|
|
|
## Why plain REST, not gRPC + REST gateway
|
|
|
|
CLAUDE.md pins the control plane to "Go, gRPC + REST gateway." This
|
|
service is plain `net/http` instead — a deliberate simplification, not a
|
|
change to the pinned stack. Wiring up `.proto` services,
|
|
`google.api.http` annotations, and `protoc-gen-grpc-gateway` codegen for
|
|
two endpoints that Phase 2 replaces outright with a real SPL-like query
|
|
layer would be exactly the kind of premature machinery this project's
|
|
conventions warn against. `api` *does* speak gRPC internally though — to
|
|
`/search` (see below) — this simplification is specifically about the
|
|
public-facing surface, not a blanket avoidance of gRPC.
|
|
|
|
## Endpoints
|
|
|
|
- `POST /query` — body `{"sql": "SELECT ..."}`, response
|
|
`{"columns": [...], "rows": [[...], ...]}` or `{"error": "..."}`.
|
|
SELECT-only, single-statement, basic keyword-based injection guarding
|
|
(see `internal/queryapi/validate.go` for exactly what that does and
|
|
doesn't catch — it's not a SQL parser).
|
|
- `POST /search` — body `{"query": "...", "limit": 100}`, same response
|
|
shape as `/query`. Calls `/search`'s `SearchService.Search` gRPC RPC to
|
|
resolve the free-text query into matching `record_id`s, then joins
|
|
those back against ClickHouse (`SELECT * FROM logs WHERE record_id IN
|
|
(...)`) to return full rows — so both endpoints return the same
|
|
`{columns, rows}` shape and `/web` can reuse one table component for
|
|
both. Every `record_id` is validated as a real UUID before being
|
|
embedded in the generated SQL (defense in depth: `record_id`s come from
|
|
an internal, trusted service, not raw user input, but a value that
|
|
fails to parse as a UUID can't contain SQL-breaking characters either
|
|
way).
|
|
- `GET /healthz` — for docker-compose/k8s liveness checks.
|
|
|
|
No auth. Not scoped yet — don't expose this beyond a trusted dev/homelab
|
|
network.
|
|
|
|
## Configuration
|
|
|
|
Environment variables (see `internal/config/config.go`):
|
|
|
|
| Var | Default | Purpose |
|
|
|---|---|---|
|
|
| `HTTP_LISTEN_ADDR` | `:8080` | |
|
|
| `CLICKHOUSE_ADDR` | `localhost:9000` | Native protocol port |
|
|
| `CLICKHOUSE_DATABASE` / `_USERNAME` / `_PASSWORD` | `sentry` / `default` / `` | |
|
|
| `SEARCH_GRPC_ADDR` | `localhost:50052` | Must match `/search`'s `GRPC_LISTEN_ADDR` |
|
|
| `QUERY_TIMEOUT_SECONDS` | `30` | Per-request timeout, both endpoints |
|
|
| `CORS_ALLOWED_ORIGIN` | `*` | Wide open by default since there's no auth yet; tighten together |
|
|
|
|
`searchclient.Dial` connects to `/search` over plain TCP, no TLS — same
|
|
trust boundary as `api`'s existing plain-TCP connection to ClickHouse.
|
|
mTLS in this project is specifically the agent↔ingest edge boundary, not
|
|
every internal hop.
|
|
|
|
## Building & testing
|
|
|
|
```sh
|
|
go build ./...
|
|
go vet ./...
|
|
go test ./...
|
|
```
|
|
|
|
```sh
|
|
# from the repo root, not api/
|
|
docker build -f api/Dockerfile -t sentry-api .
|
|
```
|
|
|
|
## Testing notes
|
|
|
|
`internal/queryapi`'s HTTP handlers depend on ClickHouse and `/search`
|
|
only through narrow interfaces (`queryExecutor`, `searchClient`), so
|
|
routing, validation, JSON encoding, error-status mapping, and the
|
|
record_id-to-SQL query building are all unit-tested against fakes — no
|
|
live ClickHouse or `/search` instance needed. `Executor` itself (the
|
|
reflection-based row scanning against `driver.Rows`) and
|
|
`internal/searchclient`'s actual gRPC dial are not unit-tested — the
|
|
former because faking ClickHouse's `driver.Rows` interface fully would be
|
|
significant test-only scaffolding the driver's own docs say isn't meant
|
|
to be implemented by adopters; the latter because it's a thin wrapper
|
|
with nothing but wiring to test. Both are exercised end-to-end via the
|
|
docker-compose flow in `/docs/phase-1-runbook.md` instead.
|