jcoffey-dev
cd8aa290ca
Extends the agent, ingest, storage, api, and web with Windows Event Log/ETW sourcing and Tantivy-backed free-text search, per the approved Phase 1 plan. - CLAUDE.md: materialized on disk (never existed as a file before) with a new Phase 1 "done looks like" section. - agent: Windows Event Log (EvtSubscribe) and ETW sources, Windows service wrapper (install/uninstall/run-service), both feature- and target_os-gated so Linux builds/tests/clippy stay unaffected. Also fixed two pre-existing Phase 0 clippy gaps (dead-code on default-features-only builds, a type-inference edge case) found while testing every feature combination properly for the first time. UNVERIFIED on real Windows -- no Windows toolchain existed anywhere in the build environment; flagged prominently in three places. - proto/ingest: new record_id field, assigned once server-side in ingest's gRPC front end so ClickHouse and Tantivy agree on the same ID for the same record. - storage: record_id column + bloom filter index, verified against a live ClickHouse. - search: new service, Tantivy index, rskafka consumer as an independent second consumer group on the same Redpanda topic ingest already reads. - api/web: new /search endpoint and page, sharing the query page's result-table shape and component. - hack/windows-fixture: sends realistic Windows-shaped data straight to ingest, so the pipeline's handling of it is verifiable without a Windows host. Verified end-to-end on the live docker-compose stack: the same record_id comes back from both /query and /search for the same log line, including for windows-fixture's synthetic Windows Event Log data. Real bugs found and fixed along the way: api/Dockerfile missing proto/ in its build context, search's logs being completely silent (RUST_LOG gap), and search/target/ missing from .gitignore/.dockerignore.
98 lines
4.6 KiB
Markdown
98 lines
4.6 KiB
Markdown
# ingest
|
|
|
|
Go service sitting between the Rust agent and ClickHouse. Two halves in one
|
|
binary, selected with `--mode`:
|
|
|
|
- **server** — mTLS gRPC front end (`LogIngest.PushBatch`) that agents
|
|
connect to. Assigns each record a server-side `record_id` (a UUID,
|
|
overwriting whatever the agent sent — agents always send it empty) and
|
|
otherwise forwards records proto-encoded onto Redpanda unchanged. Still
|
|
kept thin — one field assignment, no real normalization — so agent-
|
|
facing latency isn't coupled to ClickHouse write performance.
|
|
`record_id` has to be assigned exactly once, here, rather than
|
|
independently by each downstream consumer: Phase 1's Tantivy indexer
|
|
and the ClickHouse writer both read the same Redpanda messages and need
|
|
to agree on the same ID for the same record to join search hits back to
|
|
rows — two consumers generating their own IDs would produce mismatched
|
|
ones for what's supposed to be the same record.
|
|
- **consumer** — reads back off Redpanda, normalizes into the ClickHouse row
|
|
shape (`internal/normalize`), and batch-writes via the native protocol
|
|
driver. Commits Redpanda offsets only after a successful ClickHouse
|
|
write, so a ClickHouse outage causes redelivery on restart rather than
|
|
data loss.
|
|
- **all** (default) — both, in one process. This is what docker-compose
|
|
runs. Splitting into two deployments later (e.g. to scale them
|
|
independently in k8s) is a manifest change, not a code change — see
|
|
`--mode`.
|
|
|
|
## Why Redpanda stays in the path
|
|
|
|
Confirmed with the project owner during Phase 0 planning: the gRPC front
|
|
end produces to Redpanda rather than writing ClickHouse directly. This
|
|
exercises the pinned transport layer from day one and keeps agents from
|
|
ever needing Kafka credentials — mTLS to `ingest` is the only network
|
|
egress an agent has. See `/docs/architecture.md`.
|
|
|
|
## Dependencies worth knowing about
|
|
|
|
- **github.com/segmentio/kafka-go** — pure Go, no cgo, chosen over
|
|
franz-go/confluent-kafka-go specifically to keep the distroless build
|
|
simple (confirmed with the project owner; see git history / PR
|
|
discussion for the tradeoffs considered).
|
|
- **github.com/ClickHouse/clickhouse-go/v2** — official client, native
|
|
protocol, pure Go (no cgo).
|
|
- **golang.org/x/sync/errgroup** — used in `cmd/ingest/main.go` to run the
|
|
server and consumer halves concurrently and propagate the first error.
|
|
- **github.com/google/uuid** — was already in the dependency graph
|
|
transitively (via clickhouse-go); promoted to a direct dependency for
|
|
`record_id` generation in `internal/grpcserver`, so not a new addition
|
|
to the transitive tree.
|
|
|
|
## Configuration
|
|
|
|
All via environment variables (see `internal/config/config.go` for the
|
|
full list and defaults) — no config file format for Phase 0:
|
|
|
|
| Var | Default | Purpose |
|
|
|---|---|---|
|
|
| `GRPC_LISTEN_ADDR` | `:4317` | Agent-facing gRPC listen address |
|
|
| `TLS_CERT_FILE` / `TLS_KEY_FILE` | `/etc/sentry-ingest/server{,-key}.pem` | ingest's own mTLS identity |
|
|
| `TLS_CLIENT_CA_FILE` | `/etc/sentry-ingest/ca.pem` | CA used to verify agent client certs |
|
|
| `REDPANDA_BROKERS` | `localhost:9092` | Comma-separated broker list |
|
|
| `REDPANDA_TOPIC` | `sentry.logs.raw` | Must match the topic provisioned in `/transport` |
|
|
| `REDPANDA_CONSUMER_GROUP` | `sentry-ingest` | Consumer group id |
|
|
| `CLICKHOUSE_ADDR` | `localhost:9000` | Native protocol port, not HTTP |
|
|
| `CLICKHOUSE_DATABASE` / `_USERNAME` / `_PASSWORD` | `sentry` / `default` / `` | |
|
|
| `CONSUMER_BATCH_MAX_SIZE` | `500` | Records per ClickHouse batch insert |
|
|
| `CONSUMER_BATCH_FLUSH_INTERVAL_MS` | `2000` | Max time a partial batch waits before flushing |
|
|
|
|
## Building & testing
|
|
|
|
```sh
|
|
go build ./...
|
|
go vet ./...
|
|
go test ./...
|
|
```
|
|
|
|
Requires `google.golang.org/protobuf/cmd/protoc-gen-go` and
|
|
`google.golang.org/grpc/cmd/protoc-gen-go-grpc` only if you're
|
|
regenerating `/proto`'s Go bindings — ingest itself just imports the
|
|
already-generated `github.com/sentry/sentry/proto` module (see the
|
|
`replace` directive in `go.mod`, pointing at `../proto`).
|
|
|
|
```sh
|
|
# from the repo root, not ingest/
|
|
docker build -f ingest/Dockerfile -t sentry-ingest .
|
|
```
|
|
|
|
## Testing notes
|
|
|
|
`internal/consumer` and `internal/grpcserver` depend on Redpanda and
|
|
ClickHouse only through small interfaces (`reader`/`chWriter` in consumer,
|
|
`batchProducer` in grpcserver), so the flush/commit/error-handling logic is
|
|
unit-tested against fakes — no embedded broker or database needed. What's
|
|
*not* covered by these tests: the real `kafka.Reader`/`kafka.Writer`
|
|
wiring and the ClickHouse native-protocol driver itself. Those are only
|
|
exercised by the docker-compose end-to-end flow described in
|
|
`/docs/phase-0-runbook.md`.
|