Scaffold Phase 0: agent -> Redpanda -> ingest -> ClickHouse -> api -> web
End-to-end log pipeline for Linux hosts, per /docs/architecture.md: - proto: shared gRPC contract (agent <-> ingest), Go bindings checked in - agent: Rust, musl-targeted, journald/file sourcing, RFC5424 parser, mTLS gRPC client, no required config for the common case - ingest: Go, single binary with --mode server|consumer|all; gRPC front end forwards to Redpanda unchanged, consumer normalizes and batch-writes to ClickHouse with at-least-once delivery - storage: ClickHouse schema + a plain SQL-file migration runner - api: minimal SELECT-only query endpoint, plain REST (not gRPC+gateway yet -- see api/README.md) - web: SvelteKit static SPA, one query page - transport: Redpanda compose + topic provisioning - cli: sentryctl ping stub - hack/dev-certs: throwaway CA + cert generation for local mTLS - root docker-compose.yml + docs/phase-0-runbook.md tie it together Not yet run end-to-end against real Docker/ClickHouse/Redpanda -- see the runbook's caveats section before relying on this working as-is.
This commit is contained in:
@@ -0,0 +1,85 @@
|
||||
# ingest
|
||||
|
||||
Go service sitting between the Rust agent and ClickHouse. Two halves in one
|
||||
binary, selected with `--mode`:
|
||||
|
||||
- **server** — mTLS gRPC front end (`LogIngest.PushBatch`) that agents
|
||||
connect to. Forwards each record, proto-encoded and unchanged, onto
|
||||
Redpanda. Does no normalization — kept thin so agent-facing latency isn't
|
||||
coupled to ClickHouse write performance.
|
||||
- **consumer** — reads back off Redpanda, normalizes into the ClickHouse row
|
||||
shape (`internal/normalize`), and batch-writes via the native protocol
|
||||
driver. Commits Redpanda offsets only after a successful ClickHouse
|
||||
write, so a ClickHouse outage causes redelivery on restart rather than
|
||||
data loss.
|
||||
- **all** (default) — both, in one process. This is what docker-compose
|
||||
runs. Splitting into two deployments later (e.g. to scale them
|
||||
independently in k8s) is a manifest change, not a code change — see
|
||||
`--mode`.
|
||||
|
||||
## Why Redpanda stays in the path
|
||||
|
||||
Confirmed with the project owner during Phase 0 planning: the gRPC front
|
||||
end produces to Redpanda rather than writing ClickHouse directly. This
|
||||
exercises the pinned transport layer from day one and keeps agents from
|
||||
ever needing Kafka credentials — mTLS to `ingest` is the only network
|
||||
egress an agent has. See `/docs/architecture.md`.
|
||||
|
||||
## Dependencies worth knowing about
|
||||
|
||||
- **github.com/segmentio/kafka-go** — pure Go, no cgo, chosen over
|
||||
franz-go/confluent-kafka-go specifically to keep the distroless build
|
||||
simple (confirmed with the project owner; see git history / PR
|
||||
discussion for the tradeoffs considered).
|
||||
- **github.com/ClickHouse/clickhouse-go/v2** — official client, native
|
||||
protocol, pure Go (no cgo).
|
||||
- **golang.org/x/sync/errgroup** — used in `cmd/ingest/main.go` to run the
|
||||
server and consumer halves concurrently and propagate the first error.
|
||||
|
||||
## Configuration
|
||||
|
||||
All via environment variables (see `internal/config/config.go` for the
|
||||
full list and defaults) — no config file format for Phase 0:
|
||||
|
||||
| Var | Default | Purpose |
|
||||
|---|---|---|
|
||||
| `GRPC_LISTEN_ADDR` | `:4317` | Agent-facing gRPC listen address |
|
||||
| `TLS_CERT_FILE` / `TLS_KEY_FILE` | `/etc/sentry-ingest/server{,-key}.pem` | ingest's own mTLS identity |
|
||||
| `TLS_CLIENT_CA_FILE` | `/etc/sentry-ingest/ca.pem` | CA used to verify agent client certs |
|
||||
| `REDPANDA_BROKERS` | `localhost:9092` | Comma-separated broker list |
|
||||
| `REDPANDA_TOPIC` | `sentry.logs.raw` | Must match the topic provisioned in `/transport` |
|
||||
| `REDPANDA_CONSUMER_GROUP` | `sentry-ingest` | Consumer group id |
|
||||
| `CLICKHOUSE_ADDR` | `localhost:9000` | Native protocol port, not HTTP |
|
||||
| `CLICKHOUSE_DATABASE` / `_USERNAME` / `_PASSWORD` | `sentry` / `default` / `` | |
|
||||
| `CONSUMER_BATCH_MAX_SIZE` | `500` | Records per ClickHouse batch insert |
|
||||
| `CONSUMER_BATCH_FLUSH_INTERVAL_MS` | `2000` | Max time a partial batch waits before flushing |
|
||||
|
||||
## Building & testing
|
||||
|
||||
```sh
|
||||
go build ./...
|
||||
go vet ./...
|
||||
go test ./...
|
||||
```
|
||||
|
||||
Requires `google.golang.org/protobuf/cmd/protoc-gen-go` and
|
||||
`google.golang.org/grpc/cmd/protoc-gen-go-grpc` only if you're
|
||||
regenerating `/proto`'s Go bindings — ingest itself just imports the
|
||||
already-generated `github.com/sentry/sentry/proto` module (see the
|
||||
`replace` directive in `go.mod`, pointing at `../proto`).
|
||||
|
||||
```sh
|
||||
# from the repo root, not ingest/
|
||||
docker build -f ingest/Dockerfile -t sentry-ingest .
|
||||
```
|
||||
|
||||
## Testing notes
|
||||
|
||||
`internal/consumer` and `internal/grpcserver` depend on Redpanda and
|
||||
ClickHouse only through small interfaces (`reader`/`chWriter` in consumer,
|
||||
`batchProducer` in grpcserver), so the flush/commit/error-handling logic is
|
||||
unit-tested against fakes — no embedded broker or database needed. What's
|
||||
*not* covered by these tests: the real `kafka.Reader`/`kafka.Writer`
|
||||
wiring and the ClickHouse native-protocol driver itself. Those are only
|
||||
exercised by the docker-compose end-to-end flow described in
|
||||
`/docs/phase-0-runbook.md`.
|
||||
Reference in New Issue
Block a user