cairnobs/ingest/README.md

ingest

Go service sitting between the Rust agent and ClickHouse. Two halves in one binary, selected with --mode:

• server — mTLS gRPC front end (LogIngest.PushBatch) that agents connect to. Assigns each record a server-side record_id (a UUID, overwriting whatever the agent sent — agents always send it empty) and otherwise forwards records proto-encoded onto Redpanda unchanged. Still kept thin — one field assignment, no real normalization — so agent- facing latency isn't coupled to ClickHouse write performance. record_id has to be assigned exactly once, here, rather than independently by each downstream consumer: Phase 1's Tantivy indexer and the ClickHouse writer both read the same Redpanda messages and need to agree on the same ID for the same record to join search hits back to rows — two consumers generating their own IDs would produce mismatched ones for what's supposed to be the same record.
• consumer — reads back off Redpanda, normalizes into the ClickHouse row shape (internal/normalize), and batch-writes via the native protocol driver. Commits Redpanda offsets only after a successful ClickHouse write, so a ClickHouse outage causes redelivery on restart rather than data loss.
• all (default) — both, in one process. This is what docker-compose runs. Splitting into two deployments later (e.g. to scale them independently in k8s) is a manifest change, not a code change — see --mode.

Why Redpanda stays in the path

Confirmed with the project owner during Phase 0 planning: the gRPC front end produces to Redpanda rather than writing ClickHouse directly. This exercises the pinned transport layer from day one and keeps agents from ever needing Kafka credentials — mTLS to ingest is the only network egress an agent has. See /docs/architecture.md.

Dependencies worth knowing about

• github.com/segmentio/kafka-go — pure Go, no cgo, chosen over franz-go/confluent-kafka-go specifically to keep the distroless build simple (confirmed with the project owner; see git history / PR discussion for the tradeoffs considered).
• github.com/ClickHouse/clickhouse-go/v2 — official client, native protocol, pure Go (no cgo).
• golang.org/x/sync/errgroup — used in cmd/ingest/main.go to run the server and consumer halves concurrently and propagate the first error.
• github.com/google/uuid — was already in the dependency graph transitively (via clickhouse-go); promoted to a direct dependency for record_id generation in internal/grpcserver, so not a new addition to the transitive tree.

Configuration

All via environment variables (see internal/config/config.go for the full list and defaults) — no config file format for Phase 0:

Var	Default	Purpose
GRPC_LISTEN_ADDR	:4317	Agent-facing gRPC listen address
TLS_CERT_FILE / TLS_KEY_FILE	/etc/sentry-ingest/server{,-key}.pem	ingest's own mTLS identity
TLS_CLIENT_CA_FILE	/etc/sentry-ingest/ca.pem	CA used to verify agent client certs
REDPANDA_BROKERS	localhost:9092	Comma-separated broker list
REDPANDA_TOPIC	sentry.logs.raw	Must match the topic provisioned in /transport
REDPANDA_CONSUMER_GROUP	sentry-ingest	Consumer group id
CLICKHOUSE_ADDR	localhost:9000	Native protocol port, not HTTP
CLICKHOUSE_DATABASE / _USERNAME / _PASSWORD	sentry / default / ``
CONSUMER_BATCH_MAX_SIZE	500	Records per ClickHouse batch insert
CONSUMER_BATCH_FLUSH_INTERVAL_MS	2000	Max time a partial batch waits before flushing

Building & testing

go build ./...
go vet ./...
go test ./...

Requires google.golang.org/protobuf/cmd/protoc-gen-go and google.golang.org/grpc/cmd/protoc-gen-go-grpc only if you're regenerating /proto's Go bindings — ingest itself just imports the already-generated github.com/sentry/sentry/proto module (see the replace directive in go.mod, pointing at ../proto).

# from the repo root, not ingest/
docker build -f ingest/Dockerfile -t sentry-ingest .

Testing notes

internal/consumer and internal/grpcserver depend on Redpanda and ClickHouse only through small interfaces (reader/chWriter in consumer, batchProducer in grpcserver), so the flush/commit/error-handling logic is unit-tested against fakes — no embedded broker or database needed. What's not covered by these tests: the real kafka.Reader/kafka.Writer wiring and the ClickHouse native-protocol driver itself. Those are only exercised by the docker-compose end-to-end flow described in /docs/phase-0-runbook.md.