Scaffold Phase 0: agent -> Redpanda -> ingest -> ClickHouse -> api -> web

End-to-end log pipeline for Linux hosts, per /docs/architecture.md:

- proto: shared gRPC contract (agent <-> ingest), Go bindings checked in
- agent: Rust, musl-targeted, journald/file sourcing, RFC5424 parser,
  mTLS gRPC client, no required config for the common case
- ingest: Go, single binary with --mode server|consumer|all; gRPC front
  end forwards to Redpanda unchanged, consumer normalizes and
  batch-writes to ClickHouse with at-least-once delivery
- storage: ClickHouse schema + a plain SQL-file migration runner
- api: minimal SELECT-only query endpoint, plain REST (not gRPC+gateway
  yet -- see api/README.md)
- web: SvelteKit static SPA, one query page
- transport: Redpanda compose + topic provisioning
- cli: sentryctl ping stub
- hack/dev-certs: throwaway CA + cert generation for local mTLS
- root docker-compose.yml + docs/phase-0-runbook.md tie it together

Not yet run end-to-end against real Docker/ClickHouse/Redpanda -- see the
runbook's caveats section before relying on this working as-is.
This commit is contained in:
2026-08-13 08:25:19 -07:00
commit b6b092c912
92 changed files with 7796 additions and 0 deletions
+30
View File
@@ -0,0 +1,30 @@
# proto
Shared `.proto` contracts. Source of truth for the agent↔ingest gRPC
service; each language generates its own bindings from these files rather
than sharing generated code across languages.
- `sentry/logs/v1/logs.proto``LogIngest.PushBatch`, the only RPC an
agent ever calls.
## Go bindings
Go is the one language here with pre-generated, checked-in bindings
(`sentry/logs/v1/logs.pb.go`, `logs_grpc.pb.go`), living in this directory
as its own module (`github.com/sentry/sentry/proto`) that `/ingest` and
`/api` depend on via a local `replace` directive in their `go.mod`. Rust
(`/agent`) instead generates its bindings at build time via `tonic-build`
(see `agent/sentry-agent/build.rs`) — no checked-in Rust output.
To regenerate the Go bindings after changing `logs.proto`:
```sh
go install google.golang.org/protobuf/cmd/protoc-gen-go@latest
go install google.golang.org/grpc/cmd/protoc-gen-go-grpc@latest
cd proto
protoc --go_out=. --go_opt=paths=source_relative \
--go-grpc_out=. --go-grpc_opt=paths=source_relative \
sentry/logs/v1/logs.proto
go build ./...
```