Scaffold Phase 0: agent -> Redpanda -> ingest -> ClickHouse -> api -> web
End-to-end log pipeline for Linux hosts, per /docs/architecture.md: - proto: shared gRPC contract (agent <-> ingest), Go bindings checked in - agent: Rust, musl-targeted, journald/file sourcing, RFC5424 parser, mTLS gRPC client, no required config for the common case - ingest: Go, single binary with --mode server|consumer|all; gRPC front end forwards to Redpanda unchanged, consumer normalizes and batch-writes to ClickHouse with at-least-once delivery - storage: ClickHouse schema + a plain SQL-file migration runner - api: minimal SELECT-only query endpoint, plain REST (not gRPC+gateway yet -- see api/README.md) - web: SvelteKit static SPA, one query page - transport: Redpanda compose + topic provisioning - cli: sentryctl ping stub - hack/dev-certs: throwaway CA + cert generation for local mTLS - root docker-compose.yml + docs/phase-0-runbook.md tie it together Not yet run end-to-end against real Docker/ClickHouse/Redpanda -- see the runbook's caveats section before relying on this working as-is.
This commit is contained in:
@@ -0,0 +1,15 @@
|
||||
# Build context must be the repo root (sentry/), not ingest/, since this
|
||||
# needs both ingest/ and proto/:
|
||||
# docker build -f ingest/Dockerfile -t sentry-ingest .
|
||||
|
||||
FROM golang:1.25-alpine AS builder
|
||||
WORKDIR /src
|
||||
COPY proto ./proto
|
||||
COPY ingest ./ingest
|
||||
WORKDIR /src/ingest
|
||||
RUN go mod download
|
||||
RUN CGO_ENABLED=0 GOOS=linux go build -o /out/ingest ./cmd/ingest
|
||||
|
||||
FROM gcr.io/distroless/static-debian12
|
||||
COPY --from=builder /out/ingest /ingest
|
||||
ENTRYPOINT ["/ingest"]
|
||||
@@ -0,0 +1,85 @@
|
||||
# ingest
|
||||
|
||||
Go service sitting between the Rust agent and ClickHouse. Two halves in one
|
||||
binary, selected with `--mode`:
|
||||
|
||||
- **server** — mTLS gRPC front end (`LogIngest.PushBatch`) that agents
|
||||
connect to. Forwards each record, proto-encoded and unchanged, onto
|
||||
Redpanda. Does no normalization — kept thin so agent-facing latency isn't
|
||||
coupled to ClickHouse write performance.
|
||||
- **consumer** — reads back off Redpanda, normalizes into the ClickHouse row
|
||||
shape (`internal/normalize`), and batch-writes via the native protocol
|
||||
driver. Commits Redpanda offsets only after a successful ClickHouse
|
||||
write, so a ClickHouse outage causes redelivery on restart rather than
|
||||
data loss.
|
||||
- **all** (default) — both, in one process. This is what docker-compose
|
||||
runs. Splitting into two deployments later (e.g. to scale them
|
||||
independently in k8s) is a manifest change, not a code change — see
|
||||
`--mode`.
|
||||
|
||||
## Why Redpanda stays in the path
|
||||
|
||||
Confirmed with the project owner during Phase 0 planning: the gRPC front
|
||||
end produces to Redpanda rather than writing ClickHouse directly. This
|
||||
exercises the pinned transport layer from day one and keeps agents from
|
||||
ever needing Kafka credentials — mTLS to `ingest` is the only network
|
||||
egress an agent has. See `/docs/architecture.md`.
|
||||
|
||||
## Dependencies worth knowing about
|
||||
|
||||
- **github.com/segmentio/kafka-go** — pure Go, no cgo, chosen over
|
||||
franz-go/confluent-kafka-go specifically to keep the distroless build
|
||||
simple (confirmed with the project owner; see git history / PR
|
||||
discussion for the tradeoffs considered).
|
||||
- **github.com/ClickHouse/clickhouse-go/v2** — official client, native
|
||||
protocol, pure Go (no cgo).
|
||||
- **golang.org/x/sync/errgroup** — used in `cmd/ingest/main.go` to run the
|
||||
server and consumer halves concurrently and propagate the first error.
|
||||
|
||||
## Configuration
|
||||
|
||||
All via environment variables (see `internal/config/config.go` for the
|
||||
full list and defaults) — no config file format for Phase 0:
|
||||
|
||||
| Var | Default | Purpose |
|
||||
|---|---|---|
|
||||
| `GRPC_LISTEN_ADDR` | `:4317` | Agent-facing gRPC listen address |
|
||||
| `TLS_CERT_FILE` / `TLS_KEY_FILE` | `/etc/sentry-ingest/server{,-key}.pem` | ingest's own mTLS identity |
|
||||
| `TLS_CLIENT_CA_FILE` | `/etc/sentry-ingest/ca.pem` | CA used to verify agent client certs |
|
||||
| `REDPANDA_BROKERS` | `localhost:9092` | Comma-separated broker list |
|
||||
| `REDPANDA_TOPIC` | `sentry.logs.raw` | Must match the topic provisioned in `/transport` |
|
||||
| `REDPANDA_CONSUMER_GROUP` | `sentry-ingest` | Consumer group id |
|
||||
| `CLICKHOUSE_ADDR` | `localhost:9000` | Native protocol port, not HTTP |
|
||||
| `CLICKHOUSE_DATABASE` / `_USERNAME` / `_PASSWORD` | `sentry` / `default` / `` | |
|
||||
| `CONSUMER_BATCH_MAX_SIZE` | `500` | Records per ClickHouse batch insert |
|
||||
| `CONSUMER_BATCH_FLUSH_INTERVAL_MS` | `2000` | Max time a partial batch waits before flushing |
|
||||
|
||||
## Building & testing
|
||||
|
||||
```sh
|
||||
go build ./...
|
||||
go vet ./...
|
||||
go test ./...
|
||||
```
|
||||
|
||||
Requires `google.golang.org/protobuf/cmd/protoc-gen-go` and
|
||||
`google.golang.org/grpc/cmd/protoc-gen-go-grpc` only if you're
|
||||
regenerating `/proto`'s Go bindings — ingest itself just imports the
|
||||
already-generated `github.com/sentry/sentry/proto` module (see the
|
||||
`replace` directive in `go.mod`, pointing at `../proto`).
|
||||
|
||||
```sh
|
||||
# from the repo root, not ingest/
|
||||
docker build -f ingest/Dockerfile -t sentry-ingest .
|
||||
```
|
||||
|
||||
## Testing notes
|
||||
|
||||
`internal/consumer` and `internal/grpcserver` depend on Redpanda and
|
||||
ClickHouse only through small interfaces (`reader`/`chWriter` in consumer,
|
||||
`batchProducer` in grpcserver), so the flush/commit/error-handling logic is
|
||||
unit-tested against fakes — no embedded broker or database needed. What's
|
||||
*not* covered by these tests: the real `kafka.Reader`/`kafka.Writer`
|
||||
wiring and the ClickHouse native-protocol driver itself. Those are only
|
||||
exercised by the docker-compose end-to-end flow described in
|
||||
`/docs/phase-0-runbook.md`.
|
||||
@@ -0,0 +1,77 @@
|
||||
// Command ingest is the Sentry ingest service. It has two halves that can
|
||||
// run in one process or be split across deployments via --mode:
|
||||
//
|
||||
// - server: mTLS gRPC front end that agents push batches to; forwards
|
||||
// them onto Redpanda unchanged.
|
||||
// - consumer: reads back off Redpanda, normalizes, batch-writes to
|
||||
// ClickHouse.
|
||||
// - all (default): both, in one process — the Phase 0 / docker-compose
|
||||
// shape. Splitting into separate deployments later is a k8s manifest
|
||||
// change, not a code change.
|
||||
package main
|
||||
|
||||
import (
|
||||
"context"
|
||||
"flag"
|
||||
"fmt"
|
||||
"log/slog"
|
||||
"os"
|
||||
"os/signal"
|
||||
"syscall"
|
||||
|
||||
"golang.org/x/sync/errgroup"
|
||||
|
||||
"github.com/sentry/sentry/ingest/internal/clickhousewriter"
|
||||
"github.com/sentry/sentry/ingest/internal/config"
|
||||
"github.com/sentry/sentry/ingest/internal/consumer"
|
||||
"github.com/sentry/sentry/ingest/internal/grpcserver"
|
||||
"github.com/sentry/sentry/ingest/internal/producer"
|
||||
)
|
||||
|
||||
func main() {
|
||||
mode := flag.String("mode", "all", "which half of ingest to run: server | consumer | all")
|
||||
flag.Parse()
|
||||
|
||||
if *mode != "server" && *mode != "consumer" && *mode != "all" {
|
||||
fmt.Fprintf(os.Stderr, "unknown --mode %q, must be server|consumer|all\n", *mode)
|
||||
os.Exit(1)
|
||||
}
|
||||
|
||||
logger := slog.New(slog.NewJSONHandler(os.Stdout, nil))
|
||||
|
||||
cfg, err := config.Load()
|
||||
if err != nil {
|
||||
logger.Error("loading config", "error", err)
|
||||
os.Exit(1)
|
||||
}
|
||||
|
||||
ctx, stop := signal.NotifyContext(context.Background(), syscall.SIGINT, syscall.SIGTERM)
|
||||
defer stop()
|
||||
|
||||
g, ctx := errgroup.WithContext(ctx)
|
||||
|
||||
if *mode == "server" || *mode == "all" {
|
||||
p := producer.New(cfg.Redpanda)
|
||||
defer p.Close()
|
||||
srv := grpcserver.New(logger, cfg.GRPC, cfg.TLS, p)
|
||||
g.Go(func() error { return srv.Run(ctx) })
|
||||
}
|
||||
|
||||
if *mode == "consumer" || *mode == "all" {
|
||||
chw, err := clickhousewriter.New(ctx, cfg.ClickHouse)
|
||||
if err != nil {
|
||||
logger.Error("connecting to clickhouse", "error", err)
|
||||
os.Exit(1)
|
||||
}
|
||||
defer chw.Close()
|
||||
c := consumer.New(logger, cfg.Redpanda, cfg.Batch, chw)
|
||||
g.Go(func() error { return c.Run(ctx) })
|
||||
}
|
||||
|
||||
logger.Info("ingest started", "mode", *mode)
|
||||
|
||||
if err := g.Wait(); err != nil {
|
||||
logger.Error("ingest exited with error", "error", err)
|
||||
os.Exit(1)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,34 @@
|
||||
module github.com/sentry/sentry/ingest
|
||||
|
||||
go 1.25.0
|
||||
|
||||
replace github.com/sentry/sentry/proto => ../proto
|
||||
|
||||
require (
|
||||
github.com/ClickHouse/clickhouse-go/v2 v2.48.0
|
||||
github.com/segmentio/kafka-go v0.4.51
|
||||
github.com/sentry/sentry/proto v0.0.0-00010101000000-000000000000
|
||||
golang.org/x/sync v0.22.0
|
||||
google.golang.org/grpc v1.83.0
|
||||
google.golang.org/protobuf v1.36.12
|
||||
)
|
||||
|
||||
require (
|
||||
github.com/ClickHouse/ch-go v0.74.0 // indirect
|
||||
github.com/andybalholm/brotli v1.2.2 // indirect
|
||||
github.com/cespare/xxhash/v2 v2.3.0 // indirect
|
||||
github.com/go-faster/city v1.0.1 // indirect
|
||||
github.com/go-faster/errors v0.7.1 // indirect
|
||||
github.com/google/uuid v1.6.0 // indirect
|
||||
github.com/klauspost/compress v1.19.1 // indirect
|
||||
github.com/paulmach/orb v0.13.0 // indirect
|
||||
github.com/pierrec/lz4/v4 v4.1.27 // indirect
|
||||
github.com/segmentio/asm v1.2.1 // indirect
|
||||
github.com/shopspring/decimal v1.4.0 // indirect
|
||||
go.opentelemetry.io/otel v1.44.0 // indirect
|
||||
go.opentelemetry.io/otel/trace v1.44.0 // indirect
|
||||
golang.org/x/net v0.57.0 // indirect
|
||||
golang.org/x/sys v0.47.0 // indirect
|
||||
golang.org/x/text v0.40.0 // indirect
|
||||
google.golang.org/genproto/googleapis/rpc v0.0.0-20260526163538-3dc84a4a5aaa // indirect
|
||||
)
|
||||
@@ -0,0 +1,78 @@
|
||||
github.com/ClickHouse/ch-go v0.74.0 h1:uYs2m4wIt0ZHSM1E72rg0maCfzhR2V3xWb/vZEgpeWE=
|
||||
github.com/ClickHouse/ch-go v0.74.0/go.mod h1:sZ/r+8ttZMjyrP9PuFbgoVbth1ywIu2LIQNA2vgko6M=
|
||||
github.com/ClickHouse/clickhouse-go/v2 v2.48.0 h1:auzd4VkapQYhQF8F2Gog7s3x78Bi1JZmByxGbrw3C+4=
|
||||
github.com/ClickHouse/clickhouse-go/v2 v2.48.0/go.mod h1:lBjUCPRG6RpRQdMbkXq+JV8rY0/O5lw+Z7jShgReFjM=
|
||||
github.com/andybalholm/brotli v1.2.2 h1:HzTuoo2ErYQqf5qvcJInB8uvqSVxRttzkFexPWtnceM=
|
||||
github.com/andybalholm/brotli v1.2.2/go.mod h1:rzTDkvFWvIrjDXZHkuS16NPggd91W3kUSvPlQ1pLaKY=
|
||||
github.com/cespare/xxhash/v2 v2.3.0 h1:UL815xU9SqsFlibzuggzjXhog7bL6oX9BbNZnL2UFvs=
|
||||
github.com/cespare/xxhash/v2 v2.3.0/go.mod h1:VGX0DQ3Q6kWi7AoAeZDth3/j3BFtOZR5XLFGgcrjCOs=
|
||||
github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c=
|
||||
github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
|
||||
github.com/go-faster/city v1.0.1 h1:4WAxSZ3V2Ws4QRDrscLEDcibJY8uf41H6AhXDrNDcGw=
|
||||
github.com/go-faster/city v1.0.1/go.mod h1:jKcUJId49qdW3L1qKHH/3wPeUstCVpVSXTM6vO3VcTw=
|
||||
github.com/go-faster/errors v0.7.1 h1:MkJTnDoEdi9pDabt1dpWf7AA8/BaSYZqibYyhZ20AYg=
|
||||
github.com/go-faster/errors v0.7.1/go.mod h1:5ySTjWFiphBs07IKuiL69nxdfd5+fzh1u7FPGZP2quo=
|
||||
github.com/go-logr/logr v1.4.3 h1:CjnDlHq8ikf6E492q6eKboGOC0T8CDaOvkHCIg8idEI=
|
||||
github.com/go-logr/logr v1.4.3/go.mod h1:9T104GzyrTigFIr8wt5mBrctHMim0Nb2HLGrmQ40KvY=
|
||||
github.com/go-logr/stdr v1.2.2 h1:hSWxHoqTgW2S2qGc0LTAI563KZ5YKYRhT3MFKZMbjag=
|
||||
github.com/go-logr/stdr v1.2.2/go.mod h1:mMo/vtBO5dYbehREoey6XUKy/eSumjCCveDpRre4VKE=
|
||||
github.com/golang/protobuf v1.5.4 h1:i7eJL8qZTpSEXOPTxNKhASYpMn+8e5Q6AdndVa1dWek=
|
||||
github.com/golang/protobuf v1.5.4/go.mod h1:lnTiLA8Wa4RWRcIUkrtSVa5nRhsEGBg48fD6rSs7xps=
|
||||
github.com/google/go-cmp v0.7.0 h1:wk8382ETsv4JYUZwIsn6YpYiWiBsYLSJiTsyBybVuN8=
|
||||
github.com/google/go-cmp v0.7.0/go.mod h1:pXiqmnSA92OHEEa9HXL2W4E7lf9JzCmGVUdgjX3N/iU=
|
||||
github.com/google/uuid v1.6.0 h1:NIvaJDMOsjHA8n1jAhLSgzrAzy1Hgr+hNrb57e+94F0=
|
||||
github.com/google/uuid v1.6.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo=
|
||||
github.com/klauspost/compress v1.19.1 h1:VsB4HPswih7mmZ8WleSFQ75c/Ui1M4trX5oAsJnhSlk=
|
||||
github.com/klauspost/compress v1.19.1/go.mod h1:cwPg85FWrGar70rWktvGQj8/hthj3wpl0PGDogxkrSQ=
|
||||
github.com/paulmach/orb v0.13.0 h1:r7n7mQGGF+cj/CbcivEj9J3HGK+XR+yXnvzRdq9saIw=
|
||||
github.com/paulmach/orb v0.13.0/go.mod h1:6scRWINywA2Jf05dcjOfLfxrUIMECvTSG2MVbRLxu/k=
|
||||
github.com/pierrec/lz4/v4 v4.1.27 h1:+PhzhWDrjRj89TH2sw43nE3+4+W8lSxIuQadEHZyjUk=
|
||||
github.com/pierrec/lz4/v4 v4.1.27/go.mod h1:EoQMVJgeeEOMsCqCzqFm2O0cJvljX2nGZjcRIPL34O4=
|
||||
github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2 h1:Jamvg5psRIccs7FGNTlIRMkT8wgtp5eCXdBlqhYGL6U=
|
||||
github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
|
||||
github.com/segmentio/asm v1.2.1 h1:DTNbBqs57ioxAD4PrArqftgypG4/qNpXoJx8TVXxPR0=
|
||||
github.com/segmentio/asm v1.2.1/go.mod h1:BqMnlJP91P8d+4ibuonYZw9mfnzI9HfxselHZr5aAcs=
|
||||
github.com/segmentio/kafka-go v0.4.51 h1:JgDPPG75tC1rWIS2Me6MwcvXJ6f49UQ4HjAOef71Hno=
|
||||
github.com/segmentio/kafka-go v0.4.51/go.mod h1:Y1gn60kzLEEaW28YshXyk2+VCUKbJ3Qr6DrnT3i4+9E=
|
||||
github.com/shopspring/decimal v1.4.0 h1:bxl37RwXBklmTi0C79JfXCEBD1cqqHt0bbgBAGFp81k=
|
||||
github.com/shopspring/decimal v1.4.0/go.mod h1:gawqmDU56v4yIKSwfBSFip1HdCCXN8/+DMd9qYNcwME=
|
||||
github.com/stretchr/testify v1.11.1 h1:7s2iGBzp5EwR7/aIZr8ao5+dra3wiQyKjjFuvgVKu7U=
|
||||
github.com/stretchr/testify v1.11.1/go.mod h1:wZwfW3scLgRK+23gO65QZefKpKQRnfz6sD981Nm4B6U=
|
||||
github.com/xdg-go/pbkdf2 v1.0.0 h1:Su7DPu48wXMwC3bs7MCNG+z4FhcyEuz5dlvchbq0B0c=
|
||||
github.com/xdg-go/pbkdf2 v1.0.0/go.mod h1:jrpuAogTd400dnrH08LKmI/xc1MbPOebTwRqcT5RDeI=
|
||||
github.com/xdg-go/scram v1.1.2 h1:FHX5I5B4i4hKRVRBCFRxq1iQRej7WO3hhBuJf+UUySY=
|
||||
github.com/xdg-go/scram v1.1.2/go.mod h1:RT/sEzTbU5y00aCK8UOx6R7YryM0iF1N2MOmC3kKLN4=
|
||||
github.com/xdg-go/stringprep v1.0.4 h1:XLI/Ng3O1Atzq0oBs3TWm+5ZVgkq2aqdlvP9JtoZ6c8=
|
||||
github.com/xdg-go/stringprep v1.0.4/go.mod h1:mPGuuIYwz7CmR2bT9j4GbQqutWS1zV24gijq1dTyGkM=
|
||||
github.com/xyproto/randomstring v1.0.5 h1:YtlWPoRdgMu3NZtP45drfy1GKoojuR7hmRcnhZqKjWU=
|
||||
github.com/xyproto/randomstring v1.0.5/go.mod h1:rgmS5DeNXLivK7YprL0pY+lTuhNQW3iGxZ18UQApw/E=
|
||||
go.opentelemetry.io/auto/sdk v1.2.1 h1:jXsnJ4Lmnqd11kwkBV2LgLoFMZKizbCi5fNZ/ipaZ64=
|
||||
go.opentelemetry.io/auto/sdk v1.2.1/go.mod h1:KRTj+aOaElaLi+wW1kO/DZRXwkF4C5xPbEe3ZiIhN7Y=
|
||||
go.opentelemetry.io/otel v1.44.0 h1:JjwHmHpA4iZ3wBxluu2fbbE7j4kqlE8jXyAyPXH7HqU=
|
||||
go.opentelemetry.io/otel v1.44.0/go.mod h1:BMgjTHL9WPRlRjL2oZCBTL4whCGtXch2H4BhOPIAyYc=
|
||||
go.opentelemetry.io/otel/metric v1.44.0 h1:1w0gILTcHdr3YI+ixLyjemwrVnsMURbTZFrSYCdDdmc=
|
||||
go.opentelemetry.io/otel/metric v1.44.0/go.mod h1:8O7hanEPBNgEMmybD3s2VBKcgWOCsA6tzHBPODAiquo=
|
||||
go.opentelemetry.io/otel/sdk v1.44.0 h1:nHYwb9lK+fJPU/dnT6s7W7Z8itMWyqrnVfbheVYrZ58=
|
||||
go.opentelemetry.io/otel/sdk v1.44.0/go.mod h1:Osuydd3Se74nqjAKxid74N5eC+jfEqfTegHRnq58oK0=
|
||||
go.opentelemetry.io/otel/sdk/metric v1.44.0 h1:3LlKgI+VjbVsjNRFZJZAJ30WjXC5VkNRks6si09iEfI=
|
||||
go.opentelemetry.io/otel/sdk/metric v1.44.0/go.mod h1:5B5pMARnXxKhltooO4xUuCBorl65a4EpnTalObqOigA=
|
||||
go.opentelemetry.io/otel/trace v1.44.0 h1:jxF5CsGYCe74MCRx2X4g7WsY/VBKRqqpNvXlX/6gtIk=
|
||||
go.opentelemetry.io/otel/trace v1.44.0/go.mod h1:oLl1jrMQAVo6v3GAggN+1VH9VIz9iUSvW53sW1Q8PIE=
|
||||
golang.org/x/net v0.57.0 h1:K5+3DljvIuDG9/Jv9rvyMywYNFCQ9RSUY6OOTTkT+tE=
|
||||
golang.org/x/net v0.57.0/go.mod h1:KpXc8iv+r3XplLAG/f7Jsf9RPszJzdR0f58q9vGOuEU=
|
||||
golang.org/x/sync v0.22.0 h1:SZjpbeLmrCk4xhRSZFNZW5gFUeCeFgjekvI/+gfScek=
|
||||
golang.org/x/sync v0.22.0/go.mod h1:9xrNwdLfx4jkKbNva9FpL6vEN7evnE43NNNJQ2LF3+0=
|
||||
golang.org/x/sys v0.47.0 h1:o7XGOvZQCADBQQ4Y7VNq2dRWQR7JmOUW8Kxx4ZsNgWs=
|
||||
golang.org/x/sys v0.47.0/go.mod h1:4GL1E5IUh+htKOUEOaiffhrAeqysfVGipDYzABqnCmw=
|
||||
golang.org/x/text v0.40.0 h1:Ub2Z6/xjgF1WrYQz2nuITOEegKFtiIy+rieRJ5lHZKs=
|
||||
golang.org/x/text v0.40.0/go.mod h1:hpnzDAfGV753zIKo+wk3u1bVKCGPbrnF7+7LBF/UHVY=
|
||||
gonum.org/v1/gonum v0.17.0 h1:VbpOemQlsSMrYmn7T2OUvQ4dqxQXU+ouZFQsZOx50z4=
|
||||
gonum.org/v1/gonum v0.17.0/go.mod h1:El3tOrEuMpv2UdMrbNlKEh9vd86bmQ6vqIcDwxEOc1E=
|
||||
google.golang.org/genproto/googleapis/rpc v0.0.0-20260526163538-3dc84a4a5aaa h1:mZHHdPZl0dbGHCflZgAq/Q468DWVFcU2whhB2KAo8fk=
|
||||
google.golang.org/genproto/googleapis/rpc v0.0.0-20260526163538-3dc84a4a5aaa/go.mod h1:4Hqkh8ycfw05ld/3BWL7rJOSfebL2Q+DVDeRgYgxUU8=
|
||||
google.golang.org/grpc v1.83.0 h1:JeNZEKJFbQxArAMl+hiytHauacDNqJUllNfmIMmpqnQ=
|
||||
google.golang.org/grpc v1.83.0/go.mod h1:kDyl6SKsiHKt0uylY5gtn5cEjkrIOhQOGDgIc4JGwzQ=
|
||||
google.golang.org/protobuf v1.36.12 h1:pJOKDDOyeXErUroCihFAd5LQuwXBSpVnKGrj5o/fwxc=
|
||||
google.golang.org/protobuf v1.36.12/go.mod h1:HTf+CrKn2C3g5S8VImy6tdcUvCska2kB7j23XfzDpco=
|
||||
gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA=
|
||||
gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
|
||||
@@ -0,0 +1,60 @@
|
||||
// Package clickhousewriter batch-inserts normalized log rows into
|
||||
// ClickHouse using the native protocol driver's batch API.
|
||||
package clickhousewriter
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
|
||||
"github.com/ClickHouse/clickhouse-go/v2"
|
||||
"github.com/ClickHouse/clickhouse-go/v2/lib/driver"
|
||||
|
||||
"github.com/sentry/sentry/ingest/internal/config"
|
||||
"github.com/sentry/sentry/ingest/internal/normalize"
|
||||
logsv1 "github.com/sentry/sentry/proto/sentry/logs/v1"
|
||||
)
|
||||
|
||||
type Writer struct {
|
||||
conn driver.Conn
|
||||
}
|
||||
|
||||
func New(ctx context.Context, cfg config.ClickHouseConfig) (*Writer, error) {
|
||||
conn, err := clickhouse.Open(&clickhouse.Options{
|
||||
Addr: []string{cfg.Addr},
|
||||
Auth: clickhouse.Auth{
|
||||
Database: cfg.Database,
|
||||
Username: cfg.Username,
|
||||
Password: cfg.Password,
|
||||
},
|
||||
})
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("opening clickhouse connection: %w", err)
|
||||
}
|
||||
if err := conn.Ping(ctx); err != nil {
|
||||
return nil, fmt.Errorf("pinging clickhouse: %w", err)
|
||||
}
|
||||
return &Writer{conn: conn}, nil
|
||||
}
|
||||
|
||||
func (w *Writer) Close() error {
|
||||
return w.conn.Close()
|
||||
}
|
||||
|
||||
func (w *Writer) WriteBatch(ctx context.Context, records []*logsv1.LogRecord) error {
|
||||
batch, err := w.conn.PrepareBatch(ctx, "INSERT INTO logs (timestamp, host, service, severity, message, attributes)")
|
||||
if err != nil {
|
||||
return fmt.Errorf("preparing batch: %w", err)
|
||||
}
|
||||
|
||||
for _, rec := range records {
|
||||
row := normalize.ToRow(rec)
|
||||
if err := batch.Append(row.Timestamp, row.Host, row.Service, row.Severity, row.Message, row.Attributes); err != nil {
|
||||
return fmt.Errorf("appending row to batch: %w", err)
|
||||
}
|
||||
}
|
||||
|
||||
if err := batch.Send(); err != nil {
|
||||
return fmt.Errorf("sending batch: %w", err)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
@@ -0,0 +1,95 @@
|
||||
// Package config loads ingest's configuration from environment variables.
|
||||
// Phase 0 deliberately has no config file format of its own — env vars are
|
||||
// enough for a docker-compose/k8s deployment and avoid pulling in a config
|
||||
// library.
|
||||
package config
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"os"
|
||||
"strconv"
|
||||
"strings"
|
||||
)
|
||||
|
||||
type Config struct {
|
||||
GRPC GRPCConfig
|
||||
TLS TLSConfig
|
||||
Redpanda RedpandaConfig
|
||||
ClickHouse ClickHouseConfig
|
||||
Batch BatchConfig
|
||||
}
|
||||
|
||||
type GRPCConfig struct {
|
||||
ListenAddr string
|
||||
}
|
||||
|
||||
// TLSConfig is the server-side mTLS material: the ingest service's own
|
||||
// cert/key, and the CA used to verify agent client certs.
|
||||
type TLSConfig struct {
|
||||
CertFile string
|
||||
KeyFile string
|
||||
ClientCAFile string
|
||||
}
|
||||
|
||||
type RedpandaConfig struct {
|
||||
Brokers []string
|
||||
Topic string
|
||||
ConsumerGroup string
|
||||
}
|
||||
|
||||
type ClickHouseConfig struct {
|
||||
Addr string
|
||||
Database string
|
||||
Username string
|
||||
Password string
|
||||
}
|
||||
|
||||
type BatchConfig struct {
|
||||
MaxSize int
|
||||
FlushIntervalMS int
|
||||
}
|
||||
|
||||
func Load() (Config, error) {
|
||||
cfg := Config{
|
||||
GRPC: GRPCConfig{
|
||||
ListenAddr: getenv("GRPC_LISTEN_ADDR", ":4317"),
|
||||
},
|
||||
TLS: TLSConfig{
|
||||
CertFile: getenv("TLS_CERT_FILE", "/etc/sentry-ingest/server.pem"),
|
||||
KeyFile: getenv("TLS_KEY_FILE", "/etc/sentry-ingest/server-key.pem"),
|
||||
ClientCAFile: getenv("TLS_CLIENT_CA_FILE", "/etc/sentry-ingest/ca.pem"),
|
||||
},
|
||||
Redpanda: RedpandaConfig{
|
||||
Brokers: strings.Split(getenv("REDPANDA_BROKERS", "localhost:9092"), ","),
|
||||
Topic: getenv("REDPANDA_TOPIC", "sentry.logs.raw"),
|
||||
ConsumerGroup: getenv("REDPANDA_CONSUMER_GROUP", "sentry-ingest"),
|
||||
},
|
||||
ClickHouse: ClickHouseConfig{
|
||||
Addr: getenv("CLICKHOUSE_ADDR", "localhost:9000"),
|
||||
Database: getenv("CLICKHOUSE_DATABASE", "sentry"),
|
||||
Username: getenv("CLICKHOUSE_USERNAME", "default"),
|
||||
Password: getenv("CLICKHOUSE_PASSWORD", ""),
|
||||
},
|
||||
}
|
||||
|
||||
maxSize, err := strconv.Atoi(getenv("CONSUMER_BATCH_MAX_SIZE", "500"))
|
||||
if err != nil {
|
||||
return Config{}, fmt.Errorf("CONSUMER_BATCH_MAX_SIZE: %w", err)
|
||||
}
|
||||
cfg.Batch.MaxSize = maxSize
|
||||
|
||||
flushMS, err := strconv.Atoi(getenv("CONSUMER_BATCH_FLUSH_INTERVAL_MS", "2000"))
|
||||
if err != nil {
|
||||
return Config{}, fmt.Errorf("CONSUMER_BATCH_FLUSH_INTERVAL_MS: %w", err)
|
||||
}
|
||||
cfg.Batch.FlushIntervalMS = flushMS
|
||||
|
||||
return cfg, nil
|
||||
}
|
||||
|
||||
func getenv(key, fallback string) string {
|
||||
if v := os.Getenv(key); v != "" {
|
||||
return v
|
||||
}
|
||||
return fallback
|
||||
}
|
||||
@@ -0,0 +1,49 @@
|
||||
package config
|
||||
|
||||
import "testing"
|
||||
|
||||
func TestLoadDefaults(t *testing.T) {
|
||||
cfg, err := Load()
|
||||
if err != nil {
|
||||
t.Fatalf("Load() error = %v", err)
|
||||
}
|
||||
if cfg.GRPC.ListenAddr != ":4317" {
|
||||
t.Errorf("GRPC.ListenAddr = %q, want :4317", cfg.GRPC.ListenAddr)
|
||||
}
|
||||
if cfg.Redpanda.Topic != "sentry.logs.raw" {
|
||||
t.Errorf("Redpanda.Topic = %q, want sentry.logs.raw", cfg.Redpanda.Topic)
|
||||
}
|
||||
if cfg.Batch.MaxSize != 500 {
|
||||
t.Errorf("Batch.MaxSize = %d, want 500", cfg.Batch.MaxSize)
|
||||
}
|
||||
if cfg.Batch.FlushIntervalMS != 2000 {
|
||||
t.Errorf("Batch.FlushIntervalMS = %d, want 2000", cfg.Batch.FlushIntervalMS)
|
||||
}
|
||||
}
|
||||
|
||||
func TestLoadOverridesFromEnv(t *testing.T) {
|
||||
t.Setenv("GRPC_LISTEN_ADDR", ":9999")
|
||||
t.Setenv("REDPANDA_BROKERS", "a:9092,b:9092")
|
||||
t.Setenv("CONSUMER_BATCH_MAX_SIZE", "10")
|
||||
|
||||
cfg, err := Load()
|
||||
if err != nil {
|
||||
t.Fatalf("Load() error = %v", err)
|
||||
}
|
||||
if cfg.GRPC.ListenAddr != ":9999" {
|
||||
t.Errorf("GRPC.ListenAddr = %q, want :9999", cfg.GRPC.ListenAddr)
|
||||
}
|
||||
if len(cfg.Redpanda.Brokers) != 2 || cfg.Redpanda.Brokers[0] != "a:9092" || cfg.Redpanda.Brokers[1] != "b:9092" {
|
||||
t.Errorf("Redpanda.Brokers = %+v, want [a:9092 b:9092]", cfg.Redpanda.Brokers)
|
||||
}
|
||||
if cfg.Batch.MaxSize != 10 {
|
||||
t.Errorf("Batch.MaxSize = %d, want 10", cfg.Batch.MaxSize)
|
||||
}
|
||||
}
|
||||
|
||||
func TestLoadInvalidBatchSizeErrors(t *testing.T) {
|
||||
t.Setenv("CONSUMER_BATCH_MAX_SIZE", "not-a-number")
|
||||
if _, err := Load(); err == nil {
|
||||
t.Fatal("expected error for non-numeric CONSUMER_BATCH_MAX_SIZE, got nil")
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,124 @@
|
||||
// Package consumer reads normalized-on-write LogRecords back off Redpanda
|
||||
// and batch-writes them into ClickHouse. Offsets are committed only after
|
||||
// a successful ClickHouse write, so a ClickHouse outage causes redelivery
|
||||
// on restart rather than silent data loss (at-least-once, not exactly-once
|
||||
// — Phase 0 doesn't dedupe on the consumer side).
|
||||
package consumer
|
||||
|
||||
import (
|
||||
"context"
|
||||
"log/slog"
|
||||
"time"
|
||||
|
||||
"github.com/segmentio/kafka-go"
|
||||
"google.golang.org/protobuf/proto"
|
||||
|
||||
"github.com/sentry/sentry/ingest/internal/config"
|
||||
logsv1 "github.com/sentry/sentry/proto/sentry/logs/v1"
|
||||
)
|
||||
|
||||
// chWriter is the subset of *clickhousewriter.Writer this package depends
|
||||
// on, kept as an interface so the flush loop is unit-testable without a
|
||||
// real ClickHouse connection.
|
||||
type chWriter interface {
|
||||
WriteBatch(ctx context.Context, records []*logsv1.LogRecord) error
|
||||
}
|
||||
|
||||
// reader is the subset of *kafka.Reader used here, as an interface so the
|
||||
// flush/commit logic can be tested against a fake without a real broker.
|
||||
type reader interface {
|
||||
FetchMessage(ctx context.Context) (kafka.Message, error)
|
||||
CommitMessages(ctx context.Context, msgs ...kafka.Message) error
|
||||
Close() error
|
||||
}
|
||||
|
||||
type Consumer struct {
|
||||
logger *slog.Logger
|
||||
reader reader
|
||||
writer chWriter
|
||||
batchCfg config.BatchConfig
|
||||
}
|
||||
|
||||
func New(logger *slog.Logger, redpandaCfg config.RedpandaConfig, batchCfg config.BatchConfig, w chWriter) *Consumer {
|
||||
r := kafka.NewReader(kafka.ReaderConfig{
|
||||
Brokers: redpandaCfg.Brokers,
|
||||
Topic: redpandaCfg.Topic,
|
||||
GroupID: redpandaCfg.ConsumerGroup,
|
||||
})
|
||||
return &Consumer{logger: logger, reader: r, writer: w, batchCfg: batchCfg}
|
||||
}
|
||||
|
||||
func (c *Consumer) Run(ctx context.Context) error {
|
||||
defer c.reader.Close()
|
||||
|
||||
flushInterval := time.Duration(c.batchCfg.FlushIntervalMS) * time.Millisecond
|
||||
ticker := time.NewTicker(flushInterval)
|
||||
defer ticker.Stop()
|
||||
|
||||
msgCh := make(chan kafka.Message)
|
||||
fetchErrCh := make(chan error, 1)
|
||||
|
||||
go func() {
|
||||
for {
|
||||
m, err := c.reader.FetchMessage(ctx)
|
||||
if err != nil {
|
||||
fetchErrCh <- err
|
||||
return
|
||||
}
|
||||
select {
|
||||
case msgCh <- m:
|
||||
case <-ctx.Done():
|
||||
return
|
||||
}
|
||||
}
|
||||
}()
|
||||
|
||||
var records []*logsv1.LogRecord
|
||||
var pending []kafka.Message
|
||||
|
||||
flush := func() {
|
||||
if len(records) == 0 {
|
||||
return
|
||||
}
|
||||
if err := c.writer.WriteBatch(ctx, records); err != nil {
|
||||
c.logger.Error("clickhouse batch write failed, offsets not committed, will redeliver",
|
||||
"records", len(records), "error", err)
|
||||
} else if err := c.reader.CommitMessages(ctx, pending...); err != nil {
|
||||
c.logger.Error("committing offsets after clickhouse write", "error", err)
|
||||
} else {
|
||||
c.logger.Debug("batch flushed to clickhouse", "records", len(records))
|
||||
}
|
||||
records = records[:0]
|
||||
pending = pending[:0]
|
||||
}
|
||||
|
||||
for {
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
flush()
|
||||
return nil
|
||||
case err := <-fetchErrCh:
|
||||
flush()
|
||||
if ctx.Err() != nil {
|
||||
return nil
|
||||
}
|
||||
return err
|
||||
case <-ticker.C:
|
||||
flush()
|
||||
case m := <-msgCh:
|
||||
var rec logsv1.LogRecord
|
||||
if err := proto.Unmarshal(m.Value, &rec); err != nil {
|
||||
c.logger.Warn("skipping unparseable message", "error", err, "offset", m.Offset)
|
||||
if cerr := c.reader.CommitMessages(ctx, m); cerr != nil {
|
||||
c.logger.Error("committing offset for poison message", "error", cerr)
|
||||
}
|
||||
continue
|
||||
}
|
||||
records = append(records, &rec)
|
||||
pending = append(pending, m)
|
||||
if len(records) >= c.batchCfg.MaxSize {
|
||||
flush()
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,178 @@
|
||||
package consumer
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"io"
|
||||
"log/slog"
|
||||
"sync"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/segmentio/kafka-go"
|
||||
"google.golang.org/protobuf/proto"
|
||||
|
||||
"github.com/sentry/sentry/ingest/internal/config"
|
||||
logsv1 "github.com/sentry/sentry/proto/sentry/logs/v1"
|
||||
)
|
||||
|
||||
type fakeReader struct {
|
||||
msgs chan kafka.Message
|
||||
|
||||
mu sync.Mutex
|
||||
committed [][]kafka.Message
|
||||
}
|
||||
|
||||
func newFakeReader() *fakeReader {
|
||||
return &fakeReader{msgs: make(chan kafka.Message, 16)}
|
||||
}
|
||||
|
||||
func (f *fakeReader) push(m kafka.Message) { f.msgs <- m }
|
||||
|
||||
func (f *fakeReader) FetchMessage(ctx context.Context) (kafka.Message, error) {
|
||||
select {
|
||||
case m := <-f.msgs:
|
||||
return m, nil
|
||||
case <-ctx.Done():
|
||||
return kafka.Message{}, ctx.Err()
|
||||
}
|
||||
}
|
||||
|
||||
func (f *fakeReader) CommitMessages(_ context.Context, msgs ...kafka.Message) error {
|
||||
f.mu.Lock()
|
||||
defer f.mu.Unlock()
|
||||
f.committed = append(f.committed, msgs)
|
||||
return nil
|
||||
}
|
||||
|
||||
func (f *fakeReader) Close() error { return nil }
|
||||
|
||||
func (f *fakeReader) commitCount() int {
|
||||
f.mu.Lock()
|
||||
defer f.mu.Unlock()
|
||||
return len(f.committed)
|
||||
}
|
||||
|
||||
type fakeWriter struct {
|
||||
mu sync.Mutex
|
||||
batches [][]*logsv1.LogRecord
|
||||
failNext bool
|
||||
}
|
||||
|
||||
func (f *fakeWriter) WriteBatch(_ context.Context, records []*logsv1.LogRecord) error {
|
||||
f.mu.Lock()
|
||||
defer f.mu.Unlock()
|
||||
if f.failNext {
|
||||
f.failNext = false
|
||||
return errors.New("simulated clickhouse failure")
|
||||
}
|
||||
batch := make([]*logsv1.LogRecord, len(records))
|
||||
copy(batch, records)
|
||||
f.batches = append(f.batches, batch)
|
||||
return nil
|
||||
}
|
||||
|
||||
func (f *fakeWriter) batchCount() int {
|
||||
f.mu.Lock()
|
||||
defer f.mu.Unlock()
|
||||
return len(f.batches)
|
||||
}
|
||||
|
||||
func newTestConsumer(r reader, w chWriter, batchCfg config.BatchConfig) *Consumer {
|
||||
return &Consumer{
|
||||
logger: slog.New(slog.NewTextHandler(io.Discard, nil)),
|
||||
reader: r,
|
||||
writer: w,
|
||||
batchCfg: batchCfg,
|
||||
}
|
||||
}
|
||||
|
||||
func mustMarshal(t *testing.T, rec *logsv1.LogRecord) []byte {
|
||||
t.Helper()
|
||||
b, err := proto.Marshal(rec)
|
||||
if err != nil {
|
||||
t.Fatalf("marshal: %v", err)
|
||||
}
|
||||
return b
|
||||
}
|
||||
|
||||
func waitFor(t *testing.T, timeout time.Duration, cond func() bool) {
|
||||
t.Helper()
|
||||
deadline := time.Now().Add(timeout)
|
||||
for time.Now().Before(deadline) {
|
||||
if cond() {
|
||||
return
|
||||
}
|
||||
time.Sleep(5 * time.Millisecond)
|
||||
}
|
||||
t.Fatal("condition not met before timeout")
|
||||
}
|
||||
|
||||
func TestConsumerFlushesOnBatchSize(t *testing.T) {
|
||||
fr := newFakeReader()
|
||||
fw := &fakeWriter{}
|
||||
c := newTestConsumer(fr, fw, config.BatchConfig{MaxSize: 2, FlushIntervalMS: 60_000})
|
||||
|
||||
ctx, cancel := context.WithCancel(context.Background())
|
||||
defer cancel()
|
||||
done := make(chan error, 1)
|
||||
go func() { done <- c.Run(ctx) }()
|
||||
|
||||
fr.push(kafka.Message{Value: mustMarshal(t, &logsv1.LogRecord{Message: "a"})})
|
||||
fr.push(kafka.Message{Value: mustMarshal(t, &logsv1.LogRecord{Message: "b"})})
|
||||
|
||||
waitFor(t, time.Second, func() bool { return fw.batchCount() == 1 })
|
||||
|
||||
fw.mu.Lock()
|
||||
if len(fw.batches[0]) != 2 {
|
||||
t.Fatalf("expected batch of 2 records, got %d", len(fw.batches[0]))
|
||||
}
|
||||
fw.mu.Unlock()
|
||||
|
||||
waitFor(t, time.Second, func() bool { return fr.commitCount() == 1 })
|
||||
}
|
||||
|
||||
func TestConsumerFlushesOnTimeout(t *testing.T) {
|
||||
fr := newFakeReader()
|
||||
fw := &fakeWriter{}
|
||||
c := newTestConsumer(fr, fw, config.BatchConfig{MaxSize: 1000, FlushIntervalMS: 20})
|
||||
|
||||
ctx, cancel := context.WithCancel(context.Background())
|
||||
defer cancel()
|
||||
done := make(chan error, 1)
|
||||
go func() { done <- c.Run(ctx) }()
|
||||
|
||||
fr.push(kafka.Message{Value: mustMarshal(t, &logsv1.LogRecord{Message: "only-one"})})
|
||||
|
||||
waitFor(t, time.Second, func() bool { return fw.batchCount() == 1 })
|
||||
|
||||
fw.mu.Lock()
|
||||
if len(fw.batches[0]) != 1 {
|
||||
t.Fatalf("expected batch of 1 record, got %d", len(fw.batches[0]))
|
||||
}
|
||||
fw.mu.Unlock()
|
||||
}
|
||||
|
||||
func TestConsumerDoesNotCommitOnWriteFailure(t *testing.T) {
|
||||
fr := newFakeReader()
|
||||
fw := &fakeWriter{failNext: true}
|
||||
c := newTestConsumer(fr, fw, config.BatchConfig{MaxSize: 1, FlushIntervalMS: 60_000})
|
||||
|
||||
ctx, cancel := context.WithCancel(context.Background())
|
||||
defer cancel()
|
||||
done := make(chan error, 1)
|
||||
go func() { done <- c.Run(ctx) }()
|
||||
|
||||
fr.push(kafka.Message{Value: mustMarshal(t, &logsv1.LogRecord{Message: "will-fail"})})
|
||||
|
||||
// Give the flush a moment to run and fail.
|
||||
time.Sleep(100 * time.Millisecond)
|
||||
|
||||
if got := fr.commitCount(); got != 0 {
|
||||
t.Fatalf("expected no commits after a failed clickhouse write, got %d", got)
|
||||
}
|
||||
// The batch was attempted even though writer returned an error.
|
||||
if fw.batchCount() != 0 {
|
||||
t.Fatalf("fakeWriter should not record a failed batch, got %d recorded", fw.batchCount())
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,96 @@
|
||||
// Package grpcserver implements the agent-facing side of ingest: an mTLS
|
||||
// gRPC server accepting LogIngest.PushBatch calls, which it forwards
|
||||
// unchanged (proto-encoded) onto Redpanda. Normalization into the
|
||||
// ClickHouse row shape happens later, on the consumer side.
|
||||
package grpcserver
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"log/slog"
|
||||
"net"
|
||||
|
||||
"github.com/segmentio/kafka-go"
|
||||
"google.golang.org/grpc"
|
||||
"google.golang.org/grpc/codes"
|
||||
"google.golang.org/grpc/credentials"
|
||||
"google.golang.org/grpc/status"
|
||||
"google.golang.org/protobuf/proto"
|
||||
|
||||
"github.com/sentry/sentry/ingest/internal/config"
|
||||
logsv1 "github.com/sentry/sentry/proto/sentry/logs/v1"
|
||||
)
|
||||
|
||||
type Server struct {
|
||||
logsv1.UnimplementedLogIngestServer
|
||||
|
||||
logger *slog.Logger
|
||||
grpcCfg config.GRPCConfig
|
||||
tlsCfg config.TLSConfig
|
||||
producer batchProducer
|
||||
}
|
||||
|
||||
// batchProducer is the subset of *producer.Producer this package depends
|
||||
// on, so tests can substitute a fake without touching Redpanda.
|
||||
type batchProducer interface {
|
||||
WriteBatch(ctx context.Context, msgs []kafka.Message) error
|
||||
}
|
||||
|
||||
func New(logger *slog.Logger, grpcCfg config.GRPCConfig, tlsCfg config.TLSConfig, p batchProducer) *Server {
|
||||
return &Server{logger: logger, grpcCfg: grpcCfg, tlsCfg: tlsCfg, producer: p}
|
||||
}
|
||||
|
||||
// Run blocks serving gRPC until ctx is canceled, then gracefully stops.
|
||||
func (s *Server) Run(ctx context.Context) error {
|
||||
tlsConf, err := loadServerTLSConfig(s.tlsCfg)
|
||||
if err != nil {
|
||||
return fmt.Errorf("loading TLS config: %w", err)
|
||||
}
|
||||
|
||||
lis, err := net.Listen("tcp", s.grpcCfg.ListenAddr)
|
||||
if err != nil {
|
||||
return fmt.Errorf("listening on %s: %w", s.grpcCfg.ListenAddr, err)
|
||||
}
|
||||
|
||||
grpcSrv := grpc.NewServer(grpc.Creds(credentials.NewTLS(tlsConf)))
|
||||
logsv1.RegisterLogIngestServer(grpcSrv, s)
|
||||
|
||||
s.logger.Info("gRPC server listening", "addr", s.grpcCfg.ListenAddr)
|
||||
|
||||
errCh := make(chan error, 1)
|
||||
go func() { errCh <- grpcSrv.Serve(lis) }()
|
||||
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
grpcSrv.GracefulStop()
|
||||
return nil
|
||||
case err := <-errCh:
|
||||
return err
|
||||
}
|
||||
}
|
||||
|
||||
func (s *Server) PushBatch(ctx context.Context, req *logsv1.PushBatchRequest) (*logsv1.PushBatchResponse, error) {
|
||||
if len(req.GetRecords()) == 0 {
|
||||
return &logsv1.PushBatchResponse{Accepted: 0}, nil
|
||||
}
|
||||
|
||||
msgs := make([]kafka.Message, 0, len(req.GetRecords()))
|
||||
for _, rec := range req.GetRecords() {
|
||||
val, err := proto.Marshal(rec)
|
||||
if err != nil {
|
||||
return nil, status.Errorf(codes.InvalidArgument, "marshaling record: %v", err)
|
||||
}
|
||||
msgs = append(msgs, kafka.Message{
|
||||
Key: []byte(rec.GetHost()),
|
||||
Value: val,
|
||||
})
|
||||
}
|
||||
|
||||
if err := s.producer.WriteBatch(ctx, msgs); err != nil {
|
||||
s.logger.Error("failed to write batch to redpanda", "batch_id", req.GetBatchId(), "error", err)
|
||||
return nil, status.Errorf(codes.Unavailable, "writing to transport: %v", err)
|
||||
}
|
||||
|
||||
s.logger.Debug("batch produced to redpanda", "batch_id", req.GetBatchId(), "records", len(req.GetRecords()))
|
||||
return &logsv1.PushBatchResponse{Accepted: uint32(len(req.GetRecords()))}, nil
|
||||
}
|
||||
@@ -0,0 +1,36 @@
|
||||
package grpcserver
|
||||
|
||||
import (
|
||||
"crypto/tls"
|
||||
"crypto/x509"
|
||||
"fmt"
|
||||
"os"
|
||||
|
||||
"github.com/sentry/sentry/ingest/internal/config"
|
||||
)
|
||||
|
||||
// loadServerTLSConfig builds the mTLS server config: ingest's own
|
||||
// certificate, plus the CA used to verify agent client certificates.
|
||||
// Agents are never accepted without a client cert signed by this CA.
|
||||
func loadServerTLSConfig(cfg config.TLSConfig) (*tls.Config, error) {
|
||||
cert, err := tls.LoadX509KeyPair(cfg.CertFile, cfg.KeyFile)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("loading server cert/key: %w", err)
|
||||
}
|
||||
|
||||
caPEM, err := os.ReadFile(cfg.ClientCAFile)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("reading client CA file: %w", err)
|
||||
}
|
||||
caPool := x509.NewCertPool()
|
||||
if !caPool.AppendCertsFromPEM(caPEM) {
|
||||
return nil, fmt.Errorf("no valid certificates found in client CA file %s", cfg.ClientCAFile)
|
||||
}
|
||||
|
||||
return &tls.Config{
|
||||
Certificates: []tls.Certificate{cert},
|
||||
ClientAuth: tls.RequireAndVerifyClientCert,
|
||||
ClientCAs: caPool,
|
||||
MinVersion: tls.VersionTLS12,
|
||||
}, nil
|
||||
}
|
||||
@@ -0,0 +1,58 @@
|
||||
// Package normalize maps the wire-format LogRecord (as agents send it)
|
||||
// into the ClickHouse row shape defined in /storage. This is the "OTel-log-
|
||||
// like schema" normalization step called for in the ingest design — Phase
|
||||
// 0 keeps it to the minimal column set; full OTel field mapping (separate
|
||||
// SeverityNumber/SeverityText, resource attributes, etc.) is deferred, see
|
||||
// the open questions in /docs/architecture.md.
|
||||
package normalize
|
||||
|
||||
import (
|
||||
"time"
|
||||
|
||||
logsv1 "github.com/sentry/sentry/proto/sentry/logs/v1"
|
||||
)
|
||||
|
||||
type Row struct {
|
||||
Timestamp time.Time
|
||||
Host string
|
||||
Service string
|
||||
Severity string
|
||||
Message string
|
||||
Attributes map[string]string
|
||||
}
|
||||
|
||||
func ToRow(rec *logsv1.LogRecord) Row {
|
||||
attrs := rec.GetAttributes()
|
||||
if attrs == nil {
|
||||
attrs = map[string]string{}
|
||||
}
|
||||
return Row{
|
||||
Timestamp: time.Unix(0, rec.GetTimestampUnixNano()).UTC(),
|
||||
Host: rec.GetHost(),
|
||||
Service: rec.GetService(),
|
||||
Severity: severityText(rec.GetSeverity()),
|
||||
Message: rec.GetMessage(),
|
||||
Attributes: attrs,
|
||||
}
|
||||
}
|
||||
|
||||
// severityText maps the proto Severity enum to short OTel-style severity
|
||||
// names, stored as the `severity` column's value.
|
||||
func severityText(sev logsv1.Severity) string {
|
||||
switch sev {
|
||||
case logsv1.Severity_SEVERITY_TRACE:
|
||||
return "TRACE"
|
||||
case logsv1.Severity_SEVERITY_DEBUG:
|
||||
return "DEBUG"
|
||||
case logsv1.Severity_SEVERITY_INFO:
|
||||
return "INFO"
|
||||
case logsv1.Severity_SEVERITY_WARN:
|
||||
return "WARN"
|
||||
case logsv1.Severity_SEVERITY_ERROR:
|
||||
return "ERROR"
|
||||
case logsv1.Severity_SEVERITY_FATAL:
|
||||
return "FATAL"
|
||||
default:
|
||||
return "UNSPECIFIED"
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,68 @@
|
||||
package normalize
|
||||
|
||||
import (
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
logsv1 "github.com/sentry/sentry/proto/sentry/logs/v1"
|
||||
)
|
||||
|
||||
func TestToRowMapsFieldsAndSeverity(t *testing.T) {
|
||||
rec := &logsv1.LogRecord{
|
||||
TimestampUnixNano: time.Date(2026, 1, 2, 3, 4, 5, 0, time.UTC).UnixNano(),
|
||||
Host: "host-1",
|
||||
Service: "svc-a",
|
||||
Severity: logsv1.Severity_SEVERITY_ERROR,
|
||||
Message: "boom",
|
||||
Attributes: map[string]string{"k": "v"},
|
||||
}
|
||||
|
||||
row := ToRow(rec)
|
||||
|
||||
if row.Host != "host-1" || row.Service != "svc-a" || row.Message != "boom" {
|
||||
t.Fatalf("unexpected row: %+v", row)
|
||||
}
|
||||
if row.Severity != "ERROR" {
|
||||
t.Fatalf("expected severity ERROR, got %s", row.Severity)
|
||||
}
|
||||
if row.Attributes["k"] != "v" {
|
||||
t.Fatalf("expected attribute k=v, got %+v", row.Attributes)
|
||||
}
|
||||
if !row.Timestamp.Equal(time.Date(2026, 1, 2, 3, 4, 5, 0, time.UTC)) {
|
||||
t.Fatalf("unexpected timestamp: %v", row.Timestamp)
|
||||
}
|
||||
}
|
||||
|
||||
func TestToRowNilAttributesBecomesEmptyMap(t *testing.T) {
|
||||
rec := &logsv1.LogRecord{Host: "h", Service: "s", Message: "m"}
|
||||
row := ToRow(rec)
|
||||
if row.Attributes == nil {
|
||||
t.Fatal("expected non-nil empty map, got nil")
|
||||
}
|
||||
if len(row.Attributes) != 0 {
|
||||
t.Fatalf("expected empty map, got %+v", row.Attributes)
|
||||
}
|
||||
}
|
||||
|
||||
func TestSeverityTextCoversAllEnumValues(t *testing.T) {
|
||||
cases := map[logsv1.Severity]string{
|
||||
logsv1.Severity_SEVERITY_UNSPECIFIED: "UNSPECIFIED",
|
||||
logsv1.Severity_SEVERITY_TRACE: "TRACE",
|
||||
logsv1.Severity_SEVERITY_DEBUG: "DEBUG",
|
||||
logsv1.Severity_SEVERITY_INFO: "INFO",
|
||||
logsv1.Severity_SEVERITY_WARN: "WARN",
|
||||
logsv1.Severity_SEVERITY_ERROR: "ERROR",
|
||||
logsv1.Severity_SEVERITY_FATAL: "FATAL",
|
||||
}
|
||||
for sev, want := range cases {
|
||||
if got := severityText(sev); got != want {
|
||||
t.Errorf("severityText(%v) = %q, want %q", sev, got, want)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestSeverityTextUnknownValueFallsBackToUnspecified(t *testing.T) {
|
||||
if got := severityText(logsv1.Severity(99)); got != "UNSPECIFIED" {
|
||||
t.Fatalf("expected UNSPECIFIED for unknown severity, got %q", got)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,42 @@
|
||||
// Package producer wraps the Redpanda (Kafka API) producer used by the
|
||||
// gRPC front end to forward agent-submitted batches onto the transport
|
||||
// layer, unchanged. OTel-log-shape normalization happens later, on the
|
||||
// consumer side — see internal/normalize.
|
||||
package producer
|
||||
|
||||
import (
|
||||
"context"
|
||||
|
||||
"github.com/segmentio/kafka-go"
|
||||
|
||||
"github.com/sentry/sentry/ingest/internal/config"
|
||||
)
|
||||
|
||||
type Producer struct {
|
||||
writer *kafka.Writer
|
||||
}
|
||||
|
||||
func New(cfg config.RedpandaConfig) *Producer {
|
||||
return &Producer{
|
||||
writer: &kafka.Writer{
|
||||
Addr: kafka.TCP(cfg.Brokers...),
|
||||
Topic: cfg.Topic,
|
||||
// Partition by host so a single host's records stay in
|
||||
// relative order within a partition.
|
||||
Balancer: &kafka.Hash{},
|
||||
RequiredAcks: kafka.RequireOne,
|
||||
AllowAutoTopicCreation: false, // topics are provisioned explicitly, see /transport
|
||||
},
|
||||
}
|
||||
}
|
||||
|
||||
func (p *Producer) Close() error {
|
||||
return p.writer.Close()
|
||||
}
|
||||
|
||||
// WriteBatch writes all messages in one call. kafka-go's WriteMessages
|
||||
// either succeeds for the whole batch or returns an error, which matches
|
||||
// the PushBatch RPC's all-or-nothing contract for Phase 0.
|
||||
func (p *Producer) WriteBatch(ctx context.Context, msgs []kafka.Message) error {
|
||||
return p.writer.WriteMessages(ctx, msgs...)
|
||||
}
|
||||
Reference in New Issue
Block a user