Scaffold Phase 0: agent -> Redpanda -> ingest -> ClickHouse -> api -> web
End-to-end log pipeline for Linux hosts, per /docs/architecture.md: - proto: shared gRPC contract (agent <-> ingest), Go bindings checked in - agent: Rust, musl-targeted, journald/file sourcing, RFC5424 parser, mTLS gRPC client, no required config for the common case - ingest: Go, single binary with --mode server|consumer|all; gRPC front end forwards to Redpanda unchanged, consumer normalizes and batch-writes to ClickHouse with at-least-once delivery - storage: ClickHouse schema + a plain SQL-file migration runner - api: minimal SELECT-only query endpoint, plain REST (not gRPC+gateway yet -- see api/README.md) - web: SvelteKit static SPA, one query page - transport: Redpanda compose + topic provisioning - cli: sentryctl ping stub - hack/dev-certs: throwaway CA + cert generation for local mTLS - root docker-compose.yml + docs/phase-0-runbook.md tie it together Not yet run end-to-end against real Docker/ClickHouse/Redpanda -- see the runbook's caveats section before relying on this working as-is.
This commit is contained in:
@@ -0,0 +1,42 @@
|
||||
// Package producer wraps the Redpanda (Kafka API) producer used by the
|
||||
// gRPC front end to forward agent-submitted batches onto the transport
|
||||
// layer, unchanged. OTel-log-shape normalization happens later, on the
|
||||
// consumer side — see internal/normalize.
|
||||
package producer
|
||||
|
||||
import (
|
||||
"context"
|
||||
|
||||
"github.com/segmentio/kafka-go"
|
||||
|
||||
"github.com/sentry/sentry/ingest/internal/config"
|
||||
)
|
||||
|
||||
type Producer struct {
|
||||
writer *kafka.Writer
|
||||
}
|
||||
|
||||
func New(cfg config.RedpandaConfig) *Producer {
|
||||
return &Producer{
|
||||
writer: &kafka.Writer{
|
||||
Addr: kafka.TCP(cfg.Brokers...),
|
||||
Topic: cfg.Topic,
|
||||
// Partition by host so a single host's records stay in
|
||||
// relative order within a partition.
|
||||
Balancer: &kafka.Hash{},
|
||||
RequiredAcks: kafka.RequireOne,
|
||||
AllowAutoTopicCreation: false, // topics are provisioned explicitly, see /transport
|
||||
},
|
||||
}
|
||||
}
|
||||
|
||||
func (p *Producer) Close() error {
|
||||
return p.writer.Close()
|
||||
}
|
||||
|
||||
// WriteBatch writes all messages in one call. kafka-go's WriteMessages
|
||||
// either succeeds for the whole batch or returns an error, which matches
|
||||
// the PushBatch RPC's all-or-nothing contract for Phase 0.
|
||||
func (p *Producer) WriteBatch(ctx context.Context, msgs []kafka.Message) error {
|
||||
return p.writer.WriteMessages(ctx, msgs...)
|
||||
}
|
||||
Reference in New Issue
Block a user