Build per-tenant ClickHouse write-routing for ingest (Tantivy still deferred)

ingest tags every record with a tenant_id Kafka header (built previously),
but nothing consumed it to actually route the write. This closes that for
ClickHouse: enterprise/cmd/enterprise-ingest (a second binary, mirroring
enterprise-api) reuses ingest/consumer's own flush loop unchanged, with
enterprise/internal/chwriter.Registry -- a per-tenant clickhousewriter.Writer
registry -- swapped in as the writer. A batch pulled from the single shared
Redpanda topic can mix records from many tenants, so WriteBatch groups by
TenantID and dispatches each group to its own tenant's connection, fail-
closed on an empty or unrecognized tenant_id.

ingest/consumer and ingest/clickhousewriter move out of internal/ (same
reason api/internal/* moved earlier this phase: enterprise/ can't import
anything under another module's internal/). Their New() constructors now
take small local Config structs instead of ingest/internal/config types,
so enterprise/ doesn't need that import either.

Building this surfaced a real bug: tenantprovision.ProvisionClickHouse
only granted SELECT on a tenant's ClickHouse user, correct for chrunner's
read-only use but not enough for chwriter reusing the same credential to
write -- every real per-tenant write would have failed closed with a
permission error. Fixed by widening the grant to SELECT, INSERT; no
cross-tenant boundary is crossed by also allowing INSERT within a
tenant's own database.

Helm gates enterprise-ingest's Deployment on the same
ingest.requireTenantCredential flag that already gates tag validation --
write-routing is meaningless without tagging already being required, so
they're one decision, not two. docker-compose.yml's version is a
disclosed, weaker approximation: it can't achieve Helm's genuine
-mode=server/-mode=consumer split, so with the enterprise profile active
both ingest and enterprise-ingest independently consume every message
via different consumer groups -- harmless duplication for local
verification only.

Not built: Tantivy's independent Redpanda consumer (search/src/consumer.rs)
still doesn't read the tenant_id header at all -- every record still lands
in the one shared index regardless of tenant. Not run: the live-ClickHouse-
gated tests (chwriter's cross-tenant routing test, tenantprovision's INSERT
regression test) -- no Docker/database access in this environment; they're
correct Go that has never executed, disclosed as such in docs/security/
threat-model.md and docs/phase-4-runbook.md §14.
This commit is contained in:
2026-08-14 19:26:09 -07:00
parent 17fdc212c2
commit 1de77b969f
26 changed files with 1355 additions and 267 deletions
+11 -1
View File
@@ -9,6 +9,12 @@ go 1.25.0
// the package that defines it.
replace github.com/sentry/sentry/api => ../api
// Same allowed direction, against ingest/ instead -- enterprise/internal/
// chwriter implements ingest/consumer's chWriter interface, which
// structurally requires importing the package that defines it (see
// that package's doc comment).
replace github.com/sentry/sentry/ingest => ../ingest
// api's own go.mod replace directive for proto/ is module-local and
// doesn't propagate here -- enterprise/ needs its own, or `go build`
// tries to fetch github.com/sentry/sentry/proto from a real (nonexistent)
@@ -16,7 +22,10 @@ replace github.com/sentry/sentry/api => ../api
// the generated search gRPC stubs.
replace github.com/sentry/sentry/proto => ../proto
require github.com/sentry/sentry/api v0.0.0-00010101000000-000000000000
require (
github.com/sentry/sentry/api v0.0.0-00010101000000-000000000000
github.com/sentry/sentry/ingest v0.0.0-00010101000000-000000000000
)
require (
github.com/ClickHouse/clickhouse-go/v2 v2.48.0
@@ -71,6 +80,7 @@ require (
github.com/pkg/errors v0.9.1 // indirect
github.com/russellhaering/goxmldsig v1.4.0 // indirect
github.com/segmentio/asm v1.2.1 // indirect
github.com/segmentio/kafka-go v0.4.51 // indirect
github.com/shopspring/decimal v1.4.0 // indirect
github.com/spf13/pflag v1.0.5 // indirect
github.com/x448/float16 v0.8.4 // indirect