Build per-tenant ClickHouse write-routing for ingest (Tantivy still deferred)

ingest tags every record with a tenant_id Kafka header (built previously), but nothing consumed it to actually route the write. This closes that for ClickHouse: enterprise/cmd/enterprise-ingest (a second binary, mirroring enterprise-api) reuses ingest/consumer's own flush loop unchanged, with enterprise/internal/chwriter.Registry -- a per-tenant clickhousewriter.Writer registry -- swapped in as the writer. A batch pulled from the single shared Redpanda topic can mix records from many tenants, so WriteBatch groups by TenantID and dispatches each group to its own tenant's connection, fail- closed on an empty or unrecognized tenant_id. ingest/consumer and ingest/clickhousewriter move out of internal/ (same reason api/internal/* moved earlier this phase: enterprise/ can't import anything under another module's internal/). Their New() constructors now take small local Config structs instead of ingest/internal/config types, so enterprise/ doesn't need that import either. Building this surfaced a real bug: tenantprovision.ProvisionClickHouse only granted SELECT on a tenant's ClickHouse user, correct for chrunner's read-only use but not enough for chwriter reusing the same credential to write -- every real per-tenant write would have failed closed with a permission error. Fixed by widening the grant to SELECT, INSERT; no cross-tenant boundary is crossed by also allowing INSERT within a tenant's own database. Helm gates enterprise-ingest's Deployment on the same ingest.requireTenantCredential flag that already gates tag validation -- write-routing is meaningless without tagging already being required, so they're one decision, not two. docker-compose.yml's version is a disclosed, weaker approximation: it can't achieve Helm's genuine -mode=server/-mode=consumer split, so with the enterprise profile active both ingest and enterprise-ingest independently consume every message via different consumer groups -- harmless duplication for local verification only. Not built: Tantivy's independent Redpanda consumer (search/src/consumer.rs) still doesn't read the tenant_id header at all -- every record still lands in the one shared index regardless of tenant. Not run: the live-ClickHouse- gated tests (chwriter's cross-tenant routing test, tenantprovision's INSERT regression test) -- no Docker/database access in this environment; they're correct Go that has never executed, disclosed as such in docs/security/ threat-model.md and docs/phase-4-runbook.md §14.
2026-08-14 19:26:09 -07:00
parent 17fdc212c2
commit 1de77b969f
26 changed files with 1355 additions and 267 deletions
@@ -182,18 +182,14 @@ silently left out:
  a cross-origin `fetch` with credentials from `web`'s origin needs
  it), neither of which is verifiable in this environment without a
  live backend and a browser session to exercise.
- **Ingest write-routing, for either storage engine** -- identity is now
-  real (see "Ingest tenant identity" below), but nothing consumes it
-  yet: `chrunner`/`searchclient` prove read isolation given tenant-
-  scoped data exists, and every record `ingest` produces is now tagged
-  with a real tenant ID, but neither `ingest/internal/consumer` (the
-  ClickHouse writer) nor `search/src/consumer.rs` (a completely
-  independent Redpanda consumer) reads that tag back to route the write
-  anywhere per-tenant. Every record still lands in the single shared
-  ClickHouse database and the single shared Tantivy index regardless of
-  tenant. A newly-provisioned tenant's storage is real and isolated, and
-  permanently empty. Now scoped, disclosed remaining work, not an
-  undesigned gap -- see `/docs/security/threat-model.md`.
+- **Ingest write-routing, for Tantivy** -- `search/src/consumer.rs` (a
+  completely independent Redpanda consumer, not called through `ingest`
+  or `enterprise-ingest` at all) still writes every record into the one
+  shared (default) Tantivy index, regardless of tenant. The ClickHouse
+  half is now built (see "Ingest write-routing" below); Tantivy's is
+  real, disclosed, separate follow-up work -- a different codebase
+  (Rust) and a different consumer process, not just "the same fix
+  applied twice."

 Deployment-topology routing (does traffic actually reach `enterprise-api`
 instead of `api`) is no longer deferred -- both `deploy/helm/sentry` and
@@ -226,17 +222,67 @@ import (`ingest` is AGPL core), same "network boundary, not import
 boundary" shape `api/authz.HTTPAuthorizer` already uses for the query
 path.

-**What this does not do**: change where a record is actually written.
-See "Deliberately deferred" above -- attaching a verified tenant
-identity as early as possible (right where the credential is presented)
-was built as a self-contained first step; per-tenant write-routing for
-both storage engines is separate, scoped follow-up work.
+## Ingest write-routing (ClickHouse)
+
+`enterprise/cmd/enterprise-ingest` is `ingest -mode=consumer`'s multi-
+tenant alternative -- same "second binary" shape as `enterprise-api`
+next to `api/cmd/api` (AGPL core must never import `enterprise/`, so the
+tenant-aware wiring has to live in a binary that imports *into* core,
+not the reverse). It reuses `ingest/consumer.Consumer`'s exact flush
+loop unchanged, swapping in `enterprise/internal/chwriter.Registry` --
+chrunner's write-side counterpart -- as the writer: one fully separate
+`*ingest/clickhousewriter.Writer` (and the `driver.Conn` under it) per
+tenant, built once at startup from `rbacstore.
+ListProvisionedDataSources` (the same source of truth `chrunner` already
+uses for reads). `WriteBatch` groups a Kafka batch's records by their
+`tenant_id` tag and writes each tenant's group through its own
+dedicated connection, refusing the whole call (matching `ingest/
+consumer`'s existing all-or-nothing batch contract -- no offsets commit,
+the batch redelivers) if any record is untagged or tagged with a tenant
+that isn't provisioned. `ingest/consumer` and `ingest/clickhousewriter`
+moved out of `internal/` for this -- same Go compiler-enforced
+visibility reasoning as every other package this phase moved out of
+`internal/` for a cross-module import (see `ingest/README.md`'s "Multi-
+tenant write-routing" section).
+
+A real bug was found and fixed while wiring this up:
+`tenantprovision.ProvisionClickHouse` originally granted a tenant's
+ClickHouse user `SELECT` only -- correct for `chrunner`'s query path,
+but it would have made every real per-tenant write from `chwriter` fail
+with a permission error, since it's the *same* credential used for
+both. Fixed by granting `SELECT, INSERT` (not a second, separate
+write-only credential -- there's no cross-tenant boundary crossed by
+also granting INSERT within a tenant's own database, so one credential
+for both directions is the simpler, still-correctly-scoped choice).
+
+A real multi-tenant deployment runs `ingest -mode=server` (agent-facing,
+tags records, unchanged) alongside `enterprise-ingest` (consumer,
+per-tenant writes) *instead of* `ingest -mode=consumer` -- see `deploy/
+helm/sentry`'s `ingest.requireTenantCredential` value (gates both the
+credential-validation requirement and this mode split together, since
+write-routing is only meaningful once records actually carry a
+tenant_id to route on) and `docker-compose.yml`'s `enterprise-ingest`
+service (a simpler opt-in there -- true `-mode=server`/`-mode=consumer`
+exclusivity isn't wired in compose, a disclosed local-dev-only gap; see
+that service's own comment).
+
+Verified: `enterprise/internal/chwriter`'s fail-closed paths (empty/
+unknown `tenant_id`) run genuinely without Docker (constructing a
+`Registry` directly, bypassing `New`, which is the only part that would
+dial ClickHouse); the actual per-tenant write-isolation probe
+(`TestRegistryWritesEachTenantToItsOwnDatabase`) and the
+`tenantprovision` INSERT-grant regression test are real integration
+tests against a live ClickHouse, same `CHWRITER_TEST_CLICKHOUSE_ADDR`/
+`TENANTPROVISION_TEST_CLICKHOUSE_ADDR` convention as every other
+ClickHouse-backed test this phase -- not run against a live database in
+this environment.

 ## Package layout

 ```
 cmd/enterprise-auth/   config loading, OIDC discovery at startup, health/authorize/features/authorize-ingest endpoints, -mint-service-token, -create-tenant, -grant-membership-*, -revoke-membership-*, -list-memberships-tenant, -create-ingest-credential-tenant, -list-ingest-credentials-tenant, -revoke-ingest-credential
 cmd/enterprise-api/     multi-tenant-aware alternative to api/cmd/api -- see its own doc comment
+cmd/enterprise-ingest/   multi-tenant-aware alternative to ingest -mode=consumer -- see its own doc comment
 internal/tenant/        the ID type -- see its package doc comment before touching it
 internal/oidc/           coreos/go-oidc wiring: discovery, login redirect, code exchange + ID token verification
 internal/saml/            crewjam/saml wiring: SP setup, login redirect, response parsing/validation
@@ -247,10 +293,12 @@ internal/rbacstore/          users/tenants/tenant_memberships/data_sources/dashb
 internal/tenantprovision/     real ClickHouse CREATE DATABASE/USER/GRANT
 internal/tenantcrd/            syncs -provision-tenant's real result into deploy/operator's Tenant CRD (K8s dynamic client, no cluster needed to test)
 internal/chrunner/             tenant-scoped api/querylang/executor.SQLRunner
+internal/chwriter/              tenant-scoped ingest/consumer.chWriter -- chrunner's write-side counterpart
 internal/searchclient/          tenant-scoped api/querylang/executor.SearchClient
 internal/audit/            append-only, hash-chained query audit log, plus the
                            api/queryapi.AuditLogger adapter (queryapi_adapter.go)
 internal/apiconfig/       enterprise-api's own env-var config
+internal/ingestconfig/     enterprise-ingest's own env-var config
 internal/config/          enterprise-auth's env-var config
 ```

@@ -0,0 +1,14 @@
+# Same shape as every other Go service's Dockerfile in this repo --
+# context must be the repo root (needs ingest/, proto/, and enterprise/,
+# like enterprise-api/Dockerfile does for api/ + proto/ + enterprise/),
+# not enterprise/ alone.
+#   docker build -f enterprise/cmd/enterprise-ingest/Dockerfile -t sentry-enterprise-ingest .
+FROM golang:1.25-alpine AS builder
+WORKDIR /src
+COPY . .
+WORKDIR /src/enterprise
+RUN CGO_ENABLED=0 GOOS=linux go build -o /out/enterprise-ingest ./cmd/enterprise-ingest
+
+FROM gcr.io/distroless/static-debian12
+COPY --from=builder /out/enterprise-ingest /enterprise-ingest
+ENTRYPOINT ["/enterprise-ingest"]
@@ -0,0 +1,159 @@
+// Command enterprise-ingest is the multi-tenant-aware alternative to
+// running `ingest -mode=consumer` -- reads the same shared
+// sentry.logs.raw Redpanda topic ingest/cmd/ingest's agent-facing
+// server half (PushBatch) produces onto (see that binary's doc
+// comment), but writes each record into its own tenant's dedicated
+// ClickHouse database (enterprise/internal/chwriter) instead of the one
+// shared table `ingest -mode=consumer` always writes to.
+//
+// Why a second binary, not a flag on ingest/cmd/ingest: ingest is AGPL
+// core and must never import enterprise/ (hack/check-tenant-boundary.sh
+// enforces this) -- there is no way for ingest's own binary to
+// construct an enterprise-supplied chwriter.Registry (which needs
+// rbacstore's per-tenant ClickHouse credentials) without that import.
+// enterprise/ importing ingest/ is the allowed direction, so this
+// binary lives here instead, reusing ingest/consumer.Consumer's own
+// flush loop unchanged with a tenant-aware writer swapped in -- the
+// exact same "second binary" shape as enterprise/cmd/enterprise-api
+// next to api/cmd/api.
+//
+// A real multi-tenant deployment runs this binary INSTEAD OF (not
+// alongside) `ingest -mode=consumer` -- `ingest -mode=server` (the
+// agent-facing half, which tags records with a tenant_id via
+// TenantResolver) keeps running unchanged and unconditionally either
+// way; only which process consumes sentry.logs.raw and where it writes
+// changes.
+package main
+
+import (
+	"context"
+	"fmt"
+	"log/slog"
+	"net/http"
+	"os"
+	"os/signal"
+	"strings"
+	"syscall"
+	"time"
+
+	"github.com/jackc/pgx/v5/pgxpool"
+	"golang.org/x/sync/errgroup"
+
+	"github.com/sentry/sentry/enterprise/internal/chwriter"
+	"github.com/sentry/sentry/enterprise/internal/ingestconfig"
+	"github.com/sentry/sentry/enterprise/internal/rbacstore"
+	"github.com/sentry/sentry/ingest/consumer"
+)
+
+func main() {
+	logger := slog.New(slog.NewJSONHandler(os.Stdout, nil))
+
+	cfg, err := ingestconfig.Load()
+	if err != nil {
+		logger.Error("loading config", "error", err)
+		os.Exit(1)
+	}
+
+	if len(os.Args) > 1 && os.Args[1] == "-healthcheck" {
+		os.Exit(runHealthcheck(cfg.HTTPListenAddr))
+	}
+
+	ctx, stop := signal.NotifyContext(context.Background(), syscall.SIGINT, syscall.SIGTERM)
+	defer stop()
+
+	pgDSN := fmt.Sprintf("postgres://%s:%s@%s/%s", cfg.Postgres.Username, cfg.Postgres.Password, cfg.Postgres.Addr, cfg.Postgres.Database)
+	pgPool, err := pgxpool.New(ctx, pgDSN)
+	if err != nil {
+		logger.Error("opening postgres pool", "error", err)
+		os.Exit(1)
+	}
+	defer pgPool.Close()
+	if err := pgPool.Ping(ctx); err != nil {
+		logger.Error("pinging postgres", "error", err)
+		os.Exit(1)
+	}
+	rbac := rbacstore.NewStore(pgPool)
+
+	// Same source of truth chrunner.Registry (the read side) already
+	// uses -- active+credentialed tenants only, see
+	// rbacstore.ListProvisionedDataSources's doc comment. A tenant
+	// that's mid-provisioning simply has no writer in the registry
+	// below, so chwriter.Registry.WriteBatch refuses it the same way
+	// chrunner.Registry.RunSQL already refuses an unprovisioned tenant
+	// on the read side.
+	sources, err := rbac.ListProvisionedDataSources(ctx)
+	if err != nil {
+		logger.Error("listing provisioned data sources", "error", err)
+		os.Exit(1)
+	}
+	chwSources := make([]chwriter.DataSource, 0, len(sources))
+	for _, s := range sources {
+		if s.ClickHouseUsername == nil || s.ClickHousePassword == nil {
+			continue // ListProvisionedDataSources already filters these out; defensive only.
+		}
+		chwSources = append(chwSources, chwriter.DataSource{
+			TenantID: s.TenantID, Database: s.ClickHouseDatabaseName,
+			Username: *s.ClickHouseUsername, Password: *s.ClickHousePassword,
+		})
+	}
+	logger.Info("loaded tenant data sources", "count", len(chwSources))
+
+	registry, err := chwriter.New(ctx, cfg.ClickHouseAddr, chwSources)
+	if err != nil {
+		logger.Error("building tenant write registry", "error", err)
+		os.Exit(1)
+	}
+	defer registry.Close()
+
+	c := consumer.New(logger, consumer.Config{
+		Brokers: cfg.Redpanda.Brokers, Topic: cfg.Redpanda.Topic, ConsumerGroup: cfg.Redpanda.ConsumerGroup,
+		BatchMaxSize: cfg.Batch.MaxSize, FlushIntervalMS: cfg.Batch.FlushIntervalMS,
+	}, registry)
+
+	mux := http.NewServeMux()
+	mux.HandleFunc("GET /healthz", func(w http.ResponseWriter, _ *http.Request) { w.WriteHeader(http.StatusOK) })
+	srv := &http.Server{Addr: cfg.HTTPListenAddr, Handler: mux}
+
+	g, ctx := errgroup.WithContext(ctx)
+	g.Go(func() error { return c.Run(ctx) })
+	g.Go(func() error {
+		logger.Info("enterprise-ingest healthz listening", "addr", cfg.HTTPListenAddr)
+		if err := srv.ListenAndServe(); err != nil && err != http.ErrServerClosed {
+			return err
+		}
+		return nil
+	})
+	g.Go(func() error {
+		<-ctx.Done()
+		shutdownCtx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
+		defer cancel()
+		return srv.Shutdown(shutdownCtx)
+	})
+
+	logger.Info("enterprise-ingest started")
+	if err := g.Wait(); err != nil {
+		logger.Error("enterprise-ingest exited with error", "error", err)
+		os.Exit(1)
+	}
+}
+
+// runHealthcheck mirrors every other binary in this repo's
+// -healthcheck self-check mode -- execs the binary against itself
+// rather than using an external tool (see e.g. api/cmd/api/main.go's
+// runHealthcheck doc comment).
+func runHealthcheck(listenAddr string) int {
+	addr := listenAddr
+	if strings.HasPrefix(addr, ":") {
+		addr = "localhost" + addr
+	}
+	client := http.Client{Timeout: 3 * time.Second}
+	resp, err := client.Get("http://" + addr + "/healthz")
+	if err != nil {
+		return 1
+	}
+	defer resp.Body.Close()
+	if resp.StatusCode != http.StatusOK {
+		return 1
+	}
+	return 0
+}
@@ -9,6 +9,12 @@ go 1.25.0
 // the package that defines it.
 replace github.com/sentry/sentry/api => ../api

+// Same allowed direction, against ingest/ instead -- enterprise/internal/
+// chwriter implements ingest/consumer's chWriter interface, which
+// structurally requires importing the package that defines it (see
+// that package's doc comment).
+replace github.com/sentry/sentry/ingest => ../ingest
+
 // api's own go.mod replace directive for proto/ is module-local and
 // doesn't propagate here -- enterprise/ needs its own, or `go build`
 // tries to fetch github.com/sentry/sentry/proto from a real (nonexistent)
@@ -16,7 +22,10 @@ replace github.com/sentry/sentry/api => ../api
 // the generated search gRPC stubs.
 replace github.com/sentry/sentry/proto => ../proto

-require github.com/sentry/sentry/api v0.0.0-00010101000000-000000000000
+require (
+	github.com/sentry/sentry/api v0.0.0-00010101000000-000000000000
+	github.com/sentry/sentry/ingest v0.0.0-00010101000000-000000000000
+)

 require (
 	github.com/ClickHouse/clickhouse-go/v2 v2.48.0
@@ -71,6 +80,7 @@ require (
 	github.com/pkg/errors v0.9.1 // indirect
 	github.com/russellhaering/goxmldsig v1.4.0 // indirect
 	github.com/segmentio/asm v1.2.1 // indirect
+	github.com/segmentio/kafka-go v0.4.51 // indirect
 	github.com/shopspring/decimal v1.4.0 // indirect
 	github.com/spf13/pflag v1.0.5 // indirect
 	github.com/x448/float16 v0.8.4 // indirect
@@ -121,6 +121,8 @@ github.com/russellhaering/goxmldsig v1.4.0 h1:8UcDh/xGyQiyrW+Fq5t8f+l2DLB1+zlhYz
 github.com/russellhaering/goxmldsig v1.4.0/go.mod h1:gM4MDENBQf7M+V824SGfyIUVFWydB7n0KkEubVJl+Tw=
 github.com/segmentio/asm v1.2.1 h1:DTNbBqs57ioxAD4PrArqftgypG4/qNpXoJx8TVXxPR0=
 github.com/segmentio/asm v1.2.1/go.mod h1:BqMnlJP91P8d+4ibuonYZw9mfnzI9HfxselHZr5aAcs=
+github.com/segmentio/kafka-go v0.4.51 h1:JgDPPG75tC1rWIS2Me6MwcvXJ6f49UQ4HjAOef71Hno=
+github.com/segmentio/kafka-go v0.4.51/go.mod h1:Y1gn60kzLEEaW28YshXyk2+VCUKbJ3Qr6DrnT3i4+9E=
 github.com/shopspring/decimal v1.4.0 h1:bxl37RwXBklmTi0C79JfXCEBD1cqqHt0bbgBAGFp81k=
 github.com/shopspring/decimal v1.4.0/go.mod h1:gawqmDU56v4yIKSwfBSFip1HdCCXN8/+DMd9qYNcwME=
 github.com/spf13/pflag v1.0.5 h1:iy+VFUOCP1a+8yFto/drg2CJ5u0yRoB7fZw3DKv/JXA=
@@ -138,6 +140,12 @@ github.com/stretchr/testify v1.11.1 h1:7s2iGBzp5EwR7/aIZr8ao5+dra3wiQyKjjFuvgVKu
 github.com/stretchr/testify v1.11.1/go.mod h1:wZwfW3scLgRK+23gO65QZefKpKQRnfz6sD981Nm4B6U=
 github.com/x448/float16 v0.8.4 h1:qLwI1I70+NjRFUR3zs1JPUCgaCXSh3SW62uAKT1mSBM=
 github.com/x448/float16 v0.8.4/go.mod h1:14CWIYCyZA/cWjXOioeEpHeN/83MdbZDRQHoFcYsOfg=
+github.com/xdg-go/pbkdf2 v1.0.0 h1:Su7DPu48wXMwC3bs7MCNG+z4FhcyEuz5dlvchbq0B0c=
+github.com/xdg-go/pbkdf2 v1.0.0/go.mod h1:jrpuAogTd400dnrH08LKmI/xc1MbPOebTwRqcT5RDeI=
+github.com/xdg-go/scram v1.1.2 h1:FHX5I5B4i4hKRVRBCFRxq1iQRej7WO3hhBuJf+UUySY=
+github.com/xdg-go/scram v1.1.2/go.mod h1:RT/sEzTbU5y00aCK8UOx6R7YryM0iF1N2MOmC3kKLN4=
+github.com/xdg-go/stringprep v1.0.4 h1:XLI/Ng3O1Atzq0oBs3TWm+5ZVgkq2aqdlvP9JtoZ6c8=
+github.com/xdg-go/stringprep v1.0.4/go.mod h1:mPGuuIYwz7CmR2bT9j4GbQqutWS1zV24gijq1dTyGkM=
 github.com/xyproto/randomstring v1.0.5 h1:YtlWPoRdgMu3NZtP45drfy1GKoojuR7hmRcnhZqKjWU=
 github.com/xyproto/randomstring v1.0.5/go.mod h1:rgmS5DeNXLivK7YprL0pY+lTuhNQW3iGxZ18UQApw/E=
 github.com/yuin/goldmark v1.1.27/go.mod h1:3hX8gzYuyVAZsxl0MRgGTJEmQBFcNTphYh9decYSb74=
@@ -0,0 +1,125 @@
+// Package chwriter is enterprise/internal/chrunner's write-side
+// counterpart -- the tenant-scoped implementation ingest/consumer.
+// Consumer needs to route each batch's records into their own tenant's
+// dedicated ClickHouse database, instead of the one shared table
+// ingest/cmd/ingest's single-tenant mode always writes to. Requires
+// importing ingest/clickhousewriter and ingest/consumer directly (see
+// enterprise/go.mod's replace directive) -- same allowed
+// "enterprise -> core" import direction chrunner uses for
+// api/querylang/executor, just against a different core module.
+//
+// Design mirrors chrunner.Registry closely: one fully separate
+// *clickhousewriter.Writer (and the driver.Conn under it) per tenant,
+// built once at construction from an immutable map -- never a shared
+// pool with session-level USE, for the same concurrency reasons
+// chrunner's doc comment explains. The one real difference:
+// chrunner.RunSQL resolves exactly one tenant per call from ctx (a
+// single request always belongs to one identity); WriteBatch resolves
+// per *record*, since one Kafka batch pulled off the shared
+// sentry.logs.raw topic can freely mix records from many different
+// tenants -- see ingest/internal/grpcserver's doc comment for why
+// there's one shared topic, not topic-per-tenant.
+package chwriter
+
+import (
+	"context"
+	"fmt"
+
+	"github.com/sentry/sentry/ingest/clickhousewriter"
+	"github.com/sentry/sentry/ingest/consumer"
+	logsv1 "github.com/sentry/sentry/proto/sentry/logs/v1"
+)
+
+// DataSource mirrors chrunner.DataSource -- deliberately not
+// enterprise/internal/rbacstore.DataSource itself, so this package
+// doesn't need to import rbacstore just to describe "an address and a
+// credential." Callers (enterprise-ingest's main.go) adapt rbacstore
+// rows into this.
+type DataSource struct {
+	TenantID string
+	Database string
+	Username string
+	Password string
+}
+
+// Registry implements ingest/consumer's chWriter interface
+// (WriteBatch(ctx, []consumer.Record) error) by routing each record to
+// its tenant's dedicated connection. Immutable after New returns -- see
+// this file's doc comment.
+type Registry struct {
+	writers map[string]*clickhousewriter.Writer
+	closers []func()
+}
+
+// New opens one real ClickHouse connection per DataSource (same native
+// address for all of them, different per-tenant credentials -- tenants
+// sharing a physical ClickHouse server today, same as chrunner). Fails
+// closed: if any one tenant's connection can't be opened, the whole
+// Registry fails to construct rather than silently running with a
+// partial tenant set.
+func New(ctx context.Context, addr string, sources []DataSource) (*Registry, error) {
+	reg := &Registry{writers: make(map[string]*clickhousewriter.Writer, len(sources))}
+	for _, src := range sources {
+		w, err := clickhousewriter.New(ctx, clickhousewriter.Config{
+			Addr: addr, Database: src.Database, Username: src.Username, Password: src.Password,
+		})
+		if err != nil {
+			reg.Close()
+			return nil, fmt.Errorf("chwriter: opening connection for tenant %q: %w", src.TenantID, err)
+		}
+		reg.writers[src.TenantID] = w
+		reg.closers = append(reg.closers, func() { _ = w.Close() })
+	}
+	return reg, nil
+}
+
+// Close releases every underlying connection -- call once at process
+// shutdown, same lifecycle as chrunner.Registry.Close.
+func (r *Registry) Close() {
+	for _, c := range r.closers {
+		c()
+	}
+}
+
+// WriteBatch implements ingest/consumer's chWriter interface. Groups
+// records by TenantID and writes each tenant's group through its own
+// dedicated connection -- fails the *whole* call (matching
+// ingest/consumer's existing all-or-nothing batch contract: a failed
+// WriteBatch means no offsets are committed and the entire batch is
+// redelivered, never partial credit) if any record's tenant is empty
+// (no TenantResolver was configured for the PushBatch call that
+// produced it -- a multi-tenant deployment must never silently write an
+// untagged record somewhere) or unrecognized (not yet provisioned, or
+// provisioning failed). Fail closed, same reasoning
+// chrunner.Registry.RunSQL's doc comment gives for the read side.
+//
+// A permanently-unprovisioned or permanently-mistagged tenant would
+// stall this consumer's offset progress entirely (every redelivery of
+// that batch fails the same way) -- a real, disclosed limitation of
+// reusing ingest/consumer's existing all-or-nothing contract rather
+// than building new partial-batch-success semantics nothing else in
+// this codebase has either. See /docs/phase-4-runbook.md.
+func (r *Registry) WriteBatch(ctx context.Context, records []consumer.Record) error {
+	byTenant := make(map[string][]consumer.Record, len(records))
+	for _, rec := range records {
+		byTenant[rec.TenantID] = append(byTenant[rec.TenantID], rec)
+	}
+
+	for tenantID, group := range byTenant {
+		if tenantID == "" {
+			return fmt.Errorf("chwriter: %d record(s) in this batch have no tenant_id, refusing to write any of it", len(group))
+		}
+		writer, ok := r.writers[tenantID]
+		if !ok {
+			return fmt.Errorf("chwriter: tenant %q has no provisioned ClickHouse connection, refusing to write %d record(s)", tenantID, len(group))
+		}
+		plain := make([]*logsv1.LogRecord, len(group))
+		for i, rec := range group {
+			plain[i] = rec.Record
+		}
+		if err := writer.WriteBatch(ctx, plain); err != nil {
+			return fmt.Errorf("chwriter: writing batch for tenant %q: %w", tenantID, err)
+		}
+	}
+	return nil
+}
@@ -0,0 +1,157 @@
+// Fail-closed behavior (empty/unknown tenant_id) needs no live
+// ClickHouse at all -- Registry.WriteBatch returns before ever touching
+// a connection for those cases, so those tests run unconditionally.
+// Everything that actually writes data is a real integration test
+// against a live ClickHouse (same CHWRITER_TEST_CLICKHOUSE_ADDR
+// convention as enterprise/internal/chrunner's own tests), skipped
+// unless that's set; run via:
+//
+//	docker run --rm --network sentry_default -v $(pwd)/../../..:/src -w /src/enterprise \
+//	  -e CHWRITER_TEST_CLICKHOUSE_ADDR=clickhouse:9000 \
+//	  -e CHWRITER_TEST_CLICKHOUSE_PASSWORD=sentry-dev-only \
+//	  golang:1.25-alpine go test ./internal/chwriter/... -v
+package chwriter
+
+import (
+	"context"
+	"fmt"
+	"os"
+	"testing"
+
+	chdriver "github.com/ClickHouse/clickhouse-go/v2"
+	"github.com/google/uuid"
+
+	"github.com/sentry/sentry/enterprise/internal/tenantprovision"
+	"github.com/sentry/sentry/ingest/clickhousewriter"
+	"github.com/sentry/sentry/ingest/consumer"
+	logsv1 "github.com/sentry/sentry/proto/sentry/logs/v1"
+)
+
+// TestWriteBatchRefusesEmptyTenantID and
+// TestWriteBatchRefusesUnknownTenantWithEmptyRegistry construct a
+// Registry directly (bypassing New, which would dial ClickHouse) so
+// they genuinely run without Docker: WriteBatch's fail-closed checks
+// happen before ever touching a real connection, purely a map lookup.
+func TestWriteBatchRefusesEmptyTenantID(t *testing.T) {
+	reg := &Registry{writers: map[string]*clickhousewriter.Writer{}}
+	err := reg.WriteBatch(context.Background(), []consumer.Record{
+		{TenantID: "", Record: &logsv1.LogRecord{Message: "untagged"}},
+	})
+	if err == nil {
+		t.Fatal("expected WriteBatch to refuse a record with no tenant_id, not silently drop the tag")
+	}
+}
+
+func TestWriteBatchRefusesUnknownTenantWithEmptyRegistry(t *testing.T) {
+	reg := &Registry{writers: map[string]*clickhousewriter.Writer{}}
+	err := reg.WriteBatch(context.Background(), []consumer.Record{
+		{TenantID: "acme", Record: &logsv1.LogRecord{Message: "m"}},
+	})
+	if err == nil {
+		t.Fatal("expected WriteBatch to refuse a tenant with no entry in the registry")
+	}
+}
+
+func testAddr(t *testing.T) string {
+	t.Helper()
+	addr := os.Getenv("CHWRITER_TEST_CLICKHOUSE_ADDR")
+	if addr == "" {
+		t.Skip("CHWRITER_TEST_CLICKHOUSE_ADDR not set -- skipping live-ClickHouse integration test")
+	}
+	return addr
+}
+
+func provisionTestTenant(t *testing.T, addr string) (tenantID string, creds tenantprovision.Credentials) {
+	t.Helper()
+	admin, err := chdriver.Open(&chdriver.Options{
+		Addr: []string{addr},
+		Auth: chdriver.Auth{Database: "default", Username: "default", Password: os.Getenv("CHWRITER_TEST_CLICKHOUSE_PASSWORD")},
+	})
+	if err != nil {
+		t.Fatalf("opening admin connection: %v", err)
+	}
+	t.Cleanup(func() { admin.Close() })
+
+	tenantID = "cw" + uuid.NewString()[:8]
+	creds, err = tenantprovision.New(admin).ProvisionClickHouse(context.Background(), tenantID)
+	if err != nil {
+		t.Fatalf("provisioning tenant %s: %v", tenantID, err)
+	}
+	if err := admin.Exec(context.Background(), fmt.Sprintf(
+		"CREATE TABLE `%s`.logs (timestamp DateTime64(9), host String, service String, severity String, message String, attributes Map(String, String), record_id UUID) ENGINE = MergeTree ORDER BY timestamp",
+		tenantID)); err != nil {
+		t.Fatalf("creating logs table for tenant %s: %v", tenantID, err)
+	}
+	return tenantID, creds
+}
+
+// TestRegistryWritesEachTenantToItsOwnDatabase is the core adversarial
+// probe for the write side, complementing chrunner's own read-side
+// version: two tenants, two connections inside one Registry, one
+// WriteBatch call mixing records from both, and a direct check (via an
+// admin connection, not through Registry) that each tenant's row landed
+// only in its own database.
+func TestRegistryWritesEachTenantToItsOwnDatabase(t *testing.T) {
+	addr := testAddr(t)
+	ctx := context.Background()
+	tenantA, credsA := provisionTestTenant(t, addr)
+	tenantB, credsB := provisionTestTenant(t, addr)
+
+	reg, err := New(ctx, addr, []DataSource{
+		{TenantID: tenantA, Database: tenantA, Username: credsA.Username, Password: credsA.Password},
+		{TenantID: tenantB, Database: tenantB, Username: credsB.Username, Password: credsB.Password},
+	})
+	if err != nil {
+		t.Fatalf("New: %v", err)
+	}
+	defer reg.Close()
+
+	err = reg.WriteBatch(ctx, []consumer.Record{
+		{TenantID: tenantA, Record: &logsv1.LogRecord{Host: "h1", Message: "for-a", RecordId: uuid.NewString()}},
+		{TenantID: tenantB, Record: &logsv1.LogRecord{Host: "h1", Message: "for-b", RecordId: uuid.NewString()}},
+	})
+	if err != nil {
+		t.Fatalf("WriteBatch: %v", err)
+	}
+
+	admin, err := chdriver.Open(&chdriver.Options{
+		Addr: []string{addr},
+		Auth: chdriver.Auth{Database: "default", Username: "default", Password: os.Getenv("CHWRITER_TEST_CLICKHOUSE_PASSWORD")},
+	})
+	if err != nil {
+		t.Fatalf("opening admin connection: %v", err)
+	}
+	defer admin.Close()
+
+	for tenantID, wantMessage := range map[string]string{tenantA: "for-a", tenantB: "for-b"} {
+		row := admin.QueryRow(ctx, fmt.Sprintf("SELECT message FROM `%s`.logs", tenantID))
+		var got string
+		if err := row.Scan(&got); err != nil {
+			t.Fatalf("querying %s's logs: %v", tenantID, err)
+		}
+		if got != wantMessage {
+			t.Fatalf("tenant %s's logs.message = %q, want %q", tenantID, got, wantMessage)
+		}
+	}
+}
+
+func TestRegistryRefusesUnprovisionedTenant(t *testing.T) {
+	addr := testAddr(t)
+	ctx := context.Background()
+	tenantA, credsA := provisionTestTenant(t, addr)
+
+	reg, err := New(ctx, addr, []DataSource{
+		{TenantID: tenantA, Database: tenantA, Username: credsA.Username, Password: credsA.Password},
+	})
+	if err != nil {
+		t.Fatalf("New: %v", err)
+	}
+	defer reg.Close()
+
+	err = reg.WriteBatch(ctx, []consumer.Record{
+		{TenantID: "some-other-tenant-never-provisioned", Record: &logsv1.LogRecord{Host: "h1", Message: "m", RecordId: uuid.NewString()}},
+	})
+	if err == nil {
+		t.Fatal("expected WriteBatch to refuse a tenant with no provisioned connection, not silently drop or misroute it")
+	}
+}
@@ -0,0 +1,101 @@
+// Package ingestconfig loads enterprise-ingest's configuration from
+// environment variables -- same convention as every other Go service in
+// this repo. Named ingestconfig, not config, to avoid colliding with
+// enterprise/internal/config (enterprise-auth's own, differently-shaped
+// config) within the same module -- mirrors enterprise/internal/
+// apiconfig's own naming reasoning exactly.
+package ingestconfig
+
+import (
+	"fmt"
+	"os"
+	"strconv"
+	"strings"
+)
+
+type Config struct {
+	// HTTPListenAddr serves only /healthz -- this binary's actual job
+	// (Redpanda -> per-tenant ClickHouse) has no other HTTP surface,
+	// same "just enough for Docker's HEALTHCHECK" shape as every other
+	// binary in this repo's -healthcheck self-check mode.
+	HTTPListenAddr string
+	// ClickHouseAddr is the shared physical ClickHouse server's native
+	// address -- every tenant's connection (enterprise/internal/
+	// chwriter.Registry) dials this same address, just with different
+	// per-tenant credentials rbacstore already has on file from
+	// enterprise-api -provision-tenant. Mirrors apiconfig.Config.
+	// ClickHouseAddr's own doc comment.
+	ClickHouseAddr string
+	Postgres       PostgresConfig
+	Redpanda       RedpandaConfig
+	Batch          BatchConfig
+}
+
+type PostgresConfig struct {
+	Addr     string
+	Database string
+	Username string
+	Password string
+}
+
+type RedpandaConfig struct {
+	Brokers       []string
+	Topic         string
+	ConsumerGroup string
+}
+
+type BatchConfig struct {
+	MaxSize         int
+	FlushIntervalMS int
+}
+
+func Load() (Config, error) {
+	cfg := Config{
+		HTTPListenAddr: getenv("HTTP_LISTEN_ADDR", ":8084"),
+		ClickHouseAddr: getenv("CLICKHOUSE_ADDR", "localhost:9000"),
+		Postgres: PostgresConfig{
+			Addr:     getenv("POSTGRES_ADDR", "localhost:5432"),
+			Database: getenv("POSTGRES_DATABASE", "sentry_metadata"),
+			Username: getenv("POSTGRES_USERNAME", "sentry"),
+			Password: getenv("POSTGRES_PASSWORD", ""),
+		},
+		Redpanda: RedpandaConfig{
+			Brokers: strings.Split(getenv("REDPANDA_BROKERS", "localhost:9092"), ","),
+			// Same default topic ingest/internal/config uses -- this
+			// binary reads the identical shared sentry.logs.raw topic
+			// ingest/cmd/ingest's server half (agent-facing PushBatch)
+			// produces onto; there's no per-tenant topic, see
+			// ingest/internal/grpcserver's doc comment.
+			Topic: getenv("REDPANDA_TOPIC", "sentry.logs.raw"),
+			// A distinct consumer group from ingest/cmd/ingest's own
+			// default ("sentry-ingest") -- this binary and a
+			// single-tenant `ingest -mode=consumer` must never share a
+			// group (each message would only ever reach one of them,
+			// silently splitting traffic) even though in practice a
+			// real multi-tenant deployment runs this binary *instead
+			// of*, not alongside, `ingest -mode=consumer`.
+			ConsumerGroup: getenv("REDPANDA_CONSUMER_GROUP", "sentry-enterprise-ingest"),
+		},
+	}
+
+	maxSize, err := strconv.Atoi(getenv("CONSUMER_BATCH_MAX_SIZE", "500"))
+	if err != nil {
+		return Config{}, fmt.Errorf("CONSUMER_BATCH_MAX_SIZE: %w", err)
+	}
+	cfg.Batch.MaxSize = maxSize
+
+	flushMS, err := strconv.Atoi(getenv("CONSUMER_BATCH_FLUSH_INTERVAL_MS", "2000"))
+	if err != nil {
+		return Config{}, fmt.Errorf("CONSUMER_BATCH_FLUSH_INTERVAL_MS: %w", err)
+	}
+	cfg.Batch.FlushIntervalMS = flushMS
+
+	return cfg, nil
+}
+
+func getenv(key, fallback string) string {
+	if v := os.Getenv(key); v != "" {
+		return v
+	}
+	return fallback
+}
@@ -59,8 +59,16 @@ func New(admin driver.Conn) *Provisioner {
 }

 // ProvisionClickHouse creates tenantID's database and a fresh user for
-// it, granted SELECT on exactly that database and nothing else.
-// Database creation is idempotent (CREATE DATABASE IF NOT EXISTS --
+// it, granted SELECT and INSERT on exactly that database and nothing
+// else -- one credential covers both enterprise/internal/chrunner's
+// query path and enterprise/internal/chwriter's ingest-write path
+// (found while building chwriter: the grant here originally covered
+// SELECT only, which would have made every real per-tenant ClickHouse
+// write fail with a permission error -- there's no cross-tenant
+// boundary crossed by also granting INSERT within a tenant's own
+// database, so a single credential for both directions is the simpler,
+// still-correctly-scoped choice over provisioning a second write-only
+// credential). Database creation is idempotent (CREATE DATABASE IF NOT EXISTS --
 // harmless to repeat). User creation deliberately is NOT idempotent
 // (plain CREATE USER, no IF NOT EXISTS): ClickHouse has no way to read
 // back an existing user's password, so silently succeeding on a second
@@ -108,8 +116,8 @@ func (p *Provisioner) ProvisionClickHouse(ctx context.Context, tenantID string)
 		return Credentials{}, fmt.Errorf("tenantprovision: creating user (already provisioned? this call is not safe to retry): %w", err)
 	}

-	if err := p.admin.Exec(ctx, fmt.Sprintf("GRANT SELECT ON `%s`.* TO `%s`", database, username)); err != nil {
-		return Credentials{}, fmt.Errorf("tenantprovision: granting select: %w", err)
+	if err := p.admin.Exec(ctx, fmt.Sprintf("GRANT SELECT, INSERT ON `%s`.* TO `%s`", database, username)); err != nil {
+		return Credentials{}, fmt.Errorf("tenantprovision: granting select/insert: %w", err)
 	}

 	return Credentials{Username: username, Password: password}, nil
@@ -80,6 +80,41 @@ func TestProvisionClickHouseCreatesUsableTenantConnection(t *testing.T) {
 	}
 }

+// TestProvisionedUserCanInsertIntoOwnDatabase is the regression test for
+// a real bug found while building enterprise/internal/chwriter: this
+// credential is also the one chwriter.Registry uses to write ingested
+// records, so it must be able to INSERT into its own database, not just
+// SELECT from it -- the grant originally only covered SELECT, which
+// would have made every real per-tenant ClickHouse write fail with a
+// permission error.
+func TestProvisionedUserCanInsertIntoOwnDatabase(t *testing.T) {
+	admin := testAdminConn(t)
+	p := New(admin)
+	tenantID := testTenantID()
+	ctx := context.Background()
+
+	creds, err := p.ProvisionClickHouse(ctx, tenantID)
+	if err != nil {
+		t.Fatalf("ProvisionClickHouse: %v", err)
+	}
+	if err := admin.Exec(ctx, fmt.Sprintf("CREATE TABLE `%s`.marker (id UInt8) ENGINE = Memory", tenantID)); err != nil {
+		t.Fatalf("creating marker table: %v", err)
+	}
+
+	tenantConn, err := clickhouse.Open(&clickhouse.Options{
+		Addr: []string{os.Getenv("TENANTPROVISION_TEST_CLICKHOUSE_ADDR")},
+		Auth: clickhouse.Auth{Database: tenantID, Username: creds.Username, Password: creds.Password},
+	})
+	if err != nil {
+		t.Fatalf("opening tenant connection: %v", err)
+	}
+	defer tenantConn.Close()
+
+	if err := tenantConn.Exec(ctx, "INSERT INTO marker VALUES (42)"); err != nil {
+		t.Fatalf("INSERT as the provisioned tenant user into its own database: %v", err)
+	}
+}
+
 // TestProvisionedUserCannotReadOtherTenantDatabase is one of the four
 // adversarial probes /docs/phase-4-isolation-design.md's verification
 // plan names for Phase 4 task 8 (see api/queryapi/