Phase 4: real per-tenant ClickHouse isolation via a new enterprise-api binary

Closes the threat model's headline finding for the SQL query path:
enterprise/internal/tenantprovision does real CREATE DATABASE/USER/GRANT
against ClickHouse, and enterprise/internal/chrunner is a per-tenant
connection registry implementing api's SQLRunner interface, resolving
the tenant from the authenticated request identity -- never a
caller-suppliable parameter. Both are wired into a new binary,
enterprise/cmd/enterprise-api, alongside the unchanged single-tenant
api/cmd/api, since AGPL core can never import enterprise/ and Go's own
internal/ package visibility rules meant enterprise/ couldn't implement
core's SQLRunner interface without importing the package that defines
it. That required moving api/internal/{authz,queryapi,dashboards,
querylang/executor,searchclient,httpserver} out of internal/ -- the
minimal set enterprise-api needs to import; querylang's compiler
internals (planner/lexer/parser/ast/ir) and api's own config stay
internal, since nothing outside api needs them directly.

Also finally wires enterprise/internal/audit into queryapi.AuditLogger
(nil since Phase 4 task 4) via a new adapter, and adds live-ClickHouse
integration tests for two of the four adversarial probes named in
docs/phase-4-isolation-design.md's verification plan.

Corrected several overclaims in the docs while writing this up: an
earlier claim that rbacstore's CRUD was "verified against a live
Postgres" was never actually true in this environment (only
internal/audit was, earlier in this phase, before Docker access was
lost) -- threat-model.md, phase-4-runbook.md, CLAUDE.md, and
enterprise/README.md all now distinguish "a real integration test
exists" from "this was confirmed against a live database."

Still not built: Tantivy/free-text tenant isolation
(enterprise/internal/searchclient), and any deployment-topology
mechanism that actually routes traffic to enterprise-api instead of
plain api -- both binaries exist side by side today with nothing
enforcing or flagging which one a deployment runs.

enterprise/README.md

@@ -5,20 +5,26 @@ boundary. SSO (OIDC/SAML), tenant provisioning, and RBAC. Nothing in
 `/agent`, `/ingest`, `/storage`, `/api`, `/web` core, or `/cli` imports
 from this module — confirmed by `hack/check-tenant-boundary.sh`, run in
 CI. `enterprise/` supplies tenant-scoped implementations of core's
-already-shipped `api/internal/querylang/executor.SQLRunner`/
+already-shipped `api/querylang/executor.SQLRunner`/
 `SearchClient` interfaces rather than core growing tenant awareness —
 see `/docs/phase-4-isolation-design.md` for why.
 
 ## Status
 
-Tasks 3-5 (module skeleton, SSO library wiring, audit logging, and auth
-wiring in `/api`/`/web`/`/cli`) are built and tested. What's live
-end-to-end:
+What's built and wired end-to-end. Verification status varies by
+piece -- `internal/audit` was confirmed against a real Postgres earlier
+in this phase's work; everything else below has real integration tests
+written the same way (skipped unless a live database's connection
+details are supplied via env var, same pattern throughout this package)
+but they have **not actually been run against a live database in this
+environment** -- see `/docs/phase-4-runbook.md`'s verification-status
+section for exactly what "not yet run" means here and why. Don't read
+"has a test for this" as "this was confirmed to work."
 
 - `internal/session` issues/validates signed (HS256/JWT) tokens for both
   human sessions and `/alerting`'s `RoleService` credential.
 - `internal/authhandler` serves `POST /internal/authorize` (the endpoint
-  `api/internal/authz.HTTPAuthorizer` calls) and `GET /auth/features`
+  `api/authz.HTTPAuthorizer` calls) and `GET /auth/features`
   (the runtime-capability check `/web`'s settings page reads).
 - `api`'s `/query` and `/dashboards` endpoints enforce RBAC via
   `authz.RequireRole`/`RequireRoleOrService`, nil-safe (no-op) when
@@ -29,8 +35,30 @@ end-to-end:
 - `sentryctl` presents `$SENTRYCTL_TOKEN` as a Bearer credential on every
   request when set.
 - `internal/rbacstore`: full CRUD over `users`/`tenants`/
-  `tenant_memberships` (`metadata/migrations/0017-0023`), verified
-  against a live Postgres.
+  `tenant_memberships`/`data_sources` (`metadata/migrations/0017-0032`).
+- `internal/tenantprovision`: real `CREATE DATABASE`/`CREATE USER`/
+  `GRANT` against ClickHouse. Its tests assert a tenant A user cannot
+  read tenant B's database by fully-qualified name, and that
+  `system.query_log`/`system.tables`/`SHOW DATABASES` don't leak across
+  tenants either (task 2's finding was that the latter is
+  version-dependent) -- not yet run against a live ClickHouse in this
+  environment, see the note above.
+- `internal/chrunner`: the tenant-scoped `SQLRunner` -- a per-tenant
+  connection registry that resolves which tenant's ClickHouse connection
+  to use from the authenticated identity in request context, never a
+  parameter. Same adversarial probe, now through the actual production
+  code path (`chrunner.Registry.RunSQL`, not just tenantprovision's raw
+  grants).
+- `internal/audit.QueryAPILogger`: the real `api/queryapi.AuditLogger`
+  implementation -- wired into `enterprise-api`, no longer `nil`.
+- `cmd/enterprise-api`: a second binary (alongside `api/cmd/api`,
+  unchanged) importing *both* `api`'s handler packages and the
+  tenant-aware implementations above -- see its own doc comment for why
+  this shape exists (`enterprise → api` is the allowed import direction;
+  `api` can never import `enterprise/`). `-provision-tenant=<id>` is the
+  operator action that provisions ClickHouse and marks a tenant active,
+  same "offline action, not a network endpoint" shape as
+  `enterprise-auth -mint-service-token`.
 
 **Deliberately deferred, not half-built** -- named explicitly rather than
 silently left out:
@@ -39,37 +67,42 @@ silently left out:
   `internal/saml` do the protocol mechanics; nothing calls them from an
   HTTP handler yet). `-mint-service-token` is the only way to get a
   token today, and it only mints `RoleService` credentials.
-- `dashboard_permissions`/`data_sources` CRUD (schema exists,
-  `metadata/migrations/0024-0026`; no caller reads per-resource grants
+- `dashboard_permissions` CRUD (schema exists,
+  `metadata/migrations/0024`; no caller reads per-resource grants
   yet -- `dashboards`' handler enforces tenant-baseline role only, not
   the matrix's "(own/granted)" qualifier).
-- `internal/tenantprovision` (ClickHouse DB/user/grant + Tantivy index
-  provisioning) and the tenant-scoped `internal/chrunner`/
-  `internal/searchclient` `SQLRunner`/`SearchClient` implementations --
-  task 2's isolation model, not yet built against real per-tenant
-  connections.
-- Wiring `internal/audit` into `api`'s `queryapi.AuditLogger` extension
-  point (built in core since task 4, still passed as `nil`).
+- `internal/searchclient` (the Tantivy-side sibling of `chrunner`) --
+  `enterprise-api` shares the single, un-tenant-scoped Tantivy index
+  every deployment does today (`api/searchclient.Dial`, unchanged). See
+  `/docs/security/threat-model.md`.
+- Any deployment-topology mechanism that actually routes traffic to
+  `enterprise-api` instead of `api` -- both binaries exist,
+  `docker-compose.yml` includes `enterprise-api` available but not
+  wired into `web`'s default base URL, and the Helm chart has no
+  service for it at all yet.
 
 ## Package layout
 
 ```
 cmd/enterprise-auth/   config loading, OIDC discovery at startup, health/authorize/features endpoints, -mint-service-token
+cmd/enterprise-api/     multi-tenant-aware alternative to api/cmd/api -- see its own doc comment
 internal/tenant/        the ID type -- see its package doc comment before touching it
 internal/oidc/           coreos/go-oidc wiring: discovery, login redirect, code exchange + ID token verification
 internal/saml/            crewjam/saml wiring: SP setup, login redirect, response parsing/validation
 internal/session/          issues/validates signed session + RoleService tokens
 internal/authhandler/       POST /internal/authorize, GET /auth/features
-internal/rbacstore/          users/tenants/tenant_memberships CRUD (pgx against sentry_metadata)
-internal/audit/            append-only, hash-chained query audit log -- see its own package
-                            doc comment and /docs/phase-4-isolation-design.md's audit section
-internal/config/          env-var config, same convention as every other Go service here
+internal/rbacstore/          users/tenants/tenant_memberships/data_sources CRUD (pgx against sentry_metadata)
+internal/tenantprovision/     real ClickHouse CREATE DATABASE/USER/GRANT
+internal/chrunner/             tenant-scoped api/querylang/executor.SQLRunner
+internal/audit/            append-only, hash-chained query audit log, plus the
+                            api/queryapi.AuditLogger adapter (queryapi_adapter.go)
+internal/apiconfig/       enterprise-api's own env-var config
+internal/config/          enterprise-auth's env-var config
 ```
 
-Future additions: `internal/tenantprovision`, `internal/chrunner`/
-`internal/searchclient` (tenant-scoped `SQLRunner`/`SearchClient`
-implementations), the OIDC/SAML login/callback HTTP handlers, and
-`dashboard_permissions`/`data_sources` CRUD -- see "Status" above.
+Future additions: `internal/searchclient`, the OIDC/SAML login/callback
+HTTP handlers, `dashboard_permissions` CRUD, and real deployment-topology
+wiring for `enterprise-api` -- see "Status" above.
 
 ## Why OIDC and SAML aren't hand-rolled
 
@@ -113,6 +146,23 @@ docker run --rm --network sentry_default -v $(pwd)/..:/src -w /src/enterprise \
   golang:1.25-alpine go test ./internal/rbacstore/... -v
 ```
 
+`internal/tenantprovision` and `internal/chrunner` need a real
+ClickHouse instead (they mount the repo root, not just `enterprise/`,
+since `internal/chrunner` imports `api/authz`/`api/querylang/executor`
+via `go.mod`'s `replace` directives to `../api`):
+
+```sh
+docker run --rm --network sentry_default -v $(pwd)/..:/src -w /src/enterprise \
+  -e TENANTPROVISION_TEST_CLICKHOUSE_ADDR=clickhouse:9000 \
+  -e TENANTPROVISION_TEST_CLICKHOUSE_PASSWORD=sentry-dev-only \
+  golang:1.25-alpine go test ./internal/tenantprovision/... -v
+
+docker run --rm --network sentry_default -v $(pwd)/..:/src -w /src/enterprise \
+  -e CHRUNNER_TEST_CLICKHOUSE_ADDR=clickhouse:9000 \
+  -e CHRUNNER_TEST_CLICKHOUSE_PASSWORD=sentry-dev-only \
+  golang:1.25-alpine go test ./internal/chrunner/... -v
+```
+
 ## Turning on auth enforcement for manual testing
 
 Off by default (see "Status" above -- there's no login flow to issue a
@@ -129,7 +179,25 @@ TOKEN=$(docker compose run --rm enterprise-auth -mint-service-token=alerting)
 docker build -f Dockerfile -t sentry-enterprise-auth .   # context is enterprise/, not the repo root
 ```
 
-## Environment variables
+## Provisioning a tenant and running `enterprise-api`
+
+```sh
+docker compose build enterprise-api   # context is the repo root, not enterprise/ -- see cmd/enterprise-api/Dockerfile
+docker compose run --rm enterprise-api -provision-tenant=acme -display-name="Acme Corp"
+docker compose up -d enterprise-api
+curl -s http://localhost:8083/healthz
+```
+
+`-provision-tenant` creates the tenant/data_source rows in rbacstore if
+they don't exist, provisions ClickHouse, persists the credentials, and
+marks the tenant active -- refuses to run twice for the same tenant
+(re-provisioning would either rotate a live credential or silently fail
+to, see `tenantprovision.ProvisionClickHouse`'s doc comment). `web`
+still points at plain `api` by default (`VITE_API_BASE_URL`) --
+pointing it at `enterprise-api` instead is a manual `docker-compose.yml`
+edit today, not a supported flag.
+
+## Environment variables (`enterprise-auth`)
 
 | Var | Default |
 |---|---|
@@ -146,3 +214,22 @@ docker build -f Dockerfile -t sentry-enterprise-auth .   # context is enterprise
 | `SAML_ACS_URL` | (empty) |
 | `SAML_IDP_METADATA_URL` | (empty — presence only feeds `GET /auth/features`; not yet fetched/parsed) |
 | `ENTERPRISE_SESSION_SIGNING_KEY` | **required**, min 32 bytes |
+
+## Environment variables (`enterprise-api`)
+
+| Var | Default |
+|---|---|
+| `HTTP_LISTEN_ADDR` | `:8083` |
+| `CLICKHOUSE_ADDR` | `localhost:9000` |
+| `CLICKHOUSE_ADMIN_USERNAME` | `default` |
+| `CLICKHOUSE_ADMIN_PASSWORD` | (empty) |
+| `SEARCH_GRPC_ADDR` | `localhost:50052` |
+| `POSTGRES_ADDR` | `localhost:5432` |
+| `POSTGRES_DATABASE` | `sentry_metadata` |
+| `POSTGRES_USERNAME` | `sentry` |
+| `POSTGRES_PASSWORD` | (empty) |
+| `AUDIT_WRITER_USERNAME` | `audit_writer` |
+| `AUDIT_WRITER_PASSWORD` | (empty) |
+| `ENTERPRISE_AUTH_URL` | (empty — RBAC becomes a no-op, but `chrunner.Registry.RunSQL` still refuses every query with no resolved tenant identity, so leaving this unset does not mean "open access," it means "every query fails") |
+| `CORS_ALLOWED_ORIGIN` | `*` |
+| `QUERY_TIMEOUT_SECONDS` | `30` |