Unify the Tenant CRD with enterprise-api -provision-tenant (lightweight)

Closes a gap named across CLAUDE.md/docs/architecture.md/deploy/README.md
since early Phase 4: the operator's Tenant CRD and -provision-tenant
were two disconnected mechanisms. The operator's reconciler generated a
K8s Secret with a locally-generated random password that authenticated
against nothing (nothing ever called ClickHouse to create a matching
user), and unconditionally claimed status.phase=Active the moment a
Tenant object existed -- actively misleading, not just incomplete.

Two unification shapes were considered (surfaced to the user via
AskUserQuestion, given the real difference in blast radius): the
operator's reconcile loop becoming a second real actor (new Postgres +
ClickHouse admin credentials flowing into the K8s controller, plus real
reconcile-loop idempotency/retry design for an inherently one-shot
external side effect), or keeping -provision-tenant as the sole real
actor and having it also sync its result into the CRD. Went with the
lighter option.

enterprise/internal/tenantcrd (new): a Syncer using the K8s dynamic
client (unstructured.Unstructured + a GroupVersionResource, not
deploy/operator's typed Tenant struct -- avoids a cross-module Go
dependency between two independently-versioned modules for one type).
Upserts the Tenant object, creates/updates a Secret with the *real*
ClickHouse credentials owned by that Tenant via an OwnerReference, then
patches status.{clickHouseDatabaseName,clickHouseSecretRef,
tantivyIndexPath}. Idempotent and safe to retry: never rotates a
credential across a re-sync, never overwrites a pre-existing
spec.displayName a human/GitOps process set.

cmd/enterprise-api/main.go's runProvisionTenant calls Sync when
TENANT_CRD_NAMESPACE is set (empty = no-op, same shape as every other
optional dependency in this codebase). Its "already active" refusal is
now split: ClickHouse re-provisioning is still refused (rotating a live
credential would break every open connection for no benefit), but CR
sync alone is now retryable using the credentials already on file in
rbacstore -- needed for retrying a previously-failed sync, or
backfilling CR sync for a tenant provisioned before this existed.

deploy/operator's reconciler rewritten to match: it never claims
PhaseActive on its own initiative anymore, only once
status.ClickHouseDatabaseName is non-empty (the field -provision-tenant,
and only -provision-tenant, sets). Phase is now a pure function of
{spec.suspended, status.ClickHouseDatabaseName != ""} recomputed every
reconcile, not toggled in place -- fixes a related bug the old code
would have hit once suspension was involved: un-suspending an
already-provisioned tenant needs to return straight to Active, which
isn't derivable from "last observed phase was Suspended" alone. The
reconciler no longer creates or manages any Secret, dropped its
`secrets` RBAC grant entirely, and gained zero new dependencies.

Helm chart: enterprise-api gets its own ServiceAccount/Role/RoleBinding
(get/list/create tenants, get/update/patch tenants/status, get/create/
update secrets -- least-privilege, scoped to the release namespace, not
a ClusterRole) and a TENANT_CRD_NAMESPACE env var, both gated on
tenantOperator.enabled. tenant-operator's ClusterRole loses the
secrets grant it no longer needs.

Verified in this environment: enterprise/internal/tenantcrd's tests run
against k8s.io/client-go's fake dynamic + typed clientsets (real client
library, fake transport, no cluster needed); deploy/operator's rewritten
tenant_controller_test.go runs against controller-runtime's fake
client, including new regression tests for the "must not claim Active
without confirmation" and "un-suspending returns to Active, not
Provisioning" properties; helm template + parsing the rendered YAML
confirms the RBAC split renders exactly as designed under both
tenantOperator.enabled=true/false. Not verified: an actual
-provision-tenant run against a real cluster with the operator watching
(no live cluster in this environment, same disclosed limitation as the
rest of /deploy). Docs updated in lockstep: CLAUDE.md, docs/architecture.md,
deploy/README.md, deploy/helm/sentry/README.md (including a corrected
"Trying the two-tenant example" walkthrough), phase-4-runbook.md (new
§11), enterprise/README.md. Also fixed two unrelated stale claims found
along the way: docs/architecture.md still said docker-compose.yml ran
plain api unconditionally (fixed in an earlier commit, doc not updated
then), and enterprise-api's own main.go doc comment still said Helm/
docker-compose wiring wasn't built yet.

deploy/helm/sentry/README.md

@@ -72,20 +72,37 @@ helm install sentry . --include-crds \
   --set 'tenants[1].name=globex' --set 'tenants[1].displayName=Globex Corporation'
 
 kubectl get tenants
+# expect: both Provisioning -- the Tenant CRs above are just a
+# declarative request; nothing has actually provisioned ClickHouse for
+# either yet (see below).
+
+kubectl exec -it deploy/sentry-api -- /enterprise-api -provision-tenant=acme -display-name="Acme Corp"
+kubectl exec -it deploy/sentry-api -- /enterprise-api -provision-tenant=globex -display-name="Globex Corporation"
+
+kubectl get tenants
+# expect: both Active now.
 kubectl get secret sentry-tenant-acme-clickhouse sentry-tenant-globex-clickhouse
 ```
 
-This proves the K8s-side half of Phase 4's "two tenants... with their
-own users, roles, dashboards" exit criteria (`/CLAUDE.md`) -- a real
-per-tenant credential Secret exists for each, generated by
-`deploy/operator`'s `Tenant` controller. It does **not** by itself give
-either tenant a working ClickHouse database or Tantivy index -- that
-needs `enterprise-api -provision-tenant=<id>` (a separate, deliberately
-manual operator action; the `Tenant` CRD and `-provision-tenant` are two
-independent mechanisms today, not yet unified -- see
-`/enterprise/README.md`), and OIDC login (built, but still needs a
-manual `tenant_memberships` row -- see `/docs/phase-4-runbook.md` §3a)
-before a human can actually query as that tenant.
+This proves Phase 4's "two tenants... with their own users, roles,
+dashboards" exit criteria (`/CLAUDE.md`) end to end at the deployment-
+topology layer: `-provision-tenant` (`enterprise/internal/
+tenantprovision`) is what actually creates each tenant's ClickHouse
+database/user/grant and marks it active in `rbacstore`; running inside
+the `enterprise-api` Deployment's Pod means it automatically syncs that
+real result into the `Tenant` CRD too (`enterprise/internal/tenantcrd`,
+via the ServiceAccount/Role `tenantOperator.enabled` also grants that
+Deployment) -- the credential Secret you see above has *real*
+credentials, not a placeholder, and `Tenant.status.phase: Active` means
+the same thing `rbacstore.tenants.status='active'` does, not two
+different claims about two different systems. The `Tenant` CRD and
+`-provision-tenant` used to be genuinely disconnected (a Secret existed
+the moment the CR was created, with a password that authenticated
+against nothing) -- see `/deploy/README.md`'s "lightweight unification"
+section for the full history. OIDC/SAML login still needs a manual
+`tenant_memberships` grant (`enterprise-auth -grant-membership-*` --
+see `/docs/phase-4-runbook.md` §3a/§3b) before a human can actually
+query as either tenant.
 
 ## `web`'s image needs rebuilding per environment
 

deploy/helm/sentry/templates/enterprise-api.yaml

@@ -12,6 +12,54 @@ routes to whichever Deployment is actually rendered, with zero
 conditional logic needed in any consumer (alerting, web).
 */}}
 {{- if .Values.enterprise.enabled }}
+{{- if .Values.tenantOperator.enabled }}
+# Grants enterprise-api's -provision-tenant (enterprise/internal/
+# tenantcrd) permission to sync real provisioning results into the
+# Tenant CRD -- a Role, not a ClusterRole (unlike tenant-operator's:
+# this binary only ever provisions tenants that live in its own release
+# namespace, no reason to widen it), scoped to exactly the two resource
+# types tenantcrd.Syncer touches. Only rendered when tenantOperator is
+# also enabled -- no Tenant CRD installed, nothing to sync into.
+apiVersion: v1
+kind: ServiceAccount
+metadata:
+  name: {{ .Release.Name }}-enterprise-api
+  labels:
+    {{- include "sentry.labels" . | nindent 4 }}
+---
+apiVersion: rbac.authorization.k8s.io/v1
+kind: Role
+metadata:
+  name: {{ .Release.Name }}-enterprise-api
+  labels:
+    {{- include "sentry.labels" . | nindent 4 }}
+rules:
+  - apiGroups: ["sentry.io"]
+    resources: ["tenants"]
+    verbs: ["get", "list", "create"]
+  - apiGroups: ["sentry.io"]
+    resources: ["tenants/status"]
+    verbs: ["get", "update", "patch"]
+  - apiGroups: [""]
+    resources: ["secrets"]
+    verbs: ["get", "create", "update"]
+---
+apiVersion: rbac.authorization.k8s.io/v1
+kind: RoleBinding
+metadata:
+  name: {{ .Release.Name }}-enterprise-api
+  labels:
+    {{- include "sentry.labels" . | nindent 4 }}
+roleRef:
+  apiGroup: rbac.authorization.k8s.io
+  kind: Role
+  name: {{ .Release.Name }}-enterprise-api
+subjects:
+  - kind: ServiceAccount
+    name: {{ .Release.Name }}-enterprise-api
+    namespace: {{ .Release.Namespace }}
+---
+{{- end }}
 apiVersion: apps/v1
 kind: Deployment
 metadata:
@@ -30,6 +78,9 @@ spec:
       labels:
         {{- include "sentry.selectorLabels" (list $ "api") | nindent 8 }}
     spec:
+      {{- if .Values.tenantOperator.enabled }}
+      serviceAccountName: {{ .Release.Name }}-enterprise-api
+      {{- end }}
       initContainers:
         {{- include "sentry.waitForTCP" (list "clickhouse" (printf "%s-clickhouse" .Release.Name) "9000") | nindent 8 }}
         {{- include "sentry.waitForTCP" (list "postgres" (printf "%s-postgres" .Release.Name) "5432") | nindent 8 }}
@@ -84,6 +135,16 @@ spec:
                   key: auditWriterPassword
             - name: ENTERPRISE_AUTH_URL
               value: "http://{{ .Release.Name }}-enterprise-auth:8082"
+            {{- if .Values.tenantOperator.enabled }}
+            # Enables enterprise/internal/tenantcrd -- -provision-tenant
+            # (run via `kubectl exec` into this Deployment's Pod, using
+            # its ServiceAccount/Role above) syncs real provisioning
+            # results into the Tenant CRD this namespace's tenants live
+            # in. Unset (the default, when tenantOperator isn't enabled)
+            # is a documented no-op -- see apiconfig.Config.TenantCRDNamespace.
+            - name: TENANT_CRD_NAMESPACE
+              value: {{ .Release.Namespace | quote }}
+            {{- end }}
           ports:
             - name: http
               containerPort: 8080

deploy/helm/sentry/templates/tenant-operator.yaml

@@ -11,8 +11,12 @@ metadata:
 # doesn't assume it's the only namespace the operator might one day watch
 # -- narrowed to exactly the two resource types
 # deploy/operator/internal/controller/tenant_controller.go's
-# +kubebuilder:rbac markers name (tenants, tenants/status, secrets), not
-# a wildcard grant.
+# +kubebuilder:rbac markers name (tenants, tenants/status), not a
+# wildcard grant. No `secrets` permission -- this controller stopped
+# managing the ClickHouse credential Secret once enterprise-api
+# -provision-tenant took over creating it with real credentials (see
+# that controller's doc comment); see enterprise-api.yaml's own
+# ServiceAccount/Role for the `secrets` grant that binary needs instead.
 apiVersion: rbac.authorization.k8s.io/v1
 kind: ClusterRole
 metadata:
@@ -26,9 +30,6 @@ rules:
   - apiGroups: ["sentry.io"]
     resources: ["tenants/status"]
     verbs: ["get", "update", "patch"]
-  - apiGroups: [""]
-    resources: ["secrets"]
-    verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
 ---
 apiVersion: rbac.authorization.k8s.io/v1
 kind: ClusterRoleBinding

deploy/helm/sentry/values.yaml

@@ -165,9 +165,11 @@ enterprise:
     idpMetadataURL: ""
 
 # Installs deploy/operator (the Tenant CRD controller) alongside this
-# chart. Only meaningful when enterprise.enabled is also true --
-# gated on that, not a separate flag, since a Tenant CR with no
-# enterprise-auth deployed to consume its Secret has nothing to do.
+# chart. Only meaningful when enterprise.enabled is also true -- gated
+# on that, not a separate flag, since a Tenant CR with nothing to
+# reflect (see below) has nothing to do. Also turns on enterprise-api's
+# own Tenant-CRD-syncing permissions (a ServiceAccount/Role, and the
+# TENANT_CRD_NAMESPACE env var) -- see templates/enterprise-api.yaml.
 tenantOperator:
   enabled: false
   image:
@@ -176,11 +178,16 @@ tenantOperator:
   resources: {}
 
 # One entry per tenant to provision -- rendered as Tenant CRs
-# (templates/tenants.yaml), reconciled by the tenant-operator into a
-# per-tenant ClickHouse credential Secret. See
-# deploy/operator/internal/controller/tenant_controller.go's doc comment
-# for exactly what that does and doesn't set up. Empty by default; a
-# real two-tenant deployment (Phase 4's exit criteria) sets e.g.:
+# (templates/tenants.yaml), a declarative request an admin/GitOps
+# process makes. The operator (tenant-operator, above) only ever
+# *reflects* real state onto these objects (Phase/Conditions, derived
+# from what enterprise-api's `-provision-tenant` has reported) -- it's
+# `-provision-tenant` (run via `kubectl exec` into the enterprise-api
+# Pod), not the operator, that actually calls ClickHouse and writes the
+# credential Secret. See deploy/operator/internal/controller/
+# tenant_controller.go's and enterprise/internal/tenantcrd's doc
+# comments for the full split. Empty by default; a real two-tenant
+# deployment (Phase 4's exit criteria) sets e.g.:
 #   tenants:
 #     - name: acme
 #       displayName: "Acme Corp"