Files
cairnobs/deploy/helm/sentry/README.md
T
jcoffey-dev 823f5d48d1 Unify the Tenant CRD with enterprise-api -provision-tenant (lightweight)
Closes a gap named across CLAUDE.md/docs/architecture.md/deploy/README.md
since early Phase 4: the operator's Tenant CRD and -provision-tenant
were two disconnected mechanisms. The operator's reconciler generated a
K8s Secret with a locally-generated random password that authenticated
against nothing (nothing ever called ClickHouse to create a matching
user), and unconditionally claimed status.phase=Active the moment a
Tenant object existed -- actively misleading, not just incomplete.

Two unification shapes were considered (surfaced to the user via
AskUserQuestion, given the real difference in blast radius): the
operator's reconcile loop becoming a second real actor (new Postgres +
ClickHouse admin credentials flowing into the K8s controller, plus real
reconcile-loop idempotency/retry design for an inherently one-shot
external side effect), or keeping -provision-tenant as the sole real
actor and having it also sync its result into the CRD. Went with the
lighter option.

enterprise/internal/tenantcrd (new): a Syncer using the K8s dynamic
client (unstructured.Unstructured + a GroupVersionResource, not
deploy/operator's typed Tenant struct -- avoids a cross-module Go
dependency between two independently-versioned modules for one type).
Upserts the Tenant object, creates/updates a Secret with the *real*
ClickHouse credentials owned by that Tenant via an OwnerReference, then
patches status.{clickHouseDatabaseName,clickHouseSecretRef,
tantivyIndexPath}. Idempotent and safe to retry: never rotates a
credential across a re-sync, never overwrites a pre-existing
spec.displayName a human/GitOps process set.

cmd/enterprise-api/main.go's runProvisionTenant calls Sync when
TENANT_CRD_NAMESPACE is set (empty = no-op, same shape as every other
optional dependency in this codebase). Its "already active" refusal is
now split: ClickHouse re-provisioning is still refused (rotating a live
credential would break every open connection for no benefit), but CR
sync alone is now retryable using the credentials already on file in
rbacstore -- needed for retrying a previously-failed sync, or
backfilling CR sync for a tenant provisioned before this existed.

deploy/operator's reconciler rewritten to match: it never claims
PhaseActive on its own initiative anymore, only once
status.ClickHouseDatabaseName is non-empty (the field -provision-tenant,
and only -provision-tenant, sets). Phase is now a pure function of
{spec.suspended, status.ClickHouseDatabaseName != ""} recomputed every
reconcile, not toggled in place -- fixes a related bug the old code
would have hit once suspension was involved: un-suspending an
already-provisioned tenant needs to return straight to Active, which
isn't derivable from "last observed phase was Suspended" alone. The
reconciler no longer creates or manages any Secret, dropped its
`secrets` RBAC grant entirely, and gained zero new dependencies.

Helm chart: enterprise-api gets its own ServiceAccount/Role/RoleBinding
(get/list/create tenants, get/update/patch tenants/status, get/create/
update secrets -- least-privilege, scoped to the release namespace, not
a ClusterRole) and a TENANT_CRD_NAMESPACE env var, both gated on
tenantOperator.enabled. tenant-operator's ClusterRole loses the
secrets grant it no longer needs.

Verified in this environment: enterprise/internal/tenantcrd's tests run
against k8s.io/client-go's fake dynamic + typed clientsets (real client
library, fake transport, no cluster needed); deploy/operator's rewritten
tenant_controller_test.go runs against controller-runtime's fake
client, including new regression tests for the "must not claim Active
without confirmation" and "un-suspending returns to Active, not
Provisioning" properties; helm template + parsing the rendered YAML
confirms the RBAC split renders exactly as designed under both
tenantOperator.enabled=true/false. Not verified: an actual
-provision-tenant run against a real cluster with the operator watching
(no live cluster in this environment, same disclosed limitation as the
rest of /deploy). Docs updated in lockstep: CLAUDE.md, docs/architecture.md,
deploy/README.md, deploy/helm/sentry/README.md (including a corrected
"Trying the two-tenant example" walkthrough), phase-4-runbook.md (new
§11), enterprise/README.md. Also fixed two unrelated stale claims found
along the way: docs/architecture.md still said docker-compose.yml ran
plain api unconditionally (fixed in an earlier commit, doc not updated
then), and enterprise-api's own main.go doc comment still said Helm/
docker-compose wiring wasn't built yet.
2026-08-14 09:07:10 -07:00

129 lines
6.3 KiB
Markdown

# deploy/helm/sentry
A Helm chart covering every `docker-compose.yml` service (Redpanda,
ClickHouse, Postgres, ingest, search, alerting, web) plus, when
`enterprise.enabled: true`: enterprise-auth, the `deploy/operator`
tenant-operator, and `Tenant` CRs from `values.tenants`. See
`/deploy/README.md` for what "multi-tenant-aware" does and doesn't mean
at this layer, and its verification-status section before trusting this
against a real cluster.
This chart never builds images -- push every image its `values.yaml`
references to a registry the cluster can pull from first, same division
of labor as `docker compose build` vs. `docker compose up`.
## `api` vs `enterprise-api`: one Deployment, chosen by `enterprise.enabled`
`templates/api.yaml` and `templates/enterprise-api.yaml` are mutually
exclusive, gated on opposite sides of the same `enterprise.enabled` flag
-- exactly one of them ever renders, both under the same
`{{ .Release.Name }}-api` Service name and port 8080. This is the fix
for what `/docs/security/threat-model.md` named as Phase 4's single
largest remaining gap once both storage engines' isolation mechanisms
were built: previously nothing forced or even flagged whether a
deployment ran the tenant-isolated binary. Now it's not a second knob to
remember -- the same flag that turns on RBAC/audit/SSO also swaps which
query binary actually serves `/query` and `/dashboards` traffic. Every
consumer (`alerting`'s `API_QUERY_URL`, `web`'s build args) needs zero
conditional logic of its own, since both variants answer on the same
name/port.
`enterprise-api` starts with an empty tenant set until
`-provision-tenant` has been run for at least one tenant (see
`/enterprise/README.md`) -- until then it's up and healthy, but every
`/query` request correctly fails closed with no tenant to route to.
## Startup ordering
`docker-compose.yml` uses `depends_on: condition: service_healthy` /
`service_completed_successfully` to sequence startup (e.g. `api` waits
for `clickhouse-migrate` to actually finish, not just for `clickhouse` to
be reachable). This chart approximates that more loosely:
- Migration Jobs (`clickhouse-migrate`, `metadata-migrate`,
`redpanda-provision`) are plain `Job` resources (not Helm hooks --
making the StatefulSets they depend on into hooks too, to get
ordering, would break `helm upgrade`/`helm uninstall`'s normal
ownership tracking of stateful resources, a worse tradeoff), with
`backoffLimit: 6` so they retry a few times if their dependency isn't
up yet.
- App Deployments get an `initContainer` that busy-waits for their
dependency's **TCP port**, not for a specific Job's completion (see
`templates/_helpers.tpl`'s `sentry.waitForTCP`) -- this covers "is
ClickHouse/Postgres/Redpanda up" but not "has the migration Job
actually finished."
- The gap that leaves (a pod starts before its migration has completed)
is covered by every Go service here already calling `os.Exit(1)` on a
failed startup DB ping (see e.g. `api/cmd/api/main.go`) --
Kubernetes' pod restart policy retries with backoff until the schema
is ready. This is a real, working, but *looser* guarantee than
docker-compose's explicit ordering -- documented here rather than
implied to be equivalent.
## Trying the two-tenant example
```sh
# Quote each --set value -- zsh globs an unquoted tenants[0] as a
# pattern and fails with "no matches found."
helm install sentry . --include-crds \
--set enterprise.enabled=true \
--set tenantOperator.enabled=true \
--set 'tenants[0].name=acme' --set 'tenants[0].displayName=Acme Corp' \
--set 'tenants[1].name=globex' --set 'tenants[1].displayName=Globex Corporation'
kubectl get tenants
# expect: both Provisioning -- the Tenant CRs above are just a
# declarative request; nothing has actually provisioned ClickHouse for
# either yet (see below).
kubectl exec -it deploy/sentry-api -- /enterprise-api -provision-tenant=acme -display-name="Acme Corp"
kubectl exec -it deploy/sentry-api -- /enterprise-api -provision-tenant=globex -display-name="Globex Corporation"
kubectl get tenants
# expect: both Active now.
kubectl get secret sentry-tenant-acme-clickhouse sentry-tenant-globex-clickhouse
```
This proves Phase 4's "two tenants... with their own users, roles,
dashboards" exit criteria (`/CLAUDE.md`) end to end at the deployment-
topology layer: `-provision-tenant` (`enterprise/internal/
tenantprovision`) is what actually creates each tenant's ClickHouse
database/user/grant and marks it active in `rbacstore`; running inside
the `enterprise-api` Deployment's Pod means it automatically syncs that
real result into the `Tenant` CRD too (`enterprise/internal/tenantcrd`,
via the ServiceAccount/Role `tenantOperator.enabled` also grants that
Deployment) -- the credential Secret you see above has *real*
credentials, not a placeholder, and `Tenant.status.phase: Active` means
the same thing `rbacstore.tenants.status='active'` does, not two
different claims about two different systems. The `Tenant` CRD and
`-provision-tenant` used to be genuinely disconnected (a Secret existed
the moment the CR was created, with a password that authenticated
against nothing) -- see `/deploy/README.md`'s "lightweight unification"
section for the full history. OIDC/SAML login still needs a manual
`tenant_memberships` grant (`enterprise-auth -grant-membership-*` --
see `/docs/phase-4-runbook.md` §3a/§3b) before a human can actually
query as either tenant.
## `web`'s image needs rebuilding per environment
`web` is a static SvelteKit build (`adapter-static`) -- its three API
base URLs (`VITE_API_BASE_URL`/`VITE_ALERTING_API_BASE_URL`/
`VITE_ENTERPRISE_AUTH_BASE_URL`) are baked in at **image build time**
(`web/Dockerfile`'s build args), not read from the container's
environment at runtime. `values.yaml`'s `web.builtWithApiBaseURL` etc.
document what the image you point `web.image` at needs to have been
built with (an Ingress hostname, a LoadBalancer IP, etc.) -- this chart
has no Ingress resources and can't itself act on those values; rebuild
`web`'s image with the right build args for wherever this release is
actually reachable from a browser before pointing real users at it.
## Validating without a cluster
```sh
helm lint .
helm template sentry . --include-crds > /tmp/rendered.yaml
```
See `/deploy/README.md`'s verification section for what was actually
checked this way (and what wasn't -- no live cluster was available).