jcoffey-dev
823f5d48d1
Closes a gap named across CLAUDE.md/docs/architecture.md/deploy/README.md
since early Phase 4: the operator's Tenant CRD and -provision-tenant
were two disconnected mechanisms. The operator's reconciler generated a
K8s Secret with a locally-generated random password that authenticated
against nothing (nothing ever called ClickHouse to create a matching
user), and unconditionally claimed status.phase=Active the moment a
Tenant object existed -- actively misleading, not just incomplete.
Two unification shapes were considered (surfaced to the user via
AskUserQuestion, given the real difference in blast radius): the
operator's reconcile loop becoming a second real actor (new Postgres +
ClickHouse admin credentials flowing into the K8s controller, plus real
reconcile-loop idempotency/retry design for an inherently one-shot
external side effect), or keeping -provision-tenant as the sole real
actor and having it also sync its result into the CRD. Went with the
lighter option.
enterprise/internal/tenantcrd (new): a Syncer using the K8s dynamic
client (unstructured.Unstructured + a GroupVersionResource, not
deploy/operator's typed Tenant struct -- avoids a cross-module Go
dependency between two independently-versioned modules for one type).
Upserts the Tenant object, creates/updates a Secret with the *real*
ClickHouse credentials owned by that Tenant via an OwnerReference, then
patches status.{clickHouseDatabaseName,clickHouseSecretRef,
tantivyIndexPath}. Idempotent and safe to retry: never rotates a
credential across a re-sync, never overwrites a pre-existing
spec.displayName a human/GitOps process set.
cmd/enterprise-api/main.go's runProvisionTenant calls Sync when
TENANT_CRD_NAMESPACE is set (empty = no-op, same shape as every other
optional dependency in this codebase). Its "already active" refusal is
now split: ClickHouse re-provisioning is still refused (rotating a live
credential would break every open connection for no benefit), but CR
sync alone is now retryable using the credentials already on file in
rbacstore -- needed for retrying a previously-failed sync, or
backfilling CR sync for a tenant provisioned before this existed.
deploy/operator's reconciler rewritten to match: it never claims
PhaseActive on its own initiative anymore, only once
status.ClickHouseDatabaseName is non-empty (the field -provision-tenant,
and only -provision-tenant, sets). Phase is now a pure function of
{spec.suspended, status.ClickHouseDatabaseName != ""} recomputed every
reconcile, not toggled in place -- fixes a related bug the old code
would have hit once suspension was involved: un-suspending an
already-provisioned tenant needs to return straight to Active, which
isn't derivable from "last observed phase was Suspended" alone. The
reconciler no longer creates or manages any Secret, dropped its
`secrets` RBAC grant entirely, and gained zero new dependencies.
Helm chart: enterprise-api gets its own ServiceAccount/Role/RoleBinding
(get/list/create tenants, get/update/patch tenants/status, get/create/
update secrets -- least-privilege, scoped to the release namespace, not
a ClusterRole) and a TENANT_CRD_NAMESPACE env var, both gated on
tenantOperator.enabled. tenant-operator's ClusterRole loses the
secrets grant it no longer needs.
Verified in this environment: enterprise/internal/tenantcrd's tests run
against k8s.io/client-go's fake dynamic + typed clientsets (real client
library, fake transport, no cluster needed); deploy/operator's rewritten
tenant_controller_test.go runs against controller-runtime's fake
client, including new regression tests for the "must not claim Active
without confirmation" and "un-suspending returns to Active, not
Provisioning" properties; helm template + parsing the rendered YAML
confirms the RBAC split renders exactly as designed under both
tenantOperator.enabled=true/false. Not verified: an actual
-provision-tenant run against a real cluster with the operator watching
(no live cluster in this environment, same disclosed limitation as the
rest of /deploy). Docs updated in lockstep: CLAUDE.md, docs/architecture.md,
deploy/README.md, deploy/helm/sentry/README.md (including a corrected
"Trying the two-tenant example" walkthrough), phase-4-runbook.md (new
§11), enterprise/README.md. Also fixed two unrelated stale claims found
along the way: docs/architecture.md still said docker-compose.yml ran
plain api unconditionally (fixed in an earlier commit, doc not updated
then), and enterprise-api's own main.go doc comment still said Helm/
docker-compose wiring wasn't built yet.
deploy/operator
A small controller-runtime Operator managing one CRD: Tenant
(sentry.io/v1alpha1). See internal/controller/tenant_controller.go's
doc comment for exactly what it reconciles and -- just as importantly --
what it deliberately doesn't (no ClickHouse calls, no Tantivy filesystem
access, no enterprise/internal/rbacstore wiring; those are
enterprise/internal/tenantprovision, still unbuilt).
Not kubebuilder-scaffolded
No kubebuilder/controller-gen binary was available in this
environment, so this package is hand-written rather than generated:
api/v1alpha1/zz_generated.deepcopy.go-- normallycontroller-gen objectoutput; hand-written here, covered byapi/v1alpha1/api_test.go's round-trip tests (mutate a copy, assert the original is untouched -- exactly the class of bug a hand-writtenDeepCopyis prone to).config/crd/sentry.io_tenants.yaml-- normallycontroller-gen crdoutput from the+kubebuilder:validation:*markers onapi/v1alpha1/tenant_types.go; hand-written here and only as strong as keeping the two in sync by hand. Validated by strict-unmarshaling it into the realk8s.io/apiextensions-apiserverGo type (see/deploy/README.md's verification section) -- catches YAML/structural mistakes, not a drift between the CRD's field descriptions and the Go doc comments.+kubebuilder:rbacmarkers oninternal/controller/tenant_controller.goare present as documentation/intent (matching kubebuilder convention) but were never run throughcontroller-gen rbac-- the actual ClusterRole is hand-written in/deploy/helm/sentry/templates/tenant-operator.yaml, kept in sync with those markers by hand, same caveat as the CRD above.
Layout
api/v1alpha1/ Tenant, TenantSpec, TenantStatus -- the CRD's Go types
internal/controller/ TenantReconciler -- see its doc comment
cmd/tenant-operator/ main.go -- manager setup, matches every other
service's cmd/<name>/main.go convention in this repo
config/crd/ hand-written CRD YAML (see above)
Building & testing
go build ./...
go vet ./...
go test ./...
Tests use sigs.k8s.io/controller-runtime/pkg/client/fake, not
envtest -- envtest needs a real kube-apiserver/etcd binary pair
(setup-envtest) not available in this environment. The fake client
exercises real reconcile logic (object CRUD, owner references, status
writes) but not anything a real apiserver does for you (admission,
garbage collection, watch-triggered re-reconciliation) -- see
internal/controller/tenant_controller_test.go's doc comment.
docker build -f Dockerfile -t sentry-tenant-operator . # context is deploy/operator/, not the repo root
Not verified in this session -- see /deploy/README.md.
Trying it against a real cluster
kubectl apply -f config/crd/sentry.io_tenants.yaml
kubectl apply -f - <<'EOF'
apiVersion: sentry.io/v1alpha1
kind: Tenant
metadata:
name: acme
spec:
displayName: "Acme Corp"
EOF
kubectl get tenant acme -o yaml # status.phase should reach Active
kubectl get secret sentry-tenant-acme-clickhouse -o yaml