coffey-labs/cairnobs
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Unify the Tenant CRD with enterprise-api -provision-tenant (lightweight)

Browse Source 
Closes a gap named across CLAUDE.md/docs/architecture.md/deploy/README.md
since early Phase 4: the operator's Tenant CRD and -provision-tenant
were two disconnected mechanisms. The operator's reconciler generated a
K8s Secret with a locally-generated random password that authenticated
against nothing (nothing ever called ClickHouse to create a matching
user), and unconditionally claimed status.phase=Active the moment a
Tenant object existed -- actively misleading, not just incomplete.

Two unification shapes were considered (surfaced to the user via
AskUserQuestion, given the real difference in blast radius): the
operator's reconcile loop becoming a second real actor (new Postgres +
ClickHouse admin credentials flowing into the K8s controller, plus real
reconcile-loop idempotency/retry design for an inherently one-shot
external side effect), or keeping -provision-tenant as the sole real
actor and having it also sync its result into the CRD. Went with the
lighter option.

enterprise/internal/tenantcrd (new): a Syncer using the K8s dynamic
client (unstructured.Unstructured + a GroupVersionResource, not
deploy/operator's typed Tenant struct -- avoids a cross-module Go
dependency between two independently-versioned modules for one type).
Upserts the Tenant object, creates/updates a Secret with the *real*
ClickHouse credentials owned by that Tenant via an OwnerReference, then
patches status.{clickHouseDatabaseName,clickHouseSecretRef,
tantivyIndexPath}. Idempotent and safe to retry: never rotates a
credential across a re-sync, never overwrites a pre-existing
spec.displayName a human/GitOps process set.

cmd/enterprise-api/main.go's runProvisionTenant calls Sync when
TENANT_CRD_NAMESPACE is set (empty = no-op, same shape as every other
optional dependency in this codebase). Its "already active" refusal is
now split: ClickHouse re-provisioning is still refused (rotating a live
credential would break every open connection for no benefit), but CR
sync alone is now retryable using the credentials already on file in
rbacstore -- needed for retrying a previously-failed sync, or
backfilling CR sync for a tenant provisioned before this existed.

deploy/operator's reconciler rewritten to match: it never claims
PhaseActive on its own initiative anymore, only once
status.ClickHouseDatabaseName is non-empty (the field -provision-tenant,
and only -provision-tenant, sets). Phase is now a pure function of
{spec.suspended, status.ClickHouseDatabaseName != ""} recomputed every
reconcile, not toggled in place -- fixes a related bug the old code
would have hit once suspension was involved: un-suspending an
already-provisioned tenant needs to return straight to Active, which
isn't derivable from "last observed phase was Suspended" alone. The
reconciler no longer creates or manages any Secret, dropped its
`secrets` RBAC grant entirely, and gained zero new dependencies.

Helm chart: enterprise-api gets its own ServiceAccount/Role/RoleBinding
(get/list/create tenants, get/update/patch tenants/status, get/create/
update secrets -- least-privilege, scoped to the release namespace, not
a ClusterRole) and a TENANT_CRD_NAMESPACE env var, both gated on
tenantOperator.enabled. tenant-operator's ClusterRole loses the
secrets grant it no longer needs.

Verified in this environment: enterprise/internal/tenantcrd's tests run
against k8s.io/client-go's fake dynamic + typed clientsets (real client
library, fake transport, no cluster needed); deploy/operator's rewritten
tenant_controller_test.go runs against controller-runtime's fake
client, including new regression tests for the "must not claim Active
without confirmation" and "un-suspending returns to Active, not
Provisioning" properties; helm template + parsing the rendered YAML
confirms the RBAC split renders exactly as designed under both
tenantOperator.enabled=true/false. Not verified: an actual
-provision-tenant run against a real cluster with the operator watching
(no live cluster in this environment, same disclosed limitation as the
rest of /deploy). Docs updated in lockstep: CLAUDE.md, docs/architecture.md,
deploy/README.md, deploy/helm/sentry/README.md (including a corrected
"Trying the two-tenant example" walkthrough), phase-4-runbook.md (new
§11), enterprise/README.md. Also fixed two unrelated stale claims found
along the way: docs/architecture.md still said docker-compose.yml ran
plain api unconditionally (fixed in an earlier commit, doc not updated
then), and enterprise-api's own main.go doc comment still said Helm/
docker-compose wiring wasn't built yet.
This commit is contained in:
jcoffey-dev
2026-08-14 09:07:10 -07:00
parent 8d7326fc6a
commit 823f5d48d1
18 changed files with 1073 additions and 323 deletions
Download Patch File Download Diff File Expand all files Collapse all files
deploy/README.md
+51 -19
View File
@@ -21,26 +21,43 @@ map of per-tenant ClickHouse connection pools via `internal/chrunner`;
that's an explicit Phase 4 non-goal (see `/CLAUDE.md`). What it *does*
add:
- A `Tenant` CRD + controller that generates and manages one dedicated
ClickHouse credential Secret per tenant (`operator/internal/controller`).
- A `Tenant` CRD + controller (`operator/internal/controller`) that
reflects real provisioning state onto `status.phase`/a `Ready`
condition, derived from whether `enterprise-api -provision-tenant` has
reported real ClickHouse provisioning.
- A Helm chart that can install zero-or-more `Tenant` CRs
(`values.tenants`) alongside the rest of the stack, and — the newer
piece — swaps `api`'s Deployment for `enterprise-api`'s whenever
`enterprise.enabled` is true, so which query binary actually serves
traffic is no longer a separately-forgettable decision (see
`helm/sentry/README.md`'s "`api` vs `enterprise-api`" section).
(`values.tenants`) alongside the rest of the stack, and swaps `api`'s
Deployment for `enterprise-api`'s whenever `enterprise.enabled` is
true, so which query binary actually serves traffic is no longer a
separately-forgettable decision (see `helm/sentry/README.md`'s "`api`
vs `enterprise-api`" section).
**Two still-separate mechanisms, not yet unified**: the Operator's
`Tenant` CRD manages only the K8s-side credential Secret — it does not
call ClickHouse (no `CREATE DATABASE`/`CREATE USER`/`GRANT`) or touch
the Tantivy filesystem. `enterprise-api -provision-tenant=<id>` is what
actually does that (`enterprise/internal/tenantprovision`, built and
tested — see `/enterprise/README.md`), driven independently via
`rbacstore`, not from the `Tenant` CRD's reconcile loop. A `Tenant`
reaching `status.phase: Active` here means "this tenant has a K8s
Secret," not "this tenant's ClickHouse database/grants exist" — running
both mechanisms for the same tenant ID today requires two separate
operator actions, named explicitly rather than implied to be one.
**Now unified, in a deliberately lightweight way**: `enterprise-api
-provision-tenant=<id>` stays the sole real actor — it's the only thing
that calls ClickHouse (`CREATE DATABASE`/`CREATE USER`/`GRANT`, via
`enterprise/internal/tenantprovision`) and writes `rbacstore`. What
changed: once it succeeds, it also syncs the result into the `Tenant`
CRD (`enterprise/internal/tenantcrd`) — creating the Secret with *real*
credentials (the controller no longer generates a placeholder one that
authenticated against nothing) and setting the status fields the
controller reads to compute `Phase`/`Ready`. The controller itself
gained no new credentials and still never touches ClickHouse/Postgres --
it's a pure function of `spec.suspended` and whatever
`-provision-tenant` has reported, never an independent second guess at
"is this tenant really provisioned." A `Tenant` reaching
`status.phase: Active` now means the same thing `rbacstore.tenants.
status='active'` does, not two different claims — see
`enterprise/internal/tenantcrd`'s and `operator/internal/controller/
tenant_controller.go`'s doc comments for the full split, and
`enterprise-api -provision-tenant`'s `TENANT_CRD_NAMESPACE` env var
(set automatically by the Helm chart when `tenantOperator.enabled`) to
turn this on. Deliberately not built: the operator's reconcile loop
itself calling ClickHouse/rbacstore directly (a "full unification"
option considered and set aside — it would give the operator two new
credential sets and require real reconcile-loop idempotency design for
an inherently one-shot external side effect, a bigger and riskier
change than this repo's provisioning story needed to close the actual
gap, which was two *disconnected* sources of truth, not two actors).
## Verification status -- read before trusting this against a real cluster
@@ -59,6 +76,15 @@ access was available to fetch these tools, but no cluster):
(`internal/controller/tenant_controller_test.go`) -- real reconcile
logic exercised, but not against a real apiserver (no `envtest`
binaries available; see that test file's doc comment).
- `enterprise/internal/tenantcrd` (the "lightweight unification"
half `-provision-tenant` runs): `go test` passes against
`k8s.io/client-go`'s fake dynamic and typed clientsets -- real client
library, fake transport, same shape as `enterprise/internal/
searchclient`'s in-process gRPC tests. What this doesn't prove: that
`sentry.io/v1alpha1.Tenant`'s real CRD schema (a real apiserver's
OpenAPI validation) accepts exactly what this package writes -- the
`helm template`/kubeconform check below covers the schema shape, not
a live write against it.
- `deploy/operator/config/crd/sentry.io_tenants.yaml`: parsed with
`sigs.k8s.io/yaml` + strict-unmarshaled into the real
`k8s.io/apiextensions-apiserver` `CustomResourceDefinition` Go type --
@@ -76,7 +102,13 @@ access was available to fetch these tools, but no cluster):
parsing the rendered YAML (not just eyeballing it): exactly one
`Deployment`/`Service` named `sentry-api` renders in each mode, with
the `enterprise.enabled: true` render using the `enterprise-api` image
and the default render using plain `api`'s.
and the default render using plain `api`'s. Also confirmed for the
`tenantOperator.enabled: true` case: `enterprise-api` gets its own
ServiceAccount/Role/RoleBinding (exactly `tenants`/`tenants/status`/
`secrets`, no more), `tenant-operator`'s own ClusterRole no longer
grants `secrets` at all, and `TENANT_CRD_NAMESPACE` is set on
`enterprise-api`'s container only when `tenantOperator.enabled` is
true.
- Docker image builds (`operator/Dockerfile` and every other
`Dockerfile` this chart references) were **not** verified in this
session -- Docker's daemon wasn't reachable here either (see the
deploy/helm/sentry/README.md
+28 -11
View File
@@ -72,20 +72,37 @@ helm install sentry . --include-crds \
--set 'tenants[1].name=globex' --set 'tenants[1].displayName=Globex Corporation'
kubectl get tenants
# expect: both Provisioning -- the Tenant CRs above are just a
# declarative request; nothing has actually provisioned ClickHouse for
# either yet (see below).
kubectl exec -it deploy/sentry-api -- /enterprise-api -provision-tenant=acme -display-name="Acme Corp"
kubectl exec -it deploy/sentry-api -- /enterprise-api -provision-tenant=globex -display-name="Globex Corporation"
kubectl get tenants
# expect: both Active now.
kubectl get secret sentry-tenant-acme-clickhouse sentry-tenant-globex-clickhouse
```
This proves the K8s-side half of Phase 4's "two tenants... with their
own users, roles, dashboards" exit criteria (`/CLAUDE.md`) -- a real
per-tenant credential Secret exists for each, generated by
`deploy/operator`'s `Tenant` controller. It does **not** by itself give
either tenant a working ClickHouse database or Tantivy index -- that
needs `enterprise-api -provision-tenant=<id>` (a separate, deliberately
manual operator action; the `Tenant` CRD and `-provision-tenant` are two
independent mechanisms today, not yet unified -- see
`/enterprise/README.md`), and OIDC login (built, but still needs a
manual `tenant_memberships` row -- see `/docs/phase-4-runbook.md` §3a)
before a human can actually query as that tenant.
This proves Phase 4's "two tenants... with their own users, roles,
dashboards" exit criteria (`/CLAUDE.md`) end to end at the deployment-
topology layer: `-provision-tenant` (`enterprise/internal/
tenantprovision`) is what actually creates each tenant's ClickHouse
database/user/grant and marks it active in `rbacstore`; running inside
the `enterprise-api` Deployment's Pod means it automatically syncs that
real result into the `Tenant` CRD too (`enterprise/internal/tenantcrd`,
via the ServiceAccount/Role `tenantOperator.enabled` also grants that
Deployment) -- the credential Secret you see above has *real*
credentials, not a placeholder, and `Tenant.status.phase: Active` means
the same thing `rbacstore.tenants.status='active'` does, not two
different claims about two different systems. The `Tenant` CRD and
`-provision-tenant` used to be genuinely disconnected (a Secret existed
the moment the CR was created, with a password that authenticated
against nothing) -- see `/deploy/README.md`'s "lightweight unification"
section for the full history. OIDC/SAML login still needs a manual
`tenant_memberships` grant (`enterprise-auth -grant-membership-*` --
see `/docs/phase-4-runbook.md` §3a/§3b) before a human can actually
query as either tenant.
## `web`'s image needs rebuilding per environment
deploy/helm/sentry/templates/enterprise-api.yaml
+61
View File
@@ -12,6 +12,54 @@ routes to whichever Deployment is actually rendered, with zero
conditional logic needed in any consumer (alerting, web).
*/}}
{{- if .Values.enterprise.enabled }}
{{- if .Values.tenantOperator.enabled }}
# Grants enterprise-api's -provision-tenant (enterprise/internal/
# tenantcrd) permission to sync real provisioning results into the
# Tenant CRD -- a Role, not a ClusterRole (unlike tenant-operator's:
# this binary only ever provisions tenants that live in its own release
# namespace, no reason to widen it), scoped to exactly the two resource
# types tenantcrd.Syncer touches. Only rendered when tenantOperator is
# also enabled -- no Tenant CRD installed, nothing to sync into.
apiVersion: v1
kind: ServiceAccount
metadata:
name: {{ .Release.Name }}-enterprise-api
labels:
{{- include "sentry.labels" . | nindent 4 }}
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: {{ .Release.Name }}-enterprise-api
labels:
{{- include "sentry.labels" . | nindent 4 }}
rules:
- apiGroups: ["sentry.io"]
resources: ["tenants"]
verbs: ["get", "list", "create"]
- apiGroups: ["sentry.io"]
resources: ["tenants/status"]
verbs: ["get", "update", "patch"]
- apiGroups: [""]
resources: ["secrets"]
verbs: ["get", "create", "update"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: {{ .Release.Name }}-enterprise-api
labels:
{{- include "sentry.labels" . | nindent 4 }}
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: {{ .Release.Name }}-enterprise-api
subjects:
- kind: ServiceAccount
name: {{ .Release.Name }}-enterprise-api
namespace: {{ .Release.Namespace }}
---
{{- end }}
apiVersion: apps/v1
kind: Deployment
metadata:
@@ -30,6 +78,9 @@ spec:
labels:
{{- include "sentry.selectorLabels" (list $ "api") | nindent 8 }}
spec:
{{- if .Values.tenantOperator.enabled }}
serviceAccountName: {{ .Release.Name }}-enterprise-api
{{- end }}
initContainers:
{{- include "sentry.waitForTCP" (list "clickhouse" (printf "%s-clickhouse" .Release.Name) "9000") | nindent 8 }}
{{- include "sentry.waitForTCP" (list "postgres" (printf "%s-postgres" .Release.Name) "5432") | nindent 8 }}
@@ -84,6 +135,16 @@ spec:
key: auditWriterPassword
- name: ENTERPRISE_AUTH_URL
value: "http://{{ .Release.Name }}-enterprise-auth:8082"
{{- if .Values.tenantOperator.enabled }}
# Enables enterprise/internal/tenantcrd -- -provision-tenant
# (run via `kubectl exec` into this Deployment's Pod, using
# its ServiceAccount/Role above) syncs real provisioning
# results into the Tenant CRD this namespace's tenants live
# in. Unset (the default, when tenantOperator isn't enabled)
# is a documented no-op -- see apiconfig.Config.TenantCRDNamespace.
- name: TENANT_CRD_NAMESPACE
value: {{ .Release.Namespace | quote }}
{{- end }}
ports:
- name: http
containerPort: 8080
deploy/helm/sentry/templates/tenant-operator.yaml
+6 -5
View File
@@ -11,8 +11,12 @@ metadata:
# doesn't assume it's the only namespace the operator might one day watch
# -- narrowed to exactly the two resource types
# deploy/operator/internal/controller/tenant_controller.go's
# +kubebuilder:rbac markers name (tenants, tenants/status, secrets), not
# a wildcard grant.
# +kubebuilder:rbac markers name (tenants, tenants/status), not a
# wildcard grant. No `secrets` permission -- this controller stopped
# managing the ClickHouse credential Secret once enterprise-api
# -provision-tenant took over creating it with real credentials (see
# that controller's doc comment); see enterprise-api.yaml's own
# ServiceAccount/Role for the `secrets` grant that binary needs instead.
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
@@ -26,9 +30,6 @@ rules:
- apiGroups: ["sentry.io"]
resources: ["tenants/status"]
verbs: ["get", "update", "patch"]
- apiGroups: [""]
resources: ["secrets"]
verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
deploy/helm/sentry/values.yaml
+15 -8
View File
@@ -165,9 +165,11 @@ enterprise:
idpMetadataURL: ""
# Installs deploy/operator (the Tenant CRD controller) alongside this
# chart. Only meaningful when enterprise.enabled is also true --
# gated on that, not a separate flag, since a Tenant CR with no
# enterprise-auth deployed to consume its Secret has nothing to do.
# chart. Only meaningful when enterprise.enabled is also true -- gated
# on that, not a separate flag, since a Tenant CR with nothing to
# reflect (see below) has nothing to do. Also turns on enterprise-api's
# own Tenant-CRD-syncing permissions (a ServiceAccount/Role, and the
# TENANT_CRD_NAMESPACE env var) -- see templates/enterprise-api.yaml.
tenantOperator:
enabled: false
image:
@@ -176,11 +178,16 @@ tenantOperator:
resources: {}
# One entry per tenant to provision -- rendered as Tenant CRs
# (templates/tenants.yaml), reconciled by the tenant-operator into a
# per-tenant ClickHouse credential Secret. See
# deploy/operator/internal/controller/tenant_controller.go's doc comment
# for exactly what that does and doesn't set up. Empty by default; a
# real two-tenant deployment (Phase 4's exit criteria) sets e.g.:
# (templates/tenants.yaml), a declarative request an admin/GitOps
# process makes. The operator (tenant-operator, above) only ever
# *reflects* real state onto these objects (Phase/Conditions, derived
# from what enterprise-api's `-provision-tenant` has reported) -- it's
# `-provision-tenant` (run via `kubectl exec` into the enterprise-api
# Pod), not the operator, that actually calls ClickHouse and writes the
# credential Secret. See deploy/operator/internal/controller/
# tenant_controller.go's and enterprise/internal/tenantcrd's doc
# comments for the full split. Empty by default; a real two-tenant
# deployment (Phase 4's exit criteria) sets e.g.:
# tenants:
# - name: acme
# displayName: "Acme Corp"
deploy/operator/api/v1alpha1/tenant_types.go
+42 -17
View File
@@ -8,15 +8,20 @@ import (
// /docs/phase-4-isolation-design.md: every tenant-resolution path
// elsewhere must refuse to serve a tenant not in PhaseActive, checked
// server-side (today, against enterprise/internal/rbacstore's tenants
// table -- this CR is a K8s-native *view* of the same state machine at
// the deployment-topology layer, not a second source of truth. Reconciling
// the two together is exactly the kind of tenant-provisioning wiring
// named as deferred in /docs/phase-4-runbook.md's task 6 section: today
// this operator only manages the K8s-side artifact (a per-tenant
// ClickHouse credential Secret + a ConfigMap recording the tenant's
// database name/index path), not the actual `CREATE DATABASE`/`CREATE
// USER`/`GRANT` calls against ClickHouse -- that's
// enterprise/internal/tenantprovision, still unbuilt.
// table -- this CR is a K8s-native *view* of that same state machine,
// kept honest rather than a second, independently-guessed source of
// truth. `enterprise-api -provision-tenant` (the actual actor -- real
// `CREATE DATABASE`/`CREATE USER`/`GRANT` calls against ClickHouse, and
// the real rbacstore writes) is the only writer of
// TenantStatus.ClickHouseDatabaseName/ClickHouseSecretRef/
// TantivyIndexPath, once real provisioning succeeds -- see
// internal/controller/tenant_controller.go's doc comment for how
// TenantReconciler derives Phase/Conditions from those fields rather
// than fabricating its own "provisioned" claim. This is the "lightweight
// unification" named in CLAUDE.md/docs/phase-4-runbook.md's "two
// independent provisioning mechanisms" gap: -provision-tenant stays the
// real actor; this operator's reconcile loop never touches Postgres or
// ClickHouse and gained no new credentials.
type TenantPhase string
const (
@@ -47,23 +52,43 @@ type TenantSpec struct {
Suspended bool `json:"suspended,omitempty"`
}
// TenantStatus is observed state -- only the controller writes this.
// TenantStatus is observed state, with split ownership as of the
// "lightweight unification" (see TenantPhase's doc comment):
// ClickHouseDatabaseName/ClickHouseSecretRef/TantivyIndexPath are
// written only by enterprise-api's -provision-tenant, once real
// ClickHouse provisioning actually succeeds -- this controller only
// reads them (to compute Phase/Conditions) and never invents a value
// for them. Phase/Conditions/ObservedGeneration remain
// controller-written, computed fresh on every reconcile from Spec plus
// whatever -provision-tenant has (or hasn't) reported.
type TenantStatus struct {
// +optional
Phase TenantPhase `json:"phase,omitempty"`
// ClickHouseDatabaseName is derived (today: same as the Tenant's own
// Name) rather than settable in Spec -- see task 2's design: no
// tenant traffic authenticates as ClickHouse's `default` user, and a
// ClickHouseDatabaseName is set by enterprise-api -provision-tenant
// once ClickHouse provisioning for this tenant actually succeeds --
// empty means "not yet provisioned," which TenantReconciler reads as
// PhaseProvisioning (see tenant_controller.go). Today it's always
// equal to the Tenant's own Name (see task 2's design: no tenant
// traffic authenticates as ClickHouse's `default` user, and a
// database name that could diverge from the tenant identifier is a
// bookkeeping foot-gun this type avoids by construction.
// bookkeeping foot-gun this type avoids by construction) but isn't
// itself computed by this package -- -provision-tenant sets it
// directly from what it actually created.
// +optional
ClickHouseDatabaseName string `json:"clickHouseDatabaseName,omitempty"`
// ClickHouseSecretRef names the Secret (same namespace) holding this
// tenant's dedicated, narrowly-granted ClickHouse credentials -- see
// tenant_controller.go's reconcileSecret. Never the cluster-wide
// CLICKHOUSE_PASSWORD docker-compose.yml uses today.
// tenant's dedicated, narrowly-granted ClickHouse credentials --
// created by enterprise-api -provision-tenant (enterprise/internal/
// tenantcrd), owned by this Tenant object via an OwnerReference so
// K8s garbage-collects it on Tenant deletion regardless of which
// process created it. This controller no longer creates or manages
// any Secret itself -- see tenant_controller.go's doc comment for
// why a controller-generated placeholder credential (the pre-
// unification behavior) was actively misleading, not just
// incomplete. Never the cluster-wide CLICKHOUSE_PASSWORD
// docker-compose.yml uses today.
// +optional
ClickHouseSecretRef string `json:"clickHouseSecretRef,omitempty"`
deploy/operator/internal/controller/tenant_controller.go
+52 -131
View File
@@ -1,42 +1,39 @@
// Package controller reconciles the Tenant CRD (api/v1alpha1) into the
// K8s-native artifacts task 2/CLAUDE.md's Phase 4 exit criteria calls
// for: "real per-tenant secret management (replacing today's single
// shared CLICKHOUSE_PASSWORD)" -- see docker-compose.yml's
// CLICKHOUSE_PASSWORD comment for what that shared-secret shape looks
// like today.
// Package controller reconciles the Tenant CRD (api/v1alpha1) -- see
// TenantPhase's doc comment for the "lightweight unification" this
// controller is one half of. This reconciler never calls ClickHouse (no
// CREATE DATABASE/CREATE USER/GRANT), never touches the Tantivy index
// filesystem, and never talks to enterprise/internal/rbacstore -- those
// stay enterprise-api -provision-tenant's job (enterprise/internal/
// tenantprovision, enterprise/internal/tenantcrd). This controller's
// job is purely a function of what's already on the object: derive
// Phase and the Ready condition from Spec.Suspended and whether
// Status.ClickHouseDatabaseName has been set by -provision-tenant,
// nothing more. A Tenant reaching PhaseActive here is exactly the same
// claim as rbacstore's tenants.status='active' now, not a
// second, independently-computed one -- see docs/phase-4-isolation-design.md.
//
// What this reconciler does NOT do, named explicitly rather than
// implied: it never calls ClickHouse (no CREATE DATABASE/CREATE USER/
// GRANT), never touches the Tantivy index filesystem, and never talks to
// enterprise/internal/rbacstore. Those are enterprise/internal/
// tenantprovision's job -- unbuilt, per the task 5 summary. This
// controller's job stops at "does a K8s Secret with this tenant's
// ClickHouse credentials exist, and does the Tenant's status reflect
// that" -- the deployment-topology half of tenant provisioning, not the
// database-side half. A Tenant reaching PhaseActive here is NOT the same
// claim as rbacstore's tenants.status='active' (the actual gate every
// tenant-resolution code path checks per
// /docs/phase-4-isolation-design.md) -- reconciling those two into one
// state machine is exactly the kind of follow-up work
// /docs/phase-4-runbook.md's task 6 section names as deferred.
// Earlier versions of this controller also generated a per-tenant
// ClickHouse credential Secret with a locally-generated random
// password. That Secret authenticated against nothing (nothing in this
// controller ever called ClickHouse to create a matching user) and
// nothing else in the codebase ever read it -- a placeholder that
// actively misled ("looks provisioned") rather than one that honestly
// represented "not yet provisioned." Removed rather than fixed in
// place: the real Secret, with real credentials, is now created by
// -provision-tenant (enterprise/internal/tenantcrd) once ClickHouse
// provisioning actually succeeds.
package controller
import (
"context"
"crypto/rand"
"encoding/base64"
"fmt"
corev1 "k8s.io/api/core/v1"
apierrors "k8s.io/apimachinery/pkg/api/errors"
"k8s.io/apimachinery/pkg/api/meta"
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
"k8s.io/apimachinery/pkg/runtime"
"k8s.io/apimachinery/pkg/types"
ctrl "sigs.k8s.io/controller-runtime"
"sigs.k8s.io/controller-runtime/pkg/client"
"sigs.k8s.io/controller-runtime/pkg/controller/controllerutil"
"sigs.k8s.io/controller-runtime/pkg/log"
sentryv1alpha1 "github.com/sentry/sentry/deploy/operator/api/v1alpha1"
)
@@ -47,77 +44,52 @@ type TenantReconciler struct {
Scheme *runtime.Scheme
}
// clickHouseSecretName is deterministic from the tenant name -- never
// randomly suffixed -- so a re-run of Reconcile (or a controller
// restart) finds the same Secret it created before, rather than losing
// track of it and creating a second one.
func clickHouseSecretName(tenant *sentryv1alpha1.Tenant) string {
return fmt.Sprintf("sentry-tenant-%s-clickhouse", tenant.Name)
}
// tantivyIndexPath mirrors /docs/phase-4-isolation-design.md's Tantivy
// section: one directory per tenant under the shared search-index
// volume (search-index-data in docker-compose.yml; a PVC in the Helm
// chart -- see deploy/helm/sentry/templates/search-deployment.yaml).
func tantivyIndexPath(tenant *sentryv1alpha1.Tenant) string {
return "/var/lib/sentry-search/tenants/" + tenant.Name
}
// generatePassword returns a 32-byte random value, base64-encoded --
// same "narrowly-granted, per-tenant, never the shared default user"
// framing as /docs/phase-4-isolation-design.md's ClickHouse section,
// applied to how the credential itself is generated (crypto/rand, not
// math/rand -- this becomes a real ClickHouse user's password once
// internal/tenantprovision consumes it).
func generatePassword() (string, error) {
buf := make([]byte, 32)
if _, err := rand.Read(buf); err != nil {
return "", fmt.Errorf("generating password: %w", err)
}
return base64.RawURLEncoding.EncodeToString(buf), nil
}
// +kubebuilder:rbac:groups=sentry.io,resources=tenants,verbs=get;list;watch;create;update;patch;delete
// +kubebuilder:rbac:groups=sentry.io,resources=tenants/status,verbs=get;update;patch
// +kubebuilder:rbac:groups="",resources=secrets,verbs=get;list;watch;create;update;patch;delete
func (r *TenantReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
logger := log.FromContext(ctx)
var tenant sentryv1alpha1.Tenant
if err := r.Get(ctx, req.NamespacedName, &tenant); err != nil {
if apierrors.IsNotFound(err) {
// Deleted -- owned Secret is garbage-collected by K8s via
// its OwnerReference (set in reconcileSecret below), nothing
// else to clean up at this layer. See this file's package
// doc comment: real deprovisioning (revoking ClickHouse
// grants) isn't this controller's job.
// Deleted -- the owned Secret -provision-tenant created (if
// any) is garbage-collected by K8s via its OwnerReference,
// nothing else to clean up at this layer. Real
// deprovisioning (revoking ClickHouse grants) isn't this
// controller's job, or -provision-tenant's today -- see
// /docs/security/threat-model.md's non-goals.
return ctrl.Result{}, nil
}
return ctrl.Result{}, fmt.Errorf("getting tenant: %w", err)
}
secretName, err := r.reconcileSecret(ctx, &tenant)
if err != nil {
logger.Error(err, "reconciling clickhouse secret")
return ctrl.Result{}, err
// provisioned is true once -provision-tenant has confirmed real
// ClickHouse provisioning by setting this field -- see
// TenantStatus.ClickHouseDatabaseName's doc comment. This
// controller treats it as the sole source of truth for "has
// provisioning actually happened," never claiming PhaseActive on
// its own say-so the way the pre-unification version did.
provisioned := tenant.Status.ClickHouseDatabaseName != ""
condStatus := metav1.ConditionFalse
reason, message := "AwaitingProvisioning", "waiting for enterprise-api -provision-tenant to provision ClickHouse for this tenant"
switch {
case tenant.Spec.Suspended:
tenant.Status.Phase = sentryv1alpha1.PhaseSuspended
reason, message = "Suspended", "tenant is suspended (spec.suspended=true)"
case provisioned:
tenant.Status.Phase = sentryv1alpha1.PhaseActive
condStatus = metav1.ConditionTrue
reason, message = "Provisioned", fmt.Sprintf("ClickHouse database %q is provisioned", tenant.Status.ClickHouseDatabaseName)
default:
tenant.Status.Phase = sentryv1alpha1.PhaseProvisioning
}
desiredPhase := sentryv1alpha1.PhaseActive
if tenant.Spec.Suspended {
desiredPhase = sentryv1alpha1.PhaseSuspended
}
tenant.Status.ClickHouseDatabaseName = tenant.Name
tenant.Status.ClickHouseSecretRef = secretName
tenant.Status.TantivyIndexPath = tantivyIndexPath(&tenant)
tenant.Status.Phase = desiredPhase
tenant.Status.ObservedGeneration = tenant.Generation
meta.SetStatusCondition(&tenant.Status.Conditions, metav1.Condition{
Type: sentryv1alpha1.ConditionReady,
Status: metav1.ConditionTrue,
Reason: "SecretReconciled",
Message: fmt.Sprintf("ClickHouse credential secret %q is present", secretName),
Status: condStatus,
Reason: reason,
Message: message,
ObservedGeneration: tenant.Generation,
})
@@ -128,59 +100,8 @@ func (r *TenantReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctr
return ctrl.Result{}, nil
}
// reconcileSecret creates the tenant's ClickHouse credential Secret if
// it doesn't already exist. Deliberately never updates an existing
// Secret's password -- rotating a live tenant's ClickHouse credential
// out from under it (without first updating the ClickHouse-side grant,
// which this controller doesn't do) would just break every open
// connection for no benefit; credential rotation is real future work
// that needs to be coordinated with internal/tenantprovision, not
// something this reconcile loop can safely do alone.
func (r *TenantReconciler) reconcileSecret(ctx context.Context, tenant *sentryv1alpha1.Tenant) (string, error) {
name := clickHouseSecretName(tenant)
var existing corev1.Secret
err := r.Get(ctx, types.NamespacedName{Namespace: tenant.Namespace, Name: name}, &existing)
if err == nil {
return name, nil
}
if !apierrors.IsNotFound(err) {
return "", fmt.Errorf("getting secret: %w", err)
}
password, err := generatePassword()
if err != nil {
return "", err
}
secret := &corev1.Secret{
ObjectMeta: metav1.ObjectMeta{
Name: name,
Namespace: tenant.Namespace,
Labels: map[string]string{
"app.kubernetes.io/managed-by": "sentry-tenant-operator",
"sentry.io/tenant": tenant.Name,
},
},
Type: corev1.SecretTypeOpaque,
StringData: map[string]string{
"username": "tenant_" + tenant.Name,
"password": password,
"database": tenant.Name,
},
}
if err := controllerutil.SetControllerReference(tenant, secret, r.Scheme); err != nil {
return "", fmt.Errorf("setting owner reference: %w", err)
}
if err := r.Create(ctx, secret); err != nil {
return "", fmt.Errorf("creating secret: %w", err)
}
return name, nil
}
func (r *TenantReconciler) SetupWithManager(mgr ctrl.Manager) error {
return ctrl.NewControllerManagedBy(mgr).
For(&sentryv1alpha1.Tenant{}).
Owns(&corev1.Secret{}).
Complete(r)
}
deploy/operator/internal/controller/tenant_controller_test.go
+96 -74
View File
@@ -3,10 +3,9 @@
// real kube-apiserver/etcd binary pair (setup-envtest) that isn't
// available in this environment (see package doc comment and
// deploy/README.md's verification section). A fake client exercises
// Reconcile's actual logic (object CRUD, owner references, status
// writes) against an in-memory tracker; what it can't exercise is
// anything a real apiserver would do for you (defaulting, admission,
// actual garbage collection of owned objects, watch-triggered
// Reconcile's actual logic (status derivation, condition writes) against
// an in-memory tracker; what it can't exercise is anything a real
// apiserver would do for you (defaulting, admission, watch-triggered
// re-reconciliation) -- so passing here is real signal about this
// reconciler's logic, not proof it behaves correctly against a live
// cluster.
@@ -51,98 +50,112 @@ func testTenant(name string, suspended bool) *sentryv1alpha1.Tenant {
}
}
func TestReconcileCreatesSecretAndSetsActivePhase(t *testing.T) {
func reconcile(t *testing.T, r *TenantReconciler, name string) sentryv1alpha1.Tenant {
t.Helper()
ctx := context.Background()
if _, err := r.Reconcile(ctx, ctrl.Request{NamespacedName: types.NamespacedName{Name: name, Namespace: "default"}}); err != nil {
t.Fatalf("Reconcile: %v", err)
}
var got sentryv1alpha1.Tenant
if err := r.Get(ctx, types.NamespacedName{Name: name, Namespace: "default"}, &got); err != nil {
t.Fatalf("getting tenant: %v", err)
}
return got
}
// TestReconcileUnprovisionedTenantIsProvisioningNotActive is the
// regression test for the pre-unification bug: this controller must
// never claim PhaseActive just because a Tenant object exists -- only
// enterprise-api -provision-tenant setting Status.ClickHouseDatabaseName
// (real ClickHouse provisioning having actually succeeded) earns that.
func TestReconcileUnprovisionedTenantIsProvisioningNotActive(t *testing.T) {
tenant := testTenant("acme", false)
r := newFakeReconciler(t, tenant)
got := reconcile(t, r, "acme")
if got.Status.Phase != sentryv1alpha1.PhaseProvisioning {
t.Fatalf("Phase = %q, want Provisioning (nothing has provisioned this tenant yet)", got.Status.Phase)
}
cond := readyCondition(got)
if cond == nil || cond.Status != metav1.ConditionFalse {
t.Fatalf("Ready condition = %+v, want status False", cond)
}
}
// TestReconcileReflectsProvisioningStateProvisionTenantSets is the core
// "lightweight unification" behavior: once something external (in
// production, -provision-tenant; here, simulated directly against the
// fake client the way a real K8s API server write would land) sets
// ClickHouseDatabaseName, this controller must report PhaseActive.
func TestReconcileReflectsProvisioningStateProvisionTenantSets(t *testing.T) {
tenant := testTenant("acme", false)
r := newFakeReconciler(t, tenant)
ctx := context.Background()
if _, err := r.Reconcile(ctx, ctrl.Request{NamespacedName: types.NamespacedName{Name: "acme", Namespace: "default"}}); err != nil {
t.Fatalf("Reconcile: %v", err)
}
var secret corev1.Secret
if err := r.Get(ctx, types.NamespacedName{Name: "sentry-tenant-acme-clickhouse", Namespace: "default"}, &secret); err != nil {
t.Fatalf("expected a ClickHouse secret to be created: %v", err)
}
if secret.StringData["username"] != "tenant_acme" || secret.StringData["database"] != "acme" {
t.Fatalf("unexpected secret data: %+v", secret.StringData)
}
if secret.StringData["password"] == "" {
t.Fatal("expected a non-empty generated password")
}
if len(secret.OwnerReferences) != 1 || secret.OwnerReferences[0].Name != "acme" {
t.Fatalf("expected secret to be owned by the Tenant, got %+v", secret.OwnerReferences)
}
var got sentryv1alpha1.Tenant
if err := r.Get(ctx, types.NamespacedName{Name: "acme", Namespace: "default"}, &got); err != nil {
var toUpdate sentryv1alpha1.Tenant
if err := r.Get(ctx, types.NamespacedName{Name: "acme", Namespace: "default"}, &toUpdate); err != nil {
t.Fatalf("getting tenant: %v", err)
}
toUpdate.Status.ClickHouseDatabaseName = "acme"
toUpdate.Status.ClickHouseSecretRef = "sentry-tenant-acme-clickhouse"
toUpdate.Status.TantivyIndexPath = "/var/lib/sentry-search/tenants/acme"
if err := r.Status().Update(ctx, &toUpdate); err != nil {
t.Fatalf("simulating -provision-tenant's status write: %v", err)
}
got := reconcile(t, r, "acme")
if got.Status.Phase != sentryv1alpha1.PhaseActive {
t.Fatalf("Phase = %q, want Active", got.Status.Phase)
}
if got.Status.ClickHouseDatabaseName != "acme" {
t.Fatalf("ClickHouseDatabaseName = %q, want acme", got.Status.ClickHouseDatabaseName)
if got.Status.ClickHouseDatabaseName != "acme" || got.Status.ClickHouseSecretRef != "sentry-tenant-acme-clickhouse" || got.Status.TantivyIndexPath != "/var/lib/sentry-search/tenants/acme" {
t.Fatalf("reconcile must not clobber the fields -provision-tenant set: %+v", got.Status)
}
if got.Status.ClickHouseSecretRef != "sentry-tenant-acme-clickhouse" {
t.Fatalf("ClickHouseSecretRef = %q", got.Status.ClickHouseSecretRef)
}
if got.Status.TantivyIndexPath != "/var/lib/sentry-search/tenants/acme" {
t.Fatalf("TantivyIndexPath = %q", got.Status.TantivyIndexPath)
cond := readyCondition(got)
if cond == nil || cond.Status != metav1.ConditionTrue {
t.Fatalf("Ready condition = %+v, want status True", cond)
}
}
func TestReconcileSuspendedSetsSuspendedPhaseButKeepsSecret(t *testing.T) {
func TestReconcileSuspendedOverridesProvisionedState(t *testing.T) {
tenant := testTenant("acme", true)
tenant.Status.ClickHouseDatabaseName = "acme" // already provisioned
r := newFakeReconciler(t, tenant)
got := reconcile(t, r, "acme")
if got.Status.Phase != sentryv1alpha1.PhaseSuspended {
t.Fatalf("Phase = %q, want Suspended even though the tenant is provisioned", got.Status.Phase)
}
}
// TestReconcileUnsuspendingReturnsToActiveNotProvisioning is the
// regression test for why Phase is *derived* fresh every reconcile
// (from Spec.Suspended + whether ClickHouseDatabaseName is set) rather
// than toggled in place: un-suspending an already-provisioned tenant
// must return it straight to Active, not demote it to Provisioning just
// because the last observed Phase happened to be Suspended.
func TestReconcileUnsuspendingReturnsToActiveNotProvisioning(t *testing.T) {
tenant := testTenant("acme", true)
tenant.Status.ClickHouseDatabaseName = "acme"
r := newFakeReconciler(t, tenant)
ctx := context.Background()
if _, err := r.Reconcile(ctx, ctrl.Request{NamespacedName: types.NamespacedName{Name: "acme", Namespace: "default"}}); err != nil {
t.Fatalf("Reconcile: %v", err)
}
_ = reconcile(t, r, "acme") // establishes Suspended
var got sentryv1alpha1.Tenant
if err := r.Get(ctx, types.NamespacedName{Name: "acme", Namespace: "default"}, &got); err != nil {
var toUpdate sentryv1alpha1.Tenant
if err := r.Get(ctx, types.NamespacedName{Name: "acme", Namespace: "default"}, &toUpdate); err != nil {
t.Fatalf("getting tenant: %v", err)
}
if got.Status.Phase != sentryv1alpha1.PhaseSuspended {
t.Fatalf("Phase = %q, want Suspended", got.Status.Phase)
toUpdate.Spec.Suspended = false
if err := r.Update(ctx, &toUpdate); err != nil {
t.Fatalf("unsuspending: %v", err)
}
// A suspended tenant's credential Secret is NOT deleted -- suspension
// is reversible and this controller doesn't manage ClickHouse-side
// grants, so there's nothing at this layer to actually enforce
// suspension; deleting the Secret would just be theater.
var secret corev1.Secret
if err := r.Get(ctx, types.NamespacedName{Name: "sentry-tenant-acme-clickhouse", Namespace: "default"}, &secret); err != nil {
t.Fatalf("expected secret to still exist for a suspended tenant: %v", err)
}
}
func TestReconcileIsIdempotentAndNeverRotatesPassword(t *testing.T) {
tenant := testTenant("acme", false)
r := newFakeReconciler(t, tenant)
ctx := context.Background()
req := ctrl.Request{NamespacedName: types.NamespacedName{Name: "acme", Namespace: "default"}}
if _, err := r.Reconcile(ctx, req); err != nil {
t.Fatalf("first Reconcile: %v", err)
}
var first corev1.Secret
if err := r.Get(ctx, types.NamespacedName{Name: "sentry-tenant-acme-clickhouse", Namespace: "default"}, &first); err != nil {
t.Fatalf("getting secret after first reconcile: %v", err)
}
if _, err := r.Reconcile(ctx, req); err != nil {
t.Fatalf("second Reconcile: %v", err)
}
var second corev1.Secret
if err := r.Get(ctx, types.NamespacedName{Name: "sentry-tenant-acme-clickhouse", Namespace: "default"}, &second); err != nil {
t.Fatalf("getting secret after second reconcile: %v", err)
}
if first.StringData["password"] != second.StringData["password"] {
t.Fatal("password changed across a re-reconcile -- would break every live connection for this tenant")
got := reconcile(t, r, "acme")
if got.Status.Phase != sentryv1alpha1.PhaseActive {
t.Fatalf("Phase = %q, want Active after unsuspending an already-provisioned tenant", got.Status.Phase)
}
}
@@ -153,3 +166,12 @@ func TestReconcileMissingTenantIsNoOp(t *testing.T) {
t.Fatalf("Reconcile on a missing tenant should be a no-op, got error: %v", err)
}
}
func readyCondition(tenant sentryv1alpha1.Tenant) *metav1.Condition {
for i := range tenant.Status.Conditions {
if tenant.Status.Conditions[i].Type == sentryv1alpha1.ConditionReady {
return &tenant.Status.Conditions[i]
}
}
return nil
}