From 823f5d48d1421bf5f1e4aaed5c255b65b257c481 Mon Sep 17 00:00:00 2001 From: John Coffey Date: Fri, 14 Aug 2026 09:07:10 -0700 Subject: [PATCH] Unify the Tenant CRD with enterprise-api -provision-tenant (lightweight) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Closes a gap named across CLAUDE.md/docs/architecture.md/deploy/README.md since early Phase 4: the operator's Tenant CRD and -provision-tenant were two disconnected mechanisms. The operator's reconciler generated a K8s Secret with a locally-generated random password that authenticated against nothing (nothing ever called ClickHouse to create a matching user), and unconditionally claimed status.phase=Active the moment a Tenant object existed -- actively misleading, not just incomplete. Two unification shapes were considered (surfaced to the user via AskUserQuestion, given the real difference in blast radius): the operator's reconcile loop becoming a second real actor (new Postgres + ClickHouse admin credentials flowing into the K8s controller, plus real reconcile-loop idempotency/retry design for an inherently one-shot external side effect), or keeping -provision-tenant as the sole real actor and having it also sync its result into the CRD. Went with the lighter option. enterprise/internal/tenantcrd (new): a Syncer using the K8s dynamic client (unstructured.Unstructured + a GroupVersionResource, not deploy/operator's typed Tenant struct -- avoids a cross-module Go dependency between two independently-versioned modules for one type). Upserts the Tenant object, creates/updates a Secret with the *real* ClickHouse credentials owned by that Tenant via an OwnerReference, then patches status.{clickHouseDatabaseName,clickHouseSecretRef, tantivyIndexPath}. Idempotent and safe to retry: never rotates a credential across a re-sync, never overwrites a pre-existing spec.displayName a human/GitOps process set. cmd/enterprise-api/main.go's runProvisionTenant calls Sync when TENANT_CRD_NAMESPACE is set (empty = no-op, same shape as every other optional dependency in this codebase). Its "already active" refusal is now split: ClickHouse re-provisioning is still refused (rotating a live credential would break every open connection for no benefit), but CR sync alone is now retryable using the credentials already on file in rbacstore -- needed for retrying a previously-failed sync, or backfilling CR sync for a tenant provisioned before this existed. deploy/operator's reconciler rewritten to match: it never claims PhaseActive on its own initiative anymore, only once status.ClickHouseDatabaseName is non-empty (the field -provision-tenant, and only -provision-tenant, sets). Phase is now a pure function of {spec.suspended, status.ClickHouseDatabaseName != ""} recomputed every reconcile, not toggled in place -- fixes a related bug the old code would have hit once suspension was involved: un-suspending an already-provisioned tenant needs to return straight to Active, which isn't derivable from "last observed phase was Suspended" alone. The reconciler no longer creates or manages any Secret, dropped its `secrets` RBAC grant entirely, and gained zero new dependencies. Helm chart: enterprise-api gets its own ServiceAccount/Role/RoleBinding (get/list/create tenants, get/update/patch tenants/status, get/create/ update secrets -- least-privilege, scoped to the release namespace, not a ClusterRole) and a TENANT_CRD_NAMESPACE env var, both gated on tenantOperator.enabled. tenant-operator's ClusterRole loses the secrets grant it no longer needs. Verified in this environment: enterprise/internal/tenantcrd's tests run against k8s.io/client-go's fake dynamic + typed clientsets (real client library, fake transport, no cluster needed); deploy/operator's rewritten tenant_controller_test.go runs against controller-runtime's fake client, including new regression tests for the "must not claim Active without confirmation" and "un-suspending returns to Active, not Provisioning" properties; helm template + parsing the rendered YAML confirms the RBAC split renders exactly as designed under both tenantOperator.enabled=true/false. Not verified: an actual -provision-tenant run against a real cluster with the operator watching (no live cluster in this environment, same disclosed limitation as the rest of /deploy). Docs updated in lockstep: CLAUDE.md, docs/architecture.md, deploy/README.md, deploy/helm/sentry/README.md (including a corrected "Trying the two-tenant example" walkthrough), phase-4-runbook.md (new §11), enterprise/README.md. Also fixed two unrelated stale claims found along the way: docs/architecture.md still said docker-compose.yml ran plain api unconditionally (fixed in an earlier commit, doc not updated then), and enterprise-api's own main.go doc comment still said Helm/ docker-compose wiring wasn't built yet. --- CLAUDE.md | 21 +- deploy/README.md | 70 ++++-- deploy/helm/sentry/README.md | 39 ++- .../helm/sentry/templates/enterprise-api.yaml | 61 +++++ .../sentry/templates/tenant-operator.yaml | 11 +- deploy/helm/sentry/values.yaml | 23 +- deploy/operator/api/v1alpha1/tenant_types.go | 59 +++-- .../internal/controller/tenant_controller.go | 183 ++++---------- .../controller/tenant_controller_test.go | 170 +++++++------ docs/architecture.md | 38 +-- docs/phase-4-runbook.md | 55 ++++- enterprise/README.md | 9 + enterprise/cmd/enterprise-api/main.go | 99 ++++++-- enterprise/go.mod | 37 +++ enterprise/go.sum | 120 ++++++++- enterprise/internal/apiconfig/apiconfig.go | 14 +- enterprise/internal/tenantcrd/tenantcrd.go | 229 ++++++++++++++++++ .../internal/tenantcrd/tenantcrd_test.go | 158 ++++++++++++ 18 files changed, 1073 insertions(+), 323 deletions(-) create mode 100644 enterprise/internal/tenantcrd/tenantcrd.go create mode 100644 enterprise/internal/tenantcrd/tenantcrd_test.go diff --git a/CLAUDE.md b/CLAUDE.md index 836cc83..5e31526 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -213,14 +213,19 @@ stricter still (creator/Admin/Owner only, closing a self-escalation path). Verified against a fake store (`api/dashboards/handler_test.go`); real integration tests exist but haven't run against a live Postgres, same disclosed gap as the rest of this phase's Postgres-backed pieces. -What still keeps this phase from being done: ingest itself has no -tenant concept for either storage engine (every record lands in the one -shared ClickHouse database and Tantivy index no matter what — -undesigned, not just unbuilt), and the two -provisioning mechanisms (`deploy/operator`'s `Tenant` CRD and -`enterprise-api -provision-tenant`) still aren't unified — running both -for the same tenant ID is two separate operator actions today. Full -accounting: +`deploy/operator`'s `Tenant` CRD and `enterprise-api -provision-tenant` +are now unified too, deliberately lightweight rather than making the +K8s controller a second real actor: `-provision-tenant` stays the sole +caller of ClickHouse/`rbacstore`, and (via a new +`enterprise/internal/tenantcrd`, gated on `TENANT_CRD_NAMESPACE`) syncs +its real result into the CRD — a real credential Secret, not the +previous placeholder that authenticated against nothing, and status +fields the reconciler derives `Phase`/`Ready` from instead of +independently guessing "Active" the moment a Tenant object exists. What +still keeps this phase from being done: ingest itself has no tenant +concept for either storage engine (every record lands in the one shared +ClickHouse database and Tantivy index no matter what — undesigned, not +just unbuilt). Full accounting: `/docs/security/threat-model.md`; step-by-step verification procedure (not yet run against a live cluster in this environment): `/docs/phase-4-runbook.md`. The rest of this section describes the exit diff --git a/deploy/README.md b/deploy/README.md index af25859..0081588 100644 --- a/deploy/README.md +++ b/deploy/README.md @@ -21,26 +21,43 @@ map of per-tenant ClickHouse connection pools via `internal/chrunner`; that's an explicit Phase 4 non-goal (see `/CLAUDE.md`). What it *does* add: -- A `Tenant` CRD + controller that generates and manages one dedicated - ClickHouse credential Secret per tenant (`operator/internal/controller`). +- A `Tenant` CRD + controller (`operator/internal/controller`) that + reflects real provisioning state onto `status.phase`/a `Ready` + condition, derived from whether `enterprise-api -provision-tenant` has + reported real ClickHouse provisioning. - A Helm chart that can install zero-or-more `Tenant` CRs - (`values.tenants`) alongside the rest of the stack, and — the newer - piece — swaps `api`'s Deployment for `enterprise-api`'s whenever - `enterprise.enabled` is true, so which query binary actually serves - traffic is no longer a separately-forgettable decision (see - `helm/sentry/README.md`'s "`api` vs `enterprise-api`" section). + (`values.tenants`) alongside the rest of the stack, and swaps `api`'s + Deployment for `enterprise-api`'s whenever `enterprise.enabled` is + true, so which query binary actually serves traffic is no longer a + separately-forgettable decision (see `helm/sentry/README.md`'s "`api` + vs `enterprise-api`" section). -**Two still-separate mechanisms, not yet unified**: the Operator's -`Tenant` CRD manages only the K8s-side credential Secret — it does not -call ClickHouse (no `CREATE DATABASE`/`CREATE USER`/`GRANT`) or touch -the Tantivy filesystem. `enterprise-api -provision-tenant=` is what -actually does that (`enterprise/internal/tenantprovision`, built and -tested — see `/enterprise/README.md`), driven independently via -`rbacstore`, not from the `Tenant` CRD's reconcile loop. A `Tenant` -reaching `status.phase: Active` here means "this tenant has a K8s -Secret," not "this tenant's ClickHouse database/grants exist" — running -both mechanisms for the same tenant ID today requires two separate -operator actions, named explicitly rather than implied to be one. +**Now unified, in a deliberately lightweight way**: `enterprise-api +-provision-tenant=` stays the sole real actor — it's the only thing +that calls ClickHouse (`CREATE DATABASE`/`CREATE USER`/`GRANT`, via +`enterprise/internal/tenantprovision`) and writes `rbacstore`. What +changed: once it succeeds, it also syncs the result into the `Tenant` +CRD (`enterprise/internal/tenantcrd`) — creating the Secret with *real* +credentials (the controller no longer generates a placeholder one that +authenticated against nothing) and setting the status fields the +controller reads to compute `Phase`/`Ready`. The controller itself +gained no new credentials and still never touches ClickHouse/Postgres -- +it's a pure function of `spec.suspended` and whatever +`-provision-tenant` has reported, never an independent second guess at +"is this tenant really provisioned." A `Tenant` reaching +`status.phase: Active` now means the same thing `rbacstore.tenants. +status='active'` does, not two different claims — see +`enterprise/internal/tenantcrd`'s and `operator/internal/controller/ +tenant_controller.go`'s doc comments for the full split, and +`enterprise-api -provision-tenant`'s `TENANT_CRD_NAMESPACE` env var +(set automatically by the Helm chart when `tenantOperator.enabled`) to +turn this on. Deliberately not built: the operator's reconcile loop +itself calling ClickHouse/rbacstore directly (a "full unification" +option considered and set aside — it would give the operator two new +credential sets and require real reconcile-loop idempotency design for +an inherently one-shot external side effect, a bigger and riskier +change than this repo's provisioning story needed to close the actual +gap, which was two *disconnected* sources of truth, not two actors). ## Verification status -- read before trusting this against a real cluster @@ -59,6 +76,15 @@ access was available to fetch these tools, but no cluster): (`internal/controller/tenant_controller_test.go`) -- real reconcile logic exercised, but not against a real apiserver (no `envtest` binaries available; see that test file's doc comment). +- `enterprise/internal/tenantcrd` (the "lightweight unification" + half `-provision-tenant` runs): `go test` passes against + `k8s.io/client-go`'s fake dynamic and typed clientsets -- real client + library, fake transport, same shape as `enterprise/internal/ + searchclient`'s in-process gRPC tests. What this doesn't prove: that + `sentry.io/v1alpha1.Tenant`'s real CRD schema (a real apiserver's + OpenAPI validation) accepts exactly what this package writes -- the + `helm template`/kubeconform check below covers the schema shape, not + a live write against it. - `deploy/operator/config/crd/sentry.io_tenants.yaml`: parsed with `sigs.k8s.io/yaml` + strict-unmarshaled into the real `k8s.io/apiextensions-apiserver` `CustomResourceDefinition` Go type -- @@ -76,7 +102,13 @@ access was available to fetch these tools, but no cluster): parsing the rendered YAML (not just eyeballing it): exactly one `Deployment`/`Service` named `sentry-api` renders in each mode, with the `enterprise.enabled: true` render using the `enterprise-api` image - and the default render using plain `api`'s. + and the default render using plain `api`'s. Also confirmed for the + `tenantOperator.enabled: true` case: `enterprise-api` gets its own + ServiceAccount/Role/RoleBinding (exactly `tenants`/`tenants/status`/ + `secrets`, no more), `tenant-operator`'s own ClusterRole no longer + grants `secrets` at all, and `TENANT_CRD_NAMESPACE` is set on + `enterprise-api`'s container only when `tenantOperator.enabled` is + true. - Docker image builds (`operator/Dockerfile` and every other `Dockerfile` this chart references) were **not** verified in this session -- Docker's daemon wasn't reachable here either (see the diff --git a/deploy/helm/sentry/README.md b/deploy/helm/sentry/README.md index ef9b9ee..920daa6 100644 --- a/deploy/helm/sentry/README.md +++ b/deploy/helm/sentry/README.md @@ -72,20 +72,37 @@ helm install sentry . --include-crds \ --set 'tenants[1].name=globex' --set 'tenants[1].displayName=Globex Corporation' kubectl get tenants +# expect: both Provisioning -- the Tenant CRs above are just a +# declarative request; nothing has actually provisioned ClickHouse for +# either yet (see below). + +kubectl exec -it deploy/sentry-api -- /enterprise-api -provision-tenant=acme -display-name="Acme Corp" +kubectl exec -it deploy/sentry-api -- /enterprise-api -provision-tenant=globex -display-name="Globex Corporation" + +kubectl get tenants +# expect: both Active now. kubectl get secret sentry-tenant-acme-clickhouse sentry-tenant-globex-clickhouse ``` -This proves the K8s-side half of Phase 4's "two tenants... with their -own users, roles, dashboards" exit criteria (`/CLAUDE.md`) -- a real -per-tenant credential Secret exists for each, generated by -`deploy/operator`'s `Tenant` controller. It does **not** by itself give -either tenant a working ClickHouse database or Tantivy index -- that -needs `enterprise-api -provision-tenant=` (a separate, deliberately -manual operator action; the `Tenant` CRD and `-provision-tenant` are two -independent mechanisms today, not yet unified -- see -`/enterprise/README.md`), and OIDC login (built, but still needs a -manual `tenant_memberships` row -- see `/docs/phase-4-runbook.md` §3a) -before a human can actually query as that tenant. +This proves Phase 4's "two tenants... with their own users, roles, +dashboards" exit criteria (`/CLAUDE.md`) end to end at the deployment- +topology layer: `-provision-tenant` (`enterprise/internal/ +tenantprovision`) is what actually creates each tenant's ClickHouse +database/user/grant and marks it active in `rbacstore`; running inside +the `enterprise-api` Deployment's Pod means it automatically syncs that +real result into the `Tenant` CRD too (`enterprise/internal/tenantcrd`, +via the ServiceAccount/Role `tenantOperator.enabled` also grants that +Deployment) -- the credential Secret you see above has *real* +credentials, not a placeholder, and `Tenant.status.phase: Active` means +the same thing `rbacstore.tenants.status='active'` does, not two +different claims about two different systems. The `Tenant` CRD and +`-provision-tenant` used to be genuinely disconnected (a Secret existed +the moment the CR was created, with a password that authenticated +against nothing) -- see `/deploy/README.md`'s "lightweight unification" +section for the full history. OIDC/SAML login still needs a manual +`tenant_memberships` grant (`enterprise-auth -grant-membership-*` -- +see `/docs/phase-4-runbook.md` §3a/§3b) before a human can actually +query as either tenant. ## `web`'s image needs rebuilding per environment diff --git a/deploy/helm/sentry/templates/enterprise-api.yaml b/deploy/helm/sentry/templates/enterprise-api.yaml index e1a448e..9e23107 100644 --- a/deploy/helm/sentry/templates/enterprise-api.yaml +++ b/deploy/helm/sentry/templates/enterprise-api.yaml @@ -12,6 +12,54 @@ routes to whichever Deployment is actually rendered, with zero conditional logic needed in any consumer (alerting, web). */}} {{- if .Values.enterprise.enabled }} +{{- if .Values.tenantOperator.enabled }} +# Grants enterprise-api's -provision-tenant (enterprise/internal/ +# tenantcrd) permission to sync real provisioning results into the +# Tenant CRD -- a Role, not a ClusterRole (unlike tenant-operator's: +# this binary only ever provisions tenants that live in its own release +# namespace, no reason to widen it), scoped to exactly the two resource +# types tenantcrd.Syncer touches. Only rendered when tenantOperator is +# also enabled -- no Tenant CRD installed, nothing to sync into. +apiVersion: v1 +kind: ServiceAccount +metadata: + name: {{ .Release.Name }}-enterprise-api + labels: + {{- include "sentry.labels" . | nindent 4 }} +--- +apiVersion: rbac.authorization.k8s.io/v1 +kind: Role +metadata: + name: {{ .Release.Name }}-enterprise-api + labels: + {{- include "sentry.labels" . | nindent 4 }} +rules: + - apiGroups: ["sentry.io"] + resources: ["tenants"] + verbs: ["get", "list", "create"] + - apiGroups: ["sentry.io"] + resources: ["tenants/status"] + verbs: ["get", "update", "patch"] + - apiGroups: [""] + resources: ["secrets"] + verbs: ["get", "create", "update"] +--- +apiVersion: rbac.authorization.k8s.io/v1 +kind: RoleBinding +metadata: + name: {{ .Release.Name }}-enterprise-api + labels: + {{- include "sentry.labels" . | nindent 4 }} +roleRef: + apiGroup: rbac.authorization.k8s.io + kind: Role + name: {{ .Release.Name }}-enterprise-api +subjects: + - kind: ServiceAccount + name: {{ .Release.Name }}-enterprise-api + namespace: {{ .Release.Namespace }} +--- +{{- end }} apiVersion: apps/v1 kind: Deployment metadata: @@ -30,6 +78,9 @@ spec: labels: {{- include "sentry.selectorLabels" (list $ "api") | nindent 8 }} spec: + {{- if .Values.tenantOperator.enabled }} + serviceAccountName: {{ .Release.Name }}-enterprise-api + {{- end }} initContainers: {{- include "sentry.waitForTCP" (list "clickhouse" (printf "%s-clickhouse" .Release.Name) "9000") | nindent 8 }} {{- include "sentry.waitForTCP" (list "postgres" (printf "%s-postgres" .Release.Name) "5432") | nindent 8 }} @@ -84,6 +135,16 @@ spec: key: auditWriterPassword - name: ENTERPRISE_AUTH_URL value: "http://{{ .Release.Name }}-enterprise-auth:8082" + {{- if .Values.tenantOperator.enabled }} + # Enables enterprise/internal/tenantcrd -- -provision-tenant + # (run via `kubectl exec` into this Deployment's Pod, using + # its ServiceAccount/Role above) syncs real provisioning + # results into the Tenant CRD this namespace's tenants live + # in. Unset (the default, when tenantOperator isn't enabled) + # is a documented no-op -- see apiconfig.Config.TenantCRDNamespace. + - name: TENANT_CRD_NAMESPACE + value: {{ .Release.Namespace | quote }} + {{- end }} ports: - name: http containerPort: 8080 diff --git a/deploy/helm/sentry/templates/tenant-operator.yaml b/deploy/helm/sentry/templates/tenant-operator.yaml index eeb895b..31c077d 100644 --- a/deploy/helm/sentry/templates/tenant-operator.yaml +++ b/deploy/helm/sentry/templates/tenant-operator.yaml @@ -11,8 +11,12 @@ metadata: # doesn't assume it's the only namespace the operator might one day watch # -- narrowed to exactly the two resource types # deploy/operator/internal/controller/tenant_controller.go's -# +kubebuilder:rbac markers name (tenants, tenants/status, secrets), not -# a wildcard grant. +# +kubebuilder:rbac markers name (tenants, tenants/status), not a +# wildcard grant. No `secrets` permission -- this controller stopped +# managing the ClickHouse credential Secret once enterprise-api +# -provision-tenant took over creating it with real credentials (see +# that controller's doc comment); see enterprise-api.yaml's own +# ServiceAccount/Role for the `secrets` grant that binary needs instead. apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: @@ -26,9 +30,6 @@ rules: - apiGroups: ["sentry.io"] resources: ["tenants/status"] verbs: ["get", "update", "patch"] - - apiGroups: [""] - resources: ["secrets"] - verbs: ["get", "list", "watch", "create", "update", "patch", "delete"] --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding diff --git a/deploy/helm/sentry/values.yaml b/deploy/helm/sentry/values.yaml index 8bae114..19212d2 100644 --- a/deploy/helm/sentry/values.yaml +++ b/deploy/helm/sentry/values.yaml @@ -165,9 +165,11 @@ enterprise: idpMetadataURL: "" # Installs deploy/operator (the Tenant CRD controller) alongside this -# chart. Only meaningful when enterprise.enabled is also true -- -# gated on that, not a separate flag, since a Tenant CR with no -# enterprise-auth deployed to consume its Secret has nothing to do. +# chart. Only meaningful when enterprise.enabled is also true -- gated +# on that, not a separate flag, since a Tenant CR with nothing to +# reflect (see below) has nothing to do. Also turns on enterprise-api's +# own Tenant-CRD-syncing permissions (a ServiceAccount/Role, and the +# TENANT_CRD_NAMESPACE env var) -- see templates/enterprise-api.yaml. tenantOperator: enabled: false image: @@ -176,11 +178,16 @@ tenantOperator: resources: {} # One entry per tenant to provision -- rendered as Tenant CRs -# (templates/tenants.yaml), reconciled by the tenant-operator into a -# per-tenant ClickHouse credential Secret. See -# deploy/operator/internal/controller/tenant_controller.go's doc comment -# for exactly what that does and doesn't set up. Empty by default; a -# real two-tenant deployment (Phase 4's exit criteria) sets e.g.: +# (templates/tenants.yaml), a declarative request an admin/GitOps +# process makes. The operator (tenant-operator, above) only ever +# *reflects* real state onto these objects (Phase/Conditions, derived +# from what enterprise-api's `-provision-tenant` has reported) -- it's +# `-provision-tenant` (run via `kubectl exec` into the enterprise-api +# Pod), not the operator, that actually calls ClickHouse and writes the +# credential Secret. See deploy/operator/internal/controller/ +# tenant_controller.go's and enterprise/internal/tenantcrd's doc +# comments for the full split. Empty by default; a real two-tenant +# deployment (Phase 4's exit criteria) sets e.g.: # tenants: # - name: acme # displayName: "Acme Corp" diff --git a/deploy/operator/api/v1alpha1/tenant_types.go b/deploy/operator/api/v1alpha1/tenant_types.go index 9879f50..b005f14 100644 --- a/deploy/operator/api/v1alpha1/tenant_types.go +++ b/deploy/operator/api/v1alpha1/tenant_types.go @@ -8,15 +8,20 @@ import ( // /docs/phase-4-isolation-design.md: every tenant-resolution path // elsewhere must refuse to serve a tenant not in PhaseActive, checked // server-side (today, against enterprise/internal/rbacstore's tenants -// table -- this CR is a K8s-native *view* of the same state machine at -// the deployment-topology layer, not a second source of truth. Reconciling -// the two together is exactly the kind of tenant-provisioning wiring -// named as deferred in /docs/phase-4-runbook.md's task 6 section: today -// this operator only manages the K8s-side artifact (a per-tenant -// ClickHouse credential Secret + a ConfigMap recording the tenant's -// database name/index path), not the actual `CREATE DATABASE`/`CREATE -// USER`/`GRANT` calls against ClickHouse -- that's -// enterprise/internal/tenantprovision, still unbuilt. +// table -- this CR is a K8s-native *view* of that same state machine, +// kept honest rather than a second, independently-guessed source of +// truth. `enterprise-api -provision-tenant` (the actual actor -- real +// `CREATE DATABASE`/`CREATE USER`/`GRANT` calls against ClickHouse, and +// the real rbacstore writes) is the only writer of +// TenantStatus.ClickHouseDatabaseName/ClickHouseSecretRef/ +// TantivyIndexPath, once real provisioning succeeds -- see +// internal/controller/tenant_controller.go's doc comment for how +// TenantReconciler derives Phase/Conditions from those fields rather +// than fabricating its own "provisioned" claim. This is the "lightweight +// unification" named in CLAUDE.md/docs/phase-4-runbook.md's "two +// independent provisioning mechanisms" gap: -provision-tenant stays the +// real actor; this operator's reconcile loop never touches Postgres or +// ClickHouse and gained no new credentials. type TenantPhase string const ( @@ -47,23 +52,43 @@ type TenantSpec struct { Suspended bool `json:"suspended,omitempty"` } -// TenantStatus is observed state -- only the controller writes this. +// TenantStatus is observed state, with split ownership as of the +// "lightweight unification" (see TenantPhase's doc comment): +// ClickHouseDatabaseName/ClickHouseSecretRef/TantivyIndexPath are +// written only by enterprise-api's -provision-tenant, once real +// ClickHouse provisioning actually succeeds -- this controller only +// reads them (to compute Phase/Conditions) and never invents a value +// for them. Phase/Conditions/ObservedGeneration remain +// controller-written, computed fresh on every reconcile from Spec plus +// whatever -provision-tenant has (or hasn't) reported. type TenantStatus struct { // +optional Phase TenantPhase `json:"phase,omitempty"` - // ClickHouseDatabaseName is derived (today: same as the Tenant's own - // Name) rather than settable in Spec -- see task 2's design: no - // tenant traffic authenticates as ClickHouse's `default` user, and a + // ClickHouseDatabaseName is set by enterprise-api -provision-tenant + // once ClickHouse provisioning for this tenant actually succeeds -- + // empty means "not yet provisioned," which TenantReconciler reads as + // PhaseProvisioning (see tenant_controller.go). Today it's always + // equal to the Tenant's own Name (see task 2's design: no tenant + // traffic authenticates as ClickHouse's `default` user, and a // database name that could diverge from the tenant identifier is a - // bookkeeping foot-gun this type avoids by construction. + // bookkeeping foot-gun this type avoids by construction) but isn't + // itself computed by this package -- -provision-tenant sets it + // directly from what it actually created. // +optional ClickHouseDatabaseName string `json:"clickHouseDatabaseName,omitempty"` // ClickHouseSecretRef names the Secret (same namespace) holding this - // tenant's dedicated, narrowly-granted ClickHouse credentials -- see - // tenant_controller.go's reconcileSecret. Never the cluster-wide - // CLICKHOUSE_PASSWORD docker-compose.yml uses today. + // tenant's dedicated, narrowly-granted ClickHouse credentials -- + // created by enterprise-api -provision-tenant (enterprise/internal/ + // tenantcrd), owned by this Tenant object via an OwnerReference so + // K8s garbage-collects it on Tenant deletion regardless of which + // process created it. This controller no longer creates or manages + // any Secret itself -- see tenant_controller.go's doc comment for + // why a controller-generated placeholder credential (the pre- + // unification behavior) was actively misleading, not just + // incomplete. Never the cluster-wide CLICKHOUSE_PASSWORD + // docker-compose.yml uses today. // +optional ClickHouseSecretRef string `json:"clickHouseSecretRef,omitempty"` diff --git a/deploy/operator/internal/controller/tenant_controller.go b/deploy/operator/internal/controller/tenant_controller.go index 0320376..4661500 100644 --- a/deploy/operator/internal/controller/tenant_controller.go +++ b/deploy/operator/internal/controller/tenant_controller.go @@ -1,42 +1,39 @@ -// Package controller reconciles the Tenant CRD (api/v1alpha1) into the -// K8s-native artifacts task 2/CLAUDE.md's Phase 4 exit criteria calls -// for: "real per-tenant secret management (replacing today's single -// shared CLICKHOUSE_PASSWORD)" -- see docker-compose.yml's -// CLICKHOUSE_PASSWORD comment for what that shared-secret shape looks -// like today. +// Package controller reconciles the Tenant CRD (api/v1alpha1) -- see +// TenantPhase's doc comment for the "lightweight unification" this +// controller is one half of. This reconciler never calls ClickHouse (no +// CREATE DATABASE/CREATE USER/GRANT), never touches the Tantivy index +// filesystem, and never talks to enterprise/internal/rbacstore -- those +// stay enterprise-api -provision-tenant's job (enterprise/internal/ +// tenantprovision, enterprise/internal/tenantcrd). This controller's +// job is purely a function of what's already on the object: derive +// Phase and the Ready condition from Spec.Suspended and whether +// Status.ClickHouseDatabaseName has been set by -provision-tenant, +// nothing more. A Tenant reaching PhaseActive here is exactly the same +// claim as rbacstore's tenants.status='active' now, not a +// second, independently-computed one -- see docs/phase-4-isolation-design.md. // -// What this reconciler does NOT do, named explicitly rather than -// implied: it never calls ClickHouse (no CREATE DATABASE/CREATE USER/ -// GRANT), never touches the Tantivy index filesystem, and never talks to -// enterprise/internal/rbacstore. Those are enterprise/internal/ -// tenantprovision's job -- unbuilt, per the task 5 summary. This -// controller's job stops at "does a K8s Secret with this tenant's -// ClickHouse credentials exist, and does the Tenant's status reflect -// that" -- the deployment-topology half of tenant provisioning, not the -// database-side half. A Tenant reaching PhaseActive here is NOT the same -// claim as rbacstore's tenants.status='active' (the actual gate every -// tenant-resolution code path checks per -// /docs/phase-4-isolation-design.md) -- reconciling those two into one -// state machine is exactly the kind of follow-up work -// /docs/phase-4-runbook.md's task 6 section names as deferred. +// Earlier versions of this controller also generated a per-tenant +// ClickHouse credential Secret with a locally-generated random +// password. That Secret authenticated against nothing (nothing in this +// controller ever called ClickHouse to create a matching user) and +// nothing else in the codebase ever read it -- a placeholder that +// actively misled ("looks provisioned") rather than one that honestly +// represented "not yet provisioned." Removed rather than fixed in +// place: the real Secret, with real credentials, is now created by +// -provision-tenant (enterprise/internal/tenantcrd) once ClickHouse +// provisioning actually succeeds. package controller import ( "context" - "crypto/rand" - "encoding/base64" "fmt" - corev1 "k8s.io/api/core/v1" apierrors "k8s.io/apimachinery/pkg/api/errors" "k8s.io/apimachinery/pkg/api/meta" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/runtime" - "k8s.io/apimachinery/pkg/types" ctrl "sigs.k8s.io/controller-runtime" "sigs.k8s.io/controller-runtime/pkg/client" - "sigs.k8s.io/controller-runtime/pkg/controller/controllerutil" - "sigs.k8s.io/controller-runtime/pkg/log" sentryv1alpha1 "github.com/sentry/sentry/deploy/operator/api/v1alpha1" ) @@ -47,77 +44,52 @@ type TenantReconciler struct { Scheme *runtime.Scheme } -// clickHouseSecretName is deterministic from the tenant name -- never -// randomly suffixed -- so a re-run of Reconcile (or a controller -// restart) finds the same Secret it created before, rather than losing -// track of it and creating a second one. -func clickHouseSecretName(tenant *sentryv1alpha1.Tenant) string { - return fmt.Sprintf("sentry-tenant-%s-clickhouse", tenant.Name) -} - -// tantivyIndexPath mirrors /docs/phase-4-isolation-design.md's Tantivy -// section: one directory per tenant under the shared search-index -// volume (search-index-data in docker-compose.yml; a PVC in the Helm -// chart -- see deploy/helm/sentry/templates/search-deployment.yaml). -func tantivyIndexPath(tenant *sentryv1alpha1.Tenant) string { - return "/var/lib/sentry-search/tenants/" + tenant.Name -} - -// generatePassword returns a 32-byte random value, base64-encoded -- -// same "narrowly-granted, per-tenant, never the shared default user" -// framing as /docs/phase-4-isolation-design.md's ClickHouse section, -// applied to how the credential itself is generated (crypto/rand, not -// math/rand -- this becomes a real ClickHouse user's password once -// internal/tenantprovision consumes it). -func generatePassword() (string, error) { - buf := make([]byte, 32) - if _, err := rand.Read(buf); err != nil { - return "", fmt.Errorf("generating password: %w", err) - } - return base64.RawURLEncoding.EncodeToString(buf), nil -} - // +kubebuilder:rbac:groups=sentry.io,resources=tenants,verbs=get;list;watch;create;update;patch;delete // +kubebuilder:rbac:groups=sentry.io,resources=tenants/status,verbs=get;update;patch -// +kubebuilder:rbac:groups="",resources=secrets,verbs=get;list;watch;create;update;patch;delete func (r *TenantReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) { - logger := log.FromContext(ctx) - var tenant sentryv1alpha1.Tenant if err := r.Get(ctx, req.NamespacedName, &tenant); err != nil { if apierrors.IsNotFound(err) { - // Deleted -- owned Secret is garbage-collected by K8s via - // its OwnerReference (set in reconcileSecret below), nothing - // else to clean up at this layer. See this file's package - // doc comment: real deprovisioning (revoking ClickHouse - // grants) isn't this controller's job. + // Deleted -- the owned Secret -provision-tenant created (if + // any) is garbage-collected by K8s via its OwnerReference, + // nothing else to clean up at this layer. Real + // deprovisioning (revoking ClickHouse grants) isn't this + // controller's job, or -provision-tenant's today -- see + // /docs/security/threat-model.md's non-goals. return ctrl.Result{}, nil } return ctrl.Result{}, fmt.Errorf("getting tenant: %w", err) } - secretName, err := r.reconcileSecret(ctx, &tenant) - if err != nil { - logger.Error(err, "reconciling clickhouse secret") - return ctrl.Result{}, err + // provisioned is true once -provision-tenant has confirmed real + // ClickHouse provisioning by setting this field -- see + // TenantStatus.ClickHouseDatabaseName's doc comment. This + // controller treats it as the sole source of truth for "has + // provisioning actually happened," never claiming PhaseActive on + // its own say-so the way the pre-unification version did. + provisioned := tenant.Status.ClickHouseDatabaseName != "" + + condStatus := metav1.ConditionFalse + reason, message := "AwaitingProvisioning", "waiting for enterprise-api -provision-tenant to provision ClickHouse for this tenant" + switch { + case tenant.Spec.Suspended: + tenant.Status.Phase = sentryv1alpha1.PhaseSuspended + reason, message = "Suspended", "tenant is suspended (spec.suspended=true)" + case provisioned: + tenant.Status.Phase = sentryv1alpha1.PhaseActive + condStatus = metav1.ConditionTrue + reason, message = "Provisioned", fmt.Sprintf("ClickHouse database %q is provisioned", tenant.Status.ClickHouseDatabaseName) + default: + tenant.Status.Phase = sentryv1alpha1.PhaseProvisioning } - desiredPhase := sentryv1alpha1.PhaseActive - if tenant.Spec.Suspended { - desiredPhase = sentryv1alpha1.PhaseSuspended - } - - tenant.Status.ClickHouseDatabaseName = tenant.Name - tenant.Status.ClickHouseSecretRef = secretName - tenant.Status.TantivyIndexPath = tantivyIndexPath(&tenant) - tenant.Status.Phase = desiredPhase tenant.Status.ObservedGeneration = tenant.Generation meta.SetStatusCondition(&tenant.Status.Conditions, metav1.Condition{ Type: sentryv1alpha1.ConditionReady, - Status: metav1.ConditionTrue, - Reason: "SecretReconciled", - Message: fmt.Sprintf("ClickHouse credential secret %q is present", secretName), + Status: condStatus, + Reason: reason, + Message: message, ObservedGeneration: tenant.Generation, }) @@ -128,59 +100,8 @@ func (r *TenantReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctr return ctrl.Result{}, nil } -// reconcileSecret creates the tenant's ClickHouse credential Secret if -// it doesn't already exist. Deliberately never updates an existing -// Secret's password -- rotating a live tenant's ClickHouse credential -// out from under it (without first updating the ClickHouse-side grant, -// which this controller doesn't do) would just break every open -// connection for no benefit; credential rotation is real future work -// that needs to be coordinated with internal/tenantprovision, not -// something this reconcile loop can safely do alone. -func (r *TenantReconciler) reconcileSecret(ctx context.Context, tenant *sentryv1alpha1.Tenant) (string, error) { - name := clickHouseSecretName(tenant) - - var existing corev1.Secret - err := r.Get(ctx, types.NamespacedName{Namespace: tenant.Namespace, Name: name}, &existing) - if err == nil { - return name, nil - } - if !apierrors.IsNotFound(err) { - return "", fmt.Errorf("getting secret: %w", err) - } - - password, err := generatePassword() - if err != nil { - return "", err - } - - secret := &corev1.Secret{ - ObjectMeta: metav1.ObjectMeta{ - Name: name, - Namespace: tenant.Namespace, - Labels: map[string]string{ - "app.kubernetes.io/managed-by": "sentry-tenant-operator", - "sentry.io/tenant": tenant.Name, - }, - }, - Type: corev1.SecretTypeOpaque, - StringData: map[string]string{ - "username": "tenant_" + tenant.Name, - "password": password, - "database": tenant.Name, - }, - } - if err := controllerutil.SetControllerReference(tenant, secret, r.Scheme); err != nil { - return "", fmt.Errorf("setting owner reference: %w", err) - } - if err := r.Create(ctx, secret); err != nil { - return "", fmt.Errorf("creating secret: %w", err) - } - return name, nil -} - func (r *TenantReconciler) SetupWithManager(mgr ctrl.Manager) error { return ctrl.NewControllerManagedBy(mgr). For(&sentryv1alpha1.Tenant{}). - Owns(&corev1.Secret{}). Complete(r) } diff --git a/deploy/operator/internal/controller/tenant_controller_test.go b/deploy/operator/internal/controller/tenant_controller_test.go index 57ea6f5..688ab4d 100644 --- a/deploy/operator/internal/controller/tenant_controller_test.go +++ b/deploy/operator/internal/controller/tenant_controller_test.go @@ -3,10 +3,9 @@ // real kube-apiserver/etcd binary pair (setup-envtest) that isn't // available in this environment (see package doc comment and // deploy/README.md's verification section). A fake client exercises -// Reconcile's actual logic (object CRUD, owner references, status -// writes) against an in-memory tracker; what it can't exercise is -// anything a real apiserver would do for you (defaulting, admission, -// actual garbage collection of owned objects, watch-triggered +// Reconcile's actual logic (status derivation, condition writes) against +// an in-memory tracker; what it can't exercise is anything a real +// apiserver would do for you (defaulting, admission, watch-triggered // re-reconciliation) -- so passing here is real signal about this // reconciler's logic, not proof it behaves correctly against a live // cluster. @@ -51,98 +50,112 @@ func testTenant(name string, suspended bool) *sentryv1alpha1.Tenant { } } -func TestReconcileCreatesSecretAndSetsActivePhase(t *testing.T) { +func reconcile(t *testing.T, r *TenantReconciler, name string) sentryv1alpha1.Tenant { + t.Helper() + ctx := context.Background() + if _, err := r.Reconcile(ctx, ctrl.Request{NamespacedName: types.NamespacedName{Name: name, Namespace: "default"}}); err != nil { + t.Fatalf("Reconcile: %v", err) + } + var got sentryv1alpha1.Tenant + if err := r.Get(ctx, types.NamespacedName{Name: name, Namespace: "default"}, &got); err != nil { + t.Fatalf("getting tenant: %v", err) + } + return got +} + +// TestReconcileUnprovisionedTenantIsProvisioningNotActive is the +// regression test for the pre-unification bug: this controller must +// never claim PhaseActive just because a Tenant object exists -- only +// enterprise-api -provision-tenant setting Status.ClickHouseDatabaseName +// (real ClickHouse provisioning having actually succeeded) earns that. +func TestReconcileUnprovisionedTenantIsProvisioningNotActive(t *testing.T) { + tenant := testTenant("acme", false) + r := newFakeReconciler(t, tenant) + + got := reconcile(t, r, "acme") + + if got.Status.Phase != sentryv1alpha1.PhaseProvisioning { + t.Fatalf("Phase = %q, want Provisioning (nothing has provisioned this tenant yet)", got.Status.Phase) + } + cond := readyCondition(got) + if cond == nil || cond.Status != metav1.ConditionFalse { + t.Fatalf("Ready condition = %+v, want status False", cond) + } +} + +// TestReconcileReflectsProvisioningStateProvisionTenantSets is the core +// "lightweight unification" behavior: once something external (in +// production, -provision-tenant; here, simulated directly against the +// fake client the way a real K8s API server write would land) sets +// ClickHouseDatabaseName, this controller must report PhaseActive. +func TestReconcileReflectsProvisioningStateProvisionTenantSets(t *testing.T) { tenant := testTenant("acme", false) r := newFakeReconciler(t, tenant) ctx := context.Background() - if _, err := r.Reconcile(ctx, ctrl.Request{NamespacedName: types.NamespacedName{Name: "acme", Namespace: "default"}}); err != nil { - t.Fatalf("Reconcile: %v", err) - } - - var secret corev1.Secret - if err := r.Get(ctx, types.NamespacedName{Name: "sentry-tenant-acme-clickhouse", Namespace: "default"}, &secret); err != nil { - t.Fatalf("expected a ClickHouse secret to be created: %v", err) - } - if secret.StringData["username"] != "tenant_acme" || secret.StringData["database"] != "acme" { - t.Fatalf("unexpected secret data: %+v", secret.StringData) - } - if secret.StringData["password"] == "" { - t.Fatal("expected a non-empty generated password") - } - if len(secret.OwnerReferences) != 1 || secret.OwnerReferences[0].Name != "acme" { - t.Fatalf("expected secret to be owned by the Tenant, got %+v", secret.OwnerReferences) - } - - var got sentryv1alpha1.Tenant - if err := r.Get(ctx, types.NamespacedName{Name: "acme", Namespace: "default"}, &got); err != nil { + var toUpdate sentryv1alpha1.Tenant + if err := r.Get(ctx, types.NamespacedName{Name: "acme", Namespace: "default"}, &toUpdate); err != nil { t.Fatalf("getting tenant: %v", err) } + toUpdate.Status.ClickHouseDatabaseName = "acme" + toUpdate.Status.ClickHouseSecretRef = "sentry-tenant-acme-clickhouse" + toUpdate.Status.TantivyIndexPath = "/var/lib/sentry-search/tenants/acme" + if err := r.Status().Update(ctx, &toUpdate); err != nil { + t.Fatalf("simulating -provision-tenant's status write: %v", err) + } + + got := reconcile(t, r, "acme") + if got.Status.Phase != sentryv1alpha1.PhaseActive { t.Fatalf("Phase = %q, want Active", got.Status.Phase) } - if got.Status.ClickHouseDatabaseName != "acme" { - t.Fatalf("ClickHouseDatabaseName = %q, want acme", got.Status.ClickHouseDatabaseName) + if got.Status.ClickHouseDatabaseName != "acme" || got.Status.ClickHouseSecretRef != "sentry-tenant-acme-clickhouse" || got.Status.TantivyIndexPath != "/var/lib/sentry-search/tenants/acme" { + t.Fatalf("reconcile must not clobber the fields -provision-tenant set: %+v", got.Status) } - if got.Status.ClickHouseSecretRef != "sentry-tenant-acme-clickhouse" { - t.Fatalf("ClickHouseSecretRef = %q", got.Status.ClickHouseSecretRef) - } - if got.Status.TantivyIndexPath != "/var/lib/sentry-search/tenants/acme" { - t.Fatalf("TantivyIndexPath = %q", got.Status.TantivyIndexPath) + cond := readyCondition(got) + if cond == nil || cond.Status != metav1.ConditionTrue { + t.Fatalf("Ready condition = %+v, want status True", cond) } } -func TestReconcileSuspendedSetsSuspendedPhaseButKeepsSecret(t *testing.T) { +func TestReconcileSuspendedOverridesProvisionedState(t *testing.T) { tenant := testTenant("acme", true) + tenant.Status.ClickHouseDatabaseName = "acme" // already provisioned + r := newFakeReconciler(t, tenant) + + got := reconcile(t, r, "acme") + + if got.Status.Phase != sentryv1alpha1.PhaseSuspended { + t.Fatalf("Phase = %q, want Suspended even though the tenant is provisioned", got.Status.Phase) + } +} + +// TestReconcileUnsuspendingReturnsToActiveNotProvisioning is the +// regression test for why Phase is *derived* fresh every reconcile +// (from Spec.Suspended + whether ClickHouseDatabaseName is set) rather +// than toggled in place: un-suspending an already-provisioned tenant +// must return it straight to Active, not demote it to Provisioning just +// because the last observed Phase happened to be Suspended. +func TestReconcileUnsuspendingReturnsToActiveNotProvisioning(t *testing.T) { + tenant := testTenant("acme", true) + tenant.Status.ClickHouseDatabaseName = "acme" r := newFakeReconciler(t, tenant) ctx := context.Background() - if _, err := r.Reconcile(ctx, ctrl.Request{NamespacedName: types.NamespacedName{Name: "acme", Namespace: "default"}}); err != nil { - t.Fatalf("Reconcile: %v", err) - } + _ = reconcile(t, r, "acme") // establishes Suspended - var got sentryv1alpha1.Tenant - if err := r.Get(ctx, types.NamespacedName{Name: "acme", Namespace: "default"}, &got); err != nil { + var toUpdate sentryv1alpha1.Tenant + if err := r.Get(ctx, types.NamespacedName{Name: "acme", Namespace: "default"}, &toUpdate); err != nil { t.Fatalf("getting tenant: %v", err) } - if got.Status.Phase != sentryv1alpha1.PhaseSuspended { - t.Fatalf("Phase = %q, want Suspended", got.Status.Phase) + toUpdate.Spec.Suspended = false + if err := r.Update(ctx, &toUpdate); err != nil { + t.Fatalf("unsuspending: %v", err) } - // A suspended tenant's credential Secret is NOT deleted -- suspension - // is reversible and this controller doesn't manage ClickHouse-side - // grants, so there's nothing at this layer to actually enforce - // suspension; deleting the Secret would just be theater. - var secret corev1.Secret - if err := r.Get(ctx, types.NamespacedName{Name: "sentry-tenant-acme-clickhouse", Namespace: "default"}, &secret); err != nil { - t.Fatalf("expected secret to still exist for a suspended tenant: %v", err) - } -} - -func TestReconcileIsIdempotentAndNeverRotatesPassword(t *testing.T) { - tenant := testTenant("acme", false) - r := newFakeReconciler(t, tenant) - ctx := context.Background() - req := ctrl.Request{NamespacedName: types.NamespacedName{Name: "acme", Namespace: "default"}} - - if _, err := r.Reconcile(ctx, req); err != nil { - t.Fatalf("first Reconcile: %v", err) - } - var first corev1.Secret - if err := r.Get(ctx, types.NamespacedName{Name: "sentry-tenant-acme-clickhouse", Namespace: "default"}, &first); err != nil { - t.Fatalf("getting secret after first reconcile: %v", err) - } - - if _, err := r.Reconcile(ctx, req); err != nil { - t.Fatalf("second Reconcile: %v", err) - } - var second corev1.Secret - if err := r.Get(ctx, types.NamespacedName{Name: "sentry-tenant-acme-clickhouse", Namespace: "default"}, &second); err != nil { - t.Fatalf("getting secret after second reconcile: %v", err) - } - - if first.StringData["password"] != second.StringData["password"] { - t.Fatal("password changed across a re-reconcile -- would break every live connection for this tenant") + got := reconcile(t, r, "acme") + if got.Status.Phase != sentryv1alpha1.PhaseActive { + t.Fatalf("Phase = %q, want Active after unsuspending an already-provisioned tenant", got.Status.Phase) } } @@ -153,3 +166,12 @@ func TestReconcileMissingTenantIsNoOp(t *testing.T) { t.Fatalf("Reconcile on a missing tenant should be a no-op, got error: %v", err) } } + +func readyCondition(tenant sentryv1alpha1.Tenant) *metav1.Condition { + for i := range tenant.Status.Conditions { + if tenant.Status.Conditions[i].Type == sentryv1alpha1.ConditionReady { + return &tenant.Status.Conditions[i] + } + } + return nil +} diff --git a/docs/architecture.md b/docs/architecture.md index 50438f0..75f0667 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -146,23 +146,29 @@ escape hatch is opaque to any compiler-injected filter. query time — and permanently empty until something upstream of `chrunner`/`searchclient` becomes tenant-aware on the write side, which is undesigned, not merely unbuilt. -- `deploy/operator`'s `Tenant` CRD still manages only the K8s-side - artifact (a credential Secret); it doesn't call - `enterprise-api -provision-tenant` or otherwise trigger ClickHouse-side - provisioning. The two mechanisms are independent today, not reconciled - into one state machine. +- `deploy/operator`'s `Tenant` CRD and `enterprise-api -provision-tenant` + are now unified, deliberately lightweight: `-provision-tenant` stays + the sole real actor (ClickHouse + `rbacstore`), and now also syncs its + real result into the `Tenant` CRD (`enterprise/internal/tenantcrd`) -- + a real credential Secret, and status fields the reconciler + (`deploy/operator/internal/controller`) derives `Phase`/`Ready` from + rather than independently guessing. The reconciler itself gained no + new credentials and still never touches ClickHouse/Postgres. -**The deployment-topology gap is closed for the Helm chart**: `deploy/ -helm/sentry/templates/api.yaml`/`enterprise-api.yaml` are mutually -exclusive on `enterprise.enabled`, rendering to the same Service name -and port either way, so a Helm-deployed cluster can't accidentally run -the wrong binary — the same flag that turns on RBAC/audit/SSO now also -chooses the query binary. `docker-compose.yml` still runs plain `api` -unconditionally, so this enforcement doesn't yet extend to local/dev. -With both storage engines' connection/index-layer mechanisms built and -deployment topology enforced at the Helm layer, the largest remaining -gaps are ingest's lack of tenant-awareness (undesigned) and unifying the -`Tenant` CRD with `-provision-tenant` into one provisioning flow. +**The deployment-topology gap is closed for both Helm and +docker-compose**: `deploy/helm/sentry/templates/api.yaml`/ +`enterprise-api.yaml` are mutually exclusive on `enterprise.enabled`, +rendering to the same Service name and port either way, so a +Helm-deployed cluster can't accidentally run the wrong binary — the same +flag that turns on RBAC/audit/SSO now also chooses the query binary. +`docker-compose.yml`'s `api`/`enterprise-api` services are the same +mutually-exclusive choice via `COMPOSE_PROFILES` (`.env` defaults to +plain `api`), sharing a host-port/network-alias trick so `alerting`/ +`web` need no conditional config either way. With both storage engines' +connection/index-layer mechanisms built, deployment topology enforced at +both the Helm and docker-compose layers, and the two provisioning +mechanisms unified, the largest remaining gap is ingest's lack of +tenant-awareness, which is undesigned, not merely unbuilt. ## Licensing boundary diff --git a/docs/phase-4-runbook.md b/docs/phase-4-runbook.md index a00ad99..26a06ae 100644 --- a/docs/phase-4-runbook.md +++ b/docs/phase-4-runbook.md @@ -482,6 +482,45 @@ against a real daemon in this environment — the `config` rendering above proves the compose file's *shape* is correct, not that containers actually start and route traffic correctly end to end. +## 11. `Tenant` CRD unified with `-provision-tenant` + +`deploy/helm/sentry/README.md`'s "Trying the two-tenant example" section +has the full `helm install` → `-provision-tenant` → `kubectl get +tenants` walkthrough. What was actually run in this environment (no +live cluster, same limitation as §10): + +```sh +cd enterprise && go test ./internal/tenantcrd/... -v +# real k8s.io/client-go fake dynamic + typed clientsets, no cluster +# needed -- proves Sync() creates/updates the Tenant object and Secret +# correctly, is idempotent, never overwrites a pre-existing +# spec.displayName, and never rotates a credential across a re-sync. + +cd deploy/operator && go test ./... -v +# tenant_controller_test.go rewritten for the new split: proves an +# unprovisioned tenant reports Provisioning (not Active -- the +# regression test for the pre-unification bug where this controller +# claimed Active on its own say-so), that setting +# status.clickHouseDatabaseName (simulating what -provision-tenant +# writes) flips it to Active, and that un-suspending an +# already-provisioned tenant returns straight to Active rather than +# being demoted to Provisioning. +``` + +Helm-side wiring confirmed via `helm template` + parsing the rendered +YAML (not eyeballing it): with `tenantOperator.enabled=true`, +`enterprise-api` gets its own ServiceAccount/Role/RoleBinding scoped to +exactly `tenants`/`tenants/status`/`secrets`, `tenant-operator`'s own +ClusterRole no longer grants `secrets` at all, and `TENANT_CRD_NAMESPACE` +is only set on `enterprise-api`'s container when `tenantOperator.enabled` +is true (absent, and the ServiceAccount/Role absent too, with just +`enterprise.enabled=true`). **Not verified**: an actual `-provision-tenant` +run against a real cluster with the operator watching -- everything +above proves each half's logic and the chart's shape independently, not +the full loop (does the operator's watch actually re-trigger a reconcile +after `-provision-tenant`'s external status write the way controller- +runtime's default predicate is expected to). + ## Known gaps (do not treat this phase as done without reading these) Full accounting: `/docs/security/threat-model.md`. Headline items: @@ -494,10 +533,18 @@ Full accounting: `/docs/security/threat-model.md`. Headline items: is set. `docker-compose.yml`'s `api`/`enterprise-api` services are now the same mutually-exclusive choice via `COMPOSE_PROFILES` (§8, §10a), closing the local/dev parity gap this bullet used to name. -- The `Tenant` CRD (`deploy/operator`) and `enterprise-api - -provision-tenant` are still two independent provisioning mechanisms - -- running both for the same tenant ID today takes two separate - operator actions, not one. +- **The `Tenant` CRD (`deploy/operator`) and `enterprise-api + -provision-tenant` are now unified**, in a deliberately lightweight + way: `-provision-tenant` stays the sole real actor (ClickHouse + + `rbacstore`) and, when `TENANT_CRD_NAMESPACE` is set (the Helm chart + does this automatically when `tenantOperator.enabled`), also syncs the + real result into the Tenant CRD (`enterprise/internal/tenantcrd`) -- + see §11 below. Running `-provision-tenant` is still a separate, + deliberately manual operator action from `helm install`/`kubectl + apply -f tenant.yaml` creating the Tenant object in the first place -- + that split (declarative request vs. imperative provisioning action) + is intentional, not the "two disconnected sources of truth" gap this + bullet used to describe. - **Ingest has no tenant concept for either storage engine.** Every record `ingest` produces lands in the one shared ClickHouse database and the one shared Tantivy index no matter what. A newly-provisioned diff --git a/enterprise/README.md b/enterprise/README.md index ae2bd44..9f35239 100644 --- a/enterprise/README.md +++ b/enterprise/README.md @@ -166,6 +166,7 @@ internal/authhandler/ POST /internal/authorize, GET /auth/features internal/loginhandler/ GET /auth/oidc/{login,callback} + GET /auth/saml/login + POST /auth/saml/acs -- the human login flow internal/rbacstore/ users/tenants/tenant_memberships/data_sources/dashboard_permissions CRUD (pgx against sentry_metadata) internal/tenantprovision/ real ClickHouse CREATE DATABASE/USER/GRANT +internal/tenantcrd/ syncs -provision-tenant's real result into deploy/operator's Tenant CRD (K8s dynamic client, no cluster needed to test) internal/chrunner/ tenant-scoped api/querylang/executor.SQLRunner internal/searchclient/ tenant-scoped api/querylang/executor.SearchClient internal/audit/ append-only, hash-chained query audit log, plus the @@ -302,6 +303,14 @@ COMPOSE_PROFILES=enterprise docker compose up -d enterprise-api curl -s http://localhost:8080/healthz ``` +`docker-compose.yml` has no Kubernetes cluster to sync into, so +`TENANT_CRD_NAMESPACE` is never set here -- `-provision-tenant`'s +`internal/tenantcrd` sync step is a documented no-op in this deployment +shape, same as everywhere else this codebase has an "off unless +configured" optional dependency. It only does anything in a real +cluster with `deploy/helm/sentry`'s `tenantOperator.enabled=true` -- see +`/deploy/helm/sentry/README.md`'s "Trying the two-tenant example." + `-provision-tenant` creates the tenant/data_source rows in rbacstore if they don't exist, provisions ClickHouse, persists the credentials, and marks the tenant active -- refuses to run twice for the same tenant diff --git a/enterprise/cmd/enterprise-api/main.go b/enterprise/cmd/enterprise-api/main.go index 21d3b4d..ddf3d91 100644 --- a/enterprise/cmd/enterprise-api/main.go +++ b/enterprise/cmd/enterprise-api/main.go @@ -17,13 +17,14 @@ // keeps running plain api/cmd/api, unchanged; a real multi-tenant // deployment runs this one instead. // -// Not built yet: the actual K8s/Helm wiring to run this binary in place -// of api's (docker-compose.yml adds it available, not defaulted into -// the traffic path, same shape as enterprise-auth's own addition in -// Phase 4 task 5), and `search`'s write side (ingest, and by extension -// the Redpanda consumer search itself runs) is still not tenant-aware -- -// see enterprise/internal/searchclient and search/src/registry.rs's doc -// comments, and /docs/security/threat-model.md. +// Both Helm (deploy/helm/sentry/templates/api.yaml vs +// enterprise-api.yaml) and docker-compose.yml (COMPOSE_PROFILES) now +// make this the deployment-topology choice, not just a binary sitting +// unused alongside api's -- see this repo's CLAUDE.md. `search`'s write +// side (ingest, and by extension the Redpanda consumer search itself +// runs) is still not tenant-aware -- see enterprise/internal/searchclient +// and search/src/registry.rs's doc comments, and +// /docs/security/threat-model.md. package main import ( @@ -51,6 +52,7 @@ import ( "github.com/sentry/sentry/enterprise/internal/chrunner" "github.com/sentry/sentry/enterprise/internal/rbacstore" "github.com/sentry/sentry/enterprise/internal/searchclient" + "github.com/sentry/sentry/enterprise/internal/tenantcrd" "github.com/sentry/sentry/enterprise/internal/tenantprovision" ) @@ -193,6 +195,18 @@ func main() { // and only then mark the tenant active. Same "offline operator action, // not a network-reachable endpoint" shape as enterprise-auth's // -mint-service-token. +// +// If cfg.TenantCRDNamespace is set, this also syncs the real result into +// the Tenant CRD (enterprise/internal/tenantcrd) -- the "lightweight +// unification" of this mechanism with deploy/operator's Tenant CRD (see +// that package's doc comment). An already-active tenant no longer +// refuses outright: ClickHouse (re-)provisioning is still refused (that +// part is unchanged -- rotating a live credential would break every +// open connection for no benefit), but CR sync alone is safe to retry +// using the credentials already on file in rbacstore, which matters if +// a previous run's CR sync failed (or TENANT_CRD_NAMESPACE is being +// turned on for a tenant provisioned before this feature existed) and +// needs to be retried without touching ClickHouse again. func runProvisionTenant(ctx context.Context, logger *slog.Logger, cfg apiconfig.Config, rbac *rbacstore.Store, tenantID, displayName string) int { adminConn, err := chdriver.Open(&chdriver.Options{ Addr: []string{cfg.ClickHouseAddr}, @@ -221,9 +235,10 @@ func runProvisionTenant(ctx context.Context, logger *slog.Logger, cfg apiconfig. } logger.Info("created tenant row", "tenant_id", tenantID) } - if tenant.Status == "active" { - logger.Error("tenant is already active -- refusing to re-provision (would rotate a live credential)", "tenant_id", tenantID) - return 1 + + alreadyActive := tenant.Status == "active" + if alreadyActive { + logger.Info("tenant is already active -- skipping ClickHouse (re-)provisioning, will still sync the Tenant CRD if TENANT_CRD_NAMESPACE is set", "tenant_id", tenantID) } dataSource, err := rbac.GetDataSourceForTenant(ctx, tenantID) @@ -232,32 +247,64 @@ func runProvisionTenant(ctx context.Context, logger *slog.Logger, cfg apiconfig. logger.Error("getting data source", "error", err) return 1 } + if alreadyActive { + // An active tenant with no data_sources row at all is + // inconsistent state runProvisionTenant's own gate should + // never have allowed -- fail loudly rather than silently + // provisioning ClickHouse for a tenant already marked + // active, which SetDataSourceClickHouseCredentials's own + // doc comment says must never happen twice. + logger.Error("tenant is active but has no data source row -- inconsistent state, refusing", "tenant_id", tenantID) + return 1 + } dataSource, err = rbac.CreateDataSource(ctx, tenantID, "default", tenantID, "/var/lib/sentry-search/tenants/"+tenantID) if err != nil { logger.Error("creating data source", "error", err) return 1 } } - if dataSource.ClickHouseUsername != nil { - logger.Error("data source already has ClickHouse credentials -- refusing to re-provision", "tenant_id", tenantID) - return 1 + + var creds tenantprovision.Credentials + if alreadyActive { + if dataSource.ClickHouseUsername == nil || dataSource.ClickHousePassword == nil { + logger.Error("tenant is active but its data source has no ClickHouse credentials -- inconsistent state, refusing", "tenant_id", tenantID) + return 1 + } + creds = tenantprovision.Credentials{Username: *dataSource.ClickHouseUsername, Password: *dataSource.ClickHousePassword} + } else { + if dataSource.ClickHouseUsername != nil { + logger.Error("data source already has ClickHouse credentials -- refusing to re-provision", "tenant_id", tenantID) + return 1 + } + creds, err = tenantprovision.New(adminConn).ProvisionClickHouse(ctx, tenantID) + if err != nil { + logger.Error("provisioning clickhouse", "error", err) + return 1 + } + if err := rbac.SetDataSourceClickHouseCredentials(ctx, dataSource.ID, creds.Username, creds.Password); err != nil { + logger.Error("persisting clickhouse credentials", "error", err) + return 1 + } + if err := rbac.SetTenantStatus(ctx, tenantID, "active"); err != nil { + logger.Error("activating tenant", "error", err) + return 1 + } + logger.Info("tenant provisioned and active", "tenant_id", tenantID, "clickhouse_database", tenantID, "clickhouse_username", creds.Username) } - creds, err := tenantprovision.New(adminConn).ProvisionClickHouse(ctx, tenantID) - if err != nil { - logger.Error("provisioning clickhouse", "error", err) - return 1 - } - if err := rbac.SetDataSourceClickHouseCredentials(ctx, dataSource.ID, creds.Username, creds.Password); err != nil { - logger.Error("persisting clickhouse credentials", "error", err) - return 1 - } - if err := rbac.SetTenantStatus(ctx, tenantID, "active"); err != nil { - logger.Error("activating tenant", "error", err) - return 1 + if cfg.TenantCRDNamespace != "" { + syncer, err := tenantcrd.New(cfg.TenantCRDNamespace) + if err != nil { + logger.Error("building tenant CRD syncer", "error", err) + return 1 + } + if err := syncer.Sync(ctx, tenantID, tenant.DisplayName, dataSource.TantivyIndexPath, tenantcrd.Credentials{Username: creds.Username, Password: creds.Password}); err != nil { + logger.Error("syncing tenant CRD", "error", err) + return 1 + } + logger.Info("synced tenant CRD", "tenant_id", tenantID, "namespace", cfg.TenantCRDNamespace) } - logger.Info("tenant provisioned and active", "tenant_id", tenantID, "clickhouse_database", tenantID, "clickhouse_username", creds.Username) return 0 } diff --git a/enterprise/go.mod b/enterprise/go.mod index 6ae14d8..4cdfa15 100644 --- a/enterprise/go.mod +++ b/enterprise/go.mod @@ -28,6 +28,9 @@ require ( github.com/sentry/sentry/proto v0.0.0-00010101000000-000000000000 golang.org/x/oauth2 v0.36.0 google.golang.org/grpc v1.83.0 + k8s.io/api v0.31.0 + k8s.io/apimachinery v0.31.0 + k8s.io/client-go v0.31.0 ) require ( @@ -35,27 +38,61 @@ require ( github.com/andybalholm/brotli v1.2.2 // indirect github.com/beevik/etree v1.5.0 // indirect github.com/cespare/xxhash/v2 v2.3.0 // indirect + github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc // indirect + github.com/emicklei/go-restful/v3 v3.11.0 // indirect + github.com/fxamacker/cbor/v2 v2.7.0 // indirect github.com/go-faster/city v1.0.1 // indirect github.com/go-faster/errors v0.7.1 // indirect + github.com/go-logr/logr v1.4.3 // indirect + github.com/go-openapi/jsonpointer v0.19.6 // indirect + github.com/go-openapi/jsonreference v0.20.2 // indirect + github.com/go-openapi/swag v0.22.4 // indirect + github.com/gogo/protobuf v1.3.2 // indirect github.com/golang-jwt/jwt/v4 v4.5.2 // indirect + github.com/golang/protobuf v1.5.4 // indirect + github.com/google/gnostic-models v0.6.8 // indirect + github.com/google/go-cmp v0.7.0 // indirect + github.com/google/gofuzz v1.2.0 // indirect + github.com/imdario/mergo v0.3.6 // indirect github.com/jackc/pgpassfile v1.0.0 // indirect github.com/jackc/pgservicefile v0.0.0-20240606120523-5a60cdf6a761 // indirect github.com/jackc/puddle/v2 v2.2.2 // indirect github.com/jonboulle/clockwork v0.2.2 // indirect + github.com/josharian/intern v1.0.0 // indirect + github.com/json-iterator/go v1.1.12 // indirect github.com/klauspost/compress v1.19.1 // indirect + github.com/mailru/easyjson v0.7.7 // indirect github.com/mattermost/xml-roundtrip-validator v0.1.0 // indirect + github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd // indirect + github.com/modern-go/reflect2 v1.0.2 // indirect + github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 // indirect github.com/paulmach/orb v0.13.0 // indirect github.com/pierrec/lz4/v4 v4.1.27 // indirect + github.com/pkg/errors v0.9.1 // indirect github.com/russellhaering/goxmldsig v1.4.0 // indirect github.com/segmentio/asm v1.2.1 // indirect github.com/shopspring/decimal v1.4.0 // indirect + github.com/spf13/pflag v1.0.5 // indirect + github.com/x448/float16 v0.8.4 // indirect go.opentelemetry.io/otel v1.44.0 // indirect go.opentelemetry.io/otel/trace v1.44.0 // indirect golang.org/x/crypto v0.54.0 // indirect golang.org/x/net v0.57.0 // indirect golang.org/x/sync v0.22.0 // indirect golang.org/x/sys v0.47.0 // indirect + golang.org/x/term v0.45.0 // indirect golang.org/x/text v0.40.0 // indirect + golang.org/x/time v0.3.0 // indirect google.golang.org/genproto/googleapis/rpc v0.0.0-20260526163538-3dc84a4a5aaa // indirect google.golang.org/protobuf v1.36.12 // indirect + gopkg.in/evanphx/json-patch.v4 v4.12.0 // indirect + gopkg.in/inf.v0 v0.9.1 // indirect + gopkg.in/yaml.v2 v2.4.0 // indirect + gopkg.in/yaml.v3 v3.0.1 // indirect + k8s.io/klog/v2 v2.130.1 // indirect + k8s.io/kube-openapi v0.0.0-20240228011516-70dd3763d340 // indirect + k8s.io/utils v0.0.0-20240711033017-18e509b52bc8 // indirect + sigs.k8s.io/json v0.0.0-20221116044647-bc3834ca7abd // indirect + sigs.k8s.io/structured-merge-diff/v4 v4.4.1 // indirect + sigs.k8s.io/yaml v1.4.0 // indirect ) diff --git a/enterprise/go.sum b/enterprise/go.sum index 811e1e5..477c7b0 100644 --- a/enterprise/go.sum +++ b/enterprise/go.sum @@ -15,8 +15,13 @@ github.com/creack/pty v1.1.9/go.mod h1:oKZEueFk5CKHvIhNR5MUki03XCEU+Q6VDXinZuGJ3 github.com/crewjam/saml v0.5.1 h1:g+mfp0CrLuLRZCK793PgJcZeg5dS/0CDwoeAX2zcwNI= github.com/crewjam/saml v0.5.1/go.mod h1:r0fDkmFe5URDgPrmtH0IYokva6fac3AUdstiPhyEolQ= github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38= -github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c= github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38= +github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc h1:U9qPSI2PIWSS1VwoXQT9A3Wy9MM3WgvqSxFWenqJduM= +github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38= +github.com/emicklei/go-restful/v3 v3.11.0 h1:rAQeMHw1c7zTmncogyy8VvRZwtkmkZ4FxERmMY4rD+g= +github.com/emicklei/go-restful/v3 v3.11.0/go.mod h1:6n3XBCmQQb25CM2LCACGz8ukIrRry+4bhvbpWn3mrbc= +github.com/fxamacker/cbor/v2 v2.7.0 h1:iM5WgngdRBanHcxugY4JySA0nk1wZorNOpTgCMedv5E= +github.com/fxamacker/cbor/v2 v2.7.0/go.mod h1:pxXPTn3joSm21Gbwsv0w9OSA2y1HFR9qXEeXQVeNoDQ= github.com/go-faster/city v1.0.1 h1:4WAxSZ3V2Ws4QRDrscLEDcibJY8uf41H6AhXDrNDcGw= github.com/go-faster/city v1.0.1/go.mod h1:jKcUJId49qdW3L1qKHH/3wPeUstCVpVSXTM6vO3VcTw= github.com/go-faster/errors v0.7.1 h1:MkJTnDoEdi9pDabt1dpWf7AA8/BaSYZqibYyhZ20AYg= @@ -27,14 +32,35 @@ github.com/go-logr/logr v1.4.3 h1:CjnDlHq8ikf6E492q6eKboGOC0T8CDaOvkHCIg8idEI= github.com/go-logr/logr v1.4.3/go.mod h1:9T104GzyrTigFIr8wt5mBrctHMim0Nb2HLGrmQ40KvY= github.com/go-logr/stdr v1.2.2 h1:hSWxHoqTgW2S2qGc0LTAI563KZ5YKYRhT3MFKZMbjag= github.com/go-logr/stdr v1.2.2/go.mod h1:mMo/vtBO5dYbehREoey6XUKy/eSumjCCveDpRre4VKE= +github.com/go-openapi/jsonpointer v0.19.6 h1:eCs3fxoIi3Wh6vtgmLTOjdhSpiqphQ+DaPn38N2ZdrE= +github.com/go-openapi/jsonpointer v0.19.6/go.mod h1:osyAmYz/mB/C3I+WsTTSgw1ONzaLJoLCyoi6/zppojs= +github.com/go-openapi/jsonreference v0.20.2 h1:3sVjiK66+uXK/6oQ8xgcRKcFgQ5KXa2KvnJRumpMGbE= +github.com/go-openapi/jsonreference v0.20.2/go.mod h1:Bl1zwGIM8/wsvqjsOQLJ/SH+En5Ap4rVB5KVcIDZG2k= +github.com/go-openapi/swag v0.22.3/go.mod h1:UzaqsxGiab7freDnrUUra0MwWfN/q7tE4j+VcZ0yl14= +github.com/go-openapi/swag v0.22.4 h1:QLMzNJnMGPRNDCbySlcj1x01tzU8/9LTTL9hZZZogBU= +github.com/go-openapi/swag v0.22.4/go.mod h1:UzaqsxGiab7freDnrUUra0MwWfN/q7tE4j+VcZ0yl14= +github.com/go-task/slim-sprig/v3 v3.0.0 h1:sUs3vkvUymDpBKi3qH1YSqBQk9+9D/8M2mN1vB6EwHI= +github.com/go-task/slim-sprig/v3 v3.0.0/go.mod h1:W848ghGpv3Qj3dhTPRyJypKRiqCdHZiAzKg9hl15HA8= +github.com/gogo/protobuf v1.3.2 h1:Ov1cvc58UF3b5XjBnZv7+opcTcQFZebYjWzi34vdm4Q= +github.com/gogo/protobuf v1.3.2/go.mod h1:P1XiOD3dCwIKUDQYPy72D8LYyHL2YPYrpS2s69NZV8Q= github.com/golang-jwt/jwt/v4 v4.5.2 h1:YtQM7lnr8iZ+j5q71MGKkNw9Mn7AjHM68uc9g5fXeUI= github.com/golang-jwt/jwt/v4 v4.5.2/go.mod h1:m21LjoU+eqJr34lmDMbreY2eSTRJ1cv77w39/MY0Ch0= github.com/golang/protobuf v1.5.4 h1:i7eJL8qZTpSEXOPTxNKhASYpMn+8e5Q6AdndVa1dWek= github.com/golang/protobuf v1.5.4/go.mod h1:lnTiLA8Wa4RWRcIUkrtSVa5nRhsEGBg48fD6rSs7xps= +github.com/google/gnostic-models v0.6.8 h1:yo/ABAfM5IMRsS1VnXjTBvUb61tFIHozhlYvRgGre9I= +github.com/google/gnostic-models v0.6.8/go.mod h1:5n7qKqH0f5wFt+aWF8CW6pZLLNOfYuF5OpfBSENuI8U= +github.com/google/go-cmp v0.5.9/go.mod h1:17dUlkBOakJ0+DkrSSNjCkIjxS6bF9zb3elmeNGIjoY= github.com/google/go-cmp v0.7.0 h1:wk8382ETsv4JYUZwIsn6YpYiWiBsYLSJiTsyBybVuN8= github.com/google/go-cmp v0.7.0/go.mod h1:pXiqmnSA92OHEEa9HXL2W4E7lf9JzCmGVUdgjX3N/iU= +github.com/google/gofuzz v1.0.0/go.mod h1:dBl0BpW6vV/+mYPU4Po3pmUjxk6FQPldtuIdl/M65Eg= +github.com/google/gofuzz v1.2.0 h1:xRy4A+RhZaiKjJ1bPfwQ8sedCA+YS2YcCHW6ec7JMi0= +github.com/google/gofuzz v1.2.0/go.mod h1:dBl0BpW6vV/+mYPU4Po3pmUjxk6FQPldtuIdl/M65Eg= +github.com/google/pprof v0.0.0-20240525223248-4bfdf5a9a2af h1:kmjWCqn2qkEml422C2Rrd27c3VGxi6a/6HNq8QmHRKM= +github.com/google/pprof v0.0.0-20240525223248-4bfdf5a9a2af/go.mod h1:K1liHPHnj73Fdn/EKuT8nrFqBihUSKXoLYU0BuatOYo= github.com/google/uuid v1.6.0 h1:NIvaJDMOsjHA8n1jAhLSgzrAzy1Hgr+hNrb57e+94F0= github.com/google/uuid v1.6.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo= +github.com/imdario/mergo v0.3.6 h1:xTNEAn+kxVO7dTZGu0CegyqKZmoWFI0rF8UxjlB2d28= +github.com/imdario/mergo v0.3.6/go.mod h1:2EnlNZ0deacrJVfApfmtdGgDfMuh/nq6Ok1EcJh5FfA= github.com/jackc/pgpassfile v1.0.0 h1:/6Hmqy13Ss2zCq62VdNG8tM1wchn8zjSGOBJ6icpsIM= github.com/jackc/pgpassfile v1.0.0/go.mod h1:CEx0iS5ambNFdcRtxPj5JhEz+xB6uRky5eyVu/W2HEg= github.com/jackc/pgservicefile v0.0.0-20240606120523-5a60cdf6a761 h1:iCEnooe7UlwOQYpKFhBabPMi4aNAfoODPEFNiAnClxo= @@ -45,16 +71,38 @@ github.com/jackc/puddle/v2 v2.2.2 h1:PR8nw+E/1w0GLuRFSmiioY6UooMp6KJv0/61nB7icHo github.com/jackc/puddle/v2 v2.2.2/go.mod h1:vriiEXHvEE654aYKXXjOvZM39qJ0q+azkZFrfEOc3H4= github.com/jonboulle/clockwork v0.2.2 h1:UOGuzwb1PwsrDAObMuhUnj0p5ULPj8V/xJ7Kx9qUBdQ= github.com/jonboulle/clockwork v0.2.2/go.mod h1:Pkfl5aHPm1nk2H9h0bjmnJD/BcgbGXUBGnn1kMkgxc8= +github.com/josharian/intern v1.0.0 h1:vlS4z54oSdjm0bgjRigI+G1HpF+tI+9rE5LLzOg8HmY= +github.com/josharian/intern v1.0.0/go.mod h1:5DoeVV0s6jJacbCEi61lwdGj/aVlrQvzHFFd8Hwg//Y= +github.com/json-iterator/go v1.1.12 h1:PV8peI4a0ysnczrg+LtxykD8LfKY9ML6u2jnxaEnrnM= +github.com/json-iterator/go v1.1.12/go.mod h1:e30LSqwooZae/UwlEbR2852Gd8hjQvJoHmT4TnhNGBo= +github.com/kisielk/errcheck v1.5.0/go.mod h1:pFxgyoBC7bSaBwPgfKdkLd5X25qrDl4LWUI2bnpBCr8= +github.com/kisielk/gotool v1.0.0/go.mod h1:XhKaO+MFFWcvkIS/tQcRk01m1F5IRFswLeQ+oQHNcck= github.com/klauspost/compress v1.19.1 h1:VsB4HPswih7mmZ8WleSFQ75c/Ui1M4trX5oAsJnhSlk= github.com/klauspost/compress v1.19.1/go.mod h1:cwPg85FWrGar70rWktvGQj8/hthj3wpl0PGDogxkrSQ= github.com/kr/pretty v0.1.0/go.mod h1:dAy3ld7l9f0ibDNOQOHHMYYIIbhfbHSm3C4ZsoJORNo= github.com/kr/pretty v0.2.1/go.mod h1:ipq/a2n7PKx3OHsz4KJII5eveXtPO4qwEXGdVfWzfnI= github.com/kr/pretty v0.3.0/go.mod h1:640gp4NfQd8pI5XOwp5fnNeVWj67G7CFk/SaSQn7NBk= +github.com/kr/pretty v0.3.1 h1:flRD4NNwYAUpkphVc1HcthR4KEIFJ65n8Mw5qdRn3LE= +github.com/kr/pretty v0.3.1/go.mod h1:hoEshYVHaxMs3cyo3Yncou5ZscifuDolrwPKZanG3xk= github.com/kr/pty v1.1.1/go.mod h1:pFQYn66WHrOpPYNljwOMqo10TkYh1fy3cYio2l3bCsQ= github.com/kr/text v0.1.0/go.mod h1:4Jbv+DJW3UT/LiOwJeYQe1efqtUx/iVham/4vfdArNI= +github.com/kr/text v0.2.0 h1:5Nx0Ya0ZqY2ygV366QzturHI13Jq95ApcVaJBhpS+AY= github.com/kr/text v0.2.0/go.mod h1:eLer722TekiGuMkidMxC/pM04lWEeraHUUmBw8l2grE= +github.com/mailru/easyjson v0.7.7 h1:UGYAvKxe3sBsEDzO8ZeWOSlIQfWFlxbzLZe7hwFURr0= +github.com/mailru/easyjson v0.7.7/go.mod h1:xzfreul335JAWq5oZzymOObrkdz5UnU4kGfJJLY9Nlc= github.com/mattermost/xml-roundtrip-validator v0.1.0 h1:RXbVD2UAl7A7nOTR4u7E3ILa4IbtvKBHw64LDsmu9hU= github.com/mattermost/xml-roundtrip-validator v0.1.0/go.mod h1:qccnGMcpgwcNaBnxqpJpWWUiPNr5H3O8eDgGV9gT5To= +github.com/modern-go/concurrent v0.0.0-20180228061459-e0a39a4cb421/go.mod h1:6dJC0mAP4ikYIbvyc7fijjWJddQyLn8Ig3JB5CqoB9Q= +github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd h1:TRLaZ9cD/w8PVh93nsPXa1VrQ6jlwL5oN8l14QlcNfg= +github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd/go.mod h1:6dJC0mAP4ikYIbvyc7fijjWJddQyLn8Ig3JB5CqoB9Q= +github.com/modern-go/reflect2 v1.0.2 h1:xBagoLtFs94CBntxluKeaWgTMpvLxC4ur3nMaC9Gz0M= +github.com/modern-go/reflect2 v1.0.2/go.mod h1:yWuevngMOJpCy52FWWMvUC8ws7m/LJsjYzDa0/r8luk= +github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 h1:C3w9PqII01/Oq1c1nUAm88MOHcQC9l5mIlSMApZMrHA= +github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822/go.mod h1:+n7T8mK8HuQTcFwEeznm/DIxMOiR9yIdICNftLE1DvQ= +github.com/onsi/ginkgo/v2 v2.19.0 h1:9Cnnf7UHo57Hy3k6/m5k3dRfGTMXGvxhHFvkDTCTpvA= +github.com/onsi/ginkgo/v2 v2.19.0/go.mod h1:rlwLi9PilAFJ8jCg9UE1QP6VBpd6/xj3SRC0d6TU0To= +github.com/onsi/gomega v1.19.0 h1:4ieX6qQjPP/BfC3mpsAtIGGlxTWPeA3Inl/7DtXw1tw= +github.com/onsi/gomega v1.19.0/go.mod h1:LY+I3pBVzYsTBU1AnDwOSxaYi9WoWiqgwooUqq9yPro= github.com/paulmach/orb v0.13.0 h1:r7n7mQGGF+cj/CbcivEj9J3HGK+XR+yXnvzRdq9saIw= github.com/paulmach/orb v0.13.0/go.mod h1:6scRWINywA2Jf05dcjOfLfxrUIMECvTSG2MVbRLxu/k= github.com/pierrec/lz4/v4 v4.1.27 h1:+PhzhWDrjRj89TH2sw43nE3+4+W8lSxIuQadEHZyjUk= @@ -67,20 +115,33 @@ github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2 h1:Jamvg5psRI github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4= github.com/rogpeppe/go-internal v1.6.1/go.mod h1:xXDCJY+GAPziupqXw64V24skbSoqbTEfhy4qGm1nDQc= github.com/rogpeppe/go-internal v1.8.0/go.mod h1:WmiCO8CzOY8rg0OYDC4/i/2WRWAB6poM+XZ2dLUbcbE= +github.com/rogpeppe/go-internal v1.12.0 h1:exVL4IDcn6na9z1rAb56Vxr+CgyK3nn3O+epU5NdKM8= +github.com/rogpeppe/go-internal v1.12.0/go.mod h1:E+RYuTGaKKdloAfM02xzb0FW3Paa99yedzYV+kq4uf4= github.com/russellhaering/goxmldsig v1.4.0 h1:8UcDh/xGyQiyrW+Fq5t8f+l2DLB1+zlhYzkPUJ7Qhys= github.com/russellhaering/goxmldsig v1.4.0/go.mod h1:gM4MDENBQf7M+V824SGfyIUVFWydB7n0KkEubVJl+Tw= github.com/segmentio/asm v1.2.1 h1:DTNbBqs57ioxAD4PrArqftgypG4/qNpXoJx8TVXxPR0= github.com/segmentio/asm v1.2.1/go.mod h1:BqMnlJP91P8d+4ibuonYZw9mfnzI9HfxselHZr5aAcs= github.com/shopspring/decimal v1.4.0 h1:bxl37RwXBklmTi0C79JfXCEBD1cqqHt0bbgBAGFp81k= github.com/shopspring/decimal v1.4.0/go.mod h1:gawqmDU56v4yIKSwfBSFip1HdCCXN8/+DMd9qYNcwME= +github.com/spf13/pflag v1.0.5 h1:iy+VFUOCP1a+8yFto/drg2CJ5u0yRoB7fZw3DKv/JXA= +github.com/spf13/pflag v1.0.5/go.mod h1:McXfInJRrz4CZXVZOBLb0bTZqETkiAhM9Iw0y3An2Bg= github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME= +github.com/stretchr/objx v0.4.0/go.mod h1:YvHI0jy2hoMjB+UWwv71VJQ9isScKT/TqJzVSSt89Yw= +github.com/stretchr/objx v0.5.0/go.mod h1:Yh+to48EsGEfYuaHDzXPcE3xhTkx73EhmCGUpEOglKo= github.com/stretchr/testify v1.3.0/go.mod h1:M5WIy9Dh21IEIfnGCwXGc5bZfKNJtfHm1UVUgZn+9EI= github.com/stretchr/testify v1.6.1/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg= github.com/stretchr/testify v1.7.0/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg= +github.com/stretchr/testify v1.7.1/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg= +github.com/stretchr/testify v1.8.0/go.mod h1:yNjHg4UonilssWZ8iaSj1OCr/vHnekPRkoO+kdMU+MU= +github.com/stretchr/testify v1.8.1/go.mod h1:w2LPCIKwWwSfY2zedu0+kehJoqGctiVI29o6fzry7u4= github.com/stretchr/testify v1.11.1 h1:7s2iGBzp5EwR7/aIZr8ao5+dra3wiQyKjjFuvgVKu7U= github.com/stretchr/testify v1.11.1/go.mod h1:wZwfW3scLgRK+23gO65QZefKpKQRnfz6sD981Nm4B6U= +github.com/x448/float16 v0.8.4 h1:qLwI1I70+NjRFUR3zs1JPUCgaCXSh3SW62uAKT1mSBM= +github.com/x448/float16 v0.8.4/go.mod h1:14CWIYCyZA/cWjXOioeEpHeN/83MdbZDRQHoFcYsOfg= github.com/xyproto/randomstring v1.0.5 h1:YtlWPoRdgMu3NZtP45drfy1GKoojuR7hmRcnhZqKjWU= github.com/xyproto/randomstring v1.0.5/go.mod h1:rgmS5DeNXLivK7YprL0pY+lTuhNQW3iGxZ18UQApw/E= +github.com/yuin/goldmark v1.1.27/go.mod h1:3hX8gzYuyVAZsxl0MRgGTJEmQBFcNTphYh9decYSb74= +github.com/yuin/goldmark v1.2.1/go.mod h1:3hX8gzYuyVAZsxl0MRgGTJEmQBFcNTphYh9decYSb74= go.opentelemetry.io/auto/sdk v1.2.1 h1:jXsnJ4Lmnqd11kwkBV2LgLoFMZKizbCi5fNZ/ipaZ64= go.opentelemetry.io/auto/sdk v1.2.1/go.mod h1:KRTj+aOaElaLi+wW1kO/DZRXwkF4C5xPbEe3ZiIhN7Y= go.opentelemetry.io/otel v1.44.0 h1:JjwHmHpA4iZ3wBxluu2fbbE7j4kqlE8jXyAyPXH7HqU= @@ -93,18 +154,49 @@ go.opentelemetry.io/otel/sdk/metric v1.44.0 h1:3LlKgI+VjbVsjNRFZJZAJ30WjXC5VkNRk go.opentelemetry.io/otel/sdk/metric v1.44.0/go.mod h1:5B5pMARnXxKhltooO4xUuCBorl65a4EpnTalObqOigA= go.opentelemetry.io/otel/trace v1.44.0 h1:jxF5CsGYCe74MCRx2X4g7WsY/VBKRqqpNvXlX/6gtIk= go.opentelemetry.io/otel/trace v1.44.0/go.mod h1:oLl1jrMQAVo6v3GAggN+1VH9VIz9iUSvW53sW1Q8PIE= +golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w= +golang.org/x/crypto v0.0.0-20191011191535-87dc89f01550/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI= +golang.org/x/crypto v0.0.0-20200622213623-75b288015ac9/go.mod h1:LzIPMQfyMNhhGPhUkYOs5KpL4U8rLKemX1yGLhDgUto= golang.org/x/crypto v0.54.0 h1:YLIA59K4fiNzHzjnZt2tUJQjQtUWfWbeHBqKtk3eScw= golang.org/x/crypto v0.54.0/go.mod h1:KWL8ny2AZdGR2cWmzeHrp2azQPGogOv+HeQaVEXC2dk= +golang.org/x/mod v0.2.0/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA= +golang.org/x/mod v0.3.0/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA= +golang.org/x/net v0.0.0-20190404232315-eb5bcb51f2a3/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg= +golang.org/x/net v0.0.0-20190620200207-3b0461eec859/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s= +golang.org/x/net v0.0.0-20200226121028-0de0cce0169b/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s= +golang.org/x/net v0.0.0-20201021035429-f5854403a974/go.mod h1:sp8m0HH+o8qH0wwXwYZr8TS3Oi6o0r6Gce1SSxlDquU= golang.org/x/net v0.57.0 h1:K5+3DljvIuDG9/Jv9rvyMywYNFCQ9RSUY6OOTTkT+tE= golang.org/x/net v0.57.0/go.mod h1:KpXc8iv+r3XplLAG/f7Jsf9RPszJzdR0f58q9vGOuEU= golang.org/x/oauth2 v0.36.0 h1:peZ/1z27fi9hUOFCAZaHyrpWG5lwe0RJEEEeH0ThlIs= golang.org/x/oauth2 v0.36.0/go.mod h1:YDBUJMTkDnJS+A4BP4eZBjCqtokkg1hODuPjwiGPO7Q= +golang.org/x/sync v0.0.0-20190423024810-112230192c58/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= +golang.org/x/sync v0.0.0-20190911185100-cd5d95a43a6e/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= +golang.org/x/sync v0.0.0-20201020160332-67f06af15bc9/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= golang.org/x/sync v0.22.0 h1:SZjpbeLmrCk4xhRSZFNZW5gFUeCeFgjekvI/+gfScek= golang.org/x/sync v0.22.0/go.mod h1:9xrNwdLfx4jkKbNva9FpL6vEN7evnE43NNNJQ2LF3+0= +golang.org/x/sys v0.0.0-20190215142949-d0b11bdaac8a/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY= +golang.org/x/sys v0.0.0-20190412213103-97732733099d/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= +golang.org/x/sys v0.0.0-20200930185726-fdedc70b468f/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= golang.org/x/sys v0.47.0 h1:o7XGOvZQCADBQQ4Y7VNq2dRWQR7JmOUW8Kxx4ZsNgWs= golang.org/x/sys v0.47.0/go.mod h1:4GL1E5IUh+htKOUEOaiffhrAeqysfVGipDYzABqnCmw= +golang.org/x/term v0.45.0 h1:NwWyBmoJCbfTHpxrWoZ9C6/VxOf7ic219I8xZZFdrf0= +golang.org/x/term v0.45.0/go.mod h1:9aqxs0blBcrm/n0L9QW0aRVD+ktan8ssZromtqJC43w= +golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ= +golang.org/x/text v0.3.3/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ= golang.org/x/text v0.40.0 h1:Ub2Z6/xjgF1WrYQz2nuITOEegKFtiIy+rieRJ5lHZKs= golang.org/x/text v0.40.0/go.mod h1:hpnzDAfGV753zIKo+wk3u1bVKCGPbrnF7+7LBF/UHVY= +golang.org/x/time v0.3.0 h1:rg5rLMjNzMS1RkNLzCG38eapWhnYLFYXDXj2gOlr8j4= +golang.org/x/time v0.3.0/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ= +golang.org/x/tools v0.0.0-20180917221912-90fa682c2a6e/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ= +golang.org/x/tools v0.0.0-20191119224855-298f0cb1881e/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo= +golang.org/x/tools v0.0.0-20200619180055-7c47624df98f/go.mod h1:EkVYQZoAsY45+roYkvgYkIh4xh/qjgUK9TdY2XT94GE= +golang.org/x/tools v0.0.0-20210106214847-113979e3529a/go.mod h1:emZCQorbCU4vsT4fOWvOPXz4eW1wZW4PmDk9uLelYpA= +golang.org/x/tools v0.47.0 h1:7Kn5x/d1svx/PzryTsqeoZN4TZwqeH5pGWjefhLi/1Q= +golang.org/x/tools v0.47.0/go.mod h1:dFHnyTvFWY212G+h7ZY4Vsp/K3U4/7W9TyVaAul8uCA= +golang.org/x/xerrors v0.0.0-20190717185122-a985d3407aa7/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0= +golang.org/x/xerrors v0.0.0-20191011141410-1b5146add898/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0= +golang.org/x/xerrors v0.0.0-20191204190536-9bdfabe68543/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0= +golang.org/x/xerrors v0.0.0-20200804184101-5ec99f83aff1/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0= gonum.org/v1/gonum v0.17.0 h1:VbpOemQlsSMrYmn7T2OUvQ4dqxQXU+ouZFQsZOx50z4= gonum.org/v1/gonum v0.17.0/go.mod h1:El3tOrEuMpv2UdMrbNlKEh9vd86bmQ6vqIcDwxEOc1E= google.golang.org/genproto/googleapis/rpc v0.0.0-20260526163538-3dc84a4a5aaa h1:mZHHdPZl0dbGHCflZgAq/Q468DWVFcU2whhB2KAo8fk= @@ -115,11 +207,37 @@ google.golang.org/protobuf v1.36.12 h1:pJOKDDOyeXErUroCihFAd5LQuwXBSpVnKGrj5o/fw google.golang.org/protobuf v1.36.12/go.mod h1:HTf+CrKn2C3g5S8VImy6tdcUvCska2kB7j23XfzDpco= gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0= gopkg.in/check.v1 v1.0.0-20180628173108-788fd7840127/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0= +gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c h1:Hei/4ADfdWqJk1ZMxUNpqntNwaWcugrBjAiHlqqRiVk= gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c/go.mod h1:JHkPIbrfpd72SG/EVd6muEfDQjcINNoR0C8j2r3qZ4Q= gopkg.in/errgo.v2 v2.1.0/go.mod h1:hNsd1EY+bozCKY1Ytp96fpM3vjJbqLJn88ws8XvfDNI= +gopkg.in/evanphx/json-patch.v4 v4.12.0 h1:n6jtcsulIzXPJaxegRbvFNNrZDjbij7ny3gmSPG+6V4= +gopkg.in/evanphx/json-patch.v4 v4.12.0/go.mod h1:p8EYWUEYMpynmqDbY58zCKCFZw8pRWMG4EsWvDvM72M= +gopkg.in/inf.v0 v0.9.1 h1:73M5CoZyi3ZLMOyDlQh031Cx6N9NDJ2Vvfl76EDAgDc= +gopkg.in/inf.v0 v0.9.1/go.mod h1:cWUDdTG/fYaXco+Dcufb5Vnc6Gp2YChqWtbxRZE0mXw= +gopkg.in/yaml.v2 v2.2.8/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI= +gopkg.in/yaml.v2 v2.4.0 h1:D8xgwECY7CYvx+Y2n4sBz93Jn9JRvxdiyyo8CTfuKaY= +gopkg.in/yaml.v2 v2.4.0/go.mod h1:RDklbk79AGWmwhnvt/jBztapEOGDOx6ZbXqjP6csGnQ= gopkg.in/yaml.v3 v3.0.0-20200313102051-9f266ea9e77c/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM= gopkg.in/yaml.v3 v3.0.0-20210107192922-496545a6307b/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM= gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA= gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM= gotest.tools v2.2.0+incompatible h1:VsBPFP1AI068pPrMxtb/S8Zkgf9xEmTLJjfM+P5UIEo= gotest.tools v2.2.0+incompatible/go.mod h1:DsYFclhRJ6vuDpmuTbkuFWG+y2sxOXAzmJt81HFBacw= +k8s.io/api v0.31.0 h1:b9LiSjR2ym/SzTOlfMHm1tr7/21aD7fSkqgD/CVJBCo= +k8s.io/api v0.31.0/go.mod h1:0YiFF+JfFxMM6+1hQei8FY8M7s1Mth+z/q7eF1aJkTE= +k8s.io/apimachinery v0.31.0 h1:m9jOiSr3FoSSL5WO9bjm1n6B9KROYYgNZOb4tyZ1lBc= +k8s.io/apimachinery v0.31.0/go.mod h1:rsPdaZJfTfLsNJSQzNHQvYoTmxhoOEofxtOsF3rtsMo= +k8s.io/client-go v0.31.0 h1:QqEJzNjbN2Yv1H79SsS+SWnXkBgVu4Pj3CJQgbx0gI8= +k8s.io/client-go v0.31.0/go.mod h1:Y9wvC76g4fLjmU0BA+rV+h2cncoadjvjjkkIGoTLcGU= +k8s.io/klog/v2 v2.130.1 h1:n9Xl7H1Xvksem4KFG4PYbdQCQxqc/tTUyrgXaOhHSzk= +k8s.io/klog/v2 v2.130.1/go.mod h1:3Jpz1GvMt720eyJH1ckRHK1EDfpxISzJ7I9OYgaDtPE= +k8s.io/kube-openapi v0.0.0-20240228011516-70dd3763d340 h1:BZqlfIlq5YbRMFko6/PM7FjZpUb45WallggurYhKGag= +k8s.io/kube-openapi v0.0.0-20240228011516-70dd3763d340/go.mod h1:yD4MZYeKMBwQKVht279WycxKyM84kkAx2DPrTXaeb98= +k8s.io/utils v0.0.0-20240711033017-18e509b52bc8 h1:pUdcCO1Lk/tbT5ztQWOBi5HBgbBP1J8+AsQnQCKsi8A= +k8s.io/utils v0.0.0-20240711033017-18e509b52bc8/go.mod h1:OLgZIPagt7ERELqWJFomSt595RzquPNLL48iOWgYOg0= +sigs.k8s.io/json v0.0.0-20221116044647-bc3834ca7abd h1:EDPBXCAspyGV4jQlpZSudPeMmr1bNJefnuqLsRAsHZo= +sigs.k8s.io/json v0.0.0-20221116044647-bc3834ca7abd/go.mod h1:B8JuhiUyNFVKdsE8h686QcCxMaH6HrOAZj4vswFpcB0= +sigs.k8s.io/structured-merge-diff/v4 v4.4.1 h1:150L+0vs/8DA78h1u02ooW1/fFq/Lwr+sGiqlzvrtq4= +sigs.k8s.io/structured-merge-diff/v4 v4.4.1/go.mod h1:N8hJocpFajUSSeSJ9bOZ77VzejKZaXsTtZo4/u7Io08= +sigs.k8s.io/yaml v1.4.0 h1:Mk1wCc2gy/F0THH0TAp1QYyJNzRm2KCLy3o5ASXVI5E= +sigs.k8s.io/yaml v1.4.0/go.mod h1:Ejl7/uTz7PSA4eKMyQCUTnhZYNmLIl+5c2lQPGR2BPY= diff --git a/enterprise/internal/apiconfig/apiconfig.go b/enterprise/internal/apiconfig/apiconfig.go index 4c03495..9110179 100644 --- a/enterprise/internal/apiconfig/apiconfig.go +++ b/enterprise/internal/apiconfig/apiconfig.go @@ -41,6 +41,13 @@ type Config struct { // "never break a simpler deployment shape" reasoning used // throughout this codebase. EnterpriseAuthURL string + // TenantCRDNamespace enables enterprise/internal/tenantcrd syncing + // for -provision-tenant (cmd/enterprise-api/main.go's + // runProvisionTenant) -- empty (the default) means skip it entirely, + // same "off unless configured" shape as EnterpriseAuthURL above. + // Deployments with no Kubernetes cluster at all (docker-compose) + // never set this. + TenantCRDNamespace string } type ClickHouseAdminConfig struct { @@ -81,9 +88,10 @@ func Load() (Config, error) { Username: getenv("AUDIT_WRITER_USERNAME", "audit_writer"), Password: getenv("AUDIT_WRITER_PASSWORD", ""), }, - SearchGRPCAddr: getenv("SEARCH_GRPC_ADDR", "localhost:50052"), - CORSAllowedOrigin: getenv("CORS_ALLOWED_ORIGIN", "*"), - EnterpriseAuthURL: getenv("ENTERPRISE_AUTH_URL", ""), + SearchGRPCAddr: getenv("SEARCH_GRPC_ADDR", "localhost:50052"), + CORSAllowedOrigin: getenv("CORS_ALLOWED_ORIGIN", "*"), + EnterpriseAuthURL: getenv("ENTERPRISE_AUTH_URL", ""), + TenantCRDNamespace: getenv("TENANT_CRD_NAMESPACE", ""), } timeoutSec, err := strconv.Atoi(getenv("QUERY_TIMEOUT_SECONDS", "30")) diff --git a/enterprise/internal/tenantcrd/tenantcrd.go b/enterprise/internal/tenantcrd/tenantcrd.go new file mode 100644 index 0000000..abb7903 --- /dev/null +++ b/enterprise/internal/tenantcrd/tenantcrd.go @@ -0,0 +1,229 @@ +// Package tenantcrd syncs enterprise-api -provision-tenant's real +// provisioning result into the Tenant CRD (deploy/operator/api/ +// v1alpha1) -- the "lightweight unification" named in CLAUDE.md's "two +// independent provisioning mechanisms" gap: -provision-tenant stays the +// sole real actor (rbacstore + tenantprovision.ProvisionClickHouse, see +// runProvisionTenant's doc comment in cmd/enterprise-api/main.go); this +// package's only job is making that result observable via `kubectl get +// tenants` too, for a deployment that also runs deploy/operator's +// tenant-operator. deploy/operator/internal/controller.TenantReconciler +// derives Phase/Conditions from what this package writes -- it never +// invents "provisioned" on its own. +// +// Deliberately uses the K8s dynamic client (unstructured.Unstructured + +// a GroupVersionResource) rather than importing deploy/operator/api/ +// v1alpha1's typed Tenant struct: deploy/operator is a separate Go +// module (its own go.mod), and adding a cross-module `replace` directive +// between two independently-versioned modules is exactly the kind of +// coupling this codebase has otherwise avoided (enterprise/ already only +// imports api/, never deploy/operator/). The typed Secret API +// (k8s.io/client-go/kubernetes) is used for the credential Secret +// itself, since Secret is a stable, well-known built-in type with no +// such module-boundary concern. +package tenantcrd + +import ( + "context" + "fmt" + + corev1 "k8s.io/api/core/v1" + apierrors "k8s.io/apimachinery/pkg/api/errors" + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + "k8s.io/apimachinery/pkg/apis/meta/v1/unstructured" + "k8s.io/apimachinery/pkg/runtime/schema" + "k8s.io/apimachinery/pkg/types" + "k8s.io/client-go/dynamic" + "k8s.io/client-go/kubernetes" + "k8s.io/client-go/rest" + "k8s.io/client-go/tools/clientcmd" +) + +// tenantGVR identifies deploy/operator/config/crd/sentry.io_tenants.yaml's +// resource -- kept as a plain schema.GroupVersionResource (not the typed +// package) for the reason this file's doc comment explains. +var tenantGVR = schema.GroupVersionResource{Group: "sentry.io", Version: "v1alpha1", Resource: "tenants"} + +// Syncer talks to the K8s API. Construction (New) is the only place +// that can fail for "no cluster reachable" reasons -- Sync itself +// assumes a working client. +type Syncer struct { + dynamic dynamic.Interface + clientset kubernetes.Interface + namespace string +} + +// New builds a Syncer for namespace, trying in-cluster config first (the +// real deployment shape: -provision-tenant runs via `kubectl exec` into +// the already-running enterprise-api Pod, which carries its own +// ServiceAccount) and falling back to KUBECONFIG/the default kubeconfig +// path for local/dev convenience. Returns an error if neither is +// reachable -- callers decide whether that's fatal (see +// cmd/enterprise-api/main.go's runProvisionTenant: it is, once CR sync +// has been explicitly requested via TENANT_CRD_NAMESPACE, since a +// silent skip would leave the CRD stale/wrong again, the exact bug this +// package exists to fix). +func New(namespace string) (*Syncer, error) { + config, err := rest.InClusterConfig() + if err != nil { + config, err = clientcmd.NewNonInteractiveDeferredLoadingClientConfig( + clientcmd.NewDefaultClientConfigLoadingRules(), + &clientcmd.ConfigOverrides{}, + ).ClientConfig() + if err != nil { + return nil, fmt.Errorf("tenantcrd: no in-cluster config and no usable kubeconfig: %w", err) + } + } + dyn, err := dynamic.NewForConfig(config) + if err != nil { + return nil, fmt.Errorf("tenantcrd: building dynamic client: %w", err) + } + clientset, err := kubernetes.NewForConfig(config) + if err != nil { + return nil, fmt.Errorf("tenantcrd: building clientset: %w", err) + } + return &Syncer{dynamic: dyn, clientset: clientset, namespace: namespace}, nil +} + +// newForTest builds a Syncer directly from fake clients -- production +// code always goes through New, but tests need to inject +// k8s.io/client-go/dynamic/fake and kubernetes/fake without a real or +// in-cluster API server. +func newForTest(dyn dynamic.Interface, clientset kubernetes.Interface, namespace string) *Syncer { + return &Syncer{dynamic: dyn, clientset: clientset, namespace: namespace} +} + +// Credentials is the minimal shape Sync needs -- deliberately not +// tenantprovision.Credentials itself, matching chrunner.DataSource's +// "narrow, not the storage type" precedent. +type Credentials struct { + Username string + Password string +} + +// SecretName is deterministic from the tenant ID -- never randomly +// suffixed -- so a re-run of Sync (idempotent retry, see +// runProvisionTenant) finds and updates the same Secret rather than +// creating a second one. +func SecretName(tenantID string) string { + return fmt.Sprintf("sentry-tenant-%s-clickhouse", tenantID) +} + +// Sync upserts the Tenant object (creating it with spec.displayName if +// absent) and its credential Secret, then patches status to reflect +// real, already-confirmed provisioning -- called only after +// tenantprovision.ProvisionClickHouse has actually succeeded, never +// before. Safe to call repeatedly for the same tenant (idempotent): +// re-running overwrites the Secret with the same credentials rbacstore +// already has on file (never rotates to a *new*, different credential -- +// that would break every open connection for no benefit, same +// reasoning the operator's now-removed reconcileSecret documented) and +// re-applies the same status fields. +func (s *Syncer) Sync(ctx context.Context, tenantID, displayName, tantivyIndexPath string, creds Credentials) error { + uid, err := s.upsertTenant(ctx, tenantID, displayName) + if err != nil { + return fmt.Errorf("tenantcrd: upserting tenant object: %w", err) + } + secretName := SecretName(tenantID) + if err := s.upsertSecret(ctx, tenantID, secretName, uid, creds); err != nil { + return fmt.Errorf("tenantcrd: upserting credential secret: %w", err) + } + if err := s.patchStatus(ctx, tenantID, secretName, tantivyIndexPath); err != nil { + return fmt.Errorf("tenantcrd: patching tenant status: %w", err) + } + return nil +} + +func (s *Syncer) upsertTenant(ctx context.Context, tenantID, displayName string) (types.UID, error) { + client := s.dynamic.Resource(tenantGVR).Namespace(s.namespace) + + existing, err := client.Get(ctx, tenantID, metav1.GetOptions{}) + if err == nil { + return existing.GetUID(), nil + } + if !apierrors.IsNotFound(err) { + return "", fmt.Errorf("getting existing tenant object: %w", err) + } + + obj := &unstructured.Unstructured{Object: map[string]interface{}{ + "apiVersion": "sentry.io/v1alpha1", + "kind": "Tenant", + "metadata": map[string]interface{}{ + "name": tenantID, + "namespace": s.namespace, + }, + "spec": map[string]interface{}{ + "displayName": displayName, + }, + }} + created, err := client.Create(ctx, obj, metav1.CreateOptions{}) + if err != nil { + return "", fmt.Errorf("creating tenant object: %w", err) + } + return created.GetUID(), nil +} + +func (s *Syncer) upsertSecret(ctx context.Context, tenantID, secretName string, tenantUID types.UID, creds Credentials) error { + secrets := s.clientset.CoreV1().Secrets(s.namespace) + controllerTrue := true + secret := &corev1.Secret{ + ObjectMeta: metav1.ObjectMeta{ + Name: secretName, + Namespace: s.namespace, + Labels: map[string]string{ + "app.kubernetes.io/managed-by": "sentry-enterprise-api", + "sentry.io/tenant": tenantID, + }, + // Owned by the Tenant object even though a different + // process (this one, not the operator's controller) + // created it -- K8s's garbage collector honors + // OwnerReferences regardless of which actor set them, so + // deleting the Tenant still cleans this Secret up. + OwnerReferences: []metav1.OwnerReference{{ + APIVersion: "sentry.io/v1alpha1", + Kind: "Tenant", + Name: tenantID, + UID: tenantUID, + Controller: &controllerTrue, + BlockOwnerDeletion: &controllerTrue, + }}, + }, + Type: corev1.SecretTypeOpaque, + StringData: map[string]string{ + "username": creds.Username, + "password": creds.Password, + "database": tenantID, + }, + } + + existing, err := secrets.Get(ctx, secretName, metav1.GetOptions{}) + switch { + case err == nil: + secret.ResourceVersion = existing.ResourceVersion + _, err := secrets.Update(ctx, secret, metav1.UpdateOptions{}) + return err + case apierrors.IsNotFound(err): + _, err := secrets.Create(ctx, secret, metav1.CreateOptions{}) + return err + default: + return fmt.Errorf("getting existing secret: %w", err) + } +} + +func (s *Syncer) patchStatus(ctx context.Context, tenantID, secretName, tantivyIndexPath string) error { + client := s.dynamic.Resource(tenantGVR).Namespace(s.namespace) + existing, err := client.Get(ctx, tenantID, metav1.GetOptions{}) + if err != nil { + return fmt.Errorf("getting tenant object for status update: %w", err) + } + if err := unstructured.SetNestedField(existing.Object, tenantID, "status", "clickHouseDatabaseName"); err != nil { + return err + } + if err := unstructured.SetNestedField(existing.Object, secretName, "status", "clickHouseSecretRef"); err != nil { + return err + } + if err := unstructured.SetNestedField(existing.Object, tantivyIndexPath, "status", "tantivyIndexPath"); err != nil { + return err + } + _, err = client.UpdateStatus(ctx, existing, metav1.UpdateOptions{}) + return err +} diff --git a/enterprise/internal/tenantcrd/tenantcrd_test.go b/enterprise/internal/tenantcrd/tenantcrd_test.go new file mode 100644 index 0000000..daba990 --- /dev/null +++ b/enterprise/internal/tenantcrd/tenantcrd_test.go @@ -0,0 +1,158 @@ +// Tests use k8s.io/client-go's fake dynamic and typed clientsets, not a +// real or in-cluster API server -- genuinely runnable in an environment +// with no Kubernetes access at all, same "real client library, fake +// transport" shape as enterprise/internal/searchclient's in-process gRPC +// tests. What a fake client can't exercise: real admission/defaulting, +// or that deploy/operator's actual CRD schema accepts what this package +// writes (see /docs/phase-4-runbook.md's verification-status notes for +// the Helm/kubeconform-based schema check that covers that instead). +package tenantcrd + +import ( + "context" + "testing" + + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + "k8s.io/apimachinery/pkg/apis/meta/v1/unstructured" + "k8s.io/apimachinery/pkg/runtime" + "k8s.io/apimachinery/pkg/runtime/schema" + dynamicfake "k8s.io/client-go/dynamic/fake" + k8sfake "k8s.io/client-go/kubernetes/fake" +) + +func newTestSyncer(t *testing.T, namespace string) *Syncer { + t.Helper() + scheme := runtime.NewScheme() + listKinds := map[schema.GroupVersionResource]string{tenantGVR: "TenantList"} + dyn := dynamicfake.NewSimpleDynamicClientWithCustomListKinds(scheme, listKinds) + clientset := k8sfake.NewSimpleClientset() + return newForTest(dyn, clientset, namespace) +} + +func getTenant(t *testing.T, s *Syncer, tenantID string) *unstructured.Unstructured { + t.Helper() + obj, err := s.dynamic.Resource(tenantGVR).Namespace(s.namespace).Get(context.Background(), tenantID, metav1.GetOptions{}) + if err != nil { + t.Fatalf("getting tenant object: %v", err) + } + return obj +} + +func TestSyncCreatesTenantObjectWithDisplayName(t *testing.T) { + s := newTestSyncer(t, "sentry") + err := s.Sync(context.Background(), "acme", "Acme Corp", "/var/lib/sentry-search/tenants/acme", Credentials{Username: "tenant_acme", Password: "secret-pw"}) + if err != nil { + t.Fatalf("Sync: %v", err) + } + + obj := getTenant(t, s, "acme") + displayName, _, _ := unstructured.NestedString(obj.Object, "spec", "displayName") + if displayName != "Acme Corp" { + t.Fatalf("spec.displayName = %q, want %q", displayName, "Acme Corp") + } +} + +func TestSyncSetsRealStatusFields(t *testing.T) { + s := newTestSyncer(t, "sentry") + err := s.Sync(context.Background(), "acme", "Acme Corp", "/var/lib/sentry-search/tenants/acme", Credentials{Username: "tenant_acme", Password: "secret-pw"}) + if err != nil { + t.Fatalf("Sync: %v", err) + } + + obj := getTenant(t, s, "acme") + dbName, _, _ := unstructured.NestedString(obj.Object, "status", "clickHouseDatabaseName") + if dbName != "acme" { + t.Fatalf("status.clickHouseDatabaseName = %q, want acme", dbName) + } + secretRef, _, _ := unstructured.NestedString(obj.Object, "status", "clickHouseSecretRef") + if secretRef != "sentry-tenant-acme-clickhouse" { + t.Fatalf("status.clickHouseSecretRef = %q, want sentry-tenant-acme-clickhouse", secretRef) + } + indexPath, _, _ := unstructured.NestedString(obj.Object, "status", "tantivyIndexPath") + if indexPath != "/var/lib/sentry-search/tenants/acme" { + t.Fatalf("status.tantivyIndexPath = %q, want /var/lib/sentry-search/tenants/acme", indexPath) + } +} + +func TestSyncCreatesSecretOwnedByTenant(t *testing.T) { + s := newTestSyncer(t, "sentry") + err := s.Sync(context.Background(), "acme", "Acme Corp", "/idx", Credentials{Username: "tenant_acme", Password: "secret-pw"}) + if err != nil { + t.Fatalf("Sync: %v", err) + } + + secret, err := s.clientset.CoreV1().Secrets("sentry").Get(context.Background(), "sentry-tenant-acme-clickhouse", metav1.GetOptions{}) + if err != nil { + t.Fatalf("getting secret: %v", err) + } + if secret.StringData["username"] != "tenant_acme" || secret.StringData["password"] != "secret-pw" || secret.StringData["database"] != "acme" { + t.Fatalf("unexpected secret data: %+v", secret.StringData) + } + if len(secret.OwnerReferences) != 1 || secret.OwnerReferences[0].Name != "acme" || secret.OwnerReferences[0].Kind != "Tenant" { + t.Fatalf("expected the secret to be owned by the Tenant object, got %+v", secret.OwnerReferences) + } +} + +// TestSyncIsIdempotentAndNeverChangesCredentials is the regression test +// for the "safe to retry" property runProvisionTenant's idempotent +// CR-sync-only retry path depends on (see cmd/enterprise-api/main.go): +// running Sync twice for the same tenant must not create a duplicate +// Tenant object, must not error, and must never silently swap in +// different credentials than what was passed. +func TestSyncIsIdempotentAndNeverChangesCredentials(t *testing.T) { + s := newTestSyncer(t, "sentry") + ctx := context.Background() + creds := Credentials{Username: "tenant_acme", Password: "secret-pw"} + + if err := s.Sync(ctx, "acme", "Acme Corp", "/idx", creds); err != nil { + t.Fatalf("first Sync: %v", err) + } + if err := s.Sync(ctx, "acme", "Acme Corp", "/idx", creds); err != nil { + t.Fatalf("second Sync: %v", err) + } + + list, err := s.dynamic.Resource(tenantGVR).Namespace("sentry").List(ctx, metav1.ListOptions{}) + if err != nil { + t.Fatalf("listing tenants: %v", err) + } + if len(list.Items) != 1 { + t.Fatalf("expected exactly one Tenant object after two Syncs, got %d", len(list.Items)) + } + + secret, err := s.clientset.CoreV1().Secrets("sentry").Get(ctx, "sentry-tenant-acme-clickhouse", metav1.GetOptions{}) + if err != nil { + t.Fatalf("getting secret: %v", err) + } + if secret.StringData["password"] != "secret-pw" { + t.Fatalf("password changed across a re-sync: %q", secret.StringData["password"]) + } +} + +func TestSyncPreservesExistingTenantObjectDisplayName(t *testing.T) { + s := newTestSyncer(t, "sentry") + ctx := context.Background() + + // A human/GitOps process already created this Tenant object (e.g. + // via `kubectl apply`, per TenantSpec's doc comment) before + // -provision-tenant ever ran -- Sync must not overwrite their + // chosen displayName with its own. + pre := &unstructured.Unstructured{Object: map[string]interface{}{ + "apiVersion": "sentry.io/v1alpha1", + "kind": "Tenant", + "metadata": map[string]interface{}{"name": "acme", "namespace": "sentry"}, + "spec": map[string]interface{}{"displayName": "Human-Chosen Name"}, + }} + if _, err := s.dynamic.Resource(tenantGVR).Namespace("sentry").Create(ctx, pre, metav1.CreateOptions{}); err != nil { + t.Fatalf("pre-creating tenant: %v", err) + } + + if err := s.Sync(ctx, "acme", "Some Other Name -provision-tenant Was Called With", "/idx", Credentials{Username: "u", Password: "p"}); err != nil { + t.Fatalf("Sync: %v", err) + } + + obj := getTenant(t, s, "acme") + displayName, _, _ := unstructured.NestedString(obj.Object, "spec", "displayName") + if displayName != "Human-Chosen Name" { + t.Fatalf("spec.displayName = %q, want the pre-existing value preserved", displayName) + } +}