Files
cairnobs/deploy/operator/internal/controller/tenant_controller_test.go
T
jcoffey-dev 823f5d48d1 Unify the Tenant CRD with enterprise-api -provision-tenant (lightweight)
Closes a gap named across CLAUDE.md/docs/architecture.md/deploy/README.md
since early Phase 4: the operator's Tenant CRD and -provision-tenant
were two disconnected mechanisms. The operator's reconciler generated a
K8s Secret with a locally-generated random password that authenticated
against nothing (nothing ever called ClickHouse to create a matching
user), and unconditionally claimed status.phase=Active the moment a
Tenant object existed -- actively misleading, not just incomplete.

Two unification shapes were considered (surfaced to the user via
AskUserQuestion, given the real difference in blast radius): the
operator's reconcile loop becoming a second real actor (new Postgres +
ClickHouse admin credentials flowing into the K8s controller, plus real
reconcile-loop idempotency/retry design for an inherently one-shot
external side effect), or keeping -provision-tenant as the sole real
actor and having it also sync its result into the CRD. Went with the
lighter option.

enterprise/internal/tenantcrd (new): a Syncer using the K8s dynamic
client (unstructured.Unstructured + a GroupVersionResource, not
deploy/operator's typed Tenant struct -- avoids a cross-module Go
dependency between two independently-versioned modules for one type).
Upserts the Tenant object, creates/updates a Secret with the *real*
ClickHouse credentials owned by that Tenant via an OwnerReference, then
patches status.{clickHouseDatabaseName,clickHouseSecretRef,
tantivyIndexPath}. Idempotent and safe to retry: never rotates a
credential across a re-sync, never overwrites a pre-existing
spec.displayName a human/GitOps process set.

cmd/enterprise-api/main.go's runProvisionTenant calls Sync when
TENANT_CRD_NAMESPACE is set (empty = no-op, same shape as every other
optional dependency in this codebase). Its "already active" refusal is
now split: ClickHouse re-provisioning is still refused (rotating a live
credential would break every open connection for no benefit), but CR
sync alone is now retryable using the credentials already on file in
rbacstore -- needed for retrying a previously-failed sync, or
backfilling CR sync for a tenant provisioned before this existed.

deploy/operator's reconciler rewritten to match: it never claims
PhaseActive on its own initiative anymore, only once
status.ClickHouseDatabaseName is non-empty (the field -provision-tenant,
and only -provision-tenant, sets). Phase is now a pure function of
{spec.suspended, status.ClickHouseDatabaseName != ""} recomputed every
reconcile, not toggled in place -- fixes a related bug the old code
would have hit once suspension was involved: un-suspending an
already-provisioned tenant needs to return straight to Active, which
isn't derivable from "last observed phase was Suspended" alone. The
reconciler no longer creates or manages any Secret, dropped its
`secrets` RBAC grant entirely, and gained zero new dependencies.

Helm chart: enterprise-api gets its own ServiceAccount/Role/RoleBinding
(get/list/create tenants, get/update/patch tenants/status, get/create/
update secrets -- least-privilege, scoped to the release namespace, not
a ClusterRole) and a TENANT_CRD_NAMESPACE env var, both gated on
tenantOperator.enabled. tenant-operator's ClusterRole loses the
secrets grant it no longer needs.

Verified in this environment: enterprise/internal/tenantcrd's tests run
against k8s.io/client-go's fake dynamic + typed clientsets (real client
library, fake transport, no cluster needed); deploy/operator's rewritten
tenant_controller_test.go runs against controller-runtime's fake
client, including new regression tests for the "must not claim Active
without confirmation" and "un-suspending returns to Active, not
Provisioning" properties; helm template + parsing the rendered YAML
confirms the RBAC split renders exactly as designed under both
tenantOperator.enabled=true/false. Not verified: an actual
-provision-tenant run against a real cluster with the operator watching
(no live cluster in this environment, same disclosed limitation as the
rest of /deploy). Docs updated in lockstep: CLAUDE.md, docs/architecture.md,
deploy/README.md, deploy/helm/sentry/README.md (including a corrected
"Trying the two-tenant example" walkthrough), phase-4-runbook.md (new
§11), enterprise/README.md. Also fixed two unrelated stale claims found
along the way: docs/architecture.md still said docker-compose.yml ran
plain api unconditionally (fixed in an earlier commit, doc not updated
then), and enterprise-api's own main.go doc comment still said Helm/
docker-compose wiring wasn't built yet.
2026-08-14 09:07:10 -07:00

178 lines
6.9 KiB
Go

// Tests use controller-runtime's fake client (sigs.k8s.io/
// controller-runtime/pkg/client/fake), not envtest -- envtest needs a
// real kube-apiserver/etcd binary pair (setup-envtest) that isn't
// available in this environment (see package doc comment and
// deploy/README.md's verification section). A fake client exercises
// Reconcile's actual logic (status derivation, condition writes) against
// an in-memory tracker; what it can't exercise is anything a real
// apiserver would do for you (defaulting, admission, watch-triggered
// re-reconciliation) -- so passing here is real signal about this
// reconciler's logic, not proof it behaves correctly against a live
// cluster.
package controller
import (
"context"
"testing"
corev1 "k8s.io/api/core/v1"
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
"k8s.io/apimachinery/pkg/runtime"
"k8s.io/apimachinery/pkg/types"
ctrl "sigs.k8s.io/controller-runtime"
"sigs.k8s.io/controller-runtime/pkg/client"
"sigs.k8s.io/controller-runtime/pkg/client/fake"
sentryv1alpha1 "github.com/sentry/sentry/deploy/operator/api/v1alpha1"
)
func newFakeReconciler(t *testing.T, objs ...client.Object) *TenantReconciler {
t.Helper()
scheme := runtime.NewScheme()
if err := corev1.AddToScheme(scheme); err != nil {
t.Fatalf("adding corev1 to scheme: %v", err)
}
if err := sentryv1alpha1.AddToScheme(scheme); err != nil {
t.Fatalf("adding sentryv1alpha1 to scheme: %v", err)
}
fakeClient := fake.NewClientBuilder().
WithScheme(scheme).
WithObjects(objs...).
WithStatusSubresource(&sentryv1alpha1.Tenant{}).
Build()
return &TenantReconciler{Client: fakeClient, Scheme: scheme}
}
func testTenant(name string, suspended bool) *sentryv1alpha1.Tenant {
return &sentryv1alpha1.Tenant{
ObjectMeta: metav1.ObjectMeta{Name: name, Namespace: "default"},
Spec: sentryv1alpha1.TenantSpec{DisplayName: name, Suspended: suspended},
}
}
func reconcile(t *testing.T, r *TenantReconciler, name string) sentryv1alpha1.Tenant {
t.Helper()
ctx := context.Background()
if _, err := r.Reconcile(ctx, ctrl.Request{NamespacedName: types.NamespacedName{Name: name, Namespace: "default"}}); err != nil {
t.Fatalf("Reconcile: %v", err)
}
var got sentryv1alpha1.Tenant
if err := r.Get(ctx, types.NamespacedName{Name: name, Namespace: "default"}, &got); err != nil {
t.Fatalf("getting tenant: %v", err)
}
return got
}
// TestReconcileUnprovisionedTenantIsProvisioningNotActive is the
// regression test for the pre-unification bug: this controller must
// never claim PhaseActive just because a Tenant object exists -- only
// enterprise-api -provision-tenant setting Status.ClickHouseDatabaseName
// (real ClickHouse provisioning having actually succeeded) earns that.
func TestReconcileUnprovisionedTenantIsProvisioningNotActive(t *testing.T) {
tenant := testTenant("acme", false)
r := newFakeReconciler(t, tenant)
got := reconcile(t, r, "acme")
if got.Status.Phase != sentryv1alpha1.PhaseProvisioning {
t.Fatalf("Phase = %q, want Provisioning (nothing has provisioned this tenant yet)", got.Status.Phase)
}
cond := readyCondition(got)
if cond == nil || cond.Status != metav1.ConditionFalse {
t.Fatalf("Ready condition = %+v, want status False", cond)
}
}
// TestReconcileReflectsProvisioningStateProvisionTenantSets is the core
// "lightweight unification" behavior: once something external (in
// production, -provision-tenant; here, simulated directly against the
// fake client the way a real K8s API server write would land) sets
// ClickHouseDatabaseName, this controller must report PhaseActive.
func TestReconcileReflectsProvisioningStateProvisionTenantSets(t *testing.T) {
tenant := testTenant("acme", false)
r := newFakeReconciler(t, tenant)
ctx := context.Background()
var toUpdate sentryv1alpha1.Tenant
if err := r.Get(ctx, types.NamespacedName{Name: "acme", Namespace: "default"}, &toUpdate); err != nil {
t.Fatalf("getting tenant: %v", err)
}
toUpdate.Status.ClickHouseDatabaseName = "acme"
toUpdate.Status.ClickHouseSecretRef = "sentry-tenant-acme-clickhouse"
toUpdate.Status.TantivyIndexPath = "/var/lib/sentry-search/tenants/acme"
if err := r.Status().Update(ctx, &toUpdate); err != nil {
t.Fatalf("simulating -provision-tenant's status write: %v", err)
}
got := reconcile(t, r, "acme")
if got.Status.Phase != sentryv1alpha1.PhaseActive {
t.Fatalf("Phase = %q, want Active", got.Status.Phase)
}
if got.Status.ClickHouseDatabaseName != "acme" || got.Status.ClickHouseSecretRef != "sentry-tenant-acme-clickhouse" || got.Status.TantivyIndexPath != "/var/lib/sentry-search/tenants/acme" {
t.Fatalf("reconcile must not clobber the fields -provision-tenant set: %+v", got.Status)
}
cond := readyCondition(got)
if cond == nil || cond.Status != metav1.ConditionTrue {
t.Fatalf("Ready condition = %+v, want status True", cond)
}
}
func TestReconcileSuspendedOverridesProvisionedState(t *testing.T) {
tenant := testTenant("acme", true)
tenant.Status.ClickHouseDatabaseName = "acme" // already provisioned
r := newFakeReconciler(t, tenant)
got := reconcile(t, r, "acme")
if got.Status.Phase != sentryv1alpha1.PhaseSuspended {
t.Fatalf("Phase = %q, want Suspended even though the tenant is provisioned", got.Status.Phase)
}
}
// TestReconcileUnsuspendingReturnsToActiveNotProvisioning is the
// regression test for why Phase is *derived* fresh every reconcile
// (from Spec.Suspended + whether ClickHouseDatabaseName is set) rather
// than toggled in place: un-suspending an already-provisioned tenant
// must return it straight to Active, not demote it to Provisioning just
// because the last observed Phase happened to be Suspended.
func TestReconcileUnsuspendingReturnsToActiveNotProvisioning(t *testing.T) {
tenant := testTenant("acme", true)
tenant.Status.ClickHouseDatabaseName = "acme"
r := newFakeReconciler(t, tenant)
ctx := context.Background()
_ = reconcile(t, r, "acme") // establishes Suspended
var toUpdate sentryv1alpha1.Tenant
if err := r.Get(ctx, types.NamespacedName{Name: "acme", Namespace: "default"}, &toUpdate); err != nil {
t.Fatalf("getting tenant: %v", err)
}
toUpdate.Spec.Suspended = false
if err := r.Update(ctx, &toUpdate); err != nil {
t.Fatalf("unsuspending: %v", err)
}
got := reconcile(t, r, "acme")
if got.Status.Phase != sentryv1alpha1.PhaseActive {
t.Fatalf("Phase = %q, want Active after unsuspending an already-provisioned tenant", got.Status.Phase)
}
}
func TestReconcileMissingTenantIsNoOp(t *testing.T) {
r := newFakeReconciler(t)
_, err := r.Reconcile(context.Background(), ctrl.Request{NamespacedName: types.NamespacedName{Name: "does-not-exist", Namespace: "default"}})
if err != nil {
t.Fatalf("Reconcile on a missing tenant should be a no-op, got error: %v", err)
}
}
func readyCondition(tenant sentryv1alpha1.Tenant) *metav1.Condition {
for i := range tenant.Status.Conditions {
if tenant.Status.Conditions[i].Type == sentryv1alpha1.ConditionReady {
return &tenant.Status.Conditions[i]
}
}
return nil
}