coffey-labs/cairnobs
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Unify the Tenant CRD with enterprise-api -provision-tenant (lightweight)

Browse Source 
Closes a gap named across CLAUDE.md/docs/architecture.md/deploy/README.md
since early Phase 4: the operator's Tenant CRD and -provision-tenant
were two disconnected mechanisms. The operator's reconciler generated a
K8s Secret with a locally-generated random password that authenticated
against nothing (nothing ever called ClickHouse to create a matching
user), and unconditionally claimed status.phase=Active the moment a
Tenant object existed -- actively misleading, not just incomplete.

Two unification shapes were considered (surfaced to the user via
AskUserQuestion, given the real difference in blast radius): the
operator's reconcile loop becoming a second real actor (new Postgres +
ClickHouse admin credentials flowing into the K8s controller, plus real
reconcile-loop idempotency/retry design for an inherently one-shot
external side effect), or keeping -provision-tenant as the sole real
actor and having it also sync its result into the CRD. Went with the
lighter option.

enterprise/internal/tenantcrd (new): a Syncer using the K8s dynamic
client (unstructured.Unstructured + a GroupVersionResource, not
deploy/operator's typed Tenant struct -- avoids a cross-module Go
dependency between two independently-versioned modules for one type).
Upserts the Tenant object, creates/updates a Secret with the *real*
ClickHouse credentials owned by that Tenant via an OwnerReference, then
patches status.{clickHouseDatabaseName,clickHouseSecretRef,
tantivyIndexPath}. Idempotent and safe to retry: never rotates a
credential across a re-sync, never overwrites a pre-existing
spec.displayName a human/GitOps process set.

cmd/enterprise-api/main.go's runProvisionTenant calls Sync when
TENANT_CRD_NAMESPACE is set (empty = no-op, same shape as every other
optional dependency in this codebase). Its "already active" refusal is
now split: ClickHouse re-provisioning is still refused (rotating a live
credential would break every open connection for no benefit), but CR
sync alone is now retryable using the credentials already on file in
rbacstore -- needed for retrying a previously-failed sync, or
backfilling CR sync for a tenant provisioned before this existed.

deploy/operator's reconciler rewritten to match: it never claims
PhaseActive on its own initiative anymore, only once
status.ClickHouseDatabaseName is non-empty (the field -provision-tenant,
and only -provision-tenant, sets). Phase is now a pure function of
{spec.suspended, status.ClickHouseDatabaseName != ""} recomputed every
reconcile, not toggled in place -- fixes a related bug the old code
would have hit once suspension was involved: un-suspending an
already-provisioned tenant needs to return straight to Active, which
isn't derivable from "last observed phase was Suspended" alone. The
reconciler no longer creates or manages any Secret, dropped its
`secrets` RBAC grant entirely, and gained zero new dependencies.

Helm chart: enterprise-api gets its own ServiceAccount/Role/RoleBinding
(get/list/create tenants, get/update/patch tenants/status, get/create/
update secrets -- least-privilege, scoped to the release namespace, not
a ClusterRole) and a TENANT_CRD_NAMESPACE env var, both gated on
tenantOperator.enabled. tenant-operator's ClusterRole loses the
secrets grant it no longer needs.

Verified in this environment: enterprise/internal/tenantcrd's tests run
against k8s.io/client-go's fake dynamic + typed clientsets (real client
library, fake transport, no cluster needed); deploy/operator's rewritten
tenant_controller_test.go runs against controller-runtime's fake
client, including new regression tests for the "must not claim Active
without confirmation" and "un-suspending returns to Active, not
Provisioning" properties; helm template + parsing the rendered YAML
confirms the RBAC split renders exactly as designed under both
tenantOperator.enabled=true/false. Not verified: an actual
-provision-tenant run against a real cluster with the operator watching
(no live cluster in this environment, same disclosed limitation as the
rest of /deploy). Docs updated in lockstep: CLAUDE.md, docs/architecture.md,
deploy/README.md, deploy/helm/sentry/README.md (including a corrected
"Trying the two-tenant example" walkthrough), phase-4-runbook.md (new
§11), enterprise/README.md. Also fixed two unrelated stale claims found
along the way: docs/architecture.md still said docker-compose.yml ran
plain api unconditionally (fixed in an earlier commit, doc not updated
then), and enterprise-api's own main.go doc comment still said Helm/
docker-compose wiring wasn't built yet.
This commit is contained in:
jcoffey-dev
2026-08-14 09:07:10 -07:00
parent 8d7326fc6a
commit 823f5d48d1
18 changed files with 1073 additions and 323 deletions
Download Patch File Download Diff File Expand all files Collapse all files
deploy/operator/internal/controller/tenant_controller.go
+52 -131
View File
@@ -1,42 +1,39 @@
// Package controller reconciles the Tenant CRD (api/v1alpha1) into the
// K8s-native artifacts task 2/CLAUDE.md's Phase 4 exit criteria calls
// for: "real per-tenant secret management (replacing today's single
// shared CLICKHOUSE_PASSWORD)" -- see docker-compose.yml's
// CLICKHOUSE_PASSWORD comment for what that shared-secret shape looks
// like today.
// Package controller reconciles the Tenant CRD (api/v1alpha1) -- see
// TenantPhase's doc comment for the "lightweight unification" this
// controller is one half of. This reconciler never calls ClickHouse (no
// CREATE DATABASE/CREATE USER/GRANT), never touches the Tantivy index
// filesystem, and never talks to enterprise/internal/rbacstore -- those
// stay enterprise-api -provision-tenant's job (enterprise/internal/
// tenantprovision, enterprise/internal/tenantcrd). This controller's
// job is purely a function of what's already on the object: derive
// Phase and the Ready condition from Spec.Suspended and whether
// Status.ClickHouseDatabaseName has been set by -provision-tenant,
// nothing more. A Tenant reaching PhaseActive here is exactly the same
// claim as rbacstore's tenants.status='active' now, not a
// second, independently-computed one -- see docs/phase-4-isolation-design.md.
//
// What this reconciler does NOT do, named explicitly rather than
// implied: it never calls ClickHouse (no CREATE DATABASE/CREATE USER/
// GRANT), never touches the Tantivy index filesystem, and never talks to
// enterprise/internal/rbacstore. Those are enterprise/internal/
// tenantprovision's job -- unbuilt, per the task 5 summary. This
// controller's job stops at "does a K8s Secret with this tenant's
// ClickHouse credentials exist, and does the Tenant's status reflect
// that" -- the deployment-topology half of tenant provisioning, not the
// database-side half. A Tenant reaching PhaseActive here is NOT the same
// claim as rbacstore's tenants.status='active' (the actual gate every
// tenant-resolution code path checks per
// /docs/phase-4-isolation-design.md) -- reconciling those two into one
// state machine is exactly the kind of follow-up work
// /docs/phase-4-runbook.md's task 6 section names as deferred.
// Earlier versions of this controller also generated a per-tenant
// ClickHouse credential Secret with a locally-generated random
// password. That Secret authenticated against nothing (nothing in this
// controller ever called ClickHouse to create a matching user) and
// nothing else in the codebase ever read it -- a placeholder that
// actively misled ("looks provisioned") rather than one that honestly
// represented "not yet provisioned." Removed rather than fixed in
// place: the real Secret, with real credentials, is now created by
// -provision-tenant (enterprise/internal/tenantcrd) once ClickHouse
// provisioning actually succeeds.
package controller
import (
"context"
"crypto/rand"
"encoding/base64"
"fmt"
corev1 "k8s.io/api/core/v1"
apierrors "k8s.io/apimachinery/pkg/api/errors"
"k8s.io/apimachinery/pkg/api/meta"
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
"k8s.io/apimachinery/pkg/runtime"
"k8s.io/apimachinery/pkg/types"
ctrl "sigs.k8s.io/controller-runtime"
"sigs.k8s.io/controller-runtime/pkg/client"
"sigs.k8s.io/controller-runtime/pkg/controller/controllerutil"
"sigs.k8s.io/controller-runtime/pkg/log"
sentryv1alpha1 "github.com/sentry/sentry/deploy/operator/api/v1alpha1"
)
@@ -47,77 +44,52 @@ type TenantReconciler struct {
Scheme *runtime.Scheme
}
// clickHouseSecretName is deterministic from the tenant name -- never
// randomly suffixed -- so a re-run of Reconcile (or a controller
// restart) finds the same Secret it created before, rather than losing
// track of it and creating a second one.
func clickHouseSecretName(tenant *sentryv1alpha1.Tenant) string {
return fmt.Sprintf("sentry-tenant-%s-clickhouse", tenant.Name)
}
// tantivyIndexPath mirrors /docs/phase-4-isolation-design.md's Tantivy
// section: one directory per tenant under the shared search-index
// volume (search-index-data in docker-compose.yml; a PVC in the Helm
// chart -- see deploy/helm/sentry/templates/search-deployment.yaml).
func tantivyIndexPath(tenant *sentryv1alpha1.Tenant) string {
return "/var/lib/sentry-search/tenants/" + tenant.Name
}
// generatePassword returns a 32-byte random value, base64-encoded --
// same "narrowly-granted, per-tenant, never the shared default user"
// framing as /docs/phase-4-isolation-design.md's ClickHouse section,
// applied to how the credential itself is generated (crypto/rand, not
// math/rand -- this becomes a real ClickHouse user's password once
// internal/tenantprovision consumes it).
func generatePassword() (string, error) {
buf := make([]byte, 32)
if _, err := rand.Read(buf); err != nil {
return "", fmt.Errorf("generating password: %w", err)
}
return base64.RawURLEncoding.EncodeToString(buf), nil
}
// +kubebuilder:rbac:groups=sentry.io,resources=tenants,verbs=get;list;watch;create;update;patch;delete
// +kubebuilder:rbac:groups=sentry.io,resources=tenants/status,verbs=get;update;patch
// +kubebuilder:rbac:groups="",resources=secrets,verbs=get;list;watch;create;update;patch;delete
func (r *TenantReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
logger := log.FromContext(ctx)
var tenant sentryv1alpha1.Tenant
if err := r.Get(ctx, req.NamespacedName, &tenant); err != nil {
if apierrors.IsNotFound(err) {
// Deleted -- owned Secret is garbage-collected by K8s via
// its OwnerReference (set in reconcileSecret below), nothing
// else to clean up at this layer. See this file's package
// doc comment: real deprovisioning (revoking ClickHouse
// grants) isn't this controller's job.
// Deleted -- the owned Secret -provision-tenant created (if
// any) is garbage-collected by K8s via its OwnerReference,
// nothing else to clean up at this layer. Real
// deprovisioning (revoking ClickHouse grants) isn't this
// controller's job, or -provision-tenant's today -- see
// /docs/security/threat-model.md's non-goals.
return ctrl.Result{}, nil
}
return ctrl.Result{}, fmt.Errorf("getting tenant: %w", err)
}
secretName, err := r.reconcileSecret(ctx, &tenant)
if err != nil {
logger.Error(err, "reconciling clickhouse secret")
return ctrl.Result{}, err
// provisioned is true once -provision-tenant has confirmed real
// ClickHouse provisioning by setting this field -- see
// TenantStatus.ClickHouseDatabaseName's doc comment. This
// controller treats it as the sole source of truth for "has
// provisioning actually happened," never claiming PhaseActive on
// its own say-so the way the pre-unification version did.
provisioned := tenant.Status.ClickHouseDatabaseName != ""
condStatus := metav1.ConditionFalse
reason, message := "AwaitingProvisioning", "waiting for enterprise-api -provision-tenant to provision ClickHouse for this tenant"
switch {
case tenant.Spec.Suspended:
tenant.Status.Phase = sentryv1alpha1.PhaseSuspended
reason, message = "Suspended", "tenant is suspended (spec.suspended=true)"
case provisioned:
tenant.Status.Phase = sentryv1alpha1.PhaseActive
condStatus = metav1.ConditionTrue
reason, message = "Provisioned", fmt.Sprintf("ClickHouse database %q is provisioned", tenant.Status.ClickHouseDatabaseName)
default:
tenant.Status.Phase = sentryv1alpha1.PhaseProvisioning
}
desiredPhase := sentryv1alpha1.PhaseActive
if tenant.Spec.Suspended {
desiredPhase = sentryv1alpha1.PhaseSuspended
}
tenant.Status.ClickHouseDatabaseName = tenant.Name
tenant.Status.ClickHouseSecretRef = secretName
tenant.Status.TantivyIndexPath = tantivyIndexPath(&tenant)
tenant.Status.Phase = desiredPhase
tenant.Status.ObservedGeneration = tenant.Generation
meta.SetStatusCondition(&tenant.Status.Conditions, metav1.Condition{
Type: sentryv1alpha1.ConditionReady,
Status: metav1.ConditionTrue,
Reason: "SecretReconciled",
Message: fmt.Sprintf("ClickHouse credential secret %q is present", secretName),
Status: condStatus,
Reason: reason,
Message: message,
ObservedGeneration: tenant.Generation,
})
@@ -128,59 +100,8 @@ func (r *TenantReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctr
return ctrl.Result{}, nil
}
// reconcileSecret creates the tenant's ClickHouse credential Secret if
// it doesn't already exist. Deliberately never updates an existing
// Secret's password -- rotating a live tenant's ClickHouse credential
// out from under it (without first updating the ClickHouse-side grant,
// which this controller doesn't do) would just break every open
// connection for no benefit; credential rotation is real future work
// that needs to be coordinated with internal/tenantprovision, not
// something this reconcile loop can safely do alone.
func (r *TenantReconciler) reconcileSecret(ctx context.Context, tenant *sentryv1alpha1.Tenant) (string, error) {
name := clickHouseSecretName(tenant)
var existing corev1.Secret
err := r.Get(ctx, types.NamespacedName{Namespace: tenant.Namespace, Name: name}, &existing)
if err == nil {
return name, nil
}
if !apierrors.IsNotFound(err) {
return "", fmt.Errorf("getting secret: %w", err)
}
password, err := generatePassword()
if err != nil {
return "", err
}
secret := &corev1.Secret{
ObjectMeta: metav1.ObjectMeta{
Name: name,
Namespace: tenant.Namespace,
Labels: map[string]string{
"app.kubernetes.io/managed-by": "sentry-tenant-operator",
"sentry.io/tenant": tenant.Name,
},
},
Type: corev1.SecretTypeOpaque,
StringData: map[string]string{
"username": "tenant_" + tenant.Name,
"password": password,
"database": tenant.Name,
},
}
if err := controllerutil.SetControllerReference(tenant, secret, r.Scheme); err != nil {
return "", fmt.Errorf("setting owner reference: %w", err)
}
if err := r.Create(ctx, secret); err != nil {
return "", fmt.Errorf("creating secret: %w", err)
}
return name, nil
}
func (r *TenantReconciler) SetupWithManager(mgr ctrl.Manager) error {
return ctrl.NewControllerManagedBy(mgr).
For(&sentryv1alpha1.Tenant{}).
Owns(&corev1.Secret{}).
Complete(r)
}