coffey-labs/cairnobs
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Unify the Tenant CRD with enterprise-api -provision-tenant (lightweight)

Browse Source 
Closes a gap named across CLAUDE.md/docs/architecture.md/deploy/README.md
since early Phase 4: the operator's Tenant CRD and -provision-tenant
were two disconnected mechanisms. The operator's reconciler generated a
K8s Secret with a locally-generated random password that authenticated
against nothing (nothing ever called ClickHouse to create a matching
user), and unconditionally claimed status.phase=Active the moment a
Tenant object existed -- actively misleading, not just incomplete.

Two unification shapes were considered (surfaced to the user via
AskUserQuestion, given the real difference in blast radius): the
operator's reconcile loop becoming a second real actor (new Postgres +
ClickHouse admin credentials flowing into the K8s controller, plus real
reconcile-loop idempotency/retry design for an inherently one-shot
external side effect), or keeping -provision-tenant as the sole real
actor and having it also sync its result into the CRD. Went with the
lighter option.

enterprise/internal/tenantcrd (new): a Syncer using the K8s dynamic
client (unstructured.Unstructured + a GroupVersionResource, not
deploy/operator's typed Tenant struct -- avoids a cross-module Go
dependency between two independently-versioned modules for one type).
Upserts the Tenant object, creates/updates a Secret with the *real*
ClickHouse credentials owned by that Tenant via an OwnerReference, then
patches status.{clickHouseDatabaseName,clickHouseSecretRef,
tantivyIndexPath}. Idempotent and safe to retry: never rotates a
credential across a re-sync, never overwrites a pre-existing
spec.displayName a human/GitOps process set.

cmd/enterprise-api/main.go's runProvisionTenant calls Sync when
TENANT_CRD_NAMESPACE is set (empty = no-op, same shape as every other
optional dependency in this codebase). Its "already active" refusal is
now split: ClickHouse re-provisioning is still refused (rotating a live
credential would break every open connection for no benefit), but CR
sync alone is now retryable using the credentials already on file in
rbacstore -- needed for retrying a previously-failed sync, or
backfilling CR sync for a tenant provisioned before this existed.

deploy/operator's reconciler rewritten to match: it never claims
PhaseActive on its own initiative anymore, only once
status.ClickHouseDatabaseName is non-empty (the field -provision-tenant,
and only -provision-tenant, sets). Phase is now a pure function of
{spec.suspended, status.ClickHouseDatabaseName != ""} recomputed every
reconcile, not toggled in place -- fixes a related bug the old code
would have hit once suspension was involved: un-suspending an
already-provisioned tenant needs to return straight to Active, which
isn't derivable from "last observed phase was Suspended" alone. The
reconciler no longer creates or manages any Secret, dropped its
`secrets` RBAC grant entirely, and gained zero new dependencies.

Helm chart: enterprise-api gets its own ServiceAccount/Role/RoleBinding
(get/list/create tenants, get/update/patch tenants/status, get/create/
update secrets -- least-privilege, scoped to the release namespace, not
a ClusterRole) and a TENANT_CRD_NAMESPACE env var, both gated on
tenantOperator.enabled. tenant-operator's ClusterRole loses the
secrets grant it no longer needs.

Verified in this environment: enterprise/internal/tenantcrd's tests run
against k8s.io/client-go's fake dynamic + typed clientsets (real client
library, fake transport, no cluster needed); deploy/operator's rewritten
tenant_controller_test.go runs against controller-runtime's fake
client, including new regression tests for the "must not claim Active
without confirmation" and "un-suspending returns to Active, not
Provisioning" properties; helm template + parsing the rendered YAML
confirms the RBAC split renders exactly as designed under both
tenantOperator.enabled=true/false. Not verified: an actual
-provision-tenant run against a real cluster with the operator watching
(no live cluster in this environment, same disclosed limitation as the
rest of /deploy). Docs updated in lockstep: CLAUDE.md, docs/architecture.md,
deploy/README.md, deploy/helm/sentry/README.md (including a corrected
"Trying the two-tenant example" walkthrough), phase-4-runbook.md (new
§11), enterprise/README.md. Also fixed two unrelated stale claims found
along the way: docs/architecture.md still said docker-compose.yml ran
plain api unconditionally (fixed in an earlier commit, doc not updated
then), and enterprise-api's own main.go doc comment still said Helm/
docker-compose wiring wasn't built yet.
This commit is contained in:
jcoffey-dev
2026-08-14 09:07:10 -07:00
parent 8d7326fc6a
commit 823f5d48d1
18 changed files with 1073 additions and 323 deletions
Download Patch File Download Diff File Expand all files Collapse all files
enterprise/cmd/enterprise-api/main.go
+73 -26
View File
@@ -17,13 +17,14 @@
// keeps running plain api/cmd/api, unchanged; a real multi-tenant
// deployment runs this one instead.
//
// Not built yet: the actual K8s/Helm wiring to run this binary in place
// of api's (docker-compose.yml adds it available, not defaulted into
// the traffic path, same shape as enterprise-auth's own addition in
// Phase 4 task 5), and `search`'s write side (ingest, and by extension
// the Redpanda consumer search itself runs) is still not tenant-aware --
// see enterprise/internal/searchclient and search/src/registry.rs's doc
// comments, and /docs/security/threat-model.md.
// Both Helm (deploy/helm/sentry/templates/api.yaml vs
// enterprise-api.yaml) and docker-compose.yml (COMPOSE_PROFILES) now
// make this the deployment-topology choice, not just a binary sitting
// unused alongside api's -- see this repo's CLAUDE.md. `search`'s write
// side (ingest, and by extension the Redpanda consumer search itself
// runs) is still not tenant-aware -- see enterprise/internal/searchclient
// and search/src/registry.rs's doc comments, and
// /docs/security/threat-model.md.
package main
import (
@@ -51,6 +52,7 @@ import (
"github.com/sentry/sentry/enterprise/internal/chrunner"
"github.com/sentry/sentry/enterprise/internal/rbacstore"
"github.com/sentry/sentry/enterprise/internal/searchclient"
"github.com/sentry/sentry/enterprise/internal/tenantcrd"
"github.com/sentry/sentry/enterprise/internal/tenantprovision"
)
@@ -193,6 +195,18 @@ func main() {
// and only then mark the tenant active. Same "offline operator action,
// not a network-reachable endpoint" shape as enterprise-auth's
// -mint-service-token.
//
// If cfg.TenantCRDNamespace is set, this also syncs the real result into
// the Tenant CRD (enterprise/internal/tenantcrd) -- the "lightweight
// unification" of this mechanism with deploy/operator's Tenant CRD (see
// that package's doc comment). An already-active tenant no longer
// refuses outright: ClickHouse (re-)provisioning is still refused (that
// part is unchanged -- rotating a live credential would break every
// open connection for no benefit), but CR sync alone is safe to retry
// using the credentials already on file in rbacstore, which matters if
// a previous run's CR sync failed (or TENANT_CRD_NAMESPACE is being
// turned on for a tenant provisioned before this feature existed) and
// needs to be retried without touching ClickHouse again.
func runProvisionTenant(ctx context.Context, logger *slog.Logger, cfg apiconfig.Config, rbac *rbacstore.Store, tenantID, displayName string) int {
adminConn, err := chdriver.Open(&chdriver.Options{
Addr: []string{cfg.ClickHouseAddr},
@@ -221,9 +235,10 @@ func runProvisionTenant(ctx context.Context, logger *slog.Logger, cfg apiconfig.
}
logger.Info("created tenant row", "tenant_id", tenantID)
}
if tenant.Status == "active" {
logger.Error("tenant is already active -- refusing to re-provision (would rotate a live credential)", "tenant_id", tenantID)
return 1
alreadyActive := tenant.Status == "active"
if alreadyActive {
logger.Info("tenant is already active -- skipping ClickHouse (re-)provisioning, will still sync the Tenant CRD if TENANT_CRD_NAMESPACE is set", "tenant_id", tenantID)
}
dataSource, err := rbac.GetDataSourceForTenant(ctx, tenantID)
@@ -232,32 +247,64 @@ func runProvisionTenant(ctx context.Context, logger *slog.Logger, cfg apiconfig.
logger.Error("getting data source", "error", err)
return 1
}
if alreadyActive {
// An active tenant with no data_sources row at all is
// inconsistent state runProvisionTenant's own gate should
// never have allowed -- fail loudly rather than silently
// provisioning ClickHouse for a tenant already marked
// active, which SetDataSourceClickHouseCredentials's own
// doc comment says must never happen twice.
logger.Error("tenant is active but has no data source row -- inconsistent state, refusing", "tenant_id", tenantID)
return 1
}
dataSource, err = rbac.CreateDataSource(ctx, tenantID, "default", tenantID, "/var/lib/sentry-search/tenants/"+tenantID)
if err != nil {
logger.Error("creating data source", "error", err)
return 1
}
}
if dataSource.ClickHouseUsername != nil {
logger.Error("data source already has ClickHouse credentials -- refusing to re-provision", "tenant_id", tenantID)
return 1
var creds tenantprovision.Credentials
if alreadyActive {
if dataSource.ClickHouseUsername == nil || dataSource.ClickHousePassword == nil {
logger.Error("tenant is active but its data source has no ClickHouse credentials -- inconsistent state, refusing", "tenant_id", tenantID)
return 1
}
creds = tenantprovision.Credentials{Username: *dataSource.ClickHouseUsername, Password: *dataSource.ClickHousePassword}
} else {
if dataSource.ClickHouseUsername != nil {
logger.Error("data source already has ClickHouse credentials -- refusing to re-provision", "tenant_id", tenantID)
return 1
}
creds, err = tenantprovision.New(adminConn).ProvisionClickHouse(ctx, tenantID)
if err != nil {
logger.Error("provisioning clickhouse", "error", err)
return 1
}
if err := rbac.SetDataSourceClickHouseCredentials(ctx, dataSource.ID, creds.Username, creds.Password); err != nil {
logger.Error("persisting clickhouse credentials", "error", err)
return 1
}
if err := rbac.SetTenantStatus(ctx, tenantID, "active"); err != nil {
logger.Error("activating tenant", "error", err)
return 1
}
logger.Info("tenant provisioned and active", "tenant_id", tenantID, "clickhouse_database", tenantID, "clickhouse_username", creds.Username)
}
creds, err := tenantprovision.New(adminConn).ProvisionClickHouse(ctx, tenantID)
if err != nil {
logger.Error("provisioning clickhouse", "error", err)
return 1
}
if err := rbac.SetDataSourceClickHouseCredentials(ctx, dataSource.ID, creds.Username, creds.Password); err != nil {
logger.Error("persisting clickhouse credentials", "error", err)
return 1
}
if err := rbac.SetTenantStatus(ctx, tenantID, "active"); err != nil {
logger.Error("activating tenant", "error", err)
return 1
if cfg.TenantCRDNamespace != "" {
syncer, err := tenantcrd.New(cfg.TenantCRDNamespace)
if err != nil {
logger.Error("building tenant CRD syncer", "error", err)
return 1
}
if err := syncer.Sync(ctx, tenantID, tenant.DisplayName, dataSource.TantivyIndexPath, tenantcrd.Credentials{Username: creds.Username, Password: creds.Password}); err != nil {
logger.Error("syncing tenant CRD", "error", err)
return 1
}
logger.Info("synced tenant CRD", "tenant_id", tenantID, "namespace", cfg.TenantCRDNamespace)
}
logger.Info("tenant provisioned and active", "tenant_id", tenantID, "clickhouse_database", tenantID, "clickhouse_username", creds.Username)
return 0
}