jcoffey-dev
823f5d48d1
Closes a gap named across CLAUDE.md/docs/architecture.md/deploy/README.md
since early Phase 4: the operator's Tenant CRD and -provision-tenant
were two disconnected mechanisms. The operator's reconciler generated a
K8s Secret with a locally-generated random password that authenticated
against nothing (nothing ever called ClickHouse to create a matching
user), and unconditionally claimed status.phase=Active the moment a
Tenant object existed -- actively misleading, not just incomplete.
Two unification shapes were considered (surfaced to the user via
AskUserQuestion, given the real difference in blast radius): the
operator's reconcile loop becoming a second real actor (new Postgres +
ClickHouse admin credentials flowing into the K8s controller, plus real
reconcile-loop idempotency/retry design for an inherently one-shot
external side effect), or keeping -provision-tenant as the sole real
actor and having it also sync its result into the CRD. Went with the
lighter option.
enterprise/internal/tenantcrd (new): a Syncer using the K8s dynamic
client (unstructured.Unstructured + a GroupVersionResource, not
deploy/operator's typed Tenant struct -- avoids a cross-module Go
dependency between two independently-versioned modules for one type).
Upserts the Tenant object, creates/updates a Secret with the *real*
ClickHouse credentials owned by that Tenant via an OwnerReference, then
patches status.{clickHouseDatabaseName,clickHouseSecretRef,
tantivyIndexPath}. Idempotent and safe to retry: never rotates a
credential across a re-sync, never overwrites a pre-existing
spec.displayName a human/GitOps process set.
cmd/enterprise-api/main.go's runProvisionTenant calls Sync when
TENANT_CRD_NAMESPACE is set (empty = no-op, same shape as every other
optional dependency in this codebase). Its "already active" refusal is
now split: ClickHouse re-provisioning is still refused (rotating a live
credential would break every open connection for no benefit), but CR
sync alone is now retryable using the credentials already on file in
rbacstore -- needed for retrying a previously-failed sync, or
backfilling CR sync for a tenant provisioned before this existed.
deploy/operator's reconciler rewritten to match: it never claims
PhaseActive on its own initiative anymore, only once
status.ClickHouseDatabaseName is non-empty (the field -provision-tenant,
and only -provision-tenant, sets). Phase is now a pure function of
{spec.suspended, status.ClickHouseDatabaseName != ""} recomputed every
reconcile, not toggled in place -- fixes a related bug the old code
would have hit once suspension was involved: un-suspending an
already-provisioned tenant needs to return straight to Active, which
isn't derivable from "last observed phase was Suspended" alone. The
reconciler no longer creates or manages any Secret, dropped its
`secrets` RBAC grant entirely, and gained zero new dependencies.
Helm chart: enterprise-api gets its own ServiceAccount/Role/RoleBinding
(get/list/create tenants, get/update/patch tenants/status, get/create/
update secrets -- least-privilege, scoped to the release namespace, not
a ClusterRole) and a TENANT_CRD_NAMESPACE env var, both gated on
tenantOperator.enabled. tenant-operator's ClusterRole loses the
secrets grant it no longer needs.
Verified in this environment: enterprise/internal/tenantcrd's tests run
against k8s.io/client-go's fake dynamic + typed clientsets (real client
library, fake transport, no cluster needed); deploy/operator's rewritten
tenant_controller_test.go runs against controller-runtime's fake
client, including new regression tests for the "must not claim Active
without confirmation" and "un-suspending returns to Active, not
Provisioning" properties; helm template + parsing the rendered YAML
confirms the RBAC split renders exactly as designed under both
tenantOperator.enabled=true/false. Not verified: an actual
-provision-tenant run against a real cluster with the operator watching
(no live cluster in this environment, same disclosed limitation as the
rest of /deploy). Docs updated in lockstep: CLAUDE.md, docs/architecture.md,
deploy/README.md, deploy/helm/sentry/README.md (including a corrected
"Trying the two-tenant example" walkthrough), phase-4-runbook.md (new
§11), enterprise/README.md. Also fixed two unrelated stale claims found
along the way: docs/architecture.md still said docker-compose.yml ran
plain api unconditionally (fixed in an earlier commit, doc not updated
then), and enterprise-api's own main.go doc comment still said Helm/
docker-compose wiring wasn't built yet.
121 lines
6.8 KiB
Markdown
121 lines
6.8 KiB
Markdown
# deploy
|
|
|
|
Kubernetes deployment for Sentry, added in Phase 4 (`/deploy` was
|
|
deliberately stubbed through Phase 3 -- see `/CLAUDE.md`'s Phase 3
|
|
non-goals). Two pieces:
|
|
|
|
- `operator/` -- a small Go controller-runtime Operator managing one CRD
|
|
(`Tenant`). See `operator/README.md`.
|
|
- `helm/sentry/` -- a Helm chart covering every `docker-compose.yml`
|
|
service, plus the operator and `Tenant` CRs when
|
|
`enterprise.enabled=true`. See `helm/sentry/README.md`.
|
|
|
|
## What "multi-tenant-aware" means here, precisely
|
|
|
|
Per `/docs/phase-4-isolation-design.md`, tenant isolation itself lives at
|
|
the **application layer** inside `enterprise/` (`enterprise-api` holds a
|
|
map of per-tenant ClickHouse connection pools via `internal/chrunner`;
|
|
`search` holds a map of per-tenant Tantivy indices via
|
|
`src/registry.rs`) -- not at the Kubernetes layer. This directory is
|
|
**not** "one Deployment per tenant" or a general multi-cluster system;
|
|
that's an explicit Phase 4 non-goal (see `/CLAUDE.md`). What it *does*
|
|
add:
|
|
|
|
- A `Tenant` CRD + controller (`operator/internal/controller`) that
|
|
reflects real provisioning state onto `status.phase`/a `Ready`
|
|
condition, derived from whether `enterprise-api -provision-tenant` has
|
|
reported real ClickHouse provisioning.
|
|
- A Helm chart that can install zero-or-more `Tenant` CRs
|
|
(`values.tenants`) alongside the rest of the stack, and swaps `api`'s
|
|
Deployment for `enterprise-api`'s whenever `enterprise.enabled` is
|
|
true, so which query binary actually serves traffic is no longer a
|
|
separately-forgettable decision (see `helm/sentry/README.md`'s "`api`
|
|
vs `enterprise-api`" section).
|
|
|
|
**Now unified, in a deliberately lightweight way**: `enterprise-api
|
|
-provision-tenant=<id>` stays the sole real actor — it's the only thing
|
|
that calls ClickHouse (`CREATE DATABASE`/`CREATE USER`/`GRANT`, via
|
|
`enterprise/internal/tenantprovision`) and writes `rbacstore`. What
|
|
changed: once it succeeds, it also syncs the result into the `Tenant`
|
|
CRD (`enterprise/internal/tenantcrd`) — creating the Secret with *real*
|
|
credentials (the controller no longer generates a placeholder one that
|
|
authenticated against nothing) and setting the status fields the
|
|
controller reads to compute `Phase`/`Ready`. The controller itself
|
|
gained no new credentials and still never touches ClickHouse/Postgres --
|
|
it's a pure function of `spec.suspended` and whatever
|
|
`-provision-tenant` has reported, never an independent second guess at
|
|
"is this tenant really provisioned." A `Tenant` reaching
|
|
`status.phase: Active` now means the same thing `rbacstore.tenants.
|
|
status='active'` does, not two different claims — see
|
|
`enterprise/internal/tenantcrd`'s and `operator/internal/controller/
|
|
tenant_controller.go`'s doc comments for the full split, and
|
|
`enterprise-api -provision-tenant`'s `TENANT_CRD_NAMESPACE` env var
|
|
(set automatically by the Helm chart when `tenantOperator.enabled`) to
|
|
turn this on. Deliberately not built: the operator's reconcile loop
|
|
itself calling ClickHouse/rbacstore directly (a "full unification"
|
|
option considered and set aside — it would give the operator two new
|
|
credential sets and require real reconcile-loop idempotency design for
|
|
an inherently one-shot external side effect, a bigger and riskier
|
|
change than this repo's provisioning story needed to close the actual
|
|
gap, which was two *disconnected* sources of truth, not two actors).
|
|
|
|
## Verification status -- read before trusting this against a real cluster
|
|
|
|
**Not verified against a live Kubernetes cluster.** This environment has
|
|
no `kubectl`/`kind`/`minikube`/`kubebuilder`/cluster reachable, so
|
|
nothing here has been `kubectl apply`'d or `helm install`'d for real.
|
|
Same disclosed-limitation shape as `/agent/README.md`'s "Windows-specific
|
|
agent code remains unverified on real Windows" from Phase 1 -- a real gap
|
|
to close before shipping, not swept under the rug.
|
|
|
|
What **was** actually verified, offline, in this environment (network
|
|
access was available to fetch these tools, but no cluster):
|
|
|
|
- `deploy/operator`: `go build`/`go vet`/`go test ./...` all pass,
|
|
including reconciler tests against controller-runtime's fake client
|
|
(`internal/controller/tenant_controller_test.go`) -- real reconcile
|
|
logic exercised, but not against a real apiserver (no `envtest`
|
|
binaries available; see that test file's doc comment).
|
|
- `enterprise/internal/tenantcrd` (the "lightweight unification"
|
|
half `-provision-tenant` runs): `go test` passes against
|
|
`k8s.io/client-go`'s fake dynamic and typed clientsets -- real client
|
|
library, fake transport, same shape as `enterprise/internal/
|
|
searchclient`'s in-process gRPC tests. What this doesn't prove: that
|
|
`sentry.io/v1alpha1.Tenant`'s real CRD schema (a real apiserver's
|
|
OpenAPI validation) accepts exactly what this package writes -- the
|
|
`helm template`/kubeconform check below covers the schema shape, not
|
|
a live write against it.
|
|
- `deploy/operator/config/crd/sentry.io_tenants.yaml`: parsed with
|
|
`sigs.k8s.io/yaml` + strict-unmarshaled into the real
|
|
`k8s.io/apiextensions-apiserver` `CustomResourceDefinition` Go type --
|
|
catches YAML syntax errors and structural mistakes, not a live-cluster
|
|
admission check.
|
|
- `deploy/helm/sentry`: `helm lint` passes; `helm template` renders
|
|
cleanly under both default values and a `enterprise.enabled: true` +
|
|
two-tenant override; the rendered output was checked with `kubeconform
|
|
-strict` against the real Kubernetes 1.31 OpenAPI schema for every
|
|
built-in resource kind (22-31 resources depending on values, 0
|
|
invalid) -- this catches schema mistakes (wrong field names, wrong
|
|
types) but not whether the resources actually reconcile correctly
|
|
together on a live cluster (Job/StatefulSet startup ordering, PVC
|
|
provisioning, actual pod scheduling). Specifically confirmed by
|
|
parsing the rendered YAML (not just eyeballing it): exactly one
|
|
`Deployment`/`Service` named `sentry-api` renders in each mode, with
|
|
the `enterprise.enabled: true` render using the `enterprise-api` image
|
|
and the default render using plain `api`'s. Also confirmed for the
|
|
`tenantOperator.enabled: true` case: `enterprise-api` gets its own
|
|
ServiceAccount/Role/RoleBinding (exactly `tenants`/`tenants/status`/
|
|
`secrets`, no more), `tenant-operator`'s own ClusterRole no longer
|
|
grants `secrets` at all, and `TENANT_CRD_NAMESPACE` is set on
|
|
`enterprise-api`'s container only when `tenantOperator.enabled` is
|
|
true.
|
|
- Docker image builds (`operator/Dockerfile` and every other
|
|
`Dockerfile` this chart references) were **not** verified in this
|
|
session -- Docker's daemon wasn't reachable here either (see the
|
|
Phase 4 task 5 conversation for why). Build and push every image this
|
|
chart's `values.yaml` references before installing it.
|
|
|
|
Before relying on this in production: `kind create cluster`, `helm
|
|
install` with `--include-crds`, and walk through
|
|
`helm/sentry/README.md`'s two-tenant example end to end.
|