cairnobs/deploy/README.md

deploy

Kubernetes deployment for Cairn OBS, added in Phase 4 (/deploy was deliberately stubbed through Phase 3 -- see /PROJECT-SPEC.md's Phase 3 non-goals). Two pieces:

• operator/ -- a small Go controller-runtime Operator managing one CRD (Tenant). See operator/README.md.
• helm/cairnobs/ -- a Helm chart covering every docker-compose.yml service, plus the operator and Tenant CRs when enterprise.enabled=true. See helm/cairnobs/README.md.

What "multi-tenant-aware" means here, precisely

Per /docs/phase-4-isolation-design.md, tenant isolation itself lives at the application layer inside enterprise/ (enterprise-api holds a map of per-tenant ClickHouse connection pools via internal/chrunner; search holds a map of per-tenant Tantivy indices via src/registry.rs) -- not at the Kubernetes layer. This directory is not "one Deployment per tenant" or a general multi-cluster system; that's an explicit Phase 4 non-goal (see /PROJECT-SPEC.md). What it does add:

• A Tenant CRD + controller (operator/internal/controller) that reflects real provisioning state onto status.phase/a Ready condition, derived from whether enterprise-api -provision-tenant has reported real ClickHouse provisioning.
• A Helm chart that can install zero-or-more Tenant CRs (values.tenants) alongside the rest of the stack, and swaps api's Deployment for enterprise-api's whenever enterprise.enabled is true, so which query binary actually serves traffic is no longer a separately-forgettable decision (see helm/cairnobs/README.md's "api vs enterprise-api" section).

Now unified, in a deliberately lightweight way: enterprise-api -provision-tenant=<id> stays the sole real actor — it's the only thing that calls ClickHouse (CREATE DATABASE/CREATE USER/GRANT, via enterprise/internal/tenantprovision) and writes rbacstore. What changed: once it succeeds, it also syncs the result into the Tenant CRD (enterprise/internal/tenantcrd) — creating the Secret with real credentials (the controller no longer generates a placeholder one that authenticated against nothing) and setting the status fields the controller reads to compute Phase/Ready. The controller itself gained no new credentials and still never touches ClickHouse/Postgres -- it's a pure function of spec.suspended and whatever -provision-tenant has reported, never an independent second guess at "is this tenant really provisioned." A Tenant reaching status.phase: Active now means the same thing rbacstore.tenants. status='active' does, not two different claims — see enterprise/internal/tenantcrd's and operator/internal/controller/ tenant_controller.go's doc comments for the full split, and enterprise-api -provision-tenant's TENANT_CRD_NAMESPACE env var (set automatically by the Helm chart when tenantOperator.enabled) to turn this on. Deliberately not built: the operator's reconcile loop itself calling ClickHouse/rbacstore directly (a "full unification" option considered and set aside — it would give the operator two new credential sets and require real reconcile-loop idempotency design for an inherently one-shot external side effect, a bigger and riskier change than this repo's provisioning story needed to close the actual gap, which was two disconnected sources of truth, not two actors).

Verification status -- read before trusting this against a real cluster

Now verified against a real live Kubernetes cluster. kind/ kubectl/helm were installed without root (static binaries into ~/.local/bin), a real local cluster was created, every image this chart references was built and loaded into it, and the full two-tenant walkthrough (helm/cairnobs/README.md) was run end to end -- both tenants reached Tenant.status.phase: Active with real generated ClickHouse credentials in their Secrets. See /docs/phase-4-runbook.md §7 for the exact commands and the two real chart bugs this run found and fixed (enterprise-auth missing its Postgres connection env vars entirely, ClickHouse missing the env var that grants CREATE USER privilege) -- neither helm lint, helm template, nor the kubeconform schema check below could have caught either, since both only manifest once real pods actually try to start and connect to each other.

What was verified offline, before real cluster access existed (still true, kept as additional evidence, not superseded by the above):

• deploy/operator: go build/go vet/go test ./... all pass, including reconciler tests against controller-runtime's fake client (internal/controller/tenant_controller_test.go) -- real reconcile logic exercised, but not against a real apiserver (no envtest binaries available; see that test file's doc comment).
• enterprise/internal/tenantcrd (the "lightweight unification" half -provision-tenant runs): go test passes against k8s.io/client-go's fake dynamic and typed clientsets -- real client library, fake transport, same shape as enterprise/internal/ searchclient's in-process gRPC tests. What this doesn't prove: that cairnobs.io/v1alpha1.Tenant's real CRD schema (a real apiserver's OpenAPI validation) accepts exactly what this package writes -- the helm template/kubeconform check below covers the schema shape, not a live write against it.
• deploy/operator/config/crd/cairnobs.io_tenants.yaml: parsed with sigs.k8s.io/yaml + strict-unmarshaled into the real k8s.io/apiextensions-apiserver CustomResourceDefinition Go type -- catches YAML syntax errors and structural mistakes, not a live-cluster admission check.
• deploy/helm/cairnobs: helm lint passes; helm template renders cleanly under both default values and a enterprise.enabled: true + two-tenant override; the rendered output was checked with kubeconform -strict against the real Kubernetes 1.31 OpenAPI schema for every built-in resource kind (22-31 resources depending on values, 0 invalid) -- this catches schema mistakes (wrong field names, wrong types) but not whether the resources actually reconcile correctly together on a live cluster (Job/StatefulSet startup ordering, PVC provisioning, actual pod scheduling). Specifically confirmed by parsing the rendered YAML (not just eyeballing it): exactly one Deployment/Service named cairnobs-api renders in each mode, with the enterprise.enabled: true render using the enterprise-api image and the default render using plain api's. Also confirmed for the tenantOperator.enabled: true case: enterprise-api gets its own ServiceAccount/Role/RoleBinding (exactly tenants/tenants/status/ secrets, no more), tenant-operator's own ClusterRole no longer grants secrets at all, and TENANT_CRD_NAMESPACE is set on enterprise-api's container only when tenantOperator.enabled is true.
• Docker image builds (operator/Dockerfile and every other Dockerfile this chart references) are now confirmed working too -- all twelve images this chart needs were built and loaded into the test kind cluster above.

Before relying on this in production: ingest needs a real cert-manager (or equivalent) issued Secret, not hack/dev-certs's throwaway dev certs; ClickHouse/Postgres data isn't backed by anything durable beyond the cluster's own PVC provisioner in this chart; and only Auth0 has been tried as a real external IdP so far (see /docs/phase-4-runbook.md §3a/§3b).