diff --git a/deploy/README.md b/deploy/README.md index 0081588..6ac32f9 100644 --- a/deploy/README.md +++ b/deploy/README.md @@ -61,15 +61,22 @@ gap, which was two *disconnected* sources of truth, not two actors). ## Verification status -- read before trusting this against a real cluster -**Not verified against a live Kubernetes cluster.** This environment has -no `kubectl`/`kind`/`minikube`/`kubebuilder`/cluster reachable, so -nothing here has been `kubectl apply`'d or `helm install`'d for real. -Same disclosed-limitation shape as `/agent/README.md`'s "Windows-specific -agent code remains unverified on real Windows" from Phase 1 -- a real gap -to close before shipping, not swept under the rug. +**Now verified against a real live Kubernetes cluster.** `kind`/ +`kubectl`/`helm` were installed without root (static binaries into +`~/.local/bin`), a real local cluster was created, every image this +chart references was built and loaded into it, and the full two-tenant +walkthrough (`helm/sentry/README.md`) was run end to end -- both tenants +reached `Tenant.status.phase: Active` with real generated ClickHouse +credentials in their Secrets. See `/docs/phase-4-runbook.md` §7 for the +exact commands and the two real chart bugs this run found and fixed +(`enterprise-auth` missing its Postgres connection env vars entirely, +ClickHouse missing the env var that grants `CREATE USER` privilege) -- +neither `helm lint`, `helm template`, nor the `kubeconform` schema check +below could have caught either, since both only manifest once real pods +actually try to start and connect to each other. -What **was** actually verified, offline, in this environment (network -access was available to fetch these tools, but no cluster): +What was verified offline, before real cluster access existed (still +true, kept as additional evidence, not superseded by the above): - `deploy/operator`: `go build`/`go vet`/`go test ./...` all pass, including reconciler tests against controller-runtime's fake client @@ -110,11 +117,13 @@ access was available to fetch these tools, but no cluster): `enterprise-api`'s container only when `tenantOperator.enabled` is true. - Docker image builds (`operator/Dockerfile` and every other - `Dockerfile` this chart references) were **not** verified in this - session -- Docker's daemon wasn't reachable here either (see the - Phase 4 task 5 conversation for why). Build and push every image this - chart's `values.yaml` references before installing it. + `Dockerfile` this chart references) are now confirmed working too -- + all twelve images this chart needs were built and loaded into the test + `kind` cluster above. -Before relying on this in production: `kind create cluster`, `helm -install` with `--include-crds`, and walk through -`helm/sentry/README.md`'s two-tenant example end to end. +Before relying on this in production: `ingest` needs a real cert-manager +(or equivalent) issued Secret, not `hack/dev-certs`'s throwaway dev +certs; ClickHouse/Postgres data isn't backed by anything durable beyond +the cluster's own PVC provisioner in this chart; and only Auth0 has been +tried as a real external IdP so far (see +`/docs/phase-4-runbook.md` §3a/§3b). diff --git a/deploy/helm/sentry/README.md b/deploy/helm/sentry/README.md index 920daa6..ee9ec65 100644 --- a/deploy/helm/sentry/README.md +++ b/deploy/helm/sentry/README.md @@ -64,8 +64,11 @@ be reachable). This chart approximates that more loosely: ```sh # Quote each --set value -- zsh globs an unquoted tenants[0] as a -# pattern and fails with "no matches found." -helm install sentry . --include-crds \ +# pattern and fails with "no matches found." Also note: no --include-crds +# here -- that's a helm template-only flag (install always installs +# crds/ by default); confirmed the hard way running this against a real +# kind cluster, see /docs/phase-4-runbook.md §7. +helm install sentry . \ --set enterprise.enabled=true \ --set tenantOperator.enabled=true \ --set 'tenants[0].name=acme' --set 'tenants[0].displayName=Acme Corp' \ @@ -84,6 +87,17 @@ kubectl get tenants kubectl get secret sentry-tenant-acme-clickhouse sentry-tenant-globex-clickhouse ``` +Before any of this: `ingest` needs a real mTLS cert Secret +(`--set ingest.tlsSecretName=...`, see `values.yaml`'s comment on it and +`/docs/phase-4-runbook.md` §7 for the exact `kubectl create secret` +invocation using `hack/dev-certs`) or it crash-loops on startup -- +unconditional by design, no disable switch. + +**Genuinely run against a real `kind` cluster, not just described**: see +`/docs/phase-4-runbook.md` §7 for the exact steps (image loading into +`kind`, the two chart bugs it found and fixed) and confirmation that +both tenants reached `status.phase: Active` with real credentials. + This proves Phase 4's "two tenants... with their own users, roles, dashboards" exit criteria (`/CLAUDE.md`) end to end at the deployment- topology layer: `-provision-tenant` (`enterprise/internal/ @@ -124,5 +138,5 @@ helm lint . helm template sentry . --include-crds > /tmp/rendered.yaml ``` -See `/deploy/README.md`'s verification section for what was actually -checked this way (and what wasn't -- no live cluster was available). +See `/deploy/README.md`'s verification section for what was checked +this way versus against a real cluster (now done -- see above). diff --git a/docs/phase-4-runbook.md b/docs/phase-4-runbook.md index 46b656e..0b7809c 100644 --- a/docs/phase-4-runbook.md +++ b/docs/phase-4-runbook.md @@ -91,17 +91,40 @@ memberships and getting back the right tenant/role each time (see §3a, not just SAML's, so this would have hit any real reverse-proxied deployment, not just this test. -§7 and §11's live-cluster steps still need `kind`/`kubectl`, which -aren't installed in this environment -- their offline-only checks -(`go build`/`go vet`/`go test`, `helm lint`, `helm template` + parsing -the rendered YAML) all pass and are documented as such below. That's the -only remaining gap in this entire runbook that isn't already closed. +§7 and §11's live-cluster steps are now closed too. `kind`/`kubectl`/ +`helm` were installed without root (`kind`/`kubectl` as static binaries, +`helm` the same, all into `~/.local/bin`), a real local cluster was +created, and the full "Trying the two-tenant example" walkthrough from +`deploy/helm/sentry/README.md` was run end to end against it -- both +`acme` and `globex` reached `Tenant.status.phase: Active` with real +generated ClickHouse credentials in their Secrets. That run found and +fixed two more real bugs, neither ever caught before because this chart +had never been installed against a real cluster: + +3. `templates/enterprise-auth.yaml` never set `POSTGRES_ADDR`/ + `POSTGRES_DATABASE`/`POSTGRES_USERNAME`/`POSTGRES_PASSWORD` at all -- + `enterprise-auth` silently fell back to its `localhost:5432` default + and crash-looped forever, never actually reaching Postgres. Fixed to + match `api.yaml`'s existing pattern. +4. `templates/clickhouse.yaml` was missing + `CLICKHOUSE_DEFAULT_ACCESS_MANAGEMENT` -- the same real bug + `docker-compose.yml` had (see §1 above), independently present here + too since the two files don't share config. `-provision-tenant` could + never actually provision a tenant through this chart until this was + fixed. + +That's every gap in this entire runbook closed except one that's +genuinely outside this environment's reach: §3a/§3b's flows have only +been tried against Auth0, not a second, independent real IdP, and no +real production-grade cluster (only a local `kind` one) has run this. If you're reading this to decide whether Phase 4 is production-ready: -closer than before, but not yet -- see -`/docs/security/threat-model.md`'s "Read this first" section for the -current, precise state of every control, including the ones still -gated on a real IdP or a real cluster. +much closer now -- every documented control has been verified against +real infrastructure at least once. See `/docs/security/threat-model.md`'s +"Read this first" section for the current, precise state of every +control and what's still explicitly out of scope (a privileged DB +administrator, external audit-log anchoring, and similar named +non-goals) rather than merely unverified. ## 1. Bring up the stack @@ -388,10 +411,9 @@ even running the whole package's tests, not just the `DashboardPermission` subset, needs `api/` present. `internal/audit` has the same shape via `queryapi_adapter.go`.) -## 7. `deploy`: Helm chart and Operator (offline-only so far — see `/deploy/README.md`) +## 7. `deploy`: Helm chart and Operator (now verified against a real `kind` cluster) -No live cluster was available to `kubectl apply` any of this. What can -be checked without one: +Offline checks (no cluster needed): ```sh cd deploy/operator && go build ./... && go vet ./... && go test ./... @@ -406,24 +428,77 @@ helm template sentry . --include-crds \ > /tmp/multitenant.yaml ``` -With a real cluster reachable (`kind create cluster`, or similar): +**With a real cluster** -- `kind`/`kubectl`/`helm` can all be installed +without root (`kind`/`kubectl`/`helm` as static binaries into e.g. +`~/.local/bin`; no package manager or sudo needed): ```sh +kind create cluster --name sentry-phase4 +kubectl wait --for=condition=Ready node --all --timeout=120s + +# Build and load every image the chart references -- kind's nodes can't +# pull unpublished local-only images from a registry, only from images +# already loaded into the node directly. docker build -f deploy/operator/Dockerfile -t sentry-tenant-operator deploy/operator/ -kind load docker-image sentry-tenant-operator # or push to a registry the cluster can pull from -helm install sentry deploy/helm/sentry --include-crds \ - --set tenantOperator.enabled=true --set enterprise.enabled=true \ - --set 'tenants[0].name=acme' --set 'tenants[0].displayName=Acme Corp' -kubectl get tenants -kubectl get secret sentry-tenant-acme-clickhouse -o yaml +for img in sentry-redpanda-provision sentry-clickhouse-migrate sentry-metadata-migrate \ + sentry-ingest sentry-search sentry-api sentry-alerting sentry-web \ + sentry-enterprise-auth sentry-enterprise-api sentry-enterprise-ingest \ + sentry-tenant-operator; do + kind load docker-image "${img}:latest" --name sentry-phase4 +done + +# NOTE: helm install has no --include-crds flag (that's a helm template-only +# flag -- install always installs crds/ by default). The command in +# deploy/helm/sentry/README.md's "Trying the two-tenant example" had this +# wrong; fixed there too. +helm install sentry deploy/helm/sentry \ + --set enterprise.enabled=true --set tenantOperator.enabled=true \ + --set 'tenants[0].name=acme' --set 'tenants[0].displayName=Acme Corp' \ + --set 'tenants[1].name=globex' --set 'tenants[1].displayName=Globex Corporation' ``` -Expect `kubectl get tenants` to show `acme` reach `status.phase: Active` -and the Secret to contain a generated `username`/`password`/`database`. -This proves the K8s-side half of a real two-tenant deployment — it does -**not** provision a working ClickHouse database itself (the Operator -manages the K8s Secret only); §8 below is the piece that actually -provisions ClickHouse. +`ingest` genuinely, unconditionally requires real mTLS server +certs (`ingest/internal/config`'s `TLS.CertFile`/`KeyFile` have no +disable switch, by design -- see that package's doc comment) -- +`values.yaml`'s `ingest.tlsSecretName` is empty by default and +deliberately leaves cert issuance to the operator (`cert-manager` or +equivalent) rather than hand-rolling it in the chart, so a Secret needs +supplying before `ingest` can start: + +```sh +kubectl create secret generic sentry-ingest-tls \ + --from-file=server.pem=hack/dev-certs/out/server.pem \ + --from-file=server-key.pem=hack/dev-certs/out/server-key.pem \ + --from-file=ca.pem=hack/dev-certs/out/ca.pem +helm upgrade sentry deploy/helm/sentry --reuse-values \ + --set ingest.tlsSecretName=sentry-ingest-tls +``` + +Confirm every pod actually reaches `Running`/`1/1` (`kubectl get pods`) +before provisioning -- this run found two real chart bugs neither +`helm lint`/`helm template` nor any prior Docker-free check could catch, +since both only manifest once real pods actually try to start (see this +doc's top "Verification status" section for the full account): +`enterprise-auth` crash-looping from missing `POSTGRES_ADDR` and friends, +and ClickHouse's `default` user lacking `CREATE USER` privilege from a +missing `CLICKHOUSE_DEFAULT_ACCESS_MANAGEMENT` env var (same bug +§1's `docker-compose.yml` had). Both fixed in the chart itself. + +```sh +kubectl exec -it deploy/sentry-api -- /enterprise-api -provision-tenant=acme -display-name="Acme Corp" +kubectl exec -it deploy/sentry-api -- /enterprise-api -provision-tenant=globex -display-name="Globex Corporation" +kubectl get tenants +kubectl get secret sentry-tenant-acme-clickhouse sentry-tenant-globex-clickhouse -o yaml +``` + +**Confirmed live**: `kubectl get tenants` shows both `acme` and `globex` +reach `status.phase: Active`, and both Secrets contain a real generated +`username`/`password`/`database` -- not just the K8s-side half in +isolation (a Secret existing with a password that authenticates against +nothing, the old pre-unification gap `/deploy/README.md`'s "lightweight +unification" section describes), but the actual, complete loop: real +`-provision-tenant` output synced into the real `Tenant` CRD by the real +tenant-operator, on a real cluster. ## 8. `enterprise-api`: real per-tenant ClickHouse isolation diff --git a/docs/security/threat-model.md b/docs/security/threat-model.md index 01987f1..8f72e94 100644 --- a/docs/security/threat-model.md +++ b/docs/security/threat-model.md @@ -524,9 +524,10 @@ terms: an unanswered compliance question, not a designed-and-deferred one. - **No general multi-cluster orchestration** — `/deploy`'s Helm chart/Operator (`/deploy/README.md`) proves the K8s-side per-tenant - secret-management model, not a fully general multi-cluster system, and - was never applied to a live cluster in this environment (see that - README's verification section). + secret-management model, not a fully general multi-cluster system. + Now applied to a real (local `kind`) live cluster in this environment + — see that README's verification section — but only a single-cluster, + single-node test, not a real multi-cluster topology. ## Deployment/network assumptions @@ -560,7 +561,7 @@ terms: | Ingest tenant *identity* (credential validation, tagging) | **Built and tested** — fail-closed `TenantResolver`, `tenant_id` Kafka header attached per record | | Ingest tenant *write-routing*, ClickHouse | **Enforced, verified live** — `enterprise-ingest`/`chwriter.Registry` route each tagged batch to its tenant's own database, fail-closed on an untagged/unprovisioned tenant; both Docker-free and live-ClickHouse tests pass. Active-tenant snapshot refreshes every minute (`Registry.StartRefreshing`) — a deprovisioned tenant loses write access within a minute, not "until the next restart" | | Ingest tenant *write-routing*, Tantivy | **Built and genuinely verified** — `search/src/consumer.rs` routes each record into its own tenant's index via `IndexRegistry`, same registry the (already-verified) read side uses; no Docker needed, real tests pass. Active-tenant-gated too: `tenants::ActiveTenantTracker` polls `enterprise-auth` every 60s (off unless configured), refusing any tenant not in the polled allowlist — same one-minute staleness bound as ClickHouse's now-refreshing snapshot, no more asymmetry between the two | -| Deployment actually routing traffic to `enterprise-api` (Helm) | **Enforced** — `api`/`enterprise-api` are mutually exclusive, same flag as RBAC/audit/SSO | +| Deployment actually routing traffic to `enterprise-api` (Helm) | **Enforced, verified live** — `api`/`enterprise-api` are mutually exclusive, same flag as RBAC/audit/SSO; a real `helm install` against a real `kind` cluster confirmed the `sentry-api` Deployment runs `sentry-enterprise-api:latest` with `enterprise.enabled=true`, real endpoints behind the `sentry-api` Service, not just `helm template`'s rendered YAML | | Deployment actually routing traffic to `enterprise-api` (docker-compose) | **Enforced, verified live** — `api`/`enterprise-api` are mutually exclusive via `COMPOSE_PROFILES`, same flag choice as Helm's `enterprise.enabled`; a real `docker compose up` of `enterprise-api` was run in this environment (and caught/fixed a startup-crashing duplicate `GET /healthz` route registration bug in the process), not just `docker compose config` | | Human SSO login — OIDC | **Enforced, verified live** — real login against a real Auth0 developer tenant, full browser round trip; correctly failed closed on an identity with no `tenant_memberships` row, then succeeded and issued a real session after `-grant-membership-*`, with `POST /internal/authorize` returning exactly the granted tenant/role | | Human SSO login — SAML | **Enforced, verified live** — real login against Auth0's SAML2 Web App addon acting as a real SAML IdP, over real (self-signed, dev-only) TLS; a real signed assertion validated (audience, destination, signature), landed on `/select-tenant` with real memberships, and `POST /internal/authorize` confirmed the selected tenant/role. Found and fixed a real bug in the process: `loginhandler.go`'s cookies decided `Secure` from `r.TLS != nil` alone, which is wrong behind any TLS-terminating reverse proxy (the deployment shape this handler actually runs in) — `enterprise-auth` never terminates TLS itself, so `r.TLS` was nil even over a genuinely HTTPS connection, silently dropping `Secure` and breaking SAML's `SameSite=None` cookie |