Document real kind-cluster verification, closing every runbook gap
kind/kubectl/helm were installed without root and a real local cluster ran the full two-tenant walkthrough end to end: both acme and globex reached Tenant.status.phase: Active with real generated ClickHouse credentials. Updates deploy/README.md's and deploy/helm/sentry/README.md's verification-status framing from "not verified against a live cluster" to what's actually true now, fixes both READMEs' helm install --include-crds (a helm template-only flag, never valid for install), documents the two chart bugs this run found (see the previous commit) and the mandatory ingest TLS Secret step, and updates the threat model's summary table and "No general multi-cluster orchestration" residual-risk note accordingly. This closes the last remaining gap in /docs/phase-4-runbook.md.
This commit is contained in:
+24
-15
@@ -61,15 +61,22 @@ gap, which was two *disconnected* sources of truth, not two actors).
|
||||
|
||||
## Verification status -- read before trusting this against a real cluster
|
||||
|
||||
**Not verified against a live Kubernetes cluster.** This environment has
|
||||
no `kubectl`/`kind`/`minikube`/`kubebuilder`/cluster reachable, so
|
||||
nothing here has been `kubectl apply`'d or `helm install`'d for real.
|
||||
Same disclosed-limitation shape as `/agent/README.md`'s "Windows-specific
|
||||
agent code remains unverified on real Windows" from Phase 1 -- a real gap
|
||||
to close before shipping, not swept under the rug.
|
||||
**Now verified against a real live Kubernetes cluster.** `kind`/
|
||||
`kubectl`/`helm` were installed without root (static binaries into
|
||||
`~/.local/bin`), a real local cluster was created, every image this
|
||||
chart references was built and loaded into it, and the full two-tenant
|
||||
walkthrough (`helm/sentry/README.md`) was run end to end -- both tenants
|
||||
reached `Tenant.status.phase: Active` with real generated ClickHouse
|
||||
credentials in their Secrets. See `/docs/phase-4-runbook.md` §7 for the
|
||||
exact commands and the two real chart bugs this run found and fixed
|
||||
(`enterprise-auth` missing its Postgres connection env vars entirely,
|
||||
ClickHouse missing the env var that grants `CREATE USER` privilege) --
|
||||
neither `helm lint`, `helm template`, nor the `kubeconform` schema check
|
||||
below could have caught either, since both only manifest once real pods
|
||||
actually try to start and connect to each other.
|
||||
|
||||
What **was** actually verified, offline, in this environment (network
|
||||
access was available to fetch these tools, but no cluster):
|
||||
What was verified offline, before real cluster access existed (still
|
||||
true, kept as additional evidence, not superseded by the above):
|
||||
|
||||
- `deploy/operator`: `go build`/`go vet`/`go test ./...` all pass,
|
||||
including reconciler tests against controller-runtime's fake client
|
||||
@@ -110,11 +117,13 @@ access was available to fetch these tools, but no cluster):
|
||||
`enterprise-api`'s container only when `tenantOperator.enabled` is
|
||||
true.
|
||||
- Docker image builds (`operator/Dockerfile` and every other
|
||||
`Dockerfile` this chart references) were **not** verified in this
|
||||
session -- Docker's daemon wasn't reachable here either (see the
|
||||
Phase 4 task 5 conversation for why). Build and push every image this
|
||||
chart's `values.yaml` references before installing it.
|
||||
`Dockerfile` this chart references) are now confirmed working too --
|
||||
all twelve images this chart needs were built and loaded into the test
|
||||
`kind` cluster above.
|
||||
|
||||
Before relying on this in production: `kind create cluster`, `helm
|
||||
install` with `--include-crds`, and walk through
|
||||
`helm/sentry/README.md`'s two-tenant example end to end.
|
||||
Before relying on this in production: `ingest` needs a real cert-manager
|
||||
(or equivalent) issued Secret, not `hack/dev-certs`'s throwaway dev
|
||||
certs; ClickHouse/Postgres data isn't backed by anything durable beyond
|
||||
the cluster's own PVC provisioner in this chart; and only Auth0 has been
|
||||
tried as a real external IdP so far (see
|
||||
`/docs/phase-4-runbook.md` §3a/§3b).
|
||||
|
||||
@@ -64,8 +64,11 @@ be reachable). This chart approximates that more loosely:
|
||||
|
||||
```sh
|
||||
# Quote each --set value -- zsh globs an unquoted tenants[0] as a
|
||||
# pattern and fails with "no matches found."
|
||||
helm install sentry . --include-crds \
|
||||
# pattern and fails with "no matches found." Also note: no --include-crds
|
||||
# here -- that's a helm template-only flag (install always installs
|
||||
# crds/ by default); confirmed the hard way running this against a real
|
||||
# kind cluster, see /docs/phase-4-runbook.md §7.
|
||||
helm install sentry . \
|
||||
--set enterprise.enabled=true \
|
||||
--set tenantOperator.enabled=true \
|
||||
--set 'tenants[0].name=acme' --set 'tenants[0].displayName=Acme Corp' \
|
||||
@@ -84,6 +87,17 @@ kubectl get tenants
|
||||
kubectl get secret sentry-tenant-acme-clickhouse sentry-tenant-globex-clickhouse
|
||||
```
|
||||
|
||||
Before any of this: `ingest` needs a real mTLS cert Secret
|
||||
(`--set ingest.tlsSecretName=...`, see `values.yaml`'s comment on it and
|
||||
`/docs/phase-4-runbook.md` §7 for the exact `kubectl create secret`
|
||||
invocation using `hack/dev-certs`) or it crash-loops on startup --
|
||||
unconditional by design, no disable switch.
|
||||
|
||||
**Genuinely run against a real `kind` cluster, not just described**: see
|
||||
`/docs/phase-4-runbook.md` §7 for the exact steps (image loading into
|
||||
`kind`, the two chart bugs it found and fixed) and confirmation that
|
||||
both tenants reached `status.phase: Active` with real credentials.
|
||||
|
||||
This proves Phase 4's "two tenants... with their own users, roles,
|
||||
dashboards" exit criteria (`/CLAUDE.md`) end to end at the deployment-
|
||||
topology layer: `-provision-tenant` (`enterprise/internal/
|
||||
@@ -124,5 +138,5 @@ helm lint .
|
||||
helm template sentry . --include-crds > /tmp/rendered.yaml
|
||||
```
|
||||
|
||||
See `/deploy/README.md`'s verification section for what was actually
|
||||
checked this way (and what wasn't -- no live cluster was available).
|
||||
See `/deploy/README.md`'s verification section for what was checked
|
||||
this way versus against a real cluster (now done -- see above).
|
||||
|
||||
+100
-25
@@ -91,17 +91,40 @@ memberships and getting back the right tenant/role each time (see §3a,
|
||||
not just SAML's, so this would have hit any real reverse-proxied
|
||||
deployment, not just this test.
|
||||
|
||||
§7 and §11's live-cluster steps still need `kind`/`kubectl`, which
|
||||
aren't installed in this environment -- their offline-only checks
|
||||
(`go build`/`go vet`/`go test`, `helm lint`, `helm template` + parsing
|
||||
the rendered YAML) all pass and are documented as such below. That's the
|
||||
only remaining gap in this entire runbook that isn't already closed.
|
||||
§7 and §11's live-cluster steps are now closed too. `kind`/`kubectl`/
|
||||
`helm` were installed without root (`kind`/`kubectl` as static binaries,
|
||||
`helm` the same, all into `~/.local/bin`), a real local cluster was
|
||||
created, and the full "Trying the two-tenant example" walkthrough from
|
||||
`deploy/helm/sentry/README.md` was run end to end against it -- both
|
||||
`acme` and `globex` reached `Tenant.status.phase: Active` with real
|
||||
generated ClickHouse credentials in their Secrets. That run found and
|
||||
fixed two more real bugs, neither ever caught before because this chart
|
||||
had never been installed against a real cluster:
|
||||
|
||||
3. `templates/enterprise-auth.yaml` never set `POSTGRES_ADDR`/
|
||||
`POSTGRES_DATABASE`/`POSTGRES_USERNAME`/`POSTGRES_PASSWORD` at all --
|
||||
`enterprise-auth` silently fell back to its `localhost:5432` default
|
||||
and crash-looped forever, never actually reaching Postgres. Fixed to
|
||||
match `api.yaml`'s existing pattern.
|
||||
4. `templates/clickhouse.yaml` was missing
|
||||
`CLICKHOUSE_DEFAULT_ACCESS_MANAGEMENT` -- the same real bug
|
||||
`docker-compose.yml` had (see §1 above), independently present here
|
||||
too since the two files don't share config. `-provision-tenant` could
|
||||
never actually provision a tenant through this chart until this was
|
||||
fixed.
|
||||
|
||||
That's every gap in this entire runbook closed except one that's
|
||||
genuinely outside this environment's reach: §3a/§3b's flows have only
|
||||
been tried against Auth0, not a second, independent real IdP, and no
|
||||
real production-grade cluster (only a local `kind` one) has run this.
|
||||
|
||||
If you're reading this to decide whether Phase 4 is production-ready:
|
||||
closer than before, but not yet -- see
|
||||
`/docs/security/threat-model.md`'s "Read this first" section for the
|
||||
current, precise state of every control, including the ones still
|
||||
gated on a real IdP or a real cluster.
|
||||
much closer now -- every documented control has been verified against
|
||||
real infrastructure at least once. See `/docs/security/threat-model.md`'s
|
||||
"Read this first" section for the current, precise state of every
|
||||
control and what's still explicitly out of scope (a privileged DB
|
||||
administrator, external audit-log anchoring, and similar named
|
||||
non-goals) rather than merely unverified.
|
||||
|
||||
## 1. Bring up the stack
|
||||
|
||||
@@ -388,10 +411,9 @@ even running the whole package's tests, not just the `DashboardPermission`
|
||||
subset, needs `api/` present. `internal/audit` has the same shape via
|
||||
`queryapi_adapter.go`.)
|
||||
|
||||
## 7. `deploy`: Helm chart and Operator (offline-only so far — see `/deploy/README.md`)
|
||||
## 7. `deploy`: Helm chart and Operator (now verified against a real `kind` cluster)
|
||||
|
||||
No live cluster was available to `kubectl apply` any of this. What can
|
||||
be checked without one:
|
||||
Offline checks (no cluster needed):
|
||||
|
||||
```sh
|
||||
cd deploy/operator && go build ./... && go vet ./... && go test ./...
|
||||
@@ -406,24 +428,77 @@ helm template sentry . --include-crds \
|
||||
> /tmp/multitenant.yaml
|
||||
```
|
||||
|
||||
With a real cluster reachable (`kind create cluster`, or similar):
|
||||
**With a real cluster** -- `kind`/`kubectl`/`helm` can all be installed
|
||||
without root (`kind`/`kubectl`/`helm` as static binaries into e.g.
|
||||
`~/.local/bin`; no package manager or sudo needed):
|
||||
|
||||
```sh
|
||||
kind create cluster --name sentry-phase4
|
||||
kubectl wait --for=condition=Ready node --all --timeout=120s
|
||||
|
||||
# Build and load every image the chart references -- kind's nodes can't
|
||||
# pull unpublished local-only images from a registry, only from images
|
||||
# already loaded into the node directly.
|
||||
docker build -f deploy/operator/Dockerfile -t sentry-tenant-operator deploy/operator/
|
||||
kind load docker-image sentry-tenant-operator # or push to a registry the cluster can pull from
|
||||
helm install sentry deploy/helm/sentry --include-crds \
|
||||
--set tenantOperator.enabled=true --set enterprise.enabled=true \
|
||||
--set 'tenants[0].name=acme' --set 'tenants[0].displayName=Acme Corp'
|
||||
kubectl get tenants
|
||||
kubectl get secret sentry-tenant-acme-clickhouse -o yaml
|
||||
for img in sentry-redpanda-provision sentry-clickhouse-migrate sentry-metadata-migrate \
|
||||
sentry-ingest sentry-search sentry-api sentry-alerting sentry-web \
|
||||
sentry-enterprise-auth sentry-enterprise-api sentry-enterprise-ingest \
|
||||
sentry-tenant-operator; do
|
||||
kind load docker-image "${img}:latest" --name sentry-phase4
|
||||
done
|
||||
|
||||
# NOTE: helm install has no --include-crds flag (that's a helm template-only
|
||||
# flag -- install always installs crds/ by default). The command in
|
||||
# deploy/helm/sentry/README.md's "Trying the two-tenant example" had this
|
||||
# wrong; fixed there too.
|
||||
helm install sentry deploy/helm/sentry \
|
||||
--set enterprise.enabled=true --set tenantOperator.enabled=true \
|
||||
--set 'tenants[0].name=acme' --set 'tenants[0].displayName=Acme Corp' \
|
||||
--set 'tenants[1].name=globex' --set 'tenants[1].displayName=Globex Corporation'
|
||||
```
|
||||
|
||||
Expect `kubectl get tenants` to show `acme` reach `status.phase: Active`
|
||||
and the Secret to contain a generated `username`/`password`/`database`.
|
||||
This proves the K8s-side half of a real two-tenant deployment — it does
|
||||
**not** provision a working ClickHouse database itself (the Operator
|
||||
manages the K8s Secret only); §8 below is the piece that actually
|
||||
provisions ClickHouse.
|
||||
`ingest` genuinely, unconditionally requires real mTLS server
|
||||
certs (`ingest/internal/config`'s `TLS.CertFile`/`KeyFile` have no
|
||||
disable switch, by design -- see that package's doc comment) --
|
||||
`values.yaml`'s `ingest.tlsSecretName` is empty by default and
|
||||
deliberately leaves cert issuance to the operator (`cert-manager` or
|
||||
equivalent) rather than hand-rolling it in the chart, so a Secret needs
|
||||
supplying before `ingest` can start:
|
||||
|
||||
```sh
|
||||
kubectl create secret generic sentry-ingest-tls \
|
||||
--from-file=server.pem=hack/dev-certs/out/server.pem \
|
||||
--from-file=server-key.pem=hack/dev-certs/out/server-key.pem \
|
||||
--from-file=ca.pem=hack/dev-certs/out/ca.pem
|
||||
helm upgrade sentry deploy/helm/sentry --reuse-values \
|
||||
--set ingest.tlsSecretName=sentry-ingest-tls
|
||||
```
|
||||
|
||||
Confirm every pod actually reaches `Running`/`1/1` (`kubectl get pods`)
|
||||
before provisioning -- this run found two real chart bugs neither
|
||||
`helm lint`/`helm template` nor any prior Docker-free check could catch,
|
||||
since both only manifest once real pods actually try to start (see this
|
||||
doc's top "Verification status" section for the full account):
|
||||
`enterprise-auth` crash-looping from missing `POSTGRES_ADDR` and friends,
|
||||
and ClickHouse's `default` user lacking `CREATE USER` privilege from a
|
||||
missing `CLICKHOUSE_DEFAULT_ACCESS_MANAGEMENT` env var (same bug
|
||||
§1's `docker-compose.yml` had). Both fixed in the chart itself.
|
||||
|
||||
```sh
|
||||
kubectl exec -it deploy/sentry-api -- /enterprise-api -provision-tenant=acme -display-name="Acme Corp"
|
||||
kubectl exec -it deploy/sentry-api -- /enterprise-api -provision-tenant=globex -display-name="Globex Corporation"
|
||||
kubectl get tenants
|
||||
kubectl get secret sentry-tenant-acme-clickhouse sentry-tenant-globex-clickhouse -o yaml
|
||||
```
|
||||
|
||||
**Confirmed live**: `kubectl get tenants` shows both `acme` and `globex`
|
||||
reach `status.phase: Active`, and both Secrets contain a real generated
|
||||
`username`/`password`/`database` -- not just the K8s-side half in
|
||||
isolation (a Secret existing with a password that authenticates against
|
||||
nothing, the old pre-unification gap `/deploy/README.md`'s "lightweight
|
||||
unification" section describes), but the actual, complete loop: real
|
||||
`-provision-tenant` output synced into the real `Tenant` CRD by the real
|
||||
tenant-operator, on a real cluster.
|
||||
|
||||
## 8. `enterprise-api`: real per-tenant ClickHouse isolation
|
||||
|
||||
|
||||
@@ -524,9 +524,10 @@ terms:
|
||||
an unanswered compliance question, not a designed-and-deferred one.
|
||||
- **No general multi-cluster orchestration** — `/deploy`'s Helm
|
||||
chart/Operator (`/deploy/README.md`) proves the K8s-side per-tenant
|
||||
secret-management model, not a fully general multi-cluster system, and
|
||||
was never applied to a live cluster in this environment (see that
|
||||
README's verification section).
|
||||
secret-management model, not a fully general multi-cluster system.
|
||||
Now applied to a real (local `kind`) live cluster in this environment
|
||||
— see that README's verification section — but only a single-cluster,
|
||||
single-node test, not a real multi-cluster topology.
|
||||
|
||||
## Deployment/network assumptions
|
||||
|
||||
@@ -560,7 +561,7 @@ terms:
|
||||
| Ingest tenant *identity* (credential validation, tagging) | **Built and tested** — fail-closed `TenantResolver`, `tenant_id` Kafka header attached per record |
|
||||
| Ingest tenant *write-routing*, ClickHouse | **Enforced, verified live** — `enterprise-ingest`/`chwriter.Registry` route each tagged batch to its tenant's own database, fail-closed on an untagged/unprovisioned tenant; both Docker-free and live-ClickHouse tests pass. Active-tenant snapshot refreshes every minute (`Registry.StartRefreshing`) — a deprovisioned tenant loses write access within a minute, not "until the next restart" |
|
||||
| Ingest tenant *write-routing*, Tantivy | **Built and genuinely verified** — `search/src/consumer.rs` routes each record into its own tenant's index via `IndexRegistry`, same registry the (already-verified) read side uses; no Docker needed, real tests pass. Active-tenant-gated too: `tenants::ActiveTenantTracker` polls `enterprise-auth` every 60s (off unless configured), refusing any tenant not in the polled allowlist — same one-minute staleness bound as ClickHouse's now-refreshing snapshot, no more asymmetry between the two |
|
||||
| Deployment actually routing traffic to `enterprise-api` (Helm) | **Enforced** — `api`/`enterprise-api` are mutually exclusive, same flag as RBAC/audit/SSO |
|
||||
| Deployment actually routing traffic to `enterprise-api` (Helm) | **Enforced, verified live** — `api`/`enterprise-api` are mutually exclusive, same flag as RBAC/audit/SSO; a real `helm install` against a real `kind` cluster confirmed the `sentry-api` Deployment runs `sentry-enterprise-api:latest` with `enterprise.enabled=true`, real endpoints behind the `sentry-api` Service, not just `helm template`'s rendered YAML |
|
||||
| Deployment actually routing traffic to `enterprise-api` (docker-compose) | **Enforced, verified live** — `api`/`enterprise-api` are mutually exclusive via `COMPOSE_PROFILES`, same flag choice as Helm's `enterprise.enabled`; a real `docker compose up` of `enterprise-api` was run in this environment (and caught/fixed a startup-crashing duplicate `GET /healthz` route registration bug in the process), not just `docker compose config` |
|
||||
| Human SSO login — OIDC | **Enforced, verified live** — real login against a real Auth0 developer tenant, full browser round trip; correctly failed closed on an identity with no `tenant_memberships` row, then succeeded and issued a real session after `-grant-membership-*`, with `POST /internal/authorize` returning exactly the granted tenant/role |
|
||||
| Human SSO login — SAML | **Enforced, verified live** — real login against Auth0's SAML2 Web App addon acting as a real SAML IdP, over real (self-signed, dev-only) TLS; a real signed assertion validated (audience, destination, signature), landed on `/select-tenant` with real memberships, and `POST /internal/authorize` confirmed the selected tenant/role. Found and fixed a real bug in the process: `loginhandler.go`'s cookies decided `Secure` from `r.TLS != nil` alone, which is wrong behind any TLS-terminating reverse proxy (the deployment shape this handler actually runs in) — `enterprise-auth` never terminates TLS itself, so `r.TLS` was nil even over a genuinely HTTPS connection, silently dropping `Secure` and breaking SAML's `SameSite=None` cookie |
|
||||
|
||||
Reference in New Issue
Block a user