cairnobs/docs/security/threat-model.md

# Cairn OBS Threat Model (Phase 4)

Written for a prospective enterprise customer's security team, describing
the system **as actually built** through Phase 4 task 7 — not the target
architecture. Where a control is designed but not yet implemented, this
document says so explicitly, with a pointer to the tracking doc/task.
See `/docs/phase-4-isolation-design.md` and `/docs/phase-4-rbac-design.md`
for the full design rationale behind the controls described here.

## Read this first: the single most important open finding

**Updated a seventh time — and this time the headline actually changes.**
This section originally read "log data queried through `POST /query` is
not tenant-isolated at all," then "ClickHouse is isolated but Tantivy
isn't," then "ingest tags records with a tenant identity but nothing
routes the write," then "ClickHouse write-routing is built but Tantivy's
isn't," then "both are write-routed but neither rechecks tenant-active
status live," then "both recheck, but ClickHouse's snapshot never
refreshes while Tantivy's does," then (sixth) "both engines are built
and code-complete, but the ClickHouse half has never actually run
against a real ClickHouse." That last gap is now closed: Docker access
became available, and every ClickHouse-dependent piece named below —
`tenantprovision`, `chrunner`, `chwriter`, the `system.*` metadata
isolation check, and `enterprise-api` itself actually starting and
serving traffic — has been run against a real
`clickhouse/clickhouse-server:24.8` and a real Postgres, not just
type-checked or run against fakes. Closing this loop caught and fixed
six real bugs that no amount of Docker-free testing could have found:
a Docker build-context bug that made `enterprise-auth` fail to build at
all, a validation gap in `dashboards.Store.AddPanel`/`UpdatePanel`, a
raw Postgres error leaking past `rbacstore`'s `ErrNotFound` boundary,
`enterprise-api` panicking on startup from a duplicate `GET /healthz`
route registration, ClickHouse's `default` user genuinely lacking
`CREATE USER` privilege until `CLICKHOUSE_DEFAULT_ACCESS_MANAGEMENT=1`
was added to `docker-compose.yml`, and — most load-bearing — a freshly
provisioned tenant user was **not** actually default-denied from
`system.*` on this ClickHouse version, exactly the risk task 2's
original design flagged as needing live verification rather than trust.
See `/docs/phase-4-runbook.md` §§1–14 for the full list of what was run
and what each finding was.

Both storage engines are isolated on both the read and write paths,
**both gate writes on an active-tenant check, and both refresh that
check periodically** — `chwriter.Registry.StartRefreshing` re-lists
active tenants every minute, the same interval
`tenants.ActiveTenantTracker` uses on the Tantivy side, opening
connections for newly-active tenants and closing/removing ones no
longer active — a deprovisioned tenant loses ClickHouse write access
within a minute, not "until the next `enterprise-ingest` restart."
Neither engine does a live per-write check (a database/HTTP round trip
per record would be a real throughput cost neither implementation
accepts), so a minute-wide staleness window remains on both sides by
design, not by oversight. What's left: two tenants (`acme`, `globex`)
have been provisioned and exercised end-to-end in this environment.
Human login itself is now verified for real, via both protocols — a
real Auth0 identity logged in via OIDC and (separately) via Auth0's
SAML2 Web App addon, selected between both tenants each way, and
`POST /internal/authorize` confirmed each selection issued the right
tenant/role (see §3a/§3b/§12 below) — but that walkthrough ran against
plain `api` serving `web`'s traffic, not `enterprise-api`, so the
specific combination of "real human SSO session" and "real per-tenant
ClickHouse routing via `chrunner`" in the same request hasn't been
driven end to end yet; each half is independently confirmed (real
ClickHouse connections and Go integration tests for the routing half,
real Auth0 sessions for the human-login half), just not together in one
request. And **whether a given deployment actually runs the isolated
binaries** remains a deployment-time decision, not a code-level
guarantee — see below.

**ClickHouse (the SQL path) is built and now genuinely verified live.**
`enterprise/internal/tenantprovision` (real `CREATE DATABASE`/`CREATE
USER`/`GRANT`/`REVOKE`) and `enterprise/internal/chrunner` (a per-tenant
connection registry implementing `api/querylang/executor.SQLRunner`,
resolving the tenant from request identity, never a parameter) are wired
into `enterprise/cmd/enterprise-api`, which has itself been built,
started, and confirmed serving traffic on a real docker-compose stack.
Every ClickHouse-dependent test in both packages passes against a real
`clickhouse/clickhouse-server:24.8`, including the cross-tenant raw-SQL
probe and the corrected `system.*` metadata-isolation check (see "Known
residual risks" below for the real, version-specific wrinkle that check
found).

**Tantivy (the free-text path) is also built, and — unlike the
ClickHouse pieces — genuinely verified in this environment.**
`search/src/registry.rs`'s `IndexRegistry` resolves a `SearchRequest.
tenant_id` to its own on-disk Tantivy index, opened on demand;
`enterprise/internal/searchclient` sets that field from the
authenticated request identity, mirroring `chrunner`'s exact "read from
ctx, fail closed, never a parameter" shape. Because Tantivy is an
embedded library (no external service to fake or skip), both sides could
actually be run: `search/src/registry.rs`'s
`tenant_index_is_isolated_from_default_and_other_tenants` seeds three
real indices with the same search term and confirms a tenant-scoped
search returns only that tenant's document; `enterprise/internal/
searchclient`'s tests run a real in-process gRPC server and confirm the
wire-level `SearchRequest` carries the right `tenant_id`. All pass, for
real, no disclaimer needed for this specific claim.

**Both Helm and docker-compose now close this.**
`deploy/helm/cairnobs/templates/api.yaml` and `enterprise-api.yaml` are
mutually exclusive, gated on opposite sides of the same
`enterprise.enabled` flag, rendering to the same Service name/port — so
a Helm-deployed cluster runs exactly one of the two binaries, chosen by
the same flag that turns on RBAC/audit/SSO, not a second
independently-forgettable decision. Verified by parsing (not
eyeballing) the rendered YAML under both values: exactly one `cairnobs-api`
Deployment either way, with the right image. `docker-compose.yml`'s
`api`/`enterprise-api` services are now the analogous mutually-exclusive
choice, gated behind `COMPOSE_PROFILES` (`.env` checks in
`single-tenant`, i.e. plain `api`, as the zero-config default) and
sharing the same host port/network-alias trick to stay transparent to
`alerting`/`web` either way — verified via `docker compose config`
(renders and validates the merged YAML without a daemon; confirms
`api`/`enterprise-api` never both appear in `--services` output for the
same profile selection) *and* via an actual `docker compose up` of
`enterprise-api` in this environment, which is what caught the duplicate
`GET /healthz` route panic mentioned above — see
`/docs/phase-4-runbook.md` §10a. This still only constrains
*deployment*, not *operation*: nothing stops an
operator from manually running plain `api`'s image against a cluster
(or compose project) that has tenants
provisioned, pointing at the same ClickHouse/Postgres. The Helm chart
makes the *default*, chart-managed path correct; it isn't a runtime
guard against misconfiguration.

**Ingest now has a real tenant identity, and both storage engines'
write-routing is built.** `chrunner`/`searchclient` prove *read*
isolation given tenant-scoped data exists; an optional
`ingest/internal/grpcserver.TenantResolver` closes the "does a record
know which tenant it belongs to" half by validating a per-tenant bearer
credential an agent presents
(`enterprise-auth -create-ingest-credential-tenant=<id>` mints one; only
its SHA-256 hash is ever stored) against a `POST
/internal/authorize-ingest` endpoint, and attaching the resolved tenant
ID to every record as a `tenant_id` Kafka message header before
producing it — fail-closed: once a resolver is configured, a missing or
invalid credential refuses the whole batch, never falls back to "no
tenant."

**ClickHouse**: `enterprise/cmd/enterprise-ingest` (a second binary,
mirroring `enterprise-api`) consumes that header: it reuses
`ingest/consumer`'s own flush loop with `enterprise/internal/
chwriter.Registry` — a per-tenant `*clickhousewriter.Writer` registry,
built the same way `chrunner`'s per-tenant connections are — swapped in
as the writer, so a tagged batch's records are grouped by tenant and
each group INSERTed through that tenant's own ClickHouse connection,
fail-closed on an untagged or unprovisioned tenant. Building it surfaced
a real gap in an already-shipped control: `tenantprovision.
ProvisionClickHouse`'s grant was `SELECT`-only (correct for the
read-side credential `chrunner` uses, but `chwriter` reuses the same
credential for writes) — every real per-tenant write would have failed
with a permission error until this was widened to `SELECT, INSERT`.
`Registry.StartRefreshing` (new) re-lists active tenants every minute
and reconciles the writer map — opens a connection for a newly-active
tenant, closes and removes one no longer active — closing what was
originally a startup-only snapshot with no refresh at all. **Not yet
confirmed against a real ClickHouse**, same caveat as the read-side
chrunner claim above — the Docker-free fail-closed and refresh-error
tests pass, the live-database tests (including the two new ones proving
refresh's add/remove reconciliation against real connections) are
written but skip-gated, see `/docs/phase-4-runbook.md`.

**Tantivy**: `search/src/consumer.rs` now resolves each record's
`tenant_id` header through the *same* `IndexRegistry` the read side
already used (`search/src/registry.rs`), and writes there instead of
always into the default index — genuinely verified in this environment,
same as the read-side Tantivy claim above, since Tantivy is an embedded
library with no Docker dependency. No "second binary" was needed here,
unlike ClickHouse: Tantivy has no grant system to gate a
separately-credentialed binary behind, so `IndexRegistry` already lived
directly in this AGPL-core `search` binary, and read/write simply share
it. (This reasoning predates Phase 6's relicensing of `enterprise/` to
AGPLv3 and originally referred to a commercially-licensed credential --
restated here because the architectural point holds independent of
licensing: Tantivy still has no grant system, so the split was never
about which license `enterprise/` carried.)
This write path is now also active-tenant-gated:
`search/src/tenants.rs`'s `ActiveTenantTracker` polls a new
`GET /internal/active-tenants` endpoint on `enterprise-auth` every 60
seconds (RoleService-credentialed, the same auth shape `alerting` uses
against `api`) and `consumer.rs` refuses any tagged record whose tenant
isn't in the polled allowlist — fail-closed on the first fetch (startup
blocks until it succeeds), last-known-good on any later refresh failure.
Off unless `ENTERPRISE_AUTH_URL`/`ENTERPRISE_AUTH_SERVICE_TOKEN` are
both set. Genuinely verified here too: real HTTP round trips (the Bearer
header actually sent, JSON parsing, both fail-closed paths) against a
hand-rolled TCP test server, no live enterprise-auth needed since
`reqwest` doesn't care that the other end is real.

**Both engines share one open question now** — deployment topology,
covered above — and no longer differ in active-tenant staleness bound:
`chwriter.Registry.StartRefreshing` re-lists active tenants every
minute, matching `ActiveTenantTracker`'s interval, opening a connection
for a newly-active tenant and closing/removing one no longer active.
Neither is a live per-write check (that would mean a database/HTTP
round trip on every record, a real throughput cost neither
implementation accepts), so a roughly one-minute staleness window
remains on both sides by design, not a gap unique to either engine
anymore. A newly-provisioned tenant's ClickHouse database and Tantivy
index are both now real, isolated, and actually populated by
write-routed agent traffic (the ClickHouse claim pending live
confirmation, the Tantivy claim already verified).

## System overview

```
Browser ──▶ web (SvelteKit, static)
              │
              ▼
Browser ──▶ api OR enterprise-api ──▶ ClickHouse (log data, SQL path)
              │                    └─▶ search (gRPC) ──▶ Tantivy (log data, full-text path)
              └─▶ Postgres (control plane: dashboards, alert_rules,
                             tenants, users, tenant_memberships, audit_log)

# api: one shared ClickHouse connection, one shared (default) Tantivy
# index via api/searchclient, nil AuditLogger -- Phase 0-3 behavior.
# enterprise-api: enterprise/internal/chrunner (per-tenant ClickHouse
# connections) + enterprise/internal/searchclient (per-tenant Tantivy
# index, via search's SearchRequest.tenant_id) + enterprise/internal/
# audit.QueryAPILogger (real audit writes) wired into the SAME
# api/queryapi.Handler/api/dashboards.Handler core -- see this
# document's "Read this first" section. Either binary can be running;
# nothing forces the isolated one.

alerting ──▶ api or enterprise-api (POST /query, RoleService credential)
alerting ──▶ Postgres (rulestore, notifystore)

api/alerting ──▶ enterprise-auth (POST /internal/authorize, HTTP only —
                                    no Go import edge, see "Module
                                    boundary" below)

Browser ──▶ enterprise-auth (GET /auth/oidc/login, /auth/oidc/callback)
              └─▶ external IdP (OIDC authorization code flow)
              └─▶ Postgres (rbacstore: users, tenant_memberships)

cairnobsctl ──▶ api, alerting (Bearer token when CAIRNOBSCTL_TOKEN is set)
```

Ingest path (agent → Redpanda → ingest → ClickHouse, and Redpanda →
search → Tantivy): `ingest` resolves and tags each record with a real
tenant ID (see "Read this first" above). `enterprise/cmd/enterprise-ingest`
consumes that tag and routes ClickHouse writes to each tenant's own
database. `search/src/consumer.rs` (Tantivy's independent Redpanda
consumer) consumes the same tag and routes each record into its own
tenant's index. Both write paths' one remaining gap — no live
active-tenant recheck (ClickHouse: a startup-time snapshot; Tantivy: no
allowlist at all) — is named in "Read this first" above, not separately
designed in `/docs/phase-4-isolation-design.md`; named here as a gap that
design doc doesn't yet cover, not just an implementation gap.

## Module boundary (trust boundary #1)

`enterprise/` (SSO, RBAC storage, audit logging, session issuance — AGPLv3,
same as core as of Phase 6, see `/docs/compliance/license-audit-report.md`)
is never imported by core (`/api`, `/alerting`, `/web`, `/cli`) —
enforced in CI by `hack/check-tenant-boundary.sh`, which greps for the
import edge on every build. This is an architectural trust boundary, not
a licensing one: it keeps core buildable and deployable with zero
multi-tenant mechanism present regardless of what license either side
carries, and keeps tenant identity resolution server-side rather than
trusting a request parameter. Core calls
`enterprise-auth` over plain HTTP (`api/authz.HTTPAuthorizer`),
forwarding only the `Cookie`/`Authorization` headers, never the full
request (`api/authz/httpauthz_test.go` asserts this — an
unrelated header like `X-Forwarded-For` is never forwarded). This means
core's authorization decision is only as trustworthy as the network path
to `enterprise-auth` — see "Deployment/network assumptions" below.

## Authentication

**Implemented for both OIDC and SAML, to the same verification bar.**
`enterprise/internal/loginhandler` serves `GET /auth/oidc/login`
(redirects to the configured IdP, with a short-lived HttpOnly cookie
carrying CSRF-protection state) and `GET /auth/oidc/callback`
(validates state, exchanges the code, verifies the ID token via
`enterprise/internal/oidc`'s real `coreos/go-oidc` wiring, upserts a
`users` row keyed by SSO subject, resolves tenant/role from
`tenant_memberships`, and issues a `session.Manager`-signed session
cookie), plus the SAML equivalent, `GET /auth/saml/login` (redirects to
the configured IdP via `enterprise/internal/saml`'s
`ServiceProvider.LoginURL`, persisting the AuthnRequest ID in a
short-lived cookie — SAML's replay/unsolicited-response defense,
standing in for OIDC's `state`) and `POST /auth/saml/acs` (validates the
assertion's signature and `InResponseTo` against that cookie via
`ServiceProvider.ParseResponse`, then converges on the same
upsert/resolve/issue-session path OIDC uses). Both are verified
end-to-end with real cryptography, not mocked: OIDC's tests spin up a
real fake IdP (`coreos/go-oidc`'s own `oidctest` package) that signs
genuine RS256 ID tokens; SAML's tests spin up a real fake IdP
(`crewjam/saml/samlidp`) that builds and signs genuine SAML assertions
and XML-signs the response, exercising the same `ServiceProvider.
ParseResponse` signature-verification path production uses. Every test
in `loginhandler_test.go` and `saml_test.go` passes, including the full
login→callback/ACS→session-cookie round trip for both protocols, and
negative-path tests for each (state/`InResponseTo` mismatch, missing/
expired credential, missing required claim, no/multiple tenant
memberships). Writing the SAML test caught two real bugs in
`enterprise/internal/saml`'s `ParseResponse`, both fixed before this
verification was considered complete: it never called `r.ParseForm()`
before reading the POSTed `SAMLResponse` field (every real ACS POST
would have decoded an empty response), and its email-attribute matching
missed `urn:oid:0.9.2342.19200300.100.1.3` (the standard LDAP "mail"
OID) — what an IdP sends by default when the SP hasn't explicitly
requested an attribute literally named "email", which is exactly what
`samlidp`'s own default assertion builder does. **Not yet verified for
either protocol**: wiring this into a running `enterprise-auth`
container against a *real* external IdP (Google/Okta/etc.) — that needs
real IdP credentials and a reachable callback/ACS URL neither of which
this environment has; see `/docs/phase-4-runbook.md`.

A user with zero `tenant_memberships` rows is refused outright (403).
More than one no longer guesses or refuses: `finishLogin` issues a
short-lived `session.Manager` "pending login" token (a distinct Go/JWT
type from a real session, with its own disjoint claim name so a real
session token can't double as one — a real bug this design's own test
suite caught before it shipped, see `session.PendingLoginClaims`'s doc
comment) and redirects to `web/src/routes/select-tenant`, backed by two
endpoints (`GET /auth/memberships`, `POST /auth/select-tenant`) that
list the identity's real tenant options and, on selection, re-derive the
role for the chosen tenant server-side (never trusting a client-supplied
role) before issuing the real session. Both the *backend protocol* and
the *frontend page* that calls it are now built. The backend is verified
with the same real-fake-IdP tests as the rest of `internal/loginhandler`.
The frontend needed a second CORS posture — `httpserver.
WithCredentialedCORS`, a literal origin plus
`Access-Control-Allow-Credentials: true`, since a credentialed `fetch`
and a wildcard `Access-Control-Allow-Origin` can never be combined, so
this couldn't reuse `enterprise-api`'s wildcard-friendly `WithCORS` — and
is genuinely verified in a real browser in this environment (not just
type-checked): the full cross-origin pending-login-cookie round trip, a
real click choosing a tenant, the post-selection redirect, and the
missing/expired-cookie error path, all driven against a throwaway server
standing in for `enterprise-auth`'s exact wire contract. `GET
/auth/features` (`enterprise/internal/authhandler`) reports whether
OIDC/SAML are *configured*, for `/web`'s settings page to conditionally
render — independent of whether a login button actually exists yet in
the UI (it doesn't yet; a user still has to be sent to
`/auth/oidc/login`/`/auth/saml/login` by some means other than clicking
something in `web`, since no page links there).

**Implemented for the one machine caller.** `/alerting`'s evaluator is
the sole service-to-service caller (`POST /query`, to evaluate rule
conditions across tenants). It presents a long-lived, signed
(HS256/JWT) `RoleService` credential, minted offline via
`enterprise-auth -mint-service-token=alerting` (an operator action, not
a network-reachable endpoint) and configured via `API_SERVICE_TOKEN`.
`enterprise/internal/session.Manager` issues and validates this token;
`enterprise/internal/authhandler`'s `POST /internal/authorize` resolves
it. `RoleService` is a distinct, non-comparable lane on the `Role` type
(`api/authz.Role.Satisfies`) — a service credential can never
satisfy a human-role check and vice versa, verified by exhaustive
table-driven tests (`api/authz/authz_test.go`).

**Session/token integrity.** Tokens are HS256-signed JWTs with a single
shared signing key (`ENTERPRISE_SESSION_SIGNING_KEY`, ≥32 bytes,
required at `enterprise-auth` startup). Compromise of this key lets an
attacker forge any identity, including `RoleService` — it is the single
highest-value secret in the enterprise deployment and should be treated
accordingly (a real KMS/secrets-manager-backed value, not the
`docker-compose.yml`/Helm chart's dev-only literal). Token validation
(`enterprise/internal/session.Manager.Validate`) collapses every failure
mode — bad signature, malformed token, expired — into one
`ErrInvalidToken`, deliberately not distinguishing "expired" from
"forged" so a caller can't be tempted to treat either as a softer case.

## Authorization (RBAC)

**Live and enforced.** `POST /query` and every `/dashboards` endpoint in
`api` require a minimum role, resolved per-request via
`api/authz.RequireRole`/`RequireRoleOrService` calling
`enterprise-auth`. Roles: Viewer < Editor < Admin < Owner, plus the
separate `RoleService` lane above. `GET /dashboards` is Viewer+;
create/update/delete require Editor+ (`api/dashboards/
handler.go`). A nil `Authorizer` (no `ENTERPRISE_AUTH_URL` configured)
is a deliberate no-op, matching Phase 0-3's no-auth behavior — this is
correct default-open-for-single-tenant behavior, not an oversight, but
means an operator who forgets to set `ENTERPRISE_AUTH_URL` in a
multi-tenant deployment gets *no* enforcement at all, silently. Worth a
deployment-time check a real rollout should add (not built here).

**Now enforced:** the RBAC matrix's `(own/granted)` qualifier for
Editor-level dashboard actions. `dashboard_permissions`
(`metadata/migrations/0024_create_dashboard_permissions.sql`, tightened
by `0033_restrict_dashboard_permissions_role.sql`) is read via
`api/dashboards.PermissionStore` — a core-defined interface, same shape
as `queryapi.AuditLogger` — implemented by
`enterprise/internal/rbacstore.DashboardPermissions` and wired in only
by `enterprise/cmd/enterprise-api`. A plain Editor may now only
edit/delete a dashboard (or its panels) they created, or one where a
grant raises their effective role to Editor; Admin/Owner still act on
any dashboard in their tenant. Managing grants themselves
(`PUT`/`DELETE /dashboards/{id}/permissions/{userId}`) is deliberately
stricter than editing content — only the creator or Admin/Owner may
grant or revoke, never a user who can edit *only* because of a grant
(closes a self-escalation path a looser check would allow). Verified by
`api/dashboards/handler_test.go`'s fake-store tests (the ownership/
grant/admin matrix, plus the granted-editor-cannot-manage-grants
regression case) — real integration tests against a live Postgres exist
in `enterprise/internal/rbacstore/rbacstore_test.go` but, like the rest
of this phase's rbacstore work, have not been run against one in this
environment. A plain `api/cmd/api` deployment with RBAC enforcement on
but no enterprise permission service wired still enforces ownership/
Admin — only the "granted" bonus and grant management are unavailable
there (nil `PermissionStore` is a documented no-op, same shape as a nil
`Authorizer`).

**Application-layer tenant scoping (dashboards only).** Every
`dashboards` store query filters `WHERE tenant_id = $identity.TenantID`
(`api/dashboards/store.go`), and the handler resolves that
tenant ID from the RBAC-authenticated identity's context
(`authz.IdentityFromContext`), **never** from a client-supplied request
field. This closes a real gap found during this document's own review:
`Dashboard.TenantID` is a JSON-tagged, client-settable field
(`api/dashboards/types.go`), and the original handler/store
implementation trusted it directly on create/update and applied no
`tenant_id` filter at all on list/get/update/delete — meaning any
authenticated user could read, modify, or delete any other tenant's
dashboards simply by supplying (or guessing) their UUID, or spoof
`tenant_id` on create/import to write into a tenant they don't belong
to. Fixed as part of this task, with regression tests proving
cross-tenant access now returns 404 (not 403, which would itself leak
that the ID exists under a different tenant) —
`api/dashboards/handler_test.go`'s
`TestCrossTenant*`/`TestCreateDashboardIgnoresClientSuppliedTenantID`/
`TestImportIgnoresExportedTenantID`. **This same class of bug should be
assumed present anywhere else client-supplied identifiers cross a tenant
boundary until proven otherwise by an adversarial test** — see task 8's
adversarial test suite for what's been checked so far and what hasn't.

**Query-path tenant scoping: none** — see the top of this document.
RBAC's role check on `POST /query` answers "is this identity allowed to
run *a* query," not "does this query's result set respect tenant
boundaries" — it can't, because the executor has no tenant concept to
enforce.

## Audit logging

**Live**, and independently verified against a real Postgres (not just
written) — `enterprise/internal/audit`'s integration tests. Two
independent defenses back "no update/delete path from the application
layer":

1. A dedicated `audit_writer` Postgres role with only `INSERT`+`SELECT`
   grants (`metadata/migrations/0012-0014`), via its **own**
   `pgxpool.Pool` — never the shared `cairnobs` role/pool every other
   store uses.
2. A `BEFORE UPDATE OR DELETE ... RAISE EXCEPTION` trigger
   (`metadata/migrations/0015-0016`) that rejects the operation for
   *any* role, including the table owner — confirmed live: even the
   `cairnobs` role cannot `UPDATE` a row without first disabling the
   trigger, a privileged operation distinct from ordinary application
   access.

**Tamper detection, not tamper prevention against a privileged
attacker.** Rows are hash-chained (`prev_hash`/`row_hash =
SHA256(prev_hash || canonical_fields)`, serialized under
`pg_advisory_xact_lock` so concurrent writers can't fork the chain —
verified with a 20-goroutine concurrency test against live Postgres).
The chain alone only proves internal self-consistency: a Postgres
superuser (or anyone who compromises that credential) can wipe
`audit_log` and regenerate a perfectly self-consistent new chain from
row 1. `enterprise/internal/audit.Checkpointer` periodically ships a
rolling hash to an external `CheckpointSink` for exactly this reason —
`FileSink` (the only implementation built so far) is explicitly
documented as a dev/testing stand-in, **not** a real external-anchoring
guarantee (it writes to a local file the same privileged attacker could
also reach). A real deployment needs a genuine `CheckpointSink`
(S3 with Object Lock, or equivalent, reachable by a credential the
database administrator doesn't also hold) before the "prove nothing was
altered after the fact" claim actually holds against a privileged
insider.

**Fail-open by design for routine queries.** `queryapi.Handler.logAudit`
(`api/queryapi/handler.go`) logs a write failure and otherwise
ignores it — an audit-log outage does not take down the query path. This
is a deliberate availability-over-completeness tradeoff: it means a
brief audit outage produces an under-logged (not over-blocked) window.
No privileged/administrative action (role change, SSO config change,
notification-target secret reveal) currently exists to enforce
fail-closed on, since none of those flows are built yet
(`enterprise/internal/rbacstore` has no HTTP handlers) — when they are,
they should fail closed per `/docs/phase-4-isolation-design.md`'s
original policy, and that policy is not yet exercised by any real code
path.

**What's logged:** query text, language, row count, duration,
success/error — not result contents. `Source`/`EventType` fields exist
(`SourceAPI`/`SourceWeb`/`SourceCLI`/`SourceAlerting`,
`EventQuery`/`EventRoleChange`/`EventGrantChange`/
`EventSSOConfigChange`/`EventSecretReveal`) but only `EventQuery` from
`SourceAPI` is actually wired to a call site
(`queryapi.Handler.logAudit`) — the others are typed placeholders for
work not yet built (there's no role-change/grant-change/SSO-config
handler to call them from).

## Known residual risks (explicitly out of scope, not silently assumed away)

Per `/PROJECT-SPEC.md`'s Phase 4 non-goals, restated here in threat-model
terms:

- **A privileged ClickHouse/Postgres administrator is not defended
  against.** Every isolation and audit-integrity guarantee in this
  document is a structural defense against *application-layer* bugs and
  injection — not against someone holding database superuser
  credentials. That's an operational control (credential custody,
  infrastructure access review), out of scope for this system's own
  code.
- **`system.query_log` metadata leakage — now closed and verified live**
  against the pinned version (`clickhouse/clickhouse-server:24.8`), with
  a real, non-obvious wrinkle: a freshly created tenant user was **not**
  default-denied from `system.*` the way the original design assumed
  (`ProvisionClickHouse` now issues an explicit `REVOKE SELECT ON
  system.* FROM <user>` — see that function's doc comment). Confirming
  this live also surfaced a genuine ClickHouse behavioral split the
  design doc didn't anticipate: `system.query_log` is a real
  access-checked table (the REVOKE makes it hard-deny,
  `ACCESS_DENIED`), but `system.tables` is a filtered *catalog* view
  that ClickHouse 24.8 never denies outright regardless of grants — it
  just silently returns zero rows for a properly-revoked user. Both
  outcomes close the actual leak (no other tenant's query text or
  database/table names are visible either way);
  `tenantprovision_test.go`'s `TestProvisionedUserCannotReadSystemTables`
  was corrected to assert what each table actually does (hard error for
  `query_log`, verified-empty-and-no-foreign-database-names for
  `tables`) rather than demanding a hard error from both. Still
  contingent on the deployment-shape caveat at the top of this document:
  this only holds when `enterprise-api` (not plain `api`) is actually
  serving traffic.
- **No deny-override grants** — `dashboard_permissions` is additive-only
  by design; a full allow/deny ACL system is unbuilt, future work.
- **No data retention/deletion policy** for a deprovisioned tenant —
  the `tenants.status` state machine includes `deprovisioning`, but what
  actually happens to that tenant's ClickHouse/Tantivy/Postgres data is
  an unanswered compliance question, not a designed-and-deferred one.
- **No general multi-cluster orchestration** — `/deploy`'s Helm
  chart/Operator (`/deploy/README.md`) proves the K8s-side per-tenant
  secret-management model, not a fully general multi-cluster system.
  Now applied to a real (local `kind`) live cluster in this environment
  — see that README's verification section — but only a single-cluster,
  single-node test, not a real multi-cluster topology.

## Deployment/network assumptions

- `enterprise-auth`'s `/internal/authorize` and `/auth/features`
  endpoints have no authentication of their own beyond the credentials
  they're validating — they must be reachable only from inside the
  cluster/trusted network (`api`/`alerting`/`web`), never exposed
  publicly. Nothing in this codebase enforces that at the network layer;
  it's a deployment responsibility (NetworkPolicy, or equivalent) not
  yet codified in `/deploy/helm/cairnobs`.
- `ENTERPRISE_SESSION_SIGNING_KEY`, ClickHouse/Postgres passwords, and
  (once minted) the `alerting` service token are all K8s `Secret`
  objects in the Helm chart (`/deploy/helm/cairnobs/templates/
  secrets.yaml`) — standard K8s `Secret` semantics apply (base64, not
  encrypted at rest without a cluster-level `EncryptionConfiguration`).
  No secrets-manager integration (Vault, cloud KMS) exists; the chart
  documents this as an operator decision, not something it enforces.

## Summary: what's actually enforced today

| Control | Status |
|---|---|
| Role-based access control on `/query`, `/dashboards` | **Enforced** |
| `alerting`↔`api` service-identity credential | **Enforced** |
| Tenant scoping on dashboards (control-plane data) | **Enforced** |
| ClickHouse per-tenant provisioning (`tenantprovision`) | **Enforced, verified live** against `clickhouse/clickhouse-server:24.8` — real `CREATE DATABASE`/`CREATE USER`/`GRANT`/`REVOKE`, real two-tenant cross-read probe |
| ClickHouse query routing (`chrunner`) | **Enforced, verified live** — real per-tenant connections, cross-tenant raw-SQL probe passes; only applies when `enterprise-api` serves traffic, not plain `api` |
| `system.*` ClickHouse metadata isolation | **Enforced, verified live** — `system.query_log` hard-denies, `system.tables` returns zero foreign rows (see "Known residual risks" below for the ClickHouse-version-specific split between the two) |
| Tantivy per-tenant index routing (`search/src/registry.rs`) | **Enforced, verified live** — real Tantivy indices, real cross-tenant probe, all passing |
| Tantivy tenant_id resolution (`enterprise/internal/searchclient`) | **Enforced, verified live** — real gRPC wire-level test |
| Ingest tenant *identity* (credential validation, tagging) | **Built and tested** — fail-closed `TenantResolver`, `tenant_id` Kafka header attached per record |
| Ingest tenant *write-routing*, ClickHouse | **Enforced, verified live** — `enterprise-ingest`/`chwriter.Registry` route each tagged batch to its tenant's own database, fail-closed on an untagged/unprovisioned tenant; both Docker-free and live-ClickHouse tests pass. Active-tenant snapshot refreshes every minute (`Registry.StartRefreshing`) — a deprovisioned tenant loses write access within a minute, not "until the next restart" |
| Ingest tenant *write-routing*, Tantivy | **Built and genuinely verified** — `search/src/consumer.rs` routes each record into its own tenant's index via `IndexRegistry`, same registry the (already-verified) read side uses; no Docker needed, real tests pass. Active-tenant-gated too: `tenants::ActiveTenantTracker` polls `enterprise-auth` every 60s (off unless configured), refusing any tenant not in the polled allowlist — same one-minute staleness bound as ClickHouse's now-refreshing snapshot, no more asymmetry between the two |
| Deployment actually routing traffic to `enterprise-api` (Helm) | **Enforced, verified live** — `api`/`enterprise-api` are mutually exclusive, same flag as RBAC/audit/SSO; a real `helm install` against a real `kind` cluster confirmed the `cairnobs-api` Deployment runs `cairnobs-enterprise-api:latest` with `enterprise.enabled=true`, real endpoints behind the `cairnobs-api` Service, not just `helm template`'s rendered YAML |
| Deployment actually routing traffic to `enterprise-api` (docker-compose) | **Enforced, verified live** — `api`/`enterprise-api` are mutually exclusive via `COMPOSE_PROFILES`, same flag choice as Helm's `enterprise.enabled`; a real `docker compose up` of `enterprise-api` was run in this environment (and caught/fixed a startup-crashing duplicate `GET /healthz` route registration bug in the process), not just `docker compose config` |
| Human SSO login — OIDC | **Enforced, verified live** — real login against a real Auth0 developer tenant, full browser round trip; correctly failed closed on an identity with no `tenant_memberships` row, then succeeded and issued a real session after `-grant-membership-*`, with `POST /internal/authorize` returning exactly the granted tenant/role |
| Human SSO login — SAML | **Enforced, verified live** — real login against Auth0's SAML2 Web App addon acting as a real SAML IdP, over real (self-signed, dev-only) TLS; a real signed assertion validated (audience, destination, signature), landed on `/select-tenant` with real memberships, and `POST /internal/authorize` confirmed the selected tenant/role. Found and fixed a real bug in the process: `loginhandler.go`'s cookies decided `Secure` from `r.TLS != nil` alone, which is wrong behind any TLS-terminating reverse proxy (the deployment shape this handler actually runs in) — `enterprise-auth` never terminates TLS itself, so `r.TLS` was nil even over a genuinely HTTPS connection, silently dropping `Secure` and breaking SAML's `SameSite=None` cookie |
| Multi-tenant-membership login (tenant picker) | **Enforced, verified live** — a real Auth0 identity with two real tenant memberships (`acme` Admin, `globex` Viewer) landed on the real `/select-tenant` page against the real `enterprise-auth` container, rendered both with correct display names/roles via a real credentialed cross-origin `GET /auth/memberships`, and selecting either one issued a session that `POST /internal/authorize` confirmed matched — the selection genuinely determines the issued session's tenant, not just renders correctly. This pass also found and fixed a real bug: `web/Dockerfile` never declared `ARG`/`ENV` for `VITE_ALERTING_API_BASE_URL`/`VITE_ENTERPRISE_AUTH_BASE_URL`, so `docker-compose.yml`'s build args for them were silently dropped, leaving `enterpriseAuthBase` `undefined` in the built bundle |
| Per-resource dashboard grants (`own/granted`) | **Enforced, verified live** — real Postgres integration tests for `dashboard_permissions` CRUD and the `PermissionStore` adapter all pass (only when `enterprise-api` serves traffic — plain `api` falls back to own/Admin only) |
| Query audit logging (routine queries) | **Enforced**, fail-open, and now wired to a real writer via `enterprise-api` (`audit.QueryAPILogger`) |
| Audit log tamper detection (hash chain) | **Enforced**, verified live |
| Audit log tamper prevention (external anchoring) | **Design only** — `FileSink` is a dev stand-in |
| Mid-provisioning-race handling (evaluator ticks against a not-yet-active tenant) | **Closed on both storage engines** — see `api/queryapi/tenant_isolation_gap_test.go`; ClickHouse verified Docker-free (structural, not just tested), Tantivy fixed and verified Docker-free after finding it was a real gap, not just an unverified assumption |
| Protection against a privileged DB administrator | **Explicit non-goal** |