cairnobs/docs/security/threat-model.md

Sentry Threat Model (Phase 4)

Written for a prospective enterprise customer's security team, describing the system as actually built through Phase 4 task 7 — not the target architecture. Where a control is designed but not yet implemented, this document says so explicitly, with a pointer to the tracking doc/task. See /docs/phase-4-isolation-design.md and /docs/phase-4-rbac-design.md for the full design rationale behind the controls described here.

Read this first: the single most important open finding

Updated a third time. This section originally read "log data queried through POST /query is not tenant-isolated at all," then "ClickHouse is isolated but Tantivy isn't," then "ingest tags records with a tenant identity but nothing routes the write." Both ClickHouse and Tantivy connection/index-layer isolation are built on the read path, and ClickHouse write-routing is now built too. What's left is narrower but still real: whether a given deployment actually runs the isolated binaries, and Tantivy's write path, which still has no per-tenant routing at all.

ClickHouse (the SQL path) is built. enterprise/internal/ tenantprovision (real CREATE DATABASE/CREATE USER/GRANT) and enterprise/internal/chrunner (a per-tenant connection registry implementing api/querylang/executor.SQLRunner, resolving the tenant from request identity, never a parameter) are wired into enterprise/cmd/enterprise-api. Not yet confirmed against a real ClickHouse — this environment had no Docker/database access while these were written; the tests exist and are correct Go, but "the test exists" is not the same claim as "isolation is confirmed" (see /docs/phase-4-runbook.md).

Tantivy (the free-text path) is also built, and — unlike the ClickHouse pieces — genuinely verified in this environment. search/src/registry.rs's IndexRegistry resolves a SearchRequest. tenant_id to its own on-disk Tantivy index, opened on demand; enterprise/internal/searchclient sets that field from the authenticated request identity, mirroring chrunner's exact "read from ctx, fail closed, never a parameter" shape. Because Tantivy is an embedded library (no external service to fake or skip), both sides could actually be run: search/src/registry.rs's tenant_index_is_isolated_from_default_and_other_tenants seeds three real indices with the same search term and confirms a tenant-scoped search returns only that tenant's document; enterprise/internal/ searchclient's tests run a real in-process gRPC server and confirm the wire-level SearchRequest carries the right tenant_id. All pass, for real, no disclaimer needed for this specific claim.

Both Helm and docker-compose now close this. deploy/helm/sentry/templates/api.yaml and enterprise-api.yaml are mutually exclusive, gated on opposite sides of the same enterprise.enabled flag, rendering to the same Service name/port — so a Helm-deployed cluster runs exactly one of the two binaries, chosen by the same flag that turns on RBAC/audit/SSO, not a second independently-forgettable decision. Verified by parsing (not eyeballing) the rendered YAML under both values: exactly one sentry-api Deployment either way, with the right image. docker-compose.yml's api/enterprise-api services are now the analogous mutually-exclusive choice, gated behind COMPOSE_PROFILES (.env checks in single-tenant, i.e. plain api, as the zero-config default) and sharing the same host port/network-alias trick to stay transparent to alerting/web either way — verified via docker compose config (renders and validates the merged YAML without a daemon; confirms api/enterprise-api never both appear in --services output for the same profile selection) — see /docs/phase-4-runbook.md §10a. This still only constrains deployment, not operation: nothing stops an operator from manually running plain api's image against a cluster (or compose project) that has tenants provisioned, pointing at the same ClickHouse/Postgres. The Helm chart makes the default, chart-managed path correct; it isn't a runtime guard against misconfiguration.

Ingest now has a real tenant identity, and ClickHouse write-routing is built — Tantivy write-routing is the one gap left. chrunner/ searchclient prove read isolation given tenant-scoped data exists; an optional ingest/internal/grpcserver.TenantResolver closes the "does a record know which tenant it belongs to" half by validating a per-tenant bearer credential an agent presents (enterprise-auth -create-ingest-credential-tenant=<id> mints one; only its SHA-256 hash is ever stored) against a POST /internal/authorize-ingest endpoint, and attaching the resolved tenant ID to every record as a tenant_id Kafka message header before producing it — fail-closed: once a resolver is configured, a missing or invalid credential refuses the whole batch, never falls back to "no tenant." enterprise/cmd/enterprise-ingest (a second binary, mirroring enterprise-api) now consumes that header: it reuses ingest/consumer's own flush loop with enterprise/internal/chwriter.Registry — a per-tenant *clickhousewriter.Writer registry, built the same way chrunner's per-tenant connections are — swapped in as the writer, so a tagged batch's records are grouped by tenant and each group INSERTed through that tenant's own ClickHouse connection, fail-closed on an untagged or unprovisioned tenant. Building it surfaced a real gap in an already-shipped control: tenantprovision.ProvisionClickHouse's grant was SELECT-only (correct for the read-side credential chrunner uses, but chwriter reuses the same credential for writes) — every real per-tenant write would have failed with a permission error until this was widened to SELECT, INSERT. Not yet confirmed against a real ClickHouse, same caveat as the read-side chrunner claim above — the Docker-free fail-closed tests pass, the live-database tests are written but skip-gated, see /docs/phase-4-runbook.md. What's still missing: search's independent Redpanda consumer does not read the tenant_id header at all — every ingested record still lands in the one shared (default) Tantivy index regardless of tenant. A newly-provisioned tenant's ClickHouse database is now real, isolated, and actually populated by write-routed agent traffic (once confirmed against a real cluster); its Tantivy index remains real, isolated, and queryable through enterprise-api — and permanently empty, until Tantivy's own write-routing split is built, which is now scoped, disclosed remaining work, not an undesigned gap.

System overview

Browser ──▶ web (SvelteKit, static)
              │
              ▼
Browser ──▶ api OR enterprise-api ──▶ ClickHouse (log data, SQL path)
              │                    └─▶ search (gRPC) ──▶ Tantivy (log data, full-text path)
              └─▶ Postgres (control plane: dashboards, alert_rules,
                             tenants, users, tenant_memberships, audit_log)

# api: one shared ClickHouse connection, one shared (default) Tantivy
# index via api/searchclient, nil AuditLogger -- Phase 0-3 behavior.
# enterprise-api: enterprise/internal/chrunner (per-tenant ClickHouse
# connections) + enterprise/internal/searchclient (per-tenant Tantivy
# index, via search's SearchRequest.tenant_id) + enterprise/internal/
# audit.QueryAPILogger (real audit writes) wired into the SAME
# api/queryapi.Handler/api/dashboards.Handler core -- see this
# document's "Read this first" section. Either binary can be running;
# nothing forces the isolated one.

alerting ──▶ api or enterprise-api (POST /query, RoleService credential)
alerting ──▶ Postgres (rulestore, notifystore)

api/alerting ──▶ enterprise-auth (POST /internal/authorize, HTTP only —
                                    no Go import edge, see "Module
                                    boundary" below)

Browser ──▶ enterprise-auth (GET /auth/oidc/login, /auth/oidc/callback)
              └─▶ external IdP (OIDC authorization code flow)
              └─▶ Postgres (rbacstore: users, tenant_memberships)

sentryctl ──▶ api, alerting (Bearer token when SENTRYCTL_TOKEN is set)

Ingest path (agent → Redpanda → ingest → ClickHouse, and Redpanda → search → Tantivy): ingest resolves and tags each record with a real tenant ID (see "Read this first" above). enterprise/cmd/enterprise-ingest now consumes that tag and routes ClickHouse writes to each tenant's own database. Tantivy's independent Redpanda consumer (search/src/ consumer.rs) still does not consume the tag at all — every record still lands in the one shared Tantivy index regardless of tenant. That narrower gap is still out of scope for what's built so far and is not separately designed in /docs/phase-4-isolation-design.md; named here as a gap that design doc doesn't yet cover, not just an implementation gap.

Module boundary (trust boundary #1)

enterprise/ (commercial license: SSO, RBAC storage, audit logging, session issuance) is never imported by AGPL core (/api, /alerting, /web, /cli) — enforced in CI by hack/check-tenant-boundary.sh, which greps for the import edge on every build. Core calls enterprise-auth over plain HTTP (api/authz.HTTPAuthorizer), forwarding only the Cookie/Authorization headers, never the full request (api/authz/httpauthz_test.go asserts this — an unrelated header like X-Forwarded-For is never forwarded). This means core's authorization decision is only as trustworthy as the network path to enterprise-auth — see "Deployment/network assumptions" below.

Authentication

Implemented for both OIDC and SAML, to the same verification bar. enterprise/internal/loginhandler serves GET /auth/oidc/login (redirects to the configured IdP, with a short-lived HttpOnly cookie carrying CSRF-protection state) and GET /auth/oidc/callback (validates state, exchanges the code, verifies the ID token via enterprise/internal/oidc's real coreos/go-oidc wiring, upserts a users row keyed by SSO subject, resolves tenant/role from tenant_memberships, and issues a session.Manager-signed session cookie), plus the SAML equivalent, GET /auth/saml/login (redirects to the configured IdP via enterprise/internal/saml's ServiceProvider.LoginURL, persisting the AuthnRequest ID in a short-lived cookie — SAML's replay/unsolicited-response defense, standing in for OIDC's state) and POST /auth/saml/acs (validates the assertion's signature and InResponseTo against that cookie via ServiceProvider.ParseResponse, then converges on the same upsert/resolve/issue-session path OIDC uses). Both are verified end-to-end with real cryptography, not mocked: OIDC's tests spin up a real fake IdP (coreos/go-oidc's own oidctest package) that signs genuine RS256 ID tokens; SAML's tests spin up a real fake IdP (crewjam/saml/samlidp) that builds and signs genuine SAML assertions and XML-signs the response, exercising the same ServiceProvider. ParseResponse signature-verification path production uses. Every test in loginhandler_test.go and saml_test.go passes, including the full login→callback/ACS→session-cookie round trip for both protocols, and negative-path tests for each (state/InResponseTo mismatch, missing/ expired credential, missing required claim, no/multiple tenant memberships). Writing the SAML test caught two real bugs in enterprise/internal/saml's ParseResponse, both fixed before this verification was considered complete: it never called r.ParseForm() before reading the POSTed SAMLResponse field (every real ACS POST would have decoded an empty response), and its email-attribute matching missed urn:oid:0.9.2342.19200300.100.1.3 (the standard LDAP "mail" OID) — what an IdP sends by default when the SP hasn't explicitly requested an attribute literally named "email", which is exactly what samlidp's own default assertion builder does. Not yet verified for either protocol: wiring this into a running enterprise-auth container against a real external IdP (Google/Okta/etc.) — that needs real IdP credentials and a reachable callback/ACS URL neither of which this environment has; see /docs/phase-4-runbook.md.

A user with zero tenant_memberships rows is refused outright (403). More than one no longer guesses or refuses: finishLogin issues a short-lived session.Manager "pending login" token (a distinct Go/JWT type from a real session, with its own disjoint claim name so a real session token can't double as one — a real bug this design's own test suite caught before it shipped, see session.PendingLoginClaims's doc comment) and redirects to a not-yet-served URL instead, backed by two new endpoints (GET /auth/memberships, POST /auth/select-tenant) that list the identity's real tenant options and, on selection, re-derive the role for the chosen tenant server-side (never trusting a client-supplied role) before issuing the real session. This is the backend protocol for tenant selection, verified with the same real-fake-IdP tests as the rest of internal/loginhandler — the frontend page that would call it doesn't exist (web has no session/cookie-handling code at all today, and enterprise-auth has no CORS middleware for a cross-origin fetch with credentials to work), both real, separately-scoped gaps, not silently approximated. GET /auth/features (enterprise/internal/authhandler) reports whether OIDC/SAML are configured, for /web's settings page to conditionally render — independent of whether a login button actually exists yet in the UI (it doesn't; only the HTTP endpoints do).

Implemented for the one machine caller. /alerting's evaluator is the sole service-to-service caller (POST /query, to evaluate rule conditions across tenants). It presents a long-lived, signed (HS256/JWT) RoleService credential, minted offline via enterprise-auth -mint-service-token=alerting (an operator action, not a network-reachable endpoint) and configured via API_SERVICE_TOKEN. enterprise/internal/session.Manager issues and validates this token; enterprise/internal/authhandler's POST /internal/authorize resolves it. RoleService is a distinct, non-comparable lane on the Role type (api/authz.Role.Satisfies) — a service credential can never satisfy a human-role check and vice versa, verified by exhaustive table-driven tests (api/authz/authz_test.go).

Session/token integrity. Tokens are HS256-signed JWTs with a single shared signing key (ENTERPRISE_SESSION_SIGNING_KEY, ≥32 bytes, required at enterprise-auth startup). Compromise of this key lets an attacker forge any identity, including RoleService — it is the single highest-value secret in the enterprise deployment and should be treated accordingly (a real KMS/secrets-manager-backed value, not the docker-compose.yml/Helm chart's dev-only literal). Token validation (enterprise/internal/session.Manager.Validate) collapses every failure mode — bad signature, malformed token, expired — into one ErrInvalidToken, deliberately not distinguishing "expired" from "forged" so a caller can't be tempted to treat either as a softer case.

Authorization (RBAC)

Live and enforced. POST /query and every /dashboards endpoint in api require a minimum role, resolved per-request via api/authz.RequireRole/RequireRoleOrService calling enterprise-auth. Roles: Viewer < Editor < Admin < Owner, plus the separate RoleService lane above. GET /dashboards is Viewer+; create/update/delete require Editor+ (api/dashboards/ handler.go). A nil Authorizer (no ENTERPRISE_AUTH_URL configured) is a deliberate no-op, matching Phase 0-3's no-auth behavior — this is correct default-open-for-single-tenant behavior, not an oversight, but means an operator who forgets to set ENTERPRISE_AUTH_URL in a multi-tenant deployment gets no enforcement at all, silently. Worth a deployment-time check a real rollout should add (not built here).

Now enforced: the RBAC matrix's (own/granted) qualifier for Editor-level dashboard actions. dashboard_permissions (metadata/migrations/0024_create_dashboard_permissions.sql, tightened by 0033_restrict_dashboard_permissions_role.sql) is read via api/dashboards.PermissionStore — a core-defined interface, same shape as queryapi.AuditLogger — implemented by enterprise/internal/rbacstore.DashboardPermissions and wired in only by enterprise/cmd/enterprise-api. A plain Editor may now only edit/delete a dashboard (or its panels) they created, or one where a grant raises their effective role to Editor; Admin/Owner still act on any dashboard in their tenant. Managing grants themselves (PUT/DELETE /dashboards/{id}/permissions/{userId}) is deliberately stricter than editing content — only the creator or Admin/Owner may grant or revoke, never a user who can edit only because of a grant (closes a self-escalation path a looser check would allow). Verified by api/dashboards/handler_test.go's fake-store tests (the ownership/ grant/admin matrix, plus the granted-editor-cannot-manage-grants regression case) — real integration tests against a live Postgres exist in enterprise/internal/rbacstore/rbacstore_test.go but, like the rest of this phase's rbacstore work, have not been run against one in this environment. A plain api/cmd/api deployment with RBAC enforcement on but no enterprise permission service wired still enforces ownership/ Admin — only the "granted" bonus and grant management are unavailable there (nil PermissionStore is a documented no-op, same shape as a nil Authorizer).

Application-layer tenant scoping (dashboards only). Every dashboards store query filters WHERE tenant_id = $identity.TenantID (api/dashboards/store.go), and the handler resolves that tenant ID from the RBAC-authenticated identity's context (authz.IdentityFromContext), never from a client-supplied request field. This closes a real gap found during this document's own review: Dashboard.TenantID is a JSON-tagged, client-settable field (api/dashboards/types.go), and the original handler/store implementation trusted it directly on create/update and applied no tenant_id filter at all on list/get/update/delete — meaning any authenticated user could read, modify, or delete any other tenant's dashboards simply by supplying (or guessing) their UUID, or spoof tenant_id on create/import to write into a tenant they don't belong to. Fixed as part of this task, with regression tests proving cross-tenant access now returns 404 (not 403, which would itself leak that the ID exists under a different tenant) — api/dashboards/handler_test.go's TestCrossTenant*/TestCreateDashboardIgnoresClientSuppliedTenantID/ TestImportIgnoresExportedTenantID. This same class of bug should be assumed present anywhere else client-supplied identifiers cross a tenant boundary until proven otherwise by an adversarial test — see task 8's adversarial test suite for what's been checked so far and what hasn't.

Query-path tenant scoping: none — see the top of this document. RBAC's role check on POST /query answers "is this identity allowed to run a query," not "does this query's result set respect tenant boundaries" — it can't, because the executor has no tenant concept to enforce.

Audit logging

Live, and independently verified against a real Postgres (not just written) — enterprise/internal/audit's integration tests. Two independent defenses back "no update/delete path from the application layer":

1. A dedicated audit_writer Postgres role with only INSERT+SELECT grants (metadata/migrations/0012-0014), via its own pgxpool.Pool — never the shared sentry role/pool every other store uses.
2. A BEFORE UPDATE OR DELETE ... RAISE EXCEPTION trigger (metadata/migrations/0015-0016) that rejects the operation for any role, including the table owner — confirmed live: even the sentry role cannot UPDATE a row without first disabling the trigger, a privileged operation distinct from ordinary application access.

Tamper detection, not tamper prevention against a privileged attacker. Rows are hash-chained (prev_hash/row_hash = SHA256(prev_hash || canonical_fields), serialized under pg_advisory_xact_lock so concurrent writers can't fork the chain — verified with a 20-goroutine concurrency test against live Postgres). The chain alone only proves internal self-consistency: a Postgres superuser (or anyone who compromises that credential) can wipe audit_log and regenerate a perfectly self-consistent new chain from row 1. enterprise/internal/audit.Checkpointer periodically ships a rolling hash to an external CheckpointSink for exactly this reason — FileSink (the only implementation built so far) is explicitly documented as a dev/testing stand-in, not a real external-anchoring guarantee (it writes to a local file the same privileged attacker could also reach). A real deployment needs a genuine CheckpointSink (S3 with Object Lock, or equivalent, reachable by a credential the database administrator doesn't also hold) before the "prove nothing was altered after the fact" claim actually holds against a privileged insider.

Fail-open by design for routine queries. queryapi.Handler.logAudit (api/queryapi/handler.go) logs a write failure and otherwise ignores it — an audit-log outage does not take down the query path. This is a deliberate availability-over-completeness tradeoff: it means a brief audit outage produces an under-logged (not over-blocked) window. No privileged/administrative action (role change, SSO config change, notification-target secret reveal) currently exists to enforce fail-closed on, since none of those flows are built yet (enterprise/internal/rbacstore has no HTTP handlers) — when they are, they should fail closed per /docs/phase-4-isolation-design.md's original policy, and that policy is not yet exercised by any real code path.

What's logged: query text, language, row count, duration, success/error — not result contents. Source/EventType fields exist (SourceAPI/SourceWeb/SourceCLI/SourceAlerting, EventQuery/EventRoleChange/EventGrantChange/ EventSSOConfigChange/EventSecretReveal) but only EventQuery from SourceAPI is actually wired to a call site (queryapi.Handler.logAudit) — the others are typed placeholders for work not yet built (there's no role-change/grant-change/SSO-config handler to call them from).

Known residual risks (explicitly out of scope, not silently assumed away)

Per /CLAUDE.md's Phase 4 non-goals, restated here in threat-model terms:

• A privileged ClickHouse/Postgres administrator is not defended against. Every isolation and audit-integrity guarantee in this document is a structural defense against application-layer bugs and injection — not against someone holding database superuser credentials. That's an operational control (credential custody, infrastructure access review), out of scope for this system's own code.
• system.query_log metadata leakage — per-tenant users are now real, but the check itself hasn't run yet. Was an open verification item because there were no per-tenant ClickHouse users to check against; that blocker is gone (enterprise/internal/tenantprovision exists), and tenantprovision_test.go's TestProvisionedUserCannotReadSystemTables asserts exactly what the design calls for (system.query_log/system.tables inaccessible, SHOW DATABASES not revealing other tenants) — but this environment never had ClickHouse access to actually run it, so it remains unconfirmed against the pinned version (clickhouse/clickhouse-server:24.8) until someone with Docker access runs it (/docs/phase-4-runbook.md §8). Also still contingent on the deployment-shape caveat at the top of this document: even once confirmed, this only holds when enterprise-api (not plain api) is actually serving traffic.
• No deny-override grants — dashboard_permissions is additive-only by design; a full allow/deny ACL system is unbuilt, future work.
• No data retention/deletion policy for a deprovisioned tenant — the tenants.status state machine includes deprovisioning, but what actually happens to that tenant's ClickHouse/Tantivy/Postgres data is an unanswered compliance question, not a designed-and-deferred one.
• No general multi-cluster orchestration — /deploy's Helm chart/Operator (/deploy/README.md) proves the K8s-side per-tenant secret-management model, not a fully general multi-cluster system, and was never applied to a live cluster in this environment (see that README's verification section).

Deployment/network assumptions

• enterprise-auth's /internal/authorize and /auth/features endpoints have no authentication of their own beyond the credentials they're validating — they must be reachable only from inside the cluster/trusted network (api/alerting/web), never exposed publicly. Nothing in this codebase enforces that at the network layer; it's a deployment responsibility (NetworkPolicy, or equivalent) not yet codified in /deploy/helm/sentry.
• ENTERPRISE_SESSION_SIGNING_KEY, ClickHouse/Postgres passwords, and (once minted) the alerting service token are all K8s Secret objects in the Helm chart (/deploy/helm/sentry/templates/ secrets.yaml) — standard K8s Secret semantics apply (base64, not encrypted at rest without a cluster-level EncryptionConfiguration). No secrets-manager integration (Vault, cloud KMS) exists; the chart documents this as an operator decision, not something it enforces.

Summary: what's actually enforced today

Control	Status
Role-based access control on /query, /dashboards	Enforced
alerting↔api service-identity credential	Enforced
Tenant scoping on dashboards (control-plane data)	Enforced
ClickHouse per-tenant provisioning (tenantprovision)	Built, not live-verified — real integration test exists, not yet run against ClickHouse
ClickHouse query routing (chrunner)	Built, not live-verified — and only applies when enterprise-api serves traffic, not plain api
system.* ClickHouse metadata isolation	Built, not live-verified — same caveat as above
Tantivy per-tenant index routing (search/src/registry.rs)	Enforced, verified live — real Tantivy indices, real cross-tenant probe, all passing
Tantivy tenant_id resolution (enterprise/internal/searchclient)	Enforced, verified live — real gRPC wire-level test
Ingest tenant identity (credential validation, tagging)	Built and tested — fail-closed TenantResolver, tenant_id Kafka header attached per record
Ingest tenant write-routing, ClickHouse	Built, not yet confirmed against a real ClickHouse — enterprise-ingest/chwriter.Registry route each tagged batch to its tenant's own database, fail-closed on an untagged/unprovisioned tenant; Docker-free tests pass, live-database tests are skip-gated
Ingest tenant write-routing, Tantivy	Not implemented, now scoped — every record still lands in the single shared Tantivy index regardless of tenant; search/src/consumer.rs consuming the tenant_id header to route the write is real, disclosed remaining work
Deployment actually routing traffic to enterprise-api (Helm)	Enforced — api/enterprise-api are mutually exclusive, same flag as RBAC/audit/SSO
Deployment actually routing traffic to enterprise-api (docker-compose)	Enforced — api/enterprise-api are mutually exclusive via COMPOSE_PROFILES, same flag choice as Helm's enterprise.enabled; verified via docker compose config, not an actual docker compose up in this environment
Human SSO login — OIDC	Built, verified with a real fake IdP (not yet tried against a real external IdP)
Human SSO login — SAML	Built, verified with a real fake IdP (not yet tried against a real external IdP)
Multi-tenant-membership login (tenant picker)	Backend protocol built and verified (GET /auth/memberships, POST /auth/select-tenant, a pending-login token distinct from a real session) — no frontend page calls it yet
Per-resource dashboard grants (own/granted)	Built, unit-tested against a fake store; live-Postgres integration tests written, not run in this environment (only when enterprise-api serves traffic — plain api falls back to own/Admin only)
Query audit logging (routine queries)	Enforced, fail-open, and now wired to a real writer via enterprise-api (audit.QueryAPILogger)
Audit log tamper detection (hash chain)	Enforced, verified live
Audit log tamper prevention (external anchoring)	Design only — FileSink is a dev stand-in
Mid-provisioning-race handling (evaluator ticks against a not-yet-active tenant)	Closed on both storage engines — see api/queryapi/tenant_isolation_gap_test.go; ClickHouse verified Docker-free (structural, not just tested), Tantivy fixed and verified Docker-free after finding it was a real gap, not just an unverified assumption
Protection against a privileged DB administrator	Explicit non-goal