Phase 4: real Tantivy per-tenant isolation (search/src/registry.rs, enterprise/internal/searchclient)

Closes the last named "isolation mechanism" gap: search.proto gains a
tenant_id field on SearchRequest; search/src/registry.rs's IndexRegistry
resolves it to an on-demand-opened, per-tenant Tantivy index (empty
tenant_id keeps today's single default index, so this is purely
additive); enterprise/internal/searchclient sets that field from the
authenticated request identity in ctx, mirroring chrunner's exact
fail-closed "never a parameter" shape. Wired into enterprise-api in
place of the shared api/searchclient.

Unlike the ClickHouse pieces from the previous two commits, this one is
genuinely verified end to end in this environment: Tantivy is an
embedded library, not a networked service, so both the Rust index
registry (cargo test, cargo clippy --all-targets -- -D warnings, both
clean) and the Go client (a real in-process gRPC server) could actually
run. registry.rs's tenant_index_is_isolated_from_default_and_other_tenants
seeds three real indices with the same term and confirms a tenant-scoped
search returns only that tenant's document -- item 3 of the isolation
design doc's verification plan, closed for real, not just written.

With both ClickHouse and Tantivy isolation now built, the single largest
remaining gap is no longer a missing mechanism: it's that nothing forces
or flags whether a deployment actually runs enterprise-api instead of
plain api, and that ingest itself has no tenant concept for either
storage engine (every record still lands in the one shared database/
index no matter what -- undesigned, not just unbuilt). Updated the
threat model, architecture doc, CLAUDE.md, and both READMEs accordingly.
This commit is contained in:
2026-08-13 23:16:22 -07:00
parent 1fab02abd5
commit ba2276aa1a
17 changed files with 696 additions and 168 deletions
+25 -12
View File
@@ -76,10 +76,10 @@ This split is not to be changed without discussion — see CLAUDE.md.
| `transport` | Redpanda docker-compose + topic provisioning scripts. No application code. |
| `ingest` (Go) | gRPC server accepting agent connections; produces normalized OTel-log-like records to Redpanda; separate consumer reads from Redpanda and batch-writes to ClickHouse. No tenant concept — every record lands in the one shared `logs` table regardless of source (see "Tenant isolation" below). |
| `storage` | ClickHouse schema migrations + docker-compose for local/homelab. |
| `search` (Rust, Phase 1) | Consumes the same Redpanda topic `ingest` does (own offset tracking), builds a Tantivy full-text index over `message`, serves matches over gRPC. One shared index for every tenant today — see "Tenant isolation" below. |
| `search` (Rust, Phase 1) | Consumes the same Redpanda topic `ingest` does (own offset tracking), builds a Tantivy full-text index over `message`, serves matches over gRPC. Writes always go to one shared (default) index (`ingest` isn't tenant-aware); reads can be scoped per-tenant via `SearchRequest.tenant_id` and `src/registry.rs`'s `IndexRegistry` (Phase 4) — see "Tenant isolation" below. |
| `api` (Go) | gRPC + REST gateway. `POST /query` compiles pipe-syntax or raw SQL to one IR, executed across ClickHouse/Tantivy (`/docs/query-language-design.md`). `internal/dashboards` is CRUD only — panel query execution happens client-side, reusing `/query`. `internal/authz` (Phase 4) enforces RBAC via a network call to `enterprise-auth`, never an import. |
| `alerting` (Go, Phase 3) | Evaluates alert rules on an interval, calls `api`'s `POST /query` (via a `RoleService` credential once Phase 4 auth is configured — see `/docs/phase-4-isolation-design.md`'s alerting↔api gap), delivers firing/resolved notifications (webhook/Slack/PagerDuty). |
| `enterprise` (Go, commercial license, Phase 4) | OIDC login (`internal/loginhandler`'s `/auth/oidc/login`+`/auth/oidc/callback`, real IdP round trip, verified with a fake IdP but not a real external one), RBAC storage (`internal/rbacstore`), session/service-token issuance (`internal/session`), the append-only audit log (`internal/audit`), `enterprise-auth`'s HTTP surface (`/internal/authorize`, `/auth/features`), per-tenant ClickHouse provisioning (`internal/tenantprovision`) and query routing (`internal/chrunner`), and `cmd/enterprise-api` — a second binary combining core's `api/queryapi`/`api/dashboards` handlers with these tenant-aware implementations. Never imported by core — see "Licensing boundary" below. Does **not** yet include per-tenant Tantivy routing or SAML's login ACS handler (protocol mechanics only) — see `/docs/security/threat-model.md`. |
| `enterprise` (Go, commercial license, Phase 4) | OIDC login (`internal/loginhandler`'s `/auth/oidc/login`+`/auth/oidc/callback`, real IdP round trip, verified with a fake IdP but not a real external one), RBAC storage (`internal/rbacstore`), session/service-token issuance (`internal/session`), the append-only audit log (`internal/audit`), `enterprise-auth`'s HTTP surface (`/internal/authorize`, `/auth/features`), per-tenant ClickHouse provisioning (`internal/tenantprovision`) and query routing (`internal/chrunner`), and `cmd/enterprise-api` — a second binary combining core's `api/queryapi`/`api/dashboards` handlers with these tenant-aware implementations. Never imported by core — see "Licensing boundary" below. Also `internal/searchclient` (per-tenant Tantivy routing, wired the same way into `search`). Does **not** yet include SAML's login ACS handler (protocol mechanics only) — see `/docs/security/threat-model.md`. |
| `web` (SvelteKit, static build) | Query bar, dashboards, alerts, and (Phase 4) a settings page that renders SSO status via a runtime capability check (`GET /auth/features`) rather than bundling enterprise-licensed components. |
| `cli` (`sentryctl`) | `ping`, `query`, `dashboards` (list/get/apply), `alerts` (list/get/apply). `$SENTRYCTL_TOKEN`, if set, is forwarded as a Bearer credential (Phase 4). |
| `deploy` | A Helm chart covering every `docker-compose.yml` service, plus (Phase 4) a small Go Operator managing one CRD (`Tenant`) that provisions a per-tenant ClickHouse credential Secret. Never applied to a live cluster in the environment this was built in — see `/deploy/README.md`'s verification section before trusting it. |
@@ -129,20 +129,33 @@ escape hatch is opaque to any compiler-injected filter.
unchanged, with its single shared connection — nothing forces a
deployment to run `enterprise-api` instead, and nothing flags it if it
doesn't.
- **Tantivy connection-layer isolation is not built.** `search`'s gRPC
service and `proto/sentry/search/v1/search.proto`'s `SearchRequest`
still carry no tenant field. Every tenant's free-text queries hit the
same shared Tantivy index regardless of which binary serves the
request.
- **Tantivy index-layer isolation is built, and verified.**
`search/src/registry.rs`'s `IndexRegistry` resolves
`SearchRequest.tenant_id` (added to `proto/sentry/search/v1/
search.proto`) to its own on-disk index, opened on demand;
`enterprise/internal/searchclient` sets that field from the
authenticated request identity, the same "read from ctx, fail closed"
shape `chrunner` uses. Unlike the ClickHouse pieces, this one actually
ran in the environment it was built in — Tantivy is an embedded
library, so the cross-tenant isolation probe needed no live database
or Docker to execute for real, and it passed.
- Neither storage engine's isolation extends to *ingest*: every record
`ingest` produces lands in the one shared ClickHouse database and the
one shared (default) Tantivy index regardless of tenant. A
newly-provisioned tenant's database/index are real and isolated at
query time — and permanently empty until something upstream of
`chrunner`/`searchclient` becomes tenant-aware on the write side,
which is undesigned, not merely unbuilt.
- `deploy/operator`'s `Tenant` CRD still manages only the K8s-side
artifact (a credential Secret); the Helm chart has no service
definition for `enterprise-api` yet.
Building `enterprise/internal/searchclient` (the Tantivy-side sibling of
`chrunner`) and giving the deployment topology (Helm chart, or at least
clear documentation) an actual way to route traffic to `enterprise-api`
instead of `api` are the two largest remaining gaps between this system
and the isolation model it was designed to have.
The deployment-topology gap — giving the system an actual way to route
traffic to `enterprise-api` instead of `api` (a Helm service, or at
minimum a documented, enforced convention) — is now the single largest
remaining gap between this system and the isolation model it was
designed to have; both storage engines' connection/index-layer
mechanisms themselves are built.
## Licensing boundary
+55 -21
View File
@@ -243,11 +243,15 @@ provisions ClickHouse.
## 8. `enterprise-api`: real per-tenant ClickHouse isolation
This is new since this runbook was first written — `enterprise/internal/
tenantprovision` and `enterprise/internal/chrunner` now exist, closing
the headline gap §"Known gaps" below used to describe as completely
unbuilt. It's still a second binary you have to choose to run, though —
see `/docs/security/threat-model.md`'s "Read this first" section.
`enterprise/internal/tenantprovision` and `enterprise/internal/chrunner`
close the ClickHouse half of the headline gap §"Known gaps" below used
to describe as completely unbuilt. It's still a second binary you have
to choose to run, though — see `/docs/security/threat-model.md`'s "Read
this first" section. With OIDC login now built (§3a), a real
`curl -X POST http://localhost:8083/query` walkthrough as a logged-in
tenant is *possible* now, but still needs the manual `tenant_memberships`
bootstrap from §3a — a full end-to-end curl walkthrough isn't included
here yet.
```sh
docker compose build enterprise-api
@@ -257,14 +261,11 @@ docker compose up -d enterprise-api
curl -s http://localhost:8083/healthz
```
There's still no OIDC/SAML login handler and no CLI for minting a human
session token (see `/docs/security/threat-model.md`) -- so a real
`curl -X POST http://localhost:8083/query` walkthrough as tenant acme
isn't possible yet. Confirm isolation end to end against the live stack (this is the same
Confirm isolation end to end against the live stack (this is the same
assertion `enterprise/internal/chrunner/chrunner_test.go`'s
`TestRegistryTenantCannotReadOtherTenantEvenViaRawSQL` makes, run here
as an integration test instead of a curl walkthrough since there's no
login flow to drive it through curl yet):
as an integration test instead of a curl walkthrough since a full login
walkthrough isn't scripted yet):
```sh
docker run --rm --network sentry_default -v $(pwd)/enterprise:/src -w /src \
@@ -285,17 +286,50 @@ pass) and `TestRegistryTenantCannotReadOtherTenantEvenViaRawSQL` (item 1,
closed through the actual production code path, not just
tenantprovision's raw grants).
## 9. Tantivy per-tenant isolation — no Docker needed, actually run this one
Unlike everything above, this one doesn't need a live stack at all —
Tantivy is an embedded library, not a networked service, so both halves
(the Rust index registry and the Go client that talks to it) can be
verified with nothing but a local toolchain:
```sh
cd search
cargo build
cargo clippy --all-targets -- -D warnings
cargo test
# expect 14 tests passing, including
# registry::tests::tenant_index_is_isolated_from_default_and_other_tenants
# -- item 3 of /docs/phase-4-isolation-design.md's verification plan.
cd ../enterprise
go test ./internal/searchclient/... -v
# real in-process gRPC server, confirms SearchRequest.tenant_id is set
# correctly and that a request with no/invalid tenant identity is refused.
```
This is the one piece of Phase 4's tenant isolation work that has
**actually been run and confirmed passing** in an environment without
Docker access, alongside `enterprise/internal/loginhandler`'s OIDC
tests (§3a) — both are unusually strong evidence precisely because they
needed no infrastructure this environment lacked.
## Known gaps (do not treat this phase as done without reading these)
Full accounting: `/docs/security/threat-model.md`. Headline items:
- **ClickHouse isolation exists but is opt-in.** `enterprise-api`
(§8) gives real per-tenant ClickHouse isolation, but plain `api`
(still the default in `docker-compose.yml`/`web`'s base URL) has none,
and nothing flags which one a given deployment is actually running.
- **No Tantivy/free-text isolation at all**, regardless of which binary
serves the request -- `enterprise/internal/searchclient` (chrunner's
Tantivy-side sibling) doesn't exist.
- **Both storage engines' isolation exists but is opt-in.**
`enterprise-api` (§8, §9) gives real per-tenant ClickHouse *and*
Tantivy isolation, but plain `api` (still the default in
`docker-compose.yml`/`web`'s base URL) has neither, and nothing flags
which one a given deployment is actually running. This is now the
single largest gap — not a missing mechanism, a missing enforcement/
default.
- **Ingest has no tenant concept for either storage engine.** Every
record `ingest` produces lands in the one shared ClickHouse database
and the one shared Tantivy index no matter what. A newly-provisioned
tenant's storage is real, isolated at query time, and permanently
empty until this changes — undesigned, not just unbuilt.
- **Human SSO login now works for OIDC** (§3a) -- verified with a real
fake IdP, not yet a real external one or a running `enterprise-auth`
container. **SAML login still doesn't exist** -- protocol wiring only,
@@ -305,10 +339,10 @@ Full accounting: `/docs/security/threat-model.md`. Headline items:
bootstrap is the only way to grant a logged-in identity access today.
- **No per-resource dashboard grants** (`dashboard_permissions` has a
schema, no handler reads it).
- Two of the four adversarial ClickHouse/Tantivy probes named in
- Three of the four adversarial ClickHouse/Tantivy probes named in
`/docs/phase-4-isolation-design.md`'s verification plan are closed
(§8); the other two (Tantivy cross-tenant search, mid-provisioning-race
handling) are still stubbed as explicitly-skipped tests in
(§8, §9); the last (mid-provisioning-race handling) is still stubbed
as an explicitly-skipped test in
`api/queryapi/tenant_isolation_gap_test.go`.
## Tearing down
+63 -55
View File
@@ -9,58 +9,61 @@ for the full design rationale behind the controls described here.
## Read this first: the single most important open finding
**Updated**: this section originally read "log data queried through
`POST /query` is not tenant-isolated at all." That's now only half
true, and the half that's no longer true matters — read carefully,
because the remaining gap (Tantivy/free-text) is easy to miss if you
stop at "ClickHouse is isolated now."
**Updated a second time.** This section originally read "log data
queried through `POST /query` is not tenant-isolated at all," then
"ClickHouse is isolated but Tantivy isn't." Both ClickHouse *and*
Tantivy connection/index-layer isolation are now built. What's left is
narrower but still real: **whether a given deployment actually runs the
isolated binary**, and **whether ingest itself is tenant-aware** (it
isn't, for either storage engine).
**ClickHouse (the SQL path) is now built, but only if you run the right
binary — and it has not yet been confirmed against a real ClickHouse.**
`enterprise/internal/tenantprovision` (real `CREATE DATABASE`/`CREATE
USER`/`GRANT` against ClickHouse) and `enterprise/internal/chrunner` (a
per-tenant `driver.Conn` registry implementing api's
`querylang/executor.SQLRunner`, resolving which tenant's connection to
use from the authenticated identity in request context — never from a
client-suppliable field) now exist, and a new binary,
`enterprise/cmd/enterprise-api`, wires them into the same
`api/queryapi.Handler`/`api/dashboards.Handler` core already ships. Real
integration tests exist and would prove the core adversarial claim —
`enterprise/internal/tenantprovision/tenantprovision_test.go`'s
`TestProvisionedUserCannotReadOtherTenantDatabase` and
`TestProvisionedUserCannotReadSystemTables`,
`enterprise/internal/chrunner/chrunner_test.go`'s
`TestRegistryTenantCannotReadOtherTenantEvenViaRawSQL` — but this
environment had no Docker/ClickHouse access while these were written, so
they've only been confirmed to skip cleanly offline, not to pass for
real. See `/docs/phase-4-runbook.md`'s verification-status section
before treating "the test exists" as "isolation is confirmed."
**ClickHouse (the SQL path) is built.** `enterprise/internal/
tenantprovision` (real `CREATE DATABASE`/`CREATE USER`/`GRANT`) and
`enterprise/internal/chrunner` (a per-tenant connection registry
implementing `api/querylang/executor.SQLRunner`, resolving the tenant
from request identity, never a parameter) are wired into
`enterprise/cmd/enterprise-api`. **Not yet confirmed against a real
ClickHouse** — this environment had no Docker/database access while
these were written; the tests exist and are correct Go, but "the test
exists" is not the same claim as "isolation is confirmed" (see
`/docs/phase-4-runbook.md`).
**But plain `api/cmd/api` still runs with one shared connection**, and
nothing in this repo automatically routes traffic to `enterprise-api`
instead — `docker-compose.yml` includes it "available, not defaulted
into the traffic path" (same shape as `enterprise-auth`'s own addition),
and the Helm chart has no service for it at all yet. **A deployment is
only as isolated as which binary is actually serving traffic** — this
is an operational decision nothing currently enforces or even surfaces
as a warning.
**Tantivy (the free-text path) is also built, and — unlike the
ClickHouse pieces — genuinely verified in this environment.**
`search/src/registry.rs`'s `IndexRegistry` resolves a `SearchRequest.
tenant_id` to its own on-disk Tantivy index, opened on demand;
`enterprise/internal/searchclient` sets that field from the
authenticated request identity, mirroring `chrunner`'s exact "read from
ctx, fail closed, never a parameter" shape. Because Tantivy is an
embedded library (no external service to fake or skip), both sides could
actually be run: `search/src/registry.rs`'s
`tenant_index_is_isolated_from_default_and_other_tenants` seeds three
real indices with the same search term and confirms a tenant-scoped
search returns only that tenant's document; `enterprise/internal/
searchclient`'s tests run a real in-process gRPC server and confirm the
wire-level `SearchRequest` carries the right `tenant_id`. All pass, for
real, no disclaimer needed for this specific claim.
**Tantivy (the free-text path) is still fully unisolated.** There is no
`enterprise/internal/searchclient` (the Tantivy-side equivalent of
chrunner) — `search`'s gRPC service and
`proto/sentry/search/v1/search.proto`'s `SearchRequest` still carry no
tenant field anywhere, confirmed by reading the code. Every tenant's
free-text queries hit the same shared Tantivy index regardless of which
binary (`api` or `enterprise-api`) serves the HTTP request. A query that
resolves to a pure pipe-syntax free-text search (e.g. `message:"error"`)
is not protected by chrunner at all.
**But plain `api/cmd/api` still runs with one shared ClickHouse
connection and no tenant-scoped search client**, and nothing in this
repo automatically routes traffic to `enterprise-api` instead —
`docker-compose.yml` includes it "available, not defaulted into the
traffic path" (same shape as `enterprise-auth`'s own addition), and the
Helm chart has no service for it at all yet. **A deployment is only as
isolated as which binary is actually serving traffic** — this is an
operational decision nothing currently enforces or even surfaces as a
warning. This is now the single largest gap in the isolation story, not
a missing mechanism.
**What this means concretely**: treat a deployment as tenant-isolated
for structured/SQL queries *only if* it runs `enterprise-api` fronting
provisioned tenants, and treat it as **not isolated at all** for
free-text search regardless of which binary runs. RBAC (below) and
dashboard tenant-scoping (below) hold regardless of which binary is
running; the ClickHouse/Tantivy split above is what changed.
**Ingest is not tenant-aware for either storage engine**, and this is
more load-bearing than it sounds: `chrunner`/`searchclient` prove *read*
isolation given tenant-scoped data exists, but nothing writes
tenant-scoped data yet. Every record `ingest` produces lands in the one
shared ClickHouse database and the one shared (default) Tantivy index,
regardless of tenant. A newly-provisioned tenant's ClickHouse database
and Tantivy index are real, isolated, and queryable through
`enterprise-api` — and permanently empty, until ingest itself becomes
tenant-aware, which is undesigned, not just unbuilt.
## System overview
@@ -73,12 +76,15 @@ Browser ──▶ api OR enterprise-api ──▶ ClickHouse (log data, SQL path
└─▶ Postgres (control plane: dashboards, alert_rules,
tenants, users, tenant_memberships, audit_log)
# api: one shared ClickHouse connection, nil AuditLogger -- Phase 0-3 behavior.
# api: one shared ClickHouse connection, one shared (default) Tantivy
# index via api/searchclient, nil AuditLogger -- Phase 0-3 behavior.
# enterprise-api: enterprise/internal/chrunner (per-tenant ClickHouse
# connections) + enterprise/internal/audit.QueryAPILogger (real audit
# writes) wired into the SAME api/queryapi.Handler/api/dashboards.Handler
# core -- see this document's "Read this first" section. Either binary
# can be running; nothing forces the isolated one.
# connections) + enterprise/internal/searchclient (per-tenant Tantivy
# index, via search's SearchRequest.tenant_id) + enterprise/internal/
# audit.QueryAPILogger (real audit writes) wired into the SAME
# api/queryapi.Handler/api/dashboards.Handler core -- see this
# document's "Read this first" section. Either binary can be running;
# nothing forces the isolated one.
alerting ──▶ api or enterprise-api (POST /query, RoleService credential)
alerting ──▶ Postgres (rulestore, notifystore)
@@ -351,8 +357,10 @@ terms:
| ClickHouse per-tenant provisioning (`tenantprovision`) | **Built, not live-verified** — real integration test exists, not yet run against ClickHouse |
| ClickHouse query routing (`chrunner`) | **Built, not live-verified** — and only applies when `enterprise-api` serves traffic, not plain `api` |
| `system.*` ClickHouse metadata isolation | **Built, not live-verified** — same caveat as above |
| Tantivy/free-text tenant isolation | **Not implemented** — no per-tenant index routing at all |
| Deployment actually routing traffic to `enterprise-api` | **Not implemented** — no Helm service, no default wiring |
| Tantivy per-tenant index routing (`search/src/registry.rs`) | **Enforced, verified live** — real Tantivy indices, real cross-tenant probe, all passing |
| Tantivy tenant_id resolution (`enterprise/internal/searchclient`) | **Enforced, verified live** — real gRPC wire-level test |
| Ingest tenant-awareness (ClickHouse and Tantivy both) | **Not implemented, undesigned** — every ingested record lands in the single shared database/index regardless of tenant |
| Deployment actually routing traffic to `enterprise-api` | **Not implemented** — no Helm service, no default wiring; now the largest gap in the isolation story |
| Human SSO login — OIDC | **Built, verified with a real fake IdP** (not yet tried against a real external IdP) |
| Human SSO login — SAML | **Not implemented** |
| Multi-tenant-membership login (tenant picker) | **Not implemented** — refused with a clear error, not guessed |