Close the last tenant-isolation adversarial probe (mid-provisioning tenants)
Phase 4 task 8's verification plan named four adversarial probes; three were closed earlier this phase, the fourth (an evaluator tick, or any other caller, hitting a tenant that exists but hasn't reached the active+credentialed gate yet -- must be refused, not served) was still an explicitly-skipped stub in api/queryapi/tenant_isolation_gap_test.go. Investigating it found the two storage engines needed genuinely different treatment: - ClickHouse (enterprise/internal/chrunner) already had this property structurally, for free: Registry is built once at startup from rbacstore.ListProvisionedDataSources, which already filters to active+credentialed tenants only, so a mid-provisioning tenant is simply absent from the connection map. New test TestRegistryRefusesMidProvisioningTenant proves this without Docker -- an empty DataSource list never dials ClickHouse, so this genuinely runs in this environment, unlike every other test in that file. - Tantivy (search/src/registry.rs's IndexRegistry) was a real, different gap, not just an unverified assumption: it opens-or-creates an index for any syntactically-valid tenant_id on first request, because it's a separate process with no Postgres access and structurally can't know which tenants are actually provisioned. A query against a mid-provisioning tenant would have silently succeeded with zero results from a freshly-created empty index -- "ambient success" indistinguishable from "no matching logs," exactly the failure mode this item was worried about. Fixed the Tantivy gap with a new enterprise/internal/searchclient. TenantChecker interface (backed by a new rbacstore.TenantIsActive, implemented structurally, no new import edge needed), consulted before every gRPC call: Client.Search now refuses a non-active tenant before it ever reaches `search`. Dial's signature gained a required TenantChecker parameter; enterprise-api's main.go passes its existing rbacstore.Store (already satisfies the interface). Verified Docker-free via searchclient's existing real-in-process-gRPC-server test harness (TestSearchRefusesMidProvisioningTenant, plus TestSearchPropagatesTenantCheckerError for the fail-closed-on-error case) -- both genuinely run in this environment, same bar as the rest of the Tantivy isolation work. rbacstore.TenantIsActive itself has two new skip-gated live-Postgres tests (TestTenantIsActive, TestTenantIsActiveNonexistentTenant) -- disclosed as not run against a live database here, same gap as the rest of this phase's Postgres-backed pieces. api/queryapi/tenant_isolation_gap_test.go rewritten from a checklist with one skipped stub to a full accounting of all four now-closed probes. Docs updated in lockstep: CLAUDE.md, threat-model.md, phase-4-isolation-design.md (implementation note added after its original sign-off), phase-4-runbook.md (§9), enterprise/README.md.
This commit is contained in:
@@ -315,3 +315,25 @@ suite must include, against the live stack:
|
||||
- A simulated evaluator tick firing mid-provisioning (tenant row exists,
|
||||
grants not yet confirmed) to confirm it's refused, not silently served
|
||||
against a partially-provisioned or default-profile connection.
|
||||
|
||||
**Implementation note (Phase 4 task 8, added after this design was
|
||||
signed off):** closed for both storage engines, via
|
||||
`api/queryapi/tenant_isolation_gap_test.go`'s pointers. ClickHouse's
|
||||
refusal turned out to be structural, not something that needed new
|
||||
code: `chrunner.Registry` is built once at startup from
|
||||
`rbacstore.ListProvisionedDataSources`, which already excludes
|
||||
anything short of active+credentialed, so a mid-provisioning tenant is
|
||||
simply absent from the connection map — proven Docker-free
|
||||
(`chrunner_test.go`'s `TestRegistryRefusesMidProvisioningTenant`,
|
||||
since an empty `DataSource` list never dials ClickHouse). Tantivy was
|
||||
a real, different gap: `search/src/registry.rs`'s `IndexRegistry`
|
||||
opens-or-creates an index for *any* syntactically-valid `tenant_id` on
|
||||
first request, because it's a separate process with no Postgres
|
||||
access and structurally cannot know which tenants are provisioned --
|
||||
without a check upstream, a mid-provisioning tenant's query would have
|
||||
silently succeeded with zero results from a freshly-created empty
|
||||
index. Fixed by adding `enterprise/internal/searchclient.
|
||||
TenantChecker` (backed by a new `rbacstore.TenantIsActive`): `Client.
|
||||
Search` now refuses before the gRPC call goes out if the tenant isn't
|
||||
active, verified Docker-free via a real in-process gRPC server
|
||||
(`searchclient_test.go`'s `TestSearchRefusesMidProvisioningTenant`).
|
||||
|
||||
+33
-9
@@ -386,14 +386,27 @@ cargo test
|
||||
cd ../enterprise
|
||||
go test ./internal/searchclient/... -v
|
||||
# real in-process gRPC server, confirms SearchRequest.tenant_id is set
|
||||
# correctly and that a request with no/invalid tenant identity is refused.
|
||||
# correctly, that a request with no/invalid tenant identity is refused,
|
||||
# and (since TenantChecker was added to close verification-plan item 4)
|
||||
# that a tenant which exists but isn't active yet -- e.g. right after
|
||||
# enterprise-auth -create-tenant but before enterprise-api
|
||||
# -provision-tenant -- is refused too, not silently searched against a
|
||||
# freshly-created empty index.
|
||||
|
||||
go test ./internal/chrunner/... -run MidProvisioning -v
|
||||
# the ClickHouse half of the same item-4 probe -- also Docker-free,
|
||||
# since an empty DataSource list never dials out.
|
||||
```
|
||||
|
||||
This is the one piece of Phase 4's tenant isolation work that has
|
||||
**actually been run and confirmed passing** in an environment without
|
||||
Docker access, alongside `enterprise/internal/loginhandler`'s OIDC
|
||||
tests (§3a) — both are unusually strong evidence precisely because they
|
||||
needed no infrastructure this environment lacked.
|
||||
Docker access, alongside `enterprise/internal/loginhandler`'s OIDC/SAML
|
||||
tests (§3a/§3b) — both are unusually strong evidence precisely because
|
||||
they needed no infrastructure this environment lacked. Writing the
|
||||
`TenantChecker` test here is also what found the Tantivy mid-
|
||||
provisioning gap in the first place, not just what closed it after the
|
||||
fact -- see `api/queryapi/tenant_isolation_gap_test.go`'s item 4 for the
|
||||
full story.
|
||||
|
||||
## 10. Confirm the Helm chart actually enforces the binary swap
|
||||
|
||||
@@ -466,11 +479,22 @@ Full accounting: `/docs/security/threat-model.md`. Headline items:
|
||||
(`enterprise/internal/rbacstore/rbacstore_test.go`) haven't run
|
||||
against a live database in this environment, same gap as the rest of
|
||||
this phase's Postgres-backed pieces.
|
||||
- Three of the four adversarial ClickHouse/Tantivy probes named in
|
||||
`/docs/phase-4-isolation-design.md`'s verification plan are closed
|
||||
(§8, §9); the last (mid-provisioning-race handling) is still stubbed
|
||||
as an explicitly-skipped test in
|
||||
`api/queryapi/tenant_isolation_gap_test.go`.
|
||||
- All four adversarial ClickHouse/Tantivy probes named in
|
||||
`/docs/phase-4-isolation-design.md`'s verification plan are now closed
|
||||
-- see `api/queryapi/tenant_isolation_gap_test.go` for the full
|
||||
accounting. The fourth (mid-provisioning-race handling) turned out to
|
||||
need a real code fix on the Tantivy side, not just a test: `search/
|
||||
src/registry.rs`'s `IndexRegistry` opened-or-created an index for any
|
||||
syntactically-valid `tenant_id`, so a mid-provisioning tenant's query
|
||||
would have silently succeeded with zero results instead of being
|
||||
refused. Fixed via `enterprise/internal/searchclient.TenantChecker`
|
||||
(backed by `rbacstore.TenantIsActive`). Both the ClickHouse and
|
||||
Tantivy halves of this probe run genuinely, without Docker, in this
|
||||
environment (`chrunner_test.go`'s
|
||||
`TestRegistryRefusesMidProvisioningTenant`, `searchclient_test.go`'s
|
||||
`TestSearchRefusesMidProvisioningTenant`) -- `rbacstore.TenantIsActive`
|
||||
itself has skip-gated live-Postgres tests that haven't run here, same
|
||||
disclosed gap as the rest of this phase's Postgres-backed pieces.
|
||||
|
||||
## Tearing down
|
||||
|
||||
|
||||
@@ -412,5 +412,5 @@ terms:
|
||||
| Query audit logging (routine queries) | **Enforced**, fail-open, and now wired to a real writer via `enterprise-api` (`audit.QueryAPILogger`) |
|
||||
| Audit log tamper detection (hash chain) | **Enforced**, verified live |
|
||||
| Audit log tamper prevention (external anchoring) | **Design only** — `FileSink` is a dev stand-in |
|
||||
| Mid-provisioning-race handling (evaluator ticks against a not-yet-active tenant) | **Unverified** — see `api/queryapi/tenant_isolation_gap_test.go` |
|
||||
| Mid-provisioning-race handling (evaluator ticks against a not-yet-active tenant) | **Closed on both storage engines** — see `api/queryapi/tenant_isolation_gap_test.go`; ClickHouse verified Docker-free (structural, not just tested), Tantivy fixed and verified Docker-free after finding it was a real gap, not just an unverified assumption |
|
||||
| Protection against a privileged DB administrator | **Explicit non-goal** |
|
||||
|
||||
Reference in New Issue
Block a user