coffey-labs/cairnobs
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Build per-tenant Tantivy write-routing, closing the last ingest write gap

Browse Source 
search/src/consumer.rs now resolves each record's tenant_id Kafka header
through the same IndexRegistry the read side (search/src/registry.rs +
enterprise/internal/searchclient) already used, and writes into that
tenant's own index instead of always the default one. The periodic
Tantivy commit now commits every tenant index that's actually seen a
write (IndexRegistry::commit_all), not just the default index.

Unlike ClickHouse, this needed no "second binary": Tantivy has no grant
system to gate a commercially-licensed credential behind, so
IndexRegistry already lived directly in this AGPL-core search binary --
there was never an import-boundary reason to split the write side into
an enterprise/ binary the way chwriter/enterprise-ingest was for
ClickHouse. Read and write simply share one registry.

Because Tantivy is an embedded library, this is genuinely verified in
this environment, not just written: registry.rs's
commit_all_commits_default_and_every_opened_tenant_index writes into the
default index plus two tenant indices, confirms nothing is searchable
pre-commit, then confirms all three are post-commit. consumer.rs's
tenant_id_from_headers is factored out as a small pure helper (mirroring
ingest/consumer.tenantIDFromHeaders) with its own unit tests, plus a
guard test against the "tenant_id" header-key literal drifting from the
Go side's -- the same guard-test pattern ingest/cmd/ingest already used
for its own two Go copies of the constant, now mirrored a third time
across the language boundary.

One gap is disclosed, not fixed, by this change: unlike chwriter.Registry
(an active-tenants-only snapshot built at enterprise-ingest startup, so
an unrecognized tenant_id is refused outright) and unlike the read side
(gated by searchclient.TenantChecker), this consumer's registry.resolve()
call has no active-tenant check at all -- search has no Postgres access
to check tenant status against. A still-valid-but-should-be-revoked
ingest credential can cause an index directory to be created for a
tenant that's no longer active. Narrow blast radius (an orphan, isolated,
empty index, not cross-tenant leakage, and only reachable with a real
signed credential), but real -- see registry.rs's doc comment on
resolve(). Closing it fully would mean giving search some way to learn
which tenants are active without an enterprise/ import, which isn't
designed yet.

This closes the last of Phase 4's ingest write-routing gaps (ClickHouse
was closed last commit). The one remaining gap in the whole phase is now
the tenant-picker frontend page, deliberately deferred earlier in this
phase as out of scope for this environment.
This commit is contained in:
jcoffey-dev
2026-08-14 20:16:09 -07:00
parent 1de77b969f
commit bdd42e06f6
9 changed files with 458 additions and 151 deletions
Download Patch File Download Diff File Expand all files Collapse all files
CLAUDE.md
+27 -12
View File
@@ -249,18 +249,33 @@ ClickHouse user `SELECT` only, which would have made every real
per-tenant write fail with a permission error — fixed by granting per-tenant write fail with a permission error — fixed by granting
`SELECT, INSERT` (one credential, both directions; no cross-tenant `SELECT, INSERT` (one credential, both directions; no cross-tenant
boundary is crossed by also allowing INSERT within a tenant's own boundary is crossed by also allowing INSERT within a tenant's own
database). **What's still deferred, clearly**: Tantivy's side. database). **Tantivy write-routing is now built too** —
`search/src/consumer.rs` is a completely independent Redpanda consumer `search/src/consumer.rs` (a completely independent Redpanda consumer,
(not called through `ingest` or `enterprise-ingest` at all) and still not called through `ingest` or `enterprise-ingest` at all: a different
writes every record into the one shared (default) index regardless of codebase and process) now resolves each record's `tenant_id` header
tenant — a different codebase (Rust) and a different process, real, through the *same* `IndexRegistry` the read side already used, and
disclosed, separately-scoped remaining work, not just "the same fix routes the write there instead of always into the default index. Unlike
applied twice." What still keeps this phase from being done: the actual the ClickHouse side, this needed no "second binary": `IndexRegistry`
tenant-picker *page* doesn't exist (`web` has no session/cookie-handling already lives in this AGPL-core binary (Tantivy has no grant system to
code at all yet, and `enterprise-auth` has no CORS middleware for a gate a commercially-licensed credential behind, so there was never an
cross-origin `fetch` with credentials — both real, separately-scoped import-boundary reason to split it out), so read and write share one
frontend gaps), and Tantivy write-routing per the above. Full registry directly. The periodic Tantivy commit now commits every tenant
accounting: index that's seen a write, not just the default one
(`IndexRegistry::commit_all`). **One gap disclosed, not fixed, in this
same change**: unlike `chwriter.Registry` (built from an
active-tenants-only snapshot at startup) and unlike the read side (gated
by `searchclient.TenantChecker`), this consumer's `resolve()` call has
no active-tenant check — this process has no Postgres access to check
against, so a still-valid-but-should-be-revoked ingest credential can
cause an index directory to be created for a tenant that's no longer
active. Narrow blast radius (an orphan, isolated, empty index — not
cross-tenant leakage — and only reachable with a real signed
credential), but real; see `search/src/registry.rs`'s doc comment on
`resolve`. What still keeps this phase from being done: only the
tenant-picker *page* now — `web` has no session/cookie-handling code at
all yet, and `enterprise-auth` has no CORS middleware for a cross-origin
`fetch` with credentials, both real, separately-scoped frontend gaps.
Full accounting:
`/docs/security/threat-model.md`; step-by-step verification procedure `/docs/security/threat-model.md`; step-by-step verification procedure
(not yet run against a live cluster in this environment): (not yet run against a live cluster in this environment):
`/docs/phase-4-runbook.md`. The rest of this section describes the exit `/docs/phase-4-runbook.md`. The rest of this section describes the exit
docs/architecture.md
+28 -12
View File
@@ -139,8 +139,8 @@ escape hatch is opaque to any compiler-injected filter.
ran in the environment it was built in — Tantivy is an embedded ran in the environment it was built in — Tantivy is an embedded
library, so the cross-tenant isolation probe needed no live database library, so the cross-tenant isolation probe needed no live database
or Docker to execute for real, and it passed. or Docker to execute for real, and it passed.
- **Ingest identity and ClickHouse write-routing are now built; Tantivy's - **Ingest identity, ClickHouse write-routing, and Tantivy write-routing
isn't.** `ingest` (AGPL core) gained an optional `TenantResolver` are all now built.** `ingest` (AGPL core) gained an optional `TenantResolver`
(`ingest/internal/grpcserver`): an agent presents a per-tenant bearer (`ingest/internal/grpcserver`): an agent presents a per-tenant bearer
credential (`enterprise-auth -create-ingest-credential-tenant=<id>` credential (`enterprise-auth -create-ingest-credential-tenant=<id>`
mints one, only its hash stored), validated over the network via a new mints one, only its hash stored), validated over the network via a new
@@ -160,13 +160,28 @@ escape hatch is opaque to any compiler-injected filter.
`SELECT` only, which would have made every real per-tenant write fail `SELECT` only, which would have made every real per-tenant write fail
with a permission error — fixed by granting `SELECT, INSERT` (one with a permission error — fixed by granting `SELECT, INSERT` (one
credential, both directions; no cross-tenant boundary is crossed by credential, both directions; no cross-tenant boundary is crossed by
also allowing INSERT within a tenant's own database). What isn't built also allowing INSERT within a tenant's own database). `search`'s
yet: `search`'s independent Redpanda consumer (a different codebase — independent Redpanda consumer (a different codebase — Rust — and a
Rust — and a different process, not reachable through either `ingest` different process, not reachable through either `ingest` or
or `enterprise-ingest`) still writes every record into the one shared `enterprise-ingest`) is now write-routed too:
Tantivy index regardless of tenant. That's real, disclosed, separately `search/src/consumer.rs` resolves each record's `tenant_id` header
scoped remaining work, not something this change claims to have through the *same* `IndexRegistry` the read side
closed. (`search/src/registry.rs` + `enterprise/internal/searchclient`)
already used, and writes there instead of always into the default
index. No "second binary" needed here, unlike ClickHouse — Tantivy has
no grant system to gate a commercially-licensed credential behind, so
`IndexRegistry` already lived directly in this AGPL-core binary, and
read/write just share it. One gap is disclosed rather than closed by
this change: unlike `chwriter.Registry` (an active-tenants-only
snapshot built at startup) and unlike the read side (gated by
`searchclient.TenantChecker`), this consumer's `resolve()` call has no
active-tenant check — the process has no Postgres access to check
against — so a still-valid-but-should-be-revoked ingest credential can
cause an index directory to be created for a tenant that's no longer
active. Narrow blast radius (an orphan, isolated, empty index, not
cross-tenant leakage, and only reachable with a real signed
credential), but real; see `search/src/registry.rs`'s doc comment on
`resolve`.
- `deploy/operator`'s `Tenant` CRD and `enterprise-api -provision-tenant` - `deploy/operator`'s `Tenant` CRD and `enterprise-api -provision-tenant`
are now unified, deliberately lightweight: `-provision-tenant` stays are now unified, deliberately lightweight: `-provision-tenant` stays
the sole real actor (ClickHouse + `rbacstore`), and now also syncs its the sole real actor (ClickHouse + `rbacstore`), and now also syncs its
@@ -188,9 +203,10 @@ plain `api`), sharing a host-port/network-alias trick so `alerting`/
`web` need no conditional config either way. With both storage engines' `web` need no conditional config either way. With both storage engines'
connection/index-layer mechanisms built, deployment topology enforced at connection/index-layer mechanisms built, deployment topology enforced at
both the Helm and docker-compose layers, and the two provisioning both the Helm and docker-compose layers, and the two provisioning
mechanisms unified, the largest remaining gap is ingest's per-tenant mechanisms unified, and both storage engines' write paths now
*write-routing* (identity is now attached at ingest time; nothing per-tenant-routed too (see above), the largest remaining gap in this
downstream of Redpanda consumes it yet to isolate the write, see above). phase is the tenant-picker *frontend* page — the backend protocol is
built, but `web` has no session/cookie-handling code yet to call it.
## Licensing boundary ## Licensing boundary
docs/phase-4-runbook.md
+76 -23
View File
@@ -38,6 +38,15 @@ of what was already run and passed. Two genuine exceptions:
"email" -- crewjam's own fake IdP hit this path. Same remaining gap as "email" -- crewjam's own fake IdP hit this path. Same remaining gap as
OIDC: not yet tried against a real external IdP or a running OIDC: not yet tried against a real external IdP or a running
`enterprise-auth` container. `enterprise-auth` container.
- Tantivy tenant isolation, both directions -- `search/src/registry.rs`'s
cross-tenant read isolation (§9) and, since this pass,
`search/src/consumer.rs`'s per-tenant write-routing (§14) -- verified
live, no disclaimer needed, because Tantivy is an embedded library with
no Docker/broker dependency: real indices, real documents, real
commits, run in this environment. The one thing about it that's still
unverified is not Tantivy itself but the upstream credential/header
plumbing feeding it (ingest's `TenantResolver`, `enterprise-auth`'s
`/internal/authorize-ingest`) against a real running stack.
Everything else — `internal/rbacstore`'s CRUD, the auth-enforcement Everything else — `internal/rbacstore`'s CRUD, the auth-enforcement
walkthrough, the dashboards tenant-scoping fix, the Helm chart, the walkthrough, the dashboards tenant-scoping fix, the Helm chart, the
@@ -604,7 +613,7 @@ yet would refuse all ingest traffic outright rather than degrading
gracefully. See §14 below for what the identity is actually used for on gracefully. See §14 below for what the identity is actually used for on
the write path. the write path.
## 14. Ingest write-routing (ClickHouse built, Tantivy not yet) ## 14. Ingest write-routing (both storage engines)
`enterprise/cmd/enterprise-ingest` (mirrors `enterprise-api`'s "second `enterprise/cmd/enterprise-ingest` (mirrors `enterprise-api`'s "second
binary" shape) consumes the `tenant_id` Kafka header §13 attaches and binary" shape) consumes the `tenant_id` Kafka header §13 attaches and
@@ -643,12 +652,50 @@ never actually executed -- "the test exists" is not the same claim as
"write-routing is confirmed," same caveat §8 already states for the "write-routing is confirmed," same caveat §8 already states for the
read-side `chrunner` tests. read-side `chrunner` tests.
**Not built**: Tantivy's side of this. `search/src/consumer.rs` is a **Tantivy's side is built too, and genuinely verified.**
completely independent Redpanda consumer, not called through `ingest` or `search/src/consumer.rs` reads the same `tenant_id` Kafka header and
`enterprise-ingest` at all, and does not read the `tenant_id` header -- resolves it through `search/src/registry.rs`'s `IndexRegistry` -- the
every record still lands in the one shared (default) Tantivy index same registry the read side (§9) already uses -- routing each record's
regardless of tenant. See CLAUDE.md and `/docs/security/threat-model.md`'s write into its own tenant's Tantivy index instead of the single default
"Read this first" for the full disclosure. one. No "second binary" was needed here the way ClickHouse needed
`enterprise-ingest`: Tantivy has no grant system to gate a
commercially-licensed credential behind, so `IndexRegistry` already
lives directly in AGPL-core `search`, and read/write just share it.
Because Tantivy is an embedded library (no Docker/broker needed to
exercise real logic), this actually ran in this environment:
```sh
cd search
cargo test --quiet
# registry.rs: commit_all_commits_default_and_every_opened_tenant_index
# writes into the default index plus two tenant indices, confirms
# nothing is searchable before commit_all(), then confirms all three ARE
# searchable after -- the write-routing + periodic-commit path
# end-to-end, for real, using the same real-Tantivy-index discipline as
# every other registry.rs/index.rs test. consumer.rs:
# tenant_id_from_headers_* cover the header-extraction helper
# (missing/present/unrelated-header cases) and
# test_tenant_id_header_key_matches_go guards the "tenant_id" literal
# against drifting from ingest/consumer.TenantIDHeaderKey /
# ingest/internal/grpcserver.TenantIDHeaderKey the same way
# ingest/cmd/ingest's own guard test does on the Go side.
```
**What's still not built, for either engine**: a live active-tenant
recheck at write time. `chwriter.Registry`'s per-tenant writer map is a
snapshot built once at `enterprise-ingest` startup from
`rbacstore.ListProvisionedDataSources` (active tenants only) -- an
unrecognized `tenant_id` is refused outright, but a tenant deprovisioned
*after* startup keeps writing successfully until the next restart.
`IndexRegistry.resolve()` has no allowlist at all on either the read or
write side -- `search` has no Postgres access to check tenant status
against -- so a still-valid-but-should-be-revoked ingest credential can
cause an orphan Tantivy index directory to be created for a tenant
that's no longer active. Narrow blast radius either way (isolated, not
cross-tenant leakage, reachable only with a real signed credential), but
real and disclosed, not silently accepted -- see
`search/src/registry.rs`'s doc comment on `resolve` and
`/docs/security/threat-model.md`'s "Read this first".
**Also not built**: Helm/`docker-compose.yml` do gate *whether* **Also not built**: Helm/`docker-compose.yml` do gate *whether*
`enterprise-ingest` runs at all (`ingest.requireTenantCredential`, same `enterprise-ingest` runs at all (`ingest.requireTenantCredential`, same
@@ -688,22 +735,28 @@ Full accounting: `/docs/security/threat-model.md`. Headline items:
that split (declarative request vs. imperative provisioning action) that split (declarative request vs. imperative provisioning action)
is intentional, not the "two disconnected sources of truth" gap this is intentional, not the "two disconnected sources of truth" gap this
bullet used to describe. bullet used to describe.
- **Ingest now has a real tenant identity (§13), and ClickHouse - **Ingest now has a real tenant identity (§13), and both storage
write-routing is built (§14) -- Tantivy write-routing is the one gap engines' write-routing is built (§14).** An agent presents a bearer
left.** An agent presents a bearer credential (`enterprise-auth credential (`enterprise-auth -create-ingest-credential-tenant=<id>`),
-create-ingest-credential-tenant=<id>`), `ingest/internal/grpcserver. `ingest/internal/grpcserver.TenantResolver` validates it (fail-closed)
TenantResolver` validates it (fail-closed) and attaches the resolved and attaches the resolved tenant ID to every record as a `tenant_id`
tenant ID to every record as a `tenant_id` Kafka message header. Kafka message header. `enterprise-ingest` reads that header back and
`enterprise-ingest` reads that header back and routes each record's routes each record's ClickHouse write into its own tenant's database
ClickHouse write into its own tenant's database (not yet confirmed (not yet confirmed against a real ClickHouse in this environment --
against a real ClickHouse in this environment -- see §14). Nothing see §14). `search/src/consumer.rs` reads the same header and routes
reads that header on the Tantivy side yet -- every record still lands each record into its own tenant's Tantivy index -- genuinely verified
in the one shared Tantivy index no matter what. A newly-provisioned in this environment, unlike the ClickHouse side, since Tantivy needs
tenant's ClickHouse database is real, isolated, and actually no Docker to exercise real logic. Both engines share one remaining,
populated by write-routed traffic (once confirmed live); its Tantivy disclosed gap: neither write path rechecks tenant-active status live
index remains real, isolated at query time, and permanently empty (ClickHouse: a startup-time snapshot, stale until restart; Tantivy: no
until Tantivy's write-routing split is built (a real, scoped allowlist at all, since `search` has no Postgres access) -- narrow
follow-up, no longer an undesigned one). blast radius, not cross-tenant leakage, but real; see §14 and
`/docs/security/threat-model.md`'s "Read this first". A
newly-provisioned tenant's ClickHouse database and Tantivy index are
both now real, isolated, and actually populated by write-routed
traffic (the ClickHouse claim pending live confirmation, the Tantivy
claim already verified) -- what used to be "permanently empty" for
both is no longer true for either.
- **Human SSO login now works for both OIDC (§3a) and SAML (§3b)** -- - **Human SSO login now works for both OIDC (§3a) and SAML (§3b)** --
each verified with a real fake IdP (genuine cryptographic signing and each verified with a real fake IdP (genuine cryptographic signing and
verification), not yet a real external IdP or a running verification), not yet a real external IdP or a running
docs/security/threat-model.md
+77 -47
View File
@@ -9,15 +9,21 @@ for the full design rationale behind the controls described here.
## Read this first: the single most important open finding ## Read this first: the single most important open finding
**Updated a third time.** This section originally read "log data **Updated a fourth time.** This section originally read "log data
queried through `POST /query` is not tenant-isolated at all," then queried through `POST /query` is not tenant-isolated at all," then
"ClickHouse is isolated but Tantivy isn't," then "ingest tags records "ClickHouse is isolated but Tantivy isn't," then "ingest tags records
with a tenant identity but nothing routes the write." Both ClickHouse with a tenant identity but nothing routes the write," then "ClickHouse
*and* Tantivy connection/index-layer isolation are built on the *read* write-routing is built but Tantivy's isn't." Both storage engines are
path, and ClickHouse write-routing is now built too. What's left is now isolated on both the read and write paths. What's left is narrower:
narrower but still real: **whether a given deployment actually runs the **whether a given deployment actually runs the isolated binaries**
isolated binaries**, and **Tantivy's write path**, which still has no (deployment-time, not code-level), and two disclosed write-side gaps,
per-tenant routing at all. different in kind: ClickHouse's write registry is a startup-time
snapshot of active tenants with no live recheck (a *deprovisioned*
tenant can keep writing successfully until the next `enterprise-ingest`
restart), while Tantivy's write path has no active-tenant allowlist at
all — it opens an index for *any* syntactically-valid `tenant_id` a
record carries, active, deprovisioned, or never real. See below for
both, in full.
**ClickHouse (the SQL path) is built.** `enterprise/internal/ **ClickHouse (the SQL path) is built.** `enterprise/internal/
tenantprovision` (real `CREATE DATABASE`/`CREATE USER`/`GRANT`) and tenantprovision` (real `CREATE DATABASE`/`CREATE USER`/`GRANT`) and
@@ -70,43 +76,66 @@ provisioned, pointing at the same ClickHouse/Postgres. The Helm chart
makes the *default*, chart-managed path correct; it isn't a runtime makes the *default*, chart-managed path correct; it isn't a runtime
guard against misconfiguration. guard against misconfiguration.
**Ingest now has a real tenant identity, and ClickHouse write-routing is **Ingest now has a real tenant identity, and both storage engines'
built — Tantivy write-routing is the one gap left.** `chrunner`/ write-routing is built.** `chrunner`/`searchclient` prove *read*
`searchclient` prove *read* isolation given tenant-scoped data exists; an isolation given tenant-scoped data exists; an optional
optional `ingest/internal/grpcserver.TenantResolver` closes the "does a `ingest/internal/grpcserver.TenantResolver` closes the "does a record
record know which tenant it belongs to" half by validating a per-tenant know which tenant it belongs to" half by validating a per-tenant bearer
bearer credential an agent presents credential an agent presents
(`enterprise-auth -create-ingest-credential-tenant=<id>` mints one; only (`enterprise-auth -create-ingest-credential-tenant=<id>` mints one; only
its SHA-256 hash is ever stored) against a `POST its SHA-256 hash is ever stored) against a `POST
/internal/authorize-ingest` endpoint, and attaching the resolved tenant /internal/authorize-ingest` endpoint, and attaching the resolved tenant
ID to every record as a `tenant_id` Kafka message header before ID to every record as a `tenant_id` Kafka message header before
producing it — fail-closed: once a resolver is configured, a missing or producing it — fail-closed: once a resolver is configured, a missing or
invalid credential refuses the whole batch, never falls back to "no invalid credential refuses the whole batch, never falls back to "no
tenant." `enterprise/cmd/enterprise-ingest` (a second binary, mirroring tenant."
`enterprise-api`) now consumes that header: it reuses `ingest/consumer`'s
own flush loop with `enterprise/internal/chwriter.Registry` — a **ClickHouse**: `enterprise/cmd/enterprise-ingest` (a second binary,
per-tenant `*clickhousewriter.Writer` registry, built the same way mirroring `enterprise-api`) consumes that header: it reuses
`chrunner`'s per-tenant connections are — swapped in as the writer, so a `ingest/consumer`'s own flush loop with `enterprise/internal/
tagged batch's records are grouped by tenant and each group INSERTed chwriter.Registry` — a per-tenant `*clickhousewriter.Writer` registry,
through that tenant's own ClickHouse connection, fail-closed on an built the same way `chrunner`'s per-tenant connections are — swapped in
untagged or unprovisioned tenant. Building it surfaced a real gap in an as the writer, so a tagged batch's records are grouped by tenant and
already-shipped control: `tenantprovision.ProvisionClickHouse`'s grant each group INSERTed through that tenant's own ClickHouse connection,
was `SELECT`-only (correct for the read-side credential `chrunner` uses, fail-closed on an untagged or unprovisioned tenant. Building it surfaced
but `chwriter` reuses the same credential for writes) — every real a real gap in an already-shipped control: `tenantprovision.
per-tenant write would have failed with a permission error until this ProvisionClickHouse`'s grant was `SELECT`-only (correct for the
was widened to `SELECT, INSERT`. **Not yet confirmed against a real read-side credential `chrunner` uses, but `chwriter` reuses the same
ClickHouse**, same caveat as the read-side chrunner claim above — the credential for writes) — every real per-tenant write would have failed
Docker-free fail-closed tests pass, the live-database tests are written with a permission error until this was widened to `SELECT, INSERT`.
but skip-gated, see `/docs/phase-4-runbook.md`. What's still missing: **Not yet confirmed against a real ClickHouse**, same caveat as the
`search`'s independent Redpanda consumer does not read the `tenant_id` read-side chrunner claim above — the Docker-free fail-closed tests pass,
header at all — every ingested record still lands in the one shared the live-database tests are written but skip-gated, see
(default) Tantivy index regardless of tenant. A newly-provisioned `/docs/phase-4-runbook.md`.
tenant's ClickHouse database is now real, isolated, and actually
populated by write-routed agent traffic (once confirmed against a real **Tantivy**: `search/src/consumer.rs` now resolves each record's
cluster); its Tantivy index remains real, isolated, and queryable `tenant_id` header through the *same* `IndexRegistry` the read side
through `enterprise-api` — and permanently empty, until Tantivy's own already used (`search/src/registry.rs`), and writes there instead of
write-routing split is built, which is now scoped, disclosed remaining always into the default index — genuinely verified in this environment,
work, not an undesigned gap. same as the read-side Tantivy claim above, since Tantivy is an embedded
library with no Docker dependency. No "second binary" was needed here,
unlike ClickHouse: Tantivy has no grant system to gate a
commercially-licensed credential behind, so `IndexRegistry` already
lived directly in this AGPL-core `search` binary, and read/write simply
share it.
**Both engines share one open question** — deployment topology, covered
above — and Tantivy specifically has one gap ClickHouse's design doesn't:
`chwriter.Registry`'s map is built once at startup from an
active-tenants-only query, so an unrecognized `tenant_id` is refused
outright; `IndexRegistry.resolve()` (used for both read and write) has
no equivalent allowlist at all, because `search` has no Postgres access
to check tenant status against — a syntactically-valid `tenant_id` on a
still-valid-but-should-be-revoked ingest credential can cause an orphan
index directory to be created for a tenant that's no longer active.
Narrow blast radius (isolated, empty except for that traffic, not
cross-tenant leakage, and reachable only with a real signed credential),
but real, and not closed by this change; see
`search/src/registry.rs`'s doc comment on `resolve`. A newly-provisioned
tenant's ClickHouse database and Tantivy index are both now real,
isolated, and actually populated by write-routed agent traffic (the
ClickHouse claim pending live confirmation, the Tantivy claim already
verified).
## System overview ## System overview
@@ -146,13 +175,14 @@ sentryctl ──▶ api, alerting (Bearer token when SENTRYCTL_TOKEN is set)
Ingest path (agent → Redpanda → ingest → ClickHouse, and Redpanda → Ingest path (agent → Redpanda → ingest → ClickHouse, and Redpanda →
search → Tantivy): `ingest` resolves and tags each record with a real search → Tantivy): `ingest` resolves and tags each record with a real
tenant ID (see "Read this first" above). `enterprise/cmd/enterprise-ingest` tenant ID (see "Read this first" above). `enterprise/cmd/enterprise-ingest`
now consumes that tag and routes ClickHouse writes to each tenant's own consumes that tag and routes ClickHouse writes to each tenant's own
database. Tantivy's independent Redpanda consumer (`search/src/ database. `search/src/consumer.rs` (Tantivy's independent Redpanda
consumer.rs`) still does not consume the tag at all — every record still consumer) consumes the same tag and routes each record into its own
lands in the one shared Tantivy index regardless of tenant. That tenant's index. Both write paths' one remaining gap — no live
narrower gap is still out of scope for what's built so far and is not active-tenant recheck (ClickHouse: a startup-time snapshot; Tantivy: no
separately designed in `/docs/phase-4-isolation-design.md`; named here as allowlist at all) — is named in "Read this first" above, not separately
a gap that design doc doesn't yet cover, not just an implementation gap. designed in `/docs/phase-4-isolation-design.md`; named here as a gap that
design doc doesn't yet cover, not just an implementation gap.
## Module boundary (trust boundary #1) ## Module boundary (trust boundary #1)
@@ -454,8 +484,8 @@ terms:
| Tantivy per-tenant index routing (`search/src/registry.rs`) | **Enforced, verified live** — real Tantivy indices, real cross-tenant probe, all passing | | Tantivy per-tenant index routing (`search/src/registry.rs`) | **Enforced, verified live** — real Tantivy indices, real cross-tenant probe, all passing |
| Tantivy tenant_id resolution (`enterprise/internal/searchclient`) | **Enforced, verified live** — real gRPC wire-level test | | Tantivy tenant_id resolution (`enterprise/internal/searchclient`) | **Enforced, verified live** — real gRPC wire-level test |
| Ingest tenant *identity* (credential validation, tagging) | **Built and tested** — fail-closed `TenantResolver`, `tenant_id` Kafka header attached per record | | Ingest tenant *identity* (credential validation, tagging) | **Built and tested** — fail-closed `TenantResolver`, `tenant_id` Kafka header attached per record |
| Ingest tenant *write-routing*, ClickHouse | **Built, not yet confirmed against a real ClickHouse** — `enterprise-ingest`/`chwriter.Registry` route each tagged batch to its tenant's own database, fail-closed on an untagged/unprovisioned tenant; Docker-free tests pass, live-database tests are skip-gated | | Ingest tenant *write-routing*, ClickHouse | **Built, not yet confirmed against a real ClickHouse** — `enterprise-ingest`/`chwriter.Registry` route each tagged batch to its tenant's own database, fail-closed on an untagged/unprovisioned tenant; Docker-free tests pass, live-database tests are skip-gated. Startup-time active-tenant snapshot, no live recheck — a deprovisioned tenant can keep writing until the next restart |
| Ingest tenant *write-routing*, Tantivy | **Not implemented, now scoped** — every record still lands in the single shared Tantivy index regardless of tenant; `search/src/consumer.rs` consuming the tenant_id header to route the write is real, disclosed remaining work | | Ingest tenant *write-routing*, Tantivy | **Built and genuinely verified** — `search/src/consumer.rs` routes each record into its own tenant's index via `IndexRegistry`, same registry the (already-verified) read side uses; no Docker needed, real tests pass. No active-tenant allowlist at all on write (`search` has no Postgres access) — a syntactically-valid `tenant_id` on a still-valid credential can create an orphan index for a no-longer-active tenant; narrow, disclosed, not cross-tenant leakage |
| Deployment actually routing traffic to `enterprise-api` (Helm) | **Enforced** — `api`/`enterprise-api` are mutually exclusive, same flag as RBAC/audit/SSO | | Deployment actually routing traffic to `enterprise-api` (Helm) | **Enforced** — `api`/`enterprise-api` are mutually exclusive, same flag as RBAC/audit/SSO |
| Deployment actually routing traffic to `enterprise-api` (docker-compose) | **Enforced** — `api`/`enterprise-api` are mutually exclusive via `COMPOSE_PROFILES`, same flag choice as Helm's `enterprise.enabled`; verified via `docker compose config`, not an actual `docker compose up` in this environment | | Deployment actually routing traffic to `enterprise-api` (docker-compose) | **Enforced** — `api`/`enterprise-api` are mutually exclusive via `COMPOSE_PROFILES`, same flag choice as Helm's `enterprise.enabled`; verified via `docker compose config`, not an actual `docker compose up` in this environment |
| Human SSO login — OIDC | **Built, verified with a real fake IdP** (not yet tried against a real external IdP) | | Human SSO login — OIDC | **Built, verified with a real fake IdP** (not yet tried against a real external IdP) |
enterprise/README.md
+14 -10
View File
@@ -182,14 +182,13 @@ silently left out:
a cross-origin `fetch` with credentials from `web`'s origin needs a cross-origin `fetch` with credentials from `web`'s origin needs
it), neither of which is verifiable in this environment without a it), neither of which is verifiable in this environment without a
live backend and a browser session to exercise. live backend and a browser session to exercise.
- **Ingest write-routing, for Tantivy** -- `search/src/consumer.rs` (a
completely independent Redpanda consumer, not called through `ingest` Ingest write-routing (both ClickHouse and Tantivy) is no longer on this
or `enterprise-ingest` at all) still writes every record into the one list -- see "Ingest write-routing (ClickHouse)" below and
shared (default) Tantivy index, regardless of tenant. The ClickHouse `/search/README.md`'s "Per-tenant indices" section for Tantivy, which
half is now built (see "Ingest write-routing" below); Tantivy's is needed no code in this module at all: `search`'s `IndexRegistry` already
real, disclosed, separate follow-up work -- a different codebase lived in AGPL core, so its write side didn't need an `enterprise/`
(Rust) and a different consumer process, not just "the same fix counterpart the way ClickHouse's did.
applied twice."
Deployment-topology routing (does traffic actually reach `enterprise-api` Deployment-topology routing (does traffic actually reach `enterprise-api`
instead of `api`) is no longer deferred -- both `deploy/helm/sentry` and instead of `api`) is no longer deferred -- both `deploy/helm/sentry` and
@@ -307,8 +306,13 @@ ingest must never import enterprise/) is the client side of `internal/
authhandler`'s new `POST /internal/authorize-ingest` -- see "Ingest authhandler`'s new `POST /internal/authorize-ingest` -- see "Ingest
tenant identity" above. tenant identity" above.
Future additions: per-tenant write-routing for ingest (ClickHouse and Per-tenant write-routing for ingest is built for both ClickHouse (this
Tantivy both) -- see "Ingest tenant identity" above. module's `cmd/enterprise-ingest` + `internal/chwriter`, see "Ingest
write-routing (ClickHouse)" above) and Tantivy (`search/src/consumer.rs`
+ `search/src/registry.rs`, entirely in AGPL core -- see
`/search/README.md`'s "Per-tenant indices" section, since Tantivy's lack
of a grant system meant there was never an import-boundary reason to put
any of it here).
## Why OIDC and SAML aren't hand-rolled ## Why OIDC and SAML aren't hand-rolled
search/README.md
+39 -13
View File
@@ -38,7 +38,7 @@ ClickHouse's `logs.record_id` column (see
only ever returns IDs, never row data — it stays a pure text index, not only ever returns IDs, never row data — it stays a pure text index, not
a second copy of the row. a second copy of the row.
## Per-tenant indices (Phase 4) — read-side only ## Per-tenant indices (Phase 4) — read and write
`SearchRequest.tenant_id` (empty by default) selects which index `SearchRequest.tenant_id` (empty by default) selects which index
`src/registry.rs`'s `IndexRegistry` searches: empty resolves to the `src/registry.rs`'s `IndexRegistry` searches: empty resolves to the
@@ -50,13 +50,31 @@ convention). `tenant_id` is set only by a trusted server-side caller
(`enterprise/internal/searchclient`, from the authenticated request (`enterprise/internal/searchclient`, from the authenticated request
identity) — never a value a browser/client controls. identity) — never a value a browser/client controls.
**This is read-side isolation only.** `consumer.rs`'s Redpanda consumer `consumer.rs`'s Redpanda consumer resolves the *same* registry now,
— the only thing that ever *writes* into an index — still only ever keyed by each Kafka message's `tenant_id` header — attached server-side
writes into the single default index, because `ingest`/the log-record by `ingest/internal/grpcserver` after validating an agent's per-tenant
schema itself carries no tenant concept yet. A newly-opened tenant index credential, mirroring exactly how `enterprise/cmd/enterprise-ingest`
starts, and stays, empty until something upstream of this service routes the ClickHouse write side (see `/enterprise/README.md`'s "Ingest
becomes tenant-aware on the write side too — a real, disclosed gap, not write-routing" section). No import boundary to work around here, unlike
an oversight; see `/docs/security/threat-model.md`. Go: `IndexRegistry` already lives in this AGPL-core binary, so both the
read and write paths share one registry directly, with no "second
binary" needed. The periodic Tantivy commit (`COMMIT_INTERVAL_MS`) now
commits every tenant index that's actually seen a write, plus the
default index, via `IndexRegistry::commit_all`, not just one index.
**One residual gap, disclosed rather than fixed here**: unlike the read
side (gated by `enterprise/internal/searchclient`'s `TenantChecker`,
which refuses to search a tenant that isn't `active` in `rbacstore`) and
unlike ClickHouse's write side (`chwriter.Registry`, built from an
active-tenants-only snapshot at startup, so an unrecognized tenant has
no writer at all), this consumer's `registry.resolve()` call has no
active-tenant gate — this process has no Postgres access to check
against, the same reason `IndexRegistry` couldn't do the mid-provisioning
check itself before `TenantChecker` was added for the read side. A
still-valid (not yet revoked) ingest credential for a tenant that's no
longer active can cause an index directory to be created for it here.
See `src/registry.rs`'s doc comment on `resolve` for the full writeup,
including why closing it fully isn't scoped yet.
## Offset tracking: why this isn't a Kafka consumer group ## Offset tracking: why this isn't a Kafka consumer group
@@ -116,12 +134,20 @@ before-commit/after-commit visibility boundary. `registry.rs`'s tests
are the same shape and cover the actual adversarial claim: real per- are the same shape and cover the actual adversarial claim: real per-
tenant indices, real documents, and a search scoped to one tenant tenant indices, real documents, and a search scoped to one tenant
returning zero results for another tenant's matching document returning zero results for another tenant's matching document
(`tenant_index_is_isolated_from_default_and_other_tenants`) — all of (`tenant_index_is_isolated_from_default_and_other_tenants`), plus
this, unlike almost everything else in Phase 4, was genuinely run in `commit_all_commits_default_and_every_opened_tenant_index` proving the
the environment this was built in, not just written. `consumer.rs` (the write-routing commit path actually makes every tenant's buffered writes
rskafka wiring) is not unit-tested — that needs a real Redpanda, same searchable, not just the default index's — all of this, unlike almost
everything else in Phase 4, was genuinely run in the environment this
was built in, not just written. `consumer.rs`'s Kafka/rskafka wiring
itself is still not unit-tested — that needs a real Redpanda, same
category of gap as `/ingest`'s `kafka.Reader`/`kafka.Writer` wiring, and category of gap as `/ingest`'s `kafka.Reader`/`kafka.Writer` wiring, and
is exercised by the docker-compose end-to-end flow instead. is exercised by the docker-compose end-to-end flow instead — but the
pure `tenant_id_from_headers` header-extraction helper it uses is
factored out and unit-tested the same way `ingest/consumer`'s Go
equivalent is, including a guard test
(`test_tenant_id_header_key_matches_go`) against the header-key literal
drifting from the Go side's.
```sh ```sh
# from the repo root, not search/ # from the repo root, not search/
search/src/consumer.rs
+91 -9
View File
@@ -2,13 +2,25 @@ use anyhow::{Context, Result};
use prost::Message; use prost::Message;
use rskafka::client::partition::UnknownTopicHandling; use rskafka::client::partition::UnknownTopicHandling;
use rskafka::client::ClientBuilder; use rskafka::client::ClientBuilder;
use std::collections::BTreeMap;
use std::sync::Arc; use std::sync::Arc;
use tokio::sync::Mutex; use tokio::sync::Mutex;
use crate::config::Config; use crate::config::Config;
use crate::index::SearchIndex;
use crate::logsv1; use crate::logsv1;
use crate::offsets::OffsetStore; use crate::offsets::OffsetStore;
use crate::registry::IndexRegistry;
/// Kafka message header a resolved tenant ID rides in, attached by
/// ingest's gRPC front end. Mirrors `ingest/internal/grpcserver.
/// TenantIDHeaderKey` / `ingest/consumer.TenantIDHeaderKey` -- those two
/// Go packages duplicate the same literal rather than importing across a
/// producer/consumer boundary (see their doc comments), and this Rust
/// consumer is a third independent reader of the same header, so it
/// duplicates the literal too. `TestTenantIDHeaderKeyMatchesGo` below
/// guards against drift the same way `ingest/cmd/ingest`'s
/// `TestTenantIDHeaderKeyConstantsMatch` does on the Go side.
const TENANT_ID_HEADER_KEY: &str = "tenant_id";
/// Reads the same `sentry.logs.raw` topic ingest's ClickHouse-writer /// Reads the same `sentry.logs.raw` topic ingest's ClickHouse-writer
/// consumer reads, as an independent consumer group in spirit (its own /// consumer reads, as an independent consumer group in spirit (its own
@@ -18,7 +30,7 @@ use crate::offsets::OffsetStore;
/// discovered dynamically, since it has to match what /// discovered dynamically, since it has to match what
/// /transport/provision-topics.sh actually created anyway (documented /// /transport/provision-topics.sh actually created anyway (documented
/// cross-component contract, same as the topic name already is). /// cross-component contract, same as the topic name already is).
pub async fn run(cfg: Arc<Config>, index: Arc<SearchIndex>, partition_count: i32) -> Result<()> { pub async fn run(cfg: Arc<Config>, registry: Arc<IndexRegistry>, partition_count: i32) -> Result<()> {
let client = ClientBuilder::new(cfg.redpanda_brokers.clone()) let client = ClientBuilder::new(cfg.redpanda_brokers.clone())
.build() .build()
.await .await
@@ -32,14 +44,16 @@ pub async fn run(cfg: Arc<Config>, index: Arc<SearchIndex>, partition_count: i32
// Periodic Tantivy commit, batched for throughput the same way // Periodic Tantivy commit, batched for throughput the same way
// ingest's ClickHouse writer batches inserts rather than inserting // ingest's ClickHouse writer batches inserts rather than inserting
// per-record. // per-record. Commits every tenant index a write has actually been
// routed to (plus the default index), not just one -- see
// registry.rs's commit_all doc comment.
let commit_interval = cfg.commit_interval; let commit_interval = cfg.commit_interval;
let index_for_commit = Arc::clone(&index); let registry_for_commit = Arc::clone(&registry);
tokio::spawn(async move { tokio::spawn(async move {
let mut ticker = tokio::time::interval(commit_interval); let mut ticker = tokio::time::interval(commit_interval);
loop { loop {
ticker.tick().await; ticker.tick().await;
if let Err(e) = index_for_commit.commit().await { if let Err(e) = registry_for_commit.commit_all().await {
tracing::error!(error = %e, "periodic tantivy commit failed"); tracing::error!(error = %e, "periodic tantivy commit failed");
} }
} }
@@ -49,11 +63,11 @@ pub async fn run(cfg: Arc<Config>, index: Arc<SearchIndex>, partition_count: i32
for partition in 0..partition_count { for partition in 0..partition_count {
let start_offset = offsets.lock().await.get(partition); let start_offset = offsets.lock().await.get(partition);
let client = Arc::clone(&client); let client = Arc::clone(&client);
let index = Arc::clone(&index); let registry = Arc::clone(&registry);
let offsets = Arc::clone(&offsets); let offsets = Arc::clone(&offsets);
let topic = cfg.redpanda_topic.clone(); let topic = cfg.redpanda_topic.clone();
handles.push(tokio::spawn(async move { handles.push(tokio::spawn(async move {
consume_partition(client, topic, partition, start_offset, index, offsets).await consume_partition(client, topic, partition, start_offset, registry, offsets).await
})); }));
} }
@@ -65,13 +79,26 @@ pub async fn run(cfg: Arc<Config>, index: Arc<SearchIndex>, partition_count: i32
Ok(()) Ok(())
} }
/// Extracts the `tenant_id` header's value, or "" if absent -- an empty
/// string is exactly what `IndexRegistry::resolve` treats as "route to
/// the default index," so an untagged record (every Phase 0-3 message,
/// and any Phase 4 message from a deployment that never turned on
/// `ingest`'s `TenantResolver`) keeps landing in the same shared index
/// it always has. Mirrors `ingest/consumer.tenantIDFromHeaders` exactly.
fn tenant_id_from_headers(headers: &BTreeMap<String, Vec<u8>>) -> String {
headers
.get(TENANT_ID_HEADER_KEY)
.map(|v| String::from_utf8_lossy(v).into_owned())
.unwrap_or_default()
}
#[allow(clippy::too_many_arguments)] #[allow(clippy::too_many_arguments)]
async fn consume_partition( async fn consume_partition(
client: Arc<rskafka::client::Client>, client: Arc<rskafka::client::Client>,
topic: String, topic: String,
partition: i32, partition: i32,
start_offset: i64, start_offset: i64,
index: Arc<SearchIndex>, registry: Arc<IndexRegistry>,
offsets: Arc<Mutex<OffsetStore>>, offsets: Arc<Mutex<OffsetStore>>,
) -> Result<()> { ) -> Result<()> {
let partition_client = client let partition_client = client
@@ -115,8 +142,25 @@ async fn consume_partition(
continue; continue;
} }
let tenant_id = tenant_id_from_headers(&record_and_offset.record.headers);
let index = match registry.resolve(&tenant_id).await {
Ok(index) => index,
Err(e) => {
// Shouldn't normally happen -- ingest/grpcserver only
// ever attaches a tenant_id it validated against a
// real credential -- but an unsafe/malformed
// tenant_id is a hard skip, never a silent fall-back
// to the default or any other tenant's index. See
// registry.rs's doc comment for the one residual gap
// this consumer doesn't close (no active-tenant
// check, since this process has no Postgres access).
tracing::error!(error = %e, record_id = %rec.record_id, tenant_id, "skipping record: failed to resolve tenant index");
continue;
}
};
if let Err(e) = index.upsert(&rec.record_id, &rec.message).await { if let Err(e) = index.upsert(&rec.record_id, &rec.message).await {
tracing::error!(error = %e, record_id = %rec.record_id, "failed to index record"); tracing::error!(error = %e, record_id = %rec.record_id, tenant_id, "failed to index record");
} }
} }
@@ -132,3 +176,41 @@ async fn consume_partition(
} }
} }
} }
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn tenant_id_from_headers_returns_empty_string_when_absent() {
let headers = BTreeMap::new();
assert_eq!(tenant_id_from_headers(&headers), "");
}
#[test]
fn tenant_id_from_headers_extracts_the_tenant_id_header() {
let mut headers = BTreeMap::new();
headers.insert(TENANT_ID_HEADER_KEY.to_string(), b"acme".to_vec());
assert_eq!(tenant_id_from_headers(&headers), "acme");
}
#[test]
fn tenant_id_from_headers_ignores_unrelated_headers() {
let mut headers = BTreeMap::new();
headers.insert("some-other-header".to_string(), b"acme".to_vec());
assert_eq!(tenant_id_from_headers(&headers), "");
}
/// Guards against the exact literal drift
/// `ingest/cmd/ingest`'s `TestTenantIDHeaderKeyConstantsMatch` guards
/// against on the Go side -- three independent readers/writers of the
/// same Kafka header (`ingest/internal/grpcserver` producing,
/// `ingest/consumer` and this file both consuming) duplicate the same
/// literal by design rather than sharing an import across a
/// producer/consumer or language boundary, so nothing but a test
/// catches them drifting apart.
#[test]
fn test_tenant_id_header_key_matches_go() {
assert_eq!(TENANT_ID_HEADER_KEY, "tenant_id");
}
}
search/src/main.rs
+6 -7
View File
@@ -35,11 +35,10 @@ async fn main() -> Result<()> {
let index = Arc::new( let index = Arc::new(
SearchIndex::open_or_create(&cfg.index_path).context("opening tantivy index")?, SearchIndex::open_or_create(&cfg.index_path).context("opening tantivy index")?,
); );
// Per-tenant indices (Phase 4) are resolved on demand by // Per-tenant indices are resolved on demand by IndexRegistry, opened
// IndexRegistry, opened under cfg.tenants_index_path -- see // under cfg.tenants_index_path -- see registry.rs's doc comment for
// registry.rs's doc comment for what this does and doesn't isolate // what this isolates (both read and write now) and the one residual
// yet (read-side only; the consumer below still only ever writes // gap it doesn't close.
// into the single default `index` above).
let registry = Arc::new(IndexRegistry::new(Arc::clone(&index), cfg.tenants_index_path.clone())); let registry = Arc::new(IndexRegistry::new(Arc::clone(&index), cfg.tenants_index_path.clone()));
let partition_count: i32 = std::env::var("REDPANDA_TOPIC_PARTITIONS") let partition_count: i32 = std::env::var("REDPANDA_TOPIC_PARTITIONS")
@@ -48,9 +47,9 @@ async fn main() -> Result<()> {
.unwrap_or(DEFAULT_PARTITION_COUNT); .unwrap_or(DEFAULT_PARTITION_COUNT);
let consumer_cfg = Arc::clone(&cfg); let consumer_cfg = Arc::clone(&cfg);
let consumer_index = Arc::clone(&index); let consumer_registry = Arc::clone(&registry);
let consumer_handle = tokio::spawn(async move { let consumer_handle = tokio::spawn(async move {
if let Err(e) = consumer::run(consumer_cfg, consumer_index, partition_count).await { if let Err(e) = consumer::run(consumer_cfg, consumer_registry, partition_count).await {
tracing::error!(error = %e, "redpanda consumer exited with error"); tracing::error!(error = %e, "redpanda consumer exited with error");
} }
}); });
search/src/registry.rs
+100 -18
View File
@@ -6,26 +6,45 @@ use tokio::sync::RwLock;
use crate::index::SearchIndex; use crate::index::SearchIndex;
/// Resolves a `tenant_id` (from `SearchRequest.tenant_id`, per /// Resolves a `tenant_id` to its own `SearchIndex`, opening one on
/// search.proto's doc comment: set only by a trusted server-side caller /// demand under `<tenants_root>/<tenant_id>` the first time it's
/// -- `enterprise/internal/searchclient`, from the authenticated request /// requested. Used on both sides now: the read side
/// identity, never a value a browser/client controls directly) to its /// (`SearchRequest.tenant_id`, per search.proto's doc comment -- set
/// own `SearchIndex`, opening one on demand under `<tenants_root>/ /// only by a trusted server-side caller, `enterprise/internal/
/// <tenant_id>` the first time it's requested. /// searchclient`, from the authenticated request identity, never a
/// value a browser/client controls directly) and the write side
/// (`consumer.rs`, from a Kafka message's `tenant_id` header, attached
/// server-side by `ingest/internal/grpcserver` after validating an
/// agent's per-tenant credential -- never a value the agent's message
/// body controls directly either).
/// ///
/// An empty `tenant_id` resolves to `default_index` -- the single index /// An empty `tenant_id` resolves to `default_index` -- the single index
/// path every Phase 0-3 deployment already uses. This is intentionally /// path every Phase 0-3 deployment, and every untagged record, still
/// where the per-tenant story stops today: `consumer.rs`'s Redpanda /// uses.
/// consumer (the only thing that ever *writes* into an index) only ever ///
/// writes into `default_index`, because `ingest`/the log-record schema /// **Known residual gap, disclosed rather than silently accepted**:
/// itself carries no tenant concept yet -- see /// unlike the read side (gated by `enterprise/internal/searchclient`'s
/// /docs/security/threat-model.md's "ingest path... carries no tenant /// `TenantChecker`, which refuses to even issue a search for a tenant
/// concept" caveat. A tenant's own index therefore starts, and stays, /// that isn't `active` in `rbacstore`) and unlike ClickHouse's write
/// empty until something upstream of this service becomes tenant-aware /// side (`enterprise/internal/chwriter.Registry`, built once at startup
/// on the write side too. What this registry proves is that *read* /// from `rbacstore.ListProvisionedDataSources` -- `active` tenants
/// isolation is real once there's tenant-scoped data to isolate -- /// only, so an unrecognized `tenant_id` has no writer and the whole
/// exactly the same scope boundary enterprise/internal/chrunner drew for /// batch is refused), this registry's `resolve` has no equivalent gate
/// ClickHouse (see that package's doc comment). /// on the write path: `consumer.rs` calls it directly, with no Postgres
/// access to check tenant status against, the same reason this
/// module's doc comment used to give for the old read-side gap
/// `TenantChecker` was built to close. A syntactically-valid `tenant_id`
/// on an ingest credential that's still valid but should have been
/// revoked (deprovisioning does not yet revoke `ingest_credentials`
/// rows -- see `/CLAUDE.md`'s Phase 4 non-goals) can therefore cause an
/// index directory to be silently created here for a tenant that isn't
/// really active. The blast radius is narrow -- an orphan, isolated,
/// empty-except-for-that-tenant's-own-traffic index directory, not
/// cross-tenant data exposure, and only reachable with a real signed
/// ingest credential, not by an arbitrary caller -- but it is real, not
/// hypothetical. Closing it fully would mean giving `search` (AGPL
/// core, no `enterprise/` import allowed) some way to learn which
/// tenants are actually active; not designed yet.
pub struct IndexRegistry { pub struct IndexRegistry {
default_index: Arc<SearchIndex>, default_index: Arc<SearchIndex>,
tenants_root: PathBuf, tenants_root: PathBuf,
@@ -73,6 +92,36 @@ impl IndexRegistry {
.or_insert_with(|| Arc::new(opened)); .or_insert_with(|| Arc::new(opened));
Ok(Arc::clone(idx)) Ok(Arc::clone(idx))
} }
/// Commits `default_index` plus every tenant index opened so far --
/// the periodic-commit ticker in consumer.rs calls this instead of
/// committing a single index, now that a batch of records can span
/// several tenants' indices. An index that was never opened (no
/// write ever routed to it) is never touched, matching `resolve`'s
/// own on-demand-open behavior -- nothing to commit for a tenant
/// with no traffic yet. One tenant's commit failing does not stop
/// the others from being attempted -- a single broken index
/// shouldn't stall every other tenant's documents from becoming
/// searchable. Returns the last error encountered, if any, after
/// every index has been tried.
pub async fn commit_all(&self) -> Result<()> {
let mut last_err = self.default_index.commit().await.context("committing default index").err();
let tenants = self.tenants.read().await;
for (tenant_id, idx) in tenants.iter() {
if let Err(e) = idx
.commit()
.await
.with_context(|| format!("committing index for tenant {tenant_id:?}"))
{
tracing::error!(error = %e, tenant_id, "failed to commit tenant tantivy index");
last_err = Some(e);
}
}
match last_err {
Some(e) => Err(e),
None => Ok(()),
}
}
} }
/// Mirrors enterprise/internal/tenantprovision's tenantIdentifierPattern /// Mirrors enterprise/internal/tenantprovision's tenantIdentifierPattern
@@ -182,4 +231,37 @@ mod tests {
assert!(Arc::ptr_eq(&results[0], r), "expected every concurrent resolve to return the same Arc<SearchIndex>"); assert!(Arc::ptr_eq(&results[0], r), "expected every concurrent resolve to return the same Arc<SearchIndex>");
} }
} }
#[tokio::test]
async fn commit_all_commits_default_and_every_opened_tenant_index() {
let (registry, default_index, _dir) = new_test_registry();
default_index.upsert("default-1", "hello").await.unwrap();
let acme_idx = registry.resolve("acme").await.unwrap();
acme_idx.upsert("acme-1", "hello").await.unwrap();
let globex_idx = registry.resolve("globex").await.unwrap();
globex_idx.upsert("globex-1", "hello").await.unwrap();
// Nothing committed yet -- none of the three should be
// searchable, proving this test would actually catch commit_all
// silently skipping an index rather than passing vacuously.
assert!(default_index.search("hello", 10).unwrap().is_empty());
assert!(acme_idx.search("hello", 10).unwrap().is_empty());
assert!(globex_idx.search("hello", 10).unwrap().is_empty());
registry.commit_all().await.unwrap();
assert_eq!(default_index.search("hello", 10).unwrap(), vec!["default-1"]);
assert_eq!(acme_idx.search("hello", 10).unwrap(), vec!["acme-1"]);
assert_eq!(globex_idx.search("hello", 10).unwrap(), vec!["globex-1"]);
}
#[tokio::test]
async fn commit_all_is_a_noop_for_tenants_never_resolved() {
// A tenant with no traffic yet has no directory created at all --
// commit_all must not try to open/commit anything for it.
let (registry, _default, dir) = new_test_registry();
registry.commit_all().await.unwrap();
assert!(!dir.path().join("tenants").join("never-seen").exists());
}
} }