Build per-tenant Tantivy write-routing, closing the last ingest write gap

search/src/consumer.rs now resolves each record's tenant_id Kafka header
through the same IndexRegistry the read side (search/src/registry.rs +
enterprise/internal/searchclient) already used, and writes into that
tenant's own index instead of always the default one. The periodic
Tantivy commit now commits every tenant index that's actually seen a
write (IndexRegistry::commit_all), not just the default index.

Unlike ClickHouse, this needed no "second binary": Tantivy has no grant
system to gate a commercially-licensed credential behind, so
IndexRegistry already lived directly in this AGPL-core search binary --
there was never an import-boundary reason to split the write side into
an enterprise/ binary the way chwriter/enterprise-ingest was for
ClickHouse. Read and write simply share one registry.

Because Tantivy is an embedded library, this is genuinely verified in
this environment, not just written: registry.rs's
commit_all_commits_default_and_every_opened_tenant_index writes into the
default index plus two tenant indices, confirms nothing is searchable
pre-commit, then confirms all three are post-commit. consumer.rs's
tenant_id_from_headers is factored out as a small pure helper (mirroring
ingest/consumer.tenantIDFromHeaders) with its own unit tests, plus a
guard test against the "tenant_id" header-key literal drifting from the
Go side's -- the same guard-test pattern ingest/cmd/ingest already used
for its own two Go copies of the constant, now mirrored a third time
across the language boundary.

One gap is disclosed, not fixed, by this change: unlike chwriter.Registry
(an active-tenants-only snapshot built at enterprise-ingest startup, so
an unrecognized tenant_id is refused outright) and unlike the read side
(gated by searchclient.TenantChecker), this consumer's registry.resolve()
call has no active-tenant check at all -- search has no Postgres access
to check tenant status against. A still-valid-but-should-be-revoked
ingest credential can cause an index directory to be created for a
tenant that's no longer active. Narrow blast radius (an orphan, isolated,
empty index, not cross-tenant leakage, and only reachable with a real
signed credential), but real -- see registry.rs's doc comment on
resolve(). Closing it fully would mean giving search some way to learn
which tenants are active without an enterprise/ import, which isn't
designed yet.

This closes the last of Phase 4's ingest write-routing gaps (ClickHouse
was closed last commit). The one remaining gap in the whole phase is now
the tenant-picker frontend page, deliberately deferred earlier in this
phase as out of scope for this environment.
This commit is contained in:
2026-08-14 20:16:09 -07:00
parent 1de77b969f
commit bdd42e06f6
9 changed files with 458 additions and 151 deletions
+27 -12
View File
@@ -249,18 +249,33 @@ ClickHouse user `SELECT` only, which would have made every real
per-tenant write fail with a permission error — fixed by granting
`SELECT, INSERT` (one credential, both directions; no cross-tenant
boundary is crossed by also allowing INSERT within a tenant's own
database). **What's still deferred, clearly**: Tantivy's side.
`search/src/consumer.rs` is a completely independent Redpanda consumer
(not called through `ingest` or `enterprise-ingest` at all) and still
writes every record into the one shared (default) index regardless of
tenant — a different codebase (Rust) and a different process, real,
disclosed, separately-scoped remaining work, not just "the same fix
applied twice." What still keeps this phase from being done: the actual
tenant-picker *page* doesn't exist (`web` has no session/cookie-handling
code at all yet, and `enterprise-auth` has no CORS middleware for a
cross-origin `fetch` with credentials — both real, separately-scoped
frontend gaps), and Tantivy write-routing per the above. Full
accounting:
database). **Tantivy write-routing is now built too**
`search/src/consumer.rs` (a completely independent Redpanda consumer,
not called through `ingest` or `enterprise-ingest` at all: a different
codebase and process) now resolves each record's `tenant_id` header
through the *same* `IndexRegistry` the read side already used, and
routes the write there instead of always into the default index. Unlike
the ClickHouse side, this needed no "second binary": `IndexRegistry`
already lives in this AGPL-core binary (Tantivy has no grant system to
gate a commercially-licensed credential behind, so there was never an
import-boundary reason to split it out), so read and write share one
registry directly. The periodic Tantivy commit now commits every tenant
index that's seen a write, not just the default one
(`IndexRegistry::commit_all`). **One gap disclosed, not fixed, in this
same change**: unlike `chwriter.Registry` (built from an
active-tenants-only snapshot at startup) and unlike the read side (gated
by `searchclient.TenantChecker`), this consumer's `resolve()` call has
no active-tenant check — this process has no Postgres access to check
against, so a still-valid-but-should-be-revoked ingest credential can
cause an index directory to be created for a tenant that's no longer
active. Narrow blast radius (an orphan, isolated, empty index — not
cross-tenant leakage — and only reachable with a real signed
credential), but real; see `search/src/registry.rs`'s doc comment on
`resolve`. What still keeps this phase from being done: only the
tenant-picker *page* now — `web` has no session/cookie-handling code at
all yet, and `enterprise-auth` has no CORS middleware for a cross-origin
`fetch` with credentials, both real, separately-scoped frontend gaps.
Full accounting:
`/docs/security/threat-model.md`; step-by-step verification procedure
(not yet run against a live cluster in this environment):
`/docs/phase-4-runbook.md`. The rest of this section describes the exit
+28 -12
View File
@@ -139,8 +139,8 @@ escape hatch is opaque to any compiler-injected filter.
ran in the environment it was built in — Tantivy is an embedded
library, so the cross-tenant isolation probe needed no live database
or Docker to execute for real, and it passed.
- **Ingest identity and ClickHouse write-routing are now built; Tantivy's
isn't.** `ingest` (AGPL core) gained an optional `TenantResolver`
- **Ingest identity, ClickHouse write-routing, and Tantivy write-routing
are all now built.** `ingest` (AGPL core) gained an optional `TenantResolver`
(`ingest/internal/grpcserver`): an agent presents a per-tenant bearer
credential (`enterprise-auth -create-ingest-credential-tenant=<id>`
mints one, only its hash stored), validated over the network via a new
@@ -160,13 +160,28 @@ escape hatch is opaque to any compiler-injected filter.
`SELECT` only, which would have made every real per-tenant write fail
with a permission error — fixed by granting `SELECT, INSERT` (one
credential, both directions; no cross-tenant boundary is crossed by
also allowing INSERT within a tenant's own database). What isn't built
yet: `search`'s independent Redpanda consumer (a different codebase —
Rust — and a different process, not reachable through either `ingest`
or `enterprise-ingest`) still writes every record into the one shared
Tantivy index regardless of tenant. That's real, disclosed, separately
scoped remaining work, not something this change claims to have
closed.
also allowing INSERT within a tenant's own database). `search`'s
independent Redpanda consumer (a different codebase — Rust — and a
different process, not reachable through either `ingest` or
`enterprise-ingest`) is now write-routed too:
`search/src/consumer.rs` resolves each record's `tenant_id` header
through the *same* `IndexRegistry` the read side
(`search/src/registry.rs` + `enterprise/internal/searchclient`)
already used, and writes there instead of always into the default
index. No "second binary" needed here, unlike ClickHouse — Tantivy has
no grant system to gate a commercially-licensed credential behind, so
`IndexRegistry` already lived directly in this AGPL-core binary, and
read/write just share it. One gap is disclosed rather than closed by
this change: unlike `chwriter.Registry` (an active-tenants-only
snapshot built at startup) and unlike the read side (gated by
`searchclient.TenantChecker`), this consumer's `resolve()` call has no
active-tenant check — the process has no Postgres access to check
against — so a still-valid-but-should-be-revoked ingest credential can
cause an index directory to be created for a tenant that's no longer
active. Narrow blast radius (an orphan, isolated, empty index, not
cross-tenant leakage, and only reachable with a real signed
credential), but real; see `search/src/registry.rs`'s doc comment on
`resolve`.
- `deploy/operator`'s `Tenant` CRD and `enterprise-api -provision-tenant`
are now unified, deliberately lightweight: `-provision-tenant` stays
the sole real actor (ClickHouse + `rbacstore`), and now also syncs its
@@ -188,9 +203,10 @@ plain `api`), sharing a host-port/network-alias trick so `alerting`/
`web` need no conditional config either way. With both storage engines'
connection/index-layer mechanisms built, deployment topology enforced at
both the Helm and docker-compose layers, and the two provisioning
mechanisms unified, the largest remaining gap is ingest's per-tenant
*write-routing* (identity is now attached at ingest time; nothing
downstream of Redpanda consumes it yet to isolate the write, see above).
mechanisms unified, and both storage engines' write paths now
per-tenant-routed too (see above), the largest remaining gap in this
phase is the tenant-picker *frontend* page — the backend protocol is
built, but `web` has no session/cookie-handling code yet to call it.
## Licensing boundary
+76 -23
View File
@@ -38,6 +38,15 @@ of what was already run and passed. Two genuine exceptions:
"email" -- crewjam's own fake IdP hit this path. Same remaining gap as
OIDC: not yet tried against a real external IdP or a running
`enterprise-auth` container.
- Tantivy tenant isolation, both directions -- `search/src/registry.rs`'s
cross-tenant read isolation (§9) and, since this pass,
`search/src/consumer.rs`'s per-tenant write-routing (§14) -- verified
live, no disclaimer needed, because Tantivy is an embedded library with
no Docker/broker dependency: real indices, real documents, real
commits, run in this environment. The one thing about it that's still
unverified is not Tantivy itself but the upstream credential/header
plumbing feeding it (ingest's `TenantResolver`, `enterprise-auth`'s
`/internal/authorize-ingest`) against a real running stack.
Everything else — `internal/rbacstore`'s CRUD, the auth-enforcement
walkthrough, the dashboards tenant-scoping fix, the Helm chart, the
@@ -604,7 +613,7 @@ yet would refuse all ingest traffic outright rather than degrading
gracefully. See §14 below for what the identity is actually used for on
the write path.
## 14. Ingest write-routing (ClickHouse built, Tantivy not yet)
## 14. Ingest write-routing (both storage engines)
`enterprise/cmd/enterprise-ingest` (mirrors `enterprise-api`'s "second
binary" shape) consumes the `tenant_id` Kafka header §13 attaches and
@@ -643,12 +652,50 @@ never actually executed -- "the test exists" is not the same claim as
"write-routing is confirmed," same caveat §8 already states for the
read-side `chrunner` tests.
**Not built**: Tantivy's side of this. `search/src/consumer.rs` is a
completely independent Redpanda consumer, not called through `ingest` or
`enterprise-ingest` at all, and does not read the `tenant_id` header --
every record still lands in the one shared (default) Tantivy index
regardless of tenant. See CLAUDE.md and `/docs/security/threat-model.md`'s
"Read this first" for the full disclosure.
**Tantivy's side is built too, and genuinely verified.**
`search/src/consumer.rs` reads the same `tenant_id` Kafka header and
resolves it through `search/src/registry.rs`'s `IndexRegistry` -- the
same registry the read side (§9) already uses -- routing each record's
write into its own tenant's Tantivy index instead of the single default
one. No "second binary" was needed here the way ClickHouse needed
`enterprise-ingest`: Tantivy has no grant system to gate a
commercially-licensed credential behind, so `IndexRegistry` already
lives directly in AGPL-core `search`, and read/write just share it.
Because Tantivy is an embedded library (no Docker/broker needed to
exercise real logic), this actually ran in this environment:
```sh
cd search
cargo test --quiet
# registry.rs: commit_all_commits_default_and_every_opened_tenant_index
# writes into the default index plus two tenant indices, confirms
# nothing is searchable before commit_all(), then confirms all three ARE
# searchable after -- the write-routing + periodic-commit path
# end-to-end, for real, using the same real-Tantivy-index discipline as
# every other registry.rs/index.rs test. consumer.rs:
# tenant_id_from_headers_* cover the header-extraction helper
# (missing/present/unrelated-header cases) and
# test_tenant_id_header_key_matches_go guards the "tenant_id" literal
# against drifting from ingest/consumer.TenantIDHeaderKey /
# ingest/internal/grpcserver.TenantIDHeaderKey the same way
# ingest/cmd/ingest's own guard test does on the Go side.
```
**What's still not built, for either engine**: a live active-tenant
recheck at write time. `chwriter.Registry`'s per-tenant writer map is a
snapshot built once at `enterprise-ingest` startup from
`rbacstore.ListProvisionedDataSources` (active tenants only) -- an
unrecognized `tenant_id` is refused outright, but a tenant deprovisioned
*after* startup keeps writing successfully until the next restart.
`IndexRegistry.resolve()` has no allowlist at all on either the read or
write side -- `search` has no Postgres access to check tenant status
against -- so a still-valid-but-should-be-revoked ingest credential can
cause an orphan Tantivy index directory to be created for a tenant
that's no longer active. Narrow blast radius either way (isolated, not
cross-tenant leakage, reachable only with a real signed credential), but
real and disclosed, not silently accepted -- see
`search/src/registry.rs`'s doc comment on `resolve` and
`/docs/security/threat-model.md`'s "Read this first".
**Also not built**: Helm/`docker-compose.yml` do gate *whether*
`enterprise-ingest` runs at all (`ingest.requireTenantCredential`, same
@@ -688,22 +735,28 @@ Full accounting: `/docs/security/threat-model.md`. Headline items:
that split (declarative request vs. imperative provisioning action)
is intentional, not the "two disconnected sources of truth" gap this
bullet used to describe.
- **Ingest now has a real tenant identity (§13), and ClickHouse
write-routing is built (§14) -- Tantivy write-routing is the one gap
left.** An agent presents a bearer credential (`enterprise-auth
-create-ingest-credential-tenant=<id>`), `ingest/internal/grpcserver.
TenantResolver` validates it (fail-closed) and attaches the resolved
tenant ID to every record as a `tenant_id` Kafka message header.
`enterprise-ingest` reads that header back and routes each record's
ClickHouse write into its own tenant's database (not yet confirmed
against a real ClickHouse in this environment -- see §14). Nothing
reads that header on the Tantivy side yet -- every record still lands
in the one shared Tantivy index no matter what. A newly-provisioned
tenant's ClickHouse database is real, isolated, and actually
populated by write-routed traffic (once confirmed live); its Tantivy
index remains real, isolated at query time, and permanently empty
until Tantivy's write-routing split is built (a real, scoped
follow-up, no longer an undesigned one).
- **Ingest now has a real tenant identity (§13), and both storage
engines' write-routing is built (§14).** An agent presents a bearer
credential (`enterprise-auth -create-ingest-credential-tenant=<id>`),
`ingest/internal/grpcserver.TenantResolver` validates it (fail-closed)
and attaches the resolved tenant ID to every record as a `tenant_id`
Kafka message header. `enterprise-ingest` reads that header back and
routes each record's ClickHouse write into its own tenant's database
(not yet confirmed against a real ClickHouse in this environment --
see §14). `search/src/consumer.rs` reads the same header and routes
each record into its own tenant's Tantivy index -- genuinely verified
in this environment, unlike the ClickHouse side, since Tantivy needs
no Docker to exercise real logic. Both engines share one remaining,
disclosed gap: neither write path rechecks tenant-active status live
(ClickHouse: a startup-time snapshot, stale until restart; Tantivy: no
allowlist at all, since `search` has no Postgres access) -- narrow
blast radius, not cross-tenant leakage, but real; see §14 and
`/docs/security/threat-model.md`'s "Read this first". A
newly-provisioned tenant's ClickHouse database and Tantivy index are
both now real, isolated, and actually populated by write-routed
traffic (the ClickHouse claim pending live confirmation, the Tantivy
claim already verified) -- what used to be "permanently empty" for
both is no longer true for either.
- **Human SSO login now works for both OIDC (§3a) and SAML (§3b)** --
each verified with a real fake IdP (genuine cryptographic signing and
verification), not yet a real external IdP or a running
+77 -47
View File
@@ -9,15 +9,21 @@ for the full design rationale behind the controls described here.
## Read this first: the single most important open finding
**Updated a third time.** This section originally read "log data
**Updated a fourth time.** This section originally read "log data
queried through `POST /query` is not tenant-isolated at all," then
"ClickHouse is isolated but Tantivy isn't," then "ingest tags records
with a tenant identity but nothing routes the write." Both ClickHouse
*and* Tantivy connection/index-layer isolation are built on the *read*
path, and ClickHouse write-routing is now built too. What's left is
narrower but still real: **whether a given deployment actually runs the
isolated binaries**, and **Tantivy's write path**, which still has no
per-tenant routing at all.
with a tenant identity but nothing routes the write," then "ClickHouse
write-routing is built but Tantivy's isn't." Both storage engines are
now isolated on both the read and write paths. What's left is narrower:
**whether a given deployment actually runs the isolated binaries**
(deployment-time, not code-level), and two disclosed write-side gaps,
different in kind: ClickHouse's write registry is a startup-time
snapshot of active tenants with no live recheck (a *deprovisioned*
tenant can keep writing successfully until the next `enterprise-ingest`
restart), while Tantivy's write path has no active-tenant allowlist at
all — it opens an index for *any* syntactically-valid `tenant_id` a
record carries, active, deprovisioned, or never real. See below for
both, in full.
**ClickHouse (the SQL path) is built.** `enterprise/internal/
tenantprovision` (real `CREATE DATABASE`/`CREATE USER`/`GRANT`) and
@@ -70,43 +76,66 @@ provisioned, pointing at the same ClickHouse/Postgres. The Helm chart
makes the *default*, chart-managed path correct; it isn't a runtime
guard against misconfiguration.
**Ingest now has a real tenant identity, and ClickHouse write-routing is
built — Tantivy write-routing is the one gap left.** `chrunner`/
`searchclient` prove *read* isolation given tenant-scoped data exists; an
optional `ingest/internal/grpcserver.TenantResolver` closes the "does a
record know which tenant it belongs to" half by validating a per-tenant
bearer credential an agent presents
**Ingest now has a real tenant identity, and both storage engines'
write-routing is built.** `chrunner`/`searchclient` prove *read*
isolation given tenant-scoped data exists; an optional
`ingest/internal/grpcserver.TenantResolver` closes the "does a record
know which tenant it belongs to" half by validating a per-tenant bearer
credential an agent presents
(`enterprise-auth -create-ingest-credential-tenant=<id>` mints one; only
its SHA-256 hash is ever stored) against a `POST
/internal/authorize-ingest` endpoint, and attaching the resolved tenant
ID to every record as a `tenant_id` Kafka message header before
producing it — fail-closed: once a resolver is configured, a missing or
invalid credential refuses the whole batch, never falls back to "no
tenant." `enterprise/cmd/enterprise-ingest` (a second binary, mirroring
`enterprise-api`) now consumes that header: it reuses `ingest/consumer`'s
own flush loop with `enterprise/internal/chwriter.Registry` — a
per-tenant `*clickhousewriter.Writer` registry, built the same way
`chrunner`'s per-tenant connections are — swapped in as the writer, so a
tagged batch's records are grouped by tenant and each group INSERTed
through that tenant's own ClickHouse connection, fail-closed on an
untagged or unprovisioned tenant. Building it surfaced a real gap in an
already-shipped control: `tenantprovision.ProvisionClickHouse`'s grant
was `SELECT`-only (correct for the read-side credential `chrunner` uses,
but `chwriter` reuses the same credential for writes) — every real
per-tenant write would have failed with a permission error until this
was widened to `SELECT, INSERT`. **Not yet confirmed against a real
ClickHouse**, same caveat as the read-side chrunner claim above — the
Docker-free fail-closed tests pass, the live-database tests are written
but skip-gated, see `/docs/phase-4-runbook.md`. What's still missing:
`search`'s independent Redpanda consumer does not read the `tenant_id`
header at all — every ingested record still lands in the one shared
(default) Tantivy index regardless of tenant. A newly-provisioned
tenant's ClickHouse database is now real, isolated, and actually
populated by write-routed agent traffic (once confirmed against a real
cluster); its Tantivy index remains real, isolated, and queryable
through `enterprise-api` — and permanently empty, until Tantivy's own
write-routing split is built, which is now scoped, disclosed remaining
work, not an undesigned gap.
tenant."
**ClickHouse**: `enterprise/cmd/enterprise-ingest` (a second binary,
mirroring `enterprise-api`) consumes that header: it reuses
`ingest/consumer`'s own flush loop with `enterprise/internal/
chwriter.Registry` — a per-tenant `*clickhousewriter.Writer` registry,
built the same way `chrunner`'s per-tenant connections are — swapped in
as the writer, so a tagged batch's records are grouped by tenant and
each group INSERTed through that tenant's own ClickHouse connection,
fail-closed on an untagged or unprovisioned tenant. Building it surfaced
a real gap in an already-shipped control: `tenantprovision.
ProvisionClickHouse`'s grant was `SELECT`-only (correct for the
read-side credential `chrunner` uses, but `chwriter` reuses the same
credential for writes) — every real per-tenant write would have failed
with a permission error until this was widened to `SELECT, INSERT`.
**Not yet confirmed against a real ClickHouse**, same caveat as the
read-side chrunner claim above — the Docker-free fail-closed tests pass,
the live-database tests are written but skip-gated, see
`/docs/phase-4-runbook.md`.
**Tantivy**: `search/src/consumer.rs` now resolves each record's
`tenant_id` header through the *same* `IndexRegistry` the read side
already used (`search/src/registry.rs`), and writes there instead of
always into the default index — genuinely verified in this environment,
same as the read-side Tantivy claim above, since Tantivy is an embedded
library with no Docker dependency. No "second binary" was needed here,
unlike ClickHouse: Tantivy has no grant system to gate a
commercially-licensed credential behind, so `IndexRegistry` already
lived directly in this AGPL-core `search` binary, and read/write simply
share it.
**Both engines share one open question** — deployment topology, covered
above — and Tantivy specifically has one gap ClickHouse's design doesn't:
`chwriter.Registry`'s map is built once at startup from an
active-tenants-only query, so an unrecognized `tenant_id` is refused
outright; `IndexRegistry.resolve()` (used for both read and write) has
no equivalent allowlist at all, because `search` has no Postgres access
to check tenant status against — a syntactically-valid `tenant_id` on a
still-valid-but-should-be-revoked ingest credential can cause an orphan
index directory to be created for a tenant that's no longer active.
Narrow blast radius (isolated, empty except for that traffic, not
cross-tenant leakage, and reachable only with a real signed credential),
but real, and not closed by this change; see
`search/src/registry.rs`'s doc comment on `resolve`. A newly-provisioned
tenant's ClickHouse database and Tantivy index are both now real,
isolated, and actually populated by write-routed agent traffic (the
ClickHouse claim pending live confirmation, the Tantivy claim already
verified).
## System overview
@@ -146,13 +175,14 @@ sentryctl ──▶ api, alerting (Bearer token when SENTRYCTL_TOKEN is set)
Ingest path (agent → Redpanda → ingest → ClickHouse, and Redpanda →
search → Tantivy): `ingest` resolves and tags each record with a real
tenant ID (see "Read this first" above). `enterprise/cmd/enterprise-ingest`
now consumes that tag and routes ClickHouse writes to each tenant's own
database. Tantivy's independent Redpanda consumer (`search/src/
consumer.rs`) still does not consume the tag at all — every record still
lands in the one shared Tantivy index regardless of tenant. That
narrower gap is still out of scope for what's built so far and is not
separately designed in `/docs/phase-4-isolation-design.md`; named here as
a gap that design doc doesn't yet cover, not just an implementation gap.
consumes that tag and routes ClickHouse writes to each tenant's own
database. `search/src/consumer.rs` (Tantivy's independent Redpanda
consumer) consumes the same tag and routes each record into its own
tenant's index. Both write paths' one remaining gap — no live
active-tenant recheck (ClickHouse: a startup-time snapshot; Tantivy: no
allowlist at all) — is named in "Read this first" above, not separately
designed in `/docs/phase-4-isolation-design.md`; named here as a gap that
design doc doesn't yet cover, not just an implementation gap.
## Module boundary (trust boundary #1)
@@ -454,8 +484,8 @@ terms:
| Tantivy per-tenant index routing (`search/src/registry.rs`) | **Enforced, verified live** — real Tantivy indices, real cross-tenant probe, all passing |
| Tantivy tenant_id resolution (`enterprise/internal/searchclient`) | **Enforced, verified live** — real gRPC wire-level test |
| Ingest tenant *identity* (credential validation, tagging) | **Built and tested** — fail-closed `TenantResolver`, `tenant_id` Kafka header attached per record |
| Ingest tenant *write-routing*, ClickHouse | **Built, not yet confirmed against a real ClickHouse**`enterprise-ingest`/`chwriter.Registry` route each tagged batch to its tenant's own database, fail-closed on an untagged/unprovisioned tenant; Docker-free tests pass, live-database tests are skip-gated |
| Ingest tenant *write-routing*, Tantivy | **Not implemented, now scoped** — every record still lands in the single shared Tantivy index regardless of tenant; `search/src/consumer.rs` consuming the tenant_id header to route the write is real, disclosed remaining work |
| Ingest tenant *write-routing*, ClickHouse | **Built, not yet confirmed against a real ClickHouse**`enterprise-ingest`/`chwriter.Registry` route each tagged batch to its tenant's own database, fail-closed on an untagged/unprovisioned tenant; Docker-free tests pass, live-database tests are skip-gated. Startup-time active-tenant snapshot, no live recheck — a deprovisioned tenant can keep writing until the next restart |
| Ingest tenant *write-routing*, Tantivy | **Built and genuinely verified**`search/src/consumer.rs` routes each record into its own tenant's index via `IndexRegistry`, same registry the (already-verified) read side uses; no Docker needed, real tests pass. No active-tenant allowlist at all on write (`search` has no Postgres access) — a syntactically-valid `tenant_id` on a still-valid credential can create an orphan index for a no-longer-active tenant; narrow, disclosed, not cross-tenant leakage |
| Deployment actually routing traffic to `enterprise-api` (Helm) | **Enforced**`api`/`enterprise-api` are mutually exclusive, same flag as RBAC/audit/SSO |
| Deployment actually routing traffic to `enterprise-api` (docker-compose) | **Enforced**`api`/`enterprise-api` are mutually exclusive via `COMPOSE_PROFILES`, same flag choice as Helm's `enterprise.enabled`; verified via `docker compose config`, not an actual `docker compose up` in this environment |
| Human SSO login — OIDC | **Built, verified with a real fake IdP** (not yet tried against a real external IdP) |
+14 -10
View File
@@ -182,14 +182,13 @@ silently left out:
a cross-origin `fetch` with credentials from `web`'s origin needs
it), neither of which is verifiable in this environment without a
live backend and a browser session to exercise.
- **Ingest write-routing, for Tantivy** -- `search/src/consumer.rs` (a
completely independent Redpanda consumer, not called through `ingest`
or `enterprise-ingest` at all) still writes every record into the one
shared (default) Tantivy index, regardless of tenant. The ClickHouse
half is now built (see "Ingest write-routing" below); Tantivy's is
real, disclosed, separate follow-up work -- a different codebase
(Rust) and a different consumer process, not just "the same fix
applied twice."
Ingest write-routing (both ClickHouse and Tantivy) is no longer on this
list -- see "Ingest write-routing (ClickHouse)" below and
`/search/README.md`'s "Per-tenant indices" section for Tantivy, which
needed no code in this module at all: `search`'s `IndexRegistry` already
lived in AGPL core, so its write side didn't need an `enterprise/`
counterpart the way ClickHouse's did.
Deployment-topology routing (does traffic actually reach `enterprise-api`
instead of `api`) is no longer deferred -- both `deploy/helm/sentry` and
@@ -307,8 +306,13 @@ ingest must never import enterprise/) is the client side of `internal/
authhandler`'s new `POST /internal/authorize-ingest` -- see "Ingest
tenant identity" above.
Future additions: per-tenant write-routing for ingest (ClickHouse and
Tantivy both) -- see "Ingest tenant identity" above.
Per-tenant write-routing for ingest is built for both ClickHouse (this
module's `cmd/enterprise-ingest` + `internal/chwriter`, see "Ingest
write-routing (ClickHouse)" above) and Tantivy (`search/src/consumer.rs`
+ `search/src/registry.rs`, entirely in AGPL core -- see
`/search/README.md`'s "Per-tenant indices" section, since Tantivy's lack
of a grant system meant there was never an import-boundary reason to put
any of it here).
## Why OIDC and SAML aren't hand-rolled
+39 -13
View File
@@ -38,7 +38,7 @@ ClickHouse's `logs.record_id` column (see
only ever returns IDs, never row data — it stays a pure text index, not
a second copy of the row.
## Per-tenant indices (Phase 4) — read-side only
## Per-tenant indices (Phase 4) — read and write
`SearchRequest.tenant_id` (empty by default) selects which index
`src/registry.rs`'s `IndexRegistry` searches: empty resolves to the
@@ -50,13 +50,31 @@ convention). `tenant_id` is set only by a trusted server-side caller
(`enterprise/internal/searchclient`, from the authenticated request
identity) — never a value a browser/client controls.
**This is read-side isolation only.** `consumer.rs`'s Redpanda consumer
— the only thing that ever *writes* into an indexstill only ever
writes into the single default index, because `ingest`/the log-record
schema itself carries no tenant concept yet. A newly-opened tenant index
starts, and stays, empty until something upstream of this service
becomes tenant-aware on the write side too — a real, disclosed gap, not
an oversight; see `/docs/security/threat-model.md`.
`consumer.rs`'s Redpanda consumer resolves the *same* registry now,
keyed by each Kafka message's `tenant_id` headerattached server-side
by `ingest/internal/grpcserver` after validating an agent's per-tenant
credential, mirroring exactly how `enterprise/cmd/enterprise-ingest`
routes the ClickHouse write side (see `/enterprise/README.md`'s "Ingest
write-routing" section). No import boundary to work around here, unlike
Go: `IndexRegistry` already lives in this AGPL-core binary, so both the
read and write paths share one registry directly, with no "second
binary" needed. The periodic Tantivy commit (`COMMIT_INTERVAL_MS`) now
commits every tenant index that's actually seen a write, plus the
default index, via `IndexRegistry::commit_all`, not just one index.
**One residual gap, disclosed rather than fixed here**: unlike the read
side (gated by `enterprise/internal/searchclient`'s `TenantChecker`,
which refuses to search a tenant that isn't `active` in `rbacstore`) and
unlike ClickHouse's write side (`chwriter.Registry`, built from an
active-tenants-only snapshot at startup, so an unrecognized tenant has
no writer at all), this consumer's `registry.resolve()` call has no
active-tenant gate — this process has no Postgres access to check
against, the same reason `IndexRegistry` couldn't do the mid-provisioning
check itself before `TenantChecker` was added for the read side. A
still-valid (not yet revoked) ingest credential for a tenant that's no
longer active can cause an index directory to be created for it here.
See `src/registry.rs`'s doc comment on `resolve` for the full writeup,
including why closing it fully isn't scoped yet.
## Offset tracking: why this isn't a Kafka consumer group
@@ -116,12 +134,20 @@ before-commit/after-commit visibility boundary. `registry.rs`'s tests
are the same shape and cover the actual adversarial claim: real per-
tenant indices, real documents, and a search scoped to one tenant
returning zero results for another tenant's matching document
(`tenant_index_is_isolated_from_default_and_other_tenants`) — all of
this, unlike almost everything else in Phase 4, was genuinely run in
the environment this was built in, not just written. `consumer.rs` (the
rskafka wiring) is not unit-tested — that needs a real Redpanda, same
(`tenant_index_is_isolated_from_default_and_other_tenants`), plus
`commit_all_commits_default_and_every_opened_tenant_index` proving the
write-routing commit path actually makes every tenant's buffered writes
searchable, not just the default index's — all of this, unlike almost
everything else in Phase 4, was genuinely run in the environment this
was built in, not just written. `consumer.rs`'s Kafka/rskafka wiring
itself is still not unit-tested — that needs a real Redpanda, same
category of gap as `/ingest`'s `kafka.Reader`/`kafka.Writer` wiring, and
is exercised by the docker-compose end-to-end flow instead.
is exercised by the docker-compose end-to-end flow instead — but the
pure `tenant_id_from_headers` header-extraction helper it uses is
factored out and unit-tested the same way `ingest/consumer`'s Go
equivalent is, including a guard test
(`test_tenant_id_header_key_matches_go`) against the header-key literal
drifting from the Go side's.
```sh
# from the repo root, not search/
+91 -9
View File
@@ -2,13 +2,25 @@ use anyhow::{Context, Result};
use prost::Message;
use rskafka::client::partition::UnknownTopicHandling;
use rskafka::client::ClientBuilder;
use std::collections::BTreeMap;
use std::sync::Arc;
use tokio::sync::Mutex;
use crate::config::Config;
use crate::index::SearchIndex;
use crate::logsv1;
use crate::offsets::OffsetStore;
use crate::registry::IndexRegistry;
/// Kafka message header a resolved tenant ID rides in, attached by
/// ingest's gRPC front end. Mirrors `ingest/internal/grpcserver.
/// TenantIDHeaderKey` / `ingest/consumer.TenantIDHeaderKey` -- those two
/// Go packages duplicate the same literal rather than importing across a
/// producer/consumer boundary (see their doc comments), and this Rust
/// consumer is a third independent reader of the same header, so it
/// duplicates the literal too. `TestTenantIDHeaderKeyMatchesGo` below
/// guards against drift the same way `ingest/cmd/ingest`'s
/// `TestTenantIDHeaderKeyConstantsMatch` does on the Go side.
const TENANT_ID_HEADER_KEY: &str = "tenant_id";
/// Reads the same `sentry.logs.raw` topic ingest's ClickHouse-writer
/// consumer reads, as an independent consumer group in spirit (its own
@@ -18,7 +30,7 @@ use crate::offsets::OffsetStore;
/// discovered dynamically, since it has to match what
/// /transport/provision-topics.sh actually created anyway (documented
/// cross-component contract, same as the topic name already is).
pub async fn run(cfg: Arc<Config>, index: Arc<SearchIndex>, partition_count: i32) -> Result<()> {
pub async fn run(cfg: Arc<Config>, registry: Arc<IndexRegistry>, partition_count: i32) -> Result<()> {
let client = ClientBuilder::new(cfg.redpanda_brokers.clone())
.build()
.await
@@ -32,14 +44,16 @@ pub async fn run(cfg: Arc<Config>, index: Arc<SearchIndex>, partition_count: i32
// Periodic Tantivy commit, batched for throughput the same way
// ingest's ClickHouse writer batches inserts rather than inserting
// per-record.
// per-record. Commits every tenant index a write has actually been
// routed to (plus the default index), not just one -- see
// registry.rs's commit_all doc comment.
let commit_interval = cfg.commit_interval;
let index_for_commit = Arc::clone(&index);
let registry_for_commit = Arc::clone(&registry);
tokio::spawn(async move {
let mut ticker = tokio::time::interval(commit_interval);
loop {
ticker.tick().await;
if let Err(e) = index_for_commit.commit().await {
if let Err(e) = registry_for_commit.commit_all().await {
tracing::error!(error = %e, "periodic tantivy commit failed");
}
}
@@ -49,11 +63,11 @@ pub async fn run(cfg: Arc<Config>, index: Arc<SearchIndex>, partition_count: i32
for partition in 0..partition_count {
let start_offset = offsets.lock().await.get(partition);
let client = Arc::clone(&client);
let index = Arc::clone(&index);
let registry = Arc::clone(&registry);
let offsets = Arc::clone(&offsets);
let topic = cfg.redpanda_topic.clone();
handles.push(tokio::spawn(async move {
consume_partition(client, topic, partition, start_offset, index, offsets).await
consume_partition(client, topic, partition, start_offset, registry, offsets).await
}));
}
@@ -65,13 +79,26 @@ pub async fn run(cfg: Arc<Config>, index: Arc<SearchIndex>, partition_count: i32
Ok(())
}
/// Extracts the `tenant_id` header's value, or "" if absent -- an empty
/// string is exactly what `IndexRegistry::resolve` treats as "route to
/// the default index," so an untagged record (every Phase 0-3 message,
/// and any Phase 4 message from a deployment that never turned on
/// `ingest`'s `TenantResolver`) keeps landing in the same shared index
/// it always has. Mirrors `ingest/consumer.tenantIDFromHeaders` exactly.
fn tenant_id_from_headers(headers: &BTreeMap<String, Vec<u8>>) -> String {
headers
.get(TENANT_ID_HEADER_KEY)
.map(|v| String::from_utf8_lossy(v).into_owned())
.unwrap_or_default()
}
#[allow(clippy::too_many_arguments)]
async fn consume_partition(
client: Arc<rskafka::client::Client>,
topic: String,
partition: i32,
start_offset: i64,
index: Arc<SearchIndex>,
registry: Arc<IndexRegistry>,
offsets: Arc<Mutex<OffsetStore>>,
) -> Result<()> {
let partition_client = client
@@ -115,8 +142,25 @@ async fn consume_partition(
continue;
}
let tenant_id = tenant_id_from_headers(&record_and_offset.record.headers);
let index = match registry.resolve(&tenant_id).await {
Ok(index) => index,
Err(e) => {
// Shouldn't normally happen -- ingest/grpcserver only
// ever attaches a tenant_id it validated against a
// real credential -- but an unsafe/malformed
// tenant_id is a hard skip, never a silent fall-back
// to the default or any other tenant's index. See
// registry.rs's doc comment for the one residual gap
// this consumer doesn't close (no active-tenant
// check, since this process has no Postgres access).
tracing::error!(error = %e, record_id = %rec.record_id, tenant_id, "skipping record: failed to resolve tenant index");
continue;
}
};
if let Err(e) = index.upsert(&rec.record_id, &rec.message).await {
tracing::error!(error = %e, record_id = %rec.record_id, "failed to index record");
tracing::error!(error = %e, record_id = %rec.record_id, tenant_id, "failed to index record");
}
}
@@ -132,3 +176,41 @@ async fn consume_partition(
}
}
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn tenant_id_from_headers_returns_empty_string_when_absent() {
let headers = BTreeMap::new();
assert_eq!(tenant_id_from_headers(&headers), "");
}
#[test]
fn tenant_id_from_headers_extracts_the_tenant_id_header() {
let mut headers = BTreeMap::new();
headers.insert(TENANT_ID_HEADER_KEY.to_string(), b"acme".to_vec());
assert_eq!(tenant_id_from_headers(&headers), "acme");
}
#[test]
fn tenant_id_from_headers_ignores_unrelated_headers() {
let mut headers = BTreeMap::new();
headers.insert("some-other-header".to_string(), b"acme".to_vec());
assert_eq!(tenant_id_from_headers(&headers), "");
}
/// Guards against the exact literal drift
/// `ingest/cmd/ingest`'s `TestTenantIDHeaderKeyConstantsMatch` guards
/// against on the Go side -- three independent readers/writers of the
/// same Kafka header (`ingest/internal/grpcserver` producing,
/// `ingest/consumer` and this file both consuming) duplicate the same
/// literal by design rather than sharing an import across a
/// producer/consumer or language boundary, so nothing but a test
/// catches them drifting apart.
#[test]
fn test_tenant_id_header_key_matches_go() {
assert_eq!(TENANT_ID_HEADER_KEY, "tenant_id");
}
}
+6 -7
View File
@@ -35,11 +35,10 @@ async fn main() -> Result<()> {
let index = Arc::new(
SearchIndex::open_or_create(&cfg.index_path).context("opening tantivy index")?,
);
// Per-tenant indices (Phase 4) are resolved on demand by
// IndexRegistry, opened under cfg.tenants_index_path -- see
// registry.rs's doc comment for what this does and doesn't isolate
// yet (read-side only; the consumer below still only ever writes
// into the single default `index` above).
// Per-tenant indices are resolved on demand by IndexRegistry, opened
// under cfg.tenants_index_path -- see registry.rs's doc comment for
// what this isolates (both read and write now) and the one residual
// gap it doesn't close.
let registry = Arc::new(IndexRegistry::new(Arc::clone(&index), cfg.tenants_index_path.clone()));
let partition_count: i32 = std::env::var("REDPANDA_TOPIC_PARTITIONS")
@@ -48,9 +47,9 @@ async fn main() -> Result<()> {
.unwrap_or(DEFAULT_PARTITION_COUNT);
let consumer_cfg = Arc::clone(&cfg);
let consumer_index = Arc::clone(&index);
let consumer_registry = Arc::clone(&registry);
let consumer_handle = tokio::spawn(async move {
if let Err(e) = consumer::run(consumer_cfg, consumer_index, partition_count).await {
if let Err(e) = consumer::run(consumer_cfg, consumer_registry, partition_count).await {
tracing::error!(error = %e, "redpanda consumer exited with error");
}
});
+100 -18
View File
@@ -6,26 +6,45 @@ use tokio::sync::RwLock;
use crate::index::SearchIndex;
/// Resolves a `tenant_id` (from `SearchRequest.tenant_id`, per
/// search.proto's doc comment: set only by a trusted server-side caller
/// -- `enterprise/internal/searchclient`, from the authenticated request
/// identity, never a value a browser/client controls directly) to its
/// own `SearchIndex`, opening one on demand under `<tenants_root>/
/// <tenant_id>` the first time it's requested.
/// Resolves a `tenant_id` to its own `SearchIndex`, opening one on
/// demand under `<tenants_root>/<tenant_id>` the first time it's
/// requested. Used on both sides now: the read side
/// (`SearchRequest.tenant_id`, per search.proto's doc comment -- set
/// only by a trusted server-side caller, `enterprise/internal/
/// searchclient`, from the authenticated request identity, never a
/// value a browser/client controls directly) and the write side
/// (`consumer.rs`, from a Kafka message's `tenant_id` header, attached
/// server-side by `ingest/internal/grpcserver` after validating an
/// agent's per-tenant credential -- never a value the agent's message
/// body controls directly either).
///
/// An empty `tenant_id` resolves to `default_index` -- the single index
/// path every Phase 0-3 deployment already uses. This is intentionally
/// where the per-tenant story stops today: `consumer.rs`'s Redpanda
/// consumer (the only thing that ever *writes* into an index) only ever
/// writes into `default_index`, because `ingest`/the log-record schema
/// itself carries no tenant concept yet -- see
/// /docs/security/threat-model.md's "ingest path... carries no tenant
/// concept" caveat. A tenant's own index therefore starts, and stays,
/// empty until something upstream of this service becomes tenant-aware
/// on the write side too. What this registry proves is that *read*
/// isolation is real once there's tenant-scoped data to isolate --
/// exactly the same scope boundary enterprise/internal/chrunner drew for
/// ClickHouse (see that package's doc comment).
/// path every Phase 0-3 deployment, and every untagged record, still
/// uses.
///
/// **Known residual gap, disclosed rather than silently accepted**:
/// unlike the read side (gated by `enterprise/internal/searchclient`'s
/// `TenantChecker`, which refuses to even issue a search for a tenant
/// that isn't `active` in `rbacstore`) and unlike ClickHouse's write
/// side (`enterprise/internal/chwriter.Registry`, built once at startup
/// from `rbacstore.ListProvisionedDataSources` -- `active` tenants
/// only, so an unrecognized `tenant_id` has no writer and the whole
/// batch is refused), this registry's `resolve` has no equivalent gate
/// on the write path: `consumer.rs` calls it directly, with no Postgres
/// access to check tenant status against, the same reason this
/// module's doc comment used to give for the old read-side gap
/// `TenantChecker` was built to close. A syntactically-valid `tenant_id`
/// on an ingest credential that's still valid but should have been
/// revoked (deprovisioning does not yet revoke `ingest_credentials`
/// rows -- see `/CLAUDE.md`'s Phase 4 non-goals) can therefore cause an
/// index directory to be silently created here for a tenant that isn't
/// really active. The blast radius is narrow -- an orphan, isolated,
/// empty-except-for-that-tenant's-own-traffic index directory, not
/// cross-tenant data exposure, and only reachable with a real signed
/// ingest credential, not by an arbitrary caller -- but it is real, not
/// hypothetical. Closing it fully would mean giving `search` (AGPL
/// core, no `enterprise/` import allowed) some way to learn which
/// tenants are actually active; not designed yet.
pub struct IndexRegistry {
default_index: Arc<SearchIndex>,
tenants_root: PathBuf,
@@ -73,6 +92,36 @@ impl IndexRegistry {
.or_insert_with(|| Arc::new(opened));
Ok(Arc::clone(idx))
}
/// Commits `default_index` plus every tenant index opened so far --
/// the periodic-commit ticker in consumer.rs calls this instead of
/// committing a single index, now that a batch of records can span
/// several tenants' indices. An index that was never opened (no
/// write ever routed to it) is never touched, matching `resolve`'s
/// own on-demand-open behavior -- nothing to commit for a tenant
/// with no traffic yet. One tenant's commit failing does not stop
/// the others from being attempted -- a single broken index
/// shouldn't stall every other tenant's documents from becoming
/// searchable. Returns the last error encountered, if any, after
/// every index has been tried.
pub async fn commit_all(&self) -> Result<()> {
let mut last_err = self.default_index.commit().await.context("committing default index").err();
let tenants = self.tenants.read().await;
for (tenant_id, idx) in tenants.iter() {
if let Err(e) = idx
.commit()
.await
.with_context(|| format!("committing index for tenant {tenant_id:?}"))
{
tracing::error!(error = %e, tenant_id, "failed to commit tenant tantivy index");
last_err = Some(e);
}
}
match last_err {
Some(e) => Err(e),
None => Ok(()),
}
}
}
/// Mirrors enterprise/internal/tenantprovision's tenantIdentifierPattern
@@ -182,4 +231,37 @@ mod tests {
assert!(Arc::ptr_eq(&results[0], r), "expected every concurrent resolve to return the same Arc<SearchIndex>");
}
}
#[tokio::test]
async fn commit_all_commits_default_and_every_opened_tenant_index() {
let (registry, default_index, _dir) = new_test_registry();
default_index.upsert("default-1", "hello").await.unwrap();
let acme_idx = registry.resolve("acme").await.unwrap();
acme_idx.upsert("acme-1", "hello").await.unwrap();
let globex_idx = registry.resolve("globex").await.unwrap();
globex_idx.upsert("globex-1", "hello").await.unwrap();
// Nothing committed yet -- none of the three should be
// searchable, proving this test would actually catch commit_all
// silently skipping an index rather than passing vacuously.
assert!(default_index.search("hello", 10).unwrap().is_empty());
assert!(acme_idx.search("hello", 10).unwrap().is_empty());
assert!(globex_idx.search("hello", 10).unwrap().is_empty());
registry.commit_all().await.unwrap();
assert_eq!(default_index.search("hello", 10).unwrap(), vec!["default-1"]);
assert_eq!(acme_idx.search("hello", 10).unwrap(), vec!["acme-1"]);
assert_eq!(globex_idx.search("hello", 10).unwrap(), vec!["globex-1"]);
}
#[tokio::test]
async fn commit_all_is_a_noop_for_tenants_never_resolved() {
// A tenant with no traffic yet has no directory created at all --
// commit_all must not try to open/commit anything for it.
let (registry, _default, dir) = new_test_registry();
registry.commit_all().await.unwrap();
assert!(!dir.path().join("tenants").join("never-seen").exists());
}
}