Build per-tenant Tantivy write-routing, closing the last ingest write gap

search/src/consumer.rs now resolves each record's tenant_id Kafka header
through the same IndexRegistry the read side (search/src/registry.rs +
enterprise/internal/searchclient) already used, and writes into that
tenant's own index instead of always the default one. The periodic
Tantivy commit now commits every tenant index that's actually seen a
write (IndexRegistry::commit_all), not just the default index.

Unlike ClickHouse, this needed no "second binary": Tantivy has no grant
system to gate a commercially-licensed credential behind, so
IndexRegistry already lived directly in this AGPL-core search binary --
there was never an import-boundary reason to split the write side into
an enterprise/ binary the way chwriter/enterprise-ingest was for
ClickHouse. Read and write simply share one registry.

Because Tantivy is an embedded library, this is genuinely verified in
this environment, not just written: registry.rs's
commit_all_commits_default_and_every_opened_tenant_index writes into the
default index plus two tenant indices, confirms nothing is searchable
pre-commit, then confirms all three are post-commit. consumer.rs's
tenant_id_from_headers is factored out as a small pure helper (mirroring
ingest/consumer.tenantIDFromHeaders) with its own unit tests, plus a
guard test against the "tenant_id" header-key literal drifting from the
Go side's -- the same guard-test pattern ingest/cmd/ingest already used
for its own two Go copies of the constant, now mirrored a third time
across the language boundary.

One gap is disclosed, not fixed, by this change: unlike chwriter.Registry
(an active-tenants-only snapshot built at enterprise-ingest startup, so
an unrecognized tenant_id is refused outright) and unlike the read side
(gated by searchclient.TenantChecker), this consumer's registry.resolve()
call has no active-tenant check at all -- search has no Postgres access
to check tenant status against. A still-valid-but-should-be-revoked
ingest credential can cause an index directory to be created for a
tenant that's no longer active. Narrow blast radius (an orphan, isolated,
empty index, not cross-tenant leakage, and only reachable with a real
signed credential), but real -- see registry.rs's doc comment on
resolve(). Closing it fully would mean giving search some way to learn
which tenants are active without an enterprise/ import, which isn't
designed yet.

This closes the last of Phase 4's ingest write-routing gaps (ClickHouse
was closed last commit). The one remaining gap in the whole phase is now
the tenant-picker frontend page, deliberately deferred earlier in this
phase as out of scope for this environment.
This commit is contained in:
2026-08-14 20:16:09 -07:00
parent 1de77b969f
commit bdd42e06f6
9 changed files with 458 additions and 151 deletions
+27 -12
View File
@@ -249,18 +249,33 @@ ClickHouse user `SELECT` only, which would have made every real
per-tenant write fail with a permission error — fixed by granting
`SELECT, INSERT` (one credential, both directions; no cross-tenant
boundary is crossed by also allowing INSERT within a tenant's own
database). **What's still deferred, clearly**: Tantivy's side.
`search/src/consumer.rs` is a completely independent Redpanda consumer
(not called through `ingest` or `enterprise-ingest` at all) and still
writes every record into the one shared (default) index regardless of
tenant — a different codebase (Rust) and a different process, real,
disclosed, separately-scoped remaining work, not just "the same fix
applied twice." What still keeps this phase from being done: the actual
tenant-picker *page* doesn't exist (`web` has no session/cookie-handling
code at all yet, and `enterprise-auth` has no CORS middleware for a
cross-origin `fetch` with credentials — both real, separately-scoped
frontend gaps), and Tantivy write-routing per the above. Full
accounting:
database). **Tantivy write-routing is now built too**
`search/src/consumer.rs` (a completely independent Redpanda consumer,
not called through `ingest` or `enterprise-ingest` at all: a different
codebase and process) now resolves each record's `tenant_id` header
through the *same* `IndexRegistry` the read side already used, and
routes the write there instead of always into the default index. Unlike
the ClickHouse side, this needed no "second binary": `IndexRegistry`
already lives in this AGPL-core binary (Tantivy has no grant system to
gate a commercially-licensed credential behind, so there was never an
import-boundary reason to split it out), so read and write share one
registry directly. The periodic Tantivy commit now commits every tenant
index that's seen a write, not just the default one
(`IndexRegistry::commit_all`). **One gap disclosed, not fixed, in this
same change**: unlike `chwriter.Registry` (built from an
active-tenants-only snapshot at startup) and unlike the read side (gated
by `searchclient.TenantChecker`), this consumer's `resolve()` call has
no active-tenant check — this process has no Postgres access to check
against, so a still-valid-but-should-be-revoked ingest credential can
cause an index directory to be created for a tenant that's no longer
active. Narrow blast radius (an orphan, isolated, empty index — not
cross-tenant leakage — and only reachable with a real signed
credential), but real; see `search/src/registry.rs`'s doc comment on
`resolve`. What still keeps this phase from being done: only the
tenant-picker *page* now — `web` has no session/cookie-handling code at
all yet, and `enterprise-auth` has no CORS middleware for a cross-origin
`fetch` with credentials, both real, separately-scoped frontend gaps.
Full accounting:
`/docs/security/threat-model.md`; step-by-step verification procedure
(not yet run against a live cluster in this environment):
`/docs/phase-4-runbook.md`. The rest of this section describes the exit