Build per-tenant Tantivy write-routing, closing the last ingest write gap
search/src/consumer.rs now resolves each record's tenant_id Kafka header through the same IndexRegistry the read side (search/src/registry.rs + enterprise/internal/searchclient) already used, and writes into that tenant's own index instead of always the default one. The periodic Tantivy commit now commits every tenant index that's actually seen a write (IndexRegistry::commit_all), not just the default index. Unlike ClickHouse, this needed no "second binary": Tantivy has no grant system to gate a commercially-licensed credential behind, so IndexRegistry already lived directly in this AGPL-core search binary -- there was never an import-boundary reason to split the write side into an enterprise/ binary the way chwriter/enterprise-ingest was for ClickHouse. Read and write simply share one registry. Because Tantivy is an embedded library, this is genuinely verified in this environment, not just written: registry.rs's commit_all_commits_default_and_every_opened_tenant_index writes into the default index plus two tenant indices, confirms nothing is searchable pre-commit, then confirms all three are post-commit. consumer.rs's tenant_id_from_headers is factored out as a small pure helper (mirroring ingest/consumer.tenantIDFromHeaders) with its own unit tests, plus a guard test against the "tenant_id" header-key literal drifting from the Go side's -- the same guard-test pattern ingest/cmd/ingest already used for its own two Go copies of the constant, now mirrored a third time across the language boundary. One gap is disclosed, not fixed, by this change: unlike chwriter.Registry (an active-tenants-only snapshot built at enterprise-ingest startup, so an unrecognized tenant_id is refused outright) and unlike the read side (gated by searchclient.TenantChecker), this consumer's registry.resolve() call has no active-tenant check at all -- search has no Postgres access to check tenant status against. A still-valid-but-should-be-revoked ingest credential can cause an index directory to be created for a tenant that's no longer active. Narrow blast radius (an orphan, isolated, empty index, not cross-tenant leakage, and only reachable with a real signed credential), but real -- see registry.rs's doc comment on resolve(). Closing it fully would mean giving search some way to learn which tenants are active without an enterprise/ import, which isn't designed yet. This closes the last of Phase 4's ingest write-routing gaps (ClickHouse was closed last commit). The one remaining gap in the whole phase is now the tenant-picker frontend page, deliberately deferred earlier in this phase as out of scope for this environment.
This commit is contained in:
@@ -249,18 +249,33 @@ ClickHouse user `SELECT` only, which would have made every real
|
||||
per-tenant write fail with a permission error — fixed by granting
|
||||
`SELECT, INSERT` (one credential, both directions; no cross-tenant
|
||||
boundary is crossed by also allowing INSERT within a tenant's own
|
||||
database). **What's still deferred, clearly**: Tantivy's side.
|
||||
`search/src/consumer.rs` is a completely independent Redpanda consumer
|
||||
(not called through `ingest` or `enterprise-ingest` at all) and still
|
||||
writes every record into the one shared (default) index regardless of
|
||||
tenant — a different codebase (Rust) and a different process, real,
|
||||
disclosed, separately-scoped remaining work, not just "the same fix
|
||||
applied twice." What still keeps this phase from being done: the actual
|
||||
tenant-picker *page* doesn't exist (`web` has no session/cookie-handling
|
||||
code at all yet, and `enterprise-auth` has no CORS middleware for a
|
||||
cross-origin `fetch` with credentials — both real, separately-scoped
|
||||
frontend gaps), and Tantivy write-routing per the above. Full
|
||||
accounting:
|
||||
database). **Tantivy write-routing is now built too** —
|
||||
`search/src/consumer.rs` (a completely independent Redpanda consumer,
|
||||
not called through `ingest` or `enterprise-ingest` at all: a different
|
||||
codebase and process) now resolves each record's `tenant_id` header
|
||||
through the *same* `IndexRegistry` the read side already used, and
|
||||
routes the write there instead of always into the default index. Unlike
|
||||
the ClickHouse side, this needed no "second binary": `IndexRegistry`
|
||||
already lives in this AGPL-core binary (Tantivy has no grant system to
|
||||
gate a commercially-licensed credential behind, so there was never an
|
||||
import-boundary reason to split it out), so read and write share one
|
||||
registry directly. The periodic Tantivy commit now commits every tenant
|
||||
index that's seen a write, not just the default one
|
||||
(`IndexRegistry::commit_all`). **One gap disclosed, not fixed, in this
|
||||
same change**: unlike `chwriter.Registry` (built from an
|
||||
active-tenants-only snapshot at startup) and unlike the read side (gated
|
||||
by `searchclient.TenantChecker`), this consumer's `resolve()` call has
|
||||
no active-tenant check — this process has no Postgres access to check
|
||||
against, so a still-valid-but-should-be-revoked ingest credential can
|
||||
cause an index directory to be created for a tenant that's no longer
|
||||
active. Narrow blast radius (an orphan, isolated, empty index — not
|
||||
cross-tenant leakage — and only reachable with a real signed
|
||||
credential), but real; see `search/src/registry.rs`'s doc comment on
|
||||
`resolve`. What still keeps this phase from being done: only the
|
||||
tenant-picker *page* now — `web` has no session/cookie-handling code at
|
||||
all yet, and `enterprise-auth` has no CORS middleware for a cross-origin
|
||||
`fetch` with credentials, both real, separately-scoped frontend gaps.
|
||||
Full accounting:
|
||||
`/docs/security/threat-model.md`; step-by-step verification procedure
|
||||
(not yet run against a live cluster in this environment):
|
||||
`/docs/phase-4-runbook.md`. The rest of this section describes the exit
|
||||
|
||||
Reference in New Issue
Block a user