Build per-tenant ClickHouse write-routing for ingest (Tantivy still deferred)
ingest tags every record with a tenant_id Kafka header (built previously), but nothing consumed it to actually route the write. This closes that for ClickHouse: enterprise/cmd/enterprise-ingest (a second binary, mirroring enterprise-api) reuses ingest/consumer's own flush loop unchanged, with enterprise/internal/chwriter.Registry -- a per-tenant clickhousewriter.Writer registry -- swapped in as the writer. A batch pulled from the single shared Redpanda topic can mix records from many tenants, so WriteBatch groups by TenantID and dispatches each group to its own tenant's connection, fail- closed on an empty or unrecognized tenant_id. ingest/consumer and ingest/clickhousewriter move out of internal/ (same reason api/internal/* moved earlier this phase: enterprise/ can't import anything under another module's internal/). Their New() constructors now take small local Config structs instead of ingest/internal/config types, so enterprise/ doesn't need that import either. Building this surfaced a real bug: tenantprovision.ProvisionClickHouse only granted SELECT on a tenant's ClickHouse user, correct for chrunner's read-only use but not enough for chwriter reusing the same credential to write -- every real per-tenant write would have failed closed with a permission error. Fixed by widening the grant to SELECT, INSERT; no cross-tenant boundary is crossed by also allowing INSERT within a tenant's own database. Helm gates enterprise-ingest's Deployment on the same ingest.requireTenantCredential flag that already gates tag validation -- write-routing is meaningless without tagging already being required, so they're one decision, not two. docker-compose.yml's version is a disclosed, weaker approximation: it can't achieve Helm's genuine -mode=server/-mode=consumer split, so with the enterprise profile active both ingest and enterprise-ingest independently consume every message via different consumer groups -- harmless duplication for local verification only. Not built: Tantivy's independent Redpanda consumer (search/src/consumer.rs) still doesn't read the tenant_id header at all -- every record still lands in the one shared index regardless of tenant. Not run: the live-ClickHouse- gated tests (chwriter's cross-tenant routing test, tenantprovision's INSERT regression test) -- no Docker/database access in this environment; they're correct Go that has never executed, disclosed as such in docs/security/ threat-model.md and docs/phase-4-runbook.md §14.
This commit is contained in:
+22
-11
@@ -139,23 +139,34 @@ escape hatch is opaque to any compiler-injected filter.
|
||||
ran in the environment it was built in — Tantivy is an embedded
|
||||
library, so the cross-tenant isolation probe needed no live database
|
||||
or Docker to execute for real, and it passed.
|
||||
- **Ingest identity is now built, though write-routing isn't.** `ingest`
|
||||
(AGPL core) gained an optional `TenantResolver`
|
||||
- **Ingest identity and ClickHouse write-routing are now built; Tantivy's
|
||||
isn't.** `ingest` (AGPL core) gained an optional `TenantResolver`
|
||||
(`ingest/internal/grpcserver`): an agent presents a per-tenant bearer
|
||||
credential (`enterprise-auth -create-ingest-credential-tenant=<id>`
|
||||
mints one, only its hash stored), validated over the network via a new
|
||||
`POST /internal/authorize-ingest` endpoint (never an `enterprise/`
|
||||
import — same "network boundary, not import boundary" shape
|
||||
`api/authz.Authorizer` already uses), and the resolved tenant ID rides
|
||||
as a `tenant_id` Kafka message header on every record produced. What
|
||||
isn't built yet: neither `ingest`'s own ClickHouse writer nor
|
||||
`search`'s independent Redpanda consumer reads that header back to
|
||||
route the write anywhere per-tenant — every record still lands in the
|
||||
one shared ClickHouse database and Tantivy index regardless of tenant,
|
||||
correctly tagged but not yet isolated at write time. That per-tenant
|
||||
write-routing split is real, scoped remaining work (likely another
|
||||
"second binary," mirroring `enterprise-api`), not something this
|
||||
change claims to have closed.
|
||||
as a `tenant_id` Kafka message header on every record produced.
|
||||
`enterprise/cmd/enterprise-ingest` (another "second binary," mirroring
|
||||
`enterprise-api`) consumes that tag: it reuses `ingest/consumer`'s own
|
||||
flush loop with `enterprise/internal/chwriter.Registry` swapped in as
|
||||
the writer, routing each batch's records to a per-tenant ClickHouse
|
||||
connection (one per tenant, built from `rbacstore.
|
||||
ListProvisionedDataSources` — the same source `chrunner` already uses
|
||||
for reads), fail-closed on an untagged or unprovisioned tenant.
|
||||
Building it found and fixed a real bug: `tenantprovision.
|
||||
ProvisionClickHouse` originally granted a tenant's ClickHouse user
|
||||
`SELECT` only, which would have made every real per-tenant write fail
|
||||
with a permission error — fixed by granting `SELECT, INSERT` (one
|
||||
credential, both directions; no cross-tenant boundary is crossed by
|
||||
also allowing INSERT within a tenant's own database). What isn't built
|
||||
yet: `search`'s independent Redpanda consumer (a different codebase —
|
||||
Rust — and a different process, not reachable through either `ingest`
|
||||
or `enterprise-ingest`) still writes every record into the one shared
|
||||
Tantivy index regardless of tenant. That's real, disclosed, separately
|
||||
scoped remaining work, not something this change claims to have
|
||||
closed.
|
||||
- `deploy/operator`'s `Tenant` CRD and `enterprise-api -provision-tenant`
|
||||
are now unified, deliberately lightweight: `-provision-tenant` stays
|
||||
the sole real actor (ClickHouse + `rbacstore`), and now also syncs its
|
||||
|
||||
+88
-30
@@ -554,7 +554,7 @@ all -- a cross-origin `fetch` with credentials from `web`'s origin to
|
||||
actual picker UI is real, separately-scoped frontend work; this section
|
||||
only closes the backend half.
|
||||
|
||||
## 13. Ingest tenant identity (no per-tenant write-routing yet)
|
||||
## 13. Ingest tenant identity
|
||||
|
||||
The identity mechanism was chosen deliberately (config-supplied
|
||||
tenant_id + a shared-secret token ingest validates, not per-tenant
|
||||
@@ -592,24 +592,77 @@ go test ./internal/grpcserver/... -run 'Resolver|TenantHeader' -v
|
||||
# tenant."
|
||||
```
|
||||
|
||||
**Not built, and explicitly scoped out for now**: per-tenant write
|
||||
routing. Neither `ingest/internal/consumer` (the ClickHouse writer) nor
|
||||
`search/src/consumer.rs` (a completely independent Redpanda consumer,
|
||||
not called through `ingest` at all -- see that file) reads the
|
||||
`tenant_id` Kafka header back to route a record's write into a
|
||||
per-tenant ClickHouse database or Tantivy index. Every record still
|
||||
lands in the one shared destination regardless of tenant, correctly
|
||||
tagged but not yet isolated at write time -- see CLAUDE.md and
|
||||
`/docs/security/threat-model.md`'s "Read this first" for the full
|
||||
disclosure. Also not built: any Helm/`docker-compose.yml` wiring that
|
||||
issues an agent a real ingest credential automatically (`enterprise-
|
||||
auth -create-ingest-credential-tenant=<id>` is, like every other
|
||||
credential-minting flag in this codebase, a manual operator action) --
|
||||
`deploy/helm/sentry/values.yaml`'s `ingest.requireTenantCredential`
|
||||
(default `false`) only turns on *validation*, deliberately not folded
|
||||
into `enterprise.enabled` directly, since flipping that flag with no
|
||||
agents holding a credential yet would refuse all ingest traffic outright
|
||||
rather than degrading gracefully.
|
||||
Also not built as part of the identity mechanism itself: any Helm/
|
||||
`docker-compose.yml` wiring that issues an agent a real ingest credential
|
||||
automatically (`enterprise-auth -create-ingest-credential-tenant=<id>`
|
||||
is, like every other credential-minting flag in this codebase, a manual
|
||||
operator action) -- `deploy/helm/sentry/values.yaml`'s
|
||||
`ingest.requireTenantCredential` (default `false`) only turns on
|
||||
*validation*, deliberately not folded into `enterprise.enabled`
|
||||
directly, since flipping that flag with no agents holding a credential
|
||||
yet would refuse all ingest traffic outright rather than degrading
|
||||
gracefully. See §14 below for what the identity is actually used for on
|
||||
the write path.
|
||||
|
||||
## 14. Ingest write-routing (ClickHouse built, Tantivy not yet)
|
||||
|
||||
`enterprise/cmd/enterprise-ingest` (mirrors `enterprise-api`'s "second
|
||||
binary" shape) consumes the `tenant_id` Kafka header §13 attaches and
|
||||
routes each record's ClickHouse write into that tenant's own database,
|
||||
via `enterprise/internal/chwriter.Registry` -- a per-tenant
|
||||
`*ingest/clickhousewriter.Writer` registry that reuses `ingest/
|
||||
consumer`'s own flush loop unchanged. Verified in this environment
|
||||
without Docker, using the same fake-transport discipline as §13:
|
||||
|
||||
```sh
|
||||
cd enterprise
|
||||
go test ./internal/chwriter/... -run TestRegistry -v
|
||||
# Docker-free: constructs a Registry directly (bypassing New(), the only
|
||||
# part that dials ClickHouse) to prove the fail-closed paths -- an empty
|
||||
# TenantID, or a TenantID with no registered writer, refuses the WHOLE
|
||||
# batch rather than silently dropping just those records or falling back
|
||||
# to a default destination.
|
||||
|
||||
go test ./internal/tenantprovision/... -run TestProvisionedUserCanInsertIntoOwnDatabase -v
|
||||
# skip-gated (needs a live ClickHouse) -- regression test for the bug
|
||||
# found while building this: the per-tenant credential chwriter reuses
|
||||
# from chrunner only had SELECT granted, which would make every real
|
||||
# write fail with a permission error. Fixed by granting SELECT, INSERT
|
||||
# (tenantprovision.go). This test proves the fix, not just documents it,
|
||||
# whenever a real ClickHouse is available to run it against.
|
||||
|
||||
go test ./internal/chwriter/... -run TestRegistryRoutesToCorrectTenant -v
|
||||
# skip-gated (CHWRITER_TEST_CLICKHOUSE_ADDR) -- writes a mixed batch
|
||||
# spanning two tenants in one WriteBatch call and confirms each row
|
||||
# lands in its own tenant's database, none in the other's.
|
||||
```
|
||||
|
||||
**Not run in this environment**: no live ClickHouse was available while
|
||||
this was built, so the skip-gated tests above are correct Go that has
|
||||
never actually executed -- "the test exists" is not the same claim as
|
||||
"write-routing is confirmed," same caveat §8 already states for the
|
||||
read-side `chrunner` tests.
|
||||
|
||||
**Not built**: Tantivy's side of this. `search/src/consumer.rs` is a
|
||||
completely independent Redpanda consumer, not called through `ingest` or
|
||||
`enterprise-ingest` at all, and does not read the `tenant_id` header --
|
||||
every record still lands in the one shared (default) Tantivy index
|
||||
regardless of tenant. See CLAUDE.md and `/docs/security/threat-model.md`'s
|
||||
"Read this first" for the full disclosure.
|
||||
|
||||
**Also not built**: Helm/`docker-compose.yml` do gate *whether*
|
||||
`enterprise-ingest` runs at all (`ingest.requireTenantCredential`, same
|
||||
flag §13 uses for validation -- see `deploy/helm/sentry/templates/
|
||||
enterprise-ingest.yaml`), but `docker-compose.yml`'s version is a
|
||||
disclosed, weaker approximation of Helm's: Helm achieves genuine
|
||||
`-mode=server`/`-mode=consumer` mutual exclusivity between `ingest` and
|
||||
`enterprise-ingest`; compose's `enterprise-ingest` service is a
|
||||
profile-gated opt-in extra that does NOT split `ingest`'s own
|
||||
server/consumer halves, so with the `enterprise` profile active, both
|
||||
`ingest -mode=all` and `enterprise-ingest` independently consume every
|
||||
message via different consumer groups -- harmless duplication for local
|
||||
verification, not a topology compose actually enforces the way Helm
|
||||
does.
|
||||
|
||||
## Known gaps (do not treat this phase as done without reading these)
|
||||
|
||||
@@ -635,17 +688,22 @@ Full accounting: `/docs/security/threat-model.md`. Headline items:
|
||||
that split (declarative request vs. imperative provisioning action)
|
||||
is intentional, not the "two disconnected sources of truth" gap this
|
||||
bullet used to describe.
|
||||
- **Ingest now has a real tenant identity (§13), but no per-tenant
|
||||
write-routing yet.** An agent presents a bearer credential
|
||||
(`enterprise-auth -create-ingest-credential-tenant=<id>`),
|
||||
`ingest/internal/grpcserver.TenantResolver` validates it (fail-closed)
|
||||
and attaches the resolved tenant ID to every record as a `tenant_id`
|
||||
Kafka message header. Nothing downstream reads that header back yet --
|
||||
every record still lands in the one shared ClickHouse database and the
|
||||
one shared Tantivy index no matter what. A newly-provisioned tenant's
|
||||
storage is real, isolated at query time, and permanently empty until
|
||||
the write-routing split is built (a real, scoped follow-up, no longer
|
||||
an undesigned one).
|
||||
- **Ingest now has a real tenant identity (§13), and ClickHouse
|
||||
write-routing is built (§14) -- Tantivy write-routing is the one gap
|
||||
left.** An agent presents a bearer credential (`enterprise-auth
|
||||
-create-ingest-credential-tenant=<id>`), `ingest/internal/grpcserver.
|
||||
TenantResolver` validates it (fail-closed) and attaches the resolved
|
||||
tenant ID to every record as a `tenant_id` Kafka message header.
|
||||
`enterprise-ingest` reads that header back and routes each record's
|
||||
ClickHouse write into its own tenant's database (not yet confirmed
|
||||
against a real ClickHouse in this environment -- see §14). Nothing
|
||||
reads that header on the Tantivy side yet -- every record still lands
|
||||
in the one shared Tantivy index no matter what. A newly-provisioned
|
||||
tenant's ClickHouse database is real, isolated, and actually
|
||||
populated by write-routed traffic (once confirmed live); its Tantivy
|
||||
index remains real, isolated at query time, and permanently empty
|
||||
until Tantivy's write-routing split is built (a real, scoped
|
||||
follow-up, no longer an undesigned one).
|
||||
- **Human SSO login now works for both OIDC (§3a) and SAML (§3b)** --
|
||||
each verified with a real fake IdP (genuine cryptographic signing and
|
||||
verification), not yet a real external IdP or a running
|
||||
|
||||
@@ -9,13 +9,15 @@ for the full design rationale behind the controls described here.
|
||||
|
||||
## Read this first: the single most important open finding
|
||||
|
||||
**Updated a second time.** This section originally read "log data
|
||||
**Updated a third time.** This section originally read "log data
|
||||
queried through `POST /query` is not tenant-isolated at all," then
|
||||
"ClickHouse is isolated but Tantivy isn't." Both ClickHouse *and*
|
||||
Tantivy connection/index-layer isolation are now built. What's left is
|
||||
"ClickHouse is isolated but Tantivy isn't," then "ingest tags records
|
||||
with a tenant identity but nothing routes the write." Both ClickHouse
|
||||
*and* Tantivy connection/index-layer isolation are built on the *read*
|
||||
path, and ClickHouse write-routing is now built too. What's left is
|
||||
narrower but still real: **whether a given deployment actually runs the
|
||||
isolated binary**, and **whether ingest itself is tenant-aware** (it
|
||||
isn't, for either storage engine).
|
||||
isolated binaries**, and **Tantivy's write path**, which still has no
|
||||
per-tenant routing at all.
|
||||
|
||||
**ClickHouse (the SQL path) is built.** `enterprise/internal/
|
||||
tenantprovision` (real `CREATE DATABASE`/`CREATE USER`/`GRANT`) and
|
||||
@@ -68,30 +70,43 @@ provisioned, pointing at the same ClickHouse/Postgres. The Helm chart
|
||||
makes the *default*, chart-managed path correct; it isn't a runtime
|
||||
guard against misconfiguration.
|
||||
|
||||
**Ingest now has a real tenant identity, but no per-tenant write
|
||||
routing yet** — a narrower, more precise gap than "not tenant-aware at
|
||||
all." `chrunner`/`searchclient` prove *read* isolation given tenant-
|
||||
scoped data exists; a new optional `ingest/internal/grpcserver.
|
||||
TenantResolver` closes the "does a record know which tenant it belongs
|
||||
to" half by validating a per-tenant bearer credential an agent presents
|
||||
**Ingest now has a real tenant identity, and ClickHouse write-routing is
|
||||
built — Tantivy write-routing is the one gap left.** `chrunner`/
|
||||
`searchclient` prove *read* isolation given tenant-scoped data exists; an
|
||||
optional `ingest/internal/grpcserver.TenantResolver` closes the "does a
|
||||
record know which tenant it belongs to" half by validating a per-tenant
|
||||
bearer credential an agent presents
|
||||
(`enterprise-auth -create-ingest-credential-tenant=<id>` mints one; only
|
||||
its SHA-256 hash is ever stored) against a new `POST
|
||||
its SHA-256 hash is ever stored) against a `POST
|
||||
/internal/authorize-ingest` endpoint, and attaching the resolved tenant
|
||||
ID to every record as a `tenant_id` Kafka message header before
|
||||
producing it — fail-closed: once a resolver is configured, a missing or
|
||||
invalid credential refuses the whole batch, never falls back to "no
|
||||
tenant." What's still missing is the "does that identity actually
|
||||
change where the record is written" half: neither `ingest`'s own
|
||||
ClickHouse writer nor `search`'s independent Redpanda consumer reads
|
||||
that header back to route the write anywhere per-tenant yet. Every
|
||||
record still lands in the one shared ClickHouse database and the one
|
||||
shared (default) Tantivy index, regardless of tenant — correctly tagged,
|
||||
not yet isolated at write time. A newly-provisioned tenant's ClickHouse
|
||||
database and Tantivy index remain real, isolated, and queryable through
|
||||
`enterprise-api` — and permanently empty, until that write-routing split
|
||||
is built (likely another "second binary," mirroring `enterprise-api`
|
||||
itself), which is now scoped, disclosed remaining work, not an
|
||||
undesigned gap.
|
||||
tenant." `enterprise/cmd/enterprise-ingest` (a second binary, mirroring
|
||||
`enterprise-api`) now consumes that header: it reuses `ingest/consumer`'s
|
||||
own flush loop with `enterprise/internal/chwriter.Registry` — a
|
||||
per-tenant `*clickhousewriter.Writer` registry, built the same way
|
||||
`chrunner`'s per-tenant connections are — swapped in as the writer, so a
|
||||
tagged batch's records are grouped by tenant and each group INSERTed
|
||||
through that tenant's own ClickHouse connection, fail-closed on an
|
||||
untagged or unprovisioned tenant. Building it surfaced a real gap in an
|
||||
already-shipped control: `tenantprovision.ProvisionClickHouse`'s grant
|
||||
was `SELECT`-only (correct for the read-side credential `chrunner` uses,
|
||||
but `chwriter` reuses the same credential for writes) — every real
|
||||
per-tenant write would have failed with a permission error until this
|
||||
was widened to `SELECT, INSERT`. **Not yet confirmed against a real
|
||||
ClickHouse**, same caveat as the read-side chrunner claim above — the
|
||||
Docker-free fail-closed tests pass, the live-database tests are written
|
||||
but skip-gated, see `/docs/phase-4-runbook.md`. What's still missing:
|
||||
`search`'s independent Redpanda consumer does not read the `tenant_id`
|
||||
header at all — every ingested record still lands in the one shared
|
||||
(default) Tantivy index regardless of tenant. A newly-provisioned
|
||||
tenant's ClickHouse database is now real, isolated, and actually
|
||||
populated by write-routed agent traffic (once confirmed against a real
|
||||
cluster); its Tantivy index remains real, isolated, and queryable
|
||||
through `enterprise-api` — and permanently empty, until Tantivy's own
|
||||
write-routing split is built, which is now scoped, disclosed remaining
|
||||
work, not an undesigned gap.
|
||||
|
||||
## System overview
|
||||
|
||||
@@ -129,14 +144,15 @@ sentryctl ──▶ api, alerting (Bearer token when SENTRYCTL_TOKEN is set)
|
||||
```
|
||||
|
||||
Ingest path (agent → Redpanda → ingest → ClickHouse, and Redpanda →
|
||||
search → Tantivy): `ingest` now resolves and tags each record with a
|
||||
real tenant ID (see "Read this first" above), but nothing downstream
|
||||
routes on it yet — every ingested log record still lands in the one
|
||||
shared `logs` table/index. Tenant isolation for the *write* path is
|
||||
still out of scope for what's built so far and is not separately
|
||||
designed in
|
||||
`/docs/phase-4-isolation-design.md`; named here as a gap that design doc
|
||||
doesn't yet cover, not just an implementation gap.
|
||||
search → Tantivy): `ingest` resolves and tags each record with a real
|
||||
tenant ID (see "Read this first" above). `enterprise/cmd/enterprise-ingest`
|
||||
now consumes that tag and routes ClickHouse writes to each tenant's own
|
||||
database. Tantivy's independent Redpanda consumer (`search/src/
|
||||
consumer.rs`) still does not consume the tag at all — every record still
|
||||
lands in the one shared Tantivy index regardless of tenant. That
|
||||
narrower gap is still out of scope for what's built so far and is not
|
||||
separately designed in `/docs/phase-4-isolation-design.md`; named here as
|
||||
a gap that design doc doesn't yet cover, not just an implementation gap.
|
||||
|
||||
## Module boundary (trust boundary #1)
|
||||
|
||||
@@ -438,7 +454,8 @@ terms:
|
||||
| Tantivy per-tenant index routing (`search/src/registry.rs`) | **Enforced, verified live** — real Tantivy indices, real cross-tenant probe, all passing |
|
||||
| Tantivy tenant_id resolution (`enterprise/internal/searchclient`) | **Enforced, verified live** — real gRPC wire-level test |
|
||||
| Ingest tenant *identity* (credential validation, tagging) | **Built and tested** — fail-closed `TenantResolver`, `tenant_id` Kafka header attached per record |
|
||||
| Ingest tenant *write-routing* (ClickHouse and Tantivy both) | **Not implemented, now scoped** — every record still lands in the single shared database/index regardless of tenant; consuming the tenant_id header to route the write is real, disclosed remaining work |
|
||||
| Ingest tenant *write-routing*, ClickHouse | **Built, not yet confirmed against a real ClickHouse** — `enterprise-ingest`/`chwriter.Registry` route each tagged batch to its tenant's own database, fail-closed on an untagged/unprovisioned tenant; Docker-free tests pass, live-database tests are skip-gated |
|
||||
| Ingest tenant *write-routing*, Tantivy | **Not implemented, now scoped** — every record still lands in the single shared Tantivy index regardless of tenant; `search/src/consumer.rs` consuming the tenant_id header to route the write is real, disclosed remaining work |
|
||||
| Deployment actually routing traffic to `enterprise-api` (Helm) | **Enforced** — `api`/`enterprise-api` are mutually exclusive, same flag as RBAC/audit/SSO |
|
||||
| Deployment actually routing traffic to `enterprise-api` (docker-compose) | **Enforced** — `api`/`enterprise-api` are mutually exclusive via `COMPOSE_PROFILES`, same flag choice as Helm's `enterprise.enabled`; verified via `docker compose config`, not an actual `docker compose up` in this environment |
|
||||
| Human SSO login — OIDC | **Built, verified with a real fake IdP** (not yet tried against a real external IdP) |
|
||||
|
||||
Reference in New Issue
Block a user