Give ingest a real tenant identity (write-routing deferred, disclosed)
Ingest tenant-awareness was named "undesigned, not just unbuilt" across CLAUDE.md/threat-model.md/the runbook since early Phase 4 -- the last major standing gap. Scoping was agreed via AskUserQuestion: a config-supplied tenant_id + shared-secret token ingest validates (smaller real implementation, no new PKI), over per-tenant mTLS certs. This change builds that identity mechanism end to end and attaches it to every record at the point it enters the system; it deliberately does NOT build per-tenant write-routing for ClickHouse or Tantivy -- that's real, separately-scoped follow-up work, disclosed explicitly everywhere this was previously called undesigned, not silently left half-done. New pieces: - metadata/migrations/0034 + enterprise/internal/rbacstore/ ingest_credentials.go: a per-tenant bearer credential, only its SHA-256 hash ever persisted (same reasoning a password gets hashed, not stored raw) -- CreateIngestCredential returns the plaintext exactly once, ValidateIngestCredential/RevokeIngestCredential/ ListIngestCredentialsForTenant round it out. - enterprise-auth gains -create-ingest-credential-tenant/ -list-ingest-credentials-tenant/-revoke-ingest-credential (same offline-operator-flag shape as every other credential-minting flag in this binary) and a new POST /internal/authorize-ingest endpoint (internal/authhandler) validating a presented token and resolving its tenant -- a genuinely different credential type from session-backed /internal/authorize, so it doesn't touch session.Manager at all. - ingest (AGPL core) gains an optional TenantResolver (internal/grpcserver, nil by default) and its HTTP client implementation (internal/tenantresolver.HTTPResolver) -- a plain HTTP call to enterprise-auth's new endpoint, never an enterprise/ import, same "network boundary, not import boundary" shape api/authz.HTTPAuthorizer already uses for the query path. PushBatch now requires an `authorization: Bearer <token>` gRPC metadata entry once a resolver is configured, fails the whole batch closed on a missing/invalid credential (never falls back to "no tenant"), and attaches the resolved tenant ID to every record as a `tenant_id` Kafka message header before producing it. Verified with real round trips at every layer, no Docker needed: rbacstore's credential CRUD (skip-gated on live Postgres, same as every other rbacstore integration test this phase), authhandler's new endpoint (real HTTP via httptest, including the regression test that a session token must not validate as an ingest credential), tenantresolver (real HTTP client against httptest, same pattern as authz.HTTPAuthorizer's own tests), and grpcserver's PushBatch (fake resolver/producer -- no resolver leaves messages unchanged, a configured resolver attaches the right header or fails closed on a bad/missing token). Helm: ingest.requireTenantCredential (default false) is a deliberate, separate opt-in from enterprise.enabled -- turning ENTERPRISE_AUTH_URL on for ingest requires every agent to already hold a credential or be refused outright, so it must not default on just because enterprise.enabled does (same reasoning api.yaml's ENTERPRISE_AUTH_URL isn't tied to enterprise.enabled directly either). docker-compose.yml leaves it unset, same as ever. Docs updated everywhere this was called "undesigned": CLAUDE.md, docs/architecture.md, docs/security/threat-model.md (including its summary table, now split into "identity: built" vs "write-routing: not yet"), docs/phase-4-runbook.md (new §13), enterprise/README.md.
This commit is contained in:
+20
-9
@@ -139,13 +139,23 @@ escape hatch is opaque to any compiler-injected filter.
|
||||
ran in the environment it was built in — Tantivy is an embedded
|
||||
library, so the cross-tenant isolation probe needed no live database
|
||||
or Docker to execute for real, and it passed.
|
||||
- Neither storage engine's isolation extends to *ingest*: every record
|
||||
`ingest` produces lands in the one shared ClickHouse database and the
|
||||
one shared (default) Tantivy index regardless of tenant. A
|
||||
newly-provisioned tenant's database/index are real and isolated at
|
||||
query time — and permanently empty until something upstream of
|
||||
`chrunner`/`searchclient` becomes tenant-aware on the write side,
|
||||
which is undesigned, not merely unbuilt.
|
||||
- **Ingest identity is now built, though write-routing isn't.** `ingest`
|
||||
(AGPL core) gained an optional `TenantResolver`
|
||||
(`ingest/internal/grpcserver`): an agent presents a per-tenant bearer
|
||||
credential (`enterprise-auth -create-ingest-credential-tenant=<id>`
|
||||
mints one, only its hash stored), validated over the network via a new
|
||||
`POST /internal/authorize-ingest` endpoint (never an `enterprise/`
|
||||
import — same "network boundary, not import boundary" shape
|
||||
`api/authz.Authorizer` already uses), and the resolved tenant ID rides
|
||||
as a `tenant_id` Kafka message header on every record produced. What
|
||||
isn't built yet: neither `ingest`'s own ClickHouse writer nor
|
||||
`search`'s independent Redpanda consumer reads that header back to
|
||||
route the write anywhere per-tenant — every record still lands in the
|
||||
one shared ClickHouse database and Tantivy index regardless of tenant,
|
||||
correctly tagged but not yet isolated at write time. That per-tenant
|
||||
write-routing split is real, scoped remaining work (likely another
|
||||
"second binary," mirroring `enterprise-api`), not something this
|
||||
change claims to have closed.
|
||||
- `deploy/operator`'s `Tenant` CRD and `enterprise-api -provision-tenant`
|
||||
are now unified, deliberately lightweight: `-provision-tenant` stays
|
||||
the sole real actor (ClickHouse + `rbacstore`), and now also syncs its
|
||||
@@ -167,8 +177,9 @@ plain `api`), sharing a host-port/network-alias trick so `alerting`/
|
||||
`web` need no conditional config either way. With both storage engines'
|
||||
connection/index-layer mechanisms built, deployment topology enforced at
|
||||
both the Helm and docker-compose layers, and the two provisioning
|
||||
mechanisms unified, the largest remaining gap is ingest's lack of
|
||||
tenant-awareness, which is undesigned, not merely unbuilt.
|
||||
mechanisms unified, the largest remaining gap is ingest's per-tenant
|
||||
*write-routing* (identity is now attached at ingest time; nothing
|
||||
downstream of Redpanda consumes it yet to isolate the write, see above).
|
||||
|
||||
## Licensing boundary
|
||||
|
||||
|
||||
Reference in New Issue
Block a user