Give ingest a real tenant identity (write-routing deferred, disclosed)
Ingest tenant-awareness was named "undesigned, not just unbuilt" across CLAUDE.md/threat-model.md/the runbook since early Phase 4 -- the last major standing gap. Scoping was agreed via AskUserQuestion: a config-supplied tenant_id + shared-secret token ingest validates (smaller real implementation, no new PKI), over per-tenant mTLS certs. This change builds that identity mechanism end to end and attaches it to every record at the point it enters the system; it deliberately does NOT build per-tenant write-routing for ClickHouse or Tantivy -- that's real, separately-scoped follow-up work, disclosed explicitly everywhere this was previously called undesigned, not silently left half-done. New pieces: - metadata/migrations/0034 + enterprise/internal/rbacstore/ ingest_credentials.go: a per-tenant bearer credential, only its SHA-256 hash ever persisted (same reasoning a password gets hashed, not stored raw) -- CreateIngestCredential returns the plaintext exactly once, ValidateIngestCredential/RevokeIngestCredential/ ListIngestCredentialsForTenant round it out. - enterprise-auth gains -create-ingest-credential-tenant/ -list-ingest-credentials-tenant/-revoke-ingest-credential (same offline-operator-flag shape as every other credential-minting flag in this binary) and a new POST /internal/authorize-ingest endpoint (internal/authhandler) validating a presented token and resolving its tenant -- a genuinely different credential type from session-backed /internal/authorize, so it doesn't touch session.Manager at all. - ingest (AGPL core) gains an optional TenantResolver (internal/grpcserver, nil by default) and its HTTP client implementation (internal/tenantresolver.HTTPResolver) -- a plain HTTP call to enterprise-auth's new endpoint, never an enterprise/ import, same "network boundary, not import boundary" shape api/authz.HTTPAuthorizer already uses for the query path. PushBatch now requires an `authorization: Bearer <token>` gRPC metadata entry once a resolver is configured, fails the whole batch closed on a missing/invalid credential (never falls back to "no tenant"), and attaches the resolved tenant ID to every record as a `tenant_id` Kafka message header before producing it. Verified with real round trips at every layer, no Docker needed: rbacstore's credential CRUD (skip-gated on live Postgres, same as every other rbacstore integration test this phase), authhandler's new endpoint (real HTTP via httptest, including the regression test that a session token must not validate as an ingest credential), tenantresolver (real HTTP client against httptest, same pattern as authz.HTTPAuthorizer's own tests), and grpcserver's PushBatch (fake resolver/producer -- no resolver leaves messages unchanged, a configured resolver attaches the right header or fails closed on a bad/missing token). Helm: ingest.requireTenantCredential (default false) is a deliberate, separate opt-in from enterprise.enabled -- turning ENTERPRISE_AUTH_URL on for ingest requires every agent to already hold a credential or be refused outright, so it must not default on just because enterprise.enabled does (same reasoning api.yaml's ENTERPRISE_AUTH_URL isn't tied to enterprise.enabled directly either). docker-compose.yml leaves it unset, same as ever. Docs updated everywhere this was called "undesigned": CLAUDE.md, docs/architecture.md, docs/security/threat-model.md (including its summary table, now split into "identity: built" vs "write-routing: not yet"), docs/phase-4-runbook.md (new §13), enterprise/README.md.
This commit is contained in:
+20
-9
@@ -139,13 +139,23 @@ escape hatch is opaque to any compiler-injected filter.
|
||||
ran in the environment it was built in — Tantivy is an embedded
|
||||
library, so the cross-tenant isolation probe needed no live database
|
||||
or Docker to execute for real, and it passed.
|
||||
- Neither storage engine's isolation extends to *ingest*: every record
|
||||
`ingest` produces lands in the one shared ClickHouse database and the
|
||||
one shared (default) Tantivy index regardless of tenant. A
|
||||
newly-provisioned tenant's database/index are real and isolated at
|
||||
query time — and permanently empty until something upstream of
|
||||
`chrunner`/`searchclient` becomes tenant-aware on the write side,
|
||||
which is undesigned, not merely unbuilt.
|
||||
- **Ingest identity is now built, though write-routing isn't.** `ingest`
|
||||
(AGPL core) gained an optional `TenantResolver`
|
||||
(`ingest/internal/grpcserver`): an agent presents a per-tenant bearer
|
||||
credential (`enterprise-auth -create-ingest-credential-tenant=<id>`
|
||||
mints one, only its hash stored), validated over the network via a new
|
||||
`POST /internal/authorize-ingest` endpoint (never an `enterprise/`
|
||||
import — same "network boundary, not import boundary" shape
|
||||
`api/authz.Authorizer` already uses), and the resolved tenant ID rides
|
||||
as a `tenant_id` Kafka message header on every record produced. What
|
||||
isn't built yet: neither `ingest`'s own ClickHouse writer nor
|
||||
`search`'s independent Redpanda consumer reads that header back to
|
||||
route the write anywhere per-tenant — every record still lands in the
|
||||
one shared ClickHouse database and Tantivy index regardless of tenant,
|
||||
correctly tagged but not yet isolated at write time. That per-tenant
|
||||
write-routing split is real, scoped remaining work (likely another
|
||||
"second binary," mirroring `enterprise-api`), not something this
|
||||
change claims to have closed.
|
||||
- `deploy/operator`'s `Tenant` CRD and `enterprise-api -provision-tenant`
|
||||
are now unified, deliberately lightweight: `-provision-tenant` stays
|
||||
the sole real actor (ClickHouse + `rbacstore`), and now also syncs its
|
||||
@@ -167,8 +177,9 @@ plain `api`), sharing a host-port/network-alias trick so `alerting`/
|
||||
`web` need no conditional config either way. With both storage engines'
|
||||
connection/index-layer mechanisms built, deployment topology enforced at
|
||||
both the Helm and docker-compose layers, and the two provisioning
|
||||
mechanisms unified, the largest remaining gap is ingest's lack of
|
||||
tenant-awareness, which is undesigned, not merely unbuilt.
|
||||
mechanisms unified, the largest remaining gap is ingest's per-tenant
|
||||
*write-routing* (identity is now attached at ingest time; nothing
|
||||
downstream of Redpanda consumes it yet to isolate the write, see above).
|
||||
|
||||
## Licensing boundary
|
||||
|
||||
|
||||
+68
-5
@@ -554,6 +554,63 @@ all -- a cross-origin `fetch` with credentials from `web`'s origin to
|
||||
actual picker UI is real, separately-scoped frontend work; this section
|
||||
only closes the backend half.
|
||||
|
||||
## 13. Ingest tenant identity (no per-tenant write-routing yet)
|
||||
|
||||
The identity mechanism was chosen deliberately (config-supplied
|
||||
tenant_id + a shared-secret token ingest validates, not per-tenant
|
||||
mTLS certs -- smaller real implementation, no new PKI). Verified in
|
||||
this environment without Docker or a live enterprise-auth, using the
|
||||
same fake-client-at-every-layer discipline as everything else in this
|
||||
runbook that doesn't need a live stack:
|
||||
|
||||
```sh
|
||||
cd enterprise
|
||||
go test ./internal/rbacstore/... -run IngestCredential -v
|
||||
# skip-gated (RBACSTORE_TEST_POSTGRES_ADDR) -- CreateIngestCredential/
|
||||
# ValidateIngestCredential/RevokeIngestCredential round trip, and the
|
||||
# regression test that only a SHA-256 hash is ever persisted, never the
|
||||
# plaintext token.
|
||||
|
||||
go test ./internal/authhandler/... -run AuthorizeIngest -v
|
||||
# real HTTP round trip against POST /internal/authorize-ingest with a
|
||||
# fake credential validator -- proves a session token (service or
|
||||
# human) does NOT work as an ingest credential, since this endpoint
|
||||
# never calls session.Manager.Validate at all.
|
||||
|
||||
cd ../ingest
|
||||
go test ./internal/tenantresolver/... -v
|
||||
# real HTTP round trip (httptest), same shape as api/authz.
|
||||
# HTTPAuthorizer's own tests -- forwards the bearer token, parses
|
||||
# tenant_id, treats a non-2xx or an empty tenant_id as an error.
|
||||
|
||||
go test ./internal/grpcserver/... -run 'Resolver|TenantHeader' -v
|
||||
# PushBatch with a fake TenantResolver: no resolver configured ->
|
||||
# unchanged behavior, no tenant_id header at all; resolver configured ->
|
||||
# every produced Kafka message carries a tenant_id header matching the
|
||||
# resolved tenant; missing or invalid bearer token -> the whole batch is
|
||||
# refused (codes.Unauthenticated), fail-closed, never falls back to "no
|
||||
# tenant."
|
||||
```
|
||||
|
||||
**Not built, and explicitly scoped out for now**: per-tenant write
|
||||
routing. Neither `ingest/internal/consumer` (the ClickHouse writer) nor
|
||||
`search/src/consumer.rs` (a completely independent Redpanda consumer,
|
||||
not called through `ingest` at all -- see that file) reads the
|
||||
`tenant_id` Kafka header back to route a record's write into a
|
||||
per-tenant ClickHouse database or Tantivy index. Every record still
|
||||
lands in the one shared destination regardless of tenant, correctly
|
||||
tagged but not yet isolated at write time -- see CLAUDE.md and
|
||||
`/docs/security/threat-model.md`'s "Read this first" for the full
|
||||
disclosure. Also not built: any Helm/`docker-compose.yml` wiring that
|
||||
issues an agent a real ingest credential automatically (`enterprise-
|
||||
auth -create-ingest-credential-tenant=<id>` is, like every other
|
||||
credential-minting flag in this codebase, a manual operator action) --
|
||||
`deploy/helm/sentry/values.yaml`'s `ingest.requireTenantCredential`
|
||||
(default `false`) only turns on *validation*, deliberately not folded
|
||||
into `enterprise.enabled` directly, since flipping that flag with no
|
||||
agents holding a credential yet would refuse all ingest traffic outright
|
||||
rather than degrading gracefully.
|
||||
|
||||
## Known gaps (do not treat this phase as done without reading these)
|
||||
|
||||
Full accounting: `/docs/security/threat-model.md`. Headline items:
|
||||
@@ -578,11 +635,17 @@ Full accounting: `/docs/security/threat-model.md`. Headline items:
|
||||
that split (declarative request vs. imperative provisioning action)
|
||||
is intentional, not the "two disconnected sources of truth" gap this
|
||||
bullet used to describe.
|
||||
- **Ingest has no tenant concept for either storage engine.** Every
|
||||
record `ingest` produces lands in the one shared ClickHouse database
|
||||
and the one shared Tantivy index no matter what. A newly-provisioned
|
||||
tenant's storage is real, isolated at query time, and permanently
|
||||
empty until this changes — undesigned, not just unbuilt.
|
||||
- **Ingest now has a real tenant identity (§13), but no per-tenant
|
||||
write-routing yet.** An agent presents a bearer credential
|
||||
(`enterprise-auth -create-ingest-credential-tenant=<id>`),
|
||||
`ingest/internal/grpcserver.TenantResolver` validates it (fail-closed)
|
||||
and attaches the resolved tenant ID to every record as a `tenant_id`
|
||||
Kafka message header. Nothing downstream reads that header back yet --
|
||||
every record still lands in the one shared ClickHouse database and the
|
||||
one shared Tantivy index no matter what. A newly-provisioned tenant's
|
||||
storage is real, isolated at query time, and permanently empty until
|
||||
the write-routing split is built (a real, scoped follow-up, no longer
|
||||
an undesigned one).
|
||||
- **Human SSO login now works for both OIDC (§3a) and SAML (§3b)** --
|
||||
each verified with a real fake IdP (genuine cryptographic signing and
|
||||
verification), not yet a real external IdP or a running
|
||||
|
||||
@@ -68,15 +68,30 @@ provisioned, pointing at the same ClickHouse/Postgres. The Helm chart
|
||||
makes the *default*, chart-managed path correct; it isn't a runtime
|
||||
guard against misconfiguration.
|
||||
|
||||
**Ingest is not tenant-aware for either storage engine**, and this is
|
||||
more load-bearing than it sounds: `chrunner`/`searchclient` prove *read*
|
||||
isolation given tenant-scoped data exists, but nothing writes
|
||||
tenant-scoped data yet. Every record `ingest` produces lands in the one
|
||||
shared ClickHouse database and the one shared (default) Tantivy index,
|
||||
regardless of tenant. A newly-provisioned tenant's ClickHouse database
|
||||
and Tantivy index are real, isolated, and queryable through
|
||||
`enterprise-api` — and permanently empty, until ingest itself becomes
|
||||
tenant-aware, which is undesigned, not just unbuilt.
|
||||
**Ingest now has a real tenant identity, but no per-tenant write
|
||||
routing yet** — a narrower, more precise gap than "not tenant-aware at
|
||||
all." `chrunner`/`searchclient` prove *read* isolation given tenant-
|
||||
scoped data exists; a new optional `ingest/internal/grpcserver.
|
||||
TenantResolver` closes the "does a record know which tenant it belongs
|
||||
to" half by validating a per-tenant bearer credential an agent presents
|
||||
(`enterprise-auth -create-ingest-credential-tenant=<id>` mints one; only
|
||||
its SHA-256 hash is ever stored) against a new `POST
|
||||
/internal/authorize-ingest` endpoint, and attaching the resolved tenant
|
||||
ID to every record as a `tenant_id` Kafka message header before
|
||||
producing it — fail-closed: once a resolver is configured, a missing or
|
||||
invalid credential refuses the whole batch, never falls back to "no
|
||||
tenant." What's still missing is the "does that identity actually
|
||||
change where the record is written" half: neither `ingest`'s own
|
||||
ClickHouse writer nor `search`'s independent Redpanda consumer reads
|
||||
that header back to route the write anywhere per-tenant yet. Every
|
||||
record still lands in the one shared ClickHouse database and the one
|
||||
shared (default) Tantivy index, regardless of tenant — correctly tagged,
|
||||
not yet isolated at write time. A newly-provisioned tenant's ClickHouse
|
||||
database and Tantivy index remain real, isolated, and queryable through
|
||||
`enterprise-api` — and permanently empty, until that write-routing split
|
||||
is built (likely another "second binary," mirroring `enterprise-api`
|
||||
itself), which is now scoped, disclosed remaining work, not an
|
||||
undesigned gap.
|
||||
|
||||
## System overview
|
||||
|
||||
@@ -114,10 +129,12 @@ sentryctl ──▶ api, alerting (Bearer token when SENTRYCTL_TOKEN is set)
|
||||
```
|
||||
|
||||
Ingest path (agent → Redpanda → ingest → ClickHouse, and Redpanda →
|
||||
search → Tantivy) carries no tenant concept at all yet either — every
|
||||
ingested log record lands in the one shared `logs` table/index. Tenant
|
||||
isolation for *ingest*, not just query, is out of scope for what's built
|
||||
so far and is not separately designed in
|
||||
search → Tantivy): `ingest` now resolves and tags each record with a
|
||||
real tenant ID (see "Read this first" above), but nothing downstream
|
||||
routes on it yet — every ingested log record still lands in the one
|
||||
shared `logs` table/index. Tenant isolation for the *write* path is
|
||||
still out of scope for what's built so far and is not separately
|
||||
designed in
|
||||
`/docs/phase-4-isolation-design.md`; named here as a gap that design doc
|
||||
doesn't yet cover, not just an implementation gap.
|
||||
|
||||
@@ -420,7 +437,8 @@ terms:
|
||||
| `system.*` ClickHouse metadata isolation | **Built, not live-verified** — same caveat as above |
|
||||
| Tantivy per-tenant index routing (`search/src/registry.rs`) | **Enforced, verified live** — real Tantivy indices, real cross-tenant probe, all passing |
|
||||
| Tantivy tenant_id resolution (`enterprise/internal/searchclient`) | **Enforced, verified live** — real gRPC wire-level test |
|
||||
| Ingest tenant-awareness (ClickHouse and Tantivy both) | **Not implemented, undesigned** — every ingested record lands in the single shared database/index regardless of tenant |
|
||||
| Ingest tenant *identity* (credential validation, tagging) | **Built and tested** — fail-closed `TenantResolver`, `tenant_id` Kafka header attached per record |
|
||||
| Ingest tenant *write-routing* (ClickHouse and Tantivy both) | **Not implemented, now scoped** — every record still lands in the single shared database/index regardless of tenant; consuming the tenant_id header to route the write is real, disclosed remaining work |
|
||||
| Deployment actually routing traffic to `enterprise-api` (Helm) | **Enforced** — `api`/`enterprise-api` are mutually exclusive, same flag as RBAC/audit/SSO |
|
||||
| Deployment actually routing traffic to `enterprise-api` (docker-compose) | **Enforced** — `api`/`enterprise-api` are mutually exclusive via `COMPOSE_PROFILES`, same flag choice as Helm's `enterprise.enabled`; verified via `docker compose config`, not an actual `docker compose up` in this environment |
|
||||
| Human SSO login — OIDC | **Built, verified with a real fake IdP** (not yet tried against a real external IdP) |
|
||||
|
||||
Reference in New Issue
Block a user