088677643f16a575972157fba1b0076c1898634c
11
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 jcoffey-dev
|
088677643f |
Close search's active-tenant write-routing gap with a polled allowlist
search/src/consumer.rs's write-routing (built last pass) had no active- tenant check at all: IndexRegistry.resolve() would open-or-create an index directory for any syntactically-valid tenant_id, active or not -- unlike ClickHouse's chwriter.Registry (an active-tenants-only snapshot built at enterprise-ingest startup) or the read side (gated by searchclient.TenantChecker, a direct rbacstore query). search is AGPL core with no Postgres access and no enterprise/ import allowed, so it needed a network boundary instead -- the same shape ingest's TenantResolver already uses against enterprise-auth, just Rust calling Go instead of Go calling Go. New GET /internal/active-tenants endpoint on enterprise-auth (rbacstore.ListActiveTenantIDs + authhandler.handleActiveTenants), gated on a RoleService Bearer credential -- server-to-server auth, the same shape alerting presents to api, minted via the already-generic enterprise-auth -mint-service-token search. search/src/tenants.rs's ActiveTenantTracker polls it every 60s, blocking startup on the first fetch succeeding (fail-closed cold start -- a control-plane outage at boot must not silently accept every tenant_id) and keeping the last- known-good set on any later refresh failure (a transient blip shouldn't stop every tenant's indexing, only prevent the allowlist from growing/ shrinking until connectivity resumes). consumer.rs refuses any tagged record whose tenant isn't in the polled set, before ever calling resolve() -- IndexRegistry itself stays policy-free, matching the same mechanism/policy split clickhousewriter.Writer vs. chwriter.Registry already draws on the ClickHouse side. Off unless ENTERPRISE_AUTH_URL/ENTERPRISE_AUTH_SERVICE_TOKEN are both set (search/src/config.rs rejects exactly one being set) -- every existing deployment is unaffected. Verified with real HTTP round trips in this environment: tenants.rs's tests exercise real reqwest requests (actual Authorization: Bearer header, actual JSON parsing) against a hand-rolled dependency-free TCP test server, including both fail-closed paths (rejected first fetch, unreachable server). authhandler's new tests cover the credential-kind distinction this endpoint exists to enforce -- a real human session, even for a genuine Owner, must not satisfy a check meant for a service identity. One asymmetry remains, disclosed rather than fixed: chwriter.Registry's snapshot still never refreshes (stale until enterprise-ingest restarts), while ActiveTenantTracker's 60s poll gives Tantivy a materially tighter staleness window. Neither is a live per-write check -- that would mean a database/HTTP round trip per record, a throughput cost neither implementation accepts -- so both have some staleness window by design; the gap between the two windows is what's disclosed, not a claim either is fully live. |
||
|
jcoffey-dev
|
5a845f06ee |
Add rbacstore.TransferOwner and enterprise-auth -transfer-owner-*
RevokeMembership refuses to revoke a tenant's current Owner (would leave tenants.owner_user_id dangling), but there was no way to actually hand ownership to someone else -- only the raw SetOwner primitive, which -grant-membership-role=owner used without downgrading whoever held Owner before, leaving tenant_memberships claiming two owners while owner_user_id can only name one. Named as a real, disclosed gap in docs/phase-4-runbook.md's "Known gaps". rbacstore.TransferOwner closes it: downgrades the current owner's membership to admin, promotes the new owner, and updates tenants.owner_user_id, all in one pgx transaction -- this package's first use of one. Every other mutation here is a single independent statement because nothing else needs more than one row to agree; this does, for the same reason RevokeMembership's doc comment already gives for refusing to revoke Owner in the first place. -grant-membership-role=owner now refuses when a *different* owner already exists, pointing at the new -transfer-owner-tenant/ -transfer-owner-user-email flags instead of silently producing the inconsistent state -- it remains correct, unchanged, for a tenant's first owner assignment. Verified with the same skip-gated live-Postgres discipline as the rest of this package (TestTransferOwnerMovesOwnershipAndDowngradesPrevious Owner proves the downgrade is real by then successfully revoking the former owner's now-non-Owner membership; two refusal-path tests cover no-current-owner and transfer-to-self) -- not run against a live database in this environment, same disclosed gap as everything else here. |
||
|
jcoffey-dev
|
abeee0076b |
Build and browser-verify the tenant-picker frontend page
web/src/routes/select-tenant now calls enterprise-auth's existing
GET /auth/memberships / POST /auth/select-tenant protocol (built earlier
this phase, previously called only from Go tests) via
fetch(..., {credentials: 'include'}) -- new listMemberships/selectTenant
functions in $lib/api.ts, using a dedicated request helper that reads
plain-text error bodies (loginhandler's http.Error responses), unlike
every other request helper in that file which expects JSON.
Credentialed cross-origin fetch needed CORS enterprise-auth didn't have:
api/httpserver.WithCORS's wildcard-friendly default can't be combined
with a credentialed request at all (browsers refuse to honor
Access-Control-Allow-Origin: "*" on one) -- added WithCredentialedCORS
(literal origin, Access-Control-Allow-Credentials: true) alongside it,
wired into enterprise-auth via a new CORS_ALLOWED_ORIGIN config var
defaulting to POST_LOGIN_REDIRECT_URL (web's own origin, the same
default pattern SELECT_TENANT_REDIRECT_URL already used).
adapter-static's route crawler doesn't discover a page nothing links to
(this one is only ever reached via enterprise-auth's redirect) -- fixed
with select-tenant/+page.ts's `export const prerender = true`, the same
declaration every other route already has.
Genuinely verified in a real browser in this environment, not just
type-checked: a throwaway Node server standing in for enterprise-auth's
exact wire contract (including its plain-text error bodies), driven
through the full flow via mcp__claude-in-chrome -- cross-origin
pending-login cookie set, credentialed preflight + GET/POST round trip,
a real click choosing a tenant, the post-selection redirect, and the
missing/expired-cookie error path rendering the backend's actual
message. No Docker or live Postgres/IdP needed, since the point was
exercising web's own fetch/CORS/cookie wiring, not enterprise-auth's
internals (already covered by loginhandler's own tests).
This closes the tenant-picker as the last named gap in Phase 4. What's
left is the already-disclosed live-verification caveat shared by every
Postgres/ClickHouse-backed piece and both SSO protocols: none of this
has run against a real database, external IdP, or multi-container
deployment in this environment.
|
||
|
jcoffey-dev
|
17fdc212c2 |
Give ingest a real tenant identity (write-routing deferred, disclosed)
Ingest tenant-awareness was named "undesigned, not just unbuilt" across CLAUDE.md/threat-model.md/the runbook since early Phase 4 -- the last major standing gap. Scoping was agreed via AskUserQuestion: a config-supplied tenant_id + shared-secret token ingest validates (smaller real implementation, no new PKI), over per-tenant mTLS certs. This change builds that identity mechanism end to end and attaches it to every record at the point it enters the system; it deliberately does NOT build per-tenant write-routing for ClickHouse or Tantivy -- that's real, separately-scoped follow-up work, disclosed explicitly everywhere this was previously called undesigned, not silently left half-done. New pieces: - metadata/migrations/0034 + enterprise/internal/rbacstore/ ingest_credentials.go: a per-tenant bearer credential, only its SHA-256 hash ever persisted (same reasoning a password gets hashed, not stored raw) -- CreateIngestCredential returns the plaintext exactly once, ValidateIngestCredential/RevokeIngestCredential/ ListIngestCredentialsForTenant round it out. - enterprise-auth gains -create-ingest-credential-tenant/ -list-ingest-credentials-tenant/-revoke-ingest-credential (same offline-operator-flag shape as every other credential-minting flag in this binary) and a new POST /internal/authorize-ingest endpoint (internal/authhandler) validating a presented token and resolving its tenant -- a genuinely different credential type from session-backed /internal/authorize, so it doesn't touch session.Manager at all. - ingest (AGPL core) gains an optional TenantResolver (internal/grpcserver, nil by default) and its HTTP client implementation (internal/tenantresolver.HTTPResolver) -- a plain HTTP call to enterprise-auth's new endpoint, never an enterprise/ import, same "network boundary, not import boundary" shape api/authz.HTTPAuthorizer already uses for the query path. PushBatch now requires an `authorization: Bearer <token>` gRPC metadata entry once a resolver is configured, fails the whole batch closed on a missing/invalid credential (never falls back to "no tenant"), and attaches the resolved tenant ID to every record as a `tenant_id` Kafka message header before producing it. Verified with real round trips at every layer, no Docker needed: rbacstore's credential CRUD (skip-gated on live Postgres, same as every other rbacstore integration test this phase), authhandler's new endpoint (real HTTP via httptest, including the regression test that a session token must not validate as an ingest credential), tenantresolver (real HTTP client against httptest, same pattern as authz.HTTPAuthorizer's own tests), and grpcserver's PushBatch (fake resolver/producer -- no resolver leaves messages unchanged, a configured resolver attaches the right header or fails closed on a bad/missing token). Helm: ingest.requireTenantCredential (default false) is a deliberate, separate opt-in from enterprise.enabled -- turning ENTERPRISE_AUTH_URL on for ingest requires every agent to already hold a credential or be refused outright, so it must not default on just because enterprise.enabled does (same reasoning api.yaml's ENTERPRISE_AUTH_URL isn't tied to enterprise.enabled directly either). docker-compose.yml leaves it unset, same as ever. Docs updated everywhere this was called "undesigned": CLAUDE.md, docs/architecture.md, docs/security/threat-model.md (including its summary table, now split into "identity: built" vs "write-routing: not yet"), docs/phase-4-runbook.md (new §13), enterprise/README.md. |
||
|
jcoffey-dev
|
d2c76aa3a4 |
Build the tenant-picker backend protocol (no frontend yet, by design)
A multi-membership identity (belongs to more than one tenant) used to
get a flat 501 refusal -- named as undesigned future work across
CLAUDE.md/threat-model.md/the runbook since early Phase 4. Scope for
this change was agreed via AskUserQuestion: backend protocol only,
fully verified via real HTTP round trips, not the actual picker page --
web has zero session/cookie-handling code today (confirmed while
researching this), so building that is separately-scoped, unverifiable
frontend work in this environment (no live backend, no browser).
session.Manager gains IssuePendingLogin/ValidatePendingLogin, a second
JWT token type proving identity without committing to a tenant yet
(10-minute TTL). PendingLoginClaims is deliberately a distinct Go type
from Claims, and -- caught by this change's own test suite before it
shipped -- needed a JSON field name disjoint from Claims.UserID's
"user_id" too: go-jose's unmarshal is happy to populate a struct from
any token whose claims happen to share a key, so a real session token
would otherwise have parsed successfully as a pending login. Fixed via
"pending_user_id" instead; both directions (session-as-pending,
pending-as-session) now have regression tests.
rbacstore.ListMembershipsWithTenantForUser joins tenant_memberships
with tenants, since a picker needs display names, not just IDs.
loginhandler.resolveIdentity's multiple-membership branch no longer
errors -- finishLogin routes it into startTenantSelection instead,
which issues a pending-login cookie (Path=/auth, so it's never sent on
ordinary requests) and redirects to a new configurable
SelectTenantRedirectURL (defaults to {POST_LOGIN_REDIRECT_URL}/select-
tenant). Two new routes complete the round trip: GET /auth/memberships
lists the pending identity's real tenant options, and POST
/auth/select-tenant re-derives the role for the chosen tenant
server-side (never trusts a client-supplied role, refuses a tenant_id
outside the identity's actual memberships with 403) before issuing the
real session -- responding with JSON {"redirect_url": ...}, not a
redirect, since a POST/fetch caller should control its own navigation.
Verified with the same real-fake-IdP tests the rest of this package
uses (coreos/go-oidc's oidctest, crewjam/saml's samlidp): the full
login -> pending cookie -> GET /auth/memberships -> POST
/auth/select-tenant -> real session round trip for both protocols, plus
negative paths (missing/expired pending cookie, a tenant_id outside
membership, a real session token rejected as a pending login and vice
versa). ErrMultipleMemberships is removed -- it's not an error path
anymore.
Docs updated in lockstep: CLAUDE.md, threat-model.md (including its
summary table), phase-4-runbook.md (new §12), enterprise/README.md
(new "Tenant selection" section, explicit about what's still not built
and why: no session handling in web, no CORS on enterprise-auth).
|
||
|
jcoffey-dev
|
cfcbc77507 |
Add enterprise-auth -revoke-membership-*/-list-memberships-tenant flags
Rounds out the tenant_memberships operator flags added earlier this
phase (-create-tenant/-grant-membership-*): grant had no way to undo
itself, and there was no way to see who was actually in a tenant
without querying Postgres directly.
rbacstore.RevokeMembership deletes a tenant_memberships row, but
refuses to revoke a tenant's current Owner -- Owner is also a
dedicated tenants.owner_user_id column (SetOwner), so deleting that
membership without transferring ownership first would leave
owner_user_id pointing at a user with no membership in the tenant at
all. Ownership transfer is a deliberate, separate action per the RBAC
matrix ("Transfer tenant Owner -- Owner only"), not a side effect of
revoking access.
rbacstore.ListMembershipsForTenant is ListMembershipsForUser's
inverse -- joined with users so the result is actually useful (email,
display name), not just a bare user ID list.
enterprise-auth gains -revoke-membership-tenant/
-revoke-membership-user-email (both required together, same shape as
-grant-membership-*) and -list-memberships-tenant (prints tab-separated
id/email/display-name/role to stdout and exits -- an operator
convenience, not a machine-readable API; this binary deliberately has
no admin HTTP surface, per its own doc comment on why). Changing a
role is unchanged: re-run -grant-membership-* with a different
-grant-membership-role, SetMembership's upsert already handles it.
Covered by five new skip-gated integration tests in rbacstore_test.go
(RBACSTORE_TEST_POSTGRES_ADDR), including the Owner-revocation refusal
and cross-tenant leak check for ListMembershipsForTenant -- not run
against a live database in this environment, same disclosed gap as the
rest of this phase's Postgres-backed work. Docs updated:
phase-4-runbook.md's known-gaps bullet, enterprise/README.md's
bootstrap walkthrough and package layout table.
|
||
|
jcoffey-dev
|
b8b6a8fd7b |
Add enterprise-auth -create-tenant/-grant-membership-* operator flags
Replaces the manual psql INSERT dance phase-4-runbook.md's §3a/§3b documented for bootstrapping the very first tenant_memberships row (create the tenant, log in once so UpsertUserBySSO creates a users row, hand-write an INSERT with that user's UUID). Two new offline operator flags, same "gated by access to enterprise-auth's own environment, not a network-reachable endpoint" shape as -mint-service-token and enterprise-api's -provision-tenant: - -create-tenant=<id> [-display-name=<name>]: creates a tenant row in rbacstore (control-plane only -- pair with enterprise-api -provision-tenant separately for ClickHouse/Tantivy data-plane provisioning, still two operator actions today, a named gap this doesn't unify). Refuses to run twice for the same id. - -grant-membership-tenant/-grant-membership-user-email/ -grant-membership-role: grants a tenant_memberships row by email instead of requiring the operator to hand-look-up a UUID. The user must already exist (attempted an SSO login at least once -- this flag deliberately never creates a user itself, since that identity has to come from a real IdP round trip). role=owner also calls SetOwner, since Owner is a dedicated tenants.owner_user_id column, not just the highest tenant_memberships role. Deliberately kept as offline flags rather than an authenticated HTTP admin API: an HTTP endpoint would have to solve "who's allowed to create the very first tenant/membership" itself, a real bootstrap problem the offline-flag pattern already used elsewhere in this binary sidesteps entirely. New rbacstore.GetUserByEmail supports the email-based lookup (email is already the natural key UpsertUserBySSO upserts on). Covered by two new skip-gated integration tests in rbacstore_test.go, same RBACSTORE_TEST_POSTGRES_ADDR convention as the rest of this package -- not run against a live database in this environment, consistent with everything else in this phase's Postgres-backed work. No dedicated test for the main.go flag handlers themselves, matching the existing precedent for -mint-service-token/-provision-tenant (neither has one either). Docs updated: phase-4-runbook.md's §3a/§3b bootstrap steps and its "known gaps" list, enterprise/README.md gets a new "Bootstrapping a tenant and its first human user" section and a stale "there's no login flow to issue a human session yet" line (obsolete since OIDC/SAML login shipped) is fixed. |
||
|
jcoffey-dev
|
08a90a27aa |
Build SAML login (enterprise/internal/loginhandler), mirroring OIDC
Adds GET /auth/saml/login + POST /auth/saml/acs alongside the existing OIDC pair, both converging on the same upsert-user/resolve-tenant/ issue-session path. loginhandler.New now takes an optional *saml.ServiceProvider, RegisterRoutes registers each protocol's routes independently so either, both, or neither can be configured. SAML's replay/unsolicited-response defense (InResponseTo, standing in for OIDC's state) is carried via a SameSite=None sentry_saml_request cookie -- None because the ACS endpoint receives a cross-site POST from the IdP's origin, which SameSite=Lax cookies are never sent on. enterprise-auth's main.go now fetches+parses SAML_IDP_METADATA_URL at startup (samlsp.FetchMetadata) and wires the result through. Verified to the same bar as OIDC: a real fake IdP (crewjam/saml/samlidp, genuine XML signing/verification) drives the full login->ACS->session-cookie round trip and negative paths (bad InResponseTo, missing request cookie, missing email/NameID, no/multiple tenant memberships), all in loginhandler/saml_test.go, no Docker needed. The login-form HTML is bypassed by pre-seeding a saml.Session directly into samlidp's session store and presenting the matching `session` cookie -- an IdP-supported shortcut (confirmed by reading GetSession), the same "skip the UI, keep the crypto real" approach oidctest gave the OIDC tests. Writing that test caught two real bugs in internal/saml.ParseResponse, both fixed here: it never called r.ParseForm() before reading the POSTed SAMLResponse field, so every real ACS POST would have silently decoded an empty response; and its email-attribute matching missed urn:oid:0.9.2342.19200300.100.1.3 (the standard LDAP "mail" OID), which is what an IdP sends by default absent an explicit AttributeConsumingService request for "email" -- exactly what samlidp's own DefaultAssertionMaker does, and plausibly what real IdPs' default SAML app templates do too. Docs (CLAUDE.md, threat-model.md, architecture.md, enterprise/README.md, phase-4-runbook.md, docker-compose.yml's enterprise-auth comment) updated in lockstep: SAML login moves from "protocol mechanics only" to "built, verified with a real fake IdP, not yet tried against a real external IdP or a running enterprise-auth container" -- the same disclosed gap OIDC already carried. |
||
|
jcoffey-dev
|
1fab02abd5 |
Phase 4: real OIDC human login (enterprise/internal/loginhandler)
Closes the other major named gap from this phase: until now, there was no way for a human to actually log in -- only /alerting's RoleService credential could be minted. GET /auth/oidc/login and GET /auth/oidc/callback drive the real coreos/go-oidc flow already wired in enterprise/internal/oidc: CSRF state in a short-lived cookie, code exchange, ID token verification, upserting a users row, resolving tenant/role from exactly one tenant_memberships row (refusing outright on zero or more than one, rather than guessing), and issuing a real session cookie. Unlike everything else built this phase, this one is genuinely verified end to end: the tests spin up coreos/go-oidc's own oidctest fake IdP, which signs real RS256 ID tokens, and drive the full login->callback-> session-cookie round trip through actual signature verification -- no live database or Docker needed, so nothing here is asserted without having actually been run in this session. Also fixes a real bug caught while wiring this into enterprise-auth's main.go: assigning a nil *oidc.Provider to the handler's interface field would have produced a non-nil interface wrapping a nil pointer (Go's classic typed-nil trap), silently breaking the "OIDC not configured" no-op path -- New() now takes the concrete pointer type and checks it before ever converting to the interface, with a regression test pinning the fix down. Still missing: SAML's equivalent (ACS endpoint), a tenant-picker UI for multi-membership identities, and any admin UI to actually create a tenant_memberships row (today that's manual SQL, documented in the runbook's new bootstrap walkthrough). |
||
|
jcoffey-dev
|
1d57e697b1 |
Phase 4: real per-tenant ClickHouse isolation via a new enterprise-api binary
Closes the threat model's headline finding for the SQL query path:
enterprise/internal/tenantprovision does real CREATE DATABASE/USER/GRANT
against ClickHouse, and enterprise/internal/chrunner is a per-tenant
connection registry implementing api's SQLRunner interface, resolving
the tenant from the authenticated request identity -- never a
caller-suppliable parameter. Both are wired into a new binary,
enterprise/cmd/enterprise-api, alongside the unchanged single-tenant
api/cmd/api, since AGPL core can never import enterprise/ and Go's own
internal/ package visibility rules meant enterprise/ couldn't implement
core's SQLRunner interface without importing the package that defines
it. That required moving api/internal/{authz,queryapi,dashboards,
querylang/executor,searchclient,httpserver} out of internal/ -- the
minimal set enterprise-api needs to import; querylang's compiler
internals (planner/lexer/parser/ast/ir) and api's own config stay
internal, since nothing outside api needs them directly.
Also finally wires enterprise/internal/audit into queryapi.AuditLogger
(nil since Phase 4 task 4) via a new adapter, and adds live-ClickHouse
integration tests for two of the four adversarial probes named in
docs/phase-4-isolation-design.md's verification plan.
Corrected several overclaims in the docs while writing this up: an
earlier claim that rbacstore's CRUD was "verified against a live
Postgres" was never actually true in this environment (only
internal/audit was, earlier in this phase, before Docker access was
lost) -- threat-model.md, phase-4-runbook.md, CLAUDE.md, and
enterprise/README.md all now distinguish "a real integration test
exists" from "this was confirmed against a live database."
Still not built: Tantivy/free-text tenant isolation
(enterprise/internal/searchclient), and any deployment-topology
mechanism that actually routes traffic to enterprise-api instead of
plain api -- both binaries exist side by side today with nothing
enforcing or flagging which one a deployment runs.
|
||
|
jcoffey-dev
|
3eb0f4c589 |
Phase 4: SSO scaffolding, RBAC enforcement, tenant-scoped dashboards, audit logging, K8s deployment
RBAC (api/internal/authz) is live on /query and /dashboards, backed by a new enterprise/ module (session issuance, audit logging, RBAC storage, OIDC/SAML protocol wiring) that core never imports -- only calls over HTTP. Found and fixed a real cross-tenant vulnerability in dashboards (no tenant_id filtering at all) while writing the threat model doc. Two things are explicitly NOT done, documented rather than hidden: tenant isolation for log data itself (/query still shares one ClickHouse connection and Tantivy index across every tenant -- RBAC controls who can query, not what a query can see), and human SSO login (protocol wiring exists, no HTTP handler calls it yet). See docs/security/threat-model.md and docs/phase-4-runbook.md. Also adds deploy/ (Go Operator + Helm chart, validated offline only -- no cluster was reachable in this environment). |