b7f99b49f2043dcbb952a99cb884b90be5fa569f
24
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 jcoffey-dev
|
f756a9d4f6 |
Rename the project spec and update every reference to it
The charter file carried a tool-specific name while being the repository's own document: mission, non-negotiable constraints, the pinned stack, repo conventions and phase status, cited as authority by thirty files across the agent, api, deploy, docs, search and terraform trees. PROJECT-SPEC.md says what it is. All 42 references are updated in the same commit, including the relative link in docs/status.md, so nothing points at a filename that no longer exists. |
||
|
jcoffey-dev
|
c920e0f2c4 |
Finish the Cairn OBS rename through services, docs, and assets
The rename commit before this one covered module paths and the obvious user-facing strings; this is the rest of it -- the places where "sentry" was a default value, a filename, or a picture rather than a word in a sentence. Defaults that changed: CLICKHOUSE_DATABASE (sentry -> cairnobs), POSTGRES_DATABASE (sentry_metadata -> cairnobs_metadata), and POSTGRES_USERNAME (sentry -> cairnobs), across api/alerting/ingest and the enterprise binaries, plus the compose files and migrate scripts that create those objects. These are *defaults*, so a deployment that sets them explicitly is unaffected -- but any deployment relying on the old defaults must have its environment updated before it picks this up, or it will come up pointing at a database that doesn't exist. Also: the light-mode logo variants (the dark ones existed alone, so the landing page and sidebar rendered a dark mark on a light background), regenerated favicons, and the docs/README/threat-model prose that still said Sentry. |
||
|
jcoffey-dev
|
13cf9a30cb |
Rebrand: Sentry -> Cairn OBS
Full rebrand across cosmetic branding, code identifiers, and infrastructure/data-plane naming, using the supplied Cairn OBS logo package. Cosmetic: favicon/logo swap (also closes a stale license-audit finding -- the old favicon was SvelteKit's unreplaced scaffold logo), new centered welcome landing page, larger/legible sidebar logo, page titles, CLAUDE.md/README/docs prose. Code identifiers: Go module path github.com/sentry/sentry -> github.com/cairnobs/cairnobs across all 13 modules and ~91 files (protoc regenerated); Rust crates sentry-agent/sentry-parser/sentry-search -> cairnobs-*; CLI sentryctl -> cairnobsctl; Terraform provider fully renamed (sentry_dashboard etc. -> cairnobs_dashboard, provider type, env vars); every session/auth cookie name; agent config paths and Windows service identity. Deliberately preserved: the gRPC wire protocol's protobuf packages (sentry.logs.v1, sentry.agent.v1) and their Go import directory (proto/sentry/...) -- renaming the wire-level package would break every currently-deployed agent binary (confirmed two real hosts, including mail.inbuxa.com, are actively streaming through this exact contract) until rebuilt and redeployed in lockstep with an ingest cutover. Only the Go module path wrapping the generated code changes. Infrastructure: every docker-compose container name (root and three component-level compose files); the Helm chart (directory, Chart.yaml, named-template helpers, all templates, values.yaml image repos); Kubernetes Operator (CRD group sentry.io -> cairnobs.io, both CRD YAML files, Go identifiers, RBAC markers); the coupled enterprise/tenantcrd package. Caught and fixed real path-coupling bugs along the way: the Helm chart's search/ingest volume mounts and the dev-only-credential detection constant vs. docker-compose.yml's literal values had to move together or a security warning would have silently stopped firing. Data plane: Postgres database sentry_metadata -> cairnobs_metadata and role sentry -> cairnobs; ClickHouse database sentry -> cairnobs; Kafka topic sentry.logs.raw -> cairnobs.logs.raw and its consumer groups. Source-level defaults, docker-compose.yml, and every migrate.sh/ provision script default updated together; already-applied migration files left untouched per this repo's immutable-migration convention. Verified at every layer: all 13 Go modules build/vet/test clean, both Rust workspaces (agent, search) build/clippy/test clean, npm run check/ build clean, docker compose config validates on all four compose files. Live-verified against a real docker stack multiple times through this work, including a final fresh-volume run confirming the actual renamed Postgres database/role, ClickHouse database, and Kafka topic all work end to end with a real login and query, zero console errors. |
||
|
jcoffey-dev
|
661568085e |
Phase 6: license-compliance audit and enterprise/ relicensing to AGPLv3
Full dependency inventory across Rust/Go/npm plus Docker base images and vendored assets (776 rows, 502 unique deps), classified against AGPLv3 compatibility with real citations rather than assumptions. enterprise/ relicensed from its commercial-license stub to AGPLv3, matching core -- the one real flag (Redpanda's BSL 1.1) was evaluated against primary sources and accepted as-is rather than triggering a broker swap. CI enforcement wired up (.github/workflows/license- compliance.yml, this repo's first CI workflow), a root LICENSE file added, and every doc/comment referencing the old commercial-license boundary updated to describe it as architectural only. See /docs/compliance/ for the full report, inventory, and policy. |
||
|
jcoffey-dev
|
05d166cfa9 |
Document real kind-cluster verification, closing every runbook gap
kind/kubectl/helm were installed without root and a real local cluster ran the full two-tenant walkthrough end to end: both acme and globex reached Tenant.status.phase: Active with real generated ClickHouse credentials. Updates deploy/README.md's and deploy/helm/sentry/README.md's verification-status framing from "not verified against a live cluster" to what's actually true now, fixes both READMEs' helm install --include-crds (a helm template-only flag, never valid for install), documents the two chart bugs this run found (see the previous commit) and the mandatory ingest TLS Secret step, and updates the threat model's summary table and "No general multi-cluster orchestration" residual-risk note accordingly. This closes the last remaining gap in /docs/phase-4-runbook.md. |
||
|
jcoffey-dev
|
783d66786b |
Document real Auth0 SAML verification, closing §3b -- the last SSO gap
Auth0's SAML2 Web App addon (the same dev tenant §3a used) stood in as a real SAML IdP, over a genuine self-signed TLS proxy in front of enterprise-auth (required, not optional, for SAML's SameSite=None cookie). Full round trip confirmed: real signed assertion, audience/ destination/signature validation, correct multi-membership handling, and POST /internal/authorize returning the selected tenant/role. Updates the runbook's verification status, §3b, and the threat model's "Read this first" finding and summary table to reflect this and the isSecureRequest fix it found. §7/§11's live-cluster steps (no kind/kubectl in this environment) are now the only remaining gap in the entire runbook. |
||
|
jcoffey-dev
|
17a2fda939 |
Document real Auth0 OIDC + tenant-picker verification, closing §3a/§12
A free Auth0 developer tenant was wired into enterprise-auth via a local-only docker-compose.override.yml and driven through a real browser: login correctly failed closed with no tenant membership while still creating the users row, then succeeded after -grant-membership-* and issued a real session. With a second real membership granted, the multi-membership path landed on the real /select-tenant page, rendered both real tenants with correct roles via a real credentialed cross-origin request to the real enterprise-auth container, and selecting either one issued a session that POST /internal/authorize confirmed matched. Updates the runbook's top-level verification status, §3a, and §12, plus the threat model's "Read this first" finding and summary table to reflect what's now genuinely confirmed versus what still needs SAML's real IdP (§3b) or a real cluster (§7/§11). |
||
|
jcoffey-dev
|
010f66ec70 |
Update Phase 4 docs: the ClickHouse-side live verification actually ran
Docker access became available and §§1-10, 10a, 13, 14, and most of §8 of the runbook have now genuinely been run against a real docker- compose stack, not just documented as a procedure to run. Rewrites the runbook's "Verification status" section and the threat model's "Read this first" finding to describe what was actually confirmed (including the six real bugs this pass found and fixed) versus what's still gated on a real external IdP (§3a/§3b/§12) or a real Kubernetes cluster (§7/§11's live-cluster halves). Also fixes two stale runbook commands: §5a/§6's docker run mounts needed the repo root, not just enterprise/, for the same replace-directive reason the Dockerfile fix does; §14 cited a chwriter test name that never existed. |
||
|
jcoffey-dev
|
2e8ab1ed6a |
Give chwriter.Registry periodic refresh, matching Tantivy's tracker
Closing search's active-tenant gap last commit surfaced a real asymmetry by comparison: chwriter.Registry's per-tenant writer map was still a snapshot built once at enterprise-ingest startup with no refresh at all, while search's new ActiveTenantTracker refreshes every minute. A tenant deprovisioned after enterprise-ingest started would keep writing successfully to ClickHouse until the next restart -- a real, disclosed staleness gap, not matched by anything on the Tantivy side anymore. Registry.StartRefreshing spawns a goroutine that re-lists active tenants every minute (dataSourceRefreshInterval, same interval as search's tracker) via a new SourceLister callback and reconciles the writer map: opens a connection for a newly-active tenant, closes and removes one no longer active. New connections are dialed before taking the write lock, so a slow/unreachable ClickHouse for one newly-active tenant never blocks WriteBatch's read lock. A refresh failure (lister error, or one tenant's connection failing to open) logs and leaves the existing map untouched for that tick -- the same last-known-good posture ActiveTenantTracker already uses, so a transient rbacstore/Postgres blip doesn't evict every other tenant's already-working writer. WriteBatch now takes a read lock and Close takes a write lock -- the writer map was safe unsynchronized before only because it was immutable after New() returned; StartRefreshing makes it mutable at runtime. enterprise-ingest/main.go extracts the existing rbacstore-row-to- DataSource adaptation into tenantDataSourceLister, reused for both the initial synchronous load and StartRefreshing's periodic calls, so the two can't drift into checking different things. Verified: the lister-error-keeps-last-known-good path is Docker-free (same "construct a Registry directly, bypass New" trick the existing fail-closed tests use). The actual add/remove reconciliation against real ClickHouse connections (TestRefreshAddsNewlyActiveTenant, TestRefreshRemovesNoLongerActiveTenant) are skip-gated live-ClickHouse tests, same CHWRITER_TEST_CLICKHOUSE_ADDR convention as this package's existing integration tests -- not run against a live database in this environment. This closes the last disclosed gap from Phase 4's write-routing work: both storage engines now share the same one-minute active-tenant staleness bound instead of one being materially staler than the other. |
||
|
jcoffey-dev
|
088677643f |
Close search's active-tenant write-routing gap with a polled allowlist
search/src/consumer.rs's write-routing (built last pass) had no active- tenant check at all: IndexRegistry.resolve() would open-or-create an index directory for any syntactically-valid tenant_id, active or not -- unlike ClickHouse's chwriter.Registry (an active-tenants-only snapshot built at enterprise-ingest startup) or the read side (gated by searchclient.TenantChecker, a direct rbacstore query). search is AGPL core with no Postgres access and no enterprise/ import allowed, so it needed a network boundary instead -- the same shape ingest's TenantResolver already uses against enterprise-auth, just Rust calling Go instead of Go calling Go. New GET /internal/active-tenants endpoint on enterprise-auth (rbacstore.ListActiveTenantIDs + authhandler.handleActiveTenants), gated on a RoleService Bearer credential -- server-to-server auth, the same shape alerting presents to api, minted via the already-generic enterprise-auth -mint-service-token search. search/src/tenants.rs's ActiveTenantTracker polls it every 60s, blocking startup on the first fetch succeeding (fail-closed cold start -- a control-plane outage at boot must not silently accept every tenant_id) and keeping the last- known-good set on any later refresh failure (a transient blip shouldn't stop every tenant's indexing, only prevent the allowlist from growing/ shrinking until connectivity resumes). consumer.rs refuses any tagged record whose tenant isn't in the polled set, before ever calling resolve() -- IndexRegistry itself stays policy-free, matching the same mechanism/policy split clickhousewriter.Writer vs. chwriter.Registry already draws on the ClickHouse side. Off unless ENTERPRISE_AUTH_URL/ENTERPRISE_AUTH_SERVICE_TOKEN are both set (search/src/config.rs rejects exactly one being set) -- every existing deployment is unaffected. Verified with real HTTP round trips in this environment: tenants.rs's tests exercise real reqwest requests (actual Authorization: Bearer header, actual JSON parsing) against a hand-rolled dependency-free TCP test server, including both fail-closed paths (rejected first fetch, unreachable server). authhandler's new tests cover the credential-kind distinction this endpoint exists to enforce -- a real human session, even for a genuine Owner, must not satisfy a check meant for a service identity. One asymmetry remains, disclosed rather than fixed: chwriter.Registry's snapshot still never refreshes (stale until enterprise-ingest restarts), while ActiveTenantTracker's 60s poll gives Tantivy a materially tighter staleness window. Neither is a live per-write check -- that would mean a database/HTTP round trip per record, a throughput cost neither implementation accepts -- so both have some staleness window by design; the gap between the two windows is what's disclosed, not a claim either is fully live. |
||
|
jcoffey-dev
|
abeee0076b |
Build and browser-verify the tenant-picker frontend page
web/src/routes/select-tenant now calls enterprise-auth's existing
GET /auth/memberships / POST /auth/select-tenant protocol (built earlier
this phase, previously called only from Go tests) via
fetch(..., {credentials: 'include'}) -- new listMemberships/selectTenant
functions in $lib/api.ts, using a dedicated request helper that reads
plain-text error bodies (loginhandler's http.Error responses), unlike
every other request helper in that file which expects JSON.
Credentialed cross-origin fetch needed CORS enterprise-auth didn't have:
api/httpserver.WithCORS's wildcard-friendly default can't be combined
with a credentialed request at all (browsers refuse to honor
Access-Control-Allow-Origin: "*" on one) -- added WithCredentialedCORS
(literal origin, Access-Control-Allow-Credentials: true) alongside it,
wired into enterprise-auth via a new CORS_ALLOWED_ORIGIN config var
defaulting to POST_LOGIN_REDIRECT_URL (web's own origin, the same
default pattern SELECT_TENANT_REDIRECT_URL already used).
adapter-static's route crawler doesn't discover a page nothing links to
(this one is only ever reached via enterprise-auth's redirect) -- fixed
with select-tenant/+page.ts's `export const prerender = true`, the same
declaration every other route already has.
Genuinely verified in a real browser in this environment, not just
type-checked: a throwaway Node server standing in for enterprise-auth's
exact wire contract (including its plain-text error bodies), driven
through the full flow via mcp__claude-in-chrome -- cross-origin
pending-login cookie set, credentialed preflight + GET/POST round trip,
a real click choosing a tenant, the post-selection redirect, and the
missing/expired-cookie error path rendering the backend's actual
message. No Docker or live Postgres/IdP needed, since the point was
exercising web's own fetch/CORS/cookie wiring, not enterprise-auth's
internals (already covered by loginhandler's own tests).
This closes the tenant-picker as the last named gap in Phase 4. What's
left is the already-disclosed live-verification caveat shared by every
Postgres/ClickHouse-backed piece and both SSO protocols: none of this
has run against a real database, external IdP, or multi-container
deployment in this environment.
|
||
|
jcoffey-dev
|
bdd42e06f6 |
Build per-tenant Tantivy write-routing, closing the last ingest write gap
search/src/consumer.rs now resolves each record's tenant_id Kafka header through the same IndexRegistry the read side (search/src/registry.rs + enterprise/internal/searchclient) already used, and writes into that tenant's own index instead of always the default one. The periodic Tantivy commit now commits every tenant index that's actually seen a write (IndexRegistry::commit_all), not just the default index. Unlike ClickHouse, this needed no "second binary": Tantivy has no grant system to gate a commercially-licensed credential behind, so IndexRegistry already lived directly in this AGPL-core search binary -- there was never an import-boundary reason to split the write side into an enterprise/ binary the way chwriter/enterprise-ingest was for ClickHouse. Read and write simply share one registry. Because Tantivy is an embedded library, this is genuinely verified in this environment, not just written: registry.rs's commit_all_commits_default_and_every_opened_tenant_index writes into the default index plus two tenant indices, confirms nothing is searchable pre-commit, then confirms all three are post-commit. consumer.rs's tenant_id_from_headers is factored out as a small pure helper (mirroring ingest/consumer.tenantIDFromHeaders) with its own unit tests, plus a guard test against the "tenant_id" header-key literal drifting from the Go side's -- the same guard-test pattern ingest/cmd/ingest already used for its own two Go copies of the constant, now mirrored a third time across the language boundary. One gap is disclosed, not fixed, by this change: unlike chwriter.Registry (an active-tenants-only snapshot built at enterprise-ingest startup, so an unrecognized tenant_id is refused outright) and unlike the read side (gated by searchclient.TenantChecker), this consumer's registry.resolve() call has no active-tenant check at all -- search has no Postgres access to check tenant status against. A still-valid-but-should-be-revoked ingest credential can cause an index directory to be created for a tenant that's no longer active. Narrow blast radius (an orphan, isolated, empty index, not cross-tenant leakage, and only reachable with a real signed credential), but real -- see registry.rs's doc comment on resolve(). Closing it fully would mean giving search some way to learn which tenants are active without an enterprise/ import, which isn't designed yet. This closes the last of Phase 4's ingest write-routing gaps (ClickHouse was closed last commit). The one remaining gap in the whole phase is now the tenant-picker frontend page, deliberately deferred earlier in this phase as out of scope for this environment. |
||
|
jcoffey-dev
|
1de77b969f |
Build per-tenant ClickHouse write-routing for ingest (Tantivy still deferred)
ingest tags every record with a tenant_id Kafka header (built previously), but nothing consumed it to actually route the write. This closes that for ClickHouse: enterprise/cmd/enterprise-ingest (a second binary, mirroring enterprise-api) reuses ingest/consumer's own flush loop unchanged, with enterprise/internal/chwriter.Registry -- a per-tenant clickhousewriter.Writer registry -- swapped in as the writer. A batch pulled from the single shared Redpanda topic can mix records from many tenants, so WriteBatch groups by TenantID and dispatches each group to its own tenant's connection, fail- closed on an empty or unrecognized tenant_id. ingest/consumer and ingest/clickhousewriter move out of internal/ (same reason api/internal/* moved earlier this phase: enterprise/ can't import anything under another module's internal/). Their New() constructors now take small local Config structs instead of ingest/internal/config types, so enterprise/ doesn't need that import either. Building this surfaced a real bug: tenantprovision.ProvisionClickHouse only granted SELECT on a tenant's ClickHouse user, correct for chrunner's read-only use but not enough for chwriter reusing the same credential to write -- every real per-tenant write would have failed closed with a permission error. Fixed by widening the grant to SELECT, INSERT; no cross-tenant boundary is crossed by also allowing INSERT within a tenant's own database. Helm gates enterprise-ingest's Deployment on the same ingest.requireTenantCredential flag that already gates tag validation -- write-routing is meaningless without tagging already being required, so they're one decision, not two. docker-compose.yml's version is a disclosed, weaker approximation: it can't achieve Helm's genuine -mode=server/-mode=consumer split, so with the enterprise profile active both ingest and enterprise-ingest independently consume every message via different consumer groups -- harmless duplication for local verification only. Not built: Tantivy's independent Redpanda consumer (search/src/consumer.rs) still doesn't read the tenant_id header at all -- every record still lands in the one shared index regardless of tenant. Not run: the live-ClickHouse- gated tests (chwriter's cross-tenant routing test, tenantprovision's INSERT regression test) -- no Docker/database access in this environment; they're correct Go that has never executed, disclosed as such in docs/security/ threat-model.md and docs/phase-4-runbook.md §14. |
||
|
jcoffey-dev
|
17fdc212c2 |
Give ingest a real tenant identity (write-routing deferred, disclosed)
Ingest tenant-awareness was named "undesigned, not just unbuilt" across CLAUDE.md/threat-model.md/the runbook since early Phase 4 -- the last major standing gap. Scoping was agreed via AskUserQuestion: a config-supplied tenant_id + shared-secret token ingest validates (smaller real implementation, no new PKI), over per-tenant mTLS certs. This change builds that identity mechanism end to end and attaches it to every record at the point it enters the system; it deliberately does NOT build per-tenant write-routing for ClickHouse or Tantivy -- that's real, separately-scoped follow-up work, disclosed explicitly everywhere this was previously called undesigned, not silently left half-done. New pieces: - metadata/migrations/0034 + enterprise/internal/rbacstore/ ingest_credentials.go: a per-tenant bearer credential, only its SHA-256 hash ever persisted (same reasoning a password gets hashed, not stored raw) -- CreateIngestCredential returns the plaintext exactly once, ValidateIngestCredential/RevokeIngestCredential/ ListIngestCredentialsForTenant round it out. - enterprise-auth gains -create-ingest-credential-tenant/ -list-ingest-credentials-tenant/-revoke-ingest-credential (same offline-operator-flag shape as every other credential-minting flag in this binary) and a new POST /internal/authorize-ingest endpoint (internal/authhandler) validating a presented token and resolving its tenant -- a genuinely different credential type from session-backed /internal/authorize, so it doesn't touch session.Manager at all. - ingest (AGPL core) gains an optional TenantResolver (internal/grpcserver, nil by default) and its HTTP client implementation (internal/tenantresolver.HTTPResolver) -- a plain HTTP call to enterprise-auth's new endpoint, never an enterprise/ import, same "network boundary, not import boundary" shape api/authz.HTTPAuthorizer already uses for the query path. PushBatch now requires an `authorization: Bearer <token>` gRPC metadata entry once a resolver is configured, fails the whole batch closed on a missing/invalid credential (never falls back to "no tenant"), and attaches the resolved tenant ID to every record as a `tenant_id` Kafka message header before producing it. Verified with real round trips at every layer, no Docker needed: rbacstore's credential CRUD (skip-gated on live Postgres, same as every other rbacstore integration test this phase), authhandler's new endpoint (real HTTP via httptest, including the regression test that a session token must not validate as an ingest credential), tenantresolver (real HTTP client against httptest, same pattern as authz.HTTPAuthorizer's own tests), and grpcserver's PushBatch (fake resolver/producer -- no resolver leaves messages unchanged, a configured resolver attaches the right header or fails closed on a bad/missing token). Helm: ingest.requireTenantCredential (default false) is a deliberate, separate opt-in from enterprise.enabled -- turning ENTERPRISE_AUTH_URL on for ingest requires every agent to already hold a credential or be refused outright, so it must not default on just because enterprise.enabled does (same reasoning api.yaml's ENTERPRISE_AUTH_URL isn't tied to enterprise.enabled directly either). docker-compose.yml leaves it unset, same as ever. Docs updated everywhere this was called "undesigned": CLAUDE.md, docs/architecture.md, docs/security/threat-model.md (including its summary table, now split into "identity: built" vs "write-routing: not yet"), docs/phase-4-runbook.md (new §13), enterprise/README.md. |
||
|
jcoffey-dev
|
d2c76aa3a4 |
Build the tenant-picker backend protocol (no frontend yet, by design)
A multi-membership identity (belongs to more than one tenant) used to
get a flat 501 refusal -- named as undesigned future work across
CLAUDE.md/threat-model.md/the runbook since early Phase 4. Scope for
this change was agreed via AskUserQuestion: backend protocol only,
fully verified via real HTTP round trips, not the actual picker page --
web has zero session/cookie-handling code today (confirmed while
researching this), so building that is separately-scoped, unverifiable
frontend work in this environment (no live backend, no browser).
session.Manager gains IssuePendingLogin/ValidatePendingLogin, a second
JWT token type proving identity without committing to a tenant yet
(10-minute TTL). PendingLoginClaims is deliberately a distinct Go type
from Claims, and -- caught by this change's own test suite before it
shipped -- needed a JSON field name disjoint from Claims.UserID's
"user_id" too: go-jose's unmarshal is happy to populate a struct from
any token whose claims happen to share a key, so a real session token
would otherwise have parsed successfully as a pending login. Fixed via
"pending_user_id" instead; both directions (session-as-pending,
pending-as-session) now have regression tests.
rbacstore.ListMembershipsWithTenantForUser joins tenant_memberships
with tenants, since a picker needs display names, not just IDs.
loginhandler.resolveIdentity's multiple-membership branch no longer
errors -- finishLogin routes it into startTenantSelection instead,
which issues a pending-login cookie (Path=/auth, so it's never sent on
ordinary requests) and redirects to a new configurable
SelectTenantRedirectURL (defaults to {POST_LOGIN_REDIRECT_URL}/select-
tenant). Two new routes complete the round trip: GET /auth/memberships
lists the pending identity's real tenant options, and POST
/auth/select-tenant re-derives the role for the chosen tenant
server-side (never trusts a client-supplied role, refuses a tenant_id
outside the identity's actual memberships with 403) before issuing the
real session -- responding with JSON {"redirect_url": ...}, not a
redirect, since a POST/fetch caller should control its own navigation.
Verified with the same real-fake-IdP tests the rest of this package
uses (coreos/go-oidc's oidctest, crewjam/saml's samlidp): the full
login -> pending cookie -> GET /auth/memberships -> POST
/auth/select-tenant -> real session round trip for both protocols, plus
negative paths (missing/expired pending cookie, a tenant_id outside
membership, a real session token rejected as a pending login and vice
versa). ErrMultipleMemberships is removed -- it's not an error path
anymore.
Docs updated in lockstep: CLAUDE.md, threat-model.md (including its
summary table), phase-4-runbook.md (new §12), enterprise/README.md
(new "Tenant selection" section, explicit about what's still not built
and why: no session handling in web, no CORS on enterprise-auth).
|
||
|
jcoffey-dev
|
8d7326fc6a |
Make docker-compose.yml enforce the api/enterprise-api binary swap
Closes the local/dev half of a gap named repeatedly across this phase's docs: Helm already made api/enterprise-api mutually exclusive (same enterprise.enabled flag that turns on RBAC/audit/SSO, both rendering to the same Service name/port); docker-compose.yml let both run side by side, with nothing actually pointing at enterprise-api by default. Mechanism: both services now carry a `profiles` entry (single-tenant / enterprise), selected via COMPOSE_PROFILES -- a new checked-in .env sets single-tenant as the zero-config default (unchanged behavior for anyone who doesn't touch it), and `COMPOSE_PROFILES=enterprise docker compose up` swaps to enterprise-api instead. Docker Compose profiles are purely additive (no "profile X excludes service Y" primitive), so true exclusivity comes from both being profile-gated with no shared default profile, not from one excluding the other directly. Mirrors Helm's same-Service-name trick so alerting's API_QUERY_URL and web's VITE_API_BASE_URL need zero conditional logic either way: enterprise-api now maps host port 8080 (was 8083, its own binary default -- overridden via HTTP_LISTEN_ADDR) and carries a `networks.default.aliases: [api]` entry, so whichever binary is actually running answers on the same compose-network hostname and host port. alerting's and web's depends_on for api/enterprise-api are now `required: false` (Compose's supported "optional dependency" shape) -- without it, compose errors on the inactive one rather than just skipping it, since depends_on doesn't otherwise know about profiles. Verified for real in this environment via `docker compose config` (renders and validates the merged YAML without needing a daemon): confirmed api/enterprise-api never both appear in --services output for either profile selection, confirmed enterprise-api's rendered block has port 8080/alias "api"/HTTP_LISTEN_ADDR ":8080" when the enterprise profile is active, and confirmed `docker compose run enterprise-api ...`/`docker compose build enterprise-api` (used by enterprise/README.md's and phase-4-runbook.md's provisioning steps) still work regardless of the active profile -- explicit service references bypass profile filtering, confirmed by the commands reaching a daemon-connection permission error rather than a profile-resolution error. Not verified: an actual `docker compose up` against a real daemon, still unavailable in this environment. Docs updated in lockstep -- CLAUDE.md, threat-model.md (including its summary table), phase-4-runbook.md (new §10a, §8's provisioning commands updated for the new port/profile), enterprise/README.md. |
||
|
jcoffey-dev
|
2e698f5623 |
Close the last tenant-isolation adversarial probe (mid-provisioning tenants)
Phase 4 task 8's verification plan named four adversarial probes; three were closed earlier this phase, the fourth (an evaluator tick, or any other caller, hitting a tenant that exists but hasn't reached the active+credentialed gate yet -- must be refused, not served) was still an explicitly-skipped stub in api/queryapi/tenant_isolation_gap_test.go. Investigating it found the two storage engines needed genuinely different treatment: - ClickHouse (enterprise/internal/chrunner) already had this property structurally, for free: Registry is built once at startup from rbacstore.ListProvisionedDataSources, which already filters to active+credentialed tenants only, so a mid-provisioning tenant is simply absent from the connection map. New test TestRegistryRefusesMidProvisioningTenant proves this without Docker -- an empty DataSource list never dials ClickHouse, so this genuinely runs in this environment, unlike every other test in that file. - Tantivy (search/src/registry.rs's IndexRegistry) was a real, different gap, not just an unverified assumption: it opens-or-creates an index for any syntactically-valid tenant_id on first request, because it's a separate process with no Postgres access and structurally can't know which tenants are actually provisioned. A query against a mid-provisioning tenant would have silently succeeded with zero results from a freshly-created empty index -- "ambient success" indistinguishable from "no matching logs," exactly the failure mode this item was worried about. Fixed the Tantivy gap with a new enterprise/internal/searchclient. TenantChecker interface (backed by a new rbacstore.TenantIsActive, implemented structurally, no new import edge needed), consulted before every gRPC call: Client.Search now refuses a non-active tenant before it ever reaches `search`. Dial's signature gained a required TenantChecker parameter; enterprise-api's main.go passes its existing rbacstore.Store (already satisfies the interface). Verified Docker-free via searchclient's existing real-in-process-gRPC-server test harness (TestSearchRefusesMidProvisioningTenant, plus TestSearchPropagatesTenantCheckerError for the fail-closed-on-error case) -- both genuinely run in this environment, same bar as the rest of the Tantivy isolation work. rbacstore.TenantIsActive itself has two new skip-gated live-Postgres tests (TestTenantIsActive, TestTenantIsActiveNonexistentTenant) -- disclosed as not run against a live database here, same gap as the rest of this phase's Postgres-backed pieces. api/queryapi/tenant_isolation_gap_test.go rewritten from a checklist with one skipped stub to a full accounting of all four now-closed probes. Docs updated in lockstep: CLAUDE.md, threat-model.md, phase-4-isolation-design.md (implementation note added after its original sign-off), phase-4-runbook.md (§9), enterprise/README.md. |
||
|
jcoffey-dev
|
243f4dc2ab |
Enforce per-resource dashboard grants (RBAC matrix's own/granted qualifier)
api/dashboards' handler previously enforced only tenant-baseline role
(RoleEditor+), so any Editor could edit/delete any dashboard in their
tenant -- the matrix's "(own/granted)" qualifier was explicitly named
as unbuilt in this handler's own doc comment. This closes that gap.
New core interface api/dashboards.PermissionStore (nil-safe, same "not
wired == no-op" shape as authz.Authorizer) resolves a per-resource
dashboard_permissions grant. canEditDashboard now requires the
identity be Admin/Owner, the dashboard's creator, or hold a grant of at
least Editor; canManageGrants is deliberately stricter (creator or
Admin/Owner only, never grant-derived access) so a user who can edit a
dashboard only because of a grant can't extend or re-grant that access
to themselves or others. Wired handlers: PUT/DELETE
/dashboards/{id}/permissions/{userId}, GET .../permissions.
Two real bugs found and fixed while wiring this up, before any of it
touched a live database:
- handleCreate/handleImport never stamped created_by from the
authenticated identity, so every dashboard was owned by "anonymous"
regardless of who made it -- the ownership check would have been
meaningless. Also fixed: ImportDashboard trusted the exported JSON's
created_by verbatim, so re-importing someone else's export would
leave the actual importer unable to edit their own copy.
- metadata/migrations/0024_create_dashboard_permissions.sql's CHECK
constraint diverged from /docs/phase-4-rbac-design.md's schema
(allowed role='admin', nullable granted_by). Reconciled via
0033_restrict_dashboard_permissions_role.sql: Admin/Owner already
have tenant-wide access so a resource-level "admin" grant is
meaningless, and every real grant now always has an attributable
granter.
enterprise/internal/rbacstore gets the storage side: raw CRUD
(dashboard_permissions.go) plus DashboardPermissions
(dashboards_adapter.go), an adapter implementing
api/dashboards.PermissionStore -- same pattern as audit.QueryAPILogger
over queryapi.AuditLogger. Wired into enterprise/cmd/enterprise-api
only; plain api/cmd/api passes nil (ownership/Admin checks still work
via the nil-permissions fallback, just without the "granted" bonus).
Verified: the full own/granted/admin/creator matrix, including the
granted-editor-cannot-manage-grants regression, passes against a fake
PermissionStore (api/dashboards/handler_test.go, all existing tests
also still pass unmodified in behavior). Real integration tests exist
in enterprise/internal/rbacstore/rbacstore_test.go (skip-gated on
RBACSTORE_TEST_POSTGRES_ADDR, same convention as every other
Postgres-backed piece this phase) but have not run against a live
database in this environment -- disclosed in threat-model.md,
phase-4-runbook.md, and enterprise/README.md alongside every other
piece carrying the same gap. Also fixed a stale path in
phase-4-runbook.md's dashboards-tenant-scoping section
(./internal/dashboards/... -> ./dashboards/..., stale since that
package moved out of api/internal/ earlier in this phase).
|
||
|
jcoffey-dev
|
08a90a27aa |
Build SAML login (enterprise/internal/loginhandler), mirroring OIDC
Adds GET /auth/saml/login + POST /auth/saml/acs alongside the existing OIDC pair, both converging on the same upsert-user/resolve-tenant/ issue-session path. loginhandler.New now takes an optional *saml.ServiceProvider, RegisterRoutes registers each protocol's routes independently so either, both, or neither can be configured. SAML's replay/unsolicited-response defense (InResponseTo, standing in for OIDC's state) is carried via a SameSite=None sentry_saml_request cookie -- None because the ACS endpoint receives a cross-site POST from the IdP's origin, which SameSite=Lax cookies are never sent on. enterprise-auth's main.go now fetches+parses SAML_IDP_METADATA_URL at startup (samlsp.FetchMetadata) and wires the result through. Verified to the same bar as OIDC: a real fake IdP (crewjam/saml/samlidp, genuine XML signing/verification) drives the full login->ACS->session-cookie round trip and negative paths (bad InResponseTo, missing request cookie, missing email/NameID, no/multiple tenant memberships), all in loginhandler/saml_test.go, no Docker needed. The login-form HTML is bypassed by pre-seeding a saml.Session directly into samlidp's session store and presenting the matching `session` cookie -- an IdP-supported shortcut (confirmed by reading GetSession), the same "skip the UI, keep the crypto real" approach oidctest gave the OIDC tests. Writing that test caught two real bugs in internal/saml.ParseResponse, both fixed here: it never called r.ParseForm() before reading the POSTed SAMLResponse field, so every real ACS POST would have silently decoded an empty response; and its email-attribute matching missed urn:oid:0.9.2342.19200300.100.1.3 (the standard LDAP "mail" OID), which is what an IdP sends by default absent an explicit AttributeConsumingService request for "email" -- exactly what samlidp's own DefaultAssertionMaker does, and plausibly what real IdPs' default SAML app templates do too. Docs (CLAUDE.md, threat-model.md, architecture.md, enterprise/README.md, phase-4-runbook.md, docker-compose.yml's enterprise-auth comment) updated in lockstep: SAML login moves from "protocol mechanics only" to "built, verified with a real fake IdP, not yet tried against a real external IdP or a running enterprise-auth container" -- the same disclosed gap OIDC already carried. |
||
|
jcoffey-dev
|
3037b31b0f |
Phase 4: Helm chart enforces api vs enterprise-api, closing the deployment-topology gap
deploy/helm/sentry/templates/api.yaml and the new enterprise-api.yaml
are mutually exclusive, gated on opposite sides of the same
enterprise.enabled flag -- exactly one renders, both as a Deployment+
Service named {{ .Release.Name }}-api on port 8080, so every consumer
(alerting's API_QUERY_URL, web's build args) needs zero conditional
logic of its own. This is the concrete fix for what the threat model
named as the single largest remaining gap once both storage engines'
isolation mechanisms were built: previously nothing forced or flagged
whether a deployment ran the tenant-isolated binary. Now the same flag
that turns on RBAC/audit/SSO also chooses the query binary.
Verified by parsing (not eyeballing) helm template's rendered output
under both value sets: exactly one sentry-api Deployment/Service either
way, with the right image, and kubeconform -strict clean against the
real Kubernetes 1.31 schema. Not applied to a live cluster (still no
cluster in this environment) -- docker-compose.yml also still runs
plain api unconditionally, so this enforcement is Helm-only for now.
Updated the threat model, architecture doc, CLAUDE.md, and deploy/
READMEs to reflect this and to name what's left: ingest has no tenant
concept for either storage engine (undesigned), and the Tenant CRD
(deploy/operator) and enterprise-api -provision-tenant are still two
separate, unreconciled provisioning mechanisms.
|
||
|
jcoffey-dev
|
ba2276aa1a |
Phase 4: real Tantivy per-tenant isolation (search/src/registry.rs, enterprise/internal/searchclient)
Closes the last named "isolation mechanism" gap: search.proto gains a tenant_id field on SearchRequest; search/src/registry.rs's IndexRegistry resolves it to an on-demand-opened, per-tenant Tantivy index (empty tenant_id keeps today's single default index, so this is purely additive); enterprise/internal/searchclient sets that field from the authenticated request identity in ctx, mirroring chrunner's exact fail-closed "never a parameter" shape. Wired into enterprise-api in place of the shared api/searchclient. Unlike the ClickHouse pieces from the previous two commits, this one is genuinely verified end to end in this environment: Tantivy is an embedded library, not a networked service, so both the Rust index registry (cargo test, cargo clippy --all-targets -- -D warnings, both clean) and the Go client (a real in-process gRPC server) could actually run. registry.rs's tenant_index_is_isolated_from_default_and_other_tenants seeds three real indices with the same term and confirms a tenant-scoped search returns only that tenant's document -- item 3 of the isolation design doc's verification plan, closed for real, not just written. With both ClickHouse and Tantivy isolation now built, the single largest remaining gap is no longer a missing mechanism: it's that nothing forces or flags whether a deployment actually runs enterprise-api instead of plain api, and that ingest itself has no tenant concept for either storage engine (every record still lands in the one shared database/ index no matter what -- undesigned, not just unbuilt). Updated the threat model, architecture doc, CLAUDE.md, and both READMEs accordingly. |
||
|
jcoffey-dev
|
1fab02abd5 |
Phase 4: real OIDC human login (enterprise/internal/loginhandler)
Closes the other major named gap from this phase: until now, there was no way for a human to actually log in -- only /alerting's RoleService credential could be minted. GET /auth/oidc/login and GET /auth/oidc/callback drive the real coreos/go-oidc flow already wired in enterprise/internal/oidc: CSRF state in a short-lived cookie, code exchange, ID token verification, upserting a users row, resolving tenant/role from exactly one tenant_memberships row (refusing outright on zero or more than one, rather than guessing), and issuing a real session cookie. Unlike everything else built this phase, this one is genuinely verified end to end: the tests spin up coreos/go-oidc's own oidctest fake IdP, which signs real RS256 ID tokens, and drive the full login->callback-> session-cookie round trip through actual signature verification -- no live database or Docker needed, so nothing here is asserted without having actually been run in this session. Also fixes a real bug caught while wiring this into enterprise-auth's main.go: assigning a nil *oidc.Provider to the handler's interface field would have produced a non-nil interface wrapping a nil pointer (Go's classic typed-nil trap), silently breaking the "OIDC not configured" no-op path -- New() now takes the concrete pointer type and checks it before ever converting to the interface, with a regression test pinning the fix down. Still missing: SAML's equivalent (ACS endpoint), a tenant-picker UI for multi-membership identities, and any admin UI to actually create a tenant_memberships row (today that's manual SQL, documented in the runbook's new bootstrap walkthrough). |
||
|
jcoffey-dev
|
1d57e697b1 |
Phase 4: real per-tenant ClickHouse isolation via a new enterprise-api binary
Closes the threat model's headline finding for the SQL query path:
enterprise/internal/tenantprovision does real CREATE DATABASE/USER/GRANT
against ClickHouse, and enterprise/internal/chrunner is a per-tenant
connection registry implementing api's SQLRunner interface, resolving
the tenant from the authenticated request identity -- never a
caller-suppliable parameter. Both are wired into a new binary,
enterprise/cmd/enterprise-api, alongside the unchanged single-tenant
api/cmd/api, since AGPL core can never import enterprise/ and Go's own
internal/ package visibility rules meant enterprise/ couldn't implement
core's SQLRunner interface without importing the package that defines
it. That required moving api/internal/{authz,queryapi,dashboards,
querylang/executor,searchclient,httpserver} out of internal/ -- the
minimal set enterprise-api needs to import; querylang's compiler
internals (planner/lexer/parser/ast/ir) and api's own config stay
internal, since nothing outside api needs them directly.
Also finally wires enterprise/internal/audit into queryapi.AuditLogger
(nil since Phase 4 task 4) via a new adapter, and adds live-ClickHouse
integration tests for two of the four adversarial probes named in
docs/phase-4-isolation-design.md's verification plan.
Corrected several overclaims in the docs while writing this up: an
earlier claim that rbacstore's CRUD was "verified against a live
Postgres" was never actually true in this environment (only
internal/audit was, earlier in this phase, before Docker access was
lost) -- threat-model.md, phase-4-runbook.md, CLAUDE.md, and
enterprise/README.md all now distinguish "a real integration test
exists" from "this was confirmed against a live database."
Still not built: Tantivy/free-text tenant isolation
(enterprise/internal/searchclient), and any deployment-topology
mechanism that actually routes traffic to enterprise-api instead of
plain api -- both binaries exist side by side today with nothing
enforcing or flagging which one a deployment runs.
|
||
|
jcoffey-dev
|
3eb0f4c589 |
Phase 4: SSO scaffolding, RBAC enforcement, tenant-scoped dashboards, audit logging, K8s deployment
RBAC (api/internal/authz) is live on /query and /dashboards, backed by a new enterprise/ module (session issuance, audit logging, RBAC storage, OIDC/SAML protocol wiring) that core never imports -- only calls over HTTP. Found and fixed a real cross-tenant vulnerability in dashboards (no tenant_id filtering at all) while writing the threat model doc. Two things are explicitly NOT done, documented rather than hidden: tenant isolation for log data itself (/query still shares one ClickHouse connection and Tantivy index across every tenant -- RBAC controls who can query, not what a query can see), and human SSO login (protocol wiring exists, no HTTP handler calls it yet). See docs/security/threat-model.md and docs/phase-4-runbook.md. Also adds deploy/ (Go Operator + Helm chart, validated offline only -- no cluster was reachable in this environment). |