Merge branch 'main' into work/demo-fleet-and-display-timezone
This commit is contained in:
+1
-1
@@ -7,7 +7,7 @@ search/target/
|
||||
/ingest/ingest
|
||||
/api/api
|
||||
/cli/cairnobsctl
|
||||
/terraform/terraform-provider-sentry
|
||||
/terraform/terraform-provider-cairnobs
|
||||
/alerting/alerting
|
||||
/hack/windows-fixture/windows-fixture
|
||||
/hack/benchmark-fixture/benchmark-fixture
|
||||
|
||||
@@ -64,507 +64,13 @@ described there without flagging it to me first.
|
||||
- Prefer boring, well-understood dependencies over novel ones. This is
|
||||
infrastructure software; operators need to trust it.
|
||||
|
||||
## What "done" looks like for Phase 0 (MVP)
|
||||
|
||||
**Status: shipped.** A single log line, generated on a Linux host by the
|
||||
Rust agent, flows: agent → Redpanda → Go ingest service → ClickHouse, and
|
||||
is queryable via a minimal SQL endpoint and visible in a bare-bones
|
||||
SvelteKit table view. Verified end-to-end on real hardware, not just in
|
||||
CI — see `/docs/phase-0-runbook.md`. No alerting, no multi-tenancy, no
|
||||
dashboards — that discipline held for the whole phase.
|
||||
|
||||
## What "done" looks like for Phase 1
|
||||
|
||||
**Status: shipped.** A Windows Event Log entry and a Linux journald entry
|
||||
are both queryable via SQL (the ClickHouse path) and via free-text search
|
||||
(the Tantivy path), from the same UI, within a few seconds of being
|
||||
generated. Verified end-to-end on the live stack, including the same
|
||||
`record_id` coming back from both query paths for the same record — see
|
||||
`/docs/phase-1-runbook.md`.
|
||||
|
||||
ETW and WEF (Windows Event Forwarding) were *designed* in this phase but
|
||||
not required to be running for "done": ETW ships behind a feature flag
|
||||
most environments won't enable (it needs elevated privileges), and WEF's
|
||||
receiver-side was explicitly deferred rather than built. Only the Event
|
||||
Log source needed to actually be running end-to-end, and did. The
|
||||
Windows-specific agent code itself (`EvtSubscribe`, ETW, service
|
||||
registration) remains unverified on real Windows — no Windows toolchain
|
||||
existed anywhere in the environment this was built in; flagged
|
||||
prominently in `/agent/README.md` and the runbook.
|
||||
|
||||
## What "done" looks like for Phase 2
|
||||
|
||||
A single query bar in the web UI and a single `cairnobsctl query` command
|
||||
can express filter + free-text + stats in one query (e.g. `service=api |
|
||||
where status>=500 | stats count by host | sort -count`, or
|
||||
`message:"connection refused" | stats count by host`), execute correctly
|
||||
against both ClickHouse and Tantivy in one compiled plan, and return in
|
||||
well under a second for a 1M-row fixture dataset (rough benchmark, not a
|
||||
formal SLA — see `/docs/phase-2-runbook.md` for the actual measurement).
|
||||
Raw ClickHouse SQL remains available as an escape hatch, compiling to the
|
||||
same execution plan/IR as the pipe syntax so performance doesn't depend
|
||||
on which syntax a query uses.
|
||||
|
||||
Non-goals for this phase (same "resist scope creep" discipline as every
|
||||
phase so far): no alerting, no dashboards, no multi-tenancy — this phase
|
||||
is the query layer only. The two separate placeholder pages/endpoints
|
||||
from Phase 0/1 (`/query` raw-SQL-only, `/search` free-text-only) are
|
||||
retired, replaced by one `/query` endpoint and one query page.
|
||||
|
||||
See `/docs/query-language-design.md` for the grammar, IR, and
|
||||
ClickHouse/Tantivy routing strategy, and
|
||||
`/docs/query-language-reference.md` for the user-facing syntax reference
|
||||
once built.
|
||||
|
||||
## What "done" looks like for Phase 3
|
||||
|
||||
**Status: shipped.** A user can build a multi-panel dashboard from saved Phase 2 queries (at
|
||||
least a line chart panel and a table panel, working end-to-end against
|
||||
live data), save an alert rule that fires a Slack webhook when a
|
||||
condition is met (threshold comparison, or "absence" — the query returned
|
||||
zero rows in its own time window), and see the delivery attempt logged —
|
||||
all from the web UI, without touching the API directly. See
|
||||
`/docs/phase-3-dashboard-design.md` and `/docs/phase-3-alerting-design.md`
|
||||
for the data models and the alerting evaluator's firing/resolved state
|
||||
machine, and `/docs/phase-3-runbook.md` for the live-stack verification,
|
||||
including a load test of the alert evaluator against ~500 concurrent
|
||||
rules.
|
||||
|
||||
This phase adds PostgreSQL as a new pinned-stack component (see the
|
||||
dashboard design doc for why ClickHouse can't do this job — dashboards
|
||||
and alert state need real row-level locking and transactional
|
||||
read-modify-write, which ClickHouse's MergeTree family doesn't provide),
|
||||
scoped strictly to control-plane config: dashboards, panels, notification
|
||||
targets, alert rules, alert state, delivery log. Log data itself stays on
|
||||
ClickHouse/Tantivy only, unchanged.
|
||||
|
||||
Non-goals for this phase (same discipline as every phase so far):
|
||||
- No multi-tenancy enforcement and no `enterprise/` module work — single
|
||||
tenant/org assumed. Most new tables (`dashboards`, `alert_rules`,
|
||||
`notification_targets`) carry a `tenant_id` column so part of Phase 4's
|
||||
retrofit doesn't require a migration + backfill — but `alert_state` and
|
||||
`delivery_log` do not (an inconsistency found during Phase 4 planning,
|
||||
not caught at the time); Phase 4 adds `tenant_id` to those two and
|
||||
backfills via a join through `alert_rules.id`, and — per
|
||||
`/docs/phase-4-isolation-design.md` — tenant isolation itself turned
|
||||
out to live at the ClickHouse/Tantivy connection layer, not via these
|
||||
columns at all, since Phase 2's raw-SQL escape hatch can never be
|
||||
covered by a row filter regardless of which tables carry one.
|
||||
- No raw-SQL dashboard panels (time-range injection isn't reliable
|
||||
against arbitrary SQL) — pipe-syntax queries only.
|
||||
- No per-group/multi-row threshold alerting (e.g. "alert separately per
|
||||
host") — a threshold rule's query must resolve to a single row.
|
||||
- No debounce on the way down — a firing alert resolves on the first
|
||||
false evaluation, no symmetric "stay firing for N more minutes" hold.
|
||||
- No Kubernetes Operator/Helm deployment work — still docker-compose,
|
||||
`/deploy` remains stubbed.
|
||||
|
||||
## What "done" looks like for Phase 4
|
||||
|
||||
**Status: in progress, not shipped.** RBAC enforcement (`api/authz`), the
|
||||
`alerting`↔`api` service-identity credential, tenant-scoped dashboards,
|
||||
append-only audit logging, and — since the second pass on this phase —
|
||||
real per-tenant ClickHouse provisioning and query routing
|
||||
(`enterprise/internal/tenantprovision`, `enterprise/internal/chrunner`,
|
||||
wired into a new `enterprise/cmd/enterprise-api` binary alongside plain
|
||||
`api/cmd/api`) are all built and tested — real integration tests exist
|
||||
for the ClickHouse pieces, but this environment lost Docker/database
|
||||
access partway through the phase, so only the audit-logging guarantees
|
||||
were actually confirmed against a live database; the rest is untested
|
||||
beyond "compiles, and skips cleanly when no live database is
|
||||
configured" (see `/docs/phase-4-runbook.md`'s verification-status
|
||||
section). Human SSO login is now built for both protocols
|
||||
(`enterprise/internal/loginhandler`: `GET /auth/oidc/login` +
|
||||
`GET /auth/oidc/callback`, and `GET /auth/saml/login` +
|
||||
`POST /auth/saml/acs` via `enterprise/internal/saml`'s `crewjam/saml`
|
||||
wiring, both issuing a real session cookie after resolving tenant/role
|
||||
from `tenant_memberships`) — genuinely verified, unlike the ClickHouse
|
||||
pieces, via a real fake IdP for each protocol that performs actual
|
||||
cryptographic signing and verification (`coreos/go-oidc`'s `oidctest`
|
||||
for OIDC, `crewjam/saml/samlidp` for SAML — `loginhandler_test.go` and
|
||||
`saml_test.go`, all passing, including the full login round trip and
|
||||
negative paths for both), though never tried against a real external IdP
|
||||
or through a running `enterprise-auth` container. Writing the SAML test
|
||||
caught and fixed two real bugs in `internal/saml.ParseResponse`: a
|
||||
missing `r.ParseForm()` call that would have silently broken every real
|
||||
ACS POST, and email-attribute matching that missed the standard LDAP
|
||||
"mail" OID IdPs send by default. Tantivy per-tenant index routing is now
|
||||
built too
|
||||
(`search/src/registry.rs` + `enterprise/internal/searchclient`) —
|
||||
**genuinely verified**, like the OIDC login flow: Tantivy is an embedded
|
||||
library, not a networked service, so the isolation probe (three tenants,
|
||||
same search term, scoped search returns only that tenant's document)
|
||||
actually ran in this environment, no Docker needed. That same
|
||||
Docker-free advantage is what caught a real bug while closing the last
|
||||
of Phase 4 task 8's four adversarial probes (a mid-provisioning tenant
|
||||
must be refused, not served): `search/src/registry.rs`'s `IndexRegistry`
|
||||
opened-or-created an index for any syntactically-valid `tenant_id`,
|
||||
meaning a query against a tenant that exists in `rbacstore` but isn't
|
||||
active yet would have silently returned zero results from a
|
||||
freshly-created empty index instead of being refused --
|
||||
`chrunner`'s ClickHouse routing had the equivalent guarantee for free
|
||||
(a mid-provisioning tenant simply isn't in its startup-built connection
|
||||
map) but Tantivy, a separate process with no Postgres access, had no
|
||||
way to know. Fixed with a new `enterprise/internal/searchclient.
|
||||
TenantChecker` (backed by `rbacstore.TenantIsActive`); both halves of
|
||||
the fix verified Docker-free (`chrunner_test.go`'s and
|
||||
`searchclient_test.go`'s `TestSearchRefusesMidProvisioningTenant`-shaped
|
||||
tests) — see `api/queryapi/tenant_isolation_gap_test.go` for the full
|
||||
accounting of all four probes, now all closed. The deployment-
|
||||
topology gap that briefly was the largest one is now closed for both
|
||||
Helm and docker-compose: `deploy/helm/cairnobs/templates/api.yaml`/
|
||||
`enterprise-api.yaml` are mutually exclusive on the same
|
||||
`enterprise.enabled` flag that turns on RBAC/audit/SSO, rendering to the
|
||||
same Service name/port either way — a Helm-deployed cluster can't
|
||||
accidentally run the wrong one. `docker-compose.yml`'s `api`/
|
||||
`enterprise-api` services are now the same mutually-exclusive choice,
|
||||
gated behind `COMPOSE_PROFILES` (`.env` checks in `single-tenant` as the
|
||||
zero-config default) and sharing a host port/network-alias trick so
|
||||
`alerting`/`web` need no conditional logic either way — verified via
|
||||
`docker compose config` (renders/validates without a daemon, confirms
|
||||
the two never both appear for one profile selection), not an actual
|
||||
`docker compose up` in this environment. Per-resource dashboard grants
|
||||
(the RBAC matrix's "(own/granted)"
|
||||
qualifier) are now enforced too: `api/dashboards.PermissionStore` (core
|
||||
interface) implemented by `enterprise/internal/rbacstore.
|
||||
DashboardPermissions`, wired in only by `enterprise-api` — an Editor can
|
||||
now only edit/delete a dashboard they created or were granted access to,
|
||||
not every dashboard in their tenant; managing grants themselves is
|
||||
stricter still (creator/Admin/Owner only, closing a self-escalation
|
||||
path). Verified against a fake store (`api/dashboards/handler_test.go`);
|
||||
real integration tests exist but haven't run against a live Postgres,
|
||||
same disclosed gap as the rest of this phase's Postgres-backed pieces.
|
||||
`cairnobsctl dashboards permissions list|grant|revoke` is now the CLI
|
||||
surface for this — `PUT`/`DELETE /dashboards/{id}/permissions/{userId}`
|
||||
previously had no caller but Go tests and curl.
|
||||
`deploy/operator`'s `Tenant` CRD and `enterprise-api -provision-tenant`
|
||||
are now unified too, deliberately lightweight rather than making the
|
||||
K8s controller a second real actor: `-provision-tenant` stays the sole
|
||||
caller of ClickHouse/`rbacstore`, and (via a new
|
||||
`enterprise/internal/tenantcrd`, gated on `TENANT_CRD_NAMESPACE`) syncs
|
||||
its real result into the CRD — a real credential Secret, not the
|
||||
previous placeholder that authenticated against nothing, and status
|
||||
fields the reconciler derives `Phase`/`Ready` from instead of
|
||||
independently guessing "Active" the moment a Tenant object exists. The
|
||||
tenant-picker is now fully built, backend and frontend: an identity with
|
||||
more than one `tenant_memberships` row gets a real `GET
|
||||
/auth/memberships`/`POST /auth/select-tenant` round trip (a short-lived
|
||||
pending-login token, distinct from a real session by both Go type and
|
||||
JWT claim name — a real token-confusion bug this design's own tests
|
||||
caught before it shipped) instead of the flat refusal Phase 4 shipped
|
||||
with earlier, and `web/src/routes/select-tenant` is the page that
|
||||
actually calls it — see the Phase 4 exit-criteria paragraph below for
|
||||
what changed to make that verifiable in this environment. Ingest
|
||||
tenant-awareness — the gap this section used to
|
||||
call "undesigned" — now has a real, if intentionally partial, design:
|
||||
`ingest` (AGPL core) gained an optional `TenantResolver`
|
||||
(`ingest/internal/grpcserver`), a per-tenant bearer credential an agent
|
||||
presents (minted via `enterprise-auth
|
||||
-create-ingest-credential-tenant=<id>`, validated over the network via a
|
||||
new `POST /internal/authorize-ingest` endpoint — never an `enterprise/`
|
||||
import, same boundary shape as `api/authz.Authorizer`), and the
|
||||
resolved tenant ID is attached to every record as a `tenant_id` Kafka
|
||||
message header before it's produced. **ClickHouse write-routing is now
|
||||
built too**: `enterprise/cmd/enterprise-ingest` (another "second binary,"
|
||||
mirroring `enterprise-api`) reuses `ingest/consumer`'s own flush loop
|
||||
with `enterprise/internal/chwriter.Registry` swapped in as the writer —
|
||||
one dedicated ClickHouse connection per tenant, routing each batch's
|
||||
records by their `tenant_id` tag, fail-closed on an untagged or
|
||||
unprovisioned tenant. Building it found and fixed a real bug:
|
||||
`tenantprovision.ProvisionClickHouse` originally granted a tenant's
|
||||
ClickHouse user `SELECT` only, which would have made every real
|
||||
per-tenant write fail with a permission error — fixed by granting
|
||||
`SELECT, INSERT` (one credential, both directions; no cross-tenant
|
||||
boundary is crossed by also allowing INSERT within a tenant's own
|
||||
database). **Tantivy write-routing is now built too** —
|
||||
`search/src/consumer.rs` (a completely independent Redpanda consumer,
|
||||
not called through `ingest` or `enterprise-ingest` at all: a different
|
||||
codebase and process) now resolves each record's `tenant_id` header
|
||||
through the *same* `IndexRegistry` the read side already used, and
|
||||
routes the write there instead of always into the default index. Unlike
|
||||
the ClickHouse side, this needed no "second binary": `IndexRegistry`
|
||||
already lives in this AGPL-core binary (Tantivy has no grant system to
|
||||
gate a separately-credentialed binary behind, so there was never an
|
||||
import-boundary reason to split it out -- true regardless of licensing,
|
||||
though at the time of writing `enterprise/` was still commercially
|
||||
licensed; both sides are AGPLv3 as of Phase 6), so read and write share one
|
||||
registry directly. The periodic Tantivy commit now commits every tenant
|
||||
index that's seen a write, not just the default one
|
||||
(`IndexRegistry::commit_all`). **The active-tenant gap this same change
|
||||
originally disclosed is now closed too**: `search/src/tenants.rs`'s
|
||||
`ActiveTenantTracker` polls a new `GET /internal/active-tenants`
|
||||
endpoint on `enterprise-auth` (`search` has no Postgres access, unlike
|
||||
`chwriter.Registry`'s direct `rbacstore` query or the read side's
|
||||
`searchclient.TenantChecker`, so this needed a network call — the same
|
||||
"network boundary, not import boundary" shape `ingest`'s
|
||||
`TenantResolver` already uses against the same service, authenticated
|
||||
with a RoleService credential the same way `alerting` authenticates to
|
||||
`api`) and `consumer.rs` refuses any tagged record whose tenant isn't in
|
||||
the polled allowlist. Off unless both `ENTERPRISE_AUTH_URL` and
|
||||
`ENTERPRISE_AUTH_SERVICE_TOKEN` are set (same "off unless configured"
|
||||
default as everything else optional in this codebase); when they are,
|
||||
startup blocks on the first fetch succeeding and later refresh failures
|
||||
keep serving the last-known-good set rather than clearing it. Verified
|
||||
with real HTTP round trips against a hand-rolled TCP test server in this
|
||||
environment, no live enterprise-auth needed. **This closing move exposed
|
||||
the ClickHouse side's own gap by comparison** — `chwriter.Registry`'s
|
||||
writer map was still a startup-only snapshot with no refresh at all, a
|
||||
real asymmetry once Tantivy's tracker refreshed every minute and
|
||||
ClickHouse's didn't — so `Registry.StartRefreshing` (new) closes that
|
||||
too: same one-minute interval, same last-known-good posture on a failed
|
||||
refresh, opening connections for newly-active tenants and closing ones
|
||||
no longer active. Both engines now share the same active-tenant
|
||||
staleness bound instead of one being materially staler than the other.
|
||||
**The tenant-picker page is now built too**:
|
||||
`web/src/routes/select-tenant` calls `GET /auth/memberships`/
|
||||
`POST /auth/select-tenant` via `fetch(..., {credentials: 'include'})`
|
||||
(new `$lib/api.ts` functions), which needed a second CORS posture
|
||||
alongside the wildcard-friendly one `enterprise-api` already had —
|
||||
`api/httpserver.WithCredentialedCORS`, set to a literal origin via a new
|
||||
`CORS_ALLOWED_ORIGIN` on `enterprise-auth` — since browsers refuse to
|
||||
honor a wildcard `Access-Control-Allow-Origin` on a credentialed
|
||||
request. **Genuinely verified in a real browser in this environment**:
|
||||
a throwaway Node server standing in for `enterprise-auth`'s exact wire
|
||||
contract (including its plain-text `http.Error` bodies, not JSON) on a
|
||||
different origin than `web`'s dev server, driven through the full
|
||||
cross-origin pending-login-cookie round trip, a real click choosing a
|
||||
tenant, and the post-selection redirect — plus the missing/expired-
|
||||
pending-login error path — with no Docker or live Postgres/IdP needed,
|
||||
since the point was exercising `web`'s own fetch/CORS/cookie wiring, not
|
||||
`enterprise-auth`'s internals (already covered by that package's own
|
||||
tests). See `/web/README.md`'s "Tenant picker" section for the exact
|
||||
setup. What's left in this phase now is entirely the caveats already
|
||||
disclosed above, not an unbuilt feature: the ClickHouse/Postgres-backed
|
||||
pieces have never run against a real database in this environment, and
|
||||
nothing here has been tried against a real external IdP or a real
|
||||
running multi-container deployment. Full accounting:
|
||||
`/docs/security/threat-model.md`; step-by-step verification procedure
|
||||
(not yet run against a live cluster in this environment):
|
||||
`/docs/phase-4-runbook.md`. The rest of this section describes the exit
|
||||
bar this phase is aiming at, not a completed state.
|
||||
|
||||
Two tenants can be provisioned with SSO (OIDC or SAML), each with their
|
||||
own users, roles, dashboards, and alert rules, fully isolated at the
|
||||
ClickHouse/Tantivy connection layer — not by a row filter — with
|
||||
adversarial integration tests proving no cross-tenant data leakage,
|
||||
including via the raw-SQL escape hatch and ClickHouse's own `system.*`
|
||||
tables. A tenant admin can see a query audit trail for their tenant,
|
||||
backed by append-only storage a compromised application credential
|
||||
cannot alter (enforced by database grants, not just convention) and
|
||||
periodically anchored outside the database so tampering is detectable
|
||||
even against a privileged attacker. See `/docs/phase-4-isolation-design.md`
|
||||
for the tenant isolation model and why it lives at the connection layer,
|
||||
`/docs/phase-4-rbac-design.md` for the role/permission model, and
|
||||
`/docs/security/threat-model.md` for the auth flows and audit-log
|
||||
integrity guarantees, written for a prospective enterprise customer's
|
||||
security team.
|
||||
|
||||
The tenant-isolation, provisioning, SSO, and RBAC-enforcement mechanisms
|
||||
live entirely in `enterprise/` (commercial license at the time this
|
||||
section was written; relicensed to AGPLv3 in Phase 6, see that phase's
|
||||
section below), confirmed explicitly rather than assumed: core
|
||||
(`/api`, `/alerting`, `/web`) stays genuinely single-tenant, with no
|
||||
multi-tenant mechanism present at all — `enterprise/` supplies
|
||||
tenant-scoped implementations of core's
|
||||
already-shipped `querylang/executor.SQLRunner`/`SearchClient` interfaces
|
||||
rather than core growing tenant awareness. Query-compiler-level "compile
|
||||
time" enforcement, as originally proposed, turned out not to be
|
||||
achievable in any module once Phase 2's opaque raw-SQL passthrough is
|
||||
accounted for — the honest, implemented guarantee is that every code
|
||||
path (compiled query or raw SQL) is forced through a tenant-scoped
|
||||
database connection/index that the database's own access control
|
||||
enforces, not a compiler-injected filter.
|
||||
|
||||
Non-goals for this phase (same discipline as every phase so far):
|
||||
- No deny-override permissions — per-resource grants (e.g. a specific
|
||||
user getting edit access to one dashboard) are additive only; a full
|
||||
allow/deny ACL system is future work.
|
||||
- No data retention/deletion policy design for tenant deprovisioning —
|
||||
the provisioning state machine includes a `deprovisioning` state, but
|
||||
what actually happens to a deprovisioned tenant's data is a separate,
|
||||
not-yet-designed compliance question.
|
||||
- No general multi-cluster orchestration in `/deploy` — scoped to
|
||||
proving the per-tenant ClickHouse/Tantivy isolation model works, not a
|
||||
fully general multi-cluster system.
|
||||
- No protection against a privileged ClickHouse/Postgres administrator —
|
||||
the isolation and audit-log guarantees in this phase are structural
|
||||
defenses against application-layer bugs and injection, not against
|
||||
someone with database superuser access; that's an operational control,
|
||||
out of scope here and named explicitly, not silently assumed away.
|
||||
|
||||
## What "done" looks like for Phase 5
|
||||
|
||||
**Status: shipped.** A ground-up frontend redesign — visual direction,
|
||||
a real design system, navigation/IA, charting, dashboard panels,
|
||||
query/search, and alerting UI — plus an accessibility pass, all verified
|
||||
against a live docker-compose stack with real seeded data, not just
|
||||
`npm run check`/`npm run build` passing. See `/docs/design-system.md`
|
||||
for the token system and component library, and
|
||||
`/docs/phase-5-runbook.md` for the full verification log, including five
|
||||
real bugs this phase's live-verification discipline caught that a
|
||||
type-checked, successfully-building frontend would not have surfaced on
|
||||
its own.
|
||||
|
||||
The visual direction ("Signal": near-neutral grayscale UI, color rationed
|
||||
to the four-tier severity system plus a single interactive accent, real
|
||||
dark-mode-as-default) was picked from three proposed directions before
|
||||
any token or component work started, per an explicit stop point in this
|
||||
phase's brief. The charting library (ECharts, over Observable Plot and
|
||||
raw D3 — see the design-system doc for the reasoning and the verified
|
||||
bundle-size/perf numbers) was likewise confirmed before being wired into
|
||||
every panel type, the second explicit stop point.
|
||||
|
||||
Two of the five bugs this phase's verification caught were backend bugs
|
||||
with no connection to the frontend redesign itself, only surfaced
|
||||
because getting real dashboard/alert data to verify the new UI against
|
||||
required actually exercising write paths nothing had exercised since
|
||||
Phase 4's `tenant_id` migrations landed:
|
||||
|
||||
- `alerting`'s `rulestore.Create`/`ApplyTransition` never populated the
|
||||
`tenant_id` column Phase 4 added to `alert_state`/`delivery_log` (with
|
||||
a `NOT NULL` constraint) — every alert rule created against a
|
||||
Phase-4-or-later database silently failed. Existing rows all had a
|
||||
value from Phase 4's backfill migration, which is exactly why this
|
||||
went uncaught: Phase 4's own verification never created a *new* rule
|
||||
post-migration, and its runbook already discloses that Docker access
|
||||
was lost partway through that phase.
|
||||
- `dashboard_panels`'s `viz_type` CHECK constraint was never updated
|
||||
alongside `heatmap`'s addition to the Go/TS validators — a three-place
|
||||
change (Go validator, TS union, DB constraint), not two.
|
||||
|
||||
Both are fixed (`alerting/internal/rulestore/store.go`,
|
||||
`metadata/migrations/0035_add_heatmap_viz_type.sql`) and confirmed
|
||||
against a live stack: rule creation → evaluation → firing → a real
|
||||
(failed, to an intentionally fake webhook) delivery attempt, and a
|
||||
heatmap panel created, persisted, and rendered end to end. See the
|
||||
runbook for the other three findings (one more real product bug — a
|
||||
`findIndex`/nullish-coalescing bug in the chart-pivoting logic that made
|
||||
every `single_stat` panel render `0` — and two real accessibility
|
||||
findings caught by axe-core against live-rendered pages with real data,
|
||||
not fixture data or empty states).
|
||||
|
||||
Non-goals for this phase (same discipline as every phase so far):
|
||||
- No query-language or data-model changes beyond the one narrowly
|
||||
justified exception: `heatmap` as a `VizType`, needed to feed a new
|
||||
visualization, not a new query capability.
|
||||
- No changes to tenant isolation, RBAC, SSO, or audit logging — Phase 4's
|
||||
surface area is untouched; this phase is presentation-layer only.
|
||||
- No mobile-phone-width layout — responsive verification stops at
|
||||
tablet-landscape width, per the brief's explicit scope ("laptop/
|
||||
tablet-landscape," not phone-width).
|
||||
- No fuzzy search in the command palette, no data-grid virtualization for
|
||||
very large result sets, no chart types beyond the five built
|
||||
(time-series, bar, single-stat, heatmap, top-N) — real, disclosed
|
||||
future work, not oversights.
|
||||
|
||||
## What "done" looks like for Phase 6
|
||||
|
||||
**Status: shipped.** A full license-compliance audit and remediation
|
||||
pass across the entire monorepo. Full report:
|
||||
`/docs/compliance/license-audit-report.md`;
|
||||
machine-readable inventory: `/docs/compliance/license-inventory.{csv,json}`
|
||||
(776 rows, 502 unique dependencies across Rust/Go/npm plus Docker base
|
||||
images and vendored assets); ongoing policy:
|
||||
`/docs/compliance/license-policy.md`, now enforced in CI
|
||||
(`.github/workflows/license-compliance.yml` — this repo's first CI
|
||||
workflow file).
|
||||
|
||||
Every dependency was inventoried and classified; 774 of 776 rows
|
||||
resolved cleanly to AGPLv3-compatible with real citations, not guesses
|
||||
(see the audit report for the reasoning on each non-obvious case —
|
||||
dual-licensed crates, MPL-2.0, a license-detector false negative on
|
||||
`segmentio/asm`); `enterprise/` relicensed to AGPLv3 throughout the
|
||||
repo, with the deliberate business-model consequence recorded (anyone,
|
||||
including competitors, can now legally self-host or fork those
|
||||
features); confirmed no license-gating/entitlement logic ever existed to
|
||||
remove; a root `LICENSE` file added (there wasn't one before this
|
||||
phase); CI enforcement wired up and every command verified locally.
|
||||
|
||||
**The one real flag — Redpanda's BSL 1.1 license (confirmed against
|
||||
primary sources for the pinned v24.2.7, not assumed to still be
|
||||
Apache-2.0) — is resolved, not outstanding**: decision recorded
|
||||
2026-08-16, accept as-is. Cairn OBS's own use (internal Kafka-protocol
|
||||
transport, no resale of broker access) sits within BSL's Additional Use
|
||||
Grant; the harder question — whether a third party self-hosting Cairn OBS
|
||||
"as a service" using the bundled `docker-compose.yml` could trip BSL's
|
||||
anti-resale restriction on Redpanda specifically — was judged unlikely
|
||||
given Cairn OBS's ingest pipeline creates fixed internal topics, not
|
||||
per-end-user ones, and was accepted as a disclosed, known risk rather
|
||||
than triggering a swap to Apache Kafka (real resource-footprint cost) or
|
||||
dropping the bundled broker image (rougher local dev experience). See
|
||||
the audit report's Redpanda section for the full reasoning, the other
|
||||
two options that were considered and not chosen, and the condition under
|
||||
which this should be revisited (an official hosted/managed Cairn OBS
|
||||
offering, which would make the third-party-SaaS scenario Cairn OBS's own
|
||||
rather than a hypothetical one).
|
||||
|
||||
Non-goals for this phase: replacing permissively-licensed dependencies
|
||||
with copyleft ones (explicitly out of scope per the phase's own brief);
|
||||
per-file SPDX license headers across the monorepo's several thousand
|
||||
source files (a deliberate choice — see the audit report's "Own license
|
||||
declarations" section for why root `LICENSE` + manifest fields was
|
||||
judged sufficient); redesigning `favicon.svg` (flagged as a leftover
|
||||
SvelteKit scaffold asset, not a license blocker — a design task, not a
|
||||
compliance one).
|
||||
|
||||
## What "done" looks like for Phase 7
|
||||
|
||||
**Status: shipped.** AI-assisted query authoring: from the same query
|
||||
bar, a user can (a) get AI-assisted autocomplete, explanations, and fix
|
||||
suggestions while writing pipe-syntax or SQL queries by hand, and (b)
|
||||
type a plain-English question and get a generated structured query with
|
||||
explanation, editable before running — both paths executing through the
|
||||
unchanged Phase 2 compiler (`api/internal/querylang/planner`,
|
||||
`api/querylang/executor`) with Phase 4 tenant scoping, cost guardrails,
|
||||
and audit logging applying identically to both. No cloud dependency
|
||||
required for the default deployment (self-hosted via Ollama,
|
||||
`qwen2.5-coder:7b`/`1.5b`, both Apache-2.0 — chosen specifically to keep
|
||||
Phase 6's license-purity work intact; a pluggable, opt-in, off-by-default
|
||||
cloud adapter exists for deployments that want one).
|
||||
|
||||
Non-negotiable design principle held throughout, confirmed by inspection
|
||||
of the actual code paths rather than merely asserted: every AI-assisted
|
||||
or AI-translated query compiles down to and executes through the same
|
||||
Phase 2 IR and compiler, and passes through the same Phase 4
|
||||
tenant-scoping enforcement and cost guardrails as a hand-written query —
|
||||
no parallel execution path, no scoping shortcut, for either track. No AI
|
||||
code path anywhere constructs a `SQLRunner`, calls `executor.Execute`,
|
||||
or bypasses `authz.RequireRoleOrService`.
|
||||
|
||||
Every AI-assisted suggestion a user explicitly accepts or dismisses
|
||||
(translate/fix/optimize — deliberately not ghost-text completion or
|
||||
explain, see the design doc for why) is logged into the same
|
||||
append-only, hash-chained `audit_log` table Phase 4 built, via a new
|
||||
`event_type='ai_interaction'` rather than a new table
|
||||
(`metadata/migrations/0036`) — genuinely verified against a live
|
||||
Postgres in this environment, not just unit-tested against a fake, the
|
||||
same rigor Phase 4's own audit-logging guarantees were held to.
|
||||
|
||||
Two real product bugs were found and fixed via this phase's live
|
||||
browser verification — neither would have been caught by
|
||||
`svelte-check`/`npm run build` — and a real logic bug in the cost/safety
|
||||
guard itself (an unbounded-aggregation-vs-raw-row distinction) was found
|
||||
and fixed by the test suite written for it. Full accounting of all
|
||||
three: `/docs/phase-7-ai-design.md`. Integration tests
|
||||
(`api/ai/aiapi/integration_test.go`) wire a real `ollama.Client` through
|
||||
a real `router`/`Handler` against a mock server matching Ollama's actual
|
||||
wire contract (`hack/mock-ollama`, new — also used for this phase's live
|
||||
verification), proving the plumbing without needing real model weights;
|
||||
testing actual model *quality* is deliberately kept out of CI as a
|
||||
disclosed, periodic human-run checklist item instead — see the design
|
||||
doc's CI-testability section for the reasoning.
|
||||
|
||||
Explicit non-goals for this phase (scoped out, not deferred by oversight):
|
||||
result summarization, incident narrative generation, and proactive/
|
||||
unprompted AI suggestions — this phase is query authoring assistance
|
||||
only (structured and natural-language), not analysis or automation. Real
|
||||
future-phase candidates, not silently dropped.
|
||||
|
||||
See `/docs/phase-7-ai-design.md` for the model-provider architecture,
|
||||
shared foundation (schema grounding, cost/safety guard), both tracks'
|
||||
build-and-verification record, and the audit-logging/CI-testability
|
||||
design; `/docs/phase-7-runbook.md` for the step-by-step live-stack
|
||||
verification procedure.
|
||||
## Project status
|
||||
Phase-by-phase scope, what "done" meant for each, and the verification
|
||||
record — including what is *not* yet shipped and what remains unverified —
|
||||
live in [`/docs/status.md`](docs/status.md). Read it before assuming a
|
||||
capability works end-to-end; several are built but unconfirmed against a
|
||||
live stack, and the per-phase runbooks in `/docs` record exactly how each
|
||||
was checked.
|
||||
|
||||
## When in doubt
|
||||
Ask before: changing the pinned stack, adding a new external dependency
|
||||
|
||||
@@ -0,0 +1,157 @@
|
||||
# Cairn OBS
|
||||
|
||||
Open-core, Kubernetes-native log aggregation and observability. Built to match
|
||||
Splunk on capability while winning on cost-per-GB, with honest multi-tenant
|
||||
RBAC and a modern language stack.
|
||||
|
||||
Licensed **AGPLv3 in its entirety** — including `enterprise/`. See
|
||||
[Licensing](#licensing).
|
||||
|
||||
## What it does
|
||||
|
||||
Logs flow from a statically-linked Rust edge agent through Redpanda into a Go
|
||||
ingest pipeline, landing in ClickHouse for analytics and Tantivy for full-text
|
||||
search. One query language spans both stores, compiling to a single execution
|
||||
plan:
|
||||
|
||||
```
|
||||
service=api | where status>=500 | stats count by host | sort -count
|
||||
message:"connection refused" | stats count by host
|
||||
```
|
||||
|
||||
Raw ClickHouse SQL stays available as an escape hatch and compiles to the same
|
||||
IR, so performance doesn't depend on which syntax you write.
|
||||
|
||||
On top of that sit dashboards, an alerting evaluator with threshold and
|
||||
absence rules, a CLI (`cairnobsctl`), a Terraform provider, and AI-assisted
|
||||
query authoring that runs against a self-hosted Ollama model by default — no
|
||||
cloud dependency.
|
||||
|
||||
## Architecture
|
||||
|
||||
| Component | Stack |
|
||||
|---|---|
|
||||
| Edge agent | Rust, musl static target |
|
||||
| Transport | Redpanda (Kafka API) |
|
||||
| Ingest / parse | Go |
|
||||
| Analytical store | ClickHouse |
|
||||
| Full-text index | Tantivy (Rust) |
|
||||
| Control plane / API | Go, gRPC + REST gateway |
|
||||
| Control-plane metadata | PostgreSQL |
|
||||
| Frontend | SvelteKit + TypeScript |
|
||||
| Deployment | Kubernetes Operator (kubebuilder), Helm, docker-compose |
|
||||
|
||||
PostgreSQL is scoped strictly to control-plane config — dashboards, panels,
|
||||
alert rules and state, notification targets, delivery log — because those need
|
||||
row-level locking and transactional read-modify-write that ClickHouse's
|
||||
MergeTree family doesn't provide. Log data itself never touches it.
|
||||
|
||||
Full spec: [`docs/architecture.md`](docs/architecture.md). Read it before
|
||||
changing any component; the storage/query split in particular is deliberate.
|
||||
|
||||
## Repository layout
|
||||
|
||||
Monorepo, one top-level directory per component, each with its own `README.md`,
|
||||
unit tests, and Dockerfile.
|
||||
|
||||
```
|
||||
agent/ Rust edge agent (Linux + Windows)
|
||||
transport/ Redpanda topics and schemas
|
||||
ingest/ Go ingest and parse pipeline
|
||||
storage/ ClickHouse schema and migrations
|
||||
search/ Tantivy full-text index service
|
||||
api/ Control plane, query compiler, RBAC
|
||||
alerting/ Rule evaluator and notification delivery
|
||||
metadata/ PostgreSQL schema and migrations
|
||||
web/ SvelteKit frontend
|
||||
cli/ cairnobsctl
|
||||
proto/ gRPC service definitions
|
||||
terraform/ Terraform provider
|
||||
enterprise/ SSO, multi-tenancy, per-tenant provisioning
|
||||
deploy/ Helm charts and Kubernetes operator
|
||||
docs/ Architecture, design docs, per-phase runbooks
|
||||
hack/ Development scripts
|
||||
```
|
||||
|
||||
`enterprise/` stays a separate module that core never imports from. Since the
|
||||
Phase 6 relicensing that boundary is architectural rather than legal — it keeps
|
||||
core buildable and deployable standalone, and keeps tenant resolution
|
||||
server-side.
|
||||
|
||||
## Running locally
|
||||
|
||||
```sh
|
||||
docker compose up
|
||||
```
|
||||
|
||||
The web UI comes up on <http://localhost:3000> and the API on `:8080`;
|
||||
alerting is on `:8081` and enterprise auth on `:8082`. The agent connects to
|
||||
ingest over mTLS gRPC on `:4317`. `search` is reachable only on the compose
|
||||
network — it publishes no host port.
|
||||
|
||||
`COMPOSE_PROFILES` in `.env` selects the query-serving binary — `single-tenant`
|
||||
(default) or `enterprise` for the multi-tenant path. They're mutually
|
||||
exclusive, the same choice Helm's `enterprise.enabled` flag makes for a real
|
||||
cluster. Override per invocation:
|
||||
|
||||
```sh
|
||||
COMPOSE_PROFILES=enterprise docker compose up
|
||||
```
|
||||
|
||||
Kubernetes deployment via the Helm chart in [`deploy/`](deploy/README.md).
|
||||
|
||||
## Status
|
||||
|
||||
Built in phases; each has a runbook in `docs/` recording how it was verified.
|
||||
Full per-phase detail is in [`docs/status.md`](docs/status.md).
|
||||
|
||||
| Phase | Scope | Status |
|
||||
|---|---|---|
|
||||
| 0 | Agent → Redpanda → ingest → ClickHouse, queryable end-to-end | Shipped |
|
||||
| 1 | Windows Event Log + journald, SQL and full-text paths | Shipped |
|
||||
| 2 | Unified query language across both stores | Shipped |
|
||||
| 3 | Dashboards, alert rules, notification delivery | Shipped |
|
||||
| 4 | RBAC, tenant isolation, audit logging, per-tenant ClickHouse | **In progress** |
|
||||
| 5 | Frontend redesign and design system | Shipped |
|
||||
| 6 | License compliance audit and remediation | Shipped |
|
||||
| 7 | AI-assisted query authoring | Shipped |
|
||||
|
||||
**Phase 4 is not shipped.** The code is built and tested, but the environment
|
||||
lost Docker and database access partway through, so only the audit-logging
|
||||
guarantees were confirmed against a live database. The rest compiles and skips
|
||||
cleanly when no live database is configured, but is otherwise unverified — see
|
||||
the verification-status section of
|
||||
[`docs/phase-4-runbook.md`](docs/phase-4-runbook.md).
|
||||
|
||||
The Windows agent code (`EvtSubscribe`, ETW, service registration) has never
|
||||
run on real Windows — no Windows toolchain existed in the build environment.
|
||||
ETW additionally sits behind a feature flag, since it needs elevated
|
||||
privileges. Details in [`agent/README.md`](agent/README.md).
|
||||
|
||||
Terraform provider coverage is partial by necessity: dashboards and panels have
|
||||
full CRUD, while alert rules and notification targets are create/destroy only,
|
||||
because `alerting` exposes no `PUT /rules/{id}` or `PUT /targets/{id}` to
|
||||
update against. Tenant and RBAC resources are disclosed future work —
|
||||
[`terraform/README.md`](terraform/README.md) accounts for exactly what exists.
|
||||
|
||||
## Contributing
|
||||
|
||||
- Conventional commits. Every change should be a logically complete,
|
||||
independently revertible unit.
|
||||
- Rust: `cargo clippy --all-targets -- -D warnings` must pass.
|
||||
- Go: `go vet` and `golangci-lint`, no globals for shared state.
|
||||
- Every UI action must map to a documented REST/gRPC call — no UI-only logic.
|
||||
The CLI and Terraform provider are first-class, not afterthoughts.
|
||||
- Prefer boring, well-understood dependencies. This is infrastructure software;
|
||||
operators need to trust it.
|
||||
|
||||
## Licensing
|
||||
|
||||
AGPLv3, no exceptions — see [`LICENSE`](LICENSE). `enterprise/` was under a
|
||||
commercial-license stub from Phase 4 through Phase 5; Phase 6 relicensed it to
|
||||
match core. The full record and its business-model consequences are in
|
||||
[`docs/compliance/license-audit-report.md`](docs/compliance/license-audit-report.md).
|
||||
|
||||
The default AI deployment uses `qwen2.5-coder` (Apache-2.0) via Ollama,
|
||||
chosen specifically to keep that license purity intact. The cloud adapter is
|
||||
pluggable, opt-in, and off by default.
|
||||
+535
@@ -0,0 +1,535 @@
|
||||
# Cairn OBS — Project Status
|
||||
|
||||
Phase-by-phase record of what was built, what "done" meant for each phase,
|
||||
and how it was verified. Conventions and constraints live in
|
||||
[`/CLAUDE.md`](../CLAUDE.md); the architecture spec is
|
||||
[`/docs/architecture.md`](architecture.md).
|
||||
|
||||
Each phase has a runbook in this directory recording the actual
|
||||
verification procedure and its results.
|
||||
|
||||
## Summary
|
||||
|
||||
| Phase | Scope | Status |
|
||||
|---|---|---|
|
||||
| 0 | Agent → Redpanda → ingest → ClickHouse, queryable end-to-end | Shipped |
|
||||
| 1 | Windows Event Log + journald, SQL and full-text paths | Shipped |
|
||||
| 2 | Unified query language across both stores | Shipped |
|
||||
| 3 | Dashboards, alert rules, notification delivery | Shipped |
|
||||
| 4 | RBAC, tenant isolation, audit logging, per-tenant ClickHouse | **In progress** |
|
||||
| 5 | Frontend redesign and design system | Shipped |
|
||||
| 6 | License compliance audit and remediation | Shipped |
|
||||
| 7 | AI-assisted query authoring | Shipped |
|
||||
|
||||
**Known verification gaps**, carried forward rather than buried:
|
||||
|
||||
- **Phase 4 is not shipped.** Built and unit-tested, but the environment
|
||||
lost Docker/database access partway through; only the audit-logging
|
||||
guarantees were confirmed against a live database.
|
||||
- **The Windows agent path has never run on real Windows** — no Windows
|
||||
toolchain existed in the build environment. See `/agent/README.md`.
|
||||
- **Terraform coverage is partial** — see `/terraform/README.md` for the
|
||||
full accounting.
|
||||
|
||||
## What "done" looks like for Phase 0 (MVP)
|
||||
|
||||
**Status: shipped.** A single log line, generated on a Linux host by the
|
||||
Rust agent, flows: agent → Redpanda → Go ingest service → ClickHouse, and
|
||||
is queryable via a minimal SQL endpoint and visible in a bare-bones
|
||||
SvelteKit table view. Verified end-to-end on real hardware, not just in
|
||||
CI — see `/docs/phase-0-runbook.md`. No alerting, no multi-tenancy, no
|
||||
dashboards — that discipline held for the whole phase.
|
||||
|
||||
## What "done" looks like for Phase 1
|
||||
|
||||
**Status: shipped.** A Windows Event Log entry and a Linux journald entry
|
||||
are both queryable via SQL (the ClickHouse path) and via free-text search
|
||||
(the Tantivy path), from the same UI, within a few seconds of being
|
||||
generated. Verified end-to-end on the live stack, including the same
|
||||
`record_id` coming back from both query paths for the same record — see
|
||||
`/docs/phase-1-runbook.md`.
|
||||
|
||||
ETW and WEF (Windows Event Forwarding) were *designed* in this phase but
|
||||
not required to be running for "done": ETW ships behind a feature flag
|
||||
most environments won't enable (it needs elevated privileges), and WEF's
|
||||
receiver-side was explicitly deferred rather than built. Only the Event
|
||||
Log source needed to actually be running end-to-end, and did. The
|
||||
Windows-specific agent code itself (`EvtSubscribe`, ETW, service
|
||||
registration) remains unverified on real Windows — no Windows toolchain
|
||||
existed anywhere in the environment this was built in; flagged
|
||||
prominently in `/agent/README.md` and the runbook.
|
||||
|
||||
## What "done" looks like for Phase 2
|
||||
|
||||
A single query bar in the web UI and a single `cairnobsctl query` command
|
||||
can express filter + free-text + stats in one query (e.g. `service=api |
|
||||
where status>=500 | stats count by host | sort -count`, or
|
||||
`message:"connection refused" | stats count by host`), execute correctly
|
||||
against both ClickHouse and Tantivy in one compiled plan, and return in
|
||||
well under a second for a 1M-row fixture dataset (rough benchmark, not a
|
||||
formal SLA — see `/docs/phase-2-runbook.md` for the actual measurement).
|
||||
Raw ClickHouse SQL remains available as an escape hatch, compiling to the
|
||||
same execution plan/IR as the pipe syntax so performance doesn't depend
|
||||
on which syntax a query uses.
|
||||
|
||||
Non-goals for this phase (same "resist scope creep" discipline as every
|
||||
phase so far): no alerting, no dashboards, no multi-tenancy — this phase
|
||||
is the query layer only. The two separate placeholder pages/endpoints
|
||||
from Phase 0/1 (`/query` raw-SQL-only, `/search` free-text-only) are
|
||||
retired, replaced by one `/query` endpoint and one query page.
|
||||
|
||||
See `/docs/query-language-design.md` for the grammar, IR, and
|
||||
ClickHouse/Tantivy routing strategy, and
|
||||
`/docs/query-language-reference.md` for the user-facing syntax reference
|
||||
once built.
|
||||
|
||||
## What "done" looks like for Phase 3
|
||||
|
||||
**Status: shipped.** A user can build a multi-panel dashboard from saved Phase 2 queries (at
|
||||
least a line chart panel and a table panel, working end-to-end against
|
||||
live data), save an alert rule that fires a Slack webhook when a
|
||||
condition is met (threshold comparison, or "absence" — the query returned
|
||||
zero rows in its own time window), and see the delivery attempt logged —
|
||||
all from the web UI, without touching the API directly. See
|
||||
`/docs/phase-3-dashboard-design.md` and `/docs/phase-3-alerting-design.md`
|
||||
for the data models and the alerting evaluator's firing/resolved state
|
||||
machine, and `/docs/phase-3-runbook.md` for the live-stack verification,
|
||||
including a load test of the alert evaluator against ~500 concurrent
|
||||
rules.
|
||||
|
||||
This phase adds PostgreSQL as a new pinned-stack component (see the
|
||||
dashboard design doc for why ClickHouse can't do this job — dashboards
|
||||
and alert state need real row-level locking and transactional
|
||||
read-modify-write, which ClickHouse's MergeTree family doesn't provide),
|
||||
scoped strictly to control-plane config: dashboards, panels, notification
|
||||
targets, alert rules, alert state, delivery log. Log data itself stays on
|
||||
ClickHouse/Tantivy only, unchanged.
|
||||
|
||||
Non-goals for this phase (same discipline as every phase so far):
|
||||
- No multi-tenancy enforcement and no `enterprise/` module work — single
|
||||
tenant/org assumed. Most new tables (`dashboards`, `alert_rules`,
|
||||
`notification_targets`) carry a `tenant_id` column so part of Phase 4's
|
||||
retrofit doesn't require a migration + backfill — but `alert_state` and
|
||||
`delivery_log` do not (an inconsistency found during Phase 4 planning,
|
||||
not caught at the time); Phase 4 adds `tenant_id` to those two and
|
||||
backfills via a join through `alert_rules.id`, and — per
|
||||
`/docs/phase-4-isolation-design.md` — tenant isolation itself turned
|
||||
out to live at the ClickHouse/Tantivy connection layer, not via these
|
||||
columns at all, since Phase 2's raw-SQL escape hatch can never be
|
||||
covered by a row filter regardless of which tables carry one.
|
||||
- No raw-SQL dashboard panels (time-range injection isn't reliable
|
||||
against arbitrary SQL) — pipe-syntax queries only.
|
||||
- No per-group/multi-row threshold alerting (e.g. "alert separately per
|
||||
host") — a threshold rule's query must resolve to a single row.
|
||||
- No debounce on the way down — a firing alert resolves on the first
|
||||
false evaluation, no symmetric "stay firing for N more minutes" hold.
|
||||
- No Kubernetes Operator/Helm deployment work — still docker-compose,
|
||||
`/deploy` remains stubbed.
|
||||
|
||||
## What "done" looks like for Phase 4
|
||||
|
||||
**Status: in progress, not shipped.** RBAC enforcement (`api/authz`), the
|
||||
`alerting`↔`api` service-identity credential, tenant-scoped dashboards,
|
||||
append-only audit logging, and — since the second pass on this phase —
|
||||
real per-tenant ClickHouse provisioning and query routing
|
||||
(`enterprise/internal/tenantprovision`, `enterprise/internal/chrunner`,
|
||||
wired into a new `enterprise/cmd/enterprise-api` binary alongside plain
|
||||
`api/cmd/api`) are all built and tested — real integration tests exist
|
||||
for the ClickHouse pieces, but this environment lost Docker/database
|
||||
access partway through the phase, so only the audit-logging guarantees
|
||||
were actually confirmed against a live database; the rest is untested
|
||||
beyond "compiles, and skips cleanly when no live database is
|
||||
configured" (see `/docs/phase-4-runbook.md`'s verification-status
|
||||
section). Human SSO login is now built for both protocols
|
||||
(`enterprise/internal/loginhandler`: `GET /auth/oidc/login` +
|
||||
`GET /auth/oidc/callback`, and `GET /auth/saml/login` +
|
||||
`POST /auth/saml/acs` via `enterprise/internal/saml`'s `crewjam/saml`
|
||||
wiring, both issuing a real session cookie after resolving tenant/role
|
||||
from `tenant_memberships`) — genuinely verified, unlike the ClickHouse
|
||||
pieces, via a real fake IdP for each protocol that performs actual
|
||||
cryptographic signing and verification (`coreos/go-oidc`'s `oidctest`
|
||||
for OIDC, `crewjam/saml/samlidp` for SAML — `loginhandler_test.go` and
|
||||
`saml_test.go`, all passing, including the full login round trip and
|
||||
negative paths for both), though never tried against a real external IdP
|
||||
or through a running `enterprise-auth` container. Writing the SAML test
|
||||
caught and fixed two real bugs in `internal/saml.ParseResponse`: a
|
||||
missing `r.ParseForm()` call that would have silently broken every real
|
||||
ACS POST, and email-attribute matching that missed the standard LDAP
|
||||
"mail" OID IdPs send by default. Tantivy per-tenant index routing is now
|
||||
built too
|
||||
(`search/src/registry.rs` + `enterprise/internal/searchclient`) —
|
||||
**genuinely verified**, like the OIDC login flow: Tantivy is an embedded
|
||||
library, not a networked service, so the isolation probe (three tenants,
|
||||
same search term, scoped search returns only that tenant's document)
|
||||
actually ran in this environment, no Docker needed. That same
|
||||
Docker-free advantage is what caught a real bug while closing the last
|
||||
of Phase 4 task 8's four adversarial probes (a mid-provisioning tenant
|
||||
must be refused, not served): `search/src/registry.rs`'s `IndexRegistry`
|
||||
opened-or-created an index for any syntactically-valid `tenant_id`,
|
||||
meaning a query against a tenant that exists in `rbacstore` but isn't
|
||||
active yet would have silently returned zero results from a
|
||||
freshly-created empty index instead of being refused --
|
||||
`chrunner`'s ClickHouse routing had the equivalent guarantee for free
|
||||
(a mid-provisioning tenant simply isn't in its startup-built connection
|
||||
map) but Tantivy, a separate process with no Postgres access, had no
|
||||
way to know. Fixed with a new `enterprise/internal/searchclient.
|
||||
TenantChecker` (backed by `rbacstore.TenantIsActive`); both halves of
|
||||
the fix verified Docker-free (`chrunner_test.go`'s and
|
||||
`searchclient_test.go`'s `TestSearchRefusesMidProvisioningTenant`-shaped
|
||||
tests) — see `api/queryapi/tenant_isolation_gap_test.go` for the full
|
||||
accounting of all four probes, now all closed. The deployment-
|
||||
topology gap that briefly was the largest one is now closed for both
|
||||
Helm and docker-compose: `deploy/helm/cairnobs/templates/api.yaml`/
|
||||
`enterprise-api.yaml` are mutually exclusive on the same
|
||||
`enterprise.enabled` flag that turns on RBAC/audit/SSO, rendering to the
|
||||
same Service name/port either way — a Helm-deployed cluster can't
|
||||
accidentally run the wrong one. `docker-compose.yml`'s `api`/
|
||||
`enterprise-api` services are now the same mutually-exclusive choice,
|
||||
gated behind `COMPOSE_PROFILES` (`.env` checks in `single-tenant` as the
|
||||
zero-config default) and sharing a host port/network-alias trick so
|
||||
`alerting`/`web` need no conditional logic either way — verified via
|
||||
`docker compose config` (renders/validates without a daemon, confirms
|
||||
the two never both appear for one profile selection), not an actual
|
||||
`docker compose up` in this environment. Per-resource dashboard grants
|
||||
(the RBAC matrix's "(own/granted)"
|
||||
qualifier) are now enforced too: `api/dashboards.PermissionStore` (core
|
||||
interface) implemented by `enterprise/internal/rbacstore.
|
||||
DashboardPermissions`, wired in only by `enterprise-api` — an Editor can
|
||||
now only edit/delete a dashboard they created or were granted access to,
|
||||
not every dashboard in their tenant; managing grants themselves is
|
||||
stricter still (creator/Admin/Owner only, closing a self-escalation
|
||||
path). Verified against a fake store (`api/dashboards/handler_test.go`);
|
||||
real integration tests exist but haven't run against a live Postgres,
|
||||
same disclosed gap as the rest of this phase's Postgres-backed pieces.
|
||||
`cairnobsctl dashboards permissions list|grant|revoke` is now the CLI
|
||||
surface for this — `PUT`/`DELETE /dashboards/{id}/permissions/{userId}`
|
||||
previously had no caller but Go tests and curl.
|
||||
`deploy/operator`'s `Tenant` CRD and `enterprise-api -provision-tenant`
|
||||
are now unified too, deliberately lightweight rather than making the
|
||||
K8s controller a second real actor: `-provision-tenant` stays the sole
|
||||
caller of ClickHouse/`rbacstore`, and (via a new
|
||||
`enterprise/internal/tenantcrd`, gated on `TENANT_CRD_NAMESPACE`) syncs
|
||||
its real result into the CRD — a real credential Secret, not the
|
||||
previous placeholder that authenticated against nothing, and status
|
||||
fields the reconciler derives `Phase`/`Ready` from instead of
|
||||
independently guessing "Active" the moment a Tenant object exists. The
|
||||
tenant-picker is now fully built, backend and frontend: an identity with
|
||||
more than one `tenant_memberships` row gets a real `GET
|
||||
/auth/memberships`/`POST /auth/select-tenant` round trip (a short-lived
|
||||
pending-login token, distinct from a real session by both Go type and
|
||||
JWT claim name — a real token-confusion bug this design's own tests
|
||||
caught before it shipped) instead of the flat refusal Phase 4 shipped
|
||||
with earlier, and `web/src/routes/select-tenant` is the page that
|
||||
actually calls it — see the Phase 4 exit-criteria paragraph below for
|
||||
what changed to make that verifiable in this environment. Ingest
|
||||
tenant-awareness — the gap this section used to
|
||||
call "undesigned" — now has a real, if intentionally partial, design:
|
||||
`ingest` (AGPL core) gained an optional `TenantResolver`
|
||||
(`ingest/internal/grpcserver`), a per-tenant bearer credential an agent
|
||||
presents (minted via `enterprise-auth
|
||||
-create-ingest-credential-tenant=<id>`, validated over the network via a
|
||||
new `POST /internal/authorize-ingest` endpoint — never an `enterprise/`
|
||||
import, same boundary shape as `api/authz.Authorizer`), and the
|
||||
resolved tenant ID is attached to every record as a `tenant_id` Kafka
|
||||
message header before it's produced. **ClickHouse write-routing is now
|
||||
built too**: `enterprise/cmd/enterprise-ingest` (another "second binary,"
|
||||
mirroring `enterprise-api`) reuses `ingest/consumer`'s own flush loop
|
||||
with `enterprise/internal/chwriter.Registry` swapped in as the writer —
|
||||
one dedicated ClickHouse connection per tenant, routing each batch's
|
||||
records by their `tenant_id` tag, fail-closed on an untagged or
|
||||
unprovisioned tenant. Building it found and fixed a real bug:
|
||||
`tenantprovision.ProvisionClickHouse` originally granted a tenant's
|
||||
ClickHouse user `SELECT` only, which would have made every real
|
||||
per-tenant write fail with a permission error — fixed by granting
|
||||
`SELECT, INSERT` (one credential, both directions; no cross-tenant
|
||||
boundary is crossed by also allowing INSERT within a tenant's own
|
||||
database). **Tantivy write-routing is now built too** —
|
||||
`search/src/consumer.rs` (a completely independent Redpanda consumer,
|
||||
not called through `ingest` or `enterprise-ingest` at all: a different
|
||||
codebase and process) now resolves each record's `tenant_id` header
|
||||
through the *same* `IndexRegistry` the read side already used, and
|
||||
routes the write there instead of always into the default index. Unlike
|
||||
the ClickHouse side, this needed no "second binary": `IndexRegistry`
|
||||
already lives in this AGPL-core binary (Tantivy has no grant system to
|
||||
gate a separately-credentialed binary behind, so there was never an
|
||||
import-boundary reason to split it out -- true regardless of licensing,
|
||||
though at the time of writing `enterprise/` was still commercially
|
||||
licensed; both sides are AGPLv3 as of Phase 6), so read and write share one
|
||||
registry directly. The periodic Tantivy commit now commits every tenant
|
||||
index that's seen a write, not just the default one
|
||||
(`IndexRegistry::commit_all`). **The active-tenant gap this same change
|
||||
originally disclosed is now closed too**: `search/src/tenants.rs`'s
|
||||
`ActiveTenantTracker` polls a new `GET /internal/active-tenants`
|
||||
endpoint on `enterprise-auth` (`search` has no Postgres access, unlike
|
||||
`chwriter.Registry`'s direct `rbacstore` query or the read side's
|
||||
`searchclient.TenantChecker`, so this needed a network call — the same
|
||||
"network boundary, not import boundary" shape `ingest`'s
|
||||
`TenantResolver` already uses against the same service, authenticated
|
||||
with a RoleService credential the same way `alerting` authenticates to
|
||||
`api`) and `consumer.rs` refuses any tagged record whose tenant isn't in
|
||||
the polled allowlist. Off unless both `ENTERPRISE_AUTH_URL` and
|
||||
`ENTERPRISE_AUTH_SERVICE_TOKEN` are set (same "off unless configured"
|
||||
default as everything else optional in this codebase); when they are,
|
||||
startup blocks on the first fetch succeeding and later refresh failures
|
||||
keep serving the last-known-good set rather than clearing it. Verified
|
||||
with real HTTP round trips against a hand-rolled TCP test server in this
|
||||
environment, no live enterprise-auth needed. **This closing move exposed
|
||||
the ClickHouse side's own gap by comparison** — `chwriter.Registry`'s
|
||||
writer map was still a startup-only snapshot with no refresh at all, a
|
||||
real asymmetry once Tantivy's tracker refreshed every minute and
|
||||
ClickHouse's didn't — so `Registry.StartRefreshing` (new) closes that
|
||||
too: same one-minute interval, same last-known-good posture on a failed
|
||||
refresh, opening connections for newly-active tenants and closing ones
|
||||
no longer active. Both engines now share the same active-tenant
|
||||
staleness bound instead of one being materially staler than the other.
|
||||
**The tenant-picker page is now built too**:
|
||||
`web/src/routes/select-tenant` calls `GET /auth/memberships`/
|
||||
`POST /auth/select-tenant` via `fetch(..., {credentials: 'include'})`
|
||||
(new `$lib/api.ts` functions), which needed a second CORS posture
|
||||
alongside the wildcard-friendly one `enterprise-api` already had —
|
||||
`api/httpserver.WithCredentialedCORS`, set to a literal origin via a new
|
||||
`CORS_ALLOWED_ORIGIN` on `enterprise-auth` — since browsers refuse to
|
||||
honor a wildcard `Access-Control-Allow-Origin` on a credentialed
|
||||
request. **Genuinely verified in a real browser in this environment**:
|
||||
a throwaway Node server standing in for `enterprise-auth`'s exact wire
|
||||
contract (including its plain-text `http.Error` bodies, not JSON) on a
|
||||
different origin than `web`'s dev server, driven through the full
|
||||
cross-origin pending-login-cookie round trip, a real click choosing a
|
||||
tenant, and the post-selection redirect — plus the missing/expired-
|
||||
pending-login error path — with no Docker or live Postgres/IdP needed,
|
||||
since the point was exercising `web`'s own fetch/CORS/cookie wiring, not
|
||||
`enterprise-auth`'s internals (already covered by that package's own
|
||||
tests). See `/web/README.md`'s "Tenant picker" section for the exact
|
||||
setup. What's left in this phase now is entirely the caveats already
|
||||
disclosed above, not an unbuilt feature: the ClickHouse/Postgres-backed
|
||||
pieces have never run against a real database in this environment, and
|
||||
nothing here has been tried against a real external IdP or a real
|
||||
running multi-container deployment. Full accounting:
|
||||
`/docs/security/threat-model.md`; step-by-step verification procedure
|
||||
(not yet run against a live cluster in this environment):
|
||||
`/docs/phase-4-runbook.md`. The rest of this section describes the exit
|
||||
bar this phase is aiming at, not a completed state.
|
||||
|
||||
Two tenants can be provisioned with SSO (OIDC or SAML), each with their
|
||||
own users, roles, dashboards, and alert rules, fully isolated at the
|
||||
ClickHouse/Tantivy connection layer — not by a row filter — with
|
||||
adversarial integration tests proving no cross-tenant data leakage,
|
||||
including via the raw-SQL escape hatch and ClickHouse's own `system.*`
|
||||
tables. A tenant admin can see a query audit trail for their tenant,
|
||||
backed by append-only storage a compromised application credential
|
||||
cannot alter (enforced by database grants, not just convention) and
|
||||
periodically anchored outside the database so tampering is detectable
|
||||
even against a privileged attacker. See `/docs/phase-4-isolation-design.md`
|
||||
for the tenant isolation model and why it lives at the connection layer,
|
||||
`/docs/phase-4-rbac-design.md` for the role/permission model, and
|
||||
`/docs/security/threat-model.md` for the auth flows and audit-log
|
||||
integrity guarantees, written for a prospective enterprise customer's
|
||||
security team.
|
||||
|
||||
The tenant-isolation, provisioning, SSO, and RBAC-enforcement mechanisms
|
||||
live entirely in `enterprise/` (commercial license at the time this
|
||||
section was written; relicensed to AGPLv3 in Phase 6, see that phase's
|
||||
section below), confirmed explicitly rather than assumed: core
|
||||
(`/api`, `/alerting`, `/web`) stays genuinely single-tenant, with no
|
||||
multi-tenant mechanism present at all — `enterprise/` supplies
|
||||
tenant-scoped implementations of core's
|
||||
already-shipped `querylang/executor.SQLRunner`/`SearchClient` interfaces
|
||||
rather than core growing tenant awareness. Query-compiler-level "compile
|
||||
time" enforcement, as originally proposed, turned out not to be
|
||||
achievable in any module once Phase 2's opaque raw-SQL passthrough is
|
||||
accounted for — the honest, implemented guarantee is that every code
|
||||
path (compiled query or raw SQL) is forced through a tenant-scoped
|
||||
database connection/index that the database's own access control
|
||||
enforces, not a compiler-injected filter.
|
||||
|
||||
Non-goals for this phase (same discipline as every phase so far):
|
||||
- No deny-override permissions — per-resource grants (e.g. a specific
|
||||
user getting edit access to one dashboard) are additive only; a full
|
||||
allow/deny ACL system is future work.
|
||||
- No data retention/deletion policy design for tenant deprovisioning —
|
||||
the provisioning state machine includes a `deprovisioning` state, but
|
||||
what actually happens to a deprovisioned tenant's data is a separate,
|
||||
not-yet-designed compliance question.
|
||||
- No general multi-cluster orchestration in `/deploy` — scoped to
|
||||
proving the per-tenant ClickHouse/Tantivy isolation model works, not a
|
||||
fully general multi-cluster system.
|
||||
- No protection against a privileged ClickHouse/Postgres administrator —
|
||||
the isolation and audit-log guarantees in this phase are structural
|
||||
defenses against application-layer bugs and injection, not against
|
||||
someone with database superuser access; that's an operational control,
|
||||
out of scope here and named explicitly, not silently assumed away.
|
||||
|
||||
## What "done" looks like for Phase 5
|
||||
|
||||
**Status: shipped.** A ground-up frontend redesign — visual direction,
|
||||
a real design system, navigation/IA, charting, dashboard panels,
|
||||
query/search, and alerting UI — plus an accessibility pass, all verified
|
||||
against a live docker-compose stack with real seeded data, not just
|
||||
`npm run check`/`npm run build` passing. See `/docs/design-system.md`
|
||||
for the token system and component library, and
|
||||
`/docs/phase-5-runbook.md` for the full verification log, including five
|
||||
real bugs this phase's live-verification discipline caught that a
|
||||
type-checked, successfully-building frontend would not have surfaced on
|
||||
its own.
|
||||
|
||||
The visual direction ("Signal": near-neutral grayscale UI, color rationed
|
||||
to the four-tier severity system plus a single interactive accent, real
|
||||
dark-mode-as-default) was picked from three proposed directions before
|
||||
any token or component work started, per an explicit stop point in this
|
||||
phase's brief. The charting library (ECharts, over Observable Plot and
|
||||
raw D3 — see the design-system doc for the reasoning and the verified
|
||||
bundle-size/perf numbers) was likewise confirmed before being wired into
|
||||
every panel type, the second explicit stop point.
|
||||
|
||||
Two of the five bugs this phase's verification caught were backend bugs
|
||||
with no connection to the frontend redesign itself, only surfaced
|
||||
because getting real dashboard/alert data to verify the new UI against
|
||||
required actually exercising write paths nothing had exercised since
|
||||
Phase 4's `tenant_id` migrations landed:
|
||||
|
||||
- `alerting`'s `rulestore.Create`/`ApplyTransition` never populated the
|
||||
`tenant_id` column Phase 4 added to `alert_state`/`delivery_log` (with
|
||||
a `NOT NULL` constraint) — every alert rule created against a
|
||||
Phase-4-or-later database silently failed. Existing rows all had a
|
||||
value from Phase 4's backfill migration, which is exactly why this
|
||||
went uncaught: Phase 4's own verification never created a *new* rule
|
||||
post-migration, and its runbook already discloses that Docker access
|
||||
was lost partway through that phase.
|
||||
- `dashboard_panels`'s `viz_type` CHECK constraint was never updated
|
||||
alongside `heatmap`'s addition to the Go/TS validators — a three-place
|
||||
change (Go validator, TS union, DB constraint), not two.
|
||||
|
||||
Both are fixed (`alerting/internal/rulestore/store.go`,
|
||||
`metadata/migrations/0035_add_heatmap_viz_type.sql`) and confirmed
|
||||
against a live stack: rule creation → evaluation → firing → a real
|
||||
(failed, to an intentionally fake webhook) delivery attempt, and a
|
||||
heatmap panel created, persisted, and rendered end to end. See the
|
||||
runbook for the other three findings (one more real product bug — a
|
||||
`findIndex`/nullish-coalescing bug in the chart-pivoting logic that made
|
||||
every `single_stat` panel render `0` — and two real accessibility
|
||||
findings caught by axe-core against live-rendered pages with real data,
|
||||
not fixture data or empty states).
|
||||
|
||||
Non-goals for this phase (same discipline as every phase so far):
|
||||
- No query-language or data-model changes beyond the one narrowly
|
||||
justified exception: `heatmap` as a `VizType`, needed to feed a new
|
||||
visualization, not a new query capability.
|
||||
- No changes to tenant isolation, RBAC, SSO, or audit logging — Phase 4's
|
||||
surface area is untouched; this phase is presentation-layer only.
|
||||
- No mobile-phone-width layout — responsive verification stops at
|
||||
tablet-landscape width, per the brief's explicit scope ("laptop/
|
||||
tablet-landscape," not phone-width).
|
||||
- No fuzzy search in the command palette, no data-grid virtualization for
|
||||
very large result sets, no chart types beyond the five built
|
||||
(time-series, bar, single-stat, heatmap, top-N) — real, disclosed
|
||||
future work, not oversights.
|
||||
|
||||
## What "done" looks like for Phase 6
|
||||
|
||||
**Status: shipped.** A full license-compliance audit and remediation
|
||||
pass across the entire monorepo. Full report:
|
||||
`/docs/compliance/license-audit-report.md`;
|
||||
machine-readable inventory: `/docs/compliance/license-inventory.{csv,json}`
|
||||
(776 rows, 502 unique dependencies across Rust/Go/npm plus Docker base
|
||||
images and vendored assets); ongoing policy:
|
||||
`/docs/compliance/license-policy.md`, now enforced in CI
|
||||
(`.github/workflows/license-compliance.yml` — this repo's first CI
|
||||
workflow file).
|
||||
|
||||
Every dependency was inventoried and classified; 774 of 776 rows
|
||||
resolved cleanly to AGPLv3-compatible with real citations, not guesses
|
||||
(see the audit report for the reasoning on each non-obvious case —
|
||||
dual-licensed crates, MPL-2.0, a license-detector false negative on
|
||||
`segmentio/asm`); `enterprise/` relicensed to AGPLv3 throughout the
|
||||
repo, with the deliberate business-model consequence recorded (anyone,
|
||||
including competitors, can now legally self-host or fork those
|
||||
features); confirmed no license-gating/entitlement logic ever existed to
|
||||
remove; a root `LICENSE` file added (there wasn't one before this
|
||||
phase); CI enforcement wired up and every command verified locally.
|
||||
|
||||
**The one real flag — Redpanda's BSL 1.1 license (confirmed against
|
||||
primary sources for the pinned v24.2.7, not assumed to still be
|
||||
Apache-2.0) — is resolved, not outstanding**: decision recorded
|
||||
2026-08-16, accept as-is. Cairn OBS's own use (internal Kafka-protocol
|
||||
transport, no resale of broker access) sits within BSL's Additional Use
|
||||
Grant; the harder question — whether a third party self-hosting Cairn OBS
|
||||
"as a service" using the bundled `docker-compose.yml` could trip BSL's
|
||||
anti-resale restriction on Redpanda specifically — was judged unlikely
|
||||
given Cairn OBS's ingest pipeline creates fixed internal topics, not
|
||||
per-end-user ones, and was accepted as a disclosed, known risk rather
|
||||
than triggering a swap to Apache Kafka (real resource-footprint cost) or
|
||||
dropping the bundled broker image (rougher local dev experience). See
|
||||
the audit report's Redpanda section for the full reasoning, the other
|
||||
two options that were considered and not chosen, and the condition under
|
||||
which this should be revisited (an official hosted/managed Cairn OBS
|
||||
offering, which would make the third-party-SaaS scenario Cairn OBS's own
|
||||
rather than a hypothetical one).
|
||||
|
||||
Non-goals for this phase: replacing permissively-licensed dependencies
|
||||
with copyleft ones (explicitly out of scope per the phase's own brief);
|
||||
per-file SPDX license headers across the monorepo's several thousand
|
||||
source files (a deliberate choice — see the audit report's "Own license
|
||||
declarations" section for why root `LICENSE` + manifest fields was
|
||||
judged sufficient); redesigning `favicon.svg` (flagged as a leftover
|
||||
SvelteKit scaffold asset, not a license blocker — a design task, not a
|
||||
compliance one).
|
||||
|
||||
## What "done" looks like for Phase 7
|
||||
|
||||
**Status: shipped.** AI-assisted query authoring: from the same query
|
||||
bar, a user can (a) get AI-assisted autocomplete, explanations, and fix
|
||||
suggestions while writing pipe-syntax or SQL queries by hand, and (b)
|
||||
type a plain-English question and get a generated structured query with
|
||||
explanation, editable before running — both paths executing through the
|
||||
unchanged Phase 2 compiler (`api/internal/querylang/planner`,
|
||||
`api/querylang/executor`) with Phase 4 tenant scoping, cost guardrails,
|
||||
and audit logging applying identically to both. No cloud dependency
|
||||
required for the default deployment (self-hosted via Ollama,
|
||||
`qwen2.5-coder:7b`/`1.5b`, both Apache-2.0 — chosen specifically to keep
|
||||
Phase 6's license-purity work intact; a pluggable, opt-in, off-by-default
|
||||
cloud adapter exists for deployments that want one).
|
||||
|
||||
Non-negotiable design principle held throughout, confirmed by inspection
|
||||
of the actual code paths rather than merely asserted: every AI-assisted
|
||||
or AI-translated query compiles down to and executes through the same
|
||||
Phase 2 IR and compiler, and passes through the same Phase 4
|
||||
tenant-scoping enforcement and cost guardrails as a hand-written query —
|
||||
no parallel execution path, no scoping shortcut, for either track. No AI
|
||||
code path anywhere constructs a `SQLRunner`, calls `executor.Execute`,
|
||||
or bypasses `authz.RequireRoleOrService`.
|
||||
|
||||
Every AI-assisted suggestion a user explicitly accepts or dismisses
|
||||
(translate/fix/optimize — deliberately not ghost-text completion or
|
||||
explain, see the design doc for why) is logged into the same
|
||||
append-only, hash-chained `audit_log` table Phase 4 built, via a new
|
||||
`event_type='ai_interaction'` rather than a new table
|
||||
(`metadata/migrations/0036`) — genuinely verified against a live
|
||||
Postgres in this environment, not just unit-tested against a fake, the
|
||||
same rigor Phase 4's own audit-logging guarantees were held to.
|
||||
|
||||
Two real product bugs were found and fixed via this phase's live
|
||||
browser verification — neither would have been caught by
|
||||
`svelte-check`/`npm run build` — and a real logic bug in the cost/safety
|
||||
guard itself (an unbounded-aggregation-vs-raw-row distinction) was found
|
||||
and fixed by the test suite written for it. Full accounting of all
|
||||
three: `/docs/phase-7-ai-design.md`. Integration tests
|
||||
(`api/ai/aiapi/integration_test.go`) wire a real `ollama.Client` through
|
||||
a real `router`/`Handler` against a mock server matching Ollama's actual
|
||||
wire contract (`hack/mock-ollama`, new — also used for this phase's live
|
||||
verification), proving the plumbing without needing real model weights;
|
||||
testing actual model *quality* is deliberately kept out of CI as a
|
||||
disclosed, periodic human-run checklist item instead — see the design
|
||||
doc's CI-testability section for the reasoning.
|
||||
|
||||
Explicit non-goals for this phase (scoped out, not deferred by oversight):
|
||||
result summarization, incident narrative generation, and proactive/
|
||||
unprompted AI suggestions — this phase is query authoring assistance
|
||||
only (structured and natural-language), not analysis or automation. Real
|
||||
future-phase candidates, not silently dropped.
|
||||
|
||||
See `/docs/phase-7-ai-design.md` for the model-provider architecture,
|
||||
shared foundation (schema grounding, cost/safety guard), both tracks'
|
||||
build-and-verification record, and the audit-logging/CI-testability
|
||||
design; `/docs/phase-7-runbook.md` for the step-by-step live-stack
|
||||
verification procedure.
|
||||
|
||||
Reference in New Issue
Block a user