jcoffey-dev
ba2276aa1a
Closes the last named "isolation mechanism" gap: search.proto gains a tenant_id field on SearchRequest; search/src/registry.rs's IndexRegistry resolves it to an on-demand-opened, per-tenant Tantivy index (empty tenant_id keeps today's single default index, so this is purely additive); enterprise/internal/searchclient sets that field from the authenticated request identity in ctx, mirroring chrunner's exact fail-closed "never a parameter" shape. Wired into enterprise-api in place of the shared api/searchclient. Unlike the ClickHouse pieces from the previous two commits, this one is genuinely verified end to end in this environment: Tantivy is an embedded library, not a networked service, so both the Rust index registry (cargo test, cargo clippy --all-targets -- -D warnings, both clean) and the Go client (a real in-process gRPC server) could actually run. registry.rs's tenant_index_is_isolated_from_default_and_other_tenants seeds three real indices with the same term and confirms a tenant-scoped search returns only that tenant's document -- item 3 of the isolation design doc's verification plan, closed for real, not just written. With both ClickHouse and Tantivy isolation now built, the single largest remaining gap is no longer a missing mechanism: it's that nothing forces or flags whether a deployment actually runs enterprise-api instead of plain api, and that ingest itself has no tenant concept for either storage engine (every record still lands in the one shared database/ index no matter what -- undesigned, not just unbuilt). Updated the threat model, architecture doc, CLAUDE.md, and both READMEs accordingly.
202 lines
14 KiB
Markdown
202 lines
14 KiB
Markdown
# Sentry Architecture
|
|
|
|
> **Status:** Updated through Phase 4. The component map/diagram below is
|
|
> still the Phase 0 request path (agent → ingest → ClickHouse → api →
|
|
> web) — it was never redrawn for the full-text search, dashboards/
|
|
> alerting, or enterprise/ additions; see each phase's runbook
|
|
> (`/docs/phase-N-runbook.md`) for what was actually verified when it
|
|
> shipped. The component responsibilities table and the sections below
|
|
> the diagram are kept current. Phase 0's original framing ("draft,
|
|
> correct as needed") still applies to anything not yet built.
|
|
|
|
## Mission
|
|
|
|
Open-core, Kubernetes-native centralized logging platform. Compete with
|
|
Splunk on features; win on cost-per-GB, a modern language stack, and
|
|
multi-tenant RBAC that's actually honest about its guarantees.
|
|
|
|
## Component map
|
|
|
|
```
|
|
┌──────────┐ gRPC/mTLS ┌──────────┐ produce ┌───────────┐ consume ┌────────────┐
|
|
│ agent │ ────────────▶ │ ingest │ ──────────▶ │ Redpanda │ ────────▶ │ ingest │
|
|
│ (Rust) │ │ (Go) │ │ (Kafka API)│ │ consumer │
|
|
└──────────┘ └──────────┘ └───────────┘ │ (Go) │
|
|
└──────┬─────┘
|
|
│ batch INSERT
|
|
▼
|
|
┌────────────┐
|
|
│ ClickHouse │
|
|
└─────┬──────┘
|
|
│ SQL
|
|
┌───────────▼───────────┐
|
|
│ api (Go: gRPC+REST) │
|
|
└───────────┬───────────┘
|
|
│ REST
|
|
┌───────────▼───────────┐
|
|
│ web (SvelteKit) │
|
|
└────────────────────────┘
|
|
```
|
|
|
|
Decision (confirmed 2026-08-12): Redpanda stays in the Phase 0 path. The
|
|
`ingest` service's gRPC front end produces to Redpanda rather than writing
|
|
ClickHouse directly; a separate consumer path reads from Redpanda and batches
|
|
inserts into ClickHouse. This exercises the real transport layer from day
|
|
one instead of deferring it, and keeps Kafka credentials off the edge agent.
|
|
|
|
## Storage / query split
|
|
|
|
- **ClickHouse** is the analytical store of record for structured log data:
|
|
timestamp, host, service, severity, message, plus a `Map(String,String)`
|
|
for arbitrary structured fields. Partitioned by day, ordered by
|
|
`(service, timestamp)`.
|
|
- **Tantivy** (Phase 1) will provide full-text indexing over the `message`
|
|
field and unstructured payloads, queried out-of-band from ClickHouse and
|
|
joined by a log identifier. Not built in Phase 0.
|
|
- **Schema-on-write** using OTel semantic conventions as the default log
|
|
schema; schema-on-read fallback for unstructured/raw text that doesn't fit
|
|
the structured columns (captured via the `Map` column and/or a raw
|
|
passthrough field).
|
|
- **Postgres** (Phase 3) holds control-plane config only — dashboards,
|
|
panels, notification targets, alert rules/state, delivery log, and
|
|
(Phase 4) tenants/users/tenant_memberships/audit_log. Never log data;
|
|
ClickHouse/Tantivy stay the only place a log record itself lives. See
|
|
`/docs/phase-3-dashboard-design.md` for why ClickHouse's MergeTree
|
|
family isn't a fit for this (no real row-level locking/transactional
|
|
read-modify-write).
|
|
|
|
This split is not to be changed without discussion — see CLAUDE.md.
|
|
|
|
## Component responsibilities
|
|
|
|
| Component | Responsibility |
|
|
|---|---|
|
|
| `agent` (Rust, musl) | Tail a log file or read journald; parse RFC 5424 syslog with raw passthrough fallback; batch; ship via gRPC/mTLS to `ingest`. Windows (ETW/Event Log) code exists but is unverified on real Windows hardware — see `/agent/README.md`. |
|
|
| `proto` | Shared `.proto` contracts for agent↔ingest and api↔search gRPC, versioned independently of either side. |
|
|
| `transport` | Redpanda docker-compose + topic provisioning scripts. No application code. |
|
|
| `ingest` (Go) | gRPC server accepting agent connections; produces normalized OTel-log-like records to Redpanda; separate consumer reads from Redpanda and batch-writes to ClickHouse. No tenant concept — every record lands in the one shared `logs` table regardless of source (see "Tenant isolation" below). |
|
|
| `storage` | ClickHouse schema migrations + docker-compose for local/homelab. |
|
|
| `search` (Rust, Phase 1) | Consumes the same Redpanda topic `ingest` does (own offset tracking), builds a Tantivy full-text index over `message`, serves matches over gRPC. Writes always go to one shared (default) index (`ingest` isn't tenant-aware); reads can be scoped per-tenant via `SearchRequest.tenant_id` and `src/registry.rs`'s `IndexRegistry` (Phase 4) — see "Tenant isolation" below. |
|
|
| `api` (Go) | gRPC + REST gateway. `POST /query` compiles pipe-syntax or raw SQL to one IR, executed across ClickHouse/Tantivy (`/docs/query-language-design.md`). `internal/dashboards` is CRUD only — panel query execution happens client-side, reusing `/query`. `internal/authz` (Phase 4) enforces RBAC via a network call to `enterprise-auth`, never an import. |
|
|
| `alerting` (Go, Phase 3) | Evaluates alert rules on an interval, calls `api`'s `POST /query` (via a `RoleService` credential once Phase 4 auth is configured — see `/docs/phase-4-isolation-design.md`'s alerting↔api gap), delivers firing/resolved notifications (webhook/Slack/PagerDuty). |
|
|
| `enterprise` (Go, commercial license, Phase 4) | OIDC login (`internal/loginhandler`'s `/auth/oidc/login`+`/auth/oidc/callback`, real IdP round trip, verified with a fake IdP but not a real external one), RBAC storage (`internal/rbacstore`), session/service-token issuance (`internal/session`), the append-only audit log (`internal/audit`), `enterprise-auth`'s HTTP surface (`/internal/authorize`, `/auth/features`), per-tenant ClickHouse provisioning (`internal/tenantprovision`) and query routing (`internal/chrunner`), and `cmd/enterprise-api` — a second binary combining core's `api/queryapi`/`api/dashboards` handlers with these tenant-aware implementations. Never imported by core — see "Licensing boundary" below. Also `internal/searchclient` (per-tenant Tantivy routing, wired the same way into `search`). Does **not** yet include SAML's login ACS handler (protocol mechanics only) — see `/docs/security/threat-model.md`. |
|
|
| `web` (SvelteKit, static build) | Query bar, dashboards, alerts, and (Phase 4) a settings page that renders SSO status via a runtime capability check (`GET /auth/features`) rather than bundling enterprise-licensed components. |
|
|
| `cli` (`sentryctl`) | `ping`, `query`, `dashboards` (list/get/apply), `alerts` (list/get/apply). `$SENTRYCTL_TOKEN`, if set, is forwarded as a Bearer credential (Phase 4). |
|
|
| `deploy` | A Helm chart covering every `docker-compose.yml` service, plus (Phase 4) a small Go Operator managing one CRD (`Tenant`) that provisions a per-tenant ClickHouse credential Secret. Never applied to a live cluster in the environment this was built in — see `/deploy/README.md`'s verification section before trusting it. |
|
|
|
|
## Tenant isolation model (Phase 4)
|
|
|
|
Full design rationale: `/docs/phase-4-isolation-design.md`. Full honest
|
|
accounting of what's actually enforced vs. designed-only:
|
|
`/docs/security/threat-model.md` — read that before assuming any claim
|
|
below holds for log data specifically.
|
|
|
|
**As designed:** one dedicated ClickHouse database + narrowly-granted
|
|
user per tenant (never the shared `default`/admin credential), one
|
|
dedicated Tantivy index directory per tenant, `system.*` access revoked
|
|
per tenant, connections resolved from an immutable per-tenant map (never
|
|
a shared pool with session-level `USE`). Isolation lives at the
|
|
**connection layer** — every query, compiled or raw SQL, is forced
|
|
through a tenant-scoped connection the database's own access control
|
|
enforces — not at the query-compiler layer, since Phase 2's raw-SQL
|
|
escape hatch is opaque to any compiler-injected filter.
|
|
|
|
**As built, currently:**
|
|
|
|
- Role-based access control (`api/authz`) is live on `/query`
|
|
and `/dashboards`, resolved via `enterprise-auth` over HTTP.
|
|
- Control-plane tenant scoping is live for dashboards
|
|
(`api/dashboards`'s store filters every query by the
|
|
authenticated identity's tenant, never a client-supplied field).
|
|
- The `alerting`↔`api` service-identity gap (task 2's finding) is
|
|
closed: a `RoleService` credential, distinct from every human role.
|
|
- **ClickHouse connection-layer isolation is built**, but lives in a
|
|
second binary: `enterprise/internal/tenantprovision` (real `CREATE
|
|
DATABASE`/`CREATE USER`/`GRANT` against ClickHouse) and
|
|
`enterprise/internal/chrunner` (a per-tenant connection registry
|
|
implementing `api/querylang/executor.SQLRunner`, resolving the
|
|
right tenant's connection from the authenticated identity in request
|
|
context) are wired into `enterprise/cmd/enterprise-api` — a binary
|
|
that imports both `api`'s handler packages and enterprise's
|
|
tenant-aware implementations (the allowed `enterprise → api` import
|
|
direction; core still never imports `enterprise/`). Real integration
|
|
tests assert a tenant cannot read another tenant's database by
|
|
fully-qualified name, and that `system.query_log`/`system.tables`/
|
|
`SHOW DATABASES` don't leak across tenants either — written but not
|
|
yet run against a live ClickHouse in this environment, see
|
|
`/docs/security/threat-model.md` and `/docs/phase-4-runbook.md`'s
|
|
verification-status sections. Plain `api/cmd/api` still exists,
|
|
unchanged, with its single shared connection — nothing forces a
|
|
deployment to run `enterprise-api` instead, and nothing flags it if it
|
|
doesn't.
|
|
- **Tantivy index-layer isolation is built, and verified.**
|
|
`search/src/registry.rs`'s `IndexRegistry` resolves
|
|
`SearchRequest.tenant_id` (added to `proto/sentry/search/v1/
|
|
search.proto`) to its own on-disk index, opened on demand;
|
|
`enterprise/internal/searchclient` sets that field from the
|
|
authenticated request identity, the same "read from ctx, fail closed"
|
|
shape `chrunner` uses. Unlike the ClickHouse pieces, this one actually
|
|
ran in the environment it was built in — Tantivy is an embedded
|
|
library, so the cross-tenant isolation probe needed no live database
|
|
or Docker to execute for real, and it passed.
|
|
- Neither storage engine's isolation extends to *ingest*: every record
|
|
`ingest` produces lands in the one shared ClickHouse database and the
|
|
one shared (default) Tantivy index regardless of tenant. A
|
|
newly-provisioned tenant's database/index are real and isolated at
|
|
query time — and permanently empty until something upstream of
|
|
`chrunner`/`searchclient` becomes tenant-aware on the write side,
|
|
which is undesigned, not merely unbuilt.
|
|
- `deploy/operator`'s `Tenant` CRD still manages only the K8s-side
|
|
artifact (a credential Secret); the Helm chart has no service
|
|
definition for `enterprise-api` yet.
|
|
|
|
The deployment-topology gap — giving the system an actual way to route
|
|
traffic to `enterprise-api` instead of `api` (a Helm service, or at
|
|
minimum a documented, enforced convention) — is now the single largest
|
|
remaining gap between this system and the isolation model it was
|
|
designed to have; both storage engines' connection/index-layer
|
|
mechanisms themselves are built.
|
|
|
|
## Licensing boundary
|
|
|
|
AGPLv3 for core + agents. Enterprise features (SSO, RBAC storage, audit
|
|
logging) live under `enterprise/` (commercial license stub, added
|
|
Phase 4). AGPL code must never import from `enterprise/` — enforced in
|
|
CI by `hack/check-tenant-boundary.sh`, which greps every build for the
|
|
import edge. Where core needs a decision only `enterprise/` can make
|
|
(is this request authorized, what SSO is configured), it calls
|
|
`enterprise-auth` over plain HTTP instead
|
|
(`api/authz.HTTPAuthorizer`, `web`'s `GET /auth/features`) —
|
|
the same "network boundary, not import boundary" shape `/alerting`↔`api`
|
|
already used before `enterprise/` existed.
|
|
|
|
## Non-negotiables carried from CLAUDE.md
|
|
|
|
- Rust agent: statically linked musl, `x86_64-unknown-linux-musl` and
|
|
`aarch64-unknown-linux-musl`, no glibc runtime deps.
|
|
- Windows support via native ETW/Event Log API, not WSL — designed
|
|
(Phase 1) but still unverified on real Windows hardware.
|
|
- Every UI action maps to a documented REST/gRPC call — no UI-only logic.
|
|
- Pinned stack (see CLAUDE.md table) — no substitutions without discussion.
|
|
|
|
## Explicitly out of scope (current, Phase 4)
|
|
|
|
Per `/CLAUDE.md`'s Phase 4 non-goals and `/docs/security/threat-model.md`:
|
|
deny-override permission grants, a data retention/deletion policy for
|
|
deprovisioned tenants, general multi-cluster orchestration in `/deploy`,
|
|
and any defense against a privileged ClickHouse/Postgres administrator —
|
|
every isolation and audit-integrity guarantee here is a structural
|
|
defense against application-layer bugs, not an operational control.
|
|
|
|
## Open questions for you to resolve
|
|
|
|
- Retention/TTL policy for the ClickHouse `logs` table — not specified yet,
|
|
deferred until storage sizing is a real concern.
|
|
- Exact OTel log schema field mapping (which OTel resource/log attributes
|
|
map to which ClickHouse columns) — Phase 0 uses a minimal subset
|
|
(timestamp, host, service, severity, message, attributes map); full
|
|
mapping deferred.
|
|
- mTLS certificate provisioning/rotation story for agents — Phase 0 will use
|
|
a static dev CA and manually issued certs; production PKI design is
|
|
out of scope here.
|