jcoffey-dev
b6b092c912
End-to-end log pipeline for Linux hosts, per /docs/architecture.md: - proto: shared gRPC contract (agent <-> ingest), Go bindings checked in - agent: Rust, musl-targeted, journald/file sourcing, RFC5424 parser, mTLS gRPC client, no required config for the common case - ingest: Go, single binary with --mode server|consumer|all; gRPC front end forwards to Redpanda unchanged, consumer normalizes and batch-writes to ClickHouse with at-least-once delivery - storage: ClickHouse schema + a plain SQL-file migration runner - api: minimal SELECT-only query endpoint, plain REST (not gRPC+gateway yet -- see api/README.md) - web: SvelteKit static SPA, one query page - transport: Redpanda compose + topic provisioning - cli: sentryctl ping stub - hack/dev-certs: throwaway CA + cert generation for local mTLS - root docker-compose.yml + docs/phase-0-runbook.md tie it together Not yet run end-to-end against real Docker/ClickHouse/Redpanda -- see the runbook's caveats section before relying on this working as-is.
106 lines
6.5 KiB
Markdown
106 lines
6.5 KiB
Markdown
# Sentry Architecture
|
|
|
|
> **Status:** Draft, Phase 0 scope. Written from the project constraints and
|
|
> task list at kickoff, not transcribed from a pre-existing spec. Treat as a
|
|
> starting point to correct, not a settled design — flag anything that
|
|
> doesn't match your intent before implementation leans on it further.
|
|
|
|
## Mission
|
|
|
|
Open-core, Kubernetes-native centralized logging platform. Compete with
|
|
Splunk on features; win on cost-per-GB, a modern language stack, and
|
|
multi-tenant RBAC that's actually honest about its guarantees.
|
|
|
|
## Component map
|
|
|
|
```
|
|
┌──────────┐ gRPC/mTLS ┌──────────┐ produce ┌───────────┐ consume ┌────────────┐
|
|
│ agent │ ────────────▶ │ ingest │ ──────────▶ │ Redpanda │ ────────▶ │ ingest │
|
|
│ (Rust) │ │ (Go) │ │ (Kafka API)│ │ consumer │
|
|
└──────────┘ └──────────┘ └───────────┘ │ (Go) │
|
|
└──────┬─────┘
|
|
│ batch INSERT
|
|
▼
|
|
┌────────────┐
|
|
│ ClickHouse │
|
|
└─────┬──────┘
|
|
│ SQL
|
|
┌───────────▼───────────┐
|
|
│ api (Go: gRPC+REST) │
|
|
└───────────┬───────────┘
|
|
│ REST
|
|
┌───────────▼───────────┐
|
|
│ web (SvelteKit) │
|
|
└────────────────────────┘
|
|
```
|
|
|
|
Decision (confirmed 2026-08-12): Redpanda stays in the Phase 0 path. The
|
|
`ingest` service's gRPC front end produces to Redpanda rather than writing
|
|
ClickHouse directly; a separate consumer path reads from Redpanda and batches
|
|
inserts into ClickHouse. This exercises the real transport layer from day
|
|
one instead of deferring it, and keeps Kafka credentials off the edge agent.
|
|
|
|
## Storage / query split
|
|
|
|
- **ClickHouse** is the analytical store of record for structured log data:
|
|
timestamp, host, service, severity, message, plus a `Map(String,String)`
|
|
for arbitrary structured fields. Partitioned by day, ordered by
|
|
`(service, timestamp)`.
|
|
- **Tantivy** (Phase 1) will provide full-text indexing over the `message`
|
|
field and unstructured payloads, queried out-of-band from ClickHouse and
|
|
joined by a log identifier. Not built in Phase 0.
|
|
- **Schema-on-write** using OTel semantic conventions as the default log
|
|
schema; schema-on-read fallback for unstructured/raw text that doesn't fit
|
|
the structured columns (captured via the `Map` column and/or a raw
|
|
passthrough field).
|
|
|
|
This split is not to be changed without discussion — see CLAUDE.md.
|
|
|
|
## Component responsibilities (Phase 0)
|
|
|
|
| Component | Responsibility |
|
|
|---|---|
|
|
| `agent` (Rust, musl) | Tail a log file or read journald; parse RFC 5424 syslog with raw passthrough fallback; batch; ship via gRPC/mTLS to `ingest`. |
|
|
| `proto` | Shared `.proto` contracts for the agent↔ingest gRPC service, versioned independently of either component. |
|
|
| `transport` | Redpanda docker-compose + topic provisioning scripts. No application code. |
|
|
| `ingest` (Go) | gRPC server accepting agent connections; produces normalized OTel-log-like records to Redpanda; separate consumer reads from Redpanda and batch-writes to ClickHouse. |
|
|
| `storage` | ClickHouse schema migrations + docker-compose for local/homelab. |
|
|
| `api` (Go) | gRPC + REST gateway. Phase 0: one crude `POST /query` endpoint, SELECT-only, proxying to ClickHouse. Real SPL-like query layer is Phase 2. |
|
|
| `web` (SvelteKit) | Single page: SQL text box, submit, results table. No auth, no styling polish. |
|
|
| `cli` (`sentryctl`) | Stub. Single `ping` command for now. |
|
|
| `deploy` | Helm charts, k8s manifests. Stubbed in Phase 0; docker-compose is the real local/dev path. |
|
|
|
|
## Licensing boundary
|
|
|
|
AGPLv3 for core + agents. Enterprise features (SSO, multi-tenancy,
|
|
compliance) live under `enterprise/` (not yet created — out of scope for
|
|
Phase 0) under a commercial license stub. AGPL code must never import from
|
|
`enterprise/`. No enterprise-gated code exists yet in this repo; this
|
|
section documents the boundary so nothing added later crosses it by
|
|
accident.
|
|
|
|
## Non-negotiables carried from CLAUDE.md
|
|
|
|
- Rust agent: statically linked musl, `x86_64-unknown-linux-musl` and
|
|
`aarch64-unknown-linux-musl`, no glibc runtime deps.
|
|
- Windows support (Phase 1+) via native ETW/Event Log API, not WSL.
|
|
- Every UI action maps to a documented REST/gRPC call — no UI-only logic.
|
|
- Pinned stack (see CLAUDE.md table) — no substitutions without discussion.
|
|
|
|
## Explicitly out of scope for Phase 0
|
|
|
|
Windows agent, alerting, dashboards, multi-tenancy, Tantivy full-text
|
|
search, the real SPL-like query language, enterprise module code.
|
|
|
|
## Open questions for you to resolve
|
|
|
|
- Retention/TTL policy for the ClickHouse `logs` table — not specified yet,
|
|
deferred until storage sizing is a real concern.
|
|
- Exact OTel log schema field mapping (which OTel resource/log attributes
|
|
map to which ClickHouse columns) — Phase 0 uses a minimal subset
|
|
(timestamp, host, service, severity, message, attributes map); full
|
|
mapping deferred.
|
|
- mTLS certificate provisioning/rotation story for agents — Phase 0 will use
|
|
a static dev CA and manually issued certs; production PKI design is
|
|
out of scope here.
|