jcoffey-dev
0add3c3607
Overview, architecture, layout, local-dev quickstart, and phase status for Cairn OBS. Records the Phase 4 verification gap and the unverified Windows agent path rather than presenting them as shipped.
157 lines
6.3 KiB
Markdown
157 lines
6.3 KiB
Markdown
# Cairn OBS
|
|
|
|
Open-core, Kubernetes-native log aggregation and observability. Built to match
|
|
Splunk on capability while winning on cost-per-GB, with honest multi-tenant
|
|
RBAC and a modern language stack.
|
|
|
|
Licensed **AGPLv3 in its entirety** — including `enterprise/`. See
|
|
[Licensing](#licensing).
|
|
|
|
## What it does
|
|
|
|
Logs flow from a statically-linked Rust edge agent through Redpanda into a Go
|
|
ingest pipeline, landing in ClickHouse for analytics and Tantivy for full-text
|
|
search. One query language spans both stores, compiling to a single execution
|
|
plan:
|
|
|
|
```
|
|
service=api | where status>=500 | stats count by host | sort -count
|
|
message:"connection refused" | stats count by host
|
|
```
|
|
|
|
Raw ClickHouse SQL stays available as an escape hatch and compiles to the same
|
|
IR, so performance doesn't depend on which syntax you write.
|
|
|
|
On top of that sit dashboards, an alerting evaluator with threshold and
|
|
absence rules, a CLI (`cairnobsctl`), a Terraform provider, and AI-assisted
|
|
query authoring that runs against a self-hosted Ollama model by default — no
|
|
cloud dependency.
|
|
|
|
## Architecture
|
|
|
|
| Component | Stack |
|
|
|---|---|
|
|
| Edge agent | Rust, musl static target |
|
|
| Transport | Redpanda (Kafka API) |
|
|
| Ingest / parse | Go |
|
|
| Analytical store | ClickHouse |
|
|
| Full-text index | Tantivy (Rust) |
|
|
| Control plane / API | Go, gRPC + REST gateway |
|
|
| Control-plane metadata | PostgreSQL |
|
|
| Frontend | SvelteKit + TypeScript |
|
|
| Deployment | Kubernetes Operator (kubebuilder), Helm, docker-compose |
|
|
|
|
PostgreSQL is scoped strictly to control-plane config — dashboards, panels,
|
|
alert rules and state, notification targets, delivery log — because those need
|
|
row-level locking and transactional read-modify-write that ClickHouse's
|
|
MergeTree family doesn't provide. Log data itself never touches it.
|
|
|
|
Full spec: [`docs/architecture.md`](docs/architecture.md). Read it before
|
|
changing any component; the storage/query split in particular is deliberate.
|
|
|
|
## Repository layout
|
|
|
|
Monorepo, one top-level directory per component, each with its own `README.md`,
|
|
unit tests, and Dockerfile.
|
|
|
|
```
|
|
agent/ Rust edge agent (Linux + Windows)
|
|
transport/ Redpanda topics and schemas
|
|
ingest/ Go ingest and parse pipeline
|
|
storage/ ClickHouse schema and migrations
|
|
search/ Tantivy full-text index service
|
|
api/ Control plane, query compiler, RBAC
|
|
alerting/ Rule evaluator and notification delivery
|
|
metadata/ PostgreSQL schema and migrations
|
|
web/ SvelteKit frontend
|
|
cli/ cairnobsctl
|
|
proto/ gRPC service definitions
|
|
terraform/ Terraform provider
|
|
enterprise/ SSO, multi-tenancy, per-tenant provisioning
|
|
deploy/ Helm charts and Kubernetes operator
|
|
docs/ Architecture, design docs, per-phase runbooks
|
|
hack/ Development scripts
|
|
```
|
|
|
|
`enterprise/` stays a separate module that core never imports from. Since the
|
|
Phase 6 relicensing that boundary is architectural rather than legal — it keeps
|
|
core buildable and deployable standalone, and keeps tenant resolution
|
|
server-side.
|
|
|
|
## Running locally
|
|
|
|
```sh
|
|
docker compose up
|
|
```
|
|
|
|
The web UI comes up on <http://localhost:3000> and the API on `:8080`;
|
|
alerting is on `:8081` and enterprise auth on `:8082`. The agent connects to
|
|
ingest over mTLS gRPC on `:4317`. `search` is reachable only on the compose
|
|
network — it publishes no host port.
|
|
|
|
`COMPOSE_PROFILES` in `.env` selects the query-serving binary — `single-tenant`
|
|
(default) or `enterprise` for the multi-tenant path. They're mutually
|
|
exclusive, the same choice Helm's `enterprise.enabled` flag makes for a real
|
|
cluster. Override per invocation:
|
|
|
|
```sh
|
|
COMPOSE_PROFILES=enterprise docker compose up
|
|
```
|
|
|
|
Kubernetes deployment via the Helm chart in [`deploy/`](deploy/README.md).
|
|
|
|
## Status
|
|
|
|
Built in phases; each has a runbook in `docs/` recording how it was verified.
|
|
|
|
| Phase | Scope | Status |
|
|
|---|---|---|
|
|
| 0 | Agent → Redpanda → ingest → ClickHouse, queryable end-to-end | Shipped |
|
|
| 1 | Windows Event Log + journald, SQL and full-text paths | Shipped |
|
|
| 2 | Unified query language across both stores | Shipped |
|
|
| 3 | Dashboards, alert rules, notification delivery | Shipped |
|
|
| 4 | RBAC, tenant isolation, audit logging, per-tenant ClickHouse | **In progress** |
|
|
| 5 | Frontend redesign and design system | Shipped |
|
|
| 6 | License compliance audit and remediation | Shipped |
|
|
| 7 | AI-assisted query authoring | Shipped |
|
|
|
|
**Phase 4 is not shipped.** The code is built and tested, but the environment
|
|
lost Docker and database access partway through, so only the audit-logging
|
|
guarantees were confirmed against a live database. The rest compiles and skips
|
|
cleanly when no live database is configured, but is otherwise unverified — see
|
|
the verification-status section of
|
|
[`docs/phase-4-runbook.md`](docs/phase-4-runbook.md).
|
|
|
|
The Windows agent code (`EvtSubscribe`, ETW, service registration) has never
|
|
run on real Windows — no Windows toolchain existed in the build environment.
|
|
ETW additionally sits behind a feature flag, since it needs elevated
|
|
privileges. Details in [`agent/README.md`](agent/README.md).
|
|
|
|
Terraform provider coverage is partial by necessity: dashboards and panels have
|
|
full CRUD, while alert rules and notification targets are create/destroy only,
|
|
because `alerting` exposes no `PUT /rules/{id}` or `PUT /targets/{id}` to
|
|
update against. Tenant and RBAC resources are disclosed future work —
|
|
[`terraform/README.md`](terraform/README.md) accounts for exactly what exists.
|
|
|
|
## Contributing
|
|
|
|
- Conventional commits. Every change should be a logically complete,
|
|
independently revertible unit.
|
|
- Rust: `cargo clippy --all-targets -- -D warnings` must pass.
|
|
- Go: `go vet` and `golangci-lint`, no globals for shared state.
|
|
- Every UI action must map to a documented REST/gRPC call — no UI-only logic.
|
|
The CLI and Terraform provider are first-class, not afterthoughts.
|
|
- Prefer boring, well-understood dependencies. This is infrastructure software;
|
|
operators need to trust it.
|
|
|
|
## Licensing
|
|
|
|
AGPLv3, no exceptions — see [`LICENSE`](LICENSE). `enterprise/` was under a
|
|
commercial-license stub from Phase 4 through Phase 5; Phase 6 relicensed it to
|
|
match core. The full record and its business-model consequences are in
|
|
[`docs/compliance/license-audit-report.md`](docs/compliance/license-audit-report.md).
|
|
|
|
The default AI deployment uses `qwen2.5-coder` (Apache-2.0) via Ollama,
|
|
chosen specifically to keep that license purity intact. The cloud adapter is
|
|
pluggable, opt-in, and off by default.
|