Saved, shareable multi-panel dashboards (table/line/bar/single-stat
panels via gridstack + uPlot, global + per-panel time range, JSON
export/import) and threshold/absence alert rules with an
ok/pending/firing evaluator and webhook/Slack/PagerDuty delivery.
- New /metadata component: Postgres control-plane store for dashboards,
panels, notification targets, alert rules/state, and delivery log --
see docs/phase-3-dashboard-design.md for why ClickHouse's MergeTree
family isn't a fit for this access pattern (needs real row-level
locking and read-your-writes consistency).
- api/internal/dashboards: dashboard/panel CRUD, pure -- panel query
execution stays client-side, reusing the existing /query endpoint.
- New /alerting service: rule/target CRUD, a ticker-driven evaluator
(claim-then-evaluate concurrency control, transactional-outbox
delivery, query errors and threshold zero-rows never coerced into a
false transition) and webhook/Slack/PagerDuty delivery with
retry/backoff. See docs/phase-3-alerting-design.md for the full
state-machine design and the four correctness properties it
implements.
- web: /dashboards and /alerts UIs; cli: sentryctl dashboards/alerts
list/get/apply, seeding a future Terraform provider's JSON contract.
- hack/alert-load-test: 500 rules against real ClickHouse data, real
measured results in docs/phase-3-runbook.md.
Five real bugs found by actually running this against a live stack
(documented in the runbook, not just fixed silently): a latent Phase 2
bug where ClickHouse rejected the timestamp format used for
earliest=/latest= queries; a "now" literal token injected into query
text; a GridStack/uPlot layout-timing race; JS's Date.parse being too
lenient to use as a timestamp-detection heuristic; a rule's "enabled"
field silently defaulting to false when omitted; and the evaluator's
claim-batch-size and worker-pool-concurrency defaulting to the same
value, causing 500 concurrently-due rules to take 125s to cycle through
instead of the configured 60s.
Extends the agent, ingest, storage, api, and web with Windows Event
Log/ETW sourcing and Tantivy-backed free-text search, per the approved
Phase 1 plan.
- CLAUDE.md: materialized on disk (never existed as a file before) with
a new Phase 1 "done looks like" section.
- agent: Windows Event Log (EvtSubscribe) and ETW sources, Windows
service wrapper (install/uninstall/run-service), both feature- and
target_os-gated so Linux builds/tests/clippy stay unaffected. Also
fixed two pre-existing Phase 0 clippy gaps (dead-code on
default-features-only builds, a type-inference edge case) found while
testing every feature combination properly for the first time.
UNVERIFIED on real Windows -- no Windows toolchain existed anywhere in
the build environment; flagged prominently in three places.
- proto/ingest: new record_id field, assigned once server-side in
ingest's gRPC front end so ClickHouse and Tantivy agree on the same ID
for the same record.
- storage: record_id column + bloom filter index, verified against a
live ClickHouse.
- search: new service, Tantivy index, rskafka consumer as an independent
second consumer group on the same Redpanda topic ingest already reads.
- api/web: new /search endpoint and page, sharing the query page's
result-table shape and component.
- hack/windows-fixture: sends realistic Windows-shaped data straight to
ingest, so the pipeline's handling of it is verifiable without a
Windows host.
Verified end-to-end on the live docker-compose stack: the same record_id
comes back from both /query and /search for the same log line, including
for windows-fixture's synthetic Windows Event Log data. Real bugs found
and fixed along the way: api/Dockerfile missing proto/ in its build
context, search's logs being completely silent (RUST_LOG gap), and
search/target/ missing from .gitignore/.dockerignore.
Both surfaced only by running docker compose up for real, not from review:
- ClickHouse's official image silently disables network access for the
default user unless CLICKHOUSE_USER or CLICKHOUSE_PASSWORD is set to a
genuinely non-empty value (an explicit empty password still triggers
it). Set a dev-only password across clickhouse, clickhouse-migrate,
ingest, and api in both docker-compose.yml files.
- rpk cluster health and rpk topic ... don't accept --brokers; health
checks need -X admin.hosts=... (port 9644), topic commands need
-X brokers=... (port 9092). The old script's retry loop silently
swallowed the resulting "unknown flag" error and retried forever,
which blocked ingest from ever starting.
Verified: agent -> ingest -> Redpanda -> ClickHouse -> api round-trip
confirmed with a real log line on a real host.
End-to-end log pipeline for Linux hosts, per /docs/architecture.md:
- proto: shared gRPC contract (agent <-> ingest), Go bindings checked in
- agent: Rust, musl-targeted, journald/file sourcing, RFC5424 parser,
mTLS gRPC client, no required config for the common case
- ingest: Go, single binary with --mode server|consumer|all; gRPC front
end forwards to Redpanda unchanged, consumer normalizes and
batch-writes to ClickHouse with at-least-once delivery
- storage: ClickHouse schema + a plain SQL-file migration runner
- api: minimal SELECT-only query endpoint, plain REST (not gRPC+gateway
yet -- see api/README.md)
- web: SvelteKit static SPA, one query page
- transport: Redpanda compose + topic provisioning
- cli: sentryctl ping stub
- hack/dev-certs: throwaway CA + cert generation for local mTLS
- root docker-compose.yml + docs/phase-0-runbook.md tie it together
Not yet run end-to-end against real Docker/ClickHouse/Redpanda -- see the
runbook's caveats section before relying on this working as-is.
jcoffey-dev