coffey-labs/cairnobs
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Phase 3: dashboards and alerting

Browse Source 
Saved, shareable multi-panel dashboards (table/line/bar/single-stat
panels via gridstack + uPlot, global + per-panel time range, JSON
export/import) and threshold/absence alert rules with an
ok/pending/firing evaluator and webhook/Slack/PagerDuty delivery.

- New /metadata component: Postgres control-plane store for dashboards,
  panels, notification targets, alert rules/state, and delivery log --
  see docs/phase-3-dashboard-design.md for why ClickHouse's MergeTree
  family isn't a fit for this access pattern (needs real row-level
  locking and read-your-writes consistency).
- api/internal/dashboards: dashboard/panel CRUD, pure -- panel query
  execution stays client-side, reusing the existing /query endpoint.
- New /alerting service: rule/target CRUD, a ticker-driven evaluator
  (claim-then-evaluate concurrency control, transactional-outbox
  delivery, query errors and threshold zero-rows never coerced into a
  false transition) and webhook/Slack/PagerDuty delivery with
  retry/backoff. See docs/phase-3-alerting-design.md for the full
  state-machine design and the four correctness properties it
  implements.
- web: /dashboards and /alerts UIs; cli: sentryctl dashboards/alerts
  list/get/apply, seeding a future Terraform provider's JSON contract.
- hack/alert-load-test: 500 rules against real ClickHouse data, real
  measured results in docs/phase-3-runbook.md.

Five real bugs found by actually running this against a live stack
(documented in the runbook, not just fixed silently): a latent Phase 2
bug where ClickHouse rejected the timestamp format used for
earliest=/latest= queries; a "now" literal token injected into query
text; a GridStack/uPlot layout-timing race; JS's Date.parse being too
lenient to use as a timestamp-detection heuristic; a rule's "enabled"
field silently defaulting to false when omitted; and the evaluator's
claim-batch-size and worker-pool-concurrency defaulting to the same
value, causing 500 concurrently-due rules to take 125s to cycle through
instead of the configured 60s.
This commit is contained in:
jcoffey-dev
2026-08-13 17:29:38 -07:00
parent fb5049a747
commit 9435115ab7
88 changed files with 7463 additions and 298 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/phase-3-alerting-design.md
+358
View File
@@ -0,0 +1,358 @@
# Alerting design
> **Status:** Design only — no code written against this yet. Task 4 of
> Phase 3. Per explicit instruction, execution stops here for review
> before task 5 (building `/alerting`) begins: the firing/resolved state
> machine and debounce behavior are easy to get subtly wrong, and this is
> the artifact to sign off on before anything is built against it. If
> implementation later reveals this design is wrong somewhere, fix this
> doc in the same change — same discipline as
> `/docs/query-language-design.md` and
> `/docs/phase-3-dashboard-design.md`.
## Why this design, in one paragraph
A rule is a saved Phase 2 query plus a condition (threshold or absence)
plus an evaluation interval plus a notification target. The genuinely
hard part isn't the data model — it's the firing/resolved state machine
under concurrent evaluation and its interaction with notification
delivery, where a naive implementation can plausibly double-fire, lose a
resolve, or silently misreport an infrastructure outage as "all clear."
This doc's state machine mirrors Prometheus Alertmanager's well-understood
`pending`/`firing` `for:` model, then adds four specific correctness
properties on top of it — not because the base model is wrong, but
because the *concurrent, at-least-once* environment a Go ticker-based
evaluator actually runs in exposes gaps a single-threaded description of
the model glosses over. Each fix below is stated as: the failure it
prevents, and the mechanism, so it's reviewable as a claim rather than
just an assertion.
## Data model
A rule is a saved query, evaluated on an interval, checked against a
condition, with a debounce before it's allowed to notify, and one
notification target it notifies. Two condition types:
- **`threshold`**: the query's result must resolve to exactly one row;
the value in that row's first column is compared against
`threshold_value` via `comparator`.
- **`absence`**: the query returned zero rows. The evaluation *window*
is not a separate rule field — it's whatever `earliest=`/`latest=` the
rule's own saved query already expresses (e.g. `service=payments
severity=ERROR earliest=-5m`), reusing Phase 2's time-range syntax
rather than inventing a second one.
```sql
CREATE TABLE notification_targets (
id UUID PRIMARY KEY,
tenant_id TEXT NOT NULL DEFAULT 'default',
name TEXT NOT NULL,
kind TEXT NOT NULL CHECK (kind IN ('webhook', 'slack', 'pagerduty')),
webhook_url TEXT NOT NULL,
payload_template TEXT, -- generic ("webhook") targets only; NULL for slack/pagerduty
headers JSONB NOT NULL DEFAULT '{}',
secret TEXT, -- PagerDuty routing key / generic-webhook HMAC secret -- plaintext, see "Known gaps" below
created_by TEXT NOT NULL DEFAULT 'anonymous',
created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
updated_at TIMESTAMPTZ NOT NULL DEFAULT now()
);
CREATE TABLE alert_rules (
id UUID PRIMARY KEY,
tenant_id TEXT NOT NULL DEFAULT 'default',
name TEXT NOT NULL,
description TEXT NOT NULL DEFAULT '',
query TEXT NOT NULL,
query_language TEXT NOT NULL DEFAULT '',
condition_type TEXT NOT NULL CHECK (condition_type IN ('threshold', 'absence')),
comparator TEXT CHECK (comparator IN ('gt', 'gte', 'lt', 'lte', 'eq', 'ne')), -- NULL for absence
threshold_value DOUBLE PRECISION, -- NULL for absence
eval_interval_seconds INT NOT NULL CHECK (eval_interval_seconds >= 30),
for_minutes INT NOT NULL DEFAULT 0, -- 0 = fire on first true evaluation
renotify_interval_minutes INT, -- NULL = notify once per firing episode
notification_target_id UUID NOT NULL REFERENCES notification_targets(id),
enabled BOOLEAN NOT NULL DEFAULT true,
created_by TEXT NOT NULL DEFAULT 'anonymous',
created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
updated_at TIMESTAMPTZ NOT NULL DEFAULT now()
);
CREATE TABLE alert_state (
rule_id UUID PRIMARY KEY REFERENCES alert_rules(id) ON DELETE CASCADE,
state TEXT NOT NULL DEFAULT 'ok' CHECK (state IN ('ok', 'pending', 'firing')),
condition_true_since TIMESTAMPTZ,
fired_at TIMESTAMPTZ,
last_notified_at TIMESTAMPTZ,
last_evaluated_at TIMESTAMPTZ,
last_eval_status TEXT NOT NULL DEFAULT 'ok' CHECK (last_eval_status IN ('ok', 'error')),
last_error TEXT,
last_value DOUBLE PRECISION,
consecutive_errors INT NOT NULL DEFAULT 0,
next_eval_at TIMESTAMPTZ NOT NULL, -- the claim column, see "Concurrency" below
claimed_at TIMESTAMPTZ
);
CREATE TABLE delivery_log (
id BIGSERIAL PRIMARY KEY,
rule_id UUID NOT NULL REFERENCES alert_rules(id) ON DELETE CASCADE,
notification_target_id UUID NOT NULL REFERENCES notification_targets(id),
event_type TEXT NOT NULL CHECK (event_type IN ('firing', 'resolved')),
status TEXT NOT NULL DEFAULT 'pending' CHECK (status IN ('pending', 'sent', 'failed', 'retrying')),
attempt_count INT NOT NULL DEFAULT 0,
max_attempts INT NOT NULL DEFAULT 5,
next_attempt_at TIMESTAMPTZ, -- the delivery worker's own claim key
last_attempt_at TIMESTAMPTZ,
last_error TEXT,
response_status INT,
payload JSONB NOT NULL, -- the actual rendered payload -- needed to debug template issues
created_at TIMESTAMPTZ NOT NULL DEFAULT now()
);
CREATE INDEX ON delivery_log (rule_id, created_at DESC); -- per-rule delivery log UI
CREATE INDEX ON delivery_log (status, next_attempt_at) WHERE status IN ('pending', 'retrying'); -- delivery worker's claim query
```
**`alert_state` must be inserted in the same transaction as its owning
`alert_rules` row**, `state = 'ok'`, `next_eval_at = now()` (evaluate
immediately on creation). A rule with no `alert_state` row is silently
never picked up by the claim query below — worth stating loudly since
it's the kind of thing that "just works" in every test that remembers to
seed both rows and then silently doesn't in production the one time it's
forgotten.
Single notification target per rule for MVP. Multi-target (e.g. page AND
Slack) is a disclosed, straightforward future extension — a join table,
not a redesign.
## Notification delivery: generic webhook as the base primitive
All three `notification_targets.kind` values ultimately do the same
thing — an HTTP POST to `webhook_url` — with kind-specific *payload
formatting only*, per the explicit requirement that this stays pluggable
rather than accumulating per-vendor delivery logic:
- **`webhook`**: `payload_template` is a Go `text/template` string,
rendered against the firing/resolved event (rule name, condition,
current value, timestamp). No template = a sane default JSON shape.
- **`slack`**: fixed formatter producing Slack's incoming-webhook shape
(`{"text": "..."}`), no user template. `payload_template` is ignored
for this kind (kept nullable in the schema rather than removed, so a
target's `kind` can be changed later without a payload column migration).
- **`pagerduty`**: fixed formatter producing PagerDuty's Events API v2
shape (`{"routing_key": secret, "event_action": "trigger"|"resolve",
"payload": {...}}`) — `secret` here is the PagerDuty integration
routing key, not a delivery credential in the auth sense.
All three go through the exact same HTTP POST + retry/backoff mechanism
in `internal/delivery/webhook.go`; `slack.go`/`pagerduty.go` are payload
builders only, never their own delivery path.
## The state machine
States per rule: `ok` → `pending` → `firing`, tracked in `alert_state`.
Every evaluation produces one of three outcomes — `condition_true`,
`condition_false`, or `error` — and error is handled entirely separately
from the other two (see fix 3).
**On `condition_true`:**
- `ok` → `pending`: set `condition_true_since = now()`. No notification.
- `pending` → `firing`, once `now() - condition_true_since >=
for_minutes`: send a **firing** notification, set `fired_at =
last_notified_at = now()`. `for_minutes = 0` means this transition
happens on the very first true evaluation.
- `pending`, not yet past `for_minutes`: no transition, no notification.
- `firing` → stays `firing`: silent, unless `renotify_interval_minutes`
is set and `now() - last_notified_at >= renotify_interval_minutes`, in
which case re-send **firing** and update `last_notified_at`. Default
(`NULL`) is "notify once per firing episode, stay silent until
resolved" — stated explicitly since it's exactly the kind of default an
implementer would otherwise have to guess.
**On `condition_false`:**
- `ok` → no-op.
- `pending` → `ok`: clear `condition_true_since`. **No notification** —
this was a blip inside the debounce window, not a real alert. This is
deliberate, not an oversight: it's the entire reason `for_minutes`
exists.
- `firing` → `ok`: clear `condition_true_since`/`fired_at`, send a
**resolved** notification.
`condition_true_since` is a **wall-clock timestamp**, not a
consecutive-evaluation counter. This is what makes the debounce survive
evaluator restarts/downtime correctly: if the evaluator is down for part
of a rule's `for_minutes` window and comes back, wall-clock math
correctly resumes toward firing where it left off, while a counter would
have silently lost that progress and restarted the count. Worth
defending explicitly here since a counter looks like the "simpler"
choice and a future contributor might "simplify" it into one.
**Disclosed non-goal: no debounce on the way down.** `firing` → `ok`
happens on a single false evaluation — there's no symmetric "stay firing
for N more minutes" hold (Grafana calls this "keep firing for"). A
condition that flickers right at the threshold produces a firing/resolved
notification pair per flicker. Future work, not solved in Phase 3.
### Four correctness properties, and the concrete failure each one prevents
**1. Concurrent evaluation of the same rule (claim-then-evaluate).**
A worker-pool evaluator is required at the scale task 8 targets (~500
rules @ 60s ≈ 8+ evaluations/sec sustained). If a single evaluation's
round-trip to `api`'s `/query` ever takes longer than the rule's own
interval, the next scheduler tick can pick the *same* rule again while
the first evaluation is still in flight — two goroutines then read-
modify-write the same `alert_state` row concurrently. Depending on
timing, that produces either a duplicated firing notification (both see
`pending`, both compute "elapsed ≥ for_minutes", both fire) or a lost
resolve (a slow evaluation's stale write clobbers a faster one).
Fix: atomically claim due rules **before** the slow network call starts:
```sql
UPDATE alert_state
SET next_eval_at = now() + (eval_interval_seconds || ' seconds')::interval,
claimed_at = now()
FROM alert_rules
WHERE alert_state.rule_id = alert_rules.id
AND alert_state.next_eval_at <= now()
AND alert_rules.enabled
LIMIT $batch_size
RETURNING alert_state.rule_id, alert_state.state, alert_state.condition_true_since, ...
```
`next_eval_at` is bumped *before* the `/query` HTTP call ever starts, so
a second scheduler tick can't re-select the same rule while the first is
still running. The `/query` call itself happens **outside** any database
transaction — never hold a Postgres connection open across a network
call to another service. A second, short transaction applies the
resulting state transition once the query result is known.
This same claim pattern is what makes horizontal evaluator replicas safe
to add later (a named Phase 4+ path, see task 8) without redesigning the
state model — each replica's claim query naturally excludes rows another
replica already claimed. Worth stating positively: this is the one place
in Phase 3 that's actively built *for* that future need, not just
avoiding a trap.
**2. Notification loss/duplication on crash (transactional outbox).**
If the state transition and the webhook POST happen as separate,
sequentially-ordered steps, a crash between them is unrecoverable in one
direction or the other: crash after a successful POST but before the DB
commit → next evaluation replays the transition and double-fires; crash
after commit but before the POST → the notification is silently owed
forever with no record that it was ever decided.
Fix: the state transition and `INSERT INTO delivery_log (..., status =
'pending')` happen in the **same database transaction** — "we decided to
notify" becomes durable exactly once, atomically with the state change
itself, before any network call to a notification target is attempted. A
**separate** delivery worker polls `delivery_log WHERE status IN
('pending', 'retrying') AND next_attempt_at <= now()` (same claim
pattern as fix 1), performs the actual HTTP POST, and updates
`status`/`attempt_count`/`last_error`. This decouples "did we decide to
notify" (transactionally certain) from "did the HTTP call succeed"
(best-effort with retries) — which is exactly what task 5's retry-with-
backoff requirement needs anyway, so this isn't extra machinery bolted
on for correctness's sake, it's the same piece of work.
**3. Query errors must never be treated as `condition_false`.**
The state machine above only describes `condition_true`/
`condition_false` — what happens when the `/query` call to `api` itself
fails (timeout, ClickHouse down, a 5xx)? Coercing an error to "false" is
the tempting default and the worst possible one: a `firing` alert would
silently auto-resolve and go quiet at precisely the moment something is
broken enough that the query can't even run. Coercing to "true" is
equally wrong the other way (spurious pages on transient infra hiccups).
Fix: evaluation outcome is modeled as a three-way result, and an `error`
outcome **never transitions `state`** at all. It only updates
`alert_state.last_evaluated_at`, `last_eval_status = 'error'`,
`last_error`, and increments `consecutive_errors`. This must be explicit
in the implementation, not left to the "obvious" path — the obvious
path (an error propagating into whatever boolean the rest of the
function expects) is also the wrong one.
**4. Zero rows on a `threshold` rule is an error, not a `0`.**
`stats count by host` can legitimately return zero rows for a threshold
rule (nothing matched in the window) — that's different in kind from an
`absence` rule, where zero rows *is* the signal being checked for. If a
threshold evaluation coerces "no rows" to a scalar `0` for comparison,
`count > 100` silently reports "definitely fine" in exactly the case
where the honest answer is "the query returned nothing, which might mean
nothing happened, or might mean something upstream is broken" — often
the more alarming possibility, not the safe one.
Fix, stated as an explicit rule rather than left implicit: `threshold`
evaluation requires **exactly one** result row (first row, first numeric
column, per the dashboard/query design's single-row precedent). Zero
rows — or more than one — is treated the same as fix 3's evaluation
error, not coerced to a value. Named non-goal alongside "no per-group
alerting": a threshold rule's query must resolve to a scalar.
## Evaluator architecture
A single Go process (`/alerting`), ticker-driven — **not** a workflow
engine, per the explicit instruction not to reach for one at this stage.
Loop shape:
1. Every few seconds, run the claim query (fix 1) for due, enabled rules
up to a bounded batch size.
2. Dispatch claimed rules to a bounded worker pool (goroutines).
3. Each worker: `POST /query` against `api` (reusing the existing
endpoint — the same precedent `sentryctl query` already set, never a
second query-execution path), evaluate the condition, run the state
transition (fix 3/4 aware) and, if applicable, the fix-2 transactional
outbox insert, in one short DB transaction.
4. A separate delivery-worker loop claims and sends `delivery_log` rows
independently of the evaluation loop.
`alerting` is therefore hard-dependent on `api` being reachable — if
`api`/ClickHouse is down, every due rule's evaluation records an error
(fix 3), not a false resolve. This is documented behavior, not an
accident: you can't trust a condition you can't evaluate. `alerting`'s
docker-compose entry depends on `api`'s healthcheck (added in task 3)
for this reason.
## Component boundary
`/alerting` (new top-level Go service, own `go.mod`) owns rule CRUD,
notification-target CRUD, the delivery-log read endpoint, the evaluator
loop, and the delivery worker — all four pieces of "alerting" in one
service, since they share the same Postgres tables and the same
claim-based concurrency pattern. It does **not** import `api`'s
`querylang` package or talk to ClickHouse/Tantivy directly; it only
calls `api`'s `POST /query` over HTTP, exactly like `sentryctl query`
and the web UI's dashboard panels already do. `web` gets a second
backend base URL (`alerting`'s) alongside the existing `api` one.
## Known gaps (named, not hidden)
- **`notification_targets.secret` is stored plaintext** in Postgres —
the same posture `dashboard-design.md` already named for that domain.
Becomes both an enterprise-tier (secrets/KMS) and a multi-tenancy
(per-tenant secret isolation) concern later; naming it now avoids it
being "discovered" as a surprise security-review finding during a
future push.
- **CORS on `alerting`'s HTTP API is wide open**, matching `api`'s
existing no-auth-system posture from Phases 0–2. Not a new problem
Phase 3 introduces, just a second surface that inherits the same one.
- **No per-group/multi-row threshold alerting** (fix 4's scope decision)
and **no resolved-side debounce** (state-machine section) are both
named future work.
- **Single shared Postgres role** for `api` and `alerting`, same as
`dashboard-design.md`'s note — fine for Phase 3, a named Phase 4 item
once real auth exists.
## Load-testing plan (task 8, not run yet)
Seed ~500 rules via `alerting`'s own create API (not a direct DB insert
— exercise the real code path) at 60-second intervals, against a real
ClickHouse dataset (reusing `hack/benchmark-fixture`'s generator).
Measure: drift between `alert_state.next_eval_at` and the actual claim
timestamp under sustained load, `consecutive_errors`/`last_eval_status`
distribution (did the evaluator start erroring under load, as opposed to
just running slow), and `delivery_log.attempt_count`/`status`
distribution. Document real numbers, not projections — matching every
prior phase's benchmark discipline — and name what would need to change
for materially larger rule counts (horizontal evaluator replicas
partitioning by rule-id hash, which fix 1's claim design already makes
safe to add without a state-model change; moving off a single-process
ticker) as explicit Phase 4+ scope, not solved here.
docs/phase-3-dashboard-design.md
+288
View File
@@ -0,0 +1,288 @@
# Dashboard design
> **Status:** Approved, in progress. Task 2 of Phase 3 — see `/CLAUDE.md`'s
> "What done looks like for Phase 3" section for the exit criteria this is
> built against. Task 3 (dashboard CRUD API + web UI) implements this
> doc; if implementation reveals this design is wrong somewhere, fix this
> doc in the same change, don't let them drift apart — same discipline as
> `/docs/query-language-design.md`.
## Why this design, in one paragraph
A dashboard is a named collection of panels, each wrapping one saved
Phase 2 query plus a visualization type and a grid position. That's a
straightforward data model; the one real design decision here is *where
it's stored*. Dashboards and their panels need real create/update/delete
semantics with immediate read-your-writes consistency for a UI —
ClickHouse's MergeTree family isn't built for that access pattern (see
"Why not ClickHouse" below). This doc also covers the mechanism that
makes one query language work for both ad hoc search and reusable
dashboard panels: injecting the dashboard's time range into a stored
query at execution time, rather than baking a fixed time range into the
saved query itself.
## Why not ClickHouse: PostgreSQL for control-plane config
This phase adds PostgreSQL as a new pinned-stack component — flagged and
confirmed with the project owner before implementation, per CLAUDE.md's
"ask before... making an architectural decision not already specified in
`/docs/architecture.md`." Scope is strictly control-plane config
(dashboards, panels, and — per `/docs/phase-3-alerting-design.md` —
notification targets, alert rules, alert state, delivery log). Log data
itself is untouched: ClickHouse and Tantivy remain the only stores for
`logs`.
Two concrete reasons ClickHouse doesn't fit this job, not just "it feels
risky":
1. **No row-level locking primitive.** The alerting evaluator (see the
alerting design doc) needs to atomically claim a due rule so a second
evaluation tick can't touch it concurrently — a `SELECT ... FOR UPDATE
SKIP LOCKED` operation. `ReplacingMergeTree` + `FINAL` gives
eventually-consistent "latest write wins," not concurrency control.
That's a missing primitive, not a tuning problem.
2. **Read-your-writes consistency for a UI.** A user editing a dashboard
expects to immediately see their own edit reflected back. ClickHouse's
MergeTree engines don't guarantee this the way a transactional
database does without extra machinery (`FINAL`, careful ordering) that
amounts to reimplementing what Postgres already provides natively.
Scope of the addition: new top-level `/metadata` component (naming
mirrors existing precedent — transport=Redpanda, storage=ClickHouse,
search=Tantivy → metadata=Postgres), `postgres:16-alpine` in
docker-compose, `jackc/pgx/v5` as the Go driver in `api` and the new
`alerting` service. No ORM, no `sqlc` — hand-written SQL per store
package, matching the project's existing avoidance of query-generation
machinery (no cobra, no golang-migrate, nothing code-generated from SQL
anywhere in the repo today).
### Migrations: mirror `/storage/migrate.sh`, not a framework
`storage/README.md`'s objection to `golang-migrate` ("premature machinery"
for what's being built) is general, not specific to ClickHouse's
HTTP-interface constraint — six new tables in one change doesn't meet the
bar that would justify revisiting that call. `/metadata/migrate.sh`
mirrors it: bash + `psql -v ON_ERROR_STOP=1 -f <file>` per migration, a
`schema_migrations` tracking table, one DDL object per file (kept for
repo-wide consistency of what a migration "version" means, even though
Postgres itself supports multi-statement transactions unlike ClickHouse's
HTTP interface), applied by a one-shot init container
(`metadata-migrate`) that other services `depends_on: condition:
service_completed_successfully` — same shape as `clickhouse-migrate`.
```
metadata/
README.md Dockerfile migrate.sh docker-compose.yml
migrations/
0001_create_dashboards.sql
0002_create_dashboard_panels.sql
0003_create_notification_targets.sql (alerting design doc)
0004_create_alert_rules.sql (alerting design doc)
0005_create_alert_state.sql (alerting design doc)
0006_create_delivery_log.sql (alerting design doc)
```
All six tables live in one migrations directory / one Postgres database,
even though `api` (dashboards) and `alerting` (rules/targets/state/log)
are the two services that own different tables within it — mirrors how
ClickHouse already hosts tables conceptually "owned" by different
components (`logs` via ingest's consumer, `schema_migrations` via the
migration runner itself) inside one physical database. No shared Go
store code between `api` and `alerting` — each owns hand-written SQL for
the tables it's responsible for; nothing is shared across service
`internal/` trees in this repo except `/proto`, and that precedent holds
here too.
## Schema
All tables carry `tenant_id TEXT NOT NULL DEFAULT 'default'` (populated,
unenforced — see "Not painting Phase 4 into a corner" below) and
`created_by TEXT NOT NULL DEFAULT 'anonymous'` for the same reason.
`updated_at` is app-managed on every UPDATE, no PL/pgSQL trigger —
consistent with the project's avoidance of database-side procedural code.
```sql
CREATE TABLE dashboards (
id UUID PRIMARY KEY,
tenant_id TEXT NOT NULL DEFAULT 'default',
name TEXT NOT NULL,
description TEXT NOT NULL DEFAULT '',
default_earliest TEXT NOT NULL DEFAULT '-1h',
default_latest TEXT NOT NULL DEFAULT 'now',
created_by TEXT NOT NULL DEFAULT 'anonymous',
created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
updated_at TIMESTAMPTZ NOT NULL DEFAULT now()
);
CREATE TABLE dashboard_panels (
id UUID PRIMARY KEY,
dashboard_id UUID NOT NULL REFERENCES dashboards(id) ON DELETE CASCADE,
title TEXT NOT NULL DEFAULT '',
query TEXT NOT NULL, -- pipe syntax only, no earliest=/latest=
query_language TEXT NOT NULL DEFAULT '', -- mirrors queryRequest.Language: ''|sql|spl
viz_type TEXT NOT NULL CHECK (viz_type IN ('table','line','bar','single_stat','top_n')),
viz_config JSONB NOT NULL DEFAULT '{}', -- e.g. {"x_column":"timestamp","series_column":"host"}
position_x INT NOT NULL,
position_y INT NOT NULL,
width INT NOT NULL,
height INT NOT NULL,
earliest_override TEXT NULL, -- NULL = inherit dashboard default
latest_override TEXT NULL,
sort_order INT NOT NULL DEFAULT 0, -- deterministic export ordering
created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
updated_at TIMESTAMPTZ NOT NULL DEFAULT now()
);
```
`query_language` deliberately excludes `'sql'` from being usable in
practice for panels (enforced at the API layer, not the DB CHECK
constraint — see "Raw-SQL panels out of scope" below): the column exists
because it mirrors the existing `/query` request shape exactly, not
because raw SQL panels are supported yet.
`position_x/y/width/height` map 1:1 to the web UI's grid layout library's
own `x/y/w/h` units — no translation layer between stored position and
rendered position.
IDs are app-generated UUIDs (`google/uuid`, already an `api` dependency),
assigned server-side once, matching how `ingest` assigns `record_id` —
not `gen_random_uuid()` via a Postgres extension, to keep ID generation
in one place (Go) rather than split between the app and the database.
## Time-range mechanics
A dashboard has a default time range (`default_earliest`/
`default_latest`, in the query language's own relative syntax, e.g.
`-1h`/`now`). Each panel can override it. Critically, **panel query
strings never contain `earliest=`/`latest=` themselves** — the query a
user writes for a panel is time-range-agnostic (e.g. `service=api |
where status>=500 | stats count by host`), and the effective time range
is resolved and injected at execution time:
```
effective_earliest = panel.earliest_override ?? dashboard.default_earliest
effective_latest = panel.latest_override ?? dashboard.default_latest
executed_query = "earliest={effective_earliest} latest={effective_latest} " + panel.query
```
This works because `earliest=`/`latest=` are ordinary `base_search` terms
in Phase 2's grammar (`term := ... | "earliest" "=" time_expr | "latest"
"=" time_expr`), implicitly AND'd with whatever else is in the query,
and — like every term in `bool_expr` — order-independent. Prepending them
is syntactically identical to a user having typed them first.
**`"now"` is a UI-only sentinel, never injected literally.** The default
`default_latest` value shown in the picker is the human-readable string
`"now"`, but `time_expr` only accepts a quoted absolute timestamp or a
`"-N unit"` relative offset — there is no `now` token in the grammar.
Injecting `latest=now` produces a real compile error (`expected a quoted
absolute timestamp or a relative offset`), found by actually running a
dashboard panel against the live stack, not a hypothetical. The fix:
when the effective latest value is `"now"` (or empty), the `latest=`
clause is omitted from the injected query entirely — which is exactly
what the query language already does to mean "no upper bound" (see
`planner.go`'s handling of an absent `latest` term). `earliest=` has no
equivalent sentinel; a dashboard's `default_earliest` is always a real
`time_expr` value.
**This injection only works for pipe-syntax queries.** A raw-SQL query
(Phase 2's `SELECT ...` escape hatch) has no equivalent injection point —
there's no reliable way to splice a time bound into arbitrary SQL without
parsing it, which is exactly the work the raw-SQL escape hatch was
designed to avoid (see `/docs/query-language-design.md`'s "not parsed,
wrapped as opaque IR"). **Raw-SQL dashboard panels are out of scope for
Phase 3** — a disclosed non-goal, not a silent gap. `dashboard_panels.
query_language` exists in the schema for symmetry with the `/query`
request shape, but the dashboard API layer (task 3) rejects `'sql'` on
create/update with a clear error rather than accepting it and having the
time-range picker silently do nothing.
## Panel execution: client-side, not server-side
`GET /dashboards/{id}` returns panel *definitions* only (query, viz_type,
position, overrides) — it does not execute any queries. The web UI
resolves each panel's effective time range and calls `POST /query`
per panel directly, exactly the same endpoint the root query page already
uses. This keeps `api/internal/dashboards` pure CRUD with zero
query-execution code of its own, consistent with the project's standing
rule that nothing duplicates `querylang`'s execution path (the same
reason `sentryctl query` calls `/query` over HTTP instead of importing
`querylang` internals). It also means panels load and error
independently in the UI — one broken panel query doesn't fail the whole
dashboard.
## Visualization types
- **`table`**: renders `{columns, rows}` directly via (an extended)
`ResultsTable.svelte`. No `viz_config` needed.
- **`line`** / **`bar`**: `viz_config` names which result column is the
x-axis/category (`x_column`, typically `timestamp` or a `stats ... by`
grouping column) and which is plotted (`series_column` for
multi-series, `value_column` for the plotted value). Rendered via
uPlot (see below).
- **`single_stat`**: takes the first row's first numeric column, no
`viz_config` needed for MVP (a labeled big number).
- **`top_n`**: a table sorted/limited view — for MVP this is really
"table, but the query already did `sort`/`head`," so it renders
through the same table path as `table` with different framing in the
UI; no separate execution logic.
### New frontend dependencies
`web/package.json` currently has zero runtime dependencies. Two are
added this phase, both confirmed with the project owner before landing:
- **gridstack.js** — panel grid layout with drag-and-drop/resize.
Vanilla JS, framework-agnostic (wrapped in a thin Svelte component),
well-established, used by real dashboard products. Pre-approved
("a lightweight grid library is fine, don't build drag-and-drop from
scratch") — listed here so the concrete choice is visible in the
design record, not just in a `package.json` diff.
- **uPlot** — line/bar chart rendering. ~45KB, canvas-based, fast, no
heavy transitive dependency tree. Chosen over Chart.js (heavier) and a
hand-rolled D3/SVG approach (correct axes/scales/tooltips is a lot of
code to own well for something a library already does correctly).
Table, single-stat, and top-N panels need neither dependency.
## Export / import
`GET /dashboards/{id}/export` returns the dashboard and its panels
marshaled as one JSON document (the same `Dashboard`/`Panel` Go structs
used internally, not a separate export-specific shape). `POST
/dashboards/import` accepts that same shape and creates a new dashboard
from it (new IDs assigned, not a literal replay of the source IDs, so
importing into a different environment doesn't collide). This is
deliberately the *same* JSON contract `sentryctl dashboards apply` (task
7) consumes — "exportable/importable, Terraform-friendly" isn't a
separate design, it's one JSON shape used from two call sites (web
export button, CLI apply).
## Not painting Phase 4 into a corner
- **`tenant_id`** on every table, populated but unenforced — Phase 4's
multi-tenancy retrofit needs a partitioning/enforcement layer, not a
schema migration + backfill on every table.
- **`created_by`** on every table, defaulted to `'anonymous'` since no
auth exists yet — same reasoning, cheaper to add the column now than
during Phase 4 when real identity is also landing.
- **"Shareable" currently means "shareable within the single trusted
deployment."** There is no access-control model at all yet — not
tenant isolation, not even within-tenant per-dashboard permissions.
Stating this plainly here so it doesn't read as more solved than it is
once multi-tenancy is on the table.
- CORS stays wide open (`Access-Control-Allow-Origin: *`) on `api`,
matching the existing Phase 0–2 posture. Adding dashboards doesn't
change this tradeoff, just widens the same already-accepted surface
area — noted so it's not mistaken for a new gap introduced this phase.
## Where this lives
`api/internal/dashboards/` (`handler.go`, `store.go`, `types.go`, plus
tests) — mirrors `api/internal/queryapi`'s existing `Handler`/`Store`
shape. Requires one small prerequisite refactor to existing code:
`queryapi.Handler.Routes()` currently builds its own `http.NewServeMux()`
and applies CORS in one shot; with a second handler package, both need to
register onto one shared mux in `main.go`, with CORS applied once at the
top. `queryapi.Handler` gains a `RegisterRoutes(mux *http.ServeMux)`
method in place of `Routes()`.
docs/phase-3-runbook.md
+260
View File
@@ -0,0 +1,260 @@
# Phase 3 runbook
Extends `/docs/phase-0-runbook.md` through `/docs/phase-2-runbook.md`
with dashboards and alerting. Read those first — this assumes the stack
already works. Phase 3 adds a new Postgres-backed control-plane
(`/metadata`), a new dashboard CRUD surface in `/api`, and a new
top-level service (`/alerting`) plus its web UI and CLI subcommands.
## What's actually been verified
Every claim below was checked against the live stack, not asserted —
same discipline as every prior phase's runbook. This phase's real
findings (five of them, not hypothetical) are called out inline where
they were caught, matching the project's standing "actually run it"
rule: a passing test suite and a working feature are not the same claim.
## 1. Bring up the stack
```sh
docker compose up -d --build
```
New services beyond Phase 0–2: `metadata-postgres`, `metadata-migrate`
(one-shot, applies `/metadata/migrations/*.sql`), `alerting` (port 8081).
`api` now depends on `metadata-migrate` and exposes a `-healthcheck`
self-check mode (distroless, no shell/wget — see `api/cmd/api/main.go`)
so `alerting`'s `depends_on: api: condition: service_healthy` means
something real. Confirm everything is healthy:
```sh
docker compose ps
# api, alerting, metadata-postgres, clickhouse, redpanda should all show (healthy)
```
## 2. Dashboards
Generate a fixture dataset (reuses Phase 2's `hack/benchmark-fixture`):
```sh
cd hack/benchmark-fixture
go run . --count 500000
```
Create a dashboard and a couple of panels, either through the web UI
(`http://localhost:3000/dashboards` → "+ Create" → "+ Add panel") or via
`sentryctl`:
```sh
sentryctl dashboards apply my-dashboard.json # shape = GET /dashboards/{id}/export
```
**Verified live**: a table panel (`severity=INFO | head 10`) and a bar
chart panel (`| stats count by host`, viz_type `bar`) both render
correctly against real data, including drag/resize via the gridstack
grid.
### Real bugs this caught
Building and actually loading the dashboard UI against a live stack
(not just unit tests) found four real bugs, all now fixed:
1. **A Phase 2 bug, only surfaced now**: `earliest=`/`latest=` time-range
queries never actually worked against live ClickHouse. The executor
formatted `TimeRange` bounds with `time.RFC3339Nano`
(`2026-08-12T20:17:40.223505479Z`), but ClickHouse's *implicit*
string→`DateTime64` cast for a column comparison is strict and wants
`'YYYY-MM-DD HH:MM:SS[.fractional]'` — no `T`/`Z`. Failed with `code:
53, Cannot convert string ... to type DateTime64(9, 'UTC')`. Nothing
in Phase 2's own runbook queries or unit tests happened to exercise a
relative time filter live; the unit test asserted the broken format
without ever asking real ClickHouse whether it was valid. Fixed in
`api/internal/querylang/executor/sql.go` (`formatClickHouseDateTime64`).
2. **`"now"` as a literal query token**: the dashboard time-range picker
injected `latest=now` verbatim; the query language has no `now`
keyword (only quoted absolute timestamps or `-N unit` relative
offsets). Fixed by treating `"now"` as a UI-only sentinel that omits
the `latest=` clause entirely (`web/src/lib/api.ts`'s
`injectTimeRange`).
3. **A layout-timing race**: GridStack/uPlot measured a panel's width
before the grid had finished laying out, producing a 74px-wide chart
canvas in a 540px container. Fixed with a `ResizeObserver` in
`PanelViz.svelte` that re-renders whenever the container's actual
size changes (also fixes chart sizing after a manual panel resize).
4. **`Date.parse` is too lenient to use as a "is this a timestamp"
check**: `Date.parse("host-06")` returns a real (bogus) timestamp
rather than `NaN`, which silently misrouted a categorical `stats
count by host` column onto a numeric time axis and rendered
unreadable giant tick labels instead of host names. Fixed by requiring
a strict ISO-8601 prefix match before attempting `Date.parse`
(`PanelViz.svelte`'s `isoTimestampPrefix`).
## 3. Alerting
Bring up a local webhook receiver for testing (no real Slack/PagerDuty
needed):
```sh
docker run -d --name sentry-webhook-sink --network sentry_default \
-p 9099:9099 -v $(pwd)/hack/webhook-sink:/src -w /src golang:1.25-alpine go run .
```
Create a notification target and a rule, either via the web UI
(`http://localhost:3000/alerts/new`) or directly:
```sh
curl -X POST http://localhost:8081/targets -H 'Content-Type: application/json' -d '{
"name": "local sink", "kind": "webhook", "webhook_url": "http://sentry-webhook-sink:9099/"
}'
curl -X POST http://localhost:8081/rules -H 'Content-Type: application/json' -d '{
"name": "always fires", "query": "SELECT 1", "query_language": "sql",
"condition_type": "threshold", "comparator": "gte", "threshold_value": 1,
"eval_interval_seconds": 30, "for_minutes": 0,
"notification_target_id": "<target id>"
}'
```
**Verified live, through both curl and the actual web UI** (created a
rule via the `/alerts/new` form, watched it transition in the browser):
the rule transitions `ok` → `firing` on its first evaluation
(`for_minutes: 0`), the delivery log shows `firing / sent / 200`, and
`docker logs sentry-webhook-sink` shows the real received payload.
Also verified live: a threshold rule whose query returns **zero rows**
records `last_eval_status: "error"` with the exact expected message
(`"threshold rule query returned 0 rows, want exactly 1"`) and leaves
`state` untouched at `"ok"` — confirming fix 3/4 from
`/docs/phase-3-alerting-design.md` hold in practice, not just in the unit
tests that were written against them.
### Real bugs this caught
5. **`enabled` silently defaulted to `false`.** `rulestore.Rule.Enabled`
is a plain `bool`; a create request that simply didn't mention
`"enabled"` decoded it to Go's zero value (`false`) rather than the
intended "enabled by default." A rule created this way was silently
dead on arrival — never picked up by the evaluator's claim query, no
error anywhere. Fixed in `alerting/internal/httpapi/handler.go` with a
`createRuleRequest` wrapper using `Enabled *bool`: `nil` (omitted)
means enabled; only an explicit `"enabled": false` creates a disabled
rule. Caught by actually creating a rule through the endpoint and
checking the response, not by inspection.
## 4. Load test (task 8)
```sh
cd hack/alert-load-test
go run . --rule-count 500 --eval-interval-seconds 60 --duration 3m30s
```
Seeds 500 rules via `alerting`'s real create API (not a direct DB
insert), each querying a different host's count over the last minute
against the 500,000-row fixture dataset from §2, with an unreachably
high threshold so rules stay `ok` — isolating evaluator/ClickHouse
scheduling throughput from delivery-worker load.
**First run, before a fix**: every one of 500 rules showed
`consecutive_errors > 0`. Root cause: the load-test tool's own query
(`host=host-01`, unquoted) hit a real, pre-existing Phase 2 lexer quirk —
an unquoted comparison value containing a literal `-` fails to parse
(`unexpected MINUS after query`). Fixed by quoting the value
(`host="host-01"`) in the load-test tool itself; not a bug worth chasing
in the query language for this runbook.
**Second run, after that fix — a real, significant finding**: zero
errors, but the observed inter-evaluation interval was a suspiciously
exact **125.0s** for every one of 320 observed intervals, against a
configured `eval_interval_seconds: 60`. Root cause: `evaluator.tick()`
passed `workerPoolSize` (default 20) as *both* the claim batch size and
the concurrency limit — two genuinely different concerns conflated into
one number. With 500 rules all due at nearly the same instant (created
within ~65ms of each other), each 5-second tick could claim only 20 of
them regardless of how many more were already due, so draining the full
backlog took 500 ÷ 20 = 25 ticks × 5s = **125s** — more than double the
configured interval. Fixed by separating `EVALUATOR_CLAIM_BATCH_SIZE`
(default 1000 — how many due rules one tick can pull off the queue) from
`EVALUATOR_WORKER_POOL_SIZE` (default 20 — bounded concurrent `/query`
calls within that batch); see `alerting/internal/config/config.go`.
**Third run, after the fix**:
| Metric | Value |
|---|---|
| Rules seeded | 500 |
| Rules with `consecutive_errors > 0` | 0 |
| Configured `eval_interval_seconds` | 60 |
| Observed interval — min | 59.9s |
| Observed interval — mean | 63.3s |
| Observed interval — p95 | 65.0s |
| Observed interval — max | 65.2s |
| Drift vs. configured (p95 − configured) | 5.0s |
The remaining ~5s of "drift" is attributable to the load test's own
5-second poll granularity (an evaluation can only be observed to within
one poll interval), not to the evaluator falling behind — 500 rules at
60s intervals is comfortably within the evaluator's capacity once the
claim-batch/worker-pool conflation was fixed.
**Phase 4+ scaling paths, named but not solved here**: the claim query's
`SELECT ... FOR UPDATE SKIP LOCKED` design (see
`/docs/phase-3-alerting-design.md`'s fix 1) is specifically what makes
horizontal evaluator replicas safe to add later without a state-model
redesign — each replica's claim naturally excludes rows another replica
already claimed. Moving off a single-process ticker to a distributed
scheduler, and materially larger rule counts (10,000+), are both
explicitly out of scope for this phase.
## 5. Confirm `sentryctl`
```sh
sentryctl dashboards list
sentryctl dashboards apply exported-dashboard.json
sentryctl alerts list
sentryctl alerts apply rule.json
```
Both `dashboards` and `alerts` hit the exact same REST endpoints the web
UI does — `dashboards` against `/api` (`--api`), `alerts` against
`/alerting` (`--alerting-api`) — no separate CRUD logic to drift out of
sync.
## Tearing down
```sh
docker compose down -v # wipes the fixture dataset and all dashboards/rules
```
## Troubleshooting
**A relative `earliest=`/`latest=` query fails with `code: 53, Cannot
convert string ... to type DateTime64`.**
You've reverted or bypassed `formatClickHouseDateTime64` in
`api/internal/querylang/executor/sql.go` — ClickHouse's implicit
string→`DateTime64` cast needs `'YYYY-MM-DD HH:MM:SS[.fractional]'`, not
an ISO-8601/RFC3339 literal. See §2, bug 1 above.
**A dashboard chart panel renders with a tiny or zero-width canvas.**
Check that `PanelViz.svelte`'s `ResizeObserver` is still wired up — this
is the exact symptom of the gridstack/uPlot layout-timing race from §2,
bug 3.
**A rule created via the API never evaluates, but no error appears
anywhere.**
Check the response's `"enabled"` field — if the create request omitted
`"enabled"` entirely and the response shows `false`, the
`createRuleRequest` fix in `alerting/internal/httpapi/handler.go` has
regressed. See §3, bug 5.
**500+ rules at a short `eval_interval_seconds` fall behind the
configured interval.**
Check `EVALUATOR_CLAIM_BATCH_SIZE` hasn't been set equal to (or below)
`EVALUATOR_WORKER_POOL_SIZE` — that's the exact regression `hack/alert-
load-test` caught in §4. Claim batch size should stay well above the
worker pool size; the worker pool is what bounds concurrent `/query`
load, not the claim.
**`hack/alert-load-test` reports errors on every rule.**
Check the tool's generated query quotes any comparison value that might
contain a `-` (e.g. `host="host-01"`, not `host=host-01`) — an unquoted
value with a literal hyphen fails to parse. See §4's first-run finding.
docs/query-language-design.md
+21
View File
@@ -209,6 +209,27 @@ ClickHouse-side text indexing, a different join strategy, or simply
raising `max_query_size` server-side with matching memory sizing) is
explicitly future work, out of scope for Phase 2.
### Post-Phase-2 fix: `earliest=`/`latest=` never actually worked against live ClickHouse
Found during Phase 3's dashboard time-range picker work (the first thing
to run a relative `earliest=`/`latest=` query against real ClickHouse
end-to-end — none of Phase 2's own runbook queries or unit tests
happened to exercise it): `executor/sql.go` formatted `TimeRange` bounds
with `time.RFC3339Nano` (e.g. `2026-08-12T20:17:40.223505479Z`), which
ClickHouse's *implicit* string→`DateTime64` cast for a column-vs-literal
comparison rejects outright — `code: 53, Cannot convert string ... to
type DateTime64(9, 'UTC')`. ClickHouse's implicit cast is strict and
wants `'YYYY-MM-DD HH:MM:SS[.fractional]'` (space-separated, no `T`/`Z`);
the lenient ISO-8601-accepting `parseDateTimeBestEffort` is a different,
explicitly-invoked function, not what a plain `WHERE timestamp >= '...'`
comparison uses. Fixed by `formatClickHouseDateTime64` in `sql.go`. The
Phase 2 unit test that covered this (`TestBuildSQLTimeRange`) only
asserted the generated SQL *string*, against a fake `SQLRunner` — it
never caught this because nothing in that test actually asked real
ClickHouse whether the SQL was valid. Left here as a pointed reminder of
why this project's "actually run it" discipline exists: a passing test
suite and a working feature are not the same claim.
## Where this lives: `api/internal/querylang/`
Not a new top-level component. This subsystem always executes in-process