Phase 4 added a NOT NULL tenant_id column to both tables (migrations
0022/0023, backfilled via a join through alert_rules.id), but Store's
Create and ApplyTransition were never updated to populate it on new
inserts -- every existing row already had a value from the backfill,
which is exactly why this went uncaught: nothing created a *new* rule
against a Phase-4-or-later database until now. Every alert rule
creation since those migrations landed was silently broken.
Create's alert_state insert now passes rule.TenantID explicitly.
ApplyTransition only receives a rule ID, not a full Rule, so its
delivery_log insert resolves tenant_id via a subquery against
alert_rules. Confirmed against a live stack: rule creation, evaluation,
firing, and a real delivery attempt all completed end to end.
Saved, shareable multi-panel dashboards (table/line/bar/single-stat
panels via gridstack + uPlot, global + per-panel time range, JSON
export/import) and threshold/absence alert rules with an
ok/pending/firing evaluator and webhook/Slack/PagerDuty delivery.
- New /metadata component: Postgres control-plane store for dashboards,
panels, notification targets, alert rules/state, and delivery log --
see docs/phase-3-dashboard-design.md for why ClickHouse's MergeTree
family isn't a fit for this access pattern (needs real row-level
locking and read-your-writes consistency).
- api/internal/dashboards: dashboard/panel CRUD, pure -- panel query
execution stays client-side, reusing the existing /query endpoint.
- New /alerting service: rule/target CRUD, a ticker-driven evaluator
(claim-then-evaluate concurrency control, transactional-outbox
delivery, query errors and threshold zero-rows never coerced into a
false transition) and webhook/Slack/PagerDuty delivery with
retry/backoff. See docs/phase-3-alerting-design.md for the full
state-machine design and the four correctness properties it
implements.
- web: /dashboards and /alerts UIs; cli: sentryctl dashboards/alerts
list/get/apply, seeding a future Terraform provider's JSON contract.
- hack/alert-load-test: 500 rules against real ClickHouse data, real
measured results in docs/phase-3-runbook.md.
Five real bugs found by actually running this against a live stack
(documented in the runbook, not just fixed silently): a latent Phase 2
bug where ClickHouse rejected the timestamp format used for
earliest=/latest= queries; a "now" literal token injected into query
text; a GridStack/uPlot layout-timing race; JS's Date.parse being too
lenient to use as a timestamp-detection heuristic; a rule's "enabled"
field silently defaulting to false when omitted; and the evaluator's
claim-batch-size and worker-pool-concurrency defaulting to the same
value, causing 500 concurrently-due rules to take 125s to cycle through
instead of the configured 60s.
jcoffey-dev