Phase 3: dashboards and alerting

Saved, shareable multi-panel dashboards (table/line/bar/single-stat
panels via gridstack + uPlot, global + per-panel time range, JSON
export/import) and threshold/absence alert rules with an
ok/pending/firing evaluator and webhook/Slack/PagerDuty delivery.

- New /metadata component: Postgres control-plane store for dashboards,
  panels, notification targets, alert rules/state, and delivery log --
  see docs/phase-3-dashboard-design.md for why ClickHouse's MergeTree
  family isn't a fit for this access pattern (needs real row-level
  locking and read-your-writes consistency).
- api/internal/dashboards: dashboard/panel CRUD, pure -- panel query
  execution stays client-side, reusing the existing /query endpoint.
- New /alerting service: rule/target CRUD, a ticker-driven evaluator
  (claim-then-evaluate concurrency control, transactional-outbox
  delivery, query errors and threshold zero-rows never coerced into a
  false transition) and webhook/Slack/PagerDuty delivery with
  retry/backoff. See docs/phase-3-alerting-design.md for the full
  state-machine design and the four correctness properties it
  implements.
- web: /dashboards and /alerts UIs; cli: sentryctl dashboards/alerts
  list/get/apply, seeding a future Terraform provider's JSON contract.
- hack/alert-load-test: 500 rules against real ClickHouse data, real
  measured results in docs/phase-3-runbook.md.

Five real bugs found by actually running this against a live stack
(documented in the runbook, not just fixed silently): a latent Phase 2
bug where ClickHouse rejected the timestamp format used for
earliest=/latest= queries; a "now" literal token injected into query
text; a GridStack/uPlot layout-timing race; JS's Date.parse being too
lenient to use as a timestamp-detection heuristic; a rule's "enabled"
field silently defaulting to false when omitted; and the evaluator's
claim-batch-size and worker-pool-concurrency defaulting to the same
value, causing 500 concurrently-due rules to take 125s to cycle through
instead of the configured 60s.

alerting/internal/evaluator/transitions.go

@@ -0,0 +1,121 @@
+// Package evaluator implements the ticker-driven rule scheduler and the
+// firing/resolved state machine described in
+// /docs/phase-3-alerting-design.md. transitions.go is deliberately pure
+// (no DB, no HTTP, no clock reads beyond the `Now` passed in) --
+// evaluator.go is the only caller, and it's the single place worth
+// exhaustive table-driven testing given how easy this state machine is
+// to get subtly wrong.
+package evaluator
+
+import (
+	"time"
+
+	"github.com/sentry/sentry/alerting/internal/rulestore"
+)
+
+// TransitionInput is everything ComputeTransition needs: the rule's
+// current persisted state plus this evaluation's outcome. Evaluation
+// *errors* never reach this function at all -- per fix 3 in the design
+// doc, an error is recorded separately (rulestore.RecordError) and never
+// treated as ConditionTrue: false.
+type TransitionInput struct {
+	CurrentState       rulestore.State
+	ConditionTrueSince *time.Time
+	FiredAt            *time.Time
+	LastNotifiedAt     *time.Time
+	ForMinutes         time.Duration
+	RenotifyInterval   *time.Duration // nil = notify once per firing episode, never again
+	Now                time.Time
+	ConditionTrue      bool
+}
+
+// TransitionResult is what to persist (via rulestore.ApplyTransition)
+// and, if Notify is non-nil, which event ("firing" or "resolved") to
+// enqueue in the same transaction.
+type TransitionResult struct {
+	NextState              rulestore.State
+	NextConditionTrueSince *time.Time
+	NextFiredAt            *time.Time
+	NextLastNotifiedAt     *time.Time
+	Notify                 *string
+}
+
+// ComputeTransition implements the ok/pending/firing state machine.
+// ConditionTrueSince is always a wall-clock timestamp, never a
+// consecutive-evaluation counter -- this is what makes the for_minutes
+// debounce survive evaluator restarts correctly (a counter would
+// silently lose progress across downtime; wall-clock math resumes
+// exactly where it left off). Don't "simplify" this into a counter.
+func ComputeTransition(in TransitionInput) TransitionResult {
+	if in.ConditionTrue {
+		return computeConditionTrue(in)
+	}
+	return computeConditionFalse(in)
+}
+
+func computeConditionTrue(in TransitionInput) TransitionResult {
+	switch in.CurrentState {
+	case rulestore.StatePending:
+		since := in.ConditionTrueSince
+		if since == nil {
+			// Defensive: pending with no recorded since is inconsistent
+			// persisted state (shouldn't happen -- ok->pending always sets
+			// it) -- treat as if the condition just became true rather than
+			// dereferencing a nil or panicking.
+			now := in.Now
+			since = &now
+		}
+		if in.Now.Sub(*since) >= in.ForMinutes {
+			return fire(*since, in.Now)
+		}
+		return TransitionResult{NextState: rulestore.StatePending, NextConditionTrueSince: since}
+
+	case rulestore.StateFiring:
+		result := TransitionResult{
+			NextState:              rulestore.StateFiring,
+			NextConditionTrueSince: in.ConditionTrueSince,
+			NextFiredAt:            in.FiredAt,
+			NextLastNotifiedAt:     in.LastNotifiedAt,
+		}
+		if in.RenotifyInterval != nil && in.LastNotifiedAt != nil && in.Now.Sub(*in.LastNotifiedAt) >= *in.RenotifyInterval {
+			now := in.Now
+			event := "firing"
+			result.NextLastNotifiedAt = &now
+			result.Notify = &event
+		}
+		return result
+
+	default: // ok (or any unexpected value -- the DB CHECK constraint keeps this to 3 values)
+		since := in.Now
+		if in.ForMinutes <= 0 {
+			// for_minutes=0 means "fire on first true evaluation" --
+			// literally, not after one extra tick spent in `pending`.
+			return fire(since, in.Now)
+		}
+		return TransitionResult{NextState: rulestore.StatePending, NextConditionTrueSince: &since}
+	}
+}
+
+func computeConditionFalse(in TransitionInput) TransitionResult {
+	switch in.CurrentState {
+	case rulestore.StateFiring:
+		event := "resolved"
+		return TransitionResult{NextState: rulestore.StateOK, Notify: &event}
+	default: // ok or pending: a false evaluation while pending is a blip
+		// inside the debounce window -- deliberately no notification, that's
+		// the entire point of for_minutes existing.
+		return TransitionResult{NextState: rulestore.StateOK}
+	}
+}
+
+func fire(since, now time.Time) TransitionResult {
+	firedAt := now
+	event := "firing"
+	return TransitionResult{
+		NextState:              rulestore.StateFiring,
+		NextConditionTrueSince: &since,
+		NextFiredAt:            &firedAt,
+		NextLastNotifiedAt:     &firedAt,
+		Notify:                 &event,
+	}
+}