Phase 3: dashboards and alerting

Saved, shareable multi-panel dashboards (table/line/bar/single-stat
panels via gridstack + uPlot, global + per-panel time range, JSON
export/import) and threshold/absence alert rules with an
ok/pending/firing evaluator and webhook/Slack/PagerDuty delivery.

- New /metadata component: Postgres control-plane store for dashboards,
  panels, notification targets, alert rules/state, and delivery log --
  see docs/phase-3-dashboard-design.md for why ClickHouse's MergeTree
  family isn't a fit for this access pattern (needs real row-level
  locking and read-your-writes consistency).
- api/internal/dashboards: dashboard/panel CRUD, pure -- panel query
  execution stays client-side, reusing the existing /query endpoint.
- New /alerting service: rule/target CRUD, a ticker-driven evaluator
  (claim-then-evaluate concurrency control, transactional-outbox
  delivery, query errors and threshold zero-rows never coerced into a
  false transition) and webhook/Slack/PagerDuty delivery with
  retry/backoff. See docs/phase-3-alerting-design.md for the full
  state-machine design and the four correctness properties it
  implements.
- web: /dashboards and /alerts UIs; cli: sentryctl dashboards/alerts
  list/get/apply, seeding a future Terraform provider's JSON contract.
- hack/alert-load-test: 500 rules against real ClickHouse data, real
  measured results in docs/phase-3-runbook.md.

Five real bugs found by actually running this against a live stack
(documented in the runbook, not just fixed silently): a latent Phase 2
bug where ClickHouse rejected the timestamp format used for
earliest=/latest= queries; a "now" literal token injected into query
text; a GridStack/uPlot layout-timing race; JS's Date.parse being too
lenient to use as a timestamp-detection heuristic; a rule's "enabled"
field silently defaulting to false when omitted; and the evaluator's
claim-batch-size and worker-pool-concurrency defaulting to the same
value, causing 500 concurrently-due rules to take 125s to cycle through
instead of the configured 60s.
This commit is contained in:
2026-08-13 17:29:38 -07:00
parent fb5049a747
commit 9435115ab7
88 changed files with 7463 additions and 298 deletions
+12
View File
@@ -0,0 +1,12 @@
# alerting never needs /proto (it talks to /api over plain HTTP, no gRPC),
# so unlike api/ingest/search this build context is just alerting/ itself,
# same shape as cli/Dockerfile:
# docker build -f alerting/Dockerfile -t sentry-alerting alerting/
FROM golang:1.25-alpine AS builder
WORKDIR /src
COPY . .
RUN CGO_ENABLED=0 GOOS=linux go build -o /out/alerting ./cmd/alerting
FROM gcr.io/distroless/static-debian12
COPY --from=builder /out/alerting /alerting
ENTRYPOINT ["/alerting"]
+98
View File
@@ -0,0 +1,98 @@
# alerting
Alert rule CRUD, the ticker-driven evaluator, and webhook/Slack/PagerDuty
delivery. See `/docs/phase-3-alerting-design.md` for the full design
(data model, the ok/pending/firing state machine, and the four
correctness properties this implementation follows exactly).
## Running
```sh
POSTGRES_PASSWORD=sentry-dev-only API_QUERY_URL=http://localhost:8080 go run ./cmd/alerting
```
Talks to the same `sentry_metadata` Postgres database as `/api`
(different tables — see `/metadata/README.md`), and to `/api`'s
`POST /query` over plain HTTP for rule evaluation. Never connects to
ClickHouse or Tantivy directly.
## HTTP API
```
POST /rules create a rule
GET /rules list rules (with current state)
GET /rules/{id} get a rule (with current state)
DELETE /rules/{id}
GET /rules/{id}/deliveries delivery log for a rule, most recent first
POST /targets create a notification target
GET /targets
GET /targets/{id}
DELETE /targets/{id}
GET /healthz
```
A rule's `condition_type` is `"threshold"` (requires `comparator` +
`threshold_value`, and the query must resolve to exactly one row) or
`"absence"` (fires when the query returns zero rows in its own
`earliest=`/`latest=` window — no separate window field). A notification
target's `kind` is `"webhook"`, `"slack"`, or `"pagerduty"` — all three
deliver via the same HTTP POST + retry/backoff mechanism
(`internal/delivery/webhook.go`); slack/pagerduty are payload formatters
only, not separate delivery paths.
## Environment variables
| Var | Default |
|---|---|
| `HTTP_LISTEN_ADDR` | `:8081` |
| `POSTGRES_ADDR` | `localhost:5432` |
| `POSTGRES_DATABASE` | `sentry_metadata` |
| `POSTGRES_USERNAME` | `sentry` |
| `POSTGRES_PASSWORD` | (empty — must be set) |
| `API_QUERY_URL` | `http://localhost:8080` |
| `CORS_ALLOWED_ORIGIN` | `*` |
| `EVALUATOR_TICK_SECONDS` | `5` — how often the scheduler checks for due rules |
| `EVALUATOR_CLAIM_BATCH_SIZE` | `1000` — how many due rules one tick can pull off the queue |
| `EVALUATOR_WORKER_POOL_SIZE` | `20` — bounded concurrency for `/query` calls within a claimed batch |
| `EVALUATOR_QUERY_TIMEOUT_SECONDS` | `30` — per-evaluation `POST /query` timeout |
`EVALUATOR_CLAIM_BATCH_SIZE` and `EVALUATOR_WORKER_POOL_SIZE` are
deliberately separate knobs, not the same number — see
`internal/config/config.go`'s doc comment for the real bug this
separation fixes (found by `hack/alert-load-test`, see
`/docs/phase-3-runbook.md`): with both capped at 20, 500 rules due at
once took 125s to cycle through instead of the configured 60s.
## Package layout
```
cmd/alerting/ wires config, Postgres pool, api client; runs the
HTTP server + evaluator + delivery worker concurrently (errgroup)
internal/httpapi/ REST handlers -- Handler/RegisterRoutes, same shape as api/internal/dashboards
internal/rulestore/ pgx CRUD for alert_rules + alert_state; ClaimDueRules
(fix 1's atomic claim) and ApplyTransition (fix 2's transactional outbox)
internal/notifystore/ pgx CRUD for notification_targets
internal/queryclient/ thin HTTP client to api's POST /query -- no querylang import here
internal/evaluator/ the ticker + worker pool; transitions.go is the pure,
exhaustively-tested ok/pending/firing state machine;
condition.go implements fixes 3/4 (errors never
coerced to "condition false"; threshold zero-rows
is an error, not a 0)
internal/delivery/ webhook.go is the claim-and-send worker (all three
kinds go through it); slack.go/pagerduty.go are
payload formatters only
```
## Building & testing
```sh
go build ./...
go vet ./...
go test ./...
```
```sh
docker build -f Dockerfile -t sentry-alerting . # context is alerting/, not the repo root -- no /proto needed
```
+141
View File
@@ -0,0 +1,141 @@
// Command alerting is Sentry's alert rule evaluator and delivery
// service: rule/target CRUD, the ticker-driven ok/pending/firing
// evaluator, and the webhook/Slack/PagerDuty delivery worker. See
// /docs/phase-3-alerting-design.md. Never talks to ClickHouse/Tantivy
// directly -- rule queries run through /api's POST /query
// (internal/queryclient), same precedent sentryctl query and the web
// UI's dashboard panels already set.
package main
import (
"context"
"fmt"
"log/slog"
"net/http"
"os"
"os/signal"
"strings"
"syscall"
"time"
"github.com/jackc/pgx/v5/pgxpool"
"golang.org/x/sync/errgroup"
"github.com/sentry/sentry/alerting/internal/config"
"github.com/sentry/sentry/alerting/internal/delivery"
"github.com/sentry/sentry/alerting/internal/evaluator"
"github.com/sentry/sentry/alerting/internal/httpapi"
"github.com/sentry/sentry/alerting/internal/httpserver"
"github.com/sentry/sentry/alerting/internal/notifystore"
"github.com/sentry/sentry/alerting/internal/queryclient"
"github.com/sentry/sentry/alerting/internal/rulestore"
)
func main() {
logger := slog.New(slog.NewJSONHandler(os.Stdout, nil))
cfg, err := config.Load()
if err != nil {
logger.Error("loading config", "error", err)
os.Exit(1)
}
// -healthcheck: self-check mode for Docker's HEALTHCHECK, mirrors
// api/cmd/api/main.go's runHealthcheck -- this image is distroless too
// (no shell, no wget), so the compose healthcheck execs this binary
// against itself.
if len(os.Args) > 1 && os.Args[1] == "-healthcheck" {
os.Exit(runHealthcheck(cfg.HTTPListenAddr))
}
ctx, stop := signal.NotifyContext(context.Background(), syscall.SIGINT, syscall.SIGTERM)
defer stop()
pgDSN := fmt.Sprintf("postgres://%s:%s@%s/%s", cfg.Postgres.Username, cfg.Postgres.Password, cfg.Postgres.Addr, cfg.Postgres.Database)
pgPool, err := pgxpool.New(ctx, pgDSN)
if err != nil {
logger.Error("opening postgres pool", "error", err)
os.Exit(1)
}
defer pgPool.Close()
if err := pgPool.Ping(ctx); err != nil {
logger.Error("pinging postgres", "error", err)
os.Exit(1)
}
rules := rulestore.NewStore(pgPool)
targets := notifystore.NewStore(pgPool)
qc := queryclient.New(cfg.APIQueryURL)
handler := httpapi.NewHandler(logger, rules, targets, rules)
mux := http.NewServeMux()
handler.RegisterRoutes(mux)
srv := &http.Server{
Addr: cfg.HTTPListenAddr,
Handler: httpserver.WithCORS(mux, cfg.CORSAllowedOrigin),
}
eval := evaluator.New(rules, targets, qc, cfg.Evaluator.QueryTimeout, cfg.Evaluator.ClaimBatchSize, cfg.Evaluator.WorkerPoolSize, logger)
deliveryWorker := delivery.NewWorker(pgPool, targets, logger)
g, ctx := errgroup.WithContext(ctx)
g.Go(func() error {
logger.Info("alerting listening", "addr", cfg.HTTPListenAddr)
errCh := make(chan error, 1)
go func() { errCh <- srv.ListenAndServe() }()
select {
case <-ctx.Done():
shutdownCtx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
defer cancel()
if err := srv.Shutdown(shutdownCtx); err != nil {
return fmt.Errorf("graceful shutdown: %w", err)
}
return nil
case err := <-errCh:
if err != nil && err != http.ErrServerClosed {
return fmt.Errorf("http server exited: %w", err)
}
return nil
}
})
g.Go(func() error {
logger.Info("evaluator started", "tick_interval", cfg.Evaluator.TickInterval, "worker_pool_size", cfg.Evaluator.WorkerPoolSize)
if err := eval.Run(ctx, cfg.Evaluator.TickInterval); err != nil && ctx.Err() == nil {
return fmt.Errorf("evaluator exited: %w", err)
}
return nil
})
g.Go(func() error {
logger.Info("delivery worker started")
if err := deliveryWorker.Run(ctx, cfg.Evaluator.TickInterval); err != nil && ctx.Err() == nil {
return fmt.Errorf("delivery worker exited: %w", err)
}
return nil
})
if err := g.Wait(); err != nil {
logger.Error("alerting exited with error", "error", err)
os.Exit(1)
}
}
func runHealthcheck(listenAddr string) int {
addr := listenAddr
if strings.HasPrefix(addr, ":") {
addr = "localhost" + addr
}
client := http.Client{Timeout: 3 * time.Second}
resp, err := client.Get("http://" + addr + "/healthz")
if err != nil {
return 1
}
defer resp.Body.Close()
if resp.StatusCode != http.StatusOK {
return 1
}
return 0
}
+16
View File
@@ -0,0 +1,16 @@
module github.com/sentry/sentry/alerting
go 1.25.0
require (
github.com/google/uuid v1.6.0
github.com/jackc/pgx/v5 v5.10.0
golang.org/x/sync v0.22.0
)
require (
github.com/jackc/pgpassfile v1.0.0 // indirect
github.com/jackc/pgservicefile v0.0.0-20240606120523-5a60cdf6a761 // indirect
github.com/jackc/puddle/v2 v2.2.2 // indirect
golang.org/x/text v0.29.0 // indirect
)
+28
View File
@@ -0,0 +1,28 @@
github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c=
github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
github.com/google/uuid v1.6.0 h1:NIvaJDMOsjHA8n1jAhLSgzrAzy1Hgr+hNrb57e+94F0=
github.com/google/uuid v1.6.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo=
github.com/jackc/pgpassfile v1.0.0 h1:/6Hmqy13Ss2zCq62VdNG8tM1wchn8zjSGOBJ6icpsIM=
github.com/jackc/pgpassfile v1.0.0/go.mod h1:CEx0iS5ambNFdcRtxPj5JhEz+xB6uRky5eyVu/W2HEg=
github.com/jackc/pgservicefile v0.0.0-20240606120523-5a60cdf6a761 h1:iCEnooe7UlwOQYpKFhBabPMi4aNAfoODPEFNiAnClxo=
github.com/jackc/pgservicefile v0.0.0-20240606120523-5a60cdf6a761/go.mod h1:5TJZWKEWniPve33vlWYSoGYefn3gLQRzjfDlhSJ9ZKM=
github.com/jackc/pgx/v5 v5.10.0 h1:VhSvgU2jSli8o3AqIEOTJr7rZwAEUVo4E4XhR94Zfr0=
github.com/jackc/pgx/v5 v5.10.0/go.mod h1:mal1tBGAFfLHvZzaYh77YS/eC6IX9OWbRV1QIIM0Jn4=
github.com/jackc/puddle/v2 v2.2.2 h1:PR8nw+E/1w0GLuRFSmiioY6UooMp6KJv0/61nB7icHo=
github.com/jackc/puddle/v2 v2.2.2/go.mod h1:vriiEXHvEE654aYKXXjOvZM39qJ0q+azkZFrfEOc3H4=
github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM=
github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME=
github.com/stretchr/testify v1.3.0/go.mod h1:M5WIy9Dh21IEIfnGCwXGc5bZfKNJtfHm1UVUgZn+9EI=
github.com/stretchr/testify v1.7.0/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg=
github.com/stretchr/testify v1.11.1 h1:7s2iGBzp5EwR7/aIZr8ao5+dra3wiQyKjjFuvgVKu7U=
github.com/stretchr/testify v1.11.1/go.mod h1:wZwfW3scLgRK+23gO65QZefKpKQRnfz6sD981Nm4B6U=
golang.org/x/sync v0.22.0 h1:SZjpbeLmrCk4xhRSZFNZW5gFUeCeFgjekvI/+gfScek=
golang.org/x/sync v0.22.0/go.mod h1:9xrNwdLfx4jkKbNva9FpL6vEN7evnE43NNNJQ2LF3+0=
golang.org/x/text v0.29.0 h1:1neNs90w9YzJ9BocxfsQNHKuAT4pkghyXc4nhZ6sJvk=
golang.org/x/text v0.29.0/go.mod h1:7MhJOA9CD2qZyOKYazxdYMF85OwPdEr9jTtBpO7ydH4=
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
gopkg.in/yaml.v3 v3.0.0-20200313102051-9f266ea9e77c/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA=
gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
+96
View File
@@ -0,0 +1,96 @@
// Package config loads alerting's configuration from environment
// variables, same convention as /api and /ingest: no config file format.
package config
import (
"fmt"
"os"
"strconv"
"time"
)
type Config struct {
HTTPListenAddr string
Postgres PostgresConfig
APIQueryURL string // base URL of /api, e.g. http://api:8080 -- alerting never talks to ClickHouse/Tantivy directly
CORSAllowedOrigin string
Evaluator EvaluatorConfig
}
type PostgresConfig struct {
Addr string
Database string
Username string
Password string
}
type EvaluatorConfig struct {
TickInterval time.Duration // how often the scheduler checks for due rules
// ClaimBatchSize and WorkerPoolSize are deliberately separate knobs,
// not the same number: ClaimBatchSize bounds how many due rules one
// tick pulls off the queue (needs to be large enough to drain a
// backlog when many rules share a due time, e.g. right after bulk
// creation), while WorkerPoolSize bounds concurrent /query calls
// within that batch. Found by actually running hack/alert-load-test
// with 500 rules and both numbers defaulted to the same value
// (20): the evaluator took 125s to cycle through 500 due rules
// instead of the configured 60s eval_interval_seconds, because each
// 5s tick could only claim 20 rules regardless of how many more were
// already due -- see /docs/phase-3-runbook.md's load-test section.
ClaimBatchSize int
WorkerPoolSize int
QueryTimeout time.Duration // per-evaluation POST /query timeout
}
func Load() (Config, error) {
cfg := Config{
HTTPListenAddr: getenv("HTTP_LISTEN_ADDR", ":8081"),
Postgres: PostgresConfig{
Addr: getenv("POSTGRES_ADDR", "localhost:5432"),
Database: getenv("POSTGRES_DATABASE", "sentry_metadata"),
Username: getenv("POSTGRES_USERNAME", "sentry"),
Password: getenv("POSTGRES_PASSWORD", ""),
},
APIQueryURL: getenv("API_QUERY_URL", "http://localhost:8080"),
// Same "no auth yet" tradeoff as api's CORSAllowedOrigin default --
// see api/internal/config/config.go's comment, same reasoning here.
CORSAllowedOrigin: getenv("CORS_ALLOWED_ORIGIN", "*"),
}
tickSec, err := strconv.Atoi(getenv("EVALUATOR_TICK_SECONDS", "5"))
if err != nil {
return Config{}, fmt.Errorf("EVALUATOR_TICK_SECONDS: %w", err)
}
cfg.Evaluator.TickInterval = time.Duration(tickSec) * time.Second
poolSize, err := strconv.Atoi(getenv("EVALUATOR_WORKER_POOL_SIZE", "20"))
if err != nil {
return Config{}, fmt.Errorf("EVALUATOR_WORKER_POOL_SIZE: %w", err)
}
cfg.Evaluator.WorkerPoolSize = poolSize
// Default well above WorkerPoolSize: this is "how many due rules can
// one tick pull off the queue," not a concurrency limit -- the
// worker pool below still bounds actual concurrent /query calls
// regardless of how large a batch gets claimed.
claimBatchSize, err := strconv.Atoi(getenv("EVALUATOR_CLAIM_BATCH_SIZE", "1000"))
if err != nil {
return Config{}, fmt.Errorf("EVALUATOR_CLAIM_BATCH_SIZE: %w", err)
}
cfg.Evaluator.ClaimBatchSize = claimBatchSize
queryTimeoutSec, err := strconv.Atoi(getenv("EVALUATOR_QUERY_TIMEOUT_SECONDS", "30"))
if err != nil {
return Config{}, fmt.Errorf("EVALUATOR_QUERY_TIMEOUT_SECONDS: %w", err)
}
cfg.Evaluator.QueryTimeout = time.Duration(queryTimeoutSec) * time.Second
return cfg, nil
}
func getenv(key, fallback string) string {
if v := os.Getenv(key); v != "" {
return v
}
return fallback
}
@@ -0,0 +1,88 @@
package delivery
import (
"encoding/json"
"testing"
"time"
"github.com/sentry/sentry/alerting/internal/notifystore"
)
func TestBuildPayloadGenericDefaultShape(t *testing.T) {
target := notifystore.Target{Kind: notifystore.KindWebhook}
value := 142.0
payload, err := BuildPayload(target, Event{
RuleID: "r1", RuleName: "High error rate", EventType: "firing",
Value: &value, Timestamp: time.Date(2026, 8, 14, 0, 0, 0, 0, time.UTC),
})
if err != nil {
t.Fatalf("BuildPayload: %v", err)
}
var got defaultGenericPayload
if err := json.Unmarshal(payload, &got); err != nil {
t.Fatalf("unmarshaling payload: %v", err)
}
if got.RuleName != "High error rate" || got.EventType != "firing" || got.Value == nil || *got.Value != 142.0 {
t.Fatalf("unexpected payload: %+v", got)
}
}
func TestBuildPayloadGenericUsesTemplate(t *testing.T) {
tmpl := `{"custom": "{{.RuleName}} is {{.EventType}}"}`
target := notifystore.Target{Kind: notifystore.KindWebhook, PayloadTemplate: &tmpl}
payload, err := BuildPayload(target, Event{RuleName: "disk full", EventType: "firing"})
if err != nil {
t.Fatalf("BuildPayload: %v", err)
}
var got map[string]string
if err := json.Unmarshal(payload, &got); err != nil {
t.Fatalf("unmarshaling templated payload: %v", err)
}
if got["custom"] != "disk full is firing" {
t.Fatalf("got %q", got["custom"])
}
}
func TestBuildPayloadSlackShape(t *testing.T) {
target := notifystore.Target{Kind: notifystore.KindSlack}
payload, err := BuildPayload(target, Event{RuleName: "High error rate", EventType: "resolved"})
if err != nil {
t.Fatalf("BuildPayload: %v", err)
}
var got slackPayload
if err := json.Unmarshal(payload, &got); err != nil {
t.Fatalf("unmarshaling slack payload: %v", err)
}
if got.Text == "" {
t.Fatalf("expected non-empty Slack text")
}
}
func TestBuildPayloadPagerDutyMapsEventActionAndRoutingKey(t *testing.T) {
secret := "R0UT1NGKEY"
target := notifystore.Target{Kind: notifystore.KindPagerDuty, Secret: &secret}
firing, err := BuildPayload(target, Event{RuleID: "r1", RuleName: "disk full", EventType: "firing"})
if err != nil {
t.Fatalf("BuildPayload firing: %v", err)
}
var gotFiring pagerDutyPayload
if err := json.Unmarshal(firing, &gotFiring); err != nil {
t.Fatalf("unmarshaling: %v", err)
}
if gotFiring.EventAction != "trigger" || gotFiring.RoutingKey != secret || gotFiring.DedupKey != "r1" {
t.Fatalf("unexpected firing payload: %+v", gotFiring)
}
resolved, err := BuildPayload(target, Event{RuleID: "r1", RuleName: "disk full", EventType: "resolved"})
if err != nil {
t.Fatalf("BuildPayload resolved: %v", err)
}
var gotResolved pagerDutyPayload
if err := json.Unmarshal(resolved, &gotResolved); err != nil {
t.Fatalf("unmarshaling: %v", err)
}
if gotResolved.EventAction != "resolve" {
t.Fatalf("expected event_action=resolve, got %q", gotResolved.EventAction)
}
}
+57
View File
@@ -0,0 +1,57 @@
package delivery
import (
"encoding/json"
"fmt"
"github.com/sentry/sentry/alerting/internal/notifystore"
)
// pagerDutyPayload is PagerDuty's Events API v2 shape. routing_key comes
// from the target's `secret` field -- PagerDuty calls this an
// "integration key," not a delivery credential in the auth sense, but
// it's stored in the same plaintext `secret` column as any other
// target's credential (see /docs/phase-3-alerting-design.md's "Known
// gaps").
type pagerDutyPayload struct {
RoutingKey string `json:"routing_key"`
EventAction string `json:"event_action"` // "trigger" | "resolve"
DedupKey string `json:"dedup_key"` // rule_id -- lets PagerDuty correlate trigger/resolve as one incident
Payload pagerDutyEventPayload `json:"payload"`
}
type pagerDutyEventPayload struct {
Summary string `json:"summary"`
Source string `json:"source"`
Severity string `json:"severity"`
}
func buildPagerDutyPayload(target notifystore.Target, event Event) ([]byte, error) {
action := "trigger"
severity := "critical"
if event.EventType == "resolved" {
action = "resolve"
severity = "info"
}
var routingKey string
if target.Secret != nil {
routingKey = *target.Secret
}
summary := fmt.Sprintf("%s: %s", event.RuleName, event.EventType)
if event.Value != nil && event.ThresholdValue != nil {
summary = fmt.Sprintf("%s (value %.4g %s %.4g)", summary, *event.Value, comparatorSymbol(event.Comparator), *event.ThresholdValue)
}
return json.Marshal(pagerDutyPayload{
RoutingKey: routingKey,
EventAction: action,
DedupKey: event.RuleID,
Payload: pagerDutyEventPayload{
Summary: summary,
Source: "sentry",
Severity: severity,
},
})
}
+48
View File
@@ -0,0 +1,48 @@
package delivery
import (
"encoding/json"
"fmt"
)
// slackPayload is Slack's incoming-webhook shape: {"text": "..."}. No
// user template for this kind -- see webhook.go's BuildPayload doc
// comment for why (thin formatter, not a user-configurable path).
type slackPayload struct {
Text string `json:"text"`
}
func buildSlackPayload(event Event) ([]byte, error) {
var text string
switch event.EventType {
case "firing":
text = fmt.Sprintf(":rotating_light: *%s* is firing", event.RuleName)
if event.Value != nil && event.ThresholdValue != nil {
text += fmt.Sprintf(" (value %.4g %s %.4g)", *event.Value, comparatorSymbol(event.Comparator), *event.ThresholdValue)
}
case "resolved":
text = fmt.Sprintf(":white_check_mark: *%s* resolved", event.RuleName)
default:
text = fmt.Sprintf("%s: %s", event.RuleName, event.EventType)
}
return json.Marshal(slackPayload{Text: text})
}
func comparatorSymbol(c string) string {
switch c {
case "gt":
return ">"
case "gte":
return ">="
case "lt":
return "<"
case "lte":
return "<="
case "eq":
return "=="
case "ne":
return "!="
default:
return c
}
}
+229
View File
@@ -0,0 +1,229 @@
// Package delivery sends notifications for firing/resolved alert
// events. All three notification_targets.kind values -- webhook, slack,
// pagerduty -- go through the exact same claim-then-POST-with-backoff
// mechanism in this file; slack.go and pagerduty.go are payload
// *formatters* only, never a separate delivery path, per
// /docs/phase-3-alerting-design.md's "thin wrappers over the generic
// webhook" requirement.
package delivery
import (
"bytes"
"context"
"encoding/json"
"fmt"
"log/slog"
"net/http"
"text/template"
"time"
"github.com/jackc/pgx/v5/pgxpool"
"github.com/sentry/sentry/alerting/internal/notifystore"
)
// Event is the firing/resolved occurrence a payload is rendered from.
type Event struct {
RuleID string
RuleName string
EventType string // "firing" | "resolved"
ConditionType string
Comparator string
ThresholdValue *float64
Value *float64
Timestamp time.Time
}
// defaultGenericPayload is used when a webhook target has no
// payload_template.
type defaultGenericPayload struct {
RuleID string `json:"rule_id"`
RuleName string `json:"rule_name"`
EventType string `json:"event_type"`
Value *float64 `json:"value,omitempty"`
Timestamp string `json:"timestamp"`
}
// BuildPayload dispatches by target.Kind to the right formatter. This is
// the only place kind-specific formatting logic lives -- everything
// downstream of this (worker.go's send loop) is kind-agnostic.
func BuildPayload(target notifystore.Target, event Event) ([]byte, error) {
switch target.Kind {
case notifystore.KindSlack:
return buildSlackPayload(event)
case notifystore.KindPagerDuty:
return buildPagerDutyPayload(target, event)
default: // webhook
return buildGenericPayload(target, event)
}
}
func buildGenericPayload(target notifystore.Target, event Event) ([]byte, error) {
if target.PayloadTemplate == nil || *target.PayloadTemplate == "" {
return json.Marshal(defaultGenericPayload{
RuleID: event.RuleID, RuleName: event.RuleName, EventType: event.EventType,
Value: event.Value, Timestamp: event.Timestamp.UTC().Format(time.RFC3339),
})
}
tmpl, err := template.New("payload").Parse(*target.PayloadTemplate)
if err != nil {
return nil, fmt.Errorf("parsing payload_template: %w", err)
}
var buf bytes.Buffer
if err := tmpl.Execute(&buf, event); err != nil {
return nil, fmt.Errorf("rendering payload_template: %w", err)
}
return buf.Bytes(), nil
}
// Worker claims pending/retrying delivery_log rows and sends them.
// Deliberately separate from the evaluator: "decided to notify"
// (rulestore.ApplyTransition's transactional outbox insert) is
// transactionally certain; "the HTTP call succeeded" is best-effort with
// retries, handled entirely here.
type Worker struct {
pool *pgxpool.Pool
notifications *notifystore.Store
http *http.Client
logger *slog.Logger
}
func NewWorker(pool *pgxpool.Pool, notifications *notifystore.Store, logger *slog.Logger) *Worker {
return &Worker{pool: pool, notifications: notifications, http: &http.Client{Timeout: 10 * time.Second}, logger: logger}
}
// Run claims and sends due deliveries every tickInterval until ctx is
// cancelled.
func (w *Worker) Run(ctx context.Context, tickInterval time.Duration) error {
ticker := time.NewTicker(tickInterval)
defer ticker.Stop()
for {
select {
case <-ctx.Done():
return ctx.Err()
case <-ticker.C:
w.processDue(ctx)
}
}
}
type claimedDelivery struct {
id int64
ruleID string
notificationTargetID string
eventType string
attemptCount int
maxAttempts int
payload []byte
}
// processDue claims due rows (same SKIP LOCKED pattern as the
// evaluator's rule claim -- see rulestore.ClaimDueRules) and attempts
// delivery for each. One claim batch per tick; a stuck/slow target
// doesn't block others since each send happens independently.
func (w *Worker) processDue(ctx context.Context) {
rows, err := w.pool.Query(ctx, `
WITH due AS (
SELECT id FROM delivery_log
WHERE status IN ('pending', 'retrying') AND (next_attempt_at IS NULL OR next_attempt_at <= now())
ORDER BY created_at
LIMIT 50
FOR UPDATE SKIP LOCKED
)
UPDATE delivery_log d
SET last_attempt_at = now()
FROM due
WHERE d.id = due.id
RETURNING d.id, d.rule_id, d.notification_target_id, d.event_type, d.attempt_count, d.max_attempts, d.payload`)
if err != nil {
w.logger.Error("claiming due deliveries", "error", err)
return
}
var claimed []claimedDelivery
for rows.Next() {
var c claimedDelivery
if err := rows.Scan(&c.id, &c.ruleID, &c.notificationTargetID, &c.eventType, &c.attemptCount, &c.maxAttempts, &c.payload); err != nil {
w.logger.Error("scanning claimed delivery", "error", err)
continue
}
claimed = append(claimed, c)
}
rows.Close()
for _, c := range claimed {
w.attempt(ctx, c)
}
}
func (w *Worker) attempt(ctx context.Context, c claimedDelivery) {
target, err := w.notifications.Get(ctx, c.notificationTargetID)
if err != nil {
w.fail(ctx, c, 0, fmt.Sprintf("looking up notification target: %v", err))
return
}
req, err := http.NewRequestWithContext(ctx, http.MethodPost, target.WebhookURL, bytes.NewReader(c.payload))
if err != nil {
w.fail(ctx, c, 0, fmt.Sprintf("building request: %v", err))
return
}
req.Header.Set("Content-Type", "application/json")
resp, err := w.http.Do(req)
if err != nil {
w.fail(ctx, c, 0, err.Error())
return
}
defer resp.Body.Close()
if resp.StatusCode >= 200 && resp.StatusCode < 300 {
w.markSent(ctx, c.id, resp.StatusCode)
return
}
w.fail(ctx, c, resp.StatusCode, fmt.Sprintf("non-2xx response: %d", resp.StatusCode))
}
func (w *Worker) markSent(ctx context.Context, id int64, statusCode int) {
_, err := w.pool.Exec(ctx, `
UPDATE delivery_log SET status = 'sent', attempt_count = attempt_count + 1, response_status = $1 WHERE id = $2`,
statusCode, id)
if err != nil {
w.logger.Error("marking delivery sent", "delivery_id", id, "error", err)
}
}
// fail records a failed attempt. If attempts remain, it's rescheduled
// with exponential backoff (2^attempt_count seconds, capped at 1 hour);
// otherwise it's marked permanently failed.
func (w *Worker) fail(ctx context.Context, c claimedDelivery, statusCode int, errMsg string) {
nextAttempt := c.attemptCount + 1
if nextAttempt >= c.maxAttempts {
_, err := w.pool.Exec(ctx, `
UPDATE delivery_log SET status = 'failed', attempt_count = attempt_count + 1, response_status = $1, last_error = $2 WHERE id = $3`,
nullIfZero(statusCode), errMsg, c.id)
if err != nil {
w.logger.Error("marking delivery failed", "delivery_id", c.id, "error", err)
}
return
}
backoff := time.Duration(1<<uint(nextAttempt)) * time.Second
if backoff > time.Hour {
backoff = time.Hour
}
_, err := w.pool.Exec(ctx, `
UPDATE delivery_log
SET status = 'retrying', attempt_count = attempt_count + 1, response_status = $1, last_error = $2, next_attempt_at = now() + $3
WHERE id = $4`,
nullIfZero(statusCode), errMsg, backoff, c.id)
if err != nil {
w.logger.Error("scheduling delivery retry", "delivery_id", c.id, "error", err)
}
}
func nullIfZero(n int) *int {
if n == 0 {
return nil
}
return &n
}
+74
View File
@@ -0,0 +1,74 @@
package evaluator
import (
"fmt"
"github.com/sentry/sentry/alerting/internal/queryclient"
"github.com/sentry/sentry/alerting/internal/rulestore"
)
// evaluateCondition implements fixes 3 and 4 from
// /docs/phase-3-alerting-design.md: a threshold rule's query must
// resolve to exactly one row, and zero (or more than one) rows is
// returned as an error, never coerced to a value -- "nothing ran" and
// "ran and found nothing" are different failure/success modes, and
// conflating them can hide the more alarming case. This function never
// itself decides "condition false" on an error; it returns an error, and
// the caller (evaluator.go) routes that to rulestore.RecordError, never
// to ComputeTransition.
func evaluateCondition(rule rulestore.Rule, result *queryclient.Result) (conditionTrue bool, value *float64, err error) {
switch rule.ConditionType {
case rulestore.ConditionAbsence:
// The window is whatever earliest=/latest= the rule's own query
// already expresses -- no separate window field, per the design doc.
return len(result.Rows) == 0, nil, nil
case rulestore.ConditionThreshold:
if len(result.Rows) != 1 {
return false, nil, fmt.Errorf("threshold rule query returned %d rows, want exactly 1", len(result.Rows))
}
row := result.Rows[0]
if len(row) == 0 {
return false, nil, fmt.Errorf("threshold rule query returned a row with no columns")
}
v, ok := toFloat64(row[0])
if !ok {
return false, nil, fmt.Errorf("threshold rule's first column value %v is not numeric", row[0])
}
if rule.Comparator == nil || rule.ThresholdValue == nil {
return false, nil, fmt.Errorf("threshold rule is missing comparator or threshold_value")
}
return compare(v, *rule.Comparator, *rule.ThresholdValue), &v, nil
default:
return false, nil, fmt.Errorf("unknown condition_type %q", rule.ConditionType)
}
}
func compare(value float64, comparator rulestore.Comparator, threshold float64) bool {
switch comparator {
case rulestore.Gt:
return value > threshold
case rulestore.Gte:
return value >= threshold
case rulestore.Lt:
return value < threshold
case rulestore.Lte:
return value <= threshold
case rulestore.Eq:
return value == threshold
case rulestore.Ne:
return value != threshold
default:
return false
}
}
// toFloat64 handles the one shape /query responses actually come back
// as: Go's encoding/json always decodes JSON numbers into interface{}
// as float64, regardless of whether ClickHouse serialized an integer or
// a float -- there's no separate int64 case to handle.
func toFloat64(v any) (float64, bool) {
f, ok := v.(float64)
return f, ok
}
@@ -0,0 +1,118 @@
package evaluator
import (
"testing"
"github.com/sentry/sentry/alerting/internal/queryclient"
"github.com/sentry/sentry/alerting/internal/rulestore"
)
func thresholdRule(comparator rulestore.Comparator, threshold float64) rulestore.Rule {
return rulestore.Rule{
ConditionType: rulestore.ConditionThreshold,
Comparator: &comparator,
ThresholdValue: &threshold,
}
}
func TestEvaluateConditionThresholdTrue(t *testing.T) {
rule := thresholdRule(rulestore.Gt, 100)
result := &queryclient.Result{Columns: []string{"count"}, Rows: [][]any{{150.0}}}
got, value, err := evaluateCondition(rule, result)
if err != nil {
t.Fatalf("unexpected error: %v", err)
}
if !got {
t.Fatalf("expected condition true for 150 > 100")
}
if value == nil || *value != 150.0 {
t.Fatalf("expected value=150, got %v", value)
}
}
func TestEvaluateConditionThresholdFalse(t *testing.T) {
rule := thresholdRule(rulestore.Gt, 100)
result := &queryclient.Result{Columns: []string{"count"}, Rows: [][]any{{50.0}}}
got, _, err := evaluateCondition(rule, result)
if err != nil {
t.Fatalf("unexpected error: %v", err)
}
if got {
t.Fatalf("expected condition false for 50 > 100")
}
}
// TestEvaluateConditionThresholdZeroRowsIsError pins down fix 4: zero
// rows on a threshold rule must be an error, not silently coerced to 0
// (which would make `count > 100` falsely report "fine" when actually
// nothing ran).
func TestEvaluateConditionThresholdZeroRowsIsError(t *testing.T) {
rule := thresholdRule(rulestore.Gt, 100)
result := &queryclient.Result{Columns: []string{"count"}, Rows: [][]any{}}
_, value, err := evaluateCondition(rule, result)
if err == nil {
t.Fatalf("expected an error for zero rows on a threshold rule, got condition evaluated with value=%v", value)
}
}
func TestEvaluateConditionThresholdMultipleRowsIsError(t *testing.T) {
rule := thresholdRule(rulestore.Gt, 100)
result := &queryclient.Result{Columns: []string{"host", "count"}, Rows: [][]any{{"h1", 50.0}, {"h2", 200.0}}}
_, _, err := evaluateCondition(rule, result)
if err == nil {
t.Fatalf("expected an error for multiple rows on a threshold rule (no per-group alerting, a named non-goal)")
}
}
func TestEvaluateConditionThresholdNonNumericIsError(t *testing.T) {
rule := thresholdRule(rulestore.Gt, 100)
result := &queryclient.Result{Columns: []string{"host"}, Rows: [][]any{{"host-01"}}}
_, _, err := evaluateCondition(rule, result)
if err == nil {
t.Fatalf("expected an error for a non-numeric first column")
}
}
func TestEvaluateConditionAbsenceTrueWhenZeroRows(t *testing.T) {
rule := rulestore.Rule{ConditionType: rulestore.ConditionAbsence}
result := &queryclient.Result{Columns: []string{"message"}, Rows: [][]any{}}
got, value, err := evaluateCondition(rule, result)
if err != nil {
t.Fatalf("unexpected error: %v", err)
}
if !got {
t.Fatalf("expected absence condition true for zero rows")
}
if value != nil {
t.Fatalf("expected nil value for an absence rule, got %v", value)
}
}
func TestEvaluateConditionAbsenceFalseWhenRowsPresent(t *testing.T) {
rule := rulestore.Rule{ConditionType: rulestore.ConditionAbsence}
result := &queryclient.Result{Columns: []string{"message"}, Rows: [][]any{{"something happened"}}}
got, _, err := evaluateCondition(rule, result)
if err != nil {
t.Fatalf("unexpected error: %v", err)
}
if got {
t.Fatalf("expected absence condition false when rows are present")
}
}
func TestEvaluateConditionUnknownTypeIsError(t *testing.T) {
rule := rulestore.Rule{ConditionType: "bogus"}
result := &queryclient.Result{Columns: []string{"count"}, Rows: [][]any{{1.0}}}
_, _, err := evaluateCondition(rule, result)
if err == nil {
t.Fatalf("expected an error for an unknown condition_type")
}
}
+170
View File
@@ -0,0 +1,170 @@
package evaluator
import (
"context"
"log/slog"
"sync"
"time"
"github.com/sentry/sentry/alerting/internal/delivery"
"github.com/sentry/sentry/alerting/internal/notifystore"
"github.com/sentry/sentry/alerting/internal/queryclient"
"github.com/sentry/sentry/alerting/internal/rulestore"
)
// Evaluator is the ticker-driven scheduler -- a bounded worker pool, not
// a workflow engine, per the design doc's explicit instruction. Each
// tick claims up to claimBatchSize due rules (rulestore.ClaimDueRules,
// fix 1's atomic claim) and evaluates them concurrently up to
// workerPoolSize at a time. These are deliberately different numbers --
// see config.EvaluatorConfig's doc comment for the real bug this fixes
// (500 rules due at once, both capped at 20, took 125s to cycle through
// instead of the configured 60s).
type Evaluator struct {
rules *rulestore.Store
notifications *notifystore.Store
queryClient *queryclient.Client
queryTimeout time.Duration
claimBatchSize int
workerPoolSize int
logger *slog.Logger
}
func New(rules *rulestore.Store, notifications *notifystore.Store, queryClient *queryclient.Client, queryTimeout time.Duration, claimBatchSize, workerPoolSize int, logger *slog.Logger) *Evaluator {
return &Evaluator{
rules: rules, notifications: notifications, queryClient: queryClient,
queryTimeout: queryTimeout, claimBatchSize: claimBatchSize, workerPoolSize: workerPoolSize, logger: logger,
}
}
// Run ticks every tickInterval until ctx is cancelled.
func (e *Evaluator) Run(ctx context.Context, tickInterval time.Duration) error {
ticker := time.NewTicker(tickInterval)
defer ticker.Stop()
for {
select {
case <-ctx.Done():
return ctx.Err()
case <-ticker.C:
e.tick(ctx)
}
}
}
func (e *Evaluator) tick(ctx context.Context) {
claimed, err := e.rules.ClaimDueRules(ctx, e.claimBatchSize)
if err != nil {
e.logger.Error("claiming due rules", "error", err)
return
}
if len(claimed) == 0 {
return
}
sem := make(chan struct{}, e.workerPoolSize)
var wg sync.WaitGroup
for _, rule := range claimed {
wg.Add(1)
sem <- struct{}{}
go func(rule rulestore.RuleWithState) {
defer wg.Done()
defer func() { <-sem }()
e.evaluateOne(ctx, rule)
}(rule)
}
wg.Wait()
}
func (e *Evaluator) evaluateOne(ctx context.Context, rule rulestore.RuleWithState) {
result, err := e.queryClient.Query(ctx, rule.Query, rule.QueryLanguage, e.queryTimeout)
if err != nil {
// A failed /query call is an evaluation error, never
// "condition false" -- fix 3. Recording it here, not routing it
// through ComputeTransition at all, is what makes that guarantee
// hold structurally rather than by convention.
e.recordError(ctx, rule.ID, err.Error())
return
}
conditionTrue, value, evalErr := evaluateCondition(rule.Rule, result)
if evalErr != nil {
e.recordError(ctx, rule.ID, evalErr.Error())
return
}
var renotify *time.Duration
if rule.RenotifyIntervalMinutes != nil {
d := time.Duration(*rule.RenotifyIntervalMinutes) * time.Minute
renotify = &d
}
now := time.Now().UTC()
transition := ComputeTransition(TransitionInput{
CurrentState: rule.State.State,
ConditionTrueSince: rule.State.ConditionTrueSince,
FiredAt: rule.State.FiredAt,
LastNotifiedAt: rule.State.LastNotifiedAt,
ForMinutes: time.Duration(rule.ForMinutes) * time.Minute,
RenotifyInterval: renotify,
Now: now,
ConditionTrue: conditionTrue,
})
next := rulestore.AlertState{
State: transition.NextState,
ConditionTrueSince: transition.NextConditionTrueSince,
FiredAt: transition.NextFiredAt,
LastNotifiedAt: transition.NextLastNotifiedAt,
LastValue: value,
}
notify := e.buildNotifyEvent(ctx, rule, transition, value, now)
if err := e.rules.ApplyTransition(ctx, rule.ID, next, notify); err != nil {
e.logger.Error("applying alert state transition", "rule_id", rule.ID, "error", err)
}
}
// buildNotifyEvent resolves the notification target and renders the
// payload for a firing/resolved transition. A lookup or template
// failure here is logged and treated as "no notification this time" --
// the state still transitions correctly (that's the more important
// guarantee), it just means a misconfigured target/template silently
// drops one notification rather than blocking the whole evaluation.
func (e *Evaluator) buildNotifyEvent(ctx context.Context, rule rulestore.RuleWithState, transition TransitionResult, value *float64, now time.Time) *rulestore.NotifyEvent {
if transition.Notify == nil {
return nil
}
target, err := e.notifications.Get(ctx, rule.NotificationTargetID)
if err != nil {
e.logger.Error("looking up notification target", "rule_id", rule.ID, "target_id", rule.NotificationTargetID, "error", err)
return nil
}
comparator := ""
if rule.Comparator != nil {
comparator = string(*rule.Comparator)
}
payload, err := delivery.BuildPayload(*target, delivery.Event{
RuleID: rule.ID, RuleName: rule.Name, EventType: *transition.Notify,
ConditionType: string(rule.ConditionType), Comparator: comparator,
ThresholdValue: rule.ThresholdValue, Value: value, Timestamp: now,
})
if err != nil {
e.logger.Error("building notification payload", "rule_id", rule.ID, "error", err)
return nil
}
return &rulestore.NotifyEvent{
NotificationTargetID: rule.NotificationTargetID,
EventType: *transition.Notify,
Payload: payload,
}
}
func (e *Evaluator) recordError(ctx context.Context, ruleID, msg string) {
if err := e.rules.RecordError(ctx, ruleID, msg); err != nil {
e.logger.Error("recording evaluation error", "rule_id", ruleID, "error", err)
}
}
+121
View File
@@ -0,0 +1,121 @@
// Package evaluator implements the ticker-driven rule scheduler and the
// firing/resolved state machine described in
// /docs/phase-3-alerting-design.md. transitions.go is deliberately pure
// (no DB, no HTTP, no clock reads beyond the `Now` passed in) --
// evaluator.go is the only caller, and it's the single place worth
// exhaustive table-driven testing given how easy this state machine is
// to get subtly wrong.
package evaluator
import (
"time"
"github.com/sentry/sentry/alerting/internal/rulestore"
)
// TransitionInput is everything ComputeTransition needs: the rule's
// current persisted state plus this evaluation's outcome. Evaluation
// *errors* never reach this function at all -- per fix 3 in the design
// doc, an error is recorded separately (rulestore.RecordError) and never
// treated as ConditionTrue: false.
type TransitionInput struct {
CurrentState rulestore.State
ConditionTrueSince *time.Time
FiredAt *time.Time
LastNotifiedAt *time.Time
ForMinutes time.Duration
RenotifyInterval *time.Duration // nil = notify once per firing episode, never again
Now time.Time
ConditionTrue bool
}
// TransitionResult is what to persist (via rulestore.ApplyTransition)
// and, if Notify is non-nil, which event ("firing" or "resolved") to
// enqueue in the same transaction.
type TransitionResult struct {
NextState rulestore.State
NextConditionTrueSince *time.Time
NextFiredAt *time.Time
NextLastNotifiedAt *time.Time
Notify *string
}
// ComputeTransition implements the ok/pending/firing state machine.
// ConditionTrueSince is always a wall-clock timestamp, never a
// consecutive-evaluation counter -- this is what makes the for_minutes
// debounce survive evaluator restarts correctly (a counter would
// silently lose progress across downtime; wall-clock math resumes
// exactly where it left off). Don't "simplify" this into a counter.
func ComputeTransition(in TransitionInput) TransitionResult {
if in.ConditionTrue {
return computeConditionTrue(in)
}
return computeConditionFalse(in)
}
func computeConditionTrue(in TransitionInput) TransitionResult {
switch in.CurrentState {
case rulestore.StatePending:
since := in.ConditionTrueSince
if since == nil {
// Defensive: pending with no recorded since is inconsistent
// persisted state (shouldn't happen -- ok->pending always sets
// it) -- treat as if the condition just became true rather than
// dereferencing a nil or panicking.
now := in.Now
since = &now
}
if in.Now.Sub(*since) >= in.ForMinutes {
return fire(*since, in.Now)
}
return TransitionResult{NextState: rulestore.StatePending, NextConditionTrueSince: since}
case rulestore.StateFiring:
result := TransitionResult{
NextState: rulestore.StateFiring,
NextConditionTrueSince: in.ConditionTrueSince,
NextFiredAt: in.FiredAt,
NextLastNotifiedAt: in.LastNotifiedAt,
}
if in.RenotifyInterval != nil && in.LastNotifiedAt != nil && in.Now.Sub(*in.LastNotifiedAt) >= *in.RenotifyInterval {
now := in.Now
event := "firing"
result.NextLastNotifiedAt = &now
result.Notify = &event
}
return result
default: // ok (or any unexpected value -- the DB CHECK constraint keeps this to 3 values)
since := in.Now
if in.ForMinutes <= 0 {
// for_minutes=0 means "fire on first true evaluation" --
// literally, not after one extra tick spent in `pending`.
return fire(since, in.Now)
}
return TransitionResult{NextState: rulestore.StatePending, NextConditionTrueSince: &since}
}
}
func computeConditionFalse(in TransitionInput) TransitionResult {
switch in.CurrentState {
case rulestore.StateFiring:
event := "resolved"
return TransitionResult{NextState: rulestore.StateOK, Notify: &event}
default: // ok or pending: a false evaluation while pending is a blip
// inside the debounce window -- deliberately no notification, that's
// the entire point of for_minutes existing.
return TransitionResult{NextState: rulestore.StateOK}
}
}
func fire(since, now time.Time) TransitionResult {
firedAt := now
event := "firing"
return TransitionResult{
NextState: rulestore.StateFiring,
NextConditionTrueSince: &since,
NextFiredAt: &firedAt,
NextLastNotifiedAt: &firedAt,
Notify: &event,
}
}
@@ -0,0 +1,193 @@
package evaluator
import (
"testing"
"time"
"github.com/sentry/sentry/alerting/internal/rulestore"
)
var t0 = time.Date(2026, 8, 14, 0, 0, 0, 0, time.UTC)
func ptr[T any](v T) *T { return &v }
func TestComputeTransition(t *testing.T) {
tests := []struct {
name string
in TransitionInput
want TransitionResult
}{
{
name: "ok, condition true, for_minutes>0 -> pending, no notify",
in: TransitionInput{
CurrentState: rulestore.StateOK, ForMinutes: 5 * time.Minute,
Now: t0, ConditionTrue: true,
},
want: TransitionResult{NextState: rulestore.StatePending, NextConditionTrueSince: ptr(t0)},
},
{
name: "ok, condition true, for_minutes=0 -> fires immediately, no extra tick in pending",
in: TransitionInput{
CurrentState: rulestore.StateOK, ForMinutes: 0,
Now: t0, ConditionTrue: true,
},
want: TransitionResult{
NextState: rulestore.StateFiring, NextConditionTrueSince: ptr(t0),
NextFiredAt: ptr(t0), NextLastNotifiedAt: ptr(t0), Notify: ptr("firing"),
},
},
{
name: "pending, still within for_minutes window -> stays pending, no notify",
in: TransitionInput{
CurrentState: rulestore.StatePending, ConditionTrueSince: ptr(t0),
ForMinutes: 5 * time.Minute, Now: t0.Add(2 * time.Minute), ConditionTrue: true,
},
want: TransitionResult{NextState: rulestore.StatePending, NextConditionTrueSince: ptr(t0)},
},
{
name: "pending, exactly at for_minutes boundary -> fires (>=, not >)",
in: TransitionInput{
CurrentState: rulestore.StatePending, ConditionTrueSince: ptr(t0),
ForMinutes: 5 * time.Minute, Now: t0.Add(5 * time.Minute), ConditionTrue: true,
},
want: TransitionResult{
NextState: rulestore.StateFiring, NextConditionTrueSince: ptr(t0),
NextFiredAt: ptr(t0.Add(5 * time.Minute)), NextLastNotifiedAt: ptr(t0.Add(5 * time.Minute)), Notify: ptr("firing"),
},
},
{
name: "pending, past for_minutes -> fires",
in: TransitionInput{
CurrentState: rulestore.StatePending, ConditionTrueSince: ptr(t0),
ForMinutes: 5 * time.Minute, Now: t0.Add(10 * time.Minute), ConditionTrue: true,
},
want: TransitionResult{
NextState: rulestore.StateFiring, NextConditionTrueSince: ptr(t0),
NextFiredAt: ptr(t0.Add(10 * time.Minute)), NextLastNotifiedAt: ptr(t0.Add(10 * time.Minute)), Notify: ptr("firing"),
},
},
{
name: "firing, condition still true, no renotify interval -> stays firing, silent, preserves fired_at/last_notified",
in: TransitionInput{
CurrentState: rulestore.StateFiring, ConditionTrueSince: ptr(t0), FiredAt: ptr(t0), LastNotifiedAt: ptr(t0),
RenotifyInterval: nil, Now: t0.Add(time.Hour), ConditionTrue: true,
},
want: TransitionResult{
NextState: rulestore.StateFiring, NextConditionTrueSince: ptr(t0),
NextFiredAt: ptr(t0), NextLastNotifiedAt: ptr(t0), Notify: nil,
},
},
{
name: "firing, condition still true, renotify interval not yet elapsed -> stays firing, silent",
in: TransitionInput{
CurrentState: rulestore.StateFiring, ConditionTrueSince: ptr(t0), FiredAt: ptr(t0), LastNotifiedAt: ptr(t0),
RenotifyInterval: ptr(4 * time.Hour), Now: t0.Add(time.Hour), ConditionTrue: true,
},
want: TransitionResult{
NextState: rulestore.StateFiring, NextConditionTrueSince: ptr(t0),
NextFiredAt: ptr(t0), NextLastNotifiedAt: ptr(t0), Notify: nil,
},
},
{
name: "firing, condition still true, renotify interval elapsed -> re-fires, updates last_notified only",
in: TransitionInput{
CurrentState: rulestore.StateFiring, ConditionTrueSince: ptr(t0), FiredAt: ptr(t0), LastNotifiedAt: ptr(t0),
RenotifyInterval: ptr(4 * time.Hour), Now: t0.Add(5 * time.Hour), ConditionTrue: true,
},
want: TransitionResult{
NextState: rulestore.StateFiring, NextConditionTrueSince: ptr(t0),
// fired_at (the episode start) is NOT bumped on a renotify -- only last_notified_at moves.
NextFiredAt: ptr(t0), NextLastNotifiedAt: ptr(t0.Add(5 * time.Hour)), Notify: ptr("firing"),
},
},
{
name: "ok, condition false -> stays ok, no-op",
in: TransitionInput{CurrentState: rulestore.StateOK, Now: t0, ConditionTrue: false},
want: TransitionResult{NextState: rulestore.StateOK},
},
{
name: "pending, condition false -> back to ok, NO notification (a blip inside the debounce window)",
in: TransitionInput{
CurrentState: rulestore.StatePending, ConditionTrueSince: ptr(t0),
Now: t0.Add(time.Minute), ConditionTrue: false,
},
want: TransitionResult{NextState: rulestore.StateOK},
},
{
name: "firing, condition false -> resolves, sends resolved notification",
in: TransitionInput{
CurrentState: rulestore.StateFiring, ConditionTrueSince: ptr(t0), FiredAt: ptr(t0), LastNotifiedAt: ptr(t0),
Now: t0.Add(time.Hour), ConditionTrue: false,
},
want: TransitionResult{NextState: rulestore.StateOK, Notify: ptr("resolved")},
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
got := ComputeTransition(tt.in)
if got.NextState != tt.want.NextState {
t.Errorf("NextState = %v, want %v", got.NextState, tt.want.NextState)
}
if !timePtrEqual(got.NextConditionTrueSince, tt.want.NextConditionTrueSince) {
t.Errorf("NextConditionTrueSince = %v, want %v", got.NextConditionTrueSince, tt.want.NextConditionTrueSince)
}
if !timePtrEqual(got.NextFiredAt, tt.want.NextFiredAt) {
t.Errorf("NextFiredAt = %v, want %v", got.NextFiredAt, tt.want.NextFiredAt)
}
if !timePtrEqual(got.NextLastNotifiedAt, tt.want.NextLastNotifiedAt) {
t.Errorf("NextLastNotifiedAt = %v, want %v", got.NextLastNotifiedAt, tt.want.NextLastNotifiedAt)
}
if !strPtrEqual(got.Notify, tt.want.Notify) {
t.Errorf("Notify = %v, want %v", strPtrDeref(got.Notify), strPtrDeref(tt.want.Notify))
}
})
}
}
// TestPendingConditionTrueSinceIsWallClockNotCounter pins down the
// specific property the design doc calls out as easy to accidentally
// "simplify" away: debounce progress survives an evaluator gap (e.g.
// downtime) because condition_true_since is a timestamp, not a tick
// count. Two evaluations three hours apart, both condition=true, with a
// for_minutes=5m debounce -- the second evaluation must fire immediately
// because wall-clock time already satisfies the debounce, regardless of
// how many (or how few) evaluations happened in between.
func TestPendingConditionTrueSinceIsWallClockNotCounter(t *testing.T) {
first := ComputeTransition(TransitionInput{
CurrentState: rulestore.StateOK, ForMinutes: 5 * time.Minute, Now: t0, ConditionTrue: true,
})
if first.NextState != rulestore.StatePending {
t.Fatalf("first eval: state = %v, want pending", first.NextState)
}
// Simulate a large gap (evaluator downtime) before the next evaluation.
second := ComputeTransition(TransitionInput{
CurrentState: rulestore.StatePending, ConditionTrueSince: first.NextConditionTrueSince,
ForMinutes: 5 * time.Minute, Now: t0.Add(3 * time.Hour), ConditionTrue: true,
})
if second.NextState != rulestore.StateFiring {
t.Fatalf("second eval after gap: state = %v, want firing (wall-clock debounce should already be satisfied)", second.NextState)
}
}
func timePtrEqual(a, b *time.Time) bool {
if a == nil || b == nil {
return a == b
}
return a.Equal(*b)
}
func strPtrEqual(a, b *string) bool {
if a == nil || b == nil {
return a == b
}
return *a == *b
}
func strPtrDeref(s *string) string {
if s == nil {
return "<nil>"
}
return *s
}
+261
View File
@@ -0,0 +1,261 @@
// Package httpapi is alerting's REST surface: rule CRUD, notification
// target CRUD, and a read-only delivery log -- mirrors
// api/internal/dashboards' Handler/Store shape (narrow interfaces,
// pgx-backed production implementation, fakes in tests).
package httpapi
import (
"context"
"encoding/json"
"errors"
"log/slog"
"net/http"
"github.com/sentry/sentry/alerting/internal/notifystore"
"github.com/sentry/sentry/alerting/internal/rulestore"
)
type ruleStore interface {
Create(ctx context.Context, r *rulestore.Rule) error
List(ctx context.Context) ([]rulestore.RuleWithState, error)
Get(ctx context.Context, id string) (*rulestore.RuleWithState, error)
Delete(ctx context.Context, id string) error
}
type targetStore interface {
Create(ctx context.Context, t *notifystore.Target) error
List(ctx context.Context) ([]notifystore.Target, error)
Get(ctx context.Context, id string) (*notifystore.Target, error)
Delete(ctx context.Context, id string) error
}
type deliveryReader interface {
ListForRule(ctx context.Context, ruleID string, limit int) ([]rulestore.DeliveryLogEntry, error)
}
type Handler struct {
logger *slog.Logger
rules ruleStore
targets targetStore
deliveries deliveryReader
}
func NewHandler(logger *slog.Logger, rules ruleStore, targets targetStore, deliveries deliveryReader) *Handler {
return &Handler{logger: logger, rules: rules, targets: targets, deliveries: deliveries}
}
func (h *Handler) RegisterRoutes(mux *http.ServeMux) {
mux.HandleFunc("GET /healthz", h.handleHealthz)
mux.HandleFunc("POST /rules", h.handleCreateRule)
mux.HandleFunc("GET /rules", h.handleListRules)
mux.HandleFunc("GET /rules/{id}", h.handleGetRule)
mux.HandleFunc("DELETE /rules/{id}", h.handleDeleteRule)
mux.HandleFunc("GET /rules/{id}/deliveries", h.handleListDeliveries)
mux.HandleFunc("POST /targets", h.handleCreateTarget)
mux.HandleFunc("GET /targets", h.handleListTargets)
mux.HandleFunc("GET /targets/{id}", h.handleGetTarget)
mux.HandleFunc("DELETE /targets/{id}", h.handleDeleteTarget)
}
const maxBodyBytes = 1 << 20 // 1 MiB, same cap as api/internal/queryapi and dashboards
func (h *Handler) handleHealthz(w http.ResponseWriter, _ *http.Request) {
w.WriteHeader(http.StatusOK)
}
// createRuleRequest mirrors rulestore.Rule except Enabled is a pointer:
// a plain bool can't distinguish "omitted" from "explicitly false" on
// decode, and Go's zero value for bool is false -- without this, a
// create request that simply doesn't mention "enabled" would silently
// create a rule the evaluator's claim query never picks up (found by
// actually creating a rule through this endpoint and checking the
// response). Omitted means enabled; only an explicit "enabled": false
// creates a disabled rule.
type createRuleRequest struct {
rulestore.Rule
Enabled *bool `json:"enabled"`
}
func (h *Handler) handleCreateRule(w http.ResponseWriter, r *http.Request) {
var req createRuleRequest
if !decodeJSON(w, r, &req) {
return
}
rule := req.Rule
rule.Enabled = req.Enabled == nil || *req.Enabled
if err := validateRule(&rule); err != nil {
writeError(w, http.StatusBadRequest, err.Error())
return
}
if err := h.rules.Create(r.Context(), &rule); err != nil {
h.logger.Error("creating rule", "error", err)
writeError(w, http.StatusInternalServerError, "creating rule failed")
return
}
writeJSON(w, http.StatusCreated, rule)
}
func (h *Handler) handleListRules(w http.ResponseWriter, r *http.Request) {
rules, err := h.rules.List(r.Context())
if err != nil {
h.logger.Error("listing rules", "error", err)
writeError(w, http.StatusInternalServerError, "listing rules failed")
return
}
writeJSON(w, http.StatusOK, rules)
}
func (h *Handler) handleGetRule(w http.ResponseWriter, r *http.Request) {
rule, err := h.rules.Get(r.Context(), r.PathValue("id"))
if err != nil {
h.writeStoreErr(w, err, "fetching rule")
return
}
writeJSON(w, http.StatusOK, rule)
}
func (h *Handler) handleDeleteRule(w http.ResponseWriter, r *http.Request) {
if err := h.rules.Delete(r.Context(), r.PathValue("id")); err != nil {
h.writeStoreErr(w, err, "deleting rule")
return
}
w.WriteHeader(http.StatusNoContent)
}
func (h *Handler) handleListDeliveries(w http.ResponseWriter, r *http.Request) {
entries, err := h.deliveries.ListForRule(r.Context(), r.PathValue("id"), 100)
if err != nil {
h.logger.Error("listing deliveries", "rule_id", r.PathValue("id"), "error", err)
writeError(w, http.StatusInternalServerError, "listing deliveries failed")
return
}
writeJSON(w, http.StatusOK, entries)
}
func (h *Handler) handleCreateTarget(w http.ResponseWriter, r *http.Request) {
var target notifystore.Target
if !decodeJSON(w, r, &target) {
return
}
if target.Name == "" {
writeError(w, http.StatusBadRequest, "name must not be empty")
return
}
if !notifystore.ValidKind(target.Kind) {
writeError(w, http.StatusBadRequest, "kind must be one of webhook, slack, pagerduty")
return
}
if target.WebhookURL == "" {
writeError(w, http.StatusBadRequest, "webhook_url must not be empty")
return
}
if err := h.targets.Create(r.Context(), &target); err != nil {
h.logger.Error("creating notification target", "error", err)
writeError(w, http.StatusInternalServerError, "creating notification target failed")
return
}
writeJSON(w, http.StatusCreated, target)
}
func (h *Handler) handleListTargets(w http.ResponseWriter, r *http.Request) {
targets, err := h.targets.List(r.Context())
if err != nil {
h.logger.Error("listing notification targets", "error", err)
writeError(w, http.StatusInternalServerError, "listing notification targets failed")
return
}
writeJSON(w, http.StatusOK, targets)
}
func (h *Handler) handleGetTarget(w http.ResponseWriter, r *http.Request) {
target, err := h.targets.Get(r.Context(), r.PathValue("id"))
if err != nil {
h.writeStoreErr(w, err, "fetching notification target")
return
}
writeJSON(w, http.StatusOK, target)
}
func (h *Handler) handleDeleteTarget(w http.ResponseWriter, r *http.Request) {
if err := h.targets.Delete(r.Context(), r.PathValue("id")); err != nil {
h.writeStoreErr(w, err, "deleting notification target")
return
}
w.WriteHeader(http.StatusNoContent)
}
// validateRule enforces the shape the evaluator assumes:
// condition_type-appropriate fields present, a sane interval floor.
// Mirrors the DB CHECK constraints so bad input is rejected with a
// clear message here rather than surfacing as an opaque constraint
// violation from Postgres.
func validateRule(r *rulestore.Rule) error {
if r.Name == "" {
return errBadRequest("name must not be empty")
}
if r.Query == "" {
return errBadRequest("query must not be empty")
}
if r.NotificationTargetID == "" {
return errBadRequest("notification_target_id must not be empty")
}
if r.EvalIntervalSeconds < 30 {
return errBadRequest("eval_interval_seconds must be at least 30")
}
switch r.ConditionType {
case rulestore.ConditionThreshold:
if r.Comparator == nil || !rulestore.ValidComparator(*r.Comparator) {
return errBadRequest("threshold rules require a valid comparator (gt, gte, lt, lte, eq, ne)")
}
if r.ThresholdValue == nil {
return errBadRequest("threshold rules require threshold_value")
}
case rulestore.ConditionAbsence:
// No comparator/threshold_value needed -- absence is "the query
// returned zero rows in its own earliest=/latest= window."
default:
return errBadRequest("condition_type must be \"threshold\" or \"absence\"")
}
return nil
}
type badRequestError string
func (e badRequestError) Error() string { return string(e) }
func errBadRequest(msg string) error { return badRequestError(msg) }
func (h *Handler) writeStoreErr(w http.ResponseWriter, err error, action string) {
if errors.Is(err, rulestore.ErrNotFound) || errors.Is(err, notifystore.ErrNotFound) {
writeError(w, http.StatusNotFound, "not found")
return
}
h.logger.Error(action, "error", err)
writeError(w, http.StatusInternalServerError, action+" failed")
}
func decodeJSON(w http.ResponseWriter, r *http.Request, v any) bool {
r.Body = http.MaxBytesReader(w, r.Body, maxBodyBytes)
if err := json.NewDecoder(r.Body).Decode(v); err != nil {
writeError(w, http.StatusBadRequest, "invalid JSON body: "+err.Error())
return false
}
return true
}
func writeJSON(w http.ResponseWriter, status int, v any) {
w.Header().Set("Content-Type", "application/json")
w.WriteHeader(status)
_ = json.NewEncoder(w).Encode(v)
}
type errorResponse struct {
Error string `json:"error"`
}
func writeError(w http.ResponseWriter, status int, msg string) {
w.Header().Set("Content-Type", "application/json")
w.WriteHeader(status)
_ = json.NewEncoder(w).Encode(errorResponse{Error: msg})
}
+261
View File
@@ -0,0 +1,261 @@
package httpapi
import (
"context"
"io"
"log/slog"
"net/http"
"net/http/httptest"
"strings"
"testing"
"github.com/sentry/sentry/alerting/internal/notifystore"
"github.com/sentry/sentry/alerting/internal/rulestore"
)
type fakeRuleStore struct {
rules map[string]*rulestore.RuleWithState
}
func newFakeRuleStore() *fakeRuleStore {
return &fakeRuleStore{rules: map[string]*rulestore.RuleWithState{}}
}
func (f *fakeRuleStore) Create(_ context.Context, r *rulestore.Rule) error {
r.ID = "rule-1"
f.rules[r.ID] = &rulestore.RuleWithState{Rule: *r, State: rulestore.AlertState{RuleID: r.ID, State: rulestore.StateOK}}
return nil
}
func (f *fakeRuleStore) List(_ context.Context) ([]rulestore.RuleWithState, error) {
var out []rulestore.RuleWithState
for _, r := range f.rules {
out = append(out, *r)
}
return out, nil
}
func (f *fakeRuleStore) Get(_ context.Context, id string) (*rulestore.RuleWithState, error) {
r, ok := f.rules[id]
if !ok {
return nil, rulestore.ErrNotFound
}
return r, nil
}
func (f *fakeRuleStore) Delete(_ context.Context, id string) error {
if _, ok := f.rules[id]; !ok {
return rulestore.ErrNotFound
}
delete(f.rules, id)
return nil
}
type fakeTargetStore struct {
targets map[string]*notifystore.Target
}
func newFakeTargetStore() *fakeTargetStore {
return &fakeTargetStore{targets: map[string]*notifystore.Target{}}
}
func (f *fakeTargetStore) Create(_ context.Context, t *notifystore.Target) error {
t.ID = "target-1"
f.targets[t.ID] = t
return nil
}
func (f *fakeTargetStore) List(_ context.Context) ([]notifystore.Target, error) {
var out []notifystore.Target
for _, t := range f.targets {
out = append(out, *t)
}
return out, nil
}
func (f *fakeTargetStore) Get(_ context.Context, id string) (*notifystore.Target, error) {
t, ok := f.targets[id]
if !ok {
return nil, notifystore.ErrNotFound
}
return t, nil
}
func (f *fakeTargetStore) Delete(_ context.Context, id string) error {
if _, ok := f.targets[id]; !ok {
return notifystore.ErrNotFound
}
delete(f.targets, id)
return nil
}
type fakeDeliveryReader struct {
entries []rulestore.DeliveryLogEntry
}
func (f *fakeDeliveryReader) ListForRule(_ context.Context, _ string, _ int) ([]rulestore.DeliveryLogEntry, error) {
return f.entries, nil
}
func newTestMux(rules ruleStore, targets targetStore, deliveries deliveryReader) *http.ServeMux {
h := NewHandler(slog.New(slog.NewTextHandler(io.Discard, nil)), rules, targets, deliveries)
mux := http.NewServeMux()
h.RegisterRoutes(mux)
return mux
}
func doRequest(t *testing.T, mux *http.ServeMux, method, path, body string) *httptest.ResponseRecorder {
t.Helper()
var r io.Reader
if body != "" {
r = strings.NewReader(body)
}
req := httptest.NewRequest(method, path, r)
rec := httptest.NewRecorder()
mux.ServeHTTP(rec, req)
return rec
}
func TestCreateThresholdRule(t *testing.T) {
targets := newFakeTargetStore()
targets.targets["target-1"] = &notifystore.Target{ID: "target-1"}
mux := newTestMux(newFakeRuleStore(), targets, &fakeDeliveryReader{})
rec := doRequest(t, mux, http.MethodPost, "/rules", `{
"name": "High error rate", "query": "service=api | where status>=500 | stats count",
"condition_type": "threshold", "comparator": "gt", "threshold_value": 100,
"eval_interval_seconds": 60, "notification_target_id": "target-1"
}`)
if rec.Code != http.StatusCreated {
t.Fatalf("status = %d, want 201; body=%s", rec.Code, rec.Body.String())
}
}
// TestCreateRuleDefaultsToEnabledWhenOmitted guards against a real bug
// caught by actually calling this endpoint: a plain `bool` JSON field
// can't distinguish "omitted" from "explicitly false," and Go's zero
// value for bool is false -- without createRuleRequest's *bool handling,
// a create request that simply didn't mention "enabled" silently created
// a rule the evaluator's claim query would never pick up.
func TestCreateRuleDefaultsToEnabledWhenOmitted(t *testing.T) {
rules := newFakeRuleStore()
mux := newTestMux(rules, newFakeTargetStore(), &fakeDeliveryReader{})
rec := doRequest(t, mux, http.MethodPost, "/rules", `{
"name": "no enabled field", "query": "service=api", "condition_type": "absence",
"eval_interval_seconds": 60, "notification_target_id": "target-1"
}`)
if rec.Code != http.StatusCreated {
t.Fatalf("status = %d, want 201; body=%s", rec.Code, rec.Body.String())
}
if !rules.rules["rule-1"].Enabled {
t.Fatalf("expected a rule created without an explicit \"enabled\" field to default to enabled=true")
}
}
func TestCreateRuleRespectsExplicitDisabled(t *testing.T) {
rules := newFakeRuleStore()
mux := newTestMux(rules, newFakeTargetStore(), &fakeDeliveryReader{})
rec := doRequest(t, mux, http.MethodPost, "/rules", `{
"name": "explicitly disabled", "query": "service=api", "condition_type": "absence",
"eval_interval_seconds": 60, "notification_target_id": "target-1", "enabled": false
}`)
if rec.Code != http.StatusCreated {
t.Fatalf("status = %d, want 201; body=%s", rec.Code, rec.Body.String())
}
if rules.rules["rule-1"].Enabled {
t.Fatalf("expected an explicit \"enabled\": false to be respected")
}
}
func TestCreateThresholdRuleRejectsMissingComparator(t *testing.T) {
mux := newTestMux(newFakeRuleStore(), newFakeTargetStore(), &fakeDeliveryReader{})
rec := doRequest(t, mux, http.MethodPost, "/rules", `{
"name": "bad rule", "query": "service=api", "condition_type": "threshold",
"eval_interval_seconds": 60, "notification_target_id": "target-1"
}`)
if rec.Code != http.StatusBadRequest {
t.Fatalf("status = %d, want 400; body=%s", rec.Code, rec.Body.String())
}
}
func TestCreateAbsenceRuleDoesNotRequireComparator(t *testing.T) {
mux := newTestMux(newFakeRuleStore(), newFakeTargetStore(), &fakeDeliveryReader{})
rec := doRequest(t, mux, http.MethodPost, "/rules", `{
"name": "no heartbeat", "query": "service=payments earliest=-5m", "condition_type": "absence",
"eval_interval_seconds": 60, "notification_target_id": "target-1"
}`)
if rec.Code != http.StatusCreated {
t.Fatalf("status = %d, want 201; body=%s", rec.Code, rec.Body.String())
}
}
func TestCreateRuleRejectsShortInterval(t *testing.T) {
mux := newTestMux(newFakeRuleStore(), newFakeTargetStore(), &fakeDeliveryReader{})
rec := doRequest(t, mux, http.MethodPost, "/rules", `{
"name": "too fast", "query": "service=api", "condition_type": "absence",
"eval_interval_seconds": 5, "notification_target_id": "target-1"
}`)
if rec.Code != http.StatusBadRequest {
t.Fatalf("status = %d, want 400; body=%s", rec.Code, rec.Body.String())
}
}
func TestGetRuleNotFound(t *testing.T) {
mux := newTestMux(newFakeRuleStore(), newFakeTargetStore(), &fakeDeliveryReader{})
rec := doRequest(t, mux, http.MethodGet, "/rules/nope", "")
if rec.Code != http.StatusNotFound {
t.Fatalf("status = %d, want 404", rec.Code)
}
}
func TestDeleteRule(t *testing.T) {
rules := newFakeRuleStore()
rules.rules["rule-1"] = &rulestore.RuleWithState{Rule: rulestore.Rule{ID: "rule-1"}}
mux := newTestMux(rules, newFakeTargetStore(), &fakeDeliveryReader{})
rec := doRequest(t, mux, http.MethodDelete, "/rules/rule-1", "")
if rec.Code != http.StatusNoContent {
t.Fatalf("status = %d, want 204", rec.Code)
}
if _, ok := rules.rules["rule-1"]; ok {
t.Fatalf("expected rule to be deleted")
}
}
func TestCreateTargetRejectsInvalidKind(t *testing.T) {
mux := newTestMux(newFakeRuleStore(), newFakeTargetStore(), &fakeDeliveryReader{})
rec := doRequest(t, mux, http.MethodPost, "/targets", `{"name": "x", "kind": "carrier-pigeon", "webhook_url": "https://example.com"}`)
if rec.Code != http.StatusBadRequest {
t.Fatalf("status = %d, want 400; body=%s", rec.Code, rec.Body.String())
}
}
func TestCreateSlackTarget(t *testing.T) {
mux := newTestMux(newFakeRuleStore(), newFakeTargetStore(), &fakeDeliveryReader{})
rec := doRequest(t, mux, http.MethodPost, "/targets", `{"name": "oncall", "kind": "slack", "webhook_url": "https://hooks.slack.com/services/x"}`)
if rec.Code != http.StatusCreated {
t.Fatalf("status = %d, want 201; body=%s", rec.Code, rec.Body.String())
}
}
func TestListDeliveriesForRule(t *testing.T) {
deliveries := &fakeDeliveryReader{entries: []rulestore.DeliveryLogEntry{
{ID: 1, RuleID: "rule-1", EventType: "firing", Status: "sent"},
}}
mux := newTestMux(newFakeRuleStore(), newFakeTargetStore(), deliveries)
rec := doRequest(t, mux, http.MethodGet, "/rules/rule-1/deliveries", "")
if rec.Code != http.StatusOK {
t.Fatalf("status = %d, want 200; body=%s", rec.Code, rec.Body.String())
}
if !strings.Contains(rec.Body.String(), `"status":"sent"`) {
t.Fatalf("expected delivery entry in response, got: %s", rec.Body.String())
}
}
func TestHandleHealthz(t *testing.T) {
mux := newTestMux(newFakeRuleStore(), newFakeTargetStore(), &fakeDeliveryReader{})
rec := doRequest(t, mux, http.MethodGet, "/healthz", "")
if rec.Code != http.StatusOK {
t.Fatalf("status = %d, want 200", rec.Code)
}
}
+24
View File
@@ -0,0 +1,24 @@
// Package httpserver holds cross-handler HTTP concerns for /alerting.
// Deliberately duplicated from api/internal/httpserver rather than
// shared -- see /docs/phase-3-alerting-design.md's component-boundary
// note: no shared Go store/HTTP code between api and alerting, the same
// repo convention that only /proto is shared code.
package httpserver
import "net/http"
// WithCORS is deliberately permissive by default, same posture and
// reasoning as api's: no auth yet, the SvelteKit dev server runs on a
// different origin. Tighten alongside adding real auth.
func WithCORS(next http.Handler, allowedOrigin string) http.Handler {
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
w.Header().Set("Access-Control-Allow-Origin", allowedOrigin)
w.Header().Set("Access-Control-Allow-Methods", "GET, POST, PUT, DELETE, OPTIONS")
w.Header().Set("Access-Control-Allow-Headers", "Content-Type")
if r.Method == http.MethodOptions {
w.WriteHeader(http.StatusNoContent)
return
}
next.ServeHTTP(w, r)
})
}
+114
View File
@@ -0,0 +1,114 @@
// Package notifystore is pgx-backed CRUD for notification_targets.
package notifystore
import (
"context"
"encoding/json"
"errors"
"github.com/google/uuid"
"github.com/jackc/pgx/v5"
"github.com/jackc/pgx/v5/pgxpool"
)
var ErrNotFound = errors.New("not found")
type Kind string
const (
KindWebhook Kind = "webhook"
KindSlack Kind = "slack"
KindPagerDuty Kind = "pagerduty"
)
func ValidKind(k Kind) bool {
switch k {
case KindWebhook, KindSlack, KindPagerDuty:
return true
default:
return false
}
}
type Target struct {
ID string `json:"id"`
TenantID string `json:"tenant_id"`
Name string `json:"name"`
Kind Kind `json:"kind"`
WebhookURL string `json:"webhook_url"`
PayloadTemplate *string `json:"payload_template,omitempty"`
Headers json.RawMessage `json:"headers,omitempty"`
Secret *string `json:"secret,omitempty"`
CreatedBy string `json:"created_by"`
}
type Store struct {
pool *pgxpool.Pool
}
func NewStore(pool *pgxpool.Pool) *Store {
return &Store{pool: pool}
}
func (s *Store) Create(ctx context.Context, t *Target) error {
t.ID = uuid.NewString()
if t.TenantID == "" {
t.TenantID = "default"
}
if t.CreatedBy == "" {
t.CreatedBy = "anonymous"
}
if len(t.Headers) == 0 {
t.Headers = json.RawMessage(`{}`)
}
_, err := s.pool.Exec(ctx, `
INSERT INTO notification_targets (id, tenant_id, name, kind, webhook_url, payload_template, headers, secret, created_by)
VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9)`,
t.ID, t.TenantID, t.Name, t.Kind, t.WebhookURL, t.PayloadTemplate, t.Headers, t.Secret, t.CreatedBy)
return err
}
func (s *Store) List(ctx context.Context) ([]Target, error) {
rows, err := s.pool.Query(ctx, `
SELECT id, tenant_id, name, kind, webhook_url, payload_template, headers, secret, created_by
FROM notification_targets ORDER BY created_at DESC`)
if err != nil {
return nil, err
}
defer rows.Close()
var out []Target
for rows.Next() {
var t Target
if err := rows.Scan(&t.ID, &t.TenantID, &t.Name, &t.Kind, &t.WebhookURL, &t.PayloadTemplate, &t.Headers, &t.Secret, &t.CreatedBy); err != nil {
return nil, err
}
out = append(out, t)
}
return out, rows.Err()
}
func (s *Store) Get(ctx context.Context, id string) (*Target, error) {
var t Target
row := s.pool.QueryRow(ctx, `
SELECT id, tenant_id, name, kind, webhook_url, payload_template, headers, secret, created_by
FROM notification_targets WHERE id = $1`, id)
if err := row.Scan(&t.ID, &t.TenantID, &t.Name, &t.Kind, &t.WebhookURL, &t.PayloadTemplate, &t.Headers, &t.Secret, &t.CreatedBy); err != nil {
if errors.Is(err, pgx.ErrNoRows) {
return nil, ErrNotFound
}
return nil, err
}
return &t, nil
}
func (s *Store) Delete(ctx context.Context, id string) error {
tag, err := s.pool.Exec(ctx, `DELETE FROM notification_targets WHERE id = $1`, id)
if err != nil {
return err
}
if tag.RowsAffected() == 0 {
return ErrNotFound
}
return nil
}
+75
View File
@@ -0,0 +1,75 @@
// Package queryclient is a thin HTTP client to /api's POST /query --
// alerting never imports querylang or talks to ClickHouse/Tantivy
// directly, same precedent sentryctl query and the web UI's dashboard
// panels already set: one query-execution path, reused everywhere.
package queryclient
import (
"bytes"
"context"
"encoding/json"
"fmt"
"net/http"
"time"
)
type Result struct {
Columns []string `json:"columns"`
Rows [][]any `json:"rows"`
}
type errorResponse struct {
Error string `json:"error"`
}
type Client struct {
baseURL string
http *http.Client
}
func New(baseURL string) *Client {
return &Client{baseURL: baseURL, http: &http.Client{}}
}
// Query runs query (already time-range-injected by the caller, if
// applicable) against /api's POST /query and returns the result. A
// non-2xx response or a request/decode failure is returned as an error
// -- callers (the evaluator) treat any error here as an evaluation
// error, never as "condition false" (see
// /docs/phase-3-alerting-design.md's fix 3).
func (c *Client) Query(ctx context.Context, query, language string, timeout time.Duration) (*Result, error) {
ctx, cancel := context.WithTimeout(ctx, timeout)
defer cancel()
body, err := json.Marshal(map[string]string{"query": query, "language": language})
if err != nil {
return nil, fmt.Errorf("encoding query request: %w", err)
}
req, err := http.NewRequestWithContext(ctx, http.MethodPost, c.baseURL+"/query", bytes.NewReader(body))
if err != nil {
return nil, fmt.Errorf("building query request: %w", err)
}
req.Header.Set("Content-Type", "application/json")
resp, err := c.http.Do(req)
if err != nil {
return nil, fmt.Errorf("calling api /query: %w", err)
}
defer resp.Body.Close()
if resp.StatusCode != http.StatusOK {
var errBody errorResponse
_ = json.NewDecoder(resp.Body).Decode(&errBody)
if errBody.Error != "" {
return nil, fmt.Errorf("api /query failed (%d): %s", resp.StatusCode, errBody.Error)
}
return nil, fmt.Errorf("api /query failed with status %d", resp.StatusCode)
}
var result Result
if err := json.NewDecoder(resp.Body).Decode(&result); err != nil {
return nil, fmt.Errorf("decoding query response: %w", err)
}
return &result, nil
}
+357
View File
@@ -0,0 +1,357 @@
// Package rulestore is pgx-backed CRUD for alert_rules/alert_state, plus
// the two operations that make the evaluator's concurrency and delivery
// guarantees hold (see /docs/phase-3-alerting-design.md's fixes 1 and 2):
// ClaimDueRules (atomic claim-before-evaluate) and ApplyTransition (the
// state transition and the delivery_log outbox insert in one DB
// transaction).
package rulestore
import (
"context"
"errors"
"fmt"
"time"
"github.com/google/uuid"
"github.com/jackc/pgx/v5"
"github.com/jackc/pgx/v5/pgxpool"
)
var ErrNotFound = errors.New("not found")
type ConditionType string
const (
ConditionThreshold ConditionType = "threshold"
ConditionAbsence ConditionType = "absence"
)
func ValidConditionType(c ConditionType) bool {
return c == ConditionThreshold || c == ConditionAbsence
}
type Comparator string
const (
Gt Comparator = "gt"
Gte Comparator = "gte"
Lt Comparator = "lt"
Lte Comparator = "lte"
Eq Comparator = "eq"
Ne Comparator = "ne"
)
func ValidComparator(c Comparator) bool {
switch c {
case Gt, Gte, Lt, Lte, Eq, Ne:
return true
default:
return false
}
}
type State string
const (
StateOK State = "ok"
StatePending State = "pending"
StateFiring State = "firing"
)
type Rule struct {
ID string `json:"id"`
TenantID string `json:"tenant_id"`
Name string `json:"name"`
Description string `json:"description"`
Query string `json:"query"`
QueryLanguage string `json:"query_language"`
ConditionType ConditionType `json:"condition_type"`
Comparator *Comparator `json:"comparator,omitempty"`
ThresholdValue *float64 `json:"threshold_value,omitempty"`
EvalIntervalSeconds int `json:"eval_interval_seconds"`
ForMinutes int `json:"for_minutes"`
RenotifyIntervalMinutes *int `json:"renotify_interval_minutes,omitempty"`
NotificationTargetID string `json:"notification_target_id"`
Enabled bool `json:"enabled"`
CreatedBy string `json:"created_by"`
}
type AlertState struct {
RuleID string `json:"rule_id"`
State State `json:"state"`
ConditionTrueSince *time.Time `json:"condition_true_since,omitempty"`
FiredAt *time.Time `json:"fired_at,omitempty"`
LastNotifiedAt *time.Time `json:"last_notified_at,omitempty"`
LastEvaluatedAt *time.Time `json:"last_evaluated_at,omitempty"`
LastEvalStatus string `json:"last_eval_status"`
LastError *string `json:"last_error,omitempty"`
LastValue *float64 `json:"last_value,omitempty"`
ConsecutiveErrors int `json:"consecutive_errors"`
}
// RuleWithState is a rule joined with its current live state, exactly
// what the evaluator needs to run one evaluation and what the read API
// returns.
type RuleWithState struct {
Rule
State AlertState `json:"state"`
}
type Store struct {
pool *pgxpool.Pool
}
func NewStore(pool *pgxpool.Pool) *Store {
return &Store{pool: pool}
}
// Create inserts the rule and its initial alert_state row in one
// transaction. A rule with no alert_state row is silently never picked
// up by ClaimDueRules -- see the design doc's explicit warning about
// this exact failure mode.
func (s *Store) Create(ctx context.Context, r *Rule) error {
r.ID = uuid.NewString()
if r.TenantID == "" {
r.TenantID = "default"
}
if r.CreatedBy == "" {
r.CreatedBy = "anonymous"
}
tx, err := s.pool.Begin(ctx)
if err != nil {
return err
}
defer tx.Rollback(ctx)
_, err = tx.Exec(ctx, `
INSERT INTO alert_rules (id, tenant_id, name, description, query, query_language, condition_type,
comparator, threshold_value, eval_interval_seconds, for_minutes,
renotify_interval_minutes, notification_target_id, enabled, created_by)
VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12, $13, $14, $15)`,
r.ID, r.TenantID, r.Name, r.Description, r.Query, r.QueryLanguage, r.ConditionType,
r.Comparator, r.ThresholdValue, r.EvalIntervalSeconds, r.ForMinutes,
r.RenotifyIntervalMinutes, r.NotificationTargetID, r.Enabled, r.CreatedBy)
if err != nil {
return fmt.Errorf("inserting rule: %w", err)
}
_, err = tx.Exec(ctx, `
INSERT INTO alert_state (rule_id, state, last_eval_status, next_eval_at)
VALUES ($1, 'ok', 'ok', now())`, r.ID)
if err != nil {
return fmt.Errorf("inserting initial alert_state: %w", err)
}
return tx.Commit(ctx)
}
func (s *Store) List(ctx context.Context) ([]RuleWithState, error) {
rows, err := s.pool.Query(ctx, ruleWithStateSelect+" ORDER BY r.created_at DESC")
if err != nil {
return nil, err
}
defer rows.Close()
var out []RuleWithState
for rows.Next() {
rs, err := scanRuleWithState(rows)
if err != nil {
return nil, err
}
out = append(out, rs)
}
return out, rows.Err()
}
func (s *Store) Get(ctx context.Context, id string) (*RuleWithState, error) {
rows, err := s.pool.Query(ctx, ruleWithStateSelect+" WHERE r.id = $1", id)
if err != nil {
return nil, err
}
defer rows.Close()
if !rows.Next() {
return nil, ErrNotFound
}
rs, err := scanRuleWithState(rows)
if err != nil {
return nil, err
}
return &rs, nil
}
func (s *Store) Delete(ctx context.Context, id string) error {
tag, err := s.pool.Exec(ctx, `DELETE FROM alert_rules WHERE id = $1`, id)
if err != nil {
return err
}
if tag.RowsAffected() == 0 {
return ErrNotFound
}
return nil
}
const ruleWithStateSelect = `
SELECT r.id, r.tenant_id, r.name, r.description, r.query, r.query_language, r.condition_type,
r.comparator, r.threshold_value, r.eval_interval_seconds, r.for_minutes,
r.renotify_interval_minutes, r.notification_target_id, r.enabled, r.created_by,
s.state, s.condition_true_since, s.fired_at, s.last_notified_at, s.last_evaluated_at,
s.last_eval_status, s.last_error, s.last_value, s.consecutive_errors
FROM alert_rules r JOIN alert_state s ON s.rule_id = r.id`
func scanRuleWithState(rows pgx.Rows) (RuleWithState, error) {
var rs RuleWithState
err := rows.Scan(
&rs.ID, &rs.TenantID, &rs.Name, &rs.Description, &rs.Query, &rs.QueryLanguage, &rs.ConditionType,
&rs.Comparator, &rs.ThresholdValue, &rs.EvalIntervalSeconds, &rs.ForMinutes,
&rs.RenotifyIntervalMinutes, &rs.NotificationTargetID, &rs.Enabled, &rs.CreatedBy,
&rs.State.State, &rs.State.ConditionTrueSince, &rs.State.FiredAt, &rs.State.LastNotifiedAt, &rs.State.LastEvaluatedAt,
&rs.State.LastEvalStatus, &rs.State.LastError, &rs.State.LastValue, &rs.State.ConsecutiveErrors)
rs.State.RuleID = rs.ID
return rs, err
}
// ClaimDueRules atomically claims up to batchSize due, enabled rules --
// see /docs/phase-3-alerting-design.md's fix 1. next_eval_at is bumped
// forward *before* returning, so a second scheduler tick (or a second
// evaluator replica, later) can't re-select the same rule while this
// one is still being evaluated. SKIP LOCKED means a concurrent claim
// query never blocks on rows another claim already has locked -- it
// just moves on to the next candidate.
func (s *Store) ClaimDueRules(ctx context.Context, batchSize int) ([]RuleWithState, error) {
rows, err := s.pool.Query(ctx, `
WITH due AS (
SELECT s.rule_id
FROM alert_state s
JOIN alert_rules r ON r.id = s.rule_id
WHERE s.next_eval_at <= now() AND r.enabled
ORDER BY s.next_eval_at
LIMIT $1
FOR UPDATE OF s SKIP LOCKED
)
UPDATE alert_state s
SET next_eval_at = now() + make_interval(secs => r.eval_interval_seconds),
claimed_at = now()
FROM due, alert_rules r
WHERE s.rule_id = due.rule_id AND r.id = s.rule_id
RETURNING r.id, r.tenant_id, r.name, r.description, r.query, r.query_language, r.condition_type,
r.comparator, r.threshold_value, r.eval_interval_seconds, r.for_minutes,
r.renotify_interval_minutes, r.notification_target_id, r.enabled, r.created_by,
s.state, s.condition_true_since, s.fired_at, s.last_notified_at, s.last_evaluated_at,
s.last_eval_status, s.last_error, s.last_value, s.consecutive_errors`,
batchSize)
if err != nil {
return nil, err
}
defer rows.Close()
var out []RuleWithState
for rows.Next() {
rs, err := scanRuleWithState(rows)
if err != nil {
return nil, err
}
out = append(out, rs)
}
return out, rows.Err()
}
// NotifyEvent, if non-nil, is inserted into delivery_log in the same
// transaction as the state update -- the transactional outbox from fix
// 2. Payload is the already-rendered notification body (rulestore
// doesn't know how to format one; that's internal/delivery's job).
type NotifyEvent struct {
NotificationTargetID string
EventType string // "firing" | "resolved"
Payload []byte // JSON
}
// ApplyTransition writes a successful evaluation's outcome: the new
// alert_state fields, and -- atomically, in the same transaction -- a
// pending delivery_log row if notify is non-nil. See fix 2: "decided to
// notify" becomes durable exactly once, in lockstep with the state
// change, before any network call to a notification target is
// attempted.
func (s *Store) ApplyTransition(ctx context.Context, ruleID string, next AlertState, notify *NotifyEvent) error {
tx, err := s.pool.Begin(ctx)
if err != nil {
return err
}
defer tx.Rollback(ctx)
_, err = tx.Exec(ctx, `
UPDATE alert_state SET
state = $1, condition_true_since = $2, fired_at = $3, last_notified_at = $4,
last_evaluated_at = now(), last_eval_status = 'ok', last_error = NULL,
last_value = $5, consecutive_errors = 0
WHERE rule_id = $6`,
next.State, next.ConditionTrueSince, next.FiredAt, next.LastNotifiedAt, next.LastValue, ruleID)
if err != nil {
return fmt.Errorf("updating alert_state: %w", err)
}
if notify != nil {
_, err = tx.Exec(ctx, `
INSERT INTO delivery_log (rule_id, notification_target_id, event_type, status, next_attempt_at, payload)
VALUES ($1, $2, $3, 'pending', now(), $4)`,
ruleID, notify.NotificationTargetID, notify.EventType, notify.Payload)
if err != nil {
return fmt.Errorf("inserting delivery_log outbox row: %w", err)
}
}
return tx.Commit(ctx)
}
// DeliveryLogEntry is the read shape the web UI's delivery log view and
// the httpapi read endpoint use -- deliberately excludes `payload`
// (the rendered notification body isn't needed for the list view, and
// keeping it out of the default read path avoids echoing
// target-specific formatting, e.g. a rendered webhook secret, back over
// HTTP by default).
type DeliveryLogEntry struct {
ID int64 `json:"id"`
RuleID string `json:"rule_id"`
NotificationTargetID string `json:"notification_target_id"`
EventType string `json:"event_type"`
Status string `json:"status"`
AttemptCount int `json:"attempt_count"`
LastError *string `json:"last_error,omitempty"`
ResponseStatus *int `json:"response_status,omitempty"`
CreatedAt time.Time `json:"created_at"`
}
// ListForRule reads delivery_log's UI-visible role -- "why didn't I get
// paged" -- most recent first.
func (s *Store) ListForRule(ctx context.Context, ruleID string, limit int) ([]DeliveryLogEntry, error) {
rows, err := s.pool.Query(ctx, `
SELECT id, rule_id, notification_target_id, event_type, status, attempt_count, last_error, response_status, created_at
FROM delivery_log WHERE rule_id = $1 ORDER BY created_at DESC LIMIT $2`, ruleID, limit)
if err != nil {
return nil, err
}
defer rows.Close()
var out []DeliveryLogEntry
for rows.Next() {
var e DeliveryLogEntry
if err := rows.Scan(&e.ID, &e.RuleID, &e.NotificationTargetID, &e.EventType, &e.Status, &e.AttemptCount, &e.LastError, &e.ResponseStatus, &e.CreatedAt); err != nil {
return nil, err
}
out = append(out, e)
}
return out, rows.Err()
}
// RecordError updates only the error-tracking fields on alert_state --
// state itself is left untouched, per fix 3: an evaluation error must
// never transition state (an error is not "condition false").
func (s *Store) RecordError(ctx context.Context, ruleID, errMsg string) error {
_, err := s.pool.Exec(ctx, `
UPDATE alert_state SET
last_evaluated_at = now(), last_eval_status = 'error', last_error = $1,
consecutive_errors = consecutive_errors + 1
WHERE rule_id = $2`, errMsg, ruleID)
return err
}