Phase 3: dashboards and alerting
Saved, shareable multi-panel dashboards (table/line/bar/single-stat panels via gridstack + uPlot, global + per-panel time range, JSON export/import) and threshold/absence alert rules with an ok/pending/firing evaluator and webhook/Slack/PagerDuty delivery. - New /metadata component: Postgres control-plane store for dashboards, panels, notification targets, alert rules/state, and delivery log -- see docs/phase-3-dashboard-design.md for why ClickHouse's MergeTree family isn't a fit for this access pattern (needs real row-level locking and read-your-writes consistency). - api/internal/dashboards: dashboard/panel CRUD, pure -- panel query execution stays client-side, reusing the existing /query endpoint. - New /alerting service: rule/target CRUD, a ticker-driven evaluator (claim-then-evaluate concurrency control, transactional-outbox delivery, query errors and threshold zero-rows never coerced into a false transition) and webhook/Slack/PagerDuty delivery with retry/backoff. See docs/phase-3-alerting-design.md for the full state-machine design and the four correctness properties it implements. - web: /dashboards and /alerts UIs; cli: sentryctl dashboards/alerts list/get/apply, seeding a future Terraform provider's JSON contract. - hack/alert-load-test: 500 rules against real ClickHouse data, real measured results in docs/phase-3-runbook.md. Five real bugs found by actually running this against a live stack (documented in the runbook, not just fixed silently): a latent Phase 2 bug where ClickHouse rejected the timestamp format used for earliest=/latest= queries; a "now" literal token injected into query text; a GridStack/uPlot layout-timing race; JS's Date.parse being too lenient to use as a timestamp-detection heuristic; a rule's "enabled" field silently defaulting to false when omitted; and the evaluator's claim-batch-size and worker-pool-concurrency defaulting to the same value, causing 500 concurrently-due rules to take 125s to cycle through instead of the configured 60s.
This commit is contained in:
@@ -0,0 +1,12 @@
|
||||
# alerting never needs /proto (it talks to /api over plain HTTP, no gRPC),
|
||||
# so unlike api/ingest/search this build context is just alerting/ itself,
|
||||
# same shape as cli/Dockerfile:
|
||||
# docker build -f alerting/Dockerfile -t sentry-alerting alerting/
|
||||
FROM golang:1.25-alpine AS builder
|
||||
WORKDIR /src
|
||||
COPY . .
|
||||
RUN CGO_ENABLED=0 GOOS=linux go build -o /out/alerting ./cmd/alerting
|
||||
|
||||
FROM gcr.io/distroless/static-debian12
|
||||
COPY --from=builder /out/alerting /alerting
|
||||
ENTRYPOINT ["/alerting"]
|
||||
@@ -0,0 +1,98 @@
|
||||
# alerting
|
||||
|
||||
Alert rule CRUD, the ticker-driven evaluator, and webhook/Slack/PagerDuty
|
||||
delivery. See `/docs/phase-3-alerting-design.md` for the full design
|
||||
(data model, the ok/pending/firing state machine, and the four
|
||||
correctness properties this implementation follows exactly).
|
||||
|
||||
## Running
|
||||
|
||||
```sh
|
||||
POSTGRES_PASSWORD=sentry-dev-only API_QUERY_URL=http://localhost:8080 go run ./cmd/alerting
|
||||
```
|
||||
|
||||
Talks to the same `sentry_metadata` Postgres database as `/api`
|
||||
(different tables — see `/metadata/README.md`), and to `/api`'s
|
||||
`POST /query` over plain HTTP for rule evaluation. Never connects to
|
||||
ClickHouse or Tantivy directly.
|
||||
|
||||
## HTTP API
|
||||
|
||||
```
|
||||
POST /rules create a rule
|
||||
GET /rules list rules (with current state)
|
||||
GET /rules/{id} get a rule (with current state)
|
||||
DELETE /rules/{id}
|
||||
GET /rules/{id}/deliveries delivery log for a rule, most recent first
|
||||
|
||||
POST /targets create a notification target
|
||||
GET /targets
|
||||
GET /targets/{id}
|
||||
DELETE /targets/{id}
|
||||
|
||||
GET /healthz
|
||||
```
|
||||
|
||||
A rule's `condition_type` is `"threshold"` (requires `comparator` +
|
||||
`threshold_value`, and the query must resolve to exactly one row) or
|
||||
`"absence"` (fires when the query returns zero rows in its own
|
||||
`earliest=`/`latest=` window — no separate window field). A notification
|
||||
target's `kind` is `"webhook"`, `"slack"`, or `"pagerduty"` — all three
|
||||
deliver via the same HTTP POST + retry/backoff mechanism
|
||||
(`internal/delivery/webhook.go`); slack/pagerduty are payload formatters
|
||||
only, not separate delivery paths.
|
||||
|
||||
## Environment variables
|
||||
|
||||
| Var | Default |
|
||||
|---|---|
|
||||
| `HTTP_LISTEN_ADDR` | `:8081` |
|
||||
| `POSTGRES_ADDR` | `localhost:5432` |
|
||||
| `POSTGRES_DATABASE` | `sentry_metadata` |
|
||||
| `POSTGRES_USERNAME` | `sentry` |
|
||||
| `POSTGRES_PASSWORD` | (empty — must be set) |
|
||||
| `API_QUERY_URL` | `http://localhost:8080` |
|
||||
| `CORS_ALLOWED_ORIGIN` | `*` |
|
||||
| `EVALUATOR_TICK_SECONDS` | `5` — how often the scheduler checks for due rules |
|
||||
| `EVALUATOR_CLAIM_BATCH_SIZE` | `1000` — how many due rules one tick can pull off the queue |
|
||||
| `EVALUATOR_WORKER_POOL_SIZE` | `20` — bounded concurrency for `/query` calls within a claimed batch |
|
||||
| `EVALUATOR_QUERY_TIMEOUT_SECONDS` | `30` — per-evaluation `POST /query` timeout |
|
||||
|
||||
`EVALUATOR_CLAIM_BATCH_SIZE` and `EVALUATOR_WORKER_POOL_SIZE` are
|
||||
deliberately separate knobs, not the same number — see
|
||||
`internal/config/config.go`'s doc comment for the real bug this
|
||||
separation fixes (found by `hack/alert-load-test`, see
|
||||
`/docs/phase-3-runbook.md`): with both capped at 20, 500 rules due at
|
||||
once took 125s to cycle through instead of the configured 60s.
|
||||
|
||||
## Package layout
|
||||
|
||||
```
|
||||
cmd/alerting/ wires config, Postgres pool, api client; runs the
|
||||
HTTP server + evaluator + delivery worker concurrently (errgroup)
|
||||
internal/httpapi/ REST handlers -- Handler/RegisterRoutes, same shape as api/internal/dashboards
|
||||
internal/rulestore/ pgx CRUD for alert_rules + alert_state; ClaimDueRules
|
||||
(fix 1's atomic claim) and ApplyTransition (fix 2's transactional outbox)
|
||||
internal/notifystore/ pgx CRUD for notification_targets
|
||||
internal/queryclient/ thin HTTP client to api's POST /query -- no querylang import here
|
||||
internal/evaluator/ the ticker + worker pool; transitions.go is the pure,
|
||||
exhaustively-tested ok/pending/firing state machine;
|
||||
condition.go implements fixes 3/4 (errors never
|
||||
coerced to "condition false"; threshold zero-rows
|
||||
is an error, not a 0)
|
||||
internal/delivery/ webhook.go is the claim-and-send worker (all three
|
||||
kinds go through it); slack.go/pagerduty.go are
|
||||
payload formatters only
|
||||
```
|
||||
|
||||
## Building & testing
|
||||
|
||||
```sh
|
||||
go build ./...
|
||||
go vet ./...
|
||||
go test ./...
|
||||
```
|
||||
|
||||
```sh
|
||||
docker build -f Dockerfile -t sentry-alerting . # context is alerting/, not the repo root -- no /proto needed
|
||||
```
|
||||
@@ -0,0 +1,141 @@
|
||||
// Command alerting is Sentry's alert rule evaluator and delivery
|
||||
// service: rule/target CRUD, the ticker-driven ok/pending/firing
|
||||
// evaluator, and the webhook/Slack/PagerDuty delivery worker. See
|
||||
// /docs/phase-3-alerting-design.md. Never talks to ClickHouse/Tantivy
|
||||
// directly -- rule queries run through /api's POST /query
|
||||
// (internal/queryclient), same precedent sentryctl query and the web
|
||||
// UI's dashboard panels already set.
|
||||
package main
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"log/slog"
|
||||
"net/http"
|
||||
"os"
|
||||
"os/signal"
|
||||
"strings"
|
||||
"syscall"
|
||||
"time"
|
||||
|
||||
"github.com/jackc/pgx/v5/pgxpool"
|
||||
"golang.org/x/sync/errgroup"
|
||||
|
||||
"github.com/sentry/sentry/alerting/internal/config"
|
||||
"github.com/sentry/sentry/alerting/internal/delivery"
|
||||
"github.com/sentry/sentry/alerting/internal/evaluator"
|
||||
"github.com/sentry/sentry/alerting/internal/httpapi"
|
||||
"github.com/sentry/sentry/alerting/internal/httpserver"
|
||||
"github.com/sentry/sentry/alerting/internal/notifystore"
|
||||
"github.com/sentry/sentry/alerting/internal/queryclient"
|
||||
"github.com/sentry/sentry/alerting/internal/rulestore"
|
||||
)
|
||||
|
||||
func main() {
|
||||
logger := slog.New(slog.NewJSONHandler(os.Stdout, nil))
|
||||
|
||||
cfg, err := config.Load()
|
||||
if err != nil {
|
||||
logger.Error("loading config", "error", err)
|
||||
os.Exit(1)
|
||||
}
|
||||
|
||||
// -healthcheck: self-check mode for Docker's HEALTHCHECK, mirrors
|
||||
// api/cmd/api/main.go's runHealthcheck -- this image is distroless too
|
||||
// (no shell, no wget), so the compose healthcheck execs this binary
|
||||
// against itself.
|
||||
if len(os.Args) > 1 && os.Args[1] == "-healthcheck" {
|
||||
os.Exit(runHealthcheck(cfg.HTTPListenAddr))
|
||||
}
|
||||
|
||||
ctx, stop := signal.NotifyContext(context.Background(), syscall.SIGINT, syscall.SIGTERM)
|
||||
defer stop()
|
||||
|
||||
pgDSN := fmt.Sprintf("postgres://%s:%s@%s/%s", cfg.Postgres.Username, cfg.Postgres.Password, cfg.Postgres.Addr, cfg.Postgres.Database)
|
||||
pgPool, err := pgxpool.New(ctx, pgDSN)
|
||||
if err != nil {
|
||||
logger.Error("opening postgres pool", "error", err)
|
||||
os.Exit(1)
|
||||
}
|
||||
defer pgPool.Close()
|
||||
|
||||
if err := pgPool.Ping(ctx); err != nil {
|
||||
logger.Error("pinging postgres", "error", err)
|
||||
os.Exit(1)
|
||||
}
|
||||
|
||||
rules := rulestore.NewStore(pgPool)
|
||||
targets := notifystore.NewStore(pgPool)
|
||||
qc := queryclient.New(cfg.APIQueryURL)
|
||||
|
||||
handler := httpapi.NewHandler(logger, rules, targets, rules)
|
||||
mux := http.NewServeMux()
|
||||
handler.RegisterRoutes(mux)
|
||||
srv := &http.Server{
|
||||
Addr: cfg.HTTPListenAddr,
|
||||
Handler: httpserver.WithCORS(mux, cfg.CORSAllowedOrigin),
|
||||
}
|
||||
|
||||
eval := evaluator.New(rules, targets, qc, cfg.Evaluator.QueryTimeout, cfg.Evaluator.ClaimBatchSize, cfg.Evaluator.WorkerPoolSize, logger)
|
||||
deliveryWorker := delivery.NewWorker(pgPool, targets, logger)
|
||||
|
||||
g, ctx := errgroup.WithContext(ctx)
|
||||
|
||||
g.Go(func() error {
|
||||
logger.Info("alerting listening", "addr", cfg.HTTPListenAddr)
|
||||
errCh := make(chan error, 1)
|
||||
go func() { errCh <- srv.ListenAndServe() }()
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
shutdownCtx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
|
||||
defer cancel()
|
||||
if err := srv.Shutdown(shutdownCtx); err != nil {
|
||||
return fmt.Errorf("graceful shutdown: %w", err)
|
||||
}
|
||||
return nil
|
||||
case err := <-errCh:
|
||||
if err != nil && err != http.ErrServerClosed {
|
||||
return fmt.Errorf("http server exited: %w", err)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
})
|
||||
|
||||
g.Go(func() error {
|
||||
logger.Info("evaluator started", "tick_interval", cfg.Evaluator.TickInterval, "worker_pool_size", cfg.Evaluator.WorkerPoolSize)
|
||||
if err := eval.Run(ctx, cfg.Evaluator.TickInterval); err != nil && ctx.Err() == nil {
|
||||
return fmt.Errorf("evaluator exited: %w", err)
|
||||
}
|
||||
return nil
|
||||
})
|
||||
|
||||
g.Go(func() error {
|
||||
logger.Info("delivery worker started")
|
||||
if err := deliveryWorker.Run(ctx, cfg.Evaluator.TickInterval); err != nil && ctx.Err() == nil {
|
||||
return fmt.Errorf("delivery worker exited: %w", err)
|
||||
}
|
||||
return nil
|
||||
})
|
||||
|
||||
if err := g.Wait(); err != nil {
|
||||
logger.Error("alerting exited with error", "error", err)
|
||||
os.Exit(1)
|
||||
}
|
||||
}
|
||||
|
||||
func runHealthcheck(listenAddr string) int {
|
||||
addr := listenAddr
|
||||
if strings.HasPrefix(addr, ":") {
|
||||
addr = "localhost" + addr
|
||||
}
|
||||
client := http.Client{Timeout: 3 * time.Second}
|
||||
resp, err := client.Get("http://" + addr + "/healthz")
|
||||
if err != nil {
|
||||
return 1
|
||||
}
|
||||
defer resp.Body.Close()
|
||||
if resp.StatusCode != http.StatusOK {
|
||||
return 1
|
||||
}
|
||||
return 0
|
||||
}
|
||||
@@ -0,0 +1,16 @@
|
||||
module github.com/sentry/sentry/alerting
|
||||
|
||||
go 1.25.0
|
||||
|
||||
require (
|
||||
github.com/google/uuid v1.6.0
|
||||
github.com/jackc/pgx/v5 v5.10.0
|
||||
golang.org/x/sync v0.22.0
|
||||
)
|
||||
|
||||
require (
|
||||
github.com/jackc/pgpassfile v1.0.0 // indirect
|
||||
github.com/jackc/pgservicefile v0.0.0-20240606120523-5a60cdf6a761 // indirect
|
||||
github.com/jackc/puddle/v2 v2.2.2 // indirect
|
||||
golang.org/x/text v0.29.0 // indirect
|
||||
)
|
||||
@@ -0,0 +1,28 @@
|
||||
github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
|
||||
github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c=
|
||||
github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
|
||||
github.com/google/uuid v1.6.0 h1:NIvaJDMOsjHA8n1jAhLSgzrAzy1Hgr+hNrb57e+94F0=
|
||||
github.com/google/uuid v1.6.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo=
|
||||
github.com/jackc/pgpassfile v1.0.0 h1:/6Hmqy13Ss2zCq62VdNG8tM1wchn8zjSGOBJ6icpsIM=
|
||||
github.com/jackc/pgpassfile v1.0.0/go.mod h1:CEx0iS5ambNFdcRtxPj5JhEz+xB6uRky5eyVu/W2HEg=
|
||||
github.com/jackc/pgservicefile v0.0.0-20240606120523-5a60cdf6a761 h1:iCEnooe7UlwOQYpKFhBabPMi4aNAfoODPEFNiAnClxo=
|
||||
github.com/jackc/pgservicefile v0.0.0-20240606120523-5a60cdf6a761/go.mod h1:5TJZWKEWniPve33vlWYSoGYefn3gLQRzjfDlhSJ9ZKM=
|
||||
github.com/jackc/pgx/v5 v5.10.0 h1:VhSvgU2jSli8o3AqIEOTJr7rZwAEUVo4E4XhR94Zfr0=
|
||||
github.com/jackc/pgx/v5 v5.10.0/go.mod h1:mal1tBGAFfLHvZzaYh77YS/eC6IX9OWbRV1QIIM0Jn4=
|
||||
github.com/jackc/puddle/v2 v2.2.2 h1:PR8nw+E/1w0GLuRFSmiioY6UooMp6KJv0/61nB7icHo=
|
||||
github.com/jackc/puddle/v2 v2.2.2/go.mod h1:vriiEXHvEE654aYKXXjOvZM39qJ0q+azkZFrfEOc3H4=
|
||||
github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM=
|
||||
github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
|
||||
github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME=
|
||||
github.com/stretchr/testify v1.3.0/go.mod h1:M5WIy9Dh21IEIfnGCwXGc5bZfKNJtfHm1UVUgZn+9EI=
|
||||
github.com/stretchr/testify v1.7.0/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg=
|
||||
github.com/stretchr/testify v1.11.1 h1:7s2iGBzp5EwR7/aIZr8ao5+dra3wiQyKjjFuvgVKu7U=
|
||||
github.com/stretchr/testify v1.11.1/go.mod h1:wZwfW3scLgRK+23gO65QZefKpKQRnfz6sD981Nm4B6U=
|
||||
golang.org/x/sync v0.22.0 h1:SZjpbeLmrCk4xhRSZFNZW5gFUeCeFgjekvI/+gfScek=
|
||||
golang.org/x/sync v0.22.0/go.mod h1:9xrNwdLfx4jkKbNva9FpL6vEN7evnE43NNNJQ2LF3+0=
|
||||
golang.org/x/text v0.29.0 h1:1neNs90w9YzJ9BocxfsQNHKuAT4pkghyXc4nhZ6sJvk=
|
||||
golang.org/x/text v0.29.0/go.mod h1:7MhJOA9CD2qZyOKYazxdYMF85OwPdEr9jTtBpO7ydH4=
|
||||
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
|
||||
gopkg.in/yaml.v3 v3.0.0-20200313102051-9f266ea9e77c/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
|
||||
gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA=
|
||||
gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
|
||||
@@ -0,0 +1,96 @@
|
||||
// Package config loads alerting's configuration from environment
|
||||
// variables, same convention as /api and /ingest: no config file format.
|
||||
package config
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"os"
|
||||
"strconv"
|
||||
"time"
|
||||
)
|
||||
|
||||
type Config struct {
|
||||
HTTPListenAddr string
|
||||
Postgres PostgresConfig
|
||||
APIQueryURL string // base URL of /api, e.g. http://api:8080 -- alerting never talks to ClickHouse/Tantivy directly
|
||||
CORSAllowedOrigin string
|
||||
Evaluator EvaluatorConfig
|
||||
}
|
||||
|
||||
type PostgresConfig struct {
|
||||
Addr string
|
||||
Database string
|
||||
Username string
|
||||
Password string
|
||||
}
|
||||
|
||||
type EvaluatorConfig struct {
|
||||
TickInterval time.Duration // how often the scheduler checks for due rules
|
||||
// ClaimBatchSize and WorkerPoolSize are deliberately separate knobs,
|
||||
// not the same number: ClaimBatchSize bounds how many due rules one
|
||||
// tick pulls off the queue (needs to be large enough to drain a
|
||||
// backlog when many rules share a due time, e.g. right after bulk
|
||||
// creation), while WorkerPoolSize bounds concurrent /query calls
|
||||
// within that batch. Found by actually running hack/alert-load-test
|
||||
// with 500 rules and both numbers defaulted to the same value
|
||||
// (20): the evaluator took 125s to cycle through 500 due rules
|
||||
// instead of the configured 60s eval_interval_seconds, because each
|
||||
// 5s tick could only claim 20 rules regardless of how many more were
|
||||
// already due -- see /docs/phase-3-runbook.md's load-test section.
|
||||
ClaimBatchSize int
|
||||
WorkerPoolSize int
|
||||
QueryTimeout time.Duration // per-evaluation POST /query timeout
|
||||
}
|
||||
|
||||
func Load() (Config, error) {
|
||||
cfg := Config{
|
||||
HTTPListenAddr: getenv("HTTP_LISTEN_ADDR", ":8081"),
|
||||
Postgres: PostgresConfig{
|
||||
Addr: getenv("POSTGRES_ADDR", "localhost:5432"),
|
||||
Database: getenv("POSTGRES_DATABASE", "sentry_metadata"),
|
||||
Username: getenv("POSTGRES_USERNAME", "sentry"),
|
||||
Password: getenv("POSTGRES_PASSWORD", ""),
|
||||
},
|
||||
APIQueryURL: getenv("API_QUERY_URL", "http://localhost:8080"),
|
||||
// Same "no auth yet" tradeoff as api's CORSAllowedOrigin default --
|
||||
// see api/internal/config/config.go's comment, same reasoning here.
|
||||
CORSAllowedOrigin: getenv("CORS_ALLOWED_ORIGIN", "*"),
|
||||
}
|
||||
|
||||
tickSec, err := strconv.Atoi(getenv("EVALUATOR_TICK_SECONDS", "5"))
|
||||
if err != nil {
|
||||
return Config{}, fmt.Errorf("EVALUATOR_TICK_SECONDS: %w", err)
|
||||
}
|
||||
cfg.Evaluator.TickInterval = time.Duration(tickSec) * time.Second
|
||||
|
||||
poolSize, err := strconv.Atoi(getenv("EVALUATOR_WORKER_POOL_SIZE", "20"))
|
||||
if err != nil {
|
||||
return Config{}, fmt.Errorf("EVALUATOR_WORKER_POOL_SIZE: %w", err)
|
||||
}
|
||||
cfg.Evaluator.WorkerPoolSize = poolSize
|
||||
|
||||
// Default well above WorkerPoolSize: this is "how many due rules can
|
||||
// one tick pull off the queue," not a concurrency limit -- the
|
||||
// worker pool below still bounds actual concurrent /query calls
|
||||
// regardless of how large a batch gets claimed.
|
||||
claimBatchSize, err := strconv.Atoi(getenv("EVALUATOR_CLAIM_BATCH_SIZE", "1000"))
|
||||
if err != nil {
|
||||
return Config{}, fmt.Errorf("EVALUATOR_CLAIM_BATCH_SIZE: %w", err)
|
||||
}
|
||||
cfg.Evaluator.ClaimBatchSize = claimBatchSize
|
||||
|
||||
queryTimeoutSec, err := strconv.Atoi(getenv("EVALUATOR_QUERY_TIMEOUT_SECONDS", "30"))
|
||||
if err != nil {
|
||||
return Config{}, fmt.Errorf("EVALUATOR_QUERY_TIMEOUT_SECONDS: %w", err)
|
||||
}
|
||||
cfg.Evaluator.QueryTimeout = time.Duration(queryTimeoutSec) * time.Second
|
||||
|
||||
return cfg, nil
|
||||
}
|
||||
|
||||
func getenv(key, fallback string) string {
|
||||
if v := os.Getenv(key); v != "" {
|
||||
return v
|
||||
}
|
||||
return fallback
|
||||
}
|
||||
@@ -0,0 +1,88 @@
|
||||
package delivery
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/sentry/sentry/alerting/internal/notifystore"
|
||||
)
|
||||
|
||||
func TestBuildPayloadGenericDefaultShape(t *testing.T) {
|
||||
target := notifystore.Target{Kind: notifystore.KindWebhook}
|
||||
value := 142.0
|
||||
payload, err := BuildPayload(target, Event{
|
||||
RuleID: "r1", RuleName: "High error rate", EventType: "firing",
|
||||
Value: &value, Timestamp: time.Date(2026, 8, 14, 0, 0, 0, 0, time.UTC),
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("BuildPayload: %v", err)
|
||||
}
|
||||
var got defaultGenericPayload
|
||||
if err := json.Unmarshal(payload, &got); err != nil {
|
||||
t.Fatalf("unmarshaling payload: %v", err)
|
||||
}
|
||||
if got.RuleName != "High error rate" || got.EventType != "firing" || got.Value == nil || *got.Value != 142.0 {
|
||||
t.Fatalf("unexpected payload: %+v", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestBuildPayloadGenericUsesTemplate(t *testing.T) {
|
||||
tmpl := `{"custom": "{{.RuleName}} is {{.EventType}}"}`
|
||||
target := notifystore.Target{Kind: notifystore.KindWebhook, PayloadTemplate: &tmpl}
|
||||
payload, err := BuildPayload(target, Event{RuleName: "disk full", EventType: "firing"})
|
||||
if err != nil {
|
||||
t.Fatalf("BuildPayload: %v", err)
|
||||
}
|
||||
var got map[string]string
|
||||
if err := json.Unmarshal(payload, &got); err != nil {
|
||||
t.Fatalf("unmarshaling templated payload: %v", err)
|
||||
}
|
||||
if got["custom"] != "disk full is firing" {
|
||||
t.Fatalf("got %q", got["custom"])
|
||||
}
|
||||
}
|
||||
|
||||
func TestBuildPayloadSlackShape(t *testing.T) {
|
||||
target := notifystore.Target{Kind: notifystore.KindSlack}
|
||||
payload, err := BuildPayload(target, Event{RuleName: "High error rate", EventType: "resolved"})
|
||||
if err != nil {
|
||||
t.Fatalf("BuildPayload: %v", err)
|
||||
}
|
||||
var got slackPayload
|
||||
if err := json.Unmarshal(payload, &got); err != nil {
|
||||
t.Fatalf("unmarshaling slack payload: %v", err)
|
||||
}
|
||||
if got.Text == "" {
|
||||
t.Fatalf("expected non-empty Slack text")
|
||||
}
|
||||
}
|
||||
|
||||
func TestBuildPayloadPagerDutyMapsEventActionAndRoutingKey(t *testing.T) {
|
||||
secret := "R0UT1NGKEY"
|
||||
target := notifystore.Target{Kind: notifystore.KindPagerDuty, Secret: &secret}
|
||||
|
||||
firing, err := BuildPayload(target, Event{RuleID: "r1", RuleName: "disk full", EventType: "firing"})
|
||||
if err != nil {
|
||||
t.Fatalf("BuildPayload firing: %v", err)
|
||||
}
|
||||
var gotFiring pagerDutyPayload
|
||||
if err := json.Unmarshal(firing, &gotFiring); err != nil {
|
||||
t.Fatalf("unmarshaling: %v", err)
|
||||
}
|
||||
if gotFiring.EventAction != "trigger" || gotFiring.RoutingKey != secret || gotFiring.DedupKey != "r1" {
|
||||
t.Fatalf("unexpected firing payload: %+v", gotFiring)
|
||||
}
|
||||
|
||||
resolved, err := BuildPayload(target, Event{RuleID: "r1", RuleName: "disk full", EventType: "resolved"})
|
||||
if err != nil {
|
||||
t.Fatalf("BuildPayload resolved: %v", err)
|
||||
}
|
||||
var gotResolved pagerDutyPayload
|
||||
if err := json.Unmarshal(resolved, &gotResolved); err != nil {
|
||||
t.Fatalf("unmarshaling: %v", err)
|
||||
}
|
||||
if gotResolved.EventAction != "resolve" {
|
||||
t.Fatalf("expected event_action=resolve, got %q", gotResolved.EventAction)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,57 @@
|
||||
package delivery
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
|
||||
"github.com/sentry/sentry/alerting/internal/notifystore"
|
||||
)
|
||||
|
||||
// pagerDutyPayload is PagerDuty's Events API v2 shape. routing_key comes
|
||||
// from the target's `secret` field -- PagerDuty calls this an
|
||||
// "integration key," not a delivery credential in the auth sense, but
|
||||
// it's stored in the same plaintext `secret` column as any other
|
||||
// target's credential (see /docs/phase-3-alerting-design.md's "Known
|
||||
// gaps").
|
||||
type pagerDutyPayload struct {
|
||||
RoutingKey string `json:"routing_key"`
|
||||
EventAction string `json:"event_action"` // "trigger" | "resolve"
|
||||
DedupKey string `json:"dedup_key"` // rule_id -- lets PagerDuty correlate trigger/resolve as one incident
|
||||
Payload pagerDutyEventPayload `json:"payload"`
|
||||
}
|
||||
|
||||
type pagerDutyEventPayload struct {
|
||||
Summary string `json:"summary"`
|
||||
Source string `json:"source"`
|
||||
Severity string `json:"severity"`
|
||||
}
|
||||
|
||||
func buildPagerDutyPayload(target notifystore.Target, event Event) ([]byte, error) {
|
||||
action := "trigger"
|
||||
severity := "critical"
|
||||
if event.EventType == "resolved" {
|
||||
action = "resolve"
|
||||
severity = "info"
|
||||
}
|
||||
|
||||
var routingKey string
|
||||
if target.Secret != nil {
|
||||
routingKey = *target.Secret
|
||||
}
|
||||
|
||||
summary := fmt.Sprintf("%s: %s", event.RuleName, event.EventType)
|
||||
if event.Value != nil && event.ThresholdValue != nil {
|
||||
summary = fmt.Sprintf("%s (value %.4g %s %.4g)", summary, *event.Value, comparatorSymbol(event.Comparator), *event.ThresholdValue)
|
||||
}
|
||||
|
||||
return json.Marshal(pagerDutyPayload{
|
||||
RoutingKey: routingKey,
|
||||
EventAction: action,
|
||||
DedupKey: event.RuleID,
|
||||
Payload: pagerDutyEventPayload{
|
||||
Summary: summary,
|
||||
Source: "sentry",
|
||||
Severity: severity,
|
||||
},
|
||||
})
|
||||
}
|
||||
@@ -0,0 +1,48 @@
|
||||
package delivery
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
)
|
||||
|
||||
// slackPayload is Slack's incoming-webhook shape: {"text": "..."}. No
|
||||
// user template for this kind -- see webhook.go's BuildPayload doc
|
||||
// comment for why (thin formatter, not a user-configurable path).
|
||||
type slackPayload struct {
|
||||
Text string `json:"text"`
|
||||
}
|
||||
|
||||
func buildSlackPayload(event Event) ([]byte, error) {
|
||||
var text string
|
||||
switch event.EventType {
|
||||
case "firing":
|
||||
text = fmt.Sprintf(":rotating_light: *%s* is firing", event.RuleName)
|
||||
if event.Value != nil && event.ThresholdValue != nil {
|
||||
text += fmt.Sprintf(" (value %.4g %s %.4g)", *event.Value, comparatorSymbol(event.Comparator), *event.ThresholdValue)
|
||||
}
|
||||
case "resolved":
|
||||
text = fmt.Sprintf(":white_check_mark: *%s* resolved", event.RuleName)
|
||||
default:
|
||||
text = fmt.Sprintf("%s: %s", event.RuleName, event.EventType)
|
||||
}
|
||||
return json.Marshal(slackPayload{Text: text})
|
||||
}
|
||||
|
||||
func comparatorSymbol(c string) string {
|
||||
switch c {
|
||||
case "gt":
|
||||
return ">"
|
||||
case "gte":
|
||||
return ">="
|
||||
case "lt":
|
||||
return "<"
|
||||
case "lte":
|
||||
return "<="
|
||||
case "eq":
|
||||
return "=="
|
||||
case "ne":
|
||||
return "!="
|
||||
default:
|
||||
return c
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,229 @@
|
||||
// Package delivery sends notifications for firing/resolved alert
|
||||
// events. All three notification_targets.kind values -- webhook, slack,
|
||||
// pagerduty -- go through the exact same claim-then-POST-with-backoff
|
||||
// mechanism in this file; slack.go and pagerduty.go are payload
|
||||
// *formatters* only, never a separate delivery path, per
|
||||
// /docs/phase-3-alerting-design.md's "thin wrappers over the generic
|
||||
// webhook" requirement.
|
||||
package delivery
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"log/slog"
|
||||
"net/http"
|
||||
"text/template"
|
||||
"time"
|
||||
|
||||
"github.com/jackc/pgx/v5/pgxpool"
|
||||
|
||||
"github.com/sentry/sentry/alerting/internal/notifystore"
|
||||
)
|
||||
|
||||
// Event is the firing/resolved occurrence a payload is rendered from.
|
||||
type Event struct {
|
||||
RuleID string
|
||||
RuleName string
|
||||
EventType string // "firing" | "resolved"
|
||||
ConditionType string
|
||||
Comparator string
|
||||
ThresholdValue *float64
|
||||
Value *float64
|
||||
Timestamp time.Time
|
||||
}
|
||||
|
||||
// defaultGenericPayload is used when a webhook target has no
|
||||
// payload_template.
|
||||
type defaultGenericPayload struct {
|
||||
RuleID string `json:"rule_id"`
|
||||
RuleName string `json:"rule_name"`
|
||||
EventType string `json:"event_type"`
|
||||
Value *float64 `json:"value,omitempty"`
|
||||
Timestamp string `json:"timestamp"`
|
||||
}
|
||||
|
||||
// BuildPayload dispatches by target.Kind to the right formatter. This is
|
||||
// the only place kind-specific formatting logic lives -- everything
|
||||
// downstream of this (worker.go's send loop) is kind-agnostic.
|
||||
func BuildPayload(target notifystore.Target, event Event) ([]byte, error) {
|
||||
switch target.Kind {
|
||||
case notifystore.KindSlack:
|
||||
return buildSlackPayload(event)
|
||||
case notifystore.KindPagerDuty:
|
||||
return buildPagerDutyPayload(target, event)
|
||||
default: // webhook
|
||||
return buildGenericPayload(target, event)
|
||||
}
|
||||
}
|
||||
|
||||
func buildGenericPayload(target notifystore.Target, event Event) ([]byte, error) {
|
||||
if target.PayloadTemplate == nil || *target.PayloadTemplate == "" {
|
||||
return json.Marshal(defaultGenericPayload{
|
||||
RuleID: event.RuleID, RuleName: event.RuleName, EventType: event.EventType,
|
||||
Value: event.Value, Timestamp: event.Timestamp.UTC().Format(time.RFC3339),
|
||||
})
|
||||
}
|
||||
tmpl, err := template.New("payload").Parse(*target.PayloadTemplate)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("parsing payload_template: %w", err)
|
||||
}
|
||||
var buf bytes.Buffer
|
||||
if err := tmpl.Execute(&buf, event); err != nil {
|
||||
return nil, fmt.Errorf("rendering payload_template: %w", err)
|
||||
}
|
||||
return buf.Bytes(), nil
|
||||
}
|
||||
|
||||
// Worker claims pending/retrying delivery_log rows and sends them.
|
||||
// Deliberately separate from the evaluator: "decided to notify"
|
||||
// (rulestore.ApplyTransition's transactional outbox insert) is
|
||||
// transactionally certain; "the HTTP call succeeded" is best-effort with
|
||||
// retries, handled entirely here.
|
||||
type Worker struct {
|
||||
pool *pgxpool.Pool
|
||||
notifications *notifystore.Store
|
||||
http *http.Client
|
||||
logger *slog.Logger
|
||||
}
|
||||
|
||||
func NewWorker(pool *pgxpool.Pool, notifications *notifystore.Store, logger *slog.Logger) *Worker {
|
||||
return &Worker{pool: pool, notifications: notifications, http: &http.Client{Timeout: 10 * time.Second}, logger: logger}
|
||||
}
|
||||
|
||||
// Run claims and sends due deliveries every tickInterval until ctx is
|
||||
// cancelled.
|
||||
func (w *Worker) Run(ctx context.Context, tickInterval time.Duration) error {
|
||||
ticker := time.NewTicker(tickInterval)
|
||||
defer ticker.Stop()
|
||||
for {
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
return ctx.Err()
|
||||
case <-ticker.C:
|
||||
w.processDue(ctx)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
type claimedDelivery struct {
|
||||
id int64
|
||||
ruleID string
|
||||
notificationTargetID string
|
||||
eventType string
|
||||
attemptCount int
|
||||
maxAttempts int
|
||||
payload []byte
|
||||
}
|
||||
|
||||
// processDue claims due rows (same SKIP LOCKED pattern as the
|
||||
// evaluator's rule claim -- see rulestore.ClaimDueRules) and attempts
|
||||
// delivery for each. One claim batch per tick; a stuck/slow target
|
||||
// doesn't block others since each send happens independently.
|
||||
func (w *Worker) processDue(ctx context.Context) {
|
||||
rows, err := w.pool.Query(ctx, `
|
||||
WITH due AS (
|
||||
SELECT id FROM delivery_log
|
||||
WHERE status IN ('pending', 'retrying') AND (next_attempt_at IS NULL OR next_attempt_at <= now())
|
||||
ORDER BY created_at
|
||||
LIMIT 50
|
||||
FOR UPDATE SKIP LOCKED
|
||||
)
|
||||
UPDATE delivery_log d
|
||||
SET last_attempt_at = now()
|
||||
FROM due
|
||||
WHERE d.id = due.id
|
||||
RETURNING d.id, d.rule_id, d.notification_target_id, d.event_type, d.attempt_count, d.max_attempts, d.payload`)
|
||||
if err != nil {
|
||||
w.logger.Error("claiming due deliveries", "error", err)
|
||||
return
|
||||
}
|
||||
var claimed []claimedDelivery
|
||||
for rows.Next() {
|
||||
var c claimedDelivery
|
||||
if err := rows.Scan(&c.id, &c.ruleID, &c.notificationTargetID, &c.eventType, &c.attemptCount, &c.maxAttempts, &c.payload); err != nil {
|
||||
w.logger.Error("scanning claimed delivery", "error", err)
|
||||
continue
|
||||
}
|
||||
claimed = append(claimed, c)
|
||||
}
|
||||
rows.Close()
|
||||
|
||||
for _, c := range claimed {
|
||||
w.attempt(ctx, c)
|
||||
}
|
||||
}
|
||||
|
||||
func (w *Worker) attempt(ctx context.Context, c claimedDelivery) {
|
||||
target, err := w.notifications.Get(ctx, c.notificationTargetID)
|
||||
if err != nil {
|
||||
w.fail(ctx, c, 0, fmt.Sprintf("looking up notification target: %v", err))
|
||||
return
|
||||
}
|
||||
|
||||
req, err := http.NewRequestWithContext(ctx, http.MethodPost, target.WebhookURL, bytes.NewReader(c.payload))
|
||||
if err != nil {
|
||||
w.fail(ctx, c, 0, fmt.Sprintf("building request: %v", err))
|
||||
return
|
||||
}
|
||||
req.Header.Set("Content-Type", "application/json")
|
||||
|
||||
resp, err := w.http.Do(req)
|
||||
if err != nil {
|
||||
w.fail(ctx, c, 0, err.Error())
|
||||
return
|
||||
}
|
||||
defer resp.Body.Close()
|
||||
|
||||
if resp.StatusCode >= 200 && resp.StatusCode < 300 {
|
||||
w.markSent(ctx, c.id, resp.StatusCode)
|
||||
return
|
||||
}
|
||||
w.fail(ctx, c, resp.StatusCode, fmt.Sprintf("non-2xx response: %d", resp.StatusCode))
|
||||
}
|
||||
|
||||
func (w *Worker) markSent(ctx context.Context, id int64, statusCode int) {
|
||||
_, err := w.pool.Exec(ctx, `
|
||||
UPDATE delivery_log SET status = 'sent', attempt_count = attempt_count + 1, response_status = $1 WHERE id = $2`,
|
||||
statusCode, id)
|
||||
if err != nil {
|
||||
w.logger.Error("marking delivery sent", "delivery_id", id, "error", err)
|
||||
}
|
||||
}
|
||||
|
||||
// fail records a failed attempt. If attempts remain, it's rescheduled
|
||||
// with exponential backoff (2^attempt_count seconds, capped at 1 hour);
|
||||
// otherwise it's marked permanently failed.
|
||||
func (w *Worker) fail(ctx context.Context, c claimedDelivery, statusCode int, errMsg string) {
|
||||
nextAttempt := c.attemptCount + 1
|
||||
if nextAttempt >= c.maxAttempts {
|
||||
_, err := w.pool.Exec(ctx, `
|
||||
UPDATE delivery_log SET status = 'failed', attempt_count = attempt_count + 1, response_status = $1, last_error = $2 WHERE id = $3`,
|
||||
nullIfZero(statusCode), errMsg, c.id)
|
||||
if err != nil {
|
||||
w.logger.Error("marking delivery failed", "delivery_id", c.id, "error", err)
|
||||
}
|
||||
return
|
||||
}
|
||||
|
||||
backoff := time.Duration(1<<uint(nextAttempt)) * time.Second
|
||||
if backoff > time.Hour {
|
||||
backoff = time.Hour
|
||||
}
|
||||
_, err := w.pool.Exec(ctx, `
|
||||
UPDATE delivery_log
|
||||
SET status = 'retrying', attempt_count = attempt_count + 1, response_status = $1, last_error = $2, next_attempt_at = now() + $3
|
||||
WHERE id = $4`,
|
||||
nullIfZero(statusCode), errMsg, backoff, c.id)
|
||||
if err != nil {
|
||||
w.logger.Error("scheduling delivery retry", "delivery_id", c.id, "error", err)
|
||||
}
|
||||
}
|
||||
|
||||
func nullIfZero(n int) *int {
|
||||
if n == 0 {
|
||||
return nil
|
||||
}
|
||||
return &n
|
||||
}
|
||||
@@ -0,0 +1,74 @@
|
||||
package evaluator
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
|
||||
"github.com/sentry/sentry/alerting/internal/queryclient"
|
||||
"github.com/sentry/sentry/alerting/internal/rulestore"
|
||||
)
|
||||
|
||||
// evaluateCondition implements fixes 3 and 4 from
|
||||
// /docs/phase-3-alerting-design.md: a threshold rule's query must
|
||||
// resolve to exactly one row, and zero (or more than one) rows is
|
||||
// returned as an error, never coerced to a value -- "nothing ran" and
|
||||
// "ran and found nothing" are different failure/success modes, and
|
||||
// conflating them can hide the more alarming case. This function never
|
||||
// itself decides "condition false" on an error; it returns an error, and
|
||||
// the caller (evaluator.go) routes that to rulestore.RecordError, never
|
||||
// to ComputeTransition.
|
||||
func evaluateCondition(rule rulestore.Rule, result *queryclient.Result) (conditionTrue bool, value *float64, err error) {
|
||||
switch rule.ConditionType {
|
||||
case rulestore.ConditionAbsence:
|
||||
// The window is whatever earliest=/latest= the rule's own query
|
||||
// already expresses -- no separate window field, per the design doc.
|
||||
return len(result.Rows) == 0, nil, nil
|
||||
|
||||
case rulestore.ConditionThreshold:
|
||||
if len(result.Rows) != 1 {
|
||||
return false, nil, fmt.Errorf("threshold rule query returned %d rows, want exactly 1", len(result.Rows))
|
||||
}
|
||||
row := result.Rows[0]
|
||||
if len(row) == 0 {
|
||||
return false, nil, fmt.Errorf("threshold rule query returned a row with no columns")
|
||||
}
|
||||
v, ok := toFloat64(row[0])
|
||||
if !ok {
|
||||
return false, nil, fmt.Errorf("threshold rule's first column value %v is not numeric", row[0])
|
||||
}
|
||||
if rule.Comparator == nil || rule.ThresholdValue == nil {
|
||||
return false, nil, fmt.Errorf("threshold rule is missing comparator or threshold_value")
|
||||
}
|
||||
return compare(v, *rule.Comparator, *rule.ThresholdValue), &v, nil
|
||||
|
||||
default:
|
||||
return false, nil, fmt.Errorf("unknown condition_type %q", rule.ConditionType)
|
||||
}
|
||||
}
|
||||
|
||||
func compare(value float64, comparator rulestore.Comparator, threshold float64) bool {
|
||||
switch comparator {
|
||||
case rulestore.Gt:
|
||||
return value > threshold
|
||||
case rulestore.Gte:
|
||||
return value >= threshold
|
||||
case rulestore.Lt:
|
||||
return value < threshold
|
||||
case rulestore.Lte:
|
||||
return value <= threshold
|
||||
case rulestore.Eq:
|
||||
return value == threshold
|
||||
case rulestore.Ne:
|
||||
return value != threshold
|
||||
default:
|
||||
return false
|
||||
}
|
||||
}
|
||||
|
||||
// toFloat64 handles the one shape /query responses actually come back
|
||||
// as: Go's encoding/json always decodes JSON numbers into interface{}
|
||||
// as float64, regardless of whether ClickHouse serialized an integer or
|
||||
// a float -- there's no separate int64 case to handle.
|
||||
func toFloat64(v any) (float64, bool) {
|
||||
f, ok := v.(float64)
|
||||
return f, ok
|
||||
}
|
||||
@@ -0,0 +1,118 @@
|
||||
package evaluator
|
||||
|
||||
import (
|
||||
"testing"
|
||||
|
||||
"github.com/sentry/sentry/alerting/internal/queryclient"
|
||||
"github.com/sentry/sentry/alerting/internal/rulestore"
|
||||
)
|
||||
|
||||
func thresholdRule(comparator rulestore.Comparator, threshold float64) rulestore.Rule {
|
||||
return rulestore.Rule{
|
||||
ConditionType: rulestore.ConditionThreshold,
|
||||
Comparator: &comparator,
|
||||
ThresholdValue: &threshold,
|
||||
}
|
||||
}
|
||||
|
||||
func TestEvaluateConditionThresholdTrue(t *testing.T) {
|
||||
rule := thresholdRule(rulestore.Gt, 100)
|
||||
result := &queryclient.Result{Columns: []string{"count"}, Rows: [][]any{{150.0}}}
|
||||
|
||||
got, value, err := evaluateCondition(rule, result)
|
||||
if err != nil {
|
||||
t.Fatalf("unexpected error: %v", err)
|
||||
}
|
||||
if !got {
|
||||
t.Fatalf("expected condition true for 150 > 100")
|
||||
}
|
||||
if value == nil || *value != 150.0 {
|
||||
t.Fatalf("expected value=150, got %v", value)
|
||||
}
|
||||
}
|
||||
|
||||
func TestEvaluateConditionThresholdFalse(t *testing.T) {
|
||||
rule := thresholdRule(rulestore.Gt, 100)
|
||||
result := &queryclient.Result{Columns: []string{"count"}, Rows: [][]any{{50.0}}}
|
||||
|
||||
got, _, err := evaluateCondition(rule, result)
|
||||
if err != nil {
|
||||
t.Fatalf("unexpected error: %v", err)
|
||||
}
|
||||
if got {
|
||||
t.Fatalf("expected condition false for 50 > 100")
|
||||
}
|
||||
}
|
||||
|
||||
// TestEvaluateConditionThresholdZeroRowsIsError pins down fix 4: zero
|
||||
// rows on a threshold rule must be an error, not silently coerced to 0
|
||||
// (which would make `count > 100` falsely report "fine" when actually
|
||||
// nothing ran).
|
||||
func TestEvaluateConditionThresholdZeroRowsIsError(t *testing.T) {
|
||||
rule := thresholdRule(rulestore.Gt, 100)
|
||||
result := &queryclient.Result{Columns: []string{"count"}, Rows: [][]any{}}
|
||||
|
||||
_, value, err := evaluateCondition(rule, result)
|
||||
if err == nil {
|
||||
t.Fatalf("expected an error for zero rows on a threshold rule, got condition evaluated with value=%v", value)
|
||||
}
|
||||
}
|
||||
|
||||
func TestEvaluateConditionThresholdMultipleRowsIsError(t *testing.T) {
|
||||
rule := thresholdRule(rulestore.Gt, 100)
|
||||
result := &queryclient.Result{Columns: []string{"host", "count"}, Rows: [][]any{{"h1", 50.0}, {"h2", 200.0}}}
|
||||
|
||||
_, _, err := evaluateCondition(rule, result)
|
||||
if err == nil {
|
||||
t.Fatalf("expected an error for multiple rows on a threshold rule (no per-group alerting, a named non-goal)")
|
||||
}
|
||||
}
|
||||
|
||||
func TestEvaluateConditionThresholdNonNumericIsError(t *testing.T) {
|
||||
rule := thresholdRule(rulestore.Gt, 100)
|
||||
result := &queryclient.Result{Columns: []string{"host"}, Rows: [][]any{{"host-01"}}}
|
||||
|
||||
_, _, err := evaluateCondition(rule, result)
|
||||
if err == nil {
|
||||
t.Fatalf("expected an error for a non-numeric first column")
|
||||
}
|
||||
}
|
||||
|
||||
func TestEvaluateConditionAbsenceTrueWhenZeroRows(t *testing.T) {
|
||||
rule := rulestore.Rule{ConditionType: rulestore.ConditionAbsence}
|
||||
result := &queryclient.Result{Columns: []string{"message"}, Rows: [][]any{}}
|
||||
|
||||
got, value, err := evaluateCondition(rule, result)
|
||||
if err != nil {
|
||||
t.Fatalf("unexpected error: %v", err)
|
||||
}
|
||||
if !got {
|
||||
t.Fatalf("expected absence condition true for zero rows")
|
||||
}
|
||||
if value != nil {
|
||||
t.Fatalf("expected nil value for an absence rule, got %v", value)
|
||||
}
|
||||
}
|
||||
|
||||
func TestEvaluateConditionAbsenceFalseWhenRowsPresent(t *testing.T) {
|
||||
rule := rulestore.Rule{ConditionType: rulestore.ConditionAbsence}
|
||||
result := &queryclient.Result{Columns: []string{"message"}, Rows: [][]any{{"something happened"}}}
|
||||
|
||||
got, _, err := evaluateCondition(rule, result)
|
||||
if err != nil {
|
||||
t.Fatalf("unexpected error: %v", err)
|
||||
}
|
||||
if got {
|
||||
t.Fatalf("expected absence condition false when rows are present")
|
||||
}
|
||||
}
|
||||
|
||||
func TestEvaluateConditionUnknownTypeIsError(t *testing.T) {
|
||||
rule := rulestore.Rule{ConditionType: "bogus"}
|
||||
result := &queryclient.Result{Columns: []string{"count"}, Rows: [][]any{{1.0}}}
|
||||
|
||||
_, _, err := evaluateCondition(rule, result)
|
||||
if err == nil {
|
||||
t.Fatalf("expected an error for an unknown condition_type")
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,170 @@
|
||||
package evaluator
|
||||
|
||||
import (
|
||||
"context"
|
||||
"log/slog"
|
||||
"sync"
|
||||
"time"
|
||||
|
||||
"github.com/sentry/sentry/alerting/internal/delivery"
|
||||
"github.com/sentry/sentry/alerting/internal/notifystore"
|
||||
"github.com/sentry/sentry/alerting/internal/queryclient"
|
||||
"github.com/sentry/sentry/alerting/internal/rulestore"
|
||||
)
|
||||
|
||||
// Evaluator is the ticker-driven scheduler -- a bounded worker pool, not
|
||||
// a workflow engine, per the design doc's explicit instruction. Each
|
||||
// tick claims up to claimBatchSize due rules (rulestore.ClaimDueRules,
|
||||
// fix 1's atomic claim) and evaluates them concurrently up to
|
||||
// workerPoolSize at a time. These are deliberately different numbers --
|
||||
// see config.EvaluatorConfig's doc comment for the real bug this fixes
|
||||
// (500 rules due at once, both capped at 20, took 125s to cycle through
|
||||
// instead of the configured 60s).
|
||||
type Evaluator struct {
|
||||
rules *rulestore.Store
|
||||
notifications *notifystore.Store
|
||||
queryClient *queryclient.Client
|
||||
queryTimeout time.Duration
|
||||
claimBatchSize int
|
||||
workerPoolSize int
|
||||
logger *slog.Logger
|
||||
}
|
||||
|
||||
func New(rules *rulestore.Store, notifications *notifystore.Store, queryClient *queryclient.Client, queryTimeout time.Duration, claimBatchSize, workerPoolSize int, logger *slog.Logger) *Evaluator {
|
||||
return &Evaluator{
|
||||
rules: rules, notifications: notifications, queryClient: queryClient,
|
||||
queryTimeout: queryTimeout, claimBatchSize: claimBatchSize, workerPoolSize: workerPoolSize, logger: logger,
|
||||
}
|
||||
}
|
||||
|
||||
// Run ticks every tickInterval until ctx is cancelled.
|
||||
func (e *Evaluator) Run(ctx context.Context, tickInterval time.Duration) error {
|
||||
ticker := time.NewTicker(tickInterval)
|
||||
defer ticker.Stop()
|
||||
for {
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
return ctx.Err()
|
||||
case <-ticker.C:
|
||||
e.tick(ctx)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func (e *Evaluator) tick(ctx context.Context) {
|
||||
claimed, err := e.rules.ClaimDueRules(ctx, e.claimBatchSize)
|
||||
if err != nil {
|
||||
e.logger.Error("claiming due rules", "error", err)
|
||||
return
|
||||
}
|
||||
if len(claimed) == 0 {
|
||||
return
|
||||
}
|
||||
|
||||
sem := make(chan struct{}, e.workerPoolSize)
|
||||
var wg sync.WaitGroup
|
||||
for _, rule := range claimed {
|
||||
wg.Add(1)
|
||||
sem <- struct{}{}
|
||||
go func(rule rulestore.RuleWithState) {
|
||||
defer wg.Done()
|
||||
defer func() { <-sem }()
|
||||
e.evaluateOne(ctx, rule)
|
||||
}(rule)
|
||||
}
|
||||
wg.Wait()
|
||||
}
|
||||
|
||||
func (e *Evaluator) evaluateOne(ctx context.Context, rule rulestore.RuleWithState) {
|
||||
result, err := e.queryClient.Query(ctx, rule.Query, rule.QueryLanguage, e.queryTimeout)
|
||||
if err != nil {
|
||||
// A failed /query call is an evaluation error, never
|
||||
// "condition false" -- fix 3. Recording it here, not routing it
|
||||
// through ComputeTransition at all, is what makes that guarantee
|
||||
// hold structurally rather than by convention.
|
||||
e.recordError(ctx, rule.ID, err.Error())
|
||||
return
|
||||
}
|
||||
|
||||
conditionTrue, value, evalErr := evaluateCondition(rule.Rule, result)
|
||||
if evalErr != nil {
|
||||
e.recordError(ctx, rule.ID, evalErr.Error())
|
||||
return
|
||||
}
|
||||
|
||||
var renotify *time.Duration
|
||||
if rule.RenotifyIntervalMinutes != nil {
|
||||
d := time.Duration(*rule.RenotifyIntervalMinutes) * time.Minute
|
||||
renotify = &d
|
||||
}
|
||||
|
||||
now := time.Now().UTC()
|
||||
transition := ComputeTransition(TransitionInput{
|
||||
CurrentState: rule.State.State,
|
||||
ConditionTrueSince: rule.State.ConditionTrueSince,
|
||||
FiredAt: rule.State.FiredAt,
|
||||
LastNotifiedAt: rule.State.LastNotifiedAt,
|
||||
ForMinutes: time.Duration(rule.ForMinutes) * time.Minute,
|
||||
RenotifyInterval: renotify,
|
||||
Now: now,
|
||||
ConditionTrue: conditionTrue,
|
||||
})
|
||||
|
||||
next := rulestore.AlertState{
|
||||
State: transition.NextState,
|
||||
ConditionTrueSince: transition.NextConditionTrueSince,
|
||||
FiredAt: transition.NextFiredAt,
|
||||
LastNotifiedAt: transition.NextLastNotifiedAt,
|
||||
LastValue: value,
|
||||
}
|
||||
|
||||
notify := e.buildNotifyEvent(ctx, rule, transition, value, now)
|
||||
|
||||
if err := e.rules.ApplyTransition(ctx, rule.ID, next, notify); err != nil {
|
||||
e.logger.Error("applying alert state transition", "rule_id", rule.ID, "error", err)
|
||||
}
|
||||
}
|
||||
|
||||
// buildNotifyEvent resolves the notification target and renders the
|
||||
// payload for a firing/resolved transition. A lookup or template
|
||||
// failure here is logged and treated as "no notification this time" --
|
||||
// the state still transitions correctly (that's the more important
|
||||
// guarantee), it just means a misconfigured target/template silently
|
||||
// drops one notification rather than blocking the whole evaluation.
|
||||
func (e *Evaluator) buildNotifyEvent(ctx context.Context, rule rulestore.RuleWithState, transition TransitionResult, value *float64, now time.Time) *rulestore.NotifyEvent {
|
||||
if transition.Notify == nil {
|
||||
return nil
|
||||
}
|
||||
|
||||
target, err := e.notifications.Get(ctx, rule.NotificationTargetID)
|
||||
if err != nil {
|
||||
e.logger.Error("looking up notification target", "rule_id", rule.ID, "target_id", rule.NotificationTargetID, "error", err)
|
||||
return nil
|
||||
}
|
||||
|
||||
comparator := ""
|
||||
if rule.Comparator != nil {
|
||||
comparator = string(*rule.Comparator)
|
||||
}
|
||||
payload, err := delivery.BuildPayload(*target, delivery.Event{
|
||||
RuleID: rule.ID, RuleName: rule.Name, EventType: *transition.Notify,
|
||||
ConditionType: string(rule.ConditionType), Comparator: comparator,
|
||||
ThresholdValue: rule.ThresholdValue, Value: value, Timestamp: now,
|
||||
})
|
||||
if err != nil {
|
||||
e.logger.Error("building notification payload", "rule_id", rule.ID, "error", err)
|
||||
return nil
|
||||
}
|
||||
|
||||
return &rulestore.NotifyEvent{
|
||||
NotificationTargetID: rule.NotificationTargetID,
|
||||
EventType: *transition.Notify,
|
||||
Payload: payload,
|
||||
}
|
||||
}
|
||||
|
||||
func (e *Evaluator) recordError(ctx context.Context, ruleID, msg string) {
|
||||
if err := e.rules.RecordError(ctx, ruleID, msg); err != nil {
|
||||
e.logger.Error("recording evaluation error", "rule_id", ruleID, "error", err)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,121 @@
|
||||
// Package evaluator implements the ticker-driven rule scheduler and the
|
||||
// firing/resolved state machine described in
|
||||
// /docs/phase-3-alerting-design.md. transitions.go is deliberately pure
|
||||
// (no DB, no HTTP, no clock reads beyond the `Now` passed in) --
|
||||
// evaluator.go is the only caller, and it's the single place worth
|
||||
// exhaustive table-driven testing given how easy this state machine is
|
||||
// to get subtly wrong.
|
||||
package evaluator
|
||||
|
||||
import (
|
||||
"time"
|
||||
|
||||
"github.com/sentry/sentry/alerting/internal/rulestore"
|
||||
)
|
||||
|
||||
// TransitionInput is everything ComputeTransition needs: the rule's
|
||||
// current persisted state plus this evaluation's outcome. Evaluation
|
||||
// *errors* never reach this function at all -- per fix 3 in the design
|
||||
// doc, an error is recorded separately (rulestore.RecordError) and never
|
||||
// treated as ConditionTrue: false.
|
||||
type TransitionInput struct {
|
||||
CurrentState rulestore.State
|
||||
ConditionTrueSince *time.Time
|
||||
FiredAt *time.Time
|
||||
LastNotifiedAt *time.Time
|
||||
ForMinutes time.Duration
|
||||
RenotifyInterval *time.Duration // nil = notify once per firing episode, never again
|
||||
Now time.Time
|
||||
ConditionTrue bool
|
||||
}
|
||||
|
||||
// TransitionResult is what to persist (via rulestore.ApplyTransition)
|
||||
// and, if Notify is non-nil, which event ("firing" or "resolved") to
|
||||
// enqueue in the same transaction.
|
||||
type TransitionResult struct {
|
||||
NextState rulestore.State
|
||||
NextConditionTrueSince *time.Time
|
||||
NextFiredAt *time.Time
|
||||
NextLastNotifiedAt *time.Time
|
||||
Notify *string
|
||||
}
|
||||
|
||||
// ComputeTransition implements the ok/pending/firing state machine.
|
||||
// ConditionTrueSince is always a wall-clock timestamp, never a
|
||||
// consecutive-evaluation counter -- this is what makes the for_minutes
|
||||
// debounce survive evaluator restarts correctly (a counter would
|
||||
// silently lose progress across downtime; wall-clock math resumes
|
||||
// exactly where it left off). Don't "simplify" this into a counter.
|
||||
func ComputeTransition(in TransitionInput) TransitionResult {
|
||||
if in.ConditionTrue {
|
||||
return computeConditionTrue(in)
|
||||
}
|
||||
return computeConditionFalse(in)
|
||||
}
|
||||
|
||||
func computeConditionTrue(in TransitionInput) TransitionResult {
|
||||
switch in.CurrentState {
|
||||
case rulestore.StatePending:
|
||||
since := in.ConditionTrueSince
|
||||
if since == nil {
|
||||
// Defensive: pending with no recorded since is inconsistent
|
||||
// persisted state (shouldn't happen -- ok->pending always sets
|
||||
// it) -- treat as if the condition just became true rather than
|
||||
// dereferencing a nil or panicking.
|
||||
now := in.Now
|
||||
since = &now
|
||||
}
|
||||
if in.Now.Sub(*since) >= in.ForMinutes {
|
||||
return fire(*since, in.Now)
|
||||
}
|
||||
return TransitionResult{NextState: rulestore.StatePending, NextConditionTrueSince: since}
|
||||
|
||||
case rulestore.StateFiring:
|
||||
result := TransitionResult{
|
||||
NextState: rulestore.StateFiring,
|
||||
NextConditionTrueSince: in.ConditionTrueSince,
|
||||
NextFiredAt: in.FiredAt,
|
||||
NextLastNotifiedAt: in.LastNotifiedAt,
|
||||
}
|
||||
if in.RenotifyInterval != nil && in.LastNotifiedAt != nil && in.Now.Sub(*in.LastNotifiedAt) >= *in.RenotifyInterval {
|
||||
now := in.Now
|
||||
event := "firing"
|
||||
result.NextLastNotifiedAt = &now
|
||||
result.Notify = &event
|
||||
}
|
||||
return result
|
||||
|
||||
default: // ok (or any unexpected value -- the DB CHECK constraint keeps this to 3 values)
|
||||
since := in.Now
|
||||
if in.ForMinutes <= 0 {
|
||||
// for_minutes=0 means "fire on first true evaluation" --
|
||||
// literally, not after one extra tick spent in `pending`.
|
||||
return fire(since, in.Now)
|
||||
}
|
||||
return TransitionResult{NextState: rulestore.StatePending, NextConditionTrueSince: &since}
|
||||
}
|
||||
}
|
||||
|
||||
func computeConditionFalse(in TransitionInput) TransitionResult {
|
||||
switch in.CurrentState {
|
||||
case rulestore.StateFiring:
|
||||
event := "resolved"
|
||||
return TransitionResult{NextState: rulestore.StateOK, Notify: &event}
|
||||
default: // ok or pending: a false evaluation while pending is a blip
|
||||
// inside the debounce window -- deliberately no notification, that's
|
||||
// the entire point of for_minutes existing.
|
||||
return TransitionResult{NextState: rulestore.StateOK}
|
||||
}
|
||||
}
|
||||
|
||||
func fire(since, now time.Time) TransitionResult {
|
||||
firedAt := now
|
||||
event := "firing"
|
||||
return TransitionResult{
|
||||
NextState: rulestore.StateFiring,
|
||||
NextConditionTrueSince: &since,
|
||||
NextFiredAt: &firedAt,
|
||||
NextLastNotifiedAt: &firedAt,
|
||||
Notify: &event,
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,193 @@
|
||||
package evaluator
|
||||
|
||||
import (
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/sentry/sentry/alerting/internal/rulestore"
|
||||
)
|
||||
|
||||
var t0 = time.Date(2026, 8, 14, 0, 0, 0, 0, time.UTC)
|
||||
|
||||
func ptr[T any](v T) *T { return &v }
|
||||
|
||||
func TestComputeTransition(t *testing.T) {
|
||||
tests := []struct {
|
||||
name string
|
||||
in TransitionInput
|
||||
want TransitionResult
|
||||
}{
|
||||
{
|
||||
name: "ok, condition true, for_minutes>0 -> pending, no notify",
|
||||
in: TransitionInput{
|
||||
CurrentState: rulestore.StateOK, ForMinutes: 5 * time.Minute,
|
||||
Now: t0, ConditionTrue: true,
|
||||
},
|
||||
want: TransitionResult{NextState: rulestore.StatePending, NextConditionTrueSince: ptr(t0)},
|
||||
},
|
||||
{
|
||||
name: "ok, condition true, for_minutes=0 -> fires immediately, no extra tick in pending",
|
||||
in: TransitionInput{
|
||||
CurrentState: rulestore.StateOK, ForMinutes: 0,
|
||||
Now: t0, ConditionTrue: true,
|
||||
},
|
||||
want: TransitionResult{
|
||||
NextState: rulestore.StateFiring, NextConditionTrueSince: ptr(t0),
|
||||
NextFiredAt: ptr(t0), NextLastNotifiedAt: ptr(t0), Notify: ptr("firing"),
|
||||
},
|
||||
},
|
||||
{
|
||||
name: "pending, still within for_minutes window -> stays pending, no notify",
|
||||
in: TransitionInput{
|
||||
CurrentState: rulestore.StatePending, ConditionTrueSince: ptr(t0),
|
||||
ForMinutes: 5 * time.Minute, Now: t0.Add(2 * time.Minute), ConditionTrue: true,
|
||||
},
|
||||
want: TransitionResult{NextState: rulestore.StatePending, NextConditionTrueSince: ptr(t0)},
|
||||
},
|
||||
{
|
||||
name: "pending, exactly at for_minutes boundary -> fires (>=, not >)",
|
||||
in: TransitionInput{
|
||||
CurrentState: rulestore.StatePending, ConditionTrueSince: ptr(t0),
|
||||
ForMinutes: 5 * time.Minute, Now: t0.Add(5 * time.Minute), ConditionTrue: true,
|
||||
},
|
||||
want: TransitionResult{
|
||||
NextState: rulestore.StateFiring, NextConditionTrueSince: ptr(t0),
|
||||
NextFiredAt: ptr(t0.Add(5 * time.Minute)), NextLastNotifiedAt: ptr(t0.Add(5 * time.Minute)), Notify: ptr("firing"),
|
||||
},
|
||||
},
|
||||
{
|
||||
name: "pending, past for_minutes -> fires",
|
||||
in: TransitionInput{
|
||||
CurrentState: rulestore.StatePending, ConditionTrueSince: ptr(t0),
|
||||
ForMinutes: 5 * time.Minute, Now: t0.Add(10 * time.Minute), ConditionTrue: true,
|
||||
},
|
||||
want: TransitionResult{
|
||||
NextState: rulestore.StateFiring, NextConditionTrueSince: ptr(t0),
|
||||
NextFiredAt: ptr(t0.Add(10 * time.Minute)), NextLastNotifiedAt: ptr(t0.Add(10 * time.Minute)), Notify: ptr("firing"),
|
||||
},
|
||||
},
|
||||
{
|
||||
name: "firing, condition still true, no renotify interval -> stays firing, silent, preserves fired_at/last_notified",
|
||||
in: TransitionInput{
|
||||
CurrentState: rulestore.StateFiring, ConditionTrueSince: ptr(t0), FiredAt: ptr(t0), LastNotifiedAt: ptr(t0),
|
||||
RenotifyInterval: nil, Now: t0.Add(time.Hour), ConditionTrue: true,
|
||||
},
|
||||
want: TransitionResult{
|
||||
NextState: rulestore.StateFiring, NextConditionTrueSince: ptr(t0),
|
||||
NextFiredAt: ptr(t0), NextLastNotifiedAt: ptr(t0), Notify: nil,
|
||||
},
|
||||
},
|
||||
{
|
||||
name: "firing, condition still true, renotify interval not yet elapsed -> stays firing, silent",
|
||||
in: TransitionInput{
|
||||
CurrentState: rulestore.StateFiring, ConditionTrueSince: ptr(t0), FiredAt: ptr(t0), LastNotifiedAt: ptr(t0),
|
||||
RenotifyInterval: ptr(4 * time.Hour), Now: t0.Add(time.Hour), ConditionTrue: true,
|
||||
},
|
||||
want: TransitionResult{
|
||||
NextState: rulestore.StateFiring, NextConditionTrueSince: ptr(t0),
|
||||
NextFiredAt: ptr(t0), NextLastNotifiedAt: ptr(t0), Notify: nil,
|
||||
},
|
||||
},
|
||||
{
|
||||
name: "firing, condition still true, renotify interval elapsed -> re-fires, updates last_notified only",
|
||||
in: TransitionInput{
|
||||
CurrentState: rulestore.StateFiring, ConditionTrueSince: ptr(t0), FiredAt: ptr(t0), LastNotifiedAt: ptr(t0),
|
||||
RenotifyInterval: ptr(4 * time.Hour), Now: t0.Add(5 * time.Hour), ConditionTrue: true,
|
||||
},
|
||||
want: TransitionResult{
|
||||
NextState: rulestore.StateFiring, NextConditionTrueSince: ptr(t0),
|
||||
// fired_at (the episode start) is NOT bumped on a renotify -- only last_notified_at moves.
|
||||
NextFiredAt: ptr(t0), NextLastNotifiedAt: ptr(t0.Add(5 * time.Hour)), Notify: ptr("firing"),
|
||||
},
|
||||
},
|
||||
{
|
||||
name: "ok, condition false -> stays ok, no-op",
|
||||
in: TransitionInput{CurrentState: rulestore.StateOK, Now: t0, ConditionTrue: false},
|
||||
want: TransitionResult{NextState: rulestore.StateOK},
|
||||
},
|
||||
{
|
||||
name: "pending, condition false -> back to ok, NO notification (a blip inside the debounce window)",
|
||||
in: TransitionInput{
|
||||
CurrentState: rulestore.StatePending, ConditionTrueSince: ptr(t0),
|
||||
Now: t0.Add(time.Minute), ConditionTrue: false,
|
||||
},
|
||||
want: TransitionResult{NextState: rulestore.StateOK},
|
||||
},
|
||||
{
|
||||
name: "firing, condition false -> resolves, sends resolved notification",
|
||||
in: TransitionInput{
|
||||
CurrentState: rulestore.StateFiring, ConditionTrueSince: ptr(t0), FiredAt: ptr(t0), LastNotifiedAt: ptr(t0),
|
||||
Now: t0.Add(time.Hour), ConditionTrue: false,
|
||||
},
|
||||
want: TransitionResult{NextState: rulestore.StateOK, Notify: ptr("resolved")},
|
||||
},
|
||||
}
|
||||
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
got := ComputeTransition(tt.in)
|
||||
if got.NextState != tt.want.NextState {
|
||||
t.Errorf("NextState = %v, want %v", got.NextState, tt.want.NextState)
|
||||
}
|
||||
if !timePtrEqual(got.NextConditionTrueSince, tt.want.NextConditionTrueSince) {
|
||||
t.Errorf("NextConditionTrueSince = %v, want %v", got.NextConditionTrueSince, tt.want.NextConditionTrueSince)
|
||||
}
|
||||
if !timePtrEqual(got.NextFiredAt, tt.want.NextFiredAt) {
|
||||
t.Errorf("NextFiredAt = %v, want %v", got.NextFiredAt, tt.want.NextFiredAt)
|
||||
}
|
||||
if !timePtrEqual(got.NextLastNotifiedAt, tt.want.NextLastNotifiedAt) {
|
||||
t.Errorf("NextLastNotifiedAt = %v, want %v", got.NextLastNotifiedAt, tt.want.NextLastNotifiedAt)
|
||||
}
|
||||
if !strPtrEqual(got.Notify, tt.want.Notify) {
|
||||
t.Errorf("Notify = %v, want %v", strPtrDeref(got.Notify), strPtrDeref(tt.want.Notify))
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// TestPendingConditionTrueSinceIsWallClockNotCounter pins down the
|
||||
// specific property the design doc calls out as easy to accidentally
|
||||
// "simplify" away: debounce progress survives an evaluator gap (e.g.
|
||||
// downtime) because condition_true_since is a timestamp, not a tick
|
||||
// count. Two evaluations three hours apart, both condition=true, with a
|
||||
// for_minutes=5m debounce -- the second evaluation must fire immediately
|
||||
// because wall-clock time already satisfies the debounce, regardless of
|
||||
// how many (or how few) evaluations happened in between.
|
||||
func TestPendingConditionTrueSinceIsWallClockNotCounter(t *testing.T) {
|
||||
first := ComputeTransition(TransitionInput{
|
||||
CurrentState: rulestore.StateOK, ForMinutes: 5 * time.Minute, Now: t0, ConditionTrue: true,
|
||||
})
|
||||
if first.NextState != rulestore.StatePending {
|
||||
t.Fatalf("first eval: state = %v, want pending", first.NextState)
|
||||
}
|
||||
|
||||
// Simulate a large gap (evaluator downtime) before the next evaluation.
|
||||
second := ComputeTransition(TransitionInput{
|
||||
CurrentState: rulestore.StatePending, ConditionTrueSince: first.NextConditionTrueSince,
|
||||
ForMinutes: 5 * time.Minute, Now: t0.Add(3 * time.Hour), ConditionTrue: true,
|
||||
})
|
||||
if second.NextState != rulestore.StateFiring {
|
||||
t.Fatalf("second eval after gap: state = %v, want firing (wall-clock debounce should already be satisfied)", second.NextState)
|
||||
}
|
||||
}
|
||||
|
||||
func timePtrEqual(a, b *time.Time) bool {
|
||||
if a == nil || b == nil {
|
||||
return a == b
|
||||
}
|
||||
return a.Equal(*b)
|
||||
}
|
||||
|
||||
func strPtrEqual(a, b *string) bool {
|
||||
if a == nil || b == nil {
|
||||
return a == b
|
||||
}
|
||||
return *a == *b
|
||||
}
|
||||
|
||||
func strPtrDeref(s *string) string {
|
||||
if s == nil {
|
||||
return "<nil>"
|
||||
}
|
||||
return *s
|
||||
}
|
||||
@@ -0,0 +1,261 @@
|
||||
// Package httpapi is alerting's REST surface: rule CRUD, notification
|
||||
// target CRUD, and a read-only delivery log -- mirrors
|
||||
// api/internal/dashboards' Handler/Store shape (narrow interfaces,
|
||||
// pgx-backed production implementation, fakes in tests).
|
||||
package httpapi
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"log/slog"
|
||||
"net/http"
|
||||
|
||||
"github.com/sentry/sentry/alerting/internal/notifystore"
|
||||
"github.com/sentry/sentry/alerting/internal/rulestore"
|
||||
)
|
||||
|
||||
type ruleStore interface {
|
||||
Create(ctx context.Context, r *rulestore.Rule) error
|
||||
List(ctx context.Context) ([]rulestore.RuleWithState, error)
|
||||
Get(ctx context.Context, id string) (*rulestore.RuleWithState, error)
|
||||
Delete(ctx context.Context, id string) error
|
||||
}
|
||||
|
||||
type targetStore interface {
|
||||
Create(ctx context.Context, t *notifystore.Target) error
|
||||
List(ctx context.Context) ([]notifystore.Target, error)
|
||||
Get(ctx context.Context, id string) (*notifystore.Target, error)
|
||||
Delete(ctx context.Context, id string) error
|
||||
}
|
||||
|
||||
type deliveryReader interface {
|
||||
ListForRule(ctx context.Context, ruleID string, limit int) ([]rulestore.DeliveryLogEntry, error)
|
||||
}
|
||||
|
||||
type Handler struct {
|
||||
logger *slog.Logger
|
||||
rules ruleStore
|
||||
targets targetStore
|
||||
deliveries deliveryReader
|
||||
}
|
||||
|
||||
func NewHandler(logger *slog.Logger, rules ruleStore, targets targetStore, deliveries deliveryReader) *Handler {
|
||||
return &Handler{logger: logger, rules: rules, targets: targets, deliveries: deliveries}
|
||||
}
|
||||
|
||||
func (h *Handler) RegisterRoutes(mux *http.ServeMux) {
|
||||
mux.HandleFunc("GET /healthz", h.handleHealthz)
|
||||
|
||||
mux.HandleFunc("POST /rules", h.handleCreateRule)
|
||||
mux.HandleFunc("GET /rules", h.handleListRules)
|
||||
mux.HandleFunc("GET /rules/{id}", h.handleGetRule)
|
||||
mux.HandleFunc("DELETE /rules/{id}", h.handleDeleteRule)
|
||||
mux.HandleFunc("GET /rules/{id}/deliveries", h.handleListDeliveries)
|
||||
|
||||
mux.HandleFunc("POST /targets", h.handleCreateTarget)
|
||||
mux.HandleFunc("GET /targets", h.handleListTargets)
|
||||
mux.HandleFunc("GET /targets/{id}", h.handleGetTarget)
|
||||
mux.HandleFunc("DELETE /targets/{id}", h.handleDeleteTarget)
|
||||
}
|
||||
|
||||
const maxBodyBytes = 1 << 20 // 1 MiB, same cap as api/internal/queryapi and dashboards
|
||||
|
||||
func (h *Handler) handleHealthz(w http.ResponseWriter, _ *http.Request) {
|
||||
w.WriteHeader(http.StatusOK)
|
||||
}
|
||||
|
||||
// createRuleRequest mirrors rulestore.Rule except Enabled is a pointer:
|
||||
// a plain bool can't distinguish "omitted" from "explicitly false" on
|
||||
// decode, and Go's zero value for bool is false -- without this, a
|
||||
// create request that simply doesn't mention "enabled" would silently
|
||||
// create a rule the evaluator's claim query never picks up (found by
|
||||
// actually creating a rule through this endpoint and checking the
|
||||
// response). Omitted means enabled; only an explicit "enabled": false
|
||||
// creates a disabled rule.
|
||||
type createRuleRequest struct {
|
||||
rulestore.Rule
|
||||
Enabled *bool `json:"enabled"`
|
||||
}
|
||||
|
||||
func (h *Handler) handleCreateRule(w http.ResponseWriter, r *http.Request) {
|
||||
var req createRuleRequest
|
||||
if !decodeJSON(w, r, &req) {
|
||||
return
|
||||
}
|
||||
rule := req.Rule
|
||||
rule.Enabled = req.Enabled == nil || *req.Enabled
|
||||
if err := validateRule(&rule); err != nil {
|
||||
writeError(w, http.StatusBadRequest, err.Error())
|
||||
return
|
||||
}
|
||||
if err := h.rules.Create(r.Context(), &rule); err != nil {
|
||||
h.logger.Error("creating rule", "error", err)
|
||||
writeError(w, http.StatusInternalServerError, "creating rule failed")
|
||||
return
|
||||
}
|
||||
writeJSON(w, http.StatusCreated, rule)
|
||||
}
|
||||
|
||||
func (h *Handler) handleListRules(w http.ResponseWriter, r *http.Request) {
|
||||
rules, err := h.rules.List(r.Context())
|
||||
if err != nil {
|
||||
h.logger.Error("listing rules", "error", err)
|
||||
writeError(w, http.StatusInternalServerError, "listing rules failed")
|
||||
return
|
||||
}
|
||||
writeJSON(w, http.StatusOK, rules)
|
||||
}
|
||||
|
||||
func (h *Handler) handleGetRule(w http.ResponseWriter, r *http.Request) {
|
||||
rule, err := h.rules.Get(r.Context(), r.PathValue("id"))
|
||||
if err != nil {
|
||||
h.writeStoreErr(w, err, "fetching rule")
|
||||
return
|
||||
}
|
||||
writeJSON(w, http.StatusOK, rule)
|
||||
}
|
||||
|
||||
func (h *Handler) handleDeleteRule(w http.ResponseWriter, r *http.Request) {
|
||||
if err := h.rules.Delete(r.Context(), r.PathValue("id")); err != nil {
|
||||
h.writeStoreErr(w, err, "deleting rule")
|
||||
return
|
||||
}
|
||||
w.WriteHeader(http.StatusNoContent)
|
||||
}
|
||||
|
||||
func (h *Handler) handleListDeliveries(w http.ResponseWriter, r *http.Request) {
|
||||
entries, err := h.deliveries.ListForRule(r.Context(), r.PathValue("id"), 100)
|
||||
if err != nil {
|
||||
h.logger.Error("listing deliveries", "rule_id", r.PathValue("id"), "error", err)
|
||||
writeError(w, http.StatusInternalServerError, "listing deliveries failed")
|
||||
return
|
||||
}
|
||||
writeJSON(w, http.StatusOK, entries)
|
||||
}
|
||||
|
||||
func (h *Handler) handleCreateTarget(w http.ResponseWriter, r *http.Request) {
|
||||
var target notifystore.Target
|
||||
if !decodeJSON(w, r, &target) {
|
||||
return
|
||||
}
|
||||
if target.Name == "" {
|
||||
writeError(w, http.StatusBadRequest, "name must not be empty")
|
||||
return
|
||||
}
|
||||
if !notifystore.ValidKind(target.Kind) {
|
||||
writeError(w, http.StatusBadRequest, "kind must be one of webhook, slack, pagerduty")
|
||||
return
|
||||
}
|
||||
if target.WebhookURL == "" {
|
||||
writeError(w, http.StatusBadRequest, "webhook_url must not be empty")
|
||||
return
|
||||
}
|
||||
if err := h.targets.Create(r.Context(), &target); err != nil {
|
||||
h.logger.Error("creating notification target", "error", err)
|
||||
writeError(w, http.StatusInternalServerError, "creating notification target failed")
|
||||
return
|
||||
}
|
||||
writeJSON(w, http.StatusCreated, target)
|
||||
}
|
||||
|
||||
func (h *Handler) handleListTargets(w http.ResponseWriter, r *http.Request) {
|
||||
targets, err := h.targets.List(r.Context())
|
||||
if err != nil {
|
||||
h.logger.Error("listing notification targets", "error", err)
|
||||
writeError(w, http.StatusInternalServerError, "listing notification targets failed")
|
||||
return
|
||||
}
|
||||
writeJSON(w, http.StatusOK, targets)
|
||||
}
|
||||
|
||||
func (h *Handler) handleGetTarget(w http.ResponseWriter, r *http.Request) {
|
||||
target, err := h.targets.Get(r.Context(), r.PathValue("id"))
|
||||
if err != nil {
|
||||
h.writeStoreErr(w, err, "fetching notification target")
|
||||
return
|
||||
}
|
||||
writeJSON(w, http.StatusOK, target)
|
||||
}
|
||||
|
||||
func (h *Handler) handleDeleteTarget(w http.ResponseWriter, r *http.Request) {
|
||||
if err := h.targets.Delete(r.Context(), r.PathValue("id")); err != nil {
|
||||
h.writeStoreErr(w, err, "deleting notification target")
|
||||
return
|
||||
}
|
||||
w.WriteHeader(http.StatusNoContent)
|
||||
}
|
||||
|
||||
// validateRule enforces the shape the evaluator assumes:
|
||||
// condition_type-appropriate fields present, a sane interval floor.
|
||||
// Mirrors the DB CHECK constraints so bad input is rejected with a
|
||||
// clear message here rather than surfacing as an opaque constraint
|
||||
// violation from Postgres.
|
||||
func validateRule(r *rulestore.Rule) error {
|
||||
if r.Name == "" {
|
||||
return errBadRequest("name must not be empty")
|
||||
}
|
||||
if r.Query == "" {
|
||||
return errBadRequest("query must not be empty")
|
||||
}
|
||||
if r.NotificationTargetID == "" {
|
||||
return errBadRequest("notification_target_id must not be empty")
|
||||
}
|
||||
if r.EvalIntervalSeconds < 30 {
|
||||
return errBadRequest("eval_interval_seconds must be at least 30")
|
||||
}
|
||||
switch r.ConditionType {
|
||||
case rulestore.ConditionThreshold:
|
||||
if r.Comparator == nil || !rulestore.ValidComparator(*r.Comparator) {
|
||||
return errBadRequest("threshold rules require a valid comparator (gt, gte, lt, lte, eq, ne)")
|
||||
}
|
||||
if r.ThresholdValue == nil {
|
||||
return errBadRequest("threshold rules require threshold_value")
|
||||
}
|
||||
case rulestore.ConditionAbsence:
|
||||
// No comparator/threshold_value needed -- absence is "the query
|
||||
// returned zero rows in its own earliest=/latest= window."
|
||||
default:
|
||||
return errBadRequest("condition_type must be \"threshold\" or \"absence\"")
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
type badRequestError string
|
||||
|
||||
func (e badRequestError) Error() string { return string(e) }
|
||||
func errBadRequest(msg string) error { return badRequestError(msg) }
|
||||
|
||||
func (h *Handler) writeStoreErr(w http.ResponseWriter, err error, action string) {
|
||||
if errors.Is(err, rulestore.ErrNotFound) || errors.Is(err, notifystore.ErrNotFound) {
|
||||
writeError(w, http.StatusNotFound, "not found")
|
||||
return
|
||||
}
|
||||
h.logger.Error(action, "error", err)
|
||||
writeError(w, http.StatusInternalServerError, action+" failed")
|
||||
}
|
||||
|
||||
func decodeJSON(w http.ResponseWriter, r *http.Request, v any) bool {
|
||||
r.Body = http.MaxBytesReader(w, r.Body, maxBodyBytes)
|
||||
if err := json.NewDecoder(r.Body).Decode(v); err != nil {
|
||||
writeError(w, http.StatusBadRequest, "invalid JSON body: "+err.Error())
|
||||
return false
|
||||
}
|
||||
return true
|
||||
}
|
||||
|
||||
func writeJSON(w http.ResponseWriter, status int, v any) {
|
||||
w.Header().Set("Content-Type", "application/json")
|
||||
w.WriteHeader(status)
|
||||
_ = json.NewEncoder(w).Encode(v)
|
||||
}
|
||||
|
||||
type errorResponse struct {
|
||||
Error string `json:"error"`
|
||||
}
|
||||
|
||||
func writeError(w http.ResponseWriter, status int, msg string) {
|
||||
w.Header().Set("Content-Type", "application/json")
|
||||
w.WriteHeader(status)
|
||||
_ = json.NewEncoder(w).Encode(errorResponse{Error: msg})
|
||||
}
|
||||
@@ -0,0 +1,261 @@
|
||||
package httpapi
|
||||
|
||||
import (
|
||||
"context"
|
||||
"io"
|
||||
"log/slog"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"github.com/sentry/sentry/alerting/internal/notifystore"
|
||||
"github.com/sentry/sentry/alerting/internal/rulestore"
|
||||
)
|
||||
|
||||
type fakeRuleStore struct {
|
||||
rules map[string]*rulestore.RuleWithState
|
||||
}
|
||||
|
||||
func newFakeRuleStore() *fakeRuleStore {
|
||||
return &fakeRuleStore{rules: map[string]*rulestore.RuleWithState{}}
|
||||
}
|
||||
|
||||
func (f *fakeRuleStore) Create(_ context.Context, r *rulestore.Rule) error {
|
||||
r.ID = "rule-1"
|
||||
f.rules[r.ID] = &rulestore.RuleWithState{Rule: *r, State: rulestore.AlertState{RuleID: r.ID, State: rulestore.StateOK}}
|
||||
return nil
|
||||
}
|
||||
|
||||
func (f *fakeRuleStore) List(_ context.Context) ([]rulestore.RuleWithState, error) {
|
||||
var out []rulestore.RuleWithState
|
||||
for _, r := range f.rules {
|
||||
out = append(out, *r)
|
||||
}
|
||||
return out, nil
|
||||
}
|
||||
|
||||
func (f *fakeRuleStore) Get(_ context.Context, id string) (*rulestore.RuleWithState, error) {
|
||||
r, ok := f.rules[id]
|
||||
if !ok {
|
||||
return nil, rulestore.ErrNotFound
|
||||
}
|
||||
return r, nil
|
||||
}
|
||||
|
||||
func (f *fakeRuleStore) Delete(_ context.Context, id string) error {
|
||||
if _, ok := f.rules[id]; !ok {
|
||||
return rulestore.ErrNotFound
|
||||
}
|
||||
delete(f.rules, id)
|
||||
return nil
|
||||
}
|
||||
|
||||
type fakeTargetStore struct {
|
||||
targets map[string]*notifystore.Target
|
||||
}
|
||||
|
||||
func newFakeTargetStore() *fakeTargetStore {
|
||||
return &fakeTargetStore{targets: map[string]*notifystore.Target{}}
|
||||
}
|
||||
|
||||
func (f *fakeTargetStore) Create(_ context.Context, t *notifystore.Target) error {
|
||||
t.ID = "target-1"
|
||||
f.targets[t.ID] = t
|
||||
return nil
|
||||
}
|
||||
func (f *fakeTargetStore) List(_ context.Context) ([]notifystore.Target, error) {
|
||||
var out []notifystore.Target
|
||||
for _, t := range f.targets {
|
||||
out = append(out, *t)
|
||||
}
|
||||
return out, nil
|
||||
}
|
||||
func (f *fakeTargetStore) Get(_ context.Context, id string) (*notifystore.Target, error) {
|
||||
t, ok := f.targets[id]
|
||||
if !ok {
|
||||
return nil, notifystore.ErrNotFound
|
||||
}
|
||||
return t, nil
|
||||
}
|
||||
func (f *fakeTargetStore) Delete(_ context.Context, id string) error {
|
||||
if _, ok := f.targets[id]; !ok {
|
||||
return notifystore.ErrNotFound
|
||||
}
|
||||
delete(f.targets, id)
|
||||
return nil
|
||||
}
|
||||
|
||||
type fakeDeliveryReader struct {
|
||||
entries []rulestore.DeliveryLogEntry
|
||||
}
|
||||
|
||||
func (f *fakeDeliveryReader) ListForRule(_ context.Context, _ string, _ int) ([]rulestore.DeliveryLogEntry, error) {
|
||||
return f.entries, nil
|
||||
}
|
||||
|
||||
func newTestMux(rules ruleStore, targets targetStore, deliveries deliveryReader) *http.ServeMux {
|
||||
h := NewHandler(slog.New(slog.NewTextHandler(io.Discard, nil)), rules, targets, deliveries)
|
||||
mux := http.NewServeMux()
|
||||
h.RegisterRoutes(mux)
|
||||
return mux
|
||||
}
|
||||
|
||||
func doRequest(t *testing.T, mux *http.ServeMux, method, path, body string) *httptest.ResponseRecorder {
|
||||
t.Helper()
|
||||
var r io.Reader
|
||||
if body != "" {
|
||||
r = strings.NewReader(body)
|
||||
}
|
||||
req := httptest.NewRequest(method, path, r)
|
||||
rec := httptest.NewRecorder()
|
||||
mux.ServeHTTP(rec, req)
|
||||
return rec
|
||||
}
|
||||
|
||||
func TestCreateThresholdRule(t *testing.T) {
|
||||
targets := newFakeTargetStore()
|
||||
targets.targets["target-1"] = ¬ifystore.Target{ID: "target-1"}
|
||||
mux := newTestMux(newFakeRuleStore(), targets, &fakeDeliveryReader{})
|
||||
|
||||
rec := doRequest(t, mux, http.MethodPost, "/rules", `{
|
||||
"name": "High error rate", "query": "service=api | where status>=500 | stats count",
|
||||
"condition_type": "threshold", "comparator": "gt", "threshold_value": 100,
|
||||
"eval_interval_seconds": 60, "notification_target_id": "target-1"
|
||||
}`)
|
||||
if rec.Code != http.StatusCreated {
|
||||
t.Fatalf("status = %d, want 201; body=%s", rec.Code, rec.Body.String())
|
||||
}
|
||||
}
|
||||
|
||||
// TestCreateRuleDefaultsToEnabledWhenOmitted guards against a real bug
|
||||
// caught by actually calling this endpoint: a plain `bool` JSON field
|
||||
// can't distinguish "omitted" from "explicitly false," and Go's zero
|
||||
// value for bool is false -- without createRuleRequest's *bool handling,
|
||||
// a create request that simply didn't mention "enabled" silently created
|
||||
// a rule the evaluator's claim query would never pick up.
|
||||
func TestCreateRuleDefaultsToEnabledWhenOmitted(t *testing.T) {
|
||||
rules := newFakeRuleStore()
|
||||
mux := newTestMux(rules, newFakeTargetStore(), &fakeDeliveryReader{})
|
||||
|
||||
rec := doRequest(t, mux, http.MethodPost, "/rules", `{
|
||||
"name": "no enabled field", "query": "service=api", "condition_type": "absence",
|
||||
"eval_interval_seconds": 60, "notification_target_id": "target-1"
|
||||
}`)
|
||||
if rec.Code != http.StatusCreated {
|
||||
t.Fatalf("status = %d, want 201; body=%s", rec.Code, rec.Body.String())
|
||||
}
|
||||
if !rules.rules["rule-1"].Enabled {
|
||||
t.Fatalf("expected a rule created without an explicit \"enabled\" field to default to enabled=true")
|
||||
}
|
||||
}
|
||||
|
||||
func TestCreateRuleRespectsExplicitDisabled(t *testing.T) {
|
||||
rules := newFakeRuleStore()
|
||||
mux := newTestMux(rules, newFakeTargetStore(), &fakeDeliveryReader{})
|
||||
|
||||
rec := doRequest(t, mux, http.MethodPost, "/rules", `{
|
||||
"name": "explicitly disabled", "query": "service=api", "condition_type": "absence",
|
||||
"eval_interval_seconds": 60, "notification_target_id": "target-1", "enabled": false
|
||||
}`)
|
||||
if rec.Code != http.StatusCreated {
|
||||
t.Fatalf("status = %d, want 201; body=%s", rec.Code, rec.Body.String())
|
||||
}
|
||||
if rules.rules["rule-1"].Enabled {
|
||||
t.Fatalf("expected an explicit \"enabled\": false to be respected")
|
||||
}
|
||||
}
|
||||
|
||||
func TestCreateThresholdRuleRejectsMissingComparator(t *testing.T) {
|
||||
mux := newTestMux(newFakeRuleStore(), newFakeTargetStore(), &fakeDeliveryReader{})
|
||||
rec := doRequest(t, mux, http.MethodPost, "/rules", `{
|
||||
"name": "bad rule", "query": "service=api", "condition_type": "threshold",
|
||||
"eval_interval_seconds": 60, "notification_target_id": "target-1"
|
||||
}`)
|
||||
if rec.Code != http.StatusBadRequest {
|
||||
t.Fatalf("status = %d, want 400; body=%s", rec.Code, rec.Body.String())
|
||||
}
|
||||
}
|
||||
|
||||
func TestCreateAbsenceRuleDoesNotRequireComparator(t *testing.T) {
|
||||
mux := newTestMux(newFakeRuleStore(), newFakeTargetStore(), &fakeDeliveryReader{})
|
||||
rec := doRequest(t, mux, http.MethodPost, "/rules", `{
|
||||
"name": "no heartbeat", "query": "service=payments earliest=-5m", "condition_type": "absence",
|
||||
"eval_interval_seconds": 60, "notification_target_id": "target-1"
|
||||
}`)
|
||||
if rec.Code != http.StatusCreated {
|
||||
t.Fatalf("status = %d, want 201; body=%s", rec.Code, rec.Body.String())
|
||||
}
|
||||
}
|
||||
|
||||
func TestCreateRuleRejectsShortInterval(t *testing.T) {
|
||||
mux := newTestMux(newFakeRuleStore(), newFakeTargetStore(), &fakeDeliveryReader{})
|
||||
rec := doRequest(t, mux, http.MethodPost, "/rules", `{
|
||||
"name": "too fast", "query": "service=api", "condition_type": "absence",
|
||||
"eval_interval_seconds": 5, "notification_target_id": "target-1"
|
||||
}`)
|
||||
if rec.Code != http.StatusBadRequest {
|
||||
t.Fatalf("status = %d, want 400; body=%s", rec.Code, rec.Body.String())
|
||||
}
|
||||
}
|
||||
|
||||
func TestGetRuleNotFound(t *testing.T) {
|
||||
mux := newTestMux(newFakeRuleStore(), newFakeTargetStore(), &fakeDeliveryReader{})
|
||||
rec := doRequest(t, mux, http.MethodGet, "/rules/nope", "")
|
||||
if rec.Code != http.StatusNotFound {
|
||||
t.Fatalf("status = %d, want 404", rec.Code)
|
||||
}
|
||||
}
|
||||
|
||||
func TestDeleteRule(t *testing.T) {
|
||||
rules := newFakeRuleStore()
|
||||
rules.rules["rule-1"] = &rulestore.RuleWithState{Rule: rulestore.Rule{ID: "rule-1"}}
|
||||
mux := newTestMux(rules, newFakeTargetStore(), &fakeDeliveryReader{})
|
||||
|
||||
rec := doRequest(t, mux, http.MethodDelete, "/rules/rule-1", "")
|
||||
if rec.Code != http.StatusNoContent {
|
||||
t.Fatalf("status = %d, want 204", rec.Code)
|
||||
}
|
||||
if _, ok := rules.rules["rule-1"]; ok {
|
||||
t.Fatalf("expected rule to be deleted")
|
||||
}
|
||||
}
|
||||
|
||||
func TestCreateTargetRejectsInvalidKind(t *testing.T) {
|
||||
mux := newTestMux(newFakeRuleStore(), newFakeTargetStore(), &fakeDeliveryReader{})
|
||||
rec := doRequest(t, mux, http.MethodPost, "/targets", `{"name": "x", "kind": "carrier-pigeon", "webhook_url": "https://example.com"}`)
|
||||
if rec.Code != http.StatusBadRequest {
|
||||
t.Fatalf("status = %d, want 400; body=%s", rec.Code, rec.Body.String())
|
||||
}
|
||||
}
|
||||
|
||||
func TestCreateSlackTarget(t *testing.T) {
|
||||
mux := newTestMux(newFakeRuleStore(), newFakeTargetStore(), &fakeDeliveryReader{})
|
||||
rec := doRequest(t, mux, http.MethodPost, "/targets", `{"name": "oncall", "kind": "slack", "webhook_url": "https://hooks.slack.com/services/x"}`)
|
||||
if rec.Code != http.StatusCreated {
|
||||
t.Fatalf("status = %d, want 201; body=%s", rec.Code, rec.Body.String())
|
||||
}
|
||||
}
|
||||
|
||||
func TestListDeliveriesForRule(t *testing.T) {
|
||||
deliveries := &fakeDeliveryReader{entries: []rulestore.DeliveryLogEntry{
|
||||
{ID: 1, RuleID: "rule-1", EventType: "firing", Status: "sent"},
|
||||
}}
|
||||
mux := newTestMux(newFakeRuleStore(), newFakeTargetStore(), deliveries)
|
||||
|
||||
rec := doRequest(t, mux, http.MethodGet, "/rules/rule-1/deliveries", "")
|
||||
if rec.Code != http.StatusOK {
|
||||
t.Fatalf("status = %d, want 200; body=%s", rec.Code, rec.Body.String())
|
||||
}
|
||||
if !strings.Contains(rec.Body.String(), `"status":"sent"`) {
|
||||
t.Fatalf("expected delivery entry in response, got: %s", rec.Body.String())
|
||||
}
|
||||
}
|
||||
|
||||
func TestHandleHealthz(t *testing.T) {
|
||||
mux := newTestMux(newFakeRuleStore(), newFakeTargetStore(), &fakeDeliveryReader{})
|
||||
rec := doRequest(t, mux, http.MethodGet, "/healthz", "")
|
||||
if rec.Code != http.StatusOK {
|
||||
t.Fatalf("status = %d, want 200", rec.Code)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,24 @@
|
||||
// Package httpserver holds cross-handler HTTP concerns for /alerting.
|
||||
// Deliberately duplicated from api/internal/httpserver rather than
|
||||
// shared -- see /docs/phase-3-alerting-design.md's component-boundary
|
||||
// note: no shared Go store/HTTP code between api and alerting, the same
|
||||
// repo convention that only /proto is shared code.
|
||||
package httpserver
|
||||
|
||||
import "net/http"
|
||||
|
||||
// WithCORS is deliberately permissive by default, same posture and
|
||||
// reasoning as api's: no auth yet, the SvelteKit dev server runs on a
|
||||
// different origin. Tighten alongside adding real auth.
|
||||
func WithCORS(next http.Handler, allowedOrigin string) http.Handler {
|
||||
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
w.Header().Set("Access-Control-Allow-Origin", allowedOrigin)
|
||||
w.Header().Set("Access-Control-Allow-Methods", "GET, POST, PUT, DELETE, OPTIONS")
|
||||
w.Header().Set("Access-Control-Allow-Headers", "Content-Type")
|
||||
if r.Method == http.MethodOptions {
|
||||
w.WriteHeader(http.StatusNoContent)
|
||||
return
|
||||
}
|
||||
next.ServeHTTP(w, r)
|
||||
})
|
||||
}
|
||||
@@ -0,0 +1,114 @@
|
||||
// Package notifystore is pgx-backed CRUD for notification_targets.
|
||||
package notifystore
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
|
||||
"github.com/google/uuid"
|
||||
"github.com/jackc/pgx/v5"
|
||||
"github.com/jackc/pgx/v5/pgxpool"
|
||||
)
|
||||
|
||||
var ErrNotFound = errors.New("not found")
|
||||
|
||||
type Kind string
|
||||
|
||||
const (
|
||||
KindWebhook Kind = "webhook"
|
||||
KindSlack Kind = "slack"
|
||||
KindPagerDuty Kind = "pagerduty"
|
||||
)
|
||||
|
||||
func ValidKind(k Kind) bool {
|
||||
switch k {
|
||||
case KindWebhook, KindSlack, KindPagerDuty:
|
||||
return true
|
||||
default:
|
||||
return false
|
||||
}
|
||||
}
|
||||
|
||||
type Target struct {
|
||||
ID string `json:"id"`
|
||||
TenantID string `json:"tenant_id"`
|
||||
Name string `json:"name"`
|
||||
Kind Kind `json:"kind"`
|
||||
WebhookURL string `json:"webhook_url"`
|
||||
PayloadTemplate *string `json:"payload_template,omitempty"`
|
||||
Headers json.RawMessage `json:"headers,omitempty"`
|
||||
Secret *string `json:"secret,omitempty"`
|
||||
CreatedBy string `json:"created_by"`
|
||||
}
|
||||
|
||||
type Store struct {
|
||||
pool *pgxpool.Pool
|
||||
}
|
||||
|
||||
func NewStore(pool *pgxpool.Pool) *Store {
|
||||
return &Store{pool: pool}
|
||||
}
|
||||
|
||||
func (s *Store) Create(ctx context.Context, t *Target) error {
|
||||
t.ID = uuid.NewString()
|
||||
if t.TenantID == "" {
|
||||
t.TenantID = "default"
|
||||
}
|
||||
if t.CreatedBy == "" {
|
||||
t.CreatedBy = "anonymous"
|
||||
}
|
||||
if len(t.Headers) == 0 {
|
||||
t.Headers = json.RawMessage(`{}`)
|
||||
}
|
||||
_, err := s.pool.Exec(ctx, `
|
||||
INSERT INTO notification_targets (id, tenant_id, name, kind, webhook_url, payload_template, headers, secret, created_by)
|
||||
VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9)`,
|
||||
t.ID, t.TenantID, t.Name, t.Kind, t.WebhookURL, t.PayloadTemplate, t.Headers, t.Secret, t.CreatedBy)
|
||||
return err
|
||||
}
|
||||
|
||||
func (s *Store) List(ctx context.Context) ([]Target, error) {
|
||||
rows, err := s.pool.Query(ctx, `
|
||||
SELECT id, tenant_id, name, kind, webhook_url, payload_template, headers, secret, created_by
|
||||
FROM notification_targets ORDER BY created_at DESC`)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
defer rows.Close()
|
||||
|
||||
var out []Target
|
||||
for rows.Next() {
|
||||
var t Target
|
||||
if err := rows.Scan(&t.ID, &t.TenantID, &t.Name, &t.Kind, &t.WebhookURL, &t.PayloadTemplate, &t.Headers, &t.Secret, &t.CreatedBy); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
out = append(out, t)
|
||||
}
|
||||
return out, rows.Err()
|
||||
}
|
||||
|
||||
func (s *Store) Get(ctx context.Context, id string) (*Target, error) {
|
||||
var t Target
|
||||
row := s.pool.QueryRow(ctx, `
|
||||
SELECT id, tenant_id, name, kind, webhook_url, payload_template, headers, secret, created_by
|
||||
FROM notification_targets WHERE id = $1`, id)
|
||||
if err := row.Scan(&t.ID, &t.TenantID, &t.Name, &t.Kind, &t.WebhookURL, &t.PayloadTemplate, &t.Headers, &t.Secret, &t.CreatedBy); err != nil {
|
||||
if errors.Is(err, pgx.ErrNoRows) {
|
||||
return nil, ErrNotFound
|
||||
}
|
||||
return nil, err
|
||||
}
|
||||
return &t, nil
|
||||
}
|
||||
|
||||
func (s *Store) Delete(ctx context.Context, id string) error {
|
||||
tag, err := s.pool.Exec(ctx, `DELETE FROM notification_targets WHERE id = $1`, id)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if tag.RowsAffected() == 0 {
|
||||
return ErrNotFound
|
||||
}
|
||||
return nil
|
||||
}
|
||||
@@ -0,0 +1,75 @@
|
||||
// Package queryclient is a thin HTTP client to /api's POST /query --
|
||||
// alerting never imports querylang or talks to ClickHouse/Tantivy
|
||||
// directly, same precedent sentryctl query and the web UI's dashboard
|
||||
// panels already set: one query-execution path, reused everywhere.
|
||||
package queryclient
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"net/http"
|
||||
"time"
|
||||
)
|
||||
|
||||
type Result struct {
|
||||
Columns []string `json:"columns"`
|
||||
Rows [][]any `json:"rows"`
|
||||
}
|
||||
|
||||
type errorResponse struct {
|
||||
Error string `json:"error"`
|
||||
}
|
||||
|
||||
type Client struct {
|
||||
baseURL string
|
||||
http *http.Client
|
||||
}
|
||||
|
||||
func New(baseURL string) *Client {
|
||||
return &Client{baseURL: baseURL, http: &http.Client{}}
|
||||
}
|
||||
|
||||
// Query runs query (already time-range-injected by the caller, if
|
||||
// applicable) against /api's POST /query and returns the result. A
|
||||
// non-2xx response or a request/decode failure is returned as an error
|
||||
// -- callers (the evaluator) treat any error here as an evaluation
|
||||
// error, never as "condition false" (see
|
||||
// /docs/phase-3-alerting-design.md's fix 3).
|
||||
func (c *Client) Query(ctx context.Context, query, language string, timeout time.Duration) (*Result, error) {
|
||||
ctx, cancel := context.WithTimeout(ctx, timeout)
|
||||
defer cancel()
|
||||
|
||||
body, err := json.Marshal(map[string]string{"query": query, "language": language})
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("encoding query request: %w", err)
|
||||
}
|
||||
|
||||
req, err := http.NewRequestWithContext(ctx, http.MethodPost, c.baseURL+"/query", bytes.NewReader(body))
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("building query request: %w", err)
|
||||
}
|
||||
req.Header.Set("Content-Type", "application/json")
|
||||
|
||||
resp, err := c.http.Do(req)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("calling api /query: %w", err)
|
||||
}
|
||||
defer resp.Body.Close()
|
||||
|
||||
if resp.StatusCode != http.StatusOK {
|
||||
var errBody errorResponse
|
||||
_ = json.NewDecoder(resp.Body).Decode(&errBody)
|
||||
if errBody.Error != "" {
|
||||
return nil, fmt.Errorf("api /query failed (%d): %s", resp.StatusCode, errBody.Error)
|
||||
}
|
||||
return nil, fmt.Errorf("api /query failed with status %d", resp.StatusCode)
|
||||
}
|
||||
|
||||
var result Result
|
||||
if err := json.NewDecoder(resp.Body).Decode(&result); err != nil {
|
||||
return nil, fmt.Errorf("decoding query response: %w", err)
|
||||
}
|
||||
return &result, nil
|
||||
}
|
||||
@@ -0,0 +1,357 @@
|
||||
// Package rulestore is pgx-backed CRUD for alert_rules/alert_state, plus
|
||||
// the two operations that make the evaluator's concurrency and delivery
|
||||
// guarantees hold (see /docs/phase-3-alerting-design.md's fixes 1 and 2):
|
||||
// ClaimDueRules (atomic claim-before-evaluate) and ApplyTransition (the
|
||||
// state transition and the delivery_log outbox insert in one DB
|
||||
// transaction).
|
||||
package rulestore
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"fmt"
|
||||
"time"
|
||||
|
||||
"github.com/google/uuid"
|
||||
"github.com/jackc/pgx/v5"
|
||||
"github.com/jackc/pgx/v5/pgxpool"
|
||||
)
|
||||
|
||||
var ErrNotFound = errors.New("not found")
|
||||
|
||||
type ConditionType string
|
||||
|
||||
const (
|
||||
ConditionThreshold ConditionType = "threshold"
|
||||
ConditionAbsence ConditionType = "absence"
|
||||
)
|
||||
|
||||
func ValidConditionType(c ConditionType) bool {
|
||||
return c == ConditionThreshold || c == ConditionAbsence
|
||||
}
|
||||
|
||||
type Comparator string
|
||||
|
||||
const (
|
||||
Gt Comparator = "gt"
|
||||
Gte Comparator = "gte"
|
||||
Lt Comparator = "lt"
|
||||
Lte Comparator = "lte"
|
||||
Eq Comparator = "eq"
|
||||
Ne Comparator = "ne"
|
||||
)
|
||||
|
||||
func ValidComparator(c Comparator) bool {
|
||||
switch c {
|
||||
case Gt, Gte, Lt, Lte, Eq, Ne:
|
||||
return true
|
||||
default:
|
||||
return false
|
||||
}
|
||||
}
|
||||
|
||||
type State string
|
||||
|
||||
const (
|
||||
StateOK State = "ok"
|
||||
StatePending State = "pending"
|
||||
StateFiring State = "firing"
|
||||
)
|
||||
|
||||
type Rule struct {
|
||||
ID string `json:"id"`
|
||||
TenantID string `json:"tenant_id"`
|
||||
Name string `json:"name"`
|
||||
Description string `json:"description"`
|
||||
Query string `json:"query"`
|
||||
QueryLanguage string `json:"query_language"`
|
||||
ConditionType ConditionType `json:"condition_type"`
|
||||
Comparator *Comparator `json:"comparator,omitempty"`
|
||||
ThresholdValue *float64 `json:"threshold_value,omitempty"`
|
||||
EvalIntervalSeconds int `json:"eval_interval_seconds"`
|
||||
ForMinutes int `json:"for_minutes"`
|
||||
RenotifyIntervalMinutes *int `json:"renotify_interval_minutes,omitempty"`
|
||||
NotificationTargetID string `json:"notification_target_id"`
|
||||
Enabled bool `json:"enabled"`
|
||||
CreatedBy string `json:"created_by"`
|
||||
}
|
||||
|
||||
type AlertState struct {
|
||||
RuleID string `json:"rule_id"`
|
||||
State State `json:"state"`
|
||||
ConditionTrueSince *time.Time `json:"condition_true_since,omitempty"`
|
||||
FiredAt *time.Time `json:"fired_at,omitempty"`
|
||||
LastNotifiedAt *time.Time `json:"last_notified_at,omitempty"`
|
||||
LastEvaluatedAt *time.Time `json:"last_evaluated_at,omitempty"`
|
||||
LastEvalStatus string `json:"last_eval_status"`
|
||||
LastError *string `json:"last_error,omitempty"`
|
||||
LastValue *float64 `json:"last_value,omitempty"`
|
||||
ConsecutiveErrors int `json:"consecutive_errors"`
|
||||
}
|
||||
|
||||
// RuleWithState is a rule joined with its current live state, exactly
|
||||
// what the evaluator needs to run one evaluation and what the read API
|
||||
// returns.
|
||||
type RuleWithState struct {
|
||||
Rule
|
||||
State AlertState `json:"state"`
|
||||
}
|
||||
|
||||
type Store struct {
|
||||
pool *pgxpool.Pool
|
||||
}
|
||||
|
||||
func NewStore(pool *pgxpool.Pool) *Store {
|
||||
return &Store{pool: pool}
|
||||
}
|
||||
|
||||
// Create inserts the rule and its initial alert_state row in one
|
||||
// transaction. A rule with no alert_state row is silently never picked
|
||||
// up by ClaimDueRules -- see the design doc's explicit warning about
|
||||
// this exact failure mode.
|
||||
func (s *Store) Create(ctx context.Context, r *Rule) error {
|
||||
r.ID = uuid.NewString()
|
||||
if r.TenantID == "" {
|
||||
r.TenantID = "default"
|
||||
}
|
||||
if r.CreatedBy == "" {
|
||||
r.CreatedBy = "anonymous"
|
||||
}
|
||||
|
||||
tx, err := s.pool.Begin(ctx)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
defer tx.Rollback(ctx)
|
||||
|
||||
_, err = tx.Exec(ctx, `
|
||||
INSERT INTO alert_rules (id, tenant_id, name, description, query, query_language, condition_type,
|
||||
comparator, threshold_value, eval_interval_seconds, for_minutes,
|
||||
renotify_interval_minutes, notification_target_id, enabled, created_by)
|
||||
VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12, $13, $14, $15)`,
|
||||
r.ID, r.TenantID, r.Name, r.Description, r.Query, r.QueryLanguage, r.ConditionType,
|
||||
r.Comparator, r.ThresholdValue, r.EvalIntervalSeconds, r.ForMinutes,
|
||||
r.RenotifyIntervalMinutes, r.NotificationTargetID, r.Enabled, r.CreatedBy)
|
||||
if err != nil {
|
||||
return fmt.Errorf("inserting rule: %w", err)
|
||||
}
|
||||
|
||||
_, err = tx.Exec(ctx, `
|
||||
INSERT INTO alert_state (rule_id, state, last_eval_status, next_eval_at)
|
||||
VALUES ($1, 'ok', 'ok', now())`, r.ID)
|
||||
if err != nil {
|
||||
return fmt.Errorf("inserting initial alert_state: %w", err)
|
||||
}
|
||||
|
||||
return tx.Commit(ctx)
|
||||
}
|
||||
|
||||
func (s *Store) List(ctx context.Context) ([]RuleWithState, error) {
|
||||
rows, err := s.pool.Query(ctx, ruleWithStateSelect+" ORDER BY r.created_at DESC")
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
defer rows.Close()
|
||||
|
||||
var out []RuleWithState
|
||||
for rows.Next() {
|
||||
rs, err := scanRuleWithState(rows)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
out = append(out, rs)
|
||||
}
|
||||
return out, rows.Err()
|
||||
}
|
||||
|
||||
func (s *Store) Get(ctx context.Context, id string) (*RuleWithState, error) {
|
||||
rows, err := s.pool.Query(ctx, ruleWithStateSelect+" WHERE r.id = $1", id)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
defer rows.Close()
|
||||
|
||||
if !rows.Next() {
|
||||
return nil, ErrNotFound
|
||||
}
|
||||
rs, err := scanRuleWithState(rows)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return &rs, nil
|
||||
}
|
||||
|
||||
func (s *Store) Delete(ctx context.Context, id string) error {
|
||||
tag, err := s.pool.Exec(ctx, `DELETE FROM alert_rules WHERE id = $1`, id)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if tag.RowsAffected() == 0 {
|
||||
return ErrNotFound
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
const ruleWithStateSelect = `
|
||||
SELECT r.id, r.tenant_id, r.name, r.description, r.query, r.query_language, r.condition_type,
|
||||
r.comparator, r.threshold_value, r.eval_interval_seconds, r.for_minutes,
|
||||
r.renotify_interval_minutes, r.notification_target_id, r.enabled, r.created_by,
|
||||
s.state, s.condition_true_since, s.fired_at, s.last_notified_at, s.last_evaluated_at,
|
||||
s.last_eval_status, s.last_error, s.last_value, s.consecutive_errors
|
||||
FROM alert_rules r JOIN alert_state s ON s.rule_id = r.id`
|
||||
|
||||
func scanRuleWithState(rows pgx.Rows) (RuleWithState, error) {
|
||||
var rs RuleWithState
|
||||
err := rows.Scan(
|
||||
&rs.ID, &rs.TenantID, &rs.Name, &rs.Description, &rs.Query, &rs.QueryLanguage, &rs.ConditionType,
|
||||
&rs.Comparator, &rs.ThresholdValue, &rs.EvalIntervalSeconds, &rs.ForMinutes,
|
||||
&rs.RenotifyIntervalMinutes, &rs.NotificationTargetID, &rs.Enabled, &rs.CreatedBy,
|
||||
&rs.State.State, &rs.State.ConditionTrueSince, &rs.State.FiredAt, &rs.State.LastNotifiedAt, &rs.State.LastEvaluatedAt,
|
||||
&rs.State.LastEvalStatus, &rs.State.LastError, &rs.State.LastValue, &rs.State.ConsecutiveErrors)
|
||||
rs.State.RuleID = rs.ID
|
||||
return rs, err
|
||||
}
|
||||
|
||||
// ClaimDueRules atomically claims up to batchSize due, enabled rules --
|
||||
// see /docs/phase-3-alerting-design.md's fix 1. next_eval_at is bumped
|
||||
// forward *before* returning, so a second scheduler tick (or a second
|
||||
// evaluator replica, later) can't re-select the same rule while this
|
||||
// one is still being evaluated. SKIP LOCKED means a concurrent claim
|
||||
// query never blocks on rows another claim already has locked -- it
|
||||
// just moves on to the next candidate.
|
||||
func (s *Store) ClaimDueRules(ctx context.Context, batchSize int) ([]RuleWithState, error) {
|
||||
rows, err := s.pool.Query(ctx, `
|
||||
WITH due AS (
|
||||
SELECT s.rule_id
|
||||
FROM alert_state s
|
||||
JOIN alert_rules r ON r.id = s.rule_id
|
||||
WHERE s.next_eval_at <= now() AND r.enabled
|
||||
ORDER BY s.next_eval_at
|
||||
LIMIT $1
|
||||
FOR UPDATE OF s SKIP LOCKED
|
||||
)
|
||||
UPDATE alert_state s
|
||||
SET next_eval_at = now() + make_interval(secs => r.eval_interval_seconds),
|
||||
claimed_at = now()
|
||||
FROM due, alert_rules r
|
||||
WHERE s.rule_id = due.rule_id AND r.id = s.rule_id
|
||||
RETURNING r.id, r.tenant_id, r.name, r.description, r.query, r.query_language, r.condition_type,
|
||||
r.comparator, r.threshold_value, r.eval_interval_seconds, r.for_minutes,
|
||||
r.renotify_interval_minutes, r.notification_target_id, r.enabled, r.created_by,
|
||||
s.state, s.condition_true_since, s.fired_at, s.last_notified_at, s.last_evaluated_at,
|
||||
s.last_eval_status, s.last_error, s.last_value, s.consecutive_errors`,
|
||||
batchSize)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
defer rows.Close()
|
||||
|
||||
var out []RuleWithState
|
||||
for rows.Next() {
|
||||
rs, err := scanRuleWithState(rows)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
out = append(out, rs)
|
||||
}
|
||||
return out, rows.Err()
|
||||
}
|
||||
|
||||
// NotifyEvent, if non-nil, is inserted into delivery_log in the same
|
||||
// transaction as the state update -- the transactional outbox from fix
|
||||
// 2. Payload is the already-rendered notification body (rulestore
|
||||
// doesn't know how to format one; that's internal/delivery's job).
|
||||
type NotifyEvent struct {
|
||||
NotificationTargetID string
|
||||
EventType string // "firing" | "resolved"
|
||||
Payload []byte // JSON
|
||||
}
|
||||
|
||||
// ApplyTransition writes a successful evaluation's outcome: the new
|
||||
// alert_state fields, and -- atomically, in the same transaction -- a
|
||||
// pending delivery_log row if notify is non-nil. See fix 2: "decided to
|
||||
// notify" becomes durable exactly once, in lockstep with the state
|
||||
// change, before any network call to a notification target is
|
||||
// attempted.
|
||||
func (s *Store) ApplyTransition(ctx context.Context, ruleID string, next AlertState, notify *NotifyEvent) error {
|
||||
tx, err := s.pool.Begin(ctx)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
defer tx.Rollback(ctx)
|
||||
|
||||
_, err = tx.Exec(ctx, `
|
||||
UPDATE alert_state SET
|
||||
state = $1, condition_true_since = $2, fired_at = $3, last_notified_at = $4,
|
||||
last_evaluated_at = now(), last_eval_status = 'ok', last_error = NULL,
|
||||
last_value = $5, consecutive_errors = 0
|
||||
WHERE rule_id = $6`,
|
||||
next.State, next.ConditionTrueSince, next.FiredAt, next.LastNotifiedAt, next.LastValue, ruleID)
|
||||
if err != nil {
|
||||
return fmt.Errorf("updating alert_state: %w", err)
|
||||
}
|
||||
|
||||
if notify != nil {
|
||||
_, err = tx.Exec(ctx, `
|
||||
INSERT INTO delivery_log (rule_id, notification_target_id, event_type, status, next_attempt_at, payload)
|
||||
VALUES ($1, $2, $3, 'pending', now(), $4)`,
|
||||
ruleID, notify.NotificationTargetID, notify.EventType, notify.Payload)
|
||||
if err != nil {
|
||||
return fmt.Errorf("inserting delivery_log outbox row: %w", err)
|
||||
}
|
||||
}
|
||||
|
||||
return tx.Commit(ctx)
|
||||
}
|
||||
|
||||
// DeliveryLogEntry is the read shape the web UI's delivery log view and
|
||||
// the httpapi read endpoint use -- deliberately excludes `payload`
|
||||
// (the rendered notification body isn't needed for the list view, and
|
||||
// keeping it out of the default read path avoids echoing
|
||||
// target-specific formatting, e.g. a rendered webhook secret, back over
|
||||
// HTTP by default).
|
||||
type DeliveryLogEntry struct {
|
||||
ID int64 `json:"id"`
|
||||
RuleID string `json:"rule_id"`
|
||||
NotificationTargetID string `json:"notification_target_id"`
|
||||
EventType string `json:"event_type"`
|
||||
Status string `json:"status"`
|
||||
AttemptCount int `json:"attempt_count"`
|
||||
LastError *string `json:"last_error,omitempty"`
|
||||
ResponseStatus *int `json:"response_status,omitempty"`
|
||||
CreatedAt time.Time `json:"created_at"`
|
||||
}
|
||||
|
||||
// ListForRule reads delivery_log's UI-visible role -- "why didn't I get
|
||||
// paged" -- most recent first.
|
||||
func (s *Store) ListForRule(ctx context.Context, ruleID string, limit int) ([]DeliveryLogEntry, error) {
|
||||
rows, err := s.pool.Query(ctx, `
|
||||
SELECT id, rule_id, notification_target_id, event_type, status, attempt_count, last_error, response_status, created_at
|
||||
FROM delivery_log WHERE rule_id = $1 ORDER BY created_at DESC LIMIT $2`, ruleID, limit)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
defer rows.Close()
|
||||
|
||||
var out []DeliveryLogEntry
|
||||
for rows.Next() {
|
||||
var e DeliveryLogEntry
|
||||
if err := rows.Scan(&e.ID, &e.RuleID, &e.NotificationTargetID, &e.EventType, &e.Status, &e.AttemptCount, &e.LastError, &e.ResponseStatus, &e.CreatedAt); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
out = append(out, e)
|
||||
}
|
||||
return out, rows.Err()
|
||||
}
|
||||
|
||||
// RecordError updates only the error-tracking fields on alert_state --
|
||||
// state itself is left untouched, per fix 3: an evaluation error must
|
||||
// never transition state (an error is not "condition false").
|
||||
func (s *Store) RecordError(ctx context.Context, ruleID, errMsg string) error {
|
||||
_, err := s.pool.Exec(ctx, `
|
||||
UPDATE alert_state SET
|
||||
last_evaluated_at = now(), last_eval_status = 'error', last_error = $1,
|
||||
consecutive_errors = consecutive_errors + 1
|
||||
WHERE rule_id = $2`, errMsg, ruleID)
|
||||
return err
|
||||
}
|
||||
Reference in New Issue
Block a user