Give the demo a live synthetic fleet, dashboards, and alert rules

The demo had 75k generic records across eight host-0N/service pairs, one dashboard, one alert rule, and -- because nothing ever called AgentControl.CheckIn -- a completely empty Agents page. /hack/demo-simulator replaces the generic data with a fictional but coherent fleet: 14 hosts running nginx, an API tier, workers, Postgres, Redis, mail, Linux journals and Windows event logs, whose messages and attributes look like what those services actually write. It backfills a week (~370k records, ~20s) and then keeps running. Running continuously is the point, not an implementation detail. Three things the demo has to show are only true if data keeps arriving: the Agents page marks a host stale once check-ins stop, alert rules evaluate over trailing windows and would freeze in one state against a static dataset, and any "last 15 minutes" view is empty on data that stopped growing overnight. It also emits metrics/heartbeats and answers CheckIn faithfully enough that the remote-config editor's pending -> applied transition works end to end. Seeded incidents give the data something to find: an api-02 outage with matching slow queries on db-01, 5xx at the edge and cascading job failures; an SSH probe burst; a spam wave; a disk filling up; and one decommissioned host left deliberately stale. /hack/demo-seed holds the rest of the deployment -- the nightly reset, eight dashboards (64 panels, every viz type but line), eleven alert rules across three notification targets, and the systemd unit. Rule thresholds are calibrated against what the simulator actually produces: the first pass had four rules whose thresholds the traffic could never reach and one that fired during normal operation. No line charts: dashboard panels reject the raw-SQL escape hatch, and the pipe language has no time-bucketing, so a real time axis isn't expressible today. Noted in demo-seed/README.md rather than papered over.
2026-08-22 16:12:35 -07:00
parent e6a58f58ea
commit bcb9a01cd6
31 changed files with 3031 additions and 0 deletions
@@ -0,0 +1,11 @@
+{
+  "name": "legacy-01 agent unavailable",
+  "description": "No heartbeat from legacy-01 within its heartbeat window -- the agent or the host is gone",
+  "query": "earliest=-5m host=\"legacy-01\" cairnobs.heartbeat=true",
+  "query_language": "spl",
+  "condition_type": "absence",
+  "eval_interval_seconds": 60,
+  "for_minutes": 0,
+  "notification_target_id": "__TARGET_PLATFORM__",
+  "enabled": true
+}
@@ -0,0 +1,13 @@
+{
+  "name": "High API 5xx rate",
+  "description": "The API tier is returning server errors well above its normal background rate",
+  "query": "service=api earliest=-5m | where status>=500 | stats count",
+  "query_language": "spl",
+  "condition_type": "threshold",
+  "eval_interval_seconds": 60,
+  "for_minutes": 1,
+  "notification_target_id": "__TARGET_OPS__",
+  "enabled": true,
+  "comparator": "gt",
+  "threshold_value": 4
+}
@@ -0,0 +1,13 @@
+{
+  "name": "API latency degraded",
+  "description": "Average API response time over the last 10 minutes is above the service objective",
+  "query": "service=api earliest=-10m | stats avg(latency_ms) as avg_latency_ms",
+  "query_language": "spl",
+  "condition_type": "threshold",
+  "eval_interval_seconds": 60,
+  "for_minutes": 2,
+  "notification_target_id": "__TARGET_OPS__",
+  "enabled": true,
+  "comparator": "gt",
+  "threshold_value": 130
+}
@@ -0,0 +1,13 @@
+{
+  "name": "Edge 5xx surge",
+  "description": "nginx is serving 5xx to clients -- either upstreams are failing or the edge itself is",
+  "query": "service=nginx earliest=-5m | where status>=500 | stats count",
+  "query_language": "spl",
+  "condition_type": "threshold",
+  "eval_interval_seconds": 60,
+  "for_minutes": 1,
+  "notification_target_id": "__TARGET_OPS__",
+  "enabled": true,
+  "comparator": "gt",
+  "threshold_value": 4
+}
@@ -0,0 +1,13 @@
+{
+  "name": "Firewall block surge",
+  "description": "UFW is dropping far more inbound connections than usual -- typically a scan in progress",
+  "query": "service=system ufw_action=BLOCK earliest=-15m | stats count",
+  "query_language": "spl",
+  "condition_type": "threshold",
+  "eval_interval_seconds": 120,
+  "for_minutes": 0,
+  "notification_target_id": "__TARGET_SECURITY__",
+  "enabled": true,
+  "comparator": "gt",
+  "threshold_value": 20
+}
@@ -0,0 +1,13 @@
+{
+  "name": "Mail authentication failures",
+  "description": "Repeated SMTP auth failures on the mail host -- credential stuffing against the mail server",
+  "query": "service=smtp result=auth_failed earliest=-15m | stats count",
+  "query_language": "spl",
+  "condition_type": "threshold",
+  "eval_interval_seconds": 120,
+  "for_minutes": 0,
+  "notification_target_id": "__TARGET_SECURITY__",
+  "enabled": true,
+  "comparator": "gt",
+  "threshold_value": 6
+}
@@ -0,0 +1,13 @@
+{
+  "name": "Slow database queries",
+  "description": "Statements taking over a second are piling up, which usually shows up as API latency next",
+  "query": "service=postgres earliest=-10m | where duration_ms>1000 | stats count",
+  "query_language": "spl",
+  "condition_type": "threshold",
+  "eval_interval_seconds": 60,
+  "for_minutes": 2,
+  "notification_target_id": "__TARGET_PLATFORM__",
+  "enabled": true,
+  "comparator": "gt",
+  "threshold_value": 5
+}
@@ -0,0 +1,13 @@
+{
+  "name": "SSH brute-force attempt",
+  "description": "A burst of failed SSH authentications across the fleet, well past normal background probing",
+  "query": "service=system auth_result=failed earliest=-10m | stats count",
+  "query_language": "spl",
+  "condition_type": "threshold",
+  "eval_interval_seconds": 60,
+  "for_minutes": 0,
+  "notification_target_id": "__TARGET_SECURITY__",
+  "enabled": true,
+  "comparator": "gt",
+  "threshold_value": 12
+}
@@ -0,0 +1,13 @@
+{
+  "name": "Windows account lockout",
+  "description": "Any 4740 on the Windows hosts: an account was locked out after repeated failures",
+  "query": "winevt.event_id=4740 earliest=-15m | stats count",
+  "query_language": "spl",
+  "condition_type": "threshold",
+  "eval_interval_seconds": 120,
+  "for_minutes": 0,
+  "notification_target_id": "__TARGET_SECURITY__",
+  "enabled": true,
+  "comparator": "gte",
+  "threshold_value": 1
+}
@@ -0,0 +1,13 @@
+{
+  "name": "worker-02 disk nearly full",
+  "description": "worker-02's data volume has passed 175 GiB of 200 GiB and is still climbing",
+  "query": "host=\"worker-02\" cairnobs.metrics=true earliest=-15m | stats max(disk_used_bytes) as disk_used_bytes",
+  "query_language": "spl",
+  "condition_type": "threshold",
+  "eval_interval_seconds": 300,
+  "for_minutes": 0,
+  "notification_target_id": "__TARGET_PLATFORM__",
+  "enabled": true,
+  "comparator": "gt",
+  "threshold_value": 187904819200
+}
@@ -0,0 +1,13 @@
+{
+  "name": "Background job failures",
+  "description": "Worker jobs are failing after their retries are exhausted",
+  "query": "service=worker result=failed earliest=-15m | stats count",
+  "query_language": "spl",
+  "condition_type": "threshold",
+  "eval_interval_seconds": 120,
+  "for_minutes": 0,
+  "notification_target_id": "__TARGET_PLATFORM__",
+  "enabled": true,
+  "comparator": "gt",
+  "threshold_value": 12
+}