Give the demo a live synthetic fleet, dashboards, and alert rules

The demo had 75k generic records across eight host-0N/service pairs, one
dashboard, one alert rule, and -- because nothing ever called
AgentControl.CheckIn -- a completely empty Agents page.

/hack/demo-simulator replaces the generic data with a fictional but
coherent fleet: 14 hosts running nginx, an API tier, workers, Postgres,
Redis, mail, Linux journals and Windows event logs, whose messages and
attributes look like what those services actually write. It backfills a
week (~370k records, ~20s) and then keeps running.

Running continuously is the point, not an implementation detail. Three
things the demo has to show are only true if data keeps arriving: the
Agents page marks a host stale once check-ins stop, alert rules evaluate
over trailing windows and would freeze in one state against a static
dataset, and any "last 15 minutes" view is empty on data that stopped
growing overnight. It also emits metrics/heartbeats and answers CheckIn
faithfully enough that the remote-config editor's pending -> applied
transition works end to end.

Seeded incidents give the data something to find: an api-02 outage with
matching slow queries on db-01, 5xx at the edge and cascading job
failures; an SSH probe burst; a spam wave; a disk filling up; and one
decommissioned host left deliberately stale.

/hack/demo-seed holds the rest of the deployment -- the nightly reset,
eight dashboards (64 panels, every viz type but line), eleven alert
rules across three notification targets, and the systemd unit. Rule
thresholds are calibrated against what the simulator actually produces:
the first pass had four rules whose thresholds the traffic could never
reach and one that fired during normal operation.

No line charts: dashboard panels reject the raw-SQL escape hatch, and
the pipe language has no time-bucketing, so a real time axis isn't
expressible today. Noted in demo-seed/README.md rather than papered
over.
This commit is contained in:
2026-08-22 16:12:35 -07:00
parent e6a58f58ea
commit bcb9a01cd6
31 changed files with 3031 additions and 0 deletions
@@ -0,0 +1,113 @@
{
"name": "Errors & reliability",
"description": "Where failures are concentrated right now -- 5xx, fatal events, and failed background jobs",
"default_earliest": "-24h",
"default_latest": "now",
"panels": [
{
"title": "Error events",
"query": "severity=ERROR | stats count",
"viz_type": "single_stat",
"position_x": 0,
"position_y": 0,
"width": 4,
"height": 3,
"query_language": "spl",
"sort_order": 0
},
{
"title": "Fatal events",
"query": "severity=FATAL | stats count",
"viz_type": "single_stat",
"position_x": 4,
"position_y": 0,
"width": 4,
"height": 3,
"query_language": "spl",
"sort_order": 1
},
{
"title": "Failed jobs",
"query": "service=worker result=failed | stats count",
"viz_type": "single_stat",
"position_x": 8,
"position_y": 0,
"width": 4,
"height": 3,
"query_language": "spl",
"sort_order": 2
},
{
"title": "Errors by service",
"query": "severity=ERROR | stats count by service | sort -count",
"viz_type": "bar",
"position_x": 0,
"position_y": 3,
"width": 6,
"height": 5,
"viz_config": {
"x_column": "service",
"value_column": "count"
},
"query_language": "spl",
"sort_order": 3
},
{
"title": "Top API error codes",
"query": "service=api | where status>=500 | stats count by error_code | sort -count | head 10",
"viz_type": "top_n",
"position_x": 6,
"position_y": 3,
"width": 6,
"height": 5,
"viz_config": {
"label_column": "error_code",
"value_column": "count"
},
"query_language": "spl",
"sort_order": 4
},
{
"title": "5xx by host and status",
"query": "status>=500 | stats count by host, status",
"viz_type": "heatmap",
"position_x": 0,
"position_y": 8,
"width": 6,
"height": 5,
"viz_config": {
"x_column": "host",
"y_column": "status",
"value_column": "count"
},
"query_language": "spl",
"sort_order": 5
},
{
"title": "Failed jobs by queue",
"query": "service=worker result=failed | stats count by queue | sort -count",
"viz_type": "bar",
"position_x": 6,
"position_y": 8,
"width": 6,
"height": 5,
"viz_config": {
"x_column": "queue",
"value_column": "count"
},
"query_language": "spl",
"sort_order": 6
},
{
"title": "Recent 5xx responses",
"query": "status>=500 | sort -timestamp | head 50 | fields timestamp, host, service, status, route, message",
"viz_type": "table",
"position_x": 0,
"position_y": 13,
"width": 12,
"height": 6,
"query_language": "spl",
"sort_order": 7
}
]
}