Files
cairnobs/hack/demo-seed/reset-demo.sh
T
jcoffey-dev bcb9a01cd6 Give the demo a live synthetic fleet, dashboards, and alert rules
The demo had 75k generic records across eight host-0N/service pairs, one
dashboard, one alert rule, and -- because nothing ever called
AgentControl.CheckIn -- a completely empty Agents page.

/hack/demo-simulator replaces the generic data with a fictional but
coherent fleet: 14 hosts running nginx, an API tier, workers, Postgres,
Redis, mail, Linux journals and Windows event logs, whose messages and
attributes look like what those services actually write. It backfills a
week (~370k records, ~20s) and then keeps running.

Running continuously is the point, not an implementation detail. Three
things the demo has to show are only true if data keeps arriving: the
Agents page marks a host stale once check-ins stop, alert rules evaluate
over trailing windows and would freeze in one state against a static
dataset, and any "last 15 minutes" view is empty on data that stopped
growing overnight. It also emits metrics/heartbeats and answers CheckIn
faithfully enough that the remote-config editor's pending -> applied
transition works end to end.

Seeded incidents give the data something to find: an api-02 outage with
matching slow queries on db-01, 5xx at the edge and cascading job
failures; an SSH probe burst; a spam wave; a disk filling up; and one
decommissioned host left deliberately stale.

/hack/demo-seed holds the rest of the deployment -- the nightly reset,
eight dashboards (64 panels, every viz type but line), eleven alert
rules across three notification targets, and the systemd unit. Rule
thresholds are calibrated against what the simulator actually produces:
the first pass had four rules whose thresholds the traffic could never
reach and one that fired during normal operation.

No line charts: dashboard panels reject the raw-SQL escape hatch, and
the pipe language has no time-bucketing, so a real time axis isn't
expressible today. Noted in demo-seed/README.md rather than papered
over.
2026-08-22 16:12:35 -07:00

113 lines
5.2 KiB
Bash
Executable File

#!/usr/bin/env bash
# Nightly reset for the demo.cairnobs.org stack: wipes every data volume
# and re-seeds from scratch, so the demo's timestamps stay recent, its
# incidents stay at the same recent offsets, and storage doesn't grow
# forever. Mirrors the teardown docs/phase-0-runbook.md and
# docs/phase-2-runbook.md already document (`docker compose down -v`),
# plus the seed sequence below.
#
# Seeding is data-first, config-second: dashboards and alert rules are
# applied after the backfill exists, so nothing renders an empty panel or
# evaluates against an empty table on its first pass.
#
# The live half of the demo (/hack/demo-simulator running as
# cairnobs-demo-simulator.service) is stopped for the duration and
# started again at the end -- it must not be pushing records into a stack
# that's being torn down, and it must re-register its agents against the
# fresh, empty `agents` table afterwards.
set -euo pipefail
DEMO_ROOT=${DEMO_ROOT:-/home/john/cairnobs-demo}
SEED_DIR="$DEMO_ROOT/hack/demo-seed"
CERTS="$DEMO_ROOT/hack/dev-certs/out"
BACKFILL=${BACKFILL:-168h}
RATE_SCALE=${RATE_SCALE:-0.5}
SIMULATOR_UNIT=cairnobs-demo-simulator.service
cd "$DEMO_ROOT"
ADMIN_PASSWORD_FILE="$DEMO_ROOT/.admin-password"
# Changing DEMO_PASSWORD means rebuilding the web image too: the login
# page prefills this account's credentials, and they're baked into the
# bundle at build time from VITE_DEMO_USERNAME/VITE_DEMO_PASSWORD in the
# demo host's docker-compose.override.yml. Change one without the other
# and the demo's own login form stops working.
DEMO_PASSWORD='CairnDemo_2026!'
EVALUATOR_PASSWORD='REDACTED_ROTATED_CREDENTIAL'
echo "=== $(date -u +%FT%TZ) reset starting ==="
sudo systemctl stop "$SIMULATOR_UNIT" || true
docker compose down -v
docker compose up -d
echo "waiting for api to report healthy..."
until [ "$(docker inspect -f '{{.State.Health.Status}}' cairnobs-api 2>/dev/null)" = "healthy" ]; do sleep 2; done
# -seed-admin is idempotent and prints the password once -- capture it
# fresh each run rather than reusing a stale one from a prior reset.
ADMIN_PASSWORD=$(docker compose run --rm api -seed-admin 2>&1 | grep '^ password:' | awk '{print $2}')
echo "$ADMIN_PASSWORD" > "$ADMIN_PASSWORD_FILE"
chmod 600 "$ADMIN_PASSWORD_FILE"
export CAIRNOBSCTL_API_URL=http://localhost:8080
export CAIRNOBSCTL_ALERTING_API_URL=http://localhost:8081
ADMIN_TOKEN=$(echo "$ADMIN_PASSWORD" | ./bin/cairnobsctl users login admin)
export CAIRNOBSCTL_TOKEN="$ADMIN_TOKEN"
# The password reaches the CLI on stdin only -- `--password <value>` was
# removed deliberately (see cli/cmd/cairnobsctl/cmd_users.go).
echo "$DEMO_PASSWORD" | ./bin/cairnobsctl users create demo --role viewer >/dev/null
echo "$EVALUATOR_PASSWORD" | ./bin/cairnobsctl users create alerting-evaluator --role viewer >/dev/null
SVCTOKEN=$(echo "$EVALUATOR_PASSWORD" | ./bin/cairnobsctl users login alerting-evaluator)
printf 'COMPOSE_PROFILES=single-tenant\nALERTING_SERVICE_TOKEN=%s\n' "$SVCTOKEN" > .env && chmod 600 .env
docker compose up -d alerting
# Three notification targets so the Alerts page shows rules routed to
# different destinations, the way a real deployment splits ops/security/
# platform. The URLs are deliberately inert placeholders on a domain
# reserved for documentation -- nothing is actually notified.
create_target() {
curl -s -X POST http://localhost:8081/targets \
-H "Authorization: Bearer $ADMIN_TOKEN" -H 'Content-Type: application/json' \
-d "{\"name\":\"$1\",\"kind\":\"webhook\",\"webhook_url\":\"$2\"}" \
| python3 -c 'import json,sys; print(json.load(sys.stdin)["id"])'
}
TARGET_OPS=$(create_target 'Ops on-call (placeholder)' 'https://example.com/webhooks/ops-oncall')
TARGET_SECURITY=$(create_target 'Security team (placeholder)' 'https://example.com/webhooks/security')
TARGET_PLATFORM=$(create_target 'Platform team (placeholder)' 'https://example.com/webhooks/platform')
echo "backfilling $BACKFILL of synthetic history..."
./bin/demo-simulator \
-addr 127.0.0.1:4317 -ca "$CERTS/ca.pem" -cert "$CERTS/client.pem" -key "$CERTS/client-key.pem" \
-backfill "$BACKFILL" -rate-scale "$RATE_SCALE" -live=false
# A handful of Windows-shaped records from the dedicated fixture as well:
# it's the tool the Windows ingest path is actually verified with, so
# keeping its output present means the demo and that check agree.
docker run --rm --network host -v "$DEMO_ROOT":/src -w /src/hack/windows-fixture -e GOCACHE=/tmp/gocache golang:1.25-bookworm \
go run . --addr 127.0.0.1:4317 --ca /src/hack/dev-certs/out/ca.pem --cert /src/hack/dev-certs/out/client.pem --key /src/hack/dev-certs/out/client-key.pem --count 5
echo "applying dashboards..."
for f in "$SEED_DIR"/dashboards/*.json; do
./bin/cairnobsctl dashboards apply "$f" >/dev/null
echo " $(basename "$f")"
done
echo "applying alert rules..."
TMP=$(mktemp -d)
trap 'rm -rf "$TMP"' EXIT
for f in "$SEED_DIR"/alerts/*.json.template; do
out="$TMP/$(basename "${f%.template}")"
sed -e "s/__TARGET_OPS__/$TARGET_OPS/" \
-e "s/__TARGET_SECURITY__/$TARGET_SECURITY/" \
-e "s/__TARGET_PLATFORM__/$TARGET_PLATFORM/" "$f" > "$out"
./bin/cairnobsctl alerts apply "$out" >/dev/null
echo " $(basename "$out")"
done
sudo systemctl start "$SIMULATOR_UNIT"
echo "=== $(date -u +%FT%TZ) reset complete ==="