The demo had 75k generic records across eight host-0N/service pairs, one dashboard, one alert rule, and -- because nothing ever called AgentControl.CheckIn -- a completely empty Agents page. /hack/demo-simulator replaces the generic data with a fictional but coherent fleet: 14 hosts running nginx, an API tier, workers, Postgres, Redis, mail, Linux journals and Windows event logs, whose messages and attributes look like what those services actually write. It backfills a week (~370k records, ~20s) and then keeps running. Running continuously is the point, not an implementation detail. Three things the demo has to show are only true if data keeps arriving: the Agents page marks a host stale once check-ins stop, alert rules evaluate over trailing windows and would freeze in one state against a static dataset, and any "last 15 minutes" view is empty on data that stopped growing overnight. It also emits metrics/heartbeats and answers CheckIn faithfully enough that the remote-config editor's pending -> applied transition works end to end. Seeded incidents give the data something to find: an api-02 outage with matching slow queries on db-01, 5xx at the edge and cascading job failures; an SSH probe burst; a spam wave; a disk filling up; and one decommissioned host left deliberately stale. /hack/demo-seed holds the rest of the deployment -- the nightly reset, eight dashboards (64 panels, every viz type but line), eleven alert rules across three notification targets, and the systemd unit. Rule thresholds are calibrated against what the simulator actually produces: the first pass had four rules whose thresholds the traffic could never reach and one that fired during normal operation. No line charts: dashboard panels reject the raw-SQL escape hatch, and the pipe language has no time-bucketing, so a real time axis isn't expressible today. Noted in demo-seed/README.md rather than papered over.
113 lines
5.2 KiB
Bash
Executable File
113 lines
5.2 KiB
Bash
Executable File
#!/usr/bin/env bash
|
|
# Nightly reset for the demo.cairnobs.org stack: wipes every data volume
|
|
# and re-seeds from scratch, so the demo's timestamps stay recent, its
|
|
# incidents stay at the same recent offsets, and storage doesn't grow
|
|
# forever. Mirrors the teardown docs/phase-0-runbook.md and
|
|
# docs/phase-2-runbook.md already document (`docker compose down -v`),
|
|
# plus the seed sequence below.
|
|
#
|
|
# Seeding is data-first, config-second: dashboards and alert rules are
|
|
# applied after the backfill exists, so nothing renders an empty panel or
|
|
# evaluates against an empty table on its first pass.
|
|
#
|
|
# The live half of the demo (/hack/demo-simulator running as
|
|
# cairnobs-demo-simulator.service) is stopped for the duration and
|
|
# started again at the end -- it must not be pushing records into a stack
|
|
# that's being torn down, and it must re-register its agents against the
|
|
# fresh, empty `agents` table afterwards.
|
|
set -euo pipefail
|
|
|
|
DEMO_ROOT=${DEMO_ROOT:-/home/john/cairnobs-demo}
|
|
SEED_DIR="$DEMO_ROOT/hack/demo-seed"
|
|
CERTS="$DEMO_ROOT/hack/dev-certs/out"
|
|
BACKFILL=${BACKFILL:-168h}
|
|
RATE_SCALE=${RATE_SCALE:-0.5}
|
|
SIMULATOR_UNIT=cairnobs-demo-simulator.service
|
|
|
|
cd "$DEMO_ROOT"
|
|
|
|
ADMIN_PASSWORD_FILE="$DEMO_ROOT/.admin-password"
|
|
# Changing DEMO_PASSWORD means rebuilding the web image too: the login
|
|
# page prefills this account's credentials, and they're baked into the
|
|
# bundle at build time from VITE_DEMO_USERNAME/VITE_DEMO_PASSWORD in the
|
|
# demo host's docker-compose.override.yml. Change one without the other
|
|
# and the demo's own login form stops working.
|
|
DEMO_PASSWORD='CairnDemo_2026!'
|
|
EVALUATOR_PASSWORD='REDACTED_ROTATED_CREDENTIAL'
|
|
|
|
echo "=== $(date -u +%FT%TZ) reset starting ==="
|
|
|
|
sudo systemctl stop "$SIMULATOR_UNIT" || true
|
|
|
|
docker compose down -v
|
|
docker compose up -d
|
|
echo "waiting for api to report healthy..."
|
|
until [ "$(docker inspect -f '{{.State.Health.Status}}' cairnobs-api 2>/dev/null)" = "healthy" ]; do sleep 2; done
|
|
|
|
# -seed-admin is idempotent and prints the password once -- capture it
|
|
# fresh each run rather than reusing a stale one from a prior reset.
|
|
ADMIN_PASSWORD=$(docker compose run --rm api -seed-admin 2>&1 | grep '^ password:' | awk '{print $2}')
|
|
echo "$ADMIN_PASSWORD" > "$ADMIN_PASSWORD_FILE"
|
|
chmod 600 "$ADMIN_PASSWORD_FILE"
|
|
|
|
export CAIRNOBSCTL_API_URL=http://localhost:8080
|
|
export CAIRNOBSCTL_ALERTING_API_URL=http://localhost:8081
|
|
ADMIN_TOKEN=$(echo "$ADMIN_PASSWORD" | ./bin/cairnobsctl users login admin)
|
|
export CAIRNOBSCTL_TOKEN="$ADMIN_TOKEN"
|
|
|
|
# The password reaches the CLI on stdin only -- `--password <value>` was
|
|
# removed deliberately (see cli/cmd/cairnobsctl/cmd_users.go).
|
|
echo "$DEMO_PASSWORD" | ./bin/cairnobsctl users create demo --role viewer >/dev/null
|
|
echo "$EVALUATOR_PASSWORD" | ./bin/cairnobsctl users create alerting-evaluator --role viewer >/dev/null
|
|
|
|
SVCTOKEN=$(echo "$EVALUATOR_PASSWORD" | ./bin/cairnobsctl users login alerting-evaluator)
|
|
printf 'COMPOSE_PROFILES=single-tenant\nALERTING_SERVICE_TOKEN=%s\n' "$SVCTOKEN" > .env && chmod 600 .env
|
|
docker compose up -d alerting
|
|
|
|
# Three notification targets so the Alerts page shows rules routed to
|
|
# different destinations, the way a real deployment splits ops/security/
|
|
# platform. The URLs are deliberately inert placeholders on a domain
|
|
# reserved for documentation -- nothing is actually notified.
|
|
create_target() {
|
|
curl -s -X POST http://localhost:8081/targets \
|
|
-H "Authorization: Bearer $ADMIN_TOKEN" -H 'Content-Type: application/json' \
|
|
-d "{\"name\":\"$1\",\"kind\":\"webhook\",\"webhook_url\":\"$2\"}" \
|
|
| python3 -c 'import json,sys; print(json.load(sys.stdin)["id"])'
|
|
}
|
|
TARGET_OPS=$(create_target 'Ops on-call (placeholder)' 'https://example.com/webhooks/ops-oncall')
|
|
TARGET_SECURITY=$(create_target 'Security team (placeholder)' 'https://example.com/webhooks/security')
|
|
TARGET_PLATFORM=$(create_target 'Platform team (placeholder)' 'https://example.com/webhooks/platform')
|
|
|
|
echo "backfilling $BACKFILL of synthetic history..."
|
|
./bin/demo-simulator \
|
|
-addr 127.0.0.1:4317 -ca "$CERTS/ca.pem" -cert "$CERTS/client.pem" -key "$CERTS/client-key.pem" \
|
|
-backfill "$BACKFILL" -rate-scale "$RATE_SCALE" -live=false
|
|
|
|
# A handful of Windows-shaped records from the dedicated fixture as well:
|
|
# it's the tool the Windows ingest path is actually verified with, so
|
|
# keeping its output present means the demo and that check agree.
|
|
docker run --rm --network host -v "$DEMO_ROOT":/src -w /src/hack/windows-fixture -e GOCACHE=/tmp/gocache golang:1.25-bookworm \
|
|
go run . --addr 127.0.0.1:4317 --ca /src/hack/dev-certs/out/ca.pem --cert /src/hack/dev-certs/out/client.pem --key /src/hack/dev-certs/out/client-key.pem --count 5
|
|
|
|
echo "applying dashboards..."
|
|
for f in "$SEED_DIR"/dashboards/*.json; do
|
|
./bin/cairnobsctl dashboards apply "$f" >/dev/null
|
|
echo " $(basename "$f")"
|
|
done
|
|
|
|
echo "applying alert rules..."
|
|
TMP=$(mktemp -d)
|
|
trap 'rm -rf "$TMP"' EXIT
|
|
for f in "$SEED_DIR"/alerts/*.json.template; do
|
|
out="$TMP/$(basename "${f%.template}")"
|
|
sed -e "s/__TARGET_OPS__/$TARGET_OPS/" \
|
|
-e "s/__TARGET_SECURITY__/$TARGET_SECURITY/" \
|
|
-e "s/__TARGET_PLATFORM__/$TARGET_PLATFORM/" "$f" > "$out"
|
|
./bin/cairnobsctl alerts apply "$out" >/dev/null
|
|
echo " $(basename "$out")"
|
|
done
|
|
|
|
sudo systemctl start "$SIMULATOR_UNIT"
|
|
|
|
echo "=== $(date -u +%FT%TZ) reset complete ==="
|