Files
cairnobs/docs/development.md
T
jcoffey-dev e2c1b27cb3
ci / postgres (pull_request) Canceled after 0s
ci / check (pull_request) Successful in 9m26s
Demo: start with 30 days behind it
Every graph on the demo said "none kept yet" and the reports had
nothing to report: the container starts fresh each week. Now fleet run
-history <database> -state-key <file> first writes the last 30 days into
obs-beacond's database, from the same plan and clock the live demo runs
on:

- hosts' hours: what a host uses is now a function of its name and the
  clock (load.go): a working day in its site's time zone, quieter
  weekends, memory that creeps on some hosts until a weekly restart, and
  disks at a level of their own that fill slowly until a clean-up
  (fleet-0023's log volume warns now and then). No more every disk at
  33%, part of #40;
- the clusters' workloads, the arrays' I/O and space, the IBM i
  partitions' CPU, the Vaults' leases (on a daily curve now), the mail
  rounds;
- hosts', checks' and services' statuses, for the reports' availability;
- an incident each time a past problem came and went (pastProblems in
  scenario.go, with obs-beacond's keys and titles), notes on the stories
  saying what the cause was, and the incident numbers carrying on.

About 45 seconds for the full demo. state gains PutHostHours,
PutMailRounds and LogStatuses, to write many in one transaction.
docs/demo.md and docs/development.md describe it.

Closes #39.
2026-11-15 22:58:03 -07:00

4.9 KiB

Working on Cairn OBS

You need Go 1.26+, Node 22+ and, to see containers, Docker with your user in the docker group.

make            # build the web UI, then bin/obs-beacond and bin/obs-beacon-agent
make check      # gofmt, go vet, go test, and the UI's type check
make dev        # build, then run obs-beacond + helper + agent against this machine

make dev (dev/run.sh) keeps its state in dev/state/ and listens on 127.0.0.1 only. Open http://localhost:8150/ and either:

  • Sign in with inbuxa (fake, local): the real sign-in flow against a stand-in inbuxa on 127.0.0.1:8151 (dev/fakeinbuxa). Pick the System Administrator to get in, or the tenant admin to see a refusal.
  • Sign in for local development: a shortcut. obs-beacond only offers it when its config says dev_signin: true, and refuses to start that way on anything but a loopback address.

Sessions, incidents and the audit log are kept in dev/state/beacon.db, so restarting obs-beacond keeps you signed in. Delete that file to start over.

Alerts in make dev go to a log channel (they print in the terminal) after a 20-second hold. The dev expected-containers file names one container that doesn't exist, beacon-demo-sleeper, so an incident opens on its own; start it to watch the incident resolve:

docker run -d --rm --name beacon-demo-sleeper alpine:3 sleep 600

Working on the UI with hot reload: keep make dev running and, in web/, run npm run dev; it serves on http://localhost:5180/ and passes /api and /auth to the obs-beacond on 8150.

The web UI is built into internal/webui/dist/ and embedded in obs-beacond, so after changing web/ run make again before testing the binary.

The demo fleet

For a UI with an enterprise's worth of everything on it (screenshots, the docs, working on pages at scale), scripts/fleet fakes the agents, the clusters and the outside services they'd watch. Build it with go build -o bin/fleet ./scripts/fleet, then:

fleet config -n 120 -out demo -demo

writes demo/agents.yaml and demo/tokens, and with -demo the checks, expected units, backups and data layer for three sites of 50 hosts: demo/demo.yaml has the lines to add to obs-beacond.yaml. Then run

fleet run -server http://127.0.0.1:8150 -tokens demo/tokens -demo \
  -cloudflare demo/cf -mail demo/mail -channels demo/channels

with the same number of agents. Each of -cloudflare, -mail, -channels, -akamai and -vault serves a stand-in on loopback and writes the section for obs-beacond.yaml into its directory (cloudflare.yaml, mail.yaml, channels.yaml, …); add those once, and start fleet before obs-beacond so the first readings find them. -channels catches what the alert channels send (ntfy, Discord, PagerDuty, a webhook and email) and lists it on http://127.0.0.1:58450/. fleet run -h lists the rest: clusters, VMs, Proxmox, vSphere, Nutanix and Pure arrays.

Some things are wrong on purpose, so every page has something to show, and most of the fleet is fine, as a real one is. scripts/fleet/scenario.go has the plan: three stories with a cause and its effects (a FlashArray lost a drive; a PostgreSQL replica falls behind and the shop's checkout fails; the legacy datacenter drops out of Akamai GTM), a handful of lasting warnings (a failover nobody moved back, a weak key, an old VM snapshot, a zone on Flexible SSL, and so on), and problems that come and go: a unit that fails and is restarted, a container that exits, a check that flaps, a late backup, a host being patched. Each of those is a spell: on for a while once every so often, counted from the Unix epoch, so every part of the simulator agrees on what's happening now and a restarted demo carries on where the clock is. Incidents open and resolve through the day, and a dozen or two are open at any moment. The mail server is healthy, to show what normal looks like.

What a host uses is a function of its name and the clock (scripts/fleet/load.go), not a dice roll, so -history <database> -state-key <file> can write the last 30 days (-history-days) before anything reports (history.go): hosts' and workloads' hours, mail rounds, statuses, and an incident for each past spell of the problems in pastProblems (scenario.go), whose keys and titles are obs-beacond's own. When you change what a spell's problem says in obs-beacond, change its line there too.

Dependencies and notices

THIRD_PARTY_NOTICES.md lists every third-party component inside the binaries with its license text, and the packages install it. make check fails when it's out of date: after adding or bumping a Go module, or a package the web bundle uses, run go run ./scripts/notices and commit the result. It stops on a license it can't name, so a new one gets a person's look before it ships. The npm packages in the bundle are listed in scripts/notices/main.go (only svelte and @sveltejs/kit today; the rest are build tools).