Demo: a plan for what goes wrong, with problems that come and go #48

Merged
jcoffey-dev merged 1 commits from feature/demo-story into main 2026-11-16 05:44:17 +00:00
Owner

The simulator opened 89 alerts in its first minute and none ever resolved. scripts/fleet/scenario.go now plans it: three stories with a cause and its effects, a few lasting warnings, and problems that come and go on clock-aligned spells, so incidents open and resolve through the day. About 15 problems when it is quiet, 25 when several spells line up (was 89). docs/development.md and docs/demo.md describe it.

Also fixes the backup folder that was named after today while being 40 hours old.

Closes #35, closes #43.

The simulator opened 89 alerts in its first minute and none ever resolved. scripts/fleet/scenario.go now plans it: three stories with a cause and its effects, a few lasting warnings, and problems that come and go on clock-aligned spells, so incidents open and resolve through the day. About 15 problems when it is quiet, 25 when several spells line up (was 89). docs/development.md and docs/demo.md describe it. Also fixes the backup folder that was named after today while being 40 hours old. Closes #35, closes #43.
jcoffey-dev added 1 commit 2026-11-16 05:34:12 +00:00
Demo: a plan for what goes wrong, with problems that come and go
ci / check (pull_request) Successful in 7m15s
ci / postgres (pull_request) Successful in 9m57s
3e5e2d59f9
The simulator made every problem it knew at once and kept it forever:
89 open alerts in the first minute, nothing ever resolved. Now
scripts/fleet/scenario.go holds a plan:

- three stories with a cause and its effects: fa-prod-01 lost a drive;
  a PostgreSQL replica falls behind, HAProxy takes it out and the shop's
  checkout fails; the legacy datacenter drops out of Akamai GTM and its
  CP code fails;
- a few lasting warnings anyone would have;
- problems that come and go (spells counted from the Unix epoch, so a
  restarted demo carries on where the clock is): a Redis unit, an
  exiting container, a flapping check and block listing, ESXi and AHV
  patching, a Proxmox HA error, Pure replication paths, a blade
  evacuating, an IBM i inquiry and an ended subsystem, vault-edge losing
  a server, a late or shrunken backup, SQL Server Agent stopping,
  win-fs01 crashing, an Akamai activation that drags.

Cluster readers (Pure, vSphere, Nutanix, Proxmox, IBM i, the HMC, the
Kubernetes clusters) flap records through fakeCluster.flap and flapSet;
the HTTP stand-ins and host reports ask the spells directly. Gone: the
fan at 0 RPM, the inaccessible VM, the crashed libvirt VM, the always-
crashing pods, the ended and soon-ending Postgres certificates, a dozen
expiring Vault and Akamai certificates, the stuck staging activation.

A backup's folder is now named after the day the copy was made, so a
late one no longer says it's from today (#43).

Closes #35, closes #43.
jcoffey-dev merged commit ceaae4377a into main 2026-11-16 05:44:17 +00:00
jcoffey-dev deleted branch feature/demo-story 2026-11-16 05:44:18 +00:00
Sign in to join this conversation.