Two faults, both found by running the reset against the demo rather than
by reading it, and both fixed on the box before this commit existed.
The reset raced its own alerting container. Writing .env changes
alerting's environment, so `docker compose up -d alerting` recreates it
-- and the next line posted notification targets to it with no wait. On a
busy box that lost: curl returned nothing, json.load threw on an empty
string, and set -e killed the script. The damage is in the ordering:
`docker compose down -v` runs near the top, so any failure after it
leaves the public demo up, empty, and with the simulator still stopped,
because the unit is only restarted on the last line. It has been
surviving nightly on timing alone. It now polls /healthz for up to 60
seconds and fails loudly before the seed rather than after the wipe.
Five dashboard panels were raw ClickHouse SQL, which dashboards refuse:
validatePanel rejects query_language "sql" outright, because the
time-range picker is injected as leading query terms and a SELECT has
nowhere to put them. They were written that way because the pipe syntax
has no time bucketing -- no timechart, no bin -- so a genuine time series
is not available to a dashboard panel at all. Each is now the breakdown
the panel was actually asking for: kubelet events by kind and host, DNS
by type and result, queue depth by queue and host, IIS by site and host,
HAProxy by backend and balancer.
Both mistakes were mine and both were avoidable by reading: the rule
about pipe-syntax-only dashboards is stated in terraform/README.md, and I
had already read the line that says it.
Verified on the demo: 50 hosts, 24 services, 1,922,128 records, all 13
dashboards and all 11 alert rules applied, simulator active. `system`
appears on exactly 31 hosts, which is the Linux count -- no Windows host
was given a journald stream.
Twelve hosts running six services read as somebody's side project. Fifty
hosts across twenty-one services read as an estate, which is what a
visitor is trying to see themselves in.
Thirty-one Linux, eighteen Windows, one Linux host whose agent is gone.
The proportions are the point: Windows now carries Active Directory, IIS,
SQL Server, Exchange, file shares, Remote Desktop, print, WSUS and SCCM
rather than appearing as a Security channel on one box. Linux gains a
load-balancer tier, an outbound proxy, MySQL beside Postgres, RabbitMQ,
Elasticsearch, three Kubernetes nodes, CI, Vault, OpenLDAP, BIND and a
backup server.
Fifteen new generators, each writing what the real daemon writes --
HAProxy's timing quintuple, MySQL slow-query blocks, W3C extended format
for IIS, kubelet PLEG lines, BIND query logging with NXDOMAIN, Squid's
TCP_DENIED, SQL Server deadlock and I/O-stall messages -- with the
structured fields carried in attributes so both halves of the query
language have something to work on.
Three things found while doing it, each of which would have shipped as a
quiet wrongness:
linuxHosts() decided Windows by `service == "eventlog"`. That held while
eventlog was the only Windows role; with IIS and SQL Server on Windows it
would have given every one of them a journald system stream -- sshd and
UFW lines on a Windows box. It now decides from `os`.
worker-02 lost its filling disk when the fleet was rewritten, which is
the story worker-disk-filling thresholds on. Restored at its original
rate, with a comment saying why it cannot move.
Five dashboard panels were written as `timechart`, which this query
language does not have -- its stages are where/stats/sort/fields/head/
tail. They are raw ClickHouse SQL now, which the reference recommends for
exactly this, and a `pie` panel was dropped before it shipped because the
web's VizType union has no such member even though the API accepts one.
Five new dashboards: platform/Kubernetes, directory and DNS, messaging
and search, the Windows server estate, and edge/proxy. Every panel was
checked against the language's real stage list and the web's real viz
types.
Volume roughly quadruples: about 316 records/minute at rate-scale 1, and
1.9M per nightly reset at the demo's own settings against about 0.5M
before. ClickHouse will not notice; reset time and disk on the demo box
might, so the README says so and names RATE_SCALE as the lever.
jcoffey-dev