Grow the demo fleet to fifty hosts and a real Windows tier

Twelve hosts running six services read as somebody's side project. Fifty
hosts across twenty-one services read as an estate, which is what a
visitor is trying to see themselves in.

Thirty-one Linux, eighteen Windows, one Linux host whose agent is gone.
The proportions are the point: Windows now carries Active Directory, IIS,
SQL Server, Exchange, file shares, Remote Desktop, print, WSUS and SCCM
rather than appearing as a Security channel on one box. Linux gains a
load-balancer tier, an outbound proxy, MySQL beside Postgres, RabbitMQ,
Elasticsearch, three Kubernetes nodes, CI, Vault, OpenLDAP, BIND and a
backup server.

Fifteen new generators, each writing what the real daemon writes --
HAProxy's timing quintuple, MySQL slow-query blocks, W3C extended format
for IIS, kubelet PLEG lines, BIND query logging with NXDOMAIN, Squid's
TCP_DENIED, SQL Server deadlock and I/O-stall messages -- with the
structured fields carried in attributes so both halves of the query
language have something to work on.

Three things found while doing it, each of which would have shipped as a
quiet wrongness:

linuxHosts() decided Windows by `service == "eventlog"`. That held while
eventlog was the only Windows role; with IIS and SQL Server on Windows it
would have given every one of them a journald system stream -- sshd and
UFW lines on a Windows box. It now decides from `os`.

worker-02 lost its filling disk when the fleet was rewritten, which is
the story worker-disk-filling thresholds on. Restored at its original
rate, with a comment saying why it cannot move.

Five dashboard panels were written as `timechart`, which this query
language does not have -- its stages are where/stats/sort/fields/head/
tail. They are raw ClickHouse SQL now, which the reference recommends for
exactly this, and a `pie` panel was dropped before it shipped because the
web's VizType union has no such member even though the API accepts one.

Five new dashboards: platform/Kubernetes, directory and DNS, messaging
and search, the Windows server estate, and edge/proxy. Every panel was
checked against the language's real stage list and the web's real viz
types.

Volume roughly quadruples: about 316 records/minute at rate-scale 1, and
1.9M per nightly reset at the demo's own settings against about 0.5M
before. ClickHouse will not notice; reset time and disk on the demo box
might, so the README says so and names RATE_SCALE as the lever.
This commit is contained in:
2026-09-04 15:40:08 -07:00
parent 5742901fc5
commit e10289ee36
8 changed files with 1507 additions and 83 deletions
+34
View File
@@ -23,6 +23,40 @@ other code.
the demo, `/hack/demo-simulator`. Installed at `/etc/systemd/system/`
on the demo box.
## The fleet
Fifty hosts, shaped like an estate rather than a stack: thirty-one Linux,
eighteen Windows, and one Linux host whose agent is gone so the Agents
page has something stale to show.
| Tier | Hosts |
|---|---|
| Edge and proxy | `lb-01/02` (HAProxy), `edge-01/02` (nginx), `proxy-01` (Squid) |
| Application | `api-01``04`, `worker-01``03`, `arm-build-01` (aarch64) |
| Data | `db-01/02` (Postgres), `mysql-01`, `cache-01/02` (Redis), `mq-01/02` (RabbitMQ), `search-01/02` (Elasticsearch) |
| Platform | `k8s-node-01``03` (kubelet), `ci-01` (Jenkins), `vault-01`, `ldap-01` (OpenLDAP), `dns-01` (BIND), `backup-01`, `mail-01` |
| Windows | `DC-01/02`, `IIS-01``03`, `WIN-SQL-01/02`, `EXCH-01/02`, `FS-01/02`, `RDS-01/02`, `WIN-APP-01/02`, `PRINT-01`, `WSUS-01`, `SCCM-01` |
The Windows share is the point of the proportions. An enterprise looking
at this should recognise its own estate, which means Windows carrying
real services -- Active Directory, IIS, SQL Server, Exchange, file
shares, Remote Desktop, print, WSUS and SCCM -- rather than appearing
only as a Security channel on one box.
Two hosts carry stories the alert rules fire on and must not be moved:
`worker-02`'s disk fills at 0.04 of the volume per day, which is what
`worker-disk-filling` thresholds against, and `legacy-01` checks in once
and goes quiet, which is what `agent-legacy-01-unavailable` catches.
`api-02` is the host the outage window hits.
**Volume.** Fifty hosts generate about 316 records/minute at
`-rate-scale 1`, and the nightly reset runs at `RATE_SCALE=0.5` over a
168-hour backfill -- roughly **1.9M records per reset**, against about
0.5M when the fleet was twelve hosts. ClickHouse is untroubled by that;
what it costs is reset time and disk on the demo box. `RATE_SCALE` is the
lever if either becomes a problem, and lowering it keeps every host and
service present rather than dropping any of them.
## Prefilled login
The demo's login page comes up with the read-only `demo` account already
@@ -0,0 +1,109 @@
{
"name": "Directory and DNS",
"description": "LDAP binds, DNS resolution and the domain controllers underneath both",
"default_earliest": "-24h",
"default_latest": "now",
"panels": [
{
"title": "DNS queries",
"query": "service=bind | stats count",
"viz_type": "single_stat",
"position_x": 0,
"position_y": 0,
"width": 3,
"height": 3,
"query_language": "spl",
"sort_order": 0
},
{
"title": "NXDOMAIN",
"query": "service=bind dns_rcode=NXDOMAIN | stats count",
"viz_type": "single_stat",
"position_x": 3,
"position_y": 0,
"width": 3,
"height": 3,
"query_language": "spl",
"sort_order": 1
},
{
"title": "LDAP binds",
"query": "service=openldap event_kind=bind | stats count",
"viz_type": "single_stat",
"position_x": 6,
"position_y": 0,
"width": 3,
"height": 3,
"query_language": "spl",
"sort_order": 2
},
{
"title": "Failed binds",
"query": "service=openldap event_kind=bind_failed | stats count",
"viz_type": "single_stat",
"position_x": 9,
"position_y": 0,
"width": 3,
"height": 3,
"query_language": "spl",
"sort_order": 3
},
{
"title": "DNS query volume by type",
"query": "SELECT toStartOfInterval(timestamp, INTERVAL 30 MINUTE) AS bucket, attributes['dns_type'] AS dns_type, count(*) AS count FROM logs WHERE service = 'bind' GROUP BY bucket, dns_type ORDER BY bucket",
"viz_type": "line",
"position_x": 0,
"position_y": 3,
"width": 12,
"height": 5,
"viz_config": {
"x_column": "bucket",
"value_column": "count",
"series_column": "dns_type"
},
"query_language": "sql",
"sort_order": 4
},
{
"title": "Most queried names",
"query": "service=bind | stats count by dns_name | sort -count | head 10",
"viz_type": "top_n",
"position_x": 0,
"position_y": 8,
"width": 6,
"height": 5,
"viz_config": {
"label_column": "dns_name",
"value_column": "count"
},
"query_language": "spl",
"sort_order": 5
},
{
"title": "Bind DNs by activity",
"query": "service=openldap | stats count by bind_dn | sort -count",
"viz_type": "bar",
"position_x": 6,
"position_y": 8,
"width": 6,
"height": 5,
"viz_config": {
"x_column": "bind_dn",
"value_column": "count"
},
"query_language": "spl",
"sort_order": 6
},
{
"title": "Failed LDAP binds",
"query": "service=openldap event_kind=bind_failed | sort -timestamp | head 25 | fields timestamp, host, bind_dn, remote_addr, message",
"viz_type": "table",
"position_x": 0,
"position_y": 13,
"width": 12,
"height": 5,
"query_language": "spl",
"sort_order": 7
}
]
}
+114
View File
@@ -0,0 +1,114 @@
{
"name": "Edge and proxy",
"description": "Load balancers, reverse proxies and the outbound web proxy -- what reaches the estate and what leaves it",
"default_earliest": "-24h",
"default_latest": "now",
"panels": [
{
"title": "Balanced requests",
"query": "service=haproxy | stats count",
"viz_type": "single_stat",
"position_x": 0,
"position_y": 0,
"width": 3,
"height": 3,
"query_language": "spl",
"sort_order": 0
},
{
"title": "Backend 5xx",
"query": "service=haproxy | where status>=500 | stats count",
"viz_type": "single_stat",
"position_x": 3,
"position_y": 0,
"width": 3,
"height": 3,
"query_language": "spl",
"sort_order": 1
},
{
"title": "Proxy denials",
"query": "service=squid proxy_action=\"TCP_DENIED/403\" | stats count",
"viz_type": "single_stat",
"position_x": 6,
"position_y": 0,
"width": 3,
"height": 3,
"query_language": "spl",
"sort_order": 2
},
{
"title": "Slowest backend (avg ms)",
"query": "service=haproxy | stats avg(duration_ms) as avg_ms by backend | sort -avg_ms | head 1",
"viz_type": "single_stat",
"position_x": 9,
"position_y": 0,
"width": 3,
"height": 3,
"query_language": "spl",
"sort_order": 3
},
{
"title": "Requests per backend over time",
"query": "SELECT toStartOfInterval(timestamp, INTERVAL 30 MINUTE) AS bucket, attributes['backend'] AS backend, count(*) AS count FROM logs WHERE service = 'haproxy' GROUP BY bucket, backend ORDER BY bucket",
"viz_type": "line",
"position_x": 0,
"position_y": 3,
"width": 12,
"height": 5,
"viz_config": {
"x_column": "bucket",
"value_column": "count",
"series_column": "backend"
},
"query_language": "sql",
"sort_order": 4
},
{
"title": "Backend latency",
"query": "service=haproxy | stats avg(duration_ms) as avg_ms by backend | sort -avg_ms",
"viz_type": "bar",
"position_x": 0,
"position_y": 8,
"width": 6,
"height": 5,
"viz_config": {
"x_column": "backend",
"value_column": "avg_ms"
},
"query_language": "spl",
"sort_order": 5
},
{
"title": "Outbound destinations",
"query": "service=squid | stats count by dest_host | sort -count | head 10",
"viz_type": "top_n",
"position_x": 6,
"position_y": 8,
"width": 6,
"height": 5,
"viz_config": {
"label_column": "dest_host",
"value_column": "count"
},
"query_language": "spl",
"sort_order": 6
},
{
"title": "Status by backend",
"query": "service=haproxy | stats count by backend, status",
"viz_type": "heatmap",
"position_x": 0,
"position_y": 13,
"width": 12,
"height": 5,
"viz_config": {
"x_column": "backend",
"y_column": "status",
"value_column": "count"
},
"query_language": "spl",
"sort_order": 7
}
]
}
@@ -0,0 +1,109 @@
{
"name": "Messaging and search",
"description": "Queue depth, consumer health and the search cluster beside them",
"default_earliest": "-24h",
"default_latest": "now",
"panels": [
{
"title": "Queue events",
"query": "service=rabbitmq | stats count",
"viz_type": "single_stat",
"position_x": 0,
"position_y": 0,
"width": 3,
"height": 3,
"query_language": "spl",
"sort_order": 0
},
{
"title": "Memory alarms",
"query": "service=rabbitmq event_kind=alarm | stats count",
"viz_type": "single_stat",
"position_x": 3,
"position_y": 0,
"width": 3,
"height": 3,
"query_language": "spl",
"sort_order": 1
},
{
"title": "Searches",
"query": "service=elasticsearch event_kind=search | stats count",
"viz_type": "single_stat",
"position_x": 6,
"position_y": 0,
"width": 3,
"height": 3,
"query_language": "spl",
"sort_order": 2
},
{
"title": "Long GC pauses",
"query": "service=elasticsearch event_kind=gc | stats count",
"viz_type": "single_stat",
"position_x": 9,
"position_y": 0,
"width": 3,
"height": 3,
"query_language": "spl",
"sort_order": 3
},
{
"title": "Queue depth over time",
"query": "SELECT toStartOfInterval(timestamp, INTERVAL 30 MINUTE) AS bucket, attributes['queue'] AS queue, max(toFloat64OrZero(attributes['queue_depth'])) AS depth FROM logs WHERE service = 'rabbitmq' GROUP BY bucket, queue ORDER BY bucket",
"viz_type": "line",
"position_x": 0,
"position_y": 3,
"width": 12,
"height": 5,
"viz_config": {
"x_column": "bucket",
"value_column": "depth",
"series_column": "queue"
},
"query_language": "sql",
"sort_order": 4
},
{
"title": "Deepest queues",
"query": "service=rabbitmq | stats max(queue_depth) as max_depth by queue | sort -max_depth",
"viz_type": "bar",
"position_x": 0,
"position_y": 8,
"width": 6,
"height": 5,
"viz_config": {
"x_column": "queue",
"value_column": "max_depth"
},
"query_language": "spl",
"sort_order": 5
},
{
"title": "Search latency by index",
"query": "service=elasticsearch event_kind=search | stats avg(duration_ms) as avg_ms by index | sort -avg_ms",
"viz_type": "bar",
"position_x": 6,
"position_y": 8,
"width": 6,
"height": 5,
"viz_config": {
"x_column": "index",
"value_column": "avg_ms"
},
"query_language": "spl",
"sort_order": 6
},
{
"title": "Indexing throttles and GC",
"query": "service=elasticsearch event_kind!=search | sort -timestamp | head 25 | fields timestamp, host, index, event_kind, message",
"viz_type": "table",
"position_x": 0,
"position_y": 13,
"width": 12,
"height": 5,
"query_language": "spl",
"sort_order": 7
}
]
}
@@ -0,0 +1,137 @@
{
"name": "Platform: Kubernetes, CI and secrets",
"description": "The tier everything else runs on -- node health, pod churn, build outcomes and secret access",
"default_earliest": "-24h",
"default_latest": "now",
"panels": [
{
"title": "Pod events",
"query": "service=kubelet | stats count",
"viz_type": "single_stat",
"position_x": 0,
"position_y": 0,
"width": 3,
"height": 3,
"query_language": "spl",
"sort_order": 0
},
{
"title": "Probe failures",
"query": "service=kubelet event_kind=probe_failed | stats count",
"viz_type": "single_stat",
"position_x": 3,
"position_y": 0,
"width": 3,
"height": 3,
"query_language": "spl",
"sort_order": 1
},
{
"title": "Image pull failures",
"query": "service=kubelet event_kind=image_pull_failed | stats count",
"viz_type": "single_stat",
"position_x": 6,
"position_y": 0,
"width": 3,
"height": 3,
"query_language": "spl",
"sort_order": 2
},
{
"title": "Failed builds",
"query": "service=jenkins result=FAILURE | stats count",
"viz_type": "single_stat",
"position_x": 9,
"position_y": 0,
"width": 3,
"height": 3,
"query_language": "spl",
"sort_order": 3
},
{
"title": "Kubelet events over time",
"query": "SELECT toStartOfInterval(timestamp, INTERVAL 30 MINUTE) AS bucket, attributes['event_kind'] AS event_kind, count(*) AS count FROM logs WHERE service = 'kubelet' GROUP BY bucket, event_kind ORDER BY bucket",
"viz_type": "line",
"position_x": 0,
"position_y": 3,
"width": 12,
"height": 5,
"viz_config": {
"x_column": "bucket",
"value_column": "count",
"series_column": "event_kind"
},
"query_language": "sql",
"sort_order": 4
},
{
"title": "Busiest pods",
"query": "service=kubelet | stats count by pod | sort -count | head 10",
"viz_type": "top_n",
"position_x": 0,
"position_y": 8,
"width": 6,
"height": 5,
"viz_config": {
"label_column": "pod",
"value_column": "count"
},
"query_language": "spl",
"sort_order": 5
},
{
"title": "Build outcomes by job",
"query": "service=jenkins | stats count by job, result",
"viz_type": "bar",
"position_x": 6,
"position_y": 8,
"width": 6,
"height": 5,
"viz_config": {
"x_column": "job",
"value_column": "count",
"series_column": "result",
"stacked": "true"
},
"query_language": "spl",
"sort_order": 6
},
{
"title": "Secret reads by path",
"query": "service=vault event_kind=read | stats count by vault_path | sort -count",
"viz_type": "bar",
"position_x": 0,
"position_y": 13,
"width": 6,
"height": 5,
"viz_config": {
"x_column": "vault_path",
"value_column": "count"
},
"query_language": "spl",
"sort_order": 7
},
{
"title": "Vault authentication failures",
"query": "service=vault event_kind=auth_failed | sort -timestamp | head 25 | fields timestamp, host, vault_path, remote_addr, message",
"viz_type": "table",
"position_x": 6,
"position_y": 13,
"width": 6,
"height": 5,
"query_language": "spl",
"sort_order": 8
},
{
"title": "Recent build failures",
"query": "service=jenkins result=FAILURE | sort -timestamp | head 25 | fields timestamp, host, job, build, duration_s",
"viz_type": "table",
"position_x": 0,
"position_y": 18,
"width": 12,
"height": 5,
"query_language": "spl",
"sort_order": 9
}
]
}
@@ -0,0 +1,137 @@
{
"name": "Windows server estate",
"description": "IIS, SQL Server, Exchange and the file servers -- the Windows tier as services rather than a Security channel",
"default_earliest": "-24h",
"default_latest": "now",
"panels": [
{
"title": "IIS requests",
"query": "service=iis | stats count",
"viz_type": "single_stat",
"position_x": 0,
"position_y": 0,
"width": 3,
"height": 3,
"query_language": "spl",
"sort_order": 0
},
{
"title": "IIS 5xx",
"query": "service=iis | where status>=500 | stats count",
"viz_type": "single_stat",
"position_x": 3,
"position_y": 0,
"width": 3,
"height": 3,
"query_language": "spl",
"sort_order": 1
},
{
"title": "SQL deadlocks",
"query": "service=mssql event_kind=deadlock | stats count",
"viz_type": "single_stat",
"position_x": 6,
"position_y": 0,
"width": 3,
"height": 3,
"query_language": "spl",
"sort_order": 2
},
{
"title": "Mail bounces",
"query": "service=exchange event_kind=bounce | stats count",
"viz_type": "single_stat",
"position_x": 9,
"position_y": 0,
"width": 3,
"height": 3,
"query_language": "spl",
"sort_order": 3
},
{
"title": "IIS traffic by site",
"query": "SELECT toStartOfInterval(timestamp, INTERVAL 30 MINUTE) AS bucket, attributes['site'] AS site, count(*) AS count FROM logs WHERE service = 'iis' GROUP BY bucket, site ORDER BY bucket",
"viz_type": "line",
"position_x": 0,
"position_y": 3,
"width": 12,
"height": 5,
"viz_config": {
"x_column": "bucket",
"value_column": "count",
"series_column": "site"
},
"query_language": "sql",
"sort_order": 4
},
{
"title": "IIS status mix",
"query": "service=iis | stats count by status | sort -count",
"viz_type": "bar",
"position_x": 0,
"position_y": 8,
"width": 6,
"height": 5,
"viz_config": {
"x_column": "status",
"value_column": "count"
},
"query_language": "spl",
"sort_order": 5
},
{
"title": "SQL Server events by database",
"query": "service=mssql | stats count by db, event_kind",
"viz_type": "bar",
"position_x": 6,
"position_y": 8,
"width": 6,
"height": 5,
"viz_config": {
"x_column": "db",
"value_column": "count",
"series_column": "event_kind",
"stacked": "true"
},
"query_language": "spl",
"sort_order": 6
},
{
"title": "Busiest shares",
"query": "service=smb | stats count by share | sort -count | head 10",
"viz_type": "top_n",
"position_x": 0,
"position_y": 13,
"width": 6,
"height": 5,
"viz_config": {
"label_column": "share",
"value_column": "count"
},
"query_language": "spl",
"sort_order": 7
},
{
"title": "Denied share access",
"query": "service=smb event_kind=share_denied | sort -timestamp | head 25 | fields timestamp, host, share, winevt.target_user, message",
"viz_type": "table",
"position_x": 6,
"position_y": 13,
"width": 6,
"height": 5,
"query_language": "spl",
"sort_order": 8
},
{
"title": "SQL login failures and deadlocks",
"query": "service=mssql event_kind!=login | sort -timestamp | head 25 | fields timestamp, host, db, event_kind, message",
"viz_type": "table",
"position_x": 0,
"position_y": 18,
"width": 12,
"height": 5,
"query_language": "spl",
"sort_order": 9
}
]
}