Add two storefronts, a payment gateway, and alerts worth waking up for
The estate could show an operator their infrastructure and had nothing to say to the business paying for it. Two storefronts and the gateway behind both fix that: Magento on two hosts, WooCommerce on one, and pay-01 carrying authorisations with amount, gateway and decline reason. Orders, revenue, average order value, where checkout loses people and why a card was refused now come out of the same log lines the operators are already reading, which is the argument for not running a separate metrics stack beside this one. Two platforms rather than one deliberately. Magento and WooCommerce write about the same events differently, so a panel that groups by service instead of assuming a single shape is the honest way to build one -- and the demo shows that rather than describing it. Order totals are built from a basket of real SKUs at real prices rather than drawn from a distribution, so average order value moves the way one actually moves. Declines rise during the seeded outage window alongside the 5xx rate, because whatever fails requests fails authorisations too. Twenty-seven new alert rules, thresholds calibrated against what the fleet actually emits -- measured on the demo's own week of history rather than guessed. A rule set at the average fires constantly and one set an order of magnitude above it never fires; these sit two to three times the steady-state rate, so they are quiet in normal operation and true during the diurnal peak or the seeded incident. Four are absence rules, because a domain controller or a storefront going silent is not a threshold question. Six new dashboards: fleet health, golden signals, security posture, capacity and storage, commerce, payments. Three limits of the query language found the hard way and worth writing down, because each was discovered by a panel failing rather than by reading: `dc()` does not exist -- the functions are count, sum, avg, min and max; `or` is not supported between structured filters, so a panel spanning tiers filters on the attribute they share and groups by service; and dashboards refuse raw SQL outright. The validator run over all 169 panels and 38 rules now checks every one of those, plus stages, viz types and comparators.
This commit is contained in:
@@ -0,0 +1,11 @@
|
||||
{
|
||||
"name": "DC-01 not reporting",
|
||||
"description": "No heartbeat from the primary domain controller. Everything that authenticates depends on this host",
|
||||
"query": "earliest=-5m host=\"DC-01\" cairnobs.heartbeat=true",
|
||||
"query_language": "spl",
|
||||
"condition_type": "absence",
|
||||
"eval_interval_seconds": 60,
|
||||
"for_minutes": 0,
|
||||
"notification_target_id": "__TARGET_PLATFORM__",
|
||||
"enabled": true
|
||||
}
|
||||
@@ -0,0 +1,13 @@
|
||||
{
|
||||
"name": "Backup job failures",
|
||||
"description": "Backup jobs are erroring. Any run of these is worth attention: the failure is silent until the restore",
|
||||
"query": "service=backup result=error earliest=-1h | stats count",
|
||||
"query_language": "spl",
|
||||
"condition_type": "threshold",
|
||||
"eval_interval_seconds": 300,
|
||||
"for_minutes": 0,
|
||||
"notification_target_id": "__TARGET_PLATFORM__",
|
||||
"enabled": true,
|
||||
"comparator": "gt",
|
||||
"threshold_value": 4
|
||||
}
|
||||
@@ -0,0 +1,13 @@
|
||||
{
|
||||
"name": "CI build failures",
|
||||
"description": "Builds are failing well above the usual rate, which usually means main is broken rather than any one change being bad",
|
||||
"query": "service=jenkins result=FAILURE earliest=-30m | stats count",
|
||||
"query_language": "spl",
|
||||
"condition_type": "threshold",
|
||||
"eval_interval_seconds": 300,
|
||||
"for_minutes": 0,
|
||||
"notification_target_id": "__TARGET_PLATFORM__",
|
||||
"enabled": true,
|
||||
"comparator": "gt",
|
||||
"threshold_value": 12
|
||||
}
|
||||
@@ -0,0 +1,13 @@
|
||||
{
|
||||
"name": "Checkout errors on WooCommerce",
|
||||
"description": "Checkout is refusing customers above baseline -- an expired coupon campaign, a shipping rule change, or sessions timing out",
|
||||
"query": "service=woocommerce event_kind=checkout_error earliest=-15m | stats count",
|
||||
"query_language": "spl",
|
||||
"condition_type": "threshold",
|
||||
"eval_interval_seconds": 300,
|
||||
"for_minutes": 0,
|
||||
"notification_target_id": "__TARGET_OPS__",
|
||||
"enabled": true,
|
||||
"comparator": "gt",
|
||||
"threshold_value": 30
|
||||
}
|
||||
@@ -0,0 +1,11 @@
|
||||
{
|
||||
"name": "Orders have stopped",
|
||||
"description": "No order has been placed on either storefront for fifteen minutes. At this estate's volume that is not a quiet spell, it is an outage somebody has not noticed yet",
|
||||
"query": "event_kind=order earliest=-15m",
|
||||
"query_language": "spl",
|
||||
"condition_type": "absence",
|
||||
"eval_interval_seconds": 300,
|
||||
"for_minutes": 0,
|
||||
"notification_target_id": "__TARGET_OPS__",
|
||||
"enabled": true
|
||||
}
|
||||
@@ -0,0 +1,13 @@
|
||||
{
|
||||
"name": "Products out of stock",
|
||||
"description": "Repeated attempts to buy items with nothing on hand: the catalogue and the warehouse disagree, and customers are finding out first",
|
||||
"query": "service=magento event_kind=out_of_stock earliest=-30m | stats count",
|
||||
"query_language": "spl",
|
||||
"condition_type": "threshold",
|
||||
"eval_interval_seconds": 300,
|
||||
"for_minutes": 0,
|
||||
"notification_target_id": "__TARGET_OPS__",
|
||||
"enabled": true,
|
||||
"comparator": "gt",
|
||||
"threshold_value": 40
|
||||
}
|
||||
@@ -0,0 +1,13 @@
|
||||
{
|
||||
"name": "Storefront exceptions",
|
||||
"description": "Uncaught exceptions in the storefront code. Every one of these is a customer looking at an error page mid-purchase",
|
||||
"query": "event_kind=exception earliest=-10m | stats count",
|
||||
"query_language": "spl",
|
||||
"condition_type": "threshold",
|
||||
"eval_interval_seconds": 60,
|
||||
"for_minutes": 3,
|
||||
"notification_target_id": "__TARGET_OPS__",
|
||||
"enabled": true,
|
||||
"comparator": "gt",
|
||||
"threshold_value": 14
|
||||
}
|
||||
@@ -0,0 +1,11 @@
|
||||
{
|
||||
"name": "Primary database silent",
|
||||
"description": "db-01 has logged nothing for ten minutes. Postgres is never quiet on a live system, so silence means the service or the agent has stopped",
|
||||
"query": "earliest=-10m host=\"db-01\" service=postgres",
|
||||
"query_language": "spl",
|
||||
"condition_type": "absence",
|
||||
"eval_interval_seconds": 60,
|
||||
"for_minutes": 0,
|
||||
"notification_target_id": "__TARGET_OPS__",
|
||||
"enabled": true
|
||||
}
|
||||
@@ -0,0 +1,13 @@
|
||||
{
|
||||
"name": "DNS NXDOMAIN surge",
|
||||
"description": "A jump in non-existent domain lookups: a typo-squat campaign, a misconfigured client, or malware working through a generated domain list",
|
||||
"query": "service=bind dns_rcode=NXDOMAIN earliest=-10m | stats count",
|
||||
"query_language": "spl",
|
||||
"condition_type": "threshold",
|
||||
"eval_interval_seconds": 60,
|
||||
"for_minutes": 3,
|
||||
"notification_target_id": "__TARGET_SECURITY__",
|
||||
"enabled": true,
|
||||
"comparator": "gt",
|
||||
"threshold_value": 18
|
||||
}
|
||||
@@ -0,0 +1,11 @@
|
||||
{
|
||||
"name": "Edge tier silent",
|
||||
"description": "Neither load balancer has logged a request. Either traffic has stopped reaching the estate or both agents are down",
|
||||
"query": "earliest=-10m service=haproxy",
|
||||
"query_language": "spl",
|
||||
"condition_type": "absence",
|
||||
"eval_interval_seconds": 60,
|
||||
"for_minutes": 0,
|
||||
"notification_target_id": "__TARGET_OPS__",
|
||||
"enabled": true
|
||||
}
|
||||
@@ -0,0 +1,13 @@
|
||||
{
|
||||
"name": "Elasticsearch GC pauses",
|
||||
"description": "Old-generation collections are running long and often -- the heap is under pressure and search latency will follow",
|
||||
"query": "service=elasticsearch event_kind=gc earliest=-10m | stats count",
|
||||
"query_language": "spl",
|
||||
"condition_type": "threshold",
|
||||
"eval_interval_seconds": 60,
|
||||
"for_minutes": 5,
|
||||
"notification_target_id": "__TARGET_PLATFORM__",
|
||||
"enabled": true,
|
||||
"comparator": "gt",
|
||||
"threshold_value": 20
|
||||
}
|
||||
@@ -0,0 +1,13 @@
|
||||
{
|
||||
"name": "Exchange bounce rate",
|
||||
"description": "Mail is bouncing above the usual rate: a directory out of step with reality, or a distribution list pointing at deleted mailboxes",
|
||||
"query": "service=exchange event_kind=bounce earliest=-15m | stats count",
|
||||
"query_language": "spl",
|
||||
"condition_type": "threshold",
|
||||
"eval_interval_seconds": 300,
|
||||
"for_minutes": 0,
|
||||
"notification_target_id": "__TARGET_OPS__",
|
||||
"enabled": true,
|
||||
"comparator": "gt",
|
||||
"threshold_value": 20
|
||||
}
|
||||
@@ -0,0 +1,13 @@
|
||||
{
|
||||
"name": "HAProxy backend 5xx surge",
|
||||
"description": "The load balancers are returning 5xx from their backends well above the ~0.8 per ten minutes a healthy estate produces",
|
||||
"query": "service=haproxy earliest=-10m | where status>=500 | stats count",
|
||||
"query_language": "spl",
|
||||
"condition_type": "threshold",
|
||||
"eval_interval_seconds": 60,
|
||||
"for_minutes": 2,
|
||||
"notification_target_id": "__TARGET_OPS__",
|
||||
"enabled": true,
|
||||
"comparator": "gt",
|
||||
"threshold_value": 4
|
||||
}
|
||||
@@ -0,0 +1,13 @@
|
||||
{
|
||||
"name": "IIS 5xx surge",
|
||||
"description": "The Windows web tier is failing requests -- normally under one per ten minutes across all three IIS hosts",
|
||||
"query": "service=iis earliest=-10m | where status>=500 | stats count",
|
||||
"query_language": "spl",
|
||||
"condition_type": "threshold",
|
||||
"eval_interval_seconds": 60,
|
||||
"for_minutes": 2,
|
||||
"notification_target_id": "__TARGET_OPS__",
|
||||
"enabled": true,
|
||||
"comparator": "gt",
|
||||
"threshold_value": 5
|
||||
}
|
||||
@@ -0,0 +1,13 @@
|
||||
{
|
||||
"name": "Kubernetes image pulls failing",
|
||||
"description": "Nodes cannot pull images: a registry outage, an expired pull secret, or a tag that was deleted out from under a deployment",
|
||||
"query": "service=kubelet event_kind=image_pull_failed earliest=-10m | stats count",
|
||||
"query_language": "spl",
|
||||
"condition_type": "threshold",
|
||||
"eval_interval_seconds": 60,
|
||||
"for_minutes": 5,
|
||||
"notification_target_id": "__TARGET_PLATFORM__",
|
||||
"enabled": true,
|
||||
"comparator": "gt",
|
||||
"threshold_value": 30
|
||||
}
|
||||
@@ -0,0 +1,13 @@
|
||||
{
|
||||
"name": "Kubernetes readiness probes failing",
|
||||
"description": "Readiness probes are failing across the cluster at roughly double the normal rate -- pods are being pulled out of service",
|
||||
"query": "service=kubelet event_kind=probe_failed earliest=-10m | stats count",
|
||||
"query_language": "spl",
|
||||
"condition_type": "threshold",
|
||||
"eval_interval_seconds": 60,
|
||||
"for_minutes": 5,
|
||||
"notification_target_id": "__TARGET_PLATFORM__",
|
||||
"enabled": true,
|
||||
"comparator": "gt",
|
||||
"threshold_value": 55
|
||||
}
|
||||
@@ -0,0 +1,13 @@
|
||||
{
|
||||
"name": "LDAP bind failures",
|
||||
"description": "Failed LDAP binds are essentially zero in normal operation, so any run of them is worth waking up for",
|
||||
"query": "service=openldap event_kind=bind_failed earliest=-10m | stats count",
|
||||
"query_language": "spl",
|
||||
"condition_type": "threshold",
|
||||
"eval_interval_seconds": 60,
|
||||
"for_minutes": 0,
|
||||
"notification_target_id": "__TARGET_SECURITY__",
|
||||
"enabled": true,
|
||||
"comparator": "gt",
|
||||
"threshold_value": 5
|
||||
}
|
||||
@@ -0,0 +1,13 @@
|
||||
{
|
||||
"name": "SQL Server deadlocks",
|
||||
"description": "Transactions are being chosen as deadlock victims well above baseline; something is taking locks in a new order",
|
||||
"query": "service=mssql event_kind=deadlock earliest=-10m | stats count",
|
||||
"query_language": "spl",
|
||||
"condition_type": "threshold",
|
||||
"eval_interval_seconds": 60,
|
||||
"for_minutes": 3,
|
||||
"notification_target_id": "__TARGET_OPS__",
|
||||
"enabled": true,
|
||||
"comparator": "gt",
|
||||
"threshold_value": 12
|
||||
}
|
||||
@@ -0,0 +1,13 @@
|
||||
{
|
||||
"name": "SQL Server I/O stalls",
|
||||
"description": "SQL Server is reporting I/O requests over 15 seconds. The storage under the data files is not keeping up",
|
||||
"query": "service=mssql event_kind=io_stall earliest=-10m | stats count",
|
||||
"query_language": "spl",
|
||||
"condition_type": "threshold",
|
||||
"eval_interval_seconds": 60,
|
||||
"for_minutes": 5,
|
||||
"notification_target_id": "__TARGET_OPS__",
|
||||
"enabled": true,
|
||||
"comparator": "gt",
|
||||
"threshold_value": 35
|
||||
}
|
||||
@@ -0,0 +1,13 @@
|
||||
{
|
||||
"name": "SQL Server login failures",
|
||||
"description": "Repeated failed logins against SQL Server -- credential stuffing against a service account, or an application with a stale password",
|
||||
"query": "service=mssql event_kind=login_failed earliest=-10m | stats count",
|
||||
"query_language": "spl",
|
||||
"condition_type": "threshold",
|
||||
"eval_interval_seconds": 60,
|
||||
"for_minutes": 0,
|
||||
"notification_target_id": "__TARGET_SECURITY__",
|
||||
"enabled": true,
|
||||
"comparator": "gt",
|
||||
"threshold_value": 12
|
||||
}
|
||||
@@ -0,0 +1,13 @@
|
||||
{
|
||||
"name": "Card chargebacks",
|
||||
"description": "Chargebacks above the usual trickle, which is the first visible sign of card testing or a fraud run getting through",
|
||||
"query": "event_kind=chargeback earliest=-1h | stats count",
|
||||
"query_language": "spl",
|
||||
"condition_type": "threshold",
|
||||
"eval_interval_seconds": 300,
|
||||
"for_minutes": 0,
|
||||
"notification_target_id": "__TARGET_SECURITY__",
|
||||
"enabled": true,
|
||||
"comparator": "gt",
|
||||
"threshold_value": 8
|
||||
}
|
||||
@@ -0,0 +1,13 @@
|
||||
{
|
||||
"name": "Payment decline rate elevated",
|
||||
"description": "Declines are running well above the ~7.5% baseline. Either a gateway is unwell or something upstream is sending bad authorisations -- both cost orders directly",
|
||||
"query": "auth_result=declined earliest=-15m | stats count",
|
||||
"query_language": "spl",
|
||||
"condition_type": "threshold",
|
||||
"eval_interval_seconds": 300,
|
||||
"for_minutes": 5,
|
||||
"notification_target_id": "__TARGET_OPS__",
|
||||
"enabled": true,
|
||||
"comparator": "gt",
|
||||
"threshold_value": 120
|
||||
}
|
||||
@@ -0,0 +1,13 @@
|
||||
{
|
||||
"name": "RabbitMQ memory alarms",
|
||||
"description": "Brokers are raising memory alarms, which blocks publishers -- the queue is not draining as fast as it is filling",
|
||||
"query": "service=rabbitmq event_kind=alarm earliest=-10m | stats count",
|
||||
"query_language": "spl",
|
||||
"condition_type": "threshold",
|
||||
"eval_interval_seconds": 60,
|
||||
"for_minutes": 3,
|
||||
"notification_target_id": "__TARGET_PLATFORM__",
|
||||
"enabled": true,
|
||||
"comparator": "gt",
|
||||
"threshold_value": 15
|
||||
}
|
||||
@@ -0,0 +1,13 @@
|
||||
{
|
||||
"name": "RabbitMQ queue backing up",
|
||||
"description": "A queue has passed 2,500 ready messages. Normal depth is a few hundred; this is the shape of consumers dying rather than traffic rising",
|
||||
"query": "service=rabbitmq earliest=-15m | stats max(queue_depth) as max_depth",
|
||||
"query_language": "spl",
|
||||
"condition_type": "threshold",
|
||||
"eval_interval_seconds": 300,
|
||||
"for_minutes": 0,
|
||||
"notification_target_id": "__TARGET_PLATFORM__",
|
||||
"enabled": true,
|
||||
"comparator": "gt",
|
||||
"threshold_value": 2500
|
||||
}
|
||||
@@ -0,0 +1,13 @@
|
||||
{
|
||||
"name": "SMB access denials",
|
||||
"description": "Share access is being denied repeatedly -- someone or something is reaching for files it has no rights to",
|
||||
"query": "service=smb event_kind=share_denied earliest=-10m | stats count",
|
||||
"query_language": "spl",
|
||||
"condition_type": "threshold",
|
||||
"eval_interval_seconds": 60,
|
||||
"for_minutes": 0,
|
||||
"notification_target_id": "__TARGET_SECURITY__",
|
||||
"enabled": true,
|
||||
"comparator": "gt",
|
||||
"threshold_value": 12
|
||||
}
|
||||
@@ -0,0 +1,13 @@
|
||||
{
|
||||
"name": "Squid blocking outbound requests",
|
||||
"description": "The outbound proxy is denying requests above baseline -- something on the inside is repeatedly reaching for a blocked destination",
|
||||
"query": "service=squid status=403 earliest=-10m | stats count",
|
||||
"query_language": "spl",
|
||||
"condition_type": "threshold",
|
||||
"eval_interval_seconds": 60,
|
||||
"for_minutes": 3,
|
||||
"notification_target_id": "__TARGET_SECURITY__",
|
||||
"enabled": true,
|
||||
"comparator": "gt",
|
||||
"threshold_value": 12
|
||||
}
|
||||
@@ -0,0 +1,13 @@
|
||||
{
|
||||
"name": "Vault authentication failures",
|
||||
"description": "Something is failing to authenticate to Vault. Near-zero normally, and the one place where a handful of failures is already a story",
|
||||
"query": "service=vault event_kind=auth_failed earliest=-10m | stats count",
|
||||
"query_language": "spl",
|
||||
"condition_type": "threshold",
|
||||
"eval_interval_seconds": 60,
|
||||
"for_minutes": 0,
|
||||
"notification_target_id": "__TARGET_SECURITY__",
|
||||
"enabled": true,
|
||||
"comparator": "gt",
|
||||
"threshold_value": 3
|
||||
}
|
||||
Reference in New Issue
Block a user