{a.title}
+ {#if a.detail}{a.detail}
{/if} + {#if a.affected}What it touches {a.affected}
{/if} +diff --git a/docs/cluster-watcher.md b/docs/cluster-watcher.md index 79058c5..20358c8 100644 --- a/docs/cluster-watcher.md +++ b/docs/cluster-watcher.md @@ -327,12 +327,13 @@ hide a real outage. Built on the paged, server-side lists from #69. No page ever lists everything at once. -- **Dashboard:** one line per cluster: its version, state and a summary +- **Clusters page:** one line per cluster: its version, state and a summary in its kind's terms: "31/32 nodes ready · 3,300 workloads" for Kubernetes, "2/2 controllers ready · 600 volumes · 71% used" for an array, "CPU 41% · system ASP 85% · 4 of 5 subsystems active" for an IBM i partition. The network map's cluster boxes use the same line. - The "needs attention" list takes workloads too, capped as for hosts. + The dashboard counts the clusters with a problem, and its "needs you" + and "can wait" lists take workloads too, capped as for hosts. - **Clusters:** in sections by kind (Kubernetes, Virtualization, Storage, IBM i, Secrets), the ones that need a look first in each, with a jump list counting each kind and its problems. Filters for kind, "need a diff --git a/docs/formats.md b/docs/formats.md index c70d2de..daf8275 100644 --- a/docs/formats.md +++ b/docs/formats.md @@ -728,8 +728,29 @@ restarts; admins can preview it and send it by hand on the Reports page. ### The dashboard's briefing -The same history feeds the dashboard. `GET /api/v1/overview` carries, -besides the readings: +The dashboard (Today) is written to be read, not scanned for numbers. It +opens with a sentence on whether anything needs you and whether +customers are affected (the services with `tier: public`), then: + +- **Needs you:** critical and serious problems nobody has picked up, each + with when it started, whether people were told and what it takes down. + **I'm on it** is the acknowledgement: the repeats stop, and the problem + moves to **Being handled** with who has it. +- **Can wait:** warnings, one line each with why. +- **Your services:** each service's state and its last 24 hours, one bar + per half hour. The **Services** page lists them all by tier (what + customers use, what runs it, where the data lives, the edge); picking one + shows its parts in trouble, what it needs and what needs it. +- **Since you last looked:** what opened and closed since your last visit. +- **The fleet:** hosts reporting, containers, clusters and the busiest + host, and what's worth watching: disks filling up, hosts busier than + usual for the hour, hosts restarted in the last day. The **Hosts** page + shows the same list over one square per host, grouped by the network + map's sites and by role, the host name's first word after its site + (phx1-app05 is an app server); the table is a click away. + +The same history feeds it. `GET /api/v1/overview` carries, besides the +readings: - `today`: what happened since this person last looked. A visit ends after 30 minutes without the dashboard open, and the end of the last one is @@ -754,7 +775,8 @@ besides the readings: The strips and uptimes are read from history once a minute (30 days every ten), the watch list every ten minutes, whatever the number of dashboards -open. +open. `GET /api/v1/fleet` is the Hosts page's squares: every host's state +by site and role, the hosts that need a look per site, and the watch list. ## checks.yaml: active checks diff --git a/internal/server/fleetgrid.go b/internal/server/fleetgrid.go new file mode 100644 index 0000000..bf7da5b --- /dev/null +++ b/internal/server/fleetgrid.go @@ -0,0 +1,148 @@ +package server + +import ( + "net/http" + "sort" + "strings" + "time" +) + +// The Hosts page's grid (#72): one square per host, grouped by site (the +// network map's) and role, so at thousands of hosts the odd one out still +// stands out. GET /api/v1/fleet; not under /hosts/, where a host named +// "grid" would lose its page. + +// GridSite is a site's hosts by role. +type GridSite struct { + ID string `json:"id"` + Name string `json:"name"` + Roles []GridRole `json:"roles"` + Hosts int `json:"hosts"` + // Trouble is its hosts that need a look, worst first. + Trouble []GridHost `json:"trouble"` +} + +// GridRole is one role's hosts in a site. +type GridRole struct { + Role string `json:"role"` + Hosts []GridHost `json:"hosts"` +} + +// GridHost is one square. +type GridHost struct { + Name string `json:"name"` + Status string `json:"status"` + Word string `json:"word"` +} + +// hostRole is what a host is from its name: the first word after its +// site, without its number. phx1-app05 in phx1 is "app", web-02 is "web", +// win-dc01 is "win", host3 is "host". +func hostRole(name, site string) string { + r := name + if site != "" && strings.HasPrefix(r, site) { + r = r[len(site):] + } + r = strings.Trim(r, "-_.") + if i := strings.IndexAny(r, "-_."); i >= 0 { + r = r[:i] + } + r = strings.TrimRight(r, "0123456789") + if r == "" { + return "other" + } + return r +} + +// fleetGrid groups hosts by site and role. siteOf names each host's site +// (id → name in names); hosts in none go under "Elsewhere". +func fleetGrid(hosts []HostView, siteOf map[string]string, names map[string]string) []GridSite { + by := map[string]*GridSite{} + roles := map[string]map[string]*GridRole{} + // A host the map didn't place (a collector, one without map samples) + // goes with the site its name starts with, when one does: phx1-stor01 + // with phx1. + prefixes := map[string]string{} + for id, n := range names { + if n != "" { + prefixes[n] = id + } + } + for _, h := range hosts { + id := siteOf[h.Name] + if id == "" { + if p, _, ok := strings.Cut(h.Name, "-"); ok && prefixes[p] != "" { + id = prefixes[p] + } + } + s := by[id] + if s == nil { + name := names[id] + if id == "" { + name = "Elsewhere" + } + s = &GridSite{ID: id, Name: name, Trouble: []GridHost{}} + by[id] = s + roles[id] = map[string]*GridRole{} + } + prefix := names[id] + role := hostRole(h.Name, prefix) + r := roles[id][role] + if r == nil { + r = &GridRole{Role: role} + roles[id][role] = r + } + st := h.Status + if h.Silent { + st = "critical" + } + g := GridHost{Name: h.Name, Status: st, Word: h.Word} + r.Hosts = append(r.Hosts, g) + s.Hosts++ + if st == "warning" || st == "serious" || st == "critical" { + s.Trouble = append(s.Trouble, g) + } + } + out := []GridSite{} + for id, s := range by { + for _, r := range roles[id] { + sort.Slice(r.Hosts, func(i, j int) bool { return r.Hosts[i].Name < r.Hosts[j].Name }) + s.Roles = append(s.Roles, *r) + } + // The biggest role first (the app servers), then by name. + sort.Slice(s.Roles, func(i, j int) bool { + if len(s.Roles[i].Hosts) != len(s.Roles[j].Hosts) { + return len(s.Roles[i].Hosts) > len(s.Roles[j].Hosts) + } + return s.Roles[i].Role < s.Roles[j].Role + }) + sort.SliceStable(s.Trouble, func(i, j int) bool { return statusRank[s.Trouble[i].Status] > statusRank[s.Trouble[j].Status] }) + out = append(out, *s) + } + // Sites by size, Elsewhere last. + sort.Slice(out, func(i, j int) bool { + if (out[i].ID == "") != (out[j].ID == "") { + return out[j].ID == "" + } + if out[i].Hosts != out[j].Hosts { + return out[i].Hosts > out[j].Hosts + } + return out[i].Name < out[j].Name + }) + return out +} + +func (srv *Server) handleFleet(w http.ResponseWriter, r *http.Request) { + now := time.Now() + hosts, _ := srv.view(now) + siteOf, names := map[string]string{}, map[string]string{} + if g := srv.mapGraph(now); g != nil { + for _, s := range g.Sites { + names[s.ID] = s.Name + for _, h := range s.Hosts { + siteOf[h] = s.ID + } + } + } + writeJSON(w, http.StatusOK, map[string]any{"sites": fleetGrid(hosts, siteOf, names), "watch": srv.watch(hosts, now)}) +} diff --git a/internal/server/server.go b/internal/server/server.go index d6b31c4..7670912 100644 --- a/internal/server/server.go +++ b/internal/server/server.go @@ -428,6 +428,7 @@ func (srv *Server) Handler() http.Handler { mux.HandleFunc("GET /api/v1/map/network", srv.needSession(srv.handleMapNetwork)) mux.HandleFunc("GET /api/v1/map/services", srv.needSession(srv.handleMapServices)) mux.HandleFunc("GET /api/v1/services", srv.needSession(srv.handleServices)) + mux.HandleFunc("GET /api/v1/fleet", srv.needSession(srv.handleFleet)) mux.HandleFunc("GET /api/v1/map/find", srv.needSession(srv.handleMapFind)) mux.HandleFunc("GET /api/v1/map/found", srv.needSession(srv.handleMapFound)) mux.HandleFunc("GET /api/v1/map/corrections", srv.needSession(srv.handleMapCorrections)) diff --git a/internal/server/today_test.go b/internal/server/today_test.go index 62b35e0..c9b4c21 100644 --- a/internal/server/today_test.go +++ b/internal/server/today_test.go @@ -151,3 +151,34 @@ func TestWithIncidents(t *testing.T) { t.Errorf("others: %+v %+v", got[1].Incident, got[2].Incident) } } + +func TestFleetGrid(t *testing.T) { + for _, c := range [][3]string{{"phx1-app05", "phx1", "app"}, {"web-02", "", "web"}, {"host3", "", "host"}, {"db1.example.com", "", "db"}, {"phx1-07", "phx1", "other"}, {"win-dc01", "", "win"}, {"lb-edge-01", "", "lb"}} { + if got := hostRole(c[0], c[1]); got != c[2] { + t.Errorf("hostRole(%s, %s) = %s, want %s", c[0], c[1], got, c[2]) + } + } + hosts := []HostView{ + {Name: "phx1-app02", Status: "good"}, {Name: "phx1-app01", Status: "warning", Word: "disk 91%"}, + {Name: "phx1-db01", Status: "good"}, {Name: "lon1-app01", Status: "good"}, + {Name: "bench", Status: "good", Silent: true}, {Name: "phx1-stor01", Status: "good"}, + } + siteOf := map[string]string{"phx1-app01": "s1", "phx1-app02": "s1", "phx1-db01": "s1", "lon1-app01": "s2"} + names := map[string]string{"s1": "phx1", "s2": "lon1"} + got := fleetGrid(hosts, siteOf, names) + if len(got) != 3 || got[0].Name != "phx1" || got[1].Name != "lon1" || got[2].Name != "Elsewhere" { + t.Fatalf("sites = %+v", got) + } + if got[0].Hosts != 4 { + t.Errorf("phx1-stor01 should join phx1 by its name: %+v", got[0]) + } + if r := got[0].Roles; len(r) != 3 || r[0].Role != "app" || r[0].Hosts[0].Name != "phx1-app01" || r[1].Role != "db" || r[2].Role != "stor" { + t.Errorf("phx1 roles = %+v", r) + } + if tr := got[0].Trouble; len(tr) != 1 || tr[0].Name != "phx1-app01" { + t.Errorf("phx1 trouble = %+v", tr) + } + if tr := got[2].Trouble; len(tr) != 1 || tr[0].Status != "critical" { + t.Errorf("a silent host is trouble: %+v", tr) + } +} diff --git a/web/src/lib/FleetGrid.svelte b/web/src/lib/FleetGrid.svelte new file mode 100644 index 0000000..f8d526d --- /dev/null +++ b/web/src/lib/FleetGrid.svelte @@ -0,0 +1,144 @@ + + +{#if error}
{error}
{/if} + +{#if watch && (watch.filling.length || watch.busier.length || watch.restarted_n)} +and {watch.restarted_n - watch.restarted.length} more
{/if} +Loading…
+{:else} + +Here's how your systems are doing.
-{hello}{firstName ? ', ' + firstName : ''} · {dayFmt.format(now)}
+ {#if o} +{!needs.length && customers ? customers + ' ' : ''}{subline}
{/if} + {:else} ++ {live.error ? live.error : checked < 0 ? 'Connecting' : checked < 3 ? 'Live · checked just now' : `Live · checked ${checked} s ago`} +
+ -{#if o} -{a.detail}
{/if}{error}
{/if} + + {#if needs.length || taken.length} +Worst first. {needs.length ? 'Nobody has picked these up yet.' : 'Everything urgent has someone on it.'}
+{a.detail}
{/if} + {#if a.affected}What it touches {a.affected}
{/if} +Every host is reporting, no disk is past 80%, and every container is where it should be.
+ {#if taken.length} +Worth a look today. None of these is hurting anything yet.
+Cairn OBS tells the alert channels if that changes.{watch?.filling.length ? ` Next to look at: ${watch.filling[0].host}'s ${watch.filling[0].mount}, which fills ${days(watch.filling[0].full_days)}.` : ''}
+Last 24 hours, one bar per half hour · Service map
+{today.last_visit ? `You were last here ${ago(today.since, now)}, at ${hm(today.since)}` : 'The last 12 hours'}
+ {today.opened} opened, {today.resolved} closed{today.fixed_itself ? ` (${today.fixed_itself} on their own)` : ''}. {today.told ? `${today.told} ${today.told === 1 ? 'alert' : 'alerts'} went out to people.` : 'Nobody was alerted.'} + {#if today.opened + today.resolved > today.events.length}Everything on the Alerts page{/if} +
+ {:else} +Nothing opened or closed since then.
+ {/if} + + {/if} + +Loading…
-{/if} +Every host with an agent, the ones that need a look first. Adding and removing hosts is under Configure.
Every host with an agent, grouped by site and role so the odd one out stands out. Adding and removing hosts is under Configure.
{error}
{/if}Refreshes every 10 seconds.
diff --git a/web/src/routes/services/+page.svelte b/web/src/routes/services/+page.svelte new file mode 100644 index 0000000..83dc6db --- /dev/null +++ b/web/src/routes/services/+page.svelte @@ -0,0 +1,227 @@ + + +Grouped by who depends on them. Pick one to see what it’s made of and what breaks if it goes.
+{error}
{/if} + + {#if services === null} +Loading…
+ {:else if !topology || !all.length} +No services yet. Services come from the service map: declare them in topology.yaml, or correct what the map found (docs/maps.md).
+ {:else} +