inbuxa/inbuxa-server
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 2 Wiki Activity
Files
cd7c1a6d4c386675f385aeb5d9cb7fdeaa2faeb5
inbuxa-server/docs/spec/features/monitoring.md
T

30 KiB
Raw Blame History

Feature spec: monitoring history, live tracing and alerts

Status: draft, 2026-09-18. Feature 6 in SPEC.md §4.

Provenance

Written for the clean room (SPEC.md §3). Sources, and nothing else:

Source License Used for
Stalwart's registry schema: crates/registry/src/schema/*.rs and resources/schema/schema.json.gz (objects, fields, defaults, enums, permissions, lists, forms, the dashboards and layouts sections), upstream v0.16.22 AGPL-3.0-only OR LicenseRef-SEL, taken under the AGPL Object shapes, field meanings, defaults, what upstream flags as Enterprise, the admin views and dashboards the server has to feed
The AGPL telemetry code left after the strip: crates/common/src/telemetry/, crates/common/src/config/telemetry.rs, crates/trc (event, key and metric definitions, the collector, span tracking, the JSON serializer), crates/http/src/api/mod.rs and diagnose.rs, crates/store key layout, crates/services/src/task_manager/, crates/common/src/auth/permissions.rs, crates/common/src/expr/, crates/common/src/manager/defaults.rs, crates/jmap/src/registry/ AGPL-3.0-only OR LicenseRef-SEL, taken under the AGPL What already exists and must be kept: webhooks, exporters, the metric() expression function, span ids, storage subspaces, route and token names, default permissions
The integration suites tests/src/telemetry/*.rs AGPL-3.0-only OR LicenseRef-SEL, taken under the AGPL Tested behavior of alerts, metrics history, trace history and webhooks
The strip report, docs/fork/strip-reports/v0.16.22.json This repository Which files and snippets were removed, so which behavior is missing
Stalwart documentation (website repo): docs/telemetry/{index,alerts,history,live,management,webhooks}.md, docs/0.15/telemetry/{alerts,live,history}.md, docs/ref/object/{trace,metric,log,alert,metrics-store}.md, blog post "Announcing Dashboards" (0.9.3) Unlicensed public documentation: facts used, prose not copied Retention defaults, alert semantics, the live endpoints' parameters, which parts are Enterprise
ihasmail's admin dashboard (web/src/lib/admin/adminDashboard.ts, adminAccess.ts) AGPL-3.0, ours x:Metric query behavior as observed against INBUXA's live Enterprise server when ihasmail was built
RFC 8620, RFC 5322, RFC 3834 IETF JMAP semantics; the alert message format

No Enterprise-only file or snippet was used. The drafting session never saw Enterprise code. It writes specs only. Gaps are marked Decision or listed under "Open questions / to observe", never filled from memory of upstream code.

What it is

Three things an operator uses to see what the server is doing:

  • History. The server keeps a record of every message delivery (a trace: the inbound SMTP session or the outbound delivery attempt, with its events) and a periodic sample of its metrics. Both are kept for a set period, searchable, and drawn by the dashboards.
  • Live telemetry. An administrator watches events and metrics as they happen, filtered, in INBUXA Admin.
  • Alerts. Rules over metrics that send an email, raise an event (which a webhook can forward), or both, when a threshold is crossed.

Upstream ships these only in its Enterprise Edition. inbuxa-server ships them to everybody. Webhooks, the log tracers, OpenTelemetry and Prometheus export and the x:Log view are already AGPL and aren't part of this rebuild, except where noted.

Data model

Registry objects are unchanged from upstream, so INBUXA's existing settings open as they are (SPEC.md §7). The upstream schema flags as Enterprise: objects x:Alert, x:MetricsStore, x:Trace, x:TracingStore; fields x:DataRetention.holdMetricsFor, holdTracesFor, metricsCollectionInterval, x:Search.indexTelemetry, indexTracingFields. In inbuxa-server they're ordinary.

Settings

Where Field Default Meaning
x:TracingStore (singleton) @type schema Disabled; first boot inserts Default Disabled: no trace history. Default: the data store. FoundationDb, PostgreSql, MySql: a store of its own
x:MetricsStore (singleton) @type as above The same, for metric history
x:DataRetention holdTracesFor 30 days Duration, nullable. How long a trace is kept
holdMetricsFor 90 days Duration, nullable. How long a metric sample is kept
metricsCollectionInterval hourly, minute 0 x:Cron: Hourly, Daily or Weekly. When metric history is sampled
x:Search indexTelemetry true Whether traces are added to the search index
indexTracingFields eventType, queueId, keywords Which trace fields are indexed
x:Metrics metrics, metricsPolicy all (exclude, empty list) Which metrics are collected. Already AGPL, shared with the exporters

Clean-up of expired history runs on the existing dataCleanupSchedule.

The alert, x:Alert

Field Meaning
enable Boolean, default true
condition x:Expression (match list and else). The alert fires when it evaluates true
eventAlert Disabled, or Enabled with eventMessage (text, nullable)
emailAlert Disabled, or Enabled with fromName (nullable), fromAddress, to (a set of one or more addresses), subject, body

The trace, x:Trace (read-only)

events: a list of x:TraceEvent, each with event (an EventType), timestamp, and keyValues: a list of key (a Key) and value (a TraceValue: String, UnsignedInt, Integer, Boolean, Float, UTCDateTime, Duration, IpAddr, List, Event, Null). Server-set: timestamp, from, to (a string), size. Views: x:Trace/InboundDelivery and x:Trace/OutboundDelivery.

The metric sample, x:Metric (read-only)

Three variants by @type: Counter and Gauge (metric, count, timestamp) and Histogram (metric, count, sum, timestamp). metric is a MetricType (369 values at v0.16.22).

The index task, x:TaskIndexTrace

traceId, status, due. Task type IndexTrace, permission taskIndexTrace. Store maintenance type reindexTelemetry rebuilds the trace index. Both already exist in AGPL code.

Permissions

sysAlert{Get,Query,Create,Update,Destroy}, sysTrace{Get,Query,Create,Update,Destroy}, sysMetric{Get,Query,Create,Update,Destroy}, sysTracingStore{Get,Update}, sysMetricsStore{Get,Update}, sysDataRetention{Get,Update}, liveTracing, liveMetrics, taskIndexTrace. The AGPL default-permission table gives all of these to the superuser set only: not to tenant administrators and not to users (crates/common/src/auth/permissions.rs).

Storage

The AGPL store layout is kept: traces in subspace o (SUBSPACE_TELEMETRY_SPAN), metric samples in subspace x (SUBSPACE_TELEMETRY_METRIC), each keyed by a big-endian u64 id. A trace's id is its span id, which the server already assigns from its snowflake generator (milliseconds since an epoch in the high bits), so key order is time order. The trace search index is SearchIndex::Tracing, with fields EventType, QueueId and Keywords.

Existing data at cutover, Decision. The value encoding of stored traces and samples lived in files the strip removed (telemetry/tracers/store.rs, telemetry/metrics/store.rs, trc/src/serializers/binary.rs), so this spec doesn't know it and doesn't try to match it. inbuxa-server writes its own encoding, which starts with a format byte of its own. Records it can't decode are skipped, never an error, and are removed by the normal age purge (MON-17), so INBUXA's old history ages out within 30 and 90 days. At cutover, run reindexTelemetry once so the search index holds only readable traces. Settings, alerts and every other registry object open unchanged.

Required behavior

Switches and defaults

  • MON-1. History is on when its store isn't Disabled. The fork's first boot inserts TracingStore::Default and MetricsStore::Default when none exists (manager/defaults.rs, already in place), so a new install records trace and metric history in its data store from the start, kept 30 and 90 days. An upgraded install keeps whatever it has.
  • MON-2. A store that can't be opened (a separate PostgreSQL that's down, or a backend not compiled in) turns that history off with a build warning, as the AGPL store builder already does. Startup continues and mail flows.
  • MON-3. A change to a store setting takes effect on the next settings reload, with no restart, like every other setting. Decision, in line with undelete UD-6a: a change to the DataRetention fields in this spec or to an x:Alert takes effect without a reload, since an operator who lowers retention or disables a noisy alert expects it at once.

Metric history

  • MON-4. At each metricsCollectionInterval tick, every node writes one sample per metric selected by x:Metrics.metrics and metricsPolicy:
    • Counter: count is the increase since the node's previous sample. Counters in memory are totals since the process started, so the node keeps the last totals it wrote. The first sample after a start counts from the start. A counter that didn't move writes nothing.
    • Gauge: count is the reading at the tick.
    • Histogram: count and sum are the increases since the previous sample. A histogram that saw nothing writes nothing. All samples of one tick share its timestamp. That's the shape ihasmail already reads from INBUXA: a counter holds what happened in the interval, a gauge the reading at its end.
  • MON-5. Gauges are always written, even when unchanged, so a window with no samples at all means history is off, not a quiet period. ihasmail relies on this.
  • MON-6. There's one edition, so every gauge and histogram the collector has is collected, stored and exported: server.memory, queue.count, user.count, domain.count, the active-connection gauges, and all eleven histograms, including ingest, index, store read and write, and DNS lookup times. The is_enterprise arguments in trc and common::telemetry go. Prometheus and OpenTelemetry exports gain the same metrics.
  • MON-7. queue.count is set from the queue itself on the existing five-minute metrics calculation, not only moved by queue events, so it's right after a restart. Decision: the in-memory gauge starts at zero on restart, and nothing in the AGPL code corrects it.
  • MON-8. In a cluster, samples carry no node field (the schema has none). Each node writes its own values under the same timestamp, and dashboards aggregate them with the sum or avg their cards name. A one-node install is unaffected.
  • MON-9. After writing a tick's samples the node emits telemetry.metrics-stored.

Trace history

  • MON-10. A trace is stored for each inbound SMTP session (span opened by smtp.connection-start) and each delivery attempt (span opened by delivery.attempt-start, which includes local delivery). Other spans (IMAP, POP3, HTTP, ManageSieve) aren't stored. The upstream suite confirms this: one LMTP delivery, made while the admin was busy over HTTP, left exactly two traces.
  • MON-11. Decision: an inbound session in which no MAIL FROM was accepted or refused (a probe, a scanner, a banned address dropped at connect) isn't stored. These are most of a public server's connections and none is a message delivery. The live view and the log still show them.
  • MON-12. A trace holds the span's events, from the opening event to the closing one, that are at info level or above after x:EventTracingLevel overrides. Decision on the level. It never holds raw I/O events (*.raw-input, *.raw-output, milter read and write). Those carry message content and authentication exchanges.
  • MON-13. The trace is written once, when the span closes. A span still open after one day is dropped, matching the collector's own SPAN_MAX_HOLD.
  • MON-14. Server-set fields: timestamp is the opening event's time; from is the first from value in the trace; to is every distinct to value, comma-separated; size is the message size from the trace's events, or 0. Decision on the derivation. Compare with upstream (to observe, 3).
  • MON-15. Bounds per trace. Decision: at most 1,000 events, then one final marker event noting how many were cut; string values over 4 KiB are truncated. A trace is diagnostic, not an archive.

Search

  • MON-16. With indexTelemetry on, storing a trace schedules an IndexTrace task. The task builds one document for SearchIndex::Tracing with the fields named in indexTracingFields:
    • eventType: every event type in the trace;
    • queueId: every queueId value;
    • keywords: every address in from and to, each address's domain, every domain, hostname, remoteIp, messageId and accountName value. So searching example.org finds every trace to or from that domain, as the upstream suite expects. With indexTelemetry off nothing is indexed, and the text and queueId filters are refused (see "Interfaces").

Retention and growth

  • MON-17. On each dataCleanupSchedule run, traces older than holdTracesFor and samples older than holdMetricsFor are deleted, with their search-index documents. Keys are time-ordered, so this is a range delete. A record past its deadline is never returned, even before clean-up has run.
  • MON-18. A null holdTracesFor or holdMetricsFor means no age limit. Decision: that's what "unset" means for the other retention durations except the archive ones. INBUXA Admin and ihasmail show a warning next to a null value. Confirm upstream's meaning (to observe, 7).
  • MON-19. Growth is bounded by settings, not by a byte cap:
    • Metrics: the smallest interval is hourly, so at most 24 ticks a day, each at most one sample per selected metric per node. At the defaults that's at most 2,160 ticks over 90 days, and in practice a few hundred samples a tick.
    • Traces: one per delivered or attempted message, at most 1,000 events each (MON-15), kept 30 days. Connections that send nothing aren't stored (MON-11). The storage dashboard reports the size of both subspaces, so an operator sees what history costs.

Live telemetry

  • MON-20. Live tracing. GET /api/live/tracing returns a text/event-stream. Each frame is event: event and data: a JSON array of events, in the format webhooks already send (id, createdAt, type, data), the same framing the AGPL delivery tester uses. Query parameters: filter matches a value in any key; any Key name (for example remoteIp, domain, queueId) matches that key only; several combine with AND. Decision: keys are given in their camel-case Key names; the hyphenated names in upstream's docs (remote-ip) are accepted too.
  • MON-21. The live tracing stream never carries raw I/O events (MON-12). Decision: raw protocol lines include credentials, and anyone who needs them can set a file tracer at trace level on the host.
  • MON-22. Live metrics. GET /api/live/metrics returns a text/event-stream of the current values of the metrics listed in metrics (comma-separated names; all selected metrics when absent), every interval seconds (default 30, minimum 1). Each frame's data is a JSON array of {"id", "type", "value"} for counters and gauges, and {"id", "type", "count", "sum"} for histograms. Decision on the frame shape; check it against INBUXA Admin (to observe, 9).
  • MON-23. Tokens. Browsers can't put headers on an event stream, so as with the delivery tester, GET /api/token/tracing and /api/token/metrics return a single-use token, valid 60 seconds, bound to the account and to grant type live_tracing or live_metrics (both already defined). The stream accepts it as ?token=, or a normal Authorization header. Issuing the token needs liveTracing or liveMetrics, and on this fork a token with the inbuxa:admin scope (contract.md C-18).
  • MON-24. A live subscriber is lossy: a slow client loses events, never slows the server. At most 8 live streams run at once per node, and each ends after 30 minutes, when the client reconnects with a fresh token. Decision on both numbers. The time limit means a revoked grant (contract.md C-12) or a removed permission stops a stream within 30 minutes.

Alerts

  • MON-25. Evaluation. Each enabled alert's condition is evaluated every 60 seconds (Decision) on every node that holds the metricsCalculate task role, against that node's current values. A one-node install always holds it. The condition language is the server's expression language, and a metric is read two ways, both accepted:
    • metric('queue.count'), which the AGPL expression code already supports;
    • the name with dots and hyphens as underscores (queue_count), as upstream's docs describe. Counter values in a condition are totals since the process started, as metric() already reads them. Gauges are the current reading, histograms their average.
  • MON-26. Firing. An alert fires when its condition goes from false to true, including the first evaluation after start. While it stays true it doesn't fire again. Once false, it can fire again. Decision: upstream's docs say email goes out "each time the condition becomes true". State is in memory, so a restart while the condition holds fires once more. A condition that fails to evaluate (a metric name that doesn't exist, a type error) is rejected when the alert is saved, and never fires.
  • MON-27. Placeholders. In eventMessage, subject and body, %{metric.name}% (the dotted name) is replaced by the value used in that evaluation. Whole numbers print without decimals ("3", not "3.0"); others with at most two. An unknown name is left as written.
  • MON-28. Event notification. With eventAlert enabled, firing emits telemetry.alert-event (level warn), with details set to the rendered message and the alert's id. Webhooks subscribed to that event forward it.
  • MON-29. Email notification. With emailAlert enabled, firing queues one message to every address in to, through the normal outbound queue, so it's retried, DKIM-signed for a local sender domain, and visible in the queue like any other. Headers: From: "fromName" <fromAddress> (bare address when fromName is null), To, Subject, Date, Message-ID, and Auto-Submitted: auto-generated (RFC 3834). Body: text/plain, UTF-8. The server emits telemetry.alert-message once it's queued.
  • MON-30. An alert email is sent even when the queue itself is the problem the alert reports. It's queued like any message. If queueing fails, the event notification (MON-28) still happens and the failure is logged.

Who may see what

  • MON-31. Traces, metric samples, alerts, the two store settings and live telemetry are server-level. By default only the superuser permission set holds their permissions (see "Permissions"). A tenant administrator gets forbidden for all of them, even if a role grants the permission, unless its tenant allows it (multi-tenancy MT-13). Decision: traces carry other tenants' addresses and IP addresses, and metrics describe the whole server. A tenant-scoped trace view is a possible later addition (open question 11).
  • MON-32. x:Trace and x:Metric can't be created or updated, as the AGPL registry already enforces. Decision, an addition: a trace can be destroyed with sysTraceDestroy, so an operator can honor a request to erase someone's delivery records before they age out. Samples can't be destroyed one by one.
  • MON-33. Traces are personal data: they hold addresses, IP addresses, host names and message ids, never message content (MON-12). Retention (MON-17) is the main control, and INBUXA Admin's retention form says what's kept. Nothing in this feature sends traces off the host. Webhooks and OpenTelemetry send only what an operator configures.

Failure behavior: telemetry never blocks mail

  • MON-34. Trace and sample writes happen off the mail path, through a lossy collector subscriber with a bounded buffer. When the buffer is full, events are dropped and counted, and one error event is logged per minute at most. No SMTP, IMAP, JMAP or delivery step ever waits on history.
  • MON-35. A failing tracing or metrics store (full, unreachable, slow) loses history and logs, rate-limited. It never fails a delivery, a login or startup.
  • MON-36. A failed IndexTrace task is retried by the task manager as now. The trace stays readable by id and by date meanwhile.
  • MON-37. A failed alert evaluation or notification is logged and retried on the next evaluation. It never stops the other alerts.
  • MON-38. A failed clean-up leaves the records for the next run. Expired records are still hidden (MON-17).

Edition cleanup

  • MON-39. Remove the is_enterprise and _is_enterprise parameters and their inbuxa: signposts in telemetry/mod.rs and config/telemetry.rs; Tracers::parse uses storage again. Drop Metric and Trace from assert_enterprise_object. Replace the "Enterprise feature" stubs for /api/token/{tracing,metrics} and /api/live/{tracing,metrics}, and the cfg(not(feature = "enterprise")) branch in management_access_token.

Interfaces

  • JMAP, existing names, unchanged. Over urn:stalwart:jmap:
    • x:Alert/get, /query, /set, and the x:TracingStore, x:MetricsStore, x:DataRetention, x:Search singletons.
    • x:Trace/get and /query. Filters: text, timestamp (comparison names such as timestampIsGreaterThan), queueId, event (the type of the trace's opening event: the list views use smtp.connection-start and delivery.attempt-start). Sort by timestamp, newest first by default. /set refuses create and update, and allows destroy (MON-32).
    • x:Metric/get and /query. Filters: metric (one name or a list) and the timestamp comparisons timestampIsGreaterThan, timestampIsGreaterThanOrEqual, timestampIsLessThan, timestampIsLessThanOrEqual. A bare timestamp filter is unsupportedFilter, as upstream. Sort by timestamp either way, with position, anchor and anchorOffset paging and calculateTotal.
    • x:Trace text or queueId filters when indexTelemetry is off: unsupportedFilter, with a description saying trace search is off.
  • HTTP. /api/token/tracing, /api/token/metrics, /api/live/tracing, /api/live/metrics as MON-20 to MON-24. These are the route names in the AGPL code. Upstream's docs name /api/telemetry/traces/live and /api/telemetry/metrics/live. Decision: serve those as aliases too.
  • Webhooks, unchanged. Already AGPL, and intact after the strip: the webhook tracer is in neither the removed-file list nor the snippet list, and upstream's docs don't mark webhooks as Enterprise. POST of {"events": [...]}, HMAC-SHA256 in X-Signature, batched by throttle, stale events dropped after discardAfter.
  • Dashboards. The schema's six dashboards (Overview, Network, Security, Delivery, Performance, Storage) name, per card, live (MON-22) or history (x:Metric) and an aggregate. The server serves what they name. INBUXA Admin draws them unchanged.

ihasmail changes

These go in ihasmail-inbuxa, not public ihasmail, which stays Stalwart-facing (SPEC.md §5).

  • The dashboard already reads x:Metric for received, sent and memory. Keep it. Drop the Enterprise wording from its comments and code paths, and the "refused as forbidden" branch becomes a plain error.
  • Change the footer line that sends people to "Stalwart's own administration" to name INBUXA Admin, with its link.
  • Warn on the dashboard when metric history is off (no samples in the window, MON-5), with a link to INBUXA Admin's Metrics Store page.
  • Live tracing, trace history and alerts stay in INBUXA Admin (SPEC.md §5.4). ihasmail doesn't grow screens for them.
  • contract.md C-19 has to list x:Metric (get and query) among the object types the inbuxa:account-admin scope reaches, or the dashboard loses its message cards.
  • Translation work: two changed strings (the footer line and the error that replaces the refused case) and one new one (the history-off warning), each in ihasmail's nine languages.

Acceptance tests

Every test runs against inbuxa-server built with no Enterprise code.

  1. New install: TracingStore and MetricsStore read Default, retention 30 and 90 days, hourly collection (MON-1).
  2. Tracing store set to an unreachable PostgreSQL: the server starts, and mail is delivered (MON-2, MON-35).
  3. Two collection ticks with traffic between: counters hold the increase, gauges the reading, idle counters write nothing (MON-4, MON-5).
  4. queue.count is right after a restart with mail queued (MON-7).
  5. Prometheus output includes queue_count and the store and DNS histograms (MON-6).
  6. One LMTP delivery, with HTTP traffic alongside: exactly two traces, one smtp.connection-start, one delivery.attempt-start (MON-10).
  7. An SMTP connection that quits without MAIL FROM: no trace (MON-11).
  8. A trace holds no raw I/O event, and nothing below info (MON-12).
  9. from, to, size and timestamp set as MON-14.
  10. Text search for the sender, the recipient and their domain each finds both traces (MON-16).
  11. indexTelemetry off: text filter is unsupportedFilter, timestamp still works (MON-16, Interfaces).
  12. Purge with retention 1 second: all traces and their index entries gone. With 2 seconds, nothing gone yet (MON-17).
  13. Metric query with timestampIsGreaterThan, and paging forward, backward and by anchor, returns consistent pages (Interfaces).
  14. Live tracing with ?remoteIp= shows only that client's events, and no raw I/O (MON-20, MON-21).
  15. Live metrics with metrics=server.memory&interval=1 yields a frame a second (MON-22).
  16. Live token: expires after 60 seconds, works once, refused without liveTracing (MON-23).
  17. A ninth live stream is refused. A stream closes after 30 minutes (MON-24).
  18. Alert on metric('domain.count') > 1 && metric('cluster.publisher-error') > 3 with both conditions met: one email with the placeholders filled ("3 domains and 5 cluster errors"), From: "Alert Subsystem" <[email protected]>, one telemetry.alert-event. The opposite condition fires nothing (MON-25 to MON-29).
  19. The same alert with an underscore condition (domain_count > 1) fires the same way (MON-25).
  20. The alert fires once while the condition stays true, and again after it has been false (MON-26).
  21. An alert with an unknown metric name is refused on save (MON-26).
  22. A webhook subscribed to telemetry.alert-event receives the alert (MON-28).
  23. A tenant administrator with sysTraceGet in a role, in a tenant that doesn't allow it: forbidden (MON-31).
  24. x:Trace/set destroy by a superuser removes the trace and its index entry. Create and update are refused (MON-32).
  25. A full tracing store buffer drops events without delaying an SMTP session (MON-34).
  26. (compat) A copy of INBUXA's data opens: alerts, store settings and retention read back unchanged. Old traces and samples it can't decode are skipped, and are gone after one purge past their age (Data model).

The gated integration suites

tests/src/telemetry/mod.rs gates four suites behind pending-rebuild:

Suite Needs rebuilding Why it's gated
alerts.rs Yes It calls process_alerts(), which lived in the removed enterprise/alerts.rs. The rebuild provides a function of that shape (returns the messages it would send) so the suite runs unchanged
metrics.rs Yes It needs the metrics store's purge_metrics and the test-data generator insert_test_metrics, from the removed metrics/store.rs and metrics/test_data.rs (the latter is the dangling test_data module in metrics/mod.rs)
tracing.rs Yes It needs the tracing store's purge_spans, from the removed tracers/store.rs
webhooks.rs No Webhooks are AGPL and intact. The suite is gated only because its clean-up calls purge_spans through the shared harness. Dropping or guarding that one call lets it run now, before the rebuild

Recommended: un-gate webhooks.rs straight away (a separate change, not made here), so webhooks are tested while the rest is rebuilt.

Open questions / to observe

To check read-only against INBUXA's live Enterprise server later. None of these needs a write.

  1. How many traces a day INBUXA stores against its SMTP connection count: whether upstream stores connection-only sessions (MON-11).
  2. Which events and levels a stored trace holds, and whether raw I/O ever appears (MON-12, MON-15).
  3. How upstream fills from, to and size on a trace with several recipients (MON-14).
  4. Whether INBUXA's samples include Histogram records, which gauges appear every tick, and whether counter samples are per-interval increases (MON-4 to MON-6). ihasmail's code says they are.
  5. Whether queue.count in INBUXA's history matches the real queue after a restart (MON-7).
  6. INBUXA's current holdTracesFor, holdMetricsFor, metricsCollectionInterval, indexTelemetry and store settings, so the cutover keeps them.
  7. What a null holdTracesFor or holdMetricsFor does upstream: keep forever, or store nothing (MON-18).
  8. The x:Trace/query filters upstream accepts beyond text, timestamp, queueId and event, and the x:Metric/query comparison names.
  9. The live endpoints: the token response body, the SSE frame shape for tracing and metrics, keep-alive comments, and which path INBUXA Admin calls (MON-20 to MON-23). A GET of each is read-only.
  10. Whether INBUXA has any x:Alert objects, and whether their conditions use metric() or underscore names (MON-25). Whether INBUXA's logs show telemetry.alert-event repeating while a condition held, which settles the cadence and repeat behavior (MON-25, MON-26).
  11. Whether upstream's tenant Admin role holds any sysTrace*, sysMetric* or live permission, and whether a tenant view of its own domains' traces is wanted (MON-31).
  12. Size of INBUXA's o and x subspaces, to check MON-19's estimate.
Reference in New Issue View Git Blame Copy Permalink