Feature specs 4-9: branding and templates, AI spam classification, monitoring, SCIM, scale-out storage, per-domain directories
This commit is contained in:
@@ -0,0 +1,510 @@
|
||||
# Feature spec: monitoring history, live tracing and alerts
|
||||
|
||||
Status: draft, 2026-09-18. Feature 6 in SPEC.md §4.
|
||||
|
||||
## Provenance
|
||||
|
||||
Written for the clean room (SPEC.md §3). Sources, and nothing else:
|
||||
|
||||
| Source | License | Used for |
|
||||
|---|---|---|
|
||||
| Stalwart's registry schema: `crates/registry/src/schema/*.rs` and `resources/schema/schema.json.gz` (objects, fields, defaults, enums, permissions, lists, forms, the `dashboards` and `layouts` sections), upstream `v0.16.22` | AGPL-3.0-only OR LicenseRef-SEL, taken under the AGPL | Object shapes, field meanings, defaults, what upstream flags as Enterprise, the admin views and dashboards the server has to feed |
|
||||
| The AGPL telemetry code left after the strip: `crates/common/src/telemetry/`, `crates/common/src/config/telemetry.rs`, `crates/trc` (event, key and metric definitions, the collector, span tracking, the JSON serializer), `crates/http/src/api/mod.rs` and `diagnose.rs`, `crates/store` key layout, `crates/services/src/task_manager/`, `crates/common/src/auth/permissions.rs`, `crates/common/src/expr/`, `crates/common/src/manager/defaults.rs`, `crates/jmap/src/registry/` | AGPL-3.0-only OR LicenseRef-SEL, taken under the AGPL | What already exists and must be kept: webhooks, exporters, the `metric()` expression function, span ids, storage subspaces, route and token names, default permissions |
|
||||
| The integration suites `tests/src/telemetry/*.rs` | AGPL-3.0-only OR LicenseRef-SEL, taken under the AGPL | Tested behavior of alerts, metrics history, trace history and webhooks |
|
||||
| The strip report, `docs/fork/strip-reports/v0.16.22.json` | This repository | Which files and snippets were removed, so which behavior is missing |
|
||||
| Stalwart documentation (`website` repo): `docs/telemetry/{index,alerts,history,live,management,webhooks}.md`, `docs/0.15/telemetry/{alerts,live,history}.md`, `docs/ref/object/{trace,metric,log,alert,metrics-store}.md`, blog post "Announcing Dashboards" (0.9.3) | Unlicensed public documentation: facts used, prose not copied | Retention defaults, alert semantics, the live endpoints' parameters, which parts are Enterprise |
|
||||
| ihasmail's admin dashboard (`web/src/lib/admin/adminDashboard.ts`, `adminAccess.ts`) | AGPL-3.0, ours | `x:Metric` query behavior as observed against INBUXA's live Enterprise server when ihasmail was built |
|
||||
| RFC 8620, RFC 5322, RFC 3834 | IETF | JMAP semantics; the alert message format |
|
||||
|
||||
No Enterprise-only file or snippet was used. The drafting session never saw
|
||||
Enterprise code. It writes specs only. Gaps are marked **Decision** or listed
|
||||
under "Open questions / to observe", never filled from memory of upstream code.
|
||||
|
||||
## What it is
|
||||
|
||||
Three things an operator uses to see what the server is doing:
|
||||
|
||||
- **History.** The server keeps a record of every message delivery (a
|
||||
*trace*: the inbound SMTP session or the outbound delivery attempt, with its
|
||||
events) and a periodic sample of its metrics. Both are kept for a set
|
||||
period, searchable, and drawn by the dashboards.
|
||||
- **Live telemetry.** An administrator watches events and metrics as they
|
||||
happen, filtered, in INBUXA Admin.
|
||||
- **Alerts.** Rules over metrics that send an email, raise an event (which a
|
||||
webhook can forward), or both, when a threshold is crossed.
|
||||
|
||||
Upstream ships these only in its Enterprise Edition. inbuxa-server ships them
|
||||
to everybody. Webhooks, the log tracers, OpenTelemetry and Prometheus export
|
||||
and the `x:Log` view are already AGPL and aren't part of this rebuild, except
|
||||
where noted.
|
||||
|
||||
## Data model
|
||||
|
||||
Registry objects are unchanged from upstream, so INBUXA's existing settings
|
||||
open as they are (SPEC.md §7). The upstream schema flags as Enterprise:
|
||||
objects `x:Alert`, `x:MetricsStore`, `x:Trace`, `x:TracingStore`; fields
|
||||
`x:DataRetention.holdMetricsFor`, `holdTracesFor`, `metricsCollectionInterval`,
|
||||
`x:Search.indexTelemetry`, `indexTracingFields`. In inbuxa-server they're
|
||||
ordinary.
|
||||
|
||||
### Settings
|
||||
|
||||
| Where | Field | Default | Meaning |
|
||||
|---|---|---|---|
|
||||
| `x:TracingStore` (singleton) | `@type` | schema `Disabled`; first boot inserts `Default` | `Disabled`: no trace history. `Default`: the data store. `FoundationDb`, `PostgreSql`, `MySql`: a store of its own |
|
||||
| `x:MetricsStore` (singleton) | `@type` | as above | The same, for metric history |
|
||||
| `x:DataRetention` | `holdTracesFor` | 30 days | Duration, nullable. How long a trace is kept |
|
||||
| | `holdMetricsFor` | 90 days | Duration, nullable. How long a metric sample is kept |
|
||||
| | `metricsCollectionInterval` | hourly, minute 0 | `x:Cron`: `Hourly`, `Daily` or `Weekly`. When metric history is sampled |
|
||||
| `x:Search` | `indexTelemetry` | true | Whether traces are added to the search index |
|
||||
| | `indexTracingFields` | `eventType`, `queueId`, `keywords` | Which trace fields are indexed |
|
||||
| `x:Metrics` | `metrics`, `metricsPolicy` | all (`exclude`, empty list) | Which metrics are collected. Already AGPL, shared with the exporters |
|
||||
|
||||
Clean-up of expired history runs on the existing `dataCleanupSchedule`.
|
||||
|
||||
### The alert, `x:Alert`
|
||||
|
||||
| Field | Meaning |
|
||||
|---|---|
|
||||
| `enable` | Boolean, default true |
|
||||
| `condition` | `x:Expression` (`match` list and `else`). The alert fires when it evaluates true |
|
||||
| `eventAlert` | `Disabled`, or `Enabled` with `eventMessage` (text, nullable) |
|
||||
| `emailAlert` | `Disabled`, or `Enabled` with `fromName` (nullable), `fromAddress`, `to` (a set of one or more addresses), `subject`, `body` |
|
||||
|
||||
### The trace, `x:Trace` (read-only)
|
||||
|
||||
`events`: a list of `x:TraceEvent`, each with `event` (an `EventType`),
|
||||
`timestamp`, and `keyValues`: a list of `key` (a `Key`) and `value` (a
|
||||
`TraceValue`: `String`, `UnsignedInt`, `Integer`, `Boolean`, `Float`,
|
||||
`UTCDateTime`, `Duration`, `IpAddr`, `List`, `Event`, `Null`). Server-set:
|
||||
`timestamp`, `from`, `to` (a string), `size`. Views: `x:Trace/InboundDelivery`
|
||||
and `x:Trace/OutboundDelivery`.
|
||||
|
||||
### The metric sample, `x:Metric` (read-only)
|
||||
|
||||
Three variants by `@type`: `Counter` and `Gauge` (`metric`, `count`,
|
||||
`timestamp`) and `Histogram` (`metric`, `count`, `sum`, `timestamp`).
|
||||
`metric` is a `MetricType` (369 values at `v0.16.22`).
|
||||
|
||||
### The index task, `x:TaskIndexTrace`
|
||||
|
||||
`traceId`, `status`, `due`. Task type `IndexTrace`, permission
|
||||
`taskIndexTrace`. Store maintenance type `reindexTelemetry` rebuilds the
|
||||
trace index. Both already exist in AGPL code.
|
||||
|
||||
### Permissions
|
||||
|
||||
`sysAlert{Get,Query,Create,Update,Destroy}`, `sysTrace{Get,Query,Create,Update,Destroy}`,
|
||||
`sysMetric{Get,Query,Create,Update,Destroy}`, `sysTracingStore{Get,Update}`,
|
||||
`sysMetricsStore{Get,Update}`, `sysDataRetention{Get,Update}`, `liveTracing`,
|
||||
`liveMetrics`, `taskIndexTrace`. The AGPL default-permission table gives all
|
||||
of these to the superuser set only: not to tenant administrators and not to
|
||||
users (`crates/common/src/auth/permissions.rs`).
|
||||
|
||||
### Storage
|
||||
|
||||
The AGPL store layout is kept: traces in subspace `o`
|
||||
(`SUBSPACE_TELEMETRY_SPAN`), metric samples in subspace `x`
|
||||
(`SUBSPACE_TELEMETRY_METRIC`), each keyed by a big-endian `u64` id. A trace's
|
||||
id is its span id, which the server already assigns from its snowflake
|
||||
generator (milliseconds since an epoch in the high bits), so key order is time
|
||||
order. The trace search index is `SearchIndex::Tracing`, with fields
|
||||
`EventType`, `QueueId` and `Keywords`.
|
||||
|
||||
**Existing data at cutover, Decision.** The *value* encoding of stored traces
|
||||
and samples lived in files the strip removed (`telemetry/tracers/store.rs`,
|
||||
`telemetry/metrics/store.rs`, `trc/src/serializers/binary.rs`), so this spec
|
||||
doesn't know it and doesn't try to match it. inbuxa-server writes its own
|
||||
encoding, which starts with a format byte of its own. Records it can't decode
|
||||
are skipped, never an error, and are removed by the normal age purge (MON-17),
|
||||
so INBUXA's old history ages out within 30 and 90 days. At cutover, run
|
||||
`reindexTelemetry` once so the search index holds only readable traces.
|
||||
Settings, alerts and every other registry object open unchanged.
|
||||
|
||||
## Required behavior
|
||||
|
||||
### Switches and defaults
|
||||
|
||||
- **MON-1.** History is on when its store isn't `Disabled`. The fork's first
|
||||
boot inserts `TracingStore::Default` and `MetricsStore::Default` when none
|
||||
exists (`manager/defaults.rs`, already in place), so a new install records
|
||||
trace and metric history in its data store from the start, kept 30 and 90
|
||||
days. An upgraded install keeps whatever it has.
|
||||
- **MON-2.** A store that can't be opened (a separate PostgreSQL that's down,
|
||||
or a backend not compiled in) turns that history off with a build warning,
|
||||
as the AGPL store builder already does. Startup continues and mail flows.
|
||||
- **MON-3.** A change to a store setting takes effect on the next settings
|
||||
reload, with no restart, like every other setting. **Decision**, in line
|
||||
with undelete UD-6a: a change to the `DataRetention` fields in this spec or
|
||||
to an `x:Alert` takes effect without a reload, since an operator who lowers
|
||||
retention or disables a noisy alert expects it at once.
|
||||
|
||||
### Metric history
|
||||
|
||||
- **MON-4.** At each `metricsCollectionInterval` tick, every node writes one
|
||||
sample per metric selected by `x:Metrics.metrics` and `metricsPolicy`:
|
||||
- **Counter:** `count` is the increase since the node's previous sample.
|
||||
Counters in memory are totals since the process started, so the node keeps
|
||||
the last totals it wrote. The first sample after a start counts from the
|
||||
start. A counter that didn't move writes nothing.
|
||||
- **Gauge:** `count` is the reading at the tick.
|
||||
- **Histogram:** `count` and `sum` are the increases since the previous
|
||||
sample. A histogram that saw nothing writes nothing.
|
||||
All samples of one tick share its `timestamp`. That's the shape ihasmail
|
||||
already reads from INBUXA: a counter holds what happened in the interval, a
|
||||
gauge the reading at its end.
|
||||
- **MON-5.** Gauges are always written, even when unchanged, so a window with
|
||||
no samples at all means history is off, not a quiet period. ihasmail relies
|
||||
on this.
|
||||
- **MON-6.** There's one edition, so every gauge and histogram the collector
|
||||
has is collected, stored and exported: `server.memory`, `queue.count`,
|
||||
`user.count`, `domain.count`, the active-connection gauges, and all eleven
|
||||
histograms, including ingest, index, store read and write, and DNS lookup
|
||||
times. The `is_enterprise` arguments in `trc` and `common::telemetry` go.
|
||||
Prometheus and OpenTelemetry exports gain the same metrics.
|
||||
- **MON-7.** `queue.count` is set from the queue itself on the existing
|
||||
five-minute metrics calculation, not only moved by queue events, so it's
|
||||
right after a restart. **Decision**: the in-memory gauge starts at zero on
|
||||
restart, and nothing in the AGPL code corrects it.
|
||||
- **MON-8.** In a cluster, samples carry no node field (the schema has none).
|
||||
Each node writes its own values under the same timestamp, and dashboards
|
||||
aggregate them with the `sum` or `avg` their cards name. A one-node install
|
||||
is unaffected.
|
||||
- **MON-9.** After writing a tick's samples the node emits
|
||||
`telemetry.metrics-stored`.
|
||||
|
||||
### Trace history
|
||||
|
||||
- **MON-10.** A trace is stored for each **inbound SMTP session** (span opened
|
||||
by `smtp.connection-start`) and each **delivery attempt** (span opened by
|
||||
`delivery.attempt-start`, which includes local delivery). Other spans
|
||||
(IMAP, POP3, HTTP, ManageSieve) aren't stored. The upstream suite confirms
|
||||
this: one LMTP delivery, made while the admin was busy over HTTP, left
|
||||
exactly two traces.
|
||||
- **MON-11.** **Decision**: an inbound session in which no `MAIL FROM` was
|
||||
accepted or refused (a probe, a scanner, a banned address dropped at
|
||||
connect) isn't stored. These are most of a public server's connections and
|
||||
none is a message delivery. The live view and the log still show them.
|
||||
- **MON-12.** A trace holds the span's events, from the opening event to the
|
||||
closing one, that are at `info` level or above after `x:EventTracingLevel`
|
||||
overrides. **Decision** on the level. It never holds raw I/O events
|
||||
(`*.raw-input`, `*.raw-output`, milter read and write). Those carry message
|
||||
content and authentication exchanges.
|
||||
- **MON-13.** The trace is written once, when the span closes. A span still
|
||||
open after one day is dropped, matching the collector's own `SPAN_MAX_HOLD`.
|
||||
- **MON-14.** Server-set fields: `timestamp` is the opening event's time;
|
||||
`from` is the first `from` value in the trace; `to` is every distinct `to`
|
||||
value, comma-separated; `size` is the message size from the trace's events,
|
||||
or 0. **Decision** on the derivation. Compare with upstream (to observe, 3).
|
||||
- **MON-15.** Bounds per trace. **Decision**: at most 1,000 events, then one
|
||||
final marker event noting how many were cut; string values over 4 KiB are
|
||||
truncated. A trace is diagnostic, not an archive.
|
||||
|
||||
### Search
|
||||
|
||||
- **MON-16.** With `indexTelemetry` on, storing a trace schedules an
|
||||
`IndexTrace` task. The task builds one document for `SearchIndex::Tracing`
|
||||
with the fields named in `indexTracingFields`:
|
||||
- `eventType`: every event type in the trace;
|
||||
- `queueId`: every `queueId` value;
|
||||
- `keywords`: every address in `from` and `to`, each address's domain, every
|
||||
`domain`, `hostname`, `remoteIp`, `messageId` and `accountName` value.
|
||||
So searching `example.org` finds every trace to or from that domain, as the
|
||||
upstream suite expects. With `indexTelemetry` off nothing is indexed, and
|
||||
the `text` and `queueId` filters are refused (see "Interfaces").
|
||||
|
||||
### Retention and growth
|
||||
|
||||
- **MON-17.** On each `dataCleanupSchedule` run, traces older than
|
||||
`holdTracesFor` and samples older than `holdMetricsFor` are deleted, with
|
||||
their search-index documents. Keys are time-ordered, so this is a range
|
||||
delete. A record past its deadline is never returned, even before clean-up
|
||||
has run.
|
||||
- **MON-18.** A null `holdTracesFor` or `holdMetricsFor` means no age limit.
|
||||
**Decision**: that's what "unset" means for the other retention durations
|
||||
except the archive ones. INBUXA Admin and ihasmail show a warning next to a
|
||||
null value. Confirm upstream's meaning (to observe, 7).
|
||||
- **MON-19.** Growth is bounded by settings, not by a byte cap:
|
||||
- Metrics: the smallest interval is hourly, so at most 24 ticks a day, each
|
||||
at most one sample per selected metric per node. At the defaults that's
|
||||
at most 2,160 ticks over 90 days, and in practice a few hundred samples a
|
||||
tick.
|
||||
- Traces: one per delivered or attempted message, at most 1,000 events each
|
||||
(MON-15), kept 30 days. Connections that send nothing aren't stored
|
||||
(MON-11).
|
||||
The storage dashboard reports the size of both subspaces, so an operator
|
||||
sees what history costs.
|
||||
|
||||
### Live telemetry
|
||||
|
||||
- **MON-20.** **Live tracing.** `GET /api/live/tracing` returns a
|
||||
`text/event-stream`. Each frame is `event: event` and `data:` a JSON array
|
||||
of events, in the format webhooks already send (`id`, `createdAt`, `type`,
|
||||
`data`), the same framing the AGPL delivery tester uses. Query parameters:
|
||||
`filter` matches a value in any key; any `Key` name (for example
|
||||
`remoteIp`, `domain`, `queueId`) matches that key only; several combine with
|
||||
AND. **Decision**: keys are given in their camel-case `Key` names; the
|
||||
hyphenated names in upstream's docs (`remote-ip`) are accepted too.
|
||||
- **MON-21.** The live tracing stream never carries raw I/O events
|
||||
(MON-12). **Decision**: raw protocol lines include credentials, and anyone
|
||||
who needs them can set a file tracer at `trace` level on the host.
|
||||
- **MON-22.** **Live metrics.** `GET /api/live/metrics` returns a
|
||||
`text/event-stream` of the current values of the metrics listed in
|
||||
`metrics` (comma-separated names; all selected metrics when absent), every
|
||||
`interval` seconds (default 30, minimum 1). Each frame's data is a JSON
|
||||
array of `{"id", "type", "value"}` for counters and gauges, and `{"id",
|
||||
"type", "count", "sum"}` for histograms. **Decision** on the frame shape;
|
||||
check it against INBUXA Admin (to observe, 9).
|
||||
- **MON-23.** **Tokens.** Browsers can't put headers on an event stream, so
|
||||
as with the delivery tester, `GET /api/token/tracing` and
|
||||
`/api/token/metrics` return a single-use token, valid 60 seconds, bound to
|
||||
the account and to grant type `live_tracing` or `live_metrics` (both already
|
||||
defined). The stream accepts it as `?token=`, or a normal `Authorization`
|
||||
header. Issuing the token needs `liveTracing` or `liveMetrics`, and on this
|
||||
fork a token with the `inbuxa:admin` scope (contract.md C-18).
|
||||
- **MON-24.** A live subscriber is lossy: a slow client loses events, never
|
||||
slows the server. At most 8 live streams run at once per node, and each ends
|
||||
after 30 minutes, when the client reconnects with a fresh token.
|
||||
**Decision** on both numbers. The time limit means a revoked grant
|
||||
(contract.md C-12) or a removed permission stops a stream within 30 minutes.
|
||||
|
||||
### Alerts
|
||||
|
||||
- **MON-25.** **Evaluation.** Each enabled alert's condition is evaluated
|
||||
every 60 seconds (**Decision**) on every node that holds the
|
||||
`metricsCalculate` task role, against that node's current values. A
|
||||
one-node install always holds it. The condition language is the server's
|
||||
expression language, and a metric is read two ways, both accepted:
|
||||
- `metric('queue.count')`, which the AGPL expression code already supports;
|
||||
- the name with dots and hyphens as underscores (`queue_count`), as
|
||||
upstream's docs describe.
|
||||
Counter values in a condition are totals since the process started, as
|
||||
`metric()` already reads them. Gauges are the current reading, histograms
|
||||
their average.
|
||||
- **MON-26.** **Firing.** An alert fires when its condition goes from false to
|
||||
true, including the first evaluation after start. While it stays true it
|
||||
doesn't fire again. Once false, it can fire again. **Decision**: upstream's
|
||||
docs say email goes out "each time the condition becomes true". State is in
|
||||
memory, so a restart while the condition holds fires once more. A condition
|
||||
that fails to evaluate (a metric name that doesn't exist, a type error) is
|
||||
rejected when the alert is saved, and never fires.
|
||||
- **MON-27.** **Placeholders.** In `eventMessage`, `subject` and `body`,
|
||||
`%{metric.name}%` (the dotted name) is replaced by the value used in that
|
||||
evaluation. Whole numbers print without decimals ("3", not "3.0"); others
|
||||
with at most two. An unknown name is left as written.
|
||||
- **MON-28.** **Event notification.** With `eventAlert` enabled, firing emits
|
||||
`telemetry.alert-event` (level `warn`), with `details` set to the rendered
|
||||
message and the alert's id. Webhooks subscribed to that event forward it.
|
||||
- **MON-29.** **Email notification.** With `emailAlert` enabled, firing queues
|
||||
one message to every address in `to`, through the normal outbound queue, so
|
||||
it's retried, DKIM-signed for a local sender domain, and visible in the
|
||||
queue like any other. Headers: `From: "fromName" <fromAddress>` (bare
|
||||
address when `fromName` is null), `To`, `Subject`, `Date`, `Message-ID`,
|
||||
and `Auto-Submitted: auto-generated` (RFC 3834). Body: `text/plain`, UTF-8.
|
||||
The server emits `telemetry.alert-message` once it's queued.
|
||||
- **MON-30.** An alert email is sent even when the queue itself is the
|
||||
problem the alert reports. It's queued like any message. If queueing fails,
|
||||
the event notification (MON-28) still happens and the failure is logged.
|
||||
|
||||
### Who may see what
|
||||
|
||||
- **MON-31.** Traces, metric samples, alerts, the two store settings and live
|
||||
telemetry are server-level. By default only the superuser permission set
|
||||
holds their permissions (see "Permissions"). A tenant administrator gets
|
||||
`forbidden` for all of them, even if a role grants the permission, unless
|
||||
its tenant allows it (multi-tenancy MT-13). **Decision**: traces carry other
|
||||
tenants' addresses and IP addresses, and metrics describe the whole server.
|
||||
A tenant-scoped trace view is a possible later addition (open question 11).
|
||||
- **MON-32.** `x:Trace` and `x:Metric` can't be created or updated, as the
|
||||
AGPL registry already enforces. **Decision**, an addition: a trace can be
|
||||
destroyed with `sysTraceDestroy`, so an operator can honor a request to
|
||||
erase someone's delivery records before they age out. Samples can't be
|
||||
destroyed one by one.
|
||||
- **MON-33.** Traces are personal data: they hold addresses, IP addresses,
|
||||
host names and message ids, never message content (MON-12). Retention
|
||||
(MON-17) is the main control, and INBUXA Admin's retention form says what's
|
||||
kept. Nothing in this feature sends traces off the host. Webhooks and
|
||||
OpenTelemetry send only what an operator configures.
|
||||
|
||||
### Failure behavior: telemetry never blocks mail
|
||||
|
||||
- **MON-34.** Trace and sample writes happen off the mail path, through a
|
||||
lossy collector subscriber with a bounded buffer. When the buffer is full,
|
||||
events are dropped and counted, and one error event is logged per minute at
|
||||
most. No SMTP, IMAP, JMAP or delivery step ever waits on history.
|
||||
- **MON-35.** A failing tracing or metrics store (full, unreachable, slow)
|
||||
loses history and logs, rate-limited. It never fails a delivery, a login or
|
||||
startup.
|
||||
- **MON-36.** A failed `IndexTrace` task is retried by the task manager as
|
||||
now. The trace stays readable by id and by date meanwhile.
|
||||
- **MON-37.** A failed alert evaluation or notification is logged and retried
|
||||
on the next evaluation. It never stops the other alerts.
|
||||
- **MON-38.** A failed clean-up leaves the records for the next run. Expired
|
||||
records are still hidden (MON-17).
|
||||
|
||||
### Edition cleanup
|
||||
|
||||
- **MON-39.** Remove the `is_enterprise` and `_is_enterprise` parameters and
|
||||
their `inbuxa:` signposts in `telemetry/mod.rs` and `config/telemetry.rs`;
|
||||
`Tracers::parse` uses `storage` again. Drop `Metric` and `Trace` from
|
||||
`assert_enterprise_object`. Replace the "Enterprise feature" stubs for
|
||||
`/api/token/{tracing,metrics}` and `/api/live/{tracing,metrics}`, and the
|
||||
`cfg(not(feature = "enterprise"))` branch in `management_access_token`.
|
||||
|
||||
## Interfaces
|
||||
|
||||
- **JMAP, existing names, unchanged.** Over `urn:stalwart:jmap`:
|
||||
- `x:Alert/get`, `/query`, `/set`, and the `x:TracingStore`,
|
||||
`x:MetricsStore`, `x:DataRetention`, `x:Search` singletons.
|
||||
- `x:Trace/get` and `/query`. Filters: `text`, `timestamp` (comparison names
|
||||
such as `timestampIsGreaterThan`), `queueId`, `event` (the type of the
|
||||
trace's opening event: the list views use `smtp.connection-start` and
|
||||
`delivery.attempt-start`). Sort by `timestamp`, newest first by default.
|
||||
`/set` refuses create and update, and allows destroy (MON-32).
|
||||
- `x:Metric/get` and `/query`. Filters: `metric` (one name or a list) and
|
||||
the `timestamp` comparisons `timestampIsGreaterThan`,
|
||||
`timestampIsGreaterThanOrEqual`, `timestampIsLessThan`,
|
||||
`timestampIsLessThanOrEqual`. A bare `timestamp` filter is
|
||||
`unsupportedFilter`, as upstream. Sort by `timestamp` either way, with
|
||||
`position`, `anchor` and `anchorOffset` paging and `calculateTotal`.
|
||||
- `x:Trace` text or `queueId` filters when `indexTelemetry` is off:
|
||||
`unsupportedFilter`, with a description saying trace search is off.
|
||||
- **HTTP.** `/api/token/tracing`, `/api/token/metrics`, `/api/live/tracing`,
|
||||
`/api/live/metrics` as MON-20 to MON-24. These are the route names in the
|
||||
AGPL code. Upstream's docs name `/api/telemetry/traces/live` and
|
||||
`/api/telemetry/metrics/live`. **Decision**: serve those as aliases too.
|
||||
- **Webhooks, unchanged.** Already AGPL, and intact after the strip: the
|
||||
webhook tracer is in neither the removed-file list nor the snippet list,
|
||||
and upstream's docs don't mark webhooks as Enterprise. `POST` of
|
||||
`{"events": [...]}`, HMAC-SHA256 in `X-Signature`, batched by `throttle`,
|
||||
stale events dropped after `discardAfter`.
|
||||
- **Dashboards.** The schema's six dashboards (Overview, Network, Security,
|
||||
Delivery, Performance, Storage) name, per card, `live` (MON-22) or
|
||||
`history` (`x:Metric`) and an aggregate. The server serves what they name.
|
||||
INBUXA Admin draws them unchanged.
|
||||
|
||||
## ihasmail changes
|
||||
|
||||
These go in ihasmail-inbuxa, not public ihasmail, which stays Stalwart-facing
|
||||
(SPEC.md §5).
|
||||
|
||||
- The dashboard already reads `x:Metric` for received, sent and memory. Keep
|
||||
it. Drop the Enterprise wording from its comments and code paths, and the
|
||||
"refused as `forbidden`" branch becomes a plain error.
|
||||
- Change the footer line that sends people to "Stalwart's own
|
||||
administration" to name INBUXA Admin, with its link.
|
||||
- Warn on the dashboard when metric history is off (no samples in the window,
|
||||
MON-5), with a link to INBUXA Admin's Metrics Store page.
|
||||
- Live tracing, trace history and alerts stay in INBUXA Admin (SPEC.md §5.4).
|
||||
ihasmail doesn't grow screens for them.
|
||||
- contract.md C-19 has to list `x:Metric` (get and query) among the object
|
||||
types the `inbuxa:account-admin` scope reaches, or the dashboard loses its
|
||||
message cards.
|
||||
- Translation work: two changed strings (the footer line and the error that
|
||||
replaces the refused case) and one new one (the history-off warning), each
|
||||
in ihasmail's nine languages.
|
||||
|
||||
## Acceptance tests
|
||||
|
||||
Every test runs against inbuxa-server built with no Enterprise code.
|
||||
|
||||
1. New install: `TracingStore` and `MetricsStore` read `Default`, retention
|
||||
30 and 90 days, hourly collection (MON-1).
|
||||
2. Tracing store set to an unreachable PostgreSQL: the server starts, and
|
||||
mail is delivered (MON-2, MON-35).
|
||||
3. Two collection ticks with traffic between: counters hold the increase,
|
||||
gauges the reading, idle counters write nothing (MON-4, MON-5).
|
||||
4. `queue.count` is right after a restart with mail queued (MON-7).
|
||||
5. Prometheus output includes `queue_count` and the store and DNS histograms
|
||||
(MON-6).
|
||||
6. One LMTP delivery, with HTTP traffic alongside: exactly two traces, one
|
||||
`smtp.connection-start`, one `delivery.attempt-start` (MON-10).
|
||||
7. An SMTP connection that quits without `MAIL FROM`: no trace (MON-11).
|
||||
8. A trace holds no raw I/O event, and nothing below `info` (MON-12).
|
||||
9. `from`, `to`, `size` and `timestamp` set as MON-14.
|
||||
10. Text search for the sender, the recipient and their domain each finds both
|
||||
traces (MON-16).
|
||||
11. `indexTelemetry` off: `text` filter is `unsupportedFilter`, `timestamp`
|
||||
still works (MON-16, Interfaces).
|
||||
12. Purge with retention 1 second: all traces and their index entries gone.
|
||||
With 2 seconds, nothing gone yet (MON-17).
|
||||
13. Metric query with `timestampIsGreaterThan`, and paging forward, backward
|
||||
and by anchor, returns consistent pages (Interfaces).
|
||||
14. Live tracing with `?remoteIp=` shows only that client's events, and no
|
||||
raw I/O (MON-20, MON-21).
|
||||
15. Live metrics with `metrics=server.memory&interval=1` yields a frame a
|
||||
second (MON-22).
|
||||
16. Live token: expires after 60 seconds, works once, refused without
|
||||
`liveTracing` (MON-23).
|
||||
17. A ninth live stream is refused. A stream closes after 30 minutes
|
||||
(MON-24).
|
||||
18. Alert on `metric('domain.count') > 1 && metric('cluster.publisher-error') > 3`
|
||||
with both conditions met: one email with the placeholders filled ("3
|
||||
domains and 5 cluster errors"), `From: "Alert Subsystem"
|
||||
<[email protected]>`, one `telemetry.alert-event`. The opposite condition
|
||||
fires nothing (MON-25 to MON-29).
|
||||
19. The same alert with an underscore condition (`domain_count > 1`) fires
|
||||
the same way (MON-25).
|
||||
20. The alert fires once while the condition stays true, and again after it
|
||||
has been false (MON-26).
|
||||
21. An alert with an unknown metric name is refused on save (MON-26).
|
||||
22. A webhook subscribed to `telemetry.alert-event` receives the alert
|
||||
(MON-28).
|
||||
23. A tenant administrator with `sysTraceGet` in a role, in a tenant that
|
||||
doesn't allow it: `forbidden` (MON-31).
|
||||
24. `x:Trace/set` destroy by a superuser removes the trace and its index
|
||||
entry. Create and update are refused (MON-32).
|
||||
25. A full tracing store buffer drops events without delaying an SMTP session
|
||||
(MON-34).
|
||||
26. **(compat)** A copy of INBUXA's data opens: alerts, store settings and
|
||||
retention read back unchanged. Old traces and samples it can't decode are
|
||||
skipped, and are gone after one purge past their age (Data model).
|
||||
|
||||
### The gated integration suites
|
||||
|
||||
`tests/src/telemetry/mod.rs` gates four suites behind `pending-rebuild`:
|
||||
|
||||
| Suite | Needs rebuilding | Why it's gated |
|
||||
|---|---|---|
|
||||
| `alerts.rs` | Yes | It calls `process_alerts()`, which lived in the removed `enterprise/alerts.rs`. The rebuild provides a function of that shape (returns the messages it would send) so the suite runs unchanged |
|
||||
| `metrics.rs` | Yes | It needs the metrics store's `purge_metrics` and the test-data generator `insert_test_metrics`, from the removed `metrics/store.rs` and `metrics/test_data.rs` (the latter is the dangling `test_data` module in `metrics/mod.rs`) |
|
||||
| `tracing.rs` | Yes | It needs the tracing store's `purge_spans`, from the removed `tracers/store.rs` |
|
||||
| `webhooks.rs` | No | Webhooks are AGPL and intact. The suite is gated only because its clean-up calls `purge_spans` through the shared harness. Dropping or guarding that one call lets it run now, before the rebuild |
|
||||
|
||||
Recommended: un-gate `webhooks.rs` straight away (a separate change, not made
|
||||
here), so webhooks are tested while the rest is rebuilt.
|
||||
|
||||
## Open questions / to observe
|
||||
|
||||
To check read-only against INBUXA's live Enterprise server later. None of
|
||||
these needs a write.
|
||||
|
||||
1. How many traces a day INBUXA stores against its SMTP connection count:
|
||||
whether upstream stores connection-only sessions (MON-11).
|
||||
2. Which events and levels a stored trace holds, and whether raw I/O ever
|
||||
appears (MON-12, MON-15).
|
||||
3. How upstream fills `from`, `to` and `size` on a trace with several
|
||||
recipients (MON-14).
|
||||
4. Whether INBUXA's samples include `Histogram` records, which gauges appear
|
||||
every tick, and whether counter samples are per-interval increases
|
||||
(MON-4 to MON-6). ihasmail's code says they are.
|
||||
5. Whether `queue.count` in INBUXA's history matches the real queue after a
|
||||
restart (MON-7).
|
||||
6. INBUXA's current `holdTracesFor`, `holdMetricsFor`,
|
||||
`metricsCollectionInterval`, `indexTelemetry` and store settings, so the
|
||||
cutover keeps them.
|
||||
7. What a null `holdTracesFor` or `holdMetricsFor` does upstream: keep
|
||||
forever, or store nothing (MON-18).
|
||||
8. The `x:Trace/query` filters upstream accepts beyond `text`, `timestamp`,
|
||||
`queueId` and `event`, and the `x:Metric/query` comparison names.
|
||||
9. The live endpoints: the token response body, the SSE frame shape for
|
||||
tracing and metrics, keep-alive comments, and which path INBUXA Admin
|
||||
calls (MON-20 to MON-23). A `GET` of each is read-only.
|
||||
10. Whether INBUXA has any `x:Alert` objects, and whether their conditions use
|
||||
`metric()` or underscore names (MON-25). Whether INBUXA's logs show
|
||||
`telemetry.alert-event` repeating while a condition held, which settles
|
||||
the cadence and repeat behavior (MON-25, MON-26).
|
||||
11. Whether upstream's tenant `Admin` role holds any `sysTrace*`,
|
||||
`sysMetric*` or live permission, and whether a tenant view of its own
|
||||
domains' traces is wanted (MON-31).
|
||||
12. Size of INBUXA's `o` and `x` subspaces, to check MON-19's estimate.
|
||||
Reference in New Issue
Block a user