Specify what aggregate_count emits

The last unanswered action, and the only one whose output is not the
input with edits -- it emits a record that never existed, which is why
it was deferred twice.

It emits the window's first record unchanged, tagged with
cairnobs.aggregated, cairnobs.count, and the observed window bounds.
That follows the convention the agent already uses for heartbeat and
host-metrics records rather than inventing a second synthetic-record
mechanism, and keeping the first record intact means a reader sees a
real example of what was collapsed instead of an invented summary.

It tags even when the count is one. Emitting a bare record there would
be tidier and would make cairnobs.count present only sometimes, so
summing it silently breaks on quiet windows. window_last is the last
record that actually contributed, never window_start + window_ms,
because a window flushed early must not claim an end that never
happened.

Specifying it surfaced a problem the other nine actions do not have.
Windows are measured on record time, so a window can only be closed by a
later record arriving. suppress_duplicates never has anything pending;
aggregate_count holds state, so a matching stream that goes quiet leaves
its aggregate unemitted indefinitely -- data loss dressed as latency.
Emission therefore has a second trigger, end of stream, which the corpus
defines as an implicit flush after the last input and which production
gets from the batch flush. The cost is stated rather than hidden:
window_ms becomes a maximum, not a guarantee, and one burst can produce
more than one aggregate.

And it has a consequence nobody should meet in production first: stats
count undercounts aggregated data silently, so every panel and alert
counting rows changes meaning the moment a rule aggregates the data
behind it. Nothing here fixes that. The correct idiom is summing
cairnobs.count; teaching the query layer to do it automatically is a
Phase 2 change to the IR, recorded as the open question this decision
leaves in its place rather than quietly inherited.

Six cases added, corpus at 45. The validator's unspecified-action guard
stays in place with an empty set, still rejecting anything added to it.

Signed-off-by: John Coffey <[email protected]>
This commit is contained in:
2026-09-04 21:58:40 -07:00
parent 17256634db
commit ec4b860ba8
9 changed files with 494 additions and 16 deletions
+34 -7
View File
@@ -113,6 +113,7 @@ no later action or rule runs.
| `parse_regex` | `field`, `pattern` (named captures become attributes) |
| `sample` | `keep_one_in` |
| `suppress_duplicates` | `window_ms`, optional `key_fields` |
| `aggregate_count` | `window_ms`, optional `key_fields` |
Rules are evaluated in the order given. Every matching rule's actions
apply, to the record as left by the rule before it.
@@ -150,14 +151,40 @@ close enough at volume.
the same output regardless of how fast the test runs. This also means the
behaviour is correct under backfill, which wall-clock windows are not.
### `aggregate_count` has no cases, deliberately
### `aggregate_count` emits a record that never existed
The design lists it as an action but does not answer what it emits, or
what a query that is not expecting a synthetic record sees. Writing cases
now would invent that answer by accident and freeze it. It stays
unspecified until that question is decided —
[`phase-8-processing-design.md`](../docs/phase-8-processing-design.md)
open question 5.
The only action whose output is not the input with edits, so its shape is
worth stating here too. It emits the window's **first record, unchanged**,
plus four attributes:
| attribute | value |
|---|---|
| `cairnobs.aggregated` | `"true"` |
| `cairnobs.count` | records collapsed, as a string |
| `cairnobs.window_start_unix_nano` | first record's timestamp |
| `cairnobs.window_last_unix_nano` | last contributing record's timestamp |
This follows the convention the agent already uses for heartbeat and
host-metrics records: an ordinary record distinguished by a
`cairnobs.*` attribute, rather than a second synthetic-record mechanism.
Three behaviours the cases pin, each of which is easy to get wrong:
- **It tags even when the count is one.** Otherwise `cairnobs.count` is
present only sometimes, and summing it silently breaks on quiet
windows.
- **`window_last` is observed, never `window_start + window_ms`.** A
window flushed early must not claim an end that never happened.
- **A pending window flushes at end of stream.** Record-time windows can
only be closed by a later record, so without this a burst that stops
leaves its aggregate unemitted indefinitely. In production the batch
flush provides the same trigger, which makes `window_ms` a maximum
rather than a guarantee.
**`stats count` undercounts aggregated data.** The correct idiom is
summing `cairnobs.count`. Whether the query layer should do that
automatically is a Phase 2 question, recorded in
[`phase-8-processing-design.md`](../docs/phase-8-processing-design.md).
## Adding a case