Specify what aggregate_count emits

The last unanswered action, and the only one whose output is not the
input with edits -- it emits a record that never existed, which is why
it was deferred twice.

It emits the window's first record unchanged, tagged with
cairnobs.aggregated, cairnobs.count, and the observed window bounds.
That follows the convention the agent already uses for heartbeat and
host-metrics records rather than inventing a second synthetic-record
mechanism, and keeping the first record intact means a reader sees a
real example of what was collapsed instead of an invented summary.

It tags even when the count is one. Emitting a bare record there would
be tidier and would make cairnobs.count present only sometimes, so
summing it silently breaks on quiet windows. window_last is the last
record that actually contributed, never window_start + window_ms,
because a window flushed early must not claim an end that never
happened.

Specifying it surfaced a problem the other nine actions do not have.
Windows are measured on record time, so a window can only be closed by a
later record arriving. suppress_duplicates never has anything pending;
aggregate_count holds state, so a matching stream that goes quiet leaves
its aggregate unemitted indefinitely -- data loss dressed as latency.
Emission therefore has a second trigger, end of stream, which the corpus
defines as an implicit flush after the last input and which production
gets from the batch flush. The cost is stated rather than hidden:
window_ms becomes a maximum, not a guarantee, and one burst can produce
more than one aggregate.

And it has a consequence nobody should meet in production first: stats
count undercounts aggregated data silently, so every panel and alert
counting rows changes meaning the moment a rule aggregates the data
behind it. Nothing here fixes that. The correct idiom is summing
cairnobs.count; teaching the query layer to do it automatically is a
Phase 2 change to the IR, recorded as the open question this decision
leaves in its place rather than quietly inherited.

Six cases added, corpus at 45. The validator's unspecified-action guard
stays in place with an empty set, still rejecting anything added to it.

Signed-off-by: John Coffey <[email protected]>
This commit is contained in:
2026-09-04 21:58:40 -07:00
parent 17256634db
commit ec4b860ba8
9 changed files with 494 additions and 16 deletions
+96 -4
View File
@@ -311,6 +311,95 @@ only thing standing between a bad rule and every host at once. That
makes open question 1 below considerably less optional than it looked
when it was written.
## Decision 5: what `aggregate_count` emits
Deferred twice because it is the only action whose output is not simply
the input with edits. It emits a record that never existed.
**Shape: the window's first record, tagged.** The codebase already has a
convention for synthetic records and this follows it rather than
inventing a second one — heartbeat and host-metrics records are ordinary
`LogRecord`s distinguished by a `cairnobs.heartbeat` /
`cairnobs.metrics` attribute, with `service` left as the agent's real
service. So an aggregate is the first record of its window, unchanged,
plus:
| attribute | value |
|---|---|
| `cairnobs.aggregated` | `"true"` |
| `cairnobs.count` | number of records collapsed, as a string |
| `cairnobs.window_start_unix_nano` | timestamp of the first record |
| `cairnobs.window_last_unix_nano` | timestamp of the last contributing record |
Keeping the first record intact means a human reading the line sees a
real example of what was collapsed, not a summary someone invented.
**It tags even when the count is one.** Emitting a bare record for a
window that happened to see one event would be tidier and is wrong: it
makes `cairnobs.count` present only sometimes, so the correct way to
count aggregated data silently breaks on quiet windows. Uniformity beats
tidiness here.
**Window bounds are observed, not nominal.** `window_last` is the last
record that actually contributed, never `window_start + window_ms`. A
window flushed early must not claim an end that never happened.
### When it emits, and the problem that hides here
Windows are measured on record time (see `/processing/README.md`), which
means a window can only be *closed* by a later record arriving. That is
fine for `suppress_duplicates`, which emits the first record immediately
and drops the rest — nothing is ever pending.
`aggregate_count` holds state. If the matching stream goes quiet, the
pending aggregate has nothing to close it and sits unemitted, possibly
for hours. That is data loss dressed as latency, and it is the real
reason this action was harder to specify than the other nine.
So emission has two triggers:
1. **A later matching record** with a timestamp at or past
`window_start + window_ms`. The pending aggregate is emitted first,
then that record opens the next window.
2. **End of stream** — agent shutdown, and in production the batch flush
interval. Anything pending is emitted.
**The cost, stated plainly:** trigger 2 is wall-clock in production,
which means `window_ms` is a *maximum*, not a guarantee, and one logical
burst can produce more than one aggregate record if a flush lands in the
middle. Consumers must treat aggregates as additive — which they already
must, since a burst can span windows anyway.
The conformance corpus defines trigger 2 as an implicit flush after the
last input, which keeps the cases deterministic while describing real
behaviour.
### The consequence nobody should discover in production
**`stats count` undercounts aggregated data**, silently. A hundred
events become one record, so every existing dashboard panel and alert
rule that counts rows changes meaning the moment a rule starts
aggregating the data behind it.
Nothing in this design fixes that, and pretending otherwise would be
worse than saying it. Three things follow:
- The correct idiom over aggregated data is summing `cairnobs.count`,
not counting rows. That belongs in the query language reference before
this action ships.
- Teaching the query layer to do it automatically — making `count`
mean "sum `cairnobs.count` where present" — is a Phase 2 change to the
IR and executor, not a Phase 8 change, and it is the right long-term
answer. **Open**, and it should be decided before aggregation is
recommended for any data an alert already watches.
- It is another argument for aggregation staying opt-in per rule, which
it is.
**Choosing between the two dedup actions:** `suppress_duplicates` is
cheaper and loses the count; `aggregate_count` preserves it and creates
a synthetic record with all of the above attached. Use suppress when the
repetition is noise, aggregate when the rate is the signal.
## Where each rule runs
Agent-side is the default and the cheaper place: data reduced before the
@@ -360,7 +449,10 @@ release cannot do that, it is not finished.
3. ~~Canary rollout?~~ **Decided 2026-09-05:** no gate, ship without it;
a canary is follow-up work.
4. Explicit per-rule placement, or platform-decided?
5. Does `aggregate_count` emit a synthetic record, and if so what does it
look like to a query that is not expecting one? This design does not
answer that and should before anyone builds it. The conformance
validator refuses any case using it until it is answered.
5. ~~What does `aggregate_count` emit?~~ **Decided 2026-09-05:** the
window's first record tagged with `cairnobs.aggregated`,
`cairnobs.count` and observed window bounds — see Decision 5. It
raises one new question in its place: whether `stats count` should
learn to sum `cairnobs.count` automatically, which is a Phase 2
change to the IR rather than a Phase 8 one, and should be settled
before aggregation is pointed at data an alert already watches.