Draft the Phase 8 processing design

Starts from the distribution channel and the safety invariant it
protects, and derives the language from them, rather than designing a
rule language and asking later how to ship it.

Four decisions proposed. A rule is a matcher plus ordered typed actions,
with no expressions and nothing resembling eval -- less expressive than
Cribl on purpose, and the only shape that can be pushed to ten thousand
hosts and audited by reading it. Total evaluation is the primary safety
guarantee, with apply-then-verify as a backstop. One spec with two
implementations means a language-neutral conformance suite is the
specification and should be built first, not last. Distribution reuses
DesiredOverride, following extra_file_paths as the precedent for a
repeated field.

It also corrects something #21 got wrong. That change said a fatal rule
would strand an agent the way a corrupted ingest endpoint does. Reading
apply_override's actual semantics, overrides live only in the running
process's memory and are never written to disk, so a restarted agent
boots clean and re-syncs -- a fatal rule set crash-loops rather than
strands, and the agent keeps checking in, so it stays correctable. That
is a much better failure mode, and it was acquired by accident: the
"don't persist" choice was made to avoid filesystem writes on read-only
images, not for safety. This design promotes it to a constraint, since
persisting overrides later would silently convert every crash-loop into
a strand. positioning.md is corrected to match rather than left
disagreeing.

Five open questions are left open rather than answered to look decisive,
the sharpest being that there is no staged rollout today: an edit
reaches every matching agent at once, which for executable rules is the
difference between breaking one host and breaking all of them.

The v1 acceptance test is real data, not a fixture: two processes on the
maintainer's own workstation account for 308 of 325 journal entries in
five minutes, and a suppress rule should remove about 60% of that host's
volume.

Signed-off-by: John Coffey <[email protected]>

docs/positioning.md

@@ -148,21 +148,25 @@ bad batch size is survivable precisely because the agent still checks in
 and can be corrected.
 
 Processing rules break that invariant. A rule that panics, loops
-forever, or exhausts memory strands the agent exactly the way a corrupted
-`ingest.endpoint` would — the one channel capable of fixing the mistake
-is the thing the mistake killed, across however many hosts the rule
-reached before anyone noticed. So Phase 8 owes one of two things, chosen
-deliberately rather than discovered during a rollout:
+forever, or exhausts memory is not a degraded setting; it is a broken
+agent, across however many hosts the rule reached before anyone noticed.
 
-1. **Total evaluation** — a rule set that provably cannot panic, cannot
-   loop unboundedly, and cannot allocate without limit. A typed
-   declarative DSL can offer this; it is a third argument for one.
-2. **Apply-then-verify** — the agent treats a new rule set as
-   provisional, and reverts to the last known-good set if it crash-loops
-   before the next successful check-in.
+How badly it breaks turns out to depend on something decided for
+unrelated reasons: overrides live only in the running process's memory
+and are never written to disk, so a restarted agent boots clean and
+re-syncs. A fatal rule set therefore produces a crash-loop rather than a
+strand, and the agent still checks in on every iteration, so it remains
+correctable. That is a much better failure than the stranding an
+unfixable `ingest.endpoint` would cause — and it is load-bearing safety
+acquired by accident, which means persisting overrides to disk later
+would silently convert every crash-loop into a strand.
 
-The first is better if it can be had, because the second is a recovery
-mechanism and the first is an absence of the failure.
+Phase 8 still owes a deliberate answer here rather than a discovered
+one. [`phase-8-processing-design.md`](phase-8-processing-design.md)
+proposes total evaluation as the primary guarantee — a rule set that
+provably cannot panic, loop unboundedly or allocate without limit, which
+a typed declarative DSL can offer and which is a third argument for one
+— with apply-then-verify as a backstop.
 
 ### 2. Routing and multiple destinations
 
@@ -282,6 +286,8 @@ than appending forever to a list that was about analytics.
   ingest-side execution, the tests that prove a rule does the same thing
   in both places, and distribution: carrying rule sets through
   `DesiredOverride` without breaking the strand-safety invariant above.
+  Drafted in
+  [`phase-8-processing-design.md`](phase-8-processing-design.md).
 - **Phase 9 — Routing and sinks.** Multiple destinations, conditional
   routing, per-destination delivery guarantees, and the first three
   sinks: object storage, OTLP, Splunk HEC.