cairnobs/docs/phase-2-runbook.md

Phase 2 runbook

Extends /docs/phase-0-runbook.md and /docs/phase-1-runbook.md with the unified query language and the "modest dataset" query-latency benchmark. Read those first — this assumes the stack already works; Phase 2 replaces the two placeholder query endpoints/pages with one, and adds a way to actually measure query performance at volume.

What's actually been verified

Unlike the phrasing "define 'modest', put a rough benchmark in the runbook" might suggest, this isn't an estimate — the numbers below are from actually generating 1,022,000 rows in a live ClickHouse (via /hack/benchmark-fixture, pushed through the real agent-facing gRPC endpoint, so both ClickHouse and Tantivy have real, matching data) and timing real queries against it. One real bug turned up doing this (see "What the benchmark caught" below) that wouldn't have been found any other way.

1. Bring up the stack

docker compose up -d --build

Same as Phase 0/1. Confirm the unified endpoint works for both syntaxes:

curl -X POST http://localhost:8080/query -H 'Content-Type: application/json' \
  -d '{"query": "SELECT 1"}'
# -> {"columns":["1"],"rows":[[1]]}

curl -X POST http://localhost:8080/query -H 'Content-Type: application/json' \
  -d '{"query": "service=api"}'
# -> {"columns":[...],"rows":[]}  (empty is fine -- no data ingested yet)

2. Generate the "modest" benchmark dataset

"Modest" is defined here as 1,000,000 rows — enough to be a real volume test, small enough to generate and query interactively rather than needing a dedicated load-testing pass.

cd hack/benchmark-fixture
go run . --count 1000000 --batch-size 1000 --concurrency 16

Sends batched PushBatch calls directly to ingest's gRPC endpoint — the same path the real agent uses, just generating synthetic data instead of reading journald, so it exercises the real ingest → Redpanda → ClickHouse-writer-consumer and → search-indexer-consumer paths, not a shortcut that bypasses them. Concurrency matters: sequential single-batch calls topped out around 500 records/sec (would take ~33 minutes for 1M rows); 16 concurrent PushBatch calls over one shared gRPC connection (HTTP/2 multiplexes concurrent RPCs over it natively) got over 400,000 records/sec, so the actual measured run took about 2 seconds for the ingest call itself.

Wait for both consumers to fully drain before benchmarking — ingestion finishing doesn't mean ClickHouse/Tantivy are caught up yet:

# poll until this stops climbing
curl -s -X POST http://localhost:8080/query -H 'Content-Type: application/json' \
  -d '{"query": "SELECT count() FROM logs", "language": "sql"}'

In the run this runbook was written from, ClickHouse settled at 1,022,000 rows a few seconds after ingestion finished (the extra 22,000 were earlier smaller test runs from developing the benchmark tool itself — harmless, still real rows).

3. Run the benchmark queries

# structured filter + aggregation -- pure ClickHouse path
curl -s -o /dev/null -w "%{time_total}s\n" -X POST http://localhost:8080/query \
  -H 'Content-Type: application/json' \
  -d '{"query": "service=api | where status>=500 | stats count by host | sort -count"}'

# free-text + aggregation -- Tantivy prefilter feeding a ClickHouse GROUP BY,
# the case /docs/query-language-design.md called "the hardest part"
curl -s -o /dev/null -w "%{time_total}s\n" -X POST http://localhost:8080/query \
  -H 'Content-Type: application/json' \
  -d '{"query": "message:\"connection refused\" | stats count by host | sort -count"}'

Measured results against the 1,022,000-row dataset (this exact run, not a projection):

Query	Path	Wall time
service=api | where status>=500 | stats count by host | sort -count	Pure ClickHouse	19.7ms
message:"connection refused" | stats count by host | sort -count	Tantivy prefilter + ClickHouse aggregate	46.4ms
message:"connection refused" | head 50	Tantivy prefilter, no aggregation	49.2ms
SELECT count() FROM logs WHERE service='web' (raw SQL)	Pure ClickHouse	17.5ms

All four "well under a second" — the Phase 2 exit criteria in /PROJECT-SPEC.md. The combined text+aggregation case (the one everyone should be nervous about, since it's a two-backend query) came in at 46ms, not meaningfully slower than the pure-ClickHouse case — the Tantivy prefilter step is fast, and 5,000 UUIDs (see below) is a small IN clause by ClickHouse's standards.

4. What the benchmark caught

The original design capped the Tantivy prefilter at 10,000 matching record_ids before feeding them into ClickHouse's WHERE record_id IN (...). Running the actual combined-query benchmark against real data (where "connection refused" matched a large fraction of the 1M rows, by design — the fixture generator seeds that phrase deliberately) hit this immediately:

query failed: executing query: code: 62, message: Syntax error:
failed at position 262116 [...] Max query size exceeded

10,000 quoted UUIDs (~39 bytes each including the separating comma) produces a ~390KB query string, which exceeds ClickHouse's default max_query_size (262144 bytes / 256KiB) — a much lower ceiling than "multi-million-entry" suggested before anyone had actually tried it. The cap is now 5,000 (~195KB, safely under the default with headroom) — see api/internal/querylang/executor/executor.go's textSearchLimit and /docs/query-language-design.md's "Known scaling limitation" section, both updated with this exact finding rather than a first-principles guess.

5. Confirm the web UI and CLI both work end to end

Open http://localhost:3000 — one page now (Phase 0/1's two pages are gone), run service=api | where status>=500 | stats count by host | sort -count in the query bar, confirm results render and the query appears in the session-local history panel.

cd cli
go run ./cmd/cairnobsctl query 'service=api | where status>=500 | stats count by host | sort -count'

Both hit the exact same POST /query endpoint — there's no separate query logic to drift out of sync between the CLI, the web UI, and whatever else calls this API later.

Tearing down

docker compose down -v   # also wipes the 1M-row benchmark dataset

Troubleshooting

A text-search query with aggregation fails with a ClickHouse syntax error / "Max query size exceeded." If you've raised textSearchLimit above 5,000, you've likely reintroduced the exact failure this runbook's benchmark caught (see "What the benchmark caught" above) — lower it back down, or raise ClickHouse's max_query_size server-side with matching memory sizing if you genuinely need a larger prefilter.

Benchmark ingestion is much slower than ~400K records/sec. Check --concurrency wasn't left at a low value, and that ingest isn't CPU-starved (docker stats) — the numbers above are from a single machine running the whole stack including Redpanda/ClickHouse/search concurrently, not a dedicated load-test environment, so your exact throughput will vary with hardware. What matters is "well under a second" for the actual query latency, not the ingestion throughput number itself, which is just how the test data gets there.

stats sum(field)/avg(field) etc. on an attributes-map field returns 0 for everything. Check the field's values are actually numeric strings — non-numeric attribute values silently cast to 0 via toFloat64OrZero (see /docs/query-language-reference.md's "Field mapping" section). This is documented behavior, not a bug, but it's easy to trip over with a typo'd field name (which also "succeeds" with all zeros, since a missing map key reads as an empty string, which also casts to 0).