Phase 2: unified query language spanning ClickHouse and Tantivy

Replaces the separate SQL-only /query and text-only /search endpoints
with one pipe-syntax query language (plus raw SQL escape hatch) that
compiles to a single IR and execution plan across both backends, so a
query like `message:"connection refused" | stats count by host` runs
as one request instead of two disjoint tools.

- api/internal/querylang: lexer -> ast -> parser -> ir -> planner ->
  executor, each layer independently tested.
- Execution generalizes Phase 1's proven Tantivy-prefilter pattern
  into a 4-way routing table (pure ClickHouse / text-only / text +
  aggregation / raw SQL passthrough).
- Unified web query page and `sentryctl query`, both hitting the same
  POST /query endpoint.
- Benchmarked against a real 1,022,000-row dataset
  (hack/benchmark-fixture); caught and fixed a real bug where the
  Tantivy prefilter cap (10,000) produced an IN-clause exceeding
  ClickHouse's default max_query_size -- lowered to 5,000, documented
  in docs/query-language-design.md and docs/phase-2-runbook.md.
- docs/query-language-reference.md: customer-facing syntax reference.
This commit is contained in:
2026-08-13 12:21:42 -07:00
parent cd8aa290ca
commit fb5049a747
36 changed files with 4119 additions and 613 deletions
+8 -6
View File
@@ -1,7 +1,8 @@
// Command api is Sentry's query API: POST /query (raw SQL, SELECT-only)
// and POST /search (free-text, via the search service). See
// internal/queryapi for why these are plain REST rather than the pinned
// gRPC+gateway pattern.
// Command api is Sentry's query API: a single POST /query endpoint
// accepting either the pipe syntax or raw SQL, compiled and routed
// across ClickHouse and search by internal/querylang. See
// internal/queryapi and /docs/query-language-design.md for why this is
// plain REST rather than the pinned gRPC+gateway pattern.
package main
import (
@@ -16,6 +17,7 @@ import (
"github.com/ClickHouse/clickhouse-go/v2"
"github.com/sentry/sentry/api/internal/config"
"github.com/sentry/sentry/api/internal/querylang/executor"
"github.com/sentry/sentry/api/internal/queryapi"
"github.com/sentry/sentry/api/internal/searchclient"
)
@@ -58,8 +60,8 @@ func main() {
}
defer search.Close()
exec := queryapi.NewExecutor(conn)
handler := queryapi.NewHandler(logger, exec, search, cfg.QueryTimeout, cfg.CORSAllowedOrigin)
sqlRunner := executor.NewChRunner(conn)
handler := queryapi.NewHandler(logger, sqlRunner, search, cfg.QueryTimeout, cfg.CORSAllowedOrigin)
srv := &http.Server{
Addr: cfg.HTTPListenAddr,