Phase 2: unified query language spanning ClickHouse and Tantivy
Replaces the separate SQL-only /query and text-only /search endpoints with one pipe-syntax query language (plus raw SQL escape hatch) that compiles to a single IR and execution plan across both backends, so a query like `message:"connection refused" | stats count by host` runs as one request instead of two disjoint tools. - api/internal/querylang: lexer -> ast -> parser -> ir -> planner -> executor, each layer independently tested. - Execution generalizes Phase 1's proven Tantivy-prefilter pattern into a 4-way routing table (pure ClickHouse / text-only / text + aggregation / raw SQL passthrough). - Unified web query page and `sentryctl query`, both hitting the same POST /query endpoint. - Benchmarked against a real 1,022,000-row dataset (hack/benchmark-fixture); caught and fixed a real bug where the Tantivy prefilter cap (10,000) produced an IN-clause exceeding ClickHouse's default max_query_size -- lowered to 5,000, documented in docs/query-language-design.md and docs/phase-2-runbook.md. - docs/query-language-reference.md: customer-facing syntax reference.
This commit is contained in:
@@ -8,6 +8,7 @@ search/target/
|
|||||||
/api/api
|
/api/api
|
||||||
/cli/sentryctl
|
/cli/sentryctl
|
||||||
/hack/windows-fixture/windows-fixture
|
/hack/windows-fixture/windows-fixture
|
||||||
|
/hack/benchmark-fixture/benchmark-fixture
|
||||||
|
|
||||||
# Node / SvelteKit (web/ has its own more detailed .gitignore too)
|
# Node / SvelteKit (web/ has its own more detailed .gitignore too)
|
||||||
web/node_modules/
|
web/node_modules/
|
||||||
|
|||||||
@@ -54,21 +54,46 @@ dashboards — that discipline held for the whole phase.
|
|||||||
|
|
||||||
## What "done" looks like for Phase 1
|
## What "done" looks like for Phase 1
|
||||||
|
|
||||||
A Windows Event Log entry and a Linux journald entry should both be
|
**Status: shipped.** A Windows Event Log entry and a Linux journald entry
|
||||||
queryable via SQL (the ClickHouse path) and via free-text search (the
|
are both queryable via SQL (the ClickHouse path) and via free-text search
|
||||||
Tantivy path), from the same UI, within a few seconds of being generated.
|
(the Tantivy path), from the same UI, within a few seconds of being
|
||||||
|
generated. Verified end-to-end on the live stack, including the same
|
||||||
|
`record_id` coming back from both query paths for the same record — see
|
||||||
|
`/docs/phase-1-runbook.md`.
|
||||||
|
|
||||||
Non-goals for this phase (same "resist scope creep" discipline as Phase 0):
|
ETW and WEF (Windows Event Forwarding) were *designed* in this phase but
|
||||||
no alerting, no dashboards, no SPL-like query layer, no multi-tenancy, and
|
not required to be running for "done": ETW ships behind a feature flag
|
||||||
no unified query experience — two separate boxes on two separate pages is
|
most environments won't enable (it needs elevated privileges), and WEF's
|
||||||
correct for Phase 1; unifying them is Phase 2's job.
|
receiver-side was explicitly deferred rather than built. Only the Event
|
||||||
|
Log source needed to actually be running end-to-end, and did. The
|
||||||
|
Windows-specific agent code itself (`EvtSubscribe`, ETW, service
|
||||||
|
registration) remains unverified on real Windows — no Windows toolchain
|
||||||
|
existed anywhere in the environment this was built in; flagged
|
||||||
|
prominently in `/agent/README.md` and the runbook.
|
||||||
|
|
||||||
ETW and WEF (Windows Event Forwarding) are *designed* in this phase but not
|
## What "done" looks like for Phase 2
|
||||||
required to be running for "done": ETW ships behind a feature flag most
|
|
||||||
environments won't enable (it needs elevated privileges), and WEF's
|
A single query bar in the web UI and a single `sentryctl query` command
|
||||||
receiver-side is explicitly deferred rather than built now — see
|
can express filter + free-text + stats in one query (e.g. `service=api |
|
||||||
`/docs/phase-1-runbook.md` for both. Only the Event Log source needs to
|
where status>=500 | stats count by host | sort -count`, or
|
||||||
actually be running end-to-end for this phase to count as done.
|
`message:"connection refused" | stats count by host`), execute correctly
|
||||||
|
against both ClickHouse and Tantivy in one compiled plan, and return in
|
||||||
|
well under a second for a 1M-row fixture dataset (rough benchmark, not a
|
||||||
|
formal SLA — see `/docs/phase-2-runbook.md` for the actual measurement).
|
||||||
|
Raw ClickHouse SQL remains available as an escape hatch, compiling to the
|
||||||
|
same execution plan/IR as the pipe syntax so performance doesn't depend
|
||||||
|
on which syntax a query uses.
|
||||||
|
|
||||||
|
Non-goals for this phase (same "resist scope creep" discipline as every
|
||||||
|
phase so far): no alerting, no dashboards, no multi-tenancy — this phase
|
||||||
|
is the query layer only. The two separate placeholder pages/endpoints
|
||||||
|
from Phase 0/1 (`/query` raw-SQL-only, `/search` free-text-only) are
|
||||||
|
retired, replaced by one `/query` endpoint and one query page.
|
||||||
|
|
||||||
|
See `/docs/query-language-design.md` for the grammar, IR, and
|
||||||
|
ClickHouse/Tantivy routing strategy, and
|
||||||
|
`/docs/query-language-reference.md` for the user-facing syntax reference
|
||||||
|
once built.
|
||||||
|
|
||||||
## When in doubt
|
## When in doubt
|
||||||
Ask before: changing the pinned stack, adding a new external dependency
|
Ask before: changing the pinned stack, adding a new external dependency
|
||||||
|
|||||||
+41
-38
@@ -1,39 +1,39 @@
|
|||||||
# api
|
# api
|
||||||
|
|
||||||
Sentry's query API: two intentionally crude endpoints — raw SQL and
|
Sentry's query API: a single `POST /query` endpoint accepting either the
|
||||||
free-text search — that Phase 2's real query layer replaces outright.
|
pipe syntax or raw SQL, compiled and routed across ClickHouse and Tantivy
|
||||||
|
by `internal/querylang`. Replaces Phase 0/1's two separate placeholder
|
||||||
|
endpoints (raw-SQL-only `/query`, free-text-only `/search`) — see
|
||||||
|
`/docs/query-language-design.md` for the grammar, IR, and routing design,
|
||||||
|
and `/docs/query-language-reference.md` for the user-facing syntax.
|
||||||
|
|
||||||
## Why plain REST, not gRPC + REST gateway
|
## Why plain REST, not gRPC + REST gateway
|
||||||
|
|
||||||
CLAUDE.md pins the control plane to "Go, gRPC + REST gateway." This
|
CLAUDE.md pins the control plane to "Go, gRPC + REST gateway." This
|
||||||
service is plain `net/http` instead — a deliberate simplification, not a
|
service is plain `net/http` instead — a deliberate simplification, not a
|
||||||
change to the pinned stack. Wiring up `.proto` services,
|
change to the pinned stack. Wiring up a `.proto` service,
|
||||||
`google.api.http` annotations, and `protoc-gen-grpc-gateway` codegen for
|
`google.api.http` annotations, and `protoc-gen-grpc-gateway` codegen for
|
||||||
two endpoints that Phase 2 replaces outright with a real SPL-like query
|
one endpoint doesn't buy much at this size. `api` *does* speak gRPC
|
||||||
layer would be exactly the kind of premature machinery this project's
|
internally — to `/search` — this simplification is about the
|
||||||
conventions warn against. `api` *does* speak gRPC internally though — to
|
public-facing surface only.
|
||||||
`/search` (see below) — this simplification is specifically about the
|
|
||||||
public-facing surface, not a blanket avoidance of gRPC.
|
|
||||||
|
|
||||||
## Endpoints
|
## Endpoint
|
||||||
|
|
||||||
- `POST /query` — body `{"sql": "SELECT ..."}`, response
|
`POST /query` — body `{"query": "...", "language": ""}`, response
|
||||||
`{"columns": [...], "rows": [[...], ...]}` or `{"error": "..."}`.
|
`{"columns": [...], "rows": [[...], ...]}` or `{"error": "..."}`.
|
||||||
SELECT-only, single-statement, basic keyword-based injection guarding
|
|
||||||
(see `internal/queryapi/validate.go` for exactly what that does and
|
- `query` is either pipe syntax (`service=api | where status>=500 |
|
||||||
doesn't catch — it's not a SQL parser).
|
stats count by host`) or raw SQL (`SELECT ...`). Auto-detected by
|
||||||
- `POST /search` — body `{"query": "...", "limit": 100}`, same response
|
whether the query starts with `SELECT` (case-insensitive).
|
||||||
shape as `/query`. Calls `/search`'s `SearchService.Search` gRPC RPC to
|
- `language` optionally overrides detection: `"sql"` or `"spl"`. Exists
|
||||||
resolve the free-text query into matching `record_id`s, then joins
|
for the rare case a pipe query legitimately starts with the literal
|
||||||
those back against ClickHouse (`SELECT * FROM logs WHERE record_id IN
|
word "select" as a bare search term.
|
||||||
(...)`) to return full rows — so both endpoints return the same
|
- Both syntaxes compile to the same `querylang/ir.Plan` and execute
|
||||||
`{columns, rows}` shape and `/web` can reuse one table component for
|
through the same code path — see `internal/querylang/executor` for the
|
||||||
both. Every `record_id` is validated as a real UUID before being
|
four routing cases (pure ClickHouse; Tantivy prefilter + ClickHouse
|
||||||
embedded in the generated SQL (defense in depth: `record_id`s come from
|
rows; Tantivy prefilter + ClickHouse aggregation; raw SQL passthrough).
|
||||||
an internal, trusted service, not raw user input, but a value that
|
|
||||||
fails to parse as a UUID can't contain SQL-breaking characters either
|
`GET /healthz` — for docker-compose/k8s liveness checks.
|
||||||
way).
|
|
||||||
- `GET /healthz` — for docker-compose/k8s liveness checks.
|
|
||||||
|
|
||||||
No auth. Not scoped yet — don't expose this beyond a trusted dev/homelab
|
No auth. Not scoped yet — don't expose this beyond a trusted dev/homelab
|
||||||
network.
|
network.
|
||||||
@@ -48,7 +48,7 @@ Environment variables (see `internal/config/config.go`):
|
|||||||
| `CLICKHOUSE_ADDR` | `localhost:9000` | Native protocol port |
|
| `CLICKHOUSE_ADDR` | `localhost:9000` | Native protocol port |
|
||||||
| `CLICKHOUSE_DATABASE` / `_USERNAME` / `_PASSWORD` | `sentry` / `default` / `` | |
|
| `CLICKHOUSE_DATABASE` / `_USERNAME` / `_PASSWORD` | `sentry` / `default` / `` | |
|
||||||
| `SEARCH_GRPC_ADDR` | `localhost:50052` | Must match `/search`'s `GRPC_LISTEN_ADDR` |
|
| `SEARCH_GRPC_ADDR` | `localhost:50052` | Must match `/search`'s `GRPC_LISTEN_ADDR` |
|
||||||
| `QUERY_TIMEOUT_SECONDS` | `30` | Per-request timeout, both endpoints |
|
| `QUERY_TIMEOUT_SECONDS` | `30` | Per-request timeout |
|
||||||
| `CORS_ALLOWED_ORIGIN` | `*` | Wide open by default since there's no auth yet; tighten together |
|
| `CORS_ALLOWED_ORIGIN` | `*` | Wide open by default since there's no auth yet; tighten together |
|
||||||
|
|
||||||
`searchclient.Dial` connects to `/search` over plain TCP, no TLS — same
|
`searchclient.Dial` connects to `/search` over plain TCP, no TLS — same
|
||||||
@@ -71,15 +71,18 @@ docker build -f api/Dockerfile -t sentry-api .
|
|||||||
|
|
||||||
## Testing notes
|
## Testing notes
|
||||||
|
|
||||||
`internal/queryapi`'s HTTP handlers depend on ClickHouse and `/search`
|
`internal/queryapi`'s HTTP handler depends on ClickHouse and `/search`
|
||||||
only through narrow interfaces (`queryExecutor`, `searchClient`), so
|
only through the narrow interfaces `querylang/executor` defines
|
||||||
routing, validation, JSON encoding, error-status mapping, and the
|
(`SQLRunner`, `SearchClient`), so routing, compilation, JSON encoding,
|
||||||
record_id-to-SQL query building are all unit-tested against fakes — no
|
and error-status mapping are all unit-tested against fakes — no live
|
||||||
live ClickHouse or `/search` instance needed. `Executor` itself (the
|
ClickHouse or `/search` instance needed, and the real lexer/parser/
|
||||||
reflection-based row scanning against `driver.Rows`) and
|
planner run unmocked in these tests, only the backends are faked. See
|
||||||
`internal/searchclient`'s actual gRPC dial are not unit-tested — the
|
`internal/querylang`'s own package docs for how compilation and
|
||||||
former because faking ClickHouse's `driver.Rows` interface fully would be
|
execution are tested independently of each other. `executor.ChRunner`
|
||||||
significant test-only scaffolding the driver's own docs say isn't meant
|
(the reflection-based row scanning against ClickHouse's `driver.Rows`)
|
||||||
to be implemented by adopters; the latter because it's a thin wrapper
|
and `internal/searchclient`'s actual gRPC dial are not unit-tested — the
|
||||||
with nothing but wiring to test. Both are exercised end-to-end via the
|
former because faking `driver.Rows` fully would be significant
|
||||||
docker-compose flow in `/docs/phase-1-runbook.md` instead.
|
test-only scaffolding the driver's own docs say isn't meant to be
|
||||||
|
implemented by adopters; the latter because it's a thin wrapper with
|
||||||
|
nothing but wiring to test. Both are exercised end-to-end via the
|
||||||
|
docker-compose flow in `/docs/phase-2-runbook.md`.
|
||||||
|
|||||||
+8
-6
@@ -1,7 +1,8 @@
|
|||||||
// Command api is Sentry's query API: POST /query (raw SQL, SELECT-only)
|
// Command api is Sentry's query API: a single POST /query endpoint
|
||||||
// and POST /search (free-text, via the search service). See
|
// accepting either the pipe syntax or raw SQL, compiled and routed
|
||||||
// internal/queryapi for why these are plain REST rather than the pinned
|
// across ClickHouse and search by internal/querylang. See
|
||||||
// gRPC+gateway pattern.
|
// internal/queryapi and /docs/query-language-design.md for why this is
|
||||||
|
// plain REST rather than the pinned gRPC+gateway pattern.
|
||||||
package main
|
package main
|
||||||
|
|
||||||
import (
|
import (
|
||||||
@@ -16,6 +17,7 @@ import (
|
|||||||
"github.com/ClickHouse/clickhouse-go/v2"
|
"github.com/ClickHouse/clickhouse-go/v2"
|
||||||
|
|
||||||
"github.com/sentry/sentry/api/internal/config"
|
"github.com/sentry/sentry/api/internal/config"
|
||||||
|
"github.com/sentry/sentry/api/internal/querylang/executor"
|
||||||
"github.com/sentry/sentry/api/internal/queryapi"
|
"github.com/sentry/sentry/api/internal/queryapi"
|
||||||
"github.com/sentry/sentry/api/internal/searchclient"
|
"github.com/sentry/sentry/api/internal/searchclient"
|
||||||
)
|
)
|
||||||
@@ -58,8 +60,8 @@ func main() {
|
|||||||
}
|
}
|
||||||
defer search.Close()
|
defer search.Close()
|
||||||
|
|
||||||
exec := queryapi.NewExecutor(conn)
|
sqlRunner := executor.NewChRunner(conn)
|
||||||
handler := queryapi.NewHandler(logger, exec, search, cfg.QueryTimeout, cfg.CORSAllowedOrigin)
|
handler := queryapi.NewHandler(logger, sqlRunner, search, cfg.QueryTimeout, cfg.CORSAllowedOrigin)
|
||||||
|
|
||||||
srv := &http.Server{
|
srv := &http.Server{
|
||||||
Addr: cfg.HTTPListenAddr,
|
Addr: cfg.HTTPListenAddr,
|
||||||
|
|||||||
@@ -1,13 +1,14 @@
|
|||||||
// Package queryapi is Sentry's query API: POST /query (Phase 0, a crude
|
// Package queryapi is Sentry's query API: a single POST /query endpoint
|
||||||
// raw-SQL passthrough allowlisted to SELECT) and POST /search (Phase 1,
|
// accepting either the pipe syntax or raw SQL, compiled by
|
||||||
// free-text search via the search service, joined back against
|
// querylang/planner and executed by querylang/executor. Replaces Phase
|
||||||
// ClickHouse). This is a deliberate simplification of the pinned "gRPC +
|
// 0/1's two separate placeholder endpoints (raw-SQL-only /query,
|
||||||
// REST gateway" control-plane pattern (see CLAUDE.md's tech stack table):
|
// free-text-only /search) -- see /docs/query-language-design.md.
|
||||||
// plain net/http REST handlers, not a gRPC service transcoded through
|
//
|
||||||
// grpc-gateway. That machinery (proto definitions, googleapis
|
// Still plain net/http, not the pinned gRPC+REST-gateway pattern, for
|
||||||
// annotations, gateway codegen) doesn't buy much for two crude endpoints
|
// the same reason as Phase 0/1: this is one endpoint, and the
|
||||||
// that Phase 2's real SPL-like query layer replaces outright. Revisit
|
// proto/annotations/codegen machinery doesn't buy much at that size.
|
||||||
// gRPC+gateway once /api's endpoint count and lifespan justify it.
|
// `/api` does speak gRPC internally (to /search) — this simplification
|
||||||
|
// is about the public-facing surface only.
|
||||||
package queryapi
|
package queryapi
|
||||||
|
|
||||||
import (
|
import (
|
||||||
@@ -15,38 +16,34 @@ import (
|
|||||||
"encoding/json"
|
"encoding/json"
|
||||||
"log/slog"
|
"log/slog"
|
||||||
"net/http"
|
"net/http"
|
||||||
|
"strings"
|
||||||
"time"
|
"time"
|
||||||
)
|
|
||||||
|
|
||||||
// queryExecutor is the narrow interface handleQuery depends on, so tests
|
"github.com/sentry/sentry/api/internal/querylang/executor"
|
||||||
// can substitute a fake without a real ClickHouse connection. *Executor
|
"github.com/sentry/sentry/api/internal/querylang/planner"
|
||||||
// satisfies it.
|
)
|
||||||
type queryExecutor interface {
|
|
||||||
Execute(ctx context.Context, sql string) (*QueryResult, error)
|
|
||||||
}
|
|
||||||
|
|
||||||
type Handler struct {
|
type Handler struct {
|
||||||
logger *slog.Logger
|
logger *slog.Logger
|
||||||
exec queryExecutor
|
sqlRunner executor.SQLRunner
|
||||||
search searchClient
|
search executor.SearchClient
|
||||||
queryTimeout time.Duration
|
queryTimeout time.Duration
|
||||||
allowedOrigin string
|
allowedOrigin string
|
||||||
}
|
}
|
||||||
|
|
||||||
func NewHandler(logger *slog.Logger, exec queryExecutor, search searchClient, queryTimeout time.Duration, allowedOrigin string) *Handler {
|
func NewHandler(logger *slog.Logger, sqlRunner executor.SQLRunner, search executor.SearchClient, queryTimeout time.Duration, allowedOrigin string) *Handler {
|
||||||
return &Handler{logger: logger, exec: exec, search: search, queryTimeout: queryTimeout, allowedOrigin: allowedOrigin}
|
return &Handler{logger: logger, sqlRunner: sqlRunner, search: search, queryTimeout: queryTimeout, allowedOrigin: allowedOrigin}
|
||||||
}
|
}
|
||||||
|
|
||||||
func (h *Handler) Routes() http.Handler {
|
func (h *Handler) Routes() http.Handler {
|
||||||
mux := http.NewServeMux()
|
mux := http.NewServeMux()
|
||||||
mux.HandleFunc("POST /query", h.handleQuery)
|
mux.HandleFunc("POST /query", h.handleQuery)
|
||||||
mux.HandleFunc("POST /search", h.handleSearch)
|
|
||||||
mux.HandleFunc("GET /healthz", h.handleHealthz)
|
mux.HandleFunc("GET /healthz", h.handleHealthz)
|
||||||
return h.withCORS(mux)
|
return h.withCORS(mux)
|
||||||
}
|
}
|
||||||
|
|
||||||
// withCORS is deliberately permissive by default (see CORSAllowedOrigin in
|
// withCORS is deliberately permissive by default (see CORSAllowedOrigin in
|
||||||
// internal/config) since Phase 0 has no auth and the SvelteKit dev server
|
// internal/config) since there's no auth yet and the SvelteKit dev server
|
||||||
// runs on a different origin. Tighten alongside adding real auth.
|
// runs on a different origin. Tighten alongside adding real auth.
|
||||||
func (h *Handler) withCORS(next http.Handler) http.Handler {
|
func (h *Handler) withCORS(next http.Handler) http.Handler {
|
||||||
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||||
@@ -66,14 +63,24 @@ func (h *Handler) handleHealthz(w http.ResponseWriter, _ *http.Request) {
|
|||||||
}
|
}
|
||||||
|
|
||||||
type queryRequest struct {
|
type queryRequest struct {
|
||||||
SQL string `json:"sql"`
|
Query string `json:"query"`
|
||||||
|
// Language overrides auto-detection ("" / omitted). "sql" or "spl" --
|
||||||
|
// see planner.Language and /docs/query-language-design.md's
|
||||||
|
// "Detection" section for why this exists: the rare case a pipe
|
||||||
|
// query legitimately starts with the literal word "select".
|
||||||
|
Language string `json:"language"`
|
||||||
|
}
|
||||||
|
|
||||||
|
type queryResponse struct {
|
||||||
|
Columns []string `json:"columns"`
|
||||||
|
Rows [][]any `json:"rows"`
|
||||||
}
|
}
|
||||||
|
|
||||||
type errorResponse struct {
|
type errorResponse struct {
|
||||||
Error string `json:"error"`
|
Error string `json:"error"`
|
||||||
}
|
}
|
||||||
|
|
||||||
// maxBodyBytes caps the request body: a raw SQL string has no legitimate
|
// maxBodyBytes caps the request body: a query string has no legitimate
|
||||||
// reason to be larger than this.
|
// reason to be larger than this.
|
||||||
const maxBodyBytes = 1 << 20 // 1 MiB
|
const maxBodyBytes = 1 << 20 // 1 MiB
|
||||||
|
|
||||||
@@ -85,8 +92,19 @@ func (h *Handler) handleQuery(w http.ResponseWriter, r *http.Request) {
|
|||||||
writeError(w, http.StatusBadRequest, "invalid JSON body: "+err.Error())
|
writeError(w, http.StatusBadRequest, "invalid JSON body: "+err.Error())
|
||||||
return
|
return
|
||||||
}
|
}
|
||||||
|
if strings.TrimSpace(req.Query) == "" {
|
||||||
|
writeError(w, http.StatusBadRequest, "query must not be empty")
|
||||||
|
return
|
||||||
|
}
|
||||||
|
|
||||||
if err := validateSelectOnly(req.SQL); err != nil {
|
lang := planner.Language(req.Language)
|
||||||
|
if lang != planner.Auto && lang != planner.SQL && lang != planner.SPL {
|
||||||
|
writeError(w, http.StatusBadRequest, `language must be "sql", "spl", or omitted`)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
|
||||||
|
plan, err := planner.Compile(req.Query, lang, time.Now())
|
||||||
|
if err != nil {
|
||||||
writeError(w, http.StatusBadRequest, err.Error())
|
writeError(w, http.StatusBadRequest, err.Error())
|
||||||
return
|
return
|
||||||
}
|
}
|
||||||
@@ -94,14 +112,19 @@ func (h *Handler) handleQuery(w http.ResponseWriter, r *http.Request) {
|
|||||||
ctx, cancel := context.WithTimeout(r.Context(), h.queryTimeout)
|
ctx, cancel := context.WithTimeout(r.Context(), h.queryTimeout)
|
||||||
defer cancel()
|
defer cancel()
|
||||||
|
|
||||||
result, err := h.exec.Execute(ctx, req.SQL)
|
result, err := executor.Execute(ctx, plan, h.sqlRunner, h.search)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
h.logger.Error("query execution failed", "error", err)
|
h.logger.Error("query execution failed", "query", req.Query, "error", err)
|
||||||
writeError(w, http.StatusBadGateway, "query failed: "+err.Error())
|
writeError(w, http.StatusBadGateway, "query failed: "+err.Error())
|
||||||
return
|
return
|
||||||
}
|
}
|
||||||
|
|
||||||
writeJSON(w, result)
|
writeJSON(w, queryResponse{Columns: result.Columns, Rows: result.Rows})
|
||||||
|
}
|
||||||
|
|
||||||
|
func writeJSON(w http.ResponseWriter, v any) {
|
||||||
|
w.Header().Set("Content-Type", "application/json")
|
||||||
|
_ = json.NewEncoder(w).Encode(v)
|
||||||
}
|
}
|
||||||
|
|
||||||
func writeError(w http.ResponseWriter, status int, msg string) {
|
func writeError(w http.ResponseWriter, status int, msg string) {
|
||||||
|
|||||||
@@ -11,111 +11,179 @@ import (
|
|||||||
"strings"
|
"strings"
|
||||||
"testing"
|
"testing"
|
||||||
"time"
|
"time"
|
||||||
|
|
||||||
|
"github.com/sentry/sentry/api/internal/querylang/executor"
|
||||||
)
|
)
|
||||||
|
|
||||||
type fakeExecutor struct {
|
type fakeSQLRunner struct {
|
||||||
result *QueryResult
|
result *executor.Result
|
||||||
err error
|
err error
|
||||||
gotSQL string
|
gotSQL string
|
||||||
}
|
}
|
||||||
|
|
||||||
func (f *fakeExecutor) Execute(_ context.Context, sql string) (*QueryResult, error) {
|
func (f *fakeSQLRunner) RunSQL(_ context.Context, sql string) (*executor.Result, error) {
|
||||||
f.gotSQL = sql
|
f.gotSQL = sql
|
||||||
if f.err != nil {
|
if f.err != nil {
|
||||||
return nil, f.err
|
return nil, f.err
|
||||||
}
|
}
|
||||||
|
if f.result != nil {
|
||||||
return f.result, nil
|
return f.result, nil
|
||||||
}
|
}
|
||||||
|
return &executor.Result{Columns: []string{}, Rows: [][]any{}}, nil
|
||||||
|
}
|
||||||
|
|
||||||
type fakeSearchClient struct {
|
type fakeSearchClient struct {
|
||||||
recordIDs []string
|
recordIDs []string
|
||||||
err error
|
err error
|
||||||
|
gotQuery string
|
||||||
}
|
}
|
||||||
|
|
||||||
func (f *fakeSearchClient) Search(_ context.Context, _ string, _ uint32) ([]string, error) {
|
func (f *fakeSearchClient) Search(_ context.Context, query string, _ uint32) ([]string, error) {
|
||||||
|
f.gotQuery = query
|
||||||
if f.err != nil {
|
if f.err != nil {
|
||||||
return nil, f.err
|
return nil, f.err
|
||||||
}
|
}
|
||||||
return f.recordIDs, nil
|
return f.recordIDs, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
func newTestHandler(exec queryExecutor) *Handler {
|
func newTestHandler(sqlRunner *fakeSQLRunner, search *fakeSearchClient) *Handler {
|
||||||
return newTestHandlerWithSearch(exec, &fakeSearchClient{})
|
if search == nil {
|
||||||
|
search = &fakeSearchClient{}
|
||||||
|
}
|
||||||
|
return NewHandler(slog.New(slog.NewTextHandler(io.Discard, nil)), sqlRunner, search, time.Second, "*")
|
||||||
}
|
}
|
||||||
|
|
||||||
func newTestHandlerWithSearch(exec queryExecutor, search searchClient) *Handler {
|
func postQuery(t *testing.T, h *Handler, body string) *httptest.ResponseRecorder {
|
||||||
return NewHandler(slog.New(slog.NewTextHandler(io.Discard, nil)), exec, search, time.Second, "*")
|
t.Helper()
|
||||||
|
req := httptest.NewRequest(http.MethodPost, "/query", strings.NewReader(body))
|
||||||
|
rec := httptest.NewRecorder()
|
||||||
|
h.Routes().ServeHTTP(rec, req)
|
||||||
|
return rec
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestHandleQuerySuccess(t *testing.T) {
|
func TestHandleQuerySQLSuccess(t *testing.T) {
|
||||||
fe := &fakeExecutor{result: &QueryResult{
|
sr := &fakeSQLRunner{result: &executor.Result{
|
||||||
Columns: []string{"host", "count"},
|
Columns: []string{"host", "count"},
|
||||||
Rows: [][]any{{"h1", 3}},
|
Rows: [][]any{{"h1", 3}},
|
||||||
}}
|
}}
|
||||||
h := newTestHandler(fe)
|
h := newTestHandler(sr, nil)
|
||||||
|
|
||||||
body := strings.NewReader(`{"sql": "SELECT host, count(*) FROM logs GROUP BY host"}`)
|
rec := postQuery(t, h, `{"query": "SELECT host, count(*) FROM logs GROUP BY host"}`)
|
||||||
req := httptest.NewRequest(http.MethodPost, "/query", body)
|
|
||||||
rec := httptest.NewRecorder()
|
|
||||||
|
|
||||||
h.Routes().ServeHTTP(rec, req)
|
|
||||||
|
|
||||||
if rec.Code != http.StatusOK {
|
if rec.Code != http.StatusOK {
|
||||||
t.Fatalf("status = %d, want 200; body=%s", rec.Code, rec.Body.String())
|
t.Fatalf("status = %d, want 200; body=%s", rec.Code, rec.Body.String())
|
||||||
}
|
}
|
||||||
var got QueryResult
|
var got queryResponse
|
||||||
if err := json.Unmarshal(rec.Body.Bytes(), &got); err != nil {
|
if err := json.Unmarshal(rec.Body.Bytes(), &got); err != nil {
|
||||||
t.Fatalf("decoding response: %v", err)
|
t.Fatalf("decoding response: %v", err)
|
||||||
}
|
}
|
||||||
if len(got.Columns) != 2 || len(got.Rows) != 1 {
|
if len(got.Columns) != 2 || len(got.Rows) != 1 {
|
||||||
t.Fatalf("unexpected result: %+v", got)
|
t.Fatalf("unexpected result: %+v", got)
|
||||||
}
|
}
|
||||||
if fe.gotSQL != "SELECT host, count(*) FROM logs GROUP BY host" {
|
if sr.gotSQL != "SELECT host, count(*) FROM logs GROUP BY host" {
|
||||||
t.Fatalf("executor received unexpected SQL: %q", fe.gotSQL)
|
t.Fatalf("unexpected SQL passed through: %q", sr.gotSQL)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestHandleQueryRejectsNonSelect(t *testing.T) {
|
func TestHandleQueryPipeSyntaxSuccess(t *testing.T) {
|
||||||
fe := &fakeExecutor{}
|
sr := &fakeSQLRunner{result: &executor.Result{
|
||||||
h := newTestHandler(fe)
|
Columns: []string{"host"},
|
||||||
|
Rows: [][]any{{"api"}},
|
||||||
|
}}
|
||||||
|
h := newTestHandler(sr, nil)
|
||||||
|
|
||||||
body := strings.NewReader(`{"sql": "DELETE FROM logs"}`)
|
rec := postQuery(t, h, `{"query": "service=api"}`)
|
||||||
req := httptest.NewRequest(http.MethodPost, "/query", body)
|
|
||||||
rec := httptest.NewRecorder()
|
|
||||||
|
|
||||||
h.Routes().ServeHTTP(rec, req)
|
if rec.Code != http.StatusOK {
|
||||||
|
t.Fatalf("status = %d, want 200; body=%s", rec.Code, rec.Body.String())
|
||||||
|
}
|
||||||
|
if !strings.Contains(sr.gotSQL, "`service` = 'api'") {
|
||||||
|
t.Fatalf("expected compiled SQL to filter on service, got: %s", sr.gotSQL)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestHandleQueryTextSearchRoutesThroughSearchClient(t *testing.T) {
|
||||||
|
sr := &fakeSQLRunner{}
|
||||||
|
fs := &fakeSearchClient{recordIDs: []string{"id-1"}}
|
||||||
|
h := newTestHandler(sr, fs)
|
||||||
|
|
||||||
|
rec := postQuery(t, h, `{"query": "message:\"connection refused\""}`)
|
||||||
|
|
||||||
|
if rec.Code != http.StatusOK {
|
||||||
|
t.Fatalf("status = %d, want 200; body=%s", rec.Code, rec.Body.String())
|
||||||
|
}
|
||||||
|
if fs.gotQuery != `"connection refused"` {
|
||||||
|
t.Fatalf("search query = %q", fs.gotQuery)
|
||||||
|
}
|
||||||
|
if !strings.Contains(sr.gotSQL, "record_id IN ('id-1')") {
|
||||||
|
t.Fatalf("expected the search prefilter in the generated SQL, got: %s", sr.gotSQL)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestHandleQueryRejectsEmptyQuery(t *testing.T) {
|
||||||
|
h := newTestHandler(&fakeSQLRunner{}, nil)
|
||||||
|
rec := postQuery(t, h, `{"query": " "}`)
|
||||||
if rec.Code != http.StatusBadRequest {
|
if rec.Code != http.StatusBadRequest {
|
||||||
t.Fatalf("status = %d, want 400", rec.Code)
|
t.Fatalf("status = %d, want 400", rec.Code)
|
||||||
}
|
}
|
||||||
if fe.gotSQL != "" {
|
|
||||||
t.Fatal("executor should not have been called for a rejected query")
|
|
||||||
}
|
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestHandleQueryRejectsInvalidJSON(t *testing.T) {
|
func TestHandleQueryRejectsInvalidJSON(t *testing.T) {
|
||||||
h := newTestHandler(&fakeExecutor{})
|
h := newTestHandler(&fakeSQLRunner{}, nil)
|
||||||
|
rec := postQuery(t, h, `not json`)
|
||||||
body := strings.NewReader(`not json`)
|
|
||||||
req := httptest.NewRequest(http.MethodPost, "/query", body)
|
|
||||||
rec := httptest.NewRecorder()
|
|
||||||
|
|
||||||
h.Routes().ServeHTTP(rec, req)
|
|
||||||
|
|
||||||
if rec.Code != http.StatusBadRequest {
|
if rec.Code != http.StatusBadRequest {
|
||||||
t.Fatalf("status = %d, want 400", rec.Code)
|
t.Fatalf("status = %d, want 400", rec.Code)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func TestHandleQueryRejectsCompileError(t *testing.T) {
|
||||||
|
h := newTestHandler(&fakeSQLRunner{}, nil)
|
||||||
|
rec := postQuery(t, h, `{"query": "service=api | bogus"}`)
|
||||||
|
if rec.Code != http.StatusBadRequest {
|
||||||
|
t.Fatalf("status = %d, want 400; body=%s", rec.Code, rec.Body.String())
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestHandleQueryRejectsNonSelectSQL(t *testing.T) {
|
||||||
|
h := newTestHandler(&fakeSQLRunner{}, nil)
|
||||||
|
rec := postQuery(t, h, `{"query": "DELETE FROM logs", "language": "sql"}`)
|
||||||
|
if rec.Code != http.StatusBadRequest {
|
||||||
|
t.Fatalf("status = %d, want 400", rec.Code)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestHandleQueryRejectsInvalidLanguage(t *testing.T) {
|
||||||
|
h := newTestHandler(&fakeSQLRunner{}, nil)
|
||||||
|
rec := postQuery(t, h, `{"query": "service=api", "language": "cobol"}`)
|
||||||
|
if rec.Code != http.StatusBadRequest {
|
||||||
|
t.Fatalf("status = %d, want 400", rec.Code)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestHandleQueryExplicitLanguageOverridesAutoDetect(t *testing.T) {
|
||||||
|
sr := &fakeSQLRunner{}
|
||||||
|
fs := &fakeSearchClient{recordIDs: []string{"id-1"}}
|
||||||
|
h := newTestHandler(sr, fs)
|
||||||
|
|
||||||
|
// "select" alone would auto-detect as (nonsensical but
|
||||||
|
// syntactically-valid-looking) SQL without the override -- the
|
||||||
|
// override forces pipe-syntax parsing instead, where a bare word
|
||||||
|
// with no comparator is a free-text search term.
|
||||||
|
rec := postQuery(t, h, `{"query": "select", "language": "spl"}`)
|
||||||
|
|
||||||
|
if rec.Code != http.StatusOK {
|
||||||
|
t.Fatalf("status = %d, want 200; body=%s", rec.Code, rec.Body.String())
|
||||||
|
}
|
||||||
|
if fs.gotQuery != "select" {
|
||||||
|
t.Fatalf("expected 'select' to be treated as a free-text search term, got query=%q", fs.gotQuery)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
func TestHandleQueryExecutorErrorReturnsBadGateway(t *testing.T) {
|
func TestHandleQueryExecutorErrorReturnsBadGateway(t *testing.T) {
|
||||||
fe := &fakeExecutor{err: errors.New("boom")}
|
sr := &fakeSQLRunner{err: errors.New("boom")}
|
||||||
h := newTestHandler(fe)
|
h := newTestHandler(sr, nil)
|
||||||
|
|
||||||
body := strings.NewReader(`{"sql": "SELECT 1"}`)
|
rec := postQuery(t, h, `{"query": "SELECT 1"}`)
|
||||||
req := httptest.NewRequest(http.MethodPost, "/query", body)
|
|
||||||
rec := httptest.NewRecorder()
|
|
||||||
|
|
||||||
h.Routes().ServeHTTP(rec, req)
|
|
||||||
|
|
||||||
if rec.Code != http.StatusBadGateway {
|
if rec.Code != http.StatusBadGateway {
|
||||||
t.Fatalf("status = %d, want 502", rec.Code)
|
t.Fatalf("status = %d, want 502", rec.Code)
|
||||||
@@ -123,7 +191,7 @@ func TestHandleQueryExecutorErrorReturnsBadGateway(t *testing.T) {
|
|||||||
}
|
}
|
||||||
|
|
||||||
func TestHandleHealthz(t *testing.T) {
|
func TestHandleHealthz(t *testing.T) {
|
||||||
h := newTestHandler(&fakeExecutor{})
|
h := newTestHandler(&fakeSQLRunner{}, nil)
|
||||||
req := httptest.NewRequest(http.MethodGet, "/healthz", nil)
|
req := httptest.NewRequest(http.MethodGet, "/healthz", nil)
|
||||||
rec := httptest.NewRecorder()
|
rec := httptest.NewRecorder()
|
||||||
|
|
||||||
@@ -135,7 +203,7 @@ func TestHandleHealthz(t *testing.T) {
|
|||||||
}
|
}
|
||||||
|
|
||||||
func TestCORSPreflight(t *testing.T) {
|
func TestCORSPreflight(t *testing.T) {
|
||||||
h := newTestHandler(&fakeExecutor{})
|
h := newTestHandler(&fakeSQLRunner{}, nil)
|
||||||
req := httptest.NewRequest(http.MethodOptions, "/query", nil)
|
req := httptest.NewRequest(http.MethodOptions, "/query", nil)
|
||||||
rec := httptest.NewRecorder()
|
rec := httptest.NewRecorder()
|
||||||
|
|
||||||
|
|||||||
@@ -1,96 +0,0 @@
|
|||||||
package queryapi
|
|
||||||
|
|
||||||
import (
|
|
||||||
"context"
|
|
||||||
"encoding/json"
|
|
||||||
"fmt"
|
|
||||||
"net/http"
|
|
||||||
"strings"
|
|
||||||
|
|
||||||
"github.com/google/uuid"
|
|
||||||
)
|
|
||||||
|
|
||||||
// searchClient is the narrow interface handleSearch depends on, so tests
|
|
||||||
// can substitute a fake without a real search service. A small gRPC
|
|
||||||
// adapter in cmd/api satisfies this.
|
|
||||||
type searchClient interface {
|
|
||||||
Search(ctx context.Context, query string, limit uint32) ([]string, error)
|
|
||||||
}
|
|
||||||
|
|
||||||
type searchRequest struct {
|
|
||||||
Query string `json:"query"`
|
|
||||||
Limit uint32 `json:"limit"`
|
|
||||||
}
|
|
||||||
|
|
||||||
func (h *Handler) handleSearch(w http.ResponseWriter, r *http.Request) {
|
|
||||||
r.Body = http.MaxBytesReader(w, r.Body, maxBodyBytes)
|
|
||||||
|
|
||||||
var req searchRequest
|
|
||||||
if err := json.NewDecoder(r.Body).Decode(&req); err != nil {
|
|
||||||
writeError(w, http.StatusBadRequest, "invalid JSON body: "+err.Error())
|
|
||||||
return
|
|
||||||
}
|
|
||||||
if strings.TrimSpace(req.Query) == "" {
|
|
||||||
writeError(w, http.StatusBadRequest, "query must not be empty")
|
|
||||||
return
|
|
||||||
}
|
|
||||||
|
|
||||||
ctx, cancel := context.WithTimeout(r.Context(), h.queryTimeout)
|
|
||||||
defer cancel()
|
|
||||||
|
|
||||||
recordIDs, err := h.search.Search(ctx, req.Query, req.Limit)
|
|
||||||
if err != nil {
|
|
||||||
h.logger.Error("search failed", "error", err)
|
|
||||||
writeError(w, http.StatusBadGateway, "search failed: "+err.Error())
|
|
||||||
return
|
|
||||||
}
|
|
||||||
|
|
||||||
if len(recordIDs) == 0 {
|
|
||||||
writeJSON(w, &QueryResult{Columns: []string{}, Rows: [][]any{}})
|
|
||||||
return
|
|
||||||
}
|
|
||||||
|
|
||||||
sql, err := recordIDsQuery(recordIDs)
|
|
||||||
if err != nil {
|
|
||||||
h.logger.Error("building record_id query", "error", err)
|
|
||||||
writeError(w, http.StatusBadGateway, "search returned unusable results")
|
|
||||||
return
|
|
||||||
}
|
|
||||||
|
|
||||||
result, err := h.exec.Execute(ctx, sql)
|
|
||||||
if err != nil {
|
|
||||||
h.logger.Error("joining search results against clickhouse failed", "error", err)
|
|
||||||
writeError(w, http.StatusBadGateway, "query failed: "+err.Error())
|
|
||||||
return
|
|
||||||
}
|
|
||||||
|
|
||||||
writeJSON(w, result)
|
|
||||||
}
|
|
||||||
|
|
||||||
// recordIDsQuery builds a SELECT ... WHERE record_id IN (...) against the
|
|
||||||
// IDs the search service returned. Every ID is validated as a real UUID
|
|
||||||
// before being embedded in the query string -- record_ids come from an
|
|
||||||
// internal, trusted service (not raw user input), but a UUID that fails
|
|
||||||
// to parse can't contain SQL-breaking characters either way, so this is
|
|
||||||
// defense in depth, not a response to a specific threat.
|
|
||||||
func recordIDsQuery(recordIDs []string) (string, error) {
|
|
||||||
quoted := make([]string, 0, len(recordIDs))
|
|
||||||
for _, id := range recordIDs {
|
|
||||||
if _, err := uuid.Parse(id); err != nil {
|
|
||||||
continue // skip anything not a valid UUID rather than failing the whole query
|
|
||||||
}
|
|
||||||
quoted = append(quoted, "'"+id+"'")
|
|
||||||
}
|
|
||||||
if len(quoted) == 0 {
|
|
||||||
return "", fmt.Errorf("no valid record_ids in search response")
|
|
||||||
}
|
|
||||||
return fmt.Sprintf(
|
|
||||||
"SELECT * FROM logs WHERE record_id IN (%s) ORDER BY timestamp DESC",
|
|
||||||
strings.Join(quoted, ","),
|
|
||||||
), nil
|
|
||||||
}
|
|
||||||
|
|
||||||
func writeJSON(w http.ResponseWriter, v any) {
|
|
||||||
w.Header().Set("Content-Type", "application/json")
|
|
||||||
_ = json.NewEncoder(w).Encode(v)
|
|
||||||
}
|
|
||||||
@@ -1,125 +0,0 @@
|
|||||||
package queryapi
|
|
||||||
|
|
||||||
import (
|
|
||||||
"encoding/json"
|
|
||||||
"errors"
|
|
||||||
"net/http"
|
|
||||||
"net/http/httptest"
|
|
||||||
"strings"
|
|
||||||
"testing"
|
|
||||||
)
|
|
||||||
|
|
||||||
func TestHandleSearchSuccess(t *testing.T) {
|
|
||||||
id := "5754b062-ec8b-45b1-b1b8-a50f263adcd3"
|
|
||||||
fe := &fakeExecutor{result: &QueryResult{
|
|
||||||
Columns: []string{"message"},
|
|
||||||
Rows: [][]any{{"hello world"}},
|
|
||||||
}}
|
|
||||||
fs := &fakeSearchClient{recordIDs: []string{id}}
|
|
||||||
h := newTestHandlerWithSearch(fe, fs)
|
|
||||||
|
|
||||||
body := strings.NewReader(`{"query": "hello"}`)
|
|
||||||
req := httptest.NewRequest(http.MethodPost, "/search", body)
|
|
||||||
rec := httptest.NewRecorder()
|
|
||||||
|
|
||||||
h.Routes().ServeHTTP(rec, req)
|
|
||||||
|
|
||||||
if rec.Code != http.StatusOK {
|
|
||||||
t.Fatalf("status = %d, want 200; body=%s", rec.Code, rec.Body.String())
|
|
||||||
}
|
|
||||||
if !strings.Contains(fe.gotSQL, id) {
|
|
||||||
t.Fatalf("expected the record_id in the generated SQL, got %q", fe.gotSQL)
|
|
||||||
}
|
|
||||||
if !strings.Contains(fe.gotSQL, "WHERE record_id IN") {
|
|
||||||
t.Fatalf("expected an IN clause, got %q", fe.gotSQL)
|
|
||||||
}
|
|
||||||
|
|
||||||
var got QueryResult
|
|
||||||
if err := json.Unmarshal(rec.Body.Bytes(), &got); err != nil {
|
|
||||||
t.Fatalf("decoding response: %v", err)
|
|
||||||
}
|
|
||||||
if len(got.Rows) != 1 {
|
|
||||||
t.Fatalf("unexpected result: %+v", got)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestHandleSearchRejectsEmptyQuery(t *testing.T) {
|
|
||||||
fe := &fakeExecutor{}
|
|
||||||
fs := &fakeSearchClient{}
|
|
||||||
h := newTestHandlerWithSearch(fe, fs)
|
|
||||||
|
|
||||||
body := strings.NewReader(`{"query": " "}`)
|
|
||||||
req := httptest.NewRequest(http.MethodPost, "/search", body)
|
|
||||||
rec := httptest.NewRecorder()
|
|
||||||
|
|
||||||
h.Routes().ServeHTTP(rec, req)
|
|
||||||
|
|
||||||
if rec.Code != http.StatusBadRequest {
|
|
||||||
t.Fatalf("status = %d, want 400", rec.Code)
|
|
||||||
}
|
|
||||||
if fe.gotSQL != "" {
|
|
||||||
t.Fatal("executor should not have been called for an empty query")
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestHandleSearchNoResultsReturnsEmptyNotError(t *testing.T) {
|
|
||||||
fe := &fakeExecutor{}
|
|
||||||
fs := &fakeSearchClient{recordIDs: nil}
|
|
||||||
h := newTestHandlerWithSearch(fe, fs)
|
|
||||||
|
|
||||||
body := strings.NewReader(`{"query": "nothing matches this"}`)
|
|
||||||
req := httptest.NewRequest(http.MethodPost, "/search", body)
|
|
||||||
rec := httptest.NewRecorder()
|
|
||||||
|
|
||||||
h.Routes().ServeHTTP(rec, req)
|
|
||||||
|
|
||||||
if rec.Code != http.StatusOK {
|
|
||||||
t.Fatalf("status = %d, want 200; body=%s", rec.Code, rec.Body.String())
|
|
||||||
}
|
|
||||||
if fe.gotSQL != "" {
|
|
||||||
t.Fatal("executor should not have been called when search returns no IDs")
|
|
||||||
}
|
|
||||||
|
|
||||||
var got QueryResult
|
|
||||||
if err := json.Unmarshal(rec.Body.Bytes(), &got); err != nil {
|
|
||||||
t.Fatalf("decoding response: %v", err)
|
|
||||||
}
|
|
||||||
if len(got.Rows) != 0 {
|
|
||||||
t.Fatalf("expected empty rows, got %+v", got.Rows)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestHandleSearchServiceErrorReturnsBadGateway(t *testing.T) {
|
|
||||||
fe := &fakeExecutor{}
|
|
||||||
fs := &fakeSearchClient{err: errors.New("search service unreachable")}
|
|
||||||
h := newTestHandlerWithSearch(fe, fs)
|
|
||||||
|
|
||||||
body := strings.NewReader(`{"query": "hello"}`)
|
|
||||||
req := httptest.NewRequest(http.MethodPost, "/search", body)
|
|
||||||
rec := httptest.NewRecorder()
|
|
||||||
|
|
||||||
h.Routes().ServeHTTP(rec, req)
|
|
||||||
|
|
||||||
if rec.Code != http.StatusBadGateway {
|
|
||||||
t.Fatalf("status = %d, want 502", rec.Code)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestRecordIDsQuerySkipsInvalidUUIDs(t *testing.T) {
|
|
||||||
sql, err := recordIDsQuery([]string{"not-a-uuid", "5754b062-ec8b-45b1-b1b8-a50f263adcd3"})
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("recordIDsQuery() error = %v", err)
|
|
||||||
}
|
|
||||||
if strings.Contains(sql, "not-a-uuid") {
|
|
||||||
t.Fatalf("expected the invalid UUID to be skipped, got %q", sql)
|
|
||||||
}
|
|
||||||
if !strings.Contains(sql, "5754b062-ec8b-45b1-b1b8-a50f263adcd3") {
|
|
||||||
t.Fatalf("expected the valid UUID to be included, got %q", sql)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestRecordIDsQueryAllInvalidReturnsError(t *testing.T) {
|
|
||||||
if _, err := recordIDsQuery([]string{"not-a-uuid", "also-not-one"}); err == nil {
|
|
||||||
t.Fatal("expected an error when no IDs are valid UUIDs")
|
|
||||||
}
|
|
||||||
}
|
|
||||||
@@ -1,47 +0,0 @@
|
|||||||
package queryapi
|
|
||||||
|
|
||||||
import (
|
|
||||||
"errors"
|
|
||||||
"regexp"
|
|
||||||
"strings"
|
|
||||||
)
|
|
||||||
|
|
||||||
// disallowedKeyword is defense-in-depth on top of the SELECT-only gate: it
|
|
||||||
// catches mutating/administrative statements appearing anywhere in the
|
|
||||||
// query (e.g. smuggled into a subquery), not just at the start. This is
|
|
||||||
// word-boundary matching, not a real SQL parser.
|
|
||||||
var disallowedKeyword = regexp.MustCompile(`(?i)\b(insert|update|delete|alter|drop|truncate|create|grant|revoke|attach|detach|rename|kill|optimize|system|set|exchange|watch)\b`)
|
|
||||||
|
|
||||||
// validateSelectOnly enforces the Phase 0 query API contract: exactly one
|
|
||||||
// SELECT statement and nothing else. This is "basic injection guarding" as
|
|
||||||
// specced, not a SQL parser: it will reject some unusual-but-valid SELECTs
|
|
||||||
// (e.g. one that references a column literally named "delete") and will
|
|
||||||
// not catch every possible abuse (e.g. a syntactically pure SELECT that's
|
|
||||||
// simply expensive to run). Both are acceptable for a Phase 0 placeholder
|
|
||||||
// that's explicitly superseded by a real query layer in Phase 2 — see
|
|
||||||
// /docs/architecture.md.
|
|
||||||
func validateSelectOnly(sql string) error {
|
|
||||||
trimmed := strings.TrimSpace(sql)
|
|
||||||
if trimmed == "" {
|
|
||||||
return errors.New("query must not be empty")
|
|
||||||
}
|
|
||||||
|
|
||||||
trimmed = strings.TrimSpace(strings.TrimSuffix(trimmed, ";"))
|
|
||||||
if trimmed == "" {
|
|
||||||
return errors.New("query must not be empty")
|
|
||||||
}
|
|
||||||
if strings.Contains(trimmed, ";") {
|
|
||||||
return errors.New("only a single statement is allowed")
|
|
||||||
}
|
|
||||||
|
|
||||||
firstWord := strings.ToUpper(strings.Fields(trimmed)[0])
|
|
||||||
if firstWord != "SELECT" {
|
|
||||||
return errors.New("only SELECT queries are allowed")
|
|
||||||
}
|
|
||||||
|
|
||||||
if disallowedKeyword.MatchString(trimmed) {
|
|
||||||
return errors.New("query contains a disallowed keyword")
|
|
||||||
}
|
|
||||||
|
|
||||||
return nil
|
|
||||||
}
|
|
||||||
@@ -1,38 +0,0 @@
|
|||||||
package queryapi
|
|
||||||
|
|
||||||
import "testing"
|
|
||||||
|
|
||||||
func TestValidateSelectOnly(t *testing.T) {
|
|
||||||
cases := []struct {
|
|
||||||
name string
|
|
||||||
sql string
|
|
||||||
wantErr bool
|
|
||||||
}{
|
|
||||||
{"plain select", "SELECT * FROM logs LIMIT 10", false},
|
|
||||||
{"lowercase select", "select service, count(*) from logs group by service", false},
|
|
||||||
{"trailing semicolon allowed", "SELECT 1;", false},
|
|
||||||
{"trailing semicolon and whitespace allowed", "SELECT 1; ", false},
|
|
||||||
{"empty", "", true},
|
|
||||||
{"whitespace only", " ", true},
|
|
||||||
{"only a semicolon", ";", true},
|
|
||||||
{"multiple statements", "SELECT 1; SELECT 2", true},
|
|
||||||
{"insert", "INSERT INTO logs VALUES (1)", true},
|
|
||||||
{"delete", "DELETE FROM logs", true},
|
|
||||||
{"drop", "DROP TABLE logs", true},
|
|
||||||
{"select with drop keyword smuggled in", "SELECT * FROM logs WHERE message = 'DROP TABLE logs'", true},
|
|
||||||
{"non-select start", "WITH x AS (SELECT 1) SELECT * FROM x", true},
|
|
||||||
{"trailing garbage after semicolon", "SELECT 1; DROP TABLE logs", true},
|
|
||||||
}
|
|
||||||
|
|
||||||
for _, tc := range cases {
|
|
||||||
t.Run(tc.name, func(t *testing.T) {
|
|
||||||
err := validateSelectOnly(tc.sql)
|
|
||||||
if tc.wantErr && err == nil {
|
|
||||||
t.Errorf("validateSelectOnly(%q) = nil, want error", tc.sql)
|
|
||||||
}
|
|
||||||
if !tc.wantErr && err != nil {
|
|
||||||
t.Errorf("validateSelectOnly(%q) = %v, want nil", tc.sql, err)
|
|
||||||
}
|
|
||||||
})
|
|
||||||
}
|
|
||||||
}
|
|
||||||
@@ -0,0 +1,91 @@
|
|||||||
|
// Package ast defines the parsed pipe-syntax tree. Internal to
|
||||||
|
// querylang -- not exposed outside it. See
|
||||||
|
// /docs/query-language-design.md for the grammar this mirrors.
|
||||||
|
package ast
|
||||||
|
|
||||||
|
// Query is `base_search ("|" pipe_stage)*`.
|
||||||
|
type Query struct {
|
||||||
|
Base BoolExpr
|
||||||
|
Pipes []PipeStage
|
||||||
|
}
|
||||||
|
|
||||||
|
// PipeStage is one of WhereStage, StatsStage, SortStage, FieldsStage,
|
||||||
|
// HeadStage, TailStage.
|
||||||
|
type PipeStage interface{ isPipeStage() }
|
||||||
|
|
||||||
|
type WhereStage struct{ Expr BoolExpr }
|
||||||
|
type StatsStage struct {
|
||||||
|
Aggs []AggCall
|
||||||
|
By []string
|
||||||
|
}
|
||||||
|
type SortStage struct{ Fields []SortField }
|
||||||
|
type FieldsStage struct{ Fields []string }
|
||||||
|
type HeadStage struct {
|
||||||
|
N int
|
||||||
|
HasN bool // false => default limit, decided by the planner
|
||||||
|
}
|
||||||
|
type TailStage struct {
|
||||||
|
N int
|
||||||
|
HasN bool
|
||||||
|
}
|
||||||
|
|
||||||
|
func (WhereStage) isPipeStage() {}
|
||||||
|
func (StatsStage) isPipeStage() {}
|
||||||
|
func (SortStage) isPipeStage() {}
|
||||||
|
func (FieldsStage) isPipeStage() {}
|
||||||
|
func (HeadStage) isPipeStage() {}
|
||||||
|
func (TailStage) isPipeStage() {}
|
||||||
|
|
||||||
|
// BoolExpr is a sequence of terms joined by "and"/"or". An empty Conjs
|
||||||
|
// entry between two terms (i.e. no explicit keyword in the source) means
|
||||||
|
// implicit AND -- SPL's convention for adjacent bare search terms, e.g.
|
||||||
|
// `error timeout` means `error AND timeout`. Conjs has len(Terms)-1
|
||||||
|
// elements once Terms has more than one.
|
||||||
|
type BoolExpr struct {
|
||||||
|
Terms []Term
|
||||||
|
Conjs []string // "and" | "or", one per gap between consecutive Terms
|
||||||
|
}
|
||||||
|
|
||||||
|
// Term is one of Comparison, TimeBound, FreeText.
|
||||||
|
type Term interface{ isTerm() }
|
||||||
|
|
||||||
|
type Comparison struct {
|
||||||
|
Field string
|
||||||
|
Op string // "=", "!=", ">", ">=", "<", "<="
|
||||||
|
Value string
|
||||||
|
}
|
||||||
|
|
||||||
|
type TimeBound struct {
|
||||||
|
Kind string // "earliest" | "latest"
|
||||||
|
Expr TimeExpr
|
||||||
|
}
|
||||||
|
|
||||||
|
type FreeText struct {
|
||||||
|
Query string
|
||||||
|
}
|
||||||
|
|
||||||
|
func (Comparison) isTerm() {}
|
||||||
|
func (TimeBound) isTerm() {}
|
||||||
|
func (FreeText) isTerm() {}
|
||||||
|
|
||||||
|
// TimeExpr is either an absolute RFC3339 timestamp or a relative offset
|
||||||
|
// like -1h/-7d, resolved to an absolute time by the planner (relative to
|
||||||
|
// compile time), not the parser -- the parser has no notion of "now".
|
||||||
|
type TimeExpr struct {
|
||||||
|
Absolute string
|
||||||
|
IsRelative bool
|
||||||
|
RelativeSign int // -1 or +1
|
||||||
|
RelativeN int
|
||||||
|
RelativeUnit string // "s" | "m" | "h" | "d" | "w"
|
||||||
|
}
|
||||||
|
|
||||||
|
type AggCall struct {
|
||||||
|
Func string // count, sum, avg, min, max
|
||||||
|
Field string // empty for count()/count(*)
|
||||||
|
Alias string // empty => planner assigns a default alias
|
||||||
|
}
|
||||||
|
|
||||||
|
type SortField struct {
|
||||||
|
Field string
|
||||||
|
Desc bool
|
||||||
|
}
|
||||||
@@ -1,4 +1,4 @@
|
|||||||
package queryapi
|
package executor
|
||||||
|
|
||||||
import (
|
import (
|
||||||
"context"
|
"context"
|
||||||
@@ -8,32 +8,30 @@ import (
|
|||||||
"github.com/ClickHouse/clickhouse-go/v2/lib/driver"
|
"github.com/ClickHouse/clickhouse-go/v2/lib/driver"
|
||||||
)
|
)
|
||||||
|
|
||||||
type QueryResult struct {
|
// ChRunner runs arbitrary (pre-validated) SELECT statements against
|
||||||
Columns []string `json:"columns"`
|
|
||||||
Rows [][]any `json:"rows"`
|
|
||||||
}
|
|
||||||
|
|
||||||
// Executor runs arbitrary (pre-validated) SELECT statements against
|
|
||||||
// ClickHouse and shapes the result into JSON-friendly columns/rows,
|
// ClickHouse and shapes the result into JSON-friendly columns/rows,
|
||||||
// discovering the result's column set at query time via reflection since
|
// discovering the result's column set at query time via reflection since
|
||||||
// the query itself is arbitrary.
|
// the query itself is arbitrary. Ported from Phase 0/1's
|
||||||
type Executor struct {
|
// api/internal/queryapi.Executor, which this replaces (see task 4) --
|
||||||
|
// same logic, moved here since it's the query-execution layer's
|
||||||
|
// plumbing, not specific to the old placeholder /query handler.
|
||||||
|
type ChRunner struct {
|
||||||
conn driver.Conn
|
conn driver.Conn
|
||||||
}
|
}
|
||||||
|
|
||||||
func NewExecutor(conn driver.Conn) *Executor {
|
func NewChRunner(conn driver.Conn) *ChRunner {
|
||||||
return &Executor{conn: conn}
|
return &ChRunner{conn: conn}
|
||||||
}
|
}
|
||||||
|
|
||||||
func (e *Executor) Execute(ctx context.Context, sql string) (*QueryResult, error) {
|
func (r *ChRunner) RunSQL(ctx context.Context, sql string) (*Result, error) {
|
||||||
rows, err := e.conn.Query(ctx, sql)
|
rows, err := r.conn.Query(ctx, sql)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return nil, fmt.Errorf("executing query: %w", err)
|
return nil, fmt.Errorf("executing query: %w", err)
|
||||||
}
|
}
|
||||||
defer rows.Close()
|
defer rows.Close()
|
||||||
|
|
||||||
columnTypes := rows.ColumnTypes()
|
columnTypes := rows.ColumnTypes()
|
||||||
result := &QueryResult{
|
result := &Result{
|
||||||
Columns: rows.Columns(),
|
Columns: rows.Columns(),
|
||||||
Rows: [][]any{},
|
Rows: [][]any{},
|
||||||
}
|
}
|
||||||
@@ -0,0 +1,72 @@
|
|||||||
|
// Package executor runs a compiled ir.Plan and returns results in a
|
||||||
|
// shape consistent regardless of which backend(s) were hit -- the point
|
||||||
|
// of compiling to one IR in the first place. See
|
||||||
|
// /docs/query-language-design.md's "Execution" section for the four
|
||||||
|
// routing cases implemented here.
|
||||||
|
package executor
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"fmt"
|
||||||
|
|
||||||
|
"github.com/sentry/sentry/api/internal/querylang/ir"
|
||||||
|
)
|
||||||
|
|
||||||
|
type Result struct {
|
||||||
|
Columns []string
|
||||||
|
Rows [][]any
|
||||||
|
}
|
||||||
|
|
||||||
|
// SQLRunner executes a raw SQL statement against ClickHouse. *ChRunner
|
||||||
|
// (chrunner.go) is the production implementation; tests use a fake --
|
||||||
|
// same narrow-interface pattern used throughout /ingest and /api.
|
||||||
|
type SQLRunner interface {
|
||||||
|
RunSQL(ctx context.Context, sql string) (*Result, error)
|
||||||
|
}
|
||||||
|
|
||||||
|
// SearchClient resolves a Tantivy query into matching record_ids.
|
||||||
|
type SearchClient interface {
|
||||||
|
Search(ctx context.Context, query string, limit uint32) ([]string, error)
|
||||||
|
}
|
||||||
|
|
||||||
|
// textSearchLimit caps how many record_ids a Tantivy prefilter can feed
|
||||||
|
// into a ClickHouse `IN (...)` clause. See /docs/query-language-design.md's
|
||||||
|
// "Known scaling limitation" -- this is a real, disclosed limit on result
|
||||||
|
// completeness for very broad text searches, not an oversight.
|
||||||
|
//
|
||||||
|
// 5000, not 10000: confirmed by actually running the Phase 2 benchmark
|
||||||
|
// (see /docs/phase-2-runbook.md) that 10000 quoted UUIDs (~39 bytes each
|
||||||
|
// including the comma) produces a ~390KB query string, which exceeds
|
||||||
|
// ClickHouse's default max_query_size (262144 bytes / 256KiB) and fails
|
||||||
|
// outright with a syntax error rather than degrading gracefully. 5000
|
||||||
|
// UUIDs is ~195KB, safely under that default with headroom for the rest
|
||||||
|
// of the query. This was a real failure caught by running the benchmark,
|
||||||
|
// not a value chosen from first-principles estimation.
|
||||||
|
const textSearchLimit = 5000
|
||||||
|
|
||||||
|
// Execute runs plan against the given backends. The four cases (per the
|
||||||
|
// design doc): RawSQL passthrough; pure ClickHouse (no TextSearch); text
|
||||||
|
// search alone (Tantivy prefilter -> ClickHouse row fetch); text search
|
||||||
|
// plus aggregation (Tantivy prefilter -> ClickHouse aggregate). Cases 2-4
|
||||||
|
// share the same buildSQL/buildWhereClause code (sql.go) -- the only
|
||||||
|
// difference is whether a record_id filter is threaded in.
|
||||||
|
func Execute(ctx context.Context, plan *ir.Plan, sqlRunner SQLRunner, search SearchClient) (*Result, error) {
|
||||||
|
if plan.RawSQL != "" {
|
||||||
|
return sqlRunner.RunSQL(ctx, plan.RawSQL)
|
||||||
|
}
|
||||||
|
|
||||||
|
var recordIDFilter []string
|
||||||
|
if len(plan.TextSearch) > 0 {
|
||||||
|
ids, err := search.Search(ctx, plan.TextSearch[0].Query, textSearchLimit)
|
||||||
|
if err != nil {
|
||||||
|
return nil, fmt.Errorf("full-text search failed: %w", err)
|
||||||
|
}
|
||||||
|
if len(ids) == 0 {
|
||||||
|
return &Result{Columns: []string{}, Rows: [][]any{}}, nil
|
||||||
|
}
|
||||||
|
recordIDFilter = ids
|
||||||
|
}
|
||||||
|
|
||||||
|
sql := buildSQL(plan, recordIDFilter)
|
||||||
|
return sqlRunner.RunSQL(ctx, sql)
|
||||||
|
}
|
||||||
@@ -0,0 +1,309 @@
|
|||||||
|
package executor
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"errors"
|
||||||
|
"strings"
|
||||||
|
"testing"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"github.com/sentry/sentry/api/internal/querylang/ir"
|
||||||
|
)
|
||||||
|
|
||||||
|
func mustParseTime(t *testing.T, s string) time.Time {
|
||||||
|
t.Helper()
|
||||||
|
tm, err := time.Parse(time.RFC3339, s)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("parsing time %q: %v", s, err)
|
||||||
|
}
|
||||||
|
return tm
|
||||||
|
}
|
||||||
|
|
||||||
|
type fakeSQLRunner struct {
|
||||||
|
gotSQL string
|
||||||
|
result *Result
|
||||||
|
err error
|
||||||
|
calls int
|
||||||
|
}
|
||||||
|
|
||||||
|
func (f *fakeSQLRunner) RunSQL(_ context.Context, sql string) (*Result, error) {
|
||||||
|
f.gotSQL = sql
|
||||||
|
f.calls++
|
||||||
|
if f.err != nil {
|
||||||
|
return nil, f.err
|
||||||
|
}
|
||||||
|
if f.result != nil {
|
||||||
|
return f.result, nil
|
||||||
|
}
|
||||||
|
return &Result{Columns: []string{}, Rows: [][]any{}}, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
type fakeSearchClient struct {
|
||||||
|
gotQuery string
|
||||||
|
gotLimit uint32
|
||||||
|
ids []string
|
||||||
|
err error
|
||||||
|
calls int
|
||||||
|
}
|
||||||
|
|
||||||
|
func (f *fakeSearchClient) Search(_ context.Context, query string, limit uint32) ([]string, error) {
|
||||||
|
f.gotQuery = query
|
||||||
|
f.gotLimit = limit
|
||||||
|
f.calls++
|
||||||
|
if f.err != nil {
|
||||||
|
return nil, f.err
|
||||||
|
}
|
||||||
|
return f.ids, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestExecuteRawSQLBypassesEverythingElse(t *testing.T) {
|
||||||
|
sqlRunner := &fakeSQLRunner{}
|
||||||
|
search := &fakeSearchClient{}
|
||||||
|
plan := &ir.Plan{RawSQL: "SELECT 1"}
|
||||||
|
|
||||||
|
_, err := Execute(context.Background(), plan, sqlRunner, search)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("Execute() error = %v", err)
|
||||||
|
}
|
||||||
|
if sqlRunner.gotSQL != "SELECT 1" {
|
||||||
|
t.Fatalf("gotSQL = %q, want %q", sqlRunner.gotSQL, "SELECT 1")
|
||||||
|
}
|
||||||
|
if search.calls != 0 {
|
||||||
|
t.Fatalf("expected search not to be called for RawSQL, got %d calls", search.calls)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestExecutePureClickHousePathSkipsSearch(t *testing.T) {
|
||||||
|
sqlRunner := &fakeSQLRunner{}
|
||||||
|
search := &fakeSearchClient{}
|
||||||
|
plan := &ir.Plan{Filters: []ir.FilterPredicate{{Field: "service", Op: "=", Value: "api"}}}
|
||||||
|
|
||||||
|
_, err := Execute(context.Background(), plan, sqlRunner, search)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("Execute() error = %v", err)
|
||||||
|
}
|
||||||
|
if search.calls != 0 {
|
||||||
|
t.Fatalf("expected no search calls, got %d", search.calls)
|
||||||
|
}
|
||||||
|
if !strings.Contains(sqlRunner.gotSQL, "FROM logs") || !strings.Contains(sqlRunner.gotSQL, "`service` = 'api'") {
|
||||||
|
t.Fatalf("unexpected SQL: %s", sqlRunner.gotSQL)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestExecuteTextSearchPrefiltersThenQueriesClickHouse(t *testing.T) {
|
||||||
|
sqlRunner := &fakeSQLRunner{}
|
||||||
|
search := &fakeSearchClient{ids: []string{"id-1", "id-2"}}
|
||||||
|
plan := &ir.Plan{TextSearch: []ir.TextPredicate{{Query: "connection refused"}}}
|
||||||
|
|
||||||
|
_, err := Execute(context.Background(), plan, sqlRunner, search)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("Execute() error = %v", err)
|
||||||
|
}
|
||||||
|
if search.gotQuery != "connection refused" {
|
||||||
|
t.Fatalf("search query = %q", search.gotQuery)
|
||||||
|
}
|
||||||
|
if search.gotLimit != textSearchLimit {
|
||||||
|
t.Fatalf("search limit = %d, want %d", search.gotLimit, textSearchLimit)
|
||||||
|
}
|
||||||
|
if !strings.Contains(sqlRunner.gotSQL, "record_id IN ('id-1','id-2')") {
|
||||||
|
t.Fatalf("unexpected SQL: %s", sqlRunner.gotSQL)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestExecuteTextSearchNoMatchesSkipsClickHouseEntirely(t *testing.T) {
|
||||||
|
sqlRunner := &fakeSQLRunner{}
|
||||||
|
search := &fakeSearchClient{ids: nil}
|
||||||
|
plan := &ir.Plan{TextSearch: []ir.TextPredicate{{Query: "nothing matches"}}}
|
||||||
|
|
||||||
|
result, err := Execute(context.Background(), plan, sqlRunner, search)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("Execute() error = %v", err)
|
||||||
|
}
|
||||||
|
if sqlRunner.calls != 0 {
|
||||||
|
t.Fatalf("expected ClickHouse not to be queried when search finds nothing, got %d calls", sqlRunner.calls)
|
||||||
|
}
|
||||||
|
if len(result.Columns) != 0 || len(result.Rows) != 0 {
|
||||||
|
t.Fatalf("expected empty result, got %+v", result)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestExecuteTextSearchWithAggregation(t *testing.T) {
|
||||||
|
sqlRunner := &fakeSQLRunner{}
|
||||||
|
search := &fakeSearchClient{ids: []string{"id-1"}}
|
||||||
|
plan := &ir.Plan{
|
||||||
|
TextSearch: []ir.TextPredicate{{Query: "connection refused"}},
|
||||||
|
Aggregation: &ir.Aggregation{
|
||||||
|
Funcs: []ir.AggFunc{{Func: "count", Alias: "count"}},
|
||||||
|
GroupBy: []string{"host"},
|
||||||
|
},
|
||||||
|
}
|
||||||
|
|
||||||
|
_, err := Execute(context.Background(), plan, sqlRunner, search)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("Execute() error = %v", err)
|
||||||
|
}
|
||||||
|
if !strings.Contains(sqlRunner.gotSQL, "record_id IN ('id-1')") {
|
||||||
|
t.Fatalf("expected the text-search prefilter in the WHERE clause: %s", sqlRunner.gotSQL)
|
||||||
|
}
|
||||||
|
if !strings.Contains(sqlRunner.gotSQL, "GROUP BY `host`") {
|
||||||
|
t.Fatalf("expected GROUP BY: %s", sqlRunner.gotSQL)
|
||||||
|
}
|
||||||
|
if !strings.Contains(sqlRunner.gotSQL, "count() AS `count`") {
|
||||||
|
t.Fatalf("expected count() AS `count`: %s", sqlRunner.gotSQL)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestExecuteSearchErrorPropagates(t *testing.T) {
|
||||||
|
sqlRunner := &fakeSQLRunner{}
|
||||||
|
search := &fakeSearchClient{err: errors.New("search unavailable")}
|
||||||
|
plan := &ir.Plan{TextSearch: []ir.TextPredicate{{Query: "x"}}}
|
||||||
|
|
||||||
|
_, err := Execute(context.Background(), plan, sqlRunner, search)
|
||||||
|
if err == nil {
|
||||||
|
t.Fatal("expected the search error to propagate")
|
||||||
|
}
|
||||||
|
if sqlRunner.calls != 0 {
|
||||||
|
t.Fatalf("expected ClickHouse not to be queried after a search error, got %d calls", sqlRunner.calls)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestBuildSQLNumericCastOnAttributesField(t *testing.T) {
|
||||||
|
plan := &ir.Plan{Filters: []ir.FilterPredicate{{Field: "status", Op: ">=", Value: "500"}}}
|
||||||
|
sql := buildSQL(plan, nil)
|
||||||
|
want := "toFloat64OrZero(attributes['status']) >= 500"
|
||||||
|
if !strings.Contains(sql, want) {
|
||||||
|
t.Fatalf("SQL = %q, want it to contain %q", sql, want)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestBuildSQLStringComparisonOnAttributesField(t *testing.T) {
|
||||||
|
plan := &ir.Plan{Filters: []ir.FilterPredicate{{Field: "status", Op: "=", Value: "unknown"}}}
|
||||||
|
sql := buildSQL(plan, nil)
|
||||||
|
want := "attributes['status'] = 'unknown'"
|
||||||
|
if !strings.Contains(sql, want) {
|
||||||
|
t.Fatalf("SQL = %q, want it to contain %q", sql, want)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestBuildSQLTopLevelFieldNeverCast(t *testing.T) {
|
||||||
|
plan := &ir.Plan{Filters: []ir.FilterPredicate{{Field: "service", Op: "=", Value: "123"}}}
|
||||||
|
sql := buildSQL(plan, nil)
|
||||||
|
if strings.Contains(sql, "toFloat64OrZero") {
|
||||||
|
t.Fatalf("top-level field should never be numeric-cast: %s", sql)
|
||||||
|
}
|
||||||
|
if !strings.Contains(sql, "`service` = '123'") {
|
||||||
|
t.Fatalf("unexpected SQL: %s", sql)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestBuildSQLEscapesInjectionAttemptInValue(t *testing.T) {
|
||||||
|
plan := &ir.Plan{Filters: []ir.FilterPredicate{{Field: "service", Op: "=", Value: "x'; DROP TABLE logs; --"}}}
|
||||||
|
sql := buildSQL(plan, nil)
|
||||||
|
// The whole attacker-controlled value must land inside exactly one
|
||||||
|
// quoted literal, with its embedded quote backslash-escaped so it
|
||||||
|
// can't terminate the literal early -- checking for the escaped
|
||||||
|
// form directly, not just the absence of the raw substring (which
|
||||||
|
// is a weaker check: "\\'; DROP TABLE" still *contains* "'; DROP
|
||||||
|
// TABLE" as a substring, so that alone doesn't prove escaping
|
||||||
|
// happened).
|
||||||
|
want := `'x\'; DROP TABLE logs; --'`
|
||||||
|
if !strings.Contains(sql, want) {
|
||||||
|
t.Fatalf("expected the literal %q in SQL, got: %s", want, sql)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestBuildSQLDefaultLimitAppliedWhenNoneGiven(t *testing.T) {
|
||||||
|
plan := &ir.Plan{Filters: []ir.FilterPredicate{{Field: "service", Op: "=", Value: "api"}}}
|
||||||
|
sql := buildSQL(plan, nil)
|
||||||
|
if !strings.Contains(sql, "LIMIT 100") {
|
||||||
|
t.Fatalf("expected the default row limit, got: %s", sql)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestBuildSQLExplicitLimitOverridesDefault(t *testing.T) {
|
||||||
|
plan := &ir.Plan{
|
||||||
|
Filters: []ir.FilterPredicate{{Field: "service", Op: "=", Value: "api"}},
|
||||||
|
Limit: &ir.Limit{N: 5},
|
||||||
|
}
|
||||||
|
sql := buildSQL(plan, nil)
|
||||||
|
if !strings.Contains(sql, "LIMIT 5") || strings.Contains(sql, "LIMIT 100") {
|
||||||
|
t.Fatalf("expected LIMIT 5, got: %s", sql)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestBuildSQLTailWithoutSortOrdersAscending(t *testing.T) {
|
||||||
|
plan := &ir.Plan{Limit: &ir.Limit{N: 10, Tail: true}}
|
||||||
|
sql := buildSQL(plan, nil)
|
||||||
|
if !strings.Contains(sql, "ORDER BY `timestamp` ASC") {
|
||||||
|
t.Fatalf("expected ascending order for tail, got: %s", sql)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestBuildSQLNoSortDefaultsNewestFirst(t *testing.T) {
|
||||||
|
plan := &ir.Plan{}
|
||||||
|
sql := buildSQL(plan, nil)
|
||||||
|
if !strings.Contains(sql, "ORDER BY `timestamp` DESC") {
|
||||||
|
t.Fatalf("expected newest-first default, got: %s", sql)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestBuildSQLSortByAggregateAlias(t *testing.T) {
|
||||||
|
plan := &ir.Plan{
|
||||||
|
Aggregation: &ir.Aggregation{
|
||||||
|
Funcs: []ir.AggFunc{{Func: "count", Alias: "count"}},
|
||||||
|
GroupBy: []string{"host"},
|
||||||
|
},
|
||||||
|
Sort: []ir.SortField{{Field: "count", Desc: true}},
|
||||||
|
}
|
||||||
|
sql := buildSQL(plan, nil)
|
||||||
|
if !strings.Contains(sql, "ORDER BY `count` DESC") {
|
||||||
|
t.Fatalf("expected ORDER BY on the aggregate alias, got: %s", sql)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestBuildSQLSortByGroupByField(t *testing.T) {
|
||||||
|
plan := &ir.Plan{
|
||||||
|
Aggregation: &ir.Aggregation{
|
||||||
|
Funcs: []ir.AggFunc{{Func: "count", Alias: "count"}},
|
||||||
|
GroupBy: []string{"host"},
|
||||||
|
},
|
||||||
|
Sort: []ir.SortField{{Field: "host", Desc: false}},
|
||||||
|
}
|
||||||
|
sql := buildSQL(plan, nil)
|
||||||
|
if !strings.Contains(sql, "ORDER BY `host` ASC") {
|
||||||
|
t.Fatalf("expected ORDER BY on the group-by column, got: %s", sql)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestBuildSQLAggregationOnAttributesFieldAlwaysCasts(t *testing.T) {
|
||||||
|
plan := &ir.Plan{
|
||||||
|
Aggregation: &ir.Aggregation{
|
||||||
|
Funcs: []ir.AggFunc{{Func: "avg", Field: "latency_ms", Alias: "avg_latency"}},
|
||||||
|
},
|
||||||
|
}
|
||||||
|
sql := buildSQL(plan, nil)
|
||||||
|
if !strings.Contains(sql, "AVG(toFloat64OrZero(attributes['latency_ms'])) AS `avg_latency`") {
|
||||||
|
t.Fatalf("unexpected SQL: %s", sql)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestBuildSQLProjectionFields(t *testing.T) {
|
||||||
|
plan := &ir.Plan{Fields: []string{"host", "message"}}
|
||||||
|
sql := buildSQL(plan, nil)
|
||||||
|
if !strings.Contains(sql, "SELECT `host` AS `host`, `message` AS `message` FROM logs") {
|
||||||
|
t.Fatalf("unexpected SQL: %s", sql)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestBuildSQLTimeRange(t *testing.T) {
|
||||||
|
from := mustParseTime(t, "2026-08-14T00:00:00Z")
|
||||||
|
to := mustParseTime(t, "2026-08-14T01:00:00Z")
|
||||||
|
plan := &ir.Plan{TimeRange: &ir.TimeRange{From: from, To: to}}
|
||||||
|
sql := buildSQL(plan, nil)
|
||||||
|
if !strings.Contains(sql, "`timestamp` >= '2026-08-14T00:00:00Z'") {
|
||||||
|
t.Fatalf("missing From bound: %s", sql)
|
||||||
|
}
|
||||||
|
if !strings.Contains(sql, "`timestamp` <= '2026-08-14T01:00:00Z'") {
|
||||||
|
t.Fatalf("missing To bound: %s", sql)
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,228 @@
|
|||||||
|
package executor
|
||||||
|
|
||||||
|
import (
|
||||||
|
"fmt"
|
||||||
|
"regexp"
|
||||||
|
"strings"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"github.com/sentry/sentry/api/internal/querylang/ir"
|
||||||
|
)
|
||||||
|
|
||||||
|
// defaultRowLimit is the safety net when a raw-row query has neither an
|
||||||
|
// explicit head/tail nor an aggregation -- without it, a bare `service=api`
|
||||||
|
// with no other pipe stages would return every matching row unbounded.
|
||||||
|
// Independent of planner's own defaultLimit (same value, different
|
||||||
|
// concern: that one fills in `head`/`tail` with no N given; this one
|
||||||
|
// guards queries that never mention head/tail at all).
|
||||||
|
const defaultRowLimit = 100
|
||||||
|
|
||||||
|
// logs' real columns, per /storage. Anything else maps to
|
||||||
|
// attributes['field'] -- see /docs/query-language-design.md's "Field
|
||||||
|
// mapping" section.
|
||||||
|
var topLevelFields = map[string]bool{
|
||||||
|
"timestamp": true,
|
||||||
|
"host": true,
|
||||||
|
"service": true,
|
||||||
|
"severity": true,
|
||||||
|
"message": true,
|
||||||
|
"record_id": true,
|
||||||
|
}
|
||||||
|
|
||||||
|
func buildSQL(plan *ir.Plan, recordIDFilter []string) string {
|
||||||
|
var sb strings.Builder
|
||||||
|
|
||||||
|
sb.WriteString("SELECT ")
|
||||||
|
sb.WriteString(selectClause(plan))
|
||||||
|
sb.WriteString(" FROM logs")
|
||||||
|
|
||||||
|
if where := buildWhereClause(plan, recordIDFilter); where != "" {
|
||||||
|
sb.WriteString(" WHERE ")
|
||||||
|
sb.WriteString(where)
|
||||||
|
}
|
||||||
|
|
||||||
|
if plan.Aggregation != nil && len(plan.Aggregation.GroupBy) > 0 {
|
||||||
|
sb.WriteString(" GROUP BY ")
|
||||||
|
cols := make([]string, len(plan.Aggregation.GroupBy))
|
||||||
|
for i, g := range plan.Aggregation.GroupBy {
|
||||||
|
cols[i] = columnExpr(g)
|
||||||
|
}
|
||||||
|
sb.WriteString(strings.Join(cols, ", "))
|
||||||
|
}
|
||||||
|
|
||||||
|
writeOrderBy(&sb, plan)
|
||||||
|
|
||||||
|
if plan.Limit != nil {
|
||||||
|
fmt.Fprintf(&sb, " LIMIT %d", plan.Limit.N)
|
||||||
|
} else if plan.Aggregation == nil {
|
||||||
|
fmt.Fprintf(&sb, " LIMIT %d", defaultRowLimit)
|
||||||
|
}
|
||||||
|
|
||||||
|
return sb.String()
|
||||||
|
}
|
||||||
|
|
||||||
|
func writeOrderBy(sb *strings.Builder, plan *ir.Plan) {
|
||||||
|
switch {
|
||||||
|
case len(plan.Sort) > 0:
|
||||||
|
sb.WriteString(" ORDER BY ")
|
||||||
|
parts := make([]string, len(plan.Sort))
|
||||||
|
for i, s := range plan.Sort {
|
||||||
|
dir := "ASC"
|
||||||
|
if s.Desc {
|
||||||
|
dir = "DESC"
|
||||||
|
}
|
||||||
|
parts[i] = sortColumnExpr(plan, s.Field) + " " + dir
|
||||||
|
}
|
||||||
|
sb.WriteString(strings.Join(parts, ", "))
|
||||||
|
case plan.Limit != nil && plan.Limit.Tail:
|
||||||
|
// `tail N` with no explicit sort: order ascending so LIMIT N
|
||||||
|
// takes the chronologically *last* N rows. Callers wanting
|
||||||
|
// strict newest-first display order re-sort client-side --
|
||||||
|
// documented in the query language reference.
|
||||||
|
sb.WriteString(" ORDER BY `timestamp` ASC")
|
||||||
|
case plan.Aggregation == nil:
|
||||||
|
// Raw-row queries with no explicit sort default to newest-first,
|
||||||
|
// matching the Phase 0/1 UI default.
|
||||||
|
sb.WriteString(" ORDER BY `timestamp` DESC")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// sortColumnExpr resolves a sort field against an aggregation's own
|
||||||
|
// output columns (alias or group-by field) before falling back to the
|
||||||
|
// normal top-level/attributes mapping -- `sort -count` after `stats
|
||||||
|
// count` refers to the aggregate's alias, not a raw column.
|
||||||
|
func sortColumnExpr(plan *ir.Plan, field string) string {
|
||||||
|
if plan.Aggregation != nil {
|
||||||
|
for _, f := range plan.Aggregation.Funcs {
|
||||||
|
if f.Alias == field {
|
||||||
|
return quoteIdent(field)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
for _, g := range plan.Aggregation.GroupBy {
|
||||||
|
if g == field {
|
||||||
|
return columnExpr(field)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return columnExpr(field)
|
||||||
|
}
|
||||||
|
|
||||||
|
func selectClause(plan *ir.Plan) string {
|
||||||
|
if plan.Aggregation != nil {
|
||||||
|
parts := make([]string, 0, len(plan.Aggregation.GroupBy)+len(plan.Aggregation.Funcs))
|
||||||
|
for _, g := range plan.Aggregation.GroupBy {
|
||||||
|
parts = append(parts, columnExpr(g)+" AS "+quoteIdent(g))
|
||||||
|
}
|
||||||
|
for _, f := range plan.Aggregation.Funcs {
|
||||||
|
parts = append(parts, aggExpr(f)+" AS "+quoteIdent(f.Alias))
|
||||||
|
}
|
||||||
|
return strings.Join(parts, ", ")
|
||||||
|
}
|
||||||
|
if len(plan.Fields) > 0 {
|
||||||
|
parts := make([]string, len(plan.Fields))
|
||||||
|
for i, f := range plan.Fields {
|
||||||
|
parts[i] = columnExpr(f) + " AS " + quoteIdent(f)
|
||||||
|
}
|
||||||
|
return strings.Join(parts, ", ")
|
||||||
|
}
|
||||||
|
return "*"
|
||||||
|
}
|
||||||
|
|
||||||
|
// aggExpr always numeric-casts non-top-level (attributes-map) fields for
|
||||||
|
// sum/avg/min/max, unlike comparison predicates where casting is
|
||||||
|
// conditional on whether the compared value looks numeric -- an
|
||||||
|
// aggregate function is inherently a numeric (or, for min/max,
|
||||||
|
// order-comparable) operation, so there's no "maybe string" case the way
|
||||||
|
// there is for `field=value`. Known Phase 2 limitation: min/max on a
|
||||||
|
// non-top-level field always compares numerically, not lexicographically
|
||||||
|
// -- string min/max on attributes isn't supported this phase.
|
||||||
|
func aggExpr(f ir.AggFunc) string {
|
||||||
|
if f.Func == "count" {
|
||||||
|
return "count()"
|
||||||
|
}
|
||||||
|
col := columnExpr(f.Field)
|
||||||
|
if !topLevelFields[f.Field] {
|
||||||
|
col = "toFloat64OrZero(" + col + ")"
|
||||||
|
}
|
||||||
|
return strings.ToUpper(f.Func) + "(" + col + ")"
|
||||||
|
}
|
||||||
|
|
||||||
|
func buildWhereClause(plan *ir.Plan, recordIDFilter []string) string {
|
||||||
|
var conds []string
|
||||||
|
|
||||||
|
if len(recordIDFilter) > 0 {
|
||||||
|
quoted := make([]string, len(recordIDFilter))
|
||||||
|
for i, id := range recordIDFilter {
|
||||||
|
quoted[i] = quoteLiteral(id)
|
||||||
|
}
|
||||||
|
conds = append(conds, "record_id IN ("+strings.Join(quoted, ",")+")")
|
||||||
|
}
|
||||||
|
|
||||||
|
for _, f := range plan.Filters {
|
||||||
|
conds = append(conds, buildComparisonSQL(f))
|
||||||
|
}
|
||||||
|
|
||||||
|
if plan.TimeRange != nil {
|
||||||
|
if !plan.TimeRange.From.IsZero() {
|
||||||
|
conds = append(conds, "`timestamp` >= "+quoteLiteral(plan.TimeRange.From.UTC().Format(time.RFC3339Nano)))
|
||||||
|
}
|
||||||
|
if !plan.TimeRange.To.IsZero() {
|
||||||
|
conds = append(conds, "`timestamp` <= "+quoteLiteral(plan.TimeRange.To.UTC().Format(time.RFC3339Nano)))
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
return strings.Join(conds, " AND ")
|
||||||
|
}
|
||||||
|
|
||||||
|
// buildComparisonSQL numeric-casts a non-top-level field only when the
|
||||||
|
// compared value itself looks numeric -- `status>=500` casts (numeric
|
||||||
|
// comparison intent), `status="unknown"` doesn't (string comparison
|
||||||
|
// intent). Top-level fields are never cast; ClickHouse compares them
|
||||||
|
// against a string literal natively (DateTime64 columns parse an
|
||||||
|
// RFC3339-shaped literal, LowCardinality(String)/String compare as-is).
|
||||||
|
func buildComparisonSQL(f ir.FilterPredicate) string {
|
||||||
|
if !topLevelFields[f.Field] && isNumericLiteral(f.Value) {
|
||||||
|
return "toFloat64OrZero(" + columnExpr(f.Field) + ") " + f.Op + " " + f.Value
|
||||||
|
}
|
||||||
|
return columnExpr(f.Field) + " " + f.Op + " " + quoteLiteral(f.Value)
|
||||||
|
}
|
||||||
|
|
||||||
|
func columnExpr(field string) string {
|
||||||
|
if topLevelFields[field] {
|
||||||
|
return quoteIdent(field)
|
||||||
|
}
|
||||||
|
return "attributes[" + quoteLiteral(field) + "]"
|
||||||
|
}
|
||||||
|
|
||||||
|
func quoteIdent(name string) string {
|
||||||
|
return "`" + strings.ReplaceAll(name, "`", "``") + "`"
|
||||||
|
}
|
||||||
|
|
||||||
|
// quoteLiteral is the actual injection defense for every user-controlled
|
||||||
|
// string embedded in generated SQL (filter values, attribute keys, time
|
||||||
|
// bounds, record_ids). Field/keyword tokens from the lexer are already
|
||||||
|
// constrained to [a-zA-Z0-9_.] by construction (see lexer.isIdentPart)
|
||||||
|
// and can't carry SQL metacharacters at all, but quoted-string *values*
|
||||||
|
// can contain anything, so this can't be skipped for them.
|
||||||
|
func quoteLiteral(s string) string {
|
||||||
|
var sb strings.Builder
|
||||||
|
sb.WriteByte('\'')
|
||||||
|
for _, r := range s {
|
||||||
|
switch r {
|
||||||
|
case '\\':
|
||||||
|
sb.WriteString(`\\`)
|
||||||
|
case '\'':
|
||||||
|
sb.WriteString(`\'`)
|
||||||
|
default:
|
||||||
|
sb.WriteRune(r)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
sb.WriteByte('\'')
|
||||||
|
return sb.String()
|
||||||
|
}
|
||||||
|
|
||||||
|
var numericLiteralRe = regexp.MustCompile(`^-?\d+(\.\d+)?$`)
|
||||||
|
|
||||||
|
func isNumericLiteral(s string) bool {
|
||||||
|
return numericLiteralRe.MatchString(s)
|
||||||
|
}
|
||||||
@@ -0,0 +1,80 @@
|
|||||||
|
// Package ir defines Plan, the intermediate representation both the
|
||||||
|
// pipe-syntax parser+planner and the raw-SQL passthrough compile down
|
||||||
|
// to. This is the boundary task 3 asked for: "pipe syntax X compiles to
|
||||||
|
// IR Y" is testable in planner without any backend; "IR Y executes
|
||||||
|
// correctly" is testable in executor against fakes, independent of the
|
||||||
|
// planner. See /docs/query-language-design.md.
|
||||||
|
package ir
|
||||||
|
|
||||||
|
import "time"
|
||||||
|
|
||||||
|
type Plan struct {
|
||||||
|
// RawSQL, when non-empty, means the entire plan is this opaque
|
||||||
|
// ClickHouse SQL string, executed as-is -- every other field below
|
||||||
|
// is unused. This is the SQL escape hatch's IR representation: a
|
||||||
|
// trivial identity compilation that still flows through the same
|
||||||
|
// Plan type and the same executor code path as a parsed pipe query.
|
||||||
|
RawSQL string
|
||||||
|
|
||||||
|
// TextSearch predicates route to Tantivy as a prefilter. Empty means
|
||||||
|
// no Tantivy involvement at all -- pure ClickHouse.
|
||||||
|
TextSearch []TextPredicate
|
||||||
|
|
||||||
|
// Filters are always evaluated in ClickHouse, either directly as
|
||||||
|
// WHERE clauses (no TextSearch present) or as an additional filter
|
||||||
|
// alongside a Tantivy-sourced record_id IN (...) clause.
|
||||||
|
Filters []FilterPredicate
|
||||||
|
|
||||||
|
TimeRange *TimeRange
|
||||||
|
|
||||||
|
// Aggregation is nil for a raw-rows query (no GROUP BY).
|
||||||
|
Aggregation *Aggregation
|
||||||
|
|
||||||
|
Sort []SortField
|
||||||
|
|
||||||
|
// Fields is the projection; empty means all columns.
|
||||||
|
Fields []string
|
||||||
|
|
||||||
|
Limit *Limit
|
||||||
|
}
|
||||||
|
|
||||||
|
type TextPredicate struct {
|
||||||
|
// Query is passed to Tantivy's query parser as-is -- phrase and
|
||||||
|
// wildcard syntax already supported there (see /search).
|
||||||
|
Query string
|
||||||
|
}
|
||||||
|
|
||||||
|
type FilterPredicate struct {
|
||||||
|
Field string
|
||||||
|
Op string // "=", "!=", ">", ">=", "<", "<="
|
||||||
|
Value string
|
||||||
|
}
|
||||||
|
|
||||||
|
type Aggregation struct {
|
||||||
|
Funcs []AggFunc
|
||||||
|
GroupBy []string
|
||||||
|
}
|
||||||
|
|
||||||
|
type AggFunc struct {
|
||||||
|
Func string // count, sum, avg, min, max
|
||||||
|
Field string // empty for count
|
||||||
|
Alias string // always set by the planner (defaulted if not given explicitly)
|
||||||
|
}
|
||||||
|
|
||||||
|
type SortField struct {
|
||||||
|
Field string
|
||||||
|
Desc bool
|
||||||
|
}
|
||||||
|
|
||||||
|
type Limit struct {
|
||||||
|
N int
|
||||||
|
Tail bool // true = last N (by time), false = first N
|
||||||
|
}
|
||||||
|
|
||||||
|
type TimeRange struct {
|
||||||
|
// Absolute bounds -- any relative expression (-1h etc.) is resolved
|
||||||
|
// by the planner at compile time, since only it knows "now". A zero
|
||||||
|
// time.Time means that bound is unset.
|
||||||
|
From time.Time
|
||||||
|
To time.Time
|
||||||
|
}
|
||||||
@@ -0,0 +1,238 @@
|
|||||||
|
// Package lexer tokenizes the pipe-syntax query language. Deliberately
|
||||||
|
// simple: keywords (where/stats/sort/and/etc.) aren't distinct token
|
||||||
|
// kinds -- they're just Ident tokens whose value the parser checks
|
||||||
|
// against a keyword set, so the lexer stays context-free and the parser
|
||||||
|
// owns all the grammar decisions. See /docs/query-language-design.md.
|
||||||
|
package lexer
|
||||||
|
|
||||||
|
import "fmt"
|
||||||
|
|
||||||
|
type Kind int
|
||||||
|
|
||||||
|
const (
|
||||||
|
EOF Kind = iota
|
||||||
|
Illegal
|
||||||
|
Ident // bare words: field names, keywords, unquoted values/free-text terms
|
||||||
|
String // quoted string: "..."
|
||||||
|
Number // 123, 1.5
|
||||||
|
Pipe // |
|
||||||
|
Eq // =
|
||||||
|
Neq // !=
|
||||||
|
Gt // >
|
||||||
|
Gte // >=
|
||||||
|
Lt // <
|
||||||
|
Lte // <=
|
||||||
|
Colon // :
|
||||||
|
Comma // ,
|
||||||
|
LParen // (
|
||||||
|
RParen // )
|
||||||
|
Minus // -
|
||||||
|
Plus // +
|
||||||
|
Star // * (only meaningful inside count(*), same as SQL)
|
||||||
|
)
|
||||||
|
|
||||||
|
type Token struct {
|
||||||
|
Kind Kind
|
||||||
|
Value string
|
||||||
|
Pos int // byte offset into the original input, for error messages
|
||||||
|
}
|
||||||
|
|
||||||
|
func (t Token) String() string {
|
||||||
|
return fmt.Sprintf("%s(%q)@%d", t.Kind, t.Value, t.Pos)
|
||||||
|
}
|
||||||
|
|
||||||
|
func (k Kind) String() string {
|
||||||
|
switch k {
|
||||||
|
case EOF:
|
||||||
|
return "EOF"
|
||||||
|
case Illegal:
|
||||||
|
return "ILLEGAL"
|
||||||
|
case Ident:
|
||||||
|
return "IDENT"
|
||||||
|
case String:
|
||||||
|
return "STRING"
|
||||||
|
case Number:
|
||||||
|
return "NUMBER"
|
||||||
|
case Pipe:
|
||||||
|
return "PIPE"
|
||||||
|
case Eq:
|
||||||
|
return "EQ"
|
||||||
|
case Neq:
|
||||||
|
return "NEQ"
|
||||||
|
case Gt:
|
||||||
|
return "GT"
|
||||||
|
case Gte:
|
||||||
|
return "GTE"
|
||||||
|
case Lt:
|
||||||
|
return "LT"
|
||||||
|
case Lte:
|
||||||
|
return "LTE"
|
||||||
|
case Colon:
|
||||||
|
return "COLON"
|
||||||
|
case Comma:
|
||||||
|
return "COMMA"
|
||||||
|
case LParen:
|
||||||
|
return "LPAREN"
|
||||||
|
case RParen:
|
||||||
|
return "RPAREN"
|
||||||
|
case Minus:
|
||||||
|
return "MINUS"
|
||||||
|
case Plus:
|
||||||
|
return "PLUS"
|
||||||
|
case Star:
|
||||||
|
return "STAR"
|
||||||
|
default:
|
||||||
|
return "UNKNOWN"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
type Lexer struct {
|
||||||
|
input []rune
|
||||||
|
pos int
|
||||||
|
}
|
||||||
|
|
||||||
|
func New(input string) *Lexer {
|
||||||
|
return &Lexer{input: []rune(input)}
|
||||||
|
}
|
||||||
|
|
||||||
|
func (l *Lexer) Next() Token {
|
||||||
|
l.skipWhitespace()
|
||||||
|
if l.pos >= len(l.input) {
|
||||||
|
return Token{Kind: EOF, Pos: l.pos}
|
||||||
|
}
|
||||||
|
|
||||||
|
start := l.pos
|
||||||
|
c := l.input[l.pos]
|
||||||
|
|
||||||
|
switch {
|
||||||
|
case c == '|':
|
||||||
|
l.pos++
|
||||||
|
return Token{Kind: Pipe, Value: "|", Pos: start}
|
||||||
|
case c == '=':
|
||||||
|
l.pos++
|
||||||
|
return Token{Kind: Eq, Value: "=", Pos: start}
|
||||||
|
case c == '!' && l.peek(1) == '=':
|
||||||
|
l.pos += 2
|
||||||
|
return Token{Kind: Neq, Value: "!=", Pos: start}
|
||||||
|
case c == '>' && l.peek(1) == '=':
|
||||||
|
l.pos += 2
|
||||||
|
return Token{Kind: Gte, Value: ">=", Pos: start}
|
||||||
|
case c == '>':
|
||||||
|
l.pos++
|
||||||
|
return Token{Kind: Gt, Value: ">", Pos: start}
|
||||||
|
case c == '<' && l.peek(1) == '=':
|
||||||
|
l.pos += 2
|
||||||
|
return Token{Kind: Lte, Value: "<=", Pos: start}
|
||||||
|
case c == '<':
|
||||||
|
l.pos++
|
||||||
|
return Token{Kind: Lt, Value: "<", Pos: start}
|
||||||
|
case c == ':':
|
||||||
|
l.pos++
|
||||||
|
return Token{Kind: Colon, Value: ":", Pos: start}
|
||||||
|
case c == ',':
|
||||||
|
l.pos++
|
||||||
|
return Token{Kind: Comma, Value: ",", Pos: start}
|
||||||
|
case c == '(':
|
||||||
|
l.pos++
|
||||||
|
return Token{Kind: LParen, Value: "(", Pos: start}
|
||||||
|
case c == ')':
|
||||||
|
l.pos++
|
||||||
|
return Token{Kind: RParen, Value: ")", Pos: start}
|
||||||
|
case c == '-':
|
||||||
|
l.pos++
|
||||||
|
return Token{Kind: Minus, Value: "-", Pos: start}
|
||||||
|
case c == '+':
|
||||||
|
l.pos++
|
||||||
|
return Token{Kind: Plus, Value: "+", Pos: start}
|
||||||
|
case c == '*':
|
||||||
|
l.pos++
|
||||||
|
return Token{Kind: Star, Value: "*", Pos: start}
|
||||||
|
case c == '"':
|
||||||
|
return l.lexString()
|
||||||
|
case isDigit(c):
|
||||||
|
return l.lexNumber()
|
||||||
|
case isIdentStart(c):
|
||||||
|
return l.lexIdent()
|
||||||
|
default:
|
||||||
|
l.pos++
|
||||||
|
return Token{Kind: Illegal, Value: string(c), Pos: start}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func (l *Lexer) peek(offset int) rune {
|
||||||
|
p := l.pos + offset
|
||||||
|
if p >= len(l.input) {
|
||||||
|
return 0
|
||||||
|
}
|
||||||
|
return l.input[p]
|
||||||
|
}
|
||||||
|
|
||||||
|
func (l *Lexer) skipWhitespace() {
|
||||||
|
for l.pos < len(l.input) {
|
||||||
|
switch l.input[l.pos] {
|
||||||
|
case ' ', '\t', '\n', '\r':
|
||||||
|
l.pos++
|
||||||
|
default:
|
||||||
|
return
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func (l *Lexer) lexString() Token {
|
||||||
|
start := l.pos
|
||||||
|
l.pos++ // consume opening quote
|
||||||
|
var sb []rune
|
||||||
|
for l.pos < len(l.input) {
|
||||||
|
c := l.input[l.pos]
|
||||||
|
if c == '"' {
|
||||||
|
l.pos++
|
||||||
|
return Token{Kind: String, Value: string(sb), Pos: start}
|
||||||
|
}
|
||||||
|
if c == '\\' && l.pos+1 < len(l.input) {
|
||||||
|
l.pos++
|
||||||
|
sb = append(sb, l.input[l.pos])
|
||||||
|
l.pos++
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
sb = append(sb, c)
|
||||||
|
l.pos++
|
||||||
|
}
|
||||||
|
// unterminated string -- return what we have as Illegal so the
|
||||||
|
// parser can produce a clear "unterminated string" error rather than
|
||||||
|
// the lexer silently accepting it.
|
||||||
|
return Token{Kind: Illegal, Value: string(sb), Pos: start}
|
||||||
|
}
|
||||||
|
|
||||||
|
func (l *Lexer) lexNumber() Token {
|
||||||
|
start := l.pos
|
||||||
|
for l.pos < len(l.input) && isDigit(l.input[l.pos]) {
|
||||||
|
l.pos++
|
||||||
|
}
|
||||||
|
if l.pos < len(l.input) && l.input[l.pos] == '.' && l.pos+1 < len(l.input) && isDigit(l.input[l.pos+1]) {
|
||||||
|
l.pos++
|
||||||
|
for l.pos < len(l.input) && isDigit(l.input[l.pos]) {
|
||||||
|
l.pos++
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return Token{Kind: Number, Value: string(l.input[start:l.pos]), Pos: start}
|
||||||
|
}
|
||||||
|
|
||||||
|
func (l *Lexer) lexIdent() Token {
|
||||||
|
start := l.pos
|
||||||
|
for l.pos < len(l.input) && isIdentPart(l.input[l.pos]) {
|
||||||
|
l.pos++
|
||||||
|
}
|
||||||
|
return Token{Kind: Ident, Value: string(l.input[start:l.pos]), Pos: start}
|
||||||
|
}
|
||||||
|
|
||||||
|
func isDigit(c rune) bool { return c >= '0' && c <= '9' }
|
||||||
|
|
||||||
|
func isIdentStart(c rune) bool {
|
||||||
|
return c == '_' || (c >= 'a' && c <= 'z') || (c >= 'A' && c <= 'Z')
|
||||||
|
}
|
||||||
|
|
||||||
|
// Identifiers allow dots (e.g. winevt.event_id, a real attribute key
|
||||||
|
// shape from Phase 1) and digits after the first character.
|
||||||
|
func isIdentPart(c rune) bool {
|
||||||
|
return isIdentStart(c) || isDigit(c) || c == '.' || c == '_'
|
||||||
|
}
|
||||||
@@ -0,0 +1,140 @@
|
|||||||
|
package lexer
|
||||||
|
|
||||||
|
import "testing"
|
||||||
|
|
||||||
|
func collectKinds(input string) []Kind {
|
||||||
|
l := New(input)
|
||||||
|
var kinds []Kind
|
||||||
|
for {
|
||||||
|
tok := l.Next()
|
||||||
|
kinds = append(kinds, tok.Kind)
|
||||||
|
if tok.Kind == EOF {
|
||||||
|
return kinds
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestLexSimpleFilter(t *testing.T) {
|
||||||
|
got := collectKinds(`service=api`)
|
||||||
|
want := []Kind{Ident, Eq, Ident, EOF}
|
||||||
|
assertKinds(t, got, want)
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestLexPipeline(t *testing.T) {
|
||||||
|
got := collectKinds(`service=api | where status>=500 | stats count(*) by host`)
|
||||||
|
want := []Kind{
|
||||||
|
Ident, Eq, Ident, Pipe,
|
||||||
|
Ident, Ident, Gte, Number, Pipe,
|
||||||
|
Ident, Ident, LParen, Star, RParen, Ident, Ident,
|
||||||
|
EOF,
|
||||||
|
}
|
||||||
|
assertKinds(t, got, want)
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestLexOperators(t *testing.T) {
|
||||||
|
got := collectKinds(`= != > >= < <=`)
|
||||||
|
want := []Kind{Eq, Neq, Gt, Gte, Lt, Lte, EOF}
|
||||||
|
assertKinds(t, got, want)
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestLexQuotedString(t *testing.T) {
|
||||||
|
l := New(`"connection refused"`)
|
||||||
|
tok := l.Next()
|
||||||
|
if tok.Kind != String {
|
||||||
|
t.Fatalf("Kind = %v, want String", tok.Kind)
|
||||||
|
}
|
||||||
|
if tok.Value != "connection refused" {
|
||||||
|
t.Fatalf("Value = %q, want %q", tok.Value, "connection refused")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestLexQuotedStringWithEscapes(t *testing.T) {
|
||||||
|
l := New(`"has \"quotes\" and \\backslash"`)
|
||||||
|
tok := l.Next()
|
||||||
|
if tok.Kind != String {
|
||||||
|
t.Fatalf("Kind = %v, want String", tok.Kind)
|
||||||
|
}
|
||||||
|
want := `has "quotes" and \backslash`
|
||||||
|
if tok.Value != want {
|
||||||
|
t.Fatalf("Value = %q, want %q", tok.Value, want)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestLexUnterminatedStringIsIllegal(t *testing.T) {
|
||||||
|
l := New(`"unterminated`)
|
||||||
|
tok := l.Next()
|
||||||
|
if tok.Kind != Illegal {
|
||||||
|
t.Fatalf("Kind = %v, want Illegal", tok.Kind)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestLexFieldWithDots(t *testing.T) {
|
||||||
|
l := New(`winevt.event_id=4625`)
|
||||||
|
tok := l.Next()
|
||||||
|
if tok.Kind != Ident || tok.Value != "winevt.event_id" {
|
||||||
|
t.Fatalf("got %v, want Ident(winevt.event_id)", tok)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestLexNumber(t *testing.T) {
|
||||||
|
cases := []string{"123", "1.5", "0"}
|
||||||
|
for _, c := range cases {
|
||||||
|
l := New(c)
|
||||||
|
tok := l.Next()
|
||||||
|
if tok.Kind != Number || tok.Value != c {
|
||||||
|
t.Errorf("lexing %q: got %v, want Number(%s)", c, tok, c)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestLexNegativeTimeExpr(t *testing.T) {
|
||||||
|
// "-1h" lexes as MINUS, NUMBER, IDENT -- the parser composes these,
|
||||||
|
// not the lexer (see package doc comment).
|
||||||
|
got := collectKinds(`-1h`)
|
||||||
|
want := []Kind{Minus, Number, Ident, EOF}
|
||||||
|
assertKinds(t, got, want)
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestLexWhitespaceIsSkipped(t *testing.T) {
|
||||||
|
got := collectKinds(" service = api ")
|
||||||
|
want := []Kind{Ident, Eq, Ident, EOF}
|
||||||
|
assertKinds(t, got, want)
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestLexEmptyInput(t *testing.T) {
|
||||||
|
got := collectKinds("")
|
||||||
|
want := []Kind{EOF}
|
||||||
|
assertKinds(t, got, want)
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestLexIllegalCharacter(t *testing.T) {
|
||||||
|
l := New(`$`)
|
||||||
|
tok := l.Next()
|
||||||
|
if tok.Kind != Illegal {
|
||||||
|
t.Fatalf("Kind = %v, want Illegal", tok.Kind)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestTokenPositionsAreByteOffsets(t *testing.T) {
|
||||||
|
l := New(`service=api`)
|
||||||
|
first := l.Next()
|
||||||
|
second := l.Next()
|
||||||
|
if first.Pos != 0 {
|
||||||
|
t.Errorf("first.Pos = %d, want 0", first.Pos)
|
||||||
|
}
|
||||||
|
if second.Pos != 7 {
|
||||||
|
t.Errorf("second.Pos = %d, want 7", second.Pos)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func assertKinds(t *testing.T, got, want []Kind) {
|
||||||
|
t.Helper()
|
||||||
|
if len(got) != len(want) {
|
||||||
|
t.Fatalf("got %d tokens %v, want %d tokens %v", len(got), got, len(want), want)
|
||||||
|
}
|
||||||
|
for i := range got {
|
||||||
|
if got[i] != want[i] {
|
||||||
|
t.Fatalf("token %d: got %v, want %v (full: got=%v want=%v)", i, got[i], want[i], got, want)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,452 @@
|
|||||||
|
// Package parser is a hand-written recursive-descent parser for the
|
||||||
|
// pipe-syntax query language, per /docs/query-language-design.md's
|
||||||
|
// choice of parser approach (no combinator/generator library -- this
|
||||||
|
// grammar is small and stable, and hand-written gives full control over
|
||||||
|
// error messages, which matter for a user-facing query language).
|
||||||
|
package parser
|
||||||
|
|
||||||
|
import (
|
||||||
|
"fmt"
|
||||||
|
"strconv"
|
||||||
|
|
||||||
|
"github.com/sentry/sentry/api/internal/querylang/ast"
|
||||||
|
"github.com/sentry/sentry/api/internal/querylang/lexer"
|
||||||
|
)
|
||||||
|
|
||||||
|
// Parse parses a pipe-syntax query. Callers are responsible for routing
|
||||||
|
// SQL (queries starting with "SELECT") elsewhere before calling this --
|
||||||
|
// see planner.Plan and /docs/query-language-design.md's "SQL escape
|
||||||
|
// hatch" section for why this parser never sees SQL at all.
|
||||||
|
func Parse(input string) (*ast.Query, error) {
|
||||||
|
p := newParser(input)
|
||||||
|
return p.parseQuery()
|
||||||
|
}
|
||||||
|
|
||||||
|
type parser struct {
|
||||||
|
lex *lexer.Lexer
|
||||||
|
cur lexer.Token
|
||||||
|
next lexer.Token
|
||||||
|
}
|
||||||
|
|
||||||
|
func newParser(input string) *parser {
|
||||||
|
p := &parser{lex: lexer.New(input)}
|
||||||
|
p.next = p.lex.Next()
|
||||||
|
p.advance()
|
||||||
|
return p
|
||||||
|
}
|
||||||
|
|
||||||
|
func (p *parser) advance() {
|
||||||
|
p.cur = p.next
|
||||||
|
p.next = p.lex.Next()
|
||||||
|
}
|
||||||
|
|
||||||
|
func (p *parser) parseQuery() (*ast.Query, error) {
|
||||||
|
base, err := p.parseBoolExpr()
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
q := &ast.Query{Base: base}
|
||||||
|
for p.cur.Kind == lexer.Pipe {
|
||||||
|
p.advance()
|
||||||
|
stage, err := p.parsePipeStage()
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
q.Pipes = append(q.Pipes, stage)
|
||||||
|
}
|
||||||
|
if p.cur.Kind != lexer.EOF {
|
||||||
|
return nil, p.errorf("unexpected %s after query", p.cur)
|
||||||
|
}
|
||||||
|
return q, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func (p *parser) parseBoolExpr() (ast.BoolExpr, error) {
|
||||||
|
var expr ast.BoolExpr
|
||||||
|
term, err := p.parseTerm()
|
||||||
|
if err != nil {
|
||||||
|
return expr, err
|
||||||
|
}
|
||||||
|
expr.Terms = append(expr.Terms, term)
|
||||||
|
|
||||||
|
for {
|
||||||
|
var conj string
|
||||||
|
switch {
|
||||||
|
case p.cur.Kind == lexer.Ident && (p.cur.Value == "and" || p.cur.Value == "or"):
|
||||||
|
conj = p.cur.Value
|
||||||
|
p.advance()
|
||||||
|
case p.canStartTerm():
|
||||||
|
// Adjacent bare terms with no explicit keyword between them
|
||||||
|
// implicitly AND, matching SPL's convention (e.g. `error
|
||||||
|
// timeout` means `error AND timeout`).
|
||||||
|
conj = "and"
|
||||||
|
default:
|
||||||
|
return expr, nil
|
||||||
|
}
|
||||||
|
term, err := p.parseTerm()
|
||||||
|
if err != nil {
|
||||||
|
return expr, err
|
||||||
|
}
|
||||||
|
expr.Terms = append(expr.Terms, term)
|
||||||
|
expr.Conjs = append(expr.Conjs, conj)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func (p *parser) canStartTerm() bool {
|
||||||
|
switch p.cur.Kind {
|
||||||
|
case lexer.Ident, lexer.String:
|
||||||
|
return true
|
||||||
|
default:
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func (p *parser) parseTerm() (ast.Term, error) {
|
||||||
|
switch p.cur.Kind {
|
||||||
|
case lexer.Ident:
|
||||||
|
if p.cur.Value == "earliest" || p.cur.Value == "latest" {
|
||||||
|
return p.parseTimeBound()
|
||||||
|
}
|
||||||
|
if p.cur.Value == "message" && p.next.Kind == lexer.Colon {
|
||||||
|
return p.parseExplicitFreeText()
|
||||||
|
}
|
||||||
|
if isComparatorStart(p.next.Kind) {
|
||||||
|
return p.parseComparison()
|
||||||
|
}
|
||||||
|
// A bare word with no comparator following it is a free-text
|
||||||
|
// search term, not a malformed comparison.
|
||||||
|
val := p.cur.Value
|
||||||
|
p.advance()
|
||||||
|
return ast.FreeText{Query: val}, nil
|
||||||
|
case lexer.String:
|
||||||
|
val := p.cur.Value
|
||||||
|
p.advance()
|
||||||
|
return ast.FreeText{Query: val}, nil
|
||||||
|
default:
|
||||||
|
return nil, p.errorf("expected a filter, comparison, or search term, got %s", p.cur)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func (p *parser) parseComparison() (ast.Term, error) {
|
||||||
|
field := p.cur.Value
|
||||||
|
p.advance()
|
||||||
|
op, err := p.parseComparator()
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
value, err := p.parseValue()
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
return ast.Comparison{Field: field, Op: op, Value: value}, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func (p *parser) parseComparator() (string, error) {
|
||||||
|
if !isComparatorStart(p.cur.Kind) {
|
||||||
|
return "", p.errorf("expected a comparator (=, !=, >, >=, <, <=), got %s", p.cur)
|
||||||
|
}
|
||||||
|
op := p.cur.Value
|
||||||
|
p.advance()
|
||||||
|
return op, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func isComparatorStart(k lexer.Kind) bool {
|
||||||
|
switch k {
|
||||||
|
case lexer.Eq, lexer.Neq, lexer.Gt, lexer.Gte, lexer.Lt, lexer.Lte:
|
||||||
|
return true
|
||||||
|
default:
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func (p *parser) parseValue() (string, error) {
|
||||||
|
switch p.cur.Kind {
|
||||||
|
case lexer.Ident, lexer.String, lexer.Number:
|
||||||
|
v := p.cur.Value
|
||||||
|
p.advance()
|
||||||
|
return v, nil
|
||||||
|
default:
|
||||||
|
return "", p.errorf("expected a value, got %s", p.cur)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func (p *parser) parseTimeBound() (ast.Term, error) {
|
||||||
|
kind := p.cur.Value
|
||||||
|
p.advance()
|
||||||
|
if err := p.expect(lexer.Eq); err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
expr, err := p.parseTimeExpr()
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
return ast.TimeBound{Kind: kind, Expr: expr}, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func (p *parser) parseTimeExpr() (ast.TimeExpr, error) {
|
||||||
|
if p.cur.Kind == lexer.String {
|
||||||
|
v := p.cur.Value
|
||||||
|
p.advance()
|
||||||
|
return ast.TimeExpr{Absolute: v}, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
sign := 1
|
||||||
|
switch p.cur.Kind {
|
||||||
|
case lexer.Minus:
|
||||||
|
sign = -1
|
||||||
|
p.advance()
|
||||||
|
case lexer.Plus:
|
||||||
|
p.advance()
|
||||||
|
}
|
||||||
|
|
||||||
|
if p.cur.Kind != lexer.Number {
|
||||||
|
return ast.TimeExpr{}, p.errorf("expected a quoted absolute timestamp or a relative offset like -1h, got %s", p.cur)
|
||||||
|
}
|
||||||
|
n, err := strconv.Atoi(p.cur.Value)
|
||||||
|
if err != nil {
|
||||||
|
return ast.TimeExpr{}, p.errorf("invalid number %q in time expression", p.cur.Value)
|
||||||
|
}
|
||||||
|
p.advance()
|
||||||
|
|
||||||
|
if p.cur.Kind != lexer.Ident || !isValidTimeUnit(p.cur.Value) {
|
||||||
|
return ast.TimeExpr{}, p.errorf("expected a time unit (s/m/h/d/w) after %d, got %s", n, p.cur)
|
||||||
|
}
|
||||||
|
unit := p.cur.Value
|
||||||
|
p.advance()
|
||||||
|
|
||||||
|
return ast.TimeExpr{IsRelative: true, RelativeSign: sign, RelativeN: n, RelativeUnit: unit}, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func isValidTimeUnit(u string) bool {
|
||||||
|
switch u {
|
||||||
|
case "s", "m", "h", "d", "w":
|
||||||
|
return true
|
||||||
|
default:
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func (p *parser) parseExplicitFreeText() (ast.Term, error) {
|
||||||
|
p.advance() // "message"
|
||||||
|
if err := p.expect(lexer.Colon); err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
if p.cur.Kind != lexer.String {
|
||||||
|
return nil, p.errorf("expected a quoted string after message:, got %s", p.cur)
|
||||||
|
}
|
||||||
|
v := p.cur.Value
|
||||||
|
p.advance()
|
||||||
|
return ast.FreeText{Query: v}, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func (p *parser) parsePipeStage() (ast.PipeStage, error) {
|
||||||
|
if p.cur.Kind != lexer.Ident {
|
||||||
|
return nil, p.errorf("expected a pipe stage (where/stats/sort/fields/head/tail), got %s", p.cur)
|
||||||
|
}
|
||||||
|
switch p.cur.Value {
|
||||||
|
case "where":
|
||||||
|
p.advance()
|
||||||
|
expr, err := p.parseBoolExpr()
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
return ast.WhereStage{Expr: expr}, nil
|
||||||
|
case "stats":
|
||||||
|
return p.parseStatsStage()
|
||||||
|
case "sort":
|
||||||
|
return p.parseSortStage()
|
||||||
|
case "fields":
|
||||||
|
return p.parseFieldsStage()
|
||||||
|
case "head":
|
||||||
|
return p.parseHeadTailStage(false)
|
||||||
|
case "tail":
|
||||||
|
return p.parseHeadTailStage(true)
|
||||||
|
default:
|
||||||
|
return nil, p.errorf("unknown pipe stage %q (expected where/stats/sort/fields/head/tail)", p.cur.Value)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func (p *parser) parseStatsStage() (ast.PipeStage, error) {
|
||||||
|
p.advance() // "stats"
|
||||||
|
var stage ast.StatsStage
|
||||||
|
|
||||||
|
agg, err := p.parseAggCall()
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
stage.Aggs = append(stage.Aggs, agg)
|
||||||
|
|
||||||
|
for p.cur.Kind == lexer.Comma {
|
||||||
|
p.advance()
|
||||||
|
agg, err := p.parseAggCall()
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
stage.Aggs = append(stage.Aggs, agg)
|
||||||
|
}
|
||||||
|
|
||||||
|
if p.cur.Kind == lexer.Ident && p.cur.Value == "by" {
|
||||||
|
p.advance()
|
||||||
|
field, err := p.parseFieldIdent()
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
stage.By = append(stage.By, field)
|
||||||
|
for p.cur.Kind == lexer.Comma {
|
||||||
|
p.advance()
|
||||||
|
field, err := p.parseFieldIdent()
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
stage.By = append(stage.By, field)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
return stage, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func (p *parser) parseAggCall() (ast.AggCall, error) {
|
||||||
|
if p.cur.Kind != lexer.Ident {
|
||||||
|
return ast.AggCall{}, p.errorf("expected an aggregation function (count/sum/avg/min/max), got %s", p.cur)
|
||||||
|
}
|
||||||
|
fn := p.cur.Value
|
||||||
|
if !isValidAggFunc(fn) {
|
||||||
|
return ast.AggCall{}, p.errorf("unknown aggregation function %q (want count/sum/avg/min/max)", fn)
|
||||||
|
}
|
||||||
|
p.advance()
|
||||||
|
|
||||||
|
// Parens are optional when there's no field: `count`, `count()`, and
|
||||||
|
// `count(*)` are all equivalent. `sum(field)` etc. still need them,
|
||||||
|
// since that's the only way to name the field.
|
||||||
|
var field string
|
||||||
|
if p.cur.Kind == lexer.LParen {
|
||||||
|
p.advance()
|
||||||
|
switch p.cur.Kind {
|
||||||
|
case lexer.Ident:
|
||||||
|
field = p.cur.Value
|
||||||
|
p.advance()
|
||||||
|
case lexer.Star:
|
||||||
|
p.advance() // count(*) is the same as count()
|
||||||
|
}
|
||||||
|
if err := p.expect(lexer.RParen); err != nil {
|
||||||
|
return ast.AggCall{}, err
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
var alias string
|
||||||
|
if p.cur.Kind == lexer.Ident && p.cur.Value == "as" {
|
||||||
|
p.advance()
|
||||||
|
if p.cur.Kind != lexer.Ident {
|
||||||
|
return ast.AggCall{}, p.errorf("expected an alias after 'as', got %s", p.cur)
|
||||||
|
}
|
||||||
|
alias = p.cur.Value
|
||||||
|
p.advance()
|
||||||
|
}
|
||||||
|
|
||||||
|
return ast.AggCall{Func: fn, Field: field, Alias: alias}, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func isValidAggFunc(f string) bool {
|
||||||
|
switch f {
|
||||||
|
case "count", "sum", "avg", "min", "max":
|
||||||
|
return true
|
||||||
|
default:
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func (p *parser) parseSortStage() (ast.PipeStage, error) {
|
||||||
|
p.advance() // "sort"
|
||||||
|
var stage ast.SortStage
|
||||||
|
|
||||||
|
field, err := p.parseSortField()
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
stage.Fields = append(stage.Fields, field)
|
||||||
|
|
||||||
|
for p.cur.Kind == lexer.Comma {
|
||||||
|
p.advance()
|
||||||
|
field, err := p.parseSortField()
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
stage.Fields = append(stage.Fields, field)
|
||||||
|
}
|
||||||
|
return stage, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func (p *parser) parseSortField() (ast.SortField, error) {
|
||||||
|
desc := true // no explicit sign defaults to descending, same as an explicit "-"
|
||||||
|
switch p.cur.Kind {
|
||||||
|
case lexer.Minus:
|
||||||
|
p.advance()
|
||||||
|
case lexer.Plus:
|
||||||
|
desc = false
|
||||||
|
p.advance()
|
||||||
|
}
|
||||||
|
if p.cur.Kind != lexer.Ident {
|
||||||
|
return ast.SortField{}, p.errorf("expected a field name in sort, got %s", p.cur)
|
||||||
|
}
|
||||||
|
field := p.cur.Value
|
||||||
|
p.advance()
|
||||||
|
return ast.SortField{Field: field, Desc: desc}, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func (p *parser) parseFieldsStage() (ast.PipeStage, error) {
|
||||||
|
p.advance() // "fields"
|
||||||
|
var stage ast.FieldsStage
|
||||||
|
field, err := p.parseFieldIdent()
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
stage.Fields = append(stage.Fields, field)
|
||||||
|
for p.cur.Kind == lexer.Comma {
|
||||||
|
p.advance()
|
||||||
|
field, err := p.parseFieldIdent()
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
stage.Fields = append(stage.Fields, field)
|
||||||
|
}
|
||||||
|
return stage, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func (p *parser) parseFieldIdent() (string, error) {
|
||||||
|
if p.cur.Kind != lexer.Ident {
|
||||||
|
return "", p.errorf("expected a field name, got %s", p.cur)
|
||||||
|
}
|
||||||
|
v := p.cur.Value
|
||||||
|
p.advance()
|
||||||
|
return v, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func (p *parser) parseHeadTailStage(tail bool) (ast.PipeStage, error) {
|
||||||
|
p.advance() // "head" / "tail"
|
||||||
|
if p.cur.Kind == lexer.Number {
|
||||||
|
n, err := strconv.Atoi(p.cur.Value)
|
||||||
|
if err != nil {
|
||||||
|
return nil, p.errorf("invalid number %q", p.cur.Value)
|
||||||
|
}
|
||||||
|
p.advance()
|
||||||
|
if tail {
|
||||||
|
return ast.TailStage{N: n, HasN: true}, nil
|
||||||
|
}
|
||||||
|
return ast.HeadStage{N: n, HasN: true}, nil
|
||||||
|
}
|
||||||
|
if tail {
|
||||||
|
return ast.TailStage{}, nil
|
||||||
|
}
|
||||||
|
return ast.HeadStage{}, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func (p *parser) expect(k lexer.Kind) error {
|
||||||
|
if p.cur.Kind != k {
|
||||||
|
return p.errorf("expected %s, got %s", k, p.cur)
|
||||||
|
}
|
||||||
|
p.advance()
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func (p *parser) errorf(format string, args ...any) error {
|
||||||
|
return fmt.Errorf("query syntax error at position %d: %s", p.cur.Pos, fmt.Sprintf(format, args...))
|
||||||
|
}
|
||||||
@@ -0,0 +1,278 @@
|
|||||||
|
package parser
|
||||||
|
|
||||||
|
import (
|
||||||
|
"testing"
|
||||||
|
|
||||||
|
"github.com/sentry/sentry/api/internal/querylang/ast"
|
||||||
|
)
|
||||||
|
|
||||||
|
func TestParseSimpleFilter(t *testing.T) {
|
||||||
|
q, err := Parse(`service=api`)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("Parse() error = %v", err)
|
||||||
|
}
|
||||||
|
if len(q.Base.Terms) != 1 {
|
||||||
|
t.Fatalf("expected 1 base term, got %d", len(q.Base.Terms))
|
||||||
|
}
|
||||||
|
cmp, ok := q.Base.Terms[0].(ast.Comparison)
|
||||||
|
if !ok {
|
||||||
|
t.Fatalf("expected Comparison, got %T", q.Base.Terms[0])
|
||||||
|
}
|
||||||
|
if cmp.Field != "service" || cmp.Op != "=" || cmp.Value != "api" {
|
||||||
|
t.Fatalf("unexpected comparison: %+v", cmp)
|
||||||
|
}
|
||||||
|
if len(q.Pipes) != 0 {
|
||||||
|
t.Fatalf("expected no pipes, got %d", len(q.Pipes))
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestParseFullPipeline(t *testing.T) {
|
||||||
|
q, err := Parse(`service=api | where status>=500 | stats count(*) as errors by host | sort -errors`)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("Parse() error = %v", err)
|
||||||
|
}
|
||||||
|
if len(q.Pipes) != 3 {
|
||||||
|
t.Fatalf("expected 3 pipe stages, got %d: %+v", len(q.Pipes), q.Pipes)
|
||||||
|
}
|
||||||
|
|
||||||
|
where, ok := q.Pipes[0].(ast.WhereStage)
|
||||||
|
if !ok {
|
||||||
|
t.Fatalf("stage 0: expected WhereStage, got %T", q.Pipes[0])
|
||||||
|
}
|
||||||
|
cmp := where.Expr.Terms[0].(ast.Comparison)
|
||||||
|
if cmp.Field != "status" || cmp.Op != ">=" || cmp.Value != "500" {
|
||||||
|
t.Fatalf("unexpected where comparison: %+v", cmp)
|
||||||
|
}
|
||||||
|
|
||||||
|
stats, ok := q.Pipes[1].(ast.StatsStage)
|
||||||
|
if !ok {
|
||||||
|
t.Fatalf("stage 1: expected StatsStage, got %T", q.Pipes[1])
|
||||||
|
}
|
||||||
|
if len(stats.Aggs) != 1 || stats.Aggs[0].Func != "count" || stats.Aggs[0].Alias != "errors" {
|
||||||
|
t.Fatalf("unexpected stats aggs: %+v", stats.Aggs)
|
||||||
|
}
|
||||||
|
if len(stats.By) != 1 || stats.By[0] != "host" {
|
||||||
|
t.Fatalf("unexpected stats by: %+v", stats.By)
|
||||||
|
}
|
||||||
|
|
||||||
|
sort, ok := q.Pipes[2].(ast.SortStage)
|
||||||
|
if !ok {
|
||||||
|
t.Fatalf("stage 2: expected SortStage, got %T", q.Pipes[2])
|
||||||
|
}
|
||||||
|
if len(sort.Fields) != 1 || sort.Fields[0].Field != "errors" || !sort.Fields[0].Desc {
|
||||||
|
t.Fatalf("unexpected sort fields: %+v", sort.Fields)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestParseExplicitFreeTextWithAggregation(t *testing.T) {
|
||||||
|
q, err := Parse(`message:"connection refused" | stats count by host`)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("Parse() error = %v", err)
|
||||||
|
}
|
||||||
|
ft, ok := q.Base.Terms[0].(ast.FreeText)
|
||||||
|
if !ok {
|
||||||
|
t.Fatalf("expected FreeText, got %T", q.Base.Terms[0])
|
||||||
|
}
|
||||||
|
if ft.Query != "connection refused" {
|
||||||
|
t.Fatalf("Query = %q, want %q", ft.Query, "connection refused")
|
||||||
|
}
|
||||||
|
stats := q.Pipes[0].(ast.StatsStage)
|
||||||
|
if stats.Aggs[0].Func != "count" || stats.Aggs[0].Field != "" {
|
||||||
|
t.Fatalf("unexpected agg: %+v", stats.Aggs[0])
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestParseBareWordIsFreeText(t *testing.T) {
|
||||||
|
q, err := Parse(`timeout`)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("Parse() error = %v", err)
|
||||||
|
}
|
||||||
|
ft, ok := q.Base.Terms[0].(ast.FreeText)
|
||||||
|
if !ok || ft.Query != "timeout" {
|
||||||
|
t.Fatalf("expected FreeText(timeout), got %+v", q.Base.Terms[0])
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestParseImplicitAndBetweenBareTerms(t *testing.T) {
|
||||||
|
q, err := Parse(`error timeout`)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("Parse() error = %v", err)
|
||||||
|
}
|
||||||
|
if len(q.Base.Terms) != 2 {
|
||||||
|
t.Fatalf("expected 2 terms, got %d", len(q.Base.Terms))
|
||||||
|
}
|
||||||
|
if len(q.Base.Conjs) != 1 || q.Base.Conjs[0] != "and" {
|
||||||
|
t.Fatalf("expected implicit 'and', got %+v", q.Base.Conjs)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestParseExplicitAndOr(t *testing.T) {
|
||||||
|
q, err := Parse(`service=api and status=500`)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("Parse() error = %v", err)
|
||||||
|
}
|
||||||
|
if len(q.Base.Conjs) != 1 || q.Base.Conjs[0] != "and" {
|
||||||
|
t.Fatalf("expected explicit 'and', got %+v", q.Base.Conjs)
|
||||||
|
}
|
||||||
|
|
||||||
|
q2, err := Parse(`service=api or service=web`)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("Parse() error = %v", err)
|
||||||
|
}
|
||||||
|
if len(q2.Base.Conjs) != 1 || q2.Base.Conjs[0] != "or" {
|
||||||
|
t.Fatalf("expected 'or', got %+v", q2.Base.Conjs)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestParseTimeBoundsRelativeAndAbsolute(t *testing.T) {
|
||||||
|
q, err := Parse(`earliest=-1h latest="2026-08-14T00:00:00Z"`)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("Parse() error = %v", err)
|
||||||
|
}
|
||||||
|
if len(q.Base.Terms) != 2 {
|
||||||
|
t.Fatalf("expected 2 terms, got %d", len(q.Base.Terms))
|
||||||
|
}
|
||||||
|
earliest := q.Base.Terms[0].(ast.TimeBound)
|
||||||
|
if earliest.Kind != "earliest" || !earliest.Expr.IsRelative || earliest.Expr.RelativeSign != -1 ||
|
||||||
|
earliest.Expr.RelativeN != 1 || earliest.Expr.RelativeUnit != "h" {
|
||||||
|
t.Fatalf("unexpected earliest: %+v", earliest)
|
||||||
|
}
|
||||||
|
latest := q.Base.Terms[1].(ast.TimeBound)
|
||||||
|
if latest.Kind != "latest" || latest.Expr.Absolute != "2026-08-14T00:00:00Z" {
|
||||||
|
t.Fatalf("unexpected latest: %+v", latest)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestParseFieldsStage(t *testing.T) {
|
||||||
|
q, err := Parse(`service=api | fields host, message, severity`)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("Parse() error = %v", err)
|
||||||
|
}
|
||||||
|
fields := q.Pipes[0].(ast.FieldsStage)
|
||||||
|
want := []string{"host", "message", "severity"}
|
||||||
|
if len(fields.Fields) != len(want) {
|
||||||
|
t.Fatalf("Fields = %v, want %v", fields.Fields, want)
|
||||||
|
}
|
||||||
|
for i := range want {
|
||||||
|
if fields.Fields[i] != want[i] {
|
||||||
|
t.Fatalf("Fields = %v, want %v", fields.Fields, want)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestParseHeadTailWithAndWithoutN(t *testing.T) {
|
||||||
|
q, err := Parse(`service=api | head 10`)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("Parse() error = %v", err)
|
||||||
|
}
|
||||||
|
head := q.Pipes[0].(ast.HeadStage)
|
||||||
|
if !head.HasN || head.N != 10 {
|
||||||
|
t.Fatalf("unexpected head: %+v", head)
|
||||||
|
}
|
||||||
|
|
||||||
|
q2, err := Parse(`service=api | tail`)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("Parse() error = %v", err)
|
||||||
|
}
|
||||||
|
tail := q2.Pipes[0].(ast.TailStage)
|
||||||
|
if tail.HasN {
|
||||||
|
t.Fatalf("expected no N, got %+v", tail)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestParseDottedFieldName(t *testing.T) {
|
||||||
|
q, err := Parse(`winevt.event_id=4625`)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("Parse() error = %v", err)
|
||||||
|
}
|
||||||
|
cmp := q.Base.Terms[0].(ast.Comparison)
|
||||||
|
if cmp.Field != "winevt.event_id" || cmp.Value != "4625" {
|
||||||
|
t.Fatalf("unexpected comparison: %+v", cmp)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestParseSortAscendingWithPlus(t *testing.T) {
|
||||||
|
q, err := Parse(`service=api | sort +host`)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("Parse() error = %v", err)
|
||||||
|
}
|
||||||
|
sort := q.Pipes[0].(ast.SortStage)
|
||||||
|
if sort.Fields[0].Desc {
|
||||||
|
t.Fatalf("expected ascending sort, got %+v", sort.Fields[0])
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestParseMultipleSortFields(t *testing.T) {
|
||||||
|
q, err := Parse(`service=api | sort -severity, +host`)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("Parse() error = %v", err)
|
||||||
|
}
|
||||||
|
sort := q.Pipes[0].(ast.SortStage)
|
||||||
|
if len(sort.Fields) != 2 {
|
||||||
|
t.Fatalf("expected 2 sort fields, got %d", len(sort.Fields))
|
||||||
|
}
|
||||||
|
if sort.Fields[0].Field != "severity" || !sort.Fields[0].Desc {
|
||||||
|
t.Fatalf("unexpected first sort field: %+v", sort.Fields[0])
|
||||||
|
}
|
||||||
|
if sort.Fields[1].Field != "host" || sort.Fields[1].Desc {
|
||||||
|
t.Fatalf("unexpected second sort field: %+v", sort.Fields[1])
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestParseMultipleAggregations(t *testing.T) {
|
||||||
|
q, err := Parse(`service=api | stats count() as n, avg(latency_ms) as avg_latency by host`)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("Parse() error = %v", err)
|
||||||
|
}
|
||||||
|
stats := q.Pipes[0].(ast.StatsStage)
|
||||||
|
if len(stats.Aggs) != 2 {
|
||||||
|
t.Fatalf("expected 2 aggs, got %d: %+v", len(stats.Aggs), stats.Aggs)
|
||||||
|
}
|
||||||
|
if stats.Aggs[1].Func != "avg" || stats.Aggs[1].Field != "latency_ms" || stats.Aggs[1].Alias != "avg_latency" {
|
||||||
|
t.Fatalf("unexpected second agg: %+v", stats.Aggs[1])
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// --- error cases ---
|
||||||
|
|
||||||
|
func TestParseErrorEmptyQuery(t *testing.T) {
|
||||||
|
if _, err := Parse(``); err == nil {
|
||||||
|
t.Fatal("expected an error for an empty query")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestParseErrorUnknownPipeStage(t *testing.T) {
|
||||||
|
if _, err := Parse(`service=api | bogus`); err == nil {
|
||||||
|
t.Fatal("expected an error for an unknown pipe stage")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestParseErrorMissingComparatorValue(t *testing.T) {
|
||||||
|
if _, err := Parse(`service=`); err == nil {
|
||||||
|
t.Fatal("expected an error for a missing comparison value")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestParseErrorUnknownAggFunc(t *testing.T) {
|
||||||
|
if _, err := Parse(`service=api | stats median(latency)`); err == nil {
|
||||||
|
t.Fatal("expected an error for an unknown aggregation function")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestParseErrorInvalidTimeUnit(t *testing.T) {
|
||||||
|
if _, err := Parse(`earliest=-1x`); err == nil {
|
||||||
|
t.Fatal("expected an error for an invalid time unit")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestParseErrorTrailingGarbage(t *testing.T) {
|
||||||
|
if _, err := Parse(`service=api extra ) tokens`); err == nil {
|
||||||
|
t.Fatal("expected an error for trailing unparseable tokens")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestParseErrorUnterminatedStringInFreeText(t *testing.T) {
|
||||||
|
if _, err := Parse(`message:"unterminated`); err == nil {
|
||||||
|
t.Fatal("expected an error for an unterminated quoted string")
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,286 @@
|
|||||||
|
// Package planner compiles a query string (either syntax) into ir.Plan.
|
||||||
|
// This is the single entry point querylang exposes to callers (the /query
|
||||||
|
// HTTP handler) -- see Compile.
|
||||||
|
package planner
|
||||||
|
|
||||||
|
import (
|
||||||
|
"fmt"
|
||||||
|
"regexp"
|
||||||
|
"strings"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"github.com/sentry/sentry/api/internal/querylang/ast"
|
||||||
|
"github.com/sentry/sentry/api/internal/querylang/ir"
|
||||||
|
"github.com/sentry/sentry/api/internal/querylang/parser"
|
||||||
|
)
|
||||||
|
|
||||||
|
// Language selects which syntax a query is written in.
|
||||||
|
type Language string
|
||||||
|
|
||||||
|
const (
|
||||||
|
Auto Language = "" // detect from the query text (default)
|
||||||
|
SQL Language = "sql"
|
||||||
|
SPL Language = "spl" // the pipe syntax; named to match the query-language-reference doc
|
||||||
|
)
|
||||||
|
|
||||||
|
const defaultLimit = 100
|
||||||
|
|
||||||
|
// Compile turns a query string into a Plan. language overrides
|
||||||
|
// auto-detection; pass Auto to use the SELECT-prefix heuristic (see
|
||||||
|
// /docs/query-language-design.md's "Detection" section) -- this exists
|
||||||
|
// for the rare case a pipe query legitimately starts with the literal
|
||||||
|
// word "select" as a bare search term.
|
||||||
|
func Compile(query string, language Language, now time.Time) (*ir.Plan, error) {
|
||||||
|
isSQL := language == SQL
|
||||||
|
if language == Auto {
|
||||||
|
isSQL = looksLikeSQL(query)
|
||||||
|
}
|
||||||
|
|
||||||
|
if isSQL {
|
||||||
|
if err := validateSelectOnly(query); err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
return &ir.Plan{RawSQL: strings.TrimSpace(strings.TrimSuffix(strings.TrimSpace(query), ";"))}, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
q, err := parser.Parse(query)
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
return compileQuery(q, now)
|
||||||
|
}
|
||||||
|
|
||||||
|
func looksLikeSQL(query string) bool {
|
||||||
|
trimmed := strings.TrimSpace(query)
|
||||||
|
if trimmed == "" {
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
fields := strings.Fields(trimmed)
|
||||||
|
return len(fields) > 0 && strings.EqualFold(fields[0], "SELECT")
|
||||||
|
}
|
||||||
|
|
||||||
|
func compileQuery(q *ast.Query, now time.Time) (*ir.Plan, error) {
|
||||||
|
plan := &ir.Plan{}
|
||||||
|
|
||||||
|
textParts, err := compileBoolExpr(q.Base, plan, now)
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
|
||||||
|
for _, stage := range q.Pipes {
|
||||||
|
switch s := stage.(type) {
|
||||||
|
case ast.WhereStage:
|
||||||
|
parts, err := compileBoolExpr(s.Expr, plan, now)
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
textParts = append(textParts, parts...)
|
||||||
|
case ast.StatsStage:
|
||||||
|
agg, err := compileStats(s)
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
plan.Aggregation = agg
|
||||||
|
case ast.SortStage:
|
||||||
|
for _, f := range s.Fields {
|
||||||
|
plan.Sort = append(plan.Sort, ir.SortField{Field: f.Field, Desc: f.Desc})
|
||||||
|
}
|
||||||
|
case ast.FieldsStage:
|
||||||
|
plan.Fields = append(plan.Fields, s.Fields...)
|
||||||
|
case ast.HeadStage:
|
||||||
|
n := defaultLimit
|
||||||
|
if s.HasN {
|
||||||
|
n = s.N
|
||||||
|
}
|
||||||
|
plan.Limit = &ir.Limit{N: n, Tail: false}
|
||||||
|
case ast.TailStage:
|
||||||
|
n := defaultLimit
|
||||||
|
if s.HasN {
|
||||||
|
n = s.N
|
||||||
|
}
|
||||||
|
plan.Limit = &ir.Limit{N: n, Tail: true}
|
||||||
|
default:
|
||||||
|
return nil, fmt.Errorf("internal error: unhandled pipe stage %T", stage)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
if len(textParts) > 0 {
|
||||||
|
plan.TextSearch = []ir.TextPredicate{{Query: strings.Join(textParts, " ")}}
|
||||||
|
}
|
||||||
|
|
||||||
|
return plan, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// textPart is one piece of a combined Tantivy query string, tagged with
|
||||||
|
// the conjunction that precedes it (empty for the first piece).
|
||||||
|
type textPart struct {
|
||||||
|
conj string // "", "and", "or"
|
||||||
|
query string
|
||||||
|
}
|
||||||
|
|
||||||
|
// compileBoolExpr walks one bool_expr (the base search, or a `where`
|
||||||
|
// stage's expression), populating plan.Filters and plan.TimeRange
|
||||||
|
// directly, and returning free-text pieces for the caller to fold into
|
||||||
|
// the combined Tantivy query string.
|
||||||
|
//
|
||||||
|
// Scope decision: "or" is only supported between free-text terms, which
|
||||||
|
// Tantivy's own query parser handles natively once composed into one
|
||||||
|
// string. "or" between structured comparisons/time-bounds is rejected
|
||||||
|
// with a clear error rather than silently compiled as "and" -- see
|
||||||
|
// /docs/query-language-reference.md's limitations section. This keeps
|
||||||
|
// the executor's generated SQL a flat AND-only WHERE clause, which is
|
||||||
|
// most of what real queries need; full boolean-tree support for
|
||||||
|
// structured filters is future work if usage shows it's needed.
|
||||||
|
func compileBoolExpr(expr ast.BoolExpr, plan *ir.Plan, now time.Time) ([]string, error) {
|
||||||
|
var textParts []string
|
||||||
|
|
||||||
|
for i, term := range expr.Terms {
|
||||||
|
conj := ""
|
||||||
|
if i > 0 {
|
||||||
|
conj = expr.Conjs[i-1]
|
||||||
|
}
|
||||||
|
|
||||||
|
switch t := term.(type) {
|
||||||
|
case ast.Comparison:
|
||||||
|
if conj == "or" {
|
||||||
|
return nil, fmt.Errorf("query error: \"or\" is not supported between structured filters (%q) in Phase 2 -- only between free-text search terms", t.Field)
|
||||||
|
}
|
||||||
|
plan.Filters = append(plan.Filters, ir.FilterPredicate{Field: t.Field, Op: t.Op, Value: t.Value})
|
||||||
|
case ast.TimeBound:
|
||||||
|
if conj == "or" {
|
||||||
|
return nil, fmt.Errorf("query error: \"or\" is not supported on time bounds (%s) in Phase 2", t.Kind)
|
||||||
|
}
|
||||||
|
if err := applyTimeBound(plan, t, now); err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
case ast.FreeText:
|
||||||
|
q := t.Query
|
||||||
|
if strings.ContainsAny(q, " \t") {
|
||||||
|
q = `"` + strings.ReplaceAll(q, `"`, `\"`) + `"`
|
||||||
|
}
|
||||||
|
if conj == "or" {
|
||||||
|
textParts = append(textParts, "OR", q)
|
||||||
|
} else if len(textParts) > 0 {
|
||||||
|
textParts = append(textParts, "AND", q)
|
||||||
|
} else {
|
||||||
|
textParts = append(textParts, q)
|
||||||
|
}
|
||||||
|
default:
|
||||||
|
return nil, fmt.Errorf("internal error: unhandled term %T", term)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
return textParts, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func applyTimeBound(plan *ir.Plan, t ast.TimeBound, now time.Time) error {
|
||||||
|
when, err := resolveTimeExpr(t.Expr, now)
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
if plan.TimeRange == nil {
|
||||||
|
plan.TimeRange = &ir.TimeRange{}
|
||||||
|
}
|
||||||
|
switch t.Kind {
|
||||||
|
case "earliest":
|
||||||
|
plan.TimeRange.From = when
|
||||||
|
case "latest":
|
||||||
|
plan.TimeRange.To = when
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func resolveTimeExpr(e ast.TimeExpr, now time.Time) (time.Time, error) {
|
||||||
|
if !e.IsRelative {
|
||||||
|
t, err := time.Parse(time.RFC3339, e.Absolute)
|
||||||
|
if err != nil {
|
||||||
|
return time.Time{}, fmt.Errorf("query error: invalid absolute timestamp %q, want RFC3339 (e.g. 2026-08-14T00:00:00Z): %w", e.Absolute, err)
|
||||||
|
}
|
||||||
|
return t, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
var d time.Duration
|
||||||
|
switch e.RelativeUnit {
|
||||||
|
case "s":
|
||||||
|
d = time.Duration(e.RelativeN) * time.Second
|
||||||
|
case "m":
|
||||||
|
d = time.Duration(e.RelativeN) * time.Minute
|
||||||
|
case "h":
|
||||||
|
d = time.Duration(e.RelativeN) * time.Hour
|
||||||
|
case "d":
|
||||||
|
d = time.Duration(e.RelativeN) * 24 * time.Hour
|
||||||
|
case "w":
|
||||||
|
d = time.Duration(e.RelativeN) * 7 * 24 * time.Hour
|
||||||
|
default:
|
||||||
|
return time.Time{}, fmt.Errorf("internal error: unknown time unit %q", e.RelativeUnit)
|
||||||
|
}
|
||||||
|
if e.RelativeSign < 0 {
|
||||||
|
d = -d
|
||||||
|
}
|
||||||
|
return now.Add(d), nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func compileStats(s ast.StatsStage) (*ir.Aggregation, error) {
|
||||||
|
agg := &ir.Aggregation{GroupBy: s.By}
|
||||||
|
seen := map[string]bool{}
|
||||||
|
for _, a := range s.Aggs {
|
||||||
|
if a.Func != "count" && a.Field == "" {
|
||||||
|
return nil, fmt.Errorf("query error: %s() requires a field, e.g. %s(latency_ms)", a.Func, a.Func)
|
||||||
|
}
|
||||||
|
alias := a.Alias
|
||||||
|
if alias == "" {
|
||||||
|
alias = defaultAggAlias(a)
|
||||||
|
}
|
||||||
|
if seen[alias] && a.Alias == "" {
|
||||||
|
// Two unnamed aggs of the same shape would otherwise collide
|
||||||
|
// (e.g. `stats sum(a), sum(b)` both defaulting to "sum") --
|
||||||
|
// disambiguate by field name.
|
||||||
|
alias = alias + "_" + a.Field
|
||||||
|
}
|
||||||
|
seen[alias] = true
|
||||||
|
agg.Funcs = append(agg.Funcs, ir.AggFunc{Func: a.Func, Field: a.Field, Alias: alias})
|
||||||
|
}
|
||||||
|
return agg, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func defaultAggAlias(a ast.AggCall) string {
|
||||||
|
if a.Func == "count" {
|
||||||
|
return "count"
|
||||||
|
}
|
||||||
|
return a.Func
|
||||||
|
}
|
||||||
|
|
||||||
|
// --- SQL escape hatch validation, ported from the Phase 0/1
|
||||||
|
// api/internal/queryapi/validate.go guard it replaces (see task 4) ---
|
||||||
|
|
||||||
|
// disallowedKeyword is defense-in-depth on top of the SELECT-only gate:
|
||||||
|
// it catches mutating/administrative statements appearing anywhere in
|
||||||
|
// the query, not just at the start. Word-boundary matching, not a real
|
||||||
|
// SQL parser -- same tradeoffs as the Phase 0/1 version this replaces.
|
||||||
|
var disallowedKeyword = regexp.MustCompile(`(?i)\b(insert|update|delete|alter|drop|truncate|create|grant|revoke|attach|detach|rename|kill|optimize|system|set|exchange|watch)\b`)
|
||||||
|
|
||||||
|
func validateSelectOnly(sql string) error {
|
||||||
|
trimmed := strings.TrimSpace(sql)
|
||||||
|
if trimmed == "" {
|
||||||
|
return fmt.Errorf("query must not be empty")
|
||||||
|
}
|
||||||
|
|
||||||
|
trimmed = strings.TrimSpace(strings.TrimSuffix(trimmed, ";"))
|
||||||
|
if trimmed == "" {
|
||||||
|
return fmt.Errorf("query must not be empty")
|
||||||
|
}
|
||||||
|
if strings.Contains(trimmed, ";") {
|
||||||
|
return fmt.Errorf("only a single statement is allowed")
|
||||||
|
}
|
||||||
|
|
||||||
|
firstWord := strings.ToUpper(strings.Fields(trimmed)[0])
|
||||||
|
if firstWord != "SELECT" {
|
||||||
|
return fmt.Errorf("only SELECT queries are allowed")
|
||||||
|
}
|
||||||
|
|
||||||
|
if disallowedKeyword.MatchString(trimmed) {
|
||||||
|
return fmt.Errorf("query contains a disallowed keyword")
|
||||||
|
}
|
||||||
|
|
||||||
|
return nil
|
||||||
|
}
|
||||||
@@ -0,0 +1,221 @@
|
|||||||
|
package planner
|
||||||
|
|
||||||
|
import (
|
||||||
|
"strings"
|
||||||
|
"testing"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"github.com/sentry/sentry/api/internal/querylang/ir"
|
||||||
|
)
|
||||||
|
|
||||||
|
var fixedNow = time.Date(2026, 8, 14, 12, 0, 0, 0, time.UTC)
|
||||||
|
|
||||||
|
func TestCompileDetectsSQL(t *testing.T) {
|
||||||
|
plan, err := Compile(`SELECT * FROM logs LIMIT 10`, Auto, fixedNow)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("Compile() error = %v", err)
|
||||||
|
}
|
||||||
|
if plan.RawSQL == "" {
|
||||||
|
t.Fatalf("expected RawSQL to be set, got plan: %+v", plan)
|
||||||
|
}
|
||||||
|
if plan.RawSQL != "SELECT * FROM logs LIMIT 10" {
|
||||||
|
t.Fatalf("RawSQL = %q", plan.RawSQL)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestCompileDetectsSQLCaseInsensitive(t *testing.T) {
|
||||||
|
plan, err := Compile(`select 1`, Auto, fixedNow)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("Compile() error = %v", err)
|
||||||
|
}
|
||||||
|
if plan.RawSQL != "select 1" {
|
||||||
|
t.Fatalf("RawSQL = %q", plan.RawSQL)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestCompileRejectsNonSelectSQLKeyword(t *testing.T) {
|
||||||
|
_, err := Compile(`DELETE FROM logs`, SQL, fixedNow)
|
||||||
|
if err == nil {
|
||||||
|
t.Fatal("expected an error for a non-SELECT statement forced to SQL language")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestCompileExplicitLanguageOverridesAutoDetect(t *testing.T) {
|
||||||
|
// "select" as a bare free-text search term -- would be misdetected
|
||||||
|
// as SQL by the heuristic alone, hence the override.
|
||||||
|
plan, err := Compile(`select`, SPL, fixedNow)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("Compile() error = %v", err)
|
||||||
|
}
|
||||||
|
if plan.RawSQL != "" {
|
||||||
|
t.Fatalf("expected pipe-syntax compilation, got RawSQL = %q", plan.RawSQL)
|
||||||
|
}
|
||||||
|
if len(plan.TextSearch) != 1 || plan.TextSearch[0].Query != "select" {
|
||||||
|
t.Fatalf("expected a free-text search for 'select', got %+v", plan.TextSearch)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestCompileSimpleFilter(t *testing.T) {
|
||||||
|
plan, err := Compile(`service=api`, Auto, fixedNow)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("Compile() error = %v", err)
|
||||||
|
}
|
||||||
|
want := ir.FilterPredicate{Field: "service", Op: "=", Value: "api"}
|
||||||
|
if len(plan.Filters) != 1 || plan.Filters[0] != want {
|
||||||
|
t.Fatalf("unexpected filters: %+v, want [%+v]", plan.Filters, want)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestCompileFullPipeline(t *testing.T) {
|
||||||
|
plan, err := Compile(`service=api | where status>=500 | stats count by host | sort -count`, Auto, fixedNow)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("Compile() error = %v", err)
|
||||||
|
}
|
||||||
|
if len(plan.Filters) != 2 {
|
||||||
|
t.Fatalf("expected 2 filters (service=api, status>=500), got %+v", plan.Filters)
|
||||||
|
}
|
||||||
|
if plan.Aggregation == nil || len(plan.Aggregation.Funcs) != 1 || plan.Aggregation.Funcs[0].Alias != "count" {
|
||||||
|
t.Fatalf("unexpected aggregation: %+v", plan.Aggregation)
|
||||||
|
}
|
||||||
|
if len(plan.Aggregation.GroupBy) != 1 || plan.Aggregation.GroupBy[0] != "host" {
|
||||||
|
t.Fatalf("unexpected group by: %+v", plan.Aggregation.GroupBy)
|
||||||
|
}
|
||||||
|
if len(plan.Sort) != 1 || plan.Sort[0].Field != "count" || !plan.Sort[0].Desc {
|
||||||
|
t.Fatalf("unexpected sort: %+v", plan.Sort)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestCompileTextSearchWithAggregation(t *testing.T) {
|
||||||
|
plan, err := Compile(`message:"connection refused" | stats count by host`, Auto, fixedNow)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("Compile() error = %v", err)
|
||||||
|
}
|
||||||
|
if len(plan.TextSearch) != 1 || plan.TextSearch[0].Query != `"connection refused"` {
|
||||||
|
t.Fatalf("unexpected text search: %+v", plan.TextSearch)
|
||||||
|
}
|
||||||
|
if plan.Aggregation == nil {
|
||||||
|
t.Fatal("expected an aggregation")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestCompileImplicitAndBetweenFreeTextTerms(t *testing.T) {
|
||||||
|
plan, err := Compile(`error timeout`, Auto, fixedNow)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("Compile() error = %v", err)
|
||||||
|
}
|
||||||
|
if len(plan.TextSearch) != 1 {
|
||||||
|
t.Fatalf("expected 1 combined text predicate, got %+v", plan.TextSearch)
|
||||||
|
}
|
||||||
|
if plan.TextSearch[0].Query != "error AND timeout" {
|
||||||
|
t.Fatalf("Query = %q, want %q", plan.TextSearch[0].Query, "error AND timeout")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestCompileOrBetweenFreeTextTerms(t *testing.T) {
|
||||||
|
plan, err := Compile(`error or timeout`, Auto, fixedNow)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("Compile() error = %v", err)
|
||||||
|
}
|
||||||
|
if plan.TextSearch[0].Query != "error OR timeout" {
|
||||||
|
t.Fatalf("Query = %q, want %q", plan.TextSearch[0].Query, "error OR timeout")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestCompileOrBetweenStructuredFiltersErrors(t *testing.T) {
|
||||||
|
_, err := Compile(`service=api or service=web`, Auto, fixedNow)
|
||||||
|
if err == nil {
|
||||||
|
t.Fatal("expected an error: OR between structured filters isn't supported in Phase 2")
|
||||||
|
}
|
||||||
|
if !strings.Contains(err.Error(), "or") {
|
||||||
|
t.Fatalf("error should mention 'or', got: %v", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestCompileRelativeTimeBound(t *testing.T) {
|
||||||
|
plan, err := Compile(`earliest=-1h`, Auto, fixedNow)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("Compile() error = %v", err)
|
||||||
|
}
|
||||||
|
if plan.TimeRange == nil {
|
||||||
|
t.Fatal("expected a TimeRange")
|
||||||
|
}
|
||||||
|
want := fixedNow.Add(-1 * time.Hour)
|
||||||
|
if !plan.TimeRange.From.Equal(want) {
|
||||||
|
t.Fatalf("From = %v, want %v", plan.TimeRange.From, want)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestCompileAbsoluteTimeBound(t *testing.T) {
|
||||||
|
plan, err := Compile(`latest="2026-08-14T00:00:00Z"`, Auto, fixedNow)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("Compile() error = %v", err)
|
||||||
|
}
|
||||||
|
want := time.Date(2026, 8, 14, 0, 0, 0, 0, time.UTC)
|
||||||
|
if !plan.TimeRange.To.Equal(want) {
|
||||||
|
t.Fatalf("To = %v, want %v", plan.TimeRange.To, want)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestCompileInvalidAbsoluteTimestampErrors(t *testing.T) {
|
||||||
|
_, err := Compile(`latest="not-a-timestamp"`, Auto, fixedNow)
|
||||||
|
if err == nil {
|
||||||
|
t.Fatal("expected an error for an invalid absolute timestamp")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestCompileSumWithoutFieldErrors(t *testing.T) {
|
||||||
|
_, err := Compile(`service=api | stats sum`, Auto, fixedNow)
|
||||||
|
if err == nil {
|
||||||
|
t.Fatal("expected an error: sum() requires a field")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestCompileAggAliasDefaultsAndCollisionIsDisambiguated(t *testing.T) {
|
||||||
|
plan, err := Compile(`service=api | stats sum(a), sum(b)`, Auto, fixedNow)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("Compile() error = %v", err)
|
||||||
|
}
|
||||||
|
if len(plan.Aggregation.Funcs) != 2 {
|
||||||
|
t.Fatalf("expected 2 agg funcs, got %+v", plan.Aggregation.Funcs)
|
||||||
|
}
|
||||||
|
if plan.Aggregation.Funcs[0].Alias == plan.Aggregation.Funcs[1].Alias {
|
||||||
|
t.Fatalf("expected distinct aliases, got both %q", plan.Aggregation.Funcs[0].Alias)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestCompileHeadDefaultsLimit(t *testing.T) {
|
||||||
|
plan, err := Compile(`service=api | head`, Auto, fixedNow)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("Compile() error = %v", err)
|
||||||
|
}
|
||||||
|
if plan.Limit == nil || plan.Limit.N != defaultLimit || plan.Limit.Tail {
|
||||||
|
t.Fatalf("unexpected limit: %+v", plan.Limit)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestCompileTailSetsTailFlag(t *testing.T) {
|
||||||
|
plan, err := Compile(`service=api | tail 5`, Auto, fixedNow)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("Compile() error = %v", err)
|
||||||
|
}
|
||||||
|
if plan.Limit == nil || plan.Limit.N != 5 || !plan.Limit.Tail {
|
||||||
|
t.Fatalf("unexpected limit: %+v", plan.Limit)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestCompileFieldsProjection(t *testing.T) {
|
||||||
|
plan, err := Compile(`service=api | fields host, message`, Auto, fixedNow)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("Compile() error = %v", err)
|
||||||
|
}
|
||||||
|
if len(plan.Fields) != 2 || plan.Fields[0] != "host" || plan.Fields[1] != "message" {
|
||||||
|
t.Fatalf("unexpected fields: %+v", plan.Fields)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestCompileParseErrorPropagates(t *testing.T) {
|
||||||
|
_, err := Compile(`service=api | bogus`, Auto, fixedNow)
|
||||||
|
if err == nil {
|
||||||
|
t.Fatal("expected a parse error to propagate")
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -1,6 +1,8 @@
|
|||||||
// Package searchclient adapts the generated gRPC SearchServiceClient to
|
// Package searchclient adapts the generated gRPC SearchServiceClient to
|
||||||
// the narrow queryapi.searchClient interface, so queryapi doesn't need to
|
// the narrow querylang/executor.SearchClient interface (Search(ctx,
|
||||||
// know anything about gRPC/protobuf directly.
|
// query string, limit uint32) ([]string, error), satisfied structurally,
|
||||||
|
// no adapter type needed), so the query executor doesn't need to know
|
||||||
|
// anything about gRPC/protobuf directly.
|
||||||
package searchclient
|
package searchclient
|
||||||
|
|
||||||
import (
|
import (
|
||||||
|
|||||||
+19
-4
@@ -1,6 +1,6 @@
|
|||||||
# sentryctl
|
# sentryctl
|
||||||
|
|
||||||
Sentry's control CLI. Phase 0: a single command.
|
Sentry's control CLI.
|
||||||
|
|
||||||
```sh
|
```sh
|
||||||
sentryctl ping # checks http://localhost:8080/healthz
|
sentryctl ping # checks http://localhost:8080/healthz
|
||||||
@@ -11,9 +11,24 @@ SENTRYCTL_API_URL=http://api.internal:8080 sentryctl ping
|
|||||||
Exits 0 and prints `ok` if `/api`'s `/healthz` responds 200; exits 1 with an
|
Exits 0 and prints `ok` if `/api`'s `/healthz` responds 200; exits 1 with an
|
||||||
error on `stderr` otherwise.
|
error on `stderr` otherwise.
|
||||||
|
|
||||||
No CLI framework (cobra/urfave-cli/etc.) — a single command doesn't need
|
```sh
|
||||||
one, and stdlib `os.Args` handling is boring enough not to need a
|
sentryctl query 'service=api | where status>=500 | stats count by host'
|
||||||
dependency. Revisit once there's a real command tree to justify one.
|
sentryctl query 'SELECT * FROM logs LIMIT 10' --language sql
|
||||||
|
sentryctl query 'message:"connection refused"' --json
|
||||||
|
```
|
||||||
|
|
||||||
|
Quote the query in your shell — pipe syntax uses `|`, which your shell
|
||||||
|
interprets as an actual pipe if you don't. Hits the exact same `POST
|
||||||
|
/query` endpoint the web UI does (`internal/querylang` in `/api` does the
|
||||||
|
compiling; there's no separate query logic here to drift out of sync —
|
||||||
|
see `/docs/query-language-reference.md`). `--language` overrides
|
||||||
|
auto-detection, same optional override the HTTP API itself exposes.
|
||||||
|
Prints a table by default (stdlib `text/tabwriter`, no new dependency);
|
||||||
|
`--json` prints the raw `{columns, rows}` response instead.
|
||||||
|
|
||||||
|
No CLI framework (cobra/urfave-cli/etc.) — two commands don't need one,
|
||||||
|
and stdlib `os.Args` handling is boring enough not to need a dependency.
|
||||||
|
Revisit once there's a real command tree to justify one.
|
||||||
|
|
||||||
## Building & testing
|
## Building & testing
|
||||||
|
|
||||||
|
|||||||
+159
-9
@@ -1,13 +1,18 @@
|
|||||||
// Command sentryctl is Sentry's control CLI. Phase 0: a single "ping"
|
// Command sentryctl is Sentry's control CLI: "ping" (Phase 0) and
|
||||||
// command that checks the api service is reachable. More commands land as
|
// "query" (Phase 2), which accepts either query syntax and hits the same
|
||||||
// the control plane grows real operations to expose.
|
// POST /query endpoint the web UI does -- no separate query logic here,
|
||||||
|
// per the Phase 2 task list's explicit instruction.
|
||||||
package main
|
package main
|
||||||
|
|
||||||
import (
|
import (
|
||||||
|
"bytes"
|
||||||
|
"encoding/json"
|
||||||
"fmt"
|
"fmt"
|
||||||
"io"
|
"io"
|
||||||
"net/http"
|
"net/http"
|
||||||
"os"
|
"os"
|
||||||
|
"strings"
|
||||||
|
"text/tabwriter"
|
||||||
"time"
|
"time"
|
||||||
)
|
)
|
||||||
|
|
||||||
@@ -26,6 +31,8 @@ func run(args []string, stdout, stderr io.Writer) int {
|
|||||||
switch args[0] {
|
switch args[0] {
|
||||||
case "ping":
|
case "ping":
|
||||||
return cmdPing(args[1:], stdout, stderr)
|
return cmdPing(args[1:], stdout, stderr)
|
||||||
|
case "query":
|
||||||
|
return cmdQuery(args[1:], stdout, stderr)
|
||||||
case "-h", "--help", "help":
|
case "-h", "--help", "help":
|
||||||
usage(stdout)
|
usage(stdout)
|
||||||
return 0
|
return 0
|
||||||
@@ -37,15 +44,28 @@ func run(args []string, stdout, stderr io.Writer) int {
|
|||||||
}
|
}
|
||||||
|
|
||||||
func usage(w io.Writer) {
|
func usage(w io.Writer) {
|
||||||
fmt.Fprintln(w, `sentryctl: Sentry control CLI (Phase 0: ping only)
|
fmt.Fprintln(w, `sentryctl: Sentry control CLI
|
||||||
|
|
||||||
Usage:
|
Usage:
|
||||||
sentryctl ping [--api <url>]
|
sentryctl ping [--api <url>]
|
||||||
|
sentryctl query "<query>" [--api <url>] [--language sql|spl] [--json]
|
||||||
|
|
||||||
Commands:
|
Commands:
|
||||||
ping Checks that the api service is reachable via GET /healthz.
|
ping Checks that the api service is reachable via GET /healthz.
|
||||||
|
query Runs a query (pipe syntax or SQL) against POST /query and
|
||||||
|
prints the result as a table, or as JSON with --json. Quote
|
||||||
|
the query in your shell -- pipe syntax uses "|", which your
|
||||||
|
shell will otherwise interpret itself.
|
||||||
|
|
||||||
--api defaults to $SENTRYCTL_API_URL, or `+defaultAPIURL+` if unset.`)
|
--api defaults to $SENTRYCTL_API_URL, or `+defaultAPIURL+` if unset.
|
||||||
|
--language overrides auto-detection; omit it for the common case.`)
|
||||||
|
}
|
||||||
|
|
||||||
|
func resolveAPIURL(env func(string) string) string {
|
||||||
|
if v := env("SENTRYCTL_API_URL"); v != "" {
|
||||||
|
return v
|
||||||
|
}
|
||||||
|
return defaultAPIURL
|
||||||
}
|
}
|
||||||
|
|
||||||
// parsePingArgs resolves the api base URL for ping: --api flag wins, then
|
// parsePingArgs resolves the api base URL for ping: --api flag wins, then
|
||||||
@@ -53,10 +73,7 @@ Commands:
|
|||||||
// as a function) and separate from the HTTP call so it's unit-testable
|
// as a function) and separate from the HTTP call so it's unit-testable
|
||||||
// without a real environment or server.
|
// without a real environment or server.
|
||||||
func parsePingArgs(args []string, env func(string) string) string {
|
func parsePingArgs(args []string, env func(string) string) string {
|
||||||
apiURL := env("SENTRYCTL_API_URL")
|
apiURL := resolveAPIURL(env)
|
||||||
if apiURL == "" {
|
|
||||||
apiURL = defaultAPIURL
|
|
||||||
}
|
|
||||||
for i := 0; i < len(args); i++ {
|
for i := 0; i < len(args); i++ {
|
||||||
if args[i] == "--api" && i+1 < len(args) {
|
if args[i] == "--api" && i+1 < len(args) {
|
||||||
apiURL = args[i+1]
|
apiURL = args[i+1]
|
||||||
@@ -85,3 +102,136 @@ func cmdPing(args []string, stdout, stderr io.Writer) int {
|
|||||||
fmt.Fprintln(stdout, "ok")
|
fmt.Fprintln(stdout, "ok")
|
||||||
return 0
|
return 0
|
||||||
}
|
}
|
||||||
|
|
||||||
|
type queryArgs struct {
|
||||||
|
apiURL string
|
||||||
|
jsonOut bool
|
||||||
|
language string
|
||||||
|
query string
|
||||||
|
}
|
||||||
|
|
||||||
|
// parseQueryArgs is pure (env passed in, no I/O), same testability
|
||||||
|
// reasoning as parsePingArgs. Non-flag arguments are joined with spaces
|
||||||
|
// to form the query, so `sentryctl query service=api status=500` (no
|
||||||
|
// quotes, no shell-special characters) works without requiring users to
|
||||||
|
// quote every query -- though anything using "|" still needs shell
|
||||||
|
// quoting regardless, since that's a real shell pipe character otherwise.
|
||||||
|
func parseQueryArgs(args []string, env func(string) string) queryArgs {
|
||||||
|
qa := queryArgs{apiURL: resolveAPIURL(env)}
|
||||||
|
var rest []string
|
||||||
|
for i := 0; i < len(args); i++ {
|
||||||
|
switch args[i] {
|
||||||
|
case "--api":
|
||||||
|
if i+1 < len(args) {
|
||||||
|
qa.apiURL = args[i+1]
|
||||||
|
i++
|
||||||
|
}
|
||||||
|
case "--json":
|
||||||
|
qa.jsonOut = true
|
||||||
|
case "--language":
|
||||||
|
if i+1 < len(args) {
|
||||||
|
qa.language = args[i+1]
|
||||||
|
i++
|
||||||
|
}
|
||||||
|
default:
|
||||||
|
rest = append(rest, args[i])
|
||||||
|
}
|
||||||
|
}
|
||||||
|
qa.query = strings.Join(rest, " ")
|
||||||
|
return qa
|
||||||
|
}
|
||||||
|
|
||||||
|
type queryRequestBody struct {
|
||||||
|
Query string `json:"query"`
|
||||||
|
Language string `json:"language"`
|
||||||
|
}
|
||||||
|
|
||||||
|
type queryResponseBody struct {
|
||||||
|
Columns []string `json:"columns"`
|
||||||
|
Rows [][]any `json:"rows"`
|
||||||
|
}
|
||||||
|
|
||||||
|
type errorResponseBody struct {
|
||||||
|
Error string `json:"error"`
|
||||||
|
}
|
||||||
|
|
||||||
|
func cmdQuery(args []string, stdout, stderr io.Writer) int {
|
||||||
|
qa := parseQueryArgs(args, os.Getenv)
|
||||||
|
if strings.TrimSpace(qa.query) == "" {
|
||||||
|
fmt.Fprintln(stderr, "sentryctl query: missing query string")
|
||||||
|
return 1
|
||||||
|
}
|
||||||
|
|
||||||
|
reqBody, err := json.Marshal(queryRequestBody{Query: qa.query, Language: qa.language})
|
||||||
|
if err != nil {
|
||||||
|
fmt.Fprintf(stderr, "encoding request: %v\n", err)
|
||||||
|
return 1
|
||||||
|
}
|
||||||
|
|
||||||
|
client := &http.Client{Timeout: 30 * time.Second}
|
||||||
|
resp, err := client.Post(qa.apiURL+"/query", "application/json", bytes.NewReader(reqBody))
|
||||||
|
if err != nil {
|
||||||
|
fmt.Fprintf(stderr, "query failed: %v\n", err)
|
||||||
|
return 1
|
||||||
|
}
|
||||||
|
defer resp.Body.Close()
|
||||||
|
|
||||||
|
respBody, err := io.ReadAll(resp.Body)
|
||||||
|
if err != nil {
|
||||||
|
fmt.Fprintf(stderr, "reading response: %v\n", err)
|
||||||
|
return 1
|
||||||
|
}
|
||||||
|
|
||||||
|
if resp.StatusCode != http.StatusOK {
|
||||||
|
var errResp errorResponseBody
|
||||||
|
if json.Unmarshal(respBody, &errResp) == nil && errResp.Error != "" {
|
||||||
|
fmt.Fprintf(stderr, "query failed: %s\n", errResp.Error)
|
||||||
|
} else {
|
||||||
|
fmt.Fprintf(stderr, "query failed: api returned status %d\n", resp.StatusCode)
|
||||||
|
}
|
||||||
|
return 1
|
||||||
|
}
|
||||||
|
|
||||||
|
if qa.jsonOut {
|
||||||
|
_, _ = stdout.Write(respBody)
|
||||||
|
fmt.Fprintln(stdout)
|
||||||
|
return 0
|
||||||
|
}
|
||||||
|
|
||||||
|
var result queryResponseBody
|
||||||
|
if err := json.Unmarshal(respBody, &result); err != nil {
|
||||||
|
fmt.Fprintf(stderr, "decoding response: %v\n", err)
|
||||||
|
return 1
|
||||||
|
}
|
||||||
|
printTable(stdout, result.Columns, result.Rows)
|
||||||
|
return 0
|
||||||
|
}
|
||||||
|
|
||||||
|
func printTable(w io.Writer, columns []string, rows [][]any) {
|
||||||
|
tw := tabwriter.NewWriter(w, 0, 4, 2, ' ', 0)
|
||||||
|
fmt.Fprintln(tw, strings.Join(columns, "\t"))
|
||||||
|
for _, row := range rows {
|
||||||
|
cells := make([]string, len(row))
|
||||||
|
for i, v := range row {
|
||||||
|
cells[i] = formatCell(v)
|
||||||
|
}
|
||||||
|
fmt.Fprintln(tw, strings.Join(cells, "\t"))
|
||||||
|
}
|
||||||
|
_ = tw.Flush()
|
||||||
|
fmt.Fprintf(w, "(%d row(s))\n", len(rows))
|
||||||
|
}
|
||||||
|
|
||||||
|
func formatCell(v any) string {
|
||||||
|
switch t := v.(type) {
|
||||||
|
case nil:
|
||||||
|
return ""
|
||||||
|
case map[string]any, []any:
|
||||||
|
b, err := json.Marshal(t)
|
||||||
|
if err != nil {
|
||||||
|
return fmt.Sprintf("%v", t)
|
||||||
|
}
|
||||||
|
return string(b)
|
||||||
|
default:
|
||||||
|
return fmt.Sprintf("%v", t)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|||||||
@@ -2,6 +2,7 @@ package main
|
|||||||
|
|
||||||
import (
|
import (
|
||||||
"bytes"
|
"bytes"
|
||||||
|
"encoding/json"
|
||||||
"net/http"
|
"net/http"
|
||||||
"net/http/httptest"
|
"net/http/httptest"
|
||||||
"strings"
|
"strings"
|
||||||
@@ -121,3 +122,129 @@ func TestRunHelp(t *testing.T) {
|
|||||||
t.Fatalf("stdout should contain usage text, got %q", stdout.String())
|
t.Fatalf("stdout should contain usage text, got %q", stdout.String())
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func TestParseQueryArgsJoinsNonFlagArgsAsQuery(t *testing.T) {
|
||||||
|
env := func(string) string { return "" }
|
||||||
|
qa := parseQueryArgs([]string{"service=api", "status=500"}, env)
|
||||||
|
if qa.query != "service=api status=500" {
|
||||||
|
t.Errorf("query = %q", qa.query)
|
||||||
|
}
|
||||||
|
if qa.apiURL != defaultAPIURL {
|
||||||
|
t.Errorf("apiURL = %q, want default", qa.apiURL)
|
||||||
|
}
|
||||||
|
if qa.jsonOut {
|
||||||
|
t.Error("jsonOut should default to false")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestParseQueryArgsFlags(t *testing.T) {
|
||||||
|
env := func(string) string { return "" }
|
||||||
|
qa := parseQueryArgs([]string{"--api", "http://h:1", "--language", "sql", "--json", "SELECT", "1"}, env)
|
||||||
|
if qa.apiURL != "http://h:1" {
|
||||||
|
t.Errorf("apiURL = %q", qa.apiURL)
|
||||||
|
}
|
||||||
|
if qa.language != "sql" {
|
||||||
|
t.Errorf("language = %q", qa.language)
|
||||||
|
}
|
||||||
|
if !qa.jsonOut {
|
||||||
|
t.Error("expected jsonOut = true")
|
||||||
|
}
|
||||||
|
if qa.query != "SELECT 1" {
|
||||||
|
t.Errorf("query = %q", qa.query)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestCmdQueryMissingQueryErrors(t *testing.T) {
|
||||||
|
var stdout, stderr bytes.Buffer
|
||||||
|
code := cmdQuery(nil, &stdout, &stderr)
|
||||||
|
if code != 1 {
|
||||||
|
t.Fatalf("exit code = %d, want 1", code)
|
||||||
|
}
|
||||||
|
if !strings.Contains(stderr.String(), "missing query") {
|
||||||
|
t.Fatalf("stderr = %q", stderr.String())
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestCmdQueryTableOutput(t *testing.T) {
|
||||||
|
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||||
|
if r.URL.Path != "/query" {
|
||||||
|
t.Errorf("unexpected path %q", r.URL.Path)
|
||||||
|
}
|
||||||
|
var body queryRequestBody
|
||||||
|
if err := json.NewDecoder(r.Body).Decode(&body); err != nil {
|
||||||
|
t.Fatalf("decoding request: %v", err)
|
||||||
|
}
|
||||||
|
if body.Query != "service=api" {
|
||||||
|
t.Errorf("query = %q", body.Query)
|
||||||
|
}
|
||||||
|
w.Header().Set("Content-Type", "application/json")
|
||||||
|
_ = json.NewEncoder(w).Encode(queryResponseBody{
|
||||||
|
Columns: []string{"host", "count"},
|
||||||
|
Rows: [][]any{{"h1", float64(3)}},
|
||||||
|
})
|
||||||
|
}))
|
||||||
|
defer srv.Close()
|
||||||
|
|
||||||
|
var stdout, stderr bytes.Buffer
|
||||||
|
code := cmdQuery([]string{"--api", srv.URL, "service=api"}, &stdout, &stderr)
|
||||||
|
|
||||||
|
if code != 0 {
|
||||||
|
t.Fatalf("exit code = %d, want 0; stderr=%s", code, stderr.String())
|
||||||
|
}
|
||||||
|
out := stdout.String()
|
||||||
|
if !strings.Contains(out, "host") || !strings.Contains(out, "h1") || !strings.Contains(out, "(1 row(s))") {
|
||||||
|
t.Fatalf("unexpected table output: %q", out)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestCmdQueryJSONOutput(t *testing.T) {
|
||||||
|
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||||
|
w.Header().Set("Content-Type", "application/json")
|
||||||
|
_ = json.NewEncoder(w).Encode(queryResponseBody{Columns: []string{"c"}, Rows: [][]any{{"v"}}})
|
||||||
|
}))
|
||||||
|
defer srv.Close()
|
||||||
|
|
||||||
|
var stdout, stderr bytes.Buffer
|
||||||
|
code := cmdQuery([]string{"--api", srv.URL, "--json", "service=api"}, &stdout, &stderr)
|
||||||
|
|
||||||
|
if code != 0 {
|
||||||
|
t.Fatalf("exit code = %d, want 0; stderr=%s", code, stderr.String())
|
||||||
|
}
|
||||||
|
var got queryResponseBody
|
||||||
|
if err := json.Unmarshal(stdout.Bytes(), &got); err != nil {
|
||||||
|
t.Fatalf("stdout is not valid JSON: %v; got %q", err, stdout.String())
|
||||||
|
}
|
||||||
|
if len(got.Columns) != 1 || got.Columns[0] != "c" {
|
||||||
|
t.Fatalf("unexpected JSON output: %+v", got)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestCmdQueryServerErrorPrintsMessage(t *testing.T) {
|
||||||
|
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||||
|
w.WriteHeader(http.StatusBadRequest)
|
||||||
|
_ = json.NewEncoder(w).Encode(errorResponseBody{Error: "only SELECT queries are allowed"})
|
||||||
|
}))
|
||||||
|
defer srv.Close()
|
||||||
|
|
||||||
|
var stdout, stderr bytes.Buffer
|
||||||
|
code := cmdQuery([]string{"--api", srv.URL, "DELETE FROM logs", "--language", "sql"}, &stdout, &stderr)
|
||||||
|
|
||||||
|
if code != 1 {
|
||||||
|
t.Fatalf("exit code = %d, want 1", code)
|
||||||
|
}
|
||||||
|
if !strings.Contains(stderr.String(), "only SELECT queries are allowed") {
|
||||||
|
t.Fatalf("stderr = %q, want it to include the server's error message", stderr.String())
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestFormatCellHandlesNilMapAndSlice(t *testing.T) {
|
||||||
|
if got := formatCell(nil); got != "" {
|
||||||
|
t.Errorf("formatCell(nil) = %q, want empty", got)
|
||||||
|
}
|
||||||
|
if got := formatCell(map[string]any{"a": "b"}); got != `{"a":"b"}` {
|
||||||
|
t.Errorf("formatCell(map) = %q", got)
|
||||||
|
}
|
||||||
|
if got := formatCell(42.0); got != "42" {
|
||||||
|
t.Errorf("formatCell(42.0) = %q", got)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|||||||
@@ -0,0 +1,178 @@
|
|||||||
|
# Phase 2 runbook
|
||||||
|
|
||||||
|
Extends `/docs/phase-0-runbook.md` and `/docs/phase-1-runbook.md` with
|
||||||
|
the unified query language and the "modest dataset" query-latency
|
||||||
|
benchmark. Read those first — this assumes the stack already works;
|
||||||
|
Phase 2 replaces the two placeholder query endpoints/pages with one, and
|
||||||
|
adds a way to actually measure query performance at volume.
|
||||||
|
|
||||||
|
## What's actually been verified
|
||||||
|
|
||||||
|
Unlike the phrasing "define 'modest', put a rough benchmark in the
|
||||||
|
runbook" might suggest, this isn't an estimate — the numbers below are
|
||||||
|
from actually generating 1,022,000 rows in a live ClickHouse (via
|
||||||
|
`/hack/benchmark-fixture`, pushed through the real agent-facing gRPC
|
||||||
|
endpoint, so both ClickHouse and Tantivy have real, matching data) and
|
||||||
|
timing real queries against it. One real bug turned up doing this (see
|
||||||
|
"What the benchmark caught" below) that wouldn't have been found any
|
||||||
|
other way.
|
||||||
|
|
||||||
|
## 1. Bring up the stack
|
||||||
|
|
||||||
|
```sh
|
||||||
|
docker compose up -d --build
|
||||||
|
```
|
||||||
|
|
||||||
|
Same as Phase 0/1. Confirm the unified endpoint works for both syntaxes:
|
||||||
|
|
||||||
|
```sh
|
||||||
|
curl -X POST http://localhost:8080/query -H 'Content-Type: application/json' \
|
||||||
|
-d '{"query": "SELECT 1"}'
|
||||||
|
# -> {"columns":["1"],"rows":[[1]]}
|
||||||
|
|
||||||
|
curl -X POST http://localhost:8080/query -H 'Content-Type: application/json' \
|
||||||
|
-d '{"query": "service=api"}'
|
||||||
|
# -> {"columns":[...],"rows":[]} (empty is fine -- no data ingested yet)
|
||||||
|
```
|
||||||
|
|
||||||
|
## 2. Generate the "modest" benchmark dataset
|
||||||
|
|
||||||
|
"Modest" is defined here as **1,000,000 rows** — enough to be a real
|
||||||
|
volume test, small enough to generate and query interactively rather
|
||||||
|
than needing a dedicated load-testing pass.
|
||||||
|
|
||||||
|
```sh
|
||||||
|
cd hack/benchmark-fixture
|
||||||
|
go run . --count 1000000 --batch-size 1000 --concurrency 16
|
||||||
|
```
|
||||||
|
|
||||||
|
Sends batched `PushBatch` calls directly to `ingest`'s gRPC endpoint —
|
||||||
|
the same path the real agent uses, just generating synthetic data
|
||||||
|
instead of reading journald, so it exercises the *real* ingest →
|
||||||
|
Redpanda → ClickHouse-writer-consumer and → search-indexer-consumer
|
||||||
|
paths, not a shortcut that bypasses them. Concurrency matters:
|
||||||
|
sequential single-batch calls topped out around 500 records/sec (would
|
||||||
|
take ~33 minutes for 1M rows); 16 concurrent `PushBatch` calls over one
|
||||||
|
shared gRPC connection (HTTP/2 multiplexes concurrent RPCs over it
|
||||||
|
natively) got over 400,000 records/sec, so the actual measured run took
|
||||||
|
about 2 seconds for the ingest call itself.
|
||||||
|
|
||||||
|
Wait for both consumers to fully drain before benchmarking — ingestion
|
||||||
|
finishing doesn't mean ClickHouse/Tantivy are caught up yet:
|
||||||
|
|
||||||
|
```sh
|
||||||
|
# poll until this stops climbing
|
||||||
|
curl -s -X POST http://localhost:8080/query -H 'Content-Type: application/json' \
|
||||||
|
-d '{"query": "SELECT count() FROM logs", "language": "sql"}'
|
||||||
|
```
|
||||||
|
|
||||||
|
In the run this runbook was written from, ClickHouse settled at
|
||||||
|
1,022,000 rows a few seconds after ingestion finished (the extra 22,000
|
||||||
|
were earlier smaller test runs from developing the benchmark tool itself
|
||||||
|
— harmless, still real rows).
|
||||||
|
|
||||||
|
## 3. Run the benchmark queries
|
||||||
|
|
||||||
|
```sh
|
||||||
|
# structured filter + aggregation -- pure ClickHouse path
|
||||||
|
curl -s -o /dev/null -w "%{time_total}s\n" -X POST http://localhost:8080/query \
|
||||||
|
-H 'Content-Type: application/json' \
|
||||||
|
-d '{"query": "service=api | where status>=500 | stats count by host | sort -count"}'
|
||||||
|
|
||||||
|
# free-text + aggregation -- Tantivy prefilter feeding a ClickHouse GROUP BY,
|
||||||
|
# the case /docs/query-language-design.md called "the hardest part"
|
||||||
|
curl -s -o /dev/null -w "%{time_total}s\n" -X POST http://localhost:8080/query \
|
||||||
|
-H 'Content-Type: application/json' \
|
||||||
|
-d '{"query": "message:\"connection refused\" | stats count by host | sort -count"}'
|
||||||
|
```
|
||||||
|
|
||||||
|
**Measured results against the 1,022,000-row dataset** (this exact run,
|
||||||
|
not a projection):
|
||||||
|
|
||||||
|
| Query | Path | Wall time |
|
||||||
|
|---|---|---|
|
||||||
|
| `service=api \| where status>=500 \| stats count by host \| sort -count` | Pure ClickHouse | 19.7ms |
|
||||||
|
| `message:"connection refused" \| stats count by host \| sort -count` | Tantivy prefilter + ClickHouse aggregate | 46.4ms |
|
||||||
|
| `message:"connection refused" \| head 50` | Tantivy prefilter, no aggregation | 49.2ms |
|
||||||
|
| `SELECT count() FROM logs WHERE service='web'` (raw SQL) | Pure ClickHouse | 17.5ms |
|
||||||
|
|
||||||
|
All four "well under a second" — the Phase 2 exit criteria in
|
||||||
|
`/CLAUDE.md`. The combined text+aggregation case (the one everyone should
|
||||||
|
be nervous about, since it's a two-backend query) came in at 46ms, not
|
||||||
|
meaningfully slower than the pure-ClickHouse case — the Tantivy prefilter
|
||||||
|
step is fast, and 5,000 UUIDs (see below) is a small `IN` clause by
|
||||||
|
ClickHouse's standards.
|
||||||
|
|
||||||
|
## 4. What the benchmark caught
|
||||||
|
|
||||||
|
The original design capped the Tantivy prefilter at **10,000** matching
|
||||||
|
`record_id`s before feeding them into ClickHouse's `WHERE record_id IN
|
||||||
|
(...)`. Running the actual combined-query benchmark against real data
|
||||||
|
(where "connection refused" matched a large fraction of the 1M rows,
|
||||||
|
by design — the fixture generator seeds that phrase deliberately) hit
|
||||||
|
this immediately:
|
||||||
|
|
||||||
|
```
|
||||||
|
query failed: executing query: code: 62, message: Syntax error:
|
||||||
|
failed at position 262116 [...] Max query size exceeded
|
||||||
|
```
|
||||||
|
|
||||||
|
10,000 quoted UUIDs (~39 bytes each including the separating comma)
|
||||||
|
produces a ~390KB query string, which exceeds ClickHouse's *default*
|
||||||
|
`max_query_size` (262144 bytes / 256KiB) — a much lower ceiling than
|
||||||
|
"multi-million-entry" suggested before anyone had actually tried it. The
|
||||||
|
cap is now **5,000** (~195KB, safely under the default with headroom) —
|
||||||
|
see `api/internal/querylang/executor/executor.go`'s `textSearchLimit`
|
||||||
|
and `/docs/query-language-design.md`'s "Known scaling limitation"
|
||||||
|
section, both updated with this exact finding rather than a
|
||||||
|
first-principles guess.
|
||||||
|
|
||||||
|
## 5. Confirm the web UI and CLI both work end to end
|
||||||
|
|
||||||
|
Open `http://localhost:3000` — one page now (Phase 0/1's two pages are
|
||||||
|
gone), run `service=api | where status>=500 | stats count by host | sort
|
||||||
|
-count` in the query bar, confirm results render and the query appears
|
||||||
|
in the session-local history panel.
|
||||||
|
|
||||||
|
```sh
|
||||||
|
cd cli
|
||||||
|
go run ./cmd/sentryctl query 'service=api | where status>=500 | stats count by host | sort -count'
|
||||||
|
```
|
||||||
|
|
||||||
|
Both hit the exact same `POST /query` endpoint — there's no separate
|
||||||
|
query logic to drift out of sync between the CLI, the web UI, and
|
||||||
|
whatever else calls this API later.
|
||||||
|
|
||||||
|
## Tearing down
|
||||||
|
|
||||||
|
```sh
|
||||||
|
docker compose down -v # also wipes the 1M-row benchmark dataset
|
||||||
|
```
|
||||||
|
|
||||||
|
## Troubleshooting
|
||||||
|
|
||||||
|
**A text-search query with aggregation fails with a ClickHouse syntax
|
||||||
|
error / "Max query size exceeded."**
|
||||||
|
If you've raised `textSearchLimit` above 5,000, you've likely
|
||||||
|
reintroduced the exact failure this runbook's benchmark caught (see
|
||||||
|
"What the benchmark caught" above) — lower it back down, or raise
|
||||||
|
ClickHouse's `max_query_size` server-side with matching memory sizing if
|
||||||
|
you genuinely need a larger prefilter.
|
||||||
|
|
||||||
|
**Benchmark ingestion is much slower than ~400K records/sec.**
|
||||||
|
Check `--concurrency` wasn't left at a low value, and that `ingest`
|
||||||
|
isn't CPU-starved (`docker stats`) — the numbers above are from a single
|
||||||
|
machine running the whole stack including Redpanda/ClickHouse/search
|
||||||
|
concurrently, not a dedicated load-test environment, so your exact
|
||||||
|
throughput will vary with hardware. What matters is "well under a
|
||||||
|
second" for the actual query latency, not the ingestion throughput
|
||||||
|
number itself, which is just how the test data gets there.
|
||||||
|
|
||||||
|
**`stats sum(field)`/`avg(field)` etc. on an attributes-map field
|
||||||
|
returns 0 for everything.**
|
||||||
|
Check the field's values are actually numeric strings — non-numeric
|
||||||
|
attribute values silently cast to 0 via `toFloat64OrZero` (see
|
||||||
|
`/docs/query-language-reference.md`'s "Field mapping" section). This is
|
||||||
|
documented behavior, not a bug, but it's easy to trip over with a typo'd
|
||||||
|
field name (which also "succeeds" with all zeros, since a missing map
|
||||||
|
key reads as an empty string, which also casts to 0).
|
||||||
@@ -0,0 +1,250 @@
|
|||||||
|
# Query language design
|
||||||
|
|
||||||
|
> **Status:** Design, approved 2026-08-14, not yet implemented (that's
|
||||||
|
> Task 3). This is the reference Task 3's implementation is built against
|
||||||
|
> — if implementation reveals this design is wrong somewhere, fix this
|
||||||
|
> doc in the same change, don't let them drift apart.
|
||||||
|
|
||||||
|
## Why this design, in one paragraph
|
||||||
|
|
||||||
|
Phases 0–1 shipped two disconnected, placeholder query paths: raw SQL
|
||||||
|
against ClickHouse, and free-text against Tantivy. Phase 2 needs one
|
||||||
|
query language that can express both filter/aggregation and free-text
|
||||||
|
search in a single query, without picking a winner between "give up
|
||||||
|
structured querying" and "give up full-text search." The approach below
|
||||||
|
does that by keeping parsing and execution strictly separate (a small
|
||||||
|
pipe-syntax grammar and an "opaque SQL" passthrough both compile to the
|
||||||
|
same IR) and by generalizing a mechanism Phase 1 already built and proved
|
||||||
|
works (Tantivy-prefilter → ClickHouse `IN (...)`) rather than inventing a
|
||||||
|
new cross-backend join strategy from scratch.
|
||||||
|
|
||||||
|
## Grammar
|
||||||
|
|
||||||
|
Pipe syntax, SPL-inspired, EBNF-ish:
|
||||||
|
|
||||||
|
```
|
||||||
|
query := base_search ("|" pipe_stage)*
|
||||||
|
base_search := bool_expr // implicit filter/search, SPL convention
|
||||||
|
pipe_stage := "where" bool_expr
|
||||||
|
| "stats" agg_call ("," agg_call)* ["by" field ("," field)*]
|
||||||
|
| "sort" sort_field ("," sort_field)*
|
||||||
|
| "fields" field ("," field)*
|
||||||
|
| "head" [INT]
|
||||||
|
| "tail" [INT]
|
||||||
|
|
||||||
|
bool_expr := term (("and" | "or") term)*
|
||||||
|
term := field comparator value // structured filter -> ClickHouse
|
||||||
|
| "earliest" "=" time_expr // time range lower bound
|
||||||
|
| "latest" "=" time_expr // time range upper bound
|
||||||
|
| STRING | QUOTED_STRING // bare term -> free-text (Tantivy) on `message`
|
||||||
|
| "message" ":" QUOTED_STRING // explicit free-text (Tantivy phrase/wildcard syntax passed through)
|
||||||
|
|
||||||
|
comparator := "=" | "!=" | ">" | ">=" | "<" | "<="
|
||||||
|
agg_call := IDENT "(" [field] ")" ["as" IDENT] // count(), sum(field), avg(field), min(field), max(field)
|
||||||
|
sort_field := ["-" | "+"] field // "-" = desc (default), "+" = asc
|
||||||
|
time_expr := QUOTED_STRING // absolute RFC3339
|
||||||
|
| "-" INT ("s"|"m"|"h"|"d"|"w") // relative to query time, e.g. -1h, -7d
|
||||||
|
field := IDENT
|
||||||
|
```
|
||||||
|
|
||||||
|
### Worked examples
|
||||||
|
|
||||||
|
- `service=api | where status>=500 | stats count by host | sort -count`
|
||||||
|
— `service=api` is the base filter (structured, top-level column);
|
||||||
|
`where status>=500` filters on `status`, which isn't a top-level
|
||||||
|
column (see field mapping below); `stats count by host` aggregates;
|
||||||
|
`sort -count` orders descending by the aggregate's implicit `count`
|
||||||
|
alias.
|
||||||
|
- `message:"connection refused" | stats count by host` — free-text
|
||||||
|
predicate feeding a ClickHouse aggregation. This is the case task 2
|
||||||
|
called "the hardest part" — see Execution below.
|
||||||
|
- `SELECT host, count(*) FROM logs GROUP BY host` — detected as SQL (see
|
||||||
|
Detection below), executed directly against ClickHouse.
|
||||||
|
|
||||||
|
## Parser: hand-written recursive descent, no new dependency
|
||||||
|
|
||||||
|
This grammar is small and stable — seven pipe-stage kinds, one
|
||||||
|
expression grammar for filters. A hand-written lexer + recursive-descent
|
||||||
|
parser beats a combinator library (e.g. `participle`) or a generator
|
||||||
|
(`goyacc`) here:
|
||||||
|
|
||||||
|
- **No new dependency.** Consistent with "ask before adding a new
|
||||||
|
external dependency, there's no case for one at this grammar size.
|
||||||
|
- **Error messages matter** for a user-facing query language in a way
|
||||||
|
they don't for most internal parsing — "expected `by` after `stats
|
||||||
|
count`, got `sort`" is easy to produce by hand, harder to get right
|
||||||
|
through a combinator or generated parser.
|
||||||
|
- Generator tooling (`goyacc`) adds a codegen build step disproportionate
|
||||||
|
to a grammar this size.
|
||||||
|
- This is the standard approach for small, real query DSLs at this
|
||||||
|
scope — not a novel choice.
|
||||||
|
|
||||||
|
## The SQL escape hatch: not parsed, wrapped as opaque IR
|
||||||
|
|
||||||
|
"Both syntaxes compile to the same IR" does not mean writing a SQL
|
||||||
|
parser — reimplementing ClickHouse's SQL dialect would be a large,
|
||||||
|
pointless undertaking when ClickHouse already parses its own SQL. A query
|
||||||
|
that starts with `SELECT` (case-insensitive — see Detection) skips the
|
||||||
|
pipe-syntax parser entirely and produces an IR value that wraps the raw
|
||||||
|
SQL string as an opaque passthrough node. Both syntaxes still flow
|
||||||
|
through the same `Plan` type and the same executor code path — that's
|
||||||
|
what "same IR" actually buys (one execution and testing surface), not a
|
||||||
|
shared abstract syntax tree. The existing SELECT-only / single-statement
|
||||||
|
/ keyword-blocklist validation (`api/internal/queryapi/validate.go`) is
|
||||||
|
reused unchanged as the guard before wrapping.
|
||||||
|
|
||||||
|
## IR (`Plan`)
|
||||||
|
|
||||||
|
```go
|
||||||
|
type Plan struct {
|
||||||
|
RawSQL string // set => everything else is ignored; opaque ClickHouse passthrough
|
||||||
|
TextSearch []TextPredicate // bare terms / message: clauses -> routed to Tantivy
|
||||||
|
Filters []FilterPredicate // structured comparisons -> ClickHouse WHERE
|
||||||
|
TimeRange *TimeRange
|
||||||
|
Aggregation *Aggregation // nil => raw rows, no GROUP BY
|
||||||
|
Sort []SortField
|
||||||
|
Fields []string // projection; empty => all columns
|
||||||
|
Limit *Limit // head/tail
|
||||||
|
}
|
||||||
|
|
||||||
|
type TextPredicate struct {
|
||||||
|
Query string // passed to Tantivy's query parser as-is
|
||||||
|
}
|
||||||
|
|
||||||
|
type FilterPredicate struct {
|
||||||
|
Field string
|
||||||
|
Op string // "=", "!=", ">", ">=", "<", "<="
|
||||||
|
Value string
|
||||||
|
}
|
||||||
|
|
||||||
|
type Aggregation struct {
|
||||||
|
Funcs []AggFunc // count/sum/avg/min/max, each with an optional field + alias
|
||||||
|
GroupBy []string
|
||||||
|
}
|
||||||
|
|
||||||
|
type AggFunc struct {
|
||||||
|
Func string
|
||||||
|
Field string // empty for count()
|
||||||
|
Alias string
|
||||||
|
}
|
||||||
|
|
||||||
|
type SortField struct {
|
||||||
|
Field string
|
||||||
|
Desc bool
|
||||||
|
}
|
||||||
|
|
||||||
|
type Limit struct {
|
||||||
|
N int
|
||||||
|
Tail bool // true = last N (by time), false = first N
|
||||||
|
}
|
||||||
|
|
||||||
|
type TimeRange struct {
|
||||||
|
From, To time.Time // relative expressions (-1h etc.) resolved to absolute at compile time
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
## Field mapping: top-level columns vs. `attributes`
|
||||||
|
|
||||||
|
`logs`' real columns (per `/storage`) are `timestamp, host, service,
|
||||||
|
severity, message, attributes, record_id`. Any field name in a query
|
||||||
|
that isn't one of those maps to `attributes['<field>']` — e.g.
|
||||||
|
`status>=500` compiles to a comparison against `attributes['status']`,
|
||||||
|
not a top-level column, since `status` isn't promoted (Phase 1's decision
|
||||||
|
not to promote anything without real usage data still holds). Because
|
||||||
|
`attributes` is `Map(String,String)`, every stored value is a string;
|
||||||
|
numeric comparators against a non-top-level field cast via
|
||||||
|
`toFloat64OrZero(attributes['field'])` when the compared value looks
|
||||||
|
numeric, otherwise compare as string. This is what makes `where
|
||||||
|
status>=500` work against the existing schema with no migration —
|
||||||
|
querying an unpromoted field is always slightly more expensive than a
|
||||||
|
top-level column, which is worth knowing, not hiding.
|
||||||
|
|
||||||
|
## Execution: routing between ClickHouse and Tantivy
|
||||||
|
|
||||||
|
The core mechanism already exists and is proven: Phase 1's `/search`
|
||||||
|
endpoint (`api/internal/queryapi/search.go`, `recordIDsQuery`) already
|
||||||
|
does exactly steps 1–2 below for text-only queries. Phase 2 generalizes
|
||||||
|
it into four cases:
|
||||||
|
|
||||||
|
1. **No `TextSearch` predicates** → pure ClickHouse path. Build one SQL
|
||||||
|
statement directly from `Filters`/`TimeRange`/`Aggregation`/`Sort`/
|
||||||
|
`Fields`/`Limit`. The common case, and the fast path.
|
||||||
|
2. **`TextSearch` predicates, no `Aggregation`** → Phase 1's `/search`
|
||||||
|
behavior, generalized: Tantivy resolves matching `record_id`s, then
|
||||||
|
`SELECT ... WHERE record_id IN (...)` for the rows, with `Filters`/
|
||||||
|
`TimeRange`/`Sort`/`Fields`/`Limit` folded into that same statement.
|
||||||
|
3. **`TextSearch` predicates *and* `Aggregation`** — the genuinely new
|
||||||
|
case (`message:"connection refused" | stats count by host`): Tantivy
|
||||||
|
resolves matching `record_id`s as a *prefilter*, not a join, then one
|
||||||
|
ClickHouse statement does `WHERE record_id IN (...) AND <Filters>
|
||||||
|
GROUP BY <...>`. Aggregation always happens in ClickHouse; Tantivy
|
||||||
|
only ever narrows which rows are eligible before that.
|
||||||
|
4. **`RawSQL` set** → executed as-is against ClickHouse, no Tantivy
|
||||||
|
involvement regardless of what the SQL contains. The escape hatch is
|
||||||
|
opaque by design — no attempt to detect free-text intent inside raw
|
||||||
|
SQL.
|
||||||
|
|
||||||
|
### Known scaling limitation
|
||||||
|
|
||||||
|
Steps 2/3's `record_id IN (...)` approach breaks down if a text search
|
||||||
|
matches a large number of rows — the `IN` clause is a literal, quoted
|
||||||
|
UUID list embedded in the query string. Phase 2's mitigation: cap the
|
||||||
|
Tantivy prefilter at **5,000** results. Tantivy's `TopDocs` already
|
||||||
|
returns most-relevant-first, so the cap keeps the *best* matches rather
|
||||||
|
than an arbitrary truncation, but it's a real limitation on result
|
||||||
|
completeness for very broad text searches combined with aggregation.
|
||||||
|
Documented in `/docs/query-language-reference.md`, not silently
|
||||||
|
swallowed.
|
||||||
|
|
||||||
|
This number isn't a first-principles estimate — running the Phase 2
|
||||||
|
benchmark against a real 1M-row dataset (`/docs/phase-2-runbook.md`)
|
||||||
|
caught the original 10,000 cap failing outright: 10,000 quoted UUIDs
|
||||||
|
(~39 bytes each) produces a ~390KB query string, which exceeds
|
||||||
|
ClickHouse's *default* `max_query_size` (262144 bytes / 256KiB) and
|
||||||
|
fails with a syntax error rather than degrading gracefully — a much
|
||||||
|
lower ceiling than "multi-million-entry" suggested before anyone had
|
||||||
|
actually tried it. 5,000 UUIDs (~195KB) stays safely under that default
|
||||||
|
with headroom. The real long-term fix (streaming `record_id` batches,
|
||||||
|
ClickHouse-side text indexing, a different join strategy, or simply
|
||||||
|
raising `max_query_size` server-side with matching memory sizing) is
|
||||||
|
explicitly future work, out of scope for Phase 2.
|
||||||
|
|
||||||
|
## Where this lives: `api/internal/querylang/`
|
||||||
|
|
||||||
|
Not a new top-level component. This subsystem always executes in-process
|
||||||
|
within `/api` — it doesn't run standalone, doesn't get its own Docker
|
||||||
|
image, and needs both connections `/api` already holds (the ClickHouse
|
||||||
|
driver, the search gRPC client). A new top-level directory would imply a
|
||||||
|
new deployable service, which this isn't.
|
||||||
|
|
||||||
|
```
|
||||||
|
api/internal/querylang/
|
||||||
|
lexer/ tokenizer
|
||||||
|
ast/ parsed pipe-syntax tree
|
||||||
|
parser/ tokens -> ast (recursive descent)
|
||||||
|
ir/ Plan and supporting types
|
||||||
|
planner/ ast -> Plan (field-mapping rule, SQL-passthrough detection)
|
||||||
|
executor/ Plan -> results (the four-case routing above)
|
||||||
|
```
|
||||||
|
|
||||||
|
Mirrors the existing `internal/queryapi`, `internal/searchclient`
|
||||||
|
convention already in `/api`. Each layer is independently testable per
|
||||||
|
task 3's requirement: "pipe syntax X compiles to IR Y" tests live in
|
||||||
|
`parser`/`planner` against fixture ASTs/Plans, no backend needed; "IR Y
|
||||||
|
executes correctly" tests live in `executor` against fakes for both the
|
||||||
|
ClickHouse and search-client interfaces, same pattern already used
|
||||||
|
throughout `/ingest` and `/api`.
|
||||||
|
|
||||||
|
## `/query` endpoint: auto-detect, with an explicit override
|
||||||
|
|
||||||
|
Detection: a request body's query starting with `SELECT`
|
||||||
|
(case-insensitive, same rule `validateSelectOnly` already applies) is
|
||||||
|
SQL; otherwise pipe syntax. Covers the overwhelming common case with no
|
||||||
|
extra field required. An optional `"language": "sql" | "spl"` field in
|
||||||
|
the request body overrides detection, for the rare case a pipe query
|
||||||
|
legitimately starts with the literal word "select" as a bare search
|
||||||
|
term. Auto-detect-with-override matches the shape of other
|
||||||
|
inference-with-explicit-override choices already made in this stack
|
||||||
|
(e.g. severity hints winning over parsed values when present) — good
|
||||||
|
default ergonomics, no silent ambiguity once a caller cares enough to be
|
||||||
|
explicit.
|
||||||
@@ -0,0 +1,322 @@
|
|||||||
|
# Query language reference
|
||||||
|
|
||||||
|
Sentry has one query language for everything: filtering, free-text
|
||||||
|
search, and aggregation, in a single query, against a single endpoint
|
||||||
|
(`POST /query`), from a single query bar in the web UI or `sentryctl
|
||||||
|
query` on the command line. You don't pick a "search mode" or a
|
||||||
|
"reporting mode" first — you write one query, and Sentry figures out
|
||||||
|
which parts need ClickHouse, which parts need the full-text index, and
|
||||||
|
combines them.
|
||||||
|
|
||||||
|
If you already know Splunk's SPL, most of this will feel immediately
|
||||||
|
familiar: a base search, piped through a sequence of processing stages.
|
||||||
|
Sentry's language is a deliberately smaller subset — the operators
|
||||||
|
people actually use day to day, not SPL's full surface area — plus raw
|
||||||
|
SQL as an escape hatch for anything the pipe syntax doesn't (yet) cover.
|
||||||
|
|
||||||
|
## The shape of a query
|
||||||
|
|
||||||
|
```
|
||||||
|
<base search> | <stage> | <stage> | ...
|
||||||
|
```
|
||||||
|
|
||||||
|
Everything before the first `|` is the base search — a filter and/or a
|
||||||
|
free-text search. Everything after each `|` is a processing stage that
|
||||||
|
narrows, reshapes, or summarizes what came before it.
|
||||||
|
|
||||||
|
```
|
||||||
|
service=api | where status>=500 | stats count by host | sort -count
|
||||||
|
```
|
||||||
|
|
||||||
|
Read left to right: start with everything logged by the `api` service,
|
||||||
|
keep only the entries with `status >= 500`, count how many there are per
|
||||||
|
`host`, and show the busiest hosts first.
|
||||||
|
|
||||||
|
## Filtering
|
||||||
|
|
||||||
|
```
|
||||||
|
field=value
|
||||||
|
field!=value
|
||||||
|
field>value
|
||||||
|
field>=value
|
||||||
|
field<value
|
||||||
|
field<=value
|
||||||
|
```
|
||||||
|
|
||||||
|
```
|
||||||
|
service=api
|
||||||
|
status>=500
|
||||||
|
host!=host-03
|
||||||
|
```
|
||||||
|
|
||||||
|
Multiple filters combine with `and` (the default when you don't write a
|
||||||
|
conjunction at all — see "Combining terms" below):
|
||||||
|
|
||||||
|
```
|
||||||
|
service=api status>=500
|
||||||
|
service=api and status>=500 (equivalent)
|
||||||
|
```
|
||||||
|
|
||||||
|
## Free-text search
|
||||||
|
|
||||||
|
Three ways to search the `message` field's text:
|
||||||
|
|
||||||
|
```
|
||||||
|
timeout a single bare word
|
||||||
|
"connection refused" a quoted phrase
|
||||||
|
message:"connection refused" the same thing, explicit
|
||||||
|
```
|
||||||
|
|
||||||
|
Free-text search is powered by Sentry's full-text index (Tantivy), which
|
||||||
|
supports phrase matching and wildcards:
|
||||||
|
|
||||||
|
```
|
||||||
|
message:"exact phrase"
|
||||||
|
message:"time*"
|
||||||
|
```
|
||||||
|
|
||||||
|
Bare words and quoted phrases can be mixed freely with structured filters
|
||||||
|
in the same query — that's the whole point of having one language:
|
||||||
|
|
||||||
|
```
|
||||||
|
service=api "connection refused"
|
||||||
|
message:"connection refused" | stats count by host
|
||||||
|
```
|
||||||
|
|
||||||
|
## Combining terms: `and` / `or`
|
||||||
|
|
||||||
|
Adjacent terms with nothing between them are implicitly `and`ed, matching
|
||||||
|
what most people expect from a search bar:
|
||||||
|
|
||||||
|
```
|
||||||
|
error timeout same as: error and timeout
|
||||||
|
```
|
||||||
|
|
||||||
|
`or` works between free-text terms, and Sentry's full-text index handles
|
||||||
|
it natively:
|
||||||
|
|
||||||
|
```
|
||||||
|
error or timeout
|
||||||
|
```
|
||||||
|
|
||||||
|
**Current limitation:** `or` is not supported between structured filters
|
||||||
|
(`service=api or service=web` returns a clear error rather than silently
|
||||||
|
being treated as `and`). If you need this, use two separate queries for
|
||||||
|
now, or the raw SQL escape hatch. This is a known gap, not an oversight —
|
||||||
|
full boolean-tree support for structured filters is on the list for a
|
||||||
|
future release once there's real usage data on how much it's needed.
|
||||||
|
|
||||||
|
## Time ranges
|
||||||
|
|
||||||
|
```
|
||||||
|
earliest=-1h relative: last hour
|
||||||
|
earliest=-15m latest=-5m relative window
|
||||||
|
earliest="2026-08-14T00:00:00Z" absolute (RFC 3339)
|
||||||
|
```
|
||||||
|
|
||||||
|
Relative offsets: a number followed by `s` (seconds), `m` (minutes), `h`
|
||||||
|
(hours), `d` (days), or `w` (weeks), always relative to when the query
|
||||||
|
runs.
|
||||||
|
|
||||||
|
## Pipe stages
|
||||||
|
|
||||||
|
### `where` — additional filtering after the base search
|
||||||
|
|
||||||
|
Same syntax as the base search's filter terms:
|
||||||
|
|
||||||
|
```
|
||||||
|
service=api | where status>=500
|
||||||
|
```
|
||||||
|
|
||||||
|
### `stats` — aggregation
|
||||||
|
|
||||||
|
```
|
||||||
|
stats count by host
|
||||||
|
stats count(), avg(latency_ms) as avg_latency by host, service
|
||||||
|
```
|
||||||
|
|
||||||
|
Supported functions: `count`, `sum`, `avg`, `min`, `max`. `count` doesn't
|
||||||
|
need a field (`count`, `count()`, and `count(*)` are all equivalent);
|
||||||
|
every other function requires one (`sum(latency_ms)`). Give a result an
|
||||||
|
explicit name with `as`, or accept the default (the function name, or
|
||||||
|
`count` for a bare count).
|
||||||
|
|
||||||
|
```
|
||||||
|
stats sum(bytes_sent) as total_bytes by host
|
||||||
|
```
|
||||||
|
|
||||||
|
### `sort` — ordering
|
||||||
|
|
||||||
|
```
|
||||||
|
sort -count descending by count (the default direction)
|
||||||
|
sort +host ascending by host
|
||||||
|
sort -severity, +host descending by severity, then ascending by host
|
||||||
|
```
|
||||||
|
|
||||||
|
`-` and `+` mean the same thing they do in most search tools: `-` for
|
||||||
|
descending, `+` for ascending. No sign at all also means descending.
|
||||||
|
You can sort by any field from the base data, or by a `stats` result's
|
||||||
|
column name/alias.
|
||||||
|
|
||||||
|
### `fields` — choosing which columns come back
|
||||||
|
|
||||||
|
```
|
||||||
|
fields host, message, severity
|
||||||
|
```
|
||||||
|
|
||||||
|
Without `fields`, you get every column.
|
||||||
|
|
||||||
|
### `head` / `tail` — limiting results
|
||||||
|
|
||||||
|
```
|
||||||
|
head first 100 (the default) results
|
||||||
|
head 20 first 20
|
||||||
|
tail 50 last 50, chronologically
|
||||||
|
```
|
||||||
|
|
||||||
|
## Field mapping: what's a "real" column vs. an attribute
|
||||||
|
|
||||||
|
Sentry's structured columns are `timestamp`, `host`, `service`,
|
||||||
|
`severity`, `message`, and `record_id`. Anything else you reference by
|
||||||
|
name — `status`, `latency_ms`, `winevt.event_id`, whatever your logs
|
||||||
|
happen to carry — is looked up in the per-record attributes, which are
|
||||||
|
always stored as text.
|
||||||
|
|
||||||
|
This matters for comparisons: `status>=500` only makes sense as a number,
|
||||||
|
so Sentry casts the attribute's text value to a number for you
|
||||||
|
automatically when the value you're comparing against looks numeric.
|
||||||
|
`status="unknown"` compares as text instead, since `"unknown"` isn't a
|
||||||
|
number. You don't need to do anything differently — this happens based
|
||||||
|
on what you write on the right-hand side of the comparison — but it's
|
||||||
|
worth knowing that:
|
||||||
|
|
||||||
|
- A field that's missing entirely, or whose value isn't actually numeric,
|
||||||
|
reads as `0` in a numeric comparison or aggregation (`toFloat64OrZero`
|
||||||
|
semantics) rather than erroring. A typo'd field name will "succeed"
|
||||||
|
with everything showing as `0` — if a `stats sum(...)` looks
|
||||||
|
suspiciously empty, double-check the field name.
|
||||||
|
- `stats min()`/`max()` on an attribute field always compares
|
||||||
|
numerically, not alphabetically, in this release.
|
||||||
|
- Querying an attribute is always a little more work for ClickHouse than
|
||||||
|
querying a real column — if a field turns out to be central to how you
|
||||||
|
query your logs, that's a signal it might be worth promoting to a real
|
||||||
|
column in a future schema change (not something you can do yourself
|
||||||
|
today).
|
||||||
|
|
||||||
|
## Raw SQL
|
||||||
|
|
||||||
|
Anything starting with `SELECT` is treated as raw ClickHouse SQL and run
|
||||||
|
directly, no pipe-syntax parsing involved:
|
||||||
|
|
||||||
|
```
|
||||||
|
SELECT host, count(*) FROM logs WHERE service = 'api' GROUP BY host
|
||||||
|
```
|
||||||
|
|
||||||
|
SELECT-only, single statement — Sentry allowlists this at the API level.
|
||||||
|
Use this for anything the pipe syntax doesn't cover yet: window
|
||||||
|
functions, `WITH` clauses, ClickHouse-specific functions, joins across
|
||||||
|
other tables you've added, and so on. There's no performance penalty for
|
||||||
|
using SQL over the pipe syntax or vice versa — both compile to the same
|
||||||
|
execution plan internally.
|
||||||
|
|
||||||
|
## Which syntax am I using?
|
||||||
|
|
||||||
|
Sentry detects automatically: a query starting with `SELECT` runs as
|
||||||
|
SQL, anything else runs as the pipe syntax. This covers the overwhelming
|
||||||
|
majority of real queries with no extra step. If you're writing a pipe
|
||||||
|
query that happens to start with the literal word "select" as a search
|
||||||
|
term, set the language explicitly instead of relying on detection:
|
||||||
|
|
||||||
|
```json
|
||||||
|
{"query": "select", "language": "spl"}
|
||||||
|
```
|
||||||
|
|
||||||
|
`language` accepts `"sql"`, `"spl"`, or can be omitted entirely (the
|
||||||
|
default, auto-detect). The web UI's query bar shows which one it
|
||||||
|
detected next to the query box, with a dropdown to override it.
|
||||||
|
|
||||||
|
## Combining free-text search with aggregation
|
||||||
|
|
||||||
|
This is the case that makes Sentry's query language more than "SQL with
|
||||||
|
extra steps" — free text and aggregation, together, in one query:
|
||||||
|
|
||||||
|
```
|
||||||
|
message:"connection refused" | stats count by host
|
||||||
|
```
|
||||||
|
|
||||||
|
Under the hood: the full-text index resolves which records match the
|
||||||
|
text search first, then ClickHouse does the counting and grouping over
|
||||||
|
just those records. You don't need to know this to use it — it's
|
||||||
|
mentioned here because of the one limitation it implies:
|
||||||
|
|
||||||
|
**A single free-text search is capped at 5,000 matching records** when
|
||||||
|
it's combined with a `stats`/filter stage that needs to know exactly
|
||||||
|
which records matched (the most-relevant 5,000, not an arbitrary
|
||||||
|
truncation). A text search alone, with no aggregation, isn't affected by
|
||||||
|
this cap. If your combined query's text search is broad enough to match
|
||||||
|
more than 5,000 records, narrow it — a more specific phrase, an added
|
||||||
|
`where` filter, or a tighter time range — the same way you'd narrow an
|
||||||
|
overly broad search in any tool.
|
||||||
|
|
||||||
|
## Response shape
|
||||||
|
|
||||||
|
Every query, regardless of syntax or which backend(s) it touched,
|
||||||
|
returns the same shape:
|
||||||
|
|
||||||
|
```json
|
||||||
|
{"columns": ["host", "count"], "rows": [["api-01", 42], ["api-02", 17]]}
|
||||||
|
```
|
||||||
|
|
||||||
|
or, on error:
|
||||||
|
|
||||||
|
```json
|
||||||
|
{"error": "a description of what went wrong"}
|
||||||
|
```
|
||||||
|
|
||||||
|
## Quick reference
|
||||||
|
|
||||||
|
| Syntax | Meaning |
|
||||||
|
|---|---|
|
||||||
|
| `field=value` | equals |
|
||||||
|
| `field!=value` | not equals |
|
||||||
|
| `field>value` / `>=` / `<` / `<=` | comparison |
|
||||||
|
| `"phrase"` / bare word | free-text search on `message` |
|
||||||
|
| `message:"phrase"` | explicit free-text search |
|
||||||
|
| `earliest=-1h` / `latest=...` | time range |
|
||||||
|
| `\| where ...` | additional filter |
|
||||||
|
| `\| stats count by field` | aggregate |
|
||||||
|
| `\| sort -field` / `+field` | sort desc / asc |
|
||||||
|
| `\| fields a, b` | choose columns |
|
||||||
|
| `\| head N` / `\| tail N` | limit results |
|
||||||
|
| `SELECT ...` | raw SQL |
|
||||||
|
|
||||||
|
## Examples
|
||||||
|
|
||||||
|
```
|
||||||
|
service=api | where status>=500 | stats count by host | sort -count
|
||||||
|
```
|
||||||
|
Which hosts are producing the most 5xx errors from the `api` service?
|
||||||
|
|
||||||
|
```
|
||||||
|
message:"connection refused" | stats count by host
|
||||||
|
```
|
||||||
|
Where are connection-refused errors coming from?
|
||||||
|
|
||||||
|
```
|
||||||
|
earliest=-24h severity=ERROR | stats count by service | sort -count
|
||||||
|
```
|
||||||
|
Error volume by service over the last day.
|
||||||
|
|
||||||
|
```
|
||||||
|
winevt.event_id=4625 | fields host, message | head 20
|
||||||
|
```
|
||||||
|
Recent failed Windows logon attempts (a `winevt.*` attribute from the
|
||||||
|
Windows Event Log source — see `/docs/phase-1-runbook.md`).
|
||||||
|
|
||||||
|
```
|
||||||
|
SELECT host, avg(toFloat64OrZero(attributes['latency_ms'])) AS avg_latency
|
||||||
|
FROM logs WHERE service = 'api' GROUP BY host ORDER BY avg_latency DESC
|
||||||
|
```
|
||||||
|
The same kind of query the pipe syntax's `stats avg(latency_ms) by host`
|
||||||
|
would produce, written by hand — useful as a starting point if you need
|
||||||
|
something the pipe syntax doesn't support yet.
|
||||||
@@ -0,0 +1,18 @@
|
|||||||
|
module github.com/sentry/sentry/hack/benchmark-fixture
|
||||||
|
|
||||||
|
go 1.25.0
|
||||||
|
|
||||||
|
replace github.com/sentry/sentry/proto => ../../proto
|
||||||
|
|
||||||
|
require (
|
||||||
|
github.com/sentry/sentry/proto v0.0.0-00010101000000-000000000000
|
||||||
|
google.golang.org/grpc v1.83.0
|
||||||
|
)
|
||||||
|
|
||||||
|
require (
|
||||||
|
golang.org/x/net v0.55.0 // indirect
|
||||||
|
golang.org/x/sys v0.45.0 // indirect
|
||||||
|
golang.org/x/text v0.37.0 // indirect
|
||||||
|
google.golang.org/genproto/googleapis/rpc v0.0.0-20260526163538-3dc84a4a5aaa // indirect
|
||||||
|
google.golang.org/protobuf v1.36.12 // indirect
|
||||||
|
)
|
||||||
@@ -0,0 +1,38 @@
|
|||||||
|
github.com/cespare/xxhash/v2 v2.3.0 h1:UL815xU9SqsFlibzuggzjXhog7bL6oX9BbNZnL2UFvs=
|
||||||
|
github.com/cespare/xxhash/v2 v2.3.0/go.mod h1:VGX0DQ3Q6kWi7AoAeZDth3/j3BFtOZR5XLFGgcrjCOs=
|
||||||
|
github.com/go-logr/logr v1.4.3 h1:CjnDlHq8ikf6E492q6eKboGOC0T8CDaOvkHCIg8idEI=
|
||||||
|
github.com/go-logr/logr v1.4.3/go.mod h1:9T104GzyrTigFIr8wt5mBrctHMim0Nb2HLGrmQ40KvY=
|
||||||
|
github.com/go-logr/stdr v1.2.2 h1:hSWxHoqTgW2S2qGc0LTAI563KZ5YKYRhT3MFKZMbjag=
|
||||||
|
github.com/go-logr/stdr v1.2.2/go.mod h1:mMo/vtBO5dYbehREoey6XUKy/eSumjCCveDpRre4VKE=
|
||||||
|
github.com/golang/protobuf v1.5.4 h1:i7eJL8qZTpSEXOPTxNKhASYpMn+8e5Q6AdndVa1dWek=
|
||||||
|
github.com/golang/protobuf v1.5.4/go.mod h1:lnTiLA8Wa4RWRcIUkrtSVa5nRhsEGBg48fD6rSs7xps=
|
||||||
|
github.com/google/go-cmp v0.7.0 h1:wk8382ETsv4JYUZwIsn6YpYiWiBsYLSJiTsyBybVuN8=
|
||||||
|
github.com/google/go-cmp v0.7.0/go.mod h1:pXiqmnSA92OHEEa9HXL2W4E7lf9JzCmGVUdgjX3N/iU=
|
||||||
|
github.com/google/uuid v1.6.0 h1:NIvaJDMOsjHA8n1jAhLSgzrAzy1Hgr+hNrb57e+94F0=
|
||||||
|
github.com/google/uuid v1.6.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo=
|
||||||
|
go.opentelemetry.io/auto/sdk v1.2.1 h1:jXsnJ4Lmnqd11kwkBV2LgLoFMZKizbCi5fNZ/ipaZ64=
|
||||||
|
go.opentelemetry.io/auto/sdk v1.2.1/go.mod h1:KRTj+aOaElaLi+wW1kO/DZRXwkF4C5xPbEe3ZiIhN7Y=
|
||||||
|
go.opentelemetry.io/otel v1.44.0 h1:JjwHmHpA4iZ3wBxluu2fbbE7j4kqlE8jXyAyPXH7HqU=
|
||||||
|
go.opentelemetry.io/otel v1.44.0/go.mod h1:BMgjTHL9WPRlRjL2oZCBTL4whCGtXch2H4BhOPIAyYc=
|
||||||
|
go.opentelemetry.io/otel/metric v1.44.0 h1:1w0gILTcHdr3YI+ixLyjemwrVnsMURbTZFrSYCdDdmc=
|
||||||
|
go.opentelemetry.io/otel/metric v1.44.0/go.mod h1:8O7hanEPBNgEMmybD3s2VBKcgWOCsA6tzHBPODAiquo=
|
||||||
|
go.opentelemetry.io/otel/sdk v1.44.0 h1:nHYwb9lK+fJPU/dnT6s7W7Z8itMWyqrnVfbheVYrZ58=
|
||||||
|
go.opentelemetry.io/otel/sdk v1.44.0/go.mod h1:Osuydd3Se74nqjAKxid74N5eC+jfEqfTegHRnq58oK0=
|
||||||
|
go.opentelemetry.io/otel/sdk/metric v1.44.0 h1:3LlKgI+VjbVsjNRFZJZAJ30WjXC5VkNRks6si09iEfI=
|
||||||
|
go.opentelemetry.io/otel/sdk/metric v1.44.0/go.mod h1:5B5pMARnXxKhltooO4xUuCBorl65a4EpnTalObqOigA=
|
||||||
|
go.opentelemetry.io/otel/trace v1.44.0 h1:jxF5CsGYCe74MCRx2X4g7WsY/VBKRqqpNvXlX/6gtIk=
|
||||||
|
go.opentelemetry.io/otel/trace v1.44.0/go.mod h1:oLl1jrMQAVo6v3GAggN+1VH9VIz9iUSvW53sW1Q8PIE=
|
||||||
|
golang.org/x/net v0.55.0 h1:bcvxaJn3e1U6InsFWt1JUq1aSjnRxLzT2rtD2KfkDF8=
|
||||||
|
golang.org/x/net v0.55.0/go.mod h1:L5U2KuzuOe1lY7Z+aWVIKK6qEeJXnXV9yzGA+WCHJww=
|
||||||
|
golang.org/x/sys v0.45.0 h1:dO4czNzziLiiXplLQgBCEpCvXQ3dnkn0SdaZSYdQ+FY=
|
||||||
|
golang.org/x/sys v0.45.0/go.mod h1:4GL1E5IUh+htKOUEOaiffhrAeqysfVGipDYzABqnCmw=
|
||||||
|
golang.org/x/text v0.37.0 h1:Cqjiwd9eSg8e0QAkyCaQTNHFIIzWtidPahFWR83rTrc=
|
||||||
|
golang.org/x/text v0.37.0/go.mod h1:a5sjxXGs9hsn/AJVwuElvCAo9v8QYLzvavO5z2PiM38=
|
||||||
|
gonum.org/v1/gonum v0.17.0 h1:VbpOemQlsSMrYmn7T2OUvQ4dqxQXU+ouZFQsZOx50z4=
|
||||||
|
gonum.org/v1/gonum v0.17.0/go.mod h1:El3tOrEuMpv2UdMrbNlKEh9vd86bmQ6vqIcDwxEOc1E=
|
||||||
|
google.golang.org/genproto/googleapis/rpc v0.0.0-20260526163538-3dc84a4a5aaa h1:mZHHdPZl0dbGHCflZgAq/Q468DWVFcU2whhB2KAo8fk=
|
||||||
|
google.golang.org/genproto/googleapis/rpc v0.0.0-20260526163538-3dc84a4a5aaa/go.mod h1:4Hqkh8ycfw05ld/3BWL7rJOSfebL2Q+DVDeRgYgxUU8=
|
||||||
|
google.golang.org/grpc v1.83.0 h1:JeNZEKJFbQxArAMl+hiytHauacDNqJUllNfmIMmpqnQ=
|
||||||
|
google.golang.org/grpc v1.83.0/go.mod h1:kDyl6SKsiHKt0uylY5gtn5cEjkrIOhQOGDgIc4JGwzQ=
|
||||||
|
google.golang.org/protobuf v1.36.12 h1:pJOKDDOyeXErUroCihFAd5LQuwXBSpVnKGrj5o/fwxc=
|
||||||
|
google.golang.org/protobuf v1.36.12/go.mod h1:HTf+CrKn2C3g5S8VImy6tdcUvCska2kB7j23XfzDpco=
|
||||||
@@ -0,0 +1,172 @@
|
|||||||
|
// Command benchmark-fixture pushes a large, realistically varied
|
||||||
|
// synthetic dataset directly to ingest's gRPC endpoint, batched, so
|
||||||
|
// Phase 2's "modest dataset" query-latency benchmark
|
||||||
|
// (/docs/phase-2-runbook.md) has real data to measure against instead of
|
||||||
|
// an asserted number. Distinct from /hack/windows-fixture: that one
|
||||||
|
// sends a handful of realistic Windows events to test pipeline
|
||||||
|
// *correctness*; this one sends a lot of Linux-shaped events to test
|
||||||
|
// query *performance* at volume.
|
||||||
|
package main
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"crypto/tls"
|
||||||
|
"crypto/x509"
|
||||||
|
"flag"
|
||||||
|
"fmt"
|
||||||
|
"math/rand"
|
||||||
|
"os"
|
||||||
|
"sync"
|
||||||
|
"sync/atomic"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"google.golang.org/grpc"
|
||||||
|
"google.golang.org/grpc/credentials"
|
||||||
|
|
||||||
|
logsv1 "github.com/sentry/sentry/proto/sentry/logs/v1"
|
||||||
|
)
|
||||||
|
|
||||||
|
var (
|
||||||
|
services = []string{"api", "web", "worker", "db", "auth"}
|
||||||
|
hosts = []string{"host-01", "host-02", "host-03", "host-04", "host-05", "host-06", "host-07", "host-08"}
|
||||||
|
severites = []logsv1.Severity{
|
||||||
|
logsv1.Severity_SEVERITY_DEBUG,
|
||||||
|
logsv1.Severity_SEVERITY_INFO,
|
||||||
|
logsv1.Severity_SEVERITY_INFO,
|
||||||
|
logsv1.Severity_SEVERITY_INFO,
|
||||||
|
logsv1.Severity_SEVERITY_WARN,
|
||||||
|
logsv1.Severity_SEVERITY_ERROR,
|
||||||
|
}
|
||||||
|
// A mix of messages, some containing terms worth full-text
|
||||||
|
// searching for (connection refused, timeout) so the benchmark's
|
||||||
|
// text-search-plus-aggregation case has real matches to find, not
|
||||||
|
// just structured rows.
|
||||||
|
messages = []string{
|
||||||
|
"request completed successfully",
|
||||||
|
"connection refused by upstream",
|
||||||
|
"request timeout after 30s",
|
||||||
|
"cache miss, falling back to database",
|
||||||
|
"connection refused: too many open connections",
|
||||||
|
"user authentication succeeded",
|
||||||
|
"slow query detected: timeout approaching",
|
||||||
|
"health check passed",
|
||||||
|
"retrying after connection refused error",
|
||||||
|
"scheduled job completed",
|
||||||
|
}
|
||||||
|
)
|
||||||
|
|
||||||
|
func main() {
|
||||||
|
addr := flag.String("addr", "localhost:4317", "ingest gRPC address")
|
||||||
|
caFile := flag.String("ca", "../dev-certs/out/ca.pem", "CA cert path")
|
||||||
|
certFile := flag.String("cert", "../dev-certs/out/client.pem", "client cert path")
|
||||||
|
keyFile := flag.String("key", "../dev-certs/out/client-key.pem", "client key path")
|
||||||
|
count := flag.Int("count", 1_000_000, "total number of records to generate")
|
||||||
|
batchSize := flag.Int("batch-size", 1000, "records per PushBatch call")
|
||||||
|
concurrency := flag.Int("concurrency", 16, "concurrent PushBatch calls in flight")
|
||||||
|
flag.Parse()
|
||||||
|
|
||||||
|
tlsConf, err := loadTLSConfig(*caFile, *certFile, *keyFile)
|
||||||
|
if err != nil {
|
||||||
|
fmt.Fprintln(os.Stderr, "loading TLS config:", err)
|
||||||
|
os.Exit(1)
|
||||||
|
}
|
||||||
|
|
||||||
|
// One shared connection: gRPC multiplexes concurrent RPCs over HTTP/2
|
||||||
|
// streams on a single connection, so concurrency here comes from
|
||||||
|
// concurrent PushBatch calls, not from opening more connections.
|
||||||
|
conn, err := grpc.NewClient(*addr, grpc.WithTransportCredentials(credentials.NewTLS(tlsConf)))
|
||||||
|
if err != nil {
|
||||||
|
fmt.Fprintln(os.Stderr, "dialing ingest:", err)
|
||||||
|
os.Exit(1)
|
||||||
|
}
|
||||||
|
defer conn.Close()
|
||||||
|
|
||||||
|
client := logsv1.NewLogIngestClient(conn)
|
||||||
|
|
||||||
|
numBatches := (*count + *batchSize - 1) / *batchSize
|
||||||
|
batchIndexes := make(chan int, numBatches)
|
||||||
|
for i := 0; i < numBatches; i++ {
|
||||||
|
batchIndexes <- i
|
||||||
|
}
|
||||||
|
close(batchIndexes)
|
||||||
|
|
||||||
|
var sent atomic.Int64
|
||||||
|
var wg sync.WaitGroup
|
||||||
|
start := time.Now()
|
||||||
|
|
||||||
|
for w := 0; w < *concurrency; w++ {
|
||||||
|
wg.Add(1)
|
||||||
|
go func() {
|
||||||
|
defer wg.Done()
|
||||||
|
for batchIdx := range batchIndexes {
|
||||||
|
offset := batchIdx * *batchSize
|
||||||
|
n := *batchSize
|
||||||
|
if remaining := *count - offset; remaining < n {
|
||||||
|
n = remaining
|
||||||
|
}
|
||||||
|
records := make([]*logsv1.LogRecord, n)
|
||||||
|
for i := range records {
|
||||||
|
records[i] = randomRecord()
|
||||||
|
}
|
||||||
|
|
||||||
|
ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
|
||||||
|
resp, err := client.PushBatch(ctx, &logsv1.PushBatchRequest{
|
||||||
|
BatchId: fmt.Sprintf("benchmark-%d", batchIdx),
|
||||||
|
Records: records,
|
||||||
|
})
|
||||||
|
cancel()
|
||||||
|
if err != nil {
|
||||||
|
fmt.Fprintf(os.Stderr, "PushBatch %d failed: %v\n", batchIdx, err)
|
||||||
|
os.Exit(1)
|
||||||
|
}
|
||||||
|
|
||||||
|
total := sent.Add(int64(resp.GetAccepted()))
|
||||||
|
if total%int64(*batchSize*50) < int64(*batchSize) {
|
||||||
|
elapsed := time.Since(start)
|
||||||
|
rate := float64(total) / elapsed.Seconds()
|
||||||
|
fmt.Printf("sent %d/%d (%.0f records/sec)\n", total, *count, rate)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}()
|
||||||
|
}
|
||||||
|
wg.Wait()
|
||||||
|
|
||||||
|
elapsed := time.Since(start)
|
||||||
|
total := sent.Load()
|
||||||
|
fmt.Printf("done: %d records in %s (%.0f records/sec)\n", total, elapsed, float64(total)/elapsed.Seconds())
|
||||||
|
}
|
||||||
|
|
||||||
|
func randomRecord() *logsv1.LogRecord {
|
||||||
|
return &logsv1.LogRecord{
|
||||||
|
TimestampUnixNano: time.Now().UnixNano(),
|
||||||
|
Host: hosts[rand.Intn(len(hosts))],
|
||||||
|
Service: services[rand.Intn(len(services))],
|
||||||
|
Severity: severites[rand.Intn(len(severites))],
|
||||||
|
Message: messages[rand.Intn(len(messages))],
|
||||||
|
Attributes: map[string]string{
|
||||||
|
"status": fmt.Sprintf("%d", []int{200, 200, 200, 301, 404, 500, 503}[rand.Intn(7)]),
|
||||||
|
"latency_ms": fmt.Sprintf("%d", rand.Intn(2000)),
|
||||||
|
},
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func loadTLSConfig(caFile, certFile, keyFile string) (*tls.Config, error) {
|
||||||
|
caPEM, err := os.ReadFile(caFile)
|
||||||
|
if err != nil {
|
||||||
|
return nil, fmt.Errorf("reading CA cert %s: %w", caFile, err)
|
||||||
|
}
|
||||||
|
caPool := x509.NewCertPool()
|
||||||
|
if !caPool.AppendCertsFromPEM(caPEM) {
|
||||||
|
return nil, fmt.Errorf("no valid certificates found in %s", caFile)
|
||||||
|
}
|
||||||
|
|
||||||
|
cert, err := tls.LoadX509KeyPair(certFile, keyFile)
|
||||||
|
if err != nil {
|
||||||
|
return nil, fmt.Errorf("loading client cert/key: %w", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
return &tls.Config{
|
||||||
|
RootCAs: caPool,
|
||||||
|
Certificates: []tls.Certificate{cert},
|
||||||
|
}, nil
|
||||||
|
}
|
||||||
@@ -1,6 +1,5 @@
|
|||||||
<script lang="ts">
|
<script lang="ts">
|
||||||
import favicon from '$lib/assets/favicon.svg';
|
import favicon from '$lib/assets/favicon.svg';
|
||||||
import { page } from '$app/state';
|
|
||||||
|
|
||||||
let { children } = $props();
|
let { children } = $props();
|
||||||
</script>
|
</script>
|
||||||
@@ -10,32 +9,4 @@
|
|||||||
<link rel="icon" href={favicon} />
|
<link rel="icon" href={favicon} />
|
||||||
</svelte:head>
|
</svelte:head>
|
||||||
|
|
||||||
<nav>
|
|
||||||
<a href="/" class:active={page.url.pathname === '/'}>SQL Query</a>
|
|
||||||
<a href="/search" class:active={page.url.pathname === '/search'}>Full-Text Search</a>
|
|
||||||
</nav>
|
|
||||||
|
|
||||||
{@render children()}
|
{@render children()}
|
||||||
|
|
||||||
<style>
|
|
||||||
nav {
|
|
||||||
font-family: system-ui, sans-serif;
|
|
||||||
max-width: 960px;
|
|
||||||
margin: 1rem auto 0;
|
|
||||||
padding: 0 1rem;
|
|
||||||
display: flex;
|
|
||||||
gap: 1rem;
|
|
||||||
border-bottom: 1px solid #ccc;
|
|
||||||
}
|
|
||||||
nav a {
|
|
||||||
padding: 0.5rem 0;
|
|
||||||
text-decoration: none;
|
|
||||||
color: #555;
|
|
||||||
border-bottom: 2px solid transparent;
|
|
||||||
}
|
|
||||||
nav a.active {
|
|
||||||
color: #000;
|
|
||||||
border-bottom-color: #000;
|
|
||||||
font-weight: 600;
|
|
||||||
}
|
|
||||||
</style>
|
|
||||||
|
|||||||
+168
-14
@@ -1,19 +1,61 @@
|
|||||||
<script lang="ts">
|
<script lang="ts">
|
||||||
// Phase 0: functional only, no styling polish, no auth. One page: a raw
|
// Phase 2: single unified query page. Replaces Phase 0/1's two
|
||||||
// SQL box against POST /query on the api service, rendered as a table.
|
// separate pages (raw-SQL-only /query, free-text-only /search) --
|
||||||
// This is a placeholder for the real query UI that lands once /api grows
|
// see /docs/query-language-design.md and /docs/query-language-reference.md.
|
||||||
// a real SPL-like query layer in Phase 2.
|
|
||||||
|
|
||||||
import ResultsTable from '$lib/ResultsTable.svelte';
|
import ResultsTable from '$lib/ResultsTable.svelte';
|
||||||
|
|
||||||
const apiBase = import.meta.env.VITE_API_BASE_URL ?? 'http://localhost:8080';
|
const apiBase = import.meta.env.VITE_API_BASE_URL ?? 'http://localhost:8080';
|
||||||
|
|
||||||
let sql = $state('SELECT * FROM logs ORDER BY timestamp DESC LIMIT 100');
|
type Language = '' | 'sql' | 'spl';
|
||||||
|
type HistoryEntry = { query: string; language: Language; at: number };
|
||||||
|
|
||||||
|
const HISTORY_KEY = 'sentry.queryHistory';
|
||||||
|
const HISTORY_LIMIT = 20;
|
||||||
|
|
||||||
|
let query = $state('earliest=-1h | sort -timestamp | head 100');
|
||||||
|
let language = $state<Language>('');
|
||||||
let columns = $state<string[]>([]);
|
let columns = $state<string[]>([]);
|
||||||
let rows = $state<unknown[][]>([]);
|
let rows = $state<unknown[][]>([]);
|
||||||
let error = $state('');
|
let error = $state('');
|
||||||
let loading = $state(false);
|
let loading = $state(false);
|
||||||
let hasRun = $state(false);
|
let hasRun = $state(false);
|
||||||
|
let history = $state<HistoryEntry[]>(loadHistory());
|
||||||
|
|
||||||
|
// Client-side mirror of the backend's auto-detect heuristic
|
||||||
|
// (api/internal/querylang/planner.looksLikeSQL) -- purely a UI hint,
|
||||||
|
// the server does its own detection independently and is the
|
||||||
|
// authority on what actually runs.
|
||||||
|
function detectedLanguage(q: string): 'sql' | 'spl' {
|
||||||
|
return /^\s*select\b/i.test(q) ? 'sql' : 'spl';
|
||||||
|
}
|
||||||
|
let detected = $derived(detectedLanguage(query));
|
||||||
|
let effectiveLanguage = $derived(language === '' ? detected : language);
|
||||||
|
|
||||||
|
function loadHistory(): HistoryEntry[] {
|
||||||
|
if (typeof sessionStorage === 'undefined') return [];
|
||||||
|
try {
|
||||||
|
const raw = sessionStorage.getItem(HISTORY_KEY);
|
||||||
|
return raw ? JSON.parse(raw) : [];
|
||||||
|
} catch {
|
||||||
|
return [];
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
function saveHistory(entry: HistoryEntry) {
|
||||||
|
history = [entry, ...history.filter((h) => h.query !== entry.query)].slice(0, HISTORY_LIMIT);
|
||||||
|
try {
|
||||||
|
sessionStorage.setItem(HISTORY_KEY, JSON.stringify(history));
|
||||||
|
} catch {
|
||||||
|
// session storage unavailable/full -- history is a convenience,
|
||||||
|
// not worth failing the query over
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
function useHistoryEntry(entry: HistoryEntry) {
|
||||||
|
query = entry.query;
|
||||||
|
language = entry.language;
|
||||||
|
}
|
||||||
|
|
||||||
async function runQuery() {
|
async function runQuery() {
|
||||||
loading = true;
|
loading = true;
|
||||||
@@ -22,7 +64,7 @@
|
|||||||
const res = await fetch(`${apiBase}/query`, {
|
const res = await fetch(`${apiBase}/query`, {
|
||||||
method: 'POST',
|
method: 'POST',
|
||||||
headers: { 'Content-Type': 'application/json' },
|
headers: { 'Content-Type': 'application/json' },
|
||||||
body: JSON.stringify({ sql })
|
body: JSON.stringify({ query, language })
|
||||||
});
|
});
|
||||||
const body = await res.json();
|
const body = await res.json();
|
||||||
if (!res.ok) {
|
if (!res.ok) {
|
||||||
@@ -33,6 +75,7 @@
|
|||||||
}
|
}
|
||||||
columns = body.columns ?? [];
|
columns = body.columns ?? [];
|
||||||
rows = body.rows ?? [];
|
rows = body.rows ?? [];
|
||||||
|
saveHistory({ query, language, at: Date.now() });
|
||||||
} catch (e) {
|
} catch (e) {
|
||||||
error = e instanceof Error ? e.message : String(e);
|
error = e instanceof Error ? e.message : String(e);
|
||||||
columns = [];
|
columns = [];
|
||||||
@@ -42,21 +85,51 @@
|
|||||||
hasRun = true;
|
hasRun = true;
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
function onKeydown(e: KeyboardEvent) {
|
||||||
|
// Cmd/Ctrl+Enter runs the query -- textarea's own Enter key needs
|
||||||
|
// to stay newline-for-pipe-stage-formatting, so this isn't a bare
|
||||||
|
// Enter binding.
|
||||||
|
if ((e.metaKey || e.ctrlKey) && e.key === 'Enter') {
|
||||||
|
e.preventDefault();
|
||||||
|
runQuery();
|
||||||
|
}
|
||||||
|
}
|
||||||
</script>
|
</script>
|
||||||
|
|
||||||
<main>
|
<main>
|
||||||
<h1>Sentry — Log Query</h1>
|
<h1>Sentry — Query</h1>
|
||||||
<p>
|
<p>
|
||||||
Raw SQL only, SELECT statements against the <code>logs</code> table. No auth, no query
|
One query bar for both filter/stats queries and free-text search — see
|
||||||
builder yet — see <code>/api</code> for what's actually allowed. Looking for free-text
|
<code>/docs/query-language-reference.md</code> in the repo for the full syntax, or the cheat
|
||||||
search instead? See the <a href="/search">Full-Text Search</a> page.
|
sheet below.
|
||||||
</p>
|
</p>
|
||||||
|
|
||||||
<textarea bind:value={sql} rows="4" cols="100" spellcheck="false"></textarea>
|
<textarea
|
||||||
<div>
|
bind:value={query}
|
||||||
<button onclick={runQuery} disabled={loading}>
|
onkeydown={onKeydown}
|
||||||
|
rows="4"
|
||||||
|
cols="100"
|
||||||
|
spellcheck="false"
|
||||||
|
placeholder={'service=api | where status>=500 | stats count by host | sort -count'}
|
||||||
|
></textarea>
|
||||||
|
|
||||||
|
<div class="controls">
|
||||||
|
<label>
|
||||||
|
Language:
|
||||||
|
<select bind:value={language}>
|
||||||
|
<option value="">Auto ({detected})</option>
|
||||||
|
<option value="spl">Pipe syntax</option>
|
||||||
|
<option value="sql">SQL</option>
|
||||||
|
</select>
|
||||||
|
</label>
|
||||||
|
<span class="detected-badge" class:sql={effectiveLanguage === 'sql'}>
|
||||||
|
{effectiveLanguage === 'sql' ? 'SQL' : 'pipe syntax'}
|
||||||
|
</span>
|
||||||
|
<button onclick={runQuery} disabled={loading || query.trim() === ''}>
|
||||||
{loading ? 'Running…' : 'Run query'}
|
{loading ? 'Running…' : 'Run query'}
|
||||||
</button>
|
</button>
|
||||||
|
<span class="hint">⌘/Ctrl+Enter to run</span>
|
||||||
</div>
|
</div>
|
||||||
|
|
||||||
{#if error}
|
{#if error}
|
||||||
@@ -64,6 +137,39 @@
|
|||||||
{/if}
|
{/if}
|
||||||
|
|
||||||
<ResultsTable {columns} {rows} {hasRun} />
|
<ResultsTable {columns} {rows} {hasRun} />
|
||||||
|
|
||||||
|
{#if history.length > 0}
|
||||||
|
<details class="history">
|
||||||
|
<summary>Query history ({history.length})</summary>
|
||||||
|
<ul>
|
||||||
|
{#each history as entry (entry.at)}
|
||||||
|
<li>
|
||||||
|
<button class="history-item" onclick={() => useHistoryEntry(entry)}>
|
||||||
|
<code>{entry.query}</code>
|
||||||
|
</button>
|
||||||
|
</li>
|
||||||
|
{/each}
|
||||||
|
</ul>
|
||||||
|
</details>
|
||||||
|
{/if}
|
||||||
|
|
||||||
|
<details class="cheatsheet">
|
||||||
|
<summary>Pipe syntax cheat sheet</summary>
|
||||||
|
<table>
|
||||||
|
<tbody>
|
||||||
|
<tr><td><code>field=value</code></td><td>filter on a structured field</td></tr>
|
||||||
|
<tr><td><code>"free text"</code> / bare word</td><td>full-text search on <code>message</code></td></tr>
|
||||||
|
<tr><td><code>message:"exact phrase"</code></td><td>explicit full-text search</td></tr>
|
||||||
|
<tr><td><code>| where field>value</code></td><td>additional structured filter</td></tr>
|
||||||
|
<tr><td><code>| stats count by field</code></td><td>aggregate (count/sum/avg/min/max)</td></tr>
|
||||||
|
<tr><td><code>| sort -field</code></td><td>sort descending (<code>+field</code> for ascending)</td></tr>
|
||||||
|
<tr><td><code>| fields a, b</code></td><td>project specific columns</td></tr>
|
||||||
|
<tr><td><code>| head 50</code> / <code>| tail 50</code></td><td>limit results</td></tr>
|
||||||
|
<tr><td><code>earliest=-1h</code> / <code>latest=...</code></td><td>time range (relative or RFC3339)</td></tr>
|
||||||
|
</tbody>
|
||||||
|
</table>
|
||||||
|
<p>Full reference: <code>/docs/query-language-reference.md</code> in the repo.</p>
|
||||||
|
</details>
|
||||||
</main>
|
</main>
|
||||||
|
|
||||||
<style>
|
<style>
|
||||||
@@ -77,11 +183,59 @@
|
|||||||
width: 100%;
|
width: 100%;
|
||||||
font-family: monospace;
|
font-family: monospace;
|
||||||
font-size: 0.9rem;
|
font-size: 0.9rem;
|
||||||
|
box-sizing: border-box;
|
||||||
}
|
}
|
||||||
button {
|
.controls {
|
||||||
margin-top: 0.5rem;
|
margin-top: 0.5rem;
|
||||||
|
display: flex;
|
||||||
|
align-items: center;
|
||||||
|
gap: 0.75rem;
|
||||||
|
flex-wrap: wrap;
|
||||||
|
}
|
||||||
|
.detected-badge {
|
||||||
|
font-size: 0.75rem;
|
||||||
|
padding: 0.15rem 0.5rem;
|
||||||
|
border-radius: 1rem;
|
||||||
|
background: #eef;
|
||||||
|
color: #224;
|
||||||
|
}
|
||||||
|
.detected-badge.sql {
|
||||||
|
background: #fee;
|
||||||
|
color: #422;
|
||||||
|
}
|
||||||
|
.hint {
|
||||||
|
font-size: 0.8rem;
|
||||||
|
color: #777;
|
||||||
}
|
}
|
||||||
.error {
|
.error {
|
||||||
color: #b00020;
|
color: #b00020;
|
||||||
}
|
}
|
||||||
|
.history ul {
|
||||||
|
list-style: none;
|
||||||
|
padding: 0;
|
||||||
|
margin: 0.5rem 0 0;
|
||||||
|
}
|
||||||
|
.history-item {
|
||||||
|
background: none;
|
||||||
|
border: none;
|
||||||
|
text-align: left;
|
||||||
|
padding: 0.25rem 0;
|
||||||
|
cursor: pointer;
|
||||||
|
color: #06c;
|
||||||
|
}
|
||||||
|
.history-item:hover {
|
||||||
|
text-decoration: underline;
|
||||||
|
}
|
||||||
|
.cheatsheet {
|
||||||
|
margin-top: 1.5rem;
|
||||||
|
font-size: 0.85rem;
|
||||||
|
}
|
||||||
|
.cheatsheet table {
|
||||||
|
border-collapse: collapse;
|
||||||
|
margin-top: 0.5rem;
|
||||||
|
}
|
||||||
|
.cheatsheet td {
|
||||||
|
padding: 0.2rem 0.75rem 0.2rem 0;
|
||||||
|
vertical-align: top;
|
||||||
|
}
|
||||||
</style>
|
</style>
|
||||||
|
|||||||
@@ -1,96 +0,0 @@
|
|||||||
<script lang="ts">
|
|
||||||
// Phase 1: free-text search via POST /search on the api service, hits
|
|
||||||
// the Tantivy-backed search service and returns full rows joined back
|
|
||||||
// against ClickHouse. No unified query experience with the SQL page —
|
|
||||||
// that's Phase 2's job.
|
|
||||||
|
|
||||||
import ResultsTable from '$lib/ResultsTable.svelte';
|
|
||||||
|
|
||||||
const apiBase = import.meta.env.VITE_API_BASE_URL ?? 'http://localhost:8080';
|
|
||||||
|
|
||||||
let query = $state('');
|
|
||||||
let columns = $state<string[]>([]);
|
|
||||||
let rows = $state<unknown[][]>([]);
|
|
||||||
let error = $state('');
|
|
||||||
let loading = $state(false);
|
|
||||||
let hasRun = $state(false);
|
|
||||||
|
|
||||||
async function runSearch() {
|
|
||||||
loading = true;
|
|
||||||
error = '';
|
|
||||||
try {
|
|
||||||
const res = await fetch(`${apiBase}/search`, {
|
|
||||||
method: 'POST',
|
|
||||||
headers: { 'Content-Type': 'application/json' },
|
|
||||||
body: JSON.stringify({ query })
|
|
||||||
});
|
|
||||||
const body = await res.json();
|
|
||||||
if (!res.ok) {
|
|
||||||
error = body?.error ?? `request failed with status ${res.status}`;
|
|
||||||
columns = [];
|
|
||||||
rows = [];
|
|
||||||
return;
|
|
||||||
}
|
|
||||||
columns = body.columns ?? [];
|
|
||||||
rows = body.rows ?? [];
|
|
||||||
} catch (e) {
|
|
||||||
error = e instanceof Error ? e.message : String(e);
|
|
||||||
columns = [];
|
|
||||||
rows = [];
|
|
||||||
} finally {
|
|
||||||
loading = false;
|
|
||||||
hasRun = true;
|
|
||||||
}
|
|
||||||
}
|
|
||||||
</script>
|
|
||||||
|
|
||||||
<main>
|
|
||||||
<h1>Sentry — Full-Text Search</h1>
|
|
||||||
<p>
|
|
||||||
Free-text search over the <code>message</code> field, via Tantivy. Supports plain terms,
|
|
||||||
<code>"exact phrases"</code>, and <code>wildcard*</code> — see
|
|
||||||
<code>/search</code> for the full query syntax. Looking for structured/aggregation queries
|
|
||||||
instead? See the <a href="/">SQL Query</a> page.
|
|
||||||
</p>
|
|
||||||
|
|
||||||
<input
|
|
||||||
type="text"
|
|
||||||
bind:value={query}
|
|
||||||
placeholder="e.g. "connection refused" or timeout*"
|
|
||||||
spellcheck="false"
|
|
||||||
onkeydown={(e) => e.key === 'Enter' && runSearch()}
|
|
||||||
/>
|
|
||||||
<div>
|
|
||||||
<button onclick={runSearch} disabled={loading || query.trim() === ''}>
|
|
||||||
{loading ? 'Searching…' : 'Search'}
|
|
||||||
</button>
|
|
||||||
</div>
|
|
||||||
|
|
||||||
{#if error}
|
|
||||||
<p class="error">Error: {error}</p>
|
|
||||||
{/if}
|
|
||||||
|
|
||||||
<ResultsTable {columns} {rows} {hasRun} />
|
|
||||||
</main>
|
|
||||||
|
|
||||||
<style>
|
|
||||||
main {
|
|
||||||
font-family: system-ui, sans-serif;
|
|
||||||
max-width: 960px;
|
|
||||||
margin: 2rem auto;
|
|
||||||
padding: 0 1rem;
|
|
||||||
}
|
|
||||||
input {
|
|
||||||
width: 100%;
|
|
||||||
font-family: monospace;
|
|
||||||
font-size: 0.9rem;
|
|
||||||
padding: 0.4rem;
|
|
||||||
box-sizing: border-box;
|
|
||||||
}
|
|
||||||
button {
|
|
||||||
margin-top: 0.5rem;
|
|
||||||
}
|
|
||||||
.error {
|
|
||||||
color: #b00020;
|
|
||||||
}
|
|
||||||
</style>
|
|
||||||
@@ -1,4 +0,0 @@
|
|||||||
// Same reasoning as the root page's +page.ts: no load function (all data
|
|
||||||
// comes from a client-side fetch on submit), so a plain prerender is
|
|
||||||
// enough for the static adapter.
|
|
||||||
export const prerender = true;
|
|
||||||
Reference in New Issue
Block a user