coffey-labs/cairnobs
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Phase 1: Windows log collection + full-text search

Browse Source 
Extends the agent, ingest, storage, api, and web with Windows Event
Log/ETW sourcing and Tantivy-backed free-text search, per the approved
Phase 1 plan.

- CLAUDE.md: materialized on disk (never existed as a file before) with
  a new Phase 1 "done looks like" section.
- agent: Windows Event Log (EvtSubscribe) and ETW sources, Windows
  service wrapper (install/uninstall/run-service), both feature- and
  target_os-gated so Linux builds/tests/clippy stay unaffected. Also
  fixed two pre-existing Phase 0 clippy gaps (dead-code on
  default-features-only builds, a type-inference edge case) found while
  testing every feature combination properly for the first time.
  UNVERIFIED on real Windows -- no Windows toolchain existed anywhere in
  the build environment; flagged prominently in three places.
- proto/ingest: new record_id field, assigned once server-side in
  ingest's gRPC front end so ClickHouse and Tantivy agree on the same ID
  for the same record.
- storage: record_id column + bloom filter index, verified against a
  live ClickHouse.
- search: new service, Tantivy index, rskafka consumer as an independent
  second consumer group on the same Redpanda topic ingest already reads.
- api/web: new /search endpoint and page, sharing the query page's
  result-table shape and component.
- hack/windows-fixture: sends realistic Windows-shaped data straight to
  ingest, so the pipeline's handling of it is verifiable without a
  Windows host.

Verified end-to-end on the live docker-compose stack: the same record_id
comes back from both /query and /search for the same log line, including
for windows-fixture's synthetic Windows Event Log data. Real bugs found
and fixed along the way: api/Dockerfile missing proto/ in its build
context, search's logs being completely silent (RUST_LOG gap), and
search/target/ missing from .gitignore/.dockerignore.
This commit is contained in:
jcoffey-dev
2026-08-13 11:27:35 -07:00
parent fe854b1091
commit cd8aa290ca
66 changed files with 6084 additions and 171 deletions
Download Patch File Download Diff File Expand all files Collapse all files
api/Dockerfile
+4 -1
View File
@@ -1,8 +1,11 @@
# Build context must be the repo root (sentry/):
# Build context must be the repo root (sentry/), since this needs both
# api/ and proto/ (api now speaks gRPC to /search, using proto's checked-in
# Go bindings via the `replace` directive in api/go.mod):
# docker build -f api/Dockerfile -t sentry-api .
FROM golang:1.25-alpine AS builder
WORKDIR /src
COPY proto ./proto
COPY api ./api
WORKDIR /src/api
RUN go mod download
api/README.md
+41 -19
View File
@@ -1,17 +1,19 @@
# api
Sentry's Phase 0 query API: one crude, intentionally placeholder endpoint.
Sentry's query API: two intentionally crude endpoints — raw SQL and
free-text search — that Phase 2's real query layer replaces outright.
## Why plain REST, not gRPC + REST gateway
CLAUDE.md pins the control plane to "Go, gRPC + REST gateway." This
service is plain `net/http` instead — a deliberate Phase 0 simplification,
not a change to the pinned stack. Wiring up a `.proto` service,
`google.api.http` annotations, and `protoc-gen-grpc-gateway` codegen for a
single endpoint that Phase 2 replaces outright with a real SPL-like query
service is plain `net/http` instead — a deliberate simplification, not a
change to the pinned stack. Wiring up `.proto` services,
`google.api.http` annotations, and `protoc-gen-grpc-gateway` codegen for
two endpoints that Phase 2 replaces outright with a real SPL-like query
layer would be exactly the kind of premature machinery this project's
conventions warn against. Adopt the gRPC+gateway pattern once `/api` grows
a second real, durable endpoint.
conventions warn against. `api` *does* speak gRPC internally though — to
`/search` (see below) — this simplification is specifically about the
public-facing surface, not a blanket avoidance of gRPC.
## Endpoints
@@ -20,10 +22,21 @@ a second real, durable endpoint.
SELECT-only, single-statement, basic keyword-based injection guarding
(see `internal/queryapi/validate.go` for exactly what that does and
doesn't catch — it's not a SQL parser).
- `POST /search` — body `{"query": "...", "limit": 100}`, same response
shape as `/query`. Calls `/search`'s `SearchService.Search` gRPC RPC to
resolve the free-text query into matching `record_id`s, then joins
those back against ClickHouse (`SELECT * FROM logs WHERE record_id IN
(...)`) to return full rows — so both endpoints return the same
`{columns, rows}` shape and `/web` can reuse one table component for
both. Every `record_id` is validated as a real UUID before being
embedded in the generated SQL (defense in depth: `record_id`s come from
an internal, trusted service, not raw user input, but a value that
fails to parse as a UUID can't contain SQL-breaking characters either
way).
- `GET /healthz` — for docker-compose/k8s liveness checks.
No auth. Not scoped for Phase 0 — don't expose this beyond a trusted
dev/homelab network.
No auth. Not scoped yet — don't expose this beyond a trusted dev/homelab
network.
## Configuration
@@ -34,9 +47,15 @@ Environment variables (see `internal/config/config.go`):
| `HTTP_LISTEN_ADDR` | `:8080` | |
| `CLICKHOUSE_ADDR` | `localhost:9000` | Native protocol port |
| `CLICKHOUSE_DATABASE` / `_USERNAME` / `_PASSWORD` | `sentry` / `default` / `` | |
| `QUERY_TIMEOUT_SECONDS` | `30` | Per-request ClickHouse query timeout |
| `SEARCH_GRPC_ADDR` | `localhost:50052` | Must match `/search`'s `GRPC_LISTEN_ADDR` |
| `QUERY_TIMEOUT_SECONDS` | `30` | Per-request timeout, both endpoints |
| `CORS_ALLOWED_ORIGIN` | `*` | Wide open by default since there's no auth yet; tighten together |
`searchclient.Dial` connects to `/search` over plain TCP, no TLS — same
trust boundary as `api`'s existing plain-TCP connection to ClickHouse.
mTLS in this project is specifically the agent↔ingest edge boundary, not
every internal hop.
## Building & testing
```sh
@@ -52,12 +71,15 @@ docker build -f api/Dockerfile -t sentry-api .
## Testing notes
`internal/queryapi`'s HTTP handler depends on ClickHouse only through a
one-method `queryExecutor` interface, so routing, validation, JSON
encoding, and error-status mapping are all unit-tested against a fake —
no live ClickHouse needed. `Executor` itself (the reflection-based row
scanning against `driver.Rows`) is not unit-tested — faking ClickHouse's
`driver.Rows` interface fully would be significant test-only scaffolding
for a Phase 0 placeholder, and the driver package's own docs note it isn't
meant to be implemented by adopters. It's exercised end-to-end via the
docker-compose flow in `/docs/phase-0-runbook.md` instead.
`internal/queryapi`'s HTTP handlers depend on ClickHouse and `/search`
only through narrow interfaces (`queryExecutor`, `searchClient`), so
routing, validation, JSON encoding, error-status mapping, and the
record_id-to-SQL query building are all unit-tested against fakes — no
live ClickHouse or `/search` instance needed. `Executor` itself (the
reflection-based row scanning against `driver.Rows`) and
`internal/searchclient`'s actual gRPC dial are not unit-tested — the
former because faking ClickHouse's `driver.Rows` interface fully would be
significant test-only scaffolding the driver's own docs say isn't meant
to be implemented by adopters; the latter because it's a thin wrapper
with nothing but wiring to test. Both are exercised end-to-end via the
docker-compose flow in `/docs/phase-1-runbook.md` instead.
api/cmd/api/main.go
+13 -5
View File
@@ -1,7 +1,7 @@
// Command api is the Sentry Phase 0 query API: a single crude POST /query
// endpoint proxying allowlisted SELECT statements to ClickHouse. See
// internal/queryapi for why this is plain REST rather than the pinned
// gRPC+gateway pattern for Phase 0.
// Command api is Sentry's query API: POST /query (raw SQL, SELECT-only)
// and POST /search (free-text, via the search service). See
// internal/queryapi for why these are plain REST rather than the pinned
// gRPC+gateway pattern.
package main
import (
@@ -17,6 +17,7 @@ import (
"github.com/sentry/sentry/api/internal/config"
"github.com/sentry/sentry/api/internal/queryapi"
"github.com/sentry/sentry/api/internal/searchclient"
)
func main() {
@@ -50,8 +51,15 @@ func main() {
os.Exit(1)
}
search, err := searchclient.Dial(cfg.SearchGRPCAddr)
if err != nil {
logger.Error("dialing search service", "error", err)
os.Exit(1)
}
defer search.Close()
exec := queryapi.NewExecutor(conn)
handler := queryapi.NewHandler(logger, exec, cfg.QueryTimeout, cfg.CORSAllowedOrigin)
handler := queryapi.NewHandler(logger, exec, search, cfg.QueryTimeout, cfg.CORSAllowedOrigin)
srv := &http.Server{
Addr: cfg.HTTPListenAddr,
api/go.mod
+12 -2
View File
@@ -2,7 +2,14 @@ module github.com/sentry/sentry/api
go 1.25.0
require github.com/ClickHouse/clickhouse-go/v2 v2.48.0
require (
github.com/ClickHouse/clickhouse-go/v2 v2.48.0
github.com/google/uuid v1.6.0
github.com/sentry/sentry/proto v0.0.0-00010101000000-000000000000
google.golang.org/grpc v1.83.0
)
replace github.com/sentry/sentry/proto => ../proto
require (
github.com/ClickHouse/ch-go v0.74.0 // indirect
@@ -10,7 +17,6 @@ require (
github.com/cespare/xxhash/v2 v2.3.0 // indirect
github.com/go-faster/city v1.0.1 // indirect
github.com/go-faster/errors v0.7.1 // indirect
github.com/google/uuid v1.6.0 // indirect
github.com/klauspost/compress v1.19.1 // indirect
github.com/paulmach/orb v0.13.0 // indirect
github.com/pierrec/lz4/v4 v4.1.27 // indirect
@@ -18,5 +24,9 @@ require (
github.com/shopspring/decimal v1.4.0 // indirect
go.opentelemetry.io/otel v1.44.0 // indirect
go.opentelemetry.io/otel/trace v1.44.0 // indirect
golang.org/x/net v0.57.0 // indirect
golang.org/x/sys v0.47.0 // indirect
golang.org/x/text v0.40.0 // indirect
google.golang.org/genproto/googleapis/rpc v0.0.0-20260526163538-3dc84a4a5aaa // indirect
google.golang.org/protobuf v1.36.12 // indirect
)
api/go.sum
+26
View File
@@ -12,6 +12,12 @@ github.com/go-faster/city v1.0.1 h1:4WAxSZ3V2Ws4QRDrscLEDcibJY8uf41H6AhXDrNDcGw=
github.com/go-faster/city v1.0.1/go.mod h1:jKcUJId49qdW3L1qKHH/3wPeUstCVpVSXTM6vO3VcTw=
github.com/go-faster/errors v0.7.1 h1:MkJTnDoEdi9pDabt1dpWf7AA8/BaSYZqibYyhZ20AYg=
github.com/go-faster/errors v0.7.1/go.mod h1:5ySTjWFiphBs07IKuiL69nxdfd5+fzh1u7FPGZP2quo=
github.com/go-logr/logr v1.4.3 h1:CjnDlHq8ikf6E492q6eKboGOC0T8CDaOvkHCIg8idEI=
github.com/go-logr/logr v1.4.3/go.mod h1:9T104GzyrTigFIr8wt5mBrctHMim0Nb2HLGrmQ40KvY=
github.com/go-logr/stdr v1.2.2 h1:hSWxHoqTgW2S2qGc0LTAI563KZ5YKYRhT3MFKZMbjag=
github.com/go-logr/stdr v1.2.2/go.mod h1:mMo/vtBO5dYbehREoey6XUKy/eSumjCCveDpRre4VKE=
github.com/golang/protobuf v1.5.4 h1:i7eJL8qZTpSEXOPTxNKhASYpMn+8e5Q6AdndVa1dWek=
github.com/golang/protobuf v1.5.4/go.mod h1:lnTiLA8Wa4RWRcIUkrtSVa5nRhsEGBg48fD6rSs7xps=
github.com/google/go-cmp v0.7.0 h1:wk8382ETsv4JYUZwIsn6YpYiWiBsYLSJiTsyBybVuN8=
github.com/google/go-cmp v0.7.0/go.mod h1:pXiqmnSA92OHEEa9HXL2W4E7lf9JzCmGVUdgjX3N/iU=
github.com/google/uuid v1.6.0 h1:NIvaJDMOsjHA8n1jAhLSgzrAzy1Hgr+hNrb57e+94F0=
@@ -32,11 +38,31 @@ github.com/stretchr/testify v1.11.1 h1:7s2iGBzp5EwR7/aIZr8ao5+dra3wiQyKjjFuvgVKu
github.com/stretchr/testify v1.11.1/go.mod h1:wZwfW3scLgRK+23gO65QZefKpKQRnfz6sD981Nm4B6U=
github.com/xyproto/randomstring v1.0.5 h1:YtlWPoRdgMu3NZtP45drfy1GKoojuR7hmRcnhZqKjWU=
github.com/xyproto/randomstring v1.0.5/go.mod h1:rgmS5DeNXLivK7YprL0pY+lTuhNQW3iGxZ18UQApw/E=
go.opentelemetry.io/auto/sdk v1.2.1 h1:jXsnJ4Lmnqd11kwkBV2LgLoFMZKizbCi5fNZ/ipaZ64=
go.opentelemetry.io/auto/sdk v1.2.1/go.mod h1:KRTj+aOaElaLi+wW1kO/DZRXwkF4C5xPbEe3ZiIhN7Y=
go.opentelemetry.io/otel v1.44.0 h1:JjwHmHpA4iZ3wBxluu2fbbE7j4kqlE8jXyAyPXH7HqU=
go.opentelemetry.io/otel v1.44.0/go.mod h1:BMgjTHL9WPRlRjL2oZCBTL4whCGtXch2H4BhOPIAyYc=
go.opentelemetry.io/otel/metric v1.44.0 h1:1w0gILTcHdr3YI+ixLyjemwrVnsMURbTZFrSYCdDdmc=
go.opentelemetry.io/otel/metric v1.44.0/go.mod h1:8O7hanEPBNgEMmybD3s2VBKcgWOCsA6tzHBPODAiquo=
go.opentelemetry.io/otel/sdk v1.44.0 h1:nHYwb9lK+fJPU/dnT6s7W7Z8itMWyqrnVfbheVYrZ58=
go.opentelemetry.io/otel/sdk v1.44.0/go.mod h1:Osuydd3Se74nqjAKxid74N5eC+jfEqfTegHRnq58oK0=
go.opentelemetry.io/otel/sdk/metric v1.44.0 h1:3LlKgI+VjbVsjNRFZJZAJ30WjXC5VkNRks6si09iEfI=
go.opentelemetry.io/otel/sdk/metric v1.44.0/go.mod h1:5B5pMARnXxKhltooO4xUuCBorl65a4EpnTalObqOigA=
go.opentelemetry.io/otel/trace v1.44.0 h1:jxF5CsGYCe74MCRx2X4g7WsY/VBKRqqpNvXlX/6gtIk=
go.opentelemetry.io/otel/trace v1.44.0/go.mod h1:oLl1jrMQAVo6v3GAggN+1VH9VIz9iUSvW53sW1Q8PIE=
golang.org/x/net v0.57.0 h1:K5+3DljvIuDG9/Jv9rvyMywYNFCQ9RSUY6OOTTkT+tE=
golang.org/x/net v0.57.0/go.mod h1:KpXc8iv+r3XplLAG/f7Jsf9RPszJzdR0f58q9vGOuEU=
golang.org/x/sys v0.47.0 h1:o7XGOvZQCADBQQ4Y7VNq2dRWQR7JmOUW8Kxx4ZsNgWs=
golang.org/x/sys v0.47.0/go.mod h1:4GL1E5IUh+htKOUEOaiffhrAeqysfVGipDYzABqnCmw=
golang.org/x/text v0.40.0 h1:Ub2Z6/xjgF1WrYQz2nuITOEegKFtiIy+rieRJ5lHZKs=
golang.org/x/text v0.40.0/go.mod h1:hpnzDAfGV753zIKo+wk3u1bVKCGPbrnF7+7LBF/UHVY=
gonum.org/v1/gonum v0.17.0 h1:VbpOemQlsSMrYmn7T2OUvQ4dqxQXU+ouZFQsZOx50z4=
gonum.org/v1/gonum v0.17.0/go.mod h1:El3tOrEuMpv2UdMrbNlKEh9vd86bmQ6vqIcDwxEOc1E=
google.golang.org/genproto/googleapis/rpc v0.0.0-20260526163538-3dc84a4a5aaa h1:mZHHdPZl0dbGHCflZgAq/Q468DWVFcU2whhB2KAo8fk=
google.golang.org/genproto/googleapis/rpc v0.0.0-20260526163538-3dc84a4a5aaa/go.mod h1:4Hqkh8ycfw05ld/3BWL7rJOSfebL2Q+DVDeRgYgxUU8=
google.golang.org/grpc v1.83.0 h1:JeNZEKJFbQxArAMl+hiytHauacDNqJUllNfmIMmpqnQ=
google.golang.org/grpc v1.83.0/go.mod h1:kDyl6SKsiHKt0uylY5gtn5cEjkrIOhQOGDgIc4JGwzQ=
google.golang.org/protobuf v1.36.12 h1:pJOKDDOyeXErUroCihFAd5LQuwXBSpVnKGrj5o/fwxc=
google.golang.org/protobuf v1.36.12/go.mod h1:HTf+CrKn2C3g5S8VImy6tdcUvCska2kB7j23XfzDpco=
gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA=
gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
api/internal/config/config.go
+4
View File
@@ -12,6 +12,7 @@ import (
type Config struct {
HTTPListenAddr string
ClickHouse ClickHouseConfig
SearchGRPCAddr string
QueryTimeout time.Duration
CORSAllowedOrigin string
}
@@ -32,6 +33,9 @@ func Load() (Config, error) {
Username: getenv("CLICKHOUSE_USERNAME", "default"),
Password: getenv("CLICKHOUSE_PASSWORD", ""),
},
// Search service's gRPC address (see /search) -- default matches
// /search's own default GRPC_LISTEN_ADDR.
SearchGRPCAddr: getenv("SEARCH_GRPC_ADDR", "localhost:50052"),
// Phase 0 has no auth, so this is wide open by default to keep
// the local SvelteKit dev server (a different origin/port)
// working out of the box. Tighten before this is ever reachable
api/internal/config/config_test.go
+3
View File
@@ -19,6 +19,9 @@ func TestLoadDefaults(t *testing.T) {
if cfg.CORSAllowedOrigin != "*" {
t.Errorf("CORSAllowedOrigin = %q, want *", cfg.CORSAllowedOrigin)
}
if cfg.SearchGRPCAddr != "localhost:50052" {
t.Errorf("SearchGRPCAddr = %q, want localhost:50052", cfg.SearchGRPCAddr)
}
}
func TestLoadInvalidTimeoutErrors(t *testing.T) {
api/internal/queryapi/handler.go
+15 -16
View File
@@ -1,13 +1,13 @@
// Package queryapi is the Phase 0 query API: a single crude POST /query
// endpoint that takes a raw SQL string, allowlists it to a single SELECT
// statement, and proxies it to ClickHouse. This is a deliberate
// simplification of the pinned "gRPC + REST gateway" control-plane
// pattern (see CLAUDE.md's tech stack table): a plain net/http REST
// handler, not a gRPC service transcoded through grpc-gateway. That
// machinery (proto definitions, googleapis annotations, gateway codegen)
// buys nothing for one crude placeholder endpoint that Phase 2 replaces
// outright with the real SPL-like query layer. Revisit gRPC+gateway when
// /api grows a second real endpoint.
// Package queryapi is Sentry's query API: POST /query (Phase 0, a crude
// raw-SQL passthrough allowlisted to SELECT) and POST /search (Phase 1,
// free-text search via the search service, joined back against
// ClickHouse). This is a deliberate simplification of the pinned "gRPC +
// REST gateway" control-plane pattern (see CLAUDE.md's tech stack table):
// plain net/http REST handlers, not a gRPC service transcoded through
// grpc-gateway. That machinery (proto definitions, googleapis
// annotations, gateway codegen) doesn't buy much for two crude endpoints
// that Phase 2's real SPL-like query layer replaces outright. Revisit
// gRPC+gateway once /api's endpoint count and lifespan justify it.
package queryapi
import (
@@ -28,17 +28,19 @@ type queryExecutor interface {
type Handler struct {
logger *slog.Logger
exec queryExecutor
search searchClient
queryTimeout time.Duration
allowedOrigin string
}
func NewHandler(logger *slog.Logger, exec queryExecutor, queryTimeout time.Duration, allowedOrigin string) *Handler {
return &Handler{logger: logger, exec: exec, queryTimeout: queryTimeout, allowedOrigin: allowedOrigin}
func NewHandler(logger *slog.Logger, exec queryExecutor, search searchClient, queryTimeout time.Duration, allowedOrigin string) *Handler {
return &Handler{logger: logger, exec: exec, search: search, queryTimeout: queryTimeout, allowedOrigin: allowedOrigin}
}
func (h *Handler) Routes() http.Handler {
mux := http.NewServeMux()
mux.HandleFunc("POST /query", h.handleQuery)
mux.HandleFunc("POST /search", h.handleSearch)
mux.HandleFunc("GET /healthz", h.handleHealthz)
return h.withCORS(mux)
}
@@ -99,10 +101,7 @@ func (h *Handler) handleQuery(w http.ResponseWriter, r *http.Request) {
return
}
w.Header().Set("Content-Type", "application/json")
if err := json.NewEncoder(w).Encode(result); err != nil {
h.logger.Error("encoding response", "error", err)
}
writeJSON(w, result)
}
func writeError(w http.ResponseWriter, status int, msg string) {
api/internal/queryapi/handler_test.go
+17 -1
View File
@@ -27,8 +27,24 @@ func (f *fakeExecutor) Execute(_ context.Context, sql string) (*QueryResult, err
return f.result, nil
}
type fakeSearchClient struct {
recordIDs []string
err error
}
func (f *fakeSearchClient) Search(_ context.Context, _ string, _ uint32) ([]string, error) {
if f.err != nil {
return nil, f.err
}
return f.recordIDs, nil
}
func newTestHandler(exec queryExecutor) *Handler {
return NewHandler(slog.New(slog.NewTextHandler(io.Discard, nil)), exec, time.Second, "*")
return newTestHandlerWithSearch(exec, &fakeSearchClient{})
}
func newTestHandlerWithSearch(exec queryExecutor, search searchClient) *Handler {
return NewHandler(slog.New(slog.NewTextHandler(io.Discard, nil)), exec, search, time.Second, "*")
}
func TestHandleQuerySuccess(t *testing.T) {
api/internal/queryapi/search.go
+96
View File
@@ -0,0 +1,96 @@
package queryapi
import (
"context"
"encoding/json"
"fmt"
"net/http"
"strings"
"github.com/google/uuid"
)
// searchClient is the narrow interface handleSearch depends on, so tests
// can substitute a fake without a real search service. A small gRPC
// adapter in cmd/api satisfies this.
type searchClient interface {
Search(ctx context.Context, query string, limit uint32) ([]string, error)
}
type searchRequest struct {
Query string `json:"query"`
Limit uint32 `json:"limit"`
}
func (h *Handler) handleSearch(w http.ResponseWriter, r *http.Request) {
r.Body = http.MaxBytesReader(w, r.Body, maxBodyBytes)
var req searchRequest
if err := json.NewDecoder(r.Body).Decode(&req); err != nil {
writeError(w, http.StatusBadRequest, "invalid JSON body: "+err.Error())
return
}
if strings.TrimSpace(req.Query) == "" {
writeError(w, http.StatusBadRequest, "query must not be empty")
return
}
ctx, cancel := context.WithTimeout(r.Context(), h.queryTimeout)
defer cancel()
recordIDs, err := h.search.Search(ctx, req.Query, req.Limit)
if err != nil {
h.logger.Error("search failed", "error", err)
writeError(w, http.StatusBadGateway, "search failed: "+err.Error())
return
}
if len(recordIDs) == 0 {
writeJSON(w, &QueryResult{Columns: []string{}, Rows: [][]any{}})
return
}
sql, err := recordIDsQuery(recordIDs)
if err != nil {
h.logger.Error("building record_id query", "error", err)
writeError(w, http.StatusBadGateway, "search returned unusable results")
return
}
result, err := h.exec.Execute(ctx, sql)
if err != nil {
h.logger.Error("joining search results against clickhouse failed", "error", err)
writeError(w, http.StatusBadGateway, "query failed: "+err.Error())
return
}
writeJSON(w, result)
}
// recordIDsQuery builds a SELECT ... WHERE record_id IN (...) against the
// IDs the search service returned. Every ID is validated as a real UUID
// before being embedded in the query string -- record_ids come from an
// internal, trusted service (not raw user input), but a UUID that fails
// to parse can't contain SQL-breaking characters either way, so this is
// defense in depth, not a response to a specific threat.
func recordIDsQuery(recordIDs []string) (string, error) {
quoted := make([]string, 0, len(recordIDs))
for _, id := range recordIDs {
if _, err := uuid.Parse(id); err != nil {
continue // skip anything not a valid UUID rather than failing the whole query
}
quoted = append(quoted, "'"+id+"'")
}
if len(quoted) == 0 {
return "", fmt.Errorf("no valid record_ids in search response")
}
return fmt.Sprintf(
"SELECT * FROM logs WHERE record_id IN (%s) ORDER BY timestamp DESC",
strings.Join(quoted, ","),
), nil
}
func writeJSON(w http.ResponseWriter, v any) {
w.Header().Set("Content-Type", "application/json")
_ = json.NewEncoder(w).Encode(v)
}
api/internal/queryapi/search_test.go
+125
View File
@@ -0,0 +1,125 @@
package queryapi
import (
"encoding/json"
"errors"
"net/http"
"net/http/httptest"
"strings"
"testing"
)
func TestHandleSearchSuccess(t *testing.T) {
id := "5754b062-ec8b-45b1-b1b8-a50f263adcd3"
fe := &fakeExecutor{result: &QueryResult{
Columns: []string{"message"},
Rows: [][]any{{"hello world"}},
}}
fs := &fakeSearchClient{recordIDs: []string{id}}
h := newTestHandlerWithSearch(fe, fs)
body := strings.NewReader(`{"query": "hello"}`)
req := httptest.NewRequest(http.MethodPost, "/search", body)
rec := httptest.NewRecorder()
h.Routes().ServeHTTP(rec, req)
if rec.Code != http.StatusOK {
t.Fatalf("status = %d, want 200; body=%s", rec.Code, rec.Body.String())
}
if !strings.Contains(fe.gotSQL, id) {
t.Fatalf("expected the record_id in the generated SQL, got %q", fe.gotSQL)
}
if !strings.Contains(fe.gotSQL, "WHERE record_id IN") {
t.Fatalf("expected an IN clause, got %q", fe.gotSQL)
}
var got QueryResult
if err := json.Unmarshal(rec.Body.Bytes(), &got); err != nil {
t.Fatalf("decoding response: %v", err)
}
if len(got.Rows) != 1 {
t.Fatalf("unexpected result: %+v", got)
}
}
func TestHandleSearchRejectsEmptyQuery(t *testing.T) {
fe := &fakeExecutor{}
fs := &fakeSearchClient{}
h := newTestHandlerWithSearch(fe, fs)
body := strings.NewReader(`{"query": " "}`)
req := httptest.NewRequest(http.MethodPost, "/search", body)
rec := httptest.NewRecorder()
h.Routes().ServeHTTP(rec, req)
if rec.Code != http.StatusBadRequest {
t.Fatalf("status = %d, want 400", rec.Code)
}
if fe.gotSQL != "" {
t.Fatal("executor should not have been called for an empty query")
}
}
func TestHandleSearchNoResultsReturnsEmptyNotError(t *testing.T) {
fe := &fakeExecutor{}
fs := &fakeSearchClient{recordIDs: nil}
h := newTestHandlerWithSearch(fe, fs)
body := strings.NewReader(`{"query": "nothing matches this"}`)
req := httptest.NewRequest(http.MethodPost, "/search", body)
rec := httptest.NewRecorder()
h.Routes().ServeHTTP(rec, req)
if rec.Code != http.StatusOK {
t.Fatalf("status = %d, want 200; body=%s", rec.Code, rec.Body.String())
}
if fe.gotSQL != "" {
t.Fatal("executor should not have been called when search returns no IDs")
}
var got QueryResult
if err := json.Unmarshal(rec.Body.Bytes(), &got); err != nil {
t.Fatalf("decoding response: %v", err)
}
if len(got.Rows) != 0 {
t.Fatalf("expected empty rows, got %+v", got.Rows)
}
}
func TestHandleSearchServiceErrorReturnsBadGateway(t *testing.T) {
fe := &fakeExecutor{}
fs := &fakeSearchClient{err: errors.New("search service unreachable")}
h := newTestHandlerWithSearch(fe, fs)
body := strings.NewReader(`{"query": "hello"}`)
req := httptest.NewRequest(http.MethodPost, "/search", body)
rec := httptest.NewRecorder()
h.Routes().ServeHTTP(rec, req)
if rec.Code != http.StatusBadGateway {
t.Fatalf("status = %d, want 502", rec.Code)
}
}
func TestRecordIDsQuerySkipsInvalidUUIDs(t *testing.T) {
sql, err := recordIDsQuery([]string{"not-a-uuid", "5754b062-ec8b-45b1-b1b8-a50f263adcd3"})
if err != nil {
t.Fatalf("recordIDsQuery() error = %v", err)
}
if strings.Contains(sql, "not-a-uuid") {
t.Fatalf("expected the invalid UUID to be skipped, got %q", sql)
}
if !strings.Contains(sql, "5754b062-ec8b-45b1-b1b8-a50f263adcd3") {
t.Fatalf("expected the valid UUID to be included, got %q", sql)
}
}
func TestRecordIDsQueryAllInvalidReturnsError(t *testing.T) {
if _, err := recordIDsQuery([]string{"not-a-uuid", "also-not-one"}); err == nil {
t.Fatal("expected an error when no IDs are valid UUIDs")
}
}
api/internal/searchclient/client.go
+44
View File
@@ -0,0 +1,44 @@
// Package searchclient adapts the generated gRPC SearchServiceClient to
// the narrow queryapi.searchClient interface, so queryapi doesn't need to
// know anything about gRPC/protobuf directly.
package searchclient
import (
"context"
"fmt"
"google.golang.org/grpc"
"google.golang.org/grpc/credentials/insecure"
searchv1 "github.com/sentry/sentry/proto/sentry/search/v1"
)
type Client struct {
grpc searchv1.SearchServiceClient
conn *grpc.ClientConn
}
// Dial connects to the search service. Plain TCP, no TLS: internal
// service-to-service traffic (api <-> search), same trust boundary as
// api's existing plain-TCP connection to ClickHouse -- mTLS in this
// project is specifically the agent<->ingest edge boundary, not every
// internal hop.
func Dial(addr string) (*Client, error) {
conn, err := grpc.NewClient(addr, grpc.WithTransportCredentials(insecure.NewCredentials()))
if err != nil {
return nil, fmt.Errorf("dialing search service at %s: %w", addr, err)
}
return &Client{grpc: searchv1.NewSearchServiceClient(conn), conn: conn}, nil
}
func (c *Client) Close() error {
return c.conn.Close()
}
func (c *Client) Search(ctx context.Context, query string, limit uint32) ([]string, error) {
resp, err := c.grpc.Search(ctx, &searchv1.SearchRequest{Query: query, Limit: limit})
if err != nil {
return nil, err
}
return resp.GetRecordIds(), nil
}