Phase 7: AI-assisted query authoring (autocomplete, explain, fix, optimize, NL translation)
Adds a self-hosted (Ollama, qwen2.5-coder) model provider abstraction with a pluggable opt-in cloud adapter, schema grounding, and a shared cost/safety guard every AI-suggested query is assessed against -- compiling to and executing through the same unchanged Phase 2 IR/ compiler and Phase 4 tenant scoping as a hand-written query, no parallel execution path. Track A (built into the query bar): inline ghost-text autocomplete, "Explain this query", "Fix this query" with a diff view, and a rule-based "Optimize" suggestion. Track B: natural-language-to-query translation, always a separate review step from execution, with `sentryctl query --nl` requiring explicit confirmation to run. Every accepted/dismissed translate-fix-optimize interaction is logged into the same append-only audit_log table Phase 4 built. Two real product bugs were found and fixed via live browser verification (a Svelte effect re-running on every keystroke that silently cancelled the ghost-text debounce; a ghost-text widget positioned at document offset 0 instead of the cursor), and a real costguard logic bug (unbounded-aggregation vs. raw-row) was caught by its own test suite. New integration tests wire a real Ollama client through the real HTTP handler against a mock server matching Ollama's wire contract (hack/mock-ollama), keeping model-quality verification out of CI as a disclosed, periodic human-run check instead. See /docs/phase-7-ai-design.md and /docs/phase-7-runbook.md.
This commit is contained in:
@@ -13,6 +13,7 @@ search/target/
|
||||
/hack/benchmark-fixture/benchmark-fixture
|
||||
/hack/alert-load-test/alert-load-test
|
||||
/hack/webhook-sink/webhook-sink
|
||||
/hack/mock-ollama/mock-ollama
|
||||
|
||||
# Node / SvelteKit (web/ has its own more detailed .gitignore too)
|
||||
web/node_modules/
|
||||
|
||||
@@ -507,6 +507,65 @@ judged sufficient); redesigning `favicon.svg` (flagged as a leftover
|
||||
SvelteKit scaffold asset, not a license blocker — a design task, not a
|
||||
compliance one).
|
||||
|
||||
## What "done" looks like for Phase 7
|
||||
|
||||
**Status: shipped.** AI-assisted query authoring: from the same query
|
||||
bar, a user can (a) get AI-assisted autocomplete, explanations, and fix
|
||||
suggestions while writing pipe-syntax or SQL queries by hand, and (b)
|
||||
type a plain-English question and get a generated structured query with
|
||||
explanation, editable before running — both paths executing through the
|
||||
unchanged Phase 2 compiler (`api/internal/querylang/planner`,
|
||||
`api/querylang/executor`) with Phase 4 tenant scoping, cost guardrails,
|
||||
and audit logging applying identically to both. No cloud dependency
|
||||
required for the default deployment (self-hosted via Ollama,
|
||||
`qwen2.5-coder:7b`/`1.5b`, both Apache-2.0 — chosen specifically to keep
|
||||
Phase 6's license-purity work intact; a pluggable, opt-in, off-by-default
|
||||
cloud adapter exists for deployments that want one).
|
||||
|
||||
Non-negotiable design principle held throughout, confirmed by inspection
|
||||
of the actual code paths rather than merely asserted: every AI-assisted
|
||||
or AI-translated query compiles down to and executes through the same
|
||||
Phase 2 IR and compiler, and passes through the same Phase 4
|
||||
tenant-scoping enforcement and cost guardrails as a hand-written query —
|
||||
no parallel execution path, no scoping shortcut, for either track. No AI
|
||||
code path anywhere constructs a `SQLRunner`, calls `executor.Execute`,
|
||||
or bypasses `authz.RequireRoleOrService`.
|
||||
|
||||
Every AI-assisted suggestion a user explicitly accepts or dismisses
|
||||
(translate/fix/optimize — deliberately not ghost-text completion or
|
||||
explain, see the design doc for why) is logged into the same
|
||||
append-only, hash-chained `audit_log` table Phase 4 built, via a new
|
||||
`event_type='ai_interaction'` rather than a new table
|
||||
(`metadata/migrations/0036`) — genuinely verified against a live
|
||||
Postgres in this environment, not just unit-tested against a fake, the
|
||||
same rigor Phase 4's own audit-logging guarantees were held to.
|
||||
|
||||
Two real product bugs were found and fixed via this phase's live
|
||||
browser verification — neither would have been caught by
|
||||
`svelte-check`/`npm run build` — and a real logic bug in the cost/safety
|
||||
guard itself (an unbounded-aggregation-vs-raw-row distinction) was found
|
||||
and fixed by the test suite written for it. Full accounting of all
|
||||
three: `/docs/phase-7-ai-design.md`. Integration tests
|
||||
(`api/ai/aiapi/integration_test.go`) wire a real `ollama.Client` through
|
||||
a real `router`/`Handler` against a mock server matching Ollama's actual
|
||||
wire contract (`hack/mock-ollama`, new — also used for this phase's live
|
||||
verification), proving the plumbing without needing real model weights;
|
||||
testing actual model *quality* is deliberately kept out of CI as a
|
||||
disclosed, periodic human-run checklist item instead — see the design
|
||||
doc's CI-testability section for the reasoning.
|
||||
|
||||
Explicit non-goals for this phase (scoped out, not deferred by oversight):
|
||||
result summarization, incident narrative generation, and proactive/
|
||||
unprompted AI suggestions — this phase is query authoring assistance
|
||||
only (structured and natural-language), not analysis or automation. Real
|
||||
future-phase candidates, not silently dropped.
|
||||
|
||||
See `/docs/phase-7-ai-design.md` for the model-provider architecture,
|
||||
shared foundation (schema grounding, cost/safety guard), both tracks'
|
||||
build-and-verification record, and the audit-logging/CI-testability
|
||||
design; `/docs/phase-7-runbook.md` for the step-by-step live-stack
|
||||
verification procedure.
|
||||
|
||||
## When in doubt
|
||||
Ask before: changing the pinned stack, adding a new external dependency
|
||||
that pulls in a large transitive tree, or making an architectural decision
|
||||
|
||||
@@ -0,0 +1,514 @@
|
||||
// Package aiapi is Track A's HTTP surface: complete, explain, fix, and
|
||||
// optimize, each a thin wrapper around api/ai/router dispatching to
|
||||
// whichever provider.Provider is configured for that operation. Mirrors
|
||||
// queryapi's shape deliberately (same auth wrapper, same request-size
|
||||
// cap, same error-response shape) since this is the same kind of
|
||||
// endpoint -- a JSON-in, JSON-out operation gated by the same RoleViewer
|
||||
// requirement /query uses, nothing AI-specific about the transport.
|
||||
//
|
||||
// What this package does NOT do: execute a query. Every operation here
|
||||
// returns text (a suggestion, an explanation, a fix) for the client to
|
||||
// review -- running anything still goes through the unchanged POST
|
||||
// /query endpoint (queryapi.Handler), never through here. See
|
||||
// /docs/phase-7-ai-design.md for why that split is load-bearing, not
|
||||
// incidental.
|
||||
package aiapi
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"log/slog"
|
||||
"net/http"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"github.com/sentry/sentry/api/ai/costguard"
|
||||
"github.com/sentry/sentry/api/ai/provider"
|
||||
"github.com/sentry/sentry/api/ai/router"
|
||||
"github.com/sentry/sentry/api/authz"
|
||||
"github.com/sentry/sentry/api/internal/querylang/ir"
|
||||
"github.com/sentry/sentry/api/internal/querylang/planner"
|
||||
)
|
||||
|
||||
// SchemaContextSource resolves the calling tenant's grounding data.
|
||||
// Core wires a tenant-agnostic adapter around one grounding.Service;
|
||||
// enterprise-api wires one around groundingregistry that reads the
|
||||
// tenant from request context -- same "interface in core, tenant-aware
|
||||
// implementation supplied by whoever constructs the handler" shape as
|
||||
// queryapi.AuditLogger and dashboards.PermissionStore.
|
||||
type SchemaContextSource interface {
|
||||
SchemaContext(ctx context.Context) provider.SchemaContext
|
||||
}
|
||||
|
||||
// InteractionLogger records a translate/fix/optimize suggestion's
|
||||
// accept-or-dismiss outcome into the Phase 4 audit trail (task 12) --
|
||||
// same nil-by-default, fail-open shape as queryapi.AuditLogger: a
|
||||
// single-tenant deployment with no enterprise/ configured just doesn't
|
||||
// log these, same as it doesn't log query executions today.
|
||||
// enterprise/internal/audit supplies the real implementation, writing
|
||||
// into the same append-only audit_log table query executions use
|
||||
// (a new event_type, not a new table -- see
|
||||
// metadata/migrations/0036_add_ai_interaction_event_type.sql).
|
||||
//
|
||||
// Deliberately not wired into Complete (ghost-text): that operation
|
||||
// fires on every keystroke pause, and logging each one at the same
|
||||
// weight as a deliberate Fix/Optimize/Translate review would drown the
|
||||
// signal task 12 actually wants (real accept/reject decisions) in
|
||||
// high-frequency noise. Not wired into Explain either -- it produces no
|
||||
// suggestion to accept or reject, so "accepted vs. rejected" doesn't
|
||||
// apply to it. Both are named scope boundaries, not oversights.
|
||||
type InteractionLogger interface {
|
||||
LogInteraction(ctx context.Context, entry InteractionEntry) error
|
||||
}
|
||||
|
||||
type InteractionEntry struct {
|
||||
// Operation is "translate", "fix", or "optimize" -- the three flows
|
||||
// that produce a suggestion a user explicitly accepts or dismisses.
|
||||
Operation string
|
||||
Input string
|
||||
Output string
|
||||
// Confidence is empty for fix/optimize (provider.Confidence only
|
||||
// applies to Translate/Fix results in a way the frontend surfaces
|
||||
// today -- Optimize's phrasing has no confidence concept).
|
||||
Confidence string
|
||||
// Accepted is false for a dismissed suggestion; Output/FinalQuery
|
||||
// still carry what was offered, since a rejected suggestion is
|
||||
// itself useful signal (task 12: "useful data for improving
|
||||
// grounding/prompting later").
|
||||
Accepted bool
|
||||
// Edited is only meaningful when Accepted -- did the user change
|
||||
// the suggested text before using it. False, not omitted, when
|
||||
// Accepted is false (there's nothing to have edited).
|
||||
Edited bool
|
||||
FinalQuery string
|
||||
}
|
||||
|
||||
// completeTimeout is deliberately tight -- task 5's "low enough latency
|
||||
// to feel responsive" requirement. A slow or hung provider must not
|
||||
// stall the query bar; the frontend's fallback to deterministic
|
||||
// autocomplete (Phase 2/5) kicks in on any error, including a timeout,
|
||||
// so a short timeout here fails fast toward that fallback rather than
|
||||
// making the user wait to find out AI completion isn't going to work
|
||||
// this time.
|
||||
const completeTimeout = 1500 * time.Millisecond
|
||||
|
||||
// Explain/Fix/Optimize are user-initiated (a button press, not
|
||||
// as-you-type), so a more generous budget is the right tradeoff --
|
||||
// correctness/quality over latency here, unlike Complete.
|
||||
const operationTimeout = 15 * time.Second
|
||||
|
||||
type Handler struct {
|
||||
logger *slog.Logger
|
||||
router *router.Router
|
||||
schema SchemaContextSource
|
||||
authz authz.Authorizer
|
||||
interactions InteractionLogger
|
||||
}
|
||||
|
||||
// interactions may be nil -- see InteractionLogger's doc comment.
|
||||
func NewHandler(logger *slog.Logger, r *router.Router, schema SchemaContextSource, authorizer authz.Authorizer, interactions InteractionLogger) *Handler {
|
||||
return &Handler{logger: logger, router: r, schema: schema, authz: authorizer, interactions: interactions}
|
||||
}
|
||||
|
||||
func (h *Handler) RegisterRoutes(mux *http.ServeMux) {
|
||||
mux.HandleFunc("POST /ai/complete", authz.RequireRoleOrService(h.authz, authz.RoleViewer, h.handleComplete))
|
||||
mux.HandleFunc("POST /ai/explain", authz.RequireRoleOrService(h.authz, authz.RoleViewer, h.handleExplain))
|
||||
mux.HandleFunc("POST /ai/fix", authz.RequireRoleOrService(h.authz, authz.RoleViewer, h.handleFix))
|
||||
mux.HandleFunc("POST /ai/optimize", authz.RequireRoleOrService(h.authz, authz.RoleViewer, h.handleOptimize))
|
||||
mux.HandleFunc("POST /ai/translate", authz.RequireRoleOrService(h.authz, authz.RoleViewer, h.handleTranslate))
|
||||
mux.HandleFunc("POST /ai/log-interaction", authz.RequireRoleOrService(h.authz, authz.RoleViewer, h.handleLogInteraction))
|
||||
}
|
||||
|
||||
const maxBodyBytes = 1 << 20 // 1 MiB, same cap queryapi uses -- these bodies are smaller still
|
||||
|
||||
type errorResponse struct {
|
||||
Error string `json:"error"`
|
||||
}
|
||||
|
||||
func writeJSON(w http.ResponseWriter, v any) {
|
||||
w.Header().Set("Content-Type", "application/json")
|
||||
_ = json.NewEncoder(w).Encode(v)
|
||||
}
|
||||
|
||||
func writeError(w http.ResponseWriter, status int, msg string) {
|
||||
w.Header().Set("Content-Type", "application/json")
|
||||
w.WriteHeader(status)
|
||||
_ = json.NewEncoder(w).Encode(errorResponse{Error: msg})
|
||||
}
|
||||
|
||||
func decodeBody(w http.ResponseWriter, r *http.Request, v any) bool {
|
||||
r.Body = http.MaxBytesReader(w, r.Body, maxBodyBytes)
|
||||
if err := json.NewDecoder(r.Body).Decode(v); err != nil {
|
||||
writeError(w, http.StatusBadRequest, "invalid JSON body: "+err.Error())
|
||||
return false
|
||||
}
|
||||
return true
|
||||
}
|
||||
|
||||
// ---- complete ----
|
||||
|
||||
type completeRequest struct {
|
||||
QueryPrefix string `json:"queryPrefix"`
|
||||
Language string `json:"language"`
|
||||
}
|
||||
|
||||
type completeResponse struct {
|
||||
Suggestion string `json:"suggestion"`
|
||||
}
|
||||
|
||||
func (h *Handler) handleComplete(w http.ResponseWriter, r *http.Request) {
|
||||
var req completeRequest
|
||||
if !decodeBody(w, r, &req) {
|
||||
return
|
||||
}
|
||||
if strings.TrimSpace(req.QueryPrefix) == "" {
|
||||
writeJSON(w, completeResponse{})
|
||||
return
|
||||
}
|
||||
|
||||
ctx, cancel := context.WithTimeout(r.Context(), completeTimeout)
|
||||
defer cancel()
|
||||
|
||||
result, err := h.router.For(router.OpComplete).Complete(ctx, provider.CompleteRequest{
|
||||
QueryPrefix: req.QueryPrefix,
|
||||
Language: orDefault(req.Language, "spl"),
|
||||
Schema: h.schema.SchemaContext(r.Context()),
|
||||
})
|
||||
if err != nil {
|
||||
// Complete's whole point is to degrade gracefully -- a failed or
|
||||
// slow completion is not worth a scary error response the query
|
||||
// bar has to handle specially. An empty suggestion is exactly
|
||||
// what "no good completion available right now" looks like to
|
||||
// the frontend's fallback logic (task 5).
|
||||
h.logger.Warn("ai complete failed", "error", err)
|
||||
writeJSON(w, completeResponse{})
|
||||
return
|
||||
}
|
||||
writeJSON(w, completeResponse{Suggestion: result.Suggestion})
|
||||
}
|
||||
|
||||
// ---- explain ----
|
||||
|
||||
type explainRequest struct {
|
||||
Query string `json:"query"`
|
||||
Language string `json:"language"`
|
||||
OriginalIntent string `json:"originalIntent"`
|
||||
}
|
||||
|
||||
type explainResponse struct {
|
||||
Explanation string `json:"explanation"`
|
||||
}
|
||||
|
||||
func (h *Handler) handleExplain(w http.ResponseWriter, r *http.Request) {
|
||||
var req explainRequest
|
||||
if !decodeBody(w, r, &req) {
|
||||
return
|
||||
}
|
||||
if strings.TrimSpace(req.Query) == "" {
|
||||
writeError(w, http.StatusBadRequest, "query must not be empty")
|
||||
return
|
||||
}
|
||||
|
||||
ctx, cancel := context.WithTimeout(r.Context(), operationTimeout)
|
||||
defer cancel()
|
||||
|
||||
result, err := h.router.For(router.OpExplain).Explain(ctx, provider.ExplainRequest{
|
||||
Query: req.Query,
|
||||
Language: orDefault(req.Language, "spl"),
|
||||
OriginalIntent: req.OriginalIntent,
|
||||
})
|
||||
if err != nil {
|
||||
writeError(w, http.StatusBadGateway, "explain failed: "+err.Error())
|
||||
return
|
||||
}
|
||||
writeJSON(w, explainResponse{Explanation: result.Explanation})
|
||||
}
|
||||
|
||||
// ---- fix ----
|
||||
|
||||
type fixRequest struct {
|
||||
Query string `json:"query"`
|
||||
Language string `json:"language"`
|
||||
ParseError string `json:"parseError"`
|
||||
ExecutionError string `json:"executionError"`
|
||||
}
|
||||
|
||||
type fixResponse struct {
|
||||
SuggestedQuery string `json:"suggestedQuery"`
|
||||
Explanation string `json:"explanation"`
|
||||
Confidence string `json:"confidence"`
|
||||
// Blocked mirrors costguard's assessment on the *suggested* query --
|
||||
// task 4's stricter AI-track treatment: a reject-level suggestion is
|
||||
// still shown (so the user understands what was tried and why it's
|
||||
// not being offered outright) but the frontend must not present a
|
||||
// plain accept-and-run action for it. CostWarnings is empty unless
|
||||
// Blocked, or the suggestion has a lesser (warn-level) concern worth
|
||||
// surfacing.
|
||||
Blocked bool `json:"blocked"`
|
||||
CostWarnings []string `json:"costWarnings,omitempty"`
|
||||
}
|
||||
|
||||
func (h *Handler) handleFix(w http.ResponseWriter, r *http.Request) {
|
||||
var req fixRequest
|
||||
if !decodeBody(w, r, &req) {
|
||||
return
|
||||
}
|
||||
if strings.TrimSpace(req.Query) == "" {
|
||||
writeError(w, http.StatusBadRequest, "query must not be empty")
|
||||
return
|
||||
}
|
||||
if req.ParseError == "" && req.ExecutionError == "" {
|
||||
writeError(w, http.StatusBadRequest, "parseError or executionError must be set")
|
||||
return
|
||||
}
|
||||
|
||||
ctx, cancel := context.WithTimeout(r.Context(), operationTimeout)
|
||||
defer cancel()
|
||||
|
||||
lang := orDefault(req.Language, "spl")
|
||||
result, err := h.router.For(router.OpFix).Fix(ctx, provider.FixRequest{
|
||||
Query: req.Query,
|
||||
Language: lang,
|
||||
ParseError: req.ParseError,
|
||||
ExecutionError: req.ExecutionError,
|
||||
Schema: h.schema.SchemaContext(r.Context()),
|
||||
})
|
||||
if err != nil {
|
||||
writeError(w, http.StatusBadGateway, "fix failed: "+err.Error())
|
||||
return
|
||||
}
|
||||
|
||||
resp := fixResponse{
|
||||
SuggestedQuery: result.SuggestedQuery,
|
||||
Explanation: result.Explanation,
|
||||
Confidence: string(result.Confidence),
|
||||
}
|
||||
if resp.SuggestedQuery != "" {
|
||||
if plan, err := planner.Compile(resp.SuggestedQuery, planner.Language(lang), time.Now()); err == nil {
|
||||
if assessment := costguard.Assess(plan); assessment.Level != costguard.LevelOK {
|
||||
resp.CostWarnings = assessment.Reasons
|
||||
resp.Blocked = assessment.Level == costguard.LevelReject
|
||||
}
|
||||
}
|
||||
// A suggested query that itself fails to compile is left
|
||||
// unassessed rather than treated as an error -- an unusual
|
||||
// outcome (the model produced something that doesn't parse) the
|
||||
// frontend can still show as a suggestion text, just without a
|
||||
// cost assessment attached to it.
|
||||
}
|
||||
writeJSON(w, resp)
|
||||
}
|
||||
|
||||
// ---- optimize ----
|
||||
|
||||
type optimizeRequest struct {
|
||||
Query string `json:"query"`
|
||||
Language string `json:"language"`
|
||||
}
|
||||
|
||||
type optimizeResponse struct {
|
||||
// Findings is always populated when costguard has anything to say --
|
||||
// rule-based, instant, no model call needed to produce this part.
|
||||
Findings []string `json:"findings"`
|
||||
// Phrased is the AI-phrased version of Findings (task 8: "AI layer
|
||||
// used mainly to phrase the suggestion clearly"). Empty if the
|
||||
// provider is unavailable or fails -- graceful degradation, same as
|
||||
// Complete: the raw Findings are still useful on their own, this is
|
||||
// an enhancement layered on top, not a dependency.
|
||||
Phrased string `json:"phrased"`
|
||||
// SuggestedQuery is a mechanical rewrite, not model-generated --
|
||||
// only populated for the one case this package can safely rewrite
|
||||
// unambiguously (a missing time range; see suggestFix below). Other
|
||||
// findings (an overly large time range, an unfiltered free-text
|
||||
// search) get text-only guidance, honestly, rather than a guessed
|
||||
// rewrite.
|
||||
SuggestedQuery string `json:"suggestedQuery,omitempty"`
|
||||
}
|
||||
|
||||
func (h *Handler) handleOptimize(w http.ResponseWriter, r *http.Request) {
|
||||
var req optimizeRequest
|
||||
if !decodeBody(w, r, &req) {
|
||||
return
|
||||
}
|
||||
if strings.TrimSpace(req.Query) == "" {
|
||||
writeError(w, http.StatusBadRequest, "query must not be empty")
|
||||
return
|
||||
}
|
||||
|
||||
lang := orDefault(req.Language, "spl")
|
||||
plan, err := planner.Compile(req.Query, planner.Language(lang), time.Now())
|
||||
if err != nil {
|
||||
writeError(w, http.StatusBadRequest, "query does not compile: "+err.Error())
|
||||
return
|
||||
}
|
||||
|
||||
assessment := costguard.Assess(plan)
|
||||
resp := optimizeResponse{Findings: assessment.Reasons}
|
||||
if assessment.Level == costguard.LevelOK {
|
||||
writeJSON(w, resp)
|
||||
return
|
||||
}
|
||||
|
||||
resp.SuggestedQuery = suggestMechanicalFix(req.Query, plan)
|
||||
|
||||
ctx, cancel := context.WithTimeout(r.Context(), operationTimeout)
|
||||
defer cancel()
|
||||
if result, err := h.router.For(router.OpExplain).Explain(ctx, provider.ExplainRequest{
|
||||
Query: req.Query,
|
||||
Language: lang,
|
||||
RuleFindings: assessment.Reasons,
|
||||
}); err != nil {
|
||||
h.logger.Warn("ai optimize phrasing failed", "error", err)
|
||||
} else {
|
||||
resp.Phrased = result.Explanation
|
||||
}
|
||||
writeJSON(w, resp)
|
||||
}
|
||||
|
||||
// ---- translate (Track B, task 9) ----
|
||||
|
||||
type translateRequest struct {
|
||||
NLQuery string `json:"nlQuery"`
|
||||
}
|
||||
|
||||
type translateResponse struct {
|
||||
Query string `json:"query"`
|
||||
Confidence string `json:"confidence"`
|
||||
LowConfidenceReason string `json:"lowConfidenceReason,omitempty"`
|
||||
// Compiles is false when Query is non-empty but doesn't actually
|
||||
// parse as pipe syntax -- a real, honest outcome (the model
|
||||
// produced something invalid), not folded into "low confidence"
|
||||
// since a model can be confident and still wrong about syntax.
|
||||
// CompileError is set only then.
|
||||
Compiles bool `json:"compiles"`
|
||||
CompileError string `json:"compileError,omitempty"`
|
||||
// Blocked/CostWarnings mirror handleFix's same-named fields exactly
|
||||
// -- task 9's explicit requirement that translation results run
|
||||
// through the shared cost guard "before returning it," same
|
||||
// treatment as an AI-suggested fix gets, not a lesser one.
|
||||
Blocked bool `json:"blocked"`
|
||||
CostWarnings []string `json:"costWarnings,omitempty"`
|
||||
}
|
||||
|
||||
func (h *Handler) handleTranslate(w http.ResponseWriter, r *http.Request) {
|
||||
var req translateRequest
|
||||
if !decodeBody(w, r, &req) {
|
||||
return
|
||||
}
|
||||
if strings.TrimSpace(req.NLQuery) == "" {
|
||||
writeError(w, http.StatusBadRequest, "nlQuery must not be empty")
|
||||
return
|
||||
}
|
||||
|
||||
ctx, cancel := context.WithTimeout(r.Context(), operationTimeout)
|
||||
defer cancel()
|
||||
|
||||
result, err := h.router.For(router.OpTranslate).Translate(ctx, provider.TranslateRequest{
|
||||
NLQuery: req.NLQuery,
|
||||
Schema: h.schema.SchemaContext(r.Context()),
|
||||
})
|
||||
if err != nil {
|
||||
writeError(w, http.StatusBadGateway, "translation failed: "+err.Error())
|
||||
return
|
||||
}
|
||||
|
||||
resp := translateResponse{
|
||||
Query: result.Query,
|
||||
Confidence: string(result.Confidence),
|
||||
LowConfidenceReason: result.LowConfidenceReason,
|
||||
}
|
||||
if resp.Query != "" {
|
||||
// Always pipe syntax -- provider.TranslateResult's own doc
|
||||
// comment requires this (task 9: "prefer this over raw SQL...
|
||||
// narrower, safer surface"), so this always compiles as SPL,
|
||||
// never auto-detected/SQL.
|
||||
plan, compileErr := planner.Compile(resp.Query, planner.SPL, time.Now())
|
||||
if compileErr != nil {
|
||||
resp.Compiles = false
|
||||
resp.CompileError = compileErr.Error()
|
||||
} else {
|
||||
resp.Compiles = true
|
||||
if assessment := costguard.Assess(plan); assessment.Level != costguard.LevelOK {
|
||||
resp.CostWarnings = assessment.Reasons
|
||||
resp.Blocked = assessment.Level == costguard.LevelReject
|
||||
}
|
||||
}
|
||||
}
|
||||
writeJSON(w, resp)
|
||||
}
|
||||
|
||||
// suggestMechanicalFix handles exactly one case: no time bound at all.
|
||||
// Prepending "earliest=-1h " is always syntactically safe (another
|
||||
// AND'd base-search filter term, same as any other) and semantically
|
||||
// the single most common real fix for this specific finding -- not
|
||||
// attempted for any other finding (a too-large span, an unindexed
|
||||
// free-text pattern), which don't have one unambiguous correct rewrite.
|
||||
// Checked against the plan directly (not against costguard's Reasons
|
||||
// text), so this stays correct even if that phrasing changes later.
|
||||
func suggestMechanicalFix(originalQuery string, plan *ir.Plan) string {
|
||||
if plan.RawSQL != "" {
|
||||
return "" // no safe generic rewrite for arbitrary SQL
|
||||
}
|
||||
hasTimeBound := plan.TimeRange != nil && (!plan.TimeRange.From.IsZero() || !plan.TimeRange.To.IsZero())
|
||||
if hasTimeBound {
|
||||
return ""
|
||||
}
|
||||
return "earliest=-1h " + originalQuery
|
||||
}
|
||||
|
||||
func orDefault(s, def string) string {
|
||||
if s == "" {
|
||||
return def
|
||||
}
|
||||
return s
|
||||
}
|
||||
|
||||
// ---- log-interaction (task 12) ----
|
||||
|
||||
type logInteractionRequest struct {
|
||||
Operation string `json:"operation"`
|
||||
Input string `json:"input"`
|
||||
Output string `json:"output"`
|
||||
Confidence string `json:"confidence"`
|
||||
Accepted bool `json:"accepted"`
|
||||
Edited bool `json:"edited"`
|
||||
FinalQuery string `json:"finalQuery"`
|
||||
}
|
||||
|
||||
var validInteractionOps = map[string]bool{"translate": true, "fix": true, "optimize": true}
|
||||
|
||||
// handleLogInteraction is called by the frontend at the moment a user
|
||||
// takes a terminal action on a suggestion (accept-and-use or dismiss) --
|
||||
// see InteractionLogger's doc comment for why this is a single
|
||||
// frontend-reported event rather than a backend-correlated
|
||||
// generation-plus-outcome pair. Fail-open, same posture
|
||||
// queryapi.Handler.logAudit uses: a write failure here is logged
|
||||
// server-side and otherwise ignored, never surfaced as an error to a
|
||||
// user who just clicked a button -- audit-trail completeness is a real
|
||||
// requirement, but it shouldn't be able to break the query bar.
|
||||
func (h *Handler) handleLogInteraction(w http.ResponseWriter, r *http.Request) {
|
||||
var req logInteractionRequest
|
||||
if !decodeBody(w, r, &req) {
|
||||
return
|
||||
}
|
||||
if !validInteractionOps[req.Operation] {
|
||||
writeError(w, http.StatusBadRequest, `operation must be one of "translate", "fix", "optimize"`)
|
||||
return
|
||||
}
|
||||
|
||||
if h.interactions != nil {
|
||||
err := h.interactions.LogInteraction(r.Context(), InteractionEntry{
|
||||
Operation: req.Operation,
|
||||
Input: req.Input,
|
||||
Output: req.Output,
|
||||
Confidence: req.Confidence,
|
||||
Accepted: req.Accepted,
|
||||
Edited: req.Edited,
|
||||
FinalQuery: req.FinalQuery,
|
||||
})
|
||||
if err != nil {
|
||||
h.logger.Error("ai interaction audit log write failed", "error", err)
|
||||
}
|
||||
}
|
||||
w.WriteHeader(http.StatusNoContent)
|
||||
}
|
||||
@@ -0,0 +1,362 @@
|
||||
package aiapi
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"log/slog"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"testing"
|
||||
|
||||
"github.com/sentry/sentry/api/ai/provider"
|
||||
"github.com/sentry/sentry/api/ai/router"
|
||||
)
|
||||
|
||||
type fakeProvider struct {
|
||||
translateResult provider.TranslateResult
|
||||
completeResult provider.CompleteResult
|
||||
explainResult provider.ExplainResult
|
||||
fixResult provider.FixResult
|
||||
err error
|
||||
|
||||
gotExplainReq provider.ExplainRequest
|
||||
}
|
||||
|
||||
func (f *fakeProvider) Translate(context.Context, provider.TranslateRequest) (provider.TranslateResult, error) {
|
||||
return f.translateResult, f.err
|
||||
}
|
||||
func (f *fakeProvider) Complete(context.Context, provider.CompleteRequest) (provider.CompleteResult, error) {
|
||||
return f.completeResult, f.err
|
||||
}
|
||||
func (f *fakeProvider) Explain(_ context.Context, req provider.ExplainRequest) (provider.ExplainResult, error) {
|
||||
f.gotExplainReq = req
|
||||
return f.explainResult, f.err
|
||||
}
|
||||
func (f *fakeProvider) Fix(context.Context, provider.FixRequest) (provider.FixResult, error) {
|
||||
return f.fixResult, f.err
|
||||
}
|
||||
|
||||
type fakeSchemaSource struct{}
|
||||
|
||||
func (fakeSchemaSource) SchemaContext(context.Context) provider.SchemaContext {
|
||||
return provider.SchemaContext{Services: []string{"api"}}
|
||||
}
|
||||
|
||||
func newTestHandler(p *fakeProvider) *Handler {
|
||||
r := router.New(p)
|
||||
logger := slog.New(slog.NewTextHandler(bytesDiscard{}, nil))
|
||||
return NewHandler(logger, r, fakeSchemaSource{}, nil, nil)
|
||||
}
|
||||
|
||||
type bytesDiscard struct{}
|
||||
|
||||
func (bytesDiscard) Write(p []byte) (int, error) { return len(p), nil }
|
||||
|
||||
func doRequest(t *testing.T, h *Handler, method, path string, body any) *httptest.ResponseRecorder {
|
||||
t.Helper()
|
||||
var buf bytes.Buffer
|
||||
if body != nil {
|
||||
if err := json.NewEncoder(&buf).Encode(body); err != nil {
|
||||
t.Fatalf("encoding request body: %v", err)
|
||||
}
|
||||
}
|
||||
req := httptest.NewRequest(method, path, &buf)
|
||||
mux := http.NewServeMux()
|
||||
h.RegisterRoutes(mux)
|
||||
rec := httptest.NewRecorder()
|
||||
mux.ServeHTTP(rec, req)
|
||||
return rec
|
||||
}
|
||||
|
||||
func TestHandleCompleteReturnsSuggestion(t *testing.T) {
|
||||
p := &fakeProvider{completeResult: provider.CompleteResult{Suggestion: " | stats count"}}
|
||||
h := newTestHandler(p)
|
||||
|
||||
rec := doRequest(t, h, "POST", "/ai/complete", completeRequest{QueryPrefix: "service=api"})
|
||||
if rec.Code != http.StatusOK {
|
||||
t.Fatalf("status = %d, body = %s", rec.Code, rec.Body.String())
|
||||
}
|
||||
var resp completeResponse
|
||||
if err := json.Unmarshal(rec.Body.Bytes(), &resp); err != nil {
|
||||
t.Fatalf("decoding response: %v", err)
|
||||
}
|
||||
if resp.Suggestion != " | stats count" {
|
||||
t.Errorf("Suggestion = %q", resp.Suggestion)
|
||||
}
|
||||
}
|
||||
|
||||
func TestHandleCompleteDegradesGracefullyOnProviderError(t *testing.T) {
|
||||
p := &fakeProvider{err: errors.New("provider down")}
|
||||
h := newTestHandler(p)
|
||||
|
||||
rec := doRequest(t, h, "POST", "/ai/complete", completeRequest{QueryPrefix: "service=api"})
|
||||
if rec.Code != http.StatusOK {
|
||||
t.Fatalf("status = %d, want 200 even on provider failure (graceful degradation), body = %s", rec.Code, rec.Body.String())
|
||||
}
|
||||
var resp completeResponse
|
||||
_ = json.Unmarshal(rec.Body.Bytes(), &resp)
|
||||
if resp.Suggestion != "" {
|
||||
t.Errorf("Suggestion = %q, want empty on provider failure", resp.Suggestion)
|
||||
}
|
||||
}
|
||||
|
||||
func TestHandleCompleteEmptyPrefixSkipsProviderCall(t *testing.T) {
|
||||
p := &fakeProvider{err: errors.New("should not be called")}
|
||||
h := newTestHandler(p)
|
||||
|
||||
rec := doRequest(t, h, "POST", "/ai/complete", completeRequest{QueryPrefix: " "})
|
||||
if rec.Code != http.StatusOK {
|
||||
t.Fatalf("status = %d, body = %s", rec.Code, rec.Body.String())
|
||||
}
|
||||
}
|
||||
|
||||
func TestHandleExplainReturnsExplanation(t *testing.T) {
|
||||
p := &fakeProvider{explainResult: provider.ExplainResult{Explanation: "counts errors per host"}}
|
||||
h := newTestHandler(p)
|
||||
|
||||
rec := doRequest(t, h, "POST", "/ai/explain", explainRequest{Query: "severity=ERROR | stats count by host"})
|
||||
if rec.Code != http.StatusOK {
|
||||
t.Fatalf("status = %d, body = %s", rec.Code, rec.Body.String())
|
||||
}
|
||||
var resp explainResponse
|
||||
_ = json.Unmarshal(rec.Body.Bytes(), &resp)
|
||||
if resp.Explanation == "" {
|
||||
t.Error("expected a non-empty explanation")
|
||||
}
|
||||
}
|
||||
|
||||
func TestHandleExplainEmptyQueryIsBadRequest(t *testing.T) {
|
||||
h := newTestHandler(&fakeProvider{})
|
||||
rec := doRequest(t, h, "POST", "/ai/explain", explainRequest{Query: ""})
|
||||
if rec.Code != http.StatusBadRequest {
|
||||
t.Errorf("status = %d, want 400", rec.Code)
|
||||
}
|
||||
}
|
||||
|
||||
func TestHandleExplainProviderErrorIsBadGateway(t *testing.T) {
|
||||
p := &fakeProvider{err: errors.New("model unavailable")}
|
||||
h := newTestHandler(p)
|
||||
rec := doRequest(t, h, "POST", "/ai/explain", explainRequest{Query: "service=api"})
|
||||
if rec.Code != http.StatusBadGateway {
|
||||
t.Errorf("status = %d, want 502", rec.Code)
|
||||
}
|
||||
}
|
||||
|
||||
func TestHandleFixMissingErrorFieldsIsBadRequest(t *testing.T) {
|
||||
h := newTestHandler(&fakeProvider{})
|
||||
rec := doRequest(t, h, "POST", "/ai/fix", fixRequest{Query: "service=api"})
|
||||
if rec.Code != http.StatusBadRequest {
|
||||
t.Errorf("status = %d, want 400 (neither parseError nor executionError set)", rec.Code)
|
||||
}
|
||||
}
|
||||
|
||||
func TestHandleFixAssessesSuggestedQueryCost(t *testing.T) {
|
||||
// The provider suggests a fix that aggregates with no time bound --
|
||||
// costguard should reject it (an aggregation gets no implicit row
|
||||
// cap the way a raw-row fetch does), and the handler must mark it
|
||||
// Blocked rather than silently offering it as runnable. Needs a real
|
||||
// leading pipe stage -- bare words with no "|" parse as free-text
|
||||
// search terms, not an aggregation, per the query grammar.
|
||||
p := &fakeProvider{fixResult: provider.FixResult{
|
||||
SuggestedQuery: "service=api | stats count by host",
|
||||
Explanation: "removed the invalid field reference",
|
||||
Confidence: provider.ConfidenceHigh,
|
||||
}}
|
||||
h := newTestHandler(p)
|
||||
|
||||
rec := doRequest(t, h, "POST", "/ai/fix", fixRequest{Query: "bogus_field=1 | stats count by host", ParseError: "unknown field"})
|
||||
if rec.Code != http.StatusOK {
|
||||
t.Fatalf("status = %d, body = %s", rec.Code, rec.Body.String())
|
||||
}
|
||||
var resp fixResponse
|
||||
_ = json.Unmarshal(rec.Body.Bytes(), &resp)
|
||||
if !resp.Blocked {
|
||||
t.Errorf("resp = %+v, want Blocked=true for an unbounded aggregation suggestion", resp)
|
||||
}
|
||||
if len(resp.CostWarnings) == 0 {
|
||||
t.Error("expected non-empty CostWarnings")
|
||||
}
|
||||
}
|
||||
|
||||
func TestHandleFixBoundedSuggestionIsNotBlocked(t *testing.T) {
|
||||
p := &fakeProvider{fixResult: provider.FixResult{
|
||||
SuggestedQuery: "earliest=-1h | stats count by host",
|
||||
Confidence: provider.ConfidenceHigh,
|
||||
}}
|
||||
h := newTestHandler(p)
|
||||
|
||||
rec := doRequest(t, h, "POST", "/ai/fix", fixRequest{Query: "stats count by host", ParseError: "no time range"})
|
||||
var resp fixResponse
|
||||
_ = json.Unmarshal(rec.Body.Bytes(), &resp)
|
||||
if resp.Blocked {
|
||||
t.Errorf("resp = %+v, want Blocked=false for a properly time-bounded suggestion", resp)
|
||||
}
|
||||
}
|
||||
|
||||
func TestHandleOptimizeNoFindingsForBoundedQuery(t *testing.T) {
|
||||
h := newTestHandler(&fakeProvider{})
|
||||
rec := doRequest(t, h, "POST", "/ai/optimize", optimizeRequest{Query: "earliest=-1h severity=ERROR | stats count by host"})
|
||||
if rec.Code != http.StatusOK {
|
||||
t.Fatalf("status = %d, body = %s", rec.Code, rec.Body.String())
|
||||
}
|
||||
var resp optimizeResponse
|
||||
_ = json.Unmarshal(rec.Body.Bytes(), &resp)
|
||||
if len(resp.Findings) != 0 || resp.Phrased != "" {
|
||||
t.Errorf("resp = %+v, want no findings for a bounded query", resp)
|
||||
}
|
||||
}
|
||||
|
||||
func TestHandleOptimizeSuggestsMechanicalFixForMissingTimeRange(t *testing.T) {
|
||||
p := &fakeProvider{explainResult: provider.ExplainResult{Explanation: "add a time range to avoid scanning everything"}}
|
||||
h := newTestHandler(p)
|
||||
|
||||
rec := doRequest(t, h, "POST", "/ai/optimize", optimizeRequest{Query: "stats count by host"})
|
||||
if rec.Code != http.StatusOK {
|
||||
t.Fatalf("status = %d, body = %s", rec.Code, rec.Body.String())
|
||||
}
|
||||
var resp optimizeResponse
|
||||
_ = json.Unmarshal(rec.Body.Bytes(), &resp)
|
||||
if len(resp.Findings) == 0 {
|
||||
t.Error("expected at least one finding for an unbounded aggregation")
|
||||
}
|
||||
if resp.SuggestedQuery != "earliest=-1h stats count by host" {
|
||||
t.Errorf("SuggestedQuery = %q", resp.SuggestedQuery)
|
||||
}
|
||||
if resp.Phrased == "" {
|
||||
t.Error("expected a phrased explanation from the (fake) provider")
|
||||
}
|
||||
if len(p.gotExplainReq.RuleFindings) == 0 {
|
||||
t.Error("expected Explain to have been called with RuleFindings set")
|
||||
}
|
||||
}
|
||||
|
||||
func TestHandleOptimizeDegradesGracefullyWhenPhraseFails(t *testing.T) {
|
||||
p := &fakeProvider{err: errors.New("model down")}
|
||||
h := newTestHandler(p)
|
||||
|
||||
rec := doRequest(t, h, "POST", "/ai/optimize", optimizeRequest{Query: "stats count by host"})
|
||||
if rec.Code != http.StatusOK {
|
||||
t.Fatalf("status = %d, want 200 even when phrasing fails, body = %s", rec.Code, rec.Body.String())
|
||||
}
|
||||
var resp optimizeResponse
|
||||
_ = json.Unmarshal(rec.Body.Bytes(), &resp)
|
||||
if len(resp.Findings) == 0 {
|
||||
t.Error("Findings should still be populated (rule-based, no model needed) even if phrasing fails")
|
||||
}
|
||||
if resp.Phrased != "" {
|
||||
t.Errorf("Phrased = %q, want empty when the provider fails", resp.Phrased)
|
||||
}
|
||||
}
|
||||
|
||||
func TestHandleOptimizeInvalidQueryIsBadRequest(t *testing.T) {
|
||||
h := newTestHandler(&fakeProvider{})
|
||||
rec := doRequest(t, h, "POST", "/ai/optimize", optimizeRequest{Query: "| stats"})
|
||||
if rec.Code != http.StatusBadRequest {
|
||||
t.Errorf("status = %d, want 400 for an uncompilable query", rec.Code)
|
||||
}
|
||||
}
|
||||
|
||||
// ---- translate ----
|
||||
|
||||
func TestHandleTranslateEmptyNLQueryIsBadRequest(t *testing.T) {
|
||||
h := newTestHandler(&fakeProvider{})
|
||||
rec := doRequest(t, h, "POST", "/ai/translate", translateRequest{NLQuery: " "})
|
||||
if rec.Code != http.StatusBadRequest {
|
||||
t.Errorf("status = %d, want 400", rec.Code)
|
||||
}
|
||||
}
|
||||
|
||||
func TestHandleTranslateProviderErrorIsBadGateway(t *testing.T) {
|
||||
p := &fakeProvider{err: errors.New("model unavailable")}
|
||||
h := newTestHandler(p)
|
||||
rec := doRequest(t, h, "POST", "/ai/translate", translateRequest{NLQuery: "errors in the last hour"})
|
||||
if rec.Code != http.StatusBadGateway {
|
||||
t.Errorf("status = %d, want 502", rec.Code)
|
||||
}
|
||||
}
|
||||
|
||||
func TestHandleTranslateBoundedQueryCompilesCleanly(t *testing.T) {
|
||||
p := &fakeProvider{translateResult: provider.TranslateResult{
|
||||
Query: "earliest=-1h severity=ERROR | stats count by service",
|
||||
Confidence: provider.ConfidenceHigh,
|
||||
}}
|
||||
h := newTestHandler(p)
|
||||
rec := doRequest(t, h, "POST", "/ai/translate", translateRequest{NLQuery: "errors per service in the last hour"})
|
||||
if rec.Code != http.StatusOK {
|
||||
t.Fatalf("status = %d, body = %s", rec.Code, rec.Body.String())
|
||||
}
|
||||
var resp translateResponse
|
||||
_ = json.Unmarshal(rec.Body.Bytes(), &resp)
|
||||
if !resp.Compiles || resp.CompileError != "" {
|
||||
t.Errorf("resp = %+v, want Compiles=true, no CompileError", resp)
|
||||
}
|
||||
if resp.Blocked || len(resp.CostWarnings) != 0 {
|
||||
t.Errorf("resp = %+v, want no cost warnings for a time-bounded query", resp)
|
||||
}
|
||||
if resp.Confidence != string(provider.ConfidenceHigh) {
|
||||
t.Errorf("Confidence = %q", resp.Confidence)
|
||||
}
|
||||
}
|
||||
|
||||
func TestHandleTranslateUnboundedQueryIsBlocked(t *testing.T) {
|
||||
// The model produced a syntactically valid but unbounded aggregation
|
||||
// -- task 9's explicit requirement that translation results run
|
||||
// through the same cost guard AI-suggested fixes do.
|
||||
p := &fakeProvider{translateResult: provider.TranslateResult{
|
||||
Query: "severity=ERROR | stats count by service",
|
||||
Confidence: provider.ConfidenceHigh,
|
||||
}}
|
||||
h := newTestHandler(p)
|
||||
rec := doRequest(t, h, "POST", "/ai/translate", translateRequest{NLQuery: "errors by service"})
|
||||
var resp translateResponse
|
||||
_ = json.Unmarshal(rec.Body.Bytes(), &resp)
|
||||
if !resp.Compiles {
|
||||
t.Fatalf("resp = %+v, want Compiles=true", resp)
|
||||
}
|
||||
if !resp.Blocked || len(resp.CostWarnings) == 0 {
|
||||
t.Errorf("resp = %+v, want Blocked=true with cost warnings for an unbounded aggregation", resp)
|
||||
}
|
||||
}
|
||||
|
||||
func TestHandleTranslateNonCompilingQueryIsHonestlyReported(t *testing.T) {
|
||||
// The model returned something that doesn't actually parse -- a
|
||||
// real, distinct outcome from low confidence (a confident model can
|
||||
// still produce invalid syntax); the handler must say so plainly,
|
||||
// not silently drop it or crash.
|
||||
p := &fakeProvider{translateResult: provider.TranslateResult{
|
||||
Query: "| stats count",
|
||||
Confidence: provider.ConfidenceHigh,
|
||||
}}
|
||||
h := newTestHandler(p)
|
||||
rec := doRequest(t, h, "POST", "/ai/translate", translateRequest{NLQuery: "something odd"})
|
||||
if rec.Code != http.StatusOK {
|
||||
t.Fatalf("status = %d, want 200 (a non-compiling suggestion is a reportable outcome, not an HTTP error), body = %s", rec.Code, rec.Body.String())
|
||||
}
|
||||
var resp translateResponse
|
||||
_ = json.Unmarshal(rec.Body.Bytes(), &resp)
|
||||
if resp.Compiles || resp.CompileError == "" {
|
||||
t.Errorf("resp = %+v, want Compiles=false with a CompileError", resp)
|
||||
}
|
||||
}
|
||||
|
||||
func TestHandleTranslateLowConfidenceCarriesReason(t *testing.T) {
|
||||
p := &fakeProvider{translateResult: provider.TranslateResult{
|
||||
Confidence: provider.ConfidenceLow,
|
||||
LowConfidenceReason: "not sure what 'weird stuff' refers to",
|
||||
}}
|
||||
h := newTestHandler(p)
|
||||
rec := doRequest(t, h, "POST", "/ai/translate", translateRequest{NLQuery: "show me weird stuff"})
|
||||
var resp translateResponse
|
||||
_ = json.Unmarshal(rec.Body.Bytes(), &resp)
|
||||
if resp.Query != "" {
|
||||
t.Errorf("Query = %q, want empty for a low-confidence non-answer", resp.Query)
|
||||
}
|
||||
if resp.Confidence != string(provider.ConfidenceLow) || resp.LowConfidenceReason == "" {
|
||||
t.Errorf("resp = %+v, want low confidence with a reason", resp)
|
||||
}
|
||||
if resp.Compiles {
|
||||
t.Errorf("resp = %+v, want Compiles=false when Query is empty", resp)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,256 @@
|
||||
// Phase 7 task 13: end-to-end wiring tests distinct from handler_test.go
|
||||
// and ai/provider/ollama's own tests. Those two files each cover one
|
||||
// layer in isolation -- handler_test.go's fakeProvider satisfies
|
||||
// provider.Provider directly, bypassing HTTP/JSON/prompt construction
|
||||
// entirely; ollama_test.go exercises ollama.Client's wire-format parsing
|
||||
// against a stub server, but never through aiapi.Handler's actual HTTP
|
||||
// routes. Neither proves the seam between them actually works: a real
|
||||
// *ollama.Client wired through *router.Router into a real *Handler,
|
||||
// driven by real HTTP requests against the registered routes, with real
|
||||
// planner.Compile/costguard.Assess in the loop.
|
||||
//
|
||||
// No live Ollama or model is needed or used -- mockOllamaServer stands
|
||||
// in for Ollama's real /api/chat wire contract (same technique used for
|
||||
// this phase's live browser verification, see /docs/phase-7-ai-design.md,
|
||||
// just returning a fixed canned response instead of one selected by
|
||||
// inspecting the prompt) with a deterministic canned JSON body, which is
|
||||
// exactly what makes this suite fast and safe to run in CI -- see the
|
||||
// "CI testability" section of the design doc for why testing against a
|
||||
// real model is deliberately kept out of this suite instead.
|
||||
package aiapi
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"encoding/json"
|
||||
"io"
|
||||
"log/slog"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"testing"
|
||||
|
||||
"github.com/sentry/sentry/api/ai/provider/ollama"
|
||||
"github.com/sentry/sentry/api/ai/router"
|
||||
)
|
||||
|
||||
// jsonBody marshals v for use as an http.Post body -- the integration
|
||||
// tests below drive real HTTP requests against a real httptest.Server
|
||||
// (not handler_test.go's doRequest/ResponseRecorder shortcut), since the
|
||||
// point of this file is proving the routes are actually reachable over
|
||||
// real HTTP, not just that Handler's methods dispatch correctly.
|
||||
func jsonBody(t *testing.T, v any) io.Reader {
|
||||
t.Helper()
|
||||
b, err := json.Marshal(v)
|
||||
if err != nil {
|
||||
t.Fatalf("marshaling request body: %v", err)
|
||||
}
|
||||
return bytes.NewReader(b)
|
||||
}
|
||||
|
||||
type fakeInteractionLogger struct {
|
||||
entries []InteractionEntry
|
||||
}
|
||||
|
||||
func (f *fakeInteractionLogger) LogInteraction(_ context.Context, entry InteractionEntry) error {
|
||||
f.entries = append(f.entries, entry)
|
||||
return nil
|
||||
}
|
||||
|
||||
// mockOllamaServer returns an httptest.Server that answers any
|
||||
// POST /api/chat with the given assistant message content, matching
|
||||
// Ollama's real response envelope shape byte-for-byte (see
|
||||
// ollama.go's chatResponse).
|
||||
func mockOllamaServer(t *testing.T, content string) *httptest.Server {
|
||||
t.Helper()
|
||||
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
if r.URL.Path != "/api/chat" {
|
||||
t.Errorf("unexpected request to %s, want /api/chat", r.URL.Path)
|
||||
}
|
||||
w.Header().Set("Content-Type", "application/json")
|
||||
_ = json.NewEncoder(w).Encode(map[string]any{
|
||||
"message": map[string]string{"role": "assistant", "content": content},
|
||||
})
|
||||
}))
|
||||
t.Cleanup(srv.Close)
|
||||
return srv
|
||||
}
|
||||
|
||||
func newIntegrationHandler(t *testing.T, ollamaContent string) *Handler {
|
||||
t.Helper()
|
||||
mock := mockOllamaServer(t, ollamaContent)
|
||||
client := ollama.New(mock.URL, "test-model")
|
||||
r := router.New(client)
|
||||
logger := slog.New(slog.NewTextHandler(bytesDiscard{}, nil))
|
||||
return NewHandler(logger, r, fakeSchemaSource{}, nil, nil)
|
||||
}
|
||||
|
||||
// TestIntegrationTranslateEndToEnd proves the full path -- HTTP request
|
||||
// in, real ollama.Client HTTP call out to the mock, real JSON parsing,
|
||||
// real planner.Compile, real costguard.Assess, HTTP response out --
|
||||
// works for a query that should pass cleanly (time-bounded, no
|
||||
// aggregation).
|
||||
func TestIntegrationTranslateEndToEnd(t *testing.T) {
|
||||
h := newIntegrationHandler(t, `{"query":"earliest=-1h severity=ERROR","confidence":"high"}`)
|
||||
mux := http.NewServeMux()
|
||||
h.RegisterRoutes(mux)
|
||||
srv := httptest.NewServer(mux)
|
||||
defer srv.Close()
|
||||
|
||||
resp, err := http.Post(srv.URL+"/ai/translate", "application/json",
|
||||
jsonBody(t, map[string]string{"nlQuery": "errors in the last hour"}))
|
||||
if err != nil {
|
||||
t.Fatalf("POST /ai/translate: %v", err)
|
||||
}
|
||||
defer resp.Body.Close()
|
||||
if resp.StatusCode != http.StatusOK {
|
||||
t.Fatalf("status = %d, want 200", resp.StatusCode)
|
||||
}
|
||||
|
||||
var got translateResponse
|
||||
if err := json.NewDecoder(resp.Body).Decode(&got); err != nil {
|
||||
t.Fatalf("decoding response: %v", err)
|
||||
}
|
||||
if got.Query != "earliest=-1h severity=ERROR" {
|
||||
t.Errorf("query = %q", got.Query)
|
||||
}
|
||||
if !got.Compiles {
|
||||
t.Error("compiles = false, want true -- this query is valid pipe syntax")
|
||||
}
|
||||
if got.Blocked {
|
||||
t.Errorf("blocked = true, want false: %v", got.CostWarnings)
|
||||
}
|
||||
if got.Confidence != "high" {
|
||||
t.Errorf("confidence = %q, want high", got.Confidence)
|
||||
}
|
||||
}
|
||||
|
||||
// TestIntegrationTranslateBlockedByCostGuard proves costguard is
|
||||
// actually reached through the full HTTP stack, not just unit-tested
|
||||
// against costguard.Assess in isolation -- an unbounded aggregation
|
||||
// (stats, no time filter) must come back Blocked.
|
||||
func TestIntegrationTranslateBlockedByCostGuard(t *testing.T) {
|
||||
h := newIntegrationHandler(t, `{"query":"severity=ERROR | stats count by service","confidence":"high"}`)
|
||||
mux := http.NewServeMux()
|
||||
h.RegisterRoutes(mux)
|
||||
srv := httptest.NewServer(mux)
|
||||
defer srv.Close()
|
||||
|
||||
resp, err := http.Post(srv.URL+"/ai/translate", "application/json",
|
||||
jsonBody(t, map[string]string{"nlQuery": "error count by service"}))
|
||||
if err != nil {
|
||||
t.Fatalf("POST /ai/translate: %v", err)
|
||||
}
|
||||
defer resp.Body.Close()
|
||||
|
||||
var got translateResponse
|
||||
if err := json.NewDecoder(resp.Body).Decode(&got); err != nil {
|
||||
t.Fatalf("decoding response: %v", err)
|
||||
}
|
||||
if !got.Compiles {
|
||||
t.Fatalf("compiles = false, want true: %s", got.CompileError)
|
||||
}
|
||||
if !got.Blocked {
|
||||
t.Error("blocked = false, want true -- unbounded aggregation should be rejected by costguard")
|
||||
}
|
||||
if len(got.CostWarnings) == 0 {
|
||||
t.Error("costWarnings is empty, want at least one reason")
|
||||
}
|
||||
}
|
||||
|
||||
// TestIntegrationFixEndToEnd proves the same seam for /ai/fix.
|
||||
func TestIntegrationFixEndToEnd(t *testing.T) {
|
||||
h := newIntegrationHandler(t, `{"suggested_query":"earliest=-1h severity=ERROR","explanation":"added a time bound","confidence":"high"}`)
|
||||
mux := http.NewServeMux()
|
||||
h.RegisterRoutes(mux)
|
||||
srv := httptest.NewServer(mux)
|
||||
defer srv.Close()
|
||||
|
||||
resp, err := http.Post(srv.URL+"/ai/fix", "application/json", jsonBody(t, map[string]string{
|
||||
"query": "severity=ERROR",
|
||||
"language": "spl",
|
||||
"executionError": "query timed out: no time range specified",
|
||||
}))
|
||||
if err != nil {
|
||||
t.Fatalf("POST /ai/fix: %v", err)
|
||||
}
|
||||
defer resp.Body.Close()
|
||||
if resp.StatusCode != http.StatusOK {
|
||||
t.Fatalf("status = %d, want 200", resp.StatusCode)
|
||||
}
|
||||
|
||||
var got fixResponse
|
||||
if err := json.NewDecoder(resp.Body).Decode(&got); err != nil {
|
||||
t.Fatalf("decoding response: %v", err)
|
||||
}
|
||||
if got.SuggestedQuery != "earliest=-1h severity=ERROR" {
|
||||
t.Errorf("suggestedQuery = %q", got.SuggestedQuery)
|
||||
}
|
||||
if got.Blocked {
|
||||
t.Errorf("blocked = true, want false: %v", got.CostWarnings)
|
||||
}
|
||||
}
|
||||
|
||||
// TestIntegrationCompleteEndToEnd proves the same seam for /ai/complete
|
||||
// (Track A's ghost-text autocomplete).
|
||||
func TestIntegrationCompleteEndToEnd(t *testing.T) {
|
||||
h := newIntegrationHandler(t, `{"suggestion":" severity=ERROR"}`)
|
||||
mux := http.NewServeMux()
|
||||
h.RegisterRoutes(mux)
|
||||
srv := httptest.NewServer(mux)
|
||||
defer srv.Close()
|
||||
|
||||
resp, err := http.Post(srv.URL+"/ai/complete", "application/json", jsonBody(t, map[string]string{
|
||||
"queryPrefix": "service=api ",
|
||||
"language": "spl",
|
||||
}))
|
||||
if err != nil {
|
||||
t.Fatalf("POST /ai/complete: %v", err)
|
||||
}
|
||||
defer resp.Body.Close()
|
||||
|
||||
var got completeResponse
|
||||
if err := json.NewDecoder(resp.Body).Decode(&got); err != nil {
|
||||
t.Fatalf("decoding response: %v", err)
|
||||
}
|
||||
if got.Suggestion != " severity=ERROR" {
|
||||
t.Errorf("suggestion = %q", got.Suggestion)
|
||||
}
|
||||
}
|
||||
|
||||
// TestIntegrationLogInteractionEndToEnd proves /ai/log-interaction's
|
||||
// full HTTP decode+validate+dispatch path, using a fake InteractionLogger
|
||||
// (an in-process Go fake is the right seam here, not another mock HTTP
|
||||
// server -- the real implementation is enterprise/internal/audit, which
|
||||
// needs a live Postgres and is covered by that package's own tests).
|
||||
func TestIntegrationLogInteractionEndToEnd(t *testing.T) {
|
||||
logger := &fakeInteractionLogger{}
|
||||
r := router.New(&fakeProvider{})
|
||||
h := NewHandler(slog.New(slog.NewTextHandler(bytesDiscard{}, nil)), r, fakeSchemaSource{}, nil, logger)
|
||||
mux := http.NewServeMux()
|
||||
h.RegisterRoutes(mux)
|
||||
srv := httptest.NewServer(mux)
|
||||
defer srv.Close()
|
||||
|
||||
resp, err := http.Post(srv.URL+"/ai/log-interaction", "application/json", jsonBody(t, map[string]any{
|
||||
"operation": "translate",
|
||||
"input": "errors in the last hour",
|
||||
"output": "earliest=-1h severity=ERROR",
|
||||
"confidence": "high",
|
||||
"accepted": true,
|
||||
"edited": false,
|
||||
"finalQuery": "earliest=-1h severity=ERROR",
|
||||
}))
|
||||
if err != nil {
|
||||
t.Fatalf("POST /ai/log-interaction: %v", err)
|
||||
}
|
||||
defer resp.Body.Close()
|
||||
if resp.StatusCode != http.StatusNoContent {
|
||||
t.Fatalf("status = %d, want 204", resp.StatusCode)
|
||||
}
|
||||
if len(logger.entries) != 1 {
|
||||
t.Fatalf("got %d logged entries, want 1", len(logger.entries))
|
||||
}
|
||||
if logger.entries[0].Operation != "translate" || !logger.entries[0].Accepted {
|
||||
t.Errorf("logged entry = %+v", logger.entries[0])
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,141 @@
|
||||
// Package costguard is the shared cost/safety check task 4 asked for --
|
||||
// no such mechanism existed anywhere in Phase 2/3's compiler before this
|
||||
// (confirmed by reading planner.go/sql.go before writing this: plan.TimeRange
|
||||
// can be entirely unset, and nothing downstream rejects that). Built here
|
||||
// as a standalone, pure function operating on the same ir.Plan every
|
||||
// query -- hand-written or AI-generated -- already compiles to, so there
|
||||
// is exactly one cost check, not one per code path.
|
||||
//
|
||||
// This package does not decide what a caller *does* with a Reject-level
|
||||
// Assessment -- see /docs/phase-7-ai-design.md's "Cost/safety guard"
|
||||
// section for how the AI tracks and the existing /query handler each
|
||||
// apply this differently (AI suggestions withhold a Reject-level
|
||||
// suggestion from being offered as directly runnable; the existing
|
||||
// /query handler surfaces the same assessment as a non-blocking warning,
|
||||
// deliberately not a new hard block on hand-written queries this phase
|
||||
// didn't set out to change).
|
||||
package costguard
|
||||
|
||||
import (
|
||||
"regexp"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"github.com/sentry/sentry/api/internal/querylang/ir"
|
||||
)
|
||||
|
||||
type Level string
|
||||
|
||||
const (
|
||||
LevelOK Level = "ok"
|
||||
LevelWarn Level = "warn"
|
||||
LevelReject Level = "reject"
|
||||
)
|
||||
|
||||
type Assessment struct {
|
||||
Level Level
|
||||
Reasons []string
|
||||
}
|
||||
|
||||
// maxReasonableSpan and the two below are first-pass heuristic
|
||||
// thresholds, not benchmarked against a production-scale ClickHouse
|
||||
// cluster -- this environment's own data is far smaller than what these
|
||||
// numbers are meant to guard against. Flagged explicitly in
|
||||
// /docs/phase-7-ai-design.md rather than presented as tuned. Revisit
|
||||
// once there's real cluster-size data to check them against.
|
||||
const maxReasonableSpan = 90 * 24 * time.Hour
|
||||
|
||||
// rawSQLTimestampRe is a best-effort, deliberately loose check for
|
||||
// *some* mention of the timestamp column in a raw SQL statement's WHERE
|
||||
// clause -- not a real SQL parser. A false negative here (a query that
|
||||
// does filter by time in a way this regex doesn't recognize) just means
|
||||
// an unnecessary Warn, not a Reject, so being loose-but-safe is the
|
||||
// right failure direction. Raw SQL genuinely can't get the same
|
||||
// structural guarantee the IR-based checks below get, and this package
|
||||
// says so rather than pretending otherwise.
|
||||
var rawSQLTimestampRe = regexp.MustCompile(`(?i)\btimestamp\b\s*[<>=]`)
|
||||
|
||||
// Assess evaluates one compiled plan. Never returns an error -- a plan
|
||||
// that reached this point already parsed successfully; this is a
|
||||
// judgment call about cost, not a correctness check.
|
||||
func Assess(plan *ir.Plan) Assessment {
|
||||
if plan.RawSQL != "" {
|
||||
return assessRawSQL(plan.RawSQL)
|
||||
}
|
||||
return assessIR(plan)
|
||||
}
|
||||
|
||||
func assessRawSQL(sql string) Assessment {
|
||||
if rawSQLTimestampRe.MatchString(sql) {
|
||||
return Assessment{Level: LevelOK}
|
||||
}
|
||||
return Assessment{
|
||||
Level: LevelWarn,
|
||||
Reasons: []string{
|
||||
"no obvious timestamp filter found in this raw SQL -- this is a best-effort text check, not a real parse, so it may be wrong in either direction, but if this query has no time bound it could scan the full table's history",
|
||||
},
|
||||
}
|
||||
}
|
||||
|
||||
func assessIR(plan *ir.Plan) Assessment {
|
||||
var reasons []string
|
||||
level := LevelOK
|
||||
|
||||
hasTimeBound := plan.TimeRange != nil && (!plan.TimeRange.From.IsZero() || !plan.TimeRange.To.IsZero())
|
||||
|
||||
if !hasTimeBound {
|
||||
switch {
|
||||
case plan.Aggregation != nil:
|
||||
// Unlike a raw-row query, an aggregation gets no implicit
|
||||
// row cap from the executor regardless of plan.Limit --
|
||||
// see executor/sql.go's buildSQL: the defaultRowLimit
|
||||
// safety net only applies `else if plan.Aggregation ==
|
||||
// nil`. An unbounded aggregation is never merely
|
||||
// "capped but slow" the way a raw-row fetch is.
|
||||
level = LevelReject
|
||||
reasons = append(reasons, "no time range filter, and this query aggregates -- every matching row across the table's entire history must be scanned to compute the aggregate, regardless of how small the output is")
|
||||
default:
|
||||
// A raw-row query with no explicit Limit still gets
|
||||
// executor/sql.go's defaultRowLimit=100 safety net applied
|
||||
// automatically -- it is not actually unbounded output,
|
||||
// just potentially an expensive scan to find those rows
|
||||
// without a time bound to narrow the search. Confirmed by
|
||||
// reading buildSQL directly, not assumed: this is the same
|
||||
// risk level whether plan.Limit is nil or explicitly set,
|
||||
// so both cases share one Warn, not a Reject for one and a
|
||||
// Warn for the other.
|
||||
level = LevelWarn
|
||||
reasons = append(reasons, "no time range filter -- results are capped (explicitly, or by the default 100-row limit), but ClickHouse may still need to scan well beyond that many rows to find them without a time bound to narrow the search")
|
||||
}
|
||||
if len(plan.TextSearch) > 0 {
|
||||
reasons = append(reasons, "the free-text search stage is bounded by the existing 5,000-record Tantivy prefilter cap regardless of time range, which partially limits how bad this is, but doesn't remove the underlying ClickHouse-side cost")
|
||||
}
|
||||
} else if !plan.TimeRange.From.IsZero() && !plan.TimeRange.To.IsZero() {
|
||||
span := plan.TimeRange.To.Sub(plan.TimeRange.From)
|
||||
if span > maxReasonableSpan {
|
||||
level = maxLevel(level, LevelWarn)
|
||||
reasons = append(reasons, "time range spans more than 90 days -- this may be slow depending on data volume")
|
||||
}
|
||||
}
|
||||
|
||||
return Assessment{Level: level, Reasons: reasons}
|
||||
}
|
||||
|
||||
func maxLevel(a, b Level) Level {
|
||||
rank := map[Level]int{LevelOK: 0, LevelWarn: 1, LevelReject: 2}
|
||||
if rank[b] > rank[a] {
|
||||
return b
|
||||
}
|
||||
return a
|
||||
}
|
||||
|
||||
// Summary renders an Assessment as one human-readable line, for
|
||||
// embedding in an AI-suggestion response or a /query warnings entry --
|
||||
// one shared rendering so the two callers don't independently invent
|
||||
// slightly different phrasing for the same underlying reasons.
|
||||
func Summary(a Assessment) string {
|
||||
if a.Level == LevelOK || len(a.Reasons) == 0 {
|
||||
return ""
|
||||
}
|
||||
return strings.Join(a.Reasons, "; ")
|
||||
}
|
||||
@@ -0,0 +1,100 @@
|
||||
package costguard
|
||||
|
||||
import (
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/sentry/sentry/api/internal/querylang/ir"
|
||||
)
|
||||
|
||||
// A raw-row (non-aggregation) query with no time range and no explicit
|
||||
// Limit still gets executor/sql.go's defaultRowLimit=100 safety net
|
||||
// applied automatically -- so this is a Warn (a possibly-expensive scan
|
||||
// to find those 100 rows), not a Reject (genuinely unbounded output),
|
||||
// which only an unbounded *aggregation* actually is. See the case
|
||||
// immediately below for that contrast.
|
||||
func TestAssessNoTimeRangeNoLimitRawRowWarns(t *testing.T) {
|
||||
plan := &ir.Plan{Filters: []ir.FilterPredicate{{Field: "service", Op: "=", Value: "api"}}}
|
||||
got := Assess(plan)
|
||||
if got.Level != LevelWarn {
|
||||
t.Errorf("Level = %v, want warn (executor applies a default row limit even with no explicit Limit)", got.Level)
|
||||
}
|
||||
if len(got.Reasons) == 0 {
|
||||
t.Error("expected at least one reason")
|
||||
}
|
||||
}
|
||||
|
||||
func TestAssessNoTimeRangeWithAggregationRejects(t *testing.T) {
|
||||
plan := &ir.Plan{
|
||||
TimeRange: &ir.TimeRange{},
|
||||
Aggregation: &ir.Aggregation{Funcs: []ir.AggFunc{{Func: "count", Alias: "count"}}},
|
||||
}
|
||||
got := Assess(plan)
|
||||
if got.Level != LevelReject {
|
||||
t.Errorf("Level = %v, want reject for an unbounded aggregation", got.Level)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAssessNoTimeRangeWithLimitWarns(t *testing.T) {
|
||||
plan := &ir.Plan{
|
||||
TimeRange: &ir.TimeRange{},
|
||||
Limit: &ir.Limit{N: 100},
|
||||
}
|
||||
got := Assess(plan)
|
||||
if got.Level != LevelWarn {
|
||||
t.Errorf("Level = %v, want warn (limited, no aggregation)", got.Level)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAssessBoundedTimeRangeIsOK(t *testing.T) {
|
||||
now := time.Now()
|
||||
plan := &ir.Plan{
|
||||
TimeRange: &ir.TimeRange{From: now.Add(-1 * time.Hour), To: now},
|
||||
Limit: &ir.Limit{N: 100},
|
||||
}
|
||||
got := Assess(plan)
|
||||
if got.Level != LevelOK {
|
||||
t.Errorf("Level = %v, want ok, reasons: %v", got.Level, got.Reasons)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAssessVeryLargeTimeRangeWarns(t *testing.T) {
|
||||
now := time.Now()
|
||||
plan := &ir.Plan{
|
||||
TimeRange: &ir.TimeRange{From: now.Add(-200 * 24 * time.Hour), To: now},
|
||||
Limit: &ir.Limit{N: 100},
|
||||
}
|
||||
got := Assess(plan)
|
||||
if got.Level != LevelWarn {
|
||||
t.Errorf("Level = %v, want warn for a 200-day range", got.Level)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAssessRawSQLWithTimestampFilterIsOK(t *testing.T) {
|
||||
plan := &ir.Plan{RawSQL: "SELECT count(*) FROM logs WHERE timestamp > now() - INTERVAL 1 HOUR"}
|
||||
got := Assess(plan)
|
||||
if got.Level != LevelOK {
|
||||
t.Errorf("Level = %v, want ok, reasons: %v", got.Level, got.Reasons)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAssessRawSQLWithoutTimestampFilterWarns(t *testing.T) {
|
||||
plan := &ir.Plan{RawSQL: "SELECT service, count(*) FROM logs GROUP BY service"}
|
||||
got := Assess(plan)
|
||||
if got.Level != LevelWarn {
|
||||
t.Errorf("Level = %v, want warn for raw SQL with no detectable time filter", got.Level)
|
||||
}
|
||||
}
|
||||
|
||||
func TestSummaryEmptyForOK(t *testing.T) {
|
||||
if s := Summary(Assessment{Level: LevelOK}); s != "" {
|
||||
t.Errorf("Summary(OK) = %q, want empty", s)
|
||||
}
|
||||
}
|
||||
|
||||
func TestSummaryJoinsReasons(t *testing.T) {
|
||||
a := Assessment{Level: LevelWarn, Reasons: []string{"a", "b"}}
|
||||
if s := Summary(a); s != "a; b" {
|
||||
t.Errorf("Summary = %q, want %q", s, "a; b")
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,257 @@
|
||||
// Package grounding builds provider.SchemaContext from a tenant's own
|
||||
// ClickHouse data -- known service names, common attribute keys, and
|
||||
// example values for enum-like fields -- sourced by periodic sampling,
|
||||
// never hand-maintained (task 3). See /docs/phase-7-ai-design.md's
|
||||
// "Schema grounding" section for the embedded-in-prompt-vs-retrieved
|
||||
// tradeoff this package's shape is built around.
|
||||
//
|
||||
// Tenant scoping is structural, not a filter this package applies: a
|
||||
// Service wraps exactly one executor.SQLRunner, and that SQLRunner is
|
||||
// already tenant-scoped by whoever constructed it (the plain shared
|
||||
// runner in a single-tenant deployment, or one specific tenant's
|
||||
// chrunner-resolved connection in enterprise-api) -- the same connection-
|
||||
// layer isolation discipline Phase 4 established for query execution
|
||||
// applies here for free, because grounding queries run through the exact
|
||||
// same SQLRunner interface, never a separate admin/shared connection.
|
||||
// A multi-tenant deployment needs one Service per active tenant --
|
||||
// enterprise/internal/groundingregistry provides that, mirroring
|
||||
// enterprise/internal/chwriter.Registry's per-tenant-instance shape.
|
||||
package grounding
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"strings"
|
||||
"sync"
|
||||
"time"
|
||||
|
||||
"github.com/sentry/sentry/api/ai/provider"
|
||||
"github.com/sentry/sentry/api/querylang/executor"
|
||||
)
|
||||
|
||||
// staticFields are always present regardless of what's actually been
|
||||
// ingested -- Phase 0's schema (storage/migrations/0001_create_logs_table.sql)
|
||||
// plus record_id (0002). Listed here rather than queried: their existence
|
||||
// doesn't depend on sampling, only their *values* do (severity's examples
|
||||
// still come from a real query below, in case a deployment's severities
|
||||
// diverge from the standard OTel set).
|
||||
var staticFields = []string{"timestamp", "host", "service", "severity", "message", "record_id"}
|
||||
|
||||
// Tuning constants. First-pass values, not benchmarked against a
|
||||
// production-scale cluster -- see the "not yet verified at scale" note
|
||||
// in /docs/phase-7-ai-design.md. Deliberately conservative (short lookback,
|
||||
// small caps) since grounding data trades completeness for prompt-budget
|
||||
// and refresh-query cost, not the other way around.
|
||||
const (
|
||||
sampleWindow = 7 * 24 * time.Hour // how far back sampling queries look
|
||||
maxServices = 50
|
||||
maxAttributeKeys = 100 // how many keys we learn about at all
|
||||
maxEnumCandidateKeys = 15 // of those, how many get a real example-value query (each is a separate round trip)
|
||||
maxExamplesPerField = 20 // a field returning more distinct values than this in the capped query isn't treated as enum-like
|
||||
perFieldQueryLimit = maxExamplesPerField + 1 // +1 so "more than maxExamplesPerField" is detectable, not just silently truncated
|
||||
)
|
||||
|
||||
// Service produces provider.SchemaContext for one tenant (or, in a
|
||||
// single-tenant deployment, the whole instance) from its own ClickHouse
|
||||
// data. Safe for concurrent use: Refresh swaps a snapshot under a mutex,
|
||||
// Current reads it under the same lock -- same last-known-good pattern
|
||||
// enterprise/internal/chwriter.Registry and search/src/tenants.rs's
|
||||
// ActiveTenantTracker already use, so a slow or failing refresh never
|
||||
// blocks or blanks a caller mid-request.
|
||||
type Service struct {
|
||||
runner executor.SQLRunner
|
||||
|
||||
mu sync.RWMutex
|
||||
snapshot provider.SchemaContext
|
||||
}
|
||||
|
||||
func New(runner executor.SQLRunner) *Service {
|
||||
return &Service{runner: runner}
|
||||
}
|
||||
|
||||
// Current returns the last successfully refreshed SchemaContext --
|
||||
// possibly stale, possibly zero-valued if Refresh has never succeeded
|
||||
// yet, but never a partial/torn snapshot. Callers (the AI operation
|
||||
// handlers, task 5+) should treat a zero-valued Services/Fields as "no
|
||||
// grounding data yet available," not an error -- every operation still
|
||||
// works with an empty SchemaContext, just less well-grounded, matching
|
||||
// this codebase's "absence is a normal state, not a failure" convention
|
||||
// (e.g. AuditLogger, getAuthFeatures).
|
||||
func (s *Service) Current() provider.SchemaContext {
|
||||
s.mu.RLock()
|
||||
defer s.mu.RUnlock()
|
||||
return s.snapshot
|
||||
}
|
||||
|
||||
// SchemaContext implements aiapi.SchemaContextSource directly -- a
|
||||
// single-tenant deployment has exactly one Service, so there's no
|
||||
// per-request tenant resolution to do here (ctx is unused); it's the
|
||||
// same shape as Current, just satisfying the interface aiapi's handlers
|
||||
// depend on so main.go can wire *Service in without a separate adapter
|
||||
// type. enterprise-api's multi-tenant equivalent (groundingregistry)
|
||||
// implements this same interface by actually reading ctx.
|
||||
func (s *Service) SchemaContext(context.Context) provider.SchemaContext {
|
||||
return s.Current()
|
||||
}
|
||||
|
||||
// Refresh runs the sampling queries and swaps the cached snapshot on
|
||||
// success. A failed refresh leaves the previous snapshot in place
|
||||
// (last-known-good) rather than clearing it -- a transient ClickHouse
|
||||
// hiccup shouldn't blank out grounding for every AI request until the
|
||||
// next successful refresh.
|
||||
func (s *Service) Refresh(ctx context.Context) error {
|
||||
services, err := s.sampleServices(ctx)
|
||||
if err != nil {
|
||||
return fmt.Errorf("grounding: sampling services: %w", err)
|
||||
}
|
||||
|
||||
attrKeys, err := s.sampleAttributeKeys(ctx)
|
||||
if err != nil {
|
||||
return fmt.Errorf("grounding: sampling attribute keys: %w", err)
|
||||
}
|
||||
|
||||
fields := make([]provider.FieldInfo, 0, len(staticFields)+len(attrKeys))
|
||||
for _, name := range staticFields {
|
||||
examples, _ := s.sampleFieldExamples(ctx, name, name == "severity")
|
||||
fields = append(fields, provider.FieldInfo{Name: name, Examples: examples})
|
||||
}
|
||||
|
||||
candidateKeys := attrKeys
|
||||
if len(candidateKeys) > maxEnumCandidateKeys {
|
||||
candidateKeys = candidateKeys[:maxEnumCandidateKeys]
|
||||
}
|
||||
enumExamples := make(map[string][]string, len(candidateKeys))
|
||||
for _, key := range candidateKeys {
|
||||
examples, ok := s.sampleFieldExamples(ctx, key, false)
|
||||
if ok {
|
||||
enumExamples[key] = examples
|
||||
}
|
||||
}
|
||||
for _, key := range attrKeys {
|
||||
fields = append(fields, provider.FieldInfo{Name: key, Examples: enumExamples[key]})
|
||||
}
|
||||
|
||||
s.mu.Lock()
|
||||
s.snapshot = provider.SchemaContext{Services: services, Fields: fields}
|
||||
s.mu.Unlock()
|
||||
return nil
|
||||
}
|
||||
|
||||
// StartRefreshing runs Refresh once immediately (best-effort -- a failure
|
||||
// here just means Current() returns a zero snapshot until the first
|
||||
// successful tick, not a fatal startup error, since grounding is an
|
||||
// enhancement, not a dependency anything else blocks on) and then on
|
||||
// interval until ctx is cancelled. Same shape as chwriter.Registry.
|
||||
// StartRefreshing.
|
||||
func (s *Service) StartRefreshing(ctx context.Context, interval time.Duration, onError func(error)) {
|
||||
if err := s.Refresh(ctx); err != nil && onError != nil {
|
||||
onError(err)
|
||||
}
|
||||
go func() {
|
||||
ticker := time.NewTicker(interval)
|
||||
defer ticker.Stop()
|
||||
for {
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
return
|
||||
case <-ticker.C:
|
||||
if err := s.Refresh(ctx); err != nil && onError != nil {
|
||||
onError(err)
|
||||
}
|
||||
}
|
||||
}
|
||||
}()
|
||||
}
|
||||
|
||||
func (s *Service) sampleServices(ctx context.Context) ([]string, error) {
|
||||
sql := fmt.Sprintf(
|
||||
"SELECT service FROM logs WHERE timestamp > now() - INTERVAL %d SECOND GROUP BY service ORDER BY count() DESC LIMIT %d",
|
||||
int(sampleWindow.Seconds()), maxServices,
|
||||
)
|
||||
res, err := s.runner.RunSQL(ctx, sql)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return firstColumnStrings(res), nil
|
||||
}
|
||||
|
||||
func (s *Service) sampleAttributeKeys(ctx context.Context) ([]string, error) {
|
||||
sql := fmt.Sprintf(
|
||||
"SELECT arrayJoin(mapKeys(attributes)) AS attr_key FROM logs WHERE timestamp > now() - INTERVAL %d SECOND GROUP BY attr_key ORDER BY count() DESC LIMIT %d",
|
||||
int(sampleWindow.Seconds()), maxAttributeKeys,
|
||||
)
|
||||
res, err := s.runner.RunSQL(ctx, sql)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return firstColumnStrings(res), nil
|
||||
}
|
||||
|
||||
// sampleFieldExamples returns up to maxExamplesPerField distinct values
|
||||
// for a field, and false if the field turned out not to look enum-like
|
||||
// (more distinct values than the cap turned up, or the value column
|
||||
// wasn't usable) -- matching FieldInfo's doc comment that a
|
||||
// high-cardinality field should carry no examples rather than a
|
||||
// truncated, misleading sample. isStructuredColumn distinguishes
|
||||
// `severity` (a real column) from an attributes[...] lookup.
|
||||
func (s *Service) sampleFieldExamples(ctx context.Context, field string, isStructuredColumn bool) ([]string, bool) {
|
||||
col := "attributes[" + quoteLiteral(field) + "]"
|
||||
if isStructuredColumn {
|
||||
col = "`" + field + "`"
|
||||
}
|
||||
sql := fmt.Sprintf(
|
||||
"SELECT DISTINCT %s AS v FROM logs WHERE timestamp > now() - INTERVAL %d SECOND AND %s != '' LIMIT %d",
|
||||
col, int(sampleWindow.Seconds()), col, perFieldQueryLimit,
|
||||
)
|
||||
res, err := s.runner.RunSQL(ctx, sql)
|
||||
if err != nil {
|
||||
return nil, false
|
||||
}
|
||||
values := firstColumnStrings(res)
|
||||
if len(values) == 0 || len(values) > maxExamplesPerField {
|
||||
return nil, false
|
||||
}
|
||||
return values, true
|
||||
}
|
||||
|
||||
func firstColumnStrings(res *executor.Result) []string {
|
||||
if res == nil || len(res.Columns) == 0 {
|
||||
return nil
|
||||
}
|
||||
out := make([]string, 0, len(res.Rows))
|
||||
for _, row := range res.Rows {
|
||||
if len(row) == 0 {
|
||||
continue
|
||||
}
|
||||
if s, ok := row[0].(string); ok && s != "" {
|
||||
out = append(out, s)
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// quoteLiteral must stay byte-for-byte in sync with executor/sql.go's
|
||||
// unexported function of the same name (ClickHouse SQL string literals
|
||||
// use backslash escaping, not SQL-standard doubled quotes -- easy to get
|
||||
// wrong by assuming the more common convention, which an earlier draft
|
||||
// of this function did). Duplicated rather than exported from executor,
|
||||
// since executor's quoteLiteral is deliberately unexported (query-SQL
|
||||
// building is that package's own concern) and grounding's use is
|
||||
// narrow enough not to justify widening that package's public surface
|
||||
// for one helper.
|
||||
func quoteLiteral(s string) string {
|
||||
var sb strings.Builder
|
||||
sb.WriteByte('\'')
|
||||
for _, r := range s {
|
||||
switch r {
|
||||
case '\\':
|
||||
sb.WriteString(`\\`)
|
||||
case '\'':
|
||||
sb.WriteString(`\'`)
|
||||
default:
|
||||
sb.WriteRune(r)
|
||||
}
|
||||
}
|
||||
sb.WriteByte('\'')
|
||||
return sb.String()
|
||||
}
|
||||
@@ -0,0 +1,169 @@
|
||||
package grounding
|
||||
|
||||
import (
|
||||
"context"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/sentry/sentry/api/ai/provider"
|
||||
"github.com/sentry/sentry/api/querylang/executor"
|
||||
)
|
||||
|
||||
// routingFakeRunner returns a canned result keyed by a substring match
|
||||
// against the SQL text -- grounding.Refresh issues several structurally
|
||||
// different queries in sequence (services, attribute keys, then one
|
||||
// per candidate enum field), unlike executor's tests where a single
|
||||
// fixed result/err per call is enough.
|
||||
type routingFakeRunner struct {
|
||||
byContains []struct {
|
||||
substr string
|
||||
result *executor.Result
|
||||
err error
|
||||
}
|
||||
calls int
|
||||
}
|
||||
|
||||
func (r *routingFakeRunner) on(substr string, result *executor.Result) *routingFakeRunner {
|
||||
r.byContains = append(r.byContains, struct {
|
||||
substr string
|
||||
result *executor.Result
|
||||
err error
|
||||
}{substr, result, nil})
|
||||
return r
|
||||
}
|
||||
|
||||
func (r *routingFakeRunner) RunSQL(_ context.Context, sql string) (*executor.Result, error) {
|
||||
r.calls++
|
||||
for _, rule := range r.byContains {
|
||||
if strings.Contains(sql, rule.substr) {
|
||||
if rule.err != nil {
|
||||
return nil, rule.err
|
||||
}
|
||||
return rule.result, nil
|
||||
}
|
||||
}
|
||||
// Unmatched queries (most of the per-field example queries in a
|
||||
// small test fixture) come back empty, same as a field with no data
|
||||
// -- not an error, matching sampleFieldExamples' "unusable -> not
|
||||
// enum-like" treatment.
|
||||
return &executor.Result{Columns: []string{"v"}, Rows: nil}, nil
|
||||
}
|
||||
|
||||
func strResult(col string, vals ...string) *executor.Result {
|
||||
rows := make([][]any, len(vals))
|
||||
for i, v := range vals {
|
||||
rows[i] = []any{v}
|
||||
}
|
||||
return &executor.Result{Columns: []string{col}, Rows: rows}
|
||||
}
|
||||
|
||||
func TestRefreshPopulatesServicesAndFields(t *testing.T) {
|
||||
runner := (&routingFakeRunner{}).
|
||||
on("FROM logs WHERE timestamp > now() - INTERVAL 604800 SECOND GROUP BY service", strResult("service", "api", "web", "worker")).
|
||||
on("mapKeys(attributes)", strResult("attr_key", "status", "latency_ms")).
|
||||
on("`severity`", strResult("v", "INFO", "WARN", "ERROR")).
|
||||
on("attributes['status']", strResult("v", "200", "404", "500"))
|
||||
|
||||
svc := New(runner)
|
||||
if err := svc.Refresh(context.Background()); err != nil {
|
||||
t.Fatalf("Refresh: %v", err)
|
||||
}
|
||||
|
||||
got := svc.Current()
|
||||
if len(got.Services) != 3 || got.Services[0] != "api" {
|
||||
t.Errorf("Services = %v, want [api web worker]", got.Services)
|
||||
}
|
||||
|
||||
var severity, status *provider.FieldInfo
|
||||
for i := range got.Fields {
|
||||
f := &got.Fields[i]
|
||||
switch f.Name {
|
||||
case "severity":
|
||||
severity = f
|
||||
case "status":
|
||||
status = f
|
||||
}
|
||||
}
|
||||
if severity == nil || len(severity.Examples) != 3 {
|
||||
t.Errorf("severity field = %+v, want 3 examples", severity)
|
||||
}
|
||||
if status == nil || len(status.Examples) != 3 {
|
||||
t.Errorf("status field = %+v, want 3 examples", status)
|
||||
}
|
||||
|
||||
// Static fields with no configured example rule (host, message,
|
||||
// timestamp, record_id) should still be present, just with no
|
||||
// examples -- Refresh must not drop them.
|
||||
names := make(map[string]bool, len(got.Fields))
|
||||
for _, f := range got.Fields {
|
||||
names[f.Name] = true
|
||||
}
|
||||
for _, want := range []string{"timestamp", "host", "message", "record_id"} {
|
||||
if !names[want] {
|
||||
t.Errorf("static field %q missing from Fields", want)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestRefreshFailureLeavesPreviousSnapshot(t *testing.T) {
|
||||
good := (&routingFakeRunner{}).
|
||||
on("GROUP BY service", strResult("service", "api"))
|
||||
svc := New(good)
|
||||
if err := svc.Refresh(context.Background()); err != nil {
|
||||
t.Fatalf("first Refresh: %v", err)
|
||||
}
|
||||
first := svc.Current()
|
||||
|
||||
svc.runner = &erroringRunner{}
|
||||
if err := svc.Refresh(context.Background()); err == nil {
|
||||
t.Fatal("expected Refresh to fail with an erroring runner")
|
||||
}
|
||||
|
||||
after := svc.Current()
|
||||
if len(after.Services) != len(first.Services) || after.Services[0] != first.Services[0] {
|
||||
t.Errorf("Current() after a failed Refresh = %+v, want unchanged snapshot %+v", after, first)
|
||||
}
|
||||
}
|
||||
|
||||
type erroringRunner struct{}
|
||||
|
||||
func (erroringRunner) RunSQL(context.Context, string) (*executor.Result, error) {
|
||||
return nil, context.DeadlineExceeded
|
||||
}
|
||||
|
||||
func TestFieldExampleCapExcludesHighCardinalityFields(t *testing.T) {
|
||||
many := make([]string, maxExamplesPerField+1)
|
||||
for i := range many {
|
||||
many[i] = string(rune('a' + i%26))
|
||||
}
|
||||
runner := (&routingFakeRunner{}).
|
||||
on("GROUP BY service", strResult("service", "api")).
|
||||
on("mapKeys(attributes)", strResult("attr_key", "trace_id")).
|
||||
on("attributes['trace_id']", strResult("v", many...))
|
||||
|
||||
svc := New(runner)
|
||||
if err := svc.Refresh(context.Background()); err != nil {
|
||||
t.Fatalf("Refresh: %v", err)
|
||||
}
|
||||
for _, f := range svc.Current().Fields {
|
||||
if f.Name == "trace_id" && len(f.Examples) != 0 {
|
||||
t.Errorf("high-cardinality field trace_id got %d examples, want 0 (not treated as enum-like)", len(f.Examples))
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestStartRefreshingRunsOnInterval(t *testing.T) {
|
||||
runner := (&routingFakeRunner{}).on("GROUP BY service", strResult("service", "api"))
|
||||
svc := New(runner)
|
||||
|
||||
ctx, cancel := context.WithCancel(context.Background())
|
||||
defer cancel()
|
||||
svc.StartRefreshing(ctx, 10*time.Millisecond, nil)
|
||||
|
||||
// The immediate synchronous refresh should have already happened by
|
||||
// the time StartRefreshing returns.
|
||||
if got := svc.Current().Services; len(got) != 1 {
|
||||
t.Fatalf("Current() immediately after StartRefreshing = %v, want [api]", got)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,217 @@
|
||||
// Package ollama implements provider.Provider against a local Ollama
|
||||
// server -- the default, self-hosted primary provider (Phase 7 task 2;
|
||||
// see /docs/phase-7-ai-design.md for why Ollama over vLLM and why
|
||||
// qwen2.5-coder is the recommended model). Same thin-HTTP-client shape
|
||||
// as alerting/internal/queryclient -- net/http + encoding/json, no new
|
||||
// HTTP client dependency, matching this codebase's "boring,
|
||||
// well-understood dependencies" convention.
|
||||
package ollama
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"net/http"
|
||||
"strings"
|
||||
|
||||
"github.com/sentry/sentry/api/ai/provider"
|
||||
)
|
||||
|
||||
// Client implements provider.Provider against one Ollama server and one
|
||||
// model. The per-operation routing layer (task 2's "per-operation
|
||||
// provider/model configuration") constructs one Client per distinct
|
||||
// model a deployment configures -- e.g. one for qwen2.5-coder:1.5b
|
||||
// (Complete's fast path) and one for qwen2.5-coder:7b (everything else)
|
||||
// -- rather than this package knowing anything about operation-to-model
|
||||
// routing itself.
|
||||
type Client struct {
|
||||
baseURL string
|
||||
model string
|
||||
http *http.Client
|
||||
}
|
||||
|
||||
// New builds a Client. baseURL defaults to Ollama's standard local
|
||||
// address if empty -- the common case for the primary, self-hosted
|
||||
// deployment target.
|
||||
func New(baseURL, model string) *Client {
|
||||
if baseURL == "" {
|
||||
baseURL = "http://localhost:11434"
|
||||
}
|
||||
return &Client{baseURL: strings.TrimSuffix(baseURL, "/"), model: model, http: &http.Client{}}
|
||||
}
|
||||
|
||||
type chatMessage struct {
|
||||
Role string `json:"role"`
|
||||
Content string `json:"content"`
|
||||
}
|
||||
|
||||
type chatRequest struct {
|
||||
Model string `json:"model"`
|
||||
Messages []chatMessage `json:"messages"`
|
||||
Stream bool `json:"stream"`
|
||||
Format string `json:"format,omitempty"`
|
||||
}
|
||||
|
||||
type chatResponse struct {
|
||||
Message chatMessage `json:"message"`
|
||||
}
|
||||
|
||||
// chat calls Ollama's POST /api/chat, non-streaming, and returns the
|
||||
// assistant message content. jsonMode requests Ollama's JSON-constrained
|
||||
// output format -- used by every operation except Explain, which just
|
||||
// wants prose back.
|
||||
func (c *Client) chat(ctx context.Context, system, user string, jsonMode bool) (string, error) {
|
||||
req := chatRequest{
|
||||
Model: c.model,
|
||||
Messages: []chatMessage{
|
||||
{Role: "system", Content: system},
|
||||
{Role: "user", Content: user},
|
||||
},
|
||||
Stream: false,
|
||||
}
|
||||
if jsonMode {
|
||||
req.Format = "json"
|
||||
}
|
||||
|
||||
body, err := json.Marshal(req)
|
||||
if err != nil {
|
||||
return "", fmt.Errorf("ollama: encoding request: %w", err)
|
||||
}
|
||||
|
||||
httpReq, err := http.NewRequestWithContext(ctx, http.MethodPost, c.baseURL+"/api/chat", bytes.NewReader(body))
|
||||
if err != nil {
|
||||
return "", fmt.Errorf("ollama: building request: %w", err)
|
||||
}
|
||||
httpReq.Header.Set("Content-Type", "application/json")
|
||||
|
||||
resp, err := c.http.Do(httpReq)
|
||||
if err != nil {
|
||||
return "", fmt.Errorf("ollama: calling %s: %w", c.baseURL, err)
|
||||
}
|
||||
defer resp.Body.Close()
|
||||
|
||||
if resp.StatusCode != http.StatusOK {
|
||||
var errBody struct {
|
||||
Error string `json:"error"`
|
||||
}
|
||||
_ = json.NewDecoder(resp.Body).Decode(&errBody)
|
||||
if errBody.Error != "" {
|
||||
return "", fmt.Errorf("ollama: request failed (%d): %s", resp.StatusCode, errBody.Error)
|
||||
}
|
||||
return "", fmt.Errorf("ollama: request failed with status %d", resp.StatusCode)
|
||||
}
|
||||
|
||||
var chatResp chatResponse
|
||||
if err := json.NewDecoder(resp.Body).Decode(&chatResp); err != nil {
|
||||
return "", fmt.Errorf("ollama: decoding response: %w", err)
|
||||
}
|
||||
return chatResp.Message.Content, nil
|
||||
}
|
||||
|
||||
// stripCodeFence handles the common small-model habit of wrapping JSON
|
||||
// output in ```json ... ``` even when explicitly told not to -- a
|
||||
// best-effort cleanup, not a guarantee; a model that returns genuinely
|
||||
// malformed JSON still surfaces as a real decode error to the caller,
|
||||
// which is the correct behavior (better an explicit error than silently
|
||||
// fabricating a result).
|
||||
func stripCodeFence(s string) string {
|
||||
s = strings.TrimSpace(s)
|
||||
if !strings.HasPrefix(s, "```") {
|
||||
return s
|
||||
}
|
||||
s = strings.TrimPrefix(s, "```json")
|
||||
s = strings.TrimPrefix(s, "```")
|
||||
s = strings.TrimSuffix(s, "```")
|
||||
return strings.TrimSpace(s)
|
||||
}
|
||||
|
||||
func parseConfidence(s string) provider.Confidence {
|
||||
switch strings.ToLower(strings.TrimSpace(s)) {
|
||||
case "high":
|
||||
return provider.ConfidenceHigh
|
||||
case "medium":
|
||||
return provider.ConfidenceMedium
|
||||
default:
|
||||
// Unrecognized or missing confidence fails toward caution, not
|
||||
// toward assumed correctness -- an empty/garbled confidence
|
||||
// field from the model is itself a signal something's off.
|
||||
return provider.ConfidenceLow
|
||||
}
|
||||
}
|
||||
|
||||
func (c *Client) Translate(ctx context.Context, req provider.TranslateRequest) (provider.TranslateResult, error) {
|
||||
raw, err := c.chat(ctx, translateSystemPrompt(req.Schema), req.NLQuery, true)
|
||||
if err != nil {
|
||||
return provider.TranslateResult{}, err
|
||||
}
|
||||
var parsed struct {
|
||||
Query string `json:"query"`
|
||||
Confidence string `json:"confidence"`
|
||||
Reason string `json:"reason"`
|
||||
}
|
||||
if err := json.Unmarshal([]byte(stripCodeFence(raw)), &parsed); err != nil {
|
||||
return provider.TranslateResult{}, fmt.Errorf("ollama: parsing translate response: %w", err)
|
||||
}
|
||||
return provider.TranslateResult{
|
||||
Query: parsed.Query,
|
||||
Confidence: parseConfidence(parsed.Confidence),
|
||||
LowConfidenceReason: parsed.Reason,
|
||||
}, nil
|
||||
}
|
||||
|
||||
func (c *Client) Complete(ctx context.Context, req provider.CompleteRequest) (provider.CompleteResult, error) {
|
||||
raw, err := c.chat(ctx, completeSystemPrompt(req.Schema), req.QueryPrefix, true)
|
||||
if err != nil {
|
||||
return provider.CompleteResult{}, err
|
||||
}
|
||||
var parsed struct {
|
||||
Suggestion string `json:"suggestion"`
|
||||
}
|
||||
if err := json.Unmarshal([]byte(stripCodeFence(raw)), &parsed); err != nil {
|
||||
return provider.CompleteResult{}, fmt.Errorf("ollama: parsing complete response: %w", err)
|
||||
}
|
||||
return provider.CompleteResult{Suggestion: parsed.Suggestion}, nil
|
||||
}
|
||||
|
||||
func (c *Client) Explain(ctx context.Context, req provider.ExplainRequest) (provider.ExplainResult, error) {
|
||||
user := req.Query
|
||||
switch {
|
||||
case len(req.RuleFindings) > 0:
|
||||
user = fmt.Sprintf("Query: %s\nFindings: %s", req.Query, strings.Join(req.RuleFindings, "; "))
|
||||
case req.OriginalIntent != "":
|
||||
user = fmt.Sprintf("Original request: %q\nGenerated query: %s", req.OriginalIntent, req.Query)
|
||||
}
|
||||
raw, err := c.chat(ctx, explainSystemPrompt(req.OriginalIntent != "", len(req.RuleFindings) > 0), user, false)
|
||||
if err != nil {
|
||||
return provider.ExplainResult{}, err
|
||||
}
|
||||
return provider.ExplainResult{Explanation: strings.TrimSpace(raw)}, nil
|
||||
}
|
||||
|
||||
func (c *Client) Fix(ctx context.Context, req provider.FixRequest) (provider.FixResult, error) {
|
||||
errText := req.ParseError
|
||||
if errText == "" {
|
||||
errText = req.ExecutionError
|
||||
}
|
||||
user := fmt.Sprintf("Query: %s\nError: %s", req.Query, errText)
|
||||
raw, err := c.chat(ctx, fixSystemPrompt(req.Schema), user, true)
|
||||
if err != nil {
|
||||
return provider.FixResult{}, err
|
||||
}
|
||||
var parsed struct {
|
||||
SuggestedQuery string `json:"suggested_query"`
|
||||
Explanation string `json:"explanation"`
|
||||
Confidence string `json:"confidence"`
|
||||
}
|
||||
if err := json.Unmarshal([]byte(stripCodeFence(raw)), &parsed); err != nil {
|
||||
return provider.FixResult{}, fmt.Errorf("ollama: parsing fix response: %w", err)
|
||||
}
|
||||
return provider.FixResult{
|
||||
SuggestedQuery: parsed.SuggestedQuery,
|
||||
Explanation: parsed.Explanation,
|
||||
Confidence: parseConfidence(parsed.Confidence),
|
||||
}, nil
|
||||
}
|
||||
|
||||
var _ provider.Provider = (*Client)(nil)
|
||||
@@ -0,0 +1,152 @@
|
||||
package ollama
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"github.com/sentry/sentry/api/ai/provider"
|
||||
)
|
||||
|
||||
// fakeOllamaServer stands in for a real Ollama server, returning the
|
||||
// given assistant-message content verbatim -- same reasoning
|
||||
// queryclient's tests use httptest against a fake api instead of a real
|
||||
// one: this package's own logic (request shape, response parsing,
|
||||
// JSON-mode contract) is what's under test, not Ollama itself.
|
||||
func fakeOllamaServer(t *testing.T, content string) *httptest.Server {
|
||||
t.Helper()
|
||||
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
if r.URL.Path != "/api/chat" {
|
||||
t.Errorf("unexpected path %s", r.URL.Path)
|
||||
}
|
||||
var req chatRequest
|
||||
if err := json.NewDecoder(r.Body).Decode(&req); err != nil {
|
||||
t.Fatalf("decoding request: %v", err)
|
||||
}
|
||||
if len(req.Messages) != 2 || req.Messages[0].Role != "system" || req.Messages[1].Role != "user" {
|
||||
t.Errorf("unexpected messages shape: %+v", req.Messages)
|
||||
}
|
||||
w.Header().Set("Content-Type", "application/json")
|
||||
_ = json.NewEncoder(w).Encode(chatResponse{Message: chatMessage{Role: "assistant", Content: content}})
|
||||
}))
|
||||
t.Cleanup(srv.Close)
|
||||
return srv
|
||||
}
|
||||
|
||||
func TestTranslateParsesJSONResponse(t *testing.T) {
|
||||
srv := fakeOllamaServer(t, `{"query": "earliest=-1h severity=ERROR | stats count by service", "confidence": "high", "reason": ""}`)
|
||||
c := New(srv.URL, "test-model")
|
||||
|
||||
got, err := c.Translate(context.Background(), provider.TranslateRequest{NLQuery: "errors in the last hour by service"})
|
||||
if err != nil {
|
||||
t.Fatalf("Translate: %v", err)
|
||||
}
|
||||
if got.Confidence != provider.ConfidenceHigh {
|
||||
t.Errorf("Confidence = %v, want high", got.Confidence)
|
||||
}
|
||||
if got.Query == "" {
|
||||
t.Error("expected a non-empty query")
|
||||
}
|
||||
}
|
||||
|
||||
func TestTranslateHandlesCodeFencedJSON(t *testing.T) {
|
||||
srv := fakeOllamaServer(t, "```json\n{\"query\": \"service=api\", \"confidence\": \"medium\", \"reason\": \"\"}\n```")
|
||||
c := New(srv.URL, "test-model")
|
||||
|
||||
got, err := c.Translate(context.Background(), provider.TranslateRequest{NLQuery: "api logs"})
|
||||
if err != nil {
|
||||
t.Fatalf("Translate with fenced JSON: %v", err)
|
||||
}
|
||||
if got.Query != "service=api" {
|
||||
t.Errorf("Query = %q, want %q", got.Query, "service=api")
|
||||
}
|
||||
}
|
||||
|
||||
func TestTranslateLowConfidenceCarriesReason(t *testing.T) {
|
||||
srv := fakeOllamaServer(t, `{"query": "", "confidence": "low", "reason": "not sure what 'weird stuff' refers to"}`)
|
||||
c := New(srv.URL, "test-model")
|
||||
|
||||
got, err := c.Translate(context.Background(), provider.TranslateRequest{NLQuery: "show me weird stuff"})
|
||||
if err != nil {
|
||||
t.Fatalf("Translate: %v", err)
|
||||
}
|
||||
if got.Confidence != provider.ConfidenceLow || got.LowConfidenceReason == "" {
|
||||
t.Errorf("got = %+v, want low confidence with a reason", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestTranslateMalformedJSONIsAnError(t *testing.T) {
|
||||
srv := fakeOllamaServer(t, "not json at all, sorry")
|
||||
c := New(srv.URL, "test-model")
|
||||
|
||||
_, err := c.Translate(context.Background(), provider.TranslateRequest{NLQuery: "anything"})
|
||||
if err == nil {
|
||||
t.Fatal("expected an error for unparseable model output, got nil")
|
||||
}
|
||||
}
|
||||
|
||||
func TestCompleteReturnsSuggestionOnly(t *testing.T) {
|
||||
srv := fakeOllamaServer(t, `{"suggestion": " | stats count by host"}`)
|
||||
c := New(srv.URL, "test-model")
|
||||
|
||||
got, err := c.Complete(context.Background(), provider.CompleteRequest{QueryPrefix: "service=api", Language: "spl"})
|
||||
if err != nil {
|
||||
t.Fatalf("Complete: %v", err)
|
||||
}
|
||||
if got.Suggestion != " | stats count by host" {
|
||||
t.Errorf("Suggestion = %q", got.Suggestion)
|
||||
}
|
||||
}
|
||||
|
||||
func TestExplainReturnsPlainText(t *testing.T) {
|
||||
srv := fakeOllamaServer(t, "This counts events per host over the last hour.")
|
||||
c := New(srv.URL, "test-model")
|
||||
|
||||
got, err := c.Explain(context.Background(), provider.ExplainRequest{Query: "earliest=-1h | stats count by host"})
|
||||
if err != nil {
|
||||
t.Fatalf("Explain: %v", err)
|
||||
}
|
||||
if got.Explanation == "" {
|
||||
t.Error("expected a non-empty explanation")
|
||||
}
|
||||
}
|
||||
|
||||
func TestFixParsesJSONResponse(t *testing.T) {
|
||||
srv := fakeOllamaServer(t, `{"suggested_query": "earliest=-1h | stats count", "explanation": "added a required time range", "confidence": "high"}`)
|
||||
c := New(srv.URL, "test-model")
|
||||
|
||||
got, err := c.Fix(context.Background(), provider.FixRequest{Query: "stats count", ParseError: "no time range"})
|
||||
if err != nil {
|
||||
t.Fatalf("Fix: %v", err)
|
||||
}
|
||||
if got.SuggestedQuery == "" || got.Explanation == "" {
|
||||
t.Errorf("got = %+v, want both fields populated", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestNonOKStatusIsAnError(t *testing.T) {
|
||||
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
w.WriteHeader(http.StatusInternalServerError)
|
||||
_ = json.NewEncoder(w).Encode(map[string]string{"error": "model not found"})
|
||||
}))
|
||||
t.Cleanup(srv.Close)
|
||||
c := New(srv.URL, "missing-model")
|
||||
|
||||
_, err := c.Explain(context.Background(), provider.ExplainRequest{Query: "x"})
|
||||
if err == nil {
|
||||
t.Fatal("expected an error for a non-200 response")
|
||||
}
|
||||
if got := err.Error(); !strings.Contains(got, "model not found") {
|
||||
t.Errorf("error = %q, want it to include the server's error message", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestDefaultBaseURL(t *testing.T) {
|
||||
c := New("", "m")
|
||||
if c.baseURL != "http://localhost:11434" {
|
||||
t.Errorf("default baseURL = %q", c.baseURL)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,122 @@
|
||||
package ollama
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"strings"
|
||||
|
||||
"github.com/sentry/sentry/api/ai/provider"
|
||||
)
|
||||
|
||||
// grammarReference is a condensed version of
|
||||
// /docs/query-language-reference.md -- every operation's system prompt
|
||||
// includes this so the model is grounded in Sentry's actual pipe syntax,
|
||||
// not whatever generic log-query DSL it may have seen in training.
|
||||
// Trimmed to the parts that matter for generation/explanation (the full
|
||||
// doc's prose and examples aren't needed here); kept in sync with that
|
||||
// doc by hand -- if the grammar changes, this needs updating too, same
|
||||
// as any other place the language is described outside its own parser.
|
||||
const grammarReference = `Sentry query language (pipe syntax):
|
||||
|
||||
<base search> | <stage> | <stage> | ...
|
||||
|
||||
Base search: filter terms and/or free-text search, combined with implicit "and".
|
||||
field=value, field!=value, field>value, field>=value, field<value, field<=value
|
||||
bare word or "quoted phrase" -- free-text search on the message field
|
||||
message:"phrase" -- explicit free-text search
|
||||
earliest=-1h, latest=-5m -- relative time (s/m/h/d/w units), or earliest="2026-08-14T00:00:00Z" (RFC 3339 absolute)
|
||||
"or" only works between free-text terms, never between structured filters
|
||||
|
||||
Pipe stages, in the order they may appear:
|
||||
| where <filter terms> additional filtering, same syntax as base search filters
|
||||
| stats <func>(<field>) as <alias>, ... by <field>, ...
|
||||
functions: count (no field needed), sum, avg, min, max (all need a field)
|
||||
| sort -field, +field, ... "-" descending (default if no sign), "+" ascending
|
||||
| fields field, field, ... choose output columns
|
||||
| head N first N results (default 100)
|
||||
| tail N last N results, chronologically
|
||||
|
||||
Structured columns: timestamp, host, service, severity, message, record_id.
|
||||
Anything else is looked up in per-record attributes (always text; compared
|
||||
numerically when the right-hand side looks like a number).
|
||||
|
||||
Raw SQL (SELECT ...) is also accepted but pipe syntax is strongly preferred
|
||||
for anything generated rather than hand-written -- narrower, safer surface.`
|
||||
|
||||
func renderSchema(s provider.SchemaContext) string {
|
||||
if len(s.Services) == 0 && len(s.Fields) == 0 {
|
||||
return "(no schema grounding data available yet)"
|
||||
}
|
||||
var sb strings.Builder
|
||||
if len(s.Services) > 0 {
|
||||
fmt.Fprintf(&sb, "Known services: %s\n", strings.Join(s.Services, ", "))
|
||||
}
|
||||
if len(s.Fields) > 0 {
|
||||
sb.WriteString("Known fields:\n")
|
||||
for _, f := range s.Fields {
|
||||
if len(f.Examples) > 0 {
|
||||
fmt.Fprintf(&sb, " - %s (examples: %s)\n", f.Name, strings.Join(f.Examples, ", "))
|
||||
} else {
|
||||
fmt.Fprintf(&sb, " - %s\n", f.Name)
|
||||
}
|
||||
}
|
||||
}
|
||||
return sb.String()
|
||||
}
|
||||
|
||||
func translateSystemPrompt(schema provider.SchemaContext) string {
|
||||
return fmt.Sprintf(`You translate a plain-English question into a Sentry pipe-syntax query. You never explain, never execute anything, never write raw SQL unless the pipe syntax genuinely cannot express the request.
|
||||
|
||||
%s
|
||||
|
||||
%s
|
||||
|
||||
Respond with ONLY a JSON object, no other text, no markdown fences:
|
||||
{"query": "<the pipe-syntax query>", "confidence": "high"|"medium"|"low", "reason": "<empty unless confidence is low, in which case explain what's ambiguous or unsupported>"}
|
||||
|
||||
If you cannot produce a query you're reasonably confident in, set confidence to "low", leave query empty, and explain why in reason. Never guess with false confidence.`, grammarReference, renderSchema(schema))
|
||||
}
|
||||
|
||||
func completeSystemPrompt(schema provider.SchemaContext) string {
|
||||
return fmt.Sprintf(`You suggest how to continue a partially-typed Sentry query. You are given everything typed so far; respond with ONLY the suggested continuation text (what should appear after the cursor), not the text already typed, not an explanation.
|
||||
|
||||
%s
|
||||
|
||||
%s
|
||||
|
||||
Respond with ONLY a JSON object, no other text, no markdown fences:
|
||||
{"suggestion": "<continuation text, or empty string if you have no good suggestion>"}`, grammarReference, renderSchema(schema))
|
||||
}
|
||||
|
||||
// explainSystemPrompt covers all three contexts provider.ExplainRequest
|
||||
// supports: a plain hand-written-query explanation (both empty), a
|
||||
// post-translation review (hasIntent), or Optimize's "phrase these
|
||||
// findings" mode (hasFindings) -- mutually exclusive in practice, see
|
||||
// ExplainRequest.RuleFindings' doc comment.
|
||||
func explainSystemPrompt(hasIntent, hasFindings bool) string {
|
||||
if hasFindings {
|
||||
return fmt.Sprintf(`A rule-based check already found one or more real issues with a Sentry query's efficiency (e.g. a missing time range). Your only job is to phrase those findings as a short, clear, actionable suggestion for the person who wrote the query -- do not invent additional issues, do not restate the query's own syntax back at them, do not hedge with "might" or "could" about something the check already confirmed. One or two sentences.
|
||||
|
||||
%s`, grammarReference)
|
||||
}
|
||||
|
||||
base := fmt.Sprintf(`You explain what a Sentry query does in plain English, for someone who may not know the query language. Be concise -- two or three sentences, not a line-by-line breakdown unless the query is unusually complex.
|
||||
|
||||
%s`, grammarReference)
|
||||
if hasIntent {
|
||||
base += "\n\nYou are explaining a query that was just generated from a natural-language request. Focus on how the request became this query -- call out any interpretation choices (e.g. how a vague time phrase or field reference was resolved), not just what the query does in isolation."
|
||||
}
|
||||
return base
|
||||
}
|
||||
|
||||
func fixSystemPrompt(schema provider.SchemaContext) string {
|
||||
return fmt.Sprintf(`You fix a broken Sentry query given its error message. Produce a corrected query and a short explanation of what was wrong.
|
||||
|
||||
%s
|
||||
|
||||
%s
|
||||
|
||||
Respond with ONLY a JSON object, no other text, no markdown fences:
|
||||
{"suggested_query": "<corrected query>", "explanation": "<short explanation of what was wrong and what changed>", "confidence": "high"|"medium"|"low"}
|
||||
|
||||
If you cannot determine a fix, set confidence to "low" and suggested_query to an empty string.`, grammarReference, renderSchema(schema))
|
||||
}
|
||||
@@ -0,0 +1,179 @@
|
||||
// Package provider defines the model-provider abstraction every AI-assisted
|
||||
// query feature (Phase 7) is built on: translate, complete, explain, fix.
|
||||
// Same narrow-interface pattern as querylang/executor's SQLRunner/
|
||||
// SearchClient -- a small interface a production implementation
|
||||
// (provider/ollama, the default; a cloud adapter, opt-in) and a fake
|
||||
// (for tests) both satisfy, so nothing above this layer needs to know or
|
||||
// care which model actually answered.
|
||||
//
|
||||
// What this package deliberately does NOT do: decide *which* provider or
|
||||
// model answers a given request. That's a routing concern (per-operation
|
||||
// config, per-tenant cloud opt-in) that lives one layer up, once task 3/4
|
||||
// land -- this package only defines the shape every provider must speak.
|
||||
//
|
||||
// Every operation is grounded (SchemaContext) and every result that
|
||||
// produces a query is designed to flow through the unchanged Phase 2
|
||||
// planner.Compile -> executor.Execute path before it ever runs -- this
|
||||
// package returns query *text*, never executes anything itself. See
|
||||
// /docs/phase-7-ai-design.md for the full design this interface was
|
||||
// built against.
|
||||
package provider
|
||||
|
||||
import "context"
|
||||
|
||||
// SchemaContext is the grounding data every operation receives -- known
|
||||
// service names, field names (structured columns plus common attribute
|
||||
// keys), and value examples for enum-like fields (severity, status, and
|
||||
// so on). Sourced from ClickHouse system tables / periodic sampling
|
||||
// (task 3), never hand-maintained, and always scoped to the requesting
|
||||
// tenant's own data -- a provider implementation must never be handed
|
||||
// another tenant's grounding data, the same connection-layer-isolation
|
||||
// discipline Phase 4 applies to query execution itself. This package
|
||||
// doesn't resolve SchemaContext; callers (the schema/metadata service,
|
||||
// task 3) build it and pass it in, so a Provider implementation never
|
||||
// needs ClickHouse access of its own.
|
||||
type SchemaContext struct {
|
||||
Services []string
|
||||
// Fields covers both real columns (timestamp, host, service,
|
||||
// severity, message, record_id) and the common attribute keys seen
|
||||
// in the tenant's own data -- see /docs/query-language-reference.md's
|
||||
// "Field mapping" section for why the distinction mostly doesn't
|
||||
// matter to a query author, and shouldn't need to matter to the model
|
||||
// either.
|
||||
Fields []FieldInfo
|
||||
}
|
||||
|
||||
type FieldInfo struct {
|
||||
Name string
|
||||
// Examples is a short, representative sample of real values seen for
|
||||
// this field -- most useful for enum-like fields (severity, status)
|
||||
// where showing the model the actual vocabulary beats describing it.
|
||||
// Empty for high-cardinality fields (host, message) where examples
|
||||
// wouldn't help and would just spend context budget.
|
||||
Examples []string
|
||||
}
|
||||
|
||||
// Confidence is deliberately a small enum, not a raw float -- a model's
|
||||
// self-reported numeric confidence isn't a calibrated probability, and
|
||||
// pretending it is (via e.g. "reject anything under 0.73") invites false
|
||||
// precision. Three bands are enough to drive real UI behavior (task 10's
|
||||
// "handle low-confidence translation honestly") without pretending to
|
||||
// more precision than a model's self-assessment actually has.
|
||||
type Confidence string
|
||||
|
||||
const (
|
||||
ConfidenceHigh Confidence = "high"
|
||||
ConfidenceMedium Confidence = "medium"
|
||||
ConfidenceLow Confidence = "low"
|
||||
)
|
||||
|
||||
type TranslateRequest struct {
|
||||
NLQuery string
|
||||
Schema SchemaContext
|
||||
}
|
||||
|
||||
type TranslateResult struct {
|
||||
// Query is Phase 2 pipe-syntax, never raw SQL -- the phase brief's
|
||||
// explicit "narrower, safer surface" choice for generation targets.
|
||||
// A provider that can't produce a valid completion should return an
|
||||
// error, not a best-effort raw-SQL fallback.
|
||||
Query string
|
||||
Confidence Confidence
|
||||
// LowConfidenceReason is set (and Query may be empty) when the
|
||||
// provider can't produce a translation it's willing to stand behind
|
||||
// at all -- task 10 wants this said plainly, not papered over with a
|
||||
// guess. Empty when Confidence is High or Medium.
|
||||
LowConfidenceReason string
|
||||
}
|
||||
|
||||
type CompleteRequest struct {
|
||||
// QueryPrefix is everything the user has typed so far, cursor at the
|
||||
// end -- this operation is a full-completion suggestion (ghost text),
|
||||
// not a fill-in-the-middle edit, matching how the query bar's cursor
|
||||
// behaves (Phase 5's QueryEditor.svelte, always append-at-cursor).
|
||||
QueryPrefix string
|
||||
Language string // "spl" or "sql", never "" -- the caller has always already resolved auto-detection by this point
|
||||
Schema SchemaContext
|
||||
}
|
||||
|
||||
type CompleteResult struct {
|
||||
// Suggestion is the suggested continuation only (what ghost-text
|
||||
// should render after the cursor), not QueryPrefix+continuation
|
||||
// restated -- keeps the caller from having to diff its own input
|
||||
// back out of the result.
|
||||
Suggestion string
|
||||
// Empty Suggestion (with no error) is a legitimate response -- "no
|
||||
// good completion here" is not the same failure mode as a timeout or
|
||||
// a down provider, and the caller (task 5's fallback logic) needs to
|
||||
// tell them apart.
|
||||
}
|
||||
|
||||
type ExplainRequest struct {
|
||||
Query string
|
||||
Language string
|
||||
// OriginalIntent, when non-empty, means this Explain call is
|
||||
// reviewing a just-translated query (Track B) rather than an
|
||||
// arbitrary hand-written one (Track A) -- same operation, task 10's
|
||||
// explicit "reuse explain rather than build a separate mechanism"
|
||||
// choice, but the prompt can speak to *how the NL became this query*
|
||||
// instead of only describing the query in isolation.
|
||||
OriginalIntent string
|
||||
// RuleFindings, when non-empty, means this Explain call is task 8's
|
||||
// Optimize suggestion: rule-based detection (costguard) already found
|
||||
// something worth flagging, and the model's only job is phrasing
|
||||
// those specific findings clearly for a user -- not describing what
|
||||
// the query does, not detecting the inefficiency itself. Mutually
|
||||
// exclusive with OriginalIntent in practice (a query is either being
|
||||
// explained, reviewed post-translation, or optimized), but the type
|
||||
// doesn't need to enforce that -- three prompt-shaping contexts for
|
||||
// one operation, matching the same reuse-over-duplication choice
|
||||
// OriginalIntent already made rather than adding a fifth Provider
|
||||
// method for what is still, underneath, "explain something about
|
||||
// this query in plain English."
|
||||
RuleFindings []string
|
||||
}
|
||||
|
||||
type ExplainResult struct {
|
||||
Explanation string
|
||||
}
|
||||
|
||||
type FixRequest struct {
|
||||
Query string
|
||||
Language string
|
||||
// ParseError is set when the query never compiled at all (planner
|
||||
// error text); ExecutionError is set when it compiled but failed at
|
||||
// runtime (executor/ClickHouse error text). Exactly one is set --
|
||||
// the two failure modes want different framing ("this doesn't parse
|
||||
// because..." vs. "this ran but...").
|
||||
ParseError string
|
||||
ExecutionError string
|
||||
Schema SchemaContext
|
||||
}
|
||||
|
||||
type FixResult struct {
|
||||
// SuggestedQuery is the full corrected query text, always shown as a
|
||||
// diff against the original by the caller (task 7's explicit
|
||||
// "never silently applied" requirement) -- this package only
|
||||
// produces the suggestion, the UI owns the diff rendering and the
|
||||
// accept/dismiss decision.
|
||||
SuggestedQuery string
|
||||
// Explanation is a short plain-English note on what was wrong and
|
||||
// what changed -- distinct from Explain's job (describing what a
|
||||
// query *does*), this describes what was *fixed* and why.
|
||||
Explanation string
|
||||
Confidence Confidence
|
||||
}
|
||||
|
||||
// Provider is what every model backend implements: the default
|
||||
// self-hosted Ollama provider, the opt-in cloud adapter, and a fake for
|
||||
// tests. Every method takes a context so a caller can enforce the tight
|
||||
// latency budget Complete needs (task 5) without the interface itself
|
||||
// hard-coding a timeout -- that's a caller concern, since the right
|
||||
// timeout differs by operation (Complete's is much tighter than
|
||||
// Translate's).
|
||||
type Provider interface {
|
||||
Translate(ctx context.Context, req TranslateRequest) (TranslateResult, error)
|
||||
Complete(ctx context.Context, req CompleteRequest) (CompleteResult, error)
|
||||
Explain(ctx context.Context, req ExplainRequest) (ExplainResult, error)
|
||||
Fix(ctx context.Context, req FixRequest) (FixResult, error)
|
||||
}
|
||||
@@ -0,0 +1,62 @@
|
||||
// Package router is the per-operation provider/model dispatch layer
|
||||
// /docs/phase-7-ai-design.md's "per-operation provider/model
|
||||
// configuration" section decided to build now rather than defer --
|
||||
// Complete's tight latency budget and Translate/Fix's quality needs are
|
||||
// already in tension under a single-model-for-everything design, not a
|
||||
// hypothetical future conflict.
|
||||
//
|
||||
// Deliberately thin: a lookup table from Operation to whichever
|
||||
// provider.Provider was configured for it, falling back to one default
|
||||
// when an operation has no specific override. This package makes no
|
||||
// decisions about *which* provider is good for an operation -- that's
|
||||
// deployment configuration, resolved once at startup by whoever
|
||||
// constructs a Router (main.go, once the AI HTTP handlers exist to
|
||||
// consume it).
|
||||
package router
|
||||
|
||||
import "github.com/sentry/sentry/api/ai/provider"
|
||||
|
||||
type Operation string
|
||||
|
||||
const (
|
||||
OpTranslate Operation = "translate"
|
||||
OpComplete Operation = "complete"
|
||||
OpExplain Operation = "explain"
|
||||
OpFix Operation = "fix"
|
||||
)
|
||||
|
||||
// Router selects a provider.Provider per Operation. Not itself a
|
||||
// provider.Provider -- callers ask For(op) and then call the operation
|
||||
// they actually need on the result, rather than this type trying to
|
||||
// implement all four methods and dispatch internally, which would just
|
||||
// be an extra layer of indirection for no benefit.
|
||||
type Router struct {
|
||||
byOp map[Operation]provider.Provider
|
||||
fallback provider.Provider
|
||||
}
|
||||
|
||||
// New builds a Router. fallback must not be nil -- every operation
|
||||
// resolves to *some* provider, even a deployment that never calls
|
||||
// SetOperation and just wants one model for everything.
|
||||
func New(fallback provider.Provider) *Router {
|
||||
return &Router{byOp: make(map[Operation]provider.Provider), fallback: fallback}
|
||||
}
|
||||
|
||||
// SetOperation overrides which provider handles op. Call once per
|
||||
// operation that needs a non-default model at startup configuration
|
||||
// time -- not intended to change at runtime (a Router isn't
|
||||
// synchronized for concurrent SetOperation/For calls, matching every
|
||||
// other "assembled once in main.go, read-only after that" config shape
|
||||
// in this codebase, e.g. Handler's fields in queryapi).
|
||||
func (r *Router) SetOperation(op Operation, p provider.Provider) {
|
||||
r.byOp[op] = p
|
||||
}
|
||||
|
||||
// For returns the provider configured for op, or the fallback if none
|
||||
// was set specifically.
|
||||
func (r *Router) For(op Operation) provider.Provider {
|
||||
if p, ok := r.byOp[op]; ok {
|
||||
return p
|
||||
}
|
||||
return r.fallback
|
||||
}
|
||||
@@ -0,0 +1,52 @@
|
||||
package router
|
||||
|
||||
import (
|
||||
"context"
|
||||
"testing"
|
||||
|
||||
"github.com/sentry/sentry/api/ai/provider"
|
||||
)
|
||||
|
||||
// namedFakeProvider lets a test tell which configured provider actually
|
||||
// answered a call -- the thing router.For needs to get right.
|
||||
type namedFakeProvider struct {
|
||||
name string
|
||||
}
|
||||
|
||||
func (f *namedFakeProvider) Translate(context.Context, provider.TranslateRequest) (provider.TranslateResult, error) {
|
||||
return provider.TranslateResult{Query: f.name}, nil
|
||||
}
|
||||
func (f *namedFakeProvider) Complete(context.Context, provider.CompleteRequest) (provider.CompleteResult, error) {
|
||||
return provider.CompleteResult{Suggestion: f.name}, nil
|
||||
}
|
||||
func (f *namedFakeProvider) Explain(context.Context, provider.ExplainRequest) (provider.ExplainResult, error) {
|
||||
return provider.ExplainResult{Explanation: f.name}, nil
|
||||
}
|
||||
func (f *namedFakeProvider) Fix(context.Context, provider.FixRequest) (provider.FixResult, error) {
|
||||
return provider.FixResult{SuggestedQuery: f.name}, nil
|
||||
}
|
||||
|
||||
var _ provider.Provider = (*namedFakeProvider)(nil)
|
||||
|
||||
func TestForReturnsFallbackWhenUnconfigured(t *testing.T) {
|
||||
fallback := &namedFakeProvider{name: "default"}
|
||||
r := New(fallback)
|
||||
|
||||
if got := r.For(OpTranslate); got != fallback {
|
||||
t.Errorf("For(OpTranslate) = %v, want the fallback provider", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestSetOperationOverridesFallback(t *testing.T) {
|
||||
fallback := &namedFakeProvider{name: "default"}
|
||||
fast := &namedFakeProvider{name: "fast"}
|
||||
r := New(fallback)
|
||||
r.SetOperation(OpComplete, fast)
|
||||
|
||||
if got := r.For(OpComplete); got != fast {
|
||||
t.Errorf("For(OpComplete) = %v, want the fast override", got)
|
||||
}
|
||||
if got := r.For(OpTranslate); got != fallback {
|
||||
t.Errorf("For(OpTranslate) = %v, want unaffected fallback", got)
|
||||
}
|
||||
}
|
||||
@@ -19,6 +19,10 @@ import (
|
||||
"github.com/ClickHouse/clickhouse-go/v2"
|
||||
"github.com/jackc/pgx/v5/pgxpool"
|
||||
|
||||
"github.com/sentry/sentry/api/ai/aiapi"
|
||||
"github.com/sentry/sentry/api/ai/grounding"
|
||||
"github.com/sentry/sentry/api/ai/provider/ollama"
|
||||
"github.com/sentry/sentry/api/ai/router"
|
||||
"github.com/sentry/sentry/api/authz"
|
||||
"github.com/sentry/sentry/api/dashboards"
|
||||
"github.com/sentry/sentry/api/httpserver"
|
||||
@@ -28,6 +32,13 @@ import (
|
||||
"github.com/sentry/sentry/api/searchclient"
|
||||
)
|
||||
|
||||
// groundingRefreshInterval matches chwriter.Registry/search's
|
||||
// ActiveTenantTracker's own one-minute refresh cadence -- no strong
|
||||
// reason for a different number, and consistency means one interval to
|
||||
// reason about across every "sample something periodically" mechanism
|
||||
// in this codebase, not several slightly different ones.
|
||||
const groundingRefreshInterval = time.Minute
|
||||
|
||||
func main() {
|
||||
logger := slog.New(slog.NewJSONHandler(os.Stdout, nil))
|
||||
|
||||
@@ -116,6 +127,32 @@ func main() {
|
||||
queryHandler.RegisterRoutes(mux)
|
||||
dashboardsHandler.RegisterRoutes(mux)
|
||||
|
||||
// AI routes (Phase 7) are only registered at all when OLLAMA_BASE_URL
|
||||
// is set -- an unconfigured deployment gets a plain 404 on /ai/*
|
||||
// rather than every request failing against an unreachable
|
||||
// localhost:11434, matching "no cloud dependency required for the
|
||||
// default deployment" by not forcing a *local* model dependency on a
|
||||
// deployment that doesn't want AI features either.
|
||||
if cfg.AI.OllamaBaseURL != "" {
|
||||
groundingSvc := grounding.New(sqlRunner)
|
||||
groundingSvc.StartRefreshing(ctx, groundingRefreshInterval, func(err error) {
|
||||
logger.Warn("grounding refresh failed", "error", err)
|
||||
})
|
||||
|
||||
defaultProvider := ollama.New(cfg.AI.OllamaBaseURL, cfg.AI.OllamaModel)
|
||||
aiRouter := router.New(defaultProvider)
|
||||
if cfg.AI.OllamaFastModel != "" && cfg.AI.OllamaFastModel != cfg.AI.OllamaModel {
|
||||
aiRouter.SetOperation(router.OpComplete, ollama.New(cfg.AI.OllamaBaseURL, cfg.AI.OllamaFastModel))
|
||||
}
|
||||
|
||||
// nil interaction logger: core has no enterprise/internal/audit
|
||||
// implementation to log translate/fix/optimize interactions
|
||||
// against, same posture as queryHandler's nil audit logger above.
|
||||
aiHandler := aiapi.NewHandler(logger, aiRouter, groundingSvc, authorizer, nil)
|
||||
aiHandler.RegisterRoutes(mux)
|
||||
logger.Info("ai routes enabled", "ollama_base_url", cfg.AI.OllamaBaseURL, "model", cfg.AI.OllamaModel)
|
||||
}
|
||||
|
||||
srv := &http.Server{
|
||||
Addr: cfg.HTTPListenAddr,
|
||||
Handler: httpserver.WithCORS(mux, cfg.CORSAllowedOrigin),
|
||||
|
||||
@@ -17,6 +17,20 @@ type Config struct {
|
||||
QueryTimeout time.Duration
|
||||
CORSAllowedOrigin string
|
||||
EnterpriseAuthURL string
|
||||
AI AIConfig
|
||||
}
|
||||
|
||||
// AIConfig gates Phase 7's AI-assisted query features (Track A/B) --
|
||||
// off unless OllamaBaseURL is set, same "off unless configured"
|
||||
// convention as EnterpriseAuthURL and everything else optional in this
|
||||
// codebase. OllamaFastModel is the per-operation override for
|
||||
// Complete's tight latency budget (/docs/phase-7-ai-design.md's
|
||||
// per-operation provider/model config) -- empty means Complete uses
|
||||
// OllamaModel too, same as every other operation.
|
||||
type AIConfig struct {
|
||||
OllamaBaseURL string
|
||||
OllamaModel string
|
||||
OllamaFastModel string
|
||||
}
|
||||
|
||||
type ClickHouseConfig struct {
|
||||
@@ -65,6 +79,17 @@ func Load() (Config, error) {
|
||||
// base URL (e.g. "http://enterprise-auth:8081") to turn on
|
||||
// real session/service-token enforcement.
|
||||
EnterpriseAuthURL: getenv("ENTERPRISE_AUTH_URL", ""),
|
||||
// Empty OllamaBaseURL means AI features are entirely disabled --
|
||||
// /ai/* routes aren't even registered (see main.go), matching
|
||||
// "no cloud dependency required for the default deployment" and,
|
||||
// by the same reasoning, no *local* model dependency forced on a
|
||||
// deployment that doesn't want one either. Model names default to
|
||||
// the recommendation confirmed in /docs/phase-7-ai-design.md.
|
||||
AI: AIConfig{
|
||||
OllamaBaseURL: getenv("OLLAMA_BASE_URL", ""),
|
||||
OllamaModel: getenv("OLLAMA_MODEL", "qwen2.5-coder:7b"),
|
||||
OllamaFastModel: getenv("OLLAMA_FAST_MODEL", "qwen2.5-coder:1.5b"),
|
||||
},
|
||||
}
|
||||
|
||||
timeoutSec, err := strconv.Atoi(getenv("QUERY_TIMEOUT_SECONDS", "30"))
|
||||
|
||||
+20
-1
@@ -19,6 +19,7 @@ import (
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"github.com/sentry/sentry/api/ai/costguard"
|
||||
"github.com/sentry/sentry/api/authz"
|
||||
"github.com/sentry/sentry/api/internal/querylang/planner"
|
||||
"github.com/sentry/sentry/api/querylang/executor"
|
||||
@@ -99,6 +100,20 @@ type queryRequest struct {
|
||||
type queryResponse struct {
|
||||
Columns []string `json:"columns"`
|
||||
Rows [][]any `json:"rows"`
|
||||
// Warnings surfaces costguard's assessment (Phase 7 task 4) for
|
||||
// every query, hand-written or AI-suggested alike -- the same
|
||||
// guard, never a hard block here. AI-suggested queries get a
|
||||
// stricter treatment (a Reject-level assessment withholds the
|
||||
// suggestion entirely, see the ai package) before a query ever
|
||||
// reaches this handler; a hand-written query submitted directly
|
||||
// always runs regardless of what this says, matching every prior
|
||||
// phase's behavior -- this field is informational, not new
|
||||
// enforcement, a deliberate choice recorded in
|
||||
// /docs/phase-7-ai-design.md rather than a retrofit nobody decided
|
||||
// on. Omitted (not an empty array) when there's nothing to say, so
|
||||
// existing callers that don't look for this field see no shape
|
||||
// change at all.
|
||||
Warnings []string `json:"warnings,omitempty"`
|
||||
}
|
||||
|
||||
type errorResponse struct {
|
||||
@@ -149,7 +164,11 @@ func (h *Handler) handleQuery(w http.ResponseWriter, r *http.Request) {
|
||||
}
|
||||
|
||||
h.logAudit(r.Context(), req, len(result.Rows), duration, nil)
|
||||
writeJSON(w, queryResponse{Columns: result.Columns, Rows: result.Rows})
|
||||
resp := queryResponse{Columns: result.Columns, Rows: result.Rows}
|
||||
if assessment := costguard.Assess(plan); assessment.Level != costguard.LevelOK {
|
||||
resp.Warnings = []string{costguard.Summary(assessment)}
|
||||
}
|
||||
writeJSON(w, resp)
|
||||
}
|
||||
|
||||
// logAudit is fail-open by design (see AuditLogger's doc comment): a
|
||||
|
||||
@@ -1,6 +1,7 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"bufio"
|
||||
"bytes"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
@@ -16,6 +17,13 @@ type queryArgs struct {
|
||||
jsonOut bool
|
||||
language string
|
||||
query string
|
||||
// nlQuery, execute (Phase 7 task 11): --nl routes through
|
||||
// POST /ai/translate instead of running query text directly.
|
||||
// execute is the same "explicit opt-in to actually run this"
|
||||
// posture the web UI's confirm-to-run action enforces -- a
|
||||
// translated query never runs itself, here or there.
|
||||
nlQuery string
|
||||
execute bool
|
||||
}
|
||||
|
||||
// parseQueryArgs is pure (env passed in, no I/O), same testability
|
||||
@@ -41,6 +49,13 @@ func parseQueryArgs(args []string, env func(string) string) queryArgs {
|
||||
qa.language = args[i+1]
|
||||
i++
|
||||
}
|
||||
case "--nl":
|
||||
if i+1 < len(args) {
|
||||
qa.nlQuery = args[i+1]
|
||||
i++
|
||||
}
|
||||
case "--execute":
|
||||
qa.execute = true
|
||||
default:
|
||||
rest = append(rest, args[i])
|
||||
}
|
||||
@@ -57,22 +72,159 @@ type queryRequestBody struct {
|
||||
type queryResponseBody struct {
|
||||
Columns []string `json:"columns"`
|
||||
Rows [][]any `json:"rows"`
|
||||
Warnings []string `json:"warnings"`
|
||||
}
|
||||
|
||||
type translateRequestBody struct {
|
||||
NLQuery string `json:"nlQuery"`
|
||||
}
|
||||
|
||||
type translateResponseBody struct {
|
||||
Query string `json:"query"`
|
||||
Confidence string `json:"confidence"`
|
||||
LowConfidenceReason string `json:"lowConfidenceReason"`
|
||||
Compiles bool `json:"compiles"`
|
||||
CompileError string `json:"compileError"`
|
||||
Blocked bool `json:"blocked"`
|
||||
CostWarnings []string `json:"costWarnings"`
|
||||
}
|
||||
|
||||
func cmdQuery(args []string, stdout, stderr io.Writer) int {
|
||||
qa := parseQueryArgs(args, os.Getenv)
|
||||
|
||||
if qa.nlQuery != "" {
|
||||
return cmdQueryNL(qa, stdout, stderr, os.Stdin)
|
||||
}
|
||||
|
||||
if strings.TrimSpace(qa.query) == "" {
|
||||
fmt.Fprintln(stderr, "sentryctl query: missing query string")
|
||||
return 1
|
||||
}
|
||||
return runAndPrintQuery(qa.apiURL, qa.query, qa.language, qa.jsonOut, stdout, stderr)
|
||||
}
|
||||
|
||||
reqBody, err := json.Marshal(queryRequestBody{Query: qa.query, Language: qa.language})
|
||||
// cmdQueryNL implements --nl: translate, show the result, then only run
|
||||
// it with explicit opt-in (--execute, or an interactive "y" confirmation
|
||||
// -- never a bare unattended run). stdin is a parameter (not read from
|
||||
// os.Stdin directly) so the confirmation prompt is testable the same
|
||||
// way parseQueryArgs's env injection is.
|
||||
func cmdQueryNL(qa queryArgs, stdout, stderr io.Writer, stdin io.Reader) int {
|
||||
reqBody, err := json.Marshal(translateRequestBody{NLQuery: qa.nlQuery})
|
||||
if err != nil {
|
||||
fmt.Fprintf(stderr, "encoding request: %v\n", err)
|
||||
return 1
|
||||
}
|
||||
req, err := http.NewRequest(http.MethodPost, qa.apiURL+"/ai/translate", bytes.NewReader(reqBody))
|
||||
if err != nil {
|
||||
fmt.Fprintf(stderr, "building request: %v\n", err)
|
||||
return 1
|
||||
}
|
||||
req.Header.Set("Content-Type", "application/json")
|
||||
setAuth(req, resolveToken(os.Getenv))
|
||||
|
||||
client := &http.Client{Timeout: 30 * time.Second}
|
||||
resp, err := client.Do(req)
|
||||
if err != nil {
|
||||
fmt.Fprintf(stderr, "translation failed: %v\n", err)
|
||||
return 1
|
||||
}
|
||||
defer resp.Body.Close()
|
||||
|
||||
respBody, err := io.ReadAll(resp.Body)
|
||||
if err != nil {
|
||||
fmt.Fprintf(stderr, "reading response: %v\n", err)
|
||||
return 1
|
||||
}
|
||||
if resp.StatusCode != http.StatusOK {
|
||||
var errResp errorResponseBody
|
||||
if json.Unmarshal(respBody, &errResp) == nil && errResp.Error != "" {
|
||||
fmt.Fprintf(stderr, "translation failed: %s\n", errResp.Error)
|
||||
} else {
|
||||
fmt.Fprintf(stderr, "translation failed: api returned status %d\n", resp.StatusCode)
|
||||
}
|
||||
return 1
|
||||
}
|
||||
|
||||
var t translateResponseBody
|
||||
if err := json.Unmarshal(respBody, &t); err != nil {
|
||||
fmt.Fprintf(stderr, "decoding response: %v\n", err)
|
||||
return 1
|
||||
}
|
||||
|
||||
if t.Query == "" {
|
||||
reason := t.LowConfidenceReason
|
||||
if reason == "" {
|
||||
reason = "the model did not return a query"
|
||||
}
|
||||
fmt.Fprintf(stdout, "No confident translation available: %s\n", reason)
|
||||
return 1
|
||||
}
|
||||
|
||||
fmt.Fprintf(stdout, "Translated query (%s confidence):\n %s\n", t.Confidence, t.Query)
|
||||
if !t.Compiles {
|
||||
fmt.Fprintf(stdout, "This does not parse as a valid query: %s\n", t.CompileError)
|
||||
fmt.Fprintln(stdout, "Not running it -- copy, fix, and run manually if you want to use it.")
|
||||
return 1
|
||||
}
|
||||
if len(t.CostWarnings) > 0 {
|
||||
fmt.Fprintf(stdout, "Cost guard: %s\n", strings.Join(t.CostWarnings, "; "))
|
||||
}
|
||||
if t.Blocked {
|
||||
fmt.Fprintln(stdout, "Not offered as directly runnable -- copy and adjust manually if you want to use it.")
|
||||
return 1
|
||||
}
|
||||
|
||||
if !qa.execute {
|
||||
if !isInteractive(stdin) {
|
||||
fmt.Fprintln(stdout, "Not running (pass --execute to run automatically, or run this interactively to confirm).")
|
||||
return 0
|
||||
}
|
||||
if !confirmRun(stdin, stdout, "Run this query?") {
|
||||
fmt.Fprintln(stdout, "Not running.")
|
||||
return 0
|
||||
}
|
||||
}
|
||||
|
||||
return runAndPrintQuery(qa.apiURL, t.Query, "spl", qa.jsonOut, stdout, stderr)
|
||||
}
|
||||
|
||||
// confirmRun prompts stdin for a y/N answer -- only "y"/"yes"
|
||||
// (case-insensitive) counts as confirmation, matching the web UI's
|
||||
// posture that running an AI-generated query is opt-in, never a
|
||||
// default a blank Enter press could accidentally trigger.
|
||||
func confirmRun(stdin io.Reader, stdout io.Writer, prompt string) bool {
|
||||
fmt.Fprintf(stdout, "%s [y/N] ", prompt)
|
||||
line, _ := bufio.NewReader(stdin).ReadString('\n')
|
||||
answer := strings.ToLower(strings.TrimSpace(line))
|
||||
return answer == "y" || answer == "yes"
|
||||
}
|
||||
|
||||
// isInteractive reports whether stdin looks like a real terminal --
|
||||
// used to decide whether a confirmation prompt makes sense at all
|
||||
// (a non-interactive/piped invocation with no --execute would otherwise
|
||||
// hang forever waiting for an answer nobody can give; refusing to run
|
||||
// and exiting cleanly is the safe default there, matching --execute's
|
||||
// own opt-in-required posture rather than silently running).
|
||||
func isInteractive(stdin io.Reader) bool {
|
||||
f, ok := stdin.(*os.File)
|
||||
if !ok {
|
||||
return false
|
||||
}
|
||||
info, err := f.Stat()
|
||||
if err != nil {
|
||||
return false
|
||||
}
|
||||
return info.Mode()&os.ModeCharDevice != 0
|
||||
}
|
||||
|
||||
func runAndPrintQuery(apiURL, query, language string, jsonOut bool, stdout, stderr io.Writer) int {
|
||||
reqBody, err := json.Marshal(queryRequestBody{Query: query, Language: language})
|
||||
if err != nil {
|
||||
fmt.Fprintf(stderr, "encoding request: %v\n", err)
|
||||
return 1
|
||||
}
|
||||
|
||||
req, err := http.NewRequest(http.MethodPost, qa.apiURL+"/query", bytes.NewReader(reqBody))
|
||||
req, err := http.NewRequest(http.MethodPost, apiURL+"/query", bytes.NewReader(reqBody))
|
||||
if err != nil {
|
||||
fmt.Fprintf(stderr, "building request: %v\n", err)
|
||||
return 1
|
||||
@@ -104,7 +256,7 @@ func cmdQuery(args []string, stdout, stderr io.Writer) int {
|
||||
return 1
|
||||
}
|
||||
|
||||
if qa.jsonOut {
|
||||
if jsonOut {
|
||||
_, _ = stdout.Write(respBody)
|
||||
fmt.Fprintln(stdout)
|
||||
return 0
|
||||
@@ -116,5 +268,8 @@ func cmdQuery(args []string, stdout, stderr io.Writer) int {
|
||||
return 1
|
||||
}
|
||||
printTable(stdout, result.Columns, result.Rows)
|
||||
if len(result.Warnings) > 0 {
|
||||
fmt.Fprintf(stdout, "\nWarning: %s\n", strings.Join(result.Warnings, "; "))
|
||||
}
|
||||
return 0
|
||||
}
|
||||
|
||||
@@ -0,0 +1,145 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func TestParseQueryArgsNL(t *testing.T) {
|
||||
qa := parseQueryArgs([]string{"--nl", "errors in the last hour", "--execute"}, func(string) string { return "" })
|
||||
if qa.nlQuery != "errors in the last hour" {
|
||||
t.Errorf("nlQuery = %q", qa.nlQuery)
|
||||
}
|
||||
if !qa.execute {
|
||||
t.Error("execute = false, want true")
|
||||
}
|
||||
}
|
||||
|
||||
func TestCmdQueryNLLowConfidenceDoesNotRun(t *testing.T) {
|
||||
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
if r.URL.Path != "/ai/translate" {
|
||||
t.Errorf("unexpected request to %s, want only /ai/translate (never /query)", r.URL.Path)
|
||||
}
|
||||
w.Header().Set("Content-Type", "application/json")
|
||||
w.Write([]byte(`{"query":"","confidence":"low","lowConfidenceReason":"not sure what that means"}`))
|
||||
}))
|
||||
defer srv.Close()
|
||||
|
||||
var stdout, stderr bytes.Buffer
|
||||
code := cmdQuery([]string{"--nl", "show me weird stuff", "--execute", "--api", srv.URL}, &stdout, &stderr)
|
||||
if code != 1 {
|
||||
t.Errorf("code = %d, want 1 (no confident translation)", code)
|
||||
}
|
||||
if !strings.Contains(stdout.String(), "not sure what that means") {
|
||||
t.Errorf("stdout = %q, want the low-confidence reason", stdout.String())
|
||||
}
|
||||
}
|
||||
|
||||
func TestCmdQueryNLBlockedIsNotRunEvenWithExecute(t *testing.T) {
|
||||
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
if r.URL.Path == "/query" {
|
||||
t.Error("a blocked translation must never reach /query, even with --execute")
|
||||
}
|
||||
w.Header().Set("Content-Type", "application/json")
|
||||
w.Write([]byte(`{"query":"severity=ERROR | stats count by service","confidence":"high","compiles":true,"blocked":true,"costWarnings":["no time range filter, and this query aggregates"]}`))
|
||||
}))
|
||||
defer srv.Close()
|
||||
|
||||
var stdout, stderr bytes.Buffer
|
||||
code := cmdQuery([]string{"--nl", "errors by service", "--execute", "--api", srv.URL}, &stdout, &stderr)
|
||||
if code != 1 {
|
||||
t.Errorf("code = %d, want 1 (blocked)", code)
|
||||
}
|
||||
if !strings.Contains(stdout.String(), "Not offered as directly runnable") {
|
||||
t.Errorf("stdout = %q", stdout.String())
|
||||
}
|
||||
}
|
||||
|
||||
func TestCmdQueryNLNonCompilingDoesNotRun(t *testing.T) {
|
||||
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
if r.URL.Path == "/query" {
|
||||
t.Error("a non-compiling translation must never reach /query")
|
||||
}
|
||||
w.Header().Set("Content-Type", "application/json")
|
||||
w.Write([]byte(`{"query":"| stats count","confidence":"high","compiles":false,"compileError":"expected a filter, comparison, or search term, got PIPE"}`))
|
||||
}))
|
||||
defer srv.Close()
|
||||
|
||||
var stdout, stderr bytes.Buffer
|
||||
code := cmdQuery([]string{"--nl", "something odd", "--execute", "--api", srv.URL}, &stdout, &stderr)
|
||||
if code != 1 {
|
||||
t.Errorf("code = %d, want 1", code)
|
||||
}
|
||||
if !strings.Contains(stdout.String(), "does not parse") {
|
||||
t.Errorf("stdout = %q", stdout.String())
|
||||
}
|
||||
}
|
||||
|
||||
func TestCmdQueryNLWithExecuteRunsTheQuery(t *testing.T) {
|
||||
var sawQuery string
|
||||
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
w.Header().Set("Content-Type", "application/json")
|
||||
switch r.URL.Path {
|
||||
case "/ai/translate":
|
||||
w.Write([]byte(`{"query":"earliest=-1h severity=ERROR | stats count by service","confidence":"high","compiles":true,"blocked":false}`))
|
||||
case "/query":
|
||||
sawQuery = "called"
|
||||
w.Write([]byte(`{"columns":["service","count"],"rows":[["api",5]]}`))
|
||||
default:
|
||||
t.Errorf("unexpected request to %s", r.URL.Path)
|
||||
}
|
||||
}))
|
||||
defer srv.Close()
|
||||
|
||||
var stdout, stderr bytes.Buffer
|
||||
code := cmdQuery([]string{"--nl", "errors per service in the last hour", "--execute", "--api", srv.URL}, &stdout, &stderr)
|
||||
if code != 0 {
|
||||
t.Fatalf("code = %d, want 0; stderr=%s", code, stderr.String())
|
||||
}
|
||||
if sawQuery != "called" {
|
||||
t.Error("expected /query to be called with --execute set")
|
||||
}
|
||||
if !strings.Contains(stdout.String(), "api") {
|
||||
t.Errorf("stdout = %q, want the query results printed", stdout.String())
|
||||
}
|
||||
}
|
||||
|
||||
func TestCmdQueryNLWithoutExecuteNonInteractiveDoesNotRun(t *testing.T) {
|
||||
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
if r.URL.Path == "/query" {
|
||||
t.Error("must not run without --execute when stdin isn't a terminal")
|
||||
}
|
||||
w.Header().Set("Content-Type", "application/json")
|
||||
w.Write([]byte(`{"query":"earliest=-1h | stats count","confidence":"high","compiles":true,"blocked":false}`))
|
||||
}))
|
||||
defer srv.Close()
|
||||
|
||||
var stdout, stderr bytes.Buffer
|
||||
code := cmdQuery([]string{"--nl", "how many events", "--api", srv.URL}, &stdout, &stderr)
|
||||
if code != 0 {
|
||||
t.Errorf("code = %d, want 0 (declining to run isn't a failure)", code)
|
||||
}
|
||||
if !strings.Contains(stdout.String(), "Not running") {
|
||||
t.Errorf("stdout = %q", stdout.String())
|
||||
}
|
||||
}
|
||||
|
||||
func TestConfirmRunAcceptsY(t *testing.T) {
|
||||
var stdout bytes.Buffer
|
||||
if !confirmRun(strings.NewReader("y\n"), &stdout, "Run?") {
|
||||
t.Error("expected 'y' to confirm")
|
||||
}
|
||||
}
|
||||
|
||||
func TestConfirmRunRejectsBlankAndOther(t *testing.T) {
|
||||
var stdout bytes.Buffer
|
||||
if confirmRun(strings.NewReader("\n"), &stdout, "Run?") {
|
||||
t.Error("expected a blank line to NOT confirm")
|
||||
}
|
||||
if confirmRun(strings.NewReader("sure\n"), &stdout, "Run?") {
|
||||
t.Error("expected an unrecognized answer to NOT confirm")
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,655 @@
|
||||
# Phase 7 AI design: model provider architecture
|
||||
|
||||
Task 2 deliverable — the model-provider abstraction, presented for
|
||||
review before any implementation is built against it, per the phase
|
||||
brief's explicit stop point. Nothing past the interface itself
|
||||
(`api/ai/provider/provider.go`) has been built yet.
|
||||
|
||||
## Provider interface
|
||||
|
||||
`api/ai/provider/provider.go` (written, compiles, not yet consumed by
|
||||
anything). Four operations — `Translate`, `Complete`, `Explain`, `Fix` —
|
||||
one `Provider` interface, same narrow-interface-plus-fake pattern
|
||||
`querylang/executor`'s `SQLRunner`/`SearchClient` already established in
|
||||
this codebase, so a production implementation and a test fake both
|
||||
satisfy the same small surface.
|
||||
|
||||
Design choices worth calling out explicitly:
|
||||
|
||||
- **Every result that produces a query returns text, never executes
|
||||
anything.** The package has no dependency on `executor` or `planner`
|
||||
at all — a `TranslateResult`/`FixResult`'s query text is handed back
|
||||
to the caller, which is responsible for running it through the
|
||||
unchanged `planner.Compile` → (new) cost guard → `executor.Execute`
|
||||
pipeline. This is the mechanical enforcement of the phase's
|
||||
non-negotiable principle: there is no code path by which a
|
||||
`Provider` implementation could execute a query itself, because the
|
||||
interface doesn't give it the means to.
|
||||
- **`Confidence` is a three-value enum (`high`/`medium`/`low`), not a raw
|
||||
float.** A model's self-reported numeric confidence isn't a calibrated
|
||||
probability; treating it as one (thresholding at some specific float)
|
||||
would be false precision. Three bands are enough to drive real UI
|
||||
behavior — task 10 wants low confidence stated plainly, and a `Low`
|
||||
band with a required `LowConfidenceReason` is how that's enforced at
|
||||
the type level rather than left to prompt-following.
|
||||
- **`Complete` returns a suggestion, not a full requery.** Ghost-text
|
||||
needs exactly the continuation to render after the cursor; making the
|
||||
caller diff the model's output against its own input to find the new
|
||||
part would be fragile and unnecessary.
|
||||
- **`Explain` is reused for Track B's "explain the translation," not
|
||||
duplicated.** `ExplainRequest.OriginalIntent` is optional — empty for
|
||||
Track A's "explain this query I wrote" affordance, populated when
|
||||
Track B calls it right after `Translate` to describe *how the NL
|
||||
became this query* rather than only describing the query in
|
||||
isolation. One operation, two contexts, matching task 10's explicit
|
||||
instruction not to build a second explanation mechanism.
|
||||
- **`Fix` never has a silent-apply path.** `FixResult` is a suggestion +
|
||||
explanation; the diff rendering and accept/dismiss decision belong to
|
||||
the caller (the UI), matching task 7's explicit requirement.
|
||||
- **No streaming in this interface, deliberately.** A streamed
|
||||
token-by-token response would help perceived latency for `Complete`
|
||||
especially, but adds real complexity (partial-JSON handling,
|
||||
cancellation semantics) this design doesn't take on in v1. Flagged as
|
||||
a candidate follow-up if `Complete`'s latency budget (task 5) turns
|
||||
out not to be met by a small model's normal non-streamed response
|
||||
time — not preemptively built.
|
||||
|
||||
## Primary provider: Ollama, not vLLM
|
||||
|
||||
| | Ollama | vLLM |
|
||||
|---|---|---|
|
||||
| Hardware floor | Runs on CPU (slow) or a single consumer GPU via quantized (GGUF) models | Needs a real GPU; not practically CPU-viable |
|
||||
| Deployment complexity | Single binary/container, `ollama pull <model>`, built-in REST API | Python server, CUDA/driver management, more moving parts |
|
||||
| Throughput under concurrency | Adequate for one-user-at-a-time interactive use; not built for high concurrent QPS | Purpose-built for high-throughput serving (continuous batching, PagedAttention) |
|
||||
| Fit for this project | Matches `docker-compose for local/homelab` (CLAUDE.md's stated deployment target) — most self-hosters won't have a dedicated inference GPU | Fits a provisioned-GPU SaaS inference tier — not this phase's target (cloud is the opt-in secondary path, not primary) |
|
||||
|
||||
Sentry's actual AI workload shape is one interactive query bar per user
|
||||
at a time, not a high-QPS inference-serving problem — vLLM's real
|
||||
advantage (batched throughput at scale) isn't the bottleneck this phase
|
||||
has. Ollama's lower hardware floor and much simpler operational story
|
||||
directly serve the "self-hostable, homelab-friendly default" requirement
|
||||
task 2 sets. **Decision: Ollama**, reachable over its REST API
|
||||
(`http://localhost:11434` by default), matching every other
|
||||
external-service integration in this codebase (network boundary, not a
|
||||
linked library).
|
||||
|
||||
## Model recommendation
|
||||
|
||||
Selection criteria per the brief: license (this project just finished an
|
||||
entire phase auditing for OSI-approved-only licenses — recommending a
|
||||
model under a restricted custom license here would directly contradict
|
||||
that), hardware footprint (must have a genuinely homelab-viable size,
|
||||
not just a large flagship variant), and code/structured-output quality
|
||||
(pipe-syntax generation is closer to code/DSL generation than prose).
|
||||
|
||||
Ruled out, with reasons, rather than silently skipped:
|
||||
- **Llama 3.x** (Meta): strong quality, but Meta's Llama Community
|
||||
License is a custom license with a usage restriction (a >700M MAU
|
||||
clause) — not OSI-approved open source. Inconsistent with this
|
||||
project's own just-completed license posture.
|
||||
- **Gemma 2/CodeGemma** (Google): same shape of problem — a custom
|
||||
license with usage restrictions, not pure Apache/MIT.
|
||||
- **DeepSeek-Coder-V2**: strong at code, but its license also carries
|
||||
use restrictions beyond a standard permissive grant.
|
||||
- **StarCoder2** (BigCode): OpenRAIL-M is a "responsible AI license"
|
||||
with behavioral-use restrictions — again not a clean permissive grant.
|
||||
|
||||
**Recommended: the Qwen2.5-Coder family (Alibaba), Apache-2.0 licensed**
|
||||
— genuinely OSI-approved, no usage restrictions, strong benchmarked
|
||||
performance on code/SQL/structured-output generation specifically (not
|
||||
just general chat), and available in a real size range so the hardware
|
||||
floor is a deployment choice, not a fixed cost:
|
||||
|
||||
| Model | Approx. footprint (4-bit quantized via Ollama) | Suggested role |
|
||||
|---|---|---|
|
||||
| `qwen2.5-coder:1.5b` | ~1-2GB RAM/VRAM, CPU-viable | The fast/small end of task 2's per-operation config (see below) — candidate for `Complete`'s tight latency budget |
|
||||
| `qwen2.5-coder:7b` | ~5-6GB VRAM recommended, CPU possible but slow | **Default recommendation** for `Translate`/`Explain`/`Fix` — the balance point between quality and a realistic self-host minimum spec |
|
||||
| `qwen2.5-coder:14b` / `32b` | ~10-20GB+ VRAM | Optional upsell for deployments with more GPU headroom wanting better translation quality; not the default |
|
||||
|
||||
**Proposed default deployment**: `qwen2.5-coder:7b` for every operation
|
||||
unless per-operation config (below) is explicitly set otherwise. This is
|
||||
the one item the brief says needs your confirmation before finalizing,
|
||||
since it sets the minimum hardware bar every self-hosting operator reads
|
||||
as "what do I need to run this."
|
||||
|
||||
## Secondary provider: cloud adapter
|
||||
|
||||
A single, vendor-neutral adapter implementing `Provider` against an
|
||||
OpenAI-compatible chat-completions HTTP API (covers OpenAI itself and
|
||||
the several other providers — including some serving open-weight models
|
||||
— that expose the same wire contract), rather than a bespoke adapter per
|
||||
vendor. Concretely:
|
||||
|
||||
- **Off by default, opt-in per-tenant.** Enablement is a tenant-level
|
||||
setting (Phase 4's tenant/org model, `enterprise/`'s config surface —
|
||||
the same "core defines the interface, enterprise/ owns tenant-scoped
|
||||
policy over a network call" shape `api/authz.HTTPAuthorizer` already
|
||||
uses), not a global deployment flag. A single-tenant deployment with
|
||||
no `enterprise/` configured never has cloud access available at all,
|
||||
matching the "no cloud dependency required for the default deployment"
|
||||
exit criterion.
|
||||
- **Visible warning when enabled.** The settings UI surface that toggles
|
||||
this (extending Phase 5's Settings page) shows an explicit,
|
||||
un-dismissable-by-default notice that enabling this sends query
|
||||
content to a third-party API — not a one-time toast, a persistent
|
||||
visual indicator wherever cloud is active, so it isn't forgotten after
|
||||
the initial toggle.
|
||||
- **API key stored server-side only** (`enterprise/`'s existing
|
||||
credential-storage conventions — same posture as notification-target
|
||||
webhook secrets from Phase 3), never exposed to the browser.
|
||||
|
||||
This is architecture, not implementation — no code for this adapter is
|
||||
built in this task; it's described here so task 3/4 and the tracks can
|
||||
be designed against a stable shape.
|
||||
|
||||
## Per-operation provider/model configuration: building it in now
|
||||
|
||||
Decision: **yes, build the routing layer now**, not deferred. Reasoning:
|
||||
|
||||
`Complete`'s latency budget is a first-class requirement of task 5
|
||||
("low enough latency to feel responsive") — the brief itself flags this
|
||||
as the case where a fast small model matters most. If the interface only
|
||||
supported one model for every operation, satisfying `Complete`'s latency
|
||||
target would force *either* a small model for everything (hurting
|
||||
`Translate`/`Fix` quality) or a large model for everything (breaking
|
||||
autocomplete's responsiveness) — a real, immediate conflict, not a
|
||||
hypothetical future one.
|
||||
|
||||
What "building it in" actually means, scoped narrowly: a small
|
||||
config-driven routing table —
|
||||
`map[Operation]ProviderConfig{Provider, Model}` — resolved once at
|
||||
startup, defaulting every operation to the same provider/model unless a
|
||||
deployment explicitly overrides one. This is a routing/config concern
|
||||
sitting *above* the `Provider` interface (a thin dispatcher choosing
|
||||
which configured `Provider` instance to call per operation), not a
|
||||
change to the interface itself, and not a general multi-model
|
||||
orchestration system. A deployment that wants one model for everything
|
||||
sets one config value and never thinks about this again; a deployment
|
||||
that wants `qwen2.5-coder:1.5b` for `Complete` and `qwen2.5-coder:7b` for
|
||||
everything else sets two.
|
||||
|
||||
## Schema grounding (task 3)
|
||||
|
||||
Built: `api/ai/grounding` (core -- `Service` wraps one `executor.SQLRunner`,
|
||||
samples ClickHouse via that runner, caches one `provider.SchemaContext`
|
||||
snapshot, refreshed on an interval) and
|
||||
`enterprise/internal/groundingregistry` (multi-tenant wiring -- one
|
||||
cached snapshot per active tenant, all sampled through the same shared
|
||||
`chrunner.Registry`, which resolves the actual per-tenant ClickHouse
|
||||
connection from a context stamped via `authz.WithIdentity` -- the same
|
||||
mechanism `chrunner`'s own doc comment names this exact kind of non-HTTP
|
||||
caller as being for). Sourced by periodic sampling against `logs`
|
||||
(service names by frequency, `mapKeys(attributes)` for common attribute
|
||||
keys, `DISTINCT`-with-a-cap per candidate field for enum-like examples)
|
||||
-- nothing hand-maintained. Tenant scoping is structural: a grounding
|
||||
query is just another `RunSQL` call through the exact same tenant-scoped
|
||||
connection Phase 4 already isolates query execution behind, not a
|
||||
separate mechanism that could drift out of sync with it.
|
||||
|
||||
**Delivery mechanism: embedded in-prompt, not retrieved per-query.**
|
||||
|
||||
| | Embedded in-prompt | Retrieved per-query (RAG-style) |
|
||||
|---|---|---|
|
||||
| Latency | One model call, no extra round trip | An added retrieval/ranking step before every call |
|
||||
| Complexity | Grounding data is just serialized into the request | Needs a relevance-ranking step matching partial/NL input against a larger corpus |
|
||||
| Fit for this project's scale | A tenant's own service/field vocabulary is small (tens of services, capped at 100 attribute keys) -- full embedding doesn't meaningfully bloat the prompt | Solves a problem (a corpus too large to embed) this project doesn't have yet |
|
||||
|
||||
Decision: embed the full (capped) `SchemaContext` in every operation's
|
||||
prompt. The latency cost of an extra retrieval step would hit `Complete`
|
||||
hardest -- exactly the operation with the tightest budget (task 5) -- for
|
||||
a problem (prompt bloat from an oversized schema) this project's actual
|
||||
scale doesn't have. `grounding.go`'s caps (50 services, 100 attribute
|
||||
keys, 15 of those get real example-value queries, 20 examples max per
|
||||
field) exist specifically so this stays true even for a tenant with an
|
||||
unusually sprawling schema. If real usage ever shows a tenant blowing
|
||||
past these caps in a way that matters, per-query retrieval is the
|
||||
natural fallback design -- not built now, since nothing today needs it.
|
||||
|
||||
## Cost/safety guard (task 4)
|
||||
|
||||
**Flagging this as larger than expected, per the brief's own invitation
|
||||
to do so**: no cost-estimation mechanism existed anywhere in Phase 2/3
|
||||
before this task. Confirmed by reading the compiler, not assumed --
|
||||
`ir.Plan.TimeRange` can be entirely unset (both bounds zero), and
|
||||
nothing between the planner and ClickHouse rejects that; a bare `stats
|
||||
count by host` with no `earliest=` scans the table's full history today.
|
||||
Building this from scratch, plus deciding how it applies to *existing*
|
||||
hand-written queries (not just new AI ones), was real, unplanned design
|
||||
work beyond "check a number against a threshold."
|
||||
|
||||
Built: `api/ai/costguard`, a pure function (`Assess(*ir.Plan) Assessment`)
|
||||
with three levels (`ok`/`warn`/`reject`) and human-readable reasons:
|
||||
|
||||
- **No time bound + aggregation → reject.** An aggregation gets no
|
||||
implicit row cap the way a raw-row fetch does -- confirmed by reading
|
||||
`executor/sql.go`'s `buildSQL` directly: its `defaultRowLimit=100`
|
||||
safety net only applies `else if plan.Aggregation == nil`. An
|
||||
unbounded aggregation must scan every matching row regardless of
|
||||
output size, with nothing downstream capping that scan.
|
||||
- **No time bound, no aggregation (raw-row fetch), explicit `Limit` or
|
||||
not → warn, not reject, either way.** Real bug caught by this
|
||||
package's own tests, not shipped as originally written: an earlier
|
||||
version of this rule treated "no explicit `Limit`" as automatically
|
||||
worse (reject) than "an explicit `Limit`" (warn) -- wrong, because
|
||||
`buildSQL` applies its own `defaultRowLimit=100` to *any* non-aggregation
|
||||
query with no explicit `Limit`, so both cases already have the exact
|
||||
same real row cap and therefore the same risk level. This is also the
|
||||
common "just show me recent logs" pattern the query language's own
|
||||
documented default (`head 100`) already treats as normal -- rejecting
|
||||
it outright would have flagged a large fraction of legitimate,
|
||||
currently-working queries, not just a genuinely dangerous new class.
|
||||
- **Time range spans over 90 days → warn.**
|
||||
- **Raw SQL → a best-effort regex check** for a `timestamp` comparison
|
||||
anywhere in the text, not a real parse. Explicitly documented as
|
||||
lower-confidence than the IR-based checks, which get a structural
|
||||
guarantee raw SQL fundamentally can't (same reason Phase 2's raw-SQL
|
||||
escape hatch was always opaque to compiler-level enforcement).
|
||||
- **90-day span and every other numeric threshold here are first-pass
|
||||
heuristics**, not benchmarked against a production-scale cluster --
|
||||
this environment's own ClickHouse instance holds nowhere near enough
|
||||
data to validate them against. Flagged plainly rather than presented
|
||||
as tuned.
|
||||
|
||||
**How the two callers apply it differently, both wired now:**
|
||||
|
||||
- **AI-suggested queries** (Translate/Fix output, and the Optimize
|
||||
suggestion, tracks A/B, not yet built): a `reject`-level assessment
|
||||
means the suggestion is not offered as a normal accept-and-run action.
|
||||
This is the mechanism task 4 asked for -- "reject or flag ... before
|
||||
it's ever offered to the user."
|
||||
- **The existing `/query` handler**: now runs every query (hand-written
|
||||
or not) through the same `costguard.Assess` and surfaces the result as
|
||||
a new, additive `warnings` field on the response (`queryapi/handler.go`)
|
||||
-- never a block. This is a deliberate interpretation of the phase's
|
||||
design principle ("the *same* ... cost guardrails as a hand-written
|
||||
query"), decided here rather than left ambiguous: hand-written queries
|
||||
get the identical assessment an AI-generated one would, so there's
|
||||
real parity, but retroactively hard-blocking existing dashboard/
|
||||
`sentryctl` query patterns that happen to have no time bound is a
|
||||
behavioral change this phase didn't set out to make and could break
|
||||
real existing usage. `warnings` is `omitempty` -- a client that
|
||||
doesn't look for it sees no shape change at all. All existing
|
||||
`queryapi` tests still pass unmodified.
|
||||
|
||||
## Decisions confirmed 2026-08-16
|
||||
|
||||
All four items below were confirmed as proposed, no changes:
|
||||
|
||||
1. `qwen2.5-coder:7b` as the default model (Translate/Explain/Fix),
|
||||
`qwen2.5-coder:1.5b` as the fast-path option for `Complete`.
|
||||
2. Ollama over vLLM as the primary inference runtime.
|
||||
3. The cloud-adapter shape: single OpenAI-compatible adapter, per-tenant
|
||||
opt-in, off by default.
|
||||
4. Per-operation provider/model config built in now, not deferred.
|
||||
|
||||
Proceeding to task 3 (schema grounding) and task 4 (cost/safety guard).
|
||||
|
||||
## Ollama provider implementation (shared foundation, completion)
|
||||
|
||||
The last piece of shared foundation before either track: task 2 designed
|
||||
`provider.Provider`'s shape, but nothing implemented it until now.
|
||||
Built, tested, all green:
|
||||
|
||||
- `api/ai/provider/ollama` -- a thin `net/http`+`encoding/json` client
|
||||
against Ollama's `POST /api/chat`, same shape as
|
||||
`alerting/internal/queryclient` (this codebase's existing precedent
|
||||
for a small internal HTTP client, no new dependency). Uses Ollama's
|
||||
`format: "json"` constrained-output mode for the three operations that
|
||||
return structured data (`Translate`/`Complete`/`Fix`); `Explain` asks
|
||||
for prose directly since its result is a single string with nothing
|
||||
else to parse.
|
||||
- `prompts.go` -- each operation's system prompt embeds a condensed copy
|
||||
of `/docs/query-language-reference.md`'s grammar (kept in sync by
|
||||
hand, same as every other place the language is described outside its
|
||||
own parser) plus the caller's `SchemaContext`, rendered inline per the
|
||||
embedded-in-prompt decision above.
|
||||
- Handles the common small-model habit of wrapping JSON in a markdown
|
||||
code fence despite being told not to (`stripCodeFence`) -- a
|
||||
best-effort cleanup, not a guarantee; genuinely malformed output still
|
||||
surfaces as a real parse error to the caller rather than being
|
||||
silently papered over.
|
||||
- `Confidence` parsing fails toward `Low` on anything unrecognized, never
|
||||
toward assumed correctness -- an empty or garbled confidence field
|
||||
from the model is itself a signal something's off.
|
||||
- `api/ai/router` -- the per-operation dispatch layer task 2 decided to
|
||||
build now: a lookup from `Operation` to whichever `provider.Provider`
|
||||
was configured for it, falling back to one default. A deployment that
|
||||
wants one model for everything configures one; one that wants
|
||||
`qwen2.5-coder:1.5b` for `Complete` and `:7b` for everything else
|
||||
configures two -- exactly the scope described in task 2's writeup,
|
||||
nothing more.
|
||||
|
||||
Tested against a real `httptest.Server` standing in for Ollama's actual
|
||||
wire contract (request shape, JSON-mode response parsing, the
|
||||
code-fence-stripping fallback, non-200 error surfacing) -- not just
|
||||
type-checked. **Not yet tested against a real running Ollama server or a
|
||||
real `qwen2.5-coder` model** in this environment; that's real, disclosed
|
||||
verification work for `/docs/phase-7-runbook.md` once there's a live
|
||||
stack to test against, same "written but not run against the live thing"
|
||||
caveat this project applies throughout. Nothing in `main.go` wires any
|
||||
of this up yet -- that's deferred to when the actual AI HTTP endpoints
|
||||
(Track A/B) exist to need it, so there's no dead, unconsumed
|
||||
configuration sitting in a running binary in the meantime.
|
||||
|
||||
All shared foundation (tasks 1-4, plus this provider implementation) is
|
||||
now built and verified: `go build`/`go vet`/`go test` clean across `api`
|
||||
and `enterprise`, `hack/check-tenant-boundary.sh` still passes. Per the
|
||||
CHECKPOINT scope discussion, Track A is next.
|
||||
|
||||
## Track A (tasks 5-8): built and live-verified
|
||||
|
||||
Backend: `api/ai/aiapi` (`POST /ai/complete`/`explain`/`fix`/`optimize`),
|
||||
wired into both `api/cmd/api` and `enterprise/cmd/enterprise-api`, gated
|
||||
on `OLLAMA_BASE_URL` (empty by default -- routes aren't even registered
|
||||
when unset, matching "no cloud dependency required for the default
|
||||
deployment" and, by the same reasoning, no forced *local* model
|
||||
dependency either). `Fix`'s suggested query and `Optimize`'s mechanical
|
||||
rewrite both run through `costguard.Assess` before being returned --
|
||||
a `reject`-level assessment sets `blocked: true`, and the frontend
|
||||
disables the accept action rather than silently offering it.
|
||||
|
||||
Frontend: ghost-text completion built directly into
|
||||
`QueryEditor.svelte` on CodeMirror's own primitives (`StateField` +
|
||||
`Decoration.widget`, no new dependency), debounced (300ms) and only
|
||||
triggered when the cursor is at the document end. Explain/Fix/Optimize
|
||||
added to `QueryBar.svelte` (shared by every consumer -- Search page,
|
||||
dashboard panel editor, alert rule editor -- though only the Search page
|
||||
was wired with the full `errorMessage`/`warnings` props in this pass;
|
||||
the other two get the feature for free whenever they're updated to pass
|
||||
them, a non-breaking follow-up, not done here).
|
||||
|
||||
**Verified live in a real browser**, not just type-checked -- a mock
|
||||
Ollama server (matching its real `/api/chat` wire contract) run as a
|
||||
container on the compose network, `api` rebuilt with `OLLAMA_BASE_URL`
|
||||
pointed at it. All four operations confirmed working end-to-end through
|
||||
actual UI interaction: Explain's modal, Fix's real diff view with a
|
||||
genuine parse error and a working Accept that replaced the query bar's
|
||||
content, Optimize's real cost-guard finding (`severity=ERROR | stats
|
||||
count by host` against the live seeded ClickHouse data, showing the
|
||||
actual inline warning and populating the Optimize modal with a real
|
||||
mechanical rewrite), and ghost-text completion rendering and
|
||||
accepting correctly on Tab. Graceful degradation confirmed too: with the
|
||||
mock server stopped, ghost text silently doesn't appear (no error), and
|
||||
Explain shows a plain "not available right now" message instead of
|
||||
crashing.
|
||||
|
||||
**Two real bugs found and fixed by this live-verification pass** --
|
||||
neither would have been caught by `svelte-check`/`npm run build`, both
|
||||
type-correct code:
|
||||
|
||||
1. **The CodeMirror view was being destroyed and recreated on every
|
||||
keystroke.** `QueryEditor.svelte`'s view-creation `$effect` read
|
||||
`value` (needed for the initial `doc:` content), which made it a
|
||||
reactive dependent of `value` -- but the editor's own
|
||||
`updateListener` writes `value` on every keystroke to keep the
|
||||
bindable prop in sync. Every keystroke therefore re-ran the whole
|
||||
effect, tearing down and rebuilding the entire `EditorView`. This
|
||||
predates Phase 7 (the pattern existed since Phase 5) but never
|
||||
manifested as a visible symptom until ghost-text's debounce timer
|
||||
gave it something to silently cancel: the effect's cleanup function
|
||||
(`clearTimeout(completeTimer); view?.destroy()`) fired moments after
|
||||
`scheduleCompletion` set the timer, cancelling it before its 300ms
|
||||
elapsed -- `Complete` looked like it was doing nothing, every time.
|
||||
Fixed by wrapping the initial `value` read in Svelte 5's `untrack()`,
|
||||
so the effect now genuinely only depends on `container` (runs once,
|
||||
on mount) -- matching what the component's own second "external
|
||||
sync" effect was already documented as assuming.
|
||||
2. **Ghost text rendered at the start of the query, not after it.** The
|
||||
`Decoration.widget` was hardcoded at document position `0` (with a
|
||||
comment reasoning that position didn't matter since ghost text is
|
||||
only ever shown at the document end) -- true for the field's own
|
||||
`null`-vs-suggestion state, but the position still needs to be the
|
||||
*current* end of document, not literally position 0. Confirmed
|
||||
visually (a screenshot showing the suggestion prepended before typed
|
||||
text, not appended after it). Fixed by storing a positioned
|
||||
`DecorationSet` directly in the state field, computed inside
|
||||
`update()` using `tr.state.doc.length` -- the one place the field's
|
||||
`update` function has access to the actual current document.
|
||||
|
||||
Both fixes are in the same commit-sized unit as the rest of Track A --
|
||||
no separate patch, since neither bug shipped anywhere before this pass
|
||||
caught it.
|
||||
|
||||
## Track B (tasks 9-11): built and live-verified
|
||||
|
||||
Backend: `POST /ai/translate` in `api/ai/aiapi`, same file and same
|
||||
patterns as Track A's endpoints. Every translation is compiled
|
||||
(`planner.Compile`, always `planner.SPL` -- pipe syntax only, per task
|
||||
9's "narrower, safer surface" choice) and run through `costguard.Assess`
|
||||
before the response goes out, exactly like `Fix`'s suggested query --
|
||||
task 9's explicit requirement, not a lesser treatment for a different
|
||||
track. Three honestly-distinct failure shapes, not collapsed into one:
|
||||
low confidence (`query` empty, a reason given), a confident answer that
|
||||
doesn't compile (`compiles: false`, a real and different outcome from
|
||||
low confidence -- a model can be sure of itself and still wrong about
|
||||
syntax), and a confident, compiling answer the cost guard blocks.
|
||||
|
||||
**Detection mechanism (task 10), decided and documented, not just
|
||||
implemented**: waiting for a parse error, as the task's own phrasing
|
||||
suggests, would miss the main case this feature exists for. The pipe
|
||||
grammar's free-text rule means a plain-English question like "show me
|
||||
errors from the last day" *parses successfully* -- it becomes a
|
||||
free-text AND-search for those literal words, not a syntax error, and
|
||||
silently returns an unhelpful result instead of failing loudly. Built
|
||||
instead: a cheap client-side heuristic (`looksLikeNaturalLanguage` in
|
||||
`QueryBar.svelte`) that flags text with none of the pipe syntax's
|
||||
structural markers (`|`, a comparison operator, `:`) and four or more
|
||||
words -- long enough that it's very unlikely to be an intentional short
|
||||
free-text search, which stays untouched. Confirmed live: a real query
|
||||
like `show me errors from the last day grouped by service` correctly
|
||||
surfaced the "Interpret as natural language" affordance.
|
||||
|
||||
Frontend: a review modal (`QueryBar.svelte`) pre-filled with the current
|
||||
query bar text (since that's exactly what triggered the detection),
|
||||
auto-translates on open, shows the generated query in an **editable**
|
||||
textarea (task 10's explicit "editable inline" requirement) alongside
|
||||
an auto-fetched explanation -- reusing `Explain` via
|
||||
`ExplainRequest.OriginalIntent` rather than a separate mechanism, task
|
||||
10's explicit instruction, already built for Track A's own translation-
|
||||
review use in the provider interface. "Use this query" inserts into the
|
||||
query bar and closes the modal; it never runs anything -- the existing
|
||||
"Run query" button, calling the unchanged `runQuery`/`POST /query`, is
|
||||
the only confirm-to-run action anywhere in this flow, task 9's
|
||||
non-negotiable separation. A blocked suggestion the user has since
|
||||
edited in the textarea is treated as their own text, not the original
|
||||
flagged one -- re-blocking an edit made specifically to address the
|
||||
concern would be unhelpful, and `/query`'s own `warnings` field still
|
||||
assesses whatever they actually end up running regardless.
|
||||
|
||||
**Verified live end-to-end**: typed a natural-language-shaped query into
|
||||
the bar, clicked the affordance, got a real generated query
|
||||
(`earliest=-1h severity=ERROR | stats count by service | sort -count`)
|
||||
and a real auto-fetched explanation back from the mock provider, clicked
|
||||
"Use this query" (confirmed it replaced the query bar content **without
|
||||
running anything** -- no results table appeared), then manually clicked
|
||||
"Run query" and confirmed it executed cleanly through the unchanged
|
||||
`/query` endpoint. Low-confidence and non-compiling-suggestion rendering
|
||||
were verified via the Go-level handler tests
|
||||
(`TestHandleTranslateLowConfidenceCarriesReason`,
|
||||
`TestHandleTranslateNonCompilingQueryIsHonestlyReported`) and code
|
||||
review rather than a separate live click -- the frontend branch that
|
||||
renders them is structurally the same conditional-message pattern
|
||||
already live-verified repeatedly for Explain/Fix/Optimize's own
|
||||
"unavailable" states, not new untested UI shape.
|
||||
|
||||
CLI (task 11): `sentryctl query --nl "..."` in `cli/cmd/sentryctl/cmd_query.go`.
|
||||
Same posture as the UI, enforced identically regardless of how the
|
||||
result was produced: a low-confidence, non-compiling, or cost-guard-blocked
|
||||
translation is never run, even with `--execute` -- confirmed by
|
||||
`TestCmdQueryNLBlockedIsNotRunEvenWithExecute` and
|
||||
`TestCmdQueryNLNonCompilingDoesNotRun`, which fail the test itself if
|
||||
`/query` is ever called in those cases. Without `--execute`, a real
|
||||
terminal gets a `y/N` confirmation prompt; a non-interactive invocation
|
||||
(piped, scripted, CI) prints the translation and exits without running
|
||||
rather than hanging on a prompt nobody can answer -- detected via
|
||||
`os.Stdin`'s `ModeCharDevice` bit, no new dependency. `runAndPrintQuery`
|
||||
is shared between the plain-query path and the post-translation
|
||||
execute path, so both go through byte-for-byte the same request code
|
||||
this command already had -- not a parallel implementation.
|
||||
|
||||
All three shared-foundation guarantees hold identically for both
|
||||
tracks, confirmed by inspection of the actual code paths, not asserted:
|
||||
every generated or suggested query flows through `planner.Compile` (the
|
||||
unchanged Phase 2 compiler) and `costguard.Assess` before a human ever
|
||||
sees an offer to run it, and actual execution -- web, CLI, or a
|
||||
hand-written query -- is always the same `POST /query` handler with the
|
||||
same tenant-scoped `SQLRunner` and the same audit-logging hook Phase 4
|
||||
established. No AI code path constructs a `SQLRunner`, calls
|
||||
`executor.Execute`, or bypasses `authz.RequireRoleOrService` anywhere in
|
||||
either track.
|
||||
|
||||
## Audit logging for AI interactions (task 12): built
|
||||
|
||||
Reuses Phase 4's existing `audit_log` table rather than adding a new
|
||||
one: that table's `detail JSONB` column and extensible `event_type`
|
||||
CHECK constraint were already designed to carry event shapes other than
|
||||
"query" (`role_change`/`grant_change`/etc. already skip
|
||||
`query_text`/`row_count`/`duration_ms` in favor of `detail`) --
|
||||
`ai_interaction` (`metadata/migrations/0036_add_ai_interaction_event_type.sql`)
|
||||
is the same shape of extension, not a new table/role/trigger set. Same
|
||||
append-only, hash-chained, `audit_writer`-role-restricted protections
|
||||
apply for free.
|
||||
|
||||
Scoped to only `translate`/`fix`/`optimize` -- the three flows that
|
||||
produce a suggestion a user explicitly accepts or dismisses. Deliberately
|
||||
excludes `complete` (ghost-text fires on every keystroke pause; logging
|
||||
each one at the same weight as a deliberate review would drown the
|
||||
signal) and `explain` (produces no suggestion to accept/reject, so the
|
||||
concept doesn't apply). Both exclusions are named design boundaries, not
|
||||
oversights.
|
||||
|
||||
Chose a single frontend-reported event at the moment of a terminal user
|
||||
action (accept-and-use or dismiss/cancel) over a two-phase
|
||||
generation-plus-outcome design correlated by an ID -- simpler, and avoids
|
||||
threading interaction IDs through every generation response just to
|
||||
correlate them later.
|
||||
|
||||
`api/ai/aiapi.InteractionLogger` is a small interface
|
||||
(`LogInteraction(ctx, InteractionEntry) error`), nil-by-default on
|
||||
`Handler` -- same shape as `queryapi.AuditLogger`: a single-tenant
|
||||
deployment with no `enterprise/` configured simply doesn't log these,
|
||||
same as it doesn't log query executions today. `POST /ai/log-interaction`
|
||||
is fail-open, same posture as `queryapi.Handler`'s own audit logging: a
|
||||
write failure is logged server-side and never surfaced to the user who
|
||||
just clicked a button.
|
||||
|
||||
`enterprise/internal/audit.AIInteractionLogger`
|
||||
(`ai_interaction_adapter.go`) is the real implementation, mirroring
|
||||
`QueryAPILogger`'s exact shape: resolves tenant/user identity from ctx
|
||||
via `authz.IdentityFromContext`, refuses to write an unattributable
|
||||
entry, and writes through the same `*Store`/pool
|
||||
`enterprise-api/main.go` already opens for query auditing (one dedicated
|
||||
`audit_writer`-role pool, reused for both loggers). Operation, input,
|
||||
output, confidence, and the accepted/edited flags go into `Detail` as
|
||||
JSON; `FinalQuery` -- the suggested query, whether or not it was
|
||||
actually used -- goes into the table's existing `QueryText` column,
|
||||
since that's the one field a security reviewer scanning the audit log
|
||||
would expect to search on directly.
|
||||
|
||||
Frontend: `web/src/lib/api.ts`'s `logInteraction` is fire-and-forget
|
||||
(`.catch(() => {})` at each call site) -- an audit-write failure, like
|
||||
every other AI-operation failure in this phase, must never block or
|
||||
surface an error on the UI action that triggered it. Wired into
|
||||
`QueryBar.svelte`'s `acceptFix`/`dismissFix`, `acceptOptimize`/
|
||||
`dismissOptimize`, and `useTranslatedQuery`/`cancelTranslate`. Translate
|
||||
is the one flow with an edit affordance (the generated-query textarea),
|
||||
so it's the only one where `edited` can be `true` -- computed by
|
||||
comparing the textarea's current content against the original suggested
|
||||
query at the moment of acceptance, not tracked keystroke-by-keystroke.
|
||||
|
||||
**Genuinely verified against a live Postgres**, not just unit-tested
|
||||
against a fake `InteractionLogger`: `metadata/migrations/0036` was
|
||||
applied to the running dev stack's `sentry-metadata-postgres`
|
||||
(`docker compose up -d --build metadata-migrate`, confirmed via `\d+
|
||||
audit_log` before/after showing `ai_interaction` added to the
|
||||
`event_type` CHECK constraint), and two new tests in
|
||||
`enterprise/internal/audit/integration_test.go` --
|
||||
`TestAIInteractionLoggerWritesAttributedToContextIdentity` and
|
||||
`TestAIInteractionLoggerRefusesWithoutIdentity`, the same pattern
|
||||
`TestQueryAPILoggerWritesAttributedToContextIdentity` already
|
||||
established -- ran against that real database, through the real
|
||||
`audit_writer`-role pool, and passed: a real row lands with
|
||||
`event_type='ai_interaction'`, `query_text` carrying `FinalQuery`, and
|
||||
`detail` carrying a JSON blob whose `operation`/`accepted` fields
|
||||
round-trip correctly. This closes the one live-infrastructure gap task
|
||||
12's backend work would otherwise have shared with Phase 4's own
|
||||
disclosed "compiles and is unit-tested, never run against a real
|
||||
database" caveat.
|
||||
|
||||
## Integration tests and CI testability (task 13)
|
||||
|
||||
Before this task, coverage had a real seam nothing exercised: unit tests
|
||||
work at two separate layers that never actually touch each other in a
|
||||
test. `aiapi/handler_test.go`'s `fakeProvider` satisfies
|
||||
`provider.Provider` directly, bypassing HTTP, JSON, and prompt
|
||||
construction entirely; `ollama/ollama_test.go` exercises
|
||||
`ollama.Client`'s wire-format parsing against a stub server, but never
|
||||
through `aiapi.Handler`'s actual registered routes. Neither proves the
|
||||
seam between them -- a real `*ollama.Client`, wired through a real
|
||||
`*router.Router` into a real `*Handler`, reached over real HTTP -- was
|
||||
ever exercised end to end.
|
||||
|
||||
`api/ai/aiapi/integration_test.go` (new) closes that gap: `mockOllamaServer`
|
||||
stands in for Ollama's real `/api/chat` endpoint (matching its wire
|
||||
contract byte-for-byte, same technique this phase's live browser
|
||||
verification used, just returning one fixed canned JSON body per test
|
||||
instead of one selected by inspecting the prompt), wired into a real
|
||||
`ollama.New(...)` client, a real `router.New(...)`, and a real
|
||||
`NewHandler(...)`, then driven by real HTTP requests via `httptest.Server`
|
||||
against `/ai/translate`, `/ai/fix`, `/ai/complete`, and
|
||||
`/ai/log-interaction`. `TestIntegrationTranslateBlockedByCostGuard`
|
||||
specifically proves `costguard.Assess` is actually reached through the
|
||||
full HTTP stack for an AI-suggested query (an unbounded aggregation
|
||||
comes back `blocked: true`), not just correct in `costguard_test.go`'s
|
||||
own isolated unit tests.
|
||||
|
||||
**Why this suite is CI-safe and a real model is not**: no live Ollama
|
||||
server or model weights are needed anywhere in this repo's test suite --
|
||||
every AI-related test (unit and integration) is deterministic, runs in
|
||||
milliseconds, and needs no network egress beyond `localhost`. Testing
|
||||
against a *real* Ollama server running the actual pinned
|
||||
`qwen2.5-coder:7b` model is deliberately kept **out** of this suite and
|
||||
out of CI entirely: a multi-gigabyte model download on every run, no
|
||||
determinism guarantee even at temperature 0 across Ollama/driver
|
||||
versions, and minutes of inference time per test would make the whole
|
||||
suite both slow and flaky in a way that erodes trust in CI failures
|
||||
generally -- the same "boring, well-understood, and fast" bar this
|
||||
project already holds its dependencies to. The test pyramid this phase
|
||||
ends up with:
|
||||
|
||||
1. **Unit tests** (existing, unchanged by this task): `costguard`,
|
||||
`grounding`, `router`, `ollama`'s wire-format parsing, `aiapi`'s
|
||||
handler/routing logic via `fakeProvider` -- fast, deterministic, no
|
||||
network, all run in CI today.
|
||||
2. **Integration tests** (this task, new): the mock-Ollama-server suite
|
||||
above, plus `cli/cmd/sentryctl/cmd_query_test.go`'s existing
|
||||
`httptest.Server`-backed coverage of `--nl`/`--execute` (already
|
||||
written during Track B, task 11) -- proves the plumbing (HTTP routing,
|
||||
JSON contracts, `planner.Compile`/`costguard.Assess` integration,
|
||||
audit-log dispatch) without needing real model inference. This is
|
||||
what actually runs in CI.
|
||||
3. **Model-quality verification** (not CI, not automated, disclosed as a
|
||||
deliberate gap rather than silently skipped): whether the actual
|
||||
pinned model reliably produces valid pipe syntax for realistic
|
||||
questions, whether its confidence self-reporting is well-calibrated,
|
||||
whether Explain's prose is actually useful -- these are inherently
|
||||
non-deterministic, model-quality questions that a wire-contract mock
|
||||
cannot answer and that would make CI flaky if it tried. This project's
|
||||
established "actually run it" discipline already produced exactly
|
||||
this kind of check once (this phase's live browser verification
|
||||
against `mock_ollama.py`, and per `/docs/phase-7-runbook.md` once
|
||||
written, against a real local Ollama); the recommendation is to keep
|
||||
that as a periodic, human-run pre-release checklist item, not a CI
|
||||
gate -- the same posture this repo already takes toward the
|
||||
ClickHouse/Postgres-backed pieces of Phase 4 that only "compile and
|
||||
skip cleanly" in an environment without live infrastructure, rather
|
||||
than pretending a flaky or infeasible-to-automate check is covered
|
||||
when it isn't.
|
||||
|
||||
No new frontend test framework (vitest, Playwright, etc.) was introduced
|
||||
for this task -- `web/`'s AI-feature verification stays the same live,
|
||||
manual browser verification already used for Track A/B (see those
|
||||
sections above), consistent with this project's established frontend
|
||||
verification discipline rather than adding a new tooling dependency
|
||||
whose payoff (catching regressions in ghost-text positioning, modal
|
||||
flows) is already covered by that discipline today.
|
||||
@@ -0,0 +1,224 @@
|
||||
# Phase 7 runbook
|
||||
|
||||
Extends `/docs/phase-0-runbook.md` through `/docs/phase-5-runbook.md`
|
||||
(Phase 6 had no runbook of its own — a compliance audit, not a running
|
||||
system). Read those first. Phase 7 adds one new component category (an
|
||||
AI model provider) and touches `api`, `enterprise/`, `web`, and `cli` —
|
||||
see `/docs/phase-7-ai-design.md` for the full design record; this
|
||||
document is verification only.
|
||||
|
||||
## What's actually been verified
|
||||
|
||||
Every AI operation (`complete`, `explain`, `fix`, `optimize`,
|
||||
`translate`, and the audit-logging endpoint behind it) has been run
|
||||
end-to-end against a real `docker compose` stack — real HTTP requests
|
||||
into the real `sentry-api` container, through the real
|
||||
`api/ai/provider/ollama.Client`, over a real network call, into a real
|
||||
process answering Ollama's actual `/api/chat` wire contract. **No real
|
||||
model weights are used anywhere in this verification** — see
|
||||
"Why a mock provider, not a real model" below for why that's a
|
||||
deliberate, disclosed choice rather than a shortcut. Two real product
|
||||
bugs were found and fixed via live browser verification of the frontend
|
||||
half (`QueryEditor.svelte`'s ghost-text autocomplete) that neither
|
||||
`svelte-check` nor `npm run build` caught — see
|
||||
`/docs/phase-7-ai-design.md`'s Track A section for the full writeup;
|
||||
this runbook doesn't repeat it.
|
||||
|
||||
Also verified in this pass, against the same live stack's real
|
||||
Postgres: the Phase 4 `audit_log` table's `event_type` CHECK constraint
|
||||
was extended with migration 0036, and both new
|
||||
`enterprise/internal/audit` tests (`TestAIInteractionLoggerWritesAttributed
|
||||
ToContextIdentity`, `TestAIInteractionLoggerRefusesWithoutIdentity`)
|
||||
passed against it — a real row lands with `event_type='ai_interaction'`,
|
||||
correctly attributed to the tenant/user identity in context, with
|
||||
`detail` carrying the operation/confidence/accepted/edited fields as
|
||||
JSON.
|
||||
|
||||
**Not verified, disclosed rather than silently skipped**: this
|
||||
environment has no GPU and no downloaded model weights, so the actual
|
||||
quality of `qwen2.5-coder:7b`'s output — whether it reliably produces
|
||||
valid pipe syntax for realistic questions, how well-calibrated its
|
||||
self-reported confidence is, whether Explain's prose actually reads as
|
||||
useful — has never been checked here. See
|
||||
`/docs/phase-7-ai-design.md`'s "Integration tests and CI testability"
|
||||
section for why that's kept as a periodic human-run checklist item
|
||||
rather than something this runbook or CI can cover.
|
||||
|
||||
## 1. Bring up the stack
|
||||
|
||||
```sh
|
||||
docker compose up -d --build
|
||||
cd web && npm run dev # localhost:5183, talks to localhost:8080/8081 by default
|
||||
```
|
||||
|
||||
No new required services — `docker compose ps` shows the same set as
|
||||
Phase 5. AI routes are off by default: with no `OLLAMA_BASE_URL` set,
|
||||
`api`/`enterprise-api` never register `/ai/*` at all (confirmed live in
|
||||
this pass — `curl -X POST localhost:8080/ai/translate` returns a plain
|
||||
`404`, not a 500 or a hang against an unreachable `localhost:11434`).
|
||||
|
||||
## 2. Enable AI routes against the committed mock provider
|
||||
|
||||
`hack/mock-ollama` (new this phase) answers Ollama's real `/api/chat`
|
||||
wire contract with fixed, deterministic canned responses picked by
|
||||
inspecting the system prompt's opening line — enough to exercise every
|
||||
real code path (`ollama.Client`'s HTTP call, JSON parsing,
|
||||
`planner.Compile`, `costguard.Assess`, the HTTP response shape) without
|
||||
needing model weights, a GPU, or non-deterministic output. This is the
|
||||
same technique `api/ai/aiapi/integration_test.go` uses in Go directly;
|
||||
this tool is for manual/browser verification, where an in-process fake
|
||||
isn't an option.
|
||||
|
||||
Run it as a container on the compose network with a network alias of
|
||||
`ollama` (so `api`'s container can resolve the hostname), then point
|
||||
`OLLAMA_BASE_URL` at it via a throwaway compose override:
|
||||
|
||||
```sh
|
||||
docker run -d --rm --name sentry-mock-ollama --network sentry_default --network-alias ollama \
|
||||
-v "$(pwd)/hack/mock-ollama:/src" -w /src golang:1.25-alpine \
|
||||
sh -c "go build -o /tmp/mock-ollama . && /tmp/mock-ollama"
|
||||
|
||||
cat > /tmp/docker-compose.ai-verify.yml <<'EOF'
|
||||
services:
|
||||
api:
|
||||
environment:
|
||||
OLLAMA_BASE_URL: "http://ollama:11434"
|
||||
OLLAMA_MODEL: "test-model"
|
||||
EOF
|
||||
|
||||
docker compose -f docker-compose.yml -f /tmp/docker-compose.ai-verify.yml up -d api
|
||||
```
|
||||
|
||||
For `enterprise-api` instead of core `api` (needed to also exercise
|
||||
task 12's real audit-log write, since core has no `InteractionLogger`
|
||||
wired in), override that service's environment instead, same shape.
|
||||
|
||||
Verify the routes are live:
|
||||
|
||||
```sh
|
||||
curl -s -X POST localhost:8080/ai/translate -H 'Content-Type: application/json' \
|
||||
-d '{"nlQuery":"errors in the last hour"}'
|
||||
# {"query":"earliest=-1h severity=ERROR","confidence":"high","compiles":true,"blocked":false}
|
||||
```
|
||||
|
||||
**Clean up afterward** — don't leave the mock provider or the override
|
||||
wired into a stack anyone else might reach:
|
||||
|
||||
```sh
|
||||
docker compose up -d api # drops back to the plain env, no -f override
|
||||
docker rm -f sentry-mock-ollama
|
||||
rm /tmp/docker-compose.ai-verify.yml
|
||||
curl -s -o /dev/null -w '%{http_code}\n' -X POST localhost:8080/ai/translate -d '{}'
|
||||
# 404 -- confirms AI routes are unregistered again
|
||||
```
|
||||
|
||||
## 3. Track A — Explain / Fix / Optimize / ghost-text
|
||||
|
||||
With AI routes enabled (step 2) and the web dev server running against
|
||||
`localhost:8080`, open the Search page's query bar:
|
||||
|
||||
- Type a partial query and pause — ghost text should appear inline
|
||||
after ~300ms; Tab accepts it. Stop `sentry-mock-ollama` and confirm
|
||||
ghost text just silently stops appearing (no error toast, no
|
||||
console noise) — this is the "graceful degradation" requirement,
|
||||
not incidental behavior.
|
||||
- Run a query that produces a parse or execution error, click "Try AI
|
||||
fix" — the diff view should show the current vs. suggested query, and
|
||||
Accept should replace the query bar's content without running it.
|
||||
- Run `severity=ERROR | stats count by host` (an unbounded aggregation
|
||||
against the real seeded ClickHouse data from
|
||||
`hack/benchmark-fixture`, per Phase 2/5's runbooks) — the inline cost
|
||||
warning should appear, and clicking "Optimize" should show the real
|
||||
mechanical rewrite (`earliest=-1h ` prepended).
|
||||
- Click "Explain this query" on any query — confirm the modal shows
|
||||
prose, not raw JSON (a genuine model would return prose here too;
|
||||
the mock's canned Explain response is deliberately plain text for
|
||||
exactly this reason).
|
||||
|
||||
## 4. Track B — natural-language translation
|
||||
|
||||
Type a natural-language-shaped question into the query bar (4+ words,
|
||||
no `|`/comparison operator/`:`) — e.g. "show me errors from the last
|
||||
hour grouped by service". The "Interpret as natural language" affordance
|
||||
should appear; clicking it opens the translate modal, auto-translates,
|
||||
and shows both the generated query (editable) and an auto-fetched
|
||||
explanation. Confirm "Use this query" replaces the query bar content
|
||||
**without running anything** — no results table should appear until you
|
||||
separately click "Run query".
|
||||
|
||||
CLI:
|
||||
|
||||
```sh
|
||||
cd cli && go run ./cmd/sentryctl query --nl "errors in the last hour" --api http://localhost:8080
|
||||
# prints the translated query and, in an interactive terminal, prompts y/N before running
|
||||
```
|
||||
|
||||
## 5. Audit logging (task 12)
|
||||
|
||||
Requires `enterprise-api` (not core `api`) — core has no
|
||||
`InteractionLogger` wired in by design (see the design doc's "off unless
|
||||
configured" reasoning). With `enterprise-api` running against the
|
||||
compose profile that includes it and AI routes enabled per step 2's
|
||||
pattern applied to that service instead:
|
||||
|
||||
1. Accept or dismiss a Fix/Optimize/Translate suggestion in the web UI.
|
||||
2. Confirm a row landed in `audit_log`:
|
||||
```sh
|
||||
docker exec sentry-metadata-postgres psql -U sentry -d sentry_metadata \
|
||||
-c "SELECT event_type, query_text, detail FROM audit_log WHERE event_type='ai_interaction' ORDER BY id DESC LIMIT 5;"
|
||||
```
|
||||
`detail` should show `operation`/`accepted`/`edited` matching what you
|
||||
just did in the UI.
|
||||
|
||||
This exact path (minus the browser click, using the adapter directly)
|
||||
is what `enterprise/internal/audit/integration_test.go`'s
|
||||
`TestAIInteractionLoggerWritesAttributedToContextIdentity` already
|
||||
proves automatically — see "Running the automated suite" below to run
|
||||
it yourself instead of clicking through the UI.
|
||||
|
||||
## 6. Running the automated suite
|
||||
|
||||
```sh
|
||||
# api module -- includes the new mock-Ollama-backed integration tests
|
||||
# (api/ai/aiapi/integration_test.go), no live infra needed
|
||||
cd api && go build ./... && go vet ./... && go test ./...
|
||||
|
||||
# enterprise module -- same, plus the live-Postgres audit tests (skipped
|
||||
# automatically unless AUDIT_TEST_POSTGRES_ADDR is set)
|
||||
cd enterprise && go build ./... && go vet ./... && go test ./...
|
||||
|
||||
# live-Postgres audit tests specifically, against the real dev stack:
|
||||
docker run --rm --network sentry_default -v "$(pwd):/src" -w /src/enterprise \
|
||||
-e AUDIT_TEST_POSTGRES_ADDR=metadata-postgres:5432 \
|
||||
-e AUDIT_TEST_POSTGRES_PASSWORD=audit-writer-dev-only \
|
||||
-e AUDIT_TEST_ADMIN_PASSWORD=sentry-dev-only \
|
||||
golang:1.25-alpine go test ./internal/audit/... -v
|
||||
|
||||
# cli module
|
||||
cd cli && go build ./... && go vet ./... && go test ./...
|
||||
|
||||
# web
|
||||
cd web && npm run check && npm run build
|
||||
```
|
||||
|
||||
All of the above pass in this environment as of this runbook. The first
|
||||
three don't need Docker or a live database at all except where noted —
|
||||
that's deliberate, see the design doc's CI-testability section.
|
||||
|
||||
## Why a mock provider, not a real model
|
||||
|
||||
Testing against a real Ollama server running the actual pinned
|
||||
`qwen2.5-coder:7b` needs a multi-gigabyte model download and either a
|
||||
GPU or a slow CPU-bound wait per request — infeasible for both this
|
||||
environment and, more importantly, for CI, and non-deterministic enough
|
||||
even at temperature 0 that a failing test wouldn't reliably mean a real
|
||||
regression. `hack/mock-ollama` and `api/ai/aiapi/integration_test.go`'s
|
||||
in-process equivalent both trade away model-quality coverage for
|
||||
plumbing coverage that's actually fast and deterministic enough to run
|
||||
every time — the same tradeoff this project already made for
|
||||
ClickHouse/Postgres-backed pieces of Phase 4 that only "compile and are
|
||||
unit-tested" in environments without live infrastructure. Model-quality
|
||||
verification (does the real model produce good translations for real
|
||||
questions) is real, disclosed future work — a periodic, human-run
|
||||
checklist item against a real local Ollama with the pinned model before
|
||||
a release, not a CI gate.
|
||||
@@ -42,6 +42,9 @@ import (
|
||||
chdriver "github.com/ClickHouse/clickhouse-go/v2"
|
||||
"github.com/jackc/pgx/v5/pgxpool"
|
||||
|
||||
"github.com/sentry/sentry/api/ai/aiapi"
|
||||
"github.com/sentry/sentry/api/ai/provider/ollama"
|
||||
"github.com/sentry/sentry/api/ai/router"
|
||||
"github.com/sentry/sentry/api/authz"
|
||||
"github.com/sentry/sentry/api/dashboards"
|
||||
"github.com/sentry/sentry/api/httpserver"
|
||||
@@ -50,12 +53,17 @@ import (
|
||||
"github.com/sentry/sentry/enterprise/internal/apiconfig"
|
||||
"github.com/sentry/sentry/enterprise/internal/audit"
|
||||
"github.com/sentry/sentry/enterprise/internal/chrunner"
|
||||
"github.com/sentry/sentry/enterprise/internal/groundingregistry"
|
||||
"github.com/sentry/sentry/enterprise/internal/rbacstore"
|
||||
"github.com/sentry/sentry/enterprise/internal/searchclient"
|
||||
"github.com/sentry/sentry/enterprise/internal/tenantcrd"
|
||||
"github.com/sentry/sentry/enterprise/internal/tenantprovision"
|
||||
)
|
||||
|
||||
// groundingRefreshInterval matches api/cmd/api's own constant of the
|
||||
// same name and reasoning -- see that file's doc comment.
|
||||
const groundingRefreshInterval = time.Minute
|
||||
|
||||
func main() {
|
||||
logger := slog.New(slog.NewJSONHandler(os.Stdout, nil))
|
||||
|
||||
@@ -161,6 +169,27 @@ func main() {
|
||||
queryHandler.RegisterRoutes(mux) // also registers GET /healthz
|
||||
dashboardsHandler.RegisterRoutes(mux)
|
||||
|
||||
// Same "off unless OLLAMA_BASE_URL is set" gate as api/cmd/api --
|
||||
// see that file's doc comment.
|
||||
if cfg.AI.OllamaBaseURL != "" {
|
||||
groundingReg := groundingregistry.New(registry)
|
||||
groundingReg.StartRefreshing(ctx, rbac.ListActiveTenantIDs, groundingRefreshInterval, logger)
|
||||
|
||||
defaultProvider := ollama.New(cfg.AI.OllamaBaseURL, cfg.AI.OllamaModel)
|
||||
aiRouter := router.New(defaultProvider)
|
||||
if cfg.AI.OllamaFastModel != "" && cfg.AI.OllamaFastModel != cfg.AI.OllamaModel {
|
||||
aiRouter.SetOperation(router.OpComplete, ollama.New(cfg.AI.OllamaBaseURL, cfg.AI.OllamaFastModel))
|
||||
}
|
||||
|
||||
// Reuses the same audit_writer-role pool/store auditLogger above
|
||||
// writes through -- same append-only audit_log table, new
|
||||
// event_type (see metadata/migrations/0036).
|
||||
interactionLogger := audit.NewAIInteractionLogger(audit.NewStore(auditPool), audit.SourceAPI)
|
||||
aiHandler := aiapi.NewHandler(logger, aiRouter, groundingReg, authorizer, interactionLogger)
|
||||
aiHandler.RegisterRoutes(mux)
|
||||
logger.Info("ai routes enabled", "ollama_base_url", cfg.AI.OllamaBaseURL, "model", cfg.AI.OllamaModel)
|
||||
}
|
||||
|
||||
srv := &http.Server{
|
||||
Addr: cfg.HTTPListenAddr,
|
||||
Handler: httpserver.WithCORS(mux, cfg.CORSAllowedOrigin),
|
||||
|
||||
@@ -1,5 +1,6 @@
|
||||
// Command enterprise-auth is Sentry's SSO/tenant-provisioning/RBAC
|
||||
// service (commercial license, not AGPL) -- see
|
||||
// service (AGPLv3, same as core -- see
|
||||
// /docs/compliance/license-audit-report.md) -- see
|
||||
// /docs/phase-4-isolation-design.md and /docs/phase-4-rbac-design.md.
|
||||
//
|
||||
// Wires session issuance/validation (internal/session), the
|
||||
|
||||
@@ -48,6 +48,23 @@ type Config struct {
|
||||
// Deployments with no Kubernetes cluster at all (docker-compose)
|
||||
// never set this.
|
||||
TenantCRDNamespace string
|
||||
// AI gates Phase 7's AI-assisted query features, same shape and same
|
||||
// env var names as api/internal/config.AIConfig -- duplicated rather
|
||||
// than imported (that package is under api/internal/, which Go's
|
||||
// internal/ visibility rule blocks a separate module like this one
|
||||
// from importing at all, the same constraint that already moved
|
||||
// querylang/executor and dashboards out of internal/ in earlier
|
||||
// phases) -- matches this file's own existing pattern of
|
||||
// independently defining every field even where it overlaps with
|
||||
// api/internal/config's (Postgres, SearchGRPCAddr, and so on), not a
|
||||
// new inconsistency introduced here.
|
||||
AI AIConfig
|
||||
}
|
||||
|
||||
type AIConfig struct {
|
||||
OllamaBaseURL string
|
||||
OllamaModel string
|
||||
OllamaFastModel string
|
||||
}
|
||||
|
||||
type ClickHouseAdminConfig struct {
|
||||
@@ -84,6 +101,11 @@ func Load() (Config, error) {
|
||||
Username: getenv("POSTGRES_USERNAME", "sentry"),
|
||||
Password: getenv("POSTGRES_PASSWORD", ""),
|
||||
},
|
||||
AI: AIConfig{
|
||||
OllamaBaseURL: getenv("OLLAMA_BASE_URL", ""),
|
||||
OllamaModel: getenv("OLLAMA_MODEL", "qwen2.5-coder:7b"),
|
||||
OllamaFastModel: getenv("OLLAMA_FAST_MODEL", "qwen2.5-coder:1.5b"),
|
||||
},
|
||||
AuditWriter: AuditWriterConfig{
|
||||
Username: getenv("AUDIT_WRITER_USERNAME", "audit_writer"),
|
||||
Password: getenv("AUDIT_WRITER_PASSWORD", ""),
|
||||
|
||||
@@ -0,0 +1,79 @@
|
||||
// Adapts *Store to api/ai/aiapi.InteractionLogger -- same shape as
|
||||
// queryapi_adapter.go's QueryAPILogger, wired in by
|
||||
// enterprise/cmd/enterprise-api alongside it (Phase 7 task 12).
|
||||
package audit
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
|
||||
"github.com/sentry/sentry/api/ai/aiapi"
|
||||
"github.com/sentry/sentry/api/authz"
|
||||
)
|
||||
|
||||
// AIInteractionLogger implements aiapi.InteractionLogger by translating
|
||||
// its InteractionEntry into this package's Entry, reading tenant/user
|
||||
// identity from ctx -- same "read identity from ctx rather than the
|
||||
// interface growing tenant-awareness" shape as QueryAPILogger.
|
||||
type AIInteractionLogger struct {
|
||||
store *Store
|
||||
source Source
|
||||
}
|
||||
|
||||
func NewAIInteractionLogger(store *Store, source Source) *AIInteractionLogger {
|
||||
return &AIInteractionLogger{store: store, source: source}
|
||||
}
|
||||
|
||||
// aiInteractionDetail is what Detail carries -- Operation/Confidence/
|
||||
// Accepted/Edited don't have dedicated audit_log columns (same reasoning
|
||||
// as role_change/grant_change already using Detail instead of
|
||||
// query_text/row_count/duration_ms), only QueryText (FinalQuery) does.
|
||||
type aiInteractionDetail struct {
|
||||
Operation string `json:"operation"`
|
||||
Input string `json:"input"`
|
||||
Output string `json:"output"`
|
||||
Confidence string `json:"confidence,omitempty"`
|
||||
Accepted bool `json:"accepted"`
|
||||
Edited bool `json:"edited"`
|
||||
}
|
||||
|
||||
func (l *AIInteractionLogger) LogInteraction(ctx context.Context, entry aiapi.InteractionEntry) error {
|
||||
identity, ok := authz.IdentityFromContext(ctx)
|
||||
if !ok || identity.TenantID == "" {
|
||||
return fmt.Errorf("audit: no tenant identity in context, refusing to write an unattributable audit entry")
|
||||
}
|
||||
|
||||
var userID *string
|
||||
if identity.UserID != "" {
|
||||
userID = &identity.UserID
|
||||
}
|
||||
|
||||
detail, err := json.Marshal(aiInteractionDetail{
|
||||
Operation: entry.Operation,
|
||||
Input: entry.Input,
|
||||
Output: entry.Output,
|
||||
Confidence: entry.Confidence,
|
||||
Accepted: entry.Accepted,
|
||||
Edited: entry.Edited,
|
||||
})
|
||||
if err != nil {
|
||||
return fmt.Errorf("audit: marshaling ai interaction detail: %w", err)
|
||||
}
|
||||
|
||||
var queryText *string
|
||||
if entry.FinalQuery != "" {
|
||||
queryText = &entry.FinalQuery
|
||||
}
|
||||
|
||||
_, err = l.store.Append(ctx, Entry{
|
||||
TenantID: identity.TenantID,
|
||||
UserID: userID,
|
||||
Source: l.source,
|
||||
EventType: EventAIInteraction,
|
||||
QueryText: queryText,
|
||||
Status: StatusSuccess,
|
||||
Detail: detail,
|
||||
})
|
||||
return err
|
||||
}
|
||||
@@ -47,6 +47,12 @@ const (
|
||||
EventGrantChange EventType = "grant_change"
|
||||
EventSSOConfigChange EventType = "sso_config_change"
|
||||
EventSecretReveal EventType = "secret_reveal"
|
||||
// EventAIInteraction (Phase 7 task 12): a translate/fix/optimize
|
||||
// suggestion's accept-or-dismiss outcome. QueryText carries the
|
||||
// resulting query (if any); Detail carries the operation, the
|
||||
// original input, confidence, and whether the user edited the
|
||||
// suggestion before using it -- see ai_interaction_adapter.go.
|
||||
EventAIInteraction EventType = "ai_interaction"
|
||||
)
|
||||
|
||||
type Status string
|
||||
|
||||
@@ -13,6 +13,7 @@ package audit
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"os"
|
||||
"path/filepath"
|
||||
@@ -22,6 +23,7 @@ import (
|
||||
|
||||
"github.com/jackc/pgx/v5/pgxpool"
|
||||
|
||||
"github.com/sentry/sentry/api/ai/aiapi"
|
||||
"github.com/sentry/sentry/api/authz"
|
||||
"github.com/sentry/sentry/api/queryapi"
|
||||
)
|
||||
@@ -140,6 +142,74 @@ func TestQueryAPILoggerRefusesWithoutIdentity(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
// TestAIInteractionLoggerWritesAttributedToContextIdentity is
|
||||
// AIInteractionLogger's counterpart to TestQueryAPILoggerWritesAttributedToContextIdentity
|
||||
// above (Phase 7 task 12) -- same "reads identity from ctx" contract,
|
||||
// plus a check that Detail actually round-trips the operation/
|
||||
// confidence/accepted/edited fields that don't have dedicated columns.
|
||||
func TestAIInteractionLoggerWritesAttributedToContextIdentity(t *testing.T) {
|
||||
writerPool := testPool(t, "audit_writer", os.Getenv("AUDIT_TEST_POSTGRES_PASSWORD"))
|
||||
adminPool := testPool(t, "sentry", os.Getenv("AUDIT_TEST_ADMIN_PASSWORD"))
|
||||
cleanupAuditLog(t, adminPool)
|
||||
defer cleanupAuditLog(t, adminPool)
|
||||
|
||||
logger := NewAIInteractionLogger(NewStore(writerPool), SourceAPI)
|
||||
ctx := authz.WithIdentity(context.Background(), authz.Identity{TenantID: "acme", UserID: "22222222-2222-2222-2222-222222222222", Role: authz.RoleViewer})
|
||||
|
||||
err := logger.LogInteraction(ctx, aiapi.InteractionEntry{
|
||||
Operation: "translate",
|
||||
Input: "errors in the last hour",
|
||||
Output: "earliest=-1h severity=ERROR",
|
||||
Confidence: "high",
|
||||
Accepted: true,
|
||||
Edited: false,
|
||||
FinalQuery: "earliest=-1h severity=ERROR",
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("LogInteraction: %v", err)
|
||||
}
|
||||
|
||||
var tenantID, userID, eventType, queryText string
|
||||
var detail []byte
|
||||
row := adminPool.QueryRow(context.Background(),
|
||||
`SELECT tenant_id, user_id, event_type, query_text, detail FROM audit_log ORDER BY id DESC LIMIT 1`)
|
||||
if err := row.Scan(&tenantID, &userID, &eventType, &queryText, &detail); err != nil {
|
||||
t.Fatalf("reading back the written row: %v", err)
|
||||
}
|
||||
if tenantID != "acme" || userID != "22222222-2222-2222-2222-222222222222" {
|
||||
t.Fatalf("got tenant_id=%q user_id=%q, want acme/22222222-...", tenantID, userID)
|
||||
}
|
||||
if eventType != "ai_interaction" {
|
||||
t.Fatalf("event_type = %q, want ai_interaction", eventType)
|
||||
}
|
||||
if queryText != "earliest=-1h severity=ERROR" {
|
||||
t.Fatalf("query_text = %q, want the final query", queryText)
|
||||
}
|
||||
var parsed struct {
|
||||
Operation string `json:"operation"`
|
||||
Accepted bool `json:"accepted"`
|
||||
}
|
||||
if err := json.Unmarshal(detail, &parsed); err != nil {
|
||||
t.Fatalf("unmarshaling detail: %v", err)
|
||||
}
|
||||
if parsed.Operation != "translate" || !parsed.Accepted {
|
||||
t.Fatalf("detail = %+v, want operation=translate accepted=true", parsed)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAIInteractionLoggerRefusesWithoutIdentity(t *testing.T) {
|
||||
writerPool := testPool(t, "audit_writer", os.Getenv("AUDIT_TEST_POSTGRES_PASSWORD"))
|
||||
adminPool := testPool(t, "sentry", os.Getenv("AUDIT_TEST_ADMIN_PASSWORD"))
|
||||
cleanupAuditLog(t, adminPool)
|
||||
defer cleanupAuditLog(t, adminPool)
|
||||
|
||||
logger := NewAIInteractionLogger(NewStore(writerPool), SourceAPI)
|
||||
err := logger.LogInteraction(context.Background(), aiapi.InteractionEntry{Operation: "fix", Accepted: false})
|
||||
if err == nil {
|
||||
t.Fatal("expected LogInteraction to refuse writing an entry with no tenant identity in context")
|
||||
}
|
||||
}
|
||||
|
||||
// TestVerifyChainDetectsTampering proves the chain actually catches an
|
||||
// in-place row modification -- not just that VerifyChain runs without
|
||||
// erroring on untampered data, which a bug returning OK unconditionally
|
||||
|
||||
@@ -0,0 +1,144 @@
|
||||
// Package groundingregistry gives each active tenant its own schema-
|
||||
// grounding snapshot (Phase 7 task 3) in a multi-tenant deployment,
|
||||
// mirroring enterprise/internal/chwriter.Registry's per-tenant-instance
|
||||
// shape. It exists because api/ai/grounding.Service is deliberately
|
||||
// tenant-agnostic (it just wraps whatever executor.SQLRunner it's given
|
||||
// and caches one snapshot) -- a multi-tenant deployment needs many
|
||||
// snapshots, one per tenant, refreshed independently.
|
||||
//
|
||||
// The underlying SQLRunner every tenant's Service samples through is the
|
||||
// *same* chrunner.Registry instance shared across all of them: chrunner
|
||||
// resolves which tenant's actual ClickHouse connection to use from the
|
||||
// context.Context passed to RunSQL, not from anything this package
|
||||
// stores per tenant -- see chrunner.Registry.RunSQL's doc comment. So
|
||||
// "one grounding.Service per tenant" doesn't mean one ClickHouse
|
||||
// connection per tenant here (chrunner already owns that); it means one
|
||||
// cached snapshot per tenant, refreshed by calling that tenant's
|
||||
// Service.Refresh with a context stamped with that tenant's identity via
|
||||
// api/authz.WithIdentity -- the same "construct our own request context
|
||||
// outside an HTTP handler" pattern that function's doc comment names
|
||||
// this exact kind of caller as being for.
|
||||
package groundingregistry
|
||||
|
||||
import (
|
||||
"context"
|
||||
"log/slog"
|
||||
"sync"
|
||||
"time"
|
||||
|
||||
"github.com/sentry/sentry/api/ai/grounding"
|
||||
"github.com/sentry/sentry/api/ai/provider"
|
||||
"github.com/sentry/sentry/api/authz"
|
||||
"github.com/sentry/sentry/api/querylang/executor"
|
||||
)
|
||||
|
||||
// TenantLister returns the currently-active tenant IDs to sample --
|
||||
// a narrow function type rather than an rbacstore dependency, same
|
||||
// reasoning chwriter.Registry's SourceLister gives: this package
|
||||
// shouldn't need to import rbacstore just to know its return type.
|
||||
// enterprise-api's main.go supplies one backed by
|
||||
// rbacstore.ListProvisionedDataSources, the same source chrunner/
|
||||
// chwriter's own registries already refresh from.
|
||||
type TenantLister func(ctx context.Context) ([]string, error)
|
||||
|
||||
// Registry holds one grounding.Service per active tenant, all sharing
|
||||
// the same underlying SQLRunner (chrunner.Registry).
|
||||
type Registry struct {
|
||||
runner executor.SQLRunner
|
||||
|
||||
mu sync.RWMutex
|
||||
services map[string]*grounding.Service
|
||||
}
|
||||
|
||||
func New(runner executor.SQLRunner) *Registry {
|
||||
return &Registry{runner: runner, services: make(map[string]*grounding.Service)}
|
||||
}
|
||||
|
||||
// SchemaContextFor returns tenant's cached grounding snapshot, or a
|
||||
// zero-valued SchemaContext if that tenant hasn't been sampled yet (new
|
||||
// tenant, not yet seen by a refresh cycle) -- same "absence is normal,
|
||||
// not an error" posture grounding.Service.Current documents.
|
||||
func (r *Registry) SchemaContextFor(tenantID string) provider.SchemaContext {
|
||||
r.mu.RLock()
|
||||
svc, ok := r.services[tenantID]
|
||||
r.mu.RUnlock()
|
||||
if !ok {
|
||||
return provider.SchemaContext{}
|
||||
}
|
||||
return svc.Current()
|
||||
}
|
||||
|
||||
// SchemaContext implements aiapi.SchemaContextSource, resolving the
|
||||
// tenant from ctx the same way chrunner.RunSQL does -- the multi-tenant
|
||||
// counterpart to grounding.Service's own same-named method, which has
|
||||
// no tenant to resolve in a single-tenant deployment. An unauthenticated
|
||||
// or tenant-less context (shouldn't happen behind aiapi's RoleViewer
|
||||
// auth wrapper, but handled rather than assumed) returns a zero-valued
|
||||
// SchemaContext, same as an unseen tenant -- absence is normal here, not
|
||||
// worth a panic or a swallowed error over.
|
||||
func (r *Registry) SchemaContext(ctx context.Context) provider.SchemaContext {
|
||||
id, ok := authz.IdentityFromContext(ctx)
|
||||
if !ok || id.TenantID == "" {
|
||||
return provider.SchemaContext{}
|
||||
}
|
||||
return r.SchemaContextFor(id.TenantID)
|
||||
}
|
||||
|
||||
// StartRefreshing lists active tenants and refreshes each one's
|
||||
// grounding snapshot, immediately and then on interval, until ctx is
|
||||
// cancelled -- same shape as chwriter.Registry.StartRefreshing. A
|
||||
// newly-active tenant gets a Service the first time it appears in
|
||||
// lister's output; a tenant that's no longer listed keeps its last
|
||||
// snapshot rather than being torn down (grounding data going briefly
|
||||
// stale for a deprovisioned tenant is harmless -- unlike a ClickHouse
|
||||
// writer connection, there's no credential to leak or clean up here).
|
||||
func (r *Registry) StartRefreshing(ctx context.Context, lister TenantLister, interval time.Duration, logger *slog.Logger) {
|
||||
r.refreshAll(ctx, lister, logger)
|
||||
go func() {
|
||||
ticker := time.NewTicker(interval)
|
||||
defer ticker.Stop()
|
||||
for {
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
return
|
||||
case <-ticker.C:
|
||||
r.refreshAll(ctx, lister, logger)
|
||||
}
|
||||
}
|
||||
}()
|
||||
}
|
||||
|
||||
func (r *Registry) refreshAll(ctx context.Context, lister TenantLister, logger *slog.Logger) {
|
||||
tenantIDs, err := lister(ctx)
|
||||
if err != nil {
|
||||
if logger != nil {
|
||||
logger.Error("groundingregistry: listing active tenants", "error", err)
|
||||
}
|
||||
return
|
||||
}
|
||||
for _, tenantID := range tenantIDs {
|
||||
svc := r.serviceFor(tenantID)
|
||||
tenantCtx := authz.WithIdentity(ctx, authz.Identity{TenantID: tenantID, Role: authz.RoleService})
|
||||
if err := svc.Refresh(tenantCtx); err != nil && logger != nil {
|
||||
logger.Error("groundingregistry: refreshing tenant", "tenant", tenantID, "error", err)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func (r *Registry) serviceFor(tenantID string) *grounding.Service {
|
||||
r.mu.RLock()
|
||||
svc, ok := r.services[tenantID]
|
||||
r.mu.RUnlock()
|
||||
if ok {
|
||||
return svc
|
||||
}
|
||||
|
||||
r.mu.Lock()
|
||||
defer r.mu.Unlock()
|
||||
if svc, ok := r.services[tenantID]; ok { // re-check under write lock
|
||||
return svc
|
||||
}
|
||||
svc = grounding.New(r.runner)
|
||||
r.services[tenantID] = svc
|
||||
return svc
|
||||
}
|
||||
@@ -0,0 +1,74 @@
|
||||
package groundingregistry
|
||||
|
||||
import (
|
||||
"context"
|
||||
"testing"
|
||||
|
||||
"github.com/sentry/sentry/api/authz"
|
||||
"github.com/sentry/sentry/api/querylang/executor"
|
||||
)
|
||||
|
||||
// tenantAwareFakeRunner returns a service list keyed by the tenant
|
||||
// identity RunSQL is called with -- confirms groundingregistry actually
|
||||
// stamps a different tenant per Service.Refresh call, not the same
|
||||
// context reused for everyone (chrunner.Registry's real RunSQL resolves
|
||||
// tenant from ctx the same way, so this fake exercises the same
|
||||
// contract).
|
||||
type tenantAwareFakeRunner struct {
|
||||
byTenant map[string][]string
|
||||
}
|
||||
|
||||
func (f *tenantAwareFakeRunner) RunSQL(ctx context.Context, sql string) (*executor.Result, error) {
|
||||
id, ok := authz.IdentityFromContext(ctx)
|
||||
if !ok {
|
||||
return &executor.Result{Columns: []string{"service"}, Rows: nil}, nil
|
||||
}
|
||||
services := f.byTenant[id.TenantID]
|
||||
rows := make([][]any, len(services))
|
||||
for i, s := range services {
|
||||
rows[i] = []any{s}
|
||||
}
|
||||
return &executor.Result{Columns: []string{"service"}, Rows: rows}, nil
|
||||
}
|
||||
|
||||
func TestRefreshAllScopesEachTenantIndependently(t *testing.T) {
|
||||
runner := &tenantAwareFakeRunner{byTenant: map[string][]string{
|
||||
"tenant-a": {"api-a"},
|
||||
"tenant-b": {"api-b", "worker-b"},
|
||||
}}
|
||||
reg := New(runner)
|
||||
lister := func(context.Context) ([]string, error) {
|
||||
return []string{"tenant-a", "tenant-b"}, nil
|
||||
}
|
||||
|
||||
reg.refreshAll(context.Background(), lister, nil)
|
||||
|
||||
a := reg.SchemaContextFor("tenant-a")
|
||||
if len(a.Services) != 1 || a.Services[0] != "api-a" {
|
||||
t.Errorf("tenant-a grounding = %v, want [api-a]", a.Services)
|
||||
}
|
||||
b := reg.SchemaContextFor("tenant-b")
|
||||
if len(b.Services) != 2 {
|
||||
t.Errorf("tenant-b grounding = %v, want 2 services", b.Services)
|
||||
}
|
||||
|
||||
unknown := reg.SchemaContextFor("tenant-never-seen")
|
||||
if len(unknown.Services) != 0 || len(unknown.Fields) != 0 {
|
||||
t.Errorf("unseen tenant should get a zero-valued SchemaContext, got %+v", unknown)
|
||||
}
|
||||
}
|
||||
|
||||
func TestRefreshAllListerErrorLeavesExistingSnapshots(t *testing.T) {
|
||||
runner := &tenantAwareFakeRunner{byTenant: map[string][]string{"tenant-a": {"api-a"}}}
|
||||
reg := New(runner)
|
||||
good := func(context.Context) ([]string, error) { return []string{"tenant-a"}, nil }
|
||||
reg.refreshAll(context.Background(), good, nil)
|
||||
|
||||
failing := func(context.Context) ([]string, error) { return nil, context.DeadlineExceeded }
|
||||
reg.refreshAll(context.Background(), failing, nil)
|
||||
|
||||
a := reg.SchemaContextFor("tenant-a")
|
||||
if len(a.Services) != 1 {
|
||||
t.Errorf("tenant-a grounding after a failed lister call = %v, want unchanged [api-a]", a.Services)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,3 @@
|
||||
module github.com/sentry/sentry/hack/mock-ollama
|
||||
|
||||
go 1.25.0
|
||||
@@ -0,0 +1,100 @@
|
||||
// Command mock-ollama stands in for a real Ollama server (see
|
||||
// /docs/phase-7-ai-design.md) when manually verifying Phase 7's AI
|
||||
// features against a live docker-compose stack without needing model
|
||||
// weights or a GPU. Matches Ollama's real POST /api/chat wire contract
|
||||
// (see api/ai/provider/ollama/ollama.go's chatRequest/chatResponse)
|
||||
// closely enough that api/cmd/api and enterprise/cmd/enterprise-api
|
||||
// can't tell the difference -- picks a canned, deterministic response by
|
||||
// inspecting the system prompt's distinctive opening line (see
|
||||
// api/ai/provider/ollama/prompts.go), the same technique the
|
||||
// integration tests in api/ai/aiapi/integration_test.go use for the
|
||||
// same reason: no live model, deterministic output, fast.
|
||||
//
|
||||
// Not part of any docker-compose service by default -- run it
|
||||
// standalone (or as a throwaway container on the sentry_default
|
||||
// network) and point OLLAMA_BASE_URL at it. See
|
||||
// /docs/phase-7-runbook.md for the exact recipe.
|
||||
package main
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"flag"
|
||||
"fmt"
|
||||
"io"
|
||||
"log"
|
||||
"net/http"
|
||||
"strings"
|
||||
)
|
||||
|
||||
type chatMessage struct {
|
||||
Role string `json:"role"`
|
||||
Content string `json:"content"`
|
||||
}
|
||||
|
||||
type chatRequest struct {
|
||||
Model string `json:"model"`
|
||||
Messages []chatMessage `json:"messages"`
|
||||
}
|
||||
|
||||
type chatResponse struct {
|
||||
Message chatMessage `json:"message"`
|
||||
}
|
||||
|
||||
// canned responses, keyed by a distinctive substring of each
|
||||
// operation's system prompt (see prompts.go's opening sentence for
|
||||
// each). Content is deliberately valid, uninteresting pipe syntax --
|
||||
// this tool exists to verify plumbing, not to simulate model quality.
|
||||
var canned = []struct {
|
||||
systemPromptContains string
|
||||
response string
|
||||
}{
|
||||
{"translate a plain-English question", `{"query":"earliest=-1h severity=ERROR","confidence":"high"}`},
|
||||
{"suggest how to continue", `{"suggestion":" severity=ERROR"}`},
|
||||
{"phrase those findings", "Add a time range (e.g. earliest=-1h) to avoid scanning the entire table."},
|
||||
{"fix a broken", `{"suggested_query":"earliest=-1h severity=ERROR","explanation":"added a missing time bound","confidence":"high"}`},
|
||||
// Plain explain (no findings, no original-intent framing) falls
|
||||
// through to this last, broadest match.
|
||||
{"", "This query filters logs where severity equals ERROR from the last hour."},
|
||||
}
|
||||
|
||||
func respond(system string) string {
|
||||
for _, c := range canned {
|
||||
if strings.Contains(system, c.systemPromptContains) {
|
||||
return c.response
|
||||
}
|
||||
}
|
||||
return canned[len(canned)-1].response
|
||||
}
|
||||
|
||||
func main() {
|
||||
addr := flag.String("addr", ":11434", "listen address")
|
||||
flag.Parse()
|
||||
|
||||
http.HandleFunc("/api/chat", func(w http.ResponseWriter, r *http.Request) {
|
||||
body, err := io.ReadAll(r.Body)
|
||||
if err != nil {
|
||||
http.Error(w, "reading body: "+err.Error(), http.StatusBadRequest)
|
||||
return
|
||||
}
|
||||
var req chatRequest
|
||||
if err := json.Unmarshal(body, &req); err != nil {
|
||||
http.Error(w, "decoding request: "+err.Error(), http.StatusBadRequest)
|
||||
return
|
||||
}
|
||||
var system string
|
||||
for _, m := range req.Messages {
|
||||
if m.Role == "system" {
|
||||
system = m.Content
|
||||
break
|
||||
}
|
||||
}
|
||||
content := respond(system)
|
||||
fmt.Printf("chat: model=%s -> %s\n", req.Model, content)
|
||||
|
||||
w.Header().Set("Content-Type", "application/json")
|
||||
_ = json.NewEncoder(w).Encode(chatResponse{Message: chatMessage{Role: "assistant", Content: content}})
|
||||
})
|
||||
|
||||
log.Printf("mock-ollama listening on %s", *addr)
|
||||
log.Fatal(http.ListenAndServe(*addr, nil))
|
||||
}
|
||||
@@ -0,0 +1,14 @@
|
||||
-- Phase 7 task 12: AI-assisted interactions (translate/fix/optimize
|
||||
-- accept-or-dismiss decisions) are audited through the *same*
|
||||
-- append-only, hash-chained audit_log table Phase 4 built -- not a new
|
||||
-- table -- since audit_log's detail JSONB column and event_type
|
||||
-- discriminator were already designed to carry event shapes other than
|
||||
-- "query" (role_change/grant_change/etc. already don't populate
|
||||
-- row_count/duration_ms, relying on detail instead; ai_interaction is
|
||||
-- the same shape of extension). Postgres has no ALTER CHECK, so drop
|
||||
-- and recreate, same pattern as 0035_add_heatmap_viz_type.sql.
|
||||
ALTER TABLE audit_log DROP CONSTRAINT audit_log_event_type_check;
|
||||
|
||||
ALTER TABLE audit_log
|
||||
ADD CONSTRAINT audit_log_event_type_check
|
||||
CHECK (event_type IN ('query', 'role_change', 'grant_change', 'sso_config_change', 'secret_reveal', 'ai_interaction'));
|
||||
+467
-4
@@ -2,9 +2,23 @@
|
||||
// Extracted from the root query page in Phase 3 so the dashboard panel
|
||||
// editor and the alert rule editor can reuse the same input --
|
||||
// deliberately just the input+run affordance, not results/history,
|
||||
// which differ per consumer.
|
||||
// which differ per consumer. Phase 7 adds AI-assisted authoring
|
||||
// (explain/fix/optimize) here rather than as a separate mode/page, so
|
||||
// every consumer of this component gets it -- errorMessage/warnings
|
||||
// are optional props specifically so existing callers that don't pass
|
||||
// them see no behavior change at all.
|
||||
import type { Language } from '$lib/api';
|
||||
import { Button } from '$lib/components/ui';
|
||||
import {
|
||||
aiExplain,
|
||||
aiFix,
|
||||
aiOptimize,
|
||||
aiTranslate,
|
||||
logInteraction,
|
||||
type FixResponse,
|
||||
type OptimizeResponse,
|
||||
type TranslateResponse
|
||||
} from '$lib/api';
|
||||
import { Button, Modal } from '$lib/components/ui';
|
||||
import QueryEditor from '$lib/query-editor/QueryEditor.svelte';
|
||||
|
||||
let {
|
||||
@@ -12,13 +26,23 @@
|
||||
language = $bindable<Language>(''),
|
||||
onRun,
|
||||
loading = false,
|
||||
placeholder = 'service=api | where status>=500 | stats count by host | sort -count'
|
||||
placeholder = 'service=api | where status>=500 | stats count by host | sort -count',
|
||||
// Set by the caller after a failed run (parse or execution error)
|
||||
// -- presence alone drives the "Fix this" affordance below, this
|
||||
// component never calls runQuery itself to find out.
|
||||
errorMessage = '',
|
||||
// Set by the caller from a successful run's QueryResult.warnings
|
||||
// (costguard, Phase 7 task 4) -- presence alone drives the
|
||||
// "Optimize" affordance.
|
||||
warnings = []
|
||||
}: {
|
||||
query: string;
|
||||
language: Language;
|
||||
onRun: () => void;
|
||||
loading?: boolean;
|
||||
placeholder?: string;
|
||||
errorMessage?: string;
|
||||
warnings?: string[];
|
||||
} = $props();
|
||||
|
||||
// Client-side mirror of the backend's auto-detect heuristic
|
||||
@@ -30,10 +54,245 @@
|
||||
}
|
||||
let detected = $derived(detectedLanguage(query));
|
||||
let effectiveLanguage = $derived(language === '' ? detected : language);
|
||||
|
||||
// ---- "looks like natural language" detection (task 10) ----
|
||||
//
|
||||
// Deliberately NOT "wait for a parse error" -- the pipe grammar's
|
||||
// own free-text rule (bare words AND'd together, see
|
||||
// /docs/query-language-reference.md's "Free-text search" section)
|
||||
// means a plain-English question like "show me errors from the last
|
||||
// hour" *parses successfully* as a search for records containing
|
||||
// all of those words literally. It never fails to parse; it just
|
||||
// silently returns an unhelpful result. Waiting for a parse error
|
||||
// would miss the single most common real case this feature exists
|
||||
// for. Instead: a cheap client-side heuristic flags text that has
|
||||
// none of the pipe syntax's structural markers (`|`, a comparison
|
||||
// operator, `:`) and is long enough (4+ words) that it's very
|
||||
// unlikely to be an intentional short free-text search someone
|
||||
// actually wants run literally -- a single bare word or a quoted
|
||||
// phrase is common and legitimate, and stays untouched.
|
||||
function looksLikeNaturalLanguage(q: string): boolean {
|
||||
const trimmed = q.trim();
|
||||
if (trimmed === '' || /^\s*select\b/i.test(trimmed)) return false;
|
||||
if (/[|=<>:]/.test(trimmed)) return false;
|
||||
return trimmed.split(/\s+/).length >= 4;
|
||||
}
|
||||
let looksLikeNL = $derived(looksLikeNaturalLanguage(query));
|
||||
|
||||
// ---- interaction audit logging (task 12) ----
|
||||
// Fire-and-forget: an audit write failure here must never block or
|
||||
// surface an error on the button press that triggered it (same
|
||||
// posture as the AI operations themselves being optional). Scoped to
|
||||
// translate/fix/optimize only -- see logInteraction's/InteractionLogger's
|
||||
// doc comments for why complete and explain are excluded.
|
||||
function logFixOrOptimizeInteraction(
|
||||
operation: 'fix' | 'optimize',
|
||||
input: string,
|
||||
suggestedQuery: string,
|
||||
accepted: boolean
|
||||
) {
|
||||
logInteraction({
|
||||
operation,
|
||||
input,
|
||||
output: suggestedQuery,
|
||||
accepted,
|
||||
edited: false,
|
||||
finalQuery: suggestedQuery
|
||||
}).catch(() => {});
|
||||
}
|
||||
|
||||
// ---- explain ----
|
||||
let explainOpen = $state(false);
|
||||
let explainLoading = $state(false);
|
||||
let explainText = $state('');
|
||||
let explainError = $state('');
|
||||
|
||||
async function runExplain() {
|
||||
explainOpen = true;
|
||||
explainLoading = true;
|
||||
explainError = '';
|
||||
explainText = '';
|
||||
try {
|
||||
const res = await aiExplain(query, effectiveLanguage);
|
||||
explainText = res.explanation;
|
||||
} catch {
|
||||
// Any failure (including "AI not configured," a plain 404) is
|
||||
// shown as a quiet unavailable message, not a scary error --
|
||||
// this is an optional enhancement, not a core feature whose
|
||||
// failure should read as something broken.
|
||||
explainError = 'AI explanation is not available right now.';
|
||||
} finally {
|
||||
explainLoading = false;
|
||||
}
|
||||
}
|
||||
|
||||
// ---- fix ----
|
||||
let fixOpen = $state(false);
|
||||
let fixLoading = $state(false);
|
||||
let fixResult: FixResponse | null = $state(null);
|
||||
let fixError = $state('');
|
||||
|
||||
async function runFix() {
|
||||
fixOpen = true;
|
||||
fixLoading = true;
|
||||
fixError = '';
|
||||
fixResult = null;
|
||||
try {
|
||||
fixResult = await aiFix(query, effectiveLanguage, { executionError: errorMessage });
|
||||
} catch {
|
||||
fixError = 'AI fix suggestions are not available right now.';
|
||||
} finally {
|
||||
fixLoading = false;
|
||||
}
|
||||
}
|
||||
|
||||
function acceptFix() {
|
||||
if (!fixResult?.suggestedQuery) return;
|
||||
logFixOrOptimizeInteraction('fix', query, fixResult.suggestedQuery, true);
|
||||
query = fixResult.suggestedQuery;
|
||||
fixOpen = false;
|
||||
}
|
||||
|
||||
function dismissFix() {
|
||||
if (fixResult?.suggestedQuery) {
|
||||
logFixOrOptimizeInteraction('fix', query, fixResult.suggestedQuery, false);
|
||||
}
|
||||
fixOpen = false;
|
||||
}
|
||||
|
||||
// ---- optimize ----
|
||||
let optimizeOpen = $state(false);
|
||||
let optimizeLoading = $state(false);
|
||||
let optimizeResult: OptimizeResponse | null = $state(null);
|
||||
let optimizeError = $state('');
|
||||
|
||||
async function runOptimize() {
|
||||
optimizeOpen = true;
|
||||
optimizeLoading = true;
|
||||
optimizeError = '';
|
||||
optimizeResult = null;
|
||||
try {
|
||||
optimizeResult = await aiOptimize(query, effectiveLanguage);
|
||||
} catch {
|
||||
optimizeError = 'AI optimization suggestions are not available right now.';
|
||||
} finally {
|
||||
optimizeLoading = false;
|
||||
}
|
||||
}
|
||||
|
||||
function acceptOptimize() {
|
||||
if (!optimizeResult?.suggestedQuery) return;
|
||||
logFixOrOptimizeInteraction('optimize', query, optimizeResult.suggestedQuery, true);
|
||||
query = optimizeResult.suggestedQuery;
|
||||
optimizeOpen = false;
|
||||
}
|
||||
|
||||
function dismissOptimize() {
|
||||
if (optimizeResult?.suggestedQuery) {
|
||||
logFixOrOptimizeInteraction('optimize', query, optimizeResult.suggestedQuery, false);
|
||||
}
|
||||
optimizeOpen = false;
|
||||
}
|
||||
|
||||
// ---- translate (Track B) ----
|
||||
let translateOpen = $state(false);
|
||||
let translateLoading = $state(false);
|
||||
let translateNL = $state('');
|
||||
let translateResult: TranslateResponse | null = $state(null);
|
||||
let translateExplanation = $state('');
|
||||
let translateError = $state('');
|
||||
// Tracks edits made in the review textarea after a result arrives --
|
||||
// see editedQuery's own comment below for why this exists.
|
||||
let editedQuery = $state('');
|
||||
|
||||
function openTranslate() {
|
||||
// Pre-fill with the current query bar content -- that's exactly
|
||||
// what triggered the "looks like natural language" affordance in
|
||||
// the first place, so re-typing it would be pure friction.
|
||||
translateNL = query;
|
||||
translateOpen = true;
|
||||
runTranslate();
|
||||
}
|
||||
|
||||
async function runTranslate() {
|
||||
translateLoading = true;
|
||||
translateError = '';
|
||||
translateResult = null;
|
||||
translateExplanation = '';
|
||||
try {
|
||||
const res = await aiTranslate(translateNL);
|
||||
translateResult = res;
|
||||
editedQuery = res.query;
|
||||
if (res.query && res.compiles) {
|
||||
// Reuses Explain rather than building a separate
|
||||
// "describe the translation" mechanism -- task 10's
|
||||
// explicit instruction. OriginalIntent lets the prompt
|
||||
// speak to *how* the question became this query, not
|
||||
// just describe the query in isolation.
|
||||
try {
|
||||
const explainRes = await aiExplain(res.query, 'spl', translateNL);
|
||||
translateExplanation = explainRes.explanation;
|
||||
} catch {
|
||||
// Explanation is a nice-to-have on top of the
|
||||
// translation itself -- a failure here shouldn't
|
||||
// blank out an otherwise-successful translation.
|
||||
}
|
||||
}
|
||||
} catch {
|
||||
translateError = 'AI translation is not available right now.';
|
||||
} finally {
|
||||
translateLoading = false;
|
||||
}
|
||||
}
|
||||
|
||||
function useTranslatedQuery() {
|
||||
if (!editedQuery.trim()) return;
|
||||
if (translateResult) {
|
||||
logInteraction({
|
||||
operation: 'translate',
|
||||
input: translateNL,
|
||||
output: translateResult.query,
|
||||
confidence: translateResult.confidence,
|
||||
accepted: true,
|
||||
edited: editedQuery !== translateResult.query,
|
||||
finalQuery: editedQuery
|
||||
}).catch(() => {});
|
||||
}
|
||||
query = editedQuery;
|
||||
translateOpen = false;
|
||||
}
|
||||
|
||||
function cancelTranslate() {
|
||||
if (translateResult?.query) {
|
||||
logInteraction({
|
||||
operation: 'translate',
|
||||
input: translateNL,
|
||||
output: translateResult.query,
|
||||
confidence: translateResult.confidence,
|
||||
accepted: false,
|
||||
edited: false,
|
||||
finalQuery: translateResult.query
|
||||
}).catch(() => {});
|
||||
}
|
||||
translateOpen = false;
|
||||
}
|
||||
|
||||
// A blocked suggestion the user has since edited is their own text
|
||||
// now, not the original flagged one -- costguard will assess
|
||||
// whatever they actually run anyway (every /query response carries
|
||||
// its own warnings, Track A task 4), so re-blocking an edit they
|
||||
// made specifically to address the concern would be actively
|
||||
// unhelpful, not extra-safe.
|
||||
let translateUseDisabled = $derived.by(() => {
|
||||
if (!editedQuery.trim()) return true;
|
||||
const result = translateResult;
|
||||
if (!result) return false;
|
||||
return result.blocked && editedQuery === result.query;
|
||||
});
|
||||
</script>
|
||||
|
||||
<div class="query-bar">
|
||||
<QueryEditor bind:value={query} {onRun} {placeholder} />
|
||||
<QueryEditor bind:value={query} {onRun} {placeholder} language={effectiveLanguage} />
|
||||
<div class="controls">
|
||||
<label>
|
||||
Language:
|
||||
@@ -49,10 +308,146 @@
|
||||
<Button variant="primary" onclick={onRun} disabled={loading || query.trim() === ''}>
|
||||
{loading ? 'Running…' : 'Run query'}
|
||||
</Button>
|
||||
<Button variant="ghost" size="sm" onclick={runExplain} disabled={query.trim() === ''}>
|
||||
Explain this query
|
||||
</Button>
|
||||
{#if errorMessage}
|
||||
<Button variant="ghost" size="sm" onclick={runFix}>Try AI fix</Button>
|
||||
{/if}
|
||||
{#if warnings.length > 0}
|
||||
<Button variant="ghost" size="sm" onclick={runOptimize}>Optimize</Button>
|
||||
{/if}
|
||||
{#if looksLikeNL}
|
||||
<Button variant="ghost" size="sm" onclick={openTranslate}>Interpret as natural language</Button>
|
||||
{/if}
|
||||
<span class="hint">⌘/Ctrl+Enter to run</span>
|
||||
</div>
|
||||
{#if warnings.length > 0}
|
||||
<p class="cost-warning">⚠ {warnings.join('; ')}</p>
|
||||
{/if}
|
||||
</div>
|
||||
|
||||
<Modal bind:open={explainOpen} title="What this query does">
|
||||
{#if explainLoading}
|
||||
<p class="muted">Thinking…</p>
|
||||
{:else if explainError}
|
||||
<p class="muted">{explainError}</p>
|
||||
{:else}
|
||||
<p>{explainText}</p>
|
||||
{/if}
|
||||
</Modal>
|
||||
|
||||
<Modal bind:open={fixOpen} title="AI-suggested fix">
|
||||
{#if fixLoading}
|
||||
<p class="muted">Thinking…</p>
|
||||
{:else if fixError}
|
||||
<p class="muted">{fixError}</p>
|
||||
{:else if fixResult}
|
||||
{#if !fixResult.suggestedQuery}
|
||||
<p class="muted">
|
||||
{fixResult.explanation || "The AI couldn't determine a fix for this error."}
|
||||
</p>
|
||||
{:else}
|
||||
<div class="diff">
|
||||
<div class="diff-row removed">
|
||||
<span class="diff-label">Current</span>
|
||||
<code>{query}</code>
|
||||
</div>
|
||||
<div class="diff-row added">
|
||||
<span class="diff-label">Suggested</span>
|
||||
<code>{fixResult.suggestedQuery}</code>
|
||||
</div>
|
||||
</div>
|
||||
{#if fixResult.explanation}<p>{fixResult.explanation}</p>{/if}
|
||||
{#if fixResult.blocked}
|
||||
<p class="cost-warning">
|
||||
⚠ This suggestion isn't offered as directly runnable: {(fixResult.costWarnings ?? []).join('; ')}
|
||||
You can still copy it and adjust manually.
|
||||
</p>
|
||||
{/if}
|
||||
<div class="actions">
|
||||
<Button variant="secondary" onclick={dismissFix}>Dismiss</Button>
|
||||
<Button variant="primary" onclick={acceptFix} disabled={fixResult.blocked}>
|
||||
Accept
|
||||
</Button>
|
||||
</div>
|
||||
{/if}
|
||||
{/if}
|
||||
</Modal>
|
||||
|
||||
<Modal bind:open={optimizeOpen} title="Optimize this query">
|
||||
{#if optimizeLoading}
|
||||
<p class="muted">Thinking…</p>
|
||||
{:else if optimizeError}
|
||||
<p class="muted">{optimizeError}</p>
|
||||
{:else if optimizeResult}
|
||||
<p>{optimizeResult.phrased || optimizeResult.findings.join('; ')}</p>
|
||||
{#if optimizeResult.suggestedQuery}
|
||||
<div class="diff">
|
||||
<div class="diff-row removed">
|
||||
<span class="diff-label">Current</span>
|
||||
<code>{query}</code>
|
||||
</div>
|
||||
<div class="diff-row added">
|
||||
<span class="diff-label">Suggested</span>
|
||||
<code>{optimizeResult.suggestedQuery}</code>
|
||||
</div>
|
||||
</div>
|
||||
<div class="actions">
|
||||
<Button variant="secondary" onclick={dismissOptimize}>Dismiss</Button>
|
||||
<Button variant="primary" onclick={acceptOptimize}>Accept</Button>
|
||||
</div>
|
||||
{/if}
|
||||
{/if}
|
||||
</Modal>
|
||||
|
||||
<Modal bind:open={translateOpen} title="Ask in plain English">
|
||||
<label class="nl-label" for="nl-question">Your question</label>
|
||||
<textarea id="nl-question" class="nl-input" bind:value={translateNL} rows="2"></textarea>
|
||||
<div class="actions translate-actions">
|
||||
<Button variant="secondary" size="sm" onclick={runTranslate} disabled={translateLoading || !translateNL.trim()}>
|
||||
{translateLoading ? 'Translating…' : 'Translate again'}
|
||||
</Button>
|
||||
</div>
|
||||
|
||||
{#if translateLoading && !translateResult}
|
||||
<p class="muted">Thinking…</p>
|
||||
{:else if translateError}
|
||||
<p class="muted">{translateError}</p>
|
||||
{:else if translateResult}
|
||||
{#if !translateResult.query}
|
||||
<!-- Honest low-confidence handling (task 10): no guess shown
|
||||
with false confidence, just the reason plainly stated. -->
|
||||
<p class="muted">
|
||||
{translateResult.lowConfidenceReason || "Not confident enough to guess -- try rephrasing."}
|
||||
</p>
|
||||
{:else}
|
||||
{#if translateResult.confidence === 'low'}
|
||||
<p class="cost-warning">
|
||||
⚠ Low confidence{translateResult.lowConfidenceReason ? `: ${translateResult.lowConfidenceReason}` : ''}. Review carefully before using.
|
||||
</p>
|
||||
{/if}
|
||||
<label class="nl-label" for="nl-generated">Generated query (editable)</label>
|
||||
<textarea id="nl-generated" class="nl-input mono" bind:value={editedQuery} rows="3"></textarea>
|
||||
{#if !translateResult.compiles}
|
||||
<p class="cost-warning">⚠ This doesn't parse as a valid query: {translateResult.compileError}. Edit it above before using.</p>
|
||||
{/if}
|
||||
{#if translateExplanation}<p>{translateExplanation}</p>{/if}
|
||||
{#if translateResult.blocked && editedQuery === translateResult.query}
|
||||
<p class="cost-warning">
|
||||
⚠ Not offered as directly runnable: {(translateResult.costWarnings ?? []).join('; ')} Edit the query above to address this, or copy it manually.
|
||||
</p>
|
||||
{/if}
|
||||
<div class="actions">
|
||||
<Button variant="secondary" onclick={cancelTranslate}>Cancel</Button>
|
||||
<Button variant="primary" onclick={useTranslatedQuery} disabled={translateUseDisabled}>
|
||||
Use this query
|
||||
</Button>
|
||||
</div>
|
||||
{/if}
|
||||
{/if}
|
||||
</Modal>
|
||||
|
||||
<style>
|
||||
.controls {
|
||||
margin-top: var(--space-3);
|
||||
@@ -84,4 +479,72 @@
|
||||
font-size: var(--text-sm);
|
||||
color: var(--color-text-muted);
|
||||
}
|
||||
.cost-warning {
|
||||
margin-top: var(--space-2);
|
||||
font-size: var(--text-sm);
|
||||
color: var(--color-sev-warn);
|
||||
}
|
||||
.muted {
|
||||
color: var(--color-text-muted);
|
||||
}
|
||||
.diff {
|
||||
display: flex;
|
||||
flex-direction: column;
|
||||
gap: var(--space-2);
|
||||
font-family: var(--font-mono);
|
||||
font-size: var(--text-sm);
|
||||
margin-bottom: var(--space-3);
|
||||
}
|
||||
.diff-row {
|
||||
display: flex;
|
||||
flex-direction: column;
|
||||
gap: var(--space-1);
|
||||
padding: var(--space-2);
|
||||
border-radius: var(--radius-sm);
|
||||
}
|
||||
.diff-row code {
|
||||
white-space: pre-wrap;
|
||||
word-break: break-word;
|
||||
}
|
||||
.diff-row.removed {
|
||||
background: var(--color-sev-error-bg);
|
||||
}
|
||||
.diff-row.added {
|
||||
background: var(--color-sev-info-bg);
|
||||
}
|
||||
.diff-label {
|
||||
font-family: var(--font-ui);
|
||||
font-size: var(--text-xs);
|
||||
color: var(--color-text-muted);
|
||||
}
|
||||
.actions {
|
||||
display: flex;
|
||||
justify-content: flex-end;
|
||||
gap: var(--space-2);
|
||||
}
|
||||
.nl-label {
|
||||
display: block;
|
||||
font-size: var(--text-xs);
|
||||
color: var(--color-text-muted);
|
||||
margin-bottom: var(--space-1);
|
||||
}
|
||||
.nl-input {
|
||||
width: 100%;
|
||||
box-sizing: border-box;
|
||||
background: var(--color-surface);
|
||||
color: var(--color-text);
|
||||
border: 1px solid var(--color-border);
|
||||
border-radius: var(--radius-sm);
|
||||
padding: var(--space-2);
|
||||
font-family: var(--font-ui);
|
||||
font-size: var(--text-sm);
|
||||
resize: vertical;
|
||||
}
|
||||
.nl-input.mono {
|
||||
font-family: var(--font-mono);
|
||||
}
|
||||
.translate-actions {
|
||||
justify-content: flex-start;
|
||||
margin: var(--space-2) 0 var(--space-3);
|
||||
}
|
||||
</style>
|
||||
|
||||
+119
-1
@@ -16,7 +16,13 @@ export const enterpriseAuthBase = import.meta.env.VITE_ENTERPRISE_AUTH_BASE_URL
|
||||
|
||||
export type Language = '' | 'sql' | 'spl';
|
||||
|
||||
export type QueryResult = { columns: string[]; rows: unknown[][] };
|
||||
// warnings (Phase 7) is populated by the shared costguard package's
|
||||
// assessment of every query, hand-written or AI-suggested alike --
|
||||
// informational only, never a reason a query didn't run. Optional/absent
|
||||
// (not an empty array) when there's nothing to say, matching the
|
||||
// backend's `omitempty` -- see api/queryapi/handler.go's doc comment on
|
||||
// why this is additive, not new enforcement.
|
||||
export type QueryResult = { columns: string[]; rows: unknown[][]; warnings?: string[] };
|
||||
|
||||
export type VizType = 'table' | 'line' | 'bar' | 'single_stat' | 'top_n' | 'heatmap';
|
||||
|
||||
@@ -83,6 +89,117 @@ export function runQuery(query: string, language: Language): Promise<QueryResult
|
||||
return request('/query', { method: 'POST', body: JSON.stringify({ query, language }) });
|
||||
}
|
||||
|
||||
// ---- AI-assisted query authoring (Phase 7 Track A) ----
|
||||
// Every function here returns text/suggestions only -- running anything
|
||||
// still goes through runQuery above, unchanged, per the phase's
|
||||
// non-negotiable "no parallel execution path" design principle. A
|
||||
// deployment with no OLLAMA_BASE_URL configured has these routes
|
||||
// entirely unregistered server-side (api/cmd/api's main.go), so a 404
|
||||
// here is a normal, expected "AI isn't enabled" response, not a bug --
|
||||
// callers (QueryBar.svelte) treat any error from these functions as
|
||||
// "AI unavailable right now," never a user-facing failure.
|
||||
|
||||
export type Confidence = 'high' | 'medium' | 'low';
|
||||
|
||||
export function aiComplete(queryPrefix: string, language: string): Promise<{ suggestion: string }> {
|
||||
return request('/ai/complete', { method: 'POST', body: JSON.stringify({ queryPrefix, language }) });
|
||||
}
|
||||
|
||||
export function aiExplain(
|
||||
query: string,
|
||||
language: string,
|
||||
originalIntent?: string
|
||||
): Promise<{ explanation: string }> {
|
||||
return request('/ai/explain', {
|
||||
method: 'POST',
|
||||
body: JSON.stringify({ query, language, originalIntent: originalIntent ?? '' })
|
||||
});
|
||||
}
|
||||
|
||||
export type FixResponse = {
|
||||
suggestedQuery: string;
|
||||
explanation: string;
|
||||
confidence: Confidence | '';
|
||||
blocked: boolean;
|
||||
costWarnings?: string[];
|
||||
};
|
||||
|
||||
export function aiFix(
|
||||
query: string,
|
||||
language: string,
|
||||
opts: { parseError?: string; executionError?: string }
|
||||
): Promise<FixResponse> {
|
||||
return request('/ai/fix', {
|
||||
method: 'POST',
|
||||
body: JSON.stringify({
|
||||
query,
|
||||
language,
|
||||
parseError: opts.parseError ?? '',
|
||||
executionError: opts.executionError ?? ''
|
||||
})
|
||||
});
|
||||
}
|
||||
|
||||
export type OptimizeResponse = {
|
||||
findings: string[];
|
||||
phrased: string;
|
||||
suggestedQuery?: string;
|
||||
};
|
||||
|
||||
export function aiOptimize(query: string, language: string): Promise<OptimizeResponse> {
|
||||
return request('/ai/optimize', { method: 'POST', body: JSON.stringify({ query, language }) });
|
||||
}
|
||||
|
||||
// ---- Natural language translation (Phase 7 Track B) ----
|
||||
// Same non-negotiable split as every other AI operation: this returns a
|
||||
// query for review, never executes it. Running the result is the exact
|
||||
// same runQuery() above, reused unchanged -- task 9's explicit
|
||||
// requirement.
|
||||
export type TranslateResponse = {
|
||||
query: string;
|
||||
confidence: Confidence | '';
|
||||
lowConfidenceReason?: string;
|
||||
compiles: boolean;
|
||||
compileError?: string;
|
||||
blocked: boolean;
|
||||
costWarnings?: string[];
|
||||
};
|
||||
|
||||
export function aiTranslate(nlQuery: string): Promise<TranslateResponse> {
|
||||
return request('/ai/translate', { method: 'POST', body: JSON.stringify({ nlQuery }) });
|
||||
}
|
||||
|
||||
// ---- Interaction audit logging (task 12) ----
|
||||
// Fire-and-forget: a failure here (including AI being unconfigured
|
||||
// server-side, same 404-is-normal posture as every other /ai/* call)
|
||||
// must never block or surface an error for the accept/dismiss action
|
||||
// that triggered it, so callers should not await rejection handling
|
||||
// beyond a swallowed .catch(() => {}).
|
||||
export type InteractionOperation = 'translate' | 'fix' | 'optimize';
|
||||
|
||||
export function logInteraction(entry: {
|
||||
operation: InteractionOperation;
|
||||
input: string;
|
||||
output: string;
|
||||
confidence?: Confidence | '';
|
||||
accepted: boolean;
|
||||
edited: boolean;
|
||||
finalQuery?: string;
|
||||
}): Promise<void> {
|
||||
return request('/ai/log-interaction', {
|
||||
method: 'POST',
|
||||
body: JSON.stringify({
|
||||
operation: entry.operation,
|
||||
input: entry.input,
|
||||
output: entry.output,
|
||||
confidence: entry.confidence ?? '',
|
||||
accepted: entry.accepted,
|
||||
edited: entry.edited,
|
||||
finalQuery: entry.finalQuery ?? ''
|
||||
})
|
||||
});
|
||||
}
|
||||
|
||||
export function listDashboards(): Promise<Dashboard[]> {
|
||||
return request('/dashboards').then((d) => (d as Dashboard[]) ?? []);
|
||||
}
|
||||
@@ -355,3 +472,4 @@ export function createNotificationTarget(input: {
|
||||
}): Promise<NotificationTarget> {
|
||||
return alertingRequest('/targets', { method: 'POST', body: JSON.stringify(input) });
|
||||
}
|
||||
|
||||
|
||||
@@ -7,18 +7,161 @@
|
||||
// well-established dependency for exactly this job (the same
|
||||
// "reach for a real editor primitive" reasoning ECharts and GridStack
|
||||
// already followed elsewhere in Phase 5).
|
||||
import { EditorView, keymap, placeholder as placeholderExt } from '@codemirror/view';
|
||||
import { untrack } from 'svelte';
|
||||
import {
|
||||
EditorView,
|
||||
keymap,
|
||||
placeholder as placeholderExt,
|
||||
Decoration,
|
||||
WidgetType,
|
||||
type DecorationSet
|
||||
} from '@codemirror/view';
|
||||
import { StateField, StateEffect } from '@codemirror/state';
|
||||
import { autocompletion, closeBrackets, completionKeymap } from '@codemirror/autocomplete';
|
||||
import { defaultKeymap, history, historyKeymap } from '@codemirror/commands';
|
||||
import { pipeLanguage, pipeSyntaxHighlighting } from './language';
|
||||
import { pipeCompletions } from './completions';
|
||||
import { aiComplete } from '$lib/api';
|
||||
|
||||
let {
|
||||
value = $bindable(''),
|
||||
onRun,
|
||||
placeholder = '',
|
||||
ariaLabel = 'Query'
|
||||
}: { value?: string; onRun?: () => void; placeholder?: string; ariaLabel?: string } = $props();
|
||||
ariaLabel = 'Query',
|
||||
language = 'spl',
|
||||
aiCompleteEnabled = true
|
||||
}: {
|
||||
value?: string;
|
||||
onRun?: () => void;
|
||||
placeholder?: string;
|
||||
ariaLabel?: string;
|
||||
// language, not the bindable Language type QueryBar owns -- this
|
||||
// component only needs the string to pass through to /ai/complete,
|
||||
// never interprets it itself.
|
||||
language?: string;
|
||||
// Baseline autocomplete (pipeCompletions, above) is always on --
|
||||
// this only gates the AI ghost-text layer, so a caller embedding
|
||||
// QueryEditor somewhere AI assistance doesn't make sense (if any)
|
||||
// can opt out without losing deterministic completion. Task 5's
|
||||
// "AI assistance augments, never replaces" holds either way: the
|
||||
// two mechanisms are fully independent extensions, not one
|
||||
// swapped for the other.
|
||||
aiCompleteEnabled?: boolean;
|
||||
} = $props();
|
||||
|
||||
// ---- ghost-text AI completion (task 5) ----
|
||||
// A StateField + a Decoration.widget, not @codemirror/autocomplete's
|
||||
// dropdown machinery -- ghost text renders inline after the cursor
|
||||
// and accepts on Tab, a genuinely different interaction from a
|
||||
// completion list, so it gets its own small extension rather than
|
||||
// contorting autocompletion() into rendering it. Hand-built on
|
||||
// CodeMirror's own primitives (already a Phase 5 dependency) rather
|
||||
// than pulling in a new inline-completion package -- this codebase's
|
||||
// "boring, well-understood dependencies" convention, and the
|
||||
// primitives involved (StateField, Decoration.widget) are standard,
|
||||
// commonly-used CodeMirror 6 building blocks for exactly this pattern.
|
||||
const setGhost = StateEffect.define<string | null>();
|
||||
|
||||
class GhostTextWidget extends WidgetType {
|
||||
text: string;
|
||||
constructor(text: string) {
|
||||
super();
|
||||
this.text = text;
|
||||
}
|
||||
eq(other: GhostTextWidget) {
|
||||
return other.text === this.text;
|
||||
}
|
||||
toDOM() {
|
||||
const span = document.createElement('span');
|
||||
span.className = 'cm-ghost-text';
|
||||
span.textContent = this.text;
|
||||
span.setAttribute('aria-hidden', 'true');
|
||||
return span;
|
||||
}
|
||||
}
|
||||
|
||||
// The field stores a positioned DecorationSet directly, not a bare
|
||||
// string -- computing the widget's position (tr.state.doc.length,
|
||||
// the end of the document, since ghost text is only ever set when
|
||||
// the cursor is at the end -- see scheduleCompletion) has to happen
|
||||
// here, inside update(), where the *current* document length is
|
||||
// available. An earlier version of this field stored just the
|
||||
// suggestion string and hardcoded the widget at position 0 in a
|
||||
// separate `provide` callback that only receives the field's value,
|
||||
// not the document -- a real bug (ghost text rendered at the start
|
||||
// of the query, not after what was typed), caught by live-browser
|
||||
// verification, not by type-checking, since both are type-correct
|
||||
// CodeMirror usage.
|
||||
const ghostField = StateField.define<DecorationSet>({
|
||||
create: () => Decoration.none,
|
||||
update(deco, tr) {
|
||||
for (const effect of tr.effects) {
|
||||
if (effect.is(setGhost)) {
|
||||
if (effect.value === null) return Decoration.none;
|
||||
const pos = tr.state.doc.length;
|
||||
return Decoration.set([Decoration.widget({ widget: new GhostTextWidget(effect.value), side: 1 }).range(pos)]);
|
||||
}
|
||||
}
|
||||
// Any document change or selection move that isn't the ghost
|
||||
// effect itself invalidates a stale suggestion -- showing
|
||||
// ghost text for text that no longer reflects what's actually
|
||||
// typed would be actively misleading, worse than showing
|
||||
// nothing.
|
||||
if (tr.docChanged || tr.selection) return Decoration.none;
|
||||
return deco;
|
||||
},
|
||||
provide: (field) => EditorView.decorations.from(field)
|
||||
});
|
||||
|
||||
function currentGhostText(view: EditorView): string | null {
|
||||
let found: string | null = null;
|
||||
view.state.field(ghostField).between(0, view.state.doc.length, (_from, _to, deco) => {
|
||||
if (deco.spec.widget instanceof GhostTextWidget) found = deco.spec.widget.text;
|
||||
});
|
||||
return found;
|
||||
}
|
||||
|
||||
let completeGeneration = 0;
|
||||
let completeTimer: ReturnType<typeof setTimeout> | undefined;
|
||||
const completeDebounceMs = 300;
|
||||
|
||||
function scheduleCompletion(view: EditorView) {
|
||||
if (completeTimer) clearTimeout(completeTimer);
|
||||
const gen = ++completeGeneration;
|
||||
const sel = view.state.selection.main;
|
||||
const atEnd = sel.empty && sel.head === view.state.doc.length;
|
||||
const text = view.state.doc.toString();
|
||||
if (!atEnd || text.trim() === '') return;
|
||||
|
||||
completeTimer = setTimeout(async () => {
|
||||
let result: { suggestion: string };
|
||||
try {
|
||||
result = await aiComplete(text, language);
|
||||
} catch {
|
||||
return; // provider unavailable/slow/erroring -- silently no ghost text, never a user-facing error (task 5's graceful degradation)
|
||||
}
|
||||
// Stale response guard: the user kept typing (or the
|
||||
// component unmounted) while this request was in flight.
|
||||
if (gen !== completeGeneration || !view.hasFocus) return;
|
||||
const curSel = view.state.selection.main;
|
||||
const stillAtEnd =
|
||||
curSel.empty && curSel.head === view.state.doc.length && view.state.doc.toString() === text;
|
||||
if (!stillAtEnd || !result.suggestion) return;
|
||||
view.dispatch({ effects: setGhost.of(result.suggestion) });
|
||||
}, completeDebounceMs);
|
||||
}
|
||||
|
||||
function acceptGhost(view: EditorView): boolean {
|
||||
const ghost = currentGhostText(view);
|
||||
if (!ghost) return false;
|
||||
const end = view.state.doc.length;
|
||||
view.dispatch({
|
||||
changes: { from: end, to: end, insert: ghost },
|
||||
selection: { anchor: end + ghost.length },
|
||||
effects: setGhost.of(null)
|
||||
});
|
||||
return true;
|
||||
}
|
||||
|
||||
let container: HTMLDivElement | undefined = $state();
|
||||
let view: EditorView | undefined;
|
||||
@@ -60,14 +203,38 @@
|
||||
backgroundColor: 'var(--color-accent)',
|
||||
color: 'var(--color-on-accent)'
|
||||
},
|
||||
'.cm-selectionBackground': { backgroundColor: 'color-mix(in srgb, var(--color-accent) 25%, transparent) !important' }
|
||||
'.cm-selectionBackground': { backgroundColor: 'color-mix(in srgb, var(--color-accent) 25%, transparent) !important' },
|
||||
'.cm-ghost-text': {
|
||||
color: 'var(--color-text-faint)',
|
||||
// Not user-selectable/editable -- it's a suggestion, not real
|
||||
// document content, and must never end up copy-pasted or
|
||||
// merged into a selection as if it were part of the query.
|
||||
userSelect: 'none',
|
||||
pointerEvents: 'none'
|
||||
}
|
||||
});
|
||||
|
||||
// Reactive dependency on `container` only -- deliberately not on
|
||||
// `value`, even though the view's initial doc needs it. Reading
|
||||
// `value` normally here would make this effect a dependent of it,
|
||||
// and the updateListener below writes `value` on every keystroke to
|
||||
// keep the bindable prop in sync -- if that write re-triggered this
|
||||
// effect, it would destroy and recreate the *entire* EditorView on
|
||||
// every keystroke. That's not just wasteful: a real, confirmed bug
|
||||
// found while building task 5's ghost-text completion --
|
||||
// `scheduleCompletion`'s debounce timer lives on `view` and gets
|
||||
// silently cancelled by this effect's own cleanup
|
||||
// (`clearTimeout(completeTimer)`) moments after being set, because
|
||||
// the effect re-runs right after the triggering keystroke. Wrapping
|
||||
// the initial `value` read in `untrack` breaks that dependency --
|
||||
// the effect now only re-runs when `container` itself changes
|
||||
// (mount), matching what it actually needs to do.
|
||||
$effect(() => {
|
||||
if (!container) return;
|
||||
lastEmitted = value;
|
||||
const initialValue = untrack(() => value);
|
||||
lastEmitted = initialValue;
|
||||
view = new EditorView({
|
||||
doc: value,
|
||||
doc: initialValue,
|
||||
parent: container,
|
||||
extensions: [
|
||||
pipeLanguage,
|
||||
@@ -78,7 +245,17 @@
|
||||
autocompletion({ override: [pipeCompletions] }),
|
||||
placeholderExt(placeholder),
|
||||
EditorView.contentAttributes.of({ 'aria-label': ariaLabel }),
|
||||
ghostField,
|
||||
keymap.of([
|
||||
// Tab accepts ghost text when one is showing -- checked
|
||||
// first, ahead of completionKeymap/defaultKeymap, so it
|
||||
// never competes with the deterministic dropdown's own
|
||||
// Tab/Enter handling for which one wins; the two
|
||||
// mechanisms are mutually exclusive at any given moment
|
||||
// in practice (a dropdown being open is itself a
|
||||
// docChanged-adjacent state ghost text's own
|
||||
// invalidation logic tends to have already cleared).
|
||||
{ key: 'Tab', run: acceptGhost },
|
||||
...completionKeymap,
|
||||
...defaultKeymap,
|
||||
...historyKeymap,
|
||||
@@ -95,11 +272,15 @@
|
||||
if (update.docChanged) {
|
||||
lastEmitted = update.state.doc.toString();
|
||||
value = lastEmitted;
|
||||
if (aiCompleteEnabled) scheduleCompletion(update.view);
|
||||
}
|
||||
})
|
||||
]
|
||||
});
|
||||
return () => view?.destroy();
|
||||
return () => {
|
||||
if (completeTimer) clearTimeout(completeTimer);
|
||||
view?.destroy();
|
||||
};
|
||||
});
|
||||
|
||||
// External changes to `value` (e.g. clicking a history entry) need to
|
||||
|
||||
@@ -23,6 +23,7 @@
|
||||
let columns = $state<string[]>([]);
|
||||
let rows = $state<unknown[][]>([]);
|
||||
let error = $state('');
|
||||
let warnings = $state<string[]>([]);
|
||||
let loading = $state(false);
|
||||
let hasRun = $state(false);
|
||||
let history = $state<HistoryEntry[]>(loadHistory());
|
||||
@@ -55,10 +56,12 @@
|
||||
async function runQuery() {
|
||||
loading = true;
|
||||
error = '';
|
||||
warnings = [];
|
||||
try {
|
||||
const result = await apiRunQuery(query, language);
|
||||
columns = result.columns ?? [];
|
||||
rows = result.rows ?? [];
|
||||
warnings = result.warnings ?? [];
|
||||
saveHistory({ query, language, at: Date.now() });
|
||||
} catch (e) {
|
||||
error = e instanceof Error ? e.message : String(e);
|
||||
@@ -98,7 +101,7 @@
|
||||
sheet below. Build reusable queries into a <a href="/dashboards">dashboard</a>.
|
||||
</p>
|
||||
|
||||
<QueryBar bind:query bind:language onRun={runQuery} {loading} />
|
||||
<QueryBar bind:query bind:language onRun={runQuery} {loading} errorMessage={error} {warnings} />
|
||||
|
||||
{#if error}
|
||||
<p class="error">Error: {error}</p>
|
||||
|
||||
Reference in New Issue
Block a user