Phase 7: AI-assisted query authoring (autocomplete, explain, fix, optimize, NL translation)

Adds a self-hosted (Ollama, qwen2.5-coder) model provider abstraction
with a pluggable opt-in cloud adapter, schema grounding, and a shared
cost/safety guard every AI-suggested query is assessed against --
compiling to and executing through the same unchanged Phase 2 IR/
compiler and Phase 4 tenant scoping as a hand-written query, no
parallel execution path.

Track A (built into the query bar): inline ghost-text autocomplete,
"Explain this query", "Fix this query" with a diff view, and a
rule-based "Optimize" suggestion. Track B: natural-language-to-query
translation, always a separate review step from execution, with
`sentryctl query --nl` requiring explicit confirmation to run.
Every accepted/dismissed translate-fix-optimize interaction is logged
into the same append-only audit_log table Phase 4 built.

Two real product bugs were found and fixed via live browser
verification (a Svelte effect re-running on every keystroke that
silently cancelled the ghost-text debounce; a ghost-text widget
positioned at document offset 0 instead of the cursor), and a real
costguard logic bug (unbounded-aggregation vs. raw-row) was caught by
its own test suite. New integration tests wire a real Ollama client
through the real HTTP handler against a mock server matching Ollama's
wire contract (hack/mock-ollama), keeping model-quality verification
out of CI as a disclosed, periodic human-run check instead.

See /docs/phase-7-ai-design.md and /docs/phase-7-runbook.md.
This commit is contained in:
2026-08-16 18:06:27 -07:00
parent 661568085e
commit 7d316f92db
37 changed files with 5230 additions and 20 deletions
+217
View File
@@ -0,0 +1,217 @@
// Package ollama implements provider.Provider against a local Ollama
// server -- the default, self-hosted primary provider (Phase 7 task 2;
// see /docs/phase-7-ai-design.md for why Ollama over vLLM and why
// qwen2.5-coder is the recommended model). Same thin-HTTP-client shape
// as alerting/internal/queryclient -- net/http + encoding/json, no new
// HTTP client dependency, matching this codebase's "boring,
// well-understood dependencies" convention.
package ollama
import (
"bytes"
"context"
"encoding/json"
"fmt"
"net/http"
"strings"
"github.com/sentry/sentry/api/ai/provider"
)
// Client implements provider.Provider against one Ollama server and one
// model. The per-operation routing layer (task 2's "per-operation
// provider/model configuration") constructs one Client per distinct
// model a deployment configures -- e.g. one for qwen2.5-coder:1.5b
// (Complete's fast path) and one for qwen2.5-coder:7b (everything else)
// -- rather than this package knowing anything about operation-to-model
// routing itself.
type Client struct {
baseURL string
model string
http *http.Client
}
// New builds a Client. baseURL defaults to Ollama's standard local
// address if empty -- the common case for the primary, self-hosted
// deployment target.
func New(baseURL, model string) *Client {
if baseURL == "" {
baseURL = "http://localhost:11434"
}
return &Client{baseURL: strings.TrimSuffix(baseURL, "/"), model: model, http: &http.Client{}}
}
type chatMessage struct {
Role string `json:"role"`
Content string `json:"content"`
}
type chatRequest struct {
Model string `json:"model"`
Messages []chatMessage `json:"messages"`
Stream bool `json:"stream"`
Format string `json:"format,omitempty"`
}
type chatResponse struct {
Message chatMessage `json:"message"`
}
// chat calls Ollama's POST /api/chat, non-streaming, and returns the
// assistant message content. jsonMode requests Ollama's JSON-constrained
// output format -- used by every operation except Explain, which just
// wants prose back.
func (c *Client) chat(ctx context.Context, system, user string, jsonMode bool) (string, error) {
req := chatRequest{
Model: c.model,
Messages: []chatMessage{
{Role: "system", Content: system},
{Role: "user", Content: user},
},
Stream: false,
}
if jsonMode {
req.Format = "json"
}
body, err := json.Marshal(req)
if err != nil {
return "", fmt.Errorf("ollama: encoding request: %w", err)
}
httpReq, err := http.NewRequestWithContext(ctx, http.MethodPost, c.baseURL+"/api/chat", bytes.NewReader(body))
if err != nil {
return "", fmt.Errorf("ollama: building request: %w", err)
}
httpReq.Header.Set("Content-Type", "application/json")
resp, err := c.http.Do(httpReq)
if err != nil {
return "", fmt.Errorf("ollama: calling %s: %w", c.baseURL, err)
}
defer resp.Body.Close()
if resp.StatusCode != http.StatusOK {
var errBody struct {
Error string `json:"error"`
}
_ = json.NewDecoder(resp.Body).Decode(&errBody)
if errBody.Error != "" {
return "", fmt.Errorf("ollama: request failed (%d): %s", resp.StatusCode, errBody.Error)
}
return "", fmt.Errorf("ollama: request failed with status %d", resp.StatusCode)
}
var chatResp chatResponse
if err := json.NewDecoder(resp.Body).Decode(&chatResp); err != nil {
return "", fmt.Errorf("ollama: decoding response: %w", err)
}
return chatResp.Message.Content, nil
}
// stripCodeFence handles the common small-model habit of wrapping JSON
// output in ```json ... ``` even when explicitly told not to -- a
// best-effort cleanup, not a guarantee; a model that returns genuinely
// malformed JSON still surfaces as a real decode error to the caller,
// which is the correct behavior (better an explicit error than silently
// fabricating a result).
func stripCodeFence(s string) string {
s = strings.TrimSpace(s)
if !strings.HasPrefix(s, "```") {
return s
}
s = strings.TrimPrefix(s, "```json")
s = strings.TrimPrefix(s, "```")
s = strings.TrimSuffix(s, "```")
return strings.TrimSpace(s)
}
func parseConfidence(s string) provider.Confidence {
switch strings.ToLower(strings.TrimSpace(s)) {
case "high":
return provider.ConfidenceHigh
case "medium":
return provider.ConfidenceMedium
default:
// Unrecognized or missing confidence fails toward caution, not
// toward assumed correctness -- an empty/garbled confidence
// field from the model is itself a signal something's off.
return provider.ConfidenceLow
}
}
func (c *Client) Translate(ctx context.Context, req provider.TranslateRequest) (provider.TranslateResult, error) {
raw, err := c.chat(ctx, translateSystemPrompt(req.Schema), req.NLQuery, true)
if err != nil {
return provider.TranslateResult{}, err
}
var parsed struct {
Query string `json:"query"`
Confidence string `json:"confidence"`
Reason string `json:"reason"`
}
if err := json.Unmarshal([]byte(stripCodeFence(raw)), &parsed); err != nil {
return provider.TranslateResult{}, fmt.Errorf("ollama: parsing translate response: %w", err)
}
return provider.TranslateResult{
Query: parsed.Query,
Confidence: parseConfidence(parsed.Confidence),
LowConfidenceReason: parsed.Reason,
}, nil
}
func (c *Client) Complete(ctx context.Context, req provider.CompleteRequest) (provider.CompleteResult, error) {
raw, err := c.chat(ctx, completeSystemPrompt(req.Schema), req.QueryPrefix, true)
if err != nil {
return provider.CompleteResult{}, err
}
var parsed struct {
Suggestion string `json:"suggestion"`
}
if err := json.Unmarshal([]byte(stripCodeFence(raw)), &parsed); err != nil {
return provider.CompleteResult{}, fmt.Errorf("ollama: parsing complete response: %w", err)
}
return provider.CompleteResult{Suggestion: parsed.Suggestion}, nil
}
func (c *Client) Explain(ctx context.Context, req provider.ExplainRequest) (provider.ExplainResult, error) {
user := req.Query
switch {
case len(req.RuleFindings) > 0:
user = fmt.Sprintf("Query: %s\nFindings: %s", req.Query, strings.Join(req.RuleFindings, "; "))
case req.OriginalIntent != "":
user = fmt.Sprintf("Original request: %q\nGenerated query: %s", req.OriginalIntent, req.Query)
}
raw, err := c.chat(ctx, explainSystemPrompt(req.OriginalIntent != "", len(req.RuleFindings) > 0), user, false)
if err != nil {
return provider.ExplainResult{}, err
}
return provider.ExplainResult{Explanation: strings.TrimSpace(raw)}, nil
}
func (c *Client) Fix(ctx context.Context, req provider.FixRequest) (provider.FixResult, error) {
errText := req.ParseError
if errText == "" {
errText = req.ExecutionError
}
user := fmt.Sprintf("Query: %s\nError: %s", req.Query, errText)
raw, err := c.chat(ctx, fixSystemPrompt(req.Schema), user, true)
if err != nil {
return provider.FixResult{}, err
}
var parsed struct {
SuggestedQuery string `json:"suggested_query"`
Explanation string `json:"explanation"`
Confidence string `json:"confidence"`
}
if err := json.Unmarshal([]byte(stripCodeFence(raw)), &parsed); err != nil {
return provider.FixResult{}, fmt.Errorf("ollama: parsing fix response: %w", err)
}
return provider.FixResult{
SuggestedQuery: parsed.SuggestedQuery,
Explanation: parsed.Explanation,
Confidence: parseConfidence(parsed.Confidence),
}, nil
}
var _ provider.Provider = (*Client)(nil)
+152
View File
@@ -0,0 +1,152 @@
package ollama
import (
"context"
"encoding/json"
"net/http"
"net/http/httptest"
"strings"
"testing"
"github.com/sentry/sentry/api/ai/provider"
)
// fakeOllamaServer stands in for a real Ollama server, returning the
// given assistant-message content verbatim -- same reasoning
// queryclient's tests use httptest against a fake api instead of a real
// one: this package's own logic (request shape, response parsing,
// JSON-mode contract) is what's under test, not Ollama itself.
func fakeOllamaServer(t *testing.T, content string) *httptest.Server {
t.Helper()
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if r.URL.Path != "/api/chat" {
t.Errorf("unexpected path %s", r.URL.Path)
}
var req chatRequest
if err := json.NewDecoder(r.Body).Decode(&req); err != nil {
t.Fatalf("decoding request: %v", err)
}
if len(req.Messages) != 2 || req.Messages[0].Role != "system" || req.Messages[1].Role != "user" {
t.Errorf("unexpected messages shape: %+v", req.Messages)
}
w.Header().Set("Content-Type", "application/json")
_ = json.NewEncoder(w).Encode(chatResponse{Message: chatMessage{Role: "assistant", Content: content}})
}))
t.Cleanup(srv.Close)
return srv
}
func TestTranslateParsesJSONResponse(t *testing.T) {
srv := fakeOllamaServer(t, `{"query": "earliest=-1h severity=ERROR | stats count by service", "confidence": "high", "reason": ""}`)
c := New(srv.URL, "test-model")
got, err := c.Translate(context.Background(), provider.TranslateRequest{NLQuery: "errors in the last hour by service"})
if err != nil {
t.Fatalf("Translate: %v", err)
}
if got.Confidence != provider.ConfidenceHigh {
t.Errorf("Confidence = %v, want high", got.Confidence)
}
if got.Query == "" {
t.Error("expected a non-empty query")
}
}
func TestTranslateHandlesCodeFencedJSON(t *testing.T) {
srv := fakeOllamaServer(t, "```json\n{\"query\": \"service=api\", \"confidence\": \"medium\", \"reason\": \"\"}\n```")
c := New(srv.URL, "test-model")
got, err := c.Translate(context.Background(), provider.TranslateRequest{NLQuery: "api logs"})
if err != nil {
t.Fatalf("Translate with fenced JSON: %v", err)
}
if got.Query != "service=api" {
t.Errorf("Query = %q, want %q", got.Query, "service=api")
}
}
func TestTranslateLowConfidenceCarriesReason(t *testing.T) {
srv := fakeOllamaServer(t, `{"query": "", "confidence": "low", "reason": "not sure what 'weird stuff' refers to"}`)
c := New(srv.URL, "test-model")
got, err := c.Translate(context.Background(), provider.TranslateRequest{NLQuery: "show me weird stuff"})
if err != nil {
t.Fatalf("Translate: %v", err)
}
if got.Confidence != provider.ConfidenceLow || got.LowConfidenceReason == "" {
t.Errorf("got = %+v, want low confidence with a reason", got)
}
}
func TestTranslateMalformedJSONIsAnError(t *testing.T) {
srv := fakeOllamaServer(t, "not json at all, sorry")
c := New(srv.URL, "test-model")
_, err := c.Translate(context.Background(), provider.TranslateRequest{NLQuery: "anything"})
if err == nil {
t.Fatal("expected an error for unparseable model output, got nil")
}
}
func TestCompleteReturnsSuggestionOnly(t *testing.T) {
srv := fakeOllamaServer(t, `{"suggestion": " | stats count by host"}`)
c := New(srv.URL, "test-model")
got, err := c.Complete(context.Background(), provider.CompleteRequest{QueryPrefix: "service=api", Language: "spl"})
if err != nil {
t.Fatalf("Complete: %v", err)
}
if got.Suggestion != " | stats count by host" {
t.Errorf("Suggestion = %q", got.Suggestion)
}
}
func TestExplainReturnsPlainText(t *testing.T) {
srv := fakeOllamaServer(t, "This counts events per host over the last hour.")
c := New(srv.URL, "test-model")
got, err := c.Explain(context.Background(), provider.ExplainRequest{Query: "earliest=-1h | stats count by host"})
if err != nil {
t.Fatalf("Explain: %v", err)
}
if got.Explanation == "" {
t.Error("expected a non-empty explanation")
}
}
func TestFixParsesJSONResponse(t *testing.T) {
srv := fakeOllamaServer(t, `{"suggested_query": "earliest=-1h | stats count", "explanation": "added a required time range", "confidence": "high"}`)
c := New(srv.URL, "test-model")
got, err := c.Fix(context.Background(), provider.FixRequest{Query: "stats count", ParseError: "no time range"})
if err != nil {
t.Fatalf("Fix: %v", err)
}
if got.SuggestedQuery == "" || got.Explanation == "" {
t.Errorf("got = %+v, want both fields populated", got)
}
}
func TestNonOKStatusIsAnError(t *testing.T) {
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
w.WriteHeader(http.StatusInternalServerError)
_ = json.NewEncoder(w).Encode(map[string]string{"error": "model not found"})
}))
t.Cleanup(srv.Close)
c := New(srv.URL, "missing-model")
_, err := c.Explain(context.Background(), provider.ExplainRequest{Query: "x"})
if err == nil {
t.Fatal("expected an error for a non-200 response")
}
if got := err.Error(); !strings.Contains(got, "model not found") {
t.Errorf("error = %q, want it to include the server's error message", got)
}
}
func TestDefaultBaseURL(t *testing.T) {
c := New("", "m")
if c.baseURL != "http://localhost:11434" {
t.Errorf("default baseURL = %q", c.baseURL)
}
}
+122
View File
@@ -0,0 +1,122 @@
package ollama
import (
"fmt"
"strings"
"github.com/sentry/sentry/api/ai/provider"
)
// grammarReference is a condensed version of
// /docs/query-language-reference.md -- every operation's system prompt
// includes this so the model is grounded in Sentry's actual pipe syntax,
// not whatever generic log-query DSL it may have seen in training.
// Trimmed to the parts that matter for generation/explanation (the full
// doc's prose and examples aren't needed here); kept in sync with that
// doc by hand -- if the grammar changes, this needs updating too, same
// as any other place the language is described outside its own parser.
const grammarReference = `Sentry query language (pipe syntax):
<base search> | <stage> | <stage> | ...
Base search: filter terms and/or free-text search, combined with implicit "and".
field=value, field!=value, field>value, field>=value, field<value, field<=value
bare word or "quoted phrase" -- free-text search on the message field
message:"phrase" -- explicit free-text search
earliest=-1h, latest=-5m -- relative time (s/m/h/d/w units), or earliest="2026-08-14T00:00:00Z" (RFC 3339 absolute)
"or" only works between free-text terms, never between structured filters
Pipe stages, in the order they may appear:
| where <filter terms> additional filtering, same syntax as base search filters
| stats <func>(<field>) as <alias>, ... by <field>, ...
functions: count (no field needed), sum, avg, min, max (all need a field)
| sort -field, +field, ... "-" descending (default if no sign), "+" ascending
| fields field, field, ... choose output columns
| head N first N results (default 100)
| tail N last N results, chronologically
Structured columns: timestamp, host, service, severity, message, record_id.
Anything else is looked up in per-record attributes (always text; compared
numerically when the right-hand side looks like a number).
Raw SQL (SELECT ...) is also accepted but pipe syntax is strongly preferred
for anything generated rather than hand-written -- narrower, safer surface.`
func renderSchema(s provider.SchemaContext) string {
if len(s.Services) == 0 && len(s.Fields) == 0 {
return "(no schema grounding data available yet)"
}
var sb strings.Builder
if len(s.Services) > 0 {
fmt.Fprintf(&sb, "Known services: %s\n", strings.Join(s.Services, ", "))
}
if len(s.Fields) > 0 {
sb.WriteString("Known fields:\n")
for _, f := range s.Fields {
if len(f.Examples) > 0 {
fmt.Fprintf(&sb, " - %s (examples: %s)\n", f.Name, strings.Join(f.Examples, ", "))
} else {
fmt.Fprintf(&sb, " - %s\n", f.Name)
}
}
}
return sb.String()
}
func translateSystemPrompt(schema provider.SchemaContext) string {
return fmt.Sprintf(`You translate a plain-English question into a Sentry pipe-syntax query. You never explain, never execute anything, never write raw SQL unless the pipe syntax genuinely cannot express the request.
%s
%s
Respond with ONLY a JSON object, no other text, no markdown fences:
{"query": "<the pipe-syntax query>", "confidence": "high"|"medium"|"low", "reason": "<empty unless confidence is low, in which case explain what's ambiguous or unsupported>"}
If you cannot produce a query you're reasonably confident in, set confidence to "low", leave query empty, and explain why in reason. Never guess with false confidence.`, grammarReference, renderSchema(schema))
}
func completeSystemPrompt(schema provider.SchemaContext) string {
return fmt.Sprintf(`You suggest how to continue a partially-typed Sentry query. You are given everything typed so far; respond with ONLY the suggested continuation text (what should appear after the cursor), not the text already typed, not an explanation.
%s
%s
Respond with ONLY a JSON object, no other text, no markdown fences:
{"suggestion": "<continuation text, or empty string if you have no good suggestion>"}`, grammarReference, renderSchema(schema))
}
// explainSystemPrompt covers all three contexts provider.ExplainRequest
// supports: a plain hand-written-query explanation (both empty), a
// post-translation review (hasIntent), or Optimize's "phrase these
// findings" mode (hasFindings) -- mutually exclusive in practice, see
// ExplainRequest.RuleFindings' doc comment.
func explainSystemPrompt(hasIntent, hasFindings bool) string {
if hasFindings {
return fmt.Sprintf(`A rule-based check already found one or more real issues with a Sentry query's efficiency (e.g. a missing time range). Your only job is to phrase those findings as a short, clear, actionable suggestion for the person who wrote the query -- do not invent additional issues, do not restate the query's own syntax back at them, do not hedge with "might" or "could" about something the check already confirmed. One or two sentences.
%s`, grammarReference)
}
base := fmt.Sprintf(`You explain what a Sentry query does in plain English, for someone who may not know the query language. Be concise -- two or three sentences, not a line-by-line breakdown unless the query is unusually complex.
%s`, grammarReference)
if hasIntent {
base += "\n\nYou are explaining a query that was just generated from a natural-language request. Focus on how the request became this query -- call out any interpretation choices (e.g. how a vague time phrase or field reference was resolved), not just what the query does in isolation."
}
return base
}
func fixSystemPrompt(schema provider.SchemaContext) string {
return fmt.Sprintf(`You fix a broken Sentry query given its error message. Produce a corrected query and a short explanation of what was wrong.
%s
%s
Respond with ONLY a JSON object, no other text, no markdown fences:
{"suggested_query": "<corrected query>", "explanation": "<short explanation of what was wrong and what changed>", "confidence": "high"|"medium"|"low"}
If you cannot determine a fix, set confidence to "low" and suggested_query to an empty string.`, grammarReference, renderSchema(schema))
}