feat: implement unified search_events tool with natural language queries

[dcramer]

Summary

This PR implements a unified search_events tool that replaces the separate find_errors and find_transactions tools, providing a more flexible and powerful way to search Sentry data using natural language queries. Additionally, this PR modernizes the entire evaluation system to use vitest-evals 0.4.0 with tool-call focused testing.

Key Changes

New search_events Tool

• Natural Language Processing: Uses OpenAI GPT-4o to translate user queries into Sentry API calls
• Unified Interface: Single tool handles errors, logs, traces, and spans datasets
• Smart Dataset Detection: Automatically determines the appropriate dataset based on query content
• Parallel API Calls: Efficiently fetches custom attributes and search results concurrently

Tool Consolidation

• Removes find_errors and find_transactions from tool exports
• Updates find_errors_in_file prompt to use the new search_events tool
• Streamlines the tool interface while maintaining all functionality

Evaluation System Overhaul

• Migrated to vitest-evals 0.4.0: Moved from mock-based testing to tool-call focused evaluation
• Added ToolPredictionScorer: AI-based scorer that predicts tool calls without execution for faster testing
• Natural Language Prompts: Updated all 41 evaluation tests to use realistic user queries instead of explicit tool instructions
• Improved Test Coverage: All tests now pass with ≥0.6 confidence thresholds using natural language
• Enhanced Mock Infrastructure: Better test fixtures and data generation for realistic scenarios

Dependencies

• Adds @ai-sdk/openai for natural language query processing
• Integrates with existing Sentry API client infrastructure

Examples

• "Show me Python exceptions from the last 24 hours" → errors dataset with time filter
• "Find slow API calls over 1 second" → spans dataset with duration filter
• "Search for authentication errors" → errors dataset with keyword filter

This change improves both the user experience by allowing more intuitive, natural language queries and the development experience with a modernized, more realistic evaluation system.

[codecov]

Codecov Report

Attention: Patch coverage is 84.89388% with 121 lines in your changes missing coverage. Please review.

Project coverage is 62.94%. Comparing base (a69d3a6) to head (903600d).
Report is 1 commits behind head on main.

✅ All tests successful. No failed tests found.

Files with missing lines	Patch %	Lines
packages/mcp-server/src/tools/search-events.ts	80.10%	110 Missing ⚠️
packages/mcp-server/src/prompts.ts	0.00%	5 Missing ⚠️
packages/mcp-server/src/api-client/client.ts	96.82%	4 Missing ⚠️
packages/mcp-server/src/tools/index.ts	0.00%	2 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #375      +/-   ##
==========================================
+ Coverage   62.65%   62.94%   +0.28%     
==========================================
  Files          77       78       +1     
  Lines        6879     7491     +612     
  Branches      620      666      +46     
==========================================
+ Hits         4310     4715     +405     
- Misses       2569     2776     +207     

Flag	Coverage Δ
evals	77.14% <100.00%> (?)
unittests	62.74% <83.80%> (+0.08%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[dcramer]

Turning into a large patch as I'm trying to clean up evals in addition as I had to expand them..

[Copilot]

Pull Request Overview

This PR introduces a unified search_events tool to replace find_errors and find_transactions, updates dependencies (including vitest-evals to 0.4.0), and modernizes testing to use tool-call focused evals.

• Add new search_events tool with natural language query support
• Remove old find_errors and find_transactions exports
• Upgrade to vitest-evals 0.4.0 and switch to ToolPredictionScorer

Reviewed Changes

Copilot reviewed 53 out of 54 changed files in this pull request and generated 1 comment.

File	Description
pnpm-workspace.yaml	Added date-fns and bumped vitest-evals to 0.4.0
packages/mcp-server/src/tools/search-events.ts	Implemented new search_events tool
packages/mcp-server/src/tools/search-events.test.ts	Added tests for search_events
packages/mcp-server/src/tools/index.ts	Swapped out find_errors/find_transactions for search_events

Files not reviewed (1)

• pnpm-lock.yaml: Language not supported

Comments suppressed due to low confidence (3)

packages/mcp-server/src/tools/search-events.ts:173

• The rules string contains a stray '+-' prefix. Remove the leading '+' so each item starts with a single '-' for proper list formatting.

- Use level field for severity (error, warning, info, debug)

packages/mcp-server/src/tools/search-events.ts:188

• The rules string for logs has an extra '+' before the dash. Remove the '+' so the list marker is just '-'.

- Use severity field for log levels (fatal, error, warning, info, debug, trace)

packages/mcp-server/src/tools/search-events.ts:178

• In the examples string, remove the leading '+' before the '-' so the example list renders correctly.

- "unhandled errors in production" → error.handled:false AND environment:production

[Copilot]

[nitpick] The handler function spans hundreds of lines and mixes prompt construction, API interaction, and output formatting. Consider extracting parts into smaller helper functions to improve readability and maintainability.

[cursor]

Bug: Misleading Parameter Name Causes API Confusion

The searchEvents method's projectSlug parameter is misleadingly named, as it expects a numeric project ID (as a string) rather than a project slug. While current callers correctly pass stringified project IDs, this naming mismatch creates a confusing API interface that could lead to incorrect usage and future bugs if developers attempt to pass actual project slugs.

packages/mcp-server/src/api-client/client.ts#L1369-L1424

sentry-mcp/packages/mcp-server/src/api-client/client.ts

Lines 1369 to 1424 in 903600d

	/**
	* Searches for events in Sentry using a general query.
	* This method is used by the search_events tool for semantic search.
	*/
	async searchEvents(
	{
	organizationSlug,
	query,
	fields,
	limit = 10,
	projectSlug,
	dataset = "spans",
	statsPeriod,
	sort = "-timestamp",
	}: {
	organizationSlug: string;
	query: string;
	fields: string[];
	limit?: number;
	projectSlug?: string;
	dataset?: "spans" | "errors" | "ourlogs";
	statsPeriod?: string;
	sort?: string;
	},
	opts?: RequestOptions,
	) {
	const queryParams = new URLSearchParams();
	queryParams.set("per_page", limit.toString());
	queryParams.set("query", query);
	queryParams.set("referrer", "sentry-mcp");
	queryParams.set("dataset", dataset);
	if (statsPeriod) {
	queryParams.set("statsPeriod", statsPeriod);
	}
	queryParams.set("sort", sort);
	// Add project filter if specified
	if (projectSlug) {
	queryParams.set("project", projectSlug);
	}
	// Add dataset-specific parameters
	if (dataset === "spans") {
	queryParams.set("allowAggregateConditions", "0");
	queryParams.set("useRpc", "1");
	}
	// Add each field as a separate parameter
	for (const field of fields) {
	queryParams.append("field", field);
	}
	const apiUrl = `/organizations/${organizationSlug}/events/?${queryParams.toString()}`;
	return await this.requestJSON(apiUrl, undefined, opts);
	}

Fix in Cursor • Fix in Web

---

Was this report helpful? Give feedback by reacting with 👍 or 👎