# Run the Impact MCP search benchmark

- Instructions: https://radar.researchhub.foundation/benchmark/benchmark-instructions.md
- Manifest: https://radar.researchhub.foundation/benchmark/benchmark-manifest.v1.json
- Result schema: https://radar.researchhub.foundation/benchmark/result-schema.v1.json
- Submission contract: https://radar.researchhub.foundation/benchmark/benchmark-submissions.md
- Examples: https://radar.researchhub.foundation/benchmark/examples/
- Public MCP: https://mcp.researchhub.foundation/mcp

Read the instructions, manifest, result schema, and submission contract before
running.

Use a real MCP client or Streamable HTTP JSON-RPC client. Do not fetch `/mcp`
with a document/web fetcher: a plain GET returns `400` because it is not a web
page. For a manual HTTP client:

1. `POST https://mcp.researchhub.foundation/mcp` with `Content-Type:
   application/json`, `Accept: application/json, text/event-stream`, and an
   `initialize` JSON-RPC request using protocol version `2025-03-26`.
2. Save the `Mcp-Session-Id` response header.
3. Send `notifications/initialized`, then call tools with JSON-RPC
   `tools/call` requests and that session header. No `Authorization` header is
   required.

The required Radar calls are `get_radar_snapshot_catalog`,
`search_trending_science`, and `get_trending_science_signal`. The same session
can be used for benchmark upload tools.

Initialize payload:

```json
{"jsonrpc":"2.0","id":1,"method":"initialize","params":{"protocolVersion":"2025-03-26","capabilities":{},"clientInfo":{"name":"your-harness","version":"your-version"}}}
```

Execute all 30 frozen prompts, in all six use cases, with **three independent
repeats per lane** (150 runs total):

1. `direct_search`: use your normal web-search capability. Do not call Impact
   MCP and do not use an MCP result from another run.
2. `impact_mcp_search`: use only the public MCP at
   `https://mcp.researchhub.foundation/mcp`. Follow this sequence:
   `get_radar_snapshot_catalog` -> `search_trending_science` ->
   `get_trending_science_signal`. Do not add direct web search.

Give both lanes the identical prompt and, where supported, identical settings
and time window. Use the model and harness available to you. Record the actual
harness, model, version, settings, and tool capabilities.

For every run, emit one JSON object validating against `result-schema.v1.json`.
Record the exact prompt, lane, repeat index, harness/model metadata, timestamps,
raw output, normalized evidence, snapshot/catalog version when applicable, and
errors. If a capability is unavailable, use `status: "unavailable"`; never
invent output, evidence, IDs, dates, or scores.

Do not score or judge the benchmark. Return raw records and evidence; the
operator will run the separate JEV judging pass.

Before running, call `begin_benchmark_run` on the public MCP with your actual
harness/model metadata and `expected_records` (150 for three repeats). Preserve
the returned run ID and upload token securely in memory.

Do not convert unavailable runs into scores.

Also retain these artifacts in durable storage:

- `results.jsonl` — one validated result object per run;
- `run-manifest.json` — execution ID, model/settings, timestamps, public package
  version/checksums, and comparison limitations.

After each batch is written, call `submit_benchmark_batch` with the run ID,
upload token, zero-based batch index, and raw result objects. Retry only with
the same unchanged batch. After all 150 records are accepted, call
`finalize_benchmark_run` and return the run ID and receipts to the operator.

## Optional completion feedback

Do not send raw outputs or the dataset through `submit_feedback`; it is not a
result transport. Return the full artifacts to the operator. A feedback message
may only announce completion.

Use the MCP `submit_feedback` tool once, with one allowed type. For a benchmark
coverage observation use `data_quality_issue`; for a missing field or topic use
`data_request`; for a workflow problem use `workflow_improvement`.

Example completion notification (replace values with measured values):

```json
{
  "feedback_type": "general_feedback",
  "feedback": "Benchmark impact-mcp-search-comparison v1.1.0 execution <id> completed with <n> run records. Harness/model metadata and raw results are available to the benchmark operator at <durable artifact reference>. No scores were submitted."
}
```

If `submit_feedback` is unavailable, do not retry repeatedly. Record the
feedback error in `run-manifest.json`; the benchmark remains valid without a
feedback submission.