# Impact MCP Search Benchmark

Version: 1.1.0
Benchmark ID: `impact-mcp-search-comparison`

This package compares two isolated ways for the same language model to answer
the same search prompt:

1. **Direct Search** uses the harness's normal web-search capability without
   Impact MCP.
2. **Impact MCP Search** uses only the public Impact MCP endpoint and its
   typed tools without direct web search or frontend/page access.

The benchmark measures retrieval and answer quality. It does not establish
scientific validity, medical validity, policy impact, funding suitability, or
research priority.

Scope restriction: every frozen prompt is biomedical or life-science focused
(including medicine, biotechnology, therapeutics, clinical research, public
health, and biomedical research infrastructure). Do not substitute climate,
environmental, energy, or general science topics for benchmark prompts.

## Files

- `benchmark-manifest.v1.json` contains the 30 frozen prompts, six use cases,
  lane boundaries, three-repeat run contract, and required metadata.
- `judge-rubric.v1.json` defines the five 0–4 metrics, absolute scoring JSON,
  and blinded pairwise JSON.
- `result-schema.v1.json` defines the portable result record.
- `harness-execution-prompt.md` is a copy-paste execution and feedback handoff
  for another harness.
- `benchmark-submissions.md` defines the public MCP upload flow and the
  operator-only retrieval endpoint for raw run records.
- `examples/` contains redacted records showing the expected shapes.

Use the JSON files as the machine-readable contract. Do not infer missing
fields or create a different prompt set.

## Run rules

For every prompt, run each lane three independent times. Give both lanes the
exact same prompt, model, model version, system instruction version, settings,
output budget, and time window whenever the harness supports those controls.

The Direct Search lane:

- may use the harness's normal web search;
- must not call Impact MCP;
- must not use an Impact MCP result cached from another run;
- records each material source URL, title, retrieval time, and available dates.

The Impact MCP Search lane:

- uses only `https://mcp.researchhub.foundation/mcp`;
- must not silently add direct web search or unrelated external sources;
- starts with the relevant Impact MCP catalog tool when investigating current public data;
- uses `search_trending_science` for candidate discovery;
- uses `get_trending_science_signal` for every material signal;
- records the snapshot version, stable `trend_id`, source URLs, and retrieval
  time.

If a lane or capability is unavailable, return `status: "unavailable"` and a
safe reason. Never invent results, URLs, dates, IDs, snapshot versions, or
scores to fill a gap.

## Required run metadata

Every result must record:

- prompt ID and exact prompt text;
- lane and repeat index (`1`, `2`, or `3`);
- harness name and version;
- model and model version;
- system instruction version;
- temperature and maximum output tokens, or `null` when unsupported;
- start and completion timestamps with timezone;
- output status and raw output;
- normalized evidence records;
- errors when the run did not complete.

Do not include API keys, credentials, private data, raw monitoring captures,
internal metadata, database values, or infrastructure secrets.

## Output guidance

Return a concise answer for the user and up to five result items when the
prompt requests candidates. Every material item should include the evidence
needed by the lane: a URL for Direct Search, or a stable signal ID and source
URL for Impact MCP Search. Distinguish publication time, observation time, and
snapshot generation time. Do not call a result scientifically true merely
because it appears in search.

For the `top-50-discovery` use case, return a ranked long list with one
record per distinct item and the requested evidence fields. “Top 50” is a
maximum target, not permission to pad the output: return fewer items when the
lane cannot substantiate 50, and never invent, duplicate, or silently merge
entries. Explain the ranking basis and distinguish directly evidenced items
from weaker matches. The direct lane must use only its normal search
capability; the MCP lane must use only typed tools exposed by
`https://mcp.researchhub.foundation/mcp`. Do not fetch, scrape, or read the
Radar frontend or any other webpage. A tool name containing `radar` still
means an MCP protocol call, not a Radar-page request.

Top-50 fairness rule: both lanes receive the exact same biomedical prompt and
target. URLs, MCP stable IDs, snapshot versions, publication dates, and
observation dates are evidence formats, not different questions.

Write one result JSON record per prompt/lane/repeat using
`result-schema.v1.json`. Preserve raw output separately from normalized
evidence so another judge can re-evaluate the run.

## LLM judge procedure

The runner and judge may be separate systems. A remote harness without JEV may
execute the two lanes and return raw result records only; the benchmark
operator can perform the JEV judging pass afterward. The runner must not invent
scores or treat `submit_feedback` as a result datastore.

Use a separate pinned judge model and the rubric in `judge-rubric.v1.json`.
The judge receives the prompt, output, evidence, and retrieval timestamp, but
not the lane label during absolute scoring. It returns strict JSON with a
0–4 score and evidence-based rationale for:

- relevance;
- overall search quality;
- recency;
- traceability;
- actionability.

Do not use a manually authored answer key. Do not turn the holistic overall
quality score into a mathematical average of the other scores.

For pairwise scoring, anonymize the two outputs as A and B, randomize their
order, and ask the judge for a preference per metric and overall. Permit
`tie` and `indeterminate`; never force a winner when evidence is insufficient.

Aggregate means, medians, repeat spread, lane win rates, ties, and per-use-case
results only after preserving every individual run and judge record.

## Reproducibility and limitations

Direct web results change over time. Impact MCP results are tied to a published
snapshot that also changes. Always record timestamps and the MCP catalog or
snapshot version. Differences in model family, tool availability, search
limits, or output settings must be reported as comparison limitations.

This benchmark is portable: a harness may implement the runner in any
language, but it must follow the lane boundaries and emit the documented JSON
shape. ZIP archives and harness-specific adapters are intentionally outside
this first package version.