Benchmark retrieval quality¶

What you'll build¶

In this tutorial, you turn six labeled retrieval questions into a repeatable BM25 quality report. You calculate Hit@k, Recall@k, and mean reciprocal rank (MRR), then use the report to identify questions that lexical retrieval misses.

Time: About 30 minutes.

Before you begin¶

You need:

Python 3.12 or later.
uv.
A local clone with dependencies installed.
Implement Okapi BM25 from scratch.

You do not need an API key. The benchmark uses local chunks and deterministic labels from eval/rag_cases.jsonl.

See it work¶

Run the BM25 benchmark from the repository root:

uv run python -m eval.rag_retrieval

Corpus: 267 chunks | Cases: 6 | k=5
Mode      Cases    Hit@k   Recall@k      MRR
bm25          6    0.500      0.417    0.333
bm25 misses: conversation-persistence, citation-deduplication, system-prompt-file

The run searches 267 chunks for

6 labeled questions. The three metrics describe different parts of the resulting rankings.

How retrieval metrics work¶

Each record in eval/rag_cases.jsonl contains a question and one or more chunk IDs that count as relevant. The benchmark searches for the question, keeps the first k results, and compares those result IDs with the labels.

Metric	Question it answers	BM25 at k=5
Hit@k	Did at least one relevant chunk appear?	0.5
Recall@k	What fraction of accepted relevant chunks appeared?	0.417
MRR	How early did the first relevant chunk appear?	0.333

Hit@k is binary for each case. Recall@k matters when a question has multiple accepted chunks. Reciprocal rank is one divided by the position of the first relevant result; a miss contributes zero. MRR is the mean across all cases.

Load labeled cases¶

In eval/rag_retrieval.py, inspect RetrievalCase:
```
@dataclass(frozen=True)
class RetrievalCase:
    id: str
    query: str
    relevant_chunk_ids: frozenset[str]
```
A set of accepted IDs lets one question recognize multiple source chunks as relevant without requiring every chunk to appear.
Inspect load_cases() and validate_case_labels().

The loader rejects duplicate case IDs and empty label sets. Label validation rejects chunk IDs that no longer exist after ingestion changes, preventing a stale benchmark from silently reporting false misses.

Score one ranking¶

In score_ranking(), collect the one-based positions of relevant results:

relevant_ranks = [
    rank
    for rank, chunk_id in enumerate(ranked_chunk_ids, start=1)
    if chunk_id in case.relevant_chunk_ids
]

Calculate whether the ranking contains a hit:
```
hit = bool(relevant_ranks)
```

Divide retrieved relevant chunks by all accepted relevant chunks:

recall = len(relevant_ranks) / len(case.relevant_chunk_ids)

Invert the first relevant rank, or return zero for a miss:

reciprocal_rank = 1 / relevant_ranks[0] if relevant_ranks else 0.0

The repository implementation also ignores duplicate result IDs before scoring, so one repeated chunk cannot inflate recall.

Aggregate the cases¶

Run every labeled query through the same retrieval mode and cutoff.

Average each metric in summarize():

hit_rate = sum(metric.hit for metric in case_metrics) / case_count
recall_at_k = sum(metric.recall for metric in case_metrics) / case_count
mean_reciprocal_rank = (
    sum(metric.reciprocal_rank for metric in case_metrics) / case_count
)

Keep the missed case IDs beside the aggregate values.

Aggregate metrics support comparisons, while named misses identify the questions and code paths that need investigation.

Verify your work¶

Run the focused metric tests:

uv run pytest tests/test_rag_retrieval_eval.py -q

6 passed

Check the benchmark and its tests with Ruff:

uv run ruff check eval/rag_retrieval.py tests/test_rag_retrieval_eval.py

Break it on purpose¶

Reduce the cutoff from five results to one:

uv run python -m eval.rag_retrieval --top-k 1

Corpus: 267 chunks | Cases: 6 | k=1
bm25          6    0.167      0.167    0.167
bm25 misses: conversation-persistence, citation-deduplication, system-prompt-file, untrusted-rag-context, hybrid-rank-fusion

Hit@1 falls to

0.167 because only one case puts a relevant chunk first. The experiment shows why every retrieval metric must name its cutoff: Hit@1 and Hit@5 describe different user experiences.

Troubleshooting¶

Symptom	Resolution	Source
`unknown relevant chunk ids`	Update stale labels only after confirming the intended replacement chunk.	Observed
Metrics differ from this guide	Run from the repository root and confirm the committed fact ledger is current.	Observed
Vector or hybrid mode reports missing indexes	Build indexes and provide a Voyage key, or use the offline BM25 default.	Predicted
One case unexpectedly has full recall	Check whether it has one accepted relevant chunk; one hit then equals full recall.	Predicted

Next steps¶

Continue with Test model output with deterministic rules to apply the same repeatable-evaluation mindset to generated responses. Keyed retrieval comparisons belong in a protected environment because they use Voyage embeddings and can incur cost.

For metric definitions and evaluation context, read the Information Retrieval evaluation documentation.

Tested against¶

Item	Value
Python	3.13.14; the repository requires 3.12 or later
pytest	9.1.1
Corpus	267 chunks
Cases	6
Verified	2026-08-17