Naman Gupta
Retrieval reliability, measured
I build and evaluate RAG retrieval for small teams — chunking, reranking, and the eval harnesses that prove whether either actually worked.
Retrieval Quality Baseline Sprint— $950, 5 business days01The finding
57%
of lawyer-annotated clauses severed across chunk boundaries by fixed-width chunking
99.1%
of them kept intact by clause-aware chunking
Fixed-width chunking severs 57% of lawyer-annotated clauses across chunk boundaries. Clause-aware chunking keeps 99.1% of them intact. Neither number requires running a retriever — it’s a property of the chunker, measurable before a single query executes.
| strategy | gold spans intact in one chunk | relevant chunks per query | ceiling on precision@5 |
|---|---|---|---|
| naive (fixed 1000-char) | 43.0% | 2.70 | 0.504 |
| clause-aware | 99.1% | 1.58 | 0.312 |
02Retrieval results
20 commercial contracts from CUAD, 50 queries, 21 clause types, 114 lawyer-annotated gold spans. Embeddings via all-MiniLM-L6-v2, Qdrant, optional cross-encoder rerank. Runs on CPU, no API keys.
| config | chunks | median chars | precision@5 | recall@10 | MRR | span coverage@5 |
|---|---|---|---|---|---|---|
| naive | 1103 | 1000 | 0.276 | 0.751 | 0.607 | 0.705 |
| clause | 972 | 797 | 0.208 | 0.800 | 0.641 | 0.735 |
| clause + rerank | 972 | 797 | 0.204 | 0.827 | 0.617 | 0.709 |
03The precision@5 caveat
Why clause-aware "loses" on precision@5
Precision@5 is capped by how many chunks are relevant at all, and that count is a function of chunk size. Fixed-width chunking cuts one clause into three pieces, so three chunks count as relevant instead of one — inflating its own ceiling without retrieving one extra character of the answer. Measured against their own ceilings, clause-aware attains 66.7% and naive 54.8%. Span coverage, which counts gold characters rather than chunks and so can’t be gamed by chunk size, favours clause-aware outright: 0.735 vs 0.705.
The cross-encoder reranker is a split decision — recall@10 improves from 0.800 to 0.827, but MRR drops from 0.641 to 0.617. ms-marco-MiniLM is trained on web passage ranking, and legal clause language is outside that distribution. Reported as measured.
04Walkthrough
Full technical walkthrough (5 min) — fresh run, uncut.
namanxdev/rag-eval-harness05Production context
This is the method I used rebuilding retrieval on a production document-AI system — clause-aware chunking and cross-encoder reranking against an eval set where a domain expert labeled the correct source passage for every query. That system is under NDA, so the harness above reproduces the method on public documents instead.
Work with me
Retrieval Quality Baseline Sprint
$950 fixed·5 business days
In five business days, turn “our RAG misses obvious answers” into a reproducible golden set, baseline retrieval scores, and a ranked fix plan — without rebuilding your stack.
You get
- 40 labeled queries with relevance judgments, approved by you
- Baseline Precision@k, Recall@k, MRR and citation grounding on your current configuration
- Failure taxonomy: the top three recurring patterns, each linked to real examples
- Prioritized fix backlog across chunking, metadata filtering, hybrid retrieval and reranking
- The runnable harness, setup notes, a 30-minute readout, and one consolidated revision
Scope caps
One existing retrieval pipeline, up to 250 documents or 2 GB you supply, one current configuration.
Not included
Production implementation, model fine-tuning, UI work, ongoing monitoring.