Dense vs. hybrid retrieval for financial evidence
Seven retrieval profiles over filings and earnings calls showed that better ranking alone did not guarantee better answers. A complete answer replay passed the review thresholds after repairs, with nine failures preserved as accepted limitations. The selected approach improved coverage at a measurable latency cost.
The outcome
Better coverage and ordering near the top, bought with latency.
Hybrid retrieval alone did not beat dense retrieval. The full reranker became the first finalist because it improved early ordering, but it still lost one cross-corpus answer. Preserving evidence from both source types for multi-part questions recovered that case and improved every reported ranking and coverage metric across three clean repetitions. I selected that approach for production while accepting its additional latency.
- nDCG@10
0.860
12.9% relative vs. dense
- MRR
0.837
13.3% relative vs. dense
- Recall@5
93.1%
+9.5 pp vs. dense
- Retrieval p95
758 ms
+605 ms vs. dense
Question
Which policy retrieves the evidence an answer actually needs?
Financial research retrieval has two different jobs. It must find exact language—metric names, executives, sections, and periods—and it must also survive paraphrase. Dense search is strong at semantic similarity. Lexical search is strong at exact terms. Reranking can reorder the combined pool, but it adds provider cost and latency.
I evaluated the retrieval stage independently from answer generation. That separation let me measure whether the correct retained evidence appeared in the first five or ten results without giving fluent prose credit for a miss.
Method
One frozen corpus. One adjudicated gold set. Seven arms.
A named reviewer checked every case against the complete retained source. Answerable cases bind each necessary fact to one or more acceptable evidence identities. Ten negative cases deliberately require abstention and are not misreported as retrieval successes.
Dataset
100
human-adjudicated questions across 20 tickers; 90 answerable and 10 insufficient-evidence controls.
Controls
Fixed evidence and top-k policy
Every arm used the same release, source-policy snapshot, questions, evidence identities, candidate bound of 50, and RRF constant of 60.
Repeated runs
Three per reranked arm
Dense and plain hybrid arms ran once. Each reranked arm has three clean repetitions; the table reports means across thirteen valid outputs. Two additional rate-limited coverage runs are retained separately as tainted.
Case mix
- Filing
- 20
- Earnings call
- 20
- Cross-corpus
- 15
- Exact metric
- 15
- Multi-condition
- 10
- Speaker + section
- 10
- Insufficient evidence
- 10
Metrics and what they mean
- Recall@5 / Recall@10
- Share of required evidence groups found inside the first five or ten results.
- MRR and nDCG@10
- Whether acceptable evidence appears early and whether useful ordering is preserved.
- Complete-case rate
- Share of answerable questions for which every necessary fact was present by the cutoff.
- p50 / p95 latency and cost
- Retrieval-stage runtime and attributed provider spend, not full answer latency or serving cost.
Results
The larger embedding models lost. Hybrid needed reranking to win.
Bigger was not better on this corpus. Plain hybrid retrieval matched dense Recall@10 but ranked useful evidence worse. Both rerankers reversed that ordering problem. The full model became the first finalist; the compound-question coverage path then improved both ordering and evidence-group coverage.
| Arm | R@5 | R@10 | MRR | nDCG@10 | Complete@5 | Complete@10 | p50 | p95 | Cost |
|---|---|---|---|---|---|---|---|---|---|
| voyage-4 · dense1 run | 0.836 | 0.888 | 0.739 | 0.762 | 0.811 | 0.878 | 128 ms | 153 ms | $0.000133 |
| voyage-4-large · dense1 run | 0.767 | 0.879 | 0.700 | 0.736 | 0.756 | 0.867 | 163 ms | 193 ms | $0.000266 |
| voyage-context-4 · dense1 run | 0.681 | 0.802 | 0.580 | 0.629 | 0.667 | 0.800 | 147 ms | 175 ms | $0.000266 |
| voyage-4 · hybrid1 run | 0.810 | 0.888 | 0.711 | 0.741 | 0.778 | 0.878 | 258 ms | 352 ms | $0.000133 |
| voyage-4 · hybrid · rerank-lite3 runs | 0.836 | 0.879 | 0.798 | 0.809 | 0.833 | 0.878 | 415 ms | 518 ms | $0.1261 |
| voyage-4 · hybrid · rerank-full3 runs | 0.848 | 0.879 | 0.806 | 0.818 | 0.837 | 0.867 | 428 ms | 555 ms | $0.3145 |
| voyage-4 · hybrid · rerank-full · compoundselected3 runs | 0.931 | 0.966 | 0.837 | 0.860 | 0.911 | 0.956 | 535 ms | 758 ms | $0.4929 |
Scores and latency are means across the configured repeats. Cost is the total attributed provider spend for those repeats: three for each reranked arm and one for every other arm.
Decision
Select the measured gain and account for its cost.
I selected hybrid retrieval with full reranking and a repair for multi-part evidence coverage. The first answer replay exposed an abstention-classification defect; later human review exposed citation and grounding errors. After repairs, all 100 answers were regenerated on the same retained evidence. That full review finished September 14 and passed all four predeclared thresholds, with nine failed cases still recorded. The study ended with the reviewed baseline accepted for production, not a claim of perfect answers.
Accepted tradeoff
The selected approach retained voyage-4 embeddings and used rerank-2.5. Its p95 retrieval latency was 758 ms, about 5.0 times the dense baseline. That was acceptable for this research workflow; a faster interaction would require a different latency tradeoff.
Measured cost
$0.0018
attributed provider cost per answerable query execution in the selected arm.
$0.4929 across 270 answerable executions in three clean repetitions. The whole seven-arm comparison, including preparation and the five coverage-repair runs, cost about $1.79 in retrieval provider usage, and the three complete 100-case answer replays about $10.79. Rejected follow-up experiments are separate. None of this is a monthly serving-cost estimate.
End-to-end answer check
Measure what the answer claims, not just whether it cites something.
All ten negative cases initially refused in prose but cited related evidence. The evaluator treated a valid citation as proof of an answer, misclassifying every refusal. Explicit, validated answer outcomes repaired that distinction; malformed or unsupported responses could not become confident answers.
The corrected replay recognized 10/10 abstentions and reduced the invalid-citation case rate from 1.0% to 0.0%. It answered 89/90 answerable cases and cited 83.6% of 116 required evidence groups. These were intermediate structural results, not the later human quality scores.
One remaining refusal was correct given the evidence the model could see: neither required passage had reached its context, although both existed in the corpus. That called for a retrieval repair. Relaxing the answer's evidence requirement would have hidden the failure.
Evaluation runs retained completed work and its input and configuration identity, so interruptions could resume without repeating paid calls or mixing experiments. A later correction made prompt changes invalidate old answers too. Resuming a run safely and measuring a revised system are different requirements.
Targeted coverage repair
Fix what entered the context, not what the answerer was allowed to claim.
The corpus held both required passages. In the global candidate pool, call excerpts crowded out the filing and the full reranker demoted the correct call passage. The repair preserved coverage from both source types and the different parts of a compound question within a bounded context. The gold labels and the requirement to abstain without sufficient evidence did not change.
- Recall@5
0.931
up from 0.848
- Recall@10
0.966
up from 0.879
- MRR
0.837
up from 0.806
- nDCG@10
0.860
up from 0.818
- Complete within five
91.1%
up from 83.7%
- Recovered case
Ranks 1 + 4
both groups · all 3 clean repeats
Two additional repetitions encountered provider rate limits and used a fallback ranking. I retained their results and cost records, excluded those scores from the clean comparison, and obtained three valid repetitions.
That operational failure matters: without provider-status accounting, a safe runtime fallback can masquerade as model-quality variance inside a benchmark.
Human answer-quality gate
A full replay passed. Nine cases still need work.
The first complete review failed citation correctness and groundedness. After a prompt revision, a mixed record passed by regenerating only eight failed answers and carrying 92 older answers forward. That could miss new regressions. I bound every answer to the contract that produced it, then regenerated all 100 answers against the exact retained retrieval.
Generation finished September 11 UTC; named-human adjudication finished September 14. All four thresholds now pass with zero uncertain labels and zero carried-forward answers. The retained results identify the exact answer configuration that was reviewed. This is a full answer-layer review on frozen evidence, not a new live retrieval run.
- Answer correctness
91.1%
target 90.0% · passed on the full replay
- Citation correctness
97.8%
target 95.0% · passed on the full replay
- Groundedness
98.9%
target 95.0% · passed on the full replay
- Abstention correctness
100.0%
target 100.0% · passed on the full replay
The private adjudication preserves 8 answer-correctness failures, 2 citation failures, and 1 groundedness failures across 9 cases; the sets overlap. The review separates unsupported citations, false abstention despite available evidence, missing context, wrong periods and incomplete multi-part answers.
The review workspace supplied assistant-prefilled labels and notes. The reviewer could accept them or change them against the retained evidence; prefilled labels alone were never approval. The nine flagged cases received evidence inspection and clearer notes before the named human exported the final review. The scorer then checked all 100 cases against the exact results file.
This replay used 100 answer calls and cost $3.3241 in attributed provider usage. Retrieval was reused exactly; scoring added no provider calls.
Those case-level questions, licensed excerpts, and reviewer notes remain in the private package. This public record reports the aggregate scores and failure taxonomy needed to evaluate the engineering decision.
| Review | Correctness | Citations | Groundedness | Abstention | Failed cases |
|---|---|---|---|---|---|
| Sept 10 · first complete review | 90.0% | 91.1% | 94.4% | 100.0% | 13 |
| Sept 10 · mixed: 8 new + 92 old | 92.2% | 96.7% | 97.8% | 100.0% | 8 |
| Sept 14 · full contract: 100 new | 91.1% | 97.8% | 98.9% | 100.0% | 9 |
Correctness, citations and groundedness use 90 answerable cases; abstention uses 10 controls. The full replay scores lower on correctness than the mixed record. One earlier failed case now passes, while two others join the failure set. These measurements use the same development gold set; they are not an independent holdout or an estimate of the effect of each individual change.
Rejected follow-ups
More structure did not establish better answers.
Follow-ups used the nine failures, six passing controls and two abstention controls, selected before new outputs. A paired retrieval-and-answer comparison improved complete context coverage from 11/15 to 12/15 answerable cases, but reduced required evidence groups cited from 18/23 to 15/23. Better context coverage alone did not settle answer quality.
Two further answer candidates added more explicit evidence structure. The first raised required groups cited to 16/23 and delivered 14/15 answers, but inspection still found wrong passages, wrong reporting periods and unjustified abstention. The second delivered only 9/15 answers and cited 10/23 required groups, introducing a material omission on a previously passing control. Neither was promoted. A separate narrow retrieval experiment selected the same evidence in both arms and failed to recover the missing outlook; it was not promoted either.
These were diagnostic runs with one execution per arm, not independent holdouts. The paired run held a current corpus snapshot constant rather than recreating the historical corpus; some contexts changed, so answer changes could not all be attributed to the answer instructions. The answer-only candidates reused identical evidence. Structural counts and assistant inspection did not replace the original 100-case human scores.
On September 15 I retained the fully reviewed baseline. My stopping rule was a clear correction of a targeted failure without new material errors in passing controls, while preserving safe abstention. The candidates did not meet it. The nine failures remain documented; marginal benchmark improvements did not justify continued cost and regression risk.
Production check
A working path and a reversible change.
A smoke test exercised two authenticated answers through one account and returned ten retained source links. That verified the deployed path, not customer-scale reliability or broad answer accuracy. The setting applied across production; observation through one account did not make it an isolated account rollout.
A rollback rehearsal exposed an incomplete configuration change and a missing release identity. Startup rejected the inconsistent configuration. I corrected the coupled settings and release tracking, then verified rollback and restoration. The lesson was to test the recovery path as a whole. Operational commands and release identifiers remain in the internal deployment record.
Adverse results
Keep the adverse evidence.
I kept the frozen labels and failed cases rather than changing expectations to fit an output. These limitations belong beside the selected approach's gains.
- 01
Larger dense embeddings regressed. voyage-4-large and voyage-context-4 both scored below the voyage-4 baseline on the ranking metrics that mattered.
- 02
Hybrid fusion alone was not sufficient. It held Recall@10 but reduced Recall@5, MRR and nDCG@10 while more than doubling p95 latency.
- 03
The first answer evaluator failed every abstention control. All 10 negative cases refused in prose, but a citation-presence heuristic counted them as answers. The gate still stopped promotion until the defect was diagnosed.
- 04
Provider rate limits tainted two repetitions. Both fell back to fused ordering. They were retained, labeled and excluded from the clean comparison.
- 05
A passing review had reused answers from an earlier prompt. Only 8 answers were regenerated and 92 carried forward. Every answer now records the contract that produced it, and all 100 were regenerated and reviewed again.
- 06
The passing review still carries failed cases. 9 answerable cases fail at least one criterion. Citation selection, false abstention, missing evidence, wrong periods and incomplete answers remain explicit regression cases.
- 07
The benchmark needs precise periods and expectations. Review exposed date ambiguity in questions about current information and a gold note that needed finer distinctions. Those are benchmark issues for a new version, not reasons to relabel.
Publication boundary
Evidence for the decision, with private material protected.
This case study publishes every comparison arm, the sample composition, review method, adverse outcomes, latency, cost and limitations. Exact account questions, licensed source text, prompts, operational settings and artifact bindings remain in the retained private study records. Public aggregates explain the decision; they do not permit independent reproduction of the private cases.
Limitations
What this evaluation does not prove
- A passed human gate is not perfect accuracy. The full replay passed every predeclared threshold, but nine answerable cases still fail at least one criterion. The review measures one answer contract on retained evidence; it does not establish fresh-corpus performance or guarantee that future answers are correct.
- Corpus answerability is not context coverage. One gold-answerable cross-corpus case correctly abstained because neither required evidence group reached the five-source context. The targeted repair recovered both groups at ranks one and four without teaching the answer layer to invent what it cannot see. The later full answer replay passed the aggregate human gate, while its remaining errors stay explicit.
- It is a bounded financial corpus. One hundred cases across twenty covered tickers are useful decision evidence, not a universal claim across every issuer, source, language, or market.
- Provider fallback is not quality variance. The four non-reranked arms ran once. Reranked profiles were repeated, but two coverage-repair runs were excluded after provider rate limiting triggered the safe fused-order fallback. This report does not present confidence intervals.
- Cost is an attributed estimate. The ledger uses the configured provider prices and token counts. It excludes database, hosting, engineering, and human-adjudication cost.
- The deployment check used a later release. The smoke test and the frozen benchmark have distinct retained version records. A working deployed path does not rerun the historical evaluation or extend its quality findings to every later release.
Related case study
When better evidence changes the answer
Retrieving a passage is only part of the problem. A later 1,000-case investigation followed evidence through thesis assessments, citations and email delivery. Read what I repaired, which model and prompt changes I rejected, and what fuller sources cost.
Decision at study close
Keep the reviewed baseline. Learn from real use.
The selected approach met the study's retrieval and answer-quality targets, with its remaining limitations accepted. I stopped repeated benchmark tuning and retained the original evidence and adverse results. Further work should respond to meaningful product failures and distinguish defects from ambiguity in the benchmark itself.
Try TickerTrac free