Dense vs. hybrid retrieval for financial evidence
A 100-case evaluation of seven retrieval profiles over filings and earnings calls—followed by an answer-layer failure, a contract repair, a targeted coverage fix, named-human scoring, and production promotion with an observed rollback. A complete answer replay now passes the human thresholds, with nine residual failures preserved as accepted limitations.
The outcome
Better coverage and ordering near the top, bought with latency.
Hybrid retrieval alone did not beat dense retrieval. The full reranker became the first finalist because it improved early ordering, but it still lost one cross-corpus answer. A narrow source-balanced, clause-ranked path for explicit compound questions recovered that case and improved every reported ranking and coverage metric across three clean repetitions. It is now the production default; its additional latency remains part of the decision record.
- nDCG@10
0.860
12.9% relative vs. dense
- MRR
0.837
13.3% relative vs. dense
- Recall@5
93.1%
+9.5 pp vs. dense
- Retrieval p95
758 ms
+605 ms vs. dense
Question
Which policy retrieves the evidence an answer actually needs?
Financial research retrieval has two different jobs. It must find exact language—metric names, executives, sections, and periods—and it must also survive paraphrase. Dense search is strong at semantic similarity. Lexical search is strong at exact terms. Reranking can reorder the combined pool, but it adds provider cost and latency.
We evaluated the retrieval stage independently from answer generation. That separation lets us measure whether the correct retained evidence appeared in the first five or ten results without giving fluent prose credit for a miss.
Method
One frozen corpus. One adjudicated gold set. Seven arms.
A named reviewer checked every case against the complete retained source. Answerable cases bind each necessary fact to one or more acceptable evidence identities. Ten negative cases deliberately require abstention and are not misreported as retrieval successes.
Dataset
100
human-adjudicated questions across 20 tickers; 90 answerable and 10 insufficient-evidence controls.
Controls
Fixed evidence and top-k policy
Every arm used the same release, source-policy snapshot, questions, evidence identities, candidate bound of 50, and RRF constant of 60.
Repeated runs
Three per reranked arm
Dense and plain hybrid arms ran once. Each reranked arm has three clean repetitions; the table reports means across thirteen valid outputs. Two additional rate-limited coverage runs are retained separately as tainted.
Case mix
- Filing
- 20
- Earnings call
- 20
- Cross-corpus
- 15
- Exact metric
- 15
- Multi-condition
- 10
- Speaker + section
- 10
- Insufficient evidence
- 10
Metrics and what they mean
- Recall@5 / Recall@10
- Share of required evidence groups found inside the first five or ten results.
- MRR and nDCG@10
- Whether acceptable evidence appears early and whether useful ordering is preserved.
- Complete-case rate
- Share of answerable questions for which every necessary fact was present by the cutoff.
- p50 / p95 latency and cost
- Retrieval-stage runtime and attributed provider spend, not full answer latency or serving cost.
Results
The larger embedding models lost. Hybrid needed reranking to win.
Bigger was not better on this corpus. Plain hybrid retrieval matched dense Recall@10 but ranked useful evidence worse. Both rerankers reversed that ordering problem. The full model became the first finalist; the compound-question coverage path then improved both ordering and evidence-group coverage.
| Arm | R@5 | R@10 | MRR | nDCG@10 | Complete@5 | Complete@10 | p50 | p95 | Cost |
|---|---|---|---|---|---|---|---|---|---|
| voyage-4 · dense1 run | 0.836 | 0.888 | 0.739 | 0.762 | 0.811 | 0.878 | 128 ms | 153 ms | $0.000133 |
| voyage-4-large · dense1 run | 0.767 | 0.879 | 0.700 | 0.736 | 0.756 | 0.867 | 163 ms | 193 ms | $0.000266 |
| voyage-context-4 · dense1 run | 0.681 | 0.802 | 0.580 | 0.629 | 0.667 | 0.800 | 147 ms | 175 ms | $0.000266 |
| voyage-4 · hybrid1 run | 0.810 | 0.888 | 0.711 | 0.741 | 0.778 | 0.878 | 258 ms | 352 ms | $0.000133 |
| voyage-4 · hybrid · rerank-lite3 runs | 0.836 | 0.879 | 0.798 | 0.809 | 0.833 | 0.878 | 415 ms | 518 ms | $0.1261 |
| voyage-4 · hybrid · rerank-full3 runs | 0.848 | 0.879 | 0.806 | 0.818 | 0.837 | 0.867 | 428 ms | 555 ms | $0.3145 |
| voyage-4 · hybrid · rerank-full · compoundselected3 runs | 0.931 | 0.966 | 0.837 | 0.860 | 0.911 | 0.956 | 535 ms | 758 ms | $0.4929 |
Scores and latency are means across the configured repeats. Cost is the total attributed provider spend for those repeats: three for each reranked arm and one for every other arm.
Decision
Promote the finalist. Keep the rollback.
The original retrieval finalist entered production on September 9, 2026. The first answer replay then failed its abstention gate. We retained that adverse result, repaired the shared response contract, and replayed the identical 100-case bound. That exposed one cross-corpus coverage miss rather than an answer-policy defect. The source-balanced compound path recovered the required evidence across three clean repetitions. A fresh answer replay then failed its first complete named-human review on citation correctness and groundedness. After an answer-prompt revision, a record that regenerated eight of those failed answers passed every predeclared threshold, but it carried the other 92 judgments forward from the earlier prompt. On September 10, the exact finalist ran two successful production answers for one observed account and survived a configuration rollback drill. The environment profile applies to every Ask request, making the finalist the production default; a full replay of its revised answer prompt is still open.
Selected profile
- Strategy
- Hybrid lexical + dense retrieval
- Embedding model
- voyage-4
- Candidate pool
- 50
- RRF constant
- 60
- Reranker
- rerank-2.5
- Compound coverage
- Source-balanced clause ranking for compound questions
Measured cost
$0.0018
attributed provider cost per answerable query execution in the selected arm.
$0.4929 across 270 selected-arm executions; $1.1204 across the original six-arm preparation and bakeoff ledger: 2,035 provider calls and 18,245,286 input tokens. The five coverage-repair runs added $0.6744, including the two retained rate-limited runs.
End-to-end answer check
The failed gate became a better answer contract.
In the first run, all ten negative cases refused in prose and cited related evidence. The evaluator nevertheless labeled every response with a valid citation as answered. That 0/10 result was one shared classification defect, not ten independent retrieval failures. We moved the outcome and bounded abstention reason into the provider-side schema, validated the response locally, and made malformed, incoherent, or uncited output fail closed.
- Required-evidence citation recall
83.6%
116 required evidence groups
- Invalid-citation case rate
0.0%
down from 1.0%
- Correct structured abstentions
10 / 10
up from 0 / 10
- Answerable cases answered
89 / 90
one bounded-context retrieval miss
The corrected replay cost $3.6028 for 100 completed cases: $3.4885 for answer generation, $0.1141 for reranking, and $0.000142 for query embeddings. The initial and corrected answer runs cost $7.4690 combined.
Each completed case was appended, flushed to disk, and recorded in a durable manifest before the next paid call. A restart reuses the completed prefix only when every frozen preflight binding still matches, preventing partial runs from silently mixing configurations or repaying for completed cases.
The replay exposed a different final miss: on one answerable cross-corpus case, neither of the two required evidence groups reached the five-source answer context. The answer layer correctly abstained because it could not see the needed evidence; the gold label remains answerable because those passages exist in the frozen corpus. That is a retrieval-coverage failure, not a reason to weaken abstention. The subsequent retrieval repair recovered both groups at ranks one and four in every clean repeat. A fresh end-to-end replay then became the artifact for named-human scoring.
Automated structure was never presented as semantic approval. The first complete named review covered all 100 cases, used no uncertain labels, and failed two of the four predeclared thresholds: citation correctness and groundedness. The next record, on September 10, passed all four but regenerated only eight answers under a revised prompt, as the human-gate section below explains.
The first semantic answer review covers all 100 cases because the earlier gold-set adjudication verified retrieval evidence, not the generated answers. Later replays are incremental: a prior pass may carry forward only when the gold set and frozen corpus still match and the complete per-case review input has the same fingerprint. Changed cases, prior failures or uncertainties, and reviews without that provenance reopen automatically. The authoritative scorer still evaluates the full 100-case denominator. That rule was not enough on its own: an unchanged answer can be carried forward after the prompt that would now produce it has changed. Each answer now records the answer contract that produced it, and a replay refuses to reuse an answer from a different contract.
Targeted coverage repair
Fix what entered the context, not what the answerer was allowed to claim.
The corpus held both required passages. In the global candidate pool, call excerpts crowded out the filing and the full reranker demoted the correct call passage. For explicit compound questions only, the repair preserves bounded candidates from both source types, ranks each interrogative clause separately, and interleaves the results. The gold labels and fail-closed answer contract did not change.
- Recall@5
0.931
up from 0.848
- Recall@10
0.966
up from 0.879
- MRR
0.837
up from 0.806
- nDCG@10
0.860
up from 0.818
- Complete within five
91.1%
up from 83.7%
- Recovered case
Ranks 1 + 4
both groups · all 3 clean repeats
The three clean repetitions cost $0.4929. Two back-to-back runs hit Voyage's rerank request limit and fell back to fused ordering. Their artifacts were retained, labeled tainted, excluded from quality aggregates, and replaced after a provider cool-down. Total spend across all five runs was $0.6744.
That operational failure matters: without provider-status accounting, a safe runtime fallback can masquerade as model-quality variance inside a benchmark.
Human answer-quality gate
A full replay passed. Nine cases still need work.
The first complete review failed citation correctness and groundedness. After a prompt revision, a mixed record passed by regenerating only eight failed answers and carrying 92 older answers forward. That could miss new regressions. We bound every answer to the contract that produced it, then regenerated all 100 answers against the exact retained retrieval.
Generation finished September 11 UTC; named-human adjudication finished September 14. All four thresholds now pass with zero uncertain labels and zero carried-forward answers. The answer contract matches the scorer source at release 68b416de. This is a full answer-layer review on frozen evidence, not a new live retrieval run.
- Answer correctness
91.1%
target 90.0% · passed on the full replay
- Citation correctness
97.8%
target 95.0% · passed on the full replay
- Groundedness
98.9%
target 95.0% · passed on the full replay
- Abstention correctness
100.0%
target 100.0% · passed on the full replay
The private adjudication preserves 8 answer-correctness failures, 2 citation failures, and 1 groundedness failures across 9 cases; the sets overlap. The review separates unsupported citations, false abstention despite available evidence, missing context, wrong periods and incomplete multi-part answers.
The review workspace supplied assistant-prefilled labels and notes. The reviewer could accept them or change them against the retained evidence; prefilled labels alone were never approval. The nine flagged cases received evidence inspection and clearer notes before the named human exported the final review. The scorer then checked all 100 cases against the exact results file.
This replay used 100 answer calls and cost $3.3241 in attributed provider usage. Retrieval was reused exactly; scoring added no provider calls.
Those case-level questions, licensed excerpts, and reviewer notes remain in the private package. This public record reports the aggregate scores and failure taxonomy needed to evaluate the engineering decision.
| Review | Correctness | Citations | Groundedness | Abstention | Failed cases |
|---|---|---|---|---|---|
| Sept 10 · first complete review | 90.0% | 91.1% | 94.4% | 100.0% | 13 |
| Sept 10 · mixed: 8 new + 92 old | 92.2% | 96.7% | 97.8% | 100.0% | 8 |
| Sept 14 · full contract: 100 new | 91.1% | 97.8% | 98.9% | 100.0% | 9 |
Correctness, citations and groundedness use 90 answerable cases; abstention uses 10 controls. The full replay scores lower on correctness than the mixed record. One earlier failed case now passes, while two others join the failure set. These measurements use the same development gold set; they are not an independent holdout or an estimate of the effect of each individual change.
September 14–15 · comparisons and stopping decision
Two answer candidates stayed out of production.
We required each citation to carry a verbatim span from its exact retained passage, made validation failures visible, and strengthened period and completeness instructions. A separate retrieval change kept the full question visible while ranking each part and recorded where evidence left the context.
The first four production calls reused retained sources and cost $0.1299. Assistant review found one citation repair and three unresolved cases. That answered a narrow question about generation; it could not test the retrieval change or reveal regressions among previously passing cases.
We then selected all nine failures, six passing controls and two abstention controls before seeing new outputs. Old and new retrieval read the same current production database snapshot, each selecting five passages for its own answer. The run made 34 Anthropic calls, 34 embedding requests and 48 rerank requests, with no retries. Estimated total cost was $1.3829 under the approved $6 cap.
| Check | Before | After |
|---|---|---|
| Complete evidence context | 11/15 | 12/15 |
| Answerable cases receiving an answer | 14/15 | 13/15 |
| Required evidence groups cited | 18/23 | 15/23 |
| Correct abstention controls | 2/2 | 2/2 |
One cross-corpus case gained its missing passage. The answer contract rejected two invalid quotes: one mixed neighboring passages, and one joined separated text. Assistant inspection also found omitted facts, a missed reporting period and a dropped qualifier. One suspected citation regression was a valid alternate source that the benchmark did not recognize.
We then tested a candidate that selects numbered, exact source passages before writing claims. Seventeen more Anthropic calls reused the recent evidence, with no retrieval or retries, at an estimated cost of $0.6985 under a separate $2 cap. Answers increased from 13/15 to 14/15, required groups cited from 15/23 to 16/23, and local validation rejections fell from two to zero. Both abstention controls stayed correct.
Assistant inspection found a recovered cross-source answer and a restored qualifier, but also a wrong-passage citation, a revenue change assigned to the wrong year, an unjustified abstention and wrong-period answers. We did not promote the candidate. Valid passage IDs and more delivered answers did not establish better answers. All nine historical failures stayed recorded.
The next candidate mapped each requested fact to a scope, reporting period and short source quotes before answering. Its 17 Anthropic calls cost $0.8235 under another $2 cap, using the same saved evidence. It recovered one current-period answer, but delivered only 9/15 answerable cases and cited 10/23 required groups. Six answerable outputs failed local validation. Both abstention controls exposed no invented answer, although their structured responses also failed validation.
A previously passing control omitted a ratio the question explicitly requested, even though the supporting text was present. We kept this candidate out of production too. Our stopping rule was a clear correction of a targeted failure, no new material errors in the six passing controls, and safe abstention—not a perfect score or all nine cases solved. This candidate failed that rule. We stopped expanding the answer schema and investigated the measured losses of evidence during retrieval.
A final narrow retrieval experiment added an outlook-ranking request. Its $0.1297 comparison selected the same five sources in both arms and still missed the required outlook. The source reply was complete, so truncation did not explain the miss. We did not promote that experiment either.
On September 15, the owner accepted the fully reviewed baseline: it already met all four original targets. We restored its answer contract and clause requests, closed the unsuccessful experiments, and kept the nine failures as documented limitations. Further marginal gains did not justify the risk of new errors or the money, compute, time and attention required. The next evidence should come from product use, with meaningful failures retained for a future decision.
Runtime: 754f5232, comparing the prior answer/retrieval modules from ad8aa144 with the deployed modules. Eleven pairs had identical contexts; six differed, so their answer changes cannot be attributed to the answer contract alone. This operator run used the production image and configuration, not the signed-in Ask route. It held the current corpus constant without recreating the historical snapshot. The follow-up ran candidate 44ba5485 in production image b44733aa with identical source order, excerpts and metadata across all 17 comparisons. The requested-fact follow-up ran candidate 6d91eb0f in pinned production image 3388e2fc, with the same 17 contexts and no retrieval, retries or database writes. Each comparison used one execution per arm; none was an independent holdout. Private content and the original 100-case human scores remain intact.
Production proof
Production proof—and two operational hiccups worth keeping.
One named account exercised the exact production profile through two authenticated Ask executions, returning ten retained source links in total. Observed end-to-end latency was approximately 5500 ms and 3800 ms; attributed provider cost was $0.0557.
Observed production proof
2 answers
one observed account · 10 cited sources · exact release 5559e687
Rollback proof
Passed
API and workers healthy after restoring the promoted profile at the exact release.
The first rollback attempt followed a stale two-switch instruction: dense retrieval plus reranking off. Startup correctly failed closed because compound query coverage still required reranking. Disabling hybrid, reranking, and compound expansion together completed the rollback. A second hiccup exposed a missing release marker during direct process recreation; restoration supplied the exact release SHA. A source audit also found that the profile was selected from global runtime settings rather than a per-account branch: the proof was observed through one account, but the deployment scope was already production-wide. All three findings were added to the record.
Failures and tradeoffs
The adverse results are part of the result.
- 01
Larger dense embeddings regressed
voyage-4-large and voyage-context-4 both scored below the existing voyage-4 dense baseline on the ranking metrics that mattered.
- 02
Hybrid fusion was not sufficient
Lexical plus dense retrieval preserved Recall@10 at 0.888 but reduced Recall@5, MRR, and nDCG@10 while more than doubling p95 latency.
- 03
The first finalist still lost evidence at 10
The original full-reranker finalist's Recall@10 fell 0.9 percentage points and complete-case@10 fell 1.1 points versus dense. The compound-query repair later reversed both regressions.
- 04
Ranking quality cost latency
Selected-arm p95 was 758 ms, 5× the dense baseline. That is acceptable for the current research path, but it is not free.
- 05
The first answer classifier failed every abstention control
All 10 negative cases refused in prose, but a citation-presence heuristic mislabeled those refusals as answers. The gate correctly stopped promotion even though its failure mode needed diagnosis.
- 06
The repair exposed a different retrieval miss
The structured replay passed 10/10 negative controls and removed invalid citations, while one answerable cross-corpus case abstained because neither required evidence group reached the bounded five-source context.
- 07
Provider limits tainted two repetitions
Two back-to-back coverage replays hit the rerank request limit and safely fell back to fused order. We kept both artifacts and their 0.6744 total five-run spend, excluded the tainted scores, and obtained three clean replacement repetitions after cooling down.
- 08
Full human review retained nine failed cases
The full contract review passed without uncertain labels, but 9 answerable cases still failed at least one criterion. Citation selection, false abstention, missing evidence, wrong periods and incomplete answers remain explicit regression cases.
- 09
The passing review reused answers from an earlier prompt
The first complete review failed citation correctness and groundedness. After the answer prompt was revised, only 8 failed cases were regenerated; 92 answers and their judgments carried forward from the earlier prompt while production served the revised one. Each answer now records the contract that produced it, a replay refuses to mix contracts, and all 100 answers have now been regenerated and reviewed. The earlier mixed result remains visible as a correction to our evaluation method.
- 10
The benchmark needs precise periods and expectations
Review exposed date ambiguity in questions about current information and a gold note that needed finer separation of near-term guidance from broader growth commentary. Those are benchmark issues to clarify in a new version; we kept the frozen labels and failures rather than changing expectations to fit the output.
- 11
The rollback and scope records needed correction
The first production rollback omitted the compound-query switch, and the canary description implied per-account isolation that the runtime did not implement. Startup failed closed, the coupled configuration was corrected, and the record now states the production-wide scope and exact release identity.
Publication boundary
Publish the numbers. Protect the underlying text.
The measurements are the legitimacy of the bakeoff, so this page publishes every arm, the adverse outcomes, cohort composition, latency, cost, release, and immutable hashes.
The private package also contains exact investor questions and retained source excerpts. Some questions came from account history and some transcript text is licensed. Publishing that text would expose data that is not needed to audit the aggregate result. The hashes bind this report to the retained package without copying its contents into the public site.
- Source release
- 19cadb620919852f13736e7958c26a7b01d52a07
- Activation release
- e58faf96e8811f79c80195c70c3eb045fe870077
- Corpus snapshot
- d8a3930a06d445c051161ba20a779415e758e62efb4f866edd158efb7c7778eb
- Gold set
- 47ec16c78d94689d2f74af6381ef93669ba43ced63816f2ead39e71fe7b1dbca
- Approved plan
- 3886ab478a8e9782d20c5427083a881c7e69245fb27955c15c33fdf4302c8cad
- Comparison
- 42688e52c6914a14ab5df6984aedf8c38edd7953748b641e5cd31433a02c0cc4
- Retrieval archive
- 097c2870b0c2ff19d714b761475ec00d7a622c969de5195099c9674e229b6e1c
- Initial answer release
- 6e95d1052f426093bd779e755b0b593e5b6b0284
- Initial answer results
- 365de351f46501702d1fe13720131a6f466d446f64e384318a0e0afdf94d4774
- Initial private archive
- 93dab74c8de731697bfc9ec984fe2a096bc3d566ab4842d64b8da6b10f1f5da9
- Corrected answer implementation
- 93d94ce2b10a1b0ac7d54279415f83d589ceee6c
- Corrected production release
- 5e93ababa008421ab8dc9b2d6fa09be17a99316f
- Corrected answer results
- ab4e2a827e9df8e9a3aeb1f8570705a1f04535aa846ab3a7e8004a3d48bc6e75
- Corrected answer summary
- 85d47cf82976a7157285d8cbf242cf4ec912f0b674acb360c2ef452c752b0bcb
- Corrected human-review template
- 88e8327e24418b6e5dd65db123d312de41b999165872454d35926039df64e699
- Corrected run manifest
- 037e618a4dd3225ef78d48be62482cbc3a2bb158747d3be70f6388191f27bddc
- Corrected private archive
- b6bab8f44338f687c54ae8e8538cbcce09bc523ae6512a6850f1b353b218e4e3
- Coverage-repair release
- e58faf96e8811f79c80195c70c3eb045fe870077
- Coverage clean result 1
- 1d101238768c0d4c2fabe1a859b7f03f4b087bb5675e459a0da4ab68a901b690
- Coverage clean result 2
- 28ffb1af58d4b191519e1e4299f14067787457c3454d37603c5bf5446f7c5923
- Coverage clean result 3
- a6c9cde1bc863f9fc3abf15b26cf8dabadc0fc5a7282be13a91ea295bb348496
- Coverage tainted result 1
- 47d99eaaa52a0c988c104925997e9d35e43cdb03c616ccab2a66270d4863e4c5
- Coverage tainted result 2
- 5f1dbbd386a7b5f769253337e14e1e6b1fcb3224438f1ecc76e25c4ceb624dd5
- Coverage private archive
- f0a05348e85a8e6c66f41211643e0be83915a061e1d0baa60fcb9788065db8b0
- Coverage run manifest
- 7e4b2a309a7e91f2b5693cd51884fee0286731e1e07eac89b198f965afd9e442
- First complete answer results
- 5e8ea8489bec317f2046467e5048790009e15711a4e575dc269909654a47b945
- First complete human review
- 76af0f6d0221684a9c74a2a2493afd2252dd2d325ecdf562d7161c8aee6c4623
- Combined answer results
- 997bc3b58576ca7515ed0017cac6d1bdbc0a68b10dd909d7f042769beffcdadb
- Combined human review
- a8d8b81af33c5b4473caff98660a87faf2b35ee9163738a716bce1db43af3142
- Full replay answer contract
- dd876ab7b105a2fa95c105b60b395b91c65a01d76855e8b3e815cbeafa771ee9
- Full replay results
- 5a80798e5c13fae7f46480d54e55ebd439c454b26e827b426a58cd6701ccc718
- Full replay human review
- d038bfb2fba41911adca0e4fc714d8d258ebd2919d746320cf9c3444b4cab90d
- Full replay human summary
- 890ca6d8b48dd76ef5508ae60ddec566f4b2aa41130b792f4f596d12a218682f
- Full replay manifest
- daf86634f0f11be98470d34ef33efe88581a943268a58f30560553e821f91deb
- Full replay scorer release
- 68b416de233a34a86376a8a161e2dd35d493e6c2
- Production promotion release
- 5559e687f10f7e27bbec8909bbc99cb017ac3f65
Limitations
What this evaluation does not prove
- A passed human gate is not perfect accuracy. The full replay passed every predeclared threshold, but nine answerable cases still fail at least one criterion. The review measures one answer contract on retained evidence; it does not establish fresh-corpus performance or guarantee that future answers are correct.
- Corpus answerability is not context coverage. One gold-answerable cross-corpus case correctly abstained because neither required evidence group reached the five-source context. The targeted repair recovered both groups at ranks one and four without teaching the answer layer to invent what it cannot see. The later full answer replay passed the aggregate human gate, while its remaining errors stay explicit.
- It is a bounded financial corpus. One hundred cases across twenty covered tickers are useful decision evidence, not a universal claim across every issuer, source, language, or market.
- Provider fallback is not quality variance. The four non-reranked arms ran once. Reranked profiles were repeated, but two coverage-repair runs were excluded after provider rate limiting triggered the safe fused-order fallback. This report does not present confidence intervals.
- Cost is an attributed estimate. The ledger uses the configured provider prices and token counts. It excludes database, hosting, engineering, and human-adjudication cost.
- Production proof used a later application release. Production proof ran at release 5559e687, later than the frozen benchmark package. The live health response is authoritative for the canary release; the hashes and private package remain authoritative for the frozen evaluation. This report does not conflate the two.
Next checkpoint
Keep the reviewed baseline. Learn from real use.
The retrieval gate is complete, the finalist is the production default, and one named account has exercised it with observed rollback proof. The full answer-contract review passes, and its remaining limitations are accepted for current use. Preserve the original gold and adverse results. Use existing records to retain meaningful failures from real use, distinguish product defects from benchmark ambiguity, and revisit a repair when its user benefit warrants the cost. Routine paid replays and marginal benchmark optimization are stopped.
Try TickerTrac free