What an agent comparison revealed about our production research

The first study gave three experimental workflows the same frozen public evidence, preselected for each synthetic investment question: a single answer, a delegated investigation, and an agent that chose which evidence to read. The model and reasoning setting were held constant. Tool access, orchestration and execution scaffolding differed.

The agent completed 22 of 24 selected attempts. Three model reviewers assessed those 22 answers alongside the corresponding earlier single-answer and delegated outputs. They saw the source material with method labels hidden, although answer structure could reveal the method. All three reviewers belonged to the same model family as the candidate, so self-preference is another possible bias. A pass required correctness, preserved qualifiers, supporting citations, honest coverage limits and an answer to the actual question.

The agent's 12 passes become 12 of 24 when incomplete attempts are included. Its main remaining weakness was coverage: eight of the ten completed answers that failed review misstated what the evidence did or did not contain. A valid citation alone did not make an answer complete.

The panel also graded the same baseline answers differently from an earlier panel. Those scores cannot form a before-and-after chart. The defensible result is the comparison within this panel, with reviewer disagreement visible. These selected questions and automated judgments do not estimate accuracy across everyday use.

A second pilot compared the production Ask path with the application's own tool-using research loop on eight selected questions. This was a different implementation and a whole-workflow test. Both arms used the same model, but their reasoning settings, output allowances and access to retained evidence differed. Sharing a database snapshot did not give them identical evidence access.

Initial attempts exposed credential, structured-output and budget-accounting defects. We changed the candidate to supply its final-answer schema on each request and settle priced usage instead of accumulating full reservations. With owner approval, we reran all eight pairs. The earlier attempts remain in accounting, outside this final cohort's quality counts. The result measures the repaired candidate, not its original configuration.

The tool loop's three incomplete attempts stopped at an output limit, a tool limit and an uncited-answer check. They count against operational usefulness, but they are not three demonstrations of incorrect reasoning. Three of the five completed responses contained stray tool markup even though their cited figures were supported. One of those three also overstated source coverage; a fourth completed answer made a different coverage overclaim. In two cases the loop reached evidence that Ask missed. The new output contract did not solve delivery, and its role in the markup defect remains untested.

Codex and Claude's final diagnostic labels were three useful Ask responses and one useful loop response out of eight attempts each. Both reviewers knew the execution history; blinding was unreliable and calibration anchors were not rescored. We initially agreed on 14 of 16 grades and resolved the other two against the question and retained evidence. A strict failure could mean broken delivery, an unanswered part of the question or a missing explanation. Those labels cannot rank the models' reasoning ability.

One question asked whether Microsoft's earnings call changed the interpretation of its filing. Older filing passages reached the candidate pool, but the current earnings release was outside Ask's indexed path and the final selection contained only call excerpts. The answer compared prepared remarks with Q&A instead. Both evidence access and final selection needed repair.

An Apple question used an older quarter; the newer retained passage never reached its dense candidate pool, and the lexical search returned nothing to recover it. An Intel question needed both reported and adjusted profit, but its current release was outside Ask's indexed source path. Related cash-flow data could not substitute for the requested profit reconciliation.

Two further answers illustrate a delivery defect. The model had produced useful, cited explanations of why the evidence was insufficient. Application code replaced both with a generic sentence. A model-quality score based only on delivered text concealed that cause.

The deployed repairs let Ask read eligible retained filing passages without requiring a new embedding, match focused question terms, preserve requested source and period comparisons through final selection, and retain validated explanations of insufficient evidence. Some of the useful source access observed in the loop is now available through ordinary Ask retrieval. Source permissions and citation checks remain in force. Existing cached answers cannot masquerade as outputs of the revised contract.

Regression tests and read-only source checks confirmed the changed behavior. Replaying the two saved model responses confirmed that their explanations now survive validation. That replay made no new model call. We have not measured post-repair live answer quality, so there is no claimed improvement from 3/8 to a new score.

The earlier SDK candidate also exceeded its predeclared latency threshold: attempted 95th-percentile latency was 77.8 seconds against a 70.5-second trigger. The original write-up incorrectly said it stayed below that trigger; the attempt-level audit corrected it.

The earlier SDK estimate mixed completed and attempted populations and omitted earlier failed attempts. Reconciliation of that subscription-billed run found about $0.31 of SDK-reported estimates across attempts with usable usage receipts, plus $0.80 reserved for attempts with unknown usage. These are accounting estimates and holds, not an incremental subscription invoice or a cost-saving result. Engineering and review time were additional costs.

The later eight-pair cohort used about $0.72 at configured prices. Including setup and uncertainty holds brought the pilot's conservative budget debit to about $1.70 within its $6 allowance. We stopped there. Low provider spend alone would not justify indefinite iteration or compensate for the time spent repairing the evaluation.

The decision was to retain Ask and ship the located repairs. The application tool loop completed only five questions against its predeclared minimum of seven; it did not earn a switch. The earlier fixed-packet result favors that configuration under one advisory panel, against experimental baselines with question-selected evidence. Establishing a general quality advantage would require a separate controlled comparison. This study closes with a deployed repair and a bounded decision, while leaving that broader question open.