When better evidence changes the answer

TickerTrac checks new evidence against an investor's saved Company thesis or Investment theme. In an initial audit, some judgments invented expectations, attributed a customer's economics to a supplier, or treated a conditional risk as a contradiction. Of 38 reviewable theme cases, eight had unsupported directions; another 22 selected cases had too little retained evidence to assess reliably. This was a sample chosen to find risks, not an estimate of the overall error rate.

A separate delivery defect had allowed 192 findings into accepted email receipts despite their saved decision not to interrupt the reader. The email loader was reconstructing permission to alert from relevance alone. Better model answers could not fix that path.

Earlier evaluations had checked judgments on supplied inputs. Their theme-specific follow-up covered only ten cases, including two clear positive alerts. They did not establish reliability across causal edge cases or test the complete path from saved judgment to email. Short retained excerpts also made later diagnosis harder.

We compared 1,000 cases: 500 Company cases and 500 theme cases, using 800 retained news inputs and 200 constructed filing and transcript cases. Three model policies and multiple source lengths produced 3,600 comparisons. Source-first automated labels were reconciled before model results were revealed; they are not independent human ground truth.

The transcript path was losing useful judgments at the citation boundary and producing weaker explanations. We moved retained transcripts to Sonnet 5.5. GPT-6 Luna stayed on news; replacing it with Sonnet would have cost more while missing more useful alerts in this comparison. Filings stayed on Haiku 4.5: the tested alternative traded fewer invalid alerts for lower useful recall.

A separate 52-pair filing analysis found that longer selected passages exposed additional useful facts but sometimes removed answerable material. It did not establish that wholesale passage replacement was better. Preserving existing facts while adding missing context remains the next candidate to test.

The delivery gate now honors the stored interruption decision. An append-only correction history preserves the original judgment and what was delivered. Sixteen stored records behind nine confirmed errors were corrected without sending new alerts. Future audits retain the bounded input and prior-evidence context actually shown to the model.

Review then found a subtler defect: cutting an exact quote to its first 350 characters could remove the decisive sentence while leaving its alert intact. The repaired selector requires complete sentences around a unique verbatim evidence anchor, including adjacent context. If that passage cannot fit, the judgment remains related but undirected and cannot alert. Exact wording is still not proof of a correct causal interpretation.

Replaying 840 saved replies made no provider calls. Prefix clipping admitted 476 alerts; both strict rejection and the new selector admitted 429. The selector recovered no additional real cases over rejection. Its recovery path is covered by controlled tests, not demonstrated as a recall gain here.

Article rereads now count attempts before dispatch, including failures and denied work, with separate limits of six fetch attempts and six judgment attempts per company per pass. An article-based direction or alert requires an exact supporting passage of at most 350 characters. That passage reaches the app and digests and survives pruning of the shorter-lived audit corpus, with its retrieval time, resolved source URL and judged-text hash. It is a retained excerpt, not an archive of the whole publisher page.

We compared the original and an aligned transcript prompt on the same 140 sources, using the corrected parser and existing fallback policy. The original held-out slice has 126 cases, but these cases had already informed development. It is not fresh confirmation.

The candidate improved directional judgments but lost five useful alerts, all in the Company slice, at essentially the same model cost. The clustered 95% interval for the useful-recall change was −12.5 to +1.1 percentage points. Only 13 cases called for quiet and 20 remained unscored, so zero invalid alerts does not establish a low production false-alert rate. We kept the original prompt.

We selected 157 eligible cases whose short news inputs were related but had not produced an alert. Publisher retrieval returned 92 articles across 89 event clusters. Twenty-five sites refused access, 39 yielded too little text and one timed out. These failures stay in the denominator; inaccessible pages are not assumed to contain no useful facts.

The article exposed 33 useful opportunities, versus five in the corresponding short inputs. The model recovered 21 of those 33, with a clustered 95% interval of 47.2%–79.3%. Across all 157 eligible cases, the observed useful yield was 21. End-to-end recall remains unknown because 65 pages were unavailable. The baseline produced no alerts by selection; this was not a random sample.

The evaluation itself needed correction. Initial labels scored 16 useful and 15 invalid alerts. Some required quantified financial impact for a concrete checkpoint, assumed a recap was already known despite empty prior context, or confused partial weakening with full invalidation. A second source-only pass covered every retrieved article and its short input, with candidate answers and first labels hidden. It produced the table above.

We specified that correction after seeing the initial outputs. These are reconciled results and sensitivity analysis, not a preregistered success. Twenty-eight sources remain unscored for interruption, including eight proposed alerts. Three false alerts among 31 quiet cases also leave meaningful noise. We retained the bounded article policy with stronger provenance; these results do not justify expanding it.

The article judgment cost was about $0.00191 per confirmed additional useful alert, excluding the first reading, retrieval and research. Median fetch time was 0.715 seconds and median model time 9.43 seconds. Availability and eligibility in this selected sample do not support a monthly or per-user forecast.

The original 1,000-case study recorded $61.8319 in token-based cost plus $0.5750 reserved for uncertain outcomes. This follow-up admitted 898 requests: 897 completed and one remained uncertain. It recorded $17.5553 plus $0.072255 reserved. Most follow-up spending went to source labeling, reconciliation and output review, not the production-model article judgments. These are application-provider accounting figures, not settled invoices, and exclude the coding session.

The implemented repairs enforce stored alert decisions, preserve usable citations and bound failed work. They do not establish that every causal judgment is correct. Automated labels, reused cases, uncertain interruption thresholds and limited quiet controls constrain the quality claims. Publisher pages were fetched later; we did not independently verify that their contents were unchanged since publication.

The next useful comparison needs new quiet events, a calibrated interruption rubric and filing context that adds missing facts without discarding retained ones. We should measure useful recovery, unsupported claims, missed alerts and total cost together. Repeatedly relabeling this sample until it looks good would not answer those questions.

Download the aggregate follow-up results (JSON). The download includes all alert outcomes, both article label passes, cost accounting and a hash binding it to the internal aggregate receipt. It contains no private theses, account details, case identifiers or publisher text. A hash binds a version; it is not independent validation of the labels.