Trust sheet: AI Signal
AI Signal is the daily news tool we run on this site. It is the worked example of a trust sheet because it is ours to describe honestly, including the parts that are not proven yet, and including that its first redesign did not pass its own test.
Trust sheet
AI Signal: daily draft posts from the AI news
Every morning it reads about thirty public news sources (the frontier labs, the open-source ecosystem, industry press, the governance beat, and research trade press), scores what it finds, and drafts a small set of LinkedIn post options grounded in the most relevant items. Each option connects a few facts and, where it can, ends with the open questions they leave. It saves the reading, and it is not a substitute for choosing what to say.
- Purpose: save the daily reading of the AI news and surface the connections worth commenting on, so that a person can post something informed in a few minutes.
- Lawful basis: it handles public news content only and no personal data about readers or customers, so there is no data protection case to make beyond that.
- Accountable owner: the person who posts. The tool proposes; they publish under their own name.
- Risk accepted: a wrong claim published under that name. It is reduced by the fact-check and by a person reading every draft, and it is not eliminated.
- What good enough means: set before measuring, in a pre-registered rule: the lower bound of recall on planted errors at least 80%, the upper bound of false alarms on clean sentences at most 15%, and, for green, at least 30 spot-check labels with agreement with the owner (kappa) of at least 0.6.
- A person reads the options, chooses one, edits it and posts it. Nothing is posted automatically.
- The editorial line is set by us: what counts as a usable fact, which stories matter, what the register should be. The tool has no view of its own to add.
- Claims the checker cannot settle are contested: they are shown to the owner with the judges' views and are never rewritten automatically.
- Every draft can be rated up or down, and a few claims at a time can be labelled with the checker's verdict hidden. Ratings are reviewed after a few weeks and used to change the instructions.
Each draft is broken into claims by a call that never sees the sources, with code checking that no number or strong qualifier was lost on the way. A judge then checks each claim against the cited article and must quote the words that settle it. Code tests what it says: a quote counts only if it is really in the source, every number must appear in a source, every link must be one of the draft's own sources. A claim the judge doubts is contested and goes to a person; nothing is rewritten without agreement, and with one judge there is none. The original draft and every verdict are kept.
Measured on 64 planted errors (eight kinds, in sentences taken from real articles) and 64 clean twins, under a rule written beforehand, on a fresh set of cases:
- Recall: it flagged 61 of 64 planted errors, 95% (95% CI 87% to 98%).
- False alarms: 5 of 198 clean factual sentences were flagged, 3% (95% CI 1% to 6%). 92% (95% CI 83% to 97%) of clean drafts were left entirely alone.
- Against the original check (one model call, no code checks), on the same cases: recall 95% (95% CI 87% to 98%); false alarms 0% (95% CI 0% to 2%).
- Of the errors it found, 0 were corrected automatically and 53 were contested (put to a person).
- No class of planted error fell below 50% recall.
How it got here. The first redesign did not pass. On the pre-registered test it found 91% (95% CI 81% to 96%) of planted errors but raised false alarms on 10% (95% CI 7% to 15%) of clean sentences, just over the limit, and the original, simpler check beat it (recall 97% (95% CI 89% to 99%), false alarms 1% (95% CI 0% to 3%)). The data showed why: the step that lists the claims was quietly repairing the errors it copied, and the second judge added no recall while doubling the false alarms. We fixed the first, applied a pre-written rule to the second (the second judge raised recall by only -1.6 points (needed 3), so it is removed and production runs one judge), and measured again on fresh cases. Both runs are on the measurement page.
Why not simply keep the original check? On these constructed errors it is as good: it found 95% (95% CI 87% to 98%) and raised false alarms on 0% (95% CI 0% to 2%). We use the redesigned pipeline for what that test does not score: quotes the code has verified against the source, a record of how much of each source was read and of every verdict, the original draft kept, and nothing rewritten without agreement. The original did none of that. That is a deliberate trade, and the human labels, once there are enough of them, are the test of it.
Against a person: 0 usable spot-check labels so far; 30 are needed before the sheet can say how often the checker and a person agree.
This is amber because: only 0 of the 30 human spot-check labels needed so far.
The draft text, links to every source it used, every claim with what each judge said and the words they quoted, how much of each source the checker could read, the original wording if a correction was made, and the open questions the draft raises. Past drafts stay in an archive. The raw news stream behind them is visible on the AI Signal page.
- It processes public news headlines, summaries and article text. It does not process personal data about customers or readers.
- Article text and drafts are sent to Anthropic's API (Claude models) to draft and to check.
- Drafts, source links, verdicts, quotes and ratings are stored on Cloudflare (Workers KV). They are kept until deleted; there is currently no automatic expiry.
- Rating, labelling and refresh actions are protected by private tokens. Reading the drafts and the measurement is public.
- The measurement covers the kinds of error we thought of. A real draft goes wrong in other ways, which only a person's labels can reveal, and those are still accruing.
- The test used articles whose text could be fetched. Where a source cannot be read (a paywall, a bot wall, a PDF), the checker knows only a summary or nothing, says so, and passes the claims to a person instead of correcting them. About a quarter of candidate articles could not be fetched when the test set was built, so in daily use a good share of claims will be contested for that reason alone.
- The second run reuses articles from the first, with different sentences and different planted errors. It is fresh cases, not fresh articles.
- The check can only test a draft against what the cited page says. It cannot tell whether the page is right.
- Ranking is rule-based and has missed important stories before. A cheap open model release once scored below routine product announcements. We corrected the scoring, and the daily set is a shortlist, not a census.
- It is not for publishing without a person reading it, and nothing it produces is legal advice.
- Model versions change. The measurement is re-run when the model, a prompt or the source format changes, and on a fresh set of cases if anything was adjusted after seeing results.
Why we publish this: it is the standard we would apply to a client's automation, applied to our own, including writing down what “good enough” meant before we looked, and reporting the run we did not pass. Read the full approach, or see the tool itself at AI Signal.