I once watched a teammate clear fourteen AI review comments in about eight seconds. No reading, no expanding threads — just a rhythm of clicks down the sidebar until the PR went green. That same reviewer had caught a real null-deref three weeks earlier. It didn't matter. By then it had trained the whole team to treat its output as friction, not information.
That is the failure mode most AI code reviewers aren't built against. They compete on how many issues they can find. The question actually in front of them is whether anyone still reads the forty-first comment.
Trust, not raw accuracy, decides whether a reviewer is worth running at all — and trust doesn't decay on a smooth slope. Build the reviewer to protect the reader's attention and the accuracy tends to follow. Build it to maximize recall and you spend the attention you needed to be useful.
Trust degrades as a step function
The most useful mental model I've seen maps how developers behave at different false-positive rates:
- 0–10% false positives: developers investigate every finding.
- 10–30%: the tool is "noisy" but still read.
- 30–50%: findings are triaged with suspicion.
- 50%+: dismiss-by-default. People stop reading unless a comment blocks the merge.
For calibration: a mature static analyzer like SonarQube runs around a 3.2% false-positive rate after years of tuning, Semgrep around 12%. Untuned first-generation LLM reviewers land between 40% and 80% — inside, or past, the dismiss-by-default zone on day one.
The cruel part is what happens next. Once a tool crosses 50%, accuracy improvements alone stop recovering trust, because the team has already built the habit of ignoring it — recovery takes both a large accuracy jump and a visible signal that something changed. You don't win the room back by quietly shipping a better prompt. The cheapest trust to have is the trust you never spent, which makes the noise floor a launch requirement, not a tuning goal.
Recall is cheap. Attention is the budget you're spending.
The two numbers that matter are defined simply:
- Precision = TP / (TP + FP) — of everything flagged, what fraction is real.
- Recall = TP / (TP + FN) — of the real issues present, what fraction got caught.
A 50-PR benchmark makes the spread concrete. One tuned reviewer hit ~65% precision and ~55% recall; a popular alternative sat at 36% / 43%; a general coding assistant around 20% / 34%. Precision below ~40% is exactly the noise that produces the eight-second click-through.
The sharpest idea in that work is how "true positive" gets defined. A study across 200,000+ real pull requests treated developer behavior as ground truth: if a human changed the code after a comment, it counted; if the comment was ignored, it didn't. Precision becomes "how often did a developer actually act on this," not "how often was the model technically correct" — and only the first predicts whether the tool survives a real team.
That reframes the precision-versus-recall fight. You can run a reviewer hot on recall, but only if software, not the developer, pays for the false positives — hybrid setups that gate LLM findings through static analysis and re-checking have been reported to strip 94–98% of them while keeping recall high. Recall you can chase; the attention you burn on unverified guesses you don't get back.
Make the reviewer verify before it speaks
The design move that buys the most trust is refusing to emit a finding the system hasn't tried to disprove. Every finding carries a failure scenario and a verdict on whether that scenario was actually reproduced; anything that can't clear the bar is dropped, not downgraded to a vague "consider reviewing this."
// A finding the reviewer is allowed to surface
{
"summary": "off-by-one drops the last entry when the buffer is full",
"file": "RingBuffer.cs",
"line": 212,
"failure_scenario": "capacity=10, on the 11th append entry #1 is evicted but #11 is never written",
"verdict": "CONFIRMED", // vs PLAUSIBLE: did a check actually reproduce it?
"severity": "high"
}
Two rules make this work. First, failure_scenario is required — a finding that can't name the input and the wrong output it produces is a vibe, and vibes are where hallucinated bugs live. Second, the output is allowed to be empty. A reviewer that returns "nothing survived verification" on a clean PR is the strongest trust signal there is, because it proves the tool can not talk. Tools that must say something on every diff are the ones teams learn to mute.
Rank by blast radius, then cap the list
Even verified findings need triage; attention is finite per PR. A workable signal-to-noise framework sorts comments into tiers (critical: crashes, security, data corruption; important: architectural drift, measurable regressions; noise: style and subjective micro-optimizations) and scores the tool on signal ratio, the share landing in the first two tiers. Clear roughly 60% to be worth reading, 80% to be genuinely good. In one head-to-head the gap was stark: one tool posted 3 signal findings out of 14 comments (21%), another 7 out of 18 (61%). Same PRs, opposite verdicts on trust.
The practical consequence: a good reviewer caps its own output. Sort by severity, post the top few, hold the rest. Fourteen mixed-value comments get cleared in eight seconds; three confirmed high-severity findings get read. Under-reporting is a feature — the style nit you suppressed was never going to change the code, and it was exactly what trained your team to stop looking.
What to actually measure
Stop grading your reviewer on how many issues it finds. Grade it on the act-rate: the fraction of comments a developer actually changes code for. Watch it on the fortieth PR, not the first — trust is a repeated game. Once developers distrust a tool (in one 2025 survey, 46% actively distrusted AI-tool accuracy, 66% blaming "almost right, but not quite" output), even your technically-correct comments land as noise.
A trustworthy reviewer is a quiet one: it verifies before it speaks, it's allowed to say nothing, it caps what it surfaces to what a human can act on, and it grades itself by what got fixed. Build for the reader's attention first. Recall is the easy half of the problem — and the wrong half to optimize alone.
Sources: Deep Code Review: Why Recall Beats Precision for Agents (Augment Code), Diagnosing False Positives in AI Code Review (BDIGITAL), A Framework to Measure Signal vs. Noise in AI Code Review (DocMason), AI Code Review Benchmark: Precision, Recall, and F1 from 200,000 PRs (CodeAnt)