We ran 20 experiments to improve AI code review recall

We started with a concrete target: find more of the 136 known bugs in our evaluation set. After roughly 20 experiments, the best high-recall configuration moved our AI code review recall from about 53% to 62%.

The baseline found 72 bugs and produced 170 false positives. The high-recall configuration found 84 bugs and produced 328 false positives.

We found 12 additional bugs and almost doubled the amount of noise.

Those numbers forced us to separate what the finder discovered from what a developer would actually receive. Increasing the number of candidate findings was possible. Turning those candidates into a better review was much harder.

We were measuring the end of the pipeline

Our original benchmark looked mostly at the comments posted after the review pipeline had finished.

That is the output developers care about, but it hides where a bug disappeared.

A review comment passes through several stages before reaching a pull request:

Diff

↓

Finder

↓

Verification

↓

Anchoring and severity filters

↓

Delivered comment

The finder reads the pull request and raises candidate bugs. Verification investigates those candidates and removes findings that do not look convincing. Later stages assign severity, map the finding to a changed line, and decide whether it can be posted.

If a known bug does not appear in the pull request, any of those stages may be responsible.

We changed the benchmark to capture the candidate pool before the final delivery filters. We then measured the pool and the delivered review separately.

The vocabulary we used was small:

• Golden: a bug we knew existed in one of the evaluated pull requests.
• Pool: the findings available inside the pipeline before the final delivery filters.
• Delivered: the findings that survived and became review comments.
• Recall: the percentage of known bugs the reviewer found.
• Precision: the percentage of findings that were real bugs.
• F1: a combined measure of recall and precision.

Our golden set contained 136 labeled bugs from five real repositories. The judge matched findings against those known bugs and reported results at different stages and severity thresholds.

Once those stages were visible, “the reviewer missed the bug” stopped being a useful diagnosis.

A real bug can disappear in several places

The traces showed three recurring failure modes during the offline experiments. We later found a fourth while running the system in a self-hosted environment.

The finder reads the code and never raises the bug

This was the largest unresolved group.

The reviewer often opened the correct file and inspected the exact line containing the bug. It sometimes reasoned about the surrounding risk and raised other findings in the same region. The expected finding still never entered the pool.

The SSRF example from the previous article belongs to this group.

The reviewer saw a user-controlled URL passed to Ruby’s Kernel#open. It traced the value to the correct line and discussed nearby security concerns. It never recognized Kernel#open as a network sink, so no SSRF candidate was created.

More repository context would not have fixed that case. The relevant line was already available. The missing piece was recognition of language-specific behavior.

This distinction changed how we interpreted coverage. Our coverage ledger recorded whether the agent had read the relevant part of the pull request. Reading a line did not mean the agent had analyzed every behavior on that line.

A reviewer can achieve high file coverage while missing the bug classes that matter.

Verification rejects a valid finding

The finder sometimes raised the correct bug, and the verification stage removed it as speculative.

In one run, verification discarded seven serious findings, including an SSRF, a race condition, a FIPS bypass, and a null-related bug.

The verifier was treating uncertainty as evidence against the finding. If it could not confirm the bug confidently, it dropped it.

We changed that policy. Verification could discard a finding when it had a concrete reason to refute the claim. Uncertainty alone was no longer enough.

That change roughly doubled delivered recall in the relevant comparison. Removing a separate tendency to classify behavioral bugs as “opinion” recovered eight more findings.

The change increased recall because the filter stopped killing valid candidates. It also made the remaining precision problem more visible. A permissive verifier preserves more real bugs and more false positives.

The finding survives verification and fails anchoring

Some findings reached verification, received a high-confidence keep decision, and disappeared afterward.

The cause was anchoring.

A review comment must attach to a line that exists in the pull request diff. The reviewer may discover the problem in a related file or explain it using a line outside the changed range. If the pipeline cannot map the finding back to an eligible line, the platform cannot post the inline comment.

Our pipeline used a placeholder when this mapping failed. Later, the delivery stage silently discarded those findings with a path mismatch. They did not persist as suggestions, which made the loss difficult to see from the final output.

Fixing cross-file anchoring recovered 11 delivered true positives in one Gemini Flash evaluation.

Nothing about the model’s reasoning had improved. The system had already found and approved those bugs. We fixed the part that turned an internal finding into a valid pull request comment.

Delivery can make a working reviewer look empty

We discovered the fourth failure later, outside the golden-set experiments.

In a self-hosted run, the finder raised ten findings. The CLI delivered zero.

The severity classifier was using a broken fallback model that returned an empty result. The classifier then assigned its default severity, medium, to every finding. The delivery threshold was high, so the pipeline removed all of them.

The CLI showed an empty review even though the finder and verifier had done useful work.

This belongs to the production story in the next part of the series, but it changed how we read every benchmark result. An empty review can mean “no issues found.” It can also mean that the review failed after finding them.

Those states should never be indistinguishable.

Bigger models did not fix the system

One of the most obvious experiments was changing the model.

We ran configurations with Gemini Flash, Gemini Pro, GPT-5.4, and Claude Sonnet. The models behaved differently, found different bugs, and had very different costs. Their delivered F1 remained in roughly the same 0.33 to 0.40 range.

Gemini Flash cost about $0.29 per reviewed pull request in one paired evaluation. Gemini Pro ranged from $0.89 to $1.16. The Sonnet runs were far more expensive at the full-run level.

Paying for a larger model did not produce a corresponding improvement in delivered F1.

This does not mean model choice is irrelevant. A broken or poorly configured model can return invalid output, trigger retries, or produce an empty review. Different models also have different strengths on individual bug classes.

The experiments showed a narrower result: once the model was capable of running the review, changing model tier did not repair the architectural losses around it.

A more expensive finder can still have its valid findings removed by verification. A better candidate can still fail anchoring. A model with broader knowledge can still ignore a tool that the harness makes optional.

Prompt changes were mostly neutral

We tried several ways of changing how the finder was instructed to work.

We added examples. We opened up the reasoning. We changed coverage instructions. We added a challenge ledger and asked the model to reconsider areas it might have missed. We decomposed verdicts by hunk.

Most prompt changes produced no effect beyond normal run-to-run variation.

The hunk-level decomposition made the result worse, dropping from 36% to 31% in the relevant comparison.

The miss analysis explains part of this. The SSRF category was already present in the prompt. The reviewer still failed to connect Kernel#open with a network request.

Repeating the category or asking for more careful reasoning did not add the missing semantic knowledge.

Prompt work helps when the desired behavior is available to the model and the instructions are getting in the way. It helps less when the failure depends on a specific API behavior, a missing tool interaction, or a later pipeline stage.

The graph tool experiment never exercised the graph tool

We expected repository navigation to improve recall.

Kodus already had a persisted AST graph. We exposed the full graph through getCallers, improved the tool description, and ran the evaluation.

The result appeared to be zero improvement.

The traces showed a different failure: the agent never called getCallers. Across roughly 600 tool calls, usage of the graph tool was zero.

The agent kept using grep and readFile. A flattened call graph was already included in the prompt, so the model had little reason to request the structured version.

We had not measured whether graph navigation improved recall. We had measured whether the agent voluntarily changed its existing tool habits after receiving one additional option.

It did not.

A valid graph-navigation experiment needs to remove the static graph dump and make the structured tool the path for answering caller questions. Without that change, the treatment is never applied.

This is one of the easiest benchmark mistakes to make with agents. Adding a tool to the registry does not mean the model used it. Reading the source code confirms that the capability exists. The trace confirms whether the experiment happened.

More passes found more bugs and much more noise

The heavy configuration ran the finder several times and merged the candidate findings.

Independent runs can notice different parts of a pull request. Their union should increase coverage, and it did.

In one evaluation, the baseline found 72 of the 136 known bugs, about 53% recall. Heavy found 84, about 62%.

The false-positive count increased from 170 to 328.

Heavy added real coverage, but most of the additional output was not useful. Some candidates were paraphrases of the same bug. Others were distinct findings that did not survive human-style judgment.

The finder was no longer the only limit. The quality of the filter after it determined whether the additional recall could be used.

This is why pool recall can create the wrong impression. A larger pool looks like progress until someone measures what a developer would have to read.

Voting traded one metric for another

We also tested multiple verifiers per finding.

Each verifier tried to refute the candidate independently, and the majority decision determined whether the finding survived.

The result reduced false positives. It also removed real bugs.

In one run, the candidate pool contained 83 true positives and 294 false positives. After verification voting, 52 true positives and 159 false positives remained.

The filter became stricter. Precision improved while recall fell.

Consensus between models was useful as a confidence signal, but it did not act as an oracle. Several models can share the same blind spot, especially when they receive the same context and tools.

There is another kind of voting we have not fully tested: agreement across independent finder passes. A finding discovered in two of three passes may deserve more confidence than one found only once.

That experiment requires preserving which pass produced each finding. Our original heavy merge kept the union and discarded that attribution, so we could not measure frequency cleanly.

Coverage depth cost more without consistently helping

We tested a harder coverage policy that pushed the reviewer to inspect more of the pull request and perform recovery passes.

The result depended on the model.

With Gemini Flash, hard coverage increased pool recall from 0.374 to 0.529. Delivered performance moved in the opposite direction, from 34 true positives and 0.316 F1 under soft coverage to 31 true positives and 0.278 F1 under hard coverage.

Cost increased from about $0.29 to $0.45 per pull request.

With Gemini Pro, soft coverage performed better in the 32 pull requests completed by both configurations. Pool recall was 0.457 under soft coverage and 0.395 under hard coverage. Cost increased from $0.89 to $1.16 per pull request under the harder policy.

Reading more code did not reliably make the review better. The finder was not generally running out of steps. It was often reaching the relevant code and failing to recognize the bug.

We kept the softer, depth-first coverage policy. It matched or improved F1 while costing roughly 25% to 36% less in the paired evaluations.

The misses were not concentrated in large pull requests

We expected larger and denser pull requests to account for more misses.

The data did not support that explanation.

The correlations between a missed bug and pull request size, file count, or density were close to zero. Large pull requests create real review challenges, but they were not the main cause of the misses in this benchmark.

Severity was more informative. Low-severity bugs were missed more often, while critical bugs were detected at a higher rate. Removing low-severity labels improved measured recall, but serious misses remained.

The detailed traces kept returning to recognition.

The model saw the line. It sometimes discussed the correct region or broad risk. It failed to recognize the specific sink, side effect, or language behavior needed to raise the bug.

That explains why larger prompts and broader coverage produced limited gains. They addressed access to code while many misses happened after access had already been achieved.

Recall was constrained by precision

Our original framing treated recall as a finder problem. To find more bugs, make the finder more aggressive.

Heavy confirmed that this works at the candidate level. The finder raised more real bugs.

It also raised enough false positives to make the delivered review worse.

An aggressive finder needs a filter that removes noise without removing the extra bugs it was introduced to find. Our voting experiment did not provide that. A stricter verifier improved precision by giving back recall.

The practical limit on recall was the amount of candidate volume the later stages could verify safely.

This changed the order of work.

Increasing finder coverage was no longer the first task. Verification needed better evidence. Once the filter could distinguish a real behavioral bug from a convincing explanation, we could loosen the finder with less risk.

Why executable verification became the next bet

Our verifier used repository inspection, AST parsing, compiler checks, and model judgment. The final semantic decision still depended heavily on another model evaluating the original claim.

That creates a familiar failure mode. A second model can agree with the first model’s misunderstanding. A confidence score measures how convincing the finding appears, not whether the program behaves as claimed.

Some findings can be checked more directly.

A suspected shared-state bug can be tested by creating two instances and observing whether they share a collection. An unsafe regular expression can be exercised with a crafted input. A compiler can confirm a type claim. A data-flow bug may be reproduced with a focused request inside a sandbox.

The model still has work to do. It may need to write the check and interpret the output. The evidence comes from running the program or a deterministic tool.

This fits a BYOK product better than many N-pass strategies. Extra finder agents multiply inference and appear in the customer’s model bill. Executable verification adds a bounded amount of model work per candidate, while the check itself runs on sandbox compute that we control.

It will not apply to every review comment. Architectural concerns and team conventions often require judgment. Behavioral claims deserve a higher standard when the system can test them.

The benchmark itself failed several times

Some of the most useful findings had nothing to do with the review engine.

Agent evaluation has enough moving pieces that the benchmark can produce a convincing wrong answer.

We found several ways this happened:

• A treatment that moved five goldens was inside the normal noise floor. We observed swings of roughly six goldens between runs.
• Comparing runs from different sessions mixed configuration changes with provider and infrastructure drift.
• One repository could fail entirely because of GitHub or rate limits, making a configuration look worse even though it had reviewed fewer pull requests.
• Pull request numbers were assigned concurrently and changed between runs. Joining by PR number matched different source changes and corrupted cross-run comparisons.
• A benchmark flag set in a shell without being exported never reached the process. The run completed overnight without the intended treatment.
• The judge ran locally while the reviews ran on a remote machine. A sleeping laptop could kill scoring after the expensive part had finished.
• Looking at final CLI output hid working finder results when delivery was broken.

We added assertions after deployment, small preflight runs, stable joins based on source branch identity, and stage-level tracing.

The most useful measurement rule was simple: measure a change at the stage where it acts.

A finder experiment should be judged first on the candidate pool. A verifier experiment should be measured between candidates entering and leaving verification. Delivery bugs should not be diagnosed from finder recall.

Otherwise, several layers of unrelated variance sit between the treatment and the metric.

What we kept

A few changes survived the experiments and moved into the review engine.

Verification now requires a concrete refutation before discarding a candidate. It no longer removes a behavioral bug only because the claim seems speculative.

The finder is allowed to raise candidates more freely, leaving later stages responsible for filtering them.

Cross-file anchoring maps findings back to valid changed lines, which recovered 11 delivered bugs in one evaluation.

Soft, depth-first coverage replaced the more expensive policy. It produced similar or better F1 while reducing cost.

Heavy did not become the default. Its recall gain was real, and so was its false-positive problem.

We later brought it back as an opt-in alpha mode for users willing to trade more cost and noise for one or two additional unique bugs in a pull request. Before that could happen, we had to make it cheaper, wire the option through the actual product, and test it outside the golden set.

The experiment that failed still had a product use

Our benchmark answered whether heavy should become the default review path.

It should not.

It did not answer whether heavy could be useful when a user explicitly wanted a deeper review and accepted the trade-off.

We took it into a real self-hosted deployment next. The production run found more bugs, exposed three wiring failures that had made the option a silent no-op, and revealed a delivery bug capable of turning ten internal findings into an empty review.

It also gave us a way to remove duplicate candidates before the expensive verification stage and run independent passes in parallel.

Heavy had failed as a default. Production gave it a narrower job.

That is where the next part of this series starts.