What makes an AI code review agent actually work

If everyone can use the same models, why are some AI code reviews better?

Finding the right code was not the issue in one of our evaluations. A pull request passed a user-controlled URL to Ruby’s Kernel#open method. The reviewer inspected the right part of the diff and raised several findings nearby. It even traced the URL back to user input. It still never identified the SSRF.

The failure was easy to describe after the fact. The model did not recognize Kernel#open as a network sink in that context. Giving it more of the repository would not have helped. It already had the relevant code. Asking it to inspect the diff more carefully would probably have produced another confident pass over the same lines.

This is the kind of miss that changed how we think about AI code review.

Model choice matters, of course. We have seen different models produce very different results on the same pull request. But the model runs inside a larger system that decides what it sees, what it can inspect, how long it can work, and which findings survive long enough to reach a developer.

That system is usually called a harness.

The term comes from a test harness: the code around a component that sets it up, exercises it, and checks what happened. In an AI agent, the harness builds the prompt, provides tools, feeds tool results back to the model, manages the context window, and decides when the run is finished.

For code review, it also has to answer two difficult questions:

• How do we check that a finding is real before posting it?
• How does the reviewer learn from findings that developers accepted, ignored, or rejected?

A model can be good at reasoning about code and still perform badly when the system around it makes poor decisions. It may receive a static summary when it needs to follow a call chain. It may find a valid bug that a later classifier removes. It may produce ten versions of the same finding and spend an expensive verification call on every one. It may learn nothing when the team repeatedly rejects the same category of comment.

These are harness problems. In practice, they often matter as much as the model itself.

The reviewer is a loop

The simplest version of an AI code reviewer looks like a single request:

diff + instructions → model → review comments

That can work for small, local changes. It breaks down once the answer depends on code outside the diff.

Imagine a pull request that adds a call to should_block?. The changed file looks harmless, but the method increments a counter even when it decides to do nothing. A reviewer needs to find the method definition, inspect its side effects, understand the caller, and decide whether those effects are valid in this path.

A single prompt can include more context, but repository context grows quickly. Dumping more files into the prompt also makes it harder for the model to decide where to look.

An agentic reviewer works as a loop:

At each step, the model can read a file, search for a symbol, inspect references, run a compiler, or submit its final findings. The harness controls which tools are available and how their results enter the next step.

This sounds like a small architectural difference. It changes the task the model is being asked to perform. The model no longer has to solve the review using only the context someone guessed it would need beforehand. It can gather evidence as it works.

The quality of that evidence depends on the available tools.

A textual search can find a method with a distinctive name. It is much less useful when the relevant relationship is structural: a caller two hops away, an inherited method, an interface implementation, or a dependency that crosses package boundaries. In those cases, the reviewer needs a way to follow the code’s structure.

We had already built an AST graph covering multiple languages. The index was persisted and available to the review system. But the default agent could not query that graph during its reasoning. We flattened part of it into text and added the text to the prompt. The getCallers tool that would let the agent traverse it was disabled.

The infrastructure existed. The useful interaction did not.

This distinction appears repeatedly when working on AI agents. Having the data somewhere in the pipeline does not mean the model can use it at the moment it needs it. A static context dump and an on-demand tool can contain the same information while producing very different behavior.

Code review gives the harness a different job

General coding agents and code review agents share many components. Both need tools, context management, project instructions, permissions, and a stopping condition.

Their operating conditions are different.

A coding agent can ask the user a question, modify a file, run the tests, inspect the failure, and try again. Its work may continue across a long session. Persistence and recovery matter because the agent is changing state over time.

A code reviewer usually gets one pass over a pull request. It needs to inspect the change, produce a small set of useful findings, and stop. Developers will judge it by the comments they see, especially the wrong ones.

That makes three parts of the system dominate the result:

1. The loop that decides how the review proceeds.
2. The context and retrieval tools available inside that loop.
3. The verification between a candidate finding and a posted comment.

The last part is where code review becomes a different problem.

A coding agent can discover that its patch was wrong when the tests fail. A reviewer can produce a convincing explanation for a bug that does not exist. If the system posts that explanation directly, the developer has to disprove it.

Many review systems add a second LLM call here. The second model reads the finding, inspects some code, and gives it a confidence score. We use a version of this pattern too, alongside AST parsing and compiler checks.

It helps, but it has a hard limit. A model judging another model can repeat the same misunderstanding. Confidence is also weak evidence when the underlying question can be tested.

The better standard is a directed check: generate a small reproduction, run a focused test, follow the relevant data flow, or ask a deterministic tool whether the claimed behavior is possible. The exact proof depends on the finding. Some review comments will always require judgment, but many claims about program behavior can be checked more directly than asking an LLM how confident it feels.

That shift, from reviewing an explanation to testing a claim, became one of the main directions in our work on the Kodus review engine.

The parts that matter most in code review

General-purpose agent harnesses tend to converge on the same set of components: an iteration loop, context management, tools, subagents, project instructions, session persistence, hooks, permissions, and some way to stop.

Code review changes the weight of those components.

A coding agent may work for hours, modify dozens of files, recover from failed tests, and continue after its context has been compressed. Session persistence matters because there is a long-running task to preserve.

A reviewer has a shorter job. It receives a pull request, investigates the change, produces a limited set of findings, and exits. The quality of that run depends mostly on whether the agent can reach the relevant code and verify what it found.

Four parts carry most of that work:

• the loop that controls how the review proceeds;
• the context the reviewer can retrieve while reasoning;
• the tools it can use to inspect the repository;
• the verification that stands between a candidate finding and a posted comment.

Memory sits alongside those parts. It may not change the first review of a repository, but it should change the tenth. If developers keep rejecting the same category of comment, the reviewer should eventually stop making it. If a team repeatedly fixes a domain-specific issue, the reviewer should become more likely to look for it.

The remaining parts still matter operationally. Permissions, timeouts, concurrency limits, observability, and project instructions can all break a run. They usually do not explain why one reviewer understands a change better than another when both complete successfully.

Repository context needs to arrive at the right moment

Most code review bugs are easy to explain after someone has already found them. The difficult part is locating the code and relationships needed to recognize the problem.

A diff may show a new method call without showing its implementation. The implementation may depend on an inherited method, a configuration value, or a caller two levels higher. Loading every related file before the review begins is expensive, and the system has to guess which relationships will matter.

There are several common ways to provide repository context:

• textual search finds names and literal patterns;
• embeddings retrieve files with similar meaning;
• an AST or code graph exposes structural relationships;
• LSP operations resolve definitions, implementations, references, and types.

These methods solve different retrieval problems.

Text search works well when the reviewer knows what string to look for. Embeddings help when the relevant code uses different names for related concepts. A code graph helps when the answer depends on who calls a function or which implementation satisfies an interface. LSP diagnostics can expose type errors and language-specific behavior that a model may miss.

Combining them into one large context block wastes much of their value. The reviewer needs to choose the right operation while investigating a specific claim.

Consider a method called should_block?. A textual search can locate its definition. It may also return tests, comments, and unrelated methods with similar names. A `findReferences` or `getCallers` operation answers a narrower question: which paths can reach this method, and what happens after it returns?

The result can be smaller than a repository summary while carrying more useful information.

This was one of the gaps we found in our own system. Kodus already had a persisted AST graph covering several languages. Part of that graph was flattened into text and added to the prompt, while the tool that allowed the agent to traverse it was disabled.

From the outside, we could say that the reviewer had code graph context. Inside the loop, the model still relied mainly on `grep` and file reads.

A repository index becomes much more useful when the reviewer can query it after forming a hypothesis. The sequence matters:

1. The reviewer sees a suspicious call in the diff.

2. It decides that the callers or implementation may change the conclusion.

3. It asks for that structural relationship.

4. The result enters the next reasoning step.

5. The reviewer either keeps the candidate finding or drops it.

The agent can spend its context budget on evidence related to the current hypothesis. It also leaves a trace of why a file or symbol was inspected, which makes the review easier to debug later.

More agents can repeat the same blind spot

Running several agents in parallel is an appealing way to improve coverage. One looks for bugs, another checks security, and another focuses on performance. Their findings are merged at the end.

This can work when the agents have distinct instructions, tools, or context. It can also multiply the same failure.

The SSRF miss from the beginning of this article is a good example. The reviewer reached the right line and understood that the URL was controlled by a user. Its missing piece was language-specific knowledge about `Kernel#open`.

A separate security agent using the same model, prompt knowledge, and tools may make the same mistake. Giving the agent a security label does not teach it that a particular Ruby API can open a network connection.

The research on multi-agent software systems gives a more useful distinction. Dividing work by function can help because each agent receives a narrower operational job. One agent locates the relevant code. Another investigates a candidate. A later stage tries to disprove the claim.

That decomposition changes the evidence available at each step. A thematic split often changes only the list of issues the model is asked to consider.

There is also a direct cost. Three independent review agents can consume roughly three times the inference of one agent before verification and deduplication. Parallel execution may contain the latency when the provider allows enough concurrent requests, but the token bill still grows.

This matters more in a BYOK product. Kodus customers pay the model provider directly, so an architectural decision that adds inference appears in their bill. We cannot hide the cost inside a fixed seat price.

Extra passes make sense when they find bugs that one pass consistently misses and when the later stages can handle the additional candidates. Without a reliable filter, extra agents increase the amount of review output faster than they increase the amount of useful review output.

Verification needs evidence from outside the original claim

A candidate finding normally contains a file, a line, and an explanation of what may go wrong. At that point, the system has several ways to decide whether the developer should see it.

The simplest option is to trust the generating model. A safer option sends the finding to another model and asks it to inspect the repository before assigning a confidence score. Compiler checks and AST parsing can reject suggestions that produce invalid code.

Each layer removes some bad findings. None guarantees that the claimed runtime behavior is possible.

Two models may share the same misunderstanding about a library. A compiler can confirm that the code is valid while saying nothing about a race condition or an authorization bypass. An AST parser can validate a proposed patch without validating the bug that motivated it.

A directed check starts from the claim and asks what evidence could confirm or refute it.

For a shared-state bug, the reviewer might generate a small test that creates two instances and observes whether their arrays remain isolated. For an unsafe regular expression, it could run a crafted input and measure the behavior. A suspected type error can be passed to the native compiler. A data-flow claim may require following a value from a source to a sink.

The check does not need to prove every property of the program. It needs to test the part of the finding that can be observed.

A second model can agree with a finding. A directed check can produce evidence about the program’s actual behavior.

Executable verification has its own limits. Some architectural findings depend on team intent. A naming comment cannot be proven in a sandbox. Reproducing a concurrency bug may require an environment that is too expensive to create during every review.

The harness can choose where the extra work is justified. High-impact behavioral claims deserve more verification than a low-severity maintainability suggestion. The system can also use deterministic tools before spending another model call.

That creates a more useful ordering:

1. Reject candidates that fail basic grounding checks.

2. Use compilers, parsers, diagnostics, and existing tests where they apply.

3. Generate a directed check for behavioral claims that justify the cost.

4. Use model judgment for the residual cases that cannot be tested directly.

The order affects both cost and accuracy. Running an expensive model with repository tools on every duplicate candidate wastes money. Running a cheap deterministic check first may remove the candidate without another inference call.

The reviewer should learn from rejected comments

Repository context tells the reviewer how the code works. Team memory tells it which findings are useful in that repository.

Most review tools support some form of custom instruction. A team can add rules in a configuration file, write natural-language policies, or import instructions from files such as AGENTS.md, CLAUDE.md, and IDE-specific rule formats.

Those rules change what the reviewer looks for, but they require someone to write or import them.

The review itself produces another source of information. Developers react to comments, reply to them, implement suggested changes, or merge the pull request without addressing them. Over enough reviews, those actions reveal patterns.

A category that is repeatedly ignored may be too generic for that team. A suggestion that developers frequently implement may encode a repository-specific invariant worth checking earlier. A finding that receives negative feedback may need to be suppressed or rewritten.

Kodus already persists reactions and implementation status. We also inject manual rules, imported IDE rules, and memory rules into the agents. The missing part is a continuous path from review outcomes to the next review.

Generating a rule from every accepted comment would add noise. Many accepted suggestions describe issues that a compiler already catches or behavior the model would find without extra instructions. Repeated rules consume context and compete for the model’s attention.

A candidate memory should earn its place.

One useful test is ablation: run the relevant example without the proposed rule. If the reviewer already finds the issue, adding the rule contributes little. The next check is whether a deterministic tool handles it more reliably. A compiler or linter should own mechanical findings when possible.

The remaining rules are the valuable ones. They describe something specific to the repository that the model misses without help and that existing tools do not enforce.

Path scope matters too. A rule about domain-layer imports does not need to appear in a pull request that only changes frontend assets. Retrieving a small set of relevant memories during the review gives the model less material to sort through.

Feedback from failure deserves equal attention. A reviewer that learns only from implemented suggestions becomes better at repeating successful categories. It does not learn which comments waste the team’s time.

Security findings need different suppression rules because repeated dismissal does not necessarily make the risk irrelevant. Memory needs policy around what can be suppressed, how much evidence is required, and when an old pattern should be reconsidered.

Cost changes which techniques make sense

The common ways to increase review coverage all consume additional inference.

A second agent adds another trajectory. Resampling runs the same task several times and merges the results. Cross-model voting sends candidates through more providers. A verification agent may investigate every finding with its own tool loop.

These techniques can improve results. Their cost grows with the number of runs or candidate findings.

For a product with bundled inference, the vendor absorbs that cost and prices it into the subscription. With BYOK, the team running the review sees the model usage directly. A configuration that doubles review depth can roughly double part of the customer’s inference bill.

Token cost and latency also behave differently. Independent passes can run in parallel if the provider and the customer’s rate limits allow it. This can keep wall-clock time close to one long pass while total token usage continues to increase. A provider with a concurrency limit of one will serialize the same work.

That makes the order of operations important.

Suppose three finder runs produce twelve candidates, but four are paraphrases of the same bug. If verification runs before deduplication, the system pays to investigate all twelve. Deduplicating the candidates first may reduce the verification work without changing how many distinct bugs were found.

Caching can also reduce repeated input cost when several passes share the same repository context and instructions. It does not reduce the number of output tokens or tool calls, and the savings depend on each provider’s cache behavior.

A review system should be able to attribute cost to a run. Input tokens, cached tokens, output tokens, and tool-driven follow-up calls need to be connected to the pull request. Dividing that cost by the number of accepted findings gives a much more useful measure than total monthly token usage.

Without that instrumentation, architectural decisions about extra agents and verification are based on estimates.

Model choice still matters

None of this makes models interchangeable.

In our evaluations, different models found different bugs on the same golden set. They also behaved differently with temperature, tool use, and long contexts. Some models were better at generating candidates, while others were more useful in verification.

The surrounding system determines whether those capabilities reach the developer.

A model with good repository reasoning cannot follow a call graph if the tool is unavailable. A model that finds more bugs may produce a worse final review when the verification layer cannot separate the additional true findings from false positives. A cheaper model can become expensive when the harness repeatedly sends it duplicate work.

The ability to change models is still valuable because the best option will vary by repository, task, and cost ceiling. The harness should make those differences observable rather than assume one provider will remain the best choice for every review.

This is also why model benchmarks need to measure the whole path. Scoring only the comments that reach the pull request hides whether the finder missed the bug or a later stage removed it. Scoring only the raw candidate pool hides the false positives developers would have to read.

Both views are needed.

We needed to measure where each bug disappeared

Once we mapped the review as a pipeline, changing the model stopped being the default response to every miss.

A bug could disappear because the finder never raised it. It could be raised and then rejected during verification. A severity classifier could assign the wrong level, causing a later filter to remove it. The finding could survive internally and still fail to appear in the final CLI or pull request output.

Looking only at posted comments made all of those failures look the same.

We changed our evaluation setup to capture the candidate pool before the later filters. The judge could then measure recall and precision at several stages and across severity thresholds.

That gave us a more useful set of questions:

• Did the finder ever identify the golden bug?
• Did verification preserve or reject it?
• How many false positives were added to gain that recall?
• Did the finding survive severity filtering?
• How much did each experiment cost per pull request?

We tested more passes, different models, stricter coverage instructions, voting, graph access, and changes to verification. Some increased raw recall while making the delivered review worse. Others raised cost without changing which bugs reached the developer.

The gap between candidate quality and delivered quality became larger than we expected.

The next article covers those experiments, including where the bugs died and why finding more of them did not automatically produce a better review.