Open Source LLMs for Code Review: 2026 Benchmark
Edvaldo Freitas
For a lot of teams, getting started with AI code review is easy: connect a tool to the repository and pick a model like Claude or GPT.
The trouble starts when the code can’t leave the company. With a closed model, every pull request you review goes to the servers of whoever provides the model, and not every company can accept that, whether because of customer contracts or security policy.
In those cases, the way out is an open source model, with weights you can download and run inside the company. The question is how much quality that costs you, because reviewing a PR means the model has to read code it didn’t write and find bugs that don’t always show up in the diff.
At Kodus, we built an LLM code review benchmark that compares models by having them review real pull requests from open source projects. Here, we show how the seven open source models we tested did.
Updated: November 14, 2026
TL;DR:
- We evaluated seven open source LLMs: Kimi K3.1 Pro, GLM-5.3, DeepSeek V4.1 Flash, GLM-5.3 Flash, DeepSeek V4 Pro, Qwen3.8 27B and MiniMax M3.
- The models reviewed 30 real pull requests from five open source repositories: cal.com (TypeScript), Discourse (Ruby), Grafana (Go), Keycloak (Java) and Sentry (Python).
- Kimi K3.1 Pro found more bugs than any closed model in the test: 83 of 121 (68.6%), versus 52 for GPT-6.1 Sol, the closed model that found the most.
- GLM-5.3 and DeepSeek V4.1 Flash landed within 3 points of recall of GPT-6.1 Sol and Claude Opus 5.5, a gap that sits inside the test’s margin of error.
- On precision, the top three closed models stay ahead. The most precise open model, Qwen3.8 27B (51.5%), trails GPT-6.1 Sol (62.7%) and Claude Sonnet 5.5 (57.0%).
What is the best open source LLM for code review?
In our test, the best open source LLM for code review was Kimi K3.1 Pro, which also had the best result of all 13 models, closed ones included. It found 83 of the 121 confirmed bugs.
It isn’t right for every team, though, because it’s big, costs more per PR and posts a lot of comments. So we also picked out the models that make more sense in other situations:
- Kimi K3.1 Pro: for finding the most bugs, if you have the infrastructure for a 2.8-trillion-parameter model.
- DeepSeek V4.1 Flash: for teams that want an MIT license and low cost, with recall close to GPT-6.1 Sol and Claude Opus 5.5.
- GLM-5.3: for teams that plan to use heavy mode, where it tied GPT-6.1 Sol on bugs found.
- GLM-5.3 Flash: for the lowest cost per bug found in the entire test.
- Qwen3.8 27B: for teams with limited infrastructure that prefer fewer comments, with more of them right.
Open source models for code review compared
| Model | Company | License | Recall | Bugs found | Precision | F2 | Cost per PR |
|---|---|---|---|---|---|---|---|
| Kimi K3.1 Pro | Moonshot AI | Kimi K3.1-Pro License | 68.6% | 83 of 121 | 49.1% | 63.6 | $1.244 |
| GLM-5.3 | Z.ai | glm-5.3 | 41.3% | 50 of 121 | 43.5% | 41.7 | $1.171 |
| DeepSeek V4.1 Flash | DeepSeek | MIT | 40.5% | 49 of 121 | 43.4% | 41.0 | $0.134 |
| GLM-5.3 Flash | Z.ai | MIT | 35.5% | 43 of 121 | 39.8% | 36.3 | $0.098 |
| DeepSeek V4 Pro | DeepSeek | MIT | 32.2% | 39 of 121 | 47.6% | 34.5 | $0.490 |
| Qwen3.8 27B | Alibaba | Apache 2.0 | 28.1% | 34 of 121 | 51.5% | 30.9 | $0.198 |
| MiniMax M3 | MiniMax | minimax-community | 19.8% | 24 of 121 | 38.1% | 21.9 | $0.154 |
Source: LLM code review benchmark leaderboard, published November 7, 2026.
How we tested the models
Each model reviewed the same 30 pull requests from five open source projects: cal.com (TypeScript), Discourse (Ruby), Grafana (Go), Keycloak (Java) and Sentry (Python). Those PRs contain 121 confirmed bugs, and a judge model, Claude Haiku 5.5, checks whether each comment points to one of them.
Recall shows how many of those bugs the model found, and precision shows how many of its comments were right.
The models ran in two modes:
Balanced: a single agent reviews the PR and posts the comments. This is the mode used in the table above.
Heavy: the same agent works alongside nine specialists, covering areas like security and performance, and all of their findings become a single list of comments.
The open models ran through the vendor’s API or through providers that host the weights, such as Fireworks and AkashML. On your own infrastructure, with different quantization, the results may change.
Are open source models as good as closed models for code review?
In terms of bugs found, the best open source models landed at the same level as the closed ones, and Kimi K3.1 Pro beat all of them. For reference, these were the three closed models with the best results in the same test:
| Closed model | Recall | Bugs found | Precision | Cost per PR |
|---|---|---|---|---|
| GPT-6.1 Sol | 43.0% | 52 of 121 | 62.7% | $0.787 |
| Claude Opus 5.5 | 42.2% | 51 of 121 | 52.6% | $0.715 |
| Claude Sonnet 5.5 | 37.2% | 45 of 121 | 57.0% | $0.542 |
Kimi K3.1 Pro found 31 more bugs than GPT-6.1 Sol, the closed model that found the most. Since each model ran only once, every recall number comes with a margin of error, with confidence intervals of 10 to 25 points. Even accounting for that margin, Kimi’s lead holds, because the lower bound of its interval (59.0%) sits above the upper bound of every closed model.
GLM-5.3 and DeepSeek V4.1 Flash came within one, two or three bugs of GPT-6.1 Sol and Claude Opus 5.5, which, given that margin, is a tie. For us, this is the most important finding in the article, because it shows that a team that needs an open model, whether for contractual or security reasons, can find as many bugs as it would with the best closed models.
Closed models still win on precision. The top three got between 52.6% and 62.7% of their comments right, and no open model went above Qwen3.8 27B’s 51.5%. If you swap Claude Sonnet 5.5 for DeepSeek V4.1 Flash, for example, you’ll find a similar number of bugs, but your team will spend more time dismissing suggestions that go nowhere.
Heavy mode tells the same story. GLM-5.3 found 67 bugs, matching GPT-6.1 Sol in that mode, and Qwen3.8 27B reached 52, the same number GPT-6.1 Sol found in balanced mode.
How each model did at code review
1. Kimi K3.1 Pro: the open source model that finds the most bugs
Kimi K3.1 Pro, from Moonshot AI, had the best result in the entire benchmark. It was what surprised us most in the test, because an open model led by a wide margin, ahead of every closed one.
| Kimi K3.1 Pro | |
|---|---|
| License | Kimi K3.1-Pro License |
| Parameters | 2.8 trillion total, 104 billion active per token |
| Recall (balanced) | 68.6% (83 of 121) |
| Precision (balanced) | 49.1% |
| Recall (heavy) | 67.8% (82 of 121) |
| Cost per PR (balanced) | $1.244 |
It found more bugs than any other model in all five repositories and caught 42 of the 52 high-severity bugs, versus 28 for Claude Opus 5.5. On the other hand, it comments a lot. It posted 169 comments across the 30 PRs, and 86 of them didn’t point to a bug in the answer key, so plan for more triage time.
It’s also the biggest model on the list, with 2.8 trillion parameters, and running it on your own infrastructure takes a lot more hardware than any other open model we tested.
The Kimi K3.1 Pro license lets you use, modify and distribute the model. The extra conditions only apply to companies that offer the model as a service to third parties with more than $20 million in revenue over 12 months, or to products with more than 100 million monthly active users or more than $20 million in monthly revenue, which have to display the model’s name. If you’re only using Kimi to review your own company’s code, none of those conditions apply.
When to consider Kimi K3.1 Pro: when finding as many bugs as possible is the priority and your team has the infrastructure for a model this size, knowing it’ll be reviewing more comments.
2. GLM-5.3: the open model closest to GPT and Claude
GLM-5.3, from Z.ai, came in fourth of the 13 models, right behind GPT-6.1 Sol and Claude Opus 5.5, and was the open model that got closest to them.
| GLM-5.3 | |
|---|---|
| License | glm-5.3 |
| Parameters | 753 billion |
| Recall (balanced) | 41.3% (50 of 121) |
| Precision (balanced) | 43.5% |
| Recall (heavy) | 55.4% (67 of 121) |
| Cost per PR (balanced) | $1.171 |
In heavy mode, it tied GPT-6.1 Sol on bugs found (67) and caught more critical bugs than any other model, three of the four in the answer key. Its precision, however, dropped to 36.6%, well below OpenAI’s 55.4%.
At the Fireworks price we used in the test, each bug found in balanced mode cost $0.702, almost double Claude Sonnet 5.5 ($0.361). If you host the model yourself, that math comes down to your hardware.
GLM-5.3 has 753 billion parameters, and its license only adds one extra condition, for companies that offer the model as a service and bring in more than $10 billion over 12 months, which have to pass a security review by Z.ai.
When to consider GLM-5.3: when your team plans to use heavy mode, where it found 67 bugs and three of the four critical ones, and has the infrastructure for a 753-billion-parameter model.
3. DeepSeek V4.1 Flash: the best value among open models
If we had to recommend a single open model for most teams, it would be DeepSeek V4.1 Flash. It found almost as many bugs as GLM-5.3 at about a ninth of the cost per PR, and it’s MIT-licensed.
| DeepSeek V4.1 Flash | |
|---|---|
| License | MIT |
| Parameters | 552 billion in the backbone, with 8 billion active during prefill and 16 billion during generation |
| Recall (balanced) | 40.5% (49 of 121) |
| Precision (balanced) | 43.4% |
| Recall (heavy) | 52.9% (64 of 121) |
| Cost per PR (balanced) | $0.134 |
Through DeepSeek’s official API, each bug found cost $0.082, the second-lowest in the test. In heavy mode, it reached 64 bugs, more than Claude Opus 5.5 in the same mode (60), at $0.34 per PR.
Despite its 552 billion parameters, it only activates a small fraction of them per token, so it needs less compute than its size suggests, although all the weights still have to fit in memory. The DeepSeek V4.1 Flash MIT license only asks you to keep the copyright notice.
Its weakest result came from Sentry, the test’s Python project, where it found only 2 of the 15 bugs. That’s a single project, so you can’t draw conclusions about the language, but if your team writes a lot of Python, we recommend testing the model on your own PRs before adopting it.
When to consider DeepSeek V4.1 Flash: when your team wants an open model with no license restrictions and a low cost per review.
4. GLM-5.3 Flash: the cheapest model per bug found
GLM-5.3 Flash is the smaller version of GLM-5.3, with 320 billion parameters and an MIT license, and it was the cheapest model per bug found in the entire benchmark, closed models included.
| GLM-5.3 Flash | |
|---|---|
| License | MIT |
| Parameters | 320 billion, with 18 billion active per token |
| Recall (balanced) | 35.5% (43 of 121) |
| Precision (balanced) | 39.8% |
| Recall (heavy) | 47.9% (58 of 121) |
| Cost per PR (balanced) | $0.098 |
Each bug found cost $0.069. Compared with GLM-5.3, it found seven fewer bugs and cost less than a tenth as much per PR, a trade-off that makes sense for teams reviewing a lot of PRs.
The low price comes with more wrong comments. Its precision was 39.8%, the second-lowest among the open models, and 65 of the 108 comments it posted didn’t point to a bug in the answer key.
When to consider GLM-5.3 Flash: when your PR volume is high and your team is willing to filter more comments in exchange for the lowest cost per bug.
5. DeepSeek V4 Pro: DeepSeek’s bigger model, with fewer bugs found than Flash
DeepSeek V4 Pro is DeepSeek’s bigger model, with 1.6 trillion parameters and 49 billion active, also under an MIT license. Despite being bigger, it found fewer bugs than V4.1 Flash.
| DeepSeek V4 Pro | |
|---|---|
| License | MIT |
| Parameters | 1.6 trillion, with 49 billion active per token |
| Recall (balanced) | 32.2% (39 of 121) |
| Precision (balanced) | 47.6% |
| Recall (heavy) | 47.9% (58 of 121) |
| Cost per PR (balanced) | $0.490 |
In balanced mode, it found 10 fewer bugs than Flash at 3.7 times the cost per PR. In heavy mode, it had the biggest recall gain in the entire test, 15.7 points, and still finished behind Flash in that mode, with 58 bugs versus 64.
When to consider DeepSeek V4 Pro: when your team already runs the model for another reason and wants to reuse that infrastructure. If you’re choosing now, V4.1 Flash found more bugs and cost less in both modes, although V4 Pro has slightly higher precision.
6. Qwen3.8 27B: the best model for limited infrastructure
Qwen3.8 27B, from Alibaba, is the smallest model on the list. It has 27 billion parameters, under an Apache 2.0 license, and fits on far less hardware than the others, which makes it the easiest to host.
| Qwen3.8 27B | |
|---|---|
| License | Apache 2.0 |
| Parameters | 27 billion (dense model) |
| Recall (balanced) | 28.1% (34 of 121) |
| Precision (balanced) | 51.5% |
| Recall (heavy) | 43.0% (52 of 121) |
| Cost per PR (balanced) | $0.198 |
It commented less and got more right than the other open models. Just over half of the 66 comments it posted pointed to a confirmed bug, the highest precision among the open models. In balanced mode, it found fewer bugs than the others, except MiniMax M3, but in heavy mode it reached 52, the same number GPT-6.1 Sol found in balanced mode.
In the test, it ran in fp8, with its weights quantized to 8 bits.
When to consider Qwen3.8 27B: when your available infrastructure is limited and your team prefers fewer comments, with more of them right.
7. MiniMax M3: the weakest open source model at code review
MiniMax M3 has 428 billion parameters, with 23 billion active, and finished last among the open models.
| MiniMax M3 | |
|---|---|
| License | minimax-community |
| Parameters | 428 billion, with 23 billion active per token |
| Recall (balanced) | 19.8% (24 of 121) |
| Precision (balanced) | 38.1% |
| Recall (heavy) | 28.1% (34 of 121) |
| Cost per PR (balanced) | $0.154 |
It found fewer bugs than Qwen3.8 27B, a much smaller model, and each bug cost $0.193, more than with DeepSeek V4.1 Flash or GLM-5.3 Flash.
The MiniMax M3 license is also the most restrictive on the list. For commercial use, it requires the phrase “Built with MiniMax M3” somewhere visible in the product, an email notice to MiniMax for companies with up to $20 million in annual revenue, and prior written authorization above that.
When to consider MiniMax M3: based on the current results, we wouldn’t recommend MiniMax M3 as a first choice, because other open models find more bugs and have simpler licenses. It’s on the list for teams that already use it and want to know how it does at code review.
Which open source LLM should you choose for code review?
If your priority is finding as many bugs as possible
Kimi K3.1 Pro found the most bugs in the test, open and closed models alike. It makes sense if your team has the infrastructure for a 2.8-trillion-parameter model and is willing to review more comments.
If your priority is cutting costs
DeepSeek V4.1 Flash had recall close to the best closed models at $0.134 per PR. If your budget is even tighter, GLM-5.3 Flash cost less per bug found, with slightly lower recall.
If your infrastructure is limited
Qwen3.8 27B runs on far less hardware than the others and had the highest precision among the open models. It finds fewer bugs in balanced mode, but reached 52 in heavy mode, the same number GPT-6.1 Sol found in balanced mode.
If your license can’t come with extra conditions
DeepSeek V4.1 Flash, DeepSeek V4 Pro and GLM-5.3 Flash use the MIT license, and Qwen3.8 27B uses Apache 2.0. All four can be used commercially without an agreement with the vendor.
Can you use open source models commercially?
Yes, but the license varies from model to model, so read each one before putting the model into production. For the seven models in the test, the main conditions are:
| Model | License | Main condition |
|---|---|---|
| DeepSeek V4.1 Flash | MIT | Keep the copyright and license notice |
| DeepSeek V4 Pro | MIT | Keep the copyright and license notice |
| GLM-5.3 Flash | MIT | Keep the copyright and license notice |
| Qwen3.8 27B | Apache 2.0 | Keep license notices and mark changes in modified files |
| Kimi K3.1 Pro | Kimi K3.1-Pro License | An agreement with Moonshot AI for companies offering the model as a service with more than $20 million in revenue over 12 months |
| GLM-5.3 | glm-5.3 | A security review by Z.ai for companies offering the model as a service with more than $10 billion in revenue over 12 months |
| MiniMax M3 | minimax-community | Visible attribution plus notice to, or authorization from, MiniMax for any commercial use |
For a team that only uses the model to review its own code, without offering it to third parties, the Kimi K3.1 Pro and GLM-5.3 conditions don’t apply. The MiniMax M3 license defines commercial use more broadly, so read the full text before using it in a company setting.
Strictly speaking, Kimi K3.1 Pro, GLM-5.3 and MiniMax M3 are open-weight models, because the weights are open but the license has its own conditions. The other four use standard open source licenses.
How to run an open source model for code review
In Kodus, you set up the code review model through BYOK (Bring Your Own Key). Beyond the commercial providers, you can point reviews at any OpenAI-compatible endpoint, like a vLLM server running one of the models on this list, or at models hosted on Amazon Bedrock, Google Vertex AI and Azure OpenAI.
For teams that need to keep everything inside their own network, Kodus can also be installed on the company’s infrastructure. With the model and the review tool running in the same environment, the code never leaves your team’s control.
Conclusion
In our test, open models caught up with closed ones on the number of bugs found, and Kimi K3.1 Pro beat all of them. For a team that can’t send code outside the company, that changes the conversation with the security team, because choosing an open model no longer means settling for a worse review.
Closed models still get more of their comments right, and that gap shows up in the day-to-day work of whoever reviews the PRs. Along with hardware and licensing, it’s what your team needs to factor in before deciding.
Our suggestion is to start by testing DeepSeek V4.1 Flash on your own PRs and compare it with Kimi K3.1 Pro if your infrastructure allows. Results for all 13 models are in our article on the best LLMs for code review.
Frequently asked questions about open source LLMs for code review
What is the best open source LLM for code review?
In our test, Kimi K3.1 Pro, with 68.6% recall and an F2 of 63.6, the best result of all 13 models we evaluated. Among MIT-licensed models, DeepSeek V4.1 Flash had the best result, with 40.5% recall at $0.134 per PR.
What is the best local LLM for code review?
For limited hardware, Qwen3.8 27B, a dense 27-billion-parameter model that had the highest precision among the open models (51.5%). With more infrastructure available, DeepSeek V4.1 Flash found more bugs (49 versus 34 in balanced mode) and is MIT-licensed.
Does an open source model review code as well as a closed one?
On bugs found, yes. Kimi K3.1 Pro beat every closed model, and GLM-5.3 and DeepSeek V4.1 Flash tied GPT-6.1 Sol and Claude Opus 5.5 within the margin of error. On precision, the best closed models are still ahead: the most precise open model hit 51.5%, versus 62.7% for GPT-6.1 Sol.
What is the cheapest open source LLM for code review?
GLM-5.3 Flash, at $0.069 per bug found, the lowest in the entire benchmark. DeepSeek V4.1 Flash comes right after at $0.082 per bug, and it found six more bugs. These figures use each provider’s API price. On your own infrastructure, the cost comes down to your hardware.
Can I use these models commercially?
The MIT-licensed models (DeepSeek V4.1 Flash, DeepSeek V4 Pro and GLM-5.3 Flash) and the Apache 2.0 model (Qwen3.8 27B) can be used commercially as long as you keep the license notices. Kimi K3.1 Pro and GLM-5.3 only add conditions for companies offering the model as a service above a revenue threshold. MiniMax M3 requires attribution and notice to MiniMax for any commercial use.
Can I use an open source model for code review in Kodus?
Yes. Kodus BYOK accepts any OpenAI-compatible endpoint, which includes self-hosted servers like vLLM, as well as Amazon Bedrock, Google Vertex AI and Azure OpenAI. Kodus can also be installed on your company’s infrastructure, so the code and the model stay in the same environment.