CONNECT WITH US
Tech

Tech

Copilot tops GitHub’s own AI code review benchmark. An independent one tells a different story.

TheNewStack – Tech logo

Published on

Add as a preferred source on Google
Copilot tops GitHub’s own AI code review benchmark. An independent one tells a different story.

If there’s one thing the AI software engineering world doesn’t lack, it’s benchmarks. Want to know whether an agent can resolve real-world GitHub issues? There’s SWE-bench. Or how about seeing how well it can operate inside a terminal? Terminal-Bench has you covered.

Now, GitHub has decided AI code review needs another yardstick. On Monday, the company unveiled ReviewBench, an open benchmark developed with Microsoft for measuring how well AI code review agents find useful problems in pull requests. And sitting at the top of its inaugural leaderboard is — perhaps unsurprisingly — GitHub Copilot code review.

GitHub debuted Copilot code review in October 2024, making it generally available to all paid Copilot subscribers the following April. In the intervening months, GitHub has moved the reviewer onto an agentic architecture that pulls in wider context from the repository, started billing it in GitHub Actions minutes on private repositories, let it approve pull requests, and made a more thorough “Balanced” review mode the default on Sept. 28.

Code review has always been a crucial part of software development, and GitHub has plenty of company trying to automate more of it. Qodo, Greptile, Cubic, Devin and Cursor are among the products now offering AI-assisted reviews, while CodeRabbit also features prominently on other code-review leaderboards.

And now GitHub wants to provide another common test for comparing how well those systems actually perform.

ReviewBench takes 219 public pull requests from 187 public repositories spanning 19 programming languages, and asks competing reviewers to inspect the same changes.


Source link
Related Topics:Tech

Disclaimer

We strive to uphold the highest ethical standards in all of our reporting and coverage. We TheMorningPulse.fyi want to be transparent with our readers about any potential conflicts of interest that may arise in our work. It's possible that some of the investors we feature may have connections to other businesses, including competitors or companies we write about. However, we want to assure our readers that this will not have any impact on the integrity or impartiality of our reporting. We are committed to delivering accurate, unbiased news and information to our audience, and we will continue to uphold our ethics and principles in all of our work. Thank you for your trust and support.