Actor, Meet Critic:
RL for Code Review on Microsoft Foundry

Coding's burden of proof
My first pull request was fewer than 100 lines of code, and yet it received over 40 comments. The principal engineer pointed out issues with style conventions, downstream dependencies, and my over-reliance on LINQ1. That review took them over an hour.
ChatGPT released just a year later, and the volume of AI-generated code has since exploded. Now millions of developers use agents like Codex, Claude Code and GitHub Copilot, and their work has shifted from writing code to reviewing it.
With the rise of coding agents came code review agents. Companies like GitHub, Greptile, and CodeRabbit have built their own specialized reviewers. But these tools come with real tradeoffs. Many are expensive, with some costing around $1 to $10 per review. And adding an independent review agent is another vendor screened, another data protection agreement signed, and another company lurking in your codebase.
Why don't we try making our own?
Anatomy of a task
At its core, reviewing a pull request means reading a code diff and giving feedback through comments. Engineers carry a mental model of the codebase, so they know which modules depend on each other and which conventions the team enforces. A coding agent has to piece that together from the repo, the commit history and whatever instructions it receives.
| 6 | from .models import Customer | 6 | from .models import Customer |
| 9 | −def charge(customer, amount): | 9 | +def charge(customer, cents): |
RAreview-agent commented on line 9bug · high charge() now takes cents, but jobs/renewals.py:40 still passes plan.price, which is in dollars. Every renewal would charge 1/100 of the price. | |||
| 10 | """Charge the default card.""" | 10 | """Charge the default card.""" |
| 11 | − cents = int(amount * 100) | 11 | + if cents <= 0: |
| 12 | − return pay(customer, cents) | 12 | + raise ValueError |
RAreview-agent commented on line 12convention AGENTS.md asks billing code to raise BillingError. A ValueError here reaches the API as a 500. | |||
| 13 | + return pay(customer, cents) | ||
Why is this hard?
Detecting a bug has a definitive answer, but much of what makes a review good is harder to verify.
| 3 | −def charge(dollars): | |
| 4 | − return pay(dollars * 100) | |
| 3 | +def charge(cents): | |
RAreview-agent commented on line 3bug renewals.py still passes dollars to charge(). | ||
| 4 | + if cents <= 0: | |
| 5 | + raise ValueError("bad amount") | |
RAreview-agent commented on line 5convention AGENTS.md says to raise BillingError. | ||
| 6 | + return pay(cents) | |
charge() now takes cents.- Agents have to infer what a change is meant to do, since authors often leave the PR description empty or let AI write it.
- A change reaches beyond its diff, so agents must check downstream dependencies.
- Every repo has conventions, whether
AGENTS.mdspells them out or years of precedent imply them, and the agent has to follow them. - Agents also decide which comments are worth making, including whether nits deserve a place and, if so, how many.
Evaluation
Two reviewers can read the same PR and disagree, so it isn't obvious which comments should be saved as a gold standard. To address this, we score a reviewer’s comments based on whether the problems it flagged are ones the author later fixed.
Given a PR:
- Split its commits at the commit the review was made off of.
- Extract the review’s suggestions from its comments, and the author's fixes from the commits after the review.
- A judge model matches suggestions to fixes.
A judge model compares each review suggestion with the fixes the author made after the review. A suggestion is matched if the author addressed it in a later commit, which shows it was actionable and pointed at a real issue, and unmatched if not. During reference extraction, the judge is instructed to identify issues that already existed in the code under review and skip changes that don’t address a code issue, mitigating the inclusion of unrelated later work.
This closely follows the methodology published by Martian,2 chosen as they maintain an independent3 leaderboard for code reviews.
Their "online" methodology decides the leaderboard rankings. It works by finding real agent-authored PRs in the wild and analyzing their quality based on the commits that followed. This isn't fully compatible with our setup, since we aren't releasing a public review agent to millions of people for Martian's pipeline to find, ingest, and evaluate.
Instead, we sample PRs from a slice of the world's best coding agents according to Martian, grade them according to the online methodology, and turn each repository into an RL environment with corresponding ground truth usable for evaluation.
Measuring review quality
Our metrics of interest are:
- Completion rate: How reliably the agent navigates the repo with tools in a VM and outputs a review.
- F2: A weighted harmonic mean of precision and recall, which favors recall.
In this context, precision is the number of extracted reviewer suggestions that matched a reference action, divided by the agent’s total number of suggestions. Say an author made three fixes in the source repo PR, and our review agent identified those same three issues and then one more – that’s 75% precision.
Recall, on the other hand, is the number of distinct reference actions identified by the review agent divided by the total number of reference actions on the PR. For the same 75% precision scenario where we found all three issues, our recall would be 100%.
Why choose a weighted mean over standard F1? Our data collection prioritizes addressed suggestions (we look at the actions the PR author concretely took), but an unaddressed suggestion is weak evidence of it being a bad one. The author may have simply skipped a valid finding. Most organizations would also generally prefer a few more critical bugs found over a couple more nit suggestions prevented. So we select F2, which penalizes a missed fix more than an extra unmatched suggestion.
Raw data
The biggest improvements in post-training today are coming from better data, not new algorithms. DeepSeek's V4.1 report4 kept its training recipe standard and credited "essentially all of the observed gains" to its data and environments. While the entire corpus of public GitHub repositories is at our service, we need to ensure that our training data is high quality: diverse, realistic, and learnable.
A common way to build a code review dataset is to inject bugs into working code and ask the model to find them, which gives you unlimited training data. But injected bugs do not represent the problems that actually show up when a code review agent is running, and training data is only useful if it's realistic. So we turn to open source repos that already have real code reviews. We sample thousands of raw PRs that have bot-authored code reviews from one of the top 10 review agents according to Martian’s leaderboard as of 2026-10-04. This corpus is the top-of-funnel for our filtering and derisking.
Environment and harness
Once we have raw data, we need to turn it into production-like tasks a model can learn in.
How a task is built
Harbor5 is an open source framework for running and evaluating agents in sandboxed containers. It packages every task with an instruction, an environment, and a reward signal. Here, a Harbor task is composed of:
- An environment: a frozen pull request, meaning the repo at the reviewed commit plus the diff.
- A judge: scores the review against the fixes the author made later.
- A reward: the F2 score the judge writes.
The remaining files tell Harbor how to run the task (task.toml sets the time limit and network rules) and tell the trainer what it needs (training.json and the trial configs).
Runs authoritative grading in the separate verifier sandbox using private references.
#!/bin/sh set -eu python -m code_review.grading
The implication is that every training and evaluation example gets its own virtual world for the agent to navigate. This is necessary because we are teaching the model to get generally better at code review, rather than improving it on one particular repo.
The harness
We use OpenSWE's6 native reviewer, an open source harness by LangChain. We've made minor modifications to the harness, such as removing the ability to search the web so the agent doesn't cheat and find the answer.
What we changed for training:
- Removed web tools: so the model cannot look up the PR and its later fixes online.
- Stubbed out external services: GitHub, the cloud sandbox and the model provider are replaced by a frozen local snapshot and the model being trained. Combined with a total network lockdown at the sandbox level, HuggingFace should be safe from our experiments here!
- Published every finding: OpenSWE's severity and rank filter is off, so the judge sees everything the model found.
- Kept compaction, outside the loss: Summaries are counted with the trained model's tokenizer, and summary calls are not trained on.
- Made the source read-only: An episode that changes the code scores -1.
An example trajectory
Below is a trajectory from a frozen review of better-auth/better-auth#11110. The agent follows a new adapter entry to the table component that renders it, records a low-severity finding about the link target, and completes a simulated publication.
{}{
"success": true,
"path": "/workspace/better-auth/review-diff-560d5213175644a2.patch",
"bytes": 643,
"files": [
"docs/lib/community-adapters-data.ts"
],
"file_count": 1,
"files_truncated": false,
"base_sha": "626adf67171923c9a83fa6fac8bfcbea2d540638",
"head_sha": "dc325b572e6f2a402c620b775ca8622b00a3efdf",
"cached": true
}{
"pattern": "docs/lib/community-adapters-data.ts",
"path": "/workspace/better-auth/review-diff-560d5213175644a2.patch",
"glob": null,
"output_mode": "content",
"max_count": 10
}/workspace/repo/review-diff-560d5213175644a2.patch: 1: diff --git a/docs/lib/community-adapters-data.ts b/docs/lib/community-adapters-data.ts 3: --- a/docs/lib/community-adapters-data.ts 4: +++ b/docs/lib/community-adapters-data.ts
{
"file_path": "/workspace/better-auth/review-diff-560d5213175644a2.patch",
"offset": 0,
"limit": 100
}diff --git a/docs/lib/community-adapters-data.ts b/docs/lib/community-adapters-data.ts
index 9c7b61e..7291115 100644
--- a/docs/lib/community-adapters-data.ts
+++ b/docs/lib/community-adapters-data.ts
@@ -253,4 +253,15 @@ export const communityAdapters: CommunityAdapter[] = [
avatar: "https://github.com/ilbertt.png",
},
},
+ {
+ name: "@a77ay/better-auth-mikro-orm",
+ url: "https://www.npmjs.com/package/@a77ay/better-auth-mikro-orm",
+ database: "MikroORM",
+ databaseUrl: "https://mikro-orm.io/",
+ author: {
+ name: "A77AY",
+ url: "https://github.com/a77ay",
+ avatar: "https://github.com/a77ay.png",
+ },
+ },
];{
"file_path": "/workspace/better-auth/docs/lib/community-adapters-data.ts",
"offset": 0,
"limit": 270
}{
"pattern": "communityAdapters",
"path": "/workspace/better-auth",
"glob": "*.ts",
"output_mode": "content",
"max_count": 20
}/workspace/repo/docs/lib/community-adapters-data.ts: 13: export const communityAdapters: CommunityAdapter[] = [
export interface CommunityAdapter {
name: string;
url: string;
database: string;
databaseUrl: string;
author: {
name: string;
url: string;
avatar: string;
};
}
export const communityAdapters: CommunityAdapter[] = [
{
name: "@convex-dev/better-auth",
url: "https://github.com/get-convex/better-auth",
database: "Convex",
databaseUrl: "https://www.convex.dev/",
author: {
name: "erquhart",
url: "https://github.com/erquhart",
avatar: "https://github.com/erquhart.png",
},
},
{
name: "surreal-better-auth",
url: "https://github.com/oskar-gmerek/surreal-better-auth",
database: "SurrealDB",
databaseUrl: "https://surrealdb.com/",
author: {
name: "Oskar Gmerek",
url: "https://oskargmerek.com",
avatar: "https://github.com/oskar-gmerek.png",
},
},
{
name: "surrealdb-better-auth",
url: "https://github.com/Necmttn/surrealdb-better-auth",
database: "SurrealDB",
databaseUrl: "https://surrealdb.com/",
author: {
name: "Necmttn",
url: "https://github.com/Necmttn",
avatar: "https://github.com/Necmttn.png",
},
},
{
name: "better-auth-surrealdb",
url: "https://github.com/msanchezdev/better-auth-surrealdb",
database: "SurrealDB",
databaseUrl: "https://surrealdb.com/",
author: {
name: "msanchezdev",
url: "https://github.com/msanchezdev",
avatar: "https://github.com/msanchezdev.png",
},
},
{
name: "payload-auth",
url: "https://github.com/payload-auth/payload-auth",
database: "Payload CMS",
databaseUrl: "https://payloadcms.com/",
author: {
name: "forrestdevs",
url: "https://github.com/forrestdevs",
avatar: "https://github.com/forrestdevs.png",
},
},
{
name: "@delmaredigital/payload-better-auth",
url: "https://github.com/delmaredigital/payload-better-auth",
database: "Payload CMS",
databaseUrl: "https://payloadcms.com/",
author: {
name: "Delmare Digital",
url: "https://github.com/delmaredigital",
avatar: "https://github.com/delmaredigital.png",
},
},
{
name: "@hedystia/better-auth-typeorm",
url: "https://github.com/Zastinian/better-auth-typeorm",
database: "TypeORM",
databaseUrl: "https://typeorm.io/",
author: {
name: "Zastinian",
url: "https://github.com/Zastinian",
avatar: "https://github.com/Zastinian.png",
},
},
{
name: "better-auth-instantdb",
url: "https://github.com/daveyplate/better-auth-instantdb",
database: "InstantDB",
databaseUrl: "https://www.instantdb.com/",
author: {
name: "daveycodez",
url: "https://github.com/daveycodez",
avatar: "https://github.com/daveycodez.png",
},
},
{
name: "@nerdfolio/remult-better-auth",
url: "https://github.com/nerdfolio/remult-better-auth",
database: "Remult",
databaseUrl: "https://remult.dev/",
author: {
name: "Tai Vo",
url: "https://github.com/taivo",
avatar: "https://github.com/taivo.png",
},
},
{
name: "pocketbase-better-auth",
url: "https://github.com/LightInn/pocketbase-better-auth",
database: "PocketBase",
databaseUrl: "https://pocketbase.io/",
author: {
name: "LightInn",
url: "https://github.com/LightInn",
avatar: "https://github.com/LightInn.png",
},
},
{
name: "better-auth-firestore",
url: "https://github.com/yultyyev/better-auth-firestore",
database: "Firebase Firestore",
databaseUrl: "https://firebase.google.com/docs/firestore",
author: {
name: "yultyyev",
url: "https://github.com/yultyyev",
avatar: "https://github.com/yultyyev.png",
},
},
{
name: "@zenstackhq/better-auth",
url: "https://github.com/zenstackhq/zenstack/tree/main/packages/auth-adapters/better-auth",
database: "ZenStack",
databaseUrl: "https://zenstack.dev",
author: {
name: "zenstackhq",
url: "https://github.com/zenstackhq",
avatar: "https://github.com/zenstackhq.png",
},
},
{
name: "@strapi-community/plugin-better-auth",
url: "https://github.com/strapi-community/plugin-better-auth",
database: "Strapi CMS",
databaseUrl: "https://strapi.io/",
author: {
name: "boazpoolman",
url: "https://github.com/boazpoolman",
avatar: "https://github.com/boazpoolman.png",
},
},
{
name: "neo4j-better-auth",
url: "https://github.com/florianamette/better-auth-neo4j",
database: "Neo4j",
databaseUrl: "https://neo4j.com/",
author: {
name: "florianamette",
url: "https://github.com/florianamette",
avatar: "https://github.com/florianamette.png",
},
},
{
name: "@lubiah/better-auth-mikro-orm",
url: "https://github.com/lubiah/better-auth-mikro-orm",
database: "MikroORM",
databaseUrl: "https://mikro-orm.io/",
author: {
name: "lubiah",
url: "https://github.com/lubiah",
avatar: "https://github.com/lubiah.png",
},
},
{
name: "better-auth-mikro-orm",
url: "https://github.com/octet-stream/better-auth-mikro-orm",
database: "MikroORM",
databaseUrl: "https://mikro-orm.io/",
author: {
name: "octet-stream",
url: "https://github.com/octet-stream",
avatar: "https://github.com/octet-stream.png",
},
},
{
name: "@proofkit/better-auth",
url: "https://github.com/proofsh/proofkit/tree/main/packages/better-auth",
database: "FileMaker OData",
databaseUrl: "https://www.claris.com/filemaker/",
author: {
name: "eluce2",
url: "https://github.com/eluce2",
avatar: "https://github.com/eluce2.png",
},
},
{
name: "@datar-platform/better-auth-dynamodb",
url: "https://github.com/datar-platform/better-auth-dynamodb",
database: "DynamoDB",
databaseUrl: "https://aws.amazon.com/dynamodb/",
author: {
name: "joesome-git",
url: "https://github.com/joesome-git",
avatar: "https://github.com/joesome-git.png",
},
},
{
name: "@bjorntech/betterauth-dynamodb",
url: "https://github.com/bjorntech/betterauth-dynamodb",
database: "DynamoDB",
databaseUrl: "https://aws.amazon.com/dynamodb/",
author: {
name: "BjornTech AB",
url: "https://github.com/bjorntech",
avatar: "https://github.com/bjorntech.png",
},
},
{
name: "better-auth-azure-cosmos",
url: "https://github.com/9hsein5/better-auth-azure-cosmos",
database: "Azure Cosmos DB",
databaseUrl: "https://learn.microsoft.com/azure/cosmos-db/nosql/",
author: {
name: "9hsein5",
url: "https://github.com/9hsein5",
avatar: "https://github.com/9hsein5.png",
},
},
{
name: "better-auth-mongoose",
url: "https://github.com/AshwinSathian/better-auth-mongoose",
database: "Mongoose",
databaseUrl: "https://mongoosejs.com",
author: {
name: "Ashwin Sathian",
url: "https://github.com/AshwinSathian",
avatar: "https://github.com/AshwinSathian.png",
},
},
{
name: "@ilbertt/better-auth-bun-sql",
url: "https://github.com/ilbertt/better-auth-bun-sql",
database: "Bun SQL",
databaseUrl: "https://bun.com/docs/api/sql",
author: {
name: "ilbertt",
url: "https://github.com/ilbertt",
avatar: "https://github.com/ilbertt.png",
},
},
{
name: "@a77ay/better-auth-mikro-orm",
url: "https://www.npmjs.com/package/@a77ay/better-auth-mikro-orm",
database: "MikroORM",
databaseUrl: "https://mikro-orm.io/",
author: {
name: "A77AY",
url: "https://github.com/a77ay",
avatar: "https://github.com/a77ay.png",
},
},
];{
"pattern": "community-adapters-data",
"path": "/workspace/better-auth",
"glob": "*.tsx",
"output_mode": "content",
"max_count": 20
}/workspace/repo/docs/components/community-adapters-table.tsx:
17: import type { CommunityAdapter } from "@/lib/community-adapters-data";
18: import { communityAdapters } from "@/lib/community-adapters-data";{
"file_path": "/workspace/better-auth/docs/components/community-adapters-table.tsx",
"offset": 0,
"limit": 180
}"use client";
import type {
ColumnDef,
ColumnFiltersState,
SortingState,
} from "@tanstack/react-table";
import {
flexRender,
getCoreRowModel,
getFilteredRowModel,
getSortedRowModel,
useReactTable,
} from "@tanstack/react-table";
import { ArrowUpDown, Search } from "lucide-react";
import { useState } from "react";
import type { CommunityAdapter } from "@/lib/community-adapters-data";
import { communityAdapters } from "@/lib/community-adapters-data";
const columns: ColumnDef<CommunityAdapter>[] = [
{
accessorKey: "name",
header: ({ column }) => {
return (
<button
type="button"
className="flex items-center gap-2 font-semibold hover:text-foreground transition-colors whitespace-nowrap"
onClick={() => column.toggleSorting(column.getIsSorted() === "asc")}
>
Adapter
<ArrowUpDown className="h-4 w-4" />
</button>
);
},
cell: ({ row }) => {
return (
<div className="w-[260px]">
<a
href={row.original.url}
target="_blank"
rel="noopener noreferrer"
className="inline-flex text-muted-foreground transition-colors hover:text-foreground"
>
<code className="text-sm font-mono tracking-tight whitespace-nowrap">
{row.original.name}
</code>
</a>
</div>
);
},
},
{
accessorKey: "database",
header: ({ column }) => {
return (
<button
type="button"
className="flex items-center gap-2 font-semibold hover:text-foreground transition-colors whitespace-nowrap"
onClick={() => column.toggleSorting(column.getIsSorted() === "asc")}
>
Database
<ArrowUpDown className="h-4 w-4" />
</button>
);
},
cell: ({ row }) => {
return (
<a
href={row.original.databaseUrl}
target="_blank"
rel="noopener noreferrer"
className="text-sm text-muted-foreground hover:text-foreground transition-colors whitespace-nowrap"
>
{row.original.database}
</a>
);
},
},
{
accessorFn: (adapter) => adapter.author.name,
id: "author",
header: ({ column }) => {
return (
<button
type="button"
className="flex items-center gap-2 font-semibold hover:text-foreground transition-colors whitespace-nowrap"
onClick={() => column.toggleSorting(column.getIsSorted() === "asc")}
>
Author
<ArrowUpDown className="h-4 w-4" />
</button>
);
},
cell: ({ row }) => {
const author = row.original.author;
return (
<a
href={author.url}
target="_blank"
rel="noopener noreferrer"
className="flex items-center gap-2 hover:text-foreground transition-colors text-muted-foreground whitespace-nowrap"
>
<img
src={author.avatar}
alt={author.name}
className="rounded-full w-6 h-6 border opacity-70 m-0"
/>
<span className="text-sm">{author.name}</span>
</a>
);
},
},
];
export function CommunityAdaptersTable() {
const [sorting, setSorting] = useState<SortingState>([]);
const [columnFilters, setColumnFilters] = useState<ColumnFiltersState>([]);
const [globalFilter, setGlobalFilter] = useState("");
const table = useReactTable({
data: communityAdapters,
columns,
getCoreRowModel: getCoreRowModel(),
getSortedRowModel: getSortedRowModel(),
getFilteredRowModel: getFilteredRowModel(),
onSortingChange: setSorting,
onColumnFiltersChange: setColumnFilters,
onGlobalFilterChange: setGlobalFilter,
state: {
sorting,
columnFilters,
globalFilter,
},
});
return (
<div className="not-prose w-full space-y-4">
<div className="flex items-center justify-between text-xs text-muted-foreground">
<p>
Showing {table.getRowModel().rows.length} of{" "}
{communityAdapters.length} adapters
</p>
</div>
<div className="relative">
<Search className="absolute left-3 top-1/2 h-4 w-4 -translate-y-1/2 text-muted-foreground" />
<input
type="text"
aria-label="Search community adapters"
placeholder="Search adapters, databases, or authors..."
value={globalFilter ?? ""}
onChange={(e) => setGlobalFilter(e.target.value)}
className="w-full rounded-lg border bg-background pl-10 pr-4 py-2 text-sm focus:outline-none focus:ring-2 focus:ring-ring"
/>
</div>
<div className="rounded-lg border">
<div className="overflow-x-auto">
<table className="w-full min-w-[720px]">
<thead>
{table.getHeaderGroups().map((headerGroup) => (
<tr key={headerGroup.id} className="border-b bg-muted/50">
{headerGroup.headers.map((header) => (
<th
key={header.id}
className="px-4 py-3 text-left text-sm font-medium"
>
{header.isPlaceholder
? null
: flexRender(
header.column.columnDef.header,
header.getContext(),
)}
</th>
))}
</tr>
))}
</thead>
<tbody>
{table.getRowModel().rows?.length ? (
table.getRowModel().rows.map((row) => ({
"severity": "low",
"confidence": "high",
"category": "correctness",
"file": "docs/lib/community-adapters-data.ts",
"title": "Adapter link targets npm instead of repository",
"description": "The table renders `adapter.url` as the adapter's link. This entry points to the npm package page, while every other adapter entry in the table links to its source repository. Users clicking the adapter name get a package page instead of the adapter's code/repository, unlike the surrounding entries.",
"start_line": 257,
"end_line": 257,
"suggestion": null,
"side": "RIGHT"
}{
"success": true,
"finding_id": "f_b6a1f4e932",
"duplicate": false
}{
"status_filter": "open"
}{
"findings": [
{
"id": "f_b6a1f4e932",
"severity": "low",
"confidence": "high",
"category": "correctness",
"title": "Adapter link targets npm instead of repository",
"file": "docs/lib/community-adapters-data.ts",
"start_line": 257,
"end_line": 257,
"side": "RIGHT",
"in_diff": true,
"description": "The table renders `adapter.url` as the adapter's link. This entry points to the npm package page, while every other adapter entry in the table links to its source repository. Users clicking the adapter name get a package page instead of the adapter's code/repository, unlike the surrounding entries.",
"suggestion": null,
"status": "open",
"first_seen_sha": "dc325b572e6f2a402c620b775ca8622b00a3efdf",
"last_confirmed_sha": "dc325b572e6f2a402c620b775ca8622b00a3efdf",
"github_review_id": null,
"github_review_run_id": null,
"github_review_comment_ids": [],
"github_review_thread_ids": [],
"github_resolved_thread_ids": [],
"github_posted_resolution_comment_ids": [],
"surface_state": "not_surfaced",
"last_human_reply_at": null,
"last_human_reply_author": null,
"last_human_reply_body": null,
"last_reconciliation_note": null,
"resolution_note": null,
"diff_hunk": "@@ -253,4 +253,15 @@ export const communityAdapters: CommunityAdapter[] = [\n \t\t\tavatar: \"https://github.com/ilbertt.png\",\n \t\t},\n \t},\n+\t{\n+\t\tname: \"@a77ay/better-auth-mikro-orm\",\n+\t\turl: \"https://www.npmjs.com/package/@a77ay/better-auth-mikro-orm\",\n+\t\tdatabase: \"MikroORM\",\n+\t\tdatabaseUrl: \"https://mikro-orm.io/\",\n+\t\tauthor: {\n+\t\t\tname: \"A77AY\",\n+\t\t\turl: \"https://github.com/a77ay\",\n+\t\t\tavatar: \"https://github.com/a77ay.png\",\n+\t\t},\n+\t},\n ];",
"fingerprint": "v1:6e79f3ff80a6c223182a01535d7b1863d926ab234a4aca8c9f243344950cd245",
"interactions": [],
"rank": null
}
],
"count": 1
}{
"ranking": [
"f_b6a1f4e932"
],
"severity_threshold": "medium",
"assessment": null
}{
"success": true,
"dry_run": true,
"review_id": null,
"surfaced_count": 1,
"hidden_count": 0,
"resolved_thread_count": 0
}adapter.url as the link, but this entry points to the npm package page rather than the adapter’s source repository like the other entries.With the harness and environment set, we can move on to figuring out which data is usable for training.
Curating learnable data
Not every reviewed pull request makes a good RL task. If the task is too hard, the model never stumbles onto correct solutions and learns nothing, or it resorts to reward hacking. If it is too easy, the model solves it every time and there is no gradient. Some PRs only have dependabot comments, while others are merged without reviewer approval. So we filter on two principles: whether we can score the task honestly, and whether training on it will teach the model anything.
Can we score it honestly?
- Size: At most 50 commits and 2,000 changed lines, the same limits Martian uses.
- Reviewed commit known: We need the exact commit the bot saw. Without it we cannot split the history, and the answer key is undefined.
- At least one fix: A PR where the author fixed nothing after review has no answer key. Any review of it scores zero or is unscorable.
- No leaks: The frozen repository must not contain any later fix commit, in its objects, refs or image layers. If it did, the model could read the answer instead of reviewing the code, and RL would find that shortcut fast.
Will it move the policy?
We perform successive rounds of RL derisking by measuring the mean F2 of the model checkpoint due for training, and filtering out examples that score below 0.1 or above 0.9 across multiple trials. This is consistent with common industry practice, such as the rollout outcome filtering in Xiaomi's CodeMidas7 paper.
Training
Intelligent rollout expansion
We apply a trick that 5x'd our training throughput. The Qwen3.8-27B base model turns out to be rather terrible at this review task. Higher reasoning levels saw longer, more expensive rollouts and frequently took us beyond our context limit, triggering many compaction calls. So we used Qwen3.8-27B (no thinking) as our base model. The no-thinking model had shorter trajectories, compacted less, and was much more token efficient when it completed.
But the no-thinking model was less likely to produce valid answers. It often got stuck in what we called tool-call "doom loops": failing to grok the result of a tool, realizing it still needed the information, and invoking the tool again, until the episode timed out or hit the max horizon.
Our initial mitigations were heuristics that detected doom loops based on repetitions and argument similarity. Subtle cases still crept through, like this cyclical sequence of three calls repeated until the horizon:
After cyclical sequences came semantic equivalencies. Every form of degenerate loop wasted compute on a rollout that was doomed to fail.
We ran Microsoft-Decision-19 with a tuned criteria set for detecting rollout degeneracy early and saw nearly perfect detection rates. So we added it at rollout time, querying it millions of times, once for every new turn of every trajectory, on a 16-turn lookback window. Effectively, we can now focus our sampling budget on the rollouts most likely to produce meaningful results. Examples terminated by Microsoft-Decision-1 are assigned a reward penalty to discourage the behavior during gradient optimization.
Algorithm selection
We run cold-start SFT for early model-side doom loop mitigation, and RL with intelligent rollout expansion powered by Microsoft-Decision-1 for hill-climbing review quality thereafter. For RL, we train with group-relative advantages: the model reviews each task four times, and each review's advantage is its reward minus the group's mean. If all four reviews score the same, every advantage is zero and the group contributes no gradient. Such groups are dropped and not used for training, meaning we may sample more groups each step than we actually use.
Data for the RL run is derisked using the SFT checkpoint. We run Microsoft-Decision-1 in the loop for derisking and evaluations as well.
Results
| Model (reasoning level) | Mean F2 (completed only) | Completion rate (%) | Completion-adjusted F2 | Tokens | Tool calls | Turns |
|---|---|---|---|---|---|---|
| Qwen3.8-27B (none) | 0.1000 | 47.5 | 0.0475 | 274.46M | 7589 | 6955 |
| + SFT | 0.1537 | 90.0 | 0.1383 | 51.44M | 3220 | 2509 |
| + RLours | 0.1873 | 91.7 | 0.1718 | 21.72M | 2014 | 1098 |
| gpt-6-luna (none) | 0.0495 | 95.0 | 0.0470 | 50.92M | 3271 | 2632 |
| gpt-6-luna (high) | 0.1839 | 89.2 | 0.1640 | 97.76M | 8494 | 4353 |
The cold-start SFT is trained with 100 review trajectories from a luna-class teacher model, filtered to ensure all instances score above a minimal F2 threshold. The SFT makes a dramatic difference in boosting the completion rate of the Qwen3.8-27B model, which nearly doubles. Qualitative analysis further reveals the model learns a more efficient review workflow, outperforms at repo navigation, and better incorporates self-verification before finishing its reviews.
More efficient review workflows
Every recorded SFT read_file call supplied offset and limit arguments whereas the base model often omitted one or both. The base model consequently juggled more tokens per turn and was more likely to get lost in the context, degrading into tool doom loops. The SFT checkpoint also tended to make fewer repeated reads; repeated identical tool call count dropped 90.7% from 1204 to 112.
Better repo navigation
The base model tended to navigate the repo through repeated shell commands sent through execute, which runs a shell command inside the reviewer sandbox and returns its output. The SFT checkpoint, on the other hand, uses the dedicated grep and read_file tools from the harness. This adaptation translated into fewer turns spent on finding files and more time dedicated to reviewing code.
Better self-verification
In all but one of the SFT checkpoint’s evaluation traces it called the list_findings tool before publishing its review comments, which gave it the chance to review its work for errors and correct them before submitting the finalized review. The base model did this in less than half of its successful completions, so if errors did slip in, the model was less likely to find them.
Taken together with tool doom loop elimination, the agent became fivefold more token efficient at the task from SFT alone. Note this figure is a lower bound on the efficiency gains because we apply Microsoft-Decision-1 early termination across training and evals; had doomed trajectories not been terminated, the token consumption would have been even higher for the base model.
Cold-start SFT loss
Lower is better- Training batch loss (4 conversations per update)
- Validation loss (same 16 held-out conversations)
RL nets us another +3pp in F2 after training on a small, highly filtered set of 150 examples for two epochs.
On deeper review, we see that SFT established the navigation workflow, and RL shifted it towards faster decisions, more candidate findings, and more self-correction. The pagination discipline is retained, with all read_file calls invoked with offset and limit parameters supplied. list_findings is called relatively less – present in 99% of SFT traces but only 74% of RL ones. Yet the RL model’s update_finding frequency increased 8x over the SFT checkpoint, meaning it got more comfortable with correcting its own perceived errors when it did decide to check for them.
For examples that both the SFT and RL checkpoints were able to complete, the RL model used 50.9% fewer tokens, 32.5% fewer tool calls, and 47.6% fewer turns. This nets us an overall 2.4x improvement in token efficiency from the RL run despite not explicitly designing for this.
To contextualize our Qwen3.8-27B numbers, we evaluate gpt-6-luna with no thinking and reasoning high; our lightly RL’d review agent slightly edges out the high reasoning profile while being 4.5x more token efficient.
Mean F2 during RL
Higher is better- Mean F2 (raw)
- Smoothed
On TransformerEngine #3141, the SFT model flags an undocumented option override but misses its functional consequence (F2 = 0). The RL checkpoint identifies the missing FP8 weight-gradient operand with 7 file reads versus 59 (F2 = 1).
Parallel tool calls and results use a consistent order. Long results are excerpted; message numbers follow the recorded traces.
Conclusion
In this article we explored what it takes to hill-climb for agentic coding review on real-world GitHub PRs. With only small slices of highly filtered data for LoRA-based SFT and RL, we nearly double the mean F2 score over our base model and drive a 12.6x improvement in token efficiency.
We did so in partnership with the stellar team behind Microsoft's new Foundry Programmed Training offering; its performant GPU-based training orchestration allowed us to focus on what's often the hardest part of any post-training project: data, evals, and experimental design.
If you want to learn more or have any questions, we'd love to chat.
Disclosure: Whitney AI is a Microsoft partner; this post was produced in collaboration with Microsoft.

