Whitney AIRequest access
Journal

Actor, Meet Critic:
RL for Code Review on Microsoft Foundry

47.5% → 91.7%review completion rate
10% → 19%mean F2 on completed reviews
12.6xmore token efficient
Whitney AI

Coding's burden of proof

My first pull request was fewer than 100 lines of code, and yet it received over 40 comments. The principal engineer pointed out issues with style conventions, downstream dependencies, and my over-reliance on LINQ1. That review took them over an hour.

ChatGPT released just a year later, and the volume of AI-generated code has since exploded. Now millions of developers use agents like Codex, Claude Code and GitHub Copilot, and their work has shifted from writing code to reviewing it.

With the rise of coding agents came code review agents. Companies like GitHub, Greptile, and CodeRabbit have built their own specialized reviewers. But these tools come with real tradeoffs. Many are expensive, with some costing around $1 to $10 per review. And adding an independent review agent is another vendor screened, another data protection agreement signed, and another company lurking in your codebase.

Why don't we try making our own?

Anatomy of a task

At its core, reviewing a pull request means reading a code diff and giving feedback through comments. Engineers carry a mental model of the codebase, so they know which modules depend on each other and which conventions the team enforces. A coding agent has to piece that together from the repo, the commit history and whatever instructions it receives.

billing/charge.py+5 −3
6from .models import Customer6from .models import Customer
9−def charge(customer, amount):9+def charge(customer, cents):
RAreview-agent commented on line 9bug · high
charge() now takes cents, but jobs/renewals.py:40 still passes plan.price, which is in dollars. Every renewal would charge 1/100 of the price.
10 """Charge the default card."""10 """Charge the default card."""
11− cents = int(amount * 100)11+ if cents <= 0:
12− return pay(customer, cents)12+ raise ValueError
RAreview-agent commented on line 12convention
AGENTS.md asks billing code to raise BillingError. A ValueError here reaches the API as a 500.
13+ return pay(customer, cents)

Why is this hard?

Detecting a bug has a definitive answer, but much of what makes a review good is harder to verify.

billing/charge.py+4 −2
3−def charge(dollars):
4− return pay(dollars * 100)
3+def charge(cents):
RAreview-agent commented on line 3bug
renewals.py still passes dollars to charge().
4+ if cents <= 0:
5+ raise ValueError("bad amount")
RAreview-agent commented on line 5convention
AGENTS.md says to raise BillingError.
6+ return pay(cents)
PR descriptionempty
No description provided.
The agent reads the diff to learn that charge() now takes cents.
  • Agents have to infer what a change is meant to do, since authors often leave the PR description empty or let AI write it.
  • A change reaches beyond its diff, so agents must check downstream dependencies.
  • Every repo has conventions, whether AGENTS.md spells them out or years of precedent imply them, and the agent has to follow them.
  • Agents also decide which comments are worth making, including whether nits deserve a place and, if so, how many.

Evaluation

Two reviewers can read the same PR and disagree, so it isn't obvious which comments should be saved as a gold standard. To address this, we score a reviewer’s comments based on whether the problems it flagged are ones the author later fixed.

Given a PR:

  1. Split its commits at the commit the review was made off of.
  2. Extract the review’s suggestions from its comments, and the author's fixes from the commits after the review.
  3. A judge model matches suggestions to fixes.
WHAT THE REVIEWER SEESANSWER KEY · AUTHOR'S FIXESa1f3e9amounts in cents7c2d41update invoices3be81fadd migrationd94a0bfix renewal callere1c5aabackfill old rows9e1c07merge into mainreviewed headbot reviewreview-bot reviewed 3be81f1renewals.py:40dollars, not cents2charge.py:12rename amtno fix30042_cents.py:8old rows not filled13judge match

A judge model compares each review suggestion with the fixes the author made after the review. A suggestion is matched if the author addressed it in a later commit, which shows it was actionable and pointed at a real issue, and unmatched if not. During reference extraction, the judge is instructed to identify issues that already existed in the code under review and skip changes that don’t address a code issue, mitigating the inclusion of unrelated later work.

This closely follows the methodology published by Martian,2 chosen as they maintain an independent3 leaderboard for code reviews.

Their "online" methodology decides the leaderboard rankings. It works by finding real agent-authored PRs in the wild and analyzing their quality based on the commits that followed. This isn't fully compatible with our setup, since we aren't releasing a public review agent to millions of people for Martian's pipeline to find, ingest, and evaluate.

Instead, we sample PRs from a slice of the world's best coding agents according to Martian, grade them according to the online methodology, and turn each repository into an RL environment with corresponding ground truth usable for evaluation.

Measuring review quality

Our metrics of interest are:

  1. Completion rate: How reliably the agent navigates the repo with tools in a VM and outputs a review.
  2. F2: A weighted harmonic mean of precision and recall, which favors recall.
Fβ = (1 + β2) · precision · recallβ2 · precision + recall
F2 = 5 · precision · recall4 · precision + recall

In this context, precision is the number of extracted reviewer suggestions that matched a reference action, divided by the agent’s total number of suggestions. Say an author made three fixes in the source repo PR, and our review agent identified those same three issues and then one more – that’s 75% precision.

Recall, on the other hand, is the number of distinct reference actions identified by the review agent divided by the total number of reference actions on the PR. For the same 75% precision scenario where we found all three issues, our recall would be 100%.

Why choose a weighted mean over standard F1? Our data collection prioritizes addressed suggestions (we look at the actions the PR author concretely took), but an unaddressed suggestion is weak evidence of it being a bad one. The author may have simply skipped a valid finding. Most organizations would also generally prefer a few more critical bugs found over a couple more nit suggestions prevented. So we select F2, which penalizes a missed fix more than an extra unmatched suggestion.

Raw data

The biggest improvements in post-training today are coming from better data, not new algorithms. DeepSeek's V4.1 report4 kept its training recipe standard and credited "essentially all of the observed gains" to its data and environments. While the entire corpus of public GitHub repositories is at our service, we need to ensure that our training data is high quality: diverse, realistic, and learnable.

A common way to build a code review dataset is to inject bugs into working code and ask the model to find them, which gives you unlimited training data. But injected bugs do not represent the problems that actually show up when a code review agent is running, and training data is only useful if it's realistic. So we turn to open source repos that already have real code reviews. We sample thousands of raw PRs that have bot-authored code reviews from one of the top 10 review agents according to Martian’s leaderboard as of 2026-10-04. This corpus is the top-of-funnel for our filtering and derisking.

Environment and harness

Once we have raw data, we need to turn it into production-like tasks a model can learn in.

How a task is built

Harbor5 is an open source framework for running and evaluating agents in sandboxed containers. It packages every task with an instruction, an environment, and a reward signal. Here, a Harbor task is composed of:

  • An environment: a frozen pull request, meaning the repo at the reviewed commit plus the diff.
  • A judge: scores the review against the fixes the author made later.
  • A reward: the F2 score the judge writes.

The remaining files tell Harbor how to run the task (task.toml sets the time limit and network rules) and tell the trainer what it needs (training.json and the trial configs).

environment/reviewer
tests/verifier
tests/test.sh
verifier

Runs authoritative grading in the separate verifier sandbox using private references.

shell
#!/bin/sh
set -eu
python -m code_review.grading

The implication is that every training and evaluation example gets its own virtual world for the agent to navigate. This is necessary because we are teaching the model to get generally better at code review, rather than improving it on one particular repo.

The harness

We use OpenSWE's6 native reviewer, an open source harness by LangChain. We've made minor modifications to the harness, such as removing the ability to search the web so the agent doesn't cheat and find the answer.

instructionsTools
Reviewer agentObserve · Reason · Act

What we changed for training:

  • Removed web tools: so the model cannot look up the PR and its later fixes online.
  • Stubbed out external services: GitHub, the cloud sandbox and the model provider are replaced by a frozen local snapshot and the model being trained. Combined with a total network lockdown at the sandbox level, HuggingFace should be safe from our experiments here!
  • Published every finding: OpenSWE's severity and rank filter is off, so the judge sees everything the model found.
  • Kept compaction, outside the loss: Summaries are counted with the trained model's tokenizer, and summary calls are not trained on.
  • Made the source read-only: An episode that changes the code scores -1.

An example trajectory

Below is a trajectory from a frozen review of better-auth/better-auth#11110. The agent follows a new adapter entry to the table component that renders it, records a low-severity finding about the link target, and completes a simulated publication.

TaskAgentTool callTool result
22 messages10 model calls130,684 tokens read · 827 output
1 message omitted
Task messagemessage 2
Review pull request https://github.com/better-auth/better-auth/pull/11110. - repo: better-auth/better-auth - pr_number: 11110 - title: - base_sha: 626adf67171923c9a83fa6fac8bfcbea2d540638 - head_sha: dc325b572e6f2a402c620b775ca8622b00a3efdf - base_ref: 626adf67171923c9a83fa6fac8bfcbea2d540638 - head_ref: dc325b572e6f2a402c620b775ca8622b00a3efdf Record each issue you find with the `add_finding` tool, then call `publish_review` once at the end. Review the local frozen checkout. Do not retrieve existing PR reviews or discussions. Do not modify source or apply fixes. Review the existing frozen PR snapshot. Repository dependencies and services are not installed. Use static source review, record every finding regardless of category or severity, and do not apply fixes or install project dependencies.
Agent → toolmessage 3
fetch_review_diff
{}
Tool result · fetch_review_diffmessage 4
{
  "success": true,
  "path": "/workspace/better-auth/review-diff-560d5213175644a2.patch",
  "bytes": 643,
  "files": [
    "docs/lib/community-adapters-data.ts"
  ],
  "file_count": 1,
  "files_truncated": false,
  "base_sha": "626adf67171923c9a83fa6fac8bfcbea2d540638",
  "head_sha": "dc325b572e6f2a402c620b775ca8622b00a3efdf",
  "cached": true
}
Agent → toolmessage 5
grep
{
  "pattern": "docs/lib/community-adapters-data.ts",
  "path": "/workspace/better-auth/review-diff-560d5213175644a2.patch",
  "glob": null,
  "output_mode": "content",
  "max_count": 10
}
Tool result · grepmessage 6
/workspace/repo/review-diff-560d5213175644a2.patch:
  1: diff --git a/docs/lib/community-adapters-data.ts b/docs/lib/community-adapters-data.ts
  3: --- a/docs/lib/community-adapters-data.ts
  4: +++ b/docs/lib/community-adapters-data.ts
Agent → toolmessage 7
read_file
{
  "file_path": "/workspace/better-auth/review-diff-560d5213175644a2.patch",
  "offset": 0,
  "limit": 100
}
Tool result · read_filemessage 8
review-diff-560d5213175644a2.patchlines 1-20 of 20
diff --git a/docs/lib/community-adapters-data.ts b/docs/lib/community-adapters-data.ts
index 9c7b61e..7291115 100644
--- a/docs/lib/community-adapters-data.ts
+++ b/docs/lib/community-adapters-data.ts
@@ -253,4 +253,15 @@ export const communityAdapters: CommunityAdapter[] = [
 			avatar: "https://github.com/ilbertt.png",
 		},
 	},
+	{
+		name: "@a77ay/better-auth-mikro-orm",
+		url: "https://www.npmjs.com/package/@a77ay/better-auth-mikro-orm",
+		database: "MikroORM",
+		databaseUrl: "https://mikro-orm.io/",
+		author: {
+			name: "A77AY",
+			url: "https://github.com/a77ay",
+			avatar: "https://github.com/a77ay.png",
+		},
+	},
 ];
NoteThe diff adds one adapter entry, including its link target.
Agent → toolmessage 9
read_file
{
  "file_path": "/workspace/better-auth/docs/lib/community-adapters-data.ts",
  "offset": 0,
  "limit": 270
}
Agent → toolmessage 9
grep
{
  "pattern": "communityAdapters",
  "path": "/workspace/better-auth",
  "glob": "*.ts",
  "output_mode": "content",
  "max_count": 20
}
Tool result · grepmessage 10
/workspace/repo/docs/lib/community-adapters-data.ts:
  13: export const communityAdapters: CommunityAdapter[] = [
Tool result · read_filemessage 11
docs/lib/community-adapters-data.tslines 1-267 of 267
export interface CommunityAdapter {
	name: string;
	url: string;
	database: string;
	databaseUrl: string;
	author: {
		name: string;
		url: string;
		avatar: string;
	};
}

export const communityAdapters: CommunityAdapter[] = [
	{
		name: "@convex-dev/better-auth",
		url: "https://github.com/get-convex/better-auth",
		database: "Convex",
		databaseUrl: "https://www.convex.dev/",
		author: {
			name: "erquhart",
			url: "https://github.com/erquhart",
			avatar: "https://github.com/erquhart.png",
		},
	},
	{
		name: "surreal-better-auth",
		url: "https://github.com/oskar-gmerek/surreal-better-auth",
		database: "SurrealDB",
		databaseUrl: "https://surrealdb.com/",
		author: {
			name: "Oskar Gmerek",
			url: "https://oskargmerek.com",
			avatar: "https://github.com/oskar-gmerek.png",
		},
	},
	{
		name: "surrealdb-better-auth",
		url: "https://github.com/Necmttn/surrealdb-better-auth",
		database: "SurrealDB",
		databaseUrl: "https://surrealdb.com/",
		author: {
			name: "Necmttn",
			url: "https://github.com/Necmttn",
			avatar: "https://github.com/Necmttn.png",
		},
	},
	{
		name: "better-auth-surrealdb",
		url: "https://github.com/msanchezdev/better-auth-surrealdb",
		database: "SurrealDB",
		databaseUrl: "https://surrealdb.com/",
		author: {
			name: "msanchezdev",
			url: "https://github.com/msanchezdev",
			avatar: "https://github.com/msanchezdev.png",
		},
	},
	{
		name: "payload-auth",
		url: "https://github.com/payload-auth/payload-auth",
		database: "Payload CMS",
		databaseUrl: "https://payloadcms.com/",
		author: {
			name: "forrestdevs",
			url: "https://github.com/forrestdevs",
			avatar: "https://github.com/forrestdevs.png",
		},
	},
	{
		name: "@delmaredigital/payload-better-auth",
		url: "https://github.com/delmaredigital/payload-better-auth",
		database: "Payload CMS",
		databaseUrl: "https://payloadcms.com/",
		author: {
			name: "Delmare Digital",
			url: "https://github.com/delmaredigital",
			avatar: "https://github.com/delmaredigital.png",
		},
	},
	{
		name: "@hedystia/better-auth-typeorm",
		url: "https://github.com/Zastinian/better-auth-typeorm",
		database: "TypeORM",
		databaseUrl: "https://typeorm.io/",
		author: {
			name: "Zastinian",
			url: "https://github.com/Zastinian",
			avatar: "https://github.com/Zastinian.png",
		},
	},
	{
		name: "better-auth-instantdb",
		url: "https://github.com/daveyplate/better-auth-instantdb",
		database: "InstantDB",
		databaseUrl: "https://www.instantdb.com/",
		author: {
			name: "daveycodez",
			url: "https://github.com/daveycodez",
			avatar: "https://github.com/daveycodez.png",
		},
	},
	{
		name: "@nerdfolio/remult-better-auth",
		url: "https://github.com/nerdfolio/remult-better-auth",
		database: "Remult",
		databaseUrl: "https://remult.dev/",
		author: {
			name: "Tai Vo",
			url: "https://github.com/taivo",
			avatar: "https://github.com/taivo.png",
		},
	},
	{
		name: "pocketbase-better-auth",
		url: "https://github.com/LightInn/pocketbase-better-auth",
		database: "PocketBase",
		databaseUrl: "https://pocketbase.io/",
		author: {
			name: "LightInn",
			url: "https://github.com/LightInn",
			avatar: "https://github.com/LightInn.png",
		},
	},
	{
		name: "better-auth-firestore",
		url: "https://github.com/yultyyev/better-auth-firestore",
		database: "Firebase Firestore",
		databaseUrl: "https://firebase.google.com/docs/firestore",
		author: {
			name: "yultyyev",
			url: "https://github.com/yultyyev",
			avatar: "https://github.com/yultyyev.png",
		},
	},
	{
		name: "@zenstackhq/better-auth",
		url: "https://github.com/zenstackhq/zenstack/tree/main/packages/auth-adapters/better-auth",
		database: "ZenStack",
		databaseUrl: "https://zenstack.dev",
		author: {
			name: "zenstackhq",
			url: "https://github.com/zenstackhq",
			avatar: "https://github.com/zenstackhq.png",
		},
	},
	{
		name: "@strapi-community/plugin-better-auth",
		url: "https://github.com/strapi-community/plugin-better-auth",
		database: "Strapi CMS",
		databaseUrl: "https://strapi.io/",
		author: {
			name: "boazpoolman",
			url: "https://github.com/boazpoolman",
			avatar: "https://github.com/boazpoolman.png",
		},
	},
	{
		name: "neo4j-better-auth",
		url: "https://github.com/florianamette/better-auth-neo4j",
		database: "Neo4j",
		databaseUrl: "https://neo4j.com/",
		author: {
			name: "florianamette",
			url: "https://github.com/florianamette",
			avatar: "https://github.com/florianamette.png",
		},
	},
	{
		name: "@lubiah/better-auth-mikro-orm",
		url: "https://github.com/lubiah/better-auth-mikro-orm",
		database: "MikroORM",
		databaseUrl: "https://mikro-orm.io/",
		author: {
			name: "lubiah",
			url: "https://github.com/lubiah",
			avatar: "https://github.com/lubiah.png",
		},
	},
	{
		name: "better-auth-mikro-orm",
		url: "https://github.com/octet-stream/better-auth-mikro-orm",
		database: "MikroORM",
		databaseUrl: "https://mikro-orm.io/",
		author: {
			name: "octet-stream",
			url: "https://github.com/octet-stream",
			avatar: "https://github.com/octet-stream.png",
		},
	},
	{
		name: "@proofkit/better-auth",
		url: "https://github.com/proofsh/proofkit/tree/main/packages/better-auth",
		database: "FileMaker OData",
		databaseUrl: "https://www.claris.com/filemaker/",
		author: {
			name: "eluce2",
			url: "https://github.com/eluce2",
			avatar: "https://github.com/eluce2.png",
		},
	},
	{
		name: "@datar-platform/better-auth-dynamodb",
		url: "https://github.com/datar-platform/better-auth-dynamodb",
		database: "DynamoDB",
		databaseUrl: "https://aws.amazon.com/dynamodb/",
		author: {
			name: "joesome-git",
			url: "https://github.com/joesome-git",
			avatar: "https://github.com/joesome-git.png",
		},
	},
	{
		name: "@bjorntech/betterauth-dynamodb",
		url: "https://github.com/bjorntech/betterauth-dynamodb",
		database: "DynamoDB",
		databaseUrl: "https://aws.amazon.com/dynamodb/",
		author: {
			name: "BjornTech AB",
			url: "https://github.com/bjorntech",
			avatar: "https://github.com/bjorntech.png",
		},
	},
	{
		name: "better-auth-azure-cosmos",
		url: "https://github.com/9hsein5/better-auth-azure-cosmos",
		database: "Azure Cosmos DB",
		databaseUrl: "https://learn.microsoft.com/azure/cosmos-db/nosql/",
		author: {
			name: "9hsein5",
			url: "https://github.com/9hsein5",
			avatar: "https://github.com/9hsein5.png",
		},
	},
	{
		name: "better-auth-mongoose",
		url: "https://github.com/AshwinSathian/better-auth-mongoose",
		database: "Mongoose",
		databaseUrl: "https://mongoosejs.com",
		author: {
			name: "Ashwin Sathian",
			url: "https://github.com/AshwinSathian",
			avatar: "https://github.com/AshwinSathian.png",
		},
	},
	{
		name: "@ilbertt/better-auth-bun-sql",
		url: "https://github.com/ilbertt/better-auth-bun-sql",
		database: "Bun SQL",
		databaseUrl: "https://bun.com/docs/api/sql",
		author: {
			name: "ilbertt",
			url: "https://github.com/ilbertt",
			avatar: "https://github.com/ilbertt.png",
		},
	},
	{
		name: "@a77ay/better-auth-mikro-orm",
		url: "https://www.npmjs.com/package/@a77ay/better-auth-mikro-orm",
		database: "MikroORM",
		databaseUrl: "https://mikro-orm.io/",
		author: {
			name: "A77AY",
			url: "https://github.com/a77ay",
			avatar: "https://github.com/a77ay.png",
		},
	},
];
NoteThe agent compares the new entry with the existing adapter list.
Agent → toolmessage 12
grep
{
  "pattern": "community-adapters-data",
  "path": "/workspace/better-auth",
  "glob": "*.tsx",
  "output_mode": "content",
  "max_count": 20
}
Tool result · grepmessage 13
/workspace/repo/docs/components/community-adapters-table.tsx:
  17: import type { CommunityAdapter } from "@/lib/community-adapters-data";
  18: import { communityAdapters } from "@/lib/community-adapters-data";
Agent → toolmessage 14
read_file
{
  "file_path": "/workspace/better-auth/docs/components/community-adapters-table.tsx",
  "offset": 0,
  "limit": 180
}
Tool result · read_filemessage 15
docs/components/community-adapters-table.tsxlines 1-180 of 211 | next offset 180
"use client";

import type {
	ColumnDef,
	ColumnFiltersState,
	SortingState,
} from "@tanstack/react-table";
import {
	flexRender,
	getCoreRowModel,
	getFilteredRowModel,
	getSortedRowModel,
	useReactTable,
} from "@tanstack/react-table";
import { ArrowUpDown, Search } from "lucide-react";
import { useState } from "react";
import type { CommunityAdapter } from "@/lib/community-adapters-data";
import { communityAdapters } from "@/lib/community-adapters-data";

const columns: ColumnDef<CommunityAdapter>[] = [
	{
		accessorKey: "name",
		header: ({ column }) => {
			return (
				<button
					type="button"
					className="flex items-center gap-2 font-semibold hover:text-foreground transition-colors whitespace-nowrap"
					onClick={() => column.toggleSorting(column.getIsSorted() === "asc")}
				>
					Adapter
					<ArrowUpDown className="h-4 w-4" />
				</button>
			);
		},
		cell: ({ row }) => {
			return (
				<div className="w-[260px]">
					<a
						href={row.original.url}
						target="_blank"
						rel="noopener noreferrer"
						className="inline-flex text-muted-foreground transition-colors hover:text-foreground"
					>
						<code className="text-sm font-mono tracking-tight whitespace-nowrap">
							{row.original.name}
						</code>
					</a>
				</div>
			);
		},
	},
	{
		accessorKey: "database",
		header: ({ column }) => {
			return (
				<button
					type="button"
					className="flex items-center gap-2 font-semibold hover:text-foreground transition-colors whitespace-nowrap"
					onClick={() => column.toggleSorting(column.getIsSorted() === "asc")}
				>
					Database
					<ArrowUpDown className="h-4 w-4" />
				</button>
			);
		},
		cell: ({ row }) => {
			return (
				<a
					href={row.original.databaseUrl}
					target="_blank"
					rel="noopener noreferrer"
					className="text-sm text-muted-foreground hover:text-foreground transition-colors whitespace-nowrap"
				>
					{row.original.database}
				</a>
			);
		},
	},
	{
		accessorFn: (adapter) => adapter.author.name,
		id: "author",
		header: ({ column }) => {
			return (
				<button
					type="button"
					className="flex items-center gap-2 font-semibold hover:text-foreground transition-colors whitespace-nowrap"
					onClick={() => column.toggleSorting(column.getIsSorted() === "asc")}
				>
					Author
					<ArrowUpDown className="h-4 w-4" />
				</button>
			);
		},
		cell: ({ row }) => {
			const author = row.original.author;

			return (
				<a
					href={author.url}
					target="_blank"
					rel="noopener noreferrer"
					className="flex items-center gap-2 hover:text-foreground transition-colors text-muted-foreground whitespace-nowrap"
				>
					<img
						src={author.avatar}
						alt={author.name}
						className="rounded-full w-6 h-6 border opacity-70 m-0"
					/>
					<span className="text-sm">{author.name}</span>
				</a>
			);
		},
	},
];

export function CommunityAdaptersTable() {
	const [sorting, setSorting] = useState<SortingState>([]);
	const [columnFilters, setColumnFilters] = useState<ColumnFiltersState>([]);
	const [globalFilter, setGlobalFilter] = useState("");

	const table = useReactTable({
		data: communityAdapters,
		columns,
		getCoreRowModel: getCoreRowModel(),
		getSortedRowModel: getSortedRowModel(),
		getFilteredRowModel: getFilteredRowModel(),
		onSortingChange: setSorting,
		onColumnFiltersChange: setColumnFilters,
		onGlobalFilterChange: setGlobalFilter,
		state: {
			sorting,
			columnFilters,
			globalFilter,
		},
	});

	return (
		<div className="not-prose w-full space-y-4">
			<div className="flex items-center justify-between text-xs text-muted-foreground">
				<p>
					Showing {table.getRowModel().rows.length} of{" "}
					{communityAdapters.length} adapters
				</p>
			</div>
			<div className="relative">
				<Search className="absolute left-3 top-1/2 h-4 w-4 -translate-y-1/2 text-muted-foreground" />
				<input
					type="text"
					aria-label="Search community adapters"
					placeholder="Search adapters, databases, or authors..."
					value={globalFilter ?? ""}
					onChange={(e) => setGlobalFilter(e.target.value)}
					className="w-full rounded-lg border bg-background pl-10 pr-4 py-2 text-sm focus:outline-none focus:ring-2 focus:ring-ring"
				/>
			</div>
			<div className="rounded-lg border">
				<div className="overflow-x-auto">
					<table className="w-full min-w-[720px]">
						<thead>
							{table.getHeaderGroups().map((headerGroup) => (
								<tr key={headerGroup.id} className="border-b bg-muted/50">
									{headerGroup.headers.map((header) => (
										<th
											key={header.id}
											className="px-4 py-3 text-left text-sm font-medium"
										>
											{header.isPlaceholder
												? null
												: flexRender(
														header.column.columnDef.header,
														header.getContext(),
													)}
										</th>
									))}
								</tr>
							))}
						</thead>
						<tbody>
							{table.getRowModel().rows?.length ? (
								table.getRowModel().rows.map((row) => (
NoteThe reviewer follows the changed data to the table component, which uses the adapter URL as the link target.
Agent → toolmessage 16
add_finding
{
  "severity": "low",
  "confidence": "high",
  "category": "correctness",
  "file": "docs/lib/community-adapters-data.ts",
  "title": "Adapter link targets npm instead of repository",
  "description": "The table renders `adapter.url` as the adapter's link. This entry points to the npm package page, while every other adapter entry in the table links to its source repository. Users clicking the adapter name get a package page instead of the adapter's code/repository, unlike the surrounding entries.",
  "start_line": 257,
  "end_line": 257,
  "suggestion": null,
  "side": "RIGHT"
}
Tool result · add_findingmessage 17
{
  "success": true,
  "finding_id": "f_b6a1f4e932",
  "duplicate": false
}
Agent → toolmessage 18
list_findings
{
  "status_filter": "open"
}
Tool result · list_findingsmessage 19
{
  "findings": [
    {
      "id": "f_b6a1f4e932",
      "severity": "low",
      "confidence": "high",
      "category": "correctness",
      "title": "Adapter link targets npm instead of repository",
      "file": "docs/lib/community-adapters-data.ts",
      "start_line": 257,
      "end_line": 257,
      "side": "RIGHT",
      "in_diff": true,
      "description": "The table renders `adapter.url` as the adapter's link. This entry points to the npm package page, while every other adapter entry in the table links to its source repository. Users clicking the adapter name get a package page instead of the adapter's code/repository, unlike the surrounding entries.",
      "suggestion": null,
      "status": "open",
      "first_seen_sha": "dc325b572e6f2a402c620b775ca8622b00a3efdf",
      "last_confirmed_sha": "dc325b572e6f2a402c620b775ca8622b00a3efdf",
      "github_review_id": null,
      "github_review_run_id": null,
      "github_review_comment_ids": [],
      "github_review_thread_ids": [],
      "github_resolved_thread_ids": [],
      "github_posted_resolution_comment_ids": [],
      "surface_state": "not_surfaced",
      "last_human_reply_at": null,
      "last_human_reply_author": null,
      "last_human_reply_body": null,
      "last_reconciliation_note": null,
      "resolution_note": null,
      "diff_hunk": "@@ -253,4 +253,15 @@ export const communityAdapters: CommunityAdapter[] = [\n \t\t\tavatar: \"https://github.com/ilbertt.png\",\n \t\t},\n \t},\n+\t{\n+\t\tname: \"@a77ay/better-auth-mikro-orm\",\n+\t\turl: \"https://www.npmjs.com/package/@a77ay/better-auth-mikro-orm\",\n+\t\tdatabase: \"MikroORM\",\n+\t\tdatabaseUrl: \"https://mikro-orm.io/\",\n+\t\tauthor: {\n+\t\t\tname: \"A77AY\",\n+\t\t\turl: \"https://github.com/a77ay\",\n+\t\t\tavatar: \"https://github.com/a77ay.png\",\n+\t\t},\n+\t},\n ];",
      "fingerprint": "v1:6e79f3ff80a6c223182a01535d7b1863d926ab234a4aca8c9f243344950cd245",
      "interactions": [],
      "rank": null
    }
  ],
  "count": 1
}
Agent → toolmessage 20
publish_review
{
  "ranking": [
    "f_b6a1f4e932"
  ],
  "severity_threshold": "medium",
  "assessment": null
}
Tool result · publish_reviewmessage 21
{
  "success": true,
  "dry_run": true,
  "review_id": null,
  "surfaced_count": 1,
  "hidden_count": 0,
  "resolved_thread_count": 0
}
NotePublication is simulated in this evaluation run; nothing is posted to GitHub.
Agentmessage 22
Low: Adapter link targets npm instead of repository: The adapter table renders adapter.url as the link, but this entry points to the npm package page rather than the adapter’s source repository like the other entries.

With the harness and environment set, we can move on to figuring out which data is usable for training.

Curating learnable data

Not every reviewed pull request makes a good RL task. If the task is too hard, the model never stumbles onto correct solutions and learns nothing, or it resorts to reward hacking. If it is too easy, the model solves it every time and there is no gradient. Some PRs only have dependabot comments, while others are merged without reviewer approval. So we filter on two principles: whether we can score the task honestly, and whether training on it will teach the model anything.

Can we score it honestly?

  1. Size: At most 50 commits and 2,000 changed lines, the same limits Martian uses.
  2. Reviewed commit known: We need the exact commit the bot saw. Without it we cannot split the history, and the answer key is undefined.
  3. At least one fix: A PR where the author fixed nothing after review has no answer key. Any review of it scores zero or is unscorable.
  4. No leaks: The frozen repository must not contain any later fix commit, in its objects, refs or image layers. If it did, the model could read the answer instead of reviewing the code, and RL would find that shortcut fast.

Will it move the policy?

We perform successive rounds of RL derisking by measuring the mean F2 of the model checkpoint due for training, and filtering out examples that score below 0.1 or above 0.9 across multiple trials. This is consistent with common industry practice, such as the rollout outcome filtering in Xiaomi's CodeMidas7 paper.

Training

Whitney AI partnered with Microsoft's Foundry team to train on their new Programmed Training8 offering as one of their first private preview customers. Foundry Programmed Training let us focus on the science and RL methodology while abstracting away the complexities of managing GPU infrastructure.

Intelligent rollout expansion

We apply a trick that 5x'd our training throughput. The Qwen3.8-27B base model turns out to be rather terrible at this review task. Higher reasoning levels saw longer, more expensive rollouts and frequently took us beyond our context limit, triggering many compaction calls. So we used Qwen3.8-27B (no thinking) as our base model. The no-thinking model had shorter trajectories, compacted less, and was much more token efficient when it completed.

But the no-thinking model was less likely to produce valid answers. It often got stuck in what we called tool-call "doom loops": failing to grok the result of a tool, realizing it still needed the information, and invoking the tool again, until the episode timed out or hit the max horizon.

Our initial mitigations were heuristics that detected doom loops based on repetitions and argument similarity. Subtle cases still crept through, like this cyclical sequence of three calls repeated until the horizon:

A cyclical doom loop, turns 18 to 29: the same three calls, four times
cycle 1
18read_file(file_path="/workspace/myotis/review-diff-dca732ef4c71516f.patch", offset=200, limit=200)
19grep(pattern="fn multiaddr_port", path="/workspace/myotis/rust/myotis-net/src", glob="*.rs", output_mode="content", max_count=10)
20list_findings(status_filter="open")
cycle 2
21read_file(file_path="/workspace/myotis/review-diff-dca732ef4c71516f.patch", offset=200, limit=200)
22grep(pattern="fn multiaddr_port", path="/workspace/myotis/rust/myotis-net/src", glob="*.rs", output_mode="content", max_count=10)
23list_findings(status_filter="open")
cycle 3
24read_file(file_path="/workspace/myotis/review-diff-dca732ef4c71516f.patch", offset=200, limit=200)
25grep(pattern="fn multiaddr_port", path="/workspace/myotis/rust/myotis-net/src", glob="*.rs", output_mode="content", max_count=10)
26list_findings(status_filter="open")
cycle 4
27read_file(file_path="/workspace/myotis/review-diff-dca732ef4c71516f.patch", offset=200, limit=200)
28grep(pattern="fn multiaddr_port", path="/workspace/myotis/rust/myotis-net/src", glob="*.rs", output_mode="content", max_count=10)
29list_findings(status_filter="open")

After cyclical sequences came semantic equivalencies. Every form of degenerate loop wasted compute on a rollout that was doomed to fail.

We ran Microsoft-Decision-19 with a tuned criteria set for detecting rollout degeneracy early and saw nearly perfect detection rates. So we added it at rollout time, querying it millions of times, once for every new turn of every trajectory, on a 16-turn lookback window. Effectively, we can now focus our sampling budget on the rollouts most likely to produce meaningful results. Examples terminated by Microsoft-Decision-1 are assigned a reward penalty to discourage the behavior during gradient optimization.

Algorithm selection

We run cold-start SFT for early model-side doom loop mitigation, and RL with intelligent rollout expansion powered by Microsoft-Decision-1 for hill-climbing review quality thereafter. For RL, we train with group-relative advantages: the model reviews each task four times, and each review's advantage is its reward minus the group's mean. If all four reviews score the same, every advantage is zero and the group contributes no gradient. Such groups are dropped and not used for training, meaning we may sample more groups each step than we actually use.

Data for the RL run is derisked using the SFT checkpoint. We run Microsoft-Decision-1 in the loop for derisking and evaluations as well.

Results

Completion-adjusted F2 counts model attempts that do not successfully return reviews as 0. Tokens, tool calls, and turns are over all evaluation results, inclusive of failures.
Model (reasoning level)Mean F2 (completed only)Completion rate (%)Completion-adjusted F2TokensTool callsTurns
Qwen3.8-27B (none)0.100047.50.0475274.46M75896955
+ SFT0.153790.00.138351.44M32202509
+ RLours0.187391.70.171821.72M20141098
gpt-6-luna (none)0.049595.00.047050.92M32712632
gpt-6-luna (high)0.183989.20.164097.76M84944353

The cold-start SFT is trained with 100 review trajectories from a luna-class teacher model, filtered to ensure all instances score above a minimal F2 threshold. The SFT makes a dramatic difference in boosting the completion rate of the Qwen3.8-27B model, which nearly doubles. Qualitative analysis further reveals the model learns a more efficient review workflow, outperforms at repo navigation, and better incorporates self-verification before finishing its reviews.

More efficient review workflows

Every recorded SFT read_file call supplied offset and limit arguments whereas the base model often omitted one or both. The base model consequently juggled more tokens per turn and was more likely to get lost in the context, degrading into tool doom loops. The SFT checkpoint also tended to make fewer repeated reads; repeated identical tool call count dropped 90.7% from 1204 to 112.

Better repo navigation

The base model tended to navigate the repo through repeated shell commands sent through execute, which runs a shell command inside the reviewer sandbox and returns its output. The SFT checkpoint, on the other hand, uses the dedicated grep and read_file tools from the harness. This adaptation translated into fewer turns spent on finding files and more time dedicated to reviewing code.

Better self-verification

In all but one of the SFT checkpoint’s evaluation traces it called the list_findings tool before publishing its review comments, which gave it the chance to review its work for errors and correct them before submitting the finalized review. The base model did this in less than half of its successful completions, so if errors did slip in, the model was less likely to find them.

Taken together with tool doom loop elimination, the agent became fivefold more token efficient at the task from SFT alone. Note this figure is a lower bound on the efficiency gains because we apply Microsoft-Decision-1 early termination across training and evals; had doomed trajectories not been terminated, the token consumption would have been even higher for the base model.

Cold-start SFT loss

Lower is better
Mean NLL0.150.200.250.3001020304050Optimizer updateOptimizer update 0: 0.303
  • Training batch loss (4 conversations per update)
  • Validation loss (same 16 held-out conversations)

RL nets us another +3pp in F2 after training on a small, highly filtered set of 150 examples for two epochs.

On deeper review, we see that SFT established the navigation workflow, and RL shifted it towards faster decisions, more candidate findings, and more self-correction. The pagination discipline is retained, with all read_file calls invoked with offset and limit parameters supplied. list_findings is called relatively less – present in 99% of SFT traces but only 74% of RL ones. Yet the RL model’s update_finding frequency increased 8x over the SFT checkpoint, meaning it got more comfortable with correcting its own perceived errors when it did decide to check for them.

For examples that both the SFT and RL checkpoints were able to complete, the RL model used 50.9% fewer tokens, 32.5% fewer tool calls, and 47.6% fewer turns. This nets us an overall 2.4x improvement in token efficiency from the RL run despite not explicitly designing for this.

To contextualize our Qwen3.8-27B numbers, we evaluate gpt-6-luna with no thinking and reasoning high; our lightly RL’d review agent slightly edges out the high reasoning profile while being 4.5x more token efficient.

Mean F2 during RL

Higher is better
Mean F20.200.250.300.350.400.450102030Update stepUpdate step 0: 0.365
  • Mean F2 (raw)
  • Smoothed
TaskAgentTool callTool result

SFTF2 0.0

147 messages68 model calls1,234,155 tokens read · 5,690 output
1 message omitted
Task messagemessage 2
Review pull request https://github.com/NVIDIA/TransformerEngine/pull/3141. - repo: NVIDIA/TransformerEngine - pr_number: 3141 - title: - base_sha: 43206d9729a8988858eee303c79afc7208f518cd - head_sha: 941db9b0d5457c986f6b9a06bd960d6186c8a592 - base_ref: 43206d9729a8988858eee303c79afc7208f518cd - head_ref: 941db9b0d5457c986f6b9a06bd960d6186c8a592 Record each issue you find with the `add_finding` tool, then call `publish_review` once at the end. Review the local frozen checkout. Do not retrieve existing PR reviews or discussions. Do not modify source or apply fixes. This image supports static code review. Repository dependencies are not installed and repository tests are not guaranteed runnable. Do not install dependencies or claim execution-based validation; use the frozen source and local Git history.
Agent → toolmessage 3
fetch_review_diff
{}
Tool result · fetch_review_diffmessage 4
{"success": true, "path": "/workspace/TransformerEngine/review-diff-434ed8e16a2a16ad.patch", "bytes": 8265, "files": ["tests/pytorch/test_backward_override.py", "transformer_engine/pytorch/module/grouped_linear.py", "transformer_engine/pytorch/module/linear.py"], "file_count": 3, "files_truncated": false, "base_sha": "43206d9729a8988858eee303c79afc7208f518cd", "head_sha": "941db9b0d5457c986f6b9a06bd960d6186c8a592", "cached": true}
Agent → toolmessage 5
glob
{
  "pattern": "AGENTS.md",
  "path": "/workspace/TransformerEngine"
}
Agent → toolmessage 5
glob
{
  "pattern": "CLAUDE.md",
  "path": "/workspace/TransformerEngine"
}
Agent → toolmessage 5
read_file
{
  "file_path": "/workspace/TransformerEngine/review-diff-434ed8e16a2a16ad.patch",
  "offset": 0,
  "limit": 300
}
Tool result · globmessage 7
No files found
Tool result · globmessage 8
No files found
Tool result · read_filemessage 6
/workspace/TransformerEngine/review-diff-434ed8e16a2a16ad.patchlines 1-206
@@ lines 1-206 of 206 @@
diff --git a/tests/pytorch/test_backward_override.py b/tests/pytorch/test_backward_override.py
index 5e6f36e..ac05cc1 100644
--- a/tests/pytorch/test_backward_override.py
+++ b/tests/pytorch/test_backward_override.py
@@ -858,6 +858,175 @@ def test_backward_override_recipe_matches_requested_mode(
     assert quant_recipe.backward_override is None
 
 
+@pytest.mark.parametrize("recipe_name", _quantized_numerics_recipe_list)
+@pytest.mark.parametrize("use_bias", (False, True), ids=("no_bias", "bias"))
+def test_linear_backward_override_dequantized_ignores_save_original_input(
+    recipe_name: str,
+    use_bias: bool,
+) -> None:
+    reset_rng_states()
+    dtype = torch.bfloat16
+    input_shape = (32, 128)
+    out_features = 128
+    _maybe_skip_recipe_dtype(recipe_name, dtype, "linear")
+    _maybe_skip_unsupported_recipe_module_combo(recipe_name, "linear")
+    _maybe_skip_unsupported_recipe_shape(recipe_name, input_shape, "linear")
+
+    mode_recipe = make_recipe(recipe_name, backward_override="dequantized")
+    skip_unsupported_backward_override("linear", mode_recipe, "dequantized")
+
+    module_ref = te.Linear(
+        input_shape[-1],
+        out_features,
+        bias=use_bias,
+        params_dtype=dtype,
+        device="cuda",
+        save_original_input=False,
+    )
+    module_test = te.Linear(
+        input_shape[-1],
+        out_features,
+        bias=use_bias,
+        params_dtype=dtype,
+        device="cuda",
… 167 more lines
Agent → toolmessage 9
grep
{
  "pattern": "save_original_input",
  "path": "/workspace/TransformerEngine/transformer_engine/pytorch/module",
  "glob": "*.py",
  "output_mode": "content",
  "max_count": 100
}
Agent → toolmessage 9
read_file
{
  "file_path": "/workspace/TransformerEngine/transformer_engine/pytorch/module/grouped_linear.py",
  "offset": 350,
  "limit": 250
}
Agent → toolmessage 9
read_file
{
  "file_path": "/workspace/TransformerEngine/transformer_engine/pytorch/module/linear.py",
  "offset": 200,
  "limit": 250
}
Tool result · grepmessage 10
/workspace/repo/transformer_engine/pytorch/module/grouped_linear.py:
  94:         save_original_input: bool,
  125:             or save_original_input
  387:             ctx.save_original_input = False
  425:             save_original_input,
  433:             save_original_input = True
  435:             save_original_input = False
  444:         if save_original_input and isinstance(input_quantizers[0], Float8Quantizer):
  450:                     "save_original_input is incompatible with delayed-scaling quantizers "
  451:                     "(Float8Quantizer). Disabling save_original_input for this module.",
  454:                 save_original_input = False
  456:                 raise ValueError("DelayedScaling recipe is not supported with save_original_input")
  464:                         and not save_original_input
  504:             save_original_input=save_original_input,
  626:                 if save_original_input:
  703:             ctx.save_original_input = save_original_input
  1066:                 if ctx.save_original_input:
  1229:     save_original_input : bool, default = False
  1275:         save_original_input: bool = False,
  1293:         self.save_original_input = save_original_input
  1787:                 self.save_original_input,
/workspace/repo/transformer_engine/pytorch/module/linear.py:
  118:     save_original_input: bool
  282:     save_original_input = args.save_original_input
  287:         save_original_input = True
  289:         save_original_input = False
  327:         if save_original_input:
  330:             ), "DelayedScaling recipe is not supported with save_original_input"
  347:                 if save_original_input:
  386:                         and not save_original_input
  543:         if save_original_input:
  1456:     save_original_input : bool, default = False
  1488:         save_original_input: bool = False,
  1503:         self.save_original_input = save_original_input
  1920:                 save_original_input=self.save_original_input,
Tool result · read_filemessage 12
/workspace/TransformerEngine/transformer_engine/pytorch/module/grouped_linear.pylines 351-600
@@ lines 351-600 of 1979 | next offset 600 @@
            ctx.grad_input_quantizers = grad_input_quantizers
            ctx.grad_output_quantizers = grad_output_quantizers
            ctx.grad_weight_quantizers = grad_weight_quantizers
            ctx.weights_requires_grad = weight_requires_grad
            if fuse_wgrad_accumulation and ctx.weights_requires_grad:
                ctx.origin_weight_refs = [weakref.ref(w) for w in weights]
                ctx.origin_weights_overwrite_main_grad = getattr(
                    weights[0], "overwrite_main_grad", False
                )
                if hasattr(weights[0], "__fsdp_param__"):
                    ctx.main_grad_funcs = [weights[i].get_main_grad for i in range(num_gemms)]
                else:
                    ctx.main_grad_funcs = [
                        lambda j=i: weights[j].main_grad for i in range(num_gemms)
                    ]
            ctx.device = device
            ctx.m_splits = None
            ctx.num_gemms = num_gemms
            ctx.activation_dtype = activation_dtype
            ctx.fp8 = fp8
            ctx.fp8_recipe = FP8GlobalStateManager.get_fp8_recipe() if fp8 else None
            ctx.backward_override = None
            ctx.fuse_wgrad_accumulation = fuse_wgrad_accumulation
            ctx.cpu_offloading = False
            ctx.is_first_microbatch = is_first_microbatch
            ctx.use_bias = use_bias
            ctx.inp_shape = inp.shape
            ctx.requires_dgrad = inp.requires_grad
            ctx.reduce_and_update_bwd_fp8_tensors = False
            if ctx.fp8 and requires_grad(inp, weights[0], biases[0]):
                ctx.reduce_and_update_bwd_fp8_tensors = (
                    ctx.reduce_and_update_bwd_fp8_tensors
                    or FP8GlobalStateManager.is_first_fp8_module()
                )
            ctx.wgrad_store = wgrad_store
            ctx.debug = False
            ctx.save_original_input = False
            ctx.input_quantizers = input_quantizers
… 211 more lines
Tool result · read_filemessage 11
/workspace/TransformerEngine/transformer_engine/pytorch/module/linear.pylines 201-450
@@ lines 201-450 of 2171 | next offset 450 @@
    ub_name: Optional[str] = None
    ub_overlap_ag: bool = False
    ub_overlap_rs_dgrad: bool = False
    ub_bulk_dgrad: bool = False
    ub_bulk_wgrad: bool = False

    # --- FSDP ---
    fsdp_group: Optional[Any] = None
    fsdp_shapes: Any = None
    is_fsdp2: bool = False

    # --- Weight-grad scheduling / accumulation ---
    is_first_microbatch: Optional[bool] = None
    fuse_wgrad_accumulation: bool = False
    wgrad_store: Optional[Any] = None
    origin_weight_ref: Optional[Any] = None
    origin_weight_overwrites_main_grad: bool = False
    main_grad_func: Optional[Callable[[], torch.Tensor]] = None

    # --- FP8 reduce-and-update bookkeeping ---
    reduce_and_update_bwd_fp8_tensors: bool = False

    # --- Misc ---
    cpu_offloading: bool = False
    owns_input: bool = False

    # --- Per-backward scratch state (populated inside _linear_backward) ---
    ub_obj_gradout: Optional[Any] = None

    def setup_saved_tensors(self, ctx: torch.autograd.function.FunctionCtx) -> None:
        """Pull saved tensors from ``ctx`` into the fields backward consumes."""
        (
            self.inputmat,
            self.weight_fp8,
            self.saved_weight,
            self.bias,
        ) = restore_from_func_ctx(
            ctx
        )  # pylint: disable=unbalanced-tuple-unpacking
… 210 more lines
Agent → toolmessage 13
grep
{
  "pattern": "backward_override",
  "path": "/workspace/TransformerEngine/transformer_engine/pytorch/module",
  "glob": "*.py",
  "output_mode": "content",
  "max_count": 200
}
Agent → toolmessage 13
read_file
{
  "file_path": "/workspace/TransformerEngine/transformer_engine/pytorch/module/grouped_linear.py",
  "offset": 600,
  "limit": 180
}
Agent → toolmessage 13
read_file
{
  "file_path": "/workspace/TransformerEngine/transformer_engine/pytorch/module/linear.py",
  "offset": 450,
  "limit": 200
}
Tool result · grepmessage 16
/workspace/repo/transformer_engine/pytorch/module/base.py:
  1577:         use_fp8_bwd = ctx.fp8 and ctx.backward_override is None
/workspace/repo/transformer_engine/pytorch/module/grouped_linear.py:
  93:         backward_override: Optional[str],
  124:             or backward_override is not None
  372:             ctx.backward_override = None
  429:             backward_override = FP8GlobalStateManager.get_fp8_recipe().backward_override
  431:             backward_override = None
  432:         if backward_override == "high_precision":
  434:         elif backward_override == "dequantized":
  465:                         and backward_override is None
  469:             if backward_override is not None:
  503:             backward_override=backward_override,
  632:                             if backward_override is not None:
  646:                 if backward_override == "high_precision" and inp.requires_grad
  687:             ctx.backward_override = backward_override
  707:             if backward_override is not None:
  1011:                 if ctx.backward_override == "dequantized":
  1020:                 elif ctx.backward_override == "high_precision":
  1093:                 elif ctx.backward_override == "dequantized":
/workspace/repo/transformer_engine/pytorch/module/layernorm_linear.py:
  150:             backward_override = FP8GlobalStateManager.get_fp8_recipe().backward_override
  152:             backward_override = None
  213:                 columnwise=backward_needs_input and backward_override is None,
  227:             and backward_override is None
  251:         ln_out_hp = ln_out if backward_override == "high_precision" else None
  319:                     columnwise=is_grad_enabled and not is_fsdp2 and backward_override is None,
  441:             if backward_override == "high_precision":
  449:             if backward_needs_input and backward_override is None:
  533:             ctx.backward_override = backward_override
  566:             if backward_override is not None:
  720:             if ctx.backward_override == "dequantized":
  810:             if ctx.backward_override == "dequantized":
  815:             elif ctx.backward_override == "high_precision":
/workspace/repo/transformer_engine/pytorch/module/layernorm_mlp.py:
  246:             backward_override = FP8GlobalStateManager.get_fp8_recipe().backward_override
  248:             backward_override = None
  249:         assert backward_override is None, (
  845:             ctx.backward_override = backward_override
/workspace/repo/transformer_engine/pytorch/module/linear.py:
… 19 more lines
Tool result · read_filemessage 15
/workspace/TransformerEngine/transformer_engine/pytorch/module/grouped_linear.pylines 601-780
@@ lines 601-780 of 1979 | next offset 780 @@
            [out],
            output_quantizers,
            activation_dtype,
            single_output=True,
            m_splits=m_splits,
            bias=biases,
            use_bias=use_bias,
            use_split_accumulator=use_split_accumulator,
        )

        if fp8_calibration:
            for i in range(num_gemms):
                input_quantizers[i].calibrate(inputmats[i])
                weight_quantizers[i].calibrate(weights[i])

        if cpu_offloading:
            mark_not_offload(*weights_fp8, *weights)

        if is_grad_enabled:
            ctx.use_grouped_tensor_path = False
            ctx.weight_quantizers = weight_quantizers
            ctx.weights_shape_1 = weights[0].shape[1]

            # TODO: update after #1638 is merged. # pylint: disable=fixme
            if weight_requires_grad:
                if save_original_input:
                    inputmats = [None] * num_gemms
                    inputmats[0] = inp
                else:
                    for inputmat in inputmats:
                        if isinstance(inputmat, QuantizedTensorStorage):
                            if backward_override is not None:
                                # In dequantized mode we should dequantize directly from
                                # fprop quantized layouts without retargeting usage.
                                inputmat.update_usage(rowwise_usage=True, columnwise_usage=False)
                            else:
                                inputmat.update_usage(rowwise_usage=False, columnwise_usage=True)
            else:
                inputmats = [None] * num_gemms
… 141 more lines
Tool result · read_filemessage 14
/workspace/TransformerEngine/transformer_engine/pytorch/module/linear.pylines 451-650
@@ lines 451-650 of 2171 | next offset 650 @@
    # Calibrate quantizers if needed
    if not fp8 and args.fp8_calibration:
        if input_quantizer is not None:
            input_quantizer.calibrate(inputmat_total)
        if weight_quantizer is not None:
            weight_quantizer.calibrate(weight)

    # Choose whether to use GEMM kernel with split accumulator
    use_split_accumulator = _2X_ACC_FPROP
    if fp8:
        recipe = FP8GlobalStateManager.get_fp8_recipe()
        if hasattr(recipe, "fp8_gemm_fprop"):
            use_split_accumulator = recipe.fp8_gemm_fprop.use_split_accumulator

    # Configure output quantizer
    if output_quantizer is not None:
        output_quantizer.set_usage(rowwise=True, columnwise=False)

    # Output buffer for Userbuffers reduce-scatter
    reduce_scatter_out = None
    if ub_overlap_rs_fprop:
        out_shape = list(inp.shape)
        out_shape[0] //= tp_world_size
        out_shape[-1] = out_features
        reduce_scatter_out = torch.empty(out_shape, dtype=activation_dtype, device=inp.device)

    # ------------------------------------------------------
    # Forward GEMM
    # Note: y = x * w^T
    # ------------------------------------------------------
    nvtx_range_push(f"{nvtx_label}.gemm")
    gemm_out, *_, reduce_scatter_out = general_gemm(
        weightmat,
        inputmat_total,
        quantization_params=output_quantizer,
        out_dtype=activation_dtype,
        bias=bias,
        use_split_accumulator=use_split_accumulator,
        ub=ub_obj,
… 161 more lines
Agent → toolmessage 17
read_file
{
  "file_path": "/workspace/TransformerEngine/transformer_engine/pytorch/module/linear.py",
  "offset": 900,
  "limit": 200
}
Tool result · read_filemessage 18
/workspace/TransformerEngine/transformer_engine/pytorch/module/linear.pylines 901-1100
@@ lines 901-1100 of 2171 | next offset 1100 @@
            if bwd_args.fp8 or bwd_args.debug:
                quantizer = input_quantizer
                if quantizer.supports_only_rowwise_all_gather():
                    # If data is in FP8, we compute FP8 transposes manually
                    quantizer.set_usage(rowwise=True, columnwise=False)
                else:
                    # wgrad GEMM requires input with column-wise usage
                    quantizer.set_usage(rowwise=False, columnwise=True)
            if bwd_args.ub_bulk_dgrad:
                inputmat_total, _ = fill_userbuffers_buffer_for_all_gather(
                    ub_obj_dgrad,
                    inputmat,
                    quantizer,
                    bwd_args.tp_group,
                )
            else:
                nvtx_range_push(f"{nvtx_label}.column_parallel_comm_input")
                inputmat_total, inputmat_total_work = gather_along_first_dim(
                    inputmat,
                    bwd_args.tp_group,
                    async_op=True,
                    quantizer=quantizer,
                )
                nvtx_range_pop(f"{nvtx_label}.column_parallel_comm_input")
        else:
            inputmat_total = inputmat
        # --------------------------------------------------
        # Input tensor is ready for computing grad weight...
        # --------------------------------------------------

        # --------------------------------------------------
        # Compute grad input tensor
        # --------------------------------------------------

        dgrad = None
        dgrad_work = None
        if bwd_args.requires_dgrad:

            # FSDP2: Re-create workspace from all-gathered weight when
… 161 more lines
Agent → toolmessage 19
read_file
{
  "file_path": "/workspace/TransformerEngine/transformer_engine/pytorch/module/linear.py",
  "offset": 1100,
  "limit": 200
}
Tool result · read_filemessage 20
/workspace/TransformerEngine/transformer_engine/pytorch/module/linear.pylines 1101-1300
@@ lines 1101-1300 of 2171 | next offset 1300 @@
                # convert row-scaled MXFP8 to column-scaled, so we
                # can't reuse the grad output that was gathered
                # for the dgrad GEMM. We work around by explicitly
                # overlapping the AG operation with the dgrad GEMM.

                # Get the communication stream from the dgrad GEMM to use for the AG
                dgrad_send_stream, dgrad_recv_stream = ub_obj_dgrad.get_communication_stream()

                # This object is separate from the ub_obj_wgrad object which is passed to the GEMM
                ub_obj_overlap_wgrad = get_ub(bwd_args.ub_name + "_wgrad", bwd_args.fp8)

                grad_output_quantizer.set_usage(rowwise=False, columnwise=True)

                # We use the send stream to copy into the userbuffers.
                # This is the same stream that we will use to access the data in the AG,
                # so we dont need to add any syncs yet.
                with torch.cuda.stream(dgrad_send_stream):
                    grad_output, _ = fill_userbuffers_buffer_for_all_gather(
                        ub_obj_overlap_wgrad,
                        grad_output_arg,
                        grad_output_quantizer,
                        bwd_args.tp_group,
                    )

                # Allgather grad_outputs[0] using the dgrad streams so we can overlap with the fc2_dgrad gemm
                tex.bulk_overlap_ag_with_external_gemm(
                    ub_obj_overlap_wgrad, dgrad_send_stream, dgrad_recv_stream
                )

            if bwd_args.fp8 or bwd_args.debug:
                if isinstance(grad_output, QuantizedTensorStorage):
                    grad_output.update_usage(columnwise_usage=True)
                else:
                    grad_output_quantizer.set_usage(rowwise=False, columnwise=True)
                    grad_output = grad_output_quantizer(grad_output)

            # Figure out whether to use split accumulator
            use_split_accumulator = _2X_ACC_WGRAD
            if bwd_args.fp8:
… 161 more lines
Agent → toolmessage 21
read_file
{
  "file_path": "/workspace/TransformerEngine/transformer_engine/pytorch/module/linear.py",
  "offset": 1300,
  "limit": 150
}
Tool result · read_filemessage 22
/workspace/TransformerEngine/transformer_engine/pytorch/module/linear.pylines 1301-1450
@@ lines 1301-1450 of 2171 | next offset 1450 @@
        inp: torch.Tensor,
        bias: Optional[torch.Tensor],
        fwd_args: LinearFwdArgs,
    ) -> Tuple[torch.Tensor, Optional[torch.Tensor]]:
        """Forward pass: compute linear output and set up autograd context.

        ``weight``, ``inp`` and ``bias`` are positional Tensor arguments so
        autograd tracks them; they are immediately re-attached to ``fwd_args``
        so every downstream helper can be invoked with a single argument.

        ``weight_workspace`` is intentionally NOT a positional input: it is a
        non-differentiable cached tensor passed in via
        ``fwd_args.weight_workspace`` and the freshly produced workspace is
        returned as a separate output so the module can refresh its cache.
        """
        fwd_args.weight = weight
        fwd_args.inp = inp
        fwd_args.bias = bias
        (
            out,
            new_weight_workspace,
            tensors_to_save_from_forward,
            _,
            ctx_attrs,
        ) = _linear_forward_impl(fwd_args)
        if ctx is not None:
            bwd_args = LinearBwdArgs()
            tensors_to_save_from_setup = _linear_setup_ctx(
                bwd_args,
                fwd_args,
                out,
                ctx_attrs,
                tensors_to_save_from_forward,
            )
            tensors_to_save, tensor_objects = prepare_for_saving(*tensors_to_save_from_setup)
            ctx.save_for_backward(*tensors_to_save)
            ctx.tensor_objects = tensor_objects
            ctx.backward_objects = bwd_args
            if fwd_args.fp8 and (
… 111 more lines
Agent → toolmessage 23
read_file
{
  "file_path": "/workspace/TransformerEngine/transformer_engine/pytorch/module/linear.py",
  "offset": 0,
  "limit": 100
}
Tool result · read_filemessage 24
/workspace/TransformerEngine/transformer_engine/pytorch/module/linear.pylines 1-100
@@ lines 1-100 of 2171 | next offset 100 @@
# Copyright (c) 2022-2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved.
#
# See LICENSE for license information.

"""Linear API"""
from dataclasses import dataclass
from typing import Any, Callable, Dict, Optional, Tuple, Union, List
from functools import reduce
from operator import mul as multiply_op
import warnings
import weakref

import torch

import transformer_engine_torch as tex

from transformer_engine.common.recipe import Recipe
from transformer_engine.pytorch.torch_version import torch_version

from .base import (
    fill_userbuffers_buffer_for_all_gather,
    get_dummy_wgrad,
    get_ub,
    is_ub_initialized,
    using_cublasmp_backend,
    quantize_weight,
    TransformerEngineBaseModule,
    _2X_ACC_FPROP,
    _2X_ACC_DGRAD,
    _2X_ACC_WGRAD,
)
from ._common import noop_cat, WeightGradStore
from ..quantization import FP8GlobalStateManager, QuantizerRole
from ..utils import (
    cast_if_needed,
    clear_tensor_data,
    divide,
    init_method_constant,
    needs_quantized_gemm,
… 61 more lines
Agent → toolmessage 25
read_file
{
  "file_path": "/workspace/TransformerEngine/transformer_engine/pytorch/module/linear.py",
  "offset": 100,
  "limit": 150
}
Tool result · read_filemessage 26
/workspace/TransformerEngine/transformer_engine/pytorch/module/linear.pylines 101-250
@@ lines 101-250 of 2171 | next offset 250 @@
    input_requires_grad: bool
    weight_requires_grad: bool
    bias_requires_grad: bool

    # --- Quantizers ---
    input_quantizer: Optional[Quantizer]
    weight_quantizer: Optional[Quantizer]
    output_quantizer: Optional[Quantizer]
    grad_input_quantizer: Optional[Quantizer]
    grad_weight_quantizer: Optional[Quantizer]
    grad_output_quantizer: Optional[Quantizer]

    # --- Numerical / dtype config ---
    activation_dtype: torch.dtype
    fp8: bool
    fp8_calibration: bool
    fp8_output: bool
    save_original_input: bool
    backward_override: Optional[str]
    custom: bool
    debug: bool

    # --- Weight-workspace caching ---
    is_first_microbatch: Optional[bool]
    cache_weight: bool
    skip_fp8_weight_update: Optional[bool]

    # --- Tensor / sequence parallelism ---
    parallel_mode: Optional[str]
    tp_group: Optional[Any]
    tp_size: int
    tensor_parallel: bool
    sequence_parallel: bool
    symmetric_ar_type: Optional[str]
    backward_input_needs_gather: bool

    # --- Userbuffers (comm + GEMM overlap) ---
    ub_name: Optional[str]
    ub_overlap_ag_fprop: bool
… 110 more lines
Agent → toolmessage 27
read_file
{
  "file_path": "/workspace/TransformerEngine/transformer_engine/pytorch/module/linear.py",
  "offset": 250,
  "limit": 150
}
Tool result · read_filemessage 28
/workspace/TransformerEngine/transformer_engine/pytorch/module/linear.pylines 251-400
@@ lines 251-400 of 2171 | next offset 400 @@

def _linear_forward_impl(
    args: LinearFwdArgs,
) -> Tuple[torch.Tensor, Optional[torch.Tensor], Optional[Tuple], None, Optional[Dict]]:
    """Forward implementation for the linear layer.

    Returns ``(out, new_weight_workspace, tensors_to_save_from_forward, None,
    ctx_attrs)``. ``new_weight_workspace`` is the freshly produced FP8 weight
    workspace (returned alongside ``out`` so the caller can refresh its
    cache). The last three are ``None`` when gradients are disabled.
    """

    weight = args.weight
    inp = args.inp
    bias = args.bias
    input_quantizer = args.input_quantizer
    weight_quantizer = args.weight_quantizer
    output_quantizer = args.output_quantizer
    is_first_microbatch = args.is_first_microbatch
    fp8 = args.fp8
    cpu_offloading = args.cpu_offloading
    tp_group = args.tp_group
    sequence_parallel = args.sequence_parallel
    activation_dtype = args.activation_dtype
    parallel_mode = args.parallel_mode
    is_grad_enabled = args.is_grad_enabled
    ub_overlap_rs_fprop = args.ub_overlap_rs_fprop
    ub_overlap_ag_fprop = args.ub_overlap_ag_fprop
    ub_name = args.ub_name
    fsdp_group = args.fsdp_group
    symmetric_ar_type = args.symmetric_ar_type
    save_original_input = args.save_original_input
    debug = args.debug
    backward_override = args.backward_override
    is_fsdp2 = args.is_fsdp2
    if backward_override == "high_precision":
        save_original_input = True
    elif backward_override == "dequantized":
        save_original_input = False
… 111 more lines
Agent → toolmessage 29
read_file
{
  "file_path": "/workspace/TransformerEngine/transformer_engine/pytorch/module/linear.py",
  "offset": 400,
  "limit": 150
}
Tool result · read_filemessage 30
/workspace/TransformerEngine/transformer_engine/pytorch/module/linear.pylines 401-550
@@ lines 401-550 of 2171 | next offset 550 @@
    # ------------------------------------------------------

    # ------------------------------------------------------
    # Prepare weight tensor
    # ------------------------------------------------------
    new_weight_workspace = None
    weightmat = weight
    if fp8 or debug:
        # Configure quantizer
        # No need to set the quantizer states if weight is already quantized
        # for debug mode we create quantizer every iteration, thus we need to set the quantizer states
        if weight_quantizer is not None and (not isinstance(weight, QuantizedTensor) or debug):
            columnwise_usage = is_grad_enabled and inp.requires_grad and not is_fsdp2
            if backward_override is not None:
                columnwise_usage = False
            if not columnwise_usage:
                columnwise_usage = (
                    is_fp8_activation_recompute_enabled()
                    and not in_fp8_activation_recompute_phase()
                )
            weight_quantizer.set_usage(rowwise=True, columnwise=columnwise_usage)
        elif isinstance(weight, QuantizedTensor):
            weight_quantizer = weight._quantizer
        # Get quantized weight
        update_ws = is_first_microbatch is None or is_first_microbatch
        weightmat, new_weight_workspace = quantize_weight(
            tensor=weight,
            quantizer=weight_quantizer,
            workspace=args.weight_workspace,
            update_workspace=update_ws,
            skip_update_flag=args.skip_fp8_weight_update,
            fsdp_group=fsdp_group,
            workspace_dtype=activation_dtype,
            cache=args.cache_weight,
        )
        weightmat.update_usage(rowwise_usage=True)

    else:
        weightmat = cast_if_needed(weightmat, activation_dtype)  # Cast for AMP
… 111 more lines
Agent → toolmessage 31
read_file
{
  "file_path": "/workspace/TransformerEngine/transformer_engine/pytorch/module/linear.py",
  "offset": 550,
  "limit": 150
}
Tool result · read_filemessage 32
/workspace/TransformerEngine/transformer_engine/pytorch/module/linear.pylines 551-700
@@ lines 551-700 of 2171 | next offset 700 @@
        ):
            if backward_override is not None:
                inputmat.update_usage(rowwise_usage=True, columnwise_usage=False)
            elif (
                args.backward_input_needs_gather
                and weight_quantizer.supports_only_rowwise_all_gather()
            ):
                # All-gather is not supported with FP8 column-wise data
                inputmat.update_usage(rowwise_usage=True, columnwise_usage=False)
            else:
                # Discard row-wise data since it is not needed in backward pass
                inputmat.update_usage(rowwise_usage=False, columnwise_usage=True)

        # Cached input tensor
        saved_inputmat = None
        if backward_needs_input:
            saved_inputmat = inputmat

        if cpu_offloading and saved_inputmat is not None:
            mark_activation_offload(saved_inputmat)

        # Scatter intermediate/activation tensors saved for the backward pass
        # NOTE: FSDP sharding is not valid for models initialized with primary Fp8 weights
        nvtx_range_push(f"{nvtx_label}.fsdp_scatter")
        fsdp_shapes = _fsdp_scatter_tensors(
            fsdp_group,
            saved_inputmat,
            weightmat if fp8 and not isinstance(weight, QuantizedTensorStorage) else None,
        )
        nvtx_range_pop(f"{nvtx_label}.fsdp_scatter")

        if cpu_offloading:
            mark_not_offload(weight, weightmat, bias)

        # TODO(ksivamani): Check memory usage
        # FSDP2: Don't save FP8 workspace for non-quantized weights.
        # Backward will re-quantize from the FSDP2 all-gathered weight.
        # (Issue #2681)
        wt_save = weightmat
… 111 more lines
Agent → toolmessage 33
read_file
{
  "file_path": "/workspace/TransformerEngine/transformer_engine/pytorch/module/linear.py",
  "offset": 700,
  "limit": 150
}
Tool result · read_filemessage 34
/workspace/TransformerEngine/transformer_engine/pytorch/module/linear.pylines 701-850
@@ lines 701-850 of 2171 | next offset 850 @@

    if backward_override is not None:
        bwd_args.fp8 = False
        bwd_args.debug = False
        bwd_args.ub_overlap_ag = False
        bwd_args.ub_overlap_rs_dgrad = False
        bwd_args.ub_bulk_dgrad = False
        bwd_args.ub_bulk_wgrad = False
        bwd_args.grad_input_quantizer = None
        bwd_args.grad_weight_quantizer = None
        bwd_args.grad_output_quantizer = None

    saved_inputmat, wt_save, saved_weight, saved_bias = tensors_to_save_from_forward
    inputmat_alias, wt_save_alias, saved_weight_alias, bias_alias = ctx_attrs[
        "saved_tensor_aliases"
    ]
    bwd_args.owns_input = inputmat_alias != "inp"
    if inputmat_alias == "inp":
        saved_inputmat = inp
    if wt_save_alias == "weight":
        wt_save = weight
    if saved_weight_alias == "weight":
        saved_weight = weight
    if bias_alias == "bias":
        saved_bias = bias
    return (saved_inputmat, wt_save, saved_weight, saved_bias)


def _linear_backward(args: LinearBwdArgs) -> Tuple[Union[torch.Tensor, None], ...]:
    """Backward implementation for the linear layer.

    Caller must have populated ``args.grad_output`` and run
    ``args.setup_saved_tensors(ctx)`` before invocation.
    """
    bwd_args = args
    grad_output = args.grad_output
    assert grad_output is not None
    inputmat = args.inputmat
    weight_fp8 = args.weight_fp8
… 111 more lines
Agent → toolmessage 35
read_file
{
  "file_path": "/workspace/TransformerEngine/transformer_engine/pytorch/module/linear.py",
  "offset": 850,
  "limit": 150
}
Tool result · read_filemessage 36
/workspace/TransformerEngine/transformer_engine/pytorch/module/linear.pylines 851-1000
@@ lines 851-1000 of 2171 | next offset 1000 @@
        (
            grad_output,
            grad_bias,
        ) = TransformerEngineBaseModule.grad_output_preprocess(
            bwd_args,
            grad_output,
            bwd_args.parallel_mode == "row",
            grad_output_quantizer,
        )
        nvtx_range_pop(f"{nvtx_label}.grad_output_preprocess")

        # --------------------------------------------------
        # Grad output tensor is ready for computing grad input...
        # --------------------------------------------------

        # --------------------------------------------------
        # Prepare input tensor
        # Note: Input tensor is needed for wgrad GEMM.
        # Tensor-parallel communication is overlapped with dgrad
        # GEMM.
        # --------------------------------------------------
        inputmat_total = None
        inputmat_total_work = None
        if bwd_args.requires_wgrad:
            if bwd_args.fp8 or bwd_args.debug:
                if isinstance(inputmat, QuantizedTensorStorage):
                    # Input tensor is already quantized
                    pass
                elif bwd_args.debug or bwd_args.custom:
                    # Debug quantizer will be applied immediately before wgrad GEMM
                    pass
                else:
                    # Quantize input tensor
                    quantizer = input_quantizer
                    if quantizer.supports_only_rowwise_all_gather():
                        # All-gather is not supported with FP8 column-wise data
                        quantizer.set_usage(
                            rowwise=True,
                            columnwise=not bwd_args.backward_input_needs_gather,
… 111 more lines
Agent → toolmessage 37
read_file
{
  "file_path": "/workspace/TransformerEngine/transformer_engine/pytorch/module/linear.py",
  "offset": 1000,
  "limit": 150
}
Tool result · read_filemessage 38
/workspace/TransformerEngine/transformer_engine/pytorch/module/linear.pylines 1001-1150
@@ lines 1001-1150 of 2171 | next offset 1150 @@
                grad_output,
                layout="NN",
                grad=True,
                quantization_params=grad_input_quantizer,
                out=gemm_out,
                out_dtype=bwd_args.activation_dtype,
                use_split_accumulator=use_split_accumulator,
                ub=ub_obj_dgrad,
                ub_type=ub_type_dgrad,
                extra_output=reduce_scatter_out,
                bulk_overlap=bwd_args.ub_bulk_dgrad,
            )
            nvtx_range_pop(f"{nvtx_label}.dgrad_gemm")

            # FSDP2 only handles deallocation all-gathered weights that it allocates.
            # Columnwise data is derived from rowwise data after allgather for fp8
            # and 2d block-scaled weights in TE managed memory. So we need to clear
            # it here.
            # (Issues #2681, #2717)
            if bwd_args.is_fsdp2 and isinstance(weight_fp8, QuantizedTensorStorage):
                clear_columnwise_cache(weight_fp8)

            # Prepare grad input tensor
            # Note: Perform tensor-parallel communication
            if bwd_args.ub_overlap_rs_dgrad:
                # cuBLASMp writes the reduce-scattered dgrad directly into the
                # GEMM output tensor; Userbuffers uses the extra-output buffer.
                dgrad = (
                    gemm_out
                    if ub_obj_dgrad is not None and ub_obj_dgrad.with_cublasmp()
                    else reduce_scatter_out
                )
            elif bwd_args.ub_bulk_wgrad:
                dgrad = ub_obj_wgrad.get_buffer(local_chunk=True)
            elif bwd_args.parallel_mode == "column" and bwd_args.tp_size > 1:
                nvtx_range_push(f"{nvtx_label}.column_parallel_comm_dgrad")
                dgrad = gemm_out
                if bwd_args.sequence_parallel:
                    dgrad, dgrad_work = reduce_scatter_along_first_dim(
… 111 more lines
Agent → toolmessage 39
read_file
{
  "file_path": "/workspace/TransformerEngine/transformer_engine/pytorch/module/linear.py",
  "offset": 1150,
  "limit": 150
}
Tool result · read_filemessage 40
/workspace/TransformerEngine/transformer_engine/pytorch/module/linear.pylines 1151-1300
@@ lines 1151-1300 of 2171 | next offset 1300 @@

            # Output buffer for overlapping FP8 grad input
            # reduce-scatter with wgrad GEMM
            reduce_scatter_out = None
            if bwd_args.ub_bulk_wgrad and ub_obj_wgrad.is_fp8_ubuf():
                reduce_scatter_out = torch.empty(
                    dgrad_shape,
                    dtype=bwd_args.activation_dtype,
                    device=grad_output_arg.device,
                )

            # Arguments to include in wgrad GEMM closure
            wgrad_gemm_kwargs = {
                "out_dtype": (
                    main_grad.dtype
                    if bwd_args.fuse_wgrad_accumulation
                    else bwd_args.activation_dtype
                ),
                "quantization_params": grad_weight_quantizer,
                "accumulate": (
                    accumulate_wgrad_into_param_main_grad
                    if not origin_weight_overwrites_main_grad
                    else False
                ),
                "layout": "NT",
                "out": main_grad if bwd_args.fuse_wgrad_accumulation else None,
                "bias": (bias if (grad_bias is None and not bwd_args.fp8) else None),
                "use_split_accumulator": use_split_accumulator,
                "grad": True,
                "ub": ub_obj_wgrad,
                "ub_type": ub_type_wgrad,
                "extra_output": reduce_scatter_out,
                "bulk_overlap": bwd_args.ub_bulk_wgrad,
            }

            def wgrad_gemm(
                x: torch.Tensor,
                dy: torch.Tensor,
            ) -> Tuple[torch.Tensor, torch.Tensor, torch.Tensor]:
… 111 more lines
Agent → toolmessage 41
read_file
{
  "file_path": "/workspace/TransformerEngine/transformer_engine/pytorch/module/linear.py",
  "offset": 1300,
  "limit": 100
}
Tool result · read_filemessage 42
/workspace/TransformerEngine/transformer_engine/pytorch/module/linear.pylines 1301-1400
@@ lines 1301-1400 of 2171 | next offset 1400 @@
        inp: torch.Tensor,
        bias: Optional[torch.Tensor],
        fwd_args: LinearFwdArgs,
    ) -> Tuple[torch.Tensor, Optional[torch.Tensor]]:
        """Forward pass: compute linear output and set up autograd context.

        ``weight``, ``inp`` and ``bias`` are positional Tensor arguments so
        autograd tracks them; they are immediately re-attached to ``fwd_args``
        so every downstream helper can be invoked with a single argument.

        ``weight_workspace`` is intentionally NOT a positional input: it is a
        non-differentiable cached tensor passed in via
        ``fwd_args.weight_workspace`` and the freshly produced workspace is
        returned as a separate output so the module can refresh its cache.
        """
        fwd_args.weight = weight
        fwd_args.inp = inp
        fwd_args.bias = bias
        (
            out,
            new_weight_workspace,
            tensors_to_save_from_forward,
            _,
            ctx_attrs,
        ) = _linear_forward_impl(fwd_args)
        if ctx is not None:
            bwd_args = LinearBwdArgs()
            tensors_to_save_from_setup = _linear_setup_ctx(
                bwd_args,
                fwd_args,
                out,
                ctx_attrs,
                tensors_to_save_from_forward,
            )
            tensors_to_save, tensor_objects = prepare_for_saving(*tensors_to_save_from_setup)
            ctx.save_for_backward(*tensors_to_save)
            ctx.tensor_objects = tensor_objects
            ctx.backward_objects = bwd_args
            if fwd_args.fp8 and (
… 61 more lines
Agent → toolmessage 43
read_file
{
  "file_path": "/workspace/TransformerEngine/transformer_engine/pytorch/module/linear.py",
  "offset": 1400,
  "limit": 100
}
Tool result · read_filemessage 44
/workspace/TransformerEngine/transformer_engine/pytorch/module/linear.pylines 1401-1500
@@ lines 1401-1500 of 2171 | next offset 1500 @@
                      values as split sizes along dim 0. The resulting parameters will have
                      names that end in ``_weight`` or ``_bias``, so trailing underscores are
                      stripped from any provided names.
    device : Union[torch.device, str], default = "cuda"
          The device on which the parameters of the model will be allocated. It is the user's
          responsibility to ensure all parameters are moved to the GPU before running the
          forward pass.
    name : str, default = None
        name of the module, currently used for debugging purposes.

    Parallelism parameters
    ----------------------
    sequence_parallel : bool, default = False
                       if set to ``True``, uses sequence parallelism.
    tp_group : ProcessGroup, default = None
              tensor parallel process group.
    tp_size : int, default = 1
             used as TP (tensor parallel) world size when TP groups are not formed during
             initialization. In this case, users must call the
             ``set_tensor_parallel_group(tp_group)`` method on the initialized module before the
             forward pass to supply the tensor parallel group needed for tensor and sequence
             parallel collectives.
    parallel_mode : {None, 'column', 'row'}, default = None
                   used to decide whether this Linear layer is Column Parallel Linear or Row
                   Parallel Linear as described `here <https://arxiv.org/pdf/1909.08053.pdf>`_.
                   When set to ``None``, no communication is performed.

    Optimization parameters
    -----------------------
    fuse_wgrad_accumulation : bool, default = 'False'
                             if set to ``True``, enables fusing of creation and accumulation of
                             the weight gradient. When enabled, it is assumed that the weights
                             have an additional ``main_grad`` attribute (used instead of the
                             regular ``grad``) which is a pre-allocated buffer of the correct
                             size to accumulate gradients in. This argument along with
                             weight tensor having attribute 'overwrite_main_grad' set to True
                             will overwrite ``main_grad`` instead of accumulating.
    return_bias : bool, default = False
                 when set to ``True``, this module will not apply the additive bias itself, but
… 61 more lines
Agent → toolmessage 45
read_file
{
  "file_path": "/workspace/TransformerEngine/transformer_engine/pytorch/module/linear.py",
  "offset": 1500,
  "limit": 100
}
Tool result · read_filemessage 46
/workspace/TransformerEngine/transformer_engine/pytorch/module/linear.pylines 1501-1600
@@ lines 1501-1600 of 2171 | next offset 1600 @@
        self.rng_tracker_name = rng_tracker_name
        self.symmetric_ar_type = symmetric_ar_type
        self.save_original_input = save_original_input

        self.wgrad_store = WeightGradStore(delay_wgrad_compute, ub_bulk_wgrad)

        if device == "meta":
            assert parameters_split is None, "Cannot split module parameters on 'meta' device."
        if tp_group is None:
            self.tp_size = tp_size
            if tp_size == 1:
                self.set_tensor_parallel_group(tp_group)
        else:
            self.tp_size = get_distributed_world_size(tp_group)
            self.set_tensor_parallel_group(tp_group)
        self.set_nccl_overlap_warning_if_tp()

        self.parallel_mode = parallel_mode
        assert (
            self.parallel_mode in GemmParallelModes
        ), f"parallel_mode {parallel_mode} not supported"

        if self.parallel_mode == "column":
            self.out_features = divide(self.out_features, self.tp_size)
        elif self.parallel_mode == "row":
            self.in_features = divide(self.in_features, self.tp_size)

        self.sequence_parallel = (self.tp_size > 1) and sequence_parallel

        # Column parallel TP overlap options
        self.ub_overlap_ag_fprop = (
            self.parallel_mode == "column" and self.sequence_parallel and ub_overlap_ag
        )
        self.ub_overlap_rs_dgrad = (
            self.parallel_mode == "column" and self.sequence_parallel and ub_overlap_rs_dgrad
        )
        # Bulk overlaps require the Userbuffers backend; the cuBLASMp backend falls back on
        # DGRAD+RS overlap with no bulk overlap for WGRAD
        self.ub_bulk_dgrad = (
… 61 more lines
Agent → toolmessage 47
grep
{
  "pattern": "save_original_input",
  "path": "/workspace/TransformerEngine/transformer_engine/pytorch/module/linear.py",
  "glob": null,
  "output_mode": "content",
  "max_count": null
}
Tool result · grepmessage 48
/workspace/repo/transformer_engine/pytorch/module/linear.py:
  118:     save_original_input: bool
  282:     save_original_input = args.save_original_input
  287:         save_original_input = True
  289:         save_original_input = False
  327:         if save_original_input:
  330:             ), "DelayedScaling recipe is not supported with save_original_input"
  347:                 if save_original_input:
  386:                         and not save_original_input
  543:         if save_original_input:
  1456:     save_original_input : bool, default = False
  1488:         save_original_input: bool = False,
  1503:         self.save_original_input = save_original_input
  1920:                 save_original_input=self.save_original_input,
Agent → toolmessage 49
read_file
{
  "file_path": "/workspace/TransformerEngine/transformer_engine/pytorch/module/linear.py",
  "offset": 1900,
  "limit": 60
}
Tool result · read_filemessage 50
/workspace/TransformerEngine/transformer_engine/pytorch/module/linear.pylines 1901-1960
@@ lines 1901-1960 of 2171 | next offset 1960 @@
                weight_workspace=weight_workspace,
                # requires_grad flags
                input_requires_grad=inp.requires_grad,
                weight_requires_grad=weight_tensor.requires_grad,
                bias_requires_grad=(
                    linear_bias_tensor.requires_grad if linear_bias_tensor is not None else False
                ),
                # quantizers
                input_quantizer=input_quantizer,
                weight_quantizer=weight_quantizer,
                output_quantizer=output_quantizer,
                grad_input_quantizer=grad_input_quantizer,
                grad_weight_quantizer=grad_weight_quantizer,
                grad_output_quantizer=grad_output_quantizer,
                # numerical / dtype config
                activation_dtype=self.activation_dtype,
                fp8=self.fp8,
                fp8_calibration=self.fp8_calibration,
                fp8_output=fp8_output,
                save_original_input=self.save_original_input,
                backward_override=backward_override,
                custom=custom,
                debug=debug,
                # weight-workspace caching
                is_first_microbatch=is_first_microbatch,
                cache_weight=cache_name is not None,
                skip_fp8_weight_update=skip_fp8_weight_update,
                # tensor / sequence parallelism
                parallel_mode=self.parallel_mode,
                tp_group=self.tp_group,
                tp_size=self.tp_size,
                tensor_parallel=self.tp_size > 1,
                sequence_parallel=self.sequence_parallel,
                symmetric_ar_type=self.symmetric_ar_type,
                backward_input_needs_gather=backward_input_needs_gather,
                # userbuffers
                ub_name=self.ub_name,
                ub_overlap_ag_fprop=ub_overlap_ag_fprop,
                ub_overlap_rs_fprop=ub_overlap_rs_fprop,
… 21 more lines
Agent → toolmessage 51
add_finding
{
  "severity": "low",
  "confidence": "high",
  "category": "correctness",
  "file": "transformer_engine/pytorch/module/linear.py",
  "title": "Override silently changes input-saving behavior",
  "description": "`dequantized` forces `save_original_input=False` even when the module was configured to save the original input. Users selecting `dequantized` therefore receive a different memory-saving behavior without an explicit warning or documented interaction.",
  "start_line": 289,
  "end_line": 289,
  "suggestion": null,
  "side": "RIGHT"
}
Tool result · add_findingmessage 52
{"success": true, "finding_id": "f_edab778e3d", "duplicate": false}
Agent → toolmessage 53
add_finding
{
  "severity": "low",
  "confidence": "high",
  "category": "correctness",
  "file": "transformer_engine/pytorch/module/grouped_linear.py",
  "title": "Override silently changes input-saving behavior",
  "description": "`dequantized` forces `save_original_input=False` even when the module was configured to save the original input. Users selecting `dequantized` therefore receive a different memory-saving behavior without an explicit warning or documented interaction.",
  "start_line": 1013,
  "end_line": 1013,
  "suggestion": null,
  "side": "RIGHT"
}
Tool result · add_findingmessage 54
{"success": false, "in_diff": false, "error": "Out-of-diff findings are disabled. This finding's lines are not part of this PR's diff. Only file findings anchored to a line the PR changed. Do not re-anchor or retry."}
Agent → toolmessage 55
read_file
{
  "file_path": "/workspace/TransformerEngine/tests/pytorch/test_backward_override.py",
  "offset": 200,
  "limit": 200
}
Tool result · read_filemessage 56
/workspace/TransformerEngine/tests/pytorch/test_backward_override.pylines 201-400
@@ lines 201-400 of 2056 | next offset 400 @@


def _maybe_skip_unsupported_recipe_shape(
    recipe_name: str,
    input_shape: tuple[int, ...],
    module_type: str,
) -> None:
    flat_first_dim = math.prod(input_shape[:-1])
    last_dim = input_shape[-1]

    if module_type in ("linear", "layernorm_linear"):
        if recipe_name == "mxfp8" and (flat_first_dim % 32 != 0 or last_dim % 32 != 0):
            pytest.skip(
                "Linear/LayerNormLinear + MXFP8 requires prod(shape[:-1]) and shape[-1] divisible"
                " by 32."
            )
            return
        if recipe_name in ("nvfp4", "nvfp4_row_scaled", "nvfp4_4over6") and (
            flat_first_dim % 16 != 0 or last_dim % 16 != 0
        ):
            pytest.skip(
                "Linear/LayerNormLinear + NVFP4 requires prod(shape[:-1]) and shape[-1] divisible"
                " by 16."
            )
            return
        if flat_first_dim % 8 != 0 or last_dim % 16 != 0:
            pytest.skip(
                "Linear/LayerNormLinear FP8 execution requires prod(shape[:-1]) divisible by 8 "
                "and shape[-1] divisible by 16."
            )
    elif module_type == "ops_linear":
        if (
            recipe_name == "fp8_current_scaling"
            and not is_non_tn_fp8_gemm_supported()
            and flat_first_dim % 16 != 0
        ):
            pytest.skip(
                "te_ops.Linear + Float8CurrentScaling on pre-Blackwell requires "
                "prod(shape[:-1]) divisible by 16 for FP8 NT wgrad GEMM."
… 161 more lines
Agent → toolmessage 57
read_file
{
  "file_path": "/workspace/TransformerEngine/tests/pytorch/test_backward_override.py",
  "offset": 400,
  "limit": 200
}
Tool result · read_filemessage 58
/workspace/TransformerEngine/tests/pytorch/test_backward_override.pylines 401-600
@@ lines 401-600 of 2056 | next offset 600 @@
) -> list[dict[str, object]]:
    """Capture saved-operand layout invariants before backward runs."""
    return [
        _snapshot_saved_quantized_operand_layout(saved_operand, name=name)
        for name, saved_operand in guard_operands
    ]


def _snapshot_backward_ctx_state(
    output: torch.Tensor,
) -> tuple[str, bool, object, bool]:
    if output.grad_fn is None:
        raise RuntimeError("Output tensor has no grad_fn; cannot inspect backward context state.")
    # ``Linear`` packs backward state into ``grad_fn.backward_objects``
    # (``LinearBwdArgs``); other linear-like modules still set the attributes
    # directly on the autograd ctx.
    state_holder = getattr(output.grad_fn, "backward_objects", output.grad_fn)
    required_attrs = (
        "backward_override",
        "fp8",
        "grad_output_quantizer",
        "reduce_and_update_bwd_fp8_tensors",
    )
    missing_attrs = [attr for attr in required_attrs if not hasattr(state_holder, attr)]
    if missing_attrs:
        raise RuntimeError(
            f"Backward context does not expose required attributes: {', '.join(missing_attrs)}."
        )
    return (
        getattr(state_holder, "backward_override"),
        bool(getattr(state_holder, "fp8")),
        getattr(state_holder, "grad_output_quantizer"),
        bool(getattr(state_holder, "reduce_and_update_bwd_fp8_tensors")),
    )


def _assert_saved_quantized_operand_uses_rowwise_only(
    saved_operand: Optional[torch.Tensor],
    *,
… 161 more lines
Agent → toolmessage 59
read_file
{
  "file_path": "/workspace/TransformerEngine/tests/pytorch/test_backward_override.py",
  "offset": 600,
  "limit": 200
}
Tool result · read_filemessage 60
/workspace/TransformerEngine/tests/pytorch/test_backward_override.pylines 601-800
@@ lines 601-800 of 2056 | next offset 800 @@
    bias = getattr(module, "bias", None)
    bgrad = None if bias is None or bias.grad is None else bias.grad.detach().clone()
    return (
        y.detach().clone(),
        x_run.grad.detach().clone(),
        module.weight.grad.detach().clone(),
        bgrad,
    )


def _run_single_step_with_saved_operands(
    module: torch.nn.Module,
    x: torch.Tensor,
    fp8_recipe: recipe.Recipe,
) -> tuple[
    torch.Tensor,
    torch.Tensor,
    list[Optional[torch.Tensor]],
]:
    module.zero_grad(set_to_none=True)
    x_run = x.detach().clone().requires_grad_(True)
    with te.autocast(enabled=True, recipe=fp8_recipe):
        y = module(x_run)
        if isinstance(y, tuple):
            y = y[0]
        saved_operands = restore_from_saved(y.grad_fn.tensor_objects, list(y.grad_fn.saved_tensors))
    return y, x_run, saved_operands


def _run_grouped_linear_single_step(
    module: te.GroupedLinear,
    x: torch.Tensor,
    m_splits: list[int],
    dy: torch.Tensor,
    fp8_recipe: Optional[recipe.Recipe],
) -> tuple[torch.Tensor, torch.Tensor, list[torch.Tensor], list[Optional[torch.Tensor]]]:
    module.zero_grad(set_to_none=True)
    x_run = x.detach().clone().requires_grad_(True)
    autocast_ctx = (
… 161 more lines
Agent → toolmessage 61
read_file
{
  "file_path": "/workspace/TransformerEngine/tests/pytorch/test_backward_override.py",
  "offset": 800,
  "limit": 200
}
Tool result · read_filemessage 62
/workspace/TransformerEngine/tests/pytorch/test_backward_override.pylines 801-1000
@@ lines 801-1000 of 2056 | next offset 1000 @@
    list[torch.Tensor],
    list[Optional[torch.Tensor]],
    tuple[str, bool, bool],
]:
    module.zero_grad(set_to_none=True)
    x_run = x.detach().clone().requires_grad_(True)
    autocast_ctx = (
        te.autocast(enabled=True, recipe=fp8_recipe) if fp8_recipe is not None else nullcontext()
    )
    with autocast_ctx:
        y = module(x_run, m_splits)
        if y.grad_fn is None:
            raise RuntimeError(
                "Output tensor has no grad_fn; cannot inspect grouped backward state."
            )
        required_attrs = (
            "backward_override",
            "fp8",
            "reduce_and_update_bwd_fp8_tensors",
        )
        missing_attrs = [attr for attr in required_attrs if not hasattr(y.grad_fn, attr)]
        if missing_attrs:
            raise RuntimeError(
                "Grouped grad_fn does not expose required backward context attributes: "
                f"{', '.join(missing_attrs)}."
            )
        ctx_state = (
            getattr(y.grad_fn, "backward_override"),
            bool(getattr(y.grad_fn, "fp8")),
            bool(getattr(y.grad_fn, "reduce_and_update_bwd_fp8_tensors")),
        )
    y.backward(dy)
    assert x_run.grad is not None

    dw = [getattr(module, f"weight{i}").grad.detach().clone() for i in range(module.num_gemms)]
    db: list[Optional[torch.Tensor]] = []
    for i in range(module.num_gemms):
        if module.use_bias:
            db.append(getattr(module, f"bias{i}").grad.detach().clone())
… 160 more lines
Agent → toolmessage 63
read_file
{
  "file_path": "/workspace/TransformerEngine/tests/pytorch/test_backward_override.py",
  "offset": 1000,
  "limit": 200
}
Tool result · read_filemessage 64
/workspace/TransformerEngine/tests/pytorch/test_backward_override.pylines 1001-1200
@@ lines 1001-1200 of 2056 | next offset 1200 @@
@pytest.mark.parametrize("recipe_name", _quantized_numerics_recipe_list)
def test_linear_backward_override_high_precision_forces_save_original_input(
    recipe_name: str,
) -> None:
    reset_rng_states()
    dtype = torch.bfloat16
    input_shape = (32, 128)
    _maybe_skip_recipe_dtype(recipe_name, dtype, "linear")
    _maybe_skip_unsupported_recipe_module_combo(recipe_name, "linear")
    _maybe_skip_unsupported_recipe_shape(recipe_name, input_shape, "linear")

    mode_recipe = make_recipe(recipe_name, backward_override="high_precision")
    skip_unsupported_backward_override("linear", mode_recipe, "high_precision")

    module = te.Linear(
        input_shape[-1],
        128,
        bias=False,
        params_dtype=dtype,
        device="cuda",
        save_original_input=False,
    )
    x = torch.randn(*input_shape, dtype=dtype, device="cuda")

    _, _, saved_operands = _run_single_step_with_saved_operands(module, x, mode_recipe)

    assert isinstance(saved_operands[0], torch.Tensor)


@pytest.mark.parametrize("recipe_name", _quantized_numerics_recipe_list)
@pytest.mark.parametrize("module_type", ("linear", "layernorm_linear", "ops_linear"))
@pytest.mark.parametrize("input_shape,out_features", _shape_test_cases)
@pytest.mark.parametrize("use_bias", (False, True), ids=("no_bias", "bias"))
@pytest.mark.parametrize("dtype", _core_dtypes, ids=str)
def test_linear_like_backward_override_matches_reference(
    recipe_name: str,
    module_type: str,
    input_shape: tuple[int, ...],
    out_features: int,
… 161 more lines
Agent → toolmessage 65
read_file
{
  "file_path": "/workspace/TransformerEngine/tests/pytorch/test_backward_override.py",
  "offset": 1200,
  "limit": 200
}
Tool result · read_filemessage 66
/workspace/TransformerEngine/tests/pytorch/test_backward_override.pylines 1201-1400
@@ lines 1201-1400 of 2056 | next offset 1400 @@
        assert dx_ref is not None and dw_ref is not None and db_ref is not None

    assert_close(y_bwd_mode, y_quantized_ref, rtol=0, atol=0, check_dtype=True)
    assert_close(dx_bwd_mode, dx_ref, rtol=0, atol=0, check_dtype=True)
    assert_close(dw_bwd_mode, dw_ref, rtol=0, atol=0, check_dtype=True)
    if use_bias:
        assert db_bwd_mode is not None
        assert db_ref is not None
        assert_close(db_bwd_mode, db_ref, rtol=0, atol=0, check_dtype=True)


@pytest.mark.parametrize("recipe_name", _quantized_numerics_recipe_list)
@pytest.mark.parametrize("in_features,out_features", _linear_feature_cases)
@pytest.mark.parametrize("use_bias", (False, True), ids=("no_bias", "bias"))
@pytest.mark.parametrize("m_splits", _grouped_m_split_cases)
@pytest.mark.parametrize("dtype", _core_dtypes, ids=str)
def test_grouped_linear_backward_override_matches_reference(
    recipe_name: str,
    in_features: int,
    out_features: int,
    use_bias: bool,
    m_splits: list[int],
    dtype: torch.dtype,
    backward_override: str,
) -> None:

    reset_rng_states()
    _maybe_skip_recipe_dtype(recipe_name, dtype, "grouped_linear")
    _maybe_skip_unsupported_recipe_module_combo(recipe_name, "grouped_linear")
    _maybe_skip_unsupported_grouped_splits(recipe_name, m_splits)
    num_gemms = len(m_splits)
    num_tokens = sum(m_splits)

    quantized_ref_recipe = _make_quantized_forward_reference_recipe(recipe_name)
    mode_recipe = make_recipe(recipe_name, backward_override=backward_override)
    skip_unsupported_backward_override("grouped_linear", mode_recipe, backward_override)

    module_quantized_ref = te.GroupedLinear(
        num_gemms,
… 161 more lines
Agent → toolmessage 67
grep
{
  "pattern": "save_original_input",
  "path": "/workspace/TransformerEngine",
  "glob": "*.py",
  "output_mode": "content",
  "max_count": 200
}
Tool result · grepmessage 68
/workspace/repo/docs/features/low_precision_training/performance_considerations/save_original_input_pytorch.py:
  31:         1024, 1024, params_dtype=torch.bfloat16, save_original_input=use_save_original
  49:     print(f"save_original_input={use_save_original}: {peak:.1f} MB")
/workspace/repo/tests/pytorch/distributed/run_numerics.py:
  742:         {"save_original_input": True},
  746:         if kwargs.get("save_original_input", False) and QUANTIZATION == "fp8":
/workspace/repo/tests/pytorch/distributed/run_numerics_exact.py:
  551:         if kwargs.get("save_original_input", False) and QUANTIZATION == "fp8":
/workspace/repo/tests/pytorch/test_backward_override.py:
  863: def test_linear_backward_override_dequantized_ignores_save_original_input(
  884:         save_original_input=False,
  892:         save_original_input=True,
  926: def test_grouped_linear_backward_override_dequantized_ignores_save_original_input(
  951:         save_original_input=False,
  960:         save_original_input=True,
  1002: def test_linear_backward_override_high_precision_forces_save_original_input(
  1021:         save_original_input=False,
/workspace/repo/tests/pytorch/test_grouped_linear.py:
  321:             save_original_input=False,
  424: def test_grouped_linear_accuracy_save_original_input(
  440:         pytest.skip("DelayedScaling recipe is not supported with save_original_input")
  468:             save_original_input=True,
  686:             save_original_input=False,
  718: def test_padding_grouped_linear_accuracy_save_original_input(
  731:         pytest.skip("DelayedScaling recipe is not supported with save_original_input")
  766:             save_original_input=True,
/workspace/repo/tests/pytorch/test_numerics.py:
  1315: def test_linear_accuracy_save_original_input(dtype, model, recipe):
  1322:         pytest.skip("DelayedScaling recipe is not supported with save_original_input")
  1343:             save_original_input=False,
  1353:             save_original_input=True,
/workspace/repo/transformer_engine/pytorch/module/grouped_linear.py:
  94:         save_original_input: bool,
  125:             or save_original_input
  387:             ctx.save_original_input = False
  425:             save_original_input,
  433:             save_original_input = True
  435:             save_original_input = False
  444:         if save_original_input and isinstance(input_quantizers[0], Float8Quantizer):
  450:                     "save_original_input is incompatible with delayed-scaling quantizers "
… 26 more lines
Agent → toolmessage 69
read_file
{
  "file_path": "/workspace/TransformerEngine/transformer_engine/pytorch/module/layernorm_linear.py",
  "offset": 1,
  "limit": 220
}
Tool result · read_filemessage 70
/workspace/TransformerEngine/transformer_engine/pytorch/module/layernorm_linear.pylines 2-221
@@ lines 2-221 of 1994 | next offset 221 @@
#
# See LICENSE for license information.

"""LayerNormLinear API"""
import os
import warnings
import weakref
from typing import Callable, Dict, Optional, Tuple, Union, List
from functools import reduce
from operator import mul as multiply_op

import torch
from torch.nn import init

import transformer_engine_torch as tex

from transformer_engine.common.recipe import Recipe
from transformer_engine.pytorch.torch_version import torch_version
from transformer_engine.pytorch.tensor.utils import clear_columnwise_cache, is_custom
from .base import (
    fill_userbuffers_buffer_for_all_gather,
    get_ub,
    is_ub_initialized,
    using_cublasmp_backend,
    quantize_weight,
    TransformerEngineBaseModule,
    get_dummy_wgrad,
    _2X_ACC_FPROP,
    _2X_ACC_DGRAD,
    _2X_ACC_WGRAD,
)
from ..quantization import FP8GlobalStateManager, QuantizerRole
from ..utils import (
    assert_dim_for_fp8_exec,
    cast_if_needed,
    clear_tensor_data,
    divide,
    get_default_init_method,
    init_method_constant,
… 181 more lines
Agent → toolmessage 71
grep
{
  "pattern": "backward_override == \"dequantized\"",
  "path": "/workspace/TransformerEngine",
  "glob": "*.py",
  "output_mode": "content",
  "max_count": 100
}
Tool result · grepmessage 72
/workspace/repo/transformer_engine/pytorch/module/grouped_linear.py:
  434:         elif backward_override == "dequantized":
  1011:                 if ctx.backward_override == "dequantized":
  1093:                 elif ctx.backward_override == "dequantized":
/workspace/repo/transformer_engine/pytorch/module/layernorm_linear.py:
  720:             if ctx.backward_override == "dequantized":
  810:             if ctx.backward_override == "dequantized":
/workspace/repo/transformer_engine/pytorch/module/linear.py:
  288:     elif backward_override == "dequantized":
  990:             if bwd_args.backward_override == "dequantized":
Agent → toolmessage 73
read_file
{
  "file_path": "/workspace/TransformerEngine/transformer_engine/pytorch/module/layernorm_linear.py",
  "offset": 620,
  "limit": 240
}
Tool result · read_filemessage 74
/workspace/TransformerEngine/transformer_engine/pytorch/module/layernorm_linear.pylines 621-860
@@ lines 621-860 of 1994 | next offset 860 @@
            )
            main_grad = None
            if ctx.fuse_wgrad_accumulation and ctx.requires_wgrad:
                origin_weight_ref = ctx.origin_weight_ref
                ctx.origin_weight_ref = None
                origin_weight = origin_weight_ref() if origin_weight_ref is not None else None
                assert (
                    origin_weight is not None
                ), "weight was removed while fuse_wgrad_accumulation=True"
                # Since main_grad can be modified inplace, it should not be a part of saved_tensors
                main_grad = ctx.main_grad_func() if weight is not None else None
                if main_grad is not None:
                    origin_weight.main_grad = main_grad

            # Gather intermediate/activation tensors if needed
            # NOTE: weight_fp8 = weight when ctx.fp8 == False and torch.disttributed.FSDP already
            #       shards/unshards the base weights so we don't do it ourselves
            nvtx_range_push(f"{nvtx_label}.fsdp_gather")
            _fsdp_gather_tensors(
                ctx.fsdp_group,
                ctx.fsdp_shapes,
                mu,
                rsigma,
                weight if ctx.fp8 and not ctx.is_weight_param_quantized else None,
                ln_out,
            )
            nvtx_range_pop(f"{nvtx_label}.fsdp_gather")

            # Configure Userbuffers communication (comm+GEMM overlap)
            ctx.ub_obj_gradout = None
            ub_obj_dgrad = None
            ub_obj_wgrad = None
            ub_type_dgrad = None
            ub_type_wgrad = None
            dgrad_shape = [reduce(multiply_op, ctx.inp_shape[:-1]), ctx.inp_shape[-1]]
            if ctx.ub_overlap_ag:
                # Overlap grad_output all-gather with dgrad compute
                ctx.ub_obj_gradout = get_ub(ctx.ub_name + "_dgrad", ctx.fp8)
                ub_obj_dgrad = ctx.ub_obj_gradout
… 201 more lines
Agent → toolmessage 75
read_file
{
  "file_path": "/workspace/TransformerEngine/transformer_engine/pytorch/module/linear.py",
  "offset": 260,
  "limit": 150
}
Tool result · read_filemessage 76
/workspace/TransformerEngine/transformer_engine/pytorch/module/linear.pylines 261-410
@@ lines 261-410 of 2171 | next offset 410 @@
    """

    weight = args.weight
    inp = args.inp
    bias = args.bias
    input_quantizer = args.input_quantizer
    weight_quantizer = args.weight_quantizer
    output_quantizer = args.output_quantizer
    is_first_microbatch = args.is_first_microbatch
    fp8 = args.fp8
    cpu_offloading = args.cpu_offloading
    tp_group = args.tp_group
    sequence_parallel = args.sequence_parallel
    activation_dtype = args.activation_dtype
    parallel_mode = args.parallel_mode
    is_grad_enabled = args.is_grad_enabled
    ub_overlap_rs_fprop = args.ub_overlap_rs_fprop
    ub_overlap_ag_fprop = args.ub_overlap_ag_fprop
    ub_name = args.ub_name
    fsdp_group = args.fsdp_group
    symmetric_ar_type = args.symmetric_ar_type
    save_original_input = args.save_original_input
    debug = args.debug
    backward_override = args.backward_override
    is_fsdp2 = args.is_fsdp2
    if backward_override == "high_precision":
        save_original_input = True
    elif backward_override == "dequantized":
        save_original_input = False

    # NVTX label for profiling
    nvtx_label = "transformer_engine._Linear.forward"
    if ub_name is not None:
        nvtx_label = f"{nvtx_label}.{ub_name}"

    # Make sure input dimensions are compatible
    out_features, in_features = weight.shape
    assert inp.shape[-1] == in_features, "GEMM not possible"
… 111 more lines
Agent → toolmessage 77
read_file
{
  "file_path": "/workspace/TransformerEngine/transformer_engine/pytorch/module/linear.py",
  "offset": 500,
  "limit": 120
}
Tool result · read_filemessage 78
/workspace/TransformerEngine/transformer_engine/pytorch/module/linear.pylines 501-620
@@ lines 501-620 of 2171 | next offset 620 @@
    if with_input_all_gather_nccl:
        clear_tensor_data(inputmat_total)
        inputmat_total = None

    # ------------------------------------------------------
    # Prepare output tensor
    # Note: Perform tensor-parallel communication
    # ------------------------------------------------------
    out = None
    if ub_overlap_rs_fprop:
        # cuBLASMp writes the reduce-scattered output directly into the GEMM
        # output tensor; Userbuffers writes it into the extra-output buffer.
        out = gemm_out if ub_obj is not None and ub_obj.with_cublasmp() else reduce_scatter_out
    elif parallel_mode == "row" and args.tp_size > 1:
        nvtx_range_push(f"{nvtx_label}.row_parallel_comm")
        out = gemm_out
        if sequence_parallel:
            out, _ = reduce_scatter_along_first_dim(out, tp_group)
        elif args.tensor_parallel:
            if symmetric_ar_type is not None:
                out, _ = symmetric_all_reduce(out, tp_group, all_reduce_type=symmetric_ar_type)
            else:
                out, _ = allreduce(out, tp_group)
        nvtx_range_pop(f"{nvtx_label}.row_parallel_comm")
    else:
        out = gemm_out

    # Restore the input's logical rank (e.g., (seq, batch, hidden)) on the output.
    # This is mainly to correct for cuBLASMp comm+GEMM operators that unconditionally
    # return a 2D output buffer that ends up incompatible with downstream consumers
    # (e.g. ``bias_dropout_add`` residual connections inside ``TransformerLayer``).
    out = out.view(-1, *inp.shape[1:-1], out_features)

    # ------------------------------------------------------
    # Output tensor is ready to return...
    # ------------------------------------------------------

    # Prepare backward state
    tensors_to_save_from_forward = None
… 81 more lines
Agent → toolmessage 79
read_file
{
  "file_path": "/workspace/TransformerEngine/transformer_engine/pytorch/module/grouped_linear.py",
  "offset": 370,
  "limit": 160
}
Tool result · read_filemessage 80
/workspace/TransformerEngine/transformer_engine/pytorch/module/grouped_linear.pylines 371-530
@@ lines 371-530 of 1979 | next offset 530 @@
            ctx.fp8_recipe = FP8GlobalStateManager.get_fp8_recipe() if fp8 else None
            ctx.backward_override = None
            ctx.fuse_wgrad_accumulation = fuse_wgrad_accumulation
            ctx.cpu_offloading = False
            ctx.is_first_microbatch = is_first_microbatch
            ctx.use_bias = use_bias
            ctx.inp_shape = inp.shape
            ctx.requires_dgrad = inp.requires_grad
            ctx.reduce_and_update_bwd_fp8_tensors = False
            if ctx.fp8 and requires_grad(inp, weights[0], biases[0]):
                ctx.reduce_and_update_bwd_fp8_tensors = (
                    ctx.reduce_and_update_bwd_fp8_tensors
                    or FP8GlobalStateManager.is_first_fp8_module()
                )
            ctx.wgrad_store = wgrad_store
            ctx.debug = False
            ctx.save_original_input = False
            ctx.input_quantizers = input_quantizers

        return out.view(-1, *inp.shape[1:-1], out.shape[-1]), new_workspaces

    # pylint: disable=keyword-arg-before-vararg
    @staticmethod
    def forward(
        ctx,
        inp: torch.Tensor,
        m_splits: torch.Tensor,
        non_tensor_args: Tuple,
        *weights_and_biases,
    ) -> Tuple[torch.Tensor, list]:
        # pylint: disable=missing-function-docstring

        # Reduce number of arguments to autograd function in order
        # to reduce CPU overhead due to pytorch arg checking.
        (
            use_bias,
            is_first_microbatch,
            fp8,
            fp8_calibration,
… 121 more lines
Agent → toolmessage 81
read_file
{
  "file_path": "/workspace/TransformerEngine/transformer_engine/pytorch/module/grouped_linear.py",
  "offset": 530,
  "limit": 220
}
Tool result · read_filemessage 82
/workspace/TransformerEngine/transformer_engine/pytorch/module/grouped_linear.pylines 531-750
@@ lines 531-750 of 1979 | next offset 750 @@

        # Convert splits to list of ints for compatibility with split functions
        m_splits = m_splits.tolist()

        inp_view = inp.reshape(-1, in_features)
        inputmats: list
        if fp8 and not debug:
            # Disable bulk allocation when CPU offloading is active: offloading skips small
            # tensors (like scales), but bulk allocation shares storage across all tensors,
            # so if scales can't be offloaded, nothing in the group can be offloaded.
            inputmats = tex.split_quantize(
                inp_view,
                m_splits,
                input_quantizers,
                disable_bulk_allocation=cpu_offloading,
            )
        elif debug:
            inputmats = DebugQuantizer.multi_tensor_quantize(
                inp_view, input_quantizers, m_splits, activation_dtype
            )
        else:
            inputmats = torch.split(cast_if_needed(inp_view, activation_dtype), m_splits)

        if cpu_offloading:
            start_offload(*inputmats)

        # Initialize weights
        weights_fp8: list
        new_workspaces = [None] * num_gemms
        if fp8 or debug:
            weights_fp8 = []
            update_ws = is_first_microbatch is None or is_first_microbatch
            for i in range(num_gemms):
                weight_fp8, new_workspaces[i] = quantize_weight(
                    tensor=weights[i],
                    quantizer=weight_quantizers[i],
                    workspace=weight_workspaces[i] if weight_workspaces else None,
                    update_workspace=update_ws,
                    skip_update_flag=skip_fp8_weight_update,
… 181 more lines
Agent → toolmessage 83
read_file
{
  "file_path": "/workspace/TransformerEngine/transformer_engine/pytorch/module/layernorm_linear.py",
  "offset": 220,
  "limit": 280
}
Tool result · read_filemessage 84
/workspace/TransformerEngine/transformer_engine/pytorch/module/layernorm_linear.pylines 221-500
@@ lines 221-500 of 1994 | next offset 500 @@
        custom = is_custom(input_quantizer)
        with_quantized_norm = (
            fp8
            and not debug
            and not return_layernorm_output
            and not return_layernorm_output_gathered
            and backward_override is None
            and not custom  # TODO(negvet): and not FP8GlobalStateManager.get_fp8_recipe().custom()
        )

        # Apply normalization
        nvtx_range_push(f"{nvtx_label}.norm")
        ln_out, mu, rsigma = apply_normalization(
            inputmat,
            None,  # ln_out
            ln_weight,
            ln_bias,
            eps,
            input_quantizer if with_quantized_norm else None,
            inputmat.dtype,
            normalization,
            fwd_ln_sm_margin,
            zero_centered_gamma,
        )
        nvtx_range_pop(f"{nvtx_label}.norm")

        # Store unquantized layer norm output if we need to return it
        ln_out_return = None
        if return_layernorm_output or return_layernorm_output_gathered:
            ln_out_return = ln_out
        ln_out_hp = ln_out if backward_override == "high_precision" else None

        # ------------------------------------------------------
        # Prepare GEMM input tensor
        # Note: Cast to expected dtype and perform tensor-parallel communication
        # ------------------------------------------------------
        nvtx_range_push(f"{nvtx_label}.gemm_input_cast_comm")
        ln_out_total = None
        if with_input_all_gather:
… 241 more lines
Agent → toolmessage 85
read_file
{
  "file_path": "/workspace/TransformerEngine/transformer_engine/pytorch/module/layernorm_linear.py",
  "offset": 1,
  "limit": 220
}
Tool result · read_filemessage 86
/workspace/TransformerEngine/transformer_engine/pytorch/module/layernorm_linear.pylines 2-221
@@ lines 2-221 of 1994 | next offset 221 @@
#
# See LICENSE for license information.

"""LayerNormLinear API"""
import os
import warnings
import weakref
from typing import Callable, Dict, Optional, Tuple, Union, List
from functools import reduce
from operator import mul as multiply_op

import torch
from torch.nn import init

import transformer_engine_torch as tex

from transformer_engine.common.recipe import Recipe
from transformer_engine.pytorch.torch_version import torch_version
from transformer_engine.pytorch.tensor.utils import clear_columnwise_cache, is_custom
from .base import (
    fill_userbuffers_buffer_for_all_gather,
    get_ub,
    is_ub_initialized,
    using_cublasmp_backend,
    quantize_weight,
    TransformerEngineBaseModule,
    get_dummy_wgrad,
    _2X_ACC_FPROP,
    _2X_ACC_DGRAD,
    _2X_ACC_WGRAD,
)
from ..quantization import FP8GlobalStateManager, QuantizerRole
from ..utils import (
    assert_dim_for_fp8_exec,
    cast_if_needed,
    clear_tensor_data,
    divide,
    get_default_init_method,
    init_method_constant,
… 181 more lines
Agent → toolmessage 87
read_file
{
  "file_path": "/workspace/TransformerEngine/transformer_engine/pytorch/module/layernorm_linear.py",
  "offset": 500,
  "limit": 300
}
Tool result · read_filemessage 88
/workspace/TransformerEngine/transformer_engine/pytorch/module/layernorm_linear.pylines 501-800
@@ lines 501-800 of 1994 | next offset 800 @@
            )
            ctx.save_for_backward(*tensors_to_save)
            ctx.tensor_objects = tensor_objects
            ctx.requires_dgrad = inp_requires_grad
            ctx.requires_wgrad = weight.requires_grad
            ctx.is_weight_param_quantized = is_weight_param_quantized
            ctx.is_fsdp2 = is_fsdp2
            if fuse_wgrad_accumulation and weight.requires_grad:
                # Keep weakref to weight to preserve attributes like main_grad
                # when we need to modify the weight python object
                ctx.origin_weight_ref = weakref.ref(weight)
                # Save overwrite_main_grad flag now while we have access to weight object
                ctx.origin_weight_overwrites_main_grad = getattr(
                    weight, "overwrite_main_grad", False
                )
                # This check is needed to ensure that main_grad is not created
                # during the forward pass when using MCore FSDP as it creates
                # the main_grad buffer lazily before backprop
                if hasattr(weight, "__fsdp_param__"):
                    # MCore FSDP creates main_grad lazily before backward
                    ctx.main_grad_func = weight.get_main_grad
                else:
                    ctx.main_grad_func = lambda: weight.main_grad
            ctx.grad_input_quantizer = grad_input_quantizer
            ctx.grad_weight_quantizer = grad_weight_quantizer
            ctx.grad_output_quantizer = grad_output_quantizer
            ctx.input_quantizer = input_quantizer
            ctx.owns_input = inputmat is not inp
            ctx.weight = weight
            ctx.activation_dtype = activation_dtype
            ctx.fp8 = fp8
            ctx.fp8_recipe = FP8GlobalStateManager.get_fp8_recipe() if fp8 else None
            ctx.backward_override = backward_override
            ctx.fuse_wgrad_accumulation = fuse_wgrad_accumulation
            ctx.cpu_offloading = cpu_offloading
            ctx.is_first_microbatch = is_first_microbatch
            ctx.use_bias = bias is not None
            ctx.sequence_parallel = sequence_parallel
            ctx.tensor_parallel = tensor_parallel
… 261 more lines
Agent → toolmessage 89
read_file
{
  "file_path": "/workspace/TransformerEngine/transformer_engine/pytorch/module/layernorm_linear.py",
  "offset": 800,
  "limit": 300
}
Tool result · read_filemessage 90
/workspace/TransformerEngine/transformer_engine/pytorch/module/layernorm_linear.pylines 801-1100
@@ lines 801-1100 of 1994 | next offset 1100 @@
                    dgrad_shape, dtype=ctx.activation_dtype, device=grad_outputs[0].device
                )
            elif ctx.ub_bulk_wgrad:
                gemm_out = ub_obj_wgrad.get_buffer(local_chunk=False)

            # dgrad GEMM
            # Note: dx = dy * w
            nvtx_range_push(f"{nvtx_label}.dgrad_gemm")
            weight_for_dgrad = weight
            if ctx.backward_override == "dequantized":
                if isinstance(weight_for_dgrad, QuantizedTensorStorage):
                    weight_for_dgrad = weight_for_dgrad.dequantize(dtype=ctx.activation_dtype)
                else:
                    weight_for_dgrad = cast_if_needed(weight_for_dgrad, ctx.activation_dtype)
            elif ctx.backward_override == "high_precision":
                weight_for_dgrad = saved_weight
                if isinstance(weight_for_dgrad, QuantizedTensorStorage):
                    weight_for_dgrad = weight_for_dgrad.dequantize(dtype=ctx.activation_dtype)
            gemm_out, *_, reduce_scatter_out = general_gemm(
                weight_for_dgrad,
                grad_output,
                layout="NN",
                grad=True,
                quantization_params=ctx.grad_input_quantizer,
                out=gemm_out,
                out_dtype=ctx.activation_dtype,
                use_split_accumulator=use_split_accumulator,
                ub=ub_obj_dgrad,
                ub_type=ub_type_dgrad,
                extra_output=reduce_scatter_out,
                bulk_overlap=ctx.ub_bulk_dgrad,
            )
            nvtx_range_pop(f"{nvtx_label}.dgrad_gemm")

            # FSDP2 only handles deallocation all-gathered weights that it allocates.
            # Columnwise data is derived from rowwise data after allgather for fp8
            # and 2d block-scaled weights in TE managed memory. So we need to clear
            # it here.
            # (Issues #2681, #2717)
… 261 more lines
Agent → toolmessage 91
read_file
{
  "file_path": "/workspace/TransformerEngine/transformer_engine/pytorch/module/layernorm_linear.py",
  "offset": 1100,
  "limit": 300
}
Tool result · read_filemessage 92
/workspace/TransformerEngine/transformer_engine/pytorch/module/layernorm_linear.pylines 1101-1400
@@ lines 1101-1400 of 1994 | next offset 1400 @@
            elif ctx.normalization == "RMSNorm":
                dgrad, dgamma = tex.rmsnorm_bwd(
                    dgrad,
                    inputmat,
                    rsigma,
                    ln_weight,
                    ctx.bwd_ln_sm_margin,
                    ctx.zero_centered_gamma,
                )
                dgrad = dgrad.reshape(inputmat.size())
                dbeta = None
            nvtx_range_pop(f"{nvtx_label}.norm")
            clear_tensor_data(mu)
            clear_tensor_data(rsigma)

        if ctx.requires_wgrad:
            # Handle custom DDP from mcore.
            if ctx.fuse_wgrad_accumulation and hasattr(origin_weight, "grad_added_to_main_grad"):
                origin_weight.grad_added_to_main_grad = True
                if getattr(origin_weight, "zero_out_wgrad", False):
                    wgrad = get_dummy_wgrad(
                        list(main_grad.shape),
                        origin_weight.dtype,
                        zero=True,
                    )
                else:
                    wgrad = get_dummy_wgrad(
                        list(main_grad.shape),
                        origin_weight.dtype,
                    )
            elif ctx.fuse_wgrad_accumulation:
                wgrad = None
        else:
            wgrad = None

        if ctx.reduce_and_update_bwd_fp8_tensors and not is_graph_capturing():
            nvtx_range_push(f"{nvtx_label}.reduce_and_update_fp8_tensors")
            FP8GlobalStateManager.reduce_and_update_fp8_tensors(forward=False)
            nvtx_range_pop(f"{nvtx_label}.reduce_and_update_fp8_tensors")
… 261 more lines
Agent → toolmessage 93
read_file
{
  "file_path": "/workspace/TransformerEngine/transformer_engine/pytorch/module/layernorm_linear.py",
  "offset": 1400,
  "limit": 300
}
Tool result · read_filemessage 94
/workspace/TransformerEngine/transformer_engine/pytorch/module/layernorm_linear.pylines 1401-1700
@@ lines 1401-1700 of 1994 | next offset 1700 @@
        self.register_parameter(
            "layer_norm_weight",
            layer_norm_weight,
            init_fn=init_method_constant(float(not self.zero_centered_gamma)),
        )
        if self.normalization != "RMSNorm":
            layer_norm_bias = torch.nn.Parameter(
                torch.empty(self.in_features, device=device, dtype=params_dtype)
            )
            self.register_parameter(
                "layer_norm_bias", layer_norm_bias, init_fn=init_method_constant(0.0)
            )
        else:
            self.layer_norm_bias = None

        # Initialize params in FP8
        with_fp8_params = FP8GlobalStateManager.with_fp8_parameters()

        # Contiguous buffers for params
        weight_tensor = torch.empty(
            self.out_features,
            self.in_features,
            device=device,
            dtype=params_dtype,
        )
        bias_tensor = None
        if self.use_bias:
            bias_tensor = torch.empty(
                self.out_features,
                device=device,
                dtype=params_dtype,
            )

        # Configure parameter splits
        self.weight_names = []
        self.bias_names = []
        self.parameter_split_sizes = []
        if parameters_split is None:
            # Split into a single parameter by default
… 261 more lines
Agent → toolmessage 95
read_file
{
  "file_path": "/workspace/TransformerEngine/transformer_engine/pytorch/module/layernorm_linear.py",
  "offset": 1700,
  "limit": 294
}
Tool result · read_filemessage 96
/workspace/TransformerEngine/transformer_engine/pytorch/module/layernorm_linear.pylines 1701-1994
@@ lines 1701-1994 of 1994 @@
                weight_quantizer,
                output_quantizer,
                grad_input_quantizer,
                grad_weight_quantizer,
                grad_output_quantizer,
            ) = quantizers

            if is_grad_enabled:
                fwd_fn = _LayerNormLinear.apply
                autograd_ctx = []
            else:
                fwd_fn = _LayerNormLinear.forward
                autograd_ctx = [None]
            cache_name = None if (is_first_microbatch is None or self.is_fsdp2) else "weight"
            weight_workspace = (
                self._fp8_workspaces.get(cache_name) if cache_name is not None else None
            )

            non_tensor_args = (
                self.eps,
                is_first_microbatch,
                self.fp8,
                self.fp8_calibration,
                self.wgrad_store,
                self.fuse_wgrad_accumulation,
                input_quantizer,
                weight_quantizer,
                output_quantizer,
                grad_input_quantizer,
                grad_weight_quantizer,
                grad_output_quantizer,
                is_cpu_offload_enabled(),
                self.tp_group,
                self.tp_size,
                self.sequence_parallel,
                self.tp_size > 1,
                self.activation_dtype,
                self.parallel_mode,
                self.return_layernorm_output,
… 255 more lines
Agent → toolmessage 97
grep
{
  "pattern": "def __init__",
  "path": "/workspace/TransformerEngine/transformer_engine/pytorch/module/layernorm_linear.py",
  "glob": null,
  "output_mode": "content",
  "max_count": 5
}
Tool result · grepmessage 98
/workspace/repo/transformer_engine/pytorch/module/layernorm_linear.py:
  1255:     def __init__(
Agent → toolmessage 99
read_file
{
  "file_path": "/workspace/TransformerEngine/transformer_engine/pytorch/module/layernorm_linear.py",
  "offset": 1255,
  "limit": 240
}
Tool result · read_filemessage 100
/workspace/TransformerEngine/transformer_engine/pytorch/module/layernorm_linear.pylines 1256-1495
@@ lines 1256-1495 of 1994 | next offset 1495 @@
        self,
        in_features: int,
        out_features: int,
        eps: float = 1e-5,
        sequence_parallel: bool = False,
        fuse_wgrad_accumulation: bool = False,
        tp_group: Optional[dist_group_type] = None,
        tp_size: int = 1,
        get_rng_state_tracker: Optional[Callable] = None,
        init_method: Optional[Callable] = None,
        bias: bool = True,
        normalization: str = "LayerNorm",
        return_bias: bool = False,
        params_dtype: Optional[torch.dtype] = None,
        parallel_mode: Optional[str] = None,
        return_layernorm_output: bool = False,
        return_layernorm_output_gathered: bool = False,
        parameters_split: Optional[Union[Tuple[str, ...], Dict[str, int]]] = None,
        zero_centered_gamma: bool = False,
        device: Union[torch.device, str] = "cuda",
        ub_overlap_ag: bool = False,
        ub_overlap_rs: bool = False,
        ub_overlap_rs_dgrad: bool = False,
        ub_bulk_wgrad: bool = False,
        ub_bulk_dgrad: bool = False,
        ub_name: Optional[str] = None,
        delay_wgrad_compute: bool = False,
        symmetric_ar_type: Optional[str] = None,
        name: Optional[str] = None,
    ) -> None:
        super().__init__(name)

        params_dtype = torch.get_default_dtype() if params_dtype is None else params_dtype
        self.in_features = in_features
        self.out_features = out_features
        self.fuse_wgrad_accumulation = fuse_wgrad_accumulation
        self.normalization = normalization
        assert normalization in ["LayerNorm", "RMSNorm"], "Unsupported normalization type!"
        self.use_bias = bias
… 200 more lines
Agent → toolmessage 101
read_file
{
  "file_path": "/workspace/TransformerEngine/transformer_engine/pytorch/module/layernorm_linear.py",
  "offset": 1495,
  "limit": 205
}
Tool result · read_filemessage 102
/workspace/TransformerEngine/transformer_engine/pytorch/module/layernorm_linear.pylines 1496-1700
@@ lines 1496-1700 of 1994 | next offset 1700 @@
            # Construct weight parameter
            self.register_parameter(
                self.weight_names[i],
                torch.nn.Parameter(weight_tensor[split_start:split_end]),
                init_fn=init_method,
                get_rng_state_tracker=get_rng_state_tracker,
                fp8_meta_index=FP8FwdTensorIdx.GEMM1_WEIGHT,
            )

        # Construct bias parameters if needed
        if self.use_bias:
            offset = 0
            for i, split_size in enumerate(self.parameter_split_sizes):
                split_start = offset
                offset += split_size
                split_end = offset
                self.register_parameter(
                    self.bias_names[i],
                    torch.nn.Parameter(bias_tensor[split_start:split_end]),
                    init_fn=init_method_constant(0.0),
                )
        else:
            for name in self.bias_names:
                bias = torch.Tensor().to(dtype=params_dtype, device=device)
                setattr(self, name, bias)

        if with_fp8_params:
            self.init_fp8_metadata()

        self.reset_parameters(defer_init=device == "meta")

        # For RPL, bias has to be added after TP collectives
        # So it cannot be fused with the GEMM
        if self.parallel_mode == "row" and self.apply_bias:
            self.gemm_bias_unfused_add = True
        else:
            self.gemm_bias_unfused_add = False

        # These many SMs are subtracted from the total SM count when calling forward
… 166 more lines
Agent → toolmessage 103
grep
{
  "pattern": "save_original_input",
  "path": "/workspace/TransformerEngine/transformer_engine/pytorch/module/layernorm_linear.py",
  "glob": null,
  "output_mode": "content",
  "max_count": 20
}
Tool result · grepmessage 104
No matches found
Agent → toolmessage 105
grep
{
  "pattern": "backward_override",
  "path": "/workspace/TransformerEngine/transformer_engine/pytorch/module/layernorm_linear.py",
  "glob": null,
  "output_mode": "content",
  "max_count": 30
}
Tool result · grepmessage 106
/workspace/repo/transformer_engine/pytorch/module/layernorm_linear.py:
  150:             backward_override = FP8GlobalStateManager.get_fp8_recipe().backward_override
  152:             backward_override = None
  213:                 columnwise=backward_needs_input and backward_override is None,
  227:             and backward_override is None
  251:         ln_out_hp = ln_out if backward_override == "high_precision" else None
  319:                     columnwise=is_grad_enabled and not is_fsdp2 and backward_override is None,
  441:             if backward_override == "high_precision":
  449:             if backward_needs_input and backward_override is None:
  533:             ctx.backward_override = backward_override
  566:             if backward_override is not None:
  720:             if ctx.backward_override == "dequantized":
  810:             if ctx.backward_override == "dequantized":
  815:             elif ctx.backward_override == "high_precision":
Agent → toolmessage 107
read_file
{
  "file_path": "/workspace/TransformerEngine/transformer_engine/pytorch/module/layernorm_linear.py",
  "offset": 130,
  "limit": 210
}
Tool result · read_filemessage 108
/workspace/TransformerEngine/transformer_engine/pytorch/module/layernorm_linear.pylines 131-340
@@ lines 131-340 of 1994 | next offset 340 @@
            fwd_ln_sm_margin,
            bwd_ln_sm_margin,
            zero_centered_gamma,
            normalization,
            ub_overlap_ag_fprop,
            ub_overlap_rs_fprop,
            ub_overlap_ag_dgrad,
            ub_overlap_rs_dgrad,
            ub_bulk_wgrad,
            ub_bulk_dgrad,
            ub_name,
            fsdp_group,
            cache_weight,
            skip_fp8_weight_update,
            symmetric_ar_type,
            debug,
            is_fsdp2,
        ) = non_tensor_args
        if fp8:
            backward_override = FP8GlobalStateManager.get_fp8_recipe().backward_override
        else:
            backward_override = None

        # NVTX label for profiling
        nvtx_label = "transformer_engine._LayerNormLinear.forward"
        if ub_name is not None:
            nvtx_label = f"{nvtx_label}.{ub_name}"

        with_input_all_gather = parallel_mode == "column" and sequence_parallel

        # Make sure input dimensions are compatible
        out_features, in_features = weight.shape
        inp_shape = inp.shape
        inp_requires_grad = inp.requires_grad
        assert inp_shape[-1] == in_features, "GEMM not possible"
        inp = inp.view((-1, in_features))
        inputmat = inp
        if fp8:
            assert_dim_for_fp8_exec(inputmat, weight)
… 171 more lines
Agent → toolmessage 109
read_file
{
  "file_path": "/workspace/TransformerEngine/transformer_engine/pytorch/module/layernorm_linear.py",
  "offset": 340,
  "limit": 235
}
Tool result · read_filemessage 110
/workspace/TransformerEngine/transformer_engine/pytorch/module/layernorm_linear.pylines 341-575
@@ lines 341-575 of 1994 | next offset 575 @@
        # ------------------------------------------------------

        # Cast bias to expected dtype
        bias_dtype = activation_dtype
        if needs_quantized_gemm(ln_out_total) and activation_dtype == torch.float32:
            # cuBLAS does not support FP8 GEMM with FP32 bias, so we cast to BF16
            bias_dtype = torch.bfloat16
        bias = cast_if_needed(bias, bias_dtype) if bias is not None else bias

        # Calibrate quantizers if needed
        if not fp8 and fp8_calibration:
            if input_quantizer is not None:
                input_quantizer.calibrate(ln_out_total)
            if weight_quantizer is not None:
                weight_quantizer.calibrate(weight)

        # Choose whether to use GEMM kernel with split accumulator
        use_split_accumulator = _2X_ACC_FPROP
        if fp8:
            recipe = FP8GlobalStateManager.get_fp8_recipe()
            if hasattr(recipe, "fp8_gemm_fprop"):
                use_split_accumulator = recipe.fp8_gemm_fprop.use_split_accumulator

        # Configure output quantizer
        if output_quantizer is not None:
            output_quantizer.set_usage(rowwise=True, columnwise=False)

        # Output buffer for Userbuffers reduce-scatter
        reduce_scatter_out = None
        if ub_overlap_rs_fprop:
            out_shape = list(inp_shape)
            out_shape[0] //= tp_world_size
            out_shape[-1] = out_features
            reduce_scatter_out = torch.empty(out_shape, dtype=activation_dtype, device=inp.device)

        # ------------------------------------------------------
        # Forward GEMM
        # Note: y = x * w^T
        # ------------------------------------------------------
… 196 more lines
Agent → toolmessage 111
read_file
{
  "file_path": "/workspace/TransformerEngine/transformer_engine/pytorch/module/layernorm_linear.py",
  "offset": 575,
  "limit": 340
}
Tool result · read_filemessage 112
/workspace/TransformerEngine/transformer_engine/pytorch/module/layernorm_linear.pylines 576-915
@@ lines 576-915 of 1994 | next offset 915 @@
                ctx.reduce_and_update_bwd_fp8_tensors = False

        # ------------------------------------------------------
        # Cached state for backward pass is ready...
        # ------------------------------------------------------

        ln_out_for_return = None
        if return_layernorm_output:
            if return_layernorm_output_gathered:
                shape = list(inp_shape)
                shape[0] *= tp_size if with_input_all_gather else 1
                ln_out_for_return = ln_out_return.view(shape)
            else:
                ln_out_for_return = ln_out_return.view(inp_shape)
        return out, ln_out_for_return, new_weight_workspace

    @staticmethod
    def backward(
        ctx, *grad_outputs: Tuple[torch.Tensor, ...]
    ) -> Tuple[Union[torch.Tensor, None], ...]:
        # pylint: disable=missing-function-docstring

        # NVTX label for profiling
        nvtx_label = "transformer_engine._LayerNormLinear.backward"
        if ctx.ub_name is not None:
            nvtx_label = f"{nvtx_label}.{ctx.ub_name}"

        with get_nvtx_range_context("_LayerNormLinear_backward"):
            (  # pylint: disable=unbalanced-tuple-unpacking
                inputmat,
                weight,
                saved_weight,
                bias,
                ln_weight,
                ln_out,
                mu,
                rsigma,
            ) = restore_from_func_ctx(ctx)
… 301 more lines
Agent → toolmessage 113
grep
{
  "pattern": "backward_override",
  "path": "/workspace/TransformerEngine/transformer_engine/pytorch",
  "glob": "*.py",
  "output_mode": "content",
  "max_count": 100
}
Tool result · grepmessage 114
/workspace/repo/transformer_engine/pytorch/module/base.py:
  1577:         use_fp8_bwd = ctx.fp8 and ctx.backward_override is None
/workspace/repo/transformer_engine/pytorch/module/grouped_linear.py:
  93:         backward_override: Optional[str],
  124:             or backward_override is not None
  372:             ctx.backward_override = None
  429:             backward_override = FP8GlobalStateManager.get_fp8_recipe().backward_override
  431:             backward_override = None
  432:         if backward_override == "high_precision":
  434:         elif backward_override == "dequantized":
  465:                         and backward_override is None
  469:             if backward_override is not None:
  503:             backward_override=backward_override,
  632:                             if backward_override is not None:
  646:                 if backward_override == "high_precision" and inp.requires_grad
  687:             ctx.backward_override = backward_override
  707:             if backward_override is not None:
  1011:                 if ctx.backward_override == "dequantized":
  1020:                 elif ctx.backward_override == "high_precision":
  1093:                 elif ctx.backward_override == "dequantized":
/workspace/repo/transformer_engine/pytorch/module/layernorm_linear.py:
  150:             backward_override = FP8GlobalStateManager.get_fp8_recipe().backward_override
  152:             backward_override = None
  213:                 columnwise=backward_needs_input and backward_override is None,
  227:             and backward_override is None
  251:         ln_out_hp = ln_out if backward_override == "high_precision" else None
  319:                     columnwise=is_grad_enabled and not is_fsdp2 and backward_override is None,
  441:             if backward_override == "high_precision":
  449:             if backward_needs_input and backward_override is None:
  533:             ctx.backward_override = backward_override
  566:             if backward_override is not None:
  720:             if ctx.backward_override == "dequantized":
  810:             if ctx.backward_override == "dequantized":
  815:             elif ctx.backward_override == "high_precision":
/workspace/repo/transformer_engine/pytorch/module/layernorm_mlp.py:
  246:             backward_override = FP8GlobalStateManager.get_fp8_recipe().backward_override
  248:             backward_override = None
  249:         assert backward_override is None, (
  845:             ctx.backward_override = backward_override
/workspace/repo/transformer_engine/pytorch/module/linear.py:
… 73 more lines
Agent → toolmessage 115
read_file
{
  "file_path": "/workspace/TransformerEngine/tests/pytorch/test_backward_override.py",
  "offset": 0,
  "limit": 150
}
Agent → toolmessage 115
read_file
{
  "file_path": "/workspace/TransformerEngine/transformer_engine/pytorch/module/grouped_linear.py",
  "offset": 975,
  "limit": 130
}
Agent → toolmessage 115
read_file
{
  "file_path": "/workspace/TransformerEngine/transformer_engine/pytorch/module/layernorm_linear.py",
  "offset": 100,
  "limit": 260
}
Agent → toolmessage 115
read_file
{
  "file_path": "/workspace/TransformerEngine/transformer_engine/pytorch/module/layernorm_mlp.py",
  "offset": 225,
  "limit": 80
}
Agent → toolmessage 115
read_file
{
  "file_path": "/workspace/TransformerEngine/transformer_engine/pytorch/module/linear.py",
  "offset": 230,
  "limit": 130
}
Tool result · read_filemessage 116
/workspace/TransformerEngine/tests/pytorch/test_backward_override.pylines 1-150
@@ lines 1-150 of 2056 | next offset 150 @@
# Copyright (c) 2022-2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved.
#
# See LICENSE for license information.

from __future__ import annotations

from contextlib import nullcontext
import math
from typing import Optional

import pytest
import torch

import transformer_engine.pytorch as te
import transformer_engine.pytorch.ops as te_ops
from transformer_engine.common import recipe
from transformer_engine.pytorch.cpp_extensions import general_gemm, layernorm_bwd
from transformer_engine.pytorch.quantization import FP8GlobalStateManager
from transformer_engine.pytorch.utils import is_non_tn_fp8_gemm_supported
from transformer_engine.pytorch.ops.fused import (
    BackwardActivationBias,
    ForwardLinearBiasActivation,
    ForwardLinearBiasAdd,
    ForwardLinearScaleAdd,
    UserbuffersForwardLinear,
)
from transformer_engine.pytorch.quantized_tensor import restore_from_saved

from utils import (
    assert_close,
    make_recipe,
    reset_rng_states,
    skip_unsupported_backward_override,
)


# --------------------------
# Mode and capability config
# --------------------------
… 111 more lines
Tool result · read_filemessage 117
/workspace/TransformerEngine/transformer_engine/pytorch/module/grouped_linear.pylines 976-1105
@@ lines 976-1105 of 1979 | next offset 1105 @@
                grad_output = DebugQuantizer.multi_tensor_quantize(
                    grad_output_view,
                    ctx.grad_output_quantizers,
                    ctx.m_splits,
                    ctx.activation_dtype,
                )
            else:
                # Only split grad output. Grad bias is fused with
                # wgrad GEMM.
                grad_output = torch.split(
                    cast_if_needed(grad_output_view, ctx.activation_dtype),
                    ctx.m_splits,
                )

            if ctx.is_first_microbatch is not None:
                accumulate_wgrad_into_param_main_grad = (
                    ctx.fuse_wgrad_accumulation and not ctx.is_first_microbatch
                )
            else:
                accumulate_wgrad_into_param_main_grad = ctx.fuse_wgrad_accumulation

            if ctx.requires_dgrad:
                dgrad_gemm_use_split_accumulator = _2X_ACC_DGRAD
                if ctx.fp8 or ctx.debug:
                    recipe = ctx.fp8_recipe
                    if hasattr(recipe, "fp8_gemm_dgrad"):
                        dgrad_gemm_use_split_accumulator = (
                            recipe.fp8_gemm_dgrad.use_split_accumulator
                        )
                dgrad = torch.empty(
                    (sum(ctx.m_splits), ctx.weights_shape_1),
                    dtype=ctx.activation_dtype,
                    device=ctx.device,
                )
                weights_for_dgrad = weights
                if ctx.backward_override == "dequantized":
                    weights_for_dgrad = [
                        (
                            weight.dequantize(dtype=ctx.activation_dtype)
… 91 more lines
Tool result · read_filemessage 118
/workspace/TransformerEngine/transformer_engine/pytorch/module/layernorm_linear.pylines 101-360
@@ lines 101-360 of 1994 | next offset 360 @@
        bias: torch.Tensor,
        non_tensor_args: Tuple,
    ) -> Tuple[torch.Tensor, ...]:
        # pylint: disable=missing-function-docstring

        # Reduce number of arguments to autograd function in order
        # to reduce CPU overhead due to pytorch arg checking.
        (
            eps,
            is_first_microbatch,
            fp8,
            fp8_calibration,
            wgrad_store,
            fuse_wgrad_accumulation,
            input_quantizer,
            weight_quantizer,
            output_quantizer,
            grad_input_quantizer,
            grad_weight_quantizer,
            grad_output_quantizer,
            cpu_offloading,
            tp_group,
            tp_size,
            sequence_parallel,
            tensor_parallel,
            activation_dtype,
            parallel_mode,
            return_layernorm_output,
            return_layernorm_output_gathered,
            is_grad_enabled,
            fwd_ln_sm_margin,
            bwd_ln_sm_margin,
            zero_centered_gamma,
            normalization,
            ub_overlap_ag_fprop,
            ub_overlap_rs_fprop,
            ub_overlap_ag_dgrad,
            ub_overlap_rs_dgrad,
            ub_bulk_wgrad,
… 221 more lines
Tool result · read_filemessage 119
/workspace/TransformerEngine/transformer_engine/pytorch/module/layernorm_mlp.pylines 226-305
@@ lines 226-305 of 2783 | next offset 305 @@
            activation,
            activation_params,
            normalization,
            ub_overlap_ag,
            ub_overlap_rs,
            ub_overlap_rs_dgrad,
            ub_bulk_wgrad,
            ub_bulk_dgrad,
            gemm_gelu_fusion,
            fsdp_group,
            fp8_meta,
            cache_weight,
            skip_fp8_weight_update,
            symmetric_ar_type,
            checkpoint,
            debug,
            is_fsdp2,
            recompute_for_bwd,
        ) = non_tensor_args
        if fp8:
            backward_override = FP8GlobalStateManager.get_fp8_recipe().backward_override
        else:
            backward_override = None
        assert backward_override is None, (
            "NVTE_BACKWARD_OVERRIDE=high_precision/dequantized is not implemented in LayerNormMLP."
            " Replace LayerNormMLP with LayerNormLinear + Linear to enable"
            " high_precision/dequantized backward."
        )

        # if grad is enabled and this is not the bwd stage, we must save this so bwd knows which path to take
        if is_grad_enabled and not recompute_for_bwd:
            ctx.checkpoint = checkpoint
            if checkpoint:
                # save the state of autocast and quantizers for recomputation
                ctx.autocast_state = FP8GlobalStateManager.get_autocast_state()
                if (
                    fp8
                    and FP8GlobalStateManager.get_fp8_recipe().__class__.__name__
                    == "DelayedScaling"
… 41 more lines
Tool result · read_filemessage 120
/workspace/TransformerEngine/transformer_engine/pytorch/module/linear.pylines 231-360
@@ lines 231-360 of 2171 | next offset 360 @@
        """Pull saved tensors from ``ctx`` into the fields backward consumes."""
        (
            self.inputmat,
            self.weight_fp8,
            self.saved_weight,
            self.bias,
        ) = restore_from_func_ctx(
            ctx
        )  # pylint: disable=unbalanced-tuple-unpacking


def _check_fp8_reduce_and_update():
    """Check if this is the first FP8 module (for backward reduce-and-update)."""
    qstate = FP8GlobalStateManager.quantization_state
    _first_fp8_module = qstate.is_first_fp8_module
    result = FP8GlobalStateManager.is_first_fp8_module()
    if in_fp8_activation_recompute_phase():
        qstate.is_first_fp8_module = _first_fp8_module
    return result


def _linear_forward_impl(
    args: LinearFwdArgs,
) -> Tuple[torch.Tensor, Optional[torch.Tensor], Optional[Tuple], None, Optional[Dict]]:
    """Forward implementation for the linear layer.

    Returns ``(out, new_weight_workspace, tensors_to_save_from_forward, None,
    ctx_attrs)``. ``new_weight_workspace`` is the freshly produced FP8 weight
    workspace (returned alongside ``out`` so the caller can refresh its
    cache). The last three are ``None`` when gradients are disabled.
    """

    weight = args.weight
    inp = args.inp
    bias = args.bias
    input_quantizer = args.input_quantizer
    weight_quantizer = args.weight_quantizer
    output_quantizer = args.output_quantizer
    is_first_microbatch = args.is_first_microbatch
… 91 more lines
Agent → toolmessage 121
read_file
{
  "file_path": "/workspace/TransformerEngine/transformer_engine/pytorch/module/linear.py",
  "offset": 360,
  "limit": 170
}
Tool result · read_filemessage 122
/workspace/TransformerEngine/transformer_engine/pytorch/module/linear.pylines 361-530
@@ lines 361-530 of 2171 | next offset 530 @@
        if with_input_all_gather_nccl:  # Perform NCCL all-gather
            inputmat_total, _ = gather_along_first_dim(
                inputmat,
                tp_group,
                quantizer=quantizer,
            )
        elif ub_overlap_ag_fprop:  # Initialize Userbuffers all-gather
            inputmat_total, _ = fill_userbuffers_buffer_for_all_gather(
                ub_obj,
                inputmat,
                quantizer,
                tp_group,
            )

    else:  # Do not all-gather input tensor
        if fp8 or debug:
            if isinstance(inputmat, QuantizedTensorStorage):
                inputmat.update_usage(rowwise_usage=True)
            else:
                if input_quantizer is None:
                    raise ValueError("Missing quantizer for input tensor")
                input_quantizer.set_usage(
                    rowwise=True,
                    columnwise=(
                        backward_needs_input
                        and not save_original_input
                        and backward_override is None
                    ),
                )
                inputmat = input_quantizer(inputmat)
                own_quantized_input = True
        else:
            inputmat = cast_if_needed(inp, activation_dtype)  # Cast for AMP
        inputmat_total = inputmat

    if is_cpu_offload_enabled():
        start_offload(inputmat)
    nvtx_range_pop(f"{nvtx_label}.input_cast_comm")
    # ------------------------------------------------------
… 131 more lines
Agent → toolmessage 123
read_file
{
  "file_path": "/workspace/TransformerEngine/transformer_engine/pytorch/module/linear.py",
  "offset": 530,
  "limit": 150
}
Tool result · read_filemessage 124
/workspace/TransformerEngine/transformer_engine/pytorch/module/linear.pylines 531-680
@@ lines 531-680 of 2171 | next offset 680 @@
    # (e.g. ``bias_dropout_add`` residual connections inside ``TransformerLayer``).
    out = out.view(-1, *inp.shape[1:-1], out_features)

    # ------------------------------------------------------
    # Output tensor is ready to return...
    # ------------------------------------------------------

    # Prepare backward state
    tensors_to_save_from_forward = None
    ctx_attrs = None

    if is_grad_enabled:
        if save_original_input:
            inputmat = inp

        # Discard unneeded data in input tensor
        if (
            backward_needs_input
            and own_quantized_input
            and isinstance(inputmat, QuantizedTensorStorage)
        ):
            if backward_override is not None:
                inputmat.update_usage(rowwise_usage=True, columnwise_usage=False)
            elif (
                args.backward_input_needs_gather
                and weight_quantizer.supports_only_rowwise_all_gather()
            ):
                # All-gather is not supported with FP8 column-wise data
                inputmat.update_usage(rowwise_usage=True, columnwise_usage=False)
            else:
                # Discard row-wise data since it is not needed in backward pass
                inputmat.update_usage(rowwise_usage=False, columnwise_usage=True)

        # Cached input tensor
        saved_inputmat = None
        if backward_needs_input:
            saved_inputmat = inputmat

        if cpu_offloading and saved_inputmat is not None:
… 111 more lines
Agent → toolmessage 125
read_file
{
  "file_path": "/workspace/TransformerEngine/transformer_engine/pytorch/module/linear.py",
  "offset": 925,
  "limit": 180
}
Tool result · read_filemessage 126
/workspace/TransformerEngine/transformer_engine/pytorch/module/linear.pylines 926-1105
@@ lines 926-1105 of 2171 | next offset 1105 @@
            inputmat_total = inputmat
        # --------------------------------------------------
        # Input tensor is ready for computing grad weight...
        # --------------------------------------------------

        # --------------------------------------------------
        # Compute grad input tensor
        # --------------------------------------------------

        dgrad = None
        dgrad_work = None
        if bwd_args.requires_dgrad:

            # FSDP2: Re-create workspace from all-gathered weight when
            # workspace was not saved. (Issue #2681)
            # Use saved_weight (the original weight parameter) since
            # weight_fp8 is only set when workspace was saved.
            if weight_fp8 is None:
                if isinstance(saved_weight, QuantizedTensorStorage):
                    # saved weight is already set to right usages by
                    # fsdp2 quantized-tensor hooks when workspace was not saved.
                    weight_fp8 = saved_weight
                elif bwd_args.weight_quantizer is not None:
                    bwd_args.weight_quantizer.set_usage(rowwise=True, columnwise=True)
                    weight_fp8 = bwd_args.weight_quantizer(saved_weight)

            # Make sure required data is available
            if isinstance(grad_output, QuantizedTensorStorage):
                grad_output.update_usage(rowwise_usage=True)
            if (
                bwd_args.fp8
                and weight_quantizer is not None
                and isinstance(weight_fp8, QuantizedTensorStorage)
            ):
                weight_fp8.update_usage(columnwise_usage=True)

            # Choose whether to use GEMM kernel with split accumulator
            use_split_accumulator = _2X_ACC_DGRAD
            if bwd_args.fp8:
… 140 more lines
Agent → toolmessage 127
read_file
{
  "file_path": "/workspace/TransformerEngine/transformer_engine/pytorch/module/linear.py",
  "offset": 830,
  "limit": 100
}
Tool result · read_filemessage 128
/workspace/TransformerEngine/transformer_engine/pytorch/module/linear.pylines 831-930
@@ lines 831-930 of 2171 | next offset 930 @@

        # Adjust the quantization direction approach depending
        # on whether wgrad calculations will be performed.
        # NOTE: If requires_dgrad is False, disabling `rowwise` quantization and keeping `columnwise` quantization
        #       results in `Assertion failed: output_tensor->has_data(). Quantizing in only the columnwise direction not supported yet!`
        # NOTE: For `bias is True`, selected quantize kernel errors with
        #       `cast_kernels.cuh:1322 in function fp8_quantize_arch_l_100: Not implemented scaling mode or fusion: NVTE_DELAYED_TENSOR_SCALING or IS_DBIAS=true on GPU with compute capability < 10.0.`
        if (
            not bwd_args.use_bias
            and not bwd_args.requires_wgrad
            and grad_output_quantizer is not None
        ):
            grad_output_quantizer.set_usage(columnwise=False)

        # Prepare grad output tensor.
        # ``grad_output_preprocess`` accesses a small set of attributes
        # (sequence_parallel, fp8, backward_override, debug, ub_overlap_ag,
        # tp_group, ub_obj_gradout, use_bias). ``LinearBwdArgs`` exposes the
        # same names so we can pass it directly.
        nvtx_range_push(f"{nvtx_label}.grad_output_preprocess")
        (
            grad_output,
            grad_bias,
        ) = TransformerEngineBaseModule.grad_output_preprocess(
            bwd_args,
            grad_output,
            bwd_args.parallel_mode == "row",
            grad_output_quantizer,
        )
        nvtx_range_pop(f"{nvtx_label}.grad_output_preprocess")

        # --------------------------------------------------
        # Grad output tensor is ready for computing grad input...
        # --------------------------------------------------

        # --------------------------------------------------
        # Prepare input tensor
        # Note: Input tensor is needed for wgrad GEMM.
        # Tensor-parallel communication is overlapped with dgrad
… 60 more lines
Agent → toolmessage 129
read_file
{
  "file_path": "/workspace/TransformerEngine/transformer_engine/pytorch/module/linear.py",
  "offset": 1750,
  "limit": 180
}
Tool result · read_filemessage 130
/workspace/TransformerEngine/transformer_engine/pytorch/module/linear.pylines 1751-1930
@@ lines 1751-1930 of 2171 | next offset 1930 @@
        elif recipe.nvfp4():
            self._customize_quantizers_nvfp4(fwd, recipe)

    def reset_parameters(self, defer_init=False):
        super().reset_parameters(defer_init=defer_init)

        if not defer_init:
            # Set parallelism attributes for linear weights
            for weight in self.weight_names:
                set_tensor_model_parallel_attributes(
                    tensor=getattr(self, weight),
                    is_parallel=True,
                    dim=1 if self.parallel_mode == "row" else 0,
                    stride=1,
                )

            # Set parallelism attributes for linear biases
            if self.use_bias:
                for bias in self.bias_names:
                    if self.parallel_mode == "row":
                        setattr(getattr(self, bias), "sequence_parallel", self.sequence_parallel)
                    elif self.parallel_mode == "column":
                        set_tensor_model_parallel_attributes(getattr(self, bias), True, 0, 1)

    @no_torch_dynamo()
    def forward(
        self,
        inp: torch.Tensor,
        is_first_microbatch: Optional[bool] = None,
        fp8_output: Optional[bool] = False,
        fp8_grad: Optional[bool] = False,
    ) -> Union[torch.Tensor, Tuple[torch.Tensor, ...]]:
        """
        Apply the linear transformation to the input.

        Parameters
        ----------
        inp : torch.Tensor
             Input tensor.
… 141 more lines
Agent → toolmessage 131
grep
{
  "pattern": "save_original_input",
  "path": "/workspace/TransformerEngine/transformer_engine/pytorch/module/linear.py",
  "glob": null,
  "output_mode": "content",
  "max_count": 20
}
Tool result · grepmessage 132
/workspace/repo/transformer_engine/pytorch/module/linear.py:
  118:     save_original_input: bool
  282:     save_original_input = args.save_original_input
  287:         save_original_input = True
  289:         save_original_input = False
  327:         if save_original_input:
  330:             ), "DelayedScaling recipe is not supported with save_original_input"
  347:                 if save_original_input:
  386:                         and not save_original_input
  543:         if save_original_input:
  1456:     save_original_input : bool, default = False
  1488:         save_original_input: bool = False,
  1503:         self.save_original_input = save_original_input
  1920:                 save_original_input=self.save_original_input,
Agent → toolmessage 133
read_file
{
  "file_path": "/workspace/TransformerEngine/transformer_engine/pytorch/module/linear.py",
  "offset": 260,
  "limit": 150
}
Tool result · read_filemessage 134
/workspace/TransformerEngine/transformer_engine/pytorch/module/linear.pylines 261-410
@@ lines 261-410 of 2171 | next offset 410 @@
    """

    weight = args.weight
    inp = args.inp
    bias = args.bias
    input_quantizer = args.input_quantizer
    weight_quantizer = args.weight_quantizer
    output_quantizer = args.output_quantizer
    is_first_microbatch = args.is_first_microbatch
    fp8 = args.fp8
    cpu_offloading = args.cpu_offloading
    tp_group = args.tp_group
    sequence_parallel = args.sequence_parallel
    activation_dtype = args.activation_dtype
    parallel_mode = args.parallel_mode
    is_grad_enabled = args.is_grad_enabled
    ub_overlap_rs_fprop = args.ub_overlap_rs_fprop
    ub_overlap_ag_fprop = args.ub_overlap_ag_fprop
    ub_name = args.ub_name
    fsdp_group = args.fsdp_group
    symmetric_ar_type = args.symmetric_ar_type
    save_original_input = args.save_original_input
    debug = args.debug
    backward_override = args.backward_override
    is_fsdp2 = args.is_fsdp2
    if backward_override == "high_precision":
        save_original_input = True
    elif backward_override == "dequantized":
        save_original_input = False

    # NVTX label for profiling
    nvtx_label = "transformer_engine._Linear.forward"
    if ub_name is not None:
        nvtx_label = f"{nvtx_label}.{ub_name}"

    # Make sure input dimensions are compatible
    out_features, in_features = weight.shape
    assert inp.shape[-1] == in_features, "GEMM not possible"
… 111 more lines
Agent → toolmessage 135
read_file
{
  "file_path": "/workspace/TransformerEngine/transformer_engine/pytorch/module/linear.py",
  "offset": 520,
  "limit": 100
}
Tool result · read_filemessage 136
/workspace/TransformerEngine/transformer_engine/pytorch/module/linear.pylines 521-620
@@ lines 521-620 of 2171 | next offset 620 @@
                out, _ = symmetric_all_reduce(out, tp_group, all_reduce_type=symmetric_ar_type)
            else:
                out, _ = allreduce(out, tp_group)
        nvtx_range_pop(f"{nvtx_label}.row_parallel_comm")
    else:
        out = gemm_out

    # Restore the input's logical rank (e.g., (seq, batch, hidden)) on the output.
    # This is mainly to correct for cuBLASMp comm+GEMM operators that unconditionally
    # return a 2D output buffer that ends up incompatible with downstream consumers
    # (e.g. ``bias_dropout_add`` residual connections inside ``TransformerLayer``).
    out = out.view(-1, *inp.shape[1:-1], out_features)

    # ------------------------------------------------------
    # Output tensor is ready to return...
    # ------------------------------------------------------

    # Prepare backward state
    tensors_to_save_from_forward = None
    ctx_attrs = None

    if is_grad_enabled:
        if save_original_input:
            inputmat = inp

        # Discard unneeded data in input tensor
        if (
            backward_needs_input
            and own_quantized_input
            and isinstance(inputmat, QuantizedTensorStorage)
        ):
            if backward_override is not None:
                inputmat.update_usage(rowwise_usage=True, columnwise_usage=False)
            elif (
                args.backward_input_needs_gather
                and weight_quantizer.supports_only_rowwise_all_gather()
            ):
                # All-gather is not supported with FP8 column-wise data
                inputmat.update_usage(rowwise_usage=True, columnwise_usage=False)
… 61 more lines
Agent → toolmessage 137
read_file
{
  "file_path": "/workspace/TransformerEngine/review-diff-434ed8e16a2a16ad.patch",
  "offset": 0,
  "limit": 200
}
Tool result · read_filemessage 138
/workspace/TransformerEngine/review-diff-434ed8e16a2a16ad.patchlines 1-200
@@ lines 1-200 of 206 | next offset 200 @@
diff --git a/tests/pytorch/test_backward_override.py b/tests/pytorch/test_backward_override.py
index 5e6f36e..ac05cc1 100644
--- a/tests/pytorch/test_backward_override.py
+++ b/tests/pytorch/test_backward_override.py
@@ -858,6 +858,175 @@ def test_backward_override_recipe_matches_requested_mode(
     assert quant_recipe.backward_override is None
 
 
+@pytest.mark.parametrize("recipe_name", _quantized_numerics_recipe_list)
+@pytest.mark.parametrize("use_bias", (False, True), ids=("no_bias", "bias"))
+def test_linear_backward_override_dequantized_ignores_save_original_input(
+    recipe_name: str,
+    use_bias: bool,
+) -> None:
+    reset_rng_states()
+    dtype = torch.bfloat16
+    input_shape = (32, 128)
+    out_features = 128
+    _maybe_skip_recipe_dtype(recipe_name, dtype, "linear")
+    _maybe_skip_unsupported_recipe_module_combo(recipe_name, "linear")
+    _maybe_skip_unsupported_recipe_shape(recipe_name, input_shape, "linear")
+
+    mode_recipe = make_recipe(recipe_name, backward_override="dequantized")
+    skip_unsupported_backward_override("linear", mode_recipe, "dequantized")
+
+    module_ref = te.Linear(
+        input_shape[-1],
+        out_features,
+        bias=use_bias,
+        params_dtype=dtype,
+        device="cuda",
+        save_original_input=False,
+    )
+    module_test = te.Linear(
+        input_shape[-1],
+        out_features,
+        bias=use_bias,
+        params_dtype=dtype,
+        device="cuda",
… 161 more lines
Agent → toolmessage 139
read_file
{
  "file_path": "/workspace/TransformerEngine/review-diff-434ed8e16a2a16ad.patch",
  "offset": 190,
  "limit": 30
}
Tool result · read_filemessage 140
/workspace/TransformerEngine/review-diff-434ed8e16a2a16ad.patchlines 191-206
@@ lines 191-206 of 206 @@
 
         num_gemms = len(m_splits)
         weights = weights_and_biases[:num_gemms]
diff --git a/transformer_engine/pytorch/module/linear.py b/transformer_engine/pytorch/module/linear.py
index 6c2d98d..c68c689 100644
--- a/transformer_engine/pytorch/module/linear.py
+++ b/transformer_engine/pytorch/module/linear.py
@@ -285,6 +285,8 @@ def _linear_forward_impl(
     is_fsdp2 = args.is_fsdp2
     if backward_override == "high_precision":
         save_original_input = True
+    elif backward_override == "dequantized":
+        save_original_input = False
 
     # NVTX label for profiling
     nvtx_label = "transformer_engine._Linear.forward"
Agent → toolmessage 141
list_findings
{
  "status_filter": null
}
Tool result · list_findingsmessage 142
{"findings": [{"id": "f_edab778e3d", "severity": "low", "confidence": "high", "category": "correctness", "title": "Override silently changes input-saving behavior", "file": "transformer_engine/pytorch/module/linear.py", "start_line": 289, "end_line": 289, "side": "RIGHT", "in_diff": true, "description": "`dequantized` forces `save_original_input=False` even when the module was configured to save the original input. Users selecting `dequantized` therefore receive a different memory-saving behavior without an explicit warning or documented interaction.", "suggestion": null, "status": "open", "first_seen_sha": "941db9b0d5457c986f6b9a06bd960d6186c8a592", "last_confirmed_sha": "941db9b0d5457c986f6b9a06bd960d6186c8a592", "github_review_id": null, "github_review_run_id": null, "github_review_comment_ids": [], "github_review_thread_ids": [], "github_resolved_thread_ids": [], "github_posted_resolution_comment_ids": [], "surface_state": "not_surfaced", "last_human_reply_at": null, "last_human_reply_author": null, "last_human_reply_body": null, "last_reconciliation_note": null, "resolution_note": null, "diff_hunk": "@@ -285,6 +285,8 @@ def _linear_forward_impl(\n     is_fsdp2 = args.is_fsdp2\n     if backward_override == \"high_precision\":\n         save_original_input = True\n+    elif backward_override == \"dequantized\":\n+        save_original_input = False\n \n     # NVTX label for profiling\n     nvtx_label = \"transformer_engine._Linear.forward\"", "fingerprint": "v1:16c06dd4504ddfa3e3217b0fb9e8ed4f0cf4673fbca6a46869b0b00ac9064729", "interactions": [], "rank": null}], "count": 1}
Agent → toolmessage 143
update_finding
{
  "finding_id": "f_edab778e3d",
  "status": null,
  "severity": null,
  "confidence": null,
  "title": null,
  "description": null,
  "suggestion": null,
  "note": null
}
Tool result · update_findingmessage 144
{"success": false, "error": "No fields provided to update"}
Agent → toolmessage 145
publish_review
{
  "ranking": [
    "f_edab778e3d"
  ],
  "severity_threshold": "medium",
  "assessment": null
}
Tool result · publish_reviewmessage 146
{"success": true, "dry_run": true, "review_id": null, "surfaced_count": 1, "hidden_count": 0, "resolved_thread_count": 0}
Agentmessage 147
Simulated publish (eval mode): review not posted to GitHub.

- Low: Override silently changes input-saving behavior (transformer_engine/pytorch/module/linear.py:289): dequantized forces save_original_input=False even when the module was configured to save the original input. Users selecting dequantized therefore receive a different memory-saving behavior without an explicit warning or documented interaction.

RLF2 1.0

25 messages8 model calls144,890 tokens read · 1,273 output
1 message omitted
Task messagemessage 2
Review pull request https://github.com/NVIDIA/TransformerEngine/pull/3141. - repo: NVIDIA/TransformerEngine - pr_number: 3141 - title: - base_sha: 43206d9729a8988858eee303c79afc7208f518cd - head_sha: 941db9b0d5457c986f6b9a06bd960d6186c8a592 - base_ref: 43206d9729a8988858eee303c79afc7208f518cd - head_ref: 941db9b0d5457c986f6b9a06bd960d6186c8a592 Record each issue you find with the `add_finding` tool, then call `publish_review` once at the end. Review the local frozen checkout. Do not retrieve existing PR reviews or discussions. Do not modify source or apply fixes. This image supports static code review. Repository dependencies are not installed and repository tests are not guaranteed runnable. Do not install dependencies or claim execution-based validation; use the frozen source and local Git history.
Agent → toolmessage 3
fetch_review_diff
{}
Tool result · fetch_review_diffmessage 4
{"success": true, "path": "/workspace/TransformerEngine/review-diff-434ed8e16a2a16ad.patch", "bytes": 8265, "files": ["tests/pytorch/test_backward_override.py", "transformer_engine/pytorch/module/grouped_linear.py", "transformer_engine/pytorch/module/linear.py"], "file_count": 3, "files_truncated": false, "base_sha": "43206d9729a8988858eee303c79afc7208f518cd", "head_sha": "941db9b0d5457c986f6b9a06bd960d6186c8a592", "cached": true}
Agent → toolmessage 5
glob
{
  "pattern": "AGENTS.md",
  "path": "/workspace/TransformerEngine"
}
Agent → toolmessage 5
glob
{
  "pattern": "CLAUDE.md",
  "path": "/workspace/TransformerEngine"
}
Agent → toolmessage 5
read_file
{
  "file_path": "/workspace/TransformerEngine/review-diff-434ed8e16a2a16ad.patch",
  "offset": 0,
  "limit": 300
}
Tool result · globmessage 6
No files found
Tool result · globmessage 7
No files found
Tool result · read_filemessage 8
/workspace/TransformerEngine/review-diff-434ed8e16a2a16ad.patchlines 1-206
@@ lines 1-206 of 206 @@
diff --git a/tests/pytorch/test_backward_override.py b/tests/pytorch/test_backward_override.py
index 5e6f36e..ac05cc1 100644
--- a/tests/pytorch/test_backward_override.py
+++ b/tests/pytorch/test_backward_override.py
@@ -858,6 +858,175 @@ def test_backward_override_recipe_matches_requested_mode(
     assert quant_recipe.backward_override is None
 
 
+@pytest.mark.parametrize("recipe_name", _quantized_numerics_recipe_list)
+@pytest.mark.parametrize("use_bias", (False, True), ids=("no_bias", "bias"))
+def test_linear_backward_override_dequantized_ignores_save_original_input(
+    recipe_name: str,
+    use_bias: bool,
+) -> None:
+    reset_rng_states()
+    dtype = torch.bfloat16
+    input_shape = (32, 128)
+    out_features = 128
+    _maybe_skip_recipe_dtype(recipe_name, dtype, "linear")
+    _maybe_skip_unsupported_recipe_module_combo(recipe_name, "linear")
+    _maybe_skip_unsupported_recipe_shape(recipe_name, input_shape, "linear")
+
+    mode_recipe = make_recipe(recipe_name, backward_override="dequantized")
+    skip_unsupported_backward_override("linear", mode_recipe, "dequantized")
+
+    module_ref = te.Linear(
+        input_shape[-1],
+        out_features,
+        bias=use_bias,
+        params_dtype=dtype,
+        device="cuda",
+        save_original_input=False,
+    )
+    module_test = te.Linear(
+        input_shape[-1],
+        out_features,
+        bias=use_bias,
+        params_dtype=dtype,
+        device="cuda",
… 167 more lines
Agent → toolmessage 9
grep
{
  "pattern": "dequantized",
  "path": "/workspace/TransformerEngine/transformer_engine",
  "glob": "*.py",
  "output_mode": "content",
  "max_count": 100
}
Agent → toolmessage 9
grep
{
  "pattern": "save_original_input",
  "path": "/workspace/TransformerEngine/transformer_engine/pytorch",
  "glob": "*.py",
  "output_mode": "content",
  "max_count": 100
}
Agent → toolmessage 9
read_file
{
  "file_path": "/workspace/TransformerEngine/transformer_engine/pytorch/module/grouped_linear.py",
  "offset": 360,
  "limit": 250
}
Agent → toolmessage 9
read_file
{
  "file_path": "/workspace/TransformerEngine/transformer_engine/pytorch/module/linear.py",
  "offset": 220,
  "limit": 260
}
Tool result · grepmessage 13
/workspace/repo/transformer_engine/common/recipe/__init__.py:
  15: _BACKWARD_OVERRIDES = (None, "high_precision", "dequantized")
  236:     backward_override : {None, 'high_precision', 'dequantized'}, default = None
  267:         ), "NVTE_BACKWARD_OVERRIDE must be unset or one of: 'high_precision', 'dequantized'."
  295:     backward_override : {None, 'high_precision', 'dequantized'}, default = None
  298:             and `dequantized` dequantizes saved operands to the active high-precision
  318:         ), "NVTE_BACKWARD_OVERRIDE must be unset or one of: 'high_precision', 'dequantized'."
  359:     backward_override : {None, 'high_precision', 'dequantized'}, default = None
  362:             and `dequantized` dequantizes saved operands to the active high-precision
  376:         ), "NVTE_BACKWARD_OVERRIDE must be unset or one of: 'high_precision', 'dequantized'."
  412:     backward_override : {None, 'high_precision', 'dequantized'}, default = None
  415:             and `dequantized` dequantizes saved operands to the active high-precision
  457:         ), "NVTE_BACKWARD_OVERRIDE must be unset or one of: 'high_precision', 'dequantized'."
  540:     backward_override : {None, 'high_precision', 'dequantized'}, default = None
  543:             and `dequantized` dequantizes saved operands to the active high-precision
  571:         ), "NVTE_BACKWARD_OVERRIDE must be unset or one of: 'high_precision', 'dequantized'."
  658:     backward_override : {None, 'high_precision', 'dequantized'}, default = None
  661:         and `dequantized` dequantizes saved operands to the active high-precision
  679:         ), "NVTE_BACKWARD_OVERRIDE must be unset or one of: 'high_precision', 'dequantized'."
/workspace/repo/transformer_engine/debug/features/dump_tensors.py:
  182:         print(f"MSE between original and dequantized: {mse}")
/workspace/repo/transformer_engine/debug/features/fake_quant.py:
  26:     """Input tensor is quantized to fp8 and then dequantized."""
/workspace/repo/transformer_engine/jax/quantize/dequantizer.py:
  85:             The dequantized tensor in the specified data type
  106:             dq_dtype: The data type for dequantized values
  112:             The dequantized tensor
  148:             dq_dtype: The data type for dequantized values
  154:             The dequantized tensor
  183:             dq_dtype: The data type for dequantized values
  190:             The dequantized tensor
  242:             The dequantized tensor
  304:         List of dequantized tensors for each group
/workspace/repo/transformer_engine/jax/quantize/quantizer.py:
  153:             dq_dtype: Data type for dequantized values, default is x.dtype
  169:             dq_dtype: Data type for dequantized values
  245:             dq_dtype: Data type for dequantized values
  280:             dq_dtype: Data type for dequantized values
  374:             dq_dtype: Data type for dequantized values
  495:             dq_dtype: Data type for dequantized values
… 42 more lines
Tool result · grepmessage 12
/workspace/repo/transformer_engine/pytorch/module/grouped_linear.py:
  94:         save_original_input: bool,
  125:             or save_original_input
  387:             ctx.save_original_input = False
  425:             save_original_input,
  433:             save_original_input = True
  435:             save_original_input = False
  444:         if save_original_input and isinstance(input_quantizers[0], Float8Quantizer):
  450:                     "save_original_input is incompatible with delayed-scaling quantizers "
  451:                     "(Float8Quantizer). Disabling save_original_input for this module.",
  454:                 save_original_input = False
  456:                 raise ValueError("DelayedScaling recipe is not supported with save_original_input")
  464:                         and not save_original_input
  504:             save_original_input=save_original_input,
  626:                 if save_original_input:
  703:             ctx.save_original_input = save_original_input
  1066:                 if ctx.save_original_input:
  1229:     save_original_input : bool, default = False
  1275:         save_original_input: bool = False,
  1293:         self.save_original_input = save_original_input
  1787:                 self.save_original_input,
/workspace/repo/transformer_engine/pytorch/module/linear.py:
  118:     save_original_input: bool
  282:     save_original_input = args.save_original_input
  287:         save_original_input = True
  289:         save_original_input = False
  327:         if save_original_input:
  330:             ), "DelayedScaling recipe is not supported with save_original_input"
  347:                 if save_original_input:
  386:                         and not save_original_input
  543:         if save_original_input:
  1456:     save_original_input : bool, default = False
  1488:         save_original_input: bool = False,
  1503:         self.save_original_input = save_original_input
  1920:                 save_original_input=self.save_original_input,
Tool result · read_filemessage 11
/workspace/TransformerEngine/transformer_engine/pytorch/module/grouped_linear.pylines 361-610
@@ lines 361-610 of 1979 | next offset 610 @@
                    ctx.main_grad_funcs = [weights[i].get_main_grad for i in range(num_gemms)]
                else:
                    ctx.main_grad_funcs = [
                        lambda j=i: weights[j].main_grad for i in range(num_gemms)
                    ]
            ctx.device = device
            ctx.m_splits = None
            ctx.num_gemms = num_gemms
            ctx.activation_dtype = activation_dtype
            ctx.fp8 = fp8
            ctx.fp8_recipe = FP8GlobalStateManager.get_fp8_recipe() if fp8 else None
            ctx.backward_override = None
            ctx.fuse_wgrad_accumulation = fuse_wgrad_accumulation
            ctx.cpu_offloading = False
            ctx.is_first_microbatch = is_first_microbatch
            ctx.use_bias = use_bias
            ctx.inp_shape = inp.shape
            ctx.requires_dgrad = inp.requires_grad
            ctx.reduce_and_update_bwd_fp8_tensors = False
            if ctx.fp8 and requires_grad(inp, weights[0], biases[0]):
                ctx.reduce_and_update_bwd_fp8_tensors = (
                    ctx.reduce_and_update_bwd_fp8_tensors
                    or FP8GlobalStateManager.is_first_fp8_module()
                )
            ctx.wgrad_store = wgrad_store
            ctx.debug = False
            ctx.save_original_input = False
            ctx.input_quantizers = input_quantizers

        return out.view(-1, *inp.shape[1:-1], out.shape[-1]), new_workspaces

    # pylint: disable=keyword-arg-before-vararg
    @staticmethod
    def forward(
        ctx,
        inp: torch.Tensor,
        m_splits: torch.Tensor,
        non_tensor_args: Tuple,
        *weights_and_biases,
… 210 more lines
Tool result · read_filemessage 10
/workspace/TransformerEngine/transformer_engine/pytorch/module/linear.pylines 221-480
@@ lines 221-480 of 2171 | next offset 480 @@
    reduce_and_update_bwd_fp8_tensors: bool = False

    # --- Misc ---
    cpu_offloading: bool = False
    owns_input: bool = False

    # --- Per-backward scratch state (populated inside _linear_backward) ---
    ub_obj_gradout: Optional[Any] = None

    def setup_saved_tensors(self, ctx: torch.autograd.function.FunctionCtx) -> None:
        """Pull saved tensors from ``ctx`` into the fields backward consumes."""
        (
            self.inputmat,
            self.weight_fp8,
            self.saved_weight,
            self.bias,
        ) = restore_from_func_ctx(
            ctx
        )  # pylint: disable=unbalanced-tuple-unpacking


def _check_fp8_reduce_and_update():
    """Check if this is the first FP8 module (for backward reduce-and-update)."""
    qstate = FP8GlobalStateManager.quantization_state
    _first_fp8_module = qstate.is_first_fp8_module
    result = FP8GlobalStateManager.is_first_fp8_module()
    if in_fp8_activation_recompute_phase():
        qstate.is_first_fp8_module = _first_fp8_module
    return result


def _linear_forward_impl(
    args: LinearFwdArgs,
) -> Tuple[torch.Tensor, Optional[torch.Tensor], Optional[Tuple], None, Optional[Dict]]:
    """Forward implementation for the linear layer.

    Returns ``(out, new_weight_workspace, tensors_to_save_from_forward, None,
    ctx_attrs)``. ``new_weight_workspace`` is the freshly produced FP8 weight
    workspace (returned alongside ``out`` so the caller can refresh its
… 221 more lines
Agent → toolmessage 14
read_file
{
  "file_path": "/workspace/TransformerEngine/transformer_engine/pytorch/module/grouped_linear.py",
  "offset": 600,
  "limit": 130
}
Agent → toolmessage 14
read_file
{
  "file_path": "/workspace/TransformerEngine/transformer_engine/pytorch/module/grouped_linear.py",
  "offset": 990,
  "limit": 120
}
Agent → toolmessage 14
read_file
{
  "file_path": "/workspace/TransformerEngine/transformer_engine/pytorch/module/linear.py",
  "offset": 500,
  "limit": 100
}
Agent → toolmessage 14
read_file
{
  "file_path": "/workspace/TransformerEngine/transformer_engine/pytorch/module/linear.py",
  "offset": 950,
  "limit": 100
}
Tool result · read_filemessage 18
/workspace/TransformerEngine/transformer_engine/pytorch/module/grouped_linear.pylines 601-730
@@ lines 601-730 of 1979 | next offset 730 @@
            [out],
            output_quantizers,
            activation_dtype,
            single_output=True,
            m_splits=m_splits,
            bias=biases,
            use_bias=use_bias,
            use_split_accumulator=use_split_accumulator,
        )

        if fp8_calibration:
            for i in range(num_gemms):
                input_quantizers[i].calibrate(inputmats[i])
                weight_quantizers[i].calibrate(weights[i])

        if cpu_offloading:
            mark_not_offload(*weights_fp8, *weights)

        if is_grad_enabled:
            ctx.use_grouped_tensor_path = False
            ctx.weight_quantizers = weight_quantizers
            ctx.weights_shape_1 = weights[0].shape[1]

            # TODO: update after #1638 is merged. # pylint: disable=fixme
            if weight_requires_grad:
                if save_original_input:
                    inputmats = [None] * num_gemms
                    inputmats[0] = inp
                else:
                    for inputmat in inputmats:
                        if isinstance(inputmat, QuantizedTensorStorage):
                            if backward_override is not None:
                                # In dequantized mode we should dequantize directly from
                                # fprop quantized layouts without retargeting usage.
                                inputmat.update_usage(rowwise_usage=True, columnwise_usage=False)
                            else:
                                inputmat.update_usage(rowwise_usage=False, columnwise_usage=True)
            else:
                inputmats = [None] * num_gemms
… 91 more lines
Tool result · read_filemessage 15
/workspace/TransformerEngine/transformer_engine/pytorch/module/grouped_linear.pylines 991-1110
@@ lines 991-1110 of 1979 | next offset 1110 @@
                accumulate_wgrad_into_param_main_grad = (
                    ctx.fuse_wgrad_accumulation and not ctx.is_first_microbatch
                )
            else:
                accumulate_wgrad_into_param_main_grad = ctx.fuse_wgrad_accumulation

            if ctx.requires_dgrad:
                dgrad_gemm_use_split_accumulator = _2X_ACC_DGRAD
                if ctx.fp8 or ctx.debug:
                    recipe = ctx.fp8_recipe
                    if hasattr(recipe, "fp8_gemm_dgrad"):
                        dgrad_gemm_use_split_accumulator = (
                            recipe.fp8_gemm_dgrad.use_split_accumulator
                        )
                dgrad = torch.empty(
                    (sum(ctx.m_splits), ctx.weights_shape_1),
                    dtype=ctx.activation_dtype,
                    device=ctx.device,
                )
                weights_for_dgrad = weights
                if ctx.backward_override == "dequantized":
                    weights_for_dgrad = [
                        (
                            weight.dequantize(dtype=ctx.activation_dtype)
                            if isinstance(weight, QuantizedTensorStorage)
                            else cast_if_needed(weight, ctx.activation_dtype)
                        )
                        for weight in weights
                    ]
                elif ctx.backward_override == "high_precision":
                    weights_for_dgrad = [
                        (
                            weight.dequantize(dtype=ctx.activation_dtype)
                            if isinstance(weight, QuantizedTensorStorage)
                            else cast_if_needed(weight, ctx.activation_dtype)
                        )
                        for weight in saved_weights
                    ]
                # Make sure weights are available in column-wise format
… 81 more lines
Tool result · read_filemessage 16
/workspace/TransformerEngine/transformer_engine/pytorch/module/linear.pylines 501-600
@@ lines 501-600 of 2171 | next offset 600 @@
    if with_input_all_gather_nccl:
        clear_tensor_data(inputmat_total)
        inputmat_total = None

    # ------------------------------------------------------
    # Prepare output tensor
    # Note: Perform tensor-parallel communication
    # ------------------------------------------------------
    out = None
    if ub_overlap_rs_fprop:
        # cuBLASMp writes the reduce-scattered output directly into the GEMM
        # output tensor; Userbuffers writes it into the extra-output buffer.
        out = gemm_out if ub_obj is not None and ub_obj.with_cublasmp() else reduce_scatter_out
    elif parallel_mode == "row" and args.tp_size > 1:
        nvtx_range_push(f"{nvtx_label}.row_parallel_comm")
        out = gemm_out
        if sequence_parallel:
            out, _ = reduce_scatter_along_first_dim(out, tp_group)
        elif args.tensor_parallel:
            if symmetric_ar_type is not None:
                out, _ = symmetric_all_reduce(out, tp_group, all_reduce_type=symmetric_ar_type)
            else:
                out, _ = allreduce(out, tp_group)
        nvtx_range_pop(f"{nvtx_label}.row_parallel_comm")
    else:
        out = gemm_out

    # Restore the input's logical rank (e.g., (seq, batch, hidden)) on the output.
    # This is mainly to correct for cuBLASMp comm+GEMM operators that unconditionally
    # return a 2D output buffer that ends up incompatible with downstream consumers
    # (e.g. ``bias_dropout_add`` residual connections inside ``TransformerLayer``).
    out = out.view(-1, *inp.shape[1:-1], out_features)

    # ------------------------------------------------------
    # Output tensor is ready to return...
    # ------------------------------------------------------

    # Prepare backward state
    tensors_to_save_from_forward = None
… 61 more lines
Tool result · read_filemessage 17
/workspace/TransformerEngine/transformer_engine/pytorch/module/linear.pylines 951-1050
@@ lines 951-1050 of 2171 | next offset 1050 @@

            # Make sure required data is available
            if isinstance(grad_output, QuantizedTensorStorage):
                grad_output.update_usage(rowwise_usage=True)
            if (
                bwd_args.fp8
                and weight_quantizer is not None
                and isinstance(weight_fp8, QuantizedTensorStorage)
            ):
                weight_fp8.update_usage(columnwise_usage=True)

            # Choose whether to use GEMM kernel with split accumulator
            use_split_accumulator = _2X_ACC_DGRAD
            if bwd_args.fp8:
                recipe = bwd_args.fp8_recipe
                if hasattr(recipe, "fp8_gemm_dgrad"):
                    use_split_accumulator = recipe.fp8_gemm_dgrad.use_split_accumulator

            # Update grad input quantizer
            if grad_input_quantizer is not None:
                grad_input_quantizer.set_usage(rowwise=True, columnwise=False)

            # Output buffers for Userbuffers reduce-scatter
            gemm_out = None
            reduce_scatter_out = None
            if bwd_args.ub_overlap_rs_dgrad:
                reduce_scatter_out = torch.empty(
                    dgrad_shape,
                    dtype=bwd_args.activation_dtype,
                    device=grad_output_arg.device,
                )
            elif bwd_args.ub_bulk_wgrad:
                gemm_out = ub_obj_wgrad.get_buffer(local_chunk=False)

            # dgrad GEMM
            # Note: dx = dy * w

            nvtx_range_push(f"{nvtx_label}.dgrad_gemm")
            weight_for_dgrad = weight_fp8
… 61 more lines
Agent → toolmessage 19
add_finding
{
  "severity": "medium",
  "confidence": "high",
  "category": "correctness",
  "file": "transformer_engine/pytorch/module/linear.py",
  "title": "Dequantized override loses columnwise input data",
  "description": "Forcing `save_original_input = False` makes this forward enable columnwise quantization (`backward_needs_input and not save_original_input and backward_override is None` evaluates true only when `backward_override is None`, but the later discard logic selects the override branch). With `dequantized`, the saved quantized input is explicitly reduced to rowwise-only (`update_usage(rowwise_usage=True, columnwise_usage=False)`), so backward dequantizes the input and weight but no longer has the columnwise operand required by the FP8 wgrad path. This makes the dequantized override use a different (and potentially unsupported/incorrect) wgrad computation than the normal backward, contrary to its documented purpose of dequantizing saved operands rather than changing quantization.",
  "start_line": 289,
  "end_line": 289,
  "suggestion": null,
  "side": "RIGHT"
}
Tool result · add_findingmessage 20
{"success": true, "finding_id": "f_d339f18f5c", "duplicate": false}
Agent → toolmessage 21
list_findings
{
  "status_filter": "open"
}
Tool result · list_findingsmessage 22
{"findings": [{"id": "f_d339f18f5c", "severity": "medium", "confidence": "high", "category": "correctness", "title": "Dequantized override loses columnwise input data", "file": "transformer_engine/pytorch/module/linear.py", "start_line": 289, "end_line": 289, "side": "RIGHT", "in_diff": true, "description": "Forcing `save_original_input = False` makes this forward enable columnwise quantization (`backward_needs_input and not save_original_input and backward_override is None` evaluates true only when `backward_override is None`, but the later discard logic selects the override branch). With `dequantized`, the saved quantized input is explicitly reduced to rowwise-only (`update_usage(rowwise_usage=True, columnwise_usage=False)`), so backward dequantizes the input and weight but no longer has the columnwise operand required by the FP8 wgrad path. This makes the dequantized override use a different (and potentially unsupported/incorrect) wgrad computation than the normal backward, contrary to its documented purpose of dequantizing saved operands rather than changing quantization.", "suggestion": null, "status": "open", "first_seen_sha": "941db9b0d5457c986f6b9a06bd960d6186c8a592", "last_confirmed_sha": "941db9b0d5457c986f6b9a06bd960d6186c8a592", "github_review_id": null, "github_review_run_id": null, "github_review_comment_ids": [], "github_review_thread_ids": [], "github_resolved_thread_ids": [], "github_posted_resolution_comment_ids": [], "surface_state": "not_surfaced", "last_human_reply_at": null, "last_human_reply_author": null, "last_human_reply_body": null, "last_reconciliation_note": null, "resolution_note": null, "diff_hunk": "@@ -285,6 +285,8 @@ def _linear_forward_impl(\n     is_fsdp2 = args.is_fsdp2\n     if backward_override == \"high_precision\":\n         save_original_input = True\n+    elif backward_override == \"dequantized\":\n+        save_original_input = False\n \n     # NVTX label for profiling\n     nvtx_label = \"transformer_engine._Linear.forward\"", "fingerprint": "v1:fd6298c9c693786fbb72294e06fad2b424becdb74f8ce0c843daf290f35f4034", "interactions": [], "rank": null}], "count": 1}
Agent → toolmessage 23
publish_review
{
  "ranking": [
    "f_d339f18f5c"
  ],
  "severity_threshold": "medium",
  "assessment": null
}
Tool result · publish_reviewmessage 24
{"success": true, "dry_run": true, "review_id": null, "surfaced_count": 1, "hidden_count": 0, "resolved_thread_count": 0}
Agentmessage 25
Simulated publish (eval mode): review not posted to GitHub.

- Medium: Dequantized override loses columnwise input data (transformer_engine/pytorch/module/linear.py:289): Forcing save_original_input = False under the dequantized override causes the saved quantized input to be reduced to rowwise-only. Backward then dequantizes the operands but no longer has the columnwise input data used by the normal FP8 wgrad path, making the override’s wgrad computation differ from the documented behavior of dequantizing saved operands.

On TransformerEngine #3141, the SFT model flags an undocumented option override but misses its functional consequence (F2 = 0). The RL checkpoint identifies the missing FP8 weight-gradient operand with 7 file reads versus 59 (F2 = 1).

Parallel tool calls and results use a consistent order. Long results are excerpted; message numbers follow the recorded traces.

Conclusion

In this article we explored what it takes to hill-climb for agentic coding review on real-world GitHub PRs. With only small slices of highly filtered data for LoRA-based SFT and RL, we nearly double the mean F2 score over our base model and drive a 12.6x improvement in token efficiency.

We did so in partnership with the stellar team behind Microsoft's new Foundry Programmed Training offering; its performant GPU-based training orchestration allowed us to focus on what's often the hardest part of any post-training project: data, evals, and experimental design.

If you want to learn more or have any questions, we'd love to chat.

Disclosure: Whitney AI is a Microsoft partner; this post was produced in collaboration with Microsoft.