Scaling Code Review for the AI Era: Intent Over Line-by-Line Reading

When AI writes most of the code, does a human reviewer still need to read every line? The answer, according to engineers at Aviator, is no. Instead of scrutinizing diffs, they now verify five dimensions: the reasoning behind a change, its acceptance criteria, the invariants it cannot violate, the decisions made during development, and concrete proof that it works.

This methodological shift produced measurable gains. Nine engineers on the Aviator team began merging four times as many pull requests. The median time from submission to merge dropped from 2.1 days to 3.4 hours. Reverts fell by 40 percent. A human still approves every PR, but what that person examines has fundamentally changed.

Decisions need a person, and a diff is the worst place to make them.

The insight emerged from analyzing review comments across the team's pull requests. Most remarks fell into two categories: enforcement of rules the reviewer had articulated before, or judgment calls about design. Automated systems should handle rules consistently. Decisions, by contrast, require human judgment—and a code diff is a poor venue for making them.

Why traditional review breaks under AI volume

The team's workflow shifted from human-authored code with AI assistance to AI-generated code with human oversight. Weekly merged pull requests climbed from roughly 100 to more than 400. The median AI-generated PR reached 640 lines, far exceeding the 400-line threshold beyond which defect detection typically declines.

Yet catching bugs was never the primary function of code review. A 2013 Microsoft study by Bacchelli and Bird found that only 14 percent of review comments addressed defects. Review serves a deeper purpose: it distributes knowledge across the team and keeps engineers aligned on how the system works. When AI generates most code, this knowledge-sharing function becomes both more critical and more at risk.

Catching bugs was never the main point of code review anyway. Review is how engineers share knowledge and keep their mental model of the system current, and that's the part worth protecting.

What failed to scale

• An AI reviewer. On one engineer's PR, the bot opened 16 of 18 comment threads, flagging magic numbers, array-index React keys, and overly long components. One thread caught a genuine race condition. The two comments that mattered—design decisions from teammates—drowned in a sea of nitpicks.
• A four-hour review service level agreement. Unable to add reviewers, the team imposed a time limit instead. Initial reviews accelerated, but approvals became perfunctory as reviewers skimmed to meet the deadline while their own work languished.
• A self-review tool tuned to team conventions. Its output surpassed the generic bot's quality. Yet it still analyzed the diff after code was written, and the most valuable part of the exchange—what the tool questioned and what the team decided—evaporated in a chat log.

All three approaches attempted to read the diff more effectively. But the diff itself is the least informative part of code review. It documents what changed, never why.

Examining a PR reviewed in 2025 revealed the pattern. Fourteen threads appeared on the PR. Eight came from a bot and identified one genuine bug. Three of six human comments restated rules the reviewer had already written elsewhere, such as "Use GraphQLRetryableException so Celery retries on it." Only two represented actual decisions, like "No comments extracted: should we tell the user on the PR?"

A new approach: Verify intent, not lines

Code review historically bundled multiple responsibilities: bug detection, convention enforcement, design arbitration, and system comprehension. The team disaggregated these functions and assigned each to a more suitable mechanism.

Argue before the PR exists

A review agent cross-examines the author's agent, running on a different model, and documents every disagreement. On a recent PR, the agents surfaced four disagreements and resolved three with explicit reasoning. Only one—a naming decision—required human judgment.

Capture intent and acceptance criteria

Rather than writing specifications upfront, the team extracts intent and criteria from the coding session itself. Each time an engineer corrects the agent or answers a question, a criterion is created. Previously, this information disappeared when the session ended. Intent explains why the change exists; acceptance criteria specify how it should behave.

Intent answers why we're making the change; acceptance criteria answer how it should behave. Intent without criteria can't be checked, and criteria without intent can't be judged.

Codify repeated comments as invariants

The team mined one year of review history in three passes: eliminate noise (bot summaries, bare approvals), classify each comment as a rule, proposal, isolated fix, or question, then consolidate duplicates. 11,400 comments condensed into 212 candidate rules and 64 approved invariants. Each invariant specifies scope, verification method, source, and ownership.

Consider the invariant: "Every config change writes an audit-log entry, only when something changed." It applies to GraphQL mutations and API handlers that modify config. Verification runs a preview environment, saves twice, and checks the audit log. The rule originated from 23 review threads across 17 PRs and belongs to the platform team.

Model-based checking of every invariant on every PR is slow and imperfectly reproducible. Where feasible, agents compile invariants into ast-grep rules that run deterministically alongside the model until they converge. A comment that appeared on nine PRs—"Add a timeout, this hung a worker"—became an eight-line rule. Nineteen invariants now run deterministically in 0.3 seconds.

Prove every criterion with evidence

Each criterion becomes a scenario. An agent provisions a preview environment, interacts with it as a user would, and records screenshots, requests, and logs. For the audit-log invariant, it saves a setting, confirms one new log entry, saves again with no changes, and confirms the entry count remains one.

That second verification step is precisely what a reviewer examining the diff would overlook. Not every criterion requires a browser. An API contract is proven by a recorded request and response; a code rule by an ast-grep match or file-and-line reference. Select the least expensive proof that resolves the question. A passing CI check only confirms the tests the team anticipated.

Debate the decisions

Decisions originate from three sources: the coding session, the agents' disagreement, and failed verification runs. The PR lists each decision, its origin, and who resolved it. On a PR adding retries for failed webhook deliveries, four of five decisions were settled by the agents. The agents disagreed on whether to deduplicate retried deliveries, so a human made that call.

Reviewers quickly scan settled decisions, resolve what the agents could not, reopen anything they would handle differently, and discuss with the author. This conversation deepens understanding of the system.

Own the rules

Engineers stop reading every line. They own the rules that check every line and maintain system understanding by reading intent, evidence, and decisions.

Measured outcomes

• Pull requests merged per week: 4× increase with the same nine engineers
• Median time to merge: 2.1 days reduced to 3.4 hours
• Human comment threads per PR: 6.2 reduced to 1.3
• Reverts per 100 PRs: 40 percent decline

Review work did not vanish. It shifted toward refining invariants, debating decisions, and strengthening verification. The transition carries an initial cost: the first month involves double work, by month two invariants mature enough to shrink reviews, and by month three reviewers focus on intent and evidence rather than code.