Risk Scoring Agent Pull Requests by Architectural Impact
Score agent PRs by structural impact, not size or coverage.
Sponsored by Graphide

Architectural risk scoring is the code review alternative for large AI-generated diffs. It evaluates the boundaries, contracts, and structural invariants a change affects rather than using lines changed or test coverage as proxies for safety. Automated checks screen low-risk work, but changes with high architectural impact need a human to weigh them before they can merge.
Why line counts and test coverage cannot tell you how to review large AI-generated pull requests
“I gave up and just started hitting approve” is the endpoint described in the r/ExperiencedDevs account. It names a failure of review signals. Traditional review assumes that the author has put effort into writing code and has thought through the choices in the diff. An agent can produce locally coherent code without accounting for the system around it. Those signals therefore no longer show reviewers how much care the author took.
How agent volume has already broken the review queue
LinearB’s 2026 benchmark covers millions of PRs across thousands of teams and shows the magnitude of the mismatch. AI-assisted PRs exceed twice the line count of unassisted PRs at the 75th percentile. Agentic AI PRs also wait roughly five times longer for a first reviewer, yet review clears faster once it begins. The ability to generate code has grown faster than the ability to judge what should merge.
What agents actually break — boundaries, contracts, and structural invariants
The target of architectural scoring is structural damage that remains invisible to local checks. A shared utility can let a service access another module’s internals without making the ownership violation obvious in the diff. A migration can add a table successfully while violating the database’s read-only contract. Uncontrolled scaffolding presents a related problem, where line count multiplies several-fold and file count triples while each file remains plausible on its own. Size reveals the growth, but not whether the system needed the new structure.
Decision-layer corruption is harder to recognize because it need not change executable code. The fullsend-ai PR #6378 proposed an ADR for a new always-on skills mechanism. The review agent classified it as low risk because it was a “documentation-only PR” touching “no protected paths, no security-sensitive files.” Senior reviewers later found fundamental problems with the design, and the PR closed without merge after nine days of discussion. The linked issue documents downstream effects including wrong model selection, no inline comments across three runs, and “Looks good to me” verdicts.
The classification failed because an ADR is not merely explanatory prose. It records a commitment that constrains future implementation, so a wrong ADR can carry more downstream impact than a wrong README. Across these failures, architectural intent must become a first-class constraint.
The three signal dimensions an architecture-aware risk score must cover
A defensible score applies three lenses simultaneously, namely structural position, decision-layer impact, and boundary-crossing behavior. Structural position describes where changed code sits in the dependency graph. Shared utilities, base interfaces, and data access layers can affect callers outside the diff through fan-out. SmartCodeReview spells out the intended dependency direction in AGENTS.md as controller → service → repository → entity. The scorer evaluates drift against that repository-specific direction.
Decision-layer impact measures whether a change creates or revises an architectural commitment. ADRs, interface definitions, design documents, schema migrations, and agent instruction files can all constrain later implementation. The fullsend-ai issue #1077 proposes a medium baseline for ADR paths, with further elevation when an ADR introduces runtime behavior. That proposal remains an open issue. The paper “Architecture Without Architects” similarly treats prompt specifications as architectural artifacts, extending review beyond source code to the instructions that shape it.
Boundary-crossing behavior asks whether a change adds a dependency across a declared boundary of a module, service, or data-ownership. Detecting it requires comparing the changed import graph with the intended boundary model. Morgan Stanley’s CALM deployment gates enforce related rules by checking for approved patterns, schema validity, required placeholders, and controls for each relationship. Salesforce’s review system also checks domain-specific rules for security, quality, and architecture, with findings that point to the exact files and lines. These mechanisms show why all three dimensions matter together, since a path label cannot establish either decision impact or permitted relationships.
How to build the scoring system — architecture, inputs, and verdict gates
The first component is a repository contract that states the intended architecture. It declares constraints on package structure, dependency direction, API boundaries, transaction ownership, persistence conventions, and data-ownership rules. The contract versions with the code and remains readable by humans and agents. SmartCodeReview places AGENTS.md at the repository root to establish this shared reference.
The second component is review orchestration, which defines how the reviewer gathers evidence. SmartCodeReview places SKILL.md under .agents/skills/code-review/ and keeps REVIEW_CONTROLS.md and REPORT_FORMAT.md in its references directory. This separates the review procedure from its controls and reporting format. The procedure requires inspecting changed code in context and following callers or interfaces where necessary, without expanding into unrelated repository areas. If the repository cannot support a conclusion, the reviewer records UNVERIFIED.
The third component is scoring logic that combines path sensitivity with structural evidence. Security-sensitive paths such as auth, secrets, and payments are in Tier 1, while ADRs have a medium baseline and infrastructure configuration has a high baseline. Shared utilities, base interfaces, and data access layers have higher multipliers because they can affect more than the changed files. A cross-boundary dependency calls for escalation no matter how low the line count is. Where available, agent reasoning captured through Claude Code hooks, AGENTS.md instructions, or commit metadata becomes another input, helping explain why the change exists.
The fourth component is a verdict gate that converts findings into merge requirements. Low-risk explanatory documentation, tests, and configuration typos can pass without human sign-off if the other dimensions show no higher impact. Medium-risk changes require human reviewer assignment, with the agent leaving inline comments. High-risk changes require human review regardless of how clean the diff appears, and the agent cannot approve and merge them independently. Keeping these four components separate allows the contract, review procedure, scoring rules, and merge policy to be governed independently.
Where naive scoring systems fail and how to calibrate against those failure modes
Calibration starts with semantic category. An architectural commitment needs more than the low-risk treatment given to explanatory prose, even when both are in documentation paths. The fullsend-ai case supplies a concrete test for that distinction. Similar scrutiny applies to a shared utility whose small diff conceals broad dependency impact. Path tiers establish a baseline, but the other dimensions must be able to override it.
Independent judgment also cannot be inferred from a second agent pass. The model that produced a change may carry the same assumptions into its review, so agent findings serve as evidence. Medium- and high-risk work therefore still needs human sign-off, however confident the model is. An incomplete AGENTS.md creates another blind spot because a scorer cannot enforce an unstated boundary. The contract must remain a living document with its own review discipline.
A per-PR scorer also misses semantic merge conflicts unless it considers other open branches touching the same modules. Two agents can make changes that merge cleanly as text but conflict in meaning. More fundamentally, the strongest objection to post-generation scoring is that constraints should shape generation before code exists. “Architecture Without Architects” argues that prompt specifications belong in architectural review, while the notes identify proactive consequence guidance as still missing from existing reactive controls. A merge gate is more useful than discovering nonconformance after merge.
Enforcing the score in CI so architectural constraints cannot be bypassed at merge time

A score in a PR comment remains advisory. A failing CI check that blocks merge turns the same assessment into an enforceable constraint. This distinction matters when reviewers face more code than they can reconstruct and approval becomes a response to queue pressure. Morgan Stanley’s CALM gates demonstrate the operational pattern by rejecting nonconforming architecture before deployment. Missing image versions or network controls produce feedback in the PR pipeline.
In GitHub Actions, the workflow begins on PR open by comparing the architectural diff with the declared contract and evaluating all three signal dimensions. A score below medium passes without assigning a required reviewer. A medium score blocks merge and assigns a human reviewer, while the agent supplies file- and line-level comments. A high score blocks merge and requires named architect sign-off, supported by a structured report of boundary findings and remediation guidance. These verdicts reserve human attention for escalated decisions rather than making a human the first structural check.
What changes in how developers work when architectural risk scoring is in place
Architectural scoring changes the unit of review from lines to decisions. Senior engineers no longer need to spend the same attention on agent-generated boilerplate and changes that alter ownership or dependency direction. The automated layer does the structural analysis that can be repeated, while reviewers judge the constraints a PR challenges. Most work still needs human review because full task delegation is still rare. Judgment begins with evidence of impact rather than a reconstruction of everything the agent built.
Agent reasoning strengthens that evidence when it accompanies the code through commit metadata, impact statements, or structured PR descriptions. The “Architecture Without Architects” proposal illustrates this with the impact statement “Structured JSON output adds validator, retry handler, fallback; +330 LoC, two new failure modes”. Such a statement shows the coupling before generation, while CI checks the structure that comes out of it before merge. Neither substitutes for the other. Together they connect the proposed architectural consequences to the change that actually reaches the gate.
A living graph makes this shift sustainable by mapping modules, files, classes, functions, and call relationships into a structure developers can navigate. The reviewer can then look at the boundaries and decisions the change affects instead of searching a large diff for hidden structural changes. Agents will keep accelerating, and the gap between generation speed and review bandwidth will keep widening. The only review heuristic that scales is one that checks structural invariants automatically and reserves deliberate human judgment for consequential changes. A graph-as-source-of-truth model is therefore a necessary next step.
Current verification workflow
Large agent-written changes need both a CI verdict and review findings whose evidence a reviewer can inspect. Graphide CI runs existing GitHub Actions jobs on Graphide runners, using warm caches, and splitting long test steps only where a check shows every result stays the same, to reduce waiting while preserving the pass/fail verdict. A failing test fails the job. Graphide review is a separate product that presents changed behavior and findings with evidence; it does not block merging on its own. Consult the current documentation for supported workloads and limitations.
Product information updated November 5, 2026 to reflect Graphide’s current CI and review products.


