The Real Software Factory Problem Is the Code Review Bottleneck
A spec-to-production AI factory is easy to build. Deciding which pull requests need a human is the hard part. What I learned building one, from code freezes to Meta's Diff Risk Score and RADAR.
Kingsley Ijomah
AI Adoption Lead
Updated

Say "software factory" and most people picture the lights going off. A spec goes in one end, AI agents do the work, software comes out the other end, and no engineer touches it on the way.
I've been building one, and the first thing I learned is that the version in that picture is the easy part. You can build a spec-to-production factory in an afternoon. Point a coding agent at your main branch, let it finish with git push, and your CI/CD pipeline deploys whatever it pushed. Done.
Nobody sensible runs it that way. The reason they don't is the actual subject of this post, and it turns out to be a problem we had long before AI.
Why nobody lets the agent push to main
If the agent pushes straight to main, every change it makes goes live, including the ones that break something. The pipeline will happily deploy a bad change as fast as a good one. You would be shipping code that creates more problems than it solves, at a speed no team could clean up after.
So we put a pull request in the way, and a person reads it before it merges. That works until you look at the numbers on the other side of the pull request.
AI made code fast and cheap, but cheap is an illusion
There is an old rule in project management called the iron triangle: quality, time and cost. You can usually pick two. Fast and cheap means quality pays.
AI looks as if it has broken the rule. Code arrives faster than ever, and the cost per line has collapsed. But the triangle hasn't gone anywhere. When quality drops, the cost comes back later as reverts, incidents, rework and engineers pulled off planned work to fix production. The code was never cheap. The bill just arrived after the merge.
The only thing standing between fast code and that bill is somebody checking it, which is the same discipline I wrote about in how to check AI output before you trust it. In software, that check is code review.
The real constraint is the code review bottleneck
Meta published numbers in 2026 that show the problem plainly. Over a year, lines of code per human-landed diff rose by 105.9%, and diffs per developer per month rose by 51%. Meta attributes more than 80% of that increase to agentic AI. The number of people who can review that code did not double.
A note on vocabulary, because it confused me at first. Meta says "diff" where GitHub says "pull request". It means the same thing: the lines you've added, removed or changed, put up for review. When you read "diff risk score", read "pull request risk score".
This is the code review bottleneck, and it is the real problem a software factory has to solve. Writing code is no longer the slow step. Deciding whether code is safe to merge is.
Two bad ways out are available. You can rubber-stamp reviews and let poor code reach main, or you can slow the AI down to the speed of your reviewers and give up what you adopted it for. The third way is to triage: spend human attention on the changes that need it, and let the rest through on evidence.
How we used to keep bad changes out
Triage isn't new. Teams have always had ways to control the risk of a change, even when review capacity was the only constraint.
- Before merge: code review, CODEOWNERS so named experts must approve changes to their areas, and required CI checks such as tests, linters and security scans.
- Timing: code freezes, release trains, maintenance windows, and a rule from teams I've worked in: we don't push anything live on a Friday, because nobody wants to fix production over the weekend.
- During deploy: feature flags, canary releases and staged rollouts.
- After deploy: monitoring, alerts, automatic rollback and postmortems.
The no-Friday rule turns out to be a formal practice at Meta. Their 2024 paper on release risk lists "weekend gating" as one of four gating levels, because "on the weekend there are fewer engineers available to fix SEVs". They also freeze code "during certain periods of the year, such as holidays or major events", with Black Friday as the example.
Freezes are the bluntest tool on that list. A freeze manages risk by stopping change. Meta's earlier freezes were also scoped by path: "experts select a set of paths and freeze diffs that touch these paths." That is exactly the instinct behind protecting the parts of a codebase that hold the core business logic, where one wrong change is a serious outage.
The safest codebase is one nobody changes. That's also a business that has stopped growing. Freezing everything was always a trade, and Meta's own write-up admits the cost: "a substantial reduction in productivity". AI makes the trade worse, because the pile of waiting changes now grows faster.
From blanket freezes to a risk score for every pull request
Meta's answer came in three steps, and the order is the useful part.
First, blanket freezes. Stop most changes during sensitive periods and accept the lost productivity.
Second, the Diff Risk Score (DRS). A model that predicts how likely a diff is to cause a production incident. During a freeze, low-risk diffs could still land. Meta says it "landed 10,000+ code changes (that previously could not have landed during a freeze) with minimal production impact" during one event, and that DRS "has helped us eliminate major code freezes". The freeze stopped being a wall and became a filter.
Third, RADAR, which stands for Risk Aware Diff Auto Review. This is the step that matters for a software factory. RADAR passes each diff through a funnel: eligibility checks, static heuristics, a DRS threshold, an LLM review agent and deterministic validation. If a diff passes every stage, it lands with no human review. If it fails any stage, it goes to a person.
The thresholds are strict and tuned by who wrote the change. For human-written diffs the default is P5, meaning only the lowest-risk 5% qualify. AI and bot diffs from trusted, allowlisted sources get P50, and other bots get P20. The LLM reviewer needs a confidence of at least 8 out of 10, with every change classified as safe, and approved bot diffs wait for a configurable delay during which a human can still reject them.
The results are large. RADAR has reviewed more than 535,000 diffs and landed more than 331,000. Meta reports a revert rate one third of non-RADAR diffs and a production incident rate one fiftieth. Read those with the authors' own caveat: the comparisons are observational, and RADAR only sees diffs that were already judged low risk, so it is not a like-for-like trial.
Why lines changed is the wrong measure of risk
The obvious way to score a pull request is by size. Small change, small risk. It's wrong, and Meta has a clean example of why.
In their paper on configuration risk, a SEV1, Meta's highest severity, was triggered by "a small change to a feature-flag config path consumed by a high-criticality service". Because it modified one artifact and showed no code complexity, the code-focused DRS scored it as low risk. One line, in the wrong place, took down something important. Meta built a separate model, CORA, for config changes because of it.
Risk comes from what a change touches and what depends on it, not from how many lines it has. And this is where a general-purpose model falls short. The LLM writing your code is good at code. It does not know which service pays the bills, which file broke production last spring, or which flag three teams depend on. A general LLM doesn't know your incident history. Meta's DRS works because it was trained on Meta's own diffs and incidents.
That raises a fair question for anyone without Meta's history. There is an open-source attempt, DRS-OSS, from researchers at Concordia and the University of Tennessee: a Llama 3.1 8B model fine-tuned on the public ApacheJIT dataset, with a GitHub App that posts a risk label on pull requests. On ApacheJIT's own test data it reports that gating the riskiest 30% of commits would catch 86.4% of the changes that introduced defects. Its precision is 0.491, so about half of what it flags is a false alarm, and the paper only evaluates it on that one dataset. It's a reasonable starting score, not a substitute for knowing your own system.
What I've built so far, and what I removed
My factory takes a Jira ticket, has agents write a spec, build the change, write acceptance tests and verify the result, then opens a pull request and merges and deploys when the gate allows it.
My first risk engine was path-based. Every change got a tier from R0 to R3 based on the files it touched: docs and tests were R0, auth, payments and infrastructure were R3. The planner estimated a tier from the spec, the real diff confirmed it, and the higher of the two won. R0 merged on its own; R2 and R3 always waited for a human.
I took it out. Path rules go stale as a codebase grows, and they can't capture risk on their own. They are the same instinct as the old path-scoped freeze, and they fail the same way: they only know what somebody remembered to write down. The one-line feature flag would have sailed through.
What runs today is simpler and more honest about what it knows. A person chooses the mode for each ticket with a Jira label. A light ticket stops for a human to approve the spec and the pull request. A dark ticket has no human approvals and merges once verification passes.
Dark doesn't mean unchecked. Dark skips the human approvals, not the checks. Acceptance tests, locked specs and tests, limits on which paths the agent may write to, an agent reviewer, deploy smoke tests and automatic revert all still run, and any failure blocks the ticket. Small dark tickets have gone from label to production with nobody touching them.
The weak point is obvious. A person decides which tickets are safe enough to go dark, based on the ticket rather than the change, and anyone who can add a label can make a ticket fully automatic. That is the decision a risk engine should make.
The design I'm building next
This part isn't built yet. It's the design I'm working towards, taken from what Meta has published and what my own first attempt taught me. A pull request goes dark only if it passes every stage, and anything that fails goes to a person:
- Hard floors. Critical paths, configuration, migrations and authentication always go to a human, whatever any score says. Rules set the floor.
- Verification. Tests and checks must pass. This already exists.
- A risk score. A model such as DRS-OSS as the starting point, plus cheap signals from history like how often a file changes and how often it has needed fixing.
- An LLM reviewer with fixed categories. Like RADAR's, it classifies each change as safe or risky, and any risk signal sends the pull request to a human.
- A way to learn. A short delay before merging during which a human can still stop it, and calibration from my own reverts and incidents so the score slowly learns the history a general model can't know.
There are open problems I don't have answers to yet. A small team has very little incident history to train on. Config changes need their own scoring, as CORA shows. And a pull request written by an agent can contain text designed to talk the reviewer into approving it, so the reviewer has to treat the diff as untrusted input.
The rule I'm working to
The software factory everybody pictures is a demo. The real one is a decision about where to spend the scarcest thing in your team: the attention of the people who can tell a safe change from a dangerous one.
Freezes protected that attention by stopping change. AI makes stopping too expensive. So the rule I'm building to is this: don't stop changes; decide which ones need a human, and make that decision on what the change touches, not how big it is.
Sources
Linked so you can check the evidence and its limits.
- Adams, Banga, Mockus, Rigby, Nagappan et al. (Meta, 2026): Automating Low-Risk Code Review at Meta: RADAR, Risk Calibration, and Review Efficiency. Source of the code growth figures (+105.9% lines per human-landed diff, +51% diffs per developer, 80%+ attributed to agentic AI), the RADAR funnel, the P5/P20/P50 thresholds, the 8/10 reviewer confidence, the cancel delay, and the 535K+ reviewed and 331K+ landed diffs. The revert (1/3) and incident (1/50) comparisons are observational and compare diffs already judged low risk with all others. arXiv preprint.
- Engineering at Meta (2025): Diff Risk Score: AI-driven risk-aware software development. Describes DRS as a model predicting incident-causing diffs, the 10,000+ changes landed during a freeze, the claim that DRS "has helped us eliminate major code freezes", and the productivity cost of freezes. A company blog with no accuracy figures.
- Abreu, Murali, Rigby, Maddila, Mockus, Nagappan et al. (Meta, 2024): Moving Faster and Reducing Risk: Using LLMs in Release Deployment. Source of the four gating levels including weekend gating, the reasons for weekend and holiday freezes, and the description of earlier path-scoped freezes. Shows fine-tuned models beating Meta's earlier logistic regression at catching incident-causing diffs; single-company results.
- Sun, Mockus, Nagappan et al. (Meta, ESEM 2026): CORA: Config Risk Analyzer. Source of the SEV1 caused by a small feature-flag change that the code-focused DRS scored as low risk, and the case for a separate model for config changes. Results are from Meta only.
- Sayedsalehi, Rigby and Mockus (2026): DRS-OSS: Practical Diff Risk Scoring with LLMs. An open-source Llama 3.1 8B risk model fine-tuned on ApacheJIT: precision 0.491, recall 0.84, ROC-AUC 0.895, and the riskiest 30% of commits catching 86.4% of defect-inducing changes. Evaluated only on the ApacheJIT test split; it predicts defects, not production incidents.
- DRS-OSS code repository (MIT licence). The implementation behind the paper, including the GitHub App; each model API needs a GPU.
Share this article
Pass it on to someone who might find it useful.
From theory to practice
See how the ideas become working systems.
I’m working on hands-on examples and videos that build real agentic workflows step by step. Join the list for new articles, practical material, and the first course updates when they’re ready.
Discussion
What did this make you think about?
Share what you have seen in practice, ask a question, or add a different perspective.
