AI Code Quality Metrics: What to Measure and What to Ignore
Most teams track the wrong AI code metrics. Here are the 8 metrics that actually predict whether AI code generation helps or hurts your codebase long-term.
Most teams measuring AI code quality are tracking the wrong things. Lines of code generated per day. Acceptance rate. Time from prompt to commit. These metrics measure AI usage, not AI quality. Optimizing for them is like optimizing for how fast you can dig holes without checking if you're digging in the right place.
After setting up quality tracking for 8 engineering teams, I've identified the metrics that actually predict whether AI code generation helps or hurts your codebase long-term.
Lines of code generated. More code isn't better code. AI can generate 500 lines in seconds. That doesn't mean 500 lines was the right solution. Often, 50 lines of well-thought-out code would have been better.
Acceptance rate. A high acceptance rate means developers are accepting AI suggestions frequently. This could mean the AI is producing great code. It could also mean developers are vibe coding. You can't tell from the metric alone.
Prompt-to-commit time. Faster isn't better if the code needs to be rewritten in 3 months. This metric actively encourages skipping review.
Developer satisfaction with AI tools. Developers love tools that make them feel productive. But feeling productive and being productive aren't the same thing.
These metrics predict problems before they become expensive.[1]
1. AI Code Modification Rate
Track how often AI-generated code is modified within 30 days of being merged. This is the single most informative metric I've found.
typescript
// Pseudocode for tracking this metricinterface AICodeModificationRate { // How to calculate: // 1. Tag PRs as AI-assisted or human-written at merge time // 2. Track subsequent modifications to files from each category // 3. Calculate the percentage modified within 30 days aiAssistedFilesModified30d: number; // e.g., 34% humanWrittenFilesModified30d: number; // e.g., 18% ratio: number; // e.g., 1.89x // Target: ratio should be < 1.5x // If AI code is modified nearly 2x as often as human code, // your generation quality needs work}
Illustrative threshold: Use the 1.5x ratio as a starting point for investigation, then compare similar tasks and examine why code changed. Later edits can reflect new requirements as well as defects.
2. Defect Origin Rate
What percentage of production bugs originate from AI-generated code versus human-written code, normalized by lines of code? The following thresholds are illustrative. Record attribution uncertainty and compare with your own baseline.
Origin
Bugs per 1K Lines
Assessment
AI-generated
< 0.5
Excellent
AI-generated
0.5 - 1.5
Acceptable
AI-generated
> 1.5
Needs intervention
Human-written (baseline)
typically 0.5 - 1.0
Reference
3. Pattern Conformance Score
Run automated checks on AI-generated code to measure how well it matches your codebase conventions. I use a custom script that checks:
typescript
const patternChecks = { errorHandling: "Uses team's Result<T,E> pattern vs try/catch", imports: "Uses aliased imports (@/) vs relative paths", dataAccess: "Uses established data layer vs direct DB calls", naming: "Follows naming conventions from .eslintrc", fileStructure: "Follows colocation rules from ARCHITECTURE.md",};// Score each check as pass/fail// Target: > 80% conformance rate
These confirm trends you're already seeing in Tier 1 metrics.
4. Code Duplication Delta
Run jscpd or a similar tool monthly. Track how duplication changes over time, specifically in files that were AI-generated.[2]
bash
# Baseline measurementnpx jscpd src/ --min-lines 5 --reporters json --output reports/baseline# Monthly comparisonnpx jscpd src/ --min-lines 5 --reporters json --output reports/current# Compare reports/baseline/jscpd-report.json with reports/current/jscpd-report.json# Treat the 2% and 5% bands as proposed investigation thresholds.# State whether the change is a relative percent or percentage points.
5. Onboarding Velocity
How long does it take a new developer to make their first meaningful contribution? If this is increasing, AI code might be making your codebase harder to understand. This is a lagging indicator but a powerful one.
Track how long AI-assisted PRs take to get through code review compared to human-written PRs. If AI PRs take longer to review, reviewers are finding issues that need discussion.
Figure 2. Include the work after generation when estimating the benefit.
7. Architecture Integrity Index
Use a tool like dependency-cruiser to measure how well your codebase's dependency graph matches your intended architecture. AI code tends to create unexpected dependencies.[3]
bash
npx depcruise --config .dependency-cruiser.js src/ --output-type err# Track violation count over time
8. Net Time Benefit of AI Code
Calculate: (Time saved generating) minus (Time spent reviewing + Time spent fixing + Time spent maintaining). A negative value means the measured workflow used more time than the estimated manual alternative.
# The Contrarian Take: Some Metrics Should Be Ignored on Purpose
I deliberately don't measure code coverage for AI-generated tests. Here's why: AI can generate tests that achieve 95% code coverage and catch zero meaningful bugs. Coverage becomes a vanity metric when AI is generating both the code and the tests. It tells you how much code was executed during testing, not how much behavior was verified.[4]
Instead, I measure mutation testing scores on AI-generated code. Mutation testing modifies your code slightly (e.g., changing > to >=) and checks if any test fails. If no test fails, the test suite has a gap. Compare mutation results with coverage to find executed code whose behaviour the tests do not protect.[5]
bash
# Using Stryker for JavaScript/TypeScript mutation testingnpx stryker run --mutate "src/ai-generated/**/*.ts"# Target mutation score: > 60%
Look for correlations. Which types of AI-generated code have the highest modification rate? Which developers produce the most conformant AI code? Which prompt patterns produce the best metrics?
Now you have enough data to make informed decisions. Maybe AI-generated API routes are fine but AI-generated data migrations are problematic. Adjust your AI usage policy based on actual metrics, not gut feeling.
You can't improve what you don't measure. But measuring the wrong things is worse than measuring nothing, because it gives you false confidence. Focus on the metrics that predict long-term code health, not short-term productivity. The investment pays off within one quarter.
Frequently asked questions
Is AI suggestion acceptance rate a quality measure?
Acceptance shows that a suggestion was used. Review the resulting behaviour, rework and defects to judge whether it helped, because frequent acceptance alone does not show that code fits the system or its requirements.
Should every team use the same quality thresholds?
Treat the thresholds in the examples as starting points for investigation. Establish a baseline for comparable tasks and code, then examine the causes of a change before turning a number into a team target.
How is mutation testing different from code coverage?
Coverage shows which code tests execute. Mutation testing changes the code and checks whether tests detect the change, giving another view of the assertions and behaviours the suite can actually protect.
How should I compare AI-assisted and other changes?
Record how you label the changes and compare similar work with similar risk. Review time, later edits and defects can provide useful signals, but file-level labels and lines of code cannot precisely isolate authorship.
Five predictions for where code intelligence is heading by 2028, based on 6 years of building tools in this space. Includes what's real, what's overhyped, and a maturity model for planning.
Most code quality dashboards fail because they track too many metrics and live outside the team's workflow. Here are exactly 7 metrics and a 3-week adoption plan that actually works.
A 3-tier succession planning framework for engineering orgs, with a 90-day sprint for accelerating candidates from potential to ready. Based on succession plans built at 4 companies.
The CTO's real job in product development isn't attending every meeting or making every technical call. It's connecting feasibility, viability, and capacity while letting EMs handle the day-to-day.
Four years of data from a fully remote engineering team across 4 time zones. Specific metrics on productivity, retention, and onboarding with the frameworks that actually worked.
Small frictions cost more than big blockers. 47 micro-frictions consume 96 minutes per engineer per day. Here's the TAX Return framework for finding and eliminating the developer productivity tax.