Skip to content

AI Code Quality Metrics: What to Measure and What to Ignore

Most teams track the wrong AI code metrics. Here are the 8 metrics that actually predict whether AI code generation helps or hurts your codebase long-term.

8 min read
Illustrative precision mechanical balance on a charcoal tabletop comparing two differently shaped components against a single unmarked teal reference block.
On this page

Most teams measuring AI code quality are tracking the wrong things. Lines of code generated per day. Acceptance rate. Time from prompt to commit. These metrics measure AI usage, not AI quality. Optimizing for them is like optimizing for how fast you can dig holes without checking if you're digging in the right place.

After setting up quality tracking for 8 engineering teams, I've identified the metrics that actually predict whether AI code generation helps or hurts your codebase long-term.

The Metrics That Don't Matter

Let me save you some time. Stop tracking these:

Lines of code generated. More code isn't better code. AI can generate 500 lines in seconds. That doesn't mean 500 lines was the right solution. Often, 50 lines of well-thought-out code would have been better.

Acceptance rate. A high acceptance rate means developers are accepting AI suggestions frequently. This could mean the AI is producing great code. It could also mean developers are vibe coding. You can't tell from the metric alone.

Prompt-to-commit time. Faster isn't better if the code needs to be rewritten in 3 months. This metric actively encourages skipping review.

Developer satisfaction with AI tools. Developers love tools that make them feel productive. But feeling productive and being productive aren't the same thing.

The Metrics That Actually Matter

Weekly Leading indicators Monthly Lagging indicators Quarterly Strategic indicators
Figure 1. Use a review cadence that lets the team investigate changes in quality.

Tier 1: Leading Indicators (Measure Weekly)

These metrics predict problems before they become expensive.[1]

1. AI Code Modification Rate

Track how often AI-generated code is modified within 30 days of being merged. This is the single most informative metric I've found.

typescript
// Pseudocode for tracking this metric
interface AICodeModificationRate {
  // How to calculate:
  // 1. Tag PRs as AI-assisted or human-written at merge time
  // 2. Track subsequent modifications to files from each category
  // 3. Calculate the percentage modified within 30 days

  aiAssistedFilesModified30d: number;    // e.g., 34%
  humanWrittenFilesModified30d: number;  // e.g., 18%
  ratio: number;                          // e.g., 1.89x

  // Target: ratio should be < 1.5x
  // If AI code is modified nearly 2x as often as human code,
  // your generation quality needs work
}

Illustrative threshold: Use the 1.5x ratio as a starting point for investigation, then compare similar tasks and examine why code changed. Later edits can reflect new requirements as well as defects.

2. Defect Origin Rate

What percentage of production bugs originate from AI-generated code versus human-written code, normalized by lines of code? The following thresholds are illustrative. Record attribution uncertainty and compare with your own baseline.

Origin Bugs per 1K Lines Assessment
AI-generated < 0.5 Excellent
AI-generated 0.5 - 1.5 Acceptable
AI-generated > 1.5 Needs intervention
Human-written (baseline) typically 0.5 - 1.0 Reference

3. Pattern Conformance Score

Run automated checks on AI-generated code to measure how well it matches your codebase conventions. I use a custom script that checks:

typescript
const patternChecks = {
  errorHandling: "Uses team's Result<T,E> pattern vs try/catch",
  imports: "Uses aliased imports (@/) vs relative paths",
  dataAccess: "Uses established data layer vs direct DB calls",
  naming: "Follows naming conventions from .eslintrc",
  fileStructure: "Follows colocation rules from ARCHITECTURE.md",
};

// Score each check as pass/fail
// Target: > 80% conformance rate

Tier 2: Lagging Indicators (Measure Monthly)

These confirm trends you're already seeing in Tier 1 metrics.

4. Code Duplication Delta

Run jscpd or a similar tool monthly. Track how duplication changes over time, specifically in files that were AI-generated.[2]

bash
# Baseline measurement
npx jscpd src/ --min-lines 5 --reporters json --output reports/baseline

# Monthly comparison
npx jscpd src/ --min-lines 5 --reporters json --output reports/current

# Compare reports/baseline/jscpd-report.json with reports/current/jscpd-report.json
# Treat the 2% and 5% bands as proposed investigation thresholds.
# State whether the change is a relative percent or percentage points.

5. Onboarding Velocity

How long does it take a new developer to make their first meaningful contribution? If this is increasing, AI code might be making your codebase harder to understand. This is a lagging indicator but a powerful one.

6. Review Cycle Time for AI PRs

Track how long AI-assisted PRs take to get through code review compared to human-written PRs. If AI PRs take longer to review, reviewers are finding issues that need discussion.

Tier 3: Strategic Indicators (Measure Quarterly)

Time saved Generation Time spent Review, fixes, maintenance Net time benefit Saved minus spent
Figure 2. Include the work after generation when estimating the benefit.

7. Architecture Integrity Index

Use a tool like dependency-cruiser to measure how well your codebase's dependency graph matches your intended architecture. AI code tends to create unexpected dependencies.[3]

bash
npx depcruise --config .dependency-cruiser.js src/ --output-type err
# Track violation count over time

8. Net Time Benefit of AI Code

Calculate: (Time saved generating) minus (Time spent reviewing + Time spent fixing + Time spent maintaining). A negative value means the measured workflow used more time than the estimated manual alternative.

The AI Quality Dashboard

Here's the dashboard I set up for my teams. Four panels, updated weekly:

+----------------------------+---------------------------+
|  AI Modification Rate      |  Defect Origin Rate       |
|  Target: < 1.5x human     |  Target: < 1.0 per 1K LoC |
|  Current: [chart]          |  Current: [chart]         |
+----------------------------+---------------------------+
|  Pattern Conformance       |  Duplication Delta        |
|  Target: > 80%             |  Target: < 2% monthly    |
|  Current: [chart]          |  Current: [chart]         |
+----------------------------+---------------------------+

The Contrarian Take: Some Metrics Should Be Ignored on Purpose

I deliberately don't measure code coverage for AI-generated tests. Here's why: AI can generate tests that achieve 95% code coverage and catch zero meaningful bugs. Coverage becomes a vanity metric when AI is generating both the code and the tests. It tells you how much code was executed during testing, not how much behavior was verified.[4]

Instead, I measure mutation testing scores on AI-generated code. Mutation testing modifies your code slightly (e.g., changing > to >=) and checks if any test fails. If no test fails, the test suite has a gap. Compare mutation results with coverage to find executed code whose behaviour the tests do not protect.[5]

bash
# Using Stryker for JavaScript/TypeScript mutation testing
npx stryker run --mutate "src/ai-generated/**/*.ts"
# Target mutation score: > 60%

Setting Up Your Measurement System

Week 1: Establish Baselines

Tag all PRs as AI-assisted or human-written for one week. Measure your current duplication ratio, defect rate, and pattern conformance.

Week 2-4: Track Leading Indicators

Set up automated collection of Tier 1 metrics. Present them in your weekly standup. Don't make changes yet; just observe.

Month 2: Identify Patterns

Look for correlations. Which types of AI-generated code have the highest modification rate? Which developers produce the most conformant AI code? Which prompt patterns produce the best metrics?

Month 3: Act on Data

Now you have enough data to make informed decisions. Maybe AI-generated API routes are fine but AI-generated data migrations are problematic. Adjust your AI usage policy based on actual metrics, not gut feeling.

The Bottom Line

You can't improve what you don't measure. But measuring the wrong things is worse than measuring nothing, because it gives you false confidence. Focus on the metrics that predict long-term code health, not short-term productivity. The investment pays off within one quarter.

Frequently asked questions

Is AI suggestion acceptance rate a quality measure?

Acceptance shows that a suggestion was used. Review the resulting behaviour, rework and defects to judge whether it helped, because frequent acceptance alone does not show that code fits the system or its requirements.

Should every team use the same quality thresholds?

Treat the thresholds in the examples as starting points for investigation. Establish a baseline for comparable tasks and code, then examine the causes of a change before turning a number into a team target.

How is mutation testing different from code coverage?

Coverage shows which code tests execute. Mutation testing changes the code and checks whether tests detect the change, giving another view of the assertions and behaviours the suite can actually protect.

How should I compare AI-assisted and other changes?

Record how you label the changes and compare similar work with similar risk. Review time, later edits and defects can provide useful signals, but file-level labels and lines of code cannot precisely isolate authorship.

Sources

  1. [1]DORA’s software delivery performance metrics · DORA
  2. [2]GitHub - kucherenko/jscpd · jscpd maintainers
  3. [3]GitHub - sverweij/dependency-cruiser: Validate and visualize dependencies. Your rules. · Dependency Cruiser maintainers
  4. [4]Test Coverage · Martin Fowler
  5. [5]Configuration · Stryker Mutator

Checked against these sources on

Related Articles