AI tests hit 91% coverage but only 34% mutation score. Here's proof that AI-generated tests create dangerous false confidence, and the workflow to fix it.
I had a PR with 94% code coverage. Every test passed. The CI pipeline was green across the board. We shipped on Friday afternoon. By Saturday morning, production was down because a null pointer exception hit a path that none of our tests covered meaningfully.
The tests were all AI-generated. They looked thorough. They covered every function. And they were almost completely useless.
This post explains why AI-generated tests create a dangerous illusion of safety, and how to fix it.
Generated tests can repeat assumptions in the implementation when the prompt supplies no independent requirements. Define the required behaviour first, then review whether each assertion could detect an incorrect implementation.[1]
Here's the difference in practice:
typescript
// The implementationfunction calculateDiscount(order: Order): number { if (order.total > 100) return order.total * 0.1; if (order.items.length > 5) return order.total * 0.05; return 0;}// AI-generated test (mirrors the implementation)describe("calculateDiscount", () => { it("returns 10% for orders over $100", () => { const order = { total: 150, items: [{ id: "1" }] }; expect(calculateDiscount(order)).toBe(15); }); it("returns 5% for orders with more than 5 items", () => { const order = { total: 50, items: Array(6).fill({ id: "1" }) }; expect(calculateDiscount(order)).toBe(2.5); }); it("returns 0 for small orders", () => { const order = { total: 20, items: [{ id: "1" }] }; expect(calculateDiscount(order)).toBe(0); });});
Looks great, right? 100% coverage. All green. But this test suite misses:
What happens when total is exactly 100? (boundary condition)
What happens when total is negative? (invalid input)
What happens when items is an empty array? (edge case)
What happens when an order qualifies for both discounts? The implementation only gives one. Is that correct? (business logic question)
What happens when total is 101 and items.length is 6? (priority question)
The AI didn't test any of these because the AI doesn't understand the business rules. It only understands the code.
AI tests almost always test the success case. Error paths, timeouts, and failure modes are either missing or trivially tested.
typescript
// AI only tests the success casetest("fetches user successfully", async () => { const user = await getUser("valid-id"); expect(user).toBeDefined(); expect(user.name).toBe("Test User");});// Missing tests:// - What happens with an invalid ID format?// - What happens when the API returns 500?// - What happens when the network times out?// - What happens when the response is malformed JSON?// - What happens under rate limiting?
AI generates snapshot tests for complex outputs. These tests break every time the output format changes, even if the behavior is correct. They become noise that teams eventually stop maintaining.
// The most useless assertion AI generatestest("should work", async () => { const result = await complexOperation(input); expect(result).toBeDefined(); // Fails for undefined; accepts other values. expect(result).not.toBeNull(); // Also passes for wrong values});
A defined, non-null value can still be the wrong result. Use assertions that check the required value or behaviour, and include cases where the implementation should reject an input or operation.[2]
To prove that AI tests provide false confidence, I ran mutation testing on 3 projects.
Mutation testing changes your code slightly (flipping operators, removing lines) and checks if tests catch the change.[3]
Project
Code Coverage
Mutation Score
Gap (percentage points)
Project A (AI tests)
91%
34%
57
Project B (AI tests)
87%
41%
46
Project C (human tests)
76%
68%
8
Project C had lower code coverage but a much higher mutation score. Its tests actually caught bugs. Projects A and B had high coverage but their tests let most mutations survive. Those tests give you a green dashboard while bugs slip through.
describe("OrderProcessor", () => { // Happy paths it("should create an order with valid items and payment"); it("should apply volume discount for orders over 10 items"); // Edge cases it("should reject orders with zero items"); it("should reject orders with negative prices"); it("should handle exactly 10 items without discount"); // Failure modes it("should roll back order if payment fails"); it("should retry payment once on timeout"); it("should not send confirmation email on payment failure"); // Concurrency it("should prevent double-charging with idempotency key");});
# Step 2: Let AI Implement Test Bodies (AI-Assisted)
Give the AI your test descriptions and ask it to implement them. This constrains the AI to test what matters, not what's easy.
# Step 3: Verify with Mutation Testing (Automated)
Install the StrykerJS core package and the plugin for your test runner, then use npx stryker init to configure the project before running mutation tests.[4]
bash
npx stryker run --mutate "src/order-processor.ts"# Target: > 60% mutation score
Set thresholds in the Stryker configuration used by your CI pipeline. The following values are proposed team settings. Configure the installed runner and verify both a failing and a passing score before making the job required.[5]
javascript
// stryker.config.mjsexport default { thresholds: { high: 70, low: 50, break: 40 }, // Include the runner and other settings created for your project.};
yaml
# .github/workflows/test-quality.yml- name: Mutation Testing run: npx stryker run
Code coverage is a necessary but insufficient metric. When AI generates your tests, coverage becomes almost meaningless. Mutation testing is the metric that tells you whether your tests actually protect against bugs.
The rule is simple: humans decide what to test, AI helps with how to test it. Reverse that order, and you get a green dashboard that means nothing.
Frequently asked questions
Does high coverage prove that a test suite detects defects?
Coverage shows which code was exercised under the selected measure. Review the assertions and expected behaviour as well, then investigate surviving mutations to see where a changed implementation still passes the tests.
Are mocks always a sign of a weak test?
Mocks can isolate dependencies and make failures repeatable. Check whether the assertions verify the required interaction and outcome, and whether separate tests exercise the real integrations that the unit test replaces.
What should a team do with a surviving mutant?
Inspect the changed behaviour and decide whether it matters to the contract. Add a targeted assertion or test for a meaningful gap, and examine equivalent mutants and configuration before treating every survivor as a defect.
How does StrykerJS fail CI below a mutation score?
Set the high, low and break thresholds in the Stryker configuration and run it with the project’s installed test runner. The break value controls failure below the chosen score, while the other thresholds control report bands.
Five predictions for where code intelligence is heading by 2028, based on 6 years of building tools in this space. Includes what's real, what's overhyped, and a maturity model for planning.
Most code quality dashboards fail because they track too many metrics and live outside the team's workflow. Here are exactly 7 metrics and a 3-week adoption plan that actually works.
Architecture fitness functions are automated tests for your system's structural properties. Here's how to implement the 5 most impactful ones and integrate them into your CI pipeline.
Apply developer documentation best practices with ADRs, comments and tested examples. Use the WRITE system to keep changes reviewable and find stale guidance.
A repeatable quarterly process for measuring codebase health using the VITAL Signs Framework. Covers velocity, incidents, test effectiveness, architecture, and legacy burden with real metrics and thresholds.