QA of AI-generated test cases and code: a review method

Main test weakness

assertions that check nothing useful

Detection technique

mutation testing

Mutation score floor

70 percent on changed code

Code risk

vulnerable in about 40 percent of scenarios (2022 study)

Invented dependencies

verify every package exists

Review stance

untrusted new contributor

Passing is not the same as testing

A generated unit test is usually derived from the code it tests. That is the root of its main weakness: a test that reads the implementation and asserts what the implementation does will pass today and will keep passing if the implementation is wrong in the same way tomorrow. It adds coverage, because the lines execute, without adding the ability to detect a defect. Coverage measures what ran; it never measured what was checked. Generated suites make that old gap visible at scale.

Mutation testing is the review tool

Mutation testing, surveyed by Jia and Harman, makes small deliberate changes to the code (flip a comparison, change a constant, delete a statement) and reruns the tests. A mutant that no test fails on has survived, and a surviving mutant marks behaviour nobody checks. For machine-written tests it is the most efficient review there is: run it on the changed code, and every generated test that kills no mutant is either redundant or asserting nothing useful. Set a mutation score floor, for example 70 percent on changed lines, as a review gate rather than a vanity metric.

1. 1

   Run mutation testing on the changed code

   Restrict it to the files the change touches so it finishes in minutes. List surviving mutants.

2. 2

   Reject tests that kill nothing

   A generated test that kills no mutant is deleted or rewritten; it costs run time and maintenance and protects nothing.

3. 3

   Write tests for the survivors

   Each surviving mutant in important logic gets a human-written test that fails on it, derived from the requirement, not the code.

4. 4

   Add the boundaries

   Apply boundary value analysis to every input the generated tests only probed in the middle of its range.

5. 5

   Check the oracle

   For each remaining test, ask where the expected value came from. If the answer is the implementation, replace it with a value from the specification.

Reviewing generated code

Treat generated code as a contribution from a capable new team member who has never seen the codebase and does not know the security rules: probably fine, occasionally wrong in confident and plausible ways. Pearce and colleagues, at the IEEE Symposium on Security and Privacy in 2022, generated 1,689 programs across security-relevant scenarios with a code assistant and found about 40 percent contained a vulnerability from the MITRE CWE top 25. Perry and colleagues found in a user study presented at ACM CCS 2023 that participants with an assistant wrote less secure code than those without, and were more confident that it was secure. The combination, more flaws and less suspicion, is what a review process has to counter.

When generated tests are worth having

Generated tests earn their place in characterisation work (pinning the current behaviour of legacy code before a refactor, where asserting the implementation is exactly the point), in scaffolding data-heavy tests that a person then corrects, and in widening input variety for property-based tests. They are weakest where the value of a test lies in an independent oracle: business rules, calculations and security checks. Put the rule in the team's definition of done: generated tests are allowed, and each must survive the mutation gate. For the evaluation side of AI features, see test design for LLM features and the overview of how to test AI systems.

This guide is part of the testing AI systems hub. It is best read alongside test design for llm features and ROC curves and thresholds for AI text detectors, which cover the neighbouring questions.