Passing is not the same as testing
A generated unit test is usually derived from the code it tests. That is the root of its main weakness: a test that reads the implementation and asserts what the implementation does will pass today and will keep passing if the implementation is wrong in the same way tomorrow. It adds coverage, because the lines execute, without adding the ability to detect a defect. Coverage measures what ran; it never measured what was checked. Generated suites make that old gap visible at scale.
Mutation testing is the review tool
Mutation testing, surveyed by Jia and Harman, makes small deliberate changes to the code (flip a comparison, change a constant, delete a statement) and reruns the tests. A mutant that no test fails on has survived, and a surviving mutant marks behaviour nobody checks. For machine-written tests it is the most efficient review there is: run it on the changed code, and every generated test that kills no mutant is either redundant or asserting nothing useful. Set a mutation score floor, for example 70 percent on changed lines, as a review gate rather than a vanity metric.
Run mutation testing on the changed code
Restrict it to the files the change touches so it finishes in minutes. List surviving mutants.
Reject tests that kill nothing
A generated test that kills no mutant is deleted or rewritten; it costs run time and maintenance and protects nothing.
Write tests for the survivors
Each surviving mutant in important logic gets a human-written test that fails on it, derived from the requirement, not the code.
Add the boundaries
Apply boundary value analysis to every input the generated tests only probed in the middle of its range.
Check the oracle
For each remaining test, ask where the expected value came from. If the answer is the implementation, replace it with a value from the specification.
Reviewing generated code
Treat generated code as a contribution from a capable new team member who has never seen the codebase and does not know the security rules: probably fine, occasionally wrong in confident and plausible ways. Pearce and colleagues, at the IEEE Symposium on Security and Privacy in 2022, generated 1,689 programs across security-relevant scenarios with a code assistant and found about 40 percent contained a vulnerability from the MITRE CWE top 25. Perry and colleagues found in a user study presented at ACM CCS 2023 that participants with an assistant wrote less secure code than those without, and were more confident that it was secure. The combination, more flaws and less suspicion, is what a review process has to counter.
| Check | What to look for | How |
|---|---|---|
| Dependencies exist | Imported packages, functions or options that do not exist or are misspelled | Resolve every new import against the package index; block unknown names |
| Security patterns | Injection, unsafe deserialisation, weak cryptography, secrets in code, missing authorisation | Static analysis plus the security testing checklist |
| Error handling | Swallowed exceptions, missing timeouts, happy-path-only logic | Review against the failure modes in the specification |
| Licence and provenance | Long verbatim passages that may carry a licence | Similarity scan on large generated blocks |
| Tests for claims | Behaviour the code or its comments claim with no test | Require a requirement-derived test for each claim |
Never review generated code by reading its explanation. Read the diff, run the tests you wrote, and verify every name it imports.
When generated tests are worth having
Generated tests earn their place in characterisation work (pinning the current behaviour of legacy code before a refactor, where asserting the implementation is exactly the point), in scaffolding data-heavy tests that a person then corrects, and in widening input variety for property-based tests. They are weakest where the value of a test lies in an independent oracle: business rules, calculations and security checks. Put the rule in the team's definition of done: generated tests are allowed, and each must survive the mutation gate. For the evaluation side of AI features, see test design for LLM features and the overview of how to test AI systems.
Common questions
Are AI-generated unit tests reliable?
They are reliable at executing code and unreliable at checking it. Many assert what the implementation does rather than what it should do. Mutation testing shows which ones would catch a defect.
What is mutation testing?
A technique that introduces small deliberate faults into the code and reruns the tests. Tests that fail on a mutant have detected it; mutants that survive show behaviour no test checks. The share of mutants killed is the mutation score.
Is AI-generated code less secure?
Published studies suggest it often is in security-relevant tasks: one 2022 study found about 40 percent of generated programs in such scenarios contained a known class of vulnerability, and a 2023 user study found assistant users wrote less secure code while feeling more confident.
How should a team review generated code?
As untrusted code from a new contributor: verify every dependency exists, run static analysis and a security checklist, check error handling and timeouts, and require requirement-derived tests for each behaviour it claims.
Does coverage from generated tests count?
It counts as coverage, which only ever measured what ran. Report mutation score for the changed code alongside it, so that coverage from tests that check nothing is visible for what it is.
Sources
- Perry et al., Do Users Write More Insecure Code with AI Assistants?
- Jia and Harman, An Analysis and Survey of the Development of Mutation Testing, IEEE TSE
- OWASP Web Security Testing Guide
Further reading named in the text
- Hammond Pearce, Baleegh Ahmad, Benjamin Tan, Brendan Dolan-Gavitt and Ramesh Karri, study of the security of assistant-generated code contributions (IEEE Symposium on Security and Privacy, 2022)
- Mark Chen, Jerry Tworek, Heewoo Jun and colleagues, Evaluating Large Language Models Trained on Code (arXiv preprint, 2021)
- MITRE, CWE Top 25 Most Dangerous Software Weaknesses
- Michael Feathers, Working Effectively with Legacy Code (Prentice Hall, 2004)
This guide is part of the testing AI systems hub. It is best read alongside test design for llm features and ROC curves and thresholds for AI text detectors, which cover the neighbouring questions.