Classical mutation testing applies a small, fixed catalog of mutations to code, e.g. flipping + to – or changing True to False. But this produces mutated code that often doesn’t resemble the kinds of mistakes real developers actually make. In this project you will build a mutation-testing pipeline around a direct comparison: does learning to propose mutations (via a small contextual language model, sampling plausible-but-wrong code) find meaningfully different or better test gaps than the classical approach of generating everything mechanically and filtering afterward? We’ll look at Python tests since Python is easy to work with (no need to re-compile code after its mutated).
Outline work plan:
- Create mutation-testing harness: infrastructure code to carry out coverage-instrumented execution of the target program’s test suite, mutant injection, and kill/survive scoring. To provide a baseline this will include an existing open-source mutation-testing tool’s operator set e.g. cosmic-ray or mutmut.
- Write reachability prefilter: before any learned component runs, discard any candidate mutant sitting in code with zero test coverage; it’s guaranteed to survive trivially.
- Train a neural net mutation proposer: a small pretrained masked language model over code (CodeBERT-scale or similar), fine-tuned on the target codebase. At mutation time: mask a candidate token, sample from the model’s predicted distribution rather than taking its top prediction. That’s likely to generate code changes that don’t execute so a refinement might be to use Monte Carlo Tree Search (MCTS), with reward feedback on whether code executes or not, to pick good mutations.
- Evaluate: compare performance of neural net mutations vs classical AST mutations. Not only metrics such as mutation score achieved per unit of execution budget but also qualitative analysis e.g. different surviving mutants (different code regions or different kinds of test gaps), not just different quantities of them.