Mutation Testing
Have you ever thought about creating unit tests with good coverage but thought that it was hard or choosing the unit test data stopped you from making the optimum tests?
If only there was a method to test my test?
What is mutation testing?
Mutation testing is a software testing type that is based on changes or mutations. Miniscule changes are introduced into the source code to check whether the defined test cases can detect errors in the code. These Miniscule changes mimic software errors so that the bugs they introduce are validated by research.
The miniscule changes are Mutations
The changed source code is Mutant
In other words, It is A Test for Your Test.
The objective of Mutation Testing:
- To identify pieces of code that are not tested properly.
- To identify hidden defects that cannot be detected using other testing methods.
- To discover new kinds of errors or bugs.
- To assess the quality of the test cases(by calculating the mutation score).
Mutation Score = (killed mutants )/(total number of mutatnts ) *100
Advantages of Mutation Testing :
- It brings a good level of error detection to the program.
- It discovers ambiguities in the source code.
- High coverage of the source program is attained.
- Loopholes in test data can be identified.
Disadvantages of Mutation Testing:
- It is highly costly and time-consuming.
- Mutation testing is not applicable for black-box testing as it involves a lot of source code changes.
Note: Automation is necessary for mutation testing as it is very time-consuming.
Types of mutants & Mutations :
- Killed mutant: mutants that were killed after mutation testing (which means that our test suit can detect changes in the code and will fail accordingly )
- Survived mutant: mutants that are alive after running the test suit on the mutated source code.
- Equivalent mutant: mutants that are alive even after running test data through them, but they have the same meaning as the original source code even though they may have different syntax.
Mutations:
Value Mutations:
The values are changed to detect errors in the program. Basically, a small value is changed to a larger value, or a larger value is changed to a smaller value. basically, constants are changed.
Ex : Int x = 10000000254 -> int x = 10254
Decision Mutations:
Logical or arithmetic operators are changed to detect errors in the program.
Ex : If (a < b) -> if (a > b)
Statement Mutations:
A statement is deleted, or it is replaced by some other statements.
Ex : C = 10 -> d = 10
How to Apply Mutation Testing:
- Write a couple of unit tests for your module
- Run mutation testing automaton on it (like Pitest).
- Check the result and change the test suit accordingly by adding tests or changing test’s data.
- Repeat steps 2 and 3 until reaching good coverage.
Optional Steps:
- Try to remove the unit test and run the mutation testing automaton again
- If the coverage didn’t change Remove the test.
- Repeat steps 1 and 2 till you reach the optimum test suite.
Example:
In this example, we are using Pitest which Is a mutation testing system to automate the process.
Having a small function as
public int add(int a , int b){
if(a < 2)
{
return (a + b)* -1;
}
else{
return a+b;
}
}
we could start with a couple of small tests as
public void testAdd1() {
int result = new MutationTesting().add(0,0);
assertEquals(0,result);
} public void testAdd2() {
int result = new MutationTesting().add(3,0);
assertEquals(3,result);
}
Then run Pitest & view the report.
Pitest created 7 mutations and our test suit killed only 3, so we can calculate the mutation score as 3/7*100 = 43%
Or we can view it in the report as the percentage next to the progress bar.
Mutation Testing. java
The report displays :
The number next to the line of code is the number of mutations (changes) for each line
The mutations section indicates which mutation was used on what lines and whether it survived or died.
The report also indicates which mutators were used
Note: You can find all about the available mutators and the default mutators on Pitest website https://pitest.org/quickstart/mutators/
Using the report as our guide we start to see the lack of coverage of our test suite. For example, seeing that we didn’t have a test for the edge value of 2 or we didn’t try the addition with a number other than zero to check both the branches of the "if condition".
Test yourself whether you can reach the maximum coverage for this case.
Feel free to contact us if you get stuck or look for the solution in here https://github.com/sandraerian16/MutationTestingExample.git
Conclusion
Mutation Testing is one of the most effective ways to ensure good coverage. It is very easy to apply and there are many mutation testing systems that help.
If you want to start making a better test suit or want to be sure of your test suit data; mutation testing is your choice, Though it might be time-consuming and costly, It would be worthwhile if done correctly.
.webp)
.webp)
