Thursday, June 18, 2015

Can your tests survive the coming mutant apocalypse? [REPOST]

Are your unit tests effective against code mutations? No, you don't need to worry about your code being bombarded by gamma rays or being bitten by a radioactive spider. The question is will your tests catch minuscule changes in the code base that could have disastrous affects. Mutation testing has been around since 1971 but the majority of people in the software world haven't heard of it. So congratulations you have just learned something that can wow and amaze your peers at the lunch table.

Breaking code is easy

This may be earth shattering news but software engineers are not perfect. We make mistakes but that is why we test. The code written during a 2am coding bender all hyped up on Jolt Cola may be the stuff of legend that creates the next paradigm shifting application but one typo can result in catastrophe. Have you ever spent hours upon hours debugging a critical defect that resulted in something silly like the usage of the wrong operator or worse a missing semicolon?

100% code coverage != effective tests

Despite our best intentions test code is not treated with the same respect as the rest of your code base.  One hundred percent code coverage is a unicorn, really really rare, in complex applications but even that is not enough to ensure your code is rock solid. Even if your unit tests result in a perfect code coverage report, are you positive all your assert statements are effectively verifying the results? Mutant testing is essentially tests for your tests.

And you know what the worst part is? 

If you are taking the time to read this then there is likely no need to convince you of how vital testing is. However, effective test writing is a skill/art/science but it is one of the many critical skills in the software engineering world that they likely did not teach you in college. Formal training on unit test writing is virtually unheard of. During code reviews the test code is not given the same type of scrutiny as the production code. Yet the tests are just as critical to the long term success of your project.

So what is mutation testing?

Your tests ensure your application works as you expect it to but what ensures that your tests sufficiently protect your application. Mutation testing is effectively making minor changes to your source code and ensuring your tests catch it and none of those mutants survive. A mutant that survive could one day grow up to be a defect.
mutant survival score = number of mutants killed / total number of mutants

The creation of a mutant army

As previously discussed mutants are brought into existence by sightly modifying your source code with something as simple as changing an equals into a not equals. Doing this for every conditional, arithmetic, relational, and assignment operator is just the beginning but you can see how the number of mutations even for a fairly simple piece of code can grow quickly.

    public int additionOrSubtraction(boolean isAddition, int x, int y) {
        if (isAddition) { //mutant: replace isAddition with !isAddition
            return x + y; //mutants: replace + with -,*,/,%
        } else {
            return x - y; //mutants: replace - with +,*,/,%

The code above could result in 50 or more mutants which your unit tests would then be executed against.  If any of those mutants do not cause a test failure it "survives."

Creating a mutant army is not something you want to be doing by hand so look for tools available in your language of choice. Mutation testing is done almost exclusively with unit tests in mind. That is not to say that throwing a few mutants randomly against your integration tests would not be valuable too but throwing a fully mutant army against your functional tests would be absurdly costly.

Are you scared of the mutants yet?

Do not alarmed. These mutants are here to save you. Send in the mutants! Save your tests. Your project could be one fat fingered key stroke away from disaster.

Find out more: Give your tests mutant powers with PIT [Part 1]

Post with my own permission as the original author.  (Source)


  1. I am reading your post from the beginning, it was so interesting to read & I feel thanks to you for posting such a good blog, keep updates regularly.
    Node JS training|Node JS training in chennai

  2. Hello , I have been training students on AngularJS for past 6 months, and at times, I have used your blog as reference for the class training and also for my personal project development. It has been so much useful. Thank you, keep writing more:)
    AngularJS training

  3. a couple of years ago my partner and i turned on using THE new Vortex Diamondback 8.5 x 50 binoculars with regard to nature outings AND sporting events ALONG WITH my partner and i soon realized they were better made ALONG WITH additional reliable as compared to any other brands when i had sooner owned. all model lines involving Vortex binoculars are usually purged throughout argon gas that will provides extraordinary waterproof protection.

  4. Thanks for your informative post!!! After completing my graduation, i am confused whether to choose web design as my career. Your article helped me to make a right choice. PHP Training in Chennai

  5. Pretty content with useful information. Angularjs is a structural framework for developing the dynamic web applications. Angular's data binding and dependecy injection is capable of eliminating much of the code.
    angularjs training in chennai | angularjs training chennai

  6. Angular JS is a structural framework for dynamic web application and angular's data binding eliminates much of the code that you would otherwise write it.
    Angularjs Training in Chennai | Angularjs training Chennai

  7. The strategy you have posted on this technology helped me to get into the next level and had lot of information in it. The python programming language is very popular and most widely used.
    Python Training in Chennai | Python Course in Chennai

  8. Pretty article! I found some useful information in your blog, it was awesome to read, thanks for sharing this great content to my vision, keep sharing.
    ETL Testing Training in Chennai | SoapUI Training in Chennai

  9. Thanks for posting this useful content, Good to know about new things here, Let me share this,
    AngularJS Training in Chennai | AngularJS Training | AngularJS Training Institute in Chennai

  10. Thanks for sharing this unique and informative content which provided me the required information.
    Java Training in Chennai | JAVA Course in Chennai

  11. Great explanation. Really it will be useful for both freshers and working professionals. Thanks for you post.
    AngularJS Training In Chennai | AngularJS Training Institute in Chennai

  12. This comment has been removed by the author.

  13. Nice blog. Thank you for sharing. The information you shared is very effective for learners I have got some important suggestions from it. erp in chennai.

  14. I really enjoy simply reading all of your weblogs. Simply wanted to inform you that you have people like me who appreciate your work. 
    Software Testing Training Center in Chennai | | Best Selenium Training Institute in Chennai | Six Sigma Certification Training in Chennai

  15. Well Said, you have furnished the right information that will be useful to anyone at all time. Thanks for sharing your Ideas.
    Linux Certification Training in Chennai | Microsoft Certification Training in Chennai | MCSA Training in Chennai

  16. Nice and good article, thanks for sharing your views and ideas, it is really useful to me.. Thanks , keep updating and rocks...Java Training in Chennai | Web Designing Training Institute in Chennai | DotNet Training Institute in Chennai

  17. Excellent post. I have read your blog it's very interesting and informative. Keep sharing.
    Power System Project Center in Chennai | Power System Project Center in Velachery

  18. Great article..Thanks for sharing your excellent post with smart and cute content..keep updating..MBA Projects Center in Chennai | MBA Projects Center in Velachery

  19. Excellent post!!!. The strategy you have posted on this technology helped me to get into the next level and had lot of information in it.
    Best VLSI Project Center in Chennai | VLSI Project Center in Velachery

  20. • I enjoy what you guys are usually up too. This sort of clever work and coverage! Keep up the wonderful works guys I’ve added you guys to my blog roll.

    selinium training in bangalore

  21. Those guidelines additionally worked to become a good way to recognize that other people online have the identical fervor like mine to grasp great deal more around this condition.

    java training in bangalore

  22. I believe that there will be found more interesting and useful content for those who are coming around this area.
    Best Online Software Training Institute | Testing Tools Training

  23. Thank you a lot for providing individuals with a very spectacular possibility to read critical reviews from this site.
    Good AWS Training in Tambaram | AWS Courses in Adyar | AWS Institute in Besant Nagar

  24. I have visited to this site multiple times and everytime I find beneficial jobs for me so I would suggest please come to this site and take the chance from here.
    AWS Certification Exam Centers in Taramani | AWS Training in Shozhinganallur

  25. A log of thanks for sharing information for more websites with their sort order ranking.
    CorelDraw Certification Training in Chennai | Advance CorelDraw Training in Madipakkam

  26. This is a wonderful article, Given so much info in it, These type of articles keeps the users interest in the website, and keep on sharing more ... good luck
    Android Exam Certification Training in Chennai | No.1 Android Exam in Velachery

  27. I enjoy what you guys are usually up too. This sort of clever work and coverage! Keep up the wonderful works guys.
    Hardware and Networking Certification Center in Chennai | No.1 Hardware Exam in Medavakkam

  28. Thanks for sharing such an informative blog. I have read your blog and I gathered some needful information from your blog. Keep update your blog.
    Certified Ethical Hacking Training in Chennai | No.1 Ethical Hacking in Porur

  29. Impressive blog with lovely information. really very useful article for us thanks for sharing such a wonderful blog...
    Cisco Certification Training in Chennai | Excellent Cisco Courses in Pallavaram

  30. Thanks for posting useful information.You have provided an nice article, Thank you very much for this one. And i hope this will be useful for many people..
    CompTIA Security Plus Certifications Course in Chennai | Best CompTIA Security+ Training in Mandaveli

  31. Thank you a lot for providing individuals with a very spectacular possibility to read critical reviews from this site.
    CompTIA A Plus Certifications Coaching Center in Chennai | No.1 CompTIA Security+ Training in Pallavaram

  32. It’s great to come across a blog every once in a while that isn’t the same out of date rehashed material. Fantastic read.
    Comptia Network Plus Certifications Coaching in Chennai | Best N+ Training in Pallavaram

  33. Interesting article about "Software engineering in the eyes of the Lough
    (pr. Law)" thanks for sharing.
    Data Science Training in Chennai


  34. Hello. This post couldn’t be written any better! Reading this post reminds me of my previous roommate. He always kept chatting about this. I will forward this page to him.
    bigdata training in bangalore

  35. Appreciation for really being thoughtful and also for deciding on certain marvelous guides most people really want to be aware of.

    DevOps Training in Chennai
    DevOPs Online Training
    DevOps Training

  36. Very Nice Blog I like the way you explained these things. I’ve been looking for ways to improve my website and overall rankings. I hope your future article will help me further. Take SEO Training in our Aorta Institute, Chennai to mould yourself.
    Digital Marketing Training Institute in Chennai | SEO Training in Chennai