Thursday, June 18, 2015

Can your tests survive the coming mutant apocalypse? [REPOST]



Are your unit tests effective against code mutations? No, you don't need to worry about your code being bombarded by gamma rays or being bitten by a radioactive spider. The question is will your tests catch minuscule changes in the code base that could have disastrous affects. Mutation testing has been around since 1971 but the majority of people in the software world haven't heard of it. So congratulations you have just learned something that can wow and amaze your peers at the lunch table.



Breaking code is easy

This may be earth shattering news but software engineers are not perfect. We make mistakes but that is why we test. The code written during a 2am coding bender all hyped up on Jolt Cola may be the stuff of legend that creates the next paradigm shifting application but one typo can result in catastrophe. Have you ever spent hours upon hours debugging a critical defect that resulted in something silly like the usage of the wrong operator or worse a missing semicolon?


100% code coverage != effective tests

Despite our best intentions test code is not treated with the same respect as the rest of your code base.  One hundred percent code coverage is a unicorn, really really rare, in complex applications but even that is not enough to ensure your code is rock solid. Even if your unit tests result in a perfect code coverage report, are you positive all your assert statements are effectively verifying the results? Mutant testing is essentially tests for your tests.

And you know what the worst part is? 
I NEVER LEARNED TO READ TEST.

If you are taking the time to read this then there is likely no need to convince you of how vital testing is. However, effective test writing is a skill/art/science but it is one of the many critical skills in the software engineering world that they likely did not teach you in college. Formal training on unit test writing is virtually unheard of. During code reviews the test code is not given the same type of scrutiny as the production code. Yet the tests are just as critical to the long term success of your project.


So what is mutation testing?

Your tests ensure your application works as you expect it to but what ensures that your tests sufficiently protect your application. Mutation testing is effectively making minor changes to your source code and ensuring your tests catch it and none of those mutants survive. A mutant that survive could one day grow up to be a defect.
mutant survival score = number of mutants killed / total number of mutants

The creation of a mutant army

As previously discussed mutants are brought into existence by sightly modifying your source code with something as simple as changing an equals into a not equals. Doing this for every conditional, arithmetic, relational, and assignment operator is just the beginning but you can see how the number of mutations even for a fairly simple piece of code can grow quickly.

    public int additionOrSubtraction(boolean isAddition, int x, int y) {
        if (isAddition) { //mutant: replace isAddition with !isAddition
            return x + y; //mutants: replace + with -,*,/,%
        } else {
            return x - y; //mutants: replace - with +,*,/,%
        }
    }

The code above could result in 50 or more mutants which your unit tests would then be executed against.  If any of those mutants do not cause a test failure it "survives."

Creating a mutant army is not something you want to be doing by hand so look for tools available in your language of choice. Mutation testing is done almost exclusively with unit tests in mind. That is not to say that throwing a few mutants randomly against your integration tests would not be valuable too but throwing a fully mutant army against your functional tests would be absurdly costly.




Are you scared of the mutants yet?

Do not alarmed. These mutants are here to save you. Send in the mutants! Save your tests. Your project could be one fat fingered key stroke away from disaster.


Find out more: Give your tests mutant powers with PIT [Part 1]




Post with my own permission as the original author.  (Source)

32 comments:

  1. I am reading your post from the beginning, it was so interesting to read & I feel thanks to you for posting such a good blog, keep updates regularly.
    Regards,
    Node JS training|Node JS training in chennai

    ReplyDelete
  2. Hello , I have been training students on AngularJS for past 6 months, and at times, I have used your blog as reference for the class training and also for my personal project development. It has been so much useful. Thank you, keep writing more:)
    AngularJS training

    ReplyDelete
  3. a couple of years ago my partner and i turned on using THE new Vortex Diamondback 8.5 x 50 binoculars with regard to nature outings AND sporting events ALONG WITH my partner and i soon realized they were better made ALONG WITH additional reliable as compared to any other brands when i had sooner owned. all model lines involving Vortex binoculars are usually purged throughout argon gas that will provides extraordinary waterproof protection. http://www.mordocrosswords.com/2015/12/survived-test.html

    ReplyDelete
  4. Thanks for your informative post!!! After completing my graduation, i am confused whether to choose web design as my career. Your article helped me to make a right choice. PHP Training in Chennai

    ReplyDelete
  5. Pretty content with useful information. Angularjs is a structural framework for developing the dynamic web applications. Angular's data binding and dependecy injection is capable of eliminating much of the code.
    angularjs training in chennai | angularjs training chennai

    ReplyDelete
  6. Angular JS is a structural framework for dynamic web application and angular's data binding eliminates much of the code that you would otherwise write it.
    Angularjs Training in Chennai | Angularjs training Chennai

    ReplyDelete
  7. The strategy you have posted on this technology helped me to get into the next level and had lot of information in it. The python programming language is very popular and most widely used.
    Python Training in Chennai | Python Course in Chennai

    ReplyDelete
  8. Pretty article! I found some useful information in your blog, it was awesome to read, thanks for sharing this great content to my vision, keep sharing.
    Regards,
    ETL Testing Training in Chennai | SoapUI Training in Chennai

    ReplyDelete
  9. Thanks for posting this useful content, Good to know about new things here, Let me share this,
    AngularJS Training in Chennai | AngularJS Training | AngularJS Training Institute in Chennai

    ReplyDelete
  10. Thanks for sharing this unique and informative content which provided me the required information.
    Java Training in Chennai | JAVA Course in Chennai

    ReplyDelete
  11. Great explanation. Really it will be useful for both freshers and working professionals. Thanks for you post.
    AngularJS Training In Chennai | AngularJS Training Institute in Chennai

    ReplyDelete
  12. This comment has been removed by the author.

    ReplyDelete
  13. Nice blog. Thank you for sharing. The information you shared is very effective for learners I have got some important suggestions from it. erp in chennai.

    ReplyDelete
  14. I really enjoy simply reading all of your weblogs. Simply wanted to inform you that you have people like me who appreciate your work. 
    Software Testing Training Center in Chennai | | Best Selenium Training Institute in Chennai | Six Sigma Certification Training in Chennai

    ReplyDelete
  15. Well Said, you have furnished the right information that will be useful to anyone at all time. Thanks for sharing your Ideas.
    Linux Certification Training in Chennai | Microsoft Certification Training in Chennai | MCSA Training in Chennai

    ReplyDelete
  16. Nice and good article, thanks for sharing your views and ideas, it is really useful to me.. Thanks , keep updating and rocks...Java Training in Chennai | Web Designing Training Institute in Chennai | DotNet Training Institute in Chennai

    ReplyDelete
  17. Excellent post. I have read your blog it's very interesting and informative. Keep sharing.
    Power System Project Center in Chennai | Power System Project Center in Velachery

    ReplyDelete
  18. Great article..Thanks for sharing your excellent post with smart and cute content..keep updating..MBA Projects Center in Chennai | MBA Projects Center in Velachery

    ReplyDelete
  19. Excellent post!!!. The strategy you have posted on this technology helped me to get into the next level and had lot of information in it.
    Best VLSI Project Center in Chennai | VLSI Project Center in Velachery

    ReplyDelete
  20. • I enjoy what you guys are usually up too. This sort of clever work and coverage! Keep up the wonderful works guys I’ve added you guys to my blog roll.

    selinium training in bangalore

    ReplyDelete
  21. Those guidelines additionally worked to become a good way to recognize that other people online have the identical fervor like mine to grasp great deal more around this condition.

    java training in bangalore

    ReplyDelete
  22. I believe that there will be found more interesting and useful content for those who are coming around this area.
    Best Online Software Training Institute | Testing Tools Training

    ReplyDelete