Testing is a key element of software development processes for the management and assessment of product quality. In most development environments, the software engineers are responsible for ensuring the functional correctness of code. However, for large complex software products, there is an additional need to check that changes do not negatively impact other parts of the software and they comply with system constraints such as backward compatibility, performance, security etc. Ensuring these system constraints may require complex verification infrastructure and test procedures. Although such tests are time consuming and expensive and rarely find defects they act as an insurance process to ensure the software is compliant. However, long lasting tests increasingly conflict with strategic aims to shorten release cycles. To decrease production costs and to improve development agility, we created a generic test selection strategy called THEO that accelerates test processes without sacrificing product quality. THEO is based on a cost model, which dynamically skips tests when the expected cost of running the test exceeds the expected cost of removing it. We replayed past development periods of three major Microsoft products resulting in a reduction of 50% of test executions, saving millions of dollars per year, while maintaining product quality.

K. Herzig, M. Greiler, J. Czerwonka, and B. Murphy, “The Art of Testing Less without Sacrificing Quality,” in Proceedings of the 2015 International Conference on Software Engineering, 2015.

title = {{The Art of Testing Less without Sacrificing Quality}},
author = {Herzig, Kim and Greiler, Michaela and Czerwonka, Jacek and Murphy, Brendan},
year = {2015},
month = {May},
booktitle = {{Proceedings of the 2015 International Conference on Software Engineering}},
location = {Florence, Italy},
publisher = {IEEE},

Download author PDF.
The author PDF is posted here by permission of IEEE for your personal use. Not for redistribution. The definitive version was published in Proceedings of the 2015 International Conference on Software Engineering .

No Comment

Comments are closed.