Testing and Continuous Integration at Scale – Expectations, Cost, and Limits (Keynote @ SBST 2018)

The AST (automated software testing) workshop invited me to give a keynote about testing and continuous integration at Microsoft. Below, you find the abstract and a copy of the slides presented in Austin Texas 2016.


It should not come as a surprise but testing is not for free, it costs money and effort. Test automation is supposed to help development teams to “[…] reduce the cost and improve the effectiveness of software testing […]” [1]. But what exactly is it we are automating and does it actually help to solve a problem? Are we providing solutions that are actually worth the effort and who is going to pay for it? In this talk, I provide insights into large-scale verification processes and tool-chains that solved some pressing testing issues, but also provide new challenges that require sustainable and acceptable solutions that are worth automating.


  • [DOI] K. Herzig, “Testing and continuous integration at scale: limits, costs, and expectations,” in Proceedings of the 11th international workshop on search-based software testing, New York, NY, USA, 2018, p. 38–38.
    author = {Herzig, Kim},
    title = {Testing and Continuous Integration at Scale: Limits, Costs, and Expectations},
    booktitle = {Proceedings of the 11th International Workshop on Search-Based Software Testing},
    series = {SBST '18},
    year = {2018},
    isbn = {978-1-4503-5741-8},
    location = {Gothenburg, Sweden},
    pages = {38--38},
    numpages = {1},
    url = {http://doi.acm.org/10.1145/3194718.3194731},
    doi = {10.1145/3194718.3194731},
    acmid = {3194731},
    publisher = {ACM},
    address = {New York, NY, USA},


Link to workshop website: http://tech.brookes.ac.uk/AST2016/keynote.htm