Workshop on Industrial Software Testing @ FSE 2016

Peter Rigby and myself are organizing a workshop on Industrial Software Testing at the 24th ACM SIGSOFT International Symposium on the Foundations of Software Engineering (FSE) held in Seattle, WA, USA.


The workshop will be held Monday, November 14, 2016.
The submission deadline is July 1, 2016. Please check the workshop website for the latest news.

What is this workshop about?

The goal of this workshop is to discuss common, open issues in industrial software testing. This workshop is not about completed research studies or a showcase of the latest achievements in software testing. With this workshop, we want to offer a platform for industry and research to discuss unsolved testing and verification issues that occur in industry.

Call for contributions

In recent years, the pressure on software producers to cut their time to market for software products has increased significantly. At the same time, software products get more complex. From this perspective, software testing and verification is a lower bound on how fast we can ship software. While there have been outstanding advances in build systems, software testing processes have not changed significantly, as it rarely has been the primary bottleneck blocking software development agility.
We seek for contributions articulating and highlighting common issues in industrial testing. How do we test software, where are bottlenecks, and how should testing scale looking forward? We want to focus on open or only partially solved testing issues at industrial scale. This workshop is about problem, not solutions.
We explicitly encourage problem statements that go beyond unit and functional correctness testing, but rather target the entire verification process including the expensive field of system and integration testing.

Potential and relevant topics might include:

  • Flaky tests: Tests failing due to any other reason than code defects pose a big threat to the efficiency and effectiveness of any verification process. There are many root causes for flaky tests. However, there seem to exist very little efficient solutions that help engineers to fix flaky tests and to design them in a more robust way in the first place. We are interested in common solutions and issues around the topic of flaky tests.
  • Test selection: While there is a long list of research on test selection and test prioritization, many of these research studies did not consider the economic aspects of testing, e.g. the cost of flaky tests, the cost of test infrastructure, or the potential risk of elapsing defects into later stages of the development process. We are interested in strategies at large-scale, for both unit as well as system and integration tests (system \& integration tests are widely ignored in research).
  • Verification processes: Testing is only one step of the larger verification process for software products. We are interested in overall development process strategies and how testing fits into these processes. We seek common problems and bottlenecks in these processes as well as side effects from or to other processes, such as build.
  • Missing tool support: What test tools are currently available and are there any common gaps that need to be closed to allow engineers and testers to be more effective and efficient in creating, running, debugging, and triaging tests.
  • Migrating system & integration tests towards unit tests. By nature, system and integration tests are long running usually finding few code defects and if finding them, finding them late in the development process causing extra cost.

Companies participating?

These companies showed interest and are likely to participate:

  • Microsoft
  • Cisco
  • Erisson
  • Intel
  • Amazon
  • Sendwithus

We are still spreading the word and hope to extend the list in the coming months.

More Details?

For more details please visit the workshop website: Or send us an email with your questions.

See you all in Seattle.