Software reliability is heavily impacted by soft- ware changes. How do these changes relate to each other? By analyzing the impacted method definitions and usages, we determine dependencies between changes, resulting in a change genealogy that captures how earlier changes enable and cause later ones. Model checking this genealogy reveals temporal process patterns that encode key features of the software process such as pending development activities: “Whenever class A is changed, its test case is later updated as well.” Such patterns can be validated automatically: In an evaluation of four open source histories, our prototype would recommend pending activities with a precision of 60–72%.

Reference

BibTeX Entry

@inproceedings{herzig-issre-2011,
 title = {Mining Cause-Effect-Chains from Version Histories}, 
   author={Kim Herzig and Andreas Zeller}, 
   year = {2011}, 
   month = {November}, 
   booktitle={Proceedings of the 22nd International Symposium on Software Reliability Engineering} 
   }

Several defect prediction models have been proposed to identify which entities in a software system are likely to have defects before its release. This paper presents a replication of one such study conducted by Zimmermann and Nagappan [1] on Windows Server 2003 where the authors leveraged dependency relationships between software entities captured using social network metrics to predict whether they are likely to have defects. They found that network metrics perform significantly better than source code metrics at predicting defects. In order to corroborate the generality of their findings, we replicate their study on three open source Java projects, viz., JRuby, ArgoUML, and Eclipse. Our results are in agreement with the original study by Zimmermann and Nagappan when using a similar experimental setup as them (random sampling). However, when we evaluated the metrics using setups more suited for industrial use – forward-release and cross-project prediction – we found network metrics to offer no vantage over code metrics. Moreover, code metrics may be preferable to network metrics considering the data is easier to collect and we used only 8 code metrics compared to approximately 58 network metrics.

Reference

BibTeX Entry

@inproceedings{premraj-esem-2011,
 title = {Network versus Code Metrics to Predict Defects: A Replication Study}, 
   author={Premraj, Rahul and Herzig, Kim}, 
   year = {2011}, 
   month = {september}, 
   booktitle={Proceedings of the Fifth International Symposium on Empirical Software Engineering and Measurement}
}

Download PDF version

Changing source code in large software systems is complex and requires a good understanding of dependencies between software components. Modification to components with little regard to dependencies may have an adverse impact on the quality of the latter, i.e., increase their risk to fail. We conduct an empirical study to understand the relationship between the quality of components and the characteristics of their dependencies such as their frequency of change, their complexity, number of past failures and the like. Our study has been conducted on two large software systems: Microsoft VISTA and ECLIPSE. Our results show that components that have outgoing dependencies to components with higher object-oriented complexity tend to have fewer field failures for VISTA, but the opposite relation holds for ECLIPSE. Likewise, other notable observations have been made through our study that (a) confirm that certain characteristics of components increase the risk of their dependencies to fail and (b) some of the characteristics are project specific while some were also found to be common. We expect that such results can be leveraged for use to provide new directions for research in defect prediction, test prioritization and related research fields that utilize code dependencies in their empirical analysis. Additionally, these results provide insights to engineers on the potential reliability impacts of new component dependencies based upon the characteristics of the component.

Reference

BibTeX Entry

@inproceedings{zimmermann-icst-2011,
 title = {An Empirical Study of the Factors Relating Field Failures and Dependencies}, 
   author={Zimmerman, Thomas and Nagappan, Nachiappan and Herzig, Kim and Premraj, Rahul and Williams, Laurie}, 
   year = {2011}, 
   month = {march}, 
   booktitle={Software Testing, Verification and Validation (ICST), 2011 IEEE Fourth International Conference on Software Testing, Verification and Validation}
 pages={347--356}, 
   doi={10.1109/ICST.2011.39}
}

Download PDF version

In software development, every change induces a risk. What happens if code changes again and again in some period of time? In an empirical study on Windows Vista, we found that the features of such change bursts have the highest predictive power for defect-prone components. With precision and recall values well above 90%, change bursts significantly improve upon earlier predictors such as complexity metrics, code churn, or organizational structure. As they only rely on version history and a controlled change process, change bursts are straight-forward to detect and deploy.

Reference

BibTeX Entry

@inproceedings{nagappan-issre-2010,
 title = "Change Bursts as Defect Predictors",
 author = "Nachiappan Nagappan and Andreas Zeller and Thomas Zimmermann and Kim Herzig and Brendan Murphy",
 year = "2010",
 month = "November",
 booktitle = "Proceedings of the 21st IEEE International Symposium on Software Reliability Engineering",
 location = "San Jose, California
}

Download PDF version

Developers change source code to add new functionality, fix bugs, or refactor their code. Many of these changes have immediate impact on quality or stability. However, some impact of changes may become evident only in the long term. The goal of this thesis is to explore the long-term impact of changes by detecting dependencies between code changes and by measuring their influence on software quality, software maintainability, and development effort. Being able to identify the changes with the greatest long-term impact will strengthen our understanding of a project’s history and thus shape future code changes and decisions.

Reference

BibTeX Entry

@inproceedings{herzig-icse-2010,
 author = "Herzig, Kim Sebastian",
 title = "Capturing the long-term impact of changes",
 booktitle = "ICSE '10: Proceedings of the 32nd ACM/IEEE International Conference on Software Engineering",
 year = "2010",
 isbn = "978-1-60558-719-6",
 pages = "393--396",
 location = "Cape Town, South Africa",
 doi = "http://doi.acm.org/10.1145/1810295.1810401",
 publisher = "ACM",
 address = "New York, NY
}

Download PDF version

By integrating various development and collaboration tools into one single platform, the Jazz environment offers several opportunities for software repository miners. In particular, Jazz offers full traceability from the initial requirements via work packages and work assignments to the final changes and tests; all these features can be easily accessed and leveraged for better prediction and recommendation systems. In this paper, we share our initial experiences from mining the Jazz repository. We also give a short overview of the retrieved data sets and discuss possible problems of the Jazz repository and the platform itself.

Reference

BibTeX Entry

@inproceedings{herzig-msr-2009,
 title = "Mining the Jazz Repository: Challenges and Opportunities",
 author = "Kim Herzig and Andreas Zeller",
 year = "2009",
 month = "May",
 booktitle = "Proceedings of the 6th IEEE Working Conference on Mining Software Repositories (MSR09)",
 location = "Vancouver, BC, Canada"
}

Which components of a large software system are the most defect-prone? In a study on a large SAP Java system, we evaluated and compared a number of defect predictors, based on code features such as complexity metrics, static error detectors, change frequency, or component imports, thus replicating a number of earlier case studies in an industrial context. We found the overall predictive power to be lower than expected; still, the resulting regression models successfully predicted 50-60% of the 20% most defect-prone components.

Reference

BibTeX Entry

@inproceedings{holschuh-icse-2009,
 title = "Predicting Defects in SAP Java Code: An Experience Report",
 author = "Tilman Holschuh and Markus Paeuser and Kim Herzig and Thomas Zimmermann and Rahul Premraj and Andreas Zeller",
 year = "2009",
 month = "May",
 booktitle = "Proceedings of the 31th International Conference on Software Engineering",
 location = "Vancouver, BC
}

Download PDF version

Given a large body of code, how do we know where to focus our quality assurance effort? By mining the software’s defect history, we can automatically learn which code features correlated with defects in the past—and leverage these correlations for new predictions: “In the past, high inheritance depth was an indicator of a high number of defects. Since this new component also has a high inheritance depth, let us test it thoroughly”. Such history-based approaches work best if the new component is similar to the components learned from. But how does learning from history perform for projects with high variability between components? We ran a study on two SAP products involving a wide spectrum of functionality. We found that learning and predicting was accurate at package level, but not at product level. These results suggest that to learn from past defects, one should separate the product into component clusters with similar functionality, and make separate predictions for each cluster. Our initial approaches to form such clusters automatically, based on similarity of metrics, showed promising accuracy.

Reference

BibTeX Entry

@techreport{herzig-abap-prediction-2008,
 title = "Predicting Defects in SAP Products: A Replicated Study",
 author = "Kim Herzig and Rahul Premraj and Thomas Zimmermann and Andreas Zeller and Juergen Heymann",
 institution = "Software Engineering Chair, Saarland University"
}

Download PDF version

This thesis contains a lock flag. I am not allowed t publish this thesis neither completely nor in parts. The handout of this thesis to third persons requires a written approval of SAP.

Software products and projects can become very large and still grow over time. Building one prediction model for a whole software product might be easy but might also limit the prediction accuracy.

Different parts of a software product have different duties (GUI, database, kernel,…). We found out that for each of these different code zones there exist different code characteristics that matter when it comes to defect prediction. The main idea is to cluster code entities by their software duty. Currently we are investigating whether defect prediction taking advantage of such clustering techniques have a higher prediction accuracy than defect prediction models using no clustering technique.