While Microsoft product teams have adopted defect prediction models, they have not adopted vulnerability prediction models (VPMs). Seeking to understand this discrepancy, we replicated a VPM for two releases of the Windows Operating System, varying model granularity and statistical learners. We reproduced binary-level prediction precision (~0.75) and recall (~0.2). However, binaries often exceed 1 million lines of code, too large to practically inspect, and engineers expressed preference for source file level predictions. Our source file level models yield precision below 0.5 and recall below 0.2. We suggest that VPMs must be refined to achieve actionable performance, possibly through security-specific metrics.

P. Morrison, K. Herzig, B. Murphy, and L. Williams, “Challenges with applying vulnerability prediction models,” in Proceedings of the 2015 symposium and bootcamp on the science of security, New York, NY, USA, 2015.
[Bibtex]

@inproceedings{morrison-hotsos-2015,
author = {Morrison, Patrick and Herzig, Kim and Murphy, Brendan and Williams, Laurie},
title = {Challenges with Applying Vulnerability Prediction Models},
booktitle = {Proceedings of the 2015 Symposium and Bootcamp on the Science of Security},
series = {HotSoS '15},
year = {2015},
location = {Illinoi, USA},
publisher = {ACM},
address = {New York, NY, USA},
link={http://wp.me/p2TI1Q-k4},
pdf={http://www.kim-herzig.de/wp-content/uploads/2015/02/ChallengesVulnerabilityModelsMicrosoft_HotSOS.pdf}
}

Download author PDF.
The author PDF is posted here by permission of ACM for your personal use. Not for redistribution. The definitive version was published in Proceedings of the 2015 symposium and bootcamp on the science of security .

No Comment

Comments are closed.