Challenges with Applying Vulnerability Prediction Models @HotSoS 2015

While Microsoft product teams have adopted defect prediction models, they have not adopted vulnerability prediction models (VPMs). Seeking to understand this discrepancy, we replicated a VPM for two releases of the Windows Operating System, varying model granularity and statistical learners. We reproduced binary-level prediction precision (~0.75) and recall (~0.2). However, binaries often exceed 1 million lines of code, too large to practically inspect, and engineers expressed preference for source file level predictions. Our source file level models yield precision below 0.5 and recall below 0.2. We suggest that VPMs must be refined to achieve actionable performance, possibly through security-specific metrics.

  • [PDF] P. Morrison, K. Herzig, B. Murphy, and L. Williams, “Challenges with applying vulnerability prediction models,” in Proceedings of the 2015 symposium and bootcamp on the science of security, New York, NY, USA, 2015.
    [Bibtex]
    @inproceedings{morrison-hotsos-2015,
    author = {Morrison, Patrick and Herzig, Kim and Murphy, Brendan and Williams, Laurie},
    title = {Challenges with Applying Vulnerability Prediction Models},
    booktitle = {Proceedings of the 2015 Symposium and Bootcamp on the Science of Security},
    series = {HotSoS '15},
    year = {2015},
    location = {Illinoi, USA},
    publisher = {ACM},
    address = {New York, NY, USA},
    link={http://wp.me/p2TI1Q-k4},
    pdf={http://www.kim-herzig.de/wp-content/uploads/2015/02/ChallengesVulnerabilityModelsMicrosoft_HotSOS.pdf}
    }