User Tools
vulnerability_discovery_models
Differences
This shows you the differences between two versions of the page.
| Both sides previous revision Previous revision Next revision | Previous revision | ||
|
vulnerability_discovery_models [2017/12/13 20:45] ivan.pashchenko@unitn.it [Talks and Tutorials] |
vulnerability_discovery_models [2025/01/28 00:47] (current) fabio.massacci@unitn.it |
||
|---|---|---|---|
| Line 3: | Line 3: | ||
| Among the [[research_activities|research topics]] of the [[start|Security Group]] we focus on here on the following activities: | Among the [[research_activities|research topics]] of the [[start|Security Group]] we focus on here on the following activities: | ||
| - | * How to (automatically) find vulnerabilities in the (10 years old) version you shipped to the customers? | + | * Which vulnerable dependencies really matter? |
| + | * How to (automatically) find vulnerabilities in the deployed versions of FOSS? | ||
| * How to automatically test them (when you get a report) | * How to automatically test them (when you get a report) | ||
| * Which vulnerabilities are actually exploited in the Wild? | * Which vulnerabilities are actually exploited in the Wild? | ||
| + | * Which vulnerability scanning tool performs best on your particular project? | ||
| * Empirical Validation of Vulnerability Discovery Models? | * Empirical Validation of Vulnerability Discovery Models? | ||
| Line 12: | Line 14: | ||
| Most importantly, **Do you want data?** We know that building datasets is difficult, error prone and time consuming so we have decided to share our efforts of the past 4 years. Check our [[datasets|Security Datasets in Trento]]. | Most importantly, **Do you want data?** We know that building datasets is difficult, error prone and time consuming so we have decided to share our efforts of the past 4 years. Check our [[datasets|Security Datasets in Trento]]. | ||
| - | ===== How to (automatically) find vulnerabilities in the (10 years old) version you shipped to the customers? ===== | + | ===== Bringing order to the dependency hell: which vulnerable dependencies really matter ===== |
| - | Our [[https://link.springer.com/article/10.1007%2Fs10664-015-9408-2|new paper in the Empirical Software Engineering Journal]] proposes an automated method to determine the code evidence for the presence of vulnerabilities in retro software versions. | + | Vulnerable dependencies are a known problem in today’s open-source software ecosystems because FOSS libraries are highly interconnected and developers do not always update their dependencies. |
| - | Why should one bother? After all that code is old code. Well, this is only true if you are thinking about your free browser that is shipped to you in change of your personal data and is updated periodically possibly breaking your extensions. In all other cases, when software is proprietary, it has a actually very long shelf life (Industrial control systems last scores and for ERP systems a decade isn't rare). | + | You may want to read first a thematic analysis study ({{:research_activities:vulnerability-analysis:ccs-2020-aam.pdf|Accepted in CCS}}) in which we interviewed 25 developers all over the world provide some important insight in the choice of company to update or not update the software. |
| - | The method scans the code base of each retro version of software for the code evidence to determine whether a retro version is vulnerable or not. It identifies the lines of code that were changed to fix vulnerabilities. If an earlier version contains these deleted lines, it is highly likely that this version is vulnerable. To show the scalability of the method we performed a large scale experiments on Chrome and Firefox (spanning 7,236 vulnerable files and approximately 9,800 vulnerabilities) on the National Vulnerability Database (NVD). The elimination of spurious vulnerability claims (e.g. entries to a vulnerability database such as NVD) found by our method may change the conclusions of studies on the prevalence of foundational vulnerabilities. | + | In {{:research_activities:vulnerability-analysis:pashchenko-vuln4real.pdf |TSE 2020 Paper}} we show how to avoid the over-inflation problem of academic and industrial approaches for reporting vulnerable dependencies in FOSS software, and therefore, satisfy the needs of industrial practice for correct allocation of development and audit resources. |
| + | |||
| + | To achieve this, we carefully analysed the deployed dependencies, aggregated dependencies by their projects, and distinguished halted dependencies. All this allowed us to obtain a counting method that avoids over-inflation. | ||
| + | |||
| + | To understand the industrial impact, we considered the 200 most popular FOSS Java libraries used by SAP in its own software. Our analysis included 10905 distinct GAVs (group, artifact, version) in Maven when considering all the library versions. | ||
| + | |||
| + | We found that about 20% of the dependencies affected by a known vulnerability are not deployed, and therefore, they do not represent a danger to the analyzed library because they cannot be exploited in practice. Developers of the analyzed libraries are able to fix (and actually responsible for) 82% of the deployed vulnerable dependencies. The vast majority (81%) of vulnerable dependencies may be fixed by simply updating to a new version, while 1% of the vulnerable dependencies in our sample are halted, and therefore, potentially require a costly mitigation strategy. | ||
| + | |||
| + | Our methodology allows software development companies to receive actionable information about their library dependencies, and therefore, correctly allocate costly development and audit resources, which is spent inefficiently in case of distorted measurements. | ||
| + | |||
| + | Do you want to check if your project actually uses some vulnerable dependencies? Let us know. | ||
| + | |||
| + | |||
| + | ===== A Screening Test for Disclosed Vulnerabilities in FOSS Components ===== | ||
| + | |||
| + | Our {{:research_activities:vulnerability-analysis:camera-ready.pdf|new paper in the IEEE Transactions on Software Engineering}} proposes an automated method to determine the code evidence for the presence of vulnerabilities in retro software versions. | ||
| + | |||
| + | Why is should you worry about a disclosed vulnerabilities? Each time a vulnerability is disclosed in a FOSS component, a software vendor using this component in an application must decide whether to update the FOSS component, patch the application itself, or just do nothing as the vulnerability is not applicable to the older version of the FOSS component used. | ||
| + | |||
| + | To address this challenge, we propose a //screening test//: a novel, automatic method based on thin slicing, for estimating quickly whether a given vulnerability is present in a consumed FOSS component by looking across its entire repository. We have applied it our test suit to large open source projects (e.g., Apache Tomcat, Spring Framework, Jenkins) that are routinely used by large software | ||
| + | vendors, scanning thousands of commits and hundred thousands lines of code in a matter of minutes. | ||
| + | |||
| + | Further, we provide insights on the empirical probability that, on the above mentioned projects, a potentially vulnerable component might not actually be vulnerable after all (e.g. entries to a vulnerability database such as NVD, which says that a version is vulnerable when the code is not even there), | ||
| + | |||
| + | A previous [[https://link.springer.com/article/10.1007%2Fs10664-015-9408-2|paper in the Empirical Software Engineering Journal]] focussed on Chrome and Firefox (spanning 7,236 vulnerable files and approximately 9,800 vulnerabilities) on the National Vulnerability Database (NVD). We found out that the elimination of spurious vulnerability claims found by our method may change the conclusions of studies on the prevalence of foundational vulnerabilities. | ||
| + | |||
| + | If you are interested in getting the code for the analysis please let us know. | ||
| + | |||
| + | |||
| + | ===== Effort of security maintenance of FOSS components ===== | ||
| + | |||
| + | In our paper we investigated publicly available factors (from number of active users to commits, from code size to usage of popular programming languages, etc.) to identify which ones impact three potential effort models: Centralized (the company checks each component and propagates changes to the product groups), Distributed (each product group is in charge of evaluating and fixing its consumed FOSS components), and Hybrid (seldom used components are checked individually by each development team, the rest is centralized). | ||
| + | |||
| + | We use Grounded Theory to extract the factors from a six months study at the vendor and report the results on a sample of 152 FOSS components used by the vendor. | ||
| + | |||
| + | ===== Which static analyzer performs best on a particular FOSS project? ===== | ||
| + | |||
| + | Our {{:esem-final.pdf|paper in proceedings of International Symposium on Empirical Software Engineering and Measurement}} addresses the limitations of the existing static analysis security testing (SAST) tool benchmarks: lack of vulnerability realism, uncertain ground truth, and large amount of findings not related to analyzed vulnerability. | ||
| + | |||
| + | We propose **Delta-Bench** – a novel approach for the automatic construction of benchmarks for SAST tools based on | ||
| + | differencing vulnerable and fixed versions in Free and Open Source (FOSS) repositories. I.e., Delta-Bench allows SAST tools to be automatically evaluated on the real-world historical vulnerabilities using only the findings that a tool produced for the analyzed vulnerability. | ||
| + | |||
| + | We applied our approach to test 7 state of the art SAST tools against 70 revisions of four major versions of Apache Tomcat spanning 62 distinct Common Vulnerabilities and Exposures (CVE) fixes and vulnerable files totalling over | ||
| + | 100K lines of code as the source of ground truth vulnerabilities. | ||
| + | |||
| + | The most interesting finding we have - tools perform differently due to the selected benchmark. | ||
| + | |||
| + | The Delta-Bench was awarded silver medal in the ESEC/FSE 2017 Graduate Student Research Competition: {{https://drive.google.com/file/d/0B_rJCkKmzPjSWllQcEJpQWNOOVU/view?usp=sharing|Author's PDF}} or {{https://doi.org/10.1145/3106237.3121276|Publisher's Version}} | ||
| + | |||
| + | Let us know if you want us to select a SAST tool that suits to your needs. | ||
| Line 157: | Line 208: | ||
| ===== Talks and Tutorials ===== | ===== Talks and Tutorials ===== | ||
| - | * Ivan Pashchenko, Stanislav Dashevskyi, Fabio Massacci //Delta–Bench: Differential Benchmark for Security Testing.// Poster at ESSoS 2017. Bonn, Germany, July 2017. {{:https://drive.google.com/file/d/1js_nNe8vIYMatb-PhKijtxAmZCC-HyQA/view?usp=sharing|Poster}} | + | * Ivan Pashchenko, Stanislav Dashevskyi, Fabio Massacci //Delta–Bench: Differential Benchmark for Security Testing.// Poster at ESSoS 2017. Bonn, Germany, July 2017. {{https://drive.google.com/file/d/1js_nNe8vIYMatb-PhKijtxAmZCC-HyQA/view?usp=sharing|Poster}} |
| - | * Ivan Pashchenko, Stanislav Dashevskyi, Fabio Massacci //Design of a benchmark for static analysis security testing tools.// Presentation at ESSoS Doctoral Symposium 2016. London, UK, Apr 2016. {{:https://drive.google.com/open?id=0B_rJCkKmzPjSek90aklLY0lheXc|Slides}} | + | * Ivan Pashchenko, Stanislav Dashevskyi, Fabio Massacci //Design of a benchmark for static analysis security testing tools.// Presentation at ESSoS Doctoral Symposium 2016. London, UK, Apr 2016. {{https://drive.google.com/file/d/0B_rJCkKmzPjSek90aklLY0lheXc/view?usp=sharing|Slides}} |
| * Luca Allodi //Internet-scale vulnerability risk assessment (Extended Abstract).// Presentation at Usenix Security LEET 2013. Washington D.C., USA, Aug 2013. {{:presentation-leet-13-pdf.pdf|Slides}} | * Luca Allodi //Internet-scale vulnerability risk assessment (Extended Abstract).// Presentation at Usenix Security LEET 2013. Washington D.C., USA, Aug 2013. {{:presentation-leet-13-pdf.pdf|Slides}} | ||
| * Luca Allodi. //Quantitative assessment of risk reduction with cybercrime black market monitoring//. Talk at IEEE SS&P IWCC 2013, San Francisco, CA, USA. {{:iwcc-13.pdf|Slides}} | * Luca Allodi. //Quantitative assessment of risk reduction with cybercrime black market monitoring//. Talk at IEEE SS&P IWCC 2013, San Francisco, CA, USA. {{:iwcc-13.pdf|Slides}} | ||
vulnerability_discovery_models.1513194336.txt.gz · Last modified: 2021/01/29 10:58 (external edit)
Page Tools
