Fuzzing with Code Fragments @USENIX Security 2012

kim — Fri, 27 Apr 2012 08:03:10 +0000

Fuzz testing is an automated technique providing random data as input to a software system in the hope to expose a vulnerability. In order to be effective, the fuzzed input must be common enough to pass elementary consistency checks; a JavaScript interpreter, for instance, would only accept a semantically valid program. On the other hand, the fuzzed input must be uncommon enough to trigger exceptional behavior, such as a crash of the interpreter. The LangFuzz approach resolves this conflict by using a grammar to randomly generate valid programs; the code fragments, however, partially stem from programs known to have caused invalid behavior before. Lang- Fuzz is an effective tool for security testing. Applied on the Mozilla JavaScript interpreter, it discovered a total of 105 new severe vulnerabilities within three months of operation (and thus became one of the top security bug bounty collectors within this period); applied on the PHP interpreter, it discovered 18 new defects causing crashes.

Reference

[2012] C. Holler, K. Herzig, and A. Zeller, "Fuzzing with Code Fragments", in Proceedings of the 21st Usenix Security Symposium, 2012.

BibTeX Entry

@inproceedings{holler-usenix-2012,
 title = {Fuzzing with Code Fragments}, 
   author={Christian Holler and Kim Herzig and Andreas Zeller}, 
   year = {2012}, 
   month = {August}, 
   booktitle={Proceedings of the 21st Usenix Security Symposium}, 
   }

Fixing Google GData Library (1+)

kim — Sun, 01 Apr 2012 22:53:03 +0000

Google has many products and projects. The problem is that for many projects, the current version is deprecated and the new one not ready yet. For GData, Google’s library to access core API infrastructure such as Google’s project hosting services is not getting maintained anymore. For me this is a major drawback since APIs are the better way to access software project repositories than parsing HTML.

Currently we are working on a mining framework called “mozkito” that will combine all our mining techniques including bug tracker parsing and processing. Parsing bug reports is easy when you have working APIs at hand but tedious when you have to parse the human target front end—basically you have to parse HTML. Now, parsing public Google bug trackers from code.google.com seems to be easy since you can use the Google’s Project Hosting JAVA API to access the content of each issue tracker programmatically. So far in theory. In practice this work well unless you want to combine your mining tool with libraries that depend on a current version of Google’s Guava library.

Now, to understand the problem, we have to understand the history of the GUAVA library. GUAVA is a set of libraries developed by Google. Before GUAVA was released, the single libraries now contained in GUAVA were released separately. So far so good. The basic problem concerning Google’s project hosting library (we will use the acronym CODE LIB for the remainder of this text) is that CODE LIB depends the former and now deprecated Google collections library (short GCOLLECTIONS) instead of it’s replacement library GUAVA. If your own project (like mine) depends (direct or inherited) on GUAVA then you will run into the problem, that GUAVA and GCOLLECTIONS need to be put on the same CLASSPATH. Although GUAVA should be fully backward compatible with GCOLLECTIONS it turns out that this is simply not the case. The GCOLLECTIONS class ImmutableSet is not compatible with it’s corresponding class in GUAVA missing a method definition required for CODE LIB.

Since Google does not maintain multiple of their central API libraries anymore, this issue is unresolved since quite some time. The issue reports covering this issue are old but still unresolved. Well, let’s do the job for Google then. We won’t get the Google salary for this but at least we can continue parsing their deprecated and unmaintained services.

For those of you who do not care about the solution but simply need a version that does the trick: here you find the GUAVA.MOZKITO version and here the modified GDATA-CORE.GUAVA and GDATA-PROJECTHOSTING.GUAVA library. All modified libraries come with no warranty nor guarantees and are as they are. Maven users can use our maven nexus instance https://nexus.own-hero.net/ as repository and adding the following dependencies to their pom.xml:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

>
>com.google.gdata>
>gdata-core-1.0>
>1.41.5.guava>
>
>
>com.google.gdata>
>gdata-projecthosting-2.1>
>1.41.5.guava>
>
>
>com.google.guava>
>guava>
>11.0.2-mozkito>
>

Those of you who want to see the solution on detail just have to read the remainder. The solution has two major step: first we have to bug fix the ImmutableSet class GUAVA before using the revised version of GUAVA to build a new version of GDATA (containing CODE LIB).

Fixing GUAVA

Now first, clone the current version of GDATA from Google using git:

1	git clone https://code.google.com/p/guava-libraries/

Then, we have to path the class ImmutableSet. This class contains a number of unusual classes defining the same method over and over again just using a different number of arguments all having the same type before defining the method with a variable set of arguments.

The code looks like this (stripped JavaDoc):

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47

public abstract class ImmutableSet<E> extends ImmutableCollection<E> implements Set<E> {

public static <E> ImmutableSet<E> of() {
return (ImmutableSet<E>) EmptyImmutableSet.INSTANCE;
}

public static <E> ImmutableSet<E> of(E element) {
return new SingletonImmutableSet<E>(element);
}

public static <E> ImmutableSet<E> of(E e1, E e2) {
return construct(e1, e2);
}

public static <E> ImmutableSet<E> of(E e1, E e2, E e3) {
return construct(e1, e2, e3);
}

public static <E> ImmutableSet<E> of(E e1, E e2, E e3, E e4) {
return construct(e1, e2, e3, e4);
}

public static <E> ImmutableSet<E> of(E e1, E e2, E e3, E e4, E e5) {
return construct(e1, e2, e3, e4, e5);
}

public static <E> ImmutableSet<E> of(E e1, E e2, E e3, E e4, E e5, E e6,
E... others) {
final int paramCount = 6;
Object[] elements = new Object[paramCount + others.length];
elements[0] = e1;
elements[1] = e2;
elements[2] = e3;
elements[3] = e4;
elements[4] = e5;
elements[5] = e6;
for (int i = paramCount; i < elements.length; i++) {
elements[i] = others[i - paramCount];
}
return construct(elements);
}

private static <E> ImmutableSet<E> construct(Object... elements) {
....
}
....
}

The reason for methods like on line 19

1	public static <E> ImmutableSet<E> of(E e1, E e2, E e3, E e4)

when having a method like on line 43:

1	private static <E> ImmutableSet<E> construct(Object... elements)

is due to type safety. Let’s make things simple and remove all the existing of() methods and add a method whose signature stems from google-collections:

1
2
3

public static <E> ImmutableSet<E> of(E... elements) {
return construct(elements);
}

The patch we applied is available here: http://www.kim-herzig.de/wp-content/uploads/2012/04/ImmutableSet.diff_.zip. This should do the trick. Next, we have to compile the modified GUAVA. Which again can be complicated. On our machine, we had to compile GUAVA using JDK1.7 because the maven build system caused an JDK1.6 compiler to fail silently resulting in a compiled GUAVA that contains not all required and necessary class files. So switch to JDK1.7 and then run

1	mvn clean mvn clean package source:jar javadoc:jar

In guava/target/ you find the newly compiled GUAVA library. We strongly recommend to rename this version of GUAVA to indicate that this is your self-modified version of GUAVA. For the remainder of this text we call the library jar file: guava-11.0.2-mozkito.jar.

Fixing GDATA

The second phase of this Google code fixing session is more easy. Basically, we have to rebuild the GDATA library replacing it’s dependency on GCOLLECTIONS with our newly backed guava-11.0.2-mozkito.jar.

First, download the latest version of GDATA (currently 1.46.0) and unzip the downloaded file. Unzipping GDATA yields a set of directories and files similar to this:

1
2
3
4
5
6
7
8
9
10
11
12
13
14

kim@grid1 /path/to/gdata/java $ ls -llah
total 52K
drwxr-xr-x 11 kim staff 4.0K Mar 28 18:11 ./
drwxr-xr-x 3 kim staff 4.0K Mar 9 16:38 ../
drwxr-x--- 2 kim staff 4.0K Mar 28 18:02 build-src/
-rw------- 1 kim staff 3.7K Sep 9 2011 build-src.xml
drwxr-xr-x 25 kim staff 4.0K Mar 28 18:12 classes/
drwxr-x--- 3 kim staff 4.0K Mar 28 18:01 deps/
drwxr-x--- 4 kim staff 4.0K Sep 9 2011 doc/
drwxr-xr-x 4 kim staff 4.0K Mar 9 17:49 genfiles/
drwxr-x--- 2 kim staff 4.0K Mar 28 18:12 lib/
drwxr-x--- 2 kim staff 4.0K Sep 6 2011 manifest/
drwxr-x--- 3 kim staff 4.0K Sep 9 2011 src/
-rw------- 1 kim staff 1.3K Sep 9 2011 version.properties

To replace the dependency of GCOLLECTIONS with our own GUAVA version we have to place the guava-11.0.2-mozkito.jar file into the deps folder. To make things rock solid, we deleted the GCOLLECTIONS jar file from the deps directory.

1
2
3
4
5
6
7

kim@grid1 /path/to/gdata/java $ ls -llah deps/
total 2.9M
drwxr-x--- 3 kim staff 4.0K Mar 28 18:01 ./
drwxr-xr-x 11 kim staff 4.0K Mar 28 18:11 ../
-rw-r--r-- 1 kim staff 1.6M Mar 9 18:01 guava-11.0.2-mozkito.jar
-rw------- 1 kim staff 33K Sep 6 2011 jsr305.jar
drwxr-xr-x 3 kim staff 4.0K Feb 22 09:01 META-INF/

Next, we have to tell the build system to use OUR own GUAVA instead of GCOLLECTIONS. Open the file build-src/build.properties and replace the line google-collect.jar=deps/google-collect-1.0-rc1.jar on line 13 referring to our own GUAVA version:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18

# Points to a external library dependancies
# EDIT-THIS: If rebuilding the authsub or gbase/recipe sample point to,
# servlet jar in Sun's Servlet API library.
servlet.jar=/tmp/gdata_dep/servlet-api-2.4.jar
# EDIT-THIS: Point to mail.jar lib in Sun's Java Mail API.
mail.jar=/tmp/gdata_dep/mail.jar
# EDIT-THIS: If using version older than JDK 1.6,
# Point to activation.jar in Sun's activation framework library.
activation.jar=/tmp/gdata_dep/activation.jar

# EDIT-THIS: Point to google-collect lib
#google-collect.jar=deps/google-collect-1.0-rc1.jar
google-guava.jar=deps/guava-11.0.2.jar
google-jsr305.jar=deps/jsr305.jar

# Include debugging information in built library files. Possible values "on" or "off"
javac.debug=true
javac.debuglevel=lines,vars,source

This should be it. Compile you own version of GDATA using the following commands (requires Apache ANT.

1	ant clean build -f build-src.xml -lib deps

Done. Now, you have your own fixed version of GUAVA and GDATA and can do you own job.

Stacked Bar Plots with GGPlot2

kim — Wed, 14 Mar 2012 17:54:48 +0000

When plotting contribution shares, pie charts seem to be logic. But pie charts look ugly and do not add much more information than presenting the data in a table. But there is an alterative: stacker bar charts. In R you can use the ggplot2 package to plot them easily. Below, you find the very same data plotted using two different layout. Left the stacked bar chart. Right you see the stacked bar chart using polar coordinates.

To reproduce both charts, copy and paste the following code to your R-Shell.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16

library(RColorBrewer)
library(ggplot2)

project <- c('ArgoUML','ArgoUML','ArgoUML','ArgoUML','ArgoUML','GWT','GWT','GWT','GWT','GWT','Jaxen','Jaxen','Jaxen','Jaxen','Jaxen','JRuby','JRuby','JRuby','JRuby','JRuby')

component <- c('FileDistance','PackageDistance','DataDependency','CallGraphDistance','ChangeCouplings','FileDistance','PackageDistance','DataDependency','CallGraphDistance','ChangeCouplings','FileDistance','PackageDistance','DataDependency','CallGraphDistance','ChangeCouplings','FileDistance','PackageDistance','DataDependency','CallGraphDistance','ChangeCouplings')

numbers <- c(16,22,46,10,6,46,22,10,12,10,20,20,20,20,20,0,36,33,20,11)

df <- data.frame(project,component,numbers)

#plot the stacked bar plot
ggplot(df, aes(x = project)) + geom_bar(aes(weight=numbers, fill = component), position = 'fill') + scale_y_continuous("", breaks=NA) + scale_fill_manual(values = rev(brewer.pal(6, "Purples")))

#plot the stacked bar plot with polar coordinates
ggplot(df, aes(x = project)) + geom_bar(aes(weight=numbers, fill = component), position = 'fill') + scale_y_continuous("", breaks=NA) + scale_fill_manual(values = rev(brewer.pal(6, "Purples"))) + coord_polar()

More in the coolest plotting framework ever on http://had.co.nz/ggplot2/

―エビデンスが変えるソフトウェア開発

kim — Sun, 20 Nov 2011 21:41:26 +0000

The book Making Software is published now also available in Japan language. The article written by Andreas Zeller and myself is translated as 自力でのエビデンスマイニング (Mine your own evidence).

Thanks to O’Reilly Japan and the translators for translating this great book.

Mining Cause-Effect-Chains from Version Histories – ISSRE 2011

kim — Wed, 28 Sep 2011 14:44:43 +0000

Software reliability is heavily impacted by soft- ware changes. How do these changes relate to each other? By analyzing the impacted method definitions and usages, we determine dependencies between changes, resulting in a change genealogy that captures how earlier changes enable and cause later ones. Model checking this genealogy reveals temporal process patterns that encode key features of the software process such as pending development activities: “Whenever class A is changed, its test case is later updated as well.” Such patterns can be validated automatically: In an evaluation of four open source histories, our prototype would recommend pending activities with a precision of 60–72%.

Reference

[2011] K. Herzig and A. Zeller, "Mining Cause-Effect-Chains from Version Histories", in Proceedings of the 22nd International Symposium on Software Reliability Engineering, 2011.

BibTeX Entry

@inproceedings{herzig-issre-2011,
 title = {Mining Cause-Effect-Chains from Version Histories}, 
   author={Kim Herzig and Andreas Zeller}, 
   year = {2011}, 
   month = {November}, 
   booktitle={Proceedings of the 22nd International Symposium on Software Reliability Engineering} 
   }

Mining Cause Effect Chains from Version Archives – ISSRE 2011

View more presentations from Kim Herzig.

Network versus Code Metrics to Predict Defects: A Replication Study – ESEM 2011

kim — Sat, 21 May 2011 15:41:00 +0000

Several defect prediction models have been proposed to identify which entities in a software system are likely to have defects before its release. This paper presents a replication of one such study conducted by Zimmermann and Nagappan [1] on Windows Server 2003 where the authors leveraged dependency relationships between software entities captured using social network metrics to predict whether they are likely to have defects. They found that network metrics perform significantly better than source code metrics at predicting defects. In order to corroborate the generality of their findings, we replicate their study on three open source Java projects, viz., JRuby, ArgoUML, and Eclipse. Our results are in agreement with the original study by Zimmermann and Nagappan when using a similar experimental setup as them (random sampling). However, when we evaluated the metrics using setups more suited for industrial use – forward-release and cross-project prediction – we found network metrics to offer no vantage over code metrics. Moreover, code metrics may be preferable to network metrics considering the data is easier to collect and we used only 8 code metrics compared to approximately 58 network metrics.

Reference

[2011] R. Premraj and K. Herzig, "Network versus Code Metrics to Predict Defects: A Replication Study", in Proceedings of the Fifth International Symposium on Empirical Software Engineering and Measurement, 2011.

BibTeX Entry

@inproceedings{premraj-esem-2011,
 title = {Network versus Code Metrics to Predict Defects: A Replication Study}, 
   author={Premraj, Rahul and Herzig, Kim}, 
   year = {2011}, 
   month = {september}, 
   booktitle={Proceedings of the Fifth International Symposium on Empirical Software Engineering and Measurement}
}

Download PDF version

Network vs. Code Metrics to Predict Defects: A Replication Study

View more presentations from Kim Herzig.

Spell checking LaTeX Documents

kim — Sat, 29 Jan 2011 11:17:29 +0000

LaTeX is great and there is no better environment to write professional documents. But spell-checking text within LaTeX is difficult and all projects trying to solve this issue failed at various levels. One of the tools to check for spelling and grammar errors is Microsoft Word. To use MS Word to spell and grammar check LaTeX documents, one has to convert the LaTeX documents to a format readable by MS Word: Rich Text Format (RTF). The latex2rtf project at Sourceforge is transforming LaTeX document to RTF documents. Opening the RTF document using Word allows you to spell and grammar check your document.

There are some limitations to latex2rtf. It cannot handle LaTeX environments like

1	\usepackage{boxenminipage}

. For more advanced LaTeX documents I normally stick to Acrobat Professional converting the compiled PDF documents into RTF or MS Word format directly.

An Empirical Study of the Factors Relating Field Failures and Dependencies – ICST 2011

kim — Tue, 30 Nov 2010 08:31:26 +0000

Changing source code in large software systems is complex and requires a good understanding of dependencies between software components. Modification to components with little regard to dependencies may have an adverse impact on the quality of the latter, i.e., increase their risk to fail. We conduct an empirical study to understand the relationship between the quality of components and the characteristics of their dependencies such as their frequency of change, their complexity, number of past failures and the like. Our study has been conducted on two large software systems: Microsoft VISTA and ECLIPSE. Our results show that components that have outgoing dependencies to components with higher object-oriented complexity tend to have fewer field failures for VISTA, but the opposite relation holds for ECLIPSE. Likewise, other notable observations have been made through our study that (a) confirm that certain characteristics of components increase the risk of their dependencies to fail and (b) some of the characteristics are project specific while some were also found to be common. We expect that such results can be leveraged for use to provide new directions for research in defect prediction, test prioritization and related research fields that utilize code dependencies in their empirical analysis. Additionally, these results provide insights to engineers on the potential reliability impacts of new component dependencies based upon the characteristics of the component.

Reference

[2011] T. Zimmerman, N. Nagappan, K. Herzig, R. Premraj, and L. Williams, "An Empirical Study of the Factors Relating Field Failures and Dependencies", in Software Testing, Verification and Validation (ICST), 2011 IEEE Fourth International Conference on Software Testing, Verification and Validation, 2011, pp. 347-356.

BibTeX Entry

@inproceedings{zimmermann-icst-2011,
 title = {An Empirical Study of the Factors Relating Field Failures and Dependencies}, 
   author={Zimmerman, Thomas and Nagappan, Nachiappan and Herzig, Kim and Premraj, Rahul and Williams, Laurie}, 
   year = {2011}, 
   month = {march}, 
   booktitle={Software Testing, Verification and Validation (ICST), 2011 IEEE Fourth International Conference on Software Testing, Verification and Validation}
 pages={347--356}, 
   doi={10.1109/ICST.2011.39}
}

Download PDF version

Factors Relating Field Failures and Dependencies – ICST 2011

View more presentations from kim.herzig.

Mining Your Own Evidence – Making Software

kim — Fri, 29 Oct 2010 13:52:47 +0000

Throughout this book, you will find examples of how to gather evidence—evidence on the effectiveness of testing, the quality of bug reports, the role of complexity metrics, and so on. But do these findings actually apply to your project? The definite way to find this out is to repeat the appropriate study on your data, in your environment. This way, you will not only gather lots of insight into your own project; you will also experience the joys of experimental research. Unfortunately, you may also encounter the downside: empirical studies can be very expensive, in particular if they involve experiments with developers.

Fortunately, there is a relatively inexpensive way to gather lots of evidence about your project. Software archives, such as version or bug repositories, record much of the activity around your product, in terms of problems occurring, changes made, and problems fixed. By mining these archives automatically, you can obtain lots of initial evidence about your product—evidence that already is worthy in itself, but which may also pave the path toward further experiments and further insights. In this chapter, we give a hands-on tutorial into mining software archives, covering both the basic technical steps and possible pitfalls that you may encounter on the way.

Reference

[2010] K. Herzig and A. Zeller, "Mining Your Own Evidence." O’Reilly Media, 2010, pp. 517-529.

BibTeX Entry

@inbook{herzig-makingsoftware-2010,
 title = "Mining Your Own Evidence",
 author = "Kim Herzig and Andreas Zeller",
 year = "2010",
 month = "October",
 booktitle = "Making Software: What Really Works, and Why We Believe It",
 chapter = "Mining Your Own Evidence",
 editors = "Andy Oram and Greg Wilson",
 pages = "517--529",
 publisher = "O'Reilly Media",
 ISBN = "9780596808327"
}

Funded by Google Research Award

kim — Tue, 07 Sep 2010 14:25:52 +0000

Google rewarded Andreas Zeller‘s research proposal was successful. Google rewarded our chair with a Google Research Award. The money funds my new research project conducted with and within Google.

Fuzzing with Code Fragments @USENIX Security 2012

Reference

BibTeX Entry

Fixing Google GData Library (1+)

Fixing GUAVA

Fixing GDATA

Stacked Bar Plots with GGPlot2

―エビデンスが変えるソフトウェア開発

Mining Cause-Effect-Chains from Version Histories – ISSRE 2011

Reference

BibTeX Entry

Network versus Code Metrics to Predict Defects: A Replication Study – ESEM 2011

Reference

BibTeX Entry

Spell checking LaTeX Documents

An Empirical Study of the Factors Relating Field Failures and Dependencies – ICST 2011

Reference

BibTeX Entry

Mining Your Own Evidence – Making Software

See also

Reference

BibTeX Entry

Funded by Google Research Award