Fixing Google GData Library (1+)

kim — Sun, 01 Apr 2012 22:53:03 +0000

Google has many products and projects. The problem is that for many projects, the current version is deprecated and the new one not ready yet. For GData, Google’s library to access core API infrastructure such as Google’s project hosting services is not getting maintained anymore. For me this is a major drawback since APIs are the better way to access software project repositories than parsing HTML.

Currently we are working on a mining framework called “mozkito” that will combine all our mining techniques including bug tracker parsing and processing. Parsing bug reports is easy when you have working APIs at hand but tedious when you have to parse the human target front end—basically you have to parse HTML. Now, parsing public Google bug trackers from code.google.com seems to be easy since you can use the Google’s Project Hosting JAVA API to access the content of each issue tracker programmatically. So far in theory. In practice this work well unless you want to combine your mining tool with libraries that depend on a current version of Google’s Guava library.

Now, to understand the problem, we have to understand the history of the GUAVA library. GUAVA is a set of libraries developed by Google. Before GUAVA was released, the single libraries now contained in GUAVA were released separately. So far so good. The basic problem concerning Google’s project hosting library (we will use the acronym CODE LIB for the remainder of this text) is that CODE LIB depends the former and now deprecated Google collections library (short GCOLLECTIONS) instead of it’s replacement library GUAVA. If your own project (like mine) depends (direct or inherited) on GUAVA then you will run into the problem, that GUAVA and GCOLLECTIONS need to be put on the same CLASSPATH. Although GUAVA should be fully backward compatible with GCOLLECTIONS it turns out that this is simply not the case. The GCOLLECTIONS class ImmutableSet is not compatible with it’s corresponding class in GUAVA missing a method definition required for CODE LIB.

Since Google does not maintain multiple of their central API libraries anymore, this issue is unresolved since quite some time. The issue reports covering this issue are old but still unresolved. Well, let’s do the job for Google then. We won’t get the Google salary for this but at least we can continue parsing their deprecated and unmaintained services.

For those of you who do not care about the solution but simply need a version that does the trick: here you find the GUAVA.MOZKITO version and here the modified GDATA-CORE.GUAVA and GDATA-PROJECTHOSTING.GUAVA library. All modified libraries come with no warranty nor guarantees and are as they are. Maven users can use our maven nexus instance https://nexus.own-hero.net/ as repository and adding the following dependencies to their pom.xml:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

>
>com.google.gdata>
>gdata-core-1.0>
>1.41.5.guava>
>
>
>com.google.gdata>
>gdata-projecthosting-2.1>
>1.41.5.guava>
>
>
>com.google.guava>
>guava>
>11.0.2-mozkito>
>

Those of you who want to see the solution on detail just have to read the remainder. The solution has two major step: first we have to bug fix the ImmutableSet class GUAVA before using the revised version of GUAVA to build a new version of GDATA (containing CODE LIB).

Fixing GUAVA

Now first, clone the current version of GDATA from Google using git:

1	git clone https://code.google.com/p/guava-libraries/

Then, we have to path the class ImmutableSet. This class contains a number of unusual classes defining the same method over and over again just using a different number of arguments all having the same type before defining the method with a variable set of arguments.

The code looks like this (stripped JavaDoc):

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47

public abstract class ImmutableSet<E> extends ImmutableCollection<E> implements Set<E> {

public static <E> ImmutableSet<E> of() {
return (ImmutableSet<E>) EmptyImmutableSet.INSTANCE;
}

public static <E> ImmutableSet<E> of(E element) {
return new SingletonImmutableSet<E>(element);
}

public static <E> ImmutableSet<E> of(E e1, E e2) {
return construct(e1, e2);
}

public static <E> ImmutableSet<E> of(E e1, E e2, E e3) {
return construct(e1, e2, e3);
}

public static <E> ImmutableSet<E> of(E e1, E e2, E e3, E e4) {
return construct(e1, e2, e3, e4);
}

public static <E> ImmutableSet<E> of(E e1, E e2, E e3, E e4, E e5) {
return construct(e1, e2, e3, e4, e5);
}

public static <E> ImmutableSet<E> of(E e1, E e2, E e3, E e4, E e5, E e6,
E... others) {
final int paramCount = 6;
Object[] elements = new Object[paramCount + others.length];
elements[0] = e1;
elements[1] = e2;
elements[2] = e3;
elements[3] = e4;
elements[4] = e5;
elements[5] = e6;
for (int i = paramCount; i < elements.length; i++) {
elements[i] = others[i - paramCount];
}
return construct(elements);
}

private static <E> ImmutableSet<E> construct(Object... elements) {
....
}
....
}

The reason for methods like on line 19

1	public static <E> ImmutableSet<E> of(E e1, E e2, E e3, E e4)

when having a method like on line 43:

1	private static <E> ImmutableSet<E> construct(Object... elements)

is due to type safety. Let’s make things simple and remove all the existing of() methods and add a method whose signature stems from google-collections:

1
2
3

public static <E> ImmutableSet<E> of(E... elements) {
return construct(elements);
}

The patch we applied is available here: http://www.kim-herzig.de/wp-content/uploads/2012/04/ImmutableSet.diff_.zip. This should do the trick. Next, we have to compile the modified GUAVA. Which again can be complicated. On our machine, we had to compile GUAVA using JDK1.7 because the maven build system caused an JDK1.6 compiler to fail silently resulting in a compiled GUAVA that contains not all required and necessary class files. So switch to JDK1.7 and then run

1	mvn clean mvn clean package source:jar javadoc:jar

In guava/target/ you find the newly compiled GUAVA library. We strongly recommend to rename this version of GUAVA to indicate that this is your self-modified version of GUAVA. For the remainder of this text we call the library jar file: guava-11.0.2-mozkito.jar.

Fixing GDATA

The second phase of this Google code fixing session is more easy. Basically, we have to rebuild the GDATA library replacing it’s dependency on GCOLLECTIONS with our newly backed guava-11.0.2-mozkito.jar.

First, download the latest version of GDATA (currently 1.46.0) and unzip the downloaded file. Unzipping GDATA yields a set of directories and files similar to this:

1
2
3
4
5
6
7
8
9
10
11
12
13
14

kim@grid1 /path/to/gdata/java $ ls -llah
total 52K
drwxr-xr-x 11 kim staff 4.0K Mar 28 18:11 ./
drwxr-xr-x 3 kim staff 4.0K Mar 9 16:38 ../
drwxr-x--- 2 kim staff 4.0K Mar 28 18:02 build-src/
-rw------- 1 kim staff 3.7K Sep 9 2011 build-src.xml
drwxr-xr-x 25 kim staff 4.0K Mar 28 18:12 classes/
drwxr-x--- 3 kim staff 4.0K Mar 28 18:01 deps/
drwxr-x--- 4 kim staff 4.0K Sep 9 2011 doc/
drwxr-xr-x 4 kim staff 4.0K Mar 9 17:49 genfiles/
drwxr-x--- 2 kim staff 4.0K Mar 28 18:12 lib/
drwxr-x--- 2 kim staff 4.0K Sep 6 2011 manifest/
drwxr-x--- 3 kim staff 4.0K Sep 9 2011 src/
-rw------- 1 kim staff 1.3K Sep 9 2011 version.properties

To replace the dependency of GCOLLECTIONS with our own GUAVA version we have to place the guava-11.0.2-mozkito.jar file into the deps folder. To make things rock solid, we deleted the GCOLLECTIONS jar file from the deps directory.

1
2
3
4
5
6
7

kim@grid1 /path/to/gdata/java $ ls -llah deps/
total 2.9M
drwxr-x--- 3 kim staff 4.0K Mar 28 18:01 ./
drwxr-xr-x 11 kim staff 4.0K Mar 28 18:11 ../
-rw-r--r-- 1 kim staff 1.6M Mar 9 18:01 guava-11.0.2-mozkito.jar
-rw------- 1 kim staff 33K Sep 6 2011 jsr305.jar
drwxr-xr-x 3 kim staff 4.0K Feb 22 09:01 META-INF/

Next, we have to tell the build system to use OUR own GUAVA instead of GCOLLECTIONS. Open the file build-src/build.properties and replace the line google-collect.jar=deps/google-collect-1.0-rc1.jar on line 13 referring to our own GUAVA version:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18

# Points to a external library dependancies
# EDIT-THIS: If rebuilding the authsub or gbase/recipe sample point to,
# servlet jar in Sun's Servlet API library.
servlet.jar=/tmp/gdata_dep/servlet-api-2.4.jar
# EDIT-THIS: Point to mail.jar lib in Sun's Java Mail API.
mail.jar=/tmp/gdata_dep/mail.jar
# EDIT-THIS: If using version older than JDK 1.6,
# Point to activation.jar in Sun's activation framework library.
activation.jar=/tmp/gdata_dep/activation.jar

# EDIT-THIS: Point to google-collect lib
#google-collect.jar=deps/google-collect-1.0-rc1.jar
google-guava.jar=deps/guava-11.0.2.jar
google-jsr305.jar=deps/jsr305.jar

# Include debugging information in built library files. Possible values "on" or "off"
javac.debug=true
javac.debuglevel=lines,vars,source

This should be it. Compile you own version of GDATA using the following commands (requires Apache ANT.

1	ant clean build -f build-src.xml -lib deps

Done. Now, you have your own fixed version of GUAVA and GDATA and can do you own job.

Stacked Bar Plots with GGPlot2

kim — Wed, 14 Mar 2012 17:54:48 +0000

When plotting contribution shares, pie charts seem to be logic. But pie charts look ugly and do not add much more information than presenting the data in a table. But there is an alterative: stacker bar charts. In R you can use the ggplot2 package to plot them easily. Below, you find the very same data plotted using two different layout. Left the stacked bar chart. Right you see the stacked bar chart using polar coordinates.

To reproduce both charts, copy and paste the following code to your R-Shell.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16

library(RColorBrewer)
library(ggplot2)

project <- c('ArgoUML','ArgoUML','ArgoUML','ArgoUML','ArgoUML','GWT','GWT','GWT','GWT','GWT','Jaxen','Jaxen','Jaxen','Jaxen','Jaxen','JRuby','JRuby','JRuby','JRuby','JRuby')

component <- c('FileDistance','PackageDistance','DataDependency','CallGraphDistance','ChangeCouplings','FileDistance','PackageDistance','DataDependency','CallGraphDistance','ChangeCouplings','FileDistance','PackageDistance','DataDependency','CallGraphDistance','ChangeCouplings','FileDistance','PackageDistance','DataDependency','CallGraphDistance','ChangeCouplings')

numbers <- c(16,22,46,10,6,46,22,10,12,10,20,20,20,20,20,0,36,33,20,11)

df <- data.frame(project,component,numbers)

#plot the stacked bar plot
ggplot(df, aes(x = project)) + geom_bar(aes(weight=numbers, fill = component), position = 'fill') + scale_y_continuous("", breaks=NA) + scale_fill_manual(values = rev(brewer.pal(6, "Purples")))

#plot the stacked bar plot with polar coordinates
ggplot(df, aes(x = project)) + geom_bar(aes(weight=numbers, fill = component), position = 'fill') + scale_y_continuous("", breaks=NA) + scale_fill_manual(values = rev(brewer.pal(6, "Purples"))) + coord_polar()

More in the coolest plotting framework ever on http://had.co.nz/ggplot2/

Spell checking LaTeX Documents

kim — Sat, 29 Jan 2011 11:17:29 +0000

LaTeX is great and there is no better environment to write professional documents. But spell-checking text within LaTeX is difficult and all projects trying to solve this issue failed at various levels. One of the tools to check for spelling and grammar errors is Microsoft Word. To use MS Word to spell and grammar check LaTeX documents, one has to convert the LaTeX documents to a format readable by MS Word: Rich Text Format (RTF). The latex2rtf project at Sourceforge is transforming LaTeX document to RTF documents. Opening the RTF document using Word allows you to spell and grammar check your document.

There are some limitations to latex2rtf. It cannot handle LaTeX environments like

1	\usepackage{boxenminipage}

. For more advanced LaTeX documents I normally stick to Acrobat Professional converting the compiled PDF documents into RTF or MS Word format directly.

» Blog

Fixing Google GData Library (1+)

Fixing GUAVA

Fixing GDATA

Stacked Bar Plots with GGPlot2

Spell checking LaTeX Documents