JDOM2 and Android

This page details both how the JDOM 2.x code is tested on Android, and how to include JDOM2 in your Android project.

Testing JDOM in Android

Android has a few built-in mechanisms for testing code. I opted to use the most rigorous mechanism I could conceive, which is to actually run the full JDOM test harness on an (emulated) Android device. There are a few challenges with this approach:

0. Android uses JUnit3 as a 'foundation' for unit testing, and all the JDOM unit tests are written in JUnit4
1. A number of JDOM tests read and parse XML 'files' (and DTD's, and XSD's, etc.) and the existing mechanisms in the JDOM tests for accessing these files is 'broken' on Android ClassLoader.getSystemResource().
2. JDOM has 1800 unit tests, but the eclipse/android test interface cannot process more than a few hunded at a time, before the 'RPC' channel overflows and you get a Failed Binder Transaction.
3. Android has some missing functionality which is/was directly referenced by JDOM Unit tests (ResourceBundles)
4. Android's XML parser is different to Xerces and produces different results.

All these issues haven been worked around in some manner...

A Brief WalkThrough

Android testing is set up with two Android projects. The first project is the 'real' project, and the second project is the 'Test' project. The two projects are linked using Android project references. You build the two projects in to seperate android 'apps', and then install both apps on to the Android device. After they are installed, you then run the tests on the device. The test results are returned to the 'calling' process.

A special note must be made about 'resources'. 'Resource' has a number of meanings in Java and Android. In Android, a resource is a special construct that is available as a special static class called 'R'. JDOM uses a different type of resource accessed through ClassLoader.getSystemResource(...), and the ClassLoader.getSystemResource(...) type methods are completely unreliable on Android. So, these JDOM resources needed to be loaded in to Android in a different way. The only really logical way to do that is though what Android calls 'Assets'. Unfortunately, JDOM needs to access these resources through URL's because in many cases JDOM needs to parse documents which have internal DTD's and XSD's which themselves are relative URL's to the Document.

There are some significant logistical problems with setting up the JDOM tests in such an environment:

0. You need both a 'main' and a 'test' Android project
1. You need to have the jars that the tests use installed in the 'main' app.
2. You need all 'resources' to be in the main app.
3. All the 'resources' need to be made accessible in some form that can be expressed as a URL.
4. You need all the JUnit tests to be in the form of JUnit3 tests
5. The JUnit tests need to be installed in the test app.

All these requirements have been met in a way that makes it almost easy.... ;-)

Testing JDOM Recipe - The '13 Steps'

In essence, if you want to run the tests yourself, you need to:

0. Install a Java JDK, and set up your JAVA_HOME and PATH
1. Install Apache 'ant' and set up your ANT_HOME and PATH.
2. Install the Android SDK and add <AndroidDirectory>/tools to your PATH
3. Create a directory to work in. I recommend something like C:\jdom for windows, or $HOME/jdom for UNIX.
4. Choose a name for your Android roject. I use 'JUT' for 'JDOM Unit Testing'
5. Create the 'main' Android project: android create project --target android-15 --name JUT --path ./JUT --activity JUTActivity --package org.jdom2
6. Create the 'test' Android project: android create test-project -m ../JUT -n JUTTest -p JUTTest
7. Get the JDOM source code from GitHub: git clone git://github.com/hunterhacker/jdom.git
8. 'Populate' the projects with JDOM tests, libraries, and resources. This is a little piece of 'magic'.
   1. cd jdom
   2. ant android -Dandroid=../ -Dandroid.project=JUT -Dversion=2.android
9. Build your Android projects with:
   1. cd ../JUTTest
   2. ant debug
10. Start an Android virtual device (see the Android DevKit for instructions).
11. Install the JUT apps: ant installt
12. Run the tests: ant test

The Magic JDOM 'android' ant target

This is the part that interfaces the JDOM project with Android. It does the following:

0. compiles JDOM
1. builds the JDOM jars
2. Creates a copy of the JUnit test jar in the main Android project which contains only the classes, with no resources. These classes are the original JUnit4 tests.
3. Copies the core and contrib JDOM jars to the main Android project.
4. Copies all the 'support' jars to the main Android project (JUnit4 library, Xalan, Jaxen, etc.)
5. Copies all the test resources in to the main Android project's 'assets' folder.
6. Creates JUnit3 wrapper classes and methods in the 'test' Android project for each JUnit4 test. It first applies a limited amount of 'filtering' for some tests (like it does not test StAX-based code).
7. It adds a special 'AndroidFetch' class to the test project.

There are two items of note here. The first item is that a conversion layer is created to translate the JUnit4 tests in JDOM to JUnit3 tests in Android. This layer essentially wraps each JUnit4 test in whatever means necessary to make it behave as if it was called from a JUnit4 TestRunner. You can inspect the generated JUnit3 source code in the test Android project. The program that does this conversion is included in the JDOM contrib section: org.jdom2.contrib.android.TranslateTests

The second item of note is that all the JDOM JUnit tests now use the new class 'FidoFetch' to get test resources (URLs and Streams). This new class is in the JDOM test area called org.jdom2.test.util.FidoFetch. This class 'supplies' the tests with the resources they may need. When running on Android there is a special implementation of this FidoFetch called org.jdom2.test.util.AndroidFetch. This special implementation converts the Android 'assets' to regular Android File instances. The File instances are important because you can make File URL's, and these File URL's can be used for the JDOM tests.

Thus, it is important to note that, for regular JDOM testing, the resources are retrieved (using FidoFetch) as either ClassLoader.getResource(name) or ClassLoader.getResourceAsStream(name), but on Android they are retrieved (using a specialized FidoFetch) as new File(dir, name).toURI().toURL() and new FileInputStream(new File(dir,name)) respectively.

Test Results

As this testing process was built, a number of technical issues in JDOM became apparent. Eah of these issues could be called a 'bug' in JDOM's run-anywhere status. These issues have been resolved.

0. JUnit tests were using ResourceBundles which do not work on Android. These were replaced with equivalent, but simpler mechanisms.
1. The XMLReaders Enumeration is a critical, and significant feature in JDOM 2.x. This Enumeration was failing to initialize through because the default javax.xml.validate.SchemaFactory class does not implement an XSD Schema validation mechanism (which is against the specifications for SchemaFactory). Because the enumeration failed to initialize, all references to the Enumeration threw ClassNotFound errors!
2. The LineSeparator Enumeration is also critical. It's construction order appears to be subtly different between Java and Android. This caused the Android implementation to dereference enum members which had not yet been constructed. Changing the way that the LineSeparator DEFAULT member is constructed allows the process to work well in both Android and Java.
3. Some of the JDOM tests are for 'negative' testing of the handling of invalid Unicode 'surrogate' sequences. It is not possible to load/compile these invalid unicode sequences in Android, so these tests have been skipped.

Now that we actually have JUnit tests running on android, we can analyze the results.

The Android XML Parsers

Android ships with its own XML Parser. JDOM typically recommends using the Xerces parsers, and the JDOM tests are all written to pass when using Xerces. Android does not use the Xerces parse, and some differences are a result of that. It is thus useful to document what does not work with the Android built-in XML parser.

StAX support missing

The StAX test code simply cannot run on Android. The core StAX Libraries are not available. The StAX tests cause the Android test project fail to compile (the tests throw XMLStreamException which is not available). As a result, the StAX tests are not even generated.

DocType Internal Subset

Xerces SAX Parser provides any internal subset data from the DocType as part of the SAX Build process. The Android SAX parser does not. This is causing about a dozen tests to fail. This issue is not critical though, I don't think.

ExpandExternalEntities cannot be toggled.

The Android parser ignores the feature to turn off entity expansion (expand-general-entities). As a consequence, the Unit tests that expect to parse a document and get a JDOM EntityRef in the results are going to fail.