Dealing with the Android Market

With the G1 now available, everyone is waiting for the Android Market to come online.

Google has taken the decision to only allow free applications onto the market until some time early in 2009, which has clearly pissed off some developers who had hoped to be able to charge from day one. I am not surprised by this decision. While there might be reasons of simply not getting the payment system ready (which should have been not too difficult for Google), I assume the main reason is that it is quite easy to get lots of bad publicity if you have crappy paid-for applications in the official market. While these applications are third-party, Google would take part of the blame. With free, free as in beer, stuff that is not likely to be the case as long as no application can do any damage.

Anyway, the reason I am mentioning this is that developers have now to decide what to do with their applications. They have the following options:

1. Give away for free what they wanted to charge for. Bad idea, I think. Not because of the bad vibes that it will create later, but mainly because it gives competitors some three months to do it better.
2. Just wait. Maybe not such a bad idea. One can see how it all shapes up, but one might lose momentum against others. Of the 10,000 or so developers at least one will have had a similar idea and they will occupy the high spot.
3. Give away a 'beta' or cut-down version for now, with an upgrade later to a fully functional version. Possibly my choice - in combination with 2). It means not losing out on interest and also getting valuable feed-back.

Now how to do such a beta. 

1. Time-limit the application: bound to piss people off, but not bad if one really has a valuable application.
2. Tempt people to pay with better functionality. Of course one does not really know right now what that might be, so I decided to put a WebView into my application, so that I can change the details of what is available and also take a user then to whereever I decide to sell my app.

I embed a download link to the upgraded up, so that it would install on the emulator, but that hit a snag. For those who are not in the US and are thus pretty much unable to get a G1, the emulator unfortunately poses an unsurmountable restriction for installing apps from the web directly onto the emulator, as it is locked to only accept apps from the official store and the settings module is not compiled in. Here you can read more.

There is also a site with deals with embedding licensing information in the app: http://www.andlicensing.com/.

Dealing with large resources

I have a rather large set of resources for my application. At this point I do not really want to talk about what they are, but they could well be larger sets of built-in geographical points.

First of all why do I want this data to be inside my application when Android is, after all, a phone, ie a device where connectivity to the outside is essential to its functioning? There are several reasons:

• my app is to be taken where connectivity might not be available, e.g. out in the sticks or when connectivity fails. 
• I would like to avoid having to maintain a website that provides all this data to users on the go. The overall download would exceed the once-off one. Plus I have to deal with it.
• I would like the app to be complete and functional from the very beginning, without the need to install additional data packs from the web.

Turns out that this is not as straight-forward as I thought.

A bad way of packaging the information is to hard-code it all in Java routines. But individual routines have a limit of 65635 for their size, and hacking this by breaking this up into individual routines then results in the translator to the DEX format to fail. Hacking is never a good idea...

A better way is to put it all into an XML resource file and extract the data like this:

XmlResourceParser xpp = mContext.getResources().getXml(R.xml.extensiondata);
try {
 int eventType = xpp.getEventType();
 while (eventType != XmlPullParser.END_DOCUMENT) {
  if(eventType == XmlPullParser.START_TAG) {
   if (xpp.getName().equals("p")){
    String i = xpp.getAttributeValue(null, "h");
    String j = xpp.getAttributeValue(null, "p");
    insertData(db, i, j);
   }
  }
  eventType = xpp.next();
 }
} catch (XmlPullParserException e){
 Log.e(TAG, e.toString());
} catch (IOException e){
 Log.e(TAG, e.toString());
}

This assumed that my XML (for exhibition) contained elements named 'c' with attributes 'p' and 'h' and that I have a routine insertData() that puts this in the right table in my DB. This works fine up to a certain size, then I get the error:
'Data exceeds UNCOMPRESS_DATA_MAX', where the maximum size of a decompressed file for this purpose seems to be 1048576. Fair enough, there is a limit on memory on the unit.

You might at this point argue that that is too much data anyway. But I do not think I am actually that much implementing for the G1 phone. Quite frankly, if I was to pay that much money for a phone, I would buy an iPhone, as it has a higher sex appeal. In my view G1 is a poor attempt of competing with a design company that is way ahead of Google. The real appeal lies in the open platform and in the next year we will I would be surprised if we did not see devices much more powerful in computing power, but perhaps smaller screens.

Anyway. For now it is to downsize the XML files further, in a first attempt by breaking it into a few pieces.

Breaking my XML file down into three parts and iterating through them makes this work. That is good news as it shows that the limitation was alone at the inflated file size. 

I check with my database and the size of the database file is about 500KB, which is not bad considering how much stuff I have in there. The total size of the XML files is about the same, which is not so good, as it suggests that SQLite stores a lot of overhead: the payload size is about 150K bytes. But access speed is not bad on the emulator.

I will try to find a way of tweaking the size a bit and I will also need a good way of getting the data in at first start-up. At the moment I just go for a coffee when I load the DB. A real user would probably reboot the phone.

Testing with the Android Framework

The second installment on testing with Android. There is a dearth of decent documentation (or indeed any documentation) out on how to test Android applications.

The best starting point is to have a look at the ApiDemos sample that comes with the SDK. It has a complete and working setup for a variety of tests. The first steps are to make sure that this actually works on your machine.

1. First of all, build the officially supplied test suite. This does not work out of Eclipse as far as I can tell, so just go to a command line window and go to /samples/ApiDemos/tests and do a build with ant.
2. You might find that the ApiDemos itself was not built before, so go there first.
3. Once the ApiDemos is built, try building the test again.
4. Now you need to install the ApiDemos and the tests. This is done by going into the respective bin directories and run adb install ApiDemos-debug.apk and adb install ApiDemosTest-debug.apk. This requires that the emulator is running. Mine is, as I started it from Eclipse.
5. Now restart the emulator. This is important as otherwise it might not know that these new apks have been installed. I made the mistake of letting it run, and I always got an error when running with the suggested commands.
6. Run something like 

   $ adb shell am instrument -w com.example.android.apis.tests/android.test.InstrumentationTestRunner

   
   The tests should now run (some fail for me, but the main machinery is up and running).

Now ist the time to get my own tests in:

1. I recreate the directory structure for my tests from the Google template and copy the AndroidManifest.xml and build.xml. Both will need some modification to make sure that my classes will be tested.
2. In the src directory I adjust the files for the top-level test runners and add a directory that will exercise my own TracklogViewer.
3. The build with ant fails on a few points. -e option as usual gives a bit more information. Some directories need to be created by hand as they were expected but are not there.
4. I install my own test apk on the emulator.
5. I run the instrumentation.
6. It fails. The reason, it transpires, is that I have not restarted the emulator.
7. It finally runs. As the tests are pretty empty, there is no error. 

Now it comes to actually implementing a few tests. But that is another story.

Testing with Android

Testing with Android is a pain.

While the normal JUnit test does work, once one makes a few adjustments to the setup on Eclipse, this does not mean that any of the Android classes are available for these tests. So even if my classes are not Activities, Services etc, but plain old computations, they cannot be tested with JUnit as soon as I make use of any of the classes in Android.

I think that some of this reveals bad design decisions by those developing Android. Take android.graphics.Rect. This is a class that has keeps tab of four corner points plus gives some simple operations such as intersections. There would be nothing Android specific in them were it not for the methods to parcel it for the Android IPC. Would this not have been done better by exposing these methods with standard serialisation mechanisms via a custom toByteArray or some such?

Another solution would have been to have an Android-only jar which contains only the Android classes, with some things suitably stubbed out, so it can be linked into the testing framework on the host machine, rather than the Emulator.

Looks like I have to bite the bullet and make all my testing run on the emulator. At the same time I am making the same mistake as Google: not observing a strict layering principle which will give me much better portability in the future.

I have to go away and think about that.

Using Large Images with Android: A no-no

I want to be able to display satellite images not from Google, but from other sources on my Android GPS application.

I have already shown how to build a simple WMS client to display online satellite images. However, the use I envision for my device (which might not be a phone), is to be able to display such images offline, from the built-in disk or an SDCard.

The built-in BitmapFactory has not real support for larger images, anything over a certain size will just fail to load with a memory error.

I first thought that something like Java Advanced Imaging JAI will do the trick. Well, it would, were it not that all support for this has been disabled in Android, certainly to reduce the memory footprint and stop people doing stupid things, like processing large images.

Every other library then either uses java.awt's BufferedImage (again not available on Android) or JNI with some underlying C, C++ or whatever implementation.

So, this is all a dead end. I have thought for a long time, that there is room for a really light-weight WMS server implemented in Java, or something equally lightweight for image processing. A library that does NOT do everything, but just the bare minimals.

Now I have a new direction. As large files are really a no-no on Android, there has to be some preprocessing to make the files smaller.
Fortunately, there is gdal2tiles, a program from the GDAL Suite for geospatial processing. It turns a georeferenced file into a tree of much smaller ones together with an XML based description. Parsing the XML file will allow me to pick the tiles I need to put the overlay image back together.

Stay tuned.

An Icon in the Status Bar

A little excursion here, but now I am putting an icon into the status bar of Android to indicate whether tracklogging is enabled or not.

This is done with the NotificationManager system service.

The interface seems to be easy enough, but there is once again a pitfall that is somehow not really documented. At first sight, this is the only thing one needs to do to get it running.

protected void notify(Context context, Boolean on){
    NotificationManager nm =   (NotificationManager)context.getSystemService(Context.NOTIFICATION_SERVICE);
 if (on){
  Notification n = new Notification(R.drawable.locationloggerservice, "GPS Logging started", System.currentTimeMillis());
  nm.notify(22, n);
 } else {
  nm.cancel(22);
 }
}

However, doing this always raises an exception and looking at the message it becomes clear that the notification cannot live without an associated intent that is raised whenever someone clicks on the notification. So in this sense a 'read-only' notification that simply says the LocationLoggerService is up and running is not an option.  

We therefore need to wire an Intent into it. What should the intent be? Well, in this case, lacking any better I want to go to the preferences screen where the location logger can be turned off. This Intent has to be wrapped into a PendingIntent, which is a little parcel around all the information that can be handed back to the Notification application to launch an activity within my GPS application with the same rights as anything run within my own application natively.

Once we have created this PendingIntent and passed it into the notification, all is happy.

 protected void notify(Context context, Boolean on){
  NotificationManager nm = (NotificationManager)context.getSystemService(Context.NOTIFICATION_SERVICE);
  if (on){
   Notification n = new Notification(R.drawable.locationloggerservice, "GPS Logging started", System.currentTimeMillis());

   ComponentName comp = new ComponentName(context.getPackageName(), GPSPreferenceActivity.class.getName());
   Intent intent = new Intent().setComponent(comp);
   PendingIntent pendingIntent = PendingIntent.getActivity(context, 0, intent, Intent.FLAG_ACTIVITY_NEW_TASK);
   n.setLatestEventInfo(context, "Location Logger Service", "GPS Tracklogging Enabled", pendingIntent);
   nm.notify(22, n);
  } else {
   nm.cancel(22);
  }
 }

I call this notify() message from within my service manager that turns tracklogging on and off.

However, there remains one thing not so pretty. Now if I toggle in the preferences the Logging service on, I get the notification, and when I click on it, it takes me to the Preferences Screen again (I now have two of them stacked up). To make sure, only one is shown I also set another flag in the notification:

PendingIntent pendingIntent = PendingIntent.getActivity(context, 0, intent, Intent.FLAG_ACTIVITY_NEW_TASK + Intent.FLAG_ACTIVITY_SINGLE_TOP);

Now if my preferences screen is already top, it will not be launched again.

What is 22? Well, the notification needs an application specific unique int associated with it. 22 was as good as any. It is of course better to code this somewhere so uniqueness is guaranteed and a symbolic value can be associated with it.

An OpenStreetMap ContentProvider for Android

The Android ContentProvider is a wrapper for data access that allows applications to hide the specifics behind a small number of calls that are modelled on a database interface.
However, there is nothing that limits a ContentProvider to being implemented on top of a DB: any data source will do.

The OpenStreetMap Project is a collaborative mapping project, collecting map data all around the world for access under a liberal license (attribution/shared-alike). There are several ways of getting hold of OSM data, but one is the OSMXAPI interface. It allows getting data based on a bounding box and selection details. Returned from a query is (if successful) an XML document, which we parse with SAX.

Here, we implement a simple lookup only interface to OSMXAPI, which looks up public toilets.

Fundamentally, the query method, which is really the only we implement, goes like this:

1. First of all we expect in the selection parameter something like "amenity=toilets". OSMXAPI is not a database, but this is as close as a DB selection parameter as we get in this context.
2. Secondly in the selectionArgs we expect the four corner coordinates for which to check. This is a bit a rough-handling of this parameter, but the ContentProvider interface is a bit too tightly coupled to DBs to do this differently.
3. Then we construct the URL for the OSMXAPI call and retrieve the XML document. If anything goes wrong here, we will just return an empty cursor.
4. Now we parse the document with an XML parser, scanning for the nodes which contain the lat/long positions of our points. Every node is inserted into the data of a MatrixCursor, which is a generalised cursor essentially containing a table of data over which the user can iterate. 
5. We return this cursor to the user.

This is the full code for the ContentProvider:

package com.ucont;

import java.io.IOException;
import java.net.MalformedURLException;
import java.net.URL;

import javax.xml.parsers.ParserConfigurationException;
import javax.xml.parsers.SAXParser;
import javax.xml.parsers.SAXParserFactory;

import org.xml.sax.Attributes;
import org.xml.sax.InputSource;
import org.xml.sax.SAXException;
import org.xml.sax.XMLReader;
import org.xml.sax.helpers.DefaultHandler;

import android.content.ContentProvider;
import android.content.ContentValues;
import android.database.Cursor;
import android.database.MatrixCursor;
import android.net.Uri;
import android.util.Log;

public class OSMPointsContentProvider extends ContentProvider {

 private final String TAG = "OSMPointsContentProvider";

 public static final String AUTHORITY = "org.osm.data";
 public static final Uri CONTENT_URI = Uri.parse("content://" + AUTHORITY + "/osmdata");

 private final String osmxapiserver = "http://www.informationfreeway.org/api/0.5/";

 private class OSMPointCursor extends MatrixCursor {
  public OSMPointCursor(String[] columnNames) {
   super(columnNames);
  }
 }

 protected String constructQueryUrl(String selection, Double left, Double bottom, Double right, Double top) {
  String query = String.format("node[%][bbox=%f,%f,%f,%f]", selection, left, bottom, right, top);
  String url = osmxapiserver + query;
  return url;
 }

 protected String constructQueryUrl(String selection, String top, String left, String bottom, String right) {
  String query = String.format("node[%s][bbox=%s,%s,%s,%s]", selection, left, bottom, right, top);
  String url = osmxapiserver + query;
  return url;
 }

 
 @Override
 public int delete(Uri arg0, String arg1, String[] arg2) {
  // TODO Auto-generated method stub
  return 0;
 }

 @Override
 public String getType(Uri uri) {
  // TODO Auto-generated method stub
  return null;
 }

 @Override
 public Uri insert(Uri uri, ContentValues values) {
  // TODO Auto-generated method stub
  return null;
 }

 @Override
 public boolean onCreate() {
  // TODO Auto-generated method stub
  return false;
 }

 private class OSMHandler extends DefaultHandler {
  private OSMPointCursor cursor;
  OSMHandler(OSMPointCursor cursor){
   super();
   this.cursor = cursor;
  }
  public void startElement (String uri, String name,
    String qName, Attributes atts){
   if (name.equals("node")){
    Object[] columnValues = {atts.getValue("lat"), atts.getValue("lon")};
    cursor.addRow(columnValues);
   }
  }
 }

 @Override
 public Cursor query(Uri uri, String[] projection, String selection,
   String[] selectionArgs, String sortOrder) {
  String[] columnNames = {"lat", "lon" };
  OSMPointCursor result = new OSMPointCursor(columnNames);
  URL url = null;
  if (null == selection){
   selection = "*=*";
  }
  try {
   url = new URL (constructQueryUrl(selection, selectionArgs[0], selectionArgs[1], selectionArgs[2], selectionArgs[3]));
   SAXParserFactory spf = SAXParserFactory.newInstance();
   SAXParser sp = spf.newSAXParser();
   XMLReader xmlReader = sp.getXMLReader();
   OSMHandler handler = new OSMHandler(result);
   xmlReader.setContentHandler(handler);
   xmlReader.setErrorHandler(handler);
   xmlReader.parse(new InputSource(url.openStream()));
  } catch (MalformedURLException e){
   Log.e(TAG, e.getMessage());   
  } catch (IOException e){
   Log.e(TAG, "IOException " + e.getMessage());   
  } catch (SAXException e){
   Log.e(TAG, "SAX " + e.getMessage());
  } catch (ParserConfigurationException e) {
   Log.e(TAG, "PC Exception " + e.getMessage());
  }
    
  return result;
 }

 @Override
 public int update(Uri uri, ContentValues values, String selection,
   String[] selectionArgs) {
  // TODO Auto-generated method stub
  return 0;
 }

}

What are the lessons learned from this?

1. The ContentProvider interface is a bit too closely aligned with databases. A more general concept, based on key/values to match would have been much better. Android developers have somehow recognised this and provided the MatrixCursor for a generalised return value.
2. It would be nice if a ContentProvider could be queried which operations it supported. That way there would be a general way of implementing a read-only or write-only ContentProvider. The interface for insert does not allow a ContentProvider to throw an exception to indicate that inserts are not supported.
3. It would be good to have things like cost functions on ContentProviders (e.g. the cheapest way of providing a certain content or the most up-to-date). In that sense ContentProvider is some two decades behind what has been done in terms of service provision negotiation in distributed systems. 

Next will be to display the retrieved data on a map.