Dealing with large resources

I have a rather large set of resources for my application. At this point I do not really want to talk about what they are, but they could well be larger sets of built-in geographical points.

First of all why do I want this data to be inside my application when Android is, after all, a phone, ie a device where connectivity to the outside is essential to its functioning? There are several reasons:

• my app is to be taken where connectivity might not be available, e.g. out in the sticks or when connectivity fails. 
• I would like to avoid having to maintain a website that provides all this data to users on the go. The overall download would exceed the once-off one. Plus I have to deal with it.
• I would like the app to be complete and functional from the very beginning, without the need to install additional data packs from the web.

Turns out that this is not as straight-forward as I thought.

A bad way of packaging the information is to hard-code it all in Java routines. But individual routines have a limit of 65635 for their size, and hacking this by breaking this up into individual routines then results in the translator to the DEX format to fail. Hacking is never a good idea...

A better way is to put it all into an XML resource file and extract the data like this:

XmlResourceParser xpp = mContext.getResources().getXml(R.xml.extensiondata);
try {
 int eventType = xpp.getEventType();
 while (eventType != XmlPullParser.END_DOCUMENT) {
  if(eventType == XmlPullParser.START_TAG) {
   if (xpp.getName().equals("p")){
    String i = xpp.getAttributeValue(null, "h");
    String j = xpp.getAttributeValue(null, "p");
    insertData(db, i, j);
   }
  }
  eventType = xpp.next();
 }
} catch (XmlPullParserException e){
 Log.e(TAG, e.toString());
} catch (IOException e){
 Log.e(TAG, e.toString());
}

This assumed that my XML (for exhibition) contained elements named 'c' with attributes 'p' and 'h' and that I have a routine insertData() that puts this in the right table in my DB. This works fine up to a certain size, then I get the error:
'Data exceeds UNCOMPRESS_DATA_MAX', where the maximum size of a decompressed file for this purpose seems to be 1048576. Fair enough, there is a limit on memory on the unit.

You might at this point argue that that is too much data anyway. But I do not think I am actually that much implementing for the G1 phone. Quite frankly, if I was to pay that much money for a phone, I would buy an iPhone, as it has a higher sex appeal. In my view G1 is a poor attempt of competing with a design company that is way ahead of Google. The real appeal lies in the open platform and in the next year we will I would be surprised if we did not see devices much more powerful in computing power, but perhaps smaller screens.

Anyway. For now it is to downsize the XML files further, in a first attempt by breaking it into a few pieces.

Breaking my XML file down into three parts and iterating through them makes this work. That is good news as it shows that the limitation was alone at the inflated file size. 

I check with my database and the size of the database file is about 500KB, which is not bad considering how much stuff I have in there. The total size of the XML files is about the same, which is not so good, as it suggests that SQLite stores a lot of overhead: the payload size is about 150K bytes. But access speed is not bad on the emulator.

I will try to find a way of tweaking the size a bit and I will also need a good way of getting the data in at first start-up. At the moment I just go for a coffee when I load the DB. A real user would probably reboot the phone.