Thursday, August 13, 2009

Android and Google App Engine

As the second part to my little series on developing an application across GAE(Google App Engine) and Android, I am going to continue with the example from my previous post on creating a REST service on GAE. In this post I will go through a very simple Android application that will use Restlet to call a Restful service deployed in GAE. I will assume the user has some basic knowledge of developing an android application, although this will be a very simple example so it should be easy to follow.

First download Restlet from here, if you do not already have it. You will need at least version 2.0 milestone 3. Once this is ready add the to the classpath of the project.

First using the eclipse plugin for Android development create a new project, and an Activity. I am just going to use the Hello World activity that is created by the plugin as a starting point. I modified the layout for the page to look like the following, I just gave the TextView an id so that I could refer to it in the Activity.

<?xml version="1.0" encoding="utf-8"?>
<LinearLayout xmlns:android=""

Now on to the activity itself, in this example the application will just call the Restful service running on the GAE when starting up. If the current date is before the expiration date a message will display telling the user when the app will expire, and if the application is already expired, it will display a message telling the user so. Obviously this example is not practical, and in a real application more would be done to provide a better user experience, but this is just to show how one could call the service.

In the onCreate method a handle to the TextView is created, and the text is set to display the correct message based on the expire date. A method called getExpireDate is called to do the actual checking of the date, parse the xml returned from the server, and generate the message string. You can see the class in all it's glory below.


import java.text.ParseException;
import java.text.SimpleDateFormat;
import java.util.Date;

import javax.xml.parsers.DocumentBuilder;
import javax.xml.parsers.DocumentBuilderFactory;
import javax.xml.parsers.ParserConfigurationException;

import org.restlet.representation.Representation;
import org.restlet.resource.ClientResource;
import org.restlet.resource.ResourceException;
import org.w3c.dom.Document;
import org.w3c.dom.Element;
import org.w3c.dom.Node;
import org.w3c.dom.NodeList;
import org.xml.sax.SAXException;

import android.os.Bundle;
import android.widget.TextView;

public class Test extends Activity {
private SimpleDateFormat df = new SimpleDateFormat("MMM d, yyyy HH:mm:ss a");
/** Called when the activity is first created. */
public void onCreate(Bundle savedInstanceState) {
TextView v = (TextView) findViewById(;

private String getExpireDate(){

String expireDate = "";
boolean expired = false;
String url = "";
ClientResource resource = new ClientResource(url);
try {
Representation r = resource.get();
DocumentBuilderFactory dbf = DocumentBuilderFactory.newInstance();
DocumentBuilder builder = dbf.newDocumentBuilder();
Document dom = builder.parse(new ByteArrayInputStream(r.getText().getBytes("UTF-8")));
Element root = dom.getDocumentElement();
NodeList nodes = root.getElementsByTagName("date");
// DomRepresentation rep = new DomRepresentation(r);
// NodeList nodes = rep.getDocument().getDocumentElement().getChildNodes();
for (int i = 0; i < nodes.getLength(); i++) {
Node node = nodes.item(i);
expireDate = node.getFirstChild().getNodeValue();
Date date = df.parse(expireDate);
Date today = new Date();

} catch (ResourceException e) {
} catch (IOException e) {
// TODO Auto-generated catch block
} catch (ParserConfigurationException e) {
// TODO Auto-generated catch block
} catch (SAXException e) {
// TODO Auto-generated catch block
} catch (ParseException e) {
// TODO Auto-generated catch block

//create date string
String returnMessage="";
returnMessage = "The app expired on "+expireDate;
returnMessage = "The app will expire on "+expireDate;
return returnMessage;


I am using the Restlet libraries to call the service on GAE using a ClientResource, with the URL the service is running at, and then calling get on that URL. I am then using some DOM libraries that are included with Android to parse the xml result. Restlet does include some libraries to do this, but I couldn't get them to work properly, I'm not sure that they are fully supported in this release of Restlet for Android. Once the expire date is pulled out of the result, the comparison is done, and the display string generated. This is fairly straightforward and simple, and Restlet makes doing this very easy. The url that is used in the example should be the url that your application is running in under GAE, in the example above I was running on localhost, and is the machine address that the android emulator uses to access a url running on your local machine.

The only other thing that is left to do is to enable the internet permission on android, to let Android and the users know that the application will be accessing the internet. If you don't know how to do that refer to the Android documentation, but I have included my full manifest for the example below. That's it!

<?xml version="1.0" encoding="utf-8"?>
<manifest xmlns:android=""
<application android:icon="@drawable/icon" android:label="@string/app_name">
<activity android:name=".Test"
<action android:name="android.intent.action.MAIN" />
<category android:name="android.intent.category.LAUNCHER" />
<uses-sdk android:minSdkVersion="3" />

<uses-permission android:name="android.permission.INTERNET"></uses-permission>

Saturday, August 8, 2009

Restful Service on Google App Engine

I have been playing around with GAE(Google App Engine) and Android recently, and as part of my experimenting I wanted to to be able to call a service running on GAE from Android. I found a great library called Restlet. Restlet provides a simple framework and api for creating and consuming rest services, it even includes libraries for GAE and Android so I thought I would give it a shot. I am going to write a series of posts that will contain a basic GAE application for a rest service, an Android application to consume the Restful service, and finally I will write a post that shows a simple example of writing some information to the GAE datastore. My example will be target toward providing a simple solution to a question on stackOverFlow.

For part one I will go over an example for creating a restful service using the Restlet api. I will be using the eclipse with the GAE plugin installed. In eclipse create a new Google web application project, and enter the project details, I unchecked the use GWT box, as I am not using that. This will create a basic servlet project, we will not use the servlet that is created, but it sets up the basic project structure for us. Next download Restlet from here, and add the org.restlet.gae.jar to the war/WEB-INF/lib folder of your project, and make sure it is in the classpath. We are now ready to add Restlet functionality to our project.

This example will use two classes, so create two classes, I have created TestApplication and TestResource. As you can see below. Test application extends Application, and here we will create our Restlet to direct calls to. The TestResource extends ServerResource. This class allows us to tie our rest services into other logic, such as accessing the datastore or other various config. In this class I have created two methods, represent, and accept representation. These will handle the associated Post and Get requests marked by their corresponding annotation. Take notice of the @Get annotation on the represent method, I have specified that I will return type of "xml". And then looking at the body of the represent method, I create a simple xml document, and populate and create a date element populated with today's date. This date could be hard coded in the class or retrieved from the datastore, but this shows the basic idea of returning the date from a restful service. Take a look at the full classes below.

import org.restlet.Application;
import org.restlet.Restlet;
import org.restlet.routing.Router;

public class TestApplication extends Application{

* Creates a root Restlet that will receive all incoming calls.
public synchronized Restlet createRoot() {
// Create a router Restlet that routes each call to a
// new Resource
Router router = new Router(getContext());

return router;

import java.util.Date;

import org.restlet.ext.xml.DomRepresentation;
import org.restlet.representation.Representation;
import org.restlet.resource.Get;
import org.restlet.resource.Post;
import org.restlet.resource.ServerResource;
import org.w3c.dom.Document;
import org.w3c.dom.Element;

public class TestResource extends ServerResource{

public Representation represent(){
// Generate the right representation according to its media type.
try {
DomRepresentation domRep = new DomRepresentation(MediaType.TEXT_XML);
// Generate a DOM document representing the list of
// items.
Document d = domRep.getDocument();
Element r = d.createElement("expire_date");
Element date = d.createElement("date");
date.appendChild(d.createTextNode(new Date().toLocaleString()));

// Returns the XML representation of this document.
return domRep;
} catch (IOException e) {
return null;

public Representation acceptRepresentation(Representation entity){
//Could process some post requests here and possibly write to the datastore
return null;

The only thing left to do now is configure the web.xml to route to our servlet correctly. Open war/WEB-INF/web.xml, and delete the generated web-app config tags and add the following. Here we are intercepting all requests for simplicity, but these could be changed, and routing everything to our test.TestApplication class, change this to whatever you named your application class above.


<web-app xmlns="" version="2.5">
<display-name>first steps servlet</display-name>
<!-- Application class name -->

<!-- Restlet adapter -->

<!-- Catch all requests -->

That is it, you can now run this with the embedded server provided by the GAE eclipse plugin by running your application as a google web application, and directing your browser to http://localhost:8080/. You should see the xml returned by the request with today's date. When you are ready you can deploy the application to the app server and access it from your appspot address. In my next post I will go over how to consume this request from an android application.

I have been informed that some distributions of restlet do not include the app engine jar, it should be noted that this is only available on Version 2.0, and Milestone 3 according to the release notes. If you go here and download, it should contain the jar file.

Thursday, June 25, 2009

Importing large datasets in MySql

I have been working a new website This website will be a bicycle journal and statistics management site. Allowing a user to track all kinds of statistics about their bicycles, rides, and equipment they use, or that is part of their bicycle. The website is up and currently I am allowing beta testers, so if you're interested then go the site and request to be a beta user, you will be able to give feedback as to what features you would like, as well as any other changes you think would make the site great.

But enough about the site for now, I will post future posts about the site, and the technology I am using as well as obstacles I have had to overcome, both to give credit to those who have made what I want to achieve easier, and also as a way for me to log things I don't want to forget.

The first thing I would like to write about is how to import large data sets into a MySQL database. For cyclebrain I have some database tables where I store information to allow a user to select the country, state, and city there are from as part of the user preferences. This is great, except one of the import files is over 300MB! This has caused issues with the interface my hosting provider has provided. I have tried a few different tactics to achieve this, such as using MySQL QueryBrowser and trying to execute the scripts, this took way too long and I was losing my connection to the database. I also tried uploading the files and connecting to the database via an ssh client, this too did not work. I also tried splitting the file into smaller files and running those, this worked the best but I was still getting timeout errors, and I was tired of tracking which files I had run, and which had errors.

I just happened to be browsing dzone and came across a post where someone was running into the same issue, and in the comments there was a link to BigDump. BigDump is great. It takes a large sql file and breaks it up into smaller sessions, so issues like timeouts, are less prone to being encountered. You simply just upload a php file to your server, and put the sql files in the same, web accessible, directory. You then just bring up the page in your browser. It lists all of the sql files, and give you the option to run them. It couldn't be simpler. BigDump does have some restrictions with the files it can handle and the formats, but for my files, simple sql inserts it worked great. One problem that I did have was BigDump stopping on any errors in my sql file. I ran into primary key constraint violations, after checking my file I did not notice these inconsistencies, I'm thinking it may have something to do with the way BigDump breaks up the requests, and maybe there was some duplication in the queries at the end of one request, and the beginning of another. No matter, I just commented out the code that stops the process on this error and ran it again. No problems, this worked for me, as I can deal with a few rows missing. I may go back and log these so I can make sure they worked, but for now this works, and BigDumphas saved me some headaches.