GWT / GAE development blog Articles about GWT/GAE and the development of TeamScape: A sports team portal

Datastore frameworks: The interview

Posted on March 27, 2010

In this article we will have a look at three of the low-level wrapper frameworks for the AppEngine datastore. It's an interview-style post where the authors of each framework answers a number of questions related to the AppEngine datastore and their frameworks. There will also be a follow-up article where I solve some typical datastore scenarios with each of the frameworks and write about my experience with that.

Background

Anyone who has tried to use JDO/JPA on AppEngine will probably agree that it is too complex and has a steep learning curve. Offering a JDO/JPA solution is kind of like saying "Hey. We have a standardized API so you don't have to worry about the implementation details of the underlying datastore.". The problem is that in the case of AppEngine, there is also a fine print saying "But you should also be aware of these limitations, these GAE extensions and...etc...etc...".

This means that you MUST worry about the implementation details anyway, since the datastore <-> JDO/JPA mapping is far from optimal. If you don't know about the datastore limitations and think you can use JDO/JPA like you would on a relational datastore, you are in for a bumpy ride. And just to make this clear, once again, the root cause of this problem is not related to JDO/JPA or the DataNucleus implementation in general.

The low-level API is on the other end of the scale. It has an extremply simple API, but it's not type safe and only works with native entities, which requires lots of tedious work from the developer. What should a developer do?

The low-level wrapper frameworks to the rescue

I'm not the only one who thinks that the JDO/JPA solution is too complex and makes development harder than it has to be. There are, to my knowledge, 6 different attempts at offering an alternative, which is not a coincidence. These 6 are Objectify, Twig, SimpleDS, Siena, Slim3 and cloud2b. Update: A 7th framework called jiql which is a SQL/JDBC interface on top of AppEngine.

I have chosen to look at three of these; Objectify, Twig and SimpleDS. There is a good reason to why I chose those three. They are designed specifically for the AppEngine datastore with its features and limitations in mind. The other three frameworks are designed with other goals in mind.

Siena and cloud2b has multi-platform support and, like JDO/JPA, has a generic API to support this. Slim3 is a full-stack MVC framework that goes outside the scope of a thin low-level datastore wrapper. Update: Apparently Slim3 can be used as a standalone datastore framework, which wasn't clear from the documentation at the time. The documentation has now been updated to describe this.

I'm sure that these frameworks are great and probably good choices for a multi-platform strategy. Or, in the case of Slim3, a complete server-side framework. But I'm not at all interested in that. I want to use a simple and efficient datastore framework that mainly acts as a convenience layer on top of the AppEngine low-level API. Objectify, Twig and SimpleDS fulfills this requirement. Read this thread for some debating on this matter and why I don't want to use SQL/JDBC interfaces for my project.

The interview

Ok. Let's get on with the juicy stuff. First, let me introduce the authors. (Lots of credit and thanks to the authors for this!)

Jeff Schnitzer, lead developer Objectify.

John Patterson, lead developer Twig.

Ignacio Coloma, lead developer SimpleDS.

The original plan was to wrap this article up with a nice debate, but one of the GAE/J threads turned into a fantastic debate that covers many different aspects of datastore philosophies and designs. The authors would need to repeat themselves here if we were to keep the debate, so I decided to skip it and refer to the debate thread instead. In one of the interview sections there are links to various stuff you should have a look at.

The interview has been split into 4 subpages to make it more lucid.

Table of content

AppEngine datastore

The frameworks

Feature checklist

Summary

Next: AppEngine datastore

  • Share/Bookmark

objectify evaluated

Posted on March 3, 2010

In this post we'll evaluate objectify with regards to performance and complexity compared to our current JDO solution.

Objectify is basically a framework wrapping the GAE datastore low-level API. There are other frameworks for this (see further down), but objectify seems to be the most comprehensive one with support for transactions, caching, cursors etc. It also provides a very clean and simple API using generics and annotations.

There is no need for me to describe objectify further since the author has done an excellent job documenting it on the wiki. I have used objectify 2.0.2 for this evaluation.

So why are we evaluating JDO alternatives?

  • The JDO GAE implementation is quite complex and hard to grasp for beginners, partly due to unsupported features, workarounds and extensions.
  • JDO is designed to work "everywhere", which means it is highly abstracted and very general in its nature. Great for portability, but it also means that it might not be a perfect match for a specific engine like GAE with its quite unique datastore.
  • The Datanucleus/JDO/JPA jars are quite big (~2.3 mb in total), while the low-level frameworks are in the kb range.
  • JDO/JPA *might* perform class path scanning at startup to find and register your entities, which could add to the load time. As Jeff points out in a comment below, the time spent would be proportional to the number of classes in your project.

Do note that JDO as a specification and as an implementation by DataNucleus does not necessarily have these drawbacks. I'm only discussing JDO as implemented on GAE in this post.

What about JPA? Well, JPA is designed for RDBMS, which the GAE datastore is not. I guess the datastore supports it mainly because there is a number of frameworks that uses JPA. For apps like TeamScape where don't use any server side framework, it makes no sense to use JPA. Google doesn't exactly seem to encourage developers to use JPA for the datastore either.

Why not use the low-level API directly in that case? The objectify introduction answers this question very well.

What about other low-level wrappers, like SimpleDS or Twig? Basically, I heard about objectify first ;-) . There is an interesting objectify-SimpleDS discussion here and an interesting objectify-Twig discussion here. I considered making this post a JDO/objectify/SimpleDS/Twig comparison, but I just don't have the time or energy for that. I do however make comments about them here and there. But check out each framework and compare for yourself. If you find something useful to discuss, feel free to post a comment in this post. My initial impression is that objectify is the most mature of the three frameworks with regards to development cycles, releases and documentation. Update: I just found two more frameworks for this, Siena and Slim3. And one more, cloud2db.

Why use objectify? Answering that is what this post is about. If the evaluation shows that objectify offers an edge over JDO, we will probably switch.

Before I get started, I really need to point out that I'm in no way an expert on databases, JDO or the GAE datastore. I do however consider this as an advantage when writing blog posts like these, since I don't take knowledge and details of these things for granted. But let me know if anything is unclear so I can clarify it. Also, if you think something I write seems completely crazy, let me know as well. I refer to JDO and objectify as "frameworks" in this post, although this might not be an accurate term for them.

Ok, let's move on.

I will compare objectify and JDO (and might make remarks on the other frameworks to) with regards to:

  • API design/complexity and how it relates to the GAE Datastore implementation
  • Load time (GAE cold start time)
  • CRUD + Q performance (create, read, update, delete, query)
  • Portability

Framework/API design and complexity

Let's start by having a look at the two frameworks from a developer perspective. I had never used GAE or JDO until a couple of months ago, so I can evaluate both JDO and objectify from a beginners perspective. Well, I did have a fair amount of knowledge on the GAE datastore when I got started with objectify, but that doesn't matter much for this comparison.

JDO@GAE is actually really well documented if you combine the GAE docs/articles, Datanucleus docs and the GAE Java forum. But for a beginner, it is a LOT of information to take in before you have a decent picture on the work flow. Once you feel comfortable with JDO, you have to dig into the DataNucleus/GAE extensions and how GAE features such as transactions, owned/unowned relationships, entity groups, joins, keys, etc plays with JDO. And once you think you are ready and start testing it, you will bump into lots of problems along the way that is caused by DataNucleus/GAE implementation details that you were not aware of. For me it was a lot of write - test - fail - rewrite cycles before I had something up and running.

I first heard about objectify in the GAE forum and decided to have a look at it after reading David Chandler's interesting posts on it. When I evaluate third party frameworks, one of the most important factors for me is good documentation (which loooots of frameworks lack). Anyways, my point is that I was very impressed by the objectify wiki. It has an excellent introduction and lots of examples, howtos, best practices, etc. It answered most of my questions before I even thought of asking them...! I checked out the documentation for SimpleDS and Twig as well. SimpleDS has decent documentation, but Twig has some work to do to reach up to the same level as objectify.

After reading through the whole wiki and checking out the JavaDoc, I setup a separate track using objectify instead of JDO for my app. It took me about 30 minutes to setup objectify, define an entity and write a Datastore implementation using the common objectify operations. And it worked on the first attempt. Do note that I only use very trivial datastore operations at this point (saveModel, getModel, getModelByQuery and deleteModel). The objectify API is very straightforward and it's easy to understand the purpose of each class/method.

So let's just do a quick comparison of the JDO API and the objectify API. We'll compare the main interfaces a developer is faced with. Remember here that JDO is designed to work everywhere and is very generalized because of this. It can never be as straightforward and simple as a framework targeted specifically for one engine, like objectify is for GAE in this case.

The factory wrappers: It is common to create a PMF class if you use JDO where you have a PersistenceManagerFactory singleton. This is used to retrieve PersistenceManager instances. Objectify has a corresponding class called ObjectifyService that wraps a ObjectifyFactory singleton. The main difference is that you can use ObjectifyService.begin() to retreive a Objectify instance directly, whereas in the JDO case you do something like: PMF.get().getPersistenceManager(). Slightly easier in the objectify case since ObjectifyService provides some convenience functions and you don't have to create a PMF corresponding class yourself.

The factory: The factories contains similar functionality, although the objectify one is clearly more light-weight and easier to understand.

The primary interface: PersistenceManager vs Objectify. This is where objectify really shines in simplicity compared to JDO. Objectify contains ~15 very straightforward methods, while PersistenceManager has like ~50-60 methods. It is much easier to get a quick overview of objectify.

I will not discuss the other classes, but if you open up each JavaDoc, you can see that JDO is a pretty large library while objectify is pretty slimmed.

Let's have a look at how some of the datastore implementation details are handled in the frameworks.

Entity relationships: The frameworks differ a bit here. JDO supports what we call owned relationships by having a direct reference to another entity. This is not supported by objectify. Objectify does however support parent/child relationships using keys and annotations. While you can configure JDO to automatically fetch all children (fetch groups) when you fetch the parent, it seems you have to manually handle this with objectify. You can of course use unowned relationships by keeping another entities string key/long id in a field with both frameworks. Both frameworks also support embedded entities.

JDO does more work for you automatically in the case of owned relationships, but it is also much more complex and more error-prone. Also note that owned relationships is direcly related to entity groups and transactions. If you don't need to perform atomic operations on a set of entities, you should generally NOT use owned relationships because of the limitations entity groups results in. It is prettty straighforward to use unowned relationships in both frameworks.

Side note: One of the main differences between objectify and Twig is related to this. Twig does support owned relationships (Java style). The authors discuss this at length in the thread I referred to in the beginning of the post.

Transactions: From our perspective, the transaction handling is basically identical in JDO and objectify. Easy to use in both cases.

Entity refactoring: If you decide to refactor your entities, you can easily get into trouble in a live app if you're not careful. I know that JDO has support for schema mapping / metadata that can be used in these cases, but it is not really mentioned anywhere in the GAE docs and this thread seems to suggest that it is not implemented in GAE yet. Objectify has some basic support to handle this using annotations.

Caching: Objectify integrates very nicely with GAE memcache. All you have to do is to add @Cached annotation to your entity, and objectify takes care of the rest. Very neat. JDO has some caching support as well that apparently uses memcache, as mentioned by Erik in a comment below. See this page.

Usage with GWT: You probably want to seamlessly pass your entities over the wire to and from GWT without using DTOs. First, your entities and all its fields needs to be serializable. Secondly, your entity code must not refer to any server specific libraries (for instance, you can't use the GAE Keys directly). Third, you must keep your entities in a package known to GWT (in our example, the shared package). Fourth, the datastore framework must accept "detached" entities to be saved directly to the datastore. For JDO, you have to define your entities as detachable and then detach them via PersistenceManager before you return them to GWT. This basically works out of the box with objectify as long as you follow the rules above. Very nice!

Dependencies: JDO with all its dependencies is huge. It's something like ~2 mb in total. Objectify on the other hand has no external dependencies and the jar is ~100 kb.

Another thing that makes sense to discuss is how well JDO integrates with the GAE datastore. The GAE JDO implementation has lots of unsupported JDO features and a bunch of extensions to make it work properly in the GAE environment. The GAE datastore is quite unique in many ways, while JDO is a standardized API. Because of this, there will always be some unsupported stuff, some necessary workarounds and some extensions needed for frameworks like JDO/JPA. Frameworks like objectify that is targeted specifically for the GAE datastore doesn't have this problem. The API is basically an abstraction layer above the low-level API, so the API/method/features design makes more sense for a developer trying to relate to the GAE datastore.

Cold start time

Let's move on and talk a bit about the GAE cold start issue and startup time.

This issue has been under heavy debate lately, which is understandable considering the time it takes to cold start a JVM instance. Please read this appengine blog post, this performance faq, this forum post and this forum post to gain an understanding of the issue.

In TeamScape, I have seen load times from 5 seconds all the way up to 13 seconds. This is simply not acceptable.

Since I deploy new TeamScape versions regularly and we're developing the app, it will basically cold start every time someone access it. We basically need at least one visitor per minute to keep the instance running, and I don't think this is a likely scenario for TeamScape even when its live. This is quite frustrating, so we need to do everything we can to improve the performance. I have enabled the pre-compilation (which is default now btw), but it is still way too slow. The best way to improve this is to remove any library we don't need in WEB-INF/lib and try to minimize the work done when our servlets are initalized.

One of the reasons I looked forward to evaluating objectify was this forum post where the poster reported very interesting performance improvements by switching from JDO to objectify. JDO/JPA does something called class path scanning to automatically find and register your entities. This is done each time a new JVM instance if spawned, so it adds significant time to your load time. The objectify authors seems to have taken an active decision not to support this, which means that you have to manually register your entities. However, it also means that the cold start time for an objectify solution should be much faster than a JDO/JPA solution since classpath scanning can be avoided. Note: The text above is not entirely correct. Read the comments below.

To test this, I logged 10 cold start requests using the JDO solution. Then I removed all JDO code and dependencies, removed all DataNucleus/JDO/JPA related jars from the libs directory, turned off the enhancer, integrated objectify and deployed a new version. I logged 10 more cold start requests after that.

I'm going to report 3 values for each framework: shortest, longest and average. This should give a fairly good overview of the results.

JDO Objectify
Shortest 5.6 s 4.2 s
Longest 11.7 s 11.5 s
Average 7.8 s 7.2 s

There seems to be a slight improvement with objectify, but it is honestly very hard to tell since the range of values I got varied from 4-5 to 11 seconds in both cases. I really don't understand how the load time can vary that much. I have read all the documentation, faqs and threads I can find on this matter, and I still don't quite understand why the hell it takes up to 11 seconds to load an app like TeamScape. We only have one servlet that in my latest deployed version only does some guice stuff. I haven't measured the load time without Guice, so not sure how much it adds to the total load time. I give up on this for now, or this post will never be finished. But trust me, we will come back to this later when we look at performance and optimizations.

CRUD performance

Alright, time to look into datastore operations performance.

I have compared JDO and objectify performance for the typical cases of saving an entity, reading an entity, updating an entity, deleting an entity and querying for set of entities. Below you can find the code that I used for testing each of the operations on both JDO and objectify. The listed code is what differed between the frameworks. I have omitted common code such as try-catch, error handling etc. Note that each operation is implemented in its own method. I have just collected the code in one place here for convenience.

JDO operations

// Create/Update
  PersistenceManager pm = PMF.get().getPersistenceManager();
  Model savedModel = pm.makePersistent(model);

// Read
  PersistenceManager pm = PMF.get().getPersistenceManager();
  Model model = (Model) pm.getObjectById(className, encodedKey);
  model = pm.detachCopy(model);

// Delete
  PersistenceManager pm = PMF.get().getPersistenceManager();
  pm.deletePersistent(model);

// Query
  PersistenceManager pm = PMF.get().getPersistenceManager();
  Query query = pm.newQuery(queryString);
  List<Model> results = (List<Model>) query.executeWithArray(parameterValues);
  results = (List<Model>) pm.detachCopyAll(results);

objectify operations

// Create/Update
  Objectify ofy = ObjectifyService.begin();
  Key<? extends Model> key = ofy.put(model);

// Read
  Objectify ofy = ObjectifyService.begin();
  Model model = ofy.get(className, id);

// Delete
  Objectify ofy = ObjectifyService.begin();
  ofy.delete(model);

// Query
  Objectify ofy = ObjectifyService.begin();
  Query<? extends Model> q = ofy.query(className).filter(fieldName, fieldValue);
  List<Model> models = new ArrayList<Model>();
  for (Model m : q) {
     models.add(m);
  }

Each operation was run 50 times, and I ran the whole test scenario at 5 different occasions for each framework. This means that each operation was run 250 times in total for each framework. The datastore was of course cleared between each test run for a fair comparison. All the tests were carried out completely on server side, so no network traffic had any impact on the results. The reason I chose 50 iterations was simply because a larger number resulted in a DeadlineException since the request lasted more than 30 seconds. In order to increase the number of test runs (and possibly the accuracy of the results), each test operation would need to be handled in a separate request, which I did not have time for.

Here are the average run times for each operation. Unlike the cold start measurements, the values here were quite consistent over the different test runs, so it makes sense to present the results as an average value.

JDO Objectify
Create 57 ms 55 ms
Read 25 ms 20 ms
Update 56 ms 68 ms
Delete 71 ms 52 ms
Query 32 ms 26 ms

I didn't expect these figures to differ much since most of the time is spent in the datastore for these operations. Objectify seems to be slightly faster on read/query operations, which is what matters most. I was a bit surprised that the objectify update operation was so much slower than JDO, but perhaps there's an explanation for that. Delete is also quite faster with objectify, although that is probably the least time critical operation.

Portability

Let's just briefly discuss this. JDO is designed with portability in mind, and objectify is designed to run on GAE, so it's probably safe to say that a JDO solution is more portable by definition. However, this is not the whole truth as far as I'm concerned. The DataNucleus/GAE JDO implementation has lots of extensions and unsupported features that requires workarounds in the GAE environment. This means that any somewhat complex application probably requires some adaptation effort to run in another JDO compliant environment. The effort needed is of course related to the complexity of the database model.

Conclusion

Time to conclude what we have discussed. These observations are of course related to the GAE environment in specific, and not in general.

Documentation: Both frameworks are well documented on how they relate to GAE. It's a tie.

Features: Both frameworks has nice feature sets and most/all of the datastore features are exposed in both cases. A tie, I would presume.

Simplicity: objectify. The API, documentation and usage is much more straightforward with objectify.

Getting it up and running: objectify is a clear winner. A GAE newbie could probably get objectify up and running in a matter of hours. Not so much with JDO, no.

Cold start performance: Inconclusive. But an objectify solution can never be slower than a JDO solution, so a theoretical win for objectify :-) .

CRUD performance: Slightly in favor of objectify due to better read/query performance.

Portability: JDO. Obviously...

To sum up, if you are getting started with GAE and looking into a datastore solution, you should definitely consider objectify. The only reason to choose JDO on GAE over objectify is if you:

  • Want your app to be engine independent. JDO is the way to go for portability reasons.
  • Have an existing app that already uses JDO and you want to port it to GAE
  • Plan an application that has a very complex database model.
  • Like torturing yourself

TeamScape has a fairly simple database model and is targeted for GAE, so I have decided to make the switch. The main factors for this is simplicity and performance (which, at least in theory, outperforms that of JDO).

I will look into SimpleDS/Twig a bit more, but unless I find some extremly useful feature there that objectify is lacking, we will probably integrate objectify next.

http://sites.google.com/site/slim3appengine/
  • Share/Bookmark

Part 6-1: Datastore/JDO

Posted on February 8, 2010

In this part we will implement basic database handling based on the GAE datastore and the JDO platform.

I will not go into any details on GAE datastore or how JDO works here, since this information is well documented in GAE docs , JDO docs and DataNucleus docs.

We start by defining an interface that our database managers will implement. Let's have a look at our Datastore interface.

public interface Datastore {

	Model getModel(String encodedKey, String className);
	List<Model> getModelsByQuery(DBQuery query);
	Model saveModel(Model model);

	void createTestEntities();
}

We have defined methods to retrieve single model instances by key, or a list of models based on a custom query. We also define a method to save/update models. Nothing strange here. We will have a look at DBQuery soon.

We have a new interface Model (that replaces PersistentData), which all our models/JDO entities must implement. In many cases we will be able to work with the Model directly, but we will also need to cast them to their respective implementation regularly. I decided not to use generics for the model handling for code simplicity. I have bumped into issues with how GWT-RPC handles generics before and spent way too much time debugging that for it to be worth it. We generally don't have to worry about type checking anyways due to how we fetch models via the command pattern. I love generics, but sometimes it leads to more complexity than necessary. Perhaps I'm a duct tape programmer wannabe? ;-)

Now we add an implementation of Datastore that we name GAEDatastore to reflect that is uses GAE datastore for persistent storage.

public class GAEDatastore implements Datastore {

	public GAEDatastore() {
		// TODO: remove later
		createTestEntities();
	}

	@SuppressWarnings("unchecked")
	@Override
	public Model getModel(String encodedKey, String className) {

		System.out.println("GAE getModel");
		PersistenceManager pm = PMF.get().getPersistenceManager();
		Model model = null;

		try{
			// TODO: Class.forName performance?
			Class cl = Class.forName(className);
			model = (Model)pm.getObjectById(cl, encodedKey);
			model = (Model)pm.detachCopy(model);
		}
		catch(JDOException e) {
			e.printStackTrace();
		}
		catch(ClassNotFoundException e) {
			e.printStackTrace();
		}
		finally {
			pm.close();
		}

		return model;
	}

	@SuppressWarnings("unchecked")
	@Override
	public List<Model> getModelsByQuery(DBQuery dbQuery) {

		System.out.println("GAE getModelsByQuery");
		PersistenceManager pm = PMF.get().getPersistenceManager();
		String queryString = dbQuery.toString();
		System.out.println("query: " + queryString);
		List<Model> results = null;

		try{
			Query query = pm.newQuery(queryString);

			if(dbQuery.getParamValues() != null)
				results = (List<Model>) query.executeWithArray(dbQuery.getParamValues().toArray());
			else
				results = (List<Model>) query.execute();

			results = (List<Model>) pm.detachCopyAll(results);
		}
		catch(JDOException e) {
			e.printStackTrace();
		}
		finally {
			pm.close();
		}

		return results;
	}

	@Override
	public Model saveModel(Model model) {

		System.out.println("GAE saveModel");
		PersistenceManager pm = PMF.get().getPersistenceManager();
		Model savedModel = null;

		try {
			savedModel = pm.makePersistent(model);
		} catch (JDOException e) {
			e.printStackTrace();
		} catch (Exception e) {
			e.printStackTrace();
		} finally {
			pm.close();
		}

		return savedModel;
	}

	@Override
	public void createTestEntities() {

		DBQuery query = DBQuery.selectFrom(Team.class.getName());
		List<Model> models = getModelsByQuery(query);

		// only add test stuff if not already done
		if(models.size() != 0)
			return;

		System.out.println("createTestEntities");
		Person person = new Person();
		person.setFirstName("John");
		person.setLastName("Doe");
		person.setEmail("john.doe@db.com");
		person.setPhone("+123456789");
		person.setBirthday(new Date());
		person.setAdminLevel(0);
		person.setPassword("tempPw");

		Person savedPerson = (Person)saveModel(person);;

		Team team = new Team();
		team.setName("The testers");
		team.setOwnerPersonKey(savedPerson.getId());
		Team savedTeam = (Team)saveModel(team);

		TeamPlayer player = new TeamPlayer();
		player.setPersonKey(savedPerson.getId());
		player.setTeamKey(savedTeam.getId());
		player.setPersonName(person.toString());
		player.setActive(true);
		player.setJoinDate(new Date());
		player.setPosition("Forward");
		player.setShirtNumber("99");
		player.setOtherInfo("Test player");
		saveModel(player);
	}

}

Nothing fancy here. We haven't added any error handling yet. It might be worth noting that we detach our models before we return them to the client. This allows us to update the models on client side and then save them directly to the datastore.We also need to declare our models as detachable, which we will see in the next post.

We fetch our PersistenceManager instances via a class named PMF, for PersistanceManagerFactory. This is a standard setup for retrieving persistence managers. There are probably ways to inject the persistance manager as well, but I don't see any reason for that as of now.

We do want the Datastore implementation to be injected in our dispatch handlers however, so we must also add a binding in our server module.

	bind(Datastore.class).to(GAEDatastore.class).in(Singleton.class);

Now we must also define our Action, Result and ActionHandler implementations to use with gwt-dispatch. We define two basic types: GetModel and ModelsQuery. GetModel corresponds to fetching a model by key from the datastore. ModelsQuery uses a DBQuery for more complex scenarios where we need to fetch lists of models based on any field, etc, etc. The Actions and Results are not really interesting (browse the source if you want), but let's have a look at one of the handlers that now uses the datastore.

public class GetModelHandler implements ActionHandler<GetModel, GetModelResult> {

	private Datastore mDatastore;

	@Inject
	public GetModelHandler(Datastore datastore) {
		this.mDatastore = datastore;
	}

	@Override
	public GetModelResult execute(GetModel arg0, ExecutionContext arg1)
			throws ActionException {

		return new GetModelResult(mDatastore.getModel(arg0.getId(), arg0.getClassName()));

	}

	@Override
	public Class<GetModel> getActionType() {
		return GetModel.class;
	}

	@Override
	public void rollback(GetModel
			arg0, GetModelResult arg1,
			ExecutionContext arg2) throws ActionException {

	}

}

We inject a Datastore implementation and uses it in execute() to fetch our models.

Now, let's take a look at the DBQuery class that we can use to build JDO queries.

public class DBQuery implements Serializable {

	// Types
	public static final String SELECT = "select from ";
	public static final String DELETE = "delete from ";

	// Filter operators
	public static final String EQUALS = " == ";
	public static final String NOT_EQUAlS = " != ";
	public static final String GREATER_THAN = " > ";
	public static final String LESS_THAN = " < ";
	public static final String GREATER_THAN_OR_EQUALS = " >= ";
	public static final String LESS_THAN_OR_EQUALS = " <= ";

	// Logical operators
	public static final String AND = " && ";
	public static final String OR = " || ";

	public static final String ASCENDING = " asc ";
	public static final String DESCENDING = " desc ";

	private static final String COMMA_SEPARATOR = ", ";

	private String mType;
	private String mClassName;
	private List<QueryFilter> mQueryFilters;
	private List<Serializable> mParamValues;
	private String mOrdering;
	private Integer mRangeStart;
	private Integer mRangeEnd;

	static class QueryFilter implements Serializable {
		private String mFieldName;
		private String mOperator;
		private String mParamName;
		private String mParamType;
		private String mLogicalSeparator;

		public QueryFilter() { }

		public QueryFilter(String fn, String op, String pn, String pt,
				String ls) {
			mFieldName = fn;
			mOperator = op;
			mParamName = pn;
			mParamType = pt;
			mLogicalSeparator = ls;
		}
	}

	// Builder methods to build a complete query

	public static DBQuery selectFrom(String className) {
		return new DBQuery(DBQuery.SELECT, className);
	}

	public static DBQuery deleteFrom(String className) {
		return new DBQuery(DBQuery.DELETE, className);
	}

	public DBQuery where(String classField, String fieldType, String operator, String value) {
		addFilter(classField, operator, classField + "Param", fieldType, value);
		return this;
	}

	public DBQuery and() {
		addLogicalOperator(DBQuery.AND);
		return this;
	}

	public DBQuery or() {
		addLogicalOperator(DBQuery.OR);
		return this;
	}

	public DBQuery orderBy(String field, String order) {
		addOrdering(field, order);
		return this;
	}

	public DBQuery withRange(int start, int end) {
		setRange(start, end);
		return this;
	}

	// End builder methods

	protected DBQuery(){
		mQueryFilters = new ArrayList<QueryFilter>();
	}

	protected DBQuery(String cl) {
		this();
		mClassName = cl;
		mType = DBQuery.SELECT;
	}

	protected DBQuery(String type, String cl) {
		this(type);
		mClassName = cl;
	}

	@Override
	public String toString() {

		StringBuffer buffer = new StringBuffer();

		buffer.append(mType);
		buffer.append(mClassName);

		if (mQueryFilters.size() > 0) {

			buffer.append(" where ");

			// Add filters
			for (QueryFilter filter : mQueryFilters) {

				buffer.append(filter.mFieldName);
				buffer.append(filter.mOperator);
				buffer.append(filter.mParamName);

				if (filter.mLogicalSeparator != null)
					buffer.append(filter.mLogicalSeparator);
			}

			buffer.append(" parameters ");

			// Add param declarations
			boolean addComma = false;
			for (QueryFilter filter : mQueryFilters) {

				if (addComma)
					buffer.append(", ");
				buffer.append(filter.mParamType);
				buffer.append(" ");
				buffer.append(filter.mParamName);
				addComma = true;
			}
		}

		// Add ordering
		if (mOrdering != null) {
			buffer.append(" order by ");
			buffer.append(mOrdering);
		}

		return buffer.toString();
	}

	protected void addFilter(String fieldName, String operator,
			String paramName, String paramType, String paramValue) {
		addFilter(fieldName, operator, paramName, paramType, paramValue, null);
	}

	protected void addFilter(String fieldName, String operator,
			String paramName, String paramType, String paramValue,
			String logical) {
		addFilter(new QueryFilter(fieldName, operator, paramName,
				paramType, logical));

		if (mParamValues == null)
			mParamValues = new ArrayList<Serializable>();
		mParamValues.add(paramValue);
	}

	private void addFilter(QueryFilter filter) {
		mQueryFilters.add(filter);
	}

	private void addLogicalOperator(String logical) {
		QueryFilter lastFilter = mQueryFilters.get(mQueryFilters.size() - 1);
		lastFilter.mLogicalSeparator = logical;
	}
// removed setters/getters

}

We can now build queries on the client side without caring about anything other than the model. Let's have a quick look at how we can use it too.

DBQuery query = DBQuery.selectFrom(Team.class.getName()); // Get all teams
DBQuery query = DBQuery.selectFrom(TeamPlayer.class.getName()).
where(TeamPlayer.TEAMKEY_FIELD, Model.KEY_TYPE, DBQuery.EQUALS, teamId); // Get all players for a team
// We can also use add()/or() to add more where clause logic.
// And orderBy() to sort, and withRange() to get a specific range

Don't miss the comments below. There are some interesting comments regarding security and objectify.

That was it for this part. In the next part we will take a look at some new models and using what we implemented here.

  • Share/Bookmark
Tagged as: , , 6 Comments

Part 5: RPC with gwt-dispatch

Posted on February 2, 2010

Now that we have GIN setup, we will integrate the gwt-dispatch library to handle our client/server communication. We will also enable Guice for dependency injection on server side.

What is gwt-dispatch? It is a GWT implementation of the command pattern that wraps the GWT-RPC mechanism.

Earlier we defined a model Person which simply reprents a user in the system. Since we had no RPC setup, we had a dummy implementation on the client side to create a Person instance for testing purposes. Now we will move the dummy implementation to server side and fetch Person instances over the wire. It will be a lot of dummy stuff until we have defined a database model and implemented it.

Let's start by creating the Command/Response objects, which in gwt-dispatch are named Action/Result because there is already a Command class in GWT. It wouldn't really be a problem since they reside in different packages, but I assume the gwt-dispatch people wanted to avoid confusion, so let's just roll with it. We have created a new subpackage named rpc in the shared package where we will keep all these classes.

public class GetPerson implements Action<GetPersonResult> {

	private String id;

	GetPerson() { }

	public GetPerson(String id){
		this.id = id;
	}

	public String getId(){
		return id;
	}

}
public class GetPersonResult implements Result {

	private Person person;

	GetPersonResult() { }

	public GetPersonResult(Person person){
		this.person = person;
	}

	public Person getPerson() {
		return person;
	}

}

Nothing fancy going on here. The GetPerson class holds the id of the Person we want to retrieve. It implements the Action interface parameterized with GetPersonResult. The GetPersonResult simply holds the Person instance that was fetched from the server and it implements Result. I'm not sure which path we will take later, but I suspect that we will use some generics to avoid creating GetX/GetXResult/SetX/SetXResult objects for each model. Note that we need no-args constructors in all these classes for the serialization process.

We will now look at two more new classes on the client side which will be used for all RPC. As provided in the GreetMVP example, we define a class CachingDispatchAsync where we will keep a map of Actions that maps to Results. Each time we succesfully fetch something from the server, we cache it for subsequent queries.

public class CachingDispatchAsync implements DispatchAsync {

	private DispatchAsync mDispatcher;
	private Map<Action<Result>, Result> mCache = new HashMap<Action<Result>, Result>();

	@Inject
	public CachingDispatchAsync(final DispatchAsync dispatcher) {
		this.mDispatcher = dispatcher;
	}

	public <A extends Action<R>, R extends Result> void execute(final A action, final AsyncCallback<R> callback) {
		mDispatcher.execute(action, callback);
	}

	@SuppressWarnings("unchecked")
	public <A extends Action<R>, R extends Result> void executeWithCache(final A action, final AsyncCallback<R> callback) {
		final Result r = mCache.get(action);

		if (r != null) {
			callback.onSuccess((R) r);
		}
		else {
			execute(action, new AsyncCallback<R>() {

				public void onFailure(Throwable caught) {
					callback.onFailure(caught);
				}

				public void onSuccess(R result) {
					mCache.put((Action) action, (Result) result);
					callback.onSuccess(result);
				}

			});
		}
	}

	public void clear() {
		mCache.clear();
	}
}

We implement the DispatchAsync interface, but also have a DispatchAsync member instance that we use to actually call the server. An instance of StandardDispatchAsync will be injected for us. Besides that, the class is pretty straightforward.

Inspired by how the Apache HUPA project handles callbacks, we also define a DispatchCallback class which implements the AsyncCallback interface. We can implement application wide error handling in this class when appropriate, and clients can choose to override the default failure callbacks if needed.

public abstract class DispatchCallback<T> implements AsyncCallback<T> {

	public DispatchCallback() {

	}

	@Override
	public void onFailure(Throwable caught) {
		callbackError(caught);
	}

	@Override
	public void onSuccess(T result) {
		callback(result);
	}

	/**
	 * Must be overriden by clients to handle callbacks
	 * @param result
	 */
	public abstract void callback(T result);

	/**
	 * Should be overriden by clients who want to
	 * handle error cases themselves.
	 */
	public void callbackError(Throwable t) {
		t.printStackTrace();
		Window.alert("RPC failed: " + t.toString());
	}
}

As you can see, we don't currently do much here, but it will save us time to have it in place from the start. We will want to handle user session stuff here later among other things.

Before we can use these classes, we must also let GIN know about them in our client module, so we add the following bindings/modules.

// In TeamScapeClientModule
bind(CachingDispatchAsync.class).in(Singleton.class);

// In TeamScapeGinjector
@GinModules({TeamScapeClientModule.class, StandardDispatchModule.class})

Now we can use the dispatch service when we query for Person instances.

		mDispatcher.executeWithCache(new GetPerson("1"), new DispatchCallback<GetPersonResult>(){

			@Override
			public void callback(GetPersonResult result) {
				Person person = result.getPerson();
				ArrayList<NameValuePair<String>> list = Person.getAsList(person);
				display.setData(list);
			}

		});

OK. Client side done. Let's move on to the server side and a bit more unknown territory. We have created two new subpackages named rpc and guice. The rpc package will contain all the dispatch action handlers, and the guice package injection and binding stuff.

This is how the GetPersonHandler currently looks. It will of course change once we have the database up and running.

public class GetPersonHandler implements ActionHandler<GetPerson, GetPersonResult>{

	public GetPersonHandler() {
	}

	@Override
	public GetPersonResult execute(GetPerson arg0, ExecutionContext arg1)
			throws ActionException {

		// Dummy impl until we have setup database stuff
		Person person = new Person();
		person.setId(arg0.getId());
		person.setFirstName("John");
		person.setLastName("Doe");
		person.setEmail("john.doe@serverside.com");
		person.setPhone("+0123456");

		return new GetPersonResult(person);
	}

	@Override
	public Class<GetPerson> getActionType() {
		return GetPerson.class;
	}

	@Override
	public void rollback(GetPerson arg0, GetPersonResult arg1,
			ExecutionContext arg2) throws ActionException {
		// Nothing to do on getters
	}

}

The interesting stuff is in the execute() method where we have the Action object as input. In the cases where we write to the database, we will also want to implement the rollback() method.

Moving onto the guicy stuff. First, we have to define an instance of ServletModule where we configure our servlets. This basically corresponds to what we do in the web.xml file with servlet-mapping tags for regular GWT RPC services.

public class DispatchServletModule extends ServletModule{

	@Override
	public void configureServlets() {
		serve("/teamscape/dispatch").with(GuiceStandardDispatchServlet.class);
	}
}

The dispatch library offers a variety of dispatch servlet instances for different use cases. We will use the standard guice one for now.

Next, we define an implementation of the ActionHandlerModule which is where we will bind our Actions to handlers.

public class ServerModule extends ActionHandlerModule {

	@Override
	protected void configureHandlers() {
		bindHandler(GetPerson.class, GetPersonHandler.class);
	}

}

Finally, we will implement our guice servlet config that inherits from GuiceServletContextListener and where we override the getInjector() method.

public class GuiceServletConfig extends GuiceServletContextListener {

	@Override
	protected Injector getInjector() {
		return Guice.createInjector(new ServerModule(), new DispatchServletModule());
	}
}

Here we provide our ServletModule and ActionHandlerModule implementation as input for Guice.createInjector().

We also need to add some stuff to web.xml for Guice and add the Guice libraries to our war/WEB-INF/lib directory. Now we are ready to go!

When we visit http://dev.teamscape.se, we can now see that John Doe's email domain has changed to @serverside.com, which is our sign of success! Note that it is quite slow at the moment since Google App Engine terminates the VM during no activity, so it is likely that it has to cold start when you visit it.

  • Share/Bookmark
Tagged as: , , , 1 Comment

Part 6-2: Entities #1

Posted on January 21, 2010

In this part we will take a look at our database model and entity relationships. We will build the entire entity relationship tree in steps and this post will cover the first step where we look at how teams, persons and players relate to each other. In future posts, we will add new entities and relationships to the tree until we have satisfied all the use cases from a database perspective.

Ok. So let's have a look at the relationship diagram for these entities. I have not had time to make a fancy ER diagram, so this ugly one will have to do for now. The important thing is that you understand the relationship.

We should probably discuss the general strategy for our model relationships. We can use owned or unowned model relationships. An owned relationship is when you define an instance of the child model in the parent model. This is the normal "Java way" of defining relationships. We can also use the "typical SQL way" of defining relationships and keep a parent key reference in the child model (which is not really a child in this case from a datastore perspective). Using owned relationships also brings entity groups and fetch groups into play when using the GAE datastore.

The main factors that influence which path we take here is what kind of dependency the models have on each other, how the datastore handles owned relationships and if we need transactions. Reading material: Relationships, article serie on scaling, contention.

We might want to present data in different ways. Say we want to show information of a team. We want to display its players, leagues, news, guestbook, forum, events, statistics, etc, but it is likely that this information is presented on different pages. So, do we want to fetch all that information as soon as we enter the main team page? Probably not. Are the models strongly dependent on each other? Nope. Do we have a need for transactions here? Not really, no. So in this case, it really makes no sense to use owned relationships. Owned relationships should really only be used when the entities have some form of strong dependency between each other. Jason@Google sums this up pretty well here. So, our basic strategy will be to use unowned relationships unless we find a very good reason to use owned in certain cases.

Let's move on by taking a look at our Team model. I have stripped the classes of setters/getters as usual.

@PersistenceCapable(identityType = IdentityType.APPLICATION, detachable="true")
public class Team implements Model{

	@PrimaryKey
    @Persistent(valueStrategy = IdGeneratorStrategy.IDENTITY)
    @Extension(vendorName="datanucleus", key="gae.encoded-pk", value="true")
    private String encodedKey;

    @Persistent
	private String name;

    @Persistent
	private String ownerPersonKey;

    @NotPersistent
    private ArrayList<TeamPlayer> players;

    public Team() {

    }

    @Override
	public String getId() {
		return encodedKey;
	}

	@Override
	public String getDisplayName() {
		return name;
	}

}

We use JDO annotations with DataNucleus/GAE extensions (GAE uses the DataNucleus JDO implementation) to define our model as persistent. Let's quickly go over the different annotations this once, since we will be using them on all our models.

@PersistenceCapable(identityType = IdentityType.APPLICATION, detachable="true")

We annotate all our models with this to tell JDO that it is a JDO entity and that we want them to be detachable (which we discussed in the previous post). Let's have a look at how we define our keys.

    @PrimaryKey
    @Persistent(valueStrategy = IdGeneratorStrategy.IDENTITY)
    @Extension(vendorName="datanucleus", key="gae.encoded-pk", value="true")
    private String encodedKey;

We use a GAE extension for our model keys which is an encoded string. All these annotations has no meaning on the client side and we want our server side implementations to be as transparent as possible to the client side. We can't use the Key class for that reason and the Long key variants are too limited since we might want owned relationships. So, encoded keys is the way to go. Our model interface enforces our models to implement getId(), which should return this encoded key.

We annotate all fields we want to be persistent with the @Persistent tag. This is actually not needed for most types, but I prefer to explicitly state it anyway. For fields we don't want to be persistent, we use the @NotPersistent tag.

OK. So let's have a look at the Person model again. It now contains some more fields and JDO annotations.

@PersistenceCapable(identityType = IdentityType.APPLICATION, detachable="true")
public class Person implements Model {

	@PrimaryKey
    @Persistent(valueStrategy = IdGeneratorStrategy.IDENTITY)
    @Extension(vendorName="datanucleus", key="gae.encoded-pk", value="true")
    private String encodedKey;

	@Persistent
	private String firstName;

	@Persistent
	private String lastName;

	@Persistent
	private String email;

	@Persistent
	private String phone;

	@Persistent
	private Date birthday;

	@Persistent
	private String password;

	@Persistent
	private Integer adminLevel;

	public Person(){

	}

	@Override
	public String toString() {
		return getDisplayName();
	}

	@Override
	public String getId() {
		return encodedKey;
	}

	@Override
	public String getDisplayName() {
		return firstName + " " + lastName;
	}

}

Now we'll have a look at TeamPlayer, which is the connection between a Team and a Person.

@PersistenceCapable(identityType = IdentityType.APPLICATION, detachable="true")
public class TeamPlayer implements Model {

	public static final String TEAMKEY_FIELD = "teamKey";
	public static final String PERSONKEY_FIELD = "personKey";

	@PrimaryKey
    @Persistent(valueStrategy = IdGeneratorStrategy.IDENTITY)
    @Extension(vendorName="datanucleus", key="gae.encoded-pk", value="true")
    private String encodedKey;

	@Persistent
	private String teamKey;

    @Persistent
    private String personKey;

    @Persistent
    private String personName;

    @Persistent
    private Boolean active;

	@Persistent
	private String position;

	@Persistent
	private String shirtNumber;

	@Persistent
	private Date joinDate;

	@Persistent
	private String otherInfo;

    public TeamPlayer() {

    }

    @Override
	public String getId() {
		return encodedKey;
	}

	@Override
	public String getDisplayName() {
		return personName;
	}

}

We have key references both to a Team and a Person. A Team can have many players, and a Person can play in several teams. So, if we want to get all players for a team, we create a query where we fetch all players where teamKey = our team key. If we want to see which teams a Person is a player in, we do the same with the person key.

Also note that we denormalize person name data here. We will need to do this occasionally since GAE datastore has no support for joins. Joins can be pretty expensive anyway, so it it not a big problem as long as we make sure to minimize the amount of duplicated fields. If we didn't have a person name field in TeamPlayer, we would need to fetch the whole Player entity just to be able to present the player.

Since we perform queries based on an entities field names, we have also defined two constants TEAMKEY_FIELD and PERSONKEY_FIELD that contains the names of the fields. We use these when we build our queries.

That were all the models for this time. In the next entity part, we will look at entities for News, Forum, GuestBook, etc that we will want to have for a team page.

  • Share/Bookmark
Tagged as: , , 2 Comments