Note that this post is not as comprehensive as my previous posts about gwt-presenter. This is mainly due to the fact that gwt-platform is still evolving at a rapid pace, which means that whatever I write here will likely be outdated soon. Once gwt-platform reaches a 1.0 state with a somewhat finalized API, I will likely write another post with more details on both the framework and how TeamScape uses it.

So what is gwt-platform? It is a complete MVP + Command pattern framework forked off gwt-presenter/dispatch by Philippe Beaudoin for his project PuzzleBazar.

Why not mvp4g? One reason alone: gwt-platform is based on gwt-presenter and share many traits with it. I have spent a lot of time learning gwt-presenter and TeamScape is currently using gwt-presenter, so the transition for me to gwt-platform makes much more sense than moving to a completely unknown territory. I have however looked into mvp4g and it seems like a very nice alternative as well. If you are looking for a MVP framework for a new application, I would recommend you to spend some time evaluating the alternatives.

You might be wondering which problems I see with the current state of gwt-presenter?

• Presenter relationships: Child presenters are injected in parent which leads to tight coupling between your presenters. This also makes it harder to structure your application for certain scenarios.
• Lack of lazy loading: It does not have built in support for lazy loading of presenters/views, which means you have to to implement this in your application layer.
• Only singleton presenters: No support for non-singleton presenters which decreases the level of flexibility in your application design.
• Places implementation: Too much code to handle places (one class for each place, etc). Also, each request leads to quering each place if it is a match which can be a bottleneck (described this in Part 7).
• Code splitting: No support for code splitting.
• Development activity: The last "real" checkin was in beginning of January. Although the authors have been active in the forum, it seems that development has grinded to a halt.

I'm sure that gwt-presenter will evolve past many of these problems, but I have waited long enough. I need to get my foundation in place now so I can stop worrying about that.

I have now rewritten all TeamScape client code from scratch using gwt-platform instead of gwt-presenter/dispatch. Was it worth it? Oh yes, definitely. So what is so compelling with gwt-platform that I decided to throw out a well-established framework and spend a whole weekend getting back to where I was?

• Loose coupling between presenters. There are no direct dependencies between parents and children. All communication takes place over the event bus and the child simply fires an event that a parent presenter reacts to. Want to change parent for a child? You change the event type being fired in the child..that's it. You can also reveal the child in different parents with this based on the context, for example token parameters. Parents annotate defined events with @ContentSlot where child presenters are injected when they fire that event.
• Lazy loading. gwt-platform introduces the concept of light-weight proxies. These proxies listen to events and instantiates your presenters on demand. Your presenter/view is therefore not created until the first request.
• Non-singleton presenters. Singleton presenters does not always make sense. Sometimes you want to re-use presenters in different contexts, and this is where PresenterWidgets come in handy.
• Code splitting. Code splitting could not possibly be easier. You simply annotate your proxy with @ProxyCodeSplit, and the initialization of the presenter is behind a code split point. Incredibly easy and powerful.
• Place management. Easier than ever. With three lines of code and no new classes, you have mapped your presenter to a place. The gwt-presenter problem with quering every place if it's a match is avoided as well.
• Search engine crawling: Integrated support for letting search engines crawl your GWT app (still in experimental phase).
• Integrated command pattern for RPC communication.

I struggled a bit on how to design my application to handle the "3 pages" problem with gwt-presenter. With the code splitting feature and presenter architecture of gwt-platform, I quickly found a solution that I like. I now have three different PagePresenters that each map to a PageView that holds a DockLayoutPanel. Each PagePresenter is behind a code split point, so the user only downloads the relevant page content. If the user goes from say the main page to a team page, the MainPageView is swapped out at the root level and in comes TeamPageView.

The non-singleton presenters also makes it much easier to define a single "shared" presenter/view pair that can be instantiated for each page. This will be very useful for TeamScape since we can avoid redundant code that basically does the same thing in different contexts.

I thought long and hard about how to represent the "static" content on each page. With this I mean the content that is in the north, south, west and east directions in my layout that doesn't change as the result of a place request. The content may be updated as the result of user interaction, but it is never related to place management. The only feasible solution that I came up with was to let the PagePresenter "own" these presenters (gwt-presenter style). They are injected in the PagePresenter constructor and then initialized on the first request. Note that the views are still happily unaware of the content they are presenting at this level. I did post a question about possible solutions in the gwtp forum here. Issue 78 for gwt-platform could be an interesting alternative solution.

Let's have a look at some examples on how you use gwt-platform.

We have two types of presenters: Presenter (singletons with view/proxy/place) and PresenterWidget (non-singleton, view only). Which one you define depends entirely on what you want to accomplish. You will always use singleton presenters to handle place requests, code splitting and revealing in parents. So when do you want to use PresenterWidgets? Say you have an application where users can post comments in many different places (via different presenters). In this case, you might want to define a CommentPresenterWidget mapped to a CommentView to handle comments. Instead of duplicating code in different places, you inject a new instance of CommentPresenterWidget in each presenter where you need this logic. There are other scenarios as well, but you should be getting the idea by now.

The interesting part of gwt-platform is the presenter-proxy handling. Let's have a look at a typical singleton presenter that maps to a place and handles token parameters.

public class ShowPlayerPresenter extends TeamBasePresenter<ShowPlayerPresenter.MyView, ShowPlayerPresenter.MyProxy> {

    private String mPlayerId;
    private String mLastId;

    public interface MyView extends View {
      // methods to feed view with player data
    }

    @ProxyCodeSplit
    @NameToken(TSNameTokens.teamShowPlayer)
    public interface MyProxy extends Proxy<ShowPlayerPresenter>, Place {
    }

    @Inject
    public ShowPlayerPresenter(EventBus eventBus, MyView view, MyProxy proxy,
            DispatchAsync dispatcher, TokenFormatter formatter, CurrentTeam curTeam) {
        super(eventBus, view, proxy, dispatcher, formatter, curTeam);
    }

    @Override
    public void onReset() { // Read the comments in the post regarding onReset() / onReveal()
        super.onReset();
        if (mPlayerId != null && mPlayerId != mLastId) {

            Player player = mCurrentTeam.getPlayerById(mPlayerId);

            if (player == null) {

                mDispatcher.execute(new GetModel(mPlayerId, Player.class, Player.isNumericId()),
                        new DispatchCallback<GetModelResult>() {

                            @Override
                            public void callback(GetModelResult result) {
                                Player player = (Player) result.getModel();
                                displayPlayer(player);
                            }

                        });
            }
            else {
                displayPlayer(player);
            }

            mLastId = mPlayerId;

        }

    }

    private void displayPlayer(Player player) {
        // feed the view with data via MyView interface
    }

    @Override
    protected void revealInParent() {
        RevealContentEvent.fire(eventBus, TeamPagePresenter.TYPE_TeamCenterContent, this);
    }

    @Override
    public void prepareFromRequest(PlaceRequest request) {
        mPlayerId = request.getParameter(TSParamTokens.PLAYER_ID, null);
        super.prepareFromRequest(request);
    }

    @Override
    public PlaceRequest prepareRequest(PlaceRequest request) {
        if (mPlayerId != null) {
            request = request.with(TSParamTokens.PLAYER_ID, mPlayerId);
        }
        return super.prepareRequest(request);
    }

}

The presenter above is responsible for showing information about a player in TeamScape. Let's go over how this works.

It inherits from TeamBasePresenter (which in turn inherits from PresenterImpl). All my "center" team presenters inherit from TeamBasePresenter in order to extract team token parameters in one place and update team data accordingly. You can see that I've used @ProxyCodeSplit and @NameToken to annotate MyProxy. This tells gwt-plattform that this presenter should sit behind a code split point and also be activated on a place request with the token defined in TSNameTokens.teamShowPlayer. You can also use @ProxyStandard, which means that the code will be downloaded by default. If you don't want your presenter to be mapped to a place, you simply remove the @NameToken annotation and remove the Place extension to the MyProxy interface.

If you come from gwt-presenter, you might remember onRevealDisplay()?. With gwt-platform, we have both onReset() and onReveal() that can be overriden. Read the comment below by Philippe on this. Use onReset() if you are unsure. We override this to update the view with relevant data (perhaps based on token parameters). In this case,  I check if we already have fetched the player. If not, we dispatch a call to the server to fetch it from the datastore. Then I call displayPlayer() which updates the view with the new player data.

The method revealInParent() is a new addition in gwt-platform. The relationship between parent and child presenters are entirely different with gwt-platform, and so the logic for it is as well. With gwt-platform, you define a type parameter in your parent like this:

@ContentSlot
    public static final Type<RevealContentHandler<?>> TYPE_TeamCenterContent = new Type<RevealContentHandler<?>>();

When the child presenter is activated via a place request, revealInParent() is called and it is up to the child presenter to fire the appropriate event to get revealed. In this case, we want the presenter to be revealed in the center of the team page, hence TYPE_TeamCenterContent.

You will also recognize the last two methods, prepareFromRequest() (preparePresenter() in gwt-presenter) and prepareRequest(). These were defined in the Place-class in gwt-presenter > v1.1. This can easily lead to confusion for beginners. When a place request is triggered with token parameters, you must first extract the parameters in prepareFromRequest, and then add them again in prepareRequest. Read more about this here.

Now you have seen what a typical singleton presenter looks like. There are lots of other nifty featues in gwt-platform such as secure places, on leave confirmation, search engine crawling, etc. I might come back to all that later and write a post about the more advanced features of gwt-platform, but for now I'm going to focus my time and energy on TeamScape instead. Feel free to ask questions for clarifications or other code examples.

Unfortunately I do not have a part available for download yet and there is no code checked in. Once I'm done migrating away from JDO and implementing "Entities #2", there will be a download for Part 10 - 12.

If you want to get started with gwt-platform, you might want to check out this thread on how to include it in your project.

Interesting reading: Post about shared views, Kyle Baley's post about gwt-platform.

Do checkout the gwt-platform wiki and forum. There are lots of interesting stuff to read. Looking at the issues list, it seems there are lots of nice improvements and new features in the pipe as well.

In this section the authors got the opportunity to write about anything they wanted to summarize what their framework is about to you as a reader.

Sum it up!

Jeff Schnitzer (objectify)

With its transparent mapping to native operations, thorough documentation, clean API, and well commented code, Objectify should be the easiest way to get started using the GAE/Java datastore. Its low-level nature makes it easy to optimize your application for performance and cost without having to hunt through 20 volumes of documentation.

The specific feature set has evolved with real-world experience by several developers working on several different applications.  The strongest points:

• Short learning curve.
• Full and effective use of Java generics.
• Easy second-level caching in GAE's memcache service.
• Solid tools for performing zero-downtime schema migrations.
• Fine control of indexing, including partial indexes.
• GWT-safe entities.
• Active developer community.

John Patterson (Twig)

High performance, easy development:

• Clean data models with no requirements or dependencies
• Works out of the box with no configuration for most data models
• Parallel commands!  Multi task your application
• Many performance optimising features
• Very readable code
• Branched commands allow sorted merged OR queries
• Flexible strategies allows sharing data with JDO, Objectify, SimpleDS

Ignacio Coloma (SimpleDS)

This is the solution I developed to solve my own problems while working with GAE, after battling with Datanucleus for a couple of weeks.

I have been working with persistent relationships since JBoss 2.4 (2001), and again with JPA. That being said, I do not think that relationships are the way to go with GAE, as I prefer having ultimate control about what is loaded and what not. I honestly believe that (with proper caching) this performs better and makes me more productive in a system where you don't have to ask which datasource the data came from.

For me, a persistence framework is more about the features that you _don't_ put into it, which is the best way to let others extend it. This is what we have done with SimpleDS.

My summary

If you, like me, are looking into these frameworks as a replacement for JDO, there are mainly six things to consider. Each point has very different weight depending on your development style and your application requirements.

• Features - Which features do you REALLY need for YOUR project? Sit down and think about the data model and relationships, which queries you will be running, etc. Then write down framework features that you think would make your work easier, and possibly, more efficient.
• Simplicity - Datastore operations are essential for any web application. It should be trivial to perform the basic datastore operations. The API should speak for itself to the highest degree possible. It should be clear how the API relates to the AppEngine concepts.
• Level of control - What level of control do you want? Do you want the framework to automagically do most of the stuff for you or do you prefer to have ultimate control over the flow of events? This is more related to taste than application requirements.
• Extensibility - Do you want to be able to adapt/extend the solution to fit your needs? If so, make sure to evaluate the design and architecture of the framework. Was it designed to support this?
• Performance - If you have a data heavy application, datastore performance is obviously very important. One thing to look out for is that the framework does not add cost to basic datastore calls. Another thing to look out for is what the frameworks offers to work around the limitation/problems with the datastore that affect performance negatively.
• Documentation -  No matter how simple the API is, you still need good documentation that explains different concepts and provides code examples. JavaDoc is an advantage as well.

What it really boils down to is a matter of taste and what requirements you have for your application. Sit down, read the documentation, check out the API and the code examples. Hopefully this and the next article helps you further to find the best fit for your needs.

In the next article I will code up some solutions with each framework and write about my experience with that. I will discuss things more actively from my point of view in that article, which I tried to avoid here to let you, the readers, make up your own mind based on the authors comments.

As always, I encourage feedback and comments. I hope this article is helpful for anyone trying to make a choice in the framework jungle. And once again, big thanks to Jeff, John and Ignacio for participating in this!

This is a checklist of supported features in each framework and JDO (as implemented on GAE).

Do note that this table only is a convenience guide to which features are supported. It does not tell the whole story. If you are considering to switch to one of these frameworks, you should consider all the points mentioned in the summary.

This table will be continously updated as the frameworks evolve. The only legal answers are Yes, No and In progress. Hover the mouse over the dotted underlined features for a popup description.

Latest update: 2010-07-05 (Objectify 2.1, Twig 1.0, SimpleDS 1.0 RC1)

* = JDO/DataNucleus has support for it, but the AppEngine implementation does not. About schema migration.

** = Not directly supported, but supported via the exposed Datastore interface.

Next: Summary

These questions are meant to give a high-level introduction to each framework.

Objectify, Twig and SimpleDS are all frameworks wrapped around the datastore low-level API. When reading this, you might want to complement this text by reading:

• Objectify docs and JavaDoc
• Twig docs
• SimpleDS docs

Describe your framework with just one sentence so that a AppEngine/datastore beginner would understand it.

Jeff Schnitzer (objectify)

Objectify is an easy-to-use framework that represents the GAE datastore close to its true nature:  A persistent HashMap which allows you to put(), get(), delete(), and query() for your Java objects.

John Patterson (Twig)

A simple interface that makes difficult problems easy, faster and more readable while keeping your data models pure POJOS with no dependencies on the low-level datastore.

Ignacio Coloma (SimpleDS)

SimpleDS is a framework that uses the Datastore to persist your own Java classes instead of Entity instances.

What was the number one design goal for your solution?

Jeff Schnitzer (objectify)

Exposing all native datastore features in an easy-to-use, easy-to-understand fashion.

John Patterson (Twig)

Being “feature rich” while maintaining a simple, easy-to-explore API.  Working out of the box with no configuration while allowing optimisations later to be as fast as is possible with the low-level interface.  Actually, by hacking into async queries it is now _faster_ than possible with the low-level API!

Ignacio Coloma (SimpleDS)

Simplicity. You are as close to the Datastore API as possible.

Did you have to make any tradeoffs to support that goal?

Jeff Schnitzer (objectify)

There are features which have been deliberately avoided (managed relationships in particular) because they introduce a significant amount of configuration complexity.  We pride ourselves that the Objectify documentation is both thorough and short.  Features that change either characteristic get a lot of resistance.

John Patterson (Twig)

I specifically chose not to follow the low-level API design in any way which means that developers who are already familiar with it might not feel quite as at home.  The other frameworks follow the low-level API more closely which has the benefit of reusing their experience if they have already worked with the low-level API.

There are two main reasons behind the decision not to follow the low-level API.  Firstly, the fact that software maintenance costs are typically twice the cost of the initial development and normally not performed by you, the initial developer.  You might have needed to learn the low-level API but don't expect that of the maintenance developer who tries to quickly add a new feature in 5 years time.  I believe that the benefit of that familiarity are very low over all.

Secondly, the low-level API is not a good example to follow for an Object Oriented persistence interface.

An example of this is the low-level put() method which has direct counter-parts in Objectify and SimpleDS.  Twig has two separate methods, store() and update() for what the others combine into a single method.  The reason for this separation is that the intent of the two operations is completely different and you are making your intent clear by using one of the other.  If you make a mistake and try to update an instance that is not already persistent an exception will be thrown which helps catch such logic errors early.  If you try and store an existing instance it will not be silently clobbered.

This same pattern is tried and tested in SQL with INSERT and UPDATE and Hibernate with save() and update().

Admittedly, because the other frameworks do not manage instance identity it is not possible for them to know if an instance is already persistent without querying the datastore.  Twig works like Hibernate in this respect - it keeps an internal cache of Key->Instance to ensure that every instance is represented in memory by only one instance.  So every persistent instance in memory must also have been loaded and therefore will be in the cache.  When you call update(), Twig just needs to check that the instance and its Key are already in the Cache.

Both Objectify and SimpleDS do not manage instance identity which means that you can load the same entity into memory multiple times and the framework could never know.

Ignacio Coloma (SimpleDS)

I had to add features for common tasks that are not included in GAE: paged queries and @MultivaluedIndex, for example.

How does your framework help an application perform better in terms of latency and cost?

Jeff Schnitzer (objectify)

Objectify performs some fairly clever static analysis of your entity classes so that loading and saving data into the low-level API should be roughly as fast as equivalent handwritten code.

In terms of latency and cost,

• Batch get(), put(), and delete() operations are essential parts of Objectify's API.  It is both faster and cheaper to batch a number of operations together than to issue separate requests.
• Objectify enables a second-level memcache of your entites with a simple @Cached annotation.  This cache stores negative as well as positive results and works intelligently with batch operations.  It gracefully handles heterogeneous expiration periods.
• Objectify provides easy fine-grained control over indexing, including partial indexing.  Indexes are what make writes expensive. The api_cpu_ms cost to write an entity without indexes is 48ms. Every index adds 17ms to that figure.  This won’t affect latency (indexes are updated in parallel), but it will affect your bill at the end of the week.  If your application is write-heavy, you *must* choose your indexes carefully.
• Objectify has very little "magic" in its API; almost all operations map clearly and transparently to underlying native operations.  You will not inadvertently write code that looks simple but in fact masks several expensive queries.  Code that looks fast will be fast.

John Patterson (Twig)

Parallel queries execute N queries in the time it takes for the slowest to execute.  In other words if you run 10 queries in a request then Twig could reduce the total query time to 10% of what it otherwise would be.

Low-level filters are incredibly useful and exclude unwanted entities from being converted into instances at all.  In GAE this is sadly needed very often because you can only query or sort on a single property.  For example, the query "find all Hotels under $100 sorted by rating" required you to load every instance into memory and then discard the ones over $100.  Converting all those unneeded entities into instances can be very wasteful of CPU resource.

Using Java code and implementing StorageStrategy (or extending the default) gives you precise control over what is indexed and stored.  These flexible configuration options mean it is possible to code your own "partial indexes" solution.

Precise control over the kind name significantly reduces the size of Keys which can save storage costs.

Embedded Collections give a huge performance gain.  They cut out the need to perform many in memory joins that can kill an applications performance.

Ignacio Coloma (SimpleDS)

You can retrieve and store multiple entities using a single API call (same way as you would using the raw Datastore). In that case, unassigned entity keys will be generated using a single API call per entity kind.
We are also adding cache support to SimpleDS, but that's a work in progress.

Is your framework targeted solely for AppEngine, or did you have portability in mind when designing it?

Jeff Schnitzer (objectify)

Just AppEngine.  However, the core mechanism of being able to "objectify" key/value stores is something that could just as well apply to Cassandra, MongoDB, HBase, etc.  One of the Objectify developers, Scott, is actively contributing to MongoDB's Java API now.

While we don't wish to create a single project spanning multiple platforms, we are interested in creating Objectify-ish interfaces for other platforms which would make your porting your application easier - not necessarily automatic.

John Patterson (Twig)

Solely for GAE.  I had portability of the data models in mind – hence pure POJOs – which leads to more portable code which uses that data.  But for querying and manipulating data I want to be as specific as possible to get the most from the platform. An example of this is the method ensureUniqueKey() which is used like this:

Ignacio Coloma (SimpleDS)

It's just for AppEngine.

How easily can a developer extend/override framework logic/features in their own code base? Was this considered when designing it?

Jeff Schnitzer (objectify)

As always, it depends on what you're trying to do.  We don't have a vast API of extension hooks.  We do however, have 1) code that is extraordinarily well commented by the standards of most opensource projects, 2) many classes that are easily extended, and 3) a framework which is actually pretty simple and easy to understand.

John Patterson (Twig)

In my opinion Twig is the most configurable interface because it was designed from the start to be configured in Java code. –  Annotations were added as a convenient implementation that hooks into those extension points.  A developer can add their own TypeConverters, PropertyTranslators (which give exact control over writing/reading objects to low-level properties) and all significant behaviour can be overridden by writing your own Strategy.

A practical example of this is using an implementation of FieldStrategy which allows you to read and write data shared by JDO just by implementing this simple "strategy"  interface to use the same kind and property names:

Voila!  Twig now reads your existing JDO, Objectify, SimpleDS or raw low-level data.

Ignacio Coloma (SimpleDS)

It is trivial to add Converters for your own data types (e.g. Joda classes). SimpleDS is also strongly based on interfaces for their API classes, so a user should be able to replace implementation classes with his own.

Does your framework have any external dependencies? If so, which and why?

Jeff Schnitzer (objectify)

None.  Just add objectify.jar to your project.

John Patterson (Twig)

Only Google Collections which is just so incredibly handy.

Ignacio Coloma (SimpleDS)

We are using the following dependencies:

• spring-core to find persistent classes. We are using Spring as any other library, not as a dependency injection framework. You may use your own for that purpose, or not use class scanning at all and just add your mappings by hand.
• google collections, because life is too short to use raw Java collections. We also use Functions to implement the PagedQuery.transform() feature.
• aspectjrt and aspectjweaver for @Transactional support using aspects.
• commons-logging to, well, log.

Were you inspired by any existing framework/solution (not necessarily related to GAE)?

Jeff Schnitzer (objectify)

We think Google did a great job with the Python API, so we mimicked GAE/Python where we could.

John Patterson (Twig)

Db4o inspired the Activation concept as a way to limit object graph retrieval without needing proxies etc. Martin Fowlers writings on fluent interfaces influenced the design of the command objects.  http://www.martinfowler.com/bliki/FluentInterface.html

Ignacio Coloma (SimpleDS)

We have borrowed the generics approach from Google Collections, some architecture ideas from "Practical API Design" from Apress, and struggled to combine together a singleton-based and dependency injection approach. But mostly, we tried to keep the API as similar to the Datastore as possible, making it easier to use at some points.

Even though your solutions are different, many fundamental parts are solved in similar ways. Do you see any possiblity for a shared code base with the other frameworks? Or even a merge of the solutions? On the other hand, competition is healthy. What is your view on this?

Jeff Schnitzer (objectify)

We're happy to share code and cooperate with other developers. Ultimately, all of these projects were created to meet the everyday development needs of the authors, and as such represent "lost time". The more we can share, the better.

That said, the Objectify developers have a strong desire to keep the API small and learning curve short.  We are probably more cautious about introducing features than other projects are.  This philosophy would make it difficult to merge with more "kitchen sink"-oriented projects.

John Patterson (Twig)

Certainly collaboration on low level features such as data migration, profiling, logging this makes a lot of sense.  Scott (Objectify) suggested we try to keep annotations named the same which I agree with.

There is already some borrowing of high level features.  Objectify integrated Embedded collections from Twig and I was influenced in Transaction handling by Objectify.

A merge would not be practical with any of the other frameworks because all operate at a lower-level i.e. put, get, query, delete.

Ignacio Coloma (SimpleDS)

Twig and Objectify are putting more features on the table than SimpleDS, and that's on purpose. I see obvious value in trying to put together a solution for Indexes and Schema migration, but I don't know how far we can get until differences of criteria makes it complicated. Even the feature sets that we support using annotations are different.

My 2 cents

Complementary reading that I recommend:

• Objectify announcement with some Objectify/SimpleDS discussion: Here
• Objectify/Twig discussion: Here
• Debate thread: Here

Next: Feature checklist

This first section of interview questions focuses on the AppEngine datastore and its offering. The idea with this section is that developers that are considering AppEngine gets a chance to hear what some experienced developers has to say about the datastore.

What are the biggest strengths of the AppEngine datastore?

Jeff Schnitzer (objectify)

• That someone else manages it for you.  Server maintenance, backups, capacity, hardware failures, etc are all someone else's problem.
• That it's schemaless.  It's actually pretty easy to make structural changes to your data on-the-fly with no downtime. Despite some significant schema changes, I've only taken Mobcast (my "day job") offline once, and that had nothing to do with the datastore.
• That scaling the datastore is not an issue.  I feel pretty secure knowing that even if 10 million users all switch on my application tomorrow, they will all get about the same experience that users get today.  It means that I can spend every day writing new features instead of making sure the old ones continue to work.

John Patterson (Twig)

• No need to administer you own servers - Hard problems like clustering, security and hardware maintenance are taken off your hands.
• Scalability - Confidence that as your app grows in popularity you will have very few problems keeping pace. The limitations actually prohibit architectural practices that do not scale.
• Price - –Free to get started and investigate whether the platform suits your requirements. Very competitively priced as your resource demands increase.

Ignacio Coloma (SimpleDS)

You can deploy today, test your application and then turn your cell phone off for the weekend, knowing that if it goes down there will be an entire team working hard to get it back online again. It also has a really simple, straight-to-the-point set of features.

What about biggest weaknesses?

Jeff Schnitzer (objectify)

My biggest complaint, by far, is the lack of ad-hoc queries. In an RDBMS world you get used to firing up a SQL prompt and asking certain kinds of arbitrary questions about your data. It may take an hour to table scan a billion records, but you eventually get your answer.

The GAE datastore is great for operational systems, but it's not very convenient for analytics. You can iterate through your dataset 30s at a time using the task queue, but even simple questions require a fair amount of code.

Perhaps the looming map/reduce feature will address this issue, but it's hard to imagine it being as convenient as a SQL prompt. A SQL engine that you can run on top of an existing dataset and perform complex queries, no matter how slowly, would be extraordinarily useful. Unfortunately the folks building Cloud2db don't seem to be thinking along these lines - their solution is much more of an "all or nothing" approach.

My second biggest complaint is that the datastore is actually kinda slow. I read boasts that applications on Cassandra, Tokyo Tyrant, and MongoDB get more than 10k writes per second *per instance*. I'm lucky to get 100 writes per second in a GAE request.  Two orders of magnitude difference!

Other popular complaints are the lack of usable transactions and the lack of (automatic) joins. Not every application needs these things, so I'm more inclined to call them "constraints" rather than "weaknesses".

John Patterson (Twig)

The datastore requires a complete change in mindset and a new toolset.  Not having the ability to run complex queries means you need to pre-calculate results that you would be used to querying on the fly.

Frequent loading requests – traditional frameworks such as Wicket and Spring assume they can do work upfront to reduce per request effort.
Engine apps need to start fast and amortise initialisation work over many request or users are left waiting during the frequent loading requests.  This is probably the issue that hits more projects than any other. Often developers get quite far through the development process before being shocked to realise that their initial framework choice has made their application unusable.

Transactions are very limited and your data model must be structured to use them in a particular way from the start.

No https support for your domain and nothing a developer can do to remedy this themselves – apart from going elsewhere for this functionality.  The more general issue is that if you need something the platform doesn't offer you cannot simply install it yourself.

Many Java libraries are incompatible - there is no SOAP RPC implementation that runs on the Engine.

Ignacio Coloma (SimpleDS)

It has a really simple set of features, which quite often makes the "not allowed to do" list rather long. This is where third-party libraries should extend the capabilities of GAE.

What are the problems with the JDO/JPA solutions?

Jeff Schnitzer (objectify)

The biggest problem is that they're clumsy and unwieldy.  JDO contains a bazillion annotations, configuration options, and query language structures to allow it to straddle every conceivable kind of datastore, from an RDBMS to a object databases.  JPA is tailored specifically to the feature set of an RDBMS.  The GAE datastore is something new... so JDO gets a whole new arsenal of special extensions and JPA just gets left out in the cold.

In the context of GAE, 80% of JDO and JPA are meaningless cruft, and the GAE-specific features that you *do* need have tortured syntax.  Do you really want to type things like @Extension(vendorName = "datanucleus", key = "gae.unindexed", value="true") everywhere? The syntax for batch gets - one of the most important operations on the datastore - is atrocious:

Query q = pm.newQuery("select from " + Book.class.getName() + " where :keys.contains(key));

Using JDO/JPA on GAE effectively requires climbing not only the learning curve for JDO/JPA (and figuring out which 20% works), but also learning how the datastore works and how to bend JDO/JPA around it.  It's just not worth it - and I say this as someone who has been working extensively with Hibernate and JPA since the days of Hibernate 1.0.

Aside from the general complexity issue, there are some specific problems with JDO/JPA on GAE.  GAE has some interesting nuances (parent keys & entity groups, partial indexes, collection properties) that don't have direct mappings in either JDO or JPA.  JPA is particularly troublesome; DataNucleus uses the bytecode enhancer, but JPA doesn't have a detach() method - so you can't serialize your entities, ever!

John Patterson (Twig)

JDO is a one-size-fits-all interface that really doesn't fit the Engine so well at all.  Doing simple things that are specific to the datastore becomes incredibly complex and not at all intuitive.  You will spend a large portion of your development time hunting for the right annotations to use or figuring out if a feature is implemented.  Often you will be disappointed - no support for polymorphism?  Come on, this is Java!

More generally adhering to a spec limits how the implementation can utilize the underlying features of the datastore.  In some cases this needs to unnecessarily terrible performance.  Storing Lists of instances must update every member Entity if the order is changed so that behaviour adheres to the spec.

Startup time is terrible.

Ignacio Coloma (SimpleDS)

With JDO you are going through an extensive API thought for the features present in a relational database, or worse, any kind of datastore. GAE has a limited nature that basically chops off all those features behind the curtains. To me, it would be more natural to learn the things that are allowed in GAE instead of all the things that are forbidden (which again, is a long list).

The Datanucleus JPA implementation has JDO underneath it, so you get Yet Another Layer to understand and debug.

Are there any situations where you would recommend a developer to use JDO/JPA rather than a third party framework?

Jeff Schnitzer (objectify)

I'm tempted to say:  "If you expect to port your system to another platform, you should probably use JDO or JPA." However, the first retort that pops into my mind is "If you expect to port your system to another platform, you should stop using GAE right now." There are quite a lot of issues involved in migrating a GAE app to another environment and the datastore is just one of them. If you aren't sure that GAE will work for you, keep reading the documentation until you *are* sure one way or another. You will save yourself a lot of pain later on.

It's not that JDO/JPA doesn't have some nice features (change detection and cascading deletes, for example), it's just that I don't think these features are worth the trouble.  On the other hand, your apps may have entirely different demands than my apps, and maybe to you those features are golden.

If you're new to Appengine, I would recommend developing a spike solution with Objectify, even for JDO/JPA experts that intend to develop with JDO/JPA fulltime.  Objectify will teach you how the datastore really works, knowledge that will arm you well for your future struggles.

John Patterson (Twig)

If you have more time and money than you know what to do with I suggest wasting both by using JDO/JPA.

If you have an existing app that already uses JDO or JPA you may get some reuse of your investment by sticking with them.  But in reality your data model will need to change significantly, many if not all of your queries will need to be restructured, transactions will not work in the same way ... basically you will need to rewrite in any case.

If you want to keep your options open to move your app to another platform I still would not recommend it.  You will not be able to reuse your data models because of all the Engine specific hacks and dependencies you are forced to depend on.

Twig data models are more portable than JDO models because they do are pure Pojos with no dependencies on the low-level datastore or JDO.  Think of them as the DTOs that developers often need to write to hide implementation details.  Its just that you persist these pure DTOs directly.

Some apps are just not possible to build with JDO-GAE due to its lack of performance optimisations.  My experiments with JDO showed it was not possible to get the performance I needed for my app to be practical.  The same logical query that could not complete within the 30 second deadline using JDO now runs in 250ms thanks to Twigs embedded collections and parallel queries.  Other users have found that queries they had to run as background tasks previously now run in real time.

Ignacio Coloma (SimpleDS)

People aiming to make applications independent from the datastore will find some value in JDO/JPA. This is a case that I have problems to imagine, since your application will end tied to AppEngine one way or the other, but the biggest amount of independence is going through JPA/JDO.

Which problems do you see with using the low-level API directly?

Jeff Schnitzer (objectify)

• Untyped data structures.
• Clunky query API.
• Your data gets unceremoniously munged in odd ways (ie, all Integers go to Long, all collections to ArrayList).
• Poor documentation.
• It's simply not fun.

John Patterson (Twig)

You basically have to deal with bags of untyped values.  There is a reason we program in languages like Java with class constructs and not just Maps of Objects.

Transactions are handled inconsistently.  Due to historical reasons, all queries ignore the current transaction even if they include an ancestor filter.  This could not be changed without breaking existing low-level code.  Without that restriction Twig is free to always use the current transaction in every operation including queries.

If you have your own dynamic data models then they can map very well to low-level Entities.  But Twig can also be configured to store any type as Entities and Properties exactly as you want using its flexible PropertyTranslator chains.

Ignacio Coloma (SimpleDS)

There are no checks at all. You may query for a field that is not there anymore, or use a Key referencing the wrong kind. Of course, the biggest problem is using HashMaps to handle Entity data instead of Java classes.

There are a number of other frameworks currently available (objectify, Twig, SimpleDS, Slim3, Siena, cloud2db). If these existed when you decided to create your own framework, did you know of them? If so, why did these not live up to your requirements?

Jeff Schnitzer (objectify)

Yes, I reviewed all of these in fairly fine detail.  I would much rather build applications than tools.  I actually started working on patches to Siena in hopes of making it meet my needs, but Siena's cross-platform nature ran at odds with my need to squeeze every last ounce of performance out of the GAE datastore.

The biggest problem I had with the others is key management.  My entities have simple ids; many have natural integer keys (facebook ids, place ids).  I want to use simple POJO classes like this:

class Foo {
@Id long id;
String someData;
// ...etc
}

Placing the datastore Key class (or even a stringified version of the Key) in the entity carries redundant information and complicates working with the data.  JPA entities have simple ids, why can't mine? SimpleDS and Slim3 force the Key into the entity. Siena gets it right but can't handle entity groups.

Another issue was lack of deep support for generics.  It was tricky to weave a generic Key<?> class throughout Objectify's API, but the payoff is vastly more readable application code.

And then there are aesthetics.  Other than Siena, I just couldn't bring myself to love the other APIs.  Yes, it's subjective... but here are the money shots, you can judge for yourself:

http://objectify-appengine.googlecode.com/svn/trunk/javadoc/com/googlecode/objectify/Objectify.html
http://slim3.googlecode.com/svn/trunk/slim3/javadoc/org/slim3/datastore/Datastore.html
http://loom.sourceforge.net/docs/simpleds/javadoc/org/simpleds/EntityManager.html
http://code.google.com/p/twig-persist/source/browse/src/main/java/com/vercer/engine/persist/ObjectDatastore.java (no javadocs)

John Patterson (Twig)

Although Twig is the most recently released of the three interfaces discussed here, pre-release versions were made public that focused on getting the core functionality working well and fast.  Only in the recent final release were the user friendly fluent commands introduced with the benefit of seeing the strengths and weaknesses of the other API's as a base.

datastore.find().type(Hotel.class)
.withAncestor(chain)
.fetchResultsBy(50)
.continueFrom(myCursor)
.start From(100)
.addFilter(...)
.returnResultsNow(); //   returnKeysLater() performs a parallel query.

Ignacio Coloma (SimpleDS)

I searched for alternatives before starting up SimpleDS and couldn't find any. Anyway, I was looking for the simplest possible solution for GAE storage, not for a full JDO replacement.

My 2 cents

I had only worked with relational databases before I decided to try out AppEngine. I agree with all of the authors opinions here, but especially with John Patterson's comment about mindset. Not only did I go from standard relational SQL databases to BigTable, but also from standard PHP/HTML approach to Google Web Toolkit. Sometimes it feels like I would need behavioral therapy to;

1. Stop thinking about my model relationships in relational/SQL terms.
2. Stop thinking in terms of pages and page navigation when using GWT. There are no pages (well, one perhaps)! Embrace the dynamic world with GWT history!

I still love GWT + GAE though (well, besides the cold start time that is driving me a bit crazy). It's a great way for startups or hobby projects to get started without investing money in webservers, etc.

Next: The frameworks

Background

Anyone who has tried to use JDO/JPA on AppEngine will probably agree that it is too complex and has a steep learning curve. Offering a JDO/JPA solution is kind of like saying "Hey. We have a standardized API so you don't have to worry about the implementation details of the underlying datastore.". The problem is that in the case of AppEngine, there is also a fine print saying "But you should also be aware of these limitations, these GAE extensions and...etc...etc...".

This means that you MUST worry about the implementation details anyway, since the datastore <-> JDO/JPA mapping is far from optimal. If you don't know about the datastore limitations and think you can use JDO/JPA like you would on a relational datastore, you are in for a bumpy ride. And just to make this clear, once again, the root cause of this problem is not related to JDO/JPA or the DataNucleus implementation in general.

The low-level API is on the other end of the scale. It has an extremply simple API, but it's not type safe and only works with native entities, which requires lots of tedious work from the developer. What should a developer do?

The low-level wrapper frameworks to the rescue

I'm not the only one who thinks that the JDO/JPA solution is too complex and makes development harder than it has to be. There are, to my knowledge, 6 different attempts at offering an alternative, which is not a coincidence. These 6 are Objectify, Twig, SimpleDS, Siena, Slim3 and cloud2b. Update: A 7th framework called jiql which is a SQL/JDBC interface on top of AppEngine.

I have chosen to look at three of these; Objectify, Twig and SimpleDS. There is a good reason to why I chose those three. They are designed specifically for the AppEngine datastore with its features and limitations in mind. The other three frameworks are designed with other goals in mind.

Siena and cloud2b has multi-platform support and, like JDO/JPA, has a generic API to support this. Slim3 is a full-stack MVC framework that goes outside the scope of a thin low-level datastore wrapper. Update: Apparently Slim3 can be used as a standalone datastore framework, which wasn't clear from the documentation at the time. The documentation has now been updated to describe this.

I'm sure that these frameworks are great and probably good choices for a multi-platform strategy. Or, in the case of Slim3, a complete server-side framework. But I'm not at all interested in that. I want to use a simple and efficient datastore framework that mainly acts as a convenience layer on top of the AppEngine low-level API. Objectify, Twig and SimpleDS fulfills this requirement. Read this thread for some debating on this matter and why I don't want to use SQL/JDBC interfaces for my project.

The interview

Ok. Let's get on with the juicy stuff. First, let me introduce the authors. (Lots of credit and thanks to the authors for this!)

Jeff Schnitzer, lead developer Objectify.

John Patterson, lead developer Twig.

Ignacio Coloma, lead developer SimpleDS.

The original plan was to wrap this article up with a nice debate, but one of the GAE/J threads turned into a fantastic debate that covers many different aspects of datastore philosophies and designs. The authors would need to repeat themselves here if we were to keep the debate, so I decided to skip it and refer to the debate thread instead. In one of the interview sections there are links to various stuff you should have a look at.

The interview has been split into 4 subpages to make it more lucid.

Table of content

AppEngine datastore

The frameworks

Feature checklist

Summary

Next: AppEngine datastore

If you read my Page handling post, you know that I've been thinking about how to split TeamScape into several conceptual modules (main page, team page, association page).

I don't see how this can be solved with the current state of gwt-presenter which basically is a 1-1-1 mapping of place request -> place -> presenter.

I currently see three different ways forward:

1. Branch off gwt-presenter and adapt it
2. Try out the PuzzleBazar version of gwt-presenter
3. Try out mvp4g, which has integrated support for multiple modules (with code splitting!)

Has anyone tried mvp4g? Or does anyone have other ideas?

I have to make a decision related to how we should handle pages, so I was hoping you guys could help me out on this one.

There are two things to discuss here:

• How we should handle URL mapping of pages
• How pages should be represented in GWT with the MVP pattern

Let's start by describing what we need to do. TeamScape will have three different pages;

1. A main page which a visitor comes to by going to http://www.teamscape.se. This page presents the site and you can register a team, search for teams, etc.
2. A team page, which represents registered teams and its players, leagues, matches etc.
3. An association page, which represents registered assocations/clubs and their sport offerings.

All these pages will have different layouts, This means that there will be unique presenters/views for each page, but we also have some that needs to be shared between the pages.

Initially I had some ideas that this could be solved with servlet url-mappings and multiple GWT modules, but this turned out to be very bad ideas. First, I missed one very important detail related to servlets, and that is that the html file (read the js script) will be reloaded for each request. Second, splitting the project into several modules, entry points and host files only make the structure unnecassarily complex.

After some more research, it has become clear that GWT history + code splitting is the way to go.  I will probably postpone the implementation of this for a while, until I know more about how gwt-presenter will evolve with places and presenters. We have lots of work to do with the team page and entities before we have to bother with this anyway.

• MVP pattern with gwt-presenter
• EventBus
• GWT History mechanism
• GWT Localisation/I18N mechanism
• Dependency injection with GIN and Guice
• RPC with gwt-dispatch
• Standards mode compliance with GWT 2.0
• Layout with UiBinder and MVP
• Advanced layout with CSS
• Using client bundles to handle css and image resources
• Browser auto-complete login form
• GWT/GAE testing methodologies
• GAE Datastore (BigTable) offering and limitations
• JDO
• Objectify (framework that wraps the datastore low-level API)
• Security. Especially for client/server communication.
• Performance and optimizations client and server side
• GAE User service for Google accounts
• GAE Memcache service
• GAE Mail service
• GAE Image service
• And..more...