Tales From The Cloud

So you have a great website idea and you want to build and bring that first version online as fast as you can. You figured that node.js is the way to go. You kick-off the development and after a couple of hours of hacking you realize that although you’re progressing at breakneck speed you’re missing a few important bits:

• - How do I better structure my project?
• - I want to test this thing. I want unit tests, UI (headless browser) tests and public API tests (I want that API offering out too of course)
• - I want proper CSS and html templating
• - Looks like I need non-trivial request routing, I need more than the default provided

Oh, and after you have all of this, you want to be able to deploy it to a node-ready cloud environment like Heroku without hassle.

Enter bootstrap.js.

Location Based Services are all the rage these days. The space is still being defined and the players are trying to differentiate their service offerings in order to attract the critical mass of developers. In this post I’ll draw a side-by-side comparison of the main features provided by the major Places API providers today. While I have no hard numbers to back-up the “major provider” claim, I’ll simply go for the web companies I would look for when building an application around Location services.

Here are my candidates ordered by their first API release date:

I won’t try to identify any “best” API in the end. This post is meant to allow the reader to make an informed decision on which API(s) to use.

Handling errors in a REST way is seemingly simple enough: upon requesting a resource, when an error occurs, a proper status code and a body that contains a parseable message and using the content-type of the request should be returned. The default error pages in Tomcat are ugly. Not only they expose too much of the server internals, they are only HTML formatted and making them a poor choice if a RESTful web service is deployed in that Tomcat container. Substituting them to simple static pages is still no enough since I want a dynamic response containing error information.

Here’s how to do it in 3 simple steps:

Fetching, aggregating and transforming data for delivery is a seemingly complex task. Imagine a service that serves aggregated search results from Twitter, Google and Bing where the response has to be tailored for mobile and web. One has to fetch data from different sources, parse and compose the results then transform them into the right markup for delivery to a specific client platform. To cook this I’ll need:

• a web server
• a nice way to aggregate web service responses (pipelining would be nice)
• a component to transform the raw aggregated representation into a tailored client response.

I could take a stab at it and use Apache/Tomcat, Java (using Apache HttpClient 4.0), a servlet dispatcher (Spring WebMVC) and Velocity templating but it sounds too complex.
Enter Node.js. It’s an event-based web server built on Google’s V8 engine. It’s fast and it’s scalable and you develop on it using the familiar Javascript. While Nodejs is still new, the community has built a rich ecosystem of extensions (modules) that greatly ease the pain of using it. If you’re unfamiliar with the technology, check-out the Hello World example, it should get you started. Back to the task at hand, here are the modules I’ll need:

• Restler to get me data.
• async to allow parallelizing requests for effective data fetching.
• Haml-js for view generation

Update Mar.04 Thanks to @ewolff some of the points described below are now official feature requests. One (SPR-6928) is actually scheduled in Spring 3.1 (cool!). I’ve updated the post and added all open tickets. Please vote!

This post is somewhat a response to InfoQ’s Comparison of Spring MVC and JAX-RS. Recently I have completed a migration from a JAX-RS implementation of a web service to Spring 3.0 MVC annotation-based @Controllers. The aforementioned post on InfoQ was published a few days after my migration so I’m dumping below the list of problems I had, along with solutions.

Full list of issues:

• Same relative paths in multiple controllers not supported
• @ExceptionHandler is controller-centric
• Standard content negotiation can't respond with a fixed response type
• JSR 303 bean validation not applied in @Controllers
• Formatting responses (i.e. Date) not working when using Spring formatter annotations

Same relative paths in multiple @Controllers not supported Consider two Controllers where I use a versioned URL and a web.xml file that uses two URL mappings:

@Controller
public class AdminController {
   @RequestMapping("/v1/{userId}")
   public SomeResponse showUserDetails(String userId) {
      ...
   }
}

@Controller
public class UserController {
   @RequestMapping("/v1/{userId}")
   public SomeOtherResponse showUserStreamtring userId) {
      ...
   }
}
In web.xml:
	<servlet-mapping>
	  <servlet-name>public-api</servlet-name>
	  <url-pattern>/public</url-pattern>
	</servlet-mapping>
	<servlet-mapping>
	  <servlet-name>admin-api</servlet-name>
	  <url-pattern>/admin</url-pattern>
	</servlet-mapping>

I want to write testable code and occasionally I bump into frameworks that make it challenging to unit test. Ideally I want to inject a service stub into my code then control the stub’s behavior based on my testing needs. Commons Http Client from Jakarta facilitates integration with HTTP services but how to easily unit test code that depends on the HttpClient library? Turns out it’s not that hard. I’ll cover both 1.3 and the newer 1.4 versions of the library since the older v1.3 is still widely used. Here’s some typical service (HttpClient v1.3) we want to test. It returns the remote HTML page title:

public class RemoteHttpService {
   private HttpClient client;
   
   public String getPageTitle(String uri)  throws IOException {
     String contentHtml = fetchContent(uri);
     Pattern p = Pattern.compile("<title>(.*)</title>");
     Matcher m = p.matcher(contentHtml);
     if(m.find()) {
        return m.group(1);
     }
     return null;
   }

   private String fetchContent(String uri)  throws IOException {
      HttpMethod method = new GetMethod("http://blog.newsplore.com/" + uri);
      int responseStatus = client.executeMethod(method);
      if(responseStatus != 200) {
        throw new IllegalStateException("Expected HTTP response status 200 " +
"but instead got [" + responseStatus + "]");
      }
      byte[] responseBody = method.getResponseBody();
      return new String(responseBody, "UTF-8");
   }

   public void setHttpClient(HttpClient client) {
      this.client = client;
   }
}

with the HttpClient is injected at runtime (via some IoC container or explicitly). To be able to unit-test this code we have to come-up with a stubbed version of the HttpClient and emulate the GET method.

There are three forces that shape a project: domain, process and technology. Add the “-driven” suffix to any of them and you’ll perhaps recognize some of the methods used in projects you’ve been involved in and yet, as soon as one takes too much of a lead against the other two, failure will follow almost inevitably. At the intersection of these three forces we find familiar terms and concepts but first a word or two about each of them:

Technology Back in the days of the tech bubble, tech was allmighty. Buzzwords like Java, EJBs, PHP defined entire projects. It was the time where software became accessible to a much larger audience. The new wave of enthusiastic geeks embraced everything from new languages to professional certifications to the then-nascent open source. I admit having my share of technology-driven projects back in the day…

Process Process brings structure and pace to a project. The two complementary components of a project process are methodology and integration. We are all too familiar with methodology: waterfall, RUP or agile methods of software development are vastly documented and practiced; integration largely is defect management, testing, build, deployment and documentation. Today they all come together in what is called continuous integration where all these concepts become interrelated in a repetitive process that produces accountability and visibility into the progress of a project. Process is also the one force that tends to disappear first after a project is finished.

Domain The domain captures entities and business logic, is driven by the business requirements and is in no way influenced by the two other forces. The most successful way to employ the domain is through Domain Driven Design where the main focus of software development is neither technology nor process but the business requirements.

In the first part I’ve left-off with some good news: successful deployment in the local GAE container. In this second part I’ll talk about the following:

• - Loading data and browsing
• - Table indexing
• - Limitations of datastore queries
• - More datastore limitations and JPA issues
• - Deployment
• - Performance
• - Production monitoring
• - Usage and quotas
• - Other limitations
• - Final thoughts

Back to bulk loading, after voting on the feature request to have a bulk uploader, I decided not to use the python solution but to handcraft a loader that will fill-in two tables: COUNTRY and COUNTRY_STATE (for federated countries like US and CAN). Since there’s no way to schedule worker threads (but is on it’s way) the only mechanism to trigger the data loading is via URLs. I’m using Spring 3.0 and so it wasn’t hard to create a new @Controller, some methods to handle data loading then map them to URLs:

@Controller
public class BulkLoadController {
	@RequestMapping("/importCountries")
	public ModelAndView importCountries() {
           ...
        }
	@RequestMapping("/importProvinces")
	public ModelAndView importProvinces() {
           ...
        }
}