Tales From The Cloud

Everything Beta

Posts

October 25, 2023

177Pod Episode 2: Tech News

177Pod Episode 2 is just out. This time me & Simon ramble about new Java 21 features, programming in Golang and fixing performance issues.

September 29, 2023

New project: podcasting!

I'm delighted to announce that this summer I've kicked-off together with my good friend Simon Redfern a new podcast called the 177 Pod. Sustainability, AI, tech news are on the menu. Listen to what we're up to in the teaser episode. Teaser Podcast social links: Twitter: @177pod Instagram: @177pod Youtube: @177pod Episode 1: Cycling and Recycling is just out with two exciting guests. Enjoy!

July 31, 2017

Zero to image recognition in 60 seconds with TensorFlow and Spring Boot

Machine learning is not new and TensorFlow isn't new either. By open-sourcing the tech, Google has brought the ML domain within the grasp of software engineers. Google is refining TensorFlow's toolchain continuously but the barrier to entry for developers is still high. Not only the domain of machine learning is tough to wrap a beginner's head around, TensorFlow itself presents a relatively high upfront complexity for engineers trying out the tech, not to mention building the tools will try hard one's patience.

December 22, 2016

On Engineers and Technicians

The Software Engineering field is a liberal one with a relatively low barrier to entry. Formal education is valued less than hands-on experience but that tends to be difficult to accurately measure. The industry came up with various solutions to gauge a candidate’s skills, ranging from coding tests, whiteboard interview to showing their public work to asking the candidate for a work portfolio. Programming is also an art, isn’t it?

September 11, 2015

Five years of weather in a timelapse

This has been a long time coming, five years actually. Spincloud went live in January 2009 and I’ve added the temperature heatmap overlay a few months later. The heatmap and the corresponding map overlay is generated once an hour and it’s been faithfully doing so for six years. It does it by first generating an global temperature heatmap then cutting tiles (up to zoom level 6). Every hour. But what I have also done back then was to also save one heat map image once a day.

September 3, 2015

A TinyGPS upgrade adding NMEA v3.0 and GLONASS support

TL;DR: Checkout my forked https://github.com/florind/TinyGPS, adding GLONASS and NMEA 3.x support and that it’s fully compatible with the original TinyGPS and Arduino. It’s been too long of a dry streak on my blog (four years!) but in the mean time I’ve been actively working on several projects. About 3,5 years ago I have reconnected with hardware engineering, a hobby of mine ever since I was a kid. I am integrating GPS in one of my Arduino-compatible hardware projects and I’m using a Maestro Wireless part, called A5100 that notably comes with GLONASS support (the Russian constellation of GPS satellites).

September 9, 2011

Bootstrap your node.js project in the cloud

So you have a great website idea and you want to build and bring that first version online as fast as you can. You figured that node.js is the way to go. You kick-off the development and after a couple of hours of hacking you realize that although you’re progressing at breakneck speed you’re missing a few important bits:

  • - How do I better structure my project?
  • - I want to test this thing. I want unit tests, UI (headless browser) tests and public API tests (I want that API offering out too of course)
  • - I want proper CSS and html templating
  • - Looks like I need non-trivial request routing, I need more than the default provided

Oh, and after you have all of this, you want to be able to deploy it to a node-ready cloud environment like Heroku without hassle.

Enter bootstrap.js.

March 23, 2011

A Comparison of Places APIs

Location Based Services are all the rage these days. The space is still being defined and the players are trying to differentiate their service offerings in order to attract the critical mass of developers. In this post I’ll draw a side-by-side comparison of the main features provided by the major Places API providers today. While I have no hard numbers to back-up the “major provider” claim, I’ll simply go for the web companies I would look for when building an application around Location services.

Here are my candidates ordered by their first API release date:

ProviderNameAPI LinkFirst Released
YahooYahoo GeoPlanet APIYahoo! GeoPlanet™May 2009
FoursquareFoursquare APIFoursquare APIv2Nov. 2009
TwitterTwitter Places APIGeo methods in Twitter APIJun. 2010
FacebookFacebook Places APIScattered under the Graph APIAug. 2010
GoogleGoogle Places APIGoogle Places APINov. 2010
The features of all these APIs are designed primarily to support (and promote) the business use cases of each respective competitor. One notable exception is Yahoo's GeoPlanet API which advertises itself as being a general purpose API for referencing places.

I won’t try to identify any “best” API in the end. This post is meant to allow the reader to make an informed decision on which API(s) to use.

August 20, 2010

Towards An Open Database of Places: Location Autodiscovery

A short while back I read a challenging article titled t’s Time For An Open Database Of Places. There, Erich Schonfeld notes: A long list of companies including Twitter, Google, Foursquare, Gowalla, SimpleGeo, Loopt, and Citysearch are far along in creating separate databases of places mapped to their geo-coordinates. These efforts at creating an underlying database of places are duplicative, and any competitive advantage any single company gets from being more comprehensive than the rest will be short-lived at best.

August 4, 2010

RESTful error handling with Tomcat and SpringMVC 3.x

Handling errors in a REST way is seemingly simple enough: upon requesting a resource, when an error occurs, a proper status code and a body that contains a parseable message and using the content-type of the request should be returned. The default error pages in Tomcat are ugly. Not only they expose too much of the server internals, they are only HTML formatted and making them a poor choice if a RESTful web service is deployed in that Tomcat container. Substituting them to simple static pages is still no enough since I want a dynamic response containing error information.

Here’s how to do it in 3 simple steps:

June 27, 2010

Building a content aggregation service with node.js

Fetching, aggregating and transforming data for delivery is a seemingly complex task. Imagine a service that serves aggregated search results from Twitter, Google and Bing where the response has to be tailored for mobile and web. One has to fetch data from different sources, parse and compose the results then transform them into the right markup for delivery to a specific client platform. To cook this I’ll need:

  • a web server
  • a nice way to aggregate web service responses (pipelining would be nice)
  • a component to transform the raw aggregated representation into a tailored client response.

I could take a stab at it and use Apache/Tomcat, Java (using Apache HttpClient 4.0), a servlet dispatcher (Spring WebMVC) and Velocity templating but it sounds too complex.
Enter Node.js. It’s an event-based web server built on Google’s V8 engine. It’s fast and it’s scalable and you develop on it using the familiar Javascript. While Nodejs is still new, the community has built a rich ecosystem of extensions (modules) that greatly ease the pain of using it. If you’re unfamiliar with the technology, check-out the Hello World example, it should get you started. Back to the task at hand, here are the modules I’ll need:

  • Restler to get me data.
  • async to allow parallelizing requests for effective data fetching.
  • Haml-js for view generation

February 23, 2010

Using Spring 3.0 MVC for RESTful web services (rebuttal)

Update Mar.04 Thanks to @ewolff some of the points described below are now official feature requests. One (SPR-6928) is actually scheduled in Spring 3.1 (cool!). I’ve updated the post and added all open tickets. Please vote!

This post is somewhat a response to InfoQ’s Comparison of Spring MVC and JAX-RS. Recently I have completed a migration from a JAX-RS implementation of a web service to Spring 3.0 MVC annotation-based @Controllers. The aforementioned post on InfoQ was published a few days after my migration so I’m dumping below the list of problems I had, along with solutions.

Full list of issues:

Same relative paths in multiple @Controllers not supported Consider two Controllers where I use a versioned URL and a web.xml file that uses two URL mappings:

@Controller
public class AdminController {
   @RequestMapping("/v1/{userId}")
   public SomeResponse showUserDetails(String userId) {
      ...
   }
}

@Controller
public class UserController {
   @RequestMapping("/v1/{userId}")
   public SomeOtherResponse showUserStreamtring userId) {
      ...
   }
}
In web.xml:
	<servlet-mapping>
	  <servlet-name>public-api</servlet-name>
	  <url-pattern>/public</url-pattern>
	</servlet-mapping>
	<servlet-mapping>
	  <servlet-name>admin-api</servlet-name>
	  <url-pattern>/admin</url-pattern>
	</servlet-mapping>

February 9, 2010

Unit testing with Commons HttpClient library

I want to write testable code and occasionally I bump into frameworks that make it challenging to unit test. Ideally I want to inject a service stub into my code then control the stub’s behavior based on my testing needs. Commons Http Client from Jakarta facilitates integration with HTTP services but how to easily unit test code that depends on the HttpClient library? Turns out it’s not that hard. I’ll cover both 1.3 and the newer 1.4 versions of the library since the older v1.3 is still widely used. Here’s some typical service (HttpClient v1.3) we want to test. It returns the remote HTML page title:

public class RemoteHttpService {
   private HttpClient client;
   
   public String getPageTitle(String uri)  throws IOException {
     String contentHtml = fetchContent(uri);
     Pattern p = Pattern.compile("<title>(.*)</title>");
     Matcher m = p.matcher(contentHtml);
     if(m.find()) {
        return m.group(1);
     }
     return null;
   }

   private String fetchContent(String uri)  throws IOException {
      HttpMethod method = new GetMethod("http://blog.newsplore.com/" + uri);
      int responseStatus = client.executeMethod(method);
      if(responseStatus != 200) {
        throw new IllegalStateException("Expected HTTP response status 200 " +
"but instead got [" + responseStatus + "]");
      }
      byte[] responseBody = method.getResponseBody();
      return new String(responseBody, "UTF-8");
   }

   public void setHttpClient(HttpClient client) {
      this.client = client;
   }
}

with the HttpClient is injected at runtime (via some IoC container or explicitly). To be able to unit-test this code we have to come-up with a stubbed version of the HttpClient and emulate the GET method.

December 13, 2009

Spincloud, now with worldwide forecast

In my constant search for free weather data for Spincloud, a short while ago I have found a gem: free forecast data offered by the progressive Norwegian Meteorologic Institute. The long range forecast coverage is fairly thorough and covers most more than 2700 locations worldwide. I am happy to announce that I have extended spincloud.com to include it.     The data is refreshed every hour and the forecast range is available for the next seven days.

October 15, 2009

Continuous everything?

I admit that I regard automation as a dull but vital part in the success of a project. Automation had evolved into Continuous Integration, a powerful toolset allowing frequent and regular building and testing of the code. I won’t get into what CI is (check the internets). Instead, I am going to explore a couple of aspects of CI that can be added to the artifacts of the development process and note some others that cannot.

October 3, 2009

blog.newsplore.com is one year old!

Sep. 15 came and went and I didn’t realize that this blog is one year old. I didn’t post in the last few months but this is because I have been busy moving across the world from Toronto back to Europe (I’m in Berlin now), becoming a father (“the best job in the world”) and taking a new job (LBS, yes!). Things are happening, wheels are in motion and there’s still a lot to write about.

July 7, 2009

New Spincloud feature: heat map overlay

It took a while since the previous feature update to Spincloud. I have done a number of upgrades to the underlying tech and some intensive code refactoring but nothing visible. The time has come for another eye candy: heat maps. It is a map overlay that shows a color-translated temperature layer based on interpolated values of the current weather conditions. It gives a quick indication of the average temperature across all land masses where data is available.

July 5, 2009

A project triumvirate

There are three forces that shape a project: domain, process and technology. Add the “-driven” suffix to any of them and you’ll perhaps recognize some of the methods used in projects you’ve been involved in and yet, as soon as one takes too much of a lead against the other two, failure will follow almost inevitably. At the intersection of these three forces we find familiar terms and concepts but first a word or two about each of them:

Technology Back in the days of the tech bubble, tech was allmighty. Buzzwords like Java, EJBs, PHP defined entire projects. It was the time where software became accessible to a much larger audience. The new wave of enthusiastic geeks embraced everything from new languages to professional certifications to the then-nascent open source. I admit having my share of technology-driven projects back in the day…

Process Process brings structure and pace to a project. The two complementary components of a project process are methodology and integration. We are all too familiar with methodology: waterfall, RUP or agile methods of software development are vastly documented and practiced; integration largely is defect management, testing, build, deployment and documentation. Today they all come together in what is called continuous integration where all these concepts become interrelated in a repetitive process that produces accountability and visibility into the progress of a project. Process is also the one force that tends to disappear first after a project is finished.

Domain The domain captures entities and business logic, is driven by the business requirements and is in no way influenced by the two other forces. The most successful way to employ the domain is through Domain Driven Design where the main focus of software development is neither technology nor process but the business requirements.

June 13, 2009

Process Journalism really is Agile Journalism

There’s an interesting story rippling the news stream these days: The New York Times is questioning the ways Techcrunch reports on news. The crux of the issue is product journalism v. process journalism, the act of producing news only after all facts have been verified versus just writing the story based on what the author best knows about an event at that moment then let the story evolve and provide rigorous updates and corrections once it does.

June 6, 2009

Reviewing Google AppEngine for Java (Part 2)

In the first part I’ve left-off with some good news: successful deployment in the local GAE container. In this second part I’ll talk about the following:

Loading data and browsing After finishing-off the first successful deployment, next on the agenda was testing out the persistence tier but for this I needed some data to work with in the GAE datastore. Along with the local container Google makes available a local datastore but so far the only way to populate the datastore is described here and it uses python. Furthermore, currently there is one documented way to browse the local datastore using the local development console but this again requires the python environment. There's a hint from Google though that there will be a data viewer for the Java SDK. In the mean time I voted up the feature request.

Back to bulk loading, after voting on the feature request to have a bulk uploader, I decided not to use the python solution but to handcraft a loader that will fill-in two tables: COUNTRY and COUNTRY_STATE (for federated countries like US and CAN). Since there’s no way to schedule worker threads (but is on it’s way) the only mechanism to trigger the data loading is via URLs. I’m using Spring 3.0 and so it wasn’t hard to create a new @Controller, some methods to handle data loading then map them to URLs:

@Controller
public class BulkLoadController {
	@RequestMapping("/importCountries")
	public ModelAndView importCountries() {
           ...
        }
	@RequestMapping("/importProvinces")
	public ModelAndView importProvinces() {
           ...
        }
}

June 4, 2009

Upgrading to Spring 3.0.0.M3 and Spring Security 3.0.0.M1

A short two months back I posted an article describing how to upgrade to Spring 3.0 M2. Spring folks are releasing at breakneck speed and so I got busy again upgrading spincloud.com to Spring 3.0 M3 released at the beginning of May. Just yesterday (June 3rd) the team released Spring Security 3.0 M1 and I decided to roll this in Spincloud as well.

Upgrading Spring Security from 2.0.4 to 3.0.0 M1 For Spring Security 3.0.0 M1 I’m doing a “soft” upgrade since I had done my homework when I migrated from Acegi. I won’t use any of the new 3.0 features, just getting ready to use them.

To digress a bit, Spring Security a technology that is harder to swallow due to its breadth. To simplify the picture, Spring Security (former Acegi) provides three major security concerns to enterprise applications:

  • Authorization of method execution (either through standard JSR-250 JEE security annotations or via specific annotation-based method security).
  • HTTP Request authorization (mapping URLs to accessible roles using Ant style or regexp’ed paths, dubbed channel security).
  • Website authentication (integrating Single Sign On (SSO) services by supporting major SSO providers, HTTP BASIC authentication, OpenId authentication and other types). For Spincloud I’m using OpenId for authentication and Channel Security to secure website pages.

May 28, 2009

Proposal to standardize the Level 2 query cache configuration in JPA 2.0

Level 2 cache is one of the most powerful features of the JPA spec. It’s a transparent layer that manages out-of-session data access and cranks-up the performance of the data access tier. To my knowledge it has been first seen in Hibernate and was later adopted by the then-emerging JPA spec (driven mostly by the Hibernate guys back in the day). As annotations gained strength and adoption, L2 caching that was initially configured through XML or properties files, was brought closer to the source code, alas in different forms and shapes. This becomes apparent if you ever have to deploy your webapp across a multitude of containers as you have to painfully recode the cache configuration (or worse, hand-coded cache access). Why not standardizing the cache control in JPA? This seems to be simple enough to achieve and yet it isn’t there. Now that JPA 2.0 is standardizing on Level 2 cache access (See JSR 317 section 6.10) it is the natural thing to do. Every JPA provider has its own way of specifying cache access (both Entity and query cache). To grasp the extent of the various ways cache is configured, here are some examples:

Hibernate: Cache control for entities

@Entity
@org.hibernate.annotations.Cache(usage = CacheConcurrencyStrategy.NONSTRICT_READ_WRITE)
public class Employee {...

Cache control for named queries:

@javax.persistence.NamedQuery(name="findEmployeesInDept",
query="select emp from Employee emp where emp.department = ?1",
hints={@QueryHint(name="org.hibernate.cacheable",value="true")})

or

@org.hibernate.annotations.NamedQuery(cacheable=true, cacheRegion="employeeRegion")

OpenJPA Cache control for entities

@Entity
@org.apache.openjpa.persistence.DataCache(timeout=10000)
public class Employee {...

Query cache requires hand coded cache access:

OpenJPAEntityManagerFactory oemf = OpenJPAPersistence.cast(emf);
QueryResultCache qcache = oemf.getQueryResultCache();

May 27, 2009

Reviewing Google AppEngine for Java (Part 1)

appengine_lowres
When Google announced that Java is the second language that the Appengine will support I almost didn't believe it given the surge of the new languages and the perception that Java entered legacy but the JVM is a powerful tried-and-true environment and Google saw the potential of using it for what it is bound to
become: a runtime environment for the new and exciting languages (see JRuby and Grails). The JVM is the new gateway drug in the world of languages.

Note: I’ll break down this review into two posts as it’s too extensive to cover everything at once. This first part is about initial setup, JPA and some local webapp deployment issues. In the second part I’ll describe how to load data into the datastore, data indexing, how I reached some of the limitations of the AppEngine and how to get around some of them.

I managed to snatch an evaluation account and last few days I have been playing with it. (as of today Google opened the registrations to all). My goal was (again) simple: to port Spincloud to the AppEngine. The prize: free hosting which amounts to $20/month in my case (see my post about Java hosting providers), some cool monitoring tools that I can access on the web and of course the transparent super-scalability that Google is touting, in case I get techcrunched, slashdotted or whatever (slim chances but still…). I am a couple of weeks into the migration effort and I can conclude that the current state of the AppEngine is not quite ready for Spincloud as I’ll detail below but it comes quite close. The glaring omission was scheduling but Google announced today that the AppEngine will support it (awesome!) and I plan to use it from day one.

My stated goal is running Spincloud on AppEngine. Here are the technologies I use in Spincloud:

  • Spring 3.0 (details here)
  • SpringAOP
  • SpringSecurity with OpenId
  • Webapp management with JMX (I know, it won’t fly)
  • Job scheduling using Spring’s Timer Task abstraction (not optimistic about this one either)
  • SiteMesh (upgraded to 2.4.2 as indicated)
  • Spatial database support including spatial indexing
  • JPA 2.0 (back story here)
  • Level 2 JPA cache, query cache.
  • Native SQL query support including native SQL spatial queries.
  • Image processing (spincloud scraps images and processes pixel-level information to get some weather data)

May 23, 2009

"Thinking in" what?

“Thinking in…” anything has been a marketing quirk for a while now, being used and abused from the field of language learning to computer science. Thinking in Java is the title of a well known book written by Bruce Eckel. I am passing a “Think in Spanish” course flyer ad every time I stroll on Bloor Street West here in Toronto. Funny, thinking within the rigors of a discipline is the very definition of thinking inside the box.

May 17, 2009

Mobile internet is here to stay

Since I went back from vacation (Easter with family back home in Romania) I got quite interested in the mobile internet. This was the first time I wasn’t going to internet cafes or asking buddies to let me use their internet (and their PCs). I have a beaten Nokia 3100 that works both in RO and here in Toronto. I was playing with it in my first vacation days back home, exploring the features of my romanian mobile carrier (I love the *…# service commands, they remind me of dumb terminals) and I noticed that they had a new offering, mobile internet. Configuration was a breeze and I found myself browsing the internet a few minutes later. Email, news, even checking if my websites are still up and running; they were all there where I left them. Needless to say that from that day I didn’t step into an internet cafe anymore to get my fix.

I also got a lovely gift from Corina around the same time, an iPod Touch. Since I’m a fan of Apple’s products I couldn’t be happier and since I started using it I realized that I use my regular laptop less and less. I’m using the iPod Touch more to check Email, news, weather, live TV (gotta love France24) and play the occasional game, than to listen to music. The experience is more condensed and focused and so I spend less time to find what I’m looking for. Mobile internet experience achieved the goal that it’s parent couldn’t: ease of use.

Funny that I had to go back home to find it out.

April 4, 2009

Selecting location data from a spatial database

I have been thinking to write about this subject a while back when project Spincloud was still under development. I was even thinking about making this the first post on my blog. The idea is simple: you have location-based data (POIs for instance) stored in some database (preferably a spatial DB) and now you want to perform a select statement that will indicate the area that should include the points we want. In case of Spincloud’s weather map, we want the weather reported by the stations located within a given area determined by the Google Map viewport that the user is currently browsing. In all my examples I’ll use SQL Spatial Extensions support, specifically MSQL spatial extensions. Here’s a visual representation of the spatial select (the red grid is the area where we want to fetch the data):

select_smpl

This is quite easy to accomplish by issuing a spatial select statement on the database:

select * from POI where Contains(GeomFromText
    ('POLYGON ((-30 32, 30 -8, -89 -8, -89 32, -30 32))', LOCATION))

But what about selecting an area that crosses the 180 degrees longitude? Let’s say we want to select data in an area around New Zealand that starts at 170 degrees latitude and ends at -160 degrees latitude going East. The selected area will look like this:

March 21, 2009

Evaluating EclipseLink 1.1

As I'm using the ubiquitous Hibernate 3.3 as the JPA 1.0 provider for Spincloud, I decided to try out another one. I had tried OpenJPA (spawned from Kodo JDO) when they only supported build-time bytecode enhancement and it was a pain to make it work. It worked all right but boy what a pain. There's now an agent to provide on-the-fly enhancement but I'll take transparent enhancement anytime. I've heard about EclipseLink before. The project started when Oracle donated the respectable TopLink project to the Eclipse foundation. If the solid reputation behind TopLink was a good enough argument for me to try it, the announcement that it will be the JPA 2.0 reference implementation convinced me that I should try it out. My goal is to evaluate if EclipseLink is production-ready. I'm applying a complex set of evaluation criteria (joking): if it can run Spincloud then it is (I was inspired by Seifer's interview on Infoq about Ruby VMs; when asked what was the criteria for qualifying if a Ruby VM is production ready, he answered: if it runs Rails).

I have the following JPA requirements:

  • column mappings, one-to-one, one-to-many
  • supports BLOB fields
  • supports NamedQueries and NamedNativeQueries
  • support for object cache and query cache
  • deployment/operational nice to have: ease of maintaining compatibility with both EclipseLink and Hibernate in the source code and runtime. Ideally I should plug-in any JPA provider without changing a single line of code. This was not attainable as I’ll explain below.

I started by downloading the binaries. I’m using Maven to bring the jars so I’ve followed the instructions here. I only changed the version since I wanted to use v1.1.0:

  <dependency>
    <groupId>org.eclipse.persistence</groupId>
    <artifactId>eclipselink</artifactId>
    <version>1.1.0</version>
    <scope>provided</scope>
  </dependency>

There’s a single jar file called eclipselink-1.1.0.jar downloaded which is a nice change from the multitude of Hibernate jars I was accustomed with.

March 14, 2009

Choosing a Java hosting provider

TL;DR: Go for a VPS. I now use Digitalocean, they’re awesome.

Selecting a web hosting provider is a tough job for any web developer that wants to put a Java/JEE web application online. The choice is much simpler when it comes to publishing a PHP web site and there are a load of cheap (and sometimes quite reliable) PHP hosting providers to choose from with LAMP being the de facto standard in the web hosting world. But when it comes to Java hosting providers the picture becomes blurrier. The common thing that all these environments need is a Java container. The most popular choice is Tomcat but there are providers that use Resin, Weblogic or Websphere (the latter two are full fledged JEE containers). With the raise of lightweight J2EE servers started by the Spring folks, a little revolution began in the Java world: running enterprise-grade JEE webapps without the need for an EJB container; a servlet container is enough. Tomcat should fill the bill for just about any Java web application that doesn’t use EJBs. The advantage that comes from running a servlet container is the smaller footprint compared with an EJB container. This is critical when it comes to selecting hosting environments since you’re paying for resources (especially RAM) that have to be sized to accommodate the memory requirements of the web application.

Self hosting: Home-based dedicated server (near $0) homepcYou can use your home internet connection and a PC where the web container runs and deploys your web application. There is an obvious advantage since the server is physically located close to the development team: complete control and physical access to the machine. The other big plus is the price: you’re already paying for the internet connection and chances are you already have a PC that can be transformed into a server. And chances are that you already have a development environment that can be the basis of the production deployment. Here are the shortcomings:- Bandwidth limitations: the webserver where the webapp lives will upload content to the web browsers that access the application. The upload bandwidth is smaller than the download bandwidth and will result in longer load times for web clients.

  • Some internet providers cap the internet traffic so you may run beyond the limit.
  • You may violate contractual obligations you agreed with your Internet provider by running your own web server.
  • The internet connection is not reliable: provider’s downtime directly translates in downtime for your webapp.

March 7, 2009

Upgrading to Spring 3.0


In the spirit of beta I’m upgrading spincloud.com to Spring 3.0. I’m using version 2.5.6 currently but it’s missing REST support and I had to use Carbonfive’s REST library which worked like a charm. Now it’s time to get back under Spring’s fold and use their built-in REST support. Spring 3 opens the door to a lot of new features so I’m eager to try it.

I’m using Maven2 to get the jars and Ant to build the project. To fetch Spring 3.0 binaries, you have to add the following repository if you don’t have it:

<repository>
  <id>SpringSource Enterprise Bundle Repositorys</id>
  <url>http://repository.springsource.com/maven/bundles/milestone</url>
</repository>

and the spring packages that you need since the packaging has changed from 2.5.x. Instead of a single spring.jar file, now there is one per feature so you have to sort out what jars to include in the project. I ended up with the following:

  <properties>
      <spring.version>3.0.0.M2</spring.version>
  </properties>
  <dependencies>
   ...
  <dependency>
      <groupId>org.springframework</groupId>
      <artifactId>org.springframework.core</artifactId>
      <version>${spring.version}</version>
    </dependency>
    <dependency>
      <groupId>org.springframework</groupId>
      <artifactId>org.springframework.web</artifactId>
      <version>${spring.version}</version>
    </dependency>
    <dependency>
      <groupId>org.springframework</groupId>
      <artifactId>org.springframework.transaction</artifactId>
      <version>${spring.version}</version>
    </dependency>
    <dependency>
      <groupId>org.springframework</groupId>
      <artifactId>org.springframework.orm</artifactId>
      <version>${spring.version}</version>
    </dependency>
    <dependency>
      <groupId>org.springframework</groupId>
      <artifactId>org.springframework.jdbc</artifactId>
      <version>${spring.version}</version>
    </dependency>
    <dependency>
      <groupId>org.springframework</groupId>
      <artifactId>org.springframework.web.servlet</artifactId>
      <version>${spring.version}</version>
    </dependency>
    <dependency>
      <groupId>org.springframework</groupId>
      <artifactId>org.springframework.context</artifactId>
      <version>${spring.version}</version>
    </dependency>
    <dependency>
      <groupId>org.springframework</groupId>
      <artifactId>org.springframework.aop</artifactId>
      <version>${spring.version}</version>
    </dependency>
    <dependency>
      <groupId>org.springframework</groupId>
      <artifactId>org.springframework.expression</artifactId>
      <version>${spring.version}</version>
    </dependency>
    <dependency>
      <groupId>org.springframework</groupId>
      <artifactId>org.springframework.test</artifactId>
      <version>${spring.version}</version>
    </dependency>
  </dependencies>

Once the jars are brought, I’ve replaced my old jarfiles with the all-new M2s then fired the build target. The first issue I found was with the asm version.

March 1, 2009

Spincloud Labs: Political boundaries overlay in Google maps (Part 2)

Update Sep.21.2011: I took the code in the two parts and made a github project off of it called Gborders. The code is simpler and there are more options to generate the borders overlay based on geographic regions. Happy forks!

In Part 1 we imported world political borders into a database table. In this second part we'll use the table and generate a script that will be used to add the borders overlay to Google Maps. We'll use the cool Ruby and some fancy GIS words along the way.

We left-off with a database table containing all borders. The goal today is to produce a Javascript file that will be used for overlaying polygons representing countries, over a map using the Google Maps API.

Let’s examine the table data first (I’m using mysql through the command line):

mysql> desc world_boundaries;
+-----------+--------------+------+-----+---------+-------+
| Field     | Type         | Null | Key | Default | Extra |
+-----------+--------------+------+-----+---------+-------+
| ID        | int(11)      | NO   | PRI | NULL    |       |
| FIPS      | varchar(255) | YES  |     | NULL    |       |
| ISO2      | varchar(255) | YES  |     | NULL    |       |
| ISO3      | varchar(255) | YES  |     | NULL    |       |
| UN        | int(11)      | YES  |     | NULL    |       |
| NAME      | varchar(255) | YES  |     | NULL    |       |
| AREA      | int(11)      | YES  |     | NULL    |       |
| POP2005   | bigint(20)   | YES  |     | NULL    |       |
| REGION    | int(11)      | YES  |     | NULL    |       |
| SUBREGION | int(11)      | YES  |     | NULL    |       |
| LON       | double       | YES  |     | NULL    |       |
| LAT       | double       | YES  |     | NULL    |       |
| ogc_geom  | geometry     | YES  |     | NULL    |       |
+-----------+--------------+------+-----+---------+-------+

A lot of information here but we’ll need just these fields: name (country name), iso2 (two letter country codes) the_geom, iso2 (border geometry) and region (grouping countries by regions). To check the data-set let’s examine a small country. I’ll pick the tiny State of Vatican for its size:

mysql> select iso2, AsText(ogc_geom), region from world_boundaries where iso2='VA';
+------+---------------------------------------------------------------------------------------------------------------------------------------------------+--------+
| iso2 | AsText(ogc_geom)                                                                                                                                  | region |
+------+---------------------------------------------------------------------------------------------------------------------------------------------------+--------+
| VA   | MULTIPOLYGON(((12.445090330889 41.903117521785,12.451653339581 41.907989033391,12.456660170954 41.901426024699,12.445090330889 41.903117521785))) |    150 |
+------+---------------------------------------------------------------------------------------------------------------------------------------------------+--------+
1 row in set (0.00 sec)

The only surprise here is that the country border is described as a MULTIPLOYGON spatial type which describe a collection of polygons that don’t interset. This is in order to accommodate countries that have islands under ownership.

Let’s see how these points look on the map. We’ll use the excellent polygon encoder utility written by Mark McClure. Copy Vatican’s point set in the “Input Text” input box (choose lng/lat option):

12.445090330889, 41.903117521785
12.451653339581, 41.907989033391
12.456660170954, 41.901426024699
12.445090330889, 41.903117521785

Here’s the result: va_polygon Not very accurate but we’ll have to live with it, it’s a free data set after all…

February 22, 2009

Atlantis spotted on Google Earth?

I couldn’t miss this one. It didn’t take long after Google released its seafloor imagery to its Earth and Maps products for someone to claim the discovery of nothing less than Atlantis itself! Ah, and again it wasn’t me… Take a closer look at the Atlantic Ocean floor area: On the left, a grid-like structure is clearly visible. I also spotted a similar “grid” east of the main one (yes, I called it!

February 22, 2009

Spincloud Labs: Political boundaries overlay in Google maps (Part 1)

Update Sep.21.2011: I took the code in this post and part deux and made a github project off of it called Gborders. The code is simpler and there are more options to generate the borders overlay based on geographic regions. Happy forks!

One thing I needed when designing the Meteoalarm mashup for Spincloud were the political boundaries for all European countries. With them at hand, I would use the polygon overlay from the Coogle API and fill the country polygons with the respective weather warning colors.

This first part is a tutorial on how to import world political borders into a MySQL database table. The second part in these series will use this table to create a script that will add the country borders overlay to Google Maps.

So after learning that Google Maps API doesn’t give programatic access to the political boundaries I moved on to doing it myself. I read about the encoded polylines and figured that I only missed the country boundaries as encoded polygons so I started searching for readily-available resources. Sure enough there were nowhere to be found.

The next best thing was to find the country boundaries in some GIS format and generate the polyligons myself. The information seems to be sparse but after a bit of research I found was the CIA World Databank but the format is incomplete, it only defines the polygons with no country names or any other administrative hook that I would be able to use. I then spent hours trying to find a better resource until I remembered that Mapping Hacks (an excellent book by the way) has data set for the examples in the book. I had the first breakthrough, the website makes available updated world borders files here, they are mirrored from thematicmapping.org.

We’ll use a utility called shp2mysql to export the boundaries to SQL. I’m on a MacBook and no shp2mysql binaries for my platform were available in the original package so we’ll have to compile it first then run it to be able to export the shapefile to SQL.

Below are detailed all steps needed to to produce the borders SQL file on Leopard 10.5.x (note that you have to have the Developer Tools installed). In short, copy and run the following script on your mac in a file called process_boundaries.sh. The script creates a file called wb_dump.sql which is ready to be imported into database:

February 10, 2009

Spincloud, a month old: looking good

I launched Spincloud on New Year's day. I didn't expect a flood of traffic to hit the site since -let's face it- there was nothing groundbreaking. Spincloud was born from my idea of having access to the world weather in one step or less and I'm pleased with the results so far. Spincloud gains adoption every day and this is only due to the people that saw something new and exciting in it. There are blog posts, traffic graphs and other goodies to talk about so let's begin.

I announced spincloud’s launch on my personal blog then Keir from googlemapsmania broke the news to the community. And so the first spike of traffic was logged on Jan 2nd. 

jan_traffic

Then followed a month of relative silence during which I took a much needed break from the project. I couldn’t help but refactoring and cleaning-up the code a bit. I had two minor releases deployed in prod in the mean time and everything went smooth.

I had a new feature I wanted to have online before the initial release, the meteoalarm mashup that displays weather warnings across Europe but chose not to delay it (a Good Decision in hindsight) . I started working on it by mid January , did a lot of digging for administrative borders data then getting the meteoalarm warning data right (those guys don’t have any integration point). I was done two weeks later.

In the mean time I was trying to raise the awareness about spincloud. I submitted the site so some mapping/mashup themed websites, improved the indexing (still working on it). Then Wednesday last week the European bloggers finally found it. UK’s Mapperz ran a nice little story that seems to have set the things in motion. A number of other bloggers picked-up the story and a surprisingly good feedback came from Italy: Matteo and italiasw.com did a great job while  gisdk.blogspot.com of Denmark  noticed the RSS facilities. Then the referral machine kicked in,  killerstartups.com, programmableweb.com amongst the sites linking. And thus the second spike registered on the traffic meter:

February 5, 2009

@Meteo_alarm twitter stream now available

= Update Nov.25.2009: The twitter name has changed to @meteo_alarm. Please update your bookmark or follow list. Between code refactoring and cooking new ideas, I took some time to add a new feature to Spincloud. Inspired by the crowd-sourced weather updates triggered by really bad snow storms in the UK, I (finally) started to see the advantages to using Twitter. So I took the Meteoalarm data that I mashed-up on Spincloud's map and streamed it to Twitter.

January 31, 2009

Spincloud now integrates with Europe's weather warning service Meteoalarm

A short month after the initial public release of Spincloud, I am pleased to announce the addition of a new feature: integration with the European weather warning service Meteoalarm which provides the most relevant information needed to prepare for extreme weather, expected to occur over Europe. The colors indicate the severity of the danger and its possible impact: Red: very dangerous, Orange: dangerous, Yelllow: potentially dangerous, Green: no warnings. Currently only the country-level warnings are displayed.

January 10, 2009

@BreakingNewsOn map

I’m a bit of a news junkie and when I heard that a twitter stream called @BreakingNewsOn existed I did the natural geekish thing, mash it up. Yahoo pipes was my friend here. I haven’t used the Pipes before but that didn’t quite matter, Yahoo did a great job keeping this playgroundsy product in the child realm so even a caveman could do it. The mashup result -two pipe connections later- is beyond the link:

January 1, 2009

Announcing a new weahter mashup: Spincloud

I started working on Spincloud back in 2003 when I got interested in regular expressions. I liked them so much I started looking for large sets of freely available data that I could crunch. Weather data was quite appealing; not only I’d parse it but I could get instant feedback by just looking at the parsed data, after all we understand temperatures and wind speeds and the like. I’d get to use other technologies I couldn’t experiment at work where I was doing the regular J2EE stuff and I kept feeling the itch to do new things.

December 17, 2008

Mini DisplayPort saga: a reverse hangover

One benefit of owning a Mac is that you’re part of a never ending soap opera that Cupertino carefully directs. Sometimes the drama has bright spots such as the Mini Display Port story. The MDP became quite the feature story when Apple decided to scrap the DVI port on its new unibody lineup. I didn’t pay too much attention initially but since the Macs are so media friendly, I found myself wanting to plug it into a TV basically the second day after I bought it.

November 22, 2008

libcurl.3.dylib and Google Earth

I just installed Google Earth on my mac. It went smooth, the software was installed under Applications but when I launched it I got this: The application Google Earth quit unexpectedly. Hit the Report button and got: Library not loaded: /usr/lib/libcurl.3.dylib Referenced from: /Applications/Google Earth.app/Contents/MacOS/libge_net.dylib Reason: image not found I actually upgraded libcurl a while ago and to 7.19.0 and I now only have /usr/lib/libcurl4.dylib libcurl.3.dylib version is gone. The solution I found was to create libcurl.

November 21, 2008

Gmail themes, skin deep

Today was my lucky day according to least one popular blog. I got the Themes tab enabled in my gmail account, wohoo! I only wish they could change the font sizes too, I’m up for a microscopic look’n feel, really small and cozy-cute. My current favorite is the Shiny theme: Now, if only googlers would add a Cover Flow option instead of that dull message list. Oh, and it should come gratis as usual.

November 7, 2008

Deus ex Machina

I love Hollywood movies. They project a surreal world that in which all stories have a happy ending. Whenever the hero seems to be cornered and rapidly heading to his doom, a last-second turn of events allows him to escape. This is deus ex machina. God from the machine. To the rescue. I’m thinking back at many movies I so enjoyed. Terminator, Die Hard (yeah, right), Batman, Ironman, Indiana. Deus ex machina all over.

November 6, 2008

REST is DSL

I like RESTful URLs. They clearly express the intent of the provided resource. We humans speak a language close to REST when we talk to The Internet. Take a look at Google’s RESTful search query, probably the most used RESTful URL ever: http://www.google.com/search?q=your-terms-of-choice in freeform language this translates to: “Google, search for this term.” How about the Flickr RESTful tag cluster search service: http://api.flickr.com/services/rest/?method=flickr.tags.getClusters& api_key=a6f11b67ae348a7b3f2da0d865b8bc1d&tag=Berlin In Flickr lingo this means: “Fetch all tag clusters for the ‘Berlin’ tag”.

November 4, 2008

Mac wars

Oh, that “I’m a Mac” commercial is on the TV again. Then some ugly new ad where some webcam people defend the PCs on behalf of Microsoft: “I’m a PC”. While my brain was tricked into an attempted wash, I realized that I’m not any of it.I’m a human being. I’m not trying to help you here Redmond but voila, here’s one good ad slogan: I am not a machine. I’m a human being.

November 1, 2008

java.lang.Void

VoidI’ve considered myself a Java guy (that’s before I recently started with Ruby). I’ve stumbled upon Java in 1998 and fell in love with it at first sight. I remember quite crisply my first encounter of the third kind with Hello World. But I confess, I only learned about java.lang.Void yesterday. I was digging through the SwingWorker concepts and as I went through the tutorial I was intrigued by a mention about the Void type in the Simple Background Tasks section. I already experimented with the SwingWorker before reading the tutorial and liked it except for the annoying return and interim results types as I’m migrating an ancient Java 1.1.8 GUI to JDK 6 at work and those Runnable classes I’m thinking of converting to don’t have any return types.

October 24, 2008

My very first mac: one with the machine

I just bought my first MacBook yesterday. My current laptop (a beaten down Toshiba Satellite) was long overdue and I realized that I didn’t want yet another boring Win notebook, it just doesn’t do it for me anymore. I’m in front of a PC every day at work and I surely don’t want to see the same things while at home. And so I was determined to buy a Mac. My first Mac. I didn’t know that much about Apple and still don’t (imagine that)… I waited until the new MacBooks were announced (no rush, I don’t know much about these machines anyway) and I started to do my research. A $1500 laptop seems a reasonable price for a good laptop and I had two requirements: not that big and not that heavy. MBPs seemed too expensive so my choices were the 2.0 or 2.4 MacBook. I eventually choose the 2.4 version mostly because of the backlit keyboard.

September 5, 2008

OK, starting... now

I wanted for a while to start blogging but pretended to be busy or actually being too tired of working long hours in front of my laptop to actually start writing about things. But now the time has come. Or something.