Blog

Where we discuss our work, thoughts, and process

First we built an API, then we built a CMS

API design has been a point of discussion on Hacker News recently. The discussion has been focused on the process of adding on an API to your app. With our SaaS product, the HiFi CMS, we approached the problem from the opposite direction: flesh out the API first and then build your application on the API. There was a lot of interest in our approach to this, so we thought we would write out this post.

Our application, HiFi, is a modern web cms that is the antithesis of most cookie-cutter CMS SaaS products on the market. A core goal with HiFi has been to create a platform where professional designers, web developers, and information architects can create websites that live up to their original vision. Early in the planning process it became clear a flexible, first-class API would be needed to achieve this goal. Getting the API right would be the only way a hosted CMS could acheive the flexibility that developers love about non-hosted platforms.

Our requirements for the API were driven by common sense best practices and by the unique needs of website content models. The key requirements were:

  • Consistency - all objects in the system should be represented, manipulated, searched, versioned, and access controlled in the exact same way. This holds for content, types, users, relations, permissions, etc.
  • Simplicity - there are only two methods in our API: get and put. An object's type and its "undeleted state" are just properties of each object. JSON is the only format for V1. Further more, the requests made are in a very similar format to the data you get back. It just feels natural.
  • Versioning / Immutability - storage is cheap in 2010: why not keep everything? Objects aren't updated or deleted, new copies are created.
  • Structure - all objects are organized in a tree. Relationships between objects can introduce additional structure.
  • Queryable - all objects can be queried by criteria on their own properties and properties of structuraly related objects
  • Access Controlled - read / write permissions can be applied to any object and inherit down the tree

We built the API to satisfy these requirements first, then we built our app on top of the API. This turned out to be a great idea. We got to dogfood our API for the entire development process and it made testing a lot simpler.

Websites running on HiFi use a templating language, twig, which is very similar to Django templing. All templates have access to the API server-side. This makes it really easy to add complex features to websites without much stress. Here are a few quick examples of queries:

{"type":"page", "orderBy":"-publishedAt"}

This query pulls in the most recently published pages in HiFi.

{"type":"page","orderBy":"-publishedAt","parent":{"type":"feed","url":"/blog"}}

Fetch the most recently published pages whose parent is a blog feed with url '/blog'.

{"type":"page","orderBy":"-publishedAt","child":{"type":"comment"}}

Fetch the most recently published pages which contain a comment.

All nested queries can be combined as much as possible and be as recursive as desired. There are similar commands for working with relationships. The net result is an API that was simple, yet powerful enough to build our admin panel on.

HiFi UI

HiFi's admin user interface is 95% client-side using jQuery, Resig's JavaScript templates, and pulls data via the API. We created a HiFi JS library that creates a jQuery-like chainable DSL around the API. The next example should look familiar to jQuery users. In it we'll change the URLs of all pages on a site to be lower case using this:

hifi({type:'page'}).each(function(page) {
    hifi(page).update({url:page.url.toLowerCase()});
});

A neat side-effect of writing the API first is finding use cases you didn't plan for. One example for HiFi was in keeping old versions of all objects. The intention for this API feature was to enable easy roll backs for any content type. When we recognized that old URLs were being stored and were queryable through the API we updated our website routing code's logic to see if an unreachable URL ever existed and if so we automatically 301 redirect to the object's most recent version's URL. We got this for free. Had we built the app first we would have written a lot of special case code to achieve the same result.

Now that HiFi is open for business and over 50 websites have launched on the service we're beginning to see uses of the API in end-user work that are really exciting. An example site that has personally blown me away is PBS' CiaoItalia.com. Ciao Italia is America's longest running cooking show. Over the course of 21 seasons chef Mary Ann Esposito has compiled a lot of great recipes and videos. If we were to flatten the APIs tree it would look like: 21 Seasons -> ~20 Episodes each -> ~4 Recipes each -> YouTube Video. This advanced search page was implemented entirely using the API in user-space templates.

HiFi's API was built before we built the app on top of it and it's a better product because of it. If you're not in a greenfield situation and your app already exists consider what it would take to design an API that could eventually sit beneath your app not beside or on top of it. Investing in a great API not only cleans up your code base, but also opens up the door for awesome, unanticipated use cases coming to life.

Comments

JL's avatar
JL
Fascinating read, I am taking a similar approach yet not so unified as yours. I'll have to reconsider some of my approach.

The versioning and open data model sounds intense, are you on a SQL or no-sql (mongodb) type of technology? Or maybe something altogether different, like Redis.
Kris Jordan's avatar
Kris Jordan NMC team member
JL, We're running 'schema free' on top of MySQL. A similar approach to what FriendFeed did (http://bret.appspot.com/entry/how-friendfeed-uses-mysql). Not quite the same though. With HiFi a Type is 'just another object' that is special cased behind the scenes. When a type is created or modified the API does the work of building out the tables necessary for it. When an object of that type is created the system keeps a gzip'ed copy of the JSON blob on the central table similar to FriendFeed and fills in the fields of the flat type table so that properties can be queried against.

There is type inheritance (happens when a type object's parent is another type object, fun exploit of the tree structure) in which case extended types create tables with just their additional fields. This follows Martin's class table inheritance pattern (http://martinfowler.com/eaaCatalog/classTableInheritance.html).

At some point I'll try to write up a more technical post on the architecture behind the API.
Joey Guerra's avatar
Joey Guerra
I think it's funny when people talk like this is something so new. The fact is, when you build a RESTful application, it IS an API. But, hey, kudos to you for finally getting to this.
Pete Forde's avatar
Pete Forde
Joey, I think you actually managed to miss the entire point of the article.

*whoooooosh*
Kai's avatar
Kai
Indeed, designing the API first and then implementing the product is a fascinating way. Not unlikely to start with the documentation and then implement what's in the manual.

We did the same approach with our web imaging engine, where documentation (including api, use cases and example code) was finished before we started to implement it.

It has huge benefits in testing as well, as user-tests can start in a mockup-scenario with the dry api.

As a result, Intermedia enables you to write code like this on the server for easy pixel transmogrifications:


overlay1 = intermedia.getImage ('clipping_path/business-man.jpg');
overlay1.transform ([
intermedia.pixelProcessor ({clip : {bgcolor : "#88e" }}),
intermedia.pixelProcessor ({roundcorners : { width: 80, height: 80}}),
intermedia.pixelProcessor ({clip : {}}),
intermedia.pixelProcessor ({rotate : {angle:-45}}),
intermedia.pixelProcessor ({scale : {xsize : overlaysize, ysize : overlaysize}}),
intermedia.pixelProcessor ({text : { text : "Wow!", 'font-size': fontsize, position: "br",
'offset-x' : fontsize * -3}})
]);
JL's avatar
JL
Thanks for the reply Kris.

That sounds very cool, and complex. I would have probably given over to the MongoDB hype at that point.

Keeping things in MySQL is great for maintaing that familiar tool chain though and that is definitely worth something.

Would definitely be interested in reading a more detailed architecture post.

Leave a comment