July 15th, 2014

Beyond the CMS

NPR BuildingOn April 22, 2011, I was in Washington, DC, preparing to start my new job at NPR. At that point in my life, this was pretty much my dream job, so I was very excited and a little nervous. I did a lot of thinking that night and the conclusions I came to eventually became the basis of NPR’s technology strategy. I recently had a chance to share my thoughts from that night as part of a talk at the Integrated Media Association’s iMA 2014 conference. Here are the edited highlights.

The basic premise I started from was that all content management systems are fundamentally broken. This may sound a little harsh but I feel able to say it because I’m part of the problem – I’ve built content management systems for organizations across the public and private sectors, so I’m pretty well placed to tell you that no available CMS platform is architected for what publishers – particularly news outlets – truly need.

Most content management systems were designed years ago, for a much simpler world. We now live in an incredibly fragmented and complex world. Any piece of content tends to be sourced from a variety of places and published across a range of old and new media channels. Throughout this complex process, everything has to work seamlessly. The margin for error during breaking news or major events is pretty much zero.

In this context, what do publishers actually need from a CMS? They need:

  • An easy way to connect with many news sources
  • The ability to push content across a variety of channels
  • Guaranteed availability and scalability

So, how do we build a CMS that actually addresses these needs? To my mind, the solution has three key components. First and foremost, the whole architectural approach must be based on APIs. Second, it must specifically use hypermedia APIs and finally, the APIs must be what I’ve been calling “linked APIs”.

1. APIs First
APIs represent the only universal way to connect anything on the Web to any other online thing. Unfortunately, since we started the Web in a desktop-centric world, APIs were an afterthought. Historically, we used to build a Web site and then maybe also add an API, as a window into our content.

This is the wrong approach. Your Web site is just one of the destinations for your content. Increasingly, it’s not even the most important one, since mobile viewership is clearly on the rise. Don’t treat your Web site as special. All your content and functionality should be put into and delivered through APIs.

 2. Hypermedia
Publishers need things to just work. They don’t care about the technical details; they just can’t have their services go down at any time – so, scalability is paramount. And how do you ensure scalability? As I’ve pointed out before, the most scalable network ever created is the World Wide Web and the secret to the Web’s scalability is hypermedia.

Hypermedia is any type of content that not only carries data but also links to other documents. The hypermedia type that is most fundamental to the Web – and certainly the one we are most familiar with – is HTML. However, HTML was designed for human-centric Web sites, not for exchanging structured content via APIs.

There are, however, other hypermedia types that were designed for this very purpose. As a matter of fact, I was involved in the creation of a very robust one called Collection.Document, which was designed specifically for media organizations.

3. Linked APIs
Leveraging hypermedia as an integral part of interface design allows us to create “linked APIs”. Most current APIs are, at best, creating narrow windows into the solid walls of data silos. Even the most high-profile API will typically only provide access to a single corporate database. Hypermedia allows us to create links between these databases.

This will prove essential to the next generation of content management systems because linked APIs have the potential to give content publishers the freedom they want to seamlessly integrate content from diverse sources and push it across the full spectrum of online channels. As such, they could even come to represent the engine that drives press freedom into the coming decades. So, let’s get that engine cranking!

June 26th, 2014

APIs in the Connected Car: APIdays San Francisco

APIdays SFToday, I’m going to share some rather opinionated thoughts about APIs and the connected car. My opinions on this subject sprang from a combination of real-world experience plus (informed) speculation and came together as I prepared a talk for APIdays San Francisco.

The connected car is widely recognized as a game changer for the automotive industry. Experts all agree that just selling cars is a thing of the past. Mobility, connectivity and in-car user-experience will be leading decision considerations for car sales. Right now, automotive manufacturers, content providers and app developers are all competing to take a leading role in the connected car space. This is a matter of survival. Winners of the competition will be richly rewarded; the losers may sink into oblivion.

Car manufacturers seem understandably determined to dominate the connected car space. But this space is inherently shared with device manufacturers, content providers and app developers. Take away any one participant and you no longer have a sustainable ecosystem. If the automotive sector is not prepared to work with and accommodate the needs of other stakeholders, then no one will win. There are three things the industry can do to make things significantly better right away.

1. Implement a Standard Hypermedia Type for Automotive APIs
Right now, every car manufacturer wants to do its own thing and sees originality as a key to differentiation. This is a fallacy. There are way too many car manufacturers for content providers and app developers to keep up with the variety. Some have suggested that all manufacturers should just deploy Android as the base OS. I personally doubt they will all be able to agree on something as fundamental as the core OS. We should shoot for something much more realistic.

This is where hypermedia comes in. The most distributed system ever built — the World Wide Web — uses a hypermedia type (HTML) as its engine. There is a great opportunity to create a hypermedia format for car APIs that will energize the space just like HTML did for the Web. I believe this format could be based on an existing, generic type such as: Uber, HAL or Siren. This would be similar to the way the Collection.Document type was created for the news media industry, based on Collection.json.

2. Adopt a Standard API Security & Identity System
The prospect of connected cars getting hacked creates enormous anxiety. But connected car security can be addressed quite simply by adopting a security framework based around compartmentalization and standards-based access control.

In this context, “compartmentalization” means that core functions of the vehicle should be highly guarded. Specifically, no third-party app should have access to core driving functions like handling and braking. Meanwhile, a standards-based access control framework like OAuth will provide secure, granular access to specific system features. This would be similar to the way mobile apps currently ask for access to other parts of the device (GPs, contacts etc.)

3. Enable App Developers
Currently, only the lucky few are able to develop apps for connected cars. Generally, these are app vendors that have formal partnerships with car manufacturers. In most cases, developers can’t even get access to API documentation without a group of lawyers signing stacks of papers. The connected car space will not develop if it remains a tightly-held, closed system. On the contrary, manufacturers must build developer communities by providing the things that developers require: documentation; self-service portals; sandboxes; SDKs etc.

But That’s Not All
These are three immediate steps that can be taken to improve the connected car space significantly but as the space develops, we will have to focus not only on immediate requirements but also on the big picture. The connected car is a special case of the Internet of Things (IoT). The context of IoT is different enough that it requires a fundamentally different approach to system design and architecture. Hopefully, I will be able to delve into this context more in future.

Another aspect of the big picture is a good deal simpler: fun. If this space is going to develop as it should, manufacturers will have to make it fun for developers to experiment with the potential of automotive connectivity.

So, have fun out there!

June 7th, 2013

Hypermedia Workflow Questions

Hypermedia WorkflowI fairly often get emails following up on the workshops, articles, webinars and online tutorials I take part in. I can’t always answer these questions directly and sometimes deal with them in blog posts or online articles. Following my recent API Tech Talk on hypermedia, I got some questions from Abiel Woldu on how to handle hypermedia support when the same backend operation is called from different workflows. Here’s part of the Abiel’s email:

“Say you have an end point for validating address; call it /validateAddress. Now this endpoint is called from two work flows.

  1. When a user updates his account settings (changes a new address)
  2. When a user tries to buy a product and enters the shipment address

In both cases the /validateAddress should give different set of links and forms as part of the response of validation (next step affordances) because the flow is different. In this case what is the set of the next links and forms returned from the endpoint? Is it the union of the two workflows and the client knows how to get what it needs? Or does the client send information of which flow it is in and the server uses the information to figure out what response to give it?”

Decoupling Backend Processes from Public URIs
This kind of question comes up frequently. Essentially, there are a couple assumptions here that are worth exploring. The first is the idea that a backend operation (e.g. “validateAddress()”) is exposed over HTTP as a single endpoint, no matter the calling context. This is not a requirement. In fact, it is advantageous to decouple public addresses (URI) from private operations on the server. HTTP (whether using HTTP-CRUD, Hypermedia-REST or some other model) offers the advantage of using multiple public URIs to point to the same backend operation. For example, it is perfectly correct to publish both /validateExistingAddress and /validateNewAddress URIs each of which points to the same  “validateAddress()” operation on the server.

Not Everything Needs a URI
Just because the backend server has an operation such as “validateAddress()” does not mean there has to be a URI associated with that operation. For example, the “user updates his account settings” workflow need not have a direct URI call to “validateAddress()”. Instead, there could be an account settings resource (/account-settings/) that supports the HTTP.PUT method and accepts a body containing (among other things) a modified address. Executing this client-side operation (PUT /account-settings/) passes data to the server and – along with other operations – the server calls the “validateAddress()” operation itself and reports the results to the client.

The same can be done in the case of “user tries to buy a product and enters the shipment address”. This address validation could be a small part of the server-side operation and processing of an HTTP.POST to a /check-out/ resource.

Mapping Actions to URI & Method
In the HTTP-CRUD model, the focus is on using URIs to identify entities and/or operations and using the protocol methods to perform actions. For example, an /addresses/ resource that supports adding (POST), modifying (PUT), removing (DELETE) and retrieving (GET) addresses associated with a context (logged in user, check-out processing etc.) In this case, POSTing or PUTing a resource body to the server allows the server to call the “validateAddress()” operation (among other things) and report results to the client.

Mapping Actions to Hypermedia Controls
In the hypermedia model, actions are described using a hypermedia control such as a link or form. The URI is not important in this model. Instead the control has an identifier (e.g. “validate“), indicates a protocol action (“POST“) and lists state data to include in the payload.

In Siren it might look like this:

"actions": [
 {
     "name": "validate",
     "title": "Validate an Address",
     "method": "POST",
     "href": "...",
     "type": "application/x-www-form-urlencoded",
     "fields": [
           { "name" : "Street", "type" : "text", "value" : "123 Main Street" },
           { "name" : "City",   "type" : "text", "value" : "Byteville"},
           { "name" : "State",  "type" : "text", "value" : "MD" },
           { "name" : "ZIP",    "type" : "text", "value" : "12345"}
     ]
     }
 ]

Note that I didn’t bother to enter a value for the href in this example. It could be any valid URL; I just left it out.

Tracking Workflow Progress Within Messages
Here’s another question from Abiel Woldu’s email:

“The concept of which work flow the client is going through – is it code that should reside in the API code itself or it’s something that sits outside in some other gateway or something?”

When implementing processes over HTTP, it’s wise not to rely on stateful multi-request chains. In other words, don’t expect either the client or server to keep track of where some request belongs in a workflow. Instead, include that information in the request and response bodies themselves. This pattern of including all the important context information with each request and response not only assures that the request can be handled independently (e.g. in a load-balanced cluster), it also helps clients and servers to do work within varying time-spans (e.g. a clients can cache the last request to disk and pick things up a day later). In the REST model, Fielding described this as making messages “self-descriptive”.

For example, there might be a use case that prompts human users to provide quite a lot of information (across various UI tabs) before finally submitting this completed set of work to the server for final validation and processing. One way to support this over HTTP is to allow clients to store “work-in-progress” (WIP) records on the server. As each “tab” (or other UI affordance) is completed, the client app is free to execute a POST or PUT operation with the payload to a URI supplied by the server. The stored data would include a value that indicates how far along in the workflow the user has progressed. This same client app could also recall stored WIP records, inspect the workflow indicator and prompt the user to pick up where she left off. Once all the required elements were supplied, the work could be forwarded for final validation and processing.

Dynamic Workflow via Hypermedia
Finally, in some cases, the series of steps in a workflow might vary greatly at runtime. For example, a service might support a multi-tenant model where each instance of “supply all the details for this work” has different steps or the same steps appear in differing order. The “next step” need not be memorized by the client code. Instead, hypermedia servers can inspect the current server-side configuration, check the current progress by the user and then supply the correct “next step” for this particular instance.

In this way, the client app can support a wide range of workflow details without needing custom code ahead of time (or even downloaded code-on-demand). Instead, the client app only needs to be able to recognize the “next step” link and navigate to that resource.

In Summary
In general, when using HTTP:

  1. There is no rule that you must expose internal methods as public URIs
  2. You may use more than one URI for the same backend operation
  3. In the HTTP-CRUD model, you usually map operations by linking URIs and methods
  4. In the hypermedia model, you usually map operations by linking controls and state variables
  5. It is best to use “self-descriptive” messages to track workflow progress statelessly
  6. The hypermedia model supports dynamic workflow progress using the “next step” link pattern

Thanks to Abiel for his questions and his generous permission for me to use his email and name in this blog post. If you’ve got a question that I haven’t answered online before, feel free to ping me via twitter (@mamund) and fire away.

December 7th, 2012

Use Hypermedia to Reduce Mobile Deployment Costs

Using Hypermedia to Reduce CostsI speak about the power and flexibility of hypermedia quite often. I explain the general idea behind hypermedia, discuss its historical roots and show how it can help client applications adapt to changes in data input and application flow. Essentially, a hypermedia-based approach aims to take key elements often placed into the client’s source code and move them into the actual response messages sent by the server.

I also point out that using a hypermedia-based approach to building client and server applications takes a different kind of effort than using RPC-style approaches. And I explain that, currently, there is a limited amount of tooling available to support the process of designing, implementing and maintaining hypermedia-style systems.

If your work involves designing, building, testing and deploying a mobile client application, it is likely you need to deal with an “application store” or some other process where your packaged application must be submitted for review and approval before it is available to users for download. This can happen not only with well-known public offerings such as the Apple Store but also within any organization that provides its own application repository aimed at ensuring the safety and consistency of user-available mobile apps.

In an environment of quick-turnaround, agile-style implementations this “app store” approval can be a real bottleneck. It may be not just days but weeks before your app is tested, approved and posted. This can be especially frustrating when you want to deploy a rapid-fire series of enhancements in response to an engaged user community.

A hypermedia-based client design can often support UI, data transfer and workflow modifications by altering the server messages rather than altering the client source code. By doing this, it is possible to improve both the user experience and the system functionality without the need for re-submitting the client code for “app store” review and re-deployment. This also has the potential to reduce the need for interrupting a user’s day with download and reinstall events and can, in the process, cut down on the bandwidth costs incurred during the repeated roll outs of code modifications to a potentially large user base.

Improved agility, a better user experience and reduced bandwidth costs are all tangible benefits that are possible when investing in a hypermedia-based implementation for your mobile client application.