“Government information shall be managed as an asset throughout its life cycle to promote interoperability and openness, and, wherever possible and legally permissible, to ensure that data are released to the public in ways that make the data easy to find, accessible, and usable” (emphasis mine).

No Dumping
The usual approach to this type of work is to simply publish raw data in a directory or repository and then create some fencing around the data that helps track usage and distribution. Essentially, making government data “open” becomes a data dumping operation. This practice fails on all of President Obama’s three key points. First, data dumps make finding valuable information not at all easy. Second, even though the content might appear in a standard format like XML, CSV or JSON, it is hardly accessible (except for to geeks, who love this kind of stuff). And finally, raw data is hardly ever usable. Instead, it’s a mind-numbing pile of characters and quote marks that must be massaged and re-interpreted before it comes close to usability.

So, while this new directive offers an opportunity to make available a vast amount of the data the government collects on our behalf, the devil is in the details. And the details are in the interface – the API. As with poorly-designed kitchen appliances and cryptic entertainment center remote controls, when it takes extensive documentation to explain how to use something, the design has failed. There’s a simple principle here. Poor API design results in unusable data.

Affordable Data
It doesn’t have to be this way, of course. Government departments have the opportunity to implement designs that meet the goals set forth in the executive order. They can make it easy for people to find, access and use the data. They can publish not just data but APIs that afford searching, filtering and exploring the data in a meaningful and helpful manner; APIs that empower both users and developers to successfully interact with the data, without resorting to a dashboard featuring dozens of options or mind-numbing explanations.

In the (likely) event that the initial open data release consists of mere data, companies and individuals would be well advised to resist the temptation to build a multitude of “one-off” applications, each of which solves a single problem or answers a narrow set of questions for some subset of the data. Instead, work should be put into converting the raw data into usable API formats such as Atom, OData, HAL, Collection+JSON and HTML (to name just a few). APIs should be designed with the same care that would be given to any interactive experience.  Investment in tools and technologies that can properly represent the data in multiple formats while supporting various use cases and access requirements will yield great results.

Open Data APIs
In the end, organizations that know the importance of a good interface, the power of choice and the freedom of flexible representations will be able to convert raw data into valuable information, which can be consumed by a wide range of users, platforms and devices. These considerations are essential to building and supporting open data APIs.

Because – ultimately – data isn’t open, unless it’s “easy to find, accessible, and usable”.

If They Have to Ask…
Try this:

• Create a new Web API
• Get it up and running on some server or other
• Hand the single URL to a client dev and say: “There ya go!”

Is the API self-descriptive? Does it contain enough information in the responses to allow client devs to know what the API is for, what it is capable of and how they can make valid requests to the server and properly parse the responses?

Here are some questions for you:

• How many assumptions do you have about your API?
• Are these assumptions shared by client devs?
• All clients devs?
• Even ones who have never met you?

If your answer to any of those questions was “No” or “I’m not sure” then it’s likely that devs will need to ask you a thing or two about how to properly use your API. That’s no big deal, right?

…You Didn’t Afford It
In everyday life, if people have to ask how to use a device (television remote, toaster etc.) then you can be sure that device is “poorly afforded” – it’s a case of weak design. We all know devices (especially electronics) that come with huge manuals and complicated explanations – and we all know what a bummer it is when that happens.

In this respect, your API is the same as any other consumer device. It should be “well afforded” – developers shouldn’t have to read the technical equivalent of War & Peace before they are able to successfully use your API.

Yes, you can supply detailed instructions in prose, provide a long list of possible methods, include lots of tables etc. These resources are helpful for devs but they can be daunting to read and cumbersome to maintain.

Another approach is to include this kind of information in a machine-readable format – and one that most devs will also understand quickly. This can be achieved by providing instructions (that get automatically updated whenever your API changes) via hypermedia controls in the response. Why write a Web page of documentation to tell devs to construct a URI and use that URI to execute an HTTP GET when you can just include that (and much more) information in your API responses?

Help your client devs out. Throw ‘em a bone, here. Don’t make them read pages of documentation when you can just include simple run-time instructions as they’re needed.

In conclusion: If they have to ask, you didn’t afford it.

(Originally published on my personal blog.)

As you’ll see, all of my predictions are driven by the relentless increase of connected mobile devices. This is the dominating overall trend that will continue to affect all aspects of information systems.

In a nutshell, I predict:

• Individual service deployments on the Web will get smaller and more numerous
• Mobile client deployment will be a bottleneck
• Server mash-ups will increase but client mash-ups will decline
• The demand for seamless switching between personal devices will increase

Services on the Web Get Smaller, More Numerous
Influenced by the existence of the many mobile apps running on a single device, Web-based services will become small, single-focused offerings that (in the words of Doug Mcllroy) “do one thing and do it well.” This will also explode the number of available services. The advantage of this trend will be an increase in the agility and evolvability of service offerings. The challenge will be an increased need for governance at the “micro-service” level.

Mobile Client Deployment Becomes a Bottleneck
As more services appear on the Web and more mobile devices spread throughout the world, keeping up with mobile app deployment will become more difficult and more costly. This is especially true for cases where an app store requires approval before release. To mitigate this problem, developers and architects will look for new ways to update and modify the functionality of already-installed mobile apps without the need for full-on redeployment. Solutions will include use of in-message hypermedia designs, reliance on remote discovery documents and just-in-time plug-in style implementations.

Server-Side Mash-Ups Increase while Client-Side Mash-Ups Decline
The increasing popularity of languages like Node.js, Erlang and Closure will make implementing server-side mash-ups more efficient and easier to maintain than doing the same work within a client application; especially for the mobile platform. This will reduce the “chattiness” of client-side applications and increase the security and flexibility of server-side implementations. The result will be a perceived increase in responsiveness and a reduced use of battery power on mobile apps.

Multiple Device Form Factors Will Demand Seamless Sharing
As more users access content on multiple devices, there will be an increased need to design apps that seamlessly share user data across these devices. This will affect the both client- and server-side implementation details. Identity will need to cross devices easily and content syncing will need to be seamless and automatic. App builders will rely more on the “responsive design” pattern in order to automatically adjust displays and functionality to meet the needs of the current form factor. Servers will need to be “context-aware” and provide the most up-to-date content while users switch from one device to the next.

Finally, whether my predictions are spot on or way off, I look forward to a very interesting and challenging 2013.

Each technology, service or coding framework exhibits its own “style” for solving a problem. Sometimes we select a system component because it’s familiar (“We use SQL databases because that’s what we’ve always used”). Sometimes we include one because it’s unfamiliar (“We’ve never used Node.js before, let’s try it on this project”). And sometimes we select components based on skill set (“Our team doesn’t have any experience with WebSockets, so let’s just use HTTP instead”). It’s important to step back and get a big picture view when selecting components for a production system that will (hopefully) serve your needs for an extended period of time. And that’s where architectural styles come into play.

Architectural styles set the tone for how components in a system interact, govern the implementation details and establish lines of responsibility and maintenance over time. Setting the style early on and communicating it to the team ahead of time goes a long way toward creating a stable and successful implementation. To help clients get a handle on this topic, I commonly identify three widely varying-styles for Web solutions that people can easily recognize: Tunneling, Objects and Hypermedia.

The Tunneling style is best illustrated by SOAP-based implementations where all requests are “tunneled” through a limited set of components (user management, product services etc.) exposed on the Web. The Object style is one that uses the HTTP CRUD pattern (create-read-update-delete) where domain objects (users, products etc.) are exposed and basic read/write operations are supported for those objects. The Hypermedia style relies on a shared understanding through a message format (media type) that defines both the data elements (users, products etc.) and the possible actions (read, write, filter, report etc.) on those data elements. Each of these styles can be used to implement a solution and each of them has associated benefits and challenges.

This comes up so often that we’ve created a short API Academy video introducing the subject of architectural styles for the Web. Take a look and see if it gives you some ideas for how you can answer this question the next time you are about to embark on a major system implementation: “What architectural style is appropriate for this Web solution?”

I also point out that using a hypermedia-based approach to building client and server applications takes a different kind of effort than using RPC-style approaches. And I explain that, currently, there is a limited amount of tooling available to support the process of designing, implementing and maintaining hypermedia-style systems.

If your work involves designing, building, testing and deploying a mobile client application, it is likely you need to deal with an “application store” or some other process where your packaged application must be submitted for review and approval before it is available to users for download. This can happen not only with well-known public offerings such as the Apple Store but also within any organization that provides its own application repository aimed at ensuring the safety and consistency of user-available mobile apps.

In an environment of quick-turnaround, agile-style implementations this “app store” approval can be a real bottleneck. It may be not just days but weeks before your app is tested, approved and posted. This can be especially frustrating when you want to deploy a rapid-fire series of enhancements in response to an engaged user community.

A hypermedia-based client design can often support UI, data transfer and workflow modifications by altering the server messages rather than altering the client source code. By doing this, it is possible to improve both the user experience and the system functionality without the need for re-submitting the client code for “app store” review and re-deployment. This also has the potential to reduce the need for interrupting a user’s day with download and reinstall events and can, in the process, cut down on the bandwidth costs incurred during the repeated roll outs of code modifications to a potentially large user base.

Improved agility, a better user experience and reduced bandwidth costs are all tangible benefits that are possible when investing in a hypermedia-based implementation for your mobile client application.

I’m happy to announce the release of the first API Academy video shorts. I’ve been working with my colleague Ronnie Mitra to create a series of short (five-minute), informative videos on topics related to the Web, APIs and solution design/implementation.

These first few videos are just the start. We plan on doing more of these shorts on a wide range of topics, over the coming weeks and months. And we need your help. Please take a look at these first vids and send us your feedback.

You can comment here, on YouTube or by emailing me directly. We’re looking for feedback on the format, suggested topics and even how we could improve upon this model (hosting a separate site, adding interaction, badges etc.)

Any time you can spend on watching these and sending comments will be most appreciated. Our aim is to do something helpful, engaging and – above all – enjoyable. Thanks for your help and let’s see what this can become!

The API Interaction Model – An Introduction

—

Three Common Web Architecture Styles

—

Handle Errors on the Web

Essentially, REST (representational state transfer) is a style. Specifically, it’s a style of network-based software architecture. This style was first defined in 2000 by Roy Fielding. Fielding stated that “an architectural style is a coordinated set of architectural constraints that has been given a name for ease of reference”.

The set of architectural constraints Fielding defined in his dissertation remain the key criteria by which we judge whether or not a service is RESTful. Back in 2000, Fielding did a very good job of defining the six primary constraints: client-server; stateless; cache; uniform interface; layered system; code-on-demand.

However, REST is also defined by four “interface constraints” that are only partially defined in the dissertation: identification of resources; manipulation of resources through representations; self-descriptive messages; hypermedia as the engine of application state. In particular, the definitions of self-descriptive messages and hypermedia are still debated.

Assuming you can decide on clear definitions of all 10 constraints, all that remains is to identify each of them within the target design. If the implementation does not exhibit all ten (well nine, since code-on-demand is optional), then it is not RESTful. This last step is not difficult. It is the previous step (agreeing on definitions) that causes problems.

Still not sure if your service is RESTful? Well, I originally published this post, in expanded form, on my personal blog. If you want to dig deeper, take a look over there.

This may sound like a typical tech conference but REST Fest is hardly that. Taking its cue from OpenSpaces and similar events, REST Fest is organized by attendees, for attendees. For example, one of the days is devoted to everyone hacking on the same general topic. Another is dedicated to short workshops, all presented by selected registrants.

Similarly, all the general session talks are delivered by the attendees themselves. That’s because one of the “rules” of REST Fest is “everyone talks and everyone listens”. When you sign up to join REST Fest, you are expected to deliver at least a five-minute lightning talk – and there are no exceptions!

Notable presenters will include keynote speaker Stu Charlton (former CTO of Elastra), Matt Bishop (Senior Product Architect at Elastic Path), Pat Cappelaere (currently working on NASA’s SensorWeb project), Leonard Richardson (co-author of O’Reilly’s RESTful Web Services), Sam Ramji (Head of Strategy at Apigee) and yours truly.

I feel privileged to be co-chair of REST Fest and I’m pleased to note that Layer 7 is the event’s Head Sponsor this year. Hope to see you there!

The first thing to keep in mind is that WebSocket is a high-level protocol with its own registered schemes (WS: and WSS:). The specification describes it as: “… intended to be as close to just exposing raw TCP to script as possible.” This is very different from HTTP, which is “…  an application-level protocol for distributed, collaborative, hypermedia information systems.”

That’s good and bad news. It means you have almost the full range of TCP at your disposal. It also means you have none of the established constraints and conventions of the more detailed and focused HTTP specification. This has implications for both design and implementation of WS solutions.

Originally designed with Web browsers in mind, the WS protocol can also be implemented for mobile, desktop, and other stand-alone clients. There are quite a few checks and balances in the specification in order to make it easy (and safe) for browsers to switch from HTTP to WS conversations, all from JavaScript.

However, since many installed browsers do not yet natively support the WS protocol, these checks and balances are not always employed. Instead, WebSockets implementations often take advantage of browser workarounds and fallbacks, in order to support the real-time communications the WS protocol was designed to provide.

It’s also important to remember the specification states: “While this protocol is intended to be used by scripts in web pages, it can also be used directly by hosts [which] can therefore send fake ‘Origin’ header fields, misleading the server.” Implementations that will receive requests from non-browser clients should include additional checks to ensure these requests are valid.

Finally, as the protocol was designed to support real-time communications, it won’t scale in the same way HTTP does. Since the server will keep connections open to all active clients in order to track and broadcast content, servers will need to maintain (or persist) information about each connected client (including knowing when that client is no longer connected!)

If your current HTTP implementations rely on server-based session state, you may not see much difference in the scaling limits of WS. Remember though, the Web’s scaling success is largely based on HTTP’s ability to handle client requests without requiring server-persisted data. Also, some software and implementation patterns designed for HTTP will not work for WS.

Implementing WS is not for the faint-of-heart: it’s not yet widely supported on installed browsers; it uses a different implementation model; it takes more effort/resources to scale it up as services become popular. However, there are some good libraries for coding WS solutions and it can be relatively easy to get started on implementing WebSockets.

But be ready. If you experience great success, you’re likely to have a challenge on your hands!

The WAC’s key failure was that it attempted to standardize the wrong thing: the API. This is a common problem that occurs repeatedly. GigaOm readers may recall another example of industry-level standards going astray, summarized in the “Cloudstack-Openstack Dustup” piece from April. I suspect several readers can call to mind similar cases in the not-too-distant past. Such cases usually share a common theme: disagreement on the details of the API.

The solution is right at hand but few see it. The right way to go is to standardize the way messages are designed and shared, not the data points and actions themselves. In other words, the key to successful shared standardization is through media-types and protocols. This is especially true for any communication over HTTP but it holds true for standards operating over any application-level protocol.

We don’t need to look too far to see an example of an industry-led standardization success. VoiceXML was started by AT&T, IBM, Lucent and Motorola as a way to standardize interactive voice system communications. Not long after the first markup was defined in 1999 (a process which took a matter of a few months), the standard was turned over to the W3C for continued growth and refinement.

The goals of VoiceXML were strikingly similar to those of the WAC and Cloudstack/Openstack efforts: defining an interoperable standard that could be used across an industry group. The difference in the case of VoiceXML was that the committee focused on message design and domain-specific details shared by all players. It did not attempt to document all the data elements, function calls and workflows to be used in lockstep by all.

Most likely, the WAC meltdown won’t be the last one we’ll see. But this is not the inevitable result of competing interests in the global marketplace. This is a result of well-meaning people aiming at the wrong target. We can do better. We can learn from successful interface designs and focus on making it possible to consistently communicate a wide range of information freely instead of attempting to constrain systems to a single set of possible interactions.

The future of an effective Web, a growing and vibrant distributed network, rests in the hands of those who would take on the task of writing the vital standards that will make it work. I look forward to seeing more efforts where the focus is on improving communication between parties through well-designed message formats instead of on limiting communication though constrained APIs.