Over the past few years, many people – I amongst them – have spent countless hours convincing you that RESTful interfaces can (and have) changed the face of application integration. But it’s 2017, and we in the API integration space have held onto some practices for too long.
One such area we need to address is the perilous concept of polling. Across many REST interfaces, it’s common to see clients poll continuously as they check for changes in data they are interested in. This creates significant load on the API infrastructure, or the backend service directly and often means the integration tier has been sized to support entirely redundant traffic – which is unnecessarily costly. It’s time for an immediate solution.
Polling is the same as the refresh button. It’s not a viable solution.
Across much of today’s application integration, there is no ability for systems to react to events in real-time. Typically, system A (the Client) will ask system B (the Server) if new data is available based on a set of query parameters – a well-understood request/response pattern native to the Web. But application integration patterns must be selected based on merit versus conceptual elegance. The problem is that polling is essentially just hitting a refresh button – and depending on a “refresh button” for vital functionality and data updates is unacceptable.
Don’t Ask– Tell
An event represents something that happened within a system, as part of user interaction or even an action caused by existing request/response interfaces. Each of these events will typically involve new or changing data that described what just happened. For event-driven integration there is no need to ask a given system if an event happened – instead, the system will tell any interested system or application about it. The event and associated data will be pushed to other systems.
Enter WebHooks. An event-driven pattern is the foundation of webhooks – removing the need for clients to ask repeatedly if a given system has new data available.
Webhooks work by automatically posting new event data to a user-defined URL which is monitored by the user’s linked applications. Each time a new event is posted to the URL, the linked applications update to include the new data. Unlike polling which delivers usable data on less than two percent of requests, webhooks only update when new information is available. Because of the increased efficiency, over 82 percent of developers surveyed by Wufoo indicated that they would rather receive new data via webhooks instead of with polling. Despite this preference, from our research only 29 percent of APIs currently support webhooks.
According to Adam Duvander, webhooks equate to a better user experience: “Webhooks, particularly subscription webhooks, are a rare triple win. They benefit everybody. The developer doesn’t have to poll 50+ times to get a single result. The provider’s server gets a breather. And the end user, often forgotten in these discussions, still ends up with near-instant results.”
What Are the Alternatives?
First off, for many, the term Server-Sent Events (SSEs) may be unfamiliar. For the past few years, the specification has been in a state of flux and has definitely been overshadowed by the more fully featured communication protocol - WebSockets. However, the idea behind SSEs should be familiar: allowing a Web Application to "subscribe" to data updates originated (sent) by a server, and subsequently be notified as these data events occur. The problem here is that, while it removes the polling technically, it also removes much of the control from the user, and forces them into a passive state.
WebSockets provide a richer protocol to perform bi-directional, full-duplex communication. Having a two-way channel is more attractive for things like games, messaging apps, interactive experiences, and for cases where you need real-time updates in both directions. But here again, the solution isn’t perfect since the integration pattern effectively breaks the semantics of HTTP.
There is no single design decision that perfectly answers every integration question, but the perils of polling are clear – contributing to network overhead, cost, and poor user experience. Our recommendation is to consider alternatives like WebHooks when you’re building your next Web API or integration.