New design patterns have enhanced UX by bringing animated UI components. Material Design is one good example: when a user clicks on a button, he/she can see a slight animation which indicates that the action has been performed. Ditto with other UI components like inputs. That’s nice: the user has a visual feedback when he/she interacts with a UI: something actually happens. But can we do better? What if instead of animating UI, we also animate the data?
What Do You Mean by Animating Data?
Animating data is the idea of exposing to the end-user the change of data, provided the data is changing quite a lot.
Why Should I Care About Animating Data?
Well, to borrow a well-known motto: “animated data will eat the world.” Indeed, our world seems to be consuming more and more real-time data; news, multi-player games, social networks, e-commerce, sharing economy, collaborative apps, dashboards, and—last but not least—IoT.
While real-time data was once reserved to FinTech apps ten years ago, it’s not the case anymore. Just think about Twitter. Looking at tweets in real-time is just expected. So, to paraphrase a colleague of mine: “The more others invest in amazing UI, the more yours seems lousy.”
OK. So, What Do I Need to Animate Data?
One might think that frequent polling could do the job. But frankly, that’s a bad idea.
What’s wrong with polling?
Polling Is Inefficient.
Poll too fast, and you will get repeated data, useless calls between the client and the server that can overload the server depending on the polling frequency and the number of clients. Poll too slow, and you may lose data. You run behind. Poll at the wrong moment, and you may be blocked because the server is busy. What you really need is a technology that can push data frequently.
Fortunately, solutions have existed for quite a long time now.WebSockets and Server-Sent Events (SSE) are some well-placed candidates. MQTT is also a good one in the IoT world.
Let’s start with WebSockets.
WebSockets is a low-level protocol that enables a full duplex TCP connection. This means you can communicate in both ways: from the client to the server and from the server to the client. The initialization of the connection starts with an HTTP request. This request performs an upgrade handshake, which enables the client and the server to switch to a lower protocol level.
Once you’ve got your socket, you’re free to send whatever you want in it. In both directions. In binary or text.
GET /chat HTTP/1.1 Host: example.com:8000 Upgrade: Websocket Connection: Upgrade Sec-WebSocket-Key: dGhlIHNhbXBsZSBub25jZQ== Sec-WebSocket-Protocol: chat, superchat Sec-WebSocket-Version: 13
As you are free to send anything, you usually come to implement some kind of messaging protocol between the client and the server. The WebSocket protocol offers a subprotocol feature to ensure the client and the server speak the same language. When the WebSocket is initiated, a subprotocol can be sent in the headers during the handshake phase. If the server can speak the sub-protocol, then the connection can be established. If not, the client has to close the connection.
Some sub-protocol already exist in the WebSockets land. WAMP (Web Application Messaging Protocol) is one of them. This sub-protocol offers both RPC (Remote Procedure Call) and Publish-Subscribe. RPC offers a way to trigger actions to the server while Publish-Subscribe more naturally fits with notifications to the client.
Server-Sent Events (a.k.a. SSE) is also a push technology. Unfortunately, SSE is less well known than WebSockets. Like WebSockets, it has a W3C specification and was born a little bit earlier (2006 vs 2008). Server-Sent Events is half-duplex: data can be sent from the server to the client. This is, honestly, the only requirement to animate data. With SSE, only text can be sent.
Unlike WebSockets, SSE relies on HTTP/1.1, which, from an OPs point of view, is a bit simpler. Indeed, load-balancers and proxies were made to support HTTP. So de facto, SSE should be transparent for them. All you need is a server that supportsSSE and an HTTP connection. To connect to an SSE server, you need to initiate an HTTP connection (with a GET) by sending an
GET /stream HTTP/1.1 Host: example.com Accept: text/event-stream
accept header with text/event-stream.
But as for WebSockets, you may not really care since the implementations of SSE usually hide all this plumbing stuff. For iOS, you can use TRVSEventSource. And in the Android world, several Java libraries do the job: here and there.
Further, SSE comes with a nice feature: its reconnection ability. In case of a loss connection (wifi loss, etc.), the specification allows the client to reconnect to the server and get the missed events. This can be done if the server sends events structured as follows:
id: 37 event: news data: New DZone article is out! retry: 3000
The ID of the event received can be registered in the client side. If the client loses the signal, it can reconnect to the server with the ID of the last event received (via a Last-Event-ID HTTP header). The server can then use this ID to send the missed events. This Last-Event-ID mechanism is implemented in the browsers. Depending on the library you use, it may do the same for Native clients.
In the event sent, the server can also send a retry property, which is a delay for a reconnection. In this case, the client loses the connection, when the client should try to reconnect. retry tells the client after how long it can try a reconnection. This saves useless reconnection attempts. Once again, you may find the implementation of this feature in the client library.
You can also define an event property to define the type of the event. Defining a type for an event allows the client to filter the events received according to their type. This eases the way to handle several events in one SSE connection.
MQTT is one of the lightweight message protocol used in IoT. It’s a publish–subscribe protocol over a TCP connection. An MQTT broker is used to dispatch the messages sent by a publisher to subscribers. MQTT offers three levels of message delivery (also called QoS for Quality of Service): 0, 1, and 2. Level0 is a fire-and-forget mode, which delivers only once. There is no acknowledgement from the subscriber. Level 1 guarantees the message is delivered at least once. But this means the message can be delivered more than once too. Level 2 makes sure the message is delivered exactly once.
In case of a loss of connection and a reconnection attempt, a subscriber can get the missed messages (if the QoS is Level 1 or 2) and is not forced to re-subscribe to all the topics it has subscribed. This MQTT mode is called Persistent Session. It’s not the default mode, but it is easily configurable.
When a client subscribes to a topic, it may also wait for a while before the publisher publishes a new message. In some cases, it’s not convenient for the client: it may be desirable to get the last message in order to display some data on the UI. MQTT supports this kind of feature: it’s called Retained Message. This is configurable by a flag sent with the message. To be more accurate, it’s not the last message published but the last message sent with the retained flag.
Last Will and Testament is also an interesting MQTT feature that enables the detection of a disconnection of a client and implements some strategies like notifying the other subscribers that the client has been disconnected.
Which to Use
Depending on your application, you may ask whether you should use WebSockets, Server-Sent Events, or MQTT.
WebSockets is a bidirectional protocol while the two others aren’t. In the case of WebSockets vs. Server-Sent Events, I would wonder first whether the client needs to communicate as often as the server. If communication is mostly from server to client, you might consider using a REST call for transactional calls, along with SSE for server-client data. If the server API does not offer SSE, you can consume it via an SSE proxy (editor’s comment: this is what streamdata.io offers). This may be a quick and scalable win compared to re-implementing a message protocol in WebSockets.
MQTT can also be a nice technology to be considered. But it has impact on the infrastructure: a TCP MQTT broker needs to be installed and configured.