WebSockets as a Real-Time Data in Motion Flow

DZone 's Guide to

WebSockets as a Real-Time Data in Motion Flow

By using codeless microservices in a server-less environment, you can integrate between front-end web development and back-end in Big Data.

· Integration Zone ·
Free Resource

Microservices, Big Data, front-end technology, serverless environments, and WebSockets have come together in an interesting way. Apache NiFi 1.1 has added WebSockets as yet another interface, so you can now use your data ingestion server to host a number of zero code WebSocket backends. In the vein of Amazon Lambdas, you can also host endpoints that are event driven by MQTT, JMS, HTTP, TCP, UDP, and more.

For this particular flow, I have a web server serving up a static web page that has a JavaScript WebSockets client on it. Again, there's zero coding involved to host an active Single Page Application (SPA) that can call my WebSockets microservices hosted via Apache NiFi. From there, I can take the message in and transform, process, reply, send, store, and do any number of actions with it using zero code.  

One interesting application would be a web page application that was fed data right from Apache Hive or Apache Phoenix and stored user data in Hadoop. You can also trigger long running jobs via Livy, JMS, Kafka, and other asynchronous methods. I can also direct communication from Twitter to a WebPortal via Websockets and never permanently store the data.

I can add as many forks to the data and store it in a dozen different data stores while two-way communicating with a WebSockets, REST, TCP, UDP, Slack, email, and other channels. Adding machine learning and deep learning to these rich pipelines is very easy, making for a very interesting data plane for an entire enterprise. Even better, it's all Apache open-source, well-supported, constantly improving and integrating all the technologies you are interested in.

You can run it in your choice of IAAS, in-house or even tiny hardware like Raspberry Pis. You can cluster any number of these servers as you need to scale out. You can enable some interesting ChatOps flows right into Big Data.

Here's my quick example to show you how easy this all is.

This simple WebSockets Server and Client does the "Hello, world" of web sockets. Whatever the client sends, we send it back.

My suggested use cases for WebSockets are:

  • WebSocket client to Slack interface.
  • WebSocket client to email.
  • WebSocket chat stored to Apache Phoenix.
  • WebSocket to communicate from mobile web apps.
  • WebSocket to send detailed log details from enterprise web applications directly to log ingest platform, bypassing local filesystem.

Step 1: HandleHTTPRequest

This accepts the HTTP calls from browsers.

Step 2: ExecuteStreamCommand

This returns an HTML page. We could do getfile or any number of other ways of getting the HTML as a flowfile. 

Step 3: HandleHttpResponse

This serves up our web page to browsers. A StandardHTTPContextMap is required to store HTTP requests and responses to share them through the stream.

Step 4: PutFile

This is used just to keep logs of what's going on. You see all the flow files to the local file system.

Step 5: ListenWebSocket

This is the actual WebSocket Server listener; it is what our client will talk to.

Step 6: PutWebSocket

This is the reply to the web socket client.

Image title

WebSocket conversation on the client side:

cat /Volumes/Transcend/Apps/nifi-    

      <script type="text/javascript">
         function WebSocketTest()
            if ("WebSocket" in window)
               alert("WebSocket is supported by your Browser!");
               // Let us open a web socket
               var ws = new WebSocket("ws://localhost:9998/echo");
                  ws.send("MSG:  NIFI IS AWESOME");
               ws.onopen = function()
                  // Web Socket is connected, send data using send()
                  ws.send("Message to send");
                  alert("Message is sent...");
               ws.onmessage = function (evt)
                  var received_msg = evt.data;
                  alert("Message is received...");
               ws.onclose = function()
                  // websocket is closed.
                  alert("Connection is closed...");
               // The browser doesn't support WebSocket
               alert("WebSocket NOT supported by your Browser!");
      <div id="sse">
         <a href="javascript:WebSocketTest()">Run WebSocket</a>


integration, microservices, tutorial, web development, websockets

Opinions expressed by DZone contributors are their own.

{{ parent.title || parent.header.title}}

{{ parent.tldr }}

{{ parent.urlSource.name }}