WebSockets as a Real-Time Data in Motion Flow
By using codeless microservices in a server-less environment, you can integrate between front-end web development and back-end in Big Data.
Join the DZone community and get the full member experience.Join For Free
Microservices, Big Data, front-end technology, serverless environments, and WebSockets have come together in an interesting way. Apache NiFi 1.1 has added WebSockets as yet another interface, so you can now use your data ingestion server to host a number of zero code WebSocket backends. In the vein of Amazon Lambdas, you can also host endpoints that are event driven by MQTT, JMS, HTTP, TCP, UDP, and more.
One interesting application would be a web page application that was fed data right from Apache Hive or Apache Phoenix and stored user data in Hadoop. You can also trigger long running jobs via Livy, JMS, Kafka, and other asynchronous methods. I can also direct communication from Twitter to a WebPortal via Websockets and never permanently store the data.
I can add as many forks to the data and store it in a dozen different data stores while two-way communicating with a WebSockets, REST, TCP, UDP, Slack, email, and other channels. Adding machine learning and deep learning to these rich pipelines is very easy, making for a very interesting data plane for an entire enterprise. Even better, it's all Apache open-source, well-supported, constantly improving and integrating all the technologies you are interested in.
You can run it in your choice of IAAS, in-house or even tiny hardware like Raspberry Pis. You can cluster any number of these servers as you need to scale out. You can enable some interesting ChatOps flows right into Big Data.
Here's my quick example to show you how easy this all is.
This simple WebSockets Server and Client does the "Hello, world" of web sockets. Whatever the client sends, we send it back.
My suggested use cases for WebSockets are:
- WebSocket client to Slack interface.
- WebSocket client to email.
- WebSocket chat stored to Apache Phoenix.
- WebSocket to communicate from mobile web apps.
- WebSocket to send detailed log details from enterprise web applications directly to log ingest platform, bypassing local filesystem.
Step 1: HandleHTTPRequest
This accepts the HTTP calls from browsers.
Step 2: ExecuteStreamCommand
This returns an HTML page. We could do getfile or any number of other ways of getting the HTML as a flowfile.
Step 3: HandleHttpResponse
This serves up our web page to browsers. A StandardHTTPContextMap is required to store HTTP requests and responses to share them through the stream.
Step 4: PutFile
This is used just to keep logs of what's going on. You see all the flow files to the local file system.
Step 5: ListenWebSocket
This is the actual WebSocket Server listener; it is what our client will talk to.
Step 6: PutWebSocket
This is the reply to the web socket client.
WebSocket conversation on the client side:
Opinions expressed by DZone contributors are their own.