Persistent Connections With Node.js and Socket.io

Last week, I reported on an important milestone in our music visualizer project: Successful creation (from static files) of an HD video (with audio) using Node.js and FFMpeg. Our ultimate goal: To generate thirty client-rendered WebGL frames per second, push them across a socket to a Node.js microservice, create a video, and upload it to YouTube.

Sounds crazy? You bet. But it’s a mad, mad world, and it just might happen.

When I last left off, in addition to our browser client (built with Three.js, React, and PureMVC), we had a server-side proof of concept running on Node.js, FFMpeg, and PureMVC. This POC simply started up, created an FFMpeg command, configured it with the locations of our images and audio, then spit out a video that could be uploaded manually to YouTube with no issues.

Time to Add Sockets Into the Mix

Let’s just get this out of the way right now: Websockets are great, but there are soooo many libraries competing for your attention. The research becomes a real time-sink. While we now have Websocket support in all major browsers (except Opera Mini), unless you want to deal with low-level details such as handling reconnects, you probably just want to pick a library. And, please — do everyone a favor: Don’t get disgusted, decide to write your own, and throw it into the mix. If you feel those urges, I implore you to fork someone else’s, make it better, and send a pull request.

My research led me to choose Socket.io, even though:

• Its documentation is horrible.
• Its two on-site demos don’t work.
• Its Slack channel link is broken.
• It has features that lie outside the core business of socket management (like ‘rooms’), which add to the ongoing list of bugs and feature requests, making it difficult to determine if all the functionality you need is actually working. Nail down your scope, dudes, please.

Despite all this, the above-linked research and sheer volume of downloads led me to believe it might be the right choice for our project. At least to start out. Its implementation details are similar enough to all the other options that even if it did need to be replaced later, the refactoring wouldn’t be so bad.

Step 1: Add Socket.io to the Node Project

npm install socket.io --save

Step 2: Create a Socket Server

// Create the socket server
const PORT = 3000;
var socket = require('socket.io')(PORT);

Step 3: Connect From the Command Line

Once the above code was folded it into my POC, I had a running socket server and wanted to chat with it. Telnet connected, but didn’t get me anywhere because although I could connect, the handshake protocol is header-based. Anything I tried to send immediately closed the socket.

So I found and installed wscat, which should have allowed me to connect and send a message with:

wscat -c ws://localhost:3000 -m Allo?

But no dice.

I timeboxed my attempt to connect through the command line, leaving off the message, trying different options, researching the protocol, etc., eventually giving up and moving on to the more complicated but necessary step of creating a test HTML client.

However, I circled back around to the issue while writing this article, because I couldn’t accept that there was no easy way to connect to a Socket.io server via the command line. It turns out that the reason wscat didn’t work, is that Socket.io is not speaking the Websocket protocol. This would be a nice entry for the Socket.io docs or FAQ page, but alas, it isn’t mentioned anywhere on the site as far as I could determine.

The answer is iocat, which talks to either Websockets or Socket.io. Install it like this:

npm install iocat -g

And talk to it like this:

$ iocat --socketio ws://localhost:3000
> Allo?

That’s fine, but of course, I’d already moved on to creating an HTML test page. I just wanted to make you aware of this command line client. 

Also worth mentioning here is Monitor.io, which provides a nice telnet interface for monitoring and debugging Socket.io applications. This will come in handy while testing multiple, simultaneous connections, particularly with the massive amount of data we plan to push across each.

Step 4: Send a Message From the Browser

There is a handy-dandy resource at websocket.org that lets you connect to a socket server using the browser’s built-in WebSocket support (with or without the secure WebSocket layer), test message echo, and disconnect. Also on that page is an HTML snippet you can use to build your own test.

Starting with that code as a base, I refactored to use Socket.io instead of WebSocket:

<!-- test-io-client.html -->
<script src="https://cdn.socket.io/socket.io-1.4.5.js"></script>
<script>

    function init()
    {
        output = document.getElementById("output");
        testSocket();
    }

    function testSocket()
    {
        var socket = io.connect('http://localhost:3000/');
        socket.on('test', onMessage );
        socket.on('connect', onConnect );
        socket.on('disconnect', onDisconnect );
        socket.on('connect_error', onError );
        socket.on('reconnect_error', onError );

        function onConnect(evt)
        {
            writeToScreen("CONNECTED");
            doSend("Allo?");
        }

        function onDisconnect(evt)
        {
            writeToScreen("DISCONNECTED");
        }

        function onMessage(data)
        {
            writeToScreen('<span >RESPONSE: ' + data+'</span>');
            socket.close();
        }

        function onError(message)
        {
            writeToScreen('<span >ERROR:</span> ' + message);
        }

        function doSend(message)
        {
            writeToScreen("SENT: " + message);
            socket.emit('test', message);
        }

        function writeToScreen(message)
        {
            var pre = document.createElement("p");
            pre.style.wordWrap = "break-word";
            pre.innerHTML = message;
            output.appendChild(pre);
        }
    }

    window.addEventListener("load", init, false);
</script>

<h2>Socket.io Test</h2>

<div id="output"></div>

The above client-side code will connect to the socket server and emit a ‘test’ event with a message. It will also listen for a ‘test’ event from the server, display its contents, then disconnect. And it shows connect and disconnect messages as well as several possible errors.

Step 5: Handle Inbound Connections on Server

In the server project, I added a quick standalone counterpart to the test client, which will listen for ‘test’ events, log them to the console, and respond to the client by emitting its own ‘test’ event with a welcome message that includes the client’s connection id.

// test-io-server.js

// Create the socket server
const PORT = 3000;
var socket = require('socket.io')(PORT);
socket.on('connection', function(client) {

    // Listen for test and disconnect events
    client.on('test', onTest);
    client.on('disconnect', onDisconnect);

    // Handle a test event from the client
    function onTest(data) {
        console.log('Received: "' + data + '" from client: ' + client.id);
        client.emit('test', "Cheers, " + client.id);
    }

    // Handle a disconnection from the client
    function onDisconnect() {
        console.log('Received Disconnect from client: ' + client.id);
        client.removeListener('test', onTest);
        client.removeListener('disconnect', onDisconnect);
    }
});

Now, from Webstorm (or some inferior IDE):

• Run the test-io-server.js file to launch it in Node, and begin listening on port 3000.
• Run the test-io-client.html file to launch a server and serve the client to the browser.

That was easy enough. You can grab the the code from this Gist on GitHub.

Next Steps

Now I need to:

1. Push images across the socket (possible as of Socket.io version 1.0)

2. Store the images in a temporary folder

3. Kick off a video build (which I already figured out a couple of articles back)

4. Clean up the sources

At this point, it's reasonable to build these aspects into the microservice as opposed to creating another throwaway test rig. I'll cover that in the next article, and go into the architecture a bit more.