Building a Simple Kafka Client for the Web and Desktop

Kafka is an open-source platform that is used for event distribution, logging, and streaming. Distributions from Confluent and other vendors provide a web UI for broker management and client tasks, while those from Apache Kafka provide just command-line tools. Also, for most of the requirements, a simple client that publishes and consumes events or messages from the topic is sufficient. The main objectives of building such a client were:

• Follow a web development approach 
• Have a simple and elegant UI 
• Build a native desktop app

Being a sort of a de facto standard for most web development with a huge selection of frameworks and libraries, node.js was chosen as the platform. The UI framework was Bootstrap with a Material theme. Though some might argue that it is dated and has lesser capabilities, it fit the bill for the requirement in question. ElectronJS was used to provide the native desktop app capabilities. It has a flat learning curve with usually only a main file that it generates with functionality to manage the window being displayed. The Kafka client-specific code can be in separate files and can be included in this file. Using utilities such as electron packager, the executable code for the respective platform can be generated.

/**

 * This file creates and manages the Electron JS window

 */

const { app, BrowserWindow } = require('electron')

function createWindow () {

  // Create the browser window.

  const win = new BrowserWindow({

    width: 1250,

    height: 600,

    icon: __dirname + '/images/icon.ico',

    autoHideMenuBar: true,

    resizable: true,

    fullscreenable:false, 

    webPreferences: {

      nodeIntegration: true,

      devTools : false

    }

  })

  //no menu 

  win.removeMenu();

  // and load the index.html of the app.

  win.loadFile('index.html')

}

// This method will be called when Electron has finished

// initialization and is ready to create browser windows.

// Some APIs can only be used after this event occurs.

app.whenReady().then(createWindow)

// Quit when all windows are closed, except on macOS. There, it's common

// for applications and their menu bar to stay active until the user quits

// explicitly with Cmd + Q.

app.on('window-all-closed', () => {

  if (process.platform !== 'darwin') {

    app.quit()

  }

})

app.on('activate', () => {

  // On macOS it's common to re-create a window in the app when the

  // dock icon is clicked and there are no other windows open.

  if (BrowserWindow.getAllWindows().length === 0) {

    createWindow()

  }

})

// call the kokpitsever class to handle the Kafka client functionality

const kokpitserver = require("./kokpitserver");

For the Kafka client functionality, KafkaJS was used as the library as it is lightweight and provides a simple interface. The features built using KafkaJS were:

• Connecting automatically to a Kafka broker
• Creating a topic on the fly
• Publishing and consuming messages 

To provide a real-time view of the messages in the Kafka topic, Socket.io is used. It enables real-time communication using WebSockets from the Kafka topic to the UI. The consumer sees the messages on the topic it has subscribed to in the UI in real-time as they get published. The code to implement comprises adding a line of code to emit the event at the server and to consume it in the UI and display it.

xxxxxxxxxx

//server side

await consumer.run({

    eachMessage: async ({ topic, partition, message }) => {

        io.sockets.emit('message', message.value.toString())

    },

});

x

//script in HTML

var socket = io('http://localhost:8182');

    //on message 

    socket.on('message', function (msg) {

        alert("Message: " + msg);

    });

Adding new features to the Kafka client or making it available on more platforms is not as difficult as the underlying libraries and frameworks are robust and well adopted. This approach for building a real-time web client be used for messaging platforms such as RabbitMQ, Solace, and others. 

The code for the client is available here. The latest release is available here.