DZone
Thanks for visiting DZone today,
Edit Profile
  • Manage Email Subscriptions
  • How to Post to DZone
  • Article Submission Guidelines
Sign Out View Profile
  • Post an Article
  • Manage My Drafts
Over 2 million developers have joined DZone.
Log In / Join
Refcards Trend Reports
Events Video Library
Refcards
Trend Reports

Events

View Events Video Library

Zones

Culture and Methodologies Agile Career Development Methodologies Team Management
Data Engineering AI/ML Big Data Data Databases IoT
Software Design and Architecture Cloud Architecture Containers Integration Microservices Performance Security
Coding Frameworks Java JavaScript Languages Tools
Testing, Deployment, and Maintenance Deployment DevOps and CI/CD Maintenance Monitoring and Observability Testing, Tools, and Frameworks
Culture and Methodologies
Agile Career Development Methodologies Team Management
Data Engineering
AI/ML Big Data Data Databases IoT
Software Design and Architecture
Cloud Architecture Containers Integration Microservices Performance Security
Coding
Frameworks Java JavaScript Languages Tools
Testing, Deployment, and Maintenance
Deployment DevOps and CI/CD Maintenance Monitoring and Observability Testing, Tools, and Frameworks

How does AI transform chaos engineering from an experiment into a critical capability? Learn how to effectively operationalize the chaos.

Data quality isn't just a technical issue: It impacts an organization's compliance, operational efficiency, and customer satisfaction.

Are you a front-end or full-stack developer frustrated by front-end distractions? Learn to move forward with tooling and clear boundaries.

Developer Experience: Demand to support engineering teams has risen, and there is a shift from traditional DevOps to workflow improvements.

Related

  • What Are Protocol Buffers?
  • 6 of the Best API Testing Tools in the Market
  • Open Source Tools Options in Web Application Development
  • How to Build a Coronavirus Dashboard in Java

Trending

  • From ETL to ELT to Real-Time: Modern Data Engineering with Databricks Lakehouse
  • Secure Your Oracle Database Passwords in AWS RDS With a Password Verification Function
  • MLOps: Practical Lessons from Bridging the Gap Between ML Development and Production
  • OTel Me Why: The Case for OpenTelemetry Beyond the Shine
  1. DZone
  2. Software Design and Architecture
  3. Integration
  4. Lightweight RPC with ZeroMQ (ØMQ) and Protocol Buffers

Lightweight RPC with ZeroMQ (ØMQ) and Protocol Buffers

By 
John D'Emic user avatar
John D'Emic
·
Nov. 26, 12 · Interview
Likes (0)
Comment
Save
Tweet
Share
28.2K Views

Join the DZone community and get the full member experience.

Join For Free

A frequent issue I come across writing integration applications with Mule is deciding how to communicate back and forth between my front end application, typically a web or mobile application, and a flow hosted on Mule.

I could use web services and do something like annotate a component with JAX-RS and expose this out over HTTP.  This is potentially overkill,  particularly if I only want to host a few methods, the methods are asynchronous or I don’t want to deal with the overhead of HTTP.  It also could be a lot of extra effort if the only consumers of the API, at least initially, are internal facing applications.

Another choice is to use “synchronous” JMS with temporary reply queues.  While Mule makes this easy to do, particularly with MuleClient, I now have to deal with the overhead of spinning up a JMS infrastructure.   I could also be limited to Java only clients, depending on which JMS broker I choose.  The latter is particularly signifcant, as Java probably isn’t the technology of choice on the web or mobile layer.

ØMQ for RPC

ØMQ, or ZeroMQ, is a networking library designed from the ground up to ease integration between distributed applications.  In addition to supporting a variety of messaging patterns, which are enumerated in the extremely well written guide,  the library is written in platform agnostic C with wrappers for different languages like Java, Python and Ruby.

These features make it a good candidate to solve the challenges I introduced above, particularly since a community contributed module for ØMQ was released recently.  Let’s consider a  simple service that accepts a request for a range of stock quotes and returns the results and see how we can host this service with Mule and expose it out with the ØMQ Module.

Data Serialization with Protocol Buffers

Data is transported back and forth over ØMQ as byte arrays.  We, as such, need to decide on a way to serialize our stock quote request and responses “on the wire.”  Before we do that, however, let’s take a look at the Java canonical data model we’re using on the client and server side.  The following Gists show the important bits of the StockQuote and StockQuoteResponse classes.

public class StockQuote implements Serializable {

    String symbol;

    Date date;

    Double open;

    Double high;

    Double low;

    Double close;

    Long volume;

    Double adjustedClose;


public class StockQuoteRequest implements Serializable {

    String symbol;

    Date startDate;

    Date endDate;


public interface StockDataService {

    public List<StockQuote> getQuote(StockQuoteRequest request);

}

We could use Java serialization to get the objects into byte arrays.  Ignoring the other deficiencies of default Java serialization, the main drawback is that it limits our clients to one’s running on a JVM.  XML or JSON provide better alternatives, but for the purposes of this example we’ll assume we want a more compact representation of the data (this isn’t totally unrealistic, stock quote data can be extremely time sensitive and we probably want to minimize serialization and deserialization overhead.)

Protocol Buffers provide a good middle ground and also boast a Mule Module to provide the necessary transformers we need to move back and forth from the byte array representations.  Let’s define two .proto files to define the wire format and generate the intermediary stubs for serialization.

package com.acmesoft.zeromq;

option java_package = "com.acmesoft.stock.model.serialization.protobuf";
option optimize_for = SPEED;package com.acmesoft.zeromq;

option java_package = "com.acmesoft.stock.model.serialization.protobuf";
option optimize_for = SPEED;

option java_multiple_files = true;

message StockQuoteResponseBuffer {
   repeated StockQuoteBuffer result = 1;
}

message StockQuoteBuffer {
  required string symbol = 1;
  required int64 date = 2;
  required double open = 3;
  required double high = 4;
  required double low = 5;
  required double close = 6;
  required int64 volume = 7;
  required double adjustedClose = 8;
}

option java_multiple_files = true;

message StockQuoteRequestBuffer {
  required string symbol = 1;
  required int64 start = 2;
  required int64 end = 3;
}

You typically would use the “protoc” compiler to generate the Java stubs.  This is tedious, however, so we’ll instead modify the pom.xml of our project to compile the protoc files during the compile goals:

<?xml version="1.0" encoding="UTF-8"?>
<plugin>
   <groupId>com.google.protobuf.tools</groupId>
   <artifactId>maven-protoc-plugin</artifactId>
   <configuration>
      <protocExecutable>/usr/local/bin/protoc</protocExecutable>
   </configuration>
   <executions>
      <execution>
         <goals>
            <goal>compile</goal>
            <goal>testCompile</goal>
         </goals>
      </execution>
   </executions>
</plugin>

Since we already have a domain model we’ll add some helper classes to simplify the serialization tasks on the client side.

   public byte[] toProtocolBufferAsBytes() {
        return StockQuoteRequestBuffer.newBuilder()
                .setSymbol(symbol)
                .setStart(startDate.getTime())
                .setEnd(endDate.getTime()).build().toByteArray();
    }

  public static StockQuoteRequest fromProtocolBuffer(StockQuoteRequestBuffer buffer) {

        StockQuoteRequest request = new StockQuoteRequest();
        request.setSymbol(buffer.getSymbol());
        request.setStartDate(new Date(buffer.getStart()));
        request.setEndDate(new Date(buffer.getEnd()));

        return request;
    }


  public static StockQuoteResponseBuffer toProtocolBuffer(List<StockQuote> quotes) {
        StockQuoteResponseBuffer.Builder responseBuilder = StockQuoteResponseBuffer.newBuilder();

        for (StockQuote quote : quotes) {
            responseBuilder.addResult(StockQuoteBuffer.newBuilder()
                    .setAdjustedClose(quote.getAdjustedClose())
                    .setClose(quote.getClose())
                    .setDate(quote.getDate().getTime())
                    .setHigh(quote.getHigh())
                    .setLow(quote.getLow())
                    .setOpen(quote.getOpen())
                    .setSymbol(quote.getSymbol())
                    .setVolume(quote.getVolume()).build());
        }
        return responseBuilder.build();
    }

    public static List<StockQuote> listOfStockQuotesFromBytes(byte[] bytes) {
        List<StockQuoteBuffer> buffer;
        try {
            buffer = StockQuoteResponseBuffer.parseFrom(bytes).getResultList();
        } catch (InvalidProtocolBufferException e) {
            throw new SerializationException(e);
        }

        List<StockQuote> quotes = new ArrayList<StockQuote>();

        for (StockQuoteBuffer stockQuoteBuffer : buffer) {
            StockQuote stockQuote = new StockQuote();
            stockQuote.setClose(stockQuoteBuffer.getClose());
            stockQuote.setDate(new Date(stockQuoteBuffer.getDate()));
            stockQuote.setHigh(stockQuoteBuffer.getHigh());
            stockQuote.setOpen(stockQuoteBuffer.getOpen());
            stockQuote.setSymbol(stockQuoteBuffer.getSymbol());
            stockQuote.setVolume(stockQuoteBuffer.getVolume());
            stockQuote.setAdjustedClose(stockQuoteBuffer.getAdjustedClose());
            stockQuote.setLow(stockQuoteBuffer.getLow());
            quotes.add(stockQuote);
        }

        return quotes;
    }

Configuring StockDataService

Now that we have a canonical data model and a wire format defined we’re ready to wire up a Mule flow to expose the service out.  Note that for this to work you need to have jzmq installed locally on your system.  The following dependency needs to be added to your pom.xml once its installed:

<dependency>
   <groupId>org.zeromq</groupId>
   <artifactId>zmq</artifactId>
   <version>2.2.0</version>
   <systemPath>/usr/local/lib/zmq.jar</systemPath>
   <scope>system</scope>
</dependency>

Where systemPath is the location of the zmq.jar on your filesystem.

Once that’s out of the way we can configure the flow, as illustrated below:

<flow name="main">
        <zeromq:inbound-endpoint address="tcp://*:9090" socket-operation="bind"
                                 exchange-pattern="request-response"/>
        <protobuf:deserialize
                protobufClass="com.acmesoft.stock.model.serialization.protobuf.StockQuoteRequestBuffer"/>
        <expression-transformer
                expression="com.acmesoft.stock.model.StockQuoteRequest.fromProtocolBuffer(payload)"/>
        <component class="com.acmesoft.stock.service.StockDataServiceImpl"/>
        <expression-transformer
                expression="return com.acmesoft.stock.model.StockQuote.toProtocolBuffer(payload)"/>
    </flow>

The ZeroMQ inbound-endpoint will be bound to TCP port 9090 with a request-response exchange pattern.  The deserialize MP in the protobuf module will deserialize the byte array to the generated StockQuoteRequestBuffer class.  From there we’ll use MEL to invoke the helper method on StockQuoteRequest to transform the intermediary class to the domain model.

The List of StockQuotes returned from StockDataService will  be transformed by the MEL expression using the “toProtocolBuffer” helper method on the domain model.  The Protocol Buffer Module is then smart enough to implicitly transform the intermediary object to a byte array for the response.

Consuming the Service from the Client Side

Now that the server is ready we can turn our attention to the client side code to invoke the remote service.  Let’s take a look at how this works:

StockQuoteRequest stockQuoteRequest = new StockQuoteRequest();
stockQuoteRequest.setSymbol("FB");
stockQuoteRequest.setStartDate(new Date( new Date().getTime() - (86400000 * 7)));
stockQuoteRequest.setEndDate(new Date());

ZMQ.Socket zmqSocket = zmqContext.socket(ZMQ.REQ);
zmqSocket.setReceiveTimeOut(RECEIVE_TIMEOUT);
zmqSocket.connect("tcp://localhost:9090");
zmqSocket.send(stockQuoteRequest.toProtocolBufferAsBytes(), 0);

List<StockQuote> quotes = StockQuote.listOfStockQuotesFromBytes(zmqSocket.recv(0));

We start off by defining the StockQuoteRequest object to give us all the quotes for Facebook stock from the last week.  We can then open up a ZMQ socket, set the timeout, connect to the ZMQ socket on the remote Mule instance and send the byte representation of the StockQuoteRequest to it.

zmqSocket.recv is then used to receive the bytes back from Mule.  From here we can use the listOfStockQuotesFromBytes helper method we wrote above to convert the Protocol Buffer representation to a List of StockQuotes.  Despite the fair bit of plumbing we did above, this is a pretty concise bit of client side code to invoke the remote service.

Conclusion

This blog post only touched on the features of ØMQ and the ØMQ Mule Module.  In addition to request-reply, other exchange-patterns are supported, like one-way, push and pull.  This effectively gives you the benefits of a reliable, asynchronous messaging layer without a centralized infrastructure.  I hope to cover this in a later post.

Protocol buffers also seem like a natural fit as a wire format for ØMQ.  protobuffers echo  ØMQ’s principals of being lightweight, fast and platform agnostic.  These are also, not coincidently, principals Mule shares as an integration framework.

The project for this example is available on GitHub.

Protocol Buffers Protocol (object-oriented programming) Buffer (application) ZeroMQ mobile app Web Service Java (programming language)

Published at DZone with permission of John D'Emic, DZone MVB. See the original article here.

Opinions expressed by DZone contributors are their own.

Related

  • What Are Protocol Buffers?
  • 6 of the Best API Testing Tools in the Market
  • Open Source Tools Options in Web Application Development
  • How to Build a Coronavirus Dashboard in Java

Partner Resources

×

Comments

The likes didn't load as expected. Please refresh the page and try again.

ABOUT US

  • About DZone
  • Support and feedback
  • Community research
  • Sitemap

ADVERTISE

  • Advertise with DZone

CONTRIBUTE ON DZONE

  • Article Submission Guidelines
  • Become a Contributor
  • Core Program
  • Visit the Writers' Zone

LEGAL

  • Terms of Service
  • Privacy Policy

CONTACT US

  • 3343 Perimeter Hill Drive
  • Suite 100
  • Nashville, TN 37211
  • [email protected]

Let's be friends: