DZone
Thanks for visiting DZone today,
Edit Profile
  • Manage Email Subscriptions
  • How to Post to DZone
  • Article Submission Guidelines
Sign Out View Profile
  • Post an Article
  • Manage My Drafts
Over 2 million developers have joined DZone.
Log In / Join
Refcards Trend Reports Events Over 2 million developers have joined DZone. Join Today! Thanks for visiting DZone today,
Edit Profile Manage Email Subscriptions Moderation Admin Console How to Post to DZone Article Submission Guidelines
View Profile
Sign Out
Refcards
Trend Reports
Events
Zones
Culture and Methodologies Agile Career Development Methodologies Team Management
Data Engineering AI/ML Big Data Data Databases IoT
Software Design and Architecture Cloud Architecture Containers Integration Microservices Performance Security
Coding Frameworks Java JavaScript Languages Tools
Testing, Deployment, and Maintenance Deployment DevOps and CI/CD Maintenance Monitoring and Observability Testing, Tools, and Frameworks
Partner Zones AWS Cloud
by AWS Developer Relations
Culture and Methodologies
Agile Career Development Methodologies Team Management
Data Engineering
AI/ML Big Data Data Databases IoT
Software Design and Architecture
Cloud Architecture Containers Integration Microservices Performance Security
Coding
Frameworks Java JavaScript Languages Tools
Testing, Deployment, and Maintenance
Deployment DevOps and CI/CD Maintenance Monitoring and Observability Testing, Tools, and Frameworks
Partner Zones
AWS Cloud
by AWS Developer Relations
Building Scalable Real-Time Apps with AstraDB and Vaadin
Register Now

Trending

  • Does the OCP Exam Still Make Sense?
  • How Agile Works at Tesla [Video]
  • You’ve Got Mail… and It’s a SPAM!
  • Java String Templates Today

Trending

  • Does the OCP Exam Still Make Sense?
  • How Agile Works at Tesla [Video]
  • You’ve Got Mail… and It’s a SPAM!
  • Java String Templates Today
  1. DZone
  2. Software Design and Architecture
  3. Cloud Architecture
  4. Connecting Zato Clusters With WebSockets and Publish/Subscribe Queues

Connecting Zato Clusters With WebSockets and Publish/Subscribe Queues

See how to connect Zato clusters with WebSockets and publish/subscribe queues.

Dariusz Suchojad user avatar by
Dariusz Suchojad
·
Nov. 11, 19 · Review
Like (2)
Save
Tweet
Share
15.63K Views

Join the DZone community and get the full member experience.

Join For Free

Image title

Connecting Zato Clusters With WebSockets

Since version 3.0, it is possible to directly connect Zato clusters and exchange messages as though remote services were running in a local instance. This makes it an ideal choice for environments split into multiple parts.

You may also like:  Using RabbitMQ in Cluster

Introduction

The reasons to have more than one cluster, each with one or more servers, may vary:

  • For HA and performance, environments may be broken out geographically into a setup with one cluster per continent or a region of the world
  • CPU-extensive operations may be carried out in one cluster with another making use of the results the former produces to offer a set of APIs
  • For legal reasons, it may not be allowed to run all integration services in one cluster, using the same hardware and software infrastructure

The new feature in Zato 3.0 which allows for efficient communication between clusters are WebSocket connections — one of clusters will create a channel through with other clusters may invoke its services via their outgoing connections.

WebSockets (WSX for short) have essentially no overhead in practice but they can be used for bi-directional communication hence they are a great choice for such scenarios.

From a Zato programmer's perspective, all the communication details are hidden and a couple of lines of code suffices to invoke services or receive messages from remote clusters, for instance:

# Obtain a handle to a remote connection
with self.out.wsx.get('My Connection').conn.client() as client:

    # Invoke a remote service - expects a Python dict on input
    # and returns a Python dict on response. All the serialization
    # and network connectivity is handled automatically.
    response = client.invoke(msg)


Architecture and Configuration


Screenshots

  • Each cluster which is to become a recipient of messages from other clusters needs to have a new WebSocket channel created with service helpers.web-sockets-gateway mounted on it. A security definition should also be attached as required.
  • Each cluster that should invoke another one needs to have an outgoing WebSocket connection created - make sure Is remote end Zato checkbox is on and that credentials are provided, if required by the other side.
  • If the cluster with an outgoing connection is interested in receiving publish/subscribe messages, all topics it wants to subscribe to should be listed, one in each line. Make sure the cluster with a channel has a correct pub/sub endpoint configured for that channel.
  • The cluster which establishes the connection (here, cluster1) may also want to subscribe to events of interest via hooks services — more about it below.
  • Once an outgoing connection is created, internal tasks will start on cluster1 to establish a remote connection to server2. If successful, authentication will take place automatically. Finally, if configured, a hook service will fire to let cluster1 know that a new connection was established. Afterwards, cluster1 may start to invoke remote services.
  • There are no other steps involved. At this point, everything is configured and ready to be used.

ScreenshotsScreenshots

From a Programmer's Perspective

  • To invoke remote Zato services, programmers use WebSockets outgoing connections methods — providing a dictionary of input data to the invocation and receiving a dictionary of data on input. Note that the invocation is synchronous, your service is blocked until the remote cluster responds.
# -*- coding: utf-8 -*-

from __future__ import absolute_import, division, print_function, unicode_literals

from zato.server.service import Service

class MyService(Service):
    def handle(self):

        # Message to send - needs to be a dictionary with name
        # of the service to invoke as well as its input data, if any is required.
        # In this case, we are invoking an echo service
        # which writes back to output anything it receives on input.
        msg = {
             'service':'zato.helpers.echo',
             'request': {
                 'elem1': 'value1',
                 'elem2': 'value2',
             }
         }

        # Name of the connection to send messages through
        conn_name = 'My WSX Outconn'

        # Obtain a client from the connection pool
        with self.out.wsx.get(conn_name).conn.client() as client:

            # Send the message and read its response
            response = client.send(msg)

            # Or, client.invoke can be used with Zato WebSocket connections,
            # this method is an alias to client.send
            response = client.invoke(msg)

            # Log the response received
            self.logger.info('Response is `%s`', response.data)


INFO - Response is `{u'elem2': u'value2', u'elem1': u'value1'}`


  • To receive messages, hook services are used. There are three events for which hooks can be triggered - they can be handled by different services or the same one, it is up to users:
  1. Upon connecting to a remote cluster, including reconnects (on_connect)
  2. Upon receiving messages from remote clusters (on_message)
  3. Once a connection to the remote cluster is shut down (on_close)
  • The on_message hook can be combined with publish/subscribe topics and queues — each time the remote cluster (the one with a WSX channel) publishes a message that the local cluster (the one with a WSX outgoing connection) is interested in, the on_message hook will be called to handle it, in this manner making it possible for remote clusters to deliver messages to clusters subscribing to topics.
  • Each hook is just a Zato service with a specific SimpleIO signature, as in the on_message example below:
# -*- coding: utf-8 -*-

from __future__ import absolute_import, division, print_function, unicode_literals

from zato.server.service import Opaque, Service

class OnMessageHook(Service):
    class SimpleIO:
        input_optional = (Opaque('ctx'),)

    def handle(self):

        # Object describing incoming data
        ctx = self.request.input.ctx

        # Message type
        msg_type = ctx.type

        # Data received
        data = ctx.data

        # Log message type
        self.logger.info('Msg type: `%s`', msg_type)

        # Log actual data
        self.logger.info('Data received: `%s`', data.data)

        # Log metadata - ID and timestamp
        self.logger.info('Meta: `%s` `%s`', data.id, data.timestamp)


Now, we can use web-admin to publish a test message and confirm that the on_message service receives it:

Screenshots

In the on_message service's server logs:

INFO - Msg type: `message`
INFO - Data received: `[
  {u'delivery_count': 0,
   u'msg_id': u'zpsme26726911ffbe8cba2cca278',
   u'expiration_time_iso': u'2086-09-21T14:03:05.285470',
   u'topic_name': u'/customer/new',
   u'pub_time_iso': u'2018-09-03T10:48:58.285470',
   u'priority': 5,
   u'expiration': 2147483647000,
   u'has_gd': True,
   u'data': u'This is a sample message',
   u'sub_key': u'zpsk.websockets.6ef529f7cab64a71d8bd2878',
   u'mime_type': u'text/plain',
   u'size': 24}
  ]`
INFO - `6fd296ecf78493a3a0ce7570` `2018-09-03T10:49:00.540024`


Summary

  • In Zato3.0 and later clusters can be easily connected using efficient WebSocket connections
  • Once configured, such connections are maintained automatically under the hood
  • Connections can be used either to invoke remote services or to receive messages from publish/subscribe topics and queues in a straightforward manner
  • As always, the focus of the new functionality is to let users concentrate on their actual integration needs rather than on tedious, low-level, details

Further Reading

Redis Cluster on Java for Scaling and High Availability

Querying Neo4j Clusters

cluster WebSocket Connection (dance) remote

Opinions expressed by DZone contributors are their own.

Trending

  • Does the OCP Exam Still Make Sense?
  • How Agile Works at Tesla [Video]
  • You’ve Got Mail… and It’s a SPAM!
  • Java String Templates Today

Comments

Partner Resources

X

ABOUT US

  • About DZone
  • Send feedback
  • Careers
  • Sitemap

ADVERTISE

  • Advertise with DZone

CONTRIBUTE ON DZONE

  • Article Submission Guidelines
  • Become a Contributor
  • Visit the Writers' Zone

LEGAL

  • Terms of Service
  • Privacy Policy

CONTACT US

  • 600 Park Offices Drive
  • Suite 300
  • Durham, NC 27709
  • support@dzone.com

Let's be friends: