DZone
Thanks for visiting DZone today,
Edit Profile
  • Manage Email Subscriptions
  • How to Post to DZone
  • Article Submission Guidelines
Sign Out View Profile
  • Post an Article
  • Manage My Drafts
Over 2 million developers have joined DZone.
Log In / Join
Refcards Trend Reports
Events Video Library
Over 2 million developers have joined DZone. Join Today! Thanks for visiting DZone today,
Edit Profile Manage Email Subscriptions Moderation Admin Console How to Post to DZone Article Submission Guidelines
View Profile
Sign Out
Refcards
Trend Reports
Events
View Events Video Library
Zones
Culture and Methodologies Agile Career Development Methodologies Team Management
Data Engineering AI/ML Big Data Data Databases IoT
Software Design and Architecture Cloud Architecture Containers Integration Microservices Performance Security
Coding Frameworks Java JavaScript Languages Tools
Testing, Deployment, and Maintenance Deployment DevOps and CI/CD Maintenance Monitoring and Observability Testing, Tools, and Frameworks
Culture and Methodologies
Agile Career Development Methodologies Team Management
Data Engineering
AI/ML Big Data Data Databases IoT
Software Design and Architecture
Cloud Architecture Containers Integration Microservices Performance Security
Coding
Frameworks Java JavaScript Languages Tools
Testing, Deployment, and Maintenance
Deployment DevOps and CI/CD Maintenance Monitoring and Observability Testing, Tools, and Frameworks

Integrating PostgreSQL Databases with ANF: Join this workshop to learn how to create a PostgreSQL server using Instaclustr’s managed service

Mobile Database Essentials: Assess data needs, storage requirements, and more when leveraging databases for cloud and edge applications.

Monitoring and Observability for LLMs: Datadog and Google Cloud discuss how to achieve optimal AI model performance.

Automated Testing: The latest on architecture, TDD, and the benefits of AI and low-code tools.

Related

  • Penetration Testing: A Comprehensive Guide
  • Understanding the Basics of Neural Networks and Deep Learning
  • Simple Privacy Tweaks: Fast and Effective Methods To Secure Your Personal Information
  • Risk Assessment Using Blockchain

Trending

  • DevSecOps: Integrating Security Into Your DevOps Workflow
  • How To Use ChatGPT API in Python for Your Real-Time Data
  • OneStream Fast Data Extracts APIs
  • Monkey-Patching in Java
  1. DZone
  2. Data Engineering
  3. IoT
  4. ActiveMQ - Network of Brokers Explained (Part Four)

ActiveMQ - Network of Brokers Explained (Part Four)

Ashwini  Kuntamukkala user avatar by
Ashwini Kuntamukkala
·
Apr. 02, 14 · Interview
Like (2)
Save
Tweet
Share
10.52K Views

Join the DZone community and get the full member experience.

Join For Free

In the previous part 3, we have seen how ActiveMQ helps distinguish remote consumers from local consumers which helps in determining shorter routes from message producers to consumers. In this part 4,  we will look into how to load balance concurrent consumers on remote brokers.

Let’s consider a bit more advanced configuration to load balance concurrent message consumers on a queue in remote brokers as shown below. 
Part 4 - Network of brokers 
In the above configuration, we have a message producer sending messages into a queue moo.bar on broker-1. Broker-1 establishes network connectors to broker-2 and broker-3. Consumer C1 consumes messages from queue moo.bar on broker-2 while consumers C2 and C3 are concurrent consumers on queue moo.bar on broker-3. 

Let's see this in action

Let's create three brokers instances...
  1. Ashwinis-MacBook-Pro:bin akuntamukkala$ pwd
    /Users/akuntamukkala/apache-activemq-5.8.0/bin

  2. Ashwinis-MacBook-Pro:bin akuntamukkala$./activemq-admin create ../cluster/broker-1

  3. Ashwinis-MacBook-Pro:bin akuntamukkala$./activemq-admin create ../cluster/broker-2

  4. Ashwinis-MacBook-Pro:bin akuntamukkala$./activemq-admin create ../cluster/broker-3

  5. Fix the broker-2 and broker-3 transport, amqp connectors and jetty http port by modifying the corresponding conf/activemq.xml and conf/jetty.xml as follows:

    Broker Openwire Port Jetty HTTP Port AMQP Port
    broker-1 61616 8161 5672
    broker-2 61626 9161 5682
    broker-3 61636 10161 5692

  6. Fix network connector on broker-1 such that messages on queues can be forwarded dynamically to consumers on broker-2 and broker-3. This can be done by adding the following XML snippet into broker-1's conf/activemq.xml
    <networkConnectors>
        <networkConnector
          name="Q:broker1->broker2"
          uri="static:(tcp://localhost:61626)"
          duplex="false"
          decreaseNetworkConsumerPriority="true"
          networkTTL="2"
          dynamicOnly="true">
          <excludedDestinations>
             <topic physicalName=">" />
          </excludedDestinations>
        </networkConnector>
        <networkConnector
           name="Q:broker1->broker3"
           uri="static:(tcp://localhost:61636)"
           duplex="false"
           decreaseNetworkConsumerPriority="true"
           networkTTL="2"
           dynamicOnly="true">
           <excludedDestinations>
                <topic physicalName=">" />
           </excludedDestinations
        </networkConnector>
    </networkConnectors>

  7. Start broker-2, broker-3 and broker-1. We can start these in any order.
    1. /apache-activemq-5.8.0/cluster/broker-3/bin$ ./broker-3 console
    2. /apache-activemq-5.8.0/cluster/broker-2/bin$ ./broker-2 console
    3. /apache-activemq-5.8.0/cluster/broker-1/bin$ ./broker-1 console
  8. Let's start the consumers C1 on broker-2 and C2, C3 on broker-3 but on the same queue called "moo.bar"
    1. /apache-activemq-5.8.0/example$ ant consumer -Durl=tcp://localhost:61626 -Dsubject=moo.bar
    2. /apache-activemq-5.8.0/example$ ant consumer -Durl=tcp://localhost:61636 -Dsubject=moo.bar -DparallelThreads=2

      The consumer subscriptions are forwarded by broker-2 and broker-3 to their neighboring broker-1 which has a network connector established to both broker-2 and broker-3 by the use of advisory messages. 
  9. Let's review the broker web consoles to see the queues and corresponding consumers. 
    1. We find that broker-2's web console shows one queue "moo.bar" having 1 consumer, broker-3's web console shows one queue "moo.bar" having 2 concurrent consumers
    2. Though there are three consumers (C1 on broker-2 and C2,C3 on broker-3), broker-1 sees only two consumers (representing broker-2 and broker-3).
    3. http://localhost:8161/admin/queues.jsp










    4. This is because the network connector from broker-1 to broker-2 and to broker-3 by default has a property "conduitSubscriptions" which is true. 
      Due to which broker-3's C2 and C3 which consume messages from the same queue "moo.bar" are treated as one consumer in broker-1.
  10. Let's produce 30 messages into broker-1's queue moo.bar and see how the messages are divvied among the consumers C1, C2 and C3
Shows how the messages were propagated from producer to consumers C1, C2, C3
As seen above, even though there were three consumers and 30 messages, they didn't get to process 10 messages each as C2, C3 subscriptions were consolidated into one consumer at broker-1. 
conduitSubscriptions="true" is a useful setting if we were creating subscribers on topics as that would prevent duplicate messages. More on this in part 5.
So, in order to make C2 and C3 subscriptions on queue moo.bar propagate to broker-1, let's redo the same steps 6, 7, 8, 9 and 10 after setting conduitSubscriptions="false" in broker-1's network connector configuration in conf/activemq.xml 
Here is the new network connector configuration snippet for broker-1:
 <networkConnectors>
  <networkConnector
    name="Q:broker1->broker2"
    uri="static:(tcp://localhost:61626)"
    duplex="false"
    decreaseNetworkConsumerPriority="true"
    networkTTL="2"
    conduitSubscriptions="false"
    dynamicOnly="true">
    <excludedDestinations>
       <topic physicalName=">" />
    </excludedDestinations>
  </networkConnector>
  <networkConnector
    name="Q:broker1->broker3"
    uri="static:(tcp://localhost:61636)"
    duplex="false"
    decreaseNetworkConsumerPriority="true"
    networkTTL="2"
    conduitSubscriptions="false"
    dynamicOnly="true">
    <excludedDestinations>
       <topic physicalName=">" />
    </excludedDestinations>
  </networkConnector>
</networkConnectors>

Upon restarting the brokers and consumers C1, C2 and C3 and producing 30 messages into broker-1's moo.bar queue, we find that all of the three consumer subscriptions are visible at broker-1. As a result broker-1 dispatches 10 messages to each of the consumers in a round-robin fashion to load balance. This is depicted pictorially below. 
Shows how the messages were propagated from producer to consumers C1, C2, C3
Broker-1's web console @ http://localhost:8161/admin/queueConsumers.jsp?JMSDestination=moo.bar shows that broker-1 now sees 3 consumers and dispatches 10 messages to each consumer
Thus in this part 4 of the blog series, we have seen how we can load balance remote concurrent consumers which are consuming messages from a queue. 
As always, your comments and feedback is appreciated!
In the next part 5, we will explore how the same scenario will play out if we were to use a topic instead of a queue. Stay tuned...

References

  • http://fusesource.com/docs/esb/4.3/amq_clustering/Networks-Connectors.html

Resources

  • The configuration files (activemq.xml and jetty.xml) of all the brokers used in this blog are available here.







 
consumer Network

Published at DZone with permission of Ashwini Kuntamukkala, DZone MVB. See the original article here.

Opinions expressed by DZone contributors are their own.

Related

  • Penetration Testing: A Comprehensive Guide
  • Understanding the Basics of Neural Networks and Deep Learning
  • Simple Privacy Tweaks: Fast and Effective Methods To Secure Your Personal Information
  • Risk Assessment Using Blockchain

Comments

Partner Resources

X

ABOUT US

  • About DZone
  • Send feedback
  • Careers
  • Sitemap

ADVERTISE

  • Advertise with DZone

CONTRIBUTE ON DZONE

  • Article Submission Guidelines
  • Become a Contributor
  • Visit the Writers' Zone

LEGAL

  • Terms of Service
  • Privacy Policy

CONTACT US

  • 3343 Perimeter Hill Drive
  • Suite 100
  • Nashville, TN 37211
  • support@dzone.com

Let's be friends: