DZone
Thanks for visiting DZone today,
Edit Profile
  • Manage Email Subscriptions
  • How to Post to DZone
  • Article Submission Guidelines
Sign Out View Profile
  • Post an Article
  • Manage My Drafts
Over 2 million developers have joined DZone.
Log In / Join
Refcards Trend Reports Events Over 2 million developers have joined DZone. Join Today! Thanks for visiting DZone today,
Edit Profile Manage Email Subscriptions Moderation Admin Console How to Post to DZone Article Submission Guidelines
View Profile
Sign Out
Refcards
Trend Reports
Events
Zones
Culture and Methodologies Agile Career Development Methodologies Team Management
Data Engineering AI/ML Big Data Data Databases IoT
Software Design and Architecture Cloud Architecture Containers Integration Microservices Performance Security
Coding Frameworks Java JavaScript Languages Tools
Testing, Deployment, and Maintenance Deployment DevOps and CI/CD Maintenance Monitoring and Observability Testing, Tools, and Frameworks
Partner Zones AWS Cloud
by AWS Developer Relations
Culture and Methodologies
Agile Career Development Methodologies Team Management
Data Engineering
AI/ML Big Data Data Databases IoT
Software Design and Architecture
Cloud Architecture Containers Integration Microservices Performance Security
Coding
Frameworks Java JavaScript Languages Tools
Testing, Deployment, and Maintenance
Deployment DevOps and CI/CD Maintenance Monitoring and Observability Testing, Tools, and Frameworks
Partner Zones
AWS Cloud
by AWS Developer Relations
Securing Your Software Supply Chain with JFrog and Azure
Register Today

The Latest Data Engineering Topics

article thumbnail
Microservices Are (Not) the New Black!
The ultimate purpose of this article is to make people break out of the bubble surrounding microservices' adoption— but to expose their shortcomings.
January 8, 2020
by Issam Sedki CORE
· 9,436 Views · 6 Likes
article thumbnail
Microservices Architectures: Microservices vs. SOA
In this article, a developer looks at how microservices architectures are different from Service Oriented Architectures (SOA).
May 15, 2019
by Ranga Karanam CORE
· 8,738 Views · 2 Likes
article thumbnail
Microservices Architecture: Advantages of Microservices
We continue our introductory series on microservices by taking a look at some of the advantages they offer devs and architects.
May 13, 2019
by Ranga Karanam CORE
· 12,142 Views · 6 Likes
article thumbnail
Microservices Architecture: Challenges of Building Microservices
Building good microservices is tricky business. In this post, we cover some of the challenges in the process and some of the basic ways to conquer them.
May 14, 2019
by Ranga Karanam CORE
· 7,677 Views · 2 Likes
article thumbnail
Microservices and Docker at Scale
Microservies and Docker have become the peanut butter and jelly of modern app delivery. They allow organizations to work in a consistent, isolated runtime environment.
February 6, 2017
by Anders Wallgren
· 25,913 Views · 23 Likes
article thumbnail
Microservice Issues, Challenges, and Hurdles
With microservices' benefits becoming more important to businesses, be sure you're also familiar with the challenges they present.
June 27, 2018
by Dmitriy S
· 13,429 Views · 6 Likes
article thumbnail
Micronaut JPA Application Performance in AWS Lambda
How to deploy a Micronaut application using GET, PUT, and POST, and how its performance compares to when it's deployed with JVM runtime vs. as a native language.
September 9, 2021
by Amrut Prabhu
· 4,221 Views · 2 Likes
article thumbnail
Memoization in Cost-based Optimizers
This blog post discusses memoization - a technique that allows cost-based optimizers to consider billions of alternative plans.
June 13, 2021
by Vladimir Ozerov CORE
· 7,048 Views · 3 Likes
article thumbnail
May in IoT: The Tinkerer's Edition
I hope everyone wants a slice of Raspberry Pi because there's a lot of it this month! Here's some inspiration for your next IoT project.
April 30, 2018
by Mike Gates
· 5,733 Views · 3 Likes
article thumbnail
MaxScale for the Rest of Us, Part 3: Install and Configure MaxScale
This third post in this series of blogs about MaxScale is finally getting where you want to go: Install and configure MaxScale. The first blog in this series was an overview of what MaxScale is and the second about how to set up a Cluster of MariaDB servers, using MariaDB Replication, for MaxScale to access. But now it's time to introduce MaxScale. If you skipped the second post as you already know how to set up MariaDB with Replication and all that, be remineded that I will use the same Linux server setup as outlined there even for the MaxScale server and for a client to do some testing, and I recommend you stick with that for now (for MariaDB itself you can use any relevant setup you want, MaxSCale doesn't really care, but MaxScale is pretty new and has still not been tested on that many platforms, so try to stick to the CentOS 6.5 setup I propose. Installing MaxScale Start by setting up a CentOS 6.5 server as outlined in the previous blog in this series and then log in a root. Set up the server to run on IPADDR 192.168.0.165. When logged in as root, create an installation directory for MaxScale, download it and install it: # cd /usr/local # mkdir skysql # cd skysql # curl https://downloads.skysql.com/files/SkySQL/MaxScale/maxscale.preview.0.4.tar.gz > maxscale.preview.0.4.tar.gz # tar xvfz maxscale.preview.0.4.tar.gz # cd maxscale Now we have MaxScale downloaded, but for maxscale to work, the MAXSCALE_HOME environment variable has to be set up, and to achieve this we, let's create a shell-script that starts MaxScale for us. # vi maxstart.sh Then set up this script like this: #!/bin/bash # export MAXSCALE_HOME=/usr/local/skysql/maxscale/MaxScale $MAXSCALE_HOME/../bin/maxscale Once we have that script, let's make it executable: # chmod +x maxstart.sh Configuring MaxScale The next step then is to configure MaxScale. The default configuration file for MaxScale is called MaxScale.cnf and is located in the etc directory under where MAXSCALE_HOME is located. In this case we will edit: # vi /usr/local/skysql/maxscale/MaxScale/etc/MaxScale.cnf In this file, each module has it's own section, as well as each server and there is also a section for MaxScale itself. Let's begin with MaxScale which has just one variables that controls the # of threads MaxScale uses: [maxscale] threads=1 Following this, we set up the servers we are to manage here, of which there are three. For every configuration section, except the one for MaxScale core, we have to tell what type of entity we are defining here, in this case it is server: [srv1] type=server address=192.168.0.160 port=3306 protocol=MySQLBackend [srv2] type=server address=192.168.0.162 port=3306 protocol=MySQLBackend [srv3] type=server address=192.168.0.163 port=3306 protocol=MySQLBackend As you can see, we define the ip address of the server and the port that MariaDB runs on. In addition we define which protocol module to use, and in this case there is not much else than MySQLBackend to choose from. As you can see, we do not define the master or slave properties of the servers, instead we let MaxScale figure that out for us by using a Monitor module, so now is the time to define that. For the monitor to work, it will connect to the respective MySQL servers so we need to define a username and password for this. In the previous post I created a user for this called maxuser using the password maxpwd. Also, to simplify matters I created this user with full privileges, and this really isn't recommended practice for production use. Last, with MaxScale you have the option to obfuscate the passwords used in the configuration file for added security, but I am not using that feature here as I want to keep things simple. So, this is how we define this monitor: [MySQL Monitor] type=monitor module=mysqlmon servers=srv1,srv2,srv3 user=maxuser passwd=maxpwd This should be pretty straightforward, right? The module variable is a reference to the monitoring module that we are to use. With this in place, we now needs to set up the actual router, and there are two sections for this, one for the listener and one for the router. The router we are about to use is the read-write split router, which manages routing writes to the master and reads to the slaves. Let's start with the router that is the central point here: [RWSplitRouter] type=service router=readwritesplit servers=srv1,srv2,srv3 user=maxuser passwd=maxpwd The reason we need to have a username and password even for the router is that this needs to authenticate users connecting to MaxScale itself, and to do that it needs to connect to the server it manages, connect to the mysql database and get the user authorization data from there, which is why we need an account that can access the mysql database. Now we are close to getting started with MaxScale, what is left is to set up a listener for the router we defined above: [RWSplitListener] type=listener service=RWSplitRouter protocol=MySQLClient port=3306 That concludes the configuration of MaxScale! Testing MaxScale # ./maxstart.sh Now we should be able to connect to MaxScale and test that it works. We do this by setting up a MariaDB Client server. This is set up just like our previous servers, only that we only install the MariaDB client. So set up a server like before, set IPADDR to 192.168.0.167 and run, as root: # yum install MariaDB-client Now we are real close! The rest of this post assumes that you are connected to the MariaDB Client on 192.168.0.167 I here assume that you have created the t1 table that we used to test replication in the previous post, if not, create it and populate it now: # mysql -h 192.168.0.165 -u maxuser -pmaxpwd test -e "create table t1(c1 int)" # mysql -h 192.168.0.165 -u maxuser -pmaxpwd test -e "insert into t1 values(1)" Now we have a table to test with, so let's see what happens, and let's check that our SELECTs are being routed to our two slaves. We can do that by using the @@server_id variable, which is different one these two of course: # mysql -h 192.168.0.165 -u maxuser -pmaxpwd test -e "select c1, @@server_id from t1" +------+-------------+ | c1 | @@server_id | +------+-------------+ | 1 | 12 | +------+-------------+ Cool, this ended up with one of the slaves, so if I try it again, it shoudl end up with the other slave, right: # mysql -h 192.168.0.165 -u maxuser -pmaxpwd test -e "select c1, @@server_id from t1" +------+-------------+ | c1 | @@server_id | +------+-------------+ | 1 | 11 | +------+-------------+ Whoa!, it actually seems to work! Also, not that if you executed the create table above, then this got routed to the master! We are all set, right? Well, no, there are a few things left to do. And if you didn't get the MaxScale to work, like seeing this in the error log: /usr/local/skysql/maxscale/MaxScale/log/skygw_err1.log: Couldn't find suitable Slave from 3 candidates. Or if you can't connect to MaxScale, then the most likely issue is that you didn't stop iptables: # /etc/init.d/iptables stop But hopefully things work now, but that it works doesn't mean we are finished, there are some things with this configuration that needs fixing, both on the servers and in MaxScale itself, and there is also one thing to watch for in the replication setup. So the next post in the series will be about fine-tuning the MaxScale installation and the MariaDB Cluster it is accessing. /Karlsson
May 23, 2023
by Anders Karlsson
· 9,574 Views · 2 Likes
article thumbnail
Maximize Your Project's Success With the Right Database Selection
Learn the key considerations for choosing the right database for your project. This article will help you make an informed decision.
December 26, 2022
by Alex Vakulov
· 6,458 Views · 1 Like
article thumbnail
March in IoT: Synergy and IoT
Want to see how technologies like AI and blockchain are helping solve IoT's problems? This news and tutorial roundup will help.
March 19, 2018
by Mike Gates
· 7,609 Views · 4 Likes
article thumbnail
MapReduce and Yarn: Hadoop Processing Unit Part 1
In this article, we begin a two-part series on basic implementation of MapReduce and Yarn in the Hadoop ecosystem.
December 26, 2019
by Dheeraj Gupta CORE
· 99,267 Views · 5 Likes
article thumbnail
Managing Amazon WorkDocs With AWS Managed Microsoft Active Directory (AD)
Learn about how AWS WorkDocs provides multiple business benefits as a document management system by providing secure and limited access.
January 27, 2020
by Raja Sekhar Mandava
· 5,231 Views · 2 Likes
article thumbnail
Devs and Data, Part 3: Managing a Large Volume of Data
We take a look at what respondents to our 2019 Big Data Survey told us about data management and coping with data at enormous volumes.
February 25, 2019
by Jordan Baker
· 6,881 Views · 2 Likes
article thumbnail
Managed MQTT Broker Comparison — Product Packages and Pricing
In this article, we will compare the differences between several managed MQTT Brokers in aspects of product packages and pricing.
October 25, 2022
by David Li
· 5,047 Views · 1 Like
article thumbnail
Managed MQTT Broker Comparison — Console/Dashboard Features
I will compare the differences between several managed MQTT Brokers in aspects of console/dashboard features.
November 9, 2022
by David Li
· 5,225 Views · 1 Like
article thumbnail
Making Your SSR Sites 42x Faster With Redis Cache
Let's look at how you can leverage Redis Cache with Node.JS and Express
May 5, 2022
by Johnny Simpson CORE
· 6,867 Views · 8 Likes
article thumbnail
Making the County List Dynamic
A Zone Leader continues his case study of building a new application for a family member. In this article, read how different tax rates by county and for a particular time span introduced a challenge.
December 11, 2018
by John Vester CORE
· 10,283 Views · 3 Likes
article thumbnail
Making Machine Learning Accessible for Enterprises: Part 2
Let's take a look at discussing critical areas of machine learning-based solutions, such as model explainability and model governance.
August 8, 2018
by Ramesh Balakrishnan
· 4,570 Views · 3 Likes
  • Previous
  • ...
  • 5
  • 6
  • 7
  • 8
  • 9
  • 10
  • 11
  • 12
  • 13
  • 14
  • ...
  • Next

ABOUT US

  • About DZone
  • Send feedback
  • Careers
  • Sitemap

ADVERTISE

  • Advertise with DZone

CONTRIBUTE ON DZONE

  • Article Submission Guidelines
  • Become a Contributor
  • Visit the Writers' Zone

LEGAL

  • Terms of Service
  • Privacy Policy

CONTACT US

  • 600 Park Offices Drive
  • Suite 300
  • Durham, NC 27709
  • support@dzone.com

Let's be friends: