Over a million developers have joined DZone.
{{announcement.body}}
{{announcement.title}}

Tuning Linux for MongoDB: Automated Tuning on Redhat and CentOS

DZone's Guide to

Tuning Linux for MongoDB: Automated Tuning on Redhat and CentOS

Getting every last bit of performance out of your Mongo instance can mean the difference between success and failure.

Free Resource

In a previous blog post, Tuning Linux for MongoDB, I covered several tunings for an efficient MongoDB deployment on Linux in Production. This post expands on that one.

While I felt the tuning Linux for MongoDB was a very useful blog post that results in a great baseline tuning, something bugged me about how much effort and touch-points were required to achieve an efficient Linux installation for MongoDB. More importantly, I noticed some cases where the tunings (example: changes to disk I/O scheduler in /etc/udev.d) were ignored on some recent RedHat and CentOS versions. With these issues in mind, I started to investigate better solutions for achieving the tuned baseline.

Tuned

In RedHat (and thus CentOS) 7.0, a daemon called “tuned” was introduced as a unified system for applying tunings to Linux. Tuned operates with simple, file-based tuning profiles, and provides an admin command-line interface named tuned-adm for applying, listing, and even recommending tuned profiles.

Some operational benefits of tuned:

  • File-based configuration: Profile tunings are contained in a simple, consolidated files.
  • Swappable profiles: Profiles are easily changed back and forth.
  • Standards compliance: Using tuned profiles ensures tunings are not overridden or ignored.

Note: If you use configuration management systems like Puppet, Chef, Salt, Ansible, etc., I suggest you configure those systems to deploy tunings via tuned profiles instead of applying tunings directly, as tuned will likely start to fight this automation, overriding the changes.

The default available tuned profiles (as of RedHat 7.2.1511) are:

  • Balanced.
  • Desktop.
  • Latency-performance.
  • Network-latency.
  • Network-throughput.
  • Power save.
  • Throughput-performance.
  • Virtual-guest.
  • Virtual-host.

The profiles that are generally interesting for database usage are:

  • Latency-performance.

    “A server profile for typical latency performance tuning. This profile disables dynamic tuning mechanisms and transparent hugepages. It uses the performance governer for p-states through cpuspeed, and sets the I/O scheduler to deadline.network-latency.”

  • Throughput-performance.

    “A server profile for typical throughput performance tuning. It disables tuned and ktune power saving mechanisms, enables sysctl settings that improve the throughput performance of your disk and network I/O, and switches to the deadline scheduler. CPU governor is set to performance.”

  • Network-latency: Includes latency-performance, disables transparent_hugepages, disables NUMA balancing and enables some latency-based network tunings.
  • Network-throughput: Includes throughput-performance and increases network stack buffer sizes.

I find “network-latency” is the closest match to our recommended tunings, but some additional changes are still required.

The good news is tuned was designed to be flexible, so I decided to make a MongoDB-specific profile. Enter tuned-percona-mongodb.

Tuned-Percona-Mongodb

You can find this here.

tuned-percona-mongodb is a performance-focused tuned profile for MongoDB on Linux, and is currently considered experimental (no guarantees or warranties). It’s hosted in our Percona-Lab GitHub repo.

tuned-percona-mongodb applies the following tunings (from the previous tuning article) on a Redhat/CentOS 7+ host:

  • Disabling of transparent huge pages.
  • Kernel network tunings (sysctls).
  • Virtual memory dirty ratio changes (sysctls).
  • Virtual memory “swappiness” (sysctls).
  • Block-device readahead settings (on all disks except /dev/sda by default).
  • Block-device I/O scheduler (on all disks except /dev/sda by default).

The following tunings that our previous tuning article didn’t cover are also applied:

After a successful deployment of this profile, only these recommendations are outstanding:

  • Filesystem type and mount options. Tuned does not handle filesystem mount options; this needs to be done manually in /etc/fstab. To quickly summarize, we recommend the XFS or EXT4 filesystem type for MongoDB data when using MMAPv1 or RocksDB storage engines, and XFS ONLY when using WiredTiger. For all filesystems, using the mount options rw,noatime will reduce some activity.
  • NUMA disabling or interleaving. Tuned does not handle NUMA settings and these still need to be handled via the MongoDB init script or the BIOS on/off switch.
  • Linux ulimits. Tuned does not set Linux ulimit settings. However, Percona Server for MongoDB RPM packages do this for you at startup! See LimitNOFILE and LimitNPROC in /usr/lib/systemd/system/mongod.service for more information.
  • NTP server. Tuned does not handle installation of RPM packages or enabling of services. You will need to install the ntp package and enable and start the ntpd service manually:

sudo yum install ntp

sudo systemctl enable ntpd

sudo systemctl start ntpd

Tuned-Percona-Mongodb: Installation

The installation of this profile is as simple as checking out the repository with a Git command and then running sudo make enable. Full output here:

$git clonehttps://github.com/Percona-Lab/tuned-percona-mongodb

$cdtuned-percona-mongodb

$sudo makeenable

if[-d/etc/tuned];then

cp-dpR percona-mongodb/etc/tuned/percona-mongodb;

echo"### 'tuned-percona-mongodb' is installed. Enable with 'make enable'.";

else

echo"### ERROR: cannot find tuned config dir at /etc/tuned!";

exit1;

### 'tuned-percona-mongodb' is installed. Enable with 'make enable'.

tuned-adm profile percona-mongodb

tuned-adm active

Current active profile:percona-mongodb

In the example above, you can see that percona-mongodb is now the actively tuned profile on the system (mentioned on the last output line).

The tuned profile files are installed to /etc/tuned/percona-mongodb, as seen here:

$ls-alh/etc/tuned/percona-mongodb/*.*

-rwxrwxr-x.1root root677Nov2220:00percona-mongodb.sh

-rw-rw-r--.1root root1.4KNov2220:00tuned.conf

Let’s check that the deadline i/o scheduler is now the current scheduler on any disk that isn’t /dev/sda (sdb used below):

$cat/sys/block/sdb/queue/scheduler

noop[deadline]cfq

Transparent huge pages should be disabled (it is!):

$cat/sys/kernel/mm/transparent_hugepage/enabled

always madvise[never]

$cat/sys/kernel/mm/transparent_hugepage/defrag

always madvise[never]

Block-device readahead should be 32 (16 kb) on /dev/sdb (looks good!):

That was easy!

Tuned-Percona-Mongodb: Uninstallation

To uninstall the profile, run Sudo Make Uninstall in the github checkout directory:

if[-d/etc/tuned/percona-mongodb];then

echo"### Disabling tuned profile 'tuned-percona-mongodb'";

echo"### Changing tuned profile to 'latency-performance', adjust if necessary after!";

tuned-adm profile latency-performance;

tuned-adm active;

else

echo"tuned-percona-mongodb profile not installed!";

### Disabling tuned profile 'tuned-percona-mongodb'

### Changing tuned profile to 'latency-performance', adjust if necessary after!

Current active profile:latency-performance

if[-d/etc/tuned/percona-mongodb];thenrm-rf/etc/tuned/percona-mongodb;fi

Note: The uninstallation will enable the latency-performance tuned profile, change this after the uninstall if needed

To confirm the uninstallation, let’s check if the block device readahead is set back to default (256/128kb):

$sudo blockdev--getra/dev/sdb 
256 

Uninstall complete.

Conclusion

So far, Tuned shows a lot of promise for tuning Linux for MongoDB, providing a single, consistent interface for tuning the Linux operating system. In the future, I would like to see the documentation for tuned improve. However, its simplicity makes the need for documentation rarely necessary.

As mentioned, after applying tuned-percona-mongodb, you still need to configure an NTP server, NUMA (in some cases) and the filesystem type+tunings manually. The majority of the time, effort and room for mistakes are greatly reduced using this method.

If you have any issues with this profile for tuning Linux for MongoDB or have any questions, please create a Github issue at this URL.

Topics:
performance ,tuning ,linux ,mongo db

Published at DZone with permission of Tim Vaillencourt, DZone MVB. See the original article here.

Opinions expressed by DZone contributors are their own.

{{ parent.title || parent.header.title}}

{{ parent.tldr }}

{{ parent.urlSource.name }}