In a previous blog post, Tuning Linux for MongoDB, I covered several tunings for an efficient MongoDB deployment on Linux in Production. This post expands on that one.
While I felt the tuning Linux for MongoDB was a very useful blog post that results in a great baseline tuning, something bugged me about how much effort and touch-points were required to achieve an efficient Linux installation for MongoDB. More importantly, I noticed some cases where the tunings (example: changes to disk I/O scheduler in /etc/udev.d) were ignored on some recent RedHat and CentOS versions. With these issues in mind, I started to investigate better solutions for achieving the tuned baseline.
In RedHat (and thus CentOS) 7.0, a daemon called “tuned” was introduced as a unified system for applying tunings to Linux. Tuned operates with simple, file-based tuning profiles, and provides an admin command-line interface named tuned-adm for applying, listing, and even recommending tuned profiles.
Some operational benefits of tuned:
- File-based configuration: Profile tunings are contained in a simple, consolidated files.
- Swappable profiles: Profiles are easily changed back and forth.
- Standards compliance: Using tuned profiles ensures tunings are not overridden or ignored.
Note: If you use configuration management systems like Puppet, Chef, Salt, Ansible, etc., I suggest you configure those systems to deploy tunings via tuned profiles instead of applying tunings directly, as tuned will likely start to fight this automation, overriding the changes.
The default available tuned profiles (as of RedHat 7.2.1511) are:
- Power save.
The profiles that are generally interesting for database usage are:
“A server profile for typical latency performance tuning. This profile disables dynamic tuning mechanisms and transparent hugepages. It uses the performance governer for p-states through cpuspeed, and sets the I/O scheduler to deadline.network-latency.”
“A server profile for typical throughput performance tuning. It disables tuned and ktune power saving mechanisms, enables sysctl settings that improve the throughput performance of your disk and network I/O, and switches to the deadline scheduler. CPU governor is set to performance.”
- Network-latency: Includes latency-performance, disables transparent_hugepages, disables NUMA balancing and enables some latency-based network tunings.
- Network-throughput: Includes throughput-performance and increases network stack buffer sizes.
I find “network-latency” is the closest match to our recommended tunings, but some additional changes are still required.
The good news is tuned was designed to be flexible, so I decided to make a MongoDB-specific profile. Enter tuned-percona-mongodb.
You can find this here.
tuned-percona-mongodb is a performance-focused tuned profile for MongoDB on Linux, and is currently considered experimental (no guarantees or warranties). It’s hosted in our Percona-Lab GitHub repo.
tuned-percona-mongodb applies the following tunings (from the previous tuning article) on a Redhat/CentOS 7+ host:
- Disabling of transparent huge pages.
- Kernel network tunings (sysctls).
- Virtual memory dirty ratio changes (sysctls).
- Virtual memory “swappiness” (sysctls).
- Block-device readahead settings (on all disks except /dev/sda by default).
- Block-device I/O scheduler (on all disks except /dev/sda by default).
The following tunings that our previous tuning article didn’t cover are also applied:
- CPU C-States and Energy-Bias tunings for high performance.
- Disk Aggressive Link Power Management (ALPM) tuning, if available and supported.
After a successful deployment of this profile, only these recommendations are outstanding:
- Filesystem type and mount options. Tuned does not handle filesystem mount options; this needs to be done manually in /etc/fstab. To quickly summarize, we recommend the XFS or EXT4 filesystem type for MongoDB data when using MMAPv1 or RocksDB storage engines, and XFS ONLY when using WiredTiger. For all filesystems, using the mount options rw,noatime will reduce some activity.
- NUMA disabling or interleaving. Tuned does not handle NUMA settings and these still need to be handled via the MongoDB init script or the BIOS on/off switch.
- Linux ulimits. Tuned does not set Linux ulimit settings. However, Percona Server for MongoDB RPM packages do this for you at startup! See LimitNOFILE and LimitNPROC in /usr/lib/systemd/system/mongod.service for more information.
NTP server. Tuned does not handle installation of RPM packages or enabling of services. You will need to install the ntp package and enable and start the ntpd service manually:
sudo yum install ntp sudo systemctl enable ntpd sudo systemctl start ntpd
The installation of this profile is as simple as checking out the repository with a Git command and then running sudo make enable. Full output here:
$git clonehttps://github.com/Percona-Lab/tuned-percona-mongodb $cdtuned-percona-mongodb $sudo makeenable if[-d/etc/tuned];then cp-dpR percona-mongodb/etc/tuned/percona-mongodb; echo"### 'tuned-percona-mongodb' is installed. Enable with 'make enable'."; else echo"### ERROR: cannot find tuned config dir at /etc/tuned!"; exit1; ### 'tuned-percona-mongodb' is installed. Enable with 'make enable'. tuned-adm profile percona-mongodb tuned-adm active Current active profile:percona-mongodb
In the example above, you can see that percona-mongodb is now the actively tuned profile on the system (mentioned on the last output line).
The tuned profile files are installed to /etc/tuned/percona-mongodb, as seen here:
$ls-alh/etc/tuned/percona-mongodb/*.* -rwxrwxr-x.1root root677Nov2220:00percona-mongodb.sh -rw-rw-r--.1root root1.4KNov2220:00tuned.conf
Let’s check that the deadline i/o scheduler is now the current scheduler on any disk that isn’t /dev/sda (sdb used below):
Transparent huge pages should be disabled (it is!):
$cat/sys/kernel/mm/transparent_hugepage/enabled always madvise[never] $cat/sys/kernel/mm/transparent_hugepage/defrag always madvise[never]
Block-device readahead should be 32 (16 kb) on /dev/sdb (looks good!):
That was easy!
To uninstall the profile, run Sudo Make Uninstall in the github checkout directory:
if[-d/etc/tuned/percona-mongodb];then echo"### Disabling tuned profile 'tuned-percona-mongodb'"; echo"### Changing tuned profile to 'latency-performance', adjust if necessary after!"; tuned-adm profile latency-performance; tuned-adm active; else echo"tuned-percona-mongodb profile not installed!"; ### Disabling tuned profile 'tuned-percona-mongodb' ### Changing tuned profile to 'latency-performance', adjust if necessary after! Current active profile:latency-performance if[-d/etc/tuned/percona-mongodb];thenrm-rf/etc/tuned/percona-mongodb;fi
Note: The uninstallation will enable the latency-performance tuned profile, change this after the uninstall if needed
To confirm the uninstallation, let’s check if the block device readahead is set back to default (256/128kb):
$sudo blockdev--getra/dev/sdb 256
So far, Tuned shows a lot of promise for tuning Linux for MongoDB, providing a single, consistent interface for tuning the Linux operating system. In the future, I would like to see the documentation for tuned improve. However, its simplicity makes the need for documentation rarely necessary.
As mentioned, after applying tuned-percona-mongodb, you still need to configure an NTP server, NUMA (in some cases) and the filesystem type+tunings manually. The majority of the time, effort and room for mistakes are greatly reduced using this method.
If you have any issues with this profile for tuning Linux for MongoDB or have any questions, please create a Github issue at this URL.