{{announcement.body}}
{{announcement.title}}

Using DTrace on Oracle Linux

DZone 's Guide to

Using DTrace on Oracle Linux

This article introduces a DTrace tool that can be used to gather this information on the production environment.

Free Resource

Have you ever want to know the following application information on the production system?

  • Call stack when the function is called
  • The elapsed time used by the functions
  • The values of parameters passed to the functions
  • The return values of the functions
  • The functions’ call trace
  • OS behaviors

Typically, this information can be retrieved by enabling debug logging or attaching a debugger to the application. However, this method couldn’t be done when applications are running on the production system. Fortunately, this article introduces a DTrace tool that can be used to gather this information on the production environment.

Overview

DTrace is an exhaustive dynamic tracing toolkit officially introduced in Sun Solaris 10 in January 2005. It is designed to dynamically and safely inspect the behavior of the operating system’s kernel and applications on any environments including development, testing, and production. Therefore, it is a very efficient tool to investigate issues inside the operating system and running applications. At the time of this writing, DTrace has been ported to many operating systems including Oracle Linux, macOS, FreeBSD, and Windows 10 insider builds. 

This article only focuses on using DTrace on Oracle Linux.

DTrace on Oracle Linux

Oracle acquired Sun Microsystems in 2010. On December 12, 2012, Oracle officially announced the general availability of DTrace for Oracle Linux. Currently, DTrace supports the Unbreakable Enterprise Kernel (UEK) release 4 or release 5. Support for DTrace on UEK3 has been discontinued. The following table is from Oracle Linux and Unbreakable Enterprise Kernel (UEK) Releases representing the associations between support UEKs and Oracle Linux releases.

oracle linux

In this article, Oracle Linux 7 Update 7 (UEK5) is used for demonstration.

The following command can be used to install DTrace on Oracle Linux 7.

Shell
 




xxxxxxxxxx
1


 
1
# yum install dtrace-utils 



Then, you can run the following command to test DTrace. 

Shell
 




xxxxxxxxxx
1


 
1
# dtrace -ln 'pid$target:::entry' -c "ls"
2
   ID   PROVIDER            MODULE                          FUNCTION NAME
3
13021    pid2278 ld-linux-x86-64.so.2                  print_statistics entry
4
13022    pid2278 ld-linux-x86-64.so.2                          init_tls entry
5
13023    pid2278 ld-linux-x86-64.so.2         is_trusted_path_normalize entry
6
13024    pid2278 ld-linux-x86-64.so.2                               oom entry
7
13025    pid2278 ld-linux-x86-64.so.2                   update_get_addr entry
8
13026    pid2278 ld-linux-x86-64.so.2               check_one_fd.part.0 entry



The above DTrace command uses the pid provider to probe the ls command. For more information, please refer to the Oracle Linux DTrace Guide

The D Programming Language

To use DTrace, users need to create a D program with the D programming language for defining probe locations, conditions, and actions. Then, the source files will be compiled and sent into the operating system kernel for execution by DTrace. The syntax of the D programming language is very similar to C with its built-in variables and functions.

D programs consist of a set of clauses executed in order. Each clause describing one or more probes to be enabled, an optional predicate to check a condition before fired, and an optional set of actions to perform when the probe is fired.

D
 




xxxxxxxxxx
1


 
1
Probe descriptions
2
/Predicate/
3
{
4
   Action statements
5
/


Probe descriptions

Probe descriptions list probes enabled in the clause. Multiple probe descriptions are separated by commas. Each probe can be identified by using a unique probe ID or a human-readable name. The human-readable name comprises of four parts (provider:module:function:name). The module, function, and name fields can be blank to match any of its probes. Probe descriptions also support pattern matching characters, such as * (match any string), and ? (match any single character).

To list all available probes, the following command can be used:

Shell
 




xxxxxxxxxx
1
17


 
1
# dtrace -l 
2
   ID   PROVIDER            MODULE                          FUNCTION NAME
3
    1     dtrace                                                     BEGIN
4
    2     dtrace                                                     END
5
    3     dtrace                                                     ERROR
6
7
   18    syscall           vmlinux                              read entry
8
   19    syscall           vmlinux                              read return
9
   20    syscall           vmlinux                             write entry
10
11
 4865   lockstat            dm_mod                        run_io_job spin-release
12
 4866   lockstat            dm_mod                       dm_kvzalloc spin-release
13
 4867   lockstat            dm_mod                   dm_stats_create spin-release
14
 4868       perf            dm_mod                       map_request block_rq_remap
15
16
17095    pid2278   libselinux.so.1                security_av_string entry
17
17096    pid2278   libselinux.so.1               print_access_vector entry


Predicate

A predicate is an expression enclosed in slashes. It is evaluated at probe firing time to determine if the associated actions should be executed. If the predicate evaluates to true (non-zero value), the action statements will be executed. Otherwise, the probe firing will be ignored.

Action Statements

The actions are listed as a series of statements enclosed in braces { } following the probe name or predicate. Each statement ends with a semicolon (;). D programming language is derived from a large subset of C combined with a special set of functions and variables. It has built-in support for associative array and string variables. Unlike C, it doesn’t support conditional (if) and loop (for) control flow but it supports conditional expressions through the ? and : operators. Moreover, the explicit variable declaration is not required.

Sample Usage

This section demonstrates a DTrace sample program used to monitor behaviors of an application running on Oracle Linux 7. The application uses Elektron-SDK from Refinitiv to consume real-time financial market activities (Quotes, Trades) from the market data platform.  In the Elektron-SDK, the market data is represented in refresh and update messages. 

Measuring update rate per second (UpdateRate.d)

One method to determine the performance of this Elektron-SDK application is measuring the number of update messages that the applications can handle per second. The more update rate the application can handle, the higher performance the application has.

DTrace can be used to determine the update rate that the Elektron-SDK application can handle at run time. The program looks like:

D
 




xxxxxxxxxx
1
18


 
1
BEGIN
2
{
3
 updateCount = 0;
4
}
5
pid$1::*ItemCallbackClient*processRefreshMsg*:entry,
6
pid$1::*ItemCallbackClient*processUpdateMsg*:entry
7
{
8
 updateCount++;
9
}
10
profile:::tick-1sec
11
/updateCount > 0/
12
{
13
  printf("%Y: Number of Updates per second: %d\n", walltimestamp, updateCount);
14
  @average["Average Update Rate"] = avg(updateCount);
15
  @max["Max Update Rate"] = max(updateCount);
16
  @min["Min Update Rate"] = min(updateCount);
17
  updateCount = 0;
18
}


This program has three clauses. The first clause uses the BEGIN probe which is fired before any other probes. When fired, it sets the global updateCount variable to zero.

The second clause uses probes from the pid provider. $1 after the pid provider is a built-in variable representing the first parameter passed to the DTrace command when running this program. It is a process ID of the application. This clause sets probes at the entry point of the ItemCallbackClient::processRefreshMsg and ItemCallbackClient::processUpdateMsg methods in the Elektron-SDK. These methods are used to process refresh and update messages. When fired, it increases the value of the global updateCount variable by one. This updateCount variable represents a number of the processed refresh and update messages.

The last clause uses a timer probe from the profile provider. This probe will be fired every second. This clause has a predicate so it is fired when the value of the updateCount variable is more than zero. When fired, the value of the updateCount will be printed to the output. It also uses the aggregate functions to calculate the average, maximum, and minimum of the updateCount values. 

Use the following command to run this DTrace program. 

Shell
 




xxxxxxxxxx
1


1
dtrace -q -s UpdateRate.d <pid> 


<pid> is the process ID of the running consuming application.

The output will show the number of messages that the application can process per second. When the application exits with Ctrl+c, it will show the average, maximum, and minimum numbers of messages processed per second. Cons100 is the name of the running application. 

Shell
 




xxxxxxxxxx
1
16


1
# dtrace -q -s UpdateRate.d `pgrep Cons100`
2
2020 Apr 13 11:35:16: Number of Updates per second: 9847
3
2020 Apr 13 11:35:17: Number of Updates per second: 10770
4
2020 Apr 13 11:35:18: Number of Updates per second: 9517
5
2020 Apr 13 11:35:19: Number of Updates per second: 10469
6
2020 Apr 13 11:35:20: Number of Updates per second: 9626
7
2020 Apr 13 11:35:21: Number of Updates per second: 9104
8
2020 Apr 13 11:35:22: Number of Updates per second: 11270
9
2020 Apr 13 11:35:23: Number of Updates per second: 9875
10
2020 Apr 13 11:35:24: Number of Updates per second: 10139
11
12
2020 Apr 13 11:35:42: Number of Updates per second: 9980
13
 
          
14
  Average Update Rate                                           10021
15
  Max Update Rate                                               11270
16
  Min Update Rate                                                9104



Summary

DTrace is a tool for dynamically inspecting the behaviors of the operating system’s kernel and applications in any environment, especially the production environment. Currently, DTrace is supported by many operating systems, such as Oracle Solaris, Oracle Linux, macOS, FreeBSD, and Windows 10 insider builds. To use DTrace, we need to use the D programming language to create DTrace programs defining probe descriptions, predicates, and action statements. Then, run the programs with the DTrace command to probe applications or operating systems.

References

  1. Dtrace.org. 2008. Aggregations - Dynamic Tracing Guide. [online] Available at: http://dtrace.org/guide/chp-aggs.html [Accessed 17 April 2020].
  2. Coter, S., 2018. Oracle Linux And Unbreakable Enterprise Kernel (UEK) Releases. [online] Blogs.oracle.com. Available at: https://blogs.oracle.com/scoter/oracle-linux-and-unbreakable-enterprise-kernel-uek-releases [Accessed 17 April 2020].
  3. En.wikipedia.org. n.d. Dtrace. [online] Available at: https://en.wikipedia.org/wiki/DTrace [Accessed 17 April 2020].
  4. Koch, Z., 2012. Announcement: Dtrace For Oracle Linux General Availability. [online] Blogs.oracle.com. Available at: https://blogs.oracle.com/linux/announcement%3a-dtrace-for-oracle-linux-general-availability [Accessed 17 April 2020].
  5. Marsden, G., 2018. Tracing Linux Networking With Dtrace On Oracle Linux. [online] Blogs.oracle.com. Available at: https://blogs.oracle.com/linux/tracing-linux-networking-with-dtrace-on-oracle-linux-v2 [Accessed 17 April 2020].
  6. Oracle.com. n.d. Oracle Linux Dtrace. [online] Available at: https://www.oracle.com/linux/downloads/linux-dtrace.html [Accessed 17 April 2020].
  7. Docs.oracle.com. 2019. Oracle® Linux Dtrace Guide. [online] Available at: https://docs.oracle.com/en/operating-systems/oracle-linux/dtrace-guide [Accessed 17 April 2020].
  8. Docs.oracle.com. 2019. Oracle® Linux Dtrace Tutorial. [online] Available at: https://docs.oracle.com/en/operating-systems/oracle-linux/dtrace-tutorial/ [Accessed 17 April 2020].
  9. Pulpaka, H., 2019. Dtrace On Windows. [online] Microsoft Tech Community. Available at: https://techcommunity.microsoft.com/t5/windows-kernel-internals/dtrace-on-windows/ba-p/362902 [Accessed 17 April 2020].
Topics:
debugging, dtrace, linux, network monitoring, oracle linux 7, performance, tools

Opinions expressed by DZone contributors are their own.

{{ parent.title || parent.header.title}}

{{ parent.tldr }}

{{ parent.urlSource.name }}