This article discusses iostat and how it can identify I/O-subsystem and CPU bottlenecks. iostat works by sampling the kernel’s address space and extracting data from various counters that are updated every clock tick (1 clock tick = 10 milliseconds [ms]). The results — covering CPU, and I/O subsystem activity — are reported as per-second rates or as absolute values for the specified interval. The iostat tool in a part of the sysstat package, to install it simply type:

sudo yum install sysstat

or

sudo apt-get install sysstat

Syntax

Normally, iostat is issued with both an interval and a count specified, with the report sent to standard output or redirected to a file. To get up and running quickly you can try the following command which will output data in Kilobytes every 5 seconds


# iostat -k

Linux 2.6.el6.x86_64 (jahshaka) 11/21/2014 _x86_64_ (2 CPU)

avg-cpu: %user %nice %system %iowait %steal %idle
3.73 0.00 0.49 0.47 0.06 95.25

Device: tps kB_read/s kB_wrtn/s kB_read kB_wrtn
vda 8.84 20.17 89.07 39055085 172488380

The command syntax appears below:

iostat [-c] [-d] [Drives] [Interval [Count]] [-k]

The -d flag causes iostat to provide only disk statistics for all drives. The -c flag causes iostat to provide only CPU statistics.

NOTE: The -c and -d options are mutually exclusive.

If you specify one or more drives, the output is limited to those drives. Multiple drives can be specified; separate them in the syntax with spaces.

You can specify a time in seconds for the interval between records to be included in the reports. The initial record contains statistics for the time since system boot. Succeeding records contain data for the preceding interval. If no interval is specified, a single record is generated.

If you specify an interval, the count of the number of records to be included in the report can also be specified. If you specify an interval without a count, iostat will continue running until it is killed.

iostat -k 5

CPU statistics in the iostat output

The first report generated by the iostat command is the CPU Utilization Report. For multiprocessor systems, the CPU values are global averages among all processors.

iostat -c -k

The report has the following format:

%user
Show the percentage of CPU utilization that occurred while executing at the user level (application). A UNIX process can execute in user or system mode. When in user mode, a process executes within its own code and does not require kernel resources.

%nice
Show the percentage of CPU utilization that occurred while executing at the user level with nice priority.

%system
Show the percentage of CPU utilization that occurred while executing at the system level (kernel). This includes CPU resource consumed by kernel processes (kprocs) and others that need access to kernel resources. For example, the reading or writing of a file requires kernel resources to open the file, seek a specific location, and read or write data. A UNIX process accesses kernel resources by issuing system calls.

%steal
Show the percentage of time spent in involuntary wait by the virtual CPU or CPUs while the hypervisor was servicing another virtual processor.

%idle
Show the percentage of time that the CPU or CPUs were idle and the system did not have an outstanding disk I/O request. If there are no processes on the run queue, the system dispatches a special kernel process called wait. On most systems, the wait process ID (PID) is 516.

%iowait
Show the percentage of time that the CPU or CPUs were idle during which the system had an outstanding disk I/O request. The iowait state is different from the idle state in that at least one process is waiting for local disk I/O requests to complete. Unless the process is using asynchronous I/O, an I/O request to disk causes the calling process to block (or sleep) until the request is completed. Once a process’s I/O request completes, it is placed on the run queue.

Notes

For multiprocessor feedback you can also run mpstat. The mpstat command display activities for each available processor, processor 0 being the first one. Global average activities among all processors are also reported. The mpstat command can be used both on SMP and UP machines, but in the latter, only global average activities will be printed.:

# mpstat -P ALL

Analyzing the data

Typically, the CPU is pacing (the system is CPU bound) if the sum of user and system time exceeds 90 percent of CPU resource on a single-user system or 80 percent on a multi-user system. This condition means that the CPU is the limiting factor in system performance.

The ratio of user to system mode is determined by workload and is more important when tuning an application than when evaluating performance.

A key factor when evaluating CPU performance is the size of the run queue (provided by the vmstat command). In general, as the run queue increases, users will notice degradation (an increase) in response time.

A high iowait percentage indicates the system has a memory shortage or an inefficient I/O subsystem configuration. Understanding the I/O bottleneck and improving the efficiency of the I/O subsystem require more data than iostat can provide. However, typical solutions might include:

  • limiting the number of active logical volumes and file systems placed on a particular physical disk (The idea is to balance file I/O evenly across all physical disk drives.)
  • spreading a logical volume across multiple physical disks (This is useful when a number of different files are being accessed.)
  • creating multiple JFS logs for a volume group and assigning them to specific file systems (This is beneficial for applications that create, delete, or modify a large number of files, particularly temporary files.)
  • backing up and restoring file systems to reduce fragmentation (Fragmentation causes the drive to seek excessively and can be a large portion of overall response time.)
  • adding additional drives and rebalancing the existing I/O subsystem

On systems running a primary application, high I/O wait percentage may be related to workload. In this case, there may be no way to overcome the problem. On systems with many processes, some will be running while others wait for I/O. In this case, the iowait can be small or zero because running processes “hide” wait time. Although iowait is low, a bottleneck may still limit application performance. To understand the I/O subsystem thoroughly, examine the statistics in the next section.

Disk statistics in the iostat output

The disk statistics portion of the iostat output provides a breakdown of I/O usage. This information is useful in determining whether a physical disk is limiting performance.

iostat -d -k

Devices
The Devices: column shows the names of the physical volumes. They can vary based on your system between disk, vda, sda or other values, followed by a number.

A drive is active during data transfer and command processing, such as seeking to a new location. The disk-use percentage is directly proportional to resource contention and inversely proportional to performance. As disk use increases, performance decreases and response time increases. In general, when a disk’s use exceeds 70 percent, processes are waiting longer than necessary for I/O to complete because most UNIX processes block (or sleep) while waiting for their I/O requests to complete.

tps
Indicate the number of transfers per second that were issued to the device. A transfer is an I/O request to the device. Multiple logical requests can be combined into a single I/O request to the device. A transfer is of indeterminate size.

Blk_read/s
Indicate the amount of data read from the device expressed in a number of blocks per second. Blocks are equivalent to sectors with kernels 2.4 and later and therefore have a size of 512 bytes. With older kernels, a block is of indeterminate size.

Blk_wrtn/s
Indicate the amount of data written to the device expressed in a number of blocks per second.

Blk_read
The total number of blocks read.

Blk_wrtn
The total number of blocks written.

Additional modes

You can also displays statistics for specific device only with -p and the device name arguments. With -N (Uppercase) parameter you can view only LVM statistics.

Analyzing the data

Taken alone, there is no unacceptable value for any of the preceding fields because statistics are too closely related to application characteristics, system configuration, and types of physical disk drives and adapters. Therefore, when evaluating data, you must look for patterns and relationships. The most common relationship is between disk utilization and data transfer rate.

For example, if an application reads and writes sequentially, you should expect a high disk-transfer rate when you have a high disk-busy rate. (NOTE: Kb_read and Kb_wrtn can confirm an understanding of an application’s read and write behavior but they provide no information on the data access patterns.)

Generally you do not need to be concerned about a high disk-busy rate as long as the disk-transfer rate is also high. However, if you get a high disk-busy rate and a low data-transfer rate, you may have a fragmented logical volume, file system, or individual file.

Conclusions

The primary purpose of the iostat tool is to detect I/O bottlenecks by monitoring the disk utilization. iostat can also be used to identify CPU problems, assist in capacity planning, and provide insight into solving I/O problems. Armed with both vmstat and iostat, you can capture the data required to identify performance problems related to CPU, memory, and I/O subsystems.