Tools for Performance Evaluation in Linux

Learning how to evaluate the performance of your system is critical to scaling up your web services to a large number of users. This article will discuss a few tools that will help you with evaluating the performance of your Linux server and web applications.

Linux Command Line Tools

The Linux command line offers an excellent set of tools to help you monitor the performance of your server. When evaluating the performance of your web application you need to understand the impact your application has on the server you've deployed it on. It is important to understand where a potential bottleneck exists for your application.

There are many different factors that influence an applications performance. A good place to start is to investigate the CPU utilization and IO (disk and network) of your server. Below are a few examples of tools to start to dissect performance issues in Linux.

top

The Linux command top provides real-time view of the running system. The command is helpful for displaying high utilizing resources. To run the command simply type the name:

$ top

The output of the screen will look something like this:


top - 18:38:09 up  2:13,  3 users,  load average: 0.57, 0.13, 0.04
Tasks:  83 total,   2 running,  81 sleeping,   0 stopped,   0 zombie
Cpu(s): 49.7%us, 39.3%sy,  0.0%ni,  0.0%id,  0.0%wa,  0.0%hi, 11.0%si,  0.0%st
Mem:   1019452k total,   186040k used,   833412k free,    12000k buffers
Swap:        0k total,        0k used,        0k free,    99032k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND           
 3854 ec2-user  20   0 88776 8012 4956 R 21.0  0.8   0:03.69 ab                 
 3871 apache    20   0  334m  10m 6252 S 14.3  1.1   0:00.55 httpd              
 3868 apache    20   0  334m  10m 6252 S 14.0  1.1   0:00.55 httpd              
 3869 apache    20   0  334m  10m 6252 S 14.0  1.1   0:00.55 httpd              
 3870 apache    20   0  334m  10m 6252 S 14.0  1.1   0:00.55 httpd              
 3873 apache    20   0  334m  10m 6252 S  5.7  1.1   0:00.17 httpd              
 3874 apache    20   0  334m  10m 6252 S  0.7  1.1   0:00.02 httpd              
 3875 apache    20   0  334m  10m 6252 S  0.7  1.1   0:00.02 httpd              
    1 root      20   0 19612 2416 2092 S  0.0  0.2   0:00.78 init               
    2 root      20   0     0    0    0 S  0.0  0.0   0:00.00 kthreadd           
    3 root      20   0     0    0    0 S  0.0  0.0   0:00.03 ksoftirqd/0        
    5 root       0 -20     0    0    0 S  0.0  0.0   0:00.00 kworker/0:0H       
    6 root      20   0     0    0    0 S  0.0  0.0   0:00.00 kworker/u30:0      
    7 root      20   0     0    0    0 S  0.0  0.0   0:00.07 rcu_sched          
    8 root      20   0     0    0    0 S  0.0  0.0   0:00.00 rcu_bh             
    9 root      RT   0     0    0    0 S  0.0  0.0   0:00.00 migration/0        
   10 root       0 -20     0    0    0 S  0.0  0.0   0:00.00 khelper

Explanation of Columns:

PID – process ID
USER – username for the owner of each process
PR – process priority (RT means a Real Time priority class – used for system processes)
NI – priority set by nice utility
VIRT – the amount of virtual memory used by a process: code, data and shared libraries plus pages that have been swapped out
RES – the resident part of a process – how much of it resides in the physical memory (non-swapped memory)
SHR – shows you the size of potentially shared memory segments for a process
S – the current state of each process
%CPU – percentage of the time shares CPU spends running a particular process
%MEM – percentage of the physical memory of your system which is used by each process
%TIME+ – total time CPUs spent running each process
COMMAND – a command used to initiate each process.

This top interactive mode allows you to control the output of top and even send signals to kill a process.

From this example, we can see that the process 'ab' (which is the Apache benchmark tool) is utilizing 21% of the CPU and 0.8% of the memory.

vmstat

vmstat (virtual memory statistics) is a monitoring utility, which also provides information about block IO and CPU activity in addition to memory.

vmstat provides a number of values and is typically called with one or two parameters. For example:

$ vmstat 1 5

The first argument specifies a sampling period of 1 second, the second argument instructs vmstat to stop after 5 measurements.

The first line of the report below will contain the average values since the last reboot. The additional lines provide information based on the sampling period.

[ec2-user@ip-172-30-0-240 ~]$ vmstat 1 5
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu-----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 1  0      0 1869728  11612 112776    0    0    16     2    8   19  0  0 99  0  0	
 0  0      0 1869756  11612 112776    0    0     0     0   13   22  1  0 99  0  0	
 0  0      0 1869752  11612 112776    0    0     0     0    9   11  0  0 100  0  0	
 0  0      0 1869748  11612 112776    0    0     0     0   12   18  0  0 100  0  0	
 1  0      0 1869748  11612 112776    0    0     0     0    9    8  0  0 100  0  0	
[ec2-user@ip-172-30-0-240 ~]$

Explanation of Individual Values: (Source man vmstat)

Procs
- r: The number of processes waiting for run time.
- b: The number of processes in uninterruptible sleep.
Memory
- swpd: the amount of virtual memory used.
- free: the amount of idle memory.
- buff: the amount of memory used as buffers.
- cache: the amount of memory used as cache.
- inact: the amount of inactive memory. (-a option)
- active: the amount of active memory. (-a option)
Swap
- si: Amount of memory swapped in from disk (/s).
- so: Amount of memory swapped to disk (/s).
IO
- bi: Blocks received from a block device (blocks/s).
- bo: Blocks sent to a block device (blocks/s).
System
- in: The number of interrupts per second, including the clock.
- cs: The number of context switches per second.
CPU
- These are percentages of total CPU time.
- us: Time spent running non-kernel code. (user time, including nice time)
- sy: Time spent running kernel code. (system time)
- id: Time spent idle. Prior to Linux 2.5.41, this includes IO-wait time.
- wa: Time spent waiting for IO. Prior to Linux 2.5.41, included in idle.
- st: Time stolen from a virtual machine.

A detailed discussion of using vmstat to diagnose Linux performance can be found here Linux.

Additional System Tools

A comprehensive list of tools can be found here [1]. Depending on what aspect of the Linux OS you want to explore, an appropriate tool exists.

Application Benchmarks

Below is a list of common application benchmark tools.

Apache Benchmark Tool

The Apache Benchmark ab [2] provides a convenient way to stress test your web server. The tool is included with the default installation of Apache.

An example of using ab to send 1000 requests, 10 at a time to the URL http://example.com/ is shown below:

$ ab -c 10 –n 1000 http://example.com/

The output of the tool will look similar to this:


Document Path:          /
Document Length:        25951 bytes

Concurrency Level:      10
Time taken for tests:   11.072 seconds
Complete requests:      1000
Failed requests:        0
Write errors:           0
Total transferred:      26253056 bytes
HTML transferred:       25995542 bytes
Requests per second:    90.32 [#/sec] (mean)
Time per request:       110.722 [ms] (mean)
Time per request:       11.072 [ms] (mean, across all concurrent requests)
Transfer rate:          2315.51 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:       32   33   0.4     33      39
Processing:    67   77  22.5     68     141
Waiting:       34   35   0.9     35      44
Total:        100  110  22.4    101     174

From the tests, we see that Apache is able to serve 90 requests per second to the machine running the ab tool. We also have statistics on the min, mean, median, and max time to send a request.

In order to effectively measure the performance of your server we should execute the ab tool from a separate machine. The ab tool can utilize a decent amount of CPU, which may interfere with the performance of your web server.

Using ab in AWS

When using ab in AWS with multiple instances, we should take advantage of Amazon's Virtual Private Cloud (VPC) VPC. You actually have a VPC setup by default and to communicate with other instances, you simply use the private IP address assigned to it. Each instance by default will have a public IP address (which you ssh into) and the private IP address that is only accessible from inside the VPC. When load testing your server on AWS you should always communicate with the private IP address of your server, to avoid contention from other networks.