Tools for Performance Evaluation in Linux
Learning how to evaluate the performance of your operating system is critical to scaling up your web services to a large number of users. This article will discuss a few tools that will help you with evaluating the performance of your Linux server and web applications.
Contents
Linux Command Line Tools
The Linux command line offers an excellent set of tools to help you monitor the performance of your server. When evaluating the performance of your web application you need to understand the impact your application has on the server you've deployed it on. It is important to understand where a potential bottleneck exists for your application and the tools listed below will help you quickly gain insight into the problem.
The goal is to determine what resources are limiting the performance of my application. There are many different factors that influence an applications performance. A good place to start is to investigate the CPU utilization and IO (disk and network) of your server. Below are a few examples of two tools to start to dissect performance issues in Linux.
top
The Linux command top provides real-time view of the running system. The command is helpful for displaying high utilizing resources. To run the command simply type the name:
$ top
The output of the screen will look something like this:
top - 18:38:09 up 2:13, 3 users, load average: 0.57, 0.13, 0.04
Tasks: 83 total, 2 running, 81 sleeping, 0 stopped, 0 zombie
Cpu(s): 49.7%us, 39.3%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi, 11.0%si, 0.0%st
Mem: 1019452k total, 186040k used, 833412k free, 12000k buffers
Swap: 0k total, 0k used, 0k free, 99032k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
3854 ec2-user 20 0 88776 8012 4956 R 21.0 0.8 0:03.69 ab
3871 apache 20 0 334m 10m 6252 S 14.3 1.1 0:00.55 httpd
3868 apache 20 0 334m 10m 6252 S 14.0 1.1 0:00.55 httpd
3869 apache 20 0 334m 10m 6252 S 14.0 1.1 0:00.55 httpd
3870 apache 20 0 334m 10m 6252 S 14.0 1.1 0:00.55 httpd
3873 apache 20 0 334m 10m 6252 S 5.7 1.1 0:00.17 httpd
3874 apache 20 0 334m 10m 6252 S 0.7 1.1 0:00.02 httpd
3875 apache 20 0 334m 10m 6252 S 0.7 1.1 0:00.02 httpd
1 root 20 0 19612 2416 2092 S 0.0 0.2 0:00.78 init
2 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kthreadd
3 root 20 0 0 0 0 S 0.0 0.0 0:00.03 ksoftirqd/0
5 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/0:0H
6 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kworker/u30:0
7 root 20 0 0 0 0 S 0.0 0.0 0:00.07 rcu_sched
8 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcu_bh
9 root RT 0 0 0 0 S 0.0 0.0 0:00.00 migration/0
10 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 khelper
Explanation of Columns:
- PID – process ID
- USER – username for the owner of each process
- PR – process priority (RT means a Real Time priority class – used for system processes)
- NI – priority set by nice utility
- VIRT – the amount of virtual memory used by a process: code, data and shared libraries plus pages that have been swapped out
- RES – the resident part of a process – how much of it resides in the physical memory (non-swapped memory)
- SHR – shows you the size of potentially shared memory segments for a process
- S – the current state of each process
- %CPU – percentage of the time shares CPU spends running a particular process
- %MEM – percentage of the physical memory of your system which is used by each process
- %TIME+ – total time CPUs spent running each process
- COMMAND – a command used to initiate each process.
This top interactive mode allows you to control the output of top and even send signals to kill a process.
From this example, we can see that the process 'ab' (which is the Apache benchmark tool) is utilizing 21% of the CPU and 0.8% of the memory.
vmstat
vmstat (virtual memory statistics) is a monitoring utility, which also provides information about block IO and CPU activity in addition to memory.
vmstat provides a number of values and is typically called with one or two parameters. For example:
$ vmstat 1 5
The first argument specifies a sampling period of 1 second, the second argument instructs vmstat to stop after 5 measurements.
The first line of the report below will contain the average values since the last reboot. The additional lines provide information based on the sampling period.
[ec2-user@ip-172-30-0-240 ~]$ vmstat 1 5 procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu----- r b swpd free buff cache si so bi bo in cs us sy id wa st 1 0 0 1869728 11612 112776 0 0 16 2 8 19 0 0 99 0 0 0 0 0 1869756 11612 112776 0 0 0 0 13 22 1 0 99 0 0 0 0 0 1869752 11612 112776 0 0 0 0 9 11 0 0 100 0 0 0 0 0 1869748 11612 112776 0 0 0 0 12 18 0 0 100 0 0 1 0 0 1869748 11612 112776 0 0 0 0 9 8 0 0 100 0 0 [ec2-user@ip-172-30-0-240 ~]$
Explanation of Individual Values:
(Source man vmstat)
- Procs
- r: The number of processes waiting for run time.
- b: The number of processes in uninterruptible sleep.
- Memory
- swpd: the amount of virtual memory used.
- free: the amount of idle memory.
- buff: the amount of memory used as buffers.
- cache: the amount of memory used as cache.
- inact: the amount of inactive memory. (-a option)
- active: the amount of active memory. (-a option)
- Swap
- si: Amount of memory swapped in from disk (/s).
- so: Amount of memory swapped to disk (/s).
- IO
- bi: Blocks received from a block device (blocks/s).
- bo: Blocks sent to a block device (blocks/s).
- System
- in: The number of interrupts per second, including the clock.
- cs: The number of context switches per second.
- CPU
- These are percentages of total CPU time.
- us: Time spent running non-kernel code. (user time, including nice time)
- sy: Time spent running kernel code. (system time)
- id: Time spent idle. Prior to Linux 2.5.41, this includes IO-wait time.
- wa: Time spent waiting for IO. Prior to Linux 2.5.41, included in idle.
- st: Time stolen from a virtual machine.
Additional System Tools
A comprehensive list of tools can be found here [1]. Depending on what aspect of the Linux OS you want to explore, an appropriate tool exists.
Application Benchmarks
Below is a list of common application benchmark tools.
Apache Benchmark Tool
The Apache Benchmark ab [2] provides a convenient way to stress test your web server. The tool is included with the default installation of Apache.
An example of using ab to send 1000 requests, 10 at a time to the URL http://example.com/ is shown below:
$ ab -c 10 –n 1000 http://example.com/
The output of the tool will look similar to this:
Document Path: /
Document Length: 25951 bytes
Concurrency Level: 10
Time taken for tests: 11.072 seconds
Complete requests: 1000
Failed requests: 0
Write errors: 0
Total transferred: 26253056 bytes
HTML transferred: 25995542 bytes
Requests per second: 90.32 [#/sec] (mean)
Time per request: 110.722 [ms] (mean)
Time per request: 11.072 [ms] (mean, across all concurrent requests)
Transfer rate: 2315.51 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 32 33 0.4 33 39
Processing: 67 77 22.5 68 141
Waiting: 34 35 0.9 35 44
Total: 100 110 22.4 101 174
DISCUSS RESULTS HERE
MENTION USE OF DIFFERENT SERVER FOR TESTING
In order to effectively measure the performance of your server we should execute the ab tool from a separate machine. The ab tool uses a decent amount of CPU and that may hinder the performance of your web
Using ab in AWS
When using ab in AWS with multiple instances, we should take advantage of Amazon's Virtual Private Cloud (VPC) VPC. You actually have a VPC setup by default and to communicate with other instances, you simply use the private IP address assigned to it. Each instance by default will have a public IP address (which you ssh into) and the private IP address that is only accessible from inside the VPC. When load testing your server on AWS you should always use the VPC, to avoid contention from other networks.
