Performance Monitoring in Linux

There are few useful tools that can help find out a bottleneck of your Linux box performance.

What to monitor first?

The system load is a measure of the amount of work that a computer system performs. You can use this command to read system load:

uptime

Here is a sample output:

... load average: 1.07, 1.63, 2.81

The three values of load average refer to the past 1, 5, and 15 minutes of system operation. These numbers should be read this way: the number represents how well a single CPU can handle load, thus if the number is 1 or less - it is pretty comfortable (the 4-CPU system works well at load number 4 or less); 1.5 - means at least 50% of load is not handled on time, it is queued for processing and is a subject for attention.

System Monitoring

Real time monitoring can be observed with top and htop commands. Command htop gives you more convenient way of what top does. Particularly it is handy to add two more columns (via 'F2' Setup) related to IO read and IO write.

htop

Processors related statistics with mpstat:

watch -n 1 mpstat

Disk Monitoring

IO can be a one of possible bottleneck of system performance degradation. The tool iotop tracks disk I/O by process, and prints a summary report that is refreshed every second.

iotop

Statistic for IO devices and partitions can be monitored with iostat:

watch -n 1 iostat

Who is waiting and blocked?

It is useful to know how the system load goes across processes, however most interest is related to processes that keep waiting for the operation to complete, thus cause delays. Here is a simple command to get this kind of report every second:

watch -n 1 "(ps aux | awk '\$8 ~ /D/  { print \$0 }')"

Network Monitoring

Intensive network related operation can cause the high load as well. Here is a tool that let you have a better idea of your network traffic utilization - iftop:

iftop

How to quickly reboot you Linux

Here is a quick way to reboot you Linux with kexec command. In debian first install kexec-tools package:

apt-get install kexec-tools

Once installed make your current kernel the one you want to quickly reboot into:

kexec -e

In case you would like disable fast reboot, open file located at /etc/default/kexec and set LOAD_KEXEC to false:

# Defaults for kexec initscript
# sourced by /etc/init.d/kexec and /etc/init.d/kexec-load

# Load a kexec kernel (true/false)
LOAD_KEXEC=true

# Kernel and initrd image
KERNEL_IMAGE="/vmlinuz"
INITRD="/initrd.img"

# If empty, use current /proc/cmdline
APPEND=""

Timing disk read/write performance with dd

There is easy way to measure your partition performance with dd. Here is how to measure write performance by simply coping a file of length 1 Gb:

deby:~# time dd if=/dev/zero of=file bs=1024 count=1000000
1000000+0 records in
1000000+0 records out
1024000000 bytes (1.0 GB) copied, 13.454 s, 76.1 MB/s

real 0m13.696s
user 0m0.240s
sys 0m5.924s

Same way we can read that file back into a memory (/dev/shm):

deby:~# time dd if=file of=/dev/shm/file
2000000+0 records in
2000000+0 records out
1024000000 bytes (1.0 GB) copied, 4.19182 s, 244 MB/s

real 0m4.334s
user 0m0.592s
sys 0m3.740s

Read more about dd command here.

How to defragment XFS

There is an easy way to find out if your XFS partition needs defragmentation. Here is the command (it is part of package xfsprogs):

deby:~# xfs_db -c frag -r /dev/sda10
actual 2903, ideal 2418, fragmentation factor 16.71%

Once you see it is pretty high, e.g. above 40% you would need to issue the following command that defragment the drive:

deby:~# xfs_fsr -v /dev/sda10
/home start inode=0

Suppress kernel talkativness

Kernel log levels (defined in linux/kernel.h):

#define KERN_EMERG    "<0>"  /* system is unusable               */
#define KERN_ALERT    "<1>"  /* action must be taken immediately */
#define KERN_CRIT     "<2>"  /* critical conditions              */
#define KERN_ERR      "<3>"  /* error conditions                 */
#define KERN_WARNING  "<4>"  /* warning conditions               */
#define KERN_NOTICE   "<5>"  /* normal but significant condition */
#define KERN_INFO     "<6>"  /* informational                    */
#define KERN_DEBUG    "<7>"  /* debug-level messages             */

Check the current log level:

deby:~# cat /proc/sys/kernel/printk
7 4 1 7

sysctl.conf

Ensure you have the following in /etc/sysctl.conf:

# The four values in printk denote: 
# console_loglevel, default_message_loglevel,
# minimum_console_loglevel, default_console_loglevel
#
# Uncomment the following to stop low-level messages on console
kernel.printk = 4 4 1 6

These changes become effective after reboot. But you might need changes take place immediately:

echo "4 4 1 6" > /proc/sys/kernel/printk

grub

You can pass quiet parameter during kernel boot in /boot/grub/menu.list in order to suppress logs during boot:

kernel /boot/vmlinuz-2.6.26-2-686 root=/dev/sda1 ro quiet panic=0

Please note that after kernel upgrade the above changes to grub will be lost, so take this into account or add another section to your grub loader as shown below.

# Put static boot stanzas before and/or after AUTOMAGIC KERNEL LIST
title    Debian GNU/Linux, kernel 2.6.26-2-686 (quiet)
root     (hd0,0)
kernel   /boot/vmlinuz-2.6.26-2-686 root=/dev/sda1 ro quiet panic=0
initrd   /boot/initrd.img-2.6.26-2-686

### BEGIN AUTOMAGIC KERNELS LIST
## lines between the AUTOMAGIC KERNELS LIST markers will be modified
## by the debian update-grub script except for the default options 
## below

Timing disk read performance with hdparm

You can benchmark you hard disk performance with hdparm. You need install it first:

apt-get install hdparm

You can perform timings of cache/device reads for benchmark and comparison purposes. For meaningful results, this operation should be repeated 2-3 times on an otherwise inactive system (no other active processes) with at least a couple of megabytes of free memory.

deby:~# hdparm -Tt /dev/sda

/dev/sda:
 Timing cached reads:   858 MB in  2.00 seconds = 428.78 MB/sec
 Timing buffered disk reads:   60 MB in  3.03 seconds =  19.78 MB/sec

This measurement is essentially an indication of the throughput of the processor, cache, and memory of the system under test, as well as how fast the drive can sustain sequential data reads under Linux, without any filesystem overhead. Read more about this tool here.

Optimize EX3 filesystem performance

Suppose your /etc/fstab looks like this:

/dev/sda1 /     ext3    errors=remount-ro        0 1
/dev/sda9 /home ext3    defaults                 0 2

Turning off atime

If atime is on, every time a file is accessed, whether for read or write, a small change is written to the file detailing the last access time. To disable atime on ext3 partitions:

/dev/sda1 /     ext3  errors=remount-ro,noatime,nodiratime 0 1
/dev/sda9 /home ext3  defaults,noatime,nodiratime 0 2

Disable extended user attributes

Most likely you do not need extended user attributes, so disable them.

/dev/sda9 /home ext3  defaults,noatime,nodiratime,nouser_xattr 0 2

Use journal data writeback

By default journal data ordered option is used. According to this report, you can double read performance. You can simply turn on writeback for any filesystem (but root) by adding data=writeback.

/dev/sda1 /     ext3    errors=remount-ro        0 1
/dev/sda9 /home ext3    defaults,noatime,nodiratime,nouser_xattr,data=writeback 0 2

Journal data writeback for root filesystem

If you try to do this with root (/) filesystem you will get an error message during the boot and the root filesystem remains read-only. Remount it read-write:

mount -n -o remount, rw /

You need setup default filesystem option for root:

deby:~# tune2fs -o journal_data_writeback /dev/sda1
deby:~# tune2fs -l /dev/sda1 | grep 'Default mount option'
Default mount options:    journal_data_writeback

Now you can reboot your system so your changes take effect.

Deactivate barriers

Write barriers enforce proper on-disk ordering of journal commits, making volatile disk write caches safe to use, at some performance penalty. Deactivating barriers make sense if you have battery-backed storage only.

/dev/sda9 /home ext3  defaults,noatime,nodiratime,nouser_xattr,data=writeback,barrier=0 0 2

Avoid buffer heads association

Do not attach buffer_heads to file pagecache. This works together with data writeback option.

/dev/sda9 /home ext3  defaults,noatime,nodiratime,data=writeback,barrier=0,nobh 0 2

Extend commit interval

This options let sync all data and metadata every N seconds. The default value is 5 seconds.

/dev/sda9 /home ext3  defaults,noatime,nodiratime,nouser_xattr,data=writeback,barrier=0,nobh,commit=30 0 2

Have a look at more mount options here. Reboot your computer so all fine tunning take place.

How to disable a service in Debian

To disable a service ar runlevel 2 (that is default in Debian), rename its script in /etc/rc2.d directory so that the new name begins with a 'K' and a two-digit number, where the number is the difference between the two-digit number following the 'S' in its current name, and 100. To re-enable the service, rename the script back to its original name beginning with 'S'.

mv /etc/rcS.d/S05bootlogd /etc/rcS.d/K95bootlogd
mv /etc/rcS.d/S43portmap /etc/rcS.d/K57portmap
mv /etc/rcS.d/S44nfs-common /etc/rcS.d/K56nfs-common
mv /etc/rcS.d/S70x11-common /etc/rcS.d/K30x11-common

mv /etc/rc2.d/S12acpid /etc/rc2.d/K88acpid
mv /etc/rc2.d/S20nfs-common /etc/rc2.d/K80nfs-common
mv /etc/rc2.d/S20openbsd-inetd /etc/rc2.d/K80openbsd-inetd
mv /etc/rc2.d/S23ntp /etc/rc2.d/K77ntp
mv /etc/rc2.d/S89atd /etc/rc2.d/K11atd
mv /etc/rc2.d/S99stop-bootlogd /etc/rc2.d/K01stop-bootlogd

The other way is to use rcconf or update-rc.d tool

rcconf --off my-service-name
update-rc.d my-service-name disable

Read more about different services here.