Linux performance tuning
Simple performance tuning tips
Cpu(s): 8.6% us, 0.3% sy, 0.0% ni, 90.5% id, 0.6% wa, 0.0% hi, 0.0% si
- us: percent cpu being used by userland code
- sy: percent cpu being used by kernelspace code
- ni: like "us" but related to "niced" processes
- id: idle
- wa: cpu is idle because it waits for IO to complete
- hi: interrupts generated by hardware
- si: interrupts generated by software
Then look at more detailed virtual memory subsystem stats, again while system is performing well. For this you can use the vmstat command.
vmstat 5 will print an initial header and stats-to-date line and then an additional line every 5 seconds with differential stats. Leave this going for a while so you have a good idea what this normally looks like. e.g.:
$ vmstat 5 procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu---- r b swpd free buff cache si so bi bo in cs us sy id wa 2 0 94348 13440 13428 60304 0 0 2 2 2 1 9 0 90 1 0 0 94348 13456 13452 60304 0 0 0 86 46 70 0 0 100 0 0 0 94348 13464 13460 60304 0 0 0 5 30 65 0 0 100 0 0 0 94348 13472 13472 60304 0 0 0 17 32 66 0 0 100 0 0 0 94348 13472 13480 60304 0 0 0 3 30 64 0 0 100 0 0 0 94348 13472 13496 60304 0 0 0 17 37 68 0 0 100 0 0 0 94348 13472 13512 60304 0 0 0 18 33 67 0 0 100 0
- r: processes ready to run but are waiting for the cpu
- b: processes in uninterruptible sleep (often io-related)
- swpd: swap used (KiB)
- free: memory doing nothing (KiB)
- buff: buffers (KiB)
- cache: disk cache (KiB)
- si: memory swapped in from disk (KiB/s)
- so: memory swapped out from disk (KiB/s)
- bi: Blocks read in from block devices (blocks/sec)
- bo: Blocks written out to block devices (blocks/sec)
- in: interrupts per second (hardware, software, e.g. "the clock ticked" "the ethernet card got a packet")
- cs: context switches per second (one process stops running on CPU, another starts)
- us: Userland CPU usage %
- sy: Kernelspace CPU usage %
- id: Idle CPU %
- wa: IO wait CPU %
- Ignore the first line. You are looking for "r" and "b" to be low, certainly single digits, preferably zero. If they're not then processes generally are lacking resources, usually CPU or disk.
- Amount of swap ("swpd") is not as important as swap churn ("si", "so") which should be close to zero.
- The remaining io/system/cpu figures can give you an idea of how busy your machine normally is in terms of CPU and disk.
e.g. top says load is high, "wa" is low but "us" is high and this goes on for long periods and coincides with poor performance. Therefore your processes are CPU starved, give the machine more CPU (or run better code!) and things will improve (or you'll find a different bottleneck!)
e.g. top says load is high, "us" is low, "wa" is above 75% for a long time: your processes are waiting for IO, could be network or disk or various other things, but is most likely disk.
Well, OK so you find out your system is starved for IO, but that's a bit vague. You need to know what is wanting the IO. This can be a bit tricky but something that can really help is iostat. iostat -x 5 will show you the IO to every partition on your system updated every 5 seconds:
(output from a RHEL box as none of my play machines have significant disk IO; 2.6 kernel output may be slightly different)
$ iostat -x 5 Linux 2.4.21-37.ELhugemem (xxxpdb02) 02/02/06 avg-cpu: %user %nice %system %iowait %steal %idle 4.30 0.00 0.56 0.31 0.00 94.83 Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util sda 0.30 11.21 3.33 4.02 133.11 85.33 29.71 0.08 10.34 1.75 1.29 sdb 0.08 4.02 0.63 0.82 37.50 38.12 52.14 0.04 31.00 2.33 0.34 dm-0 0.00 0.00 0.05 3.83 0.24 7.66 2.03 0.02 5.25 0.36 0.14 dm-1 0.00 0.00 0.01 0.04 0.03 0.08 2.02 0.00 5.07 1.74 0.01 dm-2 0.00 0.00 0.15 2.83 2.12 22.66 8.31 0.03 8.54 0.47 0.14 dm-3 0.00 0.00 3.38 7.30 145.31 58.44 19.07 0.32 29.72 0.98 1.05 dm-4 0.00 0.00 0.55 2.33 10.95 4.66 5.42 0.17 58.80 1.07 0.31 dm-5 0.00 0.00 0.00 0.04 0.02 0.32 8.00 0.10 2515.02 0.20 0.00 dm-6 0.00 0.00 0.19 3.70 11.94 29.63 10.67 0.40 102.07 0.20 0.08 avg-cpu: %user %nice %system %iowait %steal %idle 33.75 0.00 2.60 0.05 0.00 63.60 Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util sda 0.00 29.74 0.00 6.79 0.00 253.89 37.41 0.02 2.26 0.38 0.26 sdb 0.00 0.00 0.00 0.40 0.00 3.19 8.00 0.00 4.00 2.00 0.08 dm-0 0.00 0.00 0.00 5.59 0.00 11.18 2.00 0.00 0.64 0.04 0.02 dm-1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 dm-2 0.00 0.00 0.00 7.19 0.00 57.49 8.00 0.01 1.06 0.06 0.04 dm-3 0.00 0.00 0.00 23.35 0.00 186.83 8.00 0.07 3.03 0.08 0.18 dm-4 0.00 0.00 0.00 0.80 0.00 1.60 2.00 0.00 0.25 0.25 0.02 dm-5 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 dm-6 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 avg-cpu: %user %nice %system %iowait %steal %idle 17.51 0.00 0.90 0.00 0.00 81.59 Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util sda 0.00 19.00 0.00 5.40 0.00 111.20 20.59 0.01 1.04 0.33 0.18 sdb 0.00 0.20 0.00 1.00 0.00 3.60 3.60 0.00 0.80 0.80 0.08 dm-0 0.00 0.00 0.00 8.40 0.00 16.80 2.00 0.00 0.33 0.07 0.06 dm-1 0.00 0.00 0.00 0.60 0.00 1.20 2.00 0.00 0.00 0.00 0.00 dm-2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 dm-3 0.00 0.00 0.00 10.60 0.00 84.80 8.00 0.02 1.53 0.08 0.08 dm-4 0.00 0.00 0.00 6.00 0.00 12.00 2.00 0.01 0.90 0.17 0.10 dm-5 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 dm-6 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00Don't be scared by this mass of info! You can read the man page for iostat to see what all of the columns are, but for simple diagnostics you only need to look at a few things. Again ignore the first set of results since that's the total since system start.
- Device
- which block device we're talking about
- rkB/s
- KiB/s read from device
- wkB/s
- KiB/s written to device
- await
- Average time, in ms, that requests completed in the last interval (i.e. 5 secs) took to be completed, including how long they waited and how long they were actually being serviced
- %util
- Percentage of the processes' CPU time that was spent dealing with IO related to this device. When this gets close to 100 then the device is doing all it can, basically.
To try and track down which processes are doing the IO you can do a ps awux and look for processes with a "D" in the "STAT" column. This indicates uninterruptible sleep, usually IO-related. If you are lucky you will also be able to use lsof -p to see exactly what they are reading/writing from/to, but usually iostat was specific enough.
Once you know which block devices are suffering, the answer is to buy more/faster disks and put those filesystems on them. Use SoftwareRAID to increase performance and maintain redundancy which is essential with multiple disks since adding more disks is multiplying chances of device failure. See Linux RAID best practices for an existing diatribe on this subject.
Basic performance diagnostics are really useful even in a home situation where you just want to know where to best spend 200 quid.
Retrieved from "http://strugglers.net/wiki/Linux_performance_tuning"
