IBM Wikis - Linux on POWER

When performance teams are asked for advice in improving performance, the requests are generally fairly generic and open-ended. In these cases, we generally do a first impressions sweep of the performance information.

In this example, we're on a RHEL 5.1 system on a Power6 system. If you're on different levels, the rpm and kernel versions will of course be different.

Check distro level

First, check the basic values to confirm the OS levels.

# cat  /etc/*release*
Red Hat Enterprise Linux Server release 5.1 (Tikanga)

# uname -r
2.6.18-53.el5

# ls /boot/vm*
/boot/vmlinuz-2.6.18-53.el5

RPMs to install.

We'll want oprofile and some extra commands available installed. Install oprofile and sysstat.

oprofile-0.9.2-6.el5

opcontrol
opreport

sysstat-7.0.0-3.el5

mpstat
iostat
sar

Open multiple windows.

Create a new /tmp directory to work out of. For example, /tmp/workdir. Open three windows. In each window, cd to that /tmp/workdir directory.

In one window, run the mpstat command. This will dump CPU run statistics for all of the individual CPUs every 5 seconds.

# mkdir /tmp/workdir
# cd /tmp/workdir
# mpstat -P ALL 5 | tee mpstat.txt

In a second window, run the iostat command. This format provides statistics in kilobytes per second and will display the extended statistics, every 5 seconds.

# cd /tmp/workdir
# iostat -xk 5 | tee iostat.txt

In a third window, fire up oprofile, see below for steps.

oprofile

To initialize oprofile, execute the following commands. Note that the vmlinux image should correspond to your kernel.

cd /tmp/workdir
opcontrol --version
opcontrol --vmlinux=/boot/vmlinuz-2.6.18-53.el5
opcontrol --reset
opcontrol --event=CYCLES:1000000
opcontrol --start
opcontrol --reset

In a fourth window, run the workload you want to profile.

After the workload runs, you can terminate oprofile with the following commands:

opcontrol --dump
opcontrol --stop
opcontrol --shutdown

opreport   --symbols         > opreport.symbols.txt
opreport   --long-filenames  > opreport.filenames.txt

cp /var/lib/oprofile/samples/oprofiled.log oprofiled.log

Then, in the remaining profiling windows, Ctrl-C out of mpstat and iostat.

Run snap

snap is a handy IBM support data gathering tool which quickly grabs applicable data for basic system definition assistance.

# cd /tmp/workdir
# snap

Tar up the results

# cd /tmp
# ls workdir/
mpstat.txt
iostat.txt
snap.tar.gz
opreport.symbols.txt
opreport.filenames.txt
oprofiled.log

# tar -cf workdir.tar workdir/
# gzip workdir.tar

SOS - son of sysreport

If available, run sosreport from RedHat.

[root@p6hpc3 scripts]# sosreport

sosreport (version 1.7)

This utility will collect some detailed  information about the
hardware and  setup of your  Red Hat Enterprise Linux  system.
The information is collected and an archive is  packaged under
/tmp, which you can send to a support representative.
Red Hat will use this information for diagnostic purposes ONLY
and it will be considered confidential information.

This process may take a while to complete.
No changes will be made to your system.

Press ENTER to continue, or CTRL-C to quit.

Please enter your first initial and last name [p6hpc3]:
Please enter the case number that you are generating this report for: 001

 Progress [###################100%##################][04:01/04:01]

Creating compressed archive...

Your sosreport has been generated and saved in:
  /tmp/sosreport-p6hpc3.001-405558-c54a09.tar.bz2

The md5sum is: 46a9ed06fc432f8d1989f2f97cc54a09

Please send this file to your support representative.

Send the tar balls

Send the /tmp/workdir.tar.gz and the /tmp/sosreport* to your support representative. Be sure to include a detailed description of the workload that was profiled.