Exercise 8: Compare BLAS/ATLAS/MKL DGEMM routines for Matrix multiplication
EXERCISE DESCRIPTION:
In this exercise you will test three different implementations of the DGEMM routine needed to perform a quite common operation in scientific computation: matrix multiplication. A small F90 program driver to the DGEMM routine is provided together with a Makefile. You will link this driver against three different libraries: the original BLAS fortran one (self-compiled), the ATLAS library and the Intel MKL library (single and multi-threaded).
During this exercise you will implement the following tasks:
- Install the ATLAS library
- Install the MKL library
- Download and benchmark the matmul package using five different benchmark configurations
- Measure and compare performances obtained.
- (Optional) modify the code to explore a wider or different range of sizes.
In the following Guide all the above steps are described in some detail:
EXERCISE GUIDE (to carefully check everything)
Part 0: Reset your machine for use your personal account
1.- Login as root
#su
2.- Edit the fstab file
#vim /etc/fstab
3.- Remove (or comment) the IP:/home /home ... line.
4.- remount the directories (or simply reboot your machine).
#mount -a
5.- Logout the actual session and login with your personal username.
Part 1: Install the mathematical Libraries (MKL and ATLAS)
Task 1: Download and install the ATLAS library
1. Login as root
#su
2. Install the atlas package from the repositories.
#apt-get install libatlas-base-dev
DONE! (... because it was precompiled for your type of machine ;-) )
Task 2: Download and install the MKL library
MKL is provided by Intel for its processors and needs an installation procedure similar to that of the compilers.
The mkl package was already downloaded the first day. Your need to login as root.
# su
the password of root is 12compaq34
Ensure that you have the necessary space on the hard disk drive. To clean temporary files run the command (as root)
#apt-get clean
and verify that you have more that 1.6Gb of space with
#df -lh
you will see some like
S.ficheros Tamaño Usado Disp Uso% Montado en /dev/sda1 6,5G 3,9G 2,3G 64% / tmpfs 498M 0 498M 0% /lib/init/rw udev 10M 84K 10M 1% /dev tmpfs 498M 0 498M 0% /dev/shm /dev/sda6 65G 187M 61G 1% /home
and the partition associated to / must have more than 1.6Gb.
1. Move to the location :
# cd /scratch/compilers/intel/Intel/mkl/10.1.1/
2. Now “untar” (that is, uncompress and unpack) the installation files:
# tar -xzvf l_mkl_p_10.1.1.019.tgz
3. Go to the installation directory:
# cd l_mkl_p_10.1.1.019
4. Run the install script:
# ./install.sh
5. You will be presented with this menu:
********************************************************************** Welcome to the Intel(R) Math Kernel Library 10.1 Update 1 for Linux*. The Intel(R) Math Kernel Library (Intel(R) MKL) for Linux* is composed of highly optimized mathematical functions for engineering, scientific and financial applications requiring high performance on Intel(R) platforms. The functional areas of the library include linear algebra consisting of LAPACK and BLAS, Discrete Fourier Transforms (DFT), vector transcendental functions (vector math library/VML), and vector statistical functions (VSL). Intel MKL is optimized for the features and capabilities of the latest Intel microprocessors. -------------------------------------------------------------------------------- You will complete the steps below during this installation: Step 1 : Welcome Step 2 : License agreement Step 3 : Activation option Step 4 : Installation configuration Step 5 : Configuration summary Step 6 : Installation Step 7 : Installation complete -------------------------------------------------------------------------------- Press "Enter" key to continue or "q" to quit:
Press "Enter" to continue.
6. You will now be presented with the following menu:
Step no: 2 of 7 | License agreement -------------------------------------------------------------------------------- To continue with the installation, you must accept the terms and conditions of the End User License Agreement (EULA). The EULA will be displayed with the "more" utility. While viewing the EULA, press the spacebar to page down or press the "q" key to quit viewing the EULA. After viewing the EULA, you will have the choice to either accept or decline the terms and conditions of the EULA. -------------------------------------------------------------------------------- 1. View the license agreement (required) [default] h. Help b. Back to the previous menu q. Quit -------------------------------------------------------------------------------- Please type a selection or press "Enter" to accept default choice [1]:
Select option 1 to view the license agreement, that is, type the 1 key and then press ENTER. Accept the license terms and conditions.
7. The computer will display the following prompt:
Step no: 3 of 7 | Activation options -------------------------------------------------------------------------------- Activation requires a serial number and internet connection. It is a secure and anonymous one-time process which verifies your software licensing rights to enable use of the product. You can also choose to install the software for evaluation purposes or activate at a later time. The software will time out in 30 days. -------------------------------------------------------------------------------- 1. Use existing license [default] 2. I want to activate and install my product 3. I want to evaluate my product or activate later 4. Alternative activation - necessary if you plan to activate remotely ( because this system may not be set for successful Internet connection), or use a license file, or use a license server 5. View additional information about software activation h. Help b. Back to the previous menu q. Quit -------------------------------------------------------------------------------- Please type a selection or press "Enter" to accept default choice [1]:
Then press <ENTER> to accept the default option, that is, use a existing license.
8. The computer will display the following option
Step no: 4 of 7 | Installation Type -------------------------------------------------------------------------------- Congratulations! Your software has been activated. Please continue the installation by choosing Typical Install (default installation options) or Custom Install to change the default installation options. -------------------------------------------------------------------------------- 1. Typical Install (Recommended) [default] 2. Custom Install (For Advanced Users) h. Help b. Back to the previous menu q. Quit -------------------------------------------------------------------------------- Please type a selection or press "Enter" to accept default choice [1]:
Press "Enter" for the typical installation procedure.
9. The computer will display
Step no: 5 of 7 | Configuration summary -------------------------------------------------------------------------------- Select option "1" below to begin installing files based on the information you provided. Use the "b" option to go back to the previous screen(s) if you need to change any of the information below. -------------------------------------------------------------------------------- Activation Type: Existing License File Component(s): Intel(R) Math Kernel Library 10.1 Update 1 for Linux* Location: /opt/intel/mkl/10.1.1.019 Install Mode: NONRPM -------------------------------------------------------------------------------- 1. Install [default] h. Help b. Back to the previous menu q. Quit -------------------------------------------------------------------------------- Please type a selection or press "Enter" to accept default choice [1]:
Press "Enter" to start installation.
10. After the "Installation successfull" message press <ENTER> to continue.
11. Type 'q' and press <ENTER> to exit.
12. Logout from the root account, you should get a prompt like a normal user.
#exit $cd
13. Edit the .bashrc file of the user
$vim ~/.bashrc
14. Add the line at the bottom fo the .bashrc file
source /opt/intel/mkl/10.1.1.019/tools/environment/mklvars32.sh
15. Reload your ".bashrc" file
$source ~/.bashrc
The Intel MKL is now installed on your system.
Part 2 Compare Library Performance with the Matmul Program
From here on, our instructions will be less remedial so when in doubt ask, use the man command, or refer to the other labs or notes.
Task 3: Download the matmul package
1. Ensure that you are using a normal account (non root)
2. Download Matmul package:
$ svn checkout --username anonsvn https://www.qe-forge.it/svn/hpc-2008/trunk/week1/Matmul password: anonsvn $ cd Matmul
Task 4: Compile the matmul benchmark using five different configurations
In order to compile the Matmul program, you will need to edit the Makefile and specify the right paths for each library and the correct combination. The Makefile will build 5 executables:
matmul0
standard dgemm
matmul1
link to external library LIB1 (ATLAS)
matmul2
link to external library LIB2 (multi-threaded MKL)
matmul3
link to external library LIB3 (single-threaded MKL)
matmul4
link to external library LIB4 (MKL with Gfortran)
By changing LIB1 to LIB4, you can get the Matmul program linked with different mathematical libraries, and compare performances.
In the following instructions, we will be using the Atlas lib as LIB1 and the Intel MKL library as LIB2. You are of course free to experiment with other libraries.
1. Edit the Matmul Makefile:
$ gedit Makefile
2. Find the lines that read:
# please indicate Blas libraries for matmul1 LIB1 = -L/home/tutors/atlas-core2/lib -lf77blas -latlas
and change them to read:
LIB1 = -L/usr/lib -lf77blas -latlas -lgfortran
Then save your file.
3. In order to link against the multi-threaded Intel MKL libraries, you will have to tell the compiler that some objects have been compiled with OpenMP support by adding the -openmp flag.
Check that the following line is present:
LIB2 = -openmp -L/opt/intel/mkl/10.1.1.019/lib/32/ -lmkl
4. The Intel MKL library is a collection of modular libraries (-lmkl and -lmkl_em64t are "wrapper"). By using individual component libraries, one can adjust its behavior.
Check that the following line is present:
LIB3 = -L/opt/intel/mkl/10.1.1.019/lib/32/ -lmkl_intel -lmkl_sequential -lmkl_core
5. To compile and link with gfortran and MKL, check that the following line is present:
LIB4 = -L/opt/intel/mkl/10.1.1.019/lib/32/ -lmkl_gf -lmkl_gnu_thread -lmkl_core -lpthread
6. Save the file and exit the gedit editor.
Task 5: Measure and compare performances obtained
1. Make the five executables matmul0, matmul1, matmul2, matmul3, matmul4:
$ make
2. Compare the output and performance of the different programs. To run the different programs you just have to type for example
$ ./matmul0
or use the run_matmul script to write the output of all five versions on file.
If you use the script be aware that the program may run for a while and you won't see any output nor have any access to the console. If you want to use the console while the program is running type z while holding the Ctrl key and then type bg. The program will now run in the background and you'll be able to use the console.
To inspect the output type for example
$ less out.matmul0
You might want to inspect the source code for the matmul program to known what's the meaning of the columns of numbers you see in the output.
3. Change the compiler options, by editing the CFLAGS and FFLAGS lines in the Makefile. Does adding optimization flags improve the performance of Matmul as much as it did with the programs in the other exercises? Can you tell why?
4. Check for multi-threading. Set the environment variable OMP_NUM_THREADS to either 1 or 2 and see how it affects the performance.
5. Check the difference between compilers. matmul4 uses gfortran instead of intel fortran. Discuss how this affects the timings.
Note: When you change the compiler in the Makefile, you need to issue the command
$ make clean
before compiling code again.
Good luck! :-)