为了进入TOP 50、100、500 HPC（高性能计算）复合体列表，使用HPL（高性能Linpack）基准测试获得的测试结果是合适的。Linpack基准测试（线性代数打包）实现了一种使用LU分解方法求解SLAE的算法。该软件包是公开可用的，易于安装和运行。适用于演示CPU性能。熟悉图形加速器体系结构的每个人都可以认为，此套件对于测试具有GPU体系结构的计算设备甚至更好。但是，可在线下载2011版的Fermi架构CUDA。在本指南中，我将举例说明如何为GPU构建和运行HPL。如何控制对软件的访问？
如何安装CUDA？
如何安装openmpi？
如何安装openblas？
如何为GPU安装HPL？安装模块包
要管理环境变量，请安装MODULES软件包并准备一个测试模块文件。$ yum install environment-modules
$ mcedit /etc/modulefailes/test/v1.0
proc ModulesHelp { } {
global version
puts stderr "Modulefile for test v1.0"
}
set version v1.0
module-whatis "Modulefile for test v1.0"
setenv MAINDIR /nfs/software/test/v1.0
prepend-path PATH $env(MAINDIR)/bin
prepend-path C_INCLUDE_PATH $env(MAINDIR)/include
prepend-path CPLUS_INCLUDE_PATH $env(MAINDIR)/include
prepend-path LIBRARY_PATH $env(MAINDIR)/lib64
prepend-path LD_LIBRARY_PATH $env(MAINDIR)/lib64
检查模块文件
准备模块时出错的可能性非常高。因此，我检查了模块文件中指定的所有路径。为了不手动检查每个路径，我准备了一个脚本。如果为0，则路径正确。$ cat check-modulefiles
ModulePath=$1
MainDir=$(cat $ModulePath | grep "setenv MAINDIR" | cut -f7 -d " ")
ListOfPaths=$(cat $ModulePath | grep path | cut -f7 -d " ")
ListOfPaths=$(echo $ListOfPaths | sed "s@\$env(MAINDIR)@$MainDir@g")
for u in $ListOfPaths; do
ls -la $u 1> /dev/null 2> /dev/null;
printf "%60s %4d\n" $u $?;
done
$ chmod +x check-modulefiles
$ ./check-modulefiles /etc/modulefiles/test/v1.0
/nfs/software/test/v1.0/bin 0
/nfs/software/test/v1.0/include 0
/nfs/software/test/v1.0/include 0
/nfs/software/test/v1.0/lib64 0
/nfs/software/test/v1.0/lib64 0
模块管理命令
$ module avail
$ module add cuda/v10.1
$ nvcc –version
Cuda compilation tools, release 10.1, V10.1.168
$ module switch cuda/v10.1 cuda/v9.2
$ nvcc –version
Cuda compilation tools, release 9.2, V9.2.88
$ module list
$ module rm cuda/v9.2
1.让我们看看可用于连接的模块列表。2.连接3-4 模块。检查版本5。更换模块6-7。让我们检查版本8。让我们看一下已连接模块的列表。9.从已连接列表中删除该模块。安装CUDA
在此处
下载适用于Centos 7的CUDA 9.2 。$ chmod +x cuda_9.2.run
$ ./cuda_9.2.run
Do you accept the previously read EULA? accept
Install the CUDA 9.2 Toolkit? yes
Enter Toolkit Location: /nfs/software/cuda/v9.2
Do you want to install a symbolic link at /usr/local/cuda? no
Install the CUDA 9.2 Samples? no
$ cat /etc/modulefiles/cuda/v9.2
proc ModulesHelp { } {
global version
puts stderr "Modulefile for cuda v9.2"
}
set version v9.2
module-whatis "Modulefile for cuda v9.2"
setenv MAINDIR /nfs/software/cuda/v9.2
prepend-path PATH $env(MAINDIR)/bin
prepend-path C_INCLUDE_PATH $env(MAINDIR)/include
prepend-path CPLUS_INCLUDE_PATH $env(MAINDIR)/include
prepend-path LIBRARY_PATH $env(MAINDIR)/lib64/stubs
prepend-path LIBRARY_PATH $env(MAINDIR)/lib64
prepend-path LD_LIBRARY_PATH $env(MAINDIR)/lib64/stubs
prepend-path LD_LIBRARY_PATH $env(MAINDIR)/lib64
$ module add cuda/v9.2
$ nvcc --version
Cuda compilation tools, release 9.2, V9.2.148
安装OpenBLAS
$ wget https://github.com/xianyi/OpenBLAS/archive/v0.3.6.tar.gz
$ tar -xzvf v0.3.6.tar.gz
$ cd OpenBLAS-0.3.6
$ mkdir -p /nfs/software/openblas/v0.3.6
$ make -j4
$ make PREFIX=/nfs/software/openblas/v0.3.6/ install
$ ls -la /nfs/software/openblas/v0.3.6/lib/
$ cat /etc/modulefiles/openblas/v0.3.6
proc ModulesHelp { } {
global version
puts stderr "Modulefile for openblas v0.3.6"
}
set version v0.3.6
module-whatis "Modulefile for openblas v0.3.6"
setenv MAINDIR /nfs/software/openblas/v0.3.6
prepend-path PATH $env(MAINDIR)/bin
prepend-path C_INCLUDE_PATH $env(MAINDIR)/include
prepend-path CPLUS_INCLUDE_PATH $env(MAINDIR)/include
prepend-path LIBRARY_PATH $env(MAINDIR)/lib
prepend-path LD_LIBRARY_PATH $env(MAINDIR)/lib
$ ls -la /nfs/software/openblas/v0.3.6/lib
安装OpenMPI
wget https://download.open-mpi.org/release/open-mpi/v2.1/openmpi-2.1.6.tar.gz
$ tar -xzvf openmpi-2.1.6.tar.gz
$ cd openmpi-2.1.6
$ mkdir -p /nfs/software/openmpi/v2.1.6
$ module add cuda/v9.2
$ ./configure --prefix=/nfs/software/openmpi/v2.1.6/ --with-cuda --enable-static
$ make
$ make install
$ cat /etc/modulefiles/openmpi/v2.1.6
proc ModulesHelp { } {
global version
puts stderr "Modulefile for openmpi v2.1.6"
}
set version v2.1.6
module-whatis "Modulefile for openmpi v2.1.6"
setenv MAINDIR /nfs/software/openmpi/v2.1.6
prepend-path PATH $env(MAINDIR)/bin
prepend-path C_INCLUDE_PATH $env(MAINDIR)/include
prepend-path CPLUS_INCLUDE_PATH $env(MAINDIR)/include
prepend-path LIBRARY_PATH $env(MAINDIR)/lib
prepend-path LD_LIBRARY_PATH $env(MAINDIR)/lib
$ module add openmpi/v2.1.6
$ mpirun --version
mpirun (Open MPI) 2.1.6
安装适用于GPU的HPL
通过连接模块来设置环境变量并下载HPL 2.0。$ module add openmpi/v2.1.6
$ module add cuda/v9.2
$ module add openblas/v0.3.6
$ wget https://developer.download.nvidia.com/assets/cuda/secure/AcceleratedLinpack/hpl-2.0_FERMI_v15.tgz
$ tar -xvf hpl-2.0_FERMI_v15.tgz
$ mv hpl-2.0_FERMI_v15.tgz hpl-2.0
$ cd hpl-2.0
组装之前，必须编辑几个文件。第一个是hpl-2.0目录中的Make.CUDA。将以下代码复制到Make.CUDA中：$ cat Make.CUDA
SHELL = /bin/sh
CD = cd
CP = cp
LN_S = ln -fs
MKDIR = mkdir -p
RM = /bin/rm -f
TOUCH = touch
ARCH = CUDA
TOPdir = /home/user/hpl-2.0
INCdir = $(TOPdir)/include
BINdir = $(TOPdir)/bin/$(ARCH)
LIBdir = $(TOPdir)/lib/$(ARCH)
HPLlib = $(LIBdir)/libhpl.a
MPdir = /nfs/software/openmpi/v2.1.6
MPinc = -I$(MPdir)/include
MPlib = -L$(MPdir)/lib -lmpi
LAdir = /nfs/software/openblas/v0.3.6
LAinc = -I$(LAdir)/include
LAlib = -L$(TOPdir)/src/cuda -ldgemm -L/nfs/software/cuda/v9.2/lib64 -lcuda -lcudart -lcublas -L$(LAdir)/lib -lopenblas
F2CDEFS = -DAdd__ -DF77_INTEGER=int -DStringSunStyle
HPL_INCLUDES = -I$(INCdir) -I$(INCdir)/$(ARCH) $(LAinc) $(MPinc)
HPL_LIBS = $(HPLlib) $(LAlib) $(MPlib)
HPL_OPTS = -DCUDA
HPL_DEFS = $(F2CDEFS) $(HPL_OPTS) $(HPL_INCLUDES)
CC = mpicc
CCFLAGS = -fopenmp -lpthread -fomit-frame-pointer -O3 -funroll-loops $(HPL_DEFS)
CCNOOPT = $(HPL_DEFS) -O0 -w
LINKER = $(CC)
LINKFLAGS = $(CCFLAGS)
ARCHIVER = ar
ARFLAGS = r
RANLIB = echo
MAKE = make TOPdir=$(TOPdir)
11. hpl-2.0目录的路径17. OpenMPI的路径21. OpenBLAS的路径23. CUDA lib64的路径替换hpl-2.0 / src / cuda / cuda_dgemm.c文件中的以下行：$ mcedit src/cuda/cuda_dgemm.c
…
// handle2 = dlopen ("libmkl_intel_lp64.so", RTLD_LAZY);
handle2 = dlopen ("libopenblas.so", RTLD_LAZY);
…
// dgemm_mkl = (void(*)())dlsym(handle, "dgemm");
dgemm_mkl = (void(*)())dlsym(handle, "dgemm_");
…
// handle = dlopen ("libmkl_intel_lp64.so", RTLD_LAZY);
handle = dlopen ("libopenblas.so", RTLD_LAZY);
…
// mkl_dtrsm = (void(*)())dlsym(handle2, "dtrsm");
mkl_dtrsm = (void(*)())dlsym(handle2, "dtrsm_");
在4倍GPU上构建并运行HPL：$ make arch=CUDA
$ cd bin/CUDA
$ export LD_LIBRARY_PATH=/home/user/hpl-2.0/src/cuda/:$LD_LIBRARY_PATH
$ mpirun -np 4 ./xhpl
================================================================================
HPLinpack 2.0 -- High-Performance Linpack benchmark -- September 10, 2008
Written by A. Petitet and R. Clint Whaley, Innovative Computing Laboratory, UTK
Modified by Piotr Luszczek, Innovative Computing Laboratory, UTK
Modified by Julien Langou, University of Colorado Denver
================================================================================
An explanation of the input/output parameters follows:
T/V : Wall time / encoded variant.
N : The order of the coefficient matrix A.
NB : The partitioning blocking factor.
P : The number of process rows.
Q : The number of process columns.
Time : Time in seconds to solve the linear system.
Gflops : Rate of execution for solving the linear system.
The following parameter values will be used:
N : 25000
NB : 768
PMAP : Row-major process mapping
P : 2
Q : 2
PFACT : Left
NBMIN : 2
NDIV : 2
RFACT : Left
BCAST : 1ring
DEPTH : 1
SWAP : Spread-roll (long)
L1 : no-transposed form
U : no-transposed form
EQUIL : yes
ALIGN : 8 double precision words
--------------------------------------------------------------------------------
- The matrix A is randomly generated for each test.
- The following scaled residual check will be computed:
||Ax-b||_oo / ( eps * ( || x ||_oo * || A ||_oo + || b ||_oo ) * N )
- The relative machine precision (eps) is taken to be 1.110223e-16
- Computational tests pass if scaled residuals are less than 16.0
================================================================================
T/V N NB P Q Time Gflops
--------------------------------------------------------------------------------
WR10L2L2 25000 768 2 2 16.72 6.232e+02
--------------------------------------------------------------------------------
||Ax-b||_oo/(eps*(||A||_oo*||x||_oo+||b||_oo)*N)= 0.0019019 ...... PASSED
================================================================================
Finished 1 tests with the following results:
1 tests completed and passed residual checks,
0 tests completed and failed residual checks,
0 tests skipped because of illegal input values.
--------------------------------------------------------------------------------
End of Tests.
================================================================================
要编辑测试参数，请使用hpl-2.0 / bin / CUDA / HPL.dat文件