Install CESM-1.0.2 using PGI compilers 13.4.0 , Intel MPI 4.1.0.024, Netcdf and Hdf5

Posted on July 8, 2013 by nishaagrawal

Contents

1.  Install Pre-requisites
1.1 zlib
1.2 hdf5
1.3 netcdf
2. Build , install and test CESM-1.0.2
3. Known errors and solutions
4. References

1.  Install Pre-requisites

Below are the additional software which are required to compile and build CESM-1.0.2 with Hdf5 .

1.1 zlib installation

1.1.1 untar the zip file
1.1.2 Read the comment section of Makefile.in
1.1.3 Export environment variables

export CC=pgcc
export FC=pgf90
export CFLAGS=”-m64 ”
export FFLAGS=”-m64″

1.1.4 ./configure –prefix=<config-path>
1.1.5 make install prefix=<install-path>

1.2. hdf5-1.8.11 installation

1.2.1 Download and untar
1.2.2 Export environment variables

export CC=pgcc
export FC=pgf90
export LDFLAGS=-L/opt/intel/impi/4.1.0.024/lib64
export LIBS=”-lmpi -lmpigf”
export CPPFLAGS=” -m64 -I/opt/intel/impi/4.1.0.024/include64″
export CFLAGS=” -m64 -I/opt/intel/impi/4.1.0.024/include64″
export FFLAGS=”-m64  -I/opt/intel/impi/4.1.0.024/include64″
export FCFLAGS=”-m64  -I/opt/intel/impi/4.1.0.024/include64″
export CXXFLAGS=”-m64  -I/opt/intel/impi/4.1.0.024/include64″

1.2.3 configure

./configure –prefix=/home/internal/sysadmin/anisha/software/lib/hdf5_1_8_11 –enable-parallel  –with-zlib=/home/internal/sysadmin/anisha/software/lib/zlib_1_2_8

2.4  make install

1.3. netcdf-4.1.3 Installation

1.3.1 Download and untar
1.3.2 Export environment variables

export CC=pgcc
export FC=pgf90
export LDFLAGS=-L/opt/intel/impi/4.1.0.024/lib64

export LIBS=”-L/opt/intel/impi/4.1.0.024/lib64 -lmpi -lmpigf -L<path-to-zlib/lib>
-L<path-to-hdf5/lib>”

export CPPFLAGS=” -m64 -I/opt/intel/impi/4.1.0.024/include64
-I<path-tozlib/include> -I<path-to-hdf5/include>”

export CFLAGS=” -m64 -I/opt/intel/impi/4.1.0.024/include64
-I<path-tozlib/include> -I<path-to-hdf5/include>”

export FFLAGS=”-m64  -I/opt/intel/impi/4.1.0.024/include64
-I<path-tozlib/include> -I<path-to-hdf5/include>”

export FCFLAGS=”-m64  -I/opt/intel/impi/4.1.0.024/include64
-I<path-tozlib/include> -I<path-to-hdf5/include>”

export CXXFLAGS=” -m64  -I/opt/intel/impi/4.1.0.024/include64
-DMPICH_IGNORE_CXX_SEEK  -I<path-tozlib/include> -I<path-to-hdf5/include>”

1.3.3 Configure

./configure –prefix=/opt/app/CESM/lib/netcdf_4.1.3  –enable-netcdf4 –enable-shared=no –with-libcf –disable-dap

1.3.4 make install

2.Build , install and test CESM-1.0.2

2.1 Create , build and execute new case

2.1.1 Use below command to set pgi compilers in PATH , if not in $PATH

source /opt/pgi/env.sh

2.1.2 Configure a new case

$ /opt/app/CESM/cesm1_0_2/scripts/create_newcase -case   $HOME/test_F_f19_f19 -res f19_f19 -compset F_2000 -mach generic_linux_pgi   -scratchroot $HOME/cesm-scratch    -din_loc_root_csmdata $HOME/inputdata   -max_tasks_per_node 4

2.1.3 Change the directory to case directory

cd  $HOME/test_F_f19_f19

2.1.4 Modify files to build case

  • Macro.<machine-name>
  • Tools/Makefile
  • In Macro.<machine-name> do the following modification

– Modify compilers
FC :=pgf90
CC :=pgcc

– Add following lines after CC :=pgcc line

HDF5_PATH=/opt/app/CESM/lib/hdf5_1.8.11
INC_HDF5=-I$(HDF5_PATH)/include
LIB_HDF5=-L$(HDF5_PATH)/lib

ZLIB_PATH=/opt/app/CESM/lib/zlib_1.2.8
INC_ZLIB=-I$(ZLIB_PATH)/include
LIB_ZLIB=-L$(ZLIB_PATH)/lib

– Set MPI and netcdf PATH

NETCDF_PATH := /opt/app/CESM/lib/netcdf_4.1.3
INC_NETCDF := $(NETCDF_PATH)/include
LIB_NETCDF := $(NETCDF_PATH)/lib $(LIB_HDF5) $(LIB_ZLIB)
MOD_NETCDF := $(NETCDF_PATH)/include
MPICH_PATH := /opt/intel/impi/4.1.0.024
INC_MPI := $(MPICH_PATH)/include64
LIB_MPI := $(MPICH_PATH)/lib64
MPI_LIB_NAME :=mpi -lmpigf

– Set mct and pio configuration parameters, replace the mct and pio CONFIG_ARGS
values with below mention parameters

ifeq ($(MODEL),mct)
#add arguments for mct configure here
CONFIG_ARGS+= FC=”$(FC)” F90=”$(FC)” FFLAGS=”$(FFLAGS)” CC=”$(CC)” MPIF90=”$(FC)” MPIHEADER=”-I $(INC_MPI)” MPILIBS=”-L$(LIB_MPI) -l$(MPI_LIB_NAME)” –host=Linux
endif

ifeq ($(MODEL),pio)
ifneq ($(strip $(PIO_CONFIG_OPTS)),)
CONFIG_ARGS += $(PIO_CONFIG_OPTS)
endif
CONFIG_ARGS += FC=”$(FC)” F90=”$(FC)” FFLAGS=”$(FFLAGS)” CC=”$(CC)” MPIF90=”$(FC)” MPICC=”$(CC)” MPI_INC=”-I$(INC_MPI)” MPI_LIB=”-L$(LIB_MPI) -l$(MPI_LIB_NAME)” NETCDF_PATH=”$(NETCDF_PATH)” PNETCDF_PATH=$(PNETCDF_PATH) –with-hdf5=”$(HDF5_PATH)” –with-zlib=”$(ZLIB_PATH)” –enable-netcdf=no –enable-netcdf4=yes –enable-mpiio=yes –enable-mpi2=yes
endif

  • Modify Tools/Makefile

Replace -lnetcdf to -lnetcdff -lnetcdf -lhdf5_hl -lhdf5 -lz at line number 113, 115 and 216. These lines should look like below provided lines after modification

113 SLIBS := -L$(LIB_NETCDF) -lnetcdff -lnetcdf -lhdf5_hl -lhdf5 -lz
115 SLIBS += -L$(LIB_NETCDF) -lnetcdff -lnetcdf -lhdf5_hl -lhdf5 -lz
216 $(LD) -o $@ test_nc.o -L$(LIB_NETCDF) -lnetcdff -lnetcdf -lhdf5_hl -lhdf5 -lz $(LDFLAGS)

2.1.5  Configure the case

./configure -case

2.1.6 Build the executable(case)

./test_F_f19_f19.generic_linux_pgi.build

2.1.7  To run the case Modify the file test_F_f19_f19.generic_linux_pgi.run
Uncomment and modify the below lines as per your requirement

9 ##PBS -N test_F_f19_f19
10 ##PBS -q batch
11 ##PBS -l nodes=16:ppn=4
12 ##PBS -l walltime=00:59:00
13 ##PBS -r n
14 ##PBS -j oe
15 ##PBS -S /bin/csh -V

108 #mpirun -np 64 ./ccsm.exe >&! ccsm.log.$LID

2.1.8 After the modification use below command to run

qsub test_F_f19_f19.generic_linux_pgi.run

2.1.9 Verify run

When the job is complete, review the following directories and files

a. $RUNDIR. This directory is set in the env_build.xml file. This is the location where CESM was run. There should be log files there for every component (i.e. of the form cpl.log.yymmdd-hhmmss). Each component writes its own log file. Also see whether any restart or history files were written. To check that a run completed successfully, check the last several lines of the cpl.log file for the string ” SUCCESSFUL TERMINATION OF CPL7-CCSM”.

b. $CASEROOT/logs. The log files should have been copied into this directory if the run completed successfully.
c. $CASEROOT. There could be a standard out and/or standard error file.

d. $CASEROOT/CaseDocs. The case namelist files are copied into this directory from the $RUNDIR.

e. $CASEROOT/timing. There should be a couple of timing files there that summarize the model performance.

f. $DOUT_S_ROOT/$CASE. This is the archive directory. If DOUT_S is FALSE, then no archive directory should exist. If DOUT_S is TRUE, then log, history, and restart files should have

2.1.10 Porting

for porting refer chapter 7 ” Porting CESM ” of CESM user guide.

3. Known errors and solutions

3.1 Error : rm : No match

Solution : This message is because you are running the model first time.

3.2 Error : Model did not complete – no cpl.log file present – exiting.

Solution: You are running the model first time and you changed the number of processes mention in the <test-case>.<mach>.run file. Do not modify the default number of processes and just uncomment the required lines (at least mpirun line to execute on a single host).

3.3 Error :undefined reference to `H5Dclose’

Solution : If you are using netcdf with hdf5 then the linking with netcdf and hdf5 should be in order . Below is the order

-lnetcdff -lnetcdf -lhdf5_hl -lhdf5 -lz

3.4 Error : co2_cycle.F90:(.data+0x7a8): undefined reference to `typesizes_’
co2_cycle.F90:(.data+0x7b0): undefined reference to `netcdf_’

Solution : Link with the library -lnetcdff also. Add the below flags in Tools/Makefile in same sequence given below

-lnetcdff -lnetcdf -lhdf5_hl -lhdf5 -lz

3.5 Error : Application terminated by signal 11 (illegal instruction)

Solution: Check for the compiler which is used to compile the CESM-1.0.2, is build for the architecture on which you are trying to execute . Also check for the architecture specific compiler flags , if used remove those flags and rebuild.

In my case pgi compilers are build for target processor sandybridge (checked with pgcc –version)  thats why terminating with the mentioned error while trying to execute on old processor.

4. References

4.1. Linking netcdf4 & hdf5  http://www.unidata.ucar.edu/software/netcdf/docs/netcdf-install/Quick-Instructions.html
4.2. Netcdf download
http://www.unidata.ucar.edu/downloads/netcdf/netcdf-4_0/index.jsp
4.3 Zlib download http://www.zlib.net/
4.4 Hdf5 download http://www.hdfgroup.org/HDF5/release/obtain5.html
4.5 Netcdf4p option setting https://github.com/yongqiangyu/FGOALS/blob/master/lnd/clm/bld/namelist_files/namelist_defaults_drv.xml
4.6 CPL7 does not support netcdf4p and netcdf4c optionshttp://www.cesm.ucar.edu/models/ccsm4.0/cpl7/ug.pdf
4.7 https://github.com/LICOMGroup/FGOALS/blob/master/drv/bld/cpl.template

4.8https://github.com/yongqiangyu/FGOALS/blob/master/lnd/clm/bld/namelist_files/namelist_defaults_drv.xml                                                                                                                   4.9 CESM http://www.cesm.ucar.edu/models/cesm1.0/
4.10 CESM 1.2.0 user guide

http://www.cesm.ucar.edu/models/cesm1.2/cesm/doc/usersguide/book1.html

Posted in HPC Applications | Leave a comment

lammps 64 bit installation with Intel compiler & Intel MPI

Posted on May 30, 2013 by nishaagrawal

Contents

1. Compiling & building lammps
2. Running lammps
3. Important points
4. Known Errors and Solutions

1 . Compiling and building lammps

1.1. Untar the lammps tar file
$tar -xvf lammps.tar.gz

1.2. Change the directory to lammps directory
$cd lammps-22Mar13

1.3. Change the directory to /src of lammps
$cd ./src

1.4. Choose a Makefile present in ./MAKE directory and modify as per environment available. For example for intel software environment
$vi MAKE/Makefile.mkl

1.5. For building lammps do from ./src directory
$make mkl

1.6. If build is sucessful it will create a executable called lmp_mkl.

NOTE : To installed the other then the optional packages of lammps refer the below url
http://lammps.sandia.gov/doc/Section_start.html#start_3

2. Running lammps

2.1. Run the test using below command or use pbs script ( Test input is available in lammps bench/ directory ) 

$mpirun -np 2 ./<path-to-lammps>/bin/lmp_mkl < in.lj

3. Important points

3.1. By diffult lammps build with following optional packages

  • MANYBODY
  • KSPACE
  • MOLECULE

package status can be viewed by the following command
$cd src
$make package-status

3.2. Each package supports some styles. For running the example code provided with lammps, first need to check what are  the packages installed with lammps and what the styles supported by those  packages, only those example can beexecuted.

  •  Browse the below link to check for the package and the example
    http://lammps.sandia.gov/doc/Section_packages.html
  •  or use the following commands  To check the packages
    $make package-status
  • To check the style                                                                                                             $./lmp_mkl -h

For example if MANYBODY package is installed you can run shear  example.If a package for a particular style is not supported you will get the following error while executing the code

ERROR: Invalid body style

4. Known Error and Solutions

4.1. Error : domain.cpp(555): error: a value of type
“LAMMPS_NS::tagint={int64_t={long}} *” cannot be used to
initialize an entity of type “int *”
int *image = atom->image;
Solution : Change a line(555) in file src/domian.cpp file
int *image = atom->image;
to
tagint *image = atom->image;

Then do make clean-all and make clean-mkl , Then do make mkl to build lammps.

4.2. ERROR: Invalid body style
Solution : The error means the package supported the particular style is not installed. Install the package.  (Refer section III of this doc.)

Posted in HPC Applications | Leave a comment

Quantum Espresso-5.0.2(QE) 64-bit installation with intel composer xe 2013 and Intel MPI

Posted on March 21, 2013 by nishaagrawal

Contents

  1. Bulid and install Quantum Espresso 5.0.2
  2. Test installation
  3. Known errors and solutions

1. Bulid and install Quantum Espresso 5.0.2

1.1  Download  Quantum Espresso tar file
espresso-5.0.2.tar.gz

1.2.  Untar it
tar -xvf espresso-5.0.2.tar.gz

1.3 Change the directory to QE directory
cd espresso-5.0.2

1.4 Configure

$ ./configure –prefix=PATH-to-install MPIF90=mpiifort FC=ifort F77=ifort F90=ifort CXX=icpc CC=icc BLAS_LIBS=”-L/opt/intel/composer_xe_2013.1.117/mkl/lib/intel64 -lmkl_intel_ilp64 -lmkl_intel_thread -lmkl_core -liomp5 -lpthread -lm” LAPACK_LIBS=” ” FFT_LIBS=”/opt/intel/composer_xe_2013.1.117/mkl/lib/intel64/libfftw3x_cdft_ilp64.a” SCALAPACK_LIBS=”-lmkl_scalapack_ilp64 -lmkl_blacs_intelmpi_ilp64″

1.5 Modify make.sys file

1.5.1 Add flags

MANUAL_DFLAGS  = -D__ISO_C_BINDING
FFLAGS   = -i8 -O2 -assume byterecl -g -traceback -par-report0 -vec-report0
FFLAGS_NOOPT  = -i8 -O0 -assume byterecl -g -traceback
LDFLAGS   = -ilp64

1.6 Packages download

Download the required packages from espresso site and put those in espresso-5.0.2/archive directory

1.7 Build

Build QE with some specific packages (For other packages download the package from QE site and keep them in espresso-5.0.2/archive and do make <pakagename>)

$make all

2. Test installation

For executing a test on single node. Change the directory to the input data directory
For example if BN.in input file is in  /test-yuva directory then change directory
$cd test-yuva
then run
$mpirun -np 2 /opt/espresso-5.0.2/bin/pw.x -in BN.in

if the job successfully go through then at the end it prints JOB DONE.
Note : Required UPF files by BN.in should be in the same directory of BN.in.

3. Known errors and solutions

3.1 Error in execution :

Warning: card / ignored
[eval_infix.c] A parsing error occurred
helper string:
–
error code:
Error: missing operand
[eval_infix.c] A parsing error occurred
helper string:
–
 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
Error in routine card_atomic_positions (1):
Error while parsing atomic position card.
error code:
Error: missing operand
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
 stopping …
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
Error in routine card_atomic_positions (1):
Error while parsing atomic position card.
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
 stopping …

Solution : Add a flag in make.sys file

MANUAL_DFLAGS  = -D__ISO_C_BINDING

Refer http://www.nag.com/nagware/np/r51_doc/iso_c_binding.html

Posted in HPC Applications | 1 Comment

DIRAC-10 64-bit Installation with Intel MPI

Posted on February 19, 2013 by nishaagrawal

Contents 

1. Prerequisites
2. Installation steps
3. Building dirac_mointegral_export.x utility for MRCC
4. Execution 
    4.1 How to execute DIRAC test cases?
    4.2 How to execute DIRAC ?
5. Known error and solutions
6. References

1. Prerequisites

  •  64-bit MPI Library  ( Recommended Intel MPI 4.1.x, OpenMPI 1.4.x)
  •  64-bit BLAS and LAPACK Library ( BLAS & LAPACK library provided with DIRAC can be used )

Environment Used in test run

  • Intel MPI 4.1.0.024
  • Intel composer_xe_2013.1.117 :  Intel compilers(icc/ifort) 13.0 , Intel MKL 11.0

2. Installation steps

2.1 Get  the tar file   DIRAC10.tar.bz2  from      
http://wiki.chem.vu.nl/dirac/index.php/Dirac_Program
by sending the mail to DIRAC program developers.

2.2 Untar the tar file
tar -xvf DIRAC10.tar.bz2

2.3 Update the PATH and LD_LIBRARY_PATH environment variables to add the MPI   compiler and library in PATH and LD_LIBRARY_PATH if these are not already added.

2.4  Change the directory to DIRAC10 directory 
$ cd DIRAC10

2.5 Configure the DIRAC10 by using configuration script provided with DIRAC

  •  To see all available options run

$ ./configure –help

  • Run configuration script ( configure for Single node)

$ ./configure

Generating files Makefile.config and pam

To see all available options run
$ ./configure –help
Your are compiling on a x86_64 system. Do you want to use 64 bit
integers inside Dirac? This is only useful if you need more than
16 GB of memory in your calculations, and requires 64 bit integer
(ILP64) versions of your MPI, BLAS and LAPACK libraries.
Running large-scale MCSCF/CI calculations using the LUCI*
modules, however, requires 64 bit integers on a x86_64 system.

Use 64 bit integers in Dirac? (y/N) Y

Configuring Dirac to use 64 bit integers (ILP64).
NB: Be aware that your external libraries _must_ also use
64 bit integers, which is typically not the default

Configuring for x86_64-unknown-linux-gnu (can be changed with –system)
=> Machine : x86_64-unknown-linux
=> Operating system : gnu

PARALLEL

Do you want to install the program in a parallel version, using MPI? (y/N) => Parallel version: yes
Will your nodes use local disks? (Y/n) n
Makefiles for parallel architecture are difficult to guess
Please always compare the generated Makefile.config with local documentation.
=> Parallel message passing: mpi
Does your MPI setting require machinefile for MPI launcher (like mpich2)? (y/N) N
Using local disks – default machinefile for distribution of files
among nodes needed!

COMPILER

The following Fortran compilers were detected on your system:

1) mpiifort (/home/anisha/software/impi/4.1.0.024/bin64)
2) mpif90 (/home/anisha/software/impi/4.1.0.024/bin64)
3) mpif77 (/home/anisha/software/impi/4.1.0.024/bin64)
4) ifort (/home/anisha/software/composer_xe_2013.1.117/bin/intel64)
5) gfortran (/usr/bin)
0) Specify compiler manually

Which one do you want to use? (Default: 1) 1 ( Or you can specify your own compiler by choosing the option 0)

The following C compilers were detected on your system:

1) mpicc (/home/anisha/software/impi/4.1.0.024/bin64)
2) mpiicc (/home/anisha/software/impi/4.1.0.024/bin64)
3) icc (/home/anisha/software/composer_xe_2013.1.117/bin/intel64)
4) gcc (/usr/bin)
5) cc (/usr/bin)
0) Specify compiler manually

Which one do you want to use? (Default: 1) Compiler for C: 2 ( Or you can specify your own compiler by choosing the option 0)

Found MPI launcher : /home/anisha/software/impi/4.1.0.024/bin64/mpirun
(Note: mpirun is in the same directory as mpif90/mpicc. Good.)

Assigned DEBUGGER (check it in the pam script): /home/anisha/software/composer_xe_2013.1.117/bin/intel64/idb

Assigned VALGRIND debugging & checking tool (http://valgrind.org): /usr/bin/valgrind

Assigned PROFILER tool: /usr/bin/gprof

DETECTION OF MPI LIBRARIES.

Found a mpi compiler. No further searching for mpi libraries.

MATHEMATICAL LIBRARIES

The use of optimized BLAS and LAPACK routines implementation
is very important, especially for the performance of MOLTRA and RELCCSD.

The following mathematical libraries were detected on your system:

2) dirac blas (builtin)
3) dirac lapack (builtin)
1) Specify a library path to search 0) Configure libraries manually.

You need to specify a BLAS AND a LAPACK libraries routines.
There are blas and lapack implementations shipped with dirac.
However they are VERY SLOW compared to optimized libraries.

Which one do you want to use? (More than one possible. Default: 2 3) 0

(or you can choose dirac blas  or dirac lapack)

No libraries are included at this moment.
To add the libraries yourself edit Makefile.config manually!
Mathematical libraries: none found

Testing of used mathematic(/parallel) libraries for integer data type compatibility enabled.

MEMORY SIZE

How many MB RAM should Dirac use as default?
-this is the maximum memory available to older parts of the program
-we do not recommend going below 64 MB
-note that memory use can be changed through the shell variable DIRWRK
Memory size (in MB, default: 488 MB) 1024
INSTALLATION DIRECTORY for binaries and scripts (dirac.x and pam + any of the optional)

Do you want to use current directory /home/anisha/dirac10/DIRAC10? (Y/n) OK. Now type your choice for the installation directory (full path, not relative path): … directory to be used for DIRAC installation:
/home/anisha/dirac10/install

SCRATCH DIRECTORY

Found scratch space among the system directories: /scratch
Press ENTER to use this path, or type a new directory for the scratch: Scratch directory: /home/anisha/dirac10/scratch/$USER

Fortran compiler in /home/anisha/software/impi/4.1.0.024/bin64/mpiifort found to be: ifort C compiler in mpiicc found to be: icc.

Checking if a MPI-2 version is installed:
/home/anisha/software/impi/4.1.0.024/bin64/mpiifort -DSYS_LINUX ./test/MPI2_test.F
=> MPI-2 is available.

============== configure completed – final words: ===============

Please check http://dirac.chem.vu.nl
to see hints or/and problems concerning your platform.

pam script has been configured for your MPI version.
If you change the MPI library, please reconfigure the pam script.
If you use 64 bit integers make sure that MPI libraries and
your math libraries are of the ILP64 type!

Before you run make, you may check and edit Makefile.config.

Then run
$ make

( or e.g. “make -j 8” if you have 8 CPU cores available and want to finish quicker )

2.6 Important step Modify the Makefile.config as per your system environment like
for Intel MKL and Intel MPI

I changed the following lines

  • If using the external Math library

If linking with Intel MKL threaded library
LIBS =LIBS = -L/home/anisha/software/composer_xe_2013.1.117/mkl/lib/intel64
-lmkl_intel_ilp64 -lmkl_intel_thread -lmkl_core -liomp5 -lpthread
OR if linking with Intel MKL sequential library
LIBS = -L/home/anisha/software/composer_xe_2013.1.117/mkl/lib/intel64 –
lmkl_intel_ilp64 -lmkl_sequential -lmkl_core -lpthread

  •   If using the Intel MPI then use the linker flag -ilp64 bit ( for other MPI  implementation refer their manuals)

LDFLAGS=-i_dynamic -cxxlib -lstdc++ -Wl,-E -ilp64

  •  Add –i8 flag in

F77FLAGS_SAFE =…… -i8
F90FLAGS_SAFE = ….. –i8

  •  Add –DMKL_ILP64 flag in

CFLAGS_SAFE=-DMKL_ILP64

2.7  Then  do make

$ make

2.8 After successful compilation it generate dirac.x executable in install directory

3. Building dirac_mointegral_export.x utility for MRCC

[anisha@gpu1-cn utils]$ mpiifort -i8 -c dirac_mointegral_export.F90

 [anisha@gpu1-cn utils]$ mpiifort -i8 -o dirac_mointegral_export.x  \

                                               dirac_mointegral_export.o

Note : Pay attention to compiler flag. While executing dirac_mointegral_export.x  
if it abort with memory leak means you missed a flag while compiling the utility.

4. Execution

4.1 How to execute DIRAC test 

4.1.1 Modify pam script generated in Install directory

 272         DIRAC_CMD=”$DIRAC_CMD -np $MPI_NPROC -wd $WRK”
 To
 272         DIRAC_CMD=”$DIRAC_CMD -np $MPI_NPROC -wdir $WRK”

4.1.2  Set variable BASDIR

$ export BASDIR= <path-to-DIRAC10>/basis:<path-to-DIRAC10>/basis_dalton

4.1.3 Execute the tests : if all tests pass then DIRAC can be used in production

[anisha@gpu1-cn DIRAC10]$ ./testlast –mpi=2 –tests=short

[anisha@gpu1-cn DIRAC10]$ ./testlast –mpi=2 –tests=medium

[anisha@gpu1-cn DIRAC10]$ ./testlast –mpi=2 –tests=long

It will generate the log file test/log directory .

4.2 How to execute DIRAC 

[anisha@gpu1-cn install]$ ./pam -mpi 2 test.<mol/xyz>  input.inp

it will generate the output file named input_test.out .  Refer this file for detail about run.

5. Known error and solutions

5.1. Error : Cannot open file “ dft/dft_cfg.F90 “ Check include “ DIRAC_MPI”
Solution : Take a DIRAC tar file then again configure and compile.

5.2.  Error : Abnormal exit . Look into the DIRAC output file “ 
inputfilename_molfilename.out” if it says -wd option not recognized .
Solution : in pam and pam.in script replace “-wd” option with “wdir”.

5.3. Error :  MEMGET ERROR, insufficient free space for next allocation( Need:  51378658, available (LFREE):  41405492 )  **** To overcome this memory crash: ****…please increase the current memory allocation (pam ‘mw’ or ‘nw’ parameter) by at least   11. megawords.  Well, this is serial run. Increase the pam ‘mw’ parameter — SEVERE ERROR, PROGRAM WILL BE ABORTED
Solution: Execute with( means increase the memory size)
                ./pam -mw 100  XXX.mol XXX.inp

5.4.   Error

WARNING: Self test failed! Call to LAPACK(DSYEVR) failed with integer*8, but  worked with integer*4  arguments. However, DIRAC is configured to use integer*8 (64 bit integers).  You can fix this mismatch by linking to a integer*8 LAPACK library, or recompile DIRAC using integer*4. Cannot continue without a working LAPACK, quitting.
— SEVERE ERROR, PROGRAM WILL BE ABORTED —
  Date and time (Linux) : Tue Dec 25 12:56:04 2012
  LAPACK test failed Intentionally causing a crash with a Segmentation Fault

Solution : Link with proper LAPACK library . Please refer the above section 2.6
on modify Makefile.config

 5.5.  Error :  If output file  inputfile_molfile.out contains the below output and the
program hang while running for MPI .
*** INFO *** No trial vectors found. Using bare nucleus approximation for initial
trial vectors.  Improved by an estimate of the electronic screening (Slater’s ules).
 ########## START ITERATION NO.   1 #######   Thu Jan 10 15:47:49 2013
It.   1    -19.05160242480      1.00D+20  0.00D+00  0.00D+00               0.07000005s   Scr. nuclei    Thu Jan 10
########## START ITERATION NO.   2 ######### Thu Jan 10 15:47:49 2013
 
* GETGAB: label “GABAO1XX” not found; calling GABGEN.
 
SCR        scr.thr.    Step1    Step2  Coulomb  Exchange   WALL-time
 
 QM-QM nuclear repulsion energy :   69.849464250732
 QM-pc interaction energy       :    0.000000000000
 pc-pc interaction energy       :    0.000000000000
 Total interaction energy       :   69.849464250732

Solution 😦 Refer above section 2.6 on Modify Makefile.config)
Issue is 32-bit and 64-bit mismatch. Trying to build DIRAC with 64-bit but the linking of MPI library is not proper even though the MPI library build with 64-bit.Refer MPI reference manual and check for MPI flag which needs to be used for fortran integer 8 (ILP64 support in case of intel MPI) compiler flag

 For Intel MPI Fortran compiler

 Compiler flag is : -i8
Linker flag is : -ilp64 should be used

 
The following things can happen: If linking is not done properly

  •  Your math library is compiled for 64-bit integers and DIRAC will link and run properly.
  • Your math library is not compiled for 64-bit integers and DIRAC will not link.
  • Your math library is not compiled for 64-bit integers and DIRAC will link but stop at runtime.
  • Your math library is not compiled for 64-bit integers and DIRAC will link and not stop at runtime because you deactivated the self-test and may produce wrong numbers.

 
For cases 2-4 you may have to either:
 

  • Verify linking.
  • Compile your own math library.
  • Use DIRAC’s internal math implementation (slow).
  • Go back to 32-bit integers.
Posted in HPC Applications | Leave a comment

Tips for Intel MKL 10.x Linking on Linux

Posted on February 18, 2013 by nishaagrawal

Intel MKL 10.x  Linking on Linux

1. Tips

To link with Intel MKL , you can choose pure layered  model or the default model which is  backward compatible on link line.  (except cluster components).

  • For the pure layered model, you need to choose one library from the Interface layer, one library from the Threading layer, the Computational layer library (no choice here), and add run-time libraries.
  • In case of the default model, you need not change the link line (see the Dummy Libraries section  for details).Dummy Library means its default links to some specified libraries. Like Dummy library libmkl.so contains references to following Intel MKL libraries

               libmkl.so  ==>         lib/em64t/libmkl_intel_lp64.so,             
                                                 lib/em64t/libmkl_intel_thread.so, and
                                                 lib/em64t/libmkl_core.so.

  • In case of employing the pure layered model for static linking, the interface layer, threading layer, and computation layer libraries must be enclosed in grouping symbols, for example:

-Wl,–start-group -lmkl_intel_ilp64.a -lmkl_intel_thread.a -lmkl_core.a -Wl,–end-group

  • ILP64 means 64 bit integer and LP means 32 bit integer
  • ILP64 interface is provided for the following two reasons:

         – To support huge data arrays (with more than 2 billion elements)
         – To enable compiling your Fortran code with the -i8 compiler option.

  • The Intel Fortran Compiler supports the -i8 option for changing behavior of the INTEGERtype. By default the standard INTEGER type is 4-byte. The -i8 option makes the compiler treat INTEGER constants,

  • To bind your Fortran code with the ILP64 interface, you must compile your code with the -i8 compiler option. And vice-versa, if your code is compiled with -i8, you can bind it only with the ILP64 interface because the LP64 binary interface requires the INTEGER type to be 4-byte.

  • For the languages of C/C++, Intel MKL provides the MKL_INT type as a counterpart of the INTEGER type for Fortran. MKL_INT is a macro defined as the standard C/C++ type int by default. However, if the MKL_ILP64 macro is defined for the code compilation, MKL_INT is defined as a 64-bit integer type. To define the MKL_ILP64 macro, you may call the compiler with the -DMKL_ILP64 command-line option.

2.  References

Posted in HPC | 1 Comment