OpenMPI, CentOS Linking Error - libmpi.so.0 :cannot open shared object file: No such file or directory

I was installing OpenMPI on CentOS 5.x. After installation, I  tried to  use pirun, I encountered this error
libmpi.so.0 :cannot open shared object file: No such file or directory

The reason for the error is that the the loader is not able to find the libmpi.so because /usr/local/lib is not in its PATH

To resolve the issues, you have to do the following
# vim /etc/ld.so.conf
(Add the /usr/local/lib into the file)
# ldconfig

Building OpenMPI with Intel Compiler (Ver 2)

This is a follow up from Thursday, April 2, 2009 Blog Entry Building OpenMPI with Intel Compiler

Step 1: Download the OpenMPI Software from http://www.open-mpi.org/ . The current stable version at point of writing is OpenMPI 1.3.2

Step 2: Download and Install the Intel Compilers from Intel Website. More information can be taken from Free Non-Commercial Intel Compiler Download

Step 3: Add the Intel Directory Binary Path to the Bash Startup
At my ~/.bash_profile directory, I've added

PATH=$PATH:/opt/intel/Compiler/11.0/081/bin/intel64
At command prompt
# source .bashrc

Step 4: Configuration Information

gunzip -c openmpi-1.2.tar.gz tar xf -
# cd openmpi-1.2
#./configure --prefix=/usr/local CC=icc CXX=icpc F77=ifort FC=ifort
# make all install

Step 5: Setting PATH environment for OpenMPI
At my ~/.bash_profile directory, I've added.

export PATH=/usr/local/bin:${PATH}  
export LD_LIBRARY_PATH=/opt/intel/Compiler/11.0/081/lib/intel64:${LD_LIBRARY_PATH}

(The LD_LIBRARY_PATH must point to /opt/intel/Compiler/11.0/081/lib/intel64/libimf.so)

Step 6: mpicc ........

Step 7: Repeat the procedures on the Compute Nodes

Changing Compilers from Intel to PGI for OpenMPI

Extending from Building OpenMPI with Intel Compiler (Ver 2). Assuming you are using Intel Compiler as the default compiler.

Step 1: If you wish to change to another compiler like PGI (in this example) from the Intel Compiler,
You can issue the commands
# export CC=pgcc
# export CXX=pgCC
# export F77=pgf77
# export FC=pgf90

To check that the openmpi is using the PGI compiler
# /usr/local/openmpi/bin/mpicc --showme
# /usr/local/openmpi/bin/mpiCC --showme
# /usr/local/openmpi/bin/mpif77 --showme
# /usr/local/openmpi/bin/mpif90 --showme


Step 2: If you wish to "return" to the default
# unset CC
# unset CXX
# unset F77
# unset FC


To check that the openmpi is using the PGI compiler
# /usr/local/openmpi/bin/mpicc --showme
# /usr/local/openmpi/bin/mpiCC --showme
# /usr/local/openmpi/bin/mpif77 --showme
# /usr/local/openmpi/bin/mpif90 --showme

OpenMPI 1.4 is ready for download

OpenMPI v1.4 is ready for download. According to the OpenMPI v1.4 README file, the different between v1.4 and the previous version 1.3.4 is the fix for Libtool 2.2.6b's issues.

A Hello World OpenMPI program with Intel

I compiled a simple parallel hello world program to test whether OpenMPI is working well with Intel Compilers using the example taken from Compiler Examples from https://wiki.mst.edu/nic/how_to/compile/openmpi-intel-compile

Step 1: Ensure your OpenMPI is compiled with Intel. Read the Building OpenMPI with Intel Compiler (Ver 2) for more information


Step 2: Cut and paste the parallel program taken from https://wiki.mst.edu/nic/how_to/compile/openmpi-intel-compile. Compile the C++ program with mpi

$ mpicxx -o openmpi-intel-hello mpi_hello.cpp

Step 3: Test on SMP Machine

$ mpirun -np 8 open-intel-hello

Step 4: Test on  Distributed Cluster

$ mpirun -np 8 -hostfile hostfile.file open-intel-hello

You should see some output something like

Returned: 0 Hello World! I am 1 of 8
Returned: 0 Hello World! I am 6 of 8
Returned: 0 Hello World! I am 3 of 8
Returned: 0 Hello World! I am 0 of 8
Returned: 0 Hello World! I am 2 of 8
Returned: 0 Hello World! I am 5 of 8
Returned: 0 Hello World! I am 4 of 8
Returned: 0 Hello World! I am 7 of 8

Compiling other compilers with OpenMPI

To compile Intel Compiler with OpenMPI, see blog entry Building OpenMPI with Intel Compiler (Ver 2)

For more information on compiling with other compilers: see FAQ No 65 "Open MPI seems to default to building with the GNU compiler set. Can I use other compilers?"

In summary, the 4 switches on the configure command line are used to specify the compiler suite:

•CC: Specifies the C compiler
•CXX: Specifies the C++ compiler
•F77: Specifies the Fortran 77 compiler
•FC: Specifies the Fortran 90 compiler

For Portland Group Compiler,

# Portland compilers 
# ./configure CC=pgcc CXX=pgCC F77=pgf77 FC=pgf90

For Pathscale Compilers

# Pathscale compilers
# ./configure CC=pathcc CXX=pathCC F77=pathf90 FC=pathf90

For Sun Compilers

# Sun compilers
# ./configure CC=cc CXX=CC F77=f77 FC=f90

Note: The Open MPI team recommends using a single compiler suite whenever possible. Unexpeced or undefined behavior can occur when you mix compiler suites in unsupported ways (e.g., mixing Fortran 77 and Fortran 90 compilers between different compiler suites is almost guaranteed not to work).

forrtl: severe (24): end-of-file during read & forrtl

forrtl: severe (24): end-of-file during read & forrtl: severe (67): input statement requires too much data....Read this forum entry for more information. http://software.intel.com/en-us/forums/showthread.php?t=63981

Solving error while loading shared libraries: libmpi_f77.so.0

If you encontered this error "error while loading shared libraries: libmpi_f77.so.0: cannot open shared object file: No such file or directory".

This is due to missing /usr/local/lib files which are not accessible by the compute nodes. After you have made the libraries available, you may have to add /usr/local/lib to your LD_LIBRARY_PATH environment


You should be able to eliminate the error.

Compiling Infiniband or OpenIB with OpenMPI and Intel Compilers on CentOS

Building on the Blog Entry "Useful Information to Compile Infinifband with OpenMPI", here is a more detailed writeup

I'm assuming you have compiled the Intel Compilers and Unzip the OpenMPI Package. Some of these information can be found on simple Ethernet-based Building OpenMPI with Intel Compiler (Ver 2) .
Don't configure and make yet for the OpenMPI Package.


Firstly to compile with OpenIB support,

#./configure --prefix=/usr/mpi/intel/ \ 
CC=icc CXX=icpc F77=ifort FC=ifort \
--with-openib \ 
--with-openib-libdir=/usr/lib64/

# make all install 

To test whether you have compiled and installed correctly, you can run the "ompi_info" command and look for components for your networks.

# ompi_info | grep openib

You should get something like this depending on your component

MCA btl: openib (MCA v2.0, API v2.0, Component v1.4.1)

You are in good shape.

For more information, see OpenMPI, see

1. 26. How do I build Open MPI with support for Open IB (Infiniband), mVAPI (Infiniband), GM (Myrinet), and/or MX (Myrinet)?

libimf.so: warning: warning: feupdateenv is not implemented and will always fail

After you have compiled OpenMPI with the Intel Compilers, as
and when execuing mpirun on your program, you enconter "libimf.so: warning: warning: feupdateenv is not implemented and will always fail"

$ mpicc hello.c -o hello
libimf.so: warning: warning: feupdateenv is not implemented and will always fail

This resolution to the issue can be found at OpenMPI FAQ to resolve this issue, you have to include the

$ mpicc hello.c -o hello -shared-intel

The default behavior of Open MPI's wrapper compilers can be changed to automatically include this -shared-intel flag so that it is unnecessary to specify it on the command line when linking MPI applications.

Understanding -DOMPI_SKIP_MPICXX Flag for compiler/linker settings

When I was compiling LAMMPS, I come across this Flag when preparing the compiler/linker settings
CCFLAGS =       -g -O -I/usr/local/fftw/include -DFFT_FFTW -DOMPI_SKIP_MPICXX -DOMPI_IGNORE_CXX_SEEK

What do -DFFT_FFTW -DOMPI_SKIP_MPICXX really mean? The answer is nicely given Rolf Vandevaart "Speeding Up C++ Compiles"

In a summary, from the blog

The issue is that if you are compiling C++ code, but you are not using the C++ bindings, the code can be slow to compile.  The workaround is to use the -DOMPI_SKIP_MPICXX flag which tells MPI not to include the C++ header files

And here are the results.  Not quite a 50% speedup, but close

Building Open MPI* with the Intel® compilers

A good hands-on Tutorial from Intel on how to Build OpenMPI with Intel Compilers.
Building Open MPI* with the Intel Compilers

A few things to note:

1. Make sure your path to your Intel Compilers are path-ed already

Understanding -DOMPI_IGNORE_CXX_SEEK Flag for compiler/linker settings

When I was compiling LAMMPS, I come across this Flag when preparing the compiler/linker settings


CCFLAGS =  -g -O -I/usr/local/fftw/include -DFFT_FFTW -DOMPI_SKIP_MPICXX -DOMPI_IGNORE_CXX_SEEK
 
What do -DOMPI_IGNORE_CXX_SEEK really seek to solve. It happens on openmpi-1.2 with Intel. This is due to BOTH stdio.h and the MPI C++ interface use SEEK_SET, SEEK_CUR, SEEK_END.
 
For more inforamtion. see this FAQs from NCSA
 I got an error message: SEEK_SET is #defined but must not be for the C++ binding of MPI. What does that mean?

Useful Information to compile Infiniband with OpenMPI

Taken from the excellent OpenMPI FAQ. "How do I build Open MPI with support for Open IB (Infiniband), mVAPI (Infiniband), GM (Myrinet), and/or MX (Myrinet)?"

To compile OpenMPI against Infiniband
you only have to specify the directory where its support header files and libraries were installed to Open MPI's configure script

--with-openib=
Build support for OpenFabrics (previously known as "Open IB", for Infiniband and iWARP networks -- note that iWARP support was added in the v1.3 series).

# ./configure --with-openib=/path/to/openib/installation

For more information on myrinet etc, do go to the FAQs.