<html><head><meta http-equiv="Content-Type" content="text/html charset=us-ascii"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><div class="">One a different machine, but with same version of code, I am getting the following error. There is some complaint about unable to open /dev/infiniband/rdma_cm. </div><div class=""><br class=""></div><div class="">Any pointers would be greatly appreciated. </div><div class=""><br class=""></div><div class="">Thanks,</div><div class="">Manav</div><div class=""><br class=""></div><div class=""><br class=""></div><div class=""><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);" class="">shadow-login[238] bhatia$ make check</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);" class="">Running test examples to verify correct installation</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);" class="">Using PETSC_DIR=/work/bhatia/codes/shadow/petsc/petsc-3.5.3 and PETSC_ARCH=arch-linux2-cxx-opt</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);" class="">Possible error running C/C++ src/snes/examples/tutorials/ex19 with 1 MPI process</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);" class="">See <a href="http://www.mcs.anl.gov/petsc/documentation/faq.html" class="">http://www.mcs.anl.gov/petsc/documentation/faq.html</a></div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);" class="">librdmacm: Warning: couldn't read ABI version.</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);" class="">librdmacm: Warning: assuming: 4</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);" class="">librdmacm: Fatal: unable to open /dev/infiniband/rdma_cm</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);" class="">librdmacm: Fatal: unable to open /dev/infiniband/rdma_cm</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);" class="">--------------------------------------------------------------------------</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);" class="">WARNING: There are more than one active ports on host 'shadow-login', but the</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);" class="">default subnet GID prefix was detected on more than one of these</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);" class="">ports. If these ports are connected to different physical IB</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);" class="">networks, this configuration will fail in Open MPI. This version of</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);" class="">Open MPI requires that every physically separate IB subnet that is</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);" class="">used between connected MPI processes must have different subnet ID</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);" class="">values.</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0); min-height: 14px;" class=""><br class=""></div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);" class="">Please see this FAQ entry for more details:</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0); min-height: 14px;" class=""><br class=""></div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);" class=""> <a href="http://www.open-mpi.org/faq/?category=openfabrics#ofa-default-subnet-gid" class="">http://www.open-mpi.org/faq/?category=openfabrics#ofa-default-subnet-gid</a></div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0); min-height: 14px;" class=""><br class=""></div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);" class="">NOTE: You can turn off this warning by setting the MCA parameter</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);" class=""> btl_openib_warn_default_gid_prefix to 0.</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);" class="">--------------------------------------------------------------------------</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);" class="">--------------------------------------------------------------------------</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);" class="">No OpenFabrics connection schemes reported that they were able to be</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);" class="">used on a specific port. As such, the openib BTL (OpenFabrics</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);" class="">support) will be disabled for this port.</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0); min-height: 14px;" class=""><br class=""></div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);" class=""> Local host: shadow-login</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);" class=""> Local device: mlx4_1</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);" class=""> Local port: 1</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);" class=""> CPCs attempted: udcm</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);" class="">--------------------------------------------------------------------------</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);" class="">--------------------------------------------------------------------------</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);" class="">WARNING: It appears that your OpenFabrics subsystem is configured to only</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);" class="">allow registering part of your physical memory. This can cause MPI jobs to</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);" class="">run with erratic performance, hang, and/or crash.</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0); min-height: 14px;" class=""><br class=""></div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);" class="">This may be caused by your OpenFabrics vendor limiting the amount of</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);" class="">physical memory that can be registered. You should investigate the</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);" class="">relevant Linux kernel module parameters that control how much physical</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);" class="">memory can be registered, and increase them to allow registering all</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);" class="">physical memory on your machine.</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0); min-height: 14px;" class=""><br class=""></div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);" class="">See this Open MPI FAQ item for more information on these Linux kernel module</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);" class="">parameters:</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0); min-height: 14px;" class=""><br class=""></div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);" class=""> <a href="http://www.open-mpi.org/faq/?category=openfabrics#ib-locked-pages" class="">http://www.open-mpi.org/faq/?category=openfabrics#ib-locked-pages</a></div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0); min-height: 14px;" class=""><br class=""></div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);" class=""> Local host: shadow-login</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);" class=""> Registerable memory: 24576 MiB</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);" class=""> Total memory: 65457 MiB</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0); min-height: 14px;" class=""><br class=""></div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);" class="">Your MPI job will continue, but may be behave poorly and/or hang.</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);" class="">--------------------------------------------------------------------------</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);" class="">lid velocity = 0.0016, prandtl # = 1, grashof # = 1</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);" class="">Number of SNES iterations = 2</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);" class="">Possible error running C/C++ src/snes/examples/tutorials/ex19 with 2 MPI processes</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);" class="">See <a href="http://www.mcs.anl.gov/petsc/documentation/faq.html" class="">http://www.mcs.anl.gov/petsc/documentation/faq.html</a></div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);" class="">librdmacm: Warning: couldn't read ABI version.</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);" class="">librdmacm: Warning: assuming: 4</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);" class="">librdmacm: Fatal: unable to open /dev/infiniband/rdma_cm</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);" class="">librdmacm: Fatal: unable to open /dev/infiniband/rdma_cm</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);" class="">librdmacm: Warning: couldn't read ABI version.</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);" class="">librdmacm: Warning: assuming: 4</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);" class="">librdmacm: Fatal: unable to open /dev/infiniband/rdma_cm</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);" class="">librdmacm: Fatal: unable to open /dev/infiniband/rdma_cm</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);" class="">--------------------------------------------------------------------------</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);" class="">WARNING: There are more than one active ports on host 'shadow-login', but the</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);" class="">default subnet GID prefix was detected on more than one of these</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);" class="">ports. If these ports are connected to different physical IB</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);" class="">networks, this configuration will fail in Open MPI. This version of</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);" class="">Open MPI requires that every physically separate IB subnet that is</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);" class="">used between connected MPI processes must have different subnet ID</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);" class="">values.</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0); min-height: 14px;" class=""><br class=""></div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);" class="">Please see this FAQ entry for more details:</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0); min-height: 14px;" class=""><br class=""></div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);" class=""> <a href="http://www.open-mpi.org/faq/?category=openfabrics#ofa-default-subnet-gid" class="">http://www.open-mpi.org/faq/?category=openfabrics#ofa-default-subnet-gid</a></div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0); min-height: 14px;" class=""><br class=""></div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);" class="">NOTE: You can turn off this warning by setting the MCA parameter</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);" class=""> btl_openib_warn_default_gid_prefix to 0.</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);" class="">--------------------------------------------------------------------------</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);" class="">--------------------------------------------------------------------------</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);" class="">No OpenFabrics connection schemes reported that they were able to be</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);" class="">used on a specific port. As such, the openib BTL (OpenFabrics</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);" class="">support) will be disabled for this port.</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0); min-height: 14px;" class=""><br class=""></div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);" class=""> Local host: shadow-login</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);" class=""> Local device: mlx4_1</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);" class=""> Local port: 1</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);" class=""> CPCs attempted: udcm</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);" class="">--------------------------------------------------------------------------</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);" class="">--------------------------------------------------------------------------</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);" class="">WARNING: It appears that your OpenFabrics subsystem is configured to only</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);" class="">allow registering part of your physical memory. This can cause MPI jobs to</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);" class="">run with erratic performance, hang, and/or crash.</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0); min-height: 14px;" class=""><br class=""></div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);" class="">This may be caused by your OpenFabrics vendor limiting the amount of</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);" class="">physical memory that can be registered. You should investigate the</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);" class="">relevant Linux kernel module parameters that control how much physical</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);" class="">memory can be registered, and increase them to allow registering all</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);" class="">physical memory on your machine.</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0); min-height: 14px;" class=""><br class=""></div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);" class="">See this Open MPI FAQ item for more information on these Linux kernel module</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);" class="">parameters:</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0); min-height: 14px;" class=""><br class=""></div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);" class=""> <a href="http://www.open-mpi.org/faq/?category=openfabrics#ib-locked-pages" class="">http://www.open-mpi.org/faq/?category=openfabrics#ib-locked-pages</a></div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0); min-height: 14px;" class=""><br class=""></div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);" class=""> Local host: shadow-login</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);" class=""> Registerable memory: 24576 MiB</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);" class=""> Total memory: 65457 MiB</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0); min-height: 14px;" class=""><br class=""></div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);" class="">Your MPI job will continue, but may be behave poorly and/or hang.</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);" class="">--------------------------------------------------------------------------</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);" class="">lid velocity = 0.0016, prandtl # = 1, grashof # = 1</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);" class="">Number of SNES iterations = 2</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);" class="">--------------------------------------------------------------------------</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);" class="">mpirun noticed that process rank 1 with PID 196567 on node shadow-login exited on signal 11 (Segmentation fault).</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);" class="">--------------------------------------------------------------------------</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);" class="">[shadow-login:196564] 1 more process has sent help message help-mpi-btl-openib.txt / default subnet prefix</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);" class="">[shadow-login:196564] Set MCA parameter "orte_base_help_aggregate" to 0 to see all help / error messages</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);" class="">[shadow-login:196564] 1 more process has sent help message help-mpi-btl-openib-cpc-base.txt / no cpcs for port</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);" class="">[shadow-login:196564] 1 more process has sent help message help-mpi-btl-openib.txt / reg mem limit low</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);" class="">Possible error running Fortran example src/snes/examples/tutorials/ex5f with 1 MPI process</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);" class="">See <a href="http://www.mcs.anl.gov/petsc/documentation/faq.html" class="">http://www.mcs.anl.gov/petsc/documentation/faq.html</a></div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);" class="">librdmacm: Warning: couldn't read ABI version.</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);" class="">librdmacm: Warning: assuming: 4</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);" class="">librdmacm: Fatal: unable to open /dev/infiniband/rdma_cm</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);" class="">librdmacm: Fatal: unable to open /dev/infiniband/rdma_cm</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);" class="">--------------------------------------------------------------------------</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);" class="">WARNING: There are more than one active ports on host 'shadow-login', but the</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);" class="">default subnet GID prefix was detected on more than one of these</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);" class="">ports. If these ports are connected to different physical IB</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);" class="">networks, this configuration will fail in Open MPI. This version of</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);" class="">Open MPI requires that every physically separate IB subnet that is</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);" class="">used between connected MPI processes must have different subnet ID</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);" class="">values.</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0); min-height: 14px;" class=""><br class=""></div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);" class="">Please see this FAQ entry for more details:</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0); min-height: 14px;" class=""><br class=""></div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);" class=""> <a href="http://www.open-mpi.org/faq/?category=openfabrics#ofa-default-subnet-gid" class="">http://www.open-mpi.org/faq/?category=openfabrics#ofa-default-subnet-gid</a></div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0); min-height: 14px;" class=""><br class=""></div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);" class="">NOTE: You can turn off this warning by setting the MCA parameter</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);" class=""> btl_openib_warn_default_gid_prefix to 0.</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);" class="">--------------------------------------------------------------------------</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);" class="">--------------------------------------------------------------------------</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);" class="">No OpenFabrics connection schemes reported that they were able to be</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);" class="">used on a specific port. As such, the openib BTL (OpenFabrics</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);" class="">support) will be disabled for this port.</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0); min-height: 14px;" class=""><br class=""></div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);" class=""> Local host: shadow-login</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);" class=""> Local device: mlx4_1</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);" class=""> Local port: 1</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);" class=""> CPCs attempted: udcm</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);" class="">--------------------------------------------------------------------------</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);" class="">--------------------------------------------------------------------------</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);" class="">WARNING: It appears that your OpenFabrics subsystem is configured to only</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);" class="">allow registering part of your physical memory. This can cause MPI jobs to</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);" class="">run with erratic performance, hang, and/or crash.</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0); min-height: 14px;" class=""><br class=""></div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);" class="">This may be caused by your OpenFabrics vendor limiting the amount of</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);" class="">physical memory that can be registered. You should investigate the</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);" class="">relevant Linux kernel module parameters that control how much physical</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);" class="">memory can be registered, and increase them to allow registering all</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);" class="">physical memory on your machine.</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0); min-height: 14px;" class=""><br class=""></div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);" class="">See this Open MPI FAQ item for more information on these Linux kernel module</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);" class="">parameters:</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0); min-height: 14px;" class=""><br class=""></div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);" class=""> <a href="http://www.open-mpi.org/faq/?category=openfabrics#ib-locked-pages" class="">http://www.open-mpi.org/faq/?category=openfabrics#ib-locked-pages</a></div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0); min-height: 14px;" class=""><br class=""></div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);" class=""> Local host: shadow-login</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);" class=""> Registerable memory: 24576 MiB</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);" class=""> Total memory: 65457 MiB</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0); min-height: 14px;" class=""><br class=""></div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);" class="">Your MPI job will continue, but may be behave poorly and/or hang.</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);" class="">--------------------------------------------------------------------------</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);" class="">Number of SNES iterations = 4</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);" class="">Completed test examples</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);" class="">=========================================</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);" class="">Now to evaluate the computer systems you plan use - do:</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);" class="">make PETSC_DIR=/work/bhatia/codes/shadow/petsc/petsc-3.5.3 PETSC_ARCH=arch-linux2-cxx-opt streams NPMAX=<number of MPI processes you intend to use></div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);" class="">shadow-login[239] bhatia$ </div></div><div class=""><br class=""></div><div class=""><br class=""></div><br class=""><div><blockquote type="cite" class=""><div class="">On Mar 11, 2015, at 1:12 PM, Manav Bhatia <<a href="mailto:bhatiamanav@gmail.com" class="">bhatiamanav@gmail.com</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><meta http-equiv="Content-Type" content="text/html charset=us-ascii" class=""><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">Greetings! <div class=""><br class=""></div><div class=""> I have come across the following error while running make check. I am not sure where to begin to sort this out. Any pointers would be greatly appreciated. </div><div class=""><br class=""></div><div class=""><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);" class="">Talon-login[114] bhatia$ make check</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);" class="">gmake[1]: Entering directory `/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3'</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);" class="">Running test examples to verify correct installation</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);" class="">Using PETSC_DIR=/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3 and PETSC_ARCH=arch-linux2-cxx-opt</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);" class="">gmake[2]: Entering directory `/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/src/snes/examples/tutorials'</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);" class="">gmake[2]: Leaving directory `/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/src/snes/examples/tutorials'</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);" class="">gmake[2]: Entering directory `/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/src/snes/examples/tutorials'</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(195, 55, 32); background-color: rgb(0, 0, 0);" class="">*******************Error detected during compile or link!*******************</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(195, 55, 32); background-color: rgb(0, 0, 0);" class="">See <a href="http://www.mcs.anl.gov/petsc/documentation/faq.html" class="">http://www.mcs.anl.gov/petsc/documentation/faq.html</a></div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(195, 55, 32); background-color: rgb(0, 0, 0);" class="">/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/src/snes/examples/tutorials ex19</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(195, 55, 32); background-color: rgb(0, 0, 0);" class="">*********************************************************************************</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);" class="">gmake[3]: Entering directory `/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/src/snes/examples/tutorials'</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);" class="">gmake[4]: Entering directory `/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/src/snes/examples/tutorials'</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);" class="">gmake[4]: Leaving directory `/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/src/snes/examples/tutorials'</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);" class="">mpicxx -o ex19.o -c -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas -O -I/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/include -I/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/arch-linux2-cxx-opt/include -I/usr/local/openssl/x86_64/include -I/usr/local/mpi/x86_64/openmpi/include -I/usr/local/mpi/x86_64/openmpi-1.4.2/include `pwd`/ex19.c</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);" class="">mpicxx -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas -O -o ex19 ex19.o -L/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/arch-linux2-cxx-opt/lib -lpetsc -Wl,-rpath,/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/arch-linux2-cxx-opt/lib -lcmumps -ldmumps -lsmumps -lzmumps -lmumps_common -lpord -lscalapack -lsuperlu_4.3 -lsuperlu_dist_3.3 -Wl,-rpath,/cavs/projects/sams/codes/raptor/lapack -L/cavs/projects/sams/codes/raptor/lapack -llapack -Wl,-rpath,/cavs/projects/sams/codes/raptor/blas -L/cavs/projects/sams/codes/raptor/blas -lblas -lparmetis -lmetis -lpthread -Wl,-rpath,/usr/local/openssl/x86_64/lib -L/usr/local/openssl/x86_64/lib -lssl -lcrypto -Wl,-rpath,/usr/local/mpi/x86_64/openmpi/lib -L/usr/local/mpi/x86_64/openmpi/lib -lmpi_cxx -lmpi -L/usr/local/mpi/x86_64/openmpi-1.4.2/lib -L/usr/local/server/x86_64-SuSE10/gnu/gcc-4.8.3/lib/gcc/x86_64-unknown-linux-gnu/4.8.3 -L/usr/local/server/x86_64-SuSE10/gnu/gcc-4.8.3/lib/gcc -L/usr/local/server/x86_64-SuSE10/gnu/gcc-4.8.3/lib64 -L/usr/local/server/x86_64-SuSE10/gnu/gcc-4.8.3/lib -lmpi_f90 -lmpi_f77 -lgfortran -lm -lgfortran -lm -lgfortran -lm -lquadmath -lm -lmpi_cxx -lstdc++ -L/usr/local/mpi/x86_64/openmpi-1.4.2/lib -L/usr/local/server/x86_64-SuSE10/gnu/gcc-4.8.3/lib/gcc/x86_64-unknown-linux-gnu/4.8.3 -L/usr/local/server/x86_64-SuSE10/gnu/gcc-4.8.3/lib/gcc -L/usr/local/server/x86_64-SuSE10/gnu/gcc-4.8.3/lib64 -L/usr/local/server/x86_64-SuSE10/gnu/gcc-4.8.3/lib -ldl -lmpi -lopen-rte -lopen-pal -lnsl -lutil -lgcc_s -lpthread -ldl </div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);" class="">/usr/local/gnu/gcc-4.8.3/binutils/bin/ld: warning: libgfortran.so.1, needed by /usr/local/mpi/x86_64/openmpi/lib/libmpi_f90.so, may conflict with libgfortran.so.3</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);" class="">/usr/local/gnu/gcc-4.8.3/binutils/bin/ld: warning: libgfortran.so.1, needed by /usr/local/mpi/x86_64/openmpi/lib/libmpi_f90.so, may conflict with libgfortran.so.3</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);" class="">/usr/local/gnu/gcc-4.8.3/binutils/bin/ld: warning: libgfortran.so.1, needed by /usr/local/mpi/x86_64/openmpi/lib/libmpi_f90.so, may conflict with libgfortran.so.3</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);" class="">/bin/rm -f ex19.o</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);" class="">gmake[3]: Leaving directory `/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/src/snes/examples/tutorials'</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);" class="">Possible error running C/C++ src/snes/examples/tutorials/ex19 with 1 MPI process</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);" class="">See <a href="http://www.mcs.anl.gov/petsc/documentation/faq.html" class="">http://www.mcs.anl.gov/petsc/documentation/faq.html</a></div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);" class="">libibverbs: Warning: RLIMIT_MEMLOCK is 32768 bytes.</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);" class=""> This will severely limit memory registrations.</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);" class="">libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'.</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);" class="">libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs0</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);" class="">CMA: unable to open /dev/infiniband/rdma_cm</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);" class="">libibverbs: Warning: RLIMIT_MEMLOCK is 32768 bytes.</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);" class=""> This will severely limit memory registrations.</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);" class="">libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'.</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);" class="">[Talon-login:18281] *** Process received signal ***</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);" class="">[Talon-login:18281] Signal: Segmentation fault (11)</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);" class="">[Talon-login:18281] Signal code: Address not mapped (1)</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);" class="">[Talon-login:18281] Failing at address: 0x100002990</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);" class="">[Talon-login:18281] [ 0] /lib64/libpthread.so.0 [0x2b8e9cbc6c00]</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);" class="">[Talon-login:18281] [ 1] /usr/ofed/lib64/libibverbs.so.1 [0x2b8e9fcad4ce]</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);" class="">[Talon-login:18281] [ 2] /usr/ofed/lib64/libibverbs.so.1(ibv_get_device_list+0x9e) [0x2b8e9fcac49e]</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);" class="">[Talon-login:18281] [ 3] /usr/local/mpi/x86_64/openmpi-1.4.2/lib/openmpi/mca_btl_openib.so [0x2b8e9fdbe15a]</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);" class="">[Talon-login:18281] [ 4] /usr/local/mpi/x86_64/openmpi/lib/libmpi.so.0(mca_btl_base_select+0x156) [0x2b8e9d223166]</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);" class="">[Talon-login:18281] [ 5] /usr/local/mpi/x86_64/openmpi-1.4.2/lib/openmpi/mca_bml_r2.so [0x2b8e9f997821]</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);" class="">[Talon-login:18281] [ 6] /usr/local/mpi/x86_64/openmpi/lib/libmpi.so.0(mca_bml_base_init+0x72) [0x2b8e9d222972]</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);" class="">[Talon-login:18281] [ 7] /usr/local/mpi/x86_64/openmpi-1.4.2/lib/openmpi/mca_pml_ob1.so [0x2b8e9f77fc8f]</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);" class="">[Talon-login:18281] [ 8] /usr/local/mpi/x86_64/openmpi/lib/libmpi.so.0(mca_pml_base_select+0x383) [0x2b8e9d22c743]</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);" class="">[Talon-login:18281] [ 9] /usr/local/mpi/x86_64/openmpi/lib/libmpi.so.0 [0x2b8e9d1f0050]</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);" class="">[Talon-login:18281] [10] /usr/local/mpi/x86_64/openmpi/lib/libmpi.so.0(PMPI_Init_thread+0xc1) [0x2b8e9d20db41]</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);" class="">[Talon-login:18281] [11] ./ex19(PetscInitialize+0x1bd) [0x4990fe]</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);" class="">[Talon-login:18281] [12] ./ex19(main+0x2a) [0x48471e]</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);" class="">[Talon-login:18281] [13] /lib64/libc.so.6(__libc_start_main+0xf4) [0x2b8e9e331164]</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);" class="">[Talon-login:18281] [14] ./ex19 [0x482639]</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);" class="">[Talon-login:18281] *** End of error message ***</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);" class="">--------------------------------------------------------------------------</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);" class="">mpirun noticed that process rank 0 with PID 18281 on node <a href="http://talon-login.hpc.msstate.edu/" class="">Talon-login.HPC.MsState.Edu</a> exited on signal 11 (Segmentation fault).</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);" class="">--------------------------------------------------------------------------</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);" class="">Possible error running C/C++ src/snes/examples/tutorials/ex19 with 2 MPI processes</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);" class="">See <a href="http://www.mcs.anl.gov/petsc/documentation/faq.html" class="">http://www.mcs.anl.gov/petsc/documentation/faq.html</a></div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);" class="">libibverbs: Warning: RLIMIT_MEMLOCK is 32768 bytes.</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);" class=""> This will severely limit memory registrations.</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);" class="">libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'.</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);" class="">libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs0</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);" class="">CMA: unable to open /dev/infiniband/rdma_cm</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);" class="">libibverbs: Warning: RLIMIT_MEMLOCK is 32768 bytes.</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);" class=""> This will severely limit memory registrations.</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);" class="">libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'.</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);" class="">[Talon-login:18286] *** Process received signal ***</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);" class="">[Talon-login:18286] Signal: Segmentation fault (11)</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);" class="">[Talon-login:18286] Signal code: Address not mapped (1)</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);" class="">[Talon-login:18286] Failing at address: 0x100002990</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);" class="">[Talon-login:18286] [ 0] /lib64/libpthread.so.0 [0x2acc0604ec00]</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);" class="">[Talon-login:18286] [ 1] /usr/ofed/lib64/libibverbs.so.1 [0x2acc091354ce]</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);" class="">[Talon-login:18286] [ 2] /usr/ofed/lib64/libibverbs.so.1(ibv_get_device_list+0x9e) [0x2acc0913449e]</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);" class="">[Talon-login:18286] [ 3] /usr/local/mpi/x86_64/openmpi-1.4.2/lib/openmpi/mca_btl_openib.so [0x2acc0924615a]</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);" class="">[Talon-login:18286] [ 4] /usr/local/mpi/x86_64/openmpi/lib/libmpi.so.0(mca_btl_base_select+0x156) [0x2acc066ab166]</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);" class="">[Talon-login:18286] [ 5] /usr/local/mpi/x86_64/openmpi-1.4.2/lib/openmpi/mca_bml_r2.so [0x2acc08e1f821]</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);" class="">[Talon-login:18286] [ 6] /usr/local/mpi/x86_64/openmpi/lib/libmpi.so.0(mca_bml_base_init+0x72) [0x2acc066aa972]</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);" class="">[Talon-login:18286] [ 7] /usr/local/mpi/x86_64/openmpi-1.4.2/lib/openmpi/mca_pml_ob1.so [0x2acc08c07c8f]</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);" class="">[Talon-login:18286] [ 8] /usr/local/mpi/x86_64/openmpi/lib/libmpi.so.0(mca_pml_base_select+0x383) [0x2acc066b4743]</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);" class="">[Talon-login:18286] [ 9] /usr/local/mpi/x86_64/openmpi/lib/libmpi.so.0 [0x2acc06678050]</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);" class="">[Talon-login:18286] [10] /usr/local/mpi/x86_64/openmpi/lib/libmpi.so.0(PMPI_Init_thread+0xc1) [0x2acc06695b41]</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);" class="">[Talon-login:18286] [11] ./ex19(PetscInitialize+0x1bd) [0x4990fe]</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);" class="">[Talon-login:18286] [12] ./ex19(main+0x2a) [0x48471e]</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);" class="">[Talon-login:18286] [13] /lib64/libc.so.6(__libc_start_main+0xf4) [0x2acc077b9164]</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);" class="">[Talon-login:18286] [14] ./ex19 [0x482639]</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);" class="">[Talon-login:18286] *** End of error message ***</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);" class="">--------------------------------------------------------------------------</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);" class="">mpirun noticed that process rank 0 with PID 18286 on node <a href="http://talon-login.hpc.msstate.edu/" class="">Talon-login.HPC.MsState.Edu</a> exited on signal 11 (Segmentation fault).</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);" class="">--------------------------------------------------------------------------</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);" class="">gmake[3]: Entering directory `/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/src/snes/examples/tutorials'</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);" class="">gmake[3]: Leaving directory `/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/src/snes/examples/tutorials'</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);" class="">gmake[2]: Leaving directory `/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/src/snes/examples/tutorials'</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);" class="">gmake[2]: Entering directory `/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/src/snes/examples/tutorials'</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(195, 55, 32); background-color: rgb(0, 0, 0);" class="">*******************Error detected during compile or link!*******************</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(195, 55, 32); background-color: rgb(0, 0, 0);" class="">See <a href="http://www.mcs.anl.gov/petsc/documentation/faq.html" class="">http://www.mcs.anl.gov/petsc/documentation/faq.html</a></div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(195, 55, 32); background-color: rgb(0, 0, 0);" class="">/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/src/snes/examples/tutorials ex5f</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(195, 55, 32); background-color: rgb(0, 0, 0);" class="">*********************************************************</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);" class="">gmake[3]: Entering directory `/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/src/snes/examples/tutorials'</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);" class="">mpif90 -c -Wall -Wno-unused-variable -ffree-line-length-0 -Wno-unused-dummy-argument -O -I/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/include -I/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/arch-linux2-cxx-opt/include -I/usr/local/openssl/x86_64/include -I/usr/local/mpi/x86_64/openmpi/include -I/usr/local/mpi/x86_64/openmpi-1.4.2/include -o ex5f.o ex5f.F</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);" class="">mpif90 -Wall -Wno-unused-variable -ffree-line-length-0 -Wno-unused-dummy-argument -O -o ex5f ex5f.o -L/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/arch-linux2-cxx-opt/lib -lpetsc -Wl,-rpath,/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/arch-linux2-cxx-opt/lib -lcmumps -ldmumps -lsmumps -lzmumps -lmumps_common -lpord -lscalapack -lsuperlu_4.3 -lsuperlu_dist_3.3 -Wl,-rpath,/cavs/projects/sams/codes/raptor/lapack -L/cavs/projects/sams/codes/raptor/lapack -llapack -Wl,-rpath,/cavs/projects/sams/codes/raptor/blas -L/cavs/projects/sams/codes/raptor/blas -lblas -lparmetis -lmetis -lpthread -Wl,-rpath,/usr/local/openssl/x86_64/lib -L/usr/local/openssl/x86_64/lib -lssl -lcrypto -Wl,-rpath,/usr/local/mpi/x86_64/openmpi/lib -L/usr/local/mpi/x86_64/openmpi/lib -lmpi_cxx -lmpi -L/usr/local/mpi/x86_64/openmpi-1.4.2/lib -L/usr/local/server/x86_64-SuSE10/gnu/gcc-4.8.3/lib/gcc/x86_64-unknown-linux-gnu/4.8.3 -L/usr/local/server/x86_64-SuSE10/gnu/gcc-4.8.3/lib/gcc -L/usr/local/server/x86_64-SuSE10/gnu/gcc-4.8.3/lib64 -L/usr/local/server/x86_64-SuSE10/gnu/gcc-4.8.3/lib -lmpi_f90 -lmpi_f77 -lgfortran -lm -lgfortran -lm -lgfortran -lm -lquadmath -lm -lmpi_cxx -lstdc++ -L/usr/local/mpi/x86_64/openmpi-1.4.2/lib -L/usr/local/server/x86_64-SuSE10/gnu/gcc-4.8.3/lib/gcc/x86_64-unknown-linux-gnu/4.8.3 -L/usr/local/server/x86_64-SuSE10/gnu/gcc-4.8.3/lib/gcc -L/usr/local/server/x86_64-SuSE10/gnu/gcc-4.8.3/lib64 -L/usr/local/server/x86_64-SuSE10/gnu/gcc-4.8.3/lib -ldl -lmpi -lopen-rte -lopen-pal -lnsl -lutil -lgcc_s -lpthread -ldl </div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);" class="">/usr/local/gnu/gcc-4.8.3/binutils/bin/ld: warning: libgfortran.so.1, needed by /usr/local/mpi/x86_64/openmpi/lib/libmpi_f90.so, may conflict with libgfortran.so.3</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);" class="">/usr/local/gnu/gcc-4.8.3/binutils/bin/ld: warning: libgfortran.so.1, needed by /usr/local/mpi/x86_64/openmpi/lib/libmpi_f90.so, may conflict with libgfortran.so.3</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);" class="">/usr/local/gnu/gcc-4.8.3/binutils/bin/ld: warning: libgfortran.so.1, needed by /usr/local/mpi/x86_64/openmpi/lib/libmpi_f90.so, may conflict with libgfortran.so.3</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);" class="">/usr/local/gnu/gcc-4.8.3/binutils/bin/ld: warning: libgfortran.so.1, needed by /usr/local/mpi/x86_64/openmpi/lib/libmpi_f90.so, may conflict with libgfortran.so.3</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);" class="">/usr/local/gnu/gcc-4.8.3/binutils/bin/ld: warning: libgfortran.so.1, needed by /usr/local/mpi/x86_64/openmpi/lib/libmpi_f90.so, may conflict with libgfortran.so.3</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);" class="">/usr/local/gnu/gcc-4.8.3/binutils/bin/ld: warning: libgfortran.so.1, needed by /usr/local/mpi/x86_64/openmpi/lib/libmpi_f90.so, may conflict with libgfortran.so.3</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);" class="">/bin/rm -f ex5f.o</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);" class="">gmake[3]: Leaving directory `/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/src/snes/examples/tutorials'</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);" class="">Possible error running Fortran example src/snes/examples/tutorials/ex5f with 1 MPI process</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);" class="">See <a href="http://www.mcs.anl.gov/petsc/documentation/faq.html" class="">http://www.mcs.anl.gov/petsc/documentation/faq.html</a></div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);" class="">libibverbs: Warning: RLIMIT_MEMLOCK is 32768 bytes.</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);" class=""> This will severely limit memory registrations.</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);" class="">libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'.</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);" class="">libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs0</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);" class="">CMA: unable to open /dev/infiniband/rdma_cm</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);" class="">libibverbs: Warning: RLIMIT_MEMLOCK is 32768 bytes.</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);" class=""> This will severely limit memory registrations.</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);" class="">libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'.</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0); min-height: 14px;" class=""><br class=""></div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);" class="">Program received signal SIGSEGV: Segmentation fault - invalid memory reference.</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0); min-height: 14px;" class=""><br class=""></div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);" class="">Backtrace for this error:</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);" class="">#0 0x2B6CD4A6F4D7</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);" class="">#1 0x2B6CD4A6FADE</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);" class="">#2 0x2B6CD5805E1F</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);" class="">#3 0x2B6CD716F4CE</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);" class="">#4 0x2B6CD716E49D</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);" class="">#5 0x2B6CD7280159</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);" class="">#6 0x2B6CD46E5165</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);" class="">#7 0x2B6CD6E59820</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);" class="">#8 0x2B6CD46E4971</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);" class="">#9 0x2B6CD6C41C8E</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);" class="">#10 0x2B6CD46EE742</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);" class="">#11 0x2B6CD46B204F</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);" class="">#12 0x2B6CD46CF99F</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);" class="">#13 0x2B6CD494AA84</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);" class="">#14 0x489A6D in petscinitialize_</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);" class="">#15 0x487CEC in MAIN__ at ex5f.F:?</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);" class="">--------------------------------------------------------------------------</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);" class="">mpirun noticed that process rank 0 with PID 18354 on node <a href="http://talon-login.hpc.msstate.edu/" class="">Talon-login.HPC.MsState.Edu</a> exited on signal 11 (Segmentation fault).</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);" class="">--------------------------------------------------------------------------</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);" class="">gmake[3]: Entering directory `/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/src/snes/examples/tutorials'</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);" class="">gmake[3]: Leaving directory `/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/src/snes/examples/tutorials'</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);" class="">gmake[2]: Leaving directory `/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/src/snes/examples/tutorials'</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);" class="">gmake[2]: Entering directory `/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/src/snes/examples/tutorials'</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);" class="">gmake[2]: Leaving directory `/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3/src/snes/examples/tutorials'</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);" class="">Completed test examples</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);" class="">gmake[1]: Leaving directory `/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3'</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);" class="">=========================================</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);" class="">Now to evaluate the computer systems you plan use - do:</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);" class="">make PETSC_DIR=/cavs/projects/sams/codes/raptor/petsc/petsc-3.5.3 PETSC_ARCH=arch-linux2-cxx-opt streams NPMAX=<number of MPI processes you intend to use></div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);" class="">Talon-login[115] bhatia$ </div></div><div class=""><br class=""></div><div class=""><br class=""></div><div class="">Thanks,</div><div class="">Manav</div><div class=""><br class=""></div></div></div></blockquote></div><br class=""></body></html>