<div dir="ltr"><div>Hi Jose,<br><br></div>Here is the complete error message:<br><br>[0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------<br>[0]PETSC ERROR: Invalid argument<br>[0]PETSC ERROR: Scalar value must be same on all processes, argument # 3<br>[0]PETSC ERROR: See <a href="http://www.mcs.anl.gov/petsc/documentation/faq.html">http://www.mcs.anl.gov/petsc/documentation/faq.html</a> for trouble shooting.<br>[0]PETSC ERROR: Petsc Release Version 3.5.2, Sep, 08, 2014<br>[0]PETSC ERROR: Unknown Name on a intel-openmpi_ib named <a href="http://comet-03-60.sdsc.edu">comet-03-60.sdsc.edu</a> by bikashk Thu Jan 28 00:09:17 2016<br>[0]PETSC ERROR: Configure options CFLAGS="-fPIC -xcore-avx2" FFLAGS="-fPIC -xcore-avx2" CXXFLAGS="-fPIC -xcore-avx2" --prefix=/opt/petsc/intel/openmpi_ib --with-mpi=true --download-pastix=../pastix_5.2.2.12.tar.bz2 --download-ptscotch=../scotch_6.0.0_esmumps.tar.gz --with-blas-lib="-Wl,--start-group /opt/intel/composer_xe_2013_sp1.2.144/mkl/lib/intel64/libmkl_intel_lp64.a /opt/intel/composer_xe_2013_sp1.2.144/mkl/lib/intel64/libmkl_sequential.a /opt/intel/composer_xe_2013_sp1.2.144/mkl/lib/intel64/libmkl_core.a -Wl,--end-group -lpthread -lm" --with-lapack-lib="-Wl,--start-group /opt/intel/composer_xe_2013_sp1.2.144/mkl/lib/intel64/libmkl_intel_lp64.a /opt/intel/composer_xe_2013_sp1.2.144/mkl/lib/intel64/libmkl_sequential.a /opt/intel/composer_xe_2013_sp1.2.144/mkl/lib/intel64/libmkl_core.a -Wl,--end-group -lpthread -lm" --with-superlu_dist-include=/opt/superlu/intel/openmpi_ib/include --with-superlu_dist-lib="-L/opt/superlu/intel/openmpi_ib/lib -lsuperlu" --with-parmetis-dir=/opt/parmetis/intel/openmpi_ib --with-metis-dir=/opt/parmetis/intel/openmpi_ib --with-mpi-dir=/opt/openmpi/intel/ib --with-scalapack-dir=/opt/scalapack/intel/openmpi_ib --download-mumps=../MUMPS_4.10.0-p3.tar.gz --download-blacs=../blacs-dev.tar.gz --download-fblaslapack=../fblaslapack-3.4.2.tar.gz --with-pic=true --with-shared-libraries=1 --with-hdf5=true --with-hdf5-dir=/opt/hdf5/intel/openmpi_ib --with-debugging=false<br>[0]PETSC ERROR: #1 BVScaleColumn() line 380 in /scratch/build/git/math-roll/BUILD/sdsc-slepc_intel_openmpi_ib-3.5.3/slepc-3.5.3/src/sys/classes/bv/interface/bvops.c<br>[0]PETSC ERROR: #2 BVOrthogonalize_GS() line 474 in /scratch/build/git/math-roll/BUILD/sdsc-slepc_intel_openmpi_ib-3.5.3/slepc-3.5.3/src/sys/classes/bv/interface/bvorthog.c<br>[0]PETSC ERROR: #3 BVOrthogonalize() line 535 in /scratch/build/git/math-roll/BUILD/sdsc-slepc_intel_openmpi_ib-3.5.3/slepc-3.5.3/src/sys/classes/bv/interface/bvorthog.c<br>[comet-03-60:27927] *** Process received signal ***<br>[comet-03-60:27927] Signal: Aborted (6)<br><br><br></div><div class="gmail_extra"><br><div class="gmail_quote">On Thu, Jan 28, 2016 at 2:56 AM, Jose E. Roman <span dir="ltr"><<a href="mailto:jroman@dsic.upv.es" target="_blank">jroman@dsic.upv.es</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div class="HOEnZb"><div class="h5"><br>
> El 28 ene 2016, a las 8:32, Bikash Kanungo <<a href="mailto:bikash@umich.edu">bikash@umich.edu</a>> escribió:<br>
><br>
> Hi,<br>
><br>
> I was trying to use BVOrthogonalize() function in SLEPc. For smaller problems (10-20 vectors of length < 20,000) I'm able to use it without any trouble. For larger problems ( > 150 vectors of length > 400,000) the code aborts citing an MPI_AllReduce error with following message:<br>
><br>
> Scalar value must be same on all processes, argument # 3.<br>
><br>
> I was skeptical that the PETSc compilation might be faulty and tried to build a minimalistic version omitting the previously used -xcore-avx2 flags in CFLAGS abd CXXFLAGS. That seemed to have done the cure.<br>
><br>
> What perplexes me is that I have been using the same code with -xcore-avx2 flags in PETSc build on a local cluster at the University of Michigan without any problem. It is only until recently when I moved to Xsede's Comet machine, that I started getting this MPI_AllReduce error with -xcore-avx2.<br>
><br>
> Do you have any clue on why the same PETSc build fails on two different machines just because of a build flag?<br>
><br>
> Regards,<br>
> Bikash<br>
><br>
> --<br>
> Bikash S. Kanungo<br>
> PhD Student<br>
> Computational Materials Physics Group<br>
> Mechanical Engineering<br>
> University of Michigan<br>
><br>
<br>
</div></div>Without the complete error message I cannot tell the exact point where it is failing.<br>
<span class="HOEnZb"><font color="#888888">Jose<br>
<br>
</font></span></blockquote></div><br><br clear="all"><br>-- <br><div class="gmail_signature"><div dir="ltr"><div><div><div><div><font color="#666666">Bikash S. Kanungo<br></font></div><font color="#666666">PhD Student<br></font></div><font color="#666666">Computational Materials Physics Group<br></font></div><font color="#666666">Mechanical Engineering <br></font></div><font color="#666666">University of Michigan<br><br></font></div></div>
</div>