<div dir="ltr"><div dir="ltr">On Wed, Oct 13, 2021 at 5:53 AM Roland Richter <<a href="mailto:roland.richter@ntnu.no">roland.richter@ntnu.no</a>> wrote:<br></div><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
  
    
  
  <div>
    <p>Hei,</p>
    <p>I noticed a difference in when the program is running, and when
      not. The code works fine if I compile it via a CMake-file and load
      PETSc there. If I use the compilation line which is included in
      the Makefiles, then the code will fail with the mentioned error.
      The cmake-generated compilation line (including armadillo, because
      my test sample contained armadillo-code) is<br>
    </p>
    <p><i></i></p></div></blockquote><div>One of these is a compile command and the other is a link command.</div><div><br></div><div>   Matt</div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div><p><i>/opt/intel/oneapi/mpi/2021.4.0/bin/mpicxx -D__INSDIR__=""
        -I/include -I/opt/petsc/include -I/opt/armadillo/include
        -std=c++0x -g -MD -MT CMakeFiles/main.dir/source/main.cpp.o -MF
        CMakeFiles/main.dir/source/main.cpp.o.d -o
        CMakeFiles/main.dir/source/main.cpp.o -c source/main.cpp</i><i><br>
      </i><i>/opt/intel/oneapi/mpi/2021.4.0/bin/mpicxx -rdynamic
        CMakeFiles/main.dir/source/main.cpp.o -o main_short 
        -Wl,-rpath,/opt/petsc/lib:/opt/armadillo/lib64
        /opt/petsc/lib/libpetsc.so /opt/armadillo/lib64/libarmadillo.so
      </i><br>
    </p>
    <p>Meanwhile, the original compilation line from PETSc is</p>
    <p><i>mpicxx -mavx2 -march=native -O3 -fPIC -fopenmp   
        -I/opt/petsc/include -I/opt/armadillo/include
        -I/opt/intel/oneapi/mkl/latest/include -I/opt/fftw3/include
        -I/opt/hdf5/include -I/opt/boost/include source/main.cpp
        -Wl,-rpath,/opt/petsc/lib -L/opt/petsc/lib
        -Wl,-rpath,/opt/petsc/lib -L/opt/petsc/lib
        -L/opt/intel/oneapi/mkl/latest/lib/intel64
        -Wl,-rpath,/opt/fftw3/lib64 -L/opt/fftw3/lib64
        -Wl,-rpath,/opt/armadillo/lib64 -L/opt/armadillo/lib64
        -Wl,-rpath,/opt/intel/oneapi/mkl/latest/lib/intel64
        -Wl,-rpath,/opt/hdf5/lib -L/opt/hdf5/lib
        -Wl,-rpath,/opt/intel/oneapi/mpi/2021.4.0/lib/release
        -L/opt/intel/oneapi/mpi/2021.4.0/lib/release
        -Wl,-rpath,/opt/intel/oneapi/mpi/2021.4.0/lib
        -L/opt/intel/oneapi/mpi/2021.4.0/lib
        -Wl,-rpath,/usr/lib64/gcc/x86_64-suse-linux/11
        -L/usr/lib64/gcc/x86_64-suse-linux/11
        -Wl,-rpath,/opt/intel/oneapi/vpl/2021.6.0/lib
        -L/opt/intel/oneapi/vpl/2021.6.0/lib
        -Wl,-rpath,/opt/intel/oneapi/tbb/2021.4.0/lib/intel64/gcc4.8
        -L/opt/intel/oneapi/tbb/2021.4.0/lib/intel64/gcc4.8
        -Wl,-rpath,/opt/intel/oneapi/mpi/2021.4.0/libfabric/lib
        -L/opt/intel/oneapi/mpi/2021.4.0/libfabric/lib
        -Wl,-rpath,/opt/intel/oneapi/mkl/2021.4.0/lib/intel64
        -L/opt/intel/oneapi/mkl/2021.4.0/lib/intel64
        -Wl,-rpath,/opt/intel/oneapi/ipp/2021.4.0/lib/intel64
        -L/opt/intel/oneapi/ipp/2021.4.0/lib/intel64
        -Wl,-rpath,/opt/intel/oneapi/ippcp/2021.4.0/lib/intel64
        -L/opt/intel/oneapi/ippcp/2021.4.0/lib/intel64
        -Wl,-rpath,/opt/intel/oneapi/dnnl/2021.4.0/cpu_dpcpp_gpu_dpcpp/lib
        -L/opt/intel/oneapi/dnnl/2021.4.0/cpu_dpcpp_gpu_dpcpp/lib
        -Wl,-rpath,/opt/intel/oneapi/dal/2021.4.0/lib/intel64
        -L/opt/intel/oneapi/dal/2021.4.0/lib/intel64
-Wl,-rpath,/opt/intel/oneapi/compiler/2021.4.0/linux/compiler/lib/intel64_lin
-L/opt/intel/oneapi/compiler/2021.4.0/linux/compiler/lib/intel64_lin
        -Wl,-rpath,/opt/intel/oneapi/compiler/2021.4.0/linux/lib
        -L/opt/intel/oneapi/compiler/2021.4.0/linux/lib
        -Wl,-rpath,/opt/intel/oneapi/clck/2021.4.0/lib/intel64
        -L/opt/intel/oneapi/clck/2021.4.0/lib/intel64
        -Wl,-rpath,/opt/intel/oneapi/ccl/2021.4.0/lib/cpu_gpu_dpcpp
        -L/opt/intel/oneapi/ccl/2021.4.0/lib/cpu_gpu_dpcpp
        -Wl,-rpath,/usr/x86_64-suse-linux/lib
        -L/usr/x86_64-suse-linux/lib -larmadillo -lpetsc -lHYPRE
        -lcmumps -ldmumps -lsmumps -lzmumps -lmumps_common -lpord
        -lmkl_scalapack_lp64 -lmkl_blacs_intelmpi_lp64 -lspqr -lumfpack
        -lklu -lcholmod -lbtf -lccolamd -lcolamd -lcamd -lamd
        -lsuitesparseconfig -lsuperlu -lsuperlu_dist -lEl
        -lElSuiteSparse -lpmrrr -lfftw3_mpi -lfftw3 -lp4est -lsc
        -lmkl_intel_lp64 -lmkl_core -lmkl_intel_thread -liomp5 -ldl
        -lpthread -lptesmumps -lptscotchparmetis -lptscotch
        -lptscotcherr -lesmumps -lscotch -lscotcherr -lhdf5_hl -lhdf5
        -lparmetis -lmetis -lm -lz -lmuparser -lX11 -lstdc++ -ldl
        -lmpifort -lmpi -lrt -lpthread -lgfortran -lm -lgfortran -lm
        -lgcc_s -lquadmath -lrt -lquadmath -lstdc++ -ldl -o main_long</i></p>
    <p>Both executables have the same libraries linked to them, but in a
      different order when comparing them with ldd.<br>
    </p>
    <p>Does that explain the observed behavior?</p>
    <p>Thanks,</p>
    <p>regards,</p>
    <p>Roland<br>
    </p>
    <div>Am 11.10.21 um 15:13 schrieb Roland
      Richter:<br>
    </div>
    <blockquote type="cite">
      
      <p>Hei,</p>
      <p>the following code works fine:</p>
      <p>#include <iostream><br>
        #include <petsc.h><br>
        <br>
        static char help[] = "Solves 2D Poisson equation using
        multigrid.\n\n";<br>
        int main(int argc,char **argv) {<br>
            PetscInitialize(&argc,&argv,(char*)0,help);<br>
            std::cout << "Hello World\n";<br>
            PetscFinalize();<br>
            return 0;<br>
        }</p>
      <p>Regards,</p>
      <p>Roland<br>
      </p>
      <div>Am 11.10.21 um 14:34 schrieb Stefano
        Zampini:<br>
      </div>
      <blockquote type="cite"> Can
        you try with a simple call that only calls
        PetscInitialize/Finalize?
        <div><br>
          <div>
            <div><br>
              <blockquote type="cite">
                <div>On Oct 11, 2021, at 3:30 PM, Roland
                  Richter <<a href="mailto:roland.richter@ntnu.no" target="_blank">roland.richter@ntnu.no</a>>
                  wrote:</div>
                <br>
                <div>
                  <div>
                    <p>At least according to configure.log
                      mpiexec was defined as</p>
                    <p>Checking for program
                      /opt/intel/oneapi/mpi/2021.4.0//bin/mpiexec...found<br>
                                        Defined make macro
                      "MPIEXECEXECUTABLE" to
                      "/opt/intel/oneapi/mpi/2021.4.0/bin/mpiexec"</p>
                    <p>When running ex19 with this mpiexec it
                      fails with the usual error, even though all
                      configuration steps worked fine. I attached the
                      configuration log.<br>
                    </p>
                    <p>Regards,</p>
                    <p>Roland<br>
                    </p>
                    <div>Am 11.10.21 um 14:24
                      schrieb Stefano Zampini:<br>
                    </div>
                    <blockquote type="cite"> You are most probably using a different
                      mpiexec then the one used to compile petsc.
                      <div><br>
                      </div>
                      <div><br>
                        <div><br>
                          <blockquote type="cite">
                            <div>On Oct 11, 2021, at 3:23 PM,
                              Roland Richter <<a href="mailto:roland.richter@ntnu.no" target="_blank">roland.richter@ntnu.no</a>>
                              wrote:</div>
                            <br>
                            <div>
                              <div>
                                <p>I tried either <i>./ex19</i>
                                  (SNES-example), <i>mpirun
                                    ./ex19</i> or <i>mpirun -n
                                    1 ./ex19</i>, all with the same
                                  result.</p>
                                <p>Regards,</p>
                                <p>Roland<br>
                                </p>
                                <div>Am 11.10.21
                                  um 14:22 schrieb Matthew Knepley:<br>
                                </div>
                                <blockquote type="cite">
                                  <div dir="ltr">
                                    <div dir="ltr">On Mon, Oct
                                      11, 2021 at 8:07 AM Roland Richter
                                      <<a href="mailto:roland.richter@ntnu.no" target="_blank">roland.richter@ntnu.no</a>>
                                      wrote:<br>
                                    </div>
                                    <div class="gmail_quote">
                                      <blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
                                        <div>
                                          <p>Hei,</p>
                                          <p>at least in gdb it
                                            fails with</p>
                                          <p><span style="font-family:monospace"><span style="background-color:rgb(255,255,255)">Attempting to
                                                use an MPI routine
                                                before initializing
                                                MPICH </span><br>
                                              [Inferior 1 (process 7854)
                                              exited with code 01] <br>
                                              (gdb) backtrace <br>
                                              No stack.<br>
                                            </span></p>
                                        </div>
                                      </blockquote>
                                      <div><br>
                                      </div>
                                      <div>What were you
                                        running? If it never makes it
                                        into PETSc code, I am not sure
                                        what we are</div>
                                      <div>doing to cause this.</div>
                                      <div><br>
                                      </div>
                                      <div>  Thanks,</div>
                                      <div><br>
                                      </div>
                                      <div>     Matt</div>
                                      <div> </div>
                                      <blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
                                        <div>
                                          <p><span style="font-family:monospace"> </span>Regards,</p>
                                          <p>Roland<br>
                                            <span style="font-family:monospace"></span></p>
                                          <div>Am 11.10.21 um
                                            13:57 schrieb Matthew
                                            Knepley:<br>
                                          </div>
                                          <blockquote type="cite">
                                            <div dir="ltr">
                                              <div dir="ltr">On
                                                Mon, Oct 11, 2021 at
                                                5:24 AM Roland Richter
                                                <<a href="mailto:roland.richter@ntnu.no" target="_blank">roland.richter@ntnu.no</a>> wrote:<br>
                                              </div>
                                              <div class="gmail_quote">
                                                <blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Hei,<br>
                                                  <br>
                                                  I compiled PETSc with
                                                  Intel MPI (MPICH) and
                                                  GCC as compiler (i.e.
                                                  using<br>
                                                  Intel OneAPI together
                                                  with the supplied
                                                  mpicxx-compiler).
                                                  Compilation<br>
                                                  and installation
                                                  worked fine, but
                                                  running the tests
                                                  resulted in the<br>
                                                  error "Attempting to
                                                  use an MPI routine
                                                  before initializing
                                                  MPICH". A<br>
                                                  simple test program
                                                  (attached) worked fine
                                                  with the same
                                                  combination.<br>
                                                  <br>
                                                  What could be the
                                                  reason for that?<br>
                                                </blockquote>
                                                <div><br>
                                                </div>
                                                <div>Hi Roland,</div>
                                                <div><br>
                                                </div>
                                                <div>Can you
                                                  get a stack trace for
                                                  this error using the
                                                  debugger?</div>
                                                <div><br>
                                                </div>
                                                <div>  Thanks,</div>
                                                <div><br>
                                                </div>
                                                <div>     Matt</div>
                                                <div> </div>
                                                <blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
                                                  Thanks!<br>
                                                  <br>
                                                  Regards,<br>
                                                  <br>
                                                  Roland Richter<br>
                                                </blockquote>
                                              </div>
                                              <br clear="all">
                                              <div><br>
                                              </div>
                                              -- <br>
                                              <div dir="ltr">
                                                <div dir="ltr">
                                                  <div>
                                                    <div dir="ltr">
                                                      <div>
                                                        <div dir="ltr">
                                                          <div>What
                                                          most
                                                          experimenters
                                                          take for
                                                          granted before
                                                          they begin
                                                          their
                                                          experiments is
                                                          infinitely
                                                          more
                                                          interesting
                                                          than any
                                                          results to
                                                          which their
                                                          experiments
                                                          lead.<br>
                                                          -- Norbert
                                                          Wiener</div>
                                                          <div><br>
                                                          </div>
                                                          <div><a href="http://www.cse.buffalo.edu/~knepley/" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br>
                                                          </div>
                                                        </div>
                                                      </div>
                                                    </div>
                                                  </div>
                                                </div>
                                              </div>
                                            </div>
                                          </blockquote>
                                        </div>
                                      </blockquote>
                                    </div>
                                    <br clear="all">
                                    <div><br>
                                    </div>
                                    -- <br>
                                    <div dir="ltr">
                                      <div dir="ltr">
                                        <div>
                                          <div dir="ltr">
                                            <div>
                                              <div dir="ltr">
                                                <div>What most
                                                  experimenters take for
                                                  granted before they
                                                  begin their
                                                  experiments is
                                                  infinitely more
                                                  interesting than any
                                                  results to which their
                                                  experiments lead.<br>
                                                  -- Norbert Wiener</div>
                                                <div><br>
                                                </div>
                                                <div><a href="http://www.cse.buffalo.edu/~knepley/" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br>
                                                </div>
                                              </div>
                                            </div>
                                          </div>
                                        </div>
                                      </div>
                                    </div>
                                  </div>
                                </blockquote>
                              </div>
                            </div>
                          </blockquote>
                        </div>
                        <br>
                      </div>
                    </blockquote>
                  </div>
                  <span id="gmail-m_5718787638988412959cid:3B71A0CF-7121-40E3-AC68-DB609803623C"><configure.log></span></div>
              </blockquote>
            </div>
            <br>
          </div>
        </div>
      </blockquote>
    </blockquote>
  </div>
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr" class="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>-- Norbert Wiener</div><div><br></div><div><a href="http://www.cse.buffalo.edu/~knepley/" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br></div></div></div></div></div></div></div></div>