<div dir="ltr"><div dir="ltr">On Thu, Sep 28, 2023 at 4:32 PM Jaysaval, Piyoosh <<a href="mailto:Piyoosh.Jaysaval@pnnl.gov">Piyoosh.Jaysaval@pnnl.gov</a>> wrote:<br></div><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div class="msg3477368075900544461">
<div lang="EN-US" style="overflow-wrap: break-word;">
<div class="m_3477368075900544461WordSection1">
<div>
<div>
<div>
<p class="MsoNormal">Thanks Matt. I don’t think so; I used the default. I will give it a try again with 64-bit indices. </p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">Do I just need to use --with-64-bit-indices during the configuration?</p></div></div></div></div></div></div></blockquote><div><br></div><div>Yes. I was worried about index overflow, so we will see. We try to protect for it, but its hard to catch every instance.</div><div><br></div><div> Thanks,</div><div><br></div><div> Matt</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div class="msg3477368075900544461"><div lang="EN-US" style="overflow-wrap: break-word;"><div class="m_3477368075900544461WordSection1"><div><div><div><p class="MsoNormal"> </p>
</div>
<div>
<p class="MsoNormal">Piyoosh</p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div id="m_3477368075900544461ms-outlook-mobile-signature">
<p class="MsoNormal">Get <a href="https://aka.ms/o0ukef" target="_blank">Outlook for iOS</a></p>
</div>
<div id="m_3477368075900544461mail-editor-reference-message-container">
<div class="MsoNormal" align="center" style="text-align:center">
<hr size="0" width="100%" align="center">
</div>
<div id="m_3477368075900544461divRplyFwdMsg">
<p class="MsoNormal"><b>From:</b> Matthew Knepley <<a href="mailto:knepley@gmail.com" target="_blank">knepley@gmail.com</a>><br>
<b>Sent:</b> Thursday, September 28, 2023 15:24<br>
<b>To:</b> Jaysaval, Piyoosh <<a href="mailto:Piyoosh.Jaysaval@pnnl.gov" target="_blank">Piyoosh.Jaysaval@pnnl.gov</a>><br>
<b>Cc:</b> <a href="mailto:petsc-dev@mcs.anl.gov" target="_blank">petsc-dev@mcs.anl.gov</a> <<a href="mailto:petsc-dev@mcs.anl.gov" target="_blank">petsc-dev@mcs.anl.gov</a>><br>
<b>Subject:</b> Re: [petsc-dev] PETSc Error during VecScatterCreate after MUMPS solve</p>
<div>
<p class="MsoNormal"> </p>
</div>
</div>
<div style="border-top:none;border-right:none;border-bottom:none;border-left:6pt solid rgb(215,118,0);padding:0in;font-size:1.15rem">
<p class="MsoNormal" align="center" style="text-align:center;background:rgb(247,227,204)">
<span style="font-family:Arial,sans-serif;color:black">Check twice before you click! This email originated from outside PNNL.</span><span style="font-family:Arial,sans-serif"><u></u><u></u></span></p>
</div>
<p class="MsoNormal"><u></u> <u></u></p>
<div>
<div>
<div>
<p class="MsoNormal">On Thu, Sep 28, 2023 at 4:09 PM Jaysaval, Piyoosh via petsc-dev <<a href="mailto:petsc-dev@mcs.anl.gov" target="_blank">petsc-dev@mcs.anl.gov</a>> wrote:</p>
</div>
<div>
<blockquote style="border-top:none;border-right:none;border-bottom:none;border-left:1pt solid rgb(204,204,204);padding:0in 0in 0in 6pt;margin:5pt 0in 5pt 4.8pt">
<div>
<div>
<div>
<p class="MsoNormal">Hello PETSc developers,</p>
<p class="MsoNormal"> </p>
<p class="MsoNormal">I am having some issues with using MUMPS solver after the solution phase from PETSc. I am solving a matrix equation with about 2.17 million DOFs and for 1764 RHSs. MUMPS successfully
solves the system for all RHS; however, after the solve phase the distributed solution from MUMPS vector/matrix is scattered back to PETSc mpi vector (done within PETSc) and this is where I am getting the error.
</p>
<p class="MsoNormal"> </p>
<p class="MsoNormal">FYI, when I use 1323 RHSs, there is no issue. Moreover, I had to use v3.18.6 (or lower) because of some compiling issue with v3.19+ and SuperLU_DIST on our cluster with older intel
mpi. </p>
<p class="MsoNormal"> </p>
<p class="MsoNormal">Any help is greatly appreciated. Thanks.</p>
</div>
</div>
</div>
</blockquote>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">Have you configured with 64-bit indices?</p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal"> Thanks,</p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal"> Matt</p>
</div>
<div>
<p class="MsoNormal"> </p>
</div>
<blockquote style="border-top:none;border-right:none;border-bottom:none;border-left:1pt solid rgb(204,204,204);padding:0in 0in 0in 6pt;margin:5pt 0in 5pt 4.8pt">
<div>
<div>
<div>
<p class="MsoNormal">Piyoosh</p>
<p class="MsoNormal"> </p>
<p class="MsoNormal">Here’s is the error message I am getting:</p>
<p class="MsoNormal"> </p>
<p class="m_3477368075900544461m-4118899637568871307p1"><span class="m_3477368075900544461m-4118899637568871307s1">Entering ZMUMPS 5.5.1 from C interface with JOB, N =
</span><span class="m_3477368075900544461m-4118899637568871307apple-converted-space"> </span><span class="m_3477368075900544461m-4118899637568871307s1">3
</span><span class="m_3477368075900544461m-4118899637568871307apple-converted-space"> </span><span class="m_3477368075900544461m-4118899637568871307s1">2165687</span></p>
<p class="m_3477368075900544461m-4118899637568871307p1"><span class="m_3477368075900544461m-4118899637568871307apple-converted-space">
</span><span class="m_3477368075900544461m-4118899637568871307s1">executing #MPI = </span><span class="m_3477368075900544461m-4118899637568871307apple-converted-space">
</span><span class="m_3477368075900544461m-4118899637568871307s1">63 and #OMP =</span><span class="m_3477368075900544461m-4118899637568871307apple-converted-space">
</span><span class="m_3477368075900544461m-4118899637568871307s1">1</span></p>
<p class="m_3477368075900544461m-4118899637568871307p2"> </p>
<p class="m_3477368075900544461m-4118899637568871307p2"> </p>
<p class="m_3477368075900544461m-4118899637568871307p2"> </p>
<p class="m_3477368075900544461m-4118899637568871307p1"><span class="m_3477368075900544461m-4118899637568871307apple-converted-space"> </span><span class="m_3477368075900544461m-4118899637568871307s1">****** SOLVE & CHECK STEP ********</span></p>
<p class="m_3477368075900544461m-4118899637568871307p2"> </p>
<p class="m_3477368075900544461m-4118899637568871307p1"><span class="m_3477368075900544461m-4118899637568871307apple-converted-space"> </span><span class="m_3477368075900544461m-4118899637568871307s1">GLOBAL STATISTICS PRIOR SOLVE PHASE ...........</span></p>
<p class="m_3477368075900544461m-4118899637568871307p1"><span class="m_3477368075900544461m-4118899637568871307apple-converted-space"> </span><span class="m_3477368075900544461m-4118899637568871307s1">Number of right-hand-sides</span><span class="m_3477368075900544461m-4118899637568871307apple-converted-space">
</span><span class="m_3477368075900544461m-4118899637568871307s1">=</span><span class="m_3477368075900544461m-4118899637568871307apple-converted-space">
</span><span class="m_3477368075900544461m-4118899637568871307s1">1764</span></p>
<p class="m_3477368075900544461m-4118899637568871307p1"><span class="m_3477368075900544461m-4118899637568871307apple-converted-space"> </span><span class="m_3477368075900544461m-4118899637568871307s1">Blocking factor for multiple rhs</span><span class="m_3477368075900544461m-4118899637568871307apple-converted-space">
</span><span class="m_3477368075900544461m-4118899637568871307s1">=</span><span class="m_3477368075900544461m-4118899637568871307apple-converted-space">
</span><span class="m_3477368075900544461m-4118899637568871307s1">32</span></p>
<p class="m_3477368075900544461m-4118899637568871307p1"><span class="m_3477368075900544461m-4118899637568871307apple-converted-space"> </span><span class="m_3477368075900544461m-4118899637568871307s1">ICNTL (9)
</span><span class="m_3477368075900544461m-4118899637568871307apple-converted-space">
</span><span class="m_3477368075900544461m-4118899637568871307s1">= </span><span class="m_3477368075900544461m-4118899637568871307apple-converted-space">
</span><span class="m_3477368075900544461m-4118899637568871307s1">1</span></p>
<p class="m_3477368075900544461m-4118899637568871307p1"><span class="m_3477368075900544461m-4118899637568871307apple-converted-space">
</span><span class="m_3477368075900544461m-4118899637568871307s1">--- (10) </span><span class="m_3477368075900544461m-4118899637568871307apple-converted-space">
</span><span class="m_3477368075900544461m-4118899637568871307s1">= </span><span class="m_3477368075900544461m-4118899637568871307apple-converted-space">
</span><span class="m_3477368075900544461m-4118899637568871307s1">0</span></p>
<p class="m_3477368075900544461m-4118899637568871307p1"><span class="m_3477368075900544461m-4118899637568871307apple-converted-space">
</span><span class="m_3477368075900544461m-4118899637568871307s1">--- (11) </span><span class="m_3477368075900544461m-4118899637568871307apple-converted-space">
</span><span class="m_3477368075900544461m-4118899637568871307s1">= </span><span class="m_3477368075900544461m-4118899637568871307apple-converted-space">
</span><span class="m_3477368075900544461m-4118899637568871307s1">0</span></p>
<p class="m_3477368075900544461m-4118899637568871307p1"><span class="m_3477368075900544461m-4118899637568871307apple-converted-space">
</span><span class="m_3477368075900544461m-4118899637568871307s1">--- (20) </span><span class="m_3477368075900544461m-4118899637568871307apple-converted-space">
</span><span class="m_3477368075900544461m-4118899637568871307s1">= </span><span class="m_3477368075900544461m-4118899637568871307apple-converted-space">
</span><span class="m_3477368075900544461m-4118899637568871307s1">1</span></p>
<p class="m_3477368075900544461m-4118899637568871307p1"><span class="m_3477368075900544461m-4118899637568871307apple-converted-space">
</span><span class="m_3477368075900544461m-4118899637568871307s1">--- (21) </span><span class="m_3477368075900544461m-4118899637568871307apple-converted-space">
</span><span class="m_3477368075900544461m-4118899637568871307s1">= </span><span class="m_3477368075900544461m-4118899637568871307apple-converted-space">
</span><span class="m_3477368075900544461m-4118899637568871307s1">1</span></p>
<p class="m_3477368075900544461m-4118899637568871307p1"><span class="m_3477368075900544461m-4118899637568871307apple-converted-space">
</span><span class="m_3477368075900544461m-4118899637568871307s1">--- (30) </span><span class="m_3477368075900544461m-4118899637568871307apple-converted-space">
</span><span class="m_3477368075900544461m-4118899637568871307s1">= </span><span class="m_3477368075900544461m-4118899637568871307apple-converted-space">
</span><span class="m_3477368075900544461m-4118899637568871307s1">0</span></p>
<p class="m_3477368075900544461m-4118899637568871307p1"><span class="m_3477368075900544461m-4118899637568871307apple-converted-space">
</span><span class="m_3477368075900544461m-4118899637568871307s1">--- (35) </span><span class="m_3477368075900544461m-4118899637568871307apple-converted-space">
</span><span class="m_3477368075900544461m-4118899637568871307s1">= </span><span class="m_3477368075900544461m-4118899637568871307apple-converted-space">
</span><span class="m_3477368075900544461m-4118899637568871307s1">0</span></p>
<p class="m_3477368075900544461m-4118899637568871307p1"><span class="m_3477368075900544461m-4118899637568871307apple-converted-space"> </span><span class="m_3477368075900544461m-4118899637568871307s1">** Rank of processor needing largest memory in solve
</span><span class="m_3477368075900544461m-4118899637568871307apple-converted-space"> </span><span class="m_3477368075900544461m-4118899637568871307s1">:
</span><span class="m_3477368075900544461m-4118899637568871307apple-converted-space"> </span><span class="m_3477368075900544461m-4118899637568871307s1">5</span></p>
<p class="m_3477368075900544461m-4118899637568871307p1"><span class="m_3477368075900544461m-4118899637568871307apple-converted-space"> </span><span class="m_3477368075900544461m-4118899637568871307s1">** Space in MBYTES used by this processor for solve</span><span class="m_3477368075900544461m-4118899637568871307apple-converted-space">
</span><span class="m_3477368075900544461m-4118899637568871307s1">:</span><span class="m_3477368075900544461m-4118899637568871307apple-converted-space">
</span><span class="m_3477368075900544461m-4118899637568871307s1">1173</span></p>
<p class="m_3477368075900544461m-4118899637568871307p1"><span class="m_3477368075900544461m-4118899637568871307apple-converted-space"> </span><span class="m_3477368075900544461m-4118899637568871307s1">** Avg. Space in MBYTES per working proc during solve</span><span class="m_3477368075900544461m-4118899637568871307apple-converted-space">
</span><span class="m_3477368075900544461m-4118899637568871307s1">: </span><span class="m_3477368075900544461m-4118899637568871307apple-converted-space">
</span><span class="m_3477368075900544461m-4118899637568871307s1">820</span></p>
<p class="m_3477368075900544461m-4118899637568871307p2"><span class="m_3477368075900544461m-4118899637568871307apple-converted-space"> </span></p>
<p class="m_3477368075900544461m-4118899637568871307p1"><span class="m_3477368075900544461m-4118899637568871307apple-converted-space"> </span><span class="m_3477368075900544461m-4118899637568871307s1">Leaving solve with ...</span></p>
<p class="m_3477368075900544461m-4118899637568871307p1"><span class="m_3477368075900544461m-4118899637568871307apple-converted-space"> </span><span class="m_3477368075900544461m-4118899637568871307s1">Time to build/scatter RHS</span><span class="m_3477368075900544461m-4118899637568871307apple-converted-space">
</span><span class="m_3477368075900544461m-4118899637568871307s1">= </span><span class="m_3477368075900544461m-4118899637568871307apple-converted-space">
</span><span class="m_3477368075900544461m-4118899637568871307s1">0.130768</span></p>
<p class="m_3477368075900544461m-4118899637568871307p1"><span class="m_3477368075900544461m-4118899637568871307apple-converted-space"> </span><span class="m_3477368075900544461m-4118899637568871307s1">Time in solution step (fwd/bwd)</span><span class="m_3477368075900544461m-4118899637568871307apple-converted-space">
</span><span class="m_3477368075900544461m-4118899637568871307s1">=</span><span class="m_3477368075900544461m-4118899637568871307apple-converted-space">
</span><span class="m_3477368075900544461m-4118899637568871307s1">41.264175</span></p>
<p class="m_3477368075900544461m-4118899637568871307p1"><span class="m_3477368075900544461m-4118899637568871307apple-converted-space">
</span><span class="m_3477368075900544461m-4118899637568871307s1">.. Time in forward (fwd) step </span>
<span class="m_3477368075900544461m-4118899637568871307apple-converted-space"> </span><span class="m_3477368075900544461m-4118899637568871307s1">=
</span><span class="m_3477368075900544461m-4118899637568871307apple-converted-space"> </span><span class="m_3477368075900544461m-4118899637568871307s1">11.261921</span></p>
<p class="m_3477368075900544461m-4118899637568871307p1"><span class="m_3477368075900544461m-4118899637568871307apple-converted-space">
</span><span class="m_3477368075900544461m-4118899637568871307s1">.. Time in ScaLAPACK root </span><span class="m_3477368075900544461m-4118899637568871307apple-converted-space">
</span><span class="m_3477368075900544461m-4118899637568871307s1">=</span><span class="m_3477368075900544461m-4118899637568871307apple-converted-space">
</span><span class="m_3477368075900544461m-4118899637568871307s1">3.584752</span></p>
<p class="m_3477368075900544461m-4118899637568871307p1"><span class="m_3477368075900544461m-4118899637568871307apple-converted-space">
</span><span class="m_3477368075900544461m-4118899637568871307s1">.. Time in backward (bwd) step</span><span class="m_3477368075900544461m-4118899637568871307apple-converted-space">
</span><span class="m_3477368075900544461m-4118899637568871307s1">= </span><span class="m_3477368075900544461m-4118899637568871307apple-converted-space">
</span><span class="m_3477368075900544461m-4118899637568871307s1">26.515026</span></p>
<p class="m_3477368075900544461m-4118899637568871307p1"><span class="m_3477368075900544461m-4118899637568871307apple-converted-space"> </span><span class="m_3477368075900544461m-4118899637568871307s1">Time to gather solution(cent.sol)=
</span><span class="m_3477368075900544461m-4118899637568871307apple-converted-space"> </span><span class="m_3477368075900544461m-4118899637568871307s1">0.000000</span></p>
<p class="m_3477368075900544461m-4118899637568871307p1"><span class="m_3477368075900544461m-4118899637568871307apple-converted-space"> </span><span class="m_3477368075900544461m-4118899637568871307s1">Time to copy/scale dist. solution=
</span><span class="m_3477368075900544461m-4118899637568871307apple-converted-space"> </span><span class="m_3477368075900544461m-4118899637568871307s1">0.724149</span></p>
<p class="m_3477368075900544461m-4118899637568871307p2"> </p>
<p class="m_3477368075900544461m-4118899637568871307p1"><span class="m_3477368075900544461m-4118899637568871307apple-converted-space"> </span><span class="m_3477368075900544461m-4118899637568871307s1">Elapsed time in solve driver=</span><span class="m_3477368075900544461m-4118899637568871307apple-converted-space">
</span><span class="m_3477368075900544461m-4118899637568871307s1">42.3461</span></p>
<p class="m_3477368075900544461m-4118899637568871307p1"> </p>
<p class="m_3477368075900544461m-4118899637568871307p1"><span class="m_3477368075900544461m-4118899637568871307s1">[23]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------</span></p>
<p class="m_3477368075900544461m-4118899637568871307p1"><span class="m_3477368075900544461m-4118899637568871307s1">[23]PETSC ERROR: Argument out of range</span></p>
<p class="m_3477368075900544461m-4118899637568871307p1"><span class="m_3477368075900544461m-4118899637568871307s1">[23]PETSC ERROR: Scatter indices in iy are out of range</span></p>
<p class="m_3477368075900544461m-4118899637568871307p1"><span class="m_3477368075900544461m-4118899637568871307s1">[23]PETSC ERROR: See
<a href="https://petsc.org/release/faq/" target="_blank">https://petsc.org/release/faq/</a> for trouble shooting.</span></p>
<p class="m_3477368075900544461m-4118899637568871307p1"><span class="m_3477368075900544461m-4118899637568871307s1">[23]PETSC ERROR: Petsc Release Version 3.18.6, unknown</span><span class="m_3477368075900544461m-4118899637568871307apple-converted-space"> </span></p>
<p class="m_3477368075900544461m-4118899637568871307p1"><span class="m_3477368075900544461m-4118899637568871307s1">[23]PETSC ERROR: /people/jays242/softwares/pgemini-em/pgemini/src/pgemini on a arch-linux-intel-opt-v3.18.6 named dc230.local by jays242 Thu Sep 28 10:44:51 2023</span></p>
<p class="m_3477368075900544461m-4118899637568871307p1"><span class="m_3477368075900544461m-4118899637568871307s1">[23]PETSC ERROR: Configure options PETSC_ARCH=arch-linux-intel-opt-v3.18.6 --with-debugging=0 COPTFLAGS=-O3 CXXOPTFLAGS=-O3 FOPTFLAGS=-O3 --with-cc=mpiicc --with-cxx=mpiicpc --with-fc=mpiifort
--with-blaslapack-dir=/share/apps/intel/2020u4/compilers_and_libraries_2020.4.304/linux/mkl/lib/intel64 -with-scalar-type=complex --download-mumps --download-metis --with-openmp --download-parmetis --download-superlu_dist --with-scalapack-lib="-L/share/apps/intel/2020u4/compilers_and_libraries_2020.4.304/linux/mkl/lib/intel64
-lmkl_scalapack_lp64 -lmkl_blacs_intelmpi_lp64"</span></p>
<p class="m_3477368075900544461m-4118899637568871307p1"><span class="m_3477368075900544461m-4118899637568871307s1">[23]PETSC ERROR: #1 VecScatterCreate() at /qfs/people/jays242/softwares/pgemini-em/petsc-intel/src/vec/is/sf/interface/vscat.c:736</span></p>
<p class="m_3477368075900544461m-4118899637568871307p1"><span class="m_3477368075900544461m-4118899637568871307s1">[23]PETSC ERROR: #2 MatMatSolve_MUMPS() at /qfs/people/jays242/softwares/pgemini-em/petsc-intel/src/mat/impls/aij/mpi/mumps/mumps.c:1449</span></p>
<p class="m_3477368075900544461m-4118899637568871307p1"><span class="m_3477368075900544461m-4118899637568871307s1">[23]PETSC ERROR: #3 MatMatTransposeSolve_MUMPS() at /qfs/people/jays242/softwares/pgemini-em/petsc-intel/src/mat/impls/aij/mpi/mumps/mumps.c:1506</span></p>
<p class="m_3477368075900544461m-4118899637568871307p1"><span class="m_3477368075900544461m-4118899637568871307s1">[23]PETSC ERROR: #4 MatMatTransposeSolve() at /qfs/people/jays242/softwares/pgemini-em/petsc-intel/src/mat/interface/matrix.c:3789</span></p>
<p class="m_3477368075900544461m-4118899637568871307p1"><span class="m_3477368075900544461m-4118899637568871307s1">[24]PETSC ERROR: --------------------- Error Message ---------------</span></p>
<p class="MsoNormal"> </p>
<p class="MsoNormal"> </p>
<p class="MsoNormal"> </p>
</div>
</div>
</div>
</blockquote>
</div>
<p class="MsoNormal"><br clear="all">
</p>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<p class="MsoNormal"><span class="m_3477368075900544461gmailsignatureprefix">-- </span></p>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<p class="MsoNormal">What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>
-- Norbert Wiener</p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal"><a href="http://www.cse.buffalo.edu/~knepley/" target="_blank">https://www.cse.buffalo.edu/~knepley/</a></p>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div></blockquote></div><br clear="all"><div><br></div><span class="gmail_signature_prefix">-- </span><br><div dir="ltr" class="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>-- Norbert Wiener</div><div><br></div><div><a href="http://www.cse.buffalo.edu/~knepley/" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br></div></div></div></div></div></div></div></div>