<html xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=Windows-1252">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Menlo;
panose-1:2 11 6 9 3 8 4 2 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
font-size:11.0pt;
font-family:"Calibri",sans-serif;
mso-ligatures:standardcontextual;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#0563C1;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal-compose;
font-family:"Calibri",sans-serif;
color:windowtext;}
p.p1, li.p1, div.p1
{mso-style-name:p1;
margin:0in;
font-size:8.5pt;
font-family:Menlo;
color:black;}
p.p2, li.p2, div.p2
{mso-style-name:p2;
margin:0in;
font-size:8.5pt;
font-family:Menlo;
color:black;}
span.s1
{mso-style-name:s1;}
span.apple-converted-space
{mso-style-name:apple-converted-space;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri",sans-serif;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style>
</head>
<body lang="EN-US" link="#0563C1" vlink="#954F72" style="word-wrap:break-word">
<div class="WordSection1">
<p class="MsoNormal">Hello PETSc developers,<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">I am having some issues with using MUMPS solver after the solution phase from PETSc. I am solving a matrix equation with about 2.17 million DOFs and for 1764 RHSs. MUMPS successfully solves the system for all RHS; however, after the solve
phase the distributed solution from MUMPS vector/matrix is scattered back to PETSc mpi vector (done within PETSc) and this is where I am getting the error.
<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">FYI, when I use 1323 RHSs, there is no issue. Moreover, I had to use v3.18.6 (or lower) because of some compiling issue with v3.19+ and SuperLU_DIST on our cluster with older intel mpi.
<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Any help is greatly appreciated. Thanks.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Piyoosh<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Here’s is the error message I am getting:<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="p1"><span class="s1">Entering ZMUMPS 5.5.1 from C interface with JOB, N =
</span><span class="apple-converted-space"> </span><span class="s1">3 </span><span class="apple-converted-space">
</span><span class="s1">2165687</span><o:p></o:p></p>
<p class="p1"><span class="apple-converted-space"> </span><span class="s1">executing #MPI =
</span><span class="apple-converted-space"> </span><span class="s1">63 and #OMP =</span><span class="apple-converted-space">
</span><span class="s1">1</span><o:p></o:p></p>
<p class="p2"><o:p> </o:p></p>
<p class="p2"><o:p> </o:p></p>
<p class="p2"><o:p> </o:p></p>
<p class="p1"><span class="apple-converted-space"> </span><span class="s1">****** SOLVE & CHECK STEP ********</span><o:p></o:p></p>
<p class="p2"><o:p> </o:p></p>
<p class="p1"><span class="apple-converted-space"> </span><span class="s1">GLOBAL STATISTICS PRIOR SOLVE PHASE ...........</span><o:p></o:p></p>
<p class="p1"><span class="apple-converted-space"> </span><span class="s1">Number of right-hand-sides</span><span class="apple-converted-space">
</span><span class="s1">=</span><span class="apple-converted-space"> </span>
<span class="s1">1764</span><o:p></o:p></p>
<p class="p1"><span class="apple-converted-space"> </span><span class="s1">Blocking factor for multiple rhs</span><span class="apple-converted-space">
</span><span class="s1">=</span><span class="apple-converted-space"> </span>
<span class="s1">32</span><o:p></o:p></p>
<p class="p1"><span class="apple-converted-space"> </span><span class="s1">ICNTL (9)
</span><span class="apple-converted-space"> </span>
<span class="s1">= </span><span class="apple-converted-space"> </span><span class="s1">1</span><o:p></o:p></p>
<p class="p1"><span class="apple-converted-space"> </span><span class="s1">--- (10)
</span><span class="apple-converted-space"> </span>
<span class="s1">= </span><span class="apple-converted-space"> </span><span class="s1">0</span><o:p></o:p></p>
<p class="p1"><span class="apple-converted-space"> </span><span class="s1">--- (11)
</span><span class="apple-converted-space"> </span>
<span class="s1">= </span><span class="apple-converted-space"> </span><span class="s1">0</span><o:p></o:p></p>
<p class="p1"><span class="apple-converted-space"> </span><span class="s1">--- (20)
</span><span class="apple-converted-space"> </span>
<span class="s1">= </span><span class="apple-converted-space"> </span><span class="s1">1</span><o:p></o:p></p>
<p class="p1"><span class="apple-converted-space"> </span><span class="s1">--- (21)
</span><span class="apple-converted-space"> </span>
<span class="s1">= </span><span class="apple-converted-space"> </span><span class="s1">1</span><o:p></o:p></p>
<p class="p1"><span class="apple-converted-space"> </span><span class="s1">--- (30)
</span><span class="apple-converted-space"> </span>
<span class="s1">= </span><span class="apple-converted-space"> </span><span class="s1">0</span><o:p></o:p></p>
<p class="p1"><span class="apple-converted-space"> </span><span class="s1">--- (35)
</span><span class="apple-converted-space"> </span>
<span class="s1">= </span><span class="apple-converted-space"> </span><span class="s1">0</span><o:p></o:p></p>
<p class="p1"><span class="apple-converted-space"> </span><span class="s1">** Rank of processor needing largest memory in solve
</span><span class="apple-converted-space"> </span><span class="s1">: </span><span class="apple-converted-space">
</span><span class="s1">5</span><o:p></o:p></p>
<p class="p1"><span class="apple-converted-space"> </span><span class="s1">** Space in MBYTES used by this processor for solve</span><span class="apple-converted-space">
</span><span class="s1">:</span><span class="apple-converted-space"> </span>
<span class="s1">1173</span><o:p></o:p></p>
<p class="p1"><span class="apple-converted-space"> </span><span class="s1">** Avg. Space in MBYTES per working proc during solve</span><span class="apple-converted-space">
</span><span class="s1">: </span><span class="apple-converted-space"> </span>
<span class="s1">820</span><o:p></o:p></p>
<p class="p2"><span class="apple-converted-space"> </span><o:p></o:p></p>
<p class="p1"><span class="apple-converted-space"> </span><span class="s1">Leaving solve with ...</span><o:p></o:p></p>
<p class="p1"><span class="apple-converted-space"> </span><span class="s1">Time to build/scatter RHS</span><span class="apple-converted-space">
</span><span class="s1">= </span><span class="apple-converted-space"> </span>
<span class="s1">0.130768</span><o:p></o:p></p>
<p class="p1"><span class="apple-converted-space"> </span><span class="s1">Time in solution step (fwd/bwd)</span><span class="apple-converted-space">
</span><span class="s1">=</span><span class="apple-converted-space"> </span>
<span class="s1">41.264175</span><o:p></o:p></p>
<p class="p1"><span class="apple-converted-space"> </span><span class="s1">.. Time in forward (fwd) step
</span><span class="apple-converted-space"> </span><span class="s1">= </span><span class="apple-converted-space">
</span><span class="s1">11.261921</span><o:p></o:p></p>
<p class="p1"><span class="apple-converted-space"> </span><span class="s1">.. Time in ScaLAPACK root
</span><span class="apple-converted-space"> </span><span class="s1">=</span><span class="apple-converted-space">
</span><span class="s1">3.584752</span><o:p></o:p></p>
<p class="p1"><span class="apple-converted-space"> </span><span class="s1">.. Time in backward (bwd) step</span><span class="apple-converted-space">
</span><span class="s1">= </span><span class="apple-converted-space"> </span>
<span class="s1">26.515026</span><o:p></o:p></p>
<p class="p1"><span class="apple-converted-space"> </span><span class="s1">Time to gather solution(cent.sol)=
</span><span class="apple-converted-space"> </span><span class="s1">0.000000</span><o:p></o:p></p>
<p class="p1"><span class="apple-converted-space"> </span><span class="s1">Time to copy/scale dist. solution=
</span><span class="apple-converted-space"> </span><span class="s1">0.724149</span><o:p></o:p></p>
<p class="p2"><o:p> </o:p></p>
<p class="p1"><span class="apple-converted-space"> </span><span class="s1">Elapsed time in solve driver=</span><span class="apple-converted-space">
</span><span class="s1">42.3461<o:p></o:p></span></p>
<p class="p1"><o:p> </o:p></p>
<p class="p1"><span class="s1">[23]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------</span><o:p></o:p></p>
<p class="p1"><span class="s1">[23]PETSC ERROR: Argument out of range</span><o:p></o:p></p>
<p class="p1"><span class="s1">[23]PETSC ERROR: Scatter indices in iy are out of range</span><o:p></o:p></p>
<p class="p1"><span class="s1">[23]PETSC ERROR: See <a href="https://petsc.org/release/faq/">
https://petsc.org/release/faq/</a> for trouble shooting.</span><o:p></o:p></p>
<p class="p1"><span class="s1">[23]PETSC ERROR: Petsc Release Version 3.18.6, unknown</span><span class="apple-converted-space"> </span><o:p></o:p></p>
<p class="p1"><span class="s1">[23]PETSC ERROR: /people/jays242/softwares/pgemini-em/pgemini/src/pgemini on a arch-linux-intel-opt-v3.18.6 named dc230.local by jays242 Thu Sep 28 10:44:51 2023</span><o:p></o:p></p>
<p class="p1"><span class="s1">[23]PETSC ERROR: Configure options PETSC_ARCH=arch-linux-intel-opt-v3.18.6 --with-debugging=0 COPTFLAGS=-O3 CXXOPTFLAGS=-O3 FOPTFLAGS=-O3 --with-cc=mpiicc --with-cxx=mpiicpc --with-fc=mpiifort --with-blaslapack-dir=/share/apps/intel/2020u4/compilers_and_libraries_2020.4.304/linux/mkl/lib/intel64
-with-scalar-type=complex --download-mumps --download-metis --with-openmp --download-parmetis --download-superlu_dist --with-scalapack-lib="-L/share/apps/intel/2020u4/compilers_and_libraries_2020.4.304/linux/mkl/lib/intel64 -lmkl_scalapack_lp64 -lmkl_blacs_intelmpi_lp64"</span><o:p></o:p></p>
<p class="p1"><span class="s1">[23]PETSC ERROR: #1 VecScatterCreate() at /qfs/people/jays242/softwares/pgemini-em/petsc-intel/src/vec/is/sf/interface/vscat.c:736</span><o:p></o:p></p>
<p class="p1"><span class="s1">[23]PETSC ERROR: #2 MatMatSolve_MUMPS() at /qfs/people/jays242/softwares/pgemini-em/petsc-intel/src/mat/impls/aij/mpi/mumps/mumps.c:1449</span><o:p></o:p></p>
<p class="p1"><span class="s1">[23]PETSC ERROR: #3 MatMatTransposeSolve_MUMPS() at /qfs/people/jays242/softwares/pgemini-em/petsc-intel/src/mat/impls/aij/mpi/mumps/mumps.c:1506</span><o:p></o:p></p>
<p class="p1"><span class="s1">[23]PETSC ERROR: #4 MatMatTransposeSolve() at /qfs/people/jays242/softwares/pgemini-em/petsc-intel/src/mat/interface/matrix.c:3789</span><o:p></o:p></p>
<p class="p1"><span class="s1">[24]PETSC ERROR: --------------------- Error Message ---------------</span><o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</body>
</html>