<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<!--[if !mso]><style>v\:* {behavior:url(#default#VML);}
o\:* {behavior:url(#default#VML);}
w\:* {behavior:url(#default#VML);}
.shape {behavior:url(#default#VML);}
</style><![endif]--><style><!--
/* Font Definitions */
@font-face
        {font-family:"Cambria Math";
        panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
        {font-family:Calibri;
        panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
        {margin:0in;
        font-size:10.0pt;
        font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
        {mso-style-priority:99;
        color:blue;
        text-decoration:underline;}
p.m3477368075900544461m-4118899637568871307p1, li.m3477368075900544461m-4118899637568871307p1, div.m3477368075900544461m-4118899637568871307p1
        {mso-style-name:m_3477368075900544461m-4118899637568871307p1;
        mso-margin-top-alt:auto;
        margin-right:0in;
        mso-margin-bottom-alt:auto;
        margin-left:0in;
        font-size:11.0pt;
        font-family:"Calibri",sans-serif;}
span.m3477368075900544461m-4118899637568871307s1
        {mso-style-name:m_3477368075900544461m-4118899637568871307s1;}
span.m3477368075900544461m-4118899637568871307apple-converted-space
        {mso-style-name:m_3477368075900544461m-4118899637568871307apple-converted-space;}
p.m3477368075900544461m-4118899637568871307p2, li.m3477368075900544461m-4118899637568871307p2, div.m3477368075900544461m-4118899637568871307p2
        {mso-style-name:m_3477368075900544461m-4118899637568871307p2;
        mso-margin-top-alt:auto;
        margin-right:0in;
        mso-margin-bottom-alt:auto;
        margin-left:0in;
        font-size:11.0pt;
        font-family:"Calibri",sans-serif;}
span.m3477368075900544461gmailsignatureprefix
        {mso-style-name:m_3477368075900544461gmailsignatureprefix;}
span.gmailsignatureprefix
        {mso-style-name:gmail_signature_prefix;}
span.EmailStyle24
        {mso-style-type:personal-reply;
        font-family:"Calibri",sans-serif;
        color:windowtext;}
.MsoChpDefault
        {mso-style-type:export-only;
        font-size:10.0pt;
        mso-ligatures:none;}
@page WordSection1
        {size:8.5in 11.0in;
        margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
        {page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="blue" vlink="purple" style="word-wrap:break-word">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:11.0pt">I think 64-bit indices is the solution. I mentioned earlier that I could solve 1323 RHSs but the DOFs for that problem is about 1.59 million so total unknowns are slightly below 2^31-1.
<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">I did compile PETSc with 64-bit indices but having some other issues with my assignment of variables in the code. I will come back again if the issue persists after fixing the assignments.   <o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Thanks for the help. <o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Piyoosh<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<div id="mail-editor-reference-message-container">
<div>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal" style="margin-bottom:12.0pt"><b><span style="font-size:12.0pt;color:black">From:
</span></b><span style="font-size:12.0pt;color:black">Matthew Knepley <knepley@gmail.com><br>
<b>Date: </b>Thursday, September 28, 2023 at 18:56<br>
<b>To: </b>Jaysaval, Piyoosh <Piyoosh.Jaysaval@pnnl.gov><br>
<b>Cc: </b>petsc-dev@mcs.anl.gov <petsc-dev@mcs.anl.gov><br>
<b>Subject: </b>Re: [petsc-dev] PETSc Error during VecScatterCreate after MUMPS solve<o:p></o:p></span></p>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt">On Thu, Sep 28, 2023 at 4:32 PM Jaysaval, Piyoosh <<a href="mailto:Piyoosh.Jaysaval@pnnl.gov">Piyoosh.Jaysaval@pnnl.gov</a>> wrote:<o:p></o:p></span></p>
</div>
<div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in">
<div>
<div>
<div>
<div>
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt">Thanks Matt. I don’t think so; I used the default. I will give it a try again with 64-bit indices.  <o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt"> <o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt">Do I just need to use --with-64-bit-indices during the configuration?<o:p></o:p></span></p>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt">Yes. I was worried about index overflow, so we will see. We try to protect for it, but its hard to catch every instance.<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt">  Thanks,<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt">     Matt<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt"> <o:p></o:p></span></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in">
<div>
<div>
<div>
<div>
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt"> <o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt">Piyoosh<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt"> <o:p></o:p></span></p>
</div>
<div id="m_3477368075900544461ms-outlook-mobile-signature">
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt">Get
<a href="https://aka.ms/o0ukef" target="_blank">Outlook for iOS</a><o:p></o:p></span></p>
</div>
<div id="m_3477368075900544461mail-editor-reference-message-container">
<div class="MsoNormal" align="center" style="text-align:center"><span style="font-size:11.0pt">
<hr size="0" width="100%" align="center">
</span></div>
<div id="m_3477368075900544461divRplyFwdMsg">
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><b><span style="font-size:11.0pt">From:</span></b><span style="font-size:11.0pt"> Matthew Knepley <<a href="mailto:knepley@gmail.com" target="_blank">knepley@gmail.com</a>><br>
<b>Sent:</b> Thursday, September 28, 2023 15:24<br>
<b>To:</b> Jaysaval, Piyoosh <<a href="mailto:Piyoosh.Jaysaval@pnnl.gov" target="_blank">Piyoosh.Jaysaval@pnnl.gov</a>><br>
<b>Cc:</b> <a href="mailto:petsc-dev@mcs.anl.gov" target="_blank">petsc-dev@mcs.anl.gov</a> <<a href="mailto:petsc-dev@mcs.anl.gov" target="_blank">petsc-dev@mcs.anl.gov</a>><br>
<b>Subject:</b> Re: [petsc-dev] PETSc Error during VecScatterCreate after MUMPS solve<o:p></o:p></span></p>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt"> <o:p></o:p></span></p>
</div>
</div>
<div style="border:none;border-left:solid #D77600 6.0pt;padding:0in 0in 0in 0in;font-size:1.15rem">
<p class="MsoNormal" align="center" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;text-align:center;background:#F7E3CC">
<span style="font-size:11.0pt;font-family:"Arial",sans-serif;color:black">Check twice before you click! This email originated from outside PNNL.</span><span style="font-size:11.0pt"><o:p></o:p></span></p>
</div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt"> <o:p></o:p></span></p>
<div>
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt">On Thu, Sep 28, 2023 at 4:09 PM Jaysaval, Piyoosh via petsc-dev <<a href="mailto:petsc-dev@mcs.anl.gov" target="_blank">petsc-dev@mcs.anl.gov</a>>
 wrote:<o:p></o:p></span></p>
</div>
<div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0in;margin-bottom:5.0pt">
<div>
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt">Hello PETSc developers,<o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt"> <o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt">I am having some issues with using MUMPS solver after the solution phase from PETSc. I am solving a matrix equation with about 2.17 million DOFs and
 for 1764 RHSs. MUMPS successfully solves the system for all RHS; however, after the solve phase the distributed solution from MUMPS vector/matrix is scattered back to PETSc mpi vector (done within PETSc) and this is where I am getting the error.
<o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt"> <o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt">FYI, when I use 1323 RHSs, there is no issue. Moreover, I had to use v3.18.6 (or lower) because of some compiling issue with v3.19+ and SuperLU_DIST
 on our cluster with older intel mpi. <o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt"> <o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt">Any help is greatly appreciated. Thanks.<o:p></o:p></span></p>
</div>
</div>
</div>
</blockquote>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt"> <o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt">Have you configured with 64-bit indices?<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt"> <o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt">  Thanks,<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt"> <o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt">     Matt<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt"> <o:p></o:p></span></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0in;margin-bottom:5.0pt">
<div>
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt">Piyoosh<o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt"> <o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt">Here’s is the error message I am getting:<o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt"> <o:p></o:p></span></p>
<p class="m3477368075900544461m-4118899637568871307p1"><span class="m3477368075900544461m-4118899637568871307s1">Entering ZMUMPS 5.5.1 from C interface with JOB, N =
</span><span class="m3477368075900544461m-4118899637568871307apple-converted-space"> 
</span><span class="m3477368075900544461m-4118899637568871307s1">3 </span><span class="m3477368075900544461m-4118899637568871307apple-converted-space">   
</span><span class="m3477368075900544461m-4118899637568871307s1">2165687</span><o:p></o:p></p>
<p class="m3477368075900544461m-4118899637568871307p1"><span class="m3477368075900544461m-4118899637568871307apple-converted-space">     
</span><span class="m3477368075900544461m-4118899637568871307s1">executing #MPI =
</span><span class="m3477368075900544461m-4118899637568871307apple-converted-space">   
</span><span class="m3477368075900544461m-4118899637568871307s1">63 and #OMP =</span><span class="m3477368075900544461m-4118899637568871307apple-converted-space">     
</span><span class="m3477368075900544461m-4118899637568871307s1">1</span><o:p></o:p></p>
<p class="m3477368075900544461m-4118899637568871307p2"> <o:p></o:p></p>
<p class="m3477368075900544461m-4118899637568871307p2"> <o:p></o:p></p>
<p class="m3477368075900544461m-4118899637568871307p2"> <o:p></o:p></p>
<p class="m3477368075900544461m-4118899637568871307p1"><span class="m3477368075900544461m-4118899637568871307apple-converted-space"> </span><span class="m3477368075900544461m-4118899637568871307s1">****** SOLVE & CHECK STEP ********</span><o:p></o:p></p>
<p class="m3477368075900544461m-4118899637568871307p2"> <o:p></o:p></p>
<p class="m3477368075900544461m-4118899637568871307p1"><span class="m3477368075900544461m-4118899637568871307apple-converted-space"> </span><span class="m3477368075900544461m-4118899637568871307s1">GLOBAL STATISTICS PRIOR SOLVE PHASE ...........</span><o:p></o:p></p>
<p class="m3477368075900544461m-4118899637568871307p1"><span class="m3477368075900544461m-4118899637568871307apple-converted-space"> </span><span class="m3477368075900544461m-4118899637568871307s1">Number of right-hand-sides</span><span class="m3477368075900544461m-4118899637568871307apple-converted-space"> 
                   </span><span class="m3477368075900544461m-4118899637568871307s1">=</span><span class="m3477368075900544461m-4118899637568871307apple-converted-space">       
</span><span class="m3477368075900544461m-4118899637568871307s1">1764</span><o:p></o:p></p>
<p class="m3477368075900544461m-4118899637568871307p1"><span class="m3477368075900544461m-4118899637568871307apple-converted-space"> </span><span class="m3477368075900544461m-4118899637568871307s1">Blocking factor for multiple rhs</span><span class="m3477368075900544461m-4118899637568871307apple-converted-space"> 
             </span><span class="m3477368075900544461m-4118899637568871307s1">=</span><span class="m3477368075900544461m-4118899637568871307apple-converted-space">         
</span><span class="m3477368075900544461m-4118899637568871307s1">32</span><o:p></o:p></p>
<p class="m3477368075900544461m-4118899637568871307p1"><span class="m3477368075900544461m-4118899637568871307apple-converted-space"> </span><span class="m3477368075900544461m-4118899637568871307s1">ICNTL (9)
</span><span class="m3477368075900544461m-4118899637568871307apple-converted-space">                                   
</span><span class="m3477368075900544461m-4118899637568871307s1">= </span><span class="m3477368075900544461m-4118899637568871307apple-converted-space">         
</span><span class="m3477368075900544461m-4118899637568871307s1">1</span><o:p></o:p></p>
<p class="m3477368075900544461m-4118899637568871307p1"><span class="m3477368075900544461m-4118899637568871307apple-converted-space"> 
</span><span class="m3477368075900544461m-4118899637568871307s1">--- (10) </span>
<span class="m3477368075900544461m-4118899637568871307apple-converted-space">                                   
</span><span class="m3477368075900544461m-4118899637568871307s1">= </span><span class="m3477368075900544461m-4118899637568871307apple-converted-space">         
</span><span class="m3477368075900544461m-4118899637568871307s1">0</span><o:p></o:p></p>
<p class="m3477368075900544461m-4118899637568871307p1"><span class="m3477368075900544461m-4118899637568871307apple-converted-space"> 
</span><span class="m3477368075900544461m-4118899637568871307s1">--- (11) </span>
<span class="m3477368075900544461m-4118899637568871307apple-converted-space">                                   
</span><span class="m3477368075900544461m-4118899637568871307s1">= </span><span class="m3477368075900544461m-4118899637568871307apple-converted-space">         
</span><span class="m3477368075900544461m-4118899637568871307s1">0</span><o:p></o:p></p>
<p class="m3477368075900544461m-4118899637568871307p1"><span class="m3477368075900544461m-4118899637568871307apple-converted-space"> 
</span><span class="m3477368075900544461m-4118899637568871307s1">--- (20) </span>
<span class="m3477368075900544461m-4118899637568871307apple-converted-space">                                   
</span><span class="m3477368075900544461m-4118899637568871307s1">= </span><span class="m3477368075900544461m-4118899637568871307apple-converted-space">         
</span><span class="m3477368075900544461m-4118899637568871307s1">1</span><o:p></o:p></p>
<p class="m3477368075900544461m-4118899637568871307p1"><span class="m3477368075900544461m-4118899637568871307apple-converted-space"> 
</span><span class="m3477368075900544461m-4118899637568871307s1">--- (21) </span>
<span class="m3477368075900544461m-4118899637568871307apple-converted-space">                                   
</span><span class="m3477368075900544461m-4118899637568871307s1">= </span><span class="m3477368075900544461m-4118899637568871307apple-converted-space">         
</span><span class="m3477368075900544461m-4118899637568871307s1">1</span><o:p></o:p></p>
<p class="m3477368075900544461m-4118899637568871307p1"><span class="m3477368075900544461m-4118899637568871307apple-converted-space"> 
</span><span class="m3477368075900544461m-4118899637568871307s1">--- (30) </span>
<span class="m3477368075900544461m-4118899637568871307apple-converted-space">                                   
</span><span class="m3477368075900544461m-4118899637568871307s1">= </span><span class="m3477368075900544461m-4118899637568871307apple-converted-space">         
</span><span class="m3477368075900544461m-4118899637568871307s1">0</span><o:p></o:p></p>
<p class="m3477368075900544461m-4118899637568871307p1"><span class="m3477368075900544461m-4118899637568871307apple-converted-space"> 
</span><span class="m3477368075900544461m-4118899637568871307s1">--- (35) </span>
<span class="m3477368075900544461m-4118899637568871307apple-converted-space">                                   
</span><span class="m3477368075900544461m-4118899637568871307s1">= </span><span class="m3477368075900544461m-4118899637568871307apple-converted-space">         
</span><span class="m3477368075900544461m-4118899637568871307s1">0</span><o:p></o:p></p>
<p class="m3477368075900544461m-4118899637568871307p1"><span class="m3477368075900544461m-4118899637568871307apple-converted-space"> </span><span class="m3477368075900544461m-4118899637568871307s1">** Rank of processor needing largest memory in solve
</span><span class="m3477368075900544461m-4118899637568871307apple-converted-space">   
</span><span class="m3477368075900544461m-4118899637568871307s1">: </span><span class="m3477368075900544461m-4118899637568871307apple-converted-space">       
</span><span class="m3477368075900544461m-4118899637568871307s1">5</span><o:p></o:p></p>
<p class="m3477368075900544461m-4118899637568871307p1"><span class="m3477368075900544461m-4118899637568871307apple-converted-space"> </span><span class="m3477368075900544461m-4118899637568871307s1">** Space in MBYTES used by this processor for solve</span><span class="m3477368075900544461m-4118899637568871307apple-converted-space"> 
     </span><span class="m3477368075900544461m-4118899637568871307s1">:</span><span class="m3477368075900544461m-4118899637568871307apple-converted-space">     
</span><span class="m3477368075900544461m-4118899637568871307s1">1173</span><o:p></o:p></p>
<p class="m3477368075900544461m-4118899637568871307p1"><span class="m3477368075900544461m-4118899637568871307apple-converted-space"> </span><span class="m3477368075900544461m-4118899637568871307s1">** Avg. Space in MBYTES per working proc during solve</span><span class="m3477368075900544461m-4118899637568871307apple-converted-space"> 
   </span><span class="m3477368075900544461m-4118899637568871307s1">: </span><span class="m3477368075900544461m-4118899637568871307apple-converted-space">     
</span><span class="m3477368075900544461m-4118899637568871307s1">820</span><o:p></o:p></p>
<p class="m3477368075900544461m-4118899637568871307p2"><span class="m3477368075900544461m-4118899637568871307apple-converted-space"> </span><o:p></o:p></p>
<p class="m3477368075900544461m-4118899637568871307p1"><span class="m3477368075900544461m-4118899637568871307apple-converted-space"> </span><span class="m3477368075900544461m-4118899637568871307s1">Leaving solve with ...</span><o:p></o:p></p>
<p class="m3477368075900544461m-4118899637568871307p1"><span class="m3477368075900544461m-4118899637568871307apple-converted-space"> </span><span class="m3477368075900544461m-4118899637568871307s1">Time to build/scatter RHS</span><span class="m3477368075900544461m-4118899637568871307apple-converted-space"> 
       </span><span class="m3477368075900544461m-4118899637568871307s1">= </span>
<span class="m3477368075900544461m-4118899637568871307apple-converted-space">     
</span><span class="m3477368075900544461m-4118899637568871307s1">0.130768</span><o:p></o:p></p>
<p class="m3477368075900544461m-4118899637568871307p1"><span class="m3477368075900544461m-4118899637568871307apple-converted-space"> </span><span class="m3477368075900544461m-4118899637568871307s1">Time in solution step (fwd/bwd)</span><span class="m3477368075900544461m-4118899637568871307apple-converted-space"> 
</span><span class="m3477368075900544461m-4118899637568871307s1">=</span><span class="m3477368075900544461m-4118899637568871307apple-converted-space">     
</span><span class="m3477368075900544461m-4118899637568871307s1">41.264175</span><o:p></o:p></p>
<p class="m3477368075900544461m-4118899637568871307p1"><span class="m3477368075900544461m-4118899637568871307apple-converted-space"> 
</span><span class="m3477368075900544461m-4118899637568871307s1">.. Time in forward (fwd) step
</span><span class="m3477368075900544461m-4118899637568871307apple-converted-space"> 
</span><span class="m3477368075900544461m-4118899637568871307s1">= </span><span class="m3477368075900544461m-4118899637568871307apple-converted-space">       
</span><span class="m3477368075900544461m-4118899637568871307s1">11.261921</span><o:p></o:p></p>
<p class="m3477368075900544461m-4118899637568871307p1"><span class="m3477368075900544461m-4118899637568871307apple-converted-space"> 
</span><span class="m3477368075900544461m-4118899637568871307s1">.. Time in ScaLAPACK root
</span><span class="m3477368075900544461m-4118899637568871307apple-converted-space">     
</span><span class="m3477368075900544461m-4118899637568871307s1">=</span><span class="m3477368075900544461m-4118899637568871307apple-converted-space">         
</span><span class="m3477368075900544461m-4118899637568871307s1">3.584752</span><o:p></o:p></p>
<p class="m3477368075900544461m-4118899637568871307p1"><span class="m3477368075900544461m-4118899637568871307apple-converted-space"> 
</span><span class="m3477368075900544461m-4118899637568871307s1">.. Time in backward (bwd) step</span><span class="m3477368075900544461m-4118899637568871307apple-converted-space"> 
</span><span class="m3477368075900544461m-4118899637568871307s1">= </span><span class="m3477368075900544461m-4118899637568871307apple-converted-space">       
</span><span class="m3477368075900544461m-4118899637568871307s1">26.515026</span><o:p></o:p></p>
<p class="m3477368075900544461m-4118899637568871307p1"><span class="m3477368075900544461m-4118899637568871307apple-converted-space"> </span><span class="m3477368075900544461m-4118899637568871307s1">Time to gather solution(cent.sol)=
</span><span class="m3477368075900544461m-4118899637568871307apple-converted-space">     
</span><span class="m3477368075900544461m-4118899637568871307s1">0.000000</span><o:p></o:p></p>
<p class="m3477368075900544461m-4118899637568871307p1"><span class="m3477368075900544461m-4118899637568871307apple-converted-space"> </span><span class="m3477368075900544461m-4118899637568871307s1">Time to copy/scale dist. solution=
</span><span class="m3477368075900544461m-4118899637568871307apple-converted-space">     
</span><span class="m3477368075900544461m-4118899637568871307s1">0.724149</span><o:p></o:p></p>
<p class="m3477368075900544461m-4118899637568871307p2"> <o:p></o:p></p>
<p class="m3477368075900544461m-4118899637568871307p1"><span class="m3477368075900544461m-4118899637568871307apple-converted-space"> </span><span class="m3477368075900544461m-4118899637568871307s1">Elapsed time in solve driver=</span><span class="m3477368075900544461m-4118899637568871307apple-converted-space"> 
     </span><span class="m3477368075900544461m-4118899637568871307s1">42.3461</span><o:p></o:p></p>
<p class="m3477368075900544461m-4118899637568871307p1"> <o:p></o:p></p>
<p class="m3477368075900544461m-4118899637568871307p1"><span class="m3477368075900544461m-4118899637568871307s1">[23]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------</span><o:p></o:p></p>
<p class="m3477368075900544461m-4118899637568871307p1"><span class="m3477368075900544461m-4118899637568871307s1">[23]PETSC ERROR: Argument out of range</span><o:p></o:p></p>
<p class="m3477368075900544461m-4118899637568871307p1"><span class="m3477368075900544461m-4118899637568871307s1">[23]PETSC ERROR: Scatter indices in iy are out of range</span><o:p></o:p></p>
<p class="m3477368075900544461m-4118899637568871307p1"><span class="m3477368075900544461m-4118899637568871307s1">[23]PETSC ERROR: See
<a href="https://petsc.org/release/faq/" target="_blank">https://petsc.org/release/faq/</a> for trouble shooting.</span><o:p></o:p></p>
<p class="m3477368075900544461m-4118899637568871307p1"><span class="m3477368075900544461m-4118899637568871307s1">[23]PETSC ERROR: Petsc Release Version 3.18.6, unknown</span><span class="m3477368075900544461m-4118899637568871307apple-converted-space"> </span><o:p></o:p></p>
<p class="m3477368075900544461m-4118899637568871307p1"><span class="m3477368075900544461m-4118899637568871307s1">[23]PETSC ERROR: /people/jays242/softwares/pgemini-em/pgemini/src/pgemini on a arch-linux-intel-opt-v3.18.6 named dc230.local by jays242 Thu Sep
 28 10:44:51 2023</span><o:p></o:p></p>
<p class="m3477368075900544461m-4118899637568871307p1"><span class="m3477368075900544461m-4118899637568871307s1">[23]PETSC ERROR: Configure options PETSC_ARCH=arch-linux-intel-opt-v3.18.6 --with-debugging=0 COPTFLAGS=-O3 CXXOPTFLAGS=-O3 FOPTFLAGS=-O3 --with-cc=mpiicc
 --with-cxx=mpiicpc --with-fc=mpiifort --with-blaslapack-dir=/share/apps/intel/2020u4/compilers_and_libraries_2020.4.304/linux/mkl/lib/intel64 -with-scalar-type=complex --download-mumps --download-metis --with-openmp --download-parmetis --download-superlu_dist
 --with-scalapack-lib="-L/share/apps/intel/2020u4/compilers_and_libraries_2020.4.304/linux/mkl/lib/intel64 -lmkl_scalapack_lp64 -lmkl_blacs_intelmpi_lp64"</span><o:p></o:p></p>
<p class="m3477368075900544461m-4118899637568871307p1"><span class="m3477368075900544461m-4118899637568871307s1">[23]PETSC ERROR: #1 VecScatterCreate() at /qfs/people/jays242/softwares/pgemini-em/petsc-intel/src/vec/is/sf/interface/vscat.c:736</span><o:p></o:p></p>
<p class="m3477368075900544461m-4118899637568871307p1"><span class="m3477368075900544461m-4118899637568871307s1">[23]PETSC ERROR: #2 MatMatSolve_MUMPS() at /qfs/people/jays242/softwares/pgemini-em/petsc-intel/src/mat/impls/aij/mpi/mumps/mumps.c:1449</span><o:p></o:p></p>
<p class="m3477368075900544461m-4118899637568871307p1"><span class="m3477368075900544461m-4118899637568871307s1">[23]PETSC ERROR: #3 MatMatTransposeSolve_MUMPS() at /qfs/people/jays242/softwares/pgemini-em/petsc-intel/src/mat/impls/aij/mpi/mumps/mumps.c:1506</span><o:p></o:p></p>
<p class="m3477368075900544461m-4118899637568871307p1"><span class="m3477368075900544461m-4118899637568871307s1">[23]PETSC ERROR: #4 MatMatTransposeSolve() at /qfs/people/jays242/softwares/pgemini-em/petsc-intel/src/mat/interface/matrix.c:3789</span><o:p></o:p></p>
<p class="m3477368075900544461m-4118899637568871307p1"><span class="m3477368075900544461m-4118899637568871307s1">[24]PETSC ERROR: --------------------- Error Message ---------------</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt"> <o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt"> <o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt"> <o:p></o:p></span></p>
</div>
</div>
</div>
</blockquote>
</div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt"><br clear="all">
<o:p></o:p></span></p>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt"> <o:p></o:p></span></p>
</div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span class="m3477368075900544461gmailsignatureprefix"><span style="font-size:11.0pt">--
</span></span><span style="font-size:11.0pt"><o:p></o:p></span></p>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt">What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments
 lead.<br>
-- Norbert Wiener<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt"> <o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt"><a href="http://www.cse.buffalo.edu/~knepley/" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><o:p></o:p></span></p>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
<p class="MsoNormal"><span style="font-size:11.0pt"><br clear="all">
<o:p></o:p></span></p>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
</div>
<p class="MsoNormal"><span class="gmailsignatureprefix"><span style="font-size:11.0pt">--
</span></span><span style="font-size:11.0pt"><o:p></o:p></span></p>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt">What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>
-- Norbert Wiener<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt"><a href="http://www.cse.buffalo.edu/~knepley/" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><o:p></o:p></span></p>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</body>
</html>