<div dir="ltr"><div dir="ltr">On Tue, Jul 6, 2021 at 9:31 AM Vijay S Kumar <<a href="mailto:vijayskumar@gmail.com">vijayskumar@gmail.com</a>> wrote:<br></div><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Hello all,<div><br></div><div>By way of background, we have a PetSc-based solver that we run on our in-house Cray system. We are carrying out performance analysis using profilers in the CrayPat suite that provide more fine-grained performance-related information than the PetSc log_view summary.</div><div><br></div><div>When instrumented using CrayPat perftools, it turns out that the MPI initialization (MPI_Init) internally invoked by PetscInitialize is not picked up by the profiler. That is, simply specifying the following:</div><div> ierr = PetscInitialize(&argc,&argv,(char*)0,NULL);if (ierr) return ierr;<br></div><div>results in the following runtime error: <br></div><div><p class="MsoNormal" style="margin:0in;font-size:11pt;font-family:Calibri,sans-serif"> CrayPat/X: Version 7.1.1 Revision 7c0ddd79b 08/19/19 16:58:46</p>
<p class="MsoNormal" style="text-indent:0.5in;margin:0in;font-size:11pt;font-family:Calibri,sans-serif"><span style="background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial;background-color:rgb(255,255,255)">Attempting to use an MPI routine before initializing
MPICH</span></p></div><div><div><br></div><div>To circumvent this, we had to explicitly call MPI_Init prior to PetscInitialize:</div><div> MPI_Init(&argc,&argv);<br> ierr = PetscInitialize(&argc,&argv,(char*)0,NULL);if (ierr) return ierr;<br></div><div><br></div><div>However, the side-effect of this above workaround seems to be several downstream runtime (assertion) errors with VecAssemblyBegin/End and MatAssemblyBeing/End statements:</div><div><br></div></div><blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px"><div><div>CrayPat/X: Version 7.1.1 Revision 7c0ddd79b 08/19/19 16:58:46</div></div><div>main.x: ../rtsum.c:5662: __pat_trsup_trace_waitsome_rtsum: Assertion `recv_count != MPI_UNDEFINED' failed.<br></div><div><div><br></div><div> main@main.c:769<br></div></div><div><div> <span style="background-color:rgb(255,255,0)">VecAssemblyEnd@0x2aaab951b3ba</span></div></div><div><div><span style="background-color:rgb(255,255,0)"> VecAssemblyEnd_MPI_BTS@0x2aaab950b179</span></div></div><div><div><span style="background-color:rgb(255,255,0)"> MPI_Waitsome@0x43a238</span></div></div><div><div><span style="background-color:rgb(255,255,0)"> __pat_trsup_trace_waitsome_rtsum@0x5f1a17</span></div></div><div><div><span style="background-color:rgb(255,255,0)"> __GI___assert_fail@0x2aaabc61e7d1</span></div></div><div><div><span style="background-color:rgb(255,255,0)"> __assert_fail_base@0x2aaabc61e759</span></div></div><div><div> __GI_abort@0x2aaabc627740</div></div><div><div> __GI_raise@0x2aaabc626160</div></div></blockquote><div><div><br></div></div><div>Interestingly, we do not see such errors when there is no explicit MPI_Init, and no instrumentation for performance.</div><div>Looking for someone to help throw more light on why PetSc Mat/Vec AssemblyEnd statements lead to such MPI-level assertion errors in cases where MPI_Init is explicitly called. <br></div><div>(Or alternatively, is there a way to call PetscInitialize in a manner that ensures that the MPI initialization is picked up by the profilers in question?)</div><div><br></div><div>We would highly appreciate any help/pointers,</div></div></blockquote><div><br></div><div>There is no problem calling MPI_Init() before PetscInitialize(), although then you also have to call MPI_Finalize() explicitly at the end.</div><div>Both errors appears to arise from the Cray instrumentation, which is evidently buggy. Did you try calling MPI_Init() yourself without</div><div>instrumentation? Also, what info are you getting from CrayPat that we do not log?</div><div><br></div><div> Thanks,</div><div><br></div><div> Matt</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>Thanks!</div><div> Vijay</div></div>
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr" class="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>-- Norbert Wiener</div><div><br></div><div><a href="http://www.cse.buffalo.edu/~knepley/" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br></div></div></div></div></div></div></div></div>