<html><head><meta http-equiv="Content-Type" content="text/html; charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class=""><div class=""><br class=""></div> Sam,<div class=""><br class=""><div> You can skip PetscFinalize() so long as you only call PetscInitialize() once. It is not desirable in general to skip the finalize because PETSc can't free all its data structures and you cannot see the PETSc logging information with -log_view but in terms of the code running correctly you do not need to call PetscFinalize.</div><div><br class=""></div><div> If your code crashes in PetscFinalize() please send the full error output and we can try to help you debug it.</div><div><br class=""></div><div><br class=""></div><div> Barry</div><div><br class=""><blockquote type="cite" class=""><div class="">On Jun 26, 2020, at 3:14 PM, Sam Guo <<a href="mailto:sam.guo@cd-adapco.com" class="">sam.guo@cd-adapco.com</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><div dir="ltr" class="">To clarify, we have a mpi wrapper (so we can switch to different mpi at runtime). I compile petsc using our mpi wrapper. <div class="">If I just call PETSc initialize once without calling finallize, it is ok. My question to you is that: can I skip finalize? </div><div class="">Our program calls mpi_finalize at end anyway.</div></div><br class=""><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, Jun 26, 2020 at 1:09 PM Sam Guo <<a href="mailto:sam.guo@cd-adapco.com" class="">sam.guo@cd-adapco.com</a>> wrote:<br class=""></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr" class="">Hi Junchao,<div class=""> Attached please find the configure.log.</div><div class=""> I also attach the pinit.c which contains your patch (I am currently using 3.11.3. I've applied your patch to 3.11.3). Your patch fixes the serial version. The error now is about the parallel.</div><div class=""> Here is the error log:</div><div class=""> </div><div class="">[1]PETSC ERROR: #1 PetscInitialize() line 969 in ../../../petsc/src/sys/objects/pinit.c<br class="">[1]PETSC ERROR: #2 checkError() line 56 in ../../../physics/src/eigensolver/SLEPc.cpp<br class="">[1]PETSC ERROR: #3 PetscInitialize() line 966 in ../../../petsc/src/sys/objects/pinit.c<br class="">[1]PETSC ERROR: #4 SlepcInitialize() line 262 in ../../../slepc/src/sys/slepcinit.c<br class="">[0]PETSC ERROR: #1 PetscInitialize() line 969 in ../../../petsc/src/sys/objects/pinit.c<br class="">[0]PETSC ERROR: #2 checkError() line 56 in ../../../physics/src/eigensolver/SLEPc.cpp<br class="">[0]PETSC ERROR: #3 PetscInitialize() line 966 in ../../../petsc/src/sys/objects/pinit.c<br class="">[0]PETSC ERROR: #4 SlepcInitialize() line 262 in ../../../slepc/src/sys/slepcinit.c<br class="">PETSC ERROR: Logging has not been enabled.<br class="">You might have forgotten to call PetscInitialize().<br class="">PETSC ERROR: Logging has not been enabled.<br class="">You might have forgotten to call PetscInitialize().<br class="">--------------------------------------------------------------------------<br class="">MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_WORLD<br class="">with errorcode 56.<br class=""><br class="">NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.<br class="">You may or may not see output from other processes, depending on<br class="">exactly when Open MPI kills them.<br class=""></div><div class=""><br class=""></div><div class="">Thanks,</div><div class="">Sam</div></div><br class=""><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Thu, Jun 25, 2020 at 7:37 PM Junchao Zhang <<a href="mailto:junchao.zhang@gmail.com" target="_blank" class="">junchao.zhang@gmail.com</a>> wrote:<br class=""></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr" class="">Sam,<div class=""> The MPI_Comm_create_keyval() error was fixed in maint/master. From the error message, it seems you need to configure --with-log=1</div><div class=""> Otherwise, please send your full error stack trace and configure.log.</div><div class=""> Thanks.<br clear="all" class=""><div class=""><div dir="ltr" class=""><div dir="ltr" class="">--Junchao Zhang</div></div></div><br class=""></div></div><br class=""><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Thu, Jun 25, 2020 at 2:18 PM Sam Guo <<a href="mailto:sam.guo@cd-adapco.com" target="_blank" class="">sam.guo@cd-adapco.com</a>> wrote:<br class=""></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr" class="">Hi Junchao,<div class=""> I now encountered the same error with parallel. I am wondering if there is a need for parallel fix as well.</div><div class="">[1]PETSC ERROR: #1 PetscInitialize() line 969 in ../../../petsc/src/sys/objects/pinit.c<br class="">PETSC ERROR: Logging has not been enabled.<br class="">You might have forgotten to call PetscInitialize().<br class="">PETSC ERROR: Logging has not been enabled.<br class="">You might have forgotten to call PetscInitialize().<br class=""></div></div><br class=""><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Sat, Jun 20, 2020 at 7:35 PM Sam Guo <<a href="mailto:sam.guo@cd-adapco.com" target="_blank" class="">sam.guo@cd-adapco.com</a>> wrote:<br class=""></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr" class="">Hi Junchao,<div class=""> Your patch works.</div><div class=""><br class=""></div><div class="">Thanks,</div><div class="">Sam</div></div><br class=""><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Sat, Jun 20, 2020 at 4:23 PM Junchao Zhang <<a href="mailto:junchao.zhang@gmail.com" target="_blank" class="">junchao.zhang@gmail.com</a>> wrote:<br class=""></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr" class=""><div dir="ltr" class=""><br class=""></div><br class=""><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Sat, Jun 20, 2020 at 12:24 PM Barry Smith <<a href="mailto:bsmith@petsc.dev" target="_blank" class="">bsmith@petsc.dev</a>> wrote:<br class=""></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div class=""><div class=""><br class=""></div> Junchao,<div class=""><br class=""></div><div class=""> This is a good bug fix. It solves the problem when PETSc initialize is called many times. </div><div class=""><br class=""></div><div class=""> There is another fix you can do to limit PETSc mpiuni running out of attributes inside a single PETSc run:</div><div class=""><br class=""></div><div class=""><br class=""></div><div class=""><div class="">int MPI_Comm_create_keyval(MPI_Copy_function *copy_fn,MPI_Delete_function *delete_fn,int *keyval,void *extra_state)</div><div class="">{</div><div class=""> </div><div class=""> if (num_attr >= MAX_ATTR){</div><div class=""> for (i=0; i<num_attr; i++) {</div><div class=""> if (!attr_keyval[i].extra_state) {</div></div></div></blockquote><div class="">attr_keyval[i].extra_state is provided by user (could be NULL). We can not rely on it. </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div class=""><div class=""><div class=""> /* reuse this slot */</div><div class=""> attr_keyval[i].extra_state = extra_state;</div><div class=""> attr_keyval[i.]del = delete_fn;</div><div class=""> *keyval = i;</div><div class=""> return MPI_SUCCESS;</div><div class=""> }</div><div class=""> }</div><div class=""> return MPIUni_Abort(MPI_COMM_WORLD,1);</div><div class="">}</div><div class=""> return MPIUni_Abort(MPI_COMM_WORLD,1);</div><div class=""> attr_keyval[num_attr].extra_state = extra_state;</div><div class=""> attr_keyval[num_attr].del = delete_fn;</div><div class=""> *keyval = num_attr++;</div><div class=""> return MPI_SUCCESS;</div><div class="">}</div></div><div class=""><br class=""></div><div class=""> This will work if the user creates tons of attributes but is constantly deleting some as they new ones. So long as the number outstanding at one time is < MAX_ATTR)</div><div class=""><br class=""></div><div class="">Barry</div><div class=""><br class=""></div><div class=""><br class=""></div><div class=""><br class=""></div><div class=""><br class=""><div class=""><br class=""><blockquote type="cite" class=""><div class="">On Jun 20, 2020, at 10:54 AM, Junchao Zhang <<a href="mailto:junchao.zhang@gmail.com" target="_blank" class="">junchao.zhang@gmail.com</a>> wrote:</div><br class=""><div class=""><div dir="ltr" class="">I don't understand what your session means. Let's try this patch<div class=""><br class=""></div><div class="">diff --git a/src/sys/mpiuni/mpi.c b/src/sys/mpiuni/mpi.c<br class="">index d559a513..c058265d 100644<br class="">--- a/src/sys/mpiuni/mpi.c<br class="">+++ b/src/sys/mpiuni/mpi.c<br class="">@@ -283,6 +283,7 @@ int MPI_Finalize(void)<br class=""> MPI_Comm_free(&comm);<br class=""> comm = MPI_COMM_SELF;<br class=""> MPI_Comm_free(&comm);<br class="">+ num_attr = 1; /* reset the counter */<br class=""> MPI_was_finalized = 1;<br class=""> return MPI_SUCCESS;<br class=""> }<br class=""><div class=""><br class=""></div><div class=""><br clear="all" class=""><div class=""><div dir="ltr" class=""><div dir="ltr" class="">--Junchao Zhang</div></div></div><br class=""></div></div></div><br class=""><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Sat, Jun 20, 2020 at 10:48 AM Sam Guo <<a href="mailto:sam.guo@cd-adapco.com" target="_blank" class="">sam.guo@cd-adapco.com</a>> wrote:<br class=""></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Typo: I mean “Assuming initializer is only needed once for entire session”<br class=""><br class="">On Saturday, June 20, 2020, Sam Guo <<a href="mailto:sam.guo@cd-adapco.com" target="_blank" class="">sam.guo@cd-adapco.com</a>> wrote:<br class=""><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Assuming finalizer is only needed once for entire session(?), I can put initializer into the static block to call it once but where do I call finalizer?<div class=""><br class=""></div><div class=""><br class="">On Saturday, June 20, 2020, Junchao Zhang <<a href="mailto:junchao.zhang@gmail.com" target="_blank" class="">junchao.zhang@gmail.com</a>> wrote:<br class=""><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr" class="">The counter num_attr should be recycled. But first try to call PETSc initialize/Finalize only once to see it fixes the error.<br clear="all" class=""><div class=""><div dir="ltr" class=""><div dir="ltr" class="">--Junchao Zhang</div></div></div><br class=""></div><br class=""><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Sat, Jun 20, 2020 at 12:48 AM Sam Guo <<a href="mailto:sam.guo@cd-adapco.com" target="_blank" class="">sam.guo@cd-adapco.com</a>> wrote:<br class=""></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr" class="">To clarify, I call PETSc initialize and PETSc finalize everytime I call SLEPc:<div class=""><br class=""><div class=""><pre width="80" style="white-space:pre-wrap" class=""> PetscInitializeNoPointers(argc,args,nullptr,nullptr);<br class=""></pre><pre width="80" style="white-space:pre-wrap" class=""> SlepcInitialize(&argc,&args,static_cast<char*>(nullptr),help);</pre><pre width="80" style="white-space:pre-wrap" class=""> //calling slepc</pre><pre width="80" style="white-space:pre-wrap" class=""> SlepcFinalize();</pre><pre width="80" style="white-space:pre-wrap" class=""><span style="font-family:Arial,Helvetica,sans-serif;color:rgb(34,34,34)" class=""> PetscFinalize();</span> </pre></div><div class=""><br class=""></div></div></div><br class=""><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, Jun 19, 2020 at 10:32 PM Sam Guo <<a href="mailto:sam.guo@cd-adapco.com" target="_blank" class="">sam.guo@cd-adapco.com</a>> wrote:<br class=""></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr" class="">Dear PETSc team,<div class=""> When I called SLEPc multiple time, I eventually got following error: </div><div class=""><br class=""></div><div class=""><span style="color:rgb(80,0,80)" class="">MPI operation not supported by PETSc's sequential MPI wrappers<br class=""></span>[0]PETSC ERROR: #1 PetscInitialize() line 967 in ../../../petsc/src/sys/objects/pinit.c<br class="">[0]PETSC ERROR: #2 SlepcInitialize() line 262 in ../../../slepc/src/sys/slepcinit.c<br class="">[0]PETSC ERROR: #3 SlepcInitializeNoPointers() line 359 in ../../../slepc/src/sys/slepcinit.c<span style="color:rgb(80,0,80)" class=""><br class="">PETSC ERROR: Logging has not been enabled.<br class="">You might have forgotten to call PetscInitialize().</span> <br class=""></div><div class=""><br class=""></div><div class=""> I debugged: it is because of following in petsc/src/sys/mpiuni/mpi.c</div><div class=""><br class=""></div><div class=""><div class="">if (num_attr >= MAX_ATTR)<br class=""></div></div><div class=""><br class=""></div><div class="">in function int MPI_Comm_create_keyval(MPI_Copy_function *copy_fn,MPI_Delete_function *delete_fn,int *keyval,void *extra_state)</div><div class=""><br class=""></div><div class="">num_attr is declared static and keeps increasing every time MPI_Comm_create_keyval is called.</div><div class=""><br class=""></div><div class="">I am using petsc 3.11.3 but found 3.13.2 has the same logic.</div><div class=""><br class=""></div><div class="">Is this a bug or I didn't use it correctly?</div><div class=""><br class=""></div><div class="">Thanks,</div><div class="">Sam</div></div>
</blockquote></div>
</blockquote></div>
</blockquote></div>
</blockquote>
</blockquote></div>
</div></blockquote></div><br class=""></div></div></blockquote></div></div>
</blockquote></div>
</blockquote></div>
</blockquote></div>
</blockquote></div>
</blockquote></div>
</div></blockquote></div><br class=""></div></body></html>