<div dir="ltr">To clarify, we have a mpi wrapper (so we can switch to different mpi at runtime). I compile petsc using our mpi wrapper. <div>If I just call PETSc initialize once without calling finallize, it is ok. My question to you is that: can I skip finalize? </div><div>Our program calls mpi_finalize at end anyway.</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, Jun 26, 2020 at 1:09 PM Sam Guo <<a href="mailto:sam.guo@cd-adapco.com">sam.guo@cd-adapco.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Hi Junchao,<div> Attached please find the configure.log.</div><div> I also attach the pinit.c which contains your patch (I am currently using 3.11.3. I've applied your patch to 3.11.3). Your patch fixes the serial version. The error now is about the parallel.</div><div> Here is the error log:</div><div> </div><div>[1]PETSC ERROR: #1 PetscInitialize() line 969 in ../../../petsc/src/sys/objects/pinit.c<br>[1]PETSC ERROR: #2 checkError() line 56 in ../../../physics/src/eigensolver/SLEPc.cpp<br>[1]PETSC ERROR: #3 PetscInitialize() line 966 in ../../../petsc/src/sys/objects/pinit.c<br>[1]PETSC ERROR: #4 SlepcInitialize() line 262 in ../../../slepc/src/sys/slepcinit.c<br>[0]PETSC ERROR: #1 PetscInitialize() line 969 in ../../../petsc/src/sys/objects/pinit.c<br>[0]PETSC ERROR: #2 checkError() line 56 in ../../../physics/src/eigensolver/SLEPc.cpp<br>[0]PETSC ERROR: #3 PetscInitialize() line 966 in ../../../petsc/src/sys/objects/pinit.c<br>[0]PETSC ERROR: #4 SlepcInitialize() line 262 in ../../../slepc/src/sys/slepcinit.c<br>PETSC ERROR: Logging has not been enabled.<br>You might have forgotten to call PetscInitialize().<br>PETSC ERROR: Logging has not been enabled.<br>You might have forgotten to call PetscInitialize().<br>--------------------------------------------------------------------------<br>MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_WORLD<br>with errorcode 56.<br><br>NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.<br>You may or may not see output from other processes, depending on<br>exactly when Open MPI kills them.<br></div><div><br></div><div>Thanks,</div><div>Sam</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Thu, Jun 25, 2020 at 7:37 PM Junchao Zhang <<a href="mailto:junchao.zhang@gmail.com" target="_blank">junchao.zhang@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Sam,<div> The MPI_Comm_create_keyval() error was fixed in maint/master. From the error message, it seems you need to configure --with-log=1</div><div> Otherwise, please send your full error stack trace and configure.log.</div><div> Thanks.<br clear="all"><div><div dir="ltr"><div dir="ltr">--Junchao Zhang</div></div></div><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Thu, Jun 25, 2020 at 2:18 PM Sam Guo <<a href="mailto:sam.guo@cd-adapco.com" target="_blank">sam.guo@cd-adapco.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Hi Junchao,<div> I now encountered the same error with parallel. I am wondering if there is a need for parallel fix as well.</div><div>[1]PETSC ERROR: #1 PetscInitialize() line 969 in ../../../petsc/src/sys/objects/pinit.c<br>PETSC ERROR: Logging has not been enabled.<br>You might have forgotten to call PetscInitialize().<br>PETSC ERROR: Logging has not been enabled.<br>You might have forgotten to call PetscInitialize().<br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Sat, Jun 20, 2020 at 7:35 PM Sam Guo <<a href="mailto:sam.guo@cd-adapco.com" target="_blank">sam.guo@cd-adapco.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Hi Junchao,<div> Your patch works.</div><div><br></div><div>Thanks,</div><div>Sam</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Sat, Jun 20, 2020 at 4:23 PM Junchao Zhang <<a href="mailto:junchao.zhang@gmail.com" target="_blank">junchao.zhang@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div dir="ltr"><br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Sat, Jun 20, 2020 at 12:24 PM Barry Smith <<a href="mailto:bsmith@petsc.dev" target="_blank">bsmith@petsc.dev</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div><div><br></div> Junchao,<div><br></div><div> This is a good bug fix. It solves the problem when PETSc initialize is called many times. </div><div><br></div><div> There is another fix you can do to limit PETSc mpiuni running out of attributes inside a single PETSc run:</div><div><br></div><div><br></div><div><div>int MPI_Comm_create_keyval(MPI_Copy_function *copy_fn,MPI_Delete_function *delete_fn,int *keyval,void *extra_state)</div><div>{</div><div> </div><div> if (num_attr >= MAX_ATTR){</div><div> for (i=0; i<num_attr; i++) {</div><div> if (!attr_keyval[i].extra_state) {</div></div></div></blockquote><div>attr_keyval[i].extra_state is provided by user (could be NULL). We can not rely on it. </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div><div><div> /* reuse this slot */</div><div> attr_keyval[i].extra_state = extra_state;</div><div> attr_keyval[i.]del = delete_fn;</div><div> *keyval = i;</div><div> return MPI_SUCCESS;</div><div> }</div><div> }</div><div> return MPIUni_Abort(MPI_COMM_WORLD,1);</div><div>}</div><div> return MPIUni_Abort(MPI_COMM_WORLD,1);</div><div> attr_keyval[num_attr].extra_state = extra_state;</div><div> attr_keyval[num_attr].del = delete_fn;</div><div> *keyval = num_attr++;</div><div> return MPI_SUCCESS;</div><div>}</div></div><div><br></div><div> This will work if the user creates tons of attributes but is constantly deleting some as they new ones. So long as the number outstanding at one time is < MAX_ATTR)</div><div><br></div><div>Barry</div><div><br></div><div><br></div><div><br></div><div><br><div><br><blockquote type="cite"><div>On Jun 20, 2020, at 10:54 AM, Junchao Zhang <<a href="mailto:junchao.zhang@gmail.com" target="_blank">junchao.zhang@gmail.com</a>> wrote:</div><br><div><div dir="ltr">I don't understand what your session means. Let's try this patch<div><br></div><div>diff --git a/src/sys/mpiuni/mpi.c b/src/sys/mpiuni/mpi.c<br>index d559a513..c058265d 100644<br>--- a/src/sys/mpiuni/mpi.c<br>+++ b/src/sys/mpiuni/mpi.c<br>@@ -283,6 +283,7 @@ int MPI_Finalize(void)<br> MPI_Comm_free(&comm);<br> comm = MPI_COMM_SELF;<br> MPI_Comm_free(&comm);<br>+ num_attr = 1; /* reset the counter */<br> MPI_was_finalized = 1;<br> return MPI_SUCCESS;<br> }<br><div><br></div><div><br clear="all"><div><div dir="ltr"><div dir="ltr">--Junchao Zhang</div></div></div><br></div></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Sat, Jun 20, 2020 at 10:48 AM Sam Guo <<a href="mailto:sam.guo@cd-adapco.com" target="_blank">sam.guo@cd-adapco.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Typo: I mean “Assuming initializer is only needed once for entire session”<br><br>On Saturday, June 20, 2020, Sam Guo <<a href="mailto:sam.guo@cd-adapco.com" target="_blank">sam.guo@cd-adapco.com</a>> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Assuming finalizer is only needed once for entire session(?), I can put initializer into the static block to call it once but where do I call finalizer?<div><br></div><div><br>On Saturday, June 20, 2020, Junchao Zhang <<a href="mailto:junchao.zhang@gmail.com" target="_blank">junchao.zhang@gmail.com</a>> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">The counter num_attr should be recycled. But first try to call PETSc initialize/Finalize only once to see it fixes the error.<br clear="all"><div><div dir="ltr"><div dir="ltr">--Junchao Zhang</div></div></div><br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Sat, Jun 20, 2020 at 12:48 AM Sam Guo <<a href="mailto:sam.guo@cd-adapco.com" target="_blank">sam.guo@cd-adapco.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">To clarify, I call PETSc initialize and PETSc finalize everytime I call SLEPc:<div><br><div><pre width="80" style="white-space:pre-wrap"> PetscInitializeNoPointers(argc,args,nullptr,nullptr);<br></pre><pre width="80" style="white-space:pre-wrap"> SlepcInitialize(&argc,&args,static_cast<char*>(nullptr),help);</pre><pre width="80" style="white-space:pre-wrap"> //calling slepc</pre><pre width="80" style="white-space:pre-wrap"> SlepcFinalize();</pre><pre width="80" style="white-space:pre-wrap"><span style="font-family:Arial,Helvetica,sans-serif;color:rgb(34,34,34)"> PetscFinalize();</span> </pre></div><div><br></div></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, Jun 19, 2020 at 10:32 PM Sam Guo <<a href="mailto:sam.guo@cd-adapco.com" target="_blank">sam.guo@cd-adapco.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Dear PETSc team,<div> When I called SLEPc multiple time, I eventually got following error: </div><div><br></div><div><span style="color:rgb(80,0,80)">MPI operation not supported by PETSc's sequential MPI wrappers<br></span>[0]PETSC ERROR: #1 PetscInitialize() line 967 in ../../../petsc/src/sys/objects/pinit.c<br>[0]PETSC ERROR: #2 SlepcInitialize() line 262 in ../../../slepc/src/sys/slepcinit.c<br>[0]PETSC ERROR: #3 SlepcInitializeNoPointers() line 359 in ../../../slepc/src/sys/slepcinit.c<span style="color:rgb(80,0,80)"><br>PETSC ERROR: Logging has not been enabled.<br>You might have forgotten to call PetscInitialize().</span> <br></div><div><br></div><div> I debugged: it is because of following in petsc/src/sys/mpiuni/mpi.c</div><div><br></div><div><div>if (num_attr >= MAX_ATTR)<br></div></div><div><br></div><div>in function int MPI_Comm_create_keyval(MPI_Copy_function *copy_fn,MPI_Delete_function *delete_fn,int *keyval,void *extra_state)</div><div><br></div><div>num_attr is declared static and keeps increasing every time MPI_Comm_create_keyval is called.</div><div><br></div><div>I am using petsc 3.11.3 but found 3.13.2 has the same logic.</div><div><br></div><div>Is this a bug or I didn't use it correctly?</div><div><br></div><div>Thanks,</div><div>Sam</div></div>
</blockquote></div>
</blockquote></div>
</blockquote></div>
</blockquote>
</blockquote></div>
</div></blockquote></div><br></div></div></blockquote></div></div>
</blockquote></div>
</blockquote></div>
</blockquote></div>
</blockquote></div>
</blockquote></div>