<div dir="ltr">Hi Junchao,<div> If you are talking about this commit of yours <a href="https://gitlab.com/petsc/petsc/-/commit/f0463fa09df52ce43e7c5bf47a1c87df0c9e5cbb">https://gitlab.com/petsc/petsc/-/commit/f0463fa09df52ce43e7c5bf47a1c87df0c9e5cbb</a> </div><h3 class="gmail-commit-title" style="box-sizing:border-box;margin:0px;line-height:1.2;color:rgb(46,46,46);font-size:1.53125rem;font-family:-apple-system,BlinkMacSystemFont,"Segoe UI",Roboto,"Noto Sans",Ubuntu,Cantarell,"Helvetica Neue",sans-serif,"Apple Color Emoji","Segoe UI Emoji","Segoe UI Symbol","Noto Color Emoji"">Recycle keyvals and fix bugs in MPI_Comm creation</h3><div> I think I got it. It fixes the serial one but parallel one is still crashing.</div><div><br></div><div>Thanks,</div><div>Sam</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, Jun 26, 2020 at 3:43 PM Sam Guo <<a href="mailto:sam.guo@cd-adapco.com">sam.guo@cd-adapco.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Hi Junchao,<div> I am not ready to upgrade petsc yet(due to the lengthy technical and legal approval process of our internal policy). Can you send me the diff file so I can apply it to petsc 3.11.3)?</div><div><br></div><div>Thanks,</div><div>Sam</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, Jun 26, 2020 at 3:33 PM Junchao Zhang <<a href="mailto:junchao.zhang@gmail.com" target="_blank">junchao.zhang@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Sam,<div> Please discard the origin patch I sent you. A better fix is already in maint/master. An test is at src/sys/tests/ex53.c</div><div> I modified that test at the end with</div><blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px"><div> for (i=0; i<500; i++) {</div><div> ierr = PetscInitializeNoPointers(argc,argv,NULL,help);if (ierr) return ierr;</div><div> ierr = SlepcInitialize(&argc,&argv,NULL,help);if (ierr) return ierr;</div><div> ierr = SlepcFinalize();if (ierr) return ierr;</div><div> ierr = PetscFinalize();if (ierr) return ierr;</div><div> }</div></blockquote><div><br></div><div> then I ran it with multiple mpi ranks and it ran correctly. So try your program with petsc master first. If not work, see if you can come up with a test example for us.</div><div><br></div><div> Thanks.</div><div><div><div><div dir="ltr"><div dir="ltr">--Junchao Zhang</div></div></div><br></div></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, Jun 26, 2020 at 3:37 PM Sam Guo <<a href="mailto:sam.guo@cd-adapco.com" target="_blank">sam.guo@cd-adapco.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">One work around for me is to call PetscInitialize once for my entire program and skip PetscFinalize (since I don't have a good place to call
PetscFinalize before ending the program).</div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, Jun 26, 2020 at 1:33 PM Sam Guo <<a href="mailto:sam.guo@cd-adapco.com" target="_blank">sam.guo@cd-adapco.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">I get the crash after calling Initialize/Finalize multiple times. Junchao fixed the bug for serial but parallel still crashes.</div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, Jun 26, 2020 at 1:28 PM Barry Smith <<a href="mailto:bsmith@petsc.dev" target="_blank">bsmith@petsc.dev</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div><div><br></div> Ah, so you get the crash the second time you call PetscInitialize()? That is a problem because we do intend to support that capability (but you much call PetscFinalize() each time also).<div><br></div><div> Barry</div><div><br><div><br><blockquote type="cite"><div>On Jun 26, 2020, at 3:25 PM, Sam Guo <<a href="mailto:sam.guo@cd-adapco.com" target="_blank">sam.guo@cd-adapco.com</a>> wrote:</div><br><div><div dir="ltr">Hi Barry,<div> Thanks for the quick response.</div><div> I will call PetscInitialize once and skip the PetscFinalize for now to avoid the crash. The crash is actually in PetscInitialize, not PetscFinalize.</div><div><br></div><div>Thanks,</div><div>Sam</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, Jun 26, 2020 at 1:21 PM Barry Smith <<a href="mailto:bsmith@petsc.dev" target="_blank">bsmith@petsc.dev</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div><div><br></div> Sam,<div><br><div> You can skip PetscFinalize() so long as you only call PetscInitialize() once. It is not desirable in general to skip the finalize because PETSc can't free all its data structures and you cannot see the PETSc logging information with -log_view but in terms of the code running correctly you do not need to call PetscFinalize.</div><div><br></div><div> If your code crashes in PetscFinalize() please send the full error output and we can try to help you debug it.</div><div><br></div><div><br></div><div> Barry</div><div><br><blockquote type="cite"><div>On Jun 26, 2020, at 3:14 PM, Sam Guo <<a href="mailto:sam.guo@cd-adapco.com" target="_blank">sam.guo@cd-adapco.com</a>> wrote:</div><br><div><div dir="ltr">To clarify, we have a mpi wrapper (so we can switch to different mpi at runtime). I compile petsc using our mpi wrapper. <div>If I just call PETSc initialize once without calling finallize, it is ok. My question to you is that: can I skip finalize? </div><div>Our program calls mpi_finalize at end anyway.</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, Jun 26, 2020 at 1:09 PM Sam Guo <<a href="mailto:sam.guo@cd-adapco.com" target="_blank">sam.guo@cd-adapco.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Hi Junchao,<div> Attached please find the configure.log.</div><div> I also attach the pinit.c which contains your patch (I am currently using 3.11.3. I've applied your patch to 3.11.3). Your patch fixes the serial version. The error now is about the parallel.</div><div> Here is the error log:</div><div> </div><div>[1]PETSC ERROR: #1 PetscInitialize() line 969 in ../../../petsc/src/sys/objects/pinit.c<br>[1]PETSC ERROR: #2 checkError() line 56 in ../../../physics/src/eigensolver/SLEPc.cpp<br>[1]PETSC ERROR: #3 PetscInitialize() line 966 in ../../../petsc/src/sys/objects/pinit.c<br>[1]PETSC ERROR: #4 SlepcInitialize() line 262 in ../../../slepc/src/sys/slepcinit.c<br>[0]PETSC ERROR: #1 PetscInitialize() line 969 in ../../../petsc/src/sys/objects/pinit.c<br>[0]PETSC ERROR: #2 checkError() line 56 in ../../../physics/src/eigensolver/SLEPc.cpp<br>[0]PETSC ERROR: #3 PetscInitialize() line 966 in ../../../petsc/src/sys/objects/pinit.c<br>[0]PETSC ERROR: #4 SlepcInitialize() line 262 in ../../../slepc/src/sys/slepcinit.c<br>PETSC ERROR: Logging has not been enabled.<br>You might have forgotten to call PetscInitialize().<br>PETSC ERROR: Logging has not been enabled.<br>You might have forgotten to call PetscInitialize().<br>--------------------------------------------------------------------------<br>MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_WORLD<br>with errorcode 56.<br><br>NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.<br>You may or may not see output from other processes, depending on<br>exactly when Open MPI kills them.<br></div><div><br></div><div>Thanks,</div><div>Sam</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Thu, Jun 25, 2020 at 7:37 PM Junchao Zhang <<a href="mailto:junchao.zhang@gmail.com" target="_blank">junchao.zhang@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Sam,<div> The MPI_Comm_create_keyval() error was fixed in maint/master. From the error message, it seems you need to configure --with-log=1</div><div> Otherwise, please send your full error stack trace and configure.log.</div><div> Thanks.<br clear="all"><div><div dir="ltr"><div dir="ltr">--Junchao Zhang</div></div></div><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Thu, Jun 25, 2020 at 2:18 PM Sam Guo <<a href="mailto:sam.guo@cd-adapco.com" target="_blank">sam.guo@cd-adapco.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Hi Junchao,<div> I now encountered the same error with parallel. I am wondering if there is a need for parallel fix as well.</div><div>[1]PETSC ERROR: #1 PetscInitialize() line 969 in ../../../petsc/src/sys/objects/pinit.c<br>PETSC ERROR: Logging has not been enabled.<br>You might have forgotten to call PetscInitialize().<br>PETSC ERROR: Logging has not been enabled.<br>You might have forgotten to call PetscInitialize().<br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Sat, Jun 20, 2020 at 7:35 PM Sam Guo <<a href="mailto:sam.guo@cd-adapco.com" target="_blank">sam.guo@cd-adapco.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Hi Junchao,<div> Your patch works.</div><div><br></div><div>Thanks,</div><div>Sam</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Sat, Jun 20, 2020 at 4:23 PM Junchao Zhang <<a href="mailto:junchao.zhang@gmail.com" target="_blank">junchao.zhang@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div dir="ltr"><br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Sat, Jun 20, 2020 at 12:24 PM Barry Smith <<a href="mailto:bsmith@petsc.dev" target="_blank">bsmith@petsc.dev</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div><div><br></div> Junchao,<div><br></div><div> This is a good bug fix. It solves the problem when PETSc initialize is called many times. </div><div><br></div><div> There is another fix you can do to limit PETSc mpiuni running out of attributes inside a single PETSc run:</div><div><br></div><div><br></div><div><div>int MPI_Comm_create_keyval(MPI_Copy_function *copy_fn,MPI_Delete_function *delete_fn,int *keyval,void *extra_state)</div><div>{</div><div> </div><div> if (num_attr >= MAX_ATTR){</div><div> for (i=0; i<num_attr; i++) {</div><div> if (!attr_keyval[i].extra_state) {</div></div></div></blockquote><div>attr_keyval[i].extra_state is provided by user (could be NULL). We can not rely on it. </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div><div><div> /* reuse this slot */</div><div> attr_keyval[i].extra_state = extra_state;</div><div> attr_keyval[i.]del = delete_fn;</div><div> *keyval = i;</div><div> return MPI_SUCCESS;</div><div> }</div><div> }</div><div> return MPIUni_Abort(MPI_COMM_WORLD,1);</div><div>}</div><div> return MPIUni_Abort(MPI_COMM_WORLD,1);</div><div> attr_keyval[num_attr].extra_state = extra_state;</div><div> attr_keyval[num_attr].del = delete_fn;</div><div> *keyval = num_attr++;</div><div> return MPI_SUCCESS;</div><div>}</div></div><div><br></div><div> This will work if the user creates tons of attributes but is constantly deleting some as they new ones. So long as the number outstanding at one time is < MAX_ATTR)</div><div><br></div><div>Barry</div><div><br></div><div><br></div><div><br></div><div><br><div><br><blockquote type="cite"><div>On Jun 20, 2020, at 10:54 AM, Junchao Zhang <<a href="mailto:junchao.zhang@gmail.com" target="_blank">junchao.zhang@gmail.com</a>> wrote:</div><br><div><div dir="ltr">I don't understand what your session means. Let's try this patch<div><br></div><div>diff --git a/src/sys/mpiuni/mpi.c b/src/sys/mpiuni/mpi.c<br>index d559a513..c058265d 100644<br>--- a/src/sys/mpiuni/mpi.c<br>+++ b/src/sys/mpiuni/mpi.c<br>@@ -283,6 +283,7 @@ int MPI_Finalize(void)<br> MPI_Comm_free(&comm);<br> comm = MPI_COMM_SELF;<br> MPI_Comm_free(&comm);<br>+ num_attr = 1; /* reset the counter */<br> MPI_was_finalized = 1;<br> return MPI_SUCCESS;<br> }<br><div><br></div><div><br clear="all"><div><div dir="ltr"><div dir="ltr">--Junchao Zhang</div></div></div><br></div></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Sat, Jun 20, 2020 at 10:48 AM Sam Guo <<a href="mailto:sam.guo@cd-adapco.com" target="_blank">sam.guo@cd-adapco.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Typo: I mean “Assuming initializer is only needed once for entire session”<br><br>On Saturday, June 20, 2020, Sam Guo <<a href="mailto:sam.guo@cd-adapco.com" target="_blank">sam.guo@cd-adapco.com</a>> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Assuming finalizer is only needed once for entire session(?), I can put initializer into the static block to call it once but where do I call finalizer?<div><br></div><div><br>On Saturday, June 20, 2020, Junchao Zhang <<a href="mailto:junchao.zhang@gmail.com" target="_blank">junchao.zhang@gmail.com</a>> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">The counter num_attr should be recycled. But first try to call PETSc initialize/Finalize only once to see it fixes the error.<br clear="all"><div><div dir="ltr"><div dir="ltr">--Junchao Zhang</div></div></div><br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Sat, Jun 20, 2020 at 12:48 AM Sam Guo <<a href="mailto:sam.guo@cd-adapco.com" target="_blank">sam.guo@cd-adapco.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">To clarify, I call PETSc initialize and PETSc finalize everytime I call SLEPc:<div><br><div><pre width="80" style="white-space:pre-wrap"> PetscInitializeNoPointers(argc,args,nullptr,nullptr);<br></pre><pre width="80" style="white-space:pre-wrap"> SlepcInitialize(&argc,&args,static_cast<char*>(nullptr),help);</pre><pre width="80" style="white-space:pre-wrap"> //calling slepc</pre><pre width="80" style="white-space:pre-wrap"> SlepcFinalize();</pre><pre width="80" style="white-space:pre-wrap"><span style="font-family:Arial,Helvetica,sans-serif;color:rgb(34,34,34)"> PetscFinalize();</span> </pre></div><div><br></div></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, Jun 19, 2020 at 10:32 PM Sam Guo <<a href="mailto:sam.guo@cd-adapco.com" target="_blank">sam.guo@cd-adapco.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Dear PETSc team,<div> When I called SLEPc multiple time, I eventually got following error: </div><div><br></div><div><span style="color:rgb(80,0,80)">MPI operation not supported by PETSc's sequential MPI wrappers<br></span>[0]PETSC ERROR: #1 PetscInitialize() line 967 in ../../../petsc/src/sys/objects/pinit.c<br>[0]PETSC ERROR: #2 SlepcInitialize() line 262 in ../../../slepc/src/sys/slepcinit.c<br>[0]PETSC ERROR: #3 SlepcInitializeNoPointers() line 359 in ../../../slepc/src/sys/slepcinit.c<span style="color:rgb(80,0,80)"><br>PETSC ERROR: Logging has not been enabled.<br>You might have forgotten to call PetscInitialize().</span> <br></div><div><br></div><div> I debugged: it is because of following in petsc/src/sys/mpiuni/mpi.c</div><div><br></div><div><div>if (num_attr >= MAX_ATTR)<br></div></div><div><br></div><div>in function int MPI_Comm_create_keyval(MPI_Copy_function *copy_fn,MPI_Delete_function *delete_fn,int *keyval,void *extra_state)</div><div><br></div><div>num_attr is declared static and keeps increasing every time MPI_Comm_create_keyval is called.</div><div><br></div><div>I am using petsc 3.11.3 but found 3.13.2 has the same logic.</div><div><br></div><div>Is this a bug or I didn't use it correctly?</div><div><br></div><div>Thanks,</div><div>Sam</div></div>
</blockquote></div>
</blockquote></div>
</blockquote></div>
</blockquote>
</blockquote></div>
</div></blockquote></div><br></div></div></blockquote></div></div>
</blockquote></div>
</blockquote></div>
</blockquote></div>
</blockquote></div>
</blockquote></div>
</div></blockquote></div><br></div></div></blockquote></div>
</div></blockquote></div><br></div></div></blockquote></div>
</blockquote></div>
</blockquote></div>
</blockquote></div>
</blockquote></div>