<div dir="ltr"><div dir="ltr">On Thu, Apr 1, 2021 at 6:24 PM Junchao Zhang <<a href="mailto:junchao.zhang@gmail.com">junchao.zhang@gmail.com</a>> wrote:<br></div><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>Matt,</div><div>It is easy for me to know it is not an SF bug, since the code passes a null pointer (as leafdata) to an SF with 9 roots and leaves. But I could not figure out what the root reason is. </div></div></blockquote><div><br></div><div>Junchao, thanks so much for looking at this. With what you show below, I think I can figure it out now.</div><div><br></div><div> Thanks,</div><div><br></div><div> Matt</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>Here is my finding.</div><div><span style="color:rgb(0,0,0);font-family:Menlo,Monaco,"Courier New",monospace;font-size:14px;white-space:pre-wrap">In </span><span style="font-family:Menlo,Monaco,"Courier New",monospace;font-size:14px;white-space:pre-wrap;color:rgb(121,94,38)">DMSetUp</span><span style="color:rgb(0,0,0);font-family:Menlo,Monaco,"Courier New",monospace;font-size:14px;white-space:pre-wrap">(forest) >> </span>DMSetUp_p4est >> DMPforestGetPlex >> DMConvert_pforest_plex >> .. >> DMShareDiscretization, we have</div><div> dmB->sectionSF = dmA->sectionSF;<br></div><div>Which makes the two DMs, forest, and ((DM_Forest_pforest*) ((DM_Forest*) forest->data)->data)->plex, share the same sectionSF. But in the test, there are two places to populate the sectionSF based on different DMs.</div><div><br></div><div>1) In <span style="font-family:Menlo,Monaco,"Courier New",monospace;font-size:14px;white-space:pre-wrap;color:rgb(121,94,38)">DMGlobalToLocal</span><span style="color:rgb(0,0,0);font-family:Menlo,Monaco,"Courier New",monospace;font-size:14px;white-space:pre-wrap">(forest, g, INSERT_VALUES, l)</span>, the SF is built based on forest and its local/globalSection.</div><div>2) In<span style="color:rgb(0,0,0);font-family:Menlo,Monaco,"Courier New",monospace;font-size:14px;white-space:pre-wrap"> </span><span style="font-family:Menlo,Monaco,"Courier New",monospace;font-size:14px;white-space:pre-wrap;color:rgb(121,94,38)">VecView</span><span style="color:rgb(0,0,0);font-family:Menlo,Monaco,"Courier New",monospace;font-size:14px;white-space:pre-wrap">(g, viewer), </span>which does<br> ierr = VecGetDM(vec,&dm);CHKERRQ(ierr);</div><blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px"><div> ierr = DMPforestGetPlex(dm,&plex);CHKERRQ(ierr);</div><div> ierr = VecSetDM(vec,plex);CHKERRQ(ierr);</div><div> ierr = VecView_Plex(vec,viewer);CHKERRQ(ierr);</div><div> ierr = VecSetDM(vec,dm);CHKERRQ(ierr);</div></blockquote><div>VecView_Plex() calls VecView_Plex_HDF5_Internal(), which builds the SF based on plex's local/globalSection. <br></div><div><br></div><div>Depending on which function is called first, we get different SFs. The crashed one did 1) first and then 2). The 'good' one did 2) and then 1). But even the good one is wrong, since it gives an empty SF (thus not crashing the code).</div><div><br></div><div><div dir="ltr"><div dir="ltr">--Junchao Zhang</div></div></div><br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, Mar 30, 2021 at 5:44 AM Matthew Knepley <<a href="mailto:knepley@gmail.com" target="_blank">knepley@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div dir="ltr">On Mon, Mar 29, 2021 at 11:05 PM Junchao Zhang <<a href="mailto:junchao.zhang@gmail.com" target="_blank">junchao.zhang@gmail.com</a>> wrote:<br></div><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Matt,<div> I can reproduce the error. Let me see what is wrong.</div><div></div></div></blockquote><div><br></div><div>Thanks! It might be a bug in Plex or Forest as well, but it is hard for me to tell.</div><div><br></div><div> Matt</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div> Thanks.<br clear="all"><div><div dir="ltr"><div dir="ltr">--Junchao Zhang</div></div></div><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, Mar 29, 2021 at 2:16 PM Matthew Knepley <<a href="mailto:knepley@gmail.com" target="_blank">knepley@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>Junchao,</div><div><br></div>I have an SF problem, which I think is a caching bug, but it is hard to see what is happening in the internals. I have made a small example which should help you see what is wrong. It is attached.<div><br></div><div>If you run without arguments, you get</div><div><br></div><div>master *:~/Downloads/tmp/Salac$ ./forestHDF<br>[0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------<br>[0]PETSC ERROR: Null argument, when expecting valid pointer<br>[0]PETSC ERROR: Trying to copy to a null pointer<br>[0]PETSC ERROR: See <a href="https://www.mcs.anl.gov/petsc/documentation/faq.html" target="_blank">https://www.mcs.anl.gov/petsc/documentation/faq.html</a> for trouble shooting.<br>[0]PETSC ERROR: Petsc Development GIT revision: v3.14.5-879-g03cacdc99d GIT Date: 2021-03-22 01:02:08 +0000<br>[0]PETSC ERROR: ./forestHDF on a arch-master-debug named MacBook-Pro.fios-router.home by knepley Mon Mar 29 15:14:16 2021<br>[0]PETSC ERROR: Configure options --PETSC_ARCH=arch-master-debug --download-bamg --download-chaco --download-ctetgen --download-egads --download-eigen --download-exodusii --download-fftw --download-hpddm --download-libpng --download-metis --download-ml --download-mumps --download-netcdf --download-opencascade --download-p4est --download-parmetis --download-pnetcdf --download-scalapack --download-slepc --download-suitesparse --download-superlu_dist --download-triangle --with-cmake-exec=/PETSc3/petsc/apple/bin/cmake --with-ctest-exec=/PETSc3/petsc/apple/bin/ctest --with-hdf5-dir=/PETSc3/petsc/apple --with-mpi-dir=/PETSc3/petsc/apple --with-shared-libraries --with-slepc --with-zlib --download-tetgen<br>[0]PETSC ERROR: #1 PetscMemcpy() at /PETSc3/petsc/petsc-dev/include/petscsys.h:1798<br>[0]PETSC ERROR: #2 UnpackAndInsert_PetscReal_1_1() at /PETSc3/petsc/petsc-dev/src/vec/is/sf/impls/basic/sfpack.c:426<br>[0]PETSC ERROR: #3 ScatterAndInsert_PetscReal_1_1() at /PETSc3/petsc/petsc-dev/src/vec/is/sf/impls/basic/sfpack.c:426<br>[0]PETSC ERROR: #4 PetscSFLinkScatterLocal() at /PETSc3/petsc/petsc-dev/src/vec/is/sf/impls/basic/sfpack.c:1248<br>[0]PETSC ERROR: #5 PetscSFBcastBegin_Basic() at /PETSc3/petsc/petsc-dev/src/vec/is/sf/impls/basic/sfbasic.c:193<br>[0]PETSC ERROR: #6 PetscSFBcastWithMemTypeBegin() at /PETSc3/petsc/petsc-dev/src/vec/is/sf/interface/sf.c:1493<br>[0]PETSC ERROR: #7 DMGlobalToLocalBegin() at /PETSc3/petsc/petsc-dev/src/dm/interface/dm.c:2565<br>[0]PETSC ERROR: #8 VecView_Plex_HDF5_Internal() at /PETSc3/petsc/petsc-dev/src/dm/impls/plex/plexhdf5.c:251<br>[0]PETSC ERROR: #9 VecView_Plex() at /PETSc3/petsc/petsc-dev/src/dm/impls/plex/plex.c:385<br>[0]PETSC ERROR: #10 VecView_p4est() at /PETSc3/petsc/petsc-dev/src/dm/impls/forest/p4est/pforest.c:4922<br>[0]PETSC ERROR: #11 VecView() at /PETSc3/petsc/petsc-dev/src/vec/vec/interface/vector.c:613<br>[0]PETSC ERROR: #12 main() at /Users/knepley/Downloads/tmp/Salac/forestHDF.c:53<br>[0]PETSC ERROR: PETSc Option Table entries:<br>[0]PETSC ERROR: -malloc_debug<br>[0]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint@mcs.anl.gov----------<br>application called MPI_Abort(MPI_COMM_SELF, 53001) - process 0<br>[unset]: write_line error; fd=-1 buf=:cmd=abort exitcode=53001<br><div><br></div><div>If you run with</div><div><br></div><div> <span style="font-variant-ligatures:no-common-ligatures;color:rgb(0,0,0);font-family:Menlo;font-size:11px">./forestHDF -write_early</span></div><div><span style="font-variant-ligatures:no-common-ligatures;color:rgb(0,0,0);font-family:Menlo;font-size:11px"><br></span></div><div>or</div><div><br></div><div>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures"> ./forestHDF -no_g2l</span></p></div><div><br></div><div>Then it is fine. Thus it appears to me that if you run a G2L at the wrong time, something is incorrectly cached.</div><div><br></div><div> Thanks,</div><div><br></div><div> Matt</div><div><br></div><div>--<br></div><div dir="ltr"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>-- Norbert Wiener</div><div><br></div><div><a href="http://www.cse.buffalo.edu/~knepley/" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br></div></div></div></div></div></div></div></div></div>
</blockquote></div>
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>-- Norbert Wiener</div><div><br></div><div><a href="http://www.cse.buffalo.edu/~knepley/" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br></div></div></div></div></div></div></div></div>
</blockquote></div>
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr" class="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>-- Norbert Wiener</div><div><br></div><div><a href="http://www.cse.buffalo.edu/~knepley/" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br></div></div></div></div></div></div></div></div>