<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body>
<div dir="ltr">Discussed with Mr. Hong. The following two new lines fix the problem. Either single line also works, but I think we should have both. Similar things happen to dmsnes and dmts. But I need DM experts' feedback before changing them. Thanks.
<div>
<div>
<div><br>
</div>
<div><font face="courier new, monospace">diff --git a/src/ksp/ksp/interface/dmksp.c b/src/ksp/ksp/interface/dmksp.c<br>
index 9ce75090..0ab69574 100644<br>
--- a/src/ksp/ksp/interface/dmksp.c<br>
+++ b/src/ksp/ksp/interface/dmksp.c<br>
@@ -80,6 +80,7 @@ PetscErrorCode DMKSPCopy(DMKSP kdm,DMKSP nkdm)<br>
nkdm->rhsctx = kdm->rhsctx;<br>
nkdm->initialguessctx = kdm->initialguessctx;<br>
nkdm->data = kdm->data;<br>
+ nkdm->originaldm = kdm->originaldm;<br>
<br>
nkdm->fortran_func_pointers[0] = kdm->fortran_func_pointers[0];<br>
nkdm->fortran_func_pointers[1] = kdm->fortran_func_pointers[1];<br>
@@ -156,6 +157,7 @@ PetscErrorCode DMGetDMKSPWrite(DM dm,DMKSP *kspdm)<br>
ierr = DMKSPCopy(oldkdm,kdm);CHKERRQ(ierr);<br>
ierr = DMKSPDestroy((DMKSP*)&dm->dmksp);CHKERRQ(ierr);<br>
dm->dmksp = (PetscObject)kdm;<br>
+ kdm->originaldm = dm;<br>
}<br>
*kspdm = kdm;<br>
PetscFunctionReturn(0);</font><br>
</div>
<div><br clear="all">
<div>
<div dir="ltr" class="m_2804231590164220251gmail_signature" data-smartmail="gmail_signature">
<div dir="ltr">--Junchao Zhang</div>
</div>
</div>
<br>
</div>
</div>
</div>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Fri, Jun 14, 2019 at 1:07 PM Junchao Zhang <<a href="mailto:jczhang@mcs.anl.gov" target="_blank">jczhang@mcs.anl.gov</a>> wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="ltr">
<div dir="ltr"><br>
</div>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Fri, Jun 14, 2019 at 1:01 PM Lawrence Mitchell <<a href="mailto:wence@gmx.li" target="_blank">wence@gmx.li</a>> wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<br>
<br>
> On 14 Jun 2019, at 18:44, Zhang, Junchao via petsc-dev <<a href="mailto:petsc-dev@mcs.anl.gov" target="_blank">petsc-dev@mcs.anl.gov</a>> wrote:<br>
> <br>
> Hello,<br>
> I am investigating petsc issue 306. One can produce the problem with src/snes/examples/tutorials/ex9.c and mpirun -n 3 ./ex9 -snes_grid_sequence 3 -snes_converged_reason -pc_type mg<br>
> The program can run to finish, or crash, or hang. The error appears either in PetscGatherMessageLengths or PetscCommBuildTwoSided(). From from my debugging, the following routine is suspicious. It claims not collective, but it might call DMKSPCreate,
which can indirectly call a collective MPI_Comm_dup().<br>
<br>
I would have thought that DMKSPCreate could only call MPI_Comm_dup (via PetscCommDuplicate) if the incoming dm has a communicator which is /not/ a PETSc communicator. Given that all PETSc objects must (?) return a PETSc communicator when calling PetscObjectComm,
this function is presumably incidentally not collective (although it is logically collective I would have thought), oh, and with PETSC_USE_DEBUG, there's a barrier even in the "return immediately with a PETSc comm" case.<br>
</blockquote>
<div><br>
</div>
<div>Yes, PetscCommDuplicate not always calls MPI_Comm_dup, but it decreases MPI tag, resulting in MPI tag mismatch and crashing the code. If I turn on PETSC_USE_DEBUG, the code sometime (but not all times) hang in PetscCommDuplicate. </div>
<div> </div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<br>
FWIW, the call site looks to be collective (PCSetUp_MG).<br>
<br>
Lawrence<br>
<br>
> Can someone familiar with the code explain it? Thanks.<br>
> <br>
> /*@C<br>
> DMGetDMKSPWrite - get write access to private DMKSP context from a DM<br>
> <br>
> Not Collective<br>
> <br>
> Input Argument:<br>
> . dm - DM to be used with KSP<br>
> <br>
> Output Argument:<br>
> . kspdm - private DMKSP context<br>
> <br>
> Level: developer<br>
> <br>
> .seealso: DMGetDMKSP()<br>
> @*/<br>
> PetscErrorCode DMGetDMKSPWrite(DM dm,DMKSP *kspdm)<br>
> {<br>
> PetscErrorCode ierr;<br>
> DMKSP kdm;<br>
> <br>
> PetscFunctionBegin;<br>
> PetscValidHeaderSpecific(dm,DM_CLASSID,1);<br>
> ierr = DMGetDMKSP(dm,&kdm);CHKERRQ(ierr);<br>
> if (!kdm->originaldm) kdm->originaldm = dm;<br>
> if (kdm->originaldm != dm) { /* Copy on write */<br>
> DMKSP oldkdm = kdm;<br>
> ierr = PetscInfo(dm,"Copying DMKSP due to write\n");CHKERRQ(ierr);<br>
> ierr = DMKSPCreate(PetscObjectComm((PetscObject)dm),&kdm);CHKERRQ(ierr);<br>
> ierr = DMKSPCopy(oldkdm,kdm);CHKERRQ(ierr);<br>
> ierr = DMKSPDestroy((DMKSP*)&dm->dmksp);CHKERRQ(ierr);<br>
> dm->dmksp = (PetscObject)kdm;<br>
> }<br>
> *kspdm = kdm;<br>
> PetscFunctionReturn(0);<br>
> }<br>
> <br>
> The calling stack is<br>
> ...<br>
> #20 0x00007fcf3e3d23e2 in PetscCommDuplicate (comm_in=comm_in@entry=-2080374782, comm_out=comm_out@entry=0x557a18304db0, first_tag=first_tag@entry=0x557a18304de4) at /home/jczhang/petsc/src/sys/objects/tagm.c:162<br>
> #21 0x00007fcf3e3d7730 in PetscHeaderCreate_Private (h=0x557a18304d70, classid=<optimized out>, class_name=class_name@entry=0x7fcf3f7f762a "DMKSP", descr=descr@entry=0x7fcf3f7f762a "DMKSP", mansec=mansec@entry=0x7fcf3f7f762a "DMKSP", comm=comm@entry=-2080374782,
destroy=0x7fcf3f350570 <DMKSPDestroy>, view=0x0) at /home/jczhang/petsc/src/sys/objects/inherit.c:64<br>
> #22 0x00007fcf3f3504c9 in DMKSPCreate (comm=-2080374782, kdm=kdm@entry=0x7ffc1d4d00f8) at /home/jczhang/petsc/src/ksp/ksp/interface/dmksp.c:24<br>
> #23 0x00007fcf3f35150f in DMGetDMKSPWrite (dm=0x557a18541a10, kspdm=kspdm@entry=0x7ffc1d4d01a8) at /home/jczhang/petsc/src/ksp/ksp/interface/dmksp.c:155<br>
> #24 0x00007fcf3f1bb20e in PCSetUp_MG (pc=<optimized out>) at /home/jczhang/petsc/src/ksp/pc/impls/mg/mg.c:682<br>
> #25 0x00007fcf3f204bea in PCSetUp (pc=0x557a17dc1860) at /home/jczhang/petsc/src/ksp/pc/interface/precon.c:894<br>
> #26 0x00007fcf3f32ba4b in KSPSetUp (ksp=0x557a17d73500) at /home/jczhang/petsc/src/ksp/ksp/interface/itfunc.c:377<br>
> #27 0x00007fcf3f41e43e in SNESSolve_VINEWTONRSLS (snes=0x557a17bff210) at /home/jczhang/petsc/src/snes/impls/vi/rs/virs.c:502<br>
> #28 0x00007fcf3f3fa191 in SNESSolve (snes=0x557a17bff210, b=0x0, x=<optimized out>) at /home/jczhang/petsc/src/snes/interface/snes.c:4433<br>
> #29 0x0000557a16432095 in main (argc=<optimized out>, argv=<optimized out>) at /home/jczhang/petsc/src/snes/examples/tutorials/ex9.c:105<br>
> <br>
> --Junchao Zhang<br>
<br>
</blockquote>
</div>
</div>
</blockquote>
</div>
</body>
</html>