Thanks Barry!<br>There is really something about the human mind and preconceptions... Since I could trace it down only so far I assumed the issue was outside my code. Turns out I was missing exactly 1 PetscFunctionBegin! that is all it took.<br>
<br>Thanks again.<br><br>Daniel Lowell<br><br><br><br><div class="gmail_quote">On Thu, Nov 3, 2011 at 5:01 PM, Barry Smith <span dir="ltr"><<a href="mailto:bsmith@mcs.anl.gov">bsmith@mcs.anl.gov</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin: 0pt 0pt 0pt 0.8ex; border-left: 1px solid rgb(204, 204, 204); padding-left: 1ex;"><br>
The stack is pushed on PetscFunctionBegin; and popped on PetscFunctionReturn(0); so if the stack is smaller than it should be that means that there is a PetscFunctionReturn() without its partner PetscFunctionBegin;<br>
<br>
Unfortunately I cannot see from your stack frames where the problem is? Maybe your "custom gpu vector" is missing a begin somewhere?<br>
<span class="HOEnZb"><font color="#888888"><br>
<br>
Barry<br>
</font></span><div class="HOEnZb"><div class="h5"><br>
On Nov 3, 2011, at 4:54 PM, Daniel Lowell wrote:<br>
<br>
> Hello all,<br>
><br>
><br>
> In working with petsc-dev in the debugger I have found that petscstack<br>
> popped empty before PETSc returns to the main of the program; i.e.,<br>
> prematurely emptied.<br>
><br>
> My version:<br>
> PETSc Dev-Version 3.2.3<br>
><br>
><br>
> Okay I'll just walk to through the debug output as I found the issue, then<br>
> show where specifically it is being consumed.<br>
> I am running src/snes/examples/tutorials/ex14.<br>
><br>
> Using command:<br>
> gdb --args ./ex14 -pc_type none -vec_type seqgpu -mat_no_inode<br>
><br>
> That Vec type is a custom gpu vector.<br>
><br>
><br>
><br>
> First I break at SNESSolve:<br>
><br>
><br>
> Breakpoint 3, SNESSolve (snes=0x8457cb0, b=0x0, x=0x8472a70) at snes.c:2673<br>
> 2673 for (grid=0; grid<snes->gridsequence; grid++) {ierr =<br>
> PetscViewerASCIIPushTab(PETSC_VIEWER_STDOUT_(((PetscObject)snes)->comm));CHKERRQ(ierr);}<br>
><br>
><br>
> I take a look at the petscstack:<br>
><br>
> (gdb) set $ii=0<br>
> (gdb) p petscstack->function[$ii++]<br>
> $35 = 0x1083450 "SNESSolve"<br>
><br>
><br>
> Yup, it is in there:<br>
><br>
><br>
> (gdb) bt<br>
> #0 SNESSolve (snes=0x8457cb0, b=0x0, x=0x8472a70) at snes.c:2673<br>
> #1 0x0000000000407765 in main (argc=6, argv=0x7fffffffdfb8) at ex14.c:165<br>
><br>
> Also it matches the trace.<br>
><br>
> Next I break at SNESSolve_LS:<br>
><br>
><br>
> Breakpoint 4, SNESSolve_LS (snes=0x8457cb0) at ls.c:142<br>
> 142 SNES_LS *neP = (SNES_LS*)snes->data;<br>
><br>
><br>
><br>
> Check the stack again:<br>
><br>
><br>
> (gdb) set $ii=0<br>
> (gdb) p petscstack->function[$ii++]<br>
> $36 = 0x0<br>
> (gdb)<br>
> $37 = 0x0<br>
><br>
><br>
> What happened? Stack is now empty!<br>
><br>
><br>
><br>
> (gdb) bt<br>
> #0 SNESSolve_LS (snes=0x8457cb0) at ls.c:142<br>
> #1 0x0000000000612df5 in SNESSolve (snes=0x8457cb0, b=0x0, x=0x8472a70) at<br>
> snes.c:2695<br>
> #2 0x0000000000407765 in main (argc=6, argv=0x7fffffffdfb8) at ex14.c:165<br>
> (gdb) c<br>
> Continuing.<br>
><br>
><br>
> Backtrace shows of course two PETSc frames.<br>
><br>
><br>
><br>
><br>
> ...........................................................................<br>
> Something is not right. I've tracked it down to something going on here.<br>
><br>
><br>
><br>
><br>
> Breakpoint 16, VecSetType (vec=0x85aea00, method=0x1072069 "seq") at<br>
> vecreg.c:55<br>
> 55 PetscFunctionReturn(0);<br>
> (gdb) bt<br>
> #0 VecSetType (vec=0x85aea00, method=0x1072069 "seq") at vecreg.c:55<br>
> #1 0x00000000004ca5e0 in VecCreate_Standard (v=0x85aea00) at pbvec.c:265<br>
> #2 0x000000000048d3a4 in VecSetType (vec=0x85aea00, method=0x8468d70<br>
> "standard") at vecreg.c:53<br>
> #3 0x00000000005b7571 in DMCreateGlobalVector_DA (da=0x8465cf0,<br>
> g=0x8457ee8) at dadist.c:35<br>
> #4 0x00000000005f2c5f in DMCreateGlobalVector (dm=0x8465cf0,<br>
> vec=0x8457ee8) at dm.c:303<br>
> #5 0x00000000005b7017 in VecDuplicate_MPI_DA (g=0x8472a70, gg=0x8457ee8)<br>
> at dadist.c:17<br>
> #6 0x0000000000481240 in VecDuplicate (v=0x8472a70, newv=0x8457ee8) at<br>
> vector.c:541<br>
> #7 0x0000000000609a58 in SNESSetUp (snes=0x8457cb0) at snes.c:1599<br>
> #8 0x0000000000612b67 in SNESSolve (snes=0x8457cb0, b=0x0, x=0x8472a70) at<br>
> snes.c:2685<br>
> #9 0x0000000000407765 in main (argc=6, argv=0x7fffffffdfb8) at ex14.c:165<br>
> (gdb) p petscstack->currentsize<br>
> $160 = 9<br>
> (gdb) n<br>
> 56 }<br>
> (gdb)<br>
> VecCreate_Standard (v=0x85aea00) at pbvec.c:269<br>
> 269 PetscFunctionReturn(0);<br>
> (gdb) p petscstack->currentsize<br>
> $161 = 8<br>
> (gdb) n<br>
> 270 }<br>
> (gdb) p petscstack->currentsize<br>
> $162 = 7<br>
> (gdb) n<br>
><br>
><br>
><br>
><br>
> The stack is being consumed...<br>
><br>
><br>
><br>
> Breakpoint 16, VecSetType (vec=0x85aea00, method=0x8468d70 "standard") at<br>
> vecreg.c:55<br>
> 55 PetscFunctionReturn(0);<br>
> (gdb) p petscstack->currentsize<br>
> $163 = 7<br>
> (gdb) n<br>
> 56 }<br>
> (gdb) p petscstack->currentsize<br>
> $164 = 6<br>
> (gdb) n<br>
> DMCreateGlobalVector_DA (da=0x8465cf0, g=0x8457ee8) at dadist.c:36<br>
> 36 ierr = VecSetFromOptions(*g);CHKERRQ(ierr);<br>
> (gdb) p petscstack->currentsize<br>
> $165 = 6<br>
> (gdb) n<br>
><br>
> Breakpoint 16, VecSetType (vec=0x85aea00, method=0x7fffffffc9e0 "seqgpu")<br>
> at vecreg.c:55<br>
> 55 PetscFunctionReturn(0);<br>
> (gdb) p petscstack->currentsize<br>
> $166 = 5<br>
> (gdb) n<br>
> 56 }<br>
> (gdb) p petscstack->currentsize<br>
> $167 = 4<br>
> (gdb) n<br>
> VecSetTypeFromOptions_Private (vec=0x85aea00) at vector.c:1382<br>
> 1382 PetscFunctionReturn(0);<br>
> (gdb) p petscstack->currentsize<br>
> $168 = 4<br>
> (gdb) n<br>
> 1383 }<br>
> (gdb) p petscstack->currentsize<br>
> $169 = 3<br>
> (gdb) n<br>
> VecSetFromOptions (vec=0x85aea00) at vector.c:1418<br>
> 1418 if (vec->ops->setfromoptions) {<br>
> (gdb) p petscstack->currentsize<br>
> $170 = 3<br>
> (gdb) n<br>
> 1423 ierr =<br>
> PetscObjectProcessOptionsHandlers((PetscObject)vec);CHKERRQ(ierr);<br>
> (gdb) p petscstack->currentsize<br>
> $171 = 3<br>
> (gdb) n<br>
> 1424 ierr = PetscOptionsEnd();CHKERRQ(ierr);<br>
> (gdb)<br>
> 1413 ierr = PetscObjectOptionsBegin((PetscObject)vec);CHKERRQ(ierr);<br>
> (gdb)<br>
> 1424 ierr = PetscOptionsEnd();CHKERRQ(ierr);<br>
> (gdb)<br>
> 1426 ierr = VecViewFromOptions(vec,<br>
> ((PetscObject)vec)->name);CHKERRQ(ierr);<br>
> (gdb)<br>
> 1427 PetscFunctionReturn(0);<br>
> (gdb)<br>
> 1428 }<br>
> (gdb)<br>
> DMCreateGlobalVector_DA (da=0x8465cf0, g=0x8457ee8) at dadist.c:37<br>
> 37 ierr =<br>
> PetscObjectCompose((PetscObject)*g,"DM",(PetscObject)da);CHKERRQ(ierr);<br>
> (gdb) p petscstack->currentsize<br>
> $172 = 2<br>
> (gdb) n<br>
> 38 ierr = VecSetLocalToGlobalMapping(*g,da->ltogmap);CHKERRQ(ierr);<br>
> (gdb)<br>
> 39 ierr =<br>
> VecSetLocalToGlobalMappingBlock(*g,da->ltogmapb);CHKERRQ(ierr);<br>
> (gdb)<br>
> 40 ierr = VecSetBlockSize(*g,dd->w);CHKERRQ(ierr);<br>
> (gdb)<br>
> 41 ierr =<br>
> VecSetOperation(*g,VECOP_VIEW,(void(*)(void))VecView_MPI_DA);CHKERRQ(ierr);<br>
> (gdb)<br>
> 42 ierr =<br>
> VecSetOperation(*g,VECOP_LOAD,(void(*)(void))VecLoad_Default_DA);CHKERRQ(ierr);<br>
> (gdb)<br>
> 43 ierr =<br>
> VecSetOperation(*g,VECOP_DUPLICATE,(void(*)(void))VecDuplicate_MPI_DA);CHKERRQ(ierr);<br>
> (gdb)<br>
> 44 PetscFunctionReturn(0);<br>
> (gdb)<br>
> 45 }<br>
> (gdb) p petscstack->currentsize<br>
> $173 = 1<br>
> (gdb) n<br>
> DMCreateGlobalVector (dm=0x8465cf0, vec=0x8457ee8) at dm.c:304<br>
> 304 PetscFunctionReturn(0);<br>
> (gdb) p petscstack->currentsize<br>
> $174 = 1<br>
> (gdb) n<br>
> 305 }<br>
> (gdb) p petscstack->currentsize<br>
> $175 = 0<br>
><br>
><br>
><br>
> The petscstack is fully used up at this point, however...<br>
><br>
> (gdb) bt<br>
> #0 DMCreateGlobalVector (dm=0x8465cf0, vec=0x8457ee8) at dm.c:305<br>
> #1 0x00000000005b7017 in VecDuplicate_MPI_DA (g=0x8472a70, gg=0x8457ee8)<br>
> at dadist.c:17<br>
> #2 0x0000000000481240 in VecDuplicate (v=0x8472a70, newv=0x8457ee8) at<br>
> vector.c:541<br>
> #3 0x0000000000609a58 in SNESSetUp (snes=0x8457cb0) at snes.c:1599<br>
> #4 0x0000000000612b67 in SNESSolve (snes=0x8457cb0, b=0x0, x=0x8472a70) at<br>
> snes.c:2685<br>
> #5 0x0000000000407765 in main (argc=6, argv=0x7fffffffdfb8) at ex14.c:165<br>
> (gdb)<br>
><br>
><br>
><br>
><br>
><br>
><br>
> At first I thought it was due to the fact I modified ex14 at line 108 and<br>
> inserted:<br>
> ierr = VecSetFromOptions(x);CHKERRQ(ierr);<br>
><br>
> And th petsc stack is being consumed prematurely here just as it is above.<br>
> However since this is also happening in SNESSolve...<br>
><br>
> VecCUSP and VecPThreads I believe are activated through configuration<br>
> options(i.e., preprocessing) so I'm not sure if anyone has hit this issue.<br>
> The gpu vector type I'm implementing is activated as command line option.<br>
> Has anyone else encountered this issue?<br>
><br>
> Any insight would be appreciated.<br>
><br>
><br>
><br>
><br>
> Thanks,<br>
><br>
> Daniel Lowell<br>
><br>
<br>
</div></div></blockquote></div><br><br clear="all"><br>-- <br>Daniel Lowell<br><br>