<div dir="ltr"><div class="gmail_extra"><div class="gmail_quote">On Fri, Jul 24, 2015 at 11:58 AM, Aaron Kitzmiller <span dir="ltr"><<a href="mailto:akitzmiller@g.harvard.edu" target="_blank">akitzmiller@g.harvard.edu</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div style="word-wrap:break-word">I'm using 1.8.3 which is Sept 2014. I'll try some others.<div><br></div><div>Do you happen to know what the bug is (or a good Google term for finding it)?</div></div></blockquote><div><br></div><div>It was fixed in 1.8.4</div><div><br></div><div> <a href="https://github.com/open-mpi/ompi-release/commit/6dcd42be28ac1a8dac887bf5e6c9ffb9b99f9511">https://github.com/open-mpi/ompi-release/commit/6dcd42be28ac1a8dac887bf5e6c9ffb9b99f9511</a></div><div><br></div><div> Matt</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div style="word-wrap:break-word"><div>ajk</div><div><br><div>
<div>Aaron Kitzmiller</div><div>Informatics and Scientific Applications</div><div><a href="mailto:aaron_kitzmiller@harvard.edu" target="_blank">aaron_kitzmiller@harvard.edu</a></div><div><br></div><br>
</div>
<br><div><blockquote type="cite"><div>On Jul 24, 2015, at 12:42 PM, Matthew Knepley <<a href="mailto:knepley@gmail.com" target="_blank">knepley@gmail.com</a>> wrote:</div><br><div><div dir="ltr" style="font-family:Helvetica;font-size:14px;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px"><div class="gmail_extra"><div class="gmail_quote">On Fri, Jul 24, 2015 at 11:36 AM, Aaron Kitzmiller<span> </span><span dir="ltr"><<a href="mailto:akitzmiller@g.harvard.edu" target="_blank">akitzmiller@g.harvard.edu</a>></span><span> </span>wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div style="word-wrap:break-word">futex is a Linux system call used for locking shared resources.<div><br></div><div>It could be indicative of an MPI problem. I wouldn't be surprised. If anyone has any idea how to get around it that would be great. We have dozens of applications on our compute cluster that use MPI, this version being our default. I'm wondering if there is something specific to the mix of MPI flavor / compiler, etc. that could be going on here.</div></div></blockquote><div><br></div><div>Yes, this is a bug in OpenMPI that has been open for years.</div><div><br></div><div>Can you please switch to MPICH and try another test? I thought the newest version of OpenMPI had fixed this, but maybe you are using an older release.</div><div><br></div><div> Thanks,</div><div><br></div><div> Matt</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div style="word-wrap:break-word"><div>This is the gdb stack trace:</div><div><br></div><div><font face="Andale Mono">#0 0x00000039c6a0e264 in __lll_lock_wait () from /lib64/libpthread.so.0<br>#1 0x00000039c6a09508 in _L_lock_854 () from /lib64/libpthread.so.0<br>#2 0x00000039c6a093d7 in pthread_mutex_lock () from /lib64/libpthread.so.0<br>#3 0x00002aaaaf13ddd4 in opal_mutex_lock (attr_hash=0x2aaaaf651c70, key=128, attribute=0x7fffffffc200, flag=0xffffffffffffffff)<br> at ../opal/threads/mutex_unix.h:104<br>#4 ompi_attr_get_c (attr_hash=0x2aaaaf651c70, key=128, attribute=0x7fffffffc200, flag=0xffffffffffffffff)<br> at attribute/attribute.c:758<br>#5 0x00002aaaaf17080e in PMPI_Attr_get (comm=0x2aaaaf651c70, keyval=128, attribute_val=0x7fffffffc200, flag=0xffffffffffffffff)<br> at pattr_get.c:61<br>#6 0x00002aaaaacad0b3 in Petsc_DelComm_Outer (comm=0x2aaaaf6d4140, keyval=13, attr_val=0x7af160, extra_state=0x0)<br> at /n/home08/lchristakis/petsc/petsc-3.5.4/src/sys/objects/pinit.c:409<br>#7 0x00002aaaaf13f1a4 in ompi_attr_delete_impl (type=2942639216, object=0x80, attr_hash=0x7fffffffc200, key=-1, predefined=112 'p')<br> at attribute/attribute.c:970<br>#8 0x00002aaaaf13ee02 in ompi_attr_delete (type=2942639216, object=0x80, attr_hash=0x7fffffffc200, key=-1, predefined=112 'p')<br> at attribute/attribute.c:1019<br>#9 0x00002aaaaf170710 in PMPI_Attr_delete (comm=0x2aaaaf651c70, keyval=128) at pattr_delete.c:59<br>#10 0x00002aaaaac61848 in PetscCommDestroy (comm=0x888cf0) at /n/home08/lchristakis/petsc/petsc-3.5.4/src/sys/objects/tagm.c:256<br>#11 0x00002aaaaac6a273 in PetscHeaderDestroy_Private (h=0x888ce0)<br> at /n/home08/lchristakis/petsc/petsc-3.5.4/src/sys/objects/inherit.c:121<br>#12 0x00002aaaaaf51512 in VecDestroy (v=0x7fffffffcbd0)<br> at /n/home08/lchristakis/petsc/petsc-3.5.4/src/vec/vec/interface/vector.c:434<br>#13 0x00002aaaab9c5c7f in DMSetUp_DA_2D (da=0x87b1b0) at /n/home08/lchristakis/petsc/petsc-3.5.4/src/dm/impls/da/da2.c:776<br>#14 0x00002aaaaba73bfd in DMSetUp_DA (da=0x87b1b0) at /n/home08/lchristakis/petsc/petsc-3.5.4/src/dm/impls/da/dareg.c:25<br>#15 0x00002aaaab93399a in DMSetUp (dm=0x87b1b0) at /n/home08/lchristakis/petsc/petsc-3.5.4/src/dm/interface/dm.c:560<br>#16 0x00002aaaab9c6941 in DMDACreate2d (comm=0x2aaaaf6d45c0, bx=DM_BOUNDARY_NONE, by=DM_BOUNDARY_NONE, <br> stencil_type=DMDA_STENCIL_STAR, M=-4, N=-4, m=-1, n=-1, dof=1, s=1, lx=0x0, ly=0x0, da=0x7fffffffd668)<br> at /n/home08/lchristakis/petsc/petsc-3.5.4/src/dm/impls/da/da2.c:862<br>#17 0x00000000004023d0 in main (argc=1, argv=0x7fffffffd8c8)<br> at /n/home08/lchristakis/petsc/petsc-3.5.4/src/snes/examples/tutorials/ex5.c:116<br></font><br></div><div><br><div><div>Aaron Kitzmiller</div><div>Informatics and Scientific Applications</div><div><a href="mailto:aaron_kitzmiller@harvard.edu" target="_blank">aaron_kitzmiller@harvard.edu</a></div><div><br></div><br></div><br><div><blockquote type="cite"><div>On Jul 24, 2015, at 12:18 PM, Matthew Knepley <<a href="mailto:knepley@gmail.com" target="_blank">knepley@gmail.com</a>> wrote:</div><br><div><div dir="ltr" style="font-family:Helvetica;font-size:14px;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px"><div class="gmail_extra"><div class="gmail_quote">On Fri, Jul 24, 2015 at 11:17 AM, Matthew Knepley<span> </span><span dir="ltr"><<a href="mailto:knepley@gmail.com" target="_blank">knepley@gmail.com</a>></span><span> </span>wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><span>On Fri, Jul 24, 2015 at 11:09 AM, Aaron Kitzmiller<span> </span><span dir="ltr"><<a href="mailto:akitzmiller@g.harvard.edu" target="_blank">akitzmiller@g.harvard.edu</a>></span><span> </span>wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div style="word-wrap:break-word">Doesn't run. Hangs just like the tests do.<div><br></div><div>I doubt it's helpful, but when I run it under strace, it hangs on a "futex". The last thing vaguely informative was an attempt to read the non-existent .petscrc.</div></div></blockquote><div><br></div></span><div>Run in the debugger and get a stack trace.</div></div></div></div></blockquote><div><br></div><div>Also futex does not appear in the PETSc source:</div><div><br></div><div> knepley/feature-snes-deflation *+$|MERGING:/PETSc3/petsc/petsc-dev$ find src -name "*.c" | xargs grep futex</div><div> find src -name "*.c" | xargs grep futex</div><div><br></div><div>You have an MPI problem.</div><div><br></div><div> Matt</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><div> <span> </span>Matt</div><span><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div style="word-wrap:break-word"><div>ajk</div><div><br><div><div>Aaron Kitzmiller</div><div>Informatics and Scientific Applications</div><div><a href="mailto:aaron_kitzmiller@harvard.edu" target="_blank">aaron_kitzmiller@harvard.edu</a></div><div><br></div><br></div><br><div><blockquote type="cite"><div>On Jul 24, 2015, at 11:21 AM, Matthew Knepley <<a href="mailto:knepley@gmail.com" target="_blank">knepley@gmail.com</a>> wrote:</div><br><div><span style="font-family:Helvetica;font-size:14px;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;float:none;display:inline!important"> <span> </span>./ex5 -snes_monitor</span></div></blockquote></div><br></div></div></blockquote></span></div><br><br clear="all"><span><font color="#888888"><div><br></div>--<span> </span><br><div>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>-- Norbert Wiener</div></font></span></div></div></blockquote></div><span><font color="#888888"><br><br clear="all"><span class="HOEnZb"><font color="#888888"><div><br></div>--<span> </span><br><div>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>-- Norbert Wiener</div></font></span></font></span></div></div></div></blockquote></div><span class="HOEnZb"><font color="#888888"><br></font></span></div></div></blockquote></div><span class="HOEnZb"><font color="#888888"><br><br clear="all"><div><br></div>--<span> </span><br><div>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>-- Norbert Wiener</div></font></span></div></div></div></blockquote></div><br></div></div></blockquote></div><br><br clear="all"><div><br></div>-- <br><div class="gmail_signature">What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>-- Norbert Wiener</div>
</div></div>