On Thu, Nov 17, 2011 at 3:54 PM, Robert Ellis <span dir="ltr"><<a href="mailto:Robert.Ellis@geosoft.com">Robert.Ellis@geosoft.com</a>></span> wrote:<br><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">
<div lang="EN-CA" link="blue" vlink="purple">
<div>
<p class="MsoNormal">Hello All,<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">I have a troubling intermittent problem with the simple VecSetValues/VecAssemblyBegin functions after porting a robust long working application to a cloud platform.
<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p><u></u><span style="font-family:Symbol"><span>ˇ<span style="font:7.0pt "Times New Roman"">
</span></span></span><u></u>I have 30M doubles on rank0<u></u><u></u></p>
<p><u></u><span style="font-family:Symbol"><span>ˇ<span style="font:7.0pt "Times New Roman"">
</span></span></span><u></u>I intend to assign them non sequentially among 32 processors, ranks 1-31.<u></u><u></u></p>
<p><u></u><span style="font-family:Symbol"><span>ˇ<span style="font:7.0pt "Times New Roman"">
</span></span></span><u></u>On rank0 only I use VecSetValues(x,...) to make the assignment. So far everything is fine.<u></u><u></u></p>
<p><u></u><span style="font-family:Symbol"><span>ˇ<span style="font:7.0pt "Times New Roman"">
</span></span></span><u></u>I call VecAssemblyBegin expecting this to distribute the values appropriately.<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">Sometimes this works, but about 50% of the time I see errors, immediately on calling VecAssemblyBegin, of the following form:<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal"> [23]PETSC ERROR: Fatal error in MPI_Allreduce: Other MPI error, error stack:<u></u><u></u></p>
<p class="MsoNormal"> MPI_Allreduce(919).........................: MPI_Allreduce(sbuf=0000000012DE29B0, rbuf=00000000069F6ED0, count=32, dtype=USER, op=0x98000000, comm=0x84000002) failed<u></u><u></u></p>
<p class="MsoNormal"> MPIR_Allreduce_impl(776)...................:<u></u><u></u></p>
<p class="MsoNormal"> MPIR_Allreduce_intra(220)..................:<u></u><u></u></p>
<p class="MsoNormal"> MPIR_Bcast_impl(1273)......................:<u></u><u></u></p>
<p class="MsoNormal"> MPIR_Bcast_intra(1107).....................:<u></u><u></u></p>
<p class="MsoNormal"> MPIR_Bcast_binomial(143)...................:<u></u><u></u></p>
<p class="MsoNormal"> MPIC_Recv(110).............................:<u></u><u></u></p>
<p class="MsoNormal"> MPIC_Wait(540).............................:<u></u><u></u></p>
<p class="MsoNormal"> MPIDI_CH3I_Progress(353)...................:<u></u><u></u></p>
<p class="MsoNormal"> MPID_nem_mpich2_blocking_recv(905).........:<u></u><u></u></p>
<p class="MsoNormal"> MPID_nem_newtcp_module_poll(37)............:<u></u><u></u></p>
<p class="MsoNormal"> MPID_nem_newtcp_module_connpoll(2655)......:<u></u><u></u></p>
<p class="MsoNormal"> recv_id_or_tmpvc_info_success_handler(1278): read from socket failed - No error<u></u><u></u></p>
<p class="MsoNormal"> --------------------- Error Message ------------------------------------<u></u><u></u></p>
<p class="MsoNormal"> [23]PETSC ERROR: Out of memory. This could be due to allocating<u></u><u></u></p>
<p class="MsoNormal"> [23]PETSC ERROR: too large an object or bleeding by not properly<u></u><u></u></p>
<p class="MsoNormal"> [23]PETSC ERROR: destroying unneeded objects.<u></u><u></u></p>
<p class="MsoNormal"> [23]PETSC ERROR: Memory allocated 0 Memory used by process 0<u></u><u></u></p>
<p class="MsoNormal"> [23]PETSC ERROR: Try running with -malloc_dump or -malloc_log for info.<u></u><u></u></p>
<p class="MsoNormal"> [23]PETSC ERROR: Memory requested 18446744066053327000!<u></u><u></u></p>
<p class="MsoNormal"> [23]PETSC ERROR: ------------------------------------------------------------------------<u></u><u></u></p>
<p class="MsoNormal"> [23]PETSC ERROR: Petsc Release Version 3.1.0, Patch 7, Mon Dec 20 14:26:37 CST 2010<u></u><u></u></p>
<p class="MsoNormal"> [23]PETSC ERROR: See docs/changes/index.html for recent updates.<u></u><u></u></p>
<p class="MsoNormal"> [23]PETSC ERROR: See docs/faq.html for hints about trouble shooting.<u></u><u></u></p>
<p class="MsoNormal"> [23]PETSC ERROR: See docs/in ...<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">My questions are (1) has anybody seen anything like this type of VecAssemblyBegin error? or (2) is it likely that splitting the VecSetValue in smaller blocks will help? or (4) is it likely that moving to mpich2 1.4p1 would help? (3) any
other thoughts?</p></div></div></blockquote><div><br></div><div>I would recommend interleaving your VecSetValues() with VecAssemblyBegin/End() calls. It certainly sounds like you are overflowing</div><div>buffers in the MPI implementation.</div>
<div><br></div><div> Matt</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;"><div lang="EN-CA" link="blue" vlink="purple"><div>
<p class="MsoNormal">Thanks in advance,<u></u><u></u></p>
<p class="MsoNormal">Rob <u></u><u></u></p>
</div>
</div>
</blockquote></div><br><br clear="all"><div><br></div>-- <br>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>
-- Norbert Wiener<br>