On Tue, Jun 29, 2010 at 19:35, Yujie <span dir="ltr"><<a href="mailto:recrusader@gmail.com">recrusader@gmail.com</a>></span> wrote:<br><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">
Dear PETSc developers,<br>
<br>
I want to output an about 36K*36K dense matrix using MatView in binary<br>
format. I use RedHat Enterprise 5 64bits system. However, when the<br>
file size of output matrix reaches about 2.7G, the codes pause and<br>
don't response for a long time (almost 3 hours). Could you help me<br>
figure out what happened? Thanks a lot.<br></blockquote><div><br></div><div>Were you writing this in serial or parallel? MPICH2 and Open MPI don't properly handle large message sizes fixes require ABI-incompatible changes that they don't want to push out in a minor release. I believe the latest versions of both will actually do the send, but MPI_Get_count does not return the correct value, and it probably would not be surprising if some MPI-IO functionality did not work correctly with large messages. Tickets that I'm familiar with:</div>
<div><br></div><div><a href="https://trac.mcs.anl.gov/projects/mpich2/ticket/1005">https://trac.mcs.anl.gov/projects/mpich2/ticket/1005</a></div><div><a href="https://svn.open-mpi.org/trac/ompi/ticket/2241">https://svn.open-mpi.org/trac/ompi/ticket/2241</a></div>
<div><br></div><div>I think they should both be fine for MPI-IO as long as each processor sends less than 2 GiB (even though the final output may be much bigger).</div><div><br></div><div>If this happens again, you could attach a debugger to the running process (gdb -pid XXX) and get a backtrace. Note that you can build "optimized" with debugging symbols at a very small runtime penalty.</div>
<div><br></div><div>Jed</div></div>