[petsc-users] MatView for huge matrix output

[Yujie]

Thank you very much! I am writing it in parallel mode. I am using
MPICH not MPICH2. I don't know the mechanism in MatView. If I write in
paralle, there is no communication between nodes?

Regards,
Yujie

On Tue, Jun 29, 2010 at 9:36 PM, Jed Brown <jed at 59a2.org> wrote:
> On Tue, Jun 29, 2010 at 19:35, Yujie <recrusader at gmail.com> wrote:
>>
>> Dear PETSc developers,
>>
>> I want to output an about 36K*36K dense matrix using MatView in binary
>> format. I use RedHat Enterprise 5 64bits system. However, when the
>> file size of output matrix reaches about 2.7G, the codes pause and
>> don't response for a long time (almost 3 hours). Could you help me
>> figure out what happened? Thanks a lot.
>
> Were you writing this in serial or parallel?  MPICH2 and Open MPI don't
> properly handle large message sizes fixes require ABI-incompatible changes
> that they don't want to push out in a minor release.  I believe the latest
> versions of both will actually do the send, but MPI_Get_count does not
> return the correct value, and it probably would not be surprising if some
> MPI-IO functionality did not work correctly with large messages.  Tickets
> that I'm familiar with:
> https://trac.mcs.anl.gov/projects/mpich2/ticket/1005
> https://svn.open-mpi.org/trac/ompi/ticket/2241
> I think they should both be fine for MPI-IO as long as each processor sends
> less than 2 GiB (even though the final output may be much bigger).
> If this happens again, you could attach a debugger to the running process
> (gdb -pid XXX) and get a backtrace.  Note that you can build "optimized"
> with debugging symbols at a very small runtime penalty.
> Jed