[petsc-users] MatView for huge matrix output

[Barry Smith]

  Until the MPI implementors fix this problem the only way you can work with such large matrices is to use more processors. Make sure that no process has more than 250,000,000 entries in the matrix. So for your 36k by 36k matrix you need to use at least 5 processors, maybe six just try them.

   Barry


On Jun 29, 2010, at 9:48 PM, Yujie wrote:

> Thank you very much! I am writing it in parallel mode. I am using
> MPICH not MPICH2. I don't know the mechanism in MatView. If I write in
> paralle, there is no communication between nodes?
> 
> Regards,
> Yujie
> 
> On Tue, Jun 29, 2010 at 9:36 PM, Jed Brown <jed at 59a2.org> wrote:
>> On Tue, Jun 29, 2010 at 19:35, Yujie <recrusader at gmail.com> wrote:
>>> 
>>> Dear PETSc developers,
>>> 
>>> I want to output an about 36K*36K dense matrix using MatView in binary
>>> format. I use RedHat Enterprise 5 64bits system. However, when the
>>> file size of output matrix reaches about 2.7G, the codes pause and
>>> don't response for a long time (almost 3 hours). Could you help me
>>> figure out what happened? Thanks a lot.
>> 
>> Were you writing this in serial or parallel?  MPICH2 and Open MPI don't
>> properly handle large message sizes fixes require ABI-incompatible changes
>> that they don't want to push out in a minor release.  I believe the latest
>> versions of both will actually do the send, but MPI_Get_count does not
>> return the correct value, and it probably would not be surprising if some
>> MPI-IO functionality did not work correctly with large messages.  Tickets
>> that I'm familiar with:
>> https://trac.mcs.anl.gov/projects/mpich2/ticket/1005
>> https://svn.open-mpi.org/trac/ompi/ticket/2241
>> I think they should both be fine for MPI-IO as long as each processor sends
>> less than 2 GiB (even though the final output may be much bigger).
>> If this happens again, you could attach a debugger to the running process
>> (gdb -pid XXX) and get a backtrace.  Note that you can build "optimized"
>> with debugging symbols at a very small runtime penalty.
>> Jed