[petsc-users] Memory requirements in SUPERLU_DIST

[Zin Lin]

Hi
I am solving a frequency domain Maxwell problem for a dielectric structure
of size 90x90x50, (the total matrix size is (90x90x50x6)^2 which includes
the three vector components as well as real and imaginary parts.)
I am using SUPERLU_DIST for the direct solver with the following options

parsymbfact = 1, (parallel symbolic factorization)
permcol = PARMETIS, (parallel METIS)
permrow = NATURAL (natural ordering).

First, I tried to use 4096 cores with 2GB / core memory which totals to
about 8 TB of memory.
I get the following error:

Using ParMETIS for parallel ordering.
Structual symmetry is:100%
   Current memory used:  1400271832 bytes
   Maximum memory used:  1575752120 bytes
***Memory allocation failed for SetupCoarseGraph: adjncy. Requested size:
148242928 bytes

So it seems to be an insufficient memory allocation problem (which
apparently happens at the METIS analysis phase?).

Then, I tried to use 64 large-memory cores which have a total of 2 TB
memory (so larger memory per each core), it seems to work fine (though the
solver takes about 900 sec ).
What I don't understand is  why memory per core matters rather than the
total memory? If the work space is distributed across the processors,
shouldn't it work as long as I choose a sufficient number of smaller-memory
cores? What kind of role does the memory per core play in the algorithm in
contrast to the total memory over all the cores?

The issue is I would rather use a large number of small-memory cores than
any number of the large-memory cores. The latter are two times more
expensive in terms of service units (I am running on STAMPEDE at TACC) and
not many cores are available either.

Any idea would be appreciated.

Zin

-- 
Zin Lin
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20140923/cfbaa172/attachment.html>