[petsc-users] Memory requirements in SUPERLU_DIST

[Barry Smith]

  This is something you better ask Sherri about. She’s the one who wrote and understands SuperLU_DIST


   Barry

On Sep 23, 2014, at 7:00 PM, Zin Lin <zinlin.zinlin at gmail.com> wrote:

> Hi 
> I am solving a frequency domain Maxwell problem for a dielectric structure of size 90x90x50, (the total matrix size is (90x90x50x6)^2 which includes the three vector components as well as real and imaginary parts.)
> I am using SUPERLU_DIST for the direct solver with the following options
> 
> parsymbfact = 1, (parallel symbolic factorization)
> permcol = PARMETIS, (parallel METIS)
> permrow = NATURAL (natural ordering).
> 
> First, I tried to use 4096 cores with 2GB / core memory which totals to about 8 TB of memory.
> I get the following error:
> 
> Using ParMETIS for parallel ordering.
> Structual symmetry is:100%
>    Current memory used:  1400271832 bytes
>    Maximum memory used:  1575752120 bytes
> ***Memory allocation failed for SetupCoarseGraph: adjncy. Requested size: 148242928 bytes
> 
> So it seems to be an insufficient memory allocation problem (which apparently happens at the METIS analysis phase?).
> 
> Then, I tried to use 64 large-memory cores which have a total of 2 TB memory (so larger memory per each core), it seems to work fine (though the solver takes about 900 sec ).
> What I don't understand is  why memory per core matters rather than the total memory? If the work space is distributed across the processors, shouldn't it work as long as I choose a sufficient number of smaller-memory cores? What kind of role does the memory per core play in the algorithm in contrast to the total memory over all the cores? 
> 
> The issue is I would rather use a large number of small-memory cores than any number of the large-memory cores. The latter are two times more expensive in terms of service units (I am running on STAMPEDE at TACC) and not many cores are available either.
> 
> Any idea would be appreciated.
> 
> Zin
> 
> -- 
> Zin Lin
>