[petsc-dev] Fwd: [mumps-dev] support for distributed right-hand vectors?

[Alexander Grayver]

So, I have tested both PaStiX and MUMPS solvers. Tests were run on 4 
inifinibanded nodes, each equipped with two 12 core AMD Opteron and 64 
GB RAM. Intel Compiler 11.1 + MKL + OpenMPI was the tool-chain.

The problem is 3D Helmholtz equation, 1.4 Mio of unknowns. The matrix is 
symmetric thus I used LDL^T for both.
First of all, both PaStiX and MUMPS gave correct solution with the 
relative residual < 1e-12, although the test case was not numerically 
difficult.

Below are tables, showing time for analysis+factorization (seconds) and 
overall memory usage (megabytes).

PASTIX:
N_cpus    T_fac memory
1    9.27E+03    27900
4    5.28E+03    33200
16    1.44E+03    77700
32    755    131377
64    471    225399

MUMPS:
N_cpus    T_fac memory
1    8009    49689
4    2821    63501
16    1375    84115
32    1081    86583
64    733    98235

According to this test, PaStiX is slightly faster when run on more 
cores, but also consumes much more memory. Which is opposite to what 
Garth said. Either I did something wrong or our matrices are very 
different.

PS Can anyone explain why direct solvers require more memory when run in 
parallel?

On 10.11.2012 14:14, Alexander Grayver wrote:
> Garth,
>
> At the time I was tested PaStiX it failed for my problem:
> https://lists.mcs.anl.gov/mailman/htdig/petsc-dev/2011-December/006887.html
>
> Since then PaStiX has been updated with several critical bug fixes, so 
> I should consider testing new version.
>
> The memory scalability of the MUMPS is not nice, that is true.
> Running MUMPS with default parameters on large amount of cores is 
> often not optimal. I don't how much you spent tweaking parameters.
> MUMPS is among the most robust distributed solvers nowadays and it is 
> still being developed and hopefully will improve.
>
> /*To petsc developers:* /are there plans to update PaStiX supplied 
> with PETSc? The current version is 5.2 from 2012-06-08 and 
> PETSc-3.3-p3 uses 5.1.8 from 2011-02-23.
>
> Here is changelog:
> https://gforge.inria.fr/frs/shownotes.php?group_id=186&release_id=7096
>
-- 
Regards,
Alexander

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20121112/ea87ac12/attachment.html>