[petsc-users] Setting up MUMPS in PETSc

[Jinquan Zhong]

Dear folks,

I have a question on how to use mumps properly in PETSc.  It appears that I didn’t set up mumps right.  I followed the example in
http://www.mcs.anl.gov/petsc/petsc-dev/src/mat/examples/tests/ex125.c.html

to set up my program.  Here is my situation on using the default setting


        PetscInt<http://www.mcs.anl.gov/petsc/petsc-dev/docs/manualpages/Sys/PetscInt.html#PetscInt> icntl_7 = 5;
        MatMumpsSetIcntl<http://www.mcs.anl.gov/petsc/petsc-dev/docs/manualpages/Mat/MatMumpsSetIcntl.html#MatMumpsSetIcntl>(F,7,icntl_7);

in the example ex125.c:


1.       The program work fine on all small models (sparse matrices at  the order of m= 894, 1097, 31k with a dense matrix included in the sparse matrix). The residuals are at the magnitude of 10^-3.



2.       The program has some issues on medium size problem  (m=460k with a dense matrix at the order of n=30k included in the sparse matrix).  The full sparse matrix is sized at 17GB.

a.       We used another software to generate sparse matrix by using 144 cores:

                                                               i.      When I used the resource from 144 cores (12 nodes with 48GB/node), it could not provide the solution.  There was a complain on the memory violation.

                                                             ii.      When I used the resource from 432 cores (36 nodes with 48GB/node), it provided the solution.

b.      We used another software to generate the same sparse matrix by using 576 cores:

                                                               i.      When I used the resource from 576 cores (48 nodes with 48GB/node), it could not provide the solution.  There was a complain on the memory violation.

                                                             ii.      When I used the resource from 1152 cores (96 nodes with 48GB/node), it provided the solution.



3.       The program could not solve the large size problem (m=640k with a dense matrix at the order of n=178k included in the sparse matrix).  The full sparse matrix is sized at 511GB.

a.       We used another software to generate sparse matrix by using 900 cores:

                                                               i.      When I used the resource from 900 cores (75 nodes with 48GB/node), it could not provide the solution.  There was a complain on the memory violation.

                                                             ii.      When I used the resource from 2400 cores (200 nodes with 48GB/node), it STILL COULD NOT provide the solution.

My confusion starts from the medium size problem:

·         It seems something was not right in the default setting in ex125.c for these problems.

·         I got the info that METIS was used instead of ParMETIS in solving these problems.

·         Furthermore, it appears that there was unreasonable demand on the solver even on the medium size problem.

·         I suspect one rank was trying to collect all data from other ranks.

What other addition setting is needed for mumps such that it could deal with medium and large size problems?

Do you guys have similar experience on that?

Thanks,

Jinquan

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20121023/b7f87d1e/attachment-0001.html>