[petsc-users] mumps freezes for bigger problems

[Jack Poulson]

It looks like it's due to mixing different MPI implementations together
(i.e., including the wrong 'mpif.h'):
http://lists.mcs.anl.gov/pipermail/mpich-discuss/2010-July/007559.html

If I recall correctly, MUMPS only uses ScaLAPACK to factor the root
separator when it is sufficiently large, and that would explain why it
works for him for smaller problems. I would double check that ScaLAPACK,
PETSc, and MUMPS are all compiled with the same MPI implementation.

Jack

On Wed, Dec 21, 2011 at 4:55 PM, Hong Zhang <hzhang at mcs.anl.gov> wrote:

> Hailong:
> I've never seen this type of error from MUMPS.
> It seems programming bug. Are you sure smaller problem runs correctly?
> Use valgrind check it.
>
> Hong
>
> > I got the error from MUMPS.
> >
> > When I run MUMPS (which requring scalapack) with matrix size (n) = 30620,
> > nonzeros (nz) = 785860,
> > I could run it. And could get result.
> > But when I run it with
> > nz=3112820
> > n =61240
> >
> >
> > I am getting the following error
> >
> >
> > 17 - <NO ERROR MESSAGE> : Could not convert index 1140850688 into a
> pointer
> > The index may be an incorrect argument.
> > Possible sources of this problem are a missing "include 'mpif.h'",
> > a misspelled MPI object (e.g., MPI_COM_WORLD instead of MPI_COMM_WORLD)
> > or a misspelled user variable for an MPI object (e.g.,
> > com instead of comm).
> > [17] [] Aborting Program!
> >
> >
> >
> > Do you know what happened?
> > Is that possible it is running out of memory?
> >
> > On Wed, Dec 21, 2011 at 7:15 AM, Hong Zhang <hzhang at mcs.anl.gov> wrote:
> >>
> >> Direct solvers often require large memory for storing matrix factors.
> >> As Jed suggests, you may try superlu_dist.
> >>
> >> With mumps, I notice you use parallel analysis, which is relative new in
> >> mumps.
> >> What happens if you use default sequential analysis with
> >> different matrix orderings?
> >> I usually use matrix ordering '-mat_mumps_icntl_7 2'.
> >>
> >> Also, you can increase fill ratio,
> >> -mat_mumps_icntl_14 <20>: ICNTL(14): percentage of estimated workspace
> >> increase (None)
> >> i.e., default ration is 20, you may try 50? (I notice that you already
> use
> >> 30).
> >>
> >> It seems you use 16 CPUs for "a mere couple thousands
> >> elements" problems, and mumps "silently freezes". I do not have this
> type
> >> of experience with mumps. I usually can solve sparse matrix of size
> >> 10k with 1 cpu using mumps.
> >> When mumps runs out of memory or gets other problems, it terminates
> >> execution and dumps out error message,
> >> not freezes.
> >> Something is wrong here. Use a debugger and figuring out where it
> freezes.
> >>
> >> Hong
> >>
> >> On Wed, Dec 21, 2011 at 7:01 AM, Jed Brown <jedbrown at mcs.anl.gov>
> wrote:
> >> > -pc_type lu -pc_factor_mat_solver_package superlu_dist
> >> >
> >> > On Dec 21, 2011 6:19 AM, "Dominik Szczerba" <dominik at itis.ethz.ch>
> >> > wrote:
> >> >>
> >> >> I am successfully solving my indefinite systems with MUMPS but only
> >> >> for very small problems. To give a feeling, a mere couple thousands
> >> >> elements. If I only double the problem size, it silently freezes,
> even
> >> >> with max verbosity via the control parameters. Did anyone succeed
> here
> >> >> with big problems? Any recommendations for a drop-in replacement for
> >> >> MUMPS?
> >> >>
> >> >> Thanks for any hints,
> >> >> Dominik
> >> >>
> >> >>
> >> >>
> >> >> Options used:
> >> >> -mat_mumps_icntl_4 3 -mat_mumps_icntl_28 2 -mat_mumps_icntl_29
> >> >>
> >> >> Output:
> >> >>
> >> >> ****** FACTORIZATION STEP ********
> >> >>
> >> >>
> >> >>  GLOBAL STATISTICS PRIOR NUMERICAL FACTORIZATION ...
> >> >>  NUMBER OF WORKING PROCESSES              =          16
> >> >>  OUT-OF-CORE OPTION (ICNTL(22))           =           0
> >> >>  REAL SPACE FOR FACTORS                   =  1438970073
> >> >>  INTEGER SPACE FOR FACTORS                =    11376442
> >> >>  MAXIMUM FRONTAL SIZE (ESTIMATED)         =       16868
> >> >>  NUMBER OF NODES IN THE TREE              =       43676
> >> >>  Convergence error after scaling for ONE-NORM (option 7/8)   =
> 0.21D+01
> >> >>  Maximum effective relaxed size of S              =   231932340
> >> >>  Average effective relaxed size of S              =   182366303
> >> >>
> >> >>  REDISTRIB: TOTAL DATA LOCAL/SENT         =     1509215    22859750
> >> >>  GLOBAL TIME FOR MATRIX DISTRIBUTION       =      0.8270
> >> >>  ** Memory relaxation parameter ( ICNTL(14)  )            :        35
> >> >>  ** Rank of processor needing largest memory in facto     :         0
> >> >>  ** Space in MBYTES used by this processor for facto      :      2017
> >> >>  ** Avg. Space in MBYTES per working proc during facto    :      1618
> >
> >
> >
> >
> > --
> > Hailong
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20111223/f097f5e6/attachment-0001.htm>