[petsc-users] mumps running out of memory, depending on an overall numerical factor?

[Matthew Knepley]

On Sat, Feb 1, 2014 at 1:35 PM, Klaus Zimmermann <
klaus.zimmermann at physik.uni-freiburg.de> wrote:

> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> Hi everyone,
>
> Am 01.02.2014 20:17, schrieb Matthew Knepley:
> > On Sat, Feb 1, 2014 at 1:16 PM, David Liu <daveliu at mit.edu
> > <mailto:daveliu at mit.edu>> wrote:
> >
> > I see. Does that s^2 memory scaling mean that sparse direct
> > solvers are not meant to be used beyond a certain point? I.e. if
> > the supercomputer I'm using doesn't have enough memory per core to
> > store even a single row of the factored matrix, then I'm out of
> > luck?
> >
> >
> > Yes exactly
> Isn't that a bit pessimistic? After all there is the out-of-core
> facility with mumps.
>

The question was "sparse direct solvers are not meant to be used beyond a
certain point"?
The answer is definitely yes.

   Matt


> In any case I believe the problem here is different:
> As you rightly noted there will be some fill-in in the factorization.
> It seems to be difficult to know how much beforehand. As I understand
> it, mumps takes an educated guess. If the amount of fill-in is large
> this just can be too small. If that happens you just should tell mumps
> to expect more fill-in by the mentioned icntl(14). From the manual:
> - ---
> ICNTL(14) is accessed by the host both during the analysis and the
> factorization phases. It corresponds
> to the percentage increase in the estimated working space. When
> significant extra fill-in is caused
> by numerical pivoting, increasing ICNTL(14) may help. Except in
> special cases, the default value
> is 20 (which corresponds to a 20 % increase).
> - ---
>
> There is an alternative: Using icntl(23) you can provide mumps with
> the maximum amount of memory in mb that it can use per processor. On
> supercomputers you usually have a good idea of the available memory.
> Subtract from that the footprint of your program right before the call
> to mumps and you should have an optimal value of memory for mumps to
> work with.
> If the memory then turns out to be insufficient, at least mumps will
> notice prior to the factorization and safe your precious cpu hours.
>
> I use this approach with an additional command line parameter to feed
> the total available memory to the program. Actually I thought about
> integrating this directly in petsc, but never got around to doing it.
> What do you think: Would this find its way into petsc? Or is it to
> mumps specific?
>
> Kind regards
> Klaus
>
> >
> > Matt
> >
> >
> > On Fri, Jan 31, 2014 at 9:51 PM, Jed Brown <jed at jedbrown.org
> > <mailto:jed at jedbrown.org>> wrote:
> >
> > David Liu <daveliu at mit.edu <mailto:daveliu at mit.edu>> writes:
> >
> >> Hi, I'm solving a 3d problem with mumps. When I increased the
> > grid size to
> >> 70x60x20 with 6 unknowns per point, I started noticing that
> > the program was
> >> crashing at runtime at the factoring stage, with the mumps
> > error code:
> >>
> >> -17 The internal send buffer that was allocated dynamically by
> > MUMPS on the
> >> processor is too small. The user should increase the value of
> >> ICNTL(14) before calling
> > MUMPS again.
> >>
> >> However, when I increase the grid spacing in the z direction
> > by about 50%,
> >> this crash does not happen.
> >>
> >> Why would how much memory an LU factorization uses depend on
> > an overall
> >> numerical factor (for part of the matrix at least) like this?
> >
> > I'm not sure exactly what you're asking, but the complexity of
> > direct solves depend on the minimal vertex separators in the
> > sparse matrix/graph.  Yours will be s=60*20*6 (more if your stencil
> > needs second neighbors).  The memory usage scales with s^2 and the
> > factorization time scales with s^3.
> >
> >
> >
> >
> >
> > -- What most experimenters take for granted before they begin
> > their experiments is infinitely more interesting than any results
> > to which their experiments lead. -- Norbert Wiener
>
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.14 (GNU/Linux)
> Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
>
> iQEcBAEBAgAGBQJS7UyKAAoJEHGtO1Agqpgxx0sIAIEVY81Sh6jyJC7ScSVuC+P7
> 2iaCKOmVwDb5yD6QaM88AGIdIiOHA5yFehU3IYBM2GhbWkvgiT3ylEgfjh5akp5R
> MNbn0FFmD4ycq83CBIg1TsiQj6V9fX+7Y6ROAABGdLlk0qNolN0apRKFfn/H7FnT
> YqExKSgoBp+ctt5/XsK2BdyHxiopTPjSx6J0xd+KjHhpbE97KxxURbm4vsPv+clj
> 7PrRpXbJnannMvHYQVfmVmmurw/yC47KeX8aqQTGWYrRZUfOnQ8K4Z3uzT5XzQ1l
> TW6lruaFPu+y1r8oJAA8+Fp1pf06Jjx85fDb9VelRwaIoNIS1B41KEmByHpsf3w=
> =nXDH
> -----END PGP SIGNATURE-----
>



-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20140201/11f69220/attachment.html>