[petsc-users] mumps running out of memory, depending on an overall numerical factor?

[Klaus Zimmermann]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hi everyone,

Am 01.02.2014 20:17, schrieb Matthew Knepley:
> On Sat, Feb 1, 2014 at 1:16 PM, David Liu <daveliu at mit.edu 
> <mailto:daveliu at mit.edu>> wrote:
> 
> I see. Does that s^2 memory scaling mean that sparse direct
> solvers are not meant to be used beyond a certain point? I.e. if
> the supercomputer I'm using doesn't have enough memory per core to
> store even a single row of the factored matrix, then I'm out of
> luck?
> 
> 
> Yes exactly
Isn’t that a bit pessimistic? After all there is the out-of-core
facility with mumps.

In any case I believe the problem here is different:
As you rightly noted there will be some fill-in in the factorization.
It seems to be difficult to know how much beforehand. As I understand
it, mumps takes an educated guess. If the amount of fill-in is large
this just can be too small. If that happens you just should tell mumps
to expect more fill-in by the mentioned icntl(14). From the manual:
- ---
ICNTL(14) is accessed by the host both during the analysis and the
factorization phases. It corresponds
to the percentage increase in the estimated working space. When
significant extra fill-in is caused
by numerical pivoting, increasing ICNTL(14) may help. Except in
special cases, the default value
is 20 (which corresponds to a 20 % increase).
- ---

There is an alternative: Using icntl(23) you can provide mumps with
the maximum amount of memory in mb that it can use per processor. On
supercomputers you usually have a good idea of the available memory.
Subtract from that the footprint of your program right before the call
to mumps and you should have an optimal value of memory for mumps to
work with.
If the memory then turns out to be insufficient, at least mumps will
notice prior to the factorization and safe your precious cpu hours.

I use this approach with an additional command line parameter to feed
the total available memory to the program. Actually I thought about
integrating this directly in petsc, but never got around to doing it.
What do you think: Would this find its way into petsc? Or is it to
mumps specific?

Kind regards
Klaus

> 
> Matt
> 
> 
> On Fri, Jan 31, 2014 at 9:51 PM, Jed Brown <jed at jedbrown.org 
> <mailto:jed at jedbrown.org>> wrote:
> 
> David Liu <daveliu at mit.edu <mailto:daveliu at mit.edu>> writes:
> 
>> Hi, I'm solving a 3d problem with mumps. When I increased the
> grid size to
>> 70x60x20 with 6 unknowns per point, I started noticing that
> the program was
>> crashing at runtime at the factoring stage, with the mumps
> error code:
>> 
>> -17 The internal send buffer that was allocated dynamically by
> MUMPS on the
>> processor is too small. The user should increase the value of
>> ICNTL(14) before calling
> MUMPS again.
>> 
>> However, when I increase the grid spacing in the z direction
> by about 50%,
>> this crash does not happen.
>> 
>> Why would how much memory an LU factorization uses depend on
> an overall
>> numerical factor (for part of the matrix at least) like this?
> 
> I'm not sure exactly what you're asking, but the complexity of 
> direct solves depend on the minimal vertex separators in the
> sparse matrix/graph.  Yours will be s=60*20*6 (more if your stencil
> needs second neighbors).  The memory usage scales with s^2 and the 
> factorization time scales with s^3.
> 
> 
> 
> 
> 
> -- What most experimenters take for granted before they begin
> their experiments is infinitely more interesting than any results
> to which their experiments lead. -- Norbert Wiener

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.14 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQEcBAEBAgAGBQJS7UyKAAoJEHGtO1Agqpgxx0sIAIEVY81Sh6jyJC7ScSVuC+P7
2iaCKOmVwDb5yD6QaM88AGIdIiOHA5yFehU3IYBM2GhbWkvgiT3ylEgfjh5akp5R
MNbn0FFmD4ycq83CBIg1TsiQj6V9fX+7Y6ROAABGdLlk0qNolN0apRKFfn/H7FnT
YqExKSgoBp+ctt5/XsK2BdyHxiopTPjSx6J0xd+KjHhpbE97KxxURbm4vsPv+clj
7PrRpXbJnannMvHYQVfmVmmurw/yC47KeX8aqQTGWYrRZUfOnQ8K4Z3uzT5XzQ1l
TW6lruaFPu+y1r8oJAA8+Fp1pf06Jjx85fDb9VelRwaIoNIS1B41KEmByHpsf3w=
=nXDH
-----END PGP SIGNATURE-----