[petsc-users] Triple increasing of allocated memory during KSPSolve calling(GMRES preconditioned by ASM)

Tue Feb 4 10:04:45 CST 2020

   Please run with the option -ksp_view so we know the exact solver options you are using. 

   From the lines 

MatCreateSubMats       1 1.0 1.9397e-01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
MatGetOrdering         1 1.0 1.1066e-02 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
MatIncreaseOvrlp       1 1.0 3.0324e-02 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0

   and the fact you have three matrices I would guess you are using the additive Schwarz preconditioner (ASM) with ILU(0) on the blocks. (which converges the same as ILU on one process but does use much more memory).

   Note: your code is still built with 32 bit integers. 

   I would guess the basic matrix formed plus the vectors in this example could take ~200 MB. It is the two matrices in the additive Schwarz that is taking the additional memory.

   What kind of PDEs are you solving and what kind of formulation?  

   ASM plus ILU is the "work mans" type preconditioner, relatively robust but not particularly fast for convergence. Depending on your problem you might be able to do much better convergence wise by using a PCFIELDSPLIT and a PCGAMG on one of the splits. In your own run you see the ILU is chugging along rather slowly to the solution. 

   With your current solvers you can use the option -sub_pc_factor_in_place which will shave off one of the matrices memories. Please try that.

   Avoiding the ASM you can avoid both extra matrices but at the cost of even slower convergence. Use, for example -pc_type sor 

    The petroleum industry also has a variety of "custom" preconditioners/solvers for particular models and formulations that can beat the convergence of general purpose solvers; and require less memory. Some of these can be implemented or simulated with PETSc. Some of these are implemented in the commercial petroleum simulation codes and it can be difficult to get a handle on exactly what they do because of proprietary issues. I think I have an old text on these approaches in my office, there may be modern books that discuss these.

   Barry

> On Feb 4, 2020, at 6:04 AM, Дмитрий Мельничук <dmitry.melnichuk at geosteertech.com> wrote:
> 
> Hello again!
> Thank you very much for your replies!
> Log is attached.
>  
> 1. The main problem now is following. To solve the matrix that is attached to my previous e-mail PETSc consumes ~550 MB.
> I know definitely that there are commercial softwares in petroleum industry (e.g., Schlumberger Petrel) that solve the same initial problem consuming only ~200 MB.
> Moreover, I am sure that when I used 32-bit PETSc (GMRES + ASM) a year ago, it also consumed ~200 MB for this matrix.
>  
> So, my question is: do you have any advice on how to decrease the amount of RAM consumed for such matrix from 550 MB to 200 MB? Maybe some specific preconditioner or other ways?
>  
> I will be very grateful for any thoughts!
>  
> 2. The second problem is more particular.
> According to resource manager in Windows 10, Fortran solver based on PETSc consumes 548 MB RAM  while solving the system of linear equations.
> As I understand it form logs, it is required 459 MB and 52 MB for matrix and vector storage respectively. After summing of all objects for which memory is  allocated we get only 517 MB.
>  
> Thank you again for your time! Have a nice day.
>  
> Kind regards,
> Dmitry
> 
> 
> 03.02.2020, 19:55, "Smith, Barry F." <bsmith at mcs.anl.gov>:
> 
>    GMRES also can by default require about 35 work vectors if it reaches the full restart. You can set a smaller restart with -ksp_gmres_restart 15 for example but this can also hurt the convergence of GMRES dramatically. People sometimes use the KSPBCGS algorithm since it does not require all the restart vectors but it can also converge more slowly.
> 
>     Depending on how much memory the sparse matrices use relative to the vectors the vector memory may matter or not.
> 
>    If you are using a recent version of PETSc you can run with -log_view -log_view_memory and it will show on the right side of the columns how much memory is being allocated for each of the operations in various ways. 
> 
>    Barry
> 
> 
> 
>  On Feb 3, 2020, at 10:34 AM, Matthew Knepley <knepley at gmail.com> wrote:
> 
>  On Mon, Feb 3, 2020 at 10:38 AM Дмитрий Мельничук <dmitry.melnichuk at geosteertech.com> wrote:
>  Hello all!
> 
>  Now I am faced with a problem associated with the memory allocation when calling of KSPSolve .
> 
>  GMRES preconditioned by ASM for solving linear algebra system (obtained by the finite element spatial discretisation of Biot poroelasticity model) was chosen.
>  According to the output value of PetscMallocGetCurrentUsage subroutine 176 MB for matrix and RHS vector storage is required (before KSPSolve calling).
>  But during solving linear algebra system 543 MB of RAM is required (during KSPSolve calling).
>  Thus, the amount of allocated memory after preconditioning stage increased three times. This kind of behaviour is critically for 3D models with several millions of cells. 
> 
>  1) In order to know anything, we have to see the output of -ksp_view, although I see you used an overlap of 2
> 
>  2) The overlap increases the size of submatrices beyond that of the original matrix. Suppose that you used LU for the sub-preconditioner.
>      You would need at least 2x memory (with ILU(0)) since the matrix dominates memory usage. Moreover, you have overlap
>      and you might have fill-in depending on the solver.
> 
>  3) The massif tool from valgrind is a good fine-grained way to look at memory allocation
> 
>    Thanks,
> 
>       Matt
> 
>  Is there a way to decrease amout of allocated memory?
>  Is that an expected behaviour for GMRES-ASM combination?
> 
>  As I remember, using previous version of PETSc didn't demonstrate so significante memory increasing. 
> 
>  ...
>  Vec :: Vec_F, Vec_U
>  Mat :: Mat_K
>  ...
>  ...
>  call MatAssemblyBegin(Mat_M,Mat_Final_Assembly,ierr)
>  call MatAssemblyEnd(Mat_M,Mat_Final_Assembly,ierr)
>  ....
>  call VecAssemblyBegin(Vec_F_mod,ierr)
>  call VecAssemblyEnd(Vec_F_mod,ierr)
>  ...
>  ...
>  call PetscMallocGetCurrentUsage(mem, ierr)
>  print *,"Memory used: ",mem
>  ...
>  ...
>  call KSPSetType(Krylov,KSPGMRES,ierr)
>  call KSPGetPC(Krylov,PreCon,ierr)
>  call PCSetType(PreCon,PCASM,ierr)
>  call KSPSetFromOptions(Krylov,ierr)
>  ...
>  call KSPSolve(Krylov,Vec_F,Vec_U,ierr)
>  ...
>  ...
>  options = "-pc_asm_overlap 2 -pc_asm_type basic -ksp_monitor -ksp_converged_reason"
> 
> 
>  Kind regards,
>  Dmitry Melnichuk
>  Matrix.dat (265288024)
> 
> 
>  -- 
>  What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
>  -- Norbert Wiener
> 
>  https://www.cse.buffalo.edu/~knepley/
> 
> 
> <Logs_26K_GMRES-ASM-log_view-log_view_memory-malloc_dump_32bit>