The code ierr = MatHeaderReplace(Y,&B);CHKERRQ(ierr); is suppose to handling freeing up that memory. So I don't think this is directly the problem. You'll need to run with valgrind for example to determine if there is lost memory and where. Is it possible that the matrices just require more memory than you have available?
Barry
> On Apr 26, 2016, at 2:25 PM, Antoine De Blois <antoine.deblois at aero.bombardier.com> wrote:
>
> Hi everyone,
>
> I am using petsc-3.6.2 and I get an Out-of-memory error, from what I believe to be a leak from petsc. I am doing successive calls to MatAXPY during an optimization loop. I use MatAXPY to blend 2 matrices together; a linear combination of the my operator and its first-order PC
> B = w*A + (1-w)*B
>
> The matrix operator and PC does not have the same non-zeros patterns, hence the DIFFERENT_NONZERO_PATTERN option. In fact, the non-zeros of the PC are a subset of the non-zeros of A, but I found out about the SUBSET_NONZERO_PATTERN too late…
>
> My matrices are pretty big (about 90 million unknown, with 33 non-zeros per row). So after the second call to MatAXPY, I already blow up the memory (on 256 cores). My code snippet would look like:
>
> PETSC_Mat_create(&( IMP_Mat_dR2dW) , 5, 0, 1 ,33,22);
> PETSC_Mat_create(&( IMP_Mat_dR1dW_PC), 5, 0, 1, 19,5); // notice the different non-zeros patterns
>
> while(1)
> {
> MatZeroEntries( IMP_Mat_dR2dW);
> MatZeroEntries( IMP_Mat_dR1dW_PC);
> // fill the dR2dW matrix ….
> // …
> ierr = MatSetValuesBlocked( IMP_Mat_dR2dW, nnz_W, dRdW_AD_blocked_cols, 1, &dRdW_AD_blocked_row, dRdW_AD_values, ADD_VALUES); // the dRdW_AD_values already contain the w factor
> // ….
> MatAssemblyBegin( IMP_Mat_dR2dW, MAT_FINAL_ASSEMBLY);
> MatAssemblyEnd( IMP_Mat_dR2dW, MAT_FINAL_ASSEMBLY);
>
> // fill the dR1dW_PC matrix ….
> // …
> ierr = MatSetValuesBlocked( IMP_Mat_dR1dW_PC, nnz_W, dRdW_AD_blocked_cols, 1, &dRdW_AD_blocked_row, dRdW_AD_values, ADD_VALUES);
> // …
> MatAssemblyBegin( IMP_Mat_dR1dW_PC, MAT_FINAL_ASSEMBLY);
> MatAssemblyEnd( IMP_Mat_dR1dW_PC, MAT_FINAL_ASSEMBLY);
>
> // blend the matrices
> // Out-of-memory appears here
> MatAXPY( IMP_Mat_dR1dW_PC,
> 1-w,
> IMP_Mat_dR2dW,
> SAME_NONZERO_PATTERN);
> }
> // KSPSetOperators
> // KSPsolve
> }
>
> I looked up the …/petsc-3.6.2/src/mat/impls/baij/mpi/mpibaij.c at line 2120. It appears that the temporary matrix B is allocated, but never freed. Note that I have not run Valgrind to confirm this leak.
> For now, I will make the non-zeros of the PC the same as the operator and use SAME_NONZERO_PATTERN.
>
> Thank you for your time and efforts,
> Antoine
>
> PETSC error below:
> [117]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
> [117]PETSC ERROR: Out of memory. This could be due to allocating
> [117]PETSC ERROR: too large an object or bleeding by not properly
> [117]PETSC ERROR: destroying unneeded objects.
> [117]PETSC ERROR: Memory allocated 0 Memory used by process 8423399424
> [117]PETSC ERROR: Try running with -malloc_dump or -malloc_log for info.
> [117]PETSC ERROR: Memory requested 704476168
> [117]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
> [117]PETSC ERROR: Petsc Release Version 3.6.2, Oct, 02, 2015
> [117]PETSC ERROR: /home/ad007804/CODES/FANSC_REPO_after_merge/fansc13.2.1_argus on a ARGUS_impi_opt named node1064 by ad007804 Tue Apr 26 09:34:52 2016
> [117]PETSC ERROR: Configure options --CFLAGS=-axAVX --CXXFLAGS="-std=c++11 -DMPICH_IGNORE_CXX_SEEK -DMPICH_IGNORE_CXX_SEEK" --FC=mpiifort --FFLAGS=-axAVX --download-hypre=/home/b0528796/libs/petsc_extpacks/hypre-2.10.1.tar.gz --FC=mpiifort --FFLAGS=-axAVX --download-metis=/home/b0528796/libs/petsc_extpacks/metis-5.1.0-p1.tar.gz --download-ml=/home/b0528796/libs/petsc_extpacks/ml-6.2-win.tar.gz --download-parmetis=/home/b0528796/libs/petsc_extpacks/parmetis-4.0.3-p1.tar.gz --download-suitesparse=/home/b0528796/libs/petsc_extpacks/SuiteSparse-4.4.3.tar.gz --download-superlu_dist --with-blas-lapack-lib=-mkl --with-cc=mpiicc --with-cxx=mpiicpc --with-debugging=no --download-mumps=/home/b0528796/libs/petsc_extpacks/MUMPS_5.0.0-p1.tar.gz --with-scalapack-lib="-L/gpfs/fs1/intel/mkl/lib/intel64 -lmkl_scalapack_lp64 -lmkl_blacs_intelmpi_lp64" --with-scalapack-include=/gpfs/fs1/intel/mkl/include
> [117]PETSC ERROR: #1 MatAXPY_BasicWithPreallocation() line 117 in /gpfs/fs2/aero/SOFTWARE/FLOW_SOLVERS/FANSC/EXT_LIB/petsc-3.6.2/src/mat/utils/axpy.c
> [117]PETSC ERROR: #2 MatAXPY_BasicWithPreallocation() line 117 in /gpfs/fs2/aero/SOFTWARE/FLOW_SOLVERS/FANSC/EXT_LIB/petsc-3.6.2/src/mat/utils/axpy.c
> [117]PETSC ERROR: #3 MatAXPY_MPIBAIJ() line 2120 in /gpfs/fs2/aero/SOFTWARE/FLOW_SOLVERS/FANSC/EXT_LIB/petsc-3.6.2/src/mat/impls/baij/mpi/mpibaij.c
> [117]PETSC ERROR: #4 MatAXPY() line 39 in /gpfs/fs2/aero/SOFTWARE/FLOW_SOLVERS/FANSC/EXT_LIB/petsc-3.6.2/src/mat/utils/axpy.c
>
> Antoine DeBlois
> Spécialiste ingénierie, MDO lead / Engineering Specialist, MDO lead
> Aéronautique / Aerospace
> 514-855-5001, x 50862
> antoine.deblois at aero.bombardier.com
>
> 2351 Blvd Alfred-Nobel
> Montreal, Qc
> H4S 1A9
>
> <image001.jpg>
> CONFIDENTIALITY NOTICE - This communication may contain privileged or confidential information.
> If you are not the intended recipient or received this communication by error, please notify the sender
> and delete the message without copying