[petsc-users] DMDA3D with custom preallocation leads to MAT_NEW_NONZERO_ALLOCATION_ERR when using pc_gamg
Fabian.Jakub
Fabian.Jakub at physik.uni-muenchen.de
Tue Jul 8 10:51:17 CDT 2014
Thank you very much for the swift reply!
I am running
git describe: v3.5-25-g0ace994
and configured with
./configure \
--with-cc=$(which mpicc) \
--with-fc=$(which mpif90) \
--with-cxx=$(which mpic++) \
--with-fortran \
--with-shared-libraries \
--with-hdf5 \
--with-hdf5-dir="$HDF5_DIR" \
--with-cmake=$(which cmake) \
\
I run the model with:
mpirun -np 2 -map-by socket bin/petsc_solver -pc_type gamg
-pc_gamg_agg_nsmooths 1 -pc_gamg_verbose 1 -info
Should I append the output when creating the DMDA?
For the KSP setup it says:
[0] VecScatterCreateCommon_PtoS(): Using blocksize 1 scatter
[0] VecScatterCreate(): General case: MPI to Seq
[1] MatAssemblyEnd_SeqAIJ(): Matrix size: 6359040 X 131520; storage
space: 0 unneeded,420864 used
[1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0
[1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 8
[1] MatCheckCompressedRow(): Found the ratio (num_zerorows
6227520)/(num_localrows 6359040) > 0.6. Use CompressedRow routines.
[0] MatAssemblyEnd_SeqAIJ(): Matrix size: 6359040 X 131520; storage
space: 0 unneeded,420864 used
[0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0
[0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 8
[0] MatCheckCompressedRow(): Found the ratio (num_zerorows
6227520)/(num_localrows 6359040) > 0.6. Use CompressedRow routines.
[1] PetscCommDuplicate(): Using internal PETSc communicator
47323925895584 15967792
Setup KSP
[0] PetscCommDuplicate(): Using internal PETSc communicator
47292596238752 42057616
[0] PCSetUp(): Setting up PC for first time [0]PCSetUp_GAMG level 0
N=12718080, n data rows=10, n data cols=10, nnz/row (ave)=10, np=2
[1] PetscCommDuplicate(): Using internal PETSc communicator
47323925894432 16179040
[0] PetscCommDuplicate(): Using internal PETSc communicator
47292596237600 41749680
[1] PetscCommDuplicate(): Using internal PETSc communicator
47323925894432 16179040
[0] PetscCommDuplicate(): Using internal PETSc communicator
47292596237600 41749680
[1]PETSC ERROR: --------------------- Error Message
--------------------------------------------------------------
[1]PETSC ERROR: Argument out of range
[1]PETSC ERROR: New nonzero at (4704,4609) caused a malloc
Use MatSetOption(A, MAT_NEW_NONZERO_ALLOCATION_ERR, PETSC_FALSE) to turn
off this check
[1]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html
for trouble shooting.
[1]PETSC ERROR: Petsc Development GIT revision: v3.5-25-g0ace994 GIT
Date: 2014-07-07 14:50:15 +0200
[1]PETSC ERROR: bin/petsc_solver on a debug named lx001 by jakub Tue
Jul 8 17:47:31 2014
[1]PETSC ERROR: Configure options
--with-cc=/home/opt/cosmo_tica_lib/ompi1.8.1/openmpi-1.8.1/install/bin/mpicc
--with-fc=/home/opt/cosmo_tica_lib/ompi1.8.1/openmpi-1.8.1/install/bin/mpif90
--with-cxx=/home/opt/cosmo_tica_lib/ompi1.8.1/openmpi-1.8.1/install/bin/mpic++
--with-fortran --with-shared-libraries --with-hdf5
--with-hdf5-dir=/home/opt/cosmo_tica_lib//ompi1.8.1/hdf5/HDF_Group/HDF5/1.8.13/
--with-cmake=/usr/bin/cmake
[1]PETSC ERROR: #1 MatSetValues_MPIAIJ() line 530 in
/home/opt/cosmo_tica_lib/ompi1.8.1/petsc-master/src/mat/impls/aij/mpi/mpiaij.c
[1]PETSC ERROR: #2 MatSetValues() line 1136 in
/home/opt/cosmo_tica_lib/ompi1.8.1/petsc-master/src/mat/interface/matrix.c
[1]PETSC ERROR: #3 PCGAMGCreateGraph() line 72 in
/home/opt/cosmo_tica_lib/ompi1.8.1/petsc-master/src/ksp/pc/impls/gamg/tools.c
[1]PETSC ERROR: #4 PCGAMGgraph_AGG() line 936 in
/home/opt/cosmo_tica_lib/ompi1.8.1/petsc-master/src/ksp/pc/impls/gamg/agg.c
[1]PETSC ERROR: #5 PCSetUp_GAMG() line 595 in
/home/opt/cosmo_tica_lib/ompi1.8.1/petsc-master/src/ksp/pc/impls/gamg/gamg.c
[1]PETSC ERROR: #6 PCSetUp() line 902 in
/home/opt/cosmo_tica_lib/ompi1.8.1/petsc-master/src/ksp/pc/interface/precon.c
[1]PETSC ERROR: #7 KSPSetUp() line 305 in
/home/opt/cosmo_tica_lib/ompi1.8.1/petsc-master/src/ksp/ksp/interface/itfunc.c
[0]PETSC ERROR: --------------------- Error Message
--------------------------------------------------------------
[0]PETSC ERROR: Argument out of range
[0]PETSC ERROR: New nonzero at (4704,4609) caused a malloc
Use MatSetOption(A, MAT_NEW_NONZERO_ALLOCATION_ERR, PETSC_FALSE) to turn
off this check
[0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html
for trouble shooting.
[0]PETSC ERROR: Petsc Development GIT revision: v3.5-25-g0ace994 GIT
Date: 2014-07-07 14:50:15 +0200
[0]PETSC ERROR: bin/petsc_solver on a debug named lx001 by jakub Tue
Jul 8 17:47:31 2014
[0]PETSC ERROR: Configure options
--with-cc=/home/opt/cosmo_tica_lib/ompi1.8.1/openmpi-1.8.1/install/bin/mpicc
--with-fc=/home/opt/cosmo_tica_lib/ompi1.8.1/openmpi-1.8.1/install/bin/mpif90
--with-cxx=/home/opt/cosmo_tica_lib/ompi1.8.1/openmpi-1.8.1/install/bin/mpic++
--with-fortran --with-shared-libraries --with-hdf5
--with-hdf5-dir=/home/opt/cosmo_tica_lib//ompi1.8.1/hdf5/HDF_Group/HDF5/1.8.13/
--with-cmake=/usr/bin/cmake
[0]PETSC ERROR: #1 MatSetValues_MPIAIJ() line 530 in
/home/opt/cosmo_tica_lib/ompi1.8.1/petsc-master/src/mat/impls/aij/mpi/mpiaij.c
[0]PETSC ERROR: #2 MatSetValues() line 1136 in
/home/opt/cosmo_tica_lib/ompi1.8.1/petsc-master/src/mat/interface/matrix.c
[0]PETSC ERROR: #3 PCGAMGCreateGraph() line 72 in
/home/opt/cosmo_tica_lib/ompi1.8.1/petsc-master/src/ksp/pc/impls/gamg/tools.c
[0]PETSC ERROR: #4 PCGAMGgraph_AGG() line 936 in
/home/opt/cosmo_tica_lib/ompi1.8.1/petsc-master/src/ksp/pc/impls/gamg/agg.c
[0]PETSC ERROR: #5 PCSetUp_GAMG() line 595 in
/home/opt/cosmo_tica_lib/ompi1.8.1/petsc-master/src/ksp/pc/impls/gamg/gamg.c
[0]PETSC ERROR: #6 PCSetUp() line 902 in
/home/opt/cosmo_tica_lib/ompi1.8.1/petsc-master/src/ksp/pc/interface/precon.c
[0]PETSC ERROR: #7 KSPSetUp() line 305 in
/home/opt/cosmo_tica_lib/ompi1.8.1/petsc-master/src/ksp/ksp/interface/itfunc.c
--------------------------------------------------------------------------
MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_WORLD
with errorcode 63.
NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.
You may or may not see output from other processes, depending on
exactly when Open MPI kills them.
-------------------------------------------------------------------------
Am 08.07.2014 17:14, schrieb Barry Smith:
> GAMG should not give a preallocation error regardless of the preallocation of the original matrix; so presumably you’ve hit on an error in GAMG. What version of PETSc are you using and what is the ENTIRE error message from GAMG? If you are using PETSc 3.4 please upgrade to 3.5 and see if GAMG still produces an error.
>
> Barry
>
>
>
> On Jul 8, 2014, at 7:39 AM, Fabian.Jakub <Fabian.Jakub at physik.uni-muenchen.de> wrote:
>
>> Hi,
>> I have a question regarding gamg where I get a wrong preallocation i.e. a MAT_NEW_NONZERO_ALLOCATION_ERR **.
>>
>> I use a 3d DMDA with 10 dof but the coupling only ever needs 2 dof on a star stencil.
>>
>> The code to setup the matrix is like this:
>>
>> call DMSetMatrixPreallocateOnly(C%da, PETSC_TRUE,ierr)
>> call DMCreateMatrix(C%da, A, ierr)
>> call MatSetFromOptions(A,ierr)
>>
>> call MatMPIAIJSetPreallocation(A, PETSC_NULL_INTEGER,d_nnz, PETSC_NULL_INTEGER, o_nnz, ierr)
>>
>> insert matrix values & assemble
>>
>> then solve.
>>
>> If I solve the system with any ''normal'' KSP/PC there is no problem and matrix info actually confirms that the preallocation is good.
>> However if I use gamg I get an allocation error when it tries to create a coarse grid.
>>
>> If I use the DMDA preallocation, it works but uses way more memory...
>>
>> Is there a possibility to use custom preallocation and at the same time let gamg create the coarse grid?
>> Is that even the problem or am I missing something?
>>
>> Thank you so very much.
>>
>> Sincerely,
>> Fabian
>>
>>
>>
>>
>>
>>
>>
More information about the petsc-users
mailing list