[petsc-dev] Fwd: [petsc-users] Local size 7 not compatible with block size 13

Wed Jan 15 17:31:30 CST 2014

On Jan 15, 2014, at 4:44 PM, Matthew Knepley <knepley at gmail.com> wrote:

> On Wed, Jan 15, 2014 at 4:36 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
> 
>    This problem comes up for MPIAIJ matrices when the “off-diagona” matrix is compressed to only the  columns that have non zeros in them. When a MatDuplicate() is done on this it detects the inconsistency that the number of local columns may no longer be divisible by the block size. We could handle this by
> 
> 1) when the matrix is compressed (removing empty columns) we just mark the block size to 1 if the new number of local columns is not divisible by the block size
> 
> but then if the more values are put into the matrix when the matrix is uncompressed it no longer has the block size > 1 (possibly hurting efficiency) so we could
> 
> 2) stash the old block size in the matrix and put it back when we uncompress it.
> 
> What do you think? Should I do 1)? 2)?
> 
> 2) seems better, but I am worried about the complexity. I do not quite understand the data structures either. I thought
> B was just a MatSeqAIJ + column map. However, there is some compression/decompression code I guess. What is
> the compressed matrix,

    It is the same matrix, just the column indices are changed to shrink down on compress and go back up on non-compression.

> and where is the compressed flag?

   It doesn’t actually have a flag, the off-diagonal is always compressed on assembly. 

   I just looked again at the code and realized that 2) is actually already handled correctly. So I can fix 1) easily and will get to it soon.

  BTW: longer term issue is TS is keeping too many matrices around (memory hog).

    Barry

> 
>    Matt
>  
>    Barry
> 
> Request-assigned: Barry   fix MatDuplicate() problem on MPIAIJ matrices where block size of off diagonal matrix is no longer valid when compressed
> 
> Begin forwarded message:
> 
> > From: "Jay J. Billings" <billingsjj at ornl.gov>
> > Subject: [petsc-users] Local size 7 not compatible with block size 13
> > Date: January 15, 2014 at 1:14:02 PM CST
> > To: <petsc-users at mcs.anl.gov>
> >
> > Everyone,
> >
> > I'm trying to run PETSc in parallel and I'm getting an error I haven't seen before that the local and block size arguments don't match. Can anyone point me in the right direction on this? It happens right after I call TSSolve and, according to the backtrace, when TSSolve is trying to form the Jacobian. It doesn't look like it is actually calling my Jacobian routine, although I could be wrong about that.
> >
> > Jay
> >
> > [1]PETSC ERROR: --------------------- Error Message ------------------------------------
> > [1]PETSC ERROR: [0]PETSC ERROR: --------------------- Error Message ------------------------------------
> > [0]PETSC ERROR: Arguments are incompatible!
> > [1]PETSC ERROR: Arguments are incompatible!
> > [0]PETSC ERROR: Local size 7 not compatible with block size 13!
> > [1]PETSC ERROR: Local size 7 not compatible with block size 13!
> > [0]PETSC ERROR: ------------------------------------------------------------------------
> > [1]PETSC ERROR: Petsc Development GIT revision: v3.4.3-2305-g52cd893  GIT Date: 2014-01-13 09:37:41 -0500
> > [1]PETSC ERROR: See docs/changes/index.html for recent updates.
> > [1]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
> > [1]PETSC ERROR: See docs/index.html for manual pages.
> > [1]PETSC ERROR: ------------------------------------------------------------------------
> > [1]PETSC ERROR: ./xolotl on a arch-linux2-c-opt named box.ornl.gov by jay Wed Jan 15 14:12:23 2014
> > [1]PETSC ERROR: Libraries linked from /opt/petsc-latest_mpich-3.0.1/lib
> > [1]PETSC ERROR: Configure run at Mon Jan 13 11:11:18 2014
> > [1]PETSC ERROR: Configure options --prefix=/opt/petsc-latest_mpich-3.0.1 --with-cc=mpicc --with-cxx=mpicxx --with-fc=mpif77 --with-debugging=no --download-f-blas-lapack=1 --FOPTFLAGS= --with-shared-libraries=1 --download-hypre=yes --with-debugging=0
> > [1]PETSC ERROR: ------------------------------------------------------------------------
> > [1]PETSC ERROR: PetscLayoutSetBlockSize() line 464 in /home/jay/Programs/petsc/src/vec/is/utils/pmap.c
> > [1]PETSC ERROR: MatSetBlockSizes() line 6772 in /home/jay/Programs/petsc/src/mat/interface/matrix.c
> > [1]PETSC ERROR: MatDuplicate_SeqAIJ() line 4158 in /home/jay/Programs/petsc/src/mat/impls/aij/seq/aij.c
> > [1]PETSC ERROR: MatDuplicate() line 4116 in /home/jay/Programs/petsc/src/mat/interface/matrix.c
> > [1]PETSC ERROR: MatDuplicate_MPIAIJ() line 3395 in /home/jay/Programs/petsc/src/mat/impls/aij/mpi/mpiaij.c
> > [1]PETSC ERROR: MatDuplicate() line 4116 in /home/jay/Programs/petsc/src/mat/interface/matrix.c
> > [1]PETSC ERROR: TSGetRHSMats_Private() line 605 in /home/jay/Programs/petsc/src/ts/interface/ts.c
> > [1]PETSC ERROR: TSComputeIJacobian() line 782 in /home/jay/Programs/petsc/src/ts/interface/ts.c
> > [1]PETSC ERROR: SNESTSFormJacobian_ARKIMEX() line 1056 in /home/jay/Programs/petsc/src/ts/impls/arkimex/arkimex.c
> > [1]PETSC ERROR: SNESTSFormJacobian() line 3541 in /home/jay/Programs/petsc/src/ts/interface/ts.c
> > [1]PETSC ERROR: SNESComputeJacobian() line 2253 in /home/jay/Programs/petsc/src/snes/interface/snes.c
> > [1]PETSC ERROR: SNESSolve_NEWTONLS() line 231 in /home/jay/Programs/petsc/src/snes/impls/ls/ls.c
> > [1]PETSC ERROR: SNESSolve() line 3812 in /home/jay/Programs/petsc/src/snes/interface/snes.c
> > [1]PETSC ERROR: TSStep_ARKIMEX() line 776 in /home/jay/Programs/petsc/src/ts/impls/arkimex/arkimex.c
> > [1]PETSC ERROR: TSStep() line 2625 in /home/jay/Programs/petsc/src/ts/interface/ts.c
> > [1]PETSC ERROR: TSSolve() line 2741 in /home/jay/Programs/petsc/src/ts/interface/ts.c
> > [1]PETSC ERROR: checkPetscError() line 65 in /home/jay/research/xolotl/xolotl_workspace/xolotl-Source at xolotl/xolotlSolver/PetscSolver.cpp
> > ------------------------------------------------------------------------
> > [0]PETSC ERROR: Petsc Development GIT revision: v3.4.3-2305-g52cd893  GIT Date: 2014-01-13 09:37:41 -0500
> > [0]PETSC ERROR: See docs/changes/index.html for recent updates.
> > [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
> > [0]PETSC ERROR: See docs/index.html for manual pages.
> > [0]PETSC ERROR: ------------------------------------------------------------------------
> > [0]PETSC ERROR: ./xolotl on a arch-linux2-c-opt named box.ornl.gov by jay Wed Jan 15 14:12:23 2014
> > [0]PETSC ERROR: Libraries linked from /opt/petsc-latest_mpich-3.0.1/lib
> > [0]PETSC ERROR: Configure run at Mon Jan 13 11:11:18 2014
> > [0]PETSC ERROR: Configure options --prefix=/opt/petsc-latest_mpich-3.0.1 --with-cc=mpicc --with-cxx=mpicxx --with-fc=mpif77 --with-debugging=no --download-f-blas-lapack=1 --FOPTFLAGS= --with-shared-libraries=1 --download-hypre=yes --with-debugging=0
> > [0]PETSC ERROR: ------------------------------------------------------------------------
> > [0]PETSC ERROR: PetscLayoutSetBlockSize() line 464 in /home/jay/Programs/petsc/src/vec/is/utils/pmap.c
> > [0]PETSC ERROR: MatSetBlockSizes() line 6772 in /home/jay/Programs/petsc/src/mat/interface/matrix.c
> > [0]PETSC ERROR: MatDuplicate_SeqAIJ() line 4158 in /home/jay/Programs/petsc/src/mat/impls/aij/seq/aij.c
> > [0]PETSC ERROR: MatDuplicate() line 4116 in /home/jay/Programs/petsc/src/mat/interface/matrix.c
> > [0]PETSC ERROR: MatDuplicate_MPIAIJ() line 3395 in /home/jay/Programs/petsc/src/mat/impls/aij/mpi/mpiaij.c
> > [0]PETSC ERROR: MatDuplicate() line 4116 in /home/jay/Programs/petsc/src/mat/interface/matrix.c
> > [0]PETSC ERROR: TSGetRHSMats_Private() line 605 in /home/jay/Programs/petsc/src/ts/interface/ts.c
> > [0]PETSC ERROR: TSComputeIJacobian() line 782 in /home/jay/Programs/petsc/src/ts/interface/ts.c
> > [0]PETSC ERROR: SNESTSFormJacobian_ARKIMEX() line 1056 in /home/jay/Programs/petsc/src/ts/impls/arkimex/arkimex.c
> > [0]PETSC ERROR: SNESTSFormJacobian() line 3541 in /home/jay/Programs/petsc/src/ts/interface/ts.c
> > [0]PETSC ERROR: SNESComputeJacobian() line 2253 in /home/jay/Programs/petsc/src/snes/interface/snes.c
> > [0]PETSC ERROR: SNESSolve_NEWTONLS() line 231 in /home/jay/Programs/petsc/src/snes/impls/ls/ls.c
> > [0]PETSC ERROR: SNESSolve() line 3812 in /home/jay/Programs/petsc/src/snes/interface/snes.c
> > [0]PETSC ERROR: TSStep_ARKIMEX() line 776 in /home/jay/Programs/petsc/src/ts/impls/arkimex/arkimex.c
> > [0]PETSC ERROR: TSStep() line 2625 in /home/jay/Programs/petsc/src/ts/interface/ts.c
> > [0]PETSC ERROR: TSSolve() line 2741 in /home/jay/Programs/petsc/src/ts/interface/ts.c
> > [0]PETSC ERROR: checkPetscError() line 65 in /home/jay/research/xolotl/xolotl_workspace/xolotl-Source at xolotl/xolotlSolver/PetscSolver.cpp
> >
> > --
> > Jay Jay Billings
> > Oak Ridge National Laboratory
> > Twitter Handle: @jayjaybillings
> >
> 
> 
> 
> 
> -- 
> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
> -- Norbert Wiener