[petsc-users] Error when switching from MPIAIJ to MPIBAIJ

[Henning Sauerland]

I'd like to get an idea of the solver performance when using mpibaij. Using my existing code based on mpiaij I simply swapped the preallocation routine and switched to mpibaij using the runtime flag. I basically perform the following function calls:


MatCreate(PETSC_COMM_WORLD, &SystMat); 
MatSetSizes(SystMat, PETSC_DECIDE, PETSC_DECIDE, nDOFg, nDOFg); 
MatSetFromOptions(SystMat);
MatMPIBAIJSetPreallocation(SystMat, 5, 30, PETSC_NULL,10, PETSC_NULL);

VecCreate(PETSC_COMM_WORLD, &Rhs);
VecSetSizes(Rhs, PETSC_DECIDE, nDOFg);
VecSetFromOptions(Rhs);

/* Filling SystMat and Rhs (currently using MatSetValues() ) */
...

VecDuplicate(*Rhs, &Sol);

KSPCreate(PETSC_COMM_WORLD, &ksp);
KSPSetOperators(ksp, *SystMat, *SystMat, DIFFERENT_NONZERO_PATTERN);
KSPSetFromOptions(ksp);
KSPSolve(ksp, *Rhs, Sol);


My runtime options are: -mat_type mpibaij -ksp_type lgmres -ksp_rtol 1.e-8 -ksp_gmres_restart 200 -pc_type asm -log_summary


This is what I get using 2 processors:

[0]PETSC ERROR: --------------------- Error Message ------------------------------------
[0]PETSC ERROR: Nonconforming object sizes!
[0]PETSC ERROR: Vector wrong size 34178 for scatter 34180 (scatter forward and vector from != ctx from size)!
[0]PETSC ERROR: ------------------------------------------------------------------------
[0]PETSC ERROR: Petsc Release Version 3.1.0, Patch 5, Mon Sep 27 11:51:54 CDT 2010
[0]PETSC ERROR: See docs/changes/index.html for recent updates.
[0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
[0]PETSC ERROR: See docs/index.html for manual pages.
[0]PETSC ERROR: ------------------------------------------------------------------------
[0]PETSC ERROR: ... on a ... Thu Jun  9 14:01:13 2011
[0]PETSC ERROR: Libraries linked from ...
[0]PETSC ERROR: Configure run at Wed Oct  6 11:03:30 2010
[0]PETSC ERROR: Configure options --with-cc=mpicc --download-scalapack=1 --with-cxx=mpicxx --download-mumps=1 --download-ml=1 --download-parmetis=1 --download-f-blas-lapack=1 PETSC_ARCH=linux-gnu-c-debug --download-blacs=1 --with-fc=mpif90 --download-hypre=1
[0]PETSC ERROR: ------------------------------------------------------------------------
[0]PETSC ERROR: VecScatterBegin() line 1549 in src/vec/vec/utils/vscat.c
[0]PETSC ERROR: PCApply_ASM() line 366 in src/ksp/pc/impls/asm/asm.c
[0]PETSC ERROR: PCApply() line 357 in src/ksp/pc/interface/precon.c
[0]PETSC ERROR: KSPInitialResidual() line 65 in src/ksp/ksp/interface/itres.c
[0]PETSC ERROR: KSPSolve_LGMRES() line 376 in src/ksp/ksp/impls/gmres/lgmres/lgmres.c
[0]PETSC ERROR: KSPSolve() line 396 in src/ksp/ksp/interface/itfunc.c
[1]PETSC ERROR: --------------------- Error Message ------------------------------------
[1]PETSC ERROR: Nonconforming object sizes!
[1]PETSC ERROR: Vector wrong size 34177 for scatter 34175 (scatter forward and vector from != ctx from size)!
[1]PETSC ERROR: ------------------------------------------------------------------------
[1]PETSC ERROR: Petsc Release Version 3.1.0, Patch 5, Mon Sep 27 11:51:54 CDT 2010
[1]PETSC ERROR: See docs/changes/index.html for recent updates.
[1]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
[1]PETSC ERROR: See docs/index.html for manual pages.
[1]PETSC ERROR: ------------------------------------------------------------------------
[1]PETSC ERROR: ... on a ... Thu Jun  9 14:01:13 2011
[1]PETSC ERROR: Libraries linked from ...
[1]PETSC ERROR: Configure run at Wed Oct  6 11:03:30 2010
[1]PETSC ERROR: Configure options --with-cc=mpicc --download-scalapack=1 --with-cxx=mpicxx --download-mumps=1 --download-ml=1 --download-parmetis=1 --download-f-blas-lapack=1 PETSC_ARCH=linux-gnu-c-debug --download-blacs=1 --with-fc=mpif90 --download-hypre=1
[1]PETSC ERROR: ------------------------------------------------------------------------
[1]PETSC ERROR: VecScatterBegin() line 1549 in src/vec/vec/utils/vscat.c
[1]PETSC ERROR: PCApply_ASM() line 366 in src/ksp/pc/impls/asm/asm.c
[1]PETSC ERROR: PCApply() line 357 in src/ksp/pc/interface/precon.c
[1]PETSC ERROR: KSPInitialResidual() line 65 in src/ksp/ksp/interface/itres.c
[1]PETSC ERROR: KSPSolve_LGMRES() line 376 in src/ksp/ksp/impls/gmres/lgmres/lgmres.c
[1]PETSC ERROR: KSPSolve() line 396 in src/ksp/ksp/interface/itfunc.c

Is there a basic mistake in my procedure?


Thanks
Henning