On Thu, Jun 9, 2011 at 7:57 AM, Henning Sauerland <span dir="ltr"><<a href="mailto:uerland@gmail.com">uerland@gmail.com</a>></span> wrote:<br><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">
I'd like to get an idea of the solver performance when using mpibaij. Using my existing code based on mpiaij I simply swapped the preallocation routine and switched to mpibaij using the runtime flag. I basically perform the following function calls:<br>
</blockquote><div><br></div><div>1) You can call both preallocation routines, but note that the block version takes smaller dnz/onz</div><div><br></div><div>2) If you want the matrix and vector to be divided the same way automatically, give them the same blocksize</div>
<div><br></div><div> Thanks,</div><div><br></div><div> Matt</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">
MatCreate(PETSC_COMM_WORLD, &SystMat);<br>
MatSetSizes(SystMat, PETSC_DECIDE, PETSC_DECIDE, nDOFg, nDOFg);<br>
MatSetFromOptions(SystMat);<br>
MatMPIBAIJSetPreallocation(SystMat, 5, 30, PETSC_NULL,10, PETSC_NULL);<br>
<br>
VecCreate(PETSC_COMM_WORLD, &Rhs);<br>
VecSetSizes(Rhs, PETSC_DECIDE, nDOFg);<br>
VecSetFromOptions(Rhs);<br>
<br>
/* Filling SystMat and Rhs (currently using MatSetValues() ) */<br>
...<br>
<br>
VecDuplicate(*Rhs, &Sol);<br>
<br>
KSPCreate(PETSC_COMM_WORLD, &ksp);<br>
KSPSetOperators(ksp, *SystMat, *SystMat, DIFFERENT_NONZERO_PATTERN);<br>
KSPSetFromOptions(ksp);<br>
KSPSolve(ksp, *Rhs, Sol);<br>
<br>
<br>
My runtime options are: -mat_type mpibaij -ksp_type lgmres -ksp_rtol 1.e-8 -ksp_gmres_restart 200 -pc_type asm -log_summary<br>
<br>
<br>
This is what I get using 2 processors:<br>
<br>
[0]PETSC ERROR: --------------------- Error Message ------------------------------------<br>
[0]PETSC ERROR: Nonconforming object sizes!<br>
[0]PETSC ERROR: Vector wrong size 34178 for scatter 34180 (scatter forward and vector from != ctx from size)!<br>
[0]PETSC ERROR: ------------------------------------------------------------------------<br>
[0]PETSC ERROR: Petsc Release Version 3.1.0, Patch 5, Mon Sep 27 11:51:54 CDT 2010<br>
[0]PETSC ERROR: See docs/changes/index.html for recent updates.<br>
[0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.<br>
[0]PETSC ERROR: See docs/index.html for manual pages.<br>
[0]PETSC ERROR: ------------------------------------------------------------------------<br>
[0]PETSC ERROR: ... on a ... Thu Jun 9 14:01:13 2011<br>
[0]PETSC ERROR: Libraries linked from ...<br>
[0]PETSC ERROR: Configure run at Wed Oct 6 11:03:30 2010<br>
[0]PETSC ERROR: Configure options --with-cc=mpicc --download-scalapack=1 --with-cxx=mpicxx --download-mumps=1 --download-ml=1 --download-parmetis=1 --download-f-blas-lapack=1 PETSC_ARCH=linux-gnu-c-debug --download-blacs=1 --with-fc=mpif90 --download-hypre=1<br>
[0]PETSC ERROR: ------------------------------------------------------------------------<br>
[0]PETSC ERROR: VecScatterBegin() line 1549 in src/vec/vec/utils/vscat.c<br>
[0]PETSC ERROR: PCApply_ASM() line 366 in src/ksp/pc/impls/asm/asm.c<br>
[0]PETSC ERROR: PCApply() line 357 in src/ksp/pc/interface/precon.c<br>
[0]PETSC ERROR: KSPInitialResidual() line 65 in src/ksp/ksp/interface/itres.c<br>
[0]PETSC ERROR: KSPSolve_LGMRES() line 376 in src/ksp/ksp/impls/gmres/lgmres/lgmres.c<br>
[0]PETSC ERROR: KSPSolve() line 396 in src/ksp/ksp/interface/itfunc.c<br>
[1]PETSC ERROR: --------------------- Error Message ------------------------------------<br>
[1]PETSC ERROR: Nonconforming object sizes!<br>
[1]PETSC ERROR: Vector wrong size 34177 for scatter 34175 (scatter forward and vector from != ctx from size)!<br>
[1]PETSC ERROR: ------------------------------------------------------------------------<br>
[1]PETSC ERROR: Petsc Release Version 3.1.0, Patch 5, Mon Sep 27 11:51:54 CDT 2010<br>
[1]PETSC ERROR: See docs/changes/index.html for recent updates.<br>
[1]PETSC ERROR: See docs/faq.html for hints about trouble shooting.<br>
[1]PETSC ERROR: See docs/index.html for manual pages.<br>
[1]PETSC ERROR: ------------------------------------------------------------------------<br>
[1]PETSC ERROR: ... on a ... Thu Jun 9 14:01:13 2011<br>
[1]PETSC ERROR: Libraries linked from ...<br>
[1]PETSC ERROR: Configure run at Wed Oct 6 11:03:30 2010<br>
[1]PETSC ERROR: Configure options --with-cc=mpicc --download-scalapack=1 --with-cxx=mpicxx --download-mumps=1 --download-ml=1 --download-parmetis=1 --download-f-blas-lapack=1 PETSC_ARCH=linux-gnu-c-debug --download-blacs=1 --with-fc=mpif90 --download-hypre=1<br>
[1]PETSC ERROR: ------------------------------------------------------------------------<br>
[1]PETSC ERROR: VecScatterBegin() line 1549 in src/vec/vec/utils/vscat.c<br>
[1]PETSC ERROR: PCApply_ASM() line 366 in src/ksp/pc/impls/asm/asm.c<br>
[1]PETSC ERROR: PCApply() line 357 in src/ksp/pc/interface/precon.c<br>
[1]PETSC ERROR: KSPInitialResidual() line 65 in src/ksp/ksp/interface/itres.c<br>
[1]PETSC ERROR: KSPSolve_LGMRES() line 376 in src/ksp/ksp/impls/gmres/lgmres/lgmres.c<br>
[1]PETSC ERROR: KSPSolve() line 396 in src/ksp/ksp/interface/itfunc.c<br>
<br>
Is there a basic mistake in my procedure?<br>
<br>
<br>
Thanks<br>
<font color="#888888">Henning</font></blockquote></div><br><br clear="all"><br>-- <br>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>
-- Norbert Wiener<br>