Running the simple code through valgrind yields the only non-python related error below:<div><br></div><div><div>==23697== Invalid read of size 4</div><div>==23697== at 0x6592465: PMPI_Attr_get (in /usr/local/lib/libmpi.so.0.0.2)</div>
<div>==23697== by 0x8BED7C8: PetscCommDuplicate (tagm.c:145)</div><div>==23697== by 0x8BF2EF6: PetscHeaderCreate_Private (inherit.c:51)</div><div>==23697== by 0x958F7C9: KSPCreate (itcreate.c:564)</div><div>==23697== by 0x93DAC39: PCMGSetLevels (mg.c:219)</div>
<div>==23697== by 0x93BC73B: pcmgsetlevels_ (zmgf.c:15)</div><div>==23697== by 0x7A5CE1C: mgtest_ (mgtest.F:15)</div><div>==23697== Address 0x8c is not stack'd, malloc'd or (recently) free'd</div><div>==23697== </div>
<div>[0]PETSC ERROR: ------------------------------------------------------------------------</div><div>[0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range</div><div>[0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger</div>
<div>[0]PETSC ERROR: or see <a href="http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[0]PETSC">http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[0]PETSC</a> ERROR: or try <a href="http://valgrind.org">http://valgrind.org</a> on GNU/linux and Apple Mac OS X to find memory corruption errors</div>
<div>[0]PETSC ERROR: likely location of problem given in stack below</div><div>[0]PETSC ERROR: --------------------- Stack Frames ------------------------------------</div><div>[0]PETSC ERROR: Note: The EXACT line numbers in the stack are not available,</div>
<div>[0]PETSC ERROR: INSTEAD the line number of the start of the function</div><div>[0]PETSC ERROR: is given.</div><div>[0]PETSC ERROR: [0] PetscCommDuplicate line 144 /nfs/mica/home/kenway/Downloads/petsc-3.3/src/sys/objects/tagm.c</div>
<div>[0]PETSC ERROR: [0] PetscHeaderCreate_Private line 30 /nfs/mica/home/kenway/Downloads/petsc-3.3/src/sys/objects/inherit.c</div><div>[0]PETSC ERROR: [0] KSPCreate line 557 /nfs/mica/home/kenway/Downloads/petsc-3.3/src/ksp/ksp/interface/itcreate.c</div>
<div>[0]PETSC ERROR: [0] PCMGSetLevels line 180 /nfs/mica/home/kenway/Downloads/petsc-3.3/src/ksp/pc/impls/mg/mg.c</div><div><br></div><div><br></div><div>Gaetan</div><br><div class="gmail_quote">On Fri, Aug 31, 2012 at 11:03 PM, Gaetan Kenway <span dir="ltr"><<a href="mailto:kenway@utias.utoronto.ca" target="_blank">kenway@utias.utoronto.ca</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hi again<div><br></div><div>I distilled the problem down further to a simple subroutine in mgtest.F, copied directly from ex8f.F</div>
<div><br></div><div><div> subroutine MGTest()</div><div> implicit none</div><div>
<br></div><div>#include "include/finclude/petsc.h"</div><div><br></div><div> KSP ksp</div><div> PC pc</div><div> PetscErrorCode ierr</div><div> PetscInt nlevels, two</div><div> two = 2</div>
<div class="im">
<div><br></div><div> call KSPCreate(PETSC_COMM_WORLD,ksp,ierr)</div><div> call KSPGetPC(ksp,pc,ierr)</div><div> call PCSetType(pc,PCMG,ierr)</div></div><div> call PCMGSetLevels(pc,two,PETSC_NULL_OBJECT,ierr)</div>
<div><br></div><div> call KSPDestroy(ksp)</div><div><br></div><div> end subroutine MGTest</div><div><br></div><div>When I ran this simple case with gdb I got the following backtrace:</div></div><div><br></div><div>
<div>#0 0xb694d465 in PMPI_Attr_get () from /usr/local/lib/libmpi.so.0</div><div>#1 0xb555d7c9 in PetscCommDuplicate (comm_in=0x0, comm_out=0x88d7378, first_tag=0x88d73a0) at /nfs/mica/home/kenway/Downloads/petsc-3.3/src/sys/objects/tagm.c:145</div>
<div>#2 0xb5562ef7 in PetscHeaderCreate_Private (h=0x88d7370, classid=1211235, type=-1, class_name=0xb61cfd70 "KSP", descr=0xb61cfd74 "Krylov Method", mansec=0xb61cfd70 "KSP", comm=0x0, </div>
<div> des=0xb5eeac70 <KSPDestroy>, vie=0xb5efafd7 <KSPView>) at /nfs/mica/home/kenway/Downloads/petsc-3.3/src/sys/objects/inherit.c:51</div><div>#3 0xb5eff7ca in KSPCreate (comm=0x0, inksp=0x88d6ee0) at /nfs/mica/home/kenway/Downloads/petsc-3.3/src/ksp/ksp/interface/itcreate.c:564</div>
<div>#4 0xb5d4ac3a in PCMGSetLevels (pc=0x89089a0, levels=2, comms=0xb4bbf5a0) at /nfs/mica/home/kenway/Downloads/petsc-3.3/src/ksp/pc/impls/mg/mg.c:219</div><div>#5 0xb5d2c73c in pcmgsetlevels_ (pc=0xbfffec98, levels=0xbfffec8c, comms=0xb4bbf5a0, ierr=0xbfffec94) at /nfs/mica/home/kenway/Downloads/petsc-3.3/src/ksp/pc/impls/mg/ftn-custom/zmgf.c:15</div>
<div>#6 0xb4affe1d in mgtest () at mgtest.F:15</div><div>#7 0xb4aad769 in f2py_rout_warp_mgtest (capi_self=0x8618758, capi_args=0xb7c3002c, capi_keywds=0x0, f2py_func=0xb4affdc0 <mgtest>) at warpmodule.c:4273</div>
<div>#8 0xb4aa58cf in fortran_call (fp=0x8618758, arg=0xb7c3002c, kw=0x0) at /usr/lib/python2.6/dist-packages/numpy/f2py/src/fortranobject.c:318</div><div>#9 0x0805fd6a in PyObject_Call ()</div><div>#10 0x080dd5b0 in PyEval_EvalFrameEx ()</div>
<div>#11 0x080dfbb2 in PyEval_EvalCodeEx ()</div><div>#12 0x080dfca7 in PyEval_EvalCode ()</div><div>#13 0x080fd956 in PyRun_FileExFlags ()</div><div>#14 0x080fdbb2 in PyRun_SimpleFileExFlags ()</div><div>#15 0x0805b6d3 in Py_Main ()</div>
<div>#16 0x0805a8ab in main ()</div></div><div><br></div><div><div>The only difference is that I'm calling the routine from python and petsc is initialized from petsc4py. </div><div><br></div><div>Any ideas?</div><div>
<br></div><div>Thanks, </div><span class="HOEnZb"><font color="#888888"><div>Gaetan</div></font></span><div><div class="h5"><div><br></div><br><div class="gmail_quote">On Fri, Aug 31, 2012 at 9:48 PM, Jed Brown <span dir="ltr"><<a href="mailto:jedbrown@mcs.anl.gov" target="_blank">jedbrown@mcs.anl.gov</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">src/ksp/pc/examples/tests/ex8f.F is a simple example, but I made the modification below and it runs fine.<div><div><br>
</div><div><div>diff --git a/src/ksp/pc/examples/tests/ex8f.F b/src/ksp/pc/examples/tests/ex8f.F</div><div>
--- a/src/ksp/pc/examples/tests/ex8f.F</div><div>+++ b/src/ksp/pc/examples/tests/ex8f.F</div><div>@@ -35,7 +35,7 @@</div><div> Vec x,b,u</div><div> PC pc</div><div> PetscInt n,dim,istart,iend</div>
<div>- PetscInt i,j,jj,ii,one,zero</div><div>+ PetscInt i,j,jj,ii,one,two,zero</div><div> PetscErrorCode ierr</div><div> PetscScalar v</div><div> external MyResidual</div><div>@@ -50,6 +50,7 @@</div>
<div> pfive = .5d0</div><div> n = 6</div><div> dim = n*n</div><div>+ two = 2</div><div> one = 1</div><div> zero = 0</div><div> </div><div>@@ -138,7 +139,7 @@</div><div> call KSPCreate(PETSC_COMM_WORLD,ksp,ierr)</div>
<div> call KSPGetPC(ksp,pc,ierr)</div><div> call PCSetType(pc,PCMG,ierr)</div><div>- call PCMGSetLevels(pc,one,PETSC_NULL_OBJECT,ierr)</div><div>+ call PCMGSetLevels(pc,two,PETSC_NULL_OBJECT,ierr)</div>
<div> call PCMGSetResidual(pc,zero,PCMGDefaultResidual,A,ierr)</div><div> </div><div> call PCMGSetResidual(pc,zero,MyResidual,A,ierr)</div><div>@@ -147,6 +148,7 @@</div><div> ! also serves as the preconditioning matrix.</div>
<div> </div><div> call KSPSetOperators(ksp,A,A,DIFFERENT_NONZERO_PATTERN,ierr)</div><div>+ call KSPView(ksp,PETSC_VIEWER_STDOUT_WORLD,ierr)</div><div> </div><div> </div><div> call KSPDestroy(ksp,ierr)</div>
<div><br><div>$ mpirun.hydra -n 2 ./ex8f</div><div>KSP Object: 2 MPI processes</div><div> type: gmres</div><div> GMRES: restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement</div>
<div> GMRES: happy breakdown tolerance 1e-30</div><div> maximum iterations=10000, initial guess is zero</div><div><div> tolerances: relative=1e-05, absolute=1e-50, divergence=10000</div><div> left preconditioning</div>
</div><div>
using DEFAULT norm type for convergence test</div><div><div>PC Object: 2 MPI processes</div><div> type: mg</div><div> MG: type is MULTIPLICATIVE, levels=2 cycles=v</div><div> Cycles per PCApply=1</div>
</div><div> Not using Galerkin computed coarse grid matrices</div><div>
<div> Coarse grid solver -- level -------------------------------</div><div> KSP Object: (mg_coarse_) 2 MPI processes</div><div> type: preonly</div><div> maximum iterations=1, initial guess is zero</div>
<div> tolerances: relative=1e-05, absolute=1e-50, divergence=10000</div><div> left preconditioning</div><div> using NONE norm type for convergence test</div></div><div> PC Object: (mg_coarse_) 2 MPI processes</div>
<div> type: redundant</div><div> Redundant preconditioner: Not yet setup</div><div><div> Down solver (pre-smoother) on level 1 -------------------------------</div><div> KSP Object: (mg_levels_1_) 2 MPI processes</div>
<div> type: chebyshev</div><div> Chebyshev: eigenvalue estimates: min = 0, max = 0</div><div> maximum iterations=2, initial guess is zero</div><div> tolerances: relative=1e-05, absolute=1e-50, divergence=10000</div>
<div> left preconditioning</div><div> using NONE norm type for convergence test</div><div> PC Object: (mg_levels_1_) 2 MPI processes</div><div> type: sor</div><div> SOR: type = local_symmetric, iterations = 1, local iterations = 1, omega = 1</div>
<div> Up solver (post-smoother) same as down solver (pre-smoother)</div></div><div> linear system matrix = precond matrix:</div><div> Matrix Object: 2 MPI processes</div><div> type: mpiaij</div><div> rows=36, cols=36</div>
<div> total: nonzeros=156, allocated nonzeros=252</div><div><div> total number of mallocs used during MatSetValues calls =0</div></div><div> not using I-node (on process 0) routines</div><div><br></div>
<br></div><br><div class="gmail_quote"><div><div>
On Fri, Aug 31, 2012 at 7:27 PM, Gaetan Kenway <span dir="ltr"><<a href="mailto:kenway@utias.utoronto.ca" target="_blank">kenway@utias.utoronto.ca</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Hi<div><br></div><div>I'm having an issue with the PCMG preconditioner in PETSc. I'm trying to use PCMG to precondition a linear system of equations resulting from the discretization of the EUler equations. It is from a multi-block code so I have the ability to easily generate restriction operators using geometric method. </div>
<div><br></div><div>I'm currently trying to setup a simple 2V cycle. For this I use:</div><div><br></div><div><div> ! Create the restriction operator between finest level and one below</div><div> call createRestrictOperator(RL1, 1)</div>
<div> </div><div> call PCSetFromOptions(pc, PETScIerr)</div><div> nlevels=2</div><div> !call PCMGSetLevels(pc, nlevels, PETSC_NULL_OBJECT,PETScIerr) ! Doesn't work from fortran!!!</div><div> call PCMGSetGalerkin(pc, PETSC_TRUE, PETScIerr)</div>
<div> call PCMGSetRestriction(pc, 1, RL1, PETScierr)</div><div><br></div><div> call PCView(pc, PETScierr)</div></div><div><br></div><div>I run with the command line argument -pc_mg_levels 2 .</div><div><br></div><div>First issue: PCMGSetLevels does not work correctly from Fortran. The traceback I get when I run (with the commented-out line above and without the command line arguments) is:</div>
<div><br></div><div><div>[0]PETSC ERROR: ------------------------------------------------------------------------</div><div>[0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range</div>
<div>[0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger</div><div>[0]PETSC ERROR: or see <a href="http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[0]PETSC" target="_blank">http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[0]PETSC</a> ERROR: or try <a href="http://valgrind.org" target="_blank">http://valgrind.org</a> on GNU/linux and Apple Mac OS X to find memory corruption errors</div>
<div>[0]PETSC ERROR: likely location of problem given in stack below</div><div>[0]PETSC ERROR: --------------------- Stack Frames ------------------------------------</div><div>[0]PETSC ERROR: Note: The EXACT line numbers in the stack are not available,</div>
<div>[0]PETSC ERROR: INSTEAD the line number of the start of the function</div><div>[0]PETSC ERROR: is given.</div><div>[0]PETSC ERROR: [0] PetscCommDuplicate line 144 /nfs/mica/home/kenway/Downloads/petsc-3.3/src/sys/objects/tagm.c</div>
<div>[0]PETSC ERROR: [0] PetscHeaderCreate_Private line 30 /nfs/mica/home/kenway/Downloads/petsc-3.3/src/sys/objects/inherit.c</div><div>[0]PETSC ERROR: [0] KSPCreate line 557 /nfs/mica/home/kenway/Downloads/petsc-3.3/src/ksp/ksp/interface/itcreate.c</div>
<div>[0]PETSC ERROR: [0] PCMGSetLevels line 180 /nfs/mica/home/kenway/Downloads/petsc-3.3/src/ksp/pc/impls/mg/mg.c</div></div><div><br></div><div>I can get around that issue by using -pc_mg_levels 2 and not calling the PCMGSetLevels() command, but I would like to actually use this command and not have set it via manually setting an option. The result of the PCView is listed below which appears to be what I expect (given the default options)</div>
<div><br></div><div><div>PC Object: 1 MPI processes</div><div> type: mg</div><div> MG: type is MULTIPLICATIVE, levels=2 cycles=v</div><div> Cycles per PCApply=1</div><div> Using Galerkin computed coarse grid matrices</div>
<div> Coarse grid solver -- level -------------------------------</div><div> KSP Object: (mg_coarse_) 1 MPI processes</div><div> type: preonly</div><div> maximum iterations=1, initial guess is zero</div>
<div> tolerances: relative=1e-05, absolute=1e-50, divergence=10000</div><div> left preconditioning</div><div> using NONE norm type for convergence test</div><div> PC Object: (mg_coarse_) 1 MPI processes</div>
<div> type: lu</div><div> LU: out-of-place factorization</div><div> tolerance for zero pivot 2.22045e-14</div><div> matrix ordering: nd</div><div> Down solver (pre-smoother) on level 1 -------------------------------</div>
<div> KSP Object: (mg_levels_1_) 1 MPI processes</div><div> type: chebyshev</div><div> Chebyshev: eigenvalue estimates: min = 0, max = 0</div><div> maximum iterations=2, initial guess is zero</div>
<div> tolerances: relative=1e-05, absolute=1e-50, divergence=10000</div><div> left preconditioning</div><div> using NONE norm type for convergence test</div><div> PC Object: (mg_levels_1_) 1 MPI processes</div>
<div> type: sor</div><div> SOR: type = local_symmetric, iterations = 1, local iterations = 1, omega = 1</div><div> Up solver (post-smoother) same as down solver (pre-smoother)</div><div> linear system matrix followed by preconditioner matrix:</div>
<div> Matrix Object: 1 MPI processes</div><div> type: seqbaij</div><div> rows=92160, cols=92160, bs=5</div><div> total: nonzeros=5582400, allocated nonzeros=5582400</div><div> total number of mallocs used during MatSetValues calls =0</div>
<div> block size is 5</div><div> Matrix Object: 1 MPI processes</div><div> type: seqbaij</div><div> rows=92160, cols=92160, bs=5</div><div> total: nonzeros=3089600, allocated nonzeros=3089600</div><div>
total number of mallocs used during MatSetValues calls =0</div><div> block size is 5</div></div><div><br></div><div><br></div><div>So that's ok. When it gets to actually solving my system with KSPSolve() I get a returned error code 73 which is "object in argument is in wrong state, e.g. unassembled mat"</div>
</blockquote><div><br></div></div></div><div>In general, please don't paraphrase error messages.</div><div><br></div><div>With these problems, try running in a debugger to get a real trace and with valgrind. The most common problem is stack corruption due to passing the wrong number or types of arguments.</div>
<div>
<div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div><br></div><div>and the traceback is:</div><div><div>[0]PETSC ERROR: Note: The EXACT line numbers in the stack are not available,</div><div>[0]PETSC ERROR: INSTEAD the line number of the start of the function</div>
<div>[0]PETSC ERROR: is given.</div><div>[0]PETSC ERROR: [0] MatGetVecs line 8142 /nfs/mica/home/kenway/Downloads/petsc-3.3/src/mat/interface/matrix.c</div><div>[0]PETSC ERROR: [0] KSPGetVecs line 774 /nfs/mica/home/kenway/Downloads/petsc-3.3/src/ksp/ksp/interface/iterativ.c</div>
<div>[0]PETSC ERROR: [0] PCSetUp_MG line 508 /nfs/mica/home/kenway/Downloads/petsc-3.3/src/ksp/pc/impls/mg/mg.c</div><div>[0]PETSC ERROR: [0] PCSetUp line 810 /nfs/mica/home/kenway/Downloads/petsc-3.3/src/ksp/pc/interface/precon.c</div>
<div>[0]PETSC ERROR: [0] KSPSetUp line 182 /nfs/mica/home/kenway/Downloads/petsc-3.3/src/ksp/ksp/interface/itfunc.c</div><div>[0]PETSC ERROR: [0] KSPSolve line 351 /nfs/mica/home/kenway/Downloads/petsc-3.3/src/ksp/ksp/interface/itfunc.c</div>
</div><div><br></div><div>Which would indicate to me its an issue with one of the matrices involved, either the linear operator, A, the preconditioner P or my provided restriction matrix RL1. A and P i'm pretty sure are OK, since it works just fine when I use my usual BlockJacobi/Additive Schwartz with ILU(k) on the subdomains. Which leaves the Restriction operator as the issue. I added the following code before PCView()</div>
<div><br></div><div> call MatGetVecs(RL1, vec1, vec2, ierr)</div><div> call VecSet(vec1, 1.0, ierr)</div><div> call MatMult(RL1, vec1, vec2, ierr)</div><div><br></div><div>which ran through just fine (I check each of the ierrs but those are not shown). </div>
<div><br></div><div>So I'm at a loss as to what is causing the error. </div><div>I'm using an up-to-date copy of petsc-3.3 from the repository. </div><div><br></div><div>Thanks,</div><div><br></div><div>Gaetan</div>
</blockquote></div></div><br></div></div>
</blockquote></div><br></div></div></div>
</blockquote></div><br></div>