This is already bad. You had an Invalid Read and Invalid Write in your Hypre. Did you build it<br>yourself? If so, let us build it. If not, please try your matrix on KSP ex10 and see if you get a<br>crash on 2 procs.<br><br>
Thanks,<br><br> Matt<br><br><div class="gmail_quote">On Sat, Nov 14, 2009 at 3:51 PM, Dominik Szczerba <span dir="ltr"><<a href="mailto:dominik@itis.ethz.ch">dominik@itis.ethz.ch</a>></span> wrote:<br><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
run onlu in single, he says things like below - but does not crash. Also, the program run with -np 1 does not crash. No clear idea though about valgrind's output, please advise if this tells you anything...<br>
<br>
Call from NS3T10::createSolverContexts() referenced therein is:<br>
<br>
ierr = KSPCreate(petsc_comm,&kspSchurVelocity);CHKERRQ(ierr);<br>
<br>
<br>
==2605== Conditional jump or move depends on uninitialised value(s)<br>
==2605== at 0x8AE720F: hypre_BoomerAMGSetPlotFileName (par_amg.c:2115)<br>
==2605== by 0x8AE7ED9: hypre_BoomerAMGCreate (par_amg.c:276)<br>
==2605== by 0x8AE4A71: HYPRE_BoomerAMGCreate (HYPRE_parcsr_amg.c:31)<br>
==2605== by 0x8562019: PCHYPRESetType_HYPRE (hypre.c:850)<br>
==2605== by 0x8563068: PCHYPRESetType (hypre.c:964)<br>
==2605== by 0x80E67BB: NS3T10::createSolverContexts()<br>
(NS3T10mpi.cxx:1980)<br>
==2605== by 0x80EA63B: NS3T10::solve() (NS3T10mpi.cxx:2306)<br>
==2605== by 0x8104860: main (ns3t10mpi_main.cxx:1516)<br>
==2605==<br>
==2605== Conditional jump or move depends on uninitialised value(s)<br>
==2605== at 0x8AE7244: hypre_BoomerAMGSetPlotFileName (par_amg.c:2120)<br>
==2605== by 0x8AE7ED9: hypre_BoomerAMGCreate (par_amg.c:276)<br>
==2605== by 0x8AE4A71: HYPRE_BoomerAMGCreate (HYPRE_parcsr_amg.c:31)<br>
==2605== by 0x8562019: PCHYPRESetType_HYPRE (hypre.c:850)<br>
==2605== by 0x8563068: PCHYPRESetType (hypre.c:964)<br>
==2605== by 0x80E67BB: NS3T10::createSolverContexts()<br>
(NS3T10mpi.cxx:1980)<br>
==2605== by 0x80EA63B: NS3T10::solve() (NS3T10mpi.cxx:2306)<br>
==2605== by 0x8104860: main (ns3t10mpi_main.cxx:1516)<br>
==2605==<br>
==2605== Conditional jump or move depends on uninitialised value(s)<br>
==2605== at 0x4025C16: strcpy (mc_replace_strmem.c:303)<br>
==2605== by 0x8AE727A: hypre_BoomerAMGSetPlotFileName (par_amg.c:2123)<br>
==2605== by 0x8AE7ED9: hypre_BoomerAMGCreate (par_amg.c:276)<br>
==2605== by 0x8AE4A71: HYPRE_BoomerAMGCreate (HYPRE_parcsr_amg.c:31)<br>
==2605== by 0x8562019: PCHYPRESetType_HYPRE (hypre.c:850)<br>
==2605== by 0x8563068: PCHYPRESetType (hypre.c:964)<br>
==2605== by 0x80E67BB: NS3T10::createSolverContexts()<br>
(NS3T10mpi.cxx:1980)<br>
==2605== by 0x80EA63B: NS3T10::solve() (NS3T10mpi.cxx:2306)<br>
==2605== by 0x8104860: main (ns3t10mpi_main.cxx:1516)<br>
==2605==<br>
==2605== Conditional jump or move depends on uninitialised value(s)<br>
==2605== at 0x4025C35: strcpy (mc_replace_strmem.c:303)<br>
==2605== by 0x8AE727A: hypre_BoomerAMGSetPlotFileName (par_amg.c:2123)<br>
==2605== by 0x8AE7ED9: hypre_BoomerAMGCreate (par_amg.c:276)<br>
==2605== by 0x8AE4A71: HYPRE_BoomerAMGCreate (HYPRE_parcsr_amg.c:31)<br>
==2605== by 0x8562019: PCHYPRESetType_HYPRE (hypre.c:850)<br>
==2605== by 0x8563068: PCHYPRESetType (hypre.c:964)<br>
==2605== by 0x80E67BB: NS3T10::createSolverContexts()<br>
(NS3T10mpi.cxx:1980)<br>
==2605== by 0x80EA63B: NS3T10::solve() (NS3T10mpi.cxx:2306)<br>
==2605== by 0x8104860: main (ns3t10mpi_main.cxx:1516)<br>
==2605==<br>
Solver contexts created in 2.520000 s<br>
Starting KSPSolve (0/1)<br>
0 KSP Residual norm 8.368803253774e-06<br>
==2605== Invalid read of size 8<br>
==2605== at 0x8B23B5A: hypre_BoomerAMGCreateS (par_strength.c:223)<br>
==2605== by 0x8AE966F: hypre_BoomerAMGSetup (par_amg_setup.c:630)<br>
==2605== by 0x8AE4A4D: HYPRE_BoomerAMGSetup (HYPRE_parcsr_amg.c:58)<br>
==2605== by 0x855A5D9: PCSetUp_HYPRE (hypre.c:134)<br>
==2605== by 0x86256A9: PCSetUp (precon.c:794)<br>
==2605== by 0x85A6E62: KSPSetUp (itfunc.c:237)<br>
==2605== by 0x85A7EAB: KSPSolve (itfunc.c:353)<br>
==2605== by 0x80F5B16: applyPrecSchur(void*, _p_Vec*, _p_Vec*)<br>
(NS3T10mpi.cxx:3741)<br>
==2605== by 0x851C47E: PCApply_Shell (shellpc.c:129)<br>
==2605== by 0x862074E: PCApply (precon.c:357)<br>
==2605== by 0x863AC4C: KSPInitialResidual (itres.c:64)<br>
==2605== by 0x85EB09A: KSPSolve_GMRES (gmres.c:241)<br>
==2605== Address 0xafae5d0 is 0 bytes after a block of size 93,488 alloc'd<br>
==2605== at 0x4023F5B: calloc (vg_replace_malloc.c:418)<br>
==2605== by 0x8B4E9C7: hypre_CAlloc (hypre_memory.c:121)<br>
==2605== by 0x8B4CA67: hypre_CSRMatrixInitialize (csr_matrix.c:91)<br>
==2605== by 0x8B32EC8: hypre_ParCSRMatrixInitialize<br>
(par_csr_matrix.c:200)<br>
==2605== by 0x8AE0C44: hypre_IJMatrixInitializeParCSR<br>
(IJMatrix_parcsr.c:272)<br>
==2605== by 0x8ADBE09: HYPRE_IJMatrixInitialize (HYPRE_IJMatrix.c:302)<br>
==2605== by 0x891AD3A: MatHYPRE_IJMatrixFastCopy_SeqAIJ (mhyp.c:174)<br>
==2605== by 0x891A2E1: MatHYPRE_IJMatrixCopy (mhyp.c:131)<br>
==2605== by 0x855A445: PCSetUp_HYPRE (hypre.c:130)<br>
==2605== by 0x86256A9: PCSetUp (precon.c:794)<br>
==2605== by 0x85A6E62: KSPSetUp (itfunc.c:237)<br>
==2605== by 0x85A7EAB: KSPSolve (itfunc.c:353)<br>
==2605==<br>
==2605== Invalid write of size 4<br>
==2605== at 0x8B23E0C: hypre_BoomerAMGCreateS (par_strength.c:301)<br>
==2605== by 0x8AE966F: hypre_BoomerAMGSetup (par_amg_setup.c:630)<br>
==2605== by 0x8AE4A4D: HYPRE_BoomerAMGSetup (HYPRE_parcsr_amg.c:58)<br>
==2605== by 0x855A5D9: PCSetUp_HYPRE (hypre.c:134)<br>
==2605== by 0x86256A9: PCSetUp (precon.c:794)<br>
==2605== by 0x85A6E62: KSPSetUp (itfunc.c:237)<br>
==2605== by 0x85A7EAB: KSPSolve (itfunc.c:353)<br>
==2605== by 0x80F5B16: applyPrecSchur(void*, _p_Vec*, _p_Vec*)<br>
(NS3T10mpi.cxx:3741)<br>
==2605== by 0x851C47E: PCApply_Shell (shellpc.c:129)<br>
==2605== by 0x862074E: PCApply (precon.c:357)<br>
==2605== by 0x863AC4C: KSPInitialResidual (itres.c:64)<br>
==2605== by 0x85EB09A: KSPSolve_GMRES (gmres.c:241)<br>
==2605== Address 0xb12a050 is 0 bytes after a block of size 46,744 alloc'd<br>
==2605== at 0x4023F5B: calloc (vg_replace_malloc.c:418)<br>
==2605== by 0x8B4E9C7: hypre_CAlloc (hypre_memory.c:121)<br>
==2605== by 0x8B23980: hypre_BoomerAMGCreateS (par_strength.c:163)<br>
==2605== by 0x8AE966F: hypre_BoomerAMGSetup (par_amg_setup.c:630)<br>
==2605== by 0x8AE4A4D: HYPRE_BoomerAMGSetup (HYPRE_parcsr_amg.c:58)<br>
==2605== by 0x855A5D9: PCSetUp_HYPRE (hypre.c:134)<br>
==2605== by 0x86256A9: PCSetUp (precon.c:794)<br>
==2605== by 0x85A6E62: KSPSetUp (itfunc.c:237)<br>
==2605== by 0x85A7EAB: KSPSolve (itfunc.c:353)<br>
==2605== by 0x80F5B16: applyPrecSchur(void*, _p_Vec*, _p_Vec*)<br>
(NS3T10mpi.cxx:3741)<br>
==2605== by 0x851C47E: PCApply_Shell (shellpc.c:129)<br>
==2605== by 0x862074E: PCApply (precon.c:357)<br>
==2605==<br>
...<br>
==2605== Invalid read of size 8<br>
==2605== at 0x8B1ACE8: hypre_BoomerAMGRelax (par_relax.c:182)<br>
==2605== by 0x8B1DFBF: hypre_BoomerAMGRelaxIF (par_relax_interface.c:110)<br>
==2605== by 0x8AFC310: hypre_BoomerAMGCycle (par_cycle.c:386)<br>
==2605== by 0x8AEE09E: hypre_BoomerAMGSolve (par_amg_solve.c:252)<br>
==2605== by 0x8AE4A25: HYPRE_BoomerAMGSolve (HYPRE_parcsr_amg.c:76)<br>
==2605== by 0x855AAA4: PCApply_HYPRE (hypre.c:172)<br>
==2605== by 0x862074E: PCApply (precon.c:357)<br>
==2605== by 0x8606095: KSPSolve_PREONLY (preonly.c:29)<br>
==2605== by 0x85A85D3: KSPSolve (itfunc.c:385)<br>
==2605== by 0x80F5B16: applyPrecSchur(void*, _p_Vec*, _p_Vec*)<br>
(NS3T10mpi.cxx:3741)<br>
==2605== by 0x851C47E: PCApply_Shell (shellpc.c:129)<br>
==2605== by 0x862074E: PCApply (precon.c:357)<br>
==2605== Address 0xafae5d0 is 0 bytes after a block of size 93,488 alloc'd<br>
==2605== at 0x4023F5B: calloc (vg_replace_malloc.c:418)<br>
==2605== by 0x8B4E9C7: hypre_CAlloc (hypre_memory.c:121)<br>
==2605== by 0x8B4CA67: hypre_CSRMatrixInitialize (csr_matrix.c:91)<br>
==2605== by 0x8B32EC8: hypre_ParCSRMatrixInitialize<br>
(par_csr_matrix.c:200)<br>
==2605== by 0x8AE0C44: hypre_IJMatrixInitializeParCSR<br>
(IJMatrix_parcsr.c:272)<br>
==2605== by 0x8ADBE09: HYPRE_IJMatrixInitialize (HYPRE_IJMatrix.c:302)<br>
==2605== by 0x891AD3A: MatHYPRE_IJMatrixFastCopy_SeqAIJ (mhyp.c:174)<br>
==2605== by 0x891A2E1: MatHYPRE_IJMatrixCopy (mhyp.c:131)<br>
==2605== by 0x855A445: PCSetUp_HYPRE (hypre.c:130)<br>
==2605== by 0x86256A9: PCSetUp (precon.c:794)<br>
==2605== by 0x85A6E62: KSPSetUp (itfunc.c:237)<br>
==2605== by 0x85A7EAB: KSPSolve (itfunc.c:353)<br>
==2605==<br>
...<br>
0 KSP Residual norm 8.368803253774e-06<br>
==2605== Invalid read of size 8<br>
==2605== at 0x8B1ADC0: hypre_BoomerAMGRelax (par_relax.c:196)<br>
==2605== by 0x8B1DFBF: hypre_BoomerAMGRelaxIF (par_relax_interface.c:110)<br>
==2605== by 0x8AFC310: hypre_BoomerAMGCycle (par_cycle.c:386)<br>
==2605== by 0x8AEE09E: hypre_BoomerAMGSolve (par_amg_solve.c:252)<br>
==2605== by 0x8AE4A25: HYPRE_BoomerAMGSolve (HYPRE_parcsr_amg.c:76)<br>
==2605== by 0x855AAA4: PCApply_HYPRE (hypre.c:172)<br>
==2605== by 0x862074E: PCApply (precon.c:357)<br>
==2605== by 0x8606095: KSPSolve_PREONLY (preonly.c:29)<br>
==2605== by 0x85A85D3: KSPSolve (itfunc.c:385)<br>
==2605== by 0x80F5B16: applyPrecSchur(void*, _p_Vec*, _p_Vec*)<br>
(NS3T10mpi.cxx:3741)<br>
==2605== by 0x851C47E: PCApply_Shell (shellpc.c:129)<br>
==2605== by 0x862074E: PCApply (precon.c:357)<br>
==2605== Address 0xcded820 is 0 bytes after a block of size 93,488 alloc'd<br>
==2605== at 0x4023F5B: calloc (vg_replace_malloc.c:418)<br>
==2605== by 0x8B4E9C7: hypre_CAlloc (hypre_memory.c:121)<br>
==2605== by 0x8B4CA67: hypre_CSRMatrixInitialize (csr_matrix.c:91)<br>
==2605== by 0x8B32EC8: hypre_ParCSRMatrixInitialize<br>
(par_csr_matrix.c:200)<br>
==2605== by 0x8AE0C44: hypre_IJMatrixInitializeParCSR<br>
(IJMatrix_parcsr.c:272)<br>
==2605== by 0x8ADBE09: HYPRE_IJMatrixInitialize (HYPRE_IJMatrix.c:302)<br>
==2605== by 0x891AD3A: MatHYPRE_IJMatrixFastCopy_SeqAIJ (mhyp.c:174)<br>
==2605== by 0x891A2E1: MatHYPRE_IJMatrixCopy (mhyp.c:131)<br>
==2605== by 0x855A445: PCSetUp_HYPRE (hypre.c:130)<br>
==2605== by 0x86256A9: PCSetUp (precon.c:794)<br>
==2605== by 0x85A6E62: KSPSetUp (itfunc.c:237)<br>
==2605== by 0x85A7EAB: KSPSolve (itfunc.c:353)<br>
==2605==<br>
<br>
<br>
<br>
<br>
Matthew Knepley wrote:<br>
<blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
Try valgrind.<br>
<br>
Matt<div><div></div><div class="h5"><br>
<br>
On Sat, Nov 14, 2009 at 3:32 PM, Dominik Szczerba <<a href="mailto:dominik@itis.ethz.ch" target="_blank">dominik@itis.ethz.ch</a> <mailto:<a href="mailto:dominik@itis.ethz.ch" target="_blank">dominik@itis.ethz.ch</a>>> wrote:<br>
<br>
Now for something more serious: I get a crash like this one:<br>
<br>
Starting KSPSolve (1/2)<br>
0 KSP Residual norm 2.964538623545e-06<br>
*** glibc detected *** /home/domel/build/solve-debug/ns3t10mpi:<br>
malloc(): memory corruption: 0x09258008 ***<br>
======= Backtrace: =========<br>
/lib/tls/i686/cmov/libc.so.6[0x5f9ff1]<br>
/lib/tls/i686/cmov/libc.so.6[0x5fcbb3]<br>
/lib/tls/i686/cmov/libc.so.6(__libc_calloc+0xa9)[0x5fe009]<br>
/home/domel/build/solve-debug/ns3t10mpi(hypre_CAlloc+0x2c)[0x8b4ea28]<br>
/home/domel/build/solve-debug/ns3t10mpi(hypre_BoomerAMGCoarsenRuge+0xb5)[0x8af2c7b]<br>
(and so on)<br>
<br>
gdb invoked as:<br>
<br>
mpiexec -np 2 ..... -on_error_attach_debugger -display :0.0<br>
<br>
does not display any backtrace after the crash.<br>
<br>
Any hints how to debug are highly appreciated.<br>
<br>
Dominik<br>
<br>
<br>
<br>
<br>
-- <br>
What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>
-- Norbert Wiener<br>
</div></div></blockquote>
<br>
</blockquote></div><br><br clear="all"><br>-- <br>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>-- Norbert Wiener<br>