<div dir="ltr">yes, I call the given code many times, what i have seen so far is that changing e.g. the order of destruction statements changes the time of failure,I will try to debug to get more details</div><div class="gmail_extra">
<br><br><div class="gmail_quote">2013/10/18 Mark F. Adams <span dir="ltr"><<a href="mailto:mfadams@lbl.gov" target="_blank">mfadams@lbl.gov</a>></span><br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div style="word-wrap:break-word">I would guess there is some memory corruption or leak someplace. Try running with -on_error_attach_debugger, and try to get some more information. Do you just repeat a call to the code below many times? Is this deterministic, does it fail on the same solve each time? <div>
<br></div><div><br><div><div><div class="h5"><div>On Oct 18, 2013, at 9:30 AM, Einar Sørheim <<a href="mailto:einar.sorheim@gmail.com" target="_blank">einar.sorheim@gmail.com</a>> wrote:</div><br></div></div><blockquote type="cite">
<div><div class="h5"><div dir="ltr">Uncommenting <span style="font-family:arial,sans-serif;font-size:13px"> KSPSetFromOptions solved my intial problem, however I got a new problem that occurs after many timesteps, in our code we use petsc gamg as a solver for several problems(e.g. thermal and mechanical) at each time step, I have a suspicion it might have something to do with cleanup after solving, the rror appears after many calls to the petsc gamg solver interface, the error message is the following:</span><div>
<font face="arial, sans-serif"><br></font><div>[0]PETSC ERROR: ------------------------------------------------------------------------</div><div>[0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range</div>
<div>[0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger</div><div>[0]PETSC ERROR: or see <a href="http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[0]PETSC" target="_blank">http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[0]PETSC</a> ERROR: or try <a href="http://valgrind.org/" target="_blank">http://valgrind.org</a> on GNU/linux and Apple Mac OS X to find memory corruption errors</div>
<div>[0]PETSC ERROR: likely location of problem given in stack below</div><div>[0]PETSC ERROR: --------------------- Stack Frames ------------------------------------</div><div>[0]PETSC ERROR: Note: The EXACT line numbers in the stack are not available,</div>
<div>[0]PETSC ERROR: INSTEAD the line number of the start of the function</div><div>[0]PETSC ERROR: is given.</div><div>[0]PETSC ERROR: [0] MatSetNearNullSpace line 7580 src/mat/interface/D:\PROGRA~1\PETSC-~1.3\src\mat\INTERF~1\matrix.c</div>
<div>[0]PETSC ERROR: --------------------- Error Message ------------------------------------</div><div>[0]PETSC ERROR: Signal received!</div><div>[0]PETSC ERROR: ------------------------------------------------------------------------</div>
<div>[0]PETSC ERROR: Petsc Release Version 3.4.3, Oct, 15, 2013 </div><div>[0]PETSC ERROR: See docs/changes/index.html for recent updates.</div><div>[0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.</div>
<div>[0]PETSC ERROR: See docs/index.html for manual pages.</div><div>[0]PETSC ERROR: ------------------------------------------------------------------------</div><div>[0]PETSC ERROR: D:\Programming\gits\Alsim\SharedModules - Copy\Source\Release\Alsim_nypetsc.exe on a arch-mswin-c-debug named CMP3 by admeinar Fri Oct 18 12:26:04 2013</div>
<div>[0]PETSC ERROR: Libraries linked from /cygdrive/d/Programming/petsc-3.4.3/arch-mswin-c-debug/lib</div><div>[0]PETSC ERROR: Configure run at Thu Oct 17 08:23:18 2013</div><div>[0]PETSC ERROR: Configure options --with-debugging=1 --with-openmp=yes --with-pthread=no --with-cc="win32fe icl" --with-fc="win32fe ifort" --with-cxx="win32fe icl" --with-blas-lapack-dir="C:/Program Files (x86)/Intel/Composer XE 2013/mkl/lib/intel64" --download-superlu --with-sowing=0 --with-c2html=0 --with-mpi-dir="C:/Program Files (x86)/Intel/MPI/<a href="http://4.1.0.028/em64t" target="_blank">4.1.0.028/em64t</a>" --useThreads=0 --useThreads=0</div>
<div><span style="font-family:arial,sans-serif;font-size:13px"><br></span></div><div><span style="font-family:arial,sans-serif;font-size:13px">The source code for the petsc interface: </span></div><div><span style="font-family:arial,sans-serif;font-size:13px"><br>
</span></div><div><div><font face="arial, sans-serif"> subroutine petscsolver(i_nodes,r_elements_,i_rowstart_,i_columnindex_,r_x_,r_b_,i_Nnz,i_Precond, i_Solver</font></div><div><font face="arial, sans-serif"> $ ,coords_,i_nodedof)</font></div>
<div><font face="arial, sans-serif">#include <finclude/petscsys.h></font></div><div><font face="arial, sans-serif">#include <finclude/petscvec.h></font></div><div><font face="arial, sans-serif">#include <finclude/petscmat.h></font></div>
<div><font face="arial, sans-serif">#include <finclude/petscpc.h></font></div><div><font face="arial, sans-serif">#include <finclude/petscksp.h></font></div><div><font face="arial, sans-serif">#include <finclude/petscviewer.h></font></div>
<div><font face="arial, sans-serif"><br></font></div><div><font face="arial, sans-serif"> integer i_nodes, i_Nnz, i_Precond, i_solver, i_nodedof</font></div><div><font face="arial, sans-serif"> integer i_rowstart_(*),i_columnindex_(*)</font></div>
<div><font face="arial, sans-serif"> double precision r_elements_(*),r_x_(*), r_b_(*),coords_(*)</font></div><div><font face="arial, sans-serif"> integer, allocatable :: i_indices_(:)</font></div><div><font face="arial, sans-serif">! Variables:</font></div>
<div><font face="arial, sans-serif">!</font></div><div><font face="arial, sans-serif">! A - matrix that defines linear system</font></div><div><font face="arial, sans-serif">! ksp - KSP context</font></div><div>
<font face="arial, sans-serif">! ksp - KSP context</font></div><div><font face="arial, sans-serif">! x, b, u - approx solution, RHS, exact solution vectors</font></div><div><font face="arial, sans-serif">!</font></div>
<div><font face="arial, sans-serif">! implicit none</font></div><div><font face="arial, sans-serif"> Vec x,u,b,vec_coords</font></div><div><font face="arial, sans-serif"> PC pc</font></div><div><font face="arial, sans-serif"> Mat A,F,Matnull</font></div>
<div><font face="arial, sans-serif"> KSP ksp</font></div><div><font face="arial, sans-serif"> PetscInt i,j,II,JJ,m,n,i1,imaxiter</font></div><div><font face="arial, sans-serif"> PetscInt Istart,Iend</font></div>
<div><font face="arial, sans-serif"> PetscInt nsteps,one, neg_one</font></div><div><font face="arial, sans-serif"> PetscErrorCode ierr</font></div><div><font face="arial, sans-serif"> PetscBool flg</font></div>
<div><font face="arial, sans-serif"> PetscScalar v,x_array_(1), norm, tol, t0,t1</font></div><div><font face="arial, sans-serif"> PetscOffset i_x</font></div><div><font face="arial, sans-serif"> PetscViewer viewer</font></div>
<div><font face="arial, sans-serif"><br></font></div><div><br></div><div><font face="arial, sans-serif"> call PetscTime(t0,ierr)</font></div><div><font face="arial, sans-serif"> write(*,*) "Enter petsc"</font></div>
<div><br></div><div><font face="arial, sans-serif"> n = 1</font></div><div><font face="arial, sans-serif"> nsteps = 1</font></div><div><font face="arial, sans-serif"> one = 1</font></div><div><font face="arial, sans-serif"> neg_one = 1</font></div>
<div><font face="arial, sans-serif"> i1 = 1</font></div><div><font face="arial, sans-serif"> imaxiter = 1200</font></div><div><br></div><div><br></div><div><font face="arial, sans-serif"> call MatCreate(PETSC_COMM_WORLD,A,ierr)</font></div>
<div><font face="arial, sans-serif"> call MatSetSizes(A,PETSC_DECIDE,PETSC_DECIDE,i_nodes,i_nodes,ierr)</font></div><div><font face="arial, sans-serif"> call MatSetType(A, MATSEQAIJ,ierr)</font></div><div><font face="arial, sans-serif"> call MatSetUp(A,ierr)</font></div>
<div><font face="arial, sans-serif"> call MatSeqAIJSetPreallocation(A,100,PETSC_NULL_INTEGER,ierr)</font></div><div><font face="arial, sans-serif">! The matrix is partitioned by contiguous chunks of rows across the</font></div>
<div><font face="arial, sans-serif">! processors. Determine which rows of the matrix are locally owned. </font></div><div><font face="arial, sans-serif"><br></font></div><div><font face="arial, sans-serif"> call MatGetOwnershipRange(A,Istart,Iend,ierr)</font></div>
<div><font face="arial, sans-serif"><br></font></div><div><br></div><div><font face="arial, sans-serif"> write(*,*) "Start of matrix fill", i_nodes, Istart, Iend</font></div><div><font face="arial, sans-serif"> do i=1,i_nodes</font></div>
<div><font face="arial, sans-serif">! write(*,*) "Start of loop",i !!, i_indices_(i+1)</font></div><div><font face="arial, sans-serif">! i_indices_(i+1) = i</font></div><div><font face="arial, sans-serif">! write(*,*) "Start of loop2"</font></div>
<div><font face="arial, sans-serif"> do j=i_rowstart_(i),i_rowstart_(i+1)-1</font></div><div><font face="arial, sans-serif">! write(*,*) "Start of loop 3"</font></div><div><font face="arial, sans-serif"> v = r_elements_(j)</font></div>
<div><font face="arial, sans-serif">! write(*,*) i1,i,i1,j-1,v</font></div><div><font face="arial, sans-serif"> call MatSetValues(A,i1,i-1,i1,i_columnindex_(j)-1,v,INSERT_VALUES,ierr)</font></div><div>
<font face="arial, sans-serif"> if (ierr.gt.0) stop</font></div><div><font face="arial, sans-serif"> end do</font></div><div><font face="arial, sans-serif"> end do</font></div><div><font face="arial, sans-serif">! write(*,*) "End of matrix fill"</font></div>
<div><font face="arial, sans-serif"><br></font></div><div><font face="arial, sans-serif">! Assemble matrix, using the 2-step process:</font></div><div><font face="arial, sans-serif">! MatAssemblyBegin(), MatAssemblyEnd()</font></div>
<div><font face="arial, sans-serif">! Computations can be done while messages are in transition</font></div><div><font face="arial, sans-serif">! by placing code between these two statements.</font></div><div><font face="arial, sans-serif"><br>
</font></div><div><font face="arial, sans-serif"> call MatAssemblyBegin(A,MAT_FINAL_ASSEMBLY,ierr)</font></div><div><font face="arial, sans-serif"> call MatAssemblyEnd(A,MAT_FINAL_ASSEMBLY,ierr)</font></div><div>
<font face="arial, sans-serif"><br></font></div><div><font face="arial, sans-serif">! Create parallel vectors.</font></div><div><font face="arial, sans-serif">! - When using VecCreate(), the parallel partitioning of the vector</font></div>
<div><font face="arial, sans-serif">! is determined by PETSc at runtime.</font></div><div><font face="arial, sans-serif">! - Note: We form 1 vector from scratch and then duplicate as needed.</font></div><div><font face="arial, sans-serif">! write(*,*) "Start of vector fill"</font></div>
<div><font face="arial, sans-serif"> call VecCreate(PETSC_COMM_WORLD,u,ierr)</font></div><div><font face="arial, sans-serif"> call VecSetType(u, VECSEQ,ierr)</font></div><div><font face="arial, sans-serif"> call VecSetSizes(u,PETSC_DECIDE,i_nodes,ierr)</font></div>
<div><font face="arial, sans-serif">! call VecSetFromOptions(u,ierr) </font></div><div><font face="arial, sans-serif"> call VecDuplicate(u,b,ierr)</font></div><div><font face="arial, sans-serif"> call VecDuplicate(b,x,ierr)</font></div>
<div><font face="arial, sans-serif"> do i=1,i_nodes</font></div><div><font face="arial, sans-serif"> call VecSetValues(x,i1,i-1,r_x_(i),INSERT_VALUES,ierr)</font></div><div><font face="arial, sans-serif"> call VecSetValues(b,i1,i-1,r_b_(i),INSERT_VALUES,ierr)</font></div>
<div><font face="arial, sans-serif"> enddo</font></div><div><font face="arial, sans-serif">! Assemble vector</font></div><div><font face="arial, sans-serif"><br></font></div><div><font face="arial, sans-serif"> call VecAssemblyBegin(x,ierr)</font></div>
<div><font face="arial, sans-serif"> call VecAssemblyEnd(x,ierr)</font></div><div><font face="arial, sans-serif"> call VecAssemblyBegin(b,ierr)</font></div><div><font face="arial, sans-serif"> call VecAssemblyEnd(b,ierr)</font></div>
<div><font face="arial, sans-serif"> call PetscTime(t1,ierr)</font></div><div><font face="arial, sans-serif">! write(*,*) "End of vec fill time spent :", t1-t0</font></div><div><font face="arial, sans-serif">! Create linear solver context</font></div>
<div><font face="arial, sans-serif"><br></font></div><div><font face="arial, sans-serif"> call KSPCreate(PETSC_COMM_WORLD,ksp,ierr)</font></div><div><font face="arial, sans-serif"><br></font></div><div><br></div><div>
<font face="arial, sans-serif">! Set operators. Here the matrix that defines the linear system</font></div><div><font face="arial, sans-serif">! also serves as the preconditioning matrix.</font></div><div><font face="arial, sans-serif"><br>
</font></div><div><font face="arial, sans-serif"> call KSPSetOperators(ksp,A,A,DIFFERENT_NONZERO_PATTERN,ierr)</font></div><div><font face="arial, sans-serif"><br></font></div><div><br></div><div><font face="arial, sans-serif"> call KSPGetPC(ksp,pc,ierr)</font></div>
<div><font face="arial, sans-serif"> tol = 1.e-20</font></div><div><font face="arial, sans-serif"> call KSPSetTolerances(ksp,tol,PETSC_DEFAULT_DOUBLE_PRECISION,PETSC_DEFAULT_DOUBLE_PRECISION,imaxiter,ierr)</font></div>
<div><font face="arial, sans-serif"> select case (i_Precond)</font></div><div><font face="arial, sans-serif"> case ( 1 )</font></div><div><font face="arial, sans-serif"> call PCSetType(pc,PCJACOBI,ierr)</font></div>
<div><font face="arial, sans-serif"> case ( 2 )</font></div><div><font face="arial, sans-serif"> call PCSetType(pc,PCILU,ierr)</font></div><div><font face="arial, sans-serif"> call PCFactorSetMatSolverPackage(pc,MATSOLVERSUPERLU,ierr);</font></div>
<div><font face="arial, sans-serif"> call PCFactorSetUpMatSolverPackage(pc,ierr); </font></div><div><font face="arial, sans-serif"> call PCFactorGetMatrix(pc,F);</font></div><div><font face="arial, sans-serif"> call MatSuperluSetILUDropTol(F,1.e-4);</font></div>
<div><font face="arial, sans-serif"> case ( 3 )</font></div><div><font face="arial, sans-serif"> call PCSetType(pc,PCGAMG,ierr)</font></div><div><font face="arial, sans-serif">! call PCGAMGInitializePackage()</font></div>
<div><font face="arial, sans-serif">! call PCCreate_GAMG(pc,ierr)</font></div><div><font face="arial, sans-serif"> case DEFAULT</font></div><div><font face="arial, sans-serif"> call PCSetType(pc,PCJACOBI,ierr)</font></div>
<div><font face="arial, sans-serif"> end select</font></div><div><font face="arial, sans-serif"><br></font></div><div><font face="arial, sans-serif">! call PCSetType(pc,PCJACOBI,ierr)</font></div><div><font face="arial, sans-serif"> select case (i_solver)</font></div>
<div><font face="arial, sans-serif"> case ( 0 )</font></div><div><font face="arial, sans-serif"> call KSPSetType(ksp,KSPCG,ierr)</font></div><div><font face="arial, sans-serif"> case DEFAULT</font></div>
<div><font face="arial, sans-serif"> call KSPSetType(ksp,KSPCG,ierr)</font></div><div><font face="arial, sans-serif"> end select</font></div><div><font face="arial, sans-serif"> call KSPCGSetType(ksp,KSP_CG_SYMMETRIC,ierr)</font></div>
<div><font face="arial, sans-serif"> if (i_nodedof==3) then</font></div><div><font face="arial, sans-serif"> write(*,*) "Set coord, number of nodes is:", i_nodes/i_nodedof</font></div><div><font face="arial, sans-serif"> call VecCreateSeqWithArray(MPI_COMM_SELF,3,i_nodes,coords_,vec_coords,ierr)</font></div>
<div><font face="arial, sans-serif"> call MatNullSpaceCreateRigidBody(vec_coords,Matnull,ierr)</font></div><div><font face="arial, sans-serif"> call MatSetNearNullSpace(A,Matnull,ierr)</font></div><div><font face="arial, sans-serif"> call MatNullSpaceDestroy(Matnull,ierr)</font></div>
<div><br></div><div><font face="arial, sans-serif"> write(*,*) "Finish setting null space ierr =", ierr</font></div><div><font face="arial, sans-serif">! call PCSetCoordinates( pc, 3,i_nodes/i_nodedof, coords_, ierr )</font></div>
<div><br></div><div><font face="arial, sans-serif"> end if</font></div><div><font face="arial, sans-serif">! call KSPMonitorSet(ksp, KSPMonitorDefault, PETSC_NULL, PETSC_NULL,ierr)</font></div><div><font face="arial, sans-serif">! Set runtime options, e.g.,</font></div>
<div><font face="arial, sans-serif">! -ksp_type <type> -pc_type <type> -ksp_monitor -ksp_rtol <rtol></font></div><div><font face="arial, sans-serif">! These options will override those specified above as long as</font></div>
<div><font face="arial, sans-serif">! KSPSetFromOptions() is called _after_ any other customization</font></div><div><font face="arial, sans-serif">! routines.</font></div><div><font face="arial, sans-serif"><br></font></div>
<div><font face="arial, sans-serif"> call KSPSetFromOptions(ksp,ierr)</font></div><div><font face="arial, sans-serif"> call KSPSetInitialGuessNonzero(ksp,PETSC_TRUE,ierr)</font></div><div><font face="arial, sans-serif"> call KSPSetUp(ksp,ierr)</font></div>
<div><font face="arial, sans-serif"> call PetscTime(t0,ierr)</font></div><div><font face="arial, sans-serif"> write(*,*) "Time setup", t0-t1</font></div><div><font face="arial, sans-serif">! - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - </font></div>
<div><font face="arial, sans-serif">! Solve the linear system</font></div><div><font face="arial, sans-serif">! - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - </font></div><div>
<font face="arial, sans-serif"><br></font></div><div><font face="arial, sans-serif"> call KSPSolve(ksp,b,x,ierr)</font></div><div><font face="arial, sans-serif"> call PetscTime(t1,ierr)</font></div><div><font face="arial, sans-serif"> write(*,*) "end solve, time used:",t1-t0</font></div>
<div><font face="arial, sans-serif"> call KSPGetSolution(ksp,x,ierr)</font></div><div><font face="arial, sans-serif"> call VecGetArray(x,x_array_,i_x,ierr)</font></div><div><font face="arial, sans-serif"> do i=1,i_nodes</font></div>
<div><font face="arial, sans-serif"> r_x_(i) = x_array_(i_x + i)</font></div><div><font face="arial, sans-serif"> enddo</font></div><div><font face="arial, sans-serif"> call VecRestoreArray(x,x_array_,i_x,ierr)</font></div>
<div><br></div><div><font face="arial, sans-serif"><br></font></div><div><font face="arial, sans-serif">! Free work space. All PETSc objects should be destroyed when they</font></div><div><font face="arial, sans-serif">! are no longer needed.</font></div>
<div><font face="arial, sans-serif">! call PCDestroy(pc,ierr)</font></div><div><font face="arial, sans-serif"><br></font></div><div><font face="arial, sans-serif"> call VecDestroy(u,ierr)</font></div><div><font face="arial, sans-serif"> call VecDestroy(x,ierr)</font></div>
<div><font face="arial, sans-serif"> call VecDestroy(b,ierr)</font></div><div><font face="arial, sans-serif"> call MatDestroy(A,ierr)</font></div><div><font face="arial, sans-serif"> if (i_nodedof==3) then</font></div>
<div><font face="arial, sans-serif"> call VecDestroy(vec_coords,ierr)</font></div><div><font face="arial, sans-serif"> call MatDestroy(Matnull,ierr)</font></div><div><font face="arial, sans-serif"> endif</font></div>
<div><font face="arial, sans-serif"> call KSPDestroy(ksp,ierr)</font></div><div><font face="arial, sans-serif"> call PetscTime(t0,ierr)</font></div><div><font face="arial, sans-serif"> write(*,*) "time celan up :", t0-t1</font></div>
<div><br></div><div><font face="arial, sans-serif"> end</font></div><div style="font-family:arial,sans-serif;font-size:13px"><br></div></div><div class="gmail_extra">-- <br>Einar Sørheim
</div></div></div>
</div></div><span><Mail Attachment></span></blockquote></div><br></div></div></blockquote></div><br><br clear="all"><div><br></div>-- <br>Einar Sørheim
</div>