[petsc-users] mac os x, problem with partitioning

Tue May 5 07:28:40 CDT 2015

   We haven't see this before, I can't think of a particular reason why it would work under Linux and not MacOSX.  You said you ran with valgrind, was that only on Linux or have you run the cases that fail on Apple with valgrind on Apple? (Yes valgrind often isn't supported on Apple but the development version of valgrind now works for both maverick and Yosemite. I would most definitely recommend first running with valgrind on Apple if you have not. You can also try running with -start_in_debugger and when PTscotch crashes look at the values to see why it thinks "arc data do not match", it could be a bug that doesn't expose itself on Linux with the different compilers.

  The error with permetis 
> Argument out of range
> [0]PETSC ERROR: Inserting a new nonzero at global row/column (71, 354) into matrix
is not a crash like with ptscotch. Could it be that you are actually inserting new non-zero locations at later time steps ? Or does it get past this point on Linux with the exact same options. If you are inserting new nonzero locations and want to then you can use the MatSetOption() to set MAT_NEW_NONZERO_LOCATION_ERR to PETSC_FALSE

  Good luck,

   Barry

> On May 5, 2015, at 5:18 AM, Lukasz Kaczmarczyk <Lukasz.Kaczmarczyk at glasgow.ac.uk> wrote:
> 
> Hello,
> 
> I struggle with problem of matrix partitioning on McOSX for some time, since Maverick. Error is random, it runs ok for couple of times and then it beak. I using the newest version of petsc, for bitbucket repository, however this problem exist for long time for older versions too.
> 
> If I load partitioned mesh, f.e. using metis or any other partitioner code work without errors. Moreover, the same code I run on linux, redhat, ubutu, tested with valgrind and no problems like that ever existed. I used as well intel compiler on linux and following problems never existed. 
> 
> *) When I use ptscoch,
> 
> It is at the point of graph partitioning (random),
> 
> (2): ERROR: dgraphCheck: arc data do not match
> [2]PETSC ERROR: #1 MatPartitioningApply_PTScotch() line 305 in /opt/petsc/src/mat/partition/impls/scotch/scotch.c
> (3): ERROR: dgraphCheck: arc data do not match
> [3]PETSC ERROR: #1 MatPartitioningApply_PTScotch() line 305 in /opt/petsc/src/mat/partition/impls/scotch/scotch.c
> [3]PETSC ERROR: #2 MatPartitioningApply() line 225 in /opt/petsc/src/mat/partition/partition.c
> [3]PETSC ERROR: #3 partition_problem() line 611 in /Users/likask/MyBuild/mofem-cephas/mofem_v0.2/src/interfaces/impl/CreateMat.cpp
> (0): ERROR: dgraphCheck: arc data do not match
> [0]PETSC ERROR: #1 MatPartitioningApply_PTScotch() line 305 in /opt/petsc/src/mat/partition/impls/scotch/scotch.c
> [0]PETSC ERROR: #2 MatPartitioningApply() line 225 in /opt/petsc/src/mat/partition/partition.c
> [0]PETSC ERROR: #3 partition_problem() line 611 in /Users/likask/MyBuild/mofem-cephas/mofem_v0.2/src/interfaces/impl/CreateMat.cpp
> (1): ERROR: dgraphCheck: arc data do not match
> [1]PETSC ERROR: #1 MatPartitioningApply_PTScotch() line 305 in /opt/petsc/src/mat/partition/impls/scotch/scotch.c
> [1]PETSC ERROR: #2 MatPartitioningApply() line 225 in /opt/petsc/src/mat/partition/partition.c
> 
> *) When I use parmetis
> 
> It is random once again, but problem appear later in the code, when matrix is assembled, 
> 
> 
> [1]PETSC ERROR: --------------------- MoFEM Error Message---------------------------------------------------------------------------
> [1]PETSC ERROR: MoFEM version 0.2.1
> [1]PETSC ERROR: MoFEM git commit id 837f38245270d65c22eafb638f023ea3476bc57a
> [1]PETSC ERROR: See http://userweb.eng.gla.ac.uk/lukasz.kaczmarczyk/MoFem/html/guidelines_bug_reporting.html for bug reporting.
> [1]PETSC ERROR: See http://userweb.eng.gla.ac.uk/lukasz.kaczmarczyk/MoFem/html/faq_and_bugs.html for trouble shooting.
> [0]PETSC ERROR: --------------------- MoFEM Error Message---------------------------------------------------------------------------
> [0]PETSC ERROR: MoFEM version 0.2.1
> [0]PETSC ERROR: MoFEM git commit id 837f38245270d65c22eafb638f023ea3476bc57a
> [0]PETSC ERROR: See http://userweb.eng.gla.ac.uk/lukasz.kaczmarczyk/MoFem/html/guidelines_bug_reporting.html for bug reporting.
> [0]PETSC ERROR: See http://userweb.eng.gla.ac.uk/lukasz.kaczmarczyk/MoFem/html/faq_and_bugs.html for trouble shooting.
> [0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
> [0]PETSC ERROR: Argument out of range
> [0]PETSC ERROR: Inserting a new nonzero at global row/column (71, 354) into matrix
> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
> [0]PETSC ERROR: Petsc Development GIT revision: v3.5.3-2812-gc9542f9  GIT Date: 2015-05-03 22:34:15 +0300
> [0]PETSC ERROR: /Users/likask/MyBuild/um_debug/nonlinear_elasticity/nonlinear_dynamics on a arch-darwin-c-debug named dh23-181.eng.gla.ac.uk by likask Tue May  5 09:49:38 2015
> [0]PETSC ERROR: Configure options --with-mpi=1 --with-debugging=1 --download-fblaslapack=1 --download-superlu_dist=1 --download-metis=1 --download-parmetis=1 --download-hypre=1 --download-mumps=1 --download-scalapack=1 --download-blacs=1 --download-moab=1 --download-hdf5=1 --download-netcdf=/opt/petsc/netcdf-4.3.3.1.tar.gz --download-openmpi=1 --CXXFLAGS=-fno-vectorize -CFLAGS=-fno-vectorize --download-mumps=1 --download-ptscotch=1
> [0]PETSC ERROR: #1 MatSetValues_MPIAIJ() line 616 in /opt/petsc/src/mat/impls/aij/mpi/mpiaij.c
> [0]PETSC ERROR: #2 MatAssemblyEnd_MPIAIJ() line 731 in /opt/petsc/src/mat/impls/aij/mpi/mpiaij.c
> [0]PETSC ERROR: [1]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
> [1]PETSC ERROR: Argument out of range
> [1]PETSC ERROR: Inserting a new nonzero at global row/column (300, 536) into matrix
> [1]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
> [1]PETSC ERROR: Petsc Development GIT revision: v3.5.3-2812-gc9542f9  GIT Date: 2015-05-03 22:34:15 +0300
> [1]PETSC ERROR: /Users/likask/MyBuild/um_debug/nonlinear_elasticity/nonlinear_dynamics on a arch-darwin-c-debug named dh23-181.eng.gla.ac.uk by likask Tue May  5 09:49:38 2015
> [1]PETSC ERROR: Configure options --with-mpi=1 --with-debugging=1 --download-fblaslapack=1 --download-superlu_dist=1 --download-metis=1 --download-parmetis=1 --download-hypre=1 --download-mumps=1 --download-scalapack=1 --download-blacs=1 --download-moab=1 --download-hdf5=1 --download-netcdf=/opt/petsc/netcdf-4.3.3.1.tar.gz --download-openmpi=1 --CXXFLAGS=-fno-vectorize -CFLAGS=-fno-vectorize --download-mumps=1 --download-ptscotch=1
> [1]PETSC ERROR: #1 MatSetValues_MPIAIJ() line 613 in /opt/petsc/src/mat/impls/aij/mpi/mpiaij.c
> [1]PETSC ERROR: #2 MatAssemblyEnd_MPIAIJ() line 731 in /opt/petsc/src/mat/impls/aij/mpi/mpiaij.c
> [1]PETSC ERROR: #3 MatAssemblyEnd() line 5097 in /opt/petsc/src/mat/interface/matrix.c
> #3 MatAssemblyEnd() line 5097 in /opt/petsc/src/mat/interface/matrix.c
> [0]PETSC ERROR: #4 postProcess() line 222 in /Users/likask/MyBuild/mofem-cephas/mofem_v0.2/finite_element_library/impl/DirichletBC.cpp
> [0]PETSC ERROR: #5 problem_basic_method_postProcess() line 2636 in /Users/likask/MyBuild/mofem-cephas/mofem_v0.2/src/interfaces/impl/FieldInterfaceCore.cpp
> [0]PETSC ERROR: #6 problem_basic_method_postProcess() line 2651 in /Users/likask/MyBuild/mofem-cephas/mofem_v0.2/src/interfaces/impl/FieldInterfaceCore.cpp
> [0]PETSC ERROR: #7 preProcess() line 2467 in /Users/likask/MyBuild/mofem-cephas/mofem_v0.2/users_modules/convective_mass_element/src/ConvectiveMassElement.hpp
> [0]PETSC ERROR: #8 problem_basic_method_preProcess() line 2615 in /Users/likask/MyBuild/mofem-cephas/mofem_v0.2/src/interfaces/impl/FieldInterfaceCore.cpp
> [0]PETSC ERROR: #9 problem_basic_method_preProcess() line 2628 in /Users/likask/MyBuild/mofem-cephas/mofem_v0.2/src/interfaces/impl/FieldInterfaceCore.cpp
> [0]PETSC ERROR: #10 f_TSSetIJacobian() line 120 in /Users/likask/MyBuild/mofem-cephas/mofem_v0.2/src/petsc/impl/TsCtx.cpp
> [0]PETSC ERROR: #11 TSComputeIJacobian() line 809 in /opt/petsc/src/ts/interface/ts.c
> [0]PETSC ERROR: #12 SNESTSFormJacobian_Alpha() line 181 in /opt/petsc/src/ts/impls/implicit/alpha/alpha.c
> [0]PETSC ERROR: #13 SNESTSFormJacobian() line 4193 in /opt/petsc/src/ts/interface/ts.c
> [0]PETSC ERROR: #14 SNESComputeJacobian() line 2213 in /opt/petsc/src/snes/interface/snes.c
> [0]PETSC ERROR: #15 SNESSolve_NEWTONLS() line 242 in /opt/petsc/src/snes/impls/ls/ls.c
> [0]PETSC ERROR: #16 SNESSolve() line 3884 in /opt/petsc/src/snes/interface/snes.c
> [0]PETSC ERROR: [1]PETSC ERROR: #4 postProcess() line 222 in /Users/likask/MyBuild/mofem-cephas/mofem_v0.2/finite_element_library/impl/DirichletBC.cpp
> [1]PETSC ERROR: #5 problem_basic_method_postProcess() line 2636 in /Users/likask/MyBuild/mofem-cephas/mofem_v0.2/src/interfaces/impl/FieldInterfaceCore.cpp
> [1]PETSC ERROR: #6 problem_basic_method_postProcess() line 2651 in /Users/likask/MyBuild/mofem-cephas/mofem_v0.2/src/interfaces/impl/FieldInterfaceCore.cpp
> [1]PETSC ERROR: #7 preProcess() line 2467 in /Users/likask/MyBuild/mofem-cephas/mofem_v0.2/users_modules/convective_mass_element/src/ConvectiveMassElement.hpp
> [1]PETSC ERROR: #8 problem_basic_method_preProcess() line 2615 in /Users/likask/MyBuild/mofem-cephas/mofem_v0.2/src/interfaces/impl/FieldInterfaceCore.cpp
> [1]PETSC ERROR: #9 problem_basic_method_preProcess() line 2628 in /Users/likask/MyBuild/mofem-cephas/mofem_v0.2/src/interfaces/impl/FieldInterfaceCore.cpp
> [1]PETSC ERROR: #10 f_TSSetIJacobian() line 120 in /Users/likask/MyBuild/mofem-cephas/mofem_v0.2/src/petsc/impl/TsCtx.cpp
> [1]PETSC ERROR: #11 TSComputeIJacobian() line 809 in /opt/petsc/src/ts/interface/ts.c
> [1]PETSC ERROR: #12 SNESTSFormJacobian_Alpha() line 181 in /opt/petsc/src/ts/impls/implicit/alpha/alpha.c
> [1]PETSC ERROR: #13 SNESTSFormJacobian() line 4193 in /opt/petsc/src/ts/interface/ts.c
> [1]PETSC ERROR: #14 SNESComputeJacobian() line 2213 in /opt/petsc/src/snes/interface/snes.c
> [1]PETSC ERROR: #15 SNESSolve_NEWTONLS() line 242 in /opt/petsc/src/snes/impls/ls/ls.c
> [1]PETSC ERROR: #16 SNESSolve() line 3884 in /opt/petsc/src/snes/interface/snes.c
> [1]PETSC ERROR: #17 TSStep_Alpha() line 57 in /opt/petsc/src/ts/impls/implicit/alpha/alpha.c
> [1]PETSC ERROR: #18 TSStep() line 3100 in /opt/petsc/src/ts/interface/ts.c
> [1]PETSC ERROR: #19 TSSolve() line 3290 in /opt/petsc/src/ts/interface/ts.c
> [1]PETSC ERROR: #20 main() line 623 in /Users/likask/MyBuild/mofem-cephas/mofem_v0.2/users_modules/nonlinear_elasticity/nonlinear_dynamics.cpp
> [1]PETSC ERROR: PETSc Option Table entries:
> [1]PETSC ERROR: -ksp_atol 1e-10
> [1]PETSC ERROR: -ksp_rtol 1e-10
> [1]PETSC ERROR: -ksp_type fgmres
> [1]PETSC ERROR: -mat_partitioning_type parmetis
> [1]PETSC ERROR: -my_disp_order 3
> [1]PETSC ERROR: -my_file LShape.cub
> [1]PETSC ERROR: -my_max_post_proc_ref_level 0
> [1]PETSC ERROR: -my_output_prt 1
> [1]PETSC ERROR: -my_time_data_file data.txt
> [1]PETSC ERROR: -my_vel_order 3
> [1]PETSC ERROR: -pc_factor_mat_solver_package mumps
> [1]PETSC ERROR: -pc_type lu
> #17 TSStep_Alpha() line 57 in /opt/petsc/src/ts/impls/implicit/alpha/alpha.c
> [0]PETSC ERROR: #18 TSStep() line 3100 in /opt/petsc/src/ts/interface/ts.c
> --------------------------------------------------------------------------
> MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_WORLD 
> with errorcode 63.
> 
> NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.
> You may or may not see output from other processes, depending on
> exactly when Open MPI kills them.
> --------------------------------------------------------------------------
> [0]PETSC ERROR: #19 TSSolve() line 3290 in /opt/petsc/src/ts/interface/ts.c
> [0]PETSC ERROR: #20 main() line 623 in /Users/likask/MyBuild/mofem-cephas/mofem_v0.2/users_modules/nonlinear_elasticity/nonlinear_dynamics.cpp
> [0]PETSC ERROR: PETSc Option Table entries:
> [0]PETSC ERROR: -ksp_atol 1e-10
> [0]PETSC ERROR: -ksp_rtol 1e-10
> [0]PETSC ERROR: -ksp_type fgmres
> [0]PETSC ERROR: -mat_partitioning_type parmetis
> [0]PETSC ERROR: -my_disp_order 3
> [0]PETSC ERROR: -my_file LShape.cub
> [0]PETSC ERROR: -my_max_post_proc_ref_level 0
> [0]PETSC ERROR: -my_output_prt 1
> [0]PETSC ERROR: -my_time_data_file data.txt
> [0]PETSC ERROR: -my_vel_order 3
> [0]PETSC ERROR: -pc_factor_mat_solver_package mumps
> [0]PETSC ERROR: -pc_type lu
> [0]PETSC ERROR: -snes_atol 1e-7
> [0]PETSC ERROR: -snes_linesearch_type basic
> [0]PETSC ERROR: -snes_max_it 100
> [0]PETSC ERROR: -snes_monitor
> [0]PETSC ERROR: -snes_rtol 1e-7
> [0]PETSC ERROR: -snes_type newtonls
> [0]PETSC ERROR: -ts_dt 0.001
> [0]PETSC ERROR: -ts_final_time 0.002
> [0]PETSC ERROR: -ts_monitor
> [0]PETSC ERROR: -ts_type alpha
> [0]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov----------
> [0]PETSC ERROR: ----------MoFEM End of Error Message -------send entire error message to CMatGU <cmatgu at googlegroups.com> ----------
> [1]PETSC ERROR: -snes_atol 1e-7
> [1]PETSC ERROR: -snes_linesearch_type basic
> [1]PETSC ERROR: -snes_max_it 100
> [1]PETSC ERROR: -snes_monitor
> [1]PETSC ERROR: -snes_rtol 1e-7
> [1]PETSC ERROR: -snes_type newtonls
> [1]PETSC ERROR: -ts_dt 0.001
> [1]PETSC ERROR: -ts_final_time 0.002
> [1]PETSC ERROR: -ts_monitor
> [1]PETSC ERROR: -ts_type alpha
> [1]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov----------
> [1]PETSC ERROR: ----------MoFEM End of Error Message -------send entire error message to CMatGU <cmatgu at googlegroups.com> ----------
> [2]PETSC ERROR: ------------------------------------------------------------------------
> [2]PETSC ERROR: Caught signal number 15 Terminate: Some process (or the batch system) has told this process to end
> [2]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
> [2]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind
> [2]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors
> [2]PETSC ERROR: likely location of problem given in stack below
> [2]PETSC ERROR: ---------------------  Stack Frames ------------------------------------
> [2]PETSC ERROR: Note: The EXACT line numbers in the stack are not available,
> [2]PETSC ERROR:       INSTEAD the line number of the start of the function
> [2]PETSC ERROR:       is given.
> [2]PETSC ERROR: [2] MatStashScatterGetMesg_Private line 615 /opt/petsc/src/mat/utils/matstash.c
> [2]PETSC ERROR: [2] MatAssemblyEnd_MPIAIJ line 717 /opt/petsc/src/mat/impls/aij/mpi/mpiaij.c
> [2]PETSC ERROR: [2] MatAssemblyEnd line 5088 /opt/petsc/src/mat/interface/matrix.c
> [2]PETSC ERROR: [2] postProcess line 164 /Users/likask/MyBuild/mofem-cephas/mofem_v0.2/finite_element_library/impl/DirichletBC.cpp
> [2]PETSC ERROR: [2] problem_basic_method_postProcess line 2632 /Users/likask/MyBuild/mofem-cephas/mofem_v0.2/src/interfaces/impl/FieldInterfaceCore.cpp
> [2]PETSC ERROR: [2] problem_basic_method_postProcess line 2642 /Users/likask/MyBuild/mofem-cephas/mofem_v0.2/src/interfaces/impl/FieldInterfaceCore.cpp
> [2]PETSC ERROR: [2] preProcess line 2441 /Users/likask/MyBuild/mofem-cephas/mofem_v0.2/users_modules/convective_mass_element/src/ConvectiveMassElement.hpp
> [2]PETSC ERROR: [2] problem_basic_method_preProcess line 2610 /Users/likask/MyBuild/mofem-cephas/mofem_v0.2/src/interfaces/impl/FieldInterfaceCore.cpp
> [2]PETSC ERROR: [2] problem_basic_method_preProcess line 2621 /Users/likask/MyBuild/mofem-cephas/mofem_v0.2/src/interfaces/impl/FieldInterfaceCore.cpp
> [2]PETSC ERROR: [2] f_TSSetIJacobian line 94 /Users/likask/MyBuild/mofem-cephas/mofem_v0.2/src/petsc/impl/TsCtx.cpp
> [2]PETSC ERROR: [2] TS user implicit Jacobian line 808 /opt/petsc/src/ts/interface/ts.c
> [2]PETSC ERROR: [2] TSComputeIJacobian line 791 /opt/petsc/src/ts/interface/ts.c
> [2]PETSC ERROR: [2] SNESTSFormJacobian_Alpha line 179 /opt/petsc/src/ts/impls/implicit/alpha/alpha.c
> [2]PETSC ERROR: [2] SNESTSFormJacobian line 4185 /opt/petsc/src/ts/interface/ts.c
> [2]PETSC ERROR: [2] SNES user Jacobian function line 2212 /opt/petsc/src/snes/interface/snes.c
> [2]PETSC ERROR: [2] SNESComputeJacobian line 2171 /opt/petsc/src/snes/interface/snes.c
> [2]PETSC ERROR: [2] SNESSolve_NEWTONLS line 152 /opt/petsc/src/snes/impls/ls/ls.c
> [2]PETSC ERROR: [2] SNESSolve line 3837 /opt/petsc/src/snes/interface/snes.c
> [2]PETSC ERROR: [2] TSStep_Alpha line 40 /opt/petsc/src/ts/impls/implicit/alpha/alpha.c
> [2]PETSC ERROR: [2] TSStep line 3081 /opt/petsc/src/ts/interface/ts.c
> [2]PETSC ERROR: [2] TSSolve line 3246 /opt/petsc/src/ts/interface/ts.c
> [2]PETSC ERROR: --------------------- MoFEM Error Message---------------------------------------------------------------------------
> [2]PETSC ERROR: MoFEM version 0.2.1
> [2]PETSC ERROR: MoFEM git commit id 837f38245270d65c22eafb638f023ea3476bc57a
> [2]PETSC ERROR: See http://userweb.eng.gla.ac.uk/lukasz.kaczmarczyk/MoFem/html/guidelines_bug_reporting.html for bug reporting.
> [2]PETSC ERROR: See http://userweb.eng.gla.ac.uk/lukasz.kaczmarczyk/MoFem/html/faq_and_bugs.html for trouble shooting.
> [2]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
> [2]PETSC ERROR: Signal received
> [2]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
> [2]PETSC ERROR: Petsc Development GIT revision: v3.5.3-2812-gc9542f9  GIT Date: 2015-05-03 22:34:15 +0300
> [2]PETSC ERROR: /Users/likask/MyBuild/um_debug/nonlinear_elasticity/nonlinear_dynamics on a arch-darwin-c-debug named dh23-181.eng.gla.ac.uk by likask Tue May  5 09:49:38 2015
> [2]PETSC ERROR: Configure options --with-mpi=1 --with-debugging=1 --do[3]PETSC ERROR: ------------------------------------------------------------------------
> [3]PETSC ERROR: Caught signal number 15 Terminate: Some process (or the batch system) has told this process to end
> [3]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
> [3]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind
> [3]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors
> [3]PETSC ERROR: likely location of problem given in stack below
> [3]PETSC ERROR: ---------------------  Stack Frames ------------------------------------
> [3]PETSC ERROR: Note: The EXACT line numbers in the stack are not available,
> [3]PETSC ERROR:       INSTEAD the line number of the start of the function
> [3]PETSC ERROR:       is given.
> [3]PETSC ERROR: [3] MatStashScatterGetMesg_Private line 615 /opt/petsc/src/mat/utils/matstash.c
> [3]PETSC ERROR: [3] MatAssemblyEnd_MPIAIJ line 717 /opt/petsc/src/mat/impls/aij/mpi/mpiaij.c
> [3]PETSC ERROR: [3] MatAssemblyEnd line 5088 /opt/petsc/src/mat/interface/matrix.c
> [3]PETSC ERROR: [3] postProcess line 164 /Users/likask/MyBuild/mofem-cephas/mofem_v0.2/finite_element_library/impl/DirichletBC.cpp
> [3]PETSC ERROR: [3] problem_basic_method_postProcess line 2632 /Users/likask/MyBuild/mofem-cephas/mofem_v0.2/src/interfaces/impl/FieldInterfaceCore.cpp
> [3]PETSC ERROR: [3] problem_basic_method_postProcess line 2642 /Users/likask/MyBuild/mofem-cephas/mofem_v0.2/src/interfaces/impl/FieldInterfaceCore.cpp
> [3]PETSC ERROR: [3] preProcess line 2441 /Users/likask/MyBuild/mofem-cephas/mofem_v0.2/users_modules/convective_mass_element/src/ConvectiveMassElement.hpp
> [3]PETSC ERROR: [3] problem_basic_method_preProcess line 2610 /Users/likask/MyBuild/mofem-cephas/mofem_v0.2/src/interfaces/impl/FieldInterfaceCore.cpp
> [3]PETSC ERROR: [3] problem_basic_method_preProcess line 2621 /Users/likask/MyBuild/mofem-cephas/mofem_v0.2/src/interfaces/impl/FieldInterfaceCore.cpp
> [3]PETSC ERROR: [3] f_TSSetIJacobian line 94 /Users/likask/MyBuild/mofem-cephas/mofem_v0.2/src/petsc/impl/TsCtx.cpp
> [3]PETSC ERROR: [3] TS user implicit Jacobian line 808 /opt/petsc/src/ts/interface/ts.c
> [3]PETSC ERROR: [3] TSComputeIJacobian line 791 /opt/petsc/src/ts/interface/ts.c
> [3]PETSC ERROR: [3] SNESTSFormJacobian_Alpha line 179 /opt/petsc/src/ts/impls/implicit/alpha/alpha.c
> [3]PETSC ERROR: [3] SNESTSFormJacobian line 4185 /opt/petsc/src/ts/interface/ts.c
> [3]PETSC ERROR: [3] SNES user Jacobian function line 2212 /opt/petsc/src/snes/interface/snes.c
> [3]PETSC ERROR: [3] SNESComputeJacobian line 2171 /opt/petsc/src/snes/interface/snes.c
> [3]PETSC ERROR: [3] SNESSolve_NEWTONLS line 152 /opt/petsc/src/snes/impls/ls/ls.c
> [3]PETSC ERROR: [3] SNESSolve line 3837 /opt/petsc/src/snes/interface/snes.c
> [3]PETSC ERROR: [3] TSStep_Alpha line 40 /opt/petsc/src/ts/impls/implicit/alpha/alpha.c
> [3]PETSC ERROR: [3] TSStep line 3081 /opt/petsc/src/ts/interface/ts.c
> [3]PETSC ERROR: [3] TSSolve line 3246 /opt/petsc/src/ts/interface/ts.c
> [3]PETSC ERROR: --------------------- MoFEM Error Message---------------------------------------------------------------------------
> [3]PETSC ERROR: MoFEM version 0.2.1
> [3]PETSC ERROR: MoFEM git commit id 837f38245270d65c22eafb638f023ea3476bc57a
> [3]PETSC ERROR: See http://userweb.eng.gla.ac.uk/lukasz.kaczmarczyk/MoFem/html/guidelines_bug_reporting.html for bug reporting.
> [3]PETSC ERROR: See http://userweb.eng.gla.ac.uk/lukasz.kaczmarczyk/MoFem/html/faq_and_bugs.html for trouble shooting.
> [3]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
> [3]PETSC ERROR: Signal received
> [3]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
> [3]PETSC ERROR: Petsc Development GIT revision: v3.5.3-2812-gc9542f9  GIT Date: 2015-05-03 22:34:15 +0300
> [3]PETSC ERROR: /Users/likask/MyBuild/um_debug/nonlinear_elasticity/nonlinear_dynamics on a arch-darwin-c-debug named dh23-181.eng.gla.ac.uk by likask Tue May  5 09:49:38 2015
> [3]PETSC ERROR: Configure options --with-mpi=1 --with-debugging=1 --download-fblaslapack=1 --download-superlu_dist=1 --download-metis=1 --download-parmetis=1 --download-hypre=1 --download-mumps=1 --download-scalapack=1 --download-blacs=1 --download-moab=1 --download-hdf5=1 --download-netcdf=/opt/petsc/netcdf-4.3.3.1.tar.gz --download-openmpi=1 --CXXFLAGS=-fno-vectorize -CFLAGS=-fno-vectorize --download-mumps=1 --download-ptscotch=1
> [2]PETSC ERROR: #1 User provided function() line 0 in  unknown file
> wnload-fblaslapack=1 --download-superlu_dist=1 --download-metis=1 --download-parmetis=1 --download-hypre=1 --download-mumps=1 --download-scalapack=1 --download-blacs=1 --download-moab=1 --download-hdf5=1 --download-netcdf=/opt/petsc/netcdf-4.3.3.1.tar.gz --download-openmpi=1 --CXXFLAGS=-fno-vectorize -CFLAGS=-fno-vectorize --download-mumps=1 --download-ptscotch=1
> [3]PETSC ERROR: #1 User provided function() line 0 in  unknown file
> [dh23-181.eng.gla.ac.uk:06691] 3 more processes have sent help message help-mpi-api.txt / mpi-abort
> [dh23-181.eng.gla.ac.uk:06691] Set MCA parameter "orte_base_help_aggregate" to 0 to see all help / error messages
> <configure.log>