<div dir="ltr">We had a previous error with <span style="font-size:12.8px">pdgssvx in SuperLU I think. Maybe searching petsc-maint would get it?</span><div><span style="font-size:12.8px"><br></span></div><div><span style="font-size:12.8px"> Matt</span></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Mon, Oct 16, 2017 at 12:21 PM, Mark Adams <span dir="ltr"><<a href="mailto:mfadams@lbl.gov" target="_blank">mfadams@lbl.gov</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div>I just ran this and have a little bit of a stack trace. This is on my laptop and MPI can be a little flaky here (eg, IBarrier does not work). I am going to move to Cori soon and I will try to reproduce this.</div><div>Thanks,</div><div><br></div><div>==68941== at 0x103A66AA8: MPIR_Process_status (mpiimpl.h:4394)</div><div>==68941== by 0x103A6852F: MPIC_Waitall (helper_fns.c:774)</div><div>==68941== by 0x1038ECE88: MPIR_Alltoallv_intra (alltoallv.c:194)</div><div>==68941== by 0x1038ED7F9: MPIR_Alltoallv (alltoallv.c:339)</div><div>==68941== by 0x1038EDA53: MPIR_Alltoallv_impl (alltoallv.c:376)</div><div>==68940== at 0x103A66AA8: MPIR_Process_status (mpiimpl.h:4394)</div><div>==68940== by 0x103A6852F: MPIC_Waitall (helper_fns.c:774)</div><div>==68940== by 0x1038ECE88: MPIR_Alltoallv_intra (alltoallv.c:194)</div><div>==68940== by 0x1038ED7F9: MPIR_Alltoallv (alltoallv.c:339)</div><div>==68940== by 0x1038EDA53: MPIR_Alltoallv_impl (alltoallv.c:376)</div><div>==68940== by 0x103719112: MPI_Alltoallv (alltoallv.c:527)</div><div>==68940== by 0x10238B87D: pdCompRow_loc_to_CompCol_<wbr>global (in /Users/markadams/Codes/petsc/<wbr>arch-macosx-gnu-g/lib/<wbr>libsuperlu_dist.5.1.3.dylib)</div><div>==68940== by 0x1023800CB: pdgssvx (in /Users/markadams/Codes/petsc/<wbr>arch-macosx-gnu-g/lib/<wbr>libsuperlu_dist.5.1.3.dylib)</div><div>==68940== by 0x100AB92DB: MatLUFactorNumeric_SuperLU_<wbr>DIST (superlu_dist.c:429)</div><div><br></div></div><div class="HOEnZb"><div class="h5"><div class="gmail_extra"><br><div class="gmail_quote">On Mon, Oct 16, 2017 at 12:05 PM, Xiaoye S. Li <span dir="ltr"><<a href="mailto:xsli@lbl.gov" target="_blank">xsli@lbl.gov</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Mark,<div>Is it possible to get the line number? </div><div>For example, the first failure is</div><span><div><br></div><div><div>==63582== Conditional jump or move depends on uninitialised value(s)</div><div>==63582== at 0x103A5FAA8: MPIR_Process_status (mpiimpl.h:4394)</div><div>==63582== by 0x103A6152F: MPIC_Waitall (helper_fns.c:774)</div><div>==63582== by 0x1038E2A34: MPIR_Alltoall_intra (alltoall.c:369)</div><div>==63582== by 0x1038E35E1: MPIR_Alltoall (alltoall.c:564)</div><div>==63582== by 0x1038E37E6: MPIR_Alltoall_impl (alltoall.c:599)</div><div>==63582== by 0x1037106AD: MPI_Alltoall (alltoall.c:722)</div><div>==63582== by 0x10236EA7C: static_schedule (in /Users/markadams/Codes/petsc/a<wbr>rch-macosx-gnu-g/lib/libsuperl<wbr>u_dist.5.1.3.dylib)</div></div><div><br></div></span><div>I checked all the MPI_alltoall in static_schedule() routine, I don't see any problem.</div><div><br></div><div>Sherry</div><div><div class="m_-1354519293765017474h5"><div><br></div><div><br></div><div class="gmail_extra"><br><div class="gmail_quote">On Mon, Oct 16, 2017 at 7:21 AM, Mark Adams <span dir="ltr"><<a href="mailto:mfadams@lbl.gov" target="_blank">mfadams@lbl.gov</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>FYI, I get this error on one processor with SuperLU under valgrind. Could this just be a valgrind issue?</div><div><br></div><div>Mark</div><div><br></div><div>/Users/markadams/Codes/petsc/a<wbr>rch-macosx-gnu-g/bin/mpiexec -n 1 valgrind --dsymutil=yes --leak-check=no --gen-suppressions=no --num-callers=20 --error-limit=no ./ex48 -debug 2 -dim 2 -dm_refine 3 -ts_monitor -implicit true -ts_type beuler -pc_type lu -pc_factor_mat_solver_package superlu_dist -ksp_type preonly -snes_monitor -snes_rtol 1.e-10 -snes_stol 1.e-10 -snes_converged_reason -snes_atol 1.e-18 -snes_converged_reason -petscspace_order 2 -petscspace_poly_tensor -ts_max_steps 1 -ts_dt 1.e-3 -eps 1.e-12 -eta 0.001 -ves 0.005 -beta 0.01 -mu 0.0002 -dm_view hdf5:sol.h5 -vec_view hdf5:sol.h5::append -dm_plex_periodic_cut -y_periodicity PERIODIC -cells 2,4 -Jop 4.99 -line_dir 1,1 -line_coord 3.14159265359,1.57079632679 -real_view :u.m:ascii_matlab -fft_view :spectra.m:ascii_matlab</div><div>==63582== Memcheck, a memory error detector</div><div>==63582== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.</div><div>==63582== Using Valgrind-3.13.0 and LibVEX; rerun with -h for copyright info</div><div>==63582== Command: ./ex48 -debug 2 -dim 2 -dm_refine 3 -ts_monitor -implicit true -ts_type beuler -pc_type lu -pc_factor_mat_solver_package superlu_dist -ksp_type preonly -snes_monitor -snes_rtol 1.e-10 -snes_stol 1.e-10 -snes_converged_reason -snes_atol 1.e-18 -snes_converged_reason -petscspace_order 2 -petscspace_poly_tensor -ts_max_steps 1 -ts_dt 1.e-3 -eps 1.e-12 -eta 0.001 -ves 0.005 -beta 0.01 -mu 0.0002 -dm_view hdf5:sol.h5 -vec_view hdf5:sol.h5::append -dm_plex_periodic_cut -y_periodicity PERIODIC -cells 2,4 -Jop 4.99 -line_dir 1,1 -line_coord 3.14159265359,1.57079632679 -real_view :u.m:ascii_matlab -fft_view :spectra.m:ascii_matlab</div><div>==63582== </div><div>==63582== Syscall param msg-><a href="http://desc.port.name" target="_blank">desc.port.name</a> points to uninitialised byte(s)</div><div>==63582== at 0x103FE134A: mach_msg_trap (in /usr/lib/system/libsystem_kern<wbr>el.dylib)</div><div>==63582== by 0x103FE0796: mach_msg (in /usr/lib/system/libsystem_kern<wbr>el.dylib)</div><div>==63582== by 0x103FDA485: task_set_special_port (in /usr/lib/system/libsystem_kern<wbr>el.dylib)</div><div>==63582== by 0x10817810E: _os_trace_create_debug_control<wbr>_port (in /usr/lib/system/libsystem_trac<wbr>e.dylib)</div><div>==63582== by 0x108178458: _libtrace_init (in /usr/lib/system/libsystem_trac<wbr>e.dylib)</div><div>==63582== by 0x1036119DF: libSystem_initializer (in /usr/lib/libSystem.B.dylib)</div><div>==63582== by 0x100034A1A: ImageLoaderMachO::doModInitFun<wbr>ctions(ImageLoader::LinkContex<wbr>t const&) (in /usr/lib/dyld)</div><div>==63582== by 0x100034C1D: ImageLoaderMachO::doInitializa<wbr>tion(ImageLoader::LinkContext const&) (in /usr/lib/dyld)</div><div>==63582== by 0x1000304A9: ImageLoader::recursiveInitiali<wbr>zation(ImageLoader::LinkContex<wbr>t const&, unsigned int, char const*, ImageLoader::InitializerTiming<wbr>List&, ImageLoader::UninitedUpwards&) (in /usr/lib/dyld)</div><div>==63582== by 0x100030440: ImageLoader::recursiveInitiali<wbr>zation(ImageLoader::LinkContex<wbr>t const&, unsigned int, char const*, ImageLoader::InitializerTiming<wbr>List&, ImageLoader::UninitedUpwards&) (in /usr/lib/dyld)</div><div>==63582== by 0x10002F523: ImageLoader::processInitialize<wbr>rs(ImageLoader::LinkContext const&, unsigned int, ImageLoader::InitializerTiming<wbr>List&, ImageLoader::UninitedUpwards&) (in /usr/lib/dyld)</div><div>==63582== by 0x10002F5B8: ImageLoader::runInitializers(I<wbr>mageLoader::LinkContext const&, ImageLoader::InitializerTiming<wbr>List&) (in /usr/lib/dyld)</div><div>==63582== by 0x100021433: dyld::initializeMainExecutable<wbr>() (in /usr/lib/dyld)</div><div>==63582== by 0x1000258C5: dyld::_main(macho_header const*, unsigned long, int, char const**, char const**, char const**, unsigned long*) (in /usr/lib/dyld)</div><div>==63582== by 0x100020248: dyldbootstrap::start(macho_hea<wbr>der const*, int, char const**, long, macho_header const*, unsigned long*) (in /usr/lib/dyld)</div><div>==63582== by 0x100020035: _dyld_start (in /usr/lib/dyld)</div><div>==63582== by 0x3E: ???</div><div>==63582== by 0x1080A84C2: ???</div><div>==63582== by 0x1080A84C9: ???</div><div>==63582== by 0x1080A84D0: ???</div><div>==63582== Address 0x1080a60fc is on thread 1's stack</div><div>==63582== in frame #2, created by task_set_special_port (???:)</div><div>==63582== </div><div>--63582-- UNKNOWN mach_msg unhandled MACH_SEND_TRAILER option</div><div>--63582-- UNKNOWN mach_msg unhandled MACH_SEND_TRAILER option (repeated 2 times)</div><div>--63582-- UNKNOWN mach_msg unhandled MACH_SEND_TRAILER option (repeated 4 times)</div><div>Jop=4.99</div><div>DeltaPrime=1.81627</div><div>eta=0.001</div><div>beta=0.01</div><div>mu=0.0002</div><div>ves=0.005</div><div>==63582== Warning: invalid file descriptor -1 in syscall read()</div><div>0) total perturbed mass = 0.</div><div>0 TS dt 0.001 time 0.</div><div> 0 SNES Function norm 5.917661770415e-01 </div><div>==63582== Conditional jump or move depends on uninitialised value(s)</div><div>==63582== at 0x103A5FAA8: MPIR_Process_status (mpiimpl.h:4394)</div><div>==63582== by 0x103A6152F: MPIC_Waitall (helper_fns.c:774)</div><div>==63582== by 0x1038E2A34: MPIR_Alltoall_intra (alltoall.c:369)</div><div>==63582== by 0x1038E35E1: MPIR_Alltoall (alltoall.c:564)</div><div>==63582== by 0x1038E37E6: MPIR_Alltoall_impl (alltoall.c:599)</div><div>==63582== by 0x1037106AD: MPI_Alltoall (alltoall.c:722)</div><div>==63582== by 0x10236EA7C: static_schedule (in /Users/markadams/Codes/petsc/a<wbr>rch-macosx-gnu-g/lib/libsuperl<wbr>u_dist.5.1.3.dylib)</div><div>==63582== by 0x10239923C: pdgstrf (in /Users/markadams/Codes/petsc/a<wbr>rch-macosx-gnu-g/lib/libsuperl<wbr>u_dist.5.1.3.dylib)</div><div>==63582== by 0x10237D696: pdgssvx_ABglobal (in /Users/markadams/Codes/petsc/a<wbr>rch-macosx-gnu-g/lib/libsuperl<wbr>u_dist.5.1.3.dylib)</div><div>==63582== by 0x100AB1F02: MatLUFactorNumeric_SuperLU_DIS<wbr>T (superlu_dist.c:423)</div><div>==63582== by 0x10053AD98: MatLUFactorNumeric (matrix.c:3039)</div><div>==63582== by 0x1012075CD: PCSetUp_LU (lu.c:131)</div><div>==63582== by 0x10134D65B: PCSetUp (precon.c:924)</div><div>==63582== by 0x101496E11: KSPSetUp (itfunc.c:378)</div><div>==63582== by 0x101499143: KSPSolve (itfunc.c:609)</div><div>==63582== by 0x1015F9410: SNESSolve_NEWTONLS (ls.c:224)</div><div>==63582== by 0x101574290: SNESSolve (snes.c:4106)</div><div>==63582== by 0x10179B43C: TS_SNESSolve (theta.c:176)</div><div>==63582== by 0x10178F7CE: TSStep_Theta (theta.c:216)</div><div>==63582== by 0x1016C1D62: TSStep (ts.c:4120)</div><div>==63582== </div><div>==63582== Conditional jump or move depends on uninitialised value(s)</div><div>==63582== at 0x103A5FAA8: MPIR_Process_status (mpiimpl.h:4394)</div><div>==63582== by 0x103A6152F: MPIC_Waitall (helper_fns.c:774)</div><div>==63582== by 0x1038E5E88: MPIR_Alltoallv_intra (alltoallv.c:194)</div><div>==63582== by 0x1038E67F9: MPIR_Alltoallv (alltoallv.c:339)</div><div>==63582== by 0x1038E6A53: MPIR_Alltoallv_impl (alltoallv.c:376)</div><div>==63582== by 0x103712112: MPI_Alltoallv (alltoallv.c:527)</div><div>==63582== by 0x10236ECF1: static_schedule (in /Users/markadams/Codes/petsc/a<wbr>rch-macosx-gnu-g/lib/libsuperl<wbr>u_dist.5.1.3.dylib)</div><div>==63582== by 0x10239923C: pdgstrf (in /Users/markadams/Codes/petsc/a<wbr>rch-macosx-gnu-g/lib/libsuperl<wbr>u_dist.5.1.3.dylib)</div><div>==63582== by 0x10237D696: pdgssvx_ABglobal (in /Users/markadams/Codes/petsc/a<wbr>rch-macosx-gnu-g/lib/libsuperl<wbr>u_dist.5.1.3.dylib)</div><div>==63582== by 0x100AB1F02: MatLUFactorNumeric_SuperLU_DIS<wbr>T (superlu_dist.c:423)</div><div>==63582== by 0x10053AD98: MatLUFactorNumeric (matrix.c:3039)</div><div>==63582== by 0x1012075CD: PCSetUp_LU (lu.c:131)</div><div>==63582== by 0x10134D65B: PCSetUp (precon.c:924)</div><div>==63582== by 0x101496E11: KSPSetUp (itfunc.c:378)</div><div>==63582== by 0x101499143: KSPSolve (itfunc.c:609)</div><div>==63582== by 0x1015F9410: SNESSolve_NEWTONLS (ls.c:224)</div><div>==63582== by 0x101574290: SNESSolve (snes.c:4106)</div><div>==63582== by 0x10179B43C: TS_SNESSolve (theta.c:176)</div><div>==63582== by 0x10178F7CE: TSStep_Theta (theta.c:216)</div><div><div>==63582== by 0x1016C1D62: TSStep (ts.c:4120)</div><div>==63582== </div><div>==63582== Thread 2:</div><div>==63582== Invalid read of size 4</div><div>==63582== at 0x10814A2B1: _pthread_wqthread (in /usr/lib/system/libsystem_pthr<wbr>ead.dylib)</div><div>==63582== by 0x10814A07C: start_wqthread (in /usr/lib/system/libsystem_pthr<wbr>ead.dylib)</div><div>==63582== Address 0x18 is not stack'd, malloc'd or (recently) free'd</div><div>==63582== </div><div>==63582== Invalid read of size 8</div><div>==63582== at 0x1081489D6: pthread_getspecific (in /usr/lib/system/libsystem_pthr<wbr>ead.dylib)</div><div>==63582== by 0x100286A5B: PetscVSNPrintf (mprint.c:132)</div><div>==63582== by 0x1002871A3: PetscVFPrintfDefault (mprint.c:241)</div><div>==63582== by 0x10028A1E6: PetscFPrintf (mprint.c:546)</div><div>==63582== by 0x1002A1BE9: PetscErrorPrintfDefault (errtrace.c:114)</div><div>==63582== by 0x1002A3C5D: PetscSignalHandlerDefault (signal.c:135)</div><div>==63582== by 0x1002A4A79: PetscSignalHandler_Private (signal.c:47)</div><div>==63582== by 0x25805BDBD: ???</div><div>==63582== by 0x10814A07C: start_wqthread (in /usr/lib/system/libsystem_pthr<wbr>ead.dylib)</div><div>==63582== Address 0x50 is not stack'd, malloc'd or (recently) free'd</div><div>==63582== </div><div>==63582== </div><div>==63582== Process terminating with default action of signal 11 (SIGSEGV)</div><div>==63582== Access not within mapped region at address 0x50</div><div>==63582== at 0x1081489D6: pthread_getspecific (in /usr/lib/system/libsystem_pthr<wbr>ead.dylib)</div><div>==63582== by 0x100286A5B: PetscVSNPrintf (mprint.c:132)</div><div>==63582== by 0x1002871A3: PetscVFPrintfDefault (mprint.c:241)</div><div>==63582== by 0x10028A1E6: PetscFPrintf (mprint.c:546)</div><div>==63582== by 0x1002A1BE9: PetscErrorPrintfDefault (errtrace.c:114)</div><div>==63582== by 0x1002A3C5D: PetscSignalHandlerDefault (signal.c:135)</div><div>==63582== by 0x1002A4A79: PetscSignalHandler_Private (signal.c:47)</div><div>==63582== by 0x25805BDBD: ???</div><div>==63582== by 0x10814A07C: start_wqthread (in /usr/lib/system/libsystem_pthr<wbr>ead.dylib)</div><div>==63582== If you believe this happened as a result of a stack</div><div>==63582== overflow in your program's main thread (unlikely but</div><div>==63582== possible), you can try to increase the size of the</div><div>==63582== main thread stack using the --main-stacksize= flag.</div><div>==63582== The main thread stack size used in this run was 67104768.</div><div><br></div><div>valgrind: m_scheduler/scheduler.c:881 (void run_thread_for_a_while(HWord *, Int *, ThreadId, HWord, Bool)): Assertion 'VG_(stats__n_xindirs_32) == 0' failed.</div><div><br></div><div>host stacktrace:</div><div>==63582== at 0x25804121C: ???</div><div>==63582== by 0x258041587: ???</div><div>==63582== by 0x25804156A: ???</div><div>==63582== by 0x2580BB25F: ???</div><div>==63582== by 0x2580B95EA: ???</div><div>==63582== by 0x2580CA83B: ???</div><div>==63582== by 0x2580CAAF8: ???</div><div><br></div><div>sched status:</div><div> running_tid=3</div><div><br></div><div>Thread 1: status = VgTs_Yielding (lwpid 771)</div><div>==63582== at 0x10239F9DE: dscatter_u (in /Users/markadams/Codes/petsc/a<wbr>rch-macosx-gnu-g/lib/libsuperl<wbr>u_dist.5.1.3.dylib)</div><div>==63582== by 0x10239EF4F: pdgstrf (in /Users/markadams/Codes/petsc/a<wbr>rch-macosx-gnu-g/lib/libsuperl<wbr>u_dist.5.1.3.dylib)</div><div>==63582== by 0x10237D696: pdgssvx_ABglobal (in /Users/markadams/Codes/petsc/a<wbr>rch-macosx-gnu-g/lib/libsuperl<wbr>u_dist.5.1.3.dylib)</div><div>==63582== by 0x100AB1F02: MatLUFactorNumeric_SuperLU_DIS<wbr>T (superlu_dist.c:423)</div><div>==63582== by 0x10053AD98: MatLUFactorNumeric (matrix.c:3039)</div><div>==63582== by 0x1012075CD: PCSetUp_LU (lu.c:131)</div><div>==63582== by 0x10134D65B: PCSetUp (precon.c:924)</div><div>==63582== by 0x101496E11: KSPSetUp (itfunc.c:378)</div><div>==63582== by 0x101499143: KSPSolve (itfunc.c:609)</div><div>==63582== by 0x1015F9410: SNESSolve_NEWTONLS (ls.c:224)</div><div>==63582== by 0x101574290: SNESSolve (snes.c:4106)</div><div>==63582== by 0x10179B43C: TS_SNESSolve (theta.c:176)</div><div>==63582== by 0x10178F7CE: TSStep_Theta (theta.c:216)</div><div>==63582== by 0x1016C1D62: TSStep (ts.c:4120)</div><div>==63582== by 0x1016C56A3: TSSolve (ts.c:4374)</div><div>==63582== by 0x100004E0E: main (ex48.c:1061)</div><div><br></div><div>Thread 2: status = VgTs_Yielding (lwpid 4099)</div><div>==63582== at 0x1081489D6: pthread_getspecific (in /usr/lib/system/libsystem_pthr<wbr>ead.dylib)</div><div>==63582== by 0x100286A5B: PetscVSNPrintf (mprint.c:132)</div><div>==63582== by 0x1002871A3: PetscVFPrintfDefault (mprint.c:241)</div><div>==63582== by 0x10028A1E6: PetscFPrintf (mprint.c:546)</div><div>==63582== by 0x1002A1BE9: PetscErrorPrintfDefault (errtrace.c:114)</div><div>==63582== by 0x1002A3C5D: PetscSignalHandlerDefault (signal.c:135)</div><div>==63582== by 0x1002A4A79: PetscSignalHandler_Private (signal.c:47)</div><div>==63582== by 0x25805BDBD: ???</div><div>==63582== by 0x10814A07C: start_wqthread (in /usr/lib/system/libsystem_pthr<wbr>ead.dylib)</div><div><br></div><div>Thread 3: status = VgTs_Runnable (lwpid 3843)</div><div>==63582== at 0x10814A070: start_wqthread (in /usr/lib/system/libsystem_pthr<wbr>ead.dylib)</div><div><br></div><div><br></div><div>Note: see also the FAQ in the source distribution.</div><div>It contains workarounds to several common problems.</div><div>In particular, if Valgrind aborted or crashed after</div><div>identifying problems in your program, there's a good chance</div><div>that fixing those problems will prevent Valgrind aborting or</div><div>crashing, especially if it happened in m_mallocfree.c.</div></div><div><br></div></div>
</blockquote></div><br></div></div></div></div>
</blockquote></div><br></div>
</div></div></blockquote></div><br><br clear="all"><div><br></div>-- <br><div class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>-- Norbert Wiener</div><div><br></div><div><a href="http://www.caam.rice.edu/~mk51/" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br></div></div></div></div></div>
</div>