<html><head><meta http-equiv="Content-Type" content="text/html; charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">I have never used gdb. <div class="">Using 0 as you suggested I got this output:<div class=""><br class=""></div><span class=""><font face="Menlo" size="1" class="">$PETSC_DIR/$PETSC_ARCH/bin/mpirun -n 2 examples/rosenbrock/rosenbrock optimize -start_in_debugger noxterm -debugger_nodes 0<br class="">** PETSc DEPRECATION WARNING ** : the option -debugger_nodes is deprecated as of version 3.14 and will be removed in a future release. Please use the option -debugger_ranks instead. (Silence this warning with -options_suppress_deprecated_warnings)<br class="">PETSC: Attaching gdb to examples/rosenbrock/rosenbrock of pid 3903 on srvulx13<br class="">GNU gdb (Ubuntu 7.7.1-0ubuntu5~14.04.3) 7.7.1<br class="">Copyright (C) 2014 Free Software Foundation, Inc.<br class="">License GPLv3+: GNU GPL version 3 or later <<a href="http://gnu.org/licenses/gpl.html" class="">http://gnu.org/licenses/gpl.html</a>><br class="">This is free software: you are free to change and redistribute it.<br class="">There is NO WARRANTY, to the extent permitted by law. Type "show copying"<br class="">and "show warranty" for details.<br class="">This GDB was configured as "x86_64-linux-gnu".<br class="">Type "show configuration" for configuration details.<br class="">For bug reporting instructions, please see:<br class=""><<a href="http://www.gnu.org/software/gdb/bugs/" class="">http://www.gnu.org/software/gdb/bugs/</a>>.<br class="">Find the GDB manual and other documentation resources online at:<br class=""><<a href="http://www.gnu.org/software/gdb/documentation/" class="">http://www.gnu.org/software/gdb/documentation/</a>>.<br class="">For help, type "help".<br class="">Type "apropos word" to search for commands related to "word"...<br class="">Reading symbols from examples/rosenbrock/rosenbrock...done.<br class="">Attaching to program: /home/fbrarda/cmdstan-petsc/examples/rosenbrock/rosenbrock, process 3903<br class="">Could not attach to process. If your uid matches the uid of the target<br class="">process, check the setting of /proc/sys/kernel/yama/ptrace_scope, or try<br class="">again as the root user. For more details, see /etc/sysctl.d/10-ptrace.conf<br class="">ptrace: Operation not permitted.<br class="">/home/fbrarda/cmdstan-petsc/3903: No such file or directory.<br class="">(gdb) method = optimize<br class=""> optimize<br class=""> algorithm = lbfgs (Default)<br class=""> lbfgs<br class="">method = optimize<br class=""> optimize<br class=""> algorithm = lbfgs (Default)<br class=""> lbfgs<br class=""> init_alpha = 0.001 (Default) init_alpha = 0.001 (Default)<br class=""> tol_obj = 9.9999999999999998e-13 (Default)<br class=""> tol_rel_obj = 10000 (Default)<br class=""> tol_grad = 1e-08 (Default)<br class=""> tol_rel_grad = 10000000 (Default)<br class=""> tol_obj = 9.9999999999999998e-13 (Default)<br class=""> tol_rel_obj = 10000 (Default)<br class=""><br class=""> tol_param = 1e-08 (Default)<br class=""> history_size = 5 (Default)<br class=""> tol_grad = 1e-08 (Default)<br class=""> iter = 2000 (Default)<br class=""> save_iterations = 0 (Default) tol_rel_grad = 10000000 (Default)<br class=""> tol_param = 1e-08 (Default)<br class=""> history_size = 5 (Default)<br class="">id = 0 (Default)<br class="">data<br class=""> file = (Default)<br class="">init = 2 (Default)<br class=""><br class=""> iter = 2000 (Default)<br class=""> save_iterations = 0 (Default)<br class="">random<br class=""> seed = 3666155654 (Default)<br class="">output<br class=""> file = output.csv (Default)<br class=""> diagnostic_file = (Default)id = 0 (Default)<br class="">data<br class=""> file = (Default)<br class="">init = 2 (Default)<br class=""><br class=""> refresh = 100 (Default)<br class=""><br class="">random<br class=""> seed = 3666155654 (Default)<br class="">output<br class=""> file = output.csv (Default)<br class=""> diagnostic_file = (Default)<br class=""> refresh = 100 (Default)<br class=""><br class="">Initial log joint probability = -158.559<br class=""> Iter log prob ||dx|| ||grad|| alpha alpha0 # evals Notes <br class=""> 12 -0.253535 0.000284499 0.0383658 0.001 0.001 46 LS failed, Hessian reset <br class=""> 13 -0.253535 0.000284499 0.0383658 6.528 0.001 111 LS failed, Hessian reset <br class="">Optimization terminated with error: <br class=""> Line search failed to achieve a sufficient decrease, no more progress can be made<br class="">[0]PETSC ERROR: PetscAbortErrorHandler: main() line 12 in src/cmdstan/main.cpp <br class=""> To prevent termination, change the error handler using PetscPushErrorHandler()<br class=""></font></span><span class=""><br class="">Using only 1 process the code works. </span></div><div class=""><span class=""><br class=""></span></div><div class=""><span class="">Francesco<br class=""><br class=""><blockquote type="cite" class="">Il giorno 24 feb 2021, alle ore 07:14, Barry Smith <<a href="mailto:bsmith@petsc.dev" class="">bsmith@petsc.dev</a>> ha scritto:<br class=""><br class=""><br class="">start_in_debugger noxterm -debugger_nodes 3<br class=""><br class="">Use -start_in_debugger noxterm -debugger_nodes 0<br class=""><br class="">when not opening windows for each debugger it is best to have the first rank associated with the tty as the debugger node<br class=""><br class=""><br class=""><br class=""><br class=""><br class=""><br class=""><blockquote type="cite" class="">On Feb 23, 2021, at 3:46 PM, Francesco Brarda <<a href="mailto:brardafrancesco@gmail.com" class="">brardafrancesco@gmail.com</a>> wrote:<br class=""><br class="">Using the command you suggested I got <br class=""><br class="">fbrarda@srvulx13:~/cmdstan-petsc$ $PETSC_DIR/$PETSC_ARCH/bin/mpirun -n 2 examples/rosenbrock/rosenbrock optimize -start_in_debugger noxterm -debugger_nodes 3<br class="">** PETSc DEPRECATION WARNING ** : the option -debugger_nodes is deprecated as of version 3.14 and will be removed in a future release. Please use the option -debugger_ranks instead. (Silence this warning with -options_suppress_deprecated_warnings)<br class="">method = optimize<br class=""> optimize<br class=""> algorithm = lbfgs (Default)<br class=""> lbfgs<br class="">method = optimize<br class=""> optimize<br class=""> algorithm = lbfgs (Default)<br class=""> lbfgs<br class=""> init_alpha = 0.001 (Default)<br class=""> tol_obj = 9.9999999999999998e-13 (Default)<br class=""> init_alpha = 0.001 (Default)<br class=""> tol_obj = 9.9999999999999998e-13 (Default)<br class=""> tol_rel_obj = 10000 (Default)<br class=""> tol_grad = 1e-08 (Default) tol_rel_obj = 10000 (Default)<br class=""> tol_grad = 1e-08 (Default)<br class=""> tol_rel_grad = 10000000 (Default)<br class=""><br class=""> tol_rel_grad = 10000000 (Default)<br class=""> tol_param = 1e-08 (Default) tol_param = 1e-08 (Default)<br class=""> history_size = 5 (Default)<br class=""> iter = 2000 (Default)<br class=""><br class=""> history_size = 5 (Default)<br class=""> iter = 2000 (Default)<br class=""> save_iterations = 0 (Default)<br class="">id = 0 (Default)<br class="">data<br class=""> save_iterations = 0 (Default)<br class="">id = 0 (Default)<br class="">data<br class=""> file = (Default)<br class=""> file = (Default)<br class="">init = 2 (Default)<br class="">random<br class=""> seed = 3623621468 (Default)<br class="">output<br class=""> file = output.csv (Default)init = 2 (Default)<br class="">random<br class=""> seed = 3623621468 (Default)<br class="">output<br class=""> file = output.csv (Default)<br class=""><br class=""> diagnostic_file = (Default)<br class=""> refresh = 100 (Default)<br class=""><br class=""> diagnostic_file = (Default)<br class=""> refresh = 100 (Default)<br class=""><br class="">Initial log joint probability = -195.984<br class=""> Iter log prob ||dx|| ||grad|| alpha alpha0 # evals Notes <br class=""> 10 -0.97101 0.00292919 1.65855 0.001 0.001 46 LS failed, Hessian reset <br class=""> 12 -0.483952 0.001316 1.18542 0.001 0.001 77 LS failed, Hessian reset <br class=""> 13 -0.477916 0.0118542 0.163518 0.01 0.001 106 LS failed, Hessian reset <br class="">[1]PETSC ERROR: #1 main() line 12 in src/cmdstan/main.cpp<br class="">[1]PETSC ERROR: PETSc Option Table entries:<br class="">[1]PETSC ERROR: -debugger_nodes 3<br class="">[1]PETSC ERROR: -start_in_debugger noxterm<br class="">[1]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint@mcs.anl.gov—————<br class=""> <br class="">And then it does not go further. With the -debugger_ranks suggested, the output is the same. What do you think, please?<br class="">I am using a cluster (one node, dual-socket system with twelve-core-CPUs), but when I do the ssh I do not use the -X flag, if that's what you mean.<br class=""><br class="">Thank you,<br class="">Francesco<br class=""><br class=""><br class=""><blockquote type="cite" class="">Il giorno 23 feb 2021, alle ore 21:59, Matthew Knepley <knepley@gmail.com> ha scritto:<br class=""><br class="">On Tue, Feb 23, 2021 at 3:55 PM Francesco Brarda <brardafrancesco@gmail.com> wrote:<br class="">Thank you for the quick response. <br class="">Sorry, you are right. Here is the complete output:<br class=""><br class="">fbrarda@srvulx13:~/cmdstan-petsc$ $PETSC_DIR/$PETSC_ARCH/bin/mpirun -n 2 examples/rosenbrock/rosenbrock optimize -start_in_debugger<br class="">PETSC: Attaching gdb to examples/rosenbrock/rosenbrock of pid 47803 on display :0.0 on machine srvulx13<br class="">PETSC: Attaching gdb to examples/rosenbrock/rosenbrock of pid 47804 on display :0.0 on machine srvulx13<br class="">xterm: Xt error: Can't open display: :0.0<br class="">xterm: DISPLAY is not set<br class="">xterm: Xt error: Can't open display: :0.0<br class="">xterm: DISPLAY is not set<br class=""><br class="">Do you have an Xserver running? If not, you can use<br class=""><br class=""> -start_in_debugger noxterm -debugger_nodes 3<br class=""><br class="">and try to get a stack trace from one node.<br class=""><br class=""> Thanks,<br class=""><br class=""> Matt<br class=""> <br class="">method = optimize<br class=""> optimize<br class=""> algorithm = lbfgs (Default)<br class=""> lbfgs<br class="">method = optimize<br class=""> optimize<br class=""> algorithm = lbfgs (Default)<br class=""> lbfgs<br class=""> init_alpha = 0.001 (Default)<br class=""> tol_obj = 9.9999999999999998e-13 (Default)<br class=""> tol_rel_obj = 10000 (Default)<br class=""> tol_grad = 1e-08 (Default)<br class=""> init_alpha = 0.001 (Default)<br class=""> tol_obj = 9.9999999999999998e-13 (Default)<br class=""> tol_rel_obj = 10000 (Default)<br class=""> tol_grad = 1e-08 (Default)<br class=""> tol_rel_grad = 10000000 (Default)<br class=""> tol_param = 1e-08 (Default)<br class=""> history_size = 5 (Default)<br class=""> tol_rel_grad = 10000000 (Default)<br class=""> tol_param = 1e-08 (Default)<br class=""> history_size = 5 (Default)<br class=""> iter = 2000 (Default)<br class=""> iter = 2000 (Default)<br class=""> save_iterations = 0 (Default)<br class="">id = 0 (Default)<br class="">data save_iterations = 0 (Default)<br class="">id = 0 (Default)<br class="">data<br class=""> file = (Default)<br class=""><br class=""> file = (Default)<br class="">init = 2 (Default)<br class="">random<br class=""> seed = 3585768430 (Default)<br class="">init = 2 (Default)<br class="">random<br class=""> seed = 3585768430 (Default)<br class="">output<br class=""> file = output.csv (Default)<br class="">output<br class=""> file = output.csv (Default)<br class=""> diagnostic_file = (Default)<br class=""> refresh = 100 (Default)<br class=""> diagnostic_file = (Default)<br class=""> refresh = 100 (Default)<br class=""><br class=""><br class="">Initial log joint probability = -731.444<br class=""> Iter log prob ||dx|| ||grad|| alpha alpha0 # evals Notes <br class="">[1]PETSC ERROR: PetscAbortErrorHandler: main() line 12 in src/cmdstan/main.cpp <br class=""> To prevent termination, change the error handler using PetscPushErrorHandler()<br class=""><br class="">===================================================================================<br class="">= BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES<br class="">= PID 47804 RUNNING AT srvulx13<br class="">= EXIT CODE: 134<br class="">= CLEANING UP REMAINING PROCESSES<br class="">= YOU CAN IGNORE THE BELOW CLEANUP MESSAGES<br class="">===================================================================================<br class="">YOUR APPLICATION TERMINATED WITH THE EXIT STRING: Aborted (signal 6)<br class="">This typically refers to a problem with your application.<br class="">Please see the FAQ page for debugging suggestions<br class=""><br class=""><br class=""><br class=""><br class=""><br class="">The code inside main.cpp is the following:<br class=""><br class="">#include <cmdstan/command.hpp><br class="">#include <stan/services/error_codes.hpp><br class=""><br class="">#include <petsc.h><br class=""><br class="">int main(int argc, char* argv[]) {<br class=""><br class=""> PetscErrorCode ierr;<br class=""> ierr = PetscInitialize(&argc, &argv, 0, 0);CHKERRQ(ierr);<br class=""><br class=""> try {<br class=""> ierr = cmdstan::command(argc, argv);CHKERRQ(ierr);<br class=""> } catch (const std::exception& e) {<br class=""> std::cout << e.what() << std::endl;<br class=""> ierr = stan::services::error_codes::SOFTWARE;CHKERRQ(ierr);<br class=""> }<br class=""><br class=""> ierr = PetscFinalize();CHKERRQ(ierr);<br class=""> return ierr;<br class="">}<br class=""><br class="">I highlighted the line 12. Although I read the page where the command PetscPushErrorHandler is explained and the example provided (src/ksp/ksp/tutorials/ex27.c), I do not understand how I should effectively use the command.<br class="">Should I change the entire try/catch with PetscPushErrorHandler(PetscIgnoreErrorHandler,NULL); ?<br class=""><br class="">Best,<br class="">Francesco<br class=""><br class=""><br class=""><blockquote type="cite" class="">Il giorno 23 feb 2021, alle ore 11:54, Matthew Knepley <knepley@gmail.com> ha scritto:<br class=""><br class="">On Tue, Feb 23, 2021 at 3:54 AM Francesco Brarda <brardafrancesco@gmail.com> wrote:<br class="">Hi!<br class=""><br class="">I am very new to the PETSc world. I am working with a GitHub repo that uses PETSc together with Stan (a statistics open source software), here you can find the discussion. <br class="">It has been defined a functor to convert EigenVector to PetscVec and viceversa, both sequentially and in parallel. <br class="">The file using these functions does the conversions with the sequential setting. I changed to those using MPI, that is from EigenVectorToPetscVecSeq to EigenVectorToPetscVecMPI and so on because I want to evaluate the scaling.<br class="">Running the example with mpirun -n 5 examples/rosenbrock/rosenbrock optimize in the debug mode I get the error Caught signal number 11 SEGV. I therefore used the option -start_in_debugger and I get the following:<br class=""><br class="">For some reason, the -start_in_debuggger option is not being seen. Are you showing all the output? Once the debugger is attached,<br class="">you run the program (conr) and then when you hit the SEGV you get a stack trace (where).<br class=""><br class=""> THanks,<br class=""><br class=""> Matt<br class=""> <br class="">[2]PETSC ERROR: ------------------------------------------------------------------------<br class="">[2]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range<br class="">[2]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger<br class="">[2]PETSC ERROR: or see https://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind<br class="">[2]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors<br class="">[2]PETSC ERROR: likely location of problem given in stack below<br class="">[2]PETSC ERROR: --------------------- Stack Frames ------------------------------------<br class="">[2]PETSC ERROR: Note: The EXACT line numbers in the stack are not available,<br class="">[2]PETSC ERROR: INSTEAD the line number of the start of the function<br class="">[2]PETSC ERROR: is given.<br class="">[3]PETSC ERROR: ------------------------------------------------------------------------<br class="">[3]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range<br class="">[3]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger<br class="">[3]PETSC ERROR: or see https://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind<br class="">[3]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors<br class="">[3]PETSC ERROR: likely location of problem given in stack below<br class="">[3]PETSC ERROR: --------------------- Stack Frames ------------------------------------<br class="">[3]PETSC ERROR: Note: The EXACT line numbers in the stack are not available,<br class="">[3]PETSC ERROR: INSTEAD the line number of the start of the function<br class="">[3]PETSC ERROR: is given.<br class="">[3]PETSC ERROR: PetscAbortErrorHandler: User provided function() line 0 in unknown file (null)<br class=""> To prevent termination, change the error handler using PetscPushErrorHandler()<br class="">[2]PETSC ERROR: PetscAbortErrorHandler: User provided function() line 0 in unknown file (null)<br class=""> To prevent termination, change the error handler using PetscPushErrorHandler()<br class=""><br class="">===================================================================================<br class="">= BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES<br class="">= PID 22939 RUNNING AT srvulx13<br class="">= EXIT CODE: 134<br class="">= CLEANING UP REMAINING PROCESSES<br class="">= YOU CAN IGNORE THE BELOW CLEANUP MESSAGES<br class="">===================================================================================<br class="">YOUR APPLICATION TERMINATED WITH THE EXIT STRING: Aborted (signal 6)<br class="">This typically refers to a problem with your application.<br class="">Please see the FAQ page for debugging suggestions<br class=""><br class="">I read the documentation regarding the PetscAbortErrorHandler, but I do not know where should I use it. How can I solve the problem? <br class="">I hope I have been clear enough.<br class="">Attached you can find also my configure.log and make.log files.<br class=""><br class="">Best,<br class="">Francesco<br class=""><br class=""><br class=""><br class=""><br class=""><br class="">-- <br class="">What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br class="">-- Norbert Wiener<br class=""><br class="">https://www.cse.buffalo.edu/~knepley/<br class=""></blockquote><br class=""><br class=""><br class="">-- <br class="">What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br class="">-- Norbert Wiener<br class=""><br class="">https://www.cse.buffalo.edu/~knepley/<br class=""></blockquote><br class=""></blockquote><br class=""></blockquote><br class=""></span></div></body></html>