Sender: LSF System <lavaadmin@compute-00-02>
Subject: Job 2323: <Backstp> Exited

Job <Backstp> was submitted from host <cmtl> by user <harish>.
Job was executed on host(s) <8*compute-00-02>, in queue <normal>, as user <harish>.
                            <8*compute-00-03>
</home/harish> was used as the home directory.
</home/harish/nek5_examples/3D_bfs/interpolate_test> was used as the working directory.
Started at Thu Apr  8 19:15:55 2010
Results reported at Thu Apr  8 19:16:25 2010

Your job looked like:

------------------------------------------------------------
# LSBATCH: User input
#BSUB -n 16 
##BSUB -m 'compute-00-00'
##BSUB -m 'compute-00-00 compute-00-01 compute-00-02 compute-00-03'
#BSUB -m 'compute-00-02 compute-00-03'
#BSUB -o Backstp.%J.o   
                        
#BSUB -J Backstp   
# End of embedded BSUB options
#

openmpi-mpirun -np 16 ./nek5000 > logfile # Run executable

------------------------------------------------------------

Exited with exit code 139.

Resource usage summary:

    CPU time   :      0.10 sec.

The output (if any) follows:

[compute-00-03:25107] *** Process received signal ***
[compute-00-03:25107] Signal: Segmentation fault (11)
[compute-00-03:25107] Signal code: Invalid permissions (2)
[compute-00-03:25107] Failing at address: 0x5f2886
[compute-00-03:25109] *** Process received signal ***
[compute-00-03:25109] Signal: Segmentation fault (11)
[compute-00-03:25109] Signal code: Invalid permissions (2)
[compute-00-03:25109] Failing at address: 0x5f2886
[compute-00-02:07648] *** Process received signal ***
[compute-00-02:07648] Signal: Segmentation fault (11)
[compute-00-02:07648] Signal code: Invalid permissions (2)
[compute-00-02:07648] Failing at address: 0x5f2886
[compute-00-02:07648] [ 0] /lib64/libpthread.so.0 [0x372640e4c0]
[compute-00-02:07648] [ 1] ./nek5000(chcopy_+0x38) [0x488768]
[compute-00-02:07648] [ 2] ./nek5000(exitti_+0xd6) [0x5cd756]
[compute-00-02:07648] [ 3] ./nek5000(set_conv_char_+0x136) [0x4b8596]
[compute-00-02:07648] [ 4] ./nek5000(setup_convect_+0xa9) [0x4b9d29]
[compute-00-02:07648] [ 5] ./nek5000(setics_+0xd0f) [0x4839bf]
[compute-00-02:07648] [ 6] ./nek5000(nek_init_+0x25a) [0x41648a]
[compute-00-02:07648] [ 7] ./nek5000(MAIN__+0x1a) [0x415bba]
[compute-00-02:07648] [ 8] ./nek5000(main+0x2c) [0x5eb8ac]
[compute-00-02:07648] [ 9] /lib64/libc.so.6(__libc_start_main+0xf4) [0x3725c1d974]
[compute-00-02:07648] [10] ./nek5000 [0x4159c9]
[compute-00-02:07648] *** End of error message ***
[compute-00-03:25110] *** Process received signal ***
[compute-00-03:25110] Signal: Segmentation fault (11)
[compute-00-03:25110] Signal code: Invalid permissions (2)
[compute-00-03:25110] Failing at address: 0x5f2886
[compute-00-02:07649] *** Process received signal ***
[compute-00-02:07649] Signal: Segmentation fault (11)
[compute-00-02:07649] Signal code: Invalid permissions (2)
[compute-00-02:07649] Failing at address: 0x5f2886
[compute-00-02:07649] [ 0] /lib64/libpthread.so.0 [0x372640e4c0]
[compute-00-02:07649] [ 1] ./nek5000(chcopy_+0x38) [0x488768]
[compute-00-02:07649] [ 2] ./nek5000(exitti_+0xd6) [0x5cd756]
[compute-00-02:07649] [ 3] ./nek5000(set_conv_char_+0x136) [0x4b8596]
[compute-00-02:07649] [ 4] ./nek5000(setup_convect_+0xa9) [0x4b9d29]
[compute-00-02:07649] [ 5] ./nek5000(setics_+0xd0f) [0x4839bf]
[compute-00-02:07649] [ 6] ./nek5000(nek_init_+0x25a) [0x41648a]
[compute-00-02:07649] [ 7] ./nek5000(MAIN__+0x1a) [0x415bba]
[compute-00-02:07649] [ 8] ./nek5000(main+0x2c) [0x5eb8ac]
[compute-00-02:07649] [ 9] /lib64/libc.so.6(__libc_start_main+0xf4) [0x3725c1d974]
[compute-00-02:07649] [10] ./nek5000 [0x4159c9]
[compute-00-02:07649] *** End of error message ***
[compute-00-03:25111] *** Process received signal ***
[compute-00-03:25111] Signal: Segmentation fault (11)
[compute-00-03:25111] Signal code: Invalid permissions (2)
[compute-00-03:25111] Failing at address: 0x5f2886
[compute-00-02:07650] *** Process received signal ***
[compute-00-02:07650] Signal: Segmentation fault (11)
[compute-00-02:07650] Signal code: Invalid permissions (2)
[compute-00-02:07650] Failing at address: 0x5f2886
[compute-00-02:07650] [ 0] /lib64/libpthread.so.0 [0x372640e4c0]
[compute-00-02:07650] [ 1] ./nek5000(chcopy_+0x38) [0x488768]
[compute-00-02:07650] [ 2] ./nek5000(exitti_+0xd6) [0x5cd756]
[compute-00-02:07650] [ 3] ./nek5000(set_conv_char_+0x136) [0x4b8596]
[compute-00-02:07650] [ 4] ./nek5000(setup_convect_+0xa9) [0x4b9d29]
[compute-00-02:07650] [ 5] ./nek5000(setics_+0xd0f) [0x4839bf]
[compute-00-02:07650] [ 6] ./nek5000(nek_init_+0x25a) [0x41648a]
[compute-00-02:07650] [ 7] ./nek5000(MAIN__+0x1a) [0x415bba]
[compute-00-02:07650] [ 8] ./nek5000(main+0x2c) [0x5eb8ac]
[compute-00-02:07650] [ 9] /lib64/libc.so.6(__libc_start_main+0xf4) [0x3725c1d974]
[compute-00-02:07650] [10] ./nek5000 [0x4159c9]
[compute-00-02:07650] *** End of error message ***
[compute-00-03:25112] *** Process received signal ***
[compute-00-03:25112] Signal: Segmentation fault (11)
[compute-00-03:25112] Signal code: Invalid permissions (2)
[compute-00-03:25112] Failing at address: 0x5f2886
[compute-00-02:07651] *** Process received signal ***
[compute-00-02:07651] Signal: Segmentation fault (11)
[compute-00-02:07651] Signal code: Invalid permissions (2)
[compute-00-02:07651] Failing at address: 0x5f2886
[compute-00-03:25113] *** Process received signal ***
[compute-00-03:25113] Signal: Segmentation fault (11)
[compute-00-03:25113] Signal code: Invalid permissions (2)
[compute-00-03:25113] Failing at address: 0x5f2886
[compute-00-02:07652] *** Process received signal ***
[compute-00-02:07652] Signal: Segmentation fault (11)
[compute-00-02:07652] Signal code: Invalid permissions (2)
[compute-00-02:07652] Failing at address: 0x5f2886
[compute-00-02:07652] [ 0] /lib64/libpthread.so.0 [0x372640e4c0]
[compute-00-02:07652] [ 1] ./nek5000(chcopy_+0x38) [0x488768]
[compute-00-02:07652] [ 2] ./nek5000(exitti_+0xd6) [0x5cd756]
[compute-00-02:07652] [ 3] ./nek5000(set_conv_char_+0x136) [0x4b8596]
[compute-00-02:07652] [ 4] ./nek5000(setup_convect_+0xa9) [0x4b9d29]
[compute-00-02:07652] [ 5] ./nek5000(setics_+0xd0f) [0x4839bf]
[compute-00-02:07652] [ 6] ./nek5000(nek_init_+0x25a) [0x41648a]
[compute-00-02:07652] [ 7] ./nek5000(MAIN__+0x1a) [0x415bba]
[compute-00-02:07652] [ 8] ./nek5000(main+0x2c) [0x5eb8ac]
[compute-00-02:07652] [ 9] /lib64/libc.so.6(__libc_start_main+0xf4) [0x3725c1d974]
[compute-00-02:07652] [10] ./nek5000 [0x4159c9]
[compute-00-02:07652] *** End of error message ***
[compute-00-02:07653] *** Process received signal ***
[compute-00-02:07653] Signal: Segmentation fault (11)
[compute-00-02:07653] Signal code: Invalid permissions (2)
[compute-00-02:07653] Failing at address: 0x5f2886
[compute-00-02:07653] [ 0] /lib64/libpthread.so.0 [0x372640e4c0]
[compute-00-02:07653] [ 1] ./nek5000(chcopy_+0x38) [0x488768]
[compute-00-02:07653] [ 2] ./nek5000(exitti_+0xd6) [0x5cd756]
[compute-00-02:07653] [ 3] ./nek5000(set_conv_char_+0x136) [0x4b8596]
[compute-00-02:07653] [ 4] ./nek5000(setup_convect_+0xa9) [0x4b9d29]
[compute-00-02:07653] [ 5] ./nek5000(setics_+0xd0f) [0x4839bf]
[compute-00-02:07653] [ 6] ./nek5000(nek_init_+0x25a) [0x41648a]
[compute-00-02:07653] [ 7] ./nek5000(MAIN__+0x1a) [0x415bba]
[compute-00-02:07653] [ 8] ./nek5000(main+0x2c) [0x5eb8ac]
[compute-00-02:07653] [ 9] /lib64/libc.so.6(__libc_start_main+0xf4) [0x3725c1d974]
[compute-00-02:07653] [10] ./nek5000 [0x4159c9]
[compute-00-02:07653] *** End of error message ***
[compute-00-02:07654] *** Process received signal ***
[compute-00-02:07654] Signal: Segmentation fault (11)
[compute-00-02:07654] Signal code: Invalid permissions (2)
[compute-00-02:07654] Failing at address: 0x5f2886
[compute-00-02:07654] [ 0] /lib64/libpthread.so.0 [0x372640e4c0]
[compute-00-02:07654] [ 1] ./nek5000(chcopy_+0x38) [0x488768]
[compute-00-02:07654] [ 2] ./nek5000(exitti_+0xd6) [0x5cd756]
[compute-00-02:07654] [ 3] ./nek5000(set_conv_char_+0x136) [0x4b8596]
[compute-00-02:07654] [ 4] ./nek5000(setup_convect_+0xa9) [0x4b9d29]
[compute-00-02:07654] [ 5] ./nek5000(setics_+0xd0f) [0x4839bf]
[compute-00-02:07654] [ 6] ./nek5000(nek_init_+0x25a) [0x41648a]
[compute-00-02:07654] [ 7] ./nek5000(MAIN__+0x1a) [0x415bba]
[compute-00-02:07654] [ 8] ./nek5000(main+0x2c) [0x5eb8ac]
[compute-00-02:07654] [ 9] /lib64/libc.so.6(__libc_start_main+0xf4) [0x3725c1d974]
[compute-00-02:07654] [10] ./nek5000 [0x4159c9]
[compute-00-02:07654] *** End of error message ***
[compute-00-02:07655] *** Process received signal ***
[compute-00-02:07655] Signal: Segmentation fault (11)
[compute-00-02:07655] Signal code: Invalid permissions (2)
[compute-00-02:07655] Failing at address: 0x5f2886
[compute-00-02:07655] [ 0] /lib64/libpthread.so.0 [0x372640e4c0]
[compute-00-02:07655] [ 1] ./nek5000(chcopy_+0x38) [0x488768]
[compute-00-02:07655] [ 2] ./nek5000(exitti_+0xd6) [0x5cd756]
[compute-00-02:07655] [ 3] ./nek5000(set_conv_char_+0x136) [0x4b8596]
[compute-00-02:07655] [ 4] ./nek5000(setup_convect_+0xa9) [0x4b9d29]
[compute-00-02:07655] [ 5] ./nek5000(setics_+0xd0f) [0x4839bf]
[compute-00-02:07655] [ 6] ./nek5000(nek_init_+0x25a) [0x41648a]
[compute-00-02:07655] [ 7] ./nek5000(MAIN__+0x1a) [0x415bba]
[compute-00-02:07655] [ 8] ./nek5000(main+0x2c) [0x5eb8ac]
[compute-00-02:07655] [ 9] /lib64/libc.so.6(__libc_start_main+0xf4) [0x3725c1d974]
[compute-00-02:07655] [10] ./nek5000 [0x4159c9]
[compute-00-02:07655] *** End of error message ***
[compute-00-02:07651] [ 0] /lib64/libpthread.so.0 [0x372640e4c0]
[compute-00-02:07651] [ 1] ./nek5000(chcopy_+0x38) [0x488768]
[compute-00-02:07651] [ 2] ./nek5000(exitti_+0xd6) [0x5cd756]
[compute-00-02:07651] [ 3] ./nek5000(set_conv_char_+0x136) [0x4b8596]
[compute-00-02:07651] [ 4] ./nek5000(setup_convect_+0xa9) [0x4b9d29]
[compute-00-02:07651] [ 5] ./nek5000(setics_+0xd0f) [0x4839bf]
[compute-00-02:07651] [ 6] ./nek5000(nek_init_+0x25a) [0x41648a]
[compute-00-02:07651] [ 7] ./nek5000(MAIN__+0x1a) [0x415bba]
[compute-00-02:07651] [ 8] ./nek5000(main+0x2c) [0x5eb8ac]
[compute-00-02:07651] [ 9] /lib64/libc.so.6(__libc_start_main+0xf4) [0x3725c1d974]
[compute-00-02:07651] [10] ./nek5000 [0x4159c9]
[compute-00-02:07651] *** End of error message ***
[compute-00-03:25107] [ 0] /lib64/libpthread.so.0 [0x34de20e4c0]
[compute-00-03:25107] [ 1] ./nek5000(chcopy_+0x38) [0x488768]
[compute-00-03:25107] [ 2] ./nek5000(exitti_+0xd6) [0x5cd756]
[compute-00-03:25107] [ 3] ./nek5000(set_conv_char_+0x136) [0x4b8596]
[compute-00-03:25107] [ 4] ./nek5000(setup_convect_+0xa9) [0x4b9d29]
[compute-00-03:25107] [ 5] ./nek5000(setics_+0xd0f) [0x4839bf]
[compute-00-03:25107] [ 6] ./nek5000(nek_init_+0x25a) [0x41648a]
[compute-00-03:25107] [ 7] ./nek5000(MAIN__+0x1a) [0x415bba]
[compute-00-03:25107] [ 8] ./nek5000(main+0x2c) [0x5eb8ac]
[compute-00-03:25107] [ 9] /lib64/libc.so.6(__libc_start_main+0xf4) [0x34dda1d974]
[compute-00-03:25107] [10] ./nek5000 [0x4159c9]
[compute-00-03:25107] *** End of error message ***
[compute-00-03:25108] *** Process received signal ***
[compute-00-03:25108] Signal: Segmentation fault (11)
[compute-00-03:25108] Signal code: Invalid permissions (2)
[compute-00-03:25108] Failing at address: 0x5f2886
[compute-00-03:25108] [ 0] /lib64/libpthread.so.0 [0x34de20e4c0]
[compute-00-03:25108] [ 1] ./nek5000(chcopy_+0x38) [0x488768]
[compute-00-03:25108] [ 2] ./nek5000(exitti_+0xd6) [0x5cd756]
[compute-00-03:25108] [ 3] ./nek5000(set_conv_char_+0x136) [0x4b8596]
[compute-00-03:25108] [ 4] ./nek5000(setup_convect_+0xa9) [0x4b9d29]
[compute-00-03:25108] [ 5] ./nek5000(setics_+0xd0f) [0x4839bf]
[compute-00-03:25108] [ 6] ./nek5000(nek_init_+0x25a) [0x41648a]
[compute-00-03:25108] [ 7] ./nek5000(MAIN__+0x1a) [0x415bba]
[compute-00-03:25108] [ 8] ./nek5000(main+0x2c) [0x5eb8ac]
[compute-00-03:25108] [ 9] /lib64/libc.so.6(__libc_start_main+0xf4) [0x34dda1d974]
[compute-00-03:25108] [10] ./nek5000 [0x4159c9]
[compute-00-03:25108] *** End of error message ***
[compute-00-03:25109] [ 0] /lib64/libpthread.so.0 [0x34de20e4c0]
[compute-00-03:25109] [ 1] ./nek5000(chcopy_+0x38) [0x488768]
[compute-00-03:25109] [ 2] ./nek5000(exitti_+0xd6) [0x5cd756]
[compute-00-03:25109] [ 3] ./nek5000(set_conv_char_+0x136) [0x4b8596]
[compute-00-03:25109] [ 4] ./nek5000(setup_convect_+0xa9) [0x4b9d29]
[compute-00-03:25109] [ 5] ./nek5000(setics_+0xd0f) [0x4839bf]
[compute-00-03:25109] [ 6] ./nek5000(nek_init_+0x25a) [0x41648a]
[compute-00-03:25109] [ 7] ./nek5000(MAIN__+0x1a) [0x415bba]
[compute-00-03:25109] [ 8] ./nek5000(main+0x2c) [0x5eb8ac]
[compute-00-03:25109] [ 9] /lib64/libc.so.6(__libc_start_main+0xf4) [0x34dda1d974]
[compute-00-03:25109] [10] ./nek5000 [0x4159c9]
[compute-00-03:25109] *** End of error message ***
[compute-00-03:25110] [ 0] /lib64/libpthread.so.0 [0x34de20e4c0]
[compute-00-03:25110] [ 1] ./nek5000(chcopy_+0x38) [0x488768]
[compute-00-03:25110] [ 2] ./nek5000(exitti_+0xd6) [0x5cd756]
[compute-00-03:25110] [ 3] ./nek5000(set_conv_char_+0x136) [0x4b8596]
[compute-00-03:25110] [ 4] ./nek5000(setup_convect_+0xa9) [0x4b9d29]
[compute-00-03:25110] [ 5] ./nek5000(setics_+0xd0f) [0x4839bf]
[compute-00-03:25110] [ 6] ./nek5000(nek_init_+0x25a) [0x41648a]
[compute-00-03:25110] [ 7] ./nek5000(MAIN__+0x1a) [0x415bba]
[compute-00-03:25110] [ 8] ./nek5000(main+0x2c) [0x5eb8ac]
[compute-00-03:25110] [ 9] /lib64/libc.so.6(__libc_start_main+0xf4) [0x34dda1d974]
[compute-00-03:25110] [10] ./nek5000 [0x4159c9]
[compute-00-03:25110] *** End of error message ***
[compute-00-03:25111] [ 0] /lib64/libpthread.so.0 [0x34de20e4c0]
[compute-00-03:25111] [ 1] ./nek5000(chcopy_+0x38) [0x488768]
[compute-00-03:25111] [ 2] ./nek5000(exitti_+0xd6) [0x5cd756]
[compute-00-03:25111] [ 3] ./nek5000(set_conv_char_+0x136) [0x4b8596]
[compute-00-03:25111] [ 4] ./nek5000(setup_convect_+0xa9) [0x4b9d29]
[compute-00-03:25111] [ 5] ./nek5000(setics_+0xd0f) [0x4839bf]
[compute-00-03:25111] [ 6] ./nek5000(nek_init_+0x25a) [0x41648a]
[compute-00-03:25111] [ 7] ./nek5000(MAIN__+0x1a) [0x415bba]
[compute-00-03:25111] [ 8] ./nek5000(main+0x2c) [0x5eb8ac]
[compute-00-03:25111] [ 9] /lib64/libc.so.6(__libc_start_main+0xf4) [0x34dda1d974]
[compute-00-03:25111] [10] ./nek5000 [0x4159c9]
[compute-00-03:25111] *** End of error message ***
[compute-00-03:25112] [ 0] /lib64/libpthread.so.0 [0x34de20e4c0]
[compute-00-03:25112] [ 1] ./nek5000(chcopy_+0x38) [0x488768]
[compute-00-03:25112] [ 2] ./nek5000(exitti_+0xd6) [0x5cd756]
[compute-00-03:25112] [ 3] ./nek5000(set_conv_char_+0x136) [0x4b8596]
[compute-00-03:25112] [ 4] ./nek5000(setup_convect_+0xa9) [0x4b9d29]
[compute-00-03:25112] [ 5] ./nek5000(setics_+0xd0f) [0x4839bf]
[compute-00-03:25112] [ 6] ./nek5000(nek_init_+0x25a) [0x41648a]
[compute-00-03:25112] [ 7] ./nek5000(MAIN__+0x1a) [0x415bba]
[compute-00-03:25112] [ 8] ./nek5000(main+0x2c) [0x5eb8ac]
[compute-00-03:25112] [ 9] /lib64/libc.so.6(__libc_start_main+0xf4) [0x34dda1d974]
[compute-00-03:25112] [10] ./nek5000 [0x4159c9]
[compute-00-03:25112] *** End of error message ***
[compute-00-03:25113] [ 0] /lib64/libpthread.so.0 [0x34de20e4c0]
[compute-00-03:25113] [ 1] ./nek5000(chcopy_+0x38) [0x488768]
[compute-00-03:25113] [ 2] ./nek5000(exitti_+0xd6) [0x5cd756]
[compute-00-03:25113] [ 3] ./nek5000(set_conv_char_+0x136) [0x4b8596]
[compute-00-03:25113] [ 4] ./nek5000(setup_convect_+0xa9) [0x4b9d29]
[compute-00-03:25113] [ 5] ./nek5000(setics_+0xd0f) [0x4839bf]
[compute-00-03:25113] [ 6] ./nek5000(nek_init_+0x25a) [0x41648a]
[compute-00-03:25113] [ 7] ./nek5000(MAIN__+0x1a) [0x415bba]
[compute-00-03:25113] [ 8] ./nek5000(main+0x2c) [0x5eb8ac]
[compute-00-03:25113] [ 9] /lib64/libc.so.6(__libc_start_main+0xf4) [0x34dda1d974]
[compute-00-03:25113] [10] ./nek5000 [0x4159c9]
[compute-00-03:25113] *** End of error message ***
[compute-00-03:25114] *** Process received signal ***
[compute-00-03:25114] Signal: Segmentation fault (11)
[compute-00-03:25114] Signal code: Invalid permissions (2)
[compute-00-03:25114] Failing at address: 0x5f2886
[compute-00-03:25114] [ 0] /lib64/libpthread.so.0 [0x34de20e4c0]
[compute-00-03:25114] [ 1] ./nek5000(chcopy_+0x38) [0x488768]
[compute-00-03:25114] [ 2] ./nek5000(exitti_+0xd6) [0x5cd756]
[compute-00-03:25114] [ 3] ./nek5000(set_conv_char_+0x136) [0x4b8596]
[compute-00-03:25114] [ 4] ./nek5000(setup_convect_+0xa9) [0x4b9d29]
[compute-00-03:25114] [ 5] ./nek5000(setics_+0xd0f) [0x4839bf]
[compute-00-03:25114] [ 6] ./nek5000(nek_init_+0x25a) [0x41648a]
[compute-00-03:25114] [ 7] ./nek5000(MAIN__+0x1a) [0x415bba]
[compute-00-03:25114] [ 8] ./nek5000(main+0x2c) [0x5eb8ac]
[compute-00-03:25114] [ 9] /lib64/libc.so.6(__libc_start_main+0xf4) [0x34dda1d974]
[compute-00-03:25114] [10] ./nek5000 [0x4159c9]
[compute-00-03:25114] *** End of error message ***
[compute-00-02.local:07644] [0,0,0] ORTE_ERROR_LOG: Timeout in file base/pls_base_orted_cmds.c at line 275
[compute-00-02.local:07644] [0,0,0] ORTE_ERROR_LOG: Timeout in file pls_rsh_module.c at line 1158
[compute-00-02.local:07644] [0,0,0] ORTE_ERROR_LOG: Timeout in file errmgr_hnp.c at line 90
mpirun noticed that job rank 3 with PID 7651 on node compute-00-02 exited on signal 11 (Segmentation fault). 
[compute-00-02.local:07644] [0,0,0] ORTE_ERROR_LOG: Timeout in file base/pls_base_orted_cmds.c at line 188
[compute-00-02.local:07644] [0,0,0] ORTE_ERROR_LOG: Timeout in file pls_rsh_module.c at line 1190
--------------------------------------------------------------------------
mpirun was unable to cleanly terminate the daemons for this job. Returned value Timeout instead of ORTE_SUCCESS.
--------------------------------------------------------------------------