[petsc-dev] failed nightly
Barry Smith
bsmith at mcs.anl.gov
Wed Apr 29 20:22:42 CDT 2015
Well ok. Report to the OpenMP developers and have --download-openmp use your patched version.
Thanks,
Barry
> On Apr 29, 2015, at 7:06 PM, Satish Balay <balay at mcs.anl.gov> wrote:
>
> The following change to openmpi appears to get rid of this error [and valgrind messages]
>
> I haven't checked closely to see what the actual bug is though..
>
> Satish
>
> -------
>
> $ diff -Nru ompi/datatype/ompi_datatype_args.c~ ompi/datatype/ompi_datatype_args.c
> --- ompi/datatype/ompi_datatype_args.c~ 2014-10-03 15:32:23.000000000 -0500
> +++ ompi/datatype/ompi_datatype_args.c 2015-04-29 19:00:03.425618061 -0500
> @@ -84,7 +84,7 @@
> do { \
> int length = sizeof(ompi_datatype_args_t) + (IC) * sizeof(int) + \
> (AC) * sizeof(OPAL_PTRDIFF_TYPE) + (DC) * sizeof(MPI_Datatype); \
> - char* buf = (char*)malloc( length ); \
> + char* buf = (char*)malloc( length+8 ); \
> ompi_datatype_args_t* pArgs = (ompi_datatype_args_t*)buf; \
> pArgs->ci = (IC); \
> pArgs->ca = (AC); \
> $
>
>
> On Fri, 10 Apr 2015, Barry Smith wrote:
>
>>
>> Not testing something that doesn't work in order to not have an error in the tests doesn't seem right to me. Shouldn't the window stuff be fixed or removed rather than leaving buggy code?
>>
>> barry
>>
>>> On Apr 10, 2015, at 2:45 AM, Lawrence Mitchell <lawrence.mitchell at imperial.ac.uk> wrote:
>>>
>>> (cc'ing petsc-dev as well)
>>>
>>>> On 10 Apr 2015, at 00:51, Satish Balay <balay at mcs.anl.gov> wrote:
>>>>
>>>> Its likely a codebug some where. MPICH build also gives a valgrind trace.
>>>
>>> It's probable that the OMPI implementation is buggy enough to completely not work. I'm a little confused by the MPICH issue. I don't understand enough about the datatype implementation in the window SF type to know if this is a PETSc issue, or an MPICH one. I note in passing that all the ex1 tests exhibit a similar valgrind trace.
>>>
>>> For ex2 at least, maybe the simplest option is to turn off the window test entirely. Like this:
>>>
>>> diff --git a/src/vec/is/sf/examples/tutorials/makefile b/src/vec/is/sf/examples/tutorials
>>> index aeaf1e4..e7774c5 100644
>>> --- a/src/vec/is/sf/examples/tutorials/makefile
>>> +++ b/src/vec/is/sf/examples/tutorials/makefile
>>> @@ -86,7 +86,7 @@ runex2_window:
>>> ${RM} -f ex2.tmp
>>>
>>> TESTEXAMPLES_C = ex1.PETSc runex1_basic runex1_2_basic runex1_3_basic runex1
>>> - ex2.PETSc runex2_basic runex2_window ex2.rm
>>> + ex2.PETSc runex2_basic ex2.rm
>>> TESTEXAMPLES_C_X =
>>> TESTEXAMPLES_FORTRAN =
>>> TESTEXAMPLES_FORTRAN_MPIUNI =
>>>
>>> Lawrence
>>>
>>>> Satish
>>>>
>>>> ----------
>>>>
>>>> balay at asterix /home/balay/petsc/src/vec/is/sf/examples/tutorials (master=)
>>>> $ mpiexec -n 2 valgrind --tool=memcheck -q --dsymutil=yes --num-callers=40 --track-origins=yes ./ex2 -sf_type window
>>>> PetscSF Object: 2 MPI processes
>>>> type: window
>>>> synchronization=FENCE sort=rank-order
>>>> [0] Number of roots=1, leaves=2, remote ranks=2
>>>> [0] 0 <- (0,0)
>>>> [0] 1 <- (1,0)
>>>> [1] Number of roots=1, leaves=2, remote ranks=2
>>>> [1] 0 <- (1,0)
>>>> [1] 1 <- (0,0)
>>>> ==29265== Syscall param writev(vector[...]) points to uninitialised byte(s)
>>>> ==29265== at 0x8F474E7: writev (in /usr/lib64/libc-2.20.so)
>>>> ==29265== by 0x894AD87: MPL_large_writev (in /home/balay/soft/mpich-3.1.3/lib/libmpi.so.12.0.4)
>>>> ==29265== by 0x8941A48: MPIDU_Sock_writev (in /home/balay/soft/mpich-3.1.3/lib/libmpi.so.12.0.4)
>>>> ==29265== by 0x892AC7D: MPIDI_CH3_iStartMsgv (in /home/balay/soft/mpich-3.1.3/lib/libmpi.so.12.0.4)
>>>> ==29265== by 0x8911D08: recv_rma_msg (in /home/balay/soft/mpich-3.1.3/lib/libmpi.so.12.0.4)
>>>> ==29265== by 0x8913D46: MPIDI_Win_fence (in /home/balay/soft/mpich-3.1.3/lib/libmpi.so.12.0.4)
>>>> ==29265== by 0x88C91EB: PMPI_Win_fence (in /home/balay/soft/mpich-3.1.3/lib/libmpi.so.12.0.4)
>>>> ==29265== by 0x50FB025: PetscSFRestoreWindow (sfwindow.c:348)
>>>> ==29265== by 0x50FD4BF: PetscSFBcastEnd_Window (sfwindow.c:510)
>>>> ==29265== by 0x5123CD9: PetscSFBcastEnd (sf.c:957)
>>>> ==29265== by 0x401CAF: main (ex2.c:81)
>>>> ==29265== Address 0x99436ec is 108 bytes inside a block of size 208 alloc'd
>>>> ==29265== at 0x4C29BCF: malloc (in /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so)
>>>> ==29265== by 0x890DA35: MPIDI_Get (in /home/balay/soft/mpich-3.1.3/lib/libmpi.so.12.0.4)
>>>> ==29265== by 0x88C406A: PMPI_Get (in /home/balay/soft/mpich-3.1.3/lib/libmpi.so.12.0.4)
>>>> ==29265== by 0x50FD0DA: PetscSFBcastBegin_Window (sfwindow.c:495)
>>>> ==29265== by 0x51235B5: PetscSFBcastBegin (sf.c:924)
>>>> ==29265== by 0x401BD3: main (ex2.c:79)
>>>> ==29265== Uninitialised value was created by a heap allocation
>>>> ==29265== at 0x4C29BCF: malloc (in /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so)
>>>> ==29265== by 0x890DA35: MPIDI_Get (in /home/balay/soft/mpich-3.1.3/lib/libmpi.so.12.0.4)
>>>> ==29265== by 0x88C406A: PMPI_Get (in /home/balay/soft/mpich-3.1.3/lib/libmpi.so.12.0.4)
>>>> ==29265== by 0x50FD0DA: PetscSFBcastBegin_Window (sfwindow.c:495)
>>>> ==29265== by 0x51235B5: PetscSFBcastBegin (sf.c:924)
>>>> ==29265== by 0x401BD3: main (ex2.c:79)
>>>> ==29265==
>>>> ==29266== Syscall param writev(vector[...]) points to uninitialised byte(s)
>>>> ==29266== at 0x8F474E7: writev (in /usr/lib64/libc-2.20.so)
>>>> ==29266== by 0x894AD87: MPL_large_writev (in /home/balay/soft/mpich-3.1.3/lib/libmpi.so.12.0.4)
>>>> ==29266== by 0x8941A48: MPIDU_Sock_writev (in /home/balay/soft/mpich-3.1.3/lib/libmpi.so.12.0.4)
>>>> ==29266== by 0x892AC7D: MPIDI_CH3_iStartMsgv (in /home/balay/soft/mpich-3.1.3/lib/libmpi.so.12.0.4)
>>>> ==29266== by 0x8911D08: recv_rma_msg (in /home/balay/soft/mpich-3.1.3/lib/libmpi.so.12.0.4)
>>>> ==29266== by 0x8913D46: MPIDI_Win_fence (in /home/balay/soft/mpich-3.1.3/lib/libmpi.so.12.0.4)
>>>> ==29266== by 0x88C91EB: PMPI_Win_fence (in /home/balay/soft/mpich-3.1.3/lib/libmpi.so.12.0.4)
>>>> ==29266== by 0x50FB025: PetscSFRestoreWindow (sfwindow.c:348)
>>>> ==29266== by 0x50FD4BF: PetscSFBcastEnd_Window (sfwindow.c:510)
>>>> ==29266== by 0x5123CD9: PetscSFBcastEnd (sf.c:957)
>>>> ==29266== by 0x401CAF: main (ex2.c:81)
>>>> ==29266== Address 0x98d16dc is 108 bytes inside a block of size 208 alloc'd
>>>> ==29266== at 0x4C29BCF: malloc (in /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so)
>>>> ==29266== by 0x890DA35: MPIDI_Get (in /home/balay/soft/mpich-3.1.3/lib/libmpi.so.12.0.4)
>>>> ==29266== by 0x88C406A: PMPI_Get (in /home/balay/soft/mpich-3.1.3/lib/libmpi.so.12.0.4)
>>>> ==29266== by 0x50FD0DA: PetscSFBcastBegin_Window (sfwindow.c:495)
>>>> ==29266== by 0x51235B5: PetscSFBcastBegin (sf.c:924)
>>>> ==29266== by 0x401BD3: main (ex2.c:79)
>>>> ==29266== Uninitialised value was created by a heap allocation
>>>> ==29266== at 0x4C29BCF: malloc (in /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so)
>>>> ==29266== by 0x890DA35: MPIDI_Get (in /home/balay/soft/mpich-3.1.3/lib/libmpi.so.12.0.4)
>>>> ==29266== by 0x88C406A: PMPI_Get (in /home/balay/soft/mpich-3.1.3/lib/libmpi.so.12.0.4)
>>>> ==29266== by 0x50FD0DA: PetscSFBcastBegin_Window (sfwindow.c:495)
>>>> ==29266== by 0x51235B5: PetscSFBcastBegin (sf.c:924)
>>>> ==29266== by 0x401BD3: main (ex2.c:79)
>>>> ==29266==
>>>> Vec Object: 2 MPI processes
>>>> type: mpi
>>>> Process [0]
>>>> 0
>>>> 1
>>>> Process [1]
>>>> 1
>>>> 0
>>>> Vec Object: 2 MPI processes
>>>> type: mpi
>>>> Process [0]
>>>> 10
>>>> 11
>>>> Process [1]
>>>> 11
>>>> 10
>>>> balay at asterix /home/balay/petsc/src/vec/is/sf/examples/tutorials (master=)
>>>> $
>>>>
>>>> On Thu, 9 Apr 2015, Satish Balay wrote:
>>>>
>>>>> here is a better valgrind trace..
>>>>>
>>>>> satish
>>>>>
>>>>> --------
>>>>> balay at asterix /home/balay/petsc/src/vec/is/sf/examples/tutorials (master=)
>>>>> $ /home/balay/petsc/arch-ompi/bin/mpiexec -n 2 valgrind --tool=memcheck -q --dsymutil=yes --num-callers=40 --track-origins=yes ./ex2 -sf_type window
>>>>> PetscSF Object: 2 MPI processes
>>>>> type: window
>>>>> synchronization=FENCE sort=rank-order
>>>>> [0] Number of roots=1, leaves=2, remote ranks=2
>>>>> [0] 0 <- (0,0)
>>>>> [0] 1 <- (1,0)
>>>>> [1] Number of roots=1, leaves=2, remote ranks=2
>>>>> [1] 0 <- (1,0)
>>>>> [1] 1 <- (0,0)
>>>>> ==14815== Invalid write of size 2
>>>>> ==14815== at 0x4C2E36B: memcpy@@GLIBC_2.14 (in /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so)
>>>>> ==14815== by 0x8AFDABD: ompi_datatype_set_args (ompi_datatype_args.c:167)
>>>>> ==14815== by 0x8AFF0F3: __ompi_datatype_create_from_args (ompi_datatype_args.c:718)
>>>>> ==14815== by 0x8AFEC0E: __ompi_datatype_create_from_packed_description (ompi_datatype_args.c:649)
>>>>> ==14815== by 0x8AFF5D6: ompi_datatype_create_from_packed_description (ompi_datatype_args.c:788)
>>>>> ==14815== by 0xF727F0E: ompi_osc_base_datatype_create (osc_base_obj_convert.h:52)
>>>>> ==14815== by 0xF728424: datatype_create (osc_rdma_data_move.c:333)
>>>>> ==14815== by 0xF72887D: process_get (osc_rdma_data_move.c:536)
>>>>> ==14815== by 0xF72A856: process_frag (osc_rdma_data_move.c:1593)
>>>>> ==14815== by 0xF72AA35: ompi_osc_rdma_callback (osc_rdma_data_move.c:1656)
>>>>> ==14815== by 0xECCF0DD: ompi_request_complete (request.h:402)
>>>>> ==14815== by 0xECCF4EA: recv_request_pml_complete (pml_ob1_recvreq.h:181)
>>>>> ==14815== by 0xECCFF87: mca_pml_ob1_recv_frag_callback_match (pml_ob1_recvfrag.c:243)
>>>>> ==14815== by 0xE68F875: mca_btl_vader_check_fboxes (btl_vader_fbox.h:220)
>>>>> ==14815== by 0xE690D82: mca_btl_vader_component_progress (btl_vader_component.c:695)
>>>>> ==14815== by 0x9A9E9F2: opal_progress (opal_progress.c:187)
>>>>> ==14815== by 0xECCA70A: opal_condition_wait (condition.h:78)
>>>>> ==14815== by 0xECCA7F4: ompi_request_wait_completion (request.h:381)
>>>>> ==14815== by 0xECCAF69: mca_pml_ob1_recv (pml_ob1_irecv.c:109)
>>>>> ==14815== by 0xFD8938D: ompi_coll_tuned_reduce_intra_basic_linear (coll_tuned_reduce.c:677)
>>>>> ==14815== by 0xFD79C26: ompi_coll_tuned_reduce_intra_dec_fixed (coll_tuned_decision_fixed.c:386)
>>>>> ==14815== by 0xF0F3B91: mca_coll_basic_reduce_scatter_block_intra (coll_basic_reduce_scatter_block.c:96)
>>>>> ==14815== by 0xF72BC58: ompi_osc_rdma_fence (osc_rdma_active_target.c:140)
>>>>> ==14815== by 0x8B47078: PMPI_Win_fence (pwin_fence.c:59)
>>>>> ==14815== by 0x5106D8F: PetscSFRestoreWindow (sfwindow.c:348)
>>>>> ==14815== by 0x51092DA: PetscSFBcastEnd_Window (sfwindow.c:510)
>>>>> ==14815== by 0x51303D6: PetscSFBcastEnd (sf.c:957)
>>>>> ==14815== by 0x401DD3: main (ex2.c:81)
>>>>> ==14815== Address 0x101c3b98 is 0 bytes after a block of size 72 alloc'd
>>>>> ==14815== at 0x4C29BCF: malloc (in /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so)
>>>>> ==14815== by 0x8AFD755: ompi_datatype_set_args (ompi_datatype_args.c:123)
>>>>> ==14815== by 0x8AFF0F3: __ompi_datatype_create_from_args (ompi_datatype_args.c:718)
>>>>> ==14815== by 0x8AFEC0E: __ompi_datatype_create_from_packed_description (ompi_datatype_args.c:649)
>>>>> ==14815== by 0x8AFF5D6: ompi_datatype_create_from_packed_description (ompi_datatype_args.c:788)
>>>>> ==14815== by 0xF727F0E: ompi_osc_base_datatype_create (osc_base_obj_convert.h:52)
>>>>> ==14815== by 0xF728424: datatype_create (osc_rdma_data_move.c:333)
>>>>> ==14815== by 0xF72887D: process_get (osc_rdma_data_move.c:536)
>>>>> ==14815== by 0xF72A856: process_frag (osc_rdma_data_move.c:1593)
>>>>> ==14815== by 0xF72AA35: ompi_osc_rdma_callback (osc_rdma_data_move.c:1656)
>>>>> ==14815== by 0xECCF0DD: ompi_request_complete (request.h:402)
>>>>> ==14815== by 0xECCF4EA: recv_request_pml_complete (pml_ob1_recvreq.h:181)
>>>>> ==14815== by 0xECCFF87: mca_pml_ob1_recv_frag_callback_match (pml_ob1_recvfrag.c:243)
>>>>> ==14815== by 0xE68F875: mca_btl_vader_check_fboxes (btl_vader_fbox.h:220)
>>>>> ==14815== by 0xE690D82: mca_btl_vader_component_progress (btl_vader_component.c:695)
>>>>> ==14815== by 0x9A9E9F2: opal_progress (opal_progress.c:187)
>>>>> ==14815== by 0xECCA70A: opal_condition_wait (condition.h:78)
>>>>> ==14815== by 0xECCA7F4: ompi_request_wait_completion (request.h:381)
>>>>> ==14815== by 0xECCAF69: mca_pml_ob1_recv (pml_ob1_irecv.c:109)
>>>>> ==14815== by 0xFD8938D: ompi_coll_tuned_reduce_intra_basic_linear (coll_tuned_reduce.c:677)
>>>>> ==14815== by 0xFD79C26: ompi_coll_tuned_reduce_intra_dec_fixed (coll_tuned_decision_fixed.c:386)
>>>>> ==14815== by 0xF0F3B91: mca_coll_basic_reduce_scatter_block_intra (coll_basic_reduce_scatter_block.c:96)
>>>>> ==14815== by 0xF72BC58: ompi_osc_rdma_fence (osc_rdma_active_target.c:140)
>>>>> ==14815== by 0x8B47078: PMPI_Win_fence (pwin_fence.c:59)
>>>>> ==14815== by 0x5106D8F: PetscSFRestoreWindow (sfwindow.c:348)
>>>>> ==14815== by 0x51092DA: PetscSFBcastEnd_Window (sfwindow.c:510)
>>>>> ==14815== by 0x51303D6: PetscSFBcastEnd (sf.c:957)
>>>>> ==14815== by 0x401DD3: main (ex2.c:81)
>>>>> ==14815==
>>>>> ==14816== Invalid write of size 2
>>>>> ==14816== at 0x4C2E36B: memcpy@@GLIBC_2.14 (in /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so)
>>>>> ==14816== by 0x8AFDABD: ompi_datatype_set_args (ompi_datatype_args.c:167)
>>>>> ==14816== by 0x8AFF0F3: __ompi_datatype_create_from_args (ompi_datatype_args.c:718)
>>>>> ==14816== by 0x8AFEC0E: __ompi_datatype_create_from_packed_description (ompi_datatype_args.c:649)
>>>>> ==14816== by 0x8AFF5D6: ompi_datatype_create_from_packed_description (ompi_datatype_args.c:788)
>>>>> ==14816== by 0xF727F0E: ompi_osc_base_datatype_create (osc_base_obj_convert.h:52)
>>>>> ==14816== by 0xF728424: datatype_create (osc_rdma_data_move.c:333)
>>>>> ==14816== by 0xF72887D: process_get (osc_rdma_data_move.c:536)
>>>>> ==14816== by 0xF72A856: process_frag (osc_rdma_data_move.c:1593)
>>>>> ==14816== by 0xF72AA35: ompi_osc_rdma_callback (osc_rdma_data_move.c:1656)
>>>>> ==14816== by 0xECCF0DD: ompi_request_complete (request.h:402)
>>>>> ==14816== by 0xECCF4EA: recv_request_pml_complete (pml_ob1_recvreq.h:181)
>>>>> ==14816== by 0xECCFF87: mca_pml_ob1_recv_frag_callback_match (pml_ob1_recvfrag.c:243)
>>>>> ==14816== by 0xE68F875: mca_btl_vader_check_fboxes (btl_vader_fbox.h:220)
>>>>> ==14816== by 0xE690D82: mca_btl_vader_component_progress (btl_vader_component.c:695)
>>>>> ==14816== by 0x9A9E9F2: opal_progress (opal_progress.c:187)
>>>>> ==14816== by 0xECCA70A: opal_condition_wait (condition.h:78)
>>>>> ==14816== by 0xECCA7F4: ompi_request_wait_completion (request.h:381)
>>>>> ==14816== by 0xECCAF69: mca_pml_ob1_recv (pml_ob1_irecv.c:109)
>>>>> ==14816== by 0xFD8D951: ompi_coll_tuned_scatter_intra_basic_linear (coll_tuned_scatter.c:231)
>>>>> ==14816== by 0xFD7A66D: ompi_coll_tuned_scatter_intra_dec_fixed (coll_tuned_decision_fixed.c:769)
>>>>> ==14816== by 0xF0F3BDB: mca_coll_basic_reduce_scatter_block_intra (coll_basic_reduce_scatter_block.c:102)
>>>>> ==14816== by 0xF72BC58: ompi_osc_rdma_fence (osc_rdma_active_target.c:140)
>>>>> ==14816== by 0x8B47078: PMPI_Win_fence (pwin_fence.c:59)
>>>>> ==14816== by 0x5106D8F: PetscSFRestoreWindow (sfwindow.c:348)
>>>>> ==14816== by 0x51092DA: PetscSFBcastEnd_Window (sfwindow.c:510)
>>>>> ==14816== by 0x51303D6: PetscSFBcastEnd (sf.c:957)
>>>>> ==14816== by 0x401DD3: main (ex2.c:81)
>>>>> ==14816== Address 0x101bb398 is 0 bytes after a block of size 72 alloc'd
>>>>> ==14816== at 0x4C29BCF: malloc (in /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so)
>>>>> ==14816== by 0x8AFD755: ompi_datatype_set_args (ompi_datatype_args.c:123)
>>>>> ==14816== by 0x8AFF0F3: __ompi_datatype_create_from_args (ompi_datatype_args.c:718)
>>>>> ==14816== by 0x8AFEC0E: __ompi_datatype_create_from_packed_description (ompi_datatype_args.c:649)
>>>>> ==14816== by 0x8AFF5D6: ompi_datatype_create_from_packed_description (ompi_datatype_args.c:788)
>>>>> ==14816== by 0xF727F0E: ompi_osc_base_datatype_create (osc_base_obj_convert.h:52)
>>>>> ==14816== by 0xF728424: datatype_create (osc_rdma_data_move.c:333)
>>>>> ==14816== by 0xF72887D: process_get (osc_rdma_data_move.c:536)
>>>>> ==14816== by 0xF72A856: process_frag (osc_rdma_data_move.c:1593)
>>>>> ==14816== by 0xF72AA35: ompi_osc_rdma_callback (osc_rdma_data_move.c:1656)
>>>>> ==14816== by 0xECCF0DD: ompi_request_complete (request.h:402)
>>>>> ==14816== by 0xECCF4EA: recv_request_pml_complete (pml_ob1_recvreq.h:181)
>>>>> ==14816== by 0xECCFF87: mca_pml_ob1_recv_frag_callback_match (pml_ob1_recvfrag.c:243)
>>>>> ==14816== by 0xE68F875: mca_btl_vader_check_fboxes (btl_vader_fbox.h:220)
>>>>> ==14816== by 0xE690D82: mca_btl_vader_component_progress (btl_vader_component.c:695)
>>>>> ==14816== by 0x9A9E9F2: opal_progress (opal_progress.c:187)
>>>>> ==14816== by 0xECCA70A: opal_condition_wait (condition.h:78)
>>>>> ==14816== by 0xECCA7F4: ompi_request_wait_completion (request.h:381)
>>>>> ==14816== by 0xECCAF69: mca_pml_ob1_recv (pml_ob1_irecv.c:109)
>>>>> ==14816== by 0xFD8D951: ompi_coll_tuned_scatter_intra_basic_linear (coll_tuned_scatter.c:231)
>>>>> ==14816== by 0xFD7A66D: ompi_coll_tuned_scatter_intra_dec_fixed (coll_tuned_decision_fixed.c:769)
>>>>> ==14816== by 0xF0F3BDB: mca_coll_basic_reduce_scatter_block_intra (coll_basic_reduce_scatter_block.c:102)
>>>>> ==14816== by 0xF72BC58: ompi_osc_rdma_fence (osc_rdma_active_target.c:140)
>>>>> ==14816== by 0x8B47078: PMPI_Win_fence (pwin_fence.c:59)
>>>>> ==14816== by 0x5106D8F: PetscSFRestoreWindow (sfwindow.c:348)
>>>>> ==14816== by 0x51092DA: PetscSFBcastEnd_Window (sfwindow.c:510)
>>>>> ==14816== by 0x51303D6: PetscSFBcastEnd (sf.c:957)
>>>>> ==14816== by 0x401DD3: main (ex2.c:81)
>>>>> ==14816==
>>>>> Vec Object: 2 MPI processes
>>>>> type: mpi
>>>>> Process [0]
>>>>> 0
>>>>> 1
>>>>> Process [1]
>>>>> 1
>>>>> 0
>>>>> Vec Object: 2 MPI processes
>>>>> type: mpi
>>>>> Process [0]
>>>>> 10
>>>>> 11
>>>>> Process [1]
>>>>> 11
>>>>> 10
>>>>> balay at asterix /home/balay/petsc/src/vec/is/sf/examples/tutorials (master=)
>>>>> $
>>>>>
>>>>>
>>>>> On Thu, 9 Apr 2015, Barry Smith wrote:
>>>>>
>>>>>>
>>>>>> Satish,
>>>>>>
>>>>>> Why are you telling me :-). Tell the person who's been pushing this stuff into PETSc and he can debug it.
>>>>>>
>>>>>> Barry
>>>>>>
>>>>>> This is why "my part" of PETSc only uses MPI 1.1 :-)
>>>>>>
>>>>>>
>>>>>>
>>>>>>> On Apr 9, 2015, at 5:48 PM, Satish Balay <balay at mcs.anl.gov> wrote:
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Thu, 9 Apr 2015, Barry Smith wrote:
>>>>>>>
>>>>>>>>
>>>>>>>> http://ftp.mcs.anl.gov/pub/petsc/nightlylogs/archive/2015/04/08/examples_master_arch-linux-pkgs-opt_crank.log
>>>>>>>
>>>>>>>
>>>>>>> The following test is hanging - perhaps --download-openmpi is the trigger.
>>>>>>>
>>>>>>>
>>>>>>> petsc 14547 0.0 0.0 12312 1220 ? S 13:56 0:00 /bin/sh -c /sandbox/petsc/petsc.clone/arch-linux-pkgs-opt/bin/mpiexec -n 2 ./ex2 -sf_type window > ex2.tmp 2>&1; \? /usr/bin/diff -w output/ex2_window.out ex2.tmp || printf "/sandbox/petsc/petsc.clone/src/vec/is/sf/examples/tutorials\nPossible problem with ex2_window, diffs above\n=========================================\n"; \? /bin/rm -f -f ex2.tmp
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> I can reproduce on my laptop [with the following trace].
>>>>>>>
>>>>>>> Satish
>>>>>>>
>>>>>>> ---------
>>>>>>>
>>>>>>> balay at asterix /home/balay/petsc/src/vec/is/sf/examples/tutorials (master=)
>>>>>>> $ /home/balay/petsc/arch-ompi/bin/mpiexec -n 2 ./ex2 -sf_type window
>>>>>>> PetscSF Object: 2 MPI processes
>>>>>>> type: window
>>>>>>> synchronization=FENCE sort=rank-order
>>>>>>> [0] Number of roots=1, leaves=2, remote ranks=2
>>>>>>> [0] 0 <- (0,0)
>>>>>>> [0] 1 <- (1,0)
>>>>>>> [1] Number of roots=1, leaves=2, remote ranks=2
>>>>>>> [1] 0 <- (1,0)
>>>>>>> [1] 1 <- (0,0)
>>>>>>> *** Error in `./ex2': free(): invalid next size (fast): 0x0000000002395ed0 ***
>>>>>>> [asterix:14290] *** Process received signal ***
>>>>>>> [asterix:14290] Signal: Aborted (6)
>>>>>>> [asterix:14290] Signal code: (-6)
>>>>>>> ======= Backtrace: =========
>>>>>>> /lib64/libc.so.6(+0x77d9e)[0x[asterix:14290] [ 0] /lib64/libpthread.so.0(+0x100d0)[0x7f331fac10d0]
>>>>>>> [asterix:14290] [ 1] /lib64/libc.so.6(gsignal+0x37)[0x7f331f7288d7]
>>>>>>> [asterix:14290] [ 2] /home/balay/petsc/arch-ompi/lib/libmpi.so.1(ompi_datatype_release_args/lib64/libc.so.6(abort+0x16a)[0x7f331f72a53a]
>>>>>>> [asterix:14290] [ 3] /home/balay/petsc/arch-ompi/lib/libmpi.so.1(/lib64/libc.so.6(+0x77da3)[0x7f331f76bda3]
>>>>>>> [asterix:14290] [ 4] +0x508e3)[0x7f9018c898e3]
>>>>>>> /home/balay/petsc/arch-ompi/lib/openmpi/mca_pml_ob1.so(+0x11773/lib64/libc.so.6(cfree+0x5b5)[0x7f331f7779f5]
>>>>>>> [asterix:14290] [ 5] /home/balay/petsc/arch-ompi/lib/openmpi/mca_pml_ob1.so(+0x12ece)[0x7f900f73aece]
>>>>>>> /home/balay/petsc/arch-ompi/lib/openmpi/mca_osc_rdma.so(+0x862a)[0x7f900ecdb62a]
>>>>>>> /home/balay/petsc/arch-ompi/lib/openmpi/mca_osc_rdma.so(+0x8a15/home/balay/petsc/arch-ompi/lib/libmpi.so.1(ompi_datatype_release_args+0x12b)[0x7f331ff33627]
>>>>>>> [asterix:14290] [ 6] /home/balay/petsc/arch-ompi/lib/openmpi/mca_osc_rdma.so(+0xbac7)[0x7f900ecdeac7]
>>>>>>> /home/balay/petsc/arch-ompi/lib/openmpi/mca_pml_ob1.so(+0xc0de)[0x7f900f7340de]
>>>>>>> /home/balay/petsc/arch-ompi/lib/openmpi/mca_pml_ob1.so(+0xc4eb)[0x7f900f7344eb]
>>>>>>> /home/balay/petsc/arch-ompi/lib/openmpi/mca_pml_ob1.so(mca_pml_ob1_recv_frag_callback_match+0x2ed)[0x7f900f734f88]
>>>>>>> /home/balay/petsc/arch-ompi/lib/openmpi/mca_btl_vader.so(+0x3876)[0x7f9014009876]
>>>>>>> /home/balay/petsc/arch-ompi/lib/openmpi/mca_btl_vader.so(+0x4d83)[0x7f901400ad83]
>>>>>>> /home/balay/petsc/arch-ompi/lib/libopen-pal.so.6(opal_progress+0xa2)[0x7f9017cca9f3]
>>>>>>> /home/balay/petsc/arch-ompi/lib/openmpi/mca_pml_ob1.so(+0x770b)[0x7f900f72f70b]
>>>>>>> /home/balay/petsc/arch-ompi/lib/openmpi/mca_pml_ob1.so(+0x77f5)[0x7f900f72f7f5]
>>>>>>> /home/balay/petsc/arch-ompi/lib/openmpi/mca_pml_ob1.so(mca_pml_ob1_recv+0x1c6)[0x7f900f72ff6a]
>>>>>>> /home/balay/petsc/arch-ompi/lib/openmpi/mca_coll_tuned.so(ompi_coll_tuned_scatter_intra_basic_linear+0x76)[0x7f900e689952]
>>>>>>> /home/balay/petsc/arch-ompi/lib/openmpi/mca_coll_tuned.so(ompi_coll_tuned_scatter_intra_dec_fixed+0x112)[0x7f900e67666e]
>>>>>>> /home/balay/petsc/arch-ompi/lib/openmpi/mca_coll_basic.so(mca_coll_basic_reduce_scatter_block_intra+0x188)[0x7f900f319bdc]
>>>>>>> /home/balay/petsc/arch-ompi/lib/openmpi/mca_osc_rdma.so(ompi_osc_rdma_fence+0x125)[0x7f900ecdfc59]
>>>>>>> /home/balay/petsc/arch-ompi/lib/libmpi.so.1(MPI_Win_fence+0x116)[0x7f9018cd1079]
>>>>>>> /home/balay/petsc/arch-ompi/lib/libmpi.so.1(+0x508e3)[0x7f331ff348e3]
>>>>>>> [asterix:14290] [ 7] /home/balay/petsc/arch-ompi/lib/openmpi/mca_pml_ob1.so(+0x11773)[0x7f3316910773]
>>>>>>> [asterix:14290] [ 8] /home/balay/petsc/arch-ompi/lib/openmpi/mca_pml_ob1.so(+0x12ece)[0x7f3316911ece]
>>>>>>> [asterix:14290] [ 9] /home/balay/petsc/arch-ompi/lib/openmpi/mca_osc_rdma.so(+0x862a)[0x7f3315eb262a]
>>>>>>> [asterix:14290] [10] /home/balay/petsc/arch-ompi/lib/openmpi/mca_osc_rdma.so(+0x8a15)[0x7f3315eb2a15]
>>>>>>> [asterix:14290] [11] /home/balay/petsc/arch-ompi/lib/openmpi/mca_osc_rdma.so(+0xbac7)[0x7f3315eb5ac7]
>>>>>>> [asterix:14290] [12] /home/balay/petsc/arch-ompi/lib/openmpi/mca_pml_ob1.so(+0xc0de)[0x7f331690b0de]
>>>>>>> [asterix:14290] [13] /home/balay/petsc/arch-ompi/lib/openmpi/mca_pml_ob1.so(+0xc4eb)[0x7f331690b4eb]
>>>>>>> [asterix:14290] [14] /home/balay/petsc/arch-ompi/lib/openmpi/mca_pml_ob1.so(mca_pml_ob1_recv_frag_callback_match+0x2ed)[0x7f331690bf88]
>>>>>>> [asterix:14290] [15] /home/balay/petsc/arch-ompi/lib/openmpi/mca_btl_vader.so(+0x3876)[0x7f3316f4a876]
>>>>>>> [asterix:14290] [16] /home/balay/petsc/arch-ompi/lib/openmpi/mca_btl_vader.so(+0x4d83)[0x7f3316f4bd83]
>>>>>>> [asterix:14290] [17] /home/balay/petsc/arch-ompi/lib/libopen-pal.so.6(opal_progress+0xa2)[0x7f331ef759f3]
>>>>>>> [asterix:14290] [18] /home/balay/petsc/arch-ompi/lib/openmpi/mca_pml_ob1.so(+0x770b)[0x7f331690670b]
>>>>>>> [asterix:14290] [19] /home/balay/petsc/arch-ompi/lib/openmpi/mca_pml_ob1.so(+0x77f5)[0x7f33169067f5]
>>>>>>> [asterix:14290] [20] /home/balay/petsc/arch-ompi/lib/openmpi/mca_pml_ob1.so(mca_pml_ob1_recv+0x1c6)[0x7f3316906f6a]
>>>>>>> [asterix:14290] [21] /home/balay/petsc/arch-ompi/lib/openmpi/mca_coll_tuned.so(ompi_coll_tuned_reduce_intra_basic_linear+0x1cb)[0x7f331585c38e]
>>>>>>> [asterix:14290] [22] /home/balay/petsc/arch-ompi/lib/openmpi/mca_coll_tuned.so(ompi_coll_tuned_reduce_intra_dec_fixed+0x1a6)[0x7f331584cc27]
>>>>>>> [asterix:14290] [23] /home/balay/petsc/arch-ompi/lib/openmpi/mca_coll_basic.so(mca_coll_basic_reduce_scatter_block_intra+0x13e)[0x7f33164f0b92]
>>>>>>> [asterix:14290] [24] /home/balay/petsc/arch-ompi/lib/openmpi/mca_osc_rdma.so(ompi_osc_rdma_fence+0x125)[0x7f3315eb6c59]
>>>>>>> [asterix:14290] [25] /home/balay/petsc/arch-ompi/lib/libmpi.so.1(MPI_Win_fence+0x116)[0x7f331ff7c079]
>>>>>>> [asterix:14290] [26] /home/balay/petsc/arch-ompi/lib/libpetsc.so.3.05(+0x2d1d90)[0x7f3322855d90]
>>>>>>> [asterix:14290] [27] /home/balay/petsc/arch-ompi/lib/libpetsc.so.3.05(PetscSFBcastEnd_Window+0x218)[0x7f33228582db]
>>>>>>> [asterix:14290] [28] /home/balay/petsc/arch-ompi/lib/libpetsc.so.3.05(PetscSFBcastEnd+0x4eb)[0x7f332287f3d7]
>>>>>>> [asterix:14290] [29] ./ex2[0x401dd4]
>>>>>>> [asterix:14290] *** End of error message ***
>>>>>>> [asterix:14291] *** Process received signal ***
>>>>>>> [asterix:14291] Signal: Aborted (6)
>>>>>>> [asterix:14291] Signal code: (-6)
>>>>>>> [asterix:14291] [ 0] /lib64/libpthread.so.0(+0x100d0)[0x7f90188160d0]
>>>>>>> [asterix:14291] [ 1] /lib64/libc.so.6(gsignal+0x37)[0x7f901847d8d7]
>>>>>>> [asterix:14291] [ 2] /lib64/libc.so.6(abort+0x16a)[0x7f901847f53a]
>>>>>>> [asterix:14291] [ 3] /lib64/libc.so.6(+0x77da3)[0x7f90184c0da3]
>>>>>>> [asterix:14291] [ 4] /lib64/libc.so.6(cfree+0x5b5)[0x7f90184cc9f5]
>>>>>>> [asterix:14291] [ 5] /home/balay/petsc/arch-ompi/lib/libmpi.so.1(ompi_datatype_release_args+0x12b)[0x7f9018c88627]
>>>>>>> [asterix:14291] [ 6] /home/balay/petsc/arch-ompi/lib/libmpi.so.1(+0x508e3)[0x7f9018c898e3]
>>>>>>> [asterix:14291] [ 7] /home/balay/petsc/arch-ompi/lib/openmpi/mca_pml_ob1.so(+0x11773)[0x7f900f739773]
>>>>>>> [asterix:14291] [ 8] /home/balay/petsc/arch-ompi/lib/openmpi/mca_pml_ob1.so(+0x12ece)[0x7f900f73aece]
>>>>>>> [asterix:14291] [ 9] /home/balay/petsc/arch-ompi/lib/openmpi/mca_osc_rdma.so(+0x862a)[0x7f900ecdb62a]
>>>>>>> [asterix:14291] [10] /home/balay/petsc/arch-ompi/lib/openmpi/mca_osc_rdma.so(+0x8a15)[0x7f900ecdba15]
>>>>>>> [asterix:14291] [11] /home/balay/petsc/arch-ompi/lib/openmpi/mca_osc_rdma.so(+0xbac7)[0x7f900ecdeac7]
>>>>>>> [asterix:14291] [12] /home/balay/petsc/arch-ompi/lib/openmpi/mca_pml_ob1.so(+0xc0de)[0x7f900f7340de]
>>>>>>> [asterix:14291] [13] /home/balay/petsc/arch-ompi/lib/openmpi/mca_pml_ob1.so(+0xc4eb)[0x7f900f7344eb]
>>>>>>> [asterix:14291] [14] /home/balay/petsc/arch-ompi/lib/openmpi/mca_pml_ob1.so(mca_pml_ob1_recv_frag_callback_match+0x2ed)[0x7f900f734f88]
>>>>>>> [asterix:14291] [15] /home/balay/petsc/arch-ompi/lib/openmpi/mca_btl_vader.so(+0x3876)[0x7f9014009876]
>>>>>>> [asterix:14291] [16] /home/balay/petsc/arch-ompi/lib/openmpi/mca_btl_vader.so(+0x4d83)[0x7f901400ad83]
>>>>>>> [asterix:14291] [17] /home/balay/petsc/arch-ompi/lib/libopen-pal.so.6(opal_progress+0xa2)[0x7f9017cca9f3]
>>>>>>> [asterix:14291] [18] /home/balay/petsc/arch-ompi/lib/openmpi/mca_pml_ob1.so(+0x770b)[0x7f900f72f70b]
>>>>>>> [asterix:14291] [19] /home/balay/petsc/arch-ompi/lib/openmpi/mca_pml_ob1.so(+0x77f5)[0x7f900f72f7f5]
>>>>>>> [asterix:14291] [20] /home/balay/petsc/arch-ompi/lib/openmpi/mca_pml_ob1.so(mca_pml_ob1_recv+0x1c6)[0x7f900f72ff6a]
>>>>>>> [asterix:14291] [21] /home/balay/petsc/arch-ompi/lib/openmpi/mca_coll_tuned.so(ompi_coll_tuned_scatter_intra_basic_linear+0x76)[0x7f900e689952]
>>>>>>> [asterix:14291] [22] /home/balay/petsc/arch-ompi/lib/openmpi/mca_coll_tuned.so(ompi_coll_tuned_scatter_intra_dec_fixed+0x112)[0x7f900e67666e]
>>>>>>> [asterix:14291] [23] /home/balay/petsc/arch-ompi/lib/openmpi/mca_coll_basic.so(mca_coll_basic_reduce_scatter_block_intra+0x188)[0x7f900f319bdc]
>>>>>>> [asterix:14291] [24] /home/balay/petsc/arch-ompi/lib/openmpi/mca_osc_rdma.so(ompi_osc_rdma_fence+0x125)[0x7f900ecdfc59]
>>>>>>> [asterix:14291] [25] /home/balay/petsc/arch-ompi/lib/libmpi.so.1(MPI_Win_fence+0x116)[0x7f9018cd1079]
>>>>>>> [asterix:14291] [26] /home/balay/petsc/arch-ompi/lib/libpetsc.so.3.05(+0x2d1d90)[0x7f901b5aad90]
>>>>>>> [asterix:14291] [27] /home/balay/petsc/arch-ompi/lib/libpetsc.so.3.05(PetscSFBcastEnd_Window+0x218)[0x7f901b5ad2db]
>>>>>>> [asterix:14291] [28] /home/balay/petsc/arch-ompi/lib/libpetsc.so.3.05(PetscSFBcastEnd+0x4eb)[0x7f901b5d43d7]
>>>>>>> [asterix:14291] [29] ./ex2[0x401dd4]
>>>>>>> [asterix:14291] *** End of error message ***
>>>>>>> --------------------------------------------------------------------------
>>>>>>> mpiexec noticed that process rank 0 with PID 14290 on node asterix exited on signal 6 (Aborted).
>>>>>>> --------------------------------------------------------------------------
>>>>>>> balay at asterix /home/balay/petsc/src/vec/is/sf/examples/tutorials (master=)
>>>>>>> $ /home/balay/petsc/arch-ompi/bin/mpiexec -n 2 valgrind --tool=memcheck -q ./ex2 -sf_type window
>>>>>>> PetscSF Object: 2 MPI processes
>>>>>>> type: window
>>>>>>> synchronization=FENCE sort=rank-order
>>>>>>> [0] Number of roots=1, leaves=2, remote ranks=2
>>>>>>> [0] 0 <- (0,0)
>>>>>>> [0] 1 <- (1,0)
>>>>>>> [1] Number of roots=1, leaves=2, remote ranks=2
>>>>>>> [1] 0 <- (1,0)
>>>>>>> [1] 1 <- (0,0)
>>>>>>> ==14349== Invalid write of size 2
>>>>>>> ==14349== at 0x4C2E36B: memcpy@@GLIBC_2.14 (in /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so)
>>>>>>> ==14349== by 0x8AFDABD: ompi_datatype_set_args (ompi_datatype_args.c:167)
>>>>>>> ==14349== by 0x8AFF0F3: __ompi_datatype_create_from_args (ompi_datatype_args.c:718)
>>>>>>> ==14349== by 0x8AFEC0E: __ompi_datatype_create_from_packed_description (ompi_datatype_args.c:649)
>>>>>>> ==14349== by 0x8AFF5D6: ompi_datatype_create_from_packed_description (ompi_datatype_args.c:788)
>>>>>>> ==14349== by 0xF727F0E: ompi_osc_base_datatype_create (osc_base_obj_convert.h:52)
>>>>>>> ==14349== by 0xF728424: datatype_create (osc_rdma_data_move.c:333)
>>>>>>> ==14349== by 0xF72887D: process_get (osc_rdma_data_move.c:536)
>>>>>>> ==14349== by 0xF72A856: process_frag (osc_rdma_data_move.c:1593)
>>>>>>> ==14349== by 0xF72AA35: ompi_osc_rdma_callback (osc_rdma_data_move.c:1656)
>>>>>>> ==14349== by 0xECCF0DD: ompi_request_complete (request.h:402)
>>>>>>> ==14349== by 0xECCF4EA: recv_request_pml_complete (pml_ob1_recvreq.h:181)
>>>>>>> ==14349== Address 0x101bf188 is 0 bytes after a block of size 72 alloc'd
>>>>>>> ==14349== at 0x4C29BCF: malloc (in /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so)
>>>>>>> ==14349== by 0x8AFD755: ompi_datatype_set_args (ompi_datatype_args.c:123)
>>>>>>> ==14349== by 0x8AFF0F3: __ompi_datatype_create_from_args (ompi_datatype_args.c:718)
>>>>>>> ==14349== by 0x8AFEC0E: __ompi_datatype_create_from_packed_description (ompi_datatype_args.c:649)
>>>>>>> ==14349== by 0x8AFF5D6: ompi_datatype_create_from_packed_description (ompi_datatype_args.c:788)
>>>>>>> ==14349== by 0xF727F0E: ompi_osc_base_datatype_create (osc_base_obj_convert.h:52)
>>>>>>> ==14349== by 0xF728424: datatype_create (osc_rdma_data_move.c:333)
>>>>>>> ==14349== by 0xF72887D: process_get (osc_rdma_data_move.c:536)
>>>>>>> ==14349== by 0xF72A856: process_frag (osc_rdma_data_move.c:1593)
>>>>>>> ==14349== by 0xF72AA35: ompi_osc_rdma_callback (osc_rdma_data_move.c:1656)
>>>>>>> ==14349== by 0xECCF0DD: ompi_request_complete (request.h:402)
>>>>>>> ==14349== by 0xECCF4EA: recv_request_pml_complete (pml_ob1_recvreq.h:181)
>>>>>>> ==14349==
>>>>>>> ==14348== Invalid write of size 2
>>>>>>> ==14348== at 0x4C2E36B: memcpy@@GLIBC_2.14 (in /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so)
>>>>>>> ==14348== by 0x8AFDABD: ompi_datatype_set_args (ompi_datatype_args.c:167)
>>>>>>> ==14348== by 0x8AFF0F3: __ompi_datatype_create_from_args (ompi_datatype_args.c:718)
>>>>>>> ==14348== by 0x8AFEC0E: __ompi_datatype_create_from_packed_description (ompi_datatype_args.c:649)
>>>>>>> ==14348== by 0x8AFF5D6: ompi_datatype_create_from_packed_description (ompi_datatype_args.c:788)
>>>>>>> ==14348== by 0xF727F0E: ompi_osc_base_datatype_create (osc_base_obj_convert.h:52)
>>>>>>> ==14348== by 0xF728424: datatype_create (osc_rdma_data_move.c:333)
>>>>>>> ==14348== by 0xF72887D: process_get (osc_rdma_data_move.c:536)
>>>>>>> ==14348== by 0xF72A856: process_frag (osc_rdma_data_move.c:1593)
>>>>>>> ==14348== by 0xF72AA35: ompi_osc_rdma_callback (osc_rdma_data_move.c:1656)
>>>>>>> ==14348== by 0xECCF0DD: ompi_request_complete (request.h:402)
>>>>>>> ==14348== by 0xECCF4EA: recv_request_pml_complete (pml_ob1_recvreq.h:181)
>>>>>>> ==14348== Address 0x101c71b8 is 0 bytes after a block of size 72 alloc'd
>>>>>>> ==14348== at 0x4C29BCF: malloc (in /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so)
>>>>>>> ==14348== by 0x8AFD755: ompi_datatype_set_args (ompi_datatype_args.c:123)
>>>>>>> ==14348== by 0x8AFF0F3: __ompi_datatype_create_from_args (ompi_datatype_args.c:718)
>>>>>>> ==14348== by 0x8AFEC0E: __ompi_datatype_create_from_packed_description (ompi_datatype_args.c:649)
>>>>>>> ==14348== by 0x8AFF5D6: ompi_datatype_create_from_packed_description (ompi_datatype_args.c:788)
>>>>>>> ==14348== by 0xF727F0E: ompi_osc_base_datatype_create (osc_base_obj_convert.h:52)
>>>>>>> ==14348== by 0xF728424: datatype_create (osc_rdma_data_move.c:333)
>>>>>>> ==14348== by 0xF72887D: process_get (osc_rdma_data_move.c:536)
>>>>>>> ==14348== by 0xF72A856: process_frag (osc_rdma_data_move.c:1593)
>>>>>>> ==14348== by 0xF72AA35: ompi_osc_rdma_callback (osc_rdma_data_move.c:1656)
>>>>>>> ==14348== by 0xECCF0DD: ompi_request_complete (request.h:402)
>>>>>>> ==14348== by 0xECCF4EA: recv_request_pml_complete (pml_ob1_recvreq.h:181)
>>>>>>> ==14348==
>>>>>>> Vec Object: 2 MPI processes
>>>>>>> type: mpi
>>>>>>> Process [0]
>>>>>>> 0
>>>>>>> 1
>>>>>>> Process [1]
>>>>>>> 1
>>>>>>> 0
>>>>>>> Vec Object: 2 MPI processes
>>>>>>> type: mpi
>>>>>>> Process [0]
>>>>>>> 10
>>>>>>> 11
>>>>>>> Process [1]
>>>>>>> 11
>>>>>>> 10
>>>>>>> balay at asterix /home/balay/petsc/src/vec/is/sf/examples/tutorials (master=)
>>>>>>> $
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>
>>>
>>
>>
>
More information about the petsc-dev
mailing list