[petsc-users] Debugging a petsc code

[Praveen C]

-malloc_test does not report anything.

Freeing all petsc vectors got rid of those error.

Now I see only MPI related errors like this

==33686== 376 (232 direct, 144 indirect) bytes in 1 blocks are definitely
lost in loss record 148 of 159

==33686==    at 0x4C2B0AF: malloc (in
/usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so)

==33686==    by 0x660D7EF: mca_bml_r2_add_procs (in
/home/spack/opt/spack/linux-opensuse20161217-x86_64/gcc-6/openmpi-2.0.1-asdjmd22cnyktv2athcx3ouhrozknk22/lib64/libmpi.so.20.0.1)

==33686==    by 0x66D11CA: mca_pml_ob1_add_procs (in
/home/spack/opt/spack/linux-opensuse20161217-x86_64/gcc-6/openmpi-2.0.1-asdjmd22cnyktv2athcx3ouhrozknk22/lib64/libmpi.so.20.0.1)

==33686==    by 0x65CE906: ompi_mpi_init (in
/home/spack/opt/spack/linux-opensuse20161217-x86_64/gcc-6/openmpi-2.0.1-asdjmd22cnyktv2athcx3ouhrozknk22/lib64/libmpi.so.20.0.1)

==33686==    by 0x65ED082: PMPI_Init (in
/home/spack/opt/spack/linux-opensuse20161217-x86_64/gcc-6/openmpi-2.0.1-asdjmd22cnyktv2athcx3ouhrozknk22/lib64/libmpi.so.20.0.1)

==33686==    by 0x6352D97: MPI_INIT (in
/home/spack/opt/spack/linux-opensuse20161217-x86_64/gcc-6/openmpi-2.0.1-asdjmd22cnyktv2athcx3ouhrozknk22/lib64/libmpi_mpifh.so.20.0.0)

==33686==    by 0x4F393C6: petscinitialize_ (zstart.c:320)

==33686==    by 0x417718: MAIN__ (all.f95:1385)

==33686==    by 0x4184B2: main (all.f95:1366)


Does this indicate some error in my code or my MPI.

This is the valgrind summary

==33686== LEAK SUMMARY:

==33686==    definitely lost: 1,378 bytes in 14 blocks

==33686==    indirectly lost: 64,882 bytes in 88 blocks

==33686==      possibly lost: 0 bytes in 0 blocks

==33686==    still reachable: 32,984 bytes in 139 blocks

==33686==         suppressed: 0 bytes in 0 blocks


I have attached the full valgrid output.


Thanks
praveen

On Tue, Jan 31, 2017 at 12:18 AM, Stefano Zampini <stefano.zampini at gmail.com
> wrote:

> It just reports that you have a memory leak. Probably you did not call
> VecDestroy on the Vec created at at initpetsc_ in line 2066 of all.f95.
>
> On Jan 30, 2017, at 8:04 PM, Praveen C <cpraveen at gmail.com> wrote:
>
> Dear all
>
> I am trying to find a possible bug in my fortran petsc code. Running
> valgrid I see messages like this
>
> ==28499== 1,596 (1,512 direct, 84 indirect) bytes in 1 blocks are
> definitely lost in loss record 174 of 194
> ==28499==    at 0x4C2D636: memalign (in /usr/lib64/valgrind/vgpreload_
> memcheck-amd64-linux.so)
> ==28499==    by 0x4F0F178: PetscMallocAlign (mal.c:28)
> ==28499==    by 0x4FF7E82: VecCreate (veccreate.c:37)
> ==28499==    by 0x4FDF198: VecCreateSeqWithArray (bvec2.c:946)
> ==28499==    by 0x4FE442E: veccreateseqwitharray_ (zbvec2f.c:12)
> ==28499==    by 0x406921: initpetsc_ (all.f95:2066)
> ==28499==    by 0x4035B1: run_ (all.f95:2817)
> ==28499==    by 0x41760C: MAIN__ (all.f95:1383)
> ==28499==    by 0x417D08: main (all.f95:1330)
>
> Does this indicate some bug in my code ?
>
> Thanks
> praveen
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20170131/043bbfff/attachment-0001.html>
-------------- next part --------------
==33706== Memcheck, a memory error detector
==33706== Copyright (C) 2002-2015, and GNU GPL'd, by Julian Seward et al.
==33706== Using Valgrind-3.12.0 and LibVEX; rerun with -h for copyright info
==33706== Command: ./ug3
==33706== 
==33706== 
==33706== HEAP SUMMARY:
==33706==     in use at exit: 99,244 bytes in 241 blocks
==33706==   total heap usage: 82,707 allocs, 82,466 frees, 23,824,048 bytes allocated
==33706== 
==33706== 10 bytes in 1 blocks are definitely lost in loss record 6 of 159
==33706==    at 0x4C2B0AF: malloc (in /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so)
==33706==    by 0xC717EEE: opal_pmix_pmix112_pmix_bfrop_unpack_string (in /home/spack/opt/spack/linux-opensuse20161217-x86_64/gcc-6/openmpi-2.0.1-asdjmd22cnyktv2athcx3ouhrozknk22/lib64/libopen-pal.so.20.1.0)
==33706==    by 0xC715FFB: opal_pmix_pmix112_pmix_bfrop_unpack (in /home/spack/opt/spack/linux-opensuse20161217-x86_64/gcc-6/openmpi-2.0.1-asdjmd22cnyktv2athcx3ouhrozknk22/lib64/libopen-pal.so.20.1.0)
==33706==    by 0xC6F74CF: job_data (in /home/spack/opt/spack/linux-opensuse20161217-x86_64/gcc-6/openmpi-2.0.1-asdjmd22cnyktv2athcx3ouhrozknk22/lib64/libopen-pal.so.20.1.0)
==33706==    by 0xC723831: opal_pmix_pmix112_pmix_usock_process_msg (in /home/spack/opt/spack/linux-opensuse20161217-x86_64/gcc-6/openmpi-2.0.1-asdjmd22cnyktv2athcx3ouhrozknk22/lib64/libopen-pal.so.20.1.0)
==33706==    by 0xC6C1ED7: event_process_active_single_queue (event.c:1370)
==33706==    by 0xC6C1ED7: event_process_active (event.c:1440)
==33706==    by 0xC6C1ED7: opal_libevent2022_event_base_loop (event.c:1644)
==33706==    by 0xC713A6C: progress_engine (in /home/spack/opt/spack/linux-opensuse20161217-x86_64/gcc-6/openmpi-2.0.1-asdjmd22cnyktv2athcx3ouhrozknk22/lib64/libopen-pal.so.20.1.0)
==33706==    by 0x73F4453: start_thread (in /lib64/libpthread-2.24.so)
==33706==    by 0x76F337E: clone (in /lib64/libc-2.24.so)
==33706== 
==33706== 25 bytes in 1 blocks are definitely lost in loss record 21 of 159
==33706==    at 0x4C2B0AF: malloc (in /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so)
==33706==    by 0x767B137: vasprintf (in /lib64/libc-2.24.so)
==33706==    by 0x765A936: asprintf (in /lib64/libc-2.24.so)
==33706==    by 0xC3A5DD4: rte_init (in /home/spack/opt/spack/linux-opensuse20161217-x86_64/gcc-6/openmpi-2.0.1-asdjmd22cnyktv2athcx3ouhrozknk22/lib64/libopen-rte.so.20.0.0)
==33706==    by 0xC3683E4: orte_init (in /home/spack/opt/spack/linux-opensuse20161217-x86_64/gcc-6/openmpi-2.0.1-asdjmd22cnyktv2athcx3ouhrozknk22/lib64/libopen-rte.so.20.0.0)
==33706==    by 0x65CE425: ompi_mpi_init (in /home/spack/opt/spack/linux-opensuse20161217-x86_64/gcc-6/openmpi-2.0.1-asdjmd22cnyktv2athcx3ouhrozknk22/lib64/libmpi.so.20.0.1)
==33706==    by 0x65ED082: PMPI_Init (in /home/spack/opt/spack/linux-opensuse20161217-x86_64/gcc-6/openmpi-2.0.1-asdjmd22cnyktv2athcx3ouhrozknk22/lib64/libmpi.so.20.0.1)
==33706==    by 0x6352D97: MPI_INIT (in /home/spack/opt/spack/linux-opensuse20161217-x86_64/gcc-6/openmpi-2.0.1-asdjmd22cnyktv2athcx3ouhrozknk22/lib64/libmpi_mpifh.so.20.0.0)
==33706==    by 0x4F393C6: petscinitialize_ (zstart.c:320)
==33706==    by 0x417718: MAIN__ (all.f95:1385)
==33706==    by 0x4184B2: main (all.f95:1366)
==33706== 
==33706== 30 bytes in 1 blocks are definitely lost in loss record 24 of 159
==33706==    at 0x4C2B0AF: malloc (in /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so)
==33706==    by 0x767B137: vasprintf (in /lib64/libc-2.24.so)
==33706==    by 0x765A936: asprintf (in /lib64/libc-2.24.so)
==33706==    by 0xC3A5E01: rte_init (in /home/spack/opt/spack/linux-opensuse20161217-x86_64/gcc-6/openmpi-2.0.1-asdjmd22cnyktv2athcx3ouhrozknk22/lib64/libopen-rte.so.20.0.0)
==33706==    by 0xC3683E4: orte_init (in /home/spack/opt/spack/linux-opensuse20161217-x86_64/gcc-6/openmpi-2.0.1-asdjmd22cnyktv2athcx3ouhrozknk22/lib64/libopen-rte.so.20.0.0)
==33706==    by 0x65CE425: ompi_mpi_init (in /home/spack/opt/spack/linux-opensuse20161217-x86_64/gcc-6/openmpi-2.0.1-asdjmd22cnyktv2athcx3ouhrozknk22/lib64/libmpi.so.20.0.1)
==33706==    by 0x65ED082: PMPI_Init (in /home/spack/opt/spack/linux-opensuse20161217-x86_64/gcc-6/openmpi-2.0.1-asdjmd22cnyktv2athcx3ouhrozknk22/lib64/libmpi.so.20.0.1)
==33706==    by 0x6352D97: MPI_INIT (in /home/spack/opt/spack/linux-opensuse20161217-x86_64/gcc-6/openmpi-2.0.1-asdjmd22cnyktv2athcx3ouhrozknk22/lib64/libmpi_mpifh.so.20.0.0)
==33706==    by 0x4F393C6: petscinitialize_ (zstart.c:320)
==33706==    by 0x417718: MAIN__ (all.f95:1385)
==33706==    by 0x4184B2: main (all.f95:1366)
==33706== 
==33706== 48 bytes in 1 blocks are definitely lost in loss record 118 of 159
==33706==    at 0x4C2B0AF: malloc (in /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so)
==33706==    by 0xC68FFFD: mca_base_component_find (in /home/spack/opt/spack/linux-opensuse20161217-x86_64/gcc-6/openmpi-2.0.1-asdjmd22cnyktv2athcx3ouhrozknk22/lib64/libopen-pal.so.20.1.0)
==33706==    by 0xC69A099: mca_base_framework_components_register (in /home/spack/opt/spack/linux-opensuse20161217-x86_64/gcc-6/openmpi-2.0.1-asdjmd22cnyktv2athcx3ouhrozknk22/lib64/libopen-pal.so.20.1.0)
==33706==    by 0xC69A483: mca_base_framework_register (in /home/spack/opt/spack/linux-opensuse20161217-x86_64/gcc-6/openmpi-2.0.1-asdjmd22cnyktv2athcx3ouhrozknk22/lib64/libopen-pal.so.20.1.0)
==33706==    by 0xC69A4F0: mca_base_framework_open (in /home/spack/opt/spack/linux-opensuse20161217-x86_64/gcc-6/openmpi-2.0.1-asdjmd22cnyktv2athcx3ouhrozknk22/lib64/libopen-pal.so.20.1.0)
==33706==    by 0xC6D4371: patcher_query (in /home/spack/opt/spack/linux-opensuse20161217-x86_64/gcc-6/openmpi-2.0.1-asdjmd22cnyktv2athcx3ouhrozknk22/lib64/libopen-pal.so.20.1.0)
==33706==    by 0xC6D4255: opal_memory_base_open (in /home/spack/opt/spack/linux-opensuse20161217-x86_64/gcc-6/openmpi-2.0.1-asdjmd22cnyktv2athcx3ouhrozknk22/lib64/libopen-pal.so.20.1.0)
==33706==    by 0xC69A560: mca_base_framework_open (in /home/spack/opt/spack/linux-opensuse20161217-x86_64/gcc-6/openmpi-2.0.1-asdjmd22cnyktv2athcx3ouhrozknk22/lib64/libopen-pal.so.20.1.0)
==33706==    by 0xC675386: opal_init (in /home/spack/opt/spack/linux-opensuse20161217-x86_64/gcc-6/openmpi-2.0.1-asdjmd22cnyktv2athcx3ouhrozknk22/lib64/libopen-pal.so.20.1.0)
==33706==    by 0xC36827A: orte_init (in /home/spack/opt/spack/linux-opensuse20161217-x86_64/gcc-6/openmpi-2.0.1-asdjmd22cnyktv2athcx3ouhrozknk22/lib64/libopen-rte.so.20.0.0)
==33706==    by 0x65CE425: ompi_mpi_init (in /home/spack/opt/spack/linux-opensuse20161217-x86_64/gcc-6/openmpi-2.0.1-asdjmd22cnyktv2athcx3ouhrozknk22/lib64/libmpi.so.20.0.1)
==33706==    by 0x65ED082: PMPI_Init (in /home/spack/opt/spack/linux-opensuse20161217-x86_64/gcc-6/openmpi-2.0.1-asdjmd22cnyktv2athcx3ouhrozknk22/lib64/libmpi.so.20.0.1)
==33706== 
==33706== 72 bytes in 1 blocks are definitely lost in loss record 129 of 159
==33706==    at 0x4C2D2CF: realloc (in /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so)
==33706==    by 0x767B10D: vasprintf (in /lib64/libc-2.24.so)
==33706==    by 0x765A936: asprintf (in /lib64/libc-2.24.so)
==33706==    by 0xC3A5D5D: rte_init (in /home/spack/opt/spack/linux-opensuse20161217-x86_64/gcc-6/openmpi-2.0.1-asdjmd22cnyktv2athcx3ouhrozknk22/lib64/libopen-rte.so.20.0.0)
==33706==    by 0xC3683E4: orte_init (in /home/spack/opt/spack/linux-opensuse20161217-x86_64/gcc-6/openmpi-2.0.1-asdjmd22cnyktv2athcx3ouhrozknk22/lib64/libopen-rte.so.20.0.0)
==33706==    by 0x65CE425: ompi_mpi_init (in /home/spack/opt/spack/linux-opensuse20161217-x86_64/gcc-6/openmpi-2.0.1-asdjmd22cnyktv2athcx3ouhrozknk22/lib64/libmpi.so.20.0.1)
==33706==    by 0x65ED082: PMPI_Init (in /home/spack/opt/spack/linux-opensuse20161217-x86_64/gcc-6/openmpi-2.0.1-asdjmd22cnyktv2athcx3ouhrozknk22/lib64/libmpi.so.20.0.1)
==33706==    by 0x6352D97: MPI_INIT (in /home/spack/opt/spack/linux-opensuse20161217-x86_64/gcc-6/openmpi-2.0.1-asdjmd22cnyktv2athcx3ouhrozknk22/lib64/libmpi_mpifh.so.20.0.0)
==33706==    by 0x4F393C6: petscinitialize_ (zstart.c:320)
==33706==    by 0x417718: MAIN__ (all.f95:1385)
==33706==    by 0x4184B2: main (all.f95:1366)
==33706== 
==33706== 113 bytes in 4 blocks are definitely lost in loss record 136 of 159
==33706==    at 0x4C2B0AF: malloc (in /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so)
==33706==    by 0x768B689: strdup (in /lib64/libc-2.24.so)
==33706==    by 0xC3A5904: rte_init (in /home/spack/opt/spack/linux-opensuse20161217-x86_64/gcc-6/openmpi-2.0.1-asdjmd22cnyktv2athcx3ouhrozknk22/lib64/libopen-rte.so.20.0.0)
==33706==    by 0xC3683E4: orte_init (in /home/spack/opt/spack/linux-opensuse20161217-x86_64/gcc-6/openmpi-2.0.1-asdjmd22cnyktv2athcx3ouhrozknk22/lib64/libopen-rte.so.20.0.0)
==33706==    by 0x65CE425: ompi_mpi_init (in /home/spack/opt/spack/linux-opensuse20161217-x86_64/gcc-6/openmpi-2.0.1-asdjmd22cnyktv2athcx3ouhrozknk22/lib64/libmpi.so.20.0.1)
==33706==    by 0x65ED082: PMPI_Init (in /home/spack/opt/spack/linux-opensuse20161217-x86_64/gcc-6/openmpi-2.0.1-asdjmd22cnyktv2athcx3ouhrozknk22/lib64/libmpi.so.20.0.1)
==33706==    by 0x6352D97: MPI_INIT (in /home/spack/opt/spack/linux-opensuse20161217-x86_64/gcc-6/openmpi-2.0.1-asdjmd22cnyktv2athcx3ouhrozknk22/lib64/libmpi_mpifh.so.20.0.0)
==33706==    by 0x4F393C6: petscinitialize_ (zstart.c:320)
==33706==    by 0x417718: MAIN__ (all.f95:1385)
==33706==    by 0x4184B2: main (all.f95:1366)
==33706== 
==33706== 317 (56 direct, 261 indirect) bytes in 1 blocks are definitely lost in loss record 146 of 159
==33706==    at 0x4C2B0AF: malloc (in /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so)
==33706==    by 0xC6F4470: _putfn (in /home/spack/opt/spack/linux-opensuse20161217-x86_64/gcc-6/openmpi-2.0.1-asdjmd22cnyktv2athcx3ouhrozknk22/lib64/libopen-pal.so.20.1.0)
==33706==    by 0xC6C1ED7: event_process_active_single_queue (event.c:1370)
==33706==    by 0xC6C1ED7: event_process_active (event.c:1440)
==33706==    by 0xC6C1ED7: opal_libevent2022_event_base_loop (event.c:1644)
==33706==    by 0xC713A6C: progress_engine (in /home/spack/opt/spack/linux-opensuse20161217-x86_64/gcc-6/openmpi-2.0.1-asdjmd22cnyktv2athcx3ouhrozknk22/lib64/libopen-pal.so.20.1.0)
==33706==    by 0x73F4453: start_thread (in /lib64/libpthread-2.24.so)
==33706==    by 0x76F337E: clone (in /lib64/libc-2.24.so)
==33706== 
==33706== 376 (232 direct, 144 indirect) bytes in 1 blocks are definitely lost in loss record 148 of 159
==33706==    at 0x4C2B0AF: malloc (in /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so)
==33706==    by 0x660D7EF: mca_bml_r2_add_procs (in /home/spack/opt/spack/linux-opensuse20161217-x86_64/gcc-6/openmpi-2.0.1-asdjmd22cnyktv2athcx3ouhrozknk22/lib64/libmpi.so.20.0.1)
==33706==    by 0x66D11CA: mca_pml_ob1_add_procs (in /home/spack/opt/spack/linux-opensuse20161217-x86_64/gcc-6/openmpi-2.0.1-asdjmd22cnyktv2athcx3ouhrozknk22/lib64/libmpi.so.20.0.1)
==33706==    by 0x65CE906: ompi_mpi_init (in /home/spack/opt/spack/linux-opensuse20161217-x86_64/gcc-6/openmpi-2.0.1-asdjmd22cnyktv2athcx3ouhrozknk22/lib64/libmpi.so.20.0.1)
==33706==    by 0x65ED082: PMPI_Init (in /home/spack/opt/spack/linux-opensuse20161217-x86_64/gcc-6/openmpi-2.0.1-asdjmd22cnyktv2athcx3ouhrozknk22/lib64/libmpi.so.20.0.1)
==33706==    by 0x6352D97: MPI_INIT (in /home/spack/opt/spack/linux-opensuse20161217-x86_64/gcc-6/openmpi-2.0.1-asdjmd22cnyktv2athcx3ouhrozknk22/lib64/libmpi_mpifh.so.20.0.0)
==33706==    by 0x4F393C6: petscinitialize_ (zstart.c:320)
==33706==    by 0x417718: MAIN__ (all.f95:1385)
==33706==    by 0x4184B2: main (all.f95:1366)
==33706== 
==33706== 632 bytes in 1 blocks are definitely lost in loss record 150 of 159
==33706==    at 0x4C2B0AF: malloc (in /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so)
==33706==    by 0xC6F7A4F: pmix_client_deregister_errhandler (in /home/spack/opt/spack/linux-opensuse20161217-x86_64/gcc-6/openmpi-2.0.1-asdjmd22cnyktv2athcx3ouhrozknk22/lib64/libopen-pal.so.20.1.0)
==33706==    by 0xC6FDA34: OPAL_PMIX_PMIX112_PMIx_Deregister_errhandler (in /home/spack/opt/spack/linux-opensuse20161217-x86_64/gcc-6/openmpi-2.0.1-asdjmd22cnyktv2athcx3ouhrozknk22/lib64/libopen-pal.so.20.1.0)
==33706==    by 0xC6DC62F: pmix1_client_finalize (in /home/spack/opt/spack/linux-opensuse20161217-x86_64/gcc-6/openmpi-2.0.1-asdjmd22cnyktv2athcx3ouhrozknk22/lib64/libopen-pal.so.20.1.0)
==33706==    by 0xC3A4EBD: rte_finalize (in /home/spack/opt/spack/linux-opensuse20161217-x86_64/gcc-6/openmpi-2.0.1-asdjmd22cnyktv2athcx3ouhrozknk22/lib64/libopen-rte.so.20.0.0)
==33706==    by 0xC3680A1: orte_finalize (in /home/spack/opt/spack/linux-opensuse20161217-x86_64/gcc-6/openmpi-2.0.1-asdjmd22cnyktv2athcx3ouhrozknk22/lib64/libopen-rte.so.20.0.0)
==33706==    by 0x65CF2BD: ompi_mpi_finalize (in /home/spack/opt/spack/linux-opensuse20161217-x86_64/gcc-6/openmpi-2.0.1-asdjmd22cnyktv2athcx3ouhrozknk22/lib64/libmpi.so.20.0.1)
==33706==    by 0x4F2D000: PetscFinalize (pinit.c:1428)
==33706==    by 0x4F39972: petscfinalize_ (zstart.c:495)
==33706==    by 0x418451: MAIN__ (all.f95:1438)
==33706==    by 0x4184B2: main (all.f95:1366)
==33706== 
==33706== 1,116 (104 direct, 1,012 indirect) bytes in 1 blocks are definitely lost in loss record 155 of 159
==33706==    at 0x4C2B0AF: malloc (in /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so)
==33706==    by 0xC6E8F5A: opal_pmix_pmix112_pmix_hash_store (in /home/spack/opt/spack/linux-opensuse20161217-x86_64/gcc-6/openmpi-2.0.1-asdjmd22cnyktv2athcx3ouhrozknk22/lib64/libopen-pal.so.20.1.0)
==33706==    by 0xC6F6F2A: opal_pmix_pmix112_pmix_client_process_nspace_blob (in /home/spack/opt/spack/linux-opensuse20161217-x86_64/gcc-6/openmpi-2.0.1-asdjmd22cnyktv2athcx3ouhrozknk22/lib64/libopen-pal.so.20.1.0)
==33706==    by 0xC6F7522: job_data (in /home/spack/opt/spack/linux-opensuse20161217-x86_64/gcc-6/openmpi-2.0.1-asdjmd22cnyktv2athcx3ouhrozknk22/lib64/libopen-pal.so.20.1.0)
==33706==    by 0xC723831: opal_pmix_pmix112_pmix_usock_process_msg (in /home/spack/opt/spack/linux-opensuse20161217-x86_64/gcc-6/openmpi-2.0.1-asdjmd22cnyktv2athcx3ouhrozknk22/lib64/libopen-pal.so.20.1.0)
==33706==    by 0xC6C1ED7: event_process_active_single_queue (event.c:1370)
==33706==    by 0xC6C1ED7: event_process_active (event.c:1440)
==33706==    by 0xC6C1ED7: opal_libevent2022_event_base_loop (event.c:1644)
==33706==    by 0xC713A6C: progress_engine (in /home/spack/opt/spack/linux-opensuse20161217-x86_64/gcc-6/openmpi-2.0.1-asdjmd22cnyktv2athcx3ouhrozknk22/lib64/libopen-pal.so.20.1.0)
==33706==    by 0x73F4453: start_thread (in /lib64/libpthread-2.24.so)
==33706==    by 0x76F337E: clone (in /lib64/libc-2.24.so)
==33706== 
==33706== 63,521 (56 direct, 63,465 indirect) bytes in 1 blocks are definitely lost in loss record 159 of 159
==33706==    at 0x4C2B0AF: malloc (in /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so)
==33706==    by 0xC6F681E: opal_pmix_pmix112_pmix_client_process_nspace_blob (in /home/spack/opt/spack/linux-opensuse20161217-x86_64/gcc-6/openmpi-2.0.1-asdjmd22cnyktv2athcx3ouhrozknk22/lib64/libopen-pal.so.20.1.0)
==33706==    by 0xC6F7522: job_data (in /home/spack/opt/spack/linux-opensuse20161217-x86_64/gcc-6/openmpi-2.0.1-asdjmd22cnyktv2athcx3ouhrozknk22/lib64/libopen-pal.so.20.1.0)
==33706==    by 0xC723831: opal_pmix_pmix112_pmix_usock_process_msg (in /home/spack/opt/spack/linux-opensuse20161217-x86_64/gcc-6/openmpi-2.0.1-asdjmd22cnyktv2athcx3ouhrozknk22/lib64/libopen-pal.so.20.1.0)
==33706==    by 0xC6C1ED7: event_process_active_single_queue (event.c:1370)
==33706==    by 0xC6C1ED7: event_process_active (event.c:1440)
==33706==    by 0xC6C1ED7: opal_libevent2022_event_base_loop (event.c:1644)
==33706==    by 0xC713A6C: progress_engine (in /home/spack/opt/spack/linux-opensuse20161217-x86_64/gcc-6/openmpi-2.0.1-asdjmd22cnyktv2athcx3ouhrozknk22/lib64/libopen-pal.so.20.1.0)
==33706==    by 0x73F4453: start_thread (in /lib64/libpthread-2.24.so)
==33706==    by 0x76F337E: clone (in /lib64/libc-2.24.so)
==33706== 
==33706== LEAK SUMMARY:
==33706==    definitely lost: 1,378 bytes in 14 blocks
==33706==    indirectly lost: 64,882 bytes in 88 blocks
==33706==      possibly lost: 0 bytes in 0 blocks
==33706==    still reachable: 32,984 bytes in 139 blocks
==33706==         suppressed: 0 bytes in 0 blocks
==33706== Reachable blocks (those to which a pointer was found) are not shown.
==33706== To see them, rerun with: --leak-check=full --show-leak-kinds=all
==33706== 
==33706== For counts of detected and suppressed errors, rerun with: -v
==33706== ERROR SUMMARY: 11 errors from 11 contexts (suppressed: 0 from 0)