[mpich-discuss] mpich-discuss Digest, Vol 44, Issue 36
Jeff Hammond
jhammond at alcf.anl.gov
Fri May 25 09:46:11 CDT 2012
The error is pretty obvious in the output:
PMPI_Gatherv(335): sendbuf cannot be MPI_IN_PLACE
You cannot use MPI_IN_PLACE with MPI_Gather
(https://svn.mpi-forum.org/trac/mpi-forum-web/ticket/163) but it is
allowed in MPI_Gatherv as of MPI 2.2, so I don't know why the
implementation does not allow this.
Jeff
On Fri, May 25, 2012 at 8:21 AM, Albert Spade <albert.spade at gmail.com> wrote:
> Thanks Rajeev and Darius,
>
> I tried to use MPI_IN_PLACE but not getting the desired results. Can you
> please tell me how to make it working.
>
> This is the previous code :
>
> //MPI::COMM_WORLD.Gatherv((const
> void*)(Data+StartFrom[nStages-1][rank]), Count[rank], MPI::CHAR,
> (void*)(Data), Count, Displ, MPI::CHAR, 0);
>
> And this is how I changed it.
>
> MPI::COMM_WORLD.Gatherv(MPI_IN_PLACE, Count[rank], MPI::CHAR,
> (void*)(Data), Count, Displ, MPI::CHAR, 0);
>
> Am I doing it wrong?
>
> Thanks.
>
> My output after making above changes.
> ==============================
> [root at beowulf programs]# mpiexec -n 1 ./output
> Time taken for 16 elements using 1 processors = 2.81334e-05 seconds
> [root at beowulf programs]# mpiexec -n 2 ./output
> Fatal error in PMPI_Gatherv: Invalid buffer pointer, error stack:
> PMPI_Gatherv(398): MPI_Gatherv failed(sbuf=MPI_IN_PLACE, scount=64,
> MPI_CHAR, rbuf=0x879d500, rcnts=0x879d6b8, displs=0x879d6c8, MPI_CHAR,
> root=0, MPI_COMM_WORLD) failed
> PMPI_Gatherv(335): sendbuf cannot be MPI_IN_PLACE
>
> =====================================================================================
> = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
> = EXIT CODE: 256
> = CLEANING UP REMAINING PROCESSES
> = YOU CAN IGNORE THE BELOW CLEANUP MESSAGES
> =====================================================================================
> *** glibc detected *** mpiexec: double free or corruption (fasttop):
> 0x094fb038 ***
> ======= Backtrace: =========
> /lib/libc.so.6[0x7d4a31]
> mpiexec[0x8077b11]
> mpiexec[0x8053c7f]
> mpiexec[0x8053e73]
> mpiexec[0x805592a]
> mpiexec[0x8077186]
> mpiexec[0x807639e]
> mpiexec[0x80518f8]
> mpiexec[0x804ad65]
> /lib/libc.so.6(__libc_start_main+0xe6)[0x77cce6]
> mpiexec[0x804a061]
> ======= Memory map: ========
> 00547000-00548000 r-xp 00000000 00:00 0 [vdso]
> 0054b000-0068f000 r-xp 00000000 fd:00 939775 /usr/lib/libxml2.so.2.7.6
> 0068f000-00694000 rw-p 00143000 fd:00 939775 /usr/lib/libxml2.so.2.7.6
> 00694000-00695000 rw-p 00000000 00:00 0
> 00740000-0075e000 r-xp 00000000 fd:00 2105890 /lib/ld-2.12.so
> 0075e000-0075f000 r--p 0001d000 fd:00 2105890 /lib/ld-2.12.so
> 0075f000-00760000 rw-p 0001e000 fd:00 2105890 /lib/ld-2.12.so
> 00766000-008ef000 r-xp 00000000 fd:00 2105891 /lib/libc-2.12.so
> 008ef000-008f0000 ---p 00189000 fd:00 2105891 /lib/libc-2.12.so
> 008f0000-008f2000 r--p 00189000 fd:00 2105891 /lib/libc-2.12.so
> 008f2000-008f3000 rw-p 0018b000 fd:00 2105891 /lib/libc-2.12.so
> 008f3000-008f6000 rw-p 00000000 00:00 0
> 008f8000-008fb000 r-xp 00000000 fd:00 2105893 /lib/libdl-2.12.so
> 008fb000-008fc000 r--p 00002000 fd:00 2105893 /lib/libdl-2.12.so
> 008fc000-008fd000 rw-p 00003000 fd:00 2105893 /lib/libdl-2.12.so
> 008ff000-00916000 r-xp 00000000 fd:00 2105900 /lib/libpthread-2.12.so
> 00916000-00917000 r--p 00016000 fd:00 2105900 /lib/libpthread-2.12.so
> 00917000-00918000 rw-p 00017000 fd:00 2105900 /lib/libpthread-2.12.so
> 00918000-0091a000 rw-p 00000000 00:00 0
> 0091c000-0092e000 r-xp 00000000 fd:00 2105904 /lib/libz.so.1.2.3
> 0092e000-0092f000 r--p 00011000 fd:00 2105904 /lib/libz.so.1.2.3
> 0092f000-00930000 rw-p 00012000 fd:00 2105904 /lib/libz.so.1.2.3
> 00932000-0095a000 r-xp 00000000 fd:00 2098429 /lib/libm-2.12.so
> 0095a000-0095b000 r--p 00027000 fd:00 2098429 /lib/libm-2.12.so
> 0095b000-0095c000 rw-p 00028000 fd:00 2098429 /lib/libm-2.12.so
> 00bb0000-00bcd000 r-xp 00000000 fd:00 2105914
> /lib/libgcc_s-4.4.6-20110824.so.1
> 00bcd000-00bce000 rw-p 0001d000 fd:00 2105914
> /lib/libgcc_s-4.4.6-20110824.so.1
> 00c18000-00c24000 r-xp 00000000 fd:00 2098123 /lib/libnss_files-2.12.so
> 00c24000-00c25000 r--p 0000b000 fd:00 2098123 /lib/libnss_files-2.12.so
> 00c25000-00c26000 rw-p 0000c000 fd:00 2098123 /lib/libnss_files-2.12.so
> 00ce9000-00d00000 r-xp 00000000 fd:00 2105929 /lib/libnsl-2.12.so
> 00d00000-00d01000 r--p 00016000 fd:00 2105929 /lib/libnsl-2.12.so
> 00d01000-00d02000 rw-p 00017000 fd:00 2105929 /lib/libnsl-2.12.so
> 00d02000-00d04000 rw-p 00000000 00:00 0
> 08048000-080a0000 r-xp 00000000 fd:00 656990
> /opt/mpich2-1.4.1p1/bin/bin/mpiexec.hydra
> 080a0000-080a1000 rw-p 00058000 fd:00 656990
> /opt/mpich2-1.4.1p1/bin/bin/mpiexec.hydra
> 080a1000-080a3000 rw-p 00000000 00:00 0
> 094ee000-0950f000 rw-p 00000000 00:00 0 [heap]
> b7893000-b7896000 rw-p 00000000 00:00 0
> b78a4000-b78a7000 rw-p 00000000 00:00 0
> bff80000-bff95000 rw-p 00000000 00:00 0 [stack]
> Aborted (core dumped)
> [root at beowulf programs]#
>
>
> On Tue, May 22, 2012 at 10:30 PM, <mpich-discuss-request at mcs.anl.gov> wrote:
>>
>> Send mpich-discuss mailing list submissions to
>> mpich-discuss at mcs.anl.gov
>>
>> To subscribe or unsubscribe via the World Wide Web, visit
>> https://lists.mcs.anl.gov/mailman/listinfo/mpich-discuss
>> or, via email, send a message with subject or body 'help' to
>> mpich-discuss-request at mcs.anl.gov
>>
>> You can reach the person managing the list at
>> mpich-discuss-owner at mcs.anl.gov
>>
>> When replying, please edit your Subject line so it is more specific
>> than "Re: Contents of mpich-discuss digest..."
>>
>>
>> Today's Topics:
>>
>> 1. Unable to run program parallely on cluster... Its running
>> properly on single machine... (Albert Spade)
>> 2. Not able to run program parallely on cluster... (Albert Spade)
>> 3. Re: Unable to run program parallely on cluster... Its
>> running properly on single machine... (Darius Buntinas)
>> 4. Re: Not able to run program parallely on cluster...
>> (Rajeev Thakur)
>> 5. replication of mpi applications (Thomas Ropars)
>>
>>
>> ----------------------------------------------------------------------
>>
>> Message: 1
>> Date: Tue, 22 May 2012 00:12:24 +0530
>> From: Albert Spade <albert.spade at gmail.com>
>> To: mpich-discuss at mcs.anl.gov
>> Subject: [mpich-discuss] Unable to run program parallely on cluster...
>> Its running properly on single machine...
>> Message-ID:
>>
>> <CAP2uaQopgOwaFNfCF49gcnW9REw8CQtWGMgf0U8RyNYStTFw1A at mail.gmail.com>
>> Content-Type: text/plain; charset="iso-8859-1"
>>
>> Hi everybody,
>>
>> I am using mpich2-1.4.1p1 and mpiexec from hydra-1.5b1
>> I have a cluster of 5 machines.
>> When I am trying to run the program for parallel fast fourier transform on
>> single machine it runs correctly but on a cluster it gives error.
>> Can you please tell me why its happening.
>>
>> Thanks.
>>
>> Here is my sample output:
>>
>> ---------------------------------------------------------------------------------------
>>
>> [root at beowulf programs]# mpiexec -n 1 ./Radix2
>> Time taken for 16 elements using 1 processors = 2.7895e-05 seconds
>> [root at beowulf programs]#
>> [root at beowulf programs]# mpiexec -n 4 ./Radix2
>> [mpiexec at beowulf.master] control_cb (./pm/pmiserv/pmiserv_cb.c:197):
>> assert
>> (!closed) failed
>> [mpiexec at beowulf.master] HYDT_dmxu_poll_wait_for_event
>> (./tools/demux/demux_poll.c:77): callback returned error status
>> [mpiexec at beowulf.master] HYD_pmci_wait_for_completion
>> (./pm/pmiserv/pmiserv_pmci.c:205): error waiting for event
>> [mpiexec at beowulf.master] main (./ui/mpich/mpiexec.c:437): process manager
>> error waiting for completion
>> [root at beowulf programs]# mpiexec -n 2 ./Radix2
>> [mpiexec at beowulf.master] control_cb (./pm/pmiserv/pmiserv_cb.c:197):
>> assert
>> (!closed) failed
>> [mpiexec at beowulf.master] HYDT_dmxu_poll_wait_for_event
>> (./tools/demux/demux_poll.c:77): callback returned error status
>> [mpiexec at beowulf.master] HYD_pmci_wait_for_completion
>> (./pm/pmiserv/pmiserv_pmci.c:205): error waiting for event
>> [mpiexec at beowulf.master] main (./ui/mpich/mpiexec.c:437): process manager
>> error waiting for completion
>> [root at beowulf programs]# mpiexec -n 4 ./Radix2
>> [mpiexec at beowulf.master] control_cb (./pm/pmiserv/pmiserv_cb.c:197):
>> assert
>> (!closed) failed
>> [mpiexec at beowulf.master] HYDT_dmxu_poll_wait_for_event
>> (./tools/demux/demux_poll.c:77): callback returned error status
>> [mpiexec at beowulf.master] HYD_pmci_wait_for_completion
>> (./pm/pmiserv/pmiserv_pmci.c:205): error waiting for event
>> [mpiexec at beowulf.master] main (./ui/mpich/mpiexec.c:437): process manager
>> error waiting for completion
>> [root at beowulf programs]#
>> -------------- next part --------------
>> An HTML attachment was scrubbed...
>> URL:
>> <http://lists.mcs.anl.gov/pipermail/mpich-discuss/attachments/20120522/25975b06/attachment-0001.html>
>>
>> ------------------------------
>>
>> Message: 2
>> Date: Tue, 22 May 2012 00:59:27 +0530
>> From: Albert Spade <albert.spade at gmail.com>
>> To: mpich-discuss at mcs.anl.gov
>> Subject: [mpich-discuss] Not able to run program parallely on
>> cluster...
>> Message-ID:
>>
>> <CAP2uaQpiMV0yqHsHfsWpgAQ=_K3M_ZGxsCm-S5BPvzbxH+Z9zQ at mail.gmail.com>
>> Content-Type: text/plain; charset="iso-8859-1"
>>
>> This is my new error after making few changes...
>> Results are quite similar... No succes with cluster...
>>
>> Sample run
>> --------------------------------------------------------
>>
>> [root at beowulf testing]# mpiexec -n 1 ./Radix
>> Time taken for 16 elements using 1 processors = 4.72069e-05 seconds
>> [root at beowulf testing]# mpiexec -n 2 ./Radix
>> Fatal error in PMPI_Gatherv: Internal MPI error!, error stack:
>> PMPI_Gatherv(398).....: MPI_Gatherv failed(sbuf=0x97d0500, scount=64,
>> MPI_CHAR, rbuf=0x97d0500, rcnts=0x97d06b8, displs=0x97d06c8, MPI_CHAR,
>> root=0, MPI_COMM_WORLD) failed
>> MPIR_Gatherv_impl(210):
>> MPIR_Gatherv(104).....:
>> MPIR_Localcopy(357)...: memcpy arguments alias each other, dst=0x97d0500
>> src=0x97d0500 len=64
>>
>>
>> =====================================================================================
>> = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
>> = EXIT CODE: 256
>> = CLEANING UP REMAINING PROCESSES
>> = YOU CAN IGNORE THE BELOW CLEANUP MESSAGES
>>
>> =====================================================================================
>> [proxy:0:1 at beowulf.node1] HYD_pmcd_pmip_control_cmd_cb
>> (./pm/pmiserv/pmip_cb.c:927): assert (!closed) failed
>> [proxy:0:1 at beowulf.node1] HYDT_dmxu_poll_wait_for_event
>> (./tools/demux/demux_poll.c:77): callback returned error status
>> [proxy:0:1 at beowulf.node1] main (./pm/pmiserv/pmip.c:221): demux engine
>> error waiting for event
>> [mpiexec at beowulf.master] HYDT_bscu_wait_for_completion
>> (./tools/bootstrap/utils/bscu_wait.c:77): one of the processes terminated
>> badly; aborting
>> [mpiexec at beowulf.master] HYDT_bsci_wait_for_completion
>> (./tools/bootstrap/src/bsci_wait.c:23): launcher returned error waiting
>> for
>> completion
>> [mpiexec at beowulf.master] HYD_pmci_wait_for_completion
>> (./pm/pmiserv/pmiserv_pmci.c:225): launcher returned error waiting for
>> completion
>> [mpiexec at beowulf.master] main (./ui/mpich/mpiexec.c:437): process manager
>> error waiting for completion
>> [root at beowulf testing]#
>> -------------- next part --------------
>> An HTML attachment was scrubbed...
>> URL:
>> <http://lists.mcs.anl.gov/pipermail/mpich-discuss/attachments/20120522/7b1db8c0/attachment-0001.html>
>>
>> ------------------------------
>>
>> Message: 3
>> Date: Tue, 22 May 2012 03:36:44 +0800
>> From: Darius Buntinas <buntinas at mcs.anl.gov>
>> To: mpich-discuss at mcs.anl.gov
>> Subject: Re: [mpich-discuss] Unable to run program parallely on
>> cluster... Its running properly on single machine...
>> Message-ID: <B411B6C1-CB5A-4A1C-AEBB-71680C9AF8C5 at mcs.anl.gov>
>> Content-Type: text/plain; charset=us-ascii
>>
>> It may be that one of your processes is failing, but also check to make
>> sure every process is calling MPI_Finalize before exiting.
>>
>> -d
>>
>> On May 22, 2012, at 2:42 AM, Albert Spade wrote:
>>
>> > Hi everybody,
>> >
>> > I am using mpich2-1.4.1p1 and mpiexec from hydra-1.5b1
>> > I have a cluster of 5 machines.
>> > When I am trying to run the program for parallel fast fourier transform
>> > on single machine it runs correctly but on a cluster it gives error.
>> > Can you please tell me why its happening.
>> >
>> > Thanks.
>> >
>> > Here is my sample output:
>> >
>> > ---------------------------------------------------------------------------------------
>> >
>> > [root at beowulf programs]# mpiexec -n 1 ./Radix2
>> > Time taken for 16 elements using 1 processors = 2.7895e-05 seconds
>> > [root at beowulf programs]#
>> > [root at beowulf programs]# mpiexec -n 4 ./Radix2
>> > [mpiexec at beowulf.master] control_cb (./pm/pmiserv/pmiserv_cb.c:197):
>> > assert (!closed) failed
>> > [mpiexec at beowulf.master] HYDT_dmxu_poll_wait_for_event
>> > (./tools/demux/demux_poll.c:77): callback returned error status
>> > [mpiexec at beowulf.master] HYD_pmci_wait_for_completion
>> > (./pm/pmiserv/pmiserv_pmci.c:205): error waiting for event
>> > [mpiexec at beowulf.master] main (./ui/mpich/mpiexec.c:437): process
>> > manager error waiting for completion
>> > [root at beowulf programs]# mpiexec -n 2 ./Radix2
>> > [mpiexec at beowulf.master] control_cb (./pm/pmiserv/pmiserv_cb.c:197):
>> > assert (!closed) failed
>> > [mpiexec at beowulf.master] HYDT_dmxu_poll_wait_for_event
>> > (./tools/demux/demux_poll.c:77): callback returned error status
>> > [mpiexec at beowulf.master] HYD_pmci_wait_for_completion
>> > (./pm/pmiserv/pmiserv_pmci.c:205): error waiting for event
>> > [mpiexec at beowulf.master] main (./ui/mpich/mpiexec.c:437): process
>> > manager error waiting for completion
>> > [root at beowulf programs]# mpiexec -n 4 ./Radix2
>> > [mpiexec at beowulf.master] control_cb (./pm/pmiserv/pmiserv_cb.c:197):
>> > assert (!closed) failed
>> > [mpiexec at beowulf.master] HYDT_dmxu_poll_wait_for_event
>> > (./tools/demux/demux_poll.c:77): callback returned error status
>> > [mpiexec at beowulf.master] HYD_pmci_wait_for_completion
>> > (./pm/pmiserv/pmiserv_pmci.c:205): error waiting for event
>> > [mpiexec at beowulf.master] main (./ui/mpich/mpiexec.c:437): process
>> > manager error waiting for completion
>> > [root at beowulf programs]#
>> > _______________________________________________
>> > mpich-discuss mailing list mpich-discuss at mcs.anl.gov
>> > To manage subscription options or unsubscribe:
>> > https://lists.mcs.anl.gov/mailman/listinfo/mpich-discuss
>>
>>
>>
>> ------------------------------
>>
>> Message: 4
>> Date: Mon, 21 May 2012 20:14:35 -0500
>> From: Rajeev Thakur <thakur at mcs.anl.gov>
>> To: mpich-discuss at mcs.anl.gov
>> Subject: Re: [mpich-discuss] Not able to run program parallely on
>> cluster...
>> Message-ID: <8C80534E-3611-40D7-BBAF-F66110D25EE1 at mcs.anl.gov>
>> Content-Type: text/plain; charset=us-ascii
>>
>> You are passing the same buffer as the sendbuf and recvbuf to MPI_Gatherv,
>> which is not allowed in MPI. Use MPI_IN_PLACE as described in the standard.
>>
>>
>> On May 21, 2012, at 2:29 PM, Albert Spade wrote:
>>
>> > This is my new error after making few changes...
>> > Results are quite similar... No succes with cluster...
>> >
>> > Sample run
>> > --------------------------------------------------------
>> >
>> > [root at beowulf testing]# mpiexec -n 1 ./Radix
>> > Time taken for 16 elements using 1 processors = 4.72069e-05 seconds
>> > [root at beowulf testing]# mpiexec -n 2 ./Radix
>> > Fatal error in PMPI_Gatherv: Internal MPI error!, error stack:
>> > PMPI_Gatherv(398).....: MPI_Gatherv failed(sbuf=0x97d0500, scount=64,
>> > MPI_CHAR, rbuf=0x97d0500, rcnts=0x97d06b8, displs=0x97d06c8, MPI_CHAR,
>> > root=0, MPI_COMM_WORLD) failed
>> > MPIR_Gatherv_impl(210):
>> > MPIR_Gatherv(104).....:
>> > MPIR_Localcopy(357)...: memcpy arguments alias each other, dst=0x97d0500
>> > src=0x97d0500 len=64
>> >
>> > =====================================================================================
>> > = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
>> > = EXIT CODE: 256
>> > = CLEANING UP REMAINING PROCESSES
>> > = YOU CAN IGNORE THE BELOW CLEANUP MESSAGES
>> >
>> > =====================================================================================
>> > [proxy:0:1 at beowulf.node1] HYD_pmcd_pmip_control_cmd_cb
>> > (./pm/pmiserv/pmip_cb.c:927): assert (!closed) failed
>> > [proxy:0:1 at beowulf.node1] HYDT_dmxu_poll_wait_for_event
>> > (./tools/demux/demux_poll.c:77): callback returned error status
>> > [proxy:0:1 at beowulf.node1] main (./pm/pmiserv/pmip.c:221): demux engine
>> > error waiting for event
>> > [mpiexec at beowulf.master] HYDT_bscu_wait_for_completion
>> > (./tools/bootstrap/utils/bscu_wait.c:77): one of the processes terminated
>> > badly; aborting
>> > [mpiexec at beowulf.master] HYDT_bsci_wait_for_completion
>> > (./tools/bootstrap/src/bsci_wait.c:23): launcher returned error waiting for
>> > completion
>> > [mpiexec at beowulf.master] HYD_pmci_wait_for_completion
>> > (./pm/pmiserv/pmiserv_pmci.c:225): launcher returned error waiting for
>> > completion
>> > [mpiexec at beowulf.master] main (./ui/mpich/mpiexec.c:437): process
>> > manager error waiting for completion
>> > [root at beowulf testing]#
>> >
>> > _______________________________________________
>> > mpich-discuss mailing list mpich-discuss at mcs.anl.gov
>> > To manage subscription options or unsubscribe:
>> > https://lists.mcs.anl.gov/mailman/listinfo/mpich-discuss
>>
>>
>>
>> ------------------------------
>>
>> Message: 5
>> Date: Tue, 22 May 2012 17:21:09 +0200
>> From: Thomas Ropars <thomas.ropars at epfl.ch>
>> To: mpich-discuss at mcs.anl.gov
>> Subject: [mpich-discuss] replication of mpi applications
>> Message-ID: <4FBBAEE5.5070000 at epfl.ch>
>> Content-Type: text/plain; charset=UTF-8; format=flowed
>>
>> Dear all,
>>
>> We are starting studying replication for MPI applications. A few papers
>> have been published in the last months on this topic.
>>
>> We were wondering if anybody has already started working on providing
>> process replication in MPICH ? And if yes, is there some code available ?
>>
>> Best regards,
>>
>> Thomas Ropars
>>
>>
>> ------------------------------
>>
>> _______________________________________________
>> mpich-discuss mailing list
>> mpich-discuss at mcs.anl.gov
>> https://lists.mcs.anl.gov/mailman/listinfo/mpich-discuss
>>
>>
>> End of mpich-discuss Digest, Vol 44, Issue 36
>> *********************************************
>
>
>
> _______________________________________________
> mpich-discuss mailing list mpich-discuss at mcs.anl.gov
> To manage subscription options or unsubscribe:
> https://lists.mcs.anl.gov/mailman/listinfo/mpich-discuss
>
--
Jeff Hammond
Argonne Leadership Computing Facility
University of Chicago Computation Institute
jhammond at alcf.anl.gov / (630) 252-5381
http://www.linkedin.com/in/jeffhammond
https://wiki.alcf.anl.gov/parts/index.php/User:Jhammond
More information about the mpich-discuss
mailing list