[mpich2-dev] MPI_COMM_WORLD failed when using pio-bench on PVFS
汪迪
otheryou at yahoo.cn
Mon Apr 13 10:20:33 CDT 2009
Hi, Rajeev
It is not a problem of my MPI installation, it is a problem of pio-bench. Actually I found someone who has the same problem here: http://newsgroups.derkeiler.com/Archive/Comp/comp.parallel.mpi/2006-04/msg00046.html. And I tried his source code on my own machine, the other MPI error occurs when running on root (while not on root it did not give the error detail).
I think it is due to the dynamically allocated address semantic incompatible. The gcc I use is (Ubuntu 4.3.2-1ubuntu12) 4.3.2. If you can run pio-bench without any problem, I want to know the edition of the gcc you use. Thanks.
By the way I check the pio-bench source code and know the only statement that causes this problem is:
MPI_Bcast( qlist_entry(p, ap_module, link), sizeof(ap_module), MPI_BYTE, 0, MPI_COMM_WORLD); in main function of pio-bench.c. Can you give me any suggestion on modify this code? As I might as well consider modifying the pio-bench source code if changing the gcc does not work.
--- 09年3月27日，周五, Rajeev Thakur <thakur at mcs.anl.gov> 写道：
发件人: Rajeev Thakur <thakur at mcs.anl.gov>
主题: RE: [mpich2-dev] MPI_COMM_WORLD failed when using pio-bench on PVFS
收件人: "'??'" <otheryou at yahoo.cn>
抄送: "'MPICH2-developer mailling-list'" <mpich2-dev at mcs.anl.gov>
日期: 2009,327,周五,2:51上午
What do you mean by "I configure PVFS system server on my own
laptop, which uses MPICH2 as its trove storage system implementation"? You have
installed MPICH2 and PVFS as two separate components, right? Does pio-bench
work if you make it access a local file directly via the Linux file
system?
Rajeev
From: ?? [mailto:otheryou at yahoo.cn]
Sent: Thursday, March 26, 2009 12:05 AM
To: Rajeev
Thakur
Cc: MPICH2-developer mailling-list
Subject: RE:
[mpich2-dev] MPI_COMM_WORLD failed when using pio-bench on
PVFS
Hi, Rajeev
Other MPI programs runs
correctly, including cpi under examples folder.
gxwangdi at WANGDI ~/D/m/examples>
pwd
~/Downloads/mpich2-1.0.8/examples
gxwangdi at WANGDI
~/D/m/examples> mpd &
gxwangdi at WANGDI ~/D/m/examples>
mpdtrace
WANGDI
gxwangdi at WANGDI ~/D/m/examples> which
mpiexec
/usr/local/bin/mpiexec
gxwangdi at WANGDI ~/D/m/examples>
mpiexec -n 10 ./cpi
Process 2 of 10 is on WANGDI
Process 1 of 10
is on WANGDI
Process 0 of 10 is on WANGDI
Process 6 of 10 is on
WANGDI
Process 4 of 10 is on WANGDI
Process 9 of 10 is on
WANGDI
Process 3 of 10 is on WANGDI
Process 7 of 10 is on
WANGDI
Process 8 of 10 is on WANGDI
Process 5 of 10 is on
WANGDI
pi is approximately 3.1415926544231256, Error is
0.0000000008333325
wall clock time = 0.012229
Thanks for your
suggestion, I run make testing in the top directory of mpich2.
There is a test that do not pass:
xwangdi at WANGDI ~/D/mpich2-1.0.8>
mpd &
gxwangdi at WANGDI ~/D/mpich2-1.0.8> make testing
(cd
test && make testing)
make[1]: Entering directory
`/home/gxwangdi/Downloads/mpich2-1.0.8/test'
(NOXMLCLOSE=YES
&& export NOXMLCLOSE && cd mpi && make
testing)
make[2]: Entering directory
`/home/gxwangdi/Downloads/mpich2-1.0.8/test/mpi'
./runtests -srcdir=.
-tests=testlist \
-mpiexec=/usr/local/bin/mpiexec \
-xmlfile=summary.xml
Looking in
./testlist
Processing directory attr
Looking in
./attr/testlist
Processing directory coll
Looking in
./coll/testlist
Processing directory comm
Looking in
./comm/testlist
Some programs (cmsplit) may still be running:
pids
= 2279
The executable (cmsplit) will not be removed.
Processing
directory datatype
Looking in ./datatype/testlist
Processing
directory errhan
Looking in ./errhan/testlist
Processing directory
group
Looking in ./group/testlist
Processing directory
info
Looking in ./info/testlist
Processing directory
init
Looking in ./init/testlist
Processing directory
pt2pt
Looking in ./pt2pt/testlist
Some programs (sendrecv3) may
still be running:
pids = 4049
The executable (sendrecv3) will not
be removed.
Processing directory rma
Looking in
./rma/testlist
Some programs (transpose3) may still be
running:
pids = 5713
The executable (transpose3) will not be
removed.
Processing directory spawn
Looking in
./spawn/testlist
Processing directory topo
Looking in
./topo/testlist
Processing directory perf
Looking in
./perf/testlist
Processing directory io
Looking in
./io/testlist
Processing directory cxx
Looking in
./cxx/testlist
Processing directory attr
Looking in
./cxx/attr/testlist
Processing directory pt2pt
Looking in
./cxx/pt2pt/testlist
Failed to build bsend1cxx; make[3]: Entering
directory
`/home/gxwangdi/Downloads/mpich2-1.0.8/test/mpi/cxx/pt2pt'
/usr/local/bin/mpicxx
-DHAVE_CONFIG_H -I. -I. -I../../include -I./../../include -c
bsend1cxx.cxx
bsend1cxx.cxx: In function ‘int main(int,
char**)’:
bsend1cxx.cxx:81: error: ‘strcmp’ was not declared in this
scope
bsend1cxx.cxx:91: error: ‘strcmp’ was not declared in this
scope
make[3]: *** [bsend1cxx.o] Error 1
make[3]: Leaving
directory
`/home/gxwangdi/Downloads/mpich2-1.0.8/test/mpi/cxx/pt2pt'
Processing
directory comm
Looking in ./cxx/comm/testlist
Processing directory
coll
Looking in ./cxx/coll/testlist
Processing directory
init
Looking in ./cxx/init/testlist
Processing directory
info
Looking in ./cxx/info/testlist
Processing directory
datatype
Looking in ./cxx/datatype/testlist
Processing directory
io
Looking in ./cxx/io/testlist
Processing directory
spawn
Looking in ./cxx/spawn/testlist
Processing directory
rma
Looking in ./cxx/rma/testlist
Processing directory
errors
Looking in ./errors/testlist
Processing directory
attr
Looking in ./errors/attr/testlist
Processing directory
coll
Looking in ./errors/coll/testlist
Processing directory
comm
Looking in ./errors/comm/testlist
Processing directory
group
Looking in ./errors/group/testlist
Processing directory
pt2pt
Looking in ./errors/pt2pt/testlist
Processing directory
topo
Looking in ./errors/topo/testlist
Processing directory
rma
Looking in ./errors/rma/testlist
Processing directory
spawn
Looking in ./errors/spawn/testlist
Processing directory
io
Looking in ./errors/io/testlist
Processing directory
cxx
Looking in ./errors/cxx/testlist
Processing directory
errhan
Looking in ./errors/cxx/errhan/testlist
Processing
directory io
Looking in ./errors/cxx/io/testlist
Processing
directory threads
Looking in ./threads/testlist
Processing
directory pt2pt
Looking in ./threads/pt2pt/testlist
Processing
directory comm
Looking in ./threads/comm/testlist
1 tests failed
out of 385
Details in
/home/gxwangdi/Downloads/mpich2-1.0.8/test/mpi/summary.xml
make[2]:
Leaving directory
`/home/gxwangdi/Downloads/mpich2-1.0.8/test/mpi'
(XMLFILE=../mpi/summary.xml
&& XMLCONTINUE=YES && \
export
XMLFILE && export XMLCONTINUE && \
cd commands && make testing)
make[2]: Entering directory
`/home/gxwangdi/Downloads/mpich2-1.0.8/test/commands'
make[2]:
Nothing to be done for `testing'.
make[2]: Leaving directory
`/home/gxwangdi/Downloads/mpich2-1.0.8/test/commands'
make[1]:
Leaving directory
`/home/gxwangdi/Downloads/mpich2-1.0.8/test'
And the attachment
is the /home/gxwangdi/Downloads/mpich2-1.0.8/test/mpi/summary.xml file,
it is too long to paste all content here. I can not understand what is
the problem yet.
From:
Rajeev Thakur <thakur at mcs.anl.gov>
Subject: RE: [mpich2-dev]
MPI_COMM_WORLD failed when using pio-bench on PVFS
To:
otheryou at yahoo.cn, mpich2-dev at mcs.anl.gov
Date:
2009,3/26,Thursday,2:41
Do the other MPICH2 tests run, such as
the cpi example in the examples directory? If you run "make testing"
in the top-level mpich2 directory it will run the entire tests suite
in test/mpi (can take more than an hour).
Rajeev
From: mpich2-dev-bounces at mcs.anl.gov
[mailto:mpich2-dev-bounces at mcs.anl.gov] On Behalf Of
??
Sent: Wednesday, March 25, 2009 12:11
AM
To: MPICH2-developer mailling-list
Subject:
[mpich2-dev] MPI_COMM_WORLD failed when using pio-bench on
PVFS
Hi,
I configure PVFS system server on
my own laptop, which uses MPICH2 as its trove storage system
implementation. And now I tend to use pio-bench to get
the trace of the server. It works as I set the number of
process as 1, but when I set the number of process to more
than 1, it does not work all the time. It tips the
MPI_COMM_WORLD failed, but I run hostname command with MPI and
it works. It is peculiar. I check the MPICH2 document and find
no particular configuration for single host MPI deployment. Do
I do anything wrong?
By the way my laptop is
IBM i386 architecture and CPU is intel centrino2 vPro, OS
UBUNTU 8.10, MPICH2 1.0.8 is installed by default under
/usr/local, and pvfs2.7.1 is under /root/pvfs-install/.
gxwangdi at WANGDI:~/Desktop/pio-bench$ sudo mpiexec -n 1
./pio-bench
[sudo] password for gxwangdi:
File under
test: /mnt/pvfs2/ftpaccess
Number of Processes: 1
Sync:
off
Averaging: Off
the nested strided pattern needs to
be run with an even amount of processes
file pio-bench.c,
line 586: access pattern initialization error:
-1
gxwangdi at WANGDI:~/Desktop/pio-bench$ sudo mpiexec -n
4 ./pio-bench
Fatal error in MPI_Barrier: Other MPI error,
error stack:
MPI_Barrier(406)..........................:
MPI_Barrier(MPI_COMM_WORLD)
failed
MPIR_Barrier(77)..........................:
MPIC_Sendrecv(126)........................:
MPIC_Wait(270)............................:
MPIDI_CH3i_Progress_wait(215).............:
an error occurred while
handling an event returned by
MPIDU_Sock_Wait()
MPIDI_CH3I_Progress_handle_sock_event(420):
MPIDU_Socki_handle_read(637)..............:
connection failure
(set=0,sock=1,errno=104:Connection reset
by peer)[cli_0]: aborting job:
Fatal error in MPI_Barrier:
Other MPI error, error
stack:
MPI_Barrier(406)..........................:
MPI_Barrier(MPI_COMM_WORLD)
failed
MPIR_Barrier(77)..........................:
MPIC_Sendrecv(126)........................:
MPIC_Wait(270)............................:
MPIDI_CH3i_Progress_wait(215).............:
an error occurred while
handling an event returned by
MPIDU_Sock_Wait()
MPIDI_CH3I_Progress_handle_sock_event(420):
MPIDU_Socki_handle_readFatal
error in MPI_Bcast: Other MPI error,
error
stack:
MPI_Bcast(786)............................:
MPI_Bcast(buf=0x1fd6ca78,
count=20, MPI_BYTE, root=0,
MPI_COMM_WORLD)
failed
MPIR_Bcast(198)...........................:
MPIC_Recv(81).............................:
MPIC_Wait(270)............................:
MPIDI_CH3i_Progress_wait(215).............:
an error occurred while
handling an event returned by
MPIDU_Sock_Wait()
MPIDI_CH3I_Progress_handle_sock_event(456):
adjust_iov(973)...........................:
ch3|sock|immedread
0x1e5a0d60 0x1f329978
0x1f3258d0
MPIDU_Sock_readv(455).....................: the
supplied buffer contains
invalid memory
(set=0,sock=1,errno=14:Bad address)[cli_1]: aborting
job:
Fatal error in MPI_Bcast: Other MPI error, error
stack:
MPI_Bcast(786)............................:
MPI_Bcast(buf=0x1fd6ca78,
count=20, MPI_BYTE, root=0,
MPI_COMM_WORLD)
failed
MPIR_Bcast(198)...........................:
MPIC_Recv(81).............................:
MPIC_Wait(270)............................(637)..............:
connection
failure (set=0,sock=1,errno=104:Connection reset by
peer)
:
MPIDI_CH3i_Progress_wait(215).............: an
error occurred while
handling an event returned by
MPIDU_Sock_Wait()
MPIDI_CH3I_Progress_handle_sock_event(456):
adjust_iov(973)...........................:
ch3|sock|immedread
0x1e5a0d60 0x1f329978
0x1f3258d0
MPIDU_Sock_readv(455).....................: the
supplied buffer contains
invalid memory
(set=0,sock=1,errno=14:Bad address)
Fatal error in
MPI_Bcast: Other MPI error, error
stack:
MPI_Bcast(786)............................:
MPI_Bcast(buf=0x1fd6ca78,
count=20, MPI_BYTE, root=0,
MPI_COMM_WORLD)
failed
MPIR_Bcast(198)...........................:
MPIC_Recv(81).............................:
MPIC_Wait(270)............................:
MPIDI_CH3i_Progress_wait(215).............:
an error occurred while
handling an event returned by
MPIDU_Sock_Wait()
MPIDI_CH3I_Progress_handle_sock_event(456):
adjust_iov(973)...........................:
ch3|sock|immedread
0x1e5a0d60 0x1eb9f978
0x1eb9b8d0
MPIDU_Sock_readv(455).....................: the
supplied buffer contains
invalid memory
(set=0,sock=1,errno=14:Bad address)[cli_2]: aborting
job:
Fatal error in MPI_Bcast: Other MPI error, error
stack:
MPI_Bcast(786)........................so....:
MPI_Bcast(buf=0x1fd6ca78,
count=20, MPI_BYTE, root=0,
MPI_COMM_WORLD)
failed
MPIR_Bcast(198)...........................:
MPIC_Recv(81).............................:
MPIC_Wait(270)............................:
MPIDI_CH3i_Progress_wait(215).............:
an error occurred while
handling an event returned by
MPIDU_Sock_Wait()
MPIDI_CH3I_Progress_handle_sock_event(456):
adjust_iov(973)...........................:
ch3|sock|immedread
0x1e5a0d60 0x1eb9f978
0x1eb9b8d0
MPIDU_Sock_readv(455).....................: the
supplied buffer contains
invalid memory
(set=0,sock=1,errno=14:Bad address)
rank 1 in job 9
WANGDI_59039 caused collective abort of all
ranks
exit status of rank 1: return code 1
rank 0
in job 9 WANGDI_59039 caused collective
abort of all ranks
exit status of rank 0: return
code 1
gxwangdi at WANGDI:~/Desktop/pio-bench$ sudo mpiexec -n
4
hostname
WANGDI
WANGDI
WANGDI
WANGDI
and
my pio-bench.conf file is like:
Testfile
"/mnt/pvfs2/ftpaccess"
OutputToFile
"/home/gxwangdi/Desktop/pio-bench/results/result"
<ap_module>
ModuleName
"Nested Strided (read)"
ModuleReps 3
ModuleSettleTime
5
</ap_module>
<ap_module>so
ModuleName
"Nested Strided (write)"
ModuleReps 3
ModuleSettleTime
5
</ap_module>
<ap_module>
ModuleName
"Nested Strided (read-modify-write)"
ModuleReps
3
ModuleSettleTime
5
</ap_module>
<ap_module>
ModuleName
"Nested Strided (re-read)"
ModuleReps 3
ModuleSettleTime
5
</ap_module>
<ap_module>
ModuleName
"Nested Strided (re-write)"
ModuleReps
3
ModuleSettleTime 5
</ap_module>
Also I
use another file that is not under /mnt/pvfs2 for test, it
tips like below:
gxwangdi at WANGDI:~/Desktop/pio-bench$
sudo mpiexec -n 4 ./pio-bench
Fatal error in MPI_Barrier:
Other MPI error, error
stack:
MPI_Barrier(406)..........................:
MPI_Barrier(MPI_COMM_WORLD)
failed
MPIR_Barrier(77)..........................:
MPIC_Sendrecv(126)........................:
MPIC_Wait(270)............................:
MPIDI_CH3i_Progress_wait(215).............:
an error occurred while
handling an event returned by
MPIDU_Sock_Wait()
MPIDI_CH3I_Progress_handle_sock_event(420):
MPIDU_Socki_handle_read(637)..............:
connection failure
(set=0,sock=1,errno=104:Connection reset
by peer)[cli_0]: aborting job:
Fatal error in MPI_Barrier:
Other MPI error, error
stack:
MPI_Barrier(406)..........................:
MPI_Barrier(MPI_COMM_WORLD)
failed
MPIR_Barrier(77)..........................:
MPIC_Sendrecv(126)........................:
MPIC_Wait(270)............................:
MPIDI_CH3i_Progress_wait(215).............:
an error occurred while
handling an event returned by
MPIDU_Sock_Wait()
MPIDI_CH3I_Progress_handle_sock_event(420):
MPIDU_Socki_handle_readFatal
error in MPI_Bcast: Other MPI error,
error
stack:
MPI_Bcast(786)............................:
MPI_Bcast(buf=0x1f60fad0,
count=20, MPI_BYTE, root=0,
MPI_COMM_WORLD)
failed
MPIR_Bcast(198)...........................:
MPIC_Recv(81).............................:
MPIC_Wait(270)............................:
MPIDI_CH3i_Progress_wait(215).............:
an error occurred while
handling an event returned by
MPIDU_Sock_Wait()
MPIDI_CH3I_Progress_handle_sock_event(456):
adjust_iov(973)...........................:
ch3|sock|immedread
0x1e5a0d60 0x1f525978
0x1f5218d0
MPIDU_Sock_readv(455).....................: the
supplied buffer contains
invalid memory
(set=0,sock=1,errno=14:Bad address)[cli_2]: aborting
job:
Fatal error in MPI_Bcast: Other MPI error, error
stack:
MPI_Bcast(786)............................:
MPI_Bcast(buf=0x1f60fad0,
count=20, MPI_BYTE, root=0,
MPI_COMM_WORLD)
failed
MPIR_Bcast(198)...........................:
MPIC_Recv(81).............................:
MPIC_Wait(270)............................(637)..............:
connection
failure (set=0,sock=1,errno=104:Connection reset by
peer)
:
MPIDI_CH3i_Progress_wait(215).............: an
error occurred while
handling an event returned by
MPIDU_Sock_Wait()
MPIDI_CH3I_Progress_handle_sock_event(456):
adjust_iov(973)...........................:
ch3|sock|immedread
0x1e5a0d60 0x1f525978
0x1f5218d0
MPIDU_Sock_readv(455).....................: the
supplied buffer contains
invalid memory
(set=0,sock=1,errno=14:Bad address)
Fatal error in
MPI_Bcast: Other MPI error, error
stack:
MPI_Bcast(786)............................:
MPI_Bcast(buf=0x1f60fad0,
count=20, MPI_BYTE, root=0,
MPI_COMM_WORLD)
failed
MPIR_Bcast(198)...........................:
MPIC_Recv(81).............................:
MPIC_Wait(270)............................:
MPIDI_CH3i_Progress_wait(215).............:
an error occurred while
handling an event returned by
MPIDU_Sock_Wait()
MPIDI_CH3I_Progress_handle_sock_event(456):
adjust_iov(973)...........................:
ch3|sock|immedread
0x1e5a0d60 0x202d2978
0x202ce8d0
MPIDU_Sock_readv(455).....................: the
supplied buffer contains
invalid memory
(set=0,sock=1,errno=14:Bad address)[cli_1]: aborting
job:
Fatal error in MPI_Bcast: Other MPI error, error
stack:
MPI_Bcast(786)............................:
MPI_Bcast(buf=0x1f60fad0,
count=20, MPI_BYTE, root=0,
MPI_COMM_WORLD)
failed
MPIR_Bcast(198)...........................:
MPIC_Recv(81).............................:
MPIC_Wait(270)............................:
MPIDI_CH3i_Progress_wait(215).............:
an error occurred while
handling an event returned by
MPIDU_Sock_Wait()
MPIDI_CH3I_Progress_handle_sock_event(456):
adjust_iov(973)...........................:
ch3|sock|immedread
0x1e5a0d60 0x202d2978
0x202ce8d0
MPIDU_Sock_readv(455).....................: the
supplied buffer contains
invalid memory
(set=0,sock=1,errno=14:Bad address)
rank 2 in job 11
WANGDI_59039 caused collective abort of all
ranks
exit status of rank 2: return code 1
rank 1
in job 11 WANGDI_59039 caused collective
abort of all ranks
exit status of rank 1: return
code 1
rank 0 in job 11 WANGDI_59039
caused collective abort of all ranks
exit status of
rank 0: return code 1
The MPI_COMM_WORLD failed
again but it caused collective abort of all
ranks at the
end, which is a little bit different, as it has no
syslog
for pio-bench to check, I do not understand what
happens and I can not
solve this problem.
Appreciate
your responses.
好玩贺卡等你发，邮箱贺卡全新上线！
好玩贺卡等你发，邮箱贺卡全新上线！
___________________________________________________________
好玩贺卡等你发，邮箱贺卡全新上线！
http://card.mail.cn.yahoo.com/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.mcs.anl.gov/mailman/private/mpich2-dev/attachments/20090413/03d8e6f3/attachment-0001.htm>
More information about the mpich2-dev
mailing list