<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN">
<HTML><HEAD>
<META content="text/html; charset=gb2312" http-equiv=Content-Type>
<STYLE>.hmmessage P {
PADDING-BOTTOM: 0px; MARGIN: 0px; PADDING-LEFT: 0px; PADDING-RIGHT: 0px; PADDING-TOP: 0px
}
BODY.hmmessage {
FONT-FAMILY: Verdana; FONT-SIZE: 10pt
}
</STYLE>
<META name=GENERATOR content="MSHTML 8.00.6001.18854"></HEAD>
<BODY class=hmmessage>
<DIV dir=ltr align=left><SPAN class=937023714-23122009><FONT color=#0000ff
face=Arial>You need to compile with mpif77 or mpif90, not just f77 or
f90.</FONT></SPAN></DIV>
<DIV dir=ltr align=left><SPAN class=937023714-23122009><FONT color=#0000ff
face=Arial></FONT></SPAN> </DIV>
<DIV dir=ltr align=left><SPAN class=937023714-23122009><FONT color=#0000ff
face=Arial>Rajeev</FONT></SPAN></DIV><BR>
<BLOCKQUOTE
style="BORDER-LEFT: #0000ff 2px solid; PADDING-LEFT: 5px; MARGIN-LEFT: 5px; MARGIN-RIGHT: 0px">
<DIV dir=ltr lang=en-us class=OutlookMessageHeader align=left>
<HR tabIndex=-1>
<FONT face=Tahoma><B>From:</B> mpich-discuss-bounces@mcs.anl.gov
[mailto:mpich-discuss-bounces@mcs.anl.gov] <B>On Behalf Of
</B>LS<BR><B>Sent:</B> Wednesday, December 23, 2009 6:04 AM<BR><B>To:</B>
MPICH<BR><B>Subject:</B> Re: [mpich-discuss] how to deal with these
errors?<BR></FONT><BR></DIV>
<DIV></DIV>Hi Rajeev,<BR> <BR>I just have a try, delete the
mpif.h file in the application directories. But in this way I can
not even compile the model successfully.<BR> <BR> <BR>Liu.
S <BR>
<HR id=stopSpelling>
From: thakur@mcs.anl.gov<BR>To: mpich-discuss@mcs.anl.gov<BR>Date: Wed, 23 Dec
2009 02:54:14 -0600<BR>Subject: Re: [mpich-discuss] how to deal with these
errors?<BR><BR>
<STYLE>.ExternalClass .ecxhmmessage P {
PADDING-BOTTOM: 0px; PADDING-LEFT: 0px; PADDING-RIGHT: 0px; PADDING-TOP: 0px
}
.ExternalClass BODY.ecxhmmessage {
FONT-FAMILY: Verdana; FONT-SIZE: 10pt
}
</STYLE>
<DIV dir=ltr align=left><SPAN class=ecx140565308-23122009><FONT color=#0000ff
face=Arial>Make sure there is no mpif.h file in any of the application
directories.</FONT></SPAN></DIV>
<DIV dir=ltr align=left><SPAN class=ecx140565308-23122009><FONT color=#0000ff
face=Arial></FONT></SPAN> </DIV>
<DIV dir=ltr align=left><SPAN class=ecx140565308-23122009><FONT color=#0000ff
face=Arial>Rajeev</FONT></SPAN></DIV><BR>
<BLOCKQUOTE
style="BORDER-LEFT: #0000ff 2px solid; PADDING-LEFT: 5px; MARGIN-LEFT: 5px; MARGIN-RIGHT: 0px">
<DIV dir=ltr lang=en-us class=ecxOutlookMessageHeader align=left>
<HR>
<FONT face=Tahoma><B>From:</B> mpich-discuss-bounces@mcs.anl.gov
[mailto:mpich-discuss-bounces@mcs.anl.gov] <B>On Behalf Of
</B>LSB<BR><B>Sent:</B> Wednesday, December 23, 2009 2:39 AM<BR><B>To:</B>
mpich-discuss@mcs.anl.gov<BR><B>Subject:</B> Re: [mpich-discuss] how to deal
with these errors?<BR></FONT><BR></DIV>
<DIV></DIV>Hi Correa,<BR> <BR>Thank you for your reply. Meet you here
agian, haha<BR>I have reorgnized my questions, and posted it on
the CGD forum.<BR><BR>I have edited the related
directories of the MPI include files and library after you answered me
through the ccsm maillist. However, the "invalid communicator" is still
there. I am sure my CCSM3 Makefiles now points to MPICH2 include files
and the library directories are also have been changed to associate to
MPICH2. Is there any other reason that can lead to this
problem?<BR> <BR>I said that I can "run" CCSM3 by delete any one of the
five components. I infact meant that in this situation I can see these
process names by the command "top". But if I run all the five,
I can not see these process names with "top", and the error
message as mentioned before will appear.<BR> <BR>The resolution I
use is T31_gx3v5. I asked for two nodes(each node with 16G memory and 16
cpus). But I am not sure whether these resources are enough for
CCSM3 running or not.<BR> <BR>Thanks for your help.<BR> <BR>Liu.
S<BR> <BR> <BR> <BR>> Date: Tue, 22 Dec 2009 16:30:40
-0500<BR>> From: gus@ldeo.columbia.edu<BR>> To:
mpich-discuss@mcs.anl.gov<BR>> Subject: Re: [mpich-discuss] how to deal
with these errors?<BR>> <BR>> Hi Liu<BR>> <BR>> As I mentioned,
probably to you, in the CCSM3 forum:<BR>> <BR>> **<BR>> <BR>>
Regarding 1),<BR>> the "Invalid communicator" error is often produced by
the use<BR>> of a wrong mpi.h or mpif.h include files, i.e.,<BR>>
include files from another MPI that may be in your system.<BR>> <BR>>
If you search this mailing list archives, or the OpenMPI mailing
list<BR>> archives, you will find other postings reporting this
error.<BR>> <BR>> For instance, in one of our computers here, the
MPICH-1<BR>> mpi.h has this:<B R>> <BR>> #define MPI_COMM_WORLD
91<BR>> <BR>> whereas the MPICH2 mpi.h has something else:<BR>>
<BR>> #define MPI_COMM_WORLD ((MPI_Comm)0x44000000)<BR>> <BR>> As
you can see, eve n MPI_COMM_WORLD is different on MPICH-1 and
MPICH2.<BR>> You cannot patch this by hand.<BR>> You must use the
correct mpi.h/mpif.h, associated to your<BR>> mpicc and mpif90.<BR>>
<BR>> You may want to compile everything again fresh.<BR>> Object
files and modules that were built with the wrong mpi.h<BR>> will only
cause you headaches, and the "Invalid communicator"<BR>> error will never
go away.<BR>> Get rid of them before you restart.<BR>> Do make
clean/cleanall, or make cleandist.<BR>> Even better: simply start from a
fresh tarball.<BR>> <BR>> To compile, you should preferably use the
MPICH2<BR>> compiler wrappers mpif90 and mpicc.<BR>> <BR>> Wherever
the CCSM3 Makefiles point to MPI include files,<BR>> make sure the direc
tories are those of MPICH2, not any other MPI.<BR>> <BR>> Likewise for
the MPI library directories:<BR>> they must be those associated to
MPICH2.<BR>> <BR>> To save you headaches, you can u se full path names
to<BR>> the MPICH2 mpicc and mpif90.<BR>> <BR>> You may need to
compile the ESMF library separately,<BR>> as their makefiles seem to be
hardwired not to use the MPI compiler<BR>> wrappers.<BR>> <BR>>
**<BR>> <BR>> As for 2), CCSM3 is an MPMD program with 5
executables.<BR>> It cannot work correctly if you delete one of
them.<BR>> You actually eliminated the flux coupler, which
coordinates<BR>> the work of all other four components.<BR>> The other
components only talk to the coupler.<BR>> Therefore, what probably
happens<BR>> is that the other four executables are waiting<BR>>
forever for the flux coupler to answer.<BR>> <BR>> **<BR>> <BR>>
As for 3), besides requiring a substantial number of CPUs ,<BR>> CCSM3
also needs a significant amount of memory.<BR>> On how many nodes, and
with how much memory on each,<BR>> are you trying to run the job?<BR>>
Which resolution (T42, T31, T85)?<BR>& gt; <BR>> In any case,
increasing the number of processors<BR>> will not solve the MPI error
message of 1),<BR>> which requires using the correct mpi.h.<BR>>
<BR>> **<BR>> <BR>> Only question 1) is a general MPI/MPICH
question.<BR>> Questions 2) and 3) are specific CCSM3 issues.<BR>> It
may be more productive to discuss them in the CCSM3 forum.<BR>> <BR>>
In any case, let's hope you can get additional help here also.<BR>>
<BR>> **<BR>> <BR>> I hope this helps.<BR>> Gus Correa<BR>>
---------------------------------------------------------------------<BR>>
Gustavo Correa<BR>> Lamont-Doherty Earth Observatory - Columbia
University<BR>> Palisades, NY, 10964-8000 - USA<BR>>
--------------------------------------
-------------------------------<BR>> <BR>> LSB wrote:<BR>> > Hi
everyone,<BR>> > <BR>> > I want to run Community Climate System
Model on our machine under <BR>> > MPICH2. I compile d it
successfully. However, I got some error message <BR>> > about mpi
during runnig it.<BR>> > <BR>> > 1) In the run script, I asked
for 32 cpus ( use PBS batch system). After <BR>> > starting up mpd
daemons, I wrote " <BR>> > /mnt/storage-space/disk1/mpich/bin/mpiexec
-l -n 2 $EXEROOT/all/cpl : -n <BR>> > 2 $EXEROOT/all/csim : -n 8
$EXEROOT/all/clm : -n 4 $EXEROOT/all/pop : -n <BR>> > 16
$EXEROOT/all/cam" . <BR>> > The process is over quite quickly after I
qsub it. With error message <BR>> > like:<BR>> > rank 5 in job 1
compute-0-10.local_46741 caused collective abort of <BR>> > all
ranks<BR>> > exit status of rank 5: return code 1<BR>> >
AND<BR>> > 14: Fatal error in MPI_Cart_shift: Invalid communicator,
error stack:<BR>> > 14: MPI_Cart_shift(172):
MPI_Cart_shift(MPI_COMM_NULL, direction=1, <BR>> > displ=1,
source=0x2582aa0, dest=0x2582aa4) failed<BR>> > 14: MPI_Cart_shif
t(80).: Null communi cator<BR>> > 15: Fatal error in MPI_Cart_shift:
Invalid communicator, error stack:<BR>> > 15: MPI_Cart_shift(172):
MPI_Cart_shift(MPI_COMM_NULL, direction=1, <BR>> > displ=1,
source=0x2582aa0, dest=0x2582aa4) failed<BR>> > 5: Assertion failed in
file helper_fns.c at line 337: 0<BR>> > 15: MPI_Cart_shift(80).: Null
communicator<BR>> > 5: memcpy argument memory ranges overlap,
dst_=0xf2c37f4 src_=0xf2c37f4 <BR>> > len_=4<BR>> > 9: Assertion
failed in file helper_fns.c at line 337: 0<BR>> > 5:<BR>> > 9:
memcpy argument memory ranges overlap, dst_=0x1880ce64 <BR>> >
src_=0x1880ce64 len_=4<BR>> > 5: internal ABORT - process 5<BR>>
> 9:<BR>> > 9: internal ABORT - proces s 9<BR>> > 4:
Assertion failed in file helper_fns.c at line 337: 0<BR>> > 4: memcpy
argument memory ranges overlap, dst_=0x1c9615d0 <BR>> >
src_=0x1c9615d0 len_=4<BR>> > 4:<BR>> > 4: internal ABORT -
process 4<BR>> > <BR>> > 2) What quite puzzeled me is that if I
delete any one of the five (cpl, <BR>> > csim, clm, pop, cam ) , the
model can running sucsessfully. For example, <BR>> > delete "cpl", I
wro te "<BR>> > /mnt/storage-space/disk1/mpich/bin/mpiexec -l -n 2
$EXEROOT/all/csim : <BR>> > -n 8 $EXEROOT/all/clm : -n 4
$EXEROOT/all/pop : -n 16 $EXEROOT/all/cam" <BR>> > will be ok.<BR>>
> but if I run all of the five at the same time, the error message as
<BR>> > mentioned above will appear.<BR>> > <BR>> > 3) If
ask for a few more cpus, things may become better, I guess. So I <BR>>
> have a try . Ask for 34 cpus but still use 2+2+8+4+16=32 cpus, mpi
<BR>> > error messa ge still exists.<BR>> > <BR>> > How
should I solve the problem?<BR>> > Anyone can give some
suggestions?<BR>> > <BR>> > Thanks in advace!<BR>> >
<BR>> > <BR>> > L. S<BR>> > <BR>>
_______________________________________________<BR>> mpich-discuss
mailing list<BR>> mpich-discuss@mcs.anl.gov<BR>>
https://lists.mcs.anl.gov/mailman/listinfo/mpich-discuss<BR><BR>
<HR>
更多热辣资讯尽在新版MSN首页！ <A href="http://cn.msn.com/">立刻访问！</A> </BLOCKQUOTE><BR>
<HR>
聊天+搜索+邮箱 想要轻松出游,手机MSN帮你搞定! <A href="http://3g.msn.cn/" target=_new>立刻下载！</A>
</B></BLOCKQUOTE></BODY></HTML>