<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN">
<HTML><HEAD><TITLE>General Scalability Question</TITLE>
<META http-equiv=Content-Type content="text/html; charset=us-ascii">
<META content="MSHTML 6.00.6000.16890" name=GENERATOR></HEAD>
<BODY>
<DIV dir=ltr align=left><SPAN class=285374115-26102009><FONT face=Arial
color=#0000ff size=2>Dave,</FONT></SPAN></DIV>
<DIV dir=ltr align=left><SPAN class=285374115-26102009><FONT face=Arial
color=#0000ff size=2>So does that imply you wrote the app from square one to use
shared memory? Or is that part of how mpi gets invoked. One of my applications
(GASP) uses lam-mpi. And it appears that I get only one mpi process per node
with multiple application instances. </FONT></SPAN></DIV>
<DIV dir=ltr align=left><SPAN class=285374115-26102009><FONT face=Arial
color=#0000ff size=2></FONT></SPAN> </DIV>
<DIV dir=ltr align=left><SPAN class=285374115-26102009><FONT face=Arial
color=#0000ff size=2>Is this accomplished via the use of the "nemisis" or "mt"
channel?</FONT></SPAN></DIV>
<DIV dir=ltr align=left><SPAN class=285374115-26102009><FONT face=Arial
color=#0000ff size=2></FONT></SPAN> </DIV>
<DIV dir=ltr align=left><SPAN class=285374115-26102009><FONT face=Arial
color=#0000ff size=2>Thanks</FONT></SPAN></DIV>
<DIV dir=ltr align=left><SPAN class=285374115-26102009><FONT face=Arial
color=#0000ff size=2>- Andy</FONT></SPAN></DIV>
<DIV> </DIV><!-- Converted from text/rtf format -->
<P><FONT face="Times New Roman">Andrew Robertson P.E.</FONT> <BR><FONT
face="Times New Roman">CFD Analyst</FONT> <BR><FONT face="Times New Roman">GASL
Operations</FONT> <BR><FONT face="Times New Roman">Tactical Propulsion and
Controls</FONT> <BR><FONT face="Times New Roman">ATK</FONT> <BR><FONT
face="Times New Roman">77 Raynor Avenue</FONT> <BR><FONT
face="Times New Roman">Ronkokoma NY 11779</FONT> <BR><FONT
face="Times New Roman">631-737-6100 Ext 120</FONT> <BR><FONT
face="Times New Roman">Fax: 631-588-7023</FONT> <BR><FONT
face="Times New Roman">www.atk.com</FONT> </P>
<P><FONT face="Times New Roman">!! Knowledge and Thoroughness Baby !!</FONT>
</P>
<DIV> </DIV><BR>
<DIV class=OutlookMessageHeader lang=en-us dir=ltr align=left>
<HR tabIndex=-1>
<FONT face=Tahoma size=2><B>From:</B> mpich-discuss-bounces@mcs.anl.gov
[mailto:mpich-discuss-bounces@mcs.anl.gov] <B>On Behalf Of </B>Hiatt, Dave M
<BR><B>Sent:</B> Monday, October 26, 2009 11:40 AM<BR><B>To:</B>
mpich-discuss@mcs.anl.gov<BR><B>Subject:</B> Re: [mpich-discuss] General
Scalability Question<BR></FONT><BR></DIV>
<DIV></DIV>
<DIV><SPAN class=621563415-26102009><FONT face=Arial color=#0000ff size=2>So far
my experience has been that the in core message transfer rate is far better than
a gigabyte switch and backbone. Infiniband would be a dramatic improvement
but it's hard to believe that it could keep up with in memory. What has
worked out best for our app is a single message thread, and then the app using
shared memory directly to distribute. That dramatically lowers the number
of open sockets and communication overhead. It may not work best in every
case, but for us it worked better regardless of very high core/process count per
node or lower count per node. So we ran only one MPI process per physical
node. It also lowers the number of sockets you have to support on node 0
if you have point to point communication. Linux at least defaults to 1048
sockets and files, and it's nice for node 0 performance to keep under
that. You can raise it with ulimit, but when you're got 15000 cores, it's
pretty expensive to have one MPI process per core.</FONT></SPAN></DIV>
<BLOCKQUOTE>
<DIV class=OutlookMessageHeader dir=ltr align=left><FONT face=Tahoma
size=2>-----Original Message-----<BR><B>From:</B>
mpich-discuss-bounces@mcs.anl.gov
[mailto:mpich-discuss-bounces@mcs.anl.gov]<B>On Behalf Of </B>Robertson,
Andrew<BR><B>Sent:</B> Monday, October 26, 2009 10:30 AM<BR><B>To:</B>
mpich-discuss@mcs.anl.gov<BR><B>Subject:</B> [mpich-discuss] General
Scalability Question<BR><BR></FONT></DIV><!-- Converted from text/rtf format -->
<P><FONT face=Arial size=2>Folks,</FONT> <BR><FONT face=Arial size=2>Our IT
staff is not particularly knowledgeable about parallel computing. Their
current upgrade plan centers around quad/quad or dual/hex boxes which would
have 16 or 12 cores respectively. I have no doubt that such a machine would
run a parallel job efficiently. My question is how well can I harness multiple
boxes together? </FONT></P>
<P><FONT face=Arial size=2>The applications are all CFD (FLUENT, GASP, STAR,
VULCAN). I am talking to the various software vendors about this but would
like some info from the programming community. </FONT></P>
<P><FONT face=Arial size=2>Assuming the same memory per core am I better off
with</FONT> </P>
<P><FONT face=Arial size=2>High core count (12-16) boxes on a gigabit
switch</FONT> <BR><FONT face=Arial size=2>Lower core count (2 -4) boxes on an
infiniband switch.</FONT> </P>
<P><FONT face=Arial size=2>I understand that if I configure mpich correctly it
will use shared memory on the mutli-core multi-processor boxes. If I end up
with the high core count boxes, should I spec the frontside bus (or whatever
it is called now) as high as possible??</FONT></P>
<P><FONT face=Arial size=2>I also have concerns that a single power supply
failure takes out more cores, though perhaps that is not such a problem</FONT>
</P>
<P><FONT face=Arial size=2>Any information is greatly appreciated</FONT>
</P><BR>
<P><FONT face=Arial size=2>Thanks</FONT> <BR><FONT face=Arial
size=2>Andy</FONT> <BR><FONT face=Arial size=2>--------------------</FONT>
<BR><FONT face="Times New Roman">Andrew Robertson P.E.</FONT> <BR><FONT
face="Times New Roman">CFD Analyst</FONT> <BR><FONT
face="Times New Roman">GASL Operations</FONT> <BR><FONT
face="Times New Roman">Tactical Propulsion and Controls</FONT> <BR><FONT
face="Times New Roman">ATK</FONT> <BR><FONT face="Times New Roman">77 Raynor
Avenue</FONT> <BR><FONT face="Times New Roman">Ronkokoma NY 11779</FONT>
<BR><FONT face="Times New Roman">631-737-6100 Ext 120</FONT> <BR><FONT
face="Times New Roman">Fax: 631-588-7023</FONT> <BR><A
href="file://www.atk.com"><U><FONT face="Times New Roman"
color=#0000ff>www.atk.com</FONT></U></A> </P>
<P><FONT face="Times New Roman">!! Knowledge and Thoroughness Baby !!</FONT>
</P><BR></BLOCKQUOTE></BODY></HTML>