<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN">
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
<title>General Scalability Question</title>
<meta content="MSHTML 6.00.2800.1634" name="GENERATOR">
</head>
<body>
<div><span class="560424915-26102009"><font face="Arial" color="#0000ff" size="2">Yes, it was. I'm on 1.07 so the default was not Nemesis. I think 1.1 defaults from ch3 to nemesis, but haven't tried that yet. But yes the app was written to attack the problem
this way and to lower the number of comm processes on purpose. We tried several experiments and that's what worked well for us. We also started on gigabyte Ethernet, but Infiniband tests were lots better. But we push through a smaller number of messages
that are pretty big, 400K.</font></span></div>
<blockquote dir="ltr" style="MARGIN-RIGHT: 0px">
<div class="OutlookMessageHeader" dir="ltr" align="left"><font face="Tahoma" size="2">-----Original Message-----<br>
<b>From:</b> mpich-discuss-bounces@mcs.anl.gov [mailto:mpich-discuss-bounces@mcs.anl.gov]<b>On Behalf Of
</b>Robertson, Andrew<br>
<b>Sent:</b> Monday, October 26, 2009 10:47 AM<br>
<b>To:</b> mpich-discuss@mcs.anl.gov<br>
<b>Subject:</b> Re: [mpich-discuss] General Scalability Question<br>
<br>
</font></div>
<div dir="ltr" align="left"><span class="285374115-26102009"><font face="Arial" color="#0000ff" size="2">Dave,</font></span></div>
<div dir="ltr" align="left"><span class="285374115-26102009"><font face="Arial" color="#0000ff" size="2">So does that imply you wrote the app from square one to use shared memory? Or is that part of how mpi gets invoked. One of my applications (GASP) uses lam-mpi.
And it appears that I get only one mpi process per node with multiple application instances.
</font></span></div>
<div dir="ltr" align="left"><span class="285374115-26102009"><font face="Arial" color="#0000ff" size="2"></font></span> </div>
<div dir="ltr" align="left"><span class="285374115-26102009"><font face="Arial" color="#0000ff" size="2">Is this accomplished via the use of the "nemisis" or "mt" channel?</font></span></div>
<div dir="ltr" align="left"><span class="285374115-26102009"><font face="Arial" color="#0000ff" size="2"></font></span> </div>
<div dir="ltr" align="left"><span class="285374115-26102009"><font face="Arial" color="#0000ff" size="2">Thanks</font></span></div>
<div dir="ltr" align="left"><span class="285374115-26102009"><font face="Arial" color="#0000ff" size="2">- Andy</font></span></div>
<div> </div>
<!-- Converted from text/rtf format -->
<p><font face="Times New Roman">Andrew Robertson P.E.</font> <br>
<font face="Times New Roman">CFD Analyst</font> <br>
<font face="Times New Roman">GASL Operations</font> <br>
<font face="Times New Roman">Tactical Propulsion and Controls</font> <br>
<font face="Times New Roman">ATK</font> <br>
<font face="Times New Roman">77 Raynor Avenue</font> <br>
<font face="Times New Roman">Ronkokoma NY 11779</font> <br>
<font face="Times New Roman">631-737-6100 Ext 120</font> <br>
<font face="Times New Roman">Fax: 631-588-7023</font> <br>
<font face="Times New Roman">www.atk.com</font> </p>
<p><font face="Times New Roman">!! Knowledge and Thoroughness Baby !!</font> </p>
<div> </div>
<br>
<div class="OutlookMessageHeader" lang="en-us" dir="ltr" align="left">
<hr tabindex="-1">
<font face="Tahoma" size="2"><b>From:</b> mpich-discuss-bounces@mcs.anl.gov [mailto:mpich-discuss-bounces@mcs.anl.gov]
<b>On Behalf Of </b>Hiatt, Dave M <br>
<b>Sent:</b> Monday, October 26, 2009 11:40 AM<br>
<b>To:</b> mpich-discuss@mcs.anl.gov<br>
<b>Subject:</b> Re: [mpich-discuss] General Scalability Question<br>
</font><br>
</div>
<div></div>
<div><span class="621563415-26102009"><font face="Arial" color="#0000ff" size="2">So far my experience has been that the in core message transfer rate is far better than a gigabyte switch and backbone. Infiniband would be a dramatic improvement but it's hard
to believe that it could keep up with in memory. What has worked out best for our app is a single message thread, and then the app using shared memory directly to distribute. That dramatically lowers the number of open sockets and communication overhead.
It may not work best in every case, but for us it worked better regardless of very high core/process count per node or lower count per node. So we ran only one MPI process per physical node. It also lowers the number of sockets you have to support on node
0 if you have point to point communication. Linux at least defaults to 1048 sockets and files, and it's nice for node 0 performance to keep under that. You can raise it with ulimit, but when you're got 15000 cores, it's pretty expensive to have one MPI process
per core.</font></span></div>
<blockquote>
<div class="OutlookMessageHeader" dir="ltr" align="left"><font face="Tahoma" size="2">-----Original Message-----<br>
<b>From:</b> mpich-discuss-bounces@mcs.anl.gov [mailto:mpich-discuss-bounces@mcs.anl.gov]<b>On Behalf Of
</b>Robertson, Andrew<br>
<b>Sent:</b> Monday, October 26, 2009 10:30 AM<br>
<b>To:</b> mpich-discuss@mcs.anl.gov<br>
<b>Subject:</b> [mpich-discuss] General Scalability Question<br>
<br>
</font></div>
<!-- Converted from text/rtf format -->
<p><font face="Arial" size="2">Folks,</font> <br>
<font face="Arial" size="2">Our IT staff is not particularly knowledgeable about parallel computing. Their current upgrade plan centers around quad/quad or dual/hex boxes which would have 16 or 12 cores respectively. I have no doubt that such a machine would
run a parallel job efficiently. My question is how well can I harness multiple boxes together?
</font></p>
<p><font face="Arial" size="2">The applications are all CFD (FLUENT, GASP, STAR, VULCAN). I am talking to the various software vendors about this but would like some info from the programming community.
</font></p>
<p><font face="Arial" size="2">Assuming the same memory per core am I better off with</font>
</p>
<p><font face="Arial" size="2">High core count (12-16) boxes on a gigabit switch</font>
<br>
<font face="Arial" size="2">Lower core count (2 -4) boxes on an infiniband switch.</font>
</p>
<p><font face="Arial" size="2">I understand that if I configure mpich correctly it will use shared memory on the mutli-core multi-processor boxes. If I end up with the high core count boxes, should I spec the frontside bus (or whatever it is called now) as
high as possible??</font></p>
<p><font face="Arial" size="2">I also have concerns that a single power supply failure takes out more cores, though perhaps that is not such a problem</font>
</p>
<p><font face="Arial" size="2">Any information is greatly appreciated</font> </p>
<br>
<p><font face="Arial" size="2">Thanks</font> <br>
<font face="Arial" size="2">Andy</font> <br>
<font face="Arial" size="2">--------------------</font> <br>
<font face="Times New Roman">Andrew Robertson P.E.</font> <br>
<font face="Times New Roman">CFD Analyst</font> <br>
<font face="Times New Roman">GASL Operations</font> <br>
<font face="Times New Roman">Tactical Propulsion and Controls</font> <br>
<font face="Times New Roman">ATK</font> <br>
<font face="Times New Roman">77 Raynor Avenue</font> <br>
<font face="Times New Roman">Ronkokoma NY 11779</font> <br>
<font face="Times New Roman">631-737-6100 Ext 120</font> <br>
<font face="Times New Roman">Fax: 631-588-7023</font> <br>
<a href="file://www.atk.com"><u><font face="Times New Roman" color="#0000ff">www.atk.com</font></u></a>
</p>
<p><font face="Times New Roman">!! Knowledge and Thoroughness Baby !!</font> </p>
<br>
</blockquote>
</blockquote>
</body>
</html>