[mpich-discuss] General Scalability Question

[Hiatt, Dave M]

Yes, it was.  I'm on 1.07 so the default was not Nemesis.  I think 1.1 defaults from ch3 to nemesis, but haven't tried that yet.  But yes the app was written to attack the problem this way and to lower the number of comm processes on purpose.  We tried several experiments and that's what worked well for us.  We also started on gigabyte Ethernet, but Infiniband tests were lots better.  But we push through a smaller number of messages that are pretty big, 400K.

-----Original Message-----
From: mpich-discuss-bounces at mcs.anl.gov [mailto:mpich-discuss-bounces at mcs.anl.gov]On Behalf Of Robertson, Andrew
Sent: Monday, October 26, 2009 10:47 AM
To: mpich-discuss at mcs.anl.gov
Subject: Re: [mpich-discuss] General Scalability Question


Dave,
So does that imply you wrote the app from square one to use shared memory? Or is that part of how mpi gets invoked. One of my applications (GASP) uses lam-mpi. And it appears that I get only one mpi process per node with multiple application instances.

Is this accomplished via the use of the "nemisis" or "mt" channel?

Thanks
- Andy


Andrew Robertson P.E.
CFD Analyst
GASL Operations
Tactical Propulsion and Controls
ATK
77 Raynor Avenue
Ronkokoma NY 11779
631-737-6100 Ext 120
Fax: 631-588-7023
www.atk.com

!! Knowledge and Thoroughness Baby !!



  _____

From: mpich-discuss-bounces at mcs.anl.gov [mailto:mpich-discuss-bounces at mcs.anl.gov] On Behalf Of Hiatt, Dave M
Sent: Monday, October 26, 2009 11:40 AM
To: mpich-discuss at mcs.anl.gov
Subject: Re: [mpich-discuss] General Scalability Question


So far my experience has been that the in core message transfer rate is far better than a gigabyte switch and backbone.  Infiniband would be a dramatic improvement but it's hard to believe that it could keep up with in memory.  What has worked out best for our app is a single message thread, and then the app using shared memory directly to distribute.  That dramatically lowers the number of open sockets and communication overhead.  It may not work best in every case, but for us it worked better regardless of very high core/process count per node or lower count per node.  So we ran only one MPI process per physical node.  It also lowers the number of sockets you have to support on node 0 if you have point to point communication.  Linux at least defaults to 1048 sockets and files, and it's nice for node 0 performance to keep under that.  You can raise it with ulimit, but when you're got 15000 cores, it's pretty expensive to have one MPI process per core.

-----Original Message-----
From: mpich-discuss-bounces at mcs.anl.gov [mailto:mpich-discuss-bounces at mcs.anl.gov]On Behalf Of Robertson, Andrew
Sent: Monday, October 26, 2009 10:30 AM
To: mpich-discuss at mcs.anl.gov
Subject: [mpich-discuss] General Scalability Question



Folks,
Our IT staff is not particularly knowledgeable about parallel computing. Their current upgrade plan centers around quad/quad or dual/hex boxes which would have 16 or 12 cores respectively. I have no doubt that such a machine would run a parallel job efficiently. My question is how well can I harness multiple boxes together?

The applications are all CFD (FLUENT, GASP, STAR, VULCAN). I am talking to the various software vendors about this but would like some info from the programming community.

Assuming the same memory per core am I better off with

High core count (12-16) boxes on a gigabit switch
Lower core count (2 -4) boxes on an infiniband switch.

I understand that if I configure mpich correctly it will use shared memory on the mutli-core multi-processor boxes. If I end up with the high core count boxes, should I spec the frontside bus (or whatever it is called now) as high as possible??

I also have concerns that a single power supply failure takes out more cores, though perhaps that is not such a problem

Any information is greatly appreciated


Thanks
Andy
--------------------
Andrew Robertson P.E.
CFD Analyst
GASL Operations
Tactical Propulsion and Controls
ATK
77 Raynor Avenue
Ronkokoma NY 11779
631-737-6100 Ext 120
Fax: 631-588-7023
www.atk.com<file://www.atk.com>

!! Knowledge and Thoroughness Baby !!


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/mpich-discuss/attachments/20091026/093bf7b0/attachment.htm>