<div>I have been watching this, and the other, scalability thread. Here is my observation</div> <div>on the dual/quad core boxes after spending the last few years playing with these boxes:</div> <div>- your SW is the main factor in scalability.</div> <div>- Scalability should be per physical CPU, not core.</div> <div>- keeping all cores busy is likely to bring down the performance/core. </div> <div>- the latest quad core CPUs are notoriusly bad in throughput if you use all/most the cores.</div> <div> -- shared cache thrashing ???</div> <div>- hyperthreading can reduce throughput</div> <div>- memory bandwidth is likely the limiting factors on multi-core boxes, even on SUN's Niagara.</div> <div>- you need to tune your algorithm according to the HW you have. You can't rely solely on MPICH to deliver the scalability.</div> <div>- If you really want performance, use the simplest MPICH
routines. I have reduced my MPICH calls to fixed point-point comms, except at entry where I have the only Barrier call. I don't even use ISend/IRecv (they are extremely bad for me for reasons I can't confirm). </div> <div> </div> <div>tan</div> <div><BR><B><I>Pavan Balaji <balaji@mcs.anl.gov></I></B> wrote:</div> <BLOCKQUOTE class=replbq style="PADDING-LEFT: 5px; MARGIN-LEFT: 5px; BORDER-LEFT: #1010ff 2px solid">Hee,<BR><BR>Can you send us this code? I'm interested in seeing what is causing the <BR>communication time to go up so much.<BR><BR>-- Pavan<BR><BR>On 03/28/2008 12:21 PM, Hee Il Kim wrote:<BR>> Thanks all,<BR>> <BR>> The Cactus code has a good scalability, especially with the latest <BR>> version of Carpet it shows a good scalabitly over 5000 cpu cores(?). I <BR>> tested both BSSN+PUGH and Whisky+Carpet benchmarks. Not an expert, I'm <BR>> depending on the timing info shown by Cactus rather than use profiling
<BR>> tools introduced by Pavan. The profiling info says that most of <BR>> communication time was taken to enforce boundary conditions. The total <BR>> wall clock time (including communication time) increases from ~ 700sec <BR>> (1CPU) to ~1500 sec (64CPU) whereas computation time only increases ~600 <BR>> to ~800 sec. Here the problem sizes were take to be proportional to the <BR>> number of cpus. So now I'm looking for the ways to reduce the <BR>> communication time.<BR>> <BR>> I'm using Harpertown 5420 (2.5GHz). What makes me disappointed more is <BR>> this newest Xeon cpu cluster is not that superior to my old Pentium D <BR>> 930 cluster (3.0GHz) which having 4 nodes (8 cores) . I tested various <BR>> combinations of (node# x cpu#) and the results somewhat depends on the <BR>> combinations.<BR>> <BR>> Hybrid run using "-openmp" option of Intel compilers made things worse <BR>> and had broke loadbalancing. Also the
optimization options (even -O2) <BR>> made runs slower but did not break the load balancing.<BR>> <BR>> I checked the bandwidth behavior mentioned by Elvedin. Could I change or <BR>> setup the message size and frequency in a runtime level or any other steps?<BR>> <BR>> I have no idea how to improve the scalability and how serious it is. <BR>> Anyway it's a bit unsatisfatory.at <HTTP: unsatisfatory.at>the moment <BR>> and I hope I can find a better way from here. I appreciate all your <BR>> kind comments and suggestions.<BR>> <BR>> Regards<BR>> <BR>> Kim, Hee Il<BR>> <BR>> <BR>> <BR>> 2008/3/28, Brian Dobbins <BDOBBINS@GMAIL.COM <mailto:bdobbins@gmail.com>>:<BR>> <BR>> Hi,<BR>> <BR>> I don't use the Cactus code myself, but from what little I /do/<BR>> know of it, this might not be unexpected. For starters, what do you<BR>> mean by 'bad scalability'? I believe most (all?) benchmark cases<BR>>
demonstrate what is called 'weak scaling' - that is, the problem<BR>> size increases along with the number of processors. So, running on<BR>> 1 processor gives you a wall-clock time of /n/ seconds, and running<BR>> on 2 processors will probably give you a problem size of /n/+<SOME<BR>> small number>. That small number is the communication time of your<BR>> code. Thus, running on 80 cores /will/ be slower than running on 1,<BR>> but it'll let you run a much larger system.<BR>> <BR>> (To clarify, unless you're specifically configuring a constant<BR>> problem size, you won't reduce your time to solution by increasing<BR>> your processors.)<BR>> <BR>> The next thing to consider is which benchmark you're using, as<BR>> some of them are more scalable than others. You're likely to get<BR>> different results when looking at the 'Whiskey Carpet' benchmark vs.<BR>> the 'BSSN_PUGH' one. You might wish to take a look at the
benchmark<BR>> database at the Cactus website, and there are some PDF files with<BR>> more information, too, including a master's thesis on benchmark<BR>> performance.<BR>> <BR>> Finally, a few other slightly more technical things to consider are:<BR>> <BR>> (1) What kind of Harpertowns are you using? Looking at the 5450 vs.<BR>> the 5472 (both 3.0 Ghz chips), the latter has more memory bandwidth,<BR>> and may scale better since the code does appear to make use of it. <BR>> Using the CPU2006fp_rate CactusADM benchmarks as a first<BR>> approximation to parallel performance, the SPEC website shows that a<BR>> 5450 gets a score of 101 and 73.1 when going from 1 -> 8 cores (and<BR>> larger is better here - this is throughput, not wall-clock time),<BR>> and the 5472 goes from 112 -> 84.8. Why does this matter? Well,<BR>> you'll probably get different results running an 8-core job when<BR>> running that as 8x1, 4x2, or 1x8
(cores x nodes). This will impact<BR>> your benchmark results somewhat.<BR>> <BR>> (2) I think the code supports running with MPI and OpenMP... I<BR>> don't know if there will be any difference in performance if you<BR>> choose to run 1 MPI process per node with 8 OpenMP threads vs.<BR>> simply using 8 MPI processes, but it might be worth looking into.<BR>> <BR>> (3) Again, I have no first-hand knowledge of the code's performance<BR>> under different interconnects, but it /does/ seem likely to make a<BR>> difference... chances are if you asked on the Cactus forums, there<BR>> might be someone with first-hand experience with this who could give<BR>> you some more specific information. <BR>> <BR>> Hope that helps, and if any of it isn't clear, I'll be happy to<BR>> try to clarify. Good luck!<BR>> <BR>> Cheers,<BR>> - Brian<BR>> <BR>> <BR><BR>-- <BR>Pavan
Balaji<BR>http://www.mcs.anl.gov/~balaji<BR><BR></BLOCKQUOTE><BR><p> 
<hr size=1>Be a better friend, newshound, and
know-it-all with Yahoo! Mobile. <a href="http://us.rd.yahoo.com/evt=51733/*http://mobile.yahoo.com/;_ylt=Ahu06i62sR8HDtDypao8Wcj9tAcJ "> Try it now.</a>