Thanks all,<br><br>The Cactus code has a good scalability, especially with the latest version of Carpet it shows a good scalabitly over 5000 cpu cores(?). I tested both BSSN+PUGH and Whisky+Carpet benchmarks. Not an expert, I'm depending on the timing info shown by Cactus rather than use profiling tools introduced by Pavan. The profiling info says that most of communication time was taken to enforce boundary conditions. The total wall clock time (including communication time) increases from ~ 700sec (1CPU) to ~1500 sec (64CPU) whereas computation time only increases ~600 to ~800 sec. Here the problem sizes were take to be proportional to the number of cpus. So now I'm looking for the ways to reduce the communication time. <br>
<br>I'm using Harpertown 5420 (2.5GHz). What makes me disappointed more is this newest Xeon cpu cluster is not that superior to my old Pentium D 930 cluster (3.0GHz) which having 4 nodes (8 cores) . I tested various combinations of (node# x cpu#) and the results somewhat depends on the combinations. <br>
<br>Hybrid run using "-openmp" option of Intel compilers made things worse and had broke loadbalancing. Also the optimization options (even -O2) made runs slower but did not break the load balancing.<br><br>I checked the bandwidth behavior mentioned by Elvedin. Could I change or setup the message size and frequency in a runtime level or any other steps?<br>
<br>I have no idea how to improve the scalability and how serious it is. Anyway it's a bit <a href="http://unsatisfatory.at">unsatisfatory.at</a> the moment and I hope I can find a better way from here. I appreciate all your kind comments and suggestions. <br>
<br>Regards<br><br>Kim, Hee Il<br> <br><br> <br><div><span class="gmail_quote">2008/3/28, Brian Dobbins <<a href="mailto:bdobbins@gmail.com">bdobbins@gmail.com</a>>:</span><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
Hi,<br><br> I don't use the Cactus code myself, but from what little I <i>do</i> know of it, this might not be unexpected. For starters, what do you mean by 'bad scalability'? I believe most (all?) benchmark cases demonstrate what is called 'weak scaling' - that is, the problem size increases along with the number of processors. So, running on 1 processor gives you a wall-clock time of <i>n</i> seconds, and running on 2 processors will probably give you a problem size of <i>n</i>+<some small number>. That small number is the communication time of your code. Thus, running on 80 cores <i>will</i> be slower than running on 1, but it'll let you run a much larger system.<br>
<br> (To clarify, unless you're specifically configuring a constant problem size, you won't reduce your time to solution by increasing your processors.)<br><br> The next thing to consider is which benchmark you're using, as some of them are more scalable than others. You're likely to get different results when looking at the 'Whiskey Carpet' benchmark vs. the 'BSSN_PUGH' one. You might wish to take a look at the benchmark database at the Cactus website, and there are some PDF files with more information, too, including a master's thesis on benchmark performance.<br>
<br> Finally, a few other slightly more technical things to consider are:<br><br>(1) What kind of Harpertowns are you using? Looking at the 5450 vs. the 5472 (both 3.0 Ghz chips), the latter has more memory bandwidth, and may scale better since the code does appear to make use of it. Using the CPU2006fp_rate CactusADM benchmarks as a first approximation to parallel performance, the SPEC website shows that a 5450 gets a score of 101 and 73.1 when going from 1 -> 8 cores (and larger is better here - this is throughput, not wall-clock time), and the 5472 goes from 112 -> 84.8. Why does this matter? Well, you'll probably get different results running an 8-core job when running that as 8x1, 4x2, or 1x8 (cores x nodes). This will impact your benchmark results somewhat.<br>
<br> (2) I think the code supports running with MPI and OpenMP... I don't know if there will be any difference in performance if you choose to run 1 MPI process per node with 8 OpenMP threads vs. simply using 8 MPI processes, but it might be worth looking into.<br>
<br> (3) Again, I have no first-hand knowledge of the code's performance under different interconnects, but it <i>does</i> seem likely to make a difference... chances are if you asked on the Cactus forums, there might be someone with first-hand experience with this who could give you some more specific information. <br>
<br> Hope that helps, and if any of it isn't clear, I'll be happy to try to clarify. Good luck!<br><br> Cheers,<br><span class="sg"> - Brian<br><br>
</span></blockquote></div><br>