Thanks for the helpful response Jed. I was not aware of the possibility to run seperate PETSC_COMM_WORLDS in the same program, at least this is not clear from the documentation (e.g. <a href="http://www.mcs.anl.gov/petsc/petsc-as/snapshots/petsc-dev/docs/manualpages/Sys/PetscInitialize.html">http://www.mcs.anl.gov/petsc/petsc-as/snapshots/petsc-dev/docs/manualpages/Sys/PetscInitialize.html</a>) I'll probably still try this out just out of curiosity. <br>
<br>About presenting scaling results, the most appealing to me seems to show two graphs, one with intra-node scaling (1-12) and the other going upwards from there (12, 24, 36, ...) <br><br>Leo<br><br><br><br><br><br><div class="gmail_quote">
2010/9/21 Jed Brown <span dir="ltr"><<a href="mailto:jed@59a2.org">jed@59a2.org</a>></span><br><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
<div class="im">On Tue, Sep 21, 2010 at 10:41, Leo van Kampenhout<br>
<<a href="mailto:lvankampenhout@gmail.com">lvankampenhout@gmail.com</a>> wrote:<br>
> At the cluster I am currently working on, each node is made up by 12 PEs and<br>
> have shared memory. When I would just reserve 1 PE for my job, the other 11<br>
> processors are given to other users, therefore giving dynamic load on the<br>
> memory system resulting into inaccurate timings. The solve-times I get are<br>
> ranging between 1 and 5 minutes. For me, this is not very scientific either.<br>
<br>
</div>First, shared memory and especially NUMA architectures are very<br>
difficult to draw meaningful intra-node scalability conclusions on.<br>
If at all possible, try to compare inter-node scalability instead<br>
since it is a far more reliable estimate and less<br>
architecture-dependent (provided the network is decent). That said,<br>
you should be looking for reproducibility much more than "good"<br>
scaling. It's well known that intra-node memory contention is a major<br>
issue, the STREAM benchmarks actually show _lower_ total bandwidth<br>
when running on all 6 cores per socket with Istanbul than when using<br>
only 4 (and 2 cores is within a few percent).<br>
<div class="im"><br>
> The second idea was to reserve all 12 PEs on the node and just let 1 PE run<br>
> the job. However, in this way the single CPU gets all the memory bandwidth<br>
> and has no waiting time, therefore giving very fast results. When I would<br>
> calculate speedup from here, the algorithm does not scale very well.<br>
<br>
</div>I say just do this and live with the poor intra-node scaling numbers.<br>
Some architectures actually scale memory within the node (e.g.<br>
BlueGene), but most don't. People expect to see the memory bottleneck<br>
in these results, it's nothing to be ashamed of.<br>
<div class="im"><br>
> Another idea would be to spawn 12 identical jobs on 12 PEs and take the<br>
> average runtime. Unfortunately, there is only one PETSC_COMM_WORLD, so I<br>
> think this is impossible to do from within one program (MPI_COMM_WORLD).<br>
<br>
</div>You could split MPI_COMM_WORLD and run a separate PETSC_COMM_WORLD on<br>
each group, but I think this option will not be reproducible (the<br>
instances will slightly out of sync, so memory and communication<br>
bottlenecks will be loaded in different ways on subsequent runs) and<br>
is a bit disingenuous because this is not a configuration that you<br>
would ever run in practice.<br>
<font color="#888888"><br>
Jed<br>
</font></blockquote></div><br>