<div class="gmail_quote">On Fri, May 11, 2012 at 4:20 PM, Jeff Hammond <span dir="ltr"><<a href="mailto:jhammond@alcf.anl.gov" target="_blank">jhammond@alcf.anl.gov</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div id=":20g">that's probably greatly underestimating the cost of this function<br>
since i assume in this test every time the function is called, both<br>
the dcache and icache hit every time.</div></blockquote><div><br></div><div>Actually, that's kinda the case I'm interested in. If the threads pull in enough data to knock the attribute out of L1, chances are that they will take at least a few microseconds (except in pathological cases that hit associativity). Suppose I'm doing BLAS level 1 flavor of vector operations with vectors of length a couple thousand, so just big enough to be out of L1 if done in serial. But with 16 or 32 threads, the actual work is very fast (order of 100 cycles) because we have enough L1 and the operations don't conflict with other cache lines. There are no atomic instructions in launching a kernel, though there is a write from one thread and a read from another, so the writer needs to get exclusive access to a cache line and then the reader need to get the line back. But that line shuffling doesn't affect whether MPI's attribute table stays in cache.</div>
<div><br></div><div>I can make an example to either validate or support the discussion above</div><div><br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div id=":20g">
</div></blockquote><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div id=":20g"><div class="im">
</div>Is "-O2" a suboption to "-pipe" or are you giving the compiler<br>
conflicting flags?<br></div></blockquote><div><br></div><div>No, MPICH2 slaps it's own -O2 on the end of whatever the user asked for, -pipe is irrelevant for optimization and perhaps not useful any more (it used to reduce file system traffic by having gcc using pipes instead of temporary files). The last optimization option is used in any case, so it doesn't matter.</div>
<div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div id=":20g">
<div class="im"><br>
> MPICH2 Version: 1.5b1<br>
> MPICH2 Release date: unreleased development copy<br>
> MPICH2 Device: ch3:nemesis<br>
> MPICH2 configure: --prefix=/homes/jedbrown/usr/mpich-intel --enable-shared<br>
> --enable-error-checking=runtime --enable-error-messages=all<br>
> --enable-timer-type=clock_gettime CC=icc CXX=icpc --enable-fc=0<br>
> --enable-f77=0 FC= F77=<br>
> MPICH2 CC: icc -O2<br>
> MPICH2 CXX: icpc -O2<br>
> MPICH2 F77: gfortran<br>
> MPICH2 FC: gfortran<br>
<br>
</div>"--enable-error-checking=runtime --enable-error-messages=all" would<br>
seem to be the kind of thing Dave is talking about that affect<br>
performance.</div></blockquote><div><br></div><div>Right, so should I turn that off? This is a development environment, so I definitely want those options. I can build a different MPI for profiling, but it's been irrelevant in other tests. Can't the overhead of run-time error checking amount to a few unlikely conditionals?</div>
<div><br></div><div>I could also build MPICH2 with all error checking turned off, but with debugging symbols so that I can determine which lines are sucking up the time?</div></div>