<html><head></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; ">Jim,<div><br></div><div>The MPI_WIN_CREATE routine would definitely be called quite regularly. I didn't realize the implementation of MPI_WIN_CREATE was so expensive, so I might have been more liberal in my use of using the routine than necessary i.e. I think I can definitely reuse the buffers since the routine is called every iteration and the window size is constant for all processes across each iteration.</div><div><br></div><div>Tim.</div><div><br><div><div>On May 30, 2012, at 12:03 PM, Jim Dinan wrote:</div><br class="Apple-interchange-newline"><blockquote type="cite"><div>Hi Tim,<br><br>How often are you creating windows? As Jed mentioned, this is expected <br>to be fairly expensive and synchronizing on most systems. The Cray XE <br>has some special sauce that can make this cheap if you go through DMAPP <br>directly, but if you want your performance tuning to be portable, taking <br>window creation off the critical path would be a good change to make.<br><br> ~Jim.<br><br>On 5/30/12 10:48 AM, Timothy Stitt wrote:<br><blockquote type="cite">Thanks Jeff...you provided some good suggestions. I'll consult the DMAPP<br></blockquote><blockquote type="cite">documentation and also go back to the code to see if I can reuse window<br></blockquote><blockquote type="cite">buffers in some way.<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">Would you happen to have links to the DMAPP docs on-hand? I couldn't<br></blockquote><blockquote type="cite">seem to find any tutorials etc. after a quick browse.<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">Cheers,<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">Tim.<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">On May 30, 2012, at 11:40 AM, Jeff Hammond wrote:<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite"><blockquote type="cite">If you don't care about portability, translating from MPI-2 RMA to<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">DMAPP is mostly trivial and you can eliminate collective window<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">creation altogether. However, I will note that my experience getting<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">MPI and DMAPP to inter-operate properly on XE6 (Hopper, in fact) was<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">terrible. And yes, I did everything the NERSC documentation and Cray<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">told me to do.<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">I wonder if you can reduce the time spent in MPI_WIN_CREATE by calling<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">it less often. Can you not allocate the window once and keep reusing<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">it? You might need to restructure your code to reuse the underlying<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">local buffers but that isn't that complicated in some cases.<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">Best,<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">Jeff<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">On Wed, May 30, 2012 at 10:36 AM, Jed Brown <<a href="mailto:jedbrown@mcs.anl.gov">jedbrown@mcs.anl.gov</a><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><<a href="mailto:jedbrown@mcs.anl.gov">mailto:jedbrown@mcs.anl.gov</a>>> wrote:<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">On Wed, May 30, 2012 at 10:29 AM, Timothy Stitt<br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><<a href="mailto:Timothy.Stitt.9@nd.edu">Timothy.Stitt.9@nd.edu</a> <<a href="mailto:Timothy.Stitt.9@nd.edu">mailto:Timothy.Stitt.9@nd.edu</a>>><br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">wrote:<br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><br></blockquote></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">Hi all,<br></blockquote></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><br></blockquote></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">I am currently trying to improve the scaling of a CFD code on some Cray<br></blockquote></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">machines at NERSC (I believe Cray systems leverage mpich2 for their MPI<br></blockquote></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">communications, hence the posting to this list) and I am running<br></blockquote></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">into some<br></blockquote></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">scalability issues with the MPI_WIN_CREATE() routine.<br></blockquote></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><br></blockquote></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">To cut a long story short, the CFD code requires each process to receive<br></blockquote></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">values from some neighborhood processes. Unfortunately, each process<br></blockquote></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">doesn't<br></blockquote></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">know who its neighbors should be in advance.<br></blockquote></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">How often do the neighbors change? By what mechanism?<br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><br></blockquote></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">To overcome this we exploit the one-sided MPI_PUT() routine to<br></blockquote></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">communicate<br></blockquote></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">data from neighbors directly.<br></blockquote></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><br></blockquote></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">Recent profiling at 256, 512 and 1024 processes shows that the<br></blockquote></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">MPI_WIN_CREATE routine is starting to dominate the walltime and<br></blockquote></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">reduce our<br></blockquote></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">scalability quite rapidly. For instance the %walltime for MPI_WIN_CREATE<br></blockquote></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">over various process sizes increases as follows:<br></blockquote></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><br></blockquote></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">256 cores - 4.0%<br></blockquote></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">512 cores - 9.8%<br></blockquote></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">1024 cores - 24.3%<br></blockquote></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">The current implementation of MPI_Win_create uses an Allgather which is<br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">synchronizing and relatively expensive.<br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><br></blockquote></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><br></blockquote></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">I was wondering if anyone in the MPICH2 community had any advice on how<br></blockquote></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">one can improve the performance of MPI_WIN_CREATE? Or maybe someone<br></blockquote></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">has a<br></blockquote></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">better strategy for communicating the data that bypasses the (poorly<br></blockquote></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">scaling?) MPI_WIN_CREATE routine.<br></blockquote></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><br></blockquote></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">Thanks in advance for any help you can provide.<br></blockquote></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><br></blockquote></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">Regards,<br></blockquote></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><br></blockquote></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">Tim.<br></blockquote></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">_______________________________________________<br></blockquote></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">mpich-discuss mailing list <a href="mailto:mpich-discuss@mcs.anl.gov">mpich-discuss@mcs.anl.gov</a><br></blockquote></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><<a href="mailto:mpich-discuss@mcs.anl.gov">mailto:mpich-discuss@mcs.anl.gov</a>><br></blockquote></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">To manage subscription options or unsubscribe:<br></blockquote></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><a href="https://lists.mcs.anl.gov/mailman/listinfo/mpich-discuss">https://lists.mcs.anl.gov/mailman/listinfo/mpich-discuss</a><br></blockquote></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">_______________________________________________<br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">mpich-discuss mailing list <a href="mailto:mpich-discuss@mcs.anl.gov">mpich-discuss@mcs.anl.gov</a><br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><<a href="mailto:mpich-discuss@mcs.anl.gov">mailto:mpich-discuss@mcs.anl.gov</a>><br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">To manage subscription options or unsubscribe:<br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><a href="https://lists.mcs.anl.gov/mailman/listinfo/mpich-discuss">https://lists.mcs.anl.gov/mailman/listinfo/mpich-discuss</a><br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">--<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">Jeff Hammond<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">Argonne Leadership Computing Facility<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">University of Chicago Computation Institute<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><a href="mailto:jhammond@alcf.anl.gov">jhammond@alcf.anl.gov</a> <<a href="mailto:jhammond@alcf.anl.gov">mailto:jhammond@alcf.anl.gov</a>> / (630) 252-5381<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><a href="http://www.linkedin.com/in/jeffhammond">http://www.linkedin.com/in/jeffhammond</a><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><a href="https://wiki.alcf.anl.gov/parts/index.php/User:Jhammond">https://wiki.alcf.anl.gov/parts/index.php/User:Jhammond</a><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">_______________________________________________<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">mpich-discuss mailing list <a href="mailto:mpich-discuss@mcs.anl.gov">mpich-discuss@mcs.anl.gov</a><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">To manage subscription options or unsubscribe:<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><a href="https://lists.mcs.anl.gov/mailman/listinfo/mpich-discuss">https://lists.mcs.anl.gov/mailman/listinfo/mpich-discuss</a><br></blockquote></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">*Tim Stitt*PhD(User Support Manager).<br></blockquote><blockquote type="cite">Center for Research Computing | University of Notre Dame |<br></blockquote><blockquote type="cite">P.O. Box 539, Notre Dame, IN 46556 | Phone: 574-631-5287 | Email:<br></blockquote><blockquote type="cite"><a href="mailto:tstitt@nd.edu">tstitt@nd.edu</a> <<a href="mailto:tstitt@nd.edu">mailto:tstitt@nd.edu</a>><br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">_______________________________________________<br></blockquote><blockquote type="cite">mpich-discuss mailing list <a href="mailto:mpich-discuss@mcs.anl.gov">mpich-discuss@mcs.anl.gov</a><br></blockquote><blockquote type="cite">To manage subscription options or unsubscribe:<br></blockquote><blockquote type="cite"><a href="https://lists.mcs.anl.gov/mailman/listinfo/mpich-discuss">https://lists.mcs.anl.gov/mailman/listinfo/mpich-discuss</a><br></blockquote>_______________________________________________<br>mpich-discuss mailing list <a href="mailto:mpich-discuss@mcs.anl.gov">mpich-discuss@mcs.anl.gov</a><br>To manage subscription options or unsubscribe:<br><a href="https://lists.mcs.anl.gov/mailman/listinfo/mpich-discuss">https://lists.mcs.anl.gov/mailman/listinfo/mpich-discuss</a><br></div></blockquote></div><br><div>
<!--Text inside such 'comments' is not included in your siggy. You may delete these text comments once you are done--><!-- This bit defines your entire box. Change the 'height' value to reduce the size of the box, the line-height value to reduce the spacing between text lines, and so on. I suggest you change the min-width style to better suit the width of your siggy-->
<title></title>
<div id="sig" style="border-top: 1px dotted rgb(153, 153, 153); border-bottom: 1px dotted rgb(153, 153, 153); margin: 6px 0pt; padding: 8px; min-height: 50px; line-height: 17px; font-family: 'Lucida Grande',Verdana,Arial,Sans-Serif; font-size: 11px; color: rgb(96, 111, 120); min-width: 250px;"><!--This is the image. Upload an image to your own server or imageshack.us and replace the url in this tag-->
<!--<img src="http://www.rostauguardian.webuda.com/photo.jpg" alt="me"
style="padding: 2px 6px 0pt 0pt; float: left; width: 46px; height: 45px;">--><!--end--><!--replace details outside the <> tag brackets. Your name, company, etc. Also change the URLs where needed.
You can also replace the text colour #606f78 to anything you choose--><strong style="color: rgb(255, 102, 0); font-weight: bold;">Tim
Stitt</strong><span style="font-weight: bold;"> </span><span style="color: rgb(255, 102, 0); font-weight: bold;">PhD</span><span style="font-weight: bold;"></span>
<span style="color: rgb(0, 153, 0);">(User Support Manager)</span>.<br>
Center for Research Computing | University of Notre Dame
| <br>
<!--the <br /> tag (above) signifies a line break. Add that tag anywhere you want the line to break into another one. Remove that to make the bottom line flow to the right of the one above it-->
P.O.
Box 539,
Notre Dame, IN 46556 | Phone:
574-631-5287
| Email: <span style="color: rgb(51, 51, 255);"><a href="mailto:tstitt@nd.edu">tstitt@nd.edu</a> </span></div>
</div>
<br></div></body></html>