<div>Sorry, I didn't realize what I said before probably didn't make sense, because I had something hardcoded</div><div>I belive jobspernode = 1 will have just one openmp job. then OMP_NUM_THREADS controls the /max/ number of threads per openmp job, but you can specify a lower number in the code if you want</div>
<div><br></div><div>I think I'm using native gcc so I believe the answer to your second question is yes<br><br><div class="gmail_quote">On Mon, Oct 17, 2011 at 12:59 PM, Michael Wilde <span dir="ltr"><<a href="mailto:wilde@mcs.anl.gov">wilde@mcs.anl.gov</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">By the way, reading through all the examples from Glen, Justin, Ketan and David, I am still curious how to get coasters to run exactly one OpenMP app per node, while ensuring that the app has access to all the node's cores.<br>
<br>
Also, on Beagle, Glen, do you know if you can compile and execute with the native gcc module, with all PrgEnv's unloaded? This *seems* to work on the sandbox node; I am about to try on a compute node.<br>
<br>
Lastly: the descriptions of Coaster parameters in the Beagle doc need to be corrected, they are not quite correct. They should be compared to the descriptions in the User Guide, which while not very easy to understand, are to my knowledge correct. It would be good to revise the User Guide to be more clear (which I think basically involves more explanation and examples) and then have the Beagle text reference and/or replicate the User Guide info as appropriate.<br>
<div class="im"><br>
- Mike<br>
<br>
<br>
<br>
----- Original Message -----<br>
</div><div><div></div><div class="h5">> From: "Michael Wilde" <<a href="mailto:wilde@mcs.anl.gov">wilde@mcs.anl.gov</a>><br>
> To: "Glen Hocky" <<a href="mailto:hockyg@uchicago.edu">hockyg@uchicago.edu</a>><br>
> Cc: "Swift Devel" <<a href="mailto:swift-devel@ci.uchicago.edu">swift-devel@ci.uchicago.edu</a>><br>
> Sent: Monday, October 17, 2011 11:52:54 AM<br>
> Subject: Re: [Swift-devel] Where is latest doc on running Swift on Beagle? Covers OpenMP apps?<br>
> I can help write a test case. Its just a for() loop with a #pragma in<br>
> front - very simple. If each parallel loop iteration could do<br>
> system("sleep N") we could readily observe that the test is working<br>
> and spawning OMP_NUM_THREADS threads and procs.<br>
><br>
> - Mike<br>
><br>
> ----- Original Message -----<br>
> > From: "Glen Hocky" <<a href="mailto:hockyg@uchicago.edu">hockyg@uchicago.edu</a>><br>
> > To: "Justin M Wozniak" <<a href="mailto:wozniak@mcs.anl.gov">wozniak@mcs.anl.gov</a>><br>
> > Cc: "Michael Wilde" <<a href="mailto:wilde@mcs.anl.gov">wilde@mcs.anl.gov</a>>, "David Kelly"<br>
> > <<a href="mailto:davidk@ci.uchicago.edu">davidk@ci.uchicago.edu</a>>, "ketan" <<a href="mailto:ketancmaheshwari@gmail.com">ketancmaheshwari@gmail.com</a>>,<br>
> > "Swift Devel" <<a href="mailto:swift-devel@ci.uchicago.edu">swift-devel@ci.uchicago.edu</a>><br>
> > Sent: Monday, October 17, 2011 11:30:17 AM<br>
> > Subject: Re: [Swift-devel] Where is latest doc on running Swift on<br>
> > Beagle? Covers OpenMP apps?<br>
> > Justin, I'm not sure my program counts as sufficiently simple for<br>
> > this<br>
> > purpose. I'd be happy to let you include it and get an example set<br>
> > up<br>
> > though if you want to use it anyway. The open mp part, which I<br>
> > haven't<br>
> > been using recently, may need a bit of debugging as well<br>
> ><br>
> ><br>
> > Glen<br>
> ><br>
> ><br>
> ><br>
> > On Mon, Oct 17, 2011 at 12:23 PM, Justin M Wozniak <<br>
> > <a href="mailto:wozniak@mcs.anl.gov">wozniak@mcs.anl.gov</a> > wrote:<br>
> ><br>
> ><br>
> ><br>
> > Glen, do you have an extremely simple but relevant OpenMP program<br>
> > that<br>
> > we could stick in the test suite?<br>
> ><br>
> ><br>
> ><br>
> ><br>
> > On Sun, 16 Oct 2011, Glen Hocky wrote:<br>
> ><br>
> ><br>
> ><br>
> > It's in my run script that creates the actual sites file that I run<br>
> > with.<br>
> > I'm not sure what you would do if you wanted more than 24 cores, so<br>
> > depth<br>
> > stays fixed at 24 (that's an aprun parameters). Then<br>
> ><br>
> > WORKERSPERNODE=$((24/$PPN))<br>
> ><br>
> > Where PPN is how many cores you want per OPENMP app and then workers<br>
> > per<br>
> > node says how many OPENMP apps you want to run. So obvious example<br>
> > would be<br>
> > you want 3 8 core OPENMP jobs, PPN = 8, WORKERSPERNODE=3<br>
> ><br>
> ><br>
> > On Sun, Oct 16, 2011 at 12:27 PM, Michael Wilde < <a href="mailto:wilde@mcs.anl.gov">wilde@mcs.anl.gov</a><br>
> > ><br>
> > wrote:<br>
> ><br>
> ><br>
> ><br>
> > Thanks, Glen!<br>
> ><br>
> > Justin, can you check the sites file below? I dont understand the<br>
> > interaction between the parameters OMP_NUM_THREDS, jobsPerNode, and<br>
> > depth.<br>
> > WHere is the best documentation on that?<br>
> ><br>
> > Thanks,<br>
> ><br>
> > - Mike<br>
> ><br>
> ><br>
> > ----- Original Message -----<br>
> ><br>
> ><br>
> > From: "Glen Hocky" < <a href="mailto:hockyg@uchicago.edu">hockyg@uchicago.edu</a> ><br>
> > To: "Michael Wilde" < <a href="mailto:wilde@mcs.anl.gov">wilde@mcs.anl.gov</a> ><br>
> > Cc: "David Kelly" < <a href="mailto:davidk@ci.uchicago.edu">davidk@ci.uchicago.edu</a> >, "ketan" <<br>
> > <a href="mailto:ketancmaheshwari@gmail.com">ketancmaheshwari@gmail.com</a> ><br>
> ><br>
> ><br>
> > Sent: Sunday, October 16, 2011 11:18:33 AM<br>
> > Subject: Re: [Swift-devel] Where is latest doc on running Swift on<br>
> > Beagle? Covers OpenMP apps?<br>
> ><br>
> ><br>
> > Yes, I'm running and yes I did test openmp a while back. Sites file<br>
> > follows. I'm using trunk from a few months ago<br>
> ><br>
> > "Swift svn swift-r4813 (swift modified locally) cog-r3175"<br>
> ><br>
> ><br>
> ><br>
> > <pool handle="pbs-beagle-coasters"><br>
> > <execution provider="coaster" jobmanager="local:pbs" url="none"/><br>
> > <filesystem provider="local"/><br>
> > <profile namespace="globus"<br>
> > key="providerAttributes">pbs. aprun;pbs.mpp;depth=24</ profile><br>
> > <profile key="jobsPerNode" namespace="globus">24</ profile><br>
> ><br>
> ><br>
> > <profile namespace="env" key="OMP_NUM_THREADS">$PPN</ profile><br>
> > <profile namespace="globus" key="maxwalltime">$TIME</ profile><br>
> > <profile namespace="globus" key="maxTime">$MAXTIME</ profile><br>
> > <profile namespace="globus" key="slots">$nodes</profile><br>
> > <profile namespace="globus" key="nodeGranularity">1</ profile><br>
> > <profile namespace="globus" key="maxNodes">1</profile><br>
> > <profile namespace="globus" key="lowOverAllocation">100</ profile><br>
> > <profile namespace="globus" key="highOverAllocation">100</ profile><br>
> > <profile namespace="karajan" key="jobThrottle">200.00</ profile><br>
> > <profile namespace="karajan" key="initialScore">10000</ profile><br>
> ><br>
> ><br>
> > <workdirectory >$swiftrundir/swiftwork</ workdirectory><br>
> > </pool><br>
> ><br>
> ><br>
> ><br>
> > On Sun, Oct 16, 2011 at 12:15 PM, Michael Wilde < <a href="mailto:wilde@mcs.anl.gov">wilde@mcs.anl.gov</a><br>
> > ><br>
> > wrote:<br>
> ><br>
> ><br>
> > David, Ketan,<br>
> ><br>
> > I need to run some things on Beagle, asap.<br>
> ><br>
> > Ketan, where is the latest and best documentation for this? I see<br>
> > your<br>
> > edits below to the 0.93 Site Guide. But I dont see that online where<br>
> > I<br>
> > would expect it:<br>
> ><br>
> ><br>
> > <a href="http://www.ci.uchicago.edu/" target="_blank">http://www.ci.uchicago.edu/</a> swift/wwwdev/guides/release-0.<br>
> > 93/siteguide/siteguide.html#_ beagle<br>
> ><br>
> ><br>
> ><br>
> > David, is it just that this document is not being correctly pushed<br>
> > to<br>
> > the wwwdev site on a nightly basis?<br>
> ><br>
> > Ketan, is the latest info on running Swift on Beagle now all in the<br>
> > siteguide? Is the info you were putting in the cookbook (I see many<br>
> > commits there) now all consolidated into the Site Guide? And is<br>
> > there<br>
> > a difference in sites.xml settings between 0.93 and trunk? Lastly,<br>
> > which release works best?<br>
> ><br>
> > Second question: I need to run a script that executes many 24-core<br>
> > OpenMP apps. Is the necessary support for this in 0.93? What if any<br>
> > declarations do I need other than to say jobsPerNode=1? Glen, are<br>
> > you<br>
> > running OpenMP on Beagle and if so what release and sites file are<br>
> > you<br>
> > using?<br>
> ><br>
> > Im assuming Justin's latest changes to sites.xml are in trunk but<br>
> > not<br>
> > 0.93? If that is correct, is there a corresponding site site for<br>
> > Beagle for trunk?<br>
> ><br>
> > Thanks,<br>
> ><br>
> > - Mike<br>
> ><br>
> ><br>
> > ----- Forwarded Message -----<br>
> > From: <a href="mailto:ketan@ci.uchicago.edu">ketan@ci.uchicago.edu</a><br>
> > To: <a href="mailto:swift-commit@ci.uchicago.edu">swift-commit@ci.uchicago.edu</a><br>
> > Sent: Sunday, September 18, 2011 10:14:10 PM<br>
> > Subject: [Swift-commit] r5126 - branches/release-0.93/docs/<br>
> > siteguide<br>
> ><br>
> > Author: ketan<br>
> > Date: 2011-09-18 22:14:10 -0500 (Sun, 18 Sep 2011)<br>
> > New Revision: 5126<br>
> ><br>
> > Modified:<br>
> > branches/release-0.93/docs/ siteguide/beagle<br>
> > Log:<br>
> > added content to beagle siteguide<br>
> ><br>
> > Modified: branches/release-0.93/docs/ siteguide/beagle<br>
> > ============================== ==============================<br>
> > =======<br>
> > --- branches/release-0.93/docs/ siteguide/beagle 2011-09-19 02:41:02<br>
> > UTC (rev 5125)<br>
> > +++ branches/release-0.93/docs/ siteguide/beagle 2011-09-19 03:14:10<br>
> > UTC (rev 5126)<br>
> > @@ -52,9 +52,38 @@<br>
> > A key factor in scaling up Swift runs on Beagle is to setup the<br>
> > sites.xml parameters.<br>
> > The following sites.xml parameters must be set to scale that is<br>
> > intended for a large run:<br>
> ><br>
> > - * walltime: The expected walltime for completion of your run. This<br>
> > parameter is accepted in seconds.<br>
> > - * slots: Number of qsub jobs needs to be submitted by swift. This<br>
> > number will determine how many qsubs swift will submit for your run.<br>
> > Typical values range between 40 and 80 for large runs.<br>
> > - * nodegranularity: Determines the number of nodes per job. Total<br>
> > nodes will thus be slots times nodegranularity. This may vary for<br>
> > advanced configurations though.<br>
> > - * maxnodes: Determines the maximum number of nodes a job must pack<br>
> > into its qsub. This parameter determines the largest single job that<br>
> > your run will submit.<br>
> > + * *maxTime* : The expected walltime for completion of your run.<br>
> > This<br>
> > parameter is accepted in seconds.<br>
> > + * *slots* : Number of qsub jobs needs to be submitted by swift.<br>
> > This<br>
> > number will determine how many qsubs swift will submit for your run.<br>
> > Typical values range between 40 and 80 for large runs.<br>
> > + * *nodeGranularity* : Determines the number of nodes per job.<br>
> > Total<br>
> > nodes will thus be slots times nodegranularity. This may vary for<br>
> > advanced configurations though.<br>
> > + * *maxNodes* : Determines the maximum number of nodes a job must<br>
> > pack into its qsub. This parameter determines the largest single job<br>
> > that your run will submit.<br>
> > + * *jobThrottle* : A factor that determines the number of tasks<br>
> > dispatched simultaneously. The intended number of simultaneous tasks<br>
> > must match the number of cores targeted. The number of tasks is<br>
> > calculated from the jobThrottle factor is as follows:<br>
> ><br>
> > +----<br>
> > +Number of Tasks = (JobThrottle x 100) + 1<br>
> > +----<br>
> ><br>
> > +Following is an example sites.xml for a 50 slots run with each slot<br>
> > occupying 4 nodes (thus, a 200 node run):<br>
> > +<br>
> > +-----<br>
> > +<config><br>
> > + <pool handle="pbs"><br>
> > + <execution provider="coaster" jobmanager="local:pbs"/><br>
> > + <profile namespace="globus" key="project">CI-CCR000013</ profile><br>
> > +<br>
> > + <profile namespace="globus" key="ppn">24:cray:pack</ profile><br>
> > +<br>
> > + <profile namespace="globus" key="jobsPerNode">24</profile><br>
> > + <profile namespace="globus" key="maxTime">50000</profile><br>
> > + <profile namespace="globus" key="slots">50</profile><br>
> > + <profile namespace="globus" key="nodeGranularity">4</ profile><br>
> > + <profile namespace="globus" key="maxNodes">4</profile><br>
> > +<br>
> > + <profile namespace="karajan" key="jobThrottle">48.00</ profile><br>
> > + <profile namespace="karajan" key="initialScore">10000</ profile><br>
> > +<br>
> > + <filesystem provider="local"/><br>
> > + <workdirectory >/lustre/beagle/ketan/swift.<br>
> > workdir</workdirectory><br>
> > + </pool><br>
> > +</config><br>
> > +-----<br>
> > +<br>
> ><br>
> > ______________________________ _________________<br>
> > Swift-commit mailing list<br>
> > <a href="mailto:Swift-commit@ci.uchicago.edu">Swift-commit@ci.uchicago.edu</a><br>
> > <a href="https://lists.ci.uchicago.edu/" target="_blank">https://lists.ci.uchicago.edu/</a> cgi-bin/mailman/listinfo/<br>
> > swift-commit<br>
> ><br>
> > --<br>
> > Michael Wilde<br>
> > Computation Institute, University of Chicago<br>
> > Mathematics and Computer Science Division<br>
> > Argonne National Laboratory<br>
> ><br>
> > ______________________________ _________________<br>
> > Swift-devel mailing list<br>
> > <a href="mailto:Swift-devel@ci.uchicago.edu">Swift-devel@ci.uchicago.edu</a><br>
> > <a href="https://lists.ci.uchicago.edu/" target="_blank">https://lists.ci.uchicago.edu/</a> cgi-bin/mailman/listinfo/ swift-devel<br>
> ><br>
> > --<br>
> > Michael Wilde<br>
> > Computation Institute, University of Chicago<br>
> > Mathematics and Computer Science Division<br>
> > Argonne National Laboratory<br>
> ><br>
> ><br>
> ><br>
> ><br>
> > --<br>
> > Justin M Wozniak<br>
><br>
> --<br>
> Michael Wilde<br>
> Computation Institute, University of Chicago<br>
> Mathematics and Computer Science Division<br>
> Argonne National Laboratory<br>
><br>
</div></div>> _______________________________________________<br>
<div class="im">> Swift-devel mailing list<br>
> <a href="mailto:Swift-devel@ci.uchicago.edu">Swift-devel@ci.uchicago.edu</a><br>
> <a href="https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel" target="_blank">https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel</a><br>
<br>
</div><div><div></div><div class="h5">--<br>
Michael Wilde<br>
Computation Institute, University of Chicago<br>
Mathematics and Computer Science Division<br>
Argonne National Laboratory<br>
<br>
</div></div></blockquote></div><br></div>