Sorry, actually I don't understand the N=9 example on rereading. But here is an example that you might want to combine with<div><br></div><div><a href="https://computing.llnl.gov/tutorials/openMP/samples/C/omp_hello.c">https://computing.llnl.gov/tutorials/openMP/samples/C/omp_hello.c</a></div>
<div><br></div><div>With output something like</div><div><span class="Apple-style-span" style="font-family: serif, times; background-color: rgb(255, 255, 255); font-size: medium; "><table border="1" cellpadding="5" cellspacing="0" width="90%">
<tbody><tr><td><pre>Hello World from thread = 0
Number of threads = 4
Hello World from thread = 3
Hello World from thread = 1
Hello World from thread = 2</pre></td></tr></tbody></table></span><br><div class="gmail_quote">On Mon, Oct 17, 2011 at 7:00 PM, Michael Wilde <span dir="ltr"><<a href="mailto:wilde@mcs.anl.gov">wilde@mcs.anl.gov</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">Im a total newbie to OpenMP (so this may be a flawed example), but this seems to work:<br>
<br>
#include <stdio.h><br>
<br>
void main(int argc, char **argv)<br>
{<br>
int i;<br>
<br>
#pragma omp parallel for private (i)<br>
for(i=0; i<10; i++){<br>
printf("i=%d sleeps\n",i);<br>
system("sleep 2");<br>
printf("i=%d wakes\n", i);<br>
}<br>
}<br>
<br>
make with:<br>
<br>
openmpapp: openmpapp.c<br>
gcc -fopenmp -o openmpapp openmpapp.c<br>
<br>
And try the same code with the #pragma commented out.<br>
<br>
Under OpenMP you can see how the threads behave:<br>
<br>
sandbox$ OMP_NUM_THREADS=2 ./openmpapp<br>
i=0 sleeps<br>
i=5 sleeps<br>
i=5 wakes<br>
i=6 sleeps<br>
i=0 wakes<br>
i=1 sleeps<br>
i=6 wakes<br>
i=7 sleeps<br>
i=1 wakes<br>
i=2 sleeps<br>
i=7 wakes<br>
i=8 sleeps<br>
i=2 wakes<br>
i=3 sleeps<br>
i=8 wakes<br>
i=9 sleeps<br>
i=3 wakes<br>
i=4 sleeps<br>
i=9 wakes<br>
i=4 wakes<br>
sandbox$ OMP_NUM_THREADS=10 ./openmpapp<br>
i=2 sleeps<br>
i=9 sleeps<br>
i=0 sleeps<br>
i=7 sleeps<br>
i=8 sleeps<br>
i=5 sleeps<br>
i=6 sleeps<br>
i=4 sleeps<br>
i=3 sleeps<br>
i=1 sleeps<br>
i=9 wakes<br>
i=5 wakes<br>
i=8 wakes<br>
i=4 wakes<br>
i=6 wakes<br>
i=3 wakes<br>
i=1 wakes<br>
i=0 wakes<br>
i=2 wakes<br>
i=7 wakes<br>
sandbox$<br>
<br>
One thing that has me stumped, though, is that asking for 9 threads shows a behavior as if it had 5 threads:<br>
<br>
sandbox$ OMP_NUM_THREADS=9 ./openmpapp<br>
i=4 sleeps<br>
i=0 sleeps<br>
i=8 sleeps<br>
i=6 sleeps<br>
i=2 sleeps<br>
i=8 wakes<br>
i=9 sleeps<br>
i=4 wakes<br>
i=5 sleeps<br>
i=0 wakes<br>
i=1 sleeps<br>
i=6 wakes<br>
i=7 sleeps<br>
i=2 wakes<br>
i=3 sleeps<br>
i=1 wakes<br>
i=7 wakes<br>
i=9 wakes<br>
i=3 wakes<br>
i=5 wakes<br>
sandbox$<br>
<br>
I also would not assume that printf() and system() are thread-safe, but at least this is a simple example to start out with for testing if we're getting the right number of cores and threads active under Swift, Coasters, and Cray ALPS/aprun.<br>
<br>
- Mike<br>
<br>
<br>
----- Original Message -----<br>
> From: "Michael Wilde" <<a href="mailto:wilde@mcs.anl.gov">wilde@mcs.anl.gov</a>><br>
> To: "Glen Hocky" <<a href="mailto:hockyg@uchicago.edu">hockyg@uchicago.edu</a>><br>
> Cc: "David Kelly" <<a href="mailto:davidk@ci.uchicago.edu">davidk@ci.uchicago.edu</a>>, "ketan" <<a href="mailto:ketancmaheshwari@gmail.com">ketancmaheshwari@gmail.com</a>>, "Swift Devel"<br>
> <<a href="mailto:swift-devel@ci.uchicago.edu">swift-devel@ci.uchicago.edu</a>>, "Justin M Wozniak" <<a href="mailto:wozniak@mcs.anl.gov">wozniak@mcs.anl.gov</a>><br>
> Sent: Monday, October 17, 2011 11:52:54 AM<br>
> Subject: Re: [Swift-devel] Where is latest doc on running Swift on Beagle? Covers OpenMP apps?<br>
> I can help write a test case. Its just a for() loop with a #pragma in<br>
> front - very simple. If each parallel loop iteration could do<br>
> system("sleep N") we could readily observe that the test is working<br>
> and spawning OMP_NUM_THREADS threads and procs.<br>
><br>
> - Mike<br>
><br>
> ----- Original Message -----<br>
> > From: "Glen Hocky" <<a href="mailto:hockyg@uchicago.edu">hockyg@uchicago.edu</a>><br>
> > To: "Justin M Wozniak" <<a href="mailto:wozniak@mcs.anl.gov">wozniak@mcs.anl.gov</a>><br>
> > Cc: "Michael Wilde" <<a href="mailto:wilde@mcs.anl.gov">wilde@mcs.anl.gov</a>>, "David Kelly"<br>
> > <<a href="mailto:davidk@ci.uchicago.edu">davidk@ci.uchicago.edu</a>>, "ketan" <<a href="mailto:ketancmaheshwari@gmail.com">ketancmaheshwari@gmail.com</a>>,<br>
> > "Swift Devel" <<a href="mailto:swift-devel@ci.uchicago.edu">swift-devel@ci.uchicago.edu</a>><br>
> > Sent: Monday, October 17, 2011 11:30:17 AM<br>
> > Subject: Re: [Swift-devel] Where is latest doc on running Swift on<br>
> > Beagle? Covers OpenMP apps?<br>
> > Justin, I'm not sure my program counts as sufficiently simple for<br>
> > this<br>
> > purpose. I'd be happy to let you include it and get an example set<br>
> > up<br>
> > though if you want to use it anyway. The open mp part, which I<br>
> > haven't<br>
> > been using recently, may need a bit of debugging as well<br>
> ><br>
> ><br>
> > Glen<br>
> ><br>
> ><br>
> ><br>
> > On Mon, Oct 17, 2011 at 12:23 PM, Justin M Wozniak <<br>
> > <a href="mailto:wozniak@mcs.anl.gov">wozniak@mcs.anl.gov</a> > wrote:<br>
> ><br>
> ><br>
> ><br>
> > Glen, do you have an extremely simple but relevant OpenMP program<br>
> > that<br>
> > we could stick in the test suite?<br>
> ><br>
> ><br>
> ><br>
> ><br>
> > On Sun, 16 Oct 2011, Glen Hocky wrote:<br>
> ><br>
> ><br>
> ><br>
> > It's in my run script that creates the actual sites file that I run<br>
> > with.<br>
> > I'm not sure what you would do if you wanted more than 24 cores, so<br>
> > depth<br>
> > stays fixed at 24 (that's an aprun parameters). Then<br>
> ><br>
> > WORKERSPERNODE=$((24/$PPN))<br>
> ><br>
> > Where PPN is how many cores you want per OPENMP app and then workers<br>
> > per<br>
> > node says how many OPENMP apps you want to run. So obvious example<br>
> > would be<br>
> > you want 3 8 core OPENMP jobs, PPN = 8, WORKERSPERNODE=3<br>
> ><br>
> ><br>
> > On Sun, Oct 16, 2011 at 12:27 PM, Michael Wilde < <a href="mailto:wilde@mcs.anl.gov">wilde@mcs.anl.gov</a><br>
> > ><br>
> > wrote:<br>
> ><br>
> ><br>
> ><br>
> > Thanks, Glen!<br>
> ><br>
> > Justin, can you check the sites file below? I dont understand the<br>
> > interaction between the parameters OMP_NUM_THREDS, jobsPerNode, and<br>
> > depth.<br>
> > WHere is the best documentation on that?<br>
> ><br>
> > Thanks,<br>
> ><br>
> > - Mike<br>
> ><br>
> ><br>
> > ----- Original Message -----<br>
> ><br>
> ><br>
> > From: "Glen Hocky" < <a href="mailto:hockyg@uchicago.edu">hockyg@uchicago.edu</a> ><br>
> > To: "Michael Wilde" < <a href="mailto:wilde@mcs.anl.gov">wilde@mcs.anl.gov</a> ><br>
> > Cc: "David Kelly" < <a href="mailto:davidk@ci.uchicago.edu">davidk@ci.uchicago.edu</a> >, "ketan" <<br>
> > <a href="mailto:ketancmaheshwari@gmail.com">ketancmaheshwari@gmail.com</a> ><br>
> ><br>
> ><br>
> > Sent: Sunday, October 16, 2011 11:18:33 AM<br>
> > Subject: Re: [Swift-devel] Where is latest doc on running Swift on<br>
> > Beagle? Covers OpenMP apps?<br>
> ><br>
> ><br>
> > Yes, I'm running and yes I did test openmp a while back. Sites file<br>
> > follows. I'm using trunk from a few months ago<br>
> ><br>
> > "Swift svn swift-r4813 (swift modified locally) cog-r3175"<br>
> ><br>
> ><br>
> ><br>
> > <pool handle="pbs-beagle-coasters"><br>
> > <execution provider="coaster" jobmanager="local:pbs" url="none"/><br>
> > <filesystem provider="local"/><br>
> > <profile namespace="globus"<br>
> > key="providerAttributes">pbs. aprun;pbs.mpp;depth=24</ profile><br>
> > <profile key="jobsPerNode" namespace="globus">24</ profile><br>
> ><br>
> ><br>
> > <profile namespace="env" key="OMP_NUM_THREADS">$PPN</ profile><br>
> > <profile namespace="globus" key="maxwalltime">$TIME</ profile><br>
> > <profile namespace="globus" key="maxTime">$MAXTIME</ profile><br>
> > <profile namespace="globus" key="slots">$nodes</profile><br>
> > <profile namespace="globus" key="nodeGranularity">1</ profile><br>
> > <profile namespace="globus" key="maxNodes">1</profile><br>
> > <profile namespace="globus" key="lowOverAllocation">100</ profile><br>
> > <profile namespace="globus" key="highOverAllocation">100</ profile><br>
> > <profile namespace="karajan" key="jobThrottle">200.00</ profile><br>
> > <profile namespace="karajan" key="initialScore">10000</ profile><br>
> ><br>
> ><br>
> > <workdirectory >$swiftrundir/swiftwork</ workdirectory><br>
> > </pool><br>
> ><br>
> ><br>
> ><br>
> > On Sun, Oct 16, 2011 at 12:15 PM, Michael Wilde < <a href="mailto:wilde@mcs.anl.gov">wilde@mcs.anl.gov</a><br>
> > ><br>
> > wrote:<br>
> ><br>
> ><br>
> > David, Ketan,<br>
> ><br>
> > I need to run some things on Beagle, asap.<br>
> ><br>
> > Ketan, where is the latest and best documentation for this? I see<br>
> > your<br>
> > edits below to the 0.93 Site Guide. But I dont see that online where<br>
> > I<br>
> > would expect it:<br>
> ><br>
> ><br>
> > <a href="http://www.ci.uchicago.edu/" target="_blank">http://www.ci.uchicago.edu/</a> swift/wwwdev/guides/release-0.<br>
> > 93/siteguide/siteguide.html#_ beagle<br>
> ><br>
> ><br>
> ><br>
> > David, is it just that this document is not being correctly pushed<br>
> > to<br>
> > the wwwdev site on a nightly basis?<br>
> ><br>
> > Ketan, is the latest info on running Swift on Beagle now all in the<br>
> > siteguide? Is the info you were putting in the cookbook (I see many<br>
> > commits there) now all consolidated into the Site Guide? And is<br>
> > there<br>
> > a difference in sites.xml settings between 0.93 and trunk? Lastly,<br>
> > which release works best?<br>
> ><br>
> > Second question: I need to run a script that executes many 24-core<br>
> > OpenMP apps. Is the necessary support for this in 0.93? What if any<br>
> > declarations do I need other than to say jobsPerNode=1? Glen, are<br>
> > you<br>
> > running OpenMP on Beagle and if so what release and sites file are<br>
> > you<br>
> > using?<br>
> ><br>
> > Im assuming Justin's latest changes to sites.xml are in trunk but<br>
> > not<br>
> > 0.93? If that is correct, is there a corresponding site site for<br>
> > Beagle for trunk?<br>
> ><br>
> > Thanks,<br>
> ><br>
> > - Mike<br>
> ><br>
> ><br>
> > ----- Forwarded Message -----<br>
> > From: <a href="mailto:ketan@ci.uchicago.edu">ketan@ci.uchicago.edu</a><br>
> > To: <a href="mailto:swift-commit@ci.uchicago.edu">swift-commit@ci.uchicago.edu</a><br>
> > Sent: Sunday, September 18, 2011 10:14:10 PM<br>
> > Subject: [Swift-commit] r5126 - branches/release-0.93/docs/<br>
> > siteguide<br>
> ><br>
> > Author: ketan<br>
> > Date: 2011-09-18 22:14:10 -0500 (Sun, 18 Sep 2011)<br>
> > New Revision: 5126<br>
> ><br>
> > Modified:<br>
> > branches/release-0.93/docs/ siteguide/beagle<br>
> > Log:<br>
> > added content to beagle siteguide<br>
> ><br>
> > Modified: branches/release-0.93/docs/ siteguide/beagle<br>
> > ============================== ==============================<br>
> > =======<br>
> > --- branches/release-0.93/docs/ siteguide/beagle 2011-09-19 02:41:02<br>
> > UTC (rev 5125)<br>
> > +++ branches/release-0.93/docs/ siteguide/beagle 2011-09-19 03:14:10<br>
> > UTC (rev 5126)<br>
> > @@ -52,9 +52,38 @@<br>
> > A key factor in scaling up Swift runs on Beagle is to setup the<br>
> > sites.xml parameters.<br>
> > The following sites.xml parameters must be set to scale that is<br>
> > intended for a large run:<br>
> ><br>
> > - * walltime: The expected walltime for completion of your run. This<br>
> > parameter is accepted in seconds.<br>
> > - * slots: Number of qsub jobs needs to be submitted by swift. This<br>
> > number will determine how many qsubs swift will submit for your run.<br>
> > Typical values range between 40 and 80 for large runs.<br>
> > - * nodegranularity: Determines the number of nodes per job. Total<br>
> > nodes will thus be slots times nodegranularity. This may vary for<br>
> > advanced configurations though.<br>
> > - * maxnodes: Determines the maximum number of nodes a job must pack<br>
> > into its qsub. This parameter determines the largest single job that<br>
> > your run will submit.<br>
> > + * *maxTime* : The expected walltime for completion of your run.<br>
> > This<br>
> > parameter is accepted in seconds.<br>
> > + * *slots* : Number of qsub jobs needs to be submitted by swift.<br>
> > This<br>
> > number will determine how many qsubs swift will submit for your run.<br>
> > Typical values range between 40 and 80 for large runs.<br>
> > + * *nodeGranularity* : Determines the number of nodes per job.<br>
> > Total<br>
> > nodes will thus be slots times nodegranularity. This may vary for<br>
> > advanced configurations though.<br>
> > + * *maxNodes* : Determines the maximum number of nodes a job must<br>
> > pack into its qsub. This parameter determines the largest single job<br>
> > that your run will submit.<br>
> > + * *jobThrottle* : A factor that determines the number of tasks<br>
> > dispatched simultaneously. The intended number of simultaneous tasks<br>
> > must match the number of cores targeted. The number of tasks is<br>
> > calculated from the jobThrottle factor is as follows:<br>
> ><br>
> > +----<br>
> > +Number of Tasks = (JobThrottle x 100) + 1<br>
> > +----<br>
> ><br>
> > +Following is an example sites.xml for a 50 slots run with each slot<br>
> > occupying 4 nodes (thus, a 200 node run):<br>
> > +<br>
> > +-----<br>
> > +<config><br>
> > + <pool handle="pbs"><br>
> > + <execution provider="coaster" jobmanager="local:pbs"/><br>
> > + <profile namespace="globus" key="project">CI-CCR000013</ profile><br>
> > +<br>
> > + <profile namespace="globus" key="ppn">24:cray:pack</ profile><br>
> > +<br>
> > + <profile namespace="globus" key="jobsPerNode">24</profile><br>
> > + <profile namespace="globus" key="maxTime">50000</profile><br>
> > + <profile namespace="globus" key="slots">50</profile><br>
> > + <profile namespace="globus" key="nodeGranularity">4</ profile><br>
> > + <profile namespace="globus" key="maxNodes">4</profile><br>
> > +<br>
> > + <profile namespace="karajan" key="jobThrottle">48.00</ profile><br>
> > + <profile namespace="karajan" key="initialScore">10000</ profile><br>
> > +<br>
> > + <filesystem provider="local"/><br>
> > + <workdirectory >/lustre/beagle/ketan/swift.<br>
> > workdir</workdirectory><br>
> > + </pool><br>
> > +</config><br>
> > +-----<br>
> > +<br>
> ><br>
> > ______________________________ _________________<br>
> > Swift-commit mailing list<br>
> > <a href="mailto:Swift-commit@ci.uchicago.edu">Swift-commit@ci.uchicago.edu</a><br>
> > <a href="https://lists.ci.uchicago.edu/" target="_blank">https://lists.ci.uchicago.edu/</a> cgi-bin/mailman/listinfo/<br>
> > swift-commit<br>
> ><br>
> > --<br>
> > Michael Wilde<br>
> > Computation Institute, University of Chicago<br>
> > Mathematics and Computer Science Division<br>
> > Argonne National Laboratory<br>
> ><br>
> > ______________________________ _________________<br>
> > Swift-devel mailing list<br>
> > <a href="mailto:Swift-devel@ci.uchicago.edu">Swift-devel@ci.uchicago.edu</a><br>
> > <a href="https://lists.ci.uchicago.edu/" target="_blank">https://lists.ci.uchicago.edu/</a> cgi-bin/mailman/listinfo/ swift-devel<br>
> ><br>
> > --<br>
> > Michael Wilde<br>
> > Computation Institute, University of Chicago<br>
> > Mathematics and Computer Science Division<br>
> > Argonne National Laboratory<br>
> ><br>
> ><br>
> ><br>
> ><br>
> > --<br>
> > Justin M Wozniak<br>
><br>
> --<br>
> Michael Wilde<br>
> Computation Institute, University of Chicago<br>
> Mathematics and Computer Science Division<br>
> Argonne National Laboratory<br>
<font color="#888888"><br>
--<br>
Michael Wilde<br>
Computation Institute, University of Chicago<br>
Mathematics and Computer Science Division<br>
Argonne National Laboratory<br>
<br>
</font></blockquote></div><br></div>