[Swift-devel] OpenMP example for Swift testing

Mon Oct 17 18:07:16 CDT 2011

Sorry, actually I don't understand the N=9 example on rereading. But here is
an example that you might want to combine with

https://computing.llnl.gov/tutorials/openMP/samples/C/omp_hello.c

With output something like

Hello World from thread = 0
Number of threads = 4
Hello World from thread = 3
Hello World from thread = 1
Hello World from thread = 2

On Mon, Oct 17, 2011 at 7:00 PM, Michael Wilde <wilde at mcs.anl.gov> wrote:

> Im a total newbie to OpenMP (so this may be a flawed example), but this
> seems to work:
>
> #include <stdio.h>
>
> void main(int argc, char **argv)
> {
>  int i;
>
>  #pragma omp parallel for private (i)
>  for(i=0; i<10; i++){
>    printf("i=%d sleeps\n",i);
>    system("sleep 2");
>    printf("i=%d wakes\n", i);
>  }
> }
>
> make with:
>
> openmpapp:      openmpapp.c
>        gcc -fopenmp -o openmpapp openmpapp.c
>
> And try the same code with the #pragma commented out.
>
> Under OpenMP you can see how the threads behave:
>
> sandbox$ OMP_NUM_THREADS=2 ./openmpapp
> i=0 sleeps
> i=5 sleeps
> i=5 wakes
> i=6 sleeps
> i=0 wakes
> i=1 sleeps
> i=6 wakes
> i=7 sleeps
> i=1 wakes
> i=2 sleeps
> i=7 wakes
> i=8 sleeps
> i=2 wakes
> i=3 sleeps
> i=8 wakes
> i=9 sleeps
> i=3 wakes
> i=4 sleeps
> i=9 wakes
> i=4 wakes
> sandbox$ OMP_NUM_THREADS=10 ./openmpapp
> i=2 sleeps
> i=9 sleeps
> i=0 sleeps
> i=7 sleeps
> i=8 sleeps
> i=5 sleeps
> i=6 sleeps
> i=4 sleeps
> i=3 sleeps
> i=1 sleeps
> i=9 wakes
> i=5 wakes
> i=8 wakes
> i=4 wakes
> i=6 wakes
> i=3 wakes
> i=1 wakes
> i=0 wakes
> i=2 wakes
> i=7 wakes
> sandbox$
>
> One thing that has me stumped, though, is that asking for 9 threads shows a
> behavior as if it had 5 threads:
>
> sandbox$ OMP_NUM_THREADS=9 ./openmpapp
> i=4 sleeps
> i=0 sleeps
> i=8 sleeps
> i=6 sleeps
> i=2 sleeps
> i=8 wakes
> i=9 sleeps
> i=4 wakes
> i=5 sleeps
> i=0 wakes
> i=1 sleeps
> i=6 wakes
> i=7 sleeps
> i=2 wakes
> i=3 sleeps
> i=1 wakes
> i=7 wakes
> i=9 wakes
> i=3 wakes
> i=5 wakes
> sandbox$
>
> I also would not assume that printf() and system() are thread-safe, but at
> least this is a simple example to start out with for testing if we're
> getting the right number of cores and threads active under Swift, Coasters,
> and Cray ALPS/aprun.
>
> - Mike
>
>
> ----- Original Message -----
> > From: "Michael Wilde" <wilde at mcs.anl.gov>
> > To: "Glen Hocky" <hockyg at uchicago.edu>
> > Cc: "David Kelly" <davidk at ci.uchicago.edu>, "ketan" <
> ketancmaheshwari at gmail.com>, "Swift Devel"
> > <swift-devel at ci.uchicago.edu>, "Justin M Wozniak" <wozniak at mcs.anl.gov>
> > Sent: Monday, October 17, 2011 11:52:54 AM
> > Subject: Re: [Swift-devel] Where is latest doc on running Swift on
> Beagle? Covers OpenMP apps?
> > I can help write a test case. Its just a for() loop with a #pragma in
> > front - very simple. If each parallel loop iteration could do
> > system("sleep N") we could readily observe that the test is working
> > and spawning OMP_NUM_THREADS threads and procs.
> >
> > - Mike
> >
> > ----- Original Message -----
> > > From: "Glen Hocky" <hockyg at uchicago.edu>
> > > To: "Justin M Wozniak" <wozniak at mcs.anl.gov>
> > > Cc: "Michael Wilde" <wilde at mcs.anl.gov>, "David Kelly"
> > > <davidk at ci.uchicago.edu>, "ketan" <ketancmaheshwari at gmail.com>,
> > > "Swift Devel" <swift-devel at ci.uchicago.edu>
> > > Sent: Monday, October 17, 2011 11:30:17 AM
> > > Subject: Re: [Swift-devel] Where is latest doc on running Swift on
> > > Beagle? Covers OpenMP apps?
> > > Justin, I'm not sure my program counts as sufficiently simple for
> > > this
> > > purpose. I'd be happy to let you include it and get an example set
> > > up
> > > though if you want to use it anyway. The open mp part, which I
> > > haven't
> > > been using recently, may need a bit of debugging as well
> > >
> > >
> > > Glen
> > >
> > >
> > >
> > > On Mon, Oct 17, 2011 at 12:23 PM, Justin M Wozniak <
> > > wozniak at mcs.anl.gov > wrote:
> > >
> > >
> > >
> > > Glen, do you have an extremely simple but relevant OpenMP program
> > > that
> > > we could stick in the test suite?
> > >
> > >
> > >
> > >
> > > On Sun, 16 Oct 2011, Glen Hocky wrote:
> > >
> > >
> > >
> > > It's in my run script that creates the actual sites file that I run
> > > with.
> > > I'm not sure what you would do if you wanted more than 24 cores, so
> > > depth
> > > stays fixed at 24 (that's an aprun parameters). Then
> > >
> > > WORKERSPERNODE=$((24/$PPN))
> > >
> > > Where PPN is how many cores you want per OPENMP app and then workers
> > > per
> > > node says how many OPENMP apps you want to run. So obvious example
> > > would be
> > > you want 3 8 core OPENMP jobs, PPN = 8, WORKERSPERNODE=3
> > >
> > >
> > > On Sun, Oct 16, 2011 at 12:27 PM, Michael Wilde < wilde at mcs.anl.gov
> > > >
> > > wrote:
> > >
> > >
> > >
> > > Thanks, Glen!
> > >
> > > Justin, can you check the sites file below? I dont understand the
> > > interaction between the parameters OMP_NUM_THREDS, jobsPerNode, and
> > > depth.
> > > WHere is the best documentation on that?
> > >
> > > Thanks,
> > >
> > > - Mike
> > >
> > >
> > > ----- Original Message -----
> > >
> > >
> > > From: "Glen Hocky" < hockyg at uchicago.edu >
> > > To: "Michael Wilde" < wilde at mcs.anl.gov >
> > > Cc: "David Kelly" < davidk at ci.uchicago.edu >, "ketan" <
> > > ketancmaheshwari at gmail.com >
> > >
> > >
> > > Sent: Sunday, October 16, 2011 11:18:33 AM
> > > Subject: Re: [Swift-devel] Where is latest doc on running Swift on
> > > Beagle? Covers OpenMP apps?
> > >
> > >
> > > Yes, I'm running and yes I did test openmp a while back. Sites file
> > > follows. I'm using trunk from a few months ago
> > >
> > > "Swift svn swift-r4813 (swift modified locally) cog-r3175"
> > >
> > >
> > >
> > > <pool handle="pbs-beagle-coasters">
> > > <execution provider="coaster" jobmanager="local:pbs" url="none"/>
> > > <filesystem provider="local"/>
> > > <profile namespace="globus"
> > > key="providerAttributes">pbs. aprun;pbs.mpp;depth=24</ profile>
> > > <profile key="jobsPerNode" namespace="globus">24</ profile>
> > >
> > >
> > > <profile namespace="env" key="OMP_NUM_THREADS">$PPN</ profile>
> > > <profile namespace="globus" key="maxwalltime">$TIME</ profile>
> > > <profile namespace="globus" key="maxTime">$MAXTIME</ profile>
> > > <profile namespace="globus" key="slots">$nodes</profile>
> > > <profile namespace="globus" key="nodeGranularity">1</ profile>
> > > <profile namespace="globus" key="maxNodes">1</profile>
> > > <profile namespace="globus" key="lowOverAllocation">100</ profile>
> > > <profile namespace="globus" key="highOverAllocation">100</ profile>
> > > <profile namespace="karajan" key="jobThrottle">200.00</ profile>
> > > <profile namespace="karajan" key="initialScore">10000</ profile>
> > >
> > >
> > > <workdirectory >$swiftrundir/swiftwork</ workdirectory>
> > > </pool>
> > >
> > >
> > >
> > > On Sun, Oct 16, 2011 at 12:15 PM, Michael Wilde < wilde at mcs.anl.gov
> > > >
> > > wrote:
> > >
> > >
> > > David, Ketan,
> > >
> > > I need to run some things on Beagle, asap.
> > >
> > > Ketan, where is the latest and best documentation for this? I see
> > > your
> > > edits below to the 0.93 Site Guide. But I dont see that online where
> > > I
> > > would expect it:
> > >
> > >
> > > http://www.ci.uchicago.edu/ swift/wwwdev/guides/release-0.
> > > 93/siteguide/siteguide.html#_ beagle
> > >
> > >
> > >
> > > David, is it just that this document is not being correctly pushed
> > > to
> > > the wwwdev site on a nightly basis?
> > >
> > > Ketan, is the latest info on running Swift on Beagle now all in the
> > > siteguide? Is the info you were putting in the cookbook (I see many
> > > commits there) now all consolidated into the Site Guide? And is
> > > there
> > > a difference in sites.xml settings between 0.93 and trunk? Lastly,
> > > which release works best?
> > >
> > > Second question: I need to run a script that executes many 24-core
> > > OpenMP apps. Is the necessary support for this in 0.93? What if any
> > > declarations do I need other than to say jobsPerNode=1? Glen, are
> > > you
> > > running OpenMP on Beagle and if so what release and sites file are
> > > you
> > > using?
> > >
> > > Im assuming Justin's latest changes to sites.xml are in trunk but
> > > not
> > > 0.93? If that is correct, is there a corresponding site site for
> > > Beagle for trunk?
> > >
> > > Thanks,
> > >
> > > - Mike
> > >
> > >
> > > ----- Forwarded Message -----
> > > From: ketan at ci.uchicago.edu
> > > To: swift-commit at ci.uchicago.edu
> > > Sent: Sunday, September 18, 2011 10:14:10 PM
> > > Subject: [Swift-commit] r5126 - branches/release-0.93/docs/
> > > siteguide
> > >
> > > Author: ketan
> > > Date: 2011-09-18 22:14:10 -0500 (Sun, 18 Sep 2011)
> > > New Revision: 5126
> > >
> > > Modified:
> > > branches/release-0.93/docs/ siteguide/beagle
> > > Log:
> > > added content to beagle siteguide
> > >
> > > Modified: branches/release-0.93/docs/ siteguide/beagle
> > > ============================== ==============================
> > > =======
> > > --- branches/release-0.93/docs/ siteguide/beagle 2011-09-19 02:41:02
> > > UTC (rev 5125)
> > > +++ branches/release-0.93/docs/ siteguide/beagle 2011-09-19 03:14:10
> > > UTC (rev 5126)
> > > @@ -52,9 +52,38 @@
> > > A key factor in scaling up Swift runs on Beagle is to setup the
> > > sites.xml parameters.
> > > The following sites.xml parameters must be set to scale that is
> > > intended for a large run:
> > >
> > > - * walltime: The expected walltime for completion of your run. This
> > > parameter is accepted in seconds.
> > > - * slots: Number of qsub jobs needs to be submitted by swift. This
> > > number will determine how many qsubs swift will submit for your run.
> > > Typical values range between 40 and 80 for large runs.
> > > - * nodegranularity: Determines the number of nodes per job. Total
> > > nodes will thus be slots times nodegranularity. This may vary for
> > > advanced configurations though.
> > > - * maxnodes: Determines the maximum number of nodes a job must pack
> > > into its qsub. This parameter determines the largest single job that
> > > your run will submit.
> > > + * *maxTime* : The expected walltime for completion of your run.
> > > This
> > > parameter is accepted in seconds.
> > > + * *slots* : Number of qsub jobs needs to be submitted by swift.
> > > This
> > > number will determine how many qsubs swift will submit for your run.
> > > Typical values range between 40 and 80 for large runs.
> > > + * *nodeGranularity* : Determines the number of nodes per job.
> > > Total
> > > nodes will thus be slots times nodegranularity. This may vary for
> > > advanced configurations though.
> > > + * *maxNodes* : Determines the maximum number of nodes a job must
> > > pack into its qsub. This parameter determines the largest single job
> > > that your run will submit.
> > > + * *jobThrottle* : A factor that determines the number of tasks
> > > dispatched simultaneously. The intended number of simultaneous tasks
> > > must match the number of cores targeted. The number of tasks is
> > > calculated from the jobThrottle factor is as follows:
> > >
> > > +----
> > > +Number of Tasks = (JobThrottle x 100) + 1
> > > +----
> > >
> > > +Following is an example sites.xml for a 50 slots run with each slot
> > > occupying 4 nodes (thus, a 200 node run):
> > > +
> > > +-----
> > > +<config>
> > > + <pool handle="pbs">
> > > + <execution provider="coaster" jobmanager="local:pbs"/>
> > > + <profile namespace="globus" key="project">CI-CCR000013</ profile>
> > > +
> > > + <profile namespace="globus" key="ppn">24:cray:pack</ profile>
> > > +
> > > + <profile namespace="globus" key="jobsPerNode">24</profile>
> > > + <profile namespace="globus" key="maxTime">50000</profile>
> > > + <profile namespace="globus" key="slots">50</profile>
> > > + <profile namespace="globus" key="nodeGranularity">4</ profile>
> > > + <profile namespace="globus" key="maxNodes">4</profile>
> > > +
> > > + <profile namespace="karajan" key="jobThrottle">48.00</ profile>
> > > + <profile namespace="karajan" key="initialScore">10000</ profile>
> > > +
> > > + <filesystem provider="local"/>
> > > + <workdirectory >/lustre/beagle/ketan/swift.
> > > workdir</workdirectory>
> > > + </pool>
> > > +</config>
> > > +-----
> > > +
> > >
> > > ______________________________ _________________
> > > Swift-commit mailing list
> > > Swift-commit at ci.uchicago.edu
> > > https://lists.ci.uchicago.edu/ cgi-bin/mailman/listinfo/
> > > swift-commit
> > >
> > > --
> > > Michael Wilde
> > > Computation Institute, University of Chicago
> > > Mathematics and Computer Science Division
> > > Argonne National Laboratory
> > >
> > > ______________________________ _________________
> > > Swift-devel mailing list
> > > Swift-devel at ci.uchicago.edu
> > > https://lists.ci.uchicago.edu/ cgi-bin/mailman/listinfo/ swift-devel
> > >
> > > --
> > > Michael Wilde
> > > Computation Institute, University of Chicago
> > > Mathematics and Computer Science Division
> > > Argonne National Laboratory
> > >
> > >
> > >
> > >
> > > --
> > > Justin M Wozniak
> >
> > --
> > Michael Wilde
> > Computation Institute, University of Chicago
> > Mathematics and Computer Science Division
> > Argonne National Laboratory
>
> --
> Michael Wilde
> Computation Institute, University of Chicago
> Mathematics and Computer Science Division
> Argonne National Laboratory
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-devel/attachments/20111017/d13a0000/attachment.html>