From aespinosa at cs.uchicago.edu  Fri Apr  1 06:38:56 2011
From: aespinosa at cs.uchicago.edu (Allan Espinosa)
Date: Fri, 1 Apr 2011 06:38:56 -0500
Subject: [Swift-user] Re: array slicing?
In-Reply-To: <AANLkTikfO4XCcwZ3kkPLmE=5wPXQdLyLaWYjLW-9_c7q@mail.gmail.com>
References: <AANLkTikfO4XCcwZ3kkPLmE=5wPXQdLyLaWYjLW-9_c7q@mail.gmail.com>
Message-ID: <AANLkTin6ix38SdQRUThJP+V5G=SvWegJiT3XoPpE99Z3@mail.gmail.com>

Filed as Bug 307 (feature enhancement):
https://bugzilla.mcs.anl.gov/swift/show_bug.cgi?id=307

2011/3/25 Allan Espinosa <aespinosa at cs.uchicago.edu>:
> Are there plans to incorporate this into the language?
>
> I want to perform some aggregate operations in subelements of my array:
>
> type file;
>
> file somearray[] <array_mapper; file='one, two, three, ..., n'>;
>
>
> file reduction1;
> file reduction2;
> file reduction3;
> file commoninput <'somefile'>;
>
> reduction1 = some_reduction(somearray[0:60], commoninput);
> reduction2 = some_reduction(somearray[61:120], commoninput);
> reduction3 = some_reduction(somearray[120:n], commoninput);
>
>
>
> I am doing this so that I have more control over the number of times
> commoninput is being staged to multiple resources.
>
>
> -Allan
>



-- 
Allan M. Espinosa <http://amespinosa.wordpress.com>
PhD student, Computer Science
University of Chicago <http://people.cs.uchicago.edu/~aespinosa>


From wilde at mcs.anl.gov  Fri Apr  1 07:42:51 2011
From: wilde at mcs.anl.gov (Michael Wilde)
Date: Fri, 1 Apr 2011 07:42:51 -0500 (CDT)
Subject: [Swift-user] how to stop swift in the code
In-Reply-To: <b6f453.3a81.12f0f2c94de.Coremail.tianyu491433909@163.com>
Message-ID: <1810118703.53969.1301661771956.JavaMail.root@zimbra.anl.gov>

Please explain what you mean.

If you are asking "how does a swift script terminate?" - it terminates when all statements, including all function calls are complete.

If a program does not complete, its most likely that some parallel function call or expression evaluation is still waiting for a data item it depends on. On rare occasions that's due to a deadlock. More often its due to an app function not returning.

----- Original Message -----
> How to stop swift in the code?
> 
> 
> _______________________________________________
> Swift-user mailing list
> Swift-user at ci.uchicago.edu
> http://mail.ci.uchicago.edu/mailman/listinfo/swift-user

-- 
Michael Wilde
Computation Institute, University of Chicago
Mathematics and Computer Science Division
Argonne National Laboratory



From wilde at mcs.anl.gov  Sat Apr  2 08:05:46 2011
From: wilde at mcs.anl.gov (Michael Wilde)
Date: Sat, 2 Apr 2011 08:05:46 -0500 (CDT)
Subject: [Swift-user] Re: [Swift-devel] Coasters Configuration Optimizations
In-Reply-To: <05C21979-33F0-4DCD-A99B-DCE5771FE26C@mcs.anl.gov>
Message-ID: <82614059.57717.1301749546117.JavaMail.root@zimbra.anl.gov>

(replying to swift-user instead of devel)

Ketan,

Your main sites.xml coaster settings were:

    <execution provider="coaster" jobmanager="local:pbs"/>
    <profile namespace="globus" key="project">CI-CCR000013</profile>
    <profile namespace="globus" key="ppn">24:cray:pack</profile>
    <profile namespace="globus" key="workersPerNode">24</profile>
    <profile namespace="globus" key="maxTime">100000</profile>
    <profile namespace="globus" key="lowOverallocation">100</profile>
    <profile namespace="globus" key="highOverallocation">100</profile>
    <profile namespace="globus" key="slots">20</profile>
    <profile namespace="globus" key="nodeGranularity">5</profile>
    <profile namespace="globus" key="maxNodes">5</profile>
    <profile namespace="karajan" key="jobThrottle">20.00</profile>
    <profile namespace="karajan" key="initialScore">10000</profile>

Your tc entry (shortened here) was:

pbs modftdock /.../modftdock.sh null null GLOBUS::maxwalltime="02:00:00"

And you said you saw in PBS: 13 jobs of 24 hours and 4 jobs of 22 hours. I suspect this was after the script had been running a while, and many jobs had been completed.

Based on your settings, I think you should have had at one time about 17 coaster block jobs running, because the throttle on your coaster pool was set to 20 (which would cause Swift to try to run about 2000 apps at once - 2001 to be precise). Since each job should have requested exactly 5 nodes (based on your maxnodes=nodegranularity=5 setting above), Swift would have had to run 17 jobs to accomodate 2000 apps (17 * (5*24) ) = 2040 apps. 24 comes from your workerspernode setting, which is a poorly-named parameter that we are renaming to what it really specifies: appsPerNode for concurrent application calls per node.

I also suspect that that when this workflow started, coasters was requesting blocks of time closer to the 100,000 seconds that you specified for maxtime? (thats ~27 hours). I think the qstat snapshot you provided showed fewer than 17 jobs and job times shorter than 27 hours (24 and 22 hours) because there was no longer enough apps remaining to run to require those higher values. But it was still going to try to run all the remaining jobs - probably fewer than 2000 jobs remained when you run the enclosed qstat. In fact the jobs remaining at the time was likely less than:
  
  13*5*(24/2) + 4*5*(22/2) = 120*12 + 20*10 signifying <= 1640 jobs remaining

Since the maxwalltime estimate for your app in tc.data was 2 hours, I think coasters will pick a wall time that is the min(time needed for jobs remaining, time needed for max throttle jobs based on maxtime and high/low overallocation settings).

A note here to Swift developers: we need to first clarify the behavior of coasters in detail in the User Guide; then we need to build suitable templates that *greatly* simplify the settings and end-user parameters, and explain those simpler settings for use by all but the most sophisticated users with complex needs.

We also need to do much more experimentation to see if coasters will run OK with far less parameter-override specification, and see if its automation and algorithmic intelligence will do the right thing in almost all cases.

Most of the time in current use we specify overrides for almost all settings so that we get a precise shape and number of jobs submitted. Doing that assumes we know better than coasters and forces the user to understand how to override all the settings.

Its a very interesting question, and a hard but critically important one to answer to make usage simpler.

- Mike



----- Original Message -----
> Hello,
> 
> Today, I successfully ran an experiment with 5000 tasks on beagle with
> Coasters. These modFTdock tasks correspond to the production grade
> modFTdock parameters and each task takes around 20 minutes to
> complete.
> 
> After some discussions with Mike, I configured my sites.xml file to
> obtain necessary resources on beagle.
> 
> However, it seems that I did not configure my sites.xml optimally as
> the resources requested exceeded my requirements.
> 
> In summary 13 blocks for 24 hours and 4 blocks for 22 hours (120
> nodes) were requested while the needed were at most 4 hours on each
> block or more on less number of blocks.
> 
> Attached are the following:
> 
> sites.xml
> tc.data
> A qstat snapshot at the completion of the experiment.
> 
> Suggestions and insights on how to optimize the configuration are
> welcome.
> 
> 
> Regards,
> Ketan
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> _______________________________________________
> Swift-devel mailing list
> Swift-devel at ci.uchicago.edu
> http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel

-- 
Michael Wilde
Computation Institute, University of Chicago
Mathematics and Computer Science Division
Argonne National Laboratory



From wilde at mcs.anl.gov  Sat Apr  2 08:25:53 2011
From: wilde at mcs.anl.gov (Michael Wilde)
Date: Sat, 2 Apr 2011 08:25:53 -0500 (CDT)
Subject: [Swift-user] Correction: Problem exists in 0.92 binary release as
	well
In-Reply-To: <1665932704.53596.1301626111997.JavaMail.root@zimbra.anl.gov>
Message-ID: <573553267.57729.1301750753832.JavaMail.root@zimbra.anl.gov>

Dear Users,

We have found that the problem I pointed out in the March 31 message below also exists in the 0.92 binary release.

The symptom is that in certain cases of arrays that are computed within a script and iterated on later in the script using foreach, iterations of the foreach are executed twice.  This was not detected earlier because most scripts iterate over constant arrays, or arrays that are set by variants of readData.

We hope to correct these problems with a 0.92 fixed release.  Until we post that, use Swift 0.91 if you suspect that these or related synchronization bugs may be affecting your script's behavior.

We're very sorry that this slipped past our testing.

- Mike

----- Forwarded Message -----
From: "Michael Wilde" <wilde at mcs.anl.gov>
To: "Swift-User" <swift-user at ci.uchicago.edu>
Sent: Thursday, March 31, 2011 9:48:31 PM
Subject: [Swift-user] Problem in 0.92 source branch - do not use till	corrected

Dear Swift Users,

On March 29 we discovered that the Release 0.92 branches of the Swift and CoG trees were changed after the release and a concurrency bug was introduced. If you are running Swift from this *source code* base, please revert back to a known-working release such as the 0.92 binary release if at all possible.

We're working on restoring the 0.92 SVN branch to the correct state and will report back to this email list when that is done.

- Mike

-- 
Michael Wilde
Computation Institute, University of Chicago
Mathematics and Computer Science Division
Argonne National Laboratory

_______________________________________________
Swift-user mailing list
Swift-user at ci.uchicago.edu
http://mail.ci.uchicago.edu/mailman/listinfo/swift-user

-- 
Michael Wilde
Computation Institute, University of Chicago
Mathematics and Computer Science Division
Argonne National Laboratory



From jon.monette at gmail.com  Sun Apr  3 19:38:10 2011
From: jon.monette at gmail.com (Jonathan Monette)
Date: Sun, 3 Apr 2011 19:38:10 -0500
Subject: [Swift-user] mapping question
Message-ID: <BANLkTimbn_Os57zLmPW2JDneCGC=SCuCAQ@mail.gmail.com>

Hello,
   I have a situation that I am not sure how to resolve.  I have an app that
is similar to this.



-- 
Any intelligent fool can make things bigger and more complex... It takes a
touch of genius - and a lot of courage to move in the opposite direction.
- Albert Einstein
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-user/attachments/20110403/c2d8428e/attachment.html>

From jon.monette at gmail.com  Sun Apr  3 19:44:39 2011
From: jon.monette at gmail.com (Jonathan Monette)
Date: Sun, 3 Apr 2011 19:44:39 -0500
Subject: [Swift-user] Re: mapping question
In-Reply-To: <BANLkTimbn_Os57zLmPW2JDneCGC=SCuCAQ@mail.gmail.com>
References: <BANLkTimbn_Os57zLmPW2JDneCGC=SCuCAQ@mail.gmail.com>
Message-ID: <BANLkTimR7YcPJ3qh1+Ubyd3g4YXh+DyhvA@mail.gmail.com>

Hello,
   I have a situation that I am not sure how to resolve.  I have an app that
has this interface.
      myapp infile outfile metadata

The Swift app declaration i have is
      app ( Image outfile) myapp( Image infile Meta metadata)
      {
          myapp @infile @outfile @metadata;
      }

Howerver, myapp creates two files.  It creates a file with @outfile as the
filename and creates another file that is @outfile_area as the file name.
 How in swift do I map to the @outfile_area file since it is created just
based on the name of outfile?  Currently, I don't map the file so the file
is not staged out so it is deleted presumably when my jobs have finished
executed.  This file is not necessary for my workflow to complete but it is
another metadata file that helps in the approximation of the final image.


On Sun, Apr 3, 2011 at 7:38 PM, Jonathan Monette <jon.monette at gmail.com>wrote:

> Hello,
>    I have a situation that I am not sure how to resolve.  I have an app
> that is similar to this.
>
>
>
> --
> Any intelligent fool can make things bigger and more complex... It takes a
> touch of genius - and a lot of courage to move in the opposite direction.
> - Albert Einstein
>
>
>


-- 
Any intelligent fool can make things bigger and more complex... It takes a
touch of genius - and a lot of courage to move in the opposite direction.
- Albert Einstein
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-user/attachments/20110403/27e79f90/attachment.html>

From hategan at mcs.anl.gov  Sun Apr  3 20:56:44 2011
From: hategan at mcs.anl.gov (Mihael Hategan)
Date: Sun, 03 Apr 2011 18:56:44 -0700
Subject: [Swift-user] Re: mapping question
In-Reply-To: <BANLkTimR7YcPJ3qh1+Ubyd3g4YXh+DyhvA@mail.gmail.com>
References: <BANLkTimbn_Os57zLmPW2JDneCGC=SCuCAQ@mail.gmail.com>
	<BANLkTimR7YcPJ3qh1+Ubyd3g4YXh+DyhvA@mail.gmail.com>
Message-ID: <1301882204.13178.0.camel@blabla2.none>

Might the regular expression mapper work?

Mihael

On Sun, 2011-04-03 at 19:44 -0500, Jonathan Monette wrote:
> Hello,
>    I have a situation that I am not sure how to resolve.  I have an
> app that has this interface.
>       myapp infile outfile metadata
> 
> 
> The Swift app declaration i have is
>       app ( Image outfile) myapp( Image infile Meta metadata)
>       {
>           myapp @infile @outfile @metadata;
>       }
> 
> 
> Howerver, myapp creates two files.  It creates a file with @outfile as
> the filename and creates another file that is @outfile_area as the
> file name.  How in swift do I map to the @outfile_area file since it
> is created just based on the name of outfile?  Currently, I don't map
> the file so the file is not staged out so it is deleted presumably
> when my jobs have finished executed.  This file is not necessary for
> my workflow to complete but it is another metadata file that helps in
> the approximation of the final image.
> 
> 
> 
> On Sun, Apr 3, 2011 at 7:38 PM, Jonathan Monette
> <jon.monette at gmail.com> wrote:
>         Hello,
>            I have a situation that I am not sure how to resolve.  I
>         have an app that is similar to this.
>         
>         
>         
>         
>         -- 
>         Any intelligent fool can make things bigger and more
>         complex... It takes a touch of genius - and a lot of courage
>         to move in the opposite direction.
>         - Albert Einstein
>         
>         
>         
> 
> 
> 
> -- 
> Any intelligent fool can make things bigger and more complex... It
> takes a touch of genius - and a lot of courage to move in the opposite
> direction.
> - Albert Einstein
> 
> 
> 
> _______________________________________________
> Swift-user mailing list
> Swift-user at ci.uchicago.edu
> http://mail.ci.uchicago.edu/mailman/listinfo/swift-user




From jon.monette at gmail.com  Sun Apr  3 20:59:21 2011
From: jon.monette at gmail.com (Jonathan Monette)
Date: Sun, 3 Apr 2011 20:59:21 -0500
Subject: [Swift-user] Re: mapping question
In-Reply-To: <1301882204.13178.0.camel@blabla2.none>
References: <BANLkTimbn_Os57zLmPW2JDneCGC=SCuCAQ@mail.gmail.com>
	<BANLkTimR7YcPJ3qh1+Ubyd3g4YXh+DyhvA@mail.gmail.com>
	<1301882204.13178.0.camel@blabla2.none>
Message-ID: <BANLkTimS0MN7bsagAsXhVbvaYFKi1R4RWw@mail.gmail.com>

Yea but how would I get it back through the app?  Would something like
      app ( Image outfile, Image outfile_area ) myapp( Image infile Meta
metadata)
       {
           myapp @infile @outfile @metadata;
       }
work?  Even though it is not a file used in the app would the app call still
map to it?  Or do I use the regexp mapper after the app call?

On Sun, Apr 3, 2011 at 8:56 PM, Mihael Hategan <hategan at mcs.anl.gov> wrote:

> Might the regular expression mapper work?
>
> Mihael
>
> On Sun, 2011-04-03 at 19:44 -0500, Jonathan Monette wrote:
> > Hello,
> >    I have a situation that I am not sure how to resolve.  I have an
> > app that has this interface.
> >       myapp infile outfile metadata
> >
> >
> > The Swift app declaration i have is
> >       app ( Image outfile) myapp( Image infile Meta metadata)
> >       {
> >           myapp @infile @outfile @metadata;
> >       }
> >
> >
> > Howerver, myapp creates two files.  It creates a file with @outfile as
> > the filename and creates another file that is @outfile_area as the
> > file name.  How in swift do I map to the @outfile_area file since it
> > is created just based on the name of outfile?  Currently, I don't map
> > the file so the file is not staged out so it is deleted presumably
> > when my jobs have finished executed.  This file is not necessary for
> > my workflow to complete but it is another metadata file that helps in
> > the approximation of the final image.
> >
> >
> >
> > On Sun, Apr 3, 2011 at 7:38 PM, Jonathan Monette
> > <jon.monette at gmail.com> wrote:
> >         Hello,
> >            I have a situation that I am not sure how to resolve.  I
> >         have an app that is similar to this.
> >
> >
> >
> >
> >         --
> >         Any intelligent fool can make things bigger and more
> >         complex... It takes a touch of genius - and a lot of courage
> >         to move in the opposite direction.
> >         - Albert Einstein
> >
> >
> >
> >
> >
> >
> > --
> > Any intelligent fool can make things bigger and more complex... It
> > takes a touch of genius - and a lot of courage to move in the opposite
> > direction.
> > - Albert Einstein
> >
> >
> >
> > _______________________________________________
> > Swift-user mailing list
> > Swift-user at ci.uchicago.edu
> > http://mail.ci.uchicago.edu/mailman/listinfo/swift-user
>
>
>


-- 
Any intelligent fool can make things bigger and more complex... It takes a
touch of genius - and a lot of courage to move in the opposite direction.
- Albert Einstein
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-user/attachments/20110403/8e9c8409/attachment.html>

From hategan at mcs.anl.gov  Sun Apr  3 21:03:05 2011
From: hategan at mcs.anl.gov (Mihael Hategan)
Date: Sun, 03 Apr 2011 19:03:05 -0700
Subject: [Swift-user] Re: mapping question
In-Reply-To: <BANLkTimS0MN7bsagAsXhVbvaYFKi1R4RWw@mail.gmail.com>
References: <BANLkTimbn_Os57zLmPW2JDneCGC=SCuCAQ@mail.gmail.com>
	<BANLkTimR7YcPJ3qh1+Ubyd3g4YXh+DyhvA@mail.gmail.com>
	<1301882204.13178.0.camel@blabla2.none>
	<BANLkTimS0MN7bsagAsXhVbvaYFKi1R4RWw@mail.gmail.com>
Message-ID: <1301882585.13511.0.camel@blabla2.none>

On Sun, 2011-04-03 at 20:59 -0500, Jonathan Monette wrote:
> Yea but how would I get it back through the app?  Would something
> like 
>       app ( Image outfile, Image outfile_area ) myapp( Image infile
> Meta metadata)
>        {
>            myapp @infile @outfile @metadata;
>        }
> work?  Even though it is not a file used in the app would the app call
> still map to it?  Or do I use the regexp mapper after the app call?

Swift will try to stage that file out if it's one of the app returns.
That is irrespective of whether it appears on the command line or not.
So yes, your example above should work.

Mihael
> 
> On Sun, Apr 3, 2011 at 8:56 PM, Mihael Hategan <hategan at mcs.anl.gov>
> wrote:
>         Might the regular expression mapper work?
>         
>         Mihael
>         
>         
>         On Sun, 2011-04-03 at 19:44 -0500, Jonathan Monette wrote:
>         > Hello,
>         >    I have a situation that I am not sure how to resolve.  I
>         have an
>         > app that has this interface.
>         >       myapp infile outfile metadata
>         >
>         >
>         > The Swift app declaration i have is
>         >       app ( Image outfile) myapp( Image infile Meta
>         metadata)
>         >       {
>         >           myapp @infile @outfile @metadata;
>         >       }
>         >
>         >
>         > Howerver, myapp creates two files.  It creates a file with
>         @outfile as
>         > the filename and creates another file that is @outfile_area
>         as the
>         > file name.  How in swift do I map to the @outfile_area file
>         since it
>         > is created just based on the name of outfile?  Currently, I
>         don't map
>         > the file so the file is not staged out so it is deleted
>         presumably
>         > when my jobs have finished executed.  This file is not
>         necessary for
>         > my workflow to complete but it is another metadata file that
>         helps in
>         > the approximation of the final image.
>         >
>         >
>         >
>         > On Sun, Apr 3, 2011 at 7:38 PM, Jonathan Monette
>         > <jon.monette at gmail.com> wrote:
>         >         Hello,
>         >            I have a situation that I am not sure how to
>         resolve.  I
>         >         have an app that is similar to this.
>         >
>         >
>         >
>         >
>         >         --
>         >         Any intelligent fool can make things bigger and more
>         >         complex... It takes a touch of genius - and a lot of
>         courage
>         >         to move in the opposite direction.
>         >         - Albert Einstein
>         >
>         >
>         >
>         >
>         >
>         >
>         > --
>         > Any intelligent fool can make things bigger and more
>         complex... It
>         > takes a touch of genius - and a lot of courage to move in
>         the opposite
>         > direction.
>         > - Albert Einstein
>         >
>         >
>         >
>         
>         > _______________________________________________
>         > Swift-user mailing list
>         > Swift-user at ci.uchicago.edu
>         > http://mail.ci.uchicago.edu/mailman/listinfo/swift-user
>         
>         
> 
> 
> 
> -- 
> Any intelligent fool can make things bigger and more complex... It
> takes a touch of genius - and a lot of courage to move in the opposite
> direction.
> - Albert Einstein
> 
> 
> 




From jon.monette at gmail.com  Sun Apr  3 21:05:33 2011
From: jon.monette at gmail.com (Jonathan Monette)
Date: Sun, 3 Apr 2011 21:05:33 -0500
Subject: [Swift-user] Re: mapping question
In-Reply-To: <1301882585.13511.0.camel@blabla2.none>
References: <BANLkTimbn_Os57zLmPW2JDneCGC=SCuCAQ@mail.gmail.com>
	<BANLkTimR7YcPJ3qh1+Ubyd3g4YXh+DyhvA@mail.gmail.com>
	<1301882204.13178.0.camel@blabla2.none>
	<BANLkTimS0MN7bsagAsXhVbvaYFKi1R4RWw@mail.gmail.com>
	<1301882585.13511.0.camel@blabla2.none>
Message-ID: <BANLkTikJRK17ARKFGj_1OAriQ05P+idUgg@mail.gmail.com>

Ok.  I thought the if the file was an app return it had to be used so it
will be closed and staged out.

On Sun, Apr 3, 2011 at 9:03 PM, Mihael Hategan <hategan at mcs.anl.gov> wrote:

> On Sun, 2011-04-03 at 20:59 -0500, Jonathan Monette wrote:
> > Yea but how would I get it back through the app?  Would something
> > like
> >       app ( Image outfile, Image outfile_area ) myapp( Image infile
> > Meta metadata)
> >        {
> >            myapp @infile @outfile @metadata;
> >        }
> > work?  Even though it is not a file used in the app would the app call
> > still map to it?  Or do I use the regexp mapper after the app call?
>
> Swift will try to stage that file out if it's one of the app returns.
> That is irrespective of whether it appears on the command line or not.
> So yes, your example above should work.
>
> Mihael
> >
> > On Sun, Apr 3, 2011 at 8:56 PM, Mihael Hategan <hategan at mcs.anl.gov>
> > wrote:
> >         Might the regular expression mapper work?
> >
> >         Mihael
> >
> >
> >         On Sun, 2011-04-03 at 19:44 -0500, Jonathan Monette wrote:
> >         > Hello,
> >         >    I have a situation that I am not sure how to resolve.  I
> >         have an
> >         > app that has this interface.
> >         >       myapp infile outfile metadata
> >         >
> >         >
> >         > The Swift app declaration i have is
> >         >       app ( Image outfile) myapp( Image infile Meta
> >         metadata)
> >         >       {
> >         >           myapp @infile @outfile @metadata;
> >         >       }
> >         >
> >         >
> >         > Howerver, myapp creates two files.  It creates a file with
> >         @outfile as
> >         > the filename and creates another file that is @outfile_area
> >         as the
> >         > file name.  How in swift do I map to the @outfile_area file
> >         since it
> >         > is created just based on the name of outfile?  Currently, I
> >         don't map
> >         > the file so the file is not staged out so it is deleted
> >         presumably
> >         > when my jobs have finished executed.  This file is not
> >         necessary for
> >         > my workflow to complete but it is another metadata file that
> >         helps in
> >         > the approximation of the final image.
> >         >
> >         >
> >         >
> >         > On Sun, Apr 3, 2011 at 7:38 PM, Jonathan Monette
> >         > <jon.monette at gmail.com> wrote:
> >         >         Hello,
> >         >            I have a situation that I am not sure how to
> >         resolve.  I
> >         >         have an app that is similar to this.
> >         >
> >         >
> >         >
> >         >
> >         >         --
> >         >         Any intelligent fool can make things bigger and more
> >         >         complex... It takes a touch of genius - and a lot of
> >         courage
> >         >         to move in the opposite direction.
> >         >         - Albert Einstein
> >         >
> >         >
> >         >
> >         >
> >         >
> >         >
> >         > --
> >         > Any intelligent fool can make things bigger and more
> >         complex... It
> >         > takes a touch of genius - and a lot of courage to move in
> >         the opposite
> >         > direction.
> >         > - Albert Einstein
> >         >
> >         >
> >         >
> >
> >         > _______________________________________________
> >         > Swift-user mailing list
> >         > Swift-user at ci.uchicago.edu
> >         > http://mail.ci.uchicago.edu/mailman/listinfo/swift-user
> >
> >
> >
> >
> >
> > --
> > Any intelligent fool can make things bigger and more complex... It
> > takes a touch of genius - and a lot of courage to move in the opposite
> > direction.
> > - Albert Einstein
> >
> >
> >
>
>
>


-- 
Any intelligent fool can make things bigger and more complex... It takes a
touch of genius - and a lot of courage to move in the opposite direction.
- Albert Einstein
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-user/attachments/20110403/b60b5428/attachment.html>

From hategan at mcs.anl.gov  Sun Apr  3 21:17:15 2011
From: hategan at mcs.anl.gov (Mihael Hategan)
Date: Sun, 03 Apr 2011 19:17:15 -0700
Subject: [Swift-user] Re: mapping question
In-Reply-To: <BANLkTikJRK17ARKFGj_1OAriQ05P+idUgg@mail.gmail.com>
References: <BANLkTimbn_Os57zLmPW2JDneCGC=SCuCAQ@mail.gmail.com>
	<BANLkTimR7YcPJ3qh1+Ubyd3g4YXh+DyhvA@mail.gmail.com>
	<1301882204.13178.0.camel@blabla2.none>
	<BANLkTimS0MN7bsagAsXhVbvaYFKi1R4RWw@mail.gmail.com>
	<1301882585.13511.0.camel@blabla2.none>
	<BANLkTikJRK17ARKFGj_1OAriQ05P+idUgg@mail.gmail.com>
Message-ID: <1301883435.13511.1.camel@blabla2.none>

On Sun, 2011-04-03 at 21:05 -0500, Jonathan Monette wrote:
> Ok.  I thought the if the file was an app return it had to be used so
> it will be closed and staged out.

Define "used"

> 
> On Sun, Apr 3, 2011 at 9:03 PM, Mihael Hategan <hategan at mcs.anl.gov>
> wrote:
>         On Sun, 2011-04-03 at 20:59 -0500, Jonathan Monette wrote:
>         > Yea but how would I get it back through the app?  Would
>         something
>         > like
>         >       app ( Image outfile, Image outfile_area ) myapp( Image
>         infile
>         > Meta metadata)
>         >        {
>         >            myapp @infile @outfile @metadata;
>         >        }
>         > work?  Even though it is not a file used in the app would
>         the app call
>         > still map to it?  Or do I use the regexp mapper after the
>         app call?
>         
>         
>         Swift will try to stage that file out if it's one of the app
>         returns.
>         That is irrespective of whether it appears on the command line
>         or not.
>         So yes, your example above should work.
>         
>         Mihael
>         
>         >
>         > On Sun, Apr 3, 2011 at 8:56 PM, Mihael Hategan
>         <hategan at mcs.anl.gov>
>         > wrote:
>         >         Might the regular expression mapper work?
>         >
>         >         Mihael
>         >
>         >
>         >         On Sun, 2011-04-03 at 19:44 -0500, Jonathan Monette
>         wrote:
>         >         > Hello,
>         >         >    I have a situation that I am not sure how to
>         resolve.  I
>         >         have an
>         >         > app that has this interface.
>         >         >       myapp infile outfile metadata
>         >         >
>         >         >
>         >         > The Swift app declaration i have is
>         >         >       app ( Image outfile) myapp( Image infile
>         Meta
>         >         metadata)
>         >         >       {
>         >         >           myapp @infile @outfile @metadata;
>         >         >       }
>         >         >
>         >         >
>         >         > Howerver, myapp creates two files.  It creates a
>         file with
>         >         @outfile as
>         >         > the filename and creates another file that is
>         @outfile_area
>         >         as the
>         >         > file name.  How in swift do I map to the
>         @outfile_area file
>         >         since it
>         >         > is created just based on the name of outfile?
>          Currently, I
>         >         don't map
>         >         > the file so the file is not staged out so it is
>         deleted
>         >         presumably
>         >         > when my jobs have finished executed.  This file is
>         not
>         >         necessary for
>         >         > my workflow to complete but it is another metadata
>         file that
>         >         helps in
>         >         > the approximation of the final image.
>         >         >
>         >         >
>         >         >
>         >         > On Sun, Apr 3, 2011 at 7:38 PM, Jonathan Monette
>         >         > <jon.monette at gmail.com> wrote:
>         >         >         Hello,
>         >         >            I have a situation that I am not sure
>         how to
>         >         resolve.  I
>         >         >         have an app that is similar to this.
>         >         >
>         >         >
>         >         >
>         >         >
>         >         >         --
>         >         >         Any intelligent fool can make things
>         bigger and more
>         >         >         complex... It takes a touch of genius -
>         and a lot of
>         >         courage
>         >         >         to move in the opposite direction.
>         >         >         - Albert Einstein
>         >         >
>         >         >
>         >         >
>         >         >
>         >         >
>         >         >
>         >         > --
>         >         > Any intelligent fool can make things bigger and
>         more
>         >         complex... It
>         >         > takes a touch of genius - and a lot of courage to
>         move in
>         >         the opposite
>         >         > direction.
>         >         > - Albert Einstein
>         >         >
>         >         >
>         >         >
>         >
>         >         > _______________________________________________
>         >         > Swift-user mailing list
>         >         > Swift-user at ci.uchicago.edu
>         >         >
>         http://mail.ci.uchicago.edu/mailman/listinfo/swift-user
>         >
>         >
>         >
>         >
>         >
>         > --
>         > Any intelligent fool can make things bigger and more
>         complex... It
>         > takes a touch of genius - and a lot of courage to move in
>         the opposite
>         > direction.
>         > - Albert Einstein
>         >
>         >
>         >
>         
>         
>         
> 
> 
> 
> -- 
> Any intelligent fool can make things bigger and more complex... It
> takes a touch of genius - and a lot of courage to move in the opposite
> direction.
> - Albert Einstein
> 
> 




From jon.monette at gmail.com  Sun Apr  3 21:18:35 2011
From: jon.monette at gmail.com (Jonathan Monette)
Date: Sun, 3 Apr 2011 21:18:35 -0500
Subject: [Swift-user] Re: mapping question
In-Reply-To: <1301883435.13511.1.camel@blabla2.none>
References: <BANLkTimbn_Os57zLmPW2JDneCGC=SCuCAQ@mail.gmail.com>
	<BANLkTimR7YcPJ3qh1+Ubyd3g4YXh+DyhvA@mail.gmail.com>
	<1301882204.13178.0.camel@blabla2.none>
	<BANLkTimS0MN7bsagAsXhVbvaYFKi1R4RWw@mail.gmail.com>
	<1301882585.13511.0.camel@blabla2.none>
	<BANLkTikJRK17ARKFGj_1OAriQ05P+idUgg@mail.gmail.com>
	<1301883435.13511.1.camel@blabla2.none>
Message-ID: <BANLkTi=VDOXYE228dzSLc__1_vSt7JqYag@mail.gmail.com>

Actually in the app call.

myapp @infile @outfile @outfile_are @metadata

On Sun, Apr 3, 2011 at 9:17 PM, Mihael Hategan <hategan at mcs.anl.gov> wrote:

> On Sun, 2011-04-03 at 21:05 -0500, Jonathan Monette wrote:
> > Ok.  I thought the if the file was an app return it had to be used so
> > it will be closed and staged out.
>
> Define "used"
>
> >
> > On Sun, Apr 3, 2011 at 9:03 PM, Mihael Hategan <hategan at mcs.anl.gov>
> > wrote:
> >         On Sun, 2011-04-03 at 20:59 -0500, Jonathan Monette wrote:
> >         > Yea but how would I get it back through the app?  Would
> >         something
> >         > like
> >         >       app ( Image outfile, Image outfile_area ) myapp( Image
> >         infile
> >         > Meta metadata)
> >         >        {
> >         >            myapp @infile @outfile @metadata;
> >         >        }
> >         > work?  Even though it is not a file used in the app would
> >         the app call
> >         > still map to it?  Or do I use the regexp mapper after the
> >         app call?
> >
> >
> >         Swift will try to stage that file out if it's one of the app
> >         returns.
> >         That is irrespective of whether it appears on the command line
> >         or not.
> >         So yes, your example above should work.
> >
> >         Mihael
> >
> >         >
> >         > On Sun, Apr 3, 2011 at 8:56 PM, Mihael Hategan
> >         <hategan at mcs.anl.gov>
> >         > wrote:
> >         >         Might the regular expression mapper work?
> >         >
> >         >         Mihael
> >         >
> >         >
> >         >         On Sun, 2011-04-03 at 19:44 -0500, Jonathan Monette
> >         wrote:
> >         >         > Hello,
> >         >         >    I have a situation that I am not sure how to
> >         resolve.  I
> >         >         have an
> >         >         > app that has this interface.
> >         >         >       myapp infile outfile metadata
> >         >         >
> >         >         >
> >         >         > The Swift app declaration i have is
> >         >         >       app ( Image outfile) myapp( Image infile
> >         Meta
> >         >         metadata)
> >         >         >       {
> >         >         >           myapp @infile @outfile @metadata;
> >         >         >       }
> >         >         >
> >         >         >
> >         >         > Howerver, myapp creates two files.  It creates a
> >         file with
> >         >         @outfile as
> >         >         > the filename and creates another file that is
> >         @outfile_area
> >         >         as the
> >         >         > file name.  How in swift do I map to the
> >         @outfile_area file
> >         >         since it
> >         >         > is created just based on the name of outfile?
> >          Currently, I
> >         >         don't map
> >         >         > the file so the file is not staged out so it is
> >         deleted
> >         >         presumably
> >         >         > when my jobs have finished executed.  This file is
> >         not
> >         >         necessary for
> >         >         > my workflow to complete but it is another metadata
> >         file that
> >         >         helps in
> >         >         > the approximation of the final image.
> >         >         >
> >         >         >
> >         >         >
> >         >         > On Sun, Apr 3, 2011 at 7:38 PM, Jonathan Monette
> >         >         > <jon.monette at gmail.com> wrote:
> >         >         >         Hello,
> >         >         >            I have a situation that I am not sure
> >         how to
> >         >         resolve.  I
> >         >         >         have an app that is similar to this.
> >         >         >
> >         >         >
> >         >         >
> >         >         >
> >         >         >         --
> >         >         >         Any intelligent fool can make things
> >         bigger and more
> >         >         >         complex... It takes a touch of genius -
> >         and a lot of
> >         >         courage
> >         >         >         to move in the opposite direction.
> >         >         >         - Albert Einstein
> >         >         >
> >         >         >
> >         >         >
> >         >         >
> >         >         >
> >         >         >
> >         >         > --
> >         >         > Any intelligent fool can make things bigger and
> >         more
> >         >         complex... It
> >         >         > takes a touch of genius - and a lot of courage to
> >         move in
> >         >         the opposite
> >         >         > direction.
> >         >         > - Albert Einstein
> >         >         >
> >         >         >
> >         >         >
> >         >
> >         >         > _______________________________________________
> >         >         > Swift-user mailing list
> >         >         > Swift-user at ci.uchicago.edu
> >         >         >
> >         http://mail.ci.uchicago.edu/mailman/listinfo/swift-user
> >         >
> >         >
> >         >
> >         >
> >         >
> >         > --
> >         > Any intelligent fool can make things bigger and more
> >         complex... It
> >         > takes a touch of genius - and a lot of courage to move in
> >         the opposite
> >         > direction.
> >         > - Albert Einstein
> >         >
> >         >
> >         >
> >
> >
> >
> >
> >
> >
> > --
> > Any intelligent fool can make things bigger and more complex... It
> > takes a touch of genius - and a lot of courage to move in the opposite
> > direction.
> > - Albert Einstein
> >
> >
>
>
>


-- 
Any intelligent fool can make things bigger and more complex... It takes a
touch of genius - and a lot of courage to move in the opposite direction.
- Albert Einstein
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-user/attachments/20110403/3ae1c8db/attachment.html>

From hategan at mcs.anl.gov  Sun Apr  3 21:27:57 2011
From: hategan at mcs.anl.gov (Mihael Hategan)
Date: Sun, 03 Apr 2011 19:27:57 -0700
Subject: [Swift-user] Re: mapping question
In-Reply-To: <BANLkTi=VDOXYE228dzSLc__1_vSt7JqYag@mail.gmail.com>
References: <BANLkTimbn_Os57zLmPW2JDneCGC=SCuCAQ@mail.gmail.com>
	<BANLkTimR7YcPJ3qh1+Ubyd3g4YXh+DyhvA@mail.gmail.com>
	<1301882204.13178.0.camel@blabla2.none>
	<BANLkTimS0MN7bsagAsXhVbvaYFKi1R4RWw@mail.gmail.com>
	<1301882585.13511.0.camel@blabla2.none>
	<BANLkTikJRK17ARKFGj_1OAriQ05P+idUgg@mail.gmail.com>
	<1301883435.13511.1.camel@blabla2.none>
	<BANLkTi=VDOXYE228dzSLc__1_vSt7JqYag@mail.gmail.com>
Message-ID: <1301884077.14415.1.camel@blabla2.none>

On Sun, 2011-04-03 at 21:18 -0500, Jonathan Monette wrote:
> Actually in the app call.  
> 
> 
> myapp @infile @outfile @outfile_are @metadata

Nope. It doesn't have to be. It can very well be empty (i.e. just
"myapp;") as long as swift is concerned. It will still try to stage in
all arguments and stage out all returns.
> 
> On Sun, Apr 3, 2011 at 9:17 PM, Mihael Hategan <hategan at mcs.anl.gov>
> wrote:
>         On Sun, 2011-04-03 at 21:05 -0500, Jonathan Monette wrote:
>         > Ok.  I thought the if the file was an app return it had to
>         be used so
>         > it will be closed and staged out.
>         
>         
>         Define "used"
>         
>         
>         >
>         > On Sun, Apr 3, 2011 at 9:03 PM, Mihael Hategan
>         <hategan at mcs.anl.gov>
>         > wrote:
>         >         On Sun, 2011-04-03 at 20:59 -0500, Jonathan Monette
>         wrote:
>         >         > Yea but how would I get it back through the app?
>          Would
>         >         something
>         >         > like
>         >         >       app ( Image outfile, Image outfile_area )
>         myapp( Image
>         >         infile
>         >         > Meta metadata)
>         >         >        {
>         >         >            myapp @infile @outfile @metadata;
>         >         >        }
>         >         > work?  Even though it is not a file used in the
>         app would
>         >         the app call
>         >         > still map to it?  Or do I use the regexp mapper
>         after the
>         >         app call?
>         >
>         >
>         >         Swift will try to stage that file out if it's one of
>         the app
>         >         returns.
>         >         That is irrespective of whether it appears on the
>         command line
>         >         or not.
>         >         So yes, your example above should work.
>         >
>         >         Mihael
>         >
>         >         >
>         >         > On Sun, Apr 3, 2011 at 8:56 PM, Mihael Hategan
>         >         <hategan at mcs.anl.gov>
>         >         > wrote:
>         >         >         Might the regular expression mapper work?
>         >         >
>         >         >         Mihael
>         >         >
>         >         >
>         >         >         On Sun, 2011-04-03 at 19:44 -0500,
>         Jonathan Monette
>         >         wrote:
>         >         >         > Hello,
>         >         >         >    I have a situation that I am not sure
>         how to
>         >         resolve.  I
>         >         >         have an
>         >         >         > app that has this interface.
>         >         >         >       myapp infile outfile metadata
>         >         >         >
>         >         >         >
>         >         >         > The Swift app declaration i have is
>         >         >         >       app ( Image outfile) myapp( Image
>         infile
>         >         Meta
>         >         >         metadata)
>         >         >         >       {
>         >         >         >           myapp @infile @outfile
>         @metadata;
>         >         >         >       }
>         >         >         >
>         >         >         >
>         >         >         > Howerver, myapp creates two files.  It
>         creates a
>         >         file with
>         >         >         @outfile as
>         >         >         > the filename and creates another file
>         that is
>         >         @outfile_area
>         >         >         as the
>         >         >         > file name.  How in swift do I map to the
>         >         @outfile_area file
>         >         >         since it
>         >         >         > is created just based on the name of
>         outfile?
>         >          Currently, I
>         >         >         don't map
>         >         >         > the file so the file is not staged out
>         so it is
>         >         deleted
>         >         >         presumably
>         >         >         > when my jobs have finished executed.
>          This file is
>         >         not
>         >         >         necessary for
>         >         >         > my workflow to complete but it is
>         another metadata
>         >         file that
>         >         >         helps in
>         >         >         > the approximation of the final image.
>         >         >         >
>         >         >         >
>         >         >         >
>         >         >         > On Sun, Apr 3, 2011 at 7:38 PM, Jonathan
>         Monette
>         >         >         > <jon.monette at gmail.com> wrote:
>         >         >         >         Hello,
>         >         >         >            I have a situation that I am
>         not sure
>         >         how to
>         >         >         resolve.  I
>         >         >         >         have an app that is similar to
>         this.
>         >         >         >
>         >         >         >
>         >         >         >
>         >         >         >
>         >         >         >         --
>         >         >         >         Any intelligent fool can make
>         things
>         >         bigger and more
>         >         >         >         complex... It takes a touch of
>         genius -
>         >         and a lot of
>         >         >         courage
>         >         >         >         to move in the opposite
>         direction.
>         >         >         >         - Albert Einstein
>         >         >         >
>         >         >         >
>         >         >         >
>         >         >         >
>         >         >         >
>         >         >         >
>         >         >         > --
>         >         >         > Any intelligent fool can make things
>         bigger and
>         >         more
>         >         >         complex... It
>         >         >         > takes a touch of genius - and a lot of
>         courage to
>         >         move in
>         >         >         the opposite
>         >         >         > direction.
>         >         >         > - Albert Einstein
>         >         >         >
>         >         >         >
>         >         >         >
>         >         >
>         >         >         >
>         _______________________________________________
>         >         >         > Swift-user mailing list
>         >         >         > Swift-user at ci.uchicago.edu
>         >         >         >
>         >
>         http://mail.ci.uchicago.edu/mailman/listinfo/swift-user
>         >         >
>         >         >
>         >         >
>         >         >
>         >         >
>         >         > --
>         >         > Any intelligent fool can make things bigger and
>         more
>         >         complex... It
>         >         > takes a touch of genius - and a lot of courage to
>         move in
>         >         the opposite
>         >         > direction.
>         >         > - Albert Einstein
>         >         >
>         >         >
>         >         >
>         >
>         >
>         >
>         >
>         >
>         >
>         > --
>         > Any intelligent fool can make things bigger and more
>         complex... It
>         > takes a touch of genius - and a lot of courage to move in
>         the opposite
>         > direction.
>         > - Albert Einstein
>         >
>         >
>         
>         
>         
> 
> 
> 
> -- 
> Any intelligent fool can make things bigger and more complex... It
> takes a touch of genius - and a lot of courage to move in the opposite
> direction.
> - Albert Einstein
> 
> 
> 




From jon.monette at gmail.com  Sun Apr  3 21:28:31 2011
From: jon.monette at gmail.com (Jonathan Monette)
Date: Sun, 3 Apr 2011 21:28:31 -0500
Subject: [Swift-user] Re: mapping question
In-Reply-To: <1301884077.14415.1.camel@blabla2.none>
References: <BANLkTimbn_Os57zLmPW2JDneCGC=SCuCAQ@mail.gmail.com>
	<BANLkTimR7YcPJ3qh1+Ubyd3g4YXh+DyhvA@mail.gmail.com>
	<1301882204.13178.0.camel@blabla2.none>
	<BANLkTimS0MN7bsagAsXhVbvaYFKi1R4RWw@mail.gmail.com>
	<1301882585.13511.0.camel@blabla2.none>
	<BANLkTikJRK17ARKFGj_1OAriQ05P+idUgg@mail.gmail.com>
	<1301883435.13511.1.camel@blabla2.none>
	<BANLkTi=VDOXYE228dzSLc__1_vSt7JqYag@mail.gmail.com>
	<1301884077.14415.1.camel@blabla2.none>
Message-ID: <BANLkTimmCFpB6_4i18mMZNx4SkHVKPmGMA@mail.gmail.com>

ok.  Thanks.

On Sun, Apr 3, 2011 at 9:27 PM, Mihael Hategan <hategan at mcs.anl.gov> wrote:

> On Sun, 2011-04-03 at 21:18 -0500, Jonathan Monette wrote:
> > Actually in the app call.
> >
> >
> > myapp @infile @outfile @outfile_are @metadata
>
> Nope. It doesn't have to be. It can very well be empty (i.e. just
> "myapp;") as long as swift is concerned. It will still try to stage in
> all arguments and stage out all returns.
> >
> > On Sun, Apr 3, 2011 at 9:17 PM, Mihael Hategan <hategan at mcs.anl.gov>
> > wrote:
> >         On Sun, 2011-04-03 at 21:05 -0500, Jonathan Monette wrote:
> >         > Ok.  I thought the if the file was an app return it had to
> >         be used so
> >         > it will be closed and staged out.
> >
> >
> >         Define "used"
> >
> >
> >         >
> >         > On Sun, Apr 3, 2011 at 9:03 PM, Mihael Hategan
> >         <hategan at mcs.anl.gov>
> >         > wrote:
> >         >         On Sun, 2011-04-03 at 20:59 -0500, Jonathan Monette
> >         wrote:
> >         >         > Yea but how would I get it back through the app?
> >          Would
> >         >         something
> >         >         > like
> >         >         >       app ( Image outfile, Image outfile_area )
> >         myapp( Image
> >         >         infile
> >         >         > Meta metadata)
> >         >         >        {
> >         >         >            myapp @infile @outfile @metadata;
> >         >         >        }
> >         >         > work?  Even though it is not a file used in the
> >         app would
> >         >         the app call
> >         >         > still map to it?  Or do I use the regexp mapper
> >         after the
> >         >         app call?
> >         >
> >         >
> >         >         Swift will try to stage that file out if it's one of
> >         the app
> >         >         returns.
> >         >         That is irrespective of whether it appears on the
> >         command line
> >         >         or not.
> >         >         So yes, your example above should work.
> >         >
> >         >         Mihael
> >         >
> >         >         >
> >         >         > On Sun, Apr 3, 2011 at 8:56 PM, Mihael Hategan
> >         >         <hategan at mcs.anl.gov>
> >         >         > wrote:
> >         >         >         Might the regular expression mapper work?
> >         >         >
> >         >         >         Mihael
> >         >         >
> >         >         >
> >         >         >         On Sun, 2011-04-03 at 19:44 -0500,
> >         Jonathan Monette
> >         >         wrote:
> >         >         >         > Hello,
> >         >         >         >    I have a situation that I am not sure
> >         how to
> >         >         resolve.  I
> >         >         >         have an
> >         >         >         > app that has this interface.
> >         >         >         >       myapp infile outfile metadata
> >         >         >         >
> >         >         >         >
> >         >         >         > The Swift app declaration i have is
> >         >         >         >       app ( Image outfile) myapp( Image
> >         infile
> >         >         Meta
> >         >         >         metadata)
> >         >         >         >       {
> >         >         >         >           myapp @infile @outfile
> >         @metadata;
> >         >         >         >       }
> >         >         >         >
> >         >         >         >
> >         >         >         > Howerver, myapp creates two files.  It
> >         creates a
> >         >         file with
> >         >         >         @outfile as
> >         >         >         > the filename and creates another file
> >         that is
> >         >         @outfile_area
> >         >         >         as the
> >         >         >         > file name.  How in swift do I map to the
> >         >         @outfile_area file
> >         >         >         since it
> >         >         >         > is created just based on the name of
> >         outfile?
> >         >          Currently, I
> >         >         >         don't map
> >         >         >         > the file so the file is not staged out
> >         so it is
> >         >         deleted
> >         >         >         presumably
> >         >         >         > when my jobs have finished executed.
> >          This file is
> >         >         not
> >         >         >         necessary for
> >         >         >         > my workflow to complete but it is
> >         another metadata
> >         >         file that
> >         >         >         helps in
> >         >         >         > the approximation of the final image.
> >         >         >         >
> >         >         >         >
> >         >         >         >
> >         >         >         > On Sun, Apr 3, 2011 at 7:38 PM, Jonathan
> >         Monette
> >         >         >         > <jon.monette at gmail.com> wrote:
> >         >         >         >         Hello,
> >         >         >         >            I have a situation that I am
> >         not sure
> >         >         how to
> >         >         >         resolve.  I
> >         >         >         >         have an app that is similar to
> >         this.
> >         >         >         >
> >         >         >         >
> >         >         >         >
> >         >         >         >
> >         >         >         >         --
> >         >         >         >         Any intelligent fool can make
> >         things
> >         >         bigger and more
> >         >         >         >         complex... It takes a touch of
> >         genius -
> >         >         and a lot of
> >         >         >         courage
> >         >         >         >         to move in the opposite
> >         direction.
> >         >         >         >         - Albert Einstein
> >         >         >         >
> >         >         >         >
> >         >         >         >
> >         >         >         >
> >         >         >         >
> >         >         >         >
> >         >         >         > --
> >         >         >         > Any intelligent fool can make things
> >         bigger and
> >         >         more
> >         >         >         complex... It
> >         >         >         > takes a touch of genius - and a lot of
> >         courage to
> >         >         move in
> >         >         >         the opposite
> >         >         >         > direction.
> >         >         >         > - Albert Einstein
> >         >         >         >
> >         >         >         >
> >         >         >         >
> >         >         >
> >         >         >         >
> >         _______________________________________________
> >         >         >         > Swift-user mailing list
> >         >         >         > Swift-user at ci.uchicago.edu
> >         >         >         >
> >         >
> >         http://mail.ci.uchicago.edu/mailman/listinfo/swift-user
> >         >         >
> >         >         >
> >         >         >
> >         >         >
> >         >         >
> >         >         > --
> >         >         > Any intelligent fool can make things bigger and
> >         more
> >         >         complex... It
> >         >         > takes a touch of genius - and a lot of courage to
> >         move in
> >         >         the opposite
> >         >         > direction.
> >         >         > - Albert Einstein
> >         >         >
> >         >         >
> >         >         >
> >         >
> >         >
> >         >
> >         >
> >         >
> >         >
> >         > --
> >         > Any intelligent fool can make things bigger and more
> >         complex... It
> >         > takes a touch of genius - and a lot of courage to move in
> >         the opposite
> >         > direction.
> >         > - Albert Einstein
> >         >
> >         >
> >
> >
> >
> >
> >
> >
> > --
> > Any intelligent fool can make things bigger and more complex... It
> > takes a touch of genius - and a lot of courage to move in the opposite
> > direction.
> > - Albert Einstein
> >
> >
> >
>
>
>


-- 
Any intelligent fool can make things bigger and more complex... It takes a
touch of genius - and a lot of courage to move in the opposite direction.
- Albert Einstein
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-user/attachments/20110403/2cf2cf42/attachment.html>

From tianyu491433909 at 163.com  Sun Apr  3 21:38:28 2011
From: tianyu491433909 at 163.com (tianyu491433909)
Date: Mon, 4 Apr 2011 10:38:28 +0800 (CST)
Subject: [Swift-user] how to terminate swift in the code
Message-ID: <53e06f7d.7d55.12f1e60270a.Coremail.tianyu491433909@163.com>

 in the code, I need to identify the value of the termination of a running workflow,how to complete this function
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-user/attachments/20110404/8206b49c/attachment.html>

From wilde at mcs.anl.gov  Sun Apr  3 22:37:51 2011
From: wilde at mcs.anl.gov (Michael Wilde)
Date: Sun, 3 Apr 2011 22:37:51 -0500 (CDT)
Subject: [Swift-user] how to terminate swift in the code
In-Reply-To: <53e06f7d.7d55.12f1e60270a.Coremail.tianyu491433909@163.com>
Message-ID: <1743400506.59319.1301888271611.JavaMail.root@zimbra.anl.gov>

Tian, your questions are very hard to understand. Can you clarify:

----- Original Message -----
> in the code, I need to identify the value of the termination of a
> running workflow, how to complete this function

Do you mean you want to pass a return code from the Swift script that the swift command will return as its exit value, so that you can process this in a shell script?

There is no mechanism for doing this currently.

I think the way it works at the moment is that if any error occurs in script execution, then the swift command returns a non-zero exit code, and if no errors occur, it returns zero.

If on the other hand you are talking about Swift function return values within a script, then that is simply determined by the way you code your Swift functions and what you return from them.

- Mike

> 
> 
> _______________________________________________
> Swift-user mailing list
> Swift-user at ci.uchicago.edu
> http://mail.ci.uchicago.edu/mailman/listinfo/swift-user

-- 
Michael Wilde
Computation Institute, University of Chicago
Mathematics and Computer Science Division
Argonne National Laboratory



From tianyu491433909 at 163.com  Mon Apr  4 00:45:04 2011
From: tianyu491433909 at 163.com (tianyu491433909)
Date: Mon, 4 Apr 2011 13:45:04 +0800 (CST)
Subject: [Swift-user] terminate the workflow
Message-ID: <100c49e.7a92.12f1f0afbbb.Coremail.tianyu491433909@163.com>

I'm sorry, I mean Can I terminate the workflow through the code?in addition to System.exit()
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-user/attachments/20110404/02ef2d38/attachment.html>

From wozniak at mcs.anl.gov  Mon Apr  4 10:47:12 2011
From: wozniak at mcs.anl.gov (Justin M Wozniak)
Date: Mon, 4 Apr 2011 10:47:12 -0500 (CDT)
Subject: [Swift-user] terminate the workflow
In-Reply-To: <100c49e.7a92.12f1f0afbbb.Coremail.tianyu491433909@163.com>
References: <100c49e.7a92.12f1f0afbbb.Coremail.tianyu491433909@163.com>
Message-ID: <alpine.DEB.2.00.1104041043100.8575@wozniak-desktop.mcs.anl.gov>

On Mon, 4 Apr 2011, tianyu491433909 wrote:

> I'm sorry, I mean Can I terminate the workflow through the code?in 
> addition to System.exit()

One thing you can try is the new assert() built-in.  assert() takes up to 
two arguments.  The first is either a boolean or an int.  If false or 0, 
it will bring the run down and swift will exit with code 2.  The optional 
second argument is a string that is reported if the assertion fails.

This is an experimental feature.

 	Justin

-- 
Justin M Wozniak

From ketancmaheshwari at gmail.com  Tue Apr  5 10:01:00 2011
From: ketancmaheshwari at gmail.com (Ketan Maheshwari)
Date: Tue, 5 Apr 2011 10:01:00 -0500
Subject: [Swift-user] tui
Message-ID: <898D8E1A-B3AA-4796-A15E-F615346EE464@gmail.com>

Hi,

I too tried the "-tui" feature and stumbled into the same issue of not being able to change tabs that Mike faced:

http://mail.ci.uchicago.edu/pipermail/swift-user/2009-November/001226.html

Any clues?

Ketan

From wilde at mcs.anl.gov  Tue Apr  5 10:14:11 2011
From: wilde at mcs.anl.gov (Michael Wilde)
Date: Tue, 5 Apr 2011 10:14:11 -0500 (CDT)
Subject: [Swift-user] Re: [SWIFT problem]Failed to transfer wrapper log on
	PBS
In-Reply-To: <BANLkTik6M-uiMu=WADxBurzBVPHRpKCAdw@mail.gmail.com>
Message-ID: <1597274208.65436.1302016451838.JavaMail.root@zimbra.anl.gov>

Hi Weiyang,

I'm cc'ing this to swift-user, where you should send all questions, so that other Swift developers and users can offer help as well, and all users can learn from the answers.

Its hard for me to debug this without seeing your .swift script and your log file.

The error message means: Swift tried to run a batch job to execute an app() call, and the attempt did not even return the per-job log file (the "wrapper log") from the execution site (ie what you defined in your pool entry).

First, please comment out the <scratch> tag. Thats an optimization and may be confusing things (or may even be cause of the error, but that is less likely).

The most likely cause of the problem here is that your application "touch" is listed in your tc under the wrong pathname: touch is /bin/touch, not /usr/bin.

The best way to get a sense of how far your script progressed is to do a full "find" under your swiftwork directory from the directory for the failing run id (test-20110405-0940-rqa4nyka), and see what is there.

The shared/ directory should contain whatever input files were processed.  If you did not use the <scratch> tag, then there should also be a "job directory" for each app swift tried to run.  This is described in the user guide.

Since these were not present, I concluded that Swift could not execute your application.

The Swift team is working to improve both these messages and the documentation for debugging such situations to make this much easier to spot.

Also, I dont know how far your workflow ran the previous time, but thus looks like a large run. You should test new workflows (or even any changes you make) on very small runs first, so that there are fewer files and parallel jobs to sort through when debugging new scripts.

- Mike


----- Original Message -----
> Hello,
> 
> 
> My swift codes just encountered new problems: After submitted jobs
> using foreach it's saying
> 
> 
> Failed to transfer wrapper log from test-20110405-0940-rqa4nyka/info/u
> on pbs
> 
> 
> My guess is there're sth wrong with pbs (the execution provider)
> 
> 
> My sites.xml and tc.data is exactly kept the same since last time you
> advised
> 
> 
> sites.xml:
> 
> 
> 
> <config>
> <pool handle="pbs">
> <execution provider="pbs" url="localhost" jobManager="local:pbs"/>
> <profile namespace="globus" key="maxwalltime">1:00:00</profile>
> <profile namespace="globus" key="workersPerNode">8</profile>
> <profile namespace="globus" key="ppn">8</profile>
> <!-- <profile namespace="globus"
> key="internalHostname">172.5.86.6</profile>-->
> <profile namespace="globus" key="nodeGranularity">1</profile>
> <profile namespace="globus" key="maxNodes">1</profile>
> <profile namespace="karajan" key="jobThrottle">1.99</profile>
> <profile namespace="karajan" key="initialScore">10000</profile>
> <profile namespace="globus" key="project">CI-SES000031</profile>
> <!--gridftp url="local://localhost" />-->
> <filesystem provider="local"/>
> <scratch>/home/frankwang/tmp</scratch>
> <workdirectory>/home/frankwang/swiftwork</workdirectory>
> </pool>
> 
> 
> </config>
> 
> pbs echo /bin/echo INSTALLED INTEL32::LINUX null
> pbs sh /bin/bash INSTALLED INTEL32::LINUX null
> pbs touch /usr/bin/touch INSTALLED INTEL32::LINUX
> GLOBUS::maxwalltime="0:1"
> 
> 
> The same problem occured when I was using various version of swift.
> 
> 
> Can you take a time to figure it out?
> 
> 
> Weiyang

-- 
Michael Wilde
Computation Institute, University of Chicago
Mathematics and Computer Science Division
Argonne National Laboratory



From ketancmaheshwari at gmail.com  Tue Apr  5 13:02:48 2011
From: ketancmaheshwari at gmail.com (Ketan Maheshwari)
Date: Tue, 5 Apr 2011 13:02:48 -0500
Subject: [Swift-user] Re: [SWIFT problem]Failed to transfer wrapper log on
	PBS
In-Reply-To: <1597274208.65436.1302016451838.JavaMail.root@zimbra.anl.gov>
References: <1597274208.65436.1302016451838.JavaMail.root@zimbra.anl.gov>
Message-ID: <73E84050-C076-43FC-9F06-2F76D2BCF3F8@gmail.com>

Hi Weiyang,

Are you, by any chance running from a non-lustre filesystem?

I too faced the same error this morning and after several different trials, I isolated the problem to be causing when running Swift on Beagle from non-lustre fileSystem.

The problem disappears on /lustre filesystem. 

This apparently means file transfer commands are failing in PBS either recently or was always the case as I do not remember ever running from non lustre system but today.


Ketan

On Apr 5, 2011, at 10:14 AM, Michael Wilde wrote:

> Hi Weiyang,
> 
> I'm cc'ing this to swift-user, where you should send all questions, so that other Swift developers and users can offer help as well, and all users can learn from the answers.
> 
> Its hard for me to debug this without seeing your .swift script and your log file.
> 
> The error message means: Swift tried to run a batch job to execute an app() call, and the attempt did not even return the per-job log file (the "wrapper log") from the execution site (ie what you defined in your pool entry).
> 
> First, please comment out the <scratch> tag. Thats an optimization and may be confusing things (or may even be cause of the error, but that is less likely).
> 
> The most likely cause of the problem here is that your application "touch" is listed in your tc under the wrong pathname: touch is /bin/touch, not /usr/bin.
> 
> The best way to get a sense of how far your script progressed is to do a full "find" under your swiftwork directory from the directory for the failing run id (test-20110405-0940-rqa4nyka), and see what is there.
> 
> The shared/ directory should contain whatever input files were processed.  If you did not use the <scratch> tag, then there should also be a "job directory" for each app swift tried to run.  This is described in the user guide.
> 
> Since these were not present, I concluded that Swift could not execute your application.
> 
> The Swift team is working to improve both these messages and the documentation for debugging such situations to make this much easier to spot.
> 
> Also, I dont know how far your workflow ran the previous time, but thus looks like a large run. You should test new workflows (or even any changes you make) on very small runs first, so that there are fewer files and parallel jobs to sort through when debugging new scripts.
> 
> - Mike
> 
> 
> ----- Original Message -----
>> Hello,
>> 
>> 
>> My swift codes just encountered new problems: After submitted jobs
>> using foreach it's saying
>> 
>> 
>> Failed to transfer wrapper log from test-20110405-0940-rqa4nyka/info/u
>> on pbs
>> 
>> 
>> My guess is there're sth wrong with pbs (the execution provider)
>> 
>> 
>> My sites.xml and tc.data is exactly kept the same since last time you
>> advised
>> 
>> 
>> sites.xml:
>> 
>> 
>> 
>> <config>
>> <pool handle="pbs">
>> <execution provider="pbs" url="localhost" jobManager="local:pbs"/>
>> <profile namespace="globus" key="maxwalltime">1:00:00</profile>
>> <profile namespace="globus" key="workersPerNode">8</profile>
>> <profile namespace="globus" key="ppn">8</profile>
>> <!-- <profile namespace="globus"
>> key="internalHostname">172.5.86.6</profile>-->
>> <profile namespace="globus" key="nodeGranularity">1</profile>
>> <profile namespace="globus" key="maxNodes">1</profile>
>> <profile namespace="karajan" key="jobThrottle">1.99</profile>
>> <profile namespace="karajan" key="initialScore">10000</profile>
>> <profile namespace="globus" key="project">CI-SES000031</profile>
>> <!--gridftp url="local://localhost" />-->
>> <filesystem provider="local"/>
>> <scratch>/home/frankwang/tmp</scratch>
>> <workdirectory>/home/frankwang/swiftwork</workdirectory>
>> </pool>
>> 
>> 
>> </config>
>> 
>> pbs echo /bin/echo INSTALLED INTEL32::LINUX null
>> pbs sh /bin/bash INSTALLED INTEL32::LINUX null
>> pbs touch /usr/bin/touch INSTALLED INTEL32::LINUX
>> GLOBUS::maxwalltime="0:1"
>> 
>> 
>> The same problem occured when I was using various version of swift.
>> 
>> 
>> Can you take a time to figure it out?
>> 
>> 
>> Weiyang
> 
> -- 
> Michael Wilde
> Computation Institute, University of Chicago
> Mathematics and Computer Science Division
> Argonne National Laboratory
> 
> _______________________________________________
> Swift-user mailing list
> Swift-user at ci.uchicago.edu
> http://mail.ci.uchicago.edu/mailman/listinfo/swift-user



From jon.monette at gmail.com  Tue Apr  5 18:43:24 2011
From: jon.monette at gmail.com (Jonathan Monette)
Date: Tue, 5 Apr 2011 18:43:24 -0500
Subject: [Swift-user] job name coaster
Message-ID: <BANLkTimgufwWA6XzYFTCsLsEtf6kEY=Myg@mail.gmail.com>

Hello,
   Can you give a PBS job a name in coasters?  Something like <profile
key="jobname" namespace="globus">MY_JOB_NAME</profile> and MY_JOB_NAME will
show up as job name under qstat?

-- 
Any intelligent fool can make things bigger and more complex... It takes a
touch of genius - and a lot of courage to move in the opposite direction.
- Albert Einstein
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-user/attachments/20110405/77fe03ee/attachment.html>

From wozniak at mcs.anl.gov  Tue Apr  5 20:38:52 2011
From: wozniak at mcs.anl.gov (Justin M Wozniak)
Date: Tue, 5 Apr 2011 20:38:52 -0500 (Central Daylight Time)
Subject: [Swift-user] job name coaster
In-Reply-To: <BANLkTimgufwWA6XzYFTCsLsEtf6kEY=Myg@mail.gmail.com>
References: <BANLkTimgufwWA6XzYFTCsLsEtf6kEY=Myg@mail.gmail.com>
Message-ID: <alpine.WNT.2.00.1104052038170.5524@JWOZNIAK-DESK>

On Tue, 5 Apr 2011, Jonathan Monette wrote:

> Hello,
>   Can you give a PBS job a name in coasters?  Something like <profile
> key="jobname" namespace="globus">MY_JOB_NAME</profile> and MY_JOB_NAME will
> show up as job name under qstat?

I think this would require a minor new feature- I'll add it to my list.
 	Justin

-- 
Justin M Wozniak


From jon.monette at gmail.com  Tue Apr  5 20:40:14 2011
From: jon.monette at gmail.com (Jonathan Monette)
Date: Tue, 5 Apr 2011 20:40:14 -0500
Subject: [Swift-user] job name coaster
In-Reply-To: <alpine.WNT.2.00.1104052038170.5524@JWOZNIAK-DESK>
References: <BANLkTimgufwWA6XzYFTCsLsEtf6kEY=Myg@mail.gmail.com>
	<alpine.WNT.2.00.1104052038170.5524@JWOZNIAK-DESK>
Message-ID: <BANLkTiniO2A+X1V9yxQjmB1rwDWFE8RMdw@mail.gmail.com>

Ok.  Was just wondering.  This isn't anything that needs to be done quickly.

On Tue, Apr 5, 2011 at 8:38 PM, Justin M Wozniak <wozniak at mcs.anl.gov>wrote:

> On Tue, 5 Apr 2011, Jonathan Monette wrote:
>
>  Hello,
>>  Can you give a PBS job a name in coasters?  Something like <profile
>> key="jobname" namespace="globus">MY_JOB_NAME</profile> and MY_JOB_NAME
>> will
>> show up as job name under qstat?
>>
>
> I think this would require a minor new feature- I'll add it to my list.
>        Justin
>
> --
> Justin M Wozniak
>



-- 
Any intelligent fool can make things bigger and more complex... It takes a
touch of genius - and a lot of courage to move in the opposite direction.
- Albert Einstein
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-user/attachments/20110405/6c92e68c/attachment.html>

From wilde at mcs.anl.gov  Tue Apr  5 22:03:48 2011
From: wilde at mcs.anl.gov (Michael Wilde)
Date: Tue, 5 Apr 2011 22:03:48 -0500 (CDT)
Subject: [Swift-user] job name coaster
In-Reply-To: <BANLkTiniO2A+X1V9yxQjmB1rwDWFE8RMdw@mail.gmail.com>
Message-ID: <1692591919.68869.1302059028571.JavaMail.root@zimbra.anl.gov>

Sarah, I see that you were working on some aspects of job naming back in October: 


http://mail.ci.uchicago.edu/pipermail/swift-devel/2010-October/006851.html 


Was this change committed, and if so how do you use it? The thread above seems to discuss jobnames for GRAM. Does it work for GRAM only, or other provider combinations? 


- Mike 

----- Original Message -----


Ok. Was just wondering. This isn't anything that needs to be done quickly. 


On Tue, Apr 5, 2011 at 8:38 PM, Justin M Wozniak < wozniak at mcs.anl.gov > wrote: 



On Tue, 5 Apr 2011, Jonathan Monette wrote: 



Hello, 
Can you give a PBS job a name in coasters? Something like <profile 
key="jobname" namespace="globus">MY_JOB_NAME</profile> and MY_JOB_NAME will 
show up as job name under qstat? 

I think this would require a minor new feature- I'll add it to my list. 
Justin 

-- 
Justin M Wozniak 



-- 
Any intelligent fool can make things bigger and more complex... It takes a touch of genius - and a lot of courage to move in the opposite direction. 
- Albert Einstein 



_______________________________________________ 
Swift-user mailing list 
Swift-user at ci.uchicago.edu 
http://mail.ci.uchicago.edu/mailman/listinfo/swift-user 


-- 
Michael Wilde 
Computation Institute, University of Chicago 
Mathematics and Computer Science Division 
Argonne National Laboratory 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-user/attachments/20110405/b5ad4741/attachment.html>

From jon.monette at gmail.com  Thu Apr  7 16:33:46 2011
From: jon.monette at gmail.com (Jonathan Monette)
Date: Thu, 7 Apr 2011 16:33:46 -0500
Subject: [Swift-user] Staging problem
Message-ID: <BANLkTikpDTChuZ8seQ+Wn1AXAXxjvwhaSw@mail.gmail.com>

Hello,
   I have a staging question. I have a metadata file that I read out of and
do some mapping. I then run an app on this mapping. However some of the
metadata is false occasionally. Because of this my app doesn't create the
outfile and swift errors because there is no outfile to stage out. How do I
tell swift that if there is no outfile, don't worry about staging out?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-user/attachments/20110407/e7fcb058/attachment.html>

From wilde at mcs.anl.gov  Thu Apr  7 17:03:01 2011
From: wilde at mcs.anl.gov (Michael Wilde)
Date: Thu, 7 Apr 2011 17:03:01 -0500 (CDT)
Subject: [Swift-user] Staging problem
In-Reply-To: <BANLkTikpDTChuZ8seQ+Wn1AXAXxjvwhaSw@mail.gmail.com>
Message-ID: <584065657.80246.1302213781254.JavaMail.root@zimbra.anl.gov>

Jon, 


The standard approach for handling apps that conditionally return a required data file is to run them through a wrapper script (typically a tiny shell script) which ensures that the conditional file is *always* created, even if its empty in cases where the real app did not create it. 


- Mike 

----- Original Message -----




Hello, 
I have a staging question. I have a metadata file that I read out of and do some mapping. I then run an app on this mapping. However some of the metadata is false occasionally. Because of this my app doesn't create the outfile and swift errors because there is no outfile to stage out. How do I tell swift that if there is no outfile, don't worry about staging out? 
_______________________________________________ 
Swift-user mailing list 
Swift-user at ci.uchicago.edu 
http://mail.ci.uchicago.edu/mailman/listinfo/swift-user 


-- 
Michael Wilde 
Computation Institute, University of Chicago 
Mathematics and Computer Science Division 
Argonne National Laboratory 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-user/attachments/20110407/c1eee1a2/attachment.html>

From aespinosa at cs.uchicago.edu  Fri Apr  8 16:00:54 2011
From: aespinosa at cs.uchicago.edu (Allan Espinosa)
Date: Fri, 8 Apr 2011 16:00:54 -0500
Subject: [Swift-user] how do I diagnose long file staging states?
Message-ID: <BANLkTim9tBKn8Ax12HHdLZAJStUw7+YJFA@mail.gmail.com>

Hi,

In the two plots attached, my workflow seem to hang staging files
after 5 hours.  At this point, all the jobs in these states fill the
foreach.maxthreads or throttle.job.factor of my pool of resources.
Where should I poke a more detailed log? On the data transfer
scheduler?

All staging operations are made with third party gridtp transfers.
(stage in from pads to OSG + pads and stage-out in the opposite
direction)

I do have anecdotal experience that sometimes after 2-3 days , the
transfers magically resume (or fail if the proxy certificates expire
in that period)

-- 
Allan M. Espinosa <http://amespinosa.wordpress.com>
PhD student, Computer Science
University of Chicago <http://people.cs.uchicago.edu/~aespinosa>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: dostagein.png
Type: image/png
Size: 4835 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/swift-user/attachments/20110408/ff8f3e0c/attachment.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: dostageout.png
Type: image/png
Size: 4848 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/swift-user/attachments/20110408/ff8f3e0c/attachment-0001.png>

From aespinosa at cs.uchicago.edu  Tue Apr 12 19:45:05 2011
From: aespinosa at cs.uchicago.edu (Allan Espinosa)
Date: Tue, 12 Apr 2011 19:45:05 -0500
Subject: logging file transfer status (was Re: [Swift-user] throttle transfers
	and vdl:stagein graphs)
Message-ID: <BANLkTik6OWxHr-D4iojt-fgXdJJqsaUC8w@mail.gmail.com>

Hi Mihael,

Bumping the thread.

What's the name of the specific class? I tried
log4j.logger.org.globus.cog.abstraction.tools.transfer.FileTransfer=DEBUG

in my log4j.properties file and I didn't see any transfer log messages
that occurred during the stagein/stageout period of my workflows.

Where is the scheduling of transfer made ?  VDSAdaptiveScheduler
doesn't seem to have entries on those transfer policies set in
throttle.transfers and throttle.file.operations.

Thanks,
-Allan

2010/11/18 Mihael Hategan <hategan at mcs.anl.gov>:
> On Thu, 2010-11-18 at 18:54 -0600, Allan Espinosa wrote:
>> Ah. ?I see that the the log entries that is nearest the actual
>> task:transfer() task is in vdl:dostageinfile (no parallelFor loop
>> here). ?But i still see the transfers being more than the throttle in
>> swift.properties.
>>
>> One of the classes invoked by task:transfer() is
>> /org/globus/cog/karajan/workflow/nodes/grid/GridTransfer.class ?
>>
>> I'll try adding this to my log4j.properties and see what will happen.
>
> The transfer task status is the best choice. That's in the abstractions
> module.
>
> Mihael
>
>
>
>



-- 
Allan M. Espinosa <http://amespinosa.wordpress.com>
PhD student, Computer Science
University of Chicago <http://people.cs.uchicago.edu/~aespinosa>


From aespinosa at cs.uchicago.edu  Tue Apr 12 19:49:36 2011
From: aespinosa at cs.uchicago.edu (Allan Espinosa)
Date: Tue, 12 Apr 2011 19:49:36 -0500
Subject: [Swift-user] guc-like log messages on transfers
Message-ID: <BANLkTinex72PGruWh04t7YZzeXPB9-QC2A@mail.gmail.com>

Hi,


Just to elaborate the my question in a bumped thread, How can I get
"globus-url-copy -vb (or -dbg)"-like messages on the data stagings
made in the workflow?

-- 
Allan M. Espinosa <http://amespinosa.wordpress.com>
PhD student, Computer Science
University of Chicago <http://people.cs.uchicago.edu/~aespinosa>


From hategan at mcs.anl.gov  Tue Apr 12 20:04:55 2011
From: hategan at mcs.anl.gov (Mihael Hategan)
Date: Tue, 12 Apr 2011 18:04:55 -0700
Subject: logging file transfer status (was Re: [Swift-user] throttle
	transfers and vdl:stagein graphs)
In-Reply-To: <BANLkTik6OWxHr-D4iojt-fgXdJJqsaUC8w@mail.gmail.com>
References: <BANLkTik6OWxHr-D4iojt-fgXdJJqsaUC8w@mail.gmail.com>
Message-ID: <1302656695.26076.0.camel@blabla2.none>

On Tue, 2011-04-12 at 19:45 -0500, Allan Espinosa wrote:
> Hi Mihael,
> 
> Bumping the thread.
> 
> What's the name of the specific class? I tried
> log4j.logger.org.globus.cog.abstraction.tools.transfer.FileTransfer=DEBUG

I'd try log4j.logger.org.globus.cog.abstraction=DEBUG and then see which
ones I like.





From hategan at mcs.anl.gov  Tue Apr 12 20:05:52 2011
From: hategan at mcs.anl.gov (Mihael Hategan)
Date: Tue, 12 Apr 2011 18:05:52 -0700
Subject: [Swift-user] guc-like log messages on transfers
In-Reply-To: <BANLkTinex72PGruWh04t7YZzeXPB9-QC2A@mail.gmail.com>
References: <BANLkTinex72PGruWh04t7YZzeXPB9-QC2A@mail.gmail.com>
Message-ID: <1302656752.26076.1.camel@blabla2.none>

On Tue, 2011-04-12 at 19:49 -0500, Allan Espinosa wrote:
> Hi,
> 
> 
> Just to elaborate the my question in a bumped thread, How can I get
> "globus-url-copy -vb (or -dbg)"-like messages on the data stagings
> made in the workflow?
> 

I don't have one of those handy. Can you be specific about what events
you'd like to see?



From aespinosa at cs.uchicago.edu  Wed Apr 13 14:48:41 2011
From: aespinosa at cs.uchicago.edu (Allan Espinosa)
Date: Wed, 13 Apr 2011 14:48:41 -0500
Subject: [Swift-user] guc-like log messages on transfers
In-Reply-To: <1302656752.26076.1.camel@blabla2.none>
References: <BANLkTinex72PGruWh04t7YZzeXPB9-QC2A@mail.gmail.com>
	<1302656752.26076.1.camel@blabla2.none>
Message-ID: <BANLkTi=1MtnmoksBqF1ri=WH4CeShx4+Vg@mail.gmail.com>

"-dbg" gives update on a transfer per buffer(?):
debug: response from
gsiftp://top.ucr.edu/data/down/osg_data/engage/scec/SgtFiles/TEST/TEST_fx_644.sgt:
111 Range Marker
386662400-386924544,387448832-387973120,388235264-622329856,622854144-623378432,623902720-624427008

debug: response from
gsiftp://top.ucr.edu/data/down/osg_data/engage/scec/SgtFiles/TEST/TEST_fx_644.sgt:
112-Perf Marker
 Timestamp:  1302723888.6
 Stripe Index: 0
 Stripe Bytes Transferred: 877658112
 Total Stripe Count: 1
112 End.


"-vb" gives an instantaenous measure of bandwidth.  perhaps it is
based from the debug message above:
Source: gsiftp://gridftp.pads.ci.uchicago.edu/gpfs/pads/swift/aespinosa/science/cybershake/RuptureVariations/100/1/
Dest:   gsiftp://ff-grid3.unl.edu/panfs/panasas/CMS/data/engage/scec/swift_scratch/guc-tests/
  100_1.txt.variation-s0000-h0005

      1304838 bytes         0.62 MB/sec avg         0.62 MB/sec inst
Source: gsiftp://gridftp.pads.ci.uchicago.edu/gpfs/pads/swift/aespinosa/science/cybershake/RuptureVariations/100/1/
Dest:   gsiftp://ff-grid3.unl.edu/panfs/panasas/CMS/data/engage/scec/swift_scratch/guc-tests/
  100_1.txt.variation-s0000-h0002

      1304838 bytes         0.44 MB/sec avg         0.44 MB/sec inst
Source: gsiftp://gridftp.pads.ci.uchicago.edu/gpfs/pads/swift/aespinosa/science/cybershake/RuptureVariations/100/1/
Dest:   gsiftp://ff-grid3.unl.edu/panfs/panasas/CMS/data/engage/scec/swift_scratch/guc-tests/
  100_1.txt.variation-s0000-h0000

      1304838 bytes         0.30 MB/sec avg         0.30 MB/sec inst
Source: gsiftp://gridftp.pads.ci.uchicago.edu/gpfs/pads/swift/aespinosa/science/cybershake/RuptureVariations/100/1/
Dest:   gsiftp://ff-grid3.unl.edu/panfs/panasas/CMS/data/engage/scec/swift_scratch/guc-tests/
  100_1.txt.variation-s0001-h0000

      1304838 bytes         0.39 MB/sec avg         0.39 MB/sec inst


>From GlobusOnline, it is somehow equivalent to the 'events' output:

$ globus help events
Usage: events [options] <taskid>

Display start, stop, performance, and error events for a task.
Type 'man events' for details.

Options:
  -h, --help        show this help message and exit
  -l LIMIT          Limit the number of results.  Default: Unlimited

  Output Formatting:
    -f FIELD[,...]  Select field(s) to display. -f 'all' selects all. -f
                    'help' displays all fields.
    -O FORMAT       Change output format: 'csv', 'csvh', 'kv'

Sample that I dumped:
$globus events 710dab8c-6539-11e0-bfc5-1231390f14a1
Task ID       : 710dab8c-6539-11e0-bfc5-1231390f14a1
Parent Task ID: 70e52342-6539-11e0-bfc5-1231390f14a1
Time          : 2011-04-13 15:07:21.965616Z
Code          : PROGRESS
Description   : Performance monitoring event
Details       : bytes=16777216 mbps=2.237

Task ID       : 7135589e-6539-11e0-bfc5-1231390f14a1
Parent Task ID: 70e52342-6539-11e0-bfc5-1231390f14a1
Time          : 2011-04-13 15:07:22.297131Z
Code          : PROGRESS
Description   : Performance monitoring event
Details       : bytes=22806528 mbps=3.041



2011/4/12 Mihael Hategan <hategan at mcs.anl.gov>:
> On Tue, 2011-04-12 at 19:49 -0500, Allan Espinosa wrote:
>> Hi,
>>
>>
>> Just to elaborate the my question in a bumped thread, How can I get
>> "globus-url-copy -vb (or -dbg)"-like messages on the data stagings
>> made in the workflow?
>>
>
> I don't have one of those handy. Can you be specific about what events
> you'd like to see?
>
>
>



-- 
Allan M. Espinosa <http://amespinosa.wordpress.com>
PhD student, Computer Science
University of Chicago <http://people.cs.uchicago.edu/~aespinosa>


From aespinosa at cs.uchicago.edu  Wed Apr 13 14:55:25 2011
From: aespinosa at cs.uchicago.edu (Allan Espinosa)
Date: Wed, 13 Apr 2011 14:55:25 -0500
Subject: logging file transfer status (was Re: [Swift-user] throttle
	transfers and vdl:stagein graphs)
In-Reply-To: <1302656695.26076.0.camel@blabla2.none>
References: <BANLkTik6OWxHr-D4iojt-fgXdJJqsaUC8w@mail.gmail.com>
	<1302656695.26076.0.camel@blabla2.none>
Message-ID: <BANLkTikcZo17gdN6VsB0Fh944x9X8WEM4A@mail.gmail.com>

This upper package doesn't seem to contain much information in terms
of the status of the transfer tasks (see my "guc-log thread" on the
type of information I need):

2011-04-13 14:04:51,028-0500 DEBUG vdl:dostageinfile
FILE_STAGE_IN_START file=TEST_fy_644.sgt
srchost=gridftp.ranger.tacc.teragrid.org
srcdir=/work/01035/tg802895/science/cybershake/data/SgtFiles/TEST
srcname=TEST_fy_644.sgt desthost=PADS
destdir=postproc-20110413-1404-yup45h9b/shared/work/01035/tg802895/science/cybershake/data/SgtFiles/TEST
provider=gsiftp policy=DEFAULT
2011-04-13 14:04:51,030-0500 DEBUG FileResourceCache Got request for
resource for gridftp.ranger.tacc.teragrid.org(gsiftp)
2011-04-13 14:04:51,030-0500 DEBUG FileResourceCache Found cached
resource (null)
2011-04-13 14:04:51,031-0500 DEBUG WeightedHostScoreScheduler
multiplyScore(PADS:10,001.000(99.854):3/366 overload: 0, -0.2)
...
...
...
2011-04-13 14:04:53,484-0500 DEBUG vdl:dostageinfile FILE_STAGE_IN_END
file=265_2.txt.variation-s0001-h0000
srchost=gridftp.ranger.tacc.teragrid.org
srcdir=/scratch/projects/tg/tera3d/CyberShake2007/ruptures/RuptureVariations_35_V2_3/265/2
srcname=265_2.txt.variation-s0001-h0000 desthost=PADS
destdir=postproc-20110413-1404-yup45h9b/shared/scratch/projects/tg/tera3d/CyberShake2007/ruptures/RuptureVariations_35_V2_3/265/2
provider=gsiftp
2011-04-13 14:04:53,485-0500 DEBUG AbstractionFactory Instantiating
org.globus.cog.abstraction.impl.file.local.FileResourceImpl for
provider local
2011-04-13 14:04:53,485-0500 DEBUG DelegatedFileTransferHandler
Directory transfer with resource local->local


-Allan

2011/4/12 Mihael Hategan <hategan at mcs.anl.gov>:
> On Tue, 2011-04-12 at 19:45 -0500, Allan Espinosa wrote:
>> Hi Mihael,
>>
>> Bumping the thread.
>>
>> What's the name of the specific class? I tried
>> log4j.logger.org.globus.cog.abstraction.tools.transfer.FileTransfer=DEBUG
>
> I'd try log4j.logger.org.globus.cog.abstraction=DEBUG and then see which
> ones I like.
>
>
>


From hategan at mcs.anl.gov  Wed Apr 13 15:15:52 2011
From: hategan at mcs.anl.gov (Mihael Hategan)
Date: Wed, 13 Apr 2011 13:15:52 -0700
Subject: [Swift-user] guc-like log messages on transfers
In-Reply-To: <BANLkTi=1MtnmoksBqF1ri=WH4CeShx4+Vg@mail.gmail.com>
References: <BANLkTinex72PGruWh04t7YZzeXPB9-QC2A@mail.gmail.com>
	<1302656752.26076.1.camel@blabla2.none>
	<BANLkTi=1MtnmoksBqF1ri=WH4CeShx4+Vg@mail.gmail.com>
Message-ID: <1302725752.29432.4.camel@blabla2.none>

If you are looking for transfer progress, then that may not be there.

You can try enabling debug on the jglobus gridftp stuff
(org.globus.ftp/gridftp). But I'm not sure it will contain what you
want. We can talk about implementing something like this.

On Wed, 2011-04-13 at 14:48 -0500, Allan Espinosa wrote:
> "-dbg" gives update on a transfer per buffer(?):
> debug: response from
> gsiftp://top.ucr.edu/data/down/osg_data/engage/scec/SgtFiles/TEST/TEST_fx_644.sgt:
> 111 Range Marker
> 386662400-386924544,387448832-387973120,388235264-622329856,622854144-623378432,623902720-624427008
> 
> debug: response from
> gsiftp://top.ucr.edu/data/down/osg_data/engage/scec/SgtFiles/TEST/TEST_fx_644.sgt:
> 112-Perf Marker
>  Timestamp:  1302723888.6
>  Stripe Index: 0
>  Stripe Bytes Transferred: 877658112
>  Total Stripe Count: 1
> 112 End.
> 
> 
> "-vb" gives an instantaenous measure of bandwidth.  perhaps it is
> based from the debug message above:
> Source: gsiftp://gridftp.pads.ci.uchicago.edu/gpfs/pads/swift/aespinosa/science/cybershake/RuptureVariations/100/1/
> Dest:   gsiftp://ff-grid3.unl.edu/panfs/panasas/CMS/data/engage/scec/swift_scratch/guc-tests/
>   100_1.txt.variation-s0000-h0005
> 
>       1304838 bytes         0.62 MB/sec avg         0.62 MB/sec inst
> Source: gsiftp://gridftp.pads.ci.uchicago.edu/gpfs/pads/swift/aespinosa/science/cybershake/RuptureVariations/100/1/
> Dest:   gsiftp://ff-grid3.unl.edu/panfs/panasas/CMS/data/engage/scec/swift_scratch/guc-tests/
>   100_1.txt.variation-s0000-h0002
> 
>       1304838 bytes         0.44 MB/sec avg         0.44 MB/sec inst
> Source: gsiftp://gridftp.pads.ci.uchicago.edu/gpfs/pads/swift/aespinosa/science/cybershake/RuptureVariations/100/1/
> Dest:   gsiftp://ff-grid3.unl.edu/panfs/panasas/CMS/data/engage/scec/swift_scratch/guc-tests/
>   100_1.txt.variation-s0000-h0000
> 
>       1304838 bytes         0.30 MB/sec avg         0.30 MB/sec inst
> Source: gsiftp://gridftp.pads.ci.uchicago.edu/gpfs/pads/swift/aespinosa/science/cybershake/RuptureVariations/100/1/
> Dest:   gsiftp://ff-grid3.unl.edu/panfs/panasas/CMS/data/engage/scec/swift_scratch/guc-tests/
>   100_1.txt.variation-s0001-h0000
> 
>       1304838 bytes         0.39 MB/sec avg         0.39 MB/sec inst
> 
> 
> From GlobusOnline, it is somehow equivalent to the 'events' output:
> 
> $ globus help events
> Usage: events [options] <taskid>
> 
> Display start, stop, performance, and error events for a task.
> Type 'man events' for details.
> 
> Options:
>   -h, --help        show this help message and exit
>   -l LIMIT          Limit the number of results.  Default: Unlimited
> 
>   Output Formatting:
>     -f FIELD[,...]  Select field(s) to display. -f 'all' selects all. -f
>                     'help' displays all fields.
>     -O FORMAT       Change output format: 'csv', 'csvh', 'kv'
> 
> Sample that I dumped:
> $globus events 710dab8c-6539-11e0-bfc5-1231390f14a1
> Task ID       : 710dab8c-6539-11e0-bfc5-1231390f14a1
> Parent Task ID: 70e52342-6539-11e0-bfc5-1231390f14a1
> Time          : 2011-04-13 15:07:21.965616Z
> Code          : PROGRESS
> Description   : Performance monitoring event
> Details       : bytes=16777216 mbps=2.237
> 
> Task ID       : 7135589e-6539-11e0-bfc5-1231390f14a1
> Parent Task ID: 70e52342-6539-11e0-bfc5-1231390f14a1
> Time          : 2011-04-13 15:07:22.297131Z
> Code          : PROGRESS
> Description   : Performance monitoring event
> Details       : bytes=22806528 mbps=3.041
> 
> 
> 
> 2011/4/12 Mihael Hategan <hategan at mcs.anl.gov>:
> > On Tue, 2011-04-12 at 19:49 -0500, Allan Espinosa wrote:
> >> Hi,
> >>
> >>
> >> Just to elaborate the my question in a bumped thread, How can I get
> >> "globus-url-copy -vb (or -dbg)"-like messages on the data stagings
> >> made in the workflow?
> >>
> >
> > I don't have one of those handy. Can you be specific about what events
> > you'd like to see?
> >
> >
> >
> 
> 
> 




From skenny at uchicago.edu  Wed Apr 13 17:14:39 2011
From: skenny at uchicago.edu (Sarah Kenny)
Date: Wed, 13 Apr 2011 15:14:39 -0700
Subject: [Swift-user] Swift Release 0.92.1 ready for use
Message-ID: <BANLkTinMY9qohbiandpjT8C9zjn1+-8UZg@mail.gmail.com>

Hi all,

We are happy to announce that the latest version of Swift (containing a fix
for the recently-discovered bug in version 0.92) is now available for
download here: http://www.ci.uchicago.edu/swift/downloads/index.php

Enjoy :)

-The Swift Development Team
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-user/attachments/20110413/b4d989b5/attachment.html>

From mickelso at mcs.anl.gov  Fri Apr 15 15:07:58 2011
From: mickelso at mcs.anl.gov (Sheri Mickelson)
Date: Fri, 15 Apr 2011 15:07:58 -0500
Subject: [Swift-user] Passing an array to swift?
Message-ID: <4DA8A59E.9000301@mcs.anl.gov>

Hi,

Is it possible to pass a c-shell array to swift?  And if so, how do you do it?

Thanks, Sheri


From wozniak at mcs.anl.gov  Fri Apr 15 15:25:29 2011
From: wozniak at mcs.anl.gov (Justin M Wozniak)
Date: Fri, 15 Apr 2011 15:25:29 -0500 (CDT)
Subject: [Swift-user] Passing an array to swift?
In-Reply-To: <4DA8A59E.9000301@mcs.anl.gov>
References: <4DA8A59E.9000301@mcs.anl.gov>
Message-ID: <alpine.DEB.2.00.1104151524010.1751@wozniak-desktop.mcs.anl.gov>

Hello
 	Not directly because the interface is the command-line.  Can you 
give us more of a use case?
 	Justin

On Fri, 15 Apr 2011, Sheri Mickelson wrote:

> Hi,
>
> Is it possible to pass a c-shell array to swift?  And if so, how do you 
> do it?
>
> Thanks, Sheri
> _______________________________________________
> Swift-user mailing list
> Swift-user at ci.uchicago.edu
> http://mail.ci.uchicago.edu/mailman/listinfo/swift-user
>

-- 
Justin M Wozniak


From hategan at mcs.anl.gov  Fri Apr 15 15:37:47 2011
From: hategan at mcs.anl.gov (Mihael Hategan)
Date: Fri, 15 Apr 2011 13:37:47 -0700
Subject: [Swift-user] Passing an array to swift?
In-Reply-To: <4DA8A59E.9000301@mcs.anl.gov>
References: <4DA8A59E.9000301@mcs.anl.gov>
Message-ID: <1302899867.26254.0.camel@blabla2.none>

As Justin says, not directly.

But you can write the array to a file and read in in swift using
readData (or readData2).

Mihael

On Fri, 2011-04-15 at 15:07 -0500, Sheri Mickelson wrote:
> Hi,
> 
> Is it possible to pass a c-shell array to swift?  And if so, how do you do it?
> 
> Thanks, Sheri
> _______________________________________________
> Swift-user mailing list
> Swift-user at ci.uchicago.edu
> http://mail.ci.uchicago.edu/mailman/listinfo/swift-user




From wilde at mcs.anl.gov  Fri Apr 15 16:20:41 2011
From: wilde at mcs.anl.gov (Michael Wilde)
Date: Fri, 15 Apr 2011 16:20:41 -0500 (CDT)
Subject: [Swift-user] Passing an array to swift?
In-Reply-To: <alpine.DEB.2.00.1104151524010.1751@wozniak-desktop.mcs.anl.gov>
Message-ID: <1161128172.108884.1302902441023.JavaMail.root@zimbra.anl.gov>

Maybe direct the array into a file (possibly with a "here document" which expands the array) and then read the file in Swift with readData() or process it with a Swift app() function?

- Mike


----- Original Message -----
> Hello
> Not directly because the interface is the command-line. Can you
> give us more of a use case?
> Justin
> 
> On Fri, 15 Apr 2011, Sheri Mickelson wrote:
> 
> > Hi,
> >
> > Is it possible to pass a c-shell array to swift? And if so, how do
> > you
> > do it?
> >
> > Thanks, Sheri
> > _______________________________________________
> > Swift-user mailing list
> > Swift-user at ci.uchicago.edu
> > http://mail.ci.uchicago.edu/mailman/listinfo/swift-user
> >
> 
> --
> Justin M Wozniak
> _______________________________________________
> Swift-user mailing list
> Swift-user at ci.uchicago.edu
> http://mail.ci.uchicago.edu/mailman/listinfo/swift-user

-- 
Michael Wilde
Computation Institute, University of Chicago
Mathematics and Computer Science Division
Argonne National Laboratory



From mickelso at mcs.anl.gov  Fri Apr 15 16:30:06 2011
From: mickelso at mcs.anl.gov (Sheri Mickelson)
Date: Fri, 15 Apr 2011 16:30:06 -0500
Subject: [Swift-user] Passing an array to swift?
In-Reply-To: <1161128172.108884.1302902441023.JavaMail.root@zimbra.anl.gov>
References: <1161128172.108884.1302902441023.JavaMail.root@zimbra.anl.gov>
Message-ID: <4DA8B8DE.1040804@mcs.anl.gov>

I have two arrays I need to read in - a long one and a short one.  I'll probably put the long one 
into a file and read it in with readData.  The smaller one will, at most, only have 4 components so 
I'll probably just pass it in as a string and use strsplit.

Thanks, Sheri

Michael Wilde wrote:
> Maybe direct the array into a file (possibly with a "here document" which expands the array) and then read the file in Swift with readData() or process it with a Swift app() function?
> 
> - Mike
> 
> 
> ----- Original Message -----
>> Hello
>> Not directly because the interface is the command-line. Can you
>> give us more of a use case?
>> Justin
>>
>> On Fri, 15 Apr 2011, Sheri Mickelson wrote:
>>
>>> Hi,
>>>
>>> Is it possible to pass a c-shell array to swift? And if so, how do
>>> you
>>> do it?
>>>
>>> Thanks, Sheri
>>> _______________________________________________
>>> Swift-user mailing list
>>> Swift-user at ci.uchicago.edu
>>> http://mail.ci.uchicago.edu/mailman/listinfo/swift-user
>>>
>> --
>> Justin M Wozniak
>> _______________________________________________
>> Swift-user mailing list
>> Swift-user at ci.uchicago.edu
>> http://mail.ci.uchicago.edu/mailman/listinfo/swift-user
> 


From hockyg at gmail.com  Fri Apr 15 16:31:25 2011
From: hockyg at gmail.com (Glen Hocky)
Date: Fri, 15 Apr 2011 17:31:25 -0400
Subject: [Swift-user] Passing an array to swift?
In-Reply-To: <1161128172.108884.1302902441023.JavaMail.root@zimbra.anl.gov>
References: <1161128172.108884.1302902441023.JavaMail.root@zimbra.anl.gov>
Message-ID: <5192034991988947702@unknownmsgid>

I've done what Mike suggested but you can also use @strsplit on a
comma separated command line arg and that works well for me.

Glen

On Apr 15, 2011, at 5:20 PM, Michael Wilde <wilde at mcs.anl.gov> wrote:

> Maybe direct the array into a file (possibly with a "here document" which expands the array) and then read the file in Swift with readData() or process it with a Swift app() function?
>
> - Mike
>
>
> ----- Original Message -----
>> Hello
>> Not directly because the interface is the command-line. Can you
>> give us more of a use case?
>> Justin
>>
>> On Fri, 15 Apr 2011, Sheri Mickelson wrote:
>>
>>> Hi,
>>>
>>> Is it possible to pass a c-shell array to swift? And if so, how do
>>> you
>>> do it?
>>>
>>> Thanks, Sheri
>>> _______________________________________________
>>> Swift-user mailing list
>>> Swift-user at ci.uchicago.edu
>>> http://mail.ci.uchicago.edu/mailman/listinfo/swift-user
>>>
>>
>> --
>> Justin M Wozniak
>> _______________________________________________
>> Swift-user mailing list
>> Swift-user at ci.uchicago.edu
>> http://mail.ci.uchicago.edu/mailman/listinfo/swift-user
>
> --
> Michael Wilde
> Computation Institute, University of Chicago
> Mathematics and Computer Science Division
> Argonne National Laboratory
>
> _______________________________________________
> Swift-user mailing list
> Swift-user at ci.uchicago.edu
> http://mail.ci.uchicago.edu/mailman/listinfo/swift-user


From aespinosa at cs.uchicago.edu  Mon Apr 18 13:42:16 2011
From: aespinosa at cs.uchicago.edu (Allan Espinosa)
Date: Mon, 18 Apr 2011 13:42:16 -0500
Subject: [Swift-user] condor_requirements "&&" operator and other possible
	literal tokens
Message-ID: <BANLkTikGkwm3wbxdUvrepj01ji3kbX64Sw@mail.gmail.com>

Hi,

I have the following entry in my site catalog:

  <pool handle="FNAL_FERMIGRID__fermigridosg1.fnal.gov">
    <execution provider="condor" url="none"/>

    <profile namespace="globus" key="jobType">grid</profile>
    <profile namespace="globus" key="gridResource">gt2
fermigridosg1.fnal.gov/jobmanager-condor</profile>

    <profile namespace="karajan" key="initialScore">20.0</profile>
    <profile namespace="karajan" key="jobThrottle">68.22</profile>

      <profile namespace="globus"
key="condor_requirements">GlueHostOperatingSystemRelease =?= "5.3" &&
GlueSubClusterName =!= GlueClusterName</profile>


    <gridftp  url="gsiftp://fermigridosg1.fnal.gov"/>
    <workdirectory>/grid/data/engage/scec/swift_scratch</workdirectory>
  </pool>




But Swift reports the following

Execution failed:
        entity reference names can not start with character '&'
(position: START_TAG seen
...="condor_requirements">GlueHostOperatingSystemRelease =?= "5.3"
&&... @50:104)


In the userguide, a sample condor classad entry is just specified with
plain "||"

-Allan

-- 
Allan M. Espinosa <http://amespinosa.wordpress.com>
PhD student, Computer Science
University of Chicago <http://people.cs.uchicago.edu/~aespinosa>


From aespinosa at cs.uchicago.edu  Mon Apr 18 14:35:12 2011
From: aespinosa at cs.uchicago.edu (Allan Espinosa)
Date: Mon, 18 Apr 2011 14:35:12 -0500
Subject: [Swift-user] Re: condor_requirements "&&" operator and other
	possible literal tokens
In-Reply-To: <BANLkTikGkwm3wbxdUvrepj01ji3kbX64Sw@mail.gmail.com>
References: <BANLkTikGkwm3wbxdUvrepj01ji3kbX64Sw@mail.gmail.com>
Message-ID: <BANLkTi=Yh+8rUi94iEEEOHs0FWJ=Gse6sg@mail.gmail.com>

Restating my question: how do I include logical expressions of condor
classads in a sites.xml entry?

2011/4/18 Allan Espinosa <aespinosa at cs.uchicago.edu>:
> Hi,
>
> I have the following entry in my site catalog:
>
> ?<pool handle="FNAL_FERMIGRID__fermigridosg1.fnal.gov">
> ? ?<execution provider="condor" url="none"/>
>
> ? ?<profile namespace="globus" key="jobType">grid</profile>
> ? ?<profile namespace="globus" key="gridResource">gt2
> fermigridosg1.fnal.gov/jobmanager-condor</profile>
>
> ? ?<profile namespace="karajan" key="initialScore">20.0</profile>
> ? ?<profile namespace="karajan" key="jobThrottle">68.22</profile>
>
> ? ? ?<profile namespace="globus"
> key="condor_requirements">GlueHostOperatingSystemRelease =?= "5.3" &&
> GlueSubClusterName =!= GlueClusterName</profile>
>
>
> ? ?<gridftp ?url="gsiftp://fermigridosg1.fnal.gov"/>
> ? ?<workdirectory>/grid/data/engage/scec/swift_scratch</workdirectory>
> ?</pool>
>
>
>
>
> But Swift reports the following
>
> Execution failed:
> ? ? ? ?entity reference names can not start with character '&'
> (position: START_TAG seen
> ...="condor_requirements">GlueHostOperatingSystemRelease =?= "5.3"
> &&... @50:104)
>
>
> In the userguide, a sample condor classad entry is just specified with
> plain "||"
>
> -Allan
>


From hategan at mcs.anl.gov  Mon Apr 18 15:30:33 2011
From: hategan at mcs.anl.gov (Mihael Hategan)
Date: Mon, 18 Apr 2011 13:30:33 -0700
Subject: [Swift-user] Re: condor_requirements "&&" operator and other
	possible literal tokens
In-Reply-To: <BANLkTi=Yh+8rUi94iEEEOHs0FWJ=Gse6sg@mail.gmail.com>
References: <BANLkTikGkwm3wbxdUvrepj01ji3kbX64Sw@mail.gmail.com>
	<BANLkTi=Yh+8rUi94iEEEOHs0FWJ=Gse6sg@mail.gmail.com>
Message-ID: <1303158633.8922.0.camel@blabla2.none>

sites.xml is in xml, so try "&amp;" instead of "&".

On Mon, 2011-04-18 at 14:35 -0500, Allan Espinosa wrote:
> Restating my question: how do I include logical expressions of condor
> classads in a sites.xml entry?
> 
> 2011/4/18 Allan Espinosa <aespinosa at cs.uchicago.edu>:
> > Hi,
> >
> > I have the following entry in my site catalog:
> >
> >  <pool handle="FNAL_FERMIGRID__fermigridosg1.fnal.gov">
> >    <execution provider="condor" url="none"/>
> >
> >    <profile namespace="globus" key="jobType">grid</profile>
> >    <profile namespace="globus" key="gridResource">gt2
> > fermigridosg1.fnal.gov/jobmanager-condor</profile>
> >
> >    <profile namespace="karajan" key="initialScore">20.0</profile>
> >    <profile namespace="karajan" key="jobThrottle">68.22</profile>
> >
> >      <profile namespace="globus"
> > key="condor_requirements">GlueHostOperatingSystemRelease =?= "5.3" &&
> > GlueSubClusterName =!= GlueClusterName</profile>
> >
> >
> >    <gridftp  url="gsiftp://fermigridosg1.fnal.gov"/>
> >    <workdirectory>/grid/data/engage/scec/swift_scratch</workdirectory>
> >  </pool>
> >
> >
> >
> >
> > But Swift reports the following
> >
> > Execution failed:
> >        entity reference names can not start with character '&'
> > (position: START_TAG seen
> > ...="condor_requirements">GlueHostOperatingSystemRelease =?= "5.3"
> > &&... @50:104)
> >
> >
> > In the userguide, a sample condor classad entry is just specified with
> > plain "||"
> >
> > -Allan
> >
> _______________________________________________
> Swift-user mailing list
> Swift-user at ci.uchicago.edu
> http://mail.ci.uchicago.edu/mailman/listinfo/swift-user




From Khushbu.Agarwal at pnl.gov  Mon Apr 18 16:29:03 2011
From: Khushbu.Agarwal at pnl.gov (Agarwal, Khushbu)
Date: Mon, 18 Apr 2011 14:29:03 -0700
Subject: [Swift-user] returning an array from function
Message-ID: <D0B71746ADD17848B85A4808D5A61667012305CA6646@EMAIL05.pnl.gov>

Hi, 

I am a beginner with swift and I am trying to write a workflow. My workflow has 2 app procedures. The first one calls a shell script which generates X number of files which act as input into the next app procedure. My question is , how do I specify that procedure returns an array of files?  

Currently, I am seeing that the files generated from first app, get deleted... I am guessing because I am not specifying them as output files etc?

Thanks,
-Khushbu



From mickelso at mcs.anl.gov  Mon Apr 25 09:50:45 2011
From: mickelso at mcs.anl.gov (Sheri Mickelson)
Date: Mon, 25 Apr 2011 09:50:45 -0500
Subject: [Swift-user] Swift never terminates
Message-ID: <4DB58A45.3020007@mcs.anl.gov>

Hi,

I've been trying to run something similar to the swift code below, but swift never terminates.  This 
example takes a file that's already on disk (netcdfFile.nc) and tries to create 1 plot (xy.ps).  It 
creates the plot, but keeps running("Progress: Active: 1").  Do you see anything I'm doing wrong in 
my swift file?

Thanks, Sheri

--------------------------------------------------------------------

type file;


(file out) callNCL (file inF, string nclFile) {

   app{ncl nclFile;}
}

file plot<simple_mapper;prefix="/my/file/xy.ps">;
file ncFile<simple_mapper;prefix="/my/file/netcdfFile.nc">;

string nclF = "/my/file/test.ncl";

plot = callNCL(ncFile,nclF);

--------------------------------------------------------------------


From hategan at mcs.anl.gov  Mon Apr 25 12:54:00 2011
From: hategan at mcs.anl.gov (Mihael Hategan)
Date: Mon, 25 Apr 2011 10:54:00 -0700
Subject: [Swift-user] Swift never terminates
In-Reply-To: <4DB58A45.3020007@mcs.anl.gov>
References: <4DB58A45.3020007@mcs.anl.gov>
Message-ID: <1303754040.7453.0.camel@blabla2.none>

Can you post your sites.xml?

Mihael

On Mon, 2011-04-25 at 09:50 -0500, Sheri Mickelson wrote:
> Hi,
> 
> I've been trying to run something similar to the swift code below, but swift never terminates.  This 
> example takes a file that's already on disk (netcdfFile.nc) and tries to create 1 plot (xy.ps).  It 
> creates the plot, but keeps running("Progress: Active: 1").  Do you see anything I'm doing wrong in 
> my swift file?
> 
> Thanks, Sheri
> 
> --------------------------------------------------------------------
> 
> type file;
> 
> 
> (file out) callNCL (file inF, string nclFile) {
> 
>    app{ncl nclFile;}
> }
> 
> file plot<simple_mapper;prefix="/my/file/xy.ps">;
> file ncFile<simple_mapper;prefix="/my/file/netcdfFile.nc">;
> 
> string nclF = "/my/file/test.ncl";
> 
> plot = callNCL(ncFile,nclF);
> 
> --------------------------------------------------------------------
> _______________________________________________
> Swift-user mailing list
> Swift-user at ci.uchicago.edu
> http://mail.ci.uchicago.edu/mailman/listinfo/swift-user




From iraicu at cs.iit.edu  Mon Apr 25 15:26:29 2011
From: iraicu at cs.iit.edu (Ioan Raicu)
Date: Mon, 25 Apr 2011 15:26:29 -0500
Subject: [Swift-user] Call for Participation: ACM High-Performance Parallel
 and Distributed
 Computing (HPDC) in San Jose, California, June 8-11, 2011
Message-ID: <4DB5D8F5.7060105@cs.iit.edu>


      Call for Participation


      The 20th International ACM Symposium on High-Performance Parallel
      and Distributed Computing 2011

San Jose, California, June 8-11, 2011
http://www.hpdc.org/2011/index.php

HPDC is the premier computer science conference for presenting new 
results relating to large scale high performance parallel and 
distributed systems used in both science and industry. For twenty years, 
HPDC has been at the center of new discoveries in clusters, grids, 
clouds, and parallel and multicore computers. HPDC is sponsored by the 
Association for Computing Machinery and the conference proceedings are 
published by the ACM Digital Library

HPDC 2011 will take place June 8-11 at the San Jose Convention Center in 
San Jose, California, in conjunction with the ACM Federated Computer 
Conference (FCRC). The conference overview 
(http://www.hpdc.org/2011/overview.php), program 
(http://www.hpdc.org/2011/program.php), and registration 
(http://www.hpdc.org/2011/venue.php) are now available. The early bird 
registration period end May 16th, 2011.

HPDC workshops will take place on Wednesday, June 8th; for more 
information, see http://www.hpdc.org/2011/workshops.php:*
*

    * 3DAPAS: Workshop on Dynamic Distributed Data-Intensive
      Applications, Programming Abstractions, and Systems
      <https://sites.google.com/site/3dapas/>
    * DIDC: The Fourth International Workshop on Data-Intensive
      Distributed Computing
      <http://www.cse.buffalo.edu/faculty/tkosar/didc11>
    * ECMLS: The Second International Emerging Computational Methods for
      the Life Sciences Workshop <http://salsahpc.indiana.edu/ECMLS2011/>
    * LSAP: Workshop on Large-Scale System and Application Performance
      <http://www.lsap2011.org>
    * MapReduce: The Second International Workshop on MapReduce and its
      Applications <http://graal.ens-lyon.fr/mapreduce>
    * ScienceCloud: 2nd Workshop on Scientific Cloud Computing
      <http://www.cs.iit.edu/%7Eiraicu/ScienceCloud2011>
    * VTDC: Virtual Technologies in Distributed Computing
      <http://www.grid-appliance.org/wiki/index.php/VTDC11>

A limited number of travel assistance grants will be available for 
students. Applications are due by May 1, 2011. For more information, 
please see http://www.hpdc.org/2011/studenttravel.php. To apply, please 
complete this form 
(https://spreadsheets.google.com/viewform?formkey=dHM1ZEhjeDE0R3A5ai1LSERjN2pQcnc6MQ) 
before May 1st.

HPDC and SIGMETRICS are running a student research poster session which 
are still accepting submissions until April 27th. For more information 
on the student research poster session, please see 
http://www.hpdc.org/2011/cfp-student.php.

-- 
=================================================================
Ioan Raicu, Ph.D.
Assistant Professor, Illinois Institute of Technology (IIT)
Guest Research Faculty, Argonne National Laboratory (ANL)
=================================================================
Data-Intensive Distributed Systems Laboratory, CS/IIT
Distributed Systems Laboratory, MCS/ANL
=================================================================
Cel:    1-847-722-0876
Office: 1-312-567-5704
Email:  iraicu at cs.iit.edu
Web:    http://www.cs.iit.edu/~iraicu/
Web:    http://datasys.cs.iit.edu/
=================================================================
=================================================================


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-user/attachments/20110425/8b960a87/attachment.html>

From iraicu at cs.iit.edu  Mon Apr 25 15:50:41 2011
From: iraicu at cs.iit.edu (Ioan Raicu)
Date: Mon, 25 Apr 2011 15:50:41 -0500
Subject: [Swift-user] CFP: ACM HPDC/SIGMETRICS Student Research Posters
 Session, due April 27th 2011
Message-ID: <4DB5DEA1.90506@cs.iit.edu>


    Call for Extended Abstracts


      HPDC/SIGMETRICS 2011 Student Research Posters Session

San Jose, California, June 8, 2011
http://www.hpdc.org/2011/cfp-student.php
http://www.sigmetrics.org/sigmetrics2011/student_posters.shtml

HPDC and SIGMETRICS encourage to submit poster proposals aimed at 
showcasing the "work-in-progress" of students attending the two 
conferences. The goal of the poster session is to present students' 
research, provide an opportunity for informal discussion and facilitate 
interaction between members of the two communities. Posters will be 
presented on the evening of Wednesday, June 8th.

The primary author(s) of the poster must be a student. Posters will be 
reviewed by members of the HPDC and SIGMETRICS organization committees. 
At the event, posters must be presented by a student. Authors of 
accepted papers must not submit a poster of the work they present in the 
conference or affiliated workshops. Topics of interest include, but are 
not limited to:

# Applications of parallel, distributed, and cloud computing
# Resource management, scheduling and load-balancing
# Performance modeling, simulation, measurement and prediction
# Network architectures, topology and routing, wireless networks
# Systems, networks, and architectures for high end computing
# Energy efficient computing systems
# Data intensive computing
# I/O, file systems, and data management
# Parallel and multicore issues and opportunities
# Virtualization of machines, networks, and storage
# Sensor networks, mobile devices, real-time systems
# Social networks, Internet servers, multimedia systems, web services

Both analytical and empirical studies are encouraged to be submitted to 
the student poster session.


      Submission Guidelines:

Student authors are asked to submit 2 pages abstracts in standard ACM 
format (for templates, see 
http://www.acm.org/sigs/publications/proceedings-templates) before the 
deadline below. All submissions should be in PDF format. Please submit 
your abstract at https://cmt.research.microsoft.com/HPDC_SIG2011 
<https://cmt.research.microsoft.com/HPDC_SIG2011> The accepted abstracts 
will appear in a special issue of ACM Performance Evaluation Review (PER).


      Important Dates

# Submission deadline April 27, 2011
# Acceptance notifications May 9, 2011


      Student Poster Session Chairs

# Giuliano Casale, Imperial College London
# Ioan Raicu, Illinois Institute of Technology & Argonne National Laboratory


      Student Poster Session Program Committee

    * Augustin Chaintreau, Columbia University
    * Abhishek Chandra, University of Minnesota
    * Alex Iosup - EWI, Delft University
    * Keith Jackson, Lawrence Berkeley National Laboratory
    * Manish Parashar, Rutgers University
    * Y. C. Tay, National University of Singapore


-- 
=================================================================
Ioan Raicu, Ph.D.
Assistant Professor, Illinois Institute of Technology (IIT)
Guest Research Faculty, Argonne National Laboratory (ANL)
=================================================================
Data-Intensive Distributed Systems Laboratory, CS/IIT
Distributed Systems Laboratory, MCS/ANL
=================================================================
Cel:    1-847-722-0876
Office: 1-312-567-5704
Email:  iraicu at cs.iit.edu
Web:    http://www.cs.iit.edu/~iraicu/
Web:    http://datasys.cs.iit.edu/
=================================================================
=================================================================


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-user/attachments/20110425/702b7f6a/attachment.html>

From iraicu at cs.iit.edu  Mon Apr 25 16:34:54 2011
From: iraicu at cs.iit.edu (Ioan Raicu)
Date: Mon, 25 Apr 2011 16:34:54 -0500
Subject: [Swift-user] CFP: Springer Journal of Grid Computing,
 Special Issue on Data Intensive Computing in the Clouds
Message-ID: <4DB5E8FE.9090402@cs.iit.edu>


    Special Issue on Data Intensive Computing in the Clouds


      Springer Journal of Grid Computing

http://datasys.cs.iit.edu/events/JGC-DataCloud-2012/index.html

Applications and experiments in all areas of science are becoming 
increasingly complex and more demanding in terms of their computational 
and data requirements. Some applications generate data volumes reaching 
hundreds of terabytes and even
petabytes. As scientific applications become more data intensive, the 
management of data resources and dataflow between the storage and 
compute resources is becoming the main bottleneck. Analyzing, 
visualizing, and disseminating these large data sets has become a major 
challenge and data intensive computing is now considered as the "fourth 
paradigm" in scientific discovery after empirical, theoretical, and 
computational scientific approaches.

The Special Issue on Data Intensive Computing in the Clouds will provide 
the scientific community a dedicated forum, within the prestigious 
Springer Journal of Grid Computing, for presenting new research, 
development, and deployment efforts in running data-intensive computing 
workloads on Cloud Computing infrastructures. This special issue will 
focus on the use of cloud-based technologies to meet the new data 
intensive scientific challenges that are not well served by the current 
supercomputers, grids or compute-intensive clouds. We believe this venue 
will be an excellent place to help the community define the current 
state, determine future goals, and present architectures and services 
for future clouds supporting data intensive computing.


        Topics

    * Data-intensive cloud computing applications, characteristics,
      challenges
    * Case studies of data intensive computing in the clouds
    * Performance evaluation of data clouds, data grids, and data centers
    * Energy-efficient data cloud design and management
    * Data placement, scheduling, and interoperability in the clouds
    * Accountability, QoS, and SLAs
    * Data privacy and protection in a public cloud environment
    * Distributed file systems for clouds
    * Data streaming and parallelization
    * New programming models for data-intensive cloud computing
    * Scalability issues in clouds
    * Social computing and massively social gaming
    * 3D Internet and implications
    * Future research challenges in data-intensive cloud computing


        Important Dates

    * Papers Due:                     July 15, 2011
    * First Round Decisions:        October 1, 2011
    * Major Revisions if needed:   November 1, 2011
    * Second Round Decisions:    December 1, 2011
    * Minor Revisions if needed:   January 13, 2012
    * Final Decision:                   February 1, 2012
    * Publication Date:               June 2012


        Paper Submission

Authors are invited to submit original and unpublished technical papers. 
All submissions will be peer-reviewed and judged on correctness, 
originality, technical strength, significance, quality of presentation, 
and relevance to the special issue topics of interest. Submitted papers 
may not have appeared in or be under consideration for another workshop, 
conference or a journal, nor may they be under review or submitted to 
another forum during the review process. Submitted papers may not exceed 
20 single-spaced double-column pages using 10-point size font on 8.5x11 
inch pages (1" margins), including figures, tables, and references; note 
that accepted papers will likely be between 15 to 20 pages, depending on 
a variety of factors; for more information for preparing the submitted 
papers, please see 
http://www.springer.com/computer/communication+networks/journal/10723, 
under "Instructions for Authors". The final papers (PDF format) must be 
submitted online at http://grid.edmgr.com/ before the deadline of July 
15th, 2011 at 11:59PM PST. For any questions on the submission process, 
please email the guest editors at jgc-datacloud-2012 at datasys.cs.iit.edu 
<mailto:jgc-datacloud-2012 at datasys.cs.iit.edu>.


        Guest Editors

Special Issue Guest Editors

    * Tevfik Kosar (tkosar at buffalo.edu <mailto:tkosar at buffalo.edu>), 
      University at Buffalo
    * Ioan Raicu (iraicu at cs.iit.edu <mailto:iraicu at cs.iit.edu>),
      Illinois Institute of Technology & Argonne National Laboratory


        Editors-in-Chief

    * Peter Kacsuk, Hungarian Academy of Sciences
    * Ian Foster, University of Chicago & Argonne National Laboratory

-- 
=================================================================
Ioan Raicu, Ph.D.
Assistant Professor, Illinois Institute of Technology (IIT)
Guest Research Faculty, Argonne National Laboratory (ANL)
=================================================================
Data-Intensive Distributed Systems Laboratory, CS/IIT
Distributed Systems Laboratory, MCS/ANL
=================================================================
Cel:    1-847-722-0876
Office: 1-312-567-5704
Email:  iraicu at cs.iit.edu
Web:    http://www.cs.iit.edu/~iraicu/
Web:    http://datasys.cs.iit.edu/
=================================================================
=================================================================


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-user/attachments/20110425/723dc323/attachment.html>

From frank.wang04 at gmail.com  Tue Apr 26 09:34:54 2011
From: frank.wang04 at gmail.com (Weiyang Wang)
Date: Tue, 26 Apr 2011 22:34:54 +0800
Subject: [Swift-user] [SWIFT on Beagle]Cannot Execute Correctly After
	Submitting Job
Message-ID: <BANLkTimpoVKQ5wky=c-txqVkqhJdoW0ZPQ@mail.gmail.com>

Hello,

I've meet problems when submitting jobs using Swift.

My Command Line is like:

swift -sites.file ./mysites.xml  -tc.file ./tc.data largescale.swift

My mysites.xml is like
<config>
<pool handle="pbs">
    <execution provider="pbs" url="none" jobManager="local:pbs"/>
    <profile namespace="globus" key="maxwalltime">0:20:00</profile>
    <profile namespace="globus" key="workersPerNode">8</profile>
    <profile namespace="globus" key="ppn">8</profile>
    <!-- <profile namespace="globus"
key="internalHostname">172.5.86.6</profile>-->
    <profile namespace="globus" key="nodeGranularity">1</profile>
    <profile namespace="globus" key="maxNodes">5</profile>
    <profile namespace="karajan" key="jobThrottle">1.99</profile>
    <profile namespace="karajan" key="initialScore">10000</profile>
    <profile namespace="globus" key="project">CI-SES000031</profile>
    <!--gridftp  url="local://localhost" />-->
    <filesystem provider="local"/>
    <!--scratch>/home/frankwang/tmp</scratch>-->

 <workdirectory>/gpfs/pads/projects/CI-SES000031/csm/tempswiftwork</workdirectory>
</pool>
</config>

My tc.data is like,
pbs             echo    /bin/echo       INSTALLED       INTEL32::LINUX  null
pbs             sh      /bin/bash       INSTALLED       INTEL32::LINUX  null


In my swift script I'm running some gfortran compiled programs(I wrap my
fortran programs in sh files), and in my error log file, it shows that

/gpfs/pads/projects/CI-SES000031/csm/DSSAT040.EXE: error while loading
shared libraries: libgfortran.so.3: cannot open shared object file: No such
file or directory

I didn't meet the same problem when I run these on PADS, and after I swap
the module to PrgEnv-gnu when logging onto Beagle I can also run this
DSSAT040.EXE.
My guess is,  I submit the jobs to a Non-GNU Environment in Beagle.

How can I correct this?

Thank you.

Weiyang
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-user/attachments/20110426/d3d40657/attachment.html>

From ketan at ci.uchicago.edu  Tue Apr  5 10:04:49 2011
From: ketan at ci.uchicago.edu (Ketan Maheshwari)
Date: Tue, 05 Apr 2011 15:04:49 -0000
Subject: [Swift-user] Re: tui
In-Reply-To: <898D8E1A-B3AA-4796-A15E-F615346EE464@gmail.com>
References: <898D8E1A-B3AA-4796-A15E-F615346EE464@gmail.com>
Message-ID: <6E50D518-1F4E-4B4A-BB00-9F81EC7C9AEB@ci.uchicago.edu>

Hi,

Just saw the reply to that message and tried the function keys which works.

So, the function keys F2 through F8 works on the tui to switch between tabs.

Ketan

On Apr 5, 2011, at 10:01 AM, Ketan Maheshwari wrote:

> Hi,
> 
> I too tried the "-tui" feature and stumbled into the same issue of not being able to change tabs that Mike faced:
> 
> http://mail.ci.uchicago.edu/pipermail/swift-user/2009-November/001226.html
> 
> Any clues?
> 
> Ketan



From leggett at ci.uchicago.edu  Tue Apr 26 09:53:13 2011
From: leggett at ci.uchicago.edu (Ti Leggett)
Date: Tue, 26 Apr 2011 09:53:13 -0500
Subject: [Swift-user] Re: [beagle-users] [SWIFT on Beagle]Cannot Execute
	Correctly After Submitting Job
In-Reply-To: <BANLkTimpoVKQ5wky=c-txqVkqhJdoW0ZPQ@mail.gmail.com>
References: <BANLkTimpoVKQ5wky=c-txqVkqhJdoW0ZPQ@mail.gmail.com>
Message-ID: <23CC8309-969A-41A7-8E5D-AEA010145DEC@ci.uchicago.edu>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Was the code you're trying to run compiled on Beagle. IT looks to me like it was compiled on PADS. You must recompile your code on Beagle using the Cray compiler wrappers. Please refer to the Beagle FAQ on how to do this:

http://www.ci.uchicago.edu/wiki/bin/view/Beagle/DevelopOnBeagle

On Apr 26, 2011, at 9:34 AM, Weiyang Wang wrote:

> Hello,
> 
> I've meet problems when submitting jobs using Swift.
> 
> My Command Line is like: 
> 
> swift -sites.file ./mysites.xml  -tc.file ./tc.data largescale.swift
> 
> My mysites.xml is like
> <config>
> <pool handle="pbs">
>     <execution provider="pbs" url="none" jobManager="local:pbs"/>
>     <profile namespace="globus" key="maxwalltime">0:20:00</profile>
>     <profile namespace="globus" key="workersPerNode">8</profile>
>     <profile namespace="globus" key="ppn">8</profile>
>     <!-- <profile namespace="globus" key="internalHostname">172.5.86.6</profile>-->
>     <profile namespace="globus" key="nodeGranularity">1</profile>
>     <profile namespace="globus" key="maxNodes">5</profile>
>     <profile namespace="karajan" key="jobThrottle">1.99</profile>
>     <profile namespace="karajan" key="initialScore">10000</profile>
>     <profile namespace="globus" key="project">CI-SES000031</profile>
>     <!--gridftp  url="local://localhost" />-->
>     <filesystem provider="local"/>
>     <!--scratch>/home/frankwang/tmp</scratch>-->
>     <workdirectory>/gpfs/pads/projects/CI-SES000031/csm/tempswiftwork</workdirectory>
> </pool>
> </config>
> 
> My tc.data is like, 
> pbs             echo    /bin/echo       INSTALLED       INTEL32::LINUX  null
> pbs             sh      /bin/bash       INSTALLED       INTEL32::LINUX  null
> 
> 
> In my swift script I'm running some gfortran compiled programs(I wrap my fortran programs in sh files), and in my error log file, it shows that
> 
> /gpfs/pads/projects/CI-SES000031/csm/DSSAT040.EXE: error while loading shared libraries: libgfortran.so.3: cannot open shared object file: No such file or directory
> 
> I didn't meet the same problem when I run these on PADS, and after I swap the module to PrgEnv-gnu when logging onto Beagle I can also run this DSSAT040.EXE.
> My guess is,  I submit the jobs to a Non-GNU Environment in Beagle.
> 
> How can I correct this?
> 
> Thank you.
> 
> Weiyang
> _______________________________________________
> beagle-users mailing list
> beagle-users at ci.uchicago.edu
> http://mail.ci.uchicago.edu/mailman/listinfo/beagle-users

-----BEGIN PGP SIGNATURE-----
Version: GnuPG/MacGPG2 v2.0.14 (Darwin)

iEYEARECAAYFAk223FkACgkQ4RgdOxQVi0AVxgCeNmnQ0e2xgmIZjQ3vkot1SpWf
H5MAnjS/VbEaH7vWSJFcSK+2hs1VHo4d
=8FUi
-----END PGP SIGNATURE-----


From ketan at mcs.anl.gov  Tue Apr 26 10:01:06 2011
From: ketan at mcs.anl.gov (Ketan Maheshwari)
Date: Tue, 26 Apr 2011 10:01:06 -0500
Subject: [Swift-user] Re: [beagle-users] [SWIFT on Beagle]Cannot Execute
	Correctly After Submitting Job
In-Reply-To: <BANLkTimpoVKQ5wky=c-txqVkqhJdoW0ZPQ@mail.gmail.com>
References: <BANLkTimpoVKQ5wky=c-txqVkqhJdoW0ZPQ@mail.gmail.com>
Message-ID: <B8689BAA-9692-497C-9D44-A87F65076644@mcs.anl.gov>

Hello Weiyang,

> /gpfs/pads/projects/CI-SES000031/csm/DSSAT040.EXE: error while loading shared libraries: libgfortran.so.3: cannot open shared object file: No such file or directory
> 
> I didn't meet the same problem when I run these on PADS, and after I swap the module to PrgEnv-gnu when logging onto Beagle I can also run this DSSAT040.EXE.
> My guess is,  I submit the jobs to a Non-GNU Environment in Beagle.
> 
> How can I correct this?

Perhaps you can put a line in your sh wrapper before executable invocation to load the gnu module. 

> 
> Thank you.
> 
> Weiyang
> _______________________________________________
> beagle-users mailing list
> beagle-users at ci.uchicago.edu
> http://mail.ci.uchicago.edu/mailman/listinfo/beagle-users



From ketan at mcs.anl.gov  Tue Apr 26 10:03:35 2011
From: ketan at mcs.anl.gov (Ketan Maheshwari)
Date: Tue, 26 Apr 2011 10:03:35 -0500
Subject: [Swift-user] Re: [beagle-users] [SWIFT on Beagle]Cannot Execute
	Correctly After Submitting Job
In-Reply-To: <BANLkTimpoVKQ5wky=c-txqVkqhJdoW0ZPQ@mail.gmail.com>
References: <BANLkTimpoVKQ5wky=c-txqVkqhJdoW0ZPQ@mail.gmail.com>
Message-ID: <2C6B1E93-CC25-4CDD-89E8-A16ECB6402F3@mcs.anl.gov>

Hello Weiyang,

I notice that your workdirectory on your sites.xml is /gpfs/.. which would not work as Beagle nodes can not write to any external filesystem. I suggest you to correct it to point to somewhere on your home on /lustre filesystem.

--Ketan


On Apr 26, 2011, at 9:34 AM, Weiyang Wang wrote:

> Hello,
> 
> I've meet problems when submitting jobs using Swift.
> 
> My Command Line is like: 
> 
> swift -sites.file ./mysites.xml  -tc.file ./tc.data largescale.swift
> 
> My mysites.xml is like
> <config>
> <pool handle="pbs">
>     <execution provider="pbs" url="none" jobManager="local:pbs"/>
>     <profile namespace="globus" key="maxwalltime">0:20:00</profile>
>     <profile namespace="globus" key="workersPerNode">8</profile>
>     <profile namespace="globus" key="ppn">8</profile>
>     <!-- <profile namespace="globus" key="internalHostname">172.5.86.6</profile>-->
>     <profile namespace="globus" key="nodeGranularity">1</profile>
>     <profile namespace="globus" key="maxNodes">5</profile>
>     <profile namespace="karajan" key="jobThrottle">1.99</profile>
>     <profile namespace="karajan" key="initialScore">10000</profile>
>     <profile namespace="globus" key="project">CI-SES000031</profile>
>     <!--gridftp  url="local://localhost" />-->
>     <filesystem provider="local"/>
>     <!--scratch>/home/frankwang/tmp</scratch>-->
>     <workdirectory>/gpfs/pads/projects/CI-SES000031/csm/tempswiftwork</workdirectory>
> </pool>
> </config>
> 
> My tc.data is like, 
> pbs             echo    /bin/echo       INSTALLED       INTEL32::LINUX  null
> pbs             sh      /bin/bash       INSTALLED       INTEL32::LINUX  null
> 
> 
> In my swift script I'm running some gfortran compiled programs(I wrap my fortran programs in sh files), and in my error log file, it shows that
> 
> /gpfs/pads/projects/CI-SES000031/csm/DSSAT040.EXE: error while loading shared libraries: libgfortran.so.3: cannot open shared object file: No such file or directory
> 
> I didn't meet the same problem when I run these on PADS, and after I swap the module to PrgEnv-gnu when logging onto Beagle I can also run this DSSAT040.EXE.
> My guess is,  I submit the jobs to a Non-GNU Environment in Beagle.
> 
> How can I correct this?
> 
> Thank you.
> 
> Weiyang
> _______________________________________________
> beagle-users mailing list
> beagle-users at ci.uchicago.edu
> http://mail.ci.uchicago.edu/mailman/listinfo/beagle-users



From frank.wang04 at gmail.com  Tue Apr 26 13:26:52 2011
From: frank.wang04 at gmail.com (Weiyang Wang)
Date: Wed, 27 Apr 2011 02:26:52 +0800
Subject: [Swift-user] Re: [beagle-users] [SWIFT on Beagle]Cannot Execute
 Correctly After Submitting Job
In-Reply-To: <23CC8309-969A-41A7-8E5D-AEA010145DEC@ci.uchicago.edu>
References: <BANLkTimpoVKQ5wky=c-txqVkqhJdoW0ZPQ@mail.gmail.com>
	<23CC8309-969A-41A7-8E5D-AEA010145DEC@ci.uchicago.edu>
Message-ID: <BANLkTi=-Na9iumvA9gNMAH83OeugSFSOkg@mail.gmail.com>

In fact I switched to PrgEnv-gnu to compile it again on Beagle, and it works
if I run it plainly on login.beagle

On Tue, Apr 26, 2011 at 10:53 PM, Ti Leggett <leggett at ci.uchicago.edu>wrote:

> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> Was the code you're trying to run compiled on Beagle. IT looks to me like
> it was compiled on PADS. You must recompile your code on Beagle using the
> Cray compiler wrappers. Please refer to the Beagle FAQ on how to do this:
>
> http://www.ci.uchicago.edu/wiki/bin/view/Beagle/DevelopOnBeagle
>
> On Apr 26, 2011, at 9:34 AM, Weiyang Wang wrote:
>
> > Hello,
> >
> > I've meet problems when submitting jobs using Swift.
> >
> > My Command Line is like:
> >
> > swift -sites.file ./mysites.xml  -tc.file ./tc.data largescale.swift
> >
> > My mysites.xml is like
> > <config>
> > <pool handle="pbs">
> >     <execution provider="pbs" url="none" jobManager="local:pbs"/>
> >     <profile namespace="globus" key="maxwalltime">0:20:00</profile>
> >     <profile namespace="globus" key="workersPerNode">8</profile>
> >     <profile namespace="globus" key="ppn">8</profile>
> >     <!-- <profile namespace="globus"
> key="internalHostname">172.5.86.6</profile>-->
> >     <profile namespace="globus" key="nodeGranularity">1</profile>
> >     <profile namespace="globus" key="maxNodes">5</profile>
> >     <profile namespace="karajan" key="jobThrottle">1.99</profile>
> >     <profile namespace="karajan" key="initialScore">10000</profile>
> >     <profile namespace="globus" key="project">CI-SES000031</profile>
> >     <!--gridftp  url="local://localhost" />-->
> >     <filesystem provider="local"/>
> >     <!--scratch>/home/frankwang/tmp</scratch>-->
> >
> <workdirectory>/gpfs/pads/projects/CI-SES000031/csm/tempswiftwork</workdirectory>
> > </pool>
> > </config>
> >
> > My tc.data is like,
> > pbs             echo    /bin/echo       INSTALLED       INTEL32::LINUX
>  null
> > pbs             sh      /bin/bash       INSTALLED       INTEL32::LINUX
>  null
> >
> >
> > In my swift script I'm running some gfortran compiled programs(I wrap my
> fortran programs in sh files), and in my error log file, it shows that
> >
> > /gpfs/pads/projects/CI-SES000031/csm/DSSAT040.EXE: error while loading
> shared libraries: libgfortran.so.3: cannot open shared object file: No such
> file or directory
> >
> > I didn't meet the same problem when I run these on PADS, and after I swap
> the module to PrgEnv-gnu when logging onto Beagle I can also run this
> DSSAT040.EXE.
> > My guess is,  I submit the jobs to a Non-GNU Environment in Beagle.
> >
> > How can I correct this?
> >
> > Thank you.
> >
> > Weiyang
> > _______________________________________________
> > beagle-users mailing list
> > beagle-users at ci.uchicago.edu
> > http://mail.ci.uchicago.edu/mailman/listinfo/beagle-users
>
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG/MacGPG2 v2.0.14 (Darwin)
>
> iEYEARECAAYFAk223FkACgkQ4RgdOxQVi0AVxgCeNmnQ0e2xgmIZjQ3vkot1SpWf
> H5MAnjS/VbEaH7vWSJFcSK+2hs1VHo4d
> =8FUi
> -----END PGP SIGNATURE-----
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-user/attachments/20110427/6ef50349/attachment.html>

From frank.wang04 at gmail.com  Tue Apr 26 13:34:48 2011
From: frank.wang04 at gmail.com (Weiyang Wang)
Date: Wed, 27 Apr 2011 02:34:48 +0800
Subject: [Swift-user] Re: [beagle-users] [SWIFT on Beagle]Cannot Execute
 Correctly After Submitting Job
In-Reply-To: <7A0517F1-5A62-420C-B755-EE6575BC29CF@ci.uchicago.edu>
References: <BANLkTimpoVKQ5wky=c-txqVkqhJdoW0ZPQ@mail.gmail.com>
	<23CC8309-969A-41A7-8E5D-AEA010145DEC@ci.uchicago.edu>
	<BANLkTi=-Na9iumvA9gNMAH83OeugSFSOkg@mail.gmail.com>
	<7A0517F1-5A62-420C-B755-EE6575BC29CF@ci.uchicago.edu>
Message-ID: <BANLkTik=am_r5B-Kgwx4u9aX84N7JypsSg@mail.gmail.com>

The fact is I changed the compiler to ftn and make them again, it's saying
"DSSAT040.EXE is up to date"

On Wed, Apr 27, 2011 at 2:33 AM, Ti Leggett <leggett at ci.uchicago.edu> wrote:

> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> But make sure you use the Cray compiler wrappers  - cc, CC and ftn - and
> not the gnu compilers (gcc, g++, gfortran) directly.
>
> On Apr 26, 2011, at 1:26 PM, Weiyang Wang wrote:
>
> > In fact I switched to PrgEnv-gnu to compile it again on Beagle, and it
> works if I run it plainly on login.beagle
> >
> > On Tue, Apr 26, 2011 at 10:53 PM, Ti Leggett <leggett at ci.uchicago.edu>
> wrote:
> > -----BEGIN PGP SIGNED MESSAGE-----
> > Hash: SHA1
> >
> > Was the code you're trying to run compiled on Beagle. IT looks to me like
> it was compiled on PADS. You must recompile your code on Beagle using the
> Cray compiler wrappers. Please refer to the Beagle FAQ on how to do this:
> >
> > http://www.ci.uchicago.edu/wiki/bin/view/Beagle/DevelopOnBeagle
> >
> > On Apr 26, 2011, at 9:34 AM, Weiyang Wang wrote:
> >
> > > Hello,
> > >
> > > I've meet problems when submitting jobs using Swift.
> > >
> > > My Command Line is like:
> > >
> > > swift -sites.file ./mysites.xml  -tc.file ./tc.data largescale.swift
> > >
> > > My mysites.xml is like
> > > <config>
> > > <pool handle="pbs">
> > >     <execution provider="pbs" url="none" jobManager="local:pbs"/>
> > >     <profile namespace="globus" key="maxwalltime">0:20:00</profile>
> > >     <profile namespace="globus" key="workersPerNode">8</profile>
> > >     <profile namespace="globus" key="ppn">8</profile>
> > >     <!-- <profile namespace="globus"
> key="internalHostname">172.5.86.6</profile>-->
> > >     <profile namespace="globus" key="nodeGranularity">1</profile>
> > >     <profile namespace="globus" key="maxNodes">5</profile>
> > >     <profile namespace="karajan" key="jobThrottle">1.99</profile>
> > >     <profile namespace="karajan" key="initialScore">10000</profile>
> > >     <profile namespace="globus" key="project">CI-SES000031</profile>
> > >     <!--gridftp  url="local://localhost" />-->
> > >     <filesystem provider="local"/>
> > >     <!--scratch>/home/frankwang/tmp</scratch>-->
> > >
> <workdirectory>/gpfs/pads/projects/CI-SES000031/csm/tempswiftwork</workdirectory>
> > > </pool>
> > > </config>
> > >
> > > My tc.data is like,
> > > pbs             echo    /bin/echo       INSTALLED       INTEL32::LINUX
>  null
> > > pbs             sh      /bin/bash       INSTALLED       INTEL32::LINUX
>  null
> > >
> > >
> > > In my swift script I'm running some gfortran compiled programs(I wrap
> my fortran programs in sh files), and in my error log file, it shows that
> > >
> > > /gpfs/pads/projects/CI-SES000031/csm/DSSAT040.EXE: error while loading
> shared libraries: libgfortran.so.3: cannot open shared object file: No such
> file or directory
> > >
> > > I didn't meet the same problem when I run these on PADS, and after I
> swap the module to PrgEnv-gnu when logging onto Beagle I can also run this
> DSSAT040.EXE.
> > > My guess is,  I submit the jobs to a Non-GNU Environment in Beagle.
> > >
> > > How can I correct this?
> > >
> > > Thank you.
> > >
> > > Weiyang
> > > _______________________________________________
> > > beagle-users mailing list
> > > beagle-users at ci.uchicago.edu
> > > http://mail.ci.uchicago.edu/mailman/listinfo/beagle-users
> >
> > -----BEGIN PGP SIGNATURE-----
> > Version: GnuPG/MacGPG2 v2.0.14 (Darwin)
> >
> > iEYEARECAAYFAk223FkACgkQ4RgdOxQVi0AVxgCeNmnQ0e2xgmIZjQ3vkot1SpWf
> > H5MAnjS/VbEaH7vWSJFcSK+2hs1VHo4d
> > =8FUi
> > -----END PGP SIGNATURE-----
> >
>
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG/MacGPG2 v2.0.14 (Darwin)
>
> iEYEARECAAYFAk23D/YACgkQ4RgdOxQVi0DGjACeJV4MG+9SuMBKKTyECpE82Exs
> eF4AoIE7bokWJQZM44tDVit4eQvj2G8j
> =y73+
> -----END PGP SIGNATURE-----
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-user/attachments/20110427/7e3aafc4/attachment.html>

From frank.wang04 at gmail.com  Tue Apr 26 14:14:02 2011
From: frank.wang04 at gmail.com (Weiyang Wang)
Date: Wed, 27 Apr 2011 03:14:02 +0800
Subject: [Swift-user] Re: [beagle-users] [SWIFT on Beagle]Cannot Execute
 Correctly After Submitting Job
In-Reply-To: <57C38529-F633-4062-A195-864D838A3180@mcs.anl.gov>
References: <BANLkTimpoVKQ5wky=c-txqVkqhJdoW0ZPQ@mail.gmail.com>
	<23CC8309-969A-41A7-8E5D-AEA010145DEC@ci.uchicago.edu>
	<BANLkTi=-Na9iumvA9gNMAH83OeugSFSOkg@mail.gmail.com>
	<57C38529-F633-4062-A195-864D838A3180@mcs.anl.gov>
Message-ID: <BANLkTikLg9M0GsRoiLTZ1p8e7JFvrBEkyg@mail.gmail.com>

OK, I've "make clean" them and  ow I switched to PrgEnv-cray, and going to
use ftn to compile some .for files, although they are currently not
recognizable by ftn. I want them to be compiled as F90 files (maybe I have
to rename them first? )

Could you kindly check out what's going on here within the makefile?


======= the first several lines====
OBJDIR = objCSMINPUT
FC     = ftn
FFLAGS = -O 3 -g
INC    = -I.
MKDIR  = mkdir -p
DIRS   = ../Common ../Input
RM     = rm -rf
===========================

===somewhere it starts compiling===

# Create a list of all source files including folder name
FSRC   = $(foreach dir,$(DIRS),$(wildcard $(dir)/*.for))

## Create the list of all object files
TOBJ   = $(foreach dir, $(DIRS), $(subst .for,.o,$(notdir $(wildcard
$(dir)/*.for))))

# Create the path to store the object files
FOBJ   = $(foreach obj,$(TOBJ),$(OBJDIR)/$(obj))

# Define the subfolders to look for source files
vpath %.for ../Common ../Input

$(OBJDIR)/%.o $(OBJDIR)/%.mod: %.for | $(OBJDIR)
        $(FC) $(FFLAGS) $(INC) -o $@ -c $<

../MINPT040.EXE : $(FOBJ)
        $(FC) $^ -o $@


============================




On Wed, Apr 27, 2011 at 2:43 AM, Ketan Maheshwari <ketan at mcs.anl.gov> wrote:

> Hello,
>
> This indicates that the executable is not running on beagle compute nodes.
> Could you kindly send the commandline and the path to your exe, data etc,
> and I can try to do some testing using aprun and qsub.
>
> Ketan
>
>
> On Apr 26, 2011, at 1:26 PM, Weiyang Wang wrote:
>
> In fact I switched to PrgEnv-gnu to compile it again on Beagle, and it
> works if I run it plainly on login.beagle
>
> On Tue, Apr 26, 2011 at 10:53 PM, Ti Leggett <leggett at ci.uchicago.edu>wrote:
>
>> -----BEGIN PGP SIGNED MESSAGE-----
>> Hash: SHA1
>>
>> Was the code you're trying to run compiled on Beagle. IT looks to me like
>> it was compiled on PADS. You must recompile your code on Beagle using the
>> Cray compiler wrappers. Please refer to the Beagle FAQ on how to do this:
>>
>> http://www.ci.uchicago.edu/wiki/bin/view/Beagle/DevelopOnBeagle
>>
>> On Apr 26, 2011, at 9:34 AM, Weiyang Wang wrote:
>>
>> > Hello,
>> >
>> > I've meet problems when submitting jobs using Swift.
>> >
>> > My Command Line is like:
>> >
>> > swift -sites.file ./mysites.xml  -tc.file ./tc.data largescale.swift
>> >
>> > My mysites.xml is like
>> > <config>
>> > <pool handle="pbs">
>> >     <execution provider="pbs" url="none" jobManager="local:pbs"/>
>> >     <profile namespace="globus" key="maxwalltime">0:20:00</profile>
>> >     <profile namespace="globus" key="workersPerNode">8</profile>
>> >     <profile namespace="globus" key="ppn">8</profile>
>> >     <!-- <profile namespace="globus"
>> key="internalHostname">172.5.86.6</profile>-->
>> >     <profile namespace="globus" key="nodeGranularity">1</profile>
>> >     <profile namespace="globus" key="maxNodes">5</profile>
>> >     <profile namespace="karajan" key="jobThrottle">1.99</profile>
>> >     <profile namespace="karajan" key="initialScore">10000</profile>
>> >     <profile namespace="globus" key="project">CI-SES000031</profile>
>> >     <!--gridftp  url="local://localhost" />-->
>> >     <filesystem provider="local"/>
>> >     <!--scratch>/home/frankwang/tmp</scratch>-->
>> >
>> <workdirectory>/gpfs/pads/projects/CI-SES000031/csm/tempswiftwork</workdirectory>
>> > </pool>
>> > </config>
>> >
>> > My tc.data is like,
>> > pbs             echo    /bin/echo       INSTALLED       INTEL32::LINUX
>>  null
>> > pbs             sh      /bin/bash       INSTALLED       INTEL32::LINUX
>>  null
>> >
>> >
>> > In my swift script I'm running some gfortran compiled programs(I wrap my
>> fortran programs in sh files), and in my error log file, it shows that
>> >
>> > /gpfs/pads/projects/CI-SES000031/csm/DSSAT040.EXE: error while loading
>> shared libraries: libgfortran.so.3: cannot open shared object file: No such
>> file or directory
>> >
>> > I didn't meet the same problem when I run these on PADS, and after I
>> swap the module to PrgEnv-gnu when logging onto Beagle I can also run this
>> DSSAT040.EXE.
>> > My guess is,  I submit the jobs to a Non-GNU Environment in Beagle.
>> >
>> > How can I correct this?
>> >
>> > Thank you.
>> >
>> > Weiyang
>> > _______________________________________________
>> > beagle-users mailing list
>> > beagle-users at ci.uchicago.edu
>> > http://mail.ci.uchicago.edu/mailman/listinfo/beagle-users
>>
>> -----BEGIN PGP SIGNATURE-----
>> Version: GnuPG/MacGPG2 v2.0.14 (Darwin)
>>
>> iEYEARECAAYFAk223FkACgkQ4RgdOxQVi0AVxgCeNmnQ0e2xgmIZjQ3vkot1SpWf
>> H5MAnjS/VbEaH7vWSJFcSK+2hs1VHo4d
>> =8FUi
>> -----END PGP SIGNATURE-----
>>
>
> _______________________________________________
> beagle-users mailing list
> beagle-users at ci.uchicago.edu
> http://mail.ci.uchicago.edu/mailman/listinfo/beagle-users
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-user/attachments/20110427/9d0a3e51/attachment.html>

From leggett at ci.uchicago.edu  Tue Apr 26 10:37:44 2011
From: leggett at ci.uchicago.edu (Ti Leggett)
Date: Tue, 26 Apr 2011 10:37:44 -0500
Subject: [Swift-user] Re: [beagle-users] [SWIFT on Beagle]Cannot Execute
	Correctly After Submitting Job
In-Reply-To: <2C6B1E93-CC25-4CDD-89E8-A16ECB6402F3@mcs.anl.gov>
References: <BANLkTimpoVKQ5wky=c-txqVkqhJdoW0ZPQ@mail.gmail.com>
	<2C6B1E93-CC25-4CDD-89E8-A16ECB6402F3@mcs.anl.gov>
Message-ID: <0D3439EA-7A37-40EC-B591-36C9528C3FEE@ci.uchicago.edu>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

That's incorrect. Compute nodes cannot write to /home but can write to /gpfs/pads; however, for performance and consistency reasons we highly recommend against doing so. Instead data should be staged to /lustre/beagle/scratch and computed on there and stage back to /gpfs/pads if necessary.

On Apr 26, 2011, at 10:03 AM, Ketan Maheshwari wrote:

> Hello Weiyang,
> 
> I notice that your workdirectory on your sites.xml is /gpfs/.. which would not work as Beagle nodes can not write to any external filesystem. I suggest you to correct it to point to somewhere on your home on /lustre filesystem.
> 
> --Ketan
> 
> 
> On Apr 26, 2011, at 9:34 AM, Weiyang Wang wrote:
> 
>> Hello,
>> 
>> I've meet problems when submitting jobs using Swift.
>> 
>> My Command Line is like: 
>> 
>> swift -sites.file ./mysites.xml  -tc.file ./tc.data largescale.swift
>> 
>> My mysites.xml is like
>> <config>
>> <pool handle="pbs">
>>    <execution provider="pbs" url="none" jobManager="local:pbs"/>
>>    <profile namespace="globus" key="maxwalltime">0:20:00</profile>
>>    <profile namespace="globus" key="workersPerNode">8</profile>
>>    <profile namespace="globus" key="ppn">8</profile>
>>    <!-- <profile namespace="globus" key="internalHostname">172.5.86.6</profile>-->
>>    <profile namespace="globus" key="nodeGranularity">1</profile>
>>    <profile namespace="globus" key="maxNodes">5</profile>
>>    <profile namespace="karajan" key="jobThrottle">1.99</profile>
>>    <profile namespace="karajan" key="initialScore">10000</profile>
>>    <profile namespace="globus" key="project">CI-SES000031</profile>
>>    <!--gridftp  url="local://localhost" />-->
>>    <filesystem provider="local"/>
>>    <!--scratch>/home/frankwang/tmp</scratch>-->
>>    <workdirectory>/gpfs/pads/projects/CI-SES000031/csm/tempswiftwork</workdirectory>
>> </pool>
>> </config>
>> 
>> My tc.data is like, 
>> pbs             echo    /bin/echo       INSTALLED       INTEL32::LINUX  null
>> pbs             sh      /bin/bash       INSTALLED       INTEL32::LINUX  null
>> 
>> 
>> In my swift script I'm running some gfortran compiled programs(I wrap my fortran programs in sh files), and in my error log file, it shows that
>> 
>> /gpfs/pads/projects/CI-SES000031/csm/DSSAT040.EXE: error while loading shared libraries: libgfortran.so.3: cannot open shared object file: No such file or directory
>> 
>> I didn't meet the same problem when I run these on PADS, and after I swap the module to PrgEnv-gnu when logging onto Beagle I can also run this DSSAT040.EXE.
>> My guess is,  I submit the jobs to a Non-GNU Environment in Beagle.
>> 
>> How can I correct this?
>> 
>> Thank you.
>> 
>> Weiyang
>> _______________________________________________
>> beagle-users mailing list
>> beagle-users at ci.uchicago.edu
>> http://mail.ci.uchicago.edu/mailman/listinfo/beagle-users
> 
> _______________________________________________
> beagle-users mailing list
> beagle-users at ci.uchicago.edu
> http://mail.ci.uchicago.edu/mailman/listinfo/beagle-users

-----BEGIN PGP SIGNATURE-----
Version: GnuPG/MacGPG2 v2.0.14 (Darwin)

iEYEARECAAYFAk225skACgkQ4RgdOxQVi0BIIgCfQh36tUPFVdAAAEk9vs6YYcVF
JHIAoJ83YKkfYDf8ddN8iENetbWFLVFR
=lYJY
-----END PGP SIGNATURE-----


From leggett at ci.uchicago.edu  Tue Apr 26 13:33:25 2011
From: leggett at ci.uchicago.edu (Ti Leggett)
Date: Tue, 26 Apr 2011 13:33:25 -0500
Subject: [Swift-user] Re: [beagle-users] [SWIFT on Beagle]Cannot Execute
	Correctly After Submitting Job
In-Reply-To: <BANLkTi=-Na9iumvA9gNMAH83OeugSFSOkg@mail.gmail.com>
References: <BANLkTimpoVKQ5wky=c-txqVkqhJdoW0ZPQ@mail.gmail.com>
	<23CC8309-969A-41A7-8E5D-AEA010145DEC@ci.uchicago.edu>
	<BANLkTi=-Na9iumvA9gNMAH83OeugSFSOkg@mail.gmail.com>
Message-ID: <7A0517F1-5A62-420C-B755-EE6575BC29CF@ci.uchicago.edu>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

But make sure you use the Cray compiler wrappers  - cc, CC and ftn - and not the gnu compilers (gcc, g++, gfortran) directly.

On Apr 26, 2011, at 1:26 PM, Weiyang Wang wrote:

> In fact I switched to PrgEnv-gnu to compile it again on Beagle, and it works if I run it plainly on login.beagle
> 
> On Tue, Apr 26, 2011 at 10:53 PM, Ti Leggett <leggett at ci.uchicago.edu> wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
> 
> Was the code you're trying to run compiled on Beagle. IT looks to me like it was compiled on PADS. You must recompile your code on Beagle using the Cray compiler wrappers. Please refer to the Beagle FAQ on how to do this:
> 
> http://www.ci.uchicago.edu/wiki/bin/view/Beagle/DevelopOnBeagle
> 
> On Apr 26, 2011, at 9:34 AM, Weiyang Wang wrote:
> 
> > Hello,
> >
> > I've meet problems when submitting jobs using Swift.
> >
> > My Command Line is like:
> >
> > swift -sites.file ./mysites.xml  -tc.file ./tc.data largescale.swift
> >
> > My mysites.xml is like
> > <config>
> > <pool handle="pbs">
> >     <execution provider="pbs" url="none" jobManager="local:pbs"/>
> >     <profile namespace="globus" key="maxwalltime">0:20:00</profile>
> >     <profile namespace="globus" key="workersPerNode">8</profile>
> >     <profile namespace="globus" key="ppn">8</profile>
> >     <!-- <profile namespace="globus" key="internalHostname">172.5.86.6</profile>-->
> >     <profile namespace="globus" key="nodeGranularity">1</profile>
> >     <profile namespace="globus" key="maxNodes">5</profile>
> >     <profile namespace="karajan" key="jobThrottle">1.99</profile>
> >     <profile namespace="karajan" key="initialScore">10000</profile>
> >     <profile namespace="globus" key="project">CI-SES000031</profile>
> >     <!--gridftp  url="local://localhost" />-->
> >     <filesystem provider="local"/>
> >     <!--scratch>/home/frankwang/tmp</scratch>-->
> >     <workdirectory>/gpfs/pads/projects/CI-SES000031/csm/tempswiftwork</workdirectory>
> > </pool>
> > </config>
> >
> > My tc.data is like,
> > pbs             echo    /bin/echo       INSTALLED       INTEL32::LINUX  null
> > pbs             sh      /bin/bash       INSTALLED       INTEL32::LINUX  null
> >
> >
> > In my swift script I'm running some gfortran compiled programs(I wrap my fortran programs in sh files), and in my error log file, it shows that
> >
> > /gpfs/pads/projects/CI-SES000031/csm/DSSAT040.EXE: error while loading shared libraries: libgfortran.so.3: cannot open shared object file: No such file or directory
> >
> > I didn't meet the same problem when I run these on PADS, and after I swap the module to PrgEnv-gnu when logging onto Beagle I can also run this DSSAT040.EXE.
> > My guess is,  I submit the jobs to a Non-GNU Environment in Beagle.
> >
> > How can I correct this?
> >
> > Thank you.
> >
> > Weiyang
> > _______________________________________________
> > beagle-users mailing list
> > beagle-users at ci.uchicago.edu
> > http://mail.ci.uchicago.edu/mailman/listinfo/beagle-users
> 
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG/MacGPG2 v2.0.14 (Darwin)
> 
> iEYEARECAAYFAk223FkACgkQ4RgdOxQVi0AVxgCeNmnQ0e2xgmIZjQ3vkot1SpWf
> H5MAnjS/VbEaH7vWSJFcSK+2hs1VHo4d
> =8FUi
> -----END PGP SIGNATURE-----
> 

-----BEGIN PGP SIGNATURE-----
Version: GnuPG/MacGPG2 v2.0.14 (Darwin)

iEYEARECAAYFAk23D/YACgkQ4RgdOxQVi0DGjACeJV4MG+9SuMBKKTyECpE82Exs
eF4AoIE7bokWJQZM44tDVit4eQvj2G8j
=y73+
-----END PGP SIGNATURE-----


From leggett at ci.uchicago.edu  Tue Apr 26 13:37:15 2011
From: leggett at ci.uchicago.edu (Ti Leggett)
Date: Tue, 26 Apr 2011 13:37:15 -0500
Subject: [Swift-user] Re: [beagle-users] [SWIFT on Beagle]Cannot Execute
	Correctly After Submitting Job
In-Reply-To: <BANLkTik=am_r5B-Kgwx4u9aX84N7JypsSg@mail.gmail.com>
References: <BANLkTimpoVKQ5wky=c-txqVkqhJdoW0ZPQ@mail.gmail.com>
	<23CC8309-969A-41A7-8E5D-AEA010145DEC@ci.uchicago.edu>
	<BANLkTi=-Na9iumvA9gNMAH83OeugSFSOkg@mail.gmail.com>
	<7A0517F1-5A62-420C-B755-EE6575BC29CF@ci.uchicago.edu>
	<BANLkTik=am_r5B-Kgwx4u9aX84N7JypsSg@mail.gmail.com>
Message-ID: <947053E8-95B4-4F1C-AD4F-A2EB76856085@ci.uchicago.edu>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Make sure to make clean or start from clean source so all of the code is compiled with the wrappers.

On Apr 26, 2011, at 1:34 PM, Weiyang Wang wrote:

> The fact is I changed the compiler to ftn and make them again, it's saying "DSSAT040.EXE is up to date"
> 
> On Wed, Apr 27, 2011 at 2:33 AM, Ti Leggett <leggett at ci.uchicago.edu> wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
> 
> But make sure you use the Cray compiler wrappers  - cc, CC and ftn - and not the gnu compilers (gcc, g++, gfortran) directly.
> 
> On Apr 26, 2011, at 1:26 PM, Weiyang Wang wrote:
> 
> > In fact I switched to PrgEnv-gnu to compile it again on Beagle, and it works if I run it plainly on login.beagle
> >
> > On Tue, Apr 26, 2011 at 10:53 PM, Ti Leggett <leggett at ci.uchicago.edu> wrote:
> > -----BEGIN PGP SIGNED MESSAGE-----
> > Hash: SHA1
> >
> > Was the code you're trying to run compiled on Beagle. IT looks to me like it was compiled on PADS. You must recompile your code on Beagle using the Cray compiler wrappers. Please refer to the Beagle FAQ on how to do this:
> >
> > http://www.ci.uchicago.edu/wiki/bin/view/Beagle/DevelopOnBeagle
> >
> > On Apr 26, 2011, at 9:34 AM, Weiyang Wang wrote:
> >
> > > Hello,
> > >
> > > I've meet problems when submitting jobs using Swift.
> > >
> > > My Command Line is like:
> > >
> > > swift -sites.file ./mysites.xml  -tc.file ./tc.data largescale.swift
> > >
> > > My mysites.xml is like
> > > <config>
> > > <pool handle="pbs">
> > >     <execution provider="pbs" url="none" jobManager="local:pbs"/>
> > >     <profile namespace="globus" key="maxwalltime">0:20:00</profile>
> > >     <profile namespace="globus" key="workersPerNode">8</profile>
> > >     <profile namespace="globus" key="ppn">8</profile>
> > >     <!-- <profile namespace="globus" key="internalHostname">172.5.86.6</profile>-->
> > >     <profile namespace="globus" key="nodeGranularity">1</profile>
> > >     <profile namespace="globus" key="maxNodes">5</profile>
> > >     <profile namespace="karajan" key="jobThrottle">1.99</profile>
> > >     <profile namespace="karajan" key="initialScore">10000</profile>
> > >     <profile namespace="globus" key="project">CI-SES000031</profile>
> > >     <!--gridftp  url="local://localhost" />-->
> > >     <filesystem provider="local"/>
> > >     <!--scratch>/home/frankwang/tmp</scratch>-->
> > >     <workdirectory>/gpfs/pads/projects/CI-SES000031/csm/tempswiftwork</workdirectory>
> > > </pool>
> > > </config>
> > >
> > > My tc.data is like,
> > > pbs             echo    /bin/echo       INSTALLED       INTEL32::LINUX  null
> > > pbs             sh      /bin/bash       INSTALLED       INTEL32::LINUX  null
> > >
> > >
> > > In my swift script I'm running some gfortran compiled programs(I wrap my fortran programs in sh files), and in my error log file, it shows that
> > >
> > > /gpfs/pads/projects/CI-SES000031/csm/DSSAT040.EXE: error while loading shared libraries: libgfortran.so.3: cannot open shared object file: No such file or directory
> > >
> > > I didn't meet the same problem when I run these on PADS, and after I swap the module to PrgEnv-gnu when logging onto Beagle I can also run this DSSAT040.EXE.
> > > My guess is,  I submit the jobs to a Non-GNU Environment in Beagle.
> > >
> > > How can I correct this?
> > >
> > > Thank you.
> > >
> > > Weiyang
> > > _______________________________________________
> > > beagle-users mailing list
> > > beagle-users at ci.uchicago.edu
> > > http://mail.ci.uchicago.edu/mailman/listinfo/beagle-users
> >
> > -----BEGIN PGP SIGNATURE-----
> > Version: GnuPG/MacGPG2 v2.0.14 (Darwin)
> >
> > iEYEARECAAYFAk223FkACgkQ4RgdOxQVi0AVxgCeNmnQ0e2xgmIZjQ3vkot1SpWf
> > H5MAnjS/VbEaH7vWSJFcSK+2hs1VHo4d
> > =8FUi
> > -----END PGP SIGNATURE-----
> >
> 
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG/MacGPG2 v2.0.14 (Darwin)
> 
> iEYEARECAAYFAk23D/YACgkQ4RgdOxQVi0DGjACeJV4MG+9SuMBKKTyECpE82Exs
> eF4AoIE7bokWJQZM44tDVit4eQvj2G8j
> =y73+
> -----END PGP SIGNATURE-----
> 

-----BEGIN PGP SIGNATURE-----
Version: GnuPG/MacGPG2 v2.0.14 (Darwin)

iEYEARECAAYFAk23ENsACgkQ4RgdOxQVi0DICgCghYevCuLZOVG+woUGZdftG5XA
LDQAnRoPIgmi8J/F/TzaiWBrya5guJeU
=DtcT
-----END PGP SIGNATURE-----


From ketan at mcs.anl.gov  Tue Apr 26 13:43:30 2011
From: ketan at mcs.anl.gov (Ketan Maheshwari)
Date: Tue, 26 Apr 2011 13:43:30 -0500
Subject: [Swift-user] Re: [beagle-users] [SWIFT on Beagle]Cannot Execute
	Correctly After Submitting Job
In-Reply-To: <BANLkTi=-Na9iumvA9gNMAH83OeugSFSOkg@mail.gmail.com>
References: <BANLkTimpoVKQ5wky=c-txqVkqhJdoW0ZPQ@mail.gmail.com>
	<23CC8309-969A-41A7-8E5D-AEA010145DEC@ci.uchicago.edu>
	<BANLkTi=-Na9iumvA9gNMAH83OeugSFSOkg@mail.gmail.com>
Message-ID: <57C38529-F633-4062-A195-864D838A3180@mcs.anl.gov>

Hello,

This indicates that the executable is not running on beagle compute nodes. Could you kindly send the commandline and the path to your exe, data etc, and I can try to do some testing using aprun and qsub.

Ketan


On Apr 26, 2011, at 1:26 PM, Weiyang Wang wrote:

> In fact I switched to PrgEnv-gnu to compile it again on Beagle, and it works if I run it plainly on login.beagle
> 
> On Tue, Apr 26, 2011 at 10:53 PM, Ti Leggett <leggett at ci.uchicago.edu> wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
> 
> Was the code you're trying to run compiled on Beagle. IT looks to me like it was compiled on PADS. You must recompile your code on Beagle using the Cray compiler wrappers. Please refer to the Beagle FAQ on how to do this:
> 
> http://www.ci.uchicago.edu/wiki/bin/view/Beagle/DevelopOnBeagle
> 
> On Apr 26, 2011, at 9:34 AM, Weiyang Wang wrote:
> 
> > Hello,
> >
> > I've meet problems when submitting jobs using Swift.
> >
> > My Command Line is like:
> >
> > swift -sites.file ./mysites.xml  -tc.file ./tc.data largescale.swift
> >
> > My mysites.xml is like
> > <config>
> > <pool handle="pbs">
> >     <execution provider="pbs" url="none" jobManager="local:pbs"/>
> >     <profile namespace="globus" key="maxwalltime">0:20:00</profile>
> >     <profile namespace="globus" key="workersPerNode">8</profile>
> >     <profile namespace="globus" key="ppn">8</profile>
> >     <!-- <profile namespace="globus" key="internalHostname">172.5.86.6</profile>-->
> >     <profile namespace="globus" key="nodeGranularity">1</profile>
> >     <profile namespace="globus" key="maxNodes">5</profile>
> >     <profile namespace="karajan" key="jobThrottle">1.99</profile>
> >     <profile namespace="karajan" key="initialScore">10000</profile>
> >     <profile namespace="globus" key="project">CI-SES000031</profile>
> >     <!--gridftp  url="local://localhost" />-->
> >     <filesystem provider="local"/>
> >     <!--scratch>/home/frankwang/tmp</scratch>-->
> >     <workdirectory>/gpfs/pads/projects/CI-SES000031/csm/tempswiftwork</workdirectory>
> > </pool>
> > </config>
> >
> > My tc.data is like,
> > pbs             echo    /bin/echo       INSTALLED       INTEL32::LINUX  null
> > pbs             sh      /bin/bash       INSTALLED       INTEL32::LINUX  null
> >
> >
> > In my swift script I'm running some gfortran compiled programs(I wrap my fortran programs in sh files), and in my error log file, it shows that
> >
> > /gpfs/pads/projects/CI-SES000031/csm/DSSAT040.EXE: error while loading shared libraries: libgfortran.so.3: cannot open shared object file: No such file or directory
> >
> > I didn't meet the same problem when I run these on PADS, and after I swap the module to PrgEnv-gnu when logging onto Beagle I can also run this DSSAT040.EXE.
> > My guess is,  I submit the jobs to a Non-GNU Environment in Beagle.
> >
> > How can I correct this?
> >
> > Thank you.
> >
> > Weiyang
> > _______________________________________________
> > beagle-users mailing list
> > beagle-users at ci.uchicago.edu
> > http://mail.ci.uchicago.edu/mailman/listinfo/beagle-users
> 
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG/MacGPG2 v2.0.14 (Darwin)
> 
> iEYEARECAAYFAk223FkACgkQ4RgdOxQVi0AVxgCeNmnQ0e2xgmIZjQ3vkot1SpWf
> H5MAnjS/VbEaH7vWSJFcSK+2hs1VHo4d
> =8FUi
> -----END PGP SIGNATURE-----
> 
> _______________________________________________
> beagle-users mailing list
> beagle-users at ci.uchicago.edu
> http://mail.ci.uchicago.edu/mailman/listinfo/beagle-users

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-user/attachments/20110426/1547da67/attachment.html>

From lpesce at uchicago.edu  Tue Apr 26 16:00:24 2011
From: lpesce at uchicago.edu (Lorenzo Pesce)
Date: Tue, 26 Apr 2011 16:00:24 -0500
Subject: [Swift-user] Re: [beagle-users] [SWIFT on Beagle]Cannot Execute
	Correctly After Submitting Job
In-Reply-To: <BANLkTikLg9M0GsRoiLTZ1p8e7JFvrBEkyg@mail.gmail.com>
References: <BANLkTimpoVKQ5wky=c-txqVkqhJdoW0ZPQ@mail.gmail.com>
	<23CC8309-969A-41A7-8E5D-AEA010145DEC@ci.uchicago.edu>
	<BANLkTi=-Na9iumvA9gNMAH83OeugSFSOkg@mail.gmail.com>
	<57C38529-F633-4062-A195-864D838A3180@mcs.anl.gov>
	<BANLkTikLg9M0GsRoiLTZ1p8e7JFvrBEkyg@mail.gmail.com>
Message-ID: <5054020D-58D7-4E6F-AE68-20046F652C9A@uchicago.edu>

Weiyang,

Can you wait until tomorrow morning? I have to run off campus right now and I do not have the time to give you a complete reply.
I should be able to look into your problem before noon or so.


Lorenzo

On Apr 26, 2011, at 2:14 PM, Weiyang Wang wrote:

> OK, I've "make clean" them and  ow I switched to PrgEnv-cray, and going to use ftn to compile some .for files, although they are currently not recognizable by ftn. I want them to be compiled as F90 files (maybe I have to rename them first? )
> 
> Could you kindly check out what's going on here within the makefile?
> 
> 
> ======= the first several lines====
> OBJDIR = objCSMINPUT
> FC     = ftn
> FFLAGS = -O 3 -g
> INC    = -I.
> MKDIR  = mkdir -p
> DIRS   = ../Common ../Input
> RM     = rm -rf
> ===========================
> 
> ===somewhere it starts compiling===
> 
> # Create a list of all source files including folder name
> FSRC   = $(foreach dir,$(DIRS),$(wildcard $(dir)/*.for))
> 
> ## Create the list of all object files
> TOBJ   = $(foreach dir, $(DIRS), $(subst .for,.o,$(notdir $(wildcard $(dir)/*.for))))
> 
> # Create the path to store the object files
> FOBJ   = $(foreach obj,$(TOBJ),$(OBJDIR)/$(obj))
> 
> # Define the subfolders to look for source files
> vpath %.for ../Common ../Input
> 
> $(OBJDIR)/%.o $(OBJDIR)/%.mod: %.for | $(OBJDIR)
>         $(FC) $(FFLAGS) $(INC) -o $@ -c $<
> 
> ../MINPT040.EXE : $(FOBJ)
>         $(FC) $^ -o $@
> 
> 
> ============================
> 
> 
> 
> 
> On Wed, Apr 27, 2011 at 2:43 AM, Ketan Maheshwari <ketan at mcs.anl.gov> wrote:
> Hello,
> 
> This indicates that the executable is not running on beagle compute nodes. Could you kindly send the commandline and the path to your exe, data etc, and I can try to do some testing using aprun and qsub.
> 
> Ketan
> 
> 
> On Apr 26, 2011, at 1:26 PM, Weiyang Wang wrote:
> 
>> In fact I switched to PrgEnv-gnu to compile it again on Beagle, and it works if I run it plainly on login.beagle
>> 
>> On Tue, Apr 26, 2011 at 10:53 PM, Ti Leggett <leggett at ci.uchicago.edu> wrote:
>> -----BEGIN PGP SIGNED MESSAGE-----
>> Hash: SHA1
>> 
>> Was the code you're trying to run compiled on Beagle. IT looks to me like it was compiled on PADS. You must recompile your code on Beagle using the Cray compiler wrappers. Please refer to the Beagle FAQ on how to do this:
>> 
>> http://www.ci.uchicago.edu/wiki/bin/view/Beagle/DevelopOnBeagle
>> 
>> On Apr 26, 2011, at 9:34 AM, Weiyang Wang wrote:
>> 
>> > Hello,
>> >
>> > I've meet problems when submitting jobs using Swift.
>> >
>> > My Command Line is like:
>> >
>> > swift -sites.file ./mysites.xml  -tc.file ./tc.data largescale.swift
>> >
>> > My mysites.xml is like
>> > <config>
>> > <pool handle="pbs">
>> >     <execution provider="pbs" url="none" jobManager="local:pbs"/>
>> >     <profile namespace="globus" key="maxwalltime">0:20:00</profile>
>> >     <profile namespace="globus" key="workersPerNode">8</profile>
>> >     <profile namespace="globus" key="ppn">8</profile>
>> >     <!-- <profile namespace="globus" key="internalHostname">172.5.86.6</profile>-->
>> >     <profile namespace="globus" key="nodeGranularity">1</profile>
>> >     <profile namespace="globus" key="maxNodes">5</profile>
>> >     <profile namespace="karajan" key="jobThrottle">1.99</profile>
>> >     <profile namespace="karajan" key="initialScore">10000</profile>
>> >     <profile namespace="globus" key="project">CI-SES000031</profile>
>> >     <!--gridftp  url="local://localhost" />-->
>> >     <filesystem provider="local"/>
>> >     <!--scratch>/home/frankwang/tmp</scratch>-->
>> >     <workdirectory>/gpfs/pads/projects/CI-SES000031/csm/tempswiftwork</workdirectory>
>> > </pool>
>> > </config>
>> >
>> > My tc.data is like,
>> > pbs             echo    /bin/echo       INSTALLED       INTEL32::LINUX  null
>> > pbs             sh      /bin/bash       INSTALLED       INTEL32::LINUX  null
>> >
>> >
>> > In my swift script I'm running some gfortran compiled programs(I wrap my fortran programs in sh files), and in my error log file, it shows that
>> >
>> > /gpfs/pads/projects/CI-SES000031/csm/DSSAT040.EXE: error while loading shared libraries: libgfortran.so.3: cannot open shared object file: No such file or directory
>> >
>> > I didn't meet the same problem when I run these on PADS, and after I swap the module to PrgEnv-gnu when logging onto Beagle I can also run this DSSAT040.EXE.
>> > My guess is,  I submit the jobs to a Non-GNU Environment in Beagle.
>> >
>> > How can I correct this?
>> >
>> > Thank you.
>> >
>> > Weiyang
>> > _______________________________________________
>> > beagle-users mailing list
>> > beagle-users at ci.uchicago.edu
>> > http://mail.ci.uchicago.edu/mailman/listinfo/beagle-users
>> 
>> -----BEGIN PGP SIGNATURE-----
>> Version: GnuPG/MacGPG2 v2.0.14 (Darwin)
>> 
>> iEYEARECAAYFAk223FkACgkQ4RgdOxQVi0AVxgCeNmnQ0e2xgmIZjQ3vkot1SpWf
>> H5MAnjS/VbEaH7vWSJFcSK+2hs1VHo4d
>> =8FUi
>> -----END PGP SIGNATURE-----
>> 
>> _______________________________________________
>> beagle-users mailing list
>> beagle-users at ci.uchicago.edu
>> http://mail.ci.uchicago.edu/mailman/listinfo/beagle-users
> 
> 
> _______________________________________________
> Swift-user mailing list
> Swift-user at ci.uchicago.edu
> http://mail.ci.uchicago.edu/mailman/listinfo/swift-user

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-user/attachments/20110426/5ae9c4a8/attachment.html>

From frank.wang04 at gmail.com  Tue Apr 26 16:02:21 2011
From: frank.wang04 at gmail.com (Weiyang Wang)
Date: Wed, 27 Apr 2011 05:02:21 +0800
Subject: [Swift-user] Re: [beagle-users] [SWIFT on Beagle]Cannot Execute
	Correctly After Submitting Job
In-Reply-To: <5054020D-58D7-4E6F-AE68-20046F652C9A@uchicago.edu>
References: <BANLkTimpoVKQ5wky=c-txqVkqhJdoW0ZPQ@mail.gmail.com>
	<23CC8309-969A-41A7-8E5D-AEA010145DEC@ci.uchicago.edu>
	<BANLkTi=-Na9iumvA9gNMAH83OeugSFSOkg@mail.gmail.com>
	<57C38529-F633-4062-A195-864D838A3180@mcs.anl.gov>
	<BANLkTikLg9M0GsRoiLTZ1p8e7JFvrBEkyg@mail.gmail.com>
	<5054020D-58D7-4E6F-AE68-20046F652C9A@uchicago.edu>
Message-ID: <BANLkTikg3qgfGJaLO4UK+=2h=aPQgL0i=w@mail.gmail.com>

OK Thanks.

I'm working on the makefile now, shall we meet tomorrow afternoon?

On Wed, Apr 27, 2011 at 5:00 AM, Lorenzo Pesce <lpesce at uchicago.edu> wrote:

> Weiyang,
>
> Can you wait until tomorrow morning? I have to run off campus right now and
> I do not have the time to give you a complete reply.
> I should be able to look into your problem before noon or so.
>
>
> Lorenzo
>
> On Apr 26, 2011, at 2:14 PM, Weiyang Wang wrote:
>
> OK, I've "make clean" them and  ow I switched to PrgEnv-cray, and going to
> use ftn to compile some .for files, although they are currently not
> recognizable by ftn. I want them to be compiled as F90 files (maybe I have
> to rename them first? )
>
> Could you kindly check out what's going on here within the makefile?
>
>
> ======= the first several lines====
> OBJDIR = objCSMINPUT
> FC     = ftn
> FFLAGS = -O 3 -g
> INC    = -I.
> MKDIR  = mkdir -p
> DIRS   = ../Common ../Input
> RM     = rm -rf
> ===========================
>
> ===somewhere it starts compiling===
>
> # Create a list of all source files including folder name
> FSRC   = $(foreach dir,$(DIRS),$(wildcard $(dir)/*.for))
>
> ## Create the list of all object files
> TOBJ   = $(foreach dir, $(DIRS), $(subst .for,.o,$(notdir $(wildcard
> $(dir)/*.for))))
>
> # Create the path to store the object files
> FOBJ   = $(foreach obj,$(TOBJ),$(OBJDIR)/$(obj))
>
> # Define the subfolders to look for source files
> vpath %.for ../Common ../Input
>
> $(OBJDIR)/%.o $(OBJDIR)/%.mod: %.for | $(OBJDIR)
>         $(FC) $(FFLAGS) $(INC) -o $@ -c $<
>
> ../MINPT040.EXE : $(FOBJ)
>         $(FC) $^ -o $@
>
>
> ============================
>
>
>
>
> On Wed, Apr 27, 2011 at 2:43 AM, Ketan Maheshwari <ketan at mcs.anl.gov>wrote:
>
>> Hello,
>>
>> This indicates that the executable is not running on beagle compute nodes.
>> Could you kindly send the commandline and the path to your exe, data etc,
>> and I can try to do some testing using aprun and qsub.
>>
>> Ketan
>>
>>
>> On Apr 26, 2011, at 1:26 PM, Weiyang Wang wrote:
>>
>> In fact I switched to PrgEnv-gnu to compile it again on Beagle, and it
>> works if I run it plainly on login.beagle
>>
>> On Tue, Apr 26, 2011 at 10:53 PM, Ti Leggett <leggett at ci.uchicago.edu>wrote:
>>
>>> -----BEGIN PGP SIGNED MESSAGE-----
>>> Hash: SHA1
>>>
>>> Was the code you're trying to run compiled on Beagle. IT looks to me like
>>> it was compiled on PADS. You must recompile your code on Beagle using the
>>> Cray compiler wrappers. Please refer to the Beagle FAQ on how to do this:
>>>
>>> http://www.ci.uchicago.edu/wiki/bin/view/Beagle/DevelopOnBeagle
>>>
>>> On Apr 26, 2011, at 9:34 AM, Weiyang Wang wrote:
>>>
>>> > Hello,
>>> >
>>> > I've meet problems when submitting jobs using Swift.
>>> >
>>> > My Command Line is like:
>>> >
>>> > swift -sites.file ./mysites.xml  -tc.file ./tc.data largescale.swift
>>> >
>>> > My mysites.xml is like
>>> > <config>
>>> > <pool handle="pbs">
>>> >     <execution provider="pbs" url="none" jobManager="local:pbs"/>
>>> >     <profile namespace="globus" key="maxwalltime">0:20:00</profile>
>>> >     <profile namespace="globus" key="workersPerNode">8</profile>
>>> >     <profile namespace="globus" key="ppn">8</profile>
>>> >     <!-- <profile namespace="globus"
>>> key="internalHostname">172.5.86.6</profile>-->
>>> >     <profile namespace="globus" key="nodeGranularity">1</profile>
>>> >     <profile namespace="globus" key="maxNodes">5</profile>
>>> >     <profile namespace="karajan" key="jobThrottle">1.99</profile>
>>> >     <profile namespace="karajan" key="initialScore">10000</profile>
>>> >     <profile namespace="globus" key="project">CI-SES000031</profile>
>>> >     <!--gridftp  url="local://localhost" />-->
>>> >     <filesystem provider="local"/>
>>> >     <!--scratch>/home/frankwang/tmp</scratch>-->
>>> >
>>> <workdirectory>/gpfs/pads/projects/CI-SES000031/csm/tempswiftwork</workdirectory>
>>> > </pool>
>>> > </config>
>>> >
>>> > My tc.data is like,
>>> > pbs             echo    /bin/echo       INSTALLED       INTEL32::LINUX
>>>  null
>>> > pbs             sh      /bin/bash       INSTALLED       INTEL32::LINUX
>>>  null
>>> >
>>> >
>>> > In my swift script I'm running some gfortran compiled programs(I wrap
>>> my fortran programs in sh files), and in my error log file, it shows that
>>> >
>>> > /gpfs/pads/projects/CI-SES000031/csm/DSSAT040.EXE: error while loading
>>> shared libraries: libgfortran.so.3: cannot open shared object file: No such
>>> file or directory
>>> >
>>> > I didn't meet the same problem when I run these on PADS, and after I
>>> swap the module to PrgEnv-gnu when logging onto Beagle I can also run this
>>> DSSAT040.EXE.
>>> > My guess is,  I submit the jobs to a Non-GNU Environment in Beagle.
>>> >
>>> > How can I correct this?
>>> >
>>> > Thank you.
>>> >
>>> > Weiyang
>>> > _______________________________________________
>>> > beagle-users mailing list
>>> > beagle-users at ci.uchicago.edu
>>> > http://mail.ci.uchicago.edu/mailman/listinfo/beagle-users
>>>
>>> -----BEGIN PGP SIGNATURE-----
>>> Version: GnuPG/MacGPG2 v2.0.14 (Darwin)
>>>
>>> iEYEARECAAYFAk223FkACgkQ4RgdOxQVi0AVxgCeNmnQ0e2xgmIZjQ3vkot1SpWf
>>> H5MAnjS/VbEaH7vWSJFcSK+2hs1VHo4d
>>> =8FUi
>>> -----END PGP SIGNATURE-----
>>>
>>
>> _______________________________________________
>> beagle-users mailing list
>> beagle-users at ci.uchicago.edu
>> http://mail.ci.uchicago.edu/mailman/listinfo/beagle-users
>>
>>
>>
> _______________________________________________
> Swift-user mailing list
> Swift-user at ci.uchicago.edu
> http://mail.ci.uchicago.edu/mailman/listinfo/swift-user
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-user/attachments/20110427/2e813ee1/attachment.html>

From tianyu491433909 at 163.com  Wed Apr 27 04:50:35 2011
From: tianyu491433909 at 163.com (tianyu491433909)
Date: Wed, 27 Apr 2011 17:50:35 +0800 (CST)
Subject: [Swift-user] swift-plot-log execution failed
Message-ID: <459eab69.bb75.12f965e085c.Coremail.tianyu491433909@163.com>

swift-plot-log /home/cloudian/workspace/.metadata/.me_tcat/webapps/cloudswift/logs/1303886516764/1303886516764-20110427-1459-46i0xjlc.log

I have installed with gnuplot 4.0, gnu m4, gnu textutils, perl.

The problem is that why does it execution failed
logs:

extract-start-time > start-time.tmp
head: Unable to open ?*.event? Read data: No such file or directory
extract-end-time > end-time.tmp
sort: Open failure: *.event: No such file or directory
whole-workflow-event /home/cloudian/workspace/.metadata/.me_tcat/webapps/cloudswift/logs/1303886516764/1303886516764-20110427-1459-46i0xjlc.log > workflow.event
cat karatasks.transitions | swap-and-sort | transitions-to-event > karatasks.event
simple-event-plot karatasks.event karatasks.png

events-in-progress < karatasks.JOB_SUBMISSION.Queue.event > karatasks.JOB_SUBMISSION.Queue.eip
event-duration-stats karatasks.JOB_SUBMISSION.Queue.event > karatasks.JOB_SUBMISSION.Queue.stats
cat karatasks.JOB_SUBMISSION.Queue.event | sort -n -k 2 > karatasks.JOB_SUBMISSION.Queue.sorted-by-duration
plot-duration-histogram karatasks.JOB_SUBMISSION.Queue.sorted-by-duration karatasks.JOB_SUBMISSION.Queue-duration-histogram.png
weights.sh /home/cloudian/workspace/.metadata/.me_tcat/webapps/cloudswift/logs/1303886516764/1303886516764-20110427-1459-46i0xjlc.log
tscore.sh /home/cloudian/workspace/.metadata/.me_tcat/webapps/cloudswift/logs/1303886516764/1303886516764-20110427-1459-46i0xjlc.log | normalise-event-start-time | sed 's/\([^ ]*\) [^ ]* [^ ]* [^ ]* \([^ ]*\) \([^ ]*\).*/\1 \2 \3/' > tscores.data
gnuplot /home/cloudian/dist/swift-svn/bin/../libexec/log-processing//tscores.plot
extract-coaster-timeline /home/cloudian/workspace/.metadata/.me_tcat/webapps/cloudswift/logs/1303886516764/1303886516764-20110427-1459-46i0xjlc.log
gnuplot /home/cloudian/dist/swift-svn/bin/../libexec/log-processing//coaster-block-timeline.plot
"/home/cloudian/dist/swift-svn/bin/../libexec/log-processing//coaster-block-timeline.plot", line 6: warning: Skipping unreadable file "coaster-blocks.data"
"/home/cloudian/dist/swift-svn/bin/../libexec/log-processing//coaster-block-timeline.plot", line 6: warning: Skipping unreadable file "coaster-blocks.data"
"/home/cloudian/dist/swift-svn/bin/../libexec/log-processing//coaster-block-timeline.plot", line 6: No data in plot
make: *** [coaster-block-timeline.png] error 1
rm karatasks.FILE_OPERATION.sorted-start.event karatasks.JOB_SUBMISSION.Active.event karatasks.JOB_SUBMISSION.Queue.sorted-start.event karatasks.last karatasks.JOB_SUBMISSION.seenstates karatasks.FILE_TRANSFER.sorted-start.event karatasks.JOB_SUBMISSION.Queue.eip karatasks.JOB_SUBMISSION.Active.sorted-start.event karatasks.FILE_OPERATION.seenstates karatasks.JOB_SUBMISSION.Queue.sorted-by-duration karatasks.FILE_OPERATION.eip karatasks.JOB_SUBMISSION.event karatasks.FILE_TRANSFER.seenstates karatasks.FILE_OPERATION.event karatasks.FILE_TRANSFER.eip karatasks.JOB_SUBMISSION.Active.sorted-by-duration karatasks.JOB_SUBMISSION.Active.eip karatasks.FILE_TRANSFER.event karatasks.JOB_SUBMISSION.sorted-start.event
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-user/attachments/20110427/fe6a11a7/attachment.html>

From wozniak at mcs.anl.gov  Thu Apr 28 18:02:34 2011
From: wozniak at mcs.anl.gov (Justin M Wozniak)
Date: Thu, 28 Apr 2011 18:02:34 -0500 (Central Daylight Time)
Subject: [Swift-user] swift-plot-log execution failed
In-Reply-To: <459eab69.bb75.12f965e085c.Coremail.tianyu491433909@163.com>
References: <459eab69.bb75.12f965e085c.Coremail.tianyu491433909@163.com>
Message-ID: <alpine.WNT.2.00.1104281801280.4776@JWOZNIAK-DESK>

Hi
 	I don't think we are supporting these scripts any more, however, 
many of us do generate various plots.  Is there a type of plot you are 
interested in?
 	Justin

On Wed, 27 Apr 2011, tianyu491433909 wrote:

> swift-plot-log 
> /home/cloudian/workspace/.metadata/.me_tcat/webapps/cloudswift/logs/1303886516764/1303886516764-20110427-1459-46i0xjlc.log
>
> I have installed with gnuplot 4.0, gnu m4, gnu textutils, perl.
>
> The problem is that why does it execution failed
> logs:
>
> extract-start-time > start-time.tmp
> head: Unable to open ?*.event? Read data: No such file or directory
> extract-end-time > end-time.tmp
> sort: Open failure: *.event: No such file or directory
> whole-workflow-event /home/cloudian/workspace/.metadata/.me_tcat/webapps/cloudswift/logs/1303886516764/1303886516764-20110427-1459-46i0xjlc.log > workflow.event
> cat karatasks.transitions | swap-and-sort | transitions-to-event > karatasks.event
> simple-event-plot karatasks.event karatasks.png
>
> events-in-progress < karatasks.JOB_SUBMISSION.Queue.event > karatasks.JOB_SUBMISSION.Queue.eip
> event-duration-stats karatasks.JOB_SUBMISSION.Queue.event > karatasks.JOB_SUBMISSION.Queue.stats
> cat karatasks.JOB_SUBMISSION.Queue.event | sort -n -k 2 > karatasks.JOB_SUBMISSION.Queue.sorted-by-duration
> plot-duration-histogram karatasks.JOB_SUBMISSION.Queue.sorted-by-duration karatasks.JOB_SUBMISSION.Queue-duration-histogram.png
> weights.sh /home/cloudian/workspace/.metadata/.me_tcat/webapps/cloudswift/logs/1303886516764/1303886516764-20110427-1459-46i0xjlc.log
> tscore.sh /home/cloudian/workspace/.metadata/.me_tcat/webapps/cloudswift/logs/1303886516764/1303886516764-20110427-1459-46i0xjlc.log | normalise-event-start-time | sed 's/\([^ ]*\) [^ ]* [^ ]* [^ ]* \([^ ]*\) \([^ ]*\).*/\1 \2 \3/' > tscores.data
> gnuplot /home/cloudian/dist/swift-svn/bin/../libexec/log-processing//tscores.plot
> extract-coaster-timeline /home/cloudian/workspace/.metadata/.me_tcat/webapps/cloudswift/logs/1303886516764/1303886516764-20110427-1459-46i0xjlc.log
> gnuplot /home/cloudian/dist/swift-svn/bin/../libexec/log-processing//coaster-block-timeline.plot
> "/home/cloudian/dist/swift-svn/bin/../libexec/log-processing//coaster-block-timeline.plot", line 6: warning: Skipping unreadable file "coaster-blocks.data"
> "/home/cloudian/dist/swift-svn/bin/../libexec/log-processing//coaster-block-timeline.plot", line 6: warning: Skipping unreadable file "coaster-blocks.data"
> "/home/cloudian/dist/swift-svn/bin/../libexec/log-processing//coaster-block-timeline.plot", line 6: No data in plot
> make: *** [coaster-block-timeline.png] error 1
> rm karatasks.FILE_OPERATION.sorted-start.event karatasks.JOB_SUBMISSION.Active.event karatasks.JOB_SUBMISSION.Queue.sorted-start.event karatasks.last karatasks.JOB_SUBMISSION.seenstates karatasks.FILE_TRANSFER.sorted-start.event karatasks.JOB_SUBMISSION.Queue.eip karatasks.JOB_SUBMISSION.Active.sorted-start.event karatasks.FILE_OPERATION.seenstates karatasks.JOB_SUBMISSION.Queue.sorted-by-duration karatasks.FILE_OPERATION.eip karatasks.JOB_SUBMISSION.event karatasks.FILE_TRANSFER.seenstates karatasks.FILE_OPERATION.event karatasks.FILE_TRANSFER.eip karatasks.JOB_SUBMISSION.Active.sorted-by-duration karatasks.JOB_SUBMISSION.Active.eip karatasks.FILE_TRANSFER.event karatasks.JOB_SUBMISSION.sorted-start.event

-- 
Justin M Wozniak

From frank.wang04 at gmail.com  Thu Apr 28 19:10:04 2011
From: frank.wang04 at gmail.com (Weiyang Wang)
Date: Thu, 28 Apr 2011 19:10:04 -0500
Subject: [Swift-user] [SWIFT JOBS]How To Efficiently Set Sites.xml
Message-ID: <BANLkTin6w8mjQML+-d8EWAgrzwtFXi4zHA@mail.gmail.com>

Hello,

Currently my submitted SWIFT jobs are seemed run single-thread, while my
sites.xml is like

<config>
<pool handle="pbs">
    <execution provider="pbs" url="none" jobManager="local:pbs"/>
    <profile namespace="globus" key="maxwalltime">0:20:00</profile>
    <profile namespace="globus" key="workersPerNode">8</profile>
    <profile namespace="globus" key="ppn">8</profile>
    <!-- <profile namespace="globus"
key="internalHostname">172.5.86.6</profile>-->
    <profile namespace="globus" key="nodeGranularity">1</profile>
    <profile namespace="globus" key="maxNodes">5</profile>
    <profile namespace="karajan" key="jobThrottle">1.99</profile>
    <profile namespace="karajan" key="initialScore">10000</profile>
    <profile namespace="globus" key="project">CI-SES000031</profile>
    <!--gridftp  url="local://localhost" />-->
    <filesystem provider="local"/>
    <!--scratch>/home/frankwang/tmp</scratch>-->
    <workdirectory>/lustre/beagle/scratch</workdirectory>
</pool>
</config>

Do any of you have ideas to adjust it so that it'll be more efficient?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-user/attachments/20110428/b5a656ac/attachment.html>

From ketancmaheshwari at gmail.com  Thu Apr 28 19:28:57 2011
From: ketancmaheshwari at gmail.com (Ketan Maheshwari)
Date: Thu, 28 Apr 2011 19:28:57 -0500
Subject: [Swift-user] [SWIFT JOBS]How To Efficiently Set Sites.xml
In-Reply-To: <BANLkTin6w8mjQML+-d8EWAgrzwtFXi4zHA@mail.gmail.com>
References: <BANLkTin6w8mjQML+-d8EWAgrzwtFXi4zHA@mail.gmail.com>
Message-ID: <BANLkTi=XUvDXUa1-y-pL3jHnPJA87W514Q@mail.gmail.com>

Hi Weiyang,

Ideally workerspernode and ppn should be 24 on beagle to reflect 24 cores
per node.

That should make maximum utilization of a node.

--
Ketan


On Thu, Apr 28, 2011 at 7:10 PM, Weiyang Wang <frank.wang04 at gmail.com>wrote:

> Hello,
>
> Currently my submitted SWIFT jobs are seemed run single-thread, while my
> sites.xml is like
>
> <config>
> <pool handle="pbs">
>     <execution provider="pbs" url="none" jobManager="local:pbs"/>
>     <profile namespace="globus" key="maxwalltime">0:20:00</profile>
>     <profile namespace="globus" key="workersPerNode">8</profile>
>     <profile namespace="globus" key="ppn">8</profile>
>     <!-- <profile namespace="globus"
> key="internalHostname">172.5.86.6</profile>-->
>     <profile namespace="globus" key="nodeGranularity">1</profile>
>     <profile namespace="globus" key="maxNodes">5</profile>
>     <profile namespace="karajan" key="jobThrottle">1.99</profile>
>     <profile namespace="karajan" key="initialScore">10000</profile>
>     <profile namespace="globus" key="project">CI-SES000031</profile>
>     <!--gridftp  url="local://localhost" />-->
>     <filesystem provider="local"/>
>     <!--scratch>/home/frankwang/tmp</scratch>-->
>     <workdirectory>/lustre/beagle/scratch</workdirectory>
> </pool>
> </config>
>
> Do any of you have ideas to adjust it so that it'll be more efficient?
>
> _______________________________________________
> Swift-user mailing list
> Swift-user at ci.uchicago.edu
> http://mail.ci.uchicago.edu/mailman/listinfo/swift-user
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-user/attachments/20110428/54d64ad9/attachment.html>

From wilde at mcs.anl.gov  Thu Apr 28 19:40:22 2011
From: wilde at mcs.anl.gov (Michael Wilde)
Date: Thu, 28 Apr 2011 19:40:22 -0500 (CDT)
Subject: [Swift-user] [SWIFT JOBS]How To Efficiently Set Sites.xml
In-Reply-To: <BANLkTi=XUvDXUa1-y-pL3jHnPJA87W514Q@mail.gmail.com>
Message-ID: <1252677112.5500.1304037622608.JavaMail.root@zimbra.anl.gov>

And also, Weiyang: 



Your sites.xml entry says "provider=pbs". That does not work on Beagle. In fact, it causes jobs to run on the login node. (Note to Swift developers: we should prevent this from happening by accident). 


change execution provider="pbs" to execution provider="coaster" 
in the line <execution provider="pbs" url="none" jobManager="local:pbs"/> 


I did not check the rest of your sites file. 


Ketan, can you point Wei to the Beagle Swift instructions? 


Thanks, 

- Mike 
----- Original Message -----


Hi Weiyang, 

Ideally workerspernode and ppn should be 24 on beagle to reflect 24 cores per node. 

That should make maximum utilization of a node. 

-- 
Ketan 



On Thu, Apr 28, 2011 at 7:10 PM, Weiyang Wang < frank.wang04 at gmail.com > wrote: 


Hello, 


Currently my submitted SWIFT jobs are seemed run single-thread, while my sites.xml is like 



<config> 
<pool handle="pbs"> 
<execution provider="pbs" url="none" jobManager="local:pbs"/> 
<profile namespace="globus" key="maxwalltime">0:20:00</profile> 
<profile namespace="globus" key="workersPerNode">8</profile> 
<profile namespace="globus" key="ppn">8</profile> 
<!-- <profile namespace="globus" key="internalHostname">172.5.86.6</profile>--> 
<profile namespace="globus" key="nodeGranularity">1</profile> 
<profile namespace="globus" key="maxNodes">5</profile> 
<profile namespace="karajan" key="jobThrottle">1.99</profile> 
<profile namespace="karajan" key="initialScore">10000</profile> 
<profile namespace="globus" key="project">CI-SES000031</profile> 
<!--gridftp url="local://localhost" />--> 
<filesystem provider="local"/> 
<!--scratch>/home/frankwang/tmp</scratch>--> 
<workdirectory>/lustre/beagle/scratch</workdirectory> 
</pool> 
</config> 


Do any of you have ideas to adjust it so that it'll be more efficient? 
_______________________________________________ 
Swift-user mailing list 
Swift-user at ci.uchicago.edu 
http://mail.ci.uchicago.edu/mailman/listinfo/swift-user 


_______________________________________________ 
Swift-user mailing list 
Swift-user at ci.uchicago.edu 
http://mail.ci.uchicago.edu/mailman/listinfo/swift-user 


-- 
Michael Wilde 
Computation Institute, University of Chicago 
Mathematics and Computer Science Division 
Argonne National Laboratory 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-user/attachments/20110428/d3ca130f/attachment.html>

From ketancmaheshwari at gmail.com  Thu Apr 28 19:56:40 2011
From: ketancmaheshwari at gmail.com (Ketan Maheshwari)
Date: Thu, 28 Apr 2011 19:56:40 -0500
Subject: [Swift-user] [SWIFT JOBS]How To Efficiently Set Sites.xml
In-Reply-To: <1252677112.5500.1304037622608.JavaMail.root@zimbra.anl.gov>
References: <1252677112.5500.1304037622608.JavaMail.root@zimbra.anl.gov>
Message-ID: <D36E9E2D-A719-4D8C-A639-326239331911@gmail.com>

Hello Weiyang,

As Mike instructed, please find a concise swift-on-beagle guide with example here:

http://www.ci.uchicago.edu/wiki/bin/view/Beagle/ComputeOnBeagle#Running_Swift_on_Beagle

Ketan

On Apr 28, 2011, at 7:40 PM, Michael Wilde wrote:

> And also, Weiyang: 
> 
> Your sites.xml entry says "provider=pbs". That does not work on Beagle. In fact, it causes jobs to run on the login node. (Note to Swift developers: we should prevent this from happening by accident). 
> 
> change execution provider="pbs" to execution provider="coaster" 
> in the line <execution provider="pbs" url="none" jobManager="local:pbs"/> 
> 
> I did not check the rest of your sites file.
> 
> Ketan, can you point Wei to the Beagle Swift instructions?
> 
> Thanks,
> 
> - Mike
> Hi Weiyang,
> 
> Ideally workerspernode and ppn should be 24 on beagle to reflect 24 cores per node.
> 
> That should make maximum utilization of a node.
> 
> --
> Ketan
> 
> 
> On Thu, Apr 28, 2011 at 7:10 PM, Weiyang Wang <frank.wang04 at gmail.com> wrote:
> Hello,
> 
> Currently my submitted SWIFT jobs are seemed run single-thread, while my sites.xml is like
> 
> <config>
> <pool handle="pbs">
>     <execution provider="pbs" url="none" jobManager="local:pbs"/>
>     <profile namespace="globus" key="maxwalltime">0:20:00</profile>
>     <profile namespace="globus" key="workersPerNode">8</profile>
>     <profile namespace="globus" key="ppn">8</profile>
>     <!-- <profile namespace="globus" key="internalHostname">172.5.86.6</profile>-->
>     <profile namespace="globus" key="nodeGranularity">1</profile>
>     <profile namespace="globus" key="maxNodes">5</profile>
>     <profile namespace="karajan" key="jobThrottle">1.99</profile>
>     <profile namespace="karajan" key="initialScore">10000</profile>
>     <profile namespace="globus" key="project">CI-SES000031</profile>
>     <!--gridftp  url="local://localhost" />-->
>     <filesystem provider="local"/>
>     <!--scratch>/home/frankwang/tmp</scratch>-->
>     <workdirectory>/lustre/beagle/scratch</workdirectory>
> </pool>
> </config>
> 
> Do any of you have ideas to adjust it so that it'll be more efficient?
> 
> _______________________________________________
> Swift-user mailing list
> Swift-user at ci.uchicago.edu
> http://mail.ci.uchicago.edu/mailman/listinfo/swift-user
> 
> 
> _______________________________________________
> Swift-user mailing list
> Swift-user at ci.uchicago.edu
> http://mail.ci.uchicago.edu/mailman/listinfo/swift-user
> 
> 
> 
> -- 
> Michael Wilde
> Computation Institute, University of Chicago
> Mathematics and Computer Science Division
> Argonne National Laboratory
> 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-user/attachments/20110428/e1ef5848/attachment.html>

From aespinosa at cs.uchicago.edu  Thu Apr 28 20:25:10 2011
From: aespinosa at cs.uchicago.edu (Allan Espinosa)
Date: Thu, 28 Apr 2011 20:25:10 -0500
Subject: [Swift-user] swift-plot-log execution failed
In-Reply-To: <alpine.WNT.2.00.1104281801280.4776@JWOZNIAK-DESK>
References: <459eab69.bb75.12f965e085c.Coremail.tianyu491433909@163.com>
	<alpine.WNT.2.00.1104281801280.4776@JWOZNIAK-DESK>
Message-ID: <BANLkTikC-nuUQFVwROZGJjHWA_h_We61AA@mail.gmail.com>

I manually specify the plots I want so I don't have to do the entire
plot process where some components are broken.

i.e for job execution:
$swift-plot-log logfile execute2.sorted-start.png execute2-total.png

2011/4/28 Justin M Wozniak <wozniak at mcs.anl.gov>:
> Hi
> ? ? ? ?I don't think we are supporting these scripts any more, however, many
> of us do generate various plots. ?Is there a type of plot you are interested
> in?
> ? ? ? ?Justin
>
> On Wed, 27 Apr 2011, tianyu491433909 wrote:
>
>> swift-plot-log
>> /home/cloudian/workspace/.metadata/.me_tcat/webapps/cloudswift/logs/1303886516764/1303886516764-20110427-1459-46i0xjlc.log
>>
>> I have installed with gnuplot 4.0, gnu m4, gnu textutils, perl.
>>
>> The problem is that why does it execution failed
>> logs:
>>
>> extract-start-time > start-time.tmp
>> head: Unable to open ?*.event? Read data: No such file or directory
>> extract-end-time > end-time.tmp
>> sort: Open failure: *.event: No such file or directory
>> whole-workflow-event
>> /home/cloudian/workspace/.metadata/.me_tcat/webapps/cloudswift/logs/1303886516764/1303886516764-20110427-1459-46i0xjlc.log
>> > workflow.event
>> cat karatasks.transitions | swap-and-sort | transitions-to-event >
>> karatasks.event
>> simple-event-plot karatasks.event karatasks.png
>>
>> events-in-progress < karatasks.JOB_SUBMISSION.Queue.event >
>> karatasks.JOB_SUBMISSION.Queue.eip
>> event-duration-stats karatasks.JOB_SUBMISSION.Queue.event >
>> karatasks.JOB_SUBMISSION.Queue.stats
>> cat karatasks.JOB_SUBMISSION.Queue.event | sort -n -k 2 >
>> karatasks.JOB_SUBMISSION.Queue.sorted-by-duration
>> plot-duration-histogram karatasks.JOB_SUBMISSION.Queue.sorted-by-duration
>> karatasks.JOB_SUBMISSION.Queue-duration-histogram.png
>> weights.sh
>> /home/cloudian/workspace/.metadata/.me_tcat/webapps/cloudswift/logs/1303886516764/1303886516764-20110427-1459-46i0xjlc.log
>> tscore.sh
>> /home/cloudian/workspace/.metadata/.me_tcat/webapps/cloudswift/logs/1303886516764/1303886516764-20110427-1459-46i0xjlc.log
>> | normalise-event-start-time | sed 's/\([^ ]*\) [^ ]* [^ ]* [^ ]* \([^ ]*\)
>> \([^ ]*\).*/\1 \2 \3/' > tscores.data
>> gnuplot
>> /home/cloudian/dist/swift-svn/bin/../libexec/log-processing//tscores.plot
>> extract-coaster-timeline
>> /home/cloudian/workspace/.metadata/.me_tcat/webapps/cloudswift/logs/1303886516764/1303886516764-20110427-1459-46i0xjlc.log
>> gnuplot
>> /home/cloudian/dist/swift-svn/bin/../libexec/log-processing//coaster-block-timeline.plot
>>
>> "/home/cloudian/dist/swift-svn/bin/../libexec/log-processing//coaster-block-timeline.plot",
>> line 6: warning: Skipping unreadable file "coaster-blocks.data"
>>
>> "/home/cloudian/dist/swift-svn/bin/../libexec/log-processing//coaster-block-timeline.plot",
>> line 6: warning: Skipping unreadable file "coaster-blocks.data"
>>
>> "/home/cloudian/dist/swift-svn/bin/../libexec/log-processing//coaster-block-timeline.plot",
>> line 6: No data in plot
>> make: *** [coaster-block-timeline.png] error 1
>> rm karatasks.FILE_OPERATION.sorted-start.event
>> karatasks.JOB_SUBMISSION.Active.event
>> karatasks.JOB_SUBMISSION.Queue.sorted-start.event karatasks.last
>> karatasks.JOB_SUBMISSION.seenstates
>> karatasks.FILE_TRANSFER.sorted-start.event
>> karatasks.JOB_SUBMISSION.Queue.eip
>> karatasks.JOB_SUBMISSION.Active.sorted-start.event
>> karatasks.FILE_OPERATION.seenstates
>> karatasks.JOB_SUBMISSION.Queue.sorted-by-duration
>> karatasks.FILE_OPERATION.eip karatasks.JOB_SUBMISSION.event
>> karatasks.FILE_TRANSFER.seenstates karatasks.FILE_OPERATION.event
>> karatasks.FILE_TRANSFER.eip
>> karatasks.JOB_SUBMISSION.Active.sorted-by-duration
>> karatasks.JOB_SUBMISSION.Active.eip karatasks.FILE_TRANSFER.event
>> karatasks.JOB_SUBMISSION.sorted-start.event
>
> --