From dk0966 at cs.ship.edu  Thu Jun  3 03:17:22 2010
From: dk0966 at cs.ship.edu (David Kelly)
Date: Thu, 3 Jun 2010 04:17:22 -0400
Subject: [Swift-user] Updated Swift Tutorial
Message-ID: <AANLkTildN21bjvm6PZioWgy13VW4ZwWp-8HFMa8B4alI@mail.gmail.com>

Hello all,

Part of my project this summer involves updating and improving the Swift
tutorials. The latest versions of Swift (updated earlier this week) includes
many corrections and some new example scripts. The corrected tutorial can be
found at http://www.ci.uchicago.edu/swift/guides/tutorial.php. If you had
problems with the previous example scripts, or haven't looked at it yet, the
tutorial may help you gain a better understanding of the software. Please
feel free to email me with any feedback, problems, or suggestions for
improvement.

Thank you!

Regards,
David
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-user/attachments/20100603/1769adf3/attachment.html>

From benc at hawaga.org.uk  Thu Jun  3 07:10:00 2010
From: benc at hawaga.org.uk (Ben Clifford)
Date: Thu, 3 Jun 2010 12:10:00 +0000 (GMT)
Subject: [Swift-user] Updated Swift Tutorial
In-Reply-To: <AANLkTildN21bjvm6PZioWgy13VW4ZwWp-8HFMa8B4alI@mail.gmail.com>
References: <AANLkTildN21bjvm6PZioWgy13VW4ZwWp-8HFMa8B4alI@mail.gmail.com>
Message-ID: <Pine.LNX.4.64.1006031204280.21645@dildano.hawaga.org.uk>


Not particularly a comment on the tutorial, but looking at that now makes 
me pretty convinced that:

  type file;

should be defined by default.

> A development version of Swift after 0.2 (revision 1230) introduces a 
> sequential iteration construct.

Thats so far in the past that you can maybe not comment about it?

> 4.3. Writing a mapper

That talks about writing stuff in java. I think experience shows that 
people prefer to express their own mappings using the ext mapper - so that 
whole 4.3 section could turn into an ext mapper tutorial.

> 5.1. Named and optional parameters

does anyone use those in real life? if not then maybe its not worth being 
in a basic tutorial.


-- 
http://www.hawaga.org.uk/ben/



From wilde at mcs.anl.gov  Thu Jun  3 10:53:30 2010
From: wilde at mcs.anl.gov (wilde at mcs.anl.gov)
Date: Thu, 3 Jun 2010 10:53:30 -0500 (CDT)
Subject: [Swift-user] Updated Swift Tutorial
In-Reply-To: <30875971.341101275579932649.JavaMail.root@zimbra>
Message-ID: <25027701.341851275580410549.JavaMail.root@zimbra>

Thanks, Ben - good points all. Some notes below.

----- "Ben Clifford" <benc at hawaga.org.uk> wrote:

> Not particularly a comment on the tutorial, but looking at that now
> makes 
> me pretty convinced that:
> 
>   type file;
> 
> should be defined by default.

Yes. Its low prio but if anyone wants to try this, feel free.

> > A development version of Swift after 0.2 (revision 1230) introduces
> a 
> > sequential iteration construct.
> 
> Thats so far in the past that you can maybe not comment about it?
> 

yes

> > 4.3. Writing a mapper
> 
> That talks about writing stuff in java. I think experience shows that
> 
> people prefer to express their own mappings using the ext mapper - so
> that 
> whole 4.3 section could turn into an ext mapper tutorial.

yes; a good action item for you, David.

We should move the java mapper text to an appendix.

> > 5.1. Named and optional parameters
> 
> does anyone use those in real life? if not then maybe its not worth
> being 
> in a basic tutorial.

Honestly, I forgot they that were there. But many of the science functions are getting pretty long arg lists and would benefit from this, so Im inclined to leave this in for the moment and see how real usage works out.

Lastly, back on the "type file" issue, I'd like to note:

Just defining file by default seems relatively simple.

It leads to the question of what we call the file reference type. And how it relates to "external". We left this issue dangling after some discussion last year.

There's a yet more complex discussion as to whether "file" should be an intrinsic type, and whether one would then say:

type textfile file;

And then consider:

type runflag boolean;
type status int;

Its best we defer anything complex in this area to a much later language summit, if and when we have the resources and user demand to consider the deeper type issues.
 
- Mike

> -- 
> http://www.hawaga.org.uk/ben/
> 
> _______________________________________________
> Swift-user mailing list
> Swift-user at ci.uchicago.edu
> http://mail.ci.uchicago.edu/mailman/listinfo/swift-user

-- 
Michael Wilde
Computation Institute, University of Chicago
Mathematics and Computer Science Division
Argonne National Laboratory



From ketancmaheshwari at gmail.com  Mon Jun  7 07:42:46 2010
From: ketancmaheshwari at gmail.com (Ketan Maheshwari)
Date: Mon, 7 Jun 2010 14:42:46 +0200
Subject: [Swift-user] swift interface with egee/glite
Message-ID: <AANLkTilkOMbhAcg614jhsVQHmYasvX6hOz3vK7hu2d1d@mail.gmail.com>

Hello

Are there any resources available that can help to interface swift/karajan
to egee/glite?

Regards
Ketan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-user/attachments/20100607/639ed5ed/attachment.html>

From benc at hawaga.org.uk  Mon Jun  7 08:14:49 2010
From: benc at hawaga.org.uk (Ben Clifford)
Date: Mon, 7 Jun 2010 13:14:49 +0000 (GMT)
Subject: [Swift-user] swift interface with egee/glite
In-Reply-To: <AANLkTilkOMbhAcg614jhsVQHmYasvX6hOz3vK7hu2d1d@mail.gmail.com>
References: <AANLkTilkOMbhAcg614jhsVQHmYasvX6hOz3vK7hu2d1d@mail.gmail.com>
Message-ID: <Pine.LNX.4.64.1006071313580.21645@dildano.hawaga.org.uk>


> Are there any resources available that can help to interface swift/karajan
> to egee/glite?

I once experimented with Swift submitting to WMS, with the south african 
grid people, who use the egee stack. The notes I wrote about are in this 
thread:

http://mail.ci.uchicago.edu/pipermail/swift-devel/2009-April/005238.html

-- 



From ketancmaheshwari at gmail.com  Mon Jun  7 08:34:02 2010
From: ketancmaheshwari at gmail.com (Ketan Maheshwari)
Date: Mon, 7 Jun 2010 15:34:02 +0200
Subject: [Swift-user] swift interface with egee/glite
In-Reply-To: <Pine.LNX.4.64.1006071313580.21645@dildano.hawaga.org.uk>
References: <AANLkTilkOMbhAcg614jhsVQHmYasvX6hOz3vK7hu2d1d@mail.gmail.com>
	<Pine.LNX.4.64.1006071313580.21645@dildano.hawaga.org.uk>
Message-ID: <AANLkTilN73JGB18fZESCe9Ir15rFXBcvXHm-N_lqw7Qx@mail.gmail.com>

Hi Ben

Thanks for the link. I went through it (H14n$=N:t)Z: your random string that
makes you owe me a free beer ;)).

You emphasize that EGEE does not have a shared file system and coasters that
can make swift system lose some of its performance.

More questions:

1. EGEE has LFC a common file catalog and a replica handling system.
Couldn't it be treated as a shared file system?

2. Is it possible to make swift talk to LFC and map my workflow data with
the LFC filesystem (accessed through lfn:/filename protocol)

3. What is the function of a coaster?

Regards
Ketan

On Mon, Jun 7, 2010 at 3:14 PM, Ben Clifford <benc at hawaga.org.uk> wrote:

>
> > Are there any resources available that can help to interface
> swift/karajan
> > to egee/glite?
>
> I once experimented with Swift submitting to WMS, with the south african
> grid people, who use the egee stack. The notes I wrote about are in this
> thread:
>
> http://mail.ci.uchicago.edu/pipermail/swift-devel/2009-April/005238.html
>
> --
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-user/attachments/20100607/26464672/attachment.html>

From benc at hawaga.org.uk  Mon Jun  7 08:47:36 2010
From: benc at hawaga.org.uk (Ben Clifford)
Date: Mon, 7 Jun 2010 13:47:36 +0000 (GMT)
Subject: [Swift-user] swift interface with egee/glite
In-Reply-To: <AANLkTilN73JGB18fZESCe9Ir15rFXBcvXHm-N_lqw7Qx@mail.gmail.com>
References: <AANLkTilkOMbhAcg614jhsVQHmYasvX6hOz3vK7hu2d1d@mail.gmail.com>
	<Pine.LNX.4.64.1006071313580.21645@dildano.hawaga.org.uk>
	<AANLkTilN73JGB18fZESCe9Ir15rFXBcvXHm-N_lqw7Qx@mail.gmail.com>
Message-ID: <Pine.LNX.4.64.1006071341060.5369@dildano.hawaga.org.uk>


> 1. EGEE has LFC a common file catalog and a replica handling system.
> Couldn't it be treated as a shared file system?

Shared file systems mean different things to different people.

In the context of Swift at the time that I wrote that prototype, the 
needed shared filesystem was posix-like (ie. you could use cp or 
fopen()) filesystem shared between where Swift uploads/downloads files and 
where programs actually run. In the traditional OSG cluster model which 
influenced the original site model, that is eg. an NFS filesystem running 
on a cluster head node.

Having replica management and a file catalog provides a different 
interface. I played a little bit with trying to get that to work once but 
ran into a stumbling block - how do I get the actual executable onto a 
worker node if I can only put files into the shared file catalog rather 
than onto the worker node's posix file system? That led me to the approach 
I took in the email that I sent you (although its probably not the only 
approach), I think.

The other questions aren't really specific to the prototype I worked on so 
I'll leave for other people to answer in line with the current orthodoxy.

-- 



From aespinosa at cs.uchicago.edu  Mon Jun 14 13:56:01 2010
From: aespinosa at cs.uchicago.edu (Allan Espinosa)
Date: Mon, 14 Jun 2010 13:56:01 -0500
Subject: [Swift-user] workflow 'pauses' after a few thousand jobs
Message-ID: <20100614185448.GA26114@communicado.ci.uchicago.edu>

 attached the coaster monitoring screenshot and the logfile.  Below is the snippet of the logfile when the workflow "hangs":

2010-06-13 16:53:55,077-0500 DEBUG vdl:dostagein CDM: file://localhost/LGU/LGU_fy_664.sgt : DIRECT
2010-06-13 16:53:55,077-0500 DEBUG vdl:dostageinfile FILE_STAGE_IN_START file=124_400.txt.variation-s0013-h0006 srchost=localhost srcdir=124/400 srcname=124_400.txt.variation-s0013-h0006 desthost=FIREFLY destdir=postproc-LGU-firefly_coast1/shared/124/400 provider=file policy=DIRECT
2010-06-13 16:53:55,077-0500 DEBUG vdl:dostageinfile FILE_STAGE_IN_SKIP file=124_400.txt.variation-s0013-h0006 policy=DIRECT
2010-06-13 16:53:55,077-0500 DEBUG vdl:dostageinfile FILE_STAGE_IN_END file=124_400.txt.variation-s0013-h0006 srchost=localhost srcdir=124/400 srcname=124_400.txt.variation-s0013-h0006 desthost=FIREFLY destdir=postproc-LGU-firefly_coast1/shared/124/400 provider=file
2010-06-13 16:53:55,077-0500 INFO  vdl:dostagein END jobid=extract-6qrl9dtj - Staging in finished
2010-06-13 16:53:55,077-0500 DEBUG vdl:execute2 JOB_START jobid=extract-6qrl9dtj tr=extract arguments=[stat=LGU, extract_sgt=1, slon=-119.06587, slat=34.10819, rupmodfile=124/400/124_400.txt.variation-s0013-h0006, sgt_xfile=LGU/LGU_fx_664.sgt, sgt_yfile=LGU/LGU_fy_664.sgt, extract_sgt_xfile=panfs/panasas/CMS/data/engage-aespinosa/swift/LGU/124/400/LGU_124_400_subfx.sgt, extract_sgt_yfile=panfs/panasas/CMS/data/engage-aespinosa/swift/LGU/124/400/LGU_124_400_subfy.sgt] tmpdir=postproc-LGU-firefly_coast1/jobs/6/extract-6qrl9dtj host=FIREFLY
2010-06-13 16:53:55,078-0500 INFO  Execute jobid=extract-6qrl9dtj task=Task(type=JOB_SUBMISSION, identity=urn:0-12-846-6-1-1276453892896)
2010-06-13 16:53:55,079-0500 DEBUG TaskImpl Task(type=JOB_SUBMISSION, identity=urn:0-12-846-6-1-1276453892896) setting status to Submitting
2010-06-13 16:53:55,080-0500 INFO  Command Sending Command(14283, SUBMITJOB) on GSSSChannel-11535022945(1)
2010-06-13 16:53:55,186-0500 INFO  AbstractKarajanChannel GSSSChannel-11535022945(1) REPL: Command(14283, SUBMITJOB)
2010-06-13 16:53:55,186-0500 DEBUG TaskImpl Task(type=JOB_SUBMISSION, identity=urn:0-12-846-6-1-1276453892896) setting status to Submitted
2010-06-13 16:53:55,186-0500 INFO  JobSubmissionTaskHandler Submitted task Task(type=JOB_SUBMISSION, identity=urn:0-12-846-6-1-1276453892896). Job id: urn:1276453892896-1276453772841-1276453772842

(At this point there are several hundred jobs queued)

2010-06-13 16:53:55,186-0500 INFO  AbstractKarajanChannel Unregistering Command(14283, SUBMITJOB)
2010-06-13 16:53:58,179-0500 INFO  AbstractStreamKarajanChannel$Multiplexer No streams
2010-06-13 16:54:04,831-0500 INFO  AbstractKarajanChannel GSSSChannel-11535022945(1) REQ: Handler(BQPSTATUS)
2010-06-13 16:54:04,834-0500 INFO  BQPStatusHandler Process BQP status update 1
2010-06-13 16:54:04,834-0500 INFO  BQPStatusHandler Process BQP status update 2
2010-06-13 16:54:08,201-0500 INFO  AbstractStreamKarajanChannel$Multiplexer No streams
2010-06-13 16:54:14,893-0500 INFO  AbstractKarajanChannel GSSSChannel-11535022945(1) REQ: Handler(BQPSTATUS)
2010-06-13 16:54:14,894-0500 INFO  AbstractStreamKarajanChannel Sender 16493645 queue size: 0
2010-06-13 16:54:14,897-0500 INFO  BQPStatusHandler Process BQP status update 1
2010-06-13 16:54:14,897-0500 INFO  BQPStatusHandler Process BQP status update 2
2010-06-13 16:54:18,221-0500 INFO  AbstractStreamKarajanChannel$Multiplexer No streams
2010-06-13 16:54:25,019-0500 INFO  AbstractKarajanChannel GSSSChannel-11535022945(1) REQ: Handler(BQPSTATUS)
2010-06-13 16:54:25,019-0500 INFO  AbstractStreamKarajanChannel Sender 16493645 queue size: 0
2010-06-13 16:54:25,022-0500 INFO  BQPStatusHandler Process BQP status update 1
2010-06-13 16:54:25,022-0500 INFO  BQPStatusHandler Process BQP status update 2
2010-06-13 16:54:28,243-0500 INFO  AbstractStreamKarajanChannel$Multiplexer No streams
2010-06-13 16:54:35,131-0500 INFO  AbstractKarajanChannel GSSSChannel-11535022945(1) REQ: Handler(BQPSTATUS)
2010-06-13 16:54:35,131-0500 INFO  AbstractStreamKarajanChannel Sender 16493645 queue size: 0
2010-06-13 16:54:35,134-0500 INFO  BQPStatusHandler Process BQP status update 1
2010-06-13 16:54:35,134-0500 INFO  BQPStatusHandler Process BQP status update 2
2010-06-13 16:54:38,245-0500 INFO  AbstractStreamKarajanChannel$Multiplexer No streams
2010-06-13 16:54:45,208-0500 INFO  AbstractKarajanChannel GSSSChannel-11535022945(1) REQ: Handler(BQPSTATUS)
...
...
...
queue size: 0
2010-06-14 13:48:07,067-0500 INFO  BQPStatusHandler Process BQP status update 1
2010-06-14 13:48:07,067-0500 INFO  BQPStatusHandler Process BQP status update 2
2010-06-14 13:48:09,623-0500 INFO  AbstractStreamKarajanChannel$Multiplexer No streams
2010-06-14 13:48:14,968-0500 INFO  Command Sending Command(10504, SHUTDOWNSERVICE) on GSSSChannel-1651098589(3)
2010-06-14 13:48:14,968-0500 INFO  AbstractStreamKarajanChannel Sender 15595149 queue size: 0
2010-06-14 13:48:15,089-0500 INFO  AbstractKarajanChannel GSSSChannel-1651098589(3) REPL: Command(10504, SHUTDOWNSERVICE)
2010-06-14 13:48:15,089-0500 INFO  AbstractKarajanChannel Unregistering Command(10504, SHUTDOWNSERVICE)
--i shutdown the workflow via Ctrl-C---

sites.xml:
  <pool handle="FIREFLY">
    <execution provider="coaster" url="ff-grid.unl.edu" jobmanager="gt2:gt2:pbs"
      />

    <profile namespace="globus" key="maxTime">86400</profile>
    <profile namespace="globus" key="maxNodes">5300</profile>
    <profile namespace="globus" key="spread">0.8</profile>
    <profile namespace="globus" key="slots">10</profile>
    <profile namespace="globus" key="remoteMonitorEnabled">true</profile>

    <profile namespace="karajan" key="initialScore">1500.0</profile>
    <profile namespace="karajan" key="jobThrottle">53.00</profile>

    <gridftp  url="gsiftp://ff-grid2.unl.edu"/>
    <workdirectory>/panfs/panasas/CMS/data/engage-aespinosa/swift</workdirectory>
  </pool>


-- 
Allan M. Espinosa <http://amespinosa.wordpress.com>
PhD student, Computer Science
University of Chicago <http://people.cs.uchicago.edu/~aespinosa>

-------------- next part --------------
A non-text attachment was scrubbed...
Name: postproc-LGU-firefly_coast1.png
Type: image/png
Size: 7438 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/swift-user/attachments/20100614/04074d50/attachment.png>

From aespinosa at cs.uchicago.edu  Mon Jun 14 13:59:26 2010
From: aespinosa at cs.uchicago.edu (Allan Espinosa)
Date: Mon, 14 Jun 2010 13:59:26 -0500
Subject: [Swift-user] Re: workflow 'pauses' after a few thousand jobs
In-Reply-To: <AANLkTimGxBtoMonzMIBEOjk10u_Q34vlEbyDouslzIdT@mail.gmail.com>
References: <AANLkTimGxBtoMonzMIBEOjk10u_Q34vlEbyDouslzIdT@mail.gmail.com>
Message-ID: <AANLkTil6x7yHyKqXZZmR8LR46ZtERaWTLTJOLkk95Kru@mail.gmail.com>

The logs were too large.  Currently it is located in
http://www.ci.uchicago.edu/~aespinosa/swift/debug/postproc-prod_pads2.log.bz2

-Allan

2010/6/14 Allan Espinosa <aespinosa at cs.uchicago.edu>:
> I attached the coaster monitoring screenshot and the logfile. ?Below is the snippet of the logfile when the workflow "hangs":
>
> 2010-06-13 16:53:55,077-0500 DEBUG vdl:dostagein CDM: file://localhost/LGU/LGU_fy_664.sgt : DIRECT
> 2010-06-13 16:53:55,077-0500 DEBUG vdl:dostageinfile FILE_STAGE_IN_START file=124_400.txt.variation-s0013-h0006 srchost=localhost srcdir=124/400 srcname=124_400.txt.variation-s0013-h0006 desthost=FIREFLY destdir=postproc-LGU-firefly_coast1/shared/124/400 provider=file policy=DIRECT
> 2010-06-13 16:53:55,077-0500 DEBUG vdl:dostageinfile FILE_STAGE_IN_SKIP file=124_400.txt.variation-s0013-h0006 policy=DIRECT
> 2010-06-13 16:53:55,077-0500 DEBUG vdl:dostageinfile FILE_STAGE_IN_END file=124_400.txt.variation-s0013-h0006 srchost=localhost srcdir=124/400 srcname=124_400.txt.variation-s0013-h0006 desthost=FIREFLY destdir=postproc-LGU-firefly_coast1/shared/124/400 provider=file
> 2010-06-13 16:53:55,077-0500 INFO ?vdl:dostagein END jobid=extract-6qrl9dtj - Staging in finished
> 2010-06-13 16:53:55,077-0500 DEBUG vdl:execute2 JOB_START jobid=extract-6qrl9dtj tr=extract arguments=[stat=LGU, extract_sgt=1, slon=-119.06587, slat=34.10819, rupmodfile=124/400/124_400.txt.variation-s0013-h0006, sgt_xfile=LGU/LGU_fx_664.sgt, sgt_yfile=LGU/LGU_fy_664.sgt, extract_sgt_xfile=panfs/panasas/CMS/data/engage-aespinosa/swift/LGU/124/400/LGU_124_400_subfx.sgt, extract_sgt_yfile=panfs/panasas/CMS/data/engage-aespinosa/swift/LGU/124/400/LGU_124_400_subfy.sgt] tmpdir=postproc-LGU-firefly_coast1/jobs/6/extract-6qrl9dtj host=FIREFLY
> 2010-06-13 16:53:55,078-0500 INFO ?Execute jobid=extract-6qrl9dtj task=Task(type=JOB_SUBMISSION, identity=urn:0-12-846-6-1-1276453892896)
> 2010-06-13 16:53:55,079-0500 DEBUG TaskImpl Task(type=JOB_SUBMISSION, identity=urn:0-12-846-6-1-1276453892896) setting status to Submitting
> 2010-06-13 16:53:55,080-0500 INFO ?Command Sending Command(14283, SUBMITJOB) on GSSSChannel-11535022945(1)
> 2010-06-13 16:53:55,186-0500 INFO ?AbstractKarajanChannel GSSSChannel-11535022945(1) REPL: Command(14283, SUBMITJOB)
> 2010-06-13 16:53:55,186-0500 DEBUG TaskImpl Task(type=JOB_SUBMISSION, identity=urn:0-12-846-6-1-1276453892896) setting status to Submitted
> 2010-06-13 16:53:55,186-0500 INFO ?JobSubmissionTaskHandler Submitted task Task(type=JOB_SUBMISSION, identity=urn:0-12-846-6-1-1276453892896). Job id: urn:1276453892896-1276453772841-1276453772842
>
> (At this point there are several hundred jobs queued)
>
> 2010-06-13 16:53:55,186-0500 INFO ?AbstractKarajanChannel Unregistering Command(14283, SUBMITJOB)
> 2010-06-13 16:53:58,179-0500 INFO ?AbstractStreamKarajanChannel$Multiplexer No streams
> 2010-06-13 16:54:04,831-0500 INFO ?AbstractKarajanChannel GSSSChannel-11535022945(1) REQ: Handler(BQPSTATUS)
> 2010-06-13 16:54:04,834-0500 INFO ?BQPStatusHandler Process BQP status update 1
> 2010-06-13 16:54:04,834-0500 INFO ?BQPStatusHandler Process BQP status update 2
> 2010-06-13 16:54:08,201-0500 INFO ?AbstractStreamKarajanChannel$Multiplexer No streams
> 2010-06-13 16:54:14,893-0500 INFO ?AbstractKarajanChannel GSSSChannel-11535022945(1) REQ: Handler(BQPSTATUS)
> 2010-06-13 16:54:14,894-0500 INFO ?AbstractStreamKarajanChannel Sender 16493645 queue size: 0
> 2010-06-13 16:54:14,897-0500 INFO ?BQPStatusHandler Process BQP status update 1
> 2010-06-13 16:54:14,897-0500 INFO ?BQPStatusHandler Process BQP status update 2
> 2010-06-13 16:54:18,221-0500 INFO ?AbstractStreamKarajanChannel$Multiplexer No streams
> 2010-06-13 16:54:25,019-0500 INFO ?AbstractKarajanChannel GSSSChannel-11535022945(1) REQ: Handler(BQPSTATUS)
> 2010-06-13 16:54:25,019-0500 INFO ?AbstractStreamKarajanChannel Sender 16493645 queue size: 0
> 2010-06-13 16:54:25,022-0500 INFO ?BQPStatusHandler Process BQP status update 1
> 2010-06-13 16:54:25,022-0500 INFO ?BQPStatusHandler Process BQP status update 2
> 2010-06-13 16:54:28,243-0500 INFO ?AbstractStreamKarajanChannel$Multiplexer No streams
> 2010-06-13 16:54:35,131-0500 INFO ?AbstractKarajanChannel GSSSChannel-11535022945(1) REQ: Handler(BQPSTATUS)
> 2010-06-13 16:54:35,131-0500 INFO ?AbstractStreamKarajanChannel Sender 16493645 queue size: 0
> 2010-06-13 16:54:35,134-0500 INFO ?BQPStatusHandler Process BQP status update 1
> 2010-06-13 16:54:35,134-0500 INFO ?BQPStatusHandler Process BQP status update 2
> 2010-06-13 16:54:38,245-0500 INFO ?AbstractStreamKarajanChannel$Multiplexer No streams
> 2010-06-13 16:54:45,208-0500 INFO ?AbstractKarajanChannel GSSSChannel-11535022945(1) REQ: Handler(BQPSTATUS)
> ...
> ...
> ...
> queue size: 0
> 2010-06-14 13:48:07,067-0500 INFO ?BQPStatusHandler Process BQP status update 1
> 2010-06-14 13:48:07,067-0500 INFO ?BQPStatusHandler Process BQP status update 2
> 2010-06-14 13:48:09,623-0500 INFO ?AbstractStreamKarajanChannel$Multiplexer No streams
> 2010-06-14 13:48:14,968-0500 INFO ?Command Sending Command(10504, SHUTDOWNSERVICE) on GSSSChannel-1651098589(3)
> 2010-06-14 13:48:14,968-0500 INFO ?AbstractStreamKarajanChannel Sender 15595149 queue size: 0
> 2010-06-14 13:48:15,089-0500 INFO ?AbstractKarajanChannel GSSSChannel-1651098589(3) REPL: Command(10504, SHUTDOWNSERVICE)
> 2010-06-14 13:48:15,089-0500 INFO ?AbstractKarajanChannel Unregistering Command(10504, SHUTDOWNSERVICE)
> --i shutdown the workflow via Ctrl-C---
>
> sites.xml:
> ?<pool handle="FIREFLY">
> ? ?<execution provider="coaster" url="ff-grid.unl.edu" jobmanager="gt2:gt2:pbs"
> ? ? ?/>
>
> ? ?<profile namespace="globus" key="maxTime">86400</profile>
> ? ?<profile namespace="globus" key="maxNodes">5300</profile>
> ? ?<profile namespace="globus" key="spread">0.8</profile>
> ? ?<profile namespace="globus" key="slots">10</profile>
> ? ?<profile namespace="globus" key="remoteMonitorEnabled">true</profile>
>
> ? ?<profile namespace="karajan" key="initialScore">1500.0</profile>
> ? ?<profile namespace="karajan" key="jobThrottle">53.00</profile>
>
> ? ?<gridftp ?url="gsiftp://ff-grid2.unl.edu"/>
> ? ?<workdirectory>/panfs/panasas/CMS/data/engage-aespinosa/swift</workdirectory>
> ?</pool>
>


From jon.monette at gmail.com  Mon Jun 21 17:19:59 2010
From: jon.monette at gmail.com (Jonathan Monette)
Date: Mon, 21 Jun 2010 17:19:59 -0500
Subject: [Swift-user] Compile Error
Message-ID: <4C1FE58F.7020100@gmail.com>

Hello,
     I am getting a compile error and not sure how to fix it.

         Could not start execution.
             Compile error in foreach statement at line 81: Type 
DiffStruct is not defined.

     I understand the error but I have defined DiffStruct at the top of 
the file.
         type DiffStruct
         {
             int cntl1;
             int cntl2;
             Image plus;
             Image minus;
             Image diff;
         }

     I have also declared Image as a type.  Using Image gives me no 
problems.  The line that the error comes on is:

         foreach diff_entry, i in diffs

     where diffs is an array of DiffStruct that was created using the 
csv_mapper.  Does anyone know what the problem is?  Thanks in advance.

-- 
Jon

Computers are incredibly fast, accurate, and stupid. Human beings are incredibly slow, inaccurate, and brilliant. Together they are powerful beyond imagination.
- Albert Einstein



From wilde at mcs.anl.gov  Tue Jun 22 15:00:45 2010
From: wilde at mcs.anl.gov (Michael Wilde)
Date: Tue, 22 Jun 2010 15:00:45 -0500 (CDT)
Subject: [Swift-user] Compile Error
In-Reply-To: <4C1FE58F.7020100@gmail.com>
Message-ID: <21226373.991651277236845748.JavaMail.root@zimbra>

Jon, can you try to re-create this problem in a simple Swift script?

If so, Mihael, can you take a look and see if its a Swift error, and/or how to work around it?

Thanks,

Mike

----- "Jonathan Monette" <jon.monette at gmail.com> wrote:

> Hello,
>      I am getting a compile error and not sure how to fix it.
> 
>          Could not start execution.
>              Compile error in foreach statement at line 81: Type 
> DiffStruct is not defined.
> 
>      I understand the error but I have defined DiffStruct at the top
> of 
> the file.
>          type DiffStruct
>          {
>              int cntl1;
>              int cntl2;
>              Image plus;
>              Image minus;
>              Image diff;
>          }
> 
>      I have also declared Image as a type.  Using Image gives me no 
> problems.  The line that the error comes on is:
> 
>          foreach diff_entry, i in diffs
> 
>      where diffs is an array of DiffStruct that was created using the
> 
> csv_mapper.  Does anyone know what the problem is?  Thanks in
> advance.
> 
> -- 
> Jon
> 
> Computers are incredibly fast, accurate, and stupid. Human beings are
> incredibly slow, inaccurate, and brilliant. Together they are powerful
> beyond imagination.
> - Albert Einstein
> 
> _______________________________________________
> Swift-user mailing list
> Swift-user at ci.uchicago.edu
> http://mail.ci.uchicago.edu/mailman/listinfo/swift-user

-- 
Michael Wilde
Computation Institute, University of Chicago
Mathematics and Computer Science Division
Argonne National Laboratory



From hategan at mcs.anl.gov  Tue Jun 22 15:04:12 2010
From: hategan at mcs.anl.gov (Mihael Hategan)
Date: Tue, 22 Jun 2010 15:04:12 -0500
Subject: [Swift-user] Compile Error
In-Reply-To: <21226373.991651277236845748.JavaMail.root@zimbra>
References: <21226373.991651277236845748.JavaMail.root@zimbra>
Message-ID: <1277237052.13480.1.camel@blabla2.none>

I thought I replied to it, but is seems not.

Yes, please post the exact script.

Mihael

On Tue, 2010-06-22 at 15:00 -0500, Michael Wilde wrote:
> Jon, can you try to re-create this problem in a simple Swift script?
> 
> If so, Mihael, can you take a look and see if its a Swift error, and/or how to work around it?
> 
> Thanks,
> 
> Mike
> 
> ----- "Jonathan Monette" <jon.monette at gmail.com> wrote:
> 
> > Hello,
> >      I am getting a compile error and not sure how to fix it.
> > 
> >          Could not start execution.
> >              Compile error in foreach statement at line 81: Type 
> > DiffStruct is not defined.
> > 
> >      I understand the error but I have defined DiffStruct at the top
> > of 
> > the file.
> >          type DiffStruct
> >          {
> >              int cntl1;
> >              int cntl2;
> >              Image plus;
> >              Image minus;
> >              Image diff;
> >          }
> > 
> >      I have also declared Image as a type.  Using Image gives me no 
> > problems.  The line that the error comes on is:
> > 
> >          foreach diff_entry, i in diffs
> > 
> >      where diffs is an array of DiffStruct that was created using the
> > 
> > csv_mapper.  Does anyone know what the problem is?  Thanks in
> > advance.
> > 
> > -- 
> > Jon
> > 
> > Computers are incredibly fast, accurate, and stupid. Human beings are
> > incredibly slow, inaccurate, and brilliant. Together they are powerful
> > beyond imagination.
> > - Albert Einstein
> > 
> > _______________________________________________
> > Swift-user mailing list
> > Swift-user at ci.uchicago.edu
> > http://mail.ci.uchicago.edu/mailman/listinfo/swift-user
> 




From hategan at mcs.anl.gov  Tue Jun 22 15:13:12 2010
From: hategan at mcs.anl.gov (Mihael Hategan)
Date: Tue, 22 Jun 2010 15:13:12 -0500
Subject: [Swift-user] Compile Error
In-Reply-To: <1277237052.13480.1.camel@blabla2.none>
References: <21226373.991651277236845748.JavaMail.root@zimbra>
	<1277237052.13480.1.camel@blabla2.none>
Message-ID: <1277237592.13480.7.camel@blabla2.none>

I may have sent this before, but I think it's useful reading. Don't take
it as meant to be patronizing. It's meant to clarify the perspective of
the other end of a bug report.

http://www.chiark.greenend.org.uk/~sgtatham/bugs.html


On Tue, 2010-06-22 at 15:04 -0500, Mihael Hategan wrote:
> I thought I replied to it, but is seems not.
> 
> Yes, please post the exact script.
> 
> Mihael
> 
> On Tue, 2010-06-22 at 15:00 -0500, Michael Wilde wrote:
> > Jon, can you try to re-create this problem in a simple Swift script?
> > 
> > If so, Mihael, can you take a look and see if its a Swift error, and/or how to work around it?
> > 
> > Thanks,
> > 
> > Mike
> > 
> > ----- "Jonathan Monette" <jon.monette at gmail.com> wrote:
> > 
> > > Hello,
> > >      I am getting a compile error and not sure how to fix it.
> > > 
> > >          Could not start execution.
> > >              Compile error in foreach statement at line 81: Type 
> > > DiffStruct is not defined.
> > > 
> > >      I understand the error but I have defined DiffStruct at the top
> > > of 
> > > the file.
> > >          type DiffStruct
> > >          {
> > >              int cntl1;
> > >              int cntl2;
> > >              Image plus;
> > >              Image minus;
> > >              Image diff;
> > >          }
> > > 
> > >      I have also declared Image as a type.  Using Image gives me no 
> > > problems.  The line that the error comes on is:
> > > 
> > >          foreach diff_entry, i in diffs
> > > 
> > >      where diffs is an array of DiffStruct that was created using the
> > > 
> > > csv_mapper.  Does anyone know what the problem is?  Thanks in
> > > advance.
> > > 
> > > -- 
> > > Jon
> > > 
> > > Computers are incredibly fast, accurate, and stupid. Human beings are
> > > incredibly slow, inaccurate, and brilliant. Together they are powerful
> > > beyond imagination.
> > > - Albert Einstein
> > > 
> > > _______________________________________________
> > > Swift-user mailing list
> > > Swift-user at ci.uchicago.edu
> > > http://mail.ci.uchicago.edu/mailman/listinfo/swift-user
> > 
> 
> 
> _______________________________________________
> Swift-user mailing list
> Swift-user at ci.uchicago.edu
> http://mail.ci.uchicago.edu/mailman/listinfo/swift-user




From jon.monette at gmail.com  Tue Jun 22 15:22:04 2010
From: jon.monette at gmail.com (Jonathan Monette)
Date: Tue, 22 Jun 2010 15:22:04 -0500
Subject: [Swift-user] Compile Error
In-Reply-To: <1277237592.13480.7.camel@blabla2.none>
References: <21226373.991651277236845748.JavaMail.root@zimbra>	<1277237052.13480.1.camel@blabla2.none>
	<1277237592.13480.7.camel@blabla2.none>
Message-ID: <4C211B6C.7050504@gmail.com>

Here is a script that recreates the problem and the two files needed to 
run.  The line commented out is the Montage call but the error comes 
before the call.

On 6/22/10 3:13 PM, Mihael Hategan wrote:
> I may have sent this before, but I think it's useful reading. Don't take
> it as meant to be patronizing. It's meant to clarify the perspective of
> the other end of a bug report.
>
> http://www.chiark.greenend.org.uk/~sgtatham/bugs.html
>
>
> On Tue, 2010-06-22 at 15:04 -0500, Mihael Hategan wrote:
>    
>> I thought I replied to it, but is seems not.
>>
>> Yes, please post the exact script.
>>
>> Mihael
>>
>> On Tue, 2010-06-22 at 15:00 -0500, Michael Wilde wrote:
>>      
>>> Jon, can you try to re-create this problem in a simple Swift script?
>>>
>>> If so, Mihael, can you take a look and see if its a Swift error, and/or how to work around it?
>>>
>>> Thanks,
>>>
>>> Mike
>>>
>>> ----- "Jonathan Monette"<jon.monette at gmail.com>  wrote:
>>>
>>>        
>>>> Hello,
>>>>       I am getting a compile error and not sure how to fix it.
>>>>
>>>>           Could not start execution.
>>>>               Compile error in foreach statement at line 81: Type
>>>> DiffStruct is not defined.
>>>>
>>>>       I understand the error but I have defined DiffStruct at the top
>>>> of
>>>> the file.
>>>>           type DiffStruct
>>>>           {
>>>>               int cntl1;
>>>>               int cntl2;
>>>>               Image plus;
>>>>               Image minus;
>>>>               Image diff;
>>>>           }
>>>>
>>>>       I have also declared Image as a type.  Using Image gives me no
>>>> problems.  The line that the error comes on is:
>>>>
>>>>           foreach diff_entry, i in diffs
>>>>
>>>>       where diffs is an array of DiffStruct that was created using the
>>>>
>>>> csv_mapper.  Does anyone know what the problem is?  Thanks in
>>>> advance.
>>>>
>>>> -- 
>>>> Jon
>>>>
>>>> Computers are incredibly fast, accurate, and stupid. Human beings are
>>>> incredibly slow, inaccurate, and brilliant. Together they are powerful
>>>> beyond imagination.
>>>> - Albert Einstein
>>>>
>>>> _______________________________________________
>>>> Swift-user mailing list
>>>> Swift-user at ci.uchicago.edu
>>>> http://mail.ci.uchicago.edu/mailman/listinfo/swift-user
>>>>          
>>>        
>>
>> _______________________________________________
>> Swift-user mailing list
>> Swift-user at ci.uchicago.edu
>> http://mail.ci.uchicago.edu/mailman/listinfo/swift-user
>>      
>
> _______________________________________________
> Swift-user mailing list
> Swift-user at ci.uchicago.edu
> http://mail.ci.uchicago.edu/mailman/listinfo/swift-user
>
>    

-- 
Jon

Computers are incredibly fast, accurate, and stupid. Human beings are incredibly slow, inaccurate, and brilliant. Together they are powerful beyond imagination.
- Albert Einstein

-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: problem_script.swift
URL: <http://lists.mcs.anl.gov/pipermail/swift-user/attachments/20100622/a224ffa2/attachment.ksh>
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: diffs.tbl
URL: <http://lists.mcs.anl.gov/pipermail/swift-user/attachments/20100622/a224ffa2/attachment-0001.ksh>
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: template.hdr
URL: <http://lists.mcs.anl.gov/pipermail/swift-user/attachments/20100622/a224ffa2/attachment-0002.ksh>

From jon.monette at gmail.com  Tue Jun 22 15:32:21 2010
From: jon.monette at gmail.com (Jonathan Monette)
Date: Tue, 22 Jun 2010 15:32:21 -0500
Subject: [Swift-user] Compile Error
In-Reply-To: <1277237592.13480.7.camel@blabla2.none>
References: <21226373.991651277236845748.JavaMail.root@zimbra>	<1277237052.13480.1.camel@blabla2.none>
	<1277237592.13480.7.camel@blabla2.none>
Message-ID: <4C211DD5.80300@gmail.com>

Also, the script was run on my macbook as localhost.  Was not run using 
coasters or anything of that sort.

On 6/22/10 3:13 PM, Mihael Hategan wrote:
> I may have sent this before, but I think it's useful reading. Don't take
> it as meant to be patronizing. It's meant to clarify the perspective of
> the other end of a bug report.
>
> http://www.chiark.greenend.org.uk/~sgtatham/bugs.html
>
>
> On Tue, 2010-06-22 at 15:04 -0500, Mihael Hategan wrote:
>    
>> I thought I replied to it, but is seems not.
>>
>> Yes, please post the exact script.
>>
>> Mihael
>>
>> On Tue, 2010-06-22 at 15:00 -0500, Michael Wilde wrote:
>>      
>>> Jon, can you try to re-create this problem in a simple Swift script?
>>>
>>> If so, Mihael, can you take a look and see if its a Swift error, and/or how to work around it?
>>>
>>> Thanks,
>>>
>>> Mike
>>>
>>> ----- "Jonathan Monette"<jon.monette at gmail.com>  wrote:
>>>
>>>        
>>>> Hello,
>>>>       I am getting a compile error and not sure how to fix it.
>>>>
>>>>           Could not start execution.
>>>>               Compile error in foreach statement at line 81: Type
>>>> DiffStruct is not defined.
>>>>
>>>>       I understand the error but I have defined DiffStruct at the top
>>>> of
>>>> the file.
>>>>           type DiffStruct
>>>>           {
>>>>               int cntl1;
>>>>               int cntl2;
>>>>               Image plus;
>>>>               Image minus;
>>>>               Image diff;
>>>>           }
>>>>
>>>>       I have also declared Image as a type.  Using Image gives me no
>>>> problems.  The line that the error comes on is:
>>>>
>>>>           foreach diff_entry, i in diffs
>>>>
>>>>       where diffs is an array of DiffStruct that was created using the
>>>>
>>>> csv_mapper.  Does anyone know what the problem is?  Thanks in
>>>> advance.
>>>>
>>>> -- 
>>>> Jon
>>>>
>>>> Computers are incredibly fast, accurate, and stupid. Human beings are
>>>> incredibly slow, inaccurate, and brilliant. Together they are powerful
>>>> beyond imagination.
>>>> - Albert Einstein
>>>>
>>>> _______________________________________________
>>>> Swift-user mailing list
>>>> Swift-user at ci.uchicago.edu
>>>> http://mail.ci.uchicago.edu/mailman/listinfo/swift-user
>>>>          
>>>        
>>
>> _______________________________________________
>> Swift-user mailing list
>> Swift-user at ci.uchicago.edu
>> http://mail.ci.uchicago.edu/mailman/listinfo/swift-user
>>      
>
> _______________________________________________
> Swift-user mailing list
> Swift-user at ci.uchicago.edu
> http://mail.ci.uchicago.edu/mailman/listinfo/swift-user
>
>    

-- 
Jon

Computers are incredibly fast, accurate, and stupid. Human beings are incredibly slow, inaccurate, and brilliant. Together they are powerful beyond imagination.
- Albert Einstein



From hategan at mcs.anl.gov  Tue Jun 22 15:35:17 2010
From: hategan at mcs.anl.gov (Mihael Hategan)
Date: Tue, 22 Jun 2010 15:35:17 -0500
Subject: [Swift-user] Compile Error
In-Reply-To: <4C211DD5.80300@gmail.com>
References: <21226373.991651277236845748.JavaMail.root@zimbra>
	<1277237052.13480.1.camel@blabla2.none>
	<1277237592.13480.7.camel@blabla2.none>  <4C211DD5.80300@gmail.com>
Message-ID: <1277238917.13480.22.camel@blabla2.none>

If it failed at compile time then it doesn't matter.

On Tue, 2010-06-22 at 15:32 -0500, Jonathan Monette wrote:
> Also, the script was run on my macbook as localhost.  Was not run using 
> coasters or anything of that sort.
> 
> On 6/22/10 3:13 PM, Mihael Hategan wrote:
> > I may have sent this before, but I think it's useful reading. Don't take
> > it as meant to be patronizing. It's meant to clarify the perspective of
> > the other end of a bug report.
> >
> > http://www.chiark.greenend.org.uk/~sgtatham/bugs.html
> >
> >
> > On Tue, 2010-06-22 at 15:04 -0500, Mihael Hategan wrote:
> >    
> >> I thought I replied to it, but is seems not.
> >>
> >> Yes, please post the exact script.
> >>
> >> Mihael
> >>
> >> On Tue, 2010-06-22 at 15:00 -0500, Michael Wilde wrote:
> >>      
> >>> Jon, can you try to re-create this problem in a simple Swift script?
> >>>
> >>> If so, Mihael, can you take a look and see if its a Swift error, and/or how to work around it?
> >>>
> >>> Thanks,
> >>>
> >>> Mike
> >>>
> >>> ----- "Jonathan Monette"<jon.monette at gmail.com>  wrote:
> >>>
> >>>        
> >>>> Hello,
> >>>>       I am getting a compile error and not sure how to fix it.
> >>>>
> >>>>           Could not start execution.
> >>>>               Compile error in foreach statement at line 81: Type
> >>>> DiffStruct is not defined.
> >>>>
> >>>>       I understand the error but I have defined DiffStruct at the top
> >>>> of
> >>>> the file.
> >>>>           type DiffStruct
> >>>>           {
> >>>>               int cntl1;
> >>>>               int cntl2;
> >>>>               Image plus;
> >>>>               Image minus;
> >>>>               Image diff;
> >>>>           }
> >>>>
> >>>>       I have also declared Image as a type.  Using Image gives me no
> >>>> problems.  The line that the error comes on is:
> >>>>
> >>>>           foreach diff_entry, i in diffs
> >>>>
> >>>>       where diffs is an array of DiffStruct that was created using the
> >>>>
> >>>> csv_mapper.  Does anyone know what the problem is?  Thanks in
> >>>> advance.
> >>>>
> >>>> -- 
> >>>> Jon
> >>>>
> >>>> Computers are incredibly fast, accurate, and stupid. Human beings are
> >>>> incredibly slow, inaccurate, and brilliant. Together they are powerful
> >>>> beyond imagination.
> >>>> - Albert Einstein
> >>>>
> >>>> _______________________________________________
> >>>> Swift-user mailing list
> >>>> Swift-user at ci.uchicago.edu
> >>>> http://mail.ci.uchicago.edu/mailman/listinfo/swift-user
> >>>>          
> >>>        
> >>
> >> _______________________________________________
> >> Swift-user mailing list
> >> Swift-user at ci.uchicago.edu
> >> http://mail.ci.uchicago.edu/mailman/listinfo/swift-user
> >>      
> >
> > _______________________________________________
> > Swift-user mailing list
> > Swift-user at ci.uchicago.edu
> > http://mail.ci.uchicago.edu/mailman/listinfo/swift-user
> >
> >    
> 




From jon.monette at gmail.com  Tue Jun 22 15:37:51 2010
From: jon.monette at gmail.com (Jonathan Monette)
Date: Tue, 22 Jun 2010 15:37:51 -0500
Subject: [Swift-user] Compile Error
In-Reply-To: <1277238917.13480.22.camel@blabla2.none>
References: <21226373.991651277236845748.JavaMail.root@zimbra>	
	<1277237052.13480.1.camel@blabla2.none>	
	<1277237592.13480.7.camel@blabla2.none> <4C211DD5.80300@gmail.com>
	<1277238917.13480.22.camel@blabla2.none>
Message-ID: <4C211F1F.5080205@gmail.com>

Alright, was not sure.  Yea.  That is when it failed.

On 6/22/10 3:35 PM, Mihael Hategan wrote:
> If it failed at compile time then it doesn't matter.
>
> On Tue, 2010-06-22 at 15:32 -0500, Jonathan Monette wrote:
>    
>> Also, the script was run on my macbook as localhost.  Was not run using
>> coasters or anything of that sort.
>>
>> On 6/22/10 3:13 PM, Mihael Hategan wrote:
>>      
>>> I may have sent this before, but I think it's useful reading. Don't take
>>> it as meant to be patronizing. It's meant to clarify the perspective of
>>> the other end of a bug report.
>>>
>>> http://www.chiark.greenend.org.uk/~sgtatham/bugs.html
>>>
>>>
>>> On Tue, 2010-06-22 at 15:04 -0500, Mihael Hategan wrote:
>>>
>>>        
>>>> I thought I replied to it, but is seems not.
>>>>
>>>> Yes, please post the exact script.
>>>>
>>>> Mihael
>>>>
>>>> On Tue, 2010-06-22 at 15:00 -0500, Michael Wilde wrote:
>>>>
>>>>          
>>>>> Jon, can you try to re-create this problem in a simple Swift script?
>>>>>
>>>>> If so, Mihael, can you take a look and see if its a Swift error, and/or how to work around it?
>>>>>
>>>>> Thanks,
>>>>>
>>>>> Mike
>>>>>
>>>>> ----- "Jonathan Monette"<jon.monette at gmail.com>   wrote:
>>>>>
>>>>>
>>>>>            
>>>>>> Hello,
>>>>>>        I am getting a compile error and not sure how to fix it.
>>>>>>
>>>>>>            Could not start execution.
>>>>>>                Compile error in foreach statement at line 81: Type
>>>>>> DiffStruct is not defined.
>>>>>>
>>>>>>        I understand the error but I have defined DiffStruct at the top
>>>>>> of
>>>>>> the file.
>>>>>>            type DiffStruct
>>>>>>            {
>>>>>>                int cntl1;
>>>>>>                int cntl2;
>>>>>>                Image plus;
>>>>>>                Image minus;
>>>>>>                Image diff;
>>>>>>            }
>>>>>>
>>>>>>        I have also declared Image as a type.  Using Image gives me no
>>>>>> problems.  The line that the error comes on is:
>>>>>>
>>>>>>            foreach diff_entry, i in diffs
>>>>>>
>>>>>>        where diffs is an array of DiffStruct that was created using the
>>>>>>
>>>>>> csv_mapper.  Does anyone know what the problem is?  Thanks in
>>>>>> advance.
>>>>>>
>>>>>> -- 
>>>>>> Jon
>>>>>>
>>>>>> Computers are incredibly fast, accurate, and stupid. Human beings are
>>>>>> incredibly slow, inaccurate, and brilliant. Together they are powerful
>>>>>> beyond imagination.
>>>>>> - Albert Einstein
>>>>>>
>>>>>> _______________________________________________
>>>>>> Swift-user mailing list
>>>>>> Swift-user at ci.uchicago.edu
>>>>>> http://mail.ci.uchicago.edu/mailman/listinfo/swift-user
>>>>>>
>>>>>>              
>>>>>
>>>>>            
>>>> _______________________________________________
>>>> Swift-user mailing list
>>>> Swift-user at ci.uchicago.edu
>>>> http://mail.ci.uchicago.edu/mailman/listinfo/swift-user
>>>>
>>>>          
>>> _______________________________________________
>>> Swift-user mailing list
>>> Swift-user at ci.uchicago.edu
>>> http://mail.ci.uchicago.edu/mailman/listinfo/swift-user
>>>
>>>
>>>        
>>      
>
>
>    

-- 
Jon

Computers are incredibly fast, accurate, and stupid. Human beings are incredibly slow, inaccurate, and brilliant. Together they are powerful beyond imagination.
- Albert Einstein



From hategan at mcs.anl.gov  Tue Jun 22 15:46:03 2010
From: hategan at mcs.anl.gov (Mihael Hategan)
Date: Tue, 22 Jun 2010 15:46:03 -0500
Subject: [Swift-user] Compile Error
In-Reply-To: <4C211F1F.5080205@gmail.com>
References: <21226373.991651277236845748.JavaMail.root@zimbra>
	<1277237052.13480.1.camel@blabla2.none>
	<1277237592.13480.7.camel@blabla2.none>  <4C211DD5.80300@gmail.com>
	<1277238917.13480.22.camel@blabla2.none> <4C211F1F.5080205@gmail.com>
Message-ID: <1277239563.13480.24.camel@blabla2.none>

On Tue, 2010-06-22 at 15:37 -0500, Jonathan Monette wrote:
> Alright, was not sure.  Yea.  That is when it failed.

It's a horrible error message for what is happening, but it's
"d_entry.plus" not "d_entry.plu".




From jon.monette at gmail.com  Tue Jun 22 15:51:14 2010
From: jon.monette at gmail.com (Jonathan Monette)
Date: Tue, 22 Jun 2010 15:51:14 -0500
Subject: [Swift-user] Compile Error
In-Reply-To: <1277239563.13480.24.camel@blabla2.none>
References: <21226373.991651277236845748.JavaMail.root@zimbra>	
	<1277237052.13480.1.camel@blabla2.none>	
	<1277237592.13480.7.camel@blabla2.none> <4C211DD5.80300@gmail.com>	
	<1277238917.13480.22.camel@blabla2.none>
	<4C211F1F.5080205@gmail.com>
	<1277239563.13480.24.camel@blabla2.none>
Message-ID: <4C212242.3090300@gmail.com>

OMG.  Thank you.  Yea that is a bad error message.  I did not even think 
about reading through those lines based on the message.  Thanks.

On 6/22/10 3:46 PM, Mihael Hategan wrote:
> On Tue, 2010-06-22 at 15:37 -0500, Jonathan Monette wrote:
>    
>> Alright, was not sure.  Yea.  That is when it failed.
>>      
> It's a horrible error message for what is happening, but it's
> "d_entry.plus" not "d_entry.plu".
>
>
>
>    

-- 
Jon

Computers are incredibly fast, accurate, and stupid. Human beings are incredibly slow, inaccurate, and brilliant. Together they are powerful beyond imagination.
- Albert Einstein



From mandaya at rose-hulman.edu  Wed Jun 23 11:26:58 2010
From: mandaya at rose-hulman.edu (Arjun Comar)
Date: Wed, 23 Jun 2010 11:26:58 -0500
Subject: [Swift-user] Error with condor provider
Message-ID: <AANLkTik2QVRc6BLxBs8UNeO9zFg8KcH8rp28SdvLmtue@mail.gmail.com>

Hey all,
I've been trying to get jobs submitted over Condor via swift, and running
into a few problems. I think I've finally hit a point where it's the Condor
provider itself that's failing over any of my configurations.
Here's the sites entry (though any sites entry over Condor will do, and I've
tried several):

<config>
  <pool handle="Nebraska_red.unl.edu">
    <gridftp url="gsiftp://red.unl.edu/"/>
    <execution provider="condor"/>
    <profile key="jobType" namespace="globus">grid</profile>
    <profile key="gridResource" namespace="globus">gt2
red.unl.edu/jobmanager-condor</profile>
    <workdirectory>/opt/osg/data/engage/tmp/red.unl.edu</workdirectory>
  </pool>
</config>

And any swift script at all fails, even a simple helloworld:

type messagefile;
app (messagefile t) greeting () {
        echo "Hello, world!" stdout=@filename(t);
}
messagefile outfile <"hello.txt">;
outfile = greeting();

With the following error:
The following errors have occurred:
1. Application "echo" failed (Cannot submit job: Could not submit job
(condor_submit reported an exit code of 1). Submitting job(s)
Found illegal unescaped double-quote: "" -e /bin/echo -out hello.txt -err
stderr.txt -i -d "" -if "" -of hello.txt -k "" -status "provider " -a
"Hello, world!"The full arguments you specified were:
/opt/osg/data/engage/tmp/
red.unl.edu/helloworld-20100623-1051-dr5v5apa/shared/_swiftwrapecho-dqt6jttj
-jobdir d -scratch "" -e /bin/echo -out hello.txt -err
stderr.txt -i -d "" -if "" -of hello.txt -k "" -status "provider " -a
"Hello, world!")

The same script runs just fine through any other submission mechanism, even
to the same site (ssh, coasters+ssh:pbs, etc).

Anyone have any thoughts on fixing the problem?

Thanks!

-- 
Arjun Comar, Rose-Hulman '12
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-user/attachments/20100623/e5553dd1/attachment.html>

From wilde at mcs.anl.gov  Wed Jun 23 12:03:12 2010
From: wilde at mcs.anl.gov (wilde at mcs.anl.gov)
Date: Wed, 23 Jun 2010 12:03:12 -0500 (CDT)
Subject: [Swift-user] Re: Error with condor provider
In-Reply-To: <16034500.1029471277312291378.JavaMail.root@zimbra>
Message-ID: <21022873.1029631277312592907.JavaMail.root@zimbra>

Arjun, I briefly provide here some tips for getting past this; may need some help from Mihael on it.

This is where (as I mentioned) we need to get the provider to leave the condor submit file, so we can see what its generating.

We also need to know where you ran this and what condor was used.

One place to start is to make your own condor submit file, try to re-create the error below, and then adjust the provider to remedy it.

Its *possible* that condor quoting conventions changed from the version that the provider was originally tested with to the version that you are running now.

Also, look at sec 5.2.3.5 in this Condor manual page:
  http://www.cs.wisc.edu/condor/manual/v6.8/5_3Grid_Universe.html#SECTION00632500000000000000

which says that " needs to be escaped as \"

Its also possible that somehow the provider stopped doing this in some cases???

- Mike


----- "Arjun Comar" <mandaya at rose-hulman.edu> wrote:

> Hey all,
> I've been trying to get jobs submitted over Condor via swift, and
> running into a few problems. I think I've finally hit a point where
> it's the Condor provider itself that's failing over any of my
> configurations.
> Here's the sites entry (though any sites entry over Condor will do,
> and I've tried several):
> 
> <config>
> <pool handle=" Nebraska_red.unl.edu ">
> <gridftp url="gsiftp:// red.unl.edu/ "/>
> <execution provider="condor"/>
> <profile key="jobType" namespace="globus">grid</profile>
> <profile key="gridResource" namespace="globus">gt2
> red.unl.edu/jobmanager-condor </profile>
> <workdirectory>/opt/osg/data/engage/tmp/ red.unl.edu </workdirectory>
> </pool>
> </config>
> 
> And any swift script at all fails, even a simple helloworld:
> 
> type messagefile;
> app (messagefile t) greeting () {
> echo "Hello, world!" stdout=@filename(t);
> }
> messagefile outfile <"hello.txt">;
> outfile = greeting();
> 
> With the following error:
> The following errors have occurred:
> 1. Application "echo" failed (Cannot submit job: Could not submit job
> (condor_submit reported an exit code of 1). Submitting job(s)
> Found illegal unescaped double-quote: "" -e /bin/echo -out hello.txt
> -err stderr.txt -i -d "" -if "" -of hello.txt -k "" -status "provider
> " -a "Hello, world!"The full arguments you specified were:
> /opt/osg/data/engage/tmp/
> red.unl.edu/helloworld-20100623-1051-dr5v5apa/shared/_swiftwrap
> echo-dqt6jttj -jobdir d -scratch "" -e /bin/echo -out hello.txt -err
> stderr.txt -i -d "" -if "" -of hello.txt -k "" -status "provider " -a
> "Hello, world!")
> 
> The same script runs just fine through any other submission mechanism,
> even to the same site (ssh, coasters+ssh:pbs, etc).
> 
> Anyone have any thoughts on fixing the problem?
> 
> Thanks!
> 
> --
> Arjun Comar, Rose-Hulman '12

-- 
Michael Wilde
Computation Institute, University of Chicago
Mathematics and Computer Science Division
Argonne National Laboratory



From benc at hawaga.org.uk  Wed Jun 23 12:05:37 2010
From: benc at hawaga.org.uk (Ben Clifford)
Date: Wed, 23 Jun 2010 17:05:37 +0000 (GMT)
Subject: [Swift-user] Re: Error with condor provider
In-Reply-To: <21022873.1029631277312592907.JavaMail.root@zimbra>
References: <21022873.1029631277312592907.JavaMail.root@zimbra>
Message-ID: <Pine.LNX.4.64.1006231703570.21645@dildano.hawaga.org.uk>

> Its *possible* that condor quoting conventions changed from the version 
> that the provider was originally tested with to the version that you are 
> running now.

I think they did change.

At one point, I think there was no way to pass in one particular form of 
command-line at all (maybe something with spaces in?).

-- 


From wilde at mcs.anl.gov  Wed Jun 23 12:20:45 2010
From: wilde at mcs.anl.gov (Michael Wilde)
Date: Wed, 23 Jun 2010 12:20:45 -0500 (CDT)
Subject: [Swift-user] Re: Error with condor provider
In-Reply-To: <Pine.LNX.4.64.1006231703570.21645@dildano.hawaga.org.uk>
Message-ID: <3468861.1030261277313645538.JavaMail.root@zimbra>

So one possibility is that we need to add the \" escapes to the provider.

Arjun, this is a good exercise:

- hunt down the debug=true flags that I think you can specify in the provider's .properties file (these might need to get fixed; see the devel list archives on some discussion on this; also check my stable tree for such changes, but mine may not be working for this yet).

- experiment with the required .submit file format manually

- then get the provider to emiyt the right quoting.

Mike


----- "Ben Clifford" <benc at hawaga.org.uk> wrote:

> > Its *possible* that condor quoting conventions changed from the
> version 
> > that the provider was originally tested with to the version that you
> are 
> > running now.
> 
> I think they did change.
> 
> At one point, I think there was no way to pass in one particular form
> of 
> command-line at all (maybe something with spaces in?).
> 
> --

-- 
Michael Wilde
Computation Institute, University of Chicago
Mathematics and Computer Science Division
Argonne National Laboratory



From wilde at mcs.anl.gov  Wed Jun 23 12:36:08 2010
From: wilde at mcs.anl.gov (wilde at mcs.anl.gov)
Date: Wed, 23 Jun 2010 12:36:08 -0500 (CDT)
Subject: [Swift-user] Re: Error with condor provider
In-Reply-To: <5346887.1031071277314410993.JavaMail.root@zimbra>
Message-ID: <31737278.1031201277314568495.JavaMail.root@zimbra>

relevant threads:

  http://mail.ci.uchicago.edu/pipermail/swift-devel/2010-May/006398.html

  http://mail.ci.uchicago.edu/pipermail/swift-user/2010-February/001360.html

There were a few more messages off list on this which I will forward to the list after this one.

- Mike

----- "Michael Wilde" <wilde at mcs.anl.gov> wrote:

> So one possibility is that we need to add the \" escapes to the
> provider.
> 
> Arjun, this is a good exercise:
> 
> - hunt down the debug=true flags that I think you can specify in the
> provider's .properties file (these might need to get fixed; see the
> devel list archives on some discussion on this; also check my stable
> tree for such changes, but mine may not be working for this yet).
> 
> - experiment with the required .submit file format manually
> 
> - then get the provider to emiyt the right quoting.
> 
> Mike
> 
> 
> ----- "Ben Clifford" <benc at hawaga.org.uk> wrote:
> 
> > > Its *possible* that condor quoting conventions changed from the
> > version 
> > > that the provider was originally tested with to the version that
> you
> > are 
> > > running now.
> > 
> > I think they did change.
> > 
> > At one point, I think there was no way to pass in one particular
> form
> > of 
> > command-line at all (maybe something with spaces in?).
> > 
> > --
> 
> -- 
> Michael Wilde
> Computation Institute, University of Chicago
> Mathematics and Computer Science Division
> Argonne National Laboratory
> 
> _______________________________________________
> Swift-user mailing list
> Swift-user at ci.uchicago.edu
> http://mail.ci.uchicago.edu/mailman/listinfo/swift-user

-- 
Michael Wilde
Computation Institute, University of Chicago
Mathematics and Computer Science Division
Argonne National Laboratory



From wilde at mcs.anl.gov  Wed Jun 23 12:37:26 2010
From: wilde at mcs.anl.gov (Michael Wilde)
Date: Wed, 23 Jun 2010 12:37:26 -0500 (CDT)
Subject: [Swift-user] Re: How to set provider-specific options?
In-Reply-To: <1270597624.15431.0.camel@localhost>
Message-ID: <24562037.1031291277314646603.JavaMail.root@zimbra>

One more message on this thread that never made it to the list:

----- Forwarded Message -----
From: "Mihael Hategan" <hategan at mcs.anl.gov>
To: "Michael Wilde" <wilde at mcs.anl.gov>
Sent: Tuesday, April 6, 2010 6:47:04 PM GMT -06:00 US/Canada Central
Subject: Re: How to set provider-specific options?

I think so. I recall either you or I committing the relevant code.

On Tue, 2010-04-06 at 18:28 -0500, Michael Wilde wrote:
> is the debug=tru thing working now to save submit and response logs from sge?
> 
> 
> ----- "Mihael Hategan" <hategan at mcs.anl.gov> wrote:
> 
> > No. But you missed:
> >  script = File.createTempFile(getName(), ".submit", scriptdir);
> >  script.deleteOnExit();
> > 
> > On Thu, 2010-02-25 at 14:58 -0600, Michael Wilde wrote:
> > > Did I mis-interpret the following code in AbstractExecutor.java?
> > > 
> > >         protected void cleanup() {
> > >                 if (!getProperties().isDebugEnabled()) {
> > >                         script.delete();
> > >                         new File(exitcode).delete();
> > >                         if (spec.getStdOutput() == null && stdout !=
> > null) {
> > >                                 new File(stdout).delete();
> > >                         }
> > >                         if (spec.getStdError() == null && stderr !=
> > null) {
> > >                                 new File(stderr).delete();
> > >                         }
> > >                 }
> > >         }
> > > 
> > > - Mike
> > > 
> > > ----- "Mihael Hategan" <hategan at mcs.anl.gov> wrote:
> > > 
> > > > On Thu, 2010-02-25 at 14:46 -0600, wilde at mcs.anl.gov wrote:
> > > > > I tried to set debug=true as the last line in
> > > > > etc/provider-pbs.properties in my swift dist directory, hoping
> > to
> > > > get
> > > > > the PBS provider to retain the .submit file in
> > ~/.globus/scripts.
> > > > > That didnt seem to work: I didnt get a .submit file there.
> > > > 
> > > > I don't see any code that would prevent the deletion of the script
> > if
> > > > debug was set to true in that file.
> > > > 
> > > > What I would suggest is submitting some long-ish job and then
> > copying
> > > > the submit scripts from ~/.globus/scripts before they go away.
> 


-- 
Michael Wilde
Computation Institute, University of Chicago
Mathematics and Computer Science Division
Argonne National Laboratory



From wilde at mcs.anl.gov  Wed Jun 23 12:44:28 2010
From: wilde at mcs.anl.gov (Michael Wilde)
Date: Wed, 23 Jun 2010 12:44:28 -0500 (CDT)
Subject: [Swift-user] Re: How to set provider-specific options?
In-Reply-To: <24562037.1031291277314646603.JavaMail.root@zimbra>
Message-ID: <32761147.1031681277315068831.JavaMail.root@zimbra>

I think I got debug fixes made to the sge provider confused with the condor provider. Its possible that the debug flag in the condor provider is either already working, or needs to be fixed in the same manner as we did for the sge provider.

Arjun, its worth checking the svn logs on both. Note that some of the providers share common source files which is confusing when you first hunt for their codes.

- Mike

----- "Michael Wilde" <wilde at mcs.anl.gov> wrote:

> One more message on this thread that never made it to the list:
> 
> ----- Forwarded Message -----
> From: "Mihael Hategan" <hategan at mcs.anl.gov>
> To: "Michael Wilde" <wilde at mcs.anl.gov>
> Sent: Tuesday, April 6, 2010 6:47:04 PM GMT -06:00 US/Canada Central
> Subject: Re: How to set provider-specific options?
> 
> I think so. I recall either you or I committing the relevant code.
> 
> On Tue, 2010-04-06 at 18:28 -0500, Michael Wilde wrote:
> > is the debug=tru thing working now to save submit and response logs
> from sge?
> > 
> > 
> > ----- "Mihael Hategan" <hategan at mcs.anl.gov> wrote:
> > 
> > > No. But you missed:
> > >  script = File.createTempFile(getName(), ".submit", scriptdir);
> > >  script.deleteOnExit();
> > > 
> > > On Thu, 2010-02-25 at 14:58 -0600, Michael Wilde wrote:
> > > > Did I mis-interpret the following code in
> AbstractExecutor.java?
> > > > 
> > > >         protected void cleanup() {
> > > >                 if (!getProperties().isDebugEnabled()) {
> > > >                         script.delete();
> > > >                         new File(exitcode).delete();
> > > >                         if (spec.getStdOutput() == null &&
> stdout !=
> > > null) {
> > > >                                 new File(stdout).delete();
> > > >                         }
> > > >                         if (spec.getStdError() == null && stderr
> !=
> > > null) {
> > > >                                 new File(stderr).delete();
> > > >                         }
> > > >                 }
> > > >         }
> > > > 
> > > > - Mike
> > > > 
> > > > ----- "Mihael Hategan" <hategan at mcs.anl.gov> wrote:
> > > > 
> > > > > On Thu, 2010-02-25 at 14:46 -0600, wilde at mcs.anl.gov wrote:
> > > > > > I tried to set debug=true as the last line in
> > > > > > etc/provider-pbs.properties in my swift dist directory,
> hoping
> > > to
> > > > > get
> > > > > > the PBS provider to retain the .submit file in
> > > ~/.globus/scripts.
> > > > > > That didnt seem to work: I didnt get a .submit file there.
> > > > > 
> > > > > I don't see any code that would prevent the deletion of the
> script
> > > if
> > > > > debug was set to true in that file.
> > > > > 
> > > > > What I would suggest is submitting some long-ish job and then
> > > copying
> > > > > the submit scripts from ~/.globus/scripts before they go
> away.
> > 
> 
> 
> -- 
> Michael Wilde
> Computation Institute, University of Chicago
> Mathematics and Computer Science Division
> Argonne National Laboratory
> 
> _______________________________________________
> Swift-user mailing list
> Swift-user at ci.uchicago.edu
> http://mail.ci.uchicago.edu/mailman/listinfo/swift-user

-- 
Michael Wilde
Computation Institute, University of Chicago
Mathematics and Computer Science Division
Argonne National Laboratory



From jon.monette at gmail.com  Wed Jun 23 14:00:29 2010
From: jon.monette at gmail.com (Jonathan Monette)
Date: Wed, 23 Jun 2010 14:00:29 -0500
Subject: [Swift-user] Runtime error
Message-ID: <4C2259CD.8020700@gmail.com>

Hello,
I am getting a runtime exception for no such file of directory.  Here is 
the exact error:

Swift svn swift-r3343 cog-r2757

RunID: 20100623-1340-76qi2b0c
Progress:
Execution failed:
     java.lang.RuntimeException: diffs.tbl (No such file or directory)

There are several function calls before the one that fails but it seems 
that none of the other calls run.  The diffs.tbl file is suppose to be 
created in one of these function calls that isn't running.  The line I 
believe that is causing the problems is:

DiffStruct diffs[] <csv_mapper; file = @diff_tbl, skip = 1, hdelim = "| ">;

If I comment out the function call that contains this line the script 
runs perfectly.  But as soon as I un-comment this line it fails with the 
runtime exception above.  Is the csv_mapper always the first thing to 
run no matter what or is there some implementation detail I have 
overlooked.   Any thoughts?

-- 
Jon

Computers are incredibly fast, accurate, and stupid. Human beings are incredibly slow, inaccurate, and brilliant. Together they are powerful beyond imagination.
- Albert Einstein



From aespinosa at cs.uchicago.edu  Wed Jun 23 16:12:50 2010
From: aespinosa at cs.uchicago.edu (Allan Espinosa)
Date: Wed, 23 Jun 2010 16:12:50 -0500
Subject: [Swift-user] Runtime error
In-Reply-To: <4C2259CD.8020700@gmail.com>
References: <4C2259CD.8020700@gmail.com>
Message-ID: <AANLkTimqbLJxFS6eiSr2qG2Q6Ep7-pmK4WDbT02oMJIW@mail.gmail.com>

how's @diff_tbl declared?

2010/6/23 Jonathan Monette <jon.monette at gmail.com>:
> Hello,
> I am getting a runtime exception for no such file of directory. ?Here is the
> exact error:
>
> Swift svn swift-r3343 cog-r2757
>
> RunID: 20100623-1340-76qi2b0c
> Progress:
> Execution failed:
> ? ?java.lang.RuntimeException: diffs.tbl (No such file or directory)
>
> There are several function calls before the one that fails but it seems that
> none of the other calls run. ?The diffs.tbl file is suppose to be created in
> one of these function calls that isn't running. ?The line I believe that is
> causing the problems is:
>
> DiffStruct diffs[] <csv_mapper; file = @diff_tbl, skip = 1, hdelim = "| ">;
>
> If I comment out the function call that contains this line the script runs
> perfectly. ?But as soon as I un-comment this line it fails with the runtime
> exception above. ?Is the csv_mapper always the first thing to run no matter
> what or is there some implementation detail I have overlooked. ? Any
> thoughts?
>
> --
> Jon
>
> Computers are incredibly fast, accurate, and stupid. Human beings are
> incredibly slow, inaccurate, and brilliant. Together they are powerful
> beyond imagination.
> - Albert Einstein


From hategan at mcs.anl.gov  Wed Jun 23 16:22:39 2010
From: hategan at mcs.anl.gov (Mihael Hategan)
Date: Wed, 23 Jun 2010 16:22:39 -0500
Subject: [Swift-user] Re: Runtime error
In-Reply-To: <4C2259CD.8020700@gmail.com>
References: <4C2259CD.8020700@gmail.com>
Message-ID: <1277328159.7366.1.camel@blabla2.none>

Try file = diff_tbl.

If you say @diff_tbl, that is the file name and it is known before the
file is generated, which means the csv mapper does not see any reason to
wait.

Mihael

On Wed, 2010-06-23 at 14:00 -0500, Jonathan Monette wrote:
> Hello,
> I am getting a runtime exception for no such file of directory.  Here is 
> the exact error:
> 
> Swift svn swift-r3343 cog-r2757
> 
> RunID: 20100623-1340-76qi2b0c
> Progress:
> Execution failed:
>      java.lang.RuntimeException: diffs.tbl (No such file or directory)
> 
> There are several function calls before the one that fails but it seems 
> that none of the other calls run.  The diffs.tbl file is suppose to be 
> created in one of these function calls that isn't running.  The line I 
> believe that is causing the problems is:
> 
> DiffStruct diffs[] <csv_mapper; file = @diff_tbl, skip = 1, hdelim = "| ">;
> 
> If I comment out the function call that contains this line the script 
> runs perfectly.  But as soon as I un-comment this line it fails with the 
> runtime exception above.  Is the csv_mapper always the first thing to 
> run no matter what or is there some implementation detail I have 
> overlooked.   Any thoughts?
> 




From jon.monette at gmail.com  Wed Jun 23 16:57:12 2010
From: jon.monette at gmail.com (Jonathan Monette)
Date: Wed, 23 Jun 2010 16:57:12 -0500
Subject: [Swift-user] Re: Runtime error
In-Reply-To: <1277328159.7366.1.camel@blabla2.none>
References: <4C2259CD.8020700@gmail.com> <1277328159.7366.1.camel@blabla2.none>
Message-ID: <4C228338.8050201@gmail.com>

Yes.  I believe the problem has been solved.  Thanks.

On 6/23/10 4:22 PM, Mihael Hategan wrote:
> Try file = diff_tbl.
>
> If you say @diff_tbl, that is the file name and it is known before the
> file is generated, which means the csv mapper does not see any reason to
> wait.
>
> Mihael
>
> On Wed, 2010-06-23 at 14:00 -0500, Jonathan Monette wrote:
>    
>> Hello,
>> I am getting a runtime exception for no such file of directory.  Here is
>> the exact error:
>>
>> Swift svn swift-r3343 cog-r2757
>>
>> RunID: 20100623-1340-76qi2b0c
>> Progress:
>> Execution failed:
>>       java.lang.RuntimeException: diffs.tbl (No such file or directory)
>>
>> There are several function calls before the one that fails but it seems
>> that none of the other calls run.  The diffs.tbl file is suppose to be
>> created in one of these function calls that isn't running.  The line I
>> believe that is causing the problems is:
>>
>> DiffStruct diffs[]<csv_mapper; file = @diff_tbl, skip = 1, hdelim = "| ">;
>>
>> If I comment out the function call that contains this line the script
>> runs perfectly.  But as soon as I un-comment this line it fails with the
>> runtime exception above.  Is the csv_mapper always the first thing to
>> run no matter what or is there some implementation detail I have
>> overlooked.   Any thoughts?
>>
>>      
>
>
>    

-- 
Jon

Computers are incredibly fast, accurate, and stupid. Human beings are incredibly slow, inaccurate, and brilliant. Together they are powerful beyond imagination.
- Albert Einstein



From wilde at mcs.anl.gov  Wed Jun 23 17:40:28 2010
From: wilde at mcs.anl.gov (wilde at mcs.anl.gov)
Date: Wed, 23 Jun 2010 17:40:28 -0500 (CDT)
Subject: [Swift-user] Re: Error with condor provider
In-Reply-To: <5863139.1048441277332730911.JavaMail.root@zimbra>
Message-ID: <22863415.1048501277332828795.JavaMail.root@zimbra>


----- "Arjun Comar" <mandaya at rose-hulman.edu> wrote:

> Alright, so I just tried again with trunk over Mike's stable
> repository and it worked, so it looks like whatever the problem was
> got fixed between stable and trunk.

Or uncommitted in my working dirs and needs testing and checkin :(

> However submitting jobs to condor
> results in the job simply getting held. Do I need any permissions over
> grid-proxy-init? Same thing happens when I run condor_submit on the
> command line.

There is a good little guide on debugging Condor problems:
  http://www.cs.wisc.edu/condor/CondorWeek2004/presentations/effective_condorg_v4.doc
(likely superseded now; talks about what causes jobs to go on hold, and how to look at log files)

Things like bad paths or args can cause jobs to fail, and get held and/or retried.

Below, the host name gsu1.uchicago.edu may be some grid school host that is down or non-existent???

- Mike

> 
> Condor submit file:
> executable=/bin/echo
> arguments=Hello World!
> output=results.output
> error=results.error
> log=results.log
> notification=never
> universe=grid
> grid_resource=gt2 gsu1.uchicago.edu/jobmanager-fork
> queue
> 
> and I submitted with:
> condor_submit myjob.submit
> 
> results in:
> -- Submitter: bridled.ci.uchicago.edu : < 128.135.125.18:49572 > :
> bridled.ci.uchicago.edu
> ID OWNER SUBMITTED RUN_TIME ST PRI SIZE CMD
> 57.0 aespinosa 6/15 11:54 0+00:00:00 I 0 7.3 condor_dagman
> 71.0 arjun 6/23 11:24 0+00:00:00 H 0 0.0 echo Hello World!
> 72.0 arjun 6/23 17:05 0+00:00:00 H 0 1.0 bash /opt/osg/data
> 
> 3 jobs; 1 idle, 0 running, 2 held
> 
> (My first job is the condor-submit job, the second is the job I
> attempted to submit via swift.)
> 
> Any thoughts?
> 
> Arjun
> 
> 
> On Wed, Jun 23, 2010 at 11:26 AM, Arjun Comar <
> mandaya at rose-hulman.edu > wrote:
> 
> 
> Hey all,
> I've been trying to get jobs submitted over Condor via swift, and
> running into a few problems. I think I've finally hit a point where
> it's the Condor provider itself that's failing over any of my
> configurations.
> Here's the sites entry (though any sites entry over Condor will do,
> and I've tried several):
> 
> <config>
> <pool handle=" Nebraska_red.unl.edu ">
> <gridftp url="gsiftp:// red.unl.edu/ "/>
> <execution provider="condor"/>
> <profile key="jobType" namespace="globus">grid</profile>
> <profile key="gridResource" namespace="globus">gt2
> red.unl.edu/jobmanager-condor </profile>
> <workdirectory>/opt/osg/data/engage/tmp/ red.unl.edu </workdirectory>
> </pool>
> </config>
> 
> And any swift script at all fails, even a simple helloworld:
> 
> type messagefile;
> app (messagefile t) greeting () {
> echo "Hello, world!" stdout=@filename(t);
> }
> messagefile outfile <"hello.txt">;
> outfile = greeting();
> 
> With the following error:
> The following errors have occurred:
> 1. Application "echo" failed (Cannot submit job: Could not submit job
> (condor_submit reported an exit code of 1). Submitting job(s)
> Found illegal unescaped double-quote: "" -e /bin/echo -out hello.txt
> -err stderr.txt -i -d "" -if "" -of hello.txt -k "" -status "provider
> " -a "Hello, world!"The full arguments you specified were:
> /opt/osg/data/engage/tmp/
> red.unl.edu/helloworld-20100623-1051-dr5v5apa/shared/_swiftwrap
> echo-dqt6jttj -jobdir d -scratch "" -e /bin/echo -out hello.txt -err
> stderr.txt -i -d "" -if "" -of hello.txt -k "" -status "provider " -a
> "Hello, world!")
> 
> The same script runs just fine through any other submission mechanism,
> even to the same site (ssh, coasters+ssh:pbs, etc).
> 
> Anyone have any thoughts on fixing the problem?
> 
> Thanks!
> 
> --
> Arjun Comar, Rose-Hulman '12
> 
> 
> 
> --
> Arjun Comar, Rose-Hulman '12

-- 
Michael Wilde
Computation Institute, University of Chicago
Mathematics and Computer Science Division
Argonne National Laboratory



From mandaya at rose-hulman.edu  Wed Jun 23 17:21:34 2010
From: mandaya at rose-hulman.edu (Arjun Comar)
Date: Wed, 23 Jun 2010 17:21:34 -0500
Subject: [Swift-user] Re: Error with condor provider
In-Reply-To: <AANLkTik2QVRc6BLxBs8UNeO9zFg8KcH8rp28SdvLmtue@mail.gmail.com>
References: <AANLkTik2QVRc6BLxBs8UNeO9zFg8KcH8rp28SdvLmtue@mail.gmail.com>
Message-ID: <AANLkTik8J1hZG80Q4kBqWaLeluYUuGRmVz4zPs_oKYgi@mail.gmail.com>

Alright, so I just tried again with trunk over Mike's stable repository and
it worked, so it looks like whatever the problem was got fixed between
stable and trunk. However submitting jobs to condor results in the job
simply getting held. Do I need any permissions over grid-proxy-init? Same
thing happens when I run condor_submit on the command line.

Condor submit file:
executable=/bin/echo
arguments=Hello World!
output=results.output
error=results.error
log=results.log
notification=never
universe=grid
grid_resource=gt2 gsu1.uchicago.edu/jobmanager-fork
queue

and I submitted with:
condor_submit myjob.submit

results in:
-- Submitter: bridled.ci.uchicago.edu : <128.135.125.18:49572> :
bridled.ci.uchicago.edu
 ID     OWNER            SUBMITTED     RUN_TIME ST PRI SIZE CMD
  57.0   aespinosa     6/15 11:54   0+00:00:00 I  0   7.3  condor_dagman
  71.0   arjun           6/23 11:24   0+00:00:00 H  0   0.0  echo Hello
World!
  72.0   arjun           6/23 17:05   0+00:00:00 H  0   1.0  bash
/opt/osg/data

3 jobs; 1 idle, 0 running, 2 held

(My first job is the condor-submit job, the second is the job I attempted to
submit via swift.)

Any thoughts?

Arjun

On Wed, Jun 23, 2010 at 11:26 AM, Arjun Comar <mandaya at rose-hulman.edu>wrote:

> Hey all,
> I've been trying to get jobs submitted over Condor via swift, and running
> into a few problems. I think I've finally hit a point where it's the Condor
> provider itself that's failing over any of my configurations.
> Here's the sites entry (though any sites entry over Condor will do, and
> I've tried several):
>
> <config>
>   <pool handle="Nebraska_red.unl.edu">
>     <gridftp url="gsiftp://red.unl.edu/"/>
>     <execution provider="condor"/>
>     <profile key="jobType" namespace="globus">grid</profile>
>     <profile key="gridResource" namespace="globus">gt2
> red.unl.edu/jobmanager-condor</profile>
>     <workdirectory>/opt/osg/data/engage/tmp/red.unl.edu</workdirectory>
>   </pool>
> </config>
>
> And any swift script at all fails, even a simple helloworld:
>
> type messagefile;
> app (messagefile t) greeting () {
>         echo "Hello, world!" stdout=@filename(t);
> }
> messagefile outfile <"hello.txt">;
> outfile = greeting();
>
> With the following error:
> The following errors have occurred:
> 1. Application "echo" failed (Cannot submit job: Could not submit job
> (condor_submit reported an exit code of 1). Submitting job(s)
> Found illegal unescaped double-quote: "" -e /bin/echo -out hello.txt -err
> stderr.txt -i -d "" -if "" -of hello.txt -k "" -status "provider " -a
> "Hello, world!"The full arguments you specified were:
> /opt/osg/data/engage/tmp/
> red.unl.edu/helloworld-20100623-1051-dr5v5apa/shared/_swiftwrapecho-dqt6jttj -jobdir d -scratch "" -e /bin/echo -out hello.txt -err
> stderr.txt -i -d "" -if "" -of hello.txt -k "" -status "provider " -a
> "Hello, world!")
>
> The same script runs just fine through any other submission mechanism, even
> to the same site (ssh, coasters+ssh:pbs, etc).
>
> Anyone have any thoughts on fixing the problem?
>
> Thanks!
>
> --
> Arjun Comar, Rose-Hulman '12
>



-- 
Arjun Comar, Rose-Hulman '12
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-user/attachments/20100623/ab27cdfd/attachment.html>

From hockyg at uchicago.edu  Thu Jun 24 10:36:56 2010
From: hockyg at uchicago.edu (Glen Hocky)
Date: Thu, 24 Jun 2010 11:36:56 -0400
Subject: [Swift-user] control number of condor nodes?
Message-ID: <AANLkTilylpJTJatrxKCTLPAYqn_pkgwInuwjq8zclTrN@mail.gmail.com>

Hey Everyone,
Quick question. I'm testing on bg/p and things are working fine, but I can't
seem to increase the number of nodes I'm using in the way I expect. I can
run fine on 64 nodes tih

     <filesystem provider="local"/>

    <execution provider="coaster" jobmanager="local:cobalt"/>

    <profile namespace="globus" key="queue">prod-devel</profile>

    <profile namespace="globus" key="slots">1</profile>

    <profile namespace="globus" key="nodeGranularity">64</profile>

    <profile namespace="globus" key="workersPerNode">4</profile>

    <profile namespace="globus" key="maxNodes">64</profile>

    <profile namespace="globus" key="project">OOPS</profile>

    <profile namespace="globus" key="kernelprofile">zeptoos</profile>

    <profile namespace="globus" key="maxtime">3559</profile>

    <profile namespace="globus" key="alcfbgpnat">true</profile>

    <profile namespace="karajan" key="jobThrottle">2.55</profile>

    <profile namespace="karajan" key="initialScore">100000</profile>

    <scratch>/tmp</scratch>


>  <workdirectory>/home/hockyg/reichman/glassy_dynamics/code/swift/work</workdirectory>


I successfully went up to 256 nodes by increasing slots to 4 but I was
unable to get a block of size 256.
When I set maxnodes to 256, slots=1 , i get a block of size 128 regardless
of several changes (and removal) of jobThrottle

Any ideas on how to fine tune this, because I will certainly need to choose
my number of jobs based on an expected ( max wall time * workers per node *
number of nodes )

Glen
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-user/attachments/20100624/2a4315d6/attachment.html>

From aespinosa at cs.uchicago.edu  Thu Jun 24 10:40:24 2010
From: aespinosa at cs.uchicago.edu (Allan Espinosa)
Date: Thu, 24 Jun 2010 10:40:24 -0500
Subject: [Swift-user] control number of condor nodes?
In-Reply-To: <AANLkTilylpJTJatrxKCTLPAYqn_pkgwInuwjq8zclTrN@mail.gmail.com>
References: <AANLkTilylpJTJatrxKCTLPAYqn_pkgwInuwjq8zclTrN@mail.gmail.com>
Message-ID: <AANLkTim5Yin9pGFsnGaS9h_39nLD4nV2N41RroeNq0Ku@mail.gmail.com>

hi Glen,

How about setting the spread to be more uniform (like 0.1)?

-Allan
2010/6/24 Glen Hocky <hockyg at uchicago.edu>:
> Hey Everyone,
> Quick question. I'm testing on bg/p and things are working fine, but I can't
> seem to increase the number of nodes I'm using in the way I expect. I can
> run fine on 64 nodes tih
>>
>> ?? ?<filesystem provider="local"/>
>>
>> ?? ?<execution provider="coaster" jobmanager="local:cobalt"/>
>>
>> ?? ?<profile namespace="globus" key="queue">prod-devel</profile>
>>
>> ?? ?<profile namespace="globus" key="slots">1</profile>
>>
>> ?? ?<profile namespace="globus" key="nodeGranularity">64</profile>
>>
>> ?? ?<profile namespace="globus" key="workersPerNode">4</profile>
>>
>> ?? ?<profile namespace="globus" key="maxNodes">64</profile>
>>
>> ?? ?<profile namespace="globus" key="project">OOPS</profile>
>>
>> ?? ?<profile namespace="globus" key="kernelprofile">zeptoos</profile>
>>
>> ?? ?<profile namespace="globus" key="maxtime">3559</profile>
>>
>> ?? ?<profile namespace="globus" key="alcfbgpnat">true</profile>
>>
>> ?? ?<profile namespace="karajan" key="jobThrottle">2.55</profile>
>>
>> ?? ?<profile namespace="karajan" key="initialScore">100000</profile>
>>
>> ?? ?<scratch>/tmp</scratch>
>>
>>
>> ?<workdirectory>/home/hockyg/reichman/glassy_dynamics/code/swift/work</workdirectory>
>
> I successfully went up to 256 nodes by increasing slots to 4 but I was
> unable to get a block of size 256.
> When I set maxnodes to 256, slots=1 , i get a block of size 128 regardless
> of several changes (and removal) of jobThrottle
> Any ideas on how to fine tune this, because I will certainly need to choose
> my number of jobs based on an expected ( max wall time * workers per node *
> number of nodes )
> Glen


From hockyg at uchicago.edu  Thu Jun 24 10:51:40 2010
From: hockyg at uchicago.edu (Glen Hocky)
Date: Thu, 24 Jun 2010 11:51:40 -0400
Subject: [Swift-user] control number of condor nodes?
In-Reply-To: <AANLkTim5Yin9pGFsnGaS9h_39nLD4nV2N41RroeNq0Ku@mail.gmail.com>
References: <AANLkTilylpJTJatrxKCTLPAYqn_pkgwInuwjq8zclTrN@mail.gmail.com> 
	<AANLkTim5Yin9pGFsnGaS9h_39nLD4nV2N41RroeNq0Ku@mail.gmail.com>
Message-ID: <AANLkTinwrMb7281u_JCxj82VQhv2vf4Gnaa8BUjuSYDi@mail.gmail.com>

Which parameter is that?

On Thu, Jun 24, 2010 at 11:40 AM, Allan Espinosa
<aespinosa at cs.uchicago.edu>wrote:

> hi Glen,
>
> How about setting the spread to be more uniform (like 0.1)?
>
> -Allan
> 2010/6/24 Glen Hocky <hockyg at uchicago.edu>:
> > Hey Everyone,
> > Quick question. I'm testing on bg/p and things are working fine, but I
> can't
> > seem to increase the number of nodes I'm using in the way I expect. I can
> > run fine on 64 nodes tih
> >>
> >>     <filesystem provider="local"/>
> >>
> >>     <execution provider="coaster" jobmanager="local:cobalt"/>
> >>
> >>     <profile namespace="globus" key="queue">prod-devel</profile>
> >>
> >>     <profile namespace="globus" key="slots">1</profile>
> >>
> >>     <profile namespace="globus" key="nodeGranularity">64</profile>
> >>
> >>     <profile namespace="globus" key="workersPerNode">4</profile>
> >>
> >>     <profile namespace="globus" key="maxNodes">64</profile>
> >>
> >>     <profile namespace="globus" key="project">OOPS</profile>
> >>
> >>     <profile namespace="globus" key="kernelprofile">zeptoos</profile>
> >>
> >>     <profile namespace="globus" key="maxtime">3559</profile>
> >>
> >>     <profile namespace="globus" key="alcfbgpnat">true</profile>
> >>
> >>     <profile namespace="karajan" key="jobThrottle">2.55</profile>
> >>
> >>     <profile namespace="karajan" key="initialScore">100000</profile>
> >>
> >>     <scratch>/tmp</scratch>
> >>
> >>
> >>
>  <workdirectory>/home/hockyg/reichman/glassy_dynamics/code/swift/work</workdirectory>
> >
> > I successfully went up to 256 nodes by increasing slots to 4 but I was
> > unable to get a block of size 256.
> > When I set maxnodes to 256, slots=1 , i get a block of size 128
> regardless
> > of several changes (and removal) of jobThrottle
> > Any ideas on how to fine tune this, because I will certainly need to
> choose
> > my number of jobs based on an expected ( max wall time * workers per node
> *
> > number of nodes )
> > Glen
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-user/attachments/20100624/08959e9f/attachment.html>

From aespinosa at cs.uchicago.edu  Thu Jun 24 10:54:02 2010
From: aespinosa at cs.uchicago.edu (Allan Espinosa)
Date: Thu, 24 Jun 2010 10:54:02 -0500
Subject: [Swift-user] control number of condor nodes?
In-Reply-To: <AANLkTinwrMb7281u_JCxj82VQhv2vf4Gnaa8BUjuSYDi@mail.gmail.com>
References: <AANLkTilylpJTJatrxKCTLPAYqn_pkgwInuwjq8zclTrN@mail.gmail.com>
	<AANLkTim5Yin9pGFsnGaS9h_39nLD4nV2N41RroeNq0Ku@mail.gmail.com>
	<AANLkTinwrMb7281u_JCxj82VQhv2vf4Gnaa8BUjuSYDi@mail.gmail.com>
Message-ID: <AANLkTin5E1QzbSWEZFe7FT87GAIf3FtDoyDtmCkgANZL@mail.gmail.com>

under the globus namespace:

<profile namespace="globus" key="spread">0.1</profile>

http://www.ci.uchicago.edu/swift/guides/userguide.php#profile.globus

2010/6/24 Glen Hocky <hockyg at uchicago.edu>:
> Which parameter is that?
>
> On Thu, Jun 24, 2010 at 11:40 AM, Allan Espinosa <aespinosa at cs.uchicago.edu>
> wrote:
>>
>> hi Glen,
>>
>> How about setting the spread to be more uniform (like 0.1)?
>>
>> -Allan
>> 2010/6/24 Glen Hocky <hockyg at uchicago.edu>:
>> > Hey Everyone,
>> > Quick question. I'm testing on bg/p and things are working fine, but I
>> > can't
>> > seem to increase the number of nodes I'm using in the way I expect. I
>> > can
>> > run fine on 64 nodes tih
>> >>
>> >> ?? ?<filesystem provider="local"/>
>> >>
>> >> ?? ?<execution provider="coaster" jobmanager="local:cobalt"/>
>> >>
>> >> ?? ?<profile namespace="globus" key="queue">prod-devel</profile>
>> >>
>> >> ?? ?<profile namespace="globus" key="slots">1</profile>
>> >>
>> >> ?? ?<profile namespace="globus" key="nodeGranularity">64</profile>
>> >>
>> >> ?? ?<profile namespace="globus" key="workersPerNode">4</profile>
>> >>
>> >> ?? ?<profile namespace="globus" key="maxNodes">64</profile>
>> >>
>> >> ?? ?<profile namespace="globus" key="project">OOPS</profile>
>> >>
>> >> ?? ?<profile namespace="globus" key="kernelprofile">zeptoos</profile>
>> >>
>> >> ?? ?<profile namespace="globus" key="maxtime">3559</profile>
>> >>
>> >> ?? ?<profile namespace="globus" key="alcfbgpnat">true</profile>
>> >>
>> >> ?? ?<profile namespace="karajan" key="jobThrottle">2.55</profile>
>> >>
>> >> ?? ?<profile namespace="karajan" key="initialScore">100000</profile>
>> >>
>> >> ?? ?<scratch>/tmp</scratch>
>> >>
>> >>
>> >>
>> >> ?<workdirectory>/home/hockyg/reichman/glassy_dynamics/code/swift/work</workdirectory>
>> >
>> > I successfully went up to 256 nodes by increasing slots to 4 but I was
>> > unable to get a block of size 256.
>> > When I set maxnodes to 256, slots=1 , i get a block of size 128
>> > regardless
>> > of several changes (and removal) of jobThrottle
>> > Any ideas on how to fine tune this, because I will certainly need to
>> > choose
>> > my number of jobs based on an expected ( max wall time * workers per
>> > node *
>> > number of nodes )
>> > Glen
>
>



-- 
Allan M. Espinosa <http://amespinosa.wordpress.com>
PhD student, Computer Science
University of Chicago <http://people.cs.uchicago.edu/~aespinosa>


From mandaya at rose-hulman.edu  Thu Jun 24 11:24:53 2010
From: mandaya at rose-hulman.edu (Arjun Comar)
Date: Thu, 24 Jun 2010 11:24:53 -0500
Subject: [Swift-user] Re: Error with condor provider
In-Reply-To: <22863415.1048501277332828795.JavaMail.root@zimbra>
References: <5863139.1048441277332730911.JavaMail.root@zimbra> 
	<22863415.1048501277332828795.JavaMail.root@zimbra>
Message-ID: <AANLkTimf8FsDgOV71gaLvJzELuaN8FS-a9K4J_tmk7W6@mail.gmail.com>

I don't think it's just the host, because the swift job was submitted to
red.unl.edu, which I know is a valid host. And Justin suggested running
without that gridresource line, but then it just fails outright. So I
switched the gridresource to gt2 red.unl.edu/jobmanager-fork, and ran again,
and this time I got an actual reason for being held in the log file. Here's
the exact line:
012 (077.000.000) 06/24 11:21:05 Job was held.
        Failed to get expiration time of proxy
        Code 0 Subcode 0
...

I have a valid proxy, so I don't know what the problem is. Does this
indicate that I need to switch to voms-proxy over grid-proxy?

Arjun

On Wed, Jun 23, 2010 at 5:40 PM, wilde at mcs.anl.gov <wilde at mcs.anl.gov>wrote:

>
> ----- "Arjun Comar" <mandaya at rose-hulman.edu> wrote:
>
> > Alright, so I just tried again with trunk over Mike's stable
> > repository and it worked, so it looks like whatever the problem was
> > got fixed between stable and trunk.
>
> Or uncommitted in my working dirs and needs testing and checkin :(
>
> > However submitting jobs to condor
> > results in the job simply getting held. Do I need any permissions over
> > grid-proxy-init? Same thing happens when I run condor_submit on the
> > command line.
>
> There is a good little guide on debugging Condor problems:
>
> http://www.cs.wisc.edu/condor/CondorWeek2004/presentations/effective_condorg_v4.doc
> (likely superseded now; talks about what causes jobs to go on hold, and how
> to look at log files)
>
> Things like bad paths or args can cause jobs to fail, and get held and/or
> retried.
>
> Below, the host name gsu1.uchicago.edu may be some grid school host that
> is down or non-existent???
>
> - Mike
>
> >
> > Condor submit file:
> > executable=/bin/echo
> > arguments=Hello World!
> > output=results.output
> > error=results.error
> > log=results.log
> > notification=never
> > universe=grid
> > grid_resource=gt2 gsu1.uchicago.edu/jobmanager-fork
> > queue
> >
> > and I submitted with:
> > condor_submit myjob.submit
> >
> > results in:
> > -- Submitter: bridled.ci.uchicago.edu : < 128.135.125.18:49572 > :
> > bridled.ci.uchicago.edu
> > ID OWNER SUBMITTED RUN_TIME ST PRI SIZE CMD
> > 57.0 aespinosa 6/15 11:54 0+00:00:00 I 0 7.3 condor_dagman
> > 71.0 arjun 6/23 11:24 0+00:00:00 H 0 0.0 echo Hello World!
> > 72.0 arjun 6/23 17:05 0+00:00:00 H 0 1.0 bash /opt/osg/data
> >
> > 3 jobs; 1 idle, 0 running, 2 held
> >
> > (My first job is the condor-submit job, the second is the job I
> > attempted to submit via swift.)
> >
> > Any thoughts?
> >
> > Arjun
> >
> >
> > On Wed, Jun 23, 2010 at 11:26 AM, Arjun Comar <
> > mandaya at rose-hulman.edu > wrote:
> >
> >
> > Hey all,
> > I've been trying to get jobs submitted over Condor via swift, and
> > running into a few problems. I think I've finally hit a point where
> > it's the Condor provider itself that's failing over any of my
> > configurations.
> > Here's the sites entry (though any sites entry over Condor will do,
> > and I've tried several):
> >
> > <config>
> > <pool handle=" Nebraska_red.unl.edu ">
> > <gridftp url="gsiftp:// red.unl.edu/ "/>
> > <execution provider="condor"/>
> > <profile key="jobType" namespace="globus">grid</profile>
> > <profile key="gridResource" namespace="globus">gt2
> > red.unl.edu/jobmanager-condor </profile>
> > <workdirectory>/opt/osg/data/engage/tmp/ red.unl.edu </workdirectory>
> > </pool>
> > </config>
> >
> > And any swift script at all fails, even a simple helloworld:
> >
> > type messagefile;
> > app (messagefile t) greeting () {
> > echo "Hello, world!" stdout=@filename(t);
> > }
> > messagefile outfile <"hello.txt">;
> > outfile = greeting();
> >
> > With the following error:
> > The following errors have occurred:
> > 1. Application "echo" failed (Cannot submit job: Could not submit job
> > (condor_submit reported an exit code of 1). Submitting job(s)
> > Found illegal unescaped double-quote: "" -e /bin/echo -out hello.txt
> > -err stderr.txt -i -d "" -if "" -of hello.txt -k "" -status "provider
> > " -a "Hello, world!"The full arguments you specified were:
> > /opt/osg/data/engage/tmp/
> > red.unl.edu/helloworld-20100623-1051-dr5v5apa/shared/_swiftwrap
> > echo-dqt6jttj -jobdir d -scratch "" -e /bin/echo -out hello.txt -err
> > stderr.txt -i -d "" -if "" -of hello.txt -k "" -status "provider " -a
> > "Hello, world!")
> >
> > The same script runs just fine through any other submission mechanism,
> > even to the same site (ssh, coasters+ssh:pbs, etc).
> >
> > Anyone have any thoughts on fixing the problem?
> >
> > Thanks!
> >
> > --
> > Arjun Comar, Rose-Hulman '12
> >
> >
> >
> > --
> > Arjun Comar, Rose-Hulman '12
>
> --
> Michael Wilde
> Computation Institute, University of Chicago
> Mathematics and Computer Science Division
> Argonne National Laboratory
>
>


-- 
Arjun Comar, Rose-Hulman '12
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-user/attachments/20100624/768673ac/attachment.html>

From jadeddragon1111 at yahoo.com  Thu Jun 24 12:14:36 2010
From: jadeddragon1111 at yahoo.com (Lee S)
Date: Thu, 24 Jun 2010 10:14:36 -0700 (PDT)
Subject: [Swift-user] Formatting in Swift
Message-ID: <785165.61857.qm@web113708.mail.gq1.yahoo.com>

Hello,

I'm trying to figure out how I can control formatting in Swift,
For example, in c-scripting there is set myVar = `printf "%04d" {$Integer}`
Is there a way to access these formatting options in swift?

Thank you


      
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-user/attachments/20100624/5dcff3e4/attachment.html>

From wilde at mcs.anl.gov  Thu Jun 24 12:52:33 2010
From: wilde at mcs.anl.gov (Michael Wilde)
Date: Thu, 24 Jun 2010 12:52:33 -0500 (CDT)
Subject: [Swift-user] Re: Error with condor provider
In-Reply-To: <AANLkTimf8FsDgOV71gaLvJzELuaN8FS-a9K4J_tmk7W6@mail.gmail.com>
Message-ID: <22733478.1073521277401953725.JavaMail.root@zimbra>


----- "Arjun Comar" <mandaya at rose-hulman.edu> wrote:

> I don't think it's just the host, because the swift job was submitted
> to red.unl.edu , which I know is a valid host. And Justin suggested
> running without that gridresource line, but then it just fails
> outright. So I switched the gridresource to gt2
> red.unl.edu/jobmanager-fork , and ran again, and this time I got an
> actual reason for being held in the log file. Here's the exact line:
> 012 (077.000.000) 06/24 11:21:05 Job was held.
> Failed to get expiration time of proxy
> Code 0 Subcode 0
> ...
> 
> I have a valid proxy, so I don't know what the problem is. Does this
> indicate that I need to switch to voms-proxy over grid-proxy?

You should test this step by step, layer by layer. Use whatever proxy type you would use when testing with the standard grid commands. 

Try a short simple job using the gt2 fork provider (jobmanager=fork in the sites entry I think it is).

Are you testing standalone condor-g?

Are you doing this from engage-submit (where there is a known-working OSG client stack)?

- Mike


> 
> Arjun
> 
> 
> On Wed, Jun 23, 2010 at 5:40 PM, wilde at mcs.anl.gov < wilde at mcs.anl.gov
> > wrote:
> 
> 
> 
> 
> ----- "Arjun Comar" < mandaya at rose-hulman.edu > wrote:
> 
> > Alright, so I just tried again with trunk over Mike's stable
> > repository and it worked, so it looks like whatever the problem was
> > got fixed between stable and trunk.
> 
> Or uncommitted in my working dirs and needs testing and checkin :(
> 
> 
> > However submitting jobs to condor
> > results in the job simply getting held. Do I need any permissions
> over
> > grid-proxy-init? Same thing happens when I run condor_submit on the
> > command line.
> 
> There is a good little guide on debugging Condor problems:
> http://www.cs.wisc.edu/condor/CondorWeek2004/presentations/effective_condorg_v4.doc
> (likely superseded now; talks about what causes jobs to go on hold,
> and how to look at log files)
> 
> Things like bad paths or args can cause jobs to fail, and get held
> and/or retried.
> 
> Below, the host name gsu1.uchicago.edu may be some grid school host
> that is down or non-existent???
> 
> - Mike
> 
> 
> 
> 
> >
> > Condor submit file:
> > executable=/bin/echo
> > arguments=Hello World!
> > output=results.output
> > error=results.error
> > log=results.log
> > notification=never
> > universe=grid
> > grid_resource=gt2 gsu1.uchicago.edu/jobmanager-fork
> > queue
> >
> > and I submitted with:
> > condor_submit myjob.submit
> >
> > results in:
> > -- Submitter: bridled.ci.uchicago.edu : < 128.135.125.18:49572 > :
> > bridled.ci.uchicago.edu
> > ID OWNER SUBMITTED RUN_TIME ST PRI SIZE CMD
> > 57.0 aespinosa 6/15 11:54 0+00:00:00 I 0 7.3 condor_dagman
> > 71.0 arjun 6/23 11:24 0+00:00:00 H 0 0.0 echo Hello World!
> > 72.0 arjun 6/23 17:05 0+00:00:00 H 0 1.0 bash /opt/osg/data
> >
> > 3 jobs; 1 idle, 0 running, 2 held
> >
> > (My first job is the condor-submit job, the second is the job I
> > attempted to submit via swift.)
> >
> > Any thoughts?
> >
> > Arjun
> >
> >
> > On Wed, Jun 23, 2010 at 11:26 AM, Arjun Comar <
> > mandaya at rose-hulman.edu > wrote:
> >
> >
> > Hey all,
> > I've been trying to get jobs submitted over Condor via swift, and
> > running into a few problems. I think I've finally hit a point where
> > it's the Condor provider itself that's failing over any of my
> > configurations.
> > Here's the sites entry (though any sites entry over Condor will do,
> > and I've tried several):
> >
> > <config>
> > <pool handle=" Nebraska_red.unl.edu ">
> > <gridftp url="gsiftp:// red.unl.edu/ "/>
> > <execution provider="condor"/>
> > <profile key="jobType" namespace="globus">grid</profile>
> > <profile key="gridResource" namespace="globus">gt2
> > red.unl.edu/jobmanager-condor </profile>
> > <workdirectory>/opt/osg/data/engage/tmp/ red.unl.edu
> </workdirectory>
> > </pool>
> > </config>
> >
> > And any swift script at all fails, even a simple helloworld:
> >
> > type messagefile;
> > app (messagefile t) greeting () {
> > echo "Hello, world!" stdout=@filename(t);
> > }
> > messagefile outfile <"hello.txt">;
> > outfile = greeting();
> >
> > With the following error:
> > The following errors have occurred:
> > 1. Application "echo" failed (Cannot submit job: Could not submit
> job
> > (condor_submit reported an exit code of 1). Submitting job(s)
> > Found illegal unescaped double-quote: "" -e /bin/echo -out hello.txt
> > -err stderr.txt -i -d "" -if "" -of hello.txt -k "" -status
> "provider
> > " -a "Hello, world!"The full arguments you specified were:
> > /opt/osg/data/engage/tmp/
> > red.unl.edu/helloworld-20100623-1051-dr5v5apa/shared/_swiftwrap
> > echo-dqt6jttj -jobdir d -scratch "" -e /bin/echo -out hello.txt -err
> > stderr.txt -i -d "" -if "" -of hello.txt -k "" -status "provider "
> -a
> > "Hello, world!")
> >
> > The same script runs just fine through any other submission
> mechanism,
> > even to the same site (ssh, coasters+ssh:pbs, etc).
> >
> > Anyone have any thoughts on fixing the problem?
> >
> > Thanks!
> >
> > --
> > Arjun Comar, Rose-Hulman '12
> >
> >
> >
> > --
> > Arjun Comar, Rose-Hulman '12
> 
> --
> Michael Wilde
> Computation Institute, University of Chicago
> Mathematics and Computer Science Division
> Argonne National Laboratory
> 
> 
> 
> 
> --
> Arjun Comar, Rose-Hulman '12

-- 
Michael Wilde
Computation Institute, University of Chicago
Mathematics and Computer Science Division
Argonne National Laboratory



From aespinosa at cs.uchicago.edu  Thu Jun 24 12:57:08 2010
From: aespinosa at cs.uchicago.edu (Allan Espinosa)
Date: Thu, 24 Jun 2010 12:57:08 -0500
Subject: [Swift-user] Re: Error with condor provider
In-Reply-To: <22733478.1073521277401953725.JavaMail.root@zimbra>
References: <AANLkTimf8FsDgOV71gaLvJzELuaN8FS-a9K4J_tmk7W6@mail.gmail.com>
	<22733478.1073521277401953725.JavaMail.root@zimbra>
Message-ID: <AANLkTikwXh8w1MXFAZqv_o0iCRDCMxmdEUY43_uhCGYc@mail.gmail.com>

Hi Arjun,

I see that you are connecting to my condor instance on bridled.ci.  It
may not have access permissions for other users other than me as I
don't know if its configured to be a general condor service.  I proxy
access problems when I was using the condor service installed in a
default debian installation in my workstation.

-Allan

2010/6/24 Michael Wilde <wilde at mcs.anl.gov>:
>
> ----- "Arjun Comar" <mandaya at rose-hulman.edu> wrote:
>
>> I don't think it's just the host, because the swift job was submitted
>> to red.unl.edu , which I know is a valid host. And Justin suggested
>> running without that gridresource line, but then it just fails
>> outright. So I switched the gridresource to gt2
>> red.unl.edu/jobmanager-fork , and ran again, and this time I got an
>> actual reason for being held in the log file. Here's the exact line:
>> 012 (077.000.000) 06/24 11:21:05 Job was held.
>> Failed to get expiration time of proxy
>> Code 0 Subcode 0
>> ...
>>
>> I have a valid proxy, so I don't know what the problem is. Does this
>> indicate that I need to switch to voms-proxy over grid-proxy?
>
> You should test this step by step, layer by layer. Use whatever proxy type you would use when testing with the standard grid commands.
>
> Try a short simple job using the gt2 fork provider (jobmanager=fork in the sites entry I think it is).
>
> Are you testing standalone condor-g?
>
> Are you doing this from engage-submit (where there is a known-working OSG client stack)?
>
> - Mike
>
>
>>
>> Arjun
>>
>>
>> On Wed, Jun 23, 2010 at 5:40 PM, wilde at mcs.anl.gov < wilde at mcs.anl.gov
>> > wrote:
>>
>>
>>
>>
>> ----- "Arjun Comar" < mandaya at rose-hulman.edu > wrote:
>>
>> > Alright, so I just tried again with trunk over Mike's stable
>> > repository and it worked, so it looks like whatever the problem was
>> > got fixed between stable and trunk.
>>
>> Or uncommitted in my working dirs and needs testing and checkin :(
>>
>>
>> > However submitting jobs to condor
>> > results in the job simply getting held. Do I need any permissions
>> over
>> > grid-proxy-init? Same thing happens when I run condor_submit on the
>> > command line.
>>
>> There is a good little guide on debugging Condor problems:
>> http://www.cs.wisc.edu/condor/CondorWeek2004/presentations/effective_condorg_v4.doc
>> (likely superseded now; talks about what causes jobs to go on hold,
>> and how to look at log files)
>>
>> Things like bad paths or args can cause jobs to fail, and get held
>> and/or retried.
>>
>> Below, the host name gsu1.uchicago.edu may be some grid school host
>> that is down or non-existent???
>>
>> - Mike
>>
>>
>>
>>
>> >
>> > Condor submit file:
>> > executable=/bin/echo
>> > arguments=Hello World!
>> > output=results.output
>> > error=results.error
>> > log=results.log
>> > notification=never
>> > universe=grid
>> > grid_resource=gt2 gsu1.uchicago.edu/jobmanager-fork
>> > queue
>> >
>> > and I submitted with:
>> > condor_submit myjob.submit
>> >
>> > results in:
>> > -- Submitter: bridled.ci.uchicago.edu : < 128.135.125.18:49572 > :
>> > bridled.ci.uchicago.edu
>> > ID OWNER SUBMITTED RUN_TIME ST PRI SIZE CMD
>> > 57.0 aespinosa 6/15 11:54 0+00:00:00 I 0 7.3 condor_dagman
>> > 71.0 arjun 6/23 11:24 0+00:00:00 H 0 0.0 echo Hello World!
>> > 72.0 arjun 6/23 17:05 0+00:00:00 H 0 1.0 bash /opt/osg/data
>> >
>> > 3 jobs; 1 idle, 0 running, 2 held
>> >
>> > (My first job is the condor-submit job, the second is the job I
>> > attempted to submit via swift.)
>> >
>> > Any thoughts?
>> >
>> > Arjun
>> >
>> >
>> > On Wed, Jun 23, 2010 at 11:26 AM, Arjun Comar <
>> > mandaya at rose-hulman.edu > wrote:
>> >
>> >
>> > Hey all,
>> > I've been trying to get jobs submitted over Condor via swift, and
>> > running into a few problems. I think I've finally hit a point where
>> > it's the Condor provider itself that's failing over any of my
>> > configurations.
>> > Here's the sites entry (though any sites entry over Condor will do,
>> > and I've tried several):
>> >
>> > <config>
>> > <pool handle=" Nebraska_red.unl.edu ">
>> > <gridftp url="gsiftp:// red.unl.edu/ "/>
>> > <execution provider="condor"/>
>> > <profile key="jobType" namespace="globus">grid</profile>
>> > <profile key="gridResource" namespace="globus">gt2
>> > red.unl.edu/jobmanager-condor </profile>
>> > <workdirectory>/opt/osg/data/engage/tmp/ red.unl.edu
>> </workdirectory>
>> > </pool>
>> > </config>
>> >
>> > And any swift script at all fails, even a simple helloworld:
>> >
>> > type messagefile;
>> > app (messagefile t) greeting () {
>> > echo "Hello, world!" stdout=@filename(t);
>> > }
>> > messagefile outfile <"hello.txt">;
>> > outfile = greeting();
>> >
>> > With the following error:
>> > The following errors have occurred:
>> > 1. Application "echo" failed (Cannot submit job: Could not submit
>> job
>> > (condor_submit reported an exit code of 1). Submitting job(s)
>> > Found illegal unescaped double-quote: "" -e /bin/echo -out hello.txt
>> > -err stderr.txt -i -d "" -if "" -of hello.txt -k "" -status
>> "provider
>> > " -a "Hello, world!"The full arguments you specified were:
>> > /opt/osg/data/engage/tmp/
>> > red.unl.edu/helloworld-20100623-1051-dr5v5apa/shared/_swiftwrap
>> > echo-dqt6jttj -jobdir d -scratch "" -e /bin/echo -out hello.txt -err
>> > stderr.txt -i -d "" -if "" -of hello.txt -k "" -status "provider "
>> -a
>> > "Hello, world!")
>> >
>> > The same script runs just fine through any other submission
>> mechanism,
>> > even to the same site (ssh, coasters+ssh:pbs, etc).
>> >
>> > Anyone have any thoughts on fixing the problem?
>> >
>> > Thanks!
>> >
>> > --
>> > Arjun Comar, Rose-Hulman '12
>> >
>> >
>> >
>> > --
>> > Arjun Comar, Rose-Hulman '12
>>
>> --
>> Michael Wilde
>> Computation Institute, University of Chicago
>> Mathematics and Computer Science Division
>> Argonne National Laboratory
>>
>>
>


From wozniak at mcs.anl.gov  Thu Jun 24 13:22:34 2010
From: wozniak at mcs.anl.gov (Justin M Wozniak)
Date: Thu, 24 Jun 2010 13:22:34 -0500 (CDT)
Subject: [Swift-user] Formatting in Swift
In-Reply-To: <785165.61857.qm@web113708.mail.gq1.yahoo.com>
References: <785165.61857.qm@web113708.mail.gq1.yahoo.com>
Message-ID: <alpine.DEB.2.00.1006241319470.14889@wozniak-desktop.mcs.anl.gov>


If you are using trunk, you can try the new tracef() function:

        Formatted trace output.
        Example: tracef("\t%s\n", "hello");
        Differences from trace():
        1) respects \t, \n and \\;
        2) allows for typechecked format specifiers;
        3) allows for consumption of variables without display (%k);
        4) does not impose any formatting (commas, etc.).
        Format specifiers:
            %%: % sign.
            %p: Not typechecked, output as in trace().
            %i: Typechecked int output.
            %s: Typechecked string output.
            %k: Variable sKipped, no output.

On Thu, 24 Jun 2010, Lee S wrote:

> Hello,
>
> I'm trying to figure out how I can control formatting in Swift,
> For example, in c-scripting there is set myVar = `printf "%04d" {$Integer}`
> Is there a way to access these formatting options in swift?
>
> Thank you
>
>
>

-- 
Justin M Wozniak


From jadeddragon1111 at yahoo.com  Fri Jun 25 08:46:25 2010
From: jadeddragon1111 at yahoo.com (Lee S)
Date: Fri, 25 Jun 2010 06:46:25 -0700 (PDT)
Subject: [Swift-user] Continuation on formatting in swift
Message-ID: <269608.88703.qm@web113711.mail.gq1.yahoo.com>

Hello,

If I am trying to pass this ass an argument to echo
(to pass to readdata)
 
(file out)yr_prntApp(int year, string conv)
{
app{
        echo
@strcat(conv,"`printf%"04d"",year);
}}
 
How do I get swift to accept the 'printf%"04d" portion ? (i
think the % and/or "04d" are creating an error

Thank you


      
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-user/attachments/20100625/09da75ed/attachment.html>

From aespinosa at cs.uchicago.edu  Fri Jun 25 16:15:42 2010
From: aespinosa at cs.uchicago.edu (Allan Espinosa)
Date: Fri, 25 Jun 2010 16:15:42 -0500
Subject: [Swift-user] Continuation on formatting in swift
In-Reply-To: <269608.88703.qm@web113711.mail.gq1.yahoo.com>
References: <269608.88703.qm@web113711.mail.gq1.yahoo.com>
Message-ID: <AANLkTilgaAmky6d06uAKIqh6-Uim86n0oQGlmBuiFbBO@mail.gmail.com>

What about using printf(1) instead of echo(1)?

-Allan

2010/6/25 Lee S <jadeddragon1111 at yahoo.com>:
> Hello,
>
> If I am trying to pass this ass an argument to echo (to pass to readdata)
>
>
>
> (file out)yr_prntApp(int year, string conv)
>
> {
>
> app{
>
> ?? ? ? ?echo @strcat(conv,"`printf%"04d"",year);
>
> }}
>
>
>
> How do I get swift to accept the 'printf%"04d" portion ? (i think the %
> and/or "04d" are creating an error
>
> Thank you
>