From aespinosa at cs.uchicago.edu  Sun Feb  1 00:38:12 2009
From: aespinosa at cs.uchicago.edu (Allan Espinosa)
Date: Sun, 1 Feb 2009 00:38:12 -0600
Subject: [Swift-user] mixing gram execution and local filesystem providers
Message-ID: <50b07b4b0901312238k55330bb7gb3a0922fb60f1bb0@mail.gmail.com>

Hi,

I mixed the two providers since my files are at the remote site
already.  The data files were being staged successfully but the
wrapper scripts for submittiing the jobs are not being transferred
successfully.  I also got the same kind of error using coasters
(gt2:gt2:pbs)

I used swift 0.8 .

Below are the logs

sites.xml
<config>

  <pool handle="TERAPORT" sysinfo="INTEL32::LINUX">
    <profile namespace="globus" key="queue">fast</profile>
    <profile namespace="globus"
key="coasterWorkerMaxwalltime">01:00:00</profile>
    <gridftp url="local://localhost" />
    <jobmanager universe="vanilla"
url="tp-grid1.uchicago.edu/jobmanager-pbs" major="2" minor="2"
patch="4" />
    <workdirectory >/disks/tp-gpfs/scratch/aespinosa</workdirectory>
  </pool>

</config>


runtime message:
rm -rf *.rlog *.d *.log blast.xml blast.kml test.out test.err
~/swift/vdsk/bin/swift -sites.file sites.xml -tc.file tc.data blast.swift
Swift 0.8 swift-r2448 cog-r2261

RunID: 20090201-0030-k9cb65gc
Progress:
Progress:  uninitialized:1
Progress:  Selecting site:3 Stage in:2
Progress:  Selecting site:3 Stage in:2
Progress:  Selecting site:3 Submitting:1 Submitted:1
Failed to transfer wrapper log from
blast-20090201-0030-k9cb65gc/info/u on TERAPORT
Failed to transfer wrapper log from
blast-20090201-0030-k9cb65gc/info/v on TERAPORT
Progress:  Selecting site:1 Stage in:1 Submitting:1 Failed but can retry:2
Progress:  Selecting site:1 Submitting:1 Submitted:1 Failed but can retry:2
Failed to transfer wrapper log from
blast-20090201-0030-k9cb65gc/info/y on TERAPORT
Progress:  Stage in:1 Submitting:1 Failed but can retry:3
Failed to transfer wrapper log from
blast-20090201-0030-k9cb65gc/info/0 on TERAPORT
Progress:  Submitted:1 Failed but can retry:4
Failed to transfer wrapper log from
blast-20090201-0030-k9cb65gc/info/2 on TERAPORT
Progress:  Failed but can retry:5


log file
2009-02-01 00:31:40,129-0600 DEBUG TaskImpl Task(type=FILE_OPERATION,
identity=urn:0-6-5-1-1233469825381) setting status to Active
2009-02-01 00:31:40,130-0600 DEBUG TaskImpl Task(type=FILE_OPERATION,
identity=urn:0-6-5-1-1233469825381) setting status to Failed
org.globus.cog.abstraction.impl.file.FileNotFoundException:
blastall-2qgpuz5j-info not found.
2009-02-01 00:31:40,130-0600 WARN  vdl:transferwrapperlog Failed to
transfer wrapper log from blast-20090201-0030-k9cb65gc/info/2 on
TERAPORT
2009-02-01 00:31:40,131-0600 DEBUG vdl:transferwrapperlog Exception
for wrapper log failure from blast-20090201-0030-k9cb65gc/info/2 on
TERAPORT: File not found:
/disks/tp-gpfs/scratch/aespinosa/blast-20090201-0030-k9cb65gc/info/2/blastall-2qgpuz5j-info
	task:transfer @ vdl-int.k, line: 356
	sys:try @ vdl-int.k, line: 356
	vdl:transferwrapperlog @ vdl-int.k, line: 466
	sys:maybe @ vdl-int.k, line: 466
	sys:discard @ vdl-int.k, line: 466
	sys:catch @ vdl-int.k, line: 457
	sys:try @ vdl-int.k, line: 394
	task:allocatehost @ vdl-int.k, line: 370
	vdl:execute2 @ execute-default.k, line: 24
	sys:try @ execute-default.k, line: 23
	sys:parallelfor @ execute-default.k, line: 21
	sys:restartonerror @ execute-default.k, line: 17
	sys:sequential @ execute-default.k, line: 15
	sys:try @ execute-default.k, line: 14
	sys:if @ execute-default.k, line: 13
	sys:then @ execute-default.k, line: 12
	sys:if @ execute-default.k, line: 11
	vdl:execute @ blast.kml, line: 109
	blastall @ blast.kml, line: 296
	sys:sequential @ blast.kml, line: 295
	sys:parallelfor @ blast.kml, line: 287
	sys:sequential @ blast.kml, line: 286
	sys:parallel @ blast.kml, line: 224
	vdl:mainp @ blast.kml, line: 223
	mainp @ vdl.k, line: 165
	vdl:mains @ blast.kml, line: 221
	vdl:mains @ blast.kml, line: 221
	rlog:restartlog @ blast.kml, line: 219
	kernel:project @ blast.kml, line: 2
	blast-20090201-0030-k9cb65gc
Caused by: org.globus.cog.abstraction.impl.file.FileNotFoundException:
File not found:
/disks/tp-gpfs/scratch/aespinosa/blast-20090201-0030-k9cb65gc/info/2/blastall-2qgpuz5j-info
2009-02-01 00:31:40,139-0600 DEBUG WeightedHostScoreScheduler
Releasing contact 6
2009-02-01 00:31:40,139-0600 DEBUG WeightedHostScoreScheduler
commitDelayedScore(TERAPORT:-10.000(0.039):0/1 overload: -99533, 0.1
2009-02-01 00:30:22,617-0600 DEBUG Loader Loader started


From aespinosa at cs.uchicago.edu  Sun Feb  1 00:58:19 2009
From: aespinosa at cs.uchicago.edu (Allan Espinosa)
Date: Sun, 1 Feb 2009 00:58:19 -0600
Subject: [Swift-user] Re: mixing gram execution and local filesystem
	providers
In-Reply-To: <50b07b4b0901312238k55330bb7gb3a0922fb60f1bb0@mail.gmail.com>
References: <50b07b4b0901312238k55330bb7gb3a0922fb60f1bb0@mail.gmail.com>
Message-ID: <50b07b4b0901312258q7f5c9771pce3b05405de0a9c0@mail.gmail.com>

i tried running the job normally via gridftp and still got the same thing:

with coasters:
log: http://www.ci.uchicago.edu/~aespinosa/swift/blast-20090201-0047-hav48kr0.log

<config>

  <pool handle="TERAPORT" sysinfo="INTEL32::LINUX">
    <profile namespace="globus" key="queue">fast</profile>
    <profile namespace="globus"
key="coasterWorkerMaxwalltime">01:00:00</profile>
    <gridftp url="tp-grid1.uchicago.edu"
storage="/disks/tp-gpfs/scratch/aespinosa" major="2" minor="2"
patch="4" />
<!--
    <jobmanager universe="vanilla"
url="tp-grid1.uchicago.edu/jobmanager-pbs" major="2" minor="2"
patch="4" />
-->
    <execution provider="coaster" url="tp-grid1.uchicago.edu"
jobmanager="gt2:gt2:pbs" />
    <filesystem provider="coaster" url="gt2://tp-grid1.uchicago.edu" />
    <workdirectory >/disks/tp-gpfs/scratch/aespinosa</workdirectory>
  </pool>

</config>

default prewsgram:
http://www.ci.uchicago.edu/~aespinosa/swift/blast-20090201-0054-ky5miqr4.log

<config>

  <pool handle="TERAPORT" sysinfo="INTEL32::LINUX">
    <profile namespace="globus" key="queue">fast</profile>
    <gridftp url="tp-grid1.uchicago.edu"
storage="/disks/tp-gpfs/scratch/aespinosa" major="2" minor="2"
patch="4" />
    <workdirectory >/disks/tp-gpfs/scratch/aespinosa</workdirectory>
  </pool>

</config>


From benc at hawaga.org.uk  Sun Feb  1 04:56:00 2009
From: benc at hawaga.org.uk (Ben Clifford)
Date: Sun, 1 Feb 2009 10:56:00 +0000 (GMT)
Subject: [Swift-user] Re: mixing gram execution and local filesystem
	providers
In-Reply-To: <50b07b4b0901312258q7f5c9771pce3b05405de0a9c0@mail.gmail.com>
References: <50b07b4b0901312238k55330bb7gb3a0922fb60f1bb0@mail.gmail.com>
	<50b07b4b0901312258q7f5c9771pce3b05405de0a9c0@mail.gmail.com>
Message-ID: <Pine.LNX.4.64.0902011047320.8995@dildano.hawaga.org.uk>


On Sun, 1 Feb 2009, Allan Espinosa wrote:

> i tried running the job normally via gridftp and still got the same thing:

Did you intend to post a log on this line?

> with coasters:
> log: http://www.ci.uchicago.edu/~aespinosa/swift/blast-20090201-0047-hav48kr0.log

That is showing errors in file put.

wrt you previous message in this thread, the local filesystem provider is 
for when the Swift submit side has access to the same filesystem as the 
worker nodes. In your case, that seems to be untrue.

The local filesystem provider works by copying things around from the 
Swift command line client basically as if you had typed 'cp'. If you can't 
cp files into place from the same place that you are running Swift, then 
you can't use the local filesystem provider.

If you have files elsewhere, you can map them using a URI. For example, 
something like this:

  file myfile <"gsiftp://tp-login.ci.uchicago.edu/etc/group">;

(or likewise for other mappers).

Then if you use the gridftp provider, third party transfer will be used, 
so the data won't actually leave the cluster.

Exciting feature creep for the coaster filesystem provider might be 
remote site-local filesystem staging, to handle your case.

-- 


From benc at hawaga.org.uk  Sun Feb  1 17:01:41 2009
From: benc at hawaga.org.uk (Ben Clifford)
Date: Sun, 1 Feb 2009 23:01:41 +0000 (GMT)
Subject: [Swift-user] mixing gram execution and local filesystem providers
In-Reply-To: <50b07b4b0901312238k55330bb7gb3a0922fb60f1bb0@mail.gmail.com>
References: <50b07b4b0901312238k55330bb7gb3a0922fb60f1bb0@mail.gmail.com>
Message-ID: <Pine.LNX.4.64.0902012259450.8995@dildano.hawaga.org.uk>


On Sun, 1 Feb 2009, Allan Espinosa wrote:

> I mixed the two providers since my files are at the remote site
> already.  The data files were being staged successfully but the
> wrapper scripts for submittiing the jobs are not being transferred
> successfully.  I also got the same kind of error using coasters
> (gt2:gt2:pbs)

In addition to previous note, if you are running Swift from a CI machine 
then you should have a bunch of the same filesystems (such as teraport's 
GPFS) mounted already. In that case, using the local filesystem provider 
should work.

Please can you send a full log file for a run using local filesystem and 
plain gt2 execution?

-- 



From aespinosa at cs.uchicago.edu  Sun Feb  1 18:27:58 2009
From: aespinosa at cs.uchicago.edu (Allan Espinosa)
Date: Sun, 1 Feb 2009 18:27:58 -0600
Subject: [Swift-user] mixing gram execution and local filesystem providers
In-Reply-To: <Pine.LNX.4.64.0902012259450.8995@dildano.hawaga.org.uk>
References: <50b07b4b0901312238k55330bb7gb3a0922fb60f1bb0@mail.gmail.com>
	<Pine.LNX.4.64.0902012259450.8995@dildano.hawaga.org.uk>
Message-ID: <50b07b4b0902011627s14087822o7f0fa2ce931e37ae@mail.gmail.com>

Hi Ben,

Here are the logs:
GT2 execution gridftp filesystem: (successful)
http://www.ci.uchicago.edu/~aespinosa/swift/blast-20090201-1519-5xtsovjd.log
sites.xml:
<config>

  <pool handle="TERAPORT" sysinfo="INTEL32::LINUX">
    <profile namespace="globus" key="queue">fast</profile>
    <gridftp url="tp-grid1.uchicago.edu"
storage="/disks/tp-gpfs/scratch/aespinosa" major="2" minor="2"
patch="4" />
    <jobmanager universe="vanilla"
url="tp-grid1.uchicago.edu/jobmanager-pbs" major="2" minor="2"
patch="4" />
    <workdirectory >/disks/tp-gpfs/scratch/aespinosa</workdirectory>
  </pool>

</config>

GT2 execution and local filesystem: (successful)
logs:  http://www.ci.uchicago.edu/~aespinosa/swift/blast-20090201-1808-ua9hhvz6.log
sites.xml:
<config>

  <pool handle="TERAPORT" sysinfo="INTEL32::LINUX">
    <profile namespace="globus" key="queue">fast</profile>
    <gridftp url="local://localhost" />
    <jobmanager universe="vanilla"
url="tp-grid1.uchicago.edu/jobmanager-pbs" major="2" minor="2"
patch="4" />
    <workdirectory >/disks/tp-gpfs/scratch/aespinosa</workdirectory>
  </pool>

</config>


coaster local:pbs and local filesystem: (errorr: Failed to transfer
wrapper log from blast-20090201-1818-e6uz834b/info/z on TERAPORT) I
aborted the run after a while...
logs: http://www.ci.uchicago.edu/~aespinosa/swift/blast-20090201-1818-e6uz834b.log
sites.xml
<config>

  <pool handle="TERAPORT" sysinfo="INTEL32::LINUX">
    <profile namespace="globus" key="queue">fast</profile>
    <gridftp url="local://localhost" />
    <execution provider="coaster" url="none"  jobmanager="local:pbs" />
    <workdirectory >/disks/tp-gpfs/scratch/aespinosa</workdirectory>
  </pool>

</config>

On Sun, Feb 1, 2009 at 5:01 PM, Ben Clifford <benc at hawaga.org.uk> wrote:
>
> On Sun, 1 Feb 2009, Allan Espinosa wrote:
>
>> I mixed the two providers since my files are at the remote site
>> already.  The data files were being staged successfully but the
>> wrapper scripts for submittiing the jobs are not being transferred
>> successfully.  I also got the same kind of error using coasters
>> (gt2:gt2:pbs)
>
> In addition to previous note, if you are running Swift from a CI machine
> then you should have a bunch of the same filesystems (such as teraport's
> GPFS) mounted already. In that case, using the local filesystem provider
> should work.
>
> Please can you send a full log file for a run using local filesystem and
> plain gt2 execution?
>
> --
>
>
>



-- 
Allan M. Espinosa <http://allan.88-mph.net/blog>
PhD student, Computer Science
University of Chicago <http://people.cs.uchicago.edu/~aespinosa>


From benc at hawaga.org.uk  Sun Feb  1 18:33:25 2009
From: benc at hawaga.org.uk (Ben Clifford)
Date: Mon, 2 Feb 2009 00:33:25 +0000 (GMT)
Subject: [Swift-user] mixing gram execution and local filesystem providers
In-Reply-To: <50b07b4b0902011627s14087822o7f0fa2ce931e37ae@mail.gmail.com>
References: <50b07b4b0901312238k55330bb7gb3a0922fb60f1bb0@mail.gmail.com> 
	<Pine.LNX.4.64.0902012259450.8995@dildano.hawaga.org.uk>
	<50b07b4b0902011627s14087822o7f0fa2ce931e37ae@mail.gmail.com>
Message-ID: <Pine.LNX.4.64.0902020032290.14259@dildano.hawaga.org.uk>


On Sun, 1 Feb 2009, Allan Espinosa wrote:

> GT2 execution and local filesystem: (successful)

your initial message reported that this combination was unsuccessful.

-- 



From aespinosa at cs.uchicago.edu  Sun Feb  1 20:01:08 2009
From: aespinosa at cs.uchicago.edu (Allan Espinosa)
Date: Sun, 1 Feb 2009 20:01:08 -0600
Subject: [Swift-user] mixing gram execution and local filesystem providers
In-Reply-To: <Pine.LNX.4.64.0902020032290.14259@dildano.hawaga.org.uk>
References: <50b07b4b0901312238k55330bb7gb3a0922fb60f1bb0@mail.gmail.com>
	<Pine.LNX.4.64.0902012259450.8995@dildano.hawaga.org.uk>
	<50b07b4b0902011627s14087822o7f0fa2ce931e37ae@mail.gmail.com>
	<Pine.LNX.4.64.0902020032290.14259@dildano.hawaga.org.uk>
Message-ID: <50b07b4b0902011801g15ce4045u38f3a1ef3c81fec4@mail.gmail.com>

I did some retests using local for that one and found out something
was wrong in my site deployment (misplaced binaries of the
application).

sorry about that.

On Sun, Feb 1, 2009 at 6:33 PM, Ben Clifford <benc at hawaga.org.uk> wrote:
>
> On Sun, 1 Feb 2009, Allan Espinosa wrote:
>
>> GT2 execution and local filesystem: (successful)
>
> your initial message reported that this combination was unsuccessful.
>
> --


From benc at hawaga.org.uk  Mon Feb  2 03:35:02 2009
From: benc at hawaga.org.uk (Ben Clifford)
Date: Mon, 2 Feb 2009 09:35:02 +0000 (GMT)
Subject: [Swift-user] mixing gram execution and local filesystem providers
In-Reply-To: <50b07b4b0902011801g15ce4045u38f3a1ef3c81fec4@mail.gmail.com>
References: <50b07b4b0901312238k55330bb7gb3a0922fb60f1bb0@mail.gmail.com> 
	<Pine.LNX.4.64.0902012259450.8995@dildano.hawaga.org.uk> 
	<50b07b4b0902011627s14087822o7f0fa2ce931e37ae@mail.gmail.com> 
	<Pine.LNX.4.64.0902020032290.14259@dildano.hawaga.org.uk>
	<50b07b4b0902011801g15ce4045u38f3a1ef3c81fec4@mail.gmail.com>
Message-ID: <Pine.LNX.4.64.0902020931320.14259@dildano.hawaga.org.uk>


On Sun, 1 Feb 2009, Allan Espinosa wrote:

> I did some retests using local for that one and found out something
> was wrong in my site deployment (misplaced binaries of the
> application).
> 
> sorry about that.

Ok, so the actual subject line of this thread is not a problem any more? 
;)

As far as I can tell, the only combination which isn't working for you is 
coasters with local:pbs?

If that is so, and you are using 0.8, then its likely that you're 
encountering a walltime formatting problem which we discovered (as a 
result of stuff you reported last week). That was working as of about 12h 
ago. So if you want to use coasters and the pbs provider then you should 
have a source build with at least cog >= r2270.

-- 


From benc at hawaga.org.uk  Mon Feb  2 03:36:14 2009
From: benc at hawaga.org.uk (Ben Clifford)
Date: Mon, 2 Feb 2009 09:36:14 +0000 (GMT)
Subject: [Swift-user] mixing gram execution and local filesystem providers
In-Reply-To: <Pine.LNX.4.64.0902020931320.14259@dildano.hawaga.org.uk>
References: <50b07b4b0901312238k55330bb7gb3a0922fb60f1bb0@mail.gmail.com> 
	<Pine.LNX.4.64.0902012259450.8995@dildano.hawaga.org.uk> 
	<50b07b4b0902011627s14087822o7f0fa2ce931e37ae@mail.gmail.com> 
	<Pine.LNX.4.64.0902020032290.14259@dildano.hawaga.org.uk>
	<50b07b4b0902011801g15ce4045u38f3a1ef3c81fec4@mail.gmail.com>
	<Pine.LNX.4.64.0902020931320.14259@dildano.hawaga.org.uk>
Message-ID: <Pine.LNX.4.64.0902020936050.14259@dildano.hawaga.org.uk>

On Mon, 2 Feb 2009, Ben Clifford wrote:

> have a source build with at least cog >= r2270.

r2271 I mean.

-- 



From aespinosa at cs.uchicago.edu  Mon Feb  2 13:40:59 2009
From: aespinosa at cs.uchicago.edu (Allan Espinosa)
Date: Mon, 2 Feb 2009 13:40:59 -0600
Subject: [Swift-user] mixing gram execution and local filesystem providers
In-Reply-To: <Pine.LNX.4.64.0902020936050.14259@dildano.hawaga.org.uk>
References: <50b07b4b0901312238k55330bb7gb3a0922fb60f1bb0@mail.gmail.com>
	<Pine.LNX.4.64.0902012259450.8995@dildano.hawaga.org.uk>
	<50b07b4b0902011627s14087822o7f0fa2ce931e37ae@mail.gmail.com>
	<Pine.LNX.4.64.0902020032290.14259@dildano.hawaga.org.uk>
	<50b07b4b0902011801g15ce4045u38f3a1ef3c81fec4@mail.gmail.com>
	<Pine.LNX.4.64.0902020931320.14259@dildano.hawaga.org.uk>
	<Pine.LNX.4.64.0902020936050.14259@dildano.hawaga.org.uk>
Message-ID: <50b07b4b0902021140l64290db1u57ad652025c5182c@mail.gmail.com>

I see.

I got local:pbs working but I am still  having problems with gt2:gt2:pbs.

gt2:gt2:pbs log:
http://www.ci.uchicago.edu/~aespinosa/swift/blast-20090202-1335-pbur79zc.log
swift invocation:
Swift svn swift-r2490 cog-r2271

RunID: 20090202-1335-pbur79zc
Progress:
Progress:  uninitialized:1
Progress:  Selecting site:3 Stage in:2
Progress:  Selecting site:3 Stage in:2
Progress:  Selecting site:3 Stage in:1 Submitting:1
Progress:  Selecting site:3 Submitting:1 Submitted:1
Failed to transfer wrapper log from
blast-20090202-1335-pbur79zc/info/8 on TERAPORT
Progress:  Selecting site:2 Submitting:1 Submitted:1 Failed but can retry:1
Progress:  Selecting site:2 Submitted:1 Active:1 Failed but can retry:1
Progress:  Selecting site:2 Active:2 Failed but can retry:1
Progress:  Selecting site:2 Active:2 Failed but can retry:1
Cleaning up...
Shutting down service at https://128.135.125.118:53827
Got channel MetaChannel: 628444586 -> GSSSChannel-null(1)
- Done

sites.xml
<config>

  <pool handle="TERAPORT" sysinfo="INTEL32::LINUX">
    <profile namespace="globus" key="queue">fast</profile>
    <profile namespace="globus"
key="coasterWorkerMaxwalltime">00:30:00</profile>
    <gridftp url="local://localhost" />
    <execution provider="coaster" url="tp-grid1.uchicago.edu"
jobmanager="gt2:gt2:pbs" />
    <workdirectory >/disks/tp-gpfs/scratch/aespinosa</workdirectory>
  </pool>

</config>


On Mon, Feb 2, 2009 at 3:36 AM, Ben Clifford <benc at hawaga.org.uk> wrote:
> On Mon, 2 Feb 2009, Ben Clifford wrote:
>
>> have a source build with at least cog >= r2270.
>
> r2271 I mean.
>


From benc at hawaga.org.uk  Mon Feb  2 14:01:11 2009
From: benc at hawaga.org.uk (Ben Clifford)
Date: Mon, 2 Feb 2009 20:01:11 +0000 (GMT)
Subject: [Swift-user] mixing gram execution and local filesystem providers
In-Reply-To: <50b07b4b0902021140l64290db1u57ad652025c5182c@mail.gmail.com>
References: <50b07b4b0901312238k55330bb7gb3a0922fb60f1bb0@mail.gmail.com> 
	<Pine.LNX.4.64.0902012259450.8995@dildano.hawaga.org.uk> 
	<50b07b4b0902011627s14087822o7f0fa2ce931e37ae@mail.gmail.com> 
	<Pine.LNX.4.64.0902020032290.14259@dildano.hawaga.org.uk> 
	<50b07b4b0902011801g15ce4045u38f3a1ef3c81fec4@mail.gmail.com> 
	<Pine.LNX.4.64.0902020931320.14259@dildano.hawaga.org.uk> 
	<Pine.LNX.4.64.0902020936050.14259@dildano.hawaga.org.uk>
	<50b07b4b0902021140l64290db1u57ad652025c5182c@mail.gmail.com>
Message-ID: <Pine.LNX.4.64.0902021947030.14259@dildano.hawaga.org.uk>


embedded deep in that log file is:

> STDOUT: Could not find a valid java executable

Have a read over Mike (Wilde)'s emails over the past few days about 
running on teraport and getting Java paths working - there are some notes 
in there.

It looks like you're getting different paths when you submit through the 
pbs provider vs submitting through GRAM2 - with direct PBS, likely giving 
you Java and with GRAM, not giving you Java.

This is something that needs some user interface improvements on, so this 
will all be quite rough at the moment, but your experiences are pretty 
useful for helping iron this out.

It looks like that coaster error needs to find its way into console output 
rather than being deep in the log files.

The way in which Java is found needs some more configurability too...

-- 


From aespinosa at cs.uchicago.edu  Mon Feb  2 14:03:03 2009
From: aespinosa at cs.uchicago.edu (Allan Espinosa)
Date: Mon, 2 Feb 2009 14:03:03 -0600
Subject: [Swift-user] mixing gram execution and local filesystem providers
In-Reply-To: <50b07b4b0902021140l64290db1u57ad652025c5182c@mail.gmail.com>
References: <50b07b4b0901312238k55330bb7gb3a0922fb60f1bb0@mail.gmail.com>
	<Pine.LNX.4.64.0902012259450.8995@dildano.hawaga.org.uk>
	<50b07b4b0902011627s14087822o7f0fa2ce931e37ae@mail.gmail.com>
	<Pine.LNX.4.64.0902020032290.14259@dildano.hawaga.org.uk>
	<50b07b4b0902011801g15ce4045u38f3a1ef3c81fec4@mail.gmail.com>
	<Pine.LNX.4.64.0902020931320.14259@dildano.hawaga.org.uk>
	<Pine.LNX.4.64.0902020936050.14259@dildano.hawaga.org.uk>
	<50b07b4b0902021140l64290db1u57ad652025c5182c@mail.gmail.com>
Message-ID: <50b07b4b0902021203m45dccb32k8e0b94272e320842@mail.gmail.com>

I got the error below when running from a remote machine.  But when i
submit the gt2:gt2:pbs from the gt2 server. I was able to execute the
job successfully.

On Mon, Feb 2, 2009 at 1:40 PM, Allan Espinosa
<aespinosa at cs.uchicago.edu> wrote:
> I see.
>
> I got local:pbs working but I am still  having problems with gt2:gt2:pbs.
>
> gt2:gt2:pbs log:
> http://www.ci.uchicago.edu/~aespinosa/swift/blast-20090202-1335-pbur79zc.log
> swift invocation:
> Swift svn swift-r2490 cog-r2271
>
> RunID: 20090202-1335-pbur79zc
> Progress:
> Progress:  uninitialized:1
> Progress:  Selecting site:3 Stage in:2
> Progress:  Selecting site:3 Stage in:2
> Progress:  Selecting site:3 Stage in:1 Submitting:1
> Progress:  Selecting site:3 Submitting:1 Submitted:1
> Failed to transfer wrapper log from
> blast-20090202-1335-pbur79zc/info/8 on TERAPORT
> Progress:  Selecting site:2 Submitting:1 Submitted:1 Failed but can retry:1
> Progress:  Selecting site:2 Submitted:1 Active:1 Failed but can retry:1
> Progress:  Selecting site:2 Active:2 Failed but can retry:1
> Progress:  Selecting site:2 Active:2 Failed but can retry:1
> Cleaning up...
> Shutting down service at https://128.135.125.118:53827
> Got channel MetaChannel: 628444586 -> GSSSChannel-null(1)
> - Done
>
> sites.xml
> <config>
>
>  <pool handle="TERAPORT" sysinfo="INTEL32::LINUX">
>    <profile namespace="globus" key="queue">fast</profile>
>    <profile namespace="globus"
> key="coasterWorkerMaxwalltime">00:30:00</profile>
>    <gridftp url="local://localhost" />
>    <execution provider="coaster" url="tp-grid1.uchicago.edu"
> jobmanager="gt2:gt2:pbs" />
>    <workdirectory >/disks/tp-gpfs/scratch/aespinosa</workdirectory>
>  </pool>
>
> </config>
>
>
> On Mon, Feb 2, 2009 at 3:36 AM, Ben Clifford <benc at hawaga.org.uk> wrote:
>> On Mon, 2 Feb 2009, Ben Clifford wrote:
>>
>>> have a source build with at least cog >= r2270.
>>
>> r2271 I mean


From aespinosa at cs.uchicago.edu  Mon Feb  2 14:04:22 2009
From: aespinosa at cs.uchicago.edu (Allan Espinosa)
Date: Mon, 2 Feb 2009 14:04:22 -0600
Subject: [Swift-user] mixing gram execution and local filesystem providers
In-Reply-To: <Pine.LNX.4.64.0902021947030.14259@dildano.hawaga.org.uk>
References: <50b07b4b0901312238k55330bb7gb3a0922fb60f1bb0@mail.gmail.com>
	<Pine.LNX.4.64.0902012259450.8995@dildano.hawaga.org.uk>
	<50b07b4b0902011627s14087822o7f0fa2ce931e37ae@mail.gmail.com>
	<Pine.LNX.4.64.0902020032290.14259@dildano.hawaga.org.uk>
	<50b07b4b0902011801g15ce4045u38f3a1ef3c81fec4@mail.gmail.com>
	<Pine.LNX.4.64.0902020931320.14259@dildano.hawaga.org.uk>
	<Pine.LNX.4.64.0902020936050.14259@dildano.hawaga.org.uk>
	<50b07b4b0902021140l64290db1u57ad652025c5182c@mail.gmail.com>
	<Pine.LNX.4.64.0902021947030.14259@dildano.hawaga.org.uk>
Message-ID: <50b07b4b0902021204j20e853cek14ef0134ee5610f7@mail.gmail.com>

I see.  so when i submit from teraport, the Java environment seems to
be detected smoothly over coasters.

Thanks Ben!
-Allan

On Mon, Feb 2, 2009 at 2:01 PM, Ben Clifford <benc at hawaga.org.uk> wrote:
>
> embedded deep in that log file is:
>
>> STDOUT: Could not find a valid java executable
>
> Have a read over Mike (Wilde)'s emails over the past few days about
> running on teraport and getting Java paths working - there are some notes
> in there.
>
> It looks like you're getting different paths when you submit through the
> pbs provider vs submitting through GRAM2 - with direct PBS, likely giving
> you Java and with GRAM, not giving you Java.
>
> This is something that needs some user interface improvements on, so this
> will all be quite rough at the moment, but your experiences are pretty
> useful for helping iron this out.
>
> It looks like that coaster error needs to find its way into console output
> rather than being deep in the log files.
>
> The way in which Java is found needs some more configurability too...


From benc at hawaga.org.uk  Mon Feb  2 14:06:40 2009
From: benc at hawaga.org.uk (Ben Clifford)
Date: Mon, 2 Feb 2009 20:06:40 +0000 (GMT)
Subject: [Swift-user] mixing gram execution and local filesystem providers
In-Reply-To: <50b07b4b0902021203m45dccb32k8e0b94272e320842@mail.gmail.com>
References: <50b07b4b0901312238k55330bb7gb3a0922fb60f1bb0@mail.gmail.com> 
	<Pine.LNX.4.64.0902012259450.8995@dildano.hawaga.org.uk> 
	<50b07b4b0902011627s14087822o7f0fa2ce931e37ae@mail.gmail.com> 
	<Pine.LNX.4.64.0902020032290.14259@dildano.hawaga.org.uk> 
	<50b07b4b0902011801g15ce4045u38f3a1ef3c81fec4@mail.gmail.com> 
	<Pine.LNX.4.64.0902020931320.14259@dildano.hawaga.org.uk> 
	<Pine.LNX.4.64.0902020936050.14259@dildano.hawaga.org.uk> 
	<50b07b4b0902021140l64290db1u57ad652025c5182c@mail.gmail.com>
	<50b07b4b0902021203m45dccb32k8e0b94272e320842@mail.gmail.com>
Message-ID: <Pine.LNX.4.64.0902022005350.14259@dildano.hawaga.org.uk>


On Mon, 2 Feb 2009, Allan Espinosa wrote:

> I got the error below when running from a remote machine.  But when i
> submit the gt2:gt2:pbs from the gt2 server. I was able to execute the
> job successfully.

Using the same configuration on both the 'remote machine' and 'the gt2 
server'?

My understanding is that wherever you submit from, you should get the same 
error no matter where you submit from; so I'd like to investigate that.

-- 


From aespinosa at cs.uchicago.edu  Mon Feb  2 14:11:54 2009
From: aespinosa at cs.uchicago.edu (Allan Espinosa)
Date: Mon, 2 Feb 2009 14:11:54 -0600
Subject: [Swift-user] mixing gram execution and local filesystem providers
In-Reply-To: <Pine.LNX.4.64.0902022005350.14259@dildano.hawaga.org.uk>
References: <50b07b4b0901312238k55330bb7gb3a0922fb60f1bb0@mail.gmail.com>
	<Pine.LNX.4.64.0902012259450.8995@dildano.hawaga.org.uk>
	<50b07b4b0902011627s14087822o7f0fa2ce931e37ae@mail.gmail.com>
	<Pine.LNX.4.64.0902020032290.14259@dildano.hawaga.org.uk>
	<50b07b4b0902011801g15ce4045u38f3a1ef3c81fec4@mail.gmail.com>
	<Pine.LNX.4.64.0902020931320.14259@dildano.hawaga.org.uk>
	<Pine.LNX.4.64.0902020936050.14259@dildano.hawaga.org.uk>
	<50b07b4b0902021140l64290db1u57ad652025c5182c@mail.gmail.com>
	<50b07b4b0902021203m45dccb32k8e0b94272e320842@mail.gmail.com>
	<Pine.LNX.4.64.0902022005350.14259@dildano.hawaga.org.uk>
Message-ID: <50b07b4b0902021211j6d11ea15t32d923464da55b1@mail.gmail.com>

yes its the same sites.xml file.  Here is the log of the successful gt2:gt2:pbs
http://www.ci.uchicago.edu/~aespinosa/swift/blast-20090202-1357-98hzn0n1.log

On Mon, Feb 2, 2009 at 2:06 PM, Ben Clifford <benc at hawaga.org.uk> wrote:
>
> On Mon, 2 Feb 2009, Allan Espinosa wrote:
>
>> I got the error below when running from a remote machine.  But when i
>> submit the gt2:gt2:pbs from the gt2 server. I was able to execute the
>> job successfully.
>
> Using the same configuration on both the 'remote machine' and 'the gt2
> server'?
>
> My understanding is that wherever you submit from, you should get the same
> error no matter where you submit from; so I'd like to investigate that.
>


From benc at hawaga.org.uk  Mon Feb  2 14:48:26 2009
From: benc at hawaga.org.uk (Ben Clifford)
Date: Mon, 2 Feb 2009 20:48:26 +0000 (GMT)
Subject: [Swift-user] mixing gram execution and local filesystem providers
In-Reply-To: <50b07b4b0902021211j6d11ea15t32d923464da55b1@mail.gmail.com>
References: <50b07b4b0901312238k55330bb7gb3a0922fb60f1bb0@mail.gmail.com> 
	<Pine.LNX.4.64.0902012259450.8995@dildano.hawaga.org.uk> 
	<50b07b4b0902011627s14087822o7f0fa2ce931e37ae@mail.gmail.com> 
	<Pine.LNX.4.64.0902020032290.14259@dildano.hawaga.org.uk> 
	<50b07b4b0902011801g15ce4045u38f3a1ef3c81fec4@mail.gmail.com> 
	<Pine.LNX.4.64.0902020931320.14259@dildano.hawaga.org.uk> 
	<Pine.LNX.4.64.0902020936050.14259@dildano.hawaga.org.uk> 
	<50b07b4b0902021140l64290db1u57ad652025c5182c@mail.gmail.com> 
	<50b07b4b0902021203m45dccb32k8e0b94272e320842@mail.gmail.com> 
	<Pine.LNX.4.64.0902022005350.14259@dildano.hawaga.org.uk>
	<50b07b4b0902021211j6d11ea15t32d923464da55b1@mail.gmail.com>
Message-ID: <Pine.LNX.4.64.0902022047540.14259@dildano.hawaga.org.uk>


On Mon, 2 Feb 2009, Allan Espinosa wrote:

> yes its the same sites.xml file.  Here is the log of the successful gt2:gt2:pbs
> http://www.ci.uchicago.edu/~aespinosa/swift/blast-20090202-1357-98hzn0n1.log

ok. So that is submitted using the same as:

http://www.ci.uchicago.edu/~aespinosa/swift/blast-20090202-1335-pbur79zc.log    

with the only difference that you are aware of being that they are 
submitted from different machines?

-- 



From aespinosa at cs.uchicago.edu  Mon Feb  2 14:52:37 2009
From: aespinosa at cs.uchicago.edu (Allan Espinosa)
Date: Mon, 2 Feb 2009 14:52:37 -0600
Subject: [Swift-user] mixing gram execution and local filesystem providers
In-Reply-To: <Pine.LNX.4.64.0902022047540.14259@dildano.hawaga.org.uk>
References: <50b07b4b0901312238k55330bb7gb3a0922fb60f1bb0@mail.gmail.com>
	<Pine.LNX.4.64.0902020032290.14259@dildano.hawaga.org.uk>
	<50b07b4b0902011801g15ce4045u38f3a1ef3c81fec4@mail.gmail.com>
	<Pine.LNX.4.64.0902020931320.14259@dildano.hawaga.org.uk>
	<Pine.LNX.4.64.0902020936050.14259@dildano.hawaga.org.uk>
	<50b07b4b0902021140l64290db1u57ad652025c5182c@mail.gmail.com>
	<50b07b4b0902021203m45dccb32k8e0b94272e320842@mail.gmail.com>
	<Pine.LNX.4.64.0902022005350.14259@dildano.hawaga.org.uk>
	<50b07b4b0902021211j6d11ea15t32d923464da55b1@mail.gmail.com>
	<Pine.LNX.4.64.0902022047540.14259@dildano.hawaga.org.uk>
Message-ID: <50b07b4b0902021252y71db65dfxf4317a323e4b5540@mail.gmail.com>

Yes they are the same.  the succesful one was submitted from tp-login1
(where tp-grid1 is) and the otherone from communicado.

-Allan

On Mon, Feb 2, 2009 at 2:48 PM, Ben Clifford <benc at hawaga.org.uk> wrote:
>
> On Mon, 2 Feb 2009, Allan Espinosa wrote:
>
>> yes its the same sites.xml file.  Here is the log of the successful gt2:gt2:pbs
>> http://www.ci.uchicago.edu/~aespinosa/swift/blast-20090202-1357-98hzn0n1.log
>
> ok. So that is submitted using the same as:
>
> http://www.ci.uchicago.edu/~aespinosa/swift/blast-20090202-1335-pbur79zc.log
>
> with the only difference that you are aware of being that they are
> submitted from different machines?
>


From wilde at mcs.anl.gov  Tue Feb  3 09:20:22 2009
From: wilde at mcs.anl.gov (Michael Wilde)
Date: Tue, 03 Feb 2009 09:20:22 -0600
Subject: [Swift-user] mixing gram execution and local filesystem providers
In-Reply-To: <50b07b4b0902021252y71db65dfxf4317a323e4b5540@mail.gmail.com>
References: <50b07b4b0901312238k55330bb7gb3a0922fb60f1bb0@mail.gmail.com>	<Pine.LNX.4.64.0902020032290.14259@dildano.hawaga.org.uk>	<50b07b4b0902011801g15ce4045u38f3a1ef3c81fec4@mail.gmail.com>	<Pine.LNX.4.64.0902020931320.14259@dildano.hawaga.org.uk>	<Pine.LNX.4.64.0902020936050.14259@dildano.hawaga.org.uk>	<50b07b4b0902021140l64290db1u57ad652025c5182c@mail.gmail.com>	<50b07b4b0902021203m45dccb32k8e0b94272e320842@mail.gmail.com>	<Pine.LNX.4.64.0902022005350.14259@dildano.hawaga.org.uk>	<50b07b4b0902021211j6d11ea15t32d923464da55b1@mail.gmail.com>	<Pine.LNX.4.64.0902022047540.14259@dildano.hawaga.org.uk>
	<50b07b4b0902021252y71db65dfxf4317a323e4b5540@mail.gmail.com>
Message-ID: <498860B6.7010307@mcs.anl.gov>

Were they submitted from the same cert, with the same cert-to-uid 
mapping in place?

Its possible that the OSG jobmanager initialization code differentiates 
between OSG VO users and users mapped to their own ID.

Try running globus-job-run tp-grid1.ci.uchicago.edu /bin/bash -l -c 
"which java", and the same without "-l", from both submission hosts.

- Mike


On 2/2/09 2:52 PM, Allan Espinosa wrote:
> Yes they are the same.  the succesful one was submitted from tp-login1
> (where tp-grid1 is) and the otherone from communicado.
> 
> -Allan
> 
> On Mon, Feb 2, 2009 at 2:48 PM, Ben Clifford <benc at hawaga.org.uk> wrote:
>> On Mon, 2 Feb 2009, Allan Espinosa wrote:
>>
>>> yes its the same sites.xml file.  Here is the log of the successful gt2:gt2:pbs
>>> http://www.ci.uchicago.edu/~aespinosa/swift/blast-20090202-1357-98hzn0n1.log
>> ok. So that is submitted using the same as:
>>
>> http://www.ci.uchicago.edu/~aespinosa/swift/blast-20090202-1335-pbur79zc.log
>>
>> with the only difference that you are aware of being that they are
>> submitted from different machines?
>>
> _______________________________________________
> Swift-user mailing list
> Swift-user at ci.uchicago.edu
> http://mail.ci.uchicago.edu/mailman/listinfo/swift-user


From wilde at mcs.anl.gov  Tue Feb  3 09:55:07 2009
From: wilde at mcs.anl.gov (Michael Wilde)
Date: Tue, 03 Feb 2009 09:55:07 -0600
Subject: [Swift-user] Expanding arrays in app function command lines
In-Reply-To: <Pine.LNX.4.64.0901271608040.14259@dildano.hawaga.org.uk>
References: <497E02DB.8040904@mcs.anl.gov>
	<Pine.LNX.4.64.0901261839360.14996@dildano.hawaga.org.uk>
	<Pine.LNX.4.64.0901271608040.14259@dildano.hawaga.org.uk>
Message-ID: <498868DB.1090801@mcs.anl.gov>

A few comments on this topic related to the vote Ben called:

Using the simple array or structure name on the app command line is good.

Having the members expanded in index and/or source order is good.

Expanding the members as separate args seems best, as that bypasses 
issues of deciding on separators and whitespace between members. Users 
who want to pass several members as one can apply some sort of 
concatenation and determine their own separators.

So I'm in favor of the approach used in this patch.

I'll cast my vote +1 as soon as I can apply the patch and test, to make 
sure Im not missing anything. I need to change my scripts to handle the 
new convention.


On 1/27/09 10:11 AM, Ben Clifford wrote:
> On Mon, 26 Jan 2009, Ben Clifford wrote:
> 
>>> But in general, the ability to expand an array of strings into the command
>>> line is useful and I think should be done in array member order.
>>> Can/should this be changed to behave like that?
>> yes, this is something I've been meaning to work on.
> 
> http://www.ci.uchicago.edu/~benc/tmp/argstar.patch
> 
> will allow you to specify arrays of strings directly as parameters.
> 
> http://www.ci.uchicago.edu/~benc/tmp/strarray.swift demonstrates the use 
> of such.
> 
> You can apply that, or you can wait a few days more me to test it more 
> thoroughly and commit.
> 
> Note that instead of saying x[*] you would say x without further 
> modifiers.
> 


From benc at hawaga.org.uk  Tue Feb  3 11:37:24 2009
From: benc at hawaga.org.uk (Ben Clifford)
Date: Tue, 3 Feb 2009 17:37:24 +0000 (GMT)
Subject: [Swift-user] Expanding arrays in app function command lines
In-Reply-To: <498868DB.1090801@mcs.anl.gov>
References: <497E02DB.8040904@mcs.anl.gov>
	<Pine.LNX.4.64.0901261839360.14996@dildano.hawaga.org.uk>
	<Pine.LNX.4.64.0901271608040.14259@dildano.hawaga.org.uk>
	<498868DB.1090801@mcs.anl.gov>
Message-ID: <Pine.LNX.4.64.0902031736310.14259@dildano.hawaga.org.uk>


On Tue, 3 Feb 2009, Michael Wilde wrote:

> I'll cast my vote +1 as soon as I can apply the patch and test, to make sure
> Im not missing anything. I need to change my scripts to handle the new
> convention.

There is a tidier version here:

http://www.ci.uchicago.edu/~benc/tmp/vdlarg-1.patch

It has the same changes to app syntax as my previous patch.

-- 


From aespinosa at cs.uchicago.edu  Tue Feb  3 12:20:46 2009
From: aespinosa at cs.uchicago.edu (Allan Espinosa)
Date: Tue, 3 Feb 2009 12:20:46 -0600
Subject: [Swift-user] mixing gram execution and local filesystem providers
In-Reply-To: <498860B6.7010307@mcs.anl.gov>
References: <50b07b4b0901312238k55330bb7gb3a0922fb60f1bb0@mail.gmail.com>
	<Pine.LNX.4.64.0902020931320.14259@dildano.hawaga.org.uk>
	<Pine.LNX.4.64.0902020936050.14259@dildano.hawaga.org.uk>
	<50b07b4b0902021140l64290db1u57ad652025c5182c@mail.gmail.com>
	<50b07b4b0902021203m45dccb32k8e0b94272e320842@mail.gmail.com>
	<Pine.LNX.4.64.0902022005350.14259@dildano.hawaga.org.uk>
	<50b07b4b0902021211j6d11ea15t32d923464da55b1@mail.gmail.com>
	<Pine.LNX.4.64.0902022047540.14259@dildano.hawaga.org.uk>
	<50b07b4b0902021252y71db65dfxf4317a323e4b5540@mail.gmail.com>
	<498860B6.7010307@mcs.anl.gov>
Message-ID: <50b07b4b0902031020r125169d2t61e828e4bea07da8@mail.gmail.com>

Hi mike
yes it gives a different java environment (but the same for both
submission hosts communicado and tp-login1:
globus-job-run tp-grid1.ci.uchicago.edu /bin/bash -l -c "which java"
SoftEnv 1.6.0: updating your software environment, one moment...
/autonfs/software/linux-rhel4-x86_64/osg-client-1.0.0-r1/jdk1.5/bin/java

vs

 globus-job-run tp-grid1.uchicago.edu /usr/bin/which java
/opt/osg-ce-0.8.0-r1/jdk1.5/bin/java


Update:  I fixed my ~/.soft file and observed that my coaster runs
(gt2:gt2:pbs local filesystem provider)  now work .

-Allan

On Tue, Feb 3, 2009 at 9:20 AM, Michael Wilde <wilde at mcs.anl.gov> wrote:
> Were they submitted from the same cert, with the same cert-to-uid mapping in
> place?
>
> Its possible that the OSG jobmanager initialization code differentiates
> between OSG VO users and users mapped to their own ID.
>
> Try running globus-job-run tp-grid1.ci.uchicago.edu /bin/bash -l -c "which
> java", and the same without "-l", from both submission hosts.
>
> - Mike
>
>
> On 2/2/09 2:52 PM, Allan Espinosa wrote:
>>
>> Yes they are the same.  the succesful one was submitted from tp-login1
>> (where tp-grid1 is) and the otherone from communicado.
>>
>> -Allan
>>
>> On Mon, Feb 2, 2009 at 2:48 PM, Ben Clifford <benc at hawaga.org.uk> wrote:
>>>
>>> On Mon, 2 Feb 2009, Allan Espinosa wrote:
>>>
>>>> yes its the same sites.xml file.  Here is the log of the successful
>>>> gt2:gt2:pbs
>>>>
>>>> http://www.ci.uchicago.edu/~aespinosa/swift/blast-20090202-1357-98hzn0n1.log
>>>
>>> ok. So that is submitted using the same as:
>>>
>>>
>>> http://www.ci.uchicago.edu/~aespinosa/swift/blast-20090202-1335-pbur79zc.log
>>>
>>> with the only difference that you are aware of being that they are
>>> submitted from different machines?
>>>


From aespinosa at cs.uchicago.edu  Tue Feb  3 15:43:24 2009
From: aespinosa at cs.uchicago.edu (Allan Espinosa)
Date: Tue, 3 Feb 2009 15:43:24 -0600
Subject: [Swift-user] gsiftp filesystem mapper removes leading slash
Message-ID: <50b07b4b0902031343g1c800451j78b0b914c944a416@mail.gmail.com>

Hi,

I am trying to stage input data from a remote site to itself using
this workflow:
type database;
type query;
type output;
type error;

(output out, error err) blastall(query i, string pref, database db[]) {
  app {
    blastall "-p" "blastp" "-F" "F" "-d" pref "-i" @filename(i) "-v"
"300" "-b" "300" "-m8" "-o" @filename(out) stderr=@filename(err);
  }
}

string pirpref = "/work/01035/tg802895/pir/UNIPROT_for_blast_14.0.seq";
database pir[] <filesys_mapper;location="gsiftp://gridftp.ranger.tacc.teragrid.org/work/01035/tg802895/pir",
pattern="UNIPROT_for_blast_14.0.seq*">;
output out[] <ext;exec="mapper-out.sh">;
error err[] <ext;exec="mapper-err.sh">;
query input[] <ext;exec="mapper-in.sh">;

foreach data ,i in input {
  (out[i],err[i]) = blastall(data, pirpref, pir);
}


But gridftp internals in swift reports that it cannot get a directory
listing of "work/01035/tg802895/pir".  I was thinking that maybe the
leading slashes were removed.  Below is the program output:
~/swift/vdsk/bin/swift -config swift.properties -sites.file sites.xml
-tc.file tc.data blast.swift
Swift svn swift-r2490 cog-r2271

RunID: 20090203-1540-f00ydqsf
Progress:
Execution failed:
        java.lang.RuntimeException:
org.globus.cog.abstraction.impl.file.FileResourceException: Could not
get list of files in work/01035/tg802895/pir from server
Caused by:
        Server refused performing the request. Custom message:  (error
code 1) [Nested exception message:  Custom message: Unexpected reply:
500-Command failed. : System error in stat: No such file or directory
500-A system call failed: No such file or directory
500-
500 End.]
make: *** [remote] Error 2

but i was able to make a regular gridftp session:
 uberftp gridftp.ranger.tacc.teragrid.org
220 gridftp2.ranger.tacc.utexas.edu GridFTP Server 2.8 (gcc64,
1217607445-63) [Globus Toolkit 4.0.8] ready.
230 User tg802895 logged in.
uberftp> cd /work/01035/tg802895/pir
uberftp> ls
-rw-------  1  tg802895  G-81621  204106872 Nov 12 21:44
UNIPROT_for_blast_14.0.seq.00.phr
-rw-------  1  tg802895  G-81621   23001752 Nov 12 21:45
UNIPROT_for_blast_14.0.seq.00.pin
-rw-------  1  tg802895  G-81621  999999669 Nov 12 21:45
UNIPROT_for_blast_14.0.seq.00.psq
...

logfile: http://www.ci.uchicago.edu/~aespinosa/swift/blast-20090203-1540-f00ydqsf.log
sites.xml:
<config>

  <pool handle="RANGER">
    <profile namespace="karajan" key="initialScore">2</profile>
     <profile namespace="karajan" key="jobThrottle">1</profile>
     <filesystem provider="coaster"
url="gt2://gatekeeper.ranger.tacc.teragrid.org"/>
     <profile namespace="globus" key="coastersPerNode">16</profile>
     <execution provider="coaster"
url="gatekeeper.ranger.tacc.teragrid.org" jobManager="gt2:gt2:SGE"/>
     <workdirectory>/work/01035/tg802895</workdirectory>
  </pool>
</config>


From benc at hawaga.org.uk  Tue Feb  3 15:52:14 2009
From: benc at hawaga.org.uk (Ben Clifford)
Date: Tue, 3 Feb 2009 21:52:14 +0000 (GMT)
Subject: [Swift-user] gsiftp filesystem mapper removes leading slash
In-Reply-To: <50b07b4b0902031343g1c800451j78b0b914c944a416@mail.gmail.com>
References: <50b07b4b0902031343g1c800451j78b0b914c944a416@mail.gmail.com>
Message-ID: <Pine.LNX.4.64.0902032147310.8995@dildano.hawaga.org.uk>


On Tue, 3 Feb 2009, Allan Espinosa wrote:

> location="gsiftp://gridftp.ranger.tacc.teragrid.org/work/01035/tg802895/pir",

> But gridftp internals in swift reports that it cannot get a directory 
> listing of "work/01035/tg802895/pir".  I was thinking that maybe the 
> leading slashes were removed.  Below is the program output:

CoG's gsiftp handling has an unconventional (as in different to the Globus 
Toolkit) understanding of gsiftp URLs; you need to use two slashes after 
the hostname in gsiftp URLs that are given to CoG.

Globus Toolkit gsiftp URLs implicitly start at /

CoG URLs start in pwd, and need an extra / to get to root.

Both interpretations violate my understand of the ftp:// URI scheme on 
which I understand the gsiftp:// scheme to be modelled.

(to be even more picky, its a violation to use gsiftp:// as a URI scheme 
as it neither begins with x- nor is registered with IANA... ho ho ho)

-- 



From aespinosa at cs.uchicago.edu  Tue Feb  3 16:12:27 2009
From: aespinosa at cs.uchicago.edu (Allan Espinosa)
Date: Tue, 3 Feb 2009 16:12:27 -0600
Subject: [Swift-user] gsiftp filesystem mapper removes leading slash
In-Reply-To: <Pine.LNX.4.64.0902032147310.8995@dildano.hawaga.org.uk>
References: <50b07b4b0902031343g1c800451j78b0b914c944a416@mail.gmail.com>
	<Pine.LNX.4.64.0902032147310.8995@dildano.hawaga.org.uk>
Message-ID: <50b07b4b0902031412t5f05bda2t4ff88326df678fb7@mail.gmail.com>

Oh i see.  now I'm getting NullPointExceptions:
database pir[] <filesys_mapper;location="gsiftp://gridftp.ranger.tacc.teragrid.org//work/01035/tg802895/pir",
pattern="UNIPROT_for_blast_14.0.seq*">;

$
Swift svn swift-r2490 cog-r2271

RunID: 20090203-1610-usa9ur25
Progress:
Execution failed:
        java.lang.RuntimeException: java.lang.NullPointerException
Caused by:
        java.lang.NullPointerException
        at org.griphyn.vdl.mapping.AbsFile.initDirAndName(AbsFile.java:60)
        at org.griphyn.vdl.mapping.AbsFile.<init>(AbsFile.java:77)
        at org.griphyn.vdl.mapping.AbsFile.listFiles(AbsFile.java:124)
        at org.griphyn.vdl.mapping.file.AbstractFileMapper.existing(AbstractFileMapper.java:166)
        at org.griphyn.vdl.mapping.RootDataNode.checkInputs(RootDataNode.java:96)
        at org.griphyn.vdl.mapping.RootArrayDataNode.checkInputs(RootArrayDataNode.java:64)
        at org.griphyn.vdl.mapping.RootArrayDataNode.innerInit(RootArrayDataNode.java:52)
        at org.griphyn.vdl.mapping.RootArrayDataNode.handleClosed(RootArrayDataNode.java:77)
        at org.griphyn.vdl.mapping.AbstractDataNode.notifyListeners(AbstractDataNode.java:486)
        at org.griphyn.vdl.mapping.AbstractDataNode.closeShallow(AbstractDataNode.java:364)
        at org.griphyn.vdl.mapping.AbstractDataNode.setValue(AbstractDataNode.java:322)
        at org.griphyn.vdl.mapping.RootDataNode.setValue(RootDataNode.java:214)
        at org.griphyn.vdl.karajan.lib.SetFieldValue.deepCopy(SetFieldValue.java:64)
        at org.griphyn.vdl.karajan.lib.SetFieldValue.function(SetFieldValue.java:47)
        at org.griphyn.vdl.karajan.lib.VDLFunction.post(VDLFunction.java:67)
        at org.globus.cog.karajan.workflow.nodes.AbstractSequentialWithArguments.childCompleted(AbstractSequentialWithArguments.java:192)
        at org.globus.cog.karajan.workflow.nodes.Sequential.notificationEvent(Sequential.java:33)
        at org.globus.cog.karajan.workflow.nodes.FlowNode.event(FlowNode.java:335)
        at org.globus.cog.karajan.workflow.events.EventBus.send(EventBus.java:125)
        at org.globus.cog.karajan.workflow.events.EventBus.sendHooked(EventBus.java:99)
        at org.globus.cog.karajan.workflow.nodes.FlowNode.fireNotificationEvent(FlowNode.java:173)
        at org.globus.cog.karajan.workflow.nodes.FlowNode.complete(FlowNode.java:299)
        at org.globus.cog.karajan.workflow.nodes.FlowContainer.post(FlowContainer.java:58)
        at org.globus.cog.karajan.workflow.nodes.functions.Argument.post(Argument.java:45)
        at org.globus.cog.karajan.workflow.nodes.AbstractSequentialWithArguments.childCompleted(AbstractSequentialWithArguments.java:192)
        at org.globus.cog.karajan.workflow.nodes.Sequential.notificationEvent(Sequential.java:33)
        at org.globus.cog.karajan.workflow.nodes.FlowNode.event(FlowNode.java:335)
        at org.globus.cog.karajan.workflow.events.EventBus.send(EventBus.java:125)
        at org.globus.cog.karajan.workflow.events.EventBus.sendHooked(EventBus.java:99)
        at org.globus.cog.karajan.workflow.nodes.FlowNode.fireNotificationEvent(FlowNode.java:173)
        at org.globus.cog.karajan.workflow.nodes.FlowNode.complete(FlowNode.java:299)
        at org.globus.cog.karajan.workflow.nodes.FlowContainer.post(FlowContainer.java:58)
        at org.globus.cog.karajan.workflow.nodes.functions.AbstractFunction.post(AbstractFunction.java:46)
        at org.globus.cog.karajan.workflow.nodes.Sequential.startNext(Sequential.java:51)
        at org.globus.cog.karajan.workflow.nodes.Sequential.executeChildren(Sequential.java:27)
        at org.globus.cog.karajan.workflow.nodes.functions.AbstractFunction.executeChildren(AbstractFunction.java:40)
        at org.globus.cog.karajan.workflow.nodes.FlowContainer.execute(FlowContainer.java:63)
        at org.globus.cog.karajan.workflow.nodes.FlowNode.restart(FlowNode.java:240)
        at org.globus.cog.karajan.workflow.nodes.FlowNode.start(FlowNode.java:281)
        at org.globus.cog.karajan.workflow.nodes.FlowNode.controlEvent(FlowNode.java:393)
        at org.globus.cog.karajan.workflow.nodes.FlowNode.event(FlowNode.java:332)
        at org.globus.cog.karajan.workflow.FlowElementWrapper.event(FlowElementWrapper.java:227)
        at org.globus.cog.karajan.workflow.events.EventBus.send(EventBus.java:125)
        at org.globus.cog.karajan.workflow.events.EventBus.sendHooked(EventBus.java:99)
        at org.globus.cog.karajan.workflow.events.EventWorker.run(EventWorker.java:69)


So I am guessing this is now a CoG bug right?


On Tue, Feb 3, 2009 at 3:52 PM, Ben Clifford <benc at hawaga.org.uk> wrote:
>
> On Tue, 3 Feb 2009, Allan Espinosa wrote:
>
>> location="gsiftp://gridftp.ranger.tacc.teragrid.org/work/01035/tg802895/pir",
>
>> But gridftp internals in swift reports that it cannot get a directory
>> listing of "work/01035/tg802895/pir".  I was thinking that maybe the
>> leading slashes were removed.  Below is the program output:
>
> CoG's gsiftp handling has an unconventional (as in different to the Globus
> Toolkit) understanding of gsiftp URLs; you need to use two slashes after
> the hostname in gsiftp URLs that are given to CoG.
>
> Globus Toolkit gsiftp URLs implicitly start at /
>
> CoG URLs start in pwd, and need an extra / to get to root.
>
> Both interpretations violate my understand of the ftp:// URI scheme on
> which I understand the gsiftp:// scheme to be modelled.
>
> (to be even more picky, its a violation to use gsiftp:// as a URI scheme
> as it neither begins with x- nor is registered with IANA... ho ho ho)
>


From benc at hawaga.org.uk  Tue Feb  3 16:19:28 2009
From: benc at hawaga.org.uk (Ben Clifford)
Date: Tue, 3 Feb 2009 22:19:28 +0000 (GMT)
Subject: [Swift-user] gsiftp filesystem mapper removes leading slash
In-Reply-To: <50b07b4b0902031412t5f05bda2t4ff88326df678fb7@mail.gmail.com>
References: <50b07b4b0902031343g1c800451j78b0b914c944a416@mail.gmail.com> 
	<Pine.LNX.4.64.0902032147310.8995@dildano.hawaga.org.uk>
	<50b07b4b0902031412t5f05bda2t4ff88326df678fb7@mail.gmail.com>
Message-ID: <Pine.LNX.4.64.0902032218480.14259@dildano.hawaga.org.uk>


On Tue, 3 Feb 2009, Allan Espinosa wrote:

> Oh i see.  now I'm getting NullPointExceptions:

[...]

> So I am guessing this is now a CoG bug right?

That's happening inside Swift code still, so probably, no - its likely 
to be something in Swift.

-- 


From benc at hawaga.org.uk  Tue Feb  3 16:21:52 2009
From: benc at hawaga.org.uk (Ben Clifford)
Date: Tue, 3 Feb 2009 22:21:52 +0000 (GMT)
Subject: [Swift-user] gsiftp filesystem mapper removes leading slash
In-Reply-To: <Pine.LNX.4.64.0902032218480.14259@dildano.hawaga.org.uk>
References: <50b07b4b0902031343g1c800451j78b0b914c944a416@mail.gmail.com> 
	<Pine.LNX.4.64.0902032147310.8995@dildano.hawaga.org.uk>
	<50b07b4b0902031412t5f05bda2t4ff88326df678fb7@mail.gmail.com>
	<Pine.LNX.4.64.0902032218480.14259@dildano.hawaga.org.uk>
Message-ID: <Pine.LNX.4.64.0902032221210.14259@dildano.hawaga.org.uk>

I've seen a bit of funniness in this area before. I'll add some more tests 
to the test suite which will probably help identify problems.
-- 




From aespinosa at cs.uchicago.edu  Tue Feb  3 17:57:38 2009
From: aespinosa at cs.uchicago.edu (Allan Espinosa)
Date: Tue, 3 Feb 2009 17:57:38 -0600
Subject: [Swift-user] mixing gram execution and local filesystem providers
In-Reply-To: <50b07b4b0902031020r125169d2t61e828e4bea07da8@mail.gmail.com>
References: <50b07b4b0901312238k55330bb7gb3a0922fb60f1bb0@mail.gmail.com>
	<Pine.LNX.4.64.0902020936050.14259@dildano.hawaga.org.uk>
	<50b07b4b0902021140l64290db1u57ad652025c5182c@mail.gmail.com>
	<50b07b4b0902021203m45dccb32k8e0b94272e320842@mail.gmail.com>
	<Pine.LNX.4.64.0902022005350.14259@dildano.hawaga.org.uk>
	<50b07b4b0902021211j6d11ea15t32d923464da55b1@mail.gmail.com>
	<Pine.LNX.4.64.0902022047540.14259@dildano.hawaga.org.uk>
	<50b07b4b0902021252y71db65dfxf4317a323e4b5540@mail.gmail.com>
	<498860B6.7010307@mcs.anl.gov>
	<50b07b4b0902031020r125169d2t61e828e4bea07da8@mail.gmail.com>
Message-ID: <50b07b4b0902031557p1e541894gb2ef11254c8bc308@mail.gmail.com>

This is weird, I was successful in submitting over 250 jobs but then
got the errors again after a while:
log:
http://www.ci.uchicago.edu/~aespinosa/swift/blast-20090203-1709-8dc38njb.log

...
Progress:  Selecting site:377 Active:1 Finished successfully:253
Failed but can retry:69
Failed to transfer wrapper log from
blast-20090203-1709-8dc38njb/info/r on TERAPORT
Progress:  Selecting site:377 Finished successfully:253 Failed but can retry:70
Progress:  Selecting site:377 Finished successfully:253 Failed but can retry:70
Progress:  Selecting site:377 Finished successfully:253 Failed but can retry:70
Progress:  Selecting site:376 Stage in:1 Finished successfully:253
Failed but can retry:70
Progress:  Selecting site:376 Active:1 Finished successfully:253
Failed but can retry:70
Failed to transfer wrapper log from
blast-20090203-1709-8dc38njb/info/t on TERAPORT

sites.xml:
  <pool handle="TERAPORT" sysinfo="INTEL32::LINUX">
    <profile namespace="globus" key="queue">fast</profile>
    <profile namespace="globus"
key="coasterWorkerMaxwalltime">00:30:00</profile>
    <gridftp url="local://localhost" />
    <execution provider="coaster" url="tp-grid1.uchicago.edu"
jobmanager="gt2:gt2:pbs" />
    <workdirectory >/disks/tp-gpfs/scratch/aespinosa</workdirectory>
  </pool>

Does this mean that the "cp" calls fail after a while?

-Allan

On Tue, Feb 3, 2009 at 12:20 PM, Allan Espinosa
<aespinosa at cs.uchicago.edu> wrote:
> Hi mike
> yes it gives a different java environment (but the same for both
> submission hosts communicado and tp-login1:
> globus-job-run tp-grid1.ci.uchicago.edu /bin/bash -l -c "which java"
> SoftEnv 1.6.0: updating your software environment, one moment...
> /autonfs/software/linux-rhel4-x86_64/osg-client-1.0.0-r1/jdk1.5/bin/java


From benc at hawaga.org.uk  Tue Feb  3 18:11:15 2009
From: benc at hawaga.org.uk (Ben Clifford)
Date: Wed, 4 Feb 2009 00:11:15 +0000 (GMT)
Subject: [Swift-user] mixing gram execution and local filesystem providers
In-Reply-To: <50b07b4b0902031557p1e541894gb2ef11254c8bc308@mail.gmail.com>
References: <50b07b4b0901312238k55330bb7gb3a0922fb60f1bb0@mail.gmail.com> 
	<Pine.LNX.4.64.0902020936050.14259@dildano.hawaga.org.uk> 
	<50b07b4b0902021140l64290db1u57ad652025c5182c@mail.gmail.com> 
	<50b07b4b0902021203m45dccb32k8e0b94272e320842@mail.gmail.com> 
	<Pine.LNX.4.64.0902022005350.14259@dildano.hawaga.org.uk> 
	<50b07b4b0902021211j6d11ea15t32d923464da55b1@mail.gmail.com> 
	<Pine.LNX.4.64.0902022047540.14259@dildano.hawaga.org.uk> 
	<50b07b4b0902021252y71db65dfxf4317a323e4b5540@mail.gmail.com> 
	<498860B6.7010307@mcs.anl.gov>
	<50b07b4b0902031020r125169d2t61e828e4bea07da8@mail.gmail.com>
	<50b07b4b0902031557p1e541894gb2ef11254c8bc308@mail.gmail.com>
Message-ID: <Pine.LNX.4.64.0902040007320.14259@dildano.hawaga.org.uk>


On Tue, 3 Feb 2009, Allan Espinosa wrote:

> Does this mean that the "cp" calls fail after a while?

It means something failed. In the log, I see this:

> org.globus.cog.abstraction.impl.common.task.InvalidSecurityContextException: 
Invalid GSSCredentials

and

> Caused by: GSSException: Defective credential detected [Caused by: 
> [JGLOBUS-5] Proxy file 
> (/home/aespinosa/.globus/job/tp-grid1.ci.uchicago.edu/11827.1233702664/x509_up) 
not found.]

which is a strange looking error.

I'd expect credential errors if your proxy just expired (please paste the 
output of grid-proxy-info from the same session that you ran this swift 
invocation from, if you still have it).

But I wouldn't expect the error message to have a path to a credential 
inside what looks like the remote job directory.

-- 


From aespinosa at cs.uchicago.edu  Tue Feb  3 18:33:25 2009
From: aespinosa at cs.uchicago.edu (Allan Espinosa)
Date: Tue, 3 Feb 2009 18:33:25 -0600
Subject: [Swift-user] mixing gram execution and local filesystem providers
In-Reply-To: <50b07b4b0902031632o16e58973pb8208fd6cffd49b9@mail.gmail.com>
References: <50b07b4b0901312238k55330bb7gb3a0922fb60f1bb0@mail.gmail.com>
	<Pine.LNX.4.64.0902022005350.14259@dildano.hawaga.org.uk>
	<50b07b4b0902021211j6d11ea15t32d923464da55b1@mail.gmail.com>
	<Pine.LNX.4.64.0902022047540.14259@dildano.hawaga.org.uk>
	<50b07b4b0902021252y71db65dfxf4317a323e4b5540@mail.gmail.com>
	<498860B6.7010307@mcs.anl.gov>
	<50b07b4b0902031020r125169d2t61e828e4bea07da8@mail.gmail.com>
	<50b07b4b0902031557p1e541894gb2ef11254c8bc308@mail.gmail.com>
	<Pine.LNX.4.64.0902040007320.14259@dildano.hawaga.org.uk>
	<50b07b4b0902031632o16e58973pb8208fd6cffd49b9@mail.gmail.com>
Message-ID: <50b07b4b0902031633o211b1fe4oecc44e73476f3907@mail.gmail.com>

My proxy certificate was freshly renewed when i submitted the job.

so it is looking for  a proxy certificate in the remote site?

[aespinosa at communicado ~]$ grid-proxy-info
subject  : /DC=org/DC=doegrids/OU=People/CN=Allan M. Espinosa
374652/CN=1242752696
issuer   : /DC=org/DC=doegrids/OU=People/CN=Allan M. Espinosa 374652
identity : /DC=org/DC=doegrids/OU=People/CN=Allan M. Espinosa 374652
type     : Proxy draft (pre-RFC) compliant impersonation proxy
strength : 512 bits
path     : /tmp/x509up_u1600
timeleft : 10:38:32


On Tue, Feb 3, 2009 at 6:11 PM, Ben Clifford <benc at hawaga.org.uk> wrote:
>
> On Tue, 3 Feb 2009, Allan Espinosa wrote:
>
>> Does this mean that the "cp" calls fail after a while?
>
> It means something failed. In the log, I see this:
>
>> org.globus.cog.abstraction.impl.common.task.InvalidSecurityContextException:
> Invalid GSSCredentials
>
> and
>
>> Caused by: GSSException: Defective credential detected [Caused by:
>> [JGLOBUS-5] Proxy file
>> (/home/aespinosa/.globus/job/tp-grid1.ci.uchicago.edu/11827.1233702664/x509_up)
> not found.]
>
> which is a strange looking error.
>
> I'd expect credential errors if your proxy just expired (please paste the
> output of grid-proxy-info from the same session that you ran this swift
> invocation from, if you still have it).
>
> But I wouldn't expect the error message to have a path to a credential
> inside what looks like the remote job directory.
>
> --
>
>



--
Allan M. Espinosa <http://allan.88-mph.net/blog>
PhD student, Computer Science
University of Chicago <http://people.cs.uchicago.edu/~aespinosa>



-- 
Allan M. Espinosa <http://allan.88-mph.net/blog>
PhD student, Computer Science
University of Chicago <http://people.cs.uchicago.edu/~aespinosa>


From hategan at mcs.anl.gov  Tue Feb  3 18:34:15 2009
From: hategan at mcs.anl.gov (Mihael Hategan)
Date: Tue, 03 Feb 2009 18:34:15 -0600
Subject: [Swift-user] mixing gram execution and local filesystem providers
In-Reply-To: <Pine.LNX.4.64.0902040007320.14259@dildano.hawaga.org.uk>
References: <50b07b4b0901312238k55330bb7gb3a0922fb60f1bb0@mail.gmail.com>
	<Pine.LNX.4.64.0902020936050.14259@dildano.hawaga.org.uk>
	<50b07b4b0902021140l64290db1u57ad652025c5182c@mail.gmail.com>
	<50b07b4b0902021203m45dccb32k8e0b94272e320842@mail.gmail.com>
	<Pine.LNX.4.64.0902022005350.14259@dildano.hawaga.org.uk>
	<50b07b4b0902021211j6d11ea15t32d923464da55b1@mail.gmail.com>
	<Pine.LNX.4.64.0902022047540.14259@dildano.hawaga.org.uk>
	<50b07b4b0902021252y71db65dfxf4317a323e4b5540@mail.gmail.com>
	<498860B6.7010307@mcs.anl.gov>
	<50b07b4b0902031020r125169d2t61e828e4bea07da8@mail.gmail.com>
	<50b07b4b0902031557p1e541894gb2ef11254c8bc308@mail.gmail.com>
	<Pine.LNX.4.64.0902040007320.14259@dildano.hawaga.org.uk>
Message-ID: <1233707655.12879.0.camel@localhost>

On Wed, 2009-02-04 at 00:11 +0000, Ben Clifford wrote:
> On Tue, 3 Feb 2009, Allan Espinosa wrote:
> 
> > Does this mean that the "cp" calls fail after a while?
> 
> It means something failed. In the log, I see this:
> 
> > org.globus.cog.abstraction.impl.common.task.InvalidSecurityContextException: 
> Invalid GSSCredentials
> 
> and
> 
> > Caused by: GSSException: Defective credential detected [Caused by: 
> > [JGLOBUS-5] Proxy file 
> > (/home/aespinosa/.globus/job/tp-grid1.ci.uchicago.edu/11827.1233702664/x509_up) 
> not found.]
> 
> which is a strange looking error.
> 
> I'd expect credential errors if your proxy just expired (please paste the 
> output of grid-proxy-info from the same session that you ran this swift 
> invocation from, if you still have it).
> 
> But I wouldn't expect the error message to have a path to a credential 
> inside what looks like the remote job directory.

Unless it's a delegated credential and the error happens in a gram job?



From aespinosa at cs.uchicago.edu  Wed Feb  4 14:43:21 2009
From: aespinosa at cs.uchicago.edu (Allan Espinosa)
Date: Wed, 4 Feb 2009 14:43:21 -0600
Subject: [Swift-user] local coaster looking for credential (was Re: mixing
	gram execution and local filesystem providers)
Message-ID: <50b07b4b0902041243w3ff221c5s69dadf6f19c22a9d@mail.gmail.com>

Do coasters always need a proxy certificate?  This was found in my log
when i executed using  the "local:pbs" job manager

Caused by: org.globus.gsi.GlobusCredentialException: [JGLOBUS-5] Proxy
file (/tmp/x509up_u1600) not found.
        at org.globus.gsi.GlobusCredential.<init>(GlobusCredential.java:114)
        at org.globus.gsi.GlobusCredential.reloadDefaultCredential(GlobusCredential.java:590)

full log in: http://www.ci.uchicago.edu/~aespinosa/swift/blast-20090204-1438-9vr02924.log
sites.xml:
  <pool handle="TERAPORT" sysinfo="INTEL32::LINUX">
    <profile namespace="globus" key="queue">fast</profile>
    <profile namespace="globus"
key="coasterWorkerMaxwalltime">00:30:00</profile>
    <execution provider="coaster" url="none"  jobmanager="local:pbs" />
    <gridftp url="local://localhost" />
    <workdirectory >/disks/tp-gpfs/scratch/aespinosa</workdirectory>
  </pool>

-Allan


On Tue, Feb 3, 2009 at 6:11 PM, Ben Clifford <benc at hawaga.org.uk> wrote:
>
> On Tue, 3 Feb 2009, Allan Espinosa wrote:
>
>> Does this mean that the "cp" calls fail after a while?
>
> It means something failed. In the log, I see this:
>
>> org.globus.cog.abstraction.impl.common.task.InvalidSecurityContextException:
> Invalid GSSCredentials
>
> and
>
>> Caused by: GSSException: Defective credential detected [Caused by:
>> [JGLOBUS-5] Proxy file
>> (/home/aespinosa/.globus/job/tp-grid1.ci.uchicago.edu/11827.1233702664/x509_up)
> not found.]
>
> which is a strange looking error.
>
> I'd expect credential errors if your proxy just expired (please paste the
> output of grid-proxy-info from the same session that you ran this swift
> invocation from, if you still have it).
>
> But I wouldn't expect the error message to have a path to a credential
> inside what looks like the remote job directory.


From wilde at mcs.anl.gov  Wed Feb  4 14:50:20 2009
From: wilde at mcs.anl.gov (Michael Wilde)
Date: Wed, 04 Feb 2009 14:50:20 -0600
Subject: [Swift-user] local coaster looking for credential (was Re: mixing
	gram execution and local filesystem providers)
In-Reply-To: <50b07b4b0902041243w3ff221c5s69dadf6f19c22a9d@mail.gmail.com>
References: <50b07b4b0902041243w3ff221c5s69dadf6f19c22a9d@mail.gmail.com>
Message-ID: <4989FF8C.3050106@mcs.anl.gov>

What filenames are you mapping?  Is it perhaps trying to use gridftp to 
access them?

What if you use the <filesystem> tag instead of <gridftp>?

- Mike


On 2/4/09 2:43 PM, Allan Espinosa wrote:
> Do coasters always need a proxy certificate?  This was found in my log
> when i executed using  the "local:pbs" job manager
> 
> Caused by: org.globus.gsi.GlobusCredentialException: [JGLOBUS-5] Proxy
> file (/tmp/x509up_u1600) not found.
>         at org.globus.gsi.GlobusCredential.<init>(GlobusCredential.java:114)
>         at org.globus.gsi.GlobusCredential.reloadDefaultCredential(GlobusCredential.java:590)
> 
> full log in: http://www.ci.uchicago.edu/~aespinosa/swift/blast-20090204-1438-9vr02924.log
> sites.xml:
>   <pool handle="TERAPORT" sysinfo="INTEL32::LINUX">
>     <profile namespace="globus" key="queue">fast</profile>
>     <profile namespace="globus"
> key="coasterWorkerMaxwalltime">00:30:00</profile>
>     <execution provider="coaster" url="none"  jobmanager="local:pbs" />
>     <gridftp url="local://localhost" />
>     <workdirectory >/disks/tp-gpfs/scratch/aespinosa</workdirectory>
>   </pool>
> 
> -Allan
> 
> 
> On Tue, Feb 3, 2009 at 6:11 PM, Ben Clifford <benc at hawaga.org.uk> wrote:
>> On Tue, 3 Feb 2009, Allan Espinosa wrote:
>>
>>> Does this mean that the "cp" calls fail after a while?
>> It means something failed. In the log, I see this:
>>
>>> org.globus.cog.abstraction.impl.common.task.InvalidSecurityContextException:
>> Invalid GSSCredentials
>>
>> and
>>
>>> Caused by: GSSException: Defective credential detected [Caused by:
>>> [JGLOBUS-5] Proxy file
>>> (/home/aespinosa/.globus/job/tp-grid1.ci.uchicago.edu/11827.1233702664/x509_up)
>> not found.]
>>
>> which is a strange looking error.
>>
>> I'd expect credential errors if your proxy just expired (please paste the
>> output of grid-proxy-info from the same session that you ran this swift
>> invocation from, if you still have it).
>>
>> But I wouldn't expect the error message to have a path to a credential
>> inside what looks like the remote job directory.
> _______________________________________________
> Swift-user mailing list
> Swift-user at ci.uchicago.edu
> http://mail.ci.uchicago.edu/mailman/listinfo/swift-user


From hategan at mcs.anl.gov  Wed Feb  4 14:58:57 2009
From: hategan at mcs.anl.gov (Mihael Hategan)
Date: Wed, 4 Feb 2009 14:58:57 -0600 (CST)
Subject: [Swift-user] local coaster looking for credential (was Re:
	mixing gram execution and local filesystem providers)
In-Reply-To: <50b07b4b0902041243w3ff221c5s69dadf6f19c22a9d@mail.gmail.com>
Message-ID: <8848533.1789051233781137932.JavaMail.root@zimbra>

Yes. Coasters always need a valid proxy certificate.


----- "Allan Espinosa" <aespinosa at cs.uchicago.edu> wrote:

> Do coasters always need a proxy certificate?  This was found in my
> log
> when i executed using  the "local:pbs" job manager
> 
> Caused by: org.globus.gsi.GlobusCredentialException: [JGLOBUS-5]
> Proxy
> file (/tmp/x509up_u1600) not found.
>         at
> org.globus.gsi.GlobusCredential.<init>(GlobusCredential.java:114)
>         at
> org.globus.gsi.GlobusCredential.reloadDefaultCredential(GlobusCredential.java:590)
> 
> full log in:
> http://www.ci.uchicago.edu/~aespinosa/swift/blast-20090204-1438-9vr02924.log
> sites.xml:
>   <pool handle="TERAPORT" sysinfo="INTEL32::LINUX">
>     <profile namespace="globus" key="queue">fast</profile>
>     <profile namespace="globus"
> key="coasterWorkerMaxwalltime">00:30:00</profile>
>     <execution provider="coaster" url="none"  jobmanager="local:pbs"
> />
>     <gridftp url="local://localhost" />
>     <workdirectory >/disks/tp-gpfs/scratch/aespinosa</workdirectory>
>   </pool>
> 
> -Allan
> 
> 
> On Tue, Feb 3, 2009 at 6:11 PM, Ben Clifford <benc at hawaga.org.uk>
> wrote:
> >
> > On Tue, 3 Feb 2009, Allan Espinosa wrote:
> >
> >> Does this mean that the "cp" calls fail after a while?
> >
> > It means something failed. In the log, I see this:
> >
> >>
> org.globus.cog.abstraction.impl.common.task.InvalidSecurityContextException:
> > Invalid GSSCredentials
> >
> > and
> >
> >> Caused by: GSSException: Defective credential detected [Caused by:
> >> [JGLOBUS-5] Proxy file
> >>
> (/home/aespinosa/.globus/job/tp-grid1.ci.uchicago.edu/11827.1233702664/x509_up)
> > not found.]
> >
> > which is a strange looking error.
> >
> > I'd expect credential errors if your proxy just expired (please
> paste the
> > output of grid-proxy-info from the same session that you ran this
> swift
> > invocation from, if you still have it).
> >
> > But I wouldn't expect the error message to have a path to a
> credential
> > inside what looks like the remote job directory.
> _______________________________________________
> Swift-user mailing list
> Swift-user at ci.uchicago.edu
> http://mail.ci.uchicago.edu/mailman/listinfo/swift-user


From wilde at mcs.anl.gov  Wed Feb  4 17:02:32 2009
From: wilde at mcs.anl.gov (Michael Wilde)
Date: Wed, 04 Feb 2009 17:02:32 -0600
Subject: [Swift-user] Expanding arrays in app function command lines
In-Reply-To: <Pine.LNX.4.64.0901271628080.14259@dildano.hawaga.org.uk>
References: <497E02DB.8040904@mcs.anl.gov>
	<Pine.LNX.4.64.0901261839360.14996@dildano.hawaga.org.uk>
	<Pine.LNX.4.64.0901271608040.14259@dildano.hawaga.org.uk>
	<497F35CE.8070709@mcs.anl.gov>
	<Pine.LNX.4.64.0901271628080.14259@dildano.hawaga.org.uk>
Message-ID: <498A1E88.5040907@mcs.anl.gov>

Is the expansion of structs into the command line not yet implemented, 
in the latest patch for expanding arrays?
(vdlarg-1.patch) which I have applied.

On 1/27/09 10:29 AM, Ben Clifford wrote:
> On Tue, 27 Jan 2009, Michael Wilde wrote:
> 
>> - array of other scalar types, coerced into string in the cmd line
> 
> I think that might already happen.
> 
> Everything apart from struct has a pretty natural expansion into string 
> args. The source-code order of fields for structs is probably the best, 
> unless someone comes up with a better idea.

This program:

//---
type file;

type struct { string a; string b; };
struct s;
s.a = "a=123";
s.b = "b=456";

app (file o) echo (struct s) { echo s stdout=@o; }

file o<"struct.out">;

o = echo(s);
//---

produces:

com$ cat struct.out
org.griphyn.vdl.mapping.RootDataNode identifier 
tag:benc at ci.uchicago.edu,2008:swift:dataset:20090204-1653-ez43h08a:720000000003 
type struct with no value at dataset=s (closed)
com$

---

I get the same behavior if I also pass s.a and s.b to echo, to make sure 
the fields are set before echo is called:

//---
type file;

type struct { string a; string b; };

struct s;
s.a = "a=123";
s.b = "b=456";

app (file o) echo (struct s, string i, string j) { echo "s=" s "s.a=" 
s.a "s.b=" s.b stdout=@o; }

file o<"struct.out">;

o = echo(s, s.a, s.b );
//---

com$ cat struct.out
s= org.griphyn.vdl.mapping.RootDataNode identifier 
tag:benc at ci.uchicago.edu,2008:swift:dataset:20090204-1700-u8uet3u4:720000000006 
type struct with no value at dataset=s (closed) s.a= a=123 s.b= b=456
com$


From benc at hawaga.org.uk  Thu Feb  5 03:27:17 2009
From: benc at hawaga.org.uk (Ben Clifford)
Date: Thu, 5 Feb 2009 09:27:17 +0000 (GMT)
Subject: [Swift-user] Expanding arrays in app function command lines
In-Reply-To: <498A1E88.5040907@mcs.anl.gov>
References: <497E02DB.8040904@mcs.anl.gov>
	<Pine.LNX.4.64.0901261839360.14996@dildano.hawaga.org.uk>
	<Pine.LNX.4.64.0901271608040.14259@dildano.hawaga.org.uk>
	<497F35CE.8070709@mcs.anl.gov>
	<Pine.LNX.4.64.0901271628080.14259@dildano.hawaga.org.uk>
	<498A1E88.5040907@mcs.anl.gov>
Message-ID: <Pine.LNX.4.64.0902050925150.14259@dildano.hawaga.org.uk>


On Wed, 4 Feb 2009, Michael Wilde wrote:

> Is the expansion of structs into the command line not yet implemented, in the
> latest patch for expanding arrays?
> (vdlarg-1.patch) which I have applied.

Its not implemented. I thought I'd made it throw a type checking error in 
that case, but apparently not.

Lines like this are the output of naively trying to turn an arbitrary 
dataset into a string:

> org.griphyn.vdl.mapping.RootDataNode identifier
> tag:benc at ci.uchicago.edu,2008:swift:dataset:20090204-1653-ez43h08a:720000000003
> type struct with no value at dataset=s (closed)

> I get the same behavior if I also pass s.a and s.b to echo, to make sure the
> fields are set before echo is called:

It should (although I'm not entirely sure if it actually does) not invoke 
the procedure until all of s is set, if you pass in s.

-- 



From wilde at mcs.anl.gov  Sat Feb  7 10:24:41 2009
From: wilde at mcs.anl.gov (Michael Wilde)
Date: Sat, 07 Feb 2009 10:24:41 -0600
Subject: [Swift-user] mapper for internet resources
In-Reply-To: <32078b1e0902070144te6862b4mfa11c2ba65fa529e@mail.gmail.com>
References: <32078b1e0902070144te6862b4mfa11c2ba65fa529e@mail.gmail.com>
Message-ID: <498DB5C9.1030108@mcs.anl.gov>

Uri, I took the liberty of replying via swift-user.
Thats the best place to send these questions.

Swift can do this already, as below.

A mapper can map an ordinary file as a local file name (relative or 
absolute), or an http:// or gsiftp:// URI. (Perhaps others a well, I 
think, eg, ftp:// depending on the underlying CoG and Globus APIs.)

- Mike

------

type file;

app (file f) afniapp (file u) { cat @u stdout=@f; }

file mydata <"http://www.ci.uchicago.edu/~wilde/mydata.txt">;
file output <"gotit.txt">;

output = afniapp(mydata);


On 2/7/09 3:44 AM, Uri Hasson wrote:
> Hi all,
> 
> Is there currently a way to use swift mappers to access internet resources?
> 
> That is, rather than have a file as input, have a "url" mapper than
> can manage something like this:
> 
> AFNI_obj normalized<simple_url_mapper;
> prefix=@strcat("http://www.mysite.com/somepath/audioTScat_nrmlzdPerChng.",subject,"+orig.")>;.
> 
> This would enable us to run a swift script on inputs from other file
> systems (for Read processes).
> 
> Writing files will need to be to the machine where swift is running from.
> 
> Best,
> Uri


From wilde at mcs.anl.gov  Sat Feb  7 10:44:41 2009
From: wilde at mcs.anl.gov (Michael Wilde)
Date: Sat, 07 Feb 2009 10:44:41 -0600
Subject: [Swift-user] mapper for internet resources
In-Reply-To: <498DB5C9.1030108@mcs.anl.gov>
References: <32078b1e0902070144te6862b4mfa11c2ba65fa529e@mail.gmail.com>
	<498DB5C9.1030108@mcs.anl.gov>
Message-ID: <498DBA79.2050709@mcs.anl.gov>

Also:

 >> Writing files will need to be to the machine where swift is running 
from.

You can write files to any gridftp server by mapping to a gsiftp:// URI, 
as with input.

As far as I understand, the gsiftp provider can support: http, https, 
ftp, and gsiftp URIs, with the gridftp data provider, <gridftp> in 
sites.xml.

If the uri works with globus-url-copy I think it will work with the 
gridftp provider.

I think that Swift can also accept other providers from the CoG toolkit, 
such as scp and webdav. See
   http://www.ci.uchicago.edu/swift/guides/userguide.php#id2745803
for using scp.  We should test and document hos to use webdav.

Perhaps Mihael or Ben can verify and clarify the above.

- Mike


On 2/7/09 10:24 AM, Michael Wilde wrote:
> Uri, I took the liberty of replying via swift-user.
> Thats the best place to send these questions.
> 
> Swift can do this already, as below.
> 
> A mapper can map an ordinary file as a local file name (relative or 
> absolute), or an http:// or gsiftp:// URI. (Perhaps others a well, I 
> think, eg, ftp:// depending on the underlying CoG and Globus APIs.)
> 
> - Mike
> 
> ------
> 
> type file;
> 
> app (file f) afniapp (file u) { cat @u stdout=@f; }
> 
> file mydata <"http://www.ci.uchicago.edu/~wilde/mydata.txt">;
> file output <"gotit.txt">;
> 
> output = afniapp(mydata);
> 
> 
> On 2/7/09 3:44 AM, Uri Hasson wrote:
>> Hi all,
>>
>> Is there currently a way to use swift mappers to access internet 
>> resources?
>>
>> That is, rather than have a file as input, have a "url" mapper than
>> can manage something like this:
>>
>> AFNI_obj normalized<simple_url_mapper;
>> prefix=@strcat("http://www.mysite.com/somepath/audioTScat_nrmlzdPerChng.",subject,"+orig.")>;. 
>>
>>
>> This would enable us to run a swift script on inputs from other file
>> systems (for Read processes).
>>
>> Writing files will need to be to the machine where swift is running from.
>>
>> Best,
>> Uri
> _______________________________________________
> Swift-user mailing list
> Swift-user at ci.uchicago.edu
> http://mail.ci.uchicago.edu/mailman/listinfo/swift-user


From hategan at mcs.anl.gov  Sat Feb  7 11:58:50 2009
From: hategan at mcs.anl.gov (Mihael Hategan)
Date: Sat, 7 Feb 2009 11:58:50 -0600 (CST)
Subject: [Swift-user] mapper for internet resources
In-Reply-To: <498DB5C9.1030108@mcs.anl.gov>
Message-ID: <32192206.1951551234029530606.JavaMail.root@zimbra>


----- Michael Wilde <wilde at mcs.anl.gov> wrote:
> Uri, I took the liberty of replying via swift-user.
> Thats the best place to send these questions.
> 
> Swift can do this already, as below.
> 
> A mapper can map an ordinary file as a local file name (relative or 
> absolute), or an http:// or gsiftp:// URI. (Perhaps others a well, I 
> think, eg, ftp:// depending on the underlying CoG and Globus APIs.)

ftp should be supported.


From hategan at mcs.anl.gov  Sat Feb  7 12:05:47 2009
From: hategan at mcs.anl.gov (Mihael Hategan)
Date: Sat, 7 Feb 2009 12:05:47 -0600 (CST)
Subject: [Swift-user] mapper for internet resources
In-Reply-To: <498DBA79.2050709@mcs.anl.gov>
Message-ID: <30478411.1951611234029947420.JavaMail.root@zimbra>


----- Michael Wilde <wilde at mcs.anl.gov> wrote:
> Also:
> 
>  >> Writing files will need to be to the machine where swift is running 
> from.
> 
> You can write files to any gridftp server by mapping to a gsiftp:// URI, 
> as with input.
> 
> As far as I understand, the gsiftp provider can support: http, https, 
> ftp, and gsiftp URIs, with the gridftp data provider, <gridftp> in 
> sites.xml.

The gsiftp provider only deals with gsiftp. There are other 
providers for ftp and http, but I think http is in read only mode.

However, you don't need to care which provider implements a protocol
if it is supported. Just write the URL and see if it works.

> 
> If the uri works with globus-url-copy I think it will work with the 
> gridftp provider.

The cog globus-url-copy is built on slightly different APIs than Swift,
so that statement is not, in general, correct.


From benc at hawaga.org.uk  Sun Feb  8 06:38:25 2009
From: benc at hawaga.org.uk (Ben Clifford)
Date: Sun, 8 Feb 2009 12:38:25 +0000 (GMT)
Subject: [Swift-user] mapper for internet resources
In-Reply-To: <498DBA79.2050709@mcs.anl.gov>
References: <32078b1e0902070144te6862b4mfa11c2ba65fa529e@mail.gmail.com>
	<498DB5C9.1030108@mcs.anl.gov> <498DBA79.2050709@mcs.anl.gov>
Message-ID: <Pine.LNX.4.64.0902081237100.8995@dildano.hawaga.org.uk>


On Sat, 7 Feb 2009, Michael Wilde wrote:

> As far as I understand, the gsiftp provider can support: http, https, ftp, and
> gsiftp URIs, with the gridftp data provider, <gridftp> in sites.xml.

In addition to the clarification that mihael gave, the above commentary is 
incorrect - the <gridftp> (or <filesystem>) entry in sites.xml specifies 
the mechanism used to access the site-side filesystem, not the mechanism 
used to access submit-side data.

-- 


From wilde at mcs.anl.gov  Sun Feb  8 10:16:19 2009
From: wilde at mcs.anl.gov (Michael Wilde)
Date: Sun, 08 Feb 2009 10:16:19 -0600
Subject: [Swift-user] mapper for internet resources
In-Reply-To: <Pine.LNX.4.64.0902081237100.8995@dildano.hawaga.org.uk>
References: <32078b1e0902070144te6862b4mfa11c2ba65fa529e@mail.gmail.com>
	<498DB5C9.1030108@mcs.anl.gov> <498DBA79.2050709@mcs.anl.gov>
	<Pine.LNX.4.64.0902081237100.8995@dildano.hawaga.org.uk>
Message-ID: <498F0553.5060008@mcs.anl.gov>


On 2/8/09 6:38 AM, Ben Clifford wrote:
> On Sat, 7 Feb 2009, Michael Wilde wrote:
> 
>> As far as I understand, the gsiftp provider can support: http, https, ftp, and
>> gsiftp URIs, with the gridftp data provider, <gridftp> in sites.xml.
> 
> In addition to the clarification that mihael gave, the above commentary is 
> incorrect - the <gridftp> (or <filesystem>) entry in sites.xml specifies 
> the mechanism used to access the site-side filesystem, not the mechanism 
> used to access submit-side data.

This would be good to clarify in the users guide in sec 3, Mappers, 
where we should say what kinds of URIs a mapper can specify to denote 
the "submit-side data"; sec 5.2, "notes on how swift implements file 
input and output"; and sec 13.2 when we describe how the site work 
directory is specified and accessed.

In 5.2 we say "The Swift client side selects a site; copies the input 
files for that procedure call to the site shared file cache if they are 
not already in the cache, using the file transfer mechanism; and then 
invokes the wrapper script on that site using the execution mechanism. "

We should clarify that mappings specify the source and destinations of 
input and output files, and that a variety of forms of URIs and 
pathnames can be specified in the mapping declaration.

We should state how and when output data is copied back to its mapped 
location.


From wilde at mcs.anl.gov  Fri Feb 13 16:08:00 2009
From: wilde at mcs.anl.gov (Michael Wilde)
Date: Fri, 13 Feb 2009 16:08:00 -0600
Subject: [Swift-user] Thoughts on a data dependency issues
Message-ID: <4995EF40.4030304@mcs.anl.gov>

I'm looking for advice on scripting the following in Swift:

One of the inputs to the DOCK app is a protein description that has been 
run through a time-intensive prep application stage called GRID to 
produce a binary description of the protein structure in a format that 
is machine-architecture specific.

Essentially what we have is the following 2-stage workflow:

   gridfile = grid (protein)
   forach c, i in compoundList
     dockfile[i] = dock (protein, gridfile, c)

To be conservative, we generate the gridfiles on the same host that the 
DOCK stage of the application will run on. Theoretically we could 
generate one gridfile per architecture, and then let send the right 
architecture file to the dock app.

I would like to trigger the generation of the "grid" file from the swift 
script.

This presents two problems:

1) we dont know what machine the dock will run on, and
2) we need a way to name the grid file so that it can be cached for 
later use by multiple docks.

This is not a problem in single-site environments, but if each dock job 
could run on a site having one of several architectures, its more 
challenging.

Certainly one way is to hide the gridfile and the grid() stage from 
Swift, and thats not a problem.

Another approach is to name the gridfiles with an architecture suffix, 
eg 1UBQ.grid.x86, .x86-64, .ppc. etc. and pass all the archs to each 
dock() app. That would cause excess data traffic, but not too bad if 
they are only moved once. Then an app wrapper dynamically picks the 
right arch file.

An interesting case/feature, perhaps, is some new swift option to say 
dont transfer the file, just pass the filename, and send a uri (returned 
by the mapper) that an app wrapper can fetch dynamically.

(this verges on a discussion for swift-devel) if no one sees other 
clever ways to express this.

This is a low-prio issue, as we work around it by pre-generating the 
grid files or running on just one arch.

- Mike





From benc at hawaga.org.uk  Fri Feb 13 16:30:26 2009
From: benc at hawaga.org.uk (Ben Clifford)
Date: Fri, 13 Feb 2009 22:30:26 +0000 (GMT)
Subject: [Swift-user] Thoughts on a data dependency issues
In-Reply-To: <4995EF40.4030304@mcs.anl.gov>
References: <4995EF40.4030304@mcs.anl.gov>
Message-ID: <Pine.LNX.4.64.0902132218550.1293@dildano.hawaga.org.uk>


There's a similar situation when staging in applications in the way that 
Mats and I did for his sample application - if you have multiple 
architectures, and are staging in an application, then it would be nice to 
parameterise which file goes in by selected site.

That does really slightly wierd stuff with dependencies, though - the set 
of sites available for a particular job then depends on which intermediate 
files have been created, and perhaps introduces something like the 
non-deterministic dependencies that have been talked about before: compute 
these 7 files on 7 different sites, and then whichever one finishes first, 
use that.

> An interesting case/feature, perhaps, is some new swift option to say 
> dont transfer the file, just pass the filename, and send a uri (returned 
> by the mapper) that an app wrapper can fetch dynamically.

There is an 'extern' type for representing dependencies on data that is 
not managed by Swift. No mapping happens, and you get to move data round 
however you want. Performing your own staging from your own data store, 
for example by pulling something from a URI that you have constructed 
yourself, using your own application-side data retrieval mechanism seems a 
reasonable use case for 'extern'.

Beyond that, I don't know if any further support is needed from Swift - 
try it and see.

-- 



From wilde at mcs.anl.gov  Fri Feb 20 07:07:31 2009
From: wilde at mcs.anl.gov (Michael Wilde)
Date: Fri, 20 Feb 2009 07:07:31 -0600
Subject: [Swift-user] Questions on use of iterate statement
Message-ID: <499EAB13.5050401@mcs.anl.gov>

Im trying to use iterate to test successive sets of parallel simulations 
for a termination condition, and can't find a way to do this.

Ive distilled, I think, the problem down to the following.

One can say this:

iterate i {
   int j = i * i;
   trace(j);
} until (i>3);

but not this:

iterate i {
   int j = i * i;
   trace(j);
} until (j>15);

because the value j, computed by the "simulation" i*i is not in scope in 
the termination condition.

In my case, the simulation is a protein fold in a parallel foreach. Each 
simulation computes a set of files. After each set of parallel 
simulations completes, I need to check the set of result files to see if 
   they have achieved a termination condition (either reached a 
threshold or a limit on the number of attempts).

I may be stumbling on some strange array closing problems in the real 
case, so Ive tried to distill the problem into a simpler test case.

The current "real" code is pasted at the end of this message (lines are 
wrapped; file is http://www.ci.uchicago.edu/~wilde/oops8.swift).

An east-to-run test case which I think shows a similar problem is this:

string protein[] = ["prot1","prot2"];
int iter[] = [0,1,2];

(boolean rc) done (int i[]) // fake termination test
{
   if (i[0] == 3) {
     rc = true;
   }
   else {
     rc = false;
   }
}

foreach p in protein {
   iterate i {  // do sets of parallel simulations
                // until convergence achieved or limit reached
     int result[]; // Holds results of 1 set of parallel simulations
     foreach j in iter { // do a set of "iter" simulations in parallel
       string out=@strcat("output/r",i,"/","prot1",".",j);
       trace (out);
       result[j] = i;
     }
   } until (i==3);
//} until (done(result));
}

This works, and produces the expected series of output trace lines.

But when i try to use the alternative termination test "done(result)" I get:

Could not start execution.

Compile error in foreach statement at line 15: Compile error in 
procedure invocation at line 25: Variable result is undefined.

because, as in the trivial initial example, the array results is out of 
scope at that point. I cant figure out how to write a termination 
expression that can access the values from within the iterate{} body.

So how do I pass the results of the set of simulation to a function like 
"done()" to check a termination condition?

My real code (below) hangs after the first set of simulations, which 
makes me think that it is (perhaps, in addition to the problem above) 
stumbling into an array closing issue. But I think that a solution to 
the mock simulation example above may help me solve the actual problem.

Thanks,

Mike
--

The current "real" code is below (lines are wrapped; file is 
http://www.ci.uchicago.edu/~wilde/oops8.swift).


type Fasta;
type SecSeq;
type PDB;
type RetValFile;

type OOPSSecStr;
type OOPSLog;
type OOPSEnergy;
type OOPSpdt;
type OOPSrmsd;
type OOPSLibrary;

app (OOPSpdt pdt, OOPSEnergy e, OOPSLibrary lib, OOPSSecStr secstr, 
OOPSrmsd rmsd, OOPSLog log)
     runoops (Fasta fasta, SecSeq secseq, PDB pdb, int jobnum, string 
cfgParams[])
{ simoops @fasta @secseq @pdb @pdt @rmsd jobnum cfgParams stdout=@log; }

app (RetValFile rv) roundDoneTest ( OOPSpdt pdts[] )
{ testifdone @filename(pdts[0]) @rv; }

(int rc) roundDone (OOPSpdt pdts[] ) { RetValFile rv = 
roundDoneTest(pdts); rc = readData(@rv); }

string config [] = readData(@arg("params"));

int iter[]=[ 0 : @toint(@arg("niter","1")) - 1 ];
string protein[] = readData(@arg("plist"));

foreach p in protein {
     iterate i {
         OOPSpdt result[];
         foreach j in iter {

             // map inputs

             Fasta fasta 
<single_file_mapper;file=@strcat("input/fasta/",p,".fasta")>;
             SecSeq secseq 
<single_file_mapper;file=@strcat("input/secseq/",p,".secseq")>;
             PDB native 
<single_file_mapper;file=@strcat("input/native/",p,".pdb")>;

             // map outputs

             string out=@strcat("output/r",i,"/",p,".",j);

             OOPSpdt pdt <single_file_mapper;file=@strcat(out,".pdt")>;
             OOPSEnergy e <single_file_mapper;file=@strcat(out,".Energy")>;
             OOPSLibrary lib 
<single_file_mapper;file=@strcat(out,".Library")>;
             OOPSSecStr secstr 
<single_file_mapper;file=@strcat(out,".SecStr")>;
             OOPSrmsd rmsd <single_file_mapper;file=@strcat(out,".rmsd")>;
             OOPSLog log <single_file_mapper;file=@strcat(out,".log")>;

             // run oops

             ( pdt, e, lib, secstr, rmsd, log) = runoops( fasta, secseq, 
native, j, config );
             result[j] = pdt;
         }
     } until (roundDone(result) == 1);
}





From benc at hawaga.org.uk  Fri Feb 20 07:45:59 2009
From: benc at hawaga.org.uk (Ben Clifford)
Date: Fri, 20 Feb 2009 13:45:59 +0000 (GMT)
Subject: [Swift-user] Questions on use of iterate statement
In-Reply-To: <499EAB13.5050401@mcs.anl.gov>
References: <499EAB13.5050401@mcs.anl.gov>
Message-ID: <Pine.LNX.4.64.0902201334000.1293@dildano.hawaga.org.uk>


In many (all?) cases you should be able to declare an array outside of the 
iteration, with each element of the array holding the result of each step, 
like this:

int j[];
iterate i {
  j[i] = i * i;
  trace(j[i]);
} until (j[i]>15);

That was the main way in which I invisioned iterate being used when I 
implemented it, i think, so that you would map steps of your iteration 
using a single mapper declaration on j[] rather than inside the iteration 
body. That comes from imagining it as something like a functional 'unfold' 
expression (eg 
http://www.haskell.org/ghc/docs/latest/html/libraries/base/Data-List.html#9) 
where entire structures such as lists are constructed as a single 
expression.

It seems reasonable that body variables be made available to the 
termination expression, though, to support the style that you attempted, 
so I'll put that in the bugzilla as a language feature request.

-- 



From wilde at mcs.anl.gov  Fri Feb 20 09:14:34 2009
From: wilde at mcs.anl.gov (Michael Wilde)
Date: Fri, 20 Feb 2009 09:14:34 -0600
Subject: [Swift-user] Questions on use of iterate statement
In-Reply-To: <Pine.LNX.4.64.0902201334000.1293@dildano.hawaga.org.uk>
References: <499EAB13.5050401@mcs.anl.gov>
	<Pine.LNX.4.64.0902201334000.1293@dildano.hawaga.org.uk>
Message-ID: <499EC8DA.70304@mcs.anl.gov>

OK. I was cinsidering that, but I'll need to use a 2-d array as the 
simulation itself generates a vector of results.

I'll try this.

Thanks,

Mike

On 2/20/09 7:45 AM, Ben Clifford wrote:
> In many (all?) cases you should be able to declare an array outside of the 
> iteration, with each element of the array holding the result of each step, 
> like this:
> 
> int j[];
> iterate i {
>   j[i] = i * i;
>   trace(j[i]);
> } until (j[i]>15);
> 
> That was the main way in which I invisioned iterate being used when I 
> implemented it, i think, so that you would map steps of your iteration 
> using a single mapper declaration on j[] rather than inside the iteration 
> body. That comes from imagining it as something like a functional 'unfold' 
> expression (eg 
> http://www.haskell.org/ghc/docs/latest/html/libraries/base/Data-List.html#9) 
> where entire structures such as lists are constructed as a single 
> expression.
> 
> It seems reasonable that body variables be made available to the 
> termination expression, though, to support the style that you attempted, 
> so I'll put that in the bugzilla as a language feature request.
> 


From benc at hawaga.org.uk  Fri Feb 20 09:17:37 2009
From: benc at hawaga.org.uk (Ben Clifford)
Date: Fri, 20 Feb 2009 15:17:37 +0000 (GMT)
Subject: [Swift-user] Questions on use of iterate statement
In-Reply-To: <499EC8DA.70304@mcs.anl.gov>
References: <499EAB13.5050401@mcs.anl.gov>
	<Pine.LNX.4.64.0902201334000.1293@dildano.hawaga.org.uk>
	<499EC8DA.70304@mcs.anl.gov>
Message-ID: <Pine.LNX.4.64.0902201517140.23512@dildano.hawaga.org.uk>


On Fri, 20 Feb 2009, Michael Wilde wrote:

> OK. I was cinsidering that, but I'll need to use a 2-d array as the simulation
> itself generates a vector of results.

You should be able to have arrays of many dimensions.

-- 


From zvalim at gmail.com  Fri Feb 20 09:52:21 2009
From: zvalim at gmail.com (U.H)
Date: Fri, 20 Feb 2009 16:52:21 +0100
Subject: [Swift-user] using swift in an IP-impoverished environment
Message-ID: <32078b1e0902200752y59dff7e7p44c6788dfd4f001c@mail.gmail.com>

Dear Swift experts,

I want to run swift in my new location, which is an institution with a
relatively restricted IP allocation and a particular knack for
centralization. As a result, the entire computer network of my center
is behind a NAT/Firewall.

I am exploring options to run swift in this scenario but before
pursuing them further wanted to consult the collective experience.

1. Perhaps run via vpn. I can open a vpn connection from a linux
machine. That way I obtain a visible IP address. But, the linux client
vpn crashes often, which has to do with the microsoft vpn system
installed here.  And I am not certain there's a proper hostname
associated with the IP obtained via VPN, which, if I understand
correctly is a Globus requirement.

2. Run behind NAT/Firewall. I found a document describing client-side
reqs for this kind of situation here
(http://dev.globus.org/wiki/FirewallHowTo section called "Network
Address Translation (NAT)").
Does anyone have experience in running swift in this mode?

Best,
Uri


From benc at hawaga.org.uk  Fri Feb 20 09:59:48 2009
From: benc at hawaga.org.uk (Ben Clifford)
Date: Fri, 20 Feb 2009 15:59:48 +0000 (GMT)
Subject: [Swift-user] using swift in an IP-impoverished environment
In-Reply-To: <32078b1e0902200752y59dff7e7p44c6788dfd4f001c@mail.gmail.com>
References: <32078b1e0902200752y59dff7e7p44c6788dfd4f001c@mail.gmail.com>
Message-ID: <Pine.LNX.4.64.0902201555070.1293@dildano.hawaga.org.uk>


> I want to run swift in my new location, which is an institution with a
> relatively restricted IP allocation and a particular knack for
> centralization. As a result, the entire computer network of my center
> is behind a NAT/Firewall.

> I am exploring options to run swift in this scenario but before
> pursuing them further wanted to consult the collective experience.

Can you run swift on a machine that is behind the firewall? (eg log into a 
cluster head node and run there?)

> 1. Perhaps run via vpn. I can open a vpn connection from a linux 
> machine. That way I obtain a visible IP address. But, the linux client 
> vpn crashes often, which has to do with the microsoft vpn system 
> installed here.  And I am not certain there's a proper hostname 
> associated with the IP obtained via VPN, which, if I understand 
> correctly is a Globus requirement.

Having an unreliable connection will make Swift runs probably a bit 
unreliable, but I don't know how bad that will be.

You don't need a real hostname on your submitting machine. You might need 
to set an environment variable GLOBUS_HOSTNAME to your VPN IP address, 
like this in the shell:

  export GLOBUS_HOSTNAME=81.187.211.41

That is what I do to submit from my laptop when I have it in various 
different network locations (where 81.187.211.41 is replaced with whatever 
my IP happens to be at that time)

> 2. Run behind NAT/Firewall. I found a document describing client-side
> reqs for this kind of situation here
> (http://dev.globus.org/wiki/FirewallHowTo section called "Network
> Address Translation (NAT)").
> Does anyone have experience in running swift in this mode?

I've never run it from behind a NAT.

-- 


From hategan at mcs.anl.gov  Fri Feb 20 10:05:19 2009
From: hategan at mcs.anl.gov (Mihael Hategan)
Date: Fri, 20 Feb 2009 10:05:19 -0600
Subject: [Swift-user] using swift in an IP-impoverished environment
In-Reply-To: <Pine.LNX.4.64.0902201555070.1293@dildano.hawaga.org.uk>
References: <32078b1e0902200752y59dff7e7p44c6788dfd4f001c@mail.gmail.com>
	<Pine.LNX.4.64.0902201555070.1293@dildano.hawaga.org.uk>
Message-ID: <1235145919.5393.16.camel@localhost>

On Fri, 2009-02-20 at 15:59 +0000, Ben Clifford wrote:
> > 2. Run behind NAT/Firewall. I found a document describing client-side
> > reqs for this kind of situation here
> > (http://dev.globus.org/wiki/FirewallHowTo section called "Network
> > Address Translation (NAT)").
> > Does anyone have experience in running swift in this mode?
> 
> I've never run it from behind a NAT.
> 

I do that fairly often.

It involves forwarding a range of ports (a hundred of them or more) to
your "submit" machine, setting GLOBUS_TCP_PORT_RANGE to that range and
making sure that GLOBUS_HOSTNAME has your external IP address.



From fedorov at cs.wm.edu  Fri Feb 20 12:32:35 2009
From: fedorov at cs.wm.edu (Andriy Fedorov)
Date: Fri, 20 Feb 2009 13:32:35 -0500
Subject: [Swift-user] using swift in an IP-impoverished environment
Message-ID: <82f536810902201032r1fb41270t9451cd46d0c45057@mail.gmail.com>

> On Fri, 2009-02-20 at 15:59 +0000, Ben Clifford wrote:
>> > 2. Run behind NAT/Firewall. I found a document describing client-side
>> > reqs for this kind of situation here
>> > (http://dev.globus.org/wiki/FirewallHowTo section called "Network
>> > Address Translation (NAT)").
>> > Does anyone have experience in running swift in this mode?
>>
>> I've never run it from behind a NAT.
>>
>
> I do that fairly often.
>
> It involves forwarding a range of ports (a hundred of them or more) to
> your "submit" machine, setting GLOBUS_TCP_PORT_RANGE to that range and
> making sure that GLOBUS_HOSTNAME has your external IP address.
>

Mihael,

I am not familiar at all with NAT, but I have a similar configuration
of the network, with all organization hosts behind the firewall, and
with no control over firewall configuration. I thought it is not
possible to run swift client with full functionality supported in such
an environment. In my organization, in order to log to a host from
outside, I need to first ge authenticated with the gatekeeper host,
which will next allow me to log on an intranet host.

It seems to me unlikely that it is possible to configure a host behind
a firewall in such a way that allows direct connection to that host
avoiding the firewall. Seems like a security breach... Unless NAT has
to be configured on the firewall host, which is not an option for me.

If I was wrong, could you give some more details on how something like
this can be configured with NAT or anything else?

Thanks

Andriy Fedorov


From benc at hawaga.org.uk  Fri Feb 20 12:38:53 2009
From: benc at hawaga.org.uk (Ben Clifford)
Date: Fri, 20 Feb 2009 18:38:53 +0000 (GMT)
Subject: [Swift-user] using swift in an IP-impoverished environment
In-Reply-To: <82f536810902201032r1fb41270t9451cd46d0c45057@mail.gmail.com>
References: <82f536810902201032r1fb41270t9451cd46d0c45057@mail.gmail.com>
Message-ID: <Pine.LNX.4.64.0902201837170.1293@dildano.hawaga.org.uk>


On Fri, 20 Feb 2009, Andriy Fedorov wrote:

> avoiding the firewall. Seems like a security breach... Unless NAT has to 
> be configured on the firewall host, which is not an option for me.

Mihael's comment:

 | It involves forwarding a range of ports (a hundred of them or more)

I think pretty much needs to be configured on the NAT/firewall machine 
(which I suspect in his case is the linksys router sitting in the corner 
of his living room, and to which he has admin privs).

-- 


From fedorov at cs.wm.edu  Fri Feb 20 12:54:32 2009
From: fedorov at cs.wm.edu (Andriy Fedorov)
Date: Fri, 20 Feb 2009 13:54:32 -0500
Subject: [Swift-user] using swift in an IP-impoverished environment
In-Reply-To: <Pine.LNX.4.64.0902201837170.1293@dildano.hawaga.org.uk>
References: <82f536810902201032r1fb41270t9451cd46d0c45057@mail.gmail.com>
	<Pine.LNX.4.64.0902201837170.1293@dildano.hawaga.org.uk>
Message-ID: <82f536810902201054g68f835bcq891cb74c33aceebf@mail.gmail.com>

On Fri, Feb 20, 2009 at 1:38 PM, Ben Clifford <benc at hawaga.org.uk> wrote:
>
> On Fri, 20 Feb 2009, Andriy Fedorov wrote:
>
>> avoiding the firewall. Seems like a security breach... Unless NAT has to
>> be configured on the firewall host, which is not an option for me.
>
> Mihael's comment:
>
>  | It involves forwarding a range of ports (a hundred of them or more)
>
> I think pretty much needs to be configured on the NAT/firewall machine
> (which I suspect in his case is the linksys router sitting in the corner
> of his living room, and to which he has admin privs).
>

That's what I thought -- not an option for me.

I found a report about using ssh tunneling to solve this
http://www.hpl.hp.com/techreports/2002/HPL-2002-278.pdf

but I didn't have time/need to really study this. The report is about
GT2. But I am skeptical this solves the problem. I've seen firewall
difficulties cited as one of the major problems related to using Grids
in practice. If it was easy to solve, it would be solved by now, with
the general solution google-able...


> --
>


From hategan at mcs.anl.gov  Fri Feb 20 13:36:09 2009
From: hategan at mcs.anl.gov (Mihael Hategan)
Date: Fri, 20 Feb 2009 13:36:09 -0600 (CST)
Subject: [Swift-user] using swift in an IP-impoverished environment
In-Reply-To: <82f536810902201032r1fb41270t9451cd46d0c45057@mail.gmail.com>
Message-ID: <31341573.363201235158569558.JavaMail.root@zimbra>


----- Andriy Fedorov <fedorov at cs.wm.edu> wrote:
> > On Fri, 2009-02-20 at 15:59 +0000, Ben Clifford wrote:
> >> > 2. Run behind NAT/Firewall. I found a document describing client-side
> >> > reqs for this kind of situation here
> >> > (http://dev.globus.org/wiki/FirewallHowTo section called "Network
> >> > Address Translation (NAT)").
> >> > Does anyone have experience in running swift in this mode?
> >>
> >> I've never run it from behind a NAT.
> >>
> >
> > I do that fairly often.
> >
> > It involves forwarding a range of ports (a hundred of them or more) to
> > your "submit" machine, setting GLOBUS_TCP_PORT_RANGE to that range and
> > making sure that GLOBUS_HOSTNAME has your external IP address.
> >
> 
> Mihael,
> 
> I am not familiar at all with NAT, but I have a similar configuration
> of the network, with all organization hosts behind the firewall, and
> with no control over firewall configuration.

That may be a problem.

There are two scenarios I have successfully used:
1. i2u2.org lives behind the Argonne firewall. We have ports 50000 to 
50100 (I think) open for the server that runs swift. This involves no
NAT
2. At home, I have a router with firewall and NAT. I routinely run swift
from my laptop, which has a private IP address. I convince the router
to forward all requests on ports 50000 to 50100 to my internal laptop
IP, and I set the globus callback IP to the external (modem) IP. I have
been employing this scheme for years (before Swift with cog and globus).
However, you need the ability to at least ask for a range of ports to 
be forwarded to your Swift machine automatically.

> I thought it is not
> possible to run swift client with full functionality supported in such
> an environment. In my organization, in order to log to a host from
> outside, I need to first ge authenticated with the gatekeeper host,
> which will next allow me to log on an intranet host.
> 
> It seems to me unlikely that it is possible to configure a host behind
> a firewall in such a way that allows direct connection to that host
> avoiding the firewall. Seems like a security breach... 

It is a security breach as much as explicitly allowing swift to function
is a security breach.

> Unless NAT has
> to be configured on the firewall host, which is not an option for me.
> 
> If I was wrong, could you give some more details on how something like
> this can be configured with NAT or anything else?

There is also SSH tunneling. You could create an SSH tunnel to an outside
machine with its own IP and no firewall (on a range of ports) and forward
that range of ports to your swift machine. Conceptually, it's the same
scheme, except instead of control over the firewall and port forwarding, 
you need (limited) control of an outside machine. But then you could run
swift on the external machine to begin with.

As for this potentially being labeled a "security breach", I can probably
say that changing the odds of a security breach occurring (and they are never
zero) is not the same as a security breach occurring.


From hategan at mcs.anl.gov  Fri Feb 20 13:37:22 2009
From: hategan at mcs.anl.gov (Mihael Hategan)
Date: Fri, 20 Feb 2009 13:37:22 -0600 (CST)
Subject: [Swift-user] using swift in an IP-impoverished environment
In-Reply-To: <Pine.LNX.4.64.0902201837170.1293@dildano.hawaga.org.uk>
Message-ID: <23284427.363301235158642388.JavaMail.root@zimbra>

----- Ben Clifford <benc at hawaga.org.uk> wrote:
> 
> On Fri, 20 Feb 2009, Andriy Fedorov wrote:
> 
> > avoiding the firewall. Seems like a security breach... Unless NAT has to 
> > be configured on the firewall host, which is not an option for me.
> 
> Mihael's comment:
> 
>  | It involves forwarding a range of ports (a hundred of them or more)
> 
> I think pretty much needs to be configured on the NAT/firewall machine 
> (which I suspect in his case is the linksys router sitting in the corner 
> of his living room, and to which he has admin privs).

:)

That or I know the admin really well.


From hategan at mcs.anl.gov  Fri Feb 20 13:39:41 2009
From: hategan at mcs.anl.gov (Mihael Hategan)
Date: Fri, 20 Feb 2009 13:39:41 -0600 (CST)
Subject: [Swift-user] using swift in an IP-impoverished environment
In-Reply-To: <82f536810902201054g68f835bcq891cb74c33aceebf@mail.gmail.com>
Message-ID: <14783524.363481235158781930.JavaMail.root@zimbra>


----- Andriy Fedorov <fedorov at cs.wm.edu> wrote:
> If it was easy to solve, it would be solved by now, with
> the general solution google-able...
> 

Not necessarily. 


From zhengxiongh at uchicago.edu  Fri Feb 20 14:37:57 2009
From: zhengxiongh at uchicago.edu (Zhengxiong Hou)
Date: Fri, 20 Feb 2009 14:37:57 -0600
Subject: [Swift-user] Re: oops failures on OSG
In-Reply-To: <499F0E2A.8040503@uchicago.edu>
References: <499DD953.6000607@uchicago.edu> <499E21A9.5040704@uchicago.edu>
	<499E2302.60606@uchicago.edu> <499EE378.5060106@uchicago.edu>
	<499F0E2A.8040503@uchicago.edu>
Message-ID: <499F14A5.8030104@uchicago.edu>

OK.
BTW,  Please send me the command to run oops, I'm going to run it on OSG VO.
            swift -tc.file tc.data -sites.file sites.xml oops7.swift 
(and -params=???)
> Should we ask the swift user group?
>
> Zhengxiong Hou wrote:
>>   Yes, I also found it from the log file.
>>   Since it failed on UCSDT2-B, Swift should retried it, but it was 
>> still failed, and I 'm not exactly clear why all of the rest jobs 
>> were failed. So, it is swift staff.
>>   Maybe, we had better use the "replication" mechanism of Swift.
>>
>> Glen Hocky wrote:
>>> Progress:  Submitted:27 Active:22 Finished successfully:51
>>> Progress:  Submitted:27 Active:22 Finished successfully:51
>>> Progress:  Submitted:27 Active:22 Finished successfully:51
>>> Progress:  Submitted:27 Active:22 Finished successfully:51
>>> Progress:  Submitted:27 Active:22 Finished successfully:51
>>> Progress:  Submitted:27 Active:21 Finished successfully:52
>>> Progress:  Submitted:27 Active:21 Finished successfully:52
>>> Progress:  Submitted:27 Active:21 Finished successfully:52
>>> Progress:  Submitted:27 Active:21 Finished successfully:52
>>> Progress:  Submitted:27 Active:21 Finished successfully:52
>>> Progress:  Submitted:27 Active:21 Finished successfully:52
>>> Progress:  Submitted:27 Active:21 Finished successfully:52
>>> Progress:  Submitted:26 Active:22 Finished successfully:52
>>> Progress:  Submitted:26 Active:22 Finished successfully:52
>>> Failed to transfer wrapper log from 
>>> oops7-20090219-1827-6gqbuh81/info/o on UCSDT2-B
>>> Progress:  Submitted:26 Active:21 Failed:1 Finished successfully:52
>>> Execution failed:
>>>        Exception in runoops:
>>> Arguments: [input/fasta/T1di2.fasta, input/secseq/T1di2.secseq, 
>>> input/native/T1di2.pdb, output/T1di2.26.pdt, output/T1di2.26.rmsd, 
>>> 26, TEMP UPDATE INTERVAL = 10, SMOOTH DEVIATION COEFFICIENT = 0.80001]
>>> Host: UCSDT2-B
>>> Directory: oops7-20090219-1827-6gqbuh81/jobs/o/runoops-on95pu6j
>>> stderr.txt:
>>>
>>> stdout.txt:
>>>
>>>
>>>
>>> Zhengxiong Hou wrote:
>>>> Hi Glen,
>>>>     For that 100 jobs run, I checked the results and turned them 
>>>> into plots, not more than 60 jobs were finished successfully.
>>>>     So, we should try to find the reasons for the failures.
>>>>
>>>> Thanks,
>>>> Zhengxiong
>>>
>>
>



From zhengxiongh at uchicago.edu  Fri Feb 20 15:56:15 2009
From: zhengxiongh at uchicago.edu (Zhengxiong Hou)
Date: Fri, 20 Feb 2009 15:56:15 -0600
Subject: [Swift-user] oops failures on OSG
In-Reply-To: <499E2302.60606@uchicago.edu>
References: <499DD953.6000607@uchicago.edu> <499E21A9.5040704@uchicago.edu>
	<499E2302.60606@uchicago.edu>
Message-ID: <499F26FF.6020105@uchicago.edu>

Hi,
    We encountered some failures when running oops within Engage VO or 
OSG VO on OSG by swift. Please have a look.
    The logs ( oops7-20090219-1827-6gqbuh81.0.rlog,  
oops7-20090219-1827-6gqbuh81.log, oops7-20090219-1827-6gqbuh81.d) have 
been saved to Ben's swift-logs directory.
    And the following is the standard output information when running it 
within Engage VO.
> Progress:  Submitted:27 Active:22 Finished successfully:51
> Progress:  Submitted:27 Active:22 Finished successfully:51
> Progress:  Submitted:27 Active:22 Finished successfully:51
> Progress:  Submitted:27 Active:22 Finished successfully:51
> Progress:  Submitted:27 Active:22 Finished successfully:51
> Progress:  Submitted:27 Active:21 Finished successfully:52
> Progress:  Submitted:27 Active:21 Finished successfully:52
> Progress:  Submitted:27 Active:21 Finished successfully:52
> Progress:  Submitted:27 Active:21 Finished successfully:52
> Progress:  Submitted:27 Active:21 Finished successfully:52
> Progress:  Submitted:27 Active:21 Finished successfully:52
> Progress:  Submitted:27 Active:21 Finished successfully:52
> Progress:  Submitted:26 Active:22 Finished successfully:52
> Progress:  Submitted:26 Active:22 Finished successfully:52
> Failed to transfer wrapper log from 
> oops7-20090219-1827-6gqbuh81/info/o on UCSDT2-B
> Progress:  Submitted:26 Active:21 Failed:1 Finished successfully:52
> Execution failed:
>        Exception in runoops:
> Arguments: [input/fasta/T1di2.fasta, input/secseq/T1di2.secseq, 
> input/native/T1di2.pdb, output/T1di2.26.pdt, output/T1di2.26.rmsd, 26, 
> TEMP UPDATE INTERVAL = 10, SMOOTH DEVIATION COEFFICIENT = 0.80001]
> Host: UCSDT2-B
> Directory: oops7-20090219-1827-6gqbuh81/jobs/o/runoops-on95pu6j
> stderr.txt:
>
> stdout.txt:
>

Thanks!
Zhengxiong


From hategan at mcs.anl.gov  Fri Feb 20 16:10:30 2009
From: hategan at mcs.anl.gov (Mihael Hategan)
Date: Fri, 20 Feb 2009 16:10:30 -0600 (CST)
Subject: [Swift-user] oops failures on OSG
In-Reply-To: <499F26FF.6020105@uchicago.edu>
Message-ID: <27162045.392121235167830246.JavaMail.root@zimbra>

I'm seeing this in the logs:
No status file was found. Check the shared filesystem on UCSDT2-B

For the short term I suggest commenting out that site in the
sites file.

Mihael

----- Zhengxiong Hou <zhengxiongh at uchicago.edu> wrote:
> Hi,
>     We encountered some failures when running oops within Engage VO or 
> OSG VO on OSG by swift. Please have a look.
>     The logs ( oops7-20090219-1827-6gqbuh81.0.rlog,  
> oops7-20090219-1827-6gqbuh81.log, oops7-20090219-1827-6gqbuh81.d) have 
> been saved to Ben's swift-logs directory.
>     And the following is the standard output information when running it 
> within Engage VO.
> > Progress:  Submitted:27 Active:22 Finished successfully:51
> > Progress:  Submitted:27 Active:22 Finished successfully:51
> > Progress:  Submitted:27 Active:22 Finished successfully:51
> > Progress:  Submitted:27 Active:22 Finished successfully:51
> > Progress:  Submitted:27 Active:22 Finished successfully:51
> > Progress:  Submitted:27 Active:21 Finished successfully:52
> > Progress:  Submitted:27 Active:21 Finished successfully:52
> > Progress:  Submitted:27 Active:21 Finished successfully:52
> > Progress:  Submitted:27 Active:21 Finished successfully:52
> > Progress:  Submitted:27 Active:21 Finished successfully:52
> > Progress:  Submitted:27 Active:21 Finished successfully:52
> > Progress:  Submitted:27 Active:21 Finished successfully:52
> > Progress:  Submitted:26 Active:22 Finished successfully:52
> > Progress:  Submitted:26 Active:22 Finished successfully:52
> > Failed to transfer wrapper log from 
> > oops7-20090219-1827-6gqbuh81/info/o on UCSDT2-B
> > Progress:  Submitted:26 Active:21 Failed:1 Finished successfully:52
> > Execution failed:
> >        Exception in runoops:
> > Arguments: [input/fasta/T1di2.fasta, input/secseq/T1di2.secseq, 
> > input/native/T1di2.pdb, output/T1di2.26.pdt, output/T1di2.26.rmsd, 26, 
> > TEMP UPDATE INTERVAL = 10, SMOOTH DEVIATION COEFFICIENT = 0.80001]
> > Host: UCSDT2-B
> > Directory: oops7-20090219-1827-6gqbuh81/jobs/o/runoops-on95pu6j
> > stderr.txt:
> >
> > stdout.txt:
> >
> 
> Thanks!
> Zhengxiong
> _______________________________________________
> Swift-user mailing list
> Swift-user at ci.uchicago.edu
> http://mail.ci.uchicago.edu/mailman/listinfo/swift-user



From rynge at renci.org  Fri Feb 20 17:59:38 2009
From: rynge at renci.org (Mats Rynge)
Date: Fri, 20 Feb 2009 18:59:38 -0500
Subject: [Swift-user] oops failures on OSG
In-Reply-To: <27162045.392121235167830246.JavaMail.root@zimbra>
References: <499F26FF.6020105@uchicago.edu>
	<27162045.392121235167830246.JavaMail.root@zimbra>
Message-ID: <20090220235938.GA8368@rynge.europa.renci.org>

On Fri, Feb 20, 2009 at 04:10:30PM -0600, Mihael Hategan wrote:
> I'm seeing this in the logs:
> No status file was found. Check the shared filesystem on UCSDT2-B
> 

This reminds me to bring up an issue other users have had on UCSDT2 and
similar sites on OSG. UCSDT2 and UCSDT2-B are two interfaces to the
same cluster (shared filesystem is the same). If you give those as two
separate sites to Swift, will Swift be confused about the data caching?
That is, is the data caching directory named the same on all sites in a
run?

There are other clusters on OSG with multiple gatekeepers.

-- 
Mats Rynge
Renaissance Computing Institute <http://www.renci.org>


From hategan at mcs.anl.gov  Fri Feb 20 18:15:51 2009
From: hategan at mcs.anl.gov (Mihael Hategan)
Date: Fri, 20 Feb 2009 18:15:51 -0600 (CST)
Subject: [Swift-user] oops failures on OSG
In-Reply-To: <20090220235938.GA8368@rynge.europa.renci.org>
Message-ID: <28696812.400361235175351843.JavaMail.root@zimbra>


----- Mats Rynge <rynge at renci.org> wrote:
> On Fri, Feb 20, 2009 at 04:10:30PM -0600, Mihael Hategan wrote:
> > I'm seeing this in the logs:
> > No status file was found. Check the shared filesystem on UCSDT2-B
> > 
> 
> This reminds me to bring up an issue other users have had on UCSDT2 and
> similar sites on OSG. UCSDT2 and UCSDT2-B are two interfaces to the
> same cluster (shared filesystem is the same). If you give those as two
> separate sites to Swift, will Swift be confused about the data caching?
> That is, is the data caching directory named the same on all sites in a
> run?

It's likely that files in the shared directory will get messed up.

I can see no benefit in swift seeing one site as two, so I would recommend
against doing this in general.


From rynge at renci.org  Fri Feb 20 20:14:38 2009
From: rynge at renci.org (Mats Rynge)
Date: Fri, 20 Feb 2009 21:14:38 -0500
Subject: [Swift-user] oops failures on OSG
In-Reply-To: <28696812.400361235175351843.JavaMail.root@zimbra>
References: <20090220235938.GA8368@rynge.europa.renci.org>
	<28696812.400361235175351843.JavaMail.root@zimbra>
Message-ID: <20090221021438.GA8716@rynge.europa.renci.org>

On Fri, Feb 20, 2009 at 06:15:51PM -0600, Mihael Hategan wrote:
> 
> ----- Mats Rynge <rynge at renci.org> wrote:
> > On Fri, Feb 20, 2009 at 04:10:30PM -0600, Mihael Hategan wrote:
> > > I'm seeing this in the logs:
> > > No status file was found. Check the shared filesystem on UCSDT2-B
> > > 
> > 
> > This reminds me to bring up an issue other users have had on UCSDT2 and
> > similar sites on OSG. UCSDT2 and UCSDT2-B are two interfaces to the
> > same cluster (shared filesystem is the same). If you give those as two
> > separate sites to Swift, will Swift be confused about the data caching?
> > That is, is the data caching directory named the same on all sites in a
> > run?
> 
> It's likely that files in the shared directory will get messed up.
> 
> I can see no benefit in swift seeing one site as two, so I would recommend
> against doing this in general.

OSG is not going to stop doing this. The benefits for the site are the
ability to spread the load on multiple gatekeepers, and the ability
to do maintenance on one node and still have the resource available
with the other one.

One fix could be to add something site specific to the path of data
cache directory. That would mean duplicate copies of the data on some
sites, but if that is how the sites are advertised, I think that is
acceptable.

For users under the OSG Engagement VO, the problematic sites are:

UCSDT2          <->  UCSDT2-B
BNL_ATLAS_1     <->  BNL_ATLAS_2
Purdue-RCAC     <->  Purdue-Steele
FNAL_FERMIGRID  <->  FNAL_GPGRID_1

In your site catalog, make sure you only have one of the sites on each
line above.

-- 
Mats Rynge
Renaissance Computing Institute <http://www.renci.org>


From hategan at mcs.anl.gov  Fri Feb 20 20:43:55 2009
From: hategan at mcs.anl.gov (Mihael Hategan)
Date: Fri, 20 Feb 2009 20:43:55 -0600 (CST)
Subject: [Swift-user] oops failures on OSG
In-Reply-To: <20090221021438.GA8716@rynge.europa.renci.org>
Message-ID: <6816313.401881235184235427.JavaMail.root@zimbra>


----- Mats Rynge <rynge at renci.org> wrote:
> > I can see no benefit in swift seeing one site as two, so I would recommend
> > against doing this in general.
> 
> OSG is not going to stop doing this. The benefits for the site are the
> ability to spread the load on multiple gatekeepers, and the ability
> to do maintenance on one node and still have the resource available
> with the other one.

I didn't mean it from an OSG perspective, but from a Swift perspective.
You are also offering a benefit, so I must reconsider what I said.

> 
> One fix could be to add something site specific to the path of data
> cache directory. That would mean duplicate copies of the data on some
> sites, but if that is how the sites are advertised, I think that is
> acceptable.

This, I think, can currently be done manually by specifying a different 
work directory in sites.xml, but swift won't check for such conflicts.



From rynge at renci.org  Fri Feb 20 21:27:03 2009
From: rynge at renci.org (Mats Rynge)
Date: Fri, 20 Feb 2009 22:27:03 -0500
Subject: [Swift-user] oops failures on OSG
In-Reply-To: <6816313.401881235184235427.JavaMail.root@zimbra>
References: <20090221021438.GA8716@rynge.europa.renci.org>
	<6816313.401881235184235427.JavaMail.root@zimbra>
Message-ID: <20090221032703.GA8983@rynge.europa.renci.org>

On Fri, Feb 20, 2009 at 08:43:55PM -0600, Mihael Hategan wrote:
> > One fix could be to add something site specific to the path of data
> > cache directory. That would mean duplicate copies of the data on some
> > sites, but if that is how the sites are advertised, I think that is
> > acceptable.
> 
> This, I think, can currently be done manually by specifying a different 
> work directory in sites.xml, but swift won't check for such conflicts.

Good point! Attached is a fix for swift-osg-ress-site-catalog

-- 
Mats Rynge
Renaissance Computing Institute <http://www.renci.org>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: swift-sitename.patch
Type: text/x-diff
Size: 560 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/swift-user/attachments/20090220/cbf27089/attachment.patch>

From zhengxiongh at uchicago.edu  Fri Feb 20 22:14:50 2009
From: zhengxiongh at uchicago.edu (Zhengxiong Hou)
Date: Fri, 20 Feb 2009 22:14:50 -0600
Subject: [Swift-user] oops failures on OSG
In-Reply-To: <20090221032703.GA8983@rynge.europa.renci.org>
References: <20090221021438.GA8716@rynge.europa.renci.org>
	<6816313.401881235184235427.JavaMail.root@zimbra>
	<20090221032703.GA8983@rynge.europa.renci.org>
Message-ID: <499F7FBA.5060506@uchicago.edu>

Mats Rynge wrote:
> On Fri, Feb 20, 2009 at 08:43:55PM -0600, Mihael Hategan wrote:
>   
>>> One fix could be to add something site specific to the path of data
>>> cache directory. That would mean duplicate copies of the data on some
>>> sites, but if that is how the sites are advertised, I think that is
>>> acceptable.
>>>       
>> This, I think, can currently be done manually by specifying a different 
>> work directory in sites.xml, but swift won't check for such conflicts.
>>     
>
> Good point! Attached is a fix for swift-osg-ress-site-catalo
Yes! This is a good one for the clusters which have one shared file 
system for multiple gatekeepers.

But I'm sorry to say that this might not be the failure reason for " 
UCSDT2-B ", because we did not use both of the "UCSDT2" and "UCSDT2-B" 
concurrently.

BTW, where is "bin/swift-osg-ress-site-catalog"?  I'm interested in it.

Thanks much!
Zhengxiong



From rynge at renci.org  Fri Feb 20 22:27:12 2009
From: rynge at renci.org (Mats Rynge)
Date: Fri, 20 Feb 2009 23:27:12 -0500
Subject: [Swift-user] oops failures on OSG
In-Reply-To: <499F7FBA.5060506@uchicago.edu>
References: <20090221021438.GA8716@rynge.europa.renci.org>
	<6816313.401881235184235427.JavaMail.root@zimbra>
	<20090221032703.GA8983@rynge.europa.renci.org>
	<499F7FBA.5060506@uchicago.edu>
Message-ID: <20090221042712.GA9194@rynge.europa.renci.org>

On Fri, Feb 20, 2009 at 10:14:50PM -0600, Zhengxiong Hou wrote:
>
> BTW, where is "bin/swift-osg-ress-site-catalog"?  I'm interested in it.

It will be included in the next Swift release. For now, you can get it
from the Subversion repository.

Example usage:

./swift-osg-ress-site-catalog --vo=engage --out=sites.xml


-- 
Mats Rynge
Renaissance Computing Institute <http://www.renci.org>


From hategan at mcs.anl.gov  Fri Feb 20 23:22:00 2009
From: hategan at mcs.anl.gov (Mihael Hategan)
Date: Fri, 20 Feb 2009 23:22:00 -0600
Subject: [Swift-user] oops failures on OSG
In-Reply-To: <20090221032703.GA8983@rynge.europa.renci.org>
References: <20090221021438.GA8716@rynge.europa.renci.org>
	<6816313.401881235184235427.JavaMail.root@zimbra>
	<20090221032703.GA8983@rynge.europa.renci.org>
Message-ID: <1235193720.20264.0.camel@localhost>

Committed in r2580

On Fri, 2009-02-20 at 22:27 -0500, Mats Rynge wrote:
> On Fri, Feb 20, 2009 at 08:43:55PM -0600, Mihael Hategan wrote:
> > > One fix could be to add something site specific to the path of data
> > > cache directory. That would mean duplicate copies of the data on some
> > > sites, but if that is how the sites are advertised, I think that is
> > > acceptable.
> > 
> > This, I think, can currently be done manually by specifying a different 
> > work directory in sites.xml, but swift won't check for such conflicts.
> 
> Good point! Attached is a fix for swift-osg-ress-site-catalog
> 



From benc at hawaga.org.uk  Sat Feb 21 03:00:47 2009
From: benc at hawaga.org.uk (Ben Clifford)
Date: Sat, 21 Feb 2009 09:00:47 +0000 (GMT)
Subject: [Swift-user] oops failures on OSG
In-Reply-To: <499F7FBA.5060506@uchicago.edu>
References: <20090221021438.GA8716@rynge.europa.renci.org>
	<6816313.401881235184235427.JavaMail.root@zimbra>
	<20090221032703.GA8983@rynge.europa.renci.org>
	<499F7FBA.5060506@uchicago.edu>
Message-ID: <Pine.LNX.4.64.0902210859590.1293@dildano.hawaga.org.uk>


On Fri, 20 Feb 2009, Zhengxiong Hou wrote:

> BTW, where is "bin/swift-osg-ress-site-catalog"?  I'm interested in it.

Build Swift from SVN source and you should see it there.

-- 



From benc at hawaga.org.uk  Sat Feb 21 03:15:28 2009
From: benc at hawaga.org.uk (Ben Clifford)
Date: Sat, 21 Feb 2009 09:15:28 +0000 (GMT)
Subject: [Swift-user] oops failures on OSG
In-Reply-To: <28696812.400361235175351843.JavaMail.root@zimbra>
References: <28696812.400361235175351843.JavaMail.root@zimbra>
Message-ID: <Pine.LNX.4.64.0902210913150.1293@dildano.hawaga.org.uk>


On Fri, 20 Feb 2009, Mihael Hategan wrote:

> I can see no benefit in swift seeing one site as two, so I would recommend
> against doing this in general.

There's a similar situation where you have multiple sites sharing file 
systems (for example, at CI) - using (for example) /home/benc/tmp as a 
shared directory on the >1 clusters that have access to CI home directory 
space will cause a similar problem if they're used simultaneously, I 
think.

-- 



From aespinosa at cs.uchicago.edu  Tue Feb 24 17:14:33 2009
From: aespinosa at cs.uchicago.edu (Allan Espinosa)
Date: Tue, 24 Feb 2009 17:14:33 -0600
Subject: [Swift-user] extern data type undefined.
Message-ID: <50b07b4b0902241514h72d2a694l526be144012f09c3@mail.gmail.com>

Hi,

I am trying to use the "extern" data type as documented in the swift
manual.  for some reason, the parser does not see this type as already
defined:

~/swift/vdsk/bin/swift -tc.file tc.data manysmall.swift

Could not start execution.
        Type extern is not defined.

Swift script:
type data;

app (extern o) dd(string input) {
  echo "if=" @input "of=/dev/null" "bs=128k";
}

data input<ext;exec="mapper.sh">;

foreach x in input {
  extern o;
  o = dd(@x);
}


-Allan


From aespinosa at cs.uchicago.edu  Tue Feb 24 17:31:37 2009
From: aespinosa at cs.uchicago.edu (Allan Espinosa)
Date: Tue, 24 Feb 2009 17:31:37 -0600
Subject: [Swift-user] Re: extern data type undefined.
In-Reply-To: <50b07b4b0902241514h72d2a694l526be144012f09c3@mail.gmail.com>
References: <50b07b4b0902241514h72d2a694l526be144012f09c3@mail.gmail.com>
Message-ID: <50b07b4b0902241531n72755d7eh2ffabb21b58756d6@mail.gmail.com>

I looked in src//tests/language-behaviour/087-external-dataset.swift
it is actually the "external" data type. the script now works.

so there's a typo in the documentation.

-Allan

On Tue, Feb 24, 2009 at 5:14 PM, Allan Espinosa
<aespinosa at cs.uchicago.edu> wrote:
> Hi,
>
> I am trying to use the "extern" data type as documented in the swift
> manual.  for some reason, the parser does not see this type as already
> defined:
>
> ~/swift/vdsk/bin/swift -tc.file tc.data manysmall.swift
>
> Could not start execution.
>        Type extern is not defined.
>
> Swift script:
> type data;
>
> app (extern o) dd(string input) {
>  echo "if=" @input "of=/dev/null" "bs=128k";
> }
>
> data input<ext;exec="mapper.sh">;
>
> foreach x in input {
>  extern o;
>  o = dd(@x);
> }
>
>
> -Allan
>



-- 
Allan M. Espinosa <http://allan.88-mph.net/blog>
PhD student, Computer Science
University of Chicago <http://people.cs.uchicago.edu/~aespinosa>


From benc at hawaga.org.uk  Wed Feb 25 03:35:52 2009
From: benc at hawaga.org.uk (Ben Clifford)
Date: Wed, 25 Feb 2009 09:35:52 +0000 (GMT)
Subject: [Swift-user] Re: extern data type undefined.
In-Reply-To: <50b07b4b0902241531n72755d7eh2ffabb21b58756d6@mail.gmail.com>
References: <50b07b4b0902241514h72d2a694l526be144012f09c3@mail.gmail.com>
	<50b07b4b0902241531n72755d7eh2ffabb21b58756d6@mail.gmail.com>
Message-ID: <Pine.LNX.4.64.0902250932280.1293@dildano.hawaga.org.uk>


On Tue, 24 Feb 2009, Allan Espinosa wrote:

> so there's a typo in the documentation.

ok, I have a fix for that in my commit queue.

-- 


From wilde at mcs.anl.gov  Wed Feb 25 08:17:45 2009
From: wilde at mcs.anl.gov (Michael Wilde)
Date: Wed, 25 Feb 2009 08:17:45 -0600
Subject: [Swift-user] assigning file variables
Message-ID: <49A55309.4050100@mcs.anl.gov>

I was surprised that while I can assign complete arrays from one var to 
another, I can not assign values of type "file".

This script:

---

type file;

app (file out) echo (string s) { echo s stdout=@out; }

file result;
file f <"echo.out">;

f = echo("output");
result = f;

---

gives:

---

Swift svn swift-r2552 cog-r2303

RunID: 20090225-0810-sz60al30
Progress:
Execution failed:
         java.lang.RuntimeException: Deep non-array structure copying 
not implemented, when trying to copy 
org.griphyn.vdl.mapping.RootDataNode identifier 
tag:benc at ci.uchicago.edu,2008:swift:dataset:20090225-0810-tibqd6aa:720000000003 
type file with no value at dataset=f (closed)

---

The reason I'd like to do such assignments (in my current script) is to 
use a single_file_mapper inside iterate and foreach loops to map a file 
to a specific name pattern (eg one not obtainable with anything but an 
ext mapper) and then assign the file object to an array element for 
further processing later in the script.

Is it likely that the above restriction on file assignment will be 
lifted in a future release, or should users code as if files can not be 
assigned?

Related question: I see that assignment of entire arrays works, which is 
very useful. The messsage above implies that limited kinds of 
"structure" assignments work, and others dont. Whats works currently, 
whats planned, and whats not likely to be implemented?





From benc at hawaga.org.uk  Wed Feb 25 09:32:27 2009
From: benc at hawaga.org.uk (Ben Clifford)
Date: Wed, 25 Feb 2009 15:32:27 +0000 (GMT)
Subject: [Swift-user] assigning file variables
In-Reply-To: <49A55309.4050100@mcs.anl.gov>
References: <49A55309.4050100@mcs.anl.gov>
Message-ID: <Pine.LNX.4.64.0902251521230.1293@dildano.hawaga.org.uk>


The semantics of file assignments have been discussed on and of for some 
years.

Those discussions have left two different ideas floating around in my 
head:

i) = as copy

 The code file a <"foo">; file b<"bar">; a = b; behaves something like cp 
bar foo.

Thats pretty straightforward, bu to implement and understandt unclear to 
me if thats the right thing to do in many cases - for example, when 
assigning into an array of files, you would end up with the members of the 
array still being mapped to the filenames that the mapper associated with 
the array gives, not the filenames of the rhs file in the assignment.

Its not clear to me if this behaviour, for example, would be useful in the 
use case you specify. Why do you want to construct the filenames in a 
difficult way? Is it because you want them to end up with that filename, 
or because your application wants to generate them that way? In the 
former, this model doesn't work, though in the latter, if you don't care 
what they end up being called, thats fine.

ii) assignment-as-aliasing

file a; file b<"bar">; a= b turns a into an alias for b, mapped still to 
the file "bar"; no copying takes place.

In your case, you'd then end up with an array of aliases to files that are 
mapped by a bunch of single-file-mappers.

I haven't really thought this second model through a huge amount but I 
probably should sit down and play with it.

In your specific case, can you show the code that your trying to make 
work? If you are running a foreachwhere the cardinality is known before 
hand (i.e. you know how many iterations you want, before the start of the 
loop) you can perhaps programatically construct a string[] array with your 
constructed filenames, and then use an array mapper to use that array of 
strings as the filenames for an array.

So one foreach loop to make string[] arraynames, then a statement to use 
arraynames to map an entire array, and then your original foreach loop.

In the case of an iterate where the cardinality is not known before hand, 
that method probably doesn't work.

-- 



From hategan at mcs.anl.gov  Wed Feb 25 09:40:22 2009
From: hategan at mcs.anl.gov (Mihael Hategan)
Date: Wed, 25 Feb 2009 09:40:22 -0600
Subject: [Swift-user] assigning file variables
In-Reply-To: <Pine.LNX.4.64.0902251521230.1293@dildano.hawaga.org.uk>
References: <49A55309.4050100@mcs.anl.gov>
	<Pine.LNX.4.64.0902251521230.1293@dildano.hawaga.org.uk>
Message-ID: <1235576422.17806.4.camel@localhost>

I think we cannot look at a solution without considering mapping. In
other words, there is a semantic distinction between file t; and file m
<"a.txt">;. One says "I don't care how it's named" and the other says
different.

So based on that, we can probably derive a few rules:
t = m -> aliasing
t = t -> aliasing
m = t -> copy; it would be preferable to map t to what m is mapped to
from the start
m = m -> copy.

Mihael

On Wed, 2009-02-25 at 15:32 +0000, Ben Clifford wrote:
> The semantics of file assignments have been discussed on and of for some 
> years.
> 
> Those discussions have left two different ideas floating around in my 
> head:
> 
> i) = as copy
> 
>  The code file a <"foo">; file b<"bar">; a = b; behaves something like cp 
> bar foo.
> 
> Thats pretty straightforward, bu to implement and understandt unclear to 
> me if thats the right thing to do in many cases - for example, when 
> assigning into an array of files, you would end up with the members of the 
> array still being mapped to the filenames that the mapper associated with 
> the array gives, not the filenames of the rhs file in the assignment.
> 
> Its not clear to me if this behaviour, for example, would be useful in the 
> use case you specify. Why do you want to construct the filenames in a 
> difficult way? Is it because you want them to end up with that filename, 
> or because your application wants to generate them that way? In the 
> former, this model doesn't work, though in the latter, if you don't care 
> what they end up being called, thats fine.
> 
> ii) assignment-as-aliasing
> 
> file a; file b<"bar">; a= b turns a into an alias for b, mapped still to 
> the file "bar"; no copying takes place.
> 
> In your case, you'd then end up with an array of aliases to files that are 
> mapped by a bunch of single-file-mappers.
> 
> I haven't really thought this second model through a huge amount but I 
> probably should sit down and play with it.
> 
> In your specific case, can you show the code that your trying to make 
> work? If you are running a foreachwhere the cardinality is known before 
> hand (i.e. you know how many iterations you want, before the start of the 
> loop) you can perhaps programatically construct a string[] array with your 
> constructed filenames, and then use an array mapper to use that array of 
> strings as the filenames for an array.
> 
> So one foreach loop to make string[] arraynames, then a statement to use 
> arraynames to map an entire array, and then your original foreach loop.
> 
> In the case of an iterate where the cardinality is not known before hand, 
> that method probably doesn't work.
> 



From benc at hawaga.org.uk  Wed Feb 25 09:50:47 2009
From: benc at hawaga.org.uk (Ben Clifford)
Date: Wed, 25 Feb 2009 15:50:47 +0000 (GMT)
Subject: [Swift-user] assigning file variables
In-Reply-To: <1235576422.17806.4.camel@localhost>
References: <49A55309.4050100@mcs.anl.gov>
	<Pine.LNX.4.64.0902251521230.1293@dildano.hawaga.org.uk>
	<1235576422.17806.4.camel@localhost>
Message-ID: <Pine.LNX.4.64.0902251545300.23512@dildano.hawaga.org.uk>


On Wed, 25 Feb 2009, Mihael Hategan wrote:

> it would be preferable to map t to what m is mapped to
> from the start

right. often (always?) the desire to do this kind of assignment comes from 
insufficient expressiveness in our mapping semantics. in the foreach case, 
I think my email suggests a reasonable alternative to assignments that 
allows mapping to be generated inside of Swift. In the iterate{} case, 
that in-swift expression is not possible at the moment, but could be. For 
example, soemthing like the ext mapper that only maps output files, not 
inputs, and calls a specified swift procedure to do that mapping. (thates 
something that has been discussed before, I think)

-- 



From wilde at mcs.anl.gov  Wed Feb 25 10:03:48 2009
From: wilde at mcs.anl.gov (Michael Wilde)
Date: Wed, 25 Feb 2009 10:03:48 -0600
Subject: [Swift-user] assigning file variables
In-Reply-To: <Pine.LNX.4.64.0902251521230.1293@dildano.hawaga.org.uk>
References: <49A55309.4050100@mcs.anl.gov>
	<Pine.LNX.4.64.0902251521230.1293@dildano.hawaga.org.uk>
Message-ID: <49A56BE4.20209@mcs.anl.gov>

I was envisioning approach (ii). I certainly dont want a file "cp" to 
take place, and I have not yet seen a need for (i), other than something 
related to retaining provenance across copies, which is not my need here.

I was envisioning (ii) more as a simple "reference" assignment, which 
made the target variable have the same state as the source variable, 
including the mapping.

Regardless, I dont think this is an issue we need to rush to resolve at 
the moment. Its more a matter of creating enough documentation and style 
examples so that people fall into paths that work for them.

My code is in transition but I put two version of it under: 
http://www.ci.uchicago.edu/~wilde/oops8.swift
http://www.ci.uchicago.edu/~wilde/oops8e.swift

oops8.swift doesnt work, but shows the structure of what I was trying to 
accomplish.

When I hit a few problems (like the extractint issue, file copy issue, 
mapping same file twice issue, and iterate condition issue) I created a 
simplified version oops8e.swift that used "echo" instead of the more 
complex app. oops8e now correctly does the nested 
foreach-iterate-foreach loops.

Next I'll be adapting the code to use structs to pass the large sets of 
files in and out of the app, so that the final functions arevery simple 
for the user to invoke. In doing this, I may need to go to ext mappers 
anyways, dont know yet.

It seems like the swift framework for this app is now working, and I'll 
post any new issues as they come up.

some notes, below...

On 2/25/09 9:32 AM, Ben Clifford wrote:
> The semantics of file assignments have been discussed on and of for some 
> years.
> 
> Those discussions have left two different ideas floating around in my 
> head:
> 
> i) = as copy
> 
>  The code file a <"foo">; file b<"bar">; a = b; behaves something like cp 
> bar foo.
> 
> Thats pretty straightforward, bu to implement and understandt unclear to 
> me if thats the right thing to do in many cases - for example, when 
> assigning into an array of files, you would end up with the members of the 
> array still being mapped to the filenames that the mapper associated with 
> the array gives, not the filenames of the rhs file in the assignment.
> 
> Its not clear to me if this behaviour, for example, would be useful in the 
> use case you specify. Why do you want to construct the filenames in a 
> difficult way? Is it because you want them to end up with that filename,

yes, in this case. I did try mapping the 2D array with simple_mapper, 
and that gave me names that were surprisingly acceptable, but not what I 
had planned. A simple variation on it might have worked though, 
something sprintf() like, which I could probably do with an equivalent 
ext mapper. So there is certainly a reasonable solution here.

> or because your application wants to generate them that way? In the 
> former, this model doesn't work, though in the latter, if you don't care 
> what they end up being called, thats fine.
> 
> ii) assignment-as-aliasing
> 
> file a; file b<"bar">; a= b turns a into an alias for b, mapped still to 
> the file "bar"; no copying takes place.
> 
> In your case, you'd then end up with an array of aliases to files that are 
> mapped by a bunch of single-file-mappers.

Yes, thats what I was looking for.

> I haven't really thought this second model through a huge amount but I 
> probably should sit down and play with it.
> 
> In your specific case, can you show the code that your trying to make 
> work? If you are running a foreachwhere the cardinality is known before 
> hand (i.e. you know how many iterations you want, before the start of the 
> loop) you can perhaps programatically construct a string[] array with your 
> constructed filenames, and then use an array mapper to use that array of 
> strings as the filenames for an array.

In this case I know the size of the inner array, the result of the inner 
foreach{} loop, at the start.  So that might work.
> 
> So one foreach loop to make string[] arraynames, then a statement to use 
> arraynames to map an entire array, and then your original foreach loop.

I can try that.
> 
> In the case of an iterate where the cardinality is not known before hand, 
> that method probably doesn't work.
> 


From wilde at mcs.anl.gov  Thu Feb 26 00:13:18 2009
From: wilde at mcs.anl.gov (Michael Wilde)
Date: Thu, 26 Feb 2009 00:13:18 -0600
Subject: [Swift-user] assigning file variables
In-Reply-To: <Pine.LNX.4.64.0902251545300.23512@dildano.hawaga.org.uk>
References: <49A55309.4050100@mcs.anl.gov>
	<Pine.LNX.4.64.0902251521230.1293@dildano.hawaga.org.uk>
	<1235576422.17806.4.camel@localhost>
	<Pine.LNX.4.64.0902251545300.23512@dildano.hawaga.org.uk>
Message-ID: <49A632FE.8070906@mcs.anl.gov>

Can you clarify how the ext mapper behaves differently from say the 
simple_mapper for output files, and if the following is correct?

It seems that for the simple_mapper, the mapper parameters define a 
prefix/suffix, and these strings are used wherever necessary at runtime 
to form a mapping for any object composed of (possibly nested) structs 
and arrays, by bracketing the dynamically-constructed object path.

But when the ext mapper is used for output, it is expected, in a single 
call, to map the entire structure (and hence can only do static mappings)?

I thought I had my problem solved using the ext mapper, but the 
combination of restrictions on assigning file variables and getting the 
right info to the ext mapper seems to be forcing me back to simple_mapper.

(I'll try to assemble examples when I have more time)

On 2/25/09 9:50 AM, Ben Clifford wrote:
> On Wed, 25 Feb 2009, Mihael Hategan wrote:
> 
>> it would be preferable to map t to what m is mapped to
>> from the start
> 
> right. often (always?) the desire to do this kind of assignment comes from 
> insufficient expressiveness in our mapping semantics. in the foreach case, 
> I think my email suggests a reasonable alternative to assignments that 
> allows mapping to be generated inside of Swift. In the iterate{} case, 
> that in-swift expression is not possible at the moment, but could be. For 
> example, soemthing like the ext mapper that only maps output files, not 
> inputs, and calls a specified swift procedure to do that mapping. (thates 
> something that has been discussed before, I think)
>