[Swift-user] Swift/PBS Scheduler Slow to Report "Finished"?
Andriy Fedorov
fedorov at cs.wm.edu
Thu Mar 19 13:55:21 CDT 2009
Andrew,
I had the same problem with some of the TeraGrid sites. You might want
to see the relevant discussion here:
http://mail.ci.uchicago.edu/pipermail/swift-user/2008-June/000440.html
Andrey Fedorov
> Date: Thu, 19 Mar 2009 02:42:25 -0500
> From: Andrew Boyce <ajboyce at jacks.sdstate.edu>
> Subject: [Swift-user] Swift/PBS Scheduler Slow to Report "Finished"?
> To: swift-user at ci.uchicago.edu
> Message-ID: <BLU0-SMTP55C21AC08072188E782F99E7960 at phx.gbl>
> Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes
>
> Hello,
>
> I am currently running Swift in conjunction with the PBS scheduler. My
> annoyance at the moment is this:
>
> When running any script, even a simple script such as first.swift
> (which normally finishes almost instantaneously), Swift always takes
> precisely five minutes to tell me that my job Finished successfully
> and copy the files back to the appropriate folder. It is always almost
> exactly five minutes; I've checked many logs - it polls the scheduler
> for five minutes. When I run a script (like first.swift) without using
> the PBS scheduler, everything happens as normal; execution and
> "Finished successfully" are nearly immediate.
>
> I think I know what the problem is: even after the scheduler says that
> the job is 'completed,' (which is generally right away) the scheduler
> keeps the job up on qstat and such for 5 minutes after (this setting
> is a PBS server attribute known as 'keep_completed', and I have
> checked that it is indeed set to 300 seconds; unfortunately I don't
> have permissions to change it). So when Swift polls the scheduler, the
> job is still up on qstat, and Swift must think that the task has not
> yet "Finished successfully."
>
> My question is this:
> Am I indeed right that Swift does not "understand" that when the PBS
> scheduler says a job is 'completed', the job really has "Finished
> successfully"?
> Can this be changed so that Swift does "understand" that a 'completed'
> job has "Finished successfully"?
>
> I have not included any files because I think I have narrowed the
> problem down to a question that does not require those that I would
> usually provide, but if I am wrong, then I can provide.
>
> Thank you and sorry for the length.
>
> Regards,
>
> Andrew Boyce
>
>
> ------------------------------
>
> Message: 2
> Date: Thu, 19 Mar 2009 07:26:19 -0500
> From: Michael Wilde <wilde at mcs.anl.gov>
> Subject: Re: [Swift-user] Swift/PBS Scheduler Slow to Report
> "Finished"?
> To: Andrew Boyce <ajboyce at jacks.sdstate.edu>
> Cc: swift-user at ci.uchicago.edu
> Message-ID: <49C239EB.9010008 at mcs.anl.gov>
> Content-Type: text/plain; charset=ISO-8859-1; format=flowed
>
> The Swift developers will need to look into this issue, which seems to
> be with the Karajan PBS provider. I dont think we see this delay on our
> local PBS cluster here.
>
> In the meantime, you might want to try the fairly new "coaster" provider:
>
> http://www.ci.uchicago.edu/swift/guides/userguide.php#coasters
>
> This starts "worker" jobs in the target cluster which stay up for the
> duration of a script, into which Swift sends jobs directly without
> involving the scheduler.
>
> If your scheduler has this 5-minute "linger" setting, the overall script
> will still wait at the end (I think), but all the jobs in the script
> should finish very quickly.
>
> If you're interested, preliminary notes on the design of coasters is at:
> http://wiki.cogkit.org/wiki/Coasters
>
> A few cautions:
>
> - coasters are a new feature, code is changing rapidly, and they are not
> yet suffciently tested.
>
> - we'd welcome your help in evaluating them
>
> - you'll want to run from a source release
>
> - I think they are a good base for a lot of interesting projects and
> studies on scheduling and resource allocation algorithms and approaches.
>
> I tested a simple 10-echo foreach loop with this sites.xml file on our
> local pbs cluster:
>
> --
>
> <config>
> <pool handle="teraport" >
> <execution provider="coaster" url="none" jobmanager="local:pbs" />
> <gridftp url="local://localhost" />
> <workdirectory>/home/wilde/swiftwork</workdirectory>
> </pool>
> </config>
>
> -- which gave:
>
> tp$ swift hellos.swift -sites.file sites.xml -tc.file tc.data
> Swift svn swift-r2701 cog-r2332
>
> RunID: 20090319-0658-3ejpl9xc
> Progress:
> Progress: Submitting:9 Submitted:1
> Progress: Submitted:9 Active:1
> Progress: Submitted:4 Active:3 Stage out:1 Finished successfully:2
> Final status: Finished successfully:10
> Cleaning up...
> Shutting down service at https://128.135.125.117:50002
> Got channel MetaChannel: 101224864 -> GSSSChannel-null(1)
> - Done
>
> --
>
>
> On 3/19/09 2:42 AM, Andrew Boyce wrote:
>> Hello,
>>
>> I am currently running Swift in conjunction with the PBS scheduler. My
>> annoyance at the moment is this:
>>
>> When running any script, even a simple script such as first.swift (which
>> normally finishes almost instantaneously), Swift always takes precisely
>> five minutes to tell me that my job Finished successfully and copy the
>> files back to the appropriate folder. It is always almost exactly five
>> minutes; I've checked many logs - it polls the scheduler for five
>> minutes. When I run a script (like first.swift) without using the PBS
>> scheduler, everything happens as normal; execution and "Finished
>> successfully" are nearly immediate.
>>
>> I think I know what the problem is: even after the scheduler says that
>> the job is 'completed,' (which is generally right away) the scheduler
>> keeps the job up on qstat and such for 5 minutes after (this setting is
>> a PBS server attribute known as 'keep_completed', and I have checked
>> that it is indeed set to 300 seconds; unfortunately I don't have
>> permissions to change it). So when Swift polls the scheduler, the job is
>> still up on qstat, and Swift must think that the task has not yet
>> "Finished successfully."
>>
>> My question is this:
>> Am I indeed right that Swift does not "understand" that when the PBS
>> scheduler says a job is 'completed', the job really has "Finished
>> successfully"?
>> Can this be changed so that Swift does "understand" that a 'completed'
>> job has "Finished successfully"?
>>
>> I have not included any files because I think I have narrowed the
>> problem down to a question that does not require those that I would
>> usually provide, but if I am wrong, then I can provide.
>>
>> Thank you and sorry for the length.
>>
>> Regards,
>>
>> Andrew Boyce
>> _______________________________________________
>> Swift-user mailing list
>> Swift-user at ci.uchicago.edu
>> http://mail.ci.uchicago.edu/mailman/listinfo/swift-user
>
>
> ------------------------------
>
> Message: 3
> Date: Thu, 19 Mar 2009 07:35:56 -0500
> From: Michael Wilde <wilde at mcs.anl.gov>
> Subject: Re: [Swift-user] Swift/PBS Scheduler Slow to Report
> "Finished"?
> To: Andrew Boyce <ajboyce at jacks.sdstate.edu>
> Cc: swift-user at ci.uchicago.edu
> Message-ID: <49C23C2C.1050507 at mcs.anl.gov>
> Content-Type: text/plain; charset=ISO-8859-1; format=flowed
>
> Sorry, one clarification:
>
> > - you'll want to run from a source release
>
> By this I meant that the coaster code is changing frequently due to
> ongoing testing. So you'll typically want to run the latest svn
> revision. Checkout and build is simple and described under "Building
> Swift" at: http://www.ci.uchicago.edu/swift/downloads/index.php
>
>
> On 3/19/09 7:26 AM, Michael Wilde wrote:
>> The Swift developers will need to look into this issue, which seems to
>> be with the Karajan PBS provider. I dont think we see this delay on our
>> local PBS cluster here.
>>
>> In the meantime, you might want to try the fairly new "coaster" provider:
>>
>> http://www.ci.uchicago.edu/swift/guides/userguide.php#coasters
>>
>> This starts "worker" jobs in the target cluster which stay up for the
>> duration of a script, into which Swift sends jobs directly without
>> involving the scheduler.
>>
>> If your scheduler has this 5-minute "linger" setting, the overall script
>> will still wait at the end (I think), but all the jobs in the script
>> should finish very quickly.
>>
>> If you're interested, preliminary notes on the design of coasters is at:
>> http://wiki.cogkit.org/wiki/Coasters
>>
>> A few cautions:
>>
>> - coasters are a new feature, code is changing rapidly, and they are not
>> yet suffciently tested.
>>
>> - we'd welcome your help in evaluating them
>>
>> - you'll want to run from a source release
>>
>> - I think they are a good base for a lot of interesting projects and
>> studies on scheduling and resource allocation algorithms and approaches.
>>
>> I tested a simple 10-echo foreach loop with this sites.xml file on our
>> local pbs cluster:
>>
>> --
>>
>> <config>
>> <pool handle="teraport" >
>> <execution provider="coaster" url="none" jobmanager="local:pbs" />
>> <gridftp url="local://localhost" />
>> <workdirectory>/home/wilde/swiftwork</workdirectory>
>> </pool>
>> </config>
>>
>> -- which gave:
>>
>> tp$ swift hellos.swift -sites.file sites.xml -tc.file tc.data
>> Swift svn swift-r2701 cog-r2332
>>
>> RunID: 20090319-0658-3ejpl9xc
>> Progress:
>> Progress: Submitting:9 Submitted:1
>> Progress: Submitted:9 Active:1
>> Progress: Submitted:4 Active:3 Stage out:1 Finished successfully:2
>> Final status: Finished successfully:10
>> Cleaning up...
>> Shutting down service at https://128.135.125.117:50002
>> Got channel MetaChannel: 101224864 -> GSSSChannel-null(1)
>> - Done
>>
>> --
>>
>>
>> On 3/19/09 2:42 AM, Andrew Boyce wrote:
>>> Hello,
>>>
>>> I am currently running Swift in conjunction with the PBS scheduler. My
>>> annoyance at the moment is this:
>>>
>>> When running any script, even a simple script such as first.swift
>>> (which normally finishes almost instantaneously), Swift always takes
>>> precisely five minutes to tell me that my job Finished successfully
>>> and copy the files back to the appropriate folder. It is always almost
>>> exactly five minutes; I've checked many logs - it polls the scheduler
>>> for five minutes. When I run a script (like first.swift) without using
>>> the PBS scheduler, everything happens as normal; execution and
>>> "Finished successfully" are nearly immediate.
>>>
>>> I think I know what the problem is: even after the scheduler says that
>>> the job is 'completed,' (which is generally right away) the scheduler
>>> keeps the job up on qstat and such for 5 minutes after (this setting
>>> is a PBS server attribute known as 'keep_completed', and I have
>>> checked that it is indeed set to 300 seconds; unfortunately I don't
>>> have permissions to change it). So when Swift polls the scheduler, the
>>> job is still up on qstat, and Swift must think that the task has not
>>> yet "Finished successfully."
>>>
>>> My question is this:
>>> Am I indeed right that Swift does not "understand" that when the PBS
>>> scheduler says a job is 'completed', the job really has "Finished
>>> successfully"?
>>> Can this be changed so that Swift does "understand" that a 'completed'
>>> job has "Finished successfully"?
>>>
>>> I have not included any files because I think I have narrowed the
>>> problem down to a question that does not require those that I would
>>> usually provide, but if I am wrong, then I can provide.
>>>
>>> Thank you and sorry for the length.
>>>
>>> Regards,
>>>
>>> Andrew Boyce
>>> _______________________________________________
>>> Swift-user mailing list
>>> Swift-user at ci.uchicago.edu
>>> http://mail.ci.uchicago.edu/mailman/listinfo/swift-user
>> _______________________________________________
>> Swift-user mailing list
>> Swift-user at ci.uchicago.edu
>> http://mail.ci.uchicago.edu/mailman/listinfo/swift-user
>
>
> ------------------------------
>
> Message: 4
> Date: Thu, 19 Mar 2009 08:27:02 -0500
> From: Michael Wilde <wilde at mcs.anl.gov>
> Subject: Re: [Swift-user] swift execution problem
> To: "Yue, Chen - BMD" <yuechen at bsd.uchicago.edu>
> Cc: swift-user at ci.uchicago.edu
> Message-ID: <49C24826.4090302 at mcs.anl.gov>
> Content-Type: text/plain; charset=ISO-8859-1; format=flowed
>
> Yue, what version of Swift are you using?
>
> Please send the first few lines of your output file, where it says
> something like:
>
> Swift svn swift-r2701 cog-r2332
>
> RunID: 20090319-0820-19zttiq9
>
> (in fact please send the whole output file, stdout/err)
>
> Ive tried to run your code in a near-identical test and I cant reproduce
> the failure. Ive tried with both swift0.8 and the latest svn rev, and
> both seem to work.
>
> Also please can you post the pathname of the directory in which you are
> testing (I assume you are running this on a CI machine?) so that I can
> look at your logfile? And make it publicly accessible?
>
> Thanks,
>
> - Mike
>
>
> On 3/18/09 6:32 PM, Yue, Chen - BMD wrote:
>> Hi,
>>
>> I'm new to Swift programming. I was able to run a swift script before,
>> but I couldn't run it now. I'm wondering if someone can help me figure
>> out why. The swift script, sites.xml, tc.data, and all the error
>> messages are copied in this email. Thank you!
>>
>> Regards,
>>
>> Chen, Yue
>>
>> *********************
>> Swift script
>> *********************
>> type Fasta {}
>> type PTMapOut {}
>> type Solution {}
>> type Inputfile {}
>> app (PTMapOut ofile) PTMap (Solution sfile, Fasta fastafile, Inputfile
>> input, Inputfile parameter)
>> {
>> PTMap @filename(sfile) @filename(fastafile) @filename(input)
>> @filename(parameter) stdout=@filename(ofile
>> <mailto:stdout=@filename(ofile>);
>> }
>> Fasta texts[] <filesys_mapper;prefix="fasta">;
>>
>> doall(Fasta texts[])
>> {
>> Solution sfile <"BSASolution.mzXML">;
>> Inputfile input <"inputs.txt">;
>> Inputfile parameter <"parameters.txt">;
>> foreach p in texts {
>> PTMapOut r <regexp_mapper;
>> source=@p <mailto:source=@p>,
>> match="fasta(.*)",
>> transform="\\1.out <file://\\1.out>"
>> >;
>> r = PTMap(sfile, p, input, parameter);
>> }
>> }
>> // Main
>> doall(texts);
>> **************
>> sites.xml
>> **************
>> <pool handle="localhost">
>> <gridftp url="local://localhost" />
>> <execution provider="local" />
>> <workdirectory >/var/tmp</workdirectory>
>> <profile namespace="karajan" key="jobThrottle">0</profile>
>> </pool>
>> **************
>> tc.data
>> **************
>> localhost echo /bin/echo INSTALLED
>> INTEL32::LINUX null
>> localhost cat /bin/cat INSTALLED
>> INTEL32::LINUX null
>> localhost ls /bin/ls INSTALLED
>> INTEL32::LINUX null
>> localhost grep /bin/grep INSTALLED
>> INTEL32::LINUX null
>> localhost sort /bin/sort INSTALLED
>> INTEL32::LINUX null
>> localhost paste /bin/paste INSTALLED
>> INTEL32::LINUX null
>> localhost PTMap /home/yuechen/PTMap/PTMap INSTALLED
>> INTEL32::LINUX null
>> **************
>> Error messages
>> **************
>> [yuechen at communicado PTMap]$ swift PTMap.swift
>> Execution failed:
>> java.lang.NullPointerException
>> at
>> org.globus.cog.abstraction.impl.common.task.ServiceImpl.toString(ServiceImpl.java:156)
>> at java.lang.String.valueOf(String.java:2577)
>> at java.lang.StringBuffer.append(StringBuffer.java:220)
>> at
>> org.globus.cog.karajan.workflow.nodes.grid.GridNode.function(GridNode.java:31)
>> at
>> org.globus.cog.karajan.workflow.nodes.functions.AbstractFunction.post(AbstractFunction.java:45)
>> at
>> org.globus.cog.karajan.workflow.nodes.AbstractSequentialWithArguments.childCompleted(AbstractSequentialWithArguments.java:192)
>> at
>> org.globus.cog.karajan.workflow.nodes.Sequential.notificationEvent(Sequential.java:33)
>> at
>> org.globus.cog.karajan.workflow.nodes.FlowNode.event(FlowNode.java:335)
>> at
>> org.globus.cog.karajan.workflow.events.EventBus.send(EventBus.java:125)
>> at
>> org.globus.cog.karajan.workflow.events.EventBus.sendHooked(EventBus.java:99)
>> at
>> org.globus.cog.karajan.workflow.nodes.FlowNode.fireNotificationEvent(FlowNode.java:173)
>> at
>> org.globus.cog.karajan.workflow.nodes.FlowNode.complete(FlowNode.java:299)
>> at
>> org.globus.cog.karajan.workflow.nodes.ExecuteFile.notificationEvent(ExecuteFile.java:163)
>> at
>> org.globus.cog.karajan.workflow.nodes.FlowNode.event(FlowNode.java:335)
>> at
>> org.globus.cog.karajan.workflow.events.EventBus.send(EventBus.java:125)
>> at
>> org.globus.cog.karajan.workflow.events.EventBus.sendHooked(EventBus.java:99)
>> at
>> org.globus.cog.karajan.workflow.nodes.FlowNode.fireNotificationEvent(FlowNode.java:173)
>> at
>> org.globus.cog.karajan.workflow.nodes.FlowNode.complete(FlowNode.java:299)
>> at
>> org.globus.cog.karajan.workflow.nodes.FlowContainer.post(FlowContainer.java:58)
>> at
>> org.globus.cog.karajan.workflow.nodes.Sequential.startNext(Sequential.java:51)
>> at
>> org.globus.cog.karajan.workflow.nodes.Sequential.childCompleted(Sequential.java:45)
>> at
>> org.globus.cog.karajan.workflow.nodes.Sequential.notificationEvent(Sequential.java:33)
>> at
>> org.globus.cog.karajan.workflow.nodes.FlowNode.event(FlowNode.java:335)
>> at
>> org.globus.cog.karajan.workflow.events.EventBus.send(EventBus.java:125)
>> at
>> org.globus.cog.karajan.workflow.events.EventBus.sendHooked(EventBus.java:99)
>> at
>> org.globus.cog.karajan.workflow.nodes.FlowNode.fireNotificationEvent(FlowNode.java:173)
>> at
>> org.globus.cog.karajan.workflow.nodes.FlowNode.complete(FlowNode.java:299)
>> at
>> org.globus.cog.karajan.workflow.nodes.FlowContainer.post(FlowContainer.java:58)
>> at
>> org.globus.cog.karajan.workflow.nodes.AbstractSequentialWithArguments.childCompleted(AbstractSequentialWithArguments.java:192)
>> at
>> org.globus.cog.karajan.workflow.nodes.user.UserDefinedElement.childCompleted(UserDefinedElement.java:283)
>> at
>> org.globus.cog.karajan.workflow.nodes.user.SequentialImplicitExecutionUDE.childCompleted(SequentialImplicitExecutionUDE.java:85)
>> at
>> org.globus.cog.karajan.workflow.nodes.Sequential.notificationEvent(Sequential.java:33)
>> at
>> org.globus.cog.karajan.workflow.nodes.FlowNode.event(FlowNode.java:335)
>> at
>> org.globus.cog.karajan.workflow.events.EventBus.send(EventBus.java:125)
>> at
>> org.globus.cog.karajan.workflow.events.EventBus.sendHooked(EventBus.java:99)
>> at
>> org.globus.cog.karajan.workflow.nodes.FlowNode.fireNotificationEvent(FlowNode.java:173)
>> at
>> org.globus.cog.karajan.workflow.nodes.FlowNode.complete(FlowNode.java:299)
>> at
>> org.globus.cog.karajan.workflow.nodes.If.childCompleted(If.java:30)
>> at
>> org.globus.cog.karajan.workflow.nodes.Sequential.notificationEvent(Sequential.java:33)
>> at
>> org.globus.cog.karajan.workflow.nodes.FlowNode.event(FlowNode.java:335)
>> at
>> org.globus.cog.karajan.workflow.events.EventBus.send(EventBus.java:125)
>> at
>> org.globus.cog.karajan.workflow.events.EventBus.sendHooked(EventBus.java:99)
>> at
>> org.globus.cog.karajan.workflow.nodes.FlowNode.fireNotificationEvent(FlowNode.java:173)
>> at
>> org.globus.cog.karajan.workflow.nodes.FlowNode.complete(FlowNode.java:299)
>> at
>> org.globus.cog.karajan.workflow.nodes.FlowContainer.post(FlowContainer.java:58)
>> at
>> org.globus.cog.karajan.workflow.nodes.functions.AbstractFunction.post(AbstractFunction.java:46)
>> at
>> org.globus.cog.karajan.workflow.nodes.AbstractSequentialWithArguments.childCompleted(AbstractSequentialWithArguments.java:192)
>> at
>> org.globus.cog.karajan.workflow.nodes.Sequential.notificationEvent(Sequential.java:33)
>> at
>> org.globus.cog.karajan.workflow.nodes.FlowNode.event(FlowNode.java:335)
>> at
>> org.globus.cog.karajan.workflow.events.EventBus.send(EventBus.java:125)
>> at
>> org.globus.cog.karajan.workflow.events.EventBus.sendHooked(EventBus.java:99)
>> at
>> org.globus.cog.karajan.workflow.nodes.FlowNode.fireNotificationEvent(FlowNode.java:173)
>> at
>> org.globus.cog.karajan.workflow.nodes.FlowNode.complete(FlowNode.java:299)
>> at
>> org.globus.cog.karajan.workflow.nodes.FlowContainer.post(FlowContainer.java:58)
>> at
>> org.globus.cog.karajan.workflow.nodes.functions.AbstractFunction.post(AbstractFunction.java:46)
>> at
>> org.globus.cog.karajan.workflow.nodes.Sequential.startNext(Sequential.java:51)
>> at
>> org.globus.cog.karajan.workflow.nodes.Sequential.executeChildren(Sequential.java:27)
>> at
>> org.globus.cog.karajan.workflow.nodes.functions.AbstractFunction.executeChildren(AbstractFunction.java:40)
>> at
>> org.globus.cog.karajan.workflow.nodes.FlowContainer.execute(FlowContainer.java:63)
>> at
>> org.globus.cog.karajan.workflow.nodes.FlowNode.restart(FlowNode.java:240)
>> at
>> org.globus.cog.karajan.workflow.nodes.FlowNode.start(FlowNode.java:281)
>> at
>> org.globus.cog.karajan.workflow.nodes.FlowNode.controlEvent(FlowNode.java:393)
>> at
>> org.globus.cog.karajan.workflow.nodes.FlowNode.event(FlowNode.java:332)
>> at
>> org.globus.cog.karajan.workflow.FlowElementWrapper.event(FlowElementWrapper.java:227)
>> at
>> org.globus.cog.karajan.workflow.events.EventBus.send(EventBus.java:125)
>> at
>> org.globus.cog.karajan.workflow.events.EventBus.sendHooked(EventBus.java:99)
>> at
>> org.globus.cog.karajan.workflow.events.EventWorker.run(EventWorker.java:69)
>>
>>
>>
>>
>>
>> This email is intended only for the use of the individual or entity to
>> which it is addressed and may contain information that is privileged and
>> confidential. If the reader of this email message is not the intended
>> recipient, you are hereby notified that any dissemination, distribution,
>> or copying of this communication is prohibited. If you have received
>> this email in error, please notify the sender and destroy/delete all
>> copies of the transmittal. Thank you.
>>
>>
>> ------------------------------------------------------------------------
>>
>> _______________________________________________
>> Swift-user mailing list
>> Swift-user at ci.uchicago.edu
>> http://mail.ci.uchicago.edu/mailman/listinfo/swift-user
>
>
> ------------------------------
>
> Message: 5
> Date: Thu, 19 Mar 2009 09:17:43 -0500
> From: Michael Wilde <wilde at mcs.anl.gov>
> Subject: Re: [Swift-user] swift execution problem
> To: "Yue, Chen - BMD" <yuechen at bsd.uchicago.edu>
> Cc: swift-user at ci.uchicago.edu
> Message-ID: <49C25407.7010000 at mcs.anl.gov>
> Content-Type: text/plain; charset=ISO-8859-1; format=flowed
>
> Yue, I found the log for the failing run in your working dir.
> I'll post it to the swift-devel list for the developers to look at.
>
> The problem seems related to your specific data (your 375 fasta files)
> but even when I copy your data I cant repro the problem, at least not
> using "echo" instead of "ptmap".
>
> - Mike
>
>
> On 3/19/09 8:27 AM, Michael Wilde wrote:
>> Yue, what version of Swift are you using?
>>
>> Please send the first few lines of your output file, where it says
>> something like:
>>
>> Swift svn swift-r2701 cog-r2332
>>
>> RunID: 20090319-0820-19zttiq9
>>
>> (in fact please send the whole output file, stdout/err)
>>
>> Ive tried to run your code in a near-identical test and I cant reproduce
>> the failure. Ive tried with both swift0.8 and the latest svn rev, and
>> both seem to work.
>>
>> Also please can you post the pathname of the directory in which you are
>> testing (I assume you are running this on a CI machine?) so that I can
>> look at your logfile? And make it publicly accessible?
>>
>> Thanks,
>>
>> - Mike
>>
>>
>> On 3/18/09 6:32 PM, Yue, Chen - BMD wrote:
>>> Hi,
>>>
>>> I'm new to Swift programming. I was able to run a swift script before,
>>> but I couldn't run it now. I'm wondering if someone can help me figure
>>> out why. The swift script, sites.xml, tc.data, and all the error
>>> messages are copied in this email. Thank you!
>>>
>>> Regards,
>>>
>>> Chen, Yue
>>>
>>> *********************
>>> Swift script
>>> *********************
>>> type Fasta {}
>>> type PTMapOut {}
>>> type Solution {}
>>> type Inputfile {}
>>> app (PTMapOut ofile) PTMap (Solution sfile, Fasta fastafile, Inputfile
>>> input, Inputfile parameter)
>>> {
>>> PTMap @filename(sfile) @filename(fastafile) @filename(input)
>>> @filename(parameter) stdout=@filename(ofile
>>> <mailto:stdout=@filename(ofile>);
>>> }
>>> Fasta texts[] <filesys_mapper;prefix="fasta">;
>>>
>>> doall(Fasta texts[])
>>> {
>>> Solution sfile <"BSASolution.mzXML">;
>>> Inputfile input <"inputs.txt">;
>>> Inputfile parameter <"parameters.txt">;
>>> foreach p in texts {
>>> PTMapOut r <regexp_mapper;
>>> source=@p <mailto:source=@p>,
>>> match="fasta(.*)",
>>> transform="\\1.out <file://\\1.out>"
>>> >;
>>> r = PTMap(sfile, p, input, parameter);
>>> }
>>> }
>>> // Main
>>> doall(texts);
>>> **************
>>> sites.xml
>>> **************
>>> <pool handle="localhost">
>>> <gridftp url="local://localhost" />
>>> <execution provider="local" />
>>> <workdirectory >/var/tmp</workdirectory>
>>> <profile namespace="karajan" key="jobThrottle">0</profile>
>>> </pool>
>>> **************
>>> tc.data
>>> **************
>>> localhost echo /bin/echo INSTALLED
>>> INTEL32::LINUX null
>>> localhost cat /bin/cat INSTALLED
>>> INTEL32::LINUX null
>>> localhost ls /bin/ls INSTALLED
>>> INTEL32::LINUX null
>>> localhost grep /bin/grep INSTALLED
>>> INTEL32::LINUX null
>>> localhost sort /bin/sort INSTALLED
>>> INTEL32::LINUX null
>>> localhost paste /bin/paste INSTALLED
>>> INTEL32::LINUX null
>>> localhost PTMap /home/yuechen/PTMap/PTMap
>>> INSTALLED INTEL32::LINUX null
>>> **************
>>> Error messages
>>> **************
>>> [yuechen at communicado PTMap]$ swift PTMap.swift
>>> Execution failed:
>>> java.lang.NullPointerException
>>> at
>>> org.globus.cog.abstraction.impl.common.task.ServiceImpl.toString(ServiceImpl.java:156)
>>>
>>> at java.lang.String.valueOf(String.java:2577)
>>> at java.lang.StringBuffer.append(StringBuffer.java:220)
>>> at
>>> org.globus.cog.karajan.workflow.nodes.grid.GridNode.function(GridNode.java:31)
>>>
>>> at
>>> org.globus.cog.karajan.workflow.nodes.functions.AbstractFunction.post(AbstractFunction.java:45)
>>>
>>> at
>>> org.globus.cog.karajan.workflow.nodes.AbstractSequentialWithArguments.childCompleted(AbstractSequentialWithArguments.java:192)
>>>
>>> at
>>> org.globus.cog.karajan.workflow.nodes.Sequential.notificationEvent(Sequential.java:33)
>>>
>>> at
>>> org.globus.cog.karajan.workflow.nodes.FlowNode.event(FlowNode.java:335)
>>> at
>>> org.globus.cog.karajan.workflow.events.EventBus.send(EventBus.java:125)
>>> at
>>> org.globus.cog.karajan.workflow.events.EventBus.sendHooked(EventBus.java:99)
>>>
>>> at
>>> org.globus.cog.karajan.workflow.nodes.FlowNode.fireNotificationEvent(FlowNode.java:173)
>>>
>>> at
>>> org.globus.cog.karajan.workflow.nodes.FlowNode.complete(FlowNode.java:299)
>>>
>>> at
>>> org.globus.cog.karajan.workflow.nodes.ExecuteFile.notificationEvent(ExecuteFile.java:163)
>>>
>>> at
>>> org.globus.cog.karajan.workflow.nodes.FlowNode.event(FlowNode.java:335)
>>> at
>>> org.globus.cog.karajan.workflow.events.EventBus.send(EventBus.java:125)
>>> at
>>> org.globus.cog.karajan.workflow.events.EventBus.sendHooked(EventBus.java:99)
>>>
>>> at
>>> org.globus.cog.karajan.workflow.nodes.FlowNode.fireNotificationEvent(FlowNode.java:173)
>>>
>>> at
>>> org.globus.cog.karajan.workflow.nodes.FlowNode.complete(FlowNode.java:299)
>>>
>>> at
>>> org.globus.cog.karajan.workflow.nodes.FlowContainer.post(FlowContainer.java:58)
>>>
>>> at
>>> org.globus.cog.karajan.workflow.nodes.Sequential.startNext(Sequential.java:51)
>>>
>>> at
>>> org.globus.cog.karajan.workflow.nodes.Sequential.childCompleted(Sequential.java:45)
>>>
>>> at
>>> org.globus.cog.karajan.workflow.nodes.Sequential.notificationEvent(Sequential.java:33)
>>>
>>> at
>>> org.globus.cog.karajan.workflow.nodes.FlowNode.event(FlowNode.java:335)
>>> at
>>> org.globus.cog.karajan.workflow.events.EventBus.send(EventBus.java:125)
>>> at
>>> org.globus.cog.karajan.workflow.events.EventBus.sendHooked(EventBus.java:99)
>>>
>>> at
>>> org.globus.cog.karajan.workflow.nodes.FlowNode.fireNotificationEvent(FlowNode.java:173)
>>>
>>> at
>>> org.globus.cog.karajan.workflow.nodes.FlowNode.complete(FlowNode.java:299)
>>>
>>> at
>>> org.globus.cog.karajan.workflow.nodes.FlowContainer.post(FlowContainer.java:58)
>>>
>>> at
>>> org.globus.cog.karajan.workflow.nodes.AbstractSequentialWithArguments.childCompleted(AbstractSequentialWithArguments.java:192)
>>>
>>> at
>>> org.globus.cog.karajan.workflow.nodes.user.UserDefinedElement.childCompleted(UserDefinedElement.java:283)
>>>
>>> at
>>> org.globus.cog.karajan.workflow.nodes.user.SequentialImplicitExecutionUDE.childCompleted(SequentialImplicitExecutionUDE.java:85)
>>>
>>> at
>>> org.globus.cog.karajan.workflow.nodes.Sequential.notificationEvent(Sequential.java:33)
>>>
>>> at
>>> org.globus.cog.karajan.workflow.nodes.FlowNode.event(FlowNode.java:335)
>>> at
>>> org.globus.cog.karajan.workflow.events.EventBus.send(EventBus.java:125)
>>> at
>>> org.globus.cog.karajan.workflow.events.EventBus.sendHooked(EventBus.java:99)
>>>
>>> at
>>> org.globus.cog.karajan.workflow.nodes.FlowNode.fireNotificationEvent(FlowNode.java:173)
>>>
>>> at
>>> org.globus.cog.karajan.workflow.nodes.FlowNode.complete(FlowNode.java:299)
>>>
>>> at
>>> org.globus.cog.karajan.workflow.nodes.If.childCompleted(If.java:30)
>>> at
>>> org.globus.cog.karajan.workflow.nodes.Sequential.notificationEvent(Sequential.java:33)
>>>
>>> at
>>> org.globus.cog.karajan.workflow.nodes.FlowNode.event(FlowNode.java:335)
>>> at
>>> org.globus.cog.karajan.workflow.events.EventBus.send(EventBus.java:125)
>>> at
>>> org.globus.cog.karajan.workflow.events.EventBus.sendHooked(EventBus.java:99)
>>>
>>> at
>>> org.globus.cog.karajan.workflow.nodes.FlowNode.fireNotificationEvent(FlowNode.java:173)
>>>
>>> at
>>> org.globus.cog.karajan.workflow.nodes.FlowNode.complete(FlowNode.java:299)
>>>
>>> at
>>> org.globus.cog.karajan.workflow.nodes.FlowContainer.post(FlowContainer.java:58)
>>>
>>> at
>>> org.globus.cog.karajan.workflow.nodes.functions.AbstractFunction.post(AbstractFunction.java:46)
>>>
>>> at
>>> org.globus.cog.karajan.workflow.nodes.AbstractSequentialWithArguments.childCompleted(AbstractSequentialWithArguments.java:192)
>>>
>>> at
>>> org.globus.cog.karajan.workflow.nodes.Sequential.notificationEvent(Sequential.java:33)
>>>
>>> at
>>> org.globus.cog.karajan.workflow.nodes.FlowNode.event(FlowNode.java:335)
>>> at
>>> org.globus.cog.karajan.workflow.events.EventBus.send(EventBus.java:125)
>>> at
>>> org.globus.cog.karajan.workflow.events.EventBus.sendHooked(EventBus.java:99)
>>>
>>> at
>>> org.globus.cog.karajan.workflow.nodes.FlowNode.fireNotificationEvent(FlowNode.java:173)
>>>
>>> at
>>> org.globus.cog.karajan.workflow.nodes.FlowNode.complete(FlowNode.java:299)
>>>
>>> at
>>> org.globus.cog.karajan.workflow.nodes.FlowContainer.post(FlowContainer.java:58)
>>>
>>> at
>>> org.globus.cog.karajan.workflow.nodes.functions.AbstractFunction.post(AbstractFunction.java:46)
>>>
>>> at
>>> org.globus.cog.karajan.workflow.nodes.Sequential.startNext(Sequential.java:51)
>>>
>>> at
>>> org.globus.cog.karajan.workflow.nodes.Sequential.executeChildren(Sequential.java:27)
>>>
>>> at
>>> org.globus.cog.karajan.workflow.nodes.functions.AbstractFunction.executeChildren(AbstractFunction.java:40)
>>>
>>> at
>>> org.globus.cog.karajan.workflow.nodes.FlowContainer.execute(FlowContainer.java:63)
>>>
>>> at
>>> org.globus.cog.karajan.workflow.nodes.FlowNode.restart(FlowNode.java:240)
>>> at
>>> org.globus.cog.karajan.workflow.nodes.FlowNode.start(FlowNode.java:281)
>>> at
>>> org.globus.cog.karajan.workflow.nodes.FlowNode.controlEvent(FlowNode.java:393)
>>>
>>> at
>>> org.globus.cog.karajan.workflow.nodes.FlowNode.event(FlowNode.java:332)
>>> at
>>> org.globus.cog.karajan.workflow.FlowElementWrapper.event(FlowElementWrapper.java:227)
>>>
>>> at
>>> org.globus.cog.karajan.workflow.events.EventBus.send(EventBus.java:125)
>>> at
>>> org.globus.cog.karajan.workflow.events.EventBus.sendHooked(EventBus.java:99)
>>>
>>> at
>>> org.globus.cog.karajan.workflow.events.EventWorker.run(EventWorker.java:69)
>>>
>>>
>>>
>>>
>>>
>>>
>>> This email is intended only for the use of the individual or entity to
>>> which it is addressed and may contain information that is privileged
>>> and confidential. If the reader of this email message is not the
>>> intended recipient, you are hereby notified that any dissemination,
>>> distribution, or copying of this communication is prohibited. If you
>>> have received this email in error, please notify the sender and
>>> destroy/delete all copies of the transmittal. Thank you.
>>>
>>>
>>> ------------------------------------------------------------------------
>>>
>>> _______________________________________________
>>> Swift-user mailing list
>>> Swift-user at ci.uchicago.edu
>>> http://mail.ci.uchicago.edu/mailman/listinfo/swift-user
>> _______________________________________________
>> Swift-user mailing list
>> Swift-user at ci.uchicago.edu
>> http://mail.ci.uchicago.edu/mailman/listinfo/swift-user
>
>
> ------------------------------
>
> _______________________________________________
> Swift-user mailing list
> Swift-user at ci.uchicago.edu
> http://mail.ci.uchicago.edu/mailman/listinfo/swift-user
>
>
> End of Swift-user Digest, Vol 24, Issue 6
> *****************************************
>
More information about the Swift-user
mailing list