<HTML dir=ltr><HEAD><TITLE>RE: [Swift-user] job waiting</TITLE>
<META http-equiv=Content-Type content="text/html; charset=unicode">
<META content="MSHTML 6.00.6000.16825" name=GENERATOR></HEAD>
<BODY>
<DIV id=idOWAReplyText66149 dir=ltr>
<DIV dir=ltr><FONT face=Arial color=#000000 size=2>Hi Mihael and Ben,</FONT></DIV>
<DIV dir=ltr><FONT face=Arial size=2></FONT> </DIV>
<DIV dir=ltr><FONT face=Arial size=2>Thanks for your information. The new version of coasters works very well on NCSA mercury and I don't receive those email any more. But I run into some problem with SDSC server. I will send separate email tomorrow after I get response from SDSC people.</FONT></DIV>
<DIV dir=ltr><FONT face=Arial size=2></FONT> </DIV>
<DIV dir=ltr><FONT face=Arial size=2>Best,</FONT></DIV>
<DIV dir=ltr><FONT face=Arial size=2>Chen, Yue</FONT></DIV>
<DIV dir=ltr> </DIV></DIV>
<DIV dir=ltr><BR>
<HR tabIndex=-1>
<FONT face=Tahoma size=2><B>From:</B> Mihael Hategan [mailto:hategan@mcs.anl.gov]<BR><B>Sent:</B> Wed 4/22/2009 11:44 AM<BR><B>To:</B> Yue, Chen - BMD<BR><B>Cc:</B> Ben Clifford; swift user<BR><B>Subject:</B> RE: [Swift-user] job waiting<BR></FONT><BR></DIV>
<DIV>
<P><FONT size=2>This behavior was observed previously with the version you have. I<BR>strongly recommend upgrading to the version Ben mentions.<BR><BR>On Wed, 2009-04-22 at 11:25 -0500, Yue, Chen - BMD wrote:<BR>> Hi Ben,<BR>> <BR>> Yesterday, I tested my application a few times on NCSA mercury only<BR>> with coaster and with the specification of globus::maxwalltime=50 in<BR>> tc.data. Similar to previous try, in several runs, the application<BR>> keeps waiting after 4076, 4052, 4099, 4048, 4051 successful returns<BR>> respectively. Does this relate to my setting? The log for the last run<BR>> is at:<BR>> <BR>> /home/yuechen/PTMap2/PTMap2-unmod-20090422-1036-07c88p47.log<BR>><BR>> I started to receive email with the following content after about 10<BR>> min of execution, <BR>> <BR>> /////////<BR>> PBS Job Id: 1947957.tg-master.ncsa.teragrid.org<BR>> Job Name: null<BR>> job deleted<BR>> Job deleted at request of root@tg-master.ncsa.teragrid.org<BR>> MOAB_INFO: job exceeded wallclock limit<BR>> /////////<BR>> <BR>> However, Swift did not indicate any job failure, so should I worry<BR>> about the success of those jobs?<BR>> <BR>> I also tried NCSA mercury only without coaster, but the submitted jobs<BR>> do not seem to return successfully. I notice that if I use coaster,<BR>> typicaly max number jobs I have on NCSA is about 130, but if I do not<BR>> use coaster, I can have more than 300 jobs queued on NCSA computer. Is<BR>> this related with the throttle setting?<BR>> <BR>> I also tried SDSC dtf server without coaster, but the jobs submitted<BR>> do not get started on SDSC dtf server. Instead, I got many error<BR>> messages like the following. Should I contact teragrid for these<BR>> errors?<BR>> <BR>> Progress: Stage in:93 Submitted:3710 Active:45 Stage out:4 Finished<BR>> successfully:230 Failed but can retry:45<BR>> Failed to transfer wrapper log from<BR>> PTMap2-unmod-20090421-2214-e6ssbye5/info/f on SDSC_dtf_prews_pbs<BR>> Failed to transfer wrapper log from<BR>> PTMap2-unmod-20090421-2214-e6ssbye5/info/o on SDSC_dtf_prews_pbs<BR>> Failed to transfer wrapper log from<BR>> PTMap2-unmod-20090421-2214-e6ssbye5/info/t on SDSC_dtf_prews_pbs<BR>> Failed to transfer wrapper log from<BR>> PTMap2-unmod-20090421-2214-e6ssbye5/info/o on SDSC_dtf_prews_pbs<BR>> Failed to transfer wrapper log from<BR>> PTMap2-unmod-20090421-2214-e6ssbye5/info/t on SDSC_dtf_prews_pbs<BR>> Failed to transfer wrapper log from<BR>> PTMap2-unmod-20090421-2214-e6ssbye5/info/t on SDSC_dtf_prews_pbs<BR>> <BR>> The following is my sites.xml content for NCSA mercury with and<BR>> without coaster and SDSC DTF:<BR>> <BR>> <pool handle="NCSAMERCURY"><BR>> <gridftp url="gsiftp://gridftp-hg.ncsa.teragrid.org"/><BR>> <execution provider="coaster" url="grid-hg.ncsa.teragrid.org"<BR>> jobManager="gt2:PBS"/><BR>> <workdirectory>/gpfs_scratch1/yuechen/swiftwork</workdirectory><BR>> </pool><BR>> <pool handle="NCSAMERCURY_nocoaster"><BR>> <gridftp url="gsiftp://gridftp-hg.ncsa.teragrid.org"/><BR>> <jobmanager universe="vanilla"<BR>> url="grid-hg.ncsa.teragrid.org:2119/jobmanager-pbs" major="2" /><BR>> <workdirectory>/gpfs_scratch1/yuechen/swiftwork</workdirectory><BR>> </pool><BR>> <pool handle="SDSC_dtf_prews_pbs"><BR>> <gridftp url="gsiftp://tg-gridftp.sdsc.teragrid.org:2811/" /><BR>> <jobmanager universe="vanilla"<BR>> url="tg-login1.sdsc.teragrid.org:2119/jobmanager-pbs" major="2" /><BR>> <workdirectory >/gpfs-wan/scratch/yuechen</workdirectory><BR>> <profile namespace="globus" key="queue">fast</profile><BR>> </pool><BR>> <BR>> The swift script I used is at:<BR>> <BR>> /home/yuechen/PTMap2/PTMap2-unmod.swift<BR>> <BR>> The tc.data I used is:<BR>> <BR>> /home/yuechen/PTMap2/tc.data<BR>> <BR>> I will start to try other servers to see if I can run all jobs<BR>> successfully.<BR>> <BR>> Thank you very much for help!<BR>> <BR>> Chen, Yue<BR>> <BR>> <BR>><BR>> <BR>><BR>><BR>><BR>> <BR>> <BR>> <BR>><BR>><BR>> ______________________________________________________________________<BR>> From: Ben Clifford [<A href="mailto:benc@hawaga.org.uk">mailto:benc@hawaga.org.uk</A>]<BR>> Sent: Sun 4/19/2009 2:07 AM<BR>> To: Yue, Chen - BMD<BR>> Cc: swift user<BR>> Subject: RE: [Swift-user] job waiting<BR>><BR>><BR>><BR>> On Sat, 18 Apr 2009, Yue, Chen - BMD wrote:<BR>><BR>> > Thanks for answering my question. This phenomena occur after half an<BR>> > hour of execution. If all the jobs finish execution at original<BR>> speed,<BR>> > it would probably take not more than 40 min. How the system figure<BR>> out<BR>> > that some jobs will take more than 1 hour? Should I request more<BR>> time<BR>> > when I execute "grid-proxy-init"?<BR>><BR>> Not with grid-proxy-init. You can specify a parameter called<BR>> maxwalltime<BR>> in your sites file or your tc.data file that will tell Swift an upper<BR>> bound on how long your job will run. In Swift 0.8, coasters assume<BR>> something like 10 minutes if you do not specify a walltime, so you<BR>> will<BR>> run into trouble.<BR>><BR>> For example, change the null at the end of your tc.data lines to<BR>> globus::maxwalltime=50 to mean 50 minutes maxwalltime.<BR>><BR>> There has been work done on coasters since Swift 0.8, and so Mihael<BR>> may<BR>> have some other recommendations.<BR>><BR>> > I did not change the default throttles. How much is more<BR>> appropriate?<BR>> > The total number of jobs in my application typically run between<BR>> 4000<BR>> > and 30000 and typically each job can be finished within a couple of<BR>> > minutes.<BR>><BR>> Where is your Swift installation? I would liek to look at it.<BR>><BR>> --<BR>><BR>><BR>><BR>><BR>> _______________________________________________<BR>> Swift-user mailing list<BR>> Swift-user@ci.uchicago.edu<BR>> <A href="http://mail.ci.uchicago.edu/mailman/listinfo/swift-user">http://mail.ci.uchicago.edu/mailman/listinfo/swift-user</A><BR><BR></FONT></P></DIV><DIV> </DIV><br><br>This email is intended only for the use of the individual or entity to which it is addressed and may contain information that is privileged and confidential. If the reader of this email message is not the intended recipient, you are hereby notified that any dissemination, distribution, or copying of this communication is prohibited. If you have received this email in error, please notify the sender and destroy/delete all copies of the transmittal. Thank you.<br></BODY></HTML>