<HTML dir=ltr><HEAD><TITLE>RE: [Swift-user] job waiting</TITLE>
<META http-equiv=Content-Type content="text/html; charset=unicode">
<META content="MSHTML 6.00.6000.16825" name=GENERATOR></HEAD>
<BODY>
<DIV id=idOWAReplyText4082 dir=ltr>
<DIV dir=ltr><FONT face=Arial color=#000000 size=2>Hi Ben,</FONT></DIV>
<DIV dir=ltr><FONT face=Arial size=2></FONT> </DIV>
<DIV dir=ltr><FONT face=Arial size=2>Yesterday, I tested my application a few times on NCSA mercury only with coaster and with the specification of globus::maxwalltime=50 in tc.data. Similar to previous try, in several runs, the application keeps waiting after 4076, 4052, 4099, 4048, 4051 successful returns respectively. Does this relate to my setting? The log for the last run is at: </FONT></DIV>
<DIV dir=ltr> </DIV>
<DIV dir=ltr><FONT face=Arial size=2>/home/yuechen/PTMap2/PTMap2-unmod-20090422-1036-07c88p47.log<BR></DIV></FONT>
<DIV dir=ltr><FONT face=Arial size=2>I started to receive email with the following content after about 10 min of execution, </FONT></DIV>
<DIV dir=ltr><FONT face=Arial size=2></FONT> </DIV>
<DIV dir=ltr><FONT face=Arial size=2>/////////</FONT></DIV>
<DIV dir=ltr><FONT size=2>PBS Job Id: 1947957.tg-master.ncsa.teragrid.org<BR>Job Name: null<BR>job deleted<BR>Job deleted at request of root@tg-master.ncsa.teragrid.org<BR>MOAB_INFO: job exceeded wallclock limit</FONT><BR><FONT face=Arial size=2>/////////</FONT></DIV>
<DIV dir=ltr><FONT face=Arial size=2></FONT> </DIV>
<DIV dir=ltr><FONT face=Arial size=2>However, Swift did not indicate any job failure, so should I worry about the success of those jobs? </FONT></DIV>
<DIV dir=ltr><FONT face=Arial size=2></FONT> </DIV>
<DIV dir=ltr><FONT face=Arial size=2>I also tried NCSA mercury only without coaster, but the submitted jobs do not seem to return successfully. I notice that if I use coaster, typicaly max number jobs I have on NCSA is about 130, but if I do not use coaster, I can have more than 300 jobs queued on NCSA computer. Is this related with the throttle setting?</FONT></DIV>
<DIV dir=ltr><FONT face=Arial size=2></FONT> </DIV>
<DIV dir=ltr><FONT face=Arial size=2>I also tried SDSC dtf server without coaster, but the jobs submitted do not get started on SDSC dtf server. Instead, I got many error messages like the following. Should I contact teragrid for these errors?</FONT></DIV>
<DIV dir=ltr><FONT face=Arial size=2></FONT> </DIV>
<DIV dir=ltr><FONT face=Arial size=2>Progress: Stage in:93 Submitted:3710 Active:45 Stage out:4 Finished successfully:230 Failed but can retry:45<BR>Failed to transfer wrapper log from PTMap2-unmod-20090421-2214-e6ssbye5/info/f on SDSC_dtf_prews_pbs<BR>Failed to transfer wrapper log from PTMap2-unmod-20090421-2214-e6ssbye5/info/o on SDSC_dtf_prews_pbs<BR>Failed to transfer wrapper log from PTMap2-unmod-20090421-2214-e6ssbye5/info/t on SDSC_dtf_prews_pbs<BR>Failed to transfer wrapper log from PTMap2-unmod-20090421-2214-e6ssbye5/info/o on SDSC_dtf_prews_pbs<BR>Failed to transfer wrapper log from PTMap2-unmod-20090421-2214-e6ssbye5/info/t on SDSC_dtf_prews_pbs<BR>Failed to transfer wrapper log from PTMap2-unmod-20090421-2214-e6ssbye5/info/t on SDSC_dtf_prews_pbs</FONT></DIV>
<DIV dir=ltr><FONT face=Arial size=2></FONT> </DIV>
<DIV dir=ltr><FONT face=Arial size=2>The following is my sites.xml content for NCSA mercury with and without coaster and SDSC DTF:</FONT></DIV>
<DIV dir=ltr><FONT face=Arial size=2></FONT> </DIV>
<DIV dir=ltr><FONT face=Arial size=2> <pool handle="NCSAMERCURY"><BR> <gridftp url="gsiftp://gridftp-hg.ncsa.teragrid.org"/><BR> <execution provider="coaster" url="grid-hg.ncsa.teragrid.org" jobManager="gt2:PBS"/><BR> <workdirectory>/gpfs_scratch1/yuechen/swiftwork</workdirectory><BR> </pool><BR> <pool handle="NCSAMERCURY_nocoaster"><BR> <gridftp url="gsiftp://gridftp-hg.ncsa.teragrid.org"/><BR> <jobmanager universe="vanilla" url="grid-hg.ncsa.teragrid.org:2119/jobmanager-pbs" major="2" /><BR> <workdirectory>/gpfs_scratch1/yuechen/swiftwork</workdirectory><BR> </pool><BR> <pool handle="SDSC_dtf_prews_pbs"><BR> <gridftp url="gsiftp://tg-gridftp.sdsc.teragrid.org:2811/" /><BR> <jobmanager universe="vanilla" url="tg-login1.sdsc.teragrid.org:2119/jobmanager-pbs" major="2" /><BR> <workdirectory >/gpfs-wan/scratch/yuechen</workdirectory><BR> <profile namespace="globus" key="queue">fast</profile><BR> </pool></FONT></DIV>
<DIV dir=ltr><FONT face=Arial size=2></FONT> </DIV>
<DIV dir=ltr><FONT face=Arial size=2>The swift script I used is at:</FONT></DIV>
<DIV dir=ltr><FONT face=Arial size=2></FONT> </DIV>
<DIV dir=ltr><FONT face=Arial size=2>/home/yuechen/PTMap2/PTMap2-unmod.swift</FONT></DIV>
<DIV dir=ltr><FONT face=Arial size=2></FONT> </DIV>
<DIV dir=ltr><FONT face=Arial size=2>The tc.data I used is:</FONT></DIV>
<DIV dir=ltr><FONT face=Arial size=2></FONT> </DIV>
<DIV dir=ltr><FONT face=Arial size=2>/home/yuechen/PTMap2/tc.data</FONT></DIV>
<DIV dir=ltr><FONT face=Arial size=2></FONT> </DIV>
<DIV dir=ltr><FONT face=Arial size=2>I will start to try other servers to see if I can run all jobs successfully.</FONT></DIV>
<DIV dir=ltr><FONT face=Arial size=2></FONT> </DIV>
<DIV dir=ltr><FONT face=Arial size=2>Thank you very much for help!</FONT></DIV>
<DIV dir=ltr><FONT face=Arial size=2></FONT> </DIV>
<DIV dir=ltr><FONT face=Arial size=2>Chen, Yue</FONT></DIV>
<DIV dir=ltr> </DIV><FONT face=Arial size=2></FONT></DIV>
<P><FONT face=Arial size=2></FONT> </P>
<P><FONT face=Arial size=2> </P>
<DIV dir=ltr><BR></DIV></FONT>
<DIV dir=ltr> </DIV>
<DIV dir=ltr> </DIV>
<DIV dir=ltr><FONT face=Arial size=2></FONT> </DIV>
<DIV dir=ltr><BR>
<HR tabIndex=-1>
<FONT face=Tahoma size=2><B>From:</B> Ben Clifford [mailto:benc@hawaga.org.uk]<BR><B>Sent:</B> Sun 4/19/2009 2:07 AM<BR><B>To:</B> Yue, Chen - BMD<BR><B>Cc:</B> swift user<BR><B>Subject:</B> RE: [Swift-user] job waiting<BR></FONT><BR></DIV>
<DIV><BR>
<P><FONT size=2>On Sat, 18 Apr 2009, Yue, Chen - BMD wrote:<BR><BR>> Thanks for answering my question. This phenomena occur after half an<BR>> hour of execution. If all the jobs finish execution at original speed,<BR>> it would probably take not more than 40 min. How the system figure out<BR>> that some jobs will take more than 1 hour? Should I request more time<BR>> when I execute "grid-proxy-init"?<BR><BR>Not with grid-proxy-init. You can specify a parameter called maxwalltime<BR>in your sites file or your tc.data file that will tell Swift an upper<BR>bound on how long your job will run. In Swift 0.8, coasters assume<BR>something like 10 minutes if you do not specify a walltime, so you will<BR>run into trouble.<BR><BR>For example, change the null at the end of your tc.data lines to<BR>globus::maxwalltime=50 to mean 50 minutes maxwalltime.<BR><BR>There has been work done on coasters since Swift 0.8, and so Mihael may<BR>have some other recommendations.<BR><BR>> I did not change the default throttles. How much is more appropriate?<BR>> The total number of jobs in my application typically run between 4000<BR>> and 30000 and typically each job can be finished within a couple of<BR>> minutes.<BR><BR>Where is your Swift installation? I would liek to look at it.<BR><BR>--<BR><BR></FONT></P></DIV></BODY></HTML>