[Swift-user] Cray XE6:: Swift appears to wait till nodes are empty before submitting new jobs.

[David Kelly]

I think something like this should do the trick:

# custom entries
pbs   GATKIntRecalWrapper /lustre/beagle/lpesce/Jason/SwiftRun3/gatkIntRecalWrapper.sh INSTALLED  AMD64::LINUX  ENV::TMP="/dev/shm/GATK_post2013-01-03_11:09:08";GLOBUS::maxwalltime="00:05:00"
pbs   GATKBQRecalWrapper /lustre/beagle/lpesce/Jason/SwiftRun3/gatkBQRecalWrapper.sh INSTALLED  AMD64::LINUX  ENV::TMP="/dev/shm/GATK_post2013-01-03_11:09:08";GLOBUS::maxwalltime="01:05:00"
pbs   PicardMarkDuplWrapper /lustre/beagle/lpesce/Jason/SwiftRun3/picardMarkDuplWrapper.sh INSTALLED  AMD64::LINUX  ENV::TMP="/dev/shm/GATK_post2013-01-03_11:09:08";GLOBUS::maxwalltime="00:15:00"


----- Original Message -----
> From: "Lorenzo Pesce" <lpesce at uchicago.edu>
> To: "David Kelly" <davidk at ci.uchicago.edu>
> Cc: "Joe Urbanski" <jurbanski at ci.uchicago.edu>, swift-user at ci.uchicago.edu
> Sent: Friday, January 4, 2013 2:13:39 PM
> Subject: Re: [Swift-user] Cray XE6:: Swift appears to wait till nodes are empty before submitting new jobs.
> My calls are a bit different, should I tc to something like this?
> 
> # custom entries
> pbs GATKIntRecalWrapper
> /lustre/beagle/lpesce/Jason/SwiftRun3/gatkIntRecalWrapper.sh INSTALLED
> AMD64::LINUX ENV::TMP="/dev/shm/GATK_post2013-01-03_11:09:08"
> GLOBUS::maxwalltime="00:05:00"
> pbs GATKBQRecalWrapper
> /lustre/beagle/lpesce/Jason/SwiftRun3/gatkBQRecalWrapper.sh INSTALLED
> AMD64::LINUX ENV::TMP="/dev/shm/GATK_post2013-01-03_11:09:08"
> GLOBUS::maxwalltime="01:05:00"
> pbs PicardMarkDuplWrapper
> /lustre/beagle/lpesce/Jason/SwiftRun3/picardMarkDuplWrapper.sh
> INSTALLED AMD64::LINUX
> ENV::TMP="/dev/shm/GATK_post2013-01-03_11:09:08"
> GLOBUS::maxwalltime="00:15:00"
> 
> 
> On Jan 4, 2013, at 1:32 PM, David Kelly wrote:
> 
> > Lorenzo,
> >
> > I think one way you can do this is by setting walltimes in tc.data.
> >
> > beagle shortjob /bin/shortjob null null
> > GLOBUS::maxwalltime="00:05:00"
> > beagle longjob /bin/longjob null null GLOBUS::maxwalltime="47:50:00"
> >
> > For multiple jobsPerNode values, you could define two pool entries
> > in sites.xml. Each entry could have a different value for
> > jobsPerNode. Then modify your tc.data to point to the appropriate
> > entry.
> 
> I will look up in the guide to understand this better, but the pointer
> you gave me should be enough. Thanks.
> 
> Thanks a lot David,
> 
> Lorenzo
> 
> >
> > David
> >
> > ----- Original Message -----
> >> From: "Lorenzo Pesce" <lpesce at uchicago.edu>
> >> To: "David Kelly" <davidk at ci.uchicago.edu>
> >> Cc: "Joe Urbanski" <jurbanski at ci.uchicago.edu>,
> >> swift-user at ci.uchicago.edu
> >> Sent: Friday, January 4, 2013 12:21:36 PM
> >> Subject: Re: [Swift-user] Cray XE6:: Swift appears to wait till
> >> nodes are empty before submitting new jobs.
> >> David,
> >>
> >> Thanks a lot for your reply.
> >>
> >>
> >> It is not an iterative call.
> >>
> >> I talked with Mike about it and I think that he solved the problem
> >> (I
> >> still need to test that):
> >> <profile namespace="globus" key="maxTime">172800</profile>
> >> <profile namespace="globus" key="maxwalltime">47:50:00</profile>
> >>
> >> prevented the second app to be send in the same coaster because
> >> there
> >> was not enough time given the maxwalltime setting.
> >>
> >> I just realized how different the app times actually are. Can you
> >> point me to where I can figure out how to instruct the swift that
> >> different apps have different maxwalltimes and can run a different
> >> number of jobs per node?
> >> I might have asked this question already, I know... I slowly make
> >> progress through the coding of all the apps I have to write.
> >>
> >> Thanks a lot,
> >>
> >> Lorenzo
> >>
> >>
> >>
> >> On Jan 4, 2013, at 12:05 PM, David Kelly wrote:
> >>
> >>> Lorenzo,
> >>>
> >>> Could you please show an example of how you are trying to do this?
> >>> Are you using iterate?
> >>>
> >>> ----- Original Message -----
> >>>> From: "Lorenzo Pesce" <lpesce at uchicago.edu>
> >>>> To: swift-user at ci.uchicago.edu
> >>>> Cc: "Joe Urbanski" <jurbanski at ci.uchicago.edu>
> >>>> Sent: Thursday, January 3, 2013 3:48:31 PM
> >>>> Subject: [Swift-user] Cray XE6:: Swift appears to wait till nodes
> >>>> are empty before submitting new jobs.
> >>>> I am making some small tests of sequential jobs and it seems like
> >>>> once
> >>>> step 1 is finished, step 2 doesn't start until enough step 1s
> >>>> have
> >>>> completed.
> >>>> I assumed that swift would be able to send a jobs to a node
> >>>> before
> >>>> completion.
> >>>> Does it have to do with submission settings?
> >>>>
> >>>> Lorenzo
> >>>> _______________________________________________
> >>>> Swift-user mailing list
> >>>> Swift-user at ci.uchicago.edu
> >>>> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user