[Swift-devel] Re: [Swft] Q about throttling

[Mihael Hategan]

On Thu, 2007-06-21 at 10:37 -0500, Ian Foster wrote:
> Well, if there is some concern that throttling might be a problem,
> then trying a run with it turned off seems good.
> 
> I'm gathering from this exchange that this is not possible?

As Tibi guessed, it is done by using sufficiently large numbers for the
throttles. There is no way to automatically turn off all throttles, but
I guess it could be done.

Mihael

> 
> Ben Clifford wrote: 
> > But that isn't the base problem being investigated, right?
> > 
> > On Thu, 21 Jun 2007, Ian Foster wrote:
> > 
> >   
> > > My original question was whether we could turn throttling off altogether. I'm
> > > not sure if that was answered?
> > > 
> > > Tiberiu Stef-Praun wrote:
> > >     
> > > > I did not look very deep into the throttling, mainly because I have to
> > > > wait for my turn at using the Argonne cluster because of the large
> > > > reservations that Ioan does for MolDyn
> > > > 
> > > > Anyway, here is my experience (which Ian asked me to write down, but
> > > > I'm still trying to improve on):
> > > > - whatever one asks from Falkon, one seems to get, with the caveat
> > > > that Falkon might release nodes when configured to look at an idle
> > > > timer. In the case of the Econ workflow, I had 26 long running jobs,
> > > > so I requested 30 nodes (which Falkon got for me)
> > > > - there is a swift config file, $DVS_HOME/libexec/scheduler.xml in which I
> > > > set
> > > > <property name="jobThrottle" value="30"/>, but that seemed not to be
> > > > enough to get all my 26 jobs running at the same time (as illustrated
> > > > by the graphing of the Falkon log that Ioan showed me).
> > > > - there are some other throttling parameters in
> > > > $VDS_HOME/etc/swift.properties (which I also set to 30)
> > > > 
> > > > The general observation is that I needed to modify the scheduler.xml
> > > > config file, and I need to set larger throttle values that the limit
> > > > of workers requested.
> > > > In the current scheme (simply add Falkon to Swift as a provider) the
> > > > Swift scheduler (the weighted site selection algorithm) adversely
> > > > influences the optimal execution of the workflow.
> > > > There might be other parameters to work with, but my opinion is that
> > > > we should use a different (non-throttling) scheduler in combination
> > > > with Falkon
> > > > 
> > > > Tibi
> > > > 
> > > > On 6/21/07, Mike Wilde <wilde at mcs.anl.gov> wrote:
> > > >       
> > > > > Ive had the same question - it seems that throttling is also problematic
> > > > > for
> > > > > Tibi in the econ workflow.
> > > > > 
> > > > > Tibi, since you have looked pretty deeply into it, could you write up a
> > > > > desription on how the algorithm works and how the parameters affect it.
> > > > > Mihael, when you are back on central time next week, could you work with
> > > > > TIbi on
> > > > > this?  If its not already, this should be part of the Swift documentation.
> > > > > 
> > > > > Then we should work on getting high-performance settings for the different
> > > > > runtime environments we use, in particular Falkon as Ian asks.
> > > > > 
> > > > > - Mike
> > > > > 
> > > > > 
> > > > > Ian Foster wrote, On 6/21/2007 6:50 AM:
> > > > >         
> > > > > > Hi,
> > > > > > 
> > > > > > I don't fully understand how throttling works in Swift/Karajan. However,
> > > > > > I understand that even when using Falkon, we may be doing some
> > > > > > throttling. Is there a reason to do that in this case, given that Falkon
> > > > > > can maintain large numbers of tasks just fine?
> > > > > > 
> > > > > > I ask this because in a recent MolDyn run, there seemed to be some
> > > > > > uncertainty as to whether throttling was slowing down job dispatch. If
> > > > > > we could turn it off altogether, that question would presumably go away.
> > > > > > 
> > > > > > Ian.
> > > > > > 
> > > > > > 
> > > > > >           
> > > > > -- 
> > > > > Mike Wilde
> > > > > Computation Institute, University of Chicago
> > > > > Math & Computer Science Division
> > > > > Argonne National Laboratory
> > > > > Argonne, IL   60439    USA
> > > > > tel 630-252-7497 fax 630-252-1997
> > > > > 
> > > > >         
> > 
> >   
> 
> -- 
> 
>    Ian Foster, Director, Computation Institute
> Argonne National Laboratory & University of Chicago
> Argonne: MCS/221, 9700 S. Cass Ave, Argonne, IL 60439
> Chicago: Rm 405, 5640 S. Ellis Ave, Chicago, IL 60637
> Tel: +1 630 252 4619.  Web: www.ci.uchicago.edu.
>       Globus Alliance: www.globus.org.