[Swift-devel] Re: [Swft] Q about throttling

[Ian Foster]

thanks... Ian.

Mihael Hategan wrote:
> On Thu, 2007-06-21 at 10:37 -0500, Ian Foster wrote:
>   
>> Well, if there is some concern that throttling might be a problem,
>> then trying a run with it turned off seems good.
>>
>> I'm gathering from this exchange that this is not possible?
>>     
>
> As Tibi guessed, it is done by using sufficiently large numbers for the
> throttles. There is no way to automatically turn off all throttles, but
> I guess it could be done.
>
> Mihael
>
>   
>> Ben Clifford wrote: 
>>     
>>> But that isn't the base problem being investigated, right?
>>>
>>> On Thu, 21 Jun 2007, Ian Foster wrote:
>>>
>>>   
>>>       
>>>> My original question was whether we could turn throttling off altogether. I'm
>>>> not sure if that was answered?
>>>>
>>>> Tiberiu Stef-Praun wrote:
>>>>     
>>>>         
>>>>> I did not look very deep into the throttling, mainly because I have to
>>>>> wait for my turn at using the Argonne cluster because of the large
>>>>> reservations that Ioan does for MolDyn
>>>>>
>>>>> Anyway, here is my experience (which Ian asked me to write down, but
>>>>> I'm still trying to improve on):
>>>>> - whatever one asks from Falkon, one seems to get, with the caveat
>>>>> that Falkon might release nodes when configured to look at an idle
>>>>> timer. In the case of the Econ workflow, I had 26 long running jobs,
>>>>> so I requested 30 nodes (which Falkon got for me)
>>>>> - there is a swift config file, $DVS_HOME/libexec/scheduler.xml in which I
>>>>> set
>>>>> <property name="jobThrottle" value="30"/>, but that seemed not to be
>>>>> enough to get all my 26 jobs running at the same time (as illustrated
>>>>> by the graphing of the Falkon log that Ioan showed me).
>>>>> - there are some other throttling parameters in
>>>>> $VDS_HOME/etc/swift.properties (which I also set to 30)
>>>>>
>>>>> The general observation is that I needed to modify the scheduler.xml
>>>>> config file, and I need to set larger throttle values that the limit
>>>>> of workers requested.
>>>>> In the current scheme (simply add Falkon to Swift as a provider) the
>>>>> Swift scheduler (the weighted site selection algorithm) adversely
>>>>> influences the optimal execution of the workflow.
>>>>> There might be other parameters to work with, but my opinion is that
>>>>> we should use a different (non-throttling) scheduler in combination
>>>>> with Falkon
>>>>>
>>>>> Tibi
>>>>>
>>>>> On 6/21/07, Mike Wilde <wilde at mcs.anl.gov> wrote:
>>>>>       
>>>>>           
>>>>>> Ive had the same question - it seems that throttling is also problematic
>>>>>> for
>>>>>> Tibi in the econ workflow.
>>>>>>
>>>>>> Tibi, since you have looked pretty deeply into it, could you write up a
>>>>>> desription on how the algorithm works and how the parameters affect it.
>>>>>> Mihael, when you are back on central time next week, could you work with
>>>>>> TIbi on
>>>>>> this?  If its not already, this should be part of the Swift documentation.
>>>>>>
>>>>>> Then we should work on getting high-performance settings for the different
>>>>>> runtime environments we use, in particular Falkon as Ian asks.
>>>>>>
>>>>>> - Mike
>>>>>>
>>>>>>
>>>>>> Ian Foster wrote, On 6/21/2007 6:50 AM:
>>>>>>         
>>>>>>             
>>>>>>> Hi,
>>>>>>>
>>>>>>> I don't fully understand how throttling works in Swift/Karajan. However,
>>>>>>> I understand that even when using Falkon, we may be doing some
>>>>>>> throttling. Is there a reason to do that in this case, given that Falkon
>>>>>>> can maintain large numbers of tasks just fine?
>>>>>>>
>>>>>>> I ask this because in a recent MolDyn run, there seemed to be some
>>>>>>> uncertainty as to whether throttling was slowing down job dispatch. If
>>>>>>> we could turn it off altogether, that question would presumably go away.
>>>>>>>
>>>>>>> Ian.
>>>>>>>
>>>>>>>
>>>>>>>           
>>>>>>>               
>>>>>> -- 
>>>>>> Mike Wilde
>>>>>> Computation Institute, University of Chicago
>>>>>> Math & Computer Science Division
>>>>>> Argonne National Laboratory
>>>>>> Argonne, IL   60439    USA
>>>>>> tel 630-252-7497 fax 630-252-1997
>>>>>>
>>>>>>         
>>>>>>             
>>>   
>>>       
>> -- 
>>
>>    Ian Foster, Director, Computation Institute
>> Argonne National Laboratory & University of Chicago
>> Argonne: MCS/221, 9700 S. Cass Ave, Argonne, IL 60439
>> Chicago: Rm 405, 5640 S. Ellis Ave, Chicago, IL 60637
>> Tel: +1 630 252 4619.  Web: www.ci.uchicago.edu.
>>       Globus Alliance: www.globus.org.
>>     
>
>   

-- 

   Ian Foster, Director, Computation Institute
Argonne National Laboratory & University of Chicago
Argonne: MCS/221, 9700 S. Cass Ave, Argonne, IL 60439
Chicago: Rm 405, 5640 S. Ellis Ave, Chicago, IL 60637
Tel: +1 630 252 4619.  Web: www.ci.uchicago.edu.
      Globus Alliance: www.globus.org.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-devel/attachments/20070623/4c11c3a0/attachment.html>