[Swift-devel] Re: GRAM and Swift discussion this week?

[Tim Freeman]

On Tue, 22 May 2007 14:34:07 -0500
Ioan Raicu <iraicu at cs.uchicago.edu> wrote:

> the many logs that we generate are quite hard for people to
>       follow, and keep track of what each one contains; we fixed this by
>       developing a GUI that  can connect to the GT4 container remotely
>       and display relevant information!

Is that something that could be used for other GT services?


>           * the provisioner currently creates a new thread for every job
>             (resource allocation) it sends to GRAM4
>                 o depending on which allocation strategy is used, this
>                   might/might not be a problem on TG nodes
>                 o in theory, we don't want more than 100 or so GRAM4
>                   jobs in parallel running, but  if we choose the policy
>                   in which each job allocates a single machine, then we
>                   can easily surpass 100 jobs in parallel... all the
>                   other policies, would be able to allocate 1K+, even
>                   10K+ machines with less than 100 jobs in parallel, so
>                   it could work perfectly fine even with the current
>                   implementation; in the long run, this might be able to
>                   be changed to a pool of threads in the provisioner!

Are these threads just waiting on notifications?  

If so: you should be able to reduce this to one thread by subscribing with the
same notification consumer EPR for each GRAM job and demuxing the result (demux
based on the producer EPR that is passed to the class implementing
NotifyCallback).  That way the thread that creates the GRAM job can disappear
once it is done with the create call. 

Tim