[Swift-devel] Re: [metrics-dev] feasability of collecting swift usage stats through the globus usage stats mechanism (fwd)

[Ben Clifford]

Its been mooted a few times over the last year or so, so I enquired with 
metrics-dev about using the globus usage stats stuff for very basic swift 
usage info. Here's the response below in case anyone is interested in 
following up.

---------- Forwarded message ----------
Date: Wed, 17 Jun 2009 09:31:19 -0600
From: Lee Liming <liming at mcs.anl.gov>
To: Ben Clifford <benc at hawaga.org.uk>
Cc: metrics-dev at globus.org
Subject: Re: [metrics-dev] feasability of collecting swift usage stats through
    the globus usage stats mechanism

Absolutely yes.  See http://dev.globus.org/wiki/Incubator/Metrics#FAQ for the
information you ask about here.  I think all of the topics are covered in the
FAQ and linked docs.  In summary, the code and mechanism needed to do this is
totally open and available.  Using the CDIGS listener service itself requires
coordination with the person who operates it (currently Joe Bester) but it is
do-able.  Running your own listener requires no coordination and is a good
option.

You may want to consider operating your own listener service.  The "global"
CDIGS listener service is experiencing growing pains at the moment, and is not
currently (this week, maybe next) available for you to experiment with because
it's being serviced.  It's also pretty heavily loaded, so your performance
(e.g., report generation) will not be stellar.  It's quite easy to bring up a
listener service, and if you have control of your code deployment (the code
being reported on), you can easily configure where it sends reports.  You could
even have it report to multiple listeners, such as a Swift-specific listener
*and* the CDIGS listener.

The largest challenge in running your own listener would be sustaining its
operation over time, and you will have to think a bit about what your
requirements are in that area.  (How badly you want to have *every* usage
report.)  If it's not vital that you have each and every usage report (but get a
good sampling, for example, and keep track of when you were vs. weren't
listening), then this should be a pretty lightweight thing to do.  CDIGS has
tried to be meticulous about high availability and not losing any data, and our
record over several years is quite good, but it can be a high-stress enterprise
and requires significant attention for short (mostly unpredictable) times.  I am
not 100% sure we know the return we're getting for such effort.

        --- Lee

On Jun 17, 2009, at 2:39 AM, Ben Clifford wrote:

> 
> I would like to investigate the feasability of collecting basic usage
> stats for Swift through the globus usage stats mechanism. Specifically:
> 
> 1) is the usage stats mechanism even open to other dev.globus projects
>   (sociopolitically, not technically)
> 
> 2) what actual code needs to be added to the client to send usage packets
>   (is there a packaged library?)
> 
> 3) what needs to happen at the receiving end to get useful reports.
> 
> The sort of information I suspect being logged would be (for each run
> where Swift ends naturally):
> 
> i) svn revision number
> ii) number of tasks executed
> 
> --