[AG-DEV] Re-using node listing data

[Andrew Rowley]

Hi,

My big worry is that if the single site goes down, do we have none of the information?

The idea of having backend databases in separate locations is that the main site will contact these databases for the node listing.  The information will be available from the main site on a read-only basis.  If a node wants to update the information, they can do this from the site that owns the information e.g. AGSC nodes will be able to edit their listing after authenticating on an AGSC site.  Also, the AGSC will be able to update QA testing information on our own site, and list the AGSC nodes on our own site (this is essential).

Please note that the reason that I bring up these concerns is that the AGSC is running a UK national service.  We are paid to provide this service and we have to make sure that the services we rely on can be provided within the AGSC hours of operation without failure (or can be brought up again quickly with very short notice).  Therefore, if this cannot be guaranteed to us, we cannot use the services.  We have been burnt by services going down in the US before, and therefore we don't want to rely on these services without some sort of guarantee.  

I can understand that as this is being offered on a voluntary basis, and I do appreciate the work that has gone into this.  However, we must put our users first when considering moving to a service like this.

Andrew :)

============================================
Access Grid Support Centre,
RSS Group,
Manchester Computing,
Kilburn Building,
University of Manchester,
Oxford Road,
Manchester,
M13 9PL,
UK
Tel: +44(0)161-275 0685
Email: Andrew.Rowley at manchester.ac.uk 
 

-----Original Message-----
From: Jason Bell [mailto:j.bell at cqu.edu.au] 
Sent: 29 May 2007 10:13
To: Paul Bonnett; Andrew.Rowley at manchester.ac.uk; Brian Corrie
Cc: AG-DEV
Subject: RE: [AG-DEV] Re-using node listing data

G'day all

I think I might ask Tom to comment on this, as he knows more about the
practically of data exchange within the Drupal database than I do.

The concern I have is having a centralise system is what we are trying
to achieve here, I.E. a single location that someone can go to, that
contains all of the AG nodes for example.  If we obtain data from
different sites, does that mean if a site goes down, we only have half
the info...  Or that we have to visit multiple sites???

Anyway, my 2 cents worth.

Cheers,
Jason.

-----Original Message-----
From: Paul Bonnett [mailto:P.Bonnett at ukerna.ac.uk] 
Sent: Tuesday, 29 May 2007 6:49 PM
To: Andrew.Rowley at manchester.ac.uk; Jason Bell; Brian Corrie
Cc: AG-DEV
Subject: RE: [AG-DEV] Re-using node listing data

All,

Am I missing something here? Could there not be a query system mirrored
in UK, USA & AU which polled the 3 databases for information on a live
basis so we're not copying data back and forth? You would not be able to
access the data directly, but only via your 'home' body which would
authenticate you and allow access to information.

I hate the idea of 'copies' of databases around the world; and I
actually think it may contravene the UKERNA policies, but I will have to
check.

Regards,
Paul 

-----Original Message-----
From: owner-ag-dev at mcs.anl.gov [mailto:owner-ag-dev at mcs.anl.gov] On
Behalf Of Andrew Rowley
Sent: 29 May 2007 09:14
To: Jason Bell; Brian Corrie
Cc: AG-DEV
Subject: RE: [AG-DEV] Re-using node listing data

Hi,

Mirroring could be complicated as it will probably only allow updating
of this from the US site - this would stop our day-to-day activities if
it went down and so would be unacceptable for us to rely on without a
guarantee of service.

Does the site use some sort of SQL database backend?  If so, couldn't
you trawl the info from various SQL databases, rather than just one?  As
well as this, you could have the information trawled stored in the
central accessgrid.org database in case the other sites were down...

Andrew :)

============================================
Access Grid Support Centre,
RSS Group,
Manchester Computing,
Kilburn Building,
University of Manchester,
Oxford Road,
Manchester,
M13 9PL,
UK
Tel: +44(0)161-275 0685
Email: Andrew.Rowley at manchester.ac.uk 
 

-----Original Message-----
From: Jason Bell [mailto:j.bell at cqu.edu.au]
Sent: 28 May 2007 00:29
To: Andrew.Rowley at manchester.ac.uk; Brian Corrie
Cc: AG-DEV
Subject: RE: [AG-DEV] Re-using node listing data

G'day All

Well, I must say that it appears I may have opened a can of worms here
:-)

Anyway, I don't know if the idea of the AG website sourcing data from
other sites is practical or what.  But here are some comments:

*	Given that a lot of the data will be static, you would only need
to update once or twice a day at a maximum.

*	Instead of having a single repository for the data, I am
wondering if it would be practical or viable to have mirrors of the
data.  If we had mirrors in the US, UK and possibly in AUS, than the
three major times zones would be covered.

Now, as part of the discussion at the retreat, I think it was mentioned
that it would be important to decide on a "Schema" on the data, so that
communication between the various "parties" could be established.  I
think it was also mentioned able possibly using "OpenID" for
authorization, but having never heard of it personally, I cannot
comment.

Anyway, I having said my 2 cents worth, I would like to hear what others
have to say and whether anyone can provide assistance to make it a
practical reality.

Cheers,
Jason.
    

-----Original Message-----
From: Andrew Rowley [mailto:Andrew.Rowley at manchester.ac.uk]
Sent: Friday, 25 May 2007 6:20 PM
To: Brian Corrie; Jason Bell
Cc: AG-DEV; qa at accessgrid.org
Subject: RE: [AG-DEV] Re-using node listing data

Hi,

I very much like the idea of having the accessgrid.org site extract the
information from our local repositories.  This means that we are not
creating a single point of failure.  The AGSC have been caught in the
past with relying on services in the US - this means that if it goes
down in our morning, it may not be back until our afternoon!

Andrew :)

============================================
Access Grid Support Centre,
RSS Group,
Manchester Computing,
Kilburn Building,
University of Manchester,
Oxford Road,
Manchester,
M13 9PL,
UK
Tel: +44(0)161-275 0685
Email: Andrew.Rowley at manchester.ac.uk 
 

-----Original Message-----
From: owner-ag-dev at mcs.anl.gov [mailto:owner-ag-dev at mcs.anl.gov] On
Behalf Of Brian Corrie
Sent: 23 May 2007 20:51
To: Jason Bell
Cc: AG-DEV
Subject: Re: [AG-DEV] Re-using node listing data

Hi Jason,

I think this is a great idea, if we can pull it off. I think Todd
probably mentioned some of our efforts here to keep track of nodes that
we collaborate with and that are within our consortium. With a national
collaboration infrastructure on the horizon within Canada, this is only
going to increase.

I think the trick here is to create a data description/schema that
describes nodes that we can all use for a baseline. It is likely that we
will all have ancillary information that we need for individual
purposes, but if there is a single schema where this baseline
information is defined then storage of the information itself is a
secondary question.

The data could be centrally stored and other groups (the Aussies, AG
Schedule, AGSC, Compute Canada) can pull this information for the nodes
of interest as required. Of course another alternative is to have each
of the groups maintain this information locally (probably makes more
sense), and we have a mechanism/filter that extracts the "common schema"
from known regionally authoritative databases for the purposes of
centralizing this information. That is, AGSC, Compute Canada, Pacific
Rim AG, Main AG site, might store regional information and we extract
that information for central purposes (AG Schedule and the main AG site
knows about all nodes) as required. This implies there is a home data
repository for each site somewhere and information gets stored there and
is used by other resources. We would need a catch all site that can
handle those sites that don't have a regional data repository.

Anyway, just some thoughts off the top of my head... Good idea in
general, how do we make it work???

Brian


Jason Bell wrote:
> G'day All
>
> I thought I would get the ball rolling after our interesting
discussion
> at the AG-Retreat Reception.
>
> An interesting topic was raised and one I would like to discuss within

> this forum, is how we can use the information stored within the AG
Node
> listing for other programs.
>
> Initially, it was conceived that it would be beneficial for other
group
> to be able to access the data so they could simply display
"Australian"
> nodes at a local website, etc.  From this, other interesting concepts 
> arose.  One such example was "wouldn't it be good if AG-Schedule could

> use the data within the node listing", thereby negating the
requirement
> to have to register in multiple locations. From this, a concept of a 
> single database, or at least mirroring of information was suggested.
>
> Now, I know there were a lot of other issues raised, but whilst the
idea
> is still in my mind and hopefully our enthusiasm created from the 
> Retreat is still fresh, I would like to see what people think of this 
> idea and how we could actually go about implementing it.
>
> I would like to hear what others have to say.
>
> Cheers,
> Jason.
>
> --------------------------------------------
>
> Jason Bell, B.I.T.
> B. Info. Tech. (Honours) Student
>
> Network Engineer
> Information Technology Division
> Central Queensland University
>
> High Performance Computing Support Officer Central Queensland 
> University
>
> E-mail : j.bell at cqu.edu.au
> Phone : 07 4930 9229
>
> --------------------------------------------
>
> Patience is a virtue.
>
> But if I wanted Patience,
> I would have become a Doctor.
>
> --------------------------------------------
>