[AG-DEV] Re-using node listing data
Jason Bell
j.bell at cqu.edu.au
Tue May 29 05:44:12 CDT 2007
Andrew, Paul and All
If I may throw a few suggestion and random thoughts in the mix, to try
and alleviate some of the concerns... I have also posted this message
to the QA list, as a lot of the issues concern both lists.
Firstly, can I state for the record, the global QA process came about
from me developing a Australian process and having a lot of demand from
people to make it a global one....
The following items will be in point form, so apologises if it isn't in
depth. Before going about the implementation, I would like to discuss
what we need.
* I really would like to see the AGSC group be involved and
included, though I have an extensive number of various regions ready to
hit the ground running, it wouldn't be optimal without having all the UK
nodes listed and QA'ed.
* We need to have a schema/data structure that is standardize
across the board. I don't think having different tests/items on
different machines/DB's will work effectively.
* In regards to the QA process, this really needs to be rolled out
now, if we simply delay to long, other tools will take over. Example, I
know there is talk over one of our national bodies using another tool
(cannot think of its name at the moment), but it is like vic... People
are screaming out for a QA process and I know that this is a global
demand! Hence why I plan to start QA'ing nodes next week!
* There is a need for people to go to a single site to view the
Node listings. I would guess that 80% of nodes will use the
http://www.accessgrid.org/nodes as the first point of call to try and
locate a particular AG node and associated information.
* The QA process (that I have been developing), to make it work
well, needs to be, and is currently, tightly coupled to the node
listing;
* I would think that a lot of the information is static,
therefore, if it goes down, does it matter??? Unlike the bridge
registry for example;
Now currently my thoughts of potential implementations are:
* Now we cannot change the fact that the node listing and QA stuff
is on the Drupal website, but I am sure that we can somehow plug into
the system.
* Now these are random thoughts but:
- I think we can already export data from Drupal which
could be saved and then used to display local (ie UK) nodes; This was
demo'ed at the retreat in which the node-listing data was used to
provide a world map and show where all the nodes are.
- The great idea in regards to the QA process is that
different people will manage different regions. I won't be inputting
the data for all QA'ed node. Therefore, the question I would like to
ask, does it really matter if you input the data on your local system or
the AG website... Because if you can list your data locally, than is
this an issue??? You can still hold data locally, just input it on a
Global system for everyone else to see.
- One thing that you may be able to use to argue the point
for using this system, is that having different regional testers, you
shouldn't be required to test any international nodes any more.
* Though, I do agree that having and operational window outside
our normal working hours is a problem... Therefore, is any of this
possible:
- Can a mirror/failover server be running;
- Can external people have access to restart etc;
Now I hope that some of the things I have said makes sense... But
please keep this thread going, as I really would like to resolve the
issue, ASAP.
Also, others are welcome and suggested to comment as well.
Thanks for your time,
Jason.
-----Original Message-----
From: Andrew Rowley [mailto:Andrew.Rowley at manchester.ac.uk]
Sent: Tuesday, 29 May 2007 7:28 PM
To: Jason Bell; Paul Bonnett; Brian Corrie
Cc: AG-DEV
Subject: RE: [AG-DEV] Re-using node listing data
Hi,
My big worry is that if the single site goes down, do we have none of
the information?
The idea of having backend databases in separate locations is that the
main site will contact these databases for the node listing. The
information will be available from the main site on a read-only basis.
If a node wants to update the information, they can do this from the
site that owns the information e.g. AGSC nodes will be able to edit
their listing after authenticating on an AGSC site. Also, the AGSC will
be able to update QA testing information on our own site, and list the
AGSC nodes on our own site (this is essential).
Please note that the reason that I bring up these concerns is that the
AGSC is running a UK national service. We are paid to provide this
service and we have to make sure that the services we rely on can be
provided within the AGSC hours of operation without failure (or can be
brought up again quickly with very short notice). Therefore, if this
cannot be guaranteed to us, we cannot use the services. We have been
burnt by services going down in the US before, and therefore we don't
want to rely on these services without some sort of guarantee.
I can understand that as this is being offered on a voluntary basis, and
I do appreciate the work that has gone into this. However, we must put
our users first when considering moving to a service like this.
Andrew :)
============================================
Access Grid Support Centre,
RSS Group,
Manchester Computing,
Kilburn Building,
University of Manchester,
Oxford Road,
Manchester,
M13 9PL,
UK
Tel: +44(0)161-275 0685
Email: Andrew.Rowley at manchester.ac.uk
-----Original Message-----
From: Jason Bell [mailto:j.bell at cqu.edu.au]
Sent: 29 May 2007 10:13
To: Paul Bonnett; Andrew.Rowley at manchester.ac.uk; Brian Corrie
Cc: AG-DEV
Subject: RE: [AG-DEV] Re-using node listing data
G'day all
I think I might ask Tom to comment on this, as he knows more about the
practically of data exchange within the Drupal database than I do.
The concern I have is having a centralise system is what we are trying
to achieve here, I.E. a single location that someone can go to, that
contains all of the AG nodes for example. If we obtain data from
different sites, does that mean if a site goes down, we only have half
the info... Or that we have to visit multiple sites???
Anyway, my 2 cents worth.
Cheers,
Jason.
-----Original Message-----
From: Paul Bonnett [mailto:P.Bonnett at ukerna.ac.uk]
Sent: Tuesday, 29 May 2007 6:49 PM
To: Andrew.Rowley at manchester.ac.uk; Jason Bell; Brian Corrie
Cc: AG-DEV
Subject: RE: [AG-DEV] Re-using node listing data
All,
Am I missing something here? Could there not be a query system mirrored
in UK, USA & AU which polled the 3 databases for information on a live
basis so we're not copying data back and forth? You would not be able to
access the data directly, but only via your 'home' body which would
authenticate you and allow access to information.
I hate the idea of 'copies' of databases around the world; and I
actually think it may contravene the UKERNA policies, but I will have to
check.
Regards,
Paul
-----Original Message-----
From: owner-ag-dev at mcs.anl.gov [mailto:owner-ag-dev at mcs.anl.gov] On
Behalf Of Andrew Rowley
Sent: 29 May 2007 09:14
To: Jason Bell; Brian Corrie
Cc: AG-DEV
Subject: RE: [AG-DEV] Re-using node listing data
Hi,
Mirroring could be complicated as it will probably only allow updating
of this from the US site - this would stop our day-to-day activities if
it went down and so would be unacceptable for us to rely on without a
guarantee of service.
Does the site use some sort of SQL database backend? If so, couldn't
you trawl the info from various SQL databases, rather than just one? As
well as this, you could have the information trawled stored in the
central accessgrid.org database in case the other sites were down...
Andrew :)
============================================
Access Grid Support Centre,
RSS Group,
Manchester Computing,
Kilburn Building,
University of Manchester,
Oxford Road,
Manchester,
M13 9PL,
UK
Tel: +44(0)161-275 0685
Email: Andrew.Rowley at manchester.ac.uk
-----Original Message-----
From: Jason Bell [mailto:j.bell at cqu.edu.au]
Sent: 28 May 2007 00:29
To: Andrew.Rowley at manchester.ac.uk; Brian Corrie
Cc: AG-DEV
Subject: RE: [AG-DEV] Re-using node listing data
G'day All
Well, I must say that it appears I may have opened a can of worms here
:-)
Anyway, I don't know if the idea of the AG website sourcing data from
other sites is practical or what. But here are some comments:
* Given that a lot of the data will be static, you would only need
to update once or twice a day at a maximum.
* Instead of having a single repository for the data, I am
wondering if it would be practical or viable to have mirrors of the
data. If we had mirrors in the US, UK and possibly in AUS, than the
three major times zones would be covered.
Now, as part of the discussion at the retreat, I think it was mentioned
that it would be important to decide on a "Schema" on the data, so that
communication between the various "parties" could be established. I
think it was also mentioned able possibly using "OpenID" for
authorization, but having never heard of it personally, I cannot
comment.
Anyway, I having said my 2 cents worth, I would like to hear what others
have to say and whether anyone can provide assistance to make it a
practical reality.
Cheers,
Jason.
-----Original Message-----
From: Andrew Rowley [mailto:Andrew.Rowley at manchester.ac.uk]
Sent: Friday, 25 May 2007 6:20 PM
To: Brian Corrie; Jason Bell
Cc: AG-DEV; qa at accessgrid.org
Subject: RE: [AG-DEV] Re-using node listing data
Hi,
I very much like the idea of having the accessgrid.org site extract the
information from our local repositories. This means that we are not
creating a single point of failure. The AGSC have been caught in the
past with relying on services in the US - this means that if it goes
down in our morning, it may not be back until our afternoon!
Andrew :)
============================================
Access Grid Support Centre,
RSS Group,
Manchester Computing,
Kilburn Building,
University of Manchester,
Oxford Road,
Manchester,
M13 9PL,
UK
Tel: +44(0)161-275 0685
Email: Andrew.Rowley at manchester.ac.uk
-----Original Message-----
From: owner-ag-dev at mcs.anl.gov [mailto:owner-ag-dev at mcs.anl.gov] On
Behalf Of Brian Corrie
Sent: 23 May 2007 20:51
To: Jason Bell
Cc: AG-DEV
Subject: Re: [AG-DEV] Re-using node listing data
Hi Jason,
I think this is a great idea, if we can pull it off. I think Todd
probably mentioned some of our efforts here to keep track of nodes that
we collaborate with and that are within our consortium. With a national
collaboration infrastructure on the horizon within Canada, this is only
going to increase.
I think the trick here is to create a data description/schema that
describes nodes that we can all use for a baseline. It is likely that we
will all have ancillary information that we need for individual
purposes, but if there is a single schema where this baseline
information is defined then storage of the information itself is a
secondary question.
The data could be centrally stored and other groups (the Aussies, AG
Schedule, AGSC, Compute Canada) can pull this information for the nodes
of interest as required. Of course another alternative is to have each
of the groups maintain this information locally (probably makes more
sense), and we have a mechanism/filter that extracts the "common schema"
from known regionally authoritative databases for the purposes of
centralizing this information. That is, AGSC, Compute Canada, Pacific
Rim AG, Main AG site, might store regional information and we extract
that information for central purposes (AG Schedule and the main AG site
knows about all nodes) as required. This implies there is a home data
repository for each site somewhere and information gets stored there and
is used by other resources. We would need a catch all site that can
handle those sites that don't have a regional data repository.
Anyway, just some thoughts off the top of my head... Good idea in
general, how do we make it work???
Brian
Jason Bell wrote:
> G'day All
>
> I thought I would get the ball rolling after our interesting
discussion
> at the AG-Retreat Reception.
>
> An interesting topic was raised and one I would like to discuss within
> this forum, is how we can use the information stored within the AG
Node
> listing for other programs.
>
> Initially, it was conceived that it would be beneficial for other
group
> to be able to access the data so they could simply display
"Australian"
> nodes at a local website, etc. From this, other interesting concepts
> arose. One such example was "wouldn't it be good if AG-Schedule could
> use the data within the node listing", thereby negating the
requirement
> to have to register in multiple locations. From this, a concept of a
> single database, or at least mirroring of information was suggested.
>
> Now, I know there were a lot of other issues raised, but whilst the
idea
> is still in my mind and hopefully our enthusiasm created from the
> Retreat is still fresh, I would like to see what people think of this
> idea and how we could actually go about implementing it.
>
> I would like to hear what others have to say.
>
> Cheers,
> Jason.
>
> --------------------------------------------
>
> Jason Bell, B.I.T.
> B. Info. Tech. (Honours) Student
>
> Network Engineer
> Information Technology Division
> Central Queensland University
>
> High Performance Computing Support Officer Central Queensland
> University
>
> E-mail : j.bell at cqu.edu.au
> Phone : 07 4930 9229
>
> --------------------------------------------
>
> Patience is a virtue.
>
> But if I wanted Patience,
> I would have become a Doctor.
>
> --------------------------------------------
>
More information about the ag-dev
mailing list