[AG-DEV] Continuation of AG services - Update 2
Christoph Willing
c.willing at uq.edu.au
Mon Apr 16 08:55:48 CDT 2012
Comments below:
On 16/04/2012, at 11:59 AM, Jason Bell wrote:
> Colleagues (resent – I haven’t received any feedback from this
> recent update)
>
> I have started to draft feedback (apologies for the roughness) from
> everyone and collated a few ideas on ways to move the AG services
> forward. I have attempted to group all comments and added a few of
> my own, but please feel free to comment, even editorial feedback
> welcomed!
>
> It is hoped that this email is only the beginning and additional
> content will be provide once new feedback has been obtained,
> particularly to the table below.
>
> Please note, that though a number of suggestions were made on how
> the Access Grid software and services can be improved, I believe it
> is important to focus on the continuation of current services,
> before additional code development or the likes can be considered.
>
> One idea that has been floated around, is to setup
> a/many Virtual Machine/s to run many of the services on.
Virtual or physical machine is a site dependent implementation detail
we shouldn't need to worry about here.
> There are many options with this idea, either use one of the many
> global hosting services, or to use an institutional system (thanks
> to all those institutional groups who have already offered their
> services!). Some things to consider though:
>
> Commercial hosting service
>
> · Someone or group has to finance the ongoing costs
> o Even if multiple groups contribute, a board or person will have
> to manage payment, etc.
> · Commercial grade network (will large downloads and site
> visits be slow or incur large costs)
> · Will the “hosting” be reliable?
> · Given the many hosting services that are starting and
> stopping, will a new host be needed every couple of years?
> · Not tied to any particular group or organisation,
> therefore if funding stops from one location, another can take over
> or replace easily, without the need to move the services to another
> location.
> · Not being tied to anyone particular, this can be seen as a
> more open source product and community.
>
> Institutional hosting
>
> · Will the services need to be transferred again, after
> support runs out?
> · Will network charges be incurred?
> · Most intuitional organisations are funded on a “term”
> basis, therefore, will services need to be moved regularly once
> support runs out?
> o For example, ANL will no longer fund/support the AG projects,
> therefore services need to be moved. Will this be something that
> occurs every few years?
> o How reliable can we count on the support?
> · Institutions generally have very large and fast networks,
> thus large data traffic should cause any performance issues.
Answers to just about all these questions is pretty obviously yes.
Basically any support system that anyone sets up will occasionally
need renewal. I guess its best to minimise the frequency of that;
we've had a good run with ANL - over 10 years.
>
> What do people think of the two options listed? Any advantages or
> disadvantages to add? Any preference?
Its probably harder with a commercial option esp finding ongoing
funding. Doing it institutionally, we can take advantage of people's
spare of after hours cycles. Of course that obviates performance
guarantees (but we've never had them previously anyway).
>
> Another question that I guess needs to be answered is domain names.
> For example, some of the default aspects of the Venue Server and
> Venue Client code points to ANL addresses. For example,
> “vv3.mcs.anl.gov”, I am wondering if it be possible to have rules in
> place that forward this address (DNS redirection) to another
> location (one to be determined of course).
According to whois, accessgrid.org belongs to University of Chicago.
They would have to (as mentioned below) either:
1. transfer ownership to the new "owners" (yet to be determined)
or 2. point DNS to new owners IP addresses
> This will prevent the need to immediately modify source code and
> roll out updates to support new internet addresses.
>
> I have attempted to table
Sorry my email client has lost the table layout - hope the comments
make sense.
> the various Access Grids Services and Systems that need to be
> transitioned, provided some general comments and possible options.
> If anyone have anything else to add or modify, please do contribute
> to this conversation.
>
> Access Grid Services and Systems
> Comments
> Priority
> Option 1
> Option 2
> Option 3
> Recommendation
> Additional Comments
> www.accessgrid.orgdomain name
> · Can the domain name be transferred?
>
> · Who will own (be responsible) for maintaining this address
>
> · Do we want to obtain a new address?
>
>
> Nice to have
> Transfer domain
> Use a new address
> ???
>
>
> vv3.mcs.anl.gov and other anl.gov domain names
> · Can all these address be forward to new location.
>
> · Thinking out loud, but standardising to a new address,
> such aswww.accessgrid.org would be useful in the long term.
>
>
> Nice to have
> Forward to new domain names
> Update Venue Client and Venue Server code and roll up new versions
> with these updates
Packaging new versions for Linux after code updates is pretty easy.
Mac packaging has been (I think) exclusively an ANL activity.
Windows packages have been based on ANL packaging too haven't they.
That probably means that those at ANL who have been making the Mac &
Windows packages will need to keep making them, or others will have to
figure out how to do it (hopefully with ANL packagers' help).
Who's volunteering to do the Mac & Windows builds? Hard to continue
without them.
> ???
>
>
> Access Grid Source Code
> · A system needs to be implemented that allows the storage
> and Access to the Access Grid Source Code.
>
> · It is expected that an international group will oversee
> updates, etc.
>
> · This is currently in a SVN repository.
>
> · Only one repository required.
> Must to have
> Using Source Forge
>
>
> Google ????
>
> Nothing further investigated?
>
> Using a hosted service (Commercial or Institutional) and implement a
> SVN repository
>
> Source Forge Instructions:http://sourceforge.net/apps/trac/sourceforge/wiki/SVN%20adminrepo#ImportingfromotherreposincludingotherSCMs
>
> ANL Jabber Server
> (could be made a regional service)
> · Currently, any newly configured Venue Server will use the
> ANL Jabber Server.
This is not exactly correct; while the ANL jabber server is the
default, this is actually set in the VenueServer.cfg file so is a
configurable setting. The catch is that this file is generated for
you, if it doesn't already exist, from defaults in the VenueServer
library. Therefore if you don't have a VenueServer.cfg already, create
a minimal one with (for instance)
# AGTk 3.2
[VenueServer]
textHost = jabberserver.somewhere.org
- the remaining entries will be created for you when you start the
server.
If you already have a VenueServer.cfg, edit the textHost entry to
whatever you like and restart the server.
>
> · Questions:
> o Do we want to use a different technology than jabber?
There are some good reasons to use something different. A recently
discovered good reason to keep it is that its logfile provides an
excellent record of venue server usage. See:
http://www.rcc.uq.edu.au/accessgrid/usage/
for a display of the APAG Venue Server usage, derived wholly from the
logfile of the Jabber server we run here.
> o Should a number of regional jabber servers be configured and a
> Venue Server config option be implemented.
As above, thats configurable in the VenueServer.cfg file right now.
There'd be no problem with adding a specific option but since its
already configurable by other means, needn't be a high priority.
> o Should each Venue Server run their own Jabber server, therefore
> removing a single point of failure?
While its pretty easy to run a venue server, its much, much harder to
set up a compatible jabber server (one of the good reasons to use
something other than jabber as the text chat protocol).
>
> · Should a Global Jabber server be implemented as a stop gap
> measure?
I'm happy for the APAG jabber server to be used globally as a more or
less indefinite stop gap measure.
> Critical
> Use a hosted service (Commercial or Institutional) and implement a
> Global Jabber server
From the above, I'm not sure its actually critical (at least not
urgent) to to implement a new jabber server. In any case it may be a
somewhat wasted effort if we end up using something other than jabber.
>
> Note: The Venue Server source code will have to be modified to point
> to new server, if DNS redirection cannot be implemented.
> Use a different technology than Jabber.
>
> One technology that has been mentioned is MQTT (Message Queue
> Telemetry Protocol).
I've suggested this in the past because I've used it myself in other
projects - other people will have their own favourites. MQTT already
has a python binding and I found it was pretty easy to use and think
it should be relatively easy to incorporate into the AG code. However
there would be a reasonable effort to actually do it.
>
> Each host running their own jabber server?
>
> · Is this Feasible or even possible?
Setup complexity make this possibility doubtful.
> · Is the development work require too much before the
> required deadline for the shutdown?
>
>
> ANL Bridge Registry
> (could be made a regional service)
> It is already possible to run your own Bridge Registry. The
> question is more, currently Venue Clients default to using
> “vv3.mcs.anl.gov:8030”
A second default (www.ap-accessgrid.org:8030) has been included as a
default for quite a while (years?) now. Others can be easily added but
would need to be part of a new package release for them to be
available by default. Otherwise they can be added by hand via
VenueClient gui (hardly automatic though).
>
> Should another global “bridge registry” be implemented, or should a
> number of “regional bridge registries” be implemented that these be
> added to the source code?
>
> Critical
> Use a hosted service (Commercial or Institutional) and implement a
> Global Bridge Registry
>
> Note: The Venue Client source code will have to be modified to point
> to new server, if DNS redirection cannot be implemented.
> Roll out a set of regional registries, which is documented on the AG
> website and provide documentation to users for adding new registries.
>
> · Additionally, update source code to contain new registries.
> ???
>
>
> AG-Dev Certificate Authority
> Currently, whenever new Venue Servers are implemented, they must
> request and install an identity certificate.
An identity will work but are pretty inconvenient to use from boot
time scripts; better to use service certificates.
> Critical
> Centralised certificates?
>
> Use a hosted service (Commercial or Institutional) and implement a
> Global Jabber server
>
> Note: The Venue Server source code will have to be modified to point
> to new server, if DNS redirection cannot be implemented.
> Self-signed certificates
> · lessens the security features available
>
> · Does anyone care about this?
Yes, I care about it and have started looking at how to set up a
certificate issuing infrastructure based on the ANL model. Its
incomplete - waiting for more time to spend on it. Needs some changes
to current AG code which points requests to particular servers running
particulars cgi scripts.
I'd recommend anyone running AG services to obtain new certificates
before June deadline so they have a year before certificate system
needs to be fully functional.
>
> ???
>
>
> The accessgrid.org web site
> The www.accessgrid.org website is a very useful resource that
> contained things like:
>
> · Global Node listing;
> · QA information and results;
> · AG Projects (http://www.accessgrid.org/projects)
> · Software downloads;
> · Documentation;
> · Hardware information;
> · Community information; and
> · New and events
>
> Things to note:
> · Website is based on Drupal 4.7 and is extremely long
> overdue for a software upgrade. Current version is 7.12 (as of
> 10/4/2012)
> · Newer versions may cause issues with some aspects of the
> current content, particularly Node listing and QA;
>
> Must to have
> Use a hosted service (Commercial or Institutional) and implement a
> Global Bridge Registry
>
> Simply move current content to new location.
>
> · Current version of drupal out-dated;
> · Content will be the same as current website.
> · Assumed less work required
> Use a hosted service (Commercial or Institutional) and implement a
> Global Bridge Registry
>
> Install newer version of drupal and port content over.
>
> · Significant amount of work required;
> · Some aspects, particularly Node listing and QA may have to
> be completely redeveloped;
> Start a fresh, possibly look towards using a different CMS system?
>
> It is assumed that the domain name www.accessgrid.org can still be
> used.
> AG-related mailing lists
> There are many community mailing list that are actively used. These
> being:
>
> ag-tech at mcs.anl.gov
> ag-users at mcs.anl.gov
> ag-announce at mcs.anl.gov
> ag-dev at mcs.anl.gov
> ag-cvs at mcs.anl.gov
>
> · Are all these mailing list still required?
>
> · Are there any other ones that should be created?
>
> · Can the history be ported?
>
>
> Must to have
> Use a hosted service (Commercial or Institutional) and implement an
> Access Grid mailing list service.
>
> Use Source Forge. If using Source Forge is decided as the best
> option for housing the source code, they have additional services
> such as mailing lists that could be used.
>
>
>
> ANL Venue Server
> It is already possible to run your own Venue Server. The question
> is more, currently Venue Clients default to using “vv3.mcs.anl.gov:
> 8000”
>
> Should another global “Venue Server” be implemented, or should a
> number of “regional Venue Server” be implemented that these be added
> to the source code?
> · Which ones become default?
>
>
>
> Nice to have
> Use a hosted service (Commercial or Institutional) and implement a
> Global Venue Server
>
> Note: The Venue Client source code will have to be modified to point
> to new server, if DNS redirection cannot be implemented.
> Roll out / utilise more a set of regional Venue Servers, which is
> documented on the AG website and provide documentation to users for
> adding Venue Server addresses.
> Set the default Venue Server to one currently listed athttp://www.accessgrid.org/communityor
> another one???
>
> Any volunteers?
We'll continue to run APAG venue server from UQ.
We could easily run a more generic vv3.accessgrid.org server too but
maybe people would prefer that to be hosted more centrally (US, Europe).
>
>
> NCSA AG Scheduler
> NCSA are interested in finding someone to take on the support of
> AGSchedule
> Critical ???
> Use a hosted service (Commercial or Institutional) and implement and
> support AGSchedule on a new server.
>
>
> Multicast Beacon
> Is a global or regional solution required?
It would be good to have some widely used system again.
We've been using dbeacon for a while in Asia Pacific region and it
seems pretty good. It has no real central server - any participating
site can host web server for its display e.g. display nominally hosted
at:
http://syd-a-ext1.aarnet.net.au/matrix/ (AARNet)
is also available at:
http://ag0-video.vislab.uq.edu.au:8888/matrix/ (UQ RCC)
>
> I would propose that we should try to break down some of the current
> AG services into 2 categories. These being Global and Regional
> Services.
>
> · Global Services - are services that only requires a single
> instance or implementation;
>
> · Regional Services – are services that can have multiple
> instances or implementations of;
I'm not necessarily against it but I'm not sure this is a particularly
important distinction. Any AG service can be run by as many different
sites as want to run them. Whether you label them global or regional
doesn't really mean much. For instance the APAG venue server has lots
of venues related to the asia pacific region but that hasn't prevented
sites from elsewhere in the world also using it (and they've been
welcome to do so).
>
> Once the various services are determined to be Global or Regional, a
> total list of global resources can be identified. For example, if
> throughout this process, it is determined that a regional list of
> bridge registries is the best option, then a global service is no
> longer needed. Thereby no resources will be required in running
> this as a global service, as it will be expected various “regional”
> group will run their own implementation.
Further to above, the concept of various groups running "their own
implementation" has been a core feature of the AG as I understand it -
it was designed to work in precisely that way. Lots of "regional"
services is the way to go, I think. No harm in having some "model
service site" as ANL has been over the years, but not sure where that
should be.
chris
>
> Many thanks for your time and I look forward to everyone’s feedback.
>
> Many regards,
> Jason.
>
> -------------------------------------------------------------------------
> Jason Bell, B.I.T. (Honours)
>
> Video Collaboration Champion
> Queensland Cyber Infrastructure Foundation
> http://www.qcif.edu.au/
>
> ARCS eResearch Collaborative Services
> http://www.arcs.org.au/
>
> Senior Research Technologies Officer
> Information Technology Directorate
> CQ University Australia
> http://www.cqu.edu.au/
>
> E-mail : j.bell at cqu.edu.au
> j.bell at qcif.edu.au
> jason.bell at arcs.org.au
> Work : +61 7 4930 9229
> Mobile : 0409 630897
> Postal : Building 19, Room 1.55
> Central Queensland University
> Bruce Highway
> Rockhampton, Queensland, Australia,
> 4702
> -------------------------------------------------------------------------
> Patience is a virtue.
>
> But if I wanted Patience,
> I would have become a Doctor.
> -------------------------------------------------------------------------
>
Christoph Willing +61 7 3365 8316
Research Computing Centre
University of Queensland
More information about the ag-dev
mailing list