[AG-DEV] AG release organization, packaging, and pkgsrc

[Derek Piper]

	Hi Brook,

	I initially have a couple of comments about such a system. I'll 
probably think of more after I send this :)

Pre-requisites are already packaged for multi-platform use by those that 
maintain the packages and those people exist outside of the AG 
community. Any work in re-packaging the prerequisites for AG use seems a 
duplication of effort in THAT regard and leads on to my other question..

How does pkg-src integrate with Linux package managers such as rpm and apt?
	- If a system already has, say, python 2.4 would I be forced to install 
'the AG toolkit's' python 2.4, just to keep it happy?
	- I quite enjoy being able to do 'apt-get install accessgrid' and have 
all the dependencies installed for me (thanks Steve ;))

For windows nodes, you install a few installer packages in order and 
you're done.

I've not found that the prerequisite software has much bearing on the 
learning curve for using the AccessGrid, it's mainly the foibles of 
client or the media tools that give people the most problem, i.e. the 
software of the AG toolkit itself. This software still needs to be 
developed to operate cross-platform, regardless of the packaging system.

I'm not saying I'm against it per se.. but rather wondering if it's 
worth it? Is it solving the problems that need to be solved? At the end 
of the day, is it really going to change how people use the AccessGrid?

	Derek


Brook Milligan wrote:
> Some time ago I exchanged some notes with Thomas Uram about some of my
> experiences with the Access Grid software.  He encouraged me to join
> ag-dev and begin a discussion on the topic.  So what follows is the
> start of that.  As I have not participated in the development process
> and am a relative newcomer into the AG world, please excuse what may
> be some misinterpretations on my part.
> 
> One of my observations about the Access Grid software is that it is
> both quite complex and requires a large number of prerequisites.
> Consequently, the learning curve is quite steep and users (or their
> sysadmins) must collect and manage a large number of packages just to
> check it out.  A variety of packages exist, but there appears to also
> be a variety of different versions being used for either Access Grid
> itself or the prerequisites.  Finally, the released code seems not to
> be particularly modular, which exacerbates the management problem.
> 
> It would seem that some steps forward that would greatly improve the
> situation would include the following:
> 
> - Put the AG software and its dependencies into a multiplatform
>   packaging system.
> 
> - Encourage the AG community to make use of that packaging system as a
>   distribution mechanism.
> 
> These steps would encourage standardization of deployed AG versions but would
> allow sites to use whatever platform was appropriate for them.  It
> would improve the modularity of the overall system, allowing it to
> grow and respond to changes in a guided fashion.
> 
> As one step along the way, we have integrated the AG v2 software and
> its prerequisites into the pkgsrc packaging system (www.pkgsrc.org).
> For those of you who are not aware of this system, it offers a number
> of features that I think would make it useful for the AG community.
> 
> - By virtue of being multiplatform, there is no duplication of
>   packaging infrastructure across different hardware/OS platforms.
>   Currently, each packaging effort must go through much of the same
>   development process, which is a needless duplication of effort.
> 
> - Multiplatform infrastructure support is being developed by people
>   who know how to do it well and are committed to the improvements and
>   support required.  Thus, the massive effort already well developed
>   for multiplatform support can be easily leveraged by a small effort
>   from the AG community to support the AG applications themselves.
> 
> - An ever expanding set of supported platforms with little or no
>   investment by the AG community.  Because the underlying
>   multiplatform infrastructure is independent of the AG software
>   itself, work on that infrastructure can proceed independently of AG
>   but will immediately benefit the community.
> 
> - Encouragement of a standardized release across many platforms,
>   potentially improving interoperability, reducing confusion, and
>   enhancing user satisfaction.  This is perhaps the most obvious
>   benefit.  If users can easily install and test the AG software, it
>   will be used in more cases and the community will flourish.
> 
> - Support for both binary and source-based packages for both those who
>   are not interested in compiling their tools as well as those who
>   are.  Binary packages, which could be distributed on the AG site,
>   are the most convenient means of beginning in many cases, but
>   management of source-based compilation cannot be ignored.  For
>   example, the latter is a means of preparing binary packages in an
>   automated fashion.  It is also a means of rapidly verifying the
>   correctness of code during the development process.
> 
> - Automatic management of the myriad of dependencies required for
>   deploying the AG software.  This is another obvious and immediate
>   benefit.  By modularizing the entire release, it will become much
>   easier to manage.
> 
> - Provisions for automated security audits of pkgsrc software.  The
>   recent discussions about security concerns and firewall issues might
>   be helped by the knowledge that users of the pkgsrc system can be
>   immediately notified, from a common database, of security issues
>   associated with the software.  In any case, an automated mechanism
>   of staying informed about security issues associated with the
>   installed software base is valuable in my opinion.
> 
> If the AG community perceives these as valuable benefits, I would be
> glad to work with you to guide the process of developing expertise in
> the use of pkgsrc for distributing AG software.  As it stands, based
> on our work most of the necessary code is already within pkgsrc or
> pkgsrc-wip; we are actually using it to manage deployment of our own
> operational AG nodes.  What is required for moving forward is simply
> an acceptance of the idea that pkgsrc could be a viable distribution
> mechanism, more testing on a wider array of platforms, and improving
> our pkgsrc code to cover cases we have not yet encountered.  None of
> this is nearly as complex as the AG code itself.  Overall, I think
> this would go a long way toward improving the standardization of
> operational AG nodes.
> 
> This is all offered in the spirit of beginning a discussion.  I hope
> there is some agreement that these are issues the AG community should
> be wrestling with and that the pkgsrc system offers enough benefits to
> the project to warrant further development.
> 
> Cheers,
> Brook
> 

-- 
Derek Piper - dcpiper at indiana.edu - (812) 856 0111
IRI 323, School of Informatics
Indiana University, Bloomington, Indiana