[AG-DEV] AG release organization, packaging, and pkgsrc

[Brook Milligan]

Some time ago I exchanged some notes with Thomas Uram about some of my
experiences with the Access Grid software.  He encouraged me to join
ag-dev and begin a discussion on the topic.  So what follows is the
start of that.  As I have not participated in the development process
and am a relative newcomer into the AG world, please excuse what may
be some misinterpretations on my part.

One of my observations about the Access Grid software is that it is
both quite complex and requires a large number of prerequisites.
Consequently, the learning curve is quite steep and users (or their
sysadmins) must collect and manage a large number of packages just to
check it out.  A variety of packages exist, but there appears to also
be a variety of different versions being used for either Access Grid
itself or the prerequisites.  Finally, the released code seems not to
be particularly modular, which exacerbates the management problem.

It would seem that some steps forward that would greatly improve the
situation would include the following:

- Put the AG software and its dependencies into a multiplatform
  packaging system.

- Encourage the AG community to make use of that packaging system as a
  distribution mechanism.

These steps would encourage standardization of deployed AG versions but would
allow sites to use whatever platform was appropriate for them.  It
would improve the modularity of the overall system, allowing it to
grow and respond to changes in a guided fashion.

As one step along the way, we have integrated the AG v2 software and
its prerequisites into the pkgsrc packaging system (www.pkgsrc.org).
For those of you who are not aware of this system, it offers a number
of features that I think would make it useful for the AG community.

- By virtue of being multiplatform, there is no duplication of
  packaging infrastructure across different hardware/OS platforms.
  Currently, each packaging effort must go through much of the same
  development process, which is a needless duplication of effort.

- Multiplatform infrastructure support is being developed by people
  who know how to do it well and are committed to the improvements and
  support required.  Thus, the massive effort already well developed
  for multiplatform support can be easily leveraged by a small effort
  from the AG community to support the AG applications themselves.

- An ever expanding set of supported platforms with little or no
  investment by the AG community.  Because the underlying
  multiplatform infrastructure is independent of the AG software
  itself, work on that infrastructure can proceed independently of AG
  but will immediately benefit the community.

- Encouragement of a standardized release across many platforms,
  potentially improving interoperability, reducing confusion, and
  enhancing user satisfaction.  This is perhaps the most obvious
  benefit.  If users can easily install and test the AG software, it
  will be used in more cases and the community will flourish.

- Support for both binary and source-based packages for both those who
  are not interested in compiling their tools as well as those who
  are.  Binary packages, which could be distributed on the AG site,
  are the most convenient means of beginning in many cases, but
  management of source-based compilation cannot be ignored.  For
  example, the latter is a means of preparing binary packages in an
  automated fashion.  It is also a means of rapidly verifying the
  correctness of code during the development process.

- Automatic management of the myriad of dependencies required for
  deploying the AG software.  This is another obvious and immediate
  benefit.  By modularizing the entire release, it will become much
  easier to manage.

- Provisions for automated security audits of pkgsrc software.  The
  recent discussions about security concerns and firewall issues might
  be helped by the knowledge that users of the pkgsrc system can be
  immediately notified, from a common database, of security issues
  associated with the software.  In any case, an automated mechanism
  of staying informed about security issues associated with the
  installed software base is valuable in my opinion.

If the AG community perceives these as valuable benefits, I would be
glad to work with you to guide the process of developing expertise in
the use of pkgsrc for distributing AG software.  As it stands, based
on our work most of the necessary code is already within pkgsrc or
pkgsrc-wip; we are actually using it to manage deployment of our own
operational AG nodes.  What is required for moving forward is simply
an acceptance of the idea that pkgsrc could be a viable distribution
mechanism, more testing on a wider array of platforms, and improving
our pkgsrc code to cover cases we have not yet encountered.  None of
this is nearly as complex as the AG code itself.  Overall, I think
this would go a long way toward improving the standardization of
operational AG nodes.

This is all offered in the spirit of beginning a discussion.  I hope
there is some agreement that these are issues the AG community should
be wrestling with and that the pkgsrc system offers enough benefits to
the project to warrant further development.

Cheers,
Brook

-- 
Brook G. Milligan                      Internet:  brook at nmsu.edu
Department of Biology
New Mexico State University            Telephone:  (505) 646-7980
Las Cruces, New Mexico  88003  U.S.A.  FAX:        (505) 646-5665