[AG-DEV] AG release organization, packaging, and pkgsrc

[Brook Milligan]

Derek Piper writes:
 > Pre-requisites are already packaged for multi-platform use by those that 
 > maintain the packages and those people exist outside of the AG 
 > community. Any work in re-packaging the prerequisites for AG use seems a 
 > duplication of effort in THAT regard and leads on to my other question..

It is true that there are externally maintained packages of AccessGrid
software and of many of the prerequisites.  Typically, there is a
whole series of packages for each of several distributions of Linux,
another for FreeBSD, etc.  This does not really constitute a
"multiplatform" set of packages in the sense defined by the pkgsrc
system.  Perhaps it is unclear what I meant by that and how it could
be useful for the AG community.

Let me start by clarifying the pkgsrc system.  The first distinction
is between source and binary packages.  In any system, there must be
distinct binary packages for each target OS/hardware platform, at
least for packages that contain, for example, native binary executable
code (which includes much of what is required for the AG system).
Consequently, the points have nothing to do with the fact that some
repository (or set of repositories) should maintain a large collection
of pre-built binary packages organized around OS/hardware platform
lines.  I would venture to suggest that this is exactly what the AG
packages mentioned on the web pages and the many external package
repositories currently accomplish for a variety of systems.

The more important points, however, are where did those binary packages
come from, how much of that work is currently being duplicated, and
how could that system be improved?

Douglas Kosovic writes:
 > I can't provide the same binary RPMs across all Fedora versions due to 
 > differences in python versions, GCC ABI, OpenSSL, Xorg, bundled system 
 > packages and i386 vs x86-64. I can't even supply the same source RPMs in 
 > some cases due to differences (e.g. I build against Apple's mDNSResponder on 
 > FC3, but use Avahi on FC4 & 5).

The general means of making binary packages involves independent
creation of each version.  That is, a person or people using a
particular OS/hardware platform create some binary packages for it.
Independently, another person or group using a different OS/hardware
platform create essentially the same (or at least overlapping) set of
binary packages for the second platform.  This process continues for
as many platforms as we have binary packages for.  Clearly, there is a
great deal of duplicated effort.  Even worse, the addition of a new
component to the system (for example, a useful AG component such as
the AGVCR software) triggers more independent and duplicative
development of binary packages; all the packagers must independently
add the new component to their respective packaging system.

Reducing that duplication would mean that developer time and effort
could be refocussed on such activities as improving the system,
identifying additional components that would be useful, etc.

One means of reducing that duplication is to create source-level
packages that have the capability of automatically creating binary
packages for any OS/hardware platform from the same source
description.  Thus, the duplication (i.e., creating each binary
package) becomes an automated process distinct from the development
work of describing, in a platform-independent way, the steps required
to produce a binary package.  Such an approach has the meaningful
characteristics of a multi-platform packaging system and is
implemented by the pkgsrc system.  As described by Douglas above, this
is currently not possible with other systems.

In this model, when a new component of interest is identified (e.g.,
AGVCR), only a single new source-package needs to be created; all the
relevant binary packages can then be automatically created.  All
duplication of packaging effort (save independent runs of the
automated portion) is eliminated.

Derek Piper writes:
 > How does pkg-src integrate with Linux package managers such as rpm and apt?
 > 	- If a system already has, say, python 2.4 would I be forced to install 
 > 'the AG toolkit's' python 2.4, just to keep it happy?
 > 	- I quite enjoy being able to do 'apt-get install accessgrid' and have 
 > all the dependencies installed for me (thanks Steve ;))

Perhaps by "integrate" you mean, can two systems be mixed together
such that one is managing one set of software and one is managing
another?  Indeed, there should be no problem with that.  One view of
pkgsrc at a binary level is that it is an easy means of installing and
removing packages.  Thus, its use has no bearing on independent
packages, other software systems, or even the operating system.

However, pkgsrc also guarrantees that the set of installed packages
are mutually compatible and takes care of all the prerequisite
checking.  Thus, you cannot expect it to deal smoothly with
non-independent packages (e.g., a prerequisite provided via another
packaging system).  Nor can you expect such dependencies to work
smoothly between any other pair of independent packaging systems.

Alternatively, by "integrate" you might mean, is there an equivalent
command to 'apt-get ...' that will do the same thing.  Yes.

There are also commands (equally simple to invoke) that will, for
example, check all your installed packages against a database of
vulnerabilities to identify potential problems.  Indeed, there are a
variety of commands, all simple to use, that provide a lot of support
for verifying packages, replacing them, examining their prerquisite
graphs, or otherwise administering them.  Some are even using the
pkgsrc system to create entire, self-consistent Linux distributions
with a single command.

 > I've not found that the prerequisite software has much bearing on the 
 > learning curve for using the AccessGrid, it's mainly the foibles of 
 > client or the media tools that give people the most problem, i.e. the 
 > software of the AG toolkit itself. This software still needs to be 
 > developed to operate cross-platform, regardless of the packaging
 > system.

Yes, the underlying software has to be well-designed.  Often one of
the limiting factors for detecting less well-designed portions is the
hurdle of installing it on a variety of platforms.  With a standard
and easy way to deploy the software on many platforms, however, this
hurdle is reduced, the software will be exercised on more platforms,
and the design will improve.  Thus, the topic at hand can aid
improvement of the underlying software.

 > I'm not saying I'm against it per se.. but rather wondering if it's 
 > worth it? Is it solving the problems that need to be solved? At the end 
 > of the day, is it really going to change how people use the AccessGrid?

At the end of the day, it may or may not.  There are at least several
scenarios in which it might.  First, as mentioned above, if the system
encourages easy deployment of standard versions on many platforms, it
will be better tested, and users may feel more comfortable with it.
Indeed, if the software quality improves more people will rely on it.
Second, if at least some developers (i.e., the packagers) need not
worry so much about the deployment, they could contribute to code
quality.  Again, the focus of extra development effort could lead to
an improved product, a better user experience, and more users.
Finally, use of a more standardized product across platforms as a
result of a core multi-platform packaging system such as pkgsrc might
reduce the incompatibities that sometimes exist between sites.  This
too would increase the utility of the AG system and attract more
users.

Thus, some of these issues are important and could have an impact on
how people end up using the AG system.  A broader discussion of the
merits of these points is probably worthwhile, even though it might
appear at first to be limited to the arcane details of packaging.

Cheers,
Brook