[petsc-dev] Making a PetSc Roll and Rocks Module

Thu Jan 24 13:09:51 CST 2013

On Thu, Jan 24, 2013 at 12:51 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:

>
>
> Begin forwarded message:
>
> *From: *Philip Papadopoulos <philip.papadopoulos at gmail.com>
> *Subject: **Re: [petsc-maint #149715] Making a PetSc Roll and Rocks Module
> *
> *Date: *January 24, 2013 9:48:36 AM CST
> *To: *"Schneider, Barry I." <bschneid at nsf.gov>
> *Cc: *Barry Smith <bsmith at mcs.anl.gov>
>
> Dear Barry^2,
>
> The major observation about the build process is that building the
> software so that it can livesw-
> in a standard OS package is more work than it should be. Here is the
> standard "meme".
> Suppose you have three directories:
> sw-src
> tmpoot/sw-install
> sw-install
>
> sw-src: the source directory tree for building the sw package (eg
> petsc-3.3-p5)
> sw-install:  the directory where one wants sw installed on the final
> system. We use something
>                  like
> /opt/petsc/compiler/mpi/interconnect/petsc-version/petsc-arch
>
> the last directory is an artifact of the way many people build software
> for packaging. Instead of
> installing into sw-install directly, you install into a
> tmproot/sw-install.  Then you direct the package
> manager to grab all files in tmproot/sw-install to put in the package.
>  The package itself will
> strip off the tmproot leading directory. In other words the package labels
> all files as sw-install/.
>
> So the build issue that I ran into is that the build directory and/or the
> tmproot directory path becomes embedded into a large number of files
> (include files, mod files, etc). To get around this,
> I did a bind mount of (tmproot/sw-install --> /sw-install) and told petsc
> to install into /sw-install.  I consider that to be a "hack", but petsc
> isn't the only software that I've run into that requires this
> mounting bit of legerdemain.
>
> Many software packages will support a dest=  or DESTDIR variable for "make
> install"  that supports the "installation" into a tmproot directory but
> leaves no trace of tmproot in any of its configuration files. (
> http://git.rocksclusters.org/cgi-bin/gitweb.cgi?p=core/python/.git;a=blob;f=src/python-2/Makefile;h=ced6cc1c6eb6a4f70dd0f3e1b0ccd1ac1e40f989;hb=HEAD)
>  shows a Makefile for python that supports this.
>
>
Okay, we should be able to do that anyway I guess. I will do it (probably
with Satish helping me).

> Ultimately we create packages of software to simplify a number of things.
> Including update of installed software on many machines.
>
> The other "issue" is not really an issue, but a request. When petsc is
> built it creates a petscvariables (or similarly named) file with lots of
> environment variables.  It would be terrific if it could also create  an
> environment modules files with this same information.  Users often want
> different versions/different configs of the same software and environment
> modules is now standard in Redhat/CentOS distributions.
>
>
We already create a PETSc.mk file for pkgconfig, so doing another one is
not hard. Here is us doing that:

https://bitbucket.org/petsc/petsc-dev/src/b94b5d6f0ee5d4de5a64fc5f0d1a4e01c743cf04/config/PETSc/Configure.py?at=default#cl-108

and the Dump() function sets up the information in the petscvariables file
by calling addMakeMacro(). Care to try it yourself? I do not know
the format for a modules file. We are always available to help.

  Thanks,

     Matt

> Hope this helps.
>
>
> On Thu, Jan 24, 2013 at 2:44 AM, Schneider, Barry I. <bschneid at nsf.gov>wrote:
>
>> Barry,
>> I am sending this to Phil Papadopoulos and he can make much more precise
>> recommendations.  Glad you have thick skins.  Being at NSF requires the
>> same physiology.
>>
>> -----Original Message-----
>> From: Barry Smith [mailto:bsmith at mcs.anl.gov]
>> Sent: Wednesday, January 23, 2013 10:19 PM
>> To: Schneider, Barry I.
>> Subject: Re: [petsc-maint #149715] Making a PetSc Roll and Rocks Module
>>
>>
>>    Barry,
>>
>>    By far the most common email we get is regarding installation, so we
>> are always trying to understand how to improve that process. If we can
>> eliminate just 20 percent of installation problems that would save us a
>> great deal of work, so yes, any critique/suggestions are greatly
>> appreciated, and after all these years we have thick skins so can survive
>> even the harshest suggestions.
>>
>>    Barry
>>
>> On Jan 23, 2013, at 6:13 PM, "Schneider, Barry I." <bschneid at nsf.gov>
>> wrote:
>>
>> > Barry,
>> > First, thanks for getting back to me so quickly.   I view this as a way
>> to make things easier for everyone for a library that is really invaluable
>> to many of us.  Phil P. is a pro at making rolls and modules for the Rocks
>> distribution of CentOS that is widely used in the NSF cyberinfrastructure
>> and by others.  I am both a program manager for the XSEDE project and a
>> large user as well so I appreciate the ability to use the library in its
>> most transparent fashion.  After Phil did his thing we tested it and it
>> worked perfectly on our cluster.  Basically, you load the module and any
>> associated modules such as the Intel compiler and appropriate MPI module
>> and then you just use it.  No screwing around with building and the rest of
>> it.  That's the good news.  Of course, you need to make a roll for every
>> specific combination of compiler and MPI but I believe what Phil has
>> learned makes that pretty straightforward to do.  While I cannot speak for
>> Phil, I would expect that anything he learned would be available to you
>> folks if that's what you wanted.  The next step for us will be to
>> incorporate what we did in the Rocks distribution which will eventually
>> propagate to lots of users.  Eventually, there will be an XSEDE
>> distribution which will be used by an even larger number of sites.  We are
>> talking about many thousands of users.  So, I guess what I am really asking
>> is whether you are interested enough in what Phil learned to perhaps modify
>> the current PetSc so that it is easier for users to install or make
>> available the technology for folks to make their own rolls.  Of course, you
>> could decide that is not on your agenda and that's fine.  But if you
>> capitalize on our experience and your great software that would be
>> wonderful and perhaps offer users alternatives to the current way of doing
>> things that would make life easier.
>> >
>> > **************************************************
>> > Dr Barry I. Schneider
>> > Program Director for Cyberinfrastructure Past Chair, APS Division of
>> > Computational Physics Physics Division National Science Foundation
>> > 4201 Wilson Blvd.
>> > Arlington, VA 22230
>> > Phone:(703) 292-7383
>> > FAX:(703) 292-9078
>> > Cell:(703) 589-2528
>> > **************************************************
>> >
>> >
>> > -----Original Message-----
>> > From: Barry Smith [mailto:bsmith at mcs.anl.gov]
>> > Sent: Wednesday, January 23, 2013 5:56 PM
>> > To: petsc-maint at mcs.anl.gov; Schneider, Barry I.
>> > Subject: Re: [petsc-maint #149715] Making a PetSc Roll and Rocks
>> > Module
>> >
>> >
>> >   Barry,
>> >
>> >     Thanks for the feedback. We actually desire to support "system"
>> installs just as easily as "home directory" installs, though our
>> documentation emphasizes "home directory" installs since we feel that is
>> most practical for more users. We'd be interested in more specifics on the
>> difficulties and any suggestions there may be to make the process
>> (especially for multiple compilers/mpis) easier. That is what could we have
>> done differently to make it easier for you?
>> >
>> >    Thanks
>> >
>> >    Barry
>> >
>> > On Jan 23, 2013, at 1:59 PM, "Schneider, Barry I." <bschneid at nsf.gov>
>> wrote:
>> >
>> >> I thought I might pass on the following to the folks maintaining
>> PetSc.  Recently we had the occasion to make a Rocks roll and module to be
>> used on a local cluster here at NSF.  Phil Papadopoulos the developer of
>> Rocks at SDSC took the time to help us to do that not simply because we
>> wanted it but also because he wanted to be able to know how to do it and
>> distribute it to a much larger set users of NSF resources and also on out
>> NSF supported platforms.  Here are his comments.  Perhaps if you feel there
>> are things that could be made easier that would be great.  It also could be
>> useful to you directly.
>> >>
>> >> BTW, Petsc is kind of a bear to package -- they really expect you to
>> build it in your home directory :-).
>> >> I took the time to make the roll pretty flexible in terms of
>> compiler/mpi/network support to build various versions.
>> >> This was modeled after other Triton rolls so that it is consistent
>> with other software -- it likely becomes part of the standard SDSC software
>> stack, so this was a good thing to do all the way around.
>> >>
>> >> I'm about ready to add this to our code repository, it will show up on
>> git.rocksclusters.org<http://git.rocksclusters.org> tomorrow morning.
>> >>
>> >>
>> >> **************************************************
>> >> Dr Barry I. Schneider
>> >> Program Director for Cyberinfrastructure Past Chair, APS Division of
>> >> Computational Physics Physics Division National Science Foundation
>> >> 4201 Wilson Blvd.
>> >> Arlington, VA 22230
>> >> Phone:(703) 292-7383
>> >> FAX:(703) 292-9078
>> >> Cell:(703) 589-2528
>> >> **************************************************
>> >>
>> >>
>> >
>>
>>
>
>
> --
> Philip Papadopoulos, PhD
> University of California, San Diego
> 858-822-3628 (Ofc)
> 619-331-2990 (Fax)
>
>
>

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20130124/bb333c52/attachment.html>