Cray and PETSc

[Barry Smith]

On Thu, 23 Aug 2007, Tom Cortese wrote:

> Hello,
> 
> Sure, I would be happy to share my experiences; I have amassed a decent-sized
> set of "when building PETSc on this sort of machine, use configure options
> that look like that" data points, and there is no sense in re-inventing the
> wheel.
> 
> As for external packages connected with PETSc, the most commonly-requested
> ones that I have heard of are HYPRE, MUMPS, and SLEPC (and also SuperLU, but
> you mention that is already part of Cray LibSci -- is it possible to access
> this SuperLU from within PETSc on Cray, or do users have to access SuperLU
> routines directly?).
> 
> I did make an installation of PETSc with BLOCKSOLVE95 for one particular user,
> and also an installation of PETSc with parMETIS, SLEPC, complex scalars,
> etc... for another specific user, but these appear to be isolated incidents.
> 
> I am a bit hesitant to ask our users if there are any more; what if they
> request 87 other things -- then not only would Cray have to figure out how to
> make all of them available, but someone (most likely me) would have to figure
> out how to install each of these 87 other things on all the OTHER types of HPC
> machines that I have to deal with (this sounds like an experience which would
> contain a less-than-optimal amount of fun).  I suppose I could do an informal
> survey, along with some sort of "we can't make any promises, but we're
> interested in knowing which PETSc external packages would be useful" caveat...
> 
> I have one more system-administrator-type question (I'm not a system
> administrator, but I play one on TV):  If PETSc is configured to download
> packages such as MUMPS, HYPRE, etc..., can users wishing to access the
> packages directly (i.e., use MUMPS without going through the PETSc interface),
> still access them?  If the answer is "yes", then the rest of this email is
> just un-necessary speculation...

   Yes

> 
> I'm asking this because we have disk-space limitations on some sites, and I
> don't want to end up having, say, MUMPS and HYPRE installed separately by
> themselves (for users who wish to access them directly), and then configuring
> PETSc with "download MUMPS" and "download HYPRE", thereby creating extra
> copies of these libraries.
> 
> Of course, I know the intelligent answer would be "install MUMPS", "install
> HYPRE", then "configure PETSc to point to these existing installations and
> then build PETSc", but this simple-sounding procedure does not always work (I
> often get error messages saying something like "You specified
> --with-hypre=blahblahblah, but blahblahblah does not work", which is why I use
> the "download HYPRE" PETSc configure option).
> 
> 	-Tom Cortese
> 	PET Computational Environment Onsite
> 
> On Thu, 23 Aug 2007, Adrian Tate wrote:
> 
> > 
> > Tom
> > 
> > Thanks for your comments - I'm sure your experiences with PETSc on Crays
> > will be helpful to us. Perhaps Keita and I could communicate directly with
> > you on these issues and to find out more about the requirements of your
> > users?
> > 
> > It is our ultimate goal to provide everything that Cray customers need with
> > PETSc, but I can't comment on how feasible that is without knowing more
> > about the packages that are widely used. For our initial release we were
> > thinking of providing HYPRE and ParMETIS, we also provide SuperLU already
> > within Cray LibSci. MUMPS is certainly something we should consider.
> > 
> > This is a perfect opportunity to ask PETSc developers that use Cray hardware
> > to suggest packages that they'd like to see provided by Cray.
> > 
> > Adrian
> > 
> > -----Original Message-----
> > From: Tom Cortese [mailto:tcortese at cs.utk.edu]
> > Sent: Thursday, August 23, 2007 10:53 AM
> > To: Adrian Tate
> > Cc: Barry Smith; petsc-developers at mcs.anl.gov; petsc-dev at mcs.anl.gov; Keita
> > Teranishi
> > Subject: RE: Cray and PETSc
> > 
> > Hello all,
> > 
> > First of all, I am NOT a PETSc developer, but as a member of the DoD HPC
> > Modernization Program I DO install PETSc on a wide variety of hardware
> > platforms (including XT3, XD1, and even X1 [that was rough; had to
> > cross-compile on one machine and transfer files]), and have had numerous
> > email exchanges working out all of the little quirks involved in
> > installing PETSc on these machines, so the PETSc folks have been kind
> > enough to let me watch this development email list.
> > 
> > I hope I'm not over-stepping my bounds by asking this, but I have two
> > comments:
> > 
> > 1)  Making PETSc available on the Cray machines sounds like a GREAT idea
> > 
> > 2)  Several of our users also like to have packages such as HYPRE and
> > MUMPS, etc... installed along with PETSc.  Would such external packages
> > also be installed by Cray, or would users requiring a combination of, say,
> > PETSc and MUMPS still have to make their own installation?
> > 
> > Thanks,
> > 
> > 	-Tom Cortese
> > 	PET Computational Environnment Onsite
> > 
> > On Thu, 23 Aug 2007, Adrian Tate wrote:
> > 
> > > 
> > > Hi Barry
> > > 
> > > Thanks for your reply.
> > > 
> > > We will expect to provide the new full releases of PETSc as soon as we
> > > possibly can. Obviously there is something of a lag because Cray need to
> > > test, package and release the library, but we'll try to do so with minimal
> > > delay. Obviously any heads-up you can give us in advance will help reduce
> > > the lag. With respect to patch versions, it is going to be very difficult
> > > for Cray to release every new version, and by looking at the logs it seems
> > > that many of the patches would not be relevant to a Cray specific build.
> > > We will endeavor to monitor all patch releases and will go with an
> > > unscheduled new release of PETSc when we see a relevant patch. One thing
> > > that we can do to minimize confusion is to informally provide the most
> > > currently patched version to any customer who is experiencing a problem,
> > > although we might not go with a full release of the version in question.
> > > 
> > > Regarding Cray modifications, we (Keita Teranishi or myself) will check in
> > > periodically with you to show you our improvements and to see which, if
> > > any need to be channeled back into the PETSc build.
> > > 
> > > I hope this support structure works OK from your perspective and I look
> > > forward to working with the PETSc team.
> > > 
> > > Thanks!
> > > Adrian
> > > 
> > > -----Original Message-----
> > > From: Barry Smith [mailto:bsmith at mcs.anl.gov]
> > > Sent: Monday, August 20, 2007 11:12 AM
> > > To: Adrian Tate
> > > Cc: petsc-developers at mcs.anl.gov; petsc-dev at mcs.anl.gov
> > > Subject: Re: Cray and PETSc
> > > 
> > > 
> > >   Adrian,
> > > 
> > >    Thank you for in the inquiry.
> > > 
> > > 
> > > On Thu, 16 Aug 2007, Adrian Tate wrote:
> > > 
> > > > Hello Barry
> > > > 
> > > > I'm not sure we met in person - I am the lead of the libraries group
> > > > at Cray. We decided some time ago that our iterative solver strategy
> > > > would be to leverage PETSc, and to hopefully provide some Cray
> > > > specific tunings to improve performance of the KSP solvers (largely
> > > > through our custom sparse BLAS and some parallel tuning for our
> > > > interconnect). I believe that John Lewis has been in communication
> > > > with you with respect to the tuning of Sparse BLAS and their
> > > > integration with the PETSc build.
> > > > 
> > > > We are however, considering packaging and supplying PETSc along with
> > > > the scientific library that we provide (libSci). This allows for
> > > > better integration of our internal kernels and also it means that we
> > > > are no longer requiring users (including benchmakers and internal
> > > > users) to build their own PETSc. I see from your online page that
> > > > doing so is acceptable as long as we use a copyright switch during
> > > > configure. By applying this switch, do we make our PETSc library
> > > > unsupported from your perspective?
> > > 
> > >   The GNU copyright code is tiny; it would not effect the usability
> > > or support of PETSc
> > > 
> > > > We do not expect to be able to
> > > > provide anywhere near the degree of support that your team provide,
> > > > and I was hoping to supply a pre-built library whose users could still
> > > > seek assistance through your normal support channels - is this
> > > > realistic?
> > > > 
> > >   Yes.
> > > 
> > >   The two isses that concern us with pre-packaged versions of PETSc are
> > > 1) keeping up to date on our releases. We generally make two releases
> > > a year and much prefer that users use the most recently release.
> > > If they are using an older release it means we are less able to help
> > > them.
> > > 2) keeping up to date on our patches. We may make several bug patches
> > > to each release. Users with a pre-packaged version have trouble keeping
> > > up with the source code patches we provide.
> > > 
> > > > 
> > > > 
> > > > 
> > > > 
> > > > Also, I would be interested to know your degree of interest in the
> > > > Cray-specific modifications that we make - would you prefer those to
> > > > be channeled back into the PETSc library?
> > > 
> > >  If they involved directly changing PETSc code, we much prefer to get
> > > it channeled back into the master copy of the source code. Makes it
> > > much easier to debug user code. If it is auxiliary code, like a faster
> > > ddot() etc then it is more appropriate to not try to include it.
> > > 
> > > > Any other comments you have
> > > > on the way that Cray can contribute to the PETSc project, I would be
> > > > very glad to hear.
> > > 
> > >  PETSc has a variety of "tuning factors" that could theoretically be
> > > set optimally for a particular machine, these range from simple things
> > > like compiler options, to a choice between C and Fortran versions of
> > > the same code (what we call them Fortran kernels), to different loop
> > > unrollings (in the inline.h file), even something like PetscMemzero()
> > > which has five possible forms. Now currently we do not tune these
> > > or even have a test harness for selecting a good tuning. One thing
> > > you could/would like to do, is determine good choices for these
> > > options on your machines. Just as a simple example, on some Linux systems
> > > the basic libraries are just compiled with the GNU compiler, hence the
> > > system memset() is not particularly effective. A version compiled of PETSc
> > > compiled using a "Fortran memset" may be much faster, or Intel provides
> > > its own _intel_fast_memset() which is better. I've seen a few percent
> > > increase in performance of entire nonlinear PDE solver applications
> > > just from using the non-default memset(). [This was specifically on
> > > an Itanium system, but is likely on other configurations also].
> > > 
> > > 
> > >   Barry
> > > 
> > > 
> > > > 
> > > > 
> > > > 
> > > > Regards,
> > > > 
> > > > 
> > > > Adrian Tate
> > > > 
> > > > ---
> > > > 
> > > > Technical Lead
> > > > 
> > > > Math Software
> > > > Cray Inc.
> > > > 
> > > > (206) 349 5868
> > > > 
> > > > 
> > > 
> > > 
> > 
> 
>