Cray and PETSc

[Tom Cortese]

Hello,

Sure, I would be happy to share my experiences; I have amassed a 
decent-sized set of "when building PETSc on this sort of machine, use 
configure options that look like that" data points, and there is no sense 
in re-inventing the wheel.

As for external packages connected with PETSc, the most commonly-requested 
ones that I have heard of are HYPRE, MUMPS, and SLEPC (and also SuperLU, 
but you mention that is already part of Cray LibSci -- is it possible to 
access this SuperLU from within PETSc on Cray, or do users have to access 
SuperLU routines directly?).

I did make an installation of PETSc with BLOCKSOLVE95 for one particular 
user, and also an installation of PETSc with parMETIS, SLEPC, complex 
scalars, etc... for another specific user, but these appear to be isolated 
incidents.

I am a bit hesitant to ask our users if there are any more; what if they 
request 87 other things -- then not only would Cray have to figure out how 
to make all of them available, but someone (most likely me) would have to 
figure out how to install each of these 87 other things on all the OTHER 
types of HPC machines that I have to deal with (this sounds like an 
experience which would contain a less-than-optimal amount of fun).  I 
suppose I could do an informal survey, along with some sort of "we can't 
make any promises, but we're interested in knowing which PETSc external 
packages would be useful" caveat...

I have one more system-administrator-type question (I'm not a system 
administrator, but I play one on TV):  If PETSc is configured to download 
packages such as MUMPS, HYPRE, etc..., can users wishing to access the 
packages directly (i.e., use MUMPS without going through the PETSc 
interface), still access them?  If the answer is "yes", then the rest of 
this email is just un-necessary speculation...

I'm asking this because we have disk-space limitations on some sites, and 
I don't want to end up having, say, MUMPS and HYPRE installed separately 
by themselves (for users who wish to access them directly), and then 
configuring PETSc with "download MUMPS" and "download HYPRE", thereby 
creating extra copies of these libraries.

Of course, I know the intelligent answer would be "install MUMPS", 
"install HYPRE", then "configure PETSc to point to these existing 
installations and then build PETSc", but this simple-sounding procedure 
does not always work (I often get error messages saying something like 
"You specified --with-hypre=blahblahblah, but blahblahblah does not work", 
which is why I use the "download HYPRE" PETSc configure option).

 	-Tom Cortese
 	PET Computational Environment Onsite

On Thu, 23 Aug 2007, Adrian Tate wrote:

>
> Tom
>
> Thanks for your comments - I'm sure your experiences with PETSc on Crays will be helpful to us. Perhaps Keita and I could communicate directly with you on these issues and to find out more about the requirements of your users?
>
> It is our ultimate goal to provide everything that Cray customers need with PETSc, but I can't comment on how feasible that is without knowing more about the packages that are widely used. For our initial release we were thinking of providing HYPRE and ParMETIS, we also provide SuperLU already within Cray LibSci. MUMPS is certainly something we should consider.
>
> This is a perfect opportunity to ask PETSc developers that use Cray hardware to suggest packages that they'd like to see provided by Cray.
>
> Adrian
>
> -----Original Message-----
> From: Tom Cortese [mailto:tcortese at cs.utk.edu]
> Sent: Thursday, August 23, 2007 10:53 AM
> To: Adrian Tate
> Cc: Barry Smith; petsc-developers at mcs.anl.gov; petsc-dev at mcs.anl.gov; Keita Teranishi
> Subject: RE: Cray and PETSc
>
> Hello all,
>
> First of all, I am NOT a PETSc developer, but as a member of the DoD HPC
> Modernization Program I DO install PETSc on a wide variety of hardware
> platforms (including XT3, XD1, and even X1 [that was rough; had to
> cross-compile on one machine and transfer files]), and have had numerous
> email exchanges working out all of the little quirks involved in
> installing PETSc on these machines, so the PETSc folks have been kind
> enough to let me watch this development email list.
>
> I hope I'm not over-stepping my bounds by asking this, but I have two
> comments:
>
> 1)  Making PETSc available on the Cray machines sounds like a GREAT idea
>
> 2)  Several of our users also like to have packages such as HYPRE and
> MUMPS, etc... installed along with PETSc.  Would such external packages
> also be installed by Cray, or would users requiring a combination of, say,
> PETSc and MUMPS still have to make their own installation?
>
> Thanks,
>
> 	-Tom Cortese
> 	PET Computational Environnment Onsite
>
> On Thu, 23 Aug 2007, Adrian Tate wrote:
>
>>
>> Hi Barry
>>
>> Thanks for your reply.
>>
>> We will expect to provide the new full releases of PETSc as soon as we possibly can. Obviously there is something of a lag because Cray need to test, package and release the library, but we'll try to do so with minimal delay. Obviously any heads-up you can give us in advance will help reduce the lag. With respect to patch versions, it is going to be very difficult for Cray to release every new version, and by looking at the logs it seems that many of the patches would not be relevant to a Cray specific build. We will endeavor to monitor all patch releases and will go with an unscheduled new release of PETSc when we see a relevant patch. One thing that we can do to minimize confusion is to informally provide the most currently patched version to any customer who is experiencing a problem, although we might not go with a full release of the version in question.
>>
>> Regarding Cray modifications, we (Keita Teranishi or myself) will check in periodically with you to show you our improvements and to see which, if any need to be channeled back into the PETSc build.
>>
>> I hope this support structure works OK from your perspective and I look forward to working with the PETSc team.
>>
>> Thanks!
>> Adrian
>>
>> -----Original Message-----
>> From: Barry Smith [mailto:bsmith at mcs.anl.gov]
>> Sent: Monday, August 20, 2007 11:12 AM
>> To: Adrian Tate
>> Cc: petsc-developers at mcs.anl.gov; petsc-dev at mcs.anl.gov
>> Subject: Re: Cray and PETSc
>>
>>
>>   Adrian,
>>
>>    Thank you for in the inquiry.
>>
>>
>> On Thu, 16 Aug 2007, Adrian Tate wrote:
>>
>>> Hello Barry
>>>
>>> I'm not sure we met in person - I am the lead of the libraries group
>>> at Cray. We decided some time ago that our iterative solver strategy
>>> would be to leverage PETSc, and to hopefully provide some Cray
>>> specific tunings to improve performance of the KSP solvers (largely
>>> through our custom sparse BLAS and some parallel tuning for our
>>> interconnect). I believe that John Lewis has been in communication
>>> with you with respect to the tuning of Sparse BLAS and their
>>> integration with the PETSc build.
>>>
>>> We are however, considering packaging and supplying PETSc along with
>>> the scientific library that we provide (libSci). This allows for
>>> better integration of our internal kernels and also it means that we
>>> are no longer requiring users (including benchmakers and internal
>>> users) to build their own PETSc. I see from your online page that
>>> doing so is acceptable as long as we use a copyright switch during
>>> configure. By applying this switch, do we make our PETSc library
>>> unsupported from your perspective?
>>
>>   The GNU copyright code is tiny; it would not effect the usability
>> or support of PETSc
>>
>>> We do not expect to be able to
>>> provide anywhere near the degree of support that your team provide,
>>> and I was hoping to supply a pre-built library whose users could still
>>> seek assistance through your normal support channels - is this
>>> realistic?
>>>
>>   Yes.
>>
>>   The two isses that concern us with pre-packaged versions of PETSc are
>> 1) keeping up to date on our releases. We generally make two releases
>> a year and much prefer that users use the most recently release.
>> If they are using an older release it means we are less able to help
>> them.
>> 2) keeping up to date on our patches. We may make several bug patches
>> to each release. Users with a pre-packaged version have trouble keeping
>> up with the source code patches we provide.
>>
>>>
>>>
>>>
>>>
>>> Also, I would be interested to know your degree of interest in the
>>> Cray-specific modifications that we make - would you prefer those to
>>> be channeled back into the PETSc library?
>>
>>  If they involved directly changing PETSc code, we much prefer to get
>> it channeled back into the master copy of the source code. Makes it
>> much easier to debug user code. If it is auxiliary code, like a faster
>> ddot() etc then it is more appropriate to not try to include it.
>>
>>> Any other comments you have
>>> on the way that Cray can contribute to the PETSc project, I would be
>>> very glad to hear.
>>
>>  PETSc has a variety of "tuning factors" that could theoretically be
>> set optimally for a particular machine, these range from simple things
>> like compiler options, to a choice between C and Fortran versions of
>> the same code (what we call them Fortran kernels), to different loop
>> unrollings (in the inline.h file), even something like PetscMemzero()
>> which has five possible forms. Now currently we do not tune these
>> or even have a test harness for selecting a good tuning. One thing
>> you could/would like to do, is determine good choices for these
>> options on your machines. Just as a simple example, on some Linux systems
>> the basic libraries are just compiled with the GNU compiler, hence the
>> system memset() is not particularly effective. A version compiled of PETSc
>> compiled using a "Fortran memset" may be much faster, or Intel provides
>> its own _intel_fast_memset() which is better. I've seen a few percent
>> increase in performance of entire nonlinear PDE solver applications
>> just from using the non-default memset(). [This was specifically on
>> an Itanium system, but is likely on other configurations also].
>>
>>
>>   Barry
>>
>>
>>>
>>>
>>>
>>> Regards,
>>>
>>>
>>> Adrian Tate
>>>
>>> ---
>>>
>>> Technical Lead
>>>
>>> Math Software
>>> Cray Inc.
>>>
>>> (206) 349 5868
>>>
>>>
>>
>>
>