Petsc on Blue Gene

[Satish Balay]

On Tue, 10 Jul 2007, Brian Biskeborn wrote:

> 
> Hello,
> 
> Does anyone here have experience with using Petsc on a Blue Gene system?
> I'm at IBM's Almaden Research Center and am currently involved in porting
> some code that uses Petsc to Blue Gene.  I've managed to compile and run
> the code, and it seems to produce the correct results, but it generates
> many floating point alignment exceptions at runtime.

Can you send a log of these messages? Is this on BGL or BGP? Does the
program abort? [on encountering these messages]

With the minimal runs I've done on BGL - I don't remember seing any
such messages.

>  I traced some of these warnings to a call to MatAssemblyBegin, so
> I'm thinking maybe the problem is in Petsc.  The Blue Gene
> architecture requires 8- or 16-byte memory alignment for floating
> point operations, depending on whether they are issued in parallel,
> so I tried bumping up the alignment value in
> src/sys/src/memory/mal.c.  This had no effect on the problem.

[Barry can confirm this] the code in mal.c attempts to make sure the
memory allocated by PETSc is aligned properly. [8 byte boundary for
doubles]

One possibility is that the data passed in to MatAssemblyBegin() is
not aligned?

Satish

> I didn't write any of the code I'm porting, and I'm not too familiar with
> it yet, so I suppose the problem could really be anywhere.  My question
> here is simply this: has anyone seen this problem?  Does it sound like an
> issue with Petsc, or is it more likely to be in the top-level code?
> 
> Thanks, and best regards,
> Brian
> 
>