Petsc on Blue Gene

[Satish Balay]

On Tue, 10 Jul 2007, Satish Balay wrote:

> On Tue, 10 Jul 2007, Brian Biskeborn wrote:
> 
> > 
> > Hello,
> > 
> > Does anyone here have experience with using Petsc on a Blue Gene system?
> > I'm at IBM's Almaden Research Center and am currently involved in porting
> > some code that uses Petsc to Blue Gene.  I've managed to compile and run
> > the code, and it seems to produce the correct results, but it generates
> > many floating point alignment exceptions at runtime.
> 
> Can you send a log of these messages? Is this on BGL or BGP? Does the
> program abort? [on encountering these messages]
> 
> With the minimal runs I've done on BGL - I don't remember seing any
> such messages.
> 
> >  I traced some of these warnings to a call to MatAssemblyBegin, so
> > I'm thinking maybe the problem is in Petsc.  The Blue Gene
> > architecture requires 8- or 16-byte memory alignment for floating
> > point operations, depending on whether they are issued in parallel,
> > so I tried bumping up the alignment value in
> > src/sys/src/memory/mal.c.  This had no effect on the problem.
> 
> [Barry can confirm this] the code in mal.c attempts to make sure the
> memory allocated by PETSc is aligned properly. [8 byte boundary for
> doubles]
> 
> One possibility is that the data passed in to MatAssemblyBegin() is
> not aligned?

Sorry - I meant to say MatSetValues()..

Satish

> 
> Satish
> 
> > I didn't write any of the code I'm porting, and I'm not too familiar with
> > it yet, so I suppose the problem could really be anywhere.  My question
> > here is simply this: has anyone seen this problem?  Does it sound like an
> > issue with Petsc, or is it more likely to be in the top-level code?
> > 
> > Thanks, and best regards,
> > Brian
> > 
> > 
> 
>