[petsc-dev] Understanding Some Parallel Results with PETSc

[Dave Nystrom]

Jed Brown writes:
 > On Thu, Feb 23, 2012 at 18:53, Nystrom, William D <wdn at lanl.gov> wrote:
 > 
 > > Rerunning the CPU case with numactl results in a 25x speedup and
 > > log_summary
 > > results that look reasonable to me now.
 > >
 > 
 > What command are you using for this?  We usually use the affinity options to
 > mpiexec instead of using numactl/taskset manually.

I was using openmpi-1.5.4 as installed by the system admins on our testbed
cluster.  I talked to a couple of our openmpi developers and they indicated
that the affinity stuff was broken in that version but should be fixed when
1.5.5 and 1.6 come out - which should be within the next month.

I also tried mvapich2-1.7 built with slurm and tried using the affinity stuff
with srun.  That also did not seem to work.  But I should probably revisit
that and try to make sure that I really understand how to use srun.

I was pretty surprised that getting the numa stuff right made such a huge
difference.  I'm also wondering if getting the affinity right will make much
of a difference for the gpu case.

 > Did you also set a specific memory policy?

I'm not sure what you mean by the above question but I'm kind of new to all
this numa stuff.

 > Which Linux kernel is this?

The OS was the latest beta of TOSS2.  If I remember, I can check next time I
am in my office.  It is probably RHEL6.