[petsc-dev] Understanding Some Parallel Results with PETSc

[Dave Nystrom]

That is what I did for the gpu case where I was running with 2 mpi processes
per node since I had 2 gpus per node.  But if that is not really the right
way to be doing this, I'll be happy to learn of a better way.  Note that for
the CPU case in the runPetscProb_2 script I use a different binding.

Anyway, if there is a better way to do this, I'll be happy to learn and try
the better way.  Also, this is recent work that has been done in a hurry
because of trying to get ready for this new machine coming in.  I probably
need to go back and revisit the details to make sure I understand the binding
I am really getting.

Jed Brown writes:
 > Hmm, I'll look more carefully, but I think your binding is incorrect.
 > 
 > numactl -l --cpunodebind=$OMPI_COMM_WORLD_LOCAL_RANK ex2 $KSP_ARGS
 > 
 > NUMA node numbering is different from MPI ranks.
 > 
 > On Fri, Feb 24, 2012 at 12:11, Nystrom, William D <wdn at lanl.gov> wrote:
 > 
 > >  Hi Jed,
 > >
 > > Attached is a gzipped tarball of the stuff I used to run these two test
 > > problems with
 > > numactl.  Actually, I hacked them a bit this morning because I was running
 > > them in
 > > our test framework for doing acceptance testing of new systems.  But the
 > > scripts in
 > > the tarball should give you all the info you need.  There is a top level
 > > script called
 > > runPetsc that just invokes mpirun from openmpi and calls the wrapper
 > > scripts for using
 > > numactl.  You could actually dispense with the top level script and just
 > > invoke the
 > > mpirun commands yourself.  I include it as an easy way to document what I
 > > did.
 > > The runPetscProb_1 script runs petsc on the gpus using numactl to control
 > > the affinities
 > > of the gpus to the cpu numa nodes.  The runPetscProb_2 script runs petsc
 > > on the cpus
 > > using numactl.  Note that both of those wrapper scripts are using openmpi
 > > variables.
 > > I'm not sure how one would do the same thing with another flavor of mpi.
 > > But I imagine
 > > it is possible.  Also, I'm not sure if there are other more elegant ways
 > > to run with numactl
 > > than using the wrapper script approach.  Perhaps there is but this is what
 > > we have been
 > > doing.
 > >
 > > I've also included a Perl script called numa-maps that is useful for
 > > actually checking the
 > > affinities that you get while running in order to make sure that numactl
 > > is doing what
 > > you think it is doing.  I'm not sure where this script comes from.  I find
 > > it on some systems
 > > and not on others.
 > >
 > > I've also include logs with the output of cpuinfo, nvidia-smi and uname -a
 > > to answer any
 > > questions you had about the system I was running on.
 > >
 > > Finally, I've included  runPetscProb_1.log and runPetscProb_2.log which
 > > contains the
 > > log_summary output for my latest runs on the gpu and cpu respectively.
 > > Using numactl
 > > reduced the runtime for the gpu case as well but not as much as for the
 > > cpu case.  So
 > > the final result was that running the same problem while using all of the
 > > gpu resources
 > > on a node was about 2.5x times faster than using all of the cpu resources
 > > on the same
 > > number of nodes.
 > >
 > > Let me know if you need more any more info.  I'm planning to use this
 > > stuff to help test
 > > a new gpu cluster that we have just started acceptance testing on.  It has
 > > the same
 > > basic hardware as the testbed cluster for these results but has 308
 > > nodes.  That
 > > should be interesting and fun.
 > >
 > >
 > > Thanks,
 > >
 > > Dave
 > >
 > >  --
 > > Dave Nystrom
 > > LANL HPC-5
 > > Phone: 505-667-7913
 > > Email: wdn at lanl.gov
 > > Smail: Mail Stop B272
 > >        Group HPC-5
 > >        Los Alamos National Laboratory
 > >        Los Alamos, NM 87545
 > >
 > >   ------------------------------
 > > *From:* petsc-dev-bounces at mcs.anl.gov [petsc-dev-bounces at mcs.anl.gov] on
 > > behalf of Jed Brown [jedbrown at mcs.anl.gov]
 > > *Sent:* Thursday, February 23, 2012 10:43 PM
 > >
 > > *To:* For users of the development version of PETSc
 > > *Cc:* Dave Nystrom
 > >
 > > *Subject:* Re: [petsc-dev] Understanding Some Parallel Results with PETSc
 > >
 > >   On Thu, Feb 23, 2012 at 23:41, Dave Nystrom <dnystrom1 at comcast.net>wrote:
 > >
 > >> I could also send you my mpi/numactl command lines for gpu and cpu when I
 > >> am
 > >> back in the office.
 > >>
 > >
 > > Yes, please.
 > >