Can PETSc detect the number of CPUs on each computer node?

[Alex Peyser]

On Tuesday 16 June 2009 01:53:35 pm Matthew Knepley wrote:
> On Tue, Jun 16, 2009 at 12:38 PM, xiaoyin ji
> <sapphire.jxy at gmail.com<mailto:sapphire.jxy at gmail.com>> wrote: Hi there,
>
> I'm using PETSc MATMPIAIJ and ksp solver. It seems that PETSc will run
> obviously faster if I set the number of CPUs close to the number of
> computer nodes in the job file. By default MPIAIJ matrix is stored in
> different processors and ksp solver will communicate for each step,
> however since on each node several CPUs share the same memory while
> ksp may still try to communicate through network card, this may mess
> up a bit. Is there any way to detect which CPUs are sharing the same
> memory? Thanks a lot.
>
> The interface for this is mpirun or the job submission mechanism.
>
>    Matt
>
>
> Best,
> Xiaoyin Ji
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead. -- Norbert Wiener

I had a question on what is the best approach for this. Most of the time is 
spent inside of BLAS, correct? So wouldn't you maximize your operations by 
running one MPI/PETSC job per board (per shared memory), and use a 
multi-threaded BLAS that matches your board? You should cut down 
communications by some factor proportional to the number of threads per 
board, and the BLAS itself should better optimize most of your operations 
across the board, rather than relying on higher order parallelisms.

Regards,
Alex Peyser
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 198 bytes
Desc: This is a digitally signed message part.
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20090616/54d8bfae/attachment.pgp>