Petsc Development Question

[Barry Smith]

   Paul,

    This is a very appropriate question at this time since multi-core seems
to be the wave of the future.

    The PETSc model will be the same, each "process" will continue have its own
"processor" (core) and each process will have its OWN address space (memory it 
can access directly from the programming language). That is, the OpenMP model
will NOT come into play. The "standard" MPI model will continue (with one MPI
rank per process (core)).

   BUT: for performance benefits from multi-core, the two "parallel 
communication" operations that dominate in PETSc: global reductions (like 
VecNorm) and neighbor to neighbor communication (VecScatter) will "by-pass"
the overhead of traditional MPI call overhead WITHIN a node (all the cores on 
the same chip). This will be completely transparent to PETSc users and in fact
almost completely transparent to PETSc (I have discussed these issues with the
MPICH group at ANL).

   I feel any other approach is completely unrealistic, there is no way we
could recode PETSc to use OpenMP on a node and require users to do so as well! 
MOST IMPORTANTLY that approach would NOT offer significent improvement over our
approach (if I am wrong about this, only after I have seen a real demonstration
of real timings on a real application on a real machine, will my thinking
change.).

   Barry

General note: sparse matrix and vector computations will not benefit much
at all from multi-core (that is two cores will not give anywhere near twice the 
performance of one core) if the memory-bandwidth to the cores is not high
enough. So when looking at multi-core, look at the bandwidth PER core, unless
that is high, don't buy the system, you'll be buying cores that don't do much 
for you but heat up the machine.




On Fri, 19 May 2006, Paul Dionne wrote:

> Barry,
>
> You visited us a while back at CFD Research Corporation, and we are finally 
> getting around to starting the project for which we will be using Petsc.  I 
> have a couple of questions about your potential future work.  My parallel 
> background is fairly weak, so I'll apologize in advance if these questions 
> are either silly or worded wrong.
>
> Do you have any plans on development for exploiting multi-core processors? 
> One of the guys here was saying that a parallel code could be sped up 
> considerably if communication was arranged to take advantage of the faster 
> communication within a computer compared to communication between computers.
>
> If you have plans for this, or would be considering it in the future, would I 
> be correct in assuming that this is something that would be controlled 
> entirely by Petsc?  In other words, we could develop our code with Petsc as 
> it is, and if a multi-core version came out later there is nothing additional 
> that we would have to do to gain the additional speed.
>
> Thanks,
> Paul Dionne
>
>