[petsc-dev] Generality of VecScatter

[Mark F. Adams]

Your instance on a library model seems reasonable if language support does not give you much better performance/programmability/etc.

And I did not see that you were talking about threads, I was thinking you were looking at an event driven thing like CHARM++.

So you want MPI to provide an incremental path to a model that will work at exa-scale and pthreads are part of that?  So you are in the MPI + pthreads camp?

Mark

On Nov 24, 2011, at 2:27 PM, Barry Smith wrote:

> 
> On Nov 24, 2011, at 12:52 PM, Mark F. Adams wrote:
> 
>> 
>> On Nov 24, 2011, at 12:31 PM, Barry Smith wrote:
>> 
>>> 
>>> On Nov 24, 2011, at 9:51 AM, Mark F. Adams wrote:
>>> 
>>>> Barry, 
>>>> 
>>>> I totally disagree.  About 5 years ago a lot of lumpenproletariat CS people were crowing that MPI was dead and after a few years of naval gazing w/o being able to compete the sentence "and will be replaced with ..." they quietly resigned themselves to the fact that MPI will be on top for the vast majority of apps for the foreseeable future.  I think anyone with their feet on the ground, even the most pro future languages people like Kathy Yelick, sees MPI being in the mix for a long time.  Now a days I see lots of "MPI + ?" on slides, or even "MPI + OpenMP + ?" but PGAS languages are still at the fringes.
>>> 
>>>  You seem to have the tact assumption that the choice is between MPI+X or some PGAS language, this I say is a major straw man!
>> 
>> Well of course it is!
>> 
>>> (And I totally agree with you PGAS language are junk and will never take off). 
>>> 
>>>   What I am suggesting is there is a third alternative that is compatible with MPI (allowing people to transition from MPI to it without rewriting all current MPI based code) and is COMPLETELY library based (as MPI is).
>> 
>> I with you so far,
>> 
>>> This new thing would provide a modern API suitable for many core, for GPUs and distributed computing and the API would focus on moving data and scheduling tasks.
>> 
>> I don't see how you can do a task based thing with anything that looks remotely like MPI or can be achieved in any kind of an incremental way, but I'd be interested in looking at what you have in mind.
> 
>    Programming with pthreads is library based programming. There is an API that you call in your language with function calls from your code to do stuff, it is in no way an extension to the language.  Now there are some issues with the pthread API and I am not advocating it but just using it as an example of how flexible parallel programming can be done on-node using a purely library based system.
> 
> 
>   Barry
> 
>> 
>> Mark
>> 
>>> The API will be suitable for C/C++/Fortran 90. Now exact details of the API and the model are not clear to me of course. I will be pushing the MCS CS folks in this direction because I don't see any other reasonable alternative (MPI+X is fundamentally not powerful enough for the new configurations and PGAS are a joke).
>>> 
>>> 
>>> Barry
>>> 
>>> Very crudely this new thing could be thought of as MPI  - all point to point stuff + neighbor communication (with all communication including Allreduce asynchronize).
>>> 
>>>> 
>>>> Mark
>>>> 
>>>> On Nov 23, 2011, at 11:22 PM, Barry Smith wrote:
>>>> 
>>>>> 
>>>>> These guys are making a very big strategic mistake. They cannot expect to get any funding at all if they continue to purse MPI when the acting and presumably next ASCR director believes with every fiber of his being that MPI is dead. They are writing the death sentence to computer science at ANL.
>>>>> 
>>>>> Barry
>>>>> 
>>>>> 
>>>>> On Nov 23, 2011, at 6:28 PM, Jed Brown wrote:
>>>>> 
>>>>>> I had a useful conversation with the MPICH guys today about the unstructured communication primitives that Mark, Matt, Barry, and I have been discussing. It's looking like we can do a very thin communication library that has the nice semantics I was looking for, namely that the local-to-global is stored only as indices in the local space. To implement the communication primitives, the owner of a global node never needs to know how many or which processes have that point in their local space. There is an MPI-3 feature that we can use to elegantly avoid two-way knowledge for pointwise gather/scatter. (We can always implement the same interface using vanilla MPI-1, but the MPI-2/3 implementation would have less overhead.)
>>>>>> 
>>>>>> If I/we write this layer, VecScatter (with arbitrary data types, etc) could be implemented very easily on top of it. I'm in no rush to do this, but I could imagine getting there eventually. The VecScatter interface allows some things that might not be important. For example, any process can specify an edge in the communication graph, even if it does not own the source or the destination vertex. Also, both source and destination vertices can have degree higher than one. We can even put a link between the same points twice (effectively multiplying that contribution by 2 when using ADD_VALUES). I don't know if this ever makes semantic sense, but the interface allows it.
>>>>>> 
>>>>>> I wonder if we can express all useful communication with a more restrictive interface similar to a local-to-global map (but mapping between any vectors) where one side (typically called "local", but doesn't need to actually be a local Vec) has at most one edge.
>>>>>> 
>>>>>> What weird things do people use VecScatter for that might break with this more restrictive model?
>>>>> 
>>>>> 
>>>> 
>>> 
>>> 
>> 
> 
>