[petsc-dev] Jed!!!

[Eric Chamberland]

On 04/29/2014 03:14 PM, Jed Brown wrote:
> Eric Chamberland <Eric.Chamberland at giref.ulaval.ca> writes:
>>   > You're supposed to use MatGetLocalSubMatrix().  Translating from global
>>   > indices to local indices is a disaster that we want to avoid.  So we go
>>   > the other way.  Speak the language of "split local" spaces during
>>   > assembly and the data structure itself can live wherever is most
>>   > efficient for the solver.
>>
>> We have a question about this:
>>
>> Where is the "disaster"?  In our mind, we think the  MatSetValues_Nest
>> should:
>>
>> 1) do the assembly for local lines in the proper submatrices
>
> How do you determine which block owns a given column?  Note that the
> column space is huge (can't store it) and arbitrary index sets can be
> used to define the splits.  It is not scalable to gather this
> problem-sized data to each process and it's a major headache to send the
> indices to the process owning that column range to translate it for us.
>

Ok.  In sequential, for a general IS, we are right now creating a vector 
that contains what is the sub-matrix for each column and a second vector 
that gives the index relation between the "complete matrix" and 
"sub-matrix".

We are thinking about a not so bad implementation with "strides" which 
could be acceptable in parallel.

But for the "general" case, it is as you stated, far more complicated. 
However, a clean solution would involves communicating only the 
neighbors column indices and then searching through them (which is true 
also for the "strides" implementation: you always only need the strides 
of your neighbors)


>
> Do you choose a special ordering so that you can determine block
> membership in a memory-scalable way?
>

Not right now (but soon ;) ).  We are rewriting things so that all the 
"linear" parts of the displacement/velocity fields will be continuous on 
each process (right now, it is split by u_x, u_y and u_z).  If you have 
a scalar field, the ordering is naturally done for the linear part first.

Thanks,

Eric