[petsc-users] Some questions regarding VecNest type.

[Vijay S. Mahadevan]

> Deciding when to have a deep hierarchy and when to flatten or reorder for
> locality is something that would be hard for the library to do
> automatically. You can always put a VecNest inside if you want to walk the
> tree that way.

Yes but petsc-ext did handle the "blocks" hierarchically only, if I
remember right. And parallel vectors in petsc were always "flat". Are
you suggesting that there is a way to interchangeably use these in the
current Nest implementation ?

> different approach is to make an ISNest which would make the matching
> problem easier.

I like this. And the interface is also consistent in the sense that a
Nest type propagates the entire hierarchy of IS/Vec/Mat/(KSP?).

> You can do VecNest that way, in which case you can do somewhat more flexible
> ghost updates and such. But VecNest will also let you have the two "blocks"
> be interlaced in the global ordering, but actually stored contiguously. I
> think this is rarely a good way to implement things, but you can do it.

I can think of cases where that might be a good thing to have. Of
course these are problem/discretization dependent and switching the
ordering/nesting at runtime will be a huge advantage to optimize
efficiency.

> How did you get a monolithic matrix that you could hit with a direct solver
> when you were using petsc-ext?

Ah. I didn't. Once I was sure enough that I was converging to the
right solution, I switched to petsc-ext and used only Krylov solves
with block diagonal lu/ilu preconditioners. And those worked very
nicely for my problems.

On Wed, Oct 12, 2011 at 2:34 PM, Jed Brown <jedbrown at mcs.anl.gov> wrote:
> On Wed, Oct 12, 2011 at 14:08, Vijay S. Mahadevan <vijay.m at gmail.com> wrote:
>>
>> > vecnest.c:639 ?
>>
>> Oops. Must have had a typo in my search.
>>
>> > What do you mean by multiple indices? The union of multiple constituent
>> > index sets? You could make a hierarchical VecNest if you really want
>> > that
>> > layout, otherwise we need to find an inexpensive way to handle exotic
>> > index
>> > sets without losing structure.
>>
>> Yes. I meant the union of indexes. i.e, if parent_vec = {child1,
>> child2, .. childn}, then VecGetSubVector([1,2]) would then return
>> subparent_vec={child1, child2} ? Not sure if I would call that exotic
>> but I feel this is natural since I think of the vector as truly
>> nested.
>
> Deciding when to have a deep hierarchy and when to flatten or reorder for
> locality is something that would be hard for the library to do
> automatically. You can always put a VecNest inside if you want to walk the
> tree that way. Solving the subset matching problem efficiently is tricky,
> but if someone wants to write support for that, we can do what you want. A
> different approach is to make an ISNest which would make the matching
> problem easier.
>
>>
>> If you are expecting the IS to contain [1..len(child1),
>> len(child1)+1..len(child1)+len(child2)], then the parent_vec isn't
>> truly nested but rather a regular parallel vector. Aren't these the
>> distinctions between -vec_type nest and otherwise ?
>
> You can do VecNest that way, in which case you can do somewhat more flexible
> ghost updates and such. But VecNest will also let you have the two "blocks"
> be interlaced in the global ordering, but actually stored contiguously. I
> think this is rarely a good way to implement things, but you can do it.
>
>>
>> This is also what
>> I gather from your explanation below. Please correct me if I am wrong.
>>
>> > Since you can write your
>> > software so that this choice is a run-time option, I don't see a good
>> > rationale for choosing an interface that only does one thing.
>>
>> Agreed. As I mentioned earlier, I originally did have both the options
>> but only compile time since I was using petsc-ext.
>
> How did you get a monolithic matrix that you could hit with a direct solver
> when you were using petsc-ext?
>