[petsc-dev] MatSetOption is logically collective even for MAT_ROW_ORIENTED.

[Barry Smith]

   No one has commented on my suggestion to make negative MatOptions logically collective and positive ones for non collective.

   Does anyone object to me implementing this?

     Thanks

    Barry

On Sep 20, 2012, at 1:21 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:

> 
>   Patrick,
> 
>    We may have gone overboard with our assertions in this routine. 
> 
>    Some MatOptions really should be collective while others like the one you are using need not be collective.
> 
>    Simple solution just remove those two collective asserts.  Harder solution, use the asserts for only the options that should be checked. We could for example use less then some number for collective asserts and greater than number for for non collective. A little hacky but would work? 
> 
>   Barry
> 
> 
> 
> On Sep 20, 2012, at 11:42 AM, Patrick Lacasse <patrick.m.lacasse at gmail.com> wrote:
> 
>> Hi,
>> 
>>    I just discovered that the way we assemble our matrix cause a deadlock in MatSetOption
>> when petsc is compiled with debugging option enabled.
>> 
>> I resume :
>> - We loop over elements.
>> - For each element we create a small elementary matrix.
>> - We assemble the elementary matrix with MatSetValues into the big one.
>> 
>> Sometimes, we have some rectangular elementary matrix that needs to be added twice,
>> the second time it is transposed. The big matrix is not symmetric for many possible reasons.
>> We are essentially doing it for every elementary matrix that needs it :
>> 
>> MatSetOption(mat,MAT_ROW_ORIENTED,true)
>> MatSetValues(...)
>> MatSetOption(mat,MAT_ROW_ORIENTED,false)
>> MatSetValues(...)
>> 
>> Of course, it fails (deadlock) in MatSetOption at assertion 
>> PetscValidLogicalCollectiveEnum(mat,op,2);
>> because different process don't loop over the same (number of) elements.
>> However, it is working fine without the assertion.
>> 
>> I have two question :
>> Is there really a risk to call MatSetOption(mat, MAT_ROW_ORIENTED, flg) on only one process?
>> What can I do now?
>> 
>> Some possible answer are
>> - Close our eyes and stop compiling petsc with debugging option enabled.
>> - We could copy our elementary matrix into a transposed one before to MatSetValues. We don't want do this cause we think it would slow down our code.
>> - Could MatSetValues has a bonus argument row_oriented?
>> - Could the assertions in MatSetOption be skipped for some options?
>> - Could we add a function MatSetRowOriented which would not be collective?
>> 
>> thanks,
>> 
>> Patrick Lacasse
>