[petsc-dev] MatSetOption is logically collective even for MAT_ROW_ORIENTED.

[Patrick Lacasse]

Hi,

    I just discovered that the way we assemble our matrix cause a deadlock
in MatSetOption
when petsc is compiled with debugging option enabled.

I resume :
- We loop over elements.
- For each element we create a small elementary matrix.
- We assemble the elementary matrix with MatSetValues into the big one.

Sometimes, we have some rectangular elementary matrix that needs to be
added twice,
the second time it is transposed. The big matrix is not symmetric for many
possible reasons.
We are essentially doing it for every elementary matrix that needs it :

MatSetOption(mat,MAT_ROW_ORIENTED,true)
MatSetValues(...)
MatSetOption(mat,MAT_ROW_ORIENTED,false)
MatSetValues(...)

Of course, it fails (deadlock) in MatSetOption at assertion
PetscValidLogicalCollectiveEnum(mat,op,2);
because different process don't loop over the same (number of) elements.
However, it is working fine without the assertion.

I have two question :
Is there really a risk to call MatSetOption(mat, MAT_ROW_ORIENTED, flg) on
only one process?
What can I do now?

Some possible answer are
- Close our eyes and stop compiling petsc with debugging option enabled.
- We could copy our elementary matrix into a transposed one before to
MatSetValues. We don't want do this cause we think it would slow down our
code.
- Could MatSetValues has a bonus argument row_oriented?
- Could the assertions in MatSetOption be skipped for some options?
- Could we add a function MatSetRowOriented which would not be collective?

thanks,

Patrick Lacasse
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20120920/916984a0/attachment.html>