<div dir="ltr"><div class="gmail_extra"><div class="gmail_quote">Jed:<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><span class="">
<br>
> I plan to implement MatTransposeMatMult_MPIDense_MPIDense via<br>
><br>
> 1. add MatTransposeMatMult_elemental_elemental()<br>
> 2. C_dense = P_dense^T * B_dense<br>
>     via MatConvert_dense_elemental() and MatConvert_elemental_dense()<br>
<br>
</span>The above involves a ton of data movement and MPIDense is a logical<br>
distribution for matrices with a modest number of columns.  I think I<br>
would just do the local GEMM and then MPI_Reduce_scatter it.<br></blockquote><div>This would work.</div><div><br></div><div>However, I recall that you did a smart ordering which allows MatConvert_mpidense_elemental() uses same physical matrix storage for petsc and elemental, but logically in the layout of elemental. As an example,</div><div>petsc/src/mat/examples/tests/ex103.c:<br></div><div>mpiexec -n 6 ./ex103<br></div></div><div class="gmail_extra"> Outplace MatConvert, A_elemental:</div><div class="gmail_extra">Mat Object: 6 MPI processes</div><div class="gmail_extra">  type: elemental</div><div class="gmail_extra">Elemental matrix (cyclic ordering)</div><div class="gmail_extra">0 0 0 0 0</div><div class="gmail_extra">1 1 1 1 1</div><div class="gmail_extra">2 2 2 2 2</div><div class="gmail_extra">3 3 3 3 3</div><div class="gmail_extra">4 4 4 4 4</div><div class="gmail_extra">5 5 5 5 5</div><div class="gmail_extra">0 0 0 0 0</div><div class="gmail_extra">1 1 1 1 1</div><div class="gmail_extra">2 2 2 2 2</div><div class="gmail_extra">3 3 3 3 3</div><div class="gmail_extra"><br></div><div class="gmail_extra">Elemental matrix (explicit ordering)</div><div class="gmail_extra">  Mat Object:   6 MPI processes</div><div class="gmail_extra">    type: mpidense</div><div class="gmail_extra">0.0000000000000000e+00 0.0000000000000000e+00 0.0000000000000000e+00 0.0000000000000000e+00 0.0000000000000000e+00</div><div class="gmail_extra">0.0000000000000000e+00 0.0000000000000000e+00 0.0000000000000000e+00 0.0000000000000000e+00 0.0000000000000000e+00</div><div class="gmail_extra">1.0000000000000000e+00 1.0000000000000000e+00 1.0000000000000000e+00 1.0000000000000000e+00 1.0000000000000000e+00</div><div class="gmail_extra">1.0000000000000000e+00 1.0000000000000000e+00 1.0000000000000000e+00 1.0000000000000000e+00 1.0000000000000000e+00</div><div class="gmail_extra">2.0000000000000000e+00 2.0000000000000000e+00 2.0000000000000000e+00 2.0000000000000000e+00 2.0000000000000000e+00</div><div class="gmail_extra">2.0000000000000000e+00 2.0000000000000000e+00 2.0000000000000000e+00 2.0000000000000000e+00 2.0000000000000000e+00</div><div class="gmail_extra">3.0000000000000000e+00 3.0000000000000000e+00 3.0000000000000000e+00 3.0000000000000000e+00 3.0000000000000000e+00</div><div class="gmail_extra">3.0000000000000000e+00 3.0000000000000000e+00 3.0000000000000000e+00 3.0000000000000000e+00 3.0000000000000000e+00</div><div class="gmail_extra">4.0000000000000000e+00 4.0000000000000000e+00 4.0000000000000000e+00 4.0000000000000000e+00 4.0000000000000000e+00</div><div class="gmail_extra">5.0000000000000000e+00 5.0000000000000000e+00 5.0000000000000000e+00 5.0000000000000000e+00 5.0000000000000000e+00</div><div class="gmail_extra"><br></div><div class="gmail_extra">i.e., elemental and petsc dense matrices have same ownership.</div><div class="gmail_extra">If there is no data movement for MatConvert(), then it would be easier to use elemental.</div><div class="gmail_extra"><br></div><div class="gmail_extra">Hong</div><div class="gmail_extra"><br></div></div></div>