<div dir="ltr"><div dir="ltr">On Wed, Jan 13, 2021 at 8:26 AM Stefano Zampini <<a href="mailto:stefano.zampini@gmail.com">stefano.zampini@gmail.com</a>> wrote:<br></div><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="auto">MATMPIDENSE does not implement any cyclic distribution. In parallel, a dense matrix is split by rows. Each process owns localrows*globalcols entries. Local sizes are to be intended as the size of the right and left vectors used in matvec operations, and are not strictly related with storage considerations. <a href="https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MatGetLocalSize.html" rel="noreferrer" target="_blank">https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MatGetLocalSize.html</a></div></blockquote><div><br></div><div>To expand on Stefano's point. MATDENSE is intended for very simple matrix handling, and the optimizations are most appropriate</div><div>for blocks of vectors. Sophisticated desne matrix operations are better handled by the MATELEMENTAL type. What you give up here</div><div>is the ability to directly manipulate the storage since it is now distributed in a crazy way. Thus most users actually want MATDENSE.</div><div><br></div><div> Thanks,</div><div><br></div><div> Matt</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div class="gmail_quote"><div dir="ltr" class="gmail_attr">Il Mer 13 Gen 2021, 15:40 Roland Richter <<a href="mailto:roland.richter@ntnu.no" rel="noreferrer" target="_blank">roland.richter@ntnu.no</a>> ha scritto:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div>
<p>Hei,</p>
<p>I am currently struggling a bit with my understanding of local
and global rows/columns in distributed dense matrices in PETSc. <br>
</p>
<p>The attached short program is creating a local matrix in every
thread with [matrix_size_rows, matrix_size_cols] as global size.
Afterwards, I flatten the matrix into a vector, and split this
vector using the boost-function split() when running using several
MPI threads. In addition I create a dense MPI-distributed matrix
based on PETSc, using the same input sizes, and set all values to
zero (after I'm not interested in the content itself, just the
shape). Finally, I retrieve the global and local values for the
number of rows and columns using MatGetLocalSize, and the number
of the first and last row in each matrix using
MatGetOwnershipRange() and print them. According to the
documentation I would expect that the PETSc-based matrix is split
up <br>
</p>
<p>Now, for an input size of [8, 8] and a single thread I get <br>
</p>
<p><i>Rank 0 has a total size of 8, 8<br>
Rank 0 has an initial size of 64<br>
Rank 0 has a local matrix size of 8, 8<br>
Rank 0 has a global matrix size of 8, 8<br>
Rank 0 has a matrix spanning from row 0 to 8<br>
Rank 0 has a vector size of 64</i><i></i></p>
<p>which is what I expect. For two threads, though, I get <br>
</p>
<i>Rank 0 has a total size of 8, 8</i><i><br>
</i><i>Rank 0 has an initial size of 64</i><i><br>
</i><i>Rank 0 has a local matrix size of 4, 4</i><i><br>
</i><i>Rank 0 has a global matrix size of 8, 8</i><i><br>
</i><i>Rank 0 has a matrix spanning from row 0 to 4</i><i><br>
</i><i>Rank 0 has a vector size of 32</i><i><br>
</i><i>Rank 1 has a total size of 8, 8</i><i><br>
</i><i>Rank 1 has an initial size of 64</i><i><br>
</i><i>Rank 1 has a local matrix size of 4, 4</i><i><br>
</i><i>Rank 1 has a global matrix size of 8, 8</i><i><br>
</i><i>Rank 1 has a matrix spanning from row 4 to 8</i><i><br>
Rank 1 has a vector size of 32</i>
<p>Here, most entries make sense, except the size of the local
matrices. Why do I get a size of [4, 4], and not a size of [4, 8]?
Each row should be contiguous in the local process, and therefore
each row should contain all columns, not only a part of it. <br>
</p>
<p>Is there a misunderstanding about how to use MatGetLocalSize(),
or something else?</p>
<p>Thanks!</p>
<p>Regards,</p>
<p>Roland Richter<br>
</p>
<p><i><br>
</i></p>
</div>
</blockquote></div>
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr" class="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>-- Norbert Wiener</div><div><br></div><div><a href="http://www.cse.buffalo.edu/~knepley/" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br></div></div></div></div></div></div></div></div>