On Mon, Feb 20, 2012 at 5:14 AM, Bibrak Qamar <span dir="ltr"><<a href="mailto:bibrakc@gmail.com">bibrakc@gmail.com</a>></span> wrote:<br><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr">Hello all,<br><br>The way PETSc MatCreateMPIAIJ distributes an N*N square matrix is row wise. And internally every processor stores its local matrix into two sub matrices one Diagonal and other part is off-Diagonal. (more here --> <a href="http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MatCreateMPIAIJ.html" target="_blank">http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MatCreateMPIAIJ.html</a>).<br>
<br><br>So when PETSc MatMul (v = A.x) is called it basically tries to hide communication of vector (x) by overlapping computation. As for as I understand every process first initiates a kind of non-Blocking Bcast of vector (x) and continues to do MVM of diagonal submatrix and then waits till communication is done and finally does MVM for the off-Diagonal matrix.<br>
<br>My question is this (since I am new) what was the historical reason PETSc opted for this Diagonal and off-Diagonal storage?<br></div></blockquote><div><br></div><div>Overlapping communication and computation.</div><div>
</div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">And what if I want to change the overlapping strategy of MVM by lets say introducing a ring based communication of vector (x), then I have to partition the local matrix into not two sub-matrices but P sub-matrices (here P = number of processors). Does PETSc provide this facility or one has to go from scratch to implement different techniques to store local matrix?<br>
</div></blockquote><div><br></div><div>1) Its unclear what that would accomplish</div><div><br></div><div>2) You can get that style of communication by altering the VecScatter that sends the data, rather than the matrix</div>
<div><br></div><div>3) You can always implement another matrix type. We have a lot (see src/mat/impl)</div><div><br></div><div> Thanks,</div><div><br></div><div> Matt</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr">Thanks<span class="HOEnZb"><font color="#888888"><br clear="all"><span style>Bibrak </span><br>
</font></span></div>
</blockquote></div><br><br clear="all"><div><br></div>-- <br>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>
-- Norbert Wiener<br>