<div class="gmail_quote">On Fri, Jan 20, 2012 at 05:30, <span dir="ltr"><<a href="mailto:S.H.Jongsma@utwente.nl">S.H.Jongsma@utwente.nl</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div lang="NL" link="blue" vlink="purple"><div><p class="MsoNormal"><u></u> <u></u></p><p class="MsoNormal"><span lang="EN-US" style="font-family:"Courier New"">Hello fellow PETSc users,<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-family:"Courier New""><u></u> <u></u></span></p><p class="MsoNormal"><span lang="EN-US" style="font-family:"Courier New"">While using PETSc I came across some conspicuous behavior regarding the use of MatTranspose. When I use this function with a MATSEQBAIJ matrix it performs considerably better than when I use it with a MATMPIBAIJ matrix. I was wondering if anyone of you noticed the same behavior and if this is the case, what did you do about it to circumvent the ‘problem’. In case no one came across this problem before, any suggestions that can help solving this performance issue are welcome. <u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-family:"Courier New""><u></u> <u></u></span></p><p class="MsoNormal"><span lang="EN-US" style="font-family:"Courier New"">To make my problem more clear, I will give some details on my implementation and the test I performed to come to my conclusion. The PETSc version I use is 3.1-p8.</span></p>
</div></div></blockquote><div><br></div><div>Just a friendly reminder to upgrade to petsc-3.2.</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div lang="NL" link="blue" vlink="purple">
<div><p class="MsoNormal"><span lang="EN-US" style="font-family:"Courier New""> I transpose the matrix in place using the following command:<u></u><u></u></span></p><p class="MsoNormal"><span lang="EN-US" style="font-family:"Courier New""><u></u> <u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-family:"Courier New"">MatTranspose(myMatrix, MAT_REUSE_MATRIX, &myMatrix);</span></p></div></div></blockquote><div><br></div><div>The code is not doing correct preallocation here. Can you try AIJ? It still uses a heuristic, but is more likely to be sufficient.</div>
<div><br></div><div>Or, still with MPIBAIJ if you have a square matrix with symmetric nonzero pattern, this out-of-place transpose should perform well.</div><div><br></div><div>MatDuplicate(myMatrix,MAT_DO_NOT_COPY_VALUES,&myMatrixTranspose);</div>
<div>MatTranspose(myMatrix,MAT_REUSE_MATRIX,&myMatrixTranspose);</div><div><br></div><div><br></div><div>Why do you need a transpose? Sparse transpose is a very expensive operation in parallel (even with correct preallocation) and it should always be avoidable (KSPSolveTranspose(), etc).</div>
<div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div lang="NL" link="blue" vlink="purple"><div><p class="MsoNormal"><span lang="EN-US" style="font-family:"Courier New""><u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-family:"Courier New""><u></u> <u></u></span></p><p class="MsoNormal"><span lang="EN-US" style="font-family:"Courier New"">The matrix I used to test the performance is square and consists of 24576 times 24576 blocks of size 5, which means that the matrix has 122880 rows (and columns, of course). The number of non-zero blocks in the matrix is: 291830. Timing the performance gives the following results:<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-family:"Courier New""><u></u> <u></u></span></p><p class="MsoNormal"><span lang="EN-US" style="font-family:"Courier New"">MATSEQBAIJ: 0.320 s<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-family:"Courier New"">MATMPIBAIJ: 1474 s (running on 1 processor) <u></u><u></u></span></p><p class="MsoNormal"><span lang="EN-US" style="font-family:"Courier New"">MATMPIBAIJ: 376 s (running on 2 processors)<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-family:"Courier New""><u></u> <u></u></span></p><p class="MsoNormal"><span lang="EN-US" style="font-family:"Courier New"">As I said, the difference in performance is quite considerable, so any suggestion that can help me solving this issue are greatly appreciated. <u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-family:"Courier New""><u></u> <u></u></span></p><p class="MsoNormal"><span lang="EN-US" style="font-family:"Courier New"">Thank you in advance,<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-family:"Courier New""><u></u> <u></u></span></p><p class="MsoNormal"><span lang="EN-US" style="font-family:"Courier New"">Kind regards,<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-family:"Courier New""><u></u> <u></u></span></p><p class="MsoNormal"><span lang="EN-US" style="font-family:"Courier New""><u></u> <u></u></span></p><p class="MsoNormal">
<span lang="EN-US" style="font-family:"Courier New"">Sietse Jongsma<u></u><u></u></span></p><p class="MsoNormal"><span lang="EN-US" style="font-family:"Courier New""><u></u> <u></u></span></p><p class="MsoNormal">
<span lang="EN-US"><u></u> <u></u></span></p></div></div></blockquote></div><br>