On Tue, Nov 10, 2009 at 6:02 AM, <span dir="ltr"><<a href="mailto:jarunan@ascomp.ch">jarunan@ascomp.ch</a>></span> wrote:<br><div class="gmail_quote"><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
<div class="im">Quoting Jed Brown <jed@59A2.org>:<br>
<br>
<blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
<a href="mailto:jarunan@ascomp.ch" target="_blank">jarunan@ascomp.ch</a> wrote:<br>
<blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
Total number of cells is 744872, divided into 40 blocks. In one<br>
processor, MatCreateMPIAIJWithArrays() takes 0.097 sec but 280 sec with<br>
4 processors. Usually, this routine has no problem with small test case.<br>
It works the same for one or more than one processors.<br>
</blockquote>
<br>
This sounds like incorrect preallocation. Is your PETSc built with<br>
debugging? Debug does some extra integrity checks that don't add<br>
significantly to the time (although other Debug checks do), but it would<br>
be useful to know that they pass. In particular, it checks that your<br>
rows are sorted. If they are not sorted then PETSc's preallocation<br>
would be wrong. (I actually don't think this requirement enables<br>
significantly faster implementation, so I'm tempted to change it to work<br>
correctly with unsorted rows.)<br>
</blockquote>
<br></div>
The code is compiled with the optimized version of PETSc. The row indices and column indices for each row are sorted. Well, they are not sorted for diagonal or off-diagonal part.<br>
</blockquote><div><br>Actually, what Jed says is the likely culprit. Please check that the column indices in each row are sorted. It is clear that the preallocation<br>for multiple procs does not match what you feed to MatSetValues().<br>
<br>Jed: I agree. I would have just written the loop to check for membership in the diagonal block (as I do elsewhere). Maybe we should change petsc-dev?<br><br> Matt<br><br></div><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
<div class="im"><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
<br>
You can also run with -info |grep malloc, there should be no mallocs in<br>
MatSetValues().<br>
<br>
</blockquote>
<br></div>
Here are the output.<br>
The first serie of MatAssemblyEnd_SeqAIJ() should be in MatCreateMPIAIJWithArrays(), which mallocs in MatSetValues() are not zero.<br>
Would MatCreateMPIAIJWithSplitArrays() be better in preallocation?<br>
<br>
[0] VecAssemblyBegin_MPI(): Stash has 0 entries, uses 0 mallocs.<br>
[0] VecAssemblyBegin_MPI(): Block-Stash has 0 entries, uses 0 mallocs.<br>
[2] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.<br>
[1] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.<br>
[3] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.<br>
[0] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.<br>
[3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0<br>
[2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 4379<br>
[1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 13859<br>
[0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 20286<br>
[1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 4592<br>
[3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 15042<br>
[2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 11715<br>
[0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0<br>
[2] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.<br>
[1] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.<br>
[3] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.<br>
[0] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.<br>
[2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0<br>
[3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0<br>
[1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0<br>
[0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0<br>
[2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0<br>
[3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0<br>
[1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0<br>
[0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0<br>
[0] MatIncreaseOverlap_MPIAIJ_Receive(): Allocated 0 bytes, required 3 bytes, no of mallocs = 0<br>
[1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0<br>
[2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0<br>
[3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0<br>
[2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0<br>
[3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0<br>
[1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0<br>
[0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0<br>
[2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0<br>
[3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0<br>
[3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0<br>
[2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0<br>
[0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0<br>
[1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0<br>
[2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0<br>
[3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0<br>
[0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0<br>
[2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0<br>
[1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0<br>
[3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0<br>
[1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0<br>
[0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0<br>
[1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0<br>
[0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0<br>
[0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0<br>
[1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0<br>
[3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0<br>
[1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0<br>
[2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0<br>
[3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0<br>
[2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0<br>
[1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0<br>
[0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0<br>
[3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0<br>
[2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0<br>
[1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0<br>
[3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0<br>
[2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0<br>
[0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0<br>
[0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0<br>
[0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0<br>
<br>
</blockquote></div><br><br clear="all"><br>-- <br>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>-- Norbert Wiener<br>