<div dir="ltr">Hello everyone,<div>Since I already established a baseline to compare the cost of inserting values in PetscMatrix.</div><div>And based on the hint of the number of values inserted in the matrix each time</div><div><span class="gmail-im" style="color:rgb(80,0,80)"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"> 2) Can you tell me how many values are inserted? </blockquote></span></div><div>I took a look at the source code for "MatSetValues_MPIAIJ" and found that it seems to be designed for </div><div>FEM assembly of the global matrix from element matrices(from what I remember from undergrad) since it requires for the setting of multiple rows that the col indices should be the same</div><div><br></div><div>So to increase the number of inserted values I need to modify the implementation of MatSetValues_MPIAIJ to allow for more values to be inserted </div><div>So I made a copy of the function "MatSetValues_MPIAIJ" in "<a title="src/mat/impls/aij/mpi/mpiaij.c" href="https://github.com/petsc/petsc/blob/a8158fb5a3692ffc34efd85261445929d2a1359c/src/mat/impls/aij/mpi/mpiaij.c">src/mat/impls/aij/mpi/mpiaij.c</a>" and named it "MatSetValues2_MPIAIJ"</div><div>I made some minor changes to allow for inserting multiple rows regardless of whether they have the same col indices</div><div><br></div><div>So what I do now is store the data for multiple rows and then insert them all together and I figured I would see how the performance would be.</div><div>I tried different number of rows to be inserted i.e.<b> nrow_buffer = [2, 5, 10, 20, 50, 100]</b></div><div>So now instead of calling "MatSetValues" for every row in the matrix , I call <b>"MatSetValues2_MPIAIJ"</b> every (nrow_buffer)th which should allow for some performance improvement</div><div>the results are as follows</div><div>First Remember that before</div><div><div>1-computation and insertion into petsc matrix</div><span class="gmail-im" style=""><div style=""><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr" style=""><div style=""><span style="color:rgb(204,0,0)">FillPetscMat_with_MatSetValues 100 1.0 </span><font color="#ff0000" style="background-color:rgb(182,215,168)">3.8820e+00 </font><span style="color:rgb(204,0,0)">1.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 23 0 0 0 0 </span><b style="color:rgb(204,0,0)"><span style="background-color:rgb(244,204,204)"> 96</span> </b><span style="color:rgb(204,0,0)"> 0 0 0 0 0</span><font color="#500050"> </font></div></div></blockquote></div></span><div>2-computation and insertion into eigen matrix </div><span class="gmail-im" style="color:rgb(80,0,80)"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">FilEigenMat 100 1.0<span style="background-color:rgb(182,215,168)"> 2.8727e+00 </span>1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 18 0 0 0 0 88 0 0 0 0 0 </blockquote></span></div><div><br></div><div>Now</div><div>nrow_buffer = 2<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><font color="#000000" style=""><span style="background-color:rgb(255,255,255)">FillPetscMat_with_</span><span style="background-color:rgb(255,255,255)">MatSetValues2 100 1.0</span><span style="background-color:rgb(182,215,168)"> </span><span style="background-color:rgb(234,153,153)">3.3321e+00</span><span style="background-color:rgb(255,255,255)"> 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 20 0 0 0 0 95 0 0 0 0 0</span></font></blockquote><div>nrow_buffer = 5<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><span style="color:rgb(0,0,0)">FillPetscMat_with_</span><span style="color:rgb(0,0,0)">MatSetValues2</span> 100 1.0<span style="background-color:rgb(249,203,156)"> 2.8842e+00</span> 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 17 0 0 0 0 94 0 0 0 0 0</blockquote><div>nrow_buffer = 10</div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><span style="color:rgb(0,0,0)">FillPetscMat_with_</span><span style="color:rgb(0,0,0)">MatSetValues2</span> 100 1.0 <span style="background-color:rgb(255,229,153)">2.7669e+00 </span>1.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 17 0 0 0 0 93 0 0 0 0 0</blockquote><div>nrow_buffer = 20<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><span style="color:rgb(0,0,0)">FillPetscMat_with_</span><span style="color:rgb(0,0,0)">MatSetValues2</span> 100 1.0 <span style="background-color:rgb(162,196,201)">2.6834e+00</span> 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 16 0 0 0 0 93 0 0 0 0 0</blockquote><div>nrow_buffer = 50<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><span style="color:rgb(0,0,0)">FillPetscMat_with_</span><span style="color:rgb(0,0,0)">MatSetValues2</span> 100 1.0 <span style="background-color:rgb(162,196,201)">2.6862e+00</span> 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 17 0 0 0 0 93 0 0 0 0 0</blockquote><div>nrow_buffer = 100<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><span style="color:rgb(0,0,0)">FillPetscMat_with_</span><span style="color:rgb(0,0,0)">MatSetValues2</span> 100 1.0<span style="background-color:rgb(180,167,214)"> 2.6170e+00</span> 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 16 0 0 0 0 93 0 0 0 0 0</blockquote><div><br></div><div><span style="background-color:rgb(234,209,220)"><i style="">As to be expected,</i> with increasing the number of rows to be inserted the overhead reduces until it basically stagnates somewhere between 20-50 </span></div><div><span style="background-color:rgb(234,209,220)">The modifications I made based on <b>MatSetValues_MPIAIJ</b> are very small but the effect is significant ( drop in insertion cost by 33%) and it is now even faster than Eigen(baseline) at insertion with my naive usage.</span></div><div><span style="background-color:rgb(234,209,220)">For now I am quite satisfied with the outcome. There is probably some room for improvement but for now this is enough.</span></div><div> </div><div>Thanks,</div><div>Kamra</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Thu, Jul 18, 2019 at 12:34 AM Mohammed Mostafa <<a href="mailto:mo7ammedmostafa@gmail.com">mo7ammedmostafa@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Regarding the first point<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">1) Are you timing only the insertion of values, or computation and insertion? </blockquote><div>I am timing both, the computation and insertion of values but as I said I timed three scenarios</div><div>1-computation only and no insertion</div><div><font color="#674ea7">Computation_no_insertion 100 1.0 1.6747e-01 1.2 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 2 0 0 0 0 22 0 0 0 0 0<br></font></div><div>2-computation and insertion into petsc matrix</div><div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div><span style="color:rgb(204,0,0)">FillPetscMat_with_MatSetValues 100 1.0 3.8820e+00 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 23 0 0 0 0 </span><b style="color:rgb(204,0,0)"><span style="background-color:rgb(244,204,204)"> 96</span> </b><span style="color:rgb(204,0,0)"> 0 0 0 0 0</span> </div></div></blockquote></div><div>3-computation and insertion into eigen matrix </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">FilEigenMat 100 1.0 2.8727e+00 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 18 0 0 0 0 88 0 0 0 0 0 </blockquote><div>I timed 100 times to get a reasonably accurate timings</div><div><br></div><div>as for the second point</div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"> 2) Can you tell me how many values are inserted? </blockquote><div>For a total of nearly 186062 rows per process (with 6 processes in total, the matrix global size is 1116376)</div><div>In most rows ( about 99.35%) 4 non-zeros per rows and in the remaining 0.35% 2 or 3 non-zeros per row</div><div>the number of off-diagonal onnz in total is 648 nnz </div><div>So I insert nearly 4 values 186062 times ~= 744248 times per mpi process</div><div><br></div><div><br></div><div>Thanks,</div><div>Kamra</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, Jul 17, 2019 at 11:59 PM Matthew Knepley <<a href="mailto:knepley@gmail.com" target="_blank">knepley@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div dir="ltr">On Wed, Jul 17, 2019 at 8:51 AM Mohammed Mostafa <<a href="mailto:mo7ammedmostafa@gmail.com" target="_blank">mo7ammedmostafa@gmail.com</a>> wrote:<br></div><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Sorry for the confusion <div>First I fully acknowledge that setting Matrix non-zeros or copying in general is not cheap and memory access pattern can play an important role.</div><div>So to establish a baseline to compare with, I tried setting the same matrix but in an Eigen Sparse Matrix and the timings are as follows</div><div><span style="color:rgb(204,0,0)">FillPetscMat_with_MatSetValues 100 1.0 3.8820e+00 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 23 0 0 0 0 </span><b style="color:rgb(204,0,0)"><span style="background-color:rgb(244,204,204)"> 96</span> </b><span style="color:rgb(204,0,0)"> 0 0 0 0 0</span> </div><div>FilEigenMat 100 1.0 2.8727e+00 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 18 0 0 0 0 88 0 0 0 0 0 </div></div></blockquote><div><br></div><div>Great. This helps. Two things would help me narrow down what is happening.</div><div><br></div><div> 1) Are you timing only the insertion of values, or computation and insertion?</div><div><br></div><div> 2) Can you tell me how many values are inserted?</div><div><br></div><div> Thanks,</div><div><br></div><div> Matt</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>I used the same code but simply filled a different Matrix something like</div><div><br></div><div>for ( i =0; i < nRows;i++)</div><div>{</div><div>//</div><div>.......</div><div>// Some code to get j_index, coefvalues</div><div>// Method1</div><div>MatSetValues(A, 1, &cell_global_index, nnz_per_row, j_index, coefvalues, INSERT_VALUES); </div><div><br></div><div>//Method2</div><div>for ( int k = 0;k < nnz_per_row; k++)</div><div> EigenMat.coeffRef(i, j_index[k] ) = coefvalues[k];</div><div><br></div><div> }<br></div><div>Please note that only one of the two methods is being used at a time. Also, I separately time the code section used to < j_index, coefvalues> but simpling disabling both Method1 and Method2.</div><div>I found the cost to be trivial in comparison to when either one of the methods is used.</div><div>I used Eigen out of convenience since I used for some vector and tensor arithmetics somewhere else in the code and it may not be the best choice.</div><div>Since in PetscMatrix we technically fill two matrices: diagonal and off-diagonal so I expected some difference but is that normal or am I missing something. ?</div><div>Maybe some setting or MatOption I should be using so far this what I have been using</div><div><br></div><div>MatCreateMPIAIJWithArrays(PETSC_COMM_WORLD, local_size, local_size, PETSC_DETERMINE,<br>PETSC_DETERMINE, ptr, j , v, A);<br> MatSetOption(A,MAT_NO_OFF_PROC_ENTRIES,PETSC_TRUE);<br>MatSetOption(A,MAT_IGNORE_OFF_PROC_ENTRIES,PETSC_TRUE);<br>MatSetOption(A,MAT_NEW_NONZERO_ALLOCATION_ERR,PETSC_TRUE);<br>MatSetOption(A,MAT_NEW_NONZERO_LOCATION_ERR,PETSC_TRUE);<br>MatSetOption(A,MAT_NEW_NONZERO_LOCATIONS,PETSC_FALSE);<br>MatSetOption(A,MAT_KEEP_NONZERO_PATTERN,PETSC_TRUE);<br>MatSetUp(A);<br>MatAssemblyBegin(A, MAT_FINAL_ASSEMBLY);<br>MatAssemblyEnd(A, MAT_FINAL_ASSEMBLY); <br></div><div><br></div><div>Thanks,</div><div>Kamra</div></div>
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr" class="gmail-m_-872446422927525695gmail-m_4694747002053173985gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>-- Norbert Wiener</div><div><br></div><div><a href="http://www.cse.buffalo.edu/~knepley/" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br></div></div></div></div></div></div></div></div>
</blockquote></div>
</blockquote></div>