<div dir="ltr"><div dir="ltr"><span class="gmail-im" style="color:rgb(80,0,80)"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">1) Times<br> a) You run the code that generates the matrix entries and simply comment out the calls to MatSetValues() and it takes <br> .16747 seconds.</blockquote></span><div>Yes </div><span class="gmail-im" style="color:rgb(80,0,80)"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"> <br> b) When you include the MatSetValue calls it takes 3.8820. <br></blockquote></span><div>Yes </div><span class="gmail-im" style="color:rgb(80,0,80)"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"> c) When you instead insert into an eigen matrix (which is also parallel?)<br></blockquote></span><div>No, The Eigen Matrix is just a local matrix to each process and it doesn't support MPI ( at least how I use it ), I just used to establish a baseline for the insertion cost in CSR matrix ( since I don't set off-processor rows anyway)</div><div>And it both cases it is a local insertion until MatAssembly where communication and inter-processor copy should happen</div><div>After I insert in Eigen and I copy again the Eigen matrix to the Petsc Matrix, using <i><b>MatUpdateMPIAIJWithArrays() </b>which costs me an additional <font color="#ff0000" style="background-color:rgb(234,209,220)">0</font></i><font color="#ff0000" style="background-color:rgb(234,209,220)">.20326 sec</font><font color="#ea9999">s</font>.<i><b>,</b></i> which is what I intend to use for linear system solution using ksp, pc</div><span class="gmail-im" style="color:rgb(80,0,80)"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"> it takes 2.8727 secs.<br> d) When you in your code put all the values into a CSR matrix and then call MatUpdateMPIAIJWithArrays() it takes<br> 2.8727 secs + .20326 secs.<br></blockquote></span><div>Yes </div><span class="gmail-im" style="color:rgb(80,0,80)"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"> e) When you modify MatSetValues_MPIAIJ into MatSetValues_MPIAIJ2() that allows multiple rows that do not need matching <br> columns it takes 2.6170 secs.</blockquote></span><div>Yes </div><span class="gmail-im" style="color:rgb(80,0,80)"><div>1e) When you use MatSetValues_MPIAIJ2() are you directly calling MatSetValues_MPIAIJ2() or are you still calling MatSetValues() and<br> having it call MatSetValues_MPIAIJ2()? That is did you replace the setvalues() pointer in the function table? <br></div></span><div>Yes, I am calling <b>MatSetValues2_MPIAIJ()</b> directly not <b>MatSetValues()</b>, I just wanted to confirm the behavior first and if I am getting any performance gain</div><div>I wanted to be as minimally intrusive to the code as possible right now to avoid causing problems elsewhere since I have just started learning how it works.</div><span class="gmail-im" style="color:rgb(80,0,80)"><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">1*) For the cases b,c, and e are calling MatSetValues*() multiple times for the same row or exactly once for a given row?</blockquote></span><div>I am only calling MatSetValues exactly once per row and each time I set 4 nonzero values although for the future I may set as many as 7 non-zero values</div><div>however, in <b>MatSetValues2_MPIAIJ() I call it once every </b>nrow_buffer</div><div>The call is like this</div><div><pre style="white-space:pre-wrap;color:rgb(0,0,0);background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial"> PetscLogEventBegin<span style="color:rgb(128,128,48)">(</span>FillPetscMatEvent,<span style="font-family:Arial,Helvetica,sans-serif"> </span><span style="font-family:Arial,Helvetica,sans-serif;color:rgb(0,140,0)">0</span><span style="font-family:Arial,Helvetica,sans-serif;color:rgb(128,128,48)">,</span><span style="font-family:Arial,Helvetica,sans-serif"> </span><span style="font-family:Arial,Helvetica,sans-serif;color:rgb(0,140,0)">0</span><span style="font-family:Arial,Helvetica,sans-serif;color:rgb(128,128,48)">,</span><span style="font-family:Arial,Helvetica,sans-serif"> </span><span style="font-family:Arial,Helvetica,sans-serif;color:rgb(0,140,0)">0</span><span style="font-family:Arial,Helvetica,sans-serif;color:rgb(128,128,48)">,</span><span style="font-family:Arial,Helvetica,sans-serif"> </span><span style="font-family:Arial,Helvetica,sans-serif;color:rgb(0,140,0)">0</span><span style="font-family:Arial,Helvetica,sans-serif;color:rgb(128,128,48)">)</span><span style="font-family:Arial,Helvetica,sans-serif;color:rgb(128,0,128)">;</span><br> <span style="color:rgb(128,0,0);font-weight:bold">auto</span> mat <span style="color:rgb(128,128,48)">=</span> <span style="color:rgb(128,128,48)">&</span>PhiEqnSolver<span style="color:rgb(128,128,48)">.</span>mat<span style="color:rgb(128,0,128)">;</span>
<span style="color:rgb(128,0,0);font-weight:bold">int</span> buffer_size <span style="color:rgb(128,128,48)">=</span> <span style="color:rgb(0,140,0)">100</span><span style="color:rgb(128,128,48)">,</span> n_inner<span style="color:rgb(128,128,48)">,</span> Rows_left<span style="color:rgb(128,128,48)">,</span>nj_per_row_i<span style="color:rgb(128,0,128)">;</span>
<span style="color:rgb(128,0,0);font-weight:bold">int</span> n_outer <span style="color:rgb(128,128,48)">=</span> <span style="color:rgb(128,0,0);font-weight:bold">int</span><span style="color:rgb(128,128,48)">(</span><span style="color:rgb(96,48,0)">ceil</span><span style="color:rgb(128,128,48)">(</span><span style="color:rgb(128,0,0);font-weight:bold">double</span><span style="color:rgb(128,128,48)">(</span>nCells<span style="color:rgb(128,128,48)">)</span> <span style="color:rgb(128,128,48)">/</span> <span style="color:rgb(128,0,0);font-weight:bold">double</span><span style="color:rgb(128,128,48)">(</span>buffer_size<span style="color:rgb(128,128,48)">)</span><span style="color:rgb(128,128,48)">)</span><span style="color:rgb(128,128,48)">)</span><span style="color:rgb(128,0,128)">;</span>
cell <span style="color:rgb(128,128,48)">=</span> <span style="color:rgb(0,140,0)">0</span><span style="color:rgb(128,0,128)">;</span>
Rows_left <span style="color:rgb(128,128,48)">=</span> nCells<span style="color:rgb(128,0,128)">;</span>
<span style="color:rgb(128,0,0);font-weight:bold">for</span> <span style="color:rgb(128,128,48)">(</span><span style="color:rgb(128,0,0);font-weight:bold">int</span> i_outer <span style="color:rgb(128,128,48)">=</span> <span style="color:rgb(0,140,0)">0</span><span style="color:rgb(128,0,128)">;</span> i_outer <span style="color:rgb(128,128,48)"><</span> n_outer<span style="color:rgb(128,0,128)">;</span> i_outer<span style="color:rgb(128,128,48)">+</span><span style="color:rgb(128,128,48)">+</span><span style="color:rgb(128,128,48)">)</span> <span style="color:rgb(128,0,128)">{</span>
n_inner <span style="color:rgb(128,128,48)">=</span> <span style="color:rgb(96,48,0)">min</span><span style="color:rgb(128,128,48)">(</span>buffer_size<span style="color:rgb(128,128,48)">,</span>Rows_left <span style="color:rgb(128,128,48)">)</span><span style="color:rgb(128,0,128)">;</span>
k <span style="color:rgb(128,128,48)">=</span> <span style="color:rgb(0,140,0)">0</span><span style="color:rgb(128,0,128)">;</span>
<span style="color:rgb(128,0,0);font-weight:bold">for</span> <span style="color:rgb(128,128,48)">(</span>i <span style="color:rgb(128,128,48)">=</span> <span style="color:rgb(0,140,0)">0</span><span style="color:rgb(128,0,128)">;</span> i <span style="color:rgb(128,128,48)"><</span> n_inner<span style="color:rgb(128,0,128)">;</span> i<span style="color:rgb(128,128,48)">+</span><span style="color:rgb(128,128,48)">+</span><span style="color:rgb(128,128,48)">)</span> <span style="color:rgb(128,0,128)">{</span>
cell_gindex <span style="color:rgb(128,128,48)">=</span> cellGIndex<span style="color:rgb(128,128,48)">[</span>cell<span style="color:rgb(128,128,48)">]</span><span style="color:rgb(128,0,128)">;</span>
i_index<span style="color:rgb(128,128,48)">[</span>i<span style="color:rgb(128,128,48)">]</span> <span style="color:rgb(128,128,48)">=</span> cell_gindex<span style="color:rgb(128,0,128)">;</span>
<span style="color:rgb(105,105,105)">// set the system RHS = f * vol</span>
Eqn_rhs_ptr<span style="color:rgb(128,128,48)">[</span>cell<span style="color:rgb(128,128,48)">]</span> <span style="color:rgb(128,128,48)">+</span><span style="color:rgb(128,128,48)">=</span> cellVol<span style="color:rgb(128,128,48)">[</span>cell<span style="color:rgb(128,128,48)">]</span> <span style="color:rgb(128,128,48)">*</span> rhsFieldptr<span style="color:rgb(128,128,48)">[</span>cell<span style="color:rgb(128,128,48)">]</span><span style="color:rgb(128,0,128)">;</span>
istart <span style="color:rgb(128,128,48)">=</span> cellIFace<span style="color:rgb(128,128,48)">[</span>cell<span style="color:rgb(128,128,48)">]</span><span style="color:rgb(128,0,128)">;</span>
iend <span style="color:rgb(128,128,48)">=</span> cellIFace<span style="color:rgb(128,128,48)">[</span>cell <span style="color:rgb(128,128,48)">+</span> <span style="color:rgb(0,140,0)">1</span><span style="color:rgb(128,128,48)">]</span><span style="color:rgb(128,0,128)">;</span>
diagCOEF <span style="color:rgb(128,128,48)">=</span> <span style="color:rgb(0,128,0)">0.0</span><span style="color:rgb(128,0,128)">;</span>
nj_per_row_i <span style="color:rgb(128,128,48)">=</span><span style="color:rgb(0,140,0)">0</span><span style="color:rgb(128,0,128)">;</span>
<span style="color:rgb(128,0,0);font-weight:bold">for</span> <span style="color:rgb(128,128,48)">(</span>fptr <span style="color:rgb(128,128,48)">=</span> istart<span style="color:rgb(128,0,128)">;</span> fptr <span style="color:rgb(128,128,48)"><</span> iend<span style="color:rgb(128,0,128)">;</span> fptr<span style="color:rgb(128,128,48)">+</span><span style="color:rgb(128,128,48)">+</span><span style="color:rgb(128,128,48)">)</span> <span style="color:rgb(128,0,128)">{</span>
face <span style="color:rgb(128,128,48)">=</span> cellFace<span style="color:rgb(128,128,48)">[</span>fptr<span style="color:rgb(128,128,48)">]</span><span style="color:rgb(128,0,128)">;</span>
nghbr <span style="color:rgb(128,128,48)">=</span> faceNeighbor<span style="color:rgb(128,128,48)">[</span>face<span style="color:rgb(128,128,48)">]</span><span style="color:rgb(128,0,128)">;</span>
COEF <span style="color:rgb(128,128,48)">=</span> Coef<span style="color:rgb(128,128,48)">[</span>face<span style="color:rgb(128,128,48)">]</span><span style="color:rgb(128,0,128)">; // flux coef computed on the faces of each cell in the domain</span>
<span style="color:rgb(128,0,0);font-weight:bold">if</span> <span style="color:rgb(128,128,48)">(</span>nghbr <span style="color:rgb(128,128,48)"><</span> <span style="color:rgb(0,140,0)">0</span><span style="color:rgb(128,128,48)">)</span> <span style="color:rgb(128,0,128)">{</span>
bc_type <span style="color:rgb(128,128,48)">=</span> faceBCType<span style="color:rgb(128,128,48)">[</span>face<span style="color:rgb(128,128,48)">]</span><span style="color:rgb(128,0,128)">;</span>
<span style="color:rgb(128,0,0);font-weight:bold">if</span> <span style="color:rgb(128,128,48)">(</span>bc_type <span style="color:rgb(128,128,48)">=</span><span style="color:rgb(128,128,48)">=</span> PROCESSOR<span style="color:rgb(128,128,48)">)</span> <span style="color:rgb(128,0,128)">{</span>
neighbor_cell <span style="color:rgb(128,128,48)">=</span> facePhysTag<span style="color:rgb(128,128,48)">[</span>face<span style="color:rgb(128,128,48)">]</span><span style="color:rgb(128,0,128)">;</span>
j_index<span style="color:rgb(128,128,48)">[</span>k<span style="color:rgb(128,128,48)">]</span> <span style="color:rgb(128,128,48)">=</span> neighbor_cell<span style="color:rgb(128,0,128)">;</span>
coefvalues<span style="color:rgb(128,128,48)">[</span>k<span style="color:rgb(128,128,48)">]</span> <span style="color:rgb(128,128,48)">=</span> COEF<span style="color:rgb(128,0,128)">;</span>
diagCOEF <span style="color:rgb(128,128,48)">-</span><span style="color:rgb(128,128,48)">=</span> COEF<span style="color:rgb(128,0,128)">;</span>
k<span style="color:rgb(128,128,48)">+</span><span style="color:rgb(128,128,48)">+</span><span style="color:rgb(128,0,128)">;</span>
nj_per_row_i<span style="color:rgb(128,128,48)">+</span><span style="color:rgb(128,128,48)">+</span><span style="color:rgb(128,0,128)">;</span>
<span style="color:rgb(128,0,128)">}</span> <span style="color:rgb(128,0,0);font-weight:bold">else</span> <span style="color:rgb(128,0,128)">{</span>
diagCOEF <span style="color:rgb(128,128,48)">-</span><span style="color:rgb(128,128,48)">=</span> COEF<span style="color:rgb(128,0,128)">;</span>
<span style="color:rgb(128,0,128)">}</span>
<span style="color:rgb(128,0,128)">}</span> <span style="color:rgb(128,0,0);font-weight:bold">else</span> <span style="color:rgb(128,0,128)">{</span>
owner <span style="color:rgb(128,128,48)">=</span> faceOwner<span style="color:rgb(128,128,48)">[</span>face<span style="color:rgb(128,128,48)">]</span><span style="color:rgb(128,0,128)">;</span>
neighbor_cell <span style="color:rgb(128,128,48)">=</span> owner <span style="color:rgb(128,128,48)">+</span> nghbr <span style="color:rgb(128,128,48)">-</span> cell<span style="color:rgb(128,0,128)">;</span>
j_index<span style="color:rgb(128,128,48)">[</span>k<span style="color:rgb(128,128,48)">]</span> <span style="color:rgb(128,128,48)">=</span> cellGIndex<span style="color:rgb(128,128,48)">[</span>neighbor_cell<span style="color:rgb(128,128,48)">]</span><span style="color:rgb(128,0,128)">;</span>
coefvalues<span style="color:rgb(128,128,48)">[</span>k<span style="color:rgb(128,128,48)">]</span> <span style="color:rgb(128,128,48)">=</span> COEF<span style="color:rgb(128,0,128)">;</span>
diagCOEF <span style="color:rgb(128,128,48)">-</span><span style="color:rgb(128,128,48)">=</span> COEF<span style="color:rgb(128,0,128)">;</span>
k<span style="color:rgb(128,128,48)">+</span><span style="color:rgb(128,128,48)">+</span><span style="color:rgb(128,0,128)">;</span>
nj_per_row_i<span style="color:rgb(128,128,48)">+</span><span style="color:rgb(128,128,48)">+</span><span style="color:rgb(128,0,128)">;</span>
<span style="color:rgb(128,0,128)">}</span>
<span style="color:rgb(128,0,128)">}</span>
j_index<span style="color:rgb(128,128,48)">[</span>k<span style="color:rgb(128,128,48)">]</span> <span style="color:rgb(128,128,48)">=</span> cell_gindex<span style="color:rgb(128,0,128)">;</span>
coefvalues<span style="color:rgb(128,128,48)">[</span>k<span style="color:rgb(128,128,48)">]</span> <span style="color:rgb(128,128,48)">=</span> diagCOEF<span style="color:rgb(128,0,128)">;</span>
k<span style="color:rgb(128,128,48)">+</span><span style="color:rgb(128,128,48)">+</span><span style="color:rgb(128,0,128)">;</span>
nj_per_row_i<span style="color:rgb(128,128,48)">+</span><span style="color:rgb(128,128,48)">+</span><span style="color:rgb(128,0,128)">;</span>
nj_perRow<span style="color:rgb(128,128,48)">[</span>i<span style="color:rgb(128,128,48)">]</span> <span style="color:rgb(128,128,48)">=</span> nj_per_row_i<span style="color:rgb(128,0,128)">;</span>
cell<span style="color:rgb(128,128,48)">+</span><span style="color:rgb(128,128,48)">+</span><span style="color:rgb(128,0,128)">;</span>
Rows_left<span style="color:rgb(128,128,48)">-</span><span style="color:rgb(128,128,48)">-</span><span style="color:rgb(128,0,128)">;</span>
<span style="color:rgb(128,0,128)">}</span>
MatSetValues2_MPIAIJ<span style="color:rgb(128,128,48)">(</span><span style="color:rgb(128,128,48)">*</span>A<span style="color:rgb(128,128,48)">,</span> n_inner<span style="color:rgb(128,128,48)">,</span> i_index<span style="color:rgb(128,128,48)">,</span> nj_perRow<span style="color:rgb(128,128,48)">,</span> j_index<span style="color:rgb(128,128,48)">,</span> coefvalues<span style="color:rgb(128,128,48)">,</span> INSERT_VALUES<span style="color:rgb(128,128,48)">)</span><span style="color:rgb(128,0,128)">;</span>
<span style="color:rgb(128,0,128)">}</span>
PetscLogEventEnd<span style="color:rgb(128,128,48)">(</span>FillPetscMatEvent,<span style="font-family:Arial,Helvetica,sans-serif"> </span><span style="font-family:Arial,Helvetica,sans-serif;color:rgb(0,140,0)">0</span><span style="font-family:Arial,Helvetica,sans-serif;color:rgb(128,128,48)">,</span><span style="font-family:Arial,Helvetica,sans-serif"> </span><span style="font-family:Arial,Helvetica,sans-serif;color:rgb(0,140,0)">0</span><span style="font-family:Arial,Helvetica,sans-serif;color:rgb(128,128,48)">,</span><span style="font-family:Arial,Helvetica,sans-serif"> </span><span style="font-family:Arial,Helvetica,sans-serif;color:rgb(0,140,0)">0</span><span style="font-family:Arial,Helvetica,sans-serif;color:rgb(128,128,48)">,</span><span style="font-family:Arial,Helvetica,sans-serif"> </span><span style="font-family:Arial,Helvetica,sans-serif;color:rgb(0,140,0)">0</span><span style="font-family:Arial,Helvetica,sans-serif;color:rgb(128,128,48)">)</span><span style="font-family:Arial,Helvetica,sans-serif;color:rgb(128,0,128)">;</span><br></pre></div><span class="gmail-im" style="color:rgb(80,0,80)"><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">2) You have done the initial matrix preallocation with MatCreateMPIAIJWithArrays(); and so have perfect preallocation, that is when you run with<br> -info and grep for malloc it says 0 mallocs are performed?</blockquote></span><div>Yes no mallocs and no unused non-zeros</div><div>check the attached file <info_log_petsc_setValues.txt></div><span class="gmail-im" style="color:rgb(80,0,80)"><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"> 3) Are you using ADD_VALUES or INSERT_VALUES?</blockquote><div><br></div></span><div>here are the times for nrow_buffer = 50</div><div>With InsertValues</div><div>FillCSTMat_with_MatSetValues 100 1.0 <span style="background-color:rgb(182,215,168)">2.5512e+00 </span>1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 16 0 0 0 0 93 0 0 0 0 <span style="background-color:rgb(182,215,168)"> 0</span><br></div><div>WithAddValues</div><div>FillPetscMat 100 1.0 <span style="background-color:rgb(208,224,227)">2.6654e+00</span> 1.0 7.44e+07 1.0 0.0e+00 0.0e+00 0.0e+00 17 53 0 0 0 90100 0 0 0 <span style="background-color:rgb(213,166,189)">167</span><br></div><div>Additionally I have to zero the matrix which costs</div><div>MatZeroEntries 100 1.0 <span style="background-color:rgb(208,224,227)">9.5259e-02</span> 1.2 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 1 0 0 0 0 3 0 0 0 0 0<br></div><div> INSERT_VALUES is abit faster than ADD_VALUES the difference I saw can be attributed to the cost computation ( addition)<br></div><span class="gmail-im" style="color:rgb(80,0,80)"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">4) In each MatSetValues() call are you setting values into all nonzero locations of that row, or only a subset of nonzero locations in that row? <br> Are the column indices for that row you pass in monotonically increasing or not?</blockquote></span><div>Yes I setting new values for all non-zero values in that row, not a subset of it </div><div>No, the column indices are Not monotonically increasing as you can see from the above code the column indices come from the mesh connectivity come.</div><div>I suppose I can sort them and make line an ordering map for each row and store it and use that to rearrange the values before inserting them. </div><div>But would it really matter that much ?! From memory access point, I suppose it could matter.</div><span class="gmail-im" style="color:rgb(80,0,80)"><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">5) Did you ./configure with --with-debugging=0?</blockquote></span><div>PETSC_ARCH=release3 <b>-with-debugging=0</b> COPTFLAGS="-O3 -march=native -mtune=native" CXXOPTFLAGS="-O3 -march=native -mtune=native" FOPTFLAGS="-O3 -march=native -mtune=native" --with-cc=mpicc --with-cxx=mpicxx --with-fc=mpif90 --download-metis --download-hypre<br></div><div><br></div><div>As for the second Message </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">1) Could you please send nrow_buffer = 1 <br></blockquote><div>Yes, Of course.</div><div>As I said I made very small changes and now I trying to dig deeper into the code to see if there is more to do </div><div> Should I attach the modified < mpiaij.c and petscmat.h > to this message thread or send it directly to you </div><span class="gmail-im" style="color:rgb(80,0,80)"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">2) What time do you get if you use MatSetValues2_MPIAIJ() with an nrow_buffer = 1 ?</blockquote></span><div>Honestly, I didn't try that before.</div><div>First I replaced the call in the code above to<b> MatSetValues() </b>and set nrow_buffer = 1 so it should be equivalent to before and here is the time </div><div>FillPetscMat 100 1.0 <b style="background-color:rgb(255,242,204)">3.6594e+00</b> 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 22 0 0 0 0 96 0 0 0 0 0<br></div><div>Now I used <b>MatSetValues2_MPIAIJ()</b> with an nrow_buffer = 1</div><div>FillPetscMat 100 1.0<b style="background-color:rgb(217,234,211)"> 3.3920e+00</b> 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 21 0 0 0 0 96 0 0 0 0 0<br></div><div><br></div><div>For some reason, <b>MatSetValues2_MPIAIJ </b>is a bit faster.</div><div>Sorry for the long reply.</div><div><br></div><div>Thanks,</div><div>Kamra</div></div><div class="gmail-yj6qo gmail-ajU" style="outline:none;padding:10px 0px;width:22px;margin:2px 0px 0px"><br class="gmail-Apple-interchange-newline"></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, Jul 19, 2019 at 9:48 AM Smith, Barry F. <<a href="mailto:bsmith@mcs.anl.gov">bsmith@mcs.anl.gov</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"> <br>
1) Could you please send MatSetValues2_MPIAIJ()?<br>
<br>
2) What time do you get if you use MatSetValues2_MPIAIJ() with an nrow_buffer = 1 ?<br>
<br>
Thanks<br>
<br>
Barry<br>
<br>
<br>
> On Jul 18, 2019, at 2:01 AM, Mohammed Mostafa via petsc-users <<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a>> wrote:<br>
> <br>
> Hello everyone,<br>
> Since I already established a baseline to compare the cost of inserting values in PetscMatrix.<br>
> And based on the hint of the number of values inserted in the matrix each time<br>
> 2) Can you tell me how many values are inserted? <br>
> I took a look at the source code for "MatSetValues_MPIAIJ" and found that it seems to be designed for <br>
> FEM assembly of the global matrix from element matrices(from what I remember from undergrad) since it requires for the setting of multiple rows that the col indices should be the same<br>
> <br>
> So to increase the number of inserted values I need to modify the implementation of MatSetValues_MPIAIJ to allow for more values to be inserted <br>
> So I made a copy of the function "MatSetValues_MPIAIJ" in "src/mat/impls/aij/mpi/mpiaij.c" and named it "MatSetValues2_MPIAIJ"<br>
> I made some minor changes to allow for inserting multiple rows regardless of whether they have the same col indices<br>
> <br>
> So what I do now is store the data for multiple rows and then insert them all together and I figured I would see how the performance would be.<br>
> I tried different number of rows to be inserted i.e. nrow_buffer = [2, 5, 10, 20, 50, 100]<br>
> So now instead of calling "MatSetValues" for every row in the matrix , I call "MatSetValues2_MPIAIJ" every (nrow_buffer)th which should allow for some performance improvement<br>
> the results are as follows<br>
> First Remember that before<br>
> 1-computation and insertion into petsc matrix<br>
> FillPetscMat_with_MatSetValues 100 1.0 3.8820e+00 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 23 0 0 0 0 96 0 0 0 0 0 <br>
> 2-computation and insertion into eigen matrix <br>
> FilEigenMat 100 1.0 2.8727e+00 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 18 0 0 0 0 88 0 0 0 0 0 <br>
> <br>
> Now<br>
> nrow_buffer = 2<br>
> FillPetscMat_with_MatSetValues2 100 1.0 3.3321e+00 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 20 0 0 0 0 95 0 0 0 0 0<br>
> nrow_buffer = 5<br>
> FillPetscMat_with_MatSetValues2 100 1.0 2.8842e+00 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 17 0 0 0 0 94 0 0 0 0 0<br>
> nrow_buffer = 10<br>
> FillPetscMat_with_MatSetValues2 100 1.0 2.7669e+00 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 17 0 0 0 0 93 0 0 0 0 0<br>
> nrow_buffer = 20<br>
> FillPetscMat_with_MatSetValues2 100 1.0 2.6834e+00 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 16 0 0 0 0 93 0 0 0 0 0<br>
> nrow_buffer = 50<br>
> FillPetscMat_with_MatSetValues2 100 1.0 2.6862e+00 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 17 0 0 0 0 93 0 0 0 0 0<br>
> nrow_buffer = 100<br>
> FillPetscMat_with_MatSetValues2 100 1.0 2.6170e+00 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 16 0 0 0 0 93 0 0 0 0 0<br>
> <br>
> As to be expected, with increasing the number of rows to be inserted the overhead reduces until it basically stagnates somewhere between 20-50 <br>
> The modifications I made based on MatSetValues_MPIAIJ are very small but the effect is significant ( drop in insertion cost by 33%) and it is now even faster than Eigen(baseline) at insertion with my naive usage.<br>
> For now I am quite satisfied with the outcome. There is probably some room for improvement but for now this is enough.<br>
> <br>
> Thanks,<br>
> Kamra<br>
> <br>
> On Thu, Jul 18, 2019 at 12:34 AM Mohammed Mostafa <<a href="mailto:mo7ammedmostafa@gmail.com" target="_blank">mo7ammedmostafa@gmail.com</a>> wrote:<br>
> Regarding the first point<br>
> 1) Are you timing only the insertion of values, or computation and insertion? <br>
> I am timing both, the computation and insertion of values but as I said I timed three scenarios<br>
> 1-computation only and no insertion<br>
> Computation_no_insertion 100 1.0 1.6747e-01 1.2 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 2 0 0 0 0 22 0 0 0 0 0<br>
> 2-computation and insertion into petsc matrix<br>
> FillPetscMat_with_MatSetValues 100 1.0 3.8820e+00 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 23 0 0 0 0 96 0 0 0 0 0 <br>
> 3-computation and insertion into eigen matrix <br>
> FilEigenMat 100 1.0 2.8727e+00 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 18 0 0 0 0 88 0 0 0 0 0 <br>
> I timed 100 times to get a reasonably accurate timings<br>
> <br>
> as for the second point<br>
> 2) Can you tell me how many values are inserted? <br>
> For a total of nearly 186062 rows per process (with 6 processes in total, the matrix global size is 1116376)<br>
> In most rows ( about 99.35%) 4 non-zeros per rows and in the remaining 0.35% 2 or 3 non-zeros per row<br>
> the number of off-diagonal onnz in total is 648 nnz <br>
> So I insert nearly 4 values 186062 times ~= 744248 times per mpi process<br>
> <br>
> <br>
> Thanks,<br>
> Kamra<br>
> <br>
> On Wed, Jul 17, 2019 at 11:59 PM Matthew Knepley <<a href="mailto:knepley@gmail.com" target="_blank">knepley@gmail.com</a>> wrote:<br>
> On Wed, Jul 17, 2019 at 8:51 AM Mohammed Mostafa <<a href="mailto:mo7ammedmostafa@gmail.com" target="_blank">mo7ammedmostafa@gmail.com</a>> wrote:<br>
> Sorry for the confusion <br>
> First I fully acknowledge that setting Matrix non-zeros or copying in general is not cheap and memory access pattern can play an important role.<br>
> So to establish a baseline to compare with, I tried setting the same matrix but in an Eigen Sparse Matrix and the timings are as follows<br>
> FillPetscMat_with_MatSetValues 100 1.0 3.8820e+00 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 23 0 0 0 0 96 0 0 0 0 0 <br>
> FilEigenMat 100 1.0 2.8727e+00 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 18 0 0 0 0 88 0 0 0 0 0 <br>
> <br>
> Great. This helps. Two things would help me narrow down what is happening.<br>
> <br>
> 1) Are you timing only the insertion of values, or computation and insertion?<br>
> <br>
> 2) Can you tell me how many values are inserted?<br>
> <br>
> Thanks,<br>
> <br>
> Matt<br>
> <br>
> I used the same code but simply filled a different Matrix something like<br>
> <br>
> for ( i =0; i < nRows;i++)<br>
> {<br>
> //<br>
> .......<br>
> // Some code to get j_index, coefvalues<br>
> // Method1<br>
> MatSetValues(A, 1, &cell_global_index, nnz_per_row, j_index, coefvalues, INSERT_VALUES); <br>
> <br>
> //Method2<br>
> for ( int k = 0;k < nnz_per_row; k++)<br>
> EigenMat.coeffRef(i, j_index[k] ) = coefvalues[k];<br>
> <br>
> }<br>
> Please note that only one of the two methods is being used at a time. Also, I separately time the code section used to < j_index, coefvalues> but simpling disabling both Method1 and Method2.<br>
> I found the cost to be trivial in comparison to when either one of the methods is used.<br>
> I used Eigen out of convenience since I used for some vector and tensor arithmetics somewhere else in the code and it may not be the best choice.<br>
> Since in PetscMatrix we technically fill two matrices: diagonal and off-diagonal so I expected some difference but is that normal or am I missing something. ?<br>
> Maybe some setting or MatOption I should be using so far this what I have been using<br>
> <br>
> MatCreateMPIAIJWithArrays(PETSC_COMM_WORLD, local_size, local_size, PETSC_DETERMINE,<br>
> PETSC_DETERMINE, ptr, j , v, A);<br>
> MatSetOption(A,MAT_NO_OFF_PROC_ENTRIES,PETSC_TRUE);<br>
> MatSetOption(A,MAT_IGNORE_OFF_PROC_ENTRIES,PETSC_TRUE);<br>
> MatSetOption(A,MAT_NEW_NONZERO_ALLOCATION_ERR,PETSC_TRUE);<br>
> MatSetOption(A,MAT_NEW_NONZERO_LOCATION_ERR,PETSC_TRUE);<br>
> MatSetOption(A,MAT_NEW_NONZERO_LOCATIONS,PETSC_FALSE);<br>
> MatSetOption(A,MAT_KEEP_NONZERO_PATTERN,PETSC_TRUE);<br>
> MatSetUp(A);<br>
> MatAssemblyBegin(A, MAT_FINAL_ASSEMBLY);<br>
> MatAssemblyEnd(A, MAT_FINAL_ASSEMBLY); <br>
> <br>
> Thanks,<br>
> Kamra<br>
> <br>
> <br>
> -- <br>
> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>
> -- Norbert Wiener<br>
> <br>
> <a href="https://www.cse.buffalo.edu/~knepley/" rel="noreferrer" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br>
<br>
</blockquote></div>