<div><div dir="auto">This is a 2D case on triangular mesh so 3+1</div></div><div dir="auto">For quad it becomes 4+1 and in 3D more and more nnz depending on the cell type</div><div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Sun, Jul 14, 2019 at 3:55 AM Mark Adams <<a href="mailto:mfadams@lbl.gov">mfadams@lbl.gov</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div dir="ltr"><br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Sat, Jul 13, 2019 at 2:39 PM Mohammed Mostafa via petsc-users <<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div><div dir="auto">I am generating the matrix using the finite volume method </div></div><div dir="auto">I basically loop over the face list instead of looping over the cells to avoid double evaluation of the fluxes of cell faces</div><div dir="auto">So I figured I would store the coefficients in a temp container ( in this case a csr sparse matrix) and then loop over rows to set in the petsc matrix </div><div dir="auto">I know it looks like a waste of memory and copy overhead but for now I can’t think of a better way. </div></blockquote><div><br></div><div>You can use PETSc's setValues, just use ADD_VALUES instead of INSERT. Our matrix class is pretty much the same as your "temp container" so I would use PETSc directly.</div><div><br></div><div>And I see "<span style="color:rgb(0,0,0);font-family:"Courier New",Courier,monospace,arial,sans-serif;font-size:14px;white-space:pre-wrap">Maximum nonzeros in any row is 4</span>" and "<span style="color:rgb(0,0,0);font-family:"Courier New",Courier,monospace,arial,sans-serif;font-size:14px;white-space:pre-wrap">Maximum nonzeros in any row is 1</span>". I guess the "1"s are from the off diagonal block matrix, but should the 4 be 2*D + 1 ?</div></div></div><div dir="ltr"><div class="gmail_quote"><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="auto">For now I will try the two routines from the master branch </div><div dir="auto">MatCreateMPIAIJWithArrays() MatUpdateMPIAIJWithArrays()</div><div dir="auto">and send the logs</div><div dir="auto"><br></div><div><div dir="auto">Thanks </div><div dir="auto">Kamra </div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Sun, Jul 14, 2019 at 1:51 AM Smith, Barry F. <<a href="mailto:bsmith@mcs.anl.gov" target="_blank">bsmith@mcs.anl.gov</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><br>
How are you generating entries in your matrix? Finite differences, finite element, finite volume, something else?<br>
<br>
If you are using finite differences you generally generate an entire row at a time and call MatSetValues() once per row. With finite elements you generates an element at a time and ADD_VALUES for a block of rows and columns. <br>
<br>
I don't know why generating directly in CSR format would be faster than than calling MatSetValues() once per row but anyways if you have the matrix in CSR format you can use <br>
<br>
MatCreateMPIAIJWithArrays() (and in the master branch of the repository) MatUpdateMPIAIJWithArrays().<br>
<br>
to build the matrix the first time, and then "refill" it with numerical values each new time. There are a few other optimizations related to matrix insertion in the master branch you might also benefit from. <br>
<br>
Generally for problems with multiple "times" or "linear solve steps" we use two stages, the first to track the initial set up and first time step and the other to capture all the other steps (since the extra overhead is only in the first step.) You could make a new stage for each time step but I don't think that is needed.<br>
<br>
After you have this going send us the new log summary.<br>
<br>
Barry<br>
<br>
<br>
<br>
> On Jul 13, 2019, at 11:20 AM, Mohammed Mostafa via petsc-users <<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a>> wrote:<br>
> <br>
> I am sorry but I don’t see what you mean by small times<br>
> Although mat assembly is relatively smaller<br>
> The cost of mat set values is still significant <br>
> The same can be said for vec assembly <br>
> Combined vec/mat assembly and matsetvalues constitute about 50% of the total cost of matrix construction<br>
> <br>
> So is this problem of my matrix setup/ preallocation <br>
> <br>
> Or is this a hardware issue, for whatever reason the copy is overly slow<br>
> The code was run on a single node <br>
> <br>
> Or is this function call overhead since matsetvalues is being called 1M times inside the for loop ( 170k times in each process)<br>
> <br>
> Thanks, Kamra<br>
> <br>
> On Sun, Jul 14, 2019 at 12:41 AM Matthew Knepley <<a href="mailto:knepley@gmail.com" target="_blank">knepley@gmail.com</a>> wrote:<br>
> On Sat, Jul 13, 2019 at 9:56 AM Mohammed Mostafa <<a href="mailto:mo7ammedmostafa@gmail.com" target="_blank">mo7ammedmostafa@gmail.com</a>> wrote:<br>
> Hello Matt,<br>
> <br>
> I revised my code and changed the way I create the rhs vector,<br>
> previosly I was using vecCreateGhost just in case I need the ghost values, but for now I changed that to <br>
> vecCreateMPI(.......)<br>
> So maybe that was the cause of the scatter<br>
> I am attaching with this email a new log output<br>
> <br>
> Okay, the times are now very small. How does it scale up?<br>
> <br>
> Thanks,<br>
> <br>
> Matt<br>
> <br>
> Also regarding how I fill my petsc matrix,<br>
> In my code I fill a temp CSR format matrix becasue otherwise I would need "MatSetValue" to fill the petsc mat element by element<br>
> which is not recommmeded in the petsc manual and probably very expensive due to function call overhead<br>
> So after I create my matrix in CSR format, I fill the PETSC mat A as follows<br>
> for (i = 0; i < nMatRows; i++) {<br>
> cffset = CSR_iptr[i];<br>
> row_index = row_gIndex[i];<br>
> nj = Eqn_nj[i];<br>
> MatSetValues(PhiEqnSolver.A, 1, &row_index, nj, CSR_jptr + offset, CSR_vptr + offset, INSERT_VALUES);<br>
> }<br>
> After That<br>
> VecAssemblyBegin(RHS);<br>
> VecAssemblyEnd(RHS);<br>
> <br>
> MatAssemblyBegin(A, MAT_FINAL_ASSEMBLY);<br>
> MatAssemblyEnd(A, MAT_FINAL_ASSEMBLY);<br>
> <br>
> I don't believe , I am doing anything special, if possible I would like to set the whole csr matrix at once in one command.<br>
> I took a look at the code for MatSetValues, if I am understanding it correctly(hopefully) I think I could do it, maybe modify it or create a new routine entirely for this pupose.<br>
> i.e. MatSetValuesFromCSR(.....)<br>
> Or is there a particular reason why it has to be this way<br>
> <br>
> I also tried ksp ex3 but I slightly tweaked it to add a logging stage around the assembly and MatSetValues and I am attaching the modified example here as well.<br>
> Although in this example the matrix stash is not empty ( means off-processor values are being set ) but the timing values for roughly the same matrix size , the command I used is<br>
> mpirun -np 6 ./mod_ksp_ex3 -m 1000 -log_view -info <br>
> <br>
> <br>
> Regards,<br>
> Kamra<br>
> <br>
> On Sat, Jul 13, 2019 at 1:43 PM Matthew Knepley <<a href="mailto:knepley@gmail.com" target="_blank">knepley@gmail.com</a>> wrote:<br>
> On Fri, Jul 12, 2019 at 10:51 PM Mohammed Mostafa <<a href="mailto:mo7ammedmostafa@gmail.com" target="_blank">mo7ammedmostafa@gmail.com</a>> wrote:<br>
> Hello Matt,<br>
> Attached is the dumped entire log output using -log_view and -info.<br>
> <br>
> In matrix construction, it looks like you have a mixture of load imbalance (see the imbalance in the Begin events)<br>
> and lots of Scatter messages in your assembly. We turn off MatSetValues() logging by default since it is usually<br>
> called many times, but you can explicitly turn it back on if you want. I don't think that is the problem here. Its easy<br>
> to see from examples (say SNES ex5) that it is not the major time sink. What is the Scatter doing?<br>
> <br>
> Thanks,<br>
> <br>
> Matt<br>
> <br>
> Thanks,<br>
> Kamra<br>
> <br>
> On Fri, Jul 12, 2019 at 9:23 PM Matthew Knepley <<a href="mailto:knepley@gmail.com" target="_blank">knepley@gmail.com</a>> wrote:<br>
> On Fri, Jul 12, 2019 at 5:19 AM Mohammed Mostafa via petsc-users <<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a>> wrote:<br>
> Hello all,<br>
> I have a few question regarding Petsc,<br>
> <br>
> Please send the entire output of a run with all the logging turned on, using -log_view and -info.<br>
> <br>
> Thanks,<br>
> <br>
> Matt<br>
> <br>
> Question 1:<br>
> For the profiling , is it possible to only show the user defined log events in the breakdown of each stage in Log-view.<br>
> I tried deactivating all ClassIDs, MAT,VEC, PC, KSP,PC,<br>
> PetscLogEventExcludeClass(MAT_CLASSID);<br>
> PetscLogEventExcludeClass(VEC_CLASSID);<br>
> PetscLogEventExcludeClass(KSP_CLASSID);<br>
> PetscLogEventExcludeClass(PC_CLASSID);<br>
> which should "Deactivates event logging for a PETSc object class in every stage" according to the manual.<br>
> however I still see them in the stage breakdown <br>
> --- Event Stage 1: Matrix Construction<br>
> <br>
> BuildTwoSidedF 4 1.0 2.7364e-02 2.4 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 18 0 0 0 0 0<br>
> VecSet 1 1.0 4.5300e-06 2.4 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0<br>
> VecAssemblyBegin 2 1.0 2.7344e-02 2.4 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 18 0 0 0 0 0<br>
> VecAssemblyEnd 2 1.0 8.3447e-06 1.5 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0<br>
> VecScatterBegin 2 1.0 7.5102e-05 1.7 0.00e+00 0.0 3.6e+01 2.1e+03 0.0e+00 0 0 3 0 0 0 0 50 80 0 0<br>
> VecScatterEnd 2 1.0 3.5286e-05 2.2 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0<br>
> MatAssemblyBegin 2 1.0 8.8930e-05 1.9 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0<br>
> MatAssemblyEnd 2 1.0 1.3566e-02 1.1 0.00e+00 0.0 3.6e+01 5.3e+02 8.0e+00 0 0 3 0 6 10 0 50 20100 0<br>
> AssembleMats 2 1.0 3.9774e-02 1.7 0.00e+00 0.0 7.2e+01 1.3e+03 8.0e+00 0 0 7 0 6 28 0100100100 0 # USER EVENT<br>
> myMatSetValues 2 1.0 2.6931e-02 1.2 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 19 0 0 0 0 0 # USER EVENT<br>
> setNativeMat 1 1.0 3.5613e-02 1.3 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 24 0 0 0 0 0 # USER EVENT<br>
> setNativeMatII 1 1.0 4.7023e-02 1.5 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 28 0 0 0 0 0 # USER EVENT<br>
> callScheme 1 1.0 2.2333e-03 1.2 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 2 0 0 0 0 0 # USER EVENT<br>
> <br>
> Also is possible to clear the logs so that I can write a separate profiling output file for each timestep ( since I am solving a transient problem and I want to know the change in performance as time goes by )<br>
> ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------<br>
> Question 2:<br>
> Regarding MatSetValues<br>
> Right now, I writing a finite volume code, due to algorithm requirement I have to write the matrix into local native format ( array of arrays) and then loop through rows and use MatSetValues to set the elements in "Mat A"<br>
> MatSetValues(A, 1, &row, nj, j_index, coefvalues, INSERT_VALUES);<br>
> but it is very slow and it is killing my performance<br>
> although the matrix was properly set using <br>
> MatCreateAIJ(PETSC_COMM_WORLD, this->local_size, this->local_size, PETSC_DETERMINE,<br>
> PETSC_DETERMINE, -1, d_nnz, -1, o_nnz, &A);<br>
> with d_nnz,and o_nnz properly assigned so no mallocs occur during matsetvalues and all inserted values are local so no off-processor values<br>
> So my question is it possible to set multiple rows at once hopefully all, I checked the manual and MatSetValues can only set dense matrix block because it seems that row by row is expensive<br>
> Or perhaps is it possible to copy all rows to the underlying matrix data, as I mentioned all values are local and no off-processor values ( stash is 0 )<br>
> [0] VecAssemblyBegin_MPI_BTS(): Stash has 0 entries, uses 0 mallocs.<br>
> [0] VecAssemblyBegin_MPI_BTS(): Block-Stash has 0 entries, uses 0 mallocs.<br>
> [0] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.<br>
> [1] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.<br>
> [2] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.<br>
> [3] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.<br>
> [4] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.<br>
> [5] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.<br>
> [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 186064; storage space: 0 unneeded,743028 used<br>
> [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742972 used<br>
> [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0<br>
> [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4<br>
> [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines.<br>
> [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0<br>
> [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4<br>
> [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186064) < 0.6. Do not use CompressedRow routines.<br>
> [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743093 used<br>
> [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,743036 used<br>
> [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0<br>
> [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4<br>
> [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines.<br>
> [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0<br>
> [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 186062; storage space: 0 unneeded,742938 used<br>
> [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0<br>
> [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4<br>
> [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines.<br>
> [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4<br>
> [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186062) < 0.6. Do not use CompressedRow routines.<br>
> [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 186063; storage space: 0 unneeded,743049 used<br>
> [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0<br>
> [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 4<br>
> [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 186063) < 0.6. Do not use CompressedRow routines.<br>
> [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 186064 X 685; storage space: 0 unneeded,685 used<br>
> [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 649; storage space: 0 unneeded,649 used<br>
> [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0<br>
> [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1<br>
> [4] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186063) > 0.6. Use CompressedRow routines.<br>
> [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0<br>
> [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1<br>
> [2] MatCheckCompressedRow(): Found the ratio (num_zerorows 185379)/(num_localrows 186064) > 0.6. Use CompressedRow routines.<br>
> [1] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1011; storage space: 0 unneeded,1011 used<br>
> [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 1137; storage space: 0 unneeded,1137 used<br>
> [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0<br>
> [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1<br>
> [5] MatCheckCompressedRow(): Found the ratio (num_zerorows 184925)/(num_localrows 186062) > 0.6. Use CompressedRow routines.<br>
> [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0<br>
> [1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1<br>
> [3] MatAssemblyEnd_SeqAIJ(): Matrix size: 186063 X 658; storage space: 0 unneeded,658 used<br>
> [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 186062 X 648; storage space: 0 unneeded,648 used<br>
> [1] MatCheckCompressedRow(): Found the ratio (num_zerorows 185051)/(num_localrows 186062) > 0.6. Use CompressedRow routines.<br>
> [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0<br>
> [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1<br>
> [0] MatCheckCompressedRow(): Found the ratio (num_zerorows 185414)/(num_localrows 186062) > 0.6. Use CompressedRow routines.<br>
> [3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0<br>
> [3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1<br>
> [3] MatCheckCompressedRow(): Found the ratio (num_zerorows 185405)/(num_localrows 186063) > 0.6. Use CompressedRow routines.<br>
> <br>
> ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------<br>
> Question 3:<br>
> If all matrix and vector inserted data are local, what part of the vec/mat assembly consumes time because matsetvalues and matassembly consume more time than matrix builder<br>
> Also this is not just for the first time MAT_FINAL_ASSEMBLY<br>
> <br>
> <br>
> For context the matrix in the above is nearly 1Mx1M partitioned over six processes and it was NOT built using DM <br>
> <br>
> Finally the configure options are:<br>
> <br>
> Configure options:<br>
> PETSC_ARCH=release3 -with-debugging=0 COPTFLAGS="-O3 -march=native -mtune=native" CXXOPTFLAGS="-O3 -march=native -mtune=native" FOPTFLAGS="-O3 -march=native -mtune=native" --with-cc=mpicc --with-cxx=mpicxx --with-fc=mpif90 --download-metis --download-hypre<br>
> <br>
> Sorry for such long question and thanks in advance<br>
> Thanks <br>
> M. Kamra<br>
> <br>
> <br>
> -- <br>
> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>
> -- Norbert Wiener<br>
> <br>
> <a href="https://www.cse.buffalo.edu/~knepley/" rel="noreferrer" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br>
> <br>
> <br>
> -- <br>
> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>
> -- Norbert Wiener<br>
> <br>
> <a href="https://www.cse.buffalo.edu/~knepley/" rel="noreferrer" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br>
> <br>
> <br>
> -- <br>
> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>
> -- Norbert Wiener<br>
> <br>
> <a href="https://www.cse.buffalo.edu/~knepley/" rel="noreferrer" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br>
<br>
</blockquote></div></div>
</blockquote></div></div>
</blockquote></div></div>