<html><head><meta http-equiv="Content-Type" content="text/html; charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">Yes, you are correct I missed that part of the run<div class=""><br class=""></div><div class="">As you can see below MatScale calls only BLAS dscal() there is really no way to make that go faster.</div><div class=""><br class=""></div><div class="">How big is the matrix.</div><div class=""><br class=""></div><div class="">What are you doing with the matrix after you scale it? The only way to improve the time is to find some way to scale it less often.</div><div class=""><br class=""></div><div class="">It is curious that VecScale has a much higher flop rate when it has the same code see below. Unless the matrices are tiny I would expect similar flop rates.</div><div class=""><br class=""></div><div class=""> Barry</div><div class=""><br class=""></div><div class=""><br class=""></div><div class=""><div class=""><br class=""></div><div class=""><br class=""><div><div>PetscErrorCode MatScale_SeqAIJ(Mat inA,PetscScalar alpha)</div><div>{</div><div> Mat_SeqAIJ *a = (Mat_SeqAIJ*)inA->data;</div><div> PetscScalar oalpha = alpha;</div><div> PetscErrorCode ierr;</div><div> PetscBLASInt one = 1,bnz;</div><div><br class=""></div><div> PetscFunctionBegin;</div><div> ierr = PetscBLASIntCast(a->nz,&bnz);CHKERRQ(ierr);</div><div> PetscStackCallBLAS("BLASscal",BLASscal_(&bnz,&oalpha,a->a,&one));</div><div> ierr = PetscLogFlops(a->nz);CHKERRQ(ierr);</div><div> ierr = MatSeqAIJInvalidateDiagonal(inA);CHKERRQ(ierr);</div><div>#if defined(PETSC_HAVE_DEVICE)</div><div> if (inA->offloadmask != PETSC_OFFLOAD_UNALLOCATED) inA->offloadmask = PETSC_OFFLOAD_CPU;</div><div>#endif</div><div> PetscFunctionReturn(0);</div><div>}</div><div class=""><br class=""></div><div class=""><div class="">PetscErrorCode VecScale_Seq(Vec xin, PetscScalar alpha)</div><div class="">{</div><div class=""> PetscErrorCode ierr;</div><div class=""> PetscBLASInt one = 1,bn;</div><div class=""><br class=""></div><div class=""> PetscFunctionBegin;</div><div class=""> ierr = PetscBLASIntCast(xin->map->n,&bn);CHKERRQ(ierr);</div><div class=""> if (alpha == (PetscScalar)0.0) {</div><div class=""> ierr = VecSet_Seq(xin,alpha);CHKERRQ(ierr);</div><div class=""> } else if (alpha != (PetscScalar)1.0) {</div><div class=""> PetscScalar a = alpha,*xarray;</div><div class=""> ierr = VecGetArray(xin,&xarray);CHKERRQ(ierr);</div><div class=""> PetscStackCallBLAS("BLASscal",BLASscal_(&bn,&a,xarray,&one));</div><div class=""> ierr = VecRestoreArray(xin,&xarray);CHKERRQ(ierr);</div><div class=""> }</div><div class=""> ierr = PetscLogFlops(xin->map->n);CHKERRQ(ierr);</div><div class=""> PetscFunctionReturn(0);</div><div class="">}</div></div><div class=""><br class=""></div><div class=""><br class=""></div><div class=""><br class=""></div><blockquote type="cite" class=""><div class="">On Oct 22, 2020, at 3:02 PM, Antoine Côté <<a href="mailto:Antoine.Cote3@USherbrooke.ca" class="">Antoine.Cote3@USherbrooke.ca</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><div style="font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; background-color: rgb(255, 255, 255);" class="">Hi,</div><div style="font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; background-color: rgb(255, 255, 255);" class=""><br class=""></div><div style="font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; background-color: rgb(255, 255, 255);" class="">See attached files for both outputs. Tell me if you need any clarification. It was run with a DMDA of 33x17x17 nodes (creating 32x16x16=8192 elements). With 3 dof per nodes, problem has a total of 28611 dof.</div><div style="font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; background-color: rgb(255, 255, 255);" class=""><br class=""></div><div style="font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; background-color: rgb(255, 255, 255);" class="">Note : Stage "Stiff_Adj" is the part of the code modifying Mat K. PetscLogStagePush/Pop was used.</div><div style="font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; background-color: rgb(255, 255, 255);" class=""><br class=""></div><div style="font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; background-color: rgb(255, 255, 255);" class="">Regards,</div><div style="font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; background-color: rgb(255, 255, 255);" class=""><br class=""></div><div style="font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; background-color: rgb(255, 255, 255);" class="">Antoine</div><div id="appendonsend" style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 18px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""></div><hr tabindex="-1" style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 18px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; display: inline-block; width: 1138.75px;" class=""><span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 18px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;" class=""></span><div id="divRplyFwdMsg" dir="ltr" style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 18px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><font face="Calibri, sans-serif" style="font-size: 11pt;" class=""><b class="">De :</b><span class="Apple-converted-space"> </span>Matthew Knepley <<a href="mailto:knepley@gmail.com" class="">knepley@gmail.com</a>><br class=""><b class="">Envoyé :</b><span class="Apple-converted-space"> </span>22 octobre 2020 15:35<br class=""><b class="">À :</b><span class="Apple-converted-space"> </span>Antoine Côté <<a href="mailto:Antoine.Cote3@USherbrooke.ca" class="">Antoine.Cote3@USherbrooke.ca</a>><br class=""><b class="">Cc :</b><span class="Apple-converted-space"> </span><a href="mailto:petsc-users@mcs.anl.gov" class="">petsc-users@mcs.anl.gov</a><span class="Apple-converted-space"> </span><<a href="mailto:petsc-users@mcs.anl.gov" class="">petsc-users@mcs.anl.gov</a>><br class=""><b class="">Objet :</b><span class="Apple-converted-space"> </span>Re: [petsc-users] Enhancing MatScale computing time</font><div class=""> </div></div><div style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 18px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><div dir="ltr" class=""><div dir="ltr" class="">On Thu, Oct 22, 2020 at 3:23 PM Antoine Côté <<a href="mailto:Antoine.Cote3@usherbrooke.ca" class="">Antoine.Cote3@usherbrooke.ca</a>> wrote:<br class=""></div><div class="x_gmail_quote"><blockquote class="x_gmail_quote" style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-style: solid; border-left-color: rgb(204, 204, 204); padding-left: 1ex;"><div dir="ltr" class=""><div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; background-color: rgb(255, 255, 255);" class="">Hi,</div><div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; background-color: rgb(255, 255, 255);" class=""><br class=""></div><div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; background-color: rgb(255, 255, 255);" class="">I'm working with a 3D DMDA, with 3 dof per "node", used to create a sparse matrix Mat K. The Mat is modified repeatedly by the program, using the commands (in that order) :</div><div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; background-color: rgb(255, 255, 255);" class=""><br class=""></div><div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; background-color: rgb(255, 255, 255);" class="">MatZeroEntries(K)<br class=""></div><div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; background-color: rgb(255, 255, 255);" class="">In a for loop : MatSetValuesLocal(K, 24, irow, 24, icol, vals, ADD_VALUES)</div><div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; background-color: rgb(255, 255, 255);" class="">MatAssemblyBegin(K, MAT_FINAL_ASSEMBLY)<br class=""></div><div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; background-color: rgb(255, 255, 255);" class=""><span style="background-color: rgb(255, 255, 255); display: inline;" class="">MatAssemblyEnd(K, MAT_FINAL_ASSEMBLY)</span><br class=""></div><div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; background-color: rgb(255, 255, 255);" class="">MatDiagonalScale(K, vec1, vec1)<br class=""></div><div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; background-color: rgb(255, 255, 255);" class="">MatDiagonalSet(K, vec2, ADD_VALUES)<br class=""></div><div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; background-color: rgb(255, 255, 255);" class=""><br class=""></div><div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; background-color: rgb(255, 255, 255);" class=""><span style="background-color: rgb(255, 255, 255); display: inline;" class="">Computing time seems high and I</span><span style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt;" class=""> would like to improve it. Running tests with "-log_view" tells me that MatScale() is the bottle neck (50% of total computing time) . From manual pages, I've tried a few tweaks :</span></div><div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; background-color: rgb(255, 255, 255);" class=""><ul class=""><li class="">DMSetMatType(da, MATMPIBAIJ) : "For problems with multiple degrees of freedom per node, ... BAIJ can significantly enhance performance", Chapter 14.2.4<br class=""></li><li class="">Used MatMissingDiagonal() to confirm there is no missing diagonal entries : "If the matrix Y is missing some diagonal entries this routine can be very slow", MatDiagonalSet() manual</li><li class="">Tried MatSetOption()</li><ul class=""><li class="">MAT_NEW_NONZERO_LOCATIONS == PETSC_FALSE : to increase assembly efficiency</li><li class="">MAT_NEW_NONZERO_LOCATION_ERR == PETSC_TRUE : "When true, assembly processes have one less global reduction"<br class=""></li><li class="">MAT_NEW_NONZERO_ALLOCATION_ERR == PETSC_TRUE : "When true, assembly processes have one less global reduction"<br class=""></li><li class="">MAT_USE_HASH_TABLE == PETSC_TRUE : "Improve the searches during matrix assembly"<br class=""></li></ul></ul><div class=""><span class="">According to<span class="Apple-converted-space"> </span></span><span style="background-color: rgb(255, 255, 255); display: inline;" class="">"-log_view"</span><span class="">, assembly is fast (0% of total time), and the use of a DMDA makes me believe preallocation isn't the cause of performance issue.</span><br class=""></div><div class=""><br class=""></div><div class=""><span style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt;" class="">I would like to know how could I improve <span style="background-color: rgb(255, 255, 255); display: inline;" class="">MatScale(). W</span>hat are the best practices (during allocation, when defining Vecs and Mats, the DMDA, etc.)? Instead of <span style="background-color: rgb(255, 255, 255); display: inline;" class="">MatDiagonalScale(), s<span style="background-color: rgb(255, 255, 255); display: inline;" class="">hould I use another command<span class=""> </span></span>to obtain the same result faster?</span></span></div></div></div></blockquote><div class=""><br class=""></div><div class="">Something is definitely strange. Can you please send the output of</div><div class=""><br class=""></div><div class=""> -log_view -info :mat</div><div class=""><br class=""></div><div class=""> Thanks,</div><div class=""><br class=""></div><div class=""> Matt</div><div class=""> </div><blockquote class="x_gmail_quote" style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-style: solid; border-left-color: rgb(204, 204, 204); padding-left: 1ex;"><div dir="ltr" class=""><div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; background-color: rgb(255, 255, 255);" class=""><div class=""><span style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt;" class="">Thank you very much!</span></div><div class=""><span style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt;" class=""><br class=""></span></div><div class=""><span style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt;" class="">Antoine Côté</span></div></div><div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; background-color: rgb(255, 255, 255);" class=""><br class=""></div></div></blockquote></div><br clear="all" class=""><div class=""><br class=""></div>--<span class="Apple-converted-space"> </span><br class=""><div dir="ltr" class="x_gmail_signature"><div dir="ltr" class=""><div class=""><div dir="ltr" class=""><div class=""><div dir="ltr" class=""><div class="">What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br class="">-- Norbert Wiener</div><div class=""><br class=""></div><div class=""><a href="https://can01.safelinks.protection.outlook.com/?url=http:%2F%2Fwww.cse.buffalo.edu%2F~knepley%2F&data=04%7C01%7CAntoine.Cote3%40usherbrooke.ca%7C6b823852b3964170f52908d876c1bb0b%7C3a5a8744593545f99423b32c3a5de082%7C0%7C0%7C637389921724846720%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=w7%2Fc%2BSzAfTa02gxTS8VbB%2FVwIPpaKw%2F%2BeiiX4K9gd1k%3D&reserved=0" originalsrc="http://www.cse.buffalo.edu/~knepley/" shash="A5TjyYp384mqAvon0t9PnnYoeIvNxn7rXoeq1ZpGAU5jQLwdr/TkqUejSYs09W94qjJB2V6IlQISWWPmO+ow7uDPEC+T1NcrsZ/dKBWpj+KFYJGOu+LfZ6kIN24RcMZVWz1owUXpQfAi9vy4eyCZt+xAJpFEDU3vGn+LfVFi3h8=" target="_blank" class="">https://www.cse.buffalo.edu/~knepley/</a><br class=""></div></div></div></div></div></div></div></div></div><span id="cid:92E7F4D1-2260-4DCD-9A47-B11ECD816830@hsd1.il.comcast.net"><LogView.out></span><span id="cid:18BE52CE-82ED-446C-9774-1E9A8F4B26AF@hsd1.il.comcast.net"><mat.0></span></div></blockquote></div><br class=""></div></div></body></html>