<html><head><meta http-equiv="Content-Type" content="text/html; charset=us-ascii"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class=""><div class=""> Thanks Mark, far more interesting. I've improved the formatting to make it easier to read (and fixed width font for email reading)</div><div class=""><br class=""></div><div class=""> * Can you do same run with say 10 iterations of Jacobi PC?</div><div class=""><br class=""></div><div class=""> * PCApply performance (looks like GAMG) is terrible! Problems too small?</div><div class=""><br class=""></div><div class=""> * VecScatter time is completely dominated by SFPack! Junchao what's up with that? Lots of little kernels in the PCApply? PCJACOBI run will help clarify where that is coming from.</div><div class=""><br class=""></div><div class=""><font face="Courier New" class=""><span style="font-style: normal; font-size: 12px;" class="">------------------------------------------------------------------------------------------------------------------------</span></font></div><div class=""><font face="Courier New" class=""><span style="font-style: normal; font-size: 12px;" class="">Event Count Time (sec) Flop --- Global --- --- Stage ---- Total GPU - CpuToGpu - - GpuToCpu - GPU</span></font></div><div class=""><font face="Courier New" class=""><span style="font-style: normal; font-size: 12px;" class=""> Max Ratio Max Ratio Max Ratio Mess AvgLen Reduct %T %F %M %L %R %T %F %M %L %R Mflop/s Mflop/s Count Size Count Size %F</span></font></div><div class=""><font face="Courier New" class=""><span style="font-style: normal; font-size: 12px;" class="">---------------------------------------------------------------------------------------------------------------------------------------------------------------</span></font></div><div class=""><font face="Courier New" class=""><span style="font-style: normal; font-size: 12px;" class=""><br class=""></span></font></div><div class=""><div class=""><font face="Courier New" class=""><span style="font-style: normal; font-size: 12px;" class="">MatMult 200 1.0 6.7831e-01 1.0 4.91e+10 1.0 1.1e+04 6.6e+04 1.0e+00 9 92 99 79 0 71 92100100 0 579,635 1,014,212 1 2.04e-04 0 0.00e+00 100</span></font></div><div class=""><font face="Courier New" class=""><span style="font-style: normal; font-size: 12px;" class="">KSPSolve 1 1.0 9.4550e-01 1.0 5.31e+10 1.0 1.1e+04 6.6e+04 6.0e+02 12100 99 79 94 100100100100100 449,667 893,741 1 2.04e-04 0 0.00e+00 100</span></font></div><div class=""><font face="Courier New" class=""><span style="font-style: normal; font-size: 12px;" class="">PCApply 201 1.0 1.6966e-01 1.0 3.09e+08 1.0 0.0e+00 0.0e+00 2.0e+00 2 1 0 0 0 18 1 0 0 0 14,558 16,3941 0 0.00e+00 0 0.00e+00 100</span></font></div><div class=""><font face="Courier New" class=""><span style="font-style: normal; font-size: 12px;" class="">VecTDot 401 1.0 5.3642e-02 1.3 1.23e+09 1.0 0.0e+00 0.0e+00 4.0e+02 1 2 0 0 62 5 2 0 0 66 183,716 353,914 0 0.00e+00 0 0.00e+00 100</span></font></div><div class=""><font face="Courier New" class=""><span style="font-style: normal; font-size: 12px;" class="">VecNorm 201 1.0 2.2219e-02 1.1 6.17e+08 1.0 0.0e+00 0.0e+00 2.0e+02 0 1 0 0 31 2 1 0 0 33 222,325 303,155 0 0.00e+00 0 0.00e+00 100</span></font></div><div class=""><font face="Courier New" class=""><span style="font-style: normal; font-size: 12px;" class="">VecAXPY 400 1.0 2.3017e-02 1.1 1.23e+09 1.0 0.0e+00 0.0e+00 0.0e+00 0 2 0 0 0 2 2 0 0 0 427,091 514,744 0 0.00e+00 0 0.00e+00 100</span></font></div><div class=""><font face="Courier New" class=""><span style="font-style: normal; font-size: 12px;" class="">VecAYPX 199 1.0 1.1312e-02 1.1 6.11e+08 1.0 0.0e+00 0.0e+00 0.0e+00 0 1 0 0 0 1 1 0 0 0 432,323 532,889 0 0.00e+00 0 0.00e+00 100</span></font></div><div class=""><font face="Courier New" class=""><span style="font-style: normal; font-size: 12px;" class="">VecPointwiseMult 201 1.0 1.0471e-02 1.1 3.09e+08 1.0 0.0e+00 0.0e+00 0.0e+00 0 1 0 0 0 1 1 0 0 0 235,882 290,088 0 0.00e+00 0 0.00e+00 100</span></font></div><div class=""><font face="Courier New" class=""><span style="font-style: normal; font-size: 12px;" class="">VecScatterBegin 200 1.0 1.8458e-01 1.1 0.00e+00 0.0 1.1e+04 6.6e+04 1.0e+00 2 0 99 79 0 19 0100100 0 0 0 1 2.04e-04 0 0.00e+00 0</span></font></div><div class=""><font face="Courier New" class=""><span style="font-style: normal; font-size: 12px;" class="">VecScatterEnd 200 1.0 1.9007e-02 3.7 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 1 0 0 0 0 0 0 0 0.00e+00 0 0.00e+00 0</span></font></div><div class=""><font face="Courier New" class=""><span style="font-style: normal; font-size: 12px;" class="">SFPack 200 1.0 1.7309e-01 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 2 0 0 0 0 18 0 0 0 0 0 0 1 2.04e-04 0 0.00e+00 0</span></font></div><div class=""><font face="Courier New" class=""><span style="font-style: normal; font-size: 12px;" class="">SFUnpack 200 1.0 2.3165e-05 1.4 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 0 0 0.00e+00 0 0.00e+00 0</span></font></div></div><div class=""><br class=""></div><div><br class=""><blockquote type="cite" class=""><div class="">On Jan 25, 2022, at 8:29 AM, Mark Adams <<a href="mailto:mfadams@lbl.gov" class="">mfadams@lbl.gov</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><div dir="ltr" class="">adding Suyash,<div class=""><br class=""></div><div class="">I found the/a problem. Using ex56, which has a crappy decomposition, using one MPI process/GPU is much faster than using 8 (64 total). (I am looking at ex13 to see how much of this is due to the decomposition)</div><div class="">If you only use 8 processes it seems that all 8 are put on the first GPU, but adding -c8 seems to fix this.</div><div class="">Now the numbers are looking reasonable.</div></div><br class=""><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, Jan 24, 2022 at 3:24 PM Barry Smith <<a href="mailto:bsmith@petsc.dev" class="">bsmith@petsc.dev</a>> wrote:<br class=""></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div style="overflow-wrap: break-word;" class=""><div class=""><br class=""></div> For this, to start, someone can run <div class=""><br class=""></div><div class=""><div style="margin:0px;font-stretch:normal;font-size:14px;line-height:normal;font-family:Menlo" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">src/vec/vec/tutorials/performance.c </span></div><div style="margin:0px;font-stretch:normal;font-size:14px;line-height:normal;font-family:Menlo" class=""><span style="font-variant-ligatures:no-common-ligatures" class=""><br class=""></span></div><div style="margin:0px;font-stretch:normal;line-height:normal" class=""><span style="font-family:Helvetica;font-size:18px" class="">and compare the performance to that in the technical report </span>Evaluation of PETSc on a Heterogeneous Architecture \\ the OLCF Summit System \\ Part I: Vector Node Performance. Google to find. One does not have to and shouldn't do an extensive study right now that compares everything, instead one should run a very small number of different size problems (make them big) and compare those sizes with what Summit gives. Note you will need to make sure that performance.c uses the Kokkos backend.</div><div style="margin:0px;font-stretch:normal;line-height:normal" class=""><br class=""></div><div style="margin:0px;font-stretch:normal;line-height:normal" class=""> One hopes for better performance than Summit; if one gets tons worse we know something is very wrong somewhere. I'd love to see some comparisons.</div><div style="margin:0px;font-stretch:normal;line-height:normal" class=""><br class=""></div><div style="margin:0px;font-stretch:normal;line-height:normal" class=""> Barry</div><div style="margin:0px;font-stretch:normal;line-height:normal" class=""><br class=""></div><div class=""><br class=""><blockquote type="cite" class=""><div class="">On Jan 24, 2022, at 3:06 PM, Justin Chang <<a href="mailto:jychang48@gmail.com" target="_blank" class="">jychang48@gmail.com</a>> wrote:</div><br class=""><div class=""><div dir="ltr" class="">Also, do you guys have an OLCF liaison? That's actually your better bet if you do. <br class=""><br class="">Performance issues with ROCm/Kokkos are pretty common in apps besides just PETSc. We have several teams actively working on rectifying this. However, I think performance issues can be quicker to identify if we had a more "official" and reproducible PETSc GPU benchmark, which I've already expressed to some folks in this thread, and as others already commented on the difficulty of such a task. Hopefully I will have more time soon to illustrate what I am thinking.</div><br class=""><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, Jan 24, 2022 at 1:57 PM Justin Chang <<a href="mailto:jychang48@gmail.com" target="_blank" class="">jychang48@gmail.com</a>> wrote:<br class=""></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr" class=""><div dir="auto" class="">My name has been called.</div><div dir="auto" class=""><br class=""></div><div dir="auto" class="">Mark, if you're having issues with Crusher, please contact Veronica Vergara (<a href="mailto:vergaravg@ornl.gov" target="_blank" class="">vergaravg@ornl.gov</a>). You can cc me (<a href="mailto:justin.chang@amd.com" target="_blank" class="">justin.chang@amd.com</a>) in those emails</div></div><div class=""><br class=""><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, Jan 24, 2022 at 1:49 PM Barry Smith <<a href="mailto:bsmith@petsc.dev" target="_blank" class="">bsmith@petsc.dev</a>> wrote:<br class=""></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div class=""><br class=""><div class=""><br class=""><blockquote type="cite" class=""><div class="">On Jan 24, 2022, at 2:46 PM, Mark Adams <<a href="mailto:mfadams@lbl.gov" target="_blank" class="">mfadams@lbl.gov</a>> wrote:</div><br class=""><div class=""><div dir="ltr" class="">Yea, CG/Jacobi is as close to a benchmark code as we could want. I could run this on one processor to get cleaner numbers.<div class=""><br class=""></div><div class="">Is there a designated ECP technical support contact?</div></div></div></blockquote><div class=""><br class=""></div> Mark, you've forgotten you work for DOE. There isn't a non-ECP technical support contact. </div><div class=""><br class=""></div><div class=""> But if this is an AMD machine then maybe contact Matt's student Justin Chang?</div></div><div class=""><div class=""><br class=""></div><div class=""><br class=""></div><div class=""><br class=""><blockquote type="cite" class=""><div class=""><div dir="ltr" class=""><div class=""><br class=""></div></div><br class=""><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, Jan 24, 2022 at 2:18 PM Barry Smith <<a href="mailto:bsmith@petsc.dev" target="_blank" class="">bsmith@petsc.dev</a>> wrote:<br class=""></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div class=""><div class=""><br class=""></div> I think you should contact the crusher ECP technical support team and tell them you are getting dismel performance and ask if you should expect better. Don't waste time flogging a dead horse. <br class=""><div class=""><br class=""><blockquote type="cite" class=""><div class="">On Jan 24, 2022, at 2:16 PM, Matthew Knepley <<a href="mailto:knepley@gmail.com" target="_blank" class="">knepley@gmail.com</a>> wrote:</div><br class=""><div class=""><div dir="ltr" class=""><div dir="ltr" class="">On Mon, Jan 24, 2022 at 2:11 PM Junchao Zhang <<a href="mailto:junchao.zhang@gmail.com" target="_blank" class="">junchao.zhang@gmail.com</a>> wrote:<br class=""></div><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr" class=""><div dir="ltr" class=""><br class=""></div><br class=""><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, Jan 24, 2022 at 12:55 PM Mark Adams <<a href="mailto:mfadams@lbl.gov" target="_blank" class="">mfadams@lbl.gov</a>> wrote:<br class=""></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr" class=""><div dir="ltr" class=""><br class=""></div><br class=""><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, Jan 24, 2022 at 1:38 PM Junchao Zhang <<a href="mailto:junchao.zhang@gmail.com" target="_blank" class="">junchao.zhang@gmail.com</a>> wrote:<br class=""></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr" class="">Mark, I think you can benchmark individual vector operations, and once we get reasonable profiling results, we can move to solvers etc.</div></blockquote><div class=""><br class=""></div><div class="">Can you suggest a code to run or are you suggesting making a vector benchmark code?</div></div></div></blockquote><div class="">Make a vector benchmark code, testing vector operations that would be used in your solver.</div><div class="">Also, we can run MatMult() to see if the profiling result is reasonable.</div><div class="">Only once we get some solid results on basic operations, it is useful to run big codes.</div></div></div></blockquote><div class=""><br class=""></div><div class="">So we have to make another throw-away code? Why not just look at the vector ops in Mark's actual code?</div><div class=""><br class=""></div><div class=""> Matt</div><div class=""> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr" class=""><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr" class=""><div class="gmail_quote"><div class=""> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr" class=""><div class=""><br class=""><div class=""><div class=""><div dir="ltr" class=""><div dir="ltr" class="">--Junchao Zhang</div></div></div><br class=""></div></div></div><br class=""><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, Jan 24, 2022 at 12:09 PM Mark Adams <<a href="mailto:mfadams@lbl.gov" target="_blank" class="">mfadams@lbl.gov</a>> wrote:<br class=""></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr" class=""><div dir="ltr" class=""><br class=""></div><br class=""><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, Jan 24, 2022 at 12:44 PM Barry Smith <<a href="mailto:bsmith@petsc.dev" target="_blank" class="">bsmith@petsc.dev</a>> wrote:<br class=""></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div class=""><div class=""><br class=""></div> Here except for VecNorm the GPU is used effectively in that most of the time is time is spent doing real work on the GPU<div class=""><br class=""></div><div class=""><div class="">VecNorm 402 1.0 4.4100e-01 6.1 1.69e+09 1.0 0.0e+00 0.0e+00 4.0e+02 0 1 0 0 20 9 1 0 0 33 30230 225393 0 0.00e+00 0 0.00e+00 100</div><div class=""><br class=""></div><div class="">Even the dots are very effective, only the VecNorm flop rate over the full time is much much lower than the vecdot. Which is somehow due to the use of the GPU or CPU MPI in the allreduce?</div></div></div></blockquote><div class=""><br class=""></div><div class="">The VecNorm GPU rate is relatively high on Crusher and the CPU rate is about the same as the other vec ops. I don't know what to make of that.</div><div class=""><br class=""></div><div class="">But Crusher is clearly not crushing it. </div><div class=""><br class=""></div><div class="">Junchao: Perhaps we should ask Kokkos if they have any experience with Crusher that they can share. They could very well find some low level magic.</div><div class=""><br class=""></div><div class=""><br class=""></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div class=""><div class=""><div class=""><br class=""></div><div class=""><br class=""></div><div class=""><br class=""><blockquote type="cite" class=""><div class="">On Jan 24, 2022, at 12:14 PM, Mark Adams <<a href="mailto:mfadams@lbl.gov" target="_blank" class="">mfadams@lbl.gov</a>> wrote:</div><br class=""><div class=""><div dir="ltr" class=""><div dir="ltr" class=""><br class=""></div><div dir="ltr" class=""><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><br class="">
Mark, can we compare with Spock?<br class=""></blockquote><div class=""><br class=""></div><div class=""> Looks much better. This puts two processes/GPU because there are only 4.</div></div></div>
</div>
<span id="gmail-m_-2479378921784158436gmail-m_6588895530181966739gmail-m_7911931152491619875m_-793860302615684860gmail-m_1602764271510560966gmail-m_7034942330692579726gmail-m_557496224917787162gmail-m_6873512096861806548gmail-m_-3802472944769362989gmail-m_2338317291810935402cid:f_kysy6if70" class=""><jac_out_001_kokkos_Spock_6_1_notpl.txt></span></div></blockquote></div><br class=""></div></div></blockquote></div></div>
</blockquote></div>
</blockquote></div></div>
</blockquote></div></div>
</blockquote></div><br clear="all" class=""><div class=""><br class=""></div>-- <br class=""><div dir="ltr" class=""><div dir="ltr" class=""><div class=""><div dir="ltr" class=""><div class=""><div dir="ltr" class=""><div class="">What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br class="">-- Norbert Wiener</div><div class=""><br class=""></div><div class=""><a href="http://www.cse.buffalo.edu/~knepley/" target="_blank" class="">https://www.cse.buffalo.edu/~knepley/</a><br class=""></div></div></div></div></div></div></div></div>
</div></blockquote></div><br class=""></div></blockquote></div>
</div></blockquote></div><br class=""></div></blockquote></div></div>
</blockquote></div>
</div></blockquote></div><br class=""></div></div></blockquote></div>
<span id="cid:f_kyu5esvo0"><jac_out_001_kokkos_Crusher_159_1.txt></span></div></blockquote></div><br class=""></body></html>