<html><head><meta http-equiv="Content-Type" content="text/html; charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class=""><div class=""><br class=""></div> PCSetUpOnBlocks() calls KSPSetUp() on each block which calls PCSetUp() on each block. This is why you are getting the confusing logging results. So the "<div class="">extra" KSPSetUp() is the one on the blocks.</div><div class=""><div><br class=""><blockquote type="cite" class=""><div class="">On Oct 5, 2021, at 12:02 PM, Karthikeyan Chockalingam - STFC UKRI <<a href="mailto:karthikeyan.chockalingam@stfc.ac.uk" class="">karthikeyan.chockalingam@stfc.ac.uk</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><meta charset="UTF-8" class=""><div class="WordSection1" style="page: WordSection1; caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 18px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;"><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span class="">Hi Matt,<o:p class=""></o:p></span></div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span class=""><o:p class=""> </o:p></span></div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span class="">I have a couple of questions;<o:p class=""></o:p></span></div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span class=""><o:p class=""> </o:p></span></div><ol start="1" type="1" style="margin-bottom: 0cm; margin-top: 0cm;" class=""><li class="MsoListParagraph" style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"><span class="">Weather I run on single core or on multiple cores I find that KSPSetUP (ksp/tutorial/ex.45.c) is always called twice. But why? Is setup not done once?<o:p class=""></o:p></span></li><li class="MsoListParagraph" style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"><span class="">I find PCSetUpOnBlocks is calling PCSetup and not the other way around. Can you shed some light? The preconditioner used is block jacobi.<o:p class=""></o:p></span></li></ol><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span class=""><o:p class=""> </o:p></span></div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span class="">Thanks,<o:p class=""></o:p></span></div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span class="">Karthik.<o:p class=""></o:p></span></div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span class=""><o:p class=""> </o:p></span></div><div style="border-style: solid none none; border-top-width: 1pt; border-top-color: rgb(181, 196, 223); padding: 3pt 0cm 0cm;" class=""><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><b class=""><span style="font-size: 12pt;" class="">From:<span class="Apple-converted-space"> </span></span></b><span style="font-size: 12pt;" class="">Matthew Knepley <<a href="mailto:knepley@gmail.com" style="color: blue; text-decoration: underline;" class="">knepley@gmail.com</a>><br class=""><b class="">Date:<span class="Apple-converted-space"> </span></b>Sunday, 3 October 2021 at 12:54<br class=""><b class="">To:<span class="Apple-converted-space"> </span></b>"Chockalingam, Karthikeyan (STFC,DL,HC)" <<a href="mailto:karthikeyan.chockalingam@stfc.ac.uk" style="color: blue; text-decoration: underline;" class="">karthikeyan.chockalingam@stfc.ac.uk</a>><br class=""><b class="">Cc:<span class="Apple-converted-space"> </span></b>"<a href="mailto:petsc-users@mcs.anl.gov" style="color: blue; text-decoration: underline;" class="">petsc-users@mcs.anl.gov</a>" <<a href="mailto:petsc-users@mcs.anl.gov" style="color: blue; text-decoration: underline;" class="">petsc-users@mcs.anl.gov</a>><br class=""><b class="">Subject:<span class="Apple-converted-space"> </span></b>Re: [petsc-users] (percent time in this phase)<o:p class=""></o:p></span></div></div><div class=""><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><o:p class=""> </o:p></div></div><div class=""><div class=""><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class="">On Sun, Oct 3, 2021 at 5:43 AM Karthikeyan Chockalingam - STFC UKRI <<a href="mailto:karthikeyan.chockalingam@stfc.ac.uk" style="color: blue; text-decoration: underline;" class="">karthikeyan.chockalingam@stfc.ac.uk</a>> wrote:<o:p class=""></o:p></div></div><div class=""><blockquote style="border-style: none none none solid; border-left-width: 1pt; border-left-color: rgb(204, 204, 204); padding: 0cm 0cm 0cm 6pt; margin-left: 4.8pt; margin-right: 0cm;" class=""><div class=""><div class=""><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class="">Hi Matt,<o:p class=""></o:p></div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> <o:p class=""></o:p></div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class="">Thank you. The flamegraph tool is helpful.<o:p class=""></o:p></div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class="">Please find the attached screen shoot and foo.txt which generated that graph (using<span class="Apple-converted-space"> </span><a href="https://www.speedscope.app/" target="_blank" style="color: blue; text-decoration: underline;" class="">https://www.speedscope.app</a>).<o:p class=""></o:p></div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class="">I find the following call sequence from the graph<o:p class=""></o:p></div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class="">KSPSolve -> PCApply -> KSPSolve -> PCApply -> MatSolve<o:p class=""></o:p></div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> <o:p class=""></o:p></div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class="">I have a couple of questions<o:p class=""></o:p></div><ol start="1" type="1" style="margin-bottom: 0cm;" class=""><li class="gmail-m-9161537463986845899msolistparagraph" style="margin-right: 0cm; margin-left: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;">The KSPSolve time listed in the file using -log_summary (or -log_view), is it the time taken by the first KSPSolve (in the above call sequence)?<o:p class=""></o:p></li></ol></div></div></blockquote><div class=""><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class="">Yes. When calls are nested, we just do not record the time for the nested call in log_view.<o:p class=""></o:p></div></div><div class=""><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> <o:p class=""></o:p></div></div><blockquote style="border-style: none none none solid; border-left-width: 1pt; border-left-color: rgb(204, 204, 204); padding: 0cm 0cm 0cm 6pt; margin-left: 4.8pt; margin-right: 0cm;" class=""><div class=""><div class=""><ol start="1" type="1" style="margin-bottom: 0cm;" class=""><li class="gmail-m-9161537463986845899msolistparagraph" style="margin-right: 0cm; margin-left: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"><o:p class=""> </o:p></li><li class="gmail-m-9161537463986845899msolistparagraph" style="margin-right: 0cm; margin-left: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;">What is the unit of measurement in flamegrah?<o:p class=""></o:p></li></ol></div></div></blockquote><div class=""><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class="">I believe it is microseconds, but I am not sure.<o:p class=""></o:p></div></div><div class=""><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><o:p class=""> </o:p></div></div><div class=""><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> Thanks,<o:p class=""></o:p></div></div><div class=""><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><o:p class=""> </o:p></div></div><div class=""><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> Matt<o:p class=""></o:p></div></div><div class=""><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> <o:p class=""></o:p></div></div><blockquote style="border-style: none none none solid; border-left-width: 1pt; border-left-color: rgb(204, 204, 204); padding: 0cm 0cm 0cm 6pt; margin-left: 4.8pt; margin-right: 0cm;" class=""><div class=""><div class=""><ol start="1" type="1" style="margin-bottom: 0cm;" class=""><li class="gmail-m-9161537463986845899msolistparagraph" style="margin-right: 0cm; margin-left: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"><o:p class=""> </o:p></li></ol><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class="">Thanks,<o:p class=""></o:p></div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class="">Karthik.<o:p class=""></o:p></div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> <o:p class=""></o:p></div><div style="border-style: solid none none; border-top-width: 1pt; border-top-color: rgb(181, 196, 223); padding: 3pt 0cm 0cm;" class=""><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><b class=""><span style="font-size: 12pt;" class="">From:<span class="Apple-converted-space"> </span></span></b><span style="font-size: 12pt;" class="">Matthew Knepley <<a href="mailto:knepley@gmail.com" target="_blank" style="color: blue; text-decoration: underline;" class="">knepley@gmail.com</a>><br class=""><b class="">Date:<span class="Apple-converted-space"> </span></b>Friday, 1 October 2021 at 14:51<br class=""><b class="">To:<span class="Apple-converted-space"> </span></b>"Chockalingam, Karthikeyan (STFC,DL,HC)" <<a href="mailto:karthikeyan.chockalingam@stfc.ac.uk" target="_blank" style="color: blue; text-decoration: underline;" class="">karthikeyan.chockalingam@stfc.ac.uk</a>><br class=""><b class="">Cc:<span class="Apple-converted-space"> </span></b>Barry Smith <<a href="mailto:bsmith@petsc.dev" target="_blank" style="color: blue; text-decoration: underline;" class="">bsmith@petsc.dev</a>>, "<a href="mailto:petsc-users@mcs.anl.gov" target="_blank" style="color: blue; text-decoration: underline;" class="">petsc-users@mcs.anl.gov</a>" <<a href="mailto:petsc-users@mcs.anl.gov" target="_blank" style="color: blue; text-decoration: underline;" class="">petsc-users@mcs.anl.gov</a>><br class=""><b class="">Subject:<span class="Apple-converted-space"> </span></b>Re: [petsc-users] (percent time in this phase)</span><o:p class=""></o:p></div></div><div class=""><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> <o:p class=""></o:p></div></div><div class=""><div class=""><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class="">On Thu, Sep 30, 2021 at 8:50 AM Karthikeyan Chockalingam - STFC UKRI <<a href="mailto:karthikeyan.chockalingam@stfc.ac.uk" target="_blank" style="color: blue; text-decoration: underline;" class="">karthikeyan.chockalingam@stfc.ac.uk</a>> wrote:<o:p class=""></o:p></div></div><div class=""><blockquote style="border-style: none none none solid; border-left-width: 1pt; border-left-color: rgb(204, 204, 204); padding: 0cm 0cm 0cm 6pt; margin: 5pt 0cm 5pt 4.8pt;" class=""><div class=""><div class=""><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class="">When comparing the MatSolve data for<o:p class=""></o:p></div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> <o:p class=""></o:p></div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class="">GPU<o:p class=""></o:p></div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> <o:p class=""></o:p></div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class="">MatSolve 341 1.0 1.3009e+02 1.6 2.96e+11 1.0 0.0e+00 0.0e+00 0.0e+00 36 34 0 0 0 36 34 0 0 0 4536 4538 1 5.34e+02 0 0.00e+00 100<o:p class=""></o:p></div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class="">MatCUSPARSSolAnl 2 1.0 3.2338e+01 1.5 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 9 0 0 0 0 9 0 0 0 0 0 0 0 0.00e+00 0 0.00e+00 0<o:p class=""></o:p></div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> <o:p class=""></o:p></div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class="">and CPU<o:p class=""></o:p></div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> <o:p class=""></o:p></div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class="">MatSolve 352 1.0 1.3553e+02 1.0 1.02e+11 1.0 0.0e+00 0.0e+00 0.0e+00 35 34 0 0 0 35 34 0 0 0 4489<o:p class=""></o:p></div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> <o:p class=""></o:p></div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class="">the time spent is almost the same for this preconditioner. Look like MatCUSPARSSolAnl is called only<span class="Apple-converted-space"> </span><u class="">twice</u><span class="Apple-converted-space"> </span>(since I am running on two cores)<span class="Apple-converted-space"> </span><o:p class=""></o:p></div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> <o:p class=""></o:p></div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span style="" class="">mpirun -n 2 ./ex45 -log_summary -da_grid_x 511 -da_grid_y 511 -da_grid_z 511 -dm_mat_type mpiaijcusparse -dm_vec_type mpicuda -ksp_type cg -pc_type bjacobi -ksp_monitor</span><o:p class=""></o:p></div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> <o:p class=""></o:p></div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class="">So would it be fair to assume MatCUSPARSSolAnl is<span class="Apple-converted-space"> </span><u class="">not<span class="Apple-converted-space"> </span></u>accounted for in MatSolve and it is an exclusive event?<o:p class=""></o:p></div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class="">KSPSolve (53%) + PCSetup (16%) + DMCreateMat (23%) + MatCUSPARSSolAnl (9%) ~ 100 %<o:p class=""></o:p></div></div></div></blockquote><div class=""><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> <o:p class=""></o:p></div></div><div class=""><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class="">I am getting so old. We have a different kind of log output if you are really concerned about inclusion. You can run with<o:p class=""></o:p></div></div><div class=""><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> <o:p class=""></o:p></div></div><div class=""><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> <span style="font-size: 10.5pt; font-family: Consolas; background-color: rgb(250, 250, 250);" class="">-log_view :foo.txt:ascii_flamegraph</span><o:p class=""></o:p></div></div><div class=""><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> <o:p class=""></o:p></div></div><div class=""><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class="">and then there are tools for plotting that output, described here<o:p class=""></o:p></div></div><div class=""><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> <o:p class=""></o:p></div></div><div class=""><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> <a href="https://firedrakeproject.org/optimising.html" target="_blank" style="color: blue; text-decoration: underline;" class="">https://firedrakeproject.org/optimising.html</a><o:p class=""></o:p></div></div><div class=""><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> <o:p class=""></o:p></div></div><div class=""><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class="">This output _guarantees_ strict inclusion, so you will not have the problems you have above adding things up.<o:p class=""></o:p></div></div><div class=""><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> <o:p class=""></o:p></div></div><div class=""><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> Thanks,<o:p class=""></o:p></div></div><div class=""><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> <o:p class=""></o:p></div></div><div class=""><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> Matt<o:p class=""></o:p></div></div><div class=""><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> <o:p class=""></o:p></div></div><blockquote style="border-style: none none none solid; border-left-width: 1pt; border-left-color: rgb(204, 204, 204); padding: 0cm 0cm 0cm 6pt; margin: 5pt 0cm 5pt 4.8pt;" class=""><div class=""><div class=""><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class="">Best,<o:p class=""></o:p></div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class="">Karthik.<o:p class=""></o:p></div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> <o:p class=""></o:p></div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> <o:p class=""></o:p></div><div style="border-style: solid none none; border-top-width: 1pt; border-top-color: rgb(181, 196, 223); padding: 3pt 0cm 0cm;" class=""><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><b class=""><span style="font-size: 12pt;" class="">From:<span class="Apple-converted-space"> </span></span></b><span style="font-size: 12pt;" class="">Matthew Knepley <<a href="mailto:knepley@gmail.com" target="_blank" style="color: blue; text-decoration: underline;" class="">knepley@gmail.com</a>><br class=""><b class="">Date:<span class="Apple-converted-space"> </span></b>Wednesday, 29 September 2021 at 16:29<br class=""><b class="">To:<span class="Apple-converted-space"> </span></b>"Chockalingam, Karthikeyan (STFC,DL,HC)" <<a href="mailto:karthikeyan.chockalingam@stfc.ac.uk" target="_blank" style="color: blue; text-decoration: underline;" class="">karthikeyan.chockalingam@stfc.ac.uk</a>><br class=""><b class="">Cc:<span class="Apple-converted-space"> </span></b>Barry Smith <<a href="mailto:bsmith@petsc.dev" target="_blank" style="color: blue; text-decoration: underline;" class="">bsmith@petsc.dev</a>>, "<a href="mailto:petsc-users@mcs.anl.gov" target="_blank" style="color: blue; text-decoration: underline;" class="">petsc-users@mcs.anl.gov</a>" <<a href="mailto:petsc-users@mcs.anl.gov" target="_blank" style="color: blue; text-decoration: underline;" class="">petsc-users@mcs.anl.gov</a>><br class=""><b class="">Subject:<span class="Apple-converted-space"> </span></b>Re: [petsc-users] %T (percent time in this phase)</span><o:p class=""></o:p></div></div><div class=""><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> <o:p class=""></o:p></div></div><div class=""><div class=""><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class="">On Wed, Sep 29, 2021 at 10:18 AM Karthikeyan Chockalingam - STFC UKRI <<a href="mailto:karthikeyan.chockalingam@stfc.ac.uk" target="_blank" style="color: blue; text-decoration: underline;" class="">karthikeyan.chockalingam@stfc.ac.uk</a>> wrote:<o:p class=""></o:p></div></div><div class=""><blockquote style="border-style: none none none solid; border-left-width: 1pt; border-left-color: rgb(204, 204, 204); padding: 0cm 0cm 0cm 6pt; margin: 5pt 0cm 5pt 4.8pt;" class=""><div class=""><div class=""><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class="">Thank you!<o:p class=""></o:p></div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> <o:p class=""></o:p></div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class="">Just to summarize<o:p class=""></o:p></div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> <o:p class=""></o:p></div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class="">KSPSolve (53%) + PCSetup (16%) + DMCreateMat (23%) + MatCUSPARSSolAnl (9%) ~ 100 %<o:p class=""></o:p></div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> <o:p class=""></o:p></div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class="">You didn’t happen to mention how MatCUSPARSSolAnl is accounted for? Am I right in accounting for it as above?<o:p class=""></o:p></div></div></div></blockquote><div class=""><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> <o:p class=""></o:p></div></div><div class=""><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class="">I am not sure.I thought it might be the GPU part of MatSolve(). I will have to look in the code. I am not as familiar with the GPU part.<o:p class=""></o:p></div></div><div class=""><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> <o:p class=""></o:p></div></div><blockquote style="border-style: none none none solid; border-left-width: 1pt; border-left-color: rgb(204, 204, 204); padding: 0cm 0cm 0cm 6pt; margin: 5pt 0cm 5pt 4.8pt;" class=""><div class=""><div class=""><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class="">MatCUSPARSSolAnl 2 1.0 3.2338e+01 1.5 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 9 0 0 0 0 9 0 0 0 0 0 0 0 0.00e+00 0 0.00e+00 0<o:p class=""></o:p></div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> <o:p class=""></o:p></div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class="">Finally, I believe the vector events, VecNorn, VecTDot, VecAXPY, and VecAYPX are mutually exclusive?<o:p class=""></o:p></div></div></div></blockquote><div class=""><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> <o:p class=""></o:p></div></div><div class=""><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class="">Yes.<o:p class=""></o:p></div></div><div class=""><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> <o:p class=""></o:p></div></div><div class=""><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> Thanks,<o:p class=""></o:p></div></div><div class=""><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> <o:p class=""></o:p></div></div><div class=""><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> Matt<o:p class=""></o:p></div></div><div class=""><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> <o:p class=""></o:p></div></div><blockquote style="border-style: none none none solid; border-left-width: 1pt; border-left-color: rgb(204, 204, 204); padding: 0cm 0cm 0cm 6pt; margin: 5pt 0cm 5pt 4.8pt;" class=""><div class=""><div class=""><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class="">Best,<o:p class=""></o:p></div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> <o:p class=""></o:p></div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class="">Karthik.<o:p class=""></o:p></div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> <o:p class=""></o:p></div><div style="border-style: solid none none; border-top-width: 1pt; border-top-color: rgb(181, 196, 223); padding: 3pt 0cm 0cm;" class=""><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><b class=""><span style="font-size: 12pt;" class="">From:<span class="Apple-converted-space"> </span></span></b><span style="font-size: 12pt;" class="">Matthew Knepley <<a href="mailto:knepley@gmail.com" target="_blank" style="color: blue; text-decoration: underline;" class="">knepley@gmail.com</a>><br class=""><b class="">Date:<span class="Apple-converted-space"> </span></b>Wednesday, 29 September 2021 at 11:58<br class=""><b class="">To:<span class="Apple-converted-space"> </span></b>"Chockalingam, Karthikeyan (STFC,DL,HC)" <<a href="mailto:karthikeyan.chockalingam@stfc.ac.uk" target="_blank" style="color: blue; text-decoration: underline;" class="">karthikeyan.chockalingam@stfc.ac.uk</a>><br class=""><b class="">Cc:<span class="Apple-converted-space"> </span></b>Barry Smith <<a href="mailto:bsmith@petsc.dev" target="_blank" style="color: blue; text-decoration: underline;" class="">bsmith@petsc.dev</a>>, "<a href="mailto:petsc-users@mcs.anl.gov" target="_blank" style="color: blue; text-decoration: underline;" class="">petsc-users@mcs.anl.gov</a>" <<a href="mailto:petsc-users@mcs.anl.gov" target="_blank" style="color: blue; text-decoration: underline;" class="">petsc-users@mcs.anl.gov</a>><br class=""><b class="">Subject:<span class="Apple-converted-space"> </span></b>Re: [petsc-users] %T (percent time in this phase)</span><o:p class=""></o:p></div></div><div class=""><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> <o:p class=""></o:p></div></div><div class=""><div class=""><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class="">On Wed, Sep 29, 2021 at 6:24 AM Karthikeyan Chockalingam - STFC UKRI <<a href="mailto:karthikeyan.chockalingam@stfc.ac.uk" target="_blank" style="color: blue; text-decoration: underline;" class="">karthikeyan.chockalingam@stfc.ac.uk</a>> wrote:<o:p class=""></o:p></div></div><div class=""><blockquote style="border-style: none none none solid; border-left-width: 1pt; border-left-color: rgb(204, 204, 204); padding: 0cm 0cm 0cm 6pt; margin: 5pt 0cm 5pt 4.8pt;" class=""><div class=""><div class=""><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class="">Thank you Mathew. Now, it is all making sense to me.<o:p class=""></o:p></div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> <o:p class=""></o:p></div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class="">From data file ksp_ex45_N511_gpu_2.txt<o:p class=""></o:p></div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> <o:p class=""></o:p></div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class="">KSPSolve (53%) + KSPSetup (0%) = PCSetup (16%) + PCApply (37%).<o:p class=""></o:p></div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> <o:p class=""></o:p></div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class="">However, you said “So an iteration would mostly consist of MatMult + PCApply, with some vector work”<o:p class=""></o:p></div></div></div></blockquote><div class=""><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> <o:p class=""></o:p></div></div><div class=""><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class="">1) You do one solve, but 2 KSPSetUp()s. You must be running on more than one process and using Block-Jacobi . Half the time is spent in the solve (53%)<o:p class=""></o:p></div></div><div class=""><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> <o:p class=""></o:p></div></div><div class=""><pre style="margin: 0cm 0cm 0.0001pt; font-size: 10pt; font-family: "Courier New"; white-space: pre-wrap;" class=""><span style="font-size: 10.5pt;" class="">KSPSetUp 2 1.0 5.3149e-01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 2.4e+01 0 0 0 0 1 0 0 0 0 1 0 0 0 0.00e+00 0 0.00e+00 0</span><o:p class=""></o:p></pre><pre style="margin: 0cm 0cm 0.0001pt; font-size: 10pt; font-family: "Courier New";" class=""><span style="font-size: 10.5pt;" class="">KSPSolve 1 1.0 1.5837e+02 1.1 8.63e+11 1.0 6.8e+02 2.1e+06 4.4e+03 53100100100 95 53100100100 96 10881 11730 1022 6.40e+03 1021 8.17e-03 100</span><o:p class=""></o:p></pre><div class=""><p class="MsoNormal" style="margin: 0cm 0cm 12pt; font-size: 11pt; font-family: Calibri, sans-serif;"> <o:p class=""></o:p></p></div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class="">2) The preconditioner look like BJacobi-ILU. The setup time is 16%, which is all setup of the individual blocks, and this is all used by the numerical ILU factorization.<o:p class=""></o:p></div></div><div class=""><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> <o:p class=""></o:p></div><div class=""><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span style="font-size: 10.5pt; font-family: "Courier New";" class="">PCSetUp 2 1.0 4.9623e+01 1.3 1.45e+09 1.0 0.0e+00 0.0e+00 0.0e+00 16 0 0 0 0 16 0 0 0 0 58 0 2 6.93e+03 0 0.00e+00 0 PCSetUpOnBlocks 1 1.0 4.9274e+01 1.3 1.45e+09 1.0 0.0e+00 0.0e+00 0.0e+00 15 0 0 0 0 15 0 0 0 0 59 0 2 6.93e+03 0 0.00e+00 0</span><o:p class=""></o:p></div><pre style="margin: 0cm 0cm 0.0001pt; font-size: 10pt; font-family: "Courier New";" class=""><span style="font-size: 10.5pt;" class="">MatLUFactorNum 1 1.0 4.6126e+01 1.3 1.45e+09 1.0 0.0e+00 0.0e+00 0.0e+00 14 0 0 0 0 14 0 0 0 0 63 0 2 6.93e+03 0 0.00e+00 0</span><o:p class=""></o:p></pre><pre style="margin: 0cm 0cm 0.0001pt; font-size: 10pt; font-family: "Courier New";" class=""><span style="font-size: 10.5pt;" class="">MatILUFactorSym 1 1.0 2.5110e+00 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 1 0 0 0 0 1 0 0 0 0 0 0 0 0.00e+00 0 0.00e+00 0</span><o:p class=""></o:p></pre><div class=""><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> <o:p class=""></o:p></div></div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class="">3) The preconditioner application takes 37% of the time, which is all solving the factors and recorded in MatSolve(). Matrix multiplication takes 4%.<o:p class=""></o:p></div></div><div class=""><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> <o:p class=""></o:p></div></div><div class=""><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span style="font-size: 10.5pt; font-family: "Courier New";" class="">PCApply 341 1.0 1.3068e+02 1.6 2.96e+11 1.0 0.0e+00 0.0e+00 0.0e+00 37 34 0 0 0 37 34 0 0 0 4516 4523 1 5.34e+02 0 0.00e+00 100</span><o:p class=""></o:p></div></div><pre style="margin: 0cm 0cm 0.0001pt; font-size: 10pt; font-family: "Courier New"; white-space: pre-wrap;" class=""><span style="font-size: 10.5pt;" class="">MatSolve 341 1.0 1.3009e+02 1.6 2.96e+11 1.0 0.0e+00 0.0e+00 0.0e+00 36 34 0 0 0 36 34 0 0 0 4536 4538 1 5.34e+02 0 0.00e+00 100</span><o:p class=""></o:p></pre><pre style="margin: 0cm 0cm 0.0001pt; font-size: 10pt; font-family: "Courier New"; white-space: pre-wrap;" class=""><span style="font-size: 10.5pt;" class="">MatMult 341 1.0 1.0774e+01 1.1 2.96e+11 1.0 6.9e+02 2.1e+06 2.0e+00 4 34100100 0 4 34100100 0 54801 66441 2 5.86e+03 0 0.00e+00 100</span><o:p class=""></o:p></pre><pre style="margin: 0cm 0cm 0.0001pt; font-size: 10pt; font-family: "Courier New";" class=""><span style="font-size: 10.5pt;" class=""> </span><o:p class=""></o:p></pre><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class="">4) The significant vector time is all in norms (11%) since they are really slow on the GPU.<o:p class=""></o:p></div></div><div class=""><p class="MsoNormal" style="margin: 0cm 0cm 12pt; font-size: 11pt; font-family: Calibri, sans-serif;"> <o:p class=""></o:p></p><pre style="margin: 0cm 0cm 0.0001pt; font-size: 10pt; font-family: "Courier New";" class=""><span style="font-size: 10.5pt;" class="">VecNorm 342 1.0 6.2261e+01129.9 4.57e+10 1.0 0.0e+00 0.0e+00 6.8e+02 11 5 0 0 15 11 5 0 0 15 1466 196884 0 0.00e+00 342 2.74e-03 100</span><o:p class=""></o:p></pre><pre style="margin: 0cm 0cm 0.0001pt; font-size: 10pt; font-family: "Courier New"; white-space: pre-wrap;" class=""><span style="font-size: 10.5pt;" class="">VecTDot 680 1.0 1.7107e+00 1.3 9.09e+10 1.0 0.0e+00 0.0e+00 1.4e+03 1 10 0 0 29 1 10 0 0 29 106079 133922 0 0.00e+00 680 5.44e-03 100</span><o:p class=""></o:p></pre><pre style="margin: 0cm 0cm 0.0001pt; font-size: 10pt; font-family: "Courier New";" class=""><span style="font-size: 10.5pt;" class="">VecAXPY 681 1.0 3.2036e+00 1.7 9.10e+10 1.0 0.0e+00 0.0e+00 0.0e+00 1 11 0 0 0 1 11 0 0 0 56728 58367 682 5.34e+02 0 0.00e+00 100</span><o:p class=""></o:p></pre><pre style="margin: 0cm 0cm 0.0001pt; font-size: 10pt; font-family: "Courier New";" class=""><span style="font-size: 10.5pt;" class="">VecAYPX 339 1.0 2.6502e+00 1.8 4.53e+10 1.0 0.0e+00 0.0e+00 0.0e+00 1 5 0 0 0 1 5 0 0 0 34136 34153 339 2.71e-03 0 0.00e+00 100</span><o:p class=""></o:p></pre><pre style="margin: 0cm 0cm 0.0001pt; font-size: 10pt; font-family: "Courier New"; white-space: pre-wrap;" class=""><span style="font-size: 10.5pt;" class=""> </span><o:p class=""></o:p></pre><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class="">So the solve time is:<o:p class=""></o:p></div></div><div class=""><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> <o:p class=""></o:p></div></div><div class=""><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> 53% ~ 37% + 4% + 11%<o:p class=""></o:p></div></div><div class=""><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> <o:p class=""></o:p></div></div><div class=""><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class="">and the setup time is about 16%. I was wrong about the SetUp time being included, as it is outside the event:<o:p class=""></o:p></div></div><div class=""><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> <o:p class=""></o:p></div></div><div class=""><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> <a href="https://gitlab.com/petsc/petsc/-/blob/main/src/ksp/ksp/interface/itfunc.c#L852" target="_blank" style="color: blue; text-decoration: underline;" class="">https://gitlab.com/petsc/petsc/-/blob/main/src/ksp/ksp/interface/itfunc.c#L852</a><o:p class=""></o:p></div></div><div class=""><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> <o:p class=""></o:p></div></div><div class=""><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class="">It looks like the remainder of the time (23%) is spent preallocating the matrix.<o:p class=""></o:p></div></div><div class=""><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> <o:p class=""></o:p></div></div><div class=""><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> Thanks,<o:p class=""></o:p></div></div><div class=""><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> <o:p class=""></o:p></div></div><div class=""><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> Matt<o:p class=""></o:p></div></div><div class=""><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> <o:p class=""></o:p></div></div><blockquote style="border-style: none none none solid; border-left-width: 1pt; border-left-color: rgb(204, 204, 204); padding: 0cm 0cm 0cm 6pt; margin: 5pt 0cm 5pt 4.8pt;" class=""><div class=""><div class=""><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class="">The MalMult event is 4 %. How does this event figure into the above equation; if preconditioning (MatMult + PCApply) is included in KSPSolve?<o:p class=""></o:p></div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> <o:p class=""></o:p></div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class="">Best,<o:p class=""></o:p></div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class="">Karthik.<o:p class=""></o:p></div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> <o:p class=""></o:p></div><div style="border-style: solid none none; border-top-width: 1pt; border-top-color: rgb(181, 196, 223); padding: 3pt 0cm 0cm;" class=""><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><b class=""><span style="font-size: 12pt;" class="">From:<span class="Apple-converted-space"> </span></span></b><span style="font-size: 12pt;" class="">Matthew Knepley <<a href="mailto:knepley@gmail.com" target="_blank" style="color: blue; text-decoration: underline;" class="">knepley@gmail.com</a>><br class=""><b class="">Date:<span class="Apple-converted-space"> </span></b>Wednesday, 29 September 2021 at 10:58<br class=""><b class="">To:<span class="Apple-converted-space"> </span></b>"Chockalingam, Karthikeyan (STFC,DL,HC)" <<a href="mailto:karthikeyan.chockalingam@stfc.ac.uk" target="_blank" style="color: blue; text-decoration: underline;" class="">karthikeyan.chockalingam@stfc.ac.uk</a>><br class=""><b class="">Cc:<span class="Apple-converted-space"> </span></b>Barry Smith <<a href="mailto:bsmith@petsc.dev" target="_blank" style="color: blue; text-decoration: underline;" class="">bsmith@petsc.dev</a>>, "<a href="mailto:petsc-users@mcs.anl.gov" target="_blank" style="color: blue; text-decoration: underline;" class="">petsc-users@mcs.anl.gov</a>" <<a href="mailto:petsc-users@mcs.anl.gov" target="_blank" style="color: blue; text-decoration: underline;" class="">petsc-users@mcs.anl.gov</a>><br class=""><b class="">Subject:<span class="Apple-converted-space"> </span></b>Re: [petsc-users] %T (percent time in this phase)</span><o:p class=""></o:p></div></div><div class=""><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> <o:p class=""></o:p></div></div><div class=""><div class=""><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class="">On Wed, Sep 29, 2021 at 5:52 AM Karthikeyan Chockalingam - STFC UKRI <<a href="mailto:karthikeyan.chockalingam@stfc.ac.uk" target="_blank" style="color: blue; text-decoration: underline;" class="">karthikeyan.chockalingam@stfc.ac.uk</a>> wrote:<o:p class=""></o:p></div></div><div class=""><blockquote style="border-style: none none none solid; border-left-width: 1pt; border-left-color: rgb(204, 204, 204); padding: 0cm 0cm 0cm 6pt; margin: 5pt 0cm 5pt 4.8pt;" class=""><div class=""><div class=""><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class="">That was helpful. I would like to provide some additional details of my run on cpus and gpus. Please find the following attachments:<o:p class=""></o:p></div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> <o:p class=""></o:p></div><ol start="1" type="1" style="margin-bottom: 0cm;" class=""><li class="gmail-m-9161537463986845899gmail-m-5797767908442767276gmail-m-3492767196240074348gmail-m391186430279648065gmail-m-3409338445930824232msolistparagraph" style="margin-right: 0cm; margin-left: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;">graph.pdf a plot showing overall time and various petsc events.<o:p class=""></o:p></li><li class="gmail-m-9161537463986845899gmail-m-5797767908442767276gmail-m-3492767196240074348gmail-m391186430279648065gmail-m-3409338445930824232msolistparagraph" style="margin-right: 0cm; margin-left: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;">ksp_ex45_N511_cpu_6.txt data file of the log_summary<span class="Apple-converted-space"> </span><o:p class=""></o:p></li><li class="gmail-m-9161537463986845899gmail-m-5797767908442767276gmail-m-3492767196240074348gmail-m391186430279648065gmail-m-3409338445930824232msolistparagraph" style="margin-right: 0cm; margin-left: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;">ksp_ex45_N511_gpu_2.txt data file of the log_summary<span class="Apple-converted-space"> </span><o:p class=""></o:p></li></ol><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> <o:p class=""></o:p></div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class="">I used the following petsc options for cpu<o:p class=""></o:p></div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span style="" class=""> </span><o:p class=""></o:p></div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span style="" class="">mpirun -n 6 ./ex45 -log_summary -da_grid_x 511 -da_grid_y 511 -da_grid_z 511 -dm_mat_type mpiaij -dm_vec_type mpi -ksp_type cg -pc_type bjacobi -ksp_monitor</span><o:p class=""></o:p></div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span style="" class=""> </span><o:p class=""></o:p></div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span style="" class="">and for gpus</span><o:p class=""></o:p></div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span style="" class=""> </span><o:p class=""></o:p></div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span style="" class="">mpirun -n 1 ./ex45 -log_summary -da_grid_x 511 -da_grid_y 511 -da_grid_z 511 -dm_mat_type mpiaijcusparse -dm_vec_type mpicuda -ksp_type cg -pc_type bjacobi -ksp_monitor</span><o:p class=""></o:p></div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span style="" class=""> </span><o:p class=""></o:p></div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span style="" class="">to run the following problem</span><o:p class=""></o:p></div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span style="" class=""> </span><o:p class=""></o:p></div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span style="font-size: 12pt;" class=""><a href="https://petsc.org/release/src/ksp/ksp/tutorials/ex45.c.html" target="_blank" style="color: blue; text-decoration: underline;" class="">https://petsc.org/release/src/ksp/ksp/tutorials/ex45.c.html</a></span><o:p class=""></o:p></div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span style="font-size: 12pt;" class=""> </span><o:p class=""></o:p></div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span style="font-size: 12pt;" class="">From the above code, I see is there no individual function called<span class="Apple-converted-space"> </span></span>KSPSetUp(), so I gather KSPSetDM, KSPSetComputeInitialGuess, KSPSetComputeRHS, kSPSetComputeOperators all are timed together as KSPSetUp.<span style="font-size: 12pt;" class=""><span class="Apple-converted-space"> </span>For this example, is KSPSetUp time and KSPSolve time mutually exclusive?</span><o:p class=""></o:p></div></div></div></blockquote><div class=""><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> <o:p class=""></o:p></div></div><div class=""><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class="">No, KSPSetUp() will be contained in KSPSolve() if it is called automatically.<o:p class=""></o:p></div></div><div class=""><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> <o:p class=""></o:p></div></div><blockquote style="border-style: none none none solid; border-left-width: 1pt; border-left-color: rgb(204, 204, 204); padding: 0cm 0cm 0cm 6pt; margin: 5pt 0cm 5pt 4.8pt;" class=""><div class=""><div class=""><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span style="font-size: 12pt;" class="">In your response you said that</span><o:p class=""></o:p></div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span style="font-size: 12pt;" class=""> </span><o:p class=""></o:p></div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> “PCSetUp() time may be in KSPSetUp() or it maybe in PCApply() it depends on how much of the preconditioner construction can take place early, so depends exactly on the preconditioner used.”<o:p class=""></o:p></div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span style="font-size: 12pt;" class=""> </span><o:p class=""></o:p></div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span style="font-size: 12pt;" class="">I don’t see a explicit call to PCSetUp() or PCApply() in ex45; so for this particular preconditioner (bjacobi) how can I tell how they are timed?</span><o:p class=""></o:p></div></div></div></blockquote><div class=""><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> <o:p class=""></o:p></div></div><div class=""><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class="">They are all inside KSPSolve(). If you have a preconditioned linear solve, the oreconditioning happens during the iteration. So an iteration would mostly<o:p class=""></o:p></div></div><div class=""><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class="">consist of MatMult + PCApply, with some vector work.<o:p class=""></o:p></div></div><div class=""><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> <o:p class=""></o:p></div></div><blockquote style="border-style: none none none solid; border-left-width: 1pt; border-left-color: rgb(204, 204, 204); padding: 0cm 0cm 0cm 6pt; margin: 5pt 0cm 5pt 4.8pt;" class=""><div class=""><div class=""><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span style="font-size: 12pt;" class="">I am hoping to time KSP solving and preconditioning mutually exclusively.</span><o:p class=""></o:p></div></div></div></blockquote><div class=""><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> <o:p class=""></o:p></div></div><div class=""><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class="">I am not sure that concept makes sense here. See above.<o:p class=""></o:p></div></div><div class=""><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> <o:p class=""></o:p></div></div><div class=""><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> Thanks,<o:p class=""></o:p></div></div><div class=""><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> <o:p class=""></o:p></div></div><div class=""><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> Matt<o:p class=""></o:p></div></div><div class=""><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> <o:p class=""></o:p></div></div><blockquote style="border-style: none none none solid; border-left-width: 1pt; border-left-color: rgb(204, 204, 204); padding: 0cm 0cm 0cm 6pt; margin: 5pt 0cm 5pt 4.8pt;" class=""><div class=""><div class=""><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span style="font-size: 12pt;" class=""> </span><o:p class=""></o:p></div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span style="font-size: 12pt;" class="">Kind regards,</span><o:p class=""></o:p></div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span style="font-size: 12pt;" class="">Karthik.</span><o:p class=""></o:p></div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span style="font-size: 12pt;" class=""> </span><o:p class=""></o:p></div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> <o:p class=""></o:p></div><div style="border-style: solid none none; border-top-width: 1pt; border-top-color: rgb(181, 196, 223); padding: 3pt 0cm 0cm;" class=""><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><b class=""><span style="font-size: 12pt;" class="">From:<span class="Apple-converted-space"> </span></span></b><span style="font-size: 12pt;" class="">Barry Smith <<a href="mailto:bsmith@petsc.dev" target="_blank" style="color: blue; text-decoration: underline;" class="">bsmith@petsc.dev</a>><br class=""><b class="">Date:<span class="Apple-converted-space"> </span></b>Tuesday, 28 September 2021 at 19:19<br class=""><b class="">To:<span class="Apple-converted-space"> </span></b>"Chockalingam, Karthikeyan (STFC,DL,HC)" <<a href="mailto:karthikeyan.chockalingam@stfc.ac.uk" target="_blank" style="color: blue; text-decoration: underline;" class="">karthikeyan.chockalingam@stfc.ac.uk</a>><br class=""><b class="">Cc:<span class="Apple-converted-space"> </span></b>"<a href="mailto:petsc-users@mcs.anl.gov" target="_blank" style="color: blue; text-decoration: underline;" class="">petsc-users@mcs.anl.gov</a>" <<a href="mailto:petsc-users@mcs.anl.gov" target="_blank" style="color: blue; text-decoration: underline;" class="">petsc-users@mcs.anl.gov</a>><br class=""><b class="">Subject:<span class="Apple-converted-space"> </span></b>Re: [petsc-users] %T (percent time in this phase)</span><o:p class=""></o:p></div></div><div class=""><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> <o:p class=""></o:p></div></div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> <o:p class=""></o:p></div><div class=""><p class="MsoNormal" style="margin: 0cm 0cm 12pt; font-size: 11pt; font-family: Calibri, sans-serif;"> <o:p class=""></o:p></p><blockquote style="margin-top: 5pt; margin-bottom: 5pt;" class=""><div class=""><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class="">On Sep 28, 2021, at 12:11 PM, Karthikeyan Chockalingam - STFC UKRI <<a href="mailto:karthikeyan.chockalingam@stfc.ac.uk" target="_blank" style="color: blue; text-decoration: underline;" class="">karthikeyan.chockalingam@stfc.ac.uk</a>> wrote:<o:p class=""></o:p></div></div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> <o:p class=""></o:p></div><div class=""><div class=""><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class="">Thanks for Barry for your response.<o:p class=""></o:p></div></div><div class=""><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> <o:p class=""></o:p></div></div><div class=""><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class="">I was just benchmarking the problem with various preconditioner on cpu and gpu. I understand, it is not possible to get mutually exclusive timing.<o:p class=""></o:p></div></div><div class=""><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class="">However, can you tell if KSPSolve time includes both PCSetup and PCApply? And if KSPSolve and KSPSetup are mutually exclusive? Likewise for PCSetUp and PCApply.<o:p class=""></o:p></div></div></div></blockquote><div class=""><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> <o:p class=""></o:p></div></div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> If you do not call KSPSetUp() separately from KSPSolve() then its time is included with KSPSolve().<o:p class=""></o:p></div></div><div class=""><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> <o:p class=""></o:p></div></div><div class=""><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> PCSetUp() time may be in KSPSetUp() or it maybe in PCApply() it depends on how much of the preconditioner construction can take place early, so depends exactly on the preconditioner used.<o:p class=""></o:p></div></div><div class=""><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> <o:p class=""></o:p></div></div><div class=""><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> So yes the answer is not totally satisfying. The one thing I would recommend is to not call KSPSetUp() directly and then KSPSolve() will always include the total time of the solve plus all setup time. PCApply will contain all the time to apply the preconditioner but may also include some setup time.<o:p class=""></o:p></div></div><div class=""><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> <o:p class=""></o:p></div></div><div class=""><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> Barry<o:p class=""></o:p></div></div><div class=""><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> <o:p class=""></o:p></div></div><div class=""><blockquote style="margin-top: 5pt; margin-bottom: 5pt;" class=""><div class=""><div class=""><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> <o:p class=""></o:p></div></div><div class=""><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class="">Best,<o:p class=""></o:p></div></div><div class=""><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class="">Karthik.<o:p class=""></o:p></div></div><div class=""><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> <o:p class=""></o:p></div></div><div class=""><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> <o:p class=""></o:p></div></div><div class=""><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> <o:p class=""></o:p></div></div><div class=""><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> <o:p class=""></o:p></div></div><div style="border-style: solid none none; border-top-width: 1pt; border-top-color: rgb(181, 196, 223); padding: 3pt 0cm 0cm;" class=""><div class=""><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><b class=""><span style="font-size: 12pt;" class="">From:<span class="gmail-m-9161537463986845899gmail-m-5797767908442767276gmail-m-3492767196240074348gmail-m391186430279648065gmail-m-3409338445930824232apple-converted-space"> </span></span></b><span style="font-size: 12pt;" class="">Barry Smith <<a href="mailto:bsmith@petsc.dev" target="_blank" style="color: blue; text-decoration: underline;" class="">bsmith@petsc.dev</a>><br class=""><b class="">Date:<span class="gmail-m-9161537463986845899gmail-m-5797767908442767276gmail-m-3492767196240074348gmail-m391186430279648065gmail-m-3409338445930824232apple-converted-space"> </span></b>Tuesday, 28 September 2021 at 16:56<br class=""><b class="">To:<span class="gmail-m-9161537463986845899gmail-m-5797767908442767276gmail-m-3492767196240074348gmail-m391186430279648065gmail-m-3409338445930824232apple-converted-space"> </span></b>"Chockalingam, Karthikeyan (STFC,DL,HC)" <<a href="mailto:karthikeyan.chockalingam@stfc.ac.uk" target="_blank" style="color: blue; text-decoration: underline;" class="">karthikeyan.chockalingam@stfc.ac.uk</a>><br class=""><b class="">Cc:<span class="gmail-m-9161537463986845899gmail-m-5797767908442767276gmail-m-3492767196240074348gmail-m391186430279648065gmail-m-3409338445930824232apple-converted-space"> </span></b>"<a href="mailto:petsc-users@mcs.anl.gov" target="_blank" style="color: blue; text-decoration: underline;" class="">petsc-users@mcs.anl.gov</a>" <<a href="mailto:petsc-users@mcs.anl.gov" target="_blank" style="color: blue; text-decoration: underline;" class="">petsc-users@mcs.anl.gov</a>><br class=""><b class="">Subject:<span class="gmail-m-9161537463986845899gmail-m-5797767908442767276gmail-m-3492767196240074348gmail-m391186430279648065gmail-m-3409338445930824232apple-converted-space"> </span></b>Re: [petsc-users] %T (percent time in this phase)</span><o:p class=""></o:p></div></div></div><div class=""><div class=""><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> <o:p class=""></o:p></div></div></div><div class=""><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> <o:p class=""></o:p></div></div><div class=""><div class=""><p class="MsoNormal" style="margin: 0cm 0cm 12pt; font-size: 11pt; font-family: Calibri, sans-serif;"> <o:p class=""></o:p></p></div><blockquote style="margin-top: 5pt; margin-bottom: 5pt;" class=""><div class=""><div class=""><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class="">On Sep 28, 2021, at 10:55 AM, Karthikeyan Chockalingam - STFC UKRI <<a href="mailto:karthikeyan.chockalingam@stfc.ac.uk" target="_blank" style="color: blue; text-decoration: underline;" class="">karthikeyan.chockalingam@stfc.ac.uk</a>> wrote:<o:p class=""></o:p></div></div></div><div class=""><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> <o:p class=""></o:p></div></div><div class=""><div class=""><div class=""><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span style="font-size: 12pt;" class="">Hello,</span><o:p class=""></o:p></div></div></div><div class=""><div class=""><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span style="font-size: 12pt;" class=""> </span><o:p class=""></o:p></div></div></div><div class=""><div class=""><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span style="font-size: 12pt;" class="">I ran ex45 in the KPS tutorial, which is a 3D finite-difference Poisson problem. I noticed from the output from using the flag -log_summary that for various events their respective %T (percent time in this phase) do not add up to 100 but rather exceeds 100. So, I gather there is some overlap among these events. I am primarily looking at the events KSPSetUp, KSPSolve, PCSetUp and PCSolve. Is it possible to get a mutually exclusive %T or Time for these individual events? I have attached the log_summary output file from my run for your reference.</span><o:p class=""></o:p></div></div></div><div class=""><div class=""><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span style="font-size: 12pt;" class=""> </span><o:p class=""></o:p></div></div></div></div></blockquote><div class=""><div class=""><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> <o:p class=""></o:p></div></div></div><div class=""><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> For nested solvers it is tricky to get the times to be mutually exclusive because some parts of the building of the preconditioner is for some preconditioners delayed until the solve has started. <o:p class=""></o:p></div></div></div><div class=""><div class=""><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> <o:p class=""></o:p></div></div></div><div class=""><div class=""><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> It looks like you are using the default preconditioner options which for this example are taking more or less no time since so many iterations are needed. It is best to use -pc_type mg to use geometric multigrid on this problem.<o:p class=""></o:p></div></div></div><div class=""><div class=""><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> <o:p class=""></o:p></div></div></div><div class=""><div class=""><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> Barry<o:p class=""></o:p></div></div></div><div class=""><div class=""><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> <o:p class=""></o:p></div></div></div><div class=""><div class=""><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> <o:p class=""></o:p></div></div></div><div class=""><div class=""><p class="MsoNormal" style="margin: 0cm 0cm 12pt; font-size: 11pt; font-family: Calibri, sans-serif;"> <o:p class=""></o:p></p></div><blockquote style="margin-top: 5pt; margin-bottom: 5pt;" class=""><div class=""><div class=""><div class=""><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span style="font-size: 12pt;" class="">Thanks!</span><o:p class=""></o:p></div></div></div><div class=""><div class=""><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span style="font-size: 12pt;" class="">Karthik.</span><o:p class=""></o:p></div></div></div><div class=""><div class=""><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> <o:p class=""></o:p></div></div></div><div class=""><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span style="font-size: 6pt; font-family: Helvetica;" class="">This email and any attachments are intended solely for the use of the named recipients. If you are not the intended recipient you must not use, disclose, copy or distribute this email or any of its attachments and should notify the sender immediately and delete this email from your system. UK Research and Innovation (UKRI) has taken every reasonable precaution to minimise risk of this email or any attachments containing viruses or malware but the recipient should carry out its own virus and malware checks before opening the attachments. UKRI does not accept any liability for any losses or damages which the recipient may sustain due to presence of any viruses. </span><o:p class=""></o:p></div></div><div class=""><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><ksp_ex45_N511_cpu_6.txt><o:p class=""></o:p></div></div></div></blockquote></div></div></blockquote></div><p class="MsoNormal" style="margin: 0cm 0cm 12pt; font-size: 11pt; font-family: Calibri, sans-serif;"> <o:p class=""></o:p></p></div></div></blockquote></div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><br clear="all" class=""><o:p class=""></o:p></div><div class=""><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> <o:p class=""></o:p></div></div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class="">--<o:p class=""></o:p></div><div class=""><div class=""><div class=""><div class=""><div class=""><div class=""><div class=""><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class="">What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br class="">-- Norbert Wiener<o:p class=""></o:p></div></div><div class=""><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> <o:p class=""></o:p></div></div><div class=""><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><a href="http://www.cse.buffalo.edu/~knepley/" target="_blank" style="color: blue; text-decoration: underline;" class="">https://www.cse.buffalo.edu/~knepley/</a><o:p class=""></o:p></div></div></div></div></div></div></div></div></div></div></div></blockquote></div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><br clear="all" class=""><o:p class=""></o:p></div><div class=""><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> <o:p class=""></o:p></div></div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class="">--<o:p class=""></o:p></div><div class=""><div class=""><div class=""><div class=""><div class=""><div class=""><div class=""><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class="">What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br class="">-- Norbert Wiener<o:p class=""></o:p></div></div><div class=""><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> <o:p class=""></o:p></div></div><div class=""><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><a href="http://www.cse.buffalo.edu/~knepley/" target="_blank" style="color: blue; text-decoration: underline;" class="">https://www.cse.buffalo.edu/~knepley/</a><o:p class=""></o:p></div></div></div></div></div></div></div></div></div></div></div></blockquote></div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><br clear="all" class=""><o:p class=""></o:p></div><div class=""><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> <o:p class=""></o:p></div></div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class="">--<o:p class=""></o:p></div><div class=""><div class=""><div class=""><div class=""><div class=""><div class=""><div class=""><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class="">What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br class="">-- Norbert Wiener<o:p class=""></o:p></div></div><div class=""><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> <o:p class=""></o:p></div></div><div class=""><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><a href="http://www.cse.buffalo.edu/~knepley/" target="_blank" style="color: blue; text-decoration: underline;" class="">https://www.cse.buffalo.edu/~knepley/</a><o:p class=""></o:p></div></div></div></div></div></div></div></div></div></div></div></blockquote></div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><br clear="all" class=""><o:p class=""></o:p></div><div class=""><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> <o:p class=""></o:p></div></div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class="">--<o:p class=""></o:p></div><div class=""><div class=""><div class=""><div class=""><div class=""><div class=""><div class=""><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class="">What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br class="">-- Norbert Wiener<o:p class=""></o:p></div></div><div class=""><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> <o:p class=""></o:p></div></div><div class=""><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><a href="http://www.cse.buffalo.edu/~knepley/" target="_blank" style="color: blue; text-decoration: underline;" class="">https://www.cse.buffalo.edu/~knepley/</a><o:p class=""></o:p></div></div></div></div></div></div></div></div></div></div></div></blockquote></div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><br clear="all" class=""><o:p class=""></o:p></div><div class=""><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><o:p class=""> </o:p></div></div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class="">--<span class="Apple-converted-space"> </span><o:p class=""></o:p></div><div class=""><div class=""><div class=""><div class=""><div class=""><div class=""><div class=""><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class="">What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br class="">-- Norbert Wiener<o:p class=""></o:p></div></div><div class=""><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><o:p class=""> </o:p></div></div><div class=""><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><a href="http://www.cse.buffalo.edu/~knepley/" target="_blank" style="color: blue; text-decoration: underline;" class="">https://www.cse.buffalo.edu/~knepley/</a></div></div></div></div></div></div></div></div></div></div></div></blockquote></div><br class=""></div></body></html>