<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} </style>
</head>
<body dir="ltr">
<div style="font-family: Consolas, Courier, monospace; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);" class="elementToProof ContentPasted0">
------------------------------------------------------------------ PETSc Performance Summary: ------------------------------------------------------------------
<div><br class="ContentPasted0">
</div>
<div class="ContentPasted0">Unknown Name on a named PC0115427 with 1 processor, by 4pf Wed Nov 16 14:36:46 2022</div>
<div class="ContentPasted0">Using Petsc Development GIT revision: v3.18.1-115-gdca010e0e9a GIT Date: 2022-10-28 14:39:41 +0000</div>
<div><br class="ContentPasted0">
</div>
<div class="ContentPasted0"> Max Max/Min Avg Total</div>
<div class="ContentPasted0">Time (sec): 6.023e+00 1.000 6.023e+00</div>
<div class="ContentPasted0">Objects: 1.020e+02 1.000 1.020e+02</div>
<div class="ContentPasted0">Flops: 1.080e+09 1.000 1.080e+09 1.080e+09</div>
<div class="ContentPasted0">Flops/sec: 1.793e+08 1.000 1.793e+08 1.793e+08</div>
<div class="ContentPasted0">MPI Msg Count: 0.000e+00 0.000 0.000e+00 0.000e+00</div>
<div class="ContentPasted0">MPI Msg Len (bytes): 0.000e+00 0.000 0.000e+00 0.000e+00</div>
<div class="ContentPasted0">MPI Reductions: 0.000e+00 0.000</div>
<div><br class="ContentPasted0">
</div>
<div class="ContentPasted0">Flop counting convention: 1 flop = 1 real number operation of type (multiply/divide/add/subtract)</div>
<div class="ContentPasted0"> e.g., VecAXPY() for real vectors of length N --> 2N flops</div>
<div class="ContentPasted0"> and VecAXPY() for complex vectors of length N --> 8N flops</div>
<div><br class="ContentPasted0">
</div>
<div class="ContentPasted0">Summary of Stages: ----- Time ------ ----- Flop ------ --- Messages --- -- Message Lengths -- -- Reductions --</div>
<div class="ContentPasted0"> Avg %Total Avg %Total Count %Total Avg %Total Count %Total</div>
<div class="ContentPasted0"> 0: Main Stage: 6.0226e+00 100.0% 1.0799e+09 100.0% 0.000e+00 0.0% 0.000e+00 0.0% 0.000e+00 0.0%</div>
<div><br class="ContentPasted0">
</div>
<div class="ContentPasted0">------------------------------------------------------------------------------------------------------------------------</div>
<div class="ContentPasted0">See the 'Profiling' chapter of the users' manual for details on interpreting output.</div>
<div class="ContentPasted0">Phase summary info:</div>
<div class="ContentPasted0"> Count: number of times phase was executed</div>
<div class="ContentPasted0"> Time and Flop: Max - maximum over all processors</div>
<div class="ContentPasted0"> Ratio - ratio of maximum to minimum over all processors</div>
<div class="ContentPasted0"> Mess: number of messages sent</div>
<div class="ContentPasted0"> AvgLen: average message length (bytes)</div>
<div class="ContentPasted0"> Reduct: number of global reductions</div>
<div class="ContentPasted0"> Global: entire computation</div>
<div class="ContentPasted0"> Stage: stages of a computation. Set stages with PetscLogStagePush() and PetscLogStagePop().</div>
<div class="ContentPasted0"> %T - percent time in this phase %F - percent flop in this phase</div>
<div class="ContentPasted0"> %M - percent messages in this phase %L - percent message lengths in this phase</div>
<div class="ContentPasted0"> %R - percent reductions in this phase</div>
<div class="ContentPasted0"> Total Mflop/s: 10e-6 * (sum of flop over all processors)/(max time over all processors)</div>
<div class="ContentPasted0"> GPU Mflop/s: 10e-6 * (sum of flop on GPU over all processors)/(max GPU time over all processors)</div>
<div class="ContentPasted0"> CpuToGpu Count: total number of CPU to GPU copies per processor</div>
<div class="ContentPasted0"> CpuToGpu Size (Mbytes): 10e-6 * (total size of CPU to GPU copies per processor)</div>
<div class="ContentPasted0"> GpuToCpu Count: total number of GPU to CPU copies per processor</div>
<div class="ContentPasted0"> GpuToCpu Size (Mbytes): 10e-6 * (total size of GPU to CPU copies per processor)</div>
<div class="ContentPasted0"> GPU %F: percent flops on GPU in this event</div>
<div class="ContentPasted0">------------------------------------------------------------------------------------------------------------------------</div>
<div class="ContentPasted0">Event Count Time (sec) Flop --- Global --- --- Stage ---- Total</div>
<div class="ContentPasted0"> GPU - CpuToGpu - - GpuToCpu - GPU</div>
<div><br class="ContentPasted0">
</div>
<div class="ContentPasted0"> Max Ratio Max Ratio Max Ratio Mess AvgLen Reduct %T %F %M %L %R %T %F %M %L %R Mflop/s</div>
<div class="ContentPasted0"> Mflop/s Count Size Count Size %F</div>
<div><br class="ContentPasted0">
</div>
<div class="ContentPasted0">------------------------------------------------------------------------------------------------------------------------</div>
<div class="ContentPasted0">---------------------------------------</div>
<div><br class="ContentPasted0">
</div>
<div><br class="ContentPasted0">
</div>
<div class="ContentPasted0">--- Event Stage 0: Main Stage</div>
<div><br class="ContentPasted0">
</div>
<div class="ContentPasted0">BuildTwoSided 3 1.0 nan nan 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 -nan</div>
<div class="ContentPasted0"> -nan 0 0.00e+00 0 0.00e+00 0</div>
<div><br class="ContentPasted0">
</div>
<div class="ContentPasted0">DMCreateMat 1 1.0 nan nan 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 1 0 0 0 0 1 0 0 0 0 -nan</div>
<div class="ContentPasted0"> -nan 0 0.00e+00 0 0.00e+00 0</div>
<div><br class="ContentPasted0">
</div>
<div class="ContentPasted0">SFSetGraph 3 1.0 nan nan 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 -nan</div>
<div class="ContentPasted0"> -nan 0 0.00e+00 0 0.00e+00 0</div>
<div><br class="ContentPasted0">
</div>
<div class="ContentPasted0">SFSetUp 3 1.0 nan nan 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 -nan</div>
<div class="ContentPasted0"> -nan 0 0.00e+00 0 0.00e+00 0</div>
<div><br class="ContentPasted0">
</div>
<div class="ContentPasted0">SFPack 4647 1.0 nan nan 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 -nan</div>
<div class="ContentPasted0"> -nan 0 0.00e+00 0 0.00e+00 0</div>
<div><br class="ContentPasted0">
</div>
<div class="ContentPasted0">SFUnpack 4647 1.0 nan nan 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 -nan</div>
<div class="ContentPasted0"> -nan 0 0.00e+00 0 0.00e+00 0</div>
<div><br class="ContentPasted0">
</div>
<div class="ContentPasted0">VecDot 190 1.0 nan nan 2.11e+06 1.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 -nan</div>
<div class="ContentPasted0"> -nan 0 0.00e+00 0 0.00e+00 100</div>
<div><br class="ContentPasted0">
</div>
<div class="ContentPasted0">VecMDot 775 1.0 nan nan 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 -nan</div>
<div class="ContentPasted0"> -nan 0 0.00e+00 0 0.00e+00 0</div>
<div><br class="ContentPasted0">
</div>
<div class="ContentPasted0">VecNorm 1728 1.0 nan nan 1.92e+07 1.0 0.0e+00 0.0e+00 0.0e+00 0 2 0 0 0 0 2 0 0 0 -nan</div>
<div class="ContentPasted0"> -nan 0 0.00e+00 0 0.00e+00 100</div>
<div><br class="ContentPasted0">
</div>
<div class="ContentPasted0">VecScale 1983 1.0 nan nan 6.24e+06 1.0 0.0e+00 0.0e+00 0.0e+00 0 1 0 0 0 0 1 0 0 0 -nan</div>
<div class="ContentPasted0"> -nan 0 0.00e+00 0 0.00e+00 100</div>
<div><br class="ContentPasted0">
</div>
<div class="ContentPasted0">VecCopy 780 1.0 nan nan 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 -nan</div>
<div class="ContentPasted0"> -nan 0 0.00e+00 0 0.00e+00 0</div>
<div><br class="ContentPasted0">
</div>
<div class="ContentPasted0">VecSet 4955 1.0 nan nan 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 2 0 0 0 0 2 0 0 0 0 -nan</div>
<div class="ContentPasted0"> -nan 0 0.00e+00 0 0.00e+00 0</div>
<div><br class="ContentPasted0">
</div>
<div class="ContentPasted0">VecAXPY 190 1.0 nan nan 2.11e+06 1.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 -nan</div>
<div class="ContentPasted0"> -nan 0 0.00e+00 0 0.00e+00 100</div>
<div><br class="ContentPasted0">
</div>
<div class="ContentPasted0">VecAYPX 597 1.0 nan nan 6.64e+06 1.0 0.0e+00 0.0e+00 0.0e+00 0 1 0 0 0 0 1 0 0 0 -nan</div>
<div class="ContentPasted0"> -nan 0 0.00e+00 0 0.00e+00 100</div>
<div><br class="ContentPasted0">
</div>
<div class="ContentPasted0">VecAXPBYCZ 643 1.0 nan nan 1.79e+07 1.0 0.0e+00 0.0e+00 0.0e+00 0 2 0 0 0 0 2 0 0 0 -nan</div>
<div class="ContentPasted0"> -nan 0 0.00e+00 0 0.00e+00 100</div>
<div><br class="ContentPasted0">
</div>
<div class="ContentPasted0">VecWAXPY 502 1.0 nan nan 5.58e+06 1.0 0.0e+00 0.0e+00 0.0e+00 0 1 0 0 0 0 1 0 0 0 -nan</div>
<div class="ContentPasted0"> -nan 0 0.00e+00 0 0.00e+00 100</div>
<div><br class="ContentPasted0">
</div>
<div class="ContentPasted0">VecMAXPY 1159 1.0 nan nan 3.68e+07 1.0 0.0e+00 0.0e+00 0.0e+00 0 3 0 0 0 0 3 0 0 0 -nan</div>
<div class="ContentPasted0"> -nan 0 0.00e+00 0 0.00e+00 100</div>
<div><br class="ContentPasted0">
</div>
<div class="ContentPasted0">VecScatterBegin 4647 1.0 nan nan 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 1 0 0 0 0 1 0 0 0 0 -nan</div>
<div class="ContentPasted0"> -nan 2 5.14e-03 0 0.00e+00 0</div>
<div><br class="ContentPasted0">
</div>
<div class="ContentPasted0">VecScatterEnd 4647 1.0 nan nan 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 -nan</div>
<div class="ContentPasted0"> -nan 0 0.00e+00 0 0.00e+00 0</div>
<div><br class="ContentPasted0">
</div>
<div class="ContentPasted0">VecReduceArith 380 1.0 nan nan 4.23e+06 1.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 -nan</div>
<div class="ContentPasted0"> -nan 0 0.00e+00 0 0.00e+00 100</div>
<div><br class="ContentPasted0">
</div>
<div class="ContentPasted0">VecReduceComm 190 1.0 nan nan 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 -nan</div>
<div class="ContentPasted0"> -nan 0 0.00e+00 0 0.00e+00 0</div>
<div><br class="ContentPasted0">
</div>
<div class="ContentPasted0">VecNormalize 965 1.0 nan nan 1.61e+07 1.0 0.0e+00 0.0e+00 0.0e+00 0 1 0 0 0 0 1 0 0 0 -nan</div>
<div class="ContentPasted0"> -nan 0 0.00e+00 0 0.00e+00 100</div>
<div><br class="ContentPasted0">
</div>
<div class="ContentPasted0">TSStep 20 1.0 5.8699e+00 1.0 1.08e+09 1.0 0.0e+00 0.0e+00 0.0e+00 97100 0 0 0 97100 0 0 0 184</div>
<div class="ContentPasted0"> -nan 2 5.14e-03 0 0.00e+00 54</div>
<div><br class="ContentPasted0">
</div>
<div class="ContentPasted0">TSFunctionEval 597 1.0 nan nan 6.64e+06 1.0 0.0e+00 0.0e+00 0.0e+00 63 1 0 0 0 63 1 0 0 0 -nan</div>
<div class="ContentPasted0"> -nan 1 3.36e-04 0 0.00e+00 100</div>
<div><br class="ContentPasted0">
</div>
<div class="ContentPasted0">TSJacobianEval 190 1.0 nan nan 3.37e+07 1.0 0.0e+00 0.0e+00 0.0e+00 24 3 0 0 0 24 3 0 0 0 -nan</div>
<div class="ContentPasted0"> -nan 0 0.00e+00 0 0.00e+00 97</div>
<div><br class="ContentPasted0">
</div>
<div class="ContentPasted0">MatMult 1930 1.0 nan nan 4.46e+08 1.0 0.0e+00 0.0e+00 0.0e+00 1 41 0 0 0 1 41 0 0 0 -nan</div>
<div class="ContentPasted0"> -nan 0 0.00e+00 0 0.00e+00 100</div>
<div><br class="ContentPasted0">
</div>
<div class="ContentPasted0">MatMultTranspose 1 1.0 nan nan 3.44e+05 1.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 -nan</div>
<div class="ContentPasted0"> -nan 0 0.00e+00 0 0.00e+00 100</div>
<div><br class="ContentPasted0">
</div>
<div class="ContentPasted0">MatSolve 965 1.0 nan nan 5.04e+07 1.0 0.0e+00 0.0e+00 0.0e+00 1 5 0 0 0 1 5 0 0 0 -nan</div>
<div class="ContentPasted0"> -nan 0 0.00e+00 0 0.00e+00 0</div>
<div><br class="ContentPasted0">
</div>
<div class="ContentPasted0">MatSOR 965 1.0 nan nan 3.33e+08 1.0 0.0e+00 0.0e+00 0.0e+00 4 31 0 0 0 4 31 0 0 0 -nan</div>
<div class="ContentPasted0"> -nan 0 0.00e+00 0 0.00e+00 0</div>
<div><br class="ContentPasted0">
</div>
<div class="ContentPasted0">MatLUFactorSym 1 1.0 nan nan 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 -nan</div>
<div class="ContentPasted0"> -nan 0 0.00e+00 0 0.00e+00 0</div>
<div><br class="ContentPasted0">
</div>
<div class="ContentPasted0">MatLUFactorNum 190 1.0 nan nan 1.16e+08 1.0 0.0e+00 0.0e+00 0.0e+00 1 11 0 0 0 1 11 0 0 0 -nan</div>
<div class="ContentPasted0"> -nan 0 0.00e+00 0 0.00e+00 0</div>
<div><br class="ContentPasted0">
</div>
<div class="ContentPasted0">MatScale 190 1.0 nan nan 3.26e+07 1.0 0.0e+00 0.0e+00 0.0e+00 0 3 0 0 0 0 3 0 0 0 -nan</div>
<div class="ContentPasted0"> -nan 0 0.00e+00 0 0.00e+00 100</div>
<div><br class="ContentPasted0">
</div>
<div class="ContentPasted0">MatAssemblyBegin 761 1.0 nan nan 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 -nan</div>
<div class="ContentPasted0"> -nan 0 0.00e+00 0 0.00e+00 0</div>
<div><br class="ContentPasted0">
</div>
<div class="ContentPasted0">MatAssemblyEnd 761 1.0 nan nan 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 -nan</div>
<div class="ContentPasted0"> -nan 0 0.00e+00 0 0.00e+00 0</div>
<div><br class="ContentPasted0">
</div>
<div class="ContentPasted0">MatGetRowIJ 1 1.0 nan nan 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 -nan</div>
<div class="ContentPasted0"> -nan 0 0.00e+00 0 0.00e+00 0</div>
<div><br class="ContentPasted0">
</div>
<div class="ContentPasted0">MatCreateSubMats 380 1.0 nan nan 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 1 0 0 0 0 1 0 0 0 0 -nan</div>
<div class="ContentPasted0"> -nan 0 0.00e+00 0 0.00e+00 0</div>
<div><br class="ContentPasted0">
</div>
<div class="ContentPasted0">MatGetOrdering 1 1.0 nan nan 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 -nan</div>
<div class="ContentPasted0"> -nan 0 0.00e+00 0 0.00e+00 0</div>
<div><br class="ContentPasted0">
</div>
<div class="ContentPasted0">MatZeroEntries 379 1.0 nan nan 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 -nan</div>
<div class="ContentPasted0"> -nan 0 0.00e+00 0 0.00e+00 0</div>
<div><br class="ContentPasted0">
</div>
<div class="ContentPasted0">MatSetPreallCOO 1 1.0 nan nan 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 -nan</div>
<div class="ContentPasted0"> -nan 0 0.00e+00 0 0.00e+00 0</div>
<div><br class="ContentPasted0">
</div>
<div class="ContentPasted0">MatSetValuesCOO 190 1.0 nan nan 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 1 0 0 0 0 1 0 0 0 0 -nan</div>
<div class="ContentPasted0"> -nan 0 0.00e+00 0 0.00e+00 0</div>
<div><br class="ContentPasted0">
</div>
<div class="ContentPasted0">KSPSetUp 760 1.0 nan nan 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 -nan</div>
<div class="ContentPasted0"> -nan 0 0.00e+00 0 0.00e+00 0</div>
<div><br class="ContentPasted0">
</div>
<div class="ContentPasted0">KSPSolve 190 1.0 5.8052e-01 1.0 9.30e+08 1.0 0.0e+00 0.0e+00 0.0e+00 10 86 0 0 0 10 86 0 0 0 1602</div>
<div class="ContentPasted0"> -nan 1 4.80e-03 0 0.00e+00 46</div>
<div><br class="ContentPasted0">
</div>
<div class="ContentPasted0">KSPGMRESOrthog 775 1.0 nan nan 2.27e+07 1.0 0.0e+00 0.0e+00 0.0e+00 1 2 0 0 0 1 2 0 0 0 -nan</div>
<div class="ContentPasted0"> -nan 0 0.00e+00 0 0.00e+00 100</div>
<div><br class="ContentPasted0">
</div>
<div class="ContentPasted0">SNESSolve 71 1.0 5.7117e+00 1.0 1.07e+09 1.0 0.0e+00 0.0e+00 0.0e+00 95 99 0 0 0 95 99 0 0 0 188</div>
<div class="ContentPasted0"> -nan 1 4.80e-03 0 0.00e+00 53</div>
<div><br class="ContentPasted0">
</div>
<div class="ContentPasted0">SNESSetUp 1 1.0 nan nan 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 -nan</div>
<div class="ContentPasted0"> -nan 0 0.00e+00 0 0.00e+00 0</div>
<div><br class="ContentPasted0">
</div>
<div class="ContentPasted0">SNESFunctionEval 573 1.0 nan nan 2.23e+07 1.0 0.0e+00 0.0e+00 0.0e+00 60 2 0 0 0 60 2 0 0 0 -nan</div>
<div class="ContentPasted0"> -nan 0 0.00e+00 0 0.00e+00 100</div>
<div><br class="ContentPasted0">
</div>
<div class="ContentPasted0">SNESJacobianEval 190 1.0 nan nan 3.37e+07 1.0 0.0e+00 0.0e+00 0.0e+00 24 3 0 0 0 24 3 0 0 0 -nan</div>
<div class="ContentPasted0"> -nan 0 0.00e+00 0 0.00e+00 97</div>
<div><br class="ContentPasted0">
</div>
<div class="ContentPasted0">SNESLineSearch 190 1.0 nan nan 1.05e+08 1.0 0.0e+00 0.0e+00 0.0e+00 53 10 0 0 0 53 10 0 0 0 -nan</div>
<div class="ContentPasted0"> -nan 0 0.00e+00 0 0.00e+00 100</div>
<div><br class="ContentPasted0">
</div>
<div class="ContentPasted0">PCSetUp 570 1.0 nan nan 1.16e+08 1.0 0.0e+00 0.0e+00 0.0e+00 2 11 0 0 0 2 11 0 0 0 -nan</div>
<div class="ContentPasted0"> -nan 0 0.00e+00 0 0.00e+00 0</div>
<div><br class="ContentPasted0">
</div>
<div class="ContentPasted0">PCApply 965 1.0 nan nan 6.14e+08 1.0 0.0e+00 0.0e+00 0.0e+00 8 57 0 0 0 8 57 0 0 0 -nan</div>
<div class="ContentPasted0"> -nan 1 4.80e-03 0 0.00e+00 19</div>
<div><br class="ContentPasted0">
</div>
<div class="ContentPasted0">KSPSolve_FS_0 965 1.0 nan nan 3.33e+08 1.0 0.0e+00 0.0e+00 0.0e+00 4 31 0 0 0 4 31 0 0 0 -nan</div>
<div class="ContentPasted0"> -nan 0 0.00e+00 0 0.00e+00 0</div>
<div><br class="ContentPasted0">
</div>
<div class="ContentPasted0">KSPSolve_FS_1 965 1.0 nan nan 1.66e+08 1.0 0.0e+00 0.0e+00 0.0e+00 2 15 0 0 0 2 15 0 0 0 -nan</div>
<div class="ContentPasted0"> -nan 0 0.00e+00 0 0.00e+00 0</div>
<div><br class="ContentPasted0">
</div>
<div><br class="ContentPasted0">
</div>
<div class="ContentPasted0">--- Event Stage 1: Unknown</div>
<div><br class="ContentPasted0">
</div>
<div class="ContentPasted0">------------------------------------------------------------------------------------------------------------------------</div>
<div class="ContentPasted0">---------------------------------------</div>
<div><br class="ContentPasted0">
</div>
<div><br class="ContentPasted0">
</div>
<div class="ContentPasted0">Object Type Creations Destructions. Reports information only for process 0.</div>
<div><br class="ContentPasted0">
</div>
<div class="ContentPasted0">--- Event Stage 0: Main Stage</div>
<div><br class="ContentPasted0">
</div>
<div class="ContentPasted0"> Container 5 5</div>
<div class="ContentPasted0"> Distributed Mesh 2 2</div>
<div class="ContentPasted0"> Index Set 11 11</div>
<div class="ContentPasted0"> IS L to G Mapping 1 1</div>
<div class="ContentPasted0"> Star Forest Graph 7 7</div>
<div class="ContentPasted0"> Discrete System 2 2</div>
<div class="ContentPasted0"> Weak Form 2 2</div>
<div class="ContentPasted0"> Vector 49 49</div>
<div class="ContentPasted0"> TSAdapt 1 1</div>
<div class="ContentPasted0"> TS 1 1</div>
<div class="ContentPasted0"> DMTS 1 1</div>
<div class="ContentPasted0"> SNES 1 1</div>
<div class="ContentPasted0"> DMSNES 3 3</div>
<div class="ContentPasted0"> SNESLineSearch 1 1</div>
<div class="ContentPasted0"> Krylov Solver 4 4</div>
<div class="ContentPasted0"> DMKSP interface 1 1</div>
<div class="ContentPasted0"> Matrix 4 4</div>
<div class="ContentPasted0"> Preconditioner 4 4</div>
<div class="ContentPasted0"> Viewer 2 1</div>
<div><br class="ContentPasted0">
</div>
<div class="ContentPasted0">--- Event Stage 1: Unknown</div>
<div><br class="ContentPasted0">
</div>
<div class="ContentPasted0">========================================================================================================================</div>
<div class="ContentPasted0">Average time to get PetscTime(): 3.14e-08</div>
<div class="ContentPasted0">#PETSc Option Table entries:</div>
<div class="ContentPasted0">-log_view</div>
<div class="ContentPasted0">-log_view_gpu_times</div>
<div class="ContentPasted0">#End of PETSc Option Table entries</div>
<div class="ContentPasted0">Compiled without FORTRAN kernels</div>
<div class="ContentPasted0">Compiled with 64 bit PetscInt</div>
<div class="ContentPasted0">Compiled with full precision matrices (default)</div>
<div class="ContentPasted0">sizeof(short) 2 sizeof(int) 4 sizeof(long) 8 sizeof(void*) 8 sizeof(PetscScalar) 8 sizeof(PetscInt) 8</div>
<div class="ContentPasted0">Configure options: PETSC_DIR=/home/4pf/repos/petsc PETSC_ARCH=arch-kokkos-cuda-no-tpls --with-cc=mpicc --with-cxx=mpicxx --with-fc=0 --with-cuda --with-debugging=0 --with-shared-libraries --prefix=/home/4pf/build/petsc/cuda-no-tpls/install
--with-64-bit-indices --COPTFLAGS=-O3 --CXXOPTFLAGS=-O3 --CUDAOPTFLAGS=-O3 --with-kokkos-dir=/home/4pf/build/kokkos/cuda/install --with-kokkos-kernels-dir=/home/4pf/build/kokkos-kernels/cuda-no-tpls/install</div>
<div><br class="ContentPasted0">
</div>
<div class="ContentPasted0">-----------------------------------------</div>
<div class="ContentPasted0">Libraries compiled on 2022-11-01 21:01:08 on PC0115427
</div>
<div class="ContentPasted0">Machine characteristics: Linux-5.15.0-52-generic-x86_64-with-glibc2.35</div>
<div class="ContentPasted0">Using PETSc directory: /home/4pf/build/petsc/cuda-no-tpls/install</div>
<div class="ContentPasted0">Using PETSc arch: </div>
<div class="ContentPasted0">-----------------------------------------</div>
<div><br class="ContentPasted0">
</div>
<div class="ContentPasted0">Using C compiler: mpicc -fPIC -Wall -Wwrite-strings -Wno-unknown-pragmas -Wno-lto-type-mismatch -Wno-stringop-overflow -fstack-protector -fvisibility=hidden -O3
</div>
<div class="ContentPasted0">-----------------------------------------</div>
<div><br class="ContentPasted0">
</div>
<div class="ContentPasted0">Using include paths: -I/home/4pf/build/petsc/cuda-no-tpls/install/include -I/home/4pf/build/kokkos-kernels/cuda-no-tpls/install/include -I/home/4pf/build/kokkos/cuda/install/include -I/usr/local/cuda-11.8/include</div>
<div class="ContentPasted0">-----------------------------------------</div>
<div><br class="ContentPasted0">
</div>
<div class="ContentPasted0">Using C linker: mpicc</div>
<div class="ContentPasted0">Using libraries: -Wl,-rpath,/home/4pf/build/petsc/cuda-no-tpls/install/lib -L/home/4pf/build/petsc/cuda-no-tpls/install/lib -lpetsc -Wl,-rpath,/home/4pf/build/kokkos-kernels/cuda-no-tpls/install/lib -L/home/4pf/build/kokkos-kernels/cuda-no-tpls/install/lib
-Wl,-rpath,/home/4pf/build/kokkos/cuda/install/lib -L/home/4pf/build/kokkos/cuda/install/lib -Wl,-rpath,/usr/local/cuda-11.8/lib64 -L/usr/local/cuda-11.8/lib64 -L/usr/local/cuda-11.8/lib64/stubs -lkokkoskernels -lkokkoscontainers -lkokkoscore -llapack -lblas
-lm -lcudart -lnvToolsExt -lcufft -lcublas -lcusparse -lcusolver -lcurand -lcuda -lquadmath -lstdc++ -ldl</div>
<div class="ContentPasted0">-----------------------------------------</div>
<br>
</div>
<div class="elementToProof">
<div style="font-family: Consolas, Courier, monospace; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div id="Signature">
<div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<span style="font-size:11pt"><strong>Philip Fackler<br>
</strong></span></div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<div><span style="font-size:11pt">Research Software Engineer, Application Engineering Group</span></div>
<div><span style="font-size:11pt">Advanced Computing Systems Research Section</span></div>
<div><span style="font-size:11pt">Computer Science and Mathematics Division<br>
</span></div>
<div><span style="font-size:11pt"><strong>Oak Ridge National Laboratory</strong></span><span style="font-size:11pt"></span><br>
</div>
</div>
</div>
</div>
</div>
<div id="appendonsend"></div>
<hr style="display:inline-block;width:98%" tabindex="-1">
<div id="divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" style="font-size:11pt" color="#000000"><b>From:</b> Junchao Zhang <junchao.zhang@gmail.com><br>
<b>Sent:</b> Tuesday, November 15, 2022 13:03<br>
<b>To:</b> Fackler, Philip <facklerpw@ornl.gov><br>
<b>Cc:</b> xolotl-psi-development@lists.sourceforge.net <xolotl-psi-development@lists.sourceforge.net>; petsc-users@mcs.anl.gov <petsc-users@mcs.anl.gov>; Blondel, Sophie <sblondel@utk.edu>; Roth, Philip <rothpc@ornl.gov><br>
<b>Subject:</b> Re: [EXTERNAL] Re: [petsc-users] Kokkos backend for Mat and Vec diverging when running on CUDA device.</font>
<div> </div>
</div>
<div>
<div dir="ltr">Can you paste -log_view result so I can see what functions are used?
<div>
<div><br clear="all">
<div>
<div dir="ltr" class="x_gmail_signature" data-smartmail="gmail_signature">
<div dir="ltr">--Junchao Zhang</div>
</div>
</div>
<br>
</div>
</div>
</div>
<br>
<div class="x_gmail_quote">
<div dir="ltr" class="x_gmail_attr">On Tue, Nov 15, 2022 at 10:24 AM Fackler, Philip <<a href="mailto:facklerpw@ornl.gov">facklerpw@ornl.gov</a>> wrote:<br>
</div>
<blockquote class="x_gmail_quote" style="margin:0px 0px 0px 0.8ex; border-left:1px solid rgb(204,204,204); padding-left:1ex">
<div class="x_msg-5191598488002563482">
<div dir="ltr">
<div style="font-family:Consolas,Courier,monospace; font-size:12pt; color:rgb(0,0,0); background-color:rgb(255,255,255)">
Yes, most (but not all) of our system test cases fail with the kokkos/cuda or cuda backends. All of them pass with the CPU-only kokkos backend.<br>
</div>
<div>
<div style="font-family:Consolas,Courier,monospace; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div id="x_m_-5191598488002563482Signature">
<div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<span style="font-size:11pt"><strong>Philip Fackler<br>
</strong></span></div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<div><span style="font-size:11pt">Research Software Engineer, Application Engineering Group</span></div>
<div><span style="font-size:11pt">Advanced Computing Systems Research Section</span></div>
<div><span style="font-size:11pt">Computer Science and Mathematics Division<br>
</span></div>
<div><span style="font-size:11pt"><strong>Oak Ridge National Laboratory</strong></span><span style="font-size:11pt"></span><br>
</div>
</div>
</div>
</div>
</div>
<div id="x_m_-5191598488002563482appendonsend"></div>
<hr style="display:inline-block; width:98%">
<div id="x_m_-5191598488002563482divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" color="#000000" style="font-size:11pt"><b>From:</b> Junchao Zhang <<a href="mailto:junchao.zhang@gmail.com" target="_blank">junchao.zhang@gmail.com</a>><br>
<b>Sent:</b> Monday, November 14, 2022 19:34<br>
<b>To:</b> Fackler, Philip <<a href="mailto:facklerpw@ornl.gov" target="_blank">facklerpw@ornl.gov</a>><br>
<b>Cc:</b> <a href="mailto:xolotl-psi-development@lists.sourceforge.net" target="_blank">
xolotl-psi-development@lists.sourceforge.net</a> <<a href="mailto:xolotl-psi-development@lists.sourceforge.net" target="_blank">xolotl-psi-development@lists.sourceforge.net</a>>;
<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a> <<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a>>; Blondel, Sophie <<a href="mailto:sblondel@utk.edu" target="_blank">sblondel@utk.edu</a>>;
Zhang, Junchao <<a href="mailto:jczhang@mcs.anl.gov" target="_blank">jczhang@mcs.anl.gov</a>>; Roth, Philip <<a href="mailto:rothpc@ornl.gov" target="_blank">rothpc@ornl.gov</a>><br>
<b>Subject:</b> [EXTERNAL] Re: [petsc-users] Kokkos backend for Mat and Vec diverging when running on CUDA device.</font>
<div> </div>
</div>
<div>
<div dir="ltr">Hi, Philip,<br>
Sorry to hear that. It seems you could run the same code on CPUs but not no GPUs (with either petsc/Kokkos backend or petsc/cuda backend, is it right?
<div style="border:0px; font-variant-numeric:inherit; font-variant-east-asian:inherit; font-stretch:inherit; font-size:12pt; line-height:inherit; font-family:Calibri,Arial,Helvetica,sans-serif; margin:0px; padding:0px; vertical-align:baseline; color:black">
<br>
</div>
<div>
<div dir="ltr">
<div dir="ltr">--Junchao Zhang</div>
</div>
</div>
<br>
</div>
<br>
<div>
<div dir="ltr">On Mon, Nov 14, 2022 at 12:13 PM Fackler, Philip via petsc-users <<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a>> wrote:<br>
</div>
<blockquote style="margin:0px 0px 0px 0.8ex; border-left:1px solid rgb(204,204,204); padding-left:1ex">
<div>
<div dir="ltr">
<div style="font-family:Consolas,Courier,monospace; font-size:12pt; color:rgb(0,0,0); background-color:rgb(255,255,255)">
This is an issue I've brought up before (and discussed in-person with Richard). I wanted to bring it up again because I'm hitting the limits of what I know to do, and I need help figuring this out.</div>
<div style="font-family:Consolas,Courier,monospace; font-size:12pt; color:rgb(0,0,0); background-color:rgb(255,255,255)">
<br>
</div>
<div style="font-family:Consolas,Courier,monospace; font-size:12pt; color:rgb(0,0,0); background-color:rgb(255,255,255)">
The problem can be reproduced using Xolotl's "develop" branch built against a petsc build with kokkos and kokkos-kernels enabled. Then, either add the relevant kokkos options to the "petscArgs=" line in the system test parameter file(s), or just replace the
system test parameter files with the ones from the "feature-petsc-kokkos" branch. See
<a title="hxxps://github.com/ORNL-Fusion/xolotl/tree/feature-petsc-kokkos/benchmarks">
here</a> the files that begin with "params_system_".</div>
<div style="font-family:Consolas,Courier,monospace; font-size:12pt; color:rgb(0,0,0); background-color:rgb(255,255,255)">
<br>
</div>
<div style="font-family:Consolas,Courier,monospace; font-size:12pt; color:rgb(0,0,0); background-color:rgb(255,255,255)">
Note that those files use the "kokkos" options, but the problem is similar using the corresponding cuda/cusparse options. I've already tried building kokkos-kernels with no TPLs and got slightly different results, but the same problem.<br>
</div>
<div style="font-family:Consolas,Courier,monospace; font-size:12pt; color:rgb(0,0,0); background-color:rgb(255,255,255)">
<br>
</div>
<div style="font-family:Consolas,Courier,monospace; font-size:12pt; color:rgb(0,0,0); background-color:rgb(255,255,255)">
Any help would be appreciated.</div>
<div style="font-family:Consolas,Courier,monospace; font-size:12pt; color:rgb(0,0,0); background-color:rgb(255,255,255)">
<br>
</div>
<div style="font-family:Consolas,Courier,monospace; font-size:12pt; color:rgb(0,0,0); background-color:rgb(255,255,255)">
Thanks,<br>
</div>
<div>
<div style="font-family:Consolas,Courier,monospace; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div id="x_m_-5191598488002563482x_m_-25761908020450439Signature">
<div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<span style="font-size:11pt"><strong>Philip Fackler<br>
</strong></span></div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<div><span style="font-size:11pt">Research Software Engineer, Application Engineering Group</span></div>
<div><span style="font-size:11pt">Advanced Computing Systems Research Section</span></div>
<div><span style="font-size:11pt">Computer Science and Mathematics Division<br>
</span></div>
<div><span style="font-size:11pt"><strong>Oak Ridge National Laboratory</strong></span><span style="font-size:11pt"></span><br>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
</div>
</div>
</div>
</blockquote>
</div>
</div>
</body>
</html>