<div dir="ltr"><div dir="ltr"><br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, Apr 14, 2023 at 8:54 AM Jeremy Theler <<a href="mailto:jeremy@seamplex.com">jeremy@seamplex.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Hi Mark. So glad you answered.<br>
<br>
> 0) what is your test problem? eg, 3D Lapacian with Q1 finite<br>
> elements.<br>
<br>
I said in my first email it was linear elasticty (and I gave a link<br>
where you can see the geometry, BCs, etc.) but I did not specifty<br>
further details.<br></blockquote><div><br></div><div>OK, this is fine. Nice 3D tet mesh with P2 (P1 will lock). I assume you have a benign Poisson ratio.</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
It is linear elasticity with displacement-based FEM formulation using<br>
unstructured curved 10-noded tetrahedra.<br>
<br>
The matrix is marked as SPD with MatSetOption() and the solver is<br>
indeed CG and not the default GMRES.<br>
<br></blockquote><div><br></div><div>good</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
> First, you can get GAMG diagnostics by running with '-info :pc' and<br>
> grep on GAMG.<br>
<br>
Great advice. Now I have a lot more of information but I'm not sure how<br>
to analyze it. Find attached for each combination of threshold and<br>
PETSc version the output of -info :pc -ksp_monitor -ksp_view<br>
<br></blockquote><div><br></div><div>(new_py-env) 07:43 ~/Downloads/log 2$ grep "grid complexity" *<br></div><div>....</div><div>infopc-0.02-17.log:[0] PCSetUp_GAMG(): (null): 7 levels, grid complexity = 1.28914<br>infopc-0.02-17.log: Complexity: grid = 1.04361 operator = 1.28914<br>infopc-0.02-18.log: Complexity: grid = 1.05658 operator = 1.64555<br>infopc-0.02-19.log: Complexity: grid = 1.05658 operator = 1.64555<br></div><div><br></div><div>I was using "grid complexity " and changed to the accepted term "operator". You can see that the new coarening method is pretty different on this problem. </div><div>For an isiotropoic problem like this a zero threshould is a good place to start, but you can use it to tune.</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
In general it looks like 3.18 and 3.19 have less KSP iterations than<br>
3.17 but the overall time is larger.<br></blockquote><div><br></div><div>We need to see the solve times.</div><div>Run with -log_view and grep for KSPSolve</div><div>We can look at the setup time separately</div><div>In practice the setup time is amortized unless you use a full Newton nonlinear solver.</div><div><br></div><div>Your iteration counts are reasonable. They go up a little with Jacobi.</div><div><br></div><div>Here are your grid sizes:</div><div><br></div><div>(new_py-env) 07:55 ~/Downloads/log 2$ grep N= infopc-0.0001-19.log<br>[0] <pc> PCSetUp_GAMG(): (null): level 0) N=568386, n data rows=3, n data cols=6, nnz/row (ave)=82, np=1<br>[0] <pc> PCSetUp_GAMG(): (null): 1) N=22206, n data cols=6, nnz/row (ave)=445, 1 active pes<br>[0] <pc> PCSetUp_GAMG(): (null): 2) N=2628, n data cols=6, nnz/row (ave)=1082, 1 active pes<br>[0] <pc> PCSetUp_GAMG(): (null): 3) N=180, n data cols=6, nnz/row (ave)=180, 1 active pes<br>[0] <pc> PCSetUp_GAMG(): (null): 4) N=18, n data cols=6, nnz/row (ave)=18, 1 active pes<br>[0] <pc> PCSetUp_GAMG(): (null): PCSetUp_GAMG: call KSPChebyshevSetEigenvalues on level 3 (N=180) with emax = 2.0785 emin = 0.0468896<br>[0] <pc> PCSetUp_GAMG(): (null): PCSetUp_GAMG: call KSPChebyshevSetEigenvalues on level 2 (N=2628) with emax = 1.78977 emin = 5.91431e-08<br>[0] <pc> PCSetUp_GAMG(): (null): PCSetUp_GAMG: call KSPChebyshevSetEigenvalues on level 1 (N=22206) with emax = 2.12541 emin = 4.27581e-09<br>[0] <pc> PCSetUp_GAMG(): (null): PCSetUp_GAMG: call KSPChebyshevSetEigenvalues on level 0 (N=568386) with emax = 3.42185 emin = 0.0914406<br>(new_py-env) 07:56 ~/Downloads/log 2$ grep N= infopc-0.0001-15.log<br>[0] PCSetUp_GAMG(): level 0) N=568386, n data rows=3, n data cols=6, nnz/row (ave)=82, np=1<br>[0] PCSetUp_GAMG(): 1) N=15642, n data cols=6, nnz/row (ave)=368, 1 active pes<br>[0] PCSetUp_GAMG(): 2) N=1266, n data cols=6, nnz/row (ave)=468, 1 active pes<br>[0] PCSetUp_GAMG(): 3) N=108, n data cols=6, nnz/row (ave)=108, 1 active pes<br>[0] PCSetUp_GAMG(): 4) N=12, n data cols=6, nnz/row (ave)=12, 1 active pes<br></div><div><br></div><div>The new version coarsens slower.</div><div><br></div><div>BTW. Use something like -pc_gamg_coarse_eq_limit 1000</div><div>Your coarse grids are too small.</div><div>You can grep on MatLUFactor to check that the coarse grid solve/factor is under control but 1000 in 3D is pretty conservative.</div><div><br></div><div>I am sure you are are going to want more aggresive coarsening (newer versions): -pc_gamg_aggressive_coarsening <1></div><div>Just try 10 (ie, all levels) to start.</div><div><br></div><div>Mark</div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<br>
> Anyway, sorry for the changes. <br>
> I hate changing GAMG for this reason and I hate AMG for this reason!<br>
<br>
No need to apologize, I just want to better understand how to better<br>
exploit your code!<br>
<br>
Thanks<br>
--<br>
jeremy<br>
<br>
> <br>
> Thanks,<br>
> Mark<br>
> <br>
> <br>
> <br>
> On Thu, Apr 13, 2023 at 8:17 AM Jeremy Theler <<a href="mailto:jeremy@seamplex.com" target="_blank">jeremy@seamplex.com</a>><br>
> wrote:<br>
> > When using GAMG+cg for linear elasticity and providing the near<br>
> > nullspace computed by MatNullSpaceCreateRigidBody(), I used to find<br>
> > "experimentally" that a small value of -pc_gamg_threshold in the<br>
> > order<br>
> > of 0.0001 would slightly decrease the solve time.<br>
> > <br>
> > Starting with 3.18, I started seeing that any positive value for<br>
> > the<br>
> > treshold would increase the solve time. I did a quick parametric<br>
> > (serial) run solving an elastic problem with a matrix size of<br>
> > approx<br>
> > 570k x 570k for different values of GAMG threshold and different<br>
> > PETSc<br>
> > versions (compiled with the same compiler, options and flags).<br>
> > <br>
> > I noted that<br>
> > <br>
> > 1. starting from 3.18, a threshold of 0.0001 that used to improve<br>
> > the<br>
> > speed now worsens it. <br>
> > 2. PETSc 3.17 looks like a "sweet spot" of speed<br>
> > <br>
> > I would like to hear any comments you might have.<br>
> > <br>
> > The wall time shown includes the time needed to read the mesh and<br>
> > assemble the stiffness matrix. It is a refined version of the<br>
> > NAFEMS<br>
> > LE10 benchmark described here:<br>
> > <a href="https://seamplex.com/feenox/examples/mechanical.html#nafems-le10-" rel="noreferrer" target="_blank">https://seamplex.com/feenox/examples/mechanical.html#nafems-le10-</a><br>
> > thick-plate-pressure-benchmark<br>
> > <br>
> > If you want, I could dump the matrix, rhs and near nullspace<br>
> > vectors<br>
> > and share them.<br>
> > <br>
> > --<br>
> > jeremy theler<br>
> > <br>
<br>
</blockquote></div></div>