<div dir="ltr">Barry,<div><br></div><div> I solved the issue, it was related to a wrong change I made in the creation of the IS for the VecScatter (used in the shell matrix). Fixing that, I reached the same performance. Thank your support.</div><div><br></div><div>Mark, Hong,</div><div><br></div><div> Thank you for your replies; I am also evaluating your suggestion on the optimal parametrization for the MG pre-conditioner.</div><div><br></div><div>Best regards,</div><div>Federico</div><div><br></div><div><br></div><div class="gmail_extra"><br><div class="gmail_quote">On 15 September 2017 at 17:45, Barry Smith <span dir="ltr"><<a href="mailto:bsmith@mcs.anl.gov" target="_blank">bsmith@mcs.anl.gov</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><br>
$ python<br>
Python 2.7.13 (default, Dec 18 2016, 07:03:39)<br>
[GCC 4.2.1 Compatible Apple LLVM 8.0.0 (clang-800.0.42.1)] on darwin<br>
Type "help", "copyright", "credits" or "license" for more information.<br>
MatMult >>> 3.6272e+03 - 2.0894e+03<br>
1537.7999999999997<br>
KSP Solve >>> 3.6329e+03 - 2.0949e+03<br>
1538.0<br>
>>><br>
<br>
You are right, all the extra time is within the MatMult() so for some reason your shell mat mult is much slower. I cannot guess why unless I can see inside your shell matmult at what it is doing.<br>
<br>
<br>
Make sure your configure options are identical and using same compiler.<br>
<br>
Barry<br>
<div><div class="h5"><br>
<br>
<br>
<br>
<br>
> On Sep 15, 2017, at 5:08 AM, Federico Golfrè Andreasi <<a href="mailto:federico.golfre@gmail.com">federico.golfre@gmail.com</a>> wrote:<br>
><br>
> Hi Barry,<br>
><br>
> I have attached an extract of the our program output for both the versions: PETSc-3.4.4 and PETSc-3.7.3.<br>
><br>
> In this program the KSP has a shell matrix as operator and a MPIAIJ matrix as pre-conditioner.<br>
> I was wondering if the slowing is related more on the operations done in the MatMult of the shell matrix;<br>
> because on a test program where I solve a similar system without shell matrix I do not see the performance degradation.<br>
><br>
> Perhaps you could give me some hints,<br>
> Thank you and best regards,<br>
> Federico<br>
><br>
><br>
><br>
><br>
> On 13 September 2017 at 17:58, Barry Smith <<a href="mailto:bsmith@mcs.anl.gov">bsmith@mcs.anl.gov</a>> wrote:<br>
><br>
> > On Sep 13, 2017, at 10:56 AM, Federico Golfrè Andreasi <<a href="mailto:federico.golfre@gmail.com">federico.golfre@gmail.com</a>> wrote:<br>
> ><br>
> > Hi Barry,<br>
> ><br>
> > I understand and perfectly agree with you that the behavior increase after the release due to better tuning.<br>
> ><br>
> > In my case, the difference in the solution is negligible, but the runtime increases up to +70% (with the same number of ksp_iterations).<br>
><br>
> Ok this is an important (and bad) difference.<br>
><br>
> > So I was wondering if maybe there were just some flags related to memory preallocation or re-usage of intermediate solution that before was defaulted.<br>
><br>
> Note likely it is this.<br>
><br>
> Are both compiled with the same level of compiler optimization?<br>
><br>
> Please run both with -log_summary and send the output, this will tell us WHAT parts are now slower.<br>
><br>
> Barry<br>
><br>
> ><br>
> > Thank you,<br>
> > Federico<br>
> ><br>
> ><br>
> ><br>
> > On 13 September 2017 at 17:29, Barry Smith <<a href="mailto:bsmith@mcs.anl.gov">bsmith@mcs.anl.gov</a>> wrote:<br>
> ><br>
> > There will likely always be slight differences in convergence over that many releases. Lots of little defaults etc get changed over time as we learn from users and increase the robustness of the defaults.<br>
> ><br>
> > So in your case do the differences matter?<br>
> ><br>
> > 1) What is the time to solution in both cases, is it a few percent different or now much slower?<br>
> ><br>
> > 2) What about number of iterations? Almost identical (say 1 or 2 different) or does it now take 30 iterations when it use to take 5?<br>
> ><br>
> > Barry<br>
> ><br>
> > > On Sep 13, 2017, at 10:25 AM, Federico Golfrè Andreasi <<a href="mailto:federico.golfre@gmail.com">federico.golfre@gmail.com</a>> wrote:<br>
> > ><br>
> > > Dear PETSc users/developers,<br>
> > ><br>
> > > I recently switched from PETSc-3.4 to PETSc-3.7 and found that some default setup for the "mg" (mutigrid) preconditioner have changed.<br>
> > ><br>
> > > We were solving a linear system passing, throug command line, the following options:<br>
> > > -ksp_type fgmres<br>
> > > -ksp_max_it 100000<br>
> > > -ksp_rtol 0.000001<br>
> > > -pc_type mg<br>
> > > -ksp_view<br>
> > ><br>
> > > The output of the KSP view is as follow:<br>
> > ><br>
> > > KSP Object: 128 MPI processes<br>
> > > type: fgmres<br>
> > > GMRES: restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement<br>
> > > GMRES: happy breakdown tolerance 1e-30<br>
> > > maximum iterations=100000, initial guess is zero<br>
> > > tolerances: relative=1e-06, absolute=1e-50, divergence=10000<br>
> > > right preconditioning<br>
> > > using UNPRECONDITIONED norm type for convergence test<br>
> > > PC Object: 128 MPI processes<br>
> > > type: mg<br>
> > > MG: type is MULTIPLICATIVE, levels=1 cycles=v<br>
> > > Cycles per PCApply=1<br>
> > > Not using Galerkin computed coarse grid matrices<br>
> > > Coarse grid solver -- level ------------------------------<wbr>-<br>
> > > KSP Object: (mg_levels_0_) 128 MPI processes<br>
> > > type: chebyshev<br>
> > > Chebyshev: eigenvalue estimates: min = 0.223549, max = 2.45903<br>
> > > Chebyshev: estimated using: [0 0.1; 0 1.1]<br>
> > > KSP Object: (mg_levels_0_est_) 128 MPI processes<br>
> > > type: gmres<br>
> > > GMRES: restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement<br>
> > > GMRES: happy breakdown tolerance 1e-30<br>
> > > maximum iterations=10, initial guess is zero<br>
> > > tolerances: relative=1e-05, absolute=1e-50, divergence=10000<br>
> > > left preconditioning<br>
> > > using NONE norm type for convergence test<br>
> > > PC Object: (mg_levels_0_) 128 MPI processes<br>
> > > type: sor<br>
> > > SOR: type = local_symmetric, iterations = 1, local iterations = 1, omega = 1<br>
> > > linear system matrix followed by preconditioner matrix:<br>
> > > Matrix Object: 128 MPI processes<br>
> > > type: mpiaij<br>
> > > rows=279669, cols=279669<br>
> > > total: nonzeros=6427943, allocated nonzeros=6427943<br>
> > > total number of mallocs used during MatSetValues calls =0<br>
> > > not using I-node (on process 0) routines<br>
> > > Matrix Object: 128 MPI processes<br>
> > > type: mpiaij<br>
> > > rows=279669, cols=279669<br>
> > > total: nonzeros=6427943, allocated nonzeros=6427943<br>
> > > total number of mallocs used during MatSetValues calls =0<br>
> > > not using I-node (on process 0) routines<br>
> > > maximum iterations=1, initial guess is zero<br>
> > > tolerances: relative=1e-05, absolute=1e-50, divergence=10000<br>
> > > left preconditioning<br>
> > > using NONE norm type for convergence test<br>
> > > PC Object: (mg_levels_0_) 128 MPI processes<br>
> > > type: sor<br>
> > > SOR: type = local_symmetric, iterations = 1, local iterations = 1, omega = 1<br>
> > > linear system matrix followed by preconditioner matrix:<br>
> > > Matrix Object: 128 MPI processes<br>
> > > type: mpiaij<br>
> > > rows=279669, cols=279669<br>
> > > total: nonzeros=6427943, allocated nonzeros=6427943<br>
> > > total number of mallocs used during MatSetValues calls =0<br>
> > > not using I-node (on process 0) routines<br>
> > > Matrix Object: 128 MPI processes<br>
> > > type: mpiaij<br>
> > > rows=279669, cols=279669<br>
> > > total: nonzeros=6427943, allocated nonzeros=6427943<br>
> > > total number of mallocs used during MatSetValues calls =0<br>
> > > not using I-node (on process 0) routines<br>
> > > linear system matrix followed by preconditioner matrix:<br>
> > > Matrix Object: 128 MPI processes<br>
> > > type: mpiaij<br>
> > > rows=279669, cols=279669<br>
> > > total: nonzeros=6427943, allocated nonzeros=6427943<br>
> > > total number of mallocs used during MatSetValues calls =0<br>
> > > not using I-node (on process 0) routines<br>
> > > Matrix Object: 128 MPI processes<br>
> > > type: mpiaij<br>
> > > rows=279669, cols=279669<br>
> > > total: nonzeros=6427943, allocated nonzeros=6427943<br>
> > > total number of mallocs used during MatSetValues calls =0<br>
> > > not using I-node (on process 0) routines<br>
> > ><br>
> > > When I build the same program using PETSc-3.7 and run it with the same options we observe that the runtime increases and the convergence is slightly different. The output of the KSP view is:<br>
> > ><br>
> > > KSP Object: 128 MPI processes<br>
> > > type: fgmres<br>
> > > GMRES: restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement<br>
> > > GMRES: happy breakdown tolerance 1e-30<br>
> > > maximum iterations=100000, initial guess is zero<br>
> > > tolerances: relative=1e-06, absolute=1e-50, divergence=10000.<br>
> > > right preconditioning<br>
> > > using UNPRECONDITIONED norm type for convergence test<br>
> > > PC Object: 128 MPI processes<br>
> > > type: mg<br>
> > > MG: type is MULTIPLICATIVE, levels=1 cycles=v<br>
> > > Cycles per PCApply=1<br>
> > > Not using Galerkin computed coarse grid matrices<br>
> > > Coarse grid solver -- level ------------------------------<wbr>-<br>
> > > KSP Object: (mg_levels_0_) 128 MPI processes<br>
> > > type: chebyshev<br>
> > > Chebyshev: eigenvalue estimates: min = 0.223549, max = 2.45903<br>
> > > Chebyshev: eigenvalues estimated using gmres with translations [0. 0.1; 0. 1.1]<br>
> > > KSP Object: (mg_levels_0_esteig_) 128 MPI processes<br>
> > > type: gmres<br>
> > > GMRES: restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement<br>
> > > GMRES: happy breakdown tolerance 1e-30<br>
> > > maximum iterations=10, initial guess is zero<br>
> > > tolerances: relative=1e-12, absolute=1e-50, divergence=10000.<br>
> > > left preconditioning<br>
> > > using PRECONDITIONED norm type for convergence test<br>
> > > maximum iterations=2, initial guess is zero<br>
> > > tolerances: relative=1e-05, absolute=1e-50, divergence=10000.<br>
> > > left preconditioning<br>
> > > using NONE norm type for convergence test<br>
> > > PC Object: (mg_levels_0_) 128 MPI processes<br>
> > > type: sor<br>
> > > SOR: type = local_symmetric, iterations = 1, local iterations = 1, omega = 1.<br>
> > > linear system matrix followed by preconditioner matrix:<br>
> > > Mat Object: 128 MPI processes<br>
> > > type: mpiaij<br>
> > > rows=279669, cols=279669<br>
> > > total: nonzeros=6427943, allocated nonzeros=6427943<br>
> > > total number of mallocs used during MatSetValues calls =0<br>
> > > not using I-node (on process 0) routines<br>
> > > Mat Object: 128 MPI processes<br>
> > > type: mpiaij<br>
> > > rows=279669, cols=279669<br>
> > > total: nonzeros=6427943, allocated nonzeros=6427943<br>
> > > total number of mallocs used during MatSetValues calls =0<br>
> > > not using I-node (on process 0) routines<br>
> > > linear system matrix followed by preconditioner matrix:<br>
> > > Mat Object: 128 MPI processes<br>
> > > type: mpiaij<br>
> > > rows=279669, cols=279669<br>
> > > total: nonzeros=6427943, allocated nonzeros=6427943<br>
> > > total number of mallocs used during MatSetValues calls =0<br>
> > > not using I-node (on process 0) routines<br>
> > > Mat Object: 128 MPI processes<br>
> > > type: mpiaij<br>
> > > rows=279669, cols=279669<br>
> > > total: nonzeros=6427943, allocated nonzeros=6427943<br>
> > > total number of mallocs used during MatSetValues calls =0<br>
> > > not using I-node (on process 0) routines<br>
> > ><br>
> > > I was able to get a closer solution adding the following options:<br>
> > > -mg_levels_0_esteig_ksp_norm_<wbr>type none<br>
> > > -mg_levels_0_esteig_ksp_rtol 1.0e-5<br>
> > > -mg_levels_ksp_max_it 1<br>
> > ><br>
> > > But I still can reach the same runtime we were observing with PETSc-3.4, could you please advice me if I should specify any other options?<br>
> > ><br>
> > > Thank you very much for your support,<br>
> > > Federico Golfre' Andreasi<br>
> > ><br>
> ><br>
> ><br>
><br>
><br>
</div></div>> <run_petsc34.txt><run_petsc37.<wbr>txt><br>
<br>
</blockquote></div><br></div></div>