[petsc-users] comparing HYPRE on CPU vs GPU

[Barry Smith]

  Probably some of these questions are best asked to the hypre folks. But I can answer some

> On Nov 24, 2022, at 11:19 AM, nicola varini <nicola.varini at gmail.com> wrote:
> 
> Dear all, I am comparing the HYPRE boomeramg preconditioner on CPU and GPU.
> It looks like the defaults are different, therefore I tried to specify the match the options.
> I have a few observations/questions:
> 1) Why are the residuals at step 0 different?

  By default, PETSc uses left preconditioning; thus, the norms reported are for the preconditioner residual and will be different for different preconditioner options (and possibly slightly different for numerical differences even if run with the same preconditioner options.  You can use -ksp_pc_side right to have the true residual norm printed, this should be the same except for possibly, hopefully, small numerics effects.

> 2) If the options are the same the number of iterations should be the same?

  Numerical changes could affect these. Hopefully, not much if all the algorithms are truly the same.

> 3) Why on the GPU side the PCApply doesn't seem to be done on GPU?
> I noticed that if I change certain options, like pc_hypre_boomeramg_relax_type_all symmetric SOR/Jacobi, the KSPSolve get slower and performs on CPU.

   Hypre probably does not have, by default, a GPU version of SOR, so it must be run on a CPU (Petsc also does not currently have a GPU SOR).

> However, how can I diagnose if the computations are perfomed on GPU or not?

  Run with -log_view -log_view_gpu_time and look at the final columns that give some information on time and communication to the GPU. But since hypre does not log flops, the flop rates for computations done by hypre are meaningless.


> 
> Thanks very much,
> 
> Nicola
> <log_cpu><log_gpu>