[petsc-users] Function evaluation slowness ?

[Barry Smith]

> On Aug 25, 2015, at 12:45 AM, Timothée Nicolas <timothee.nicolas at gmail.com> wrote:
> 
> Hi,
> 
> I am testing PETSc on the supercomputer where I used to run my explicit MHD code. For my tests I use 256 processes on a problem of size 128*128*640 = 10485760, that is, 40960 grid points per process, and 8 degrees of freedom (or physical fields). The explicit code was using Runge-Kutta 4 for the time scheme, which means 4 function evaluation per time step (plus one operation to put everything together, but let's forget this one). 
> 
> I could thus easily determine that the typical time required for a function evaluation was of the order of 50 ms.
> 
> Now with the implicit Newton-Krylov solver written in PETSc, in the present state where for now I have not implemented any Jacobian or preconditioner whatsoever (so I run with -snes_mf), I measure a typical time between two time steps of between 5 and 20 seconds, and the number of function evaluations for each time step obtained with SNESGetNumberFunctionEvals is 17 (I am speaking of a particular case of course)
> 
> This means a time per function evaluation of about 0.5 to 1 second, that is, 10 to 20 times slower.
> 
> So I have some questions about this.
> 
> 1. First does SNESGetNumberFunctionEvals take into account the function evaluations required to evaluate the Jacobian when -snes_mf is used, as well as the operations required by the GMRES (Krylov) method ? If it were the case, I would somehow intuitively expect a number larger than 17, which could explain the increase in time.

PetscErrorCode  SNESGetNumberFunctionEvals(SNES snes, PetscInt *nfuncs)
{
  *nfuncs = snes->nfuncs;
}

PetscErrorCode  SNESComputeFunction(SNES snes,Vec x,Vec y)
{
...
  snes->nfuncs++;
}

PetscErrorCode  MatCreateSNESMF(SNES snes,Mat *J)
{
.....
  if (snes->pc && snes->pcside == PC_LEFT) {
    ierr = MatMFFDSetFunction(*J,(PetscErrorCode (*)(void*,Vec,Vec))SNESComputeFunctionDefaultNPC,snes);CHKERRQ(ierr);
  } else {
    ierr = MatMFFDSetFunction(*J,(PetscErrorCode (*)(void*,Vec,Vec))SNESComputeFunction,snes);CHKERRQ(ierr);
  }
}

  So, yes I would expect all the function evaluations needed for the matrix-free Jacobian matrix vector product to be counted. You can also look at the number of GMRES Krylov iterations it took (which should have one multiply per iteration) to double check that the numbers make sense.

  What does your -log_summary output look like? One thing that GMRES does is it introduces a global reduction with each multiple (hence a barrier across all your processes) on some systems this can be deadly.

  Barry


> 
> 2. In any case, I thought that all things considered, the function evaluation would be the most time consuming part of a Newton-Krylov solver, am I completely wrong about that ? Is the 10-20 factor legit ?
> 
> I realize of course that preconditioning should make all this smoother, in particular allowing larger time steps, but here I am just concerned about the sheer Function evaluation time.
> 
> Best regards
> 
> Timothee NICOLAS