[petsc-users] Performance of PETSc TS solver

[Jed Brown]

"Jin, Shuangshuang" <Shuangshuang.Jin at pnnl.gov> writes:

> Hello, I'm trying to update some of my status here. I just managed to" _distribute_ the work of computing the Jacobian matrix" as you suggested, so each processor only computes a part of elements for the Jacobian matrix instead of a global Jacobian matrix. I observed a reduction of the computation time from 351 seconds to 55 seconds, which is much better but still slower than I expected given the problem size is small. (4n functions in IFunction, and 4n*4n Jacobian matrix in IJacobian, n = 288).
>
> I looked at the log profile again, and saw that most of the computation time are still for Functioan Eval and Jacobian Eval:
>
> TSStep               600 1.0 5.6103e+01 1.0 9.42e+0825.6 3.0e+06 2.9e+02 7.0e+04 93100 99 99 92 152100 99 99110   279
> TSFunctionEval      2996 1.0 2.9608e+01 4.1 0.00e+00 0.0 0.0e+00 0.0e+00 3.0e+04 30  0  0  0 39  50  0  0  0 47     0

The load imbalance is pretty significant here, so maybe you can
distribute the work for residual evaluation better?

> TSJacobianEval      1796 1.0 2.3436e+01 1.0 0.00e+00 0.0 5.4e+02 3.8e+01 1.3e+04 39  0  0  0 16  64  0  0  0 20     0
> Warning -- total time of even greater than time of entire stage -- something is wrong with the timer

SNESSolve contains the Jacobian and residual evaluations, as well as
KSPSolve.  Pretty much all the cost is in those three things.

> SNESSolve            600 1.0 5.5692e+01 1.1 9.42e+0825.7 3.0e+06 2.9e+02 6.4e+04 88100 99 99 84 144100 99 99101   281
> SNESFunctionEval    2396 1.0 2.3715e+01 3.4 1.04e+06 1.0 0.0e+00 0.0e+00 2.4e+04 25  0  0  0 31  41  0  0  0 38     1
> SNESJacobianEval    1796 1.0 2.3447e+01 1.0 0.00e+00 0.0 5.4e+02 3.8e+01 1.3e+04 39  0  0  0 16  64  0  0  0 20     0
> SNESLineSearch      1796 1.0 1.8313e+01 1.0 1.54e+0831.4 4.9e+05 2.9e+02 2.5e+04 30 16 16 16 33  50 16 16 16 39   139
> KSPGMRESOrthog      9090 1.0 1.1399e+00 4.1 1.60e+07 1.0 0.0e+00 0.0e+00 9.1e+03  1  3  0  0 12   2  3  0  0 14   450
> KSPSetUp            3592 1.0 2.8342e-02 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 3.0e+01  0  0  0  0  0   0  0  0  0  0     0
> KSPSolve            1796 1.0 2.3052e+00 1.0 7.87e+0825.2 2.5e+06 2.9e+02 2.0e+04  4 84 83 83 26   6 84 83 83 31  5680
> PCSetUp             3592 1.0 9.1255e-02 1.7 6.47e+05 2.5 0.0e+00 0.0e+00 1.8e+01  0  0  0  0  0   0  0  0  0  0   159
> PCSetUpOnBlocks     1796 1.0 6.6802e-02 2.3 6.47e+05 2.5 0.0e+00 0.0e+00 1.2e+01  0  0  0  0  0   0  0  0  0  0   217
> PCApply            10886 1.0 2.6064e-01 1.3 4.70e+06 1.5 0.0e+00 0.0e+00 0.0e+00  0  1  0  0  0   1  1  0  0  0   481
>
> I was wondering why SNESFunctionEval and SNESJacobianEval took over 23
> seconds each, however, the KSPSolve only took 2.3 seconds, which is 10
> times faster. Is this normal? Do you have any more suggestion on how
> to reduce the FunctionEval and JacobianEval time?

It means that the linear systems are easy to solve (probably because
they are small), but the IFunction and IJacobian are expensive.  As
Barry says, you might be able to speed it up by sequential optimization.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 835 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20130830/a09e5589/attachment.pgp>