[petsc-users] Performance of PETSc TS solver

Tue Aug 13 18:30:27 CDT 2013

Hi, Shri,

>From the log_summary, we can see that the TSJacobianEval/SNESJacobianEval dominates the computation time as you mentioned.

Event                Count      Time (sec)     Flops                             --- Global ---  --- Stage ---   Total
                   Max Ratio  Max     Ratio   Max  Ratio  Mess   Avg len Reduct  %T %f %M %L %R  %T %f %M %L %R Mflop/s
------------------------------------------------------------------------------------------------------------------------
TSJacobianEval      1782 1.0 3.1640e+02 1.0 0.00e+00 0.0 2.0e+03 7.4e+01 1.4e+04 90  0  0  0 21  90  0  0  0 21     0
SNESJacobianEval    1782 1.0 3.1641e+02 1.0 0.00e+00 0.0 2.0e+03 7.4e+01 1.4e+04 90  0  0  0 21  90  0  0  0 21     0

It takes 316 seconds for the total Jacobian Eval out of the 350 seconds of SNESSolve,which is the total simulation time.

So I look into my IJacobian Function. However, all it does is forming a 1152*1152 dimension Jacobian matrix. I don't understand why it takes such a long time. And if it's the reason of poor performance, is there any way to polish it to accelerate the speed in this part?

I also noticed that the KSPSolve part in the DAE process only takes 8.56 seconds:

KSPSolve            1782 1.0 8.5647e+00 1.0 2.28e+09 1.0 6.5e+06 5.8e+02 1.5e+04  2 79 78 78 22   2 79 78 78 22  8505

Comparing to the 316 seconds for JacobianEval, it's pretty fast. Does that also imply the default KSP solver is good enough to solve the problem? And I should focus on reduce the IJacobian Eval time without worrying about different KSP solver here?

Thanks,
Shuangshuang

-----Original Message-----
From: Shri [mailto:abhyshr at mcs.anl.gov] 
Sent: Tuesday, August 13, 2013 11:01 AM
To: Jin, Shuangshuang
Cc: Barry Smith; Jed Brown; petsc-users at mcs.anl.gov
Subject: Re: [petsc-users] Performance of PETSc TS solver

90% of the time is spent in your Jacobian evaluation routine which is clearly a bottleneck.

On Aug 13, 2013, at 12:34 PM, Jin, Shuangshuang wrote:

> Hello, Jed and Barry, thanks for your reply.
> 
> We are solving a power system dynamic simulation problem. We set up the DAE equations and its Jacobian matrix, and would like to use the Trapezoid method to solve it. 
> 
> That's also the reason why we chose TSTHETA. From the PETSc manual, we read that:
> 
> "-ts_type theta -ts_theta_theta 0.5 -ts_theta_endpoint corresponds to Crank-Nicholson (TSCN). This method can be applied to DAE.
> For the default Theta=0.5, this is the trapezoid rule (also known as Crank-Nicolson, see TSCN)."
> 
> I haven't heard of ARKIMEX or ROSW before. Are they some external packages or DAE solvers that implement the Trapezoid method?
> 
> I have also tried the -ksp_type preonly -pc_type lu option you indicated but failed. The PETSC ERROR messages are: No support for this operation for this object type! Matrix format mpiaij does not have a built-in PETSc LU!
PETSc does not have a native parallel direct solver. You can use MUMPS (--download-mumps) or superlu_dist (--download_superlu_dist)
> 
> Attached please see the log_summary for running the TSTHETA with -ts_theta_theta 0.5 and its default ksp solver. Please help me to evaluate the performance and see what's the bottleneck of the slow computation speed.
> 
> Thanks a lot!
> 
> Shuangshuang
> 
> 
> 
> 
> 
> 
> -----Original Message-----
> From: Barry Smith [mailto:bsmith at mcs.anl.gov]
> Sent: Monday, August 12, 2013 6:39 PM
> To: Jed Brown
> Cc: Jin, Shuangshuang; petsc-users at mcs.anl.gov
> Subject: Re: [petsc-users] Performance of PETSc TS solver
> 
> 
>   Also always send the output from running with -log_summary whenever you ask performance questions so we know what kind of performance it is getting.
> 
>   Barry
> 
> On Aug 12, 2013, at 8:14 PM, Jed Brown <jedbrown at mcs.anl.gov> wrote:
> 
>> "Jin, Shuangshuang" <Shuangshuang.Jin at pnnl.gov> writes:
>> 
>>> Hello, PETSc developers,
>>>       I have a question regarding the performance of PETSc TS solver
>>>       expecially the TSTHETA. I used it to solve my DAE equations. 
>> 
>> TSTHETA is not L-stable and not stiffly accurate, so it's not 
>> normally something that you'd want to use for a DAE.  Make sure 
>> you're getting meaningful results and try switching to something like 
>> an ARKIMEX or ROSW since those are likely better for your problem.
>> 
>>> I have recorded the solution times when different numbers of processors are used:
>>> 
>>> 2 processors: 1021 seconds,
>>> 4 processors: 587.244 seconds,
>>> 8 processors: 421.565 seconds,
>>> 16 processors: 355.594 seconds,
>>> 32 processors: 322.28 seconds,
>>> 64 processors: 382.967 seconds.
>>> 
>>> It seems like with 32 processors, it reaches the best performance. 
>>> However, 322.28 seconds to solve such DAE equations is too slow than 
>>> I expected.
>> 
>> The number of equations (1152) is quite small, so I'm not surprised 
>> there is no further speedup.  Can you explain more about your equations?
>> 
>>> 
>>> I have the following questions based on the above results:
>>> 1.      Is this the usual DAE solving time in PETSc to for the problem with this dimension?
>> 
>> That depends what your function is.
>> 
>>> 2.      I was told that in TS, by default, ksp uses GMRES, and the
>>> preconditioner is ILU(0), is there any other alterative ksp solver 
>>> or options I should use in the command line to solve the problem 
>>> much faster?
>> 
>> I would use -ksp_type preonly -pc_type lu for such small problems.  
>> Is the system dense?
>> 
>>> 3.      Do you have any other suggestion for me to speed up the DAE computation in PETSc?
>> 
>> Can you describe what sort of problem you're dealing with, what 
>> causes the stiffness in your equations, what accuracy you want, etc.
> 
> <job.out.summary>