[petsc-users] TSSolve problems

[Li, Xinya]

Thank you both very much.
I will modify the codes according to the suggestions and retest the performance.


xinya

-----Original Message-----
From: Abhyankar, Shrirang G. [mailto:abhyshr at anl.gov] 
Sent: Tuesday, June 30, 2015 7:00 AM
To: Li, Xinya
Cc: petsc-users at mcs.anl.gov
Subject: Re: [petsc-users] TSSolve problems

Note also that the use of DMDA is incorrect in your example. DMDA is for managing structured grids, while your example application (power grid dynamics simulation) has an unstructured network. Using DMNetwork or DMPlex would be more appropriate.

Shri

-----Original Message-----
From: barry smith <bsmith at mcs.anl.gov>
Date: Monday, June 29, 2015 at 10:28 PM
To: "Li, Xinya" <Xinya.Li at pnnl.gov>
Cc: "petsc-users at mcs.anl.gov" <petsc-users at mcs.anl.gov>
Subject: Re: [petsc-users] TSSolve problems

>
>> On Jun 29, 2015, at 1:35 PM, Li, Xinya <Xinya.Li at pnnl.gov> wrote:
>> 
>> Yes. it is C++.
>> I attached the major routines for TSSolve.
>
>   Thanks. There is not much in the functions to explain why they are 
>taking the vast bulk of the time but I am going to go out on a limb and 
>guess that it is the C++ implementation of complex numbers and the use 
>of the cos() and sin() of those complex numbers that it causing the 
>terrible performance.
>
>  Since your code has actually very little C++ specific stuff in it I 
>would suggest building PETSc without the --with-clanguage=c++ and use C 
>for everything, changing your code as needed to remove the C++ isms. In
>C99 complex is a native data type (not a C++ class) so can potentially 
>give better performance.
>
>  Barry
>
>> 
>> Thanks
>> Xinya
>> 
>> 
>> -----Original Message-----
>> From: Barry Smith [mailto:bsmith at mcs.anl.gov]
>> Sent: Monday, June 29, 2015 11:27 AM
>> To: Li, Xinya
>> Cc: petsc-users at mcs.anl.gov
>> Subject: Re: [petsc-users] TSSolve problems
>> 
>> 
>>  Is your code C++?
>> 
>>                             count        time               percent time
>> ---------------------------------------------------------------------
>> TSStep                     600    3.1174e+02    100
>> TSFunctionEval      2937    1.4288e+02     46
>> TSJacobianEval      1737    1.3074e+02    42
>> KSPSolve                1737    3.5144e+01    11
>> 
>>  Ok I pulled out the important time from the table.  46 percent of 
>>the time is in your function evaluation, 42 percent in the Jacobian 
>>evaluation and 11 percent in the linear solve.
>> 
>>  The only way to improve the time significantly is by speeding up the 
>>function and Jacobian computations. What is happening in those 
>>routines, can you email them?
>> 
>>  Barry
>> 
>> 
>> 
>> 
>> 
>>> On Jun 29, 2015, at 11:57 AM, Li, Xinya <Xinya.Li at pnnl.gov> wrote:
>>> 
>>> Barry,
>>> 
>>> Here is the new output without debugging.
>>> 
>>> Thank you.
>>> 
>>> Xinya
>>> 
>>> ********************************************************************
>>> 
>>> 
>>> -----Original Message-----
>>> From: Barry Smith [mailto:bsmith at mcs.anl.gov]
>>> Sent: Friday, June 26, 2015 12:04 PM
>>> To: Li, Xinya
>>> Cc: petsc-users at mcs.anl.gov
>>> Subject: Re: [petsc-users] TSSolve problems
>>> 
>>> 
>>> ##########################################################
>>>     #                                                        #
>>>     #                          WARNING!!!                    #
>>>     #                                                        #
>>>     #   This code was compiled with a debugging option,      #
>>>     #   To get timing results run ./configure                #
>>>     #   using --with-debugging=no, the performance will      #
>>>     #   be generally two or three times faster.              #
>>>     #                                                        #
>>>     ##########################################################
>>> 
>>> First you need to configure PETSc again without all the debugging. 
>>>So do, for example,
>>> 
>>> export PETSC=arch-opt
>>> ./configure --with-cc=gcc --with-fc=gfortran --with-cxx=g++ 
>>>--with-scalar-type=complex --with-clanguage=C++
>>>--with-cxx-dialect=C++11  --download-mpich --download-superlu_dist 
>>>--download-mumps --download-scalapack --download-parmetis 
>>>--download-metis --download-elemental make all test
>>> 
>>> then recompile and rerun your example again with -log_summary and 
>>>send the output.
>>> 
>>> Note that you should not pass --download-fblaslapack nor the fortran 
>>>kernel stuff.
>>> 
>>> Barry
>>> 
>>> 
>>>> On Jun 26, 2015, at 12:16 PM, Li, Xinya <Xinya.Li at pnnl.gov> wrote:
>>>> 
>>>> Barry,
>>>> 
>>>> Thank you for your response.
>>>> 
>>>> Attached is the output. SNESSolve was taking most of the time.
>>>> 
>>>> 
>>>> Xinya
>>>> 
>>>> 
>>>> 
>>>> -----Original Message-----
>>>> From: Barry Smith [mailto:bsmith at mcs.anl.gov]
>>>> Sent: Thursday, June 25, 2015 5:47 PM
>>>> To: Li, Xinya
>>>> Cc: petsc-users at mcs.anl.gov
>>>> Subject: Re: [petsc-users] TSSolve problems
>>>> 
>>>> 
>>>> Run with -ts_view -log_summary and send the output. This will tell 
>>>>the current solvers and where the time is being spent.
>>>> 
>>>> Barry
>>>> 
>>>>> On Jun 25, 2015, at 6:37 PM, Li, Xinya <Xinya.Li at pnnl.gov> wrote:
>>>>> 
>>>>> Dear Sir,
>>>>> 
>>>>> I am using the ts solver to solve a set of ODE and DAE. The 
>>>>>Jacobian matrix is a 1152 *1152 sparse complex matrix.
>>>>> Each TSStep in TSSolve is taking nearly 1 second. Thus, I need  to 
>>>>>improve the speed of TSSolve.
>>>>> Which parts should be taking into account to accelerate TSSolve?
>>>>> Thank you very much.
>>>>> Regards
>>>>> __________________________________________________
>>>>> Xinya Li
>>>>> Scientist
>>>>> EED/Hydrology
>>>>> Pacific Northwest National Laboratory
>>>>> 902 Battelle Boulevard
>>>>> P.O. Box 999, MSIN K9-33
>>>>> Richland, WA  99352 USA
>>>>> Tel:  509-372-6248
>>>>> Fax: 509-372-6089
>>>>> Xinya.Li at pnl.gov
>>>> 
>>>> <288g1081b_short.log>
>>> 
>>> <d288gen.log>
>> 
>> <simulation.C>
>