[petsc-users] saving results

Wed Feb 20 03:43:18 CST 2019

Hi Matthew you were right,

The matrix I have is very ill conditioned and my supervisor gave it for
testing purposes. Having said that, I was able to solve it previously
however, for some reason it said convergence reached at e-3 even though the
default convergence tol is e-5 which is why I put rel_tol to e-7 and it
crashed.

Just to get something solving so you can check the discretization, I would
>
>   a) Use SuperLU or MUMPS, -pc_type lu
>
>   b) Then to get bigger use ASM/LU, -pc_type asm -pc_sub_type lu
>
> Then finally
>
>   c) For very big problems, I suspect that PCPATCHcould be made to work
> with the right choices, but this is a research problem
>

That is very interesting, I have tried a) before and it does not work (lack
of memory presumably). I will try to do b), but I am very interested in c).
the final BOSS fight so to say is the 30million finite element model of the
JET A2 Antenna so it would be great if I can utilise PETSc solving it.
(Perhaps I should open another ticket for that later on)

But to stay on topic I have only used VecView to actually view the solution
*after* it was done solving. Is there an example on how people actually
utilise VecView or Jed's suggestion of KSPMonitor to have checkpoints
*during* iterations so we can pick up wherever it crashed?

Thanks

On Tue, Feb 19, 2019 at 1:49 PM Matthew Knepley <knepley at gmail.com> wrote:

> On Tue, Feb 19, 2019 at 8:21 AM Sal Am <tempohoper at gmail.com> wrote:
>
>> Matthew maybe indeed I was not clear and wrong in the wording. What I
>> meant is the matrix is undefined, but since it is a finite element method
>> (using second order elements). The pattern is like this:
>>
>
> Okay, the paper says that the formulation of Maxwell in frequency space is
> "well conditioned", and in the paper they solve using
> BiCG/Jacobi:
>
>   -ksp_type bicg -pc_type jacobi
>
> Since you are taking thousands of iterates:
>
>   a) They are mistaken and the formulation is not well-conditioned
>
>   b) You have a bug in the discretization
>
>   c) You are outside the parameter range/boundary conditions/forcing they
> were talking about for conditioning
>
> Just to get something solving so you can check the discretization, I would
>
>   a) Use SuperLU or MUMPS, -pc_type lu
>
>   b) Then to get bigger use ASM/LU, -pc_type asm -pc_sub_type lu
>
> Then finally
>
>   c) For very big problems, I suspect that PCPATCHcould be made to work
> with the right choices, but this is a research problem
>
>   Thanks,
>
>      Matt
>
>
>> [image: image.png]
>> We are indeed solving Maxwell's equations. I have added the paper.
>>
>> On Tue, Feb 19, 2019 at 12:15 PM Matthew Knepley <knepley at gmail.com>
>> wrote:
>>
>>> On Tue, Feb 19, 2019 at 4:12 AM Sal Am via petsc-users <
>>> petsc-users at mcs.anl.gov> wrote:
>>>
>>>> It is a finite-element problem of an RF antenna dielectric interaction
>>>> where all the non-zero elements are on the diagonal of the sparse matrix
>>>> (if that is relevant).
>>>>
>>>
>>> I am unsure whether this is what you mean. If you matrix is diagonal,
>>> you can invert it exactly in a single step so you
>>> do not need a Krylov solver. What equations are you solving? Maxwell?
>>> Electrostatic? What finite element are you using?
>>>   Thanks,
>>>
>>>     Matt
>>>
>>>
>>>> More about the matrix: 25Mx25M, 2B non-zeros. I checked the KSPMonitor,
>>>> it seems that I have to write my own routine?
>>>>
>>>>> Running for a Krylov method for tens of thousands of iterations is
>>>>> very rarely recommended.
>>>>>
>>>> GAMG and BCGS is the only ones that have actually worked for me so far,
>>>> I increased the max_it because it was not enough with the default one. With
>>>> the default tolerance I got ||r(i)||/||b|| in the order of 10^-3, but I
>>>> need more accuracy so I increased the tol to 10^-7 and that is when it
>>>> crashed after ~51000 iterations.
>>>>
>>>> @Barry
>>>>
>>>>> I'm guessing you mange your own time stepping (and nonlinear solver if
>>>>> there is one)?
>>>>>
>>>>
>>>> It is the default from the solver (attached the short code).
>>>>
>>>>> As Jed says it doesn't make sense to save "partial" solutions within
>>>>> the linear solver.  Just save it every several time-steps.
>>>>>
>>>>
>>>> Yes maybe I worded it wrong. I did mean to save checkpoints basically
>>>> such that the next time it is running it can pick up from where it crashed.
>>>>
>>>>    Also the code should not be "crashing" at seemingly long times
>>>>> (after hours) with a Segmentation fault. Send us the full error message and
>>>>> we'll see if there is some way we can help you fix this problem.
>>>>>
>>>> I have added this before, but seemingly they conclusion was that it is
>>>> memory error...although I am not sure how, but I have added the entire
>>>> error.
>>>>
>>>> On Mon, Feb 18, 2019 at 8:21 PM Smith, Barry F. <bsmith at mcs.anl.gov>
>>>> wrote:
>>>>
>>>>>
>>>>>    I'm guessing you mange your own time stepping (and nonlinear solver
>>>>> if there is one)?
>>>>>
>>>>>    You can save the solution with a call to VecView() and then reload
>>>>> the solution with a VecLoad() but you need to manage any other restart data
>>>>> that you may need, like the value of the current time etc.
>>>>>
>>>>>    As Jed says it doesn't make sense to save "partial" solutions
>>>>> within the linear solver.  Just save it every several time-steps.
>>>>>
>>>>>    Barry
>>>>>
>>>>>    Also the code should not be "crashing" at seemingly long times
>>>>> (after hours) with a Segmentation fault. Send us the full error message and
>>>>> we'll see if there is some way we can help you fix this problem.
>>>>>
>>>>>
>>>>> > On Feb 18, 2019, at 9:47 AM, Sal Am via petsc-users <
>>>>> petsc-users at mcs.anl.gov> wrote:
>>>>> >
>>>>> > Is there a function/command line option to save the solution as it
>>>>> is solving (and read in the file from where it crashed and keep iterating
>>>>> from there perhaps)?
>>>>> > Had a seg fault and all the results until that point seems to have
>>>>> been lost.
>>>>> >
>>>>> >
>>>>> >
>>>>> >
>>>>>
>>>>>
>>>
>>> --
>>> What most experimenters take for granted before they begin their
>>> experiments is infinitely more interesting than any results to which their
>>> experiments lead.
>>> -- Norbert Wiener
>>>
>>> https://www.cse.buffalo.edu/~knepley/
>>> <http://www.cse.buffalo.edu/~knepley/>
>>>
>>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
> https://www.cse.buffalo.edu/~knepley/
> <http://www.cse.buffalo.edu/~knepley/>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190220/405f75f5/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image.png
Type: image/png
Size: 14363 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190220/405f75f5/attachment-0001.png>