[petsc-users] saving results

Wed Feb 20 05:29:48 CST 2019

On Wed, Feb 20, 2019 at 4:43 AM Sal Am <tempohoper at gmail.com> wrote:

> Hi Matthew you were right,
>
> The matrix I have is very ill conditioned and my supervisor gave it for
> testing purposes. Having said that, I was able to solve it previously
> however, for some reason it said convergence reached at e-3 even though the
> default convergence tol is e-5 which is why I put rel_tol to e-7 and it
> crashed.
>
> Just to get something solving so you can check the discretization, I would
>>
>>   a) Use SuperLU or MUMPS, -pc_type lu
>>
>>   b) Then to get bigger use ASM/LU, -pc_type asm -pc_sub_type lu
>>
>> Then finally
>>
>>   c) For very big problems, I suspect that PCPATCHcould be made to work
>> with the right choices, but this is a research problem
>>
>
> That is very interesting, I have tried a) before and it does not work
> (lack of memory presumably). I will try to do b), but I am very interested
> in c). the final BOSS fight so to say is the 30million finite element model
> of the JET A2 Antenna so it would be great if I can utilise PETSc solving
> it. (Perhaps I should open another ticket for that later on)
>

Yep. The reason I thought it might work is that there is theory for it
working on positive semi-definite matrices (matrices which are
almost nice but can have a large null space). The idea is to capture the
local nullspace in each patch. There is an excellent writeup
of the idea, with all appropriate references, here:
https://arxiv.org/abs/1810.03315

But to stay on topic I have only used VecView to actually view the solution
> *after* it was done solving. Is there an example on how people actually
> utilise VecView or Jed's suggestion of KSPMonitor to have checkpoints
> *during* iterations so we can pick up wherever it crashed?
>

Sure. Here we set a custom monitor

https://bitbucket.org/petsc/petsc/src/6f3214441ec7b7dca9916d2bd112d01eef3e7185/src/ksp/ksp/examples/tutorials/ex9.c#lines-111

Just stick your VecView in there. In fact, I would use VecViewFromOptions()
so you can decide what viewer to use on the command line.

  Thanks,

     Matt

> Thanks
>
> On Tue, Feb 19, 2019 at 1:49 PM Matthew Knepley <knepley at gmail.com> wrote:
>
>> On Tue, Feb 19, 2019 at 8:21 AM Sal Am <tempohoper at gmail.com> wrote:
>>
>>> Matthew maybe indeed I was not clear and wrong in the wording. What I
>>> meant is the matrix is undefined, but since it is a finite element method
>>> (using second order elements). The pattern is like this:
>>>
>>
>> Okay, the paper says that the formulation of Maxwell in frequency space
>> is "well conditioned", and in the paper they solve using
>> BiCG/Jacobi:
>>
>>   -ksp_type bicg -pc_type jacobi
>>
>> Since you are taking thousands of iterates:
>>
>>   a) They are mistaken and the formulation is not well-conditioned
>>
>>   b) You have a bug in the discretization
>>
>>   c) You are outside the parameter range/boundary conditions/forcing they
>> were talking about for conditioning
>>
>> Just to get something solving so you can check the discretization, I would
>>
>>   a) Use SuperLU or MUMPS, -pc_type lu
>>
>>   b) Then to get bigger use ASM/LU, -pc_type asm -pc_sub_type lu
>>
>> Then finally
>>
>>   c) For very big problems, I suspect that PCPATCHcould be made to work
>> with the right choices, but this is a research problem
>>
>>   Thanks,
>>
>>      Matt
>>
>>
>>> [image: image.png]
>>> We are indeed solving Maxwell's equations. I have added the paper.
>>>
>>> On Tue, Feb 19, 2019 at 12:15 PM Matthew Knepley <knepley at gmail.com>
>>> wrote:
>>>
>>>> On Tue, Feb 19, 2019 at 4:12 AM Sal Am via petsc-users <
>>>> petsc-users at mcs.anl.gov> wrote:
>>>>
>>>>> It is a finite-element problem of an RF antenna dielectric interaction
>>>>> where all the non-zero elements are on the diagonal of the sparse matrix
>>>>> (if that is relevant).
>>>>>
>>>>
>>>> I am unsure whether this is what you mean. If you matrix is diagonal,
>>>> you can invert it exactly in a single step so you
>>>> do not need a Krylov solver. What equations are you solving? Maxwell?
>>>> Electrostatic? What finite element are you using?
>>>>   Thanks,
>>>>
>>>>     Matt
>>>>
>>>>
>>>>> More about the matrix: 25Mx25M, 2B non-zeros. I checked the
>>>>> KSPMonitor, it seems that I have to write my own routine?
>>>>>
>>>>>> Running for a Krylov method for tens of thousands of iterations is
>>>>>> very rarely recommended.
>>>>>>
>>>>> GAMG and BCGS is the only ones that have actually worked for me so
>>>>> far, I increased the max_it because it was not enough with the default one.
>>>>> With the default tolerance I got ||r(i)||/||b|| in the order of 10^-3, but
>>>>> I need more accuracy so I increased the tol to 10^-7 and that is when it
>>>>> crashed after ~51000 iterations.
>>>>>
>>>>> @Barry
>>>>>
>>>>>> I'm guessing you mange your own time stepping (and nonlinear solver
>>>>>> if there is one)?
>>>>>>
>>>>>
>>>>> It is the default from the solver (attached the short code).
>>>>>
>>>>>> As Jed says it doesn't make sense to save "partial" solutions within
>>>>>> the linear solver.  Just save it every several time-steps.
>>>>>>
>>>>>
>>>>> Yes maybe I worded it wrong. I did mean to save checkpoints basically
>>>>> such that the next time it is running it can pick up from where it crashed.
>>>>>
>>>>>    Also the code should not be "crashing" at seemingly long times
>>>>>> (after hours) with a Segmentation fault. Send us the full error message and
>>>>>> we'll see if there is some way we can help you fix this problem.
>>>>>>
>>>>> I have added this before, but seemingly they conclusion was that it is
>>>>> memory error...although I am not sure how, but I have added the entire
>>>>> error.
>>>>>
>>>>> On Mon, Feb 18, 2019 at 8:21 PM Smith, Barry F. <bsmith at mcs.anl.gov>
>>>>> wrote:
>>>>>
>>>>>>
>>>>>>    I'm guessing you mange your own time stepping (and nonlinear
>>>>>> solver if there is one)?
>>>>>>
>>>>>>    You can save the solution with a call to VecView() and then reload
>>>>>> the solution with a VecLoad() but you need to manage any other restart data
>>>>>> that you may need, like the value of the current time etc.
>>>>>>
>>>>>>    As Jed says it doesn't make sense to save "partial" solutions
>>>>>> within the linear solver.  Just save it every several time-steps.
>>>>>>
>>>>>>    Barry
>>>>>>
>>>>>>    Also the code should not be "crashing" at seemingly long times
>>>>>> (after hours) with a Segmentation fault. Send us the full error message and
>>>>>> we'll see if there is some way we can help you fix this problem.
>>>>>>
>>>>>>
>>>>>> > On Feb 18, 2019, at 9:47 AM, Sal Am via petsc-users <
>>>>>> petsc-users at mcs.anl.gov> wrote:
>>>>>> >
>>>>>> > Is there a function/command line option to save the solution as it
>>>>>> is solving (and read in the file from where it crashed and keep iterating
>>>>>> from there perhaps)?
>>>>>> > Had a seg fault and all the results until that point seems to have
>>>>>> been lost.
>>>>>> >
>>>>>> >
>>>>>> >
>>>>>> >
>>>>>>
>>>>>>
>>>>
>>>> --
>>>> What most experimenters take for granted before they begin their
>>>> experiments is infinitely more interesting than any results to which their
>>>> experiments lead.
>>>> -- Norbert Wiener
>>>>
>>>> https://www.cse.buffalo.edu/~knepley/
>>>> <http://www.cse.buffalo.edu/~knepley/>
>>>>
>>>
>>
>> --
>> What most experimenters take for granted before they begin their
>> experiments is infinitely more interesting than any results to which their
>> experiments lead.
>> -- Norbert Wiener
>>
>> https://www.cse.buffalo.edu/~knepley/
>> <http://www.cse.buffalo.edu/~knepley/>
>>
>

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190220/b2ccc135/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image.png
Type: image/png
Size: 14363 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190220/b2ccc135/attachment-0001.png>