<div dir="ltr"><div>Hi Matthew you were right, <br></div><div><br></div><div>The matrix I have is very ill conditioned and my supervisor gave it for testing purposes. Having said that, I was able to solve it previously however, for some reason it said convergence reached at e-3 even though the default convergence tol is e-5 which is why I put rel_tol to e-7 and it crashed. <br></div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div>
<div>Just to get something solving so you can check the discretization, I would</div><div><br></div><div> a) Use SuperLU or MUMPS, -pc_type lu</div><div><br></div><div> b) Then to get bigger use ASM/LU, -pc_type asm -pc_sub_type lu</div><div><br></div><div>Then finally</div><div><br></div><div> c) For very big problems, I suspect that PCPATCHcould be made to work with the right choices, but this is a research problem</div></div></blockquote><div><br></div><div>That is very interesting, I have tried a) before and it does not work (lack of memory presumably). I will try to do b), but I am very interested in c). the final BOSS fight so to say is the 30million finite element model of the JET A2 Antenna so it would be great if I can utilise PETSc solving it. (Perhaps I should open another ticket for that later on) <br></div><div><br></div><div>But to stay on topic I have only used VecView to actually view the solution <i>after</i> it was done solving. Is there an example on how people actually utilise VecView or Jed's suggestion of KSPMonitor to have checkpoints <i>during</i> iterations so we can pick up wherever it crashed?</div><div><br></div><div>Thanks<br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, Feb 19, 2019 at 1:49 PM Matthew Knepley <<a href="mailto:knepley@gmail.com">knepley@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div dir="ltr">On Tue, Feb 19, 2019 at 8:21 AM Sal Am <<a href="mailto:tempohoper@gmail.com" target="_blank">tempohoper@gmail.com</a>> wrote:<br></div><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>Matthew maybe indeed I was not clear and wrong in the wording. What I meant is the matrix is undefined, but since it is a finite element method (using second order elements). The pattern is like this: <br></div></div></blockquote><div><br></div><div>Okay, the paper says that the formulation of Maxwell in frequency space is "well conditioned", and in the paper they solve using</div><div>BiCG/Jacobi:</div><div><br></div><div> -ksp_type bicg -pc_type jacobi</div><div><br></div><div>Since you are taking thousands of iterates:</div><div><br></div><div> a) They are mistaken and the formulation is not well-conditioned</div><div><br></div><div> b) You have a bug in the discretization</div><div><br></div><div> c) You are outside the parameter range/boundary conditions/forcing they were talking about for conditioning</div><div><br></div><div>Just to get something solving so you can check the discretization, I would</div><div><br></div><div> a) Use SuperLU or MUMPS, -pc_type lu</div><div><br></div><div> b) Then to get bigger use ASM/LU, -pc_type asm -pc_sub_type lu</div><div><br></div><div>Then finally</div><div><br></div><div> c) For very big problems, I suspect that PCPATCHcould be made to work with the right choices, but this is a research problem</div><div><br></div><div> Thanks,</div><div><br></div><div> Matt</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div></div><div><div><img src="cid:ii_jsbsn1kc0" alt="image.png" width="561" height="421"></div><div>We are indeed solving Maxwell's equations. I have added the paper. <br></div></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, Feb 19, 2019 at 12:15 PM Matthew Knepley <<a href="mailto:knepley@gmail.com" target="_blank">knepley@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div dir="ltr">On Tue, Feb 19, 2019 at 4:12 AM Sal Am via petsc-users <<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a>> wrote:<br></div><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">
<div>It is a finite-element problem of an RF antenna dielectric
interaction where all the non-zero elements are on the diagonal of the
sparse matrix (if that is relevant).</div><div></div></div></blockquote><div><br></div><div>I am unsure whether this is what you mean. If you matrix is diagonal, you can invert it exactly in a single step so you</div><div>do not need a Krylov solver. What equations are you solving? Maxwell? Electrostatic? What finite element are you using?</div><div> Thanks,</div><div><br></div><div> Matt</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>More about the matrix: 25Mx25M, 2B non-zeros. I checked the KSPMonitor, it seems that I have to write my own routine? <br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div>
Running for a Krylov method for tens of thousands of iterations is very rarely recommended.<br></div></blockquote><div>GAMG and BCGS is the only ones that have actually worked for me so far, I increased the max_it because it was not enough with the default one. With the default tolerance I got ||r(i)||/||b|| in the order of 10^-3, but I need more accuracy so I increased the tol to 10^-7 and that is when it crashed after ~51000 iterations.</div><div><br></div><div>@Barry</div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div>
I'm guessing you mange your own time stepping (and nonlinear solver if there is one)?
</div></blockquote><div><br></div><div>It is the default from the solver (attached the short code).</div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div>
As Jed says it doesn't make sense to save "partial" solutions within the linear solver. Just save it every several time-steps.
</div></blockquote><div> </div><div>Yes maybe I worded it wrong. I did mean to save checkpoints basically such that the next time it is running it can pick up from where it crashed.</div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div>
Also the code should not be "crashing" at seemingly long times (after
hours) with a Segmentation fault. Send us the full error message and
we'll see if there is some way we can help you fix this problem.
</div></blockquote><div>I have added this before, but seemingly they conclusion was that it is memory error...although I am not sure how, but I have added the entire error. <br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, Feb 18, 2019 at 8:21 PM Smith, Barry F. <<a href="mailto:bsmith@mcs.anl.gov" target="_blank">bsmith@mcs.anl.gov</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><br>
I'm guessing you mange your own time stepping (and nonlinear solver if there is one)? <br>
<br>
You can save the solution with a call to VecView() and then reload the solution with a VecLoad() but you need to manage any other restart data that you may need, like the value of the current time etc. <br>
<br>
As Jed says it doesn't make sense to save "partial" solutions within the linear solver. Just save it every several time-steps.<br>
<br>
Barry<br>
<br>
Also the code should not be "crashing" at seemingly long times (after hours) with a Segmentation fault. Send us the full error message and we'll see if there is some way we can help you fix this problem.<br>
<br>
<br>
> On Feb 18, 2019, at 9:47 AM, Sal Am via petsc-users <<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a>> wrote:<br>
> <br>
> Is there a function/command line option to save the solution as it is solving (and read in the file from where it crashed and keep iterating from there perhaps)?<br>
> Had a seg fault and all the results until that point seems to have been lost. <br>
> <br>
> <br>
> <br>
> <br>
<br>
</blockquote></div>
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr" class="gmail-m_8695453846741553527gmail-m_-5359575751150626700gmail-m_-7287144247319992673gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>-- Norbert Wiener</div><div><br></div><div><a href="http://www.cse.buffalo.edu/~knepley/" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br></div></div></div></div></div></div></div></div>
</blockquote></div>
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr" class="gmail-m_8695453846741553527gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>-- Norbert Wiener</div><div><br></div><div><a href="http://www.cse.buffalo.edu/~knepley/" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br></div></div></div></div></div></div></div></div>
</blockquote></div>