<div dir="ltr"><div dir="ltr">On Fri, Aug 20, 2021 at 6:55 AM dazza simplythebest <<a href="mailto:sayosale@hotmail.com">sayosale@hotmail.com</a>> wrote:<br></div><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="ltr">
<div style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
Dear Jose,</div>
<div style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
Many thanks for your response, I have been investigating this issue with a few more calculations
<br>
</div>
<div style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
today, hence the slightly delayed response.</div>
<div style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
The problem is actually derived from a fluid dynamics problem, so to allow an easier exploration of things
<br>
</div>
<div style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
I first downsized the resolution of the underlying fluid solver while keeping all the physical parameters</div>
<div style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
the same - i.e. I would get a smaller matrix that should be solving the same physical problem as the original</div>
<div style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
larger matrix but to lower accuracy. <br>
</div>
<div style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<u><b>Results</b></u></div>
<div style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<b>Small matrix (N= 21168) - everything good!</b><br>
</div>
<div style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
This converged when using the -eps_largest_real approach (taking 92 iterations for nev=10,<br>
</div>
<div style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
tol= 5.0000E-06 and ncv = 300), and also when using the shift-invert approach, converging
<br>
</div>
<div style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
very impressively in a single iteration ! Interestingly it did this both for a non-zero -eps_target<br>
</div>
<div style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
and also for a zero -eps_target.<br>
</div>
<div style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<div>
<div id="gmail-m_-6879576846774134416appendonsend"></div>
<div style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<b>Large matrix (N=50400)- works for -eps_largest_real , fails for st_type sinvert
</b><br>
</div>
<div style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
I have just double checked again that the code does run properly when we use the -eps_largest_real
<br>
</div>
<div style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
option - indeed I ran it with a small nev and large tolerance (nev = 4, tol= -eps_tol 5.0e-4 , ncv = 300)</div>
<div style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
and with these parameters convergence was obtained in 164 iterations, which took 6 hours on the
<br>
</div>
<div style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
machine I was running it on. Furthermore the eigenvalues seem to be ballpark correct; for this large</div>
<div style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
higher resolution case (although with lower slepc tolerance) we obtain 1789.56816314173 -4724.51319554773i</div>
<div style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
as the eigenvalue with largest real part, while the smaller matrix (same physical problem but at lower resolution case)</div>
<div style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
found this eigenvalue to be 1831.11845726501 -4787.54519511345i , which means the agreement is in line</div>
<div style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
with expectations.<br>
</div>
<div style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<i>Unfortunately though the code does still crash though when I try to do shift-invert for the large matrix case
</i>,</div>
<div style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
whether or not I use a non-zero -eps_target. For reference this is the command line used :</div>
<div style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
-eps_nev 10 -eps_ncv 300 -log_view -eps_view -eps_target 0.1 -st_type sinvert -eps_monitor :monitor_output05.txt
<br>
</div>
<div style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
To be precise the code crashes soon after calling EPSSolve (it successfully calls
<br>
</div>
<div style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
MatCreateVecs, EPSCreate, EPSSetOperators, EPSSetProblemType and EPSSetFromOptions).</div>
<div style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
By crashes I mean that I do not even get any error messages from slepc/PETSC, and do not even get the
<br>
</div>
<div style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
'EPS Object: 16 MPI processes' message - I simply get a MPI/Fortran 'KILLED BY SIGNAL: 9 (Killed)' message</div>
<div style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
as soon as EPSsolve is called.</div></div></div></blockquote><div><br></div><div>Hi Dan,</div><div><br></div><div>It would help track this error down if we had a stack trace. You can get a stack trace from the debugger. You run with</div><div><br></div><div> -start_in_debugger</div><div><br></div><div>which should launch the debugger (usually), and then type</div><div><br></div><div> cont</div><div><br></div><div>to continue, and then</div><div><br></div><div> where</div><div><br></div><div>to get the stack trace when it crashes, or 'bt' on lldb.</div><div><br></div><div> Thanks,</div><div><br></div><div> Matt</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>
<div style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
Do you have any ideas as to why this larger matrix case should fail when using shift-invert but succeed when using
<br>
</div>
<div style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
-eps_largest_real ? The fact that the program works and produces correct results <br>
</div>
<div style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
when using the -eps_largest_real option suggests that there is probably nothing wrong with the specification
<br>
</div>
<div style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
of the problem or the matrices ? It is strange how there is no error message from slepc / Petsc ... the
<br>
</div>
<div style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
only idea I have at the moment is that perhaps max memory has been exceeded, which could cause such a sudden
<br>
</div>
<div style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
shutdown? For your reference when running the large matrix case with the -eps_largest_real option I am using
<br>
</div>
<div style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
about 36 GB of the 148GB available on this machine - does the shift invert approach require substantially
<br>
</div>
<div style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
more memory for example ?<br>
</div>
<div style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
I would be very grateful if you have any suggestions to resolve this issue or even ways to clarify it further,</div>
<div style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
the performance I have seen with the shift-invert for the small matrix is so impressive it would be great to</div>
<div style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
get that working for the full-size problem.<br>
</div>
<div style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<div><br>
</div>
<div> Many thanks and best wishes,</div>
<div> Dan.<br>
</div>
</div>
<div style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<hr style="display:inline-block;width:98%">
<div id="gmail-m_-6879576846774134416divRplyFwdMsg" dir="ltr"><font style="font-size:11pt" face="Calibri, sans-serif" color="#000000"><b>From:</b> Jose E. Roman <<a href="mailto:jroman@dsic.upv.es" target="_blank">jroman@dsic.upv.es</a>><br>
<b>Sent:</b> Thursday, August 19, 2021 7:58 AM<br>
<b>To:</b> dazza simplythebest <<a href="mailto:sayosale@hotmail.com" target="_blank">sayosale@hotmail.com</a>><br>
<b>Cc:</b> PETSc <<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a>><br>
<b>Subject:</b> Re: [petsc-users] Improving efficiency of slepc usage</font>
<div> </div>
</div>
<div><font size="2"><span style="font-size:11pt">
<div>In A) convergence may be slow, especially if the wanted eigenvalues have small magnitude. I would not say 600 iterations is a lot, you probably need many more. In most cases, approach B) is better because it improves convergence of eigenvalues
close to the target, but it requires prior knowledge of your spectrum distribution in order to choose an appropriate target.<br>
<br>
In B) what do you mean that it crashes. If you get an error about factorization, it means that your A-matrix is singular, In that case, try using a nonzero target -eps_target 0.1<br>
<br>
Jose<br>
<br>
<br>
> El 19 ago 2021, a las 7:12, dazza simplythebest <<a href="mailto:sayosale@hotmail.com" target="_blank">sayosale@hotmail.com</a>> escribió:<br>
> <br>
> Dear All,<br>
> I am planning on using slepc to do a large number of eigenvalue calculations<br>
> of a generalized eigenvalue problem, called from a program written in fortran using MPI.<br>
> Thus far I have successfully installed the slepc/PETSc software, both locally and on a cluster,<br>
> and on smaller test problems everything is working well; the matrices are efficiently and
<br>
> correctly constructed and slepc returns the correct spectrum. I am just now starting to move<br>
> towards now solving the full-size 'production run' problems, and would appreciate some
<br>
> general advice on how to improve the solver's performance.<br>
> <br>
> In particular, I am currently trying to solve the problem Ax = lambda Bx whose matrices
<br>
> are of size 50000 (this is the smallest 'production run' problem I will be tackling), and are
<br>
> complex, non-Hermitian. In most cases I aim to find the eigenvalues with the largest real part,
<br>
> although in other cases I will also be interested in finding the eigenvalues whose real part
<br>
> is close to zero.<br>
> <br>
> A)<br>
> Calling slepc 's EPS solver with the following options:<br>
> <br>
> -eps_nev 10 -log_view -eps_view -eps_max_it 600 -eps_ncv 140 -eps_tol 5.0e-6 -eps_largest_real -eps_monitor :monitor_output.txt<br>
> <br>
> <br>
> led to the code successfully running, but failing to find any eigenvalues within the maximum 600 iterations
<br>
> (examining the monitor output it did appear to be very slowly approaching convergence).<br>
> <br>
> B)<br>
> On the same problem I have also tried a shift-invert transformation using the options<br>
> <br>
> -eps_nev 10 -eps_ncv 140 -eps_target 0.0+0.0i -st_type sinvert<br>
> <br>
> -in this case the code crashed at the point it tried to call slepc, so perhaps I have incorrectly specified these options ?<br>
> <br>
> <br>
> Does anyone have any suggestions as to how to improve this performance ( or find out more about the problem) ?<br>
> In the case of A) I can see from watching the slepc videos that increasing ncv
<br>
> may help, but I am wondering , since 600 is a large number of iterations, whether there
<br>
> maybe something else going on - e.g. perhaps some alternative preconditioner may help ?<br>
> In the case of B), I guess there must be some mistake in these command line options?<br>
> Again, any advice will be greatly appreciated.<br>
> Best wishes, Dan.<br>
<br>
</div>
</span></font></div>
</div>
</div>
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr" class="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>-- Norbert Wiener</div><div><br></div><div><a href="http://www.cse.buffalo.edu/~knepley/" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br></div></div></div></div></div></div></div></div>