<div dir="ltr">Dear Matthew,<div><br></div><div>Thanks for your answer, but this is something I do not really know much about... The node I used has 12 cores and about 24GB of RAM. <br><br>But for these test cases, isn't the distribution of memory over cores handled automatically by SLEPC?<br><br>Regards</div></div><br><div class="gmail_quote"><div dir="ltr">On Tue, Apr 4, 2017 at 1:40 PM Matthew Knepley <<a href="mailto:knepley@gmail.com">knepley@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr" class="gmail_msg"><div class="gmail_extra gmail_msg"><div class="gmail_quote gmail_msg">On Tue, Apr 4, 2017 at 2:20 AM, Toon Weyens <span dir="ltr" class="gmail_msg"><<a href="mailto:toon.weyens@gmail.com" class="gmail_msg" target="_blank">toon.weyens@gmail.com</a>></span> wrote:<br class="gmail_msg"><blockquote class="gmail_quote gmail_msg" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr" class="gmail_msg">Dear Jose and Matthew,<div class="gmail_msg"><br class="gmail_msg"></div><div class="gmail_msg">Thank you so much for the effort!<br class="gmail_msg"><br class="gmail_msg">I still don't manage to converge using the range interval technique to filter out the positive eigenvalues, but using shift-invert combined with a target eigenvalue does true miracles. I get extremely fast convergence.</div><div class="gmail_msg"><br class="gmail_msg"></div><div class="gmail_msg">The truth of the matter is that we are mainly interested in negative eigenvalues (unstable modes), and from physical considerations they are more or less situated in -0.2<lambda<0 in the normalized quantities that we use. So we will just use guesses.<br class="gmail_msg"></div><div class="gmail_msg"><br class="gmail_msg"></div><div class="gmail_msg">Thank you so much again!<br class="gmail_msg"><br class="gmail_msg">Also, I have finally managed to run streams (the cluster is quite full atm). These are the outputs:<br class="gmail_msg"></div></div></blockquote><div class="gmail_msg"><br class="gmail_msg"></div></div></div></div><div dir="ltr" class="gmail_msg"><div class="gmail_extra gmail_msg"><div class="gmail_quote gmail_msg"><div class="gmail_msg">1) This shows you have a bad process mapping. You could get much more speedup for 1-4 procs by properly mapping processes to cores, perhaps with numactl.</div><div class="gmail_msg"><br class="gmail_msg"></div><div class="gmail_msg">2) Essentially 3 processes can saturate your memory bandwidth, so I would not expect much gain from using more than 4.</div><div class="gmail_msg"><br class="gmail_msg"></div><div class="gmail_msg"> Thanks,</div><div class="gmail_msg"><br class="gmail_msg"></div><div class="gmail_msg"> Matt</div></div></div></div><div dir="ltr" class="gmail_msg"><div class="gmail_extra gmail_msg"><div class="gmail_quote gmail_msg"><div class="gmail_msg"> </div><blockquote class="gmail_quote gmail_msg" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr" class="gmail_msg"><div class="gmail_msg"><div class="gmail_msg">1 processes</div><div class="gmail_msg">Number of MPI processes 1 Processor names c04b27 </div><div class="gmail_msg">Triad: 12352.0825 Rate (MB/s) </div><div class="gmail_msg">2 processes</div><div class="gmail_msg">Number of MPI processes 2 Processor names c04b27 c04b27 </div><div class="gmail_msg">Triad: 18968.0226 Rate (MB/s) </div><div class="gmail_msg">3 processes</div><div class="gmail_msg">Number of MPI processes 3 Processor names c04b27 c04b27 c04b27 </div><div class="gmail_msg">Triad: 21106.8580 Rate (MB/s) </div><div class="gmail_msg">4 processes</div><div class="gmail_msg">Number of MPI processes 4 Processor names c04b27 c04b27 c04b27 c04b27 </div><div class="gmail_msg">Triad: 21655.5885 Rate (MB/s) </div><div class="gmail_msg">5 processes</div><div class="gmail_msg">Number of MPI processes 5 Processor names c04b27 c04b27 c04b27 c04b27 c04b27 </div><div class="gmail_msg">Triad: 21627.5559 Rate (MB/s) </div><div class="gmail_msg">6 processes</div><div class="gmail_msg">Number of MPI processes 6 Processor names c04b27 c04b27 c04b27 c04b27 c04b27 c04b27 </div><div class="gmail_msg">Triad: 21394.9620 Rate (MB/s) </div><div class="gmail_msg">7 processes</div><div class="gmail_msg">Number of MPI processes 7 Processor names c04b27 c04b27 c04b27 c04b27 c04b27 c04b27 c04b27 </div><div class="gmail_msg">Triad: 24952.7076 Rate (MB/s) </div><div class="gmail_msg">8 processes</div><div class="gmail_msg">Number of MPI processes 8 Processor names c04b27 c04b27 c04b27 c04b27 c04b27 c04b27 c04b27 c04b27 </div><div class="gmail_msg">Triad: 28357.1062 Rate (MB/s) </div><div class="gmail_msg">9 processes</div><div class="gmail_msg">Number of MPI processes 9 Processor names c04b27 c04b27 c04b27 c04b27 c04b27 c04b27 c04b27 c04b27 c04b27 </div><div class="gmail_msg">Triad: 31720.4545 Rate (MB/s) </div><div class="gmail_msg">10 processes</div><div class="gmail_msg">Number of MPI processes 10 Processor names c04b27 c04b27 c04b27 c04b27 c04b27 c04b27 c04b27 c04b27 c04b27 c04b27 </div><div class="gmail_msg">Triad: 35198.7412 Rate (MB/s) </div><div class="gmail_msg">11 processes</div><div class="gmail_msg">Number of MPI processes 11 Processor names c04b27 c04b27 c04b27 c04b27 c04b27 c04b27 c04b27 c04b27 c04b27 c04b27 c04b27 </div><div class="gmail_msg">Triad: 38616.0615 Rate (MB/s) </div><div class="gmail_msg">12 processes</div><div class="gmail_msg">Number of MPI processes 12 Processor names c04b27 c04b27 c04b27 c04b27 c04b27 c04b27 c04b27 c04b27 c04b27 c04b27 c04b27 c04b27 </div><div class="gmail_msg">Triad: 41939.3994 Rate (MB/s) </div></div><div class="gmail_msg"><br class="gmail_msg"></div><div class="gmail_msg">I attach a figure.<br class="gmail_msg"><br class="gmail_msg">Thanks again!</div><div class="gmail_msg"><div class="m_8153743028333072116h5 gmail_msg"><br class="gmail_msg"><div class="gmail_quote gmail_msg"><div dir="ltr" class="gmail_msg">On Mon, Apr 3, 2017 at 8:29 PM Jose E. Roman <<a href="mailto:jroman@dsic.upv.es" class="gmail_msg" target="_blank">jroman@dsic.upv.es</a>> wrote:<br class="gmail_msg"></div><blockquote class="gmail_quote gmail_msg" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><br class="m_8153743028333072116m_-159342154222569541gmail_msg gmail_msg">
> El 1 abr 2017, a las 0:01, Toon Weyens <<a href="mailto:toon.weyens@gmail.com" class="m_8153743028333072116m_-159342154222569541gmail_msg gmail_msg" target="_blank">toon.weyens@gmail.com</a>> escribió:<br class="m_8153743028333072116m_-159342154222569541gmail_msg gmail_msg">
><br class="m_8153743028333072116m_-159342154222569541gmail_msg gmail_msg">
> Dear jose,<br class="m_8153743028333072116m_-159342154222569541gmail_msg gmail_msg">
><br class="m_8153743028333072116m_-159342154222569541gmail_msg gmail_msg">
> I have saved the matrices in Matlab format and am sending them to you using pCloud. If you want another format, please tell me. Please also note that they are about 1.4GB each.<br class="m_8153743028333072116m_-159342154222569541gmail_msg gmail_msg">
><br class="m_8153743028333072116m_-159342154222569541gmail_msg gmail_msg">
> I also attach a typical output of eps_view and log_view in output.txt, for 8 processes.<br class="m_8153743028333072116m_-159342154222569541gmail_msg gmail_msg">
><br class="m_8153743028333072116m_-159342154222569541gmail_msg gmail_msg">
> Thanks so much for helping me out! I think Petsc and Slepc are amazing inventions that really have saved me many months of work!<br class="m_8153743028333072116m_-159342154222569541gmail_msg gmail_msg">
><br class="m_8153743028333072116m_-159342154222569541gmail_msg gmail_msg">
> Regards<br class="m_8153743028333072116m_-159342154222569541gmail_msg gmail_msg">
<br class="m_8153743028333072116m_-159342154222569541gmail_msg gmail_msg">
I played a little bit with your matrices.<br class="m_8153743028333072116m_-159342154222569541gmail_msg gmail_msg">
<br class="m_8153743028333072116m_-159342154222569541gmail_msg gmail_msg">
With Krylov-Schur I can solve the problem quite easily. Note that in generalized eigenvalue problems it is always better to use STSINVERT because you have to invert a matrix anyway. So instead of setting which=smallest_real, use shift-and-invert with a target that is close to the wanted eigenvalue. For instance, with target=-0.005 I get convergence with just one iteration:<br class="m_8153743028333072116m_-159342154222569541gmail_msg gmail_msg">
<br class="m_8153743028333072116m_-159342154222569541gmail_msg gmail_msg">
$ ./ex7 -f1 A.bin -f2 B.bin -st_ksp_type preonly -st_pc_type lu -st_pc_factor_mat_solver_package mumps -eps_tol 1e-5 -st_type sinvert -eps_target -0.005<br class="m_8153743028333072116m_-159342154222569541gmail_msg gmail_msg">
<br class="m_8153743028333072116m_-159342154222569541gmail_msg gmail_msg">
Generalized eigenproblem stored in file.<br class="m_8153743028333072116m_-159342154222569541gmail_msg gmail_msg">
<br class="m_8153743028333072116m_-159342154222569541gmail_msg gmail_msg">
Reading COMPLEX matrices from binary files...<br class="m_8153743028333072116m_-159342154222569541gmail_msg gmail_msg">
Number of iterations of the method: 1<br class="m_8153743028333072116m_-159342154222569541gmail_msg gmail_msg">
Number of linear iterations of the method: 16<br class="m_8153743028333072116m_-159342154222569541gmail_msg gmail_msg">
Solution method: krylovschur<br class="m_8153743028333072116m_-159342154222569541gmail_msg gmail_msg">
<br class="m_8153743028333072116m_-159342154222569541gmail_msg gmail_msg">
Number of requested eigenvalues: 1<br class="m_8153743028333072116m_-159342154222569541gmail_msg gmail_msg">
Stopping condition: tol=1e-05, maxit=7500<br class="m_8153743028333072116m_-159342154222569541gmail_msg gmail_msg">
Linear eigensolve converged (1 eigenpair) due to CONVERGED_TOL; iterations 1<br class="m_8153743028333072116m_-159342154222569541gmail_msg gmail_msg">
---------------------- --------------------<br class="m_8153743028333072116m_-159342154222569541gmail_msg gmail_msg">
k ||Ax-kBx||/||kx||<br class="m_8153743028333072116m_-159342154222569541gmail_msg gmail_msg">
---------------------- --------------------<br class="m_8153743028333072116m_-159342154222569541gmail_msg gmail_msg">
-0.004809-0.000000i 8.82085e-05<br class="m_8153743028333072116m_-159342154222569541gmail_msg gmail_msg">
---------------------- --------------------<br class="m_8153743028333072116m_-159342154222569541gmail_msg gmail_msg">
<br class="m_8153743028333072116m_-159342154222569541gmail_msg gmail_msg">
<br class="m_8153743028333072116m_-159342154222569541gmail_msg gmail_msg">
Of course, you don't know a priori where your eigenvalue is. Alternatively, you can set the target at 0 and get rid of positive eigenvalues with a region filtering. For instance:<br class="m_8153743028333072116m_-159342154222569541gmail_msg gmail_msg">
<br class="m_8153743028333072116m_-159342154222569541gmail_msg gmail_msg">
$ ./ex7 -f1 A.bin -f2 B.bin -st_ksp_type preonly -st_pc_type lu -st_pc_factor_mat_solver_package mumps -eps_tol 1e-5 -st_type sinvert -eps_target 0 -rg_type interval -rg_interval_endpoints -1,0,-.05,.05 -eps_nev 2<br class="m_8153743028333072116m_-159342154222569541gmail_msg gmail_msg">
<br class="m_8153743028333072116m_-159342154222569541gmail_msg gmail_msg">
Generalized eigenproblem stored in file.<br class="m_8153743028333072116m_-159342154222569541gmail_msg gmail_msg">
<br class="m_8153743028333072116m_-159342154222569541gmail_msg gmail_msg">
Reading COMPLEX matrices from binary files...<br class="m_8153743028333072116m_-159342154222569541gmail_msg gmail_msg">
Number of iterations of the method: 8<br class="m_8153743028333072116m_-159342154222569541gmail_msg gmail_msg">
Number of linear iterations of the method: 74<br class="m_8153743028333072116m_-159342154222569541gmail_msg gmail_msg">
Solution method: krylovschur<br class="m_8153743028333072116m_-159342154222569541gmail_msg gmail_msg">
<br class="m_8153743028333072116m_-159342154222569541gmail_msg gmail_msg">
Number of requested eigenvalues: 2<br class="m_8153743028333072116m_-159342154222569541gmail_msg gmail_msg">
Stopping condition: tol=1e-05, maxit=7058<br class="m_8153743028333072116m_-159342154222569541gmail_msg gmail_msg">
Linear eigensolve converged (2 eigenpairs) due to CONVERGED_TOL; iterations 8<br class="m_8153743028333072116m_-159342154222569541gmail_msg gmail_msg">
---------------------- --------------------<br class="m_8153743028333072116m_-159342154222569541gmail_msg gmail_msg">
k ||Ax-kBx||/||kx||<br class="m_8153743028333072116m_-159342154222569541gmail_msg gmail_msg">
---------------------- --------------------<br class="m_8153743028333072116m_-159342154222569541gmail_msg gmail_msg">
-0.000392-0.000000i 2636.4<br class="m_8153743028333072116m_-159342154222569541gmail_msg gmail_msg">
-0.004809+0.000000i 318441<br class="m_8153743028333072116m_-159342154222569541gmail_msg gmail_msg">
---------------------- --------------------<br class="m_8153743028333072116m_-159342154222569541gmail_msg gmail_msg">
<br class="m_8153743028333072116m_-159342154222569541gmail_msg gmail_msg">
In this case, the residuals seem very bad. But this is due to the fact that your matrices have huge norms. Adding the option -eps_error_backward ::ascii_info_detail will show residuals relative to the matrix norms:<br class="m_8153743028333072116m_-159342154222569541gmail_msg gmail_msg">
---------------------- --------------------<br class="m_8153743028333072116m_-159342154222569541gmail_msg gmail_msg">
k eta(x,k)<br class="m_8153743028333072116m_-159342154222569541gmail_msg gmail_msg">
---------------------- --------------------<br class="m_8153743028333072116m_-159342154222569541gmail_msg gmail_msg">
-0.000392-0.000000i 3.78647e-11<br class="m_8153743028333072116m_-159342154222569541gmail_msg gmail_msg">
-0.004809+0.000000i 5.61419e-08<br class="m_8153743028333072116m_-159342154222569541gmail_msg gmail_msg">
---------------------- --------------------<br class="m_8153743028333072116m_-159342154222569541gmail_msg gmail_msg">
<br class="m_8153743028333072116m_-159342154222569541gmail_msg gmail_msg">
<br class="m_8153743028333072116m_-159342154222569541gmail_msg gmail_msg">
Regarding the GD solver, I am also getting the correct solution. I don't know why you are not getting convergence to the wanted eigenvalue:<br class="m_8153743028333072116m_-159342154222569541gmail_msg gmail_msg">
<br class="m_8153743028333072116m_-159342154222569541gmail_msg gmail_msg">
$ ./ex7 -f1 A.bin -f2 B.bin -st_ksp_type preonly -st_pc_type lu -st_pc_factor_mat_solver_package mumps -eps_tol 1e-5 -eps_smallest_real -eps_ncv 32 -eps_type gd<br class="m_8153743028333072116m_-159342154222569541gmail_msg gmail_msg">
<br class="m_8153743028333072116m_-159342154222569541gmail_msg gmail_msg">
Generalized eigenproblem stored in file.<br class="m_8153743028333072116m_-159342154222569541gmail_msg gmail_msg">
<br class="m_8153743028333072116m_-159342154222569541gmail_msg gmail_msg">
Reading COMPLEX matrices from binary files...<br class="m_8153743028333072116m_-159342154222569541gmail_msg gmail_msg">
Number of iterations of the method: 132<br class="m_8153743028333072116m_-159342154222569541gmail_msg gmail_msg">
Number of linear iterations of the method: 0<br class="m_8153743028333072116m_-159342154222569541gmail_msg gmail_msg">
Solution method: gd<br class="m_8153743028333072116m_-159342154222569541gmail_msg gmail_msg">
<br class="m_8153743028333072116m_-159342154222569541gmail_msg gmail_msg">
Number of requested eigenvalues: 1<br class="m_8153743028333072116m_-159342154222569541gmail_msg gmail_msg">
Stopping condition: tol=1e-05, maxit=120000<br class="m_8153743028333072116m_-159342154222569541gmail_msg gmail_msg">
Linear eigensolve converged (1 eigenpair) due to CONVERGED_TOL; iterations 132<br class="m_8153743028333072116m_-159342154222569541gmail_msg gmail_msg">
---------------------- --------------------<br class="m_8153743028333072116m_-159342154222569541gmail_msg gmail_msg">
k ||Ax-kBx||/||kx||<br class="m_8153743028333072116m_-159342154222569541gmail_msg gmail_msg">
---------------------- --------------------<br class="m_8153743028333072116m_-159342154222569541gmail_msg gmail_msg">
-0.004809+0.000000i 2.16223e-05<br class="m_8153743028333072116m_-159342154222569541gmail_msg gmail_msg">
---------------------- --------------------<br class="m_8153743028333072116m_-159342154222569541gmail_msg gmail_msg">
<br class="m_8153743028333072116m_-159342154222569541gmail_msg gmail_msg">
<br class="m_8153743028333072116m_-159342154222569541gmail_msg gmail_msg">
Again, it is much better to use a target instead of smallest_real:<br class="m_8153743028333072116m_-159342154222569541gmail_msg gmail_msg">
<br class="m_8153743028333072116m_-159342154222569541gmail_msg gmail_msg">
$ ./ex7 -f1 A.bin -f2 B.bin -st_ksp_type preonly -st_pc_type lu -st_pc_factor_mat_solver_package mumps -eps_tol 1e-5 -eps_type gd -eps_target -0.005<br class="m_8153743028333072116m_-159342154222569541gmail_msg gmail_msg">
<br class="m_8153743028333072116m_-159342154222569541gmail_msg gmail_msg">
Generalized eigenproblem stored in file.<br class="m_8153743028333072116m_-159342154222569541gmail_msg gmail_msg">
<br class="m_8153743028333072116m_-159342154222569541gmail_msg gmail_msg">
Reading COMPLEX matrices from binary files...<br class="m_8153743028333072116m_-159342154222569541gmail_msg gmail_msg">
Number of iterations of the method: 23<br class="m_8153743028333072116m_-159342154222569541gmail_msg gmail_msg">
Number of linear iterations of the method: 0<br class="m_8153743028333072116m_-159342154222569541gmail_msg gmail_msg">
Solution method: gd<br class="m_8153743028333072116m_-159342154222569541gmail_msg gmail_msg">
<br class="m_8153743028333072116m_-159342154222569541gmail_msg gmail_msg">
Number of requested eigenvalues: 1<br class="m_8153743028333072116m_-159342154222569541gmail_msg gmail_msg">
Stopping condition: tol=1e-05, maxit=120000<br class="m_8153743028333072116m_-159342154222569541gmail_msg gmail_msg">
Linear eigensolve converged (1 eigenpair) due to CONVERGED_TOL; iterations 23<br class="m_8153743028333072116m_-159342154222569541gmail_msg gmail_msg">
---------------------- --------------------<br class="m_8153743028333072116m_-159342154222569541gmail_msg gmail_msg">
k ||Ax-kBx||/||kx||<br class="m_8153743028333072116m_-159342154222569541gmail_msg gmail_msg">
---------------------- --------------------<br class="m_8153743028333072116m_-159342154222569541gmail_msg gmail_msg">
-0.004809-0.000000i 2.06572e-05<br class="m_8153743028333072116m_-159342154222569541gmail_msg gmail_msg">
---------------------- --------------------<br class="m_8153743028333072116m_-159342154222569541gmail_msg gmail_msg">
<br class="m_8153743028333072116m_-159342154222569541gmail_msg gmail_msg">
<br class="m_8153743028333072116m_-159342154222569541gmail_msg gmail_msg">
Jose<br class="m_8153743028333072116m_-159342154222569541gmail_msg gmail_msg">
<br class="m_8153743028333072116m_-159342154222569541gmail_msg gmail_msg">
<br class="m_8153743028333072116m_-159342154222569541gmail_msg gmail_msg">
</blockquote></div></div></div></div>
</blockquote></div></div></div><div dir="ltr" class="gmail_msg"><div class="gmail_extra gmail_msg"><div class="gmail_msg"><br class="gmail_msg"></div>-- <br class="gmail_msg"><div class="m_8153743028333072116gmail_signature gmail_msg" data-smartmail="gmail_signature">What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br class="gmail_msg">-- Norbert Wiener</div>
</div></div></blockquote></div>