<div dir="ltr"><div dir="ltr"><div>This is the error message i get from my program: </div><div>Shape of the dynamical matrix: (1500000, 1500000)</div><div><br></div><div>===================================================================================</div><div>= BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES</div><div>= PID 122676 RUNNING AT n3512.nemo.privat</div><div>= EXIT CODE: 9</div><div>= CLEANING UP REMAINING PROCESSES</div><div>= YOU CAN IGNORE THE BELOW CLEANUP MESSAGES</div><div>===================================================================================</div><div> Intel(R) MPI Library troubleshooting guide:</div><div> <a href="https://software.intel.com/node/561764">https://software.intel.com/node/561764</a></div><div>===================================================================================</div><div><br></div></div></div><br><div class="gmail_quote"><div dir="ltr">Am Mi., 24. Okt. 2018 um 16:01 Uhr schrieb Matthew Knepley <<a href="mailto:knepley@gmail.com">knepley@gmail.com</a>>:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div class="gmail_quote"><div dir="ltr">On Wed, Oct 24, 2018 at 9:38 AM Jan Grießer <<a href="mailto:griesser.jan@googlemail.com" target="_blank">griesser.jan@googlemail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div dir="ltr">Hey,<div>i tried to run my program as you said with -bv_type vecs and/or -bv_type mat, but instead of the PETScInt overflow i now get an MPI Error 9 </div></div></div></blockquote><div><br></div><div>Send the actual error.</div><div><br></div><div> Thanks,</div><div><br></div><div> Matt</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div dir="ltr"><div>message, which i assume (after googling a little bit around) should be a memory problem. I tried to run it also on slightly bigger compute nodes on our cluster with 20 cores with each 12 GB and 24 GB but the problem still remains. The actual limit appears to be a 1.5 Million x 1.5 Million where i searched for NEV = 1500 and MPD = 500/ 200 eigenvalues. <br>Do you have maybe an idea what the error could be? I appended also the python method i used to solve the problem. I also tried to solve the problem with spectrum solving but the error message remains the same. </div><div><br></div><div><div>def solve_eigensystem(DynMatrix_nn, NEV, MPD, Dimension):</div><div><span style="white-space:pre-wrap"> </span># Create the solver </div><div><span style="white-space:pre-wrap"> </span># E is used as an acronym for the EPS solver (EPS = Eigenvalue problem solver)</div><div><span style="white-space:pre-wrap"> </span>E = SLEPc.EPS().create()</div><div><br></div><div><span style="white-space:pre-wrap"> </span># Set the preconditioner </div><div><span style="white-space:pre-wrap"> </span>pc = PETSc.PC().create()</div><div><span style="white-space:pre-wrap"> </span>pc.setType(pc.Type.BJACOBI)</div><div><br></div><div><span style="white-space:pre-wrap"> </span># Set the linear solver </div><div><span style="white-space:pre-wrap"> </span># Create the KSP object</div><div><span style="white-space:pre-wrap"> </span>ksp = PETSc.KSP().create()</div><div><span style="white-space:pre-wrap"> </span># Create the solver, in this case GMRES</div><div><span style="white-space:pre-wrap"> </span>ksp.setType(ksp.Type.GMRES)</div><div><span style="white-space:pre-wrap"> </span># Set the tolerances of the GMRES solver</div><div> # Link it to the PC</div><div><span style="white-space:pre-wrap"> </span>ksp.setPC(pc)</div><div><br></div><div><span style="white-space:pre-wrap"> </span># Set up the spectral transformations </div><div><span style="white-space:pre-wrap"> </span>st = SLEPc.ST().create()</div><div><span style="white-space:pre-wrap"> </span>st.setType("shift")</div><div><span style="white-space:pre-wrap"> </span>st.setKSP(ksp)</div><div><span style="white-space:pre-wrap"> </span></div><div><span style="white-space:pre-wrap"> </span># MPD stands for "maximum projected dimension". It has to due with computational cost, please read Chap. 2.6.5 of SLEPc docu for </div><div><span style="white-space:pre-wrap"> </span># an explanation. At the moment mpd is only a guess</div><div><span style="white-space:pre-wrap"> </span>E.setDimensions(nev=NEV, mpd = MPD)</div><div><span style="white-space:pre-wrap"> </span># Eigenvalues should be real, therefore we start to order them from the smallest real value |l.real|</div><div><span style="white-space:pre-wrap"> </span>E.setWhichEigenpairs(E.Which.SMALLEST_REAL)</div><div><span style="white-space:pre-wrap"> </span># Since the dynamical matrix is symmetric and real it is hermitian </div><div><span style="white-space:pre-wrap"> </span>E.setProblemType(SLEPc.EPS.ProblemType.HEP)</div><div><span style="white-space:pre-wrap"> </span># Use the Krylov Schur method to solve the eigenvalue problem</div><div><span style="white-space:pre-wrap"> </span>E.setType(E.Type.KRYLOVSCHUR)</div><div><span style="white-space:pre-wrap"> </span># Set the convergence criterion to relative to the eigenvalue and the maximal number of iterations</div><div><span style="white-space:pre-wrap"> </span>E.setConvergenceTest(E.Conv.REL)</div><div><span style="white-space:pre-wrap"> </span>E.setTolerances(tol = 1e-8, max_it = 5000)</div><div><span style="white-space:pre-wrap"> </span># Set the matrix in order to solve </div><div><span style="white-space:pre-wrap"> </span>E.setOperators(DynMatrix_nn, None)</div><div><span style="white-space:pre-wrap"> </span># Sets EPS options from the options database. This routine must be called before `setUp()` if the user is to be allowed to set dthe solver type.</div><div><span style="white-space:pre-wrap"> </span>E.setFromOptions()</div><div><span style="white-space:pre-wrap"> </span># Sets up all the internal data structures necessary for the execution of the eigensolver.</div><div><span style="white-space:pre-wrap"> </span>E.setUp()</div><div><br></div><div><span style="white-space:pre-wrap"> </span># Solve eigenvalue problem </div><div><span style="white-space:pre-wrap"> </span>E.solve()</div><div><br></div><div><span style="white-space:pre-wrap"> </span>Print = PETSc.Sys.Print</div><div><br></div><div><span style="white-space:pre-wrap"> </span>Print()</div><div><span style="white-space:pre-wrap"> </span>Print("****************************")</div><div><span style="white-space:pre-wrap"> </span>Print("***SLEPc Solution Results***")</div><div><span style="white-space:pre-wrap"> </span>Print("****************************")</div><div><br></div><div><span style="white-space:pre-wrap"> </span>its = E.getIterationNumber()</div><div><span style="white-space:pre-wrap"> </span>Print("Number of iterations of the method: ", its)</div><div><span style="white-space:pre-wrap"> </span>eps_type = E.getType()</div><div><span style="white-space:pre-wrap"> </span>Print("Solution method: ", eps_type)</div><div><span style="white-space:pre-wrap"> </span>nev, ncv, mpd = E.getDimensions()</div><div><span style="white-space:pre-wrap"> </span>Print("Number of requested eigenvalues: ", nev)</div><div><span style="white-space:pre-wrap"> </span>Print("Number of computeded eigenvectors: ", ncv)</div><div><span style="white-space:pre-wrap"> </span>tol, maxit = E.getTolerances()</div><div><span style="white-space:pre-wrap"> </span>Print("Stopping condition: (tol, maxit)", (tol, maxit))</div><div><span style="white-space:pre-wrap"> </span># Get the type of convergence</div><div><span style="white-space:pre-wrap"> </span>conv_test = E.getConvergenceTest()</div><div><span style="white-space:pre-wrap"> </span>Print("Selected convergence test: ", conv_test)</div><div><span style="white-space:pre-wrap"> </span># Get the used spectral transformation</div><div><span style="white-space:pre-wrap"> </span>get_st = E.getST()</div><div><span style="white-space:pre-wrap"> </span>Print("Selected spectral transformation: ", get_st)</div><div><span style="white-space:pre-wrap"> </span># Get the applied direct solver</div><div><span style="white-space:pre-wrap"> </span>get_ksp = E.getDS()</div><div><span style="white-space:pre-wrap"> </span>Print("Selected direct solver: ", get_ksp)</div><div><span style="white-space:pre-wrap"> </span>nconv = E.getConverged()</div><div><span style="white-space:pre-wrap"> </span>Print("Number of converged eigenpairs: ", nconv)</div></div><div>.....</div><div><br></div><div><br></div></div></div><br><div class="gmail_quote"><div dir="ltr">Am Fr., 19. Okt. 2018 um 21:00 Uhr schrieb Smith, Barry F. <<a href="mailto:bsmith@mcs.anl.gov" target="_blank">bsmith@mcs.anl.gov</a>>:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><br>
<br>
> On Oct 19, 2018, at 7:56 AM, Zhang, Junchao <<a href="mailto:jczhang@mcs.anl.gov" target="_blank">jczhang@mcs.anl.gov</a>> wrote:<br>
> <br>
> <br>
> On Fri, Oct 19, 2018 at 4:02 AM Jan Grießer <<a href="mailto:griesser.jan@googlemail.com" target="_blank">griesser.jan@googlemail.com</a>> wrote:<br>
> With more than 1 MPI process you mean i should use spectrum slicing in divide the full problem in smaller subproblems? <br>
> The --with-64-bit-indices is not a possibility for me since i configured petsc with mumps, which does not allow to use the 64-bit version (At least this was the error message when i tried to configure PETSc )<br>
> <br>
> MUMPS 5.1.2 manual chapter 2.4.2 says it supports "Selective 64-bit integer feature" and "full 64-bit integer version" as well. <br>
<br>
They use to achieve this by compiling with special Fortran flags to promote integers to 64 bit; this is too fragile for our taste so we never hooked PETSc up wit it. If they have a dependable way of using 64 bit integers we should add that to our mumps.py and test it.<br>
<br>
Barry<br>
<br>
> <br>
> Am Mi., 17. Okt. 2018 um 18:24 Uhr schrieb Jose E. Roman <<a href="mailto:jroman@dsic.upv.es" target="_blank">jroman@dsic.upv.es</a>>:<br>
> To use BVVECS just add the command-line option -bv_type vecs<br>
> This causes to use a separate Vec for each column, instead of a single long Vec of size n*m. But it is considerably slower than the default.<br>
> <br>
> Anyway, for such large problems you should consider using more than 1 MPI process. In that case the error may disappear because the local size is smaller than 768000.<br>
> <br>
> Jose<br>
> <br>
> <br>
> > El 17 oct 2018, a las 17:58, Matthew Knepley <<a href="mailto:knepley@gmail.com" target="_blank">knepley@gmail.com</a>> escribió:<br>
> > <br>
> > On Wed, Oct 17, 2018 at 11:54 AM Jan Grießer <<a href="mailto:griesser.jan@googlemail.com" target="_blank">griesser.jan@googlemail.com</a>> wrote:<br>
> > Hi all,<br>
> > i am using slepc4py and petsc4py to solve for the smallest real eigenvalues and eigenvectors. For my test cases with a matrix A of the size 30k x 30k solving for the smallest soutions works quite well, but when i increase the dimension of my system to around A = 768000 x 768000 or 3 million x 3 million and ask for the smallest real 3000 (the number is increasing with increasing system size) eigenvalues and eigenvectors i get the output (for the 768000): <br>
> > The product 4001 times 768000 overflows the size of PetscInt; consider reducing the number of columns, or use BVVECS instead<br>
> > i understand that the requested number of eigenvectors and eigenvalues is causing an overflow but i do not understand the solution of the problem which is stated in the error message. Can someone tell me what exactly BVVECS is and how i can use it? Or is there any other solution to my problem ?<br>
> > <br>
> > You can also reconfigure with 64-bit integers: --with-64-bit-indices<br>
> > <br>
> > Thanks,<br>
> > <br>
> > Matt<br>
> > <br>
> > Thank you very much in advance,<br>
> > Jan <br>
> > <br>
> > <br>
> > <br>
> > -- <br>
> > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>
> > -- Norbert Wiener<br>
> > <br>
> > <a href="https://www.cse.buffalo.edu/~knepley/" rel="noreferrer" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br>
> <br>
<br>
</blockquote></div>
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr" class="m_-1120326649911867329gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>-- Norbert Wiener</div><div><br></div><div><a href="http://www.cse.buffalo.edu/~knepley/" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br></div></div></div></div></div></div></div></div>
</blockquote></div>