<div dir="ltr"><div class="gmail_quote"><div dir="ltr">On Tue, Nov 13, 2018 at 6:40 AM Jan Grießer <<a href="mailto:griesser.jan@googlemail.com">griesser.jan@googlemail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div dir="ltr">I am finally back from holidays and continued working on the problem. I think i figured out that the problem is not related to SLEPc but rather to PETSc. Until now i precomputed my matrix in python and saved it as a scipy.csr matrix( The matrix is of order 6 Mio x 6 Mio with around 0.004% nonzero elements). The code i used to convert the scipy sparse matrix to a PETSc matrix is appended below. But i am wondering if maybe the problem is that the memory of the individual processors(i use at the moment 60 cores with 24 GB each ) is simply dumped by the size of the matrix (It is around 13 Gb) or if the PETSc matrix is not correctly distributed. <br>I am sorry for the late answer and hope someone has any suggestion. <br></div></div></blockquote><div><br></div><div>The code below makes it look like you are loading the entire CSR matrix on every process (load_npz) and then</div><div>selecting a portion of it to stick into PETSc. The memory footprint of this strategy will explode.</div><div><br></div><div> Thanks,</div><div><br></div><div> Matt</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div dir="ltr"><div><div>def construct_mat():</div><div><span style="white-space:pre-wrap"> </span>D_nn = scipy.sparse.load_npz("../PBC_DynamicalMatrix/DynamicalMatrixPB.npz")</div><div><span style="white-space:pre-wrap"> </span># Create a matrix Ds in parallel </div><div><span style="white-space:pre-wrap"> </span>Dyn = PETSc.Mat().create()</div><div><span style="white-space:pre-wrap"> </span>Dyn.setSizes(CSRMatrix.shape)</div><div><span style="white-space:pre-wrap"> </span>Dyn.setFromOptions()</div><div><span style="white-space:pre-wrap"> </span>Dyn.setUp()</div><div><span style="white-space:pre-wrap"> </span>Rstart, Rend = Dyn.getOwnershipRange()</div><div><span style="white-space:pre-wrap"> </span></div><div><span style="white-space:pre-wrap"> </span># Fill the matrix</div><div><span style="white-space:pre-wrap"> </span>csr = (</div><div><span style="white-space:pre-wrap"> </span>CSRMatrix.indptr[Rstart:Rend+1] - CSRMatrix.indptr[Rstart],</div><div><span style="white-space:pre-wrap"> </span>CSRMatrix.indices[CSRMatrix.indptr[Rstart]:CSRMatrix.indptr[Rend]],</div><div><span style="white-space:pre-wrap"> </span>CSRMatrix.data[CSRMatrix.indptr[Rstart]:CSRMatrix.indptr[Rend]]</div><div><span style="white-space:pre-wrap"> </span>)</div><div><span style="white-space:pre-wrap"> </span>D = PETSc.Mat().createAIJ(size=CSRMatrix.shape, csr=csr)</div><div><span style="white-space:pre-wrap"> </span>D.assemble()</div><div><br></div><div><span style="white-space:pre-wrap"> </span># Free the memory </div><div><span style="white-space:pre-wrap"> </span>del CSRDynamicalMatrix_nn</div><div><br></div><div><span style="white-space:pre-wrap"> </span># Return the PETSc dynamical matrix </div><div><span style="white-space:pre-wrap"> </span>return D</div></div></div></div><br><div class="gmail_quote"><div dir="ltr">Am Mi., 24. Okt. 2018 um 18:46 Uhr schrieb Matthew Knepley <<a href="mailto:knepley@gmail.com" target="_blank">knepley@gmail.com</a>>:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">As it says, if you are looking at performance, you should configure using --with-debugging=0.<div><br></div><div>It says SLEPc is using 8GB. Is this what you see?</div><div><br></div><div> Matt</div></div><br><div class="gmail_quote"><div dir="ltr">On Wed, Oct 24, 2018 at 12:30 PM Jan Grießer <<a href="mailto:griesser.jan@googlemail.com" target="_blank">griesser.jan@googlemail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div dir="ltr">I also run it with the -log_summary : <div><div>---------------------------------------------- PETSc Performance Summary: ----------------------------------------------</div><div><br></div><div><br></div><div><br></div><div> ##########################################################</div><div> # #</div><div> # WARNING!!! #</div><div> # #</div><div> # This code was compiled with a debugging option, #</div><div> # To get timing results run ./configure #</div><div> # using --with-debugging=no, the performance will #</div><div> # be generally two or three times faster. #</div><div> # #</div><div> ##########################################################</div><div><br></div><div><br></div><div>/work/ws/nemo/fr_jg1080-FreeSurface_Glass-0/GlassSystems/PeriodicSystems/N500000T0.001/SolveEigenvalueProblem_par/Test/Eigensolver_petsc_slepc_no_argparse.py on a arch-linux2-c-debug named int02.nemo.privat with 20 processors, by fr_jg1080 Wed Oct 24 18:26:30 2018</div><div>Using Petsc Release Version 3.9.4, Sep, 11, 2018 </div><div><br></div><div> Max Max/Min Avg Total </div><div>Time (sec): 7.474e+02 1.00000 7.474e+02</div><div>Objects: 3.600e+01 1.00000 3.600e+01</div><div>Flop: 1.090e+11 1.00346 1.089e+11 2.177e+12</div><div>Flop/sec: 1.459e+08 1.00346 1.457e+08 2.913e+09</div><div>Memory: 3.950e+08 1.00296 7.891e+09</div><div>MPI Messages: 3.808e+04 1.00000 3.808e+04 7.615e+05</div><div>MPI Message Lengths: 2.211e+10 1.00217 5.802e+05 4.419e+11</div><div>MPI Reductions: 1.023e+05 1.00000</div><div><br></div><div>Flop counting convention: 1 flop = 1 real number operation of type (multiply/divide/add/subtract)</div><div> e.g., VecAXPY() for real vectors of length N --> 2N flop</div><div> and VecAXPY() for complex vectors of length N --> 8N flop</div><div><br></div><div>Summary of Stages: ----- Time ------ ----- Flop ----- --- Messages --- -- Message Lengths -- -- Reductions --</div><div> Avg %Total Avg %Total counts %Total Avg %Total counts %Total </div><div> 0: Main Stage: 7.4739e+02 100.0% 2.1773e+12 100.0% 7.615e+05 100.0% 5.802e+05 100.0% 1.022e+05 100.0% </div><div><br></div><div>------------------------------------------------------------------------------------------------------------------------</div><div>See the 'Profiling' chapter of the users' manual for details on interpreting output.</div><div>Phase summary info:</div><div> Count: number of times phase was executed</div><div> Time and Flop: Max - maximum over all processors</div><div> Ratio - ratio of maximum to minimum over all processors</div><div> Mess: number of messages sent</div><div> Avg. len: average message length (bytes)</div><div> Reduct: number of global reductions</div><div> Global: entire computation</div><div> Stage: stages of a computation. Set stages with PetscLogStagePush() and PetscLogStagePop().</div><div> %T - percent time in this phase %F - percent flop in this phase</div><div> %M - percent messages in this phase %L - percent message lengths in this phase</div><div> %R - percent reductions in this phase</div><div> Total Mflop/s: 10e-6 * (sum of flop over all processors)/(max time over all processors)</div><div>------------------------------------------------------------------------------------------------------------------------</div><div><br></div><div><br></div><div> ##########################################################</div><div> # #</div><div> # WARNING!!! #</div><div> # #</div><div> # This code was compiled with a debugging option, #</div><div> # To get timing results run ./configure #</div><div> # using --with-debugging=no, the performance will #</div><div> # be generally two or three times faster. #</div><div> # #</div><div> ##########################################################</div><div><br></div><div><br></div><div>Event Count Time (sec) Flop --- Global --- --- Stage --- Total</div><div> Max Ratio Max Ratio Max Ratio Mess Avg len Reduct %T %F %M %L %R %T %F %M %L %R Mflop/s</div><div>------------------------------------------------------------------------------------------------------------------------</div><div><br></div><div>--- Event Stage 0: Main Stage</div><div><br></div><div>BuildTwoSidedF 2 1.0 2.6670e-01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0</div><div>VecSet 2 1.0 6.8650e-03 1.8 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0</div><div>VecScatterBegin 2002 1.0 1.4380e+01 1.0 0.00e+00 0.0 7.6e+05 5.8e+05 0.0e+00 2 0100100 0 2 0100100 0 0</div><div>VecScatterEnd 2002 1.0 3.7604e+01 1.5 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 4 0 0 0 0 4 0 0 0 0 0</div><div>VecSetRandom 1 1.0 1.6440e-01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0</div><div>MatMult 2002 1.0 6.0846e+02 1.2 1.03e+11 1.0 7.6e+05 5.8e+05 0.0e+00 71 94100100 0 71 94100100 0 3376</div><div>MatAssemblyBegin 3 1.0 2.8129e-01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 6.0e+00 0 0 0 0 0 0 0 0 0 0 0</div><div>MatAssemblyEnd 3 1.0 8.5094e+00 1.0 0.00e+00 0.0 7.6e+02 1.5e+05 3.6e+01 1 0 0 0 0 1 0 0 0 0 0</div><div>EPSSetUp 1 1.0 1.7351e-02 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 7.6e+01 0 0 0 0 0 0 0 0 0 0 0</div><div>EPSSolve 1 1.0 6.7891e+02 1.0 1.09e+11 1.0 7.6e+05 5.8e+05 1.0e+05 91100100100100 91100100100100 3207</div><div>STSetUp 1 1.0 2.2221e-04 1.3 0.00e+00 0.0 0.0e+00 0.0e+00 6.0e+00 0 0 0 0 0 0 0 0 0 0 0</div><div>STApply 2002 1.0 6.0879e+02 1.2 1.03e+11 1.0 7.6e+05 5.8e+05 0.0e+00 71 94100100 0 71 94100100 0 3374</div><div>BVCopy 999 1.0 2.7157e-01 1.2 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0</div><div>BVMultVec 4004 1.0 2.2918e+00 1.0 2.10e+09 1.0 0.0e+00 0.0e+00 1.6e+04 0 2 0 0 16 0 2 0 0 16 18332</div><div>BVMultInPlace 999 1.0 4.8399e+01 1.0 1.20e+09 1.0 0.0e+00 0.0e+00 0.0e+00 6 1 0 0 0 6 1 0 0 0 495</div><div>BVDotVec 4004 1.0 1.0835e+01 1.0 2.70e+09 1.0 0.0e+00 0.0e+00 2.0e+04 1 2 0 0 20 1 2 0 0 20 4986</div><div>BVOrthogonalizeV 2003 1.0 1.3272e+01 1.0 4.80e+09 1.0 0.0e+00 0.0e+00 5.2e+04 2 4 0 0 51 2 4 0 0 51 7236</div><div>BVScale 2003 1.0 2.3521e-01 1.0 1.50e+08 1.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 12773</div><div>BVSetRandom 1 1.0 1.6456e-01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 4.0e+00 0 0 0 0 0 0 0 0 0 0 0</div><div>DSSolve 1000 1.0 3.3338e+00 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0</div><div>DSVectors 1000 1.0 6.0029e-03 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0</div><div>DSOther 2999 1.0 7.8770e-01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0</div><div>------------------------------------------------------------------------------------------------------------------------</div><div><br></div><div>Memory usage is given in bytes:</div><div><br></div><div>Object Type Creations Destructions Memory Descendants' Mem.</div><div>Reports information only for process 0.</div><div><br></div><div>--- Event Stage 0: Main Stage</div><div><br></div><div> Viewer 2 1 840 0.</div><div> PetscRandom 1 1 646 0.</div><div> Index Set 4 4 5510472 0.</div><div> Vector 9 9 11629608 0.</div><div> Vec Scatter 2 2 1936 0.</div><div> Matrix 10 10 331855732 0.</div><div> Preconditioner 1 1 1000 0.</div><div> Krylov Solver 1 1 1176 0.</div><div> EPS Solver 1 1 1600 0.</div><div> Spectral Transform 2 2 1624 0.</div><div> Basis Vectors 1 1 2168 0.</div><div> Direct Solver 1 1 2520 0.</div><div> Region 1 1 672 0.</div><div>========================================================================================================================</div><div>Average time to get PetscTime(): 1.19209e-07</div><div>Average time for MPI_Barrier(): 2.67982e-05</div><div>Average time for zero size MPI_Send(): 1.08957e-05</div><div>#PETSc Option Table entries:</div><div>-bv_type mat</div><div>-eps_view_pre</div><div>-log_summary</div><div>#End of PETSc Option Table entries</div><div>Compiled without FORTRAN kernels</div><div>Compiled with full precision matrices (default)</div><div>sizeof(short) 2 sizeof(int) 4 sizeof(long) 8 sizeof(void*) 8 sizeof(PetscScalar) 8 sizeof(PetscInt) 4</div><div>Configure options: --with-cc=mpicc --with-cxx=mpicxx --with-fc=mpif90 --download-mumps --with-shared-libraries=True --download-scalapack</div><div>-----------------------------------------</div><div>Libraries compiled on 2018-10-17 20:02:31 on login2.nemo.privat </div><div>Machine characteristics: Linux-3.10.0-693.21.1.el7.x86_64-x86_64-with-centos-7.4.1708-Core</div><div>Using PETSc directory: /home/fr/fr_fr/fr_jg1080/Libaries/petsc-3.9.4</div><div>Using PETSc arch: arch-linux2-c-debug</div><div>-----------------------------------------</div><div><br></div><div>Using C compiler: mpicc -fPIC -wd1572 -g </div><div>Using Fortran compiler: mpif90 -fPIC -g </div><div>-----------------------------------------</div><div><br></div><div>Using include paths: -I/home/fr/fr_fr/fr_jg1080/Libaries/petsc-3.9.4/include -I/home/fr/fr_fr/fr_jg1080/Libaries/petsc-3.9.4/arch-linux2-c-debug/include</div><div>-----------------------------------------</div><div><br></div><div>Using C linker: mpicc</div><div>Using Fortran linker: mpif90</div><div>Using libraries: -Wl,-rpath,/home/fr/fr_fr/fr_jg1080/Libaries/petsc-3.9.4/arch-linux2-c-debug/lib -L/home/fr/fr_fr/fr_jg1080/Libaries/petsc-3.9.4/arch-linux2-c-debug/lib -lpetsc -Wl,-rpath,/home/fr/fr_fr/fr_jg1080/Libaries/petsc-3.9.4/arch-linux2-c-debug/lib -L/home/fr/fr_fr/fr_jg1080/Libaries/petsc-3.9.4/arch-linux2-c-debug/lib -Wl,-rpath,/opt/bwhpc/common/compiler/intel/2018.3.222/compilers_and_libraries/linux/mpi/intel64/lib/debug_mt -L/opt/bwhpc/common/compiler/intel/2018.3.222/compilers_and_libraries/linux/mpi/intel64/lib/debug_mt -Wl,-rpath,/opt/bwhpc/common/compiler/intel/2018.3.222/compilers_and_libraries/linux/mpi/intel64/lib -L/opt/bwhpc/common/compiler/intel/2018.3.222/compilers_and_libraries/linux/mpi/intel64/lib -Wl,-rpath,/opt/bwhpc/common/compiler/intel/2018.3.222/compilers_and_libraries/linux/lib/intel64 -L/opt/bwhpc/common/compiler/intel/2018.3.222/compilers_and_libraries/linux/lib/intel64 -Wl,-rpath,/opt/bwhpc/common/compiler/intel/2018.3.222/compilers_and_libraries_2018.3.222/linux/compiler/lib/intel64_lin -L/opt/bwhpc/common/compiler/intel/2018.3.222/compilers_and_libraries_2018.3.222/linux/compiler/lib/intel64_lin -Wl,-rpath,/usr/lib/gcc/x86_64-redhat-linux/4.8.5 -L/usr/lib/gcc/x86_64-redhat-linux/4.8.5 -Wl,-rpath,/opt/intel/mpi-rt/2017.0.0/intel64/lib/debug_mt -Wl,-rpath,/opt/intel/mpi-rt/2017.0.0/intel64/lib -lcmumps -ldmumps -lsmumps -lzmumps -lmumps_common -lpord -lscalapack -llapack -lblas -lX11 -lstdc++ -ldl -lmpifort -lmpi -lmpigi -lrt -lpthread -lifport -lifcoremt_pic -limf -lsvml -lm -lipgo -lirc -lgcc_s -lirc_s -lstdc++ -ldl</div><div>-----------------------------------------</div><div><br></div><div><br></div><div><br></div><div> ##########################################################</div><div> # #</div><div> # WARNING!!! #</div><div> # #</div><div> # This code was compiled with a debugging option, #</div><div> # To get timing results run ./configure #</div><div> # using --with-debugging=no, the performance will #</div><div> # be generally two or three times faster. #</div><div> # #</div><div> ##########################################################</div><div><br></div></div></div></div><br><div class="gmail_quote"><div dir="ltr">Am Mi., 24. Okt. 2018 um 18:07 Uhr schrieb Jan Grießer <<a href="mailto:griesser.jan@googlemail.com" target="_blank">griesser.jan@googlemail.com</a>>:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div dir="ltr">For some reason i get only this error message, also when is use the -eps_view_pre. I started the program with nev=1, there the output is (with -bv_type vecs -bv_type mat -eps_view_pre)<div><div>EPS Object: 20 MPI processes</div><div> type: krylovschur</div><div> 50% of basis vectors kept after restart</div><div> using the locking variant</div><div> problem type: symmetric eigenvalue problem</div><div> selected portion of the spectrum: smallest real parts</div><div> number of eigenvalues (nev): 1</div><div> number of column vectors (ncv): 3</div><div> maximum dimension of projected problem (mpd): 2</div><div> maximum number of iterations: 1000</div><div> tolerance: 1e-08</div><div> convergence test: relative to the eigenvalue</div><div>BV Object: 20 MPI processes</div><div> type: mat</div><div> 4 columns of global length 1500000</div><div> vector orthogonalization method: classical Gram-Schmidt</div><div> orthogonalization refinement: if needed (eta: 0.7071)</div><div> block orthogonalization method: GS</div><div> doing matmult as a single matrix-matrix product</div><div>DS Object: 20 MPI processes</div><div> type: hep</div><div> parallel operation mode: REDUNDANT</div><div> solving the problem with: Implicit QR method (_steqr)</div><div>ST Object: 20 MPI processes</div><div> type: shift</div><div> shift: 0.</div><div> number of matrices: 1</div></div><div><br></div></div></div><div dir="ltr"><div dir="ltr"><div><br></div><div><br></div></div></div><br><div class="gmail_quote"><div dir="ltr">Am Mi., 24. Okt. 2018 um 16:14 Uhr schrieb Matthew Knepley <<a href="mailto:knepley@gmail.com" target="_blank">knepley@gmail.com</a>>:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div class="gmail_quote"><div dir="ltr">On Wed, Oct 24, 2018 at 10:03 AM Jan Grießer <<a href="mailto:griesser.jan@googlemail.com" target="_blank">griesser.jan@googlemail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div dir="ltr"><div>This is the error message i get from my program: </div><div>Shape of the dynamical matrix: (1500000, 1500000)</div></div></div></blockquote><div><br></div><div>Petsc installs a signal handler, so there should be a PETSc-specific message before this one. Is something eating</div><div>your output?</div><div><br></div><div> Thanks,</div><div><br></div><div> Matt</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div dir="ltr"><div>===================================================================================</div><div>= BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES</div><div>= PID 122676 RUNNING AT n3512.nemo.privat</div><div>= EXIT CODE: 9</div><div>= CLEANING UP REMAINING PROCESSES</div><div>= YOU CAN IGNORE THE BELOW CLEANUP MESSAGES</div><div>===================================================================================</div><div> Intel(R) MPI Library troubleshooting guide:</div><div> <a href="https://software.intel.com/node/561764" target="_blank">https://software.intel.com/node/561764</a></div><div>===================================================================================</div><div><br></div></div></div><br><div class="gmail_quote"><div dir="ltr">Am Mi., 24. Okt. 2018 um 16:01 Uhr schrieb Matthew Knepley <<a href="mailto:knepley@gmail.com" target="_blank">knepley@gmail.com</a>>:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div class="gmail_quote"><div dir="ltr">On Wed, Oct 24, 2018 at 9:38 AM Jan Grießer <<a href="mailto:griesser.jan@googlemail.com" target="_blank">griesser.jan@googlemail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div dir="ltr">Hey,<div>i tried to run my program as you said with -bv_type vecs and/or -bv_type mat, but instead of the PETScInt overflow i now get an MPI Error 9 </div></div></div></blockquote><div><br></div><div>Send the actual error.</div><div><br></div><div> Thanks,</div><div><br></div><div> Matt</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div dir="ltr"><div>message, which i assume (after googling a little bit around) should be a memory problem. I tried to run it also on slightly bigger compute nodes on our cluster with 20 cores with each 12 GB and 24 GB but the problem still remains. The actual limit appears to be a 1.5 Million x 1.5 Million where i searched for NEV = 1500 and MPD = 500/ 200 eigenvalues. <br>Do you have maybe an idea what the error could be? I appended also the python method i used to solve the problem. I also tried to solve the problem with spectrum solving but the error message remains the same. </div><div><br></div><div><div>def solve_eigensystem(DynMatrix_nn, NEV, MPD, Dimension):</div><div><span style="white-space:pre-wrap"> </span># Create the solver </div><div><span style="white-space:pre-wrap"> </span># E is used as an acronym for the EPS solver (EPS = Eigenvalue problem solver)</div><div><span style="white-space:pre-wrap"> </span>E = SLEPc.EPS().create()</div><div><br></div><div><span style="white-space:pre-wrap"> </span># Set the preconditioner </div><div><span style="white-space:pre-wrap"> </span>pc = PETSc.PC().create()</div><div><span style="white-space:pre-wrap"> </span>pc.setType(pc.Type.BJACOBI)</div><div><br></div><div><span style="white-space:pre-wrap"> </span># Set the linear solver </div><div><span style="white-space:pre-wrap"> </span># Create the KSP object</div><div><span style="white-space:pre-wrap"> </span>ksp = PETSc.KSP().create()</div><div><span style="white-space:pre-wrap"> </span># Create the solver, in this case GMRES</div><div><span style="white-space:pre-wrap"> </span>ksp.setType(ksp.Type.GMRES)</div><div><span style="white-space:pre-wrap"> </span># Set the tolerances of the GMRES solver</div><div> # Link it to the PC</div><div><span style="white-space:pre-wrap"> </span>ksp.setPC(pc)</div><div><br></div><div><span style="white-space:pre-wrap"> </span># Set up the spectral transformations </div><div><span style="white-space:pre-wrap"> </span>st = SLEPc.ST().create()</div><div><span style="white-space:pre-wrap"> </span>st.setType("shift")</div><div><span style="white-space:pre-wrap"> </span>st.setKSP(ksp)</div><div><span style="white-space:pre-wrap"> </span></div><div><span style="white-space:pre-wrap"> </span># MPD stands for "maximum projected dimension". It has to due with computational cost, please read Chap. 2.6.5 of SLEPc docu for </div><div><span style="white-space:pre-wrap"> </span># an explanation. At the moment mpd is only a guess</div><div><span style="white-space:pre-wrap"> </span>E.setDimensions(nev=NEV, mpd = MPD)</div><div><span style="white-space:pre-wrap"> </span># Eigenvalues should be real, therefore we start to order them from the smallest real value |l.real|</div><div><span style="white-space:pre-wrap"> </span>E.setWhichEigenpairs(E.Which.SMALLEST_REAL)</div><div><span style="white-space:pre-wrap"> </span># Since the dynamical matrix is symmetric and real it is hermitian </div><div><span style="white-space:pre-wrap"> </span>E.setProblemType(SLEPc.EPS.ProblemType.HEP)</div><div><span style="white-space:pre-wrap"> </span># Use the Krylov Schur method to solve the eigenvalue problem</div><div><span style="white-space:pre-wrap"> </span>E.setType(E.Type.KRYLOVSCHUR)</div><div><span style="white-space:pre-wrap"> </span># Set the convergence criterion to relative to the eigenvalue and the maximal number of iterations</div><div><span style="white-space:pre-wrap"> </span>E.setConvergenceTest(E.Conv.REL)</div><div><span style="white-space:pre-wrap"> </span>E.setTolerances(tol = 1e-8, max_it = 5000)</div><div><span style="white-space:pre-wrap"> </span># Set the matrix in order to solve </div><div><span style="white-space:pre-wrap"> </span>E.setOperators(DynMatrix_nn, None)</div><div><span style="white-space:pre-wrap"> </span># Sets EPS options from the options database. This routine must be called before `setUp()` if the user is to be allowed to set dthe solver type.</div><div><span style="white-space:pre-wrap"> </span>E.setFromOptions()</div><div><span style="white-space:pre-wrap"> </span># Sets up all the internal data structures necessary for the execution of the eigensolver.</div><div><span style="white-space:pre-wrap"> </span>E.setUp()</div><div><br></div><div><span style="white-space:pre-wrap"> </span># Solve eigenvalue problem </div><div><span style="white-space:pre-wrap"> </span>E.solve()</div><div><br></div><div><span style="white-space:pre-wrap"> </span>Print = PETSc.Sys.Print</div><div><br></div><div><span style="white-space:pre-wrap"> </span>Print()</div><div><span style="white-space:pre-wrap"> </span>Print("****************************")</div><div><span style="white-space:pre-wrap"> </span>Print("***SLEPc Solution Results***")</div><div><span style="white-space:pre-wrap"> </span>Print("****************************")</div><div><br></div><div><span style="white-space:pre-wrap"> </span>its = E.getIterationNumber()</div><div><span style="white-space:pre-wrap"> </span>Print("Number of iterations of the method: ", its)</div><div><span style="white-space:pre-wrap"> </span>eps_type = E.getType()</div><div><span style="white-space:pre-wrap"> </span>Print("Solution method: ", eps_type)</div><div><span style="white-space:pre-wrap"> </span>nev, ncv, mpd = E.getDimensions()</div><div><span style="white-space:pre-wrap"> </span>Print("Number of requested eigenvalues: ", nev)</div><div><span style="white-space:pre-wrap"> </span>Print("Number of computeded eigenvectors: ", ncv)</div><div><span style="white-space:pre-wrap"> </span>tol, maxit = E.getTolerances()</div><div><span style="white-space:pre-wrap"> </span>Print("Stopping condition: (tol, maxit)", (tol, maxit))</div><div><span style="white-space:pre-wrap"> </span># Get the type of convergence</div><div><span style="white-space:pre-wrap"> </span>conv_test = E.getConvergenceTest()</div><div><span style="white-space:pre-wrap"> </span>Print("Selected convergence test: ", conv_test)</div><div><span style="white-space:pre-wrap"> </span># Get the used spectral transformation</div><div><span style="white-space:pre-wrap"> </span>get_st = E.getST()</div><div><span style="white-space:pre-wrap"> </span>Print("Selected spectral transformation: ", get_st)</div><div><span style="white-space:pre-wrap"> </span># Get the applied direct solver</div><div><span style="white-space:pre-wrap"> </span>get_ksp = E.getDS()</div><div><span style="white-space:pre-wrap"> </span>Print("Selected direct solver: ", get_ksp)</div><div><span style="white-space:pre-wrap"> </span>nconv = E.getConverged()</div><div><span style="white-space:pre-wrap"> </span>Print("Number of converged eigenpairs: ", nconv)</div></div><div>.....</div><div><br></div><div><br></div></div></div><br><div class="gmail_quote"><div dir="ltr">Am Fr., 19. Okt. 2018 um 21:00 Uhr schrieb Smith, Barry F. <<a href="mailto:bsmith@mcs.anl.gov" target="_blank">bsmith@mcs.anl.gov</a>>:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><br>
<br>
> On Oct 19, 2018, at 7:56 AM, Zhang, Junchao <<a href="mailto:jczhang@mcs.anl.gov" target="_blank">jczhang@mcs.anl.gov</a>> wrote:<br>
> <br>
> <br>
> On Fri, Oct 19, 2018 at 4:02 AM Jan Grießer <<a href="mailto:griesser.jan@googlemail.com" target="_blank">griesser.jan@googlemail.com</a>> wrote:<br>
> With more than 1 MPI process you mean i should use spectrum slicing in divide the full problem in smaller subproblems? <br>
> The --with-64-bit-indices is not a possibility for me since i configured petsc with mumps, which does not allow to use the 64-bit version (At least this was the error message when i tried to configure PETSc )<br>
> <br>
> MUMPS 5.1.2 manual chapter 2.4.2 says it supports "Selective 64-bit integer feature" and "full 64-bit integer version" as well. <br>
<br>
They use to achieve this by compiling with special Fortran flags to promote integers to 64 bit; this is too fragile for our taste so we never hooked PETSc up wit it. If they have a dependable way of using 64 bit integers we should add that to our mumps.py and test it.<br>
<br>
Barry<br>
<br>
> <br>
> Am Mi., 17. Okt. 2018 um 18:24 Uhr schrieb Jose E. Roman <<a href="mailto:jroman@dsic.upv.es" target="_blank">jroman@dsic.upv.es</a>>:<br>
> To use BVVECS just add the command-line option -bv_type vecs<br>
> This causes to use a separate Vec for each column, instead of a single long Vec of size n*m. But it is considerably slower than the default.<br>
> <br>
> Anyway, for such large problems you should consider using more than 1 MPI process. In that case the error may disappear because the local size is smaller than 768000.<br>
> <br>
> Jose<br>
> <br>
> <br>
> > El 17 oct 2018, a las 17:58, Matthew Knepley <<a href="mailto:knepley@gmail.com" target="_blank">knepley@gmail.com</a>> escribió:<br>
> > <br>
> > On Wed, Oct 17, 2018 at 11:54 AM Jan Grießer <<a href="mailto:griesser.jan@googlemail.com" target="_blank">griesser.jan@googlemail.com</a>> wrote:<br>
> > Hi all,<br>
> > i am using slepc4py and petsc4py to solve for the smallest real eigenvalues and eigenvectors. For my test cases with a matrix A of the size 30k x 30k solving for the smallest soutions works quite well, but when i increase the dimension of my system to around A = 768000 x 768000 or 3 million x 3 million and ask for the smallest real 3000 (the number is increasing with increasing system size) eigenvalues and eigenvectors i get the output (for the 768000): <br>
> > The product 4001 times 768000 overflows the size of PetscInt; consider reducing the number of columns, or use BVVECS instead<br>
> > i understand that the requested number of eigenvectors and eigenvalues is causing an overflow but i do not understand the solution of the problem which is stated in the error message. Can someone tell me what exactly BVVECS is and how i can use it? Or is there any other solution to my problem ?<br>
> > <br>
> > You can also reconfigure with 64-bit integers: --with-64-bit-indices<br>
> > <br>
> > Thanks,<br>
> > <br>
> > Matt<br>
> > <br>
> > Thank you very much in advance,<br>
> > Jan <br>
> > <br>
> > <br>
> > <br>
> > -- <br>
> > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>
> > -- Norbert Wiener<br>
> > <br>
> > <a href="https://www.cse.buffalo.edu/~knepley/" rel="noreferrer" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br>
> <br>
<br>
</blockquote></div>
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr" class="m_2025512012978410376m_-6572347849915068202m_2285612461815204390m_4902418352757928713m_2802985989218999710m_1486663904307675610m_4090158591574469003m_9117256379228905931m_-1120326649911867329gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>-- Norbert Wiener</div><div><br></div><div><a href="http://www.cse.buffalo.edu/~knepley/" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br></div></div></div></div></div></div></div></div>
</blockquote></div>
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr" class="m_2025512012978410376m_-6572347849915068202m_2285612461815204390m_4902418352757928713m_2802985989218999710m_1486663904307675610m_4090158591574469003gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>-- Norbert Wiener</div><div><br></div><div><a href="http://www.cse.buffalo.edu/~knepley/" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br></div></div></div></div></div></div></div></div>
</blockquote></div>
</blockquote></div>
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr" class="m_2025512012978410376m_-6572347849915068202m_2285612461815204390gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>-- Norbert Wiener</div><div><br></div><div><a href="http://www.cse.buffalo.edu/~knepley/" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br></div></div></div></div></div></div></div>
</blockquote></div>
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr" class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>-- Norbert Wiener</div><div><br></div><div><a href="http://www.cse.buffalo.edu/~knepley/" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br></div></div></div></div></div></div></div></div>