[petsc-users] PetscInt overflow

[Matthew Knepley]

On Wed, Oct 24, 2018 at 9:38 AM Jan Grießer <griesser.jan at googlemail.com>
wrote:

> Hey,
> i tried to run my program as you said with -bv_type vecs and/or -bv_type
> mat, but instead of the PETScInt overflow i now get an MPI Error 9
>

Send the actual error.

  Thanks,

    Matt


> message, which i assume (after googling a little bit around) should be a
> memory problem. I tried to run it also on slightly bigger compute nodes on
> our cluster with 20 cores with each 12 GB and 24 GB but the problem still
> remains. The actual limit appears to be a 1.5 Million x 1.5 Million where
> i  searched for NEV = 1500 and MPD = 500/ 200 eigenvalues.
> Do you have maybe an idea what the error could be? I appended also the
> python method i used to solve the problem. I also tried to solve the
> problem with spectrum solving but the error message remains the same.
>
> def solve_eigensystem(DynMatrix_nn, NEV, MPD, Dimension):
> # Create the solver
> # E is used as an acronym for the EPS solver (EPS = Eigenvalue problem
> solver)
> E = SLEPc.EPS().create()
>
> # Set the preconditioner
> pc = PETSc.PC().create()
> pc.setType(pc.Type.BJACOBI)
>
> # Set the linear solver
> # Create the KSP object
> ksp = PETSc.KSP().create()
> # Create the solver, in this case GMRES
> ksp.setType(ksp.Type.GMRES)
> # Set the tolerances of the GMRES solver
>         # Link it to the PC
> ksp.setPC(pc)
>
> # Set up the spectral transformations
> st = SLEPc.ST().create()
> st.setType("shift")
> st.setKSP(ksp)
> # MPD stands for "maximum projected dimension". It has to due with
> computational cost, please read Chap. 2.6.5 of SLEPc docu for
> # an explanation. At the moment mpd is only a guess
> E.setDimensions(nev=NEV, mpd = MPD)
> # Eigenvalues should be real, therefore we start to order them from the
> smallest real value |l.real|
> E.setWhichEigenpairs(E.Which.SMALLEST_REAL)
> # Since the dynamical matrix is symmetric and real it is hermitian
> E.setProblemType(SLEPc.EPS.ProblemType.HEP)
> # Use the Krylov Schur method to solve the eigenvalue problem
> E.setType(E.Type.KRYLOVSCHUR)
> # Set the convergence criterion to relative to the eigenvalue and the
> maximal number of iterations
> E.setConvergenceTest(E.Conv.REL)
> E.setTolerances(tol = 1e-8, max_it = 5000)
> # Set the matrix in order to solve
> E.setOperators(DynMatrix_nn, None)
> # Sets EPS options from the options database. This routine must be called
> before `setUp()` if the user is to be allowed to set dthe solver type.
> E.setFromOptions()
> # Sets up all the internal data structures necessary for the execution of
> the eigensolver.
> E.setUp()
>
> # Solve eigenvalue problem
> E.solve()
>
> Print = PETSc.Sys.Print
>
> Print()
> Print("****************************")
> Print("***SLEPc Solution Results***")
> Print("****************************")
>
> its = E.getIterationNumber()
> Print("Number of iterations of the method: ", its)
> eps_type = E.getType()
> Print("Solution method: ", eps_type)
> nev, ncv, mpd = E.getDimensions()
> Print("Number of requested eigenvalues: ", nev)
> Print("Number of computeded eigenvectors: ", ncv)
> tol, maxit = E.getTolerances()
> Print("Stopping condition: (tol, maxit)", (tol, maxit))
> # Get the type of convergence
> conv_test = E.getConvergenceTest()
> Print("Selected convergence test: ", conv_test)
> # Get the used spectral transformation
> get_st = E.getST()
> Print("Selected spectral transformation: ", get_st)
> # Get the applied direct solver
> get_ksp = E.getDS()
> Print("Selected direct solver: ", get_ksp)
> nconv = E.getConverged()
> Print("Number of converged eigenpairs: ", nconv)
> .....
>
>
>
> Am Fr., 19. Okt. 2018 um 21:00 Uhr schrieb Smith, Barry F. <
> bsmith at mcs.anl.gov>:
>
>>
>>
>> > On Oct 19, 2018, at 7:56 AM, Zhang, Junchao <jczhang at mcs.anl.gov>
>> wrote:
>> >
>> >
>> > On Fri, Oct 19, 2018 at 4:02 AM Jan Grießer <
>> griesser.jan at googlemail.com> wrote:
>> > With more than 1 MPI process you mean i should use spectrum slicing in
>> divide the full problem in smaller subproblems?
>> > The --with-64-bit-indices is not a possibility for me since i
>> configured petsc with mumps, which does not allow to use the 64-bit version
>> (At least this was the error message when i tried to configure PETSc )
>> >
>> > MUMPS 5.1.2 manual chapter 2.4.2 says it supports "Selective 64-bit
>> integer feature" and "full 64-bit integer version" as well.
>>
>>     They use to achieve this by compiling with special Fortran flags to
>> promote integers to 64 bit; this is too fragile for our taste so we never
>> hooked PETSc up wit it. If they have a dependable way of using 64 bit
>> integers we should add that to our mumps.py and test it.
>>
>>    Barry
>>
>> >
>> > Am Mi., 17. Okt. 2018 um 18:24 Uhr schrieb Jose E. Roman <
>> jroman at dsic.upv.es>:
>> > To use BVVECS just add the command-line option -bv_type vecs
>> > This causes to use a separate Vec for each column, instead of a single
>> long Vec of size n*m. But it is considerably slower than the default.
>> >
>> > Anyway, for such large problems you should consider using more than 1
>> MPI process. In that case the error may disappear because the local size is
>> smaller than 768000.
>> >
>> > Jose
>> >
>> >
>> > > El 17 oct 2018, a las 17:58, Matthew Knepley <knepley at gmail.com>
>> escribió:
>> > >
>> > > On Wed, Oct 17, 2018 at 11:54 AM Jan Grießer <
>> griesser.jan at googlemail.com> wrote:
>> > > Hi all,
>> > > i am using slepc4py and petsc4py to solve for the smallest real
>> eigenvalues and eigenvectors. For my test cases with a matrix A of the size
>> 30k x 30k solving for the smallest soutions works quite well, but when i
>> increase the dimension of my system to around A = 768000 x 768000 or 3
>> million x 3 million and ask for the smallest real 3000 (the number is
>> increasing with increasing system size) eigenvalues and eigenvectors i get
>> the output (for the 768000):
>> > >  The product 4001 times 768000 overflows the size of PetscInt;
>> consider reducing the number of columns, or use BVVECS instead
>> > > i understand that the requested number of eigenvectors and
>> eigenvalues is causing an overflow but i do not understand the solution of
>> the problem which is stated in the error message. Can someone tell me what
>> exactly BVVECS is and how i can use it? Or is there any other solution to
>> my problem ?
>> > >
>> > > You can also reconfigure with 64-bit integers: --with-64-bit-indices
>> > >
>> > >   Thanks,
>> > >
>> > >     Matt
>> > >
>> > > Thank you very much in advance,
>> > > Jan
>> > >
>> > >
>> > >
>> > > --
>> > > What most experimenters take for granted before they begin their
>> experiments is infinitely more interesting than any results to which their
>> experiments lead.
>> > > -- Norbert Wiener
>> > >
>> > > https://www.cse.buffalo.edu/~knepley/
>> >
>>
>>

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181024/af7ad114/attachment.html>