[petsc-users] PetscInt overflow

Wed Oct 24 09:03:08 CDT 2018

This is the error message i get from my program:
Shape of the dynamical matrix:  (1500000, 1500000)

===================================================================================
=   BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
=   PID 122676 RUNNING AT n3512.nemo.privat
=   EXIT CODE: 9
=   CLEANING UP REMAINING PROCESSES
=   YOU CAN IGNORE THE BELOW CLEANUP MESSAGES
===================================================================================
   Intel(R) MPI Library troubleshooting guide:
      https://software.intel.com/node/561764
===================================================================================


Am Mi., 24. Okt. 2018 um 16:01 Uhr schrieb Matthew Knepley <
knepley at gmail.com>:

> On Wed, Oct 24, 2018 at 9:38 AM Jan Grießer <griesser.jan at googlemail.com>
> wrote:
>
>> Hey,
>> i tried to run my program as you said with -bv_type vecs and/or -bv_type
>> mat, but instead of the PETScInt overflow i now get an MPI Error 9
>>
>
> Send the actual error.
>
>   Thanks,
>
>     Matt
>
>
>> message, which i assume (after googling a little bit around) should be a
>> memory problem. I tried to run it also on slightly bigger compute nodes on
>> our cluster with 20 cores with each 12 GB and 24 GB but the problem still
>> remains. The actual limit appears to be a 1.5 Million x 1.5 Million where
>> i  searched for NEV = 1500 and MPD = 500/ 200 eigenvalues.
>> Do you have maybe an idea what the error could be? I appended also the
>> python method i used to solve the problem. I also tried to solve the
>> problem with spectrum solving but the error message remains the same.
>>
>> def solve_eigensystem(DynMatrix_nn, NEV, MPD, Dimension):
>> # Create the solver
>> # E is used as an acronym for the EPS solver (EPS = Eigenvalue problem
>> solver)
>> E = SLEPc.EPS().create()
>>
>> # Set the preconditioner
>> pc = PETSc.PC().create()
>> pc.setType(pc.Type.BJACOBI)
>>
>> # Set the linear solver
>> # Create the KSP object
>> ksp = PETSc.KSP().create()
>> # Create the solver, in this case GMRES
>> ksp.setType(ksp.Type.GMRES)
>> # Set the tolerances of the GMRES solver
>>         # Link it to the PC
>> ksp.setPC(pc)
>>
>> # Set up the spectral transformations
>> st = SLEPc.ST().create()
>> st.setType("shift")
>> st.setKSP(ksp)
>> # MPD stands for "maximum projected dimension". It has to due with
>> computational cost, please read Chap. 2.6.5 of SLEPc docu for
>> # an explanation. At the moment mpd is only a guess
>> E.setDimensions(nev=NEV, mpd = MPD)
>> # Eigenvalues should be real, therefore we start to order them from the
>> smallest real value |l.real|
>> E.setWhichEigenpairs(E.Which.SMALLEST_REAL)
>> # Since the dynamical matrix is symmetric and real it is hermitian
>> E.setProblemType(SLEPc.EPS.ProblemType.HEP)
>> # Use the Krylov Schur method to solve the eigenvalue problem
>> E.setType(E.Type.KRYLOVSCHUR)
>> # Set the convergence criterion to relative to the eigenvalue and the
>> maximal number of iterations
>> E.setConvergenceTest(E.Conv.REL)
>> E.setTolerances(tol = 1e-8, max_it = 5000)
>> # Set the matrix in order to solve
>> E.setOperators(DynMatrix_nn, None)
>> # Sets EPS options from the options database. This routine must be called
>> before `setUp()` if the user is to be allowed to set dthe solver type.
>> E.setFromOptions()
>> # Sets up all the internal data structures necessary for the execution of
>> the eigensolver.
>> E.setUp()
>>
>> # Solve eigenvalue problem
>> E.solve()
>>
>> Print = PETSc.Sys.Print
>>
>> Print()
>> Print("****************************")
>> Print("***SLEPc Solution Results***")
>> Print("****************************")
>>
>> its = E.getIterationNumber()
>> Print("Number of iterations of the method: ", its)
>> eps_type = E.getType()
>> Print("Solution method: ", eps_type)
>> nev, ncv, mpd = E.getDimensions()
>> Print("Number of requested eigenvalues: ", nev)
>> Print("Number of computeded eigenvectors: ", ncv)
>> tol, maxit = E.getTolerances()
>> Print("Stopping condition: (tol, maxit)", (tol, maxit))
>> # Get the type of convergence
>> conv_test = E.getConvergenceTest()
>> Print("Selected convergence test: ", conv_test)
>> # Get the used spectral transformation
>> get_st = E.getST()
>> Print("Selected spectral transformation: ", get_st)
>> # Get the applied direct solver
>> get_ksp = E.getDS()
>> Print("Selected direct solver: ", get_ksp)
>> nconv = E.getConverged()
>> Print("Number of converged eigenpairs: ", nconv)
>> .....
>>
>>
>>
>> Am Fr., 19. Okt. 2018 um 21:00 Uhr schrieb Smith, Barry F. <
>> bsmith at mcs.anl.gov>:
>>
>>>
>>>
>>> > On Oct 19, 2018, at 7:56 AM, Zhang, Junchao <jczhang at mcs.anl.gov>
>>> wrote:
>>> >
>>> >
>>> > On Fri, Oct 19, 2018 at 4:02 AM Jan Grießer <
>>> griesser.jan at googlemail.com> wrote:
>>> > With more than 1 MPI process you mean i should use spectrum slicing in
>>> divide the full problem in smaller subproblems?
>>> > The --with-64-bit-indices is not a possibility for me since i
>>> configured petsc with mumps, which does not allow to use the 64-bit version
>>> (At least this was the error message when i tried to configure PETSc )
>>> >
>>> > MUMPS 5.1.2 manual chapter 2.4.2 says it supports "Selective 64-bit
>>> integer feature" and "full 64-bit integer version" as well.
>>>
>>>     They use to achieve this by compiling with special Fortran flags to
>>> promote integers to 64 bit; this is too fragile for our taste so we never
>>> hooked PETSc up wit it. If they have a dependable way of using 64 bit
>>> integers we should add that to our mumps.py and test it.
>>>
>>>    Barry
>>>
>>> >
>>> > Am Mi., 17. Okt. 2018 um 18:24 Uhr schrieb Jose E. Roman <
>>> jroman at dsic.upv.es>:
>>> > To use BVVECS just add the command-line option -bv_type vecs
>>> > This causes to use a separate Vec for each column, instead of a single
>>> long Vec of size n*m. But it is considerably slower than the default.
>>> >
>>> > Anyway, for such large problems you should consider using more than 1
>>> MPI process. In that case the error may disappear because the local size is
>>> smaller than 768000.
>>> >
>>> > Jose
>>> >
>>> >
>>> > > El 17 oct 2018, a las 17:58, Matthew Knepley <knepley at gmail.com>
>>> escribió:
>>> > >
>>> > > On Wed, Oct 17, 2018 at 11:54 AM Jan Grießer <
>>> griesser.jan at googlemail.com> wrote:
>>> > > Hi all,
>>> > > i am using slepc4py and petsc4py to solve for the smallest real
>>> eigenvalues and eigenvectors. For my test cases with a matrix A of the size
>>> 30k x 30k solving for the smallest soutions works quite well, but when i
>>> increase the dimension of my system to around A = 768000 x 768000 or 3
>>> million x 3 million and ask for the smallest real 3000 (the number is
>>> increasing with increasing system size) eigenvalues and eigenvectors i get
>>> the output (for the 768000):
>>> > >  The product 4001 times 768000 overflows the size of PetscInt;
>>> consider reducing the number of columns, or use BVVECS instead
>>> > > i understand that the requested number of eigenvectors and
>>> eigenvalues is causing an overflow but i do not understand the solution of
>>> the problem which is stated in the error message. Can someone tell me what
>>> exactly BVVECS is and how i can use it? Or is there any other solution to
>>> my problem ?
>>> > >
>>> > > You can also reconfigure with 64-bit integers: --with-64-bit-indices
>>> > >
>>> > >   Thanks,
>>> > >
>>> > >     Matt
>>> > >
>>> > > Thank you very much in advance,
>>> > > Jan
>>> > >
>>> > >
>>> > >
>>> > > --
>>> > > What most experimenters take for granted before they begin their
>>> experiments is infinitely more interesting than any results to which their
>>> experiments lead.
>>> > > -- Norbert Wiener
>>> > >
>>> > > https://www.cse.buffalo.edu/~knepley/
>>> >
>>>
>>>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
> https://www.cse.buffalo.edu/~knepley/
> <http://www.cse.buffalo.edu/~knepley/>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181024/20c1fd46/attachment-0001.html>