[petsc-users] MUMPS error
Matthew Knepley
knepley at gmail.com
Sun May 24 09:06:21 CDT 2015
On Sun, May 24, 2015 at 8:57 AM, venkatesh g <venkateshgk.j at gmail.com>
wrote:
> I am using Matload option as in the ex7.c code given by the Slepc.
> ierr = MatLoad(A,viewer);CHKERRQ(ierr);
>
>
> There is no problem here right ? or any additional option is required for
> very large matrices while running the eigensolver in parallel ?
>
This will load the matrix from the viewer (presumably disk). There are no
options for large matrices.
Thanks,
Matt
> cheers,
> Venkatesh
>
> On Sat, May 23, 2015 at 5:43 PM, Matthew Knepley <knepley at gmail.com>
> wrote:
>
>> On Sat, May 23, 2015 at 7:09 AM, venkatesh g <venkateshgk.j at gmail.com>
>> wrote:
>>
>>> Hi,
>>> Thanks.
>>> Per node it has 24 cores and each core has 4 GB RAM. And the job was
>>> submitted in 10 nodes.
>>>
>>> So, does it mean it requires 10G for one core ? or for 1 node ?
>>>
>>
>> The error message from MUMPS said that it tried to allocate 10G. We must
>> assume each process
>> tried to do the same thing. That means if you scheduled 24 processes on a
>> node, it would try to
>> allocate at least 240G, which is in excess of what you specify above.
>>
>> Note that this has nothing to do with PETSc. It is all in the
>> documentation for that machine and its
>> scheduling policy.
>>
>> Thanks,
>>
>> Matt
>>
>>
>>> cheers,
>>>
>>> Venkatesh
>>>
>>> On Sat, May 23, 2015 at 5:17 PM, Matthew Knepley <knepley at gmail.com>
>>> wrote:
>>>
>>>> On Sat, May 23, 2015 at 6:44 AM, venkatesh g <venkateshgk.j at gmail.com>
>>>> wrote:
>>>>
>>>>> Hi,
>>>>> The same eigenproblem runs with 120 GB RAM in a serial machine in
>>>>> Matlab.
>>>>>
>>>>> In Cray I fired with 240*4 GB RAM in parallel. So it has to go in
>>>>> right ?
>>>>>
>>>>
>>>> I do not know how MUMPS allocates memory, but the message is
>>>> unambiguous. Also,
>>>> this is concerned with the memory available per node. Do you know how
>>>> many processes
>>>> per node were scheduled? The message below indicates that it was trying
>>>> to allocate 10G
>>>> for one process.
>>>>
>>>>
>>>>> And for small matrices it is having negative scaling i.e 24 core is
>>>>> running faster.
>>>>>
>>>>
>>>> Yes, for strong scaling you always get slowdown eventually since
>>>> overheads dominate
>>>> work, see Amdahl's Law.
>>>>
>>>> Thanks,
>>>>
>>>> Matt
>>>>
>>>>
>>>>> I have attached the submission script.
>>>>>
>>>>> Pls see.. Kindly let me know
>>>>>
>>>>> cheers,
>>>>> Venkatesh
>>>>>
>>>>>
>>>>> On Sat, May 23, 2015 at 4:58 PM, Matthew Knepley <knepley at gmail.com>
>>>>> wrote:
>>>>>
>>>>>> On Sat, May 23, 2015 at 2:39 AM, venkatesh g <venkateshgk.j at gmail.com
>>>>>> > wrote:
>>>>>>
>>>>>>> Hi again,
>>>>>>>
>>>>>>> I have installed the Petsc and Slepc in Cray with intel compilers
>>>>>>> with Mumps.
>>>>>>>
>>>>>>> I am getting this error when I solve eigenvalue problem with large
>>>>>>> matrices: [201]PETSC ERROR: Error reported by MUMPS in numerical
>>>>>>> factorization phase: Cannot allocate required memory 9632 megabytes
>>>>>>>
>>>>>>
>>>>>> It ran out of memory on the node.
>>>>>>
>>>>>>
>>>>>>> Also it is again not scaling well for small matrices.
>>>>>>>
>>>>>>
>>>>>> MUMPS strong scaling for small matrices is not very good. Weak
>>>>>> scaling is looking at big matrices.
>>>>>>
>>>>>> Thanks,
>>>>>>
>>>>>> Matt
>>>>>>
>>>>>>
>>>>>>> Kindly let me know what to do.
>>>>>>>
>>>>>>> cheers,
>>>>>>>
>>>>>>> Venkatesh
>>>>>>>
>>>>>>>
>>>>>>> On Tue, May 19, 2015 at 3:02 PM, Matthew Knepley <knepley at gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> On Tue, May 19, 2015 at 1:04 AM, venkatesh g <
>>>>>>>> venkateshgk.j at gmail.com> wrote:
>>>>>>>>
>>>>>>>>> Hi,
>>>>>>>>>
>>>>>>>>> I have attached the log of the command which I gave in the master
>>>>>>>>> node: make streams NPMAX=32
>>>>>>>>>
>>>>>>>>> I dont know why it says 'It appears you have only 1 node'. But
>>>>>>>>> other codes run in parallel with good scaling on 8 nodes.
>>>>>>>>>
>>>>>>>>
>>>>>>>> If you look at the STREAMS numbers, you can see that your system is
>>>>>>>> only able to support about 2 cores with the
>>>>>>>> available memory bandwidth. Thus for bandwidth constrained
>>>>>>>> operations (almost everything in sparse linear algebra
>>>>>>>> and solvers), your speedup will not be bigger than 2.
>>>>>>>>
>>>>>>>> Other codes may do well on this machine, but they would be compute
>>>>>>>> constrained, using things like DGEMM.
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>>
>>>>>>>> Matt
>>>>>>>>
>>>>>>>>
>>>>>>>>> Kindly let me know.
>>>>>>>>>
>>>>>>>>> Venkatesh
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Mon, May 18, 2015 at 11:21 PM, Barry Smith <bsmith at mcs.anl.gov>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Run the streams benchmark on this system and send the results.
>>>>>>>>>> http://www.mcs.anl.gov/petsc/documentation/faq.html#computers
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> > On May 18, 2015, at 11:14 AM, venkatesh g <
>>>>>>>>>> venkateshgk.j at gmail.com> wrote:
>>>>>>>>>> >
>>>>>>>>>> > Hi,
>>>>>>>>>> > I have emailed the mumps-user list.
>>>>>>>>>> > Actually the cluster has 8 nodes with 16 cores, and other codes
>>>>>>>>>> scale well.
>>>>>>>>>> > I wanted to ask if this job takes much time, then if I submit
>>>>>>>>>> on more cores, I have to increase the icntl(14).. which would again take
>>>>>>>>>> long time.
>>>>>>>>>> >
>>>>>>>>>> > So is there another way ?
>>>>>>>>>> >
>>>>>>>>>> > cheers,
>>>>>>>>>> > Venkatesh
>>>>>>>>>> >
>>>>>>>>>> > On Mon, May 18, 2015 at 7:16 PM, Matthew Knepley <
>>>>>>>>>> knepley at gmail.com> wrote:
>>>>>>>>>> > On Mon, May 18, 2015 at 8:29 AM, venkatesh g <
>>>>>>>>>> venkateshgk.j at gmail.com> wrote:
>>>>>>>>>> > Hi I have attached the performance logs for 2 jobs on different
>>>>>>>>>> processors. I had to increase the workspace icntl(14) when I submit on more
>>>>>>>>>> cores since it is failing with small value of icntl(14).
>>>>>>>>>> >
>>>>>>>>>> > 1. performance_log1.txt is run on 8 cores (option given
>>>>>>>>>> -mat_mumps_icntl_14 200)
>>>>>>>>>> > 2. performance_log2.txt is run on 2 cores (option given
>>>>>>>>>> -mat_mumps_icntl_14 85 )
>>>>>>>>>> >
>>>>>>>>>> > 1) Your number of iterates increased from 7600 to 9600, but
>>>>>>>>>> that is a relatively small effect
>>>>>>>>>> >
>>>>>>>>>> > 2) MUMPS is just taking a lot longer to do forward/backward
>>>>>>>>>> solve. You might try emailing
>>>>>>>>>> > the list for them. However, I would bet that your system has
>>>>>>>>>> enough bandwidth for 2 procs
>>>>>>>>>> > and not much more.
>>>>>>>>>> >
>>>>>>>>>> > Thanks,
>>>>>>>>>> >
>>>>>>>>>> > Matt
>>>>>>>>>> >
>>>>>>>>>> > Venkatesh
>>>>>>>>>> >
>>>>>>>>>> > On Sun, May 17, 2015 at 6:13 PM, Matthew Knepley <
>>>>>>>>>> knepley at gmail.com> wrote:
>>>>>>>>>> > On Sun, May 17, 2015 at 1:38 AM, venkatesh g <
>>>>>>>>>> venkateshgk.j at gmail.com> wrote:
>>>>>>>>>> > Hi, Thanks for the information. I now increased the workspace
>>>>>>>>>> by adding '-mat_mumps_icntl_14 100'
>>>>>>>>>> >
>>>>>>>>>> > It works. However, the problem is, if I submit in 1 core I get
>>>>>>>>>> the answer in 200 secs, but with 4 cores and '-mat_mumps_icntl_14 100' it
>>>>>>>>>> takes 3500secs.
>>>>>>>>>> >
>>>>>>>>>> > Send the output of -log_summary for all performance queries.
>>>>>>>>>> Otherwise we are just guessing.
>>>>>>>>>> >
>>>>>>>>>> > Matt
>>>>>>>>>> >
>>>>>>>>>> > My command line is: 'mpiexec -np 4 ./ex7 -f1 a2 -f2 b2 -eps_nev
>>>>>>>>>> 1 -st_type sinvert -eps_max_it 5000 -st_ksp_type preonly -st_pc_type lu
>>>>>>>>>> -st_pc_factor_mat_solver_package mumps -mat_mumps_icntl_14 100'
>>>>>>>>>> >
>>>>>>>>>> > Kindly let me know.
>>>>>>>>>> >
>>>>>>>>>> > Venkatesh
>>>>>>>>>> >
>>>>>>>>>> >
>>>>>>>>>> >
>>>>>>>>>> > On Sat, May 16, 2015 at 7:10 PM, David Knezevic <
>>>>>>>>>> david.knezevic at akselos.com> wrote:
>>>>>>>>>> > On Sat, May 16, 2015 at 8:08 AM, venkatesh g <
>>>>>>>>>> venkateshgk.j at gmail.com> wrote:
>>>>>>>>>> > Hi,
>>>>>>>>>> > I am trying to solving AX=lambda BX eigenvalue problem.
>>>>>>>>>> >
>>>>>>>>>> > A and B are of sizes 3600x3600
>>>>>>>>>> >
>>>>>>>>>> > I run with this command :
>>>>>>>>>> >
>>>>>>>>>> > 'mpiexec -np 4 ./ex7 -f1 a2 -f2 b2 -eps_nev 1 -st_type sinvert
>>>>>>>>>> -eps_max_it 5000 -st_ksp_type preonly -st_pc_type lu
>>>>>>>>>> -st_pc_factor_mat_solver_package mumps'
>>>>>>>>>> >
>>>>>>>>>> > I get this error: (I get result only when I give 1 or 2
>>>>>>>>>> processors)
>>>>>>>>>> > Reading COMPLEX matrices from binary files...
>>>>>>>>>> > [0]PETSC ERROR: --------------------- Error Message
>>>>>>>>>> ------------------------------------
>>>>>>>>>> > [0]PETSC ERROR: Error in external library!
>>>>>>>>>> > [0]PETSC ERROR: Error reported by MUMPS in numerical
>>>>>>>>>> factorization phase: INFO(1)=-9, INFO(2)=2024
>>>>>>>>>> >
>>>>>>>>>> >
>>>>>>>>>> > The MUMPS error types are described in Chapter 7 of the MUMPS
>>>>>>>>>> manual. In this case you have INFO(1)=-9, which is explained in the manual
>>>>>>>>>> as:
>>>>>>>>>> >
>>>>>>>>>> > "–9 Main internal real/complex workarray S too small. If
>>>>>>>>>> INFO(2) is positive, then the number of entries that are missing in S at
>>>>>>>>>> the moment when the error is raised is available in INFO(2). If INFO(2) is
>>>>>>>>>> negative, then its absolute value should be multiplied by 1 million. If an
>>>>>>>>>> error –9 occurs, the user should increase the value of ICNTL(14) before
>>>>>>>>>> calling the factorization (JOB=2) again, except if ICNTL(23) is provided,
>>>>>>>>>> in which case ICNTL(23) should be increased."
>>>>>>>>>> >
>>>>>>>>>> > This says that you should use ICTNL(14) to increase the working
>>>>>>>>>> space size:
>>>>>>>>>> >
>>>>>>>>>> > "ICNTL(14) is accessed by the host both during the analysis and
>>>>>>>>>> the factorization phases. It corresponds to the percentage increase in the
>>>>>>>>>> estimated working space. When significant extra fill-in is caused by
>>>>>>>>>> numerical pivoting, increasing ICNTL(14) may help. Except in special cases,
>>>>>>>>>> the default value is 20 (which corresponds to a 20 % increase)."
>>>>>>>>>> >
>>>>>>>>>> > So, for example, you can avoid this error via the following
>>>>>>>>>> command line argument to PETSc: "-mat_mumps_icntl_14 30", where 30
>>>>>>>>>> indicates that we allow a 30% increase in the workspace instead of the
>>>>>>>>>> default 20%.
>>>>>>>>>> >
>>>>>>>>>> > David
>>>>>>>>>> >
>>>>>>>>>> >
>>>>>>>>>> >
>>>>>>>>>> >
>>>>>>>>>> >
>>>>>>>>>> >
>>>>>>>>>> > --
>>>>>>>>>> > What most experimenters take for granted before they begin
>>>>>>>>>> their experiments is infinitely more interesting than any results to which
>>>>>>>>>> their experiments lead.
>>>>>>>>>> > -- Norbert Wiener
>>>>>>>>>> >
>>>>>>>>>> >
>>>>>>>>>> >
>>>>>>>>>> >
>>>>>>>>>> > --
>>>>>>>>>> > What most experimenters take for granted before they begin
>>>>>>>>>> their experiments is infinitely more interesting than any results to which
>>>>>>>>>> their experiments lead.
>>>>>>>>>> > -- Norbert Wiener
>>>>>>>>>> >
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> What most experimenters take for granted before they begin their
>>>>>>>> experiments is infinitely more interesting than any results to which their
>>>>>>>> experiments lead.
>>>>>>>> -- Norbert Wiener
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> What most experimenters take for granted before they begin their
>>>>>> experiments is infinitely more interesting than any results to which their
>>>>>> experiments lead.
>>>>>> -- Norbert Wiener
>>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> What most experimenters take for granted before they begin their
>>>> experiments is infinitely more interesting than any results to which their
>>>> experiments lead.
>>>> -- Norbert Wiener
>>>>
>>>
>>>
>>
>>
>> --
>> What most experimenters take for granted before they begin their
>> experiments is infinitely more interesting than any results to which their
>> experiments lead.
>> -- Norbert Wiener
>>
>
>
--
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150524/27265309/attachment.html>
More information about the petsc-users
mailing list