[petsc-users] MUMPS error

Matthew Knepley knepley at gmail.com
Sat May 23 07:13:27 CDT 2015 

On Sat, May 23, 2015 at 7:09 AM, venkatesh g <venkateshgk.j at gmail.com>
wrote:

> Hi,
> Thanks.
> Per node it has 24 cores and each core has 4 GB RAM. And the job was
> submitted in 10 nodes.
>
> So, does it mean it requires 10G for one core ? or for 1 node ?
>

The error message from MUMPS said that it tried to allocate 10G. We must
assume each process
tried to do the same thing. That means if you scheduled 24 processes on a
node, it would try to
allocate at least 240G, which is in excess of what you specify above.

Note that this has nothing to do with PETSc. It is all in the documentation
for that machine and its
scheduling policy.

  Thanks,

     Matt


> cheers,
>
> Venkatesh
>
> On Sat, May 23, 2015 at 5:17 PM, Matthew Knepley <knepley at gmail.com>
> wrote:
>
>> On Sat, May 23, 2015 at 6:44 AM, venkatesh g <venkateshgk.j at gmail.com>
>> wrote:
>>
>>> Hi,
>>> The same eigenproblem runs with 120 GB RAM in a serial machine in
>>> Matlab.
>>>
>>> In Cray I fired with 240*4 GB RAM in parallel. So it has to go in right
>>> ?
>>>
>>
>> I do not know how MUMPS allocates memory, but the message is unambiguous.
>> Also,
>> this is concerned with the memory available per node. Do you know how
>> many processes
>> per node were scheduled? The message below indicates that it was trying
>> to allocate 10G
>> for one process.
>>
>>
>>> And for small matrices it is having negative scaling i.e 24 core is
>>> running faster.
>>>
>>
>> Yes, for strong scaling you always get slowdown eventually since
>> overheads dominate
>> work, see Amdahl's Law.
>>
>>   Thanks,
>>
>>      Matt
>>
>>
>>> I have attached the submission script.
>>>
>>> Pls see.. Kindly let me know
>>>
>>> cheers,
>>> Venkatesh
>>>
>>>
>>> On Sat, May 23, 2015 at 4:58 PM, Matthew Knepley <knepley at gmail.com>
>>> wrote:
>>>
>>>> On Sat, May 23, 2015 at 2:39 AM, venkatesh g <venkateshgk.j at gmail.com>
>>>> wrote:
>>>>
>>>>> Hi again,
>>>>>
>>>>> I have installed the Petsc and Slepc in Cray with intel compilers with
>>>>> Mumps.
>>>>>
>>>>> I am getting this error when I solve eigenvalue problem with large
>>>>> matrices: [201]PETSC ERROR: Error reported by MUMPS in numerical
>>>>> factorization phase: Cannot allocate required memory 9632 megabytes
>>>>>
>>>>
>>>> It ran out of memory on the node.
>>>>
>>>>
>>>>> Also it is again not scaling well for small matrices.
>>>>>
>>>>
>>>> MUMPS strong scaling for small matrices is not very good. Weak scaling
>>>> is looking at big matrices.
>>>>
>>>>   Thanks,
>>>>
>>>>      Matt
>>>>
>>>>
>>>>> Kindly let me know what to do.
>>>>>
>>>>> cheers,
>>>>>
>>>>> Venkatesh
>>>>>
>>>>>
>>>>> On Tue, May 19, 2015 at 3:02 PM, Matthew Knepley <knepley at gmail.com>
>>>>> wrote:
>>>>>
>>>>>> On Tue, May 19, 2015 at 1:04 AM, venkatesh g <venkateshgk.j at gmail.com
>>>>>> > wrote:
>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>> I have attached the log of the command which I gave in the master
>>>>>>> node: make streams NPMAX=32
>>>>>>>
>>>>>>> I dont know why it says 'It appears you have only 1 node'. But other
>>>>>>> codes run in parallel with good scaling on 8 nodes.
>>>>>>>
>>>>>>
>>>>>> If you look at the STREAMS numbers, you can see that your system is
>>>>>> only able to support about 2 cores with the
>>>>>> available memory bandwidth. Thus for bandwidth constrained operations
>>>>>> (almost everything in sparse linear algebra
>>>>>> and solvers), your speedup will not be bigger than 2.
>>>>>>
>>>>>> Other codes may do well on this machine, but they would be compute
>>>>>> constrained, using things like DGEMM.
>>>>>>
>>>>>>   Thanks,
>>>>>>
>>>>>>      Matt
>>>>>>
>>>>>>
>>>>>>> Kindly let me know.
>>>>>>>
>>>>>>> Venkatesh
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Mon, May 18, 2015 at 11:21 PM, Barry Smith <bsmith at mcs.anl.gov>
>>>>>>> wrote:
>>>>>>>
>>>>>>>>
>>>>>>>>    Run the streams benchmark on this system and send the results.
>>>>>>>> http://www.mcs.anl.gov/petsc/documentation/faq.html#computers
>>>>>>>>
>>>>>>>>
>>>>>>>> > On May 18, 2015, at 11:14 AM, venkatesh g <
>>>>>>>> venkateshgk.j at gmail.com> wrote:
>>>>>>>> >
>>>>>>>> > Hi,
>>>>>>>> > I have emailed the mumps-user list.
>>>>>>>> > Actually the cluster has 8 nodes with 16 cores, and other codes
>>>>>>>> scale well.
>>>>>>>> > I wanted to ask if this job takes much time, then if I submit on
>>>>>>>> more cores, I have to increase the icntl(14).. which would again take long
>>>>>>>> time.
>>>>>>>> >
>>>>>>>> > So is there another way ?
>>>>>>>> >
>>>>>>>> > cheers,
>>>>>>>> > Venkatesh
>>>>>>>> >
>>>>>>>> > On Mon, May 18, 2015 at 7:16 PM, Matthew Knepley <
>>>>>>>> knepley at gmail.com> wrote:
>>>>>>>> > On Mon, May 18, 2015 at 8:29 AM, venkatesh g <
>>>>>>>> venkateshgk.j at gmail.com> wrote:
>>>>>>>> > Hi I have attached the performance logs for 2 jobs on different
>>>>>>>> processors. I had to increase the workspace icntl(14) when I submit on more
>>>>>>>> cores since it is failing with small value of icntl(14).
>>>>>>>> >
>>>>>>>> > 1. performance_log1.txt is run on 8 cores (option given
>>>>>>>> -mat_mumps_icntl_14 200)
>>>>>>>> > 2. performance_log2.txt is run on 2 cores (option given
>>>>>>>> -mat_mumps_icntl_14 85  )
>>>>>>>> >
>>>>>>>> > 1) Your number of iterates increased from 7600 to 9600, but that
>>>>>>>> is a relatively small effect
>>>>>>>> >
>>>>>>>> > 2) MUMPS is just taking a lot longer to do forward/backward
>>>>>>>> solve. You might try emailing
>>>>>>>> > the list for them. However, I would bet that your system has
>>>>>>>> enough bandwidth for 2 procs
>>>>>>>> > and not much more.
>>>>>>>> >
>>>>>>>> >   Thanks,
>>>>>>>> >
>>>>>>>> >      Matt
>>>>>>>> >
>>>>>>>> > Venkatesh
>>>>>>>> >
>>>>>>>> > On Sun, May 17, 2015 at 6:13 PM, Matthew Knepley <
>>>>>>>> knepley at gmail.com> wrote:
>>>>>>>> > On Sun, May 17, 2015 at 1:38 AM, venkatesh g <
>>>>>>>> venkateshgk.j at gmail.com> wrote:
>>>>>>>> > Hi, Thanks for the information. I now increased the workspace by
>>>>>>>> adding '-mat_mumps_icntl_14 100'
>>>>>>>> >
>>>>>>>> > It works. However, the problem is, if I submit in 1 core I get
>>>>>>>> the answer in 200 secs, but with 4 cores and '-mat_mumps_icntl_14 100' it
>>>>>>>> takes 3500secs.
>>>>>>>> >
>>>>>>>> > Send the output of -log_summary for all performance queries.
>>>>>>>> Otherwise we are just guessing.
>>>>>>>> >
>>>>>>>> >     Matt
>>>>>>>> >
>>>>>>>> > My command line is: 'mpiexec -np 4 ./ex7 -f1 a2 -f2 b2 -eps_nev 1
>>>>>>>> -st_type sinvert -eps_max_it 5000 -st_ksp_type preonly -st_pc_type lu
>>>>>>>> -st_pc_factor_mat_solver_package mumps -mat_mumps_icntl_14 100'
>>>>>>>> >
>>>>>>>> > Kindly let me know.
>>>>>>>> >
>>>>>>>> > Venkatesh
>>>>>>>> >
>>>>>>>> >
>>>>>>>> >
>>>>>>>> > On Sat, May 16, 2015 at 7:10 PM, David Knezevic <
>>>>>>>> david.knezevic at akselos.com> wrote:
>>>>>>>> > On Sat, May 16, 2015 at 8:08 AM, venkatesh g <
>>>>>>>> venkateshgk.j at gmail.com> wrote:
>>>>>>>> > Hi,
>>>>>>>> > I am trying to solving AX=lambda BX eigenvalue problem.
>>>>>>>> >
>>>>>>>> > A and B are of sizes 3600x3600
>>>>>>>> >
>>>>>>>> > I run with this command :
>>>>>>>> >
>>>>>>>> > 'mpiexec -np 4 ./ex7 -f1 a2 -f2 b2 -eps_nev 1 -st_type sinvert
>>>>>>>> -eps_max_it 5000 -st_ksp_type preonly -st_pc_type lu
>>>>>>>> -st_pc_factor_mat_solver_package mumps'
>>>>>>>> >
>>>>>>>> > I get this error: (I get result only when I give 1 or 2
>>>>>>>> processors)
>>>>>>>> > Reading COMPLEX matrices from binary files...
>>>>>>>> > [0]PETSC ERROR: --------------------- Error Message
>>>>>>>> ------------------------------------
>>>>>>>> > [0]PETSC ERROR: Error in external library!
>>>>>>>> > [0]PETSC ERROR: Error reported by MUMPS in numerical
>>>>>>>> factorization phase: INFO(1)=-9, INFO(2)=2024
>>>>>>>> >
>>>>>>>> >
>>>>>>>> > The MUMPS error types are described in Chapter 7 of the MUMPS
>>>>>>>> manual. In this case you have INFO(1)=-9, which is explained in the manual
>>>>>>>> as:
>>>>>>>> >
>>>>>>>> > "–9 Main internal real/complex workarray S too small. If INFO(2)
>>>>>>>> is positive, then the number of entries that are missing in S at the moment
>>>>>>>> when the error is raised is available in INFO(2). If INFO(2) is negative,
>>>>>>>> then its absolute value should be multiplied by 1 million. If an error –9
>>>>>>>> occurs, the user should increase the value of ICNTL(14) before calling the
>>>>>>>> factorization (JOB=2) again, except if ICNTL(23) is provided, in which case
>>>>>>>> ICNTL(23) should be increased."
>>>>>>>> >
>>>>>>>> > This says that you should use ICTNL(14) to increase the working
>>>>>>>> space size:
>>>>>>>> >
>>>>>>>> > "ICNTL(14) is accessed by the host both during the analysis and
>>>>>>>> the factorization phases. It corresponds to the percentage increase in the
>>>>>>>> estimated working space. When significant extra fill-in is caused by
>>>>>>>> numerical pivoting, increasing ICNTL(14) may help. Except in special cases,
>>>>>>>> the default value is 20 (which corresponds to a 20 % increase)."
>>>>>>>> >
>>>>>>>> > So, for example, you can avoid this error via the following
>>>>>>>> command line argument to PETSc: "-mat_mumps_icntl_14 30", where 30
>>>>>>>> indicates that we allow a 30% increase in the workspace instead of the
>>>>>>>> default 20%.
>>>>>>>> >
>>>>>>>> > David
>>>>>>>> >
>>>>>>>> >
>>>>>>>> >
>>>>>>>> >
>>>>>>>> >
>>>>>>>> >
>>>>>>>> > --
>>>>>>>> > What most experimenters take for granted before they begin their
>>>>>>>> experiments is infinitely more interesting than any results to which their
>>>>>>>> experiments lead.
>>>>>>>> > -- Norbert Wiener
>>>>>>>> >
>>>>>>>> >
>>>>>>>> >
>>>>>>>> >
>>>>>>>> > --
>>>>>>>> > What most experimenters take for granted before they begin their
>>>>>>>> experiments is infinitely more interesting than any results to which their
>>>>>>>> experiments lead.
>>>>>>>> > -- Norbert Wiener
>>>>>>>> >
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> What most experimenters take for granted before they begin their
>>>>>> experiments is infinitely more interesting than any results to which their
>>>>>> experiments lead.
>>>>>> -- Norbert Wiener
>>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> What most experimenters take for granted before they begin their
>>>> experiments is infinitely more interesting than any results to which their
>>>> experiments lead.
>>>> -- Norbert Wiener
>>>>
>>>
>>>
>>
>>
>> --
>> What most experimenters take for granted before they begin their
>> experiments is infinitely more interesting than any results to which their
>> experiments lead.
>> -- Norbert Wiener
>>
>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150523/afcc901f/attachment-0001.html>

More information about the petsc-users mailing list