[petsc-users] SuperLU MPI-problem

[Matthew Knepley]

On Mon, Jul 20, 2015 at 11:14 AM, Mahir.Ulker-Kaustell at tyrens.se <
Mahir.Ulker-Kaustell at tyrens.se> wrote:

>  I am roughly guessing that my sparse matrix will have 1000 non-zeros on
> each row. As the matrix I symmetric I divide by half:
>
>
>
> 1000*1000000(rows)*8(bytes)*2(complex numbers)/2 = 8GB
>

1) You have missed the memory to represent the columns, so multiply by 1.5

2) Are you using a symmetric format?

3) This is only the space to represent the matrix, not its factors.

  Matt


>
>
However, it turns out to be much less: I create the matrices in another
> program and the binary file holding my matrix K is only 800MB. M is a
> lumped mass matrix…
>
>
>
> By the way, I tried using MUMPS and have the same problem there.
>
>
>
> Mahir
>
>
>
>
>
> *From:* Matthew Knepley [mailto:knepley at gmail.com]
> *Sent:* den 20 juli 2015 18:05
> *To:* Ülker-Kaustell, Mahir
> *Cc:* Hong; petsc-users
> *Subject:* Re: [petsc-users] SuperLU MPI-problem
>
>
>
> On Mon, Jul 20, 2015 at 10:59 AM, Mahir.Ulker-Kaustell at tyrens.se <
> Mahir.Ulker-Kaustell at tyrens.se> wrote:
>
> Hong:
>
>
>
> Previous experiences with this equation have shown that it is very
> difficult to solve it iteratively. Hence the use of a direct solver.
>
>
>
> The large test problem I am trying to solve has slightly less than 10^6
> degrees of freedom. The matrices are derived from finite elements so they
> are sparse.
>
>
>
> Estimated how? It is very difficult to estimate flll-in.
>
>
>
>   Matt
>
>
>
>  The machine I am working on has 128GB ram. I have estimated the memory
> needed to less than 20GB, so if the solver needs twice or even three times
> as much, it should still work well. Or have I completely misunderstood
> something here?
>
>
>
> Mahir
>
>
>
>
>
>
>
> *From:* Hong [mailto:hzhang at mcs.anl.gov]
> *Sent:* den 20 juli 2015 17:39
> *To:* Ülker-Kaustell, Mahir
> *Cc:* petsc-users
> *Subject:* Re: [petsc-users] SuperLU MPI-problem
>
>
>
> Mahir:
>
> Direct solvers consume large amount of memory. Suggest to try followings:
>
>
>
> 1. A sparse iterative solver if  [-omega^2M + K] is not too
> ill-conditioned. You may test it using the small matrix.
>
>
>
> 2. Incrementally increase your matrix sizes. Try different matrix
> orderings.
>
> Do you get memory crash in the 1st symbolic factorization?
>
> In your case, matrix data structure stays same when omega changes, so you
> only need to do one matrix symbolic factorization and reuse it.
>
>
>
> 3. Use a machine that gives larger memory.
>
>
>
> Hong
>
>
>
> Dear Petsc-Users,
>
>
>
> I am trying to use PETSc to solve a set of linear equations arising from
> Naviers equation (elastodynamics) in the frequency domain.
>
> The frequency dependency of the problem requires that the system
>
>
>
>                              [-omega^2M + K]u = F
>
>
>
> where M and K are constant, square, positive definite matrices (mass and
> stiffness respectively) is solved for each frequency omega of interest.
>
> K is a complex matrix, including material damping.
>
>
>
> I have written a PETSc program which solves this problem for a small (1000
> degrees of freedom) test problem on one or several processors, but it keeps
> crashing when I try it on my full scale (in the order of 10^6 degrees of
> freedom) problem.
>
>
>
> The program crashes at KSPSetUp() and from what I can see in the error
> messages, it appears as if it consumes too much memory.
>
>
>
> I would guess that similar problems have occurred in this mail-list, so I
> am hoping that someone can push  me in the right direction…
>
>
>
> Mahir
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> --
>
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>



-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150720/fe32f09c/attachment.html>