[petsc-users] Why use MATMPIBAIJ?

[Matthew Knepley]

On Fri, Jan 22, 2016 at 3:47 PM, Hom Nath Gharti <hng.email at gmail.com>
wrote:

> Hi Matt,
>
> SPECFEM currently has only an explicit time scheme and does not have
> full gravity implemented. I am adding implicit time scheme and full
> gravity so that it can be used for interesting quasistatic problems
> such as glacial rebound, post seismic relaxation etc. I am using Petsc
> as a linear solver which I would like to see GPU implemented.
>

Why? It really does not make sense for those operations.

It is an unfortunate fact, but the usefulness of GPUs has been oversold.
You can certainly
get some mileage out of a SpMV on the GPU, but there the maximum win is
maybe 2x or
less for a nice CPU, and then you have to account for transfer time and
other latencies.
Unless you have a really compelling case, I would not waste your time.

To come to this opinion, I used years of my own time looking at GPUs.

  Thanks,

     Matt


> Thanks,
> Hom Nath
>
> On Fri, Jan 22, 2016 at 4:33 PM, Matthew Knepley <knepley at gmail.com>
> wrote:
> > On Fri, Jan 22, 2016 at 12:17 PM, Hom Nath Gharti <hng.email at gmail.com>
> > wrote:
> >>
> >> Thanks Matt for great suggestion. One last question, do you know
> >> whether the GPU capability of current PETSC version is matured enough
> >> to try for my problem?
> >
> >
> > The only thing that would really make sense to do on the GPU is the SEM
> > integration, which
> > would not be part of PETSc. This is what SPECFEM has optimized.
> >
> >   Thanks,
> >
> >     Matt
> >
> >>
> >> Thanks again for your help.
> >> Hom Nath
> >>
> >> On Fri, Jan 22, 2016 at 1:07 PM, Matthew Knepley <knepley at gmail.com>
> >> wrote:
> >> > On Fri, Jan 22, 2016 at 11:47 AM, Hom Nath Gharti <
> hng.email at gmail.com>
> >> > wrote:
> >> >>
> >> >> Thanks a lot.
> >> >>
> >> >> With AMG it did not converge within the iteration limit of 3000.
> >> >>
> >> >> In solid: elastic wave equation with added gravity term \rho
> \nabla\phi
> >> >> In fluid: acoustic wave equation with added gravity term \rho
> >> >> \nabla\phi
> >> >> Both solid and fluid: Poisson's equation for gravity
> >> >> Outer space: Laplace's equation for gravity
> >> >>
> >> >> We combine so called mapped infinite element with spectral-element
> >> >> method (higher order FEM that uses nodal quadrature) and solve in
> >> >> frequency domain.
> >> >
> >> >
> >> > 1) The Poisson and Laplace equation should be using MG, however you
> are
> >> > using SEM, so
> >> >     you would need to use a low order PC for the high order problem,
> >> > also
> >> > called p-MG (Paul Fischer), see
> >> >
> >> >       http://epubs.siam.org/doi/abs/10.1137/110834512
> >> >
> >> > 2) The acoustic wave equation is Helmholtz to us, and that needs
> special
> >> > MG
> >> > tweaks that
> >> >      are still research material so I can understand using ASM.
> >> >
> >> > 3) Same thing for the elastic wave equations. Some people say they
> have
> >> > this
> >> > solved using
> >> >     hierarchical matrix methods, something like
> >> >
> >> >       http://portal.nersc.gov/project/sparse/strumpack/
> >> >
> >> >     However, I think the jury is still out.
> >> >
> >> > If you can do 100 iterations of plain vanilla solvers, that seems
> like a
> >> > win
> >> > right now. You might improve
> >> > the time using FS, but I am not sure about the iterations on the
> smaller
> >> > problem.
> >> >
> >> >   Thanks,
> >> >
> >> >     Matt
> >> >
> >> >>
> >> >> Hom Nath
> >> >>
> >> >> On Fri, Jan 22, 2016 at 12:16 PM, Matthew Knepley <knepley at gmail.com
> >
> >> >> wrote:
> >> >> > On Fri, Jan 22, 2016 at 11:10 AM, Hom Nath Gharti
> >> >> > <hng.email at gmail.com>
> >> >> > wrote:
> >> >> >>
> >> >> >> Thanks Matt.
> >> >> >>
> >> >> >> Attached detailed info on ksp of a much smaller test. This is a
> >> >> >> multiphysics problem.
> >> >> >
> >> >> >
> >> >> > You are using FGMRES/ASM(ILU0). From your description below, this
> >> >> > sounds
> >> >> > like
> >> >> > an elliptic system. I would at least try AMG (-pc_type gamg) to see
> >> >> > how
> >> >> > it
> >> >> > does. Any
> >> >> > other advice would have to be based on seeing the equations.
> >> >> >
> >> >> >   Thanks,
> >> >> >
> >> >> >     Matt
> >> >> >
> >> >> >>
> >> >> >> Hom Nath
> >> >> >>
> >> >> >> On Fri, Jan 22, 2016 at 12:01 PM, Matthew Knepley
> >> >> >> <knepley at gmail.com>
> >> >> >> wrote:
> >> >> >> > On Fri, Jan 22, 2016 at 10:52 AM, Hom Nath Gharti
> >> >> >> > <hng.email at gmail.com>
> >> >> >> > wrote:
> >> >> >> >>
> >> >> >> >> Dear all,
> >> >> >> >>
> >> >> >> >> I take this opportunity to ask for your important suggestion.
> >> >> >> >>
> >> >> >> >> I am solving an elastic-acoustic-gravity equation on the
> planet.
> >> >> >> >> I
> >> >> >> >> have displacement vector (ux,uy,uz) in solid region,
> displacement
> >> >> >> >> potential (\xi) and pressure (p) in fluid region, and
> >> >> >> >> gravitational
> >> >> >> >> potential (\phi) in all of space. All these variables are
> >> >> >> >> coupled.
> >> >> >> >>
> >> >> >> >> Currently, I am using MATMPIAIJ and form a single global
> matrix.
> >> >> >> >> Does
> >> >> >> >> using a MATMPIBIJ or MATNEST improve the convergence/efficiency
> >> >> >> >> in
> >> >> >> >> this case? For your information, total degrees of freedoms are
> >> >> >> >> about
> >> >> >> >> a
> >> >> >> >> billion.
> >> >> >> >
> >> >> >> >
> >> >> >> > 1) For any solver question, we need to see the output of
> >> >> >> > -ksp_view,
> >> >> >> > and
> >> >> >> > we
> >> >> >> > would also like
> >> >> >> >
> >> >> >> >   -ksp_monitor_true_residual -ksp_converged_reason
> >> >> >> >
> >> >> >> > 2) MATNEST does not affect convergence, and MATMPIBAIJ only in
> the
> >> >> >> > blocksize
> >> >> >> > which you
> >> >> >> >     could set without that format
> >> >> >> >
> >> >> >> > 3) However, you might see benefit from using something like
> >> >> >> > PCFIELDSPLIT
> >> >> >> > if
> >> >> >> > you have multiphysics here
> >> >> >> >
> >> >> >> >    Matt
> >> >> >> >
> >> >> >> >>
> >> >> >> >> Any suggestion would be greatly appreciated.
> >> >> >> >>
> >> >> >> >> Thanks,
> >> >> >> >> Hom Nath
> >> >> >> >>
> >> >> >> >> On Fri, Jan 22, 2016 at 10:32 AM, Matthew Knepley
> >> >> >> >> <knepley at gmail.com>
> >> >> >> >> wrote:
> >> >> >> >> > On Fri, Jan 22, 2016 at 9:27 AM, Mark Adams <mfadams at lbl.gov
> >
> >> >> >> >> > wrote:
> >> >> >> >> >>>
> >> >> >> >> >>>
> >> >> >> >> >>>
> >> >> >> >> >>> I said the Hypre setup cost is not scalable,
> >> >> >> >> >>
> >> >> >> >> >>
> >> >> >> >> >> I'd be a little careful here.  Scaling for the matrix triple
> >> >> >> >> >> product
> >> >> >> >> >> is
> >> >> >> >> >> hard and hypre does put effort into scaling. I don't have
> any
> >> >> >> >> >> data
> >> >> >> >> >> however.
> >> >> >> >> >> Do you?
> >> >> >> >> >
> >> >> >> >> >
> >> >> >> >> > I used it for PyLith and saw this. I did not think any AMG
> had
> >> >> >> >> > scalable
> >> >> >> >> > setup time.
> >> >> >> >> >
> >> >> >> >> >    Matt
> >> >> >> >> >
> >> >> >> >> >>>
> >> >> >> >> >>> but it can be amortized over the iterations. You can
> quantify
> >> >> >> >> >>> this
> >> >> >> >> >>> just by looking at the PCSetUp time as your increase the
> >> >> >> >> >>> number
> >> >> >> >> >>> of
> >> >> >> >> >>> processes. I don't think they have a good
> >> >> >> >> >>> model for the memory usage, and if they do, I do not know
> >> >> >> >> >>> what
> >> >> >> >> >>> it
> >> >> >> >> >>> is.
> >> >> >> >> >>> However, generally Hypre takes more
> >> >> >> >> >>> memory than the agglomeration MG like ML or GAMG.
> >> >> >> >> >>>
> >> >> >> >> >>
> >> >> >> >> >> agglomerations methods tend to have lower "grid complexity",
> >> >> >> >> >> that
> >> >> >> >> >> is
> >> >> >> >> >> smaller coarse grids, than classic AMG like in hypre. THis
> is
> >> >> >> >> >> more
> >> >> >> >> >> of a
> >> >> >> >> >> constant complexity and not a scaling issue though.  You can
> >> >> >> >> >> address
> >> >> >> >> >> this
> >> >> >> >> >> with parameters to some extent. But for elasticity, you want
> >> >> >> >> >> to
> >> >> >> >> >> at
> >> >> >> >> >> least
> >> >> >> >> >> try, if not start with, GAMG or ML.
> >> >> >> >> >>
> >> >> >> >> >>>
> >> >> >> >> >>>   Thanks,
> >> >> >> >> >>>
> >> >> >> >> >>>     Matt
> >> >> >> >> >>>
> >> >> >> >> >>>>
> >> >> >> >> >>>>
> >> >> >> >> >>>> Giang
> >> >> >> >> >>>>
> >> >> >> >> >>>> On Mon, Jan 18, 2016 at 5:25 PM, Jed Brown
> >> >> >> >> >>>> <jed at jedbrown.org>
> >> >> >> >> >>>> wrote:
> >> >> >> >> >>>>>
> >> >> >> >> >>>>> Hoang Giang Bui <hgbk2008 at gmail.com> writes:
> >> >> >> >> >>>>>
> >> >> >> >> >>>>> > Why P2/P2 is not for co-located discretization?
> >> >> >> >> >>>>>
> >> >> >> >> >>>>> Matt typed "P2/P2" when me meant "P2/P1".
> >> >> >> >> >>>>
> >> >> >> >> >>>>
> >> >> >> >> >>>
> >> >> >> >> >>>
> >> >> >> >> >>>
> >> >> >> >> >>> --
> >> >> >> >> >>> What most experimenters take for granted before they begin
> >> >> >> >> >>> their
> >> >> >> >> >>> experiments is infinitely more interesting than any results
> >> >> >> >> >>> to
> >> >> >> >> >>> which
> >> >> >> >> >>> their
> >> >> >> >> >>> experiments lead.
> >> >> >> >> >>> -- Norbert Wiener
> >> >> >> >> >>
> >> >> >> >> >>
> >> >> >> >> >
> >> >> >> >> >
> >> >> >> >> >
> >> >> >> >> > --
> >> >> >> >> > What most experimenters take for granted before they begin
> >> >> >> >> > their
> >> >> >> >> > experiments
> >> >> >> >> > is infinitely more interesting than any results to which
> their
> >> >> >> >> > experiments
> >> >> >> >> > lead.
> >> >> >> >> > -- Norbert Wiener
> >> >> >> >
> >> >> >> >
> >> >> >> >
> >> >> >> >
> >> >> >> > --
> >> >> >> > What most experimenters take for granted before they begin their
> >> >> >> > experiments
> >> >> >> > is infinitely more interesting than any results to which their
> >> >> >> > experiments
> >> >> >> > lead.
> >> >> >> > -- Norbert Wiener
> >> >> >
> >> >> >
> >> >> >
> >> >> >
> >> >> > --
> >> >> > What most experimenters take for granted before they begin their
> >> >> > experiments
> >> >> > is infinitely more interesting than any results to which their
> >> >> > experiments
> >> >> > lead.
> >> >> > -- Norbert Wiener
> >> >
> >> >
> >> >
> >> >
> >> > --
> >> > What most experimenters take for granted before they begin their
> >> > experiments
> >> > is infinitely more interesting than any results to which their
> >> > experiments
> >> > lead.
> >> > -- Norbert Wiener
> >
> >
> >
> >
> > --
> > What most experimenters take for granted before they begin their
> experiments
> > is infinitely more interesting than any results to which their
> experiments
> > lead.
> > -- Norbert Wiener
>



-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20160122/fb380622/attachment.html>