[petsc-users] Why use MATMPIBAIJ?

[Hom Nath Gharti]

Thanks Matt for great suggestion. One last question, do you know
whether the GPU capability of current PETSC version is matured enough
to try for my problem?

Thanks again for your help.
Hom Nath

On Fri, Jan 22, 2016 at 1:07 PM, Matthew Knepley <knepley at gmail.com> wrote:
> On Fri, Jan 22, 2016 at 11:47 AM, Hom Nath Gharti <hng.email at gmail.com>
> wrote:
>>
>> Thanks a lot.
>>
>> With AMG it did not converge within the iteration limit of 3000.
>>
>> In solid: elastic wave equation with added gravity term \rho \nabla\phi
>> In fluid: acoustic wave equation with added gravity term \rho \nabla\phi
>> Both solid and fluid: Poisson's equation for gravity
>> Outer space: Laplace's equation for gravity
>>
>> We combine so called mapped infinite element with spectral-element
>> method (higher order FEM that uses nodal quadrature) and solve in
>> frequency domain.
>
>
> 1) The Poisson and Laplace equation should be using MG, however you are
> using SEM, so
>     you would need to use a low order PC for the high order problem, also
> called p-MG (Paul Fischer), see
>
>       http://epubs.siam.org/doi/abs/10.1137/110834512
>
> 2) The acoustic wave equation is Helmholtz to us, and that needs special MG
> tweaks that
>      are still research material so I can understand using ASM.
>
> 3) Same thing for the elastic wave equations. Some people say they have this
> solved using
>     hierarchical matrix methods, something like
>
>       http://portal.nersc.gov/project/sparse/strumpack/
>
>     However, I think the jury is still out.
>
> If you can do 100 iterations of plain vanilla solvers, that seems like a win
> right now. You might improve
> the time using FS, but I am not sure about the iterations on the smaller
> problem.
>
>   Thanks,
>
>     Matt
>
>>
>> Hom Nath
>>
>> On Fri, Jan 22, 2016 at 12:16 PM, Matthew Knepley <knepley at gmail.com>
>> wrote:
>> > On Fri, Jan 22, 2016 at 11:10 AM, Hom Nath Gharti <hng.email at gmail.com>
>> > wrote:
>> >>
>> >> Thanks Matt.
>> >>
>> >> Attached detailed info on ksp of a much smaller test. This is a
>> >> multiphysics problem.
>> >
>> >
>> > You are using FGMRES/ASM(ILU0). From your description below, this sounds
>> > like
>> > an elliptic system. I would at least try AMG (-pc_type gamg) to see how
>> > it
>> > does. Any
>> > other advice would have to be based on seeing the equations.
>> >
>> >   Thanks,
>> >
>> >     Matt
>> >
>> >>
>> >> Hom Nath
>> >>
>> >> On Fri, Jan 22, 2016 at 12:01 PM, Matthew Knepley <knepley at gmail.com>
>> >> wrote:
>> >> > On Fri, Jan 22, 2016 at 10:52 AM, Hom Nath Gharti
>> >> > <hng.email at gmail.com>
>> >> > wrote:
>> >> >>
>> >> >> Dear all,
>> >> >>
>> >> >> I take this opportunity to ask for your important suggestion.
>> >> >>
>> >> >> I am solving an elastic-acoustic-gravity equation on the planet. I
>> >> >> have displacement vector (ux,uy,uz) in solid region, displacement
>> >> >> potential (\xi) and pressure (p) in fluid region, and gravitational
>> >> >> potential (\phi) in all of space. All these variables are coupled.
>> >> >>
>> >> >> Currently, I am using MATMPIAIJ and form a single global matrix.
>> >> >> Does
>> >> >> using a MATMPIBIJ or MATNEST improve the convergence/efficiency in
>> >> >> this case? For your information, total degrees of freedoms are about
>> >> >> a
>> >> >> billion.
>> >> >
>> >> >
>> >> > 1) For any solver question, we need to see the output of -ksp_view,
>> >> > and
>> >> > we
>> >> > would also like
>> >> >
>> >> >   -ksp_monitor_true_residual -ksp_converged_reason
>> >> >
>> >> > 2) MATNEST does not affect convergence, and MATMPIBAIJ only in the
>> >> > blocksize
>> >> > which you
>> >> >     could set without that format
>> >> >
>> >> > 3) However, you might see benefit from using something like
>> >> > PCFIELDSPLIT
>> >> > if
>> >> > you have multiphysics here
>> >> >
>> >> >    Matt
>> >> >
>> >> >>
>> >> >> Any suggestion would be greatly appreciated.
>> >> >>
>> >> >> Thanks,
>> >> >> Hom Nath
>> >> >>
>> >> >> On Fri, Jan 22, 2016 at 10:32 AM, Matthew Knepley
>> >> >> <knepley at gmail.com>
>> >> >> wrote:
>> >> >> > On Fri, Jan 22, 2016 at 9:27 AM, Mark Adams <mfadams at lbl.gov>
>> >> >> > wrote:
>> >> >> >>>
>> >> >> >>>
>> >> >> >>>
>> >> >> >>> I said the Hypre setup cost is not scalable,
>> >> >> >>
>> >> >> >>
>> >> >> >> I'd be a little careful here.  Scaling for the matrix triple
>> >> >> >> product
>> >> >> >> is
>> >> >> >> hard and hypre does put effort into scaling. I don't have any
>> >> >> >> data
>> >> >> >> however.
>> >> >> >> Do you?
>> >> >> >
>> >> >> >
>> >> >> > I used it for PyLith and saw this. I did not think any AMG had
>> >> >> > scalable
>> >> >> > setup time.
>> >> >> >
>> >> >> >    Matt
>> >> >> >
>> >> >> >>>
>> >> >> >>> but it can be amortized over the iterations. You can quantify
>> >> >> >>> this
>> >> >> >>> just by looking at the PCSetUp time as your increase the number
>> >> >> >>> of
>> >> >> >>> processes. I don't think they have a good
>> >> >> >>> model for the memory usage, and if they do, I do not know what
>> >> >> >>> it
>> >> >> >>> is.
>> >> >> >>> However, generally Hypre takes more
>> >> >> >>> memory than the agglomeration MG like ML or GAMG.
>> >> >> >>>
>> >> >> >>
>> >> >> >> agglomerations methods tend to have lower "grid complexity", that
>> >> >> >> is
>> >> >> >> smaller coarse grids, than classic AMG like in hypre. THis is
>> >> >> >> more
>> >> >> >> of a
>> >> >> >> constant complexity and not a scaling issue though.  You can
>> >> >> >> address
>> >> >> >> this
>> >> >> >> with parameters to some extent. But for elasticity, you want to
>> >> >> >> at
>> >> >> >> least
>> >> >> >> try, if not start with, GAMG or ML.
>> >> >> >>
>> >> >> >>>
>> >> >> >>>   Thanks,
>> >> >> >>>
>> >> >> >>>     Matt
>> >> >> >>>
>> >> >> >>>>
>> >> >> >>>>
>> >> >> >>>> Giang
>> >> >> >>>>
>> >> >> >>>> On Mon, Jan 18, 2016 at 5:25 PM, Jed Brown <jed at jedbrown.org>
>> >> >> >>>> wrote:
>> >> >> >>>>>
>> >> >> >>>>> Hoang Giang Bui <hgbk2008 at gmail.com> writes:
>> >> >> >>>>>
>> >> >> >>>>> > Why P2/P2 is not for co-located discretization?
>> >> >> >>>>>
>> >> >> >>>>> Matt typed "P2/P2" when me meant "P2/P1".
>> >> >> >>>>
>> >> >> >>>>
>> >> >> >>>
>> >> >> >>>
>> >> >> >>>
>> >> >> >>> --
>> >> >> >>> What most experimenters take for granted before they begin their
>> >> >> >>> experiments is infinitely more interesting than any results to
>> >> >> >>> which
>> >> >> >>> their
>> >> >> >>> experiments lead.
>> >> >> >>> -- Norbert Wiener
>> >> >> >>
>> >> >> >>
>> >> >> >
>> >> >> >
>> >> >> >
>> >> >> > --
>> >> >> > What most experimenters take for granted before they begin their
>> >> >> > experiments
>> >> >> > is infinitely more interesting than any results to which their
>> >> >> > experiments
>> >> >> > lead.
>> >> >> > -- Norbert Wiener
>> >> >
>> >> >
>> >> >
>> >> >
>> >> > --
>> >> > What most experimenters take for granted before they begin their
>> >> > experiments
>> >> > is infinitely more interesting than any results to which their
>> >> > experiments
>> >> > lead.
>> >> > -- Norbert Wiener
>> >
>> >
>> >
>> >
>> > --
>> > What most experimenters take for granted before they begin their
>> > experiments
>> > is infinitely more interesting than any results to which their
>> > experiments
>> > lead.
>> > -- Norbert Wiener
>
>
>
>
> --
> What most experimenters take for granted before they begin their experiments
> is infinitely more interesting than any results to which their experiments
> lead.
> -- Norbert Wiener