[petsc-dev] Fwd: [PPPL-SCOREC application project] (fwd)

[Hong Zhang]

Barry,

See attached below.  Hong


---------- Forwarded message ----------
From: Hong Zhang <hzhang at mcs.anl.gov>
Date: Wed, Jan 12, 2011 at 9:57 PM
Subject: Re: [PPPL-SCOREC application project] (fwd)
To: fabien delalondre <delalf at scorec.rpi.edu>
Cc: Satish Balay <balay at mcs.anl.gov>, Barry Smith
<bsmith at mcs.anl.gov>, Fan Zhang <zhangf4 at rpi.edu>, Mark Shephard
<shephard at scorec.rpi.edu>, Hong Zhang <HZhang at mcs.anl.gov>


fabien:

For eqn(0.4), do all diagonal blocks B have same size?
If so, you can implement bjacobi+superlu_dist with

mpiexec -n <np> ./your_prog -pc_type bjacobi -pc_bjacobi_blocks <nBblocks>
-sub_pc_type lu -sub_pc_factor_mat_solver_package superlu_dist

When np > nBblocks, subcommunicators are formed by petsc to implement
parallel LU within each block,
otherwise, sequential LU are applied for each block.

Hong

>> At application level, the outer KSP is for solving global equation,
>> e.g., your Eqn0.4, while block jacobian preconditioner, i.e.,
>> diag(B's) from Eqn0.4, is applied. MatGetMultiProcBlock() is used
>> internally by
>> petsc to get preconditioner diag(B's). My understanding is that you
>> want apply superlu_dist
>> on each of block B (2D solution), and use it as a preconditioner for
>> global 3D Eqn0.4.
>
> That would be our first attempt although we are not yet sure what
> solver/precondition methods would best suit the problem we are trying to
> solve.
>
>>
>> For this, you need set your own subcommtype
>> (see "type == PETSC_SUBCOMM_GENERAL" in ex37.c).
>> I can provide an example that apply user's subcommtype to a global KSP
>> solver,
>> if it is helpful.
>
> I think I understand but would appreciate if you could provide such an
> example.
> If I understand correctly the MatGetMultiProcBlock generates a bjacobi
> submatrix from a larger matrix (that is usually the complete matrix spanning
> all processors). That submatrix spans the processors included in the
> provided subCommunicator.
> Thank you.
> Fabien
>
>>
>> Hong
>>
>> >
>> > Fabien
>> >
>> > On Mon, Jan 10, 2011 at 10:51 PM, Hong Zhang <hzhang at mcs.anl.gov> wrote:
>> >>
>> >> Dear fabien:
>> >>
>> >> Yes, petsc-dev now supports block jacobian preconditioner with
>> >> each block distributed into multiple processors as I told you in my
>> >> previous email (cannot believe my
>> >> email was dated 2009!)
>> >>
>> >> For example, you can run petsc-dev/src/ksp/ksp/examples/tutorials/ex2
>> >> with
>> >> mpiexec -n 4 ./ex2 -pc_type bjacobi -pc_bjacobi_blocks 2 -sub_pc_type
>> >> lu -sub_pc_factor_mat_solver_package superlu_dist -ksp_view
>> >> i.e., with np=4, using block jacobi preconditoner with 2 blocks, each
>> >> block running LU using superlu_dist with np=2 processors.
>> >>
>> >> For your application, I guess you may want use your own matrix and
>> >> processor partitioning in stead of petsc default.
>> >> petsc-dev/src/ksp/ksp/examples/tests/ex37.c might be relevant.
>> >> Let us know what you need from here.
>> >>
>> >> I'll let other petsc maintainers to answer your question regarding
>> >> matrix/mesh partitioning.
>> >>
>> >> Hong
>> >>
>> >> >
>> >> > As mentioned in a pretty old email, the SCOREC team is helping Steve
>> >> > Jardin's team at Princeton Plasma Physics Lab in developing M3DC1
>> >> > code
>> >> > to
>> >> > support simulation on plasma. As for now, we can build a partitioned
>> >> > 3D
>> >> > mesh (torus) made of joint 2D planes that are evenly distributed
>> >> > around
>> >> > the toroidal direction.  Each 2D plane can be distributed over one or
>> >> > several processors.
>> >> >
>> >> > The 3D matrix we end up with has a periodic tridiagonal block pattern
>> >> > where each block represents either a single 2D plane (see pages 2-3
>> >> > of
>> >> > the
>> >> > attached document from Steve Jardin for the matrix pattern). As for
>> >> > now,
>> >> > PPPL is testing different combinations to solve (gmres) and
>> >> > precondition
>> >> > (superlu, ilu, ...) such a the system using both available Petsc and
>> >> > Superlu_dist distributions.
>> >> >
>> >> > Since you mentioned in a previous communication that you were
>> >> > starting
>> >> > to
>> >> > develop methods to support solving such a type of system with several
>> >> > processors by plane (several processors by matrix block), we were
>> >> > wondering if that functionality is already available in the latest
>> >> > petsc
>> >> > distribution. In the long run, the scorec team would be interested in
>> >> > interacting more closely with the petsc team on a couple of topics
>> >> > that
>> >> > would potentially include:
>> >> > * Increasing PPPL application resolution performance by selecting a
>> >> > better
>> >> > combination of solver/preconditioner methods (the development of a
>> >> > preconditioner might be something we might want to look at). We would
>> >> > appreciate your feedback and expertise to work toward the best
>> >> > solution
>> >> > in
>> >> > this regards.
>> >> > * Working on understanding how mesh adaptation could be more tightly
>> >> > linked to the solver/preconditioner to reduce the required number of
>> >> > operations to be carried out.
>> >> >
>> >> > On a side note, I was wondering if you guys could give me some
>> >> > information
>> >> > on matrix partitioning. I would like to get the Matrix partitioned in
>> >> > the
>> >> > same way I already have my mesh partitioned (not a petsc mesh). I was
>> >> > wondering if there already exists some API functions that would
>> >> > support
>> >> > such an operation (assuming I can provide the needed information from
>> >> > my
>> >> > mesh partitioning).
>> >> >
>> >> > Best regards,
>> >> >
>> >> > Fabien Delalondre
>> >> >
>> >> >> Fabien,
>> >> >>
>> >> >>> Thank you for your reply. We are currently working on the design of
>> >> >>> an
>> >> >>> implementation that will support PPPL activities in parallel for
>> >> >>> multiple
>> >> >>> planes. Obviously, we will need to support multiple parts by plane
>> >> >>> (running a block on multiple processors) in the near future
>> >> >>> (probably
>> >> >>> some
>> >> >>> few months from now).
>> >> >>> We will then first begin with the one part (one block) by processor
>> >> >>> that
>> >> >>> will be considered as a specific case of the more general approach
>> >> >>> that
>> >> >>> will support multiple parts by processors (running a block on
>> >> >>> multiple
>> >> >>> processors). Having this running within a few months (2-3) will
>> >> >>> consequently be advantageous. Do you think this would be a
>> >> >>> reasonable
>> >> >>> time
>> >> >>> frame for you ?
>> >> >>
>> >> >> Let us know after you get the 1st phase of your work done.
>> >> >> If we still have not accomplished our task then, we'll put it
>> >> >> at the top of our development and get it done in few days.
>> >> >>
>> >> >> Sorry for late reply,
>> >> >>
>> >> >> Hong
>> >> >>>
>> >> >>>>
>> >> >>>> Fabien,
>> >> >>>>
>> >> >>>> Your matrix has pattern of periodic tridiagonal blocks,
>> >> >>>> each block represents a single 2D plane previously
>> >> >>>> computed using superlu_dist in parallel.
>> >> >>>>
>> >> >>>> Our understanding is that you want use
>> >> >>>> the diagonal block matrice as a preconditoner.
>> >> >>>> We support bjacobi preconditioner, this means,
>> >> >>>> if you use a sigle process for each block,
>> >> >>>> then current petsc can easily invok pc=jacobi
>> >> >>>> and use superlu within the block.
>> >> >>>> As block size becomes too large for a single processor,
>> >> >>>> then you want run each block on several processes,
>> >> >>>> for which we are still developing.
>> >> >>>> We can accelerate this development if you've reached
>> >> >>>> this stage of computation.
>> >> >>>> Let us know your need.
>> >> >>>>
>> >> >>>> Sorry for late reply. We are moving to a new building ...
>> >> >>>>
>> >> >>>> Hong
>> >> >>>>
>> >> >>>>> ---------- Forwarded message ----------
>> >> >>>>> Date: Wed, 9 Sep 2009 23:36:48 -0400 (EDT)
>> >> >>>>> From: delalf at scorec.rpi.edu
>> >> >>>>> To: hzhang at mcs.anl.gov
>> >> >>>>> Cc: shephard at scorec.rpi.edu
>> >> >>>>> Subject: [PPPL-SCOREC application project]
>> >> >>>>>
>> >> >>>>> Dear Hong,
>> >> >>>>>
>> >> >>>>> My name is Fabien Delalondre. I am currently working at SCOREC
>> >> >>>>> with
>> >> >>>>> Mark
>> >> >>>>> Shephard on the development of parallel adaptive multiscale
>> >> >>>>> simulation.
>> >> >>>>>
>> >> >>>>> I am contacting you regarding one of the projects we are working
>> >> >>>>> on
>> >> >>>>> that
>> >> >>>>> involves the Princeton Plasma Physics lab research team. Mark let
>> >> >>>>> me
>> >> >>>>> know
>> >> >>>>> that you already discussed some aspects of that project. I
>> >> >>>>> attached
>> >> >>>>> some
>> >> >>>>> documents to remind you the problematic we will soon be facing:
>> >> >>>>> - PPPL_Jardin-Solvers.pdf: Description of MHD equations the PPPL
>> >> >>>>> research
>> >> >>>>> team is solving (2D)
>> >> >>>>> - 3DExtension.pdf: Description of the 3D extension of MHD
>> >> >>>>> equations.
>> >> >>>>> - PPPL – SCOREC.pdf: Brief description of the matrix pattern
>> >> >>>>> that
>> >> >>>>> will
>> >> >>>>> be
>> >> >>>>> obtained using the 3D extension of MHD equations (banded matrix
>> >> >>>>> with
>> >> >>>>> some
>> >> >>>>> extra diagonal coupling terms).
>> >> >>>>>
>> >> >>>>> Mark let me know that the petsc team might already have developed
>> >> >>>>> something for supporting the resolution of algebraic system that
>> >> >>>>> includes
>> >> >>>>> such matrix pattern. Could you please give me some information
>> >> >>>>> regarding
>> >> >>>>> this.
>> >> >>>>>
>> >> >>>>> Regards,
>> >> >>>>>
>> >> >>>>> Fabien Delalondre<3DExtension.pdf><PPPL      –
>> >> >>>>> SCOREC.pdf><PPPL_Jardin-Solvers.pdf>
>> >> >>>
>> >> >>>
>> >> >>>
>> >> >
>> >
>> >
>> >
>> > --
>> > Fabien Delalondre, PhD.
>> > Senior Research Associate, Scientific Computation Research Center
>> > (SCOREC).
>> > Rensselaer Polytechnic Institute (RPI), Troy, NY.
>> > Email: delalf at scorec.rpi.edu, Phone: (518)-276-8045
>> >
>
>
>
> --
> Fabien Delalondre, PhD.
> Senior Research Associate, Scientific Computation Research Center (SCOREC).
> Rensselaer Polytechnic Institute (RPI), Troy, NY.
> Email: delalf at scorec.rpi.edu, Phone: (518)-276-8045
>