[petsc-users] Distributing already assembled stiffness matrix

[Mark Adams]

On Fri, Oct 20, 2017 at 6:42 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:

>
> > On Oct 18, 2017, at 4:14 AM, Jaganathan, Srikrishna <
> srikrishna.jaganathan at fau.de> wrote:
> >
> > Hello,
> >
> >
> > I have been trying to distribute a already existing stiffness matrix in
> my FEM code to petsc parallel matrix object , but I am unable to find any
> documentation regarding it.
>
>    I really really don't recommend doing this. If you code is sequential
> and you want to speed up the linear solves then use some linear solver
> library that uses OpenMP for parallelism and be done with it. Mix
> sequential finite element assembly with MPI parallel solvers is just not
> worth going within 100 meters of.
>

I agree with Barry. And, now that it has come up, let me just drop another
approach. Run your serial code on every processor and have PETSC ignore off
processor entries. This saves time because you have no communication at the
end of the assembly but use more "power" because everyone will be busy
doing (mostly useless) work the whole time. Now you can start trimming the
useless work away: first mark elements that do not touch local vertices and
skip these, next make a list of the active elements and iterate over those,
etc.

At some point you want to partition your grid intelligently. It is not
clear to me where you should do this, but there is code in GAMG that takes
an existing distributed matrix, partitions it in parallel, does a scatter
gather to reconstitute the matrix with the good partitioning. This is all
very scalable. This code is a bit involved and you would have to adapt it.

Next you will want to distribute your metadata ...



>
>    Barry
>
>
>
> > It was quite straightforward to create a sequential petsc matrix object
> and everything was working as intended.I have read some of the user
> comments in the mailing lists regarding similar situation and most of the
> times the solution suggested is to create stiffness matrix from the the
> mesh in distributed format. Since its a little difficult in my case to pass
> the mesh data in the code , is there anyway to distribute already existing
> stiffness matrix ?
> >
> > Thanks and Regards
> >
> > Srikrishna Jaganathan
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20171020/2f0abf3b/attachment.html>