[petsc-users] Fieldsplit with redistribute

Sun Apr 23 17:48:43 CDT 2023

   I have added support for PCREDISTRIBUTE to propogate your PCFieldSplitSetIS() down to an inner PCFIELDSPLIT. You can 
access it with

   git fetch
   git checkout barry/2023-04-22/fieldsplit-fields-propogate
  ./configure
   make all check

   Take a look at src/ksp/ksp/tutorials/ex84.c and run with the options at the bottom of the file.

   Please let us know at  https://gitlab.com/petsc/petsc/-/merge_requests/6366 if it works for you or you have any difficulties.

  Barry

> On Apr 20, 2023, at 10:14 AM, Carl-Johan Thore <carl-johan.thore at liu.se> wrote:
> 
> Great, thanks! I’ve attached the code, a makefile, and a 1-page power-point which hopefully explains
> what I’m trying to do on this little toy-problem. There is obviously (?) something I need to add around
> line 327 in the code in order to move the indices to the correct rank.
>  
> Output should be something like this when running:
>  

>  
> Let me know if you need any more info, or if the code is incomprehensible or so
> (it’s long because I’ve copied a lot from redistribute.c)
>  
> Kind regards,
> Carl-Johan
>  
> From: Barry Smith <bsmith at petsc.dev <mailto:bsmith at petsc.dev>> 
> Sent: Thursday, April 20, 2023 3:17 PM
> To: Carl-Johan Thore <carl-johan.thore at liu.se <mailto:carl-johan.thore at liu.se>>
> Subject: Re: [petsc-users] Fieldsplit with redistribute
>  
>  
>   Sure
> 
> 
> On Apr 20, 2023, at 4:09 AM, Carl-Johan Thore <carl-johan.thore at liu.se <mailto:carl-johan.thore at liu.se>> wrote:
>  
> Hi Barry,
>  
> In the conversation below you mentioned that I could send code to you to take a look. I’ve written
> up what I think is a minimally working example for this. It’s almost there in the sense of distributing
> the correct number of indices to the ranks to match the reduced matrix, but it’s the wrong indices.
> Would it be okay if I sent you the code to have look?
>  
> Kind regards,
> Carl-Johan 
>  
> From: Barry Smith <bsmith at petsc.dev <mailto:bsmith at petsc.dev>> 
> Sent: Sunday, April 16, 2023 10:31 PM
> To: Carl-Johan Thore <carl-johan.thore at liu.se <mailto:carl-johan.thore at liu.se>>
> Cc: petsc-users at mcs.anl.gov <mailto:petsc-users at mcs.anl.gov>
> Subject: Re: [petsc-users] Fieldsplit with redistribute
>  
>  
>    The manual page for ISEmbed is incomprehensible to me. Anyways no matter what, you need to know what degrees of freedom are removed by PCDistribute() in order to produce the reduced IS which is why I think you need information only available inside PCSetUp_Redistribute(). (Sorry it is PCSetUp_Redistribute() not PCApply_Redistribute())
>  
>   Barry
>  
> 
> 
> 
> On Apr 16, 2023, at 3:36 PM, Carl-Johan Thore <carl-johan.thore at liu.se <mailto:carl-johan.thore at liu.se>> wrote:
>  
> Thanks for the quick reply Barry!
> I have not tried the version with PCApply_Redistribute that you suggest, but I have a code that does roughly what you describe. It works when running on one rank, but fails on multiple ranks. I suspect the issue is with the use of ISEmbed as, quoting the PETSc-manual, "the resulting IS is sequential, since the index substitution it encodes is purely local" (admittedly I don't fully understand what that means). If you think using ISEmbed is not a good idea, I'll try PCApply_Redistribute()
> From: Barry Smith <bsmith at petsc.dev <mailto:bsmith at petsc.dev>>
> Sent: 16 April 2023 21:11:18
> To: Carl-Johan Thore <carl-johan.thore at liu.se <mailto:carl-johan.thore at liu.se>>
> Cc: petsc-users at mcs.anl.gov <mailto:petsc-users at mcs.anl.gov> <petsc-users at mcs.anl.gov <mailto:petsc-users at mcs.anl.gov>>
> Subject: Re: [petsc-users] Fieldsplit with redistribute 
>  
> 
>    There is no code to do this currently. 
> 
>     I would start by building your IS for each split before the PCRedistribute and then adding to the PCApply_Redistribute() code that "fixes" these IS by "removing" the entries of the IS associated with removed degrees of freedom and then shifting the entries indices of the IS by taking into account the removed indices. But you have probably already been trying this? It does require digging directly into the PCApply_Redistribute() to get the needed information (which degrees of freedom are removed by the redistribute code), plus it requires shifting the MPI rank ownership of the entries of the IS in the same way the MPI rank ownership of the degrees of freedom of the vector are moved.
> 
>    If you have some code that you think should be doing this but doesn't work feel free to send it to us and we may be able to fix it.
> 
>   Barry
> 
> 
> > On Apr 16, 2023, at 2:50 PM, Carl-Johan Thore via petsc-users <petsc-users at mcs.anl.gov <mailto:petsc-users at mcs.anl.gov>> wrote:
> > 
> > Hello,
> > I'm solving a blocksystem
> > [A C;
> > C' D],
> > where D is not zero, using the PCFIELDSPLIT preconditioner and set the split using PetscFieldSplitSetIS. This works very well until I try PCREDISTRIBUTE (which is attractive as I have many locked DOFs). I suspect something goes wrong when constructing the IS for the split (I've tried various things using the IS-routines). Can PETSc do this automatically? Or else, any hints?
> > Kind regards,
> > Carl-Johan
> 
>  

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20230423/5f361438/attachment-0005.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image001.png
Type: image/png
Size: 25214 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20230423/5f361438/attachment-0001.png>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20230423/5f361438/attachment-0006.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: explanation.pptx
Type: application/vnd.openxmlformats-officedocument.presentationml.presentation
Size: 515540 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20230423/5f361438/attachment-0001.pptx>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20230423/5f361438/attachment-0007.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: IStest.cc
Type: application/octet-stream
Size: 15346 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20230423/5f361438/attachment-0002.obj>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20230423/5f361438/attachment-0008.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: makefile
Type: application/octet-stream
Size: 451 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20230423/5f361438/attachment-0003.obj>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20230423/5f361438/attachment-0009.html>