<html><head><meta http-equiv="content-type" content="text/html; charset=utf-8"></head><body style="overflow-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;"><div><br></div> What happens if you pass in your IS using directly PCFIELDSPLIT and not using PCREDISTRIBUTE? <br><div><br><blockquote type="cite"><div>On Apr 26, 2023, at 2:27 AM, Carl-Johan Thore <carl-johan.thore@liu.se> wrote:</div><br class="Apple-interchange-newline"><div><meta charset="UTF-8"><div class="WordSection1" style="page: WordSection1; caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 18px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;"><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;">Hi again,<o:p></o:p></div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"><o:p> </o:p></div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;">I now think I got my IS in order (it’s just one IS because unlike in your ex84.c I don’t provide the complement of the IS explicitly but let fieldsplit compute it, but ex84.c works fine if I do the same there).<o:p></o:p></div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;">As before, my code works with pcredistribute and pcfieldsplit on 1 core. I then try<o:p></o:p></div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;">with 2 cores. First I check the IS in Matlab, and it looks fine as far as I can tell, with identical content<o:p></o:p></div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;">as in the 1-core case, which it should?. Then I try running the code, but it fails with (the first two lines are mine)<o:p></o:p></div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"><o:p> </o:p></div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;">“<o:p></o:p></div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"><span lang="SV">[1]ISdata: min= 6301, max=10639, freeudofs= 3244. min= 6293, max=10639, freedofs= 4347: row 2629<o:p></o:p></span></div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"><span lang="SV">[0]ISdata: min= 139, max= 6292, freeudofs= 4459. min= 0, max= 6292, freedofs= 6293: row 2629<o:p></o:p></span></div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;">[0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------<o:p></o:p></div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;">[0]PETSC ERROR: Argument out of range<o:p></o:p></div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;">[0]PETSC ERROR: Index 3748's value 5320 is larger than maximum given 5320<o:p></o:p></div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;">[0]PETSC ERROR: See<span class="Apple-converted-space"> </span><a href="https://petsc.org/release/faq/" style="color: blue; text-decoration: underline;">https://petsc.org/release/faq/</a><span class="Apple-converted-space"> </span>for trouble shooting.<o:p></o:p></div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;">[0]PETSC ERROR: Petsc Development GIT revision: v3.19.0-236-gee39b84cc03 GIT Date: 2023-04-23 18:43:23 -0400<o:p></o:p></div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;">[0]PETSC ERROR: topopt on a arch-linux-c-debug named win01705 by carlthore Wed Apr 26 08:01:49 2023<o:p></o:p></div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;">[0]PETSC ERROR: Configure options -f --with-cuda --with-cusp --download-scalapack --download-hdf5 --download-zlib --download-mumps --download-parmetis --download-metis --download-ptscotch --download-hypre --download-spai<o:p></o:p></div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;">[0]PETSC ERROR: #1 ISComplement() at /mnt/c/mathware/petsc/src/vec/is/is/utils/iscoloring.c:804<o:p></o:p></div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;">[0]PETSC ERROR: #2 PCFieldSplitSetDefaults() at /mnt/c/mathware/petsc/src/ksp/pc/impls/fieldsplit/fieldsplit.c:544<o:p></o:p></div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;">[0]PETSC ERROR: #3 PCSetUp_FieldSplit() at /mnt/c/mathware/petsc/src/ksp/pc/impls/fieldsplit/fieldsplit.c:587<o:p></o:p></div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;">[0]PETSC ERROR: #4 PCSetUp() at /mnt/c/mathware/petsc/src/ksp/pc/interface/precon.c:994<o:p></o:p></div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;">[0]PETSC ERROR: #5 KSPSetUp() at /mnt/c/mathware/petsc/src/ksp/ksp/interface/itfunc.c:406<o:p></o:p></div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;">[0]PETSC ERROR: #6 PCSetUp_Redistribute() at /mnt/c/mathware/petsc/src/ksp/pc/impls/redistribute/redistribute.c:327<o:p></o:p></div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;">[0]PETSC ERROR: #7 PCSetUp() at /mnt/c/mathware/petsc/src/ksp/pc/interface/precon.c:994<o:p></o:p></div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;">[0]PETSC ERROR: #8 KSPSetUp() at /mnt/c/mathware/petsc/src/ksp/ksp/interface/itfunc.c:406<o:p></o:p></div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;">[0]PETSC ERROR: #9 SetUpSolver() at /mnt/c/TOPet/fts_topopt_in_petsc-master/<a href="http://mixedstokes.cc:2650/" style="color: blue; text-decoration: underline;">MixedStokes.cc:2650</a><o:p></o:p></div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;">“<o:p></o:p></div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"><o:p> </o:p></div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;">This is all the errors I see, I’m not sure why the error message is not written out in full. I guess the error is on<o:p></o:p></div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;">my side, with the IS still not being constructed correctly, but what do you think?<o:p></o:p></div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"><o:p> </o:p></div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;">Kind regards,<o:p></o:p></div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;">Carl-Johan<o:p></o:p></div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"><o:p> </o:p></div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"><o:p> </o:p></div><div><div style="border-width: 1pt medium medium; border-style: solid none none; border-color: rgb(225, 225, 225) currentcolor currentcolor; border-image: none; padding: 3pt 0cm 0cm;"><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"><b>From:</b><span class="Apple-converted-space"> </span>Barry Smith <<a href="mailto:bsmith@petsc.dev" style="color: blue; text-decoration: underline;">bsmith@petsc.dev</a>><span class="Apple-converted-space"> </span><br><b>Sent:</b><span class="Apple-converted-space"> </span>Monday, April 24, 2023 8:27 PM<br><b>To:</b><span class="Apple-converted-space"> </span>Carl-Johan Thore <<a href="mailto:carl-johan.thore@liu.se" style="color: blue; text-decoration: underline;">carl-johan.thore@liu.se</a>><br><b>Cc:</b><span class="Apple-converted-space"> </span>PETSc <<a href="mailto:petsc-users@mcs.anl.gov" style="color: blue; text-decoration: underline;">petsc-users@mcs.anl.gov</a>><br><b>Subject:</b><span class="Apple-converted-space"> </span>Re: [petsc-users] Fieldsplit with redistribute<o:p></o:p></div></div></div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"><o:p> </o:p></div><div><div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"><o:p> </o:p></div></div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"> PCREDISTRIBUTE looks like it is not yet GPU friendly. This needs to be fixed, but it should be a separate fix and MR from my current one. <o:p></o:p></div><div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"><o:p> </o:p></div></div><div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"> Please just check if the PCREDISTRIBUTE followed by PCFIELDSPLIT just works on CPUs for your code.<o:p></o:p></div></div><div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"><o:p> </o:p></div></div><div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"> Barry<o:p></o:p></div></div><div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"><o:p> </o:p></div><div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"><br><br><o:p></o:p></div><blockquote style="margin-top: 5pt; margin-bottom: 5pt;"><div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;">On Apr 24, 2023, at 2:23 PM, Carl-Johan Thore <<a href="mailto:carl-johan.thore@liu.se" style="color: blue; text-decoration: underline;">carl-johan.thore@liu.se</a>> wrote:<o:p></o:p></div></div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"><o:p> </o:p></div><div><div><div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;">I wasn’t sure if I was going to bother you again with this, but since it looks like you plan to merge this with the<o:p></o:p></div></div><div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;">main branch (?) I thought it might be interesting to know that I’ve tried this with my code running with CUDA but got the<o:p></o:p></div></div><div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;">attached error. I suspect it’s related to the RHS red->b but I’m not sure. If I switch of redistribute my code runs fine<o:p></o:p></div></div><div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;">with CUDA.<o:p></o:p></div></div><div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"> <o:p></o:p></div></div><div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;">Kind regards,<o:p></o:p></div></div><div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;">Carl-Johan<o:p></o:p></div></div><div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"> <o:p></o:p></div></div><div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"> <o:p></o:p></div></div><div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"> <o:p></o:p></div></div><div><div style="border-width: 1pt medium medium; border-style: solid none none; border-image: none; padding: 3pt 0cm 0cm; border-color: currentcolor;"><div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"><b>From:</b><span class="xapple-converted-space"> </span>Carl-Johan Thore<span class="xapple-converted-space"> </span><br><b>Sent:</b><span class="xapple-converted-space"> </span>Monday, April 24, 2023 5:08 PM<br><b>To:</b><span class="xapple-converted-space"> </span>Barry Smith <<a href="mailto:bsmith@petsc.dev" style="color: blue; text-decoration: underline;">bsmith@petsc.dev</a>><br><b>Subject:</b><span class="xapple-converted-space"> </span>RE: [petsc-users] Fieldsplit with redistribute<o:p></o:p></div></div></div></div><div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"> <o:p></o:p></div></div><div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;">Ok, that worked great with my code on 1 core! (I haven’t been able to try the multi-core case yet due to issues with my own code<o:p></o:p></div></div><div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;">mentioned below)<o:p></o:p></div></div><div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"> <o:p></o:p></div></div><div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;">I’m not sure if you forgot to remove the freeing of the map object<o:p></o:p></div></div><div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;">outside or if I messed up with the pull somehow, but I had to outcomment that line manually:<o:p></o:p></div></div><div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"> <o:p></o:p></div></div></div></div></blockquote></div></div></div><div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"><img alt="image001.png" src="cid:image001.png@01D97816.887DA6D0"><o:p></o:p></div></div><div><div><div><blockquote style="margin-top: 5pt; margin-bottom: 5pt;"><div><div><div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"> <o:p></o:p></div></div><div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;">/Carl-Johan<o:p></o:p></div></div><div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"> <o:p></o:p></div></div><div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"> <o:p></o:p></div></div><div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"> <o:p></o:p></div></div><div><div style="border-width: 1pt medium medium; border-style: solid none none; border-image: none; padding: 3pt 0cm 0cm; border-color: currentcolor;"><div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"><b>From:</b><span class="xapple-converted-space"> </span>Barry Smith <<a href="mailto:bsmith@petsc.dev" style="color: blue; text-decoration: underline;">bsmith@petsc.dev</a>><span class="xapple-converted-space"> </span><br><b>Sent:</b><span class="xapple-converted-space"> </span>Monday, April 24, 2023 4:26 PM<br><b>To:</b><span class="xapple-converted-space"> </span>Carl-Johan Thore <<a href="mailto:carl-johan.thore@liu.se" style="color: blue; text-decoration: underline;">carl-johan.thore@liu.se</a>><br><b>Cc:</b><span class="xapple-converted-space"> </span>PETSc <<a href="mailto:petsc-users@mcs.anl.gov" style="color: blue; text-decoration: underline;">petsc-users@mcs.anl.gov</a>><br><b>Subject:</b><span class="xapple-converted-space"> </span>Re: [petsc-users] Fieldsplit with redistribute<o:p></o:p></div></div></div></div><div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"> <o:p></o:p></div></div><div><div><div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"> <o:p></o:p></div></div></div><div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"> The bug was mine; I was freeing the map object outside of the if () instead of inside. You can do<o:p></o:p></div></div><div><div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"> <o:p></o:p></div></div></div><div><div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"> git pull<o:p></o:p></div></div></div><div><div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"> make all<o:p></o:p></div></div></div><div><div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"> <o:p></o:p></div></div></div><div><div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"> and then try again.<o:p></o:p></div></div></div><div><div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"> <o:p></o:p></div></div></div><div><div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"> Barry<o:p></o:p></div></div></div><div><div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"> <o:p></o:p></div></div><div><p class="xmsonormal" style="margin-right: 0cm; margin-left: 0cm; font-size: 11pt; font-family: Calibri, sans-serif; margin-bottom: 12pt;"> <o:p></o:p></p><blockquote style="margin-top: 5pt; margin-bottom: 5pt;"><div><div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;">On Apr 24, 2023, at 5:39 AM, Carl-Johan Thore <<a href="mailto:carl-johan.thore@liu.se" style="color: blue; text-decoration: underline;">carl-johan.thore@liu.se</a>> wrote:<o:p></o:p></div></div></div><div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"> <o:p></o:p></div></div><div><div><div><div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;">Hi Barry!<o:p></o:p></div></div></div><div><div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"> <o:p></o:p></div></div></div><div><div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;">First of all, thank you very very much for this! I was expecting maybe a few hints and pointers on how to proceed<o:p></o:p></div></div></div><div><div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;">with my work, but then you did a complete implementation …<span class="xxapple-converted-space"> </span><o:p></o:p></div></div></div><div><div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"> <o:p></o:p></div></div></div><div><div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;">Your code ran fine with<span class="xxapple-converted-space"> </span><a href="http://ex84.cc/" style="color: blue; text-decoration: underline;">ex84.cc</a>. Unfortunately it crashed when running on my main code (a mixed Stokes solver).<o:p></o:p></div></div></div><div><div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;">When running on 1 core I get a crash which is maybe related to your code, so I’ve attached the error message for that<o:p></o:p></div></div></div><div><div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;">case. However, on multiple cores I think the issue is mainly that I’m not constructing the original IS correctly, so I’ll<o:p></o:p></div></div></div><div><div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;">look into that myself.<span class="xxapple-converted-space"> </span><o:p></o:p></div></div></div><div><div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"> <o:p></o:p></div></div></div><div><div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;">Regarding reporting to<span class="xxapple-converted-space"> </span><a href="https://gitlab.com/petsc/petsc/-/merge_requests/6366" style="color: blue; text-decoration: underline;">https://gitlab.com/petsc/petsc/-/merge_requests/6366</a>, should it be done here?:<o:p></o:p></div></div></div></div></div></blockquote></div></div></div></div></div></blockquote></div></div></div><div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"><img alt="image002.png" src="cid:image002.png@01D97816.887DA6D0"><o:p></o:p></div></div><div><div><div><blockquote style="margin-top: 5pt; margin-bottom: 5pt;"><div><div><div><div><div><blockquote style="margin-top: 5pt; margin-bottom: 5pt;"><div><div><div><div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"> <o:p></o:p></div></div></div><div><div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;">By the way, I managed yesterday to make a working implementation of my own example and was planning to send it<o:p></o:p></div></div></div><div><div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;">after cleaning it up and maybe optimizing a bit. I’ve attached it if your curious (or just want to have a good laugh :))<o:p></o:p></div></div></div><div><div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"> <o:p></o:p></div></div></div><div><div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;">Kind regards,<o:p></o:p></div></div></div><div><div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;">Carl-Johan<o:p></o:p></div></div></div><div><div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"> <o:p></o:p></div></div></div><div><div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"> <o:p></o:p></div></div></div><div><div style="border-width: 1pt medium medium; border-style: solid none none; border-image: none; padding: 3pt 0cm 0cm; border-color: currentcolor;"><div><div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"><b>From:</b><span class="xxapple-converted-space"> </span>Barry Smith <<a href="mailto:bsmith@petsc.dev" style="color: blue; text-decoration: underline;">bsmith@petsc.dev</a>><span class="xxapple-converted-space"> </span><br><b>Sent:</b><span class="xxapple-converted-space"> </span>Monday, April 24, 2023 12:49 AM<br><b>To:</b><span class="xxapple-converted-space"> </span>Carl-Johan Thore <<a href="mailto:carl-johan.thore@liu.se" style="color: blue; text-decoration: underline;">carl-johan.thore@liu.se</a>><br><b>Cc:</b><span class="xxapple-converted-space"> </span>PETSc <<a href="mailto:petsc-users@mcs.anl.gov" style="color: blue; text-decoration: underline;">petsc-users@mcs.anl.gov</a>><br><b>Subject:</b><span class="xxapple-converted-space"> </span>Re: [petsc-users] Fieldsplit with redistribute<o:p></o:p></div></div></div></div></div><div><div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"> <o:p></o:p></div></div></div><div><div><div><div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"> <o:p></o:p></div></div></div></div><div><div><div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"> I have added support for PCREDISTRIBUTE to propogate your PCFieldSplitSetIS() down to an inner PCFIELDSPLIT. You can <o:p></o:p></div></div></div></div><div><div><div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;">access it with<o:p></o:p></div></div></div></div><div><div><div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"> <o:p></o:p></div></div></div></div><div><div><div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"> git fetch<o:p></o:p></div></div></div></div><div><div><div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"> git checkout <b><span style="font-size: 10.5pt; font-family: "Times New Roman", serif; color: rgb(200, 20, 201);">barry/2023-04-22/fieldsplit-fields-propogate</span></b><o:p></o:p></div></div></div></div><div><div><div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"><b><span style="font-size: 10.5pt; font-family: "Times New Roman", serif; color: rgb(200, 20, 201);"> </span></b>./configure<o:p></o:p></div></div></div></div><div><div><div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"> make all check<o:p></o:p></div></div></div></div><div><div><div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"> <o:p></o:p></div></div></div></div><div><div><div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"> Take a look at src/ksp/ksp/tutorials/ex84.c and run with the options at the bottom of the file.<o:p></o:p></div></div></div></div><div><div><div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"> <o:p></o:p></div></div></div></div><div><div><div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"> Please let us know at <a href="https://gitlab.com/petsc/petsc/-/merge_requests/6366" style="color: blue; text-decoration: underline;">https://gitlab.com/petsc/petsc/-/merge_requests/6366</a> if it works for you or you have any difficulties.<o:p></o:p></div></div></div></div><div><div><div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"> <o:p></o:p></div></div></div></div><div><div><div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"> Barry<o:p></o:p></div></div></div></div><div><div><div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"> <o:p></o:p></div></div></div></div><div><div><div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"> <o:p></o:p></div></div></div><div><div><p class="xmsonormal" style="margin-right: 0cm; margin-left: 0cm; font-size: 11pt; font-family: Calibri, sans-serif; margin-bottom: 12pt;"> <o:p></o:p></p></div><blockquote style="margin-top: 5pt; margin-bottom: 5pt;"><div><div><div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;">On Apr 20, 2023, at 10:14 AM, Carl-Johan Thore <<a href="mailto:carl-johan.thore@liu.se" style="color: blue; text-decoration: underline;">carl-johan.thore@liu.se</a>> wrote:<o:p></o:p></div></div></div></div><div><div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"> <o:p></o:p></div></div></div><div><div><div><div><div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;">Great, thanks! I’ve attached the code, a makefile, and a 1-page power-point which hopefully explains<o:p></o:p></div></div></div></div><div><div><div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;">what I’m trying to do on this little toy-problem. There is obviously (?) something I need to add around<o:p></o:p></div></div></div></div><div><div><div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;">line 327 in the code in order to move the indices to the correct rank.<o:p></o:p></div></div></div></div><div><div><div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"> <o:p></o:p></div></div></div></div><div><div><div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;">Output should be something like this when running:<o:p></o:p></div></div></div></div><div><div><div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"> <o:p></o:p></div></div></div></div></div></div></blockquote></div></div></div></div></div></blockquote></div></div></div></div></div></blockquote></div></div></div><div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"><img alt="image003.png" src="cid:image003.png@01D97816.887DA6D0"><o:p></o:p></div></div><div><div><div><blockquote style="margin-top: 5pt; margin-bottom: 5pt;"><div><div><div><div><div><blockquote style="margin-top: 5pt; margin-bottom: 5pt;"><div><div><div><div><div><blockquote style="margin-top: 5pt; margin-bottom: 5pt;"><div><div><div><div><div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"> <o:p></o:p></div></div></div></div><div><div><div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;">Let me know if you need any more info, or if the code is incomprehensible or so<o:p></o:p></div></div></div></div><div><div><div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;">(it’s long because I’ve copied a lot from redistribute.c)<o:p></o:p></div></div></div></div><div><div><div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"> <o:p></o:p></div></div></div></div><div><div><div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;">Kind regards,<o:p></o:p></div></div></div></div><div><div><div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;">Carl-Johan<o:p></o:p></div></div></div></div><div><div><div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"> <o:p></o:p></div></div></div></div><div><div style="border-width: 1pt medium medium; border-style: solid none none; border-image: none; padding: 3pt 0cm 0cm; border-color: currentcolor;"><div><div><div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"><b>From:</b><span class="xxxapple-converted-space"> </span>Barry Smith <<a href="mailto:bsmith@petsc.dev" style="color: blue; text-decoration: underline;">bsmith@petsc.dev</a>><span class="xxxapple-converted-space"> </span><br><b>Sent:</b><span class="xxxapple-converted-space"> </span>Thursday, April 20, 2023 3:17 PM<br><b>To:</b><span class="xxxapple-converted-space"> </span>Carl-Johan Thore <<a href="mailto:carl-johan.thore@liu.se" style="color: blue; text-decoration: underline;">carl-johan.thore@liu.se</a>><br><b>Subject:</b><span class="xxxapple-converted-space"> </span>Re: [petsc-users] Fieldsplit with redistribute<o:p></o:p></div></div></div></div></div></div><div><div><div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"> <o:p></o:p></div></div></div></div><div><div><div><div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"> <o:p></o:p></div></div></div></div></div><div><div><div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"> Sure<o:p></o:p></div></div></div></div><div><div><p class="xxmsonormal" style="margin-right: 0cm; margin-left: 0cm; font-size: 11pt; font-family: Calibri, sans-serif; margin-bottom: 12pt;"> <o:p></o:p></p></div><blockquote style="margin-top: 5pt; margin-bottom: 5pt;"><div><div><div><div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;">On Apr 20, 2023, at 4:09 AM, Carl-Johan Thore <<a href="mailto:carl-johan.thore@liu.se" style="color: blue; text-decoration: underline;">carl-johan.thore@liu.se</a>> wrote:<o:p></o:p></div></div></div></div></div><div><div><div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"> <o:p></o:p></div></div></div></div><div><div><div><div><div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;">Hi Barry,<o:p></o:p></div></div></div></div></div><div><div><div><div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"> <o:p></o:p></div></div></div></div></div><div><div><div><div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;">In the conversation below you mentioned that I could send code to you to take a look. I’ve written<o:p></o:p></div></div></div></div></div><div><div><div><div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;">up what I think is a minimally working example for this. It’s almost there in the sense of distributing<o:p></o:p></div></div></div></div></div><div><div><div><div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;">the correct number of indices to the ranks to match the reduced matrix, but it’s the wrong indices.<o:p></o:p></div></div></div></div></div><div><div><div><div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;">Would it be okay if I sent you the code to have look?<o:p></o:p></div></div></div></div></div><div><div><div><div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"> <o:p></o:p></div></div></div></div></div><div><div><div><div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;">Kind regards,<o:p></o:p></div></div></div></div></div><div><div><div><div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;">Carl-Johan<span class="xxxapple-converted-space"> </span><o:p></o:p></div></div></div></div></div><div><div><div><div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"> <o:p></o:p></div></div></div></div></div><div><div style="border-width: 1pt medium medium; border-style: solid none none; border-image: none; padding: 3pt 0cm 0cm; border-color: currentcolor;"><div><div><div><div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"><b>From:</b><span class="xxxapple-converted-space"> </span>Barry Smith <<a href="mailto:bsmith@petsc.dev" style="color: blue; text-decoration: underline;"><span style="color: rgb(5, 99, 193);">bsmith@petsc.dev</span></a>><span class="xxxapple-converted-space"> </span><br><b>Sent:</b><span class="xxxapple-converted-space"> </span>Sunday, April 16, 2023 10:31 PM<br><b>To:</b><span class="xxxapple-converted-space"> </span>Carl-Johan Thore <<a href="mailto:carl-johan.thore@liu.se" style="color: blue; text-decoration: underline;"><span style="color: rgb(5, 99, 193);">carl-johan.thore@liu.se</span></a>><br><b>Cc:</b><span class="xxxapple-converted-space"> </span><a href="mailto:petsc-users@mcs.anl.gov" style="color: blue; text-decoration: underline;"><span style="color: rgb(5, 99, 193);">petsc-users@mcs.anl.gov</span></a><br><b>Subject:</b><span class="xxxapple-converted-space"> </span>Re: [petsc-users] Fieldsplit with redistribute<o:p></o:p></div></div></div></div></div></div></div><div><div><div><div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"> <o:p></o:p></div></div></div></div></div><div><div><div><div><div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"> <o:p></o:p></div></div></div></div></div></div><div><div><div><div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"> The manual page for ISEmbed is incomprehensible to me. Anyways no matter what, you need to know what degrees of freedom are removed by PCDistribute() in order to produce the reduced IS which is why I think you need information only available inside PCSetUp_Redistribute(). (Sorry it is PCSetUp_Redistribute() not PCApply_Redistribute())<o:p></o:p></div></div></div></div></div><div><div><div><div><div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"> <o:p></o:p></div></div></div></div></div></div><div><div><div><div><div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"> Barry<o:p></o:p></div></div></div></div></div></div><div><div><div><div><div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"> <o:p></o:p></div></div></div></div></div><div><div><div><p class="xxmsonormal" style="margin-right: 0cm; margin-left: 0cm; font-size: 11pt; font-family: Calibri, sans-serif; margin-bottom: 12pt;"> <o:p></o:p></p></div></div><blockquote style="margin-top: 5pt; margin-bottom: 5pt;"><div><div><div><div><div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;">On Apr 16, 2023, at 3:36 PM, Carl-Johan Thore <<a href="mailto:carl-johan.thore@liu.se" style="color: blue; text-decoration: underline;"><span style="color: rgb(5, 99, 193);">carl-johan.thore@liu.se</span></a>> wrote:<o:p></o:p></div></div></div></div></div></div><div><div><div><div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"> <o:p></o:p></div></div></div></div></div><div><div><div><div><div><div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;">Thanks for the quick reply Barry!<br>I have not tried the version with PCApply_Redistribute that you suggest, but I have a code that does roughly what you describe. It works when running on one rank, but fails on multiple ranks. I suspect the issue is with the use of ISEmbed as, quoting the PETSc-manual, "the resulting IS is sequential, since the index substitution it encodes is purely local" (admittedly I don't fully understand what that means). If you think using ISEmbed is not a good idea, I'll try PCApply_Redistribute()<o:p></o:p></div></div></div></div></div><div class="MsoNormal" align="center" style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif; text-align: center;"><hr size="2" width="98%" align="center"></div><div id="x_x_x_divRplyFwdMsg"><div><div><div><div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"><b>From:</b><span class="xxxapple-converted-space"> </span>Barry Smith <<a href="mailto:bsmith@petsc.dev" style="color: blue; text-decoration: underline;"><span style="color: rgb(5, 99, 193);">bsmith@petsc.dev</span></a>><br><b>Sent:</b><span class="xxxapple-converted-space"> </span>16 April 2023 21:11:18<br><b>To:</b><span class="xxxapple-converted-space"> </span>Carl-Johan Thore <<a href="mailto:carl-johan.thore@liu.se" style="color: blue; text-decoration: underline;"><span style="color: rgb(5, 99, 193);">carl-johan.thore@liu.se</span></a>><br><b>Cc:</b><span class="xxxapple-converted-space"> </span><a href="mailto:petsc-users@mcs.anl.gov" style="color: blue; text-decoration: underline;"><span style="color: rgb(5, 99, 193);">petsc-users@mcs.anl.gov</span></a><span class="xxxapple-converted-space"> </span><<a href="mailto:petsc-users@mcs.anl.gov" style="color: blue; text-decoration: underline;"><span style="color: rgb(5, 99, 193);">petsc-users@mcs.anl.gov</span></a>><br><b>Subject:</b><span class="xxxapple-converted-space"> </span>Re: [petsc-users] Fieldsplit with redistribute<span class="xxxapple-converted-space"> </span><o:p></o:p></div></div></div></div></div><div><div><div><div><div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"> <o:p></o:p></div></div></div></div></div></div></div><div><div><p class="xxxmsonormal" style="margin-right: 0cm; margin-left: 0cm; font-size: 11pt; font-family: Calibri, sans-serif; margin-bottom: 12pt;"><br> There is no code to do this currently.<span class="xxxapple-converted-space"> </span><br><br> I would start by building your IS for each split before the PCRedistribute and then adding to the PCApply_Redistribute() code that "fixes" these IS by "removing" the entries of the IS associated with removed degrees of freedom and then shifting the entries indices of the IS by taking into account the removed indices. But you have probably already been trying this? It does require digging directly into the PCApply_Redistribute() to get the needed information (which degrees of freedom are removed by the redistribute code), plus it requires shifting the MPI rank ownership of the entries of the IS in the same way the MPI rank ownership of the degrees of freedom of the vector are moved.<br><br> If you have some code that you think should be doing this but doesn't work feel free to send it to us and we may be able to fix it.<br><br> Barry<br><br><br>> On Apr 16, 2023, at 2:50 PM, Carl-Johan Thore via petsc-users <<a href="mailto:petsc-users@mcs.anl.gov" style="color: blue; text-decoration: underline;"><span style="color: rgb(5, 99, 193);">petsc-users@mcs.anl.gov</span></a>> wrote:<br>><span class="xxxapple-converted-space"> </span><br>> Hello,<br>> I'm solving a blocksystem<br>> [A C;<br>> C' D],<br>> where D is not zero, using the PCFIELDSPLIT preconditioner and set the split using PetscFieldSplitSetIS. This works very well until I try PCREDISTRIBUTE (which is attractive as I have many locked DOFs). I suspect something goes wrong when constructing the IS for the split (I've tried various things using the IS-routines). Can PETSc do this automatically? Or else, any hints?<br>> Kind regards,<br>> Carl-Johan</p></div></div></div></div></blockquote></div></div></div></blockquote></div></div></div></blockquote></div></div></div></div></div></blockquote></div></div></div></div></div></blockquote></div></div></div></div></div></blockquote></div><br></body></html>