<html><head><meta http-equiv="content-type" content="text/html; charset=utf-8"></head><body style="overflow-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;"><div><br></div> Thanks, seems we are moving to the same page.<br><div><br><blockquote type="cite"><div>On Apr 26, 2023, at 11:55 AM, Carl-Johan Thore <carl-johan.thore@liu.se> wrote:</div><br class="Apple-interchange-newline"><div><meta charset="UTF-8"><div class="WordSection1" style="page: WordSection1; caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 18px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;"><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;">Ok yes, I thought one was supposed to provide the “reduced” indices. When I did that, the sequential case worked ..<o:p></o:p></div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;">But it is clearly much more convenient to only have to provide the original indices<o:p></o:p></div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"><o:p> </o:p></div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;">Just to be very clear: to me, “the IS for the original problem without any concern…” is an IS containing ALL velocity DOFs, including locked<o:p></o:p></div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;">ones, and pcfieldsplit should compute the pressure DOFs automatically as the complement.</div></div></div></blockquote><div><br></div> Yes</div><div><br><blockquote type="cite"><div><div class="WordSection1" style="page: WordSection1; caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 18px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;"><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"><o:p></o:p></div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"><o:p> </o:p></div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;">If I provide this IS with pcfieldsplit WITHOUT PCREDISTRIBUTE, everything works fine both sequentially and in parallel.<o:p></o:p></div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"><o:p> </o:p></div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;">If I provide this IS with pcfieldsplit WITH PCREDISTRIBUTE neither the sequential nor the parallel case works. The attached<o:p></o:p></div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;">error message comes from the sequential case</div></div></div></blockquote><div><br></div> I'll take a look and see what might be causing this. Seems more likely a problem on our end of things.<br><blockquote type="cite"><div><div class="WordSection1" style="page: WordSection1; caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 18px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;"><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"><o:p></o:p></div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"><o:p> </o:p></div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"><o:p> </o:p></div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;">Not sure it is of any use, but here is how I construct the original IS, called isu, and pass it to pcfieldsplit:<o:p></o:p></div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"><o:p> </o:p></div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;">DMStagStencil stencil[24]; <span class="Apple-converted-space"> </span><o:p></o:p></div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;">for (PetscInt i=0; i<8; i++) {<o:p></o:p></div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"> for (PetscInt j=0; j<3; j++) {<o:p></o:p></div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"> stencil[j+i*3].loc = enodes[i];<o:p></o:p></div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"> stencil[j+i*3].c = j;<o:p></o:p></div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"> stencil[j+i*3].i = 0;<o:p></o:p></div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"> stencil[j+i*3].j = 0;<o:p></o:p></div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"> stencil[j+i*3].k = 0;<o:p></o:p></div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"> }<o:p></o:p></div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;">}<o:p></o:p></div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;">ierr = DMStagCreateISFromStencils(dm_state,24,stencil,&isu); CHKERRQ(ierr);<o:p></o:p></div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;">ierr = PCFieldSplitSetIS(ipc,"0",isu); CHKERRQ(ierr);<o:p></o:p></div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"><o:p> </o:p></div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"><o:p> </o:p></div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"><o:p> </o:p></div><div><div style="border-width: 1pt medium medium; border-style: solid none none; border-color: rgb(225, 225, 225) currentcolor currentcolor; border-image: none; padding: 3pt 0cm 0cm;"><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"><b>From:</b><span class="Apple-converted-space"> </span>Barry Smith <<a href="mailto:bsmith@petsc.dev" style="color: blue; text-decoration: underline;">bsmith@petsc.dev</a>><span class="Apple-converted-space"> </span><br><b>Sent:</b><span class="Apple-converted-space"> </span>Wednesday, April 26, 2023 5:21 PM<br><b>To:</b><span class="Apple-converted-space"> </span>Carl-Johan Thore <<a href="mailto:carl-johan.thore@liu.se" style="color: blue; text-decoration: underline;">carl-johan.thore@liu.se</a>><br><b>Cc:</b><span class="Apple-converted-space"> </span>PETSc <<a href="mailto:petsc-users@mcs.anl.gov" style="color: blue; text-decoration: underline;">petsc-users@mcs.anl.gov</a>><br><b>Subject:</b><span class="Apple-converted-space"> </span>Re: [petsc-users] Fieldsplit with redistribute<o:p></o:p></div></div></div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"><o:p> </o:p></div><div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"><o:p> </o:p></div></div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"> Perhaps there is a misunderstanding. With the code I added you always just provide IS for the original problem without any concern for what rows will be zeroed and what rows will be redistributed. PCREDISTRIBUTE manages "fixing" things. <o:p></o:p></div><div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"><o:p> </o:p></div></div><div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"> I want you to use PCFIEDLSPLIT and not PCREDISTRIBUTE at all (so -pc_type fieldsplit) with your IS and see if that works correctly sequentially and in parallel. <o:p></o:p></div></div><div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"><o:p> </o:p></div></div><div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"> Then I want you to use -pc_type redistribute -redistribute_pc_type fieldsplit with your IS sequentially and in parallel (note you do not change your code, or even recompiled it for all the cases.)<o:p></o:p></div></div><div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"><o:p> </o:p></div></div><div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"> Barry<o:p></o:p></div></div><div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"><o:p> </o:p></div><div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"><o:p> </o:p></div></div><div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"><o:p> </o:p></div><div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"><br><br><o:p></o:p></div><blockquote style="margin-top: 5pt; margin-bottom: 5pt;"><div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;">On Apr 26, 2023, at 10:58 AM, Carl-Johan Thore <<a href="mailto:carl-johan.thore@liu.se" style="color: blue; text-decoration: underline;">carl-johan.thore@liu.se</a>> wrote:<o:p></o:p></div></div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"><o:p> </o:p></div><div><div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;">Because without redistribute, pcfieldsplit expects number pointing to rows in the “big” matrix, whereas when<o:p></o:p></div></div><div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;">I construct my IS for redistribute it will have indices pointing to rows in the “reduced matrix”. Coming back to my small<o:p></o:p></div></div><div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;">example below, I construct an IS with 0 2 4 as expected (?) for the reduced matrix.<o:p></o:p></div></div><div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"></div></div></div></blockquote></div></div></div></div></div></blockquote></div></body></html>