<div dir="ltr"><div dir="ltr">On Thu, Jan 31, 2019 at 12:22 AM Dave Lee <<a href="mailto:davelee2804@gmail.com">davelee2804@gmail.com</a>> wrote:<br></div><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr">Hi Barry and Matt,<div><br></div><div>This seems to work for the outer (composite multiplicative) field split, but fails for the inner (schur) field split (output error file attached).</div></div></div></div></div></blockquote><div><br></div><div>The inner FS numbering is with respect to the reduced problem, not the original global problem (it can't know you did one FS on the outside).</div><div><br></div><div>  THanks,</div><div><br></div><div>    Matt</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div>I'm setting up the nested field splits as:</div><div><br></div><div><p style="margin:0px;font-stretch:normal;font-size:12px;line-height:normal;font-family:Menlo;color:rgb(0,249,0);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">  SNESGetKSP(snes, &ksp);</span></p>
<p style="margin:0px;font-stretch:normal;font-size:12px;line-height:normal;font-family:Menlo;color:rgb(0,249,0);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">  KSPGetPC(ksp, &pc);</span></p>
<p style="margin:0px;font-stretch:normal;font-size:12px;line-height:normal;font-family:Menlo;color:rgb(0,249,0);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">  PCSetType(pc, PCFIELDSPLIT);</span></p>
<p style="margin:0px;font-stretch:normal;font-size:12px;line-height:normal;font-family:Menlo;color:rgb(0,249,0);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">  PCFieldSplitSetType(pc, PC_COMPOSITE_MULTIPLICATIVE);</span></p><p style="margin:0px;font-stretch:normal;font-size:12px;line-height:normal;font-family:Menlo;color:rgb(0,249,0);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures"><br></span></p></div><div><p style="margin:0px;font-stretch:normal;font-size:12px;line-height:normal;font-family:Menlo;color:rgb(0,249,0);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">  </span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(206,121,36)">for</span><span style="font-variant-ligatures:no-common-ligatures">(</span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(52,189,38)">int</span><span style="font-variant-ligatures:no-common-ligatures"> slice_i = </span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(195,55,32)">0</span><span style="font-variant-ligatures:no-common-ligatures">; slice_i < nSlice; slice_i++) {</span></p>
<p style="margin:0px;font-stretch:normal;font-size:12px;line-height:normal;font-family:Menlo;color:rgb(0,249,0);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">    </span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(206,121,36)">if</span><span style="font-variant-ligatures:no-common-ligatures">(Geometry::procID() == slice_i) {</span></p>
<p style="margin:0px;font-stretch:normal;font-size:12px;line-height:normal;font-family:Menlo;color:rgb(0,249,0);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">      ISCreateStride(MPI_COMM_SELF, nDofsSlice, slice_i * nDofsSlice, </span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(195,55,32)">1</span><span style="font-variant-ligatures:no-common-ligatures">, &is_s[slice_i]);</span></p>
<p style="margin:0px;font-stretch:normal;font-size:12px;line-height:normal;font-family:Menlo;color:rgb(0,249,0);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">    } </span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(206,121,36)">else</span><span style="font-variant-ligatures:no-common-ligatures"> {</span></p>
<p style="margin:0px;font-stretch:normal;font-size:12px;line-height:normal;font-family:Menlo;color:rgb(0,249,0);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">      ISCreateStride(MPI_COMM_SELF, </span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(195,55,32)">0</span><span style="font-variant-ligatures:no-common-ligatures">, </span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(195,55,32)">0</span><span style="font-variant-ligatures:no-common-ligatures">, </span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(195,55,32)">1</span><span style="font-variant-ligatures:no-common-ligatures">, &is_s[slice_i]);</span></p>
<p style="margin:0px;font-stretch:normal;font-size:12px;line-height:normal;font-family:Menlo;color:rgb(0,249,0);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">    }</span></p>
<p style="margin:0px;font-stretch:normal;font-size:12px;line-height:normal;font-family:Menlo;color:rgb(0,249,0);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">    MPI_Barrier(MPI_COMM_WORLD);</span></p>
<p style="margin:0px;font-stretch:normal;font-size:12px;line-height:normal;font-family:Menlo;color:rgb(0,249,0);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">    PCFieldSplitSetIS(pc, </span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(195,55,32)">NULL</span><span style="font-variant-ligatures:no-common-ligatures">, is_s[slice_i]);</span></p>
<p style="margin:0px;font-stretch:normal;font-size:12px;line-height:normal;font-family:Menlo;color:rgb(0,249,0);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">  }</span></p>
<p style="margin:0px;font-stretch:normal;font-size:12px;line-height:normal;font-family:Menlo;color:rgb(0,249,0);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">  PCSetUp(pc);</span></p>
<p style="margin:0px;font-stretch:normal;font-size:12px;line-height:normal;font-family:Menlo;color:rgb(0,249,0);background-color:rgb(0,0,0);min-height:14px"><span style="font-variant-ligatures:no-common-ligatures"></span><br></p>
<p style="margin:0px;font-stretch:normal;font-size:12px;line-height:normal;font-family:Menlo;color:rgb(0,249,0);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">  PCFieldSplitGetSubKSP(pc, &n_split, &ksp_i);</span></p>
<p style="margin:0px;font-stretch:normal;font-size:12px;line-height:normal;font-family:Menlo;color:rgb(0,249,0);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">  </span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(206,121,36)">for</span><span style="font-variant-ligatures:no-common-ligatures">(</span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(52,189,38)">int</span><span style="font-variant-ligatures:no-common-ligatures"> slice_i = </span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(195,55,32)">0</span><span style="font-variant-ligatures:no-common-ligatures">; slice_i < nSlice; slice_i++) {</span></p>
<p style="margin:0px;font-stretch:normal;font-size:12px;line-height:normal;font-family:Menlo;color:rgb(0,249,0);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">    KSPGetPC(ksp_i[slice_i], &pc_i);</span></p>
<p style="margin:0px;font-stretch:normal;font-size:12px;line-height:normal;font-family:Menlo;color:rgb(0,249,0);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">    PCSetType(pc_i, PCFIELDSPLIT);</span></p>
<p style="margin:0px;font-stretch:normal;font-size:12px;line-height:normal;font-family:Menlo;color:rgb(0,249,0);background-color:rgb(0,0,0);min-height:14px"><span style="font-variant-ligatures:no-common-ligatures"></span><br></p>
<p style="margin:0px;font-stretch:normal;font-size:12px;line-height:normal;font-family:Menlo;color:rgb(0,249,0);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">    </span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(206,121,36)">if</span><span style="font-variant-ligatures:no-common-ligatures">(Geometry::procID() == slice_i) {</span></p>
<p style="margin:0px;font-stretch:normal;font-size:12px;line-height:normal;font-family:Menlo;color:rgb(0,249,0);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">      ISCreateStride(MPI_COMM_SELF, nDofs_u, slice_i * nDofsSlice, </span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(195,55,32)">1</span><span style="font-variant-ligatures:no-common-ligatures">, &is_u[slice_i]);</span></p>
<p style="margin:0px;font-stretch:normal;font-size:12px;line-height:normal;font-family:Menlo;color:rgb(0,249,0);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">      ISCreateStride(MPI_COMM_SELF, nDofs_p, slice_i * nDofsSlice + nDofs_u, </span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(195,55,32)">1</span><span style="font-variant-ligatures:no-common-ligatures">, &is_p[slice_i]);</span></p>
<p style="margin:0px;font-stretch:normal;font-size:12px;line-height:normal;font-family:Menlo;color:rgb(0,249,0);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">    } </span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(206,121,36)">else</span><span style="font-variant-ligatures:no-common-ligatures"> {</span></p>
<p style="margin:0px;font-stretch:normal;font-size:12px;line-height:normal;font-family:Menlo;color:rgb(0,249,0);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">      ISCreateStride(MPI_COMM_SELF, </span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(195,55,32)">0</span><span style="font-variant-ligatures:no-common-ligatures">, </span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(195,55,32)">0</span><span style="font-variant-ligatures:no-common-ligatures">, </span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(195,55,32)">1</span><span style="font-variant-ligatures:no-common-ligatures">, &is_u[slice_i]);</span></p>
<p style="margin:0px;font-stretch:normal;font-size:12px;line-height:normal;font-family:Menlo;color:rgb(0,249,0);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">      ISCreateStride(MPI_COMM_SELF, </span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(195,55,32)">0</span><span style="font-variant-ligatures:no-common-ligatures">, </span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(195,55,32)">0</span><span style="font-variant-ligatures:no-common-ligatures">, </span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(195,55,32)">1</span><span style="font-variant-ligatures:no-common-ligatures">, &is_p[slice_i]);</span></p>
<p style="margin:0px;font-stretch:normal;font-size:12px;line-height:normal;font-family:Menlo;color:rgb(0,249,0);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">    }</span></p>
<p style="margin:0px;font-stretch:normal;font-size:12px;line-height:normal;font-family:Menlo;color:rgb(0,249,0);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">    MPI_Barrier(MPI_COMM_WORLD);</span></p>
<p style="margin:0px;font-stretch:normal;font-size:12px;line-height:normal;font-family:Menlo;color:rgb(0,249,0);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">    PCFieldSplitSetIS(pc_i, </span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(195,55,32)">"u"</span><span style="font-variant-ligatures:no-common-ligatures">, is_u[slice_i]);</span></p>
<p style="margin:0px;font-stretch:normal;font-size:12px;line-height:normal;font-family:Menlo;color:rgb(0,249,0);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">    PCFieldSplitSetIS(pc_i, </span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(195,55,32)">"p"</span><span style="font-variant-ligatures:no-common-ligatures">, is_p[slice_i]);</span></p>
<p style="margin:0px;font-stretch:normal;font-size:12px;line-height:normal;font-family:Menlo;color:rgb(0,249,0);background-color:rgb(0,0,0);min-height:14px"><br><span style="font-variant-ligatures:no-common-ligatures"></span></p>
<p style="margin:0px;font-stretch:normal;font-size:12px;line-height:normal;font-family:Menlo;color:rgb(0,249,0);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">    PCFieldSplitSetType(pc_i, PC_COMPOSITE_SCHUR);</span></p>
<p style="margin:0px;font-stretch:normal;font-size:12px;line-height:normal;font-family:Menlo;color:rgb(0,249,0);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">    PCSetUp(pc_i);</span></p>
<p style="margin:0px;font-stretch:normal;font-size:12px;line-height:normal;font-family:Menlo;color:rgb(0,249,0);background-color:rgb(0,0,0);min-height:14px"><span style="font-variant-ligatures:no-common-ligatures"></span><br></p>
<p style="margin:0px;font-stretch:normal;font-size:12px;line-height:normal;font-family:Menlo;color:rgb(0,249,0);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">    </span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(206,121,36)">if</span><span style="font-variant-ligatures:no-common-ligatures">(Geometry::procID() == slice_i) {</span></p>
<p style="margin:0px;font-stretch:normal;font-size:12px;line-height:normal;font-family:Menlo;color:rgb(0,249,0);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">      PCFieldSplitGetSubKSP(pc_i, &m_split, &ksp_j);</span></p>
<p style="margin:0px;font-stretch:normal;font-size:12px;line-height:normal;font-family:Menlo;color:rgb(0,249,0);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">      KSPSetType(ksp_j</span><span style="font-variant-ligatures:no-common-ligatures;background-color:rgb(230,230,0)">[</span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(153,18,0);background-color:rgb(230,230,0)">0</span><span style="font-variant-ligatures:no-common-ligatures;background-color:rgb(230,230,0)">]</span><span style="font-variant-ligatures:no-common-ligatures">, KSPGMRES);</span></p>
<p style="margin:0px;font-stretch:normal;font-size:12px;line-height:normal;font-family:Menlo;color:rgb(0,249,0);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">      KSPSetType(ksp_j[</span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(195,55,32)">1</span><span style="font-variant-ligatures:no-common-ligatures">], KSPGMRES);</span></p>
<p style="margin:0px;font-stretch:normal;font-size:12px;line-height:normal;font-family:Menlo;color:rgb(0,249,0);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">      KSPGetPC(ksp_j</span><span style="font-variant-ligatures:no-common-ligatures;background-color:rgb(230,230,0)">[</span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(153,18,0);background-color:rgb(230,230,0)">0</span><span style="font-variant-ligatures:no-common-ligatures;background-color:rgb(230,230,0)">]</span><span style="font-variant-ligatures:no-common-ligatures">, &pc_j);</span></p>
<p style="margin:0px;font-stretch:normal;font-size:12px;line-height:normal;font-family:Menlo;color:rgb(0,249,0);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">      PCSetType(pc_j, PCJACOBI);</span></p>
<p style="margin:0px;font-stretch:normal;font-size:12px;line-height:normal;font-family:Menlo;color:rgb(0,249,0);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">      KSPGetPC(ksp_j[</span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(195,55,32)">1</span><span style="font-variant-ligatures:no-common-ligatures">], &pc_j);</span></p>
<p style="margin:0px;font-stretch:normal;font-size:12px;line-height:normal;font-family:Menlo;color:rgb(0,249,0);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">      PCSetType(pc_j, PCJACOBI);</span></p>
<p style="margin:0px;font-stretch:normal;font-size:12px;line-height:normal;font-family:Menlo;color:rgb(0,249,0);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">    }</span></p>
<p style="margin:0px;font-stretch:normal;font-size:12px;line-height:normal;font-family:Menlo;color:rgb(0,249,0);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">    MPI_Barrier(MPI_COMM_WORLD);</span></p>
<p style="margin:0px;font-stretch:normal;font-size:12px;line-height:normal;font-family:Menlo;color:rgb(0,249,0);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">  }</span></p></div><div><br></div><div><br></div><div>I could re-arrange the indexing and then use PCFieldSplitSetBlockSize(), however this will undermine the existing block diagonal structure of the system.</div><div><br></div></div></div></div>Cheers, Dave.<br><div class="gmail_quote"><div dir="ltr" class="gmail_attr"><br></div><div dir="ltr" class="gmail_attr">On Thu, Jan 31, 2019 at 1:28 PM Smith, Barry F. <<a href="mailto:bsmith@mcs.anl.gov" target="_blank">bsmith@mcs.anl.gov</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><br>
<br>
> On Jan 30, 2019, at 7:36 PM, Dave Lee via petsc-users <<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a>> wrote:<br>
> <br>
> Thanks Matt,<br>
> <br>
> I'm currently working through the details of the nested field split. I've decided to re-organise my application data layout so that the local portions are contiguous in the global space for consistency with the default PETSc layout.<br>
> <br>
> Does the IndexSet need to be the same size on each processor in order for this IndexSet to be supplied to a FieldSplit PC? <br>
<br>
   No, since it is the local size each process can have a different one.<br>
> <br>
> Moreover can an IndexSet have 0 entries on a particular processor and still be part of a global FieldSplit PC?<br>
<br>
   Yes (at least in theory, hopefully there are not bugs for this corner case).<br>
> <br>
> Cheers, Dave. <br>
> <br>
> <br>
> On Wed, Jan 30, 2019 at 11:10 PM Matthew Knepley <<a href="mailto:knepley@gmail.com" target="_blank">knepley@gmail.com</a>> wrote:<br>
> On Tue, Jan 29, 2019 at 7:58 PM Dave Lee via petsc-users <<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a>> wrote:<br>
> Hi,<br>
> <br>
> just wondering if its possible to manually specify the parallel decomposition of data within PETSc Vecs and Mats? And if so, will the FieldSplit preconditioning handle this also?<br>
> <br>
> I have an application (non-PETSc) data layout as:<br>
> DATA[16][4][num_procs][3472]<br>
> <br>
> and if possible I would like to use FieldSplit preconditioning without re-arranging this layout.<br>
> <br>
> Yes. You need to call<br>
> <br>
>   <a href="https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/PC/PCFieldSplitSetIS.html" rel="noreferrer" target="_blank">https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/PC/PCFieldSplitSetIS.html</a><br>
> <br>
> in order to define the splits. Note that if you have nested splits, you will have to also call it on the<br>
> nested PC, which stinks, but we have no other way of getting the information.<br>
> <br>
> Nested splits can also be handled by implementing the DM interface, but that is a lot of work. It is,<br>
> however, the way we handle it internally because it makes things much easier and more flexible.<br>
> <br>
>   Thanks,<br>
> <br>
>      Matt<br>
>  <br>
> Currently I have a nested FieldSplit preconditioning with 16 outer field splits, and a schur complement field split within each. however this throws up an indexing error as FieldSplit assumes that the parallel decomposition is contiguous (I think).<br>
> <br>
> [0]PETSC ERROR: #1 ISComplement() line 750 in /home/dlee0035/soft/petsc-3.9.3/src/vec/is/is/utils/iscoloring.c<br>
> [0]PETSC ERROR: #2 PCSetUp_FieldSplit() line 703 in /home/dlee0035/soft/petsc-3.9.3/src/ksp/pc/impls/fieldsplit/fieldsplit.c<br>
> [0]PETSC ERROR: #3 PCSetUp() line 923 in /home/dlee0035/soft/petsc-3.9.3/src/ksp/pc/interface/precon.c<br>
> <br>
> Cheers, Dave.<br>
> <br>
> <br>
> <br>
> <br>
> -- <br>
> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>
> -- Norbert Wiener<br>
> <br>
> <a href="https://www.cse.buffalo.edu/~knepley/" rel="noreferrer" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br>
<br>
</blockquote></div>
</div>
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr" class="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>-- Norbert Wiener</div><div><br></div><div><a href="http://www.cse.buffalo.edu/~knepley/" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br></div></div></div></div></div></div></div></div>