<div dir="ltr"><div dir="ltr">On Sat, Sep 26, 2020 at 4:13 PM Krishnan, Gautham <<a href="mailto:gautham3@illinois.edu">gautham3@illinois.edu</a>> wrote:<br></div><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="ltr">
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<div style="margin:0px;font-size:12pt;font-family:Calibri,Arial,Helvetica,sans-serif;color:black;background-color:rgb(255,255,255)">
In parallel, I have passed the global indices that I want on the PE. However, the result I obtain seems to change based on whether I divide the initial domain between PEs along the x axis or the y axis. This is what I am trying to resolve- for example:</div></div></div></blockquote><div><br></div><div>You mean that you have a DMDA, and you change the pattern of division across processes? This will change the global numbers of given vertices. Remember that these are not</div><div>"natural numbers", meaning a lexicographic ordering of vertices, but rather "global numbers", meaning the PETSc ordering of unknowns which is contiguous across processes. There</div><div>is a discussion of this in the manual.</div><div><br></div><div> Thanks,</div><div><br></div><div> Matt</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<div style="margin:0px;font-size:12pt;font-family:Calibri,Arial,Helvetica,sans-serif;color:black;background-color:rgb(255,255,255)">
the<span> </span><b>FORTRAN</b><span> </span>code I use is basically:</div>
<div style="margin:0px;font-size:12pt;font-family:Calibri,Arial,Helvetica,sans-serif;color:black;background-color:rgb(255,255,255)">
PetscInt :: idx(ngx*ngy*ngz)<br>
</div>
<div style="margin:0px;font-size:12pt;font-family:Calibri,Arial,Helvetica,sans-serif;color:black;background-color:rgb(255,255,255)">
Mat :: D2_t, lapl, , lapl_t</div>
<div style="margin:0px;font-size:12pt;font-family:Calibri,Arial,Helvetica,sans-serif;color:black;background-color:rgb(255,255,255)">
IS :: irow, irow1, icol1<br>
<br>
</div>
<div style="margin:0px;font-size:12pt;font-family:Calibri,Arial,Helvetica,sans-serif;color:black;background-color:rgb(255,255,255)">
<div style="margin:0px;color:black">call DMCreateMatrix(grid_pressure, lapl, ierr)
<div style="margin:0px"><span style="margin:0px">call MatSetOption(lapl_t, MAT_NEW_NONZERO_LOCATIONS,PETSC_TRUE,ierr)</span><br>
</div>
call MatSetOption(lapl_t, MAT_IGNORE_ZERO_ENTRIES,PETSC_TRUE,ierr) </div>
<div style="margin:0px;color:black"><span style="margin:0px;background-color:white">call DMCreateMatrix(grid_pressure, lapl_t, ierr)</span>
<div style="margin:0px;background-color:white"><span style="margin:0px">call MatSetOption(lapl_t, MAT_NEW_NONZERO_LOCATIONS,PETSC_TRUE,ierr)</span><br>
</div>
<span style="margin:0px;background-color:white">call MatSetOption(lapl_t, MAT_IGNORE_ZERO_ENTRIES,PETSC_TRUE,ierr)<span style="margin:0px"> </span></span><br>
</div>
<div style="margin:0px;color:black">.</div>
<div style="margin:0px;color:black">.</div>
<div style="margin:0px;color:black">. !(Assembling matrix lapl)<br>
</div>
<div style="margin:0px;color:black"><br>
</div>
<div style="margin:0px;color:black">call VecCreate(petsc_xyzcom, vec_b, ierr)
<div style="margin:0px">call VecSetSizes(vec_b, PETSC_DECIDE, ngx*ngy*ngz-1, ierr)</div>
<div style="margin:0px">call VecSetFromOptions(vec_b, ierr)</div>
<div style="margin:0px">call VecSetUp(vec_b, ierr)</div>
<div style="margin:0px">call VecGetLocalSize(vec_b,vecsx,ierr)</div>
<div style="margin:0px"><span style="margin:0px">call VecGetOwnershipRange(vec_b,veclo,vechi,ierr)</span><br>
</div>
<div style="margin:0px"><span style="margin:0px">call ISCreateStride(petsc_xyzcom, vecsx,veclo,1,icol1,ierr)</span><br>
</div>
</div>
<div style="margin:0px;color:black">idx = (/ (j, j=veclo,vechi-1)/)<br>
call ISCreateGeneral(petsc_xyzcom, vecsx,idx, PETSC_COPY_VALUES,irow,ierr)<br>
call MatTranspose(lapl, MAT_INPLACE_MATRIX, lapl_t,ierr) !transpose the laplacian</div>
<div style="margin:0px;color:black"><span style="margin:0px;background-color:white;color:black">call MatGetSubMatrix(lapl_t, irow, icol1, MAT_INITIAL_MATRIX, D2_t,ierr)</span><br>
<div style="margin:0px">call MatView(lapl_t, PETSC_VIEWER_STDOUT_WORLD, ierr) </div>
</div>
<div style="margin:0px;color:black">
<div style="margin:0px;background-color:white">call MatView(D2_t, PETSC_VIEWER_STDOUT_WORLD, ierr) </div>
</div>
<br>
</div>
<div style="margin:0px;font-size:12pt;font-family:Calibri,Arial,Helvetica,sans-serif;color:black;background-color:rgb(255,255,255)">
<b>Output</b>:</div>
<div style="margin:0px;font-size:12pt;font-family:Calibri,Arial,Helvetica,sans-serif;color:black;background-color:rgb(255,255,255)">
<br>
</div>
<div style="margin:0px;font-size:12pt;font-family:Calibri,Arial,Helvetica,sans-serif;color:black;background-color:rgb(255,255,255)">
<span style="margin:0px;background-color:white;color:black">ngx=ngy=4; ngz=1; such that n=4*4*1=16</span></div>
<div style="margin:0px;font-size:12pt;font-family:Calibri,Arial,Helvetica,sans-serif;color:black;background-color:rgb(255,255,255)">
<br>
</div>
<div style="margin:0px;font-size:12pt;font-family:Calibri,Arial,Helvetica,sans-serif;color:black;background-color:rgb(255,255,255)">
<span style="margin:0px;background-color:white;color:black">
<div style="margin:0px;color:black">lapl_t:</div>
<div style="margin:0px;color:black">row 0: (0, -3.)
<div style="margin:0px">row 1: (0, 3.) (1, -3.) (5, -9.)</div>
<div style="margin:0px">row 2: (2, -3.) (3, -3.) (6, -9.)</div>
<div style="margin:0px">row 3: (3, 3.)</div>
<div style="margin:0px">row 4: (4, -3.) (5, -9.)</div>
<div style="margin:0px">row 5: (1, 3.) (4, 3.) (5, 36.) (6, -9.) (9, -9.)</div>
<div style="margin:0px">row 6: (2, 3.) (5, -9.) (6, 36.) (7, -3.) (10, -9.)</div>
<div style="margin:0px">row 7: (6, -9.) (7, 3.)</div>
<div style="margin:0px">row 8: (8, -3.) (9, -9.)</div>
<div style="margin:0px">row 9: (5, -9.) (8, 3.) (9, 36.) (10, -9.) (13, -3.)</div>
<div style="margin:0px">row 10: (6, -9.) (9, -9.) (10, 36.) (11, -3.) (14, -3.)</div>
<div style="margin:0px">row 11: (10, -9.) (11, 3.)</div>
<div style="margin:0px">row 12: (12, -3.)</div>
<div style="margin:0px">row 13: (9, -9.) (12, 3.) (13, 3.)</div>
<div style="margin:0px">row 14: (10, -9.) (14, 3.) (15, -3.)</div>
<div style="margin:0px">row 15: (15, 3.)</div>
</div>
<br>
</span></div>
<div style="margin:0px;font-size:12pt;font-family:Calibri,Arial,Helvetica,sans-serif;color:black;background-color:rgb(255,255,255)">
<span style="margin:0px;background-color:white;color:black"><br>
</span></div>
<div style="margin:0px;font-size:12pt;font-family:Calibri,Arial,Helvetica,sans-serif;color:black;background-color:rgb(255,255,255)">
<b>Case 1:</b></div>
<div style="margin:0px;font-size:12pt;font-family:Calibri,Arial,Helvetica,sans-serif;color:black;background-color:rgb(255,255,255)">
nprocx =1; nprocy=2; ! number of processors in x and y among which to divide the domain</div>
<div style="margin:0px;font-size:12pt;font-family:Calibri,Arial,Helvetica,sans-serif;color:black;background-color:rgb(255,255,255)">
<div style="margin:0px"></div>
<br>
Here, the (n-1 x n-1) submatrix is extracted correctly as</div>
<div style="margin:0px;font-size:12pt;font-family:Calibri,Arial,Helvetica,sans-serif;color:black;background-color:rgb(255,255,255)">
D2_t:</div>
<div style="margin:0px;font-size:12pt;font-family:Calibri,Arial,Helvetica,sans-serif;color:black;background-color:rgb(255,255,255)">
row 0: (0, -3.)
<div style="margin:0px">row 1: (0, 3.) (1, -3.) (5, -9.)</div>
<div style="margin:0px">row 2: (2, -3.) (3, -3.) (6, -9.)</div>
<div style="margin:0px">row 3: (3, 3.)</div>
<div style="margin:0px">row 4: (4, -3.) (5, -9.)</div>
<div style="margin:0px">row 5: (1, 3.) (4, 3.) (5, 36.) (6, -9.) (9, -9.)</div>
<div style="margin:0px">row 6: (2, 3.) (5, -9.) (6, 36.) (7, -3.) (10, -9.)</div>
<div style="margin:0px">row 7: (6, -9.) (7, 3.)</div>
<div style="margin:0px">row 8: (8, -3.) (9, -9.)</div>
<div style="margin:0px">row 9: (5, -9.) (8, 3.) (9, 36.) (10, -9.) (13, -3.)</div>
<div style="margin:0px">row 10: (6, -9.) (9, -9.) (10, 36.) (11, -3.) (14, -3.)</div>
<div style="margin:0px">row 11: (10, -9.) (11, 3.)</div>
<div style="margin:0px">row 12: (12, -3.)</div>
<div style="margin:0px">row 13: (9, -9.) (12, 3.) (13, 3.)</div>
<div style="margin:0px">row 14: (10, -9.) (14, 3.)</div>
<br>
However, for </div>
<div style="margin:0px;font-size:12pt;font-family:Calibri,Arial,Helvetica,sans-serif;color:black;background-color:rgb(255,255,255)">
<div style="margin:0px;color:black"><b>Case 2:</b></div>
</div>
<div style="margin:0px;font-size:12pt;font-family:Calibri,Arial,Helvetica,sans-serif;color:black;background-color:rgb(255,255,255)">
<div style="margin:0px;color:black">nprocx =2; nprocy=1; </div>
<div style="margin:0px;color:black"><br>
</div>
lapl_t is correctly assembled and transposed but the <span style="margin:0px;background-color:white;color:black">(n-1 x n-1) submatrix is extracted incorrectly as</span></div>
<div style="margin:0px;font-size:12pt;font-family:Calibri,Arial,Helvetica,sans-serif;color:black;background-color:rgb(255,255,255)">
<span style="margin:0px;background-color:white;color:black">D2_t:</span></div>
<div style="margin:0px;font-size:12pt;font-family:Calibri,Arial,Helvetica,sans-serif;color:black;background-color:rgb(255,255,255)">
<span style="margin:0px;background-color:white;color:black">row 0: (0, -3.)
<div style="margin:0px">row 1: (0, 3.) (1, -3.) (3, -9.)</div>
<div style="margin:0px">row 2: (2, -3.) (3, -9.)</div>
<div style="margin:0px">row 3: (1, 3.) (2, 3.) (3, 36.) (5, -9.) (10, -9.)</div>
<div style="margin:0px">row 4: (4, -3.) (5, -9.)</div>
<div style="margin:0px">row 5: (3, -9.) (4, 3.) (5, 36.) (7, -3.) (12, -9.)</div>
<div style="margin:0px">row 6: (6, -3.)</div>
<div style="margin:0px">row 7: (5, -9.) (6, 3.) (7, 3.)</div>
<div style="margin:0px">row 8: (8, -3.) (9, -3.) (10, -9.)</div>
<div style="margin:0px">row 9: (9, 3.)</div>
<div style="margin:0px">row 10: (3, -9.) (8, 3.) (10, 36.) (11, -3.) (12, -9.)</div>
<div style="margin:0px">row 11: (10, -9.) (11, 3.)</div>
<div style="margin:0px">row 12: (5, -9.) (10, -9.) (12, 36.) (13, -3.) (14, -3.)</div>
<div style="margin:0px">row 13: (12, -9.) (13, 3.)</div>
<div style="margin:0px">row 14: (12, -9.) (14, 3.)</div>
<br>
I am unable to understand why the extracted submatrix is incorrect when nprocx>1 but works when nprocx=1 and nprocy>=1.</span></div>
<div style="margin:0px;font-size:12pt;font-family:Calibri,Arial,Helvetica,sans-serif;color:black;background-color:rgb(255,255,255)">
<span style="margin:0px;background-color:white;color:black"><br>
</span></div>
<div style="margin:0px;font-size:12pt;font-family:Calibri,Arial,Helvetica,sans-serif;color:black;background-color:rgb(255,255,255)">
<span style="margin:0px;background-color:white;color:black">P.S. the parallel IS in cases 1 and 2 are the same and are as follows:</span></div>
<div style="margin:0px;font-size:12pt;font-family:Calibri,Arial,Helvetica,sans-serif;color:black;background-color:rgb(255,255,255)">
<span style="margin:0px;background-color:white;color:black">irow:</span></div>
<div style="margin:0px;font-size:12pt;font-family:Calibri,Arial,Helvetica,sans-serif;color:black;background-color:rgb(255,255,255)">
<span style="margin:0px;background-color:white;color:black">[0] Number of indices in set 8
<div style="margin:0px">[0] 0 0</div>
<div style="margin:0px">[0] 1 1</div>
<div style="margin:0px">[0] 2 2</div>
<div style="margin:0px">[0] 3 3</div>
<div style="margin:0px">[0] 4 4</div>
<div style="margin:0px">[0] 5 5</div>
<div style="margin:0px">[0] 6 6</div>
<div style="margin:0px">[0] 7 7</div>
<div style="margin:0px">[1] Number of indices in set 7</div>
<div style="margin:0px">[1] 0 8</div>
<div style="margin:0px">[1] 1 9</div>
<div style="margin:0px">[1] 2 10</div>
<div style="margin:0px">[1] 3 11</div>
<div style="margin:0px">[1] 4 12</div>
<div style="margin:0px">[1] 5 13</div>
<div style="margin:0px">[1] 6 14</div>
<br>
icol1:</span></div>
<div style="margin:0px;font-size:12pt;font-family:Calibri,Arial,Helvetica,sans-serif;color:black;background-color:rgb(255,255,255)">
<span style="margin:0px;background-color:white;color:black">[0] Index set is permutation
<div style="margin:0px">[0] Number of indices in (stride) set 8</div>
<div style="margin:0px">[0] 0 0</div>
<div style="margin:0px">[0] 1 1</div>
<div style="margin:0px">[0] 2 2</div>
<div style="margin:0px">[0] 3 3</div>
<div style="margin:0px">[0] 4 4</div>
<div style="margin:0px">[0] 5 5</div>
<div style="margin:0px">[0] 6 6</div>
<div style="margin:0px">[0] 7 7</div>
<div style="margin:0px">[1] Number of indices in (stride) set 7</div>
<div style="margin:0px">[1] 0 8</div>
<div style="margin:0px">[1] 1 9</div>
<div style="margin:0px">[1] 2 10</div>
<div style="margin:0px">[1] 3 11</div>
<div style="margin:0px">[1] 4 12</div>
<div style="margin:0px">[1] 5 13</div>
<div style="margin:0px">[1] 6 14</div>
<br>
Could you please help me find out what is going wrong here?</span></div>
<div style="margin:0px;font-size:12pt;font-family:Calibri,Arial,Helvetica,sans-serif;color:black;background-color:rgb(255,255,255)">
<br>
</div>
<div style="margin:0px;font-size:12pt;font-family:Calibri,Arial,Helvetica,sans-serif;color:black;background-color:rgb(255,255,255)">
Regards,</div>
<div style="margin:0px;font-size:15px;color:rgb(0,0,0);background-color:rgb(255,255,255)">
<div style="margin:0px">
<div style="margin:0px;font-size:13px;font-family:Tahoma"><span style="margin:0px;font-family:sans-serif;color:rgb(34,34,34);background-color:white">Gautham</span> </div>
</div>
</div>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<div id="gmail-m_3104415965768461056Signature">
<div></div>
</div>
<div id="gmail-m_3104415965768461056appendonsend"></div>
<hr style="display:inline-block;width:98%">
<div id="gmail-m_3104415965768461056divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" style="font-size:11pt" color="#000000"><b>From:</b> Krishnan, Gautham <<a href="mailto:gautham3@illinois.edu" target="_blank">gautham3@illinois.edu</a>><br>
<b>Sent:</b> Saturday, September 26, 2020 2:01 PM<br>
<b>To:</b> Matthew Knepley <<a href="mailto:knepley@gmail.com" target="_blank">knepley@gmail.com</a>><br>
<b>Cc:</b> <a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a> <<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a>><br>
<b>Subject:</b> Re: [petsc-users] Regarding help with MatGetSubMatrix parallel use</font>
<div> </div>
</div>
<div dir="ltr">
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
In parallel I have passed the global indices that I want on the PE. However, the result I obtain seems to change based on whether I divide the initial domain between PEs along the x axis or the y axis. This is what I am trying to resolve- for example:</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
the <b>FORTRAN</b> code I use is basically:</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
PetscInt :: idx(ngx*ngy*ngz)<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
Mat :: D2_t, lapl, , lapl_t</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
IS :: irow, irow1, icol1<br>
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<div style="margin:0px;font-size:12pt;font-family:Calibri,Arial,Helvetica,sans-serif;color:rgb(0,0,0)">
call DMCreateMatrix(grid_pressure, lapl, ierr)
<div style="margin:0px"><span style="margin:0px">call MatSetOption(lapl_t, MAT_NEW_NONZERO_LOCATIONS,PETSC_TRUE,ierr)</span><br>
</div>
call MatSetOption(lapl_t, MAT_IGNORE_ZERO_ENTRIES,PETSC_TRUE,ierr) </div>
<div style="margin:0px;font-size:12pt;font-family:Calibri,Arial,Helvetica,sans-serif;color:rgb(0,0,0)">
<span style="margin:0px;background-color:rgb(255,255,255);display:inline">call DMCreateMatrix(grid_pressure, lapl_t, ierr)</span>
<div style="margin:0px;background-color:rgb(255,255,255)"><span style="margin:0px">call MatSetOption(lapl_t, MAT_NEW_NONZERO_LOCATIONS,PETSC_TRUE,ierr)</span><br>
</div>
<span style="margin:0px;background-color:rgb(255,255,255);display:inline">call MatSetOption(lapl_t, MAT_IGNORE_ZERO_ENTRIES,PETSC_TRUE,ierr)<span style="margin:0px"> </span></span><br>
</div>
<div style="margin:0px;font-size:12pt;font-family:Calibri,Arial,Helvetica,sans-serif;color:rgb(0,0,0)">
.</div>
<div style="margin:0px;font-size:12pt;font-family:Calibri,Arial,Helvetica,sans-serif;color:rgb(0,0,0)">
.</div>
<div style="margin:0px;font-size:12pt;font-family:Calibri,Arial,Helvetica,sans-serif;color:rgb(0,0,0)">
. !(Assembling matrix lapl)<br>
</div>
<div style="margin:0px;font-size:12pt;font-family:Calibri,Arial,Helvetica,sans-serif;color:rgb(0,0,0)">
<br>
</div>
<div style="margin:0px;font-size:12pt;font-family:Calibri,Arial,Helvetica,sans-serif;color:rgb(0,0,0)">
call VecCreate(petsc_xyzcom, vec_b, ierr)
<div style="margin:0px">call VecSetSizes(vec_b, PETSC_DECIDE, ngx*ngy*ngz-1, ierr)</div>
<div style="margin:0px">call VecSetFromOptions(vec_b, ierr)</div>
<div style="margin:0px">call VecSetUp(vec_b, ierr)</div>
<div style="margin:0px">call VecGetLocalSize(vec_b,vecsx,ierr)</div>
<div style="margin:0px"><span style="margin:0px">call VecGetOwnershipRange(vec_b,veclo,vechi,ierr)</span><br>
</div>
<div style="margin:0px"><span style="margin:0px">call ISCreateStride(petsc_xyzcom, vecsx,veclo,1,icol1,ierr)</span><br>
</div>
</div>
<div style="margin:0px;font-size:12pt;font-family:Calibri,Arial,Helvetica,sans-serif;color:rgb(0,0,0)">
idx = (/ (j, j=veclo,vechi-1)/)<br>
call ISCreateGeneral(petsc_xyzcom, vecsx,idx, PETSC_COPY_VALUES,irow,ierr)<br>
call MatTranspose(lapl, MAT_INPLACE_MATRIX, lapl_t,ierr) !transpose the laplacian</div>
<div style="margin:0px;font-size:12pt;font-family:Calibri,Arial,Helvetica,sans-serif;color:rgb(0,0,0)">
<span style="color:rgb(0,0,0);font-family:Calibri,Arial,Helvetica,sans-serif;background-color:rgb(255,255,255);display:inline">call MatGetSubMatrix(lapl_t, irow, icol1, MAT_INITIAL_MATRIX, D2_t,ierr)</span><br>
<div style="margin:0px">call MatView(lapl_t, PETSC_VIEWER_STDOUT_WORLD, ierr) </div>
</div>
<div style="margin:0px;font-size:12pt;font-family:Calibri,Arial,Helvetica,sans-serif;color:rgb(0,0,0)">
<div style="margin:0px;background-color:rgb(255,255,255)">call MatView(D2_t, PETSC_VIEWER_STDOUT_WORLD, ierr) </div>
</div>
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<b>Output</b>:</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<span style="color:rgb(0,0,0);font-family:Calibri,Arial,Helvetica,sans-serif;background-color:rgb(255,255,255);display:inline">ngx=ngy=4; ngz=1; such that n=4*4*1=16</span></div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<span style="color:rgb(0,0,0);font-family:Calibri,Arial,Helvetica,sans-serif;background-color:rgb(255,255,255);display:inline">
<div style="margin:0px;font-size:12pt;font-family:Calibri,Arial,Helvetica,sans-serif;color:rgb(0,0,0)">
lapl_t:</div>
<div style="margin:0px;font-size:12pt;font-family:Calibri,Arial,Helvetica,sans-serif;color:rgb(0,0,0)">
row 0: (0, -3.)
<div style="margin:0px">row 1: (0, 3.) (1, -3.) (5, -9.)</div>
<div style="margin:0px">row 2: (2, -3.) (3, -3.) (6, -9.)</div>
<div style="margin:0px">row 3: (3, 3.)</div>
<div style="margin:0px">row 4: (4, -3.) (5, -9.)</div>
<div style="margin:0px">row 5: (1, 3.) (4, 3.) (5, 36.) (6, -9.) (9, -9.)</div>
<div style="margin:0px">row 6: (2, 3.) (5, -9.) (6, 36.) (7, -3.) (10, -9.)</div>
<div style="margin:0px">row 7: (6, -9.) (7, 3.)</div>
<div style="margin:0px">row 8: (8, -3.) (9, -9.)</div>
<div style="margin:0px">row 9: (5, -9.) (8, 3.) (9, 36.) (10, -9.) (13, -3.)</div>
<div style="margin:0px">row 10: (6, -9.) (9, -9.) (10, 36.) (11, -3.) (14, -3.)</div>
<div style="margin:0px">row 11: (10, -9.) (11, 3.)</div>
<div style="margin:0px">row 12: (12, -3.)</div>
<div style="margin:0px">row 13: (9, -9.) (12, 3.) (13, 3.)</div>
<div style="margin:0px">row 14: (10, -9.) (14, 3.) (15, -3.)</div>
<div style="margin:0px">row 15: (15, 3.)</div>
</div>
<br>
</span></div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<span style="color:rgb(0,0,0);font-family:Calibri,Arial,Helvetica,sans-serif;background-color:rgb(255,255,255);display:inline"><br>
</span></div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<b>Case 1:</b></div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
nprocx =1; nprocy=2; ! number of processors in x and y among which to divide the domain</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<div></div>
<br>
Here, the (n-1 x n-1) submatrix is extracted correctly as</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
D2_t:</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
row 0: (0, -3.)
<div>row 1: (0, 3.) (1, -3.) (5, -9.) </div>
<div>row 2: (2, -3.) (3, -3.) (6, -9.) </div>
<div>row 3: (3, 3.) </div>
<div>row 4: (4, -3.) (5, -9.) </div>
<div>row 5: (1, 3.) (4, 3.) (5, 36.) (6, -9.) (9, -9.) </div>
<div>row 6: (2, 3.) (5, -9.) (6, 36.) (7, -3.) (10, -9.) </div>
<div>row 7: (6, -9.) (7, 3.) </div>
<div>row 8: (8, -3.) (9, -9.) </div>
<div>row 9: (5, -9.) (8, 3.) (9, 36.) (10, -9.) (13, -3.) </div>
<div>row 10: (6, -9.) (9, -9.) (10, 36.) (11, -3.) (14, -3.) </div>
<div>row 11: (10, -9.) (11, 3.) </div>
<div>row 12: (12, -3.) </div>
<div>row 13: (9, -9.) (12, 3.) (13, 3.) </div>
<div>row 14: (10, -9.) (14, 3.) </div>
<br>
However, for </div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<div style="margin:0px;font-size:12pt;font-family:Calibri,Arial,Helvetica,sans-serif;color:rgb(0,0,0)">
<b>Case 2:</b></div>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<div style="margin:0px;font-size:12pt;font-family:Calibri,Arial,Helvetica,sans-serif;color:rgb(0,0,0)">
nprocx =2; nprocy=1; </div>
<div style="margin:0px;font-size:12pt;font-family:Calibri,Arial,Helvetica,sans-serif;color:rgb(0,0,0)">
<br style="font-size:16px;background-color:rgb(255,255,255)">
</div>
lapl_t is correctly assembled and transposed but the <span style="color:rgb(0,0,0);font-family:Calibri,Arial,Helvetica,sans-serif;background-color:rgb(255,255,255);display:inline">(n-1 x n-1) submatrix is extracted incorrectly as</span></div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<span style="color:rgb(0,0,0);font-family:Calibri,Arial,Helvetica,sans-serif;background-color:rgb(255,255,255);display:inline">D2_t:</span></div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<span style="color:rgb(0,0,0);font-family:Calibri,Arial,Helvetica,sans-serif;background-color:rgb(255,255,255);display:inline">row 0: (0, -3.)
<div>row 1: (0, 3.) (1, -3.) (3, -9.) </div>
<div>row 2: (2, -3.) (3, -9.) </div>
<div>row 3: (1, 3.) (2, 3.) (3, 36.) (5, -9.) (10, -9.) </div>
<div>row 4: (4, -3.) (5, -9.) </div>
<div>row 5: (3, -9.) (4, 3.) (5, 36.) (7, -3.) (12, -9.) </div>
<div>row 6: (6, -3.) </div>
<div>row 7: (5, -9.) (6, 3.) (7, 3.) </div>
<div>row 8: (8, -3.) (9, -3.) (10, -9.) </div>
<div>row 9: (9, 3.) </div>
<div>row 10: (3, -9.) (8, 3.) (10, 36.) (11, -3.) (12, -9.) </div>
<div>row 11: (10, -9.) (11, 3.) </div>
<div>row 12: (5, -9.) (10, -9.) (12, 36.) (13, -3.) (14, -3.) </div>
<div>row 13: (12, -9.) (13, 3.) </div>
<div>row 14: (12, -9.) (14, 3.) </div>
<br>
I am unable to understand why the extracted submatrix is incorrect when nprocx>1 but works when nprocx=1 and nprocy>=1.</span></div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<span style="color:rgb(0,0,0);font-family:Calibri,Arial,Helvetica,sans-serif;background-color:rgb(255,255,255);display:inline"><br>
</span></div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<span style="color:rgb(0,0,0);font-family:Calibri,Arial,Helvetica,sans-serif;background-color:rgb(255,255,255);display:inline">P.S. the parallel IS in cases 1 and 2 are the same and are as follows:</span></div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<span style="color:rgb(0,0,0);font-family:Calibri,Arial,Helvetica,sans-serif;background-color:rgb(255,255,255);display:inline">irow:</span></div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<span style="color:rgb(0,0,0);font-family:Calibri,Arial,Helvetica,sans-serif;background-color:rgb(255,255,255);display:inline">[0] Number of indices in set 8
<div>[0] 0 0</div>
<div>[0] 1 1</div>
<div>[0] 2 2</div>
<div>[0] 3 3</div>
<div>[0] 4 4</div>
<div>[0] 5 5</div>
<div>[0] 6 6</div>
<div>[0] 7 7</div>
<div>[1] Number of indices in set 7</div>
<div>[1] 0 8</div>
<div>[1] 1 9</div>
<div>[1] 2 10</div>
<div>[1] 3 11</div>
<div>[1] 4 12</div>
<div>[1] 5 13</div>
<div>[1] 6 14</div>
<br>
icol1:</span></div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<span style="color:rgb(0,0,0);font-family:Calibri,Arial,Helvetica,sans-serif;background-color:rgb(255,255,255);display:inline">[0] Index set is permutation
<div>[0] Number of indices in (stride) set 8</div>
<div>[0] 0 0</div>
<div>[0] 1 1</div>
<div>[0] 2 2</div>
<div>[0] 3 3</div>
<div>[0] 4 4</div>
<div>[0] 5 5</div>
<div>[0] 6 6</div>
<div>[0] 7 7</div>
<div>[1] Number of indices in (stride) set 7</div>
<div>[1] 0 8</div>
<div>[1] 1 9</div>
<div>[1] 2 10</div>
<div>[1] 3 11</div>
<div>[1] 4 12</div>
<div>[1] 5 13</div>
<div>[1] 6 14</div>
<br>
Could you please help me find out what is going wrong here?</span></div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
Regards,</div>
<div id="gmail-m_3104415965768461056x_Signature">
<div>
<div style="font-family:Tahoma;font-size:13px"><span style="color:rgb(34,34,34);font-family:sans-serif;background-color:rgb(255,255,255)">Gautham</span> <br>
</div>
</div>
</div>
<div id="gmail-m_3104415965768461056x_appendonsend"></div>
<hr style="display:inline-block;width:98%">
<div id="gmail-m_3104415965768461056x_divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" color="#000000" style="font-size:11pt"><b>From:</b> Matthew Knepley <<a href="mailto:knepley@gmail.com" target="_blank">knepley@gmail.com</a>><br>
<b>Sent:</b> Wednesday, September 23, 2020 3:55 PM<br>
<b>To:</b> Krishnan, Gautham <<a href="mailto:gautham3@illinois.edu" target="_blank">gautham3@illinois.edu</a>><br>
<b>Cc:</b> <a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a> <<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a>><br>
<b>Subject:</b> Re: [petsc-users] Regarding help with MatGetSubMatrix parallel use</font>
<div> </div>
</div>
<div>
<div dir="ltr">
<div dir="ltr">On Wed, Sep 23, 2020 at 4:12 PM Krishnan, Gautham <<a href="mailto:gautham3@illinois.edu" target="_blank">gautham3@illinois.edu</a>> wrote:<br>
</div>
<div>
<blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="ltr">
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0);background-color:rgb(255,255,255)">
Hello,</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0);background-color:rgb(255,255,255)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0);background-color:rgb(255,255,255)">
For a CFD code being developed with FORTRAN and MPI, I am using PETSC matrices and for a particular operation, I require to extract a submatrix(n-1 x n-1) of a matrix created (n x n). However using the petsc MatGetSubMatrix works for serial runs but fails when
the domain is split up over PEs- I suspect the indexing changed for parallel runs and hence the global indexing that worked for serial case just shuffles around matrix entries in parallel undesirably. I would like to ask whether anybody could offer some guidance
regarding this. I would like to note that the 2D domain is split along both x and y axes for parallel runs on multiple PEs.</div>
</div>
</blockquote>
<div><br>
</div>
<div>In parallel, you pass MatGetSubmatrix() the global indices that you want on your process.</div>
<div><br>
</div>
<div> Thanks,</div>
<div><br>
</div>
<div> Matt</div>
<div> </div>
<blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="ltr">
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
Regards,</div>
<div id="gmail-m_3104415965768461056x_x_gmail-m_789255861923840103Signature">
<div style="font-family:Tahoma;font-size:13px"><span style="color:rgb(34,34,34);font-family:sans-serif;background-color:rgb(255,255,255)">Gautham Krishnan,</span>
<div style="color:rgb(34,34,34);font-family:sans-serif;background-color:rgb(255,255,255)">
<br>
</div>
</div>
</div>
</div>
</blockquote>
</div>
<br clear="all">
<div><br>
</div>
-- <br>
<div dir="ltr">
<div dir="ltr">
<div>
<div dir="ltr">
<div>
<div dir="ltr">
<div>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>
-- Norbert Wiener</div>
<div><br>
</div>
<div><a href="http://www.cse.buffalo.edu/~knepley/" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr" class="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>-- Norbert Wiener</div><div><br></div><div><a href="http://www.cse.buffalo.edu/~knepley/" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br></div></div></div></div></div></div></div></div>