<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} </style>
</head>
<body dir="ltr">
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
In parallel I have passed the global indices that I want on the PE. However, the result I obtain seems to change based on whether I divide the initial domain between PEs along the x axis or the y axis. This is what I am trying to resolve- for example:</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
the <b>FORTRAN</b> code I use is basically:</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
PetscInt :: idx(ngx*ngy*ngz)<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Mat :: D2_t, lapl, , lapl_t</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
IS :: irow, irow1, icol1<br>
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<div style="margin: 0px; font-size: 12pt; font-family: Calibri, Arial, Helvetica, sans-serif; color: rgb(0, 0, 0);">
call DMCreateMatrix(grid_pressure, lapl, ierr)
<div style="margin: 0px"><span style="margin: 0px">call MatSetOption(lapl_t, MAT_NEW_NONZERO_LOCATIONS,PETSC_TRUE,ierr)</span><br>
</div>
call MatSetOption(lapl_t, MAT_IGNORE_ZERO_ENTRIES,PETSC_TRUE,ierr) </div>
<div style="margin: 0px; font-size: 12pt; font-family: Calibri, Arial, Helvetica, sans-serif; color: rgb(0, 0, 0);">
<span style="margin: 0px; background-color: rgb(255, 255, 255); display: inline !important;">call DMCreateMatrix(grid_pressure, lapl_t, ierr)</span>
<div style="margin: 0px; background-color: rgb(255, 255, 255);"><span style="margin: 0px">call MatSetOption(lapl_t, MAT_NEW_NONZERO_LOCATIONS,PETSC_TRUE,ierr)</span><br>
</div>
<span style="margin: 0px; background-color: rgb(255, 255, 255); display: inline !important;">call MatSetOption(lapl_t, MAT_IGNORE_ZERO_ENTRIES,PETSC_TRUE,ierr)<span style="margin: 0px"> </span></span><br>
</div>
<div style="margin: 0px; font-size: 12pt; font-family: Calibri, Arial, Helvetica, sans-serif; color: rgb(0, 0, 0);">
.</div>
<div style="margin: 0px; font-size: 12pt; font-family: Calibri, Arial, Helvetica, sans-serif; color: rgb(0, 0, 0);">
.</div>
<div style="margin: 0px; font-size: 12pt; font-family: Calibri, Arial, Helvetica, sans-serif; color: rgb(0, 0, 0);">
. !(Assembling matrix lapl)<br>
</div>
<div style="margin: 0px; font-size: 12pt; font-family: Calibri, Arial, Helvetica, sans-serif; color: rgb(0, 0, 0);">
<br>
</div>
<div style="margin: 0px; font-size: 12pt; font-family: Calibri, Arial, Helvetica, sans-serif; color: rgb(0, 0, 0);">
call VecCreate(petsc_xyzcom, vec_b, ierr)
<div style="margin: 0px">call VecSetSizes(vec_b, PETSC_DECIDE, ngx*ngy*ngz-1, ierr)</div>
<div style="margin: 0px">call VecSetFromOptions(vec_b, ierr)</div>
<div style="margin: 0px">call VecSetUp(vec_b, ierr)</div>
<div style="margin: 0px">call VecGetLocalSize(vec_b,vecsx,ierr)</div>
<div style="margin: 0px"><span style="margin: 0px">call VecGetOwnershipRange(vec_b,veclo,vechi,ierr)</span><br>
</div>
<div style="margin: 0px"><span style="margin: 0px">call ISCreateStride(petsc_xyzcom, vecsx,veclo,1,icol1,ierr)</span><br>
</div>
</div>
<div style="margin: 0px; font-size: 12pt; font-family: Calibri, Arial, Helvetica, sans-serif; color: rgb(0, 0, 0);">
idx = (/ (j, j=veclo,vechi-1)/)<br>
call ISCreateGeneral(petsc_xyzcom, vecsx,idx, PETSC_COPY_VALUES,irow,ierr)<br>
call MatTranspose(lapl, MAT_INPLACE_MATRIX, lapl_t,ierr) !transpose the laplacian</div>
<div style="margin: 0px; font-size: 12pt; font-family: Calibri, Arial, Helvetica, sans-serif; color: rgb(0, 0, 0);">
<span style="color: rgb(0, 0, 0); font-family: Calibri, Arial, Helvetica, sans-serif; background-color: rgb(255, 255, 255); display: inline !important;">call MatGetSubMatrix(lapl_t, irow, icol1, MAT_INITIAL_MATRIX, D2_t,ierr)</span><br>
<div style="margin: 0px">call MatView(lapl_t, PETSC_VIEWER_STDOUT_WORLD, ierr) </div>
</div>
<div style="margin: 0px; font-size: 12pt; font-family: Calibri, Arial, Helvetica, sans-serif; color: rgb(0, 0, 0);">
<div style="margin: 0px; background-color: rgb(255, 255, 255);">call MatView(D2_t, PETSC_VIEWER_STDOUT_WORLD, ierr) </div>
</div>
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<b>Output</b>:</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<span style="color: rgb(0, 0, 0); font-family: Calibri, Arial, Helvetica, sans-serif; background-color: rgb(255, 255, 255); display: inline !important;">ngx=ngy=4; ngz=1; such that n=4*4*1=16</span></div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<span style="color: rgb(0, 0, 0); font-family: Calibri, Arial, Helvetica, sans-serif; background-color: rgb(255, 255, 255); display: inline !important;">
<div style="margin: 0px; font-size: 12pt; font-family: Calibri, Arial, Helvetica, sans-serif; color: rgb(0, 0, 0);">
lapl_t:</div>
<div style="margin: 0px; font-size: 12pt; font-family: Calibri, Arial, Helvetica, sans-serif; color: rgb(0, 0, 0);">
row 0: (0, -3.)
<div style="margin: 0px">row 1: (0, 3.) (1, -3.) (5, -9.)</div>
<div style="margin: 0px">row 2: (2, -3.) (3, -3.) (6, -9.)</div>
<div style="margin: 0px">row 3: (3, 3.)</div>
<div style="margin: 0px">row 4: (4, -3.) (5, -9.)</div>
<div style="margin: 0px">row 5: (1, 3.) (4, 3.) (5, 36.) (6, -9.) (9, -9.)</div>
<div style="margin: 0px">row 6: (2, 3.) (5, -9.) (6, 36.) (7, -3.) (10, -9.)</div>
<div style="margin: 0px">row 7: (6, -9.) (7, 3.)</div>
<div style="margin: 0px">row 8: (8, -3.) (9, -9.)</div>
<div style="margin: 0px">row 9: (5, -9.) (8, 3.) (9, 36.) (10, -9.) (13, -3.)</div>
<div style="margin: 0px">row 10: (6, -9.) (9, -9.) (10, 36.) (11, -3.) (14, -3.)</div>
<div style="margin: 0px">row 11: (10, -9.) (11, 3.)</div>
<div style="margin: 0px">row 12: (12, -3.)</div>
<div style="margin: 0px">row 13: (9, -9.) (12, 3.) (13, 3.)</div>
<div style="margin: 0px">row 14: (10, -9.) (14, 3.) (15, -3.)</div>
<div style="margin: 0px">row 15: (15, 3.)</div>
</div>
<br>
</span></div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<span style="color: rgb(0, 0, 0); font-family: Calibri, Arial, Helvetica, sans-serif; background-color: rgb(255, 255, 255); display: inline !important;"><br>
</span></div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<b>Case 1:</b></div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
nprocx =1; nprocy=2; ! number of processors in x and y among which to divide the domain</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<div></div>
<br>
Here, the (n-1 x n-1) submatrix is extracted correctly as</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
D2_t:</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
row 0: (0, -3.)
<div>row 1: (0, 3.) (1, -3.) (5, -9.) </div>
<div>row 2: (2, -3.) (3, -3.) (6, -9.) </div>
<div>row 3: (3, 3.) </div>
<div>row 4: (4, -3.) (5, -9.) </div>
<div>row 5: (1, 3.) (4, 3.) (5, 36.) (6, -9.) (9, -9.) </div>
<div>row 6: (2, 3.) (5, -9.) (6, 36.) (7, -3.) (10, -9.) </div>
<div>row 7: (6, -9.) (7, 3.) </div>
<div>row 8: (8, -3.) (9, -9.) </div>
<div>row 9: (5, -9.) (8, 3.) (9, 36.) (10, -9.) (13, -3.) </div>
<div>row 10: (6, -9.) (9, -9.) (10, 36.) (11, -3.) (14, -3.) </div>
<div>row 11: (10, -9.) (11, 3.) </div>
<div>row 12: (12, -3.) </div>
<div>row 13: (9, -9.) (12, 3.) (13, 3.) </div>
<div>row 14: (10, -9.) (14, 3.) </div>
<br>
However, for </div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<div style="margin: 0px; font-size: 12pt; font-family: Calibri, Arial, Helvetica, sans-serif; color: rgb(0, 0, 0);">
<b>Case 2:</b></div>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<div style="margin: 0px; font-size: 12pt; font-family: Calibri, Arial, Helvetica, sans-serif; color: rgb(0, 0, 0);">
nprocx =2; nprocy=1; </div>
<div style="margin: 0px; font-size: 12pt; font-family: Calibri, Arial, Helvetica, sans-serif; color: rgb(0, 0, 0);">
<br style="font-size: 16px; background-color: rgb(255, 255, 255);">
</div>
lapl_t is correctly assembled and transposed but the <span style="color: rgb(0, 0, 0); font-family: Calibri, Arial, Helvetica, sans-serif; background-color: rgb(255, 255, 255); display: inline !important;">(n-1 x n-1) submatrix is extracted incorrectly as</span></div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<span style="color: rgb(0, 0, 0); font-family: Calibri, Arial, Helvetica, sans-serif; background-color: rgb(255, 255, 255); display: inline !important;">D2_t:</span></div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<span style="color: rgb(0, 0, 0); font-family: Calibri, Arial, Helvetica, sans-serif; background-color: rgb(255, 255, 255); display: inline !important;">row 0: (0, -3.)
<div>row 1: (0, 3.) (1, -3.) (3, -9.) </div>
<div>row 2: (2, -3.) (3, -9.) </div>
<div>row 3: (1, 3.) (2, 3.) (3, 36.) (5, -9.) (10, -9.) </div>
<div>row 4: (4, -3.) (5, -9.) </div>
<div>row 5: (3, -9.) (4, 3.) (5, 36.) (7, -3.) (12, -9.) </div>
<div>row 6: (6, -3.) </div>
<div>row 7: (5, -9.) (6, 3.) (7, 3.) </div>
<div>row 8: (8, -3.) (9, -3.) (10, -9.) </div>
<div>row 9: (9, 3.) </div>
<div>row 10: (3, -9.) (8, 3.) (10, 36.) (11, -3.) (12, -9.) </div>
<div>row 11: (10, -9.) (11, 3.) </div>
<div>row 12: (5, -9.) (10, -9.) (12, 36.) (13, -3.) (14, -3.) </div>
<div>row 13: (12, -9.) (13, 3.) </div>
<div>row 14: (12, -9.) (14, 3.) </div>
<br>
I am unable to understand why the extracted submatrix is incorrect when nprocx>1 but works when nprocx=1 and nprocy>=1.</span></div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<span style="color: rgb(0, 0, 0); font-family: Calibri, Arial, Helvetica, sans-serif; background-color: rgb(255, 255, 255); display: inline !important;"><br>
</span></div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<span style="color: rgb(0, 0, 0); font-family: Calibri, Arial, Helvetica, sans-serif; background-color: rgb(255, 255, 255); display: inline !important;">P.S. the parallel IS in cases 1 and 2 are the same and are as follows:</span></div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<span style="color: rgb(0, 0, 0); font-family: Calibri, Arial, Helvetica, sans-serif; background-color: rgb(255, 255, 255); display: inline !important;">irow:</span></div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<span style="color: rgb(0, 0, 0); font-family: Calibri, Arial, Helvetica, sans-serif; background-color: rgb(255, 255, 255); display: inline !important;">[0] Number of indices in set 8
<div>[0] 0 0</div>
<div>[0] 1 1</div>
<div>[0] 2 2</div>
<div>[0] 3 3</div>
<div>[0] 4 4</div>
<div>[0] 5 5</div>
<div>[0] 6 6</div>
<div>[0] 7 7</div>
<div>[1] Number of indices in set 7</div>
<div>[1] 0 8</div>
<div>[1] 1 9</div>
<div>[1] 2 10</div>
<div>[1] 3 11</div>
<div>[1] 4 12</div>
<div>[1] 5 13</div>
<div>[1] 6 14</div>
<br>
icol1:</span></div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<span style="color: rgb(0, 0, 0); font-family: Calibri, Arial, Helvetica, sans-serif; background-color: rgb(255, 255, 255); display: inline !important;">[0] Index set is permutation
<div>[0] Number of indices in (stride) set 8</div>
<div>[0] 0 0</div>
<div>[0] 1 1</div>
<div>[0] 2 2</div>
<div>[0] 3 3</div>
<div>[0] 4 4</div>
<div>[0] 5 5</div>
<div>[0] 6 6</div>
<div>[0] 7 7</div>
<div>[1] Number of indices in (stride) set 7</div>
<div>[1] 0 8</div>
<div>[1] 1 9</div>
<div>[1] 2 10</div>
<div>[1] 3 11</div>
<div>[1] 4 12</div>
<div>[1] 5 13</div>
<div>[1] 6 14</div>
<br>
Could you please help me find out what is going wrong here?</span></div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Regards,</div>
<div id="Signature">
<div>
<div style="font-family:Tahoma; font-size:13px"><span style="color: rgb(34, 34, 34); font-family: sans-serif; background-color: rgb(255, 255, 255);">Gautham</span> <br>
</div>
</div>
</div>
<div id="appendonsend"></div>
<hr style="display:inline-block;width:98%" tabindex="-1">
<div id="divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" style="font-size:11pt" color="#000000"><b>From:</b> Matthew Knepley <knepley@gmail.com><br>
<b>Sent:</b> Wednesday, September 23, 2020 3:55 PM<br>
<b>To:</b> Krishnan, Gautham <gautham3@illinois.edu><br>
<b>Cc:</b> petsc-users@mcs.anl.gov <petsc-users@mcs.anl.gov><br>
<b>Subject:</b> Re: [petsc-users] Regarding help with MatGetSubMatrix parallel use</font>
<div> </div>
</div>
<div>
<div dir="ltr">
<div dir="ltr">On Wed, Sep 23, 2020 at 4:12 PM Krishnan, Gautham <<a href="mailto:gautham3@illinois.edu">gautham3@illinois.edu</a>> wrote:<br>
</div>
<div class="x_gmail_quote">
<blockquote class="x_gmail_quote" style="margin:0px 0px 0px 0.8ex; border-left:1px solid rgb(204,204,204); padding-left:1ex">
<div dir="ltr">
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0); background-color:rgb(255,255,255)">
Hello,</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0); background-color:rgb(255,255,255)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0); background-color:rgb(255,255,255)">
For a CFD code being developed with FORTRAN and MPI, I am using PETSC matrices and for a particular operation, I require to extract a submatrix(n-1 x n-1) of a matrix created (n x n). However using the petsc MatGetSubMatrix works for serial runs but fails when
the domain is split up over PEs- I suspect the indexing changed for parallel runs and hence the global indexing that worked for serial case just shuffles around matrix entries in parallel undesirably. I would like to ask whether anybody could offer some guidance
regarding this. I would like to note that the 2D domain is split along both x and y axes for parallel runs on multiple PEs.</div>
</div>
</blockquote>
<div><br>
</div>
<div>In parallel, you pass MatGetSubmatrix() the global indices that you want on your process.</div>
<div><br>
</div>
<div> Thanks,</div>
<div><br>
</div>
<div> Matt</div>
<div> </div>
<blockquote class="x_gmail_quote" style="margin:0px 0px 0px 0.8ex; border-left:1px solid rgb(204,204,204); padding-left:1ex">
<div dir="ltr">
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
Regards,</div>
<div id="x_gmail-m_789255861923840103Signature">
<div style="font-family:Tahoma; font-size:13px"><span style="color:rgb(34,34,34); font-family:sans-serif; background-color:rgb(255,255,255)">Gautham Krishnan,</span>
<div style="color:rgb(34,34,34); font-family:sans-serif; background-color:rgb(255,255,255)">
<br>
</div>
</div>
</div>
</div>
</blockquote>
</div>
<br clear="all">
<div><br>
</div>
-- <br>
<div dir="ltr" class="x_gmail_signature">
<div dir="ltr">
<div>
<div dir="ltr">
<div>
<div dir="ltr">
<div>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>
-- Norbert Wiener</div>
<div><br>
</div>
<div><a href="http://www.cse.buffalo.edu/~knepley/" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</body>
</html>