<div dir="ltr"><span style="border-collapse:collapse;font-family:Calibri,sans-serif">Shuangshuang,</span><br><div><span style="border-collapse:collapse;font-family:Calibri,sans-serif">In ex38.c, m and n are local block sizes, not global. When running np=2 on a matrix with global sizes 12 by 12, you should give</span></div>
<div><span style="border-collapse:collapse;font-family:Calibri,sans-serif">mpiexec -n 2 ./ex38 -Cexp_view_ownership 1 -m 6 -n 6<br></span></div><div><span style="border-collapse:collapse;font-family:Calibri,sans-serif"><br>
</span></div><div><span style="border-collapse:collapse;font-family:Calibri,sans-serif">Hong</span></div>
<div><font face="Calibri, sans-serif"><span style="border-collapse:collapse"><br></span></font></div></div><div class="gmail_extra"><br><br><div class="gmail_quote">On Tue, Jun 11, 2013 at 3:22 PM, Jin, Shuangshuang <span dir="ltr"><<a href="mailto:Shuangshuang.Jin@pnnl.gov" target="_blank">Shuangshuang.Jin@pnnl.gov</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div>
<font face="Calibri, sans-serif">
<div>Hello, I’m trying to set my Jacobian matrix to MATELEMENTAL type.</div>
<div><font face="Times New Roman, serif"> </font></div>
<div>According to an Elemental matrix example code: /src/mat/examples/tests/ex38.c, I wrote my piece of code as following:</div>
<div><font face="Times New Roman, serif"> </font></div>
<div> ierr = MatCreate(PETSC_COMM_WORLD, &J); CHKERRQ(ierr); // J: Jacobian matrix</div>
<div> ierr = MatSetSizes(J, 12, 12, PETSC_DECIDE, PETSC_DECIDE); CHKERRQ(ierr);</div>
<div> ierr = MatSetType(J, MATELEMENTAL); CHKERRQ(ierr);</div>
<div> ierr = MatSetFromOptions(J); CHKERRQ(ierr);</div>
<div> ierr = MatSetUp(J); CHKERRQ(ierr);</div>
<div><font face="Times New Roman, serif"> </font></div>
<div> PetscInt nrows,ncols;</div>
<div> const PetscInt *rows,*cols;</div>
<div> IS isrows,iscols;</div>
<div> PetscScalar *v;</div>
<div> ierr = MatGetOwnershipIS(J, &isrows, &iscols); CHKERRQ(ierr);</div>
<div> </div>
<div> // Set local matrix entries</div>
<div> ierr = ISGetLocalSize(isrows, &nrows); CHKERRQ(ierr);</div>
<div> ierr = ISGetIndices(isrows, &rows); CHKERRQ(ierr);</div>
<div> ierr = ISGetLocalSize(iscols, &ncols); CHKERRQ(ierr);</div>
<div> ierr = ISGetIndices(iscols, &cols); CHKERRQ(ierr);</div>
<div> ierr = PetscMalloc(nrows*ncols*sizeof(*v), &v); CHKERRQ(ierr);</div>
<div> printf("nrows=%d, ncols=%d\n",nrows,ncols);</div>
<div> for (i = 0; i < nrows; i++) {</div>
<div> for (j = 0; j < ncols; j++) {</div>
<div> v[i*ncols+j] = jacVal[i][j]; // jacVal stores the value of the 12*12 Jacobian matrix</div>
<div> }</div>
<div> }</div>
<div> ierr = MatSetValues(J, nrows, rows, ncols, cols, v, INSERT_VALUES); CHKERRQ(ierr);</div>
<div> ierr = ISRestoreIndices(isrows, &rows); CHKERRQ(ierr);</div>
<div> ierr = ISRestoreIndices(iscols, &cols); CHKERRQ(ierr);</div>
<div> ierr = MatAssemblyBegin(J, MAT_FINAL_ASSEMBLY); CHKERRQ(ierr);</div>
<div> ierr = MatAssemblyEnd(J, MAT_FINAL_ASSEMBLY); CHKERRQ(ierr);</div>
<div> ierr = ISDestroy(&isrows); CHKERRQ(ierr);</div>
<div> ierr = ISDestroy(&iscols); CHKERRQ(ierr);</div>
<div><font face="Times New Roman, serif"> </font></div>
<div>This works for one processor. When it goes to multiple processor, it will fail. </div>
<div> </div>
<div>I notice that when the # of processor equals to 1, the nrows and ncols will be 12 each, which are the dimension of the Jacobian matrix.</div>
<div>However, when the # of processor goes up, for example, to 2, nrows=24, ncols=12, then it will fail because the index of my jac[i][j] are withn [0, 11].</div>
<div> </div>
<div>My question is, to use MATELEMENTAL, what is the rule to fit the values of my 12x12 matrix into a 24X12 arrays by the two level for loops?</div>
<div><font face="Times New Roman, serif"> </font></div>
<div>And I don’t understand what the set means here:</div>
<div>[d3m956@olympus tests]$ mpiexec -n 1 ex38 -Cexp_view_ownership 1 -m 12 -n 12</div>
<div>Ownership of explicit C:</div>
<div>Row index set:</div>
<div>Number of indices in set 12</div>
<div>0 0</div>
<div>1 1</div>
<div>2 2</div>
<div>3 3</div>
<div>4 4</div>
<div>5 5</div>
<div>6 6</div>
<div>7 7</div>
<div>8 8</div>
<div>9 9</div>
<div>10 10</div>
<div>11 11</div>
<div>Column index set:</div>
<div>Number of indices in set 12</div>
<div>0 0</div>
<div>1 1</div>
<div>2 2</div>
<div>3 3</div>
<div>4 4</div>
<div>5 5</div>
<div>6 6</div>
<div>7 7</div>
<div>8 8</div>
<div>9 9</div>
<div>10 10</div>
<div>11 11</div>
<div>nrows=12, ncols=12</div>
<div><font face="Times New Roman, serif"> </font></div>
<div>[d3m956@olympus tests]$ mpiexec -n 2 ex38 -Cexp_view_ownership 1 -m 12 -n 12</div>
<div>Ownership of explicit C:</div>
<div>Row index set:</div>
<div>[0] Number of indices in set 24</div>
<div>[0] 0 0</div>
<div>[0] 1 12</div>
<div>[0] 2 1</div>
<div>[0] 3 13</div>
<div>[0] 4 2</div>
<div>[0] 5 14</div>
<div>[0] 6 3</div>
<div>[0] 7 15</div>
<div>[0] 8 4</div>
<div>[0] 9 16</div>
<div>[0] 10 5</div>
<div>[0] 11 17</div>
<div>[0] 12 6</div>
<div>[0] 13 18</div>
<div>[0] 14 7</div>
<div>[0] 15 19</div>
<div>[0] 16 8</div>
<div>[0] 17 20</div>
<div>[0] 18 9</div>
<div>[0] 19 21</div>
<div>[0] 20 10</div>
<div>[0] 21 22</div>
<div>[0] 22 11</div>
<div>[0] 23 23</div>
<div>nrows=24, ncols=12</div>
<div>[1] Number of indices in set 24</div>
<div>[1] 0 0</div>
<div>[1] 1 12</div>
<div>[1] 2 1</div>
<div>[1] 3 13</div>
<div>[1] 4 2</div>
<div>[1] 5 14</div>
<div>[1] 6 3</div>
<div>[1] 7 15</div>
<div>[1] 8 4</div>
<div>[1] 9 16</div>
<div>[1] 10 5</div>
<div>[1] 11 17</div>
<div>[1] 12 6</div>
<div>[1] 13 18</div>
<div>[1] 14 7</div>
<div>[1] 15 19</div>
<div>[1] 16 8</div>
<div>[1] 17 20</div>
<div>[1] 18 9</div>
<div>[1] 19 21</div>
<div>[1] 20 10</div>
<div>[1] 21 22</div>
<div>[1] 22 11</div>
<div>[1] 23 23</div>
<div>Column index set:</div>
<div>[0] Number of indices in set 12</div>
<div>[0] 0 0</div>
<div>[0] 1 1</div>
<div>[0] 2 2</div>
<div>[0] 3 3</div>
<div>[0] 4 4</div>
<div>[0] 5 5</div>
<div>[0] 6 6</div>
<div>[0] 7 7</div>
<div>[0] 8 8</div>
<div>[0] 9 9</div>
<div>[0] 10 10</div>
<div>[0] 11 11</div>
<div>[1] Number of indices in set 12</div>
<div>[1] 0 12</div>
<div>[1] 1 13</div>
<div>[1] 2 14</div>
<div>[1] 3 15</div>
<div>[1] 4 16</div>
<div>[1] 5 17</div>
<div>[1] 6 18</div>
<div>[1] 7 19</div>
<div>[1] 8 20</div>
<div>[1] 9 21</div>
<div>[1] 10 22</div>
<div>[1] 11 23</div>
<div>nrows=24, ncols=12</div>
<div><font face="Times New Roman, serif"> </font></div>
<div>Thanks,</div>
<div>Shuangshuang</div>
<div><font face="Times New Roman, serif"> </font></div>
<div><font face="Times New Roman, serif"> </font></div>
<div><font face="Times New Roman, serif"> </font></div>
<div><font face="Times New Roman, serif"> </font></div>
<div><font face="Times New Roman, serif"> </font></div>
<div><font face="Times New Roman, serif"> </font></div>
</font>
</div>
</blockquote></div><br></div>