<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<meta name="Generator" content="Microsoft Word 14 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri","sans-serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p.MsoListParagraph, li.MsoListParagraph, div.MsoListParagraph
{mso-style-priority:34;
margin-top:0cm;
margin-right:0cm;
margin-bottom:0cm;
margin-left:36.0pt;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri","sans-serif";}
span.E-MailFormatvorlage17
{mso-style-type:personal-compose;
font-family:"Calibri","sans-serif";
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:70.85pt 70.85pt 2.0cm 70.85pt;}
div.WordSection1
{page:WordSection1;}
/* List Definitions */
@list l0
{mso-list-id:332270819;
mso-list-type:hybrid;
mso-list-template-ids:1362402704 67698705 67698713 67698715 67698703 67698713 67698715 67698703 67698713 67698715;}
@list l0:level1
{mso-level-text:"%1\)";
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;}
@list l0:level2
{mso-level-number-format:alpha-lower;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;}
@list l0:level3
{mso-level-number-format:roman-lower;
mso-level-tab-stop:none;
mso-level-number-position:right;
text-indent:-9.0pt;}
@list l0:level4
{mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;}
@list l0:level5
{mso-level-number-format:alpha-lower;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;}
@list l0:level6
{mso-level-number-format:roman-lower;
mso-level-tab-stop:none;
mso-level-number-position:right;
text-indent:-9.0pt;}
@list l0:level7
{mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;}
@list l0:level8
{mso-level-number-format:alpha-lower;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;}
@list l0:level9
{mso-level-number-format:roman-lower;
mso-level-tab-stop:none;
mso-level-number-position:right;
text-indent:-9.0pt;}
ol
{margin-bottom:0cm;}
ul
{margin-bottom:0cm;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal">Dear PETSc community!<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">I would like to read in a (large) sparse SPD matrix from a file in parallel. More precisely my plan was to do the following:<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">1) Read matrix size N from file.<o:p></o:p></p>
<p class="MsoNormal">2) Create PETSc matrix.<o:p></o:p></p>
<p class="MsoNormal">3) Set option MAT_SPD=PETSC_TRUE.<o:p></o:p></p>
<p class="MsoNormal">4) Set global size N x N, local sizes PETSC_DECIDE.<o:p></o:p></p>
<p class="MsoNormal">5) Read in only those rows from file, which are owned by the local process.<o:p></o:p></p>
<p class="MsoNormal">6) Preallocate the matrix using statistics collected in the previous step.<o:p></o:p></p>
<p class="MsoNormal">7) Insert the values read into the matrix row-by-row.<o:p></o:p></p>
<p class="MsoNormal">8) Begin and finish matrix assembly.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">My problem is in step 5, leading to 3 questions:<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">QUESTION 1: How can I let PETSc decide, which rows of the global matrix will be local to the process BEFORE prealloction?<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">In the manual pages I have found so far:<o:p></o:p></p>
<p class="MsoNormal">A) MatGetOwnershipRange(): <o:p></o:p></p>
<p class="MsoNormal">“requires that the matrix be preallocated”.<o:p></o:p></p>
<p class="MsoNormal">B) MatGetOwnershipRanges(): <o:p></o:p></p>
<p class="MsoNormal">“Not collective, unless matrix has not been allocated, then collective on Mat”<o:p></o:p></p>
<p class="MsoNormal">However, when running the program, I get the error message: “Must call MatXXXSetPreallocation() or MatSetUp() … before MatGetOwnershipRanges()!”<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">QUESTION 2: Is the documentation of MatGetOwnershipRanges() incorrect or am I misinterpreting it?<o:p></o:p></p>
<p class="MsoNormal">-> <a href="http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MatGetOwnershipRanges.html">
http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MatGetOwnershipRanges.html</a><o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">I finally got the program running by calling MatSetUp() before MatGetOwnershipRange(). Still I cannot fight the feeling that I am doing things not as they have been intended by the developers, since preallocation is now done twice.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">The alternative seems to be to use PetscSplitOwnership() and MPI_Scan() to calculate the row ranges for each process before creating the matrix with MatCreate(). But this leads in any case to a very even distribution of row counts among
the processes. Assuming that only the upper triangular part of the symmetric matrix needs to be stored (IS THIS CORRECT?), I would guess that consequently this leads to an imbalance regarding the number of (nonzero) matrix entries owned by each process (Processes
with higher rank will own fewer nonzeros).<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">QUESTION 3: For SPD matrices, is it in general a good strategy to have every process owning approximately the same number of rows? (In this case, I can of course forget about PetscSplitOwnership() and MPI_Scan() and do the distribution
myself).<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Thank you and kind regards,<o:p></o:p></p>
<p class="MsoNormal">Thomas Ponweiser<o:p></o:p></p>
</div>
</body>
</html>