<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 14 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
@font-face
{font-family:Consolas;
panose-1:2 11 6 9 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman","serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p.MsoPlainText, li.MsoPlainText, div.MsoPlainText
{mso-style-priority:99;
mso-style-link:"Plain Text Char";
margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri","sans-serif";}
span.EmailStyle17
{mso-style-type:personal-reply;
font-family:"Calibri","sans-serif";
color:#1F497D;}
span.PlainTextChar
{mso-style-name:"Plain Text Char";
mso-style-priority:99;
mso-style-link:"Plain Text";
font-family:"Calibri","sans-serif";}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri","sans-serif";}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Thanks, Jed.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Your solution seems to be comprehensive. I would love to have it in the next phase of implementation. But I am running out of time till this Friday. What
I am trying to do is to use a quick fix to improve the performance of building sparse based on the lines we have as follows:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal" style="text-autospace:none"><span style="font-size:12.5pt;font-family:Consolas"> ierr = MatCreate(PETSC_COMM_WORLD,&A);CHKERRQ(ierr);<o:p></o:p></span></p>
<p class="MsoNormal" style="text-autospace:none"><span style="font-size:12.5pt;font-family:Consolas"> ierr = MatSetSizes(A,PETSC_DECIDE,PETSC_DECIDE,m*n,m*n);CHKERRQ(ierr);<o:p></o:p></span></p>
<p class="MsoNormal" style="text-autospace:none"><span style="font-size:12.5pt;font-family:Consolas"> ierr = MatSetFromOptions(A);CHKERRQ(ierr);<o:p></o:p></span></p>
<p class="MsoNormal" style="text-autospace:none"><span style="font-size:12.5pt;font-family:Consolas"> ierr = MatSetUp(A);CHKERRQ(ierr);<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">
<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:2.0in;text-indent:.5in"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">…compute</span><span style="font-size:12.5pt;font-family:Consolas"> Ii,J, and v</span><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:1.5in;text-indent:.5in;text-autospace:none">
<span style="font-size:12.5pt;font-family:Consolas"> ierr = MatSetValues(A,1,&Ii,1,&J,&v,INSERT_VALUES);CHKERRQ(ierr);<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:1.5in;text-indent:.5in;text-autospace:none">
<span style="font-size:12.5pt;font-family:Consolas"><o:p> </o:p></span></p>
<p class="MsoPlainText">“For my application, I have LDA=LDB=3 for local A and N=9 in the following
<o:p></o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoNormal"> ierr = MatCreateAIJ(PETSC_COMM_WORLD, LDA,LDB, N,N, LDA,0,0,0, &A); CHKERRQ(ierr);<span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoPlainText">I got the following error message:<o:p></o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText" style="margin-left:.5in;text-indent:.5in">[0]PETSC ERROR: --------------------- Error Message ------------------------------------<o:p></o:p></p>
<p class="MsoPlainText" style="margin-left:.5in;text-indent:.5in">[0]PETSC ERROR: Nonconforming object sizes!<o:p></o:p></p>
<p class="MsoPlainText" style="margin-left:.5in;text-indent:.5in">[0]PETSC ERROR: Sum of local lengths 27 does not equal global length 9, my local length 3<o:p></o:p></p>
<p class="MsoPlainText"> likely a call to VecSetSizes() or MatSetSizes() is wrong.<o:p></o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText">It does NOT appear PETSc had the 9x9 matrix. It tried to build 27*27 since I am using 9 procs. Is there a quick way to resolve this local size and global size issue?”<o:p></o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText">Thanks a lot anyway.<o:p></o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText">Jinquan<o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">From:</span></b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif""> petsc-users-bounces@mcs.anl.gov [mailto:petsc-users-bounces@mcs.anl.gov]
<b>On Behalf Of </b>Jed Brown<br>
<b>Sent:</b> Wednesday, August 08, 2012 2:05 PM<br>
<b>To:</b> PETSc users list<br>
<b>Subject:</b> Re: [petsc-users] Customizeing MatSetValuesBlocked(...)<o:p></o:p></span></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal">On Wed, Aug 8, 2012 at 2:53 PM, Jinquan Zhong <<a href="mailto:jzhong@scsolutions.com" target="_blank">jzhong@scsolutions.com</a>> wrote:<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">The dense matrix is predominantly so big ( 800GB) that it would cost too much to move it around in
the distributed memory. What I tried to do would be to retain the 800GB already in core from ScaLAPACK untouched and repartition their indices. The next step would be to repartition the indices for each block after I complete the construction of the sparse
matrix. This sparse matrix is mainly populated with 800GB entries from dense matrix, along with about 3 GB entries from other matrices.
</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"> </span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Had it be true as you said, wouldn’t that be a lot of exchange of data among distributed memory?
Or I am missing something huge.</span><o:p></o:p></p>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal">You can change indexing so that Elemental's matrix uses the same memory. PETSc uses yet another indexing. You can look at petsc-dev/src/mat/impls/elemental/matelem.cxx for the index translation between the ordering PETSc presents to the
user and the one Elemental uses internally. Anyway, if you configure an Elemental matrix to use your memory, while the (simpler) PETSc ordering has contiguous vectors [1], then you can combine it with traditional sparse matrices using MATNEST.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">[1] Note that PETSc's ownership ranges really fundamentally refer to the Vec distribution; the matrix entries can be stored elsewhere.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">There is nothing especially hard about doing the above, but you will have to do some careful indexing math. Look at the MATNEST man page and the Elemental documentation.<o:p></o:p></p>
</div>
</div>
</body>
</html>