First question:<div>Are your matrices sparse or dense?</div><div>How large? What type of math equations, e.g., solve Ax=b or eigenvalue problems?</div><div><br></div><div>Hong<br><br><div class="gmail_quote">On Wed, Aug 8, 2012 at 5:36 PM, Jinquan Zhong <span dir="ltr"><<a href="mailto:jzhong@scsolutions.com" target="_blank">jzhong@scsolutions.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div lang="EN-US" link="blue" vlink="purple">
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">Thanks, Prof. Zhang.<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">I could have used PETSc directly for this application. However, I have to use this procedure for QA (quality assurance) purposes since this parallelizing process
for a program I am working on is a standard program widely used in industry. That is, QA purposes have driven us to use ScaLAPACK for inversion first then PETSc for solution. I wish I could a big move now to eliminate ScaLAPACK.<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">Jinquan<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">From:</span></b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif""> <a href="mailto:petsc-users-bounces@mcs.anl.gov" target="_blank">petsc-users-bounces@mcs.anl.gov</a> [mailto:<a href="mailto:petsc-users-bounces@mcs.anl.gov" target="_blank">petsc-users-bounces@mcs.anl.gov</a>]
<b>On Behalf Of </b>Hong Zhang<br>
<b>Sent:</b> Wednesday, August 08, 2012 3:29 PM</span></p><div><div class="h5"><br>
<b>To:</b> PETSc users list<br>
<b>Subject:</b> Re: [petsc-users] Customizeing MatSetValuesBlocked(...)<u></u><u></u></div></div><p></p><div><div class="h5">
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">Jinquan:<u></u><u></u></p>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
<div>
<p class="MsoNormal">You need understand some basic concepts in numerical computations, for example:<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">- dense matrix vs. sparse matrix and their data structures<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">- algorithms for solve Ax=b<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">- available software packages, their targeted applications<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">See <span><span style="font-size:13.5pt;font-family:"Times","serif""><a href="http://www.mcs.anl.gov/~hzhang/teach/cs595" target="_blank">www.mcs.anl.gov/~hzhang/teach/cs595</a> for a list of references.</span></span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal"><span><span style="font-size:13.5pt;font-family:"Times","serif"">Hong</span></span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
<div>
<blockquote style="border:none;border-left:solid #cccccc 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in">
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">What I am trying to say is that
</span><u></u><u></u></p>
<p><span style="font-size:11.0pt;font-family:Symbol;color:#1f497d">·</span><span style="font-size:7.0pt;color:#1f497d">
</span><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">allocated 1D array A in PETSc</span><u></u><u></u></p>
<p><span style="font-size:11.0pt;font-family:Symbol;color:#1f497d">·</span><span style="font-size:7.0pt;color:#1f497d">
</span><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">passed 1D A into ScaLAPACK</span><u></u><u></u></p>
<p><span style="font-size:11.0pt;font-family:Symbol;color:#1f497d">·</span><span style="font-size:7.0pt;color:#1f497d">
</span><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">obtained the indices of A^-1 computed from ScaLAPACK and stored them in a 1D array</span><u></u><u></u></p>
<p><span style="font-size:11.0pt;font-family:Symbol;color:#1f497d">·</span><span style="font-size:7.0pt;color:#1f497d">
</span><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">indexed A^-1 as a 1D array in PETSc using the indices obtained from ScaLAPACK</span><u></u><u></u></p>
<p><span style="font-size:11.0pt;font-family:Symbol;color:#1f497d">·</span><span style="font-size:7.0pt;color:#1f497d">
</span><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">constructed a sparse matrix, including A^-1 and others, using a new set of indices for A^-1.
</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">I can have the last step done in using
</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"> </span><u></u><u></u></p>
<p class="MsoNormal" style="text-autospace:none">
<span style="font-size:12.5pt;font-family:Consolas"> ierr = MatCreate(PETSC_COMM_WORLD,&A);CHKERRQ(ierr);</span><u></u><u></u></p>
<p class="MsoNormal" style="text-autospace:none">
<span style="font-size:12.5pt;font-family:Consolas"> ierr = MatSetSizes(A,PETSC_DECIDE,PETSC_DECIDE,N,N);CHKERRQ(ierr);</span><u></u><u></u></p>
<div>
<p class="MsoNormal" style="text-autospace:none">
<span style="font-size:12.5pt;font-family:Consolas"> ierr = MatSetFromOptions(A);CHKERRQ(ierr);</span><u></u><u></u></p>
<p class="MsoNormal" style="text-autospace:none">
<span style="font-size:12.5pt;font-family:Consolas"> ierr = MatSetUp(A);CHKERRQ(ierr);</span><u></u><u></u></p>
<p class="MsoNormal" style="margin-left:1.5in;text-indent:.5in;text-autospace:none">
<span style="font-size:12.5pt;font-family:Consolas"> </span><u></u><u></u></p>
</div>
<p class="MsoNormal" style="margin-left:1.5in;text-indent:.5in;text-autospace:none">
<span style="font-size:12.5pt;font-family:Consolas"> ierr = MatSetValues(A,1,&Ii,1,&J,&v,INSERT_VALUES);CHKERRQ(ierr);</span><u></u><u></u></p>
<p class="MsoNormal" style="text-autospace:none">
<span style="font-size:12.5pt;font-family:Consolas"> </span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">It just took too long. This is the performance issue I am having.
</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"> </span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">To improve the performance, it is necessary for preallocation. The dilemma is:
<b>MatCreateAIJ</b> and <b>MatXCreateAIJ</b> complained the setting of <b>d_nz, d_nnz,o_nz, o_nnz</b> since there are no diagonal and off-diagonal parts defined in A^-1.</span><u></u><u></u></p>
<p class="MsoNormal" style="text-autospace:none">
<span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"> </span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">Jinquan</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"> </span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"> </span><u></u><u></u></p>
<p class="MsoNormal"><b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">From:</span></b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">
<a href="mailto:petsc-users-bounces@mcs.anl.gov" target="_blank">petsc-users-bounces@mcs.anl.gov</a> [mailto:<a href="mailto:petsc-users-bounces@mcs.anl.gov" target="_blank">petsc-users-bounces@mcs.anl.gov</a>]
<b>On Behalf Of </b>Jed Brown<br>
<b>Sent:</b> Wednesday, August 08, 2012 3:04 PM</span><u></u><u></u></p>
<div>
<p class="MsoNormal"><br>
<b>To:</b> PETSc users list<br>
<b>Subject:</b> Re: [petsc-users] Customizeing MatSetValuesBlocked(...)<u></u><u></u></p>
</div>
<p class="MsoNormal"> <u></u><u></u></p>
<div>
<p class="MsoNormal">On Wed, Aug 8, 2012 at 3:45 PM, Jinquan Zhong <<a href="mailto:jzhong@scsolutions.com" target="_blank">jzhong@scsolutions.com</a>> wrote:<u></u><u></u></p>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">It is a 1D matrix in PETSc. I just passed it as a 2D matrix in ScaLAPACK. It is still a 1D matrix
when get back into PETSc.</span><u></u><u></u></p>
</div>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
<div>
<p class="MsoNormal">What are you trying to say?<u></u><u></u></p>
</div>
</div>
</div>
</div>
</blockquote>
</div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
</div>
</div></div></div>
</div>
</blockquote></div><br></div>