Jinquan:<div>Has this problem or similar problem been solved using scalapack?</div><div>Your problem is virtually a dense problem, why choose petsc library?</div><div>What type of computer, number of cores, to be used for this problem?</div>
<div><br></div><div>Hong</div><div><br><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div lang="EN-US" link="blue" vlink="purple">
<div>
<p class="MsoNormal">First question:<u></u><u></u></p>
<div><div class="im">
<p class="MsoNormal">Are your matrices sparse or dense?<u></u><u></u></p>
</div><p><u></u><span style="font-size:11.0pt;font-family:Symbol;color:#1f497d"><span>·<span style="font:7.0pt "Times New Roman"">
</span></span></span><u></u><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">A is a dense matrix at the order of 225,000 of complex*16;</span> </p></div></div></div></blockquote>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div lang="EN-US" link="blue" vlink="purple"><div><div><p><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u> <u></u></span></p>
</div>
<div>
<p class="MsoNormal">How large? <span style="color:#1f497d"><u></u><u></u></span></p>
<p><u></u><span style="font-size:11.0pt;font-family:Symbol;color:#1f497d"><span>·<span style="font:7.0pt "Times New Roman"">
</span></span></span><u></u><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">800GB on disk<u></u><u></u></span></p><div class="im">
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal">What type of math equations, e.g., solve Ax=b or eigenvalue problems?<u></u><u></u></p>
</div><p><u></u><span style="font-size:11.0pt;font-family:Symbol;color:#1f497d"><span>·<span style="font:7.0pt "Times New Roman"">
</span></span></span><u></u><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">A’x=b, where A’=[A^-1 U B]= sparse matrix at the order of 2 million. B = sparse matrix to be mixed and mingled with A^-1.<u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-bottom:12.0pt">Hong<span class="HOEnZb"><font color="#888888"><u></u><u></u></font></span></p><span class="HOEnZb"><font color="#888888">
<p class="MsoNormal" style="margin-bottom:12.0pt"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal" style="margin-bottom:12.0pt"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">Jinquan<u></u><u></u></span></p></font></span><div><div class="h5">
<div>
<p class="MsoNormal">On Wed, Aug 8, 2012 at 5:36 PM, Jinquan Zhong <<a href="mailto:jzhong@scsolutions.com" target="_blank">jzhong@scsolutions.com</a>> wrote:<u></u><u></u></p>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">Thanks, Prof. Zhang.</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"> </span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">I could have used PETSc directly for this application. However, I have to use this procedure for
QA (quality assurance) purposes since this parallelizing process for a program I am working on is a standard program widely used in industry. That is, QA purposes have driven us to use ScaLAPACK for inversion first then PETSc for solution. I wish I could
a big move now to eliminate ScaLAPACK.</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"> </span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">Jinquan</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"> </span><u></u><u></u></p>
<p class="MsoNormal"><b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">From:</span></b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">
<a href="mailto:petsc-users-bounces@mcs.anl.gov" target="_blank">petsc-users-bounces@mcs.anl.gov</a> [mailto:<a href="mailto:petsc-users-bounces@mcs.anl.gov" target="_blank">petsc-users-bounces@mcs.anl.gov</a>]
<b>On Behalf Of </b>Hong Zhang<br>
<b>Sent:</b> Wednesday, August 08, 2012 3:29 PM</span><u></u><u></u></p>
<div>
<div>
<p class="MsoNormal"><br>
<b>To:</b> PETSc users list<br>
<b>Subject:</b> Re: [petsc-users] Customizeing MatSetValuesBlocked(...)<u></u><u></u></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
<p class="MsoNormal">Jinquan:<u></u><u></u></p>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
<div>
<p class="MsoNormal">You need understand some basic concepts in numerical computations, for example:<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">- dense matrix vs. sparse matrix and their data structures<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">- algorithms for solve Ax=b<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">- available software packages, their targeted applications<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">See <span style="font-size:13.5pt;font-family:"Times","serif""><a href="http://www.mcs.anl.gov/~hzhang/teach/cs595" target="_blank">www.mcs.anl.gov/~hzhang/teach/cs595</a> for a
list of references.</span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:13.5pt;font-family:"Times","serif"">Hong</span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
<div>
<blockquote style="border:none;border-left:solid #cccccc 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0in;margin-bottom:5.0pt">
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">What I am trying to say is that
</span><u></u><u></u></p>
<p><span style="font-size:11.0pt;font-family:Symbol;color:#1f497d">·</span><span style="font-size:7.0pt;color:#1f497d">
</span><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">allocated 1D array A in PETSc</span><u></u><u></u></p>
<p><span style="font-size:11.0pt;font-family:Symbol;color:#1f497d">·</span><span style="font-size:7.0pt;color:#1f497d">
</span><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">passed 1D A into ScaLAPACK</span><u></u><u></u></p>
<p><span style="font-size:11.0pt;font-family:Symbol;color:#1f497d">·</span><span style="font-size:7.0pt;color:#1f497d">
</span><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">obtained the indices of A^-1 computed from ScaLAPACK and stored them in a 1D array</span><u></u><u></u></p>
<p><span style="font-size:11.0pt;font-family:Symbol;color:#1f497d">·</span><span style="font-size:7.0pt;color:#1f497d">
</span><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">indexed A^-1 as a 1D array in PETSc using the indices obtained from ScaLAPACK</span><u></u><u></u></p>
<p><span style="font-size:11.0pt;font-family:Symbol;color:#1f497d">·</span><span style="font-size:7.0pt;color:#1f497d">
</span><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">constructed a sparse matrix, including A^-1 and others, using a new set of indices for A^-1.
</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">I can have the last step done in using
</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"> </span><u></u><u></u></p>
<p class="MsoNormal" style="text-autospace:none">
<span style="font-size:12.5pt;font-family:Consolas"> ierr = MatCreate(PETSC_COMM_WORLD,&A);CHKERRQ(ierr);</span><u></u><u></u></p>
<p class="MsoNormal" style="text-autospace:none">
<span style="font-size:12.5pt;font-family:Consolas"> ierr = MatSetSizes(A,PETSC_DECIDE,PETSC_DECIDE,N,N);CHKERRQ(ierr);</span><u></u><u></u></p>
<div>
<p class="MsoNormal" style="text-autospace:none">
<span style="font-size:12.5pt;font-family:Consolas"> ierr = MatSetFromOptions(A);CHKERRQ(ierr);</span><u></u><u></u></p>
<p class="MsoNormal" style="text-autospace:none">
<span style="font-size:12.5pt;font-family:Consolas"> ierr = MatSetUp(A);CHKERRQ(ierr);</span><u></u><u></u></p>
<p class="MsoNormal" style="margin-left:1.5in;text-indent:.5in;text-autospace:none">
<span style="font-size:12.5pt;font-family:Consolas"> </span><u></u><u></u></p>
</div>
<p class="MsoNormal" style="margin-left:1.5in;text-indent:.5in;text-autospace:none">
<span style="font-size:12.5pt;font-family:Consolas"> ierr = MatSetValues(A,1,&Ii,1,&J,&v,INSERT_VALUES);CHKERRQ(ierr);</span><u></u><u></u></p>
<p class="MsoNormal" style="text-autospace:none">
<span style="font-size:12.5pt;font-family:Consolas"> </span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">It just took too long. This is the performance issue I am having.
</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"> </span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">To improve the performance, it is necessary for preallocation. The dilemma is:
<b>MatCreateAIJ</b> and <b>MatXCreateAIJ</b> complained the setting of <b>d_nz, d_nnz,o_nz, o_nnz</b> since there are no diagonal and off-diagonal parts defined in A^-1.</span><u></u><u></u></p>
<p class="MsoNormal" style="text-autospace:none">
<span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"> </span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">Jinquan</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"> </span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"> </span><u></u><u></u></p>
<p class="MsoNormal"><b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">From:</span></b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">
<a href="mailto:petsc-users-bounces@mcs.anl.gov" target="_blank">petsc-users-bounces@mcs.anl.gov</a> [mailto:<a href="mailto:petsc-users-bounces@mcs.anl.gov" target="_blank">petsc-users-bounces@mcs.anl.gov</a>]
<b>On Behalf Of </b>Jed Brown<br>
<b>Sent:</b> Wednesday, August 08, 2012 3:04 PM</span><u></u><u></u></p>
<div>
<p class="MsoNormal"><br>
<b>To:</b> PETSc users list<br>
<b>Subject:</b> Re: [petsc-users] Customizeing MatSetValuesBlocked(...)<u></u><u></u></p>
</div>
<p class="MsoNormal"> <u></u><u></u></p>
<div>
<p class="MsoNormal">On Wed, Aug 8, 2012 at 3:45 PM, Jinquan Zhong <<a href="mailto:jzhong@scsolutions.com" target="_blank">jzhong@scsolutions.com</a>> wrote:<u></u><u></u></p>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">It is a 1D matrix in PETSc. I just passed it as a 2D matrix in ScaLAPACK. It is still a 1D matrix
when get back into PETSc.</span><u></u><u></u></p>
</div>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
<div>
<p class="MsoNormal">What are you trying to say?<u></u><u></u></p>
</div>
</div>
</div>
</div>
</blockquote>
</div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<p class="MsoNormal"><u></u> <u></u></p>
</div></div></div>
</div>
</div>
</blockquote></div><br></div>