<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 14 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:Wingdings;
panose-1:5 0 0 0 0 0 0 0 0 0;}
@font-face
{font-family:Wingdings;
panose-1:5 0 0 0 0 0 0 0 0 0;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman","serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p
{mso-style-priority:99;
mso-margin-top-alt:auto;
margin-right:0in;
mso-margin-bottom-alt:auto;
margin-left:0in;
font-size:12.0pt;
font-family:"Times New Roman","serif";}
span.EmailStyle18
{mso-style-type:personal-reply;
font-family:"Calibri","sans-serif";
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri","sans-serif";}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<div>
<div>
<div>
<div>
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
</div>
</div>
</div>
</div>
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">"mingled" is not a very useful term. How about using an actual equation so we can have some clue what you mean.<o:p></o:p></p>
</div>
<p><span style="font-size:11.0pt;font-family:"Courier New";color:#1F497D">o</span><span style="font-size:7.0pt;color:#1F497D">
</span><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">I agree. It is hard to have an accurate equation to for the structure of A’. But if you are familiar with FEM, you may know global stiffness matrix K=SIGMA_ij( K_ij ),
the assembling process of elemental stiffness. Here A’=K. </span><o:p></o:p></p>
</div>
</div>
<div>
<p class="MsoNormal">This cannot be true because you say A' is the larger part sparse and part dense matrix. Maybe you mean A rather than A'?<o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">**Jed, A’=[A^-1 U B], not the transpose of A. We need to solve for A’*x=b. A is dense matrix. B is a sparse matrix. Therefore, A’ is a sparse matrix when
the rule </span><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">K=A’=SIGMA_ij (A^-1_ij)+SIGMA_i’j’(B_i’j’)
</span><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">applies.<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in">
<div>
<div>
<p><span style="font-size:11.0pt;font-family:"Courier New";color:#1F497D">o</span><span style="font-size:7.0pt;color:#1F497D">
</span><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">We partition the FEM domain into two physical pieces due to different operations they are defined for. After completing each operation, we need to operate on A^-1_ij and
B_i’j’ such that we can have K=A’=SIGMA_ij (A^-1_ij)+SIGMA_i’j’(B_i’j’).</span><o:p></o:p></p>
</div>
</div>
</blockquote>
<div>
<p class="MsoNormal">The sum of the inverses are not the inverses of the sum. Can you back up further to explain what underlying problem you are trying to solve?<o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">** As I mentioned A’=K is the assembled global matrix using the rule
</span><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">K=A’=SIGMA_ij (A^-1_ij)+SIGMA_i’j’(B_i’j’)
</span><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">from FEM. Here,
</span><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">SIGMA_ij (A^-1_ij) denotes the assembling process for A^-1_ij into K according the DOFs of each node in the FEM model, so does SIGMA_i’j’(B_i’j’). in this application,
A^-1 and B have different indices sets. Some of them may locate at the same locations in K.</span><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p></o:p></span></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in">
<div>
<div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0in;margin-bottom:5.0pt">
<div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"> </span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">If you have an "inverse" floating around, what sort of operator is it the inverse of.<o:p></o:p></p>
<p><span style="font-size:11.0pt;font-family:Wingdings;color:#1F497D">ü</span><span style="font-size:7.0pt;color:#1F497D">
</span><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">A is a double complex matrix we have generated from a practical application using FEM. </span><o:p></o:p></p>
</div>
</div>
</blockquote>
</div>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Then there is no way A^{-1} should be stored as a dense matrix.
<o:p></o:p></p>
</div>
<p><span style="font-size:11.0pt;font-family:"Courier New";color:#1F497D">o</span><span style="font-size:7.0pt;color:#1F497D">
</span><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">It is stored as a dense matrix in scalapack. It stays in the same way in cores as scalapack finishes its inversion. We could define A^-1 as dense matrix in PETSc. </span><o:p></o:p></p>
</blockquote>
<div>
<p class="MsoNormal">It should not be done this way. A^{-1} should not be stored explicitly. Store the sparse finite element matrix A. Then when you want to "apply A^{-1}", solve with the sparse matrix.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"> <span style="color:#1F497D">** Jed, you are getting close to understand the problem related to QA. A has to be inverted explicitly. A^-1 has to be known entry by entry such that each entry in B could be assembled with A^-1 to form A’=K.
This is a requirement beyond technical issue. This is a QA issue.</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in">
<p><span style="font-size:11.0pt;font-family:"Courier New";color:#1F497D">o</span><span style="font-size:7.0pt;color:#1F497D">
</span><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Another constraint we have is we have to use direct solver for QA purposes. This brings up the needs to convert the dense matrix into MATAIJ or MATMPIAIJ. However, in PETSc,
converting dense matrix-formatted A^-1 into MATAIJ/ MATMPIAIJ also takes lots of time.</span><o:p></o:p></p>
</blockquote>
<div>
<p class="MsoNormal">Choose the right tool for the job. When I cook dinner tonight, I plan to use a stove and a frying pan instead of a dump truck radiator and 200 feet of 18 gauge copper wire woven into a basket.<o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">** That why I came here for. PETSc is the right tool for solver in the last step A’*x=b. However, due to the differences in addressing the matrix between
PETSc and ScaLAPACK, I am asking for the kind help from this community to current issue.<o:p></o:p></span></p>
</div>
</div>
</div>
</body>
</html>