<div dir="ltr"><div dir="ltr"><br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, Jan 17, 2022 at 6:40 PM Ferrand, Jesus A. <<a href="mailto:FERRANJ2@my.erau.edu">FERRANJ2@my.erau.edu</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="ltr">
<div style="font-family:"Times New Roman",Times,serif;font-size:12pt;color:rgb(0,0,0);background-color:rgb(255,255,255)">
Greetings!</div>
<div style="font-family:"Times New Roman",Times,serif;font-size:12pt;color:rgb(0,0,0);background-color:rgb(255,255,255)">
<br>
</div>
<div style="font-family:"Times New Roman",Times,serif;font-size:12pt;color:rgb(0,0,0);background-color:rgb(255,255,255)">
I have a question about preallocating memory for a parallel BAIJ matrix. From the documentation of MatMPIBAIJSetPreallocation(), the preallocation is divided between the so-called "diagonal" and "off-diagonal" sub matrices. In the example from the documentation,
the following illustration is given for the portion of a matrix that owns rows 3,4, and 5 of some 12-column matrix:<br>
</div>
<div style="font-family:"Times New Roman",Times,serif;font-size:12pt;color:rgb(0,0,0);background-color:rgb(255,255,255)">
<br>
</div>
<div style="font-family:"Times New Roman",Times,serif;font-size:12pt;color:rgb(0,0,0);background-color:rgb(255,255,255)">
<pre> 0 1 2 3 4 5 6 7 8 9 10 11
--------------------------
row 3 |o o o d d d o o o o o o
row 4 |o o o d d d o o o o o o
row 5 |o o o d d d o o o o o o
--------------------------</pre>
I have a few questions: about MPIBAIJ matrices, I am somewhat confused by the description:<br>
</div>
<div style="font-family:"Times New Roman",Times,serif;font-size:12pt;color:rgb(0,0,0);background-color:rgb(255,255,255)">
<ol>
<li><span>If the rank owned, say 5 rows instead of 3 on the same matrix, would the "d" block become a 5 by 5? as in:</span></li></ol></div></div></blockquote><div><br></div><div>When you create a matrix you specify how many local rows -- and columns (n) -- that you want.</div><div><br></div><pre style="color:rgb(0,0,0)">int MatCreateMPIBAIJ(MPI_Comm comm,int bs,int m,int n,int M,int N,int d_nz,int *d_nnz,int o_nz,int *o_nnz,Mat *A)
</pre><span style="color:rgb(0,0,0);font-family:Times;font-size:medium"></span><br class="gmail-Apple-interchange-newline"><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div style="font-family:"Times New Roman",Times,serif;font-size:12pt;color:rgb(0,0,0);background-color:rgb(255,255,255)"><ol><li style="display:block"><span>
<pre> 0 1 2 3 4 5 6 7 8 9 10 11
--------------------------
row 3 |o o o d d d d d o o o o
row 4 |o o o d d d d d o o o o
row 5 |o o o d d d d d o o o o<br> row 6 |o o o d d d d d o o o o<br> row 7 |o o o d d d d d o o o o
--------------------------</pre>
</span></li></ol>
<ol start="2">
<li><span>Is there a way to not have to deal with "d" blocks and "o" blocks separately? I know how many nonzero blocks are in each row of my matrix but I can't easily determine which of those zeros are in the "d" block or "o" block of my ranks.</span></li></ol></div></div></blockquote><div><br></div><div>No. It is best for performance to give an upper bound for each. </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div style="font-family:"Times New Roman",Times,serif;font-size:12pt;color:rgb(0,0,0);background-color:rgb(255,255,255)"><ol start="2"><li><span>The block size "bs", that is the size of each individual "d" or "o" as shown in the diagrams right? </span></li></ol></div></div></blockquote><div>d and o have the same "size" but the API is such that </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div style="font-family:"Times New Roman",Times,serif;font-size:12pt;color:rgb(0,0,0);background-color:rgb(255,255,255)"><ol start="2"><li><span>TRUE/FALSE: The parameters d_nz, o_nz, d_nnz, and o_nnz, those specify the number of "d" and "o" blocks that are nonzero (where each "d" or "o" is its own dense submatrix), not the actual number of nonzeros in the rows, right? Say the block size
is 3 and I set d_nz to 2, then for the diagram from the documentation this translates to 54 actual nonzeros in the diagonal block, right?<br></span></li></ol></div></div></blockquote><div><br></div><div>FALSE. This is a little confusing but the interface is the same for blocked or unblocked versions of the same matrix. Thus you can set from the command line -mat_type aij or baij and they both work. Keep that in mind to figure out the API.</div><div>This first example would work with a bs==3 or bs==1, which is an unblocked matrix. Your bs==5 example will not work with a blocked BAIJ matrix.</div><div> </div><div>Now performance sensitive functions like MatSetValues need blocked versions (<a href="https://petsc.org/main/docs/manualpages/Mat/MatSetValuesBlocked.html">https://petsc.org/main/docs/manualpages/Mat/MatSetValuesBlocked.html</a>). MatSetValues will work for a blocked matrix, but it is slower than exploiting the blocks with the blocked version. To be flexible, as I described with command line options, you would need to query the matrix to figure out which type it is and have two versions of the assembly (MatSetValues) code.</div><div>In practice, people often hardwire the code to a BAIJ matrix to avoid this complexity if they don't care about this flexibility.</div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div style="font-family:"Times New Roman",Times,serif;font-size:12pt;color:rgb(0,0,0);background-color:rgb(255,255,255)"><ol start="2"><li><span>
</span></li></ol>
</div>
<div style="font-family:"Times New Roman",Times,serif;font-size:12pt;color:rgb(0,0,0);background-color:rgb(255,255,255)">
<br>
</div>
<div>
<div style="font-family:"Times New Roman",Times,serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<div id="gmail-m_-3891779308173001060Signature">
<div>
<div style="font-family:"Times New Roman",Times,serif;font-size:12pt;color:rgb(0,0,0)">
</div>
<div></div>
<div></div>
<div id="gmail-m_-3891779308173001060divtagdefaultwrapper" dir="ltr" style="font-size:12pt;color:rgb(0,0,0);font-family:Calibri,Helvetica,sans-serif">
<p style="margin-top:0px;margin-bottom:0px"></p>
<div>
<p style="margin:0px 0px 10.66px"></p>
<div>
<p style="background:none 0% 0% repeat scroll white;margin:0px 0px 11px;line-height:normal">
<span style="margin:0px;color:black;font-family:"Times New Roman",Times,serif;font-size:12pt">Sincerely:</span></p>
<p style="background:none 0% 0% repeat scroll white;margin:0px;line-height:normal">
<span style="margin:0px;color:red;font-family:"Courier New",monospace;font-size:14pt"><b><span style="font-family:"Courier New",monospace">J.A. Ferrand</span></b></span><u><span style="margin:0px;color:red;font-family:"Swis721 BlkCn BT",sans-serif;font-size:12pt"></span></u></p>
<p style="background:none 0% 0% repeat scroll white;margin:0px;line-height:normal">
<span style="margin:0px;color:black;font-family:Courant;font-size:12pt">Embry-Riddle Aeronautical University - Daytona Beach FL</span></p>
<p style="background:none 0% 0% repeat scroll white;margin:0px;line-height:normal">
<span style="margin:0px;color:black;font-family:Courant;font-size:12pt">M.Sc. Aerospace Engineering | May 2022</span></p>
<p style="background:none 0% 0% repeat scroll white;margin:0px;line-height:normal">
<span style="margin:0px;color:black;font-family:Courant;font-size:12pt">B.Sc. Aerospace Engineering</span></p>
<p style="background:none 0% 0% repeat scroll white;margin:0px;line-height:normal">
<span style="margin:0px;color:black;font-family:Courant;font-size:12pt">B.Sc. Computational Mathematics</span></p>
<p style="background:none 0% 0% repeat scroll white;margin:0px;line-height:normal">
<span style="margin:0px;color:rgb(0,0,0);font-family:Courant;font-size:12pt"> </span></p>
<p style="background:none 0% 0% repeat scroll white;margin:0px;line-height:normal">
<span style="margin:0px;color:black;font-family:Courant;font-size:12pt"><span style="color:rgb(0,0,0);font-family:Courant;font-size:12pt;background-color:rgb(255,255,255);display:inline">Sigma Gamma Tau</span><span style="color:rgb(0,0,0);font-family:Courant;font-size:12pt;background-color:rgb(255,255,255);display:inline"> </span><br>
</span></p>
<p style="background:none 0% 0% repeat scroll white;margin:0px;line-height:normal">
<span style="margin:0px;color:rgb(0,0,0);font-family:Courant;font-size:12pt">Tau Beta Pi</span></p>
<p style="background:none 0% 0% repeat scroll white;margin:0px;line-height:normal">
<span style="margin:0px;color:black;font-family:Courant;font-size:12pt"><span style="margin:0px;color:rgb(0,0,0);font-family:Courant;font-size:12pt">Honors Program</span></span></p>
<p style="background:none 0% 0% repeat scroll white;margin:0px;line-height:normal">
<span style="margin:0px;color:black;font-family:Courant;font-size:12pt"> </span></p>
<p style="background:none 0% 0% repeat scroll white;margin:0px;line-height:normal">
<u><span style="margin:0px;color:black;font-family:Courant;font-size:12pt">Phone:</span></u><span style="margin:0px;color:black;font-family:Courant;font-size:12pt"> (386)-843-1829</span></p>
<p style="background:none 0% 0% repeat scroll white;margin:0px;line-height:normal">
<u><span style="margin:0px;color:black;font-family:Courant;font-size:12pt">Email(s):</span></u><span style="margin:0px;color:black;font-family:Courant;font-size:12pt"> <a href="mailto:ferranj2@my.erau.edu" target="_blank">ferranj2@my.erau.edu</a></span></p>
<p style="background:none 0% 0% repeat scroll white;margin:0px;line-height:normal;text-indent:0.5in">
<span style="margin:0px;color:black;font-family:Courant;font-size:12pt"><span style="margin:0px">
</span><a href="mailto:jesus.ferrand@gmail.com" target="_blank">jesus.ferrand@gmail.com</a></span></p>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote></div></div>