<html><head><meta http-equiv="Content-Type" content="text/html; charset=us-ascii"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class=""><div class=""><br class=""></div> My mistake, yes rank 0 does not need to put any values of x into slvec0. rank 1 and 2 need to put parts of y and w in because they will be needed by the other ranks.<div class=""><br class=""></div><div class=""><br class=""><div><br class=""><blockquote type="cite" class=""><div class="">On Mar 23, 2022, at 4:51 PM, Sam Guo <<a href="mailto:sam.guo@cd-adapco.com" class="">sam.guo@cd-adapco.com</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><div dir="ltr" class=""><div class="">Hi Barry,</div><div class=""> I try to understand your example. Why does Rank 0 put the values of x into slvec0? x is needed on rank 0. Other ranks need B'x and D'x. Is it because we need
slvec0 to be the same size as (x,y,z) on all ranks?</div><div class=""><br class=""></div><div class="">Thanks,</div><div class="">Sam</div><div class=""><br class=""></div></div><br class=""><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, Mar 22, 2022 at 2:24 PM Sam Guo <<a href="mailto:sam.guo@cd-adapco.com" class="">sam.guo@cd-adapco.com</a>> wrote:<br class=""></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr" class="">Hi Barry,<div class=""> This is the total memory summed over all the ranks.</div><div class=""> Same problem size on different np.</div><div class=""> I call MUMPS in parallel with distributed input and centralized rhs.</div><div class=""><br class=""></div><div class="">Thanks,</div><div class="">Sam</div></div><br class=""><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, Mar 22, 2022 at 2:11 PM Barry Smith <<a href="mailto:bsmith@petsc.dev" target="_blank" class="">bsmith@petsc.dev</a>> wrote:<br class=""></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div class=""><div class=""><br class=""></div><div class=""> I don't understand the numbers in the table.</div><div class=""><br class=""></div><div class=""> Is this memory summed over all the ranks or the maximum over the ranks?<br class=""></div><div class=""><br class=""></div><div class=""> Is this for the same problem size on different np or are you increasing the problem size with more ranks?</div><div class=""><br class=""></div><div class=""> Are you storing and factoring the matrix on each rank or is the solution of a single linear system done in parallel?</div><div class=""><br class=""></div> <div class=""><br class=""></div><div class=""> <div class=""> <br class=""><div class=""><br class=""></div><div class=""><br class=""><div class=""><br class=""><blockquote type="cite" class=""><div class="">On Mar 22, 2022, at 4:16 PM, Sam Guo <<a href="mailto:sam.guo@cd-adapco.com" target="_blank" class="">sam.guo@cd-adapco.com</a>> wrote:</div><br class=""><div class=""><div dir="ltr" class="">Here is one memory comparison (memory in MB)<div class=""><table style="border-collapse:collapse;margin:5px 0px 5px 2px;width:auto;color:rgb(23,43,77);font-family:-apple-system,BlinkMacSystemFont,"Segoe UI",Roboto,Oxygen,Ubuntu,"Fira Sans","Droid Sans","Helvetica Neue",sans-serif;font-size:14px" class=""><tbody class=""><tr class=""><th style="border:1px solid rgb(193,199,208);background:rgb(244,245,247);padding:3px 4px;text-align:center" class=""> </th><th style="border:1px solid rgb(193,199,208);background:rgb(244,245,247);padding:3px 4px;text-align:center" class="">np=1</th><th style="border:1px solid rgb(193,199,208);background:rgb(244,245,247);padding:3px 4px;text-align:center" class="">np=2</th><th style="border:1px solid rgb(193,199,208);background:rgb(244,245,247);padding:3px 4px;text-align:center" class="">np=4</th><th style="border:1px solid rgb(193,199,208);background:rgb(244,245,247);padding:3px 4px;text-align:center" class="">np=8</th><th style="border:1px solid rgb(193,199,208);background:rgb(244,245,247);padding:3px 4px;text-align:center" class="">np=16</th></tr><tr class=""><td style="border:1px solid rgb(193,199,208);padding:3px 4px" class="">shell</td><td style="border:1px solid rgb(193,199,208);padding:3px 4px" class="">1614</td><td style="border:1px solid rgb(193,199,208);padding:3px 4px" class="">1720</td><td style="border:1px solid rgb(193,199,208);padding:3px 4px" class="">1874</td><td style="border:1px solid rgb(193,199,208);padding:3px 4px" class="">1673</td><td style="border:1px solid rgb(193,199,208);padding:3px 4px" class="">1248</td></tr><tr class=""><td style="border:1px solid rgb(193,199,208);padding:3px 4px" class="">PETSc(using full matrix)</td><td style="border:1px solid rgb(193,199,208);padding:3px 4px" class="">2108</td><td style="border:1px solid rgb(193,199,208);padding:3px 4px" class="">2260</td><td style="border:1px solid rgb(193,199,208);padding:3px 4px" class="">2364</td><td style="border:1px solid rgb(193,199,208);padding:3px 4px" class="">2215</td><td style="border:1px solid rgb(193,199,208);padding:3px 4px" class="">1734</td></tr><tr class=""><td style="border:1px solid rgb(193,199,208);padding:3px 4px" class="">PETSc(using symmetric matrix)</td><td style="border:1px solid rgb(193,199,208);padding:3px 4px" class="">1750</td><td style="border:1px solid rgb(193,199,208);padding:3px 4px" class="">2100</td><td style="border:1px solid rgb(193,199,208);padding:3px 4px" class="">2189</td><td style="border:1px solid rgb(193,199,208);padding:3px 4px" class="">2094</td><td style="border:1px solid rgb(193,199,208);padding:3px 4px" class="">1727</td></tr></tbody></table>Those are the total water mark memory added. </div></div><br class=""><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, Mar 22, 2022 at 1:10 PM Barry Smith <<a href="mailto:bsmith@petsc.dev" target="_blank" class="">bsmith@petsc.dev</a>> wrote:<br class=""></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div class=""><div class=""><br class=""></div><div class=""> Sam,</div><div class=""><br class=""></div> MUMPS is a direct solver, as such, it requires much more memory than the original matrix (stored as a PETSc matrix) to store the factored matrix. The savings you will get by not having a PETSc copy of the matrix and a MUMPS copy of the matrix at the same time is unlikely to be significant. Do you have memory footprint measurements indicating that not having the PETSc copy of the matrix in memory will allow you to run measurably larger simulations? <div class=""><br class=""></div><div class=""> Barry</div><div class=""><br class=""><div class=""><br class=""></div><div class=""> <br class=""><div class=""><br class=""><blockquote type="cite" class=""><div class="">On Mar 22, 2022, at 3:58 PM, Sam Guo <<a href="mailto:sam.guo@cd-adapco.com" target="_blank" class="">sam.guo@cd-adapco.com</a>> wrote:</div><br class=""><div class=""><div dir="ltr" class="">The reason I want to use shell matrix is to reduce memory footprint. If I create a PETSc matrix and use MUMPS, I understand PETSc will create another copy of the matrix for MUMPS. Is there any way to avoid the extra copy of MUMPS?</div><br class=""><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, Mar 22, 2022 at 12:53 PM Sam Guo <<a href="mailto:sam.guo@cd-adapco.com" target="_blank" class="">sam.guo@cd-adapco.com</a>> wrote:<br class=""></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr" class="">Barry,<div class=""> Thanks for the illustration. Is there an easy way to mimic the implementation using shell matrix? I have been studying how the sMvctx is created and it seems pretty involved.</div><div class=""><br class=""></div><div class="">Thanks,</div><div class="">Sam</div></div><br class=""><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, Mar 21, 2022 at 2:48 PM Barry Smith <<a href="mailto:bsmith@petsc.dev" target="_blank" class="">bsmith@petsc.dev</a>> wrote:<br class=""></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div class=""><br class=""><div class=""><br class=""><blockquote type="cite" class=""><div class="">On Mar 21, 2022, at 4:36 PM, Sam Guo <<a href="mailto:sam.guo@cd-adapco.com" target="_blank" class="">sam.guo@cd-adapco.com</a>> wrote:</div><br class=""><div class=""><div dir="ltr" class="">Barry,<div class=""> Thanks. Could you elaborate? I try to implement the matrix-vector multiplication for a symmetric matrix using shell matrix.</div></div></div></blockquote><div class=""><br class=""></div><div class=""> Consider with three ranks </div><div class=""><br class=""></div> (a) = ( A B D) (x)</div><div class=""> (b) ( B' C E) (y)</div><div class=""> (c) ( D' E' F) (w)</div><div class=""><br class=""></div><div class=""> Only the ones without the ' are stored on the rank. So for example B is stored on rank 0.</div><div class=""><br class=""></div><div class=""> Rank 0 computes A x and keeps it in a. Rank 1 computes Cy and keeps it in b Rank 2 computes Fw and keeps it in c</div><div class=""><br class=""></div><div class=""> Rank 0 computes B'x and D'x. It puts the nonzero entries of these values as well as the values of x into slvec0</div><div class=""><br class=""></div><div class=""> Rank 1 computes E'y and puts the nonzero entries as well as the values into slvec0</div><div class=""><br class=""></div><div class=""> Rank 2 puts the values of we needed by the other ranks into slvec0</div><div class=""><br class=""></div><div class=""> Rank 0 does B y_h + D z_h where it gets the y_h and z_h values from slvec1 and adds it to a</div><div class=""><br class=""></div><div class=""> Rank 1 takes the B'x from slvec1 and adds it to b it then takes the E y_h values where the y_h are pulled from slvec1 and adds them b</div><div class=""><br class=""></div><div class=""> Rank 2 takes the B'x and E'y from slvec0 and adds them to c. </div><div class=""><br class=""></div><div class=""><br class=""></div><div class=""><blockquote type="cite" class=""><div class=""><div dir="ltr" class=""><div class=""><br class=""></div><div class="">Thanks,</div><div class="">Sam</div></div><br class=""><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, Mar 21, 2022 at 12:56 PM Barry Smith <<a href="mailto:bsmith@petsc.dev" target="_blank" class="">bsmith@petsc.dev</a>> wrote:<br class=""></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div class=""><div class=""><br class=""></div> The "trick" is that though "more" communication is needed to complete the product the communication can still be done in a single VecScatter instead of two separate calls to VecScatter. We simply pack both pieces of information that needs to be sent into a single vector. <div class=""><br class=""></div><div class=""><pre width="80" class=""> <font color="#B22222" class="">/* copy x into the vec slvec0 */</font>
<a name="m_-8997436359524451786_m_5408433206362835710_m_-6721792415453726673_m_-6880554628075345529_m_-6002104729051566256_m_-1423813778787502823_line1111" class="">1111: </a> <a href="https://petsc.org/main/docs/manualpages/Vec/VecGetArray.html#VecGetArray" target="_blank" class="">VecGetArray</a>(a->slvec0,&from);
<a name="m_-8997436359524451786_m_5408433206362835710_m_-6721792415453726673_m_-6880554628075345529_m_-6002104729051566256_m_-1423813778787502823_line1112" class="">1112: </a> <a href="https://petsc.org/main/docs/manualpages/Vec/VecGetArrayRead.html#VecGetArrayRead" target="_blank" class="">VecGetArrayRead</a>(xx,&x);
<a name="m_-8997436359524451786_m_5408433206362835710_m_-6721792415453726673_m_-6880554628075345529_m_-6002104729051566256_m_-1423813778787502823_line1114" class="">1114: </a> <a href="https://petsc.org/main/docs/manualpages/Sys/PetscArraycpy.html#PetscArraycpy" target="_blank" class="">PetscArraycpy</a>(from,x,bs*mbs);
<a name="m_-8997436359524451786_m_5408433206362835710_m_-6721792415453726673_m_-6880554628075345529_m_-6002104729051566256_m_-1423813778787502823_line1115" class="">1115: </a> <a href="https://petsc.org/main/docs/manualpages/Vec/VecRestoreArray.html#VecRestoreArray" target="_blank" class="">VecRestoreArray</a>(a->slvec0,&from);
<a name="m_-8997436359524451786_m_5408433206362835710_m_-6721792415453726673_m_-6880554628075345529_m_-6002104729051566256_m_-1423813778787502823_line1116" class="">1116: </a> <a href="https://petsc.org/main/docs/manualpages/Vec/VecRestoreArrayRead.html#VecRestoreArrayRead" target="_blank" class="">VecRestoreArrayRead</a>(xx,&x);
<a name="m_-8997436359524451786_m_5408433206362835710_m_-6721792415453726673_m_-6880554628075345529_m_-6002104729051566256_m_-1423813778787502823_line1118" class="">1118: </a> <a href="https://petsc.org/main/docs/manualpages/PetscSF/VecScatterBegin.html#VecScatterBegin" target="_blank" class="">VecScatterBegin</a>(a->sMvctx,a->slvec0,a->slvec1,<a href="https://petsc.org/main/docs/manualpages/Sys/ADD_VALUES.html#ADD_VALUES" target="_blank" class="">ADD_VALUES</a>,<a href="https://petsc.org/main/docs/manualpages/Vec/SCATTER_FORWARD.html#SCATTER_FORWARD" target="_blank" class="">SCATTER_FORWARD</a>);
<a name="m_-8997436359524451786_m_5408433206362835710_m_-6721792415453726673_m_-6880554628075345529_m_-6002104729051566256_m_-1423813778787502823_line1119" class="">1119: </a> <a href="https://petsc.org/main/docs/manualpages/PetscSF/VecScatterEnd.html#VecScatterEnd" target="_blank" class="">VecScatterEnd</a>(a->sMvctx,a->slvec0,a->slvec1,<a href="https://petsc.org/main/docs/manualpages/Sys/ADD_VALUES.html#ADD_VALUES" target="_blank" class="">ADD_VALUES</a>,<a href="https://petsc.org/main/docs/manualpages/Vec/SCATTER_FORWARD.html#SCATTER_FORWARD" target="_blank" class="">SCATTER_FORWARD</a>);</pre><div class="">If you create two symmetric matrices, one with SBAIJ and one with BAIJ and compare the time to do the product you will find that the SBAIJ is not significantly slower but does save memory.</div><div class=""><br class=""></div><div class=""><br class=""><blockquote type="cite" class=""><div class="">On Mar 21, 2022, at 3:26 PM, Sam Guo <<a href="mailto:sam.guo@cd-adapco.com" target="_blank" class="">sam.guo@cd-adapco.com</a>> wrote:</div><br class=""><div class=""><div dir="ltr" class="">Using following example from the
<span style="font-family:"Times New Roman";font-size:inherit" class="">MatCreateSBAIJ documentation</span><div class=""><pre class=""> 0 1 2 3 4 5 6 7 8 9 10 11
--------------------------
row 3 |. . . d d d o o o o o o
row 4 |. . . d d d o o o o o o
row 5 |. . . d d d o o o o o o
--------------------------</pre><pre class=""><br class=""></pre><pre class="">On a processor that owns rows 3, 4 and 5, rows 0-2 info are still needed. Is is that the processor that owns rows 0-2 will apply B symmetrical and then send the result </pre><pre class="">to the processor that owns 3-5?</pre><pre class=""></pre></div></div><br class=""><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, Mar 21, 2022 at 12:14 PM Mark Adams <<a href="mailto:mfadams@lbl.gov" target="_blank" class="">mfadams@lbl.gov</a>> wrote:<br class=""></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr" class="">PETSc stores parallel matrices as two serial matrices. One for the diagonal (d or A) block and one for the rest (o or B).<div class="">I would guess that for symmetric matrices it has a symmetric matrix for the diagonal and a full AIJ matrix for the (upper) off-diagonal.</div><div class="">So the <span style="white-space:pre-wrap" class="">multtranspose is applying B symmetrically. This lower off-diagonal and the diagonal block can be done without communication.</span></div><div class=""><span style="white-space:pre-wrap" class="">Then the off processor values are collected, and the upper off-diagonal is applied.</span></div></div><br class=""><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, Mar 21, 2022 at 2:35 PM Sam Guo <<a href="mailto:sam.guo@cd-adapco.com" target="_blank" class="">sam.guo@cd-adapco.com</a>> wrote:<br class=""></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr" class="">I am most interested in how the lower triangular part is redistributed. It seems that SBAJI saves memory but requires more communication than BAIJ.</div><br class=""><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, Mar 21, 2022 at 11:27 AM Sam Guo <<a href="mailto:sam.guo@cd-adapco.com" target="_blank" class="">sam.guo@cd-adapco.com</a>> wrote:<br class=""></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr" class="">Mark, thanks for the quick response. I am more interested in parallel implementation of MatMult for SBAIJ. I found following<div class=""><pre width="80" class=""><a name="m_-8997436359524451786_m_5408433206362835710_m_-6721792415453726673_m_-6880554628075345529_m_-6002104729051566256_m_-1423813778787502823_m_-1097518819297141460_m_2758397486421008689_m_-8016313848878634063_line1094" class="">1094: </a><strong class=""><font color="#4169E1" class=""><a name="m_-8997436359524451786_m_5408433206362835710_m_-6721792415453726673_m_-6880554628075345529_m_-6002104729051566256_m_-1423813778787502823_m_-1097518819297141460_m_2758397486421008689_m_-8016313848878634063_MatMult_MPISBAIJ" class=""></a><a href="https://petsc.org/main/docs/manualpages/Sys/PetscErrorCode.html#PetscErrorCode" target="_blank" class="">PetscErrorCode</a> MatMult_MPISBAIJ(<a href="https://petsc.org/main/docs/manualpages/Mat/Mat.html#Mat" target="_blank" class="">Mat</a> A,<a href="https://petsc.org/main/docs/manualpages/Vec/Vec.html#Vec" target="_blank" class="">Vec</a> xx,<a href="https://petsc.org/main/docs/manualpages/Vec/Vec.html#Vec" target="_blank" class="">Vec</a> yy)</font></strong>
<a name="m_-8997436359524451786_m_5408433206362835710_m_-6721792415453726673_m_-6880554628075345529_m_-6002104729051566256_m_-1423813778787502823_m_-1097518819297141460_m_2758397486421008689_m_-8016313848878634063_line1095" class="">1095: </a>{
<a name="m_-8997436359524451786_m_5408433206362835710_m_-6721792415453726673_m_-6880554628075345529_m_-6002104729051566256_m_-1423813778787502823_m_-1097518819297141460_m_2758397486421008689_m_-8016313848878634063_line1096" class="">1096: </a> Mat_MPISBAIJ *a = (Mat_MPISBAIJ*)A->data;
<a name="m_-8997436359524451786_m_5408433206362835710_m_-6721792415453726673_m_-6880554628075345529_m_-6002104729051566256_m_-1423813778787502823_m_-1097518819297141460_m_2758397486421008689_m_-8016313848878634063_line1097" class="">1097: </a> <a href="https://petsc.org/main/docs/manualpages/Sys/PetscErrorCode.html#PetscErrorCode" target="_blank" class="">PetscErrorCode</a> ierr;
<a name="m_-8997436359524451786_m_5408433206362835710_m_-6721792415453726673_m_-6880554628075345529_m_-6002104729051566256_m_-1423813778787502823_m_-1097518819297141460_m_2758397486421008689_m_-8016313848878634063_line1098" class="">1098: </a> <a href="https://petsc.org/main/docs/manualpages/Sys/PetscInt.html#PetscInt" target="_blank" class="">PetscInt</a> mbs=a->mbs,bs=A->rmap->bs;
<a name="m_-8997436359524451786_m_5408433206362835710_m_-6721792415453726673_m_-6880554628075345529_m_-6002104729051566256_m_-1423813778787502823_m_-1097518819297141460_m_2758397486421008689_m_-8016313848878634063_line1099" class="">1099: </a> <a href="https://petsc.org/main/docs/manualpages/Sys/PetscScalar.html#PetscScalar" target="_blank" class="">PetscScalar</a> *from;
<a name="m_-8997436359524451786_m_5408433206362835710_m_-6721792415453726673_m_-6880554628075345529_m_-6002104729051566256_m_-1423813778787502823_m_-1097518819297141460_m_2758397486421008689_m_-8016313848878634063_line1100" class="">1100: </a> const <a href="https://petsc.org/main/docs/manualpages/Sys/PetscScalar.html#PetscScalar" target="_blank" class="">PetscScalar</a> *x;
<a name="m_-8997436359524451786_m_5408433206362835710_m_-6721792415453726673_m_-6880554628075345529_m_-6002104729051566256_m_-1423813778787502823_m_-1097518819297141460_m_2758397486421008689_m_-8016313848878634063_line1103" class="">1103: </a> <font color="#B22222" class="">/* diagonal part */</font>
<a name="m_-8997436359524451786_m_5408433206362835710_m_-6721792415453726673_m_-6880554628075345529_m_-6002104729051566256_m_-1423813778787502823_m_-1097518819297141460_m_2758397486421008689_m_-8016313848878634063_line1104" class="">1104: </a> (*a->A->ops->mult)(a->A,xx,a->slvec1a);
<a name="m_-8997436359524451786_m_5408433206362835710_m_-6721792415453726673_m_-6880554628075345529_m_-6002104729051566256_m_-1423813778787502823_m_-1097518819297141460_m_2758397486421008689_m_-8016313848878634063_line1105" class="">1105: </a> <a href="https://petsc.org/main/docs/manualpages/Vec/VecSet.html#VecSet" target="_blank" class="">VecSet</a>(a->slvec1b,0.0);
<a name="m_-8997436359524451786_m_5408433206362835710_m_-6721792415453726673_m_-6880554628075345529_m_-6002104729051566256_m_-1423813778787502823_m_-1097518819297141460_m_2758397486421008689_m_-8016313848878634063_line1107" class="">1107: </a> <font color="#B22222" class="">/* subdiagonal part */</font>
<a name="m_-8997436359524451786_m_5408433206362835710_m_-6721792415453726673_m_-6880554628075345529_m_-6002104729051566256_m_-1423813778787502823_m_-1097518819297141460_m_2758397486421008689_m_-8016313848878634063_line1108" class="">1108: </a> (*a->B->ops->multtranspose)(a->B,xx,a->slvec0b);
<a name="m_-8997436359524451786_m_5408433206362835710_m_-6721792415453726673_m_-6880554628075345529_m_-6002104729051566256_m_-1423813778787502823_m_-1097518819297141460_m_2758397486421008689_m_-8016313848878634063_line1110" class="">1110: </a> <font color="#B22222" class="">/* copy x into the vec slvec0 */</font>
<a name="m_-8997436359524451786_m_5408433206362835710_m_-6721792415453726673_m_-6880554628075345529_m_-6002104729051566256_m_-1423813778787502823_m_-1097518819297141460_m_2758397486421008689_m_-8016313848878634063_line1111" class="">1111: </a> <a href="https://petsc.org/main/docs/manualpages/Vec/VecGetArray.html#VecGetArray" target="_blank" class="">VecGetArray</a>(a->slvec0,&from);
<a name="m_-8997436359524451786_m_5408433206362835710_m_-6721792415453726673_m_-6880554628075345529_m_-6002104729051566256_m_-1423813778787502823_m_-1097518819297141460_m_2758397486421008689_m_-8016313848878634063_line1112" class="">1112: </a> <a href="https://petsc.org/main/docs/manualpages/Vec/VecGetArrayRead.html#VecGetArrayRead" target="_blank" class="">VecGetArrayRead</a>(xx,&x);
<a name="m_-8997436359524451786_m_5408433206362835710_m_-6721792415453726673_m_-6880554628075345529_m_-6002104729051566256_m_-1423813778787502823_m_-1097518819297141460_m_2758397486421008689_m_-8016313848878634063_line1114" class="">1114: </a> <a href="https://petsc.org/main/docs/manualpages/Sys/PetscArraycpy.html#PetscArraycpy" target="_blank" class="">PetscArraycpy</a>(from,x,bs*mbs);
<a name="m_-8997436359524451786_m_5408433206362835710_m_-6721792415453726673_m_-6880554628075345529_m_-6002104729051566256_m_-1423813778787502823_m_-1097518819297141460_m_2758397486421008689_m_-8016313848878634063_line1115" class="">1115: </a> <a href="https://petsc.org/main/docs/manualpages/Vec/VecRestoreArray.html#VecRestoreArray" target="_blank" class="">VecRestoreArray</a>(a->slvec0,&from);
<a name="m_-8997436359524451786_m_5408433206362835710_m_-6721792415453726673_m_-6880554628075345529_m_-6002104729051566256_m_-1423813778787502823_m_-1097518819297141460_m_2758397486421008689_m_-8016313848878634063_line1116" class="">1116: </a> <a href="https://petsc.org/main/docs/manualpages/Vec/VecRestoreArrayRead.html#VecRestoreArrayRead" target="_blank" class="">VecRestoreArrayRead</a>(xx,&x);
<a name="m_-8997436359524451786_m_5408433206362835710_m_-6721792415453726673_m_-6880554628075345529_m_-6002104729051566256_m_-1423813778787502823_m_-1097518819297141460_m_2758397486421008689_m_-8016313848878634063_line1118" class="">1118: </a> <a href="https://petsc.org/main/docs/manualpages/PetscSF/VecScatterBegin.html#VecScatterBegin" target="_blank" class="">VecScatterBegin</a>(a->sMvctx,a->slvec0,a->slvec1,<a href="https://petsc.org/main/docs/manualpages/Sys/ADD_VALUES.html#ADD_VALUES" target="_blank" class="">ADD_VALUES</a>,<a href="https://petsc.org/main/docs/manualpages/Vec/SCATTER_FORWARD.html#SCATTER_FORWARD" target="_blank" class="">SCATTER_FORWARD</a>);
<a name="m_-8997436359524451786_m_5408433206362835710_m_-6721792415453726673_m_-6880554628075345529_m_-6002104729051566256_m_-1423813778787502823_m_-1097518819297141460_m_2758397486421008689_m_-8016313848878634063_line1119" class="">1119: </a> <a href="https://petsc.org/main/docs/manualpages/PetscSF/VecScatterEnd.html#VecScatterEnd" target="_blank" class="">VecScatterEnd</a>(a->sMvctx,a->slvec0,a->slvec1,<a href="https://petsc.org/main/docs/manualpages/Sys/ADD_VALUES.html#ADD_VALUES" target="_blank" class="">ADD_VALUES</a>,<a href="https://petsc.org/main/docs/manualpages/Vec/SCATTER_FORWARD.html#SCATTER_FORWARD" target="_blank" class="">SCATTER_FORWARD</a>);
<a name="m_-8997436359524451786_m_5408433206362835710_m_-6721792415453726673_m_-6880554628075345529_m_-6002104729051566256_m_-1423813778787502823_m_-1097518819297141460_m_2758397486421008689_m_-8016313848878634063_line1120" class="">1120: </a> <font color="#B22222" class="">/* supperdiagonal part */</font>
<a name="m_-8997436359524451786_m_5408433206362835710_m_-6721792415453726673_m_-6880554628075345529_m_-6002104729051566256_m_-1423813778787502823_m_-1097518819297141460_m_2758397486421008689_m_-8016313848878634063_line1121" class="">1121: </a> (*a->B->ops->multadd)(a->B,a->slvec1b,a->slvec1a,yy);
<a name="m_-8997436359524451786_m_5408433206362835710_m_-6721792415453726673_m_-6880554628075345529_m_-6002104729051566256_m_-1423813778787502823_m_-1097518819297141460_m_2758397486421008689_m_-8016313848878634063_line1122" class="">1122: </a> <font color="#4169E1" class="">return</font>(0);
<a name="m_-8997436359524451786_m_5408433206362835710_m_-6721792415453726673_m_-6880554628075345529_m_-6002104729051566256_m_-1423813778787502823_m_-1097518819297141460_m_2758397486421008689_m_-8016313848878634063_line1123" class="">1123: </a>}</pre><pre width="80" class=""> I try to understand the algorithm.</pre><pre width="80" class=""><br class=""></pre><pre width="80" class="">Thanks,</pre><pre width="80" class="">Sam</pre></div></div><br class=""><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, Mar 21, 2022 at 11:14 AM Mark Adams <<a href="mailto:mfadams@lbl.gov" target="_blank" class="">mfadams@lbl.gov</a>> wrote:<br class=""></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr" class="">This code looks fine to me and the code is in src/mat/impls/sbaij/seq/sbaij2.c</div><br class=""><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, Mar 21, 2022 at 2:02 PM Sam Guo <<a href="mailto:sam.guo@cd-adapco.com" target="_blank" class="">sam.guo@cd-adapco.com</a>> wrote:<br class=""></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr" class=""><span style="font-family:"Times New Roman";font-size:inherit" class="">Dear PETSc dev team,</span><div class=""><span style="font-family:"Times New Roman";font-size:inherit" class=""> The documentation about MatCreateSBAIJ has following</span></div><div class=""><span style="font-family:"Times New Roman";font-size:inherit" class="">"It is recommended that one use the </span><a href="https://petsc.org/main/docs/manualpages/Mat/MatCreate.html#MatCreate" style="font-family:"Times New Roman";font-size:inherit" target="_blank" class="">MatCreate</a><span style="font-family:"Times New Roman";font-size:inherit" class="">(), </span><a href="https://petsc.org/main/docs/manualpages/Mat/MatSetType.html#MatSetType" style="font-family:"Times New Roman";font-size:inherit" target="_blank" class="">MatSetType</a><span style="font-family:"Times New Roman";font-size:inherit" class="">() and/or </span><a href="https://petsc.org/main/docs/manualpages/Mat/MatSetFromOptions.html#MatSetFromOptions" style="font-family:"Times New Roman";font-size:inherit" target="_blank" class="">MatSetFromOptions</a><span style="font-family:"Times New Roman";font-size:inherit" class="">(), MatXXXXSetPreallocation() paradigm instead of this routine directly. [MatXXXXSetPreallocation() is, for example, </span><a href="https://petsc.org/main/docs/manualpages/Mat/MatSeqAIJSetPreallocation.html#MatSeqAIJSetPreallocation" style="font-family:"Times New Roman";font-size:inherit" target="_blank" class="">MatSeqAIJSetPreallocation</a><span style="font-family:"Times New Roman";font-size:inherit" class="">]"</span><br class=""></div><div class=""><span style="font-family:"Times New Roman";font-size:inherit" class=""> I currently call </span>MatCreateSBAIJ directly as follows:</div><div class="">MatCreateSBAIJ (with d_nnz and o_nnz)<br class="">MatSetValues (to add row by row)</div><div class="">MatAssemblyBegin(A,MAT_FINAL_ASSEMBLY);</div><div class="">MatAssemblyEnd(A,MAT_FINAL_ASSEMBLY);<br class="">MatSetOption(A, MAT_SYMMETRIC, PETSC_TRUE);<span style="font-family:"Times New Roman";font-size:inherit" class=""><br class=""></span></div><div class=""><br class=""></div><div class=""> Two questions:</div><div class=""> (1) I am wondering whether what I am doing is the most efficient. </div><div class=""><br class=""></div><div class=""> (2) I try to find out how the matrix vector multiplication is implemented in PETSc for SBAIJ storage.</div><div class=""><br class=""></div><div class="">Thanks,</div><div class="">Sam</div></div>
</blockquote></div>
</blockquote></div>
</blockquote></div>
</blockquote></div>
</blockquote></div>
</div></blockquote></div><br class=""></div></div></blockquote></div>
</div></blockquote></div><br class=""></div></blockquote></div>
</blockquote></div>
</div></blockquote></div><br class=""></div></div></div></blockquote></div>
</div></blockquote></div><br class=""></div></div></div></div></blockquote></div>
</blockquote></div>
</div></blockquote></div><br class=""></div></body></html>