<div dir="ltr">Hello Pierre,<div><br></div><div>I am trying out the KSPMatSolve with the BoomerAMG preconditioner. However, I am noticing that it is still solving column by column (this is stated explicitly in the info dump attached). I looked at the code for KSPMatSolve_Private() and saw that as long as <span style="color:rgb(0,0,0)">ksp->ops->matsolve is true, it should do the batched solve, though I'm not sure where that gets set. </span></div><div><span style="color:rgb(0,0,0)"><br></span></div><div><span style="color:rgb(0,0,0)">I am using the options -pc_type hypre -pc_hypre_type boomeramg when running the code.</span></div><div><span style="color:rgb(0,0,0)"><br></span></div><div><span style="color:rgb(0,0,0)">Can you please help me with this?</span></div><div><span style="color:rgb(0,0,0)"><br></span></div><div><span style="color:rgb(0,0,0)">Thanks,</span></div><div><span style="color:rgb(0,0,0)">Sreeram</span></div><div><span style="color:rgb(0,0,0)"><br></span></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Thu, Dec 7, 2023 at 4:04 PM Mark Adams <<a href="mailto:mfadams@lbl.gov">mfadams@lbl.gov</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">N.B., AMGX interface is a bit experimental.<div><div>Mark</div></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Thu, Dec 7, 2023 at 4:11 PM Sreeram R Venkat <<a href="mailto:srvenkat@utexas.edu" target="_blank">srvenkat@utexas.edu</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="auto">Oh, in that case I will try out BoomerAMG. Getting AMGX to build correctly was also tricky so hopefully the HYPRE build will be easier.<div dir="auto"><br></div><div dir="auto">Thanks,</div><div dir="auto">Sreeram</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Thu, Dec 7, 2023, 3:03 PM Pierre Jolivet <<a href="mailto:pierre@joliv.et" rel="noreferrer" target="_blank">pierre@joliv.et</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div><br id="m_4661579025182849266m_858150001532199206m_7082342255457086237m_-8161104843020054705lineBreakAtBeginningOfMessage"><div><br><blockquote type="cite"><div>On 7 Dec 2023, at 9:37 PM, Sreeram R Venkat <<a href="mailto:srvenkat@utexas.edu" rel="noreferrer noreferrer" target="_blank">srvenkat@utexas.edu</a>> wrote:</div><br><div><div dir="ltr">Thank you Barry and Pierre; I will proceed with the first option. <div><br></div><div>I want to use the AMGX preconditioner for the KSP. I will try it out and see how it performs.</div></div></div></blockquote><div><br></div><div>Just FYI, AMGX does not handle systems with multiple RHS, and thus has no PCMatApply() implementation.</div><div>BoomerAMG does, and there is a PCMatApply_HYPRE_BoomerAMG() implementation.</div><div>But let us know if you need assistance figuring things out.</div><div><br></div><div>Thanks,</div><div>Pierre</div><br><blockquote type="cite"><div><div dir="ltr"><div>Thanks,</div><div>Sreeram</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Thu, Dec 7, 2023 at 2:02 PM Pierre Jolivet <<a href="mailto:pierre@joliv.et" rel="noreferrer noreferrer" target="_blank">pierre@joliv.et</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div>To expand on Barry’s answer, we have observed repeatedly that MatMatMult with MatAIJ performs better than MatMult with MatMAIJ, you can reproduce this on your own with <a href="https://petsc.org/release/src/mat/tests/ex237.c.html" rel="noreferrer noreferrer" target="_blank">https://petsc.org/release/src/mat/tests/ex237.c.html</a>.<div>Also, I’m guessing you are using some sort of preconditioner within your KSP.</div><div>Not all are “KSPMatSolve-ready”, i.e., they may treat blocks of right-hand sides column by column, which is very inefficient.</div><div>You could run your code with -info dump and send us dump.0 to see what needs to be done on our end to make things more efficient, should you not be satisfied with the current performance of the code.</div><div><br></div><div>Thanks,</div><div>Pierre<br><div><br><div><blockquote type="cite"><div>On 7 Dec 2023, at 8:34 PM, Barry Smith <<a href="mailto:bsmith@petsc.dev" rel="noreferrer noreferrer" target="_blank">bsmith@petsc.dev</a>> wrote:</div><br><div><div style="font-family:Helvetica;font-size:12px;font-style:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none"><br><br><blockquote type="cite"><div>On Dec 7, 2023, at 1:17 PM, Sreeram R Venkat <<a href="mailto:srvenkat@utexas.edu" rel="noreferrer noreferrer" target="_blank">srvenkat@utexas.edu</a>> wrote:</div><br><div><div dir="ltr"><div dir="ltr" class="gmail_signature"><div dir="ltr"><div style="color:rgb(34,34,34)">I have 2 sequential matrices M and R (both MATSEQAIJCUSPARSE of size n x n) and a vector v of size n*m. v = [v_1 , v_2 ,... , v_m] where v_i has size n. The data for v can be stored either in column-major or row-major order. Now, I want to do 2 types of operations:</div><div style="color:rgb(34,34,34)"><br></div><div style="color:rgb(34,34,34)">1. Matvecs of the form M*v_i = w_i, for i = 1..m. </div><div style="color:rgb(34,34,34)">2. KSPSolves of the form R*x_i = v_i, for i = 1..m.</div><div style="color:rgb(34,34,34)"><br></div><div style="color:rgb(34,34,34)">From what I have read on the documentation, I can think of 2 approaches. </div><div style="color:rgb(34,34,34)"><br></div><div style="color:rgb(34,34,34)">1. Get the pointer to the data in v (column-major) and use it to create a dense matrix V. Then do a MatMatMult with M*V = W, and take the data pointer of W to create the vector w. For KSPSolves, use KSPMatSolve with R and V.</div><div style="color:rgb(34,34,34)"><br></div><div style="color:rgb(34,34,34)">2. Create a MATMAIJ using M/R and use that for matvecs directly with the vector v. I don't know if KSPSolve with the MATMAIJ will know that it is a multiple RHS system and act accordingly.</div><div style="color:rgb(34,34,34)"><br></div><div style="color:rgb(34,34,34)">Which would be the more efficient option?</div></div></div></div></div></blockquote><div><br></div>Use 1. <br><blockquote type="cite"><div><div dir="ltr"><div dir="ltr" class="gmail_signature"><div dir="ltr"><div style="color:rgb(34,34,34)"><br></div><div style="color:rgb(34,34,34)">As a side-note, I am also wondering if there is a way to use row-major storage of the vector v.</div></div></div></div></div></blockquote><div><br></div>No</div><div style="font-family:Helvetica;font-size:12px;font-style:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none"><br><blockquote type="cite"><div><div dir="ltr"><div dir="ltr" class="gmail_signature"><div dir="ltr"><div style="color:rgb(34,34,34)">The reason is that this could allow for more coalesced memory access when doing matvecs.</div></div></div></div></div></blockquote><div><br></div> PETSc matrix-vector products use BLAS GMEV matrix-vector products for the computation so in theory they should already be well-optimized</div><div style="font-family:Helvetica;font-size:12px;font-style:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none"><br><blockquote type="cite"><div><div dir="ltr"><div dir="ltr" class="gmail_signature"><div dir="ltr"><div style="color:rgb(34,34,34)"><br></div><div style="color:rgb(34,34,34)">Thanks,</div><div style="color:rgb(34,34,34)">Sreeram</div></div></div></div></div></blockquote></div></div></blockquote></div><br></div></div></div></blockquote></div>
</div></blockquote></div><br></div></blockquote></div>
</blockquote></div>
</blockquote></div>