[petsc-users] Matvecs and KSPSolves with multiple vectors

[Sreeram R Venkat]

Hello Pierre,

I am trying out the KSPMatSolve with the BoomerAMG preconditioner. However,
I am noticing that it is still solving column by column (this is stated
explicitly in the info dump attached). I looked at the code for
KSPMatSolve_Private() and saw that as long as ksp->ops->matsolve is true,
it should do the batched solve, though I'm not sure where that gets set.

I am using the options -pc_type hypre -pc_hypre_type boomeramg when running
the code.

Can you please help me with this?

Thanks,
Sreeram


On Thu, Dec 7, 2023 at 4:04 PM Mark Adams <mfadams at lbl.gov> wrote:

> N.B., AMGX interface is a bit experimental.
> Mark
>
> On Thu, Dec 7, 2023 at 4:11 PM Sreeram R Venkat <srvenkat at utexas.edu>
> wrote:
>
>> Oh, in that case I will try out BoomerAMG. Getting AMGX to build
>> correctly was also tricky so hopefully the HYPRE build will be easier.
>>
>> Thanks,
>> Sreeram
>>
>> On Thu, Dec 7, 2023, 3:03 PM Pierre Jolivet <pierre at joliv.et> wrote:
>>
>>>
>>>
>>> On 7 Dec 2023, at 9:37 PM, Sreeram R Venkat <srvenkat at utexas.edu> wrote:
>>>
>>> Thank you Barry and Pierre; I will proceed with the first option.
>>>
>>> I want to use the AMGX preconditioner for the KSP. I will try it out and
>>> see how it performs.
>>>
>>>
>>> Just FYI, AMGX does not handle systems with multiple RHS, and thus has
>>> no PCMatApply() implementation.
>>> BoomerAMG does, and there is a PCMatApply_HYPRE_BoomerAMG()
>>> implementation.
>>> But let us know if you need assistance figuring things out.
>>>
>>> Thanks,
>>> Pierre
>>>
>>> Thanks,
>>> Sreeram
>>>
>>> On Thu, Dec 7, 2023 at 2:02 PM Pierre Jolivet <pierre at joliv.et> wrote:
>>>
>>>> To expand on Barry’s answer, we have observed repeatedly that
>>>> MatMatMult with MatAIJ performs better than MatMult with MatMAIJ, you can
>>>> reproduce this on your own with
>>>> https://petsc.org/release/src/mat/tests/ex237.c.html.
>>>> Also, I’m guessing you are using some sort of preconditioner within
>>>> your KSP.
>>>> Not all are “KSPMatSolve-ready”, i.e., they may treat blocks of
>>>> right-hand sides column by column, which is very inefficient.
>>>> You could run your code with -info dump and send us dump.0 to see what
>>>> needs to be done on our end to make things more efficient, should you not
>>>> be satisfied with the current performance of the code.
>>>>
>>>> Thanks,
>>>> Pierre
>>>>
>>>> On 7 Dec 2023, at 8:34 PM, Barry Smith <bsmith at petsc.dev> wrote:
>>>>
>>>>
>>>>
>>>> On Dec 7, 2023, at 1:17 PM, Sreeram R Venkat <srvenkat at utexas.edu>
>>>> wrote:
>>>>
>>>> I have 2 sequential matrices M and R (both MATSEQAIJCUSPARSE of size n
>>>> x n) and a vector v of size n*m. v = [v_1 , v_2 ,... , v_m] where v_i has
>>>> size n. The data for v can be stored either in column-major or row-major
>>>> order.  Now, I want to do 2 types of operations:
>>>>
>>>> 1. Matvecs of the form M*v_i = w_i, for i = 1..m.
>>>> 2. KSPSolves of the form R*x_i = v_i, for i = 1..m.
>>>>
>>>> From what I have read on the documentation, I can think of 2
>>>> approaches.
>>>>
>>>> 1. Get the pointer to the data in v (column-major) and use it to create
>>>> a dense matrix V. Then do a MatMatMult with M*V = W, and take the data
>>>> pointer of W to create the vector w. For KSPSolves, use KSPMatSolve with R
>>>> and V.
>>>>
>>>> 2. Create a MATMAIJ using M/R and use that for matvecs directly with
>>>> the vector v. I don't know if KSPSolve with the MATMAIJ will know that it
>>>> is a multiple RHS system and act accordingly.
>>>>
>>>> Which would be the more efficient option?
>>>>
>>>>
>>>> Use 1.
>>>>
>>>>
>>>> As a side-note, I am also wondering if there is a way to use row-major
>>>> storage of the vector v.
>>>>
>>>>
>>>> No
>>>>
>>>> The reason is that this could allow for more coalesced memory access
>>>> when doing matvecs.
>>>>
>>>>
>>>>   PETSc matrix-vector products use BLAS GMEV matrix-vector products for
>>>> the computation so in theory they should already be well-optimized
>>>>
>>>>
>>>> Thanks,
>>>> Sreeram
>>>>
>>>>
>>>>
>>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20231213/c6559e20/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: dump.0
Type: application/octet-stream
Size: 1940230 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20231213/c6559e20/attachment-0001.obj>