[petsc-users] Hypre freezing with Mat type mpiaij and Vec type mpi

[Junchao Zhang]

On Fri, Jan 5, 2024 at 9:33 AM Edoardo Centofanti <
edoardo.centofanti01 at universitadipavia.it> wrote:

>
>
> Il giorno ven 5 gen 2024 alle ore 16:21 Junchao Zhang <
> junchao.zhang at gmail.com> ha scritto:
>
>>
>>
>>
>> On Fri, Jan 5, 2024 at 4:40 AM Edoardo Centofanti <
>> edoardo.centofanti01 at universitadipavia.it> wrote:
>>
>>> Dear all,
>>>
>>> I have a code running both on GPU and CPU. This code has both cuda
>>> kernels and calls to PETSc KSP and related PC. In particular, I am trying
>>> to perform tests with Hypre BoomerAMG both on CPU and GPU. In order to do
>>> that, on CPU I am running the code with -mat_type mpiaij and -vec_type mpi,
>>> while on GPU I am using respectively aijcusparse and cuda.
>>>
>>> The configuration ran for PETSc (version is 3.20) is
>>> ./configure PETSC_ARCH=arch-linux-cuda --with-cuda --download-mumps
>>> --download-hypre --with-debugging=0 --download-scalapack
>>> --download-parmetis --download-metis --download-fblaslapack=1
>>> --download-mpich --download-make --download-cmake
>>>
>>> My problem is that when I try to run my code on GPU it works well, while
>>> on CPU with mat_type mpiaij and -vec_type mpi it works regularly until the
>>> call to Hypre, then freezes (I have to kill it myself), while with GAMG it
>>> works on CPU with the same configuration (and the same code, just PC is
>>> changed).
>>>
>> Probably because if hypre is configured with GPU support, even petsc is
>> running on CPU (with -mat_type mpiaij and -vec_type mpi), hypre is still
>> trying to run on GPU, causing the hanging problem. The petsc/hypre
>> interface is not able to support both CPU/GPU with the same build.
>>
>>
>>> On another machine running PETSc version 3.17 everything worked smoothly
>>> with the same code and the same configuration, also on Hypre.
>>>
>> I guess with petsc-3.17, you also used a different hypre version
>> (downloaded by petsc).
>>
> Yes, this is exactly what I concluded after a bit of debugging and
> recompiling. In the new version it seems that CPU/GPU with the same build
> is not supported anymore. Since the CUDA kernels I have in my code do not
> interfere with Hypre, the only workaround I found was removing kernels from
> my code and performing an only-CPU run. It slows down a bit overall, but
> nothing dramatic. Anyway, in another test with version 3.18 I encountered
> the same problem, but the program crashes and throws "Invalid options"
> error, while in 3.20 it just hangs.
>
Improving the petsc/hypre interface to support both CPU/GPU is WIP.


> Thanks,
> Edoardo
>
>>
>>
>>
>>> Do you have any insights on what is happening?
>>>
>>> Best regards,
>>> Edoardo
>>>
>>>
>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20240105/4250aa0c/attachment.html>