[petsc-users] Hypre freezing with Mat type mpiaij and Vec type mpi

[Edoardo Centofanti]

Il giorno ven 5 gen 2024 alle ore 16:21 Junchao Zhang <
junchao.zhang at gmail.com> ha scritto:

>
>
>
> On Fri, Jan 5, 2024 at 4:40 AM Edoardo Centofanti <
> edoardo.centofanti01 at universitadipavia.it> wrote:
>
>> Dear all,
>>
>> I have a code running both on GPU and CPU. This code has both cuda
>> kernels and calls to PETSc KSP and related PC. In particular, I am trying
>> to perform tests with Hypre BoomerAMG both on CPU and GPU. In order to do
>> that, on CPU I am running the code with -mat_type mpiaij and -vec_type mpi,
>> while on GPU I am using respectively aijcusparse and cuda.
>>
>> The configuration ran for PETSc (version is 3.20) is
>> ./configure PETSC_ARCH=arch-linux-cuda --with-cuda --download-mumps
>> --download-hypre --with-debugging=0 --download-scalapack
>> --download-parmetis --download-metis --download-fblaslapack=1
>> --download-mpich --download-make --download-cmake
>>
>> My problem is that when I try to run my code on GPU it works well, while
>> on CPU with mat_type mpiaij and -vec_type mpi it works regularly until the
>> call to Hypre, then freezes (I have to kill it myself), while with GAMG it
>> works on CPU with the same configuration (and the same code, just PC is
>> changed).
>>
> Probably because if hypre is configured with GPU support, even petsc is
> running on CPU (with -mat_type mpiaij and -vec_type mpi), hypre is still
> trying to run on GPU, causing the hanging problem. The petsc/hypre
> interface is not able to support both CPU/GPU with the same build.
>
>
>> On another machine running PETSc version 3.17 everything worked smoothly
>> with the same code and the same configuration, also on Hypre.
>>
> I guess with petsc-3.17, you also used a different hypre version
> (downloaded by petsc).
>
Yes, this is exactly what I concluded after a bit of debugging and
recompiling. In the new version it seems that CPU/GPU with the same build
is not supported anymore. Since the CUDA kernels I have in my code do not
interfere with Hypre, the only workaround I found was removing kernels from
my code and performing an only-CPU run. It slows down a bit overall, but
nothing dramatic. Anyway, in another test with version 3.18 I encountered
the same problem, but the program crashes and throws "Invalid options"
error, while in 3.20 it just hangs.

Thanks,
Edoardo

>
>
>
>> Do you have any insights on what is happening?
>>
>> Best regards,
>> Edoardo
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20240105/0b5bc245/attachment-0001.html>