[petsc-users] hypre / hip usage

[Mark Adams]

Thanks,
'-mat_type hypre' was failing for me. I could not find a test that used it
and I was not sure it was considered functional.
I will look at it again and collect a bug report if needed.

On Sat, Jan 22, 2022 at 11:31 AM Stefano Zampini <stefano.zampini at gmail.com>
wrote:

> Mark
>
> the two options are only there to test the code in CI, and are not needed
> in general
>
>    '--download-hypre-configure-arguments=--enable-unified-memory',
> This is only here to test the unified memory code path
>
>     '--with-hypre-gpuarch=gfx90a',
> This is not needed if rocminfo is in PATH
>
> Our interface code with HYPRE GPU works fine for HIP, it is tested in CI.
> The -mat_type hypre assembling for ex19 does not work because ex19 uses
> FDColoring. Just assemble in mpiaij format (look at  runex19_hypre_hip in
> src/snes/tutorials/makefile); the interface code will copy the matrix to
> the GPU
>
> Il giorno ven 21 gen 2022 alle ore 19:24 Mark Adams <mfadams at lbl.gov> ha
> scritto:
>
>>
>>
>> On Fri, Jan 21, 2022 at 11:14 AM Jed Brown <jed at jedbrown.org> wrote:
>>
>>> "Paul T. Bauman" <ptbauman at gmail.com> writes:
>>>
>>> > On Fri, Jan 21, 2022 at 8:52 AM Paul T. Bauman <ptbauman at gmail.com>
>>> wrote:
>>> >> Yes. The way HYPRE's memory model is setup is that ALL GPU
>>> allocations are
>>> >> "native" (i.e. [cuda,hip]Malloc) or, if unified memory is enabled,
>>> then ALL
>>> >> GPU allocations are unified memory (i.e. [cuda,hip]MallocManaged).
>>> >> Regarding HIP, there is an HMM implementation of hipMallocManaged
>>> planned,
>>> >> but is it not yet delivered AFAIK (and it will *not* support gfx906,
>>> e.g.
>>> >> RVII, FYI), so, today, under the covers, hipMallocManaged is calling
>>> >> hipHostMalloc. So, today, all your unified memory allocations in
>>> HYPRE on
>>> >> HIP are doing CPU-pinned memory accesses. And performance is just
>>> truly
>>> >> terrible (as you might expect).
>>>
>>> Thanks for this important bit of information.
>>>
>>> And it sounds like when we add support to hand off Kokkos matrices and
>>> vectors (our current support for matrices on ROCm devices uses Kokkos) or
>>> add direct support for hipSparse, we'll avoid touching host memory in
>>> assembly-to-solve with hypre.
>>>
>>
>> It does not look like anyone has made Hypre work with HIP. Stafano added
>> a runex19_hypre_hip target 4 months ago and hypre.py has some HIP things.
>>
>> I have a user that would like to try this, no hurry but, can I get an
>> idea of a plan for this?
>>
>> Thanks,
>> Mark
>>
>>
>
>
> --
> Stefano
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20220123/85cf260b/attachment.html>