[petsc-users] hypre / hip usage

[Stefano Zampini]

Mark

the two options are only there to test the code in CI, and are not needed
in general

   '--download-hypre-configure-arguments=--enable-unified-memory',
This is only here to test the unified memory code path

    '--with-hypre-gpuarch=gfx90a',
This is not needed if rocminfo is in PATH

Our interface code with HYPRE GPU works fine for HIP, it is tested in CI.
The -mat_type hypre assembling for ex19 does not work because ex19 uses
FDColoring. Just assemble in mpiaij format (look at  runex19_hypre_hip in
src/snes/tutorials/makefile); the interface code will copy the matrix to
the GPU

Il giorno ven 21 gen 2022 alle ore 19:24 Mark Adams <mfadams at lbl.gov> ha
scritto:

>
>
> On Fri, Jan 21, 2022 at 11:14 AM Jed Brown <jed at jedbrown.org> wrote:
>
>> "Paul T. Bauman" <ptbauman at gmail.com> writes:
>>
>> > On Fri, Jan 21, 2022 at 8:52 AM Paul T. Bauman <ptbauman at gmail.com>
>> wrote:
>> >> Yes. The way HYPRE's memory model is setup is that ALL GPU allocations
>> are
>> >> "native" (i.e. [cuda,hip]Malloc) or, if unified memory is enabled,
>> then ALL
>> >> GPU allocations are unified memory (i.e. [cuda,hip]MallocManaged).
>> >> Regarding HIP, there is an HMM implementation of hipMallocManaged
>> planned,
>> >> but is it not yet delivered AFAIK (and it will *not* support gfx906,
>> e.g.
>> >> RVII, FYI), so, today, under the covers, hipMallocManaged is calling
>> >> hipHostMalloc. So, today, all your unified memory allocations in HYPRE
>> on
>> >> HIP are doing CPU-pinned memory accesses. And performance is just truly
>> >> terrible (as you might expect).
>>
>> Thanks for this important bit of information.
>>
>> And it sounds like when we add support to hand off Kokkos matrices and
>> vectors (our current support for matrices on ROCm devices uses Kokkos) or
>> add direct support for hipSparse, we'll avoid touching host memory in
>> assembly-to-solve with hypre.
>>
>
> It does not look like anyone has made Hypre work with HIP. Stafano added a
> runex19_hypre_hip target 4 months ago and hypre.py has some HIP things.
>
> I have a user that would like to try this, no hurry but, can I get an idea
> of a plan for this?
>
> Thanks,
> Mark
>
>


-- 
Stefano
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20220122/2c305c85/attachment-0001.html>