<div dir="ltr"><div dir="ltr"><br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Sat, Jan 22, 2022 at 11:31 AM Stefano Zampini <<a href="mailto:stefano.zampini@gmail.com">stefano.zampini@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Mark<div><br></div><div>the two options are only there to test the code in CI, and are not needed in general</div><div><br></div><div> '--download-hypre-configure-arguments=--enable-unified-memory',<br>This is only here to test the unified memory code path</div><div><br></div><div> '--with-hypre-gpuarch=gfx90a',<br></div><div>This is not needed if rocminfo is in PATH</div><div><br></div><div>Our interface code with HYPRE GPU works fine for HIP, it is tested in CI.</div></div></blockquote><div><br></div><div>I think there may be a problem with Crusher (he says after trying to debug this all day). I was not able to get to the error. rocgdb hung and I did not manage to get a print statement from hypre. </div><div>It could be that this error happens _at_ the call to HYPRE_IJMatrixAddToValues in PETSc (ie, it never gets to hypre code). Not sure.</div><div>Oddly HYPRE_IJMatrixSetToValues worked in snes/ex56.</div><div>I didn't figure out how to simple do this until now:</div><div><br></div><div>14:21 adams/aijkokkos-gpu-logging *= crusher:/gpfs/alpine/csc314/scratch/adams/petsc$ make -f gmakefile PETSC_ARCH=arch-olcf-crusher-g test search='ksp_ksp_tutorials-ex55_hypre_device'<br>Using MAKEFLAGS: -- search=ksp_ksp_tutorials-ex55_hypre_device PETSC_ARCH=arch-olcf-crusher-g<br> TEST arch-olcf-crusher-g/tests/counts/ksp_ksp_tutorials-ex55_hypre_device.counts<br># retrying ksp_ksp_tutorials-ex55_hypre_device<br>not ok ksp_ksp_tutorials-ex55_hypre_device # Error code: 134<br># :0:rocdevice.cpp :2589: 360810731350 us: Device::callbackQueue aborting with error : HSA_STATUS_ERROR_INVALID_ISA: The instruction set architecture is invalid. code: 0x100f<br># :0:rocdevice.cpp :2589: 360810732560 us: Device::callbackQueue aborting with error : HSA_STATUS_ERROR_INVALID_ISA: The instruction set architecture is invalid. code: 0x100f<br># :0:rocdevice.cpp :2589: 360810735300 us: Device::callbackQueue aborting with error : HSA_STATUS_ERROR_INVALID_ISA: The instruction set architecture is invalid. code: 0x100f<br># :0:rocdevice.cpp :2589: 360810736352 us: Device::callbackQueue aborting with error : HSA_STATUS_ERROR_INVALID_ISA: The instruction set architecture is invalid. code: 0x100f<br># srun: error: crusher002: tasks 0-3: Aborted<br># srun: launch/slurm: _step_signal: Terminating StepId=66195.4<br> ok ksp_ksp_tutorials-ex55_hypre_device # SKIP Command failed so no diff<br><br><br># FAILED ksp_ksp_tutorials-ex55_hypre_device<br>#<br># To rerun failed tests:<br># /usr/bin/gmake -f gmakefile test test-fail=1<br></div><div><br></div><div><br></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>The -mat_type hypre assembling for ex19 does not work because ex19 uses FDColoring. Just assemble in mpiaij format (look at runex19_hypre_hip in src/snes/tutorials/makefile); the interface code will copy the matrix to the GPU</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">Il giorno ven 21 gen 2022 alle ore 19:24 Mark Adams <<a href="mailto:mfadams@lbl.gov" target="_blank">mfadams@lbl.gov</a>> ha scritto:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div dir="ltr"><br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, Jan 21, 2022 at 11:14 AM Jed Brown <<a href="mailto:jed@jedbrown.org" target="_blank">jed@jedbrown.org</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">"Paul T. Bauman" <<a href="mailto:ptbauman@gmail.com" target="_blank">ptbauman@gmail.com</a>> writes:<br>
<br>
> On Fri, Jan 21, 2022 at 8:52 AM Paul T. Bauman <<a href="mailto:ptbauman@gmail.com" target="_blank">ptbauman@gmail.com</a>> wrote:<br>
>> Yes. The way HYPRE's memory model is setup is that ALL GPU allocations are<br>
>> "native" (i.e. [cuda,hip]Malloc) or, if unified memory is enabled, then ALL<br>
>> GPU allocations are unified memory (i.e. [cuda,hip]MallocManaged).<br>
>> Regarding HIP, there is an HMM implementation of hipMallocManaged planned,<br>
>> but is it not yet delivered AFAIK (and it will *not* support gfx906, e.g.<br>
>> RVII, FYI), so, today, under the covers, hipMallocManaged is calling<br>
>> hipHostMalloc. So, today, all your unified memory allocations in HYPRE on<br>
>> HIP are doing CPU-pinned memory accesses. And performance is just truly<br>
>> terrible (as you might expect).<br>
<br>
Thanks for this important bit of information.<br>
<br>
And it sounds like when we add support to hand off Kokkos matrices and vectors (our current support for matrices on ROCm devices uses Kokkos) or add direct support for hipSparse, we'll avoid touching host memory in assembly-to-solve with hypre.<br></blockquote><div><br></div><div>It does not look like anyone has made Hypre work with HIP. Stafano added a runex19_hypre_hip target 4 months ago and hypre.py has some HIP things.</div><div><br></div><div>I have a user that would like to try this, no hurry but, can I get an idea of a plan for this?</div><div><br></div><div>Thanks,</div><div>Mark <br></div><div> </div></div></div>
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr">Stefano</div>
</blockquote></div></div>