<div dir="ltr"><div dir="ltr">On Mon, Jan 24, 2022 at 9:24 AM Mark Adams <<a href="mailto:mfadams@lbl.gov">mfadams@lbl.gov</a>> wrote:<br></div><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div><div>What is the fastest way to rebuild hypre? reconfiguring did not work and is slow.</div><div><br></div></div><div>I am printf debugging to find this HSA_STATUS_ERROR_MEMORY_FAULT (no debuggers other than valgrind on Crusher??!?!) and I get to a hypre call:</div><div><br></div><div> PetscStackCallStandard(HYPRE_IJMatrixAddToValues,(hA->ij,1,&hnc,(HYPRE_BigInt*)(rows+i),(HYPRE_BigInt*)cscr[0],sscr));<br></div><div><br></div><div>This is from DMPlexComputeJacobian_Internal and MatSetClosure. HYPRE_IJMatrixAddToValues is successfully called in earlier parts of the run.</div></div></blockquote><div><br></div><div>So MatSetClosure() just calls MatSetValues(). That should find any out of range index. I guess it is possible that the element matrix storage is</div><div>invalid, but that is a hard thing to mess up. Hopefully, debugging shows the SEGV in Hypre.</div><div><br></div><div> Thanks,</div><div><br></div><div> Matt</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>The args look OK, so I am going into HYPRE_IJMatrixAddToValues.</div><div><br></div><div>Thanks,</div><div>Mark</div><div><br></div><div><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Sun, Jan 23, 2022 at 9:55 AM Mark Adams <<a href="mailto:mfadams@lbl.gov" target="_blank">mfadams@lbl.gov</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Stefano and Matt, This segv looks like a Plexism.<div><br></div><div><div>+ srun -n1 -N1 --ntasks-per-gpu=1 --gpu-bind=closest ../ex13 -dm_plex_box_faces 2,2,2 -petscpartitioner_simple_process_grid 1,1,1 -dm_plex_box_upper 1,1,1 -petscpartitioner_simple_node_grid 1,1,1 -dm_refine 2 -dm_view -malloc_debug -log_trace -pc_type hypre -dm_vec_type hip -dm_mat_type hypre<br>+ tee out_001_kokkos_Crusher_2_8_hypre.txt<br> [0] 1.293e-06 Event begin: DMPlexSymmetrize<br>[0] 8.9463e-05 Event end: DMPlexSymmetrize<br></div><div> .....</div><div>[0] 0.554529 Event end: VecHIPCopyFrom<br>[0] 0.559891 Event begin: DMCreateInterp<br> [0] 0.560603 Event begin: DMPlexInterpFE<br> [0] 0.566707 Event begin: MatAssemblyBegin<br> [0] 0.566962 Event begin: BuildTwoSidedF<br> [0] 0.567068 Event begin: BuildTwoSided<br> [0] 0.567119 Event end: BuildTwoSided<br> [0] 0.567154 Event end: BuildTwoSidedF<br> [0] 0.567162 Event end: MatAssemblyBegin<br> [0] 0.567164 Event begin: MatAssemblyEnd<br> [0] 0.567356 Event end: MatAssemblyEnd<br> [0] 0.572884 Event begin: MatAssemblyBegin<br> [0] 0.57289 Event end: MatAssemblyBegin<br> [0] 0.572892 Event begin: MatAssemblyEnd<br> [0] 0.573978 Event end: MatAssemblyEnd<br> [0] 0.574428 Event begin: MatZeroEntries<br> [0] 0.579998 Event end: MatZeroEntries<br>:0:rocdevice.cpp :2589: 257935591316 us: Device::callbackQueue aborting with error : HSA_STATUS_ERROR_MEMORY_FAULT: Agent attempted to access an inaccessible address. code: 0x2b<br>srun: error: crusher001: task 0: Aborted<br>srun: launch/slurm: _step_signal: Terminating StepId=65929.4<br>+ date<br>Sun 23 Jan 2022 09:46:55 AM EST<br></div></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Sun, Jan 23, 2022 at 8:15 AM Mark Adams <<a href="mailto:mfadams@lbl.gov" target="_blank">mfadams@lbl.gov</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Thanks, <div>'-mat_type hypre' was failing for me. I could not find a test that used it and I was not sure it was considered functional.<div>I will look at it again and collect a bug report if needed.</div></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Sat, Jan 22, 2022 at 11:31 AM Stefano Zampini <<a href="mailto:stefano.zampini@gmail.com" target="_blank">stefano.zampini@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Mark<div><br></div><div>the two options are only there to test the code in CI, and are not needed in general</div><div><br></div><div> '--download-hypre-configure-arguments=--enable-unified-memory',<br>This is only here to test the unified memory code path</div><div><br></div><div> '--with-hypre-gpuarch=gfx90a',<br></div><div>This is not needed if rocminfo is in PATH</div><div><br></div><div>Our interface code with HYPRE GPU works fine for HIP, it is tested in CI.</div><div>The -mat_type hypre assembling for ex19 does not work because ex19 uses FDColoring. Just assemble in mpiaij format (look at runex19_hypre_hip in src/snes/tutorials/makefile); the interface code will copy the matrix to the GPU</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">Il giorno ven 21 gen 2022 alle ore 19:24 Mark Adams <<a href="mailto:mfadams@lbl.gov" target="_blank">mfadams@lbl.gov</a>> ha scritto:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div dir="ltr"><br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, Jan 21, 2022 at 11:14 AM Jed Brown <<a href="mailto:jed@jedbrown.org" target="_blank">jed@jedbrown.org</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">"Paul T. Bauman" <<a href="mailto:ptbauman@gmail.com" target="_blank">ptbauman@gmail.com</a>> writes:<br>
<br>
> On Fri, Jan 21, 2022 at 8:52 AM Paul T. Bauman <<a href="mailto:ptbauman@gmail.com" target="_blank">ptbauman@gmail.com</a>> wrote:<br>
>> Yes. The way HYPRE's memory model is setup is that ALL GPU allocations are<br>
>> "native" (i.e. [cuda,hip]Malloc) or, if unified memory is enabled, then ALL<br>
>> GPU allocations are unified memory (i.e. [cuda,hip]MallocManaged).<br>
>> Regarding HIP, there is an HMM implementation of hipMallocManaged planned,<br>
>> but is it not yet delivered AFAIK (and it will *not* support gfx906, e.g.<br>
>> RVII, FYI), so, today, under the covers, hipMallocManaged is calling<br>
>> hipHostMalloc. So, today, all your unified memory allocations in HYPRE on<br>
>> HIP are doing CPU-pinned memory accesses. And performance is just truly<br>
>> terrible (as you might expect).<br>
<br>
Thanks for this important bit of information.<br>
<br>
And it sounds like when we add support to hand off Kokkos matrices and vectors (our current support for matrices on ROCm devices uses Kokkos) or add direct support for hipSparse, we'll avoid touching host memory in assembly-to-solve with hypre.<br></blockquote><div><br></div><div>It does not look like anyone has made Hypre work with HIP. Stafano added a runex19_hypre_hip target 4 months ago and hypre.py has some HIP things.</div><div><br></div><div>I have a user that would like to try this, no hurry but, can I get an idea of a plan for this?</div><div><br></div><div>Thanks,</div><div>Mark <br></div><div> </div></div></div>
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr">Stefano</div>
</blockquote></div>
</blockquote></div>
</blockquote></div>
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr" class="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>-- Norbert Wiener</div><div><br></div><div><a href="http://www.cse.buffalo.edu/~knepley/" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br></div></div></div></div></div></div></div></div>