<div dir="ltr">Ok, I was wrong about hypre registering its flops. Our KSP is 100 % GPU, but there are very few flops recorded (ie, no hypre flops).<div>No flops in PCApply. That implies that hypre is in fact being used.</div><div>I don't know of a way to verify from log_view if hypre is running on the GPU. You could try -ksp_view to see if there is anything, but you can just look at the KSPSOlve time on decent sized problems with and without GPUs in hypre.</div><div>Now, I am not sure what you should do to get a CPU hypre. You can configure a PETSc without cuda. That is the surest way to go, but simply removing -mat_type hypre (or change to -mat_type aij) might work and give you CPU hypre. I am not sure.</div><div>Anyway you should see a difference in solve times with the CPU solves if in fact you are getting the GPU solver.</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, Nov 22, 2021 at 12:41 PM Karthikeyan Chockalingam - STFC UKRI <<a href="mailto:karthikeyan.chockalingam@stfc.ac.uk">karthikeyan.chockalingam@stfc.ac.uk</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div lang="EN-GB" style="overflow-wrap: break-word;">
<div class="gmail-m_-1600957560393275553WordSection1">
<p class="MsoNormal"><span><a id="gmail-m_-1600957560393275553OWAAM27767981EC43904FA162658CBD412BEF" href="mailto:knepley@gmail.com" target="_blank"><span style="font-family:Calibri,sans-serif;text-decoration:none">@Matthew Knepley</span></a> Yes, it works with main
build!<u></u><u></u></span></p>
<p class="MsoNormal"><span><u></u> <u></u></span></p>
<p class="MsoNormal"><span><a id="gmail-m_-1600957560393275553OWAAM20BFFEB3E5274A41A16043677C50F8FF" href="mailto:mfadams@lbl.gov" target="_blank"><span style="font-family:Calibri,sans-serif;text-decoration:none">@Mark Adams</span></a> I have attached the log_view
output of ex4 for your reference. But pcapply or pcsetup did not record any gpu flops.
<u></u><u></u></span></p>
<p class="MsoNormal"><span>If it is not too much trouble can you send me the log_view output of ksp/ksp/tutorial/ex45 using hypre?<u></u><u></u></span></p>
<p class="MsoNormal"><span><u></u> <u></u></span></p>
<p class="MsoNormal"><span>Many thanks!<u></u><u></u></span></p>
<p class="MsoNormal"><span>Karthik.<u></u><u></u></span></p>
<p class="MsoNormal"><span><u></u> <u></u></span></p>
<div style="border-right:none;border-bottom:none;border-left:none;border-top:1pt solid rgb(181,196,223);padding:3pt 0cm 0cm">
<p class="MsoNormal"><b><span style="font-size:12pt;color:black">From: </span></b><span style="font-size:12pt;color:black">Matthew Knepley <<a href="mailto:knepley@gmail.com" target="_blank">knepley@gmail.com</a>><br>
<b>Date: </b>Monday, 22 November 2021 at 15:26<br>
<b>To: </b>"Chockalingam, Karthikeyan (STFC,DL,HC)" <<a href="mailto:karthikeyan.chockalingam@stfc.ac.uk" target="_blank">karthikeyan.chockalingam@stfc.ac.uk</a>><br>
<b>Cc: </b>Stefano Zampini <<a href="mailto:stefano.zampini@gmail.com" target="_blank">stefano.zampini@gmail.com</a>>, "<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a>" <<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a>><br>
<b>Subject: </b>Re: [petsc-users] hypre on gpus<u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<div>
<p class="MsoNormal">On Mon, Nov 22, 2021 at 10:20 AM Karthikeyan Chockalingam - STFC UKRI <<a href="mailto:karthikeyan.chockalingam@stfc.ac.uk" target="_blank">karthikeyan.chockalingam@stfc.ac.uk</a>> wrote:<u></u><u></u></p>
</div>
<div>
<blockquote style="border-top:none;border-right:none;border-bottom:none;border-left:1pt solid rgb(204,204,204);padding:0cm 0cm 0cm 6pt;margin-left:4.8pt;margin-right:0cm">
<div>
<div>
<p class="MsoNormal">Thanks Stefano,<u></u><u></u></p>
</div>
</div>
</blockquote>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">I just did a clean build with no problems. Can you try a clean build of main?<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal"> Thanks,<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal"> Matt<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<blockquote style="border-top:none;border-right:none;border-bottom:none;border-left:1pt solid rgb(204,204,204);padding:0cm 0cm 0cm 6pt;margin-left:4.8pt;margin-right:0cm">
<div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
<p class="MsoNormal">I have another build without the
<span class="gmail-m_-1600957560393275553gmail-m-5318220838648288847gmail-m136675646475516123s1"><span style="font-size:10pt">--download-hypre-commit=origin/hypre_petsc
</span>but that gives a different error.</span><u></u><u></u></p>
<p class="MsoNormal"> <u></u><u></u></p>
<p class="gmail-m_-1600957560393275553gmail-m-5318220838648288847p1"><span class="gmail-m_-1600957560393275553gmail-m-5318220838648288847s1"><span style="font-size:8pt">[kchockalingam@glados tutorials]$ ./ex4 -ksp_view -ksp_type cg -mat_type hypre -pc_type hypre</span></span><u></u><u></u></p>
<p class="gmail-m_-1600957560393275553gmail-m-5318220838648288847p2"><span class="gmail-m_-1600957560393275553gmail-m-5318220838648288847s1"><b><span style="font-size:8pt">[0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------</span></b></span><u></u><u></u></p>
<p class="gmail-m_-1600957560393275553gmail-m-5318220838648288847p1"><span class="gmail-m_-1600957560393275553gmail-m-5318220838648288847s1"><span style="font-size:8pt">[0]PETSC ERROR: Error in external library</span></span><u></u><u></u></p>
<p class="gmail-m_-1600957560393275553gmail-m-5318220838648288847p1"><span class="gmail-m_-1600957560393275553gmail-m-5318220838648288847s1"><span style="font-size:8pt">[0]PETSC ERROR: Error in HYPRE_IJMatrixAssemble(): error code 12</span></span><u></u><u></u></p>
<p class="gmail-m_-1600957560393275553gmail-m-5318220838648288847p1"><span class="gmail-m_-1600957560393275553gmail-m-5318220838648288847s1"><span style="font-size:8pt">[0]PETSC ERROR: See
<a href="https://www.mcs.anl.gov/petsc/documentation/faq.html" target="_blank">https://www.mcs.anl.gov/petsc/documentation/faq.html</a> for trouble shooting.</span></span><u></u><u></u></p>
<p class="gmail-m_-1600957560393275553gmail-m-5318220838648288847p1"><span class="gmail-m_-1600957560393275553gmail-m-5318220838648288847s1"><span style="font-size:8pt">[0]PETSC ERROR: Petsc Release Version 3.15.3, Aug 06, 2021</span></span><span class="gmail-m_-1600957560393275553gmail-m-5318220838648288847apple-converted-space"><span style="font-size:8pt"> </span></span><u></u><u></u></p>
<p class="gmail-m_-1600957560393275553gmail-m-5318220838648288847p1"><span class="gmail-m_-1600957560393275553gmail-m-5318220838648288847s1"><span style="font-size:8pt">[0]PETSC ERROR: ./ex4 on a</span></span><span class="gmail-m_-1600957560393275553gmail-m-5318220838648288847apple-converted-space"><span style="font-size:8pt">
</span></span><span class="gmail-m_-1600957560393275553gmail-m-5318220838648288847s1"><span style="font-size:8pt">named
<a href="http://glados.dl.ac.uk" target="_blank">glados.dl.ac.uk</a> by kchockalingam Mon Nov 22 15:07:46 2021</span></span><u></u><u></u></p>
<p class="gmail-m_-1600957560393275553gmail-m-5318220838648288847p1"><span class="gmail-m_-1600957560393275553gmail-m-5318220838648288847s1"><span style="font-size:8pt">[0]PETSC ERROR: Configure options --package-prefix-hash=/home/kchockalingam/petsc-hash-pkgs --with-make-test-np=2 COPTFLAGS="-g -O3 -fno-omit-frame-pointer"
FOPTFLAGS="-g -O3 -fno-omit-frame-pointer" CXXOPTFLAGS="-g -O3 -fno-omit-frame-pointer" --with-cuda=1 --with-cuda-arch=70 --with-blaslapack=1 --with-cuda-dir=/apps/packages/cuda/10.1/ --with-mpi-dir=/apps/packages/gcc/7.3.0/openmpi/3.1.2 --download-hypre=1
--download-hypre-configure-arguments=HYPRE_CUDA_SM=70 --with-debugging=no PETSC_ARCH=arch-ci-linux-cuda11-double</span></span><u></u><u></u></p>
<p class="gmail-m_-1600957560393275553gmail-m-5318220838648288847p1"><span class="gmail-m_-1600957560393275553gmail-m-5318220838648288847s1"><span style="font-size:8pt">[0]PETSC ERROR: #1 MatAssemblyEnd_HYPRE() at /home/kchockalingam/tools/petsc-3.15.3/src/mat/impls/hypre/mhypre.c:1212</span></span><u></u><u></u></p>
<p class="gmail-m_-1600957560393275553gmail-m-5318220838648288847p1"><span class="gmail-m_-1600957560393275553gmail-m-5318220838648288847s1"><span style="font-size:8pt">[0]PETSC ERROR: #2 MatAssemblyEnd() at /home/kchockalingam/tools/petsc-3.15.3/src/mat/interface/matrix.c:5652</span></span><u></u><u></u></p>
<p class="gmail-m_-1600957560393275553gmail-m-5318220838648288847p1"><span class="gmail-m_-1600957560393275553gmail-m-5318220838648288847s1"><span style="font-size:8pt">[0]PETSC ERROR: #3 main() at ex4.c:84</span></span><u></u><u></u></p>
<p class="gmail-m_-1600957560393275553gmail-m-5318220838648288847p1"><span class="gmail-m_-1600957560393275553gmail-m-5318220838648288847s1"><span style="font-size:8pt">[0]PETSC ERROR: PETSc Option Table entries:</span></span><u></u><u></u></p>
<p class="gmail-m_-1600957560393275553gmail-m-5318220838648288847p1"><span class="gmail-m_-1600957560393275553gmail-m-5318220838648288847s1"><span style="font-size:8pt">[0]PETSC ERROR: -ksp_type cg</span></span><u></u><u></u></p>
<p class="gmail-m_-1600957560393275553gmail-m-5318220838648288847p1"><span class="gmail-m_-1600957560393275553gmail-m-5318220838648288847s1"><span style="font-size:8pt">[0]PETSC ERROR: -ksp_view</span></span><u></u><u></u></p>
<p class="gmail-m_-1600957560393275553gmail-m-5318220838648288847p1"><span class="gmail-m_-1600957560393275553gmail-m-5318220838648288847s1"><span style="font-size:8pt">[0]PETSC ERROR: -mat_type hypre</span></span><u></u><u></u></p>
<p class="gmail-m_-1600957560393275553gmail-m-5318220838648288847p1"><span class="gmail-m_-1600957560393275553gmail-m-5318220838648288847s1"><span style="font-size:8pt">[0]PETSC ERROR: -pc_type hypre</span></span><u></u><u></u></p>
<p class="gmail-m_-1600957560393275553gmail-m-5318220838648288847p2"><span class="gmail-m_-1600957560393275553gmail-m-5318220838648288847s1"><b><span style="font-size:8pt">[0]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint@mcs.anl.gov----------</span></b></span><u></u><u></u></p>
<p class="gmail-m_-1600957560393275553gmail-m-5318220838648288847p1"><span class="gmail-m_-1600957560393275553gmail-m-5318220838648288847s1"><span style="font-size:8pt">--------------------------------------------------------------------------</span></span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:8pt"> </span><u></u><u></u></p>
<p class="MsoNormal">Best,<u></u><u></u></p>
<p class="MsoNormal">Karthik.<u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:8pt"> </span><u></u><u></u></p>
<div style="border-right:none;border-bottom:none;border-left:none;border-top:1pt solid rgb(181,196,223);padding:3pt 0cm 0cm">
<p class="MsoNormal"><b><span style="font-size:12pt;color:black">From:
</span></b><span style="font-size:12pt;color:black">Stefano Zampini <<a href="mailto:stefano.zampini@gmail.com" target="_blank">stefano.zampini@gmail.com</a>><br>
<b>Date: </b>Monday, 22 November 2021 at 14:46<br>
<b>To: </b>Matthew Knepley <<a href="mailto:knepley@gmail.com" target="_blank">knepley@gmail.com</a>><br>
<b>Cc: </b>"Chockalingam, Karthikeyan (STFC,DL,HC)" <<a href="mailto:karthikeyan.chockalingam@stfc.ac.uk" target="_blank">karthikeyan.chockalingam@stfc.ac.uk</a>>, "<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a>" <<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a>><br>
<b>Subject: </b>Re: [petsc-users] hypre on gpus</span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">You don't need to specify the HYPRE commit. Remove <span class="gmail-m_-1600957560393275553gmail-m-5318220838648288847gmail-m136675646475516123s1"><span style="font-size:8pt">
</span></span><span class="gmail-m_-1600957560393275553gmail-m-5318220838648288847gmail-m136675646475516123s1"><span style="font-size:10pt">--download-hypre-commit=origin/hypre_petsc from the configuration options</span></span><u></u><u></u></p>
</div>
<p class="MsoNormal"> <u></u><u></u></p>
<div>
<div>
<p class="MsoNormal">Il giorno lun 22 nov 2021 alle ore 17:29 Matthew Knepley <<a href="mailto:knepley@gmail.com" target="_blank">knepley@gmail.com</a>> ha scritto:<u></u><u></u></p>
</div>
<blockquote style="border-top:none;border-right:none;border-bottom:none;border-left:1pt solid rgb(204,204,204);padding:0cm 0cm 0cm 6pt;margin:5pt 0cm 5pt 4.8pt">
<div>
<div>
<p class="MsoNormal">On Mon, Nov 22, 2021 at 8:50 AM Karthikeyan Chockalingam - STFC UKRI <<a href="mailto:karthikeyan.chockalingam@stfc.ac.uk" target="_blank">karthikeyan.chockalingam@stfc.ac.uk</a>>
wrote:<u></u><u></u></p>
</div>
<div>
<blockquote style="border-top:none;border-right:none;border-bottom:none;border-left:1pt solid rgb(204,204,204);padding:0cm 0cm 0cm 6pt;margin:5pt 0cm 5pt 4.8pt">
<div>
<div>
<p class="MsoNormal">Hi Matt,<u></u><u></u></p>
<p class="MsoNormal"> <u></u><u></u></p>
<p class="MsoNormal">Below is the entire error message:<u></u><u></u></p>
</div>
</div>
</blockquote>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">I cannot reproduce this:<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-bottom:12pt">main $:/PETSc3/petsc/petsc-dev/src/ksp/ksp/tutorials$ ./ex4 -ksp_view -ksp_type cg -mat_type hypre -pc_type hypre<br>
KSP Object: 1 MPI processes<br>
type: cg<br>
maximum iterations=10000, initial guess is zero<br>
tolerances: relative=0.000138889, absolute=1e-50, divergence=10000.<br>
left preconditioning<br>
using PRECONDITIONED norm type for convergence test<br>
PC Object: 1 MPI processes<br>
type: hypre<br>
HYPRE BoomerAMG preconditioning<br>
Cycle type V<br>
Maximum number of levels 25<br>
Maximum number of iterations PER hypre call 1<br>
Convergence tolerance PER hypre call 0.<br>
Threshold for strong coupling 0.25<br>
Interpolation truncation factor 0.<br>
Interpolation: max elements per row 0<br>
Number of levels of aggressive coarsening 0<br>
Number of paths for aggressive coarsening 1<br>
Maximum row sums 0.9<br>
Sweeps down 1<br>
Sweeps up 1<br>
Sweeps on coarse 1<br>
Relax down symmetric-SOR/Jacobi<br>
Relax up symmetric-SOR/Jacobi<br>
Relax on coarse Gaussian-elimination<br>
Relax weight (all) 1.<br>
Outer relax weight (all) 1.<br>
Using CF-relaxation<br>
Not using more complex smoothers.<br>
Measure type local<br>
Coarsen type Falgout<br>
Interpolation type classical<br>
linear system matrix = precond matrix:<br>
Mat Object: 1 MPI processes<br>
type: hypre<br>
rows=56, cols=56<br>
Norm of error 8.69801e-05 iterations 2<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">This is on the 'main' branch. So either there is some bug in release, or something is strange on your end. Since we run Hypre tests for the CI,<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">I am leaning toward the latter. Can you try the 'main' branch? We will have to use this anyway if we want any fixes.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> Thanks,<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> Matt<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<blockquote style="border-top:none;border-right:none;border-bottom:none;border-left:1pt solid rgb(204,204,204);padding:0cm 0cm 0cm 6pt;margin:5pt 0cm 5pt 4.8pt">
<div>
<div>
<p class="MsoNormal"><span style="font-size:8pt"> <b>[0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------</b></span><u></u><u></u></p>
<p><span style="font-size:8pt">[0]PETSC ERROR: Object is in wrong state</span><u></u><u></u></p>
<p><span style="font-size:8pt">[0]PETSC ERROR: Must call MatXXXSetPreallocation(), MatSetUp() or the matrix has not yet been factored on argument 1 "mat" before MatGetOwnershipRange()</span><u></u><u></u></p>
<p><span style="font-size:8pt">[0]PETSC ERROR: See <a href="https://petsc.org/release/faq/" target="_blank">
https://petsc.org/release/faq/</a> for trouble shooting.</span><u></u><u></u></p>
<p><span style="font-size:8pt">[0]PETSC ERROR: Petsc Release Version 3.16.0, Sep 29, 2021 </span><u></u><u></u></p>
<p><span style="font-size:8pt">[0]PETSC ERROR: ./ex4 on a named sqg2b4.bullx by kxc07-lxm25 Mon Nov 22 11:33:41 2021</span><u></u><u></u></p>
<p><span style="font-size:8pt">[0]PETSC ERROR: Configure options --prefix=/lustre/scafellpike/local/apps/gcc7/petsc/3.16.0-cuda11.2 --with-debugging=yes --with-blaslapack-dir=/lustre/scafellpike/local/apps/intel/intel_cs/2018.0.128/mkl --with-cuda=1 --with-cuda-arch=70
--download-hypre=yes --download-hypre-configure-arguments=HYPRE_CUDA_SM=70 --download-hypre-commit=origin/hypre_petsc --with-shared-libraries=1 --known-mpi-shared-libraries=1 --with-cc=mpicc --with-cxx=mpicxx -with-fc=mpif90</span><u></u><u></u></p>
<p><span style="font-size:8pt">[0]PETSC ERROR: #1 MatGetOwnershipRange() at /netfs/smain01/scafellpike/local/package_build/build/rja87-build/petsc-cuda-3.16.0/src/mat/interface/matrix.c:6784</span><u></u><u></u></p>
<p><span style="font-size:8pt">[0]PETSC ERROR: #2 main() at ex4.c:40</span><u></u><u></u></p>
<p><span style="font-size:8pt">[0]PETSC ERROR: PETSc Option Table entries:</span><u></u><u></u></p>
<p><span style="font-size:8pt">[0]PETSC ERROR: -ksp_type cg</span><u></u><u></u></p>
<p><span style="font-size:8pt">[0]PETSC ERROR: -ksp_view</span><u></u><u></u></p>
<p><span style="font-size:8pt">[0]PETSC ERROR: -mat_type hypre</span><u></u><u></u></p>
<p><span style="font-size:8pt">[0]PETSC ERROR: -pc_type hypre</span><u></u><u></u></p>
<p><span style="font-size:8pt">[0]PETSC ERROR: -use_gpu_aware_mpi 0</span><u></u><u></u></p>
<p><b><span style="font-size:8pt">[0]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint@mcs.anl.gov----------</span></b><u></u><u></u></p>
<p><span style="font-size:8pt">--------------------------------------------------------------------------</span><u></u><u></u></p>
<p class="MsoNormal"> <u></u><u></u></p>
<p class="MsoNormal">I have also attached the make.log. Thank you for having a look.<u></u><u></u></p>
<p class="MsoNormal"> <u></u><u></u></p>
<p class="MsoNormal">Best,<u></u><u></u></p>
<p class="MsoNormal">Karthik.<u></u><u></u></p>
<p class="MsoNormal"> <u></u><u></u></p>
<div>
<p class="MsoNormal"><b><span style="font-size:12pt;color:black">From:
</span></b><span style="font-size:12pt;color:black">Matthew Knepley <<a href="mailto:knepley@gmail.com" target="_blank">knepley@gmail.com</a>><br>
<b>Date: </b>Monday, 22 November 2021 at 13:41<br>
<b>To: </b>"Chockalingam, Karthikeyan (STFC,DL,HC)" <<a href="mailto:karthikeyan.chockalingam@stfc.ac.uk" target="_blank">karthikeyan.chockalingam@stfc.ac.uk</a>><br>
<b>Cc: </b>Mark Adams <<a href="mailto:mfadams@lbl.gov" target="_blank">mfadams@lbl.gov</a>>, "<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a>" <<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a>><br>
<b>Subject: </b>Re: [petsc-users] hypre on gpus</span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<div>
<p class="MsoNormal">On Mon, Nov 22, 2021 at 6:47 AM Karthikeyan Chockalingam - STFC UKRI <<a href="mailto:karthikeyan.chockalingam@stfc.ac.uk" target="_blank">karthikeyan.chockalingam@stfc.ac.uk</a>>
wrote:<u></u><u></u></p>
</div>
<div>
<blockquote style="border-left:1pt solid rgb(204,204,204);padding:0cm 0cm 0cm 6pt;margin:5pt 0cm 5pt 4.8pt;border-top:currentcolor;border-right:currentcolor;border-bottom:currentcolor">
<div>
<div>
<p><span style="font-size:10pt">Thank you for your response. I tried to run the same example</span><u></u><u></u></p>
<p><span style="font-size:10pt"> </span><u></u><u></u></p>
<p><span style="font-size:10pt">petsc/src/ksp/ksp/tutorials$ <b>./ex4 -ksp_type cg -mat_type hypre -ksp_view -pc_type hypre</b></span><u></u><u></u></p>
<p><b><span style="font-size:10pt"> </span></b><u></u><u></u></p>
<p><span style="font-size:10pt">but it crashed with the below error</span><u></u><u></u></p>
<p><span style="font-size:12pt"> </span><u></u><u></u></p>
<p class="MsoNormal"><b><span style="font-size:10pt;font-family:Menlo;color:rgb(180,36,25)">[0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------</span></b><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:10pt;font-family:Menlo;color:black">[0]PETSC ERROR: Object is in wrong state</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:10pt;font-family:Menlo;color:black">[0]PETSC ERROR: Must call MatXXXSetPreallocation(), MatSetUp() or the matrix has not yet been factored on argument 1
"mat" before MatGetOwnershipRange()</span><u></u><u></u></p>
</div>
</div>
</blockquote>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">Hi Karthik,<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">Please do not clip the error message. We need the entire output. This seems strange that you would get a logic error,<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">since that should be the same on any architecture. So could you also send the make.log?<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> Thanks,<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> Matt<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<blockquote style="border-left:1pt solid rgb(204,204,204);padding:0cm 0cm 0cm 6pt;margin:5pt 0cm 5pt 4.8pt;border-top:currentcolor;border-right:currentcolor;border-bottom:currentcolor">
<div>
<div>
<p><span style="font-size:10pt">Below are the options used to configure hypre with cuda support. Do you spot any mistakes?</span><u></u><u></u></p>
<p><span style="font-size:12pt"> </span><u></u><u></u></p>
<p><span style="font-size:10pt">--with-blaslapack-dir=/lustre/scafellpike/local/apps/intel/intel_cs/2018.0.128/mkl
</span><u></u><u></u></p>
<p><span style="font-size:10pt">--with-cuda=1 </span><u></u><u></u></p>
<p><span style="font-size:10pt">--with-cuda-arch=70 </span><u></u><u></u></p>
<p><span style="font-size:10pt">--download-hypre=yes </span><u></u><u></u></p>
<p><span style="font-size:10pt">--download-hypre-configure-arguments=HYPRE_CUDA_SM=70
</span><u></u><u></u></p>
<p><span style="font-size:10pt">--download-hypre-commit=origin/hypre_petsc </span>
<u></u><u></u></p>
<p><span style="font-size:10pt">--with-shared-libraries=1 </span><u></u><u></u></p>
<p><span style="font-size:10pt">--known-mpi-shared-libraries=1 </span><u></u><u></u></p>
<p><span style="font-size:10pt">--with-cc=mpicc </span><u></u><u></u></p>
<p><span style="font-size:10pt">--with-cxx=mpicxx </span><u></u><u></u></p>
<p><span style="font-size:10pt">-with-fc=mpif90</span><u></u><u></u></p>
<p><span style="font-size:10pt"> </span><u></u><u></u></p>
<p><span style="font-size:10pt">Best,</span><u></u><u></u></p>
<p><span style="font-size:10pt">Karthik.</span><u></u><u></u></p>
<p class="MsoNormal"> <u></u><u></u></p>
<p class="MsoNormal"> <u></u><u></u></p>
<div>
<p class="MsoNormal"><b><span style="font-size:12pt;color:black">From:
</span></b><span style="font-size:12pt;color:black">Mark Adams <<a href="mailto:mfadams@lbl.gov" target="_blank">mfadams@lbl.gov</a>><br>
<b>Date: </b>Friday, 19 November 2021 at 16:31<br>
<b>To: </b>"Chockalingam, Karthikeyan (STFC,DL,HC)" <<a href="mailto:karthikeyan.chockalingam@stfc.ac.uk" target="_blank">karthikeyan.chockalingam@stfc.ac.uk</a>><br>
<b>Cc: </b>"<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a>" <<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a>><br>
<b>Subject: </b>Re: [petsc-users] hypre on gpus</span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<div>
<p class="MsoNormal">You should run with -options_left to check that your options are being used. It may be -mat_type hypre.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<p class="MsoNormal">I have tested this:<u></u><u></u></p>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">petsc/src/ksp/ksp/tutorials$ <b>srun -n2 ./ex4 -ksp_type cg -mat_type hypre -ksp_view -pc_type hypre</b><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">You can add -log_view and that will print performance data for each method like KSPSolve.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">If you are configured with a GPU there will be some extra columns that give the percent of flops on the GPU.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">In the past hypre has not registered flops with us, but I do get flops from hypre now, and -ksp_view showed that it did indeed use hypre.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">I saw that the flops were 100% GPU in hypre.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
</div>
<p class="MsoNormal"> <u></u><u></u></p>
<div>
<div>
<p class="MsoNormal">On Fri, Nov 19, 2021 at 10:47 AM Karthikeyan Chockalingam - STFC UKRI <<a href="mailto:karthikeyan.chockalingam@stfc.ac.uk" target="_blank">karthikeyan.chockalingam@stfc.ac.uk</a>>
wrote:<u></u><u></u></p>
</div>
<blockquote style="border-left:1pt solid rgb(204,204,204);padding:0cm 0cm 0cm 6pt;margin:5pt 0cm 5pt 4.8pt;border-top:currentcolor;border-right:currentcolor;border-bottom:currentcolor">
<div>
<div>
<p class="MsoNormal">Hello,<u></u><u></u></p>
<p class="MsoNormal"> <u></u><u></u></p>
<p class="MsoNormal">I tried to solve a 3D Poisson (ksp/tutorial/ex45) using -pc_type hypre on gpus<u></u><u></u></p>
<p class="MsoNormal"> <u></u><u></u></p>
<p class="MsoNormal">./ex45 -da_grid_x 128 -da_grid_y 128 -da_grid_z 128 -dm_mat_type hypre -dm_vec_type cuda -ksp_type cg -pc_type hypre -pc_hypre_type boomeramg -ksp_monitor -log_view<u></u><u></u></p>
<p class="MsoNormal"> <u></u><u></u></p>
<p class="MsoNormal">I profiled the run using NSYS - attached you find all the relevant files.
<u></u><u></u></p>
<p class="MsoNormal">Looking at the profile I doubt if hypre is running on gpus. The cuda kernels are barely active.<u></u><u></u></p>
<p class="MsoNormal">I don’t see any cuda kernel relevant to solve.<u></u><u></u></p>
<p class="MsoNormal">Is my assessment correct? How can I verify if hypre indeed is running on gpus?<u></u><u></u></p>
<p class="MsoNormal"> <u></u><u></u></p>
<p class="MsoNormal">Best,<u></u><u></u></p>
<p class="MsoNormal">Karthik.<u></u><u></u></p>
<p class="MsoNormal"> <u></u><u></u></p>
<div>
<p class="MsoNormal"><b><span style="font-size:12pt;color:black">From:
</span></b><span style="font-size:12pt;color:black">Mark Adams <<a href="mailto:mfadams@lbl.gov" target="_blank">mfadams@lbl.gov</a>><br>
<b>Date: </b>Friday, 8 October 2021 at 18:47<br>
<b>To: </b>"Chockalingam, Karthikeyan (STFC,DL,HC)" <<a href="mailto:karthikeyan.chockalingam@stfc.ac.uk" target="_blank">karthikeyan.chockalingam@stfc.ac.uk</a>><br>
<b>Cc: </b>"<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a>" <<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a>><br>
<b>Subject: </b>Re: [petsc-users] hypre on gpus</span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">I think you would want to use 'cuda' vec_type, but I .<u></u><u></u></p>
<div>
<p class="MsoNormal">You might ask Hypre how one verifies that the GPU is used.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">Mark<u></u><u></u></p>
</div>
</div>
<p class="MsoNormal"> <u></u><u></u></p>
<div>
<div>
<p class="MsoNormal">On Fri, Oct 8, 2021 at 1:35 PM Karthikeyan Chockalingam - STFC UKRI <<a href="mailto:karthikeyan.chockalingam@stfc.ac.uk" target="_blank">karthikeyan.chockalingam@stfc.ac.uk</a>>
wrote:<u></u><u></u></p>
</div>
<blockquote style="border-left:1pt solid rgb(204,204,204);padding:0cm 0cm 0cm 6pt;margin:5pt 0cm 5pt 4.8pt;border-top:currentcolor;border-right:currentcolor;border-bottom:currentcolor">
<div>
<div>
<p class="MsoNormal">Yes, I used it for both cpu and gpu. Is that not okay?<u></u><u></u></p>
<p class="MsoNormal"><span style="font-family:Arial,sans-serif"> </span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-family:Arial,sans-serif">For gpu:
<span style="color:black">-dm_mat_type hypre -dm_vec_type mpicuda </span></span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-family:Arial,sans-serif;color:black"> </span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-family:Arial,sans-serif;color:black">For cpu: -dm_mat_type hypre -dm_vec_type mpi</span><u></u><u></u></p>
<p class="MsoNormal"> <u></u><u></u></p>
<div>
<p class="MsoNormal"><b><span style="font-size:12pt;color:black">From:
</span></b><span style="font-size:12pt;color:black">Mark Adams <</span><a href="mailto:mfadams@lbl.gov" target="_blank"><span style="font-size:12pt">mfadams@lbl.gov</span></a><span style="font-size:12pt;color:black">><br>
<b>Date: </b>Friday, 8 October 2021 at 18:28<br>
<b>To: </b>"Chockalingam, Karthikeyan (STFC,DL,HC)" <</span><a href="mailto:karthikeyan.chockalingam@stfc.ac.uk" target="_blank"><span style="font-size:12pt">karthikeyan.chockalingam@stfc.ac.uk</span></a><span style="font-size:12pt;color:black">><br>
<b>Cc: </b>"</span><a href="mailto:petsc-users@mcs.anl.gov" target="_blank"><span style="font-size:12pt">petsc-users@mcs.anl.gov</span></a><span style="font-size:12pt;color:black">" <</span><a href="mailto:petsc-users@mcs.anl.gov" target="_blank"><span style="font-size:12pt">petsc-users@mcs.anl.gov</span></a><span style="font-size:12pt;color:black">><br>
<b>Subject: </b>Re: [petsc-users] hypre on gpus</span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">Did you use -dm_mat_type hypre on the GPU case ?<u></u><u></u></p>
</div>
<p class="MsoNormal"> <u></u><u></u></p>
<div>
<div>
<p class="MsoNormal">On Fri, Oct 8, 2021 at 12:19 PM Karthikeyan Chockalingam - STFC UKRI <<a href="mailto:karthikeyan.chockalingam@stfc.ac.uk" target="_blank">karthikeyan.chockalingam@stfc.ac.uk</a>>
wrote:<u></u><u></u></p>
</div>
<blockquote style="border-left:1pt solid rgb(204,204,204);padding:0cm 0cm 0cm 6pt;margin:5pt 0cm 5pt 4.8pt;border-top:currentcolor;border-right:currentcolor;border-bottom:currentcolor">
<div>
<div>
<p class="MsoNormal">I tried a different exercise ran the same problem on two cpu cores and on two gpu:<u></u><u></u></p>
<p class="MsoNormal"> <u></u><u></u></p>
<p class="MsoNormal">On gpu
<u></u><u></u></p>
<p class="MsoNormal"> <u></u><u></u></p>
<p class="MsoNormal">PCApply 6 1.0 6.0335e+00 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 6.0e+00 15 0 0 0 1 15 0 0 0 1 0 0 0 0.00e+00 5 9.65e+01 0<u></u><u></u></p>
<p class="MsoNormal"> <u></u><u></u></p>
<p class="MsoNormal">and on cpu<u></u><u></u></p>
<p class="MsoNormal"> <u></u><u></u></p>
<p class="MsoNormal">PCApply 6 1.0 5.6348e+00 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 16 0 0 0 0 16 0 0 0 0 0<u></u><u></u></p>
<p class="MsoNormal"> <u></u><u></u></p>
<p class="MsoNormal">timings again are close but gpu version did a reduction 6.0e+00 but the cpu version did not 0.0e+00.
<u></u><u></u></p>
<p class="MsoNormal">I am not sure if that is any indication if hypre ran on gpus?<u></u><u></u></p>
<p class="MsoNormal"> <u></u><u></u></p>
<p class="MsoNormal">Thanks,<u></u><u></u></p>
<p class="MsoNormal">Karthik.<u></u><u></u></p>
<p class="MsoNormal"> <u></u><u></u></p>
<p class="MsoNormal"> <u></u><u></u></p>
<div>
<p class="MsoNormal"><b><span style="font-size:12pt;color:black">From:
</span></b><span style="font-size:12pt;color:black">Mark Adams <</span><a href="mailto:mfadams@lbl.gov" target="_blank"><span style="font-size:12pt">mfadams@lbl.gov</span></a><span style="font-size:12pt;color:black">><br>
<b>Date: </b>Friday, 8 October 2021 at 16:36<br>
<b>To: </b>"Chockalingam, Karthikeyan (STFC,DL,HC)" <</span><a href="mailto:karthikeyan.chockalingam@stfc.ac.uk" target="_blank"><span style="font-size:12pt">karthikeyan.chockalingam@stfc.ac.uk</span></a><span style="font-size:12pt;color:black">><br>
<b>Cc: </b>"</span><a href="mailto:petsc-users@mcs.anl.gov" target="_blank"><span style="font-size:12pt">petsc-users@mcs.anl.gov</span></a><span style="font-size:12pt;color:black">" <</span><a href="mailto:petsc-users@mcs.anl.gov" target="_blank"><span style="font-size:12pt">petsc-users@mcs.anl.gov</span></a><span style="font-size:12pt;color:black">><br>
<b>Subject: </b>Re: [petsc-users] hypre on gpus</span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<p class="MsoNormal"> <u></u><u></u></p>
<div>
<div>
<p class="MsoNormal">On Fri, Oct 8, 2021 at 10:29 AM Karthikeyan Chockalingam - STFC UKRI <<a href="mailto:karthikeyan.chockalingam@stfc.ac.uk" target="_blank">karthikeyan.chockalingam@stfc.ac.uk</a>>
wrote:<u></u><u></u></p>
</div>
<blockquote style="border-left:1pt solid rgb(204,204,204);padding:0cm 0cm 0cm 6pt;margin:5pt 0cm 5pt 4.8pt;border-top:currentcolor;border-right:currentcolor;border-bottom:currentcolor">
<div>
<div>
<p class="MsoNormal">The PCApply timing on<u></u><u></u></p>
<p class="MsoNormal"> <u></u><u></u></p>
<p class="MsoNormal">gpu<u></u><u></u></p>
<p class="MsoNormal"> <u></u><u></u></p>
<p>PCApply 6 1.0 1.0235e+01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 39 0 0 0 0 39 0 0 0 0 0 0 0 0.00e+00 0 0.00e+00 0<u></u><u></u></p>
<p> <u></u><u></u></p>
<p>and cpu <u></u><u></u></p>
<p> <u></u><u></u></p>
<p>PCApply 6 1.0 1.0242e+01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 41 0 0 0 0 41 0 0 0 0 0<u></u><u></u></p>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
</div>
</blockquote>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">You don't have GPUs. probably.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">Use <span style="font-family:Menlo;color:black">-dm_mat_type hypre</span>. <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<blockquote style="border-left:1pt solid rgb(204,204,204);padding:0cm 0cm 0cm 6pt;margin:5pt 0cm 5pt 4.8pt;border-top:currentcolor;border-right:currentcolor;border-bottom:currentcolor">
<div>
<div>
<p class="MsoNormal">are close. It is hard for me tell if hypre on gpu is on or not.<u></u><u></u></p>
<p class="MsoNormal"> <u></u><u></u></p>
<p class="MsoNormal">Best,<u></u><u></u></p>
<p class="MsoNormal">Karthik.<u></u><u></u></p>
<p class="MsoNormal"> <u></u><u></u></p>
<p class="MsoNormal"> <u></u><u></u></p>
<div>
<p class="MsoNormal"><b><span style="font-size:12pt;color:black">From:
</span></b><span style="font-size:12pt;color:black">"Chockalingam, Karthikeyan (STFC,DL,HC)" <</span><a href="mailto:karthikeyan.chockalingam@stfc.ac.uk" target="_blank"><span style="font-size:12pt">karthikeyan.chockalingam@stfc.ac.uk</span></a><span style="font-size:12pt;color:black">><br>
<b>Date: </b>Friday, 8 October 2021 at 14:55<br>
<b>To: </b>Mark Adams <</span><a href="mailto:mfadams@lbl.gov" target="_blank"><span style="font-size:12pt">mfadams@lbl.gov</span></a><span style="font-size:12pt;color:black">><br>
<b>Cc: </b>"</span><a href="mailto:petsc-users@mcs.anl.gov" target="_blank"><span style="font-size:12pt">petsc-users@mcs.anl.gov</span></a><span style="font-size:12pt;color:black">" <</span><a href="mailto:petsc-users@mcs.anl.gov" target="_blank"><span style="font-size:12pt">petsc-users@mcs.anl.gov</span></a><span style="font-size:12pt;color:black">><br>
<b>Subject: </b>Re: [petsc-users] hypre on gpus</span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<p class="MsoNormal">Thanks Mark, I will try your recommendations.<u></u><u></u></p>
<p class="MsoNormal">Should I also change
<span style="font-family:Menlo;color:black">-dm_vec_type to hypre </span>currently I have it as mpicuda?<u></u><u></u></p>
<p class="MsoNormal"> <u></u><u></u></p>
<p class="MsoNormal">Karthik.<u></u><u></u></p>
<p class="MsoNormal"> <u></u><u></u></p>
<p class="MsoNormal"> <u></u><u></u></p>
<div>
<p class="MsoNormal"><b><span style="font-size:12pt;color:black">From:
</span></b><span style="font-size:12pt;color:black">Mark Adams <</span><a href="mailto:mfadams@lbl.gov" target="_blank"><span style="font-size:12pt">mfadams@lbl.gov</span></a><span style="font-size:12pt;color:black">><br>
<b>Date: </b>Friday, 8 October 2021 at 14:33<br>
<b>To: </b>"Chockalingam, Karthikeyan (STFC,DL,HC)" <</span><a href="mailto:karthikeyan.chockalingam@stfc.ac.uk" target="_blank"><span style="font-size:12pt">karthikeyan.chockalingam@stfc.ac.uk</span></a><span style="font-size:12pt;color:black">><br>
<b>Cc: </b>"</span><a href="mailto:petsc-users@mcs.anl.gov" target="_blank"><span style="font-size:12pt">petsc-users@mcs.anl.gov</span></a><span style="font-size:12pt;color:black">" <</span><a href="mailto:petsc-users@mcs.anl.gov" target="_blank"><span style="font-size:12pt">petsc-users@mcs.anl.gov</span></a><span style="font-size:12pt;color:black">><br>
<b>Subject: </b>Re: [petsc-users] hypre on gpus</span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">Hypre does not record its flops with PETSc's timers.<u></u><u></u></p>
<div>
<p class="MsoNormal">Configure with and without CUDA and see if the timings change in PCApply.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">Hypre does not dynamically switch between CUDA and CPU solves at this time, but you want to use <span style="font-family:Menlo;color:black">-dm_mat_type hypre.</span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:Menlo;color:black">Mark</span><u></u><u></u></p>
</div>
</div>
<p class="MsoNormal"> <u></u><u></u></p>
<div>
<div>
<p class="MsoNormal">On Fri, Oct 8, 2021 at 6:59 AM Karthikeyan Chockalingam - STFC UKRI <<a href="mailto:karthikeyan.chockalingam@stfc.ac.uk" target="_blank">karthikeyan.chockalingam@stfc.ac.uk</a>>
wrote:<u></u><u></u></p>
</div>
<blockquote style="border-left:1pt solid rgb(204,204,204);padding:0cm 0cm 0cm 6pt;margin:5pt 0cm 5pt 4.8pt;border-top:currentcolor;border-right:currentcolor;border-bottom:currentcolor">
<div>
<div>
<p class="MsoNormal">Hello,<u></u><u></u></p>
<p class="MsoNormal"> <u></u><u></u></p>
<p class="MsoNormal">I am trying to run ex45 (in KSP tutorial) using hypre on gpus. I have attached the python configuration file and -log_view output from running the below command options<u></u><u></u></p>
<p class="MsoNormal"> <u></u><u></u></p>
<p class="MsoNormal"><span style="font-family:Menlo;color:black">mpirun -n 2 ./ex45 -log_view -da_grid_x 169 -da_grid_y 169 -da_grid_z 169 -dm_mat_type mpiaijcusparse -dm_vec_type mpicuda -ksp_type
gmres -pc_type hypre -pc_hypre_type boomeramg -ksp_gmres_restart 31 -pc_hypre_boomeramg_strong_threshold 0.7 -ksp_monitor</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-family:Menlo;color:black"> </span><u></u><u></u></p>
<p class="MsoNormal">The problem was solved and converged but from the output file I suspect hypre is not running on gpus as PCApply and DMCreate does
<b>not</b> record any gpu Mflop/s. However, some events such KSPSolve, MatMult etc are running on gpus.<u></u><u></u></p>
<p class="MsoNormal"> <u></u><u></u></p>
<p class="MsoNormal">Can you please let me know if I need to add any extra flag to the attached arch-ci-linux-cuda11-double-xx.py script file to get hypre working on gpus?<u></u><u></u></p>
<p class="MsoNormal"> <u></u><u></u></p>
<p class="MsoNormal">Thanks,<u></u><u></u></p>
<p class="MsoNormal">Karthik.<u></u><u></u></p>
<p class="MsoNormal"><span style="font-family:Menlo;color:rgb(192,0,0)"> </span><u></u><u></u></p>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<p><span style="font-size:6pt">This email and any attachments are intended solely for the use of the named recipients. If you are not the intended recipient you must not use, disclose, copy or distribute this email or any of its attachments and should notify
the sender immediately and delete this email from your system. UK Research and Innovation (UKRI) has taken every reasonable precaution to minimise risk of this email or any attachments containing viruses or malware but the recipient should carry out its own
virus and malware checks before opening the attachments. UKRI does not accept any liability for any losses or damages which the recipient may sustain due to presence of any viruses. </span><u></u><u></u></p>
</div>
</blockquote>
</div>
</div>
</div>
</blockquote>
</div>
</div>
</div>
</div>
</blockquote>
</div>
</div>
</div>
</blockquote>
</div>
</div>
</div>
</blockquote>
</div>
</div>
</div>
</blockquote>
</div>
<p class="MsoNormal"><br clear="all">
<u></u><u></u></p>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<p class="MsoNormal">--
<u></u><u></u></p>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<p class="MsoNormal">What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>
-- Norbert Wiener<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><a href="http://www.cse.buffalo.edu/~knepley/" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><u></u><u></u></p>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
<p class="MsoNormal"><br clear="all">
<u></u><u></u></p>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<p class="MsoNormal">--
<u></u><u></u></p>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<p class="MsoNormal">What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>
-- Norbert Wiener<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><a href="http://www.cse.buffalo.edu/~knepley/" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><u></u><u></u></p>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
<p class="MsoNormal"><br clear="all">
<br>
-- <u></u><u></u></p>
<div>
<p class="MsoNormal">Stefano<u></u><u></u></p>
</div>
</div>
</div>
</blockquote>
</div>
<p class="MsoNormal"><br clear="all">
<u></u><u></u></p>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<p class="MsoNormal">-- <u></u><u></u></p>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<p class="MsoNormal">What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>
-- Norbert Wiener<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal"><a href="http://www.cse.buffalo.edu/~knepley/" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><u></u><u></u></p>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote></div>