<div dir="ltr">Just a note, I think you can get meaningful hypre data by not timing the first solve and perhaps by simply calling KSPSetup before KSPSolve to get the conversion to a hypre matrix out of the way and not timed in KSPSolve.<div>ksp/ksp/tutorials/ex56.c does do this but I see snes does not. You could add KSPSetup before KSPSolve and you should see that the time reported for setup + solve == original solve.</div><div>Mark</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, Dec 15, 2021 at 3:31 PM Karthikeyan Chockalingam - STFC UKRI <<a href="mailto:karthikeyan.chockalingam@stfc.ac.uk">karthikeyan.chockalingam@stfc.ac.uk</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div lang="EN-GB" style="overflow-wrap: break-word;">
<div class="gmail-m_-1009239739655857453WordSection1">
<p class="MsoNormal"><span>Thank you for your detailed response. Yes, using
</span>aijcusparse worked (as you mentioned it is not optimal).<u></u><u></u></p>
<p class="MsoNormal"><span><u></u> <u></u></span></p>
<p class="MsoNormal"><span>My objective was to run an unstructured mesh (which I believe would be of type DMPlex), for varying size using different preconditioners by comparing their performance on cpus and gpus. I understand
that would not be possible using hypre. Leaving out hypre - I need some recommendation to move onto other unstructured (DMPlex) example problems, where I change the size of the domain via command line input.
<u></u><u></u></span></p>
<p class="MsoNormal"><span><u></u> <u></u></span></p>
<p class="MsoNormal"><span>Kind regards,<u></u><u></u></span></p>
<p class="MsoNormal"><span>Karthik.<u></u><u></u></span></p>
<p class="MsoNormal"><span><u></u> <u></u></span></p>
<p class="MsoNormal"><span><u></u> <u></u></span></p>
<p class="MsoNormal"><span><u></u> <u></u></span></p>
<p class="MsoNormal"><span><u></u> <u></u></span></p>
<p class="MsoNormal"><span><u></u> <u></u></span></p>
<div style="border-right:none;border-bottom:none;border-left:none;border-top:1pt solid rgb(181,196,223);padding:3pt 0cm 0cm">
<p class="MsoNormal"><b><span style="font-size:12pt;color:black">From: </span></b><span style="font-size:12pt;color:black">Mark Adams <<a href="mailto:mfadams@lbl.gov" target="_blank">mfadams@lbl.gov</a>><br>
<b>Date: </b>Tuesday, 14 December 2021 at 18:58<br>
<b>To: </b>"Chockalingam, Karthikeyan (STFC,DL,HC)" <<a href="mailto:karthikeyan.chockalingam@stfc.ac.uk" target="_blank">karthikeyan.chockalingam@stfc.ac.uk</a>><br>
<b>Cc: </b>Matthew Knepley <<a href="mailto:knepley@gmail.com" target="_blank">knepley@gmail.com</a>>, "<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a>" <<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a>><br>
<b>Subject: </b>Re: [petsc-users] Unstructured mesh<u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">I was able to get hypre to work on ex56 (snes) with -ex56_dm_mat_type aijcusparse -ex56_dm_vec_type cuda (not hypre matrix).<u></u><u></u></p>
<div>
<p class="MsoNormal">This should copy the cusparse matrix to a hypre matrix before the solve. So not optimal but the actual solve should be the same.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">Hypre is not yet supported on this example so you might not want to spend too much time on it.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">In particular, we do not have an example that uses DMPlex and hypre on a GPU.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">src/ksp/ksp/tutotials/ex56 is old and does not use DMPLex, but it is missing a call like ksp ex4 for hypre like this:<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">#if defined(PETSC_HAVE_HYPRE)<br>
ierr = MatHYPRESetPreallocation(A,5,NULL,5,NULL);CHKERRQ(ierr);<br>
#endif<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">If you add that it might work, but again this is all pretty fragile at this point.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">Mark<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">Mark<u></u><u></u></p>
</div>
</div>
<p class="MsoNormal"><u></u> <u></u></p>
<div>
<div>
<p class="MsoNormal">On Tue, Dec 14, 2021 at 2:42 AM Karthikeyan Chockalingam - STFC UKRI <<a href="mailto:karthikeyan.chockalingam@stfc.ac.uk" target="_blank">karthikeyan.chockalingam@stfc.ac.uk</a>> wrote:<u></u><u></u></p>
</div>
<blockquote style="border-top:none;border-right:none;border-bottom:none;border-left:1pt solid rgb(204,204,204);padding:0cm 0cm 0cm 6pt;margin:5pt 0cm 5pt 4.8pt">
<div>
<div>
<p class="MsoNormal">I tried adding the -mat_block_size 3 but I still get the same error message.</p>
<p class="MsoNormal"> </p>
<p class="MsoNormal">Thanks,</p>
<p class="MsoNormal">Karthik.</p>
<p class="MsoNormal"> </p>
<div style="border-right:none;border-bottom:none;border-left:none;border-top:1pt solid rgb(181,196,223);padding:3pt 0cm 0cm">
<p class="MsoNormal"><b><span style="font-size:12pt;color:black">From:
</span></b><span style="font-size:12pt;color:black">Mark Adams <</span><a href="mailto:mfadams@lbl.gov" target="_blank"><span style="font-size:12pt">mfadams@lbl.gov</span></a><span style="font-size:12pt;color:black">><br>
<b>Date: </b>Monday, 13 December 2021 at 19:54<br>
<b>To: </b>"Chockalingam, Karthikeyan (STFC,DL,HC)" <</span><a href="mailto:karthikeyan.chockalingam@stfc.ac.uk" target="_blank"><span style="font-size:12pt">karthikeyan.chockalingam@stfc.ac.uk</span></a><span style="font-size:12pt;color:black">><br>
<b>Cc: </b>Matthew Knepley <</span><a href="mailto:knepley@gmail.com" target="_blank"><span style="font-size:12pt">knepley@gmail.com</span></a><span style="font-size:12pt;color:black">>, "</span><a href="mailto:petsc-users@mcs.anl.gov" target="_blank"><span style="font-size:12pt">petsc-users@mcs.anl.gov</span></a><span style="font-size:12pt;color:black">"
<</span><a href="mailto:petsc-users@mcs.anl.gov" target="_blank"><span style="font-size:12pt">petsc-users@mcs.anl.gov</span></a><span style="font-size:12pt;color:black">><br>
<b>Subject: </b>Re: [petsc-users] Unstructured mesh</span></p>
</div>
<div>
<p class="MsoNormal"> </p>
</div>
<div>
<p class="MsoNormal">Try adding -mat_block_size 3</p>
<div>
<p class="MsoNormal"> </p>
</div>
</div>
<p class="MsoNormal"> </p>
<div>
<div>
<p class="MsoNormal">On Mon, Dec 13, 2021 at 11:57 AM Karthikeyan Chockalingam - STFC UKRI <<a href="mailto:karthikeyan.chockalingam@stfc.ac.uk" target="_blank">karthikeyan.chockalingam@stfc.ac.uk</a>>
wrote:</p>
</div>
<blockquote style="border-top:none;border-right:none;border-bottom:none;border-left:1pt solid rgb(204,204,204);padding:0cm 0cm 0cm 6pt;margin:5pt 0cm 5pt 4.8pt">
<div>
<div>
<p class="gmail-m_-1009239739655857453gmail-m8741764795963433052gmail-m-1073648265619177122p1"><span class="gmail-m_-1009239739655857453gmail-m8741764795963433052gmail-m-1073648265619177122s1"><span style="font-size:10pt;color:black">I tried to run the problem using -pc_type hypre but it errored out:</span></span></p>
<p class="gmail-m_-1009239739655857453gmail-m8741764795963433052gmail-m-1073648265619177122p1"><span class="gmail-m_-1009239739655857453gmail-m8741764795963433052gmail-m-1073648265619177122s1"><span style="font-size:10pt;color:black"> </span></span></p>
<p class="MsoNormal"><span style="font-size:10pt;font-family:Menlo;color:black">./ex56 -cells 4,4,2 -max_conv_its 1 -lx 1. -alpha .01 -petscspace_degree 1 -ksp_type cg -ksp_monitor -ksp_rtol 1.e-8
-pc_type hypre -pc_hypre_type<b> </b>boomeramg -snes_monitor -use_mat_nearnullspace true -snes_rtol 1.e-10 -ex56_dm_view -log_view -ex56_dm_vec_type cuda
</span><b><span style="font-size:10pt;font-family:Menlo;color:red">-ex56_dm_mat_type hypre</span></b><span style="font-size:10pt;font-family:Menlo;color:red">
</span><span style="font-size:10pt;font-family:Menlo;color:black">-options_left</span></p>
<p class="gmail-m_-1009239739655857453gmail-m8741764795963433052gmail-m-1073648265619177122p1"><span class="gmail-m_-1009239739655857453gmail-m8741764795963433052gmail-m-1073648265619177122s1"><span style="font-size:10pt;color:black"> </span></span></p>
<p class="gmail-m_-1009239739655857453gmail-m8741764795963433052gmail-m-1073648265619177122p1"><span class="gmail-m_-1009239739655857453gmail-m8741764795963433052gmail-m-1073648265619177122s1"><b><span style="font-size:8pt"> </span></b></span></p>
<p class="gmail-m_-1009239739655857453gmail-m8741764795963433052gmail-m-1073648265619177122p1"><span class="gmail-m_-1009239739655857453gmail-m8741764795963433052gmail-m-1073648265619177122s1"><b><span style="font-size:8pt"> </span></b></span></p>
<p class="gmail-m_-1009239739655857453gmail-m8741764795963433052gmail-m-1073648265619177122p1"><span class="gmail-m_-1009239739655857453gmail-m8741764795963433052gmail-m-1073648265619177122s1"><b><span style="font-size:8pt">[0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------</span></b></span></p>
<p class="gmail-m_-1009239739655857453gmail-m8741764795963433052gmail-m-1073648265619177122p2"><span class="gmail-m_-1009239739655857453gmail-m8741764795963433052gmail-m-1073648265619177122s1"><span style="font-size:8pt">[0]PETSC ERROR: Petsc has generated inconsistent data</span></span></p>
<p class="gmail-m_-1009239739655857453gmail-m8741764795963433052gmail-m-1073648265619177122p2"><span class="gmail-m_-1009239739655857453gmail-m8741764795963433052gmail-m-1073648265619177122s1"><span style="font-size:8pt">[0]PETSC ERROR: Blocksize of layout 1 must match that of mapping 3 (or the latter must be
1)</span></span></p>
<p class="gmail-m_-1009239739655857453gmail-m8741764795963433052gmail-m-1073648265619177122p2"><span class="gmail-m_-1009239739655857453gmail-m8741764795963433052gmail-m-1073648265619177122s1"><span style="font-size:8pt">[0]PETSC ERROR: See
</span></span><a href="https://petsc.org/release/faq/" target="_blank"><span style="font-size:8pt">https://petsc.org/release/faq/</span></a><span class="gmail-m_-1009239739655857453gmail-m8741764795963433052gmail-m-1073648265619177122s1"><span style="font-size:8pt"> for trouble shooting.</span></span></p>
<p class="gmail-m_-1009239739655857453gmail-m8741764795963433052gmail-m-1073648265619177122p2"><span class="gmail-m_-1009239739655857453gmail-m8741764795963433052gmail-m-1073648265619177122s1"><span style="font-size:8pt">[0]PETSC ERROR: Petsc Development GIT revision: v3.16.1-353-g887dddf386</span></span><span class="gmail-m_-1009239739655857453gmail-m8741764795963433052gmail-m-1073648265619177122apple-converted-space"><span style="font-size:8pt">
</span></span><span class="gmail-m_-1009239739655857453gmail-m8741764795963433052gmail-m-1073648265619177122s1"><span style="font-size:8pt">GIT Date: 2021-11-19 20:24:41 +0000</span></span></p>
<p class="gmail-m_-1009239739655857453gmail-m8741764795963433052gmail-m-1073648265619177122p2"><span class="gmail-m_-1009239739655857453gmail-m8741764795963433052gmail-m-1073648265619177122s1"><span style="font-size:8pt">[0]PETSC ERROR: ./ex56 on a arch-linux2-c-opt named sqg2b13.bullx by kxc07-lxm25 Mon Dec 13
16:50:02 2021</span></span></p>
<p class="gmail-m_-1009239739655857453gmail-m8741764795963433052gmail-m-1073648265619177122p2"><span class="gmail-m_-1009239739655857453gmail-m8741764795963433052gmail-m-1073648265619177122s1"><span style="font-size:8pt">[0]PETSC ERROR: Configure options --with-debugging=0 --with-blaslapack-dir=/lustre/scafellpike/local/apps/intel/intel_cs/2018.0.128/mkl
--with-cuda=1 --with-cuda-arch=70 --download-hypre=yes --download-hypre-configure-arguments="--with-cuda=yes --enable-gpu-profiling=yes --enable-cusparse=yes --enable-cublas=yes --enable-curand=yes</span></span><span class="gmail-m_-1009239739655857453gmail-m8741764795963433052gmail-m-1073648265619177122apple-converted-space"><span style="font-size:8pt">
</span></span><span class="gmail-m_-1009239739655857453gmail-m8741764795963433052gmail-m-1073648265619177122s1"><span style="font-size:8pt">--enable-unified-memory=yes HYPRE_CUDA_SM=70" --with-shared-libraries=1 --known-mpi-shared-libraries=1 --with-cc=mpicc --with-cxx=mpicxx -with-fc=mpif90</span></span></p>
<p class="gmail-m_-1009239739655857453gmail-m8741764795963433052gmail-m-1073648265619177122p2"><span class="gmail-m_-1009239739655857453gmail-m8741764795963433052gmail-m-1073648265619177122s1"><span style="font-size:8pt">[0]PETSC ERROR: #1 PetscLayoutSetISLocalToGlobalMapping() at /lustre/scafellpike/local/HT04048/lxm25/kxc07-lxm25/petsc-main/petsc/src/vec/is/utils/pmap.c:371</span></span></p>
<p class="gmail-m_-1009239739655857453gmail-m8741764795963433052gmail-m-1073648265619177122p2"><span class="gmail-m_-1009239739655857453gmail-m8741764795963433052gmail-m-1073648265619177122s1"><span style="font-size:8pt">[0]PETSC ERROR: #2 MatSetLocalToGlobalMapping() at /lustre/scafellpike/local/HT04048/lxm25/kxc07-lxm25/petsc-main/petsc/src/mat/interface/matrix.c:2089</span></span></p>
<p class="gmail-m_-1009239739655857453gmail-m8741764795963433052gmail-m-1073648265619177122p2"><span class="gmail-m_-1009239739655857453gmail-m8741764795963433052gmail-m-1073648265619177122s1"><span style="font-size:8pt">[0]PETSC ERROR: #3 DMCreateMatrix_Plex() at /lustre/scafellpike/local/HT04048/lxm25/kxc07-lxm25/petsc-main/petsc/src/dm/impls/plex/plex.c:2460</span></span></p>
<p class="gmail-m_-1009239739655857453gmail-m8741764795963433052gmail-m-1073648265619177122p2"><span class="gmail-m_-1009239739655857453gmail-m8741764795963433052gmail-m-1073648265619177122s1"><span style="font-size:8pt">[0]PETSC ERROR: #4 DMCreateMatrix() at /lustre/scafellpike/local/HT04048/lxm25/kxc07-lxm25/petsc-main/petsc/src/dm/interface/dm.c:1445</span></span></p>
<p class="gmail-m_-1009239739655857453gmail-m8741764795963433052gmail-m-1073648265619177122p2"><span class="gmail-m_-1009239739655857453gmail-m8741764795963433052gmail-m-1073648265619177122s1"><span style="font-size:8pt">[0]PETSC ERROR: #5 main() at ex56.c:439</span></span></p>
<p class="gmail-m_-1009239739655857453gmail-m8741764795963433052gmail-m-1073648265619177122p2"><span class="gmail-m_-1009239739655857453gmail-m8741764795963433052gmail-m-1073648265619177122s1"><span style="font-size:8pt">[0]PETSC ERROR: PETSc Option Table entries:</span></span></p>
<p class="gmail-m_-1009239739655857453gmail-m8741764795963433052gmail-m-1073648265619177122p2"><span class="gmail-m_-1009239739655857453gmail-m8741764795963433052gmail-m-1073648265619177122s1"><span style="font-size:8pt">[0]PETSC ERROR: -alpha .01</span></span></p>
<p class="gmail-m_-1009239739655857453gmail-m8741764795963433052gmail-m-1073648265619177122p2"><span class="gmail-m_-1009239739655857453gmail-m8741764795963433052gmail-m-1073648265619177122s1"><span style="font-size:8pt">[0]PETSC ERROR: -cells 4,4,2</span></span></p>
<p class="gmail-m_-1009239739655857453gmail-m8741764795963433052gmail-m-1073648265619177122p2"><span class="gmail-m_-1009239739655857453gmail-m8741764795963433052gmail-m-1073648265619177122s1"><span style="font-size:8pt">[0]PETSC ERROR: -ex56_dm_mat_type hypre</span></span></p>
<p class="gmail-m_-1009239739655857453gmail-m8741764795963433052gmail-m-1073648265619177122p2"><span class="gmail-m_-1009239739655857453gmail-m8741764795963433052gmail-m-1073648265619177122s1"><span style="font-size:8pt">[0]PETSC ERROR: -ex56_dm_vec_type cuda</span></span></p>
<p class="gmail-m_-1009239739655857453gmail-m8741764795963433052gmail-m-1073648265619177122p2"><span class="gmail-m_-1009239739655857453gmail-m8741764795963433052gmail-m-1073648265619177122s1"><span style="font-size:8pt">[0]PETSC ERROR: -ex56_dm_view</span></span></p>
<p class="gmail-m_-1009239739655857453gmail-m8741764795963433052gmail-m-1073648265619177122p2"><span class="gmail-m_-1009239739655857453gmail-m8741764795963433052gmail-m-1073648265619177122s1"><span style="font-size:8pt">[0]PETSC ERROR: -ksp_monitor</span></span></p>
<p class="gmail-m_-1009239739655857453gmail-m8741764795963433052gmail-m-1073648265619177122p2"><span class="gmail-m_-1009239739655857453gmail-m8741764795963433052gmail-m-1073648265619177122s1"><span style="font-size:8pt">[0]PETSC ERROR: -ksp_rtol 1.e-8</span></span></p>
<p class="gmail-m_-1009239739655857453gmail-m8741764795963433052gmail-m-1073648265619177122p2"><span class="gmail-m_-1009239739655857453gmail-m8741764795963433052gmail-m-1073648265619177122s1"><span style="font-size:8pt">[0]PETSC ERROR: -ksp_type cg</span></span></p>
<p class="gmail-m_-1009239739655857453gmail-m8741764795963433052gmail-m-1073648265619177122p2"><span class="gmail-m_-1009239739655857453gmail-m8741764795963433052gmail-m-1073648265619177122s1"><span style="font-size:8pt">[0]PETSC ERROR: -log_view</span></span></p>
<p class="gmail-m_-1009239739655857453gmail-m8741764795963433052gmail-m-1073648265619177122p2"><span class="gmail-m_-1009239739655857453gmail-m8741764795963433052gmail-m-1073648265619177122s1"><span style="font-size:8pt">[0]PETSC ERROR: -lx 1.</span></span></p>
<p class="gmail-m_-1009239739655857453gmail-m8741764795963433052gmail-m-1073648265619177122p2"><span class="gmail-m_-1009239739655857453gmail-m8741764795963433052gmail-m-1073648265619177122s1"><span style="font-size:8pt">[0]PETSC ERROR: -max_conv_its 1</span></span></p>
<p class="gmail-m_-1009239739655857453gmail-m8741764795963433052gmail-m-1073648265619177122p2"><span class="gmail-m_-1009239739655857453gmail-m8741764795963433052gmail-m-1073648265619177122s1"><span style="font-size:8pt">[0]PETSC ERROR: -options_left</span></span></p>
<p class="gmail-m_-1009239739655857453gmail-m8741764795963433052gmail-m-1073648265619177122p2"><span class="gmail-m_-1009239739655857453gmail-m8741764795963433052gmail-m-1073648265619177122s1"><span style="font-size:8pt">[0]PETSC ERROR: -pc_hypre_type boomeramg</span></span></p>
<p class="gmail-m_-1009239739655857453gmail-m8741764795963433052gmail-m-1073648265619177122p2"><span class="gmail-m_-1009239739655857453gmail-m8741764795963433052gmail-m-1073648265619177122s1"><span style="font-size:8pt">[0]PETSC ERROR: -pc_type hypre</span></span></p>
<p class="gmail-m_-1009239739655857453gmail-m8741764795963433052gmail-m-1073648265619177122p2"><span class="gmail-m_-1009239739655857453gmail-m8741764795963433052gmail-m-1073648265619177122s1"><span style="font-size:8pt">[0]PETSC ERROR: -petscspace_degree 1</span></span></p>
<p class="gmail-m_-1009239739655857453gmail-m8741764795963433052gmail-m-1073648265619177122p2"><span class="gmail-m_-1009239739655857453gmail-m8741764795963433052gmail-m-1073648265619177122s1"><span style="font-size:8pt">[0]PETSC ERROR: -snes_monitor</span></span></p>
<p class="gmail-m_-1009239739655857453gmail-m8741764795963433052gmail-m-1073648265619177122p2"><span class="gmail-m_-1009239739655857453gmail-m8741764795963433052gmail-m-1073648265619177122s1"><span style="font-size:8pt">[0]PETSC ERROR: -snes_rtol 1.e-10</span></span></p>
<p class="gmail-m_-1009239739655857453gmail-m8741764795963433052gmail-m-1073648265619177122p2"><span class="gmail-m_-1009239739655857453gmail-m8741764795963433052gmail-m-1073648265619177122s1"><span style="font-size:8pt">[0]PETSC ERROR: -use_gpu_aware_mpi 0</span></span></p>
<p class="gmail-m_-1009239739655857453gmail-m8741764795963433052gmail-m-1073648265619177122p2"><span class="gmail-m_-1009239739655857453gmail-m8741764795963433052gmail-m-1073648265619177122s1"><span style="font-size:8pt">[0]PETSC ERROR: -use_mat_nearnullspace true</span></span></p>
<p class="gmail-m_-1009239739655857453gmail-m8741764795963433052gmail-m-1073648265619177122p1"><span class="gmail-m_-1009239739655857453gmail-m8741764795963433052gmail-m-1073648265619177122s1"><b><span style="font-size:8pt">[0]PETSC ERROR: ----------------End of Error Message -------send entire error message to
petsc-maint@mcs.anl.gov----------</span></b></span></p>
<p class="gmail-m_-1009239739655857453gmail-m8741764795963433052gmail-m-1073648265619177122p2"><span class="gmail-m_-1009239739655857453gmail-m8741764795963433052gmail-m-1073648265619177122s1"><span style="font-size:8pt">--------------------------------------------------------------------------</span></span></p>
<p class="gmail-m_-1009239739655857453gmail-m8741764795963433052gmail-m-1073648265619177122p2"><span class="gmail-m_-1009239739655857453gmail-m8741764795963433052gmail-m-1073648265619177122s1"><span style="font-size:8pt">MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_WORLD</span></span></p>
<p class="gmail-m_-1009239739655857453gmail-m8741764795963433052gmail-m-1073648265619177122p2"><span class="gmail-m_-1009239739655857453gmail-m8741764795963433052gmail-m-1073648265619177122s1"><span style="font-size:8pt">with errorcode 77.</span></span></p>
<p class="gmail-m_-1009239739655857453gmail-m8741764795963433052gmail-m-1073648265619177122p3"><span style="font-size:8pt"> </span></p>
<p class="gmail-m_-1009239739655857453gmail-m8741764795963433052gmail-m-1073648265619177122p2"><span class="gmail-m_-1009239739655857453gmail-m8741764795963433052gmail-m-1073648265619177122s1"><span style="font-size:8pt">NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.</span></span></p>
<p class="gmail-m_-1009239739655857453gmail-m8741764795963433052gmail-m-1073648265619177122p2"><span class="gmail-m_-1009239739655857453gmail-m8741764795963433052gmail-m-1073648265619177122s1"><span style="font-size:8pt">You may or may not see output from other processes, depending on</span></span></p>
<p class="gmail-m_-1009239739655857453gmail-m8741764795963433052gmail-m-1073648265619177122p2"><span class="gmail-m_-1009239739655857453gmail-m8741764795963433052gmail-m-1073648265619177122s1"><span style="font-size:8pt">exactly when Open MPI kills them.</span></span></p>
<p class="gmail-m_-1009239739655857453gmail-m8741764795963433052gmail-m-1073648265619177122p2"><span class="gmail-m_-1009239739655857453gmail-m8741764795963433052gmail-m-1073648265619177122s1"><span style="font-size:8pt">--------------------------------------------------------------------------</span></span></p>
<p class="MsoNormal"><span style="font-size:8pt"> </span></p>
<p class="MsoNormal"> </p>
<div style="border-right:none;border-bottom:none;border-left:none;border-top:1pt solid rgb(181,196,223);padding:3pt 0cm 0cm">
<p class="MsoNormal"><b><span style="font-size:12pt;color:black">From:
</span></b><span style="font-size:12pt;color:black">Mark Adams <</span><a href="mailto:mfadams@lbl.gov" target="_blank"><span style="font-size:12pt">mfadams@lbl.gov</span></a><span style="font-size:12pt;color:black">><br>
<b>Date: </b>Monday, 13 December 2021 at 13:58<br>
<b>To: </b>"Chockalingam, Karthikeyan (STFC,DL,HC)" <</span><a href="mailto:karthikeyan.chockalingam@stfc.ac.uk" target="_blank"><span style="font-size:12pt">karthikeyan.chockalingam@stfc.ac.uk</span></a><span style="font-size:12pt;color:black">><br>
<b>Cc: </b>Matthew Knepley <</span><a href="mailto:knepley@gmail.com" target="_blank"><span style="font-size:12pt">knepley@gmail.com</span></a><span style="font-size:12pt;color:black">>, "</span><a href="mailto:petsc-users@mcs.anl.gov" target="_blank"><span style="font-size:12pt">petsc-users@mcs.anl.gov</span></a><span style="font-size:12pt;color:black">"
<</span><a href="mailto:petsc-users@mcs.anl.gov" target="_blank"><span style="font-size:12pt">petsc-users@mcs.anl.gov</span></a><span style="font-size:12pt;color:black">><br>
<b>Subject: </b>Re: [petsc-users] Unstructured mesh</span></p>
</div>
<div>
<p class="MsoNormal"> </p>
</div>
<div>
<div>
<p class="MsoNormal"> </p>
</div>
<p class="MsoNormal"> </p>
<div>
<div>
<p class="MsoNormal">On Mon, Dec 13, 2021 at 8:35 AM Karthikeyan Chockalingam - STFC UKRI <<a href="mailto:karthikeyan.chockalingam@stfc.ac.uk" target="_blank">karthikeyan.chockalingam@stfc.ac.uk</a>>
wrote:</p>
</div>
<blockquote style="border-top:none;border-right:none;border-bottom:none;border-left:1pt solid rgb(204,204,204);padding:0cm 0cm 0cm 6pt;margin:5pt 0cm 5pt 4.8pt">
<div>
<div>
<p class="MsoNormal">Thanks Matt. Couple of weeks back you mentioned</p>
<p class="MsoNormal"><span style="font-size:10pt;font-family:Menlo">“<span style="color:black">There are many unstructured grid examples, e.g. SNES ex13, ex17, ex56. The solver can run on the GPU,
but the vector/matrix FEM assembly does not. I am working on that now.”</span></span></p>
<p class="MsoNormal"> </p>
<p class="MsoNormal">I am able to run other examples in ksp/tutorials on gpus. I complied ex56 in snes/tutorials no differently. The only difference being I didn’t specify _dm_vec_type and _dm_vec_type
(as you mentioned they are not assembled on gpus anyways plus I am working on an unstructured grid thought _dm is not right type for this problem). I was hoping to see gpu flops recorded for KSPSolve, which I didn’t.
</p>
<p class="MsoNormal"> </p>
<p class="MsoNormal">Okay, I will wait for Mark to comment.</p>
</div>
</div>
</blockquote>
<div>
<p class="MsoNormal"> </p>
</div>
<div>
<p class="MsoNormal">This (DM) example works like any other, with a prefix, as far as GPU: -ex56_dm_vec_type cuda and -ex56_dm_mat_type cusparse, or aijkokkos/kokkos, etc.</p>
</div>
<div>
<p class="MsoNormal">Run with -options_left to verify that these are used. </p>
</div>
<div>
<p class="MsoNormal"> </p>
</div>
<blockquote style="border-top:none;border-right:none;border-bottom:none;border-left:1pt solid rgb(204,204,204);padding:0cm 0cm 0cm 6pt;margin:5pt 0cm 5pt 4.8pt">
<div>
<div>
<p class="MsoNormal"> </p>
<p class="MsoNormal">Kind regards,</p>
<p class="MsoNormal">Karthik.</p>
<p class="MsoNormal"> </p>
<div style="border-right:none;border-bottom:none;border-left:none;border-top:1pt solid rgb(181,196,223);padding:3pt 0cm 0cm">
<p class="MsoNormal"><b><span style="font-size:12pt;color:black">From:
</span></b><span style="font-size:12pt;color:black">Matthew Knepley <</span><a href="mailto:knepley@gmail.com" target="_blank"><span style="font-size:12pt">knepley@gmail.com</span></a><span style="font-size:12pt;color:black">><br>
<b>Date: </b>Monday, 13 December 2021 at 13:17<br>
<b>To: </b>"Chockalingam, Karthikeyan (STFC,DL,HC)" <</span><a href="mailto:karthikeyan.chockalingam@stfc.ac.uk" target="_blank"><span style="font-size:12pt">karthikeyan.chockalingam@stfc.ac.uk</span></a><span style="font-size:12pt;color:black">><br>
<b>Cc: </b>Mark Adams <</span><a href="mailto:mfadams@lbl.gov" target="_blank"><span style="font-size:12pt">mfadams@lbl.gov</span></a><span style="font-size:12pt;color:black">>, "</span><a href="mailto:petsc-users@mcs.anl.gov" target="_blank"><span style="font-size:12pt">petsc-users@mcs.anl.gov</span></a><span style="font-size:12pt;color:black">"
<</span><a href="mailto:petsc-users@mcs.anl.gov" target="_blank"><span style="font-size:12pt">petsc-users@mcs.anl.gov</span></a><span style="font-size:12pt;color:black">><br>
<b>Subject: </b>Re: [petsc-users] Unstructured mesh</span></p>
</div>
<div>
<p class="MsoNormal"> </p>
</div>
<div>
<div>
<p class="MsoNormal">On Mon, Dec 13, 2021 at 7:15 AM Karthikeyan Chockalingam - STFC UKRI <<a href="mailto:karthikeyan.chockalingam@stfc.ac.uk" target="_blank">karthikeyan.chockalingam@stfc.ac.uk</a>>
wrote:</p>
</div>
<div>
<blockquote style="border-top:none;border-right:none;border-bottom:none;border-left:1pt solid rgb(204,204,204);padding:0cm 0cm 0cm 6pt;margin:5pt 0cm 5pt 4.8pt">
<div>
<div>
<p class="gmail-m_-1009239739655857453gmail-m8741764795963433052gmail-m-1073648265619177122gmail-m-743708965511543589gmail-m4613828314074599431p1">
<span class="gmail-m_-1009239739655857453gmail-m8741764795963433052gmail-m-1073648265619177122gmail-m-743708965511543589gmail-m4613828314074599431apple-converted-space"><span style="font-size:10pt">Thank you. I was able to confirm both the below options produced the same mesh</span></span></p>
<p class="gmail-m_-1009239739655857453gmail-m8741764795963433052gmail-m-1073648265619177122gmail-m-743708965511543589gmail-m4613828314074599431p1">
<span class="gmail-m_-1009239739655857453gmail-m8741764795963433052gmail-m-1073648265619177122gmail-m-743708965511543589gmail-m4613828314074599431apple-converted-space"><span style="font-size:10pt"> </span></span></p>
<p class="gmail-m_-1009239739655857453gmail-m8741764795963433052gmail-m-1073648265619177122gmail-m-743708965511543589gmail-m4613828314074599431p1">
<span class="gmail-m_-1009239739655857453gmail-m8741764795963433052gmail-m-1073648265619177122gmail-m-743708965511543589gmail-m4613828314074599431apple-converted-space"><span style="font-size:10pt"> </span></span><span class="gmail-m_-1009239739655857453gmail-m8741764795963433052gmail-m-1073648265619177122gmail-m-743708965511543589gmail-m4613828314074599431s1"><span style="font-size:10pt">./ex56
-cells 2,2,1 -max_conv_its 2</span></span></p>
<p class="gmail-m_-1009239739655857453gmail-m8741764795963433052gmail-m-1073648265619177122gmail-m-743708965511543589gmail-m4613828314074599431p1">
<span class="gmail-m_-1009239739655857453gmail-m8741764795963433052gmail-m-1073648265619177122gmail-m-743708965511543589gmail-m4613828314074599431s1"><span style="font-size:10pt">./ex56 -cells 4,4,2 -max_conv_its 1</span></span></p>
</div>
</div>
</blockquote>
<div>
<p class="MsoNormal">Good </p>
</div>
<blockquote style="border-top:none;border-right:none;border-bottom:none;border-left:1pt solid rgb(204,204,204);padding:0cm 0cm 0cm 6pt;margin:5pt 0cm 5pt 4.8pt">
<div>
<div>
<p class="gmail-m_-1009239739655857453gmail-m8741764795963433052gmail-m-1073648265619177122gmail-m-743708965511543589gmail-m4613828314074599431p1">
<span class="gmail-m_-1009239739655857453gmail-m8741764795963433052gmail-m-1073648265619177122gmail-m-743708965511543589gmail-m4613828314074599431s1"><span style="font-size:10pt">But I didn’t get how is
</span></span><span style="font-size:10pt">-cells i,j,k <1,1,1><span class="gmail-m_-1009239739655857453gmail-m8741764795963433052gmail-m-1073648265619177122gmail-m-743708965511543589gmail-m4613828314074599431s1"> is related to the number of MPI processes.</span></span></p>
</div>
</div>
</blockquote>
<div>
<p class="MsoNormal">It is not. The number of processes is specified independently using 'mpiexec -n <p>' or when using the test system NP=<p>. </p>
</div>
<blockquote style="border-top:none;border-right:none;border-bottom:none;border-left:1pt solid rgb(204,204,204);padding:0cm 0cm 0cm 6pt;margin:5pt 0cm 5pt 4.8pt">
<div>
<div>
<p class="gmail-m_-1009239739655857453gmail-m8741764795963433052gmail-m-1073648265619177122gmail-m-743708965511543589gmail-m4613828314074599431msolistparagraph" style="margin-left:54pt">
<span class="gmail-m_-1009239739655857453gmail-m8741764795963433052gmail-m-1073648265619177122gmail-m-743708965511543589gmail-m4613828314074599431s1"><span style="font-size:10pt;font-family:Menlo">(i)</span></span><span class="gmail-m_-1009239739655857453gmail-m8741764795963433052gmail-m-1073648265619177122gmail-m-743708965511543589gmail-m4613828314074599431s1"><span style="font-size:7pt;font-family:"Times New Roman",serif">
</span></span><span style="font-size:10pt;font-family:Menlo">Say I start with -cells 1,1,1 -max_conv its<span class="gmail-m_-1009239739655857453gmail-m8741764795963433052gmail-m-1073648265619177122gmail-m-743708965511543589gmail-m4613828314074599431s1"> 7; that would eventually leave
all refinement on level 7 running on 1 MPI process?</span></span></p>
<p class="gmail-m_-1009239739655857453gmail-m8741764795963433052gmail-m-1073648265619177122gmail-m-743708965511543589gmail-m4613828314074599431msolistparagraph" style="margin-left:54pt">
<span style="font-size:10pt;font-family:Menlo">(ii)</span><span style="font-size:7pt;font-family:"Times New Roman",serif">
</span><span style="font-size:10pt;font-family:Menlo">Say I start with -cells 2,2,1 -max_conv its<span class="gmail-m_-1009239739655857453gmail-m8741764795963433052gmail-m-1073648265619177122gmail-m-743708965511543589gmail-m4613828314074599431s1"> n</span>; is it recommended to run
on 4 MPI processes?</span></p>
</div>
</div>
</blockquote>
<div>
<p class="MsoNormal">No, those options do not influence the number of processes.</p>
</div>
<div>
<p class="MsoNormal"> </p>
</div>
<blockquote style="border-top:none;border-right:none;border-bottom:none;border-left:1pt solid rgb(204,204,204);padding:0cm 0cm 0cm 6pt;margin:5pt 0cm 5pt 4.8pt">
<div>
<div>
<p class="MsoNormal">I am running ex56 on gpu; I am looking at KSPSolve (or any other event) but no gpu flops are recorded in the -log_view?</p>
</div>
</div>
</blockquote>
<div>
<p class="MsoNormal"> </p>
</div>
<div>
<p class="MsoNormal">I do not think you are running on the GPU then. Mark can comment, but we usually specify GPU execution using the Vec and Mat types</p>
</div>
<div>
<p class="MsoNormal">through -dm_vec_type and -dm_mat_type.</p>
</div>
<div>
<p class="MsoNormal"> </p>
</div>
<div>
<p class="MsoNormal"> Thanks,</p>
</div>
<div>
<p class="MsoNormal"> </p>
</div>
<div>
<p class="MsoNormal"> Matt</p>
</div>
<div>
<p class="MsoNormal"> </p>
</div>
<blockquote style="border-top:none;border-right:none;border-bottom:none;border-left:1pt solid rgb(204,204,204);padding:0cm 0cm 0cm 6pt;margin:5pt 0cm 5pt 4.8pt">
<div>
<div>
<p class="MsoNormal">For your reference I used the below flags:</p>
<p class="gmail-m_-1009239739655857453gmail-m8741764795963433052gmail-m-1073648265619177122gmail-m-743708965511543589gmail-m4613828314074599431p1">
<span class="gmail-m_-1009239739655857453gmail-m8741764795963433052gmail-m-1073648265619177122gmail-m-743708965511543589gmail-m4613828314074599431s1"><span style="font-size:10pt">./ex56 -cells 1,1,1 -max_conv_its 3 -lx 1. -alpha .01 -petscspace_degree 1 -ksp_type cg -ksp_monitor -ksp_rtol
1.e-8 -pc_type asm -snes_monitor -use_mat_nearnullspace true -snes_rtol 1.e-10 -ex56_dm_view -log_view</span></span></p>
<p class="MsoNormal"> </p>
<p class="MsoNormal">Kind regards,</p>
<p class="MsoNormal">Karthik.</p>
<p class="MsoNormal"> </p>
<p class="MsoNormal"> </p>
<div style="border-right:none;border-bottom:none;border-left:none;border-top:1pt solid rgb(181,196,223);padding:3pt 0cm 0cm">
<p class="MsoNormal"><b><span style="font-size:12pt;color:black">From:
</span></b><span style="font-size:12pt;color:black">Mark Adams <</span><a href="mailto:mfadams@lbl.gov" target="_blank"><span style="font-size:12pt">mfadams@lbl.gov</span></a><span style="font-size:12pt;color:black">><br>
<b>Date: </b>Sunday, 12 December 2021 at 23:00<br>
<b>To: </b>"Chockalingam, Karthikeyan (STFC,DL,HC)" <</span><a href="mailto:karthikeyan.chockalingam@stfc.ac.uk" target="_blank"><span style="font-size:12pt">karthikeyan.chockalingam@stfc.ac.uk</span></a><span style="font-size:12pt;color:black">><br>
<b>Cc: </b>Matthew Knepley <</span><a href="mailto:knepley@gmail.com" target="_blank"><span style="font-size:12pt">knepley@gmail.com</span></a><span style="font-size:12pt;color:black">>, "</span><a href="mailto:petsc-users@mcs.anl.gov" target="_blank"><span style="font-size:12pt">petsc-users@mcs.anl.gov</span></a><span style="font-size:12pt;color:black">"
<</span><a href="mailto:petsc-users@mcs.anl.gov" target="_blank"><span style="font-size:12pt">petsc-users@mcs.anl.gov</span></a><span style="font-size:12pt;color:black">><br>
<b>Subject: </b>Re: [petsc-users] Unstructured mesh</span></p>
</div>
<div>
<p class="MsoNormal"> </p>
</div>
<div>
<div>
<p class="MsoNormal"> </p>
</div>
<p class="MsoNormal"> </p>
<div>
<div>
<p class="MsoNormal">On Sun, Dec 12, 2021 at 3:19 PM Karthikeyan Chockalingam - STFC UKRI <<a href="mailto:karthikeyan.chockalingam@stfc.ac.uk" target="_blank">karthikeyan.chockalingam@stfc.ac.uk</a>>
wrote:</p>
</div>
<blockquote style="border-top:none;border-right:none;border-bottom:none;border-left:1pt solid rgb(204,204,204);padding:0cm 0cm 0cm 6pt;margin:5pt 0cm 5pt 4.8pt">
<div>
<div>
<p class="MsoNormal">Thank for your response that was helpful. I have a couple of questions:</p>
<p class="MsoNormal"> </p>
<p style="margin-left:54pt">(i)<span style="font-size:7pt;font-family:"Times New Roman",serif">
</span>How can I control the level of refinement? I tried to pass the flag “-ex56_dm_refine 0” but that didn’t stop the refinement from 8 giving 32 cubes.</p>
</div>
</div>
</blockquote>
<div>
<p class="MsoNormal"> </p>
</div>
<div>
<p class="MsoNormal">I answered this question recently but ex56 clobbers ex56_dm_refine in the convergence loop. I have an MR that prints a warning if you provide a ex56_dm_refine. </p>
</div>
<div>
<p class="MsoNormal"> </p>
</div>
<div>
<p class="MsoNormal">* snes/ex56 runs a convergence study and confusingly sets the options manually, thus erasing your -ex56_dm_refine.</p>
<div>
<p class="MsoNormal"> </p>
</div>
<div>
<p class="MsoNormal">* To refine, use -max_conv_its N <3>, this sets the number of steps of refinement. That is, the length of the convergence study</p>
</div>
<div>
<p class="MsoNormal"> </p>
</div>
<div>
<p class="MsoNormal">* You can adjust where it starts from with -cells i,j,k <1,1,1></p>
</div>
<div>
<div>
<p class="MsoNormal">You do want to set this if you have multiple MPI processes so that the size of this mesh is the number of processes. That way it starts with one cell per process and refines from
there.</p>
</div>
<div>
<p class="MsoNormal"> </p>
</div>
</div>
</div>
<blockquote style="border-top:none;border-right:none;border-bottom:none;border-left:1pt solid rgb(204,204,204);padding:0cm 0cm 0cm 6pt;margin:5pt 0cm 5pt 4.8pt">
<div>
<div>
<p style="margin-left:54pt">(ii)<span style="font-size:7pt;font-family:"Times New Roman",serif">
</span>What does -cell 2,2,1 correspond to? </p>
</div>
</div>
</blockquote>
<div>
<p class="MsoNormal"> </p>
</div>
<div>
<p class="MsoNormal">The initial mesh or mesh_0. The convergence test uniformly refines this mesh. So if you want to refine this twice you could use -cells 8,8,4</p>
</div>
<div>
<p class="MsoNormal"> </p>
</div>
<blockquote style="border-top:none;border-right:none;border-bottom:none;border-left:1pt solid rgb(204,204,204);padding:0cm 0cm 0cm 6pt;margin:5pt 0cm 5pt 4.8pt">
<div>
<div>
<p style="margin-left:54pt">How can I determine the total number of dofs? </p>
</div>
</div>
</blockquote>
<div>
<p class="MsoNormal">Unfortunately, that is not printed but you can calculate from the initial cell grid, the order of the element and the refinement in each iteration of the convergence tests.</p>
</div>
<div>
<p class="MsoNormal"> </p>
</div>
<blockquote style="border-top:none;border-right:none;border-bottom:none;border-left:1pt solid rgb(204,204,204);padding:0cm 0cm 0cm 6pt;margin:5pt 0cm 5pt 4.8pt">
<div>
<div>
<p style="margin-left:54pt">So that I can perform a scaling study by changing the input of the flag -cells.</p>
<p class="MsoNormal"> </p>
</div>
</div>
</blockquote>
<div>
<p class="MsoNormal"> </p>
</div>
<div>
<p class="MsoNormal">You can and the convergence test gives you data for a strong speedup study in one run. Each solve is put in its own "stage" of the output and you want to look at KSPSolve lines
in the log_view output. </p>
</div>
</div>
</div>
</div>
<p><span style="font-size:6pt">This email and any attachments are intended solely for the use of the named recipients. If you are not the intended recipient you must not use, disclose, copy or distribute this email or any of its attachments and should notify
the sender immediately and delete this email from your system. UK Research and Innovation (UKRI) has taken every reasonable precaution to minimise risk of this email or any attachments containing viruses or malware but the recipient should carry out its own
virus and malware checks before opening the attachments. UKRI does not accept any liability for any losses or damages which the recipient may sustain due to presence of any viruses. </span></p>
</div>
</blockquote>
</div>
<p class="MsoNormal"><br clear="all">
</p>
<div>
<p class="MsoNormal"> </p>
</div>
<p class="MsoNormal">--
</p>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<p class="MsoNormal">What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>
-- Norbert Wiener</p>
</div>
<div>
<p class="MsoNormal"> </p>
</div>
<div>
<p class="MsoNormal"><a href="http://www.cse.buffalo.edu/~knepley/" target="_blank">https://www.cse.buffalo.edu/~knepley/</a></p>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
</div>
</div>
</div>
</blockquote>
</div>
</div>
</div>
</blockquote>
</div>
</div>
</div>
</blockquote></div>