<div dir="ltr"><div dir="ltr"><br><br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, Oct 27, 2021 at 3:37 PM Karthikeyan Chockalingam - STFC UKRI <<a href="mailto:karthikeyan.chockalingam@stfc.ac.uk">karthikeyan.chockalingam@stfc.ac.uk</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div lang="EN-GB" style="overflow-wrap: break-word;">
<div class="gmail-m_9076948005455290591WordSection1">
<p class="MsoNormal"><span>Thank you for your response.<u></u><u></u></span></p>
<p class="MsoNormal"><span><u></u> <u></u></span></p>
<p class="MsoNormal"><span>It tried running ksp/ex2.c using<u></u><u></u></span></p>
<p class="MsoNormal"><span><u></u> <u></u></span></p>
<p class="gmail-m_9076948005455290591p1"><span class="gmail-m_9076948005455290591s1"><span style="font-size:10pt">./ex2</span></span><span class="gmail-m_9076948005455290591apple-converted-space"><span style="font-size:10pt"> </span></span><span class="gmail-m_9076948005455290591s1"><span style="font-size:10pt">-m 9 -n 9 –vec_type cuda -mat_type aijcusparse</span></span><span class="gmail-m_9076948005455290591apple-converted-space"><span style="font-size:10pt"> </span></span><span class="gmail-m_9076948005455290591s1"><span style="font-size:10pt">-ksp_type
cg</span></span><span class="gmail-m_9076948005455290591apple-converted-space"><span style="font-size:10pt"> </span></span><span class="gmail-m_9076948005455290591s1"><span style="font-size:10pt">-pc_type jacobi -log_view</span></span></p></div></div></blockquote><div>under src/ksp/ksp/tutorials, run this command (your old command line has a weird character <span style="font-size:13.3333px">–</span>)</div><div>./ex2 -m 9 -n 9 -vec_type cuda -mat_type aijcusparse -ksp_type cg -pc_type jacobi -log_view </div><div><br></div><div><span style="font-family:monospace">Event Count Time (sec) Flop --- Global --- --- Stage ---- Total GPU - CpuToGpu - - GpuToCpu - GPU</span><br></div><div><font face="monospace"> Max Ratio Max Ratio Max Ratio Mess AvgLen Reduct %T %F %M %L %R %T %F %M %L %R Mflop/s Mflop/s Count Size Count Size %F<br>---------------------------------------------------------------------------------------------------------------------------------------------------------------<br><br>--- Event Stage 0: Main Stage<br><br>MatMult 14 1.0 5.5454e-04 1.0 9.20e+03 1.0 0.0e+00 0.0e+00 0.0e+00 0 40 0 0 0 0 40 0 0 0 17 40 1 4.78e-03 0 0.00e+00 100<br>MatAssemblyBegin 1 1.0 1.9960e-06 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 0 0 0.00e+00 0 0.00e+00 0<br>MatAssemblyEnd 1 1.0 2.2575e-05 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 0 0 0.00e+00 0 0.00e+00 0<br>MatCUSPARSCopyTo 1 1.0 1.9215e-04 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 0 1 4.78e-03 0 0.00e+00 0<br>VecTDot 26 1.0 7.3121e-04 1.0 4.19e+03 1.0 0.0e+00 0.0e+00 0.0e+00 0 18 0 0 0 0 18 0 0 0 6 9 0 0.00e+00 0 0.00e+00 100<br>VecNorm 15 1.0 8.1064e-04 1.0 2.42e+03 1.0 0.0e+00 0.0e+00 0.0e+00 0 10 0 0 0 0 10 0 0 0 3 4 0 0.00e+00 0 0.00e+00 100</font><br></div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div lang="EN-GB" style="overflow-wrap: break-word;"><div class="gmail-m_9076948005455290591WordSection1"><p class="gmail-m_9076948005455290591p1"><span class="gmail-m_9076948005455290591s1"><span style="font-size:10pt"><u></u><u></u></span></span></p>
<p class="gmail-m_9076948005455290591p1"><span style="font-size:10pt"><u></u> <u></u></span></p>
<p class="MsoNormal"><span>but the log file didn’t record any gpu flops.
<u></u><u></u></span></p>
<p class="MsoNormal"><span><u></u> <u></u></span></p>
<p class="MsoNormal"><span>Sorry, my next question doesn’t belong to this thread.
<u></u><u></u></span></p>
<p class="MsoNormal"><span>Does </span>DMDA only work on structured grid/mesh and not on unstructured grid/mesh?<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">Best,<u></u><u></u></p>
<p class="MsoNormal">Karthik.<span><u></u><u></u></span></p>
<p class="MsoNormal"><span><u></u> <u></u></span></p>
<div style="border-right:none;border-bottom:none;border-left:none;border-top:1pt solid rgb(181,196,223);padding:3pt 0cm 0cm">
<p class="MsoNormal"><b><span style="font-size:12pt;color:black">From: </span></b><span style="font-size:12pt;color:black">Junchao Zhang <<a href="mailto:junchao.zhang@gmail.com" target="_blank">junchao.zhang@gmail.com</a>><br>
<b>Date: </b>Wednesday, 27 October 2021 at 21:13<br>
<b>To: </b>"Chockalingam, Karthikeyan (STFC,DL,HC)" <<a href="mailto:karthikeyan.chockalingam@stfc.ac.uk" target="_blank">karthikeyan.chockalingam@stfc.ac.uk</a>><br>
<b>Cc: </b>"<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a>" <<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a>><br>
<b>Subject: </b>Re: [petsc-users] Cuda: Vec and Mat types<u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<div>
<p class="MsoNormal" style="margin-bottom:12pt"><u></u> <u></u></p>
</div>
<p class="MsoNormal"><u></u> <u></u></p>
<div>
<div>
<p class="MsoNormal">On Wed, Oct 27, 2021 at 2:24 PM Karthikeyan Chockalingam - STFC UKRI <<a href="mailto:karthikeyan.chockalingam@stfc.ac.uk" target="_blank">karthikeyan.chockalingam@stfc.ac.uk</a>> wrote:<u></u><u></u></p>
</div>
<blockquote style="border-top:none;border-right:none;border-bottom:none;border-left:1pt solid rgb(204,204,204);padding:0cm 0cm 0cm 6pt;margin-left:4.8pt;margin-right:0cm">
<div>
<div>
<p class="MsoNormal">Hello,<u></u><u></u></p>
<p class="MsoNormal"> <u></u><u></u></p>
<p class="MsoNormal">I hope, I am framing the question currently.
<u></u><u></u></p>
<p class="MsoNormal">Are only distributed arrays (DMDA) of -vec_type and -mat_type only supported by CUDA?<u></u><u></u></p>
</div>
</div>
</blockquote>
<p class="MsoNormal">I don't understand this question. Currently, CUDA-capable types include VECCUDA, MATAIJCUDA and MATDENSECUDA, either sequential or MPI. <u></u><u></u></p>
<blockquote style="border-top:none;border-right:none;border-bottom:none;border-left:1pt solid rgb(204,204,204);padding:0cm 0cm 0cm 6pt;margin-left:4.8pt;margin-right:0cm">
<div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
<p class="MsoNormal">I am reading the petsc user manual in section 2.4 distributed arrays are introduced but at the start of chapter two there are other vector and matrix types as well. I wonder if
these types (I don’t how they are referred by) are also CUDA supported?<u></u><u></u></p>
<p class="MsoNormal"> <u></u><u></u></p>
<p class="MsoNormal">Can you please point me to some tutorial examples in KSP and SNES that can run on gpus?<u></u><u></u></p>
</div>
</div>
</blockquote>
<div>
<p class="MsoNormal">search "-mat_type aijcusparse" or "-dm_mat_type aijcusparse" in petsc tests/tutorials, you will find many.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<blockquote style="border-top:none;border-right:none;border-bottom:none;border-left:1pt solid rgb(204,204,204);padding:0cm 0cm 0cm 6pt;margin-left:4.8pt;margin-right:0cm">
<div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
<p class="MsoNormal">At the moment I am testing KSP/ex45.c with different preconditioners on cpus and gpus.<u></u><u></u></p>
<p class="MsoNormal"> <u></u><u></u></p>
<p class="MsoNormal">I tried to run KSP/ex2.c with -vec_type cuda and -mat_type aijcuda noticed there was no gpu flops recorded in my log file.<u></u><u></u></p>
</div>
</div>
</blockquote>
<div>
<p class="MsoNormal">It is -mat_type aijcusparse<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<blockquote style="border-top:none;border-right:none;border-bottom:none;border-left:1pt solid rgb(204,204,204);padding:0cm 0cm 0cm 6pt;margin-left:4.8pt;margin-right:0cm">
<div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
<p class="MsoNormal">Many thanks,<u></u><u></u></p>
<p class="MsoNormal">Karthik.<u></u><u></u></p>
</div>
<p><span style="font-size:6pt">This email and any attachments are intended solely for the use of the named recipients. If you are not the intended recipient you must not use, disclose, copy or distribute this email or any of its attachments and should notify
the sender immediately and delete this email from your system. UK Research and Innovation (UKRI) has taken every reasonable precaution to minimise risk of this email or any attachments containing viruses or malware but the recipient should carry out its own
virus and malware checks before opening the attachments. UKRI does not accept any liability for any losses or damages which the recipient may sustain due to presence of any viruses. </span><u></u><u></u></p>
</div>
</blockquote>
</div>
</div>
</div>
</div>
</blockquote></div></div>