<div dir="ltr">Thanks, it looks correct. I am getting memory leaks (appended)<div><br></div><div>And something horrible is going on with performance:</div><div><br></div><div>MatLUFactorNum 130 1.0 9.2220e+00 1.0 6.51e+08 1.0 0.0e+00 0.0e+00 0.0e+00 30 0 0 0 0 30 0 0 0 0 71 0 390 3.33e+02 0 0.00e+00 0<br></div><div><br></div><div>MatLUFactorNum 130 1.0 6.5177e-01 1.0 1.28e+09 1.0 0.0e+00 0.0e+00 0.0e+00 4 1 0 0 0 4 1 0 0 0 1966 0 0 0.00e+00 0 0.00e+00 0<br></div><div><br></div><div>This is not urgent, but I'd like to get a serial LU GPU solver at some point.</div><div><br></div><div>Thanks again,</div><div>Mark</div><div><br></div><div>Lots of these:</div><div>[ 0]32 bytes VecCUDAAllocateCheck() line 34 in /autofs/nccs-svm1_home1/adams/petsc/src/vec/vec/impls/seq/seqcuda/<a href="http://veccuda2.cu">veccuda2.cu</a><br>[ 0]32 bytes VecCUDAAllocateCheck() line 34 in /autofs/nccs-svm1_home1/adams/petsc/src/vec/vec/impls/seq/seqcuda/<a href="http://veccuda2.cu">veccuda2.cu</a><br>[ 0]32 bytes VecCUDAAllocateCheck() line 34 in /autofs/nccs-svm1_home1/adams/petsc/src/vec/vec/impls/seq/seqcuda/<a href="http://veccuda2.cu">veccuda2.cu</a><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, Apr 15, 2020 at 12:47 PM Stefano Zampini <<a href="mailto:stefano.zampini@gmail.com">stefano.zampini@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Mark<div><br></div><div>attached is the patch. I will open an MR in the next days if you confirm it is working for you</div><div>The issue is that CUSPARSE does not have a way to compute the triangular factors, so we demand the computation of the factors to PETSc (CPU). These factors are then copied to the GPU.</div><div>What was happening in the second step of SNES, was that the factors were never updated since the offloadmask was never updated.</div><div><br></div><div>The real issue is that the CUSPARSE support in PETSc is really in bad shape and mostly untested, with coding solutions that are probably outdated now.</div><div>I'll see what I can do to fix the class if I have time in the next weeks.</div><div><br></div><div>Stefano</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">Il giorno mer 15 apr 2020 alle ore 17:21 Mark Adams <<a href="mailto:mfadams@lbl.gov" target="_blank">mfadams@lbl.gov</a>> ha scritto:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div dir="ltr"><br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, Apr 15, 2020 at 8:24 AM Stefano Zampini <<a href="mailto:stefano.zampini@gmail.com" target="_blank">stefano.zampini@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div>Mark<div><br></div><div>I have fixed few things in the solver and it is tested with the current master.</div></div></blockquote><div><br></div><div>I rebased with master over the weekend ....</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div><div>Can you write a MWE to reproduce the issue? Which version of CUDA and CUSPARSE are you using?</div></div></blockquote><div><br></div><div>You can use mark/feature-xgc-interface-rebase branch and add '-mat_type seqaijcusparse -fp_pc_factor_mat_solver_type cusparse -mat_cusparse_storage_format ell -vec_type cuda' to dm/impls/plex/tutorials/ex10.c</div><div><br></div><div>The first stage, SNES solve, actually looks OK here. Maybe.</div><div><br></div><div>Thanks,</div><div><br></div><div>10:01 mark/feature-xgc-interface-rebase *= ~/petsc$ make -f gmakefile test search='dm_impls_plex_tutorials-ex10_0'<br>/usr/bin/python /ccs/home/adams/petsc/config/gmakegentest.py --petsc-dir=/ccs/home/adams/petsc --petsc-arch=arch-summit-opt64-gnu-cuda --testdir=./arch-summit-opt64-gnu-cuda/tests<br>Using MAKEFLAGS: search=dm_impls_plex_tutorials-ex10_0<br> CC arch-summit-opt64-gnu-cuda/tests/dm/impls/plex/tutorials/ex10.o<br> CLINKER arch-summit-opt64-gnu-cuda/tests/dm/impls/plex/tutorials/ex10<br> TEST arch-summit-opt64-gnu-cuda/tests/counts/dm_impls_plex_tutorials-ex10_0.counts<br> ok dm_impls_plex_tutorials-ex10_0<br>not ok diff-dm_impls_plex_tutorials-ex10_0 # Error code: 1<br># 14,16c14,16<br># < 0 SNES Function norm 6.184233768573e-04 <br># < 1 SNES Function norm 1.467479466750e-08 <br># < 2 SNES Function norm 7.863111141350e-12 <br># ---<br># > 0 SNES Function norm 6.184233768572e-04 <br># > 1 SNES Function norm 1.467479466739e-08 <br># > 2 SNES Function norm 7.863102870090e-12 <br># 18,31c18,256<br># < 0 SNES Function norm 6.182952107532e-04 <br># < 1 SNES Function norm 7.336382211149e-09 <br># < 2 SNES Function norm 1.566979901443e-11 <br># < Nonlinear fp_ solve converged due to CONVERGED_FNORM_RELATIVE iterations 2<br># < 0 SNES Function norm 6.183592738545e-04 <br># < 1 SNES Function norm 7.337681407420e-09 <br># < 2 SNES Function norm 1.408823933908e-11 <br># < Nonlinear fp_ solve converged due to CONVERGED_FNORM_RELATIVE iterations 2<br># < [0] TSAdaptChoose_Basic(): Estimated scaled local truncation error 0.0396569, accepting step of size 1e-06<br># < 1 TS dt 1.25e-06 time 1e-06<br># < 1) species-0: charge density= -1.6024814608984e+01 z-momentum= 2.0080682964364e-19 energy= 1.2018000284846e+05<br># < 1) species-1: charge density= 1.6021676653316e+01 z-momentum= 1.4964483981137e-17 energy= 1.2017223215083e+05<br># < 1) species-2: charge density= 2.8838441139703e-03 z-momentum= -1.1062018110807e-23 energy= 1.2019641370376e-03<br># < 1) Total: charge density= -2.5411155383649e-04, momentum= 1.5165279748763e-17, energy= 2.4035223620125e+05 (m_i[0]/m_e = 3670.94, 140 cells), 1 sub threads<br># ---<br><font color="#ff0000"># > 0 SNES Function norm 6.182952107531e-04 <br># > 1 SNES Function norm 6.181600164904e-04 <br># > 2 SNES Function norm 6.180249471739e-04 <br># > 3 SNES Function norm 6.178899987549e-04 </font><br></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div><div>I was planning to reorganize the factor code in AIJCUSPARSE in the next days.</div><div><br></div><div><div style="margin:0px;font-stretch:normal;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures"><font size="1">kl-18967:petsc zampins$ git grep "solver_type cusparse"</font></span></div><div style="margin:0px;font-stretch:normal;line-height:normal;font-family:Menlo"><font size="1"><span style="font-variant-ligatures:no-common-ligatures">src/ksp/ksp/examples/tests/ex43.c</span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(46,174,187)">:</span><span style="font-variant-ligatures:no-common-ligatures"> args: -f ${DATAFILESPATH}/matrices/cfd.2.10 -mat_type seqaijcusparse -pc_factor_mat_</span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(180,36,25)"><b>solver_type cusparse</b></span><span style="font-variant-ligatures:no-common-ligatures"> -mat_cusparse_storage_format ell -vec_type cuda -pc_type ilu</span></font></div><div style="margin:0px;font-stretch:normal;line-height:normal;font-family:Menlo"><font size="1"><span style="font-variant-ligatures:no-common-ligatures">src/ksp/ksp/examples/tests/ex43.c</span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(46,174,187)">:</span><span style="font-variant-ligatures:no-common-ligatures"> args: -f ${DATAFILESPATH}/matrices/shallow_water1 -mat_type seqaijcusparse -pc_factor_mat_</span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(180,36,25)"><b>solver_type cusparse</b></span><span style="font-variant-ligatures:no-common-ligatures"> -mat_cusparse_storage_format hyb -vec_type cuda -ksp_type cg -pc_type icc</span></font></div><div style="margin:0px;font-stretch:normal;line-height:normal;font-family:Menlo"><font size="1"><span style="font-variant-ligatures:no-common-ligatures">src/ksp/ksp/examples/tests/ex43.c</span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(46,174,187)">:</span><span style="font-variant-ligatures:no-common-ligatures"> args: -f ${DATAFILESPATH}/matrices/cfd.2.10 -mat_type seqaijcusparse -pc_factor_mat_</span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(180,36,25)"><b>solver_type cusparse</b></span><span style="font-variant-ligatures:no-common-ligatures"> -mat_cusparse_storage_format csr -vec_type cuda -ksp_type bicg -pc_type ilu</span></font></div><div style="margin:0px;font-stretch:normal;line-height:normal;font-family:Menlo"><font size="1"><span style="font-variant-ligatures:no-common-ligatures">src/ksp/ksp/examples/tests/ex43.c</span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(46,174,187)">:</span><span style="font-variant-ligatures:no-common-ligatures"> args: -f ${DATAFILESPATH}/matrices/cfd.2.10 -mat_type seqaijcusparse -pc_factor_mat_</span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(180,36,25)"><b>solver_type cusparse</b></span><span style="font-variant-ligatures:no-common-ligatures"> -mat_cusparse_storage_format csr -vec_type cuda -ksp_type bicg -pc_type ilu -pc_factor_mat_ordering_type nd</span></font></div><div style="margin:0px;font-stretch:normal;line-height:normal;font-family:Menlo"><font size="1"><span style="font-variant-ligatures:no-common-ligatures">src/ksp/ksp/examples/tutorials/ex46.c</span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(46,174,187)">:</span><span style="font-variant-ligatures:no-common-ligatures"> args: -dm_mat_type aijcusparse -dm_vec_type cuda -random_exact_sol -pc_type ilu -pc_factor_mat_</span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(180,36,25)"><b>solver_type cusparse</b></span></font></div><div style="margin:0px;font-stretch:normal;line-height:normal;font-family:Menlo"><font size="1"><span style="font-variant-ligatures:no-common-ligatures">src/ksp/ksp/examples/tutorials/ex59.c</span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(46,174,187)">:</span><span style="font-variant-ligatures:no-common-ligatures"> args: -subdomain_mat_type aijcusparse -physical_pc_bddc_dirichlet_pc_factor_mat_</span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(180,36,25)"><b>solver_type cusparse</b></span></font></div><div style="margin:0px;font-stretch:normal;line-height:normal;font-family:Menlo"><font size="1"><span style="font-variant-ligatures:no-common-ligatures">src/ksp/ksp/examples/tutorials/ex7.c</span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(46,174,187)">:</span><span style="font-variant-ligatures:no-common-ligatures"> args: -ksp_monitor_short -mat_type aijcusparse -sub_pc_factor_mat_</span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(180,36,25)"><b>solver_type cusparse</b></span><span style="font-variant-ligatures:no-common-ligatures"> -vec_type cuda -sub_ksp_type preonly -sub_pc_type ilu</span></font></div><div style="margin:0px;font-stretch:normal;line-height:normal;font-family:Menlo"><font size="1"><span style="font-variant-ligatures:no-common-ligatures">src/ksp/ksp/examples/tutorials/ex7.c</span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(46,174,187)">:</span><span style="font-variant-ligatures:no-common-ligatures"> args: -ksp_monitor_short -mat_type aijcusparse -sub_pc_factor_mat_</span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(180,36,25)"><b>solver_type cusparse</b></span><span style="font-variant-ligatures:no-common-ligatures"> -vec_type cuda -sub_ksp_type preonly -sub_pc_type ilu</span></font></div><div style="margin:0px;font-stretch:normal;line-height:normal;font-family:Menlo"><font size="1"><span style="font-variant-ligatures:no-common-ligatures">src/ksp/ksp/examples/tutorials/ex7.c</span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(46,174,187)">:</span><span style="font-variant-ligatures:no-common-ligatures"> args: -ksp_monitor_short -mat_type aijcusparse -sub_pc_factor_mat_</span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(180,36,25)"><b>solver_type cusparse</b></span><span style="font-variant-ligatures:no-common-ligatures"> -vec_type cuda</span></font></div><div style="margin:0px;font-stretch:normal;line-height:normal;font-family:Menlo"><font size="1"><span style="font-variant-ligatures:no-common-ligatures">src/ksp/ksp/examples/tutorials/ex7.c</span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(46,174,187)">:</span><span style="font-variant-ligatures:no-common-ligatures"> args: -ksp_monitor_short -mat_type aijcusparse -sub_pc_factor_mat_</span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(180,36,25)"><b>solver_type cusparse</b></span><span style="font-variant-ligatures:no-common-ligatures"> -vec_type cuda</span></font></div><div style="margin:0px;font-stretch:normal;line-height:normal;font-family:Menlo"><font size="1"><span style="font-variant-ligatures:no-common-ligatures">src/ksp/ksp/examples/tutorials/ex71.c</span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(46,174,187)">:</span><span style="font-variant-ligatures:no-common-ligatures"> args: -pde_type Poisson -cells 7,9,8 -dim 3 -ksp_view -pc_bddc_coarse_redundant_pc_type svd -ksp_error_if_not_converged -pc_bddc_dirichlet_pc_type cholesky -pc_bddc_dirichlet_pc_factor_mat_</span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(180,36,25)"><b>solver_type cusparse</b></span><span style="font-variant-ligatures:no-common-ligatures"> -pc_bddc_dirichlet_pc_factor_mat_ordering_type nd -pc_bddc_neumann_pc_type cholesky -pc_bddc_neumann_pc_factor_mat_</span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(180,36,25)"><b>solver_type cusparse</b></span><span style="font-variant-ligatures:no-common-ligatures"> -pc_bddc_neumann_pc_factor_mat_ordering_type nd -matis_localmat_type aijcusparse</span></font></div><div style="margin:0px;font-stretch:normal;line-height:normal;font-family:Menlo"><font size="1"><span style="font-variant-ligatures:no-common-ligatures">src/ksp/ksp/examples/tutorials/ex72.c</span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(46,174,187)">:</span><span style="font-variant-ligatures:no-common-ligatures"> args: -f0 ${DATAFILESPATH}/matrices/medium -ksp_monitor_short -ksp_view -mat_view ascii::ascii_info -mat_type aijcusparse -pc_factor_mat_</span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(180,36,25)"><b>solver_type cusparse</b></span><span style="font-variant-ligatures:no-common-ligatures"> -pc_type ilu -vec_type cuda</span></font></div><div style="margin:0px;font-stretch:normal;line-height:normal;font-family:Menlo"><font size="1"><span style="font-variant-ligatures:no-common-ligatures">src/snes/examples/tutorials/ex12.c</span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(46,174,187)">:</span><span style="font-variant-ligatures:no-common-ligatures"> args: -matis_localmat_type aijcusparse -pc_bddc_dirichlet_pc_factor_mat_</span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(180,36,25)"><b>solver_type cusparse</b></span><span style="font-variant-ligatures:no-common-ligatures"> -pc_bddc_neumann_pc_factor_mat_</span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(180,36,25)"><b>solver_type cusparse</b></span></font></div><div><br><blockquote type="cite"><div>On Apr 15, 2020, at 2:20 PM, Mark Adams <<a href="mailto:mfadams@lbl.gov" target="_blank">mfadams@lbl.gov</a>> wrote:</div><br><div><div dir="ltr">I tried using a serial direct solver in cusparse and got bad numerics:<div><br></div><div>-vector_type cuda -mat_type aijcusparse -pc_factor_mat_solver_type cusparse <br></div><div><br></div><div>Before I start debugging this I wanted to see if there are any known issues that I should be aware of.</div><div><br></div><div>Thanks,</div></div>
</div></blockquote></div><br></div></div></blockquote></div></div>
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr">Stefano</div>
</blockquote></div>