<div dir="ltr">(Junchao), Ruipeng said this was an OOM error and suggested trying the in-house with<div><br></div><div>SpGEMM: HYPRE_SetSpGemmUseCusparse(FALSE);<div><div><br></div><div>Should I clone a hypre argument to make a -pc_hypre_use_tpl_spgemm or -pc_hypre_use_cusparse_spgemm ?<br><div><br></div></div></div></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, Jan 28, 2022 at 1:42 PM Mark Adams <<a href="mailto:mfadams@lbl.gov" target="_blank">mfadams@lbl.gov</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Moving this to the users list (We can not talk about Crusher on public forums, but this is on Summit. I had to check this thread carefully!)<div><br></div><div>Treb is using hypre on Summit and getting this error:</div><div><br></div><div>CUSPARSE ERROR (code = 11, insufficient resources) at csr_spgemm_device_cusparse.c:128</div><div><br></div><div>This is probably from Hypre's RAP.</div><div><br></div><div>He has contacted OLCF, which seems like the right place to go, but does anyone have any ideas?</div><div><br></div><div>Treb: You might ask Hypre also. We do actually have a fair amount of experience with hypre but hypre has more!</div><div><br></div><div>Thanks,</div><div>Mark</div><div><br><div><br style="color:rgb(80,0,80)"></div><div><br></div><div><br></div></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, Jan 28, 2022 at 1:31 PM David Trebotich <<a href="mailto:dptrebotich@lbl.gov" target="_blank">dptrebotich@lbl.gov</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="auto"><div dir="ltr">Thought I sent you this...will change the order of MatSetOption to see if that helps</div><div dir="ltr"><br></div><div dir="ltr">I ran it and get that error which I have already sent a ticket to OLCF:<div>CUSPARSE ERROR (code = 11, insufficient resources) at csr_spgemm_device_cusparse.c:128<br></div><div><br></div><div>Here's my petscrc</div><div>#do not use -mat_view with hypre-cuda if running on gpu<br>#-mat_view :A.m:ascii_matlab<br>-help<br>-proj_mac_pc_type hypre<br>-proj_mac_pc_hypre_type boomeramg<br>-proj_mac_pc_hypre_boomeramg_no_CF<br>-proj_mac_pc_hypre_boomeramg_agg_nl 0<br>-proj_mac_pc_hypre_boomeramg_coarsen_type PMIS<br>-proj_mac_pc_hypre_boomeramg_interp_type ext+i<br>-proj_mac_pc_hypre_boomeramg_print_statistics<br>-proj_mac_pc_hypre_boomeramg_relax_type_all l1scaled-Jacobi<br>-proj_mac_ksp_type gmres<br>-proj_mac_ksp_max_it 50<br>-proj_mac_ksp_rtol 1.e-12<br>-proj_mac_ksp_atol 1.e-30<br>-mat_type hypre<br>-use_gpu_aware_mpi 0</div><div>-log_view<br>-history PETSc.history<br>-visc_ksp_rtol 1.e-12<br>-visc_pc_type jacobi<br>-visc_ksp_type gmres<br>-visc_ksp_max_it 50<br>-diff_ksp_rtol 1.e-6<br>-diff_pc_type jacobi<br></div><div>-diff_ksp_max_it 50<br>-proj_mac_ksp_converged_reason<br>-visc_ksp_converged_reason<br>-diff_ksp_converged_reason<br>-proj_mac_ksp_norm_type unpreconditioned<br>-diff_ksp_norm_type unpreconditioned<br>-visc_ksp_norm_type unpreconditioned<br></div><div><br></div><div>And here's my code:</div><div> ierr = MatSetSizes(m_mat,NN,NN,PETSC_DECIDE,PETSC_DECIDE);CHKERRQ(ierr);<br> ierr = MatSetBlockSize(m_mat,nc);CHKERRQ(ierr);<br> ierr = MatSetType(m_mat,MATAIJ);CHKERRQ(ierr);<br> // ierr = MatSetOption(m_mat,MAT_NEW_NONZERO_ALLOCATION_ERR,PETSC_FALSE) ;CHKERRQ(ierr);<br> ierr = MatSetFromOptions( m_mat ); CHKERRQ(ierr);<br> ierr = MatSeqAIJSetPreallocation(m_mat,nnzrow, d_nnz);CHKERRQ(ierr);<br> ierr = MatMPIAIJSetPreallocation(m_mat,nnzrow, d_nnz, nnzrow/2, o_nnz);CHKERRQ(ierr);<br> ierr = MatSetOption(m_mat,MAT_NEW_NONZERO_ALLOCATION_ERR,PETSC_FALSE) ;CHKERRQ(ierr);<br><br>#if defined(PETSC_HAVE_HYPRE)<br> ierr = MatHYPRESetPreallocation(m_mat,nnzrow, d_nnz, nnzrow/2, o_nnz);CHKERRQ(ierr);<br>#endif<br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, Jan 26, 2022 at 6:18 PM Mark Adams <<a href="mailto:mfadams@lbl.gov" rel="noreferrer" target="_blank">mfadams@lbl.gov</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div dir="ltr"><br></div><div dir="ltr"><div dir="ltr"><br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, Jan 26, 2022 at 7:43 PM David Trebotich <<a href="mailto:dptrebotich@lbl.gov" rel="noreferrer" target="_blank">dptrebotich@lbl.gov</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Can you confirm with me on the settings in .petscrc for Summit with -pc_type hypre? We were using<div>-mat_type aijcusparse</div><div>which worked a few months ago and now is not working.</div><div><br></div><div>I don't know the difference between cusparse, aijcusparse and hypre as -mat_type</div></div><div class="gmail_quote"><div dir="ltr" class="gmail_attr"><br></div></div></blockquote><div> </div><div>cusparse is not a matrix type. The other two are and they both should work.</div><div> </div><div>I tested the builds that I just sent in another email (eg, PETSC_DIR=/gpfs/alpine/world-shared/geo127/petsc/arch-summit-dbg-gcc-cuda PETSC_ARCH="")</div><div><br></div><div>$ make PETSC_DIR=$PWD PETSC_ARCH=arch-summit-opt-gnu-hypre-cuda -f gmakefile test search='ksp_ksp_tutorials-ex55_hypre_device'<br>Using MAKEFLAGS: -- search=ksp_ksp_tutorials-ex55_hypre_device PETSC_ARCH=arch-summit-opt-gnu-hypre-cuda PETSC_DIR=/gpfs/alpine/csc314/scratch/adams/petsc2<br> CC arch-summit-opt-gnu-hypre-cuda/tests/ksp/ksp/tutorials/ex55.o<br> CLINKER arch-summit-opt-gnu-hypre-cuda/tests/ksp/ksp/tutorials/ex55<br> TEST arch-summit-opt-gnu-hypre-cuda/tests/counts/ksp_ksp_tutorials-ex55_hypre_device.counts<br> ok ksp_ksp_tutorials-ex55_hypre_device<br> ok diff-ksp_ksp_tutorials-ex55_hypre_device<br></div><div><br></div><div>So this work. In this file (attached) you will see an example of a construction a matrix that we have gone over before:</div><div><br></div><div> /* create stiffness matrix */<br> ierr = MatCreate(comm,&Amat);CHKERRQ(ierr);<br> ierr = MatSetSizes(Amat,m,m,M,M);CHKERRQ(ierr);<br> ierr = MatSetType(Amat,MATAIJ);CHKERRQ(ierr);<br> ierr = MatSetOption(Amat,MAT_SPD,PETSC_TRUE);CHKERRQ(ierr);<br> ierr = MatSetFromOptions(Amat);CHKERRQ(ierr);<br> ierr = MatSetBlockSize(Amat,2);CHKERRQ(ierr);<br> ierr = MatSeqAIJSetPreallocation(Amat,18,NULL);CHKERRQ(ierr);<br> ierr = MatMPIAIJSetPreallocation(Amat,18,NULL,18,NULL);CHKERRQ(ierr);<br>#if defined(PETSC_HAVE_HYPRE)<br> ierr = MatHYPRESetPreallocation(Amat,18,NULL,18,NULL);CHKERRQ(ierr);<br>#endif<br></div><div><br></div><div>At the end of the file you will what is executed with this test for "hypre_device":</div><div><br></div><div> # command line options match GPU defaults<br> test:<br> suffix: hypre_device<br> nsize: 4<br> requires: hypre !complex<br> args: <b>-mat_type hypre</b> -ksp_view -ne 29 -alpha 1.e-3 -ksp_type cg <b>-pc_type hypre </b><i><font color="#9900ff">-pc_hypre_type boomeramg </font></i>-ksp_monitor_short <i><font color="#9900ff">-pc_hypre_boomeramg_relax_type_all l1scaled-Jacobi -pc_hypre_boomeramg_interp_type ext+i -pc_hypre_boomeramg_coarsen_type PMIS -pc_hypre_boomeramg_no_CF</font></i><br></div><div><br></div><div>All you need is <b>-mat_type hypre and -pc_type hypre</b>. <i><font color="#9900ff">You could also add these hypre arguments.</font></i></div><div><b><br></b></div><div>If this is not working please send me a description of the problem, like any error output on your screen and your petsc.history file.</div><div><br></div><div> </div></div></div>
</div>
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr">----------------------<br>David Trebotich<br>Lawrence Berkeley National Laboratory<br>
Computational Research Division<br>Applied Numerical Algorithms Group<br><a href="mailto:treb@lbl.gov" rel="noreferrer" target="_blank">treb@lbl.gov</a><br>(510) 486-5984 office<br>(510) 384-6868 mobile<br></div></div>
</blockquote></div>
</blockquote></div>