[petsc-users] Use of hypre in your application

Mark Adams mfadams at lbl.gov
Fri Jan 28 15:27:47 CST 2022 

(Junchao), Ruipeng said this was an OOM error and suggested trying the
in-house with

SpGEMM: HYPRE_SetSpGemmUseCusparse(FALSE);

Should I clone a hypre argument to make a -pc_hypre_use_tpl_spgemm or
-pc_hypre_use_cusparse_spgemm ?


On Fri, Jan 28, 2022 at 1:42 PM Mark Adams <mfadams at lbl.gov> wrote:

> Moving this to the users list (We can not talk about Crusher on public
> forums, but this is on Summit. I had to check this thread carefully!)
>
> Treb is using hypre on Summit and getting this error:
>
> CUSPARSE ERROR (code = 11, insufficient resources) at
> csr_spgemm_device_cusparse.c:128
>
> This is probably from Hypre's RAP.
>
> He has contacted OLCF, which seems like the right place to go, but does
> anyone have any ideas?
>
> Treb: You might ask Hypre also. We do actually have a fair amount of
> experience with hypre but hypre has more!
>
> Thanks,
> Mark
>
>
>
>
>
> On Fri, Jan 28, 2022 at 1:31 PM David Trebotich <dptrebotich at lbl.gov>
> wrote:
>
>> Thought I sent you this...will change the order of MatSetOption to see if
>> that helps
>>
>> I ran it and get that error which I have already sent a ticket to OLCF:
>> CUSPARSE ERROR (code = 11, insufficient resources) at
>> csr_spgemm_device_cusparse.c:128
>>
>> Here's my petscrc
>> #do not use -mat_view with hypre-cuda if running on gpu
>> #-mat_view :A.m:ascii_matlab
>> -help
>> -proj_mac_pc_type hypre
>> -proj_mac_pc_hypre_type boomeramg
>> -proj_mac_pc_hypre_boomeramg_no_CF
>> -proj_mac_pc_hypre_boomeramg_agg_nl 0
>> -proj_mac_pc_hypre_boomeramg_coarsen_type PMIS
>> -proj_mac_pc_hypre_boomeramg_interp_type ext+i
>> -proj_mac_pc_hypre_boomeramg_print_statistics
>> -proj_mac_pc_hypre_boomeramg_relax_type_all l1scaled-Jacobi
>> -proj_mac_ksp_type gmres
>> -proj_mac_ksp_max_it 50
>> -proj_mac_ksp_rtol 1.e-12
>> -proj_mac_ksp_atol 1.e-30
>> -mat_type hypre
>> -use_gpu_aware_mpi 0
>> -log_view
>> -history PETSc.history
>> -visc_ksp_rtol 1.e-12
>> -visc_pc_type jacobi
>> -visc_ksp_type gmres
>> -visc_ksp_max_it 50
>> -diff_ksp_rtol 1.e-6
>> -diff_pc_type jacobi
>> -diff_ksp_max_it 50
>> -proj_mac_ksp_converged_reason
>> -visc_ksp_converged_reason
>> -diff_ksp_converged_reason
>> -proj_mac_ksp_norm_type unpreconditioned
>> -diff_ksp_norm_type unpreconditioned
>> -visc_ksp_norm_type unpreconditioned
>>
>> And here's my code:
>>       ierr =
>> MatSetSizes(m_mat,NN,NN,PETSC_DECIDE,PETSC_DECIDE);CHKERRQ(ierr);
>>       ierr = MatSetBlockSize(m_mat,nc);CHKERRQ(ierr);
>>       ierr = MatSetType(m_mat,MATAIJ);CHKERRQ(ierr);
>>       // ierr =
>> MatSetOption(m_mat,MAT_NEW_NONZERO_ALLOCATION_ERR,PETSC_FALSE)
>> ;CHKERRQ(ierr);
>>       ierr = MatSetFromOptions( m_mat ); CHKERRQ(ierr);
>>       ierr = MatSeqAIJSetPreallocation(m_mat,nnzrow, d_nnz);CHKERRQ(ierr);
>>       ierr = MatMPIAIJSetPreallocation(m_mat,nnzrow, d_nnz, nnzrow/2,
>> o_nnz);CHKERRQ(ierr);
>>       ierr =
>> MatSetOption(m_mat,MAT_NEW_NONZERO_ALLOCATION_ERR,PETSC_FALSE)
>> ;CHKERRQ(ierr);
>>
>> #if defined(PETSC_HAVE_HYPRE)
>>   ierr = MatHYPRESetPreallocation(m_mat,nnzrow, d_nnz, nnzrow/2,
>> o_nnz);CHKERRQ(ierr);
>> #endif
>>
>> On Wed, Jan 26, 2022 at 6:18 PM Mark Adams <mfadams at lbl.gov> wrote:
>>
>>>
>>>
>>>
>>> On Wed, Jan 26, 2022 at 7:43 PM David Trebotich <dptrebotich at lbl.gov>
>>> wrote:
>>>
>>>> Can you confirm with me on the settings in .petscrc for Summit with
>>>> -pc_type hypre?  We were using
>>>> -mat_type aijcusparse
>>>> which worked a few months ago and now is not working.
>>>>
>>>> I don't know the difference between cusparse, aijcusparse and hypre as
>>>> -mat_type
>>>>
>>>>
>>> cusparse is not a matrix type. The other two are and they both should
>>> work.
>>>
>>> I tested the builds that I just sent in another email (eg,
>>> PETSC_DIR=/gpfs/alpine/world-shared/geo127/petsc/arch-summit-dbg-gcc-cuda
>>> PETSC_ARCH="")
>>>
>>> $ make PETSC_DIR=$PWD PETSC_ARCH=arch-summit-opt-gnu-hypre-cuda -f
>>> gmakefile test search='ksp_ksp_tutorials-ex55_hypre_device'
>>> Using MAKEFLAGS: -- search=ksp_ksp_tutorials-ex55_hypre_device
>>> PETSC_ARCH=arch-summit-opt-gnu-hypre-cuda
>>> PETSC_DIR=/gpfs/alpine/csc314/scratch/adams/petsc2
>>>           CC
>>> arch-summit-opt-gnu-hypre-cuda/tests/ksp/ksp/tutorials/ex55.o
>>>      CLINKER arch-summit-opt-gnu-hypre-cuda/tests/ksp/ksp/tutorials/ex55
>>>         TEST
>>> arch-summit-opt-gnu-hypre-cuda/tests/counts/ksp_ksp_tutorials-ex55_hypre_device.counts
>>>  ok ksp_ksp_tutorials-ex55_hypre_device
>>>  ok diff-ksp_ksp_tutorials-ex55_hypre_device
>>>
>>> So this work. In this file (attached) you will see an example of a
>>> construction a matrix that we have gone over before:
>>>
>>>   /* create stiffness matrix */
>>>   ierr = MatCreate(comm,&Amat);CHKERRQ(ierr);
>>>   ierr = MatSetSizes(Amat,m,m,M,M);CHKERRQ(ierr);
>>>   ierr = MatSetType(Amat,MATAIJ);CHKERRQ(ierr);
>>>   ierr = MatSetOption(Amat,MAT_SPD,PETSC_TRUE);CHKERRQ(ierr);
>>>   ierr = MatSetFromOptions(Amat);CHKERRQ(ierr);
>>>   ierr = MatSetBlockSize(Amat,2);CHKERRQ(ierr);
>>>   ierr = MatSeqAIJSetPreallocation(Amat,18,NULL);CHKERRQ(ierr);
>>>   ierr = MatMPIAIJSetPreallocation(Amat,18,NULL,18,NULL);CHKERRQ(ierr);
>>> #if defined(PETSC_HAVE_HYPRE)
>>>   ierr = MatHYPRESetPreallocation(Amat,18,NULL,18,NULL);CHKERRQ(ierr);
>>> #endif
>>>
>>> At the end of the file you will what is executed with this test for
>>> "hypre_device":
>>>
>>>    # command line options match GPU defaults
>>>    test:
>>>       suffix: hypre_device
>>>       nsize: 4
>>>       requires: hypre !complex
>>>       args: *-mat_type hypre* -ksp_view -ne 29 -alpha 1.e-3 -ksp_type
>>> cg *-pc_type hypre **-pc_hypre_type boomeramg *-ksp_monitor_short *-pc_hypre_boomeramg_relax_type_all
>>> l1scaled-Jacobi -pc_hypre_boomeramg_interp_type ext+i
>>> -pc_hypre_boomeramg_coarsen_type PMIS -pc_hypre_boomeramg_no_CF*
>>>
>>> All you need is *-mat_type hypre and -pc_type hypre*. *You could also
>>> add these hypre arguments.*
>>>
>>> If this is not working please send me a description of the problem, like
>>> any error output on your screen and your petsc.history file.
>>>
>>>
>>>
>>
>>
>> --
>> ----------------------
>> David Trebotich
>> Lawrence Berkeley National Laboratory
>> Computational Research Division
>> Applied Numerical Algorithms Group
>> treb at lbl.gov
>> (510) 486-5984 office
>> (510) 384-6868 mobile
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20220128/bdbe3948/attachment-0001.html>

More information about the petsc-users mailing list