[petsc-users] Use of hypre in your application

Junchao Zhang junchao.zhang at gmail.com
Fri Jan 28 20:37:26 CST 2022 

On Fri, Jan 28, 2022 at 3:27 PM Mark Adams <mfadams at lbl.gov> wrote:

> (Junchao), Ruipeng said this was an OOM error and suggested trying the
> in-house with
>
> SpGEMM: HYPRE_SetSpGemmUseCusparse(FALSE);
>
> Should I clone a hypre argument to make a -pc_hypre_use_tpl_spgemm or
> -pc_hypre_use_cusparse_spgemm ?
>
Better use the same words as hypre, e.g., -pc_hypre_set_spgemm_use_cusparse


>
>
> On Fri, Jan 28, 2022 at 1:42 PM Mark Adams <mfadams at lbl.gov> wrote:
>
>> Moving this to the users list (We can not talk about Crusher on public
>> forums, but this is on Summit. I had to check this thread carefully!)
>>
>> Treb is using hypre on Summit and getting this error:
>>
>> CUSPARSE ERROR (code = 11, insufficient resources) at
>> csr_spgemm_device_cusparse.c:128
>>
>> This is probably from Hypre's RAP.
>>
>> He has contacted OLCF, which seems like the right place to go, but does
>> anyone have any ideas?
>>
>> Treb: You might ask Hypre also. We do actually have a fair amount of
>> experience with hypre but hypre has more!
>>
>> Thanks,
>> Mark
>>
>>
>>
>>
>>
>> On Fri, Jan 28, 2022 at 1:31 PM David Trebotich <dptrebotich at lbl.gov>
>> wrote:
>>
>>> Thought I sent you this...will change the order of MatSetOption to see
>>> if that helps
>>>
>>> I ran it and get that error which I have already sent a ticket to OLCF:
>>> CUSPARSE ERROR (code = 11, insufficient resources) at
>>> csr_spgemm_device_cusparse.c:128
>>>
>>> Here's my petscrc
>>> #do not use -mat_view with hypre-cuda if running on gpu
>>> #-mat_view :A.m:ascii_matlab
>>> -help
>>> -proj_mac_pc_type hypre
>>> -proj_mac_pc_hypre_type boomeramg
>>> -proj_mac_pc_hypre_boomeramg_no_CF
>>> -proj_mac_pc_hypre_boomeramg_agg_nl 0
>>> -proj_mac_pc_hypre_boomeramg_coarsen_type PMIS
>>> -proj_mac_pc_hypre_boomeramg_interp_type ext+i
>>> -proj_mac_pc_hypre_boomeramg_print_statistics
>>> -proj_mac_pc_hypre_boomeramg_relax_type_all l1scaled-Jacobi
>>> -proj_mac_ksp_type gmres
>>> -proj_mac_ksp_max_it 50
>>> -proj_mac_ksp_rtol 1.e-12
>>> -proj_mac_ksp_atol 1.e-30
>>> -mat_type hypre
>>> -use_gpu_aware_mpi 0
>>> -log_view
>>> -history PETSc.history
>>> -visc_ksp_rtol 1.e-12
>>> -visc_pc_type jacobi
>>> -visc_ksp_type gmres
>>> -visc_ksp_max_it 50
>>> -diff_ksp_rtol 1.e-6
>>> -diff_pc_type jacobi
>>> -diff_ksp_max_it 50
>>> -proj_mac_ksp_converged_reason
>>> -visc_ksp_converged_reason
>>> -diff_ksp_converged_reason
>>> -proj_mac_ksp_norm_type unpreconditioned
>>> -diff_ksp_norm_type unpreconditioned
>>> -visc_ksp_norm_type unpreconditioned
>>>
>>> And here's my code:
>>>       ierr =
>>> MatSetSizes(m_mat,NN,NN,PETSC_DECIDE,PETSC_DECIDE);CHKERRQ(ierr);
>>>       ierr = MatSetBlockSize(m_mat,nc);CHKERRQ(ierr);
>>>       ierr = MatSetType(m_mat,MATAIJ);CHKERRQ(ierr);
>>>       // ierr =
>>> MatSetOption(m_mat,MAT_NEW_NONZERO_ALLOCATION_ERR,PETSC_FALSE)
>>> ;CHKERRQ(ierr);
>>>       ierr = MatSetFromOptions( m_mat ); CHKERRQ(ierr);
>>>       ierr = MatSeqAIJSetPreallocation(m_mat,nnzrow,
>>> d_nnz);CHKERRQ(ierr);
>>>       ierr = MatMPIAIJSetPreallocation(m_mat,nnzrow, d_nnz, nnzrow/2,
>>> o_nnz);CHKERRQ(ierr);
>>>       ierr =
>>> MatSetOption(m_mat,MAT_NEW_NONZERO_ALLOCATION_ERR,PETSC_FALSE)
>>> ;CHKERRQ(ierr);
>>>
>>> #if defined(PETSC_HAVE_HYPRE)
>>>   ierr = MatHYPRESetPreallocation(m_mat,nnzrow, d_nnz, nnzrow/2,
>>> o_nnz);CHKERRQ(ierr);
>>> #endif
>>>
>>> On Wed, Jan 26, 2022 at 6:18 PM Mark Adams <mfadams at lbl.gov> wrote:
>>>
>>>>
>>>>
>>>>
>>>> On Wed, Jan 26, 2022 at 7:43 PM David Trebotich <dptrebotich at lbl.gov>
>>>> wrote:
>>>>
>>>>> Can you confirm with me on the settings in .petscrc for Summit with
>>>>> -pc_type hypre?  We were using
>>>>> -mat_type aijcusparse
>>>>> which worked a few months ago and now is not working.
>>>>>
>>>>> I don't know the difference between cusparse, aijcusparse and hypre as
>>>>> -mat_type
>>>>>
>>>>>
>>>> cusparse is not a matrix type. The other two are and they both should
>>>> work.
>>>>
>>>> I tested the builds that I just sent in another email (eg,
>>>> PETSC_DIR=/gpfs/alpine/world-shared/geo127/petsc/arch-summit-dbg-gcc-cuda
>>>> PETSC_ARCH="")
>>>>
>>>> $ make PETSC_DIR=$PWD PETSC_ARCH=arch-summit-opt-gnu-hypre-cuda -f
>>>> gmakefile test search='ksp_ksp_tutorials-ex55_hypre_device'
>>>> Using MAKEFLAGS: -- search=ksp_ksp_tutorials-ex55_hypre_device
>>>> PETSC_ARCH=arch-summit-opt-gnu-hypre-cuda
>>>> PETSC_DIR=/gpfs/alpine/csc314/scratch/adams/petsc2
>>>>           CC
>>>> arch-summit-opt-gnu-hypre-cuda/tests/ksp/ksp/tutorials/ex55.o
>>>>      CLINKER arch-summit-opt-gnu-hypre-cuda/tests/ksp/ksp/tutorials/ex55
>>>>         TEST
>>>> arch-summit-opt-gnu-hypre-cuda/tests/counts/ksp_ksp_tutorials-ex55_hypre_device.counts
>>>>  ok ksp_ksp_tutorials-ex55_hypre_device
>>>>  ok diff-ksp_ksp_tutorials-ex55_hypre_device
>>>>
>>>> So this work. In this file (attached) you will see an example of a
>>>> construction a matrix that we have gone over before:
>>>>
>>>>   /* create stiffness matrix */
>>>>   ierr = MatCreate(comm,&Amat);CHKERRQ(ierr);
>>>>   ierr = MatSetSizes(Amat,m,m,M,M);CHKERRQ(ierr);
>>>>   ierr = MatSetType(Amat,MATAIJ);CHKERRQ(ierr);
>>>>   ierr = MatSetOption(Amat,MAT_SPD,PETSC_TRUE);CHKERRQ(ierr);
>>>>   ierr = MatSetFromOptions(Amat);CHKERRQ(ierr);
>>>>   ierr = MatSetBlockSize(Amat,2);CHKERRQ(ierr);
>>>>   ierr = MatSeqAIJSetPreallocation(Amat,18,NULL);CHKERRQ(ierr);
>>>>   ierr = MatMPIAIJSetPreallocation(Amat,18,NULL,18,NULL);CHKERRQ(ierr);
>>>> #if defined(PETSC_HAVE_HYPRE)
>>>>   ierr = MatHYPRESetPreallocation(Amat,18,NULL,18,NULL);CHKERRQ(ierr);
>>>> #endif
>>>>
>>>> At the end of the file you will what is executed with this test for
>>>> "hypre_device":
>>>>
>>>>    # command line options match GPU defaults
>>>>    test:
>>>>       suffix: hypre_device
>>>>       nsize: 4
>>>>       requires: hypre !complex
>>>>       args: *-mat_type hypre* -ksp_view -ne 29 -alpha 1.e-3 -ksp_type
>>>> cg *-pc_type hypre **-pc_hypre_type boomeramg *-ksp_monitor_short *-pc_hypre_boomeramg_relax_type_all
>>>> l1scaled-Jacobi -pc_hypre_boomeramg_interp_type ext+i
>>>> -pc_hypre_boomeramg_coarsen_type PMIS -pc_hypre_boomeramg_no_CF*
>>>>
>>>> All you need is *-mat_type hypre and -pc_type hypre*. *You could also
>>>> add these hypre arguments.*
>>>>
>>>> If this is not working please send me a description of the problem,
>>>> like any error output on your screen and your petsc.history file.
>>>>
>>>>
>>>>
>>>
>>>
>>> --
>>> ----------------------
>>> David Trebotich
>>> Lawrence Berkeley National Laboratory
>>> Computational Research Division
>>> Applied Numerical Algorithms Group
>>> treb at lbl.gov
>>> (510) 486-5984 office
>>> (510) 384-6868 mobile
>>>
>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20220128/c4a522c1/attachment.html>

More information about the petsc-users mailing list