[petsc-users] Use of hypre in your application

[Mark Adams]

Moving this to the users list (We can not talk about Crusher on public
forums, but this is on Summit. I had to check this thread carefully!)

Treb is using hypre on Summit and getting this error:

CUSPARSE ERROR (code = 11, insufficient resources) at
csr_spgemm_device_cusparse.c:128

This is probably from Hypre's RAP.

He has contacted OLCF, which seems like the right place to go, but does
anyone have any ideas?

Treb: You might ask Hypre also. We do actually have a fair amount of
experience with hypre but hypre has more!

Thanks,
Mark





On Fri, Jan 28, 2022 at 1:31 PM David Trebotich <dptrebotich at lbl.gov> wrote:

> Thought I sent you this...will change the order of MatSetOption to see if
> that helps
>
> I ran it and get that error which I have already sent a ticket to OLCF:
> CUSPARSE ERROR (code = 11, insufficient resources) at
> csr_spgemm_device_cusparse.c:128
>
> Here's my petscrc
> #do not use -mat_view with hypre-cuda if running on gpu
> #-mat_view :A.m:ascii_matlab
> -help
> -proj_mac_pc_type hypre
> -proj_mac_pc_hypre_type boomeramg
> -proj_mac_pc_hypre_boomeramg_no_CF
> -proj_mac_pc_hypre_boomeramg_agg_nl 0
> -proj_mac_pc_hypre_boomeramg_coarsen_type PMIS
> -proj_mac_pc_hypre_boomeramg_interp_type ext+i
> -proj_mac_pc_hypre_boomeramg_print_statistics
> -proj_mac_pc_hypre_boomeramg_relax_type_all l1scaled-Jacobi
> -proj_mac_ksp_type gmres
> -proj_mac_ksp_max_it 50
> -proj_mac_ksp_rtol 1.e-12
> -proj_mac_ksp_atol 1.e-30
> -mat_type hypre
> -use_gpu_aware_mpi 0
> -log_view
> -history PETSc.history
> -visc_ksp_rtol 1.e-12
> -visc_pc_type jacobi
> -visc_ksp_type gmres
> -visc_ksp_max_it 50
> -diff_ksp_rtol 1.e-6
> -diff_pc_type jacobi
> -diff_ksp_max_it 50
> -proj_mac_ksp_converged_reason
> -visc_ksp_converged_reason
> -diff_ksp_converged_reason
> -proj_mac_ksp_norm_type unpreconditioned
> -diff_ksp_norm_type unpreconditioned
> -visc_ksp_norm_type unpreconditioned
>
> And here's my code:
>       ierr =
> MatSetSizes(m_mat,NN,NN,PETSC_DECIDE,PETSC_DECIDE);CHKERRQ(ierr);
>       ierr = MatSetBlockSize(m_mat,nc);CHKERRQ(ierr);
>       ierr = MatSetType(m_mat,MATAIJ);CHKERRQ(ierr);
>       // ierr =
> MatSetOption(m_mat,MAT_NEW_NONZERO_ALLOCATION_ERR,PETSC_FALSE)
> ;CHKERRQ(ierr);
>       ierr = MatSetFromOptions( m_mat ); CHKERRQ(ierr);
>       ierr = MatSeqAIJSetPreallocation(m_mat,nnzrow, d_nnz);CHKERRQ(ierr);
>       ierr = MatMPIAIJSetPreallocation(m_mat,nnzrow, d_nnz, nnzrow/2,
> o_nnz);CHKERRQ(ierr);
>       ierr =
> MatSetOption(m_mat,MAT_NEW_NONZERO_ALLOCATION_ERR,PETSC_FALSE)
> ;CHKERRQ(ierr);
>
> #if defined(PETSC_HAVE_HYPRE)
>   ierr = MatHYPRESetPreallocation(m_mat,nnzrow, d_nnz, nnzrow/2,
> o_nnz);CHKERRQ(ierr);
> #endif
>
> On Wed, Jan 26, 2022 at 6:18 PM Mark Adams <mfadams at lbl.gov> wrote:
>
>>
>>
>>
>> On Wed, Jan 26, 2022 at 7:43 PM David Trebotich <dptrebotich at lbl.gov>
>> wrote:
>>
>>> Can you confirm with me on the settings in .petscrc for Summit with
>>> -pc_type hypre?  We were using
>>> -mat_type aijcusparse
>>> which worked a few months ago and now is not working.
>>>
>>> I don't know the difference between cusparse, aijcusparse and hypre as
>>> -mat_type
>>>
>>>
>> cusparse is not a matrix type. The other two are and they both should
>> work.
>>
>> I tested the builds that I just sent in another email (eg,
>> PETSC_DIR=/gpfs/alpine/world-shared/geo127/petsc/arch-summit-dbg-gcc-cuda
>> PETSC_ARCH="")
>>
>> $ make PETSC_DIR=$PWD PETSC_ARCH=arch-summit-opt-gnu-hypre-cuda -f
>> gmakefile test search='ksp_ksp_tutorials-ex55_hypre_device'
>> Using MAKEFLAGS: -- search=ksp_ksp_tutorials-ex55_hypre_device
>> PETSC_ARCH=arch-summit-opt-gnu-hypre-cuda
>> PETSC_DIR=/gpfs/alpine/csc314/scratch/adams/petsc2
>>           CC arch-summit-opt-gnu-hypre-cuda/tests/ksp/ksp/tutorials/ex55.o
>>      CLINKER arch-summit-opt-gnu-hypre-cuda/tests/ksp/ksp/tutorials/ex55
>>         TEST
>> arch-summit-opt-gnu-hypre-cuda/tests/counts/ksp_ksp_tutorials-ex55_hypre_device.counts
>>  ok ksp_ksp_tutorials-ex55_hypre_device
>>  ok diff-ksp_ksp_tutorials-ex55_hypre_device
>>
>> So this work. In this file (attached) you will see an example of a
>> construction a matrix that we have gone over before:
>>
>>   /* create stiffness matrix */
>>   ierr = MatCreate(comm,&Amat);CHKERRQ(ierr);
>>   ierr = MatSetSizes(Amat,m,m,M,M);CHKERRQ(ierr);
>>   ierr = MatSetType(Amat,MATAIJ);CHKERRQ(ierr);
>>   ierr = MatSetOption(Amat,MAT_SPD,PETSC_TRUE);CHKERRQ(ierr);
>>   ierr = MatSetFromOptions(Amat);CHKERRQ(ierr);
>>   ierr = MatSetBlockSize(Amat,2);CHKERRQ(ierr);
>>   ierr = MatSeqAIJSetPreallocation(Amat,18,NULL);CHKERRQ(ierr);
>>   ierr = MatMPIAIJSetPreallocation(Amat,18,NULL,18,NULL);CHKERRQ(ierr);
>> #if defined(PETSC_HAVE_HYPRE)
>>   ierr = MatHYPRESetPreallocation(Amat,18,NULL,18,NULL);CHKERRQ(ierr);
>> #endif
>>
>> At the end of the file you will what is executed with this test for
>> "hypre_device":
>>
>>    # command line options match GPU defaults
>>    test:
>>       suffix: hypre_device
>>       nsize: 4
>>       requires: hypre !complex
>>       args: *-mat_type hypre* -ksp_view -ne 29 -alpha 1.e-3 -ksp_type cg *-pc_type
>> hypre **-pc_hypre_type boomeramg *-ksp_monitor_short *-pc_hypre_boomeramg_relax_type_all
>> l1scaled-Jacobi -pc_hypre_boomeramg_interp_type ext+i
>> -pc_hypre_boomeramg_coarsen_type PMIS -pc_hypre_boomeramg_no_CF*
>>
>> All you need is *-mat_type hypre and -pc_type hypre*. *You could also
>> add these hypre arguments.*
>>
>> If this is not working please send me a description of the problem, like
>> any error output on your screen and your petsc.history file.
>>
>>
>>
>
>
> --
> ----------------------
> David Trebotich
> Lawrence Berkeley National Laboratory
> Computational Research Division
> Applied Numerical Algorithms Group
> treb at lbl.gov
> (510) 486-5984 office
> (510) 384-6868 mobile
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20220128/aa939ae8/attachment.html>