<div dir="ltr">Hi, Maruthi, <div> I could not reproduce it. I used the attached slightly modified code (note the added PetscFunctionBeginUser; PetscFunctionReturn(PETSC_SUCCESS); in bc_const_temp_both_sides)<div> Could you try it?<br><div><div><div dir="ltr" class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr">--Junchao Zhang</div></div></div><br></div></div></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, Sep 13, 2023 at 1:07 AM Maruthi NH <<a href="mailto:maruthinh@gmail.com">maruthinh@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>Hi Junchao Zhang,</div><div><br></div><div>I also build petsc with --download-hypre and I too have 2.29.0 version. Not sure why I am facing this issue. This is the configuration file I created for building petsc. <br></div><div><br></div><div>#!/usr/bin/env python3<br><br>import os<br>petsc_hash_pkgs=os.path.join(os.getenv('HOME'),'petsc-hash-pkgs')<br><br>if __name__ == '__main__':<br> import sys<br> import os<br> sys.path.insert(0, os.path.abspath('config'))<br> import configure<br> configure_options = [<br> '--package-prefix-hash='+petsc_hash_pkgs,<br> '--with-debugging=0',<br> '--with-shared-libraries=0',<br> '--with-cc=mpiicc',<br> '--with-cxx=mpiicpc',<br> '--with-fc=mpiifort',<br> '--with-mpiexec=mpiexec.hydra',<br> '--with-cudac=nvcc', # nvcc-12.2 via ccache<br> '--with-cuda-dir=/usr/local/cuda-12.2',<br> # Intel compilers enable GCC/clangs equivalent of -ffast-math *by default*. This is<br> # bananas, so we make sure they use the same model as everyone else<br> 'COPTFLAGS=-O3 -fPIE -fp-model=precise',<br> 'FOPTFLAGS=-O3 -fPIE -fp-model=precise',<br> 'CXXOPTFLAGS=-O3 -fPIE -fp-model=precise',<br> 'CUDAOPTFLAGS=-O3',<br> '--with-precision=double',<br> '--with-blaslapack-dir='+os.environ['MKLROOT'],<br> '--with-mkl_pardiso-dir='+os.environ['MKLROOT'],<br> '--with-mkl_cpardiso-dir='+os.environ['MKLROOT'],<br> '--download-hypre',<br> ]<br> configure.petsc_configure(configure_options)</div><div><br></div><div>Regards,</div><div>Maruthi<br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, Sep 12, 2023 at 10:20 PM Junchao Zhang <<a href="mailto:junchao.zhang@gmail.com" target="_blank">junchao.zhang@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>Which version of hypre do you use? I used petsc's --download-hypre, which automatically installed hypre-2.29.0. I did not see the error.</div><div><br></div><div>$ ./heat_diff_cu -ksp_type gmres -pc_type hypre -pc_hypre_type boomeramg -use_gpu_aware_mpi 0 -mat_type aijcusparse -vec_type cuda<br></div><div>The start and end indices of mat for each rank: 0 100<br>Total time taken for KSP solve: rank: 0 0.0112464<br></div><br clear="all"><div><div dir="ltr" class="gmail_signature"><div dir="ltr">--Junchao Zhang</div></div></div><br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, Sep 12, 2023 at 10:15 AM Maruthi NH <<a href="mailto:maruthinh@gmail.com" target="_blank">maruthinh@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>Hi Junchao Zhang,</div><div><br></div><div>Thanks for the help. Updating PETSc fixed the problem. However, if I use boomeramg from hypre as follows, I get a similar error. <br></div><div><br></div><div>./heat_diff_cu --ksp_type gmres -pc_type hypre -pc_hypre_type boomeramg -use_gpu_aware_mpi 0 -mat_type aijcusparse -vec_type cuda</div><div><br></div><div>** On entry to cusparseCreateCsr() parameter number 5 (csrRowOffsets) had an illegal value: NULL pointer CUSPARSE ERROR (code = 3, invalid value) at csr_matrix_cuda_utils.c:57<br></div><div><br></div><div>Regards,</div><div>Maruthi<br></div><div><br></div><div><br></div><div><br></div><div><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, Sep 12, 2023 at 1:09 AM Junchao Zhang <<a href="mailto:junchao.zhang@gmail.com" target="_blank">junchao.zhang@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>Hi, Maruthi,</div><div> I could run your example on my machine. BTW, I added these at the end of main() to free petsc objects.</div><div><div style="color:rgb(0,0,0);font-family:Menlo,Monaco,"Courier New",monospace;font-size:14px;line-height:21px;white-space:pre-wrap"><div> VecDestroy(&vout);</div><div> VecDestroy(&x);</div><div> VecDestroy(&b);</div><div> VecDestroy(&u);</div><div> MatDestroy(&A);</div><div> VecScatterDestroy(&ctx);</div><div> KSPDestroy(&ksp);</div></div></div><div><br></div><div>If you use cuda-12.2, maybe the problem is already fixed by MR <a href="https://gitlab.com/petsc/petsc/-/merge_requests/6828" target="_blank">https://gitlab.com/petsc/petsc/-/merge_requests/6828</a></div><div>You can use petsc/main branch to try. Note your petsc version is from Date: 2023-08-13</div><div><br></div><div>Thanks.</div><div><div><div dir="ltr" class="gmail_signature"><div dir="ltr">--Junchao Zhang</div></div></div><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, Sep 11, 2023 at 12:10 PM Maruthi NH <<a href="mailto:maruthinh@gmail.com" target="_blank">maruthinh@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>Hi Barry Smith, <br></div><div><br></div><div>Thanks for the quick response. <br></div><div><br></div><div>Here is the code I used to test PETSc on GPU. <br></div><div>This is the command I used to run</div><div>mpiexec.hydra -n 1 ./heat_diff_cu -Nx 10000000 -ksp_type gmres -mat_type aijcusparse -vec_type cuda -use_gpu_aware_mpi 0 -pc_type gamg -ksp_converged_reason</div><div><br></div><div>Regards,</div><div>Maruthi <br></div><div><br></div><div><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Sun, Sep 10, 2023 at 11:37 PM Barry Smith <<a href="mailto:bsmith@petsc.dev" target="_blank">bsmith@petsc.dev</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div><br><div><br><blockquote type="cite"><div>On Sep 10, 2023, at 5:54 AM, Maruthi NH <<a href="mailto:maruthinh@gmail.com" target="_blank">maruthinh@gmail.com</a>> wrote:</div><br><div><div dir="ltr"><div>Hi all,</div><div><br></div><div>I am trying to accelerate the linear solver with PETSc GPU backend. For testing I have a simple 1D heat diffusion solver, here are some observations. <br></div><div>1. If I use -pc_type gamg it throws the following error <br></div><div> ** On entry to cusparseCreateCsr() parameter number 5 (csrRowOffsets) had an illegal value: NULL pointer<br><br>[0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------<br>[0]PETSC ERROR: GPU error<br>[0]PETSC ERROR: cuSPARSE errorcode 3 (CUSPARSE_STATUS_INVALID_VALUE) : invalid value<br>[0]PETSC ERROR: See <a href="https://petsc.org/release/faq/" target="_blank">https://petsc.org/release/faq/</a> for trouble shooting.<br>[0]PETSC ERROR: Petsc Development GIT revision: v3.19.4-959-g92f1e92e88 GIT Date: 2023-08-13 19:43:04 +0000</div><div><br></div></div></div></blockquote> Can you share the code that triggers this?</div><div><br><blockquote type="cite"><div><div dir="ltr"><div>2. Default pc ilu takes about 1.2 seconds on a single CPU and it takes about 105.9 seconds on a GPU. Similar observations with pc_type asm </div><div>I have NVIDIA RTX A2000 8GB Laptop GPU <br></div></div></div></blockquote><div><br></div> This is expected. The triangular solves sequentialize on the GPU so naturally are extremely slow since they cannot take advantage of the massive parallelism of the GPU.</div><div><br><blockquote type="cite"><div><div dir="ltr"><div><br></div><div>3. What I could be missing? Also, are there any general guidelines for better GPU performance using PETSc? <br></div><div><br></div><div>Regards,</div><div>Maruthi<br></div></div>
</div></blockquote></div><br></div></blockquote></div>
</blockquote></div>
</blockquote></div>
</blockquote></div>
</blockquote></div>
</blockquote></div>