<html><head><meta http-equiv="Content-Type" content="text/html; charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">MatShellSetMatProductOperation looks really nice to me, thanks!<div class="">Pierre<br class=""><div class=""><div><br class=""><blockquote type="cite" class=""><div class="">On 12 May 2020, at 12:13 PM, Stefano Zampini <<a href="mailto:stefano.zampini@gmail.com" class="">stefano.zampini@gmail.com</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><div dir="ltr" class="">Pierre and Jose<div class=""><br class=""></div><div class="">I have added support for MatMat callbacks operations for MATSHELL, you may want to take a look here for how to use it <a href="https://gitlab.com/petsc/petsc/-/merge_requests/2712/diffs?commit_id=7f809910e2bafe055242a87d70afd114664ffaf8" class="">https://gitlab.com/petsc/petsc/-/merge_requests/2712/diffs?commit_id=7f809910e2bafe055242a87d70afd114664ffaf8</a></div><div class="">This is the relevant commit <a href="https://gitlab.com/petsc/petsc/-/merge_requests/2712/diffs?commit_id=e01f573d2ca0ec07c54db508a94a042fba4038de" class="">https://gitlab.com/petsc/petsc/-/merge_requests/2712/diffs?commit_id=e01f573d2ca0ec07c54db508a94a042fba4038de</a></div><div class=""><br class=""></div><div class="">Let me know if you need more customization (e.g. attach data to the product in a more systematic way) or if it can already fit your frameworks.</div><div class=""><br class=""></div><div class="">Best</div><div class="">Stefano</div></div><br class=""><div class="gmail_quote"><div dir="ltr" class="gmail_attr">Il giorno dom 10 mag 2020 alle ore 21:04 Stefano Zampini <<a href="mailto:stefano.zampini@gmail.com" class="">stefano.zampini@gmail.com</a>> ha scritto:<br class=""></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div style="overflow-wrap: break-word;" class=""><br class=""><div class=""><br class=""><blockquote type="cite" class=""><div class="">On May 10, 2020, at 8:56 PM, Jose E. Roman <<a href="mailto:jroman@dsic.upv.es" target="_blank" class="">jroman@dsic.upv.es</a>> wrote:</div><br class=""><div class=""><div class=""><br class=""><br class=""><blockquote type="cite" class="">El 10 may 2020, a las 19:42, Stefano Zampini <<a href="mailto:stefano.zampini@gmail.com" target="_blank" class="">stefano.zampini@gmail.com</a>> escribió:<br class=""><br class="">Glad to hear it works. Anyway, without the MatShellVecSetType call the code was erroring for me, not leaking memory.<br class="">Where you also providing -vec_type cuda at command line or what? <br class="">Mark recently noted a similar leak, and I was wondering what was the cause for yours. A MWE would be great.<br class=""></blockquote><br class="">I never use -vec_type cuda because all SLEPc tests that use CUDA create vectors with MatCreateVecs(). Until now I had never tested examples with shell matrices.<br class=""><br class="">This kind of issues would be more easy to detect if macros such as PetscCheckSameTypeAndComm() would actually compare type_name.</div></div></blockquote><div class=""><br class=""></div>VECCUDA specific code checks for type names (either VECSEQCUDA or VECMPICUDA) and these errored in my case. (Try by simply removing MatShellSetVecType)</div><div class=""><br class=""></div><div class=""><div style="margin:0px;font-stretch:normal;line-height:normal;font-family:Menlo;color:rgb(64,11,217)" class=""><span style="font-variant-ligatures:no-common-ligatures;color:rgb(47,180,29)" class=""><b class="">zampins@jasmine</b></span><span style="font-variant-ligatures: no-common-ligatures;" class="">:</span><span style="font-variant-ligatures:no-common-ligatures" class=""><b class="">~/petsc/src/mat/tests</b></span><span style="font-variant-ligatures: no-common-ligatures;" class="">$ git diff</span></div><div style="margin:0px;font-stretch:normal;line-height:normal;font-family:Menlo" class=""><span style="font-variant-ligatures:no-common-ligatures" class=""><b class="">diff --git a/src/mat/tests/ex69.c b/src/mat/tests/ex69.c</b></span></div><div style="margin:0px;font-stretch:normal;line-height:normal;font-family:Menlo" class=""><span style="font-variant-ligatures:no-common-ligatures" class=""><b class="">index b04652d..2a9374d 100644</b></span></div><div style="margin:0px;font-stretch:normal;line-height:normal;font-family:Menlo" class=""><span style="font-variant-ligatures:no-common-ligatures" class=""><b class="">--- a/src/mat/tests/ex69.c</b></span></div><div style="margin:0px;font-stretch:normal;line-height:normal;font-family:Menlo" class=""><span style="font-variant-ligatures:no-common-ligatures" class=""><b class="">+++ b/src/mat/tests/ex69.c</b></span></div><div style="margin:0px;font-stretch:normal;line-height:normal;font-family:Menlo" class=""><span style="font-variant-ligatures:no-common-ligatures;color:rgb(46,174,187)" class="">@@ -89,7 +89,7 @@</span><span style="font-variant-ligatures:no-common-ligatures" class=""> int main(int argc,char **argv)</span></div><div style="margin:0px;font-stretch:normal;line-height:normal;font-family:Menlo" class=""><span style="font-variant-ligatures:no-common-ligatures" class=""> if (use_shell) {</span></div><div style="margin:0px;font-stretch:normal;line-height:normal;font-family:Menlo" class=""><span style="font-variant-ligatures:no-common-ligatures" class=""> ierr = MatCreateShell(PetscObjectComm((PetscObject)v),nloc,nloc,n,n,A,&S);CHKERRQ(ierr);</span></div><div style="margin:0px;font-stretch:normal;line-height:normal;font-family:Menlo" class=""><span style="font-variant-ligatures:no-common-ligatures" class=""> ierr = MatShellSetOperation(S,MATOP_MULT,(void(*)(void))MatMult_S);CHKERRQ(ierr);</span></div><div style="margin:0px;font-stretch:normal;line-height:normal;font-family:Menlo;color:rgb(180,36,25)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">- ierr = MatShellSetVecType(S,vtype);CHKERRQ(ierr);</span></div><div style="margin:0px;font-stretch:normal;line-height:normal;font-family:Menlo;color:rgb(47,180,29)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">+ //ierr = MatShellSetVecType(S,vtype);CHKERRQ(ierr);</span></div><div style="margin:0px;font-stretch:normal;line-height:normal;font-family:Menlo" class=""><span style="font-variant-ligatures:no-common-ligatures" class=""> /* we could have called the general convertor also */</span></div><div style="margin:0px;font-stretch:normal;line-height:normal;font-family:Menlo" class=""><span style="font-variant-ligatures:no-common-ligatures" class=""> /* ierr = MatConvert(A,MATSHELL,MAT_INITIAL_MATRIX,&S);CHKERRQ(ierr); */</span></div><div style="margin:0px;font-stretch:normal;line-height:normal;font-family:Menlo" class=""><span style="font-variant-ligatures:no-common-ligatures" class=""> } else {</span></div><div style="margin:0px;font-stretch:normal;line-height:normal;font-family:Menlo;color:rgb(64,11,217)" class=""><span style="font-variant-ligatures:no-common-ligatures;color:rgb(47,180,29)" class=""><b class="">zampins@jasmine</b></span><span style="font-variant-ligatures: no-common-ligatures;" class="">:</span><span style="font-variant-ligatures:no-common-ligatures" class=""><b class="">~/petsc/src/mat/tests</b></span><span style="font-variant-ligatures: no-common-ligatures;" class="">$ ./ex69 -use_shell</span></div><div style="margin:0px;font-stretch:normal;line-height:normal;font-family:Menlo;color:rgb(180,36,25)" class=""><span style="font-variant-ligatures:no-common-ligatures" class=""><b class="">[0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------</b></span></div><div style="margin:0px;font-stretch:normal;line-height:normal;font-family:Menlo" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">[0]PETSC ERROR: Invalid argument</span></div><div style="margin:0px;font-stretch:normal;line-height:normal;font-family:Menlo" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">[0]PETSC ERROR: Object (seq) is not seqcuda or mpicuda</span></div><div style="margin:0px;font-stretch:normal;line-height:normal;font-family:Menlo" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">[0]PETSC ERROR: See <a href="https://www.mcs.anl.gov/petsc/documentation/faq.html" target="_blank" class="">https://www.mcs.anl.gov/petsc/documentation/faq.html</a> for trouble shooting.</span></div><div style="margin:0px;font-stretch:normal;line-height:normal;font-family:Menlo" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">[0]PETSC ERROR: Petsc Development GIT revision: v3.13-213-g441560f GIT Date: 2020-04-25 17:01:13 +0300</span></div><div style="margin:0px;font-stretch:normal;line-height:normal;font-family:Menlo" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">[0]PETSC ERROR: ./ex69 on a arch-gpu-double-openmp-openblas named jasmine by zampins Sun May 10 21:03:09 2020</span></div><div style="margin:0px;font-stretch:normal;line-height:normal;font-family:Menlo" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">[0]PETSC ERROR: Configure options --download-cub=1 --download-hara-commit=HEAD --download-hara=1 --download-kblas=1 --download-magma=1 --download-openblas-use-pthreads=1 --download-openblas=1 --download-superlu_dist-commit=HEAD --download-superlu_dist=1 --download-mumps=1 --download-scalapack=1 --with-cc=/opt/ecrc/intel/2018/compilers_and_libraries/linux/mpi/intel64/bin/mpicc --with-cuda-gencodearch=37 --with-cuda=1 --with-cudac=/opt/ecrc/cuda/10.1/bin/nvcc --with-cxx-dialect=C++11 --with-cxx=/opt/ecrc/intel/2018/compilers_and_libraries/linux/mpi/intel64/bin/mpicxx --with-fc=/opt/ecrc/intel/2018/compilers_and_libraries/linux/mpi/intel64/bin/mpif90 --with-fortran-bindings=0 --with-magma-fortran-bindings=0 --with-opencl-include=/opt/ecrc/cuda/10.1/include --with-opencl-lib="-L/opt/ecrc/cuda/10.1/lib64 -lOpenCL" --with-openmp=1 --with-precision=double PETSC_ARCH=arch-gpu-double-openmp-openblas</span></div><div style="margin:0px;font-stretch:normal;line-height:normal;font-family:Menlo" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">[0]PETSC ERROR: #1 VecCUDAGetArrayRead() line 2206 in /home/zampins/petsc/src/vec/vec/interface/rvector.c</span></div><div style="margin:0px;font-stretch:normal;line-height:normal;font-family:Menlo" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">[0]PETSC ERROR: #2 MatMultAdd_SeqAIJCUSPARSE() line 1448 in /home/zampins/petsc/src/mat/impls/aij/seq/seqcusparse/<a href="http://aijcusparse.cu/" target="_blank" class="">aijcusparse.cu</a></span></div><div style="margin:0px;font-stretch:normal;line-height:normal;font-family:Menlo" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">[0]PETSC ERROR: #3 MatMult_SeqAIJCUSPARSE() line 1375 in /home/zampins/petsc/src/mat/impls/aij/seq/seqcusparse/<a href="http://aijcusparse.cu/" target="_blank" class="">aijcusparse.cu</a></span></div><div style="margin:0px;font-stretch:normal;line-height:normal;font-family:Menlo" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">[0]PETSC ERROR: #4 MatMult() line 2392 in /home/zampins/petsc/src/mat/interface/matrix.c</span></div><div style="margin:0px;font-stretch:normal;line-height:normal;font-family:Menlo" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">[0]PETSC ERROR: #5 MatMult_S() line 12 in ex69.c</span></div><div style="margin:0px;font-stretch:normal;line-height:normal;font-family:Menlo" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">[0]PETSC ERROR: #6 MatMult_Shell() line 586 in /home/zampins/petsc/src/mat/impls/shell/shell.c</span></div><div style="margin:0px;font-stretch:normal;line-height:normal;font-family:Menlo" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">[0]PETSC ERROR: #7 MatMult() line 2392 in /home/zampins/petsc/src/mat/interface/matrix.c</span></div><div style="margin:0px;font-stretch:normal;line-height:normal;font-family:Menlo" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">[0]PETSC ERROR: #8 MatProductNumeric_X_Dense() line 305 in /home/zampins/petsc/src/mat/interface/matproduct.c</span></div><div style="margin:0px;font-stretch:normal;line-height:normal;font-family:Menlo" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">[0]PETSC ERROR: #9 MatProductNumeric() line 976 in /home/zampins/petsc/src/mat/interface/matproduct.c</span></div><div style="margin:0px;font-stretch:normal;line-height:normal;font-family:Menlo" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">[0]PETSC ERROR: #10 main() line 109 in ex69.c</span></div><div style="margin:0px;font-stretch:normal;line-height:normal;font-family:Menlo" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">[0]PETSC ERROR: PETSc Option Table entries:</span></div><div style="margin:0px;font-stretch:normal;line-height:normal;font-family:Menlo" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">[0]PETSC ERROR: -use_gpu_aware_mpi 0</span></div><div style="margin:0px;font-stretch:normal;line-height:normal;font-family:Menlo" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">[0]PETSC ERROR: -use_shell</span></div><div style="margin:0px;font-stretch:normal;line-height:normal;font-family:Menlo;color:rgb(180,36,25)" class=""><span style="font-variant-ligatures:no-common-ligatures" class=""><b class="">[0]PETSC ERROR: ----------------End of Error Message -------send entire error message to <a href="mailto:petsc-maint@mcs.anl.gov" target="_blank" class="">petsc-maint@mcs.anl.gov</a>----------</b></span></div><div style="margin:0px;font-stretch:normal;line-height:normal;font-family:Menlo" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">application called MPI_Abort(MPI_COMM_SELF, 109062) - process 0</span></div><div style="margin:0px;font-stretch:normal;line-height:normal;font-family:Menlo" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">[unset]: write_line error; fd=-1 buf=:cmd=abort exitcode=109062</span></div><div style="margin:0px;font-stretch:normal;line-height:normal;font-family:Menlo" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">:</span></div><div style="margin:0px;font-stretch:normal;line-height:normal;font-family:Menlo" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">system msg for write_line failure : Bad file descriptor</span></div><div class=""><span style="font-variant-ligatures:no-common-ligatures" class=""><br class=""></span></div></div><div class=""><br class=""><blockquote type="cite" class=""><div class=""><div class=""> <br class=""><br class=""><blockquote type="cite" class=""><br class="">BTW, the branch also provide MatDenseReplaceArray() now.<br class=""></blockquote><br class="">Yes, I saw it. Thanks.<br class=""><br class=""><blockquote type="cite" class="">Last thing I want to do is to support the user to provide MatMat (AB and AtB at least) callbacks for MATSHELL <br class="">Can SLEPc benefit from such a feature ?<br class=""></blockquote><br class="">Some solvers yes.<br class=""><br class=""><blockquote type="cite" class=""><br class=""><blockquote type="cite" class="">On May 10, 2020, at 7:47 PM, Jose E. Roman <<a href="mailto:jroman@dsic.upv.es" target="_blank" class="">jroman@dsic.upv.es</a>> wrote:<br class=""><br class="">Thanks for the hints. I have modified my branch. I was missing the MatShellSetVecType() call. Now everything works fine and all tests are clean.<br class=""><br class="">Jose<br class=""><br class=""><br class=""><blockquote type="cite" class="">El 9 may 2020, a las 21:32, Stefano Zampini <<a href="mailto:stefano.zampini@gmail.com" target="_blank" class="">stefano.zampini@gmail.com</a>> escribió:<br class=""><br class="">Jose<br class=""><br class="">I have just pushed an updated example with the MatMat operation, and I do not see the memory leak. Can you check?<br class=""><br class="">zampins@jasmine:~/petsc$ make -f gmakefile.test test search='mat%' searchin='ex69' PETSC_OPTIONS='-malloc -malloc_dump -malloc_debug' <br class="">/usr/bin/python /home/zampins/petsc/config/gmakegentest.py --petsc-dir=/home/zampins/petsc --petsc-arch=arch-gpu-double-openmp-openblas --testdir=./arch-gpu-double-openmp-openblas/tests<br class="">Using MAKEFLAGS: -- PETSC_OPTIONS=-malloc -malloc_dump -malloc_debug searchin=ex69 search=mat%<br class=""> CC arch-gpu-double-openmp-openblas/tests/mat/tests/ex69.o<br class=""> CLINKER arch-gpu-double-openmp-openblas/tests/mat/tests/ex69<br class=""> TEST arch-gpu-double-openmp-openblas/tests/counts/mat_tests-ex69_1.counts<br class="">ok mat_tests-ex69_1+nsize-1test-0_l-0_use_shell-0<br class="">ok diff-mat_tests-ex69_1+nsize-1test-0_l-0_use_shell-0<br class="">ok mat_tests-ex69_1+nsize-1test-0_l-0_use_shell-1<br class="">ok diff-mat_tests-ex69_1+nsize-1test-0_l-0_use_shell-1<br class="">ok mat_tests-ex69_1+nsize-1test-0_l-5_use_shell-0<br class="">ok diff-mat_tests-ex69_1+nsize-1test-0_l-5_use_shell-0<br class="">ok mat_tests-ex69_1+nsize-1test-0_l-5_use_shell-1<br class="">ok diff-mat_tests-ex69_1+nsize-1test-0_l-5_use_shell-1<br class="">ok mat_tests-ex69_1+nsize-1test-1_l-0_use_shell-0<br class="">ok diff-mat_tests-ex69_1+nsize-1test-1_l-0_use_shell-0<br class="">ok mat_tests-ex69_1+nsize-1test-1_l-0_use_shell-1<br class="">ok diff-mat_tests-ex69_1+nsize-1test-1_l-0_use_shell-1<br class="">ok mat_tests-ex69_1+nsize-1test-1_l-5_use_shell-0<br class="">ok diff-mat_tests-ex69_1+nsize-1test-1_l-5_use_shell-0<br class="">ok mat_tests-ex69_1+nsize-1test-1_l-5_use_shell-1<br class="">ok diff-mat_tests-ex69_1+nsize-1test-1_l-5_use_shell-1<br class="">ok mat_tests-ex69_1+nsize-1test-2_l-0_use_shell-0<br class="">ok diff-mat_tests-ex69_1+nsize-1test-2_l-0_use_shell-0<br class="">ok mat_tests-ex69_1+nsize-1test-2_l-0_use_shell-1<br class="">ok diff-mat_tests-ex69_1+nsize-1test-2_l-0_use_shell-1<br class="">ok mat_tests-ex69_1+nsize-1test-2_l-5_use_shell-0<br class="">ok diff-mat_tests-ex69_1+nsize-1test-2_l-5_use_shell-0<br class="">ok mat_tests-ex69_1+nsize-1test-2_l-5_use_shell-1<br class="">ok diff-mat_tests-ex69_1+nsize-1test-2_l-5_use_shell-1<br class="">ok mat_tests-ex69_1+nsize-2test-0_l-0_use_shell-0<br class="">ok diff-mat_tests-ex69_1+nsize-2test-0_l-0_use_shell-0<br class="">ok mat_tests-ex69_1+nsize-2test-0_l-0_use_shell-1<br class="">ok diff-mat_tests-ex69_1+nsize-2test-0_l-0_use_shell-1<br class="">ok mat_tests-ex69_1+nsize-2test-0_l-5_use_shell-0<br class="">ok diff-mat_tests-ex69_1+nsize-2test-0_l-5_use_shell-0<br class="">ok mat_tests-ex69_1+nsize-2test-0_l-5_use_shell-1<br class="">ok diff-mat_tests-ex69_1+nsize-2test-0_l-5_use_shell-1<br class="">ok mat_tests-ex69_1+nsize-2test-1_l-0_use_shell-0<br class="">ok diff-mat_tests-ex69_1+nsize-2test-1_l-0_use_shell-0<br class="">ok mat_tests-ex69_1+nsize-2test-1_l-0_use_shell-1<br class="">ok diff-mat_tests-ex69_1+nsize-2test-1_l-0_use_shell-1<br class="">ok mat_tests-ex69_1+nsize-2test-1_l-5_use_shell-0<br class="">ok diff-mat_tests-ex69_1+nsize-2test-1_l-5_use_shell-0<br class="">ok mat_tests-ex69_1+nsize-2test-1_l-5_use_shell-1<br class="">ok diff-mat_tests-ex69_1+nsize-2test-1_l-5_use_shell-1<br class="">ok mat_tests-ex69_1+nsize-2test-2_l-0_use_shell-0<br class="">ok diff-mat_tests-ex69_1+nsize-2test-2_l-0_use_shell-0<br class="">ok mat_tests-ex69_1+nsize-2test-2_l-0_use_shell-1<br class="">ok diff-mat_tests-ex69_1+nsize-2test-2_l-0_use_shell-1<br class="">ok mat_tests-ex69_1+nsize-2test-2_l-5_use_shell-0<br class="">ok diff-mat_tests-ex69_1+nsize-2test-2_l-5_use_shell-0<br class="">ok mat_tests-ex69_1+nsize-2test-2_l-5_use_shell-1<br class="">ok diff-mat_tests-ex69_1+nsize-2test-2_l-5_use_shell-1<br class=""><br class=""># -------------<br class=""># Summary <br class=""># -------------<br class=""># success 48/48 tests (100.0%)<br class=""># failed 0/48 tests (0.0%)<br class=""># todo 0/48 tests (0.0%)<br class=""># skip 0/48 tests (0.0%)<br class="">#<br class=""># Wall clock time for tests: 58 sec<br class=""># Approximate CPU time (not incl. build time): 62.11 sec<br class="">#<br class=""># Timing summary (actual test time / total CPU time): <br class=""># mat_tests-ex69_1: 2.30 sec / 62.11 sec<br class=""><br class=""><blockquote type="cite" class="">On May 9, 2020, at 9:28 PM, Jose E. Roman <<a href="mailto:jroman@dsic.upv.es" target="_blank" class="">jroman@dsic.upv.es</a>> wrote:<br class=""><br class=""><br class=""><br class=""><blockquote type="cite" class="">El 9 may 2020, a las 20:00, Stefano Zampini <<a href="mailto:stefano.zampini@gmail.com" target="_blank" class="">stefano.zampini@gmail.com</a>> escribió:<br class=""><br class=""><br class=""><br class="">Il giorno sab 9 mag 2020 alle ore 19:43 Jose E. Roman <<a href="mailto:jroman@dsic.upv.es" target="_blank" class="">jroman@dsic.upv.es</a>> ha scritto:<br class=""><br class=""><br class=""><blockquote type="cite" class="">El 9 may 2020, a las 12:45, Stefano Zampini <<a href="mailto:stefano.zampini@gmail.com" target="_blank" class="">stefano.zampini@gmail.com</a>> escribió:<br class=""><br class="">Jose<br class=""><br class="">I have just pushed a test <a href="https://gitlab.com/petsc/petsc/-/blob/d64c2bc63c8d5d1a8c689f1abc762ae2722bba26/src/mat/tests/ex69.c" target="_blank" class="">https://gitlab.com/petsc/petsc/-/blob/d64c2bc63c8d5d1a8c689f1abc762ae2722bba26/src/mat/tests/ex69.c</a><br class="">See if it fits your framework, and feel free to modify the test to add more checks<br class=""></blockquote><br class="">Almost good. The following modification of the example fails with -test 1:<br class=""><br class=""><br class="">diff --git a/src/mat/tests/ex69.c b/src/mat/tests/ex69.c<br class="">index e562f1e2e3..2df2c89be1 100644<br class="">--- a/src/mat/tests/ex69.c<br class="">+++ b/src/mat/tests/ex69.c<br class="">@@ -84,6 +84,10 @@ int main(int argc,char **argv)<br class="">}<br class="">ierr = VecCUDARestoreArray(v,&vv);CHKERRQ(ierr);<br class=""><br class="">+ if (test==1) {<br class="">+ ierr = MatDenseCUDAGetArray(B,&aa);CHKERRQ(ierr);<br class="">+ if (aa) SETERRQ(PETSC_COMM_WORLD,PETSC_ERR_USER,"Expected a null pointer");<br class="">+ }<br class=""><br class="">/* free work space */<br class="">ierr = MatDestroy(&B);CHKERRQ(ierr);<br class=""><br class=""><br class=""><br class="">I would expect that after MatDenseCUDAResetArray() the pointer is NULL because it was set so in line 60. In the CPU counterpart it works as expected.<br class=""><br class="">Pushed a fix for this, thanks.<br class=""><br class="">Another comment is: in line 60 you have changed MatDenseCUDAPlaceArray() to MatDenseCUDAReplaceArray(). This is ok, but it is strange because MatDenseReplaceArray() does not exist. So the interface is different in GPU vs CPU, but I guess it is necessary here.<br class=""><br class="">I think we do not support calling PlaceArray twice anywhere PETSc. This is why I have added MatDenseCUDAReplaceArray(). If you need support for the CPU case too, I can add it.<br class=""></blockquote><br class="">Yes, please. It is better to have the same thing in both cases.<br class=""><br class="">I am attaching the modified example, now performs a mat-mat product. If I do A*B it works well, but if I replace A with a shell matrix I get a memory leak.<br class=""><br class="">[ 0]32 bytes VecCUDAAllocateCheck() line 34 in /home/users/proy/copa/jroman/soft/petsc/src/vec/vec/impls/seq/seqcuda/<a href="http://veccuda2.cu/" target="_blank" class="">veccuda2.cu</a><br class="">[ 0]32 bytes VecCUDAAllocateCheck() line 34 in /home/users/proy/copa/jroman/soft/petsc/src/vec/vec/impls/seq/seqcuda/<a href="http://veccuda2.cu/" target="_blank" class="">veccuda2.cu</a><br class=""><br class=""><br class=""><br class=""><blockquote type="cite" class=""><br class="">Thanks.<br class="">Jose<br class=""><br class=""><br class=""><blockquote type="cite" class=""><br class=""><br class="">Il giorno ven 8 mag 2020 alle ore 18:48 Jose E. Roman <<a href="mailto:jroman@dsic.upv.es" target="_blank" class="">jroman@dsic.upv.es</a>> ha scritto:<br class="">Attached. Run with -test 1 or -test 2<br class=""><br class=""><blockquote type="cite" class="">El 8 may 2020, a las 17:14, Stefano Zampini <<a href="mailto:stefano.zampini@gmail.com" target="_blank" class="">stefano.zampini@gmail.com</a>> escribió:<br class=""><br class="">Jose<br class=""><br class="">Just send me a MWE and I’ll fix the case for you<br class=""><br class="">Thanks<br class="">Stefano<br class=""></blockquote><br class=""><br class="">-- <br class="">Stefano<br class=""></blockquote><br class=""><br class=""><br class="">-- <br class="">Stefano<br class=""></blockquote><ex69.c><br class=""></blockquote><br class=""></blockquote><br class=""></blockquote><br class=""></blockquote><br class=""></div></div></blockquote></div><br class=""></div></blockquote></div><br clear="all" class=""><div class=""><br class=""></div>-- <br class=""><div dir="ltr" class="gmail_signature">Stefano</div>
</div></blockquote></div><br class=""></div></div></body></html>