<div dir="ltr"><div dir="ltr"><br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, Jun 9, 2021 at 1:40 AM Barry Smith <<a href="mailto:bsmith@petsc.dev">bsmith@petsc.dev</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div style="overflow-wrap: break-word;"><div><br></div> You need to set the type before calling the preallocation routines. Otherwise when you set to a new type all the previous information is essentially removed when the new type is created. </div></blockquote><div><br></div><div>I do <b> set the type b</b>efore calling the <i>preallocation routines</i></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div style="overflow-wrap: break-word;"><div><br></div><div> Barry</div><div><br></div><div>A slightly more clever MatCreate_MPIAIJCUSPARSE() could check if the matrix was already of the type MPIAIJ and reuse that information but currently it does not do that check.</div><div><br><div><br><blockquote type="cite"><div>On Jun 8, 2021, at 11:11 PM, Mark Adams <<a href="mailto:mfadams@lbl.gov" target="_blank">mfadams@lbl.gov</a>> wrote:</div><br><div><div dir="ltr">I have an old code that fails with -mat_type aijcusparse<div>This is the code that creates the matrix:<br></div><div><br></div><div> ierr = PetscPrintf(PETSC_COMM_SELF,"\nPetscSolve.H MatCreate adding csr \n");CHKERRQ(ierr);<br> ierr = MatCreate(wcomm,&m_mat);CHKERRQ(ierr);<br> ierr = MatSetOptionsPrefix(m_mat,"csr_");CHKERRQ(ierr); // prefix is new but does not matter<br> ierr = MatSetSizes(m_mat,NN,NN,PETSC_DECIDE,PETSC_DECIDE);CHKERRQ(ierr);<br> ierr = MatSetBlockSize(m_mat,nc);CHKERRQ(ierr);<br><b> ierr = MatSetType(m_mat,MATAIJ);CHKERRQ(ierr);<br></b><i> ierr = MatSeqAIJSetPreallocation(m_mat,nnzrow, d_nnz);CHKERRQ(ierr);<br> ierr = MatMPIAIJSetPreallocation(m_mat,nnzrow, d_nnz, nnzrow/2, o_nnz);CHKERRQ(ierr);</i><br> ierr = MatSetOption(m_mat,MAT_NEW_NONZERO_ALLOCATION_ERR,PETSC_FALSE) ;CHKERRQ(ierr);<br> ierr = MatSetFromOptions( m_mat ); CHKERRQ(ierr);</div><div><br></div><div><div>With -csr_mat_type aijcusprase I get this output.</div><div>Note, this apparent recursive structure in the call stack. I have no idea where that is coming from. I can't see that from the code.</div><div><br></div><div>Any ideas?</div><div><br></div><div>Thanks,</div><div>Mark</div><div></div></div><div><br></div><div>PetscSolve.H MatCreate .H adding csr<br>[0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------<br>[0]PETSC ERROR: Object is in wrong state<br>[0]PETSC ERROR: Must call MatXXXSetPreallocation(), MatSetUp() or the matrix has not yet been factored on argument 1 "mat" before MatSetValues()<br>[0]PETSC ERROR: See <a href="https://www.mcs.anl.gov/petsc/documentation/faq.html" target="_blank">https://www.mcs.anl.gov/petsc/documentation/faq.html</a> for trouble shooting.<br>[0]PETSC ERROR: Petsc Development GIT revision: v3.15.0-734-gc2c1a54 GIT Date: 2021-06-07 02:44:02 +0000<br>[0]PETSC ERROR: ../viscousDriver3d.Linux.64.mpiCC.mpifort.OPT.MPI.PETSC.ex on a arch-summit-dbg-gnu-cuda named a15n03 by adams Tue Jun 8 23:39:49 2021<br>[0]PETSC ERROR: Configure options --with-fc=mpifort --with-cc=mpicc --with-cxx=mpiCC --CFLAGS="-fPIC -g -mcpu=native -Wno-unused-but-set-variable -Wno-long-long -Wno-sign-compare -Wno-deprecated -Wno-unused-local-typedefs -Wno-variadic-macros -Wno-unknown-pragmas" --CXXFLAGS="-fPIC -g -std=c++11 -mcpu=native -Wno-unused-but-set-variable -Wno-long-long -Wno-sign-compare -Wno-deprecated -ftemplate-depth-99 -Wno-unused-local-typedefs -Wno-literal-suffix -Wno-invalid-offsetof -Wno-variadic-macros -Wno-unknown-pragmas" --CUDAFLAGS="-g -Xcompiler -lineinfo" --FCFLAGS="-fPIC -g" --FCOPTFLAGS=-O0 --COPTFLAGS=-O0 --CXXOPTFLAGS=-O0 --with-ssl=0 --with-batch=0 --with-mpiexec="jsrun -g1" --with-cuda=1 --with-cudac=nvcc --with-cuda-gencodearch=70 --download-metis --download-parmetis --with-blaslapack-lib="-L/autofs/nccs-svm1_sw/summit/.swci/1-compute/opt/spack/20180914/linux-rhel7-ppc64le/gcc-6.4.0/netlib-lapack-3.8.0-wcabdyqhdi5rooxbkqa6x5d7hxyxwdkm/lib64 -lblas -llapack" --with-x=0 --with-64-bit-indices=0 --with-debugging=1 PETSC_ARCH=arch-summit-dbg-gnu-cuda --force<br>[0]PETSC ERROR: #1 MatSetValues() at /gpfs/alpine/csc314/scratch/adams/petsc/src/mat/interface/matrix.c:1366<br>[0]PETSC ERROR: #2 formMatrix() at ../srcLib/EBPetscLinearSolverPoisson.cpp:126<br>[0]PETSC ERROR: #3 setup_solver() at ../../../Chombo/lib/src/AMRElliptic/PetscSolverI.H:283<br>[0]PETSC ERROR: #4 solve_private() at ../../../Chombo/lib/src/AMRElliptic/PetscSolverI.H:427<br>[0]PETSC ERROR: #5 KSPSolve() at /gpfs/alpine/csc314/scratch/adams/petsc/src/ksp/ksp/interface/itfunc.c:1080<br>[0]PETSC ERROR: #6 solve_private() at ../../../Chombo/lib/src/AMRElliptic/PetscSolverI.H:508<br>[0]PETSC ERROR: #7 KSPSolve() at /gpfs/alpine/csc314/scratch/adams/petsc/src/ksp/ksp/interface/itfunc.c:1080<br>[0]PETSC ERROR: #8 solve_private() at ../../../Chombo/lib/src/AMRElliptic/PetscSolverI.H:508<br>[0]PETSC ERROR: #9 KSPSolve() at /gpfs/alpine/csc314/scratch/adams/petsc/src/ksp/ksp/interface/itfunc.c:1080<br>[0]PETSC ERROR: #10 solve_private() at ../../../Chombo/lib/src/AMRElliptic/PetscSolverI.H:508<br>[0]PETSC ERROR: #11 KSPSolve() at /gpfs/alpine/csc314/scratch/adams/petsc/src/ksp/ksp/interface/itfunc.c:1080<br>[0]PETSC ERROR: #12 solve_private() at ../../../Chombo/lib/src/AMRElliptic/PetscSolverI.H:508<br> <br></div><div><br></div></div>
</div></blockquote></div><br></div></div></blockquote></div></div>