<div dir="ltr">Ok,<div><br></div><div>I removed the MatSetType call and added DMSetMatrixPreallocateOnly before creating the matrix and this worked out the error, but the same situation persist, when i try to run with more than one processor it says:</div><div><br></div><div><div>[0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------</div><div>[0]PETSC ERROR: No support for this operation for this object type</div><div>[0]PETSC ERROR: Currently only handles ViennaCL matrices</div><div>[0]PETSC ERROR: See <a href="http://www.mcs.anl.gov/petsc/documentation/faq.html">http://www.mcs.anl.gov/petsc/documentation/faq.html</a> for trouble shooting.</div><div>[0]PETSC ERROR: Petsc Development GIT revision: v3.9.2-549-g779ab53 GIT Date: 2018-05-31 17:31:13 +0300</div><div>[0]PETSC ERROR: ./gcmLEP.GPU on a cuda-debug named node50 by valera Wed Aug 15 14:09:31 2018</div><div>[0]PETSC ERROR: Configure options PETSC_ARCH=cuda-debug --with-mpi-dir=/usr/lib64/openmpi --COPTFLAGS=-O2 --CXXOPTFLAGS=-O2 --FOPTFLAGS=-O2 --with-shared-libraries=1 --with-debugging=1 --with-cuda=1 --CUDAFLAGS=-arch=sm_60 --with-blaslapack-dir=/usr/lib64 --download-viennacl</div><div>[0]PETSC ERROR: #1 PCSetUp_SAVIENNACL() line 47 in /home/valera/petsc/src/ksp/pc/impls/saviennaclcuda/<a href="http://saviennacl.cu">saviennacl.cu</a></div><div>[0]PETSC ERROR: #2 PCSetUp() line 932 in /home/valera/petsc/src/ksp/pc/interface/precon.c</div><div>[0]PETSC ERROR: #3 KSPSetUp() line 381 in /home/valera/petsc/src/ksp/ksp/interface/itfunc.c</div><div>[1]PETSC ERROR: ------------------------------------------------------------------------</div><div>[1]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range</div><div>[1]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger</div><div>[1]PETSC ERROR: or see <a href="http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind">http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind</a></div><div>[1]PETSC ERROR: or try <a href="http://valgrind.org">http://valgrind.org</a> on GNU/linux and Apple Mac OS X to find memory corruption errors</div><div>[1]PETSC ERROR: likely location of problem given in stack below</div><div>[1]PETSC ERROR: --------------------- Stack Frames ------------------------------------</div><div>[1]PETSC ERROR: Note: The EXACT line numbers in the stack are not available,</div><div>[1]PETSC ERROR: INSTEAD the line number of the start of the function</div><div>[1]PETSC ERROR: is given.</div><div>[1]PETSC ERROR: [1] PetscTraceBackErrorHandler line 182 /home/valera/petsc/src/sys/error/errtrace.c</div><div>[1]PETSC ERROR: [1] PetscError line 352 /home/valera/petsc/src/sys/error/err.c</div><div>[1]PETSC ERROR: [1] PCSetUp_SAVIENNACL line 45 /home/valera/petsc/src/ksp/pc/impls/saviennaclcuda/<a href="http://saviennacl.cu">saviennacl.cu</a></div><div>[1]PETSC ERROR: [1] PCSetUp line 894 /home/valera/petsc/src/ksp/pc/interface/precon.c</div><div>[1]PETSC ERROR: [1] KSPSetUp line 294 /home/valera/petsc/src/ksp/ksp/interface/itfunc.c</div><div>[1]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------</div><div>[1]PETSC ERROR: Signal received</div><div>[1]PETSC ERROR: See <a href="http://www.mcs.anl.gov/petsc/documentation/faq.html">http://www.mcs.anl.gov/petsc/documentation/faq.html</a> for trouble shooting.</div><div>[1]PETSC ERROR: Petsc Development GIT revision: v3.9.2-549-g779ab53 GIT Date: 2018-05-31 17:31:13 +0300</div><div>[1]PETSC ERROR: ./gcmLEP.GPU on a cuda-debug named node50 by valera Wed Aug 15 14:09:31 2018</div><div>[1]PETSC ERROR: Configure options PETSC_ARCH=cuda-debug --with-mpi-dir=/usr/lib64/openmpi --COPTFLAGS=-O2 --CXXOPTFLAGS=-O2 --FOPTFLAGS=-O2 --with-shared-libraries=1 --with-debugging=1 --with-cuda=1 --CUDAFLAGS=-arch=sm_60 --with-blaslapack-dir=/usr/lib64 --download-viennacl</div></div><div><br></div><div>Thanks,</div><div><br></div><div><br></div><div><br></div><div><br></div><div><br></div><div><br></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Wed, Aug 15, 2018 at 1:53 PM, Manuel Valera <span dir="ltr"><<a href="mailto:mvalera-w@sdsu.edu" target="_blank">mvalera-w@sdsu.edu</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Thanks Matthew, <div><br></div><div>I try to do that when calling:</div><div><br></div><div><span style="font-size:12.8px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">call MatMPIAIJSetPreallocation(A,</span><span style="font-size:12.8px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">19<wbr>,PETSC_NULL_INTEGER,19,</span><span style="font-size:12.8px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">PETSC_<wbr>NULL_INTEGER,ierr)</span><br></div><div><span style="font-size:12.8px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline"><br></span></div><div><span style="font-size:12.8px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">But i am not aware on how to do this for the DM if it needs something more specific/different,</span></div><div><span style="font-size:12.8px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline"><br></span></div><div><span style="font-size:12.8px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">Thanks,</span></div></div><div class="HOEnZb"><div class="h5"><div class="gmail_extra"><br><div class="gmail_quote">On Wed, Aug 15, 2018 at 1:51 PM, Matthew Knepley <span dir="ltr"><<a href="mailto:knepley@gmail.com" target="_blank">knepley@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div class="gmail_quote"><span><div dir="ltr">On Wed, Aug 15, 2018 at 4:39 PM Manuel Valera <<a href="mailto:mvalera-w@sdsu.edu" target="_blank">mvalera-w@sdsu.edu</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Hello PETSc devs,<div><br></div><div>I am running into an error when trying to use the MATMPIAIJVIENNACL Matrix type in MPI calls, the same code runs for MATSEQAIJVIENNACL type in one processor. The error happens when calling MatSetValues for this specific configuration. It does not occur when using MPI DMMatrix types only.</div></div></blockquote><div><br></div></span><div>The DM properly preallocates the matrix. I am assuming you do not here.</div><div><br></div><div> Matt</div><div><div class="m_-4647800331915645351h5"><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div>Any help will be appreciated, </div><div><br></div><div>Thanks,</div><div><br></div><div><br></div><div><br></div><div>My program call:</div><div><br></div><div>mpirun -n 2 ./gcmLEP.GPU tc=TestCases/LockRelease/LE_6x<wbr>6x6/ jid=tiny_cuda_test_n2 -ksp_type cg -dm_vec_type viennacl -dm_mat_type aijviennacl -pc_type saviennacl -log_view <br></div><div><br></div><div><br></div><div>The error (repeats after each MatSetValues call):</div><div><br></div><div><div>[1]PETSC ERROR: --------------------- Error Message ------------------------------<wbr>------------------------------<wbr>--</div><div>[1]PETSC ERROR: Argument out of range</div><div>[1]PETSC ERROR: Inserting a new nonzero at global row/column (75, 50) into matrix</div><div>[1]PETSC ERROR: See <a href="http://www.mcs.anl.gov/petsc/documentation/faq.html" target="_blank">http://www.mcs.anl.gov/petsc/d<wbr>ocumentation/faq.html</a> for trouble shooting.</div><div>[1]PETSC ERROR: Petsc Development GIT revision: v3.9.2-549-g779ab53 GIT Date: 2018-05-31 17:31:13 +0300</div><div>[1]PETSC ERROR: ./gcmLEP.GPU on a cuda-debug named node50 by valera Wed Aug 15 13:10:44 2018</div><div>[1]PETSC ERROR: Configure options PETSC_ARCH=cuda-debug --with-mpi-dir=/usr/lib64/open<wbr>mpi --COPTFLAGS=-O2 --CXXOPTFLAGS=-O2 --FOPTFLAGS=-O2 --with-shared-libraries=1 --with-debugging=1 --with-cuda=1 --CUDAFLAGS=-arch=sm_60 --with-blaslapack-dir=/usr/lib<wbr>64 --download-viennacl</div><div>[1]PETSC ERROR: #1 MatSetValues_MPIAIJ() line 608 in /home/valera/petsc/src/mat/imp<wbr>ls/aij/mpi/mpiaij.c</div><div>[1]PETSC ERROR: #2 MatSetValues() line 1339 in /home/valera/petsc/src/mat/int<wbr>erface/matrix.c</div></div><div><br></div><div><br></div><div>My Code structure:</div><div><br></div><div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">call DMCreateMatrix(daDummy,A,ierr)<br>call MatSetFromOptions(A,ierr)<br>call MPI_Comm_size(PETSC_COMM_WORLD<wbr>, numprocs, ierr)<br>if (numprocs > 1) then ! set matrix type parallel<br> ! Get local size<br> call DMDACreateNaturalVector(daDumm<wbr>y,Tmpnat,ierr)<br> call VecGetLocalSize(Tmpnat,locsize<wbr>,ierr)<br> call VecDestroy(Tmpnat,ierr)<br> ! Set matrix<br>#ifdef GPU<br> call MatSetType(A,MATAIJVIENNACL,ie<wbr>rr)<br> call DMSetMatType(daDummy,MATMPIAIJ<wbr>VIENNACL,ierr)<br> call DMSetVecType(daDummy,VECMPIVIE<wbr>NNACL,ierr)<br> print*,'SETTING GPU TYPES'<br>#else<br> call DMSetMatType(daDummy,MATMPIAIJ<wbr>,ierr)<br> call DMSetMatType(daDummy,VECMPI,ie<wbr>rr)<br> call MatSetType(A,MATMPIAIJ,ierr)!<br>#endif<br> call MatMPIAIJSetPreallocation(A,19<wbr>,PETSC_NULL_INTEGER,19,PETSC_<wbr>NULL_INTEGER,ierr)<br>else ! set matrix type sequential<br>#ifdef GPU<br> call DMSetMatType(daDummy,MATSEQAIJ<wbr>VIENNACL,ierr)<br> call DMSetVecType(daDummy,VECSEQVIE<wbr>NNACL,ierr)<br> call MatSetType(A,MATSEQAIJVIENNACL<wbr>,ierr)<br> print*,'SETTING GPU TYPES'<br>#else<br> call DMSetMatType(daDummy,MATSEQAIJ<wbr>,ierr)<br> call DMSetMatType(daDummy,VECSEQ,ie<wbr>rr)<br> call MatSetType(A,MATSEQAIJ,ierr)<br>#endif<br>call MatSetUp(A,ierr)<br>call getCenterInfo(daGrid,xstart,ys<wbr>tart,zstart,xend,yend,zend)<br></blockquote><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">do k=zstart,zend-1<br> do j=ystart,yend-1<br> do i=xstart,xend-1<br>[..]<br> call MatSetValues(A,1,row,sumpos,po<wbr>s(0:iter-1),vals(0:iter-1),INS<wbr>ERT_VALUES,ierr)<br>[..]</blockquote></div><div><br></div><div><br></div><div><br></div><div><br></div></div>
</blockquote></div></div></div><span class="m_-4647800331915645351HOEnZb"><font color="#888888"><br clear="all"><div><br></div>-- <br><div dir="ltr" class="m_-4647800331915645351m_3778373861107327029gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>-- Norbert Wiener</div><div><br></div><div><a href="http://www.caam.rice.edu/~mk51/" target="_blank">https://www.cse.buffalo.edu/~k<wbr>nepley/</a><br></div></div></div></div></div></font></span></div>
</blockquote></div><br></div>
</div></div></blockquote></div><br></div>