<div dir="ltr">Dear Matt,<div><br></div><div>I solved my problem by trying with 9*9 matrix as you said. I made indexing mistake.</div><div><br></div><div>Thank you for answering.</div><div><br></div><div>Eda</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">Matthew Knepley <<a href="mailto:knepley@gmail.com">knepley@gmail.com</a>>, 24 Nis 2019 Çar, 14:30 tarihinde şunu yazdı:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div dir="ltr">On Wed, Apr 24, 2019 at 6:35 AM Stefano Zampini via petsc-users <<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a>> wrote:<br></div><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Dump the index sets and the matrix in binary and send them</div></blockquote><div><br></div><div>First, reduce your problem size to about 10.</div><div><br></div><div> Matt</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div class="gmail_quote"><div dir="ltr" class="gmail_attr">Il giorno mer 24 apr 2019 alle ore 13:21 Eda Oktay <<a href="mailto:eda.oktay@metu.edu.tr" target="_blank">eda.oktay@metu.edu.tr</a>> ha scritto:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Since I am using square matrix, isn't the row local size of the matrix equal to the local size of IS? They are both 2127 and 2126.</div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">Stefano Zampini <<a href="mailto:stefano.zampini@gmail.com" target="_blank">stefano.zampini@gmail.com</a>>, 24 Nis 2019 Çar, 13:15 tarihinde şunu yazdı:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div dir="ltr">I don't understand your code. However, I suspect we have a bug here <a href="https://bitbucket.org/petsc/petsc/src/dba4d5898a4236b42fbe8dff209f1243d2f2582a/src/mat/impls/aij/mpi/mpiaij.c#lines-1652" target="_blank">https://bitbucket.org/petsc/petsc/src/dba4d5898a4236b42fbe8dff209f1243d2f2582a/src/mat/impls/aij/mpi/mpiaij.c#lines-1652</a></div><div>The code assumes the number of leaves is A->rmap->n (the row local size of the matrix). Shouldn't this be the local size of the IS (rowp)? Barry? </div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">Il giorno mer 24 apr 2019 alle ore 13:03 Eda Oktay <<a href="mailto:eda.oktay@metu.edu.tr" target="_blank">eda.oktay@metu.edu.tr</a>> ha scritto:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div dir="ltr"><div dir="ltr">Dear Stefano, <div><br></div><div>Thank you for answering. When I used MatSetSizes, I got an error since the matrix is read from outside, but from the error I understood that actually the local sizes are 2127 and 2126, so I misunderstood the problem. I am sorry for my mistake.</div><div><br></div><div>However, I still cannot understand where is the error. Because both communicators and local sizes of IS and the matrix are the same. I still get the same error in MatPermute:</div><div><br></div><div><div>[0]PETSC ERROR: Argument out of range</div><div>[0]PETSC ERROR: Index -1081207334 is out of range</div><div>[0]PETSC ERROR: See <a href="http://www.mcs.anl.gov/petsc/documentation/faq.html" target="_blank">http://www.mcs.anl.gov/petsc/documentation/faq.html</a> for trouble shooting.</div><div>[0]PETSC ERROR: Petsc Release Version 3.10.3, Dec, 18, 2018 </div><div>[0]PETSC ERROR: ./TEKRAR_TEK_SAYI_SON_YENI_DENEME_TEMIZ_ENYENI_FINAL on a arch-linux2-c-debug named <a href="http://53d.wls.metu.edu.tr" target="_blank">53d.wls.metu.edu.tr</a> by edaoktay Wed Apr 24 11:22:15 2019</div><div>[0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --with-cxx-dialect=C++11 --download-openblas --download-metis --download-parmetis --download-superlu_dist --download-slepc --download-mpich</div><div>[0]PETSC ERROR: #1 PetscLayoutFindOwner() line 248 in /home/edaoktay/petsc-3.10.3/include/petscis.h</div><div>[0]PETSC ERROR: #2 MatPermute_MPIAIJ() line 1685 in /home/edaoktay/petsc-3.10.3/src/mat/impls/aij/mpi/mpiaij.c</div><div>[0]PETSC ERROR: #3 MatPermute() line 4997 in /home/edaoktay/petsc-3.10.3/src/mat/interface/matrix.c</div><div>[0]PETSC ERROR: #4 main() line 361 in /home/edaoktay/petsc-3.10.3/arch-linux2-c-debug/share/slepc/examples/src/eda/TEKRAR_TEK_SAYI_SON_YENI_DENEME_TEMIZ_ENYENI_FINAL.c</div></div><div><br></div><div>This is the part of my program:</div><div><br></div><div><div> ierr = MatCreateVecs(L,&vr,NULL);CHKERRQ(ierr);</div><div> ierr = EPSGetEigenpair(eps,0,&kr,NULL,vr,NULL); </div><div> ierr = PetscPrintf(PETSC_COMM_WORLD," The second smallest eigenvalue: %g\n",kr);CHKERRQ(ierr); </div><div> </div><div> /* sort second smallest eigenvector */</div><div><br></div><div> ierr = VecGetSize(vr,&siz);CHKERRQ(ierr);</div><div> ierr = PetscMalloc1(siz,&idx);CHKERRQ(ierr);</div><div> for (i=0; i<siz;i++) idx[i] = i;</div><div> </div><div> VecScatter ctx;</div><div> Vec vout;</div><div> VecScatterCreateToAll(vr,&ctx,&vout);</div><div> VecScatterBegin(ctx,vr,vout,INSERT_VALUES,SCATTER_FORWARD);</div><div> VecScatterEnd(ctx,vr,vout,INSERT_VALUES,SCATTER_FORWARD);</div><div> VecScatterDestroy(&ctx);</div><div> </div><div> PetscScalar *avr;</div><div> ierr = VecGetArray(vout,&avr);CHKERRQ(ierr);</div><div> ierr = PetscSortRealWithPermutation(siz,avr,idx);CHKERRQ(ierr); </div><div> </div><div> /*Select out a piece of the resulting indices idx on each process; for example with two processes I think rank = 0 would get the first half of the idx and rank = 1 would get the second half.*/</div><div> </div><div> PetscMPIInt rank,size;</div><div> MPI_Comm_rank(PETSC_COMM_WORLD, &rank);</div><div> MPI_Comm_size(PETSC_COMM_WORLD, &size);</div><div> </div><div> PetscInt mod;</div><div> mod = siz % size;</div><div> </div><div> PetscInt *idxx,ss;</div><div> ss = (siz-mod)/size; </div><div><br></div><div> if (mod != 0){</div><div> if (rank<mod){</div><div> PetscMalloc1(ss+1,&idxx); </div><div> } else{</div><div> PetscMalloc1(ss,&idxx);</div><div> }</div><div> } else{</div><div> PetscMalloc1(ss,&idxx);</div><div> }</div><div><br></div><div><br></div><div> j =0;</div><div> for (i=rank*ss; i<(rank+1)*ss; i++) {</div><div> idxx[j] = idx[i];</div><div> //PetscPrintf(PETSC_COMM_WORLD," idxx: %D\n",idxx[j]);</div><div> j++;</div><div> </div><div> if (mod != 0){</div><div> if (rank<mod){</div><div> idxx[ss+1] = idx[ss*size+rank+1];</div><div> }</div><div> }</div><div><br></div><div> /*Permute matrix L (spy(A(p1,p1))*/</div><div> </div><div> if (mod != 0){</div><div> if (rank<mod){</div><div> ierr = ISCreateGeneral(PETSC_COMM_WORLD,ss+1,idxx,PETSC_COPY_VALUES,&is);CHKERRQ(ierr);</div><div> } else{</div><div> ierr = ISCreateGeneral(PETSC_COMM_WORLD,ss,idxx,PETSC_COPY_VALUES,&is);CHKERRQ(ierr);</div><div> } </div><div> </div><div> }else {</div><div> ierr = ISCreateGeneral(PETSC_COMM_WORLD,ss,idxx,PETSC_COPY_VALUES,&is);CHKERRQ(ierr);</div><div> }</div><div><br></div><div> ierr = ISSetPermutation(is);CHKERRQ(ierr); </div><div> ierr = MatPermute(A,is,is,&PL);CHKERRQ(ierr); </div></div><div><br></div><div>I printed IS and idxx, IS is a permutation and numbers seem to be correct. The program also works when mod==0.</div><div><br></div><div>Thanks,</div><div><br></div><div>Eda</div></div></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">Stefano Zampini <<a href="mailto:stefano.zampini@gmail.com" target="_blank">stefano.zampini@gmail.com</a>>, 22 Nis 2019 Pzt, 18:13 tarihinde şunu yazdı:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr">If you are using PETSC_DECIDE for the local sizes in MatSetSizes, the default local sizes should be 2127 and 2126. </div></div></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">Il giorno lun 22 apr 2019 alle ore 11:28 Eda Oktay via petsc-users <<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a>> ha scritto:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Thank you for your answers. I figured out that for a matrix of size 4253*4253, local size of Mat is 2127 and 2127 for 2 processors. However, I wrote the program such that the local sizes of IS 2127 and 2126. <div><br></div><div>Is local size of Mat being 2127 on both processors correct? If it is, then I will change the local size of IS but then it will exceed the global size of Mat. Isn't this also a problem?</div><div><br></div><div>Thanks a lot,</div><div><br></div><div>Eda</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">Smith, Barry F. <<a href="mailto:bsmith@mcs.anl.gov" target="_blank">bsmith@mcs.anl.gov</a>>, 9 Nis 2019 Sal, 00:31 tarihinde şunu yazdı:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><br>
Suggest printing out the IS with ISView after it is created and confirming that 1) it is a permutation and 2) that the size of the IS on each process matches the number of rows on that process. <br>
<br>
Note from the manual page: The index sets should be on the same communicator as Mat and have the same local sizes.<br>
<br>
Barry<br>
<br>
<br>
> On Apr 8, 2019, at 3:21 AM, Eda Oktay via petsc-users <<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a>> wrote:<br>
> <br>
> Hello again,<br>
> <br>
> I solved the problem for even numbered sized matrices. However when the matrix size is odd, then number of elements in each index set at each processor are different. (For example, for size 4253*4253 and 2 processors, size of index set at processor 0 is 2127 where at processor 1, it is 2126) I think this is why, MatPermute again gives the same "Argument out of range" error. Index sets look like correct but I still did not get why I get this error.<br>
> <br>
> This is the part of my program:<br>
> <br>
> PetscMPIInt rank,size;<br>
> MPI_Comm_rank(PETSC_COMM_WORLD, &rank);<br>
> MPI_Comm_size(PETSC_COMM_WORLD, &size);<br>
> <br>
> PetscInt mod;<br>
> mod = siz % size;<br>
> <br>
> PetscInt *idxx,ss;<br>
> ss = (siz-mod)/size; <br>
> <br>
> if (mod != 0){<br>
> if (rank<mod){<br>
> PetscMalloc1(ss+1,&idxx); <br>
> } else{<br>
> PetscMalloc1(ss,&idxx);<br>
> }<br>
> }<br>
> <br>
> if (rank != size-1) {<br>
> j =0;<br>
> for (i=rank*ss; i<(rank+1)*ss; i++) {<br>
> idxx[j] = idx[i];<br>
> j++;<br>
> }<br>
> <br>
> } else {<br>
> <br>
> j =0;<br>
> for (i=rank*ss; i<siz; i++) {<br>
> idxx[j] = idx[i];<br>
> j++;<br>
> }<br>
> <br>
> }<br>
> <br>
> if (mod != 0){<br>
> if (rank<mod){ <br>
> idxx[ss+1] = idx[ss*size+rank+1];<br>
> }<br>
> }<br>
> <br>
> /*Permute matrix L (spy(A(p1,p1))*/<br>
> <br>
> if (mod != 0){<br>
> if (rank<mod){<br>
> ierr = ISCreateGeneral(PETSC_COMM_WORLD,ss+1,idxx,PETSC_COPY_VALUES,&is);CHKERRQ(ierr);<br>
> } else{<br>
> ierr = ISCreateGeneral(PETSC_COMM_WORLD,ss,idxx,PETSC_COPY_VALUES,&is);CHKERRQ(ierr);<br>
> }<br>
> }<br>
> ierr = ISSetPermutation(is);CHKERRQ(ierr);<br>
> <br>
> ierr = MatPermute(A,is,is,&PL);CHKERRQ(ierr); <br>
> <br>
> And I get the following error even if I use MatSetOption :<br>
> <br>
> [0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------<br>
> [0]PETSC ERROR: Argument out of range<br>
> [0]PETSC ERROR: New nonzero at (0,4252) caused a malloc<br>
> Use MatSetOption(A, MAT_NEW_NONZERO_ALLOCATION_ERR, PETSC_FALSE) to turn off this check<br>
> [0]PETSC ERROR: See <a href="http://www.mcs.anl.gov/petsc/documentation/faq.html" rel="noreferrer" target="_blank">http://www.mcs.anl.gov/petsc/documentation/faq.html</a> for trouble shooting.<br>
> [0]PETSC ERROR: Petsc Release Version 3.10.3, Dec, 18, 2018 <br>
> [0]PETSC ERROR: ./SON_YENI_DENEME_TEMIZ_ENYENI_FINAL on a arch-linux2-c-debug named <a href="http://dd2b.wls.metu.edu.tr" rel="noreferrer" target="_blank">dd2b.wls.metu.edu.tr</a> by edaoktay Mon Apr 8 11:10:59 2019<br>
> [0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --with-cxx-dialect=C++11 --download-openblas --download-metis --download-parmetis --download-superlu_dist --download-slepc --download-mpich<br>
> [0]PETSC ERROR: #1 MatSetValues_MPIAIJ() line 617 in /home/edaoktay/petsc-3.10.3/src/mat/impls/aij/mpi/mpiaij.c<br>
> [0]PETSC ERROR: #2 MatSetValues() line 1349 in /home/edaoktay/petsc-3.10.3/src/mat/interface/matrix.c<br>
> [0]PETSC ERROR: #3 MatPermute_MPIAIJ() line 1714 in /home/edaoktay/petsc-3.10.3/src/mat/impls/aij/mpi/mpiaij.c<br>
> [0]PETSC ERROR: #4 MatPermute() line 4997 in /home/edaoktay/petsc-3.10.3/src/mat/interface/matrix.c<br>
> [0]PETSC ERROR: #5 main() line 352 in /home/edaoktay/petsc-3.10.3/arch-linux2-c-debug/share/slepc/examples/src/eda/SON_YENI_DENEME_TEMIZ_ENYENI_FINAL.c<br>
> [0]PETSC ERROR: PETSc Option Table entries:<br>
> [0]PETSC ERROR: -f /home/edaoktay/petsc-3.10.3/share/petsc/datafiles/matrices/binary_files/airfoil1_binary<br>
> [0]PETSC ERROR: -mat_partitioning_type parmetis<br>
> [0]PETSC ERROR: -unweighted<br>
> [0]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint@mcs.anl.gov----------<br>
> <br>
> Thanks!<br>
> <br>
> Eda<br>
> <br>
> Eda Oktay <<a href="mailto:eda.oktay@metu.edu.tr" target="_blank">eda.oktay@metu.edu.tr</a>>, 25 Mar 2019 Pzt, 13:53 tarihinde şunu yazdı:<br>
> I attached whole program I wrote where the problem is in line 285. One of the matrices I used was airfoil1_binary, included in the folder. Also, I included makefile. Is that what you want?<br>
> <br>
> Matthew Knepley <<a href="mailto:knepley@gmail.com" target="_blank">knepley@gmail.com</a>>, 25 Mar 2019 Pzt, 13:41 tarihinde şunu yazdı:<br>
> That should not happen. Can you send in a small example that we can debug.<br>
> <br>
> Thanks,<br>
> <br>
> Matt<br>
> <br>
> On Mon, Mar 25, 2019 at 12:38 AM Eda Oktay via petsc-users <<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a>> wrote:<br>
> Hello,<br>
> <br>
> I am trying to permute a vector A using following lines: <br>
> <br>
> ierr = ISCreateGeneral(PETSC_COMM_SELF,siz,idx,PETSC_COPY_VALUES,&is);CHKERRQ(ierr);<br>
> ierr = ISSetPermutation(is);CHKERRQ(ierr);<br>
> ierr = ISDuplicate(is,&newIS);CHKERRQ(ierr);<br>
> ierr = MatSetOption(A,MAT_NEW_NONZERO_ALLOCATION_ERR,PETSC_FALSE);CHKERRQ(ierr); <br>
> ierr = MatPermute(A,is,newIS,&PL);CHKERRQ(ierr); <br>
> <br>
> However, in MatPermute line, I get the following error even if I used MatSetOption before this line:<br>
> <br>
> [0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------<br>
> [0]PETSC ERROR: Argument out of range<br>
> [0]PETSC ERROR: New nonzero at (0,485) caused a malloc<br>
> Use MatSetOption(A, MAT_NEW_NONZERO_ALLOCATION_ERR, PETSC_FALSE) to turn off this check<br>
> [0]PETSC ERROR: See <a href="http://www.mcs.anl.gov/petsc/documentation/faq.html" rel="noreferrer" target="_blank">http://www.mcs.anl.gov/petsc/documentation/faq.html</a> for trouble shooting.<br>
> [0]PETSC ERROR: Petsc Release Version 3.10.3, Dec, 18, 2018 <br>
> [0]PETSC ERROR: ./DENEME_TEMIZ_ENYENI_FINAL on a arch-linux2-c-debug named <a href="http://1232.wls.metu.edu.tr" rel="noreferrer" target="_blank">1232.wls.metu.edu.tr</a> by edaoktay Mon Mar 25 12:15:14 2019<br>
> [0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --with-cxx-dialect=C++11 --download-openblas --download-metis --download-parmetis --download-superlu_dist --download-slepc --download-mpich<br>
> [0]PETSC ERROR: #1 MatSetValues_MPIAIJ() line 579 in /home/edaoktay/petsc-3.10.3/src/mat/impls/aij/mpi/mpiaij.c<br>
> [0]PETSC ERROR: #2 MatAssemblyEnd_MPIAIJ() line 807 in /home/edaoktay/petsc-3.10.3/src/mat/impls/aij/mpi/mpiaij.c<br>
> [0]PETSC ERROR: #3 MatAssemblyEnd() line 5340 in /home/edaoktay/petsc-3.10.3/src/mat/interface/matrix.c<br>
> [0]PETSC ERROR: #4 MatPermute_MPIAIJ() line 1723 in /home/edaoktay/petsc-3.10.3/src/mat/impls/aij/mpi/mpiaij.c<br>
> [0]PETSC ERROR: #5 MatPermute() line 4997 in /home/edaoktay/petsc-3.10.3/src/mat/interface/matrix.c<br>
> [0]PETSC ERROR: #6 main() line 285 in /home/edaoktay/petsc-3.10.3/arch-linux2-c-debug/share/slepc/examples/src/eda/DENEME_TEMIZ_ENYENI_FINAL.c<br>
> [0]PETSC ERROR: PETSc Option Table entries:<br>
> [0]PETSC ERROR: -f /home/edaoktay/petsc-3.10.3/share/petsc/datafiles/matrices/binary_files/airfoil1_binary<br>
> [0]PETSC ERROR: -mat_partitioning_type parmetis<br>
> [0]PETSC ERROR: -weighted<br>
> [0]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint@mcs.anl.gov----------<br>
> <br>
> I'll be glad if you can help me.<br>
> <br>
> Thanks!<br>
> <br>
> Eda<br>
> <br>
> <br>
> -- <br>
> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>
> -- Norbert Wiener<br>
> <br>
> <a href="https://www.cse.buffalo.edu/~knepley/" rel="noreferrer" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br>
<br>
</blockquote></div>
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr" class="gmail-m_-5726192447822237761gmail-m_606514280929872134gmail-m_-5761846145796336703gmail-m_-6245399490948643678gmail-m_2031354379316320429gmail-m_-1213966221332933972gmail_signature">Stefano</div>
</blockquote></div>
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr" class="gmail-m_-5726192447822237761gmail-m_606514280929872134gmail-m_-5761846145796336703gmail-m_-6245399490948643678gmail_signature">Stefano</div>
</blockquote></div>
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr" class="gmail-m_-5726192447822237761gmail-m_606514280929872134gmail_signature">Stefano</div>
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr" class="gmail-m_-5726192447822237761gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>-- Norbert Wiener</div><div><br></div><div><a href="http://www.cse.buffalo.edu/~knepley/" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br></div></div></div></div></div></div></div></div>
</blockquote></div>