<div dir="auto">I got it, thanks so much!<div dir="auto"><br></div><div dir="auto">Eda</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, Jul 1, 2020, 2:42 PM Matthew Knepley <<a href="mailto:knepley@gmail.com">knepley@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div dir="ltr">On Wed, Jul 1, 2020 at 7:32 AM Eda Oktay <<a href="mailto:eda.oktay@metu.edu.tr" target="_blank" rel="noreferrer">eda.oktay@metu.edu.tr</a>> wrote:<br></div><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">I'm sorry but I still can't understand how to put an input as these<br>
parameters. It is the first time I saw 'const' for IS. If it doesn't<br>
refer IS, then as a parameter for row and column indices, how should I<br>
give them as a valid input? It still should be an IS, right?<br>
<br>
Lastly, I tried to use in the form:<br>
<br>
IS idUi;<br>
ISCreateStride(PETSC_COMM_WORLD,siz,0,1,&idUi);<br>
Mat *submat;<br>
MatCreateSubMatrices(U,nev,&idUi,&idUi,MAT_INITIAL_MATRIX,&submat);<br></blockquote><div><br></div><div>You gave n = 4 (you call it nev), but you only have 1 rowIS. You are supposed to give an array of 4.</div><div><br></div><div> Thanks,</div><div><br></div><div> Matt</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
where nev = 4, siz = 72 and U is 72*4 matrix.<br>
<br>
But then, the error becomes about the matrix, U:<br>
<br>
[0]PETSC ERROR: --------------------- Error Message<br>
--------------------------------------------------------------<br>
[0]PETSC ERROR: [1]PETSC ERROR: --------------------- Error Message<br>
--------------------------------------------------------------<br>
[1]PETSC ERROR: Invalid argument<br>
[1]PETSC ERROR: Wrong type of object: Parameter # 1<br>
Invalid argument<br>
[0]PETSC ERROR: Wrong type of object: Parameter # 1<br>
[0]PETSC ERROR: See<br>
<a href="https://www.mcs.anl.gov/petsc/documentation/faq.html" rel="noreferrer noreferrer" target="_blank">https://www.mcs.anl.gov/petsc/documentation/faq.html</a> for trouble<br>
shooting.<br>
[0]PETSC ERROR: Petsc Release Version 3.13.2, Jun 02, 2020<br>
[0]PETSC ERROR: ./deneme_new_vecscatter_arastep on a<br>
arch-linux2-c-debug named <a href="http://b342.wls.metu.edu.tr" rel="noreferrer noreferrer" target="_blank">b342.wls.metu.edu.tr</a> by edaoktay Wed Jul 1<br>
14:25:00 2020<br>
[0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++<br>
--download-mpich --download-openblas --download-slepc --download-metis<br>
--download-parmetis --download-chaco --with-X=1<br>
[1]PETSC ERROR: See<br>
<a href="https://www.mcs.anl.gov/petsc/documentation/faq.html" rel="noreferrer noreferrer" target="_blank">https://www.mcs.anl.gov/petsc/documentation/faq.html</a> for trouble<br>
shooting.<br>
[1]PETSC ERROR: Petsc Release Version 3.13.2, Jun 02, 2020<br>
[1]PETSC ERROR: [0]PETSC ERROR: #1 ISSorted() line 1777 in<br>
/home/edaoktay/petsc-3.13.2/src/vec/is/is/interface/index.c<br>
[0]PETSC ERROR: ./deneme_new_vecscatter_arastep on a<br>
arch-linux2-c-debug named <a href="http://b342.wls.metu.edu.tr" rel="noreferrer noreferrer" target="_blank">b342.wls.metu.edu.tr</a> by edaoktay Wed Jul 1<br>
14:25:00 2020<br>
[1]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++<br>
--download-mpich --download-openblas --download-slepc --download-metis<br>
--download-parmetis --download-chaco --with-X=1<br>
[1]PETSC ERROR: #2 MatCreateSubMatrices_MPIDense_Local() line 104 in<br>
/home/edaoktay/petsc-3.13.2/src/mat/impls/dense/mpi/mmdense.c<br>
[0]PETSC ERROR: #3 MatCreateSubMatrices_MPIDense() line 66 in<br>
/home/edaoktay/petsc-3.13.2/src/mat/impls/dense/mpi/mmdense.c<br>
#1 ISSorted() line 1777 in<br>
/home/edaoktay/petsc-3.13.2/src/vec/is/is/interface/index.c<br>
[1]PETSC ERROR: #2 MatCreateSubMatrices_MPIDense_Local() line 104 in<br>
/home/edaoktay/petsc-3.13.2/src/mat/impls/dense/mpi/mmdense.c<br>
[0]PETSC ERROR: #4 MatCreateSubMatrices() line 6758 in<br>
/home/edaoktay/petsc-3.13.2/src/mat/interface/matrix.c<br>
[1]PETSC ERROR: #3 MatCreateSubMatrices_MPIDense() line 66 in<br>
/home/edaoktay/petsc-3.13.2/src/mat/impls/dense/mpi/mmdense.c<br>
[1]PETSC ERROR: #4 MatCreateSubMatrices() line 6758 in<br>
/home/edaoktay/petsc-3.13.2/src/mat/interface/matrix.c<br>
<br>
Before I used &idUi, I tried to use just idUi (without & sign) and it<br>
didn't work, so I decided to try using this way, since may be 'const'<br>
refers this one. But how can my matrix be of wrong type? It is MPI<br>
Dense.<br>
<br>
Thanks!<br>
<br>
Eda<br>
<br>
Matthew Knepley <<a href="mailto:knepley@gmail.com" target="_blank" rel="noreferrer">knepley@gmail.com</a>>, 1 Tem 2020 Çar, 13:43 tarihinde şunu yazdı:<br>
><br>
> On Wed, Jul 1, 2020 at 6:34 AM Eda Oktay <<a href="mailto:eda.oktay@metu.edu.tr" target="_blank" rel="noreferrer">eda.oktay@metu.edu.tr</a>> wrote:<br>
>><br>
>> Der Barry,<br>
>><br>
>> ı am trying to use your way but I couldn't understand how I create sub<br>
>> matrices by using MatCreateSubMatrices() since as input, the function<br>
>> needs const IS instead of IS and I couldn't understand how to get a<br>
>> const IS. I tried to use ISCreateStride since this IS should be 0:71<br>
>> because as you mentioned, the sub matrix should consist of the entire<br>
>> matrix. However, since ISCreateStride produces IS, not const IS, I<br>
>> couldn't use it in MatCreateSubMatrices().<br>
><br>
><br>
> The 'const' refers to the array, not the IS:<br>
><br>
> <a href="https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MatCreateSubMatrices.html" rel="noreferrer noreferrer" target="_blank">https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MatCreateSubMatrices.html</a><br>
><br>
> Thanks,<br>
><br>
> Matt<br>
><br>
>><br>
>> Thanks<br>
>><br>
>> Eda<br>
>><br>
>> Barry Smith <<a href="mailto:bsmith@petsc.dev" target="_blank" rel="noreferrer">bsmith@petsc.dev</a>>, 10 Haz 2020 Çar, 19:59 tarihinde şunu yazdı:<br>
>> ><br>
>> ><br>
>> > You can use MatCreateSubMatrices() with each process getting a single sequential dense sub matrix that consists of the entire matrix.<br>
>> ><br>
>> > Use VecDuplicateVecs() to create an array of 72 vectors (create a single seq vector of size 4 as the input to this routine)<br>
>> ><br>
>> > Then use MatDenseGetArrayRead() to access the upper left corner of the new sequential dense matrix<br>
>> ><br>
>> > Loop over the vectors calling VecGetArray()<br>
>> > Then loop over the row of the dense array filling up the vector<br>
>> ><br>
>> > Because dense matrices are stored by column, you have to do this looping to fill up the vectors, they can't share the space with the matrix.<br>
>> ><br>
>> > Barry<br>
>> ><br>
>> ><br>
>> ><br>
>> > On Jun 10, 2020, at 11:36 AM, Matthew Knepley <<a href="mailto:knepley@gmail.com" target="_blank" rel="noreferrer">knepley@gmail.com</a>> wrote:<br>
>> ><br>
>> > On Wed, Jun 10, 2020 at 12:26 PM Eda Oktay <<a href="mailto:eda.oktay@metu.edu.tr" target="_blank" rel="noreferrer">eda.oktay@metu.edu.tr</a>> wrote:<br>
>> >><br>
>> >> Matthew Knepley <<a href="mailto:knepley@gmail.com" target="_blank" rel="noreferrer">knepley@gmail.com</a>>, 10 Haz 2020 Çar, 19:13 tarihinde<br>
>> >> şunu yazdı:<br>
>> >> ><br>
>> >> > On Wed, Jun 10, 2020 at 12:07 PM Eda Oktay <<a href="mailto:eda.oktay@metu.edu.tr" target="_blank" rel="noreferrer">eda.oktay@metu.edu.tr</a>> wrote:<br>
>> >> >><br>
>> >> >> Der Matt,<br>
>> >> >><br>
>> >> >> When I looked at the results, I found that there are some problems I<br>
>> >> >> couldn't understand.<br>
>> >> >><br>
>> >> >> First of all, I am working on a 72*4 matrix and as I said before, I<br>
>> >> >> want to have 72 different vectors having size 4 each, whose elements<br>
>> >> >> consist of the elements in the same row. And of course, all vectors<br>
>> >> >> should be in all processors (currently I am using 4 processors).<br>
>> >> >><br>
>> >> >> When I use your scatter code, the output vector is divided into 4<br>
>> >> >> parts for 4 processors and each vector consists of 18 row vectors<br>
>> >> >> whose elements are arranged in a way that if I want to find zeroth row<br>
>> >> >> vector, its elements are located in 0th,18th,36th,54th elements.<br>
>> >> ><br>
>> >> ><br>
>> >> > Was the global size of the vector you wrapped around the dense matrix 72*4?<br>
>> >><br>
>> >> Yes it is. I set up its global size to 72*4.<br>
>> >><br>
>> >> ><br>
>> >> > If you use CreateToAll(), it will make a vector on each process which has the global size of the original vector.<br>
>> >><br>
>> >> Although I set 72*4, the size of the vectors in each process is 72.<br>
>> ><br>
>> ><br>
>> > You can understand how it is hard to accept, as this code is tested every night. Can you VecView() the input vector<br>
>> > to CreateToAll and the output vector, and send that output?<br>
>> ><br>
>> > Thanks,<br>
>> ><br>
>> > Matt<br>
>> ><br>
>> >><br>
>> >> Thanks,<br>
>> >><br>
>> >> Eda<br>
>> >><br>
>> >> ><br>
>> >> > Thanks,<br>
>> >> ><br>
>> >> > Matt<br>
>> >> ><br>
>> >> >><br>
>> >> >> So, isn't scatter's goal is to scatter all values to all processors?<br>
>> >> >><br>
>> >> >> Furthermore, I am trying to use my vectors in that way but isn't there<br>
>> >> >> any possible way that I can reach my goal entirely?<br>
>> >> >><br>
>> >> >> Thanks so much for your help,<br>
>> >> >><br>
>> >> >> Eda<br>
>> >> >><br>
>> >> >> Matthew Knepley <<a href="mailto:knepley@gmail.com" target="_blank" rel="noreferrer">knepley@gmail.com</a>>, 10 Haz 2020 Çar, 18:11 tarihinde<br>
>> >> >> şunu yazdı:<br>
>> >> >> ><br>
>> >> >> > On Wed, Jun 10, 2020 at 10:09 AM Eda Oktay <<a href="mailto:eda.oktay@metu.edu.tr" target="_blank" rel="noreferrer">eda.oktay@metu.edu.tr</a>> wrote:<br>
>> >> >> >><br>
>> >> >> >> Dear Matt,<br>
>> >> >> >><br>
>> >> >> >> I have one last question I believe. Up to creating a dense matrix I<br>
>> >> >> >> did what you've suggested. Thank you so much for that.<br>
>> >> >> >><br>
>> >> >> >> I created a new dense matrix. Now, how should I wrap each vector in a<br>
>> >> >> >> MatDense again? I mean, what is wrapping vectors in a matrix? To put<br>
>> >> >> >> each of them again as rows?<br>
>> >> >> ><br>
>> >> >> ><br>
>> >> >> > I thought you need a dense matrix for something, since you started with one. If you<br>
>> >> >> > do not, just do VecGetArray() on the vector from CreateToAll and use the values.<br>
>> >> >> ><br>
>> >> >> > Thanks,<br>
>> >> >> ><br>
>> >> >> > Matt<br>
>> >> >> ><br>
>> >> >> >><br>
>> >> >> >> Thanks!<br>
>> >> >> >><br>
>> >> >> >> Eda<br>
>> >> >> >><br>
>> >> >> >> Matthew Knepley <<a href="mailto:knepley@gmail.com" target="_blank" rel="noreferrer">knepley@gmail.com</a>>, 10 Haz 2020 Çar, 16:16 tarihinde<br>
>> >> >> >> şunu yazdı:<br>
>> >> >> >> ><br>
>> >> >> >> > On Wed, Jun 10, 2020 at 9:08 AM Eda Oktay <<a href="mailto:eda.oktay@metu.edu.tr" target="_blank" rel="noreferrer">eda.oktay@metu.edu.tr</a>> wrote:<br>
>> >> >> >> >><br>
>> >> >> >> >> Dear Matt,<br>
>> >> >> >> >><br>
>> >> >> >> >> Matthew Knepley <<a href="mailto:knepley@gmail.com" target="_blank" rel="noreferrer">knepley@gmail.com</a>>, 10 Haz 2020 Çar, 16:03 tarihinde<br>
>> >> >> >> >> şunu yazdı:<br>
>> >> >> >> >> ><br>
>> >> >> >> >> > On Wed, Jun 10, 2020 at 8:56 AM Eda Oktay <<a href="mailto:eda.oktay@metu.edu.tr" target="_blank" rel="noreferrer">eda.oktay@metu.edu.tr</a>> wrote:<br>
>> >> >> >> >> >><br>
>> >> >> >> >> >> Hi all,<br>
>> >> >> >> >> >><br>
>> >> >> >> >> >> I am trying to get all the rows of a parallel matrix as individual<br>
>> >> >> >> >> >> vectors. For instance, if I have 72*4 matrix, I want to get 72<br>
>> >> >> >> >> >> different vectors having size 4.<br>
>> >> >> >> >> >><br>
>> >> >> >> >> >> As far as I understood, MatGetRow is only for local rows, so<br>
>> >> >> >> >> >> MatGetOwnershipRange is used, however, when I tried this one, I<br>
>> >> >> >> >> >> couldn't get the whole and desired row vectors.<br>
>> >> >> >> >> >><br>
>> >> >> >> >> >> In MatGetRow explanation, it is written that I should use<br>
>> >> >> >> >> >> MatCreateSubMatrices first, then use MatGetRow. But I couldn't<br>
>> >> >> >> >> >> understand to which extent I should create submatrices. I just need to<br>
>> >> >> >> >> >> have all 72 rows as 72 different vectors each having 4 elements.<br>
>> >> >> >> >> ><br>
>> >> >> >> >> ><br>
>> >> >> >> >> > 1) For sparse matrices, the storage is always divided by row, so that values can only be retrieved for local rows with MatGetRow()<br>
>> >> >> >> >> ><br>
>> >> >> >> >> > 2) Is this matrix sparse? It sounds like it is dense.<br>
>> >> >> >> >><br>
>> >> >> >> >> Matrix is dense.<br>
>> >> >> >> >><br>
>> >> >> >> >> ><br>
>> >> >> >> >> > 3) Are you asking to get all matrix values on all processes? If so, I think the easiest thing to do is first wrap a Vec around the<br>
>> >> >> >> >> > values, then use VecScatterToAll(), then wrap each one in a MatDense again.<br>
>> >> >> >> >><br>
>> >> >> >> >> Yes, I want all row vectors on all processes. In a dense matrix,<br>
>> >> >> >> >> should I still wrap a Vec around the values? I know I should use<br>
>> >> >> >> >> scatter but I couldn't even wrap a Vec around them.<br>
>> >> >> >> ><br>
>> >> >> >> ><br>
>> >> >> >> > I would do<br>
>> >> >> >> ><br>
>> >> >> >> > MatGetSize(&N);<br>
>> >> >> >> > MatGetLocalSize(&m);<br>
>> >> >> >> > <a href="https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MatDenseGetArray.html" rel="noreferrer noreferrer" target="_blank">https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MatDenseGetArray.html</a><br>
>> >> >> >> > <create vector of local size m*N><br>
>> >> >> >> > <a href="https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Vec/VecPlaceArray.html" rel="noreferrer noreferrer" target="_blank">https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Vec/VecPlaceArray.html</a><br>
>> >> >> >> > <a href="https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Vec/VecScatterCreateToAll.html" rel="noreferrer noreferrer" target="_blank">https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Vec/VecScatterCreateToAll.html</a><br>
>> >> >> >> > <do scatter><br>
>> >> >> >> > <a href="https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Vec/VecResetArray.html#VecResetArray" rel="noreferrer noreferrer" target="_blank">https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Vec/VecResetArray.html#VecResetArray</a><br>
>> >> >> >> > <a href="https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MatCreateDense.html" rel="noreferrer noreferrer" target="_blank">https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MatCreateDense.html</a><br>
>> >> >> >> > <use it><br>
>> >> >> >> > <destroy matrix><br>
>> >> >> >> > <destroy vector from CreateToAll><br>
>> >> >> >> ><br>
>> >> >> >> > Thanks,<br>
>> >> >> >> ><br>
>> >> >> >> > Matt<br>
>> >> >> >> ><br>
>> >> >> >> >><br>
>> >> >> >> >> Thanks so much!<br>
>> >> >> >> >><br>
>> >> >> >> >> Eda<br>
>> >> >> >> >><br>
>> >> >> >> >> ><br>
>> >> >> >> >> > Thanks,<br>
>> >> >> >> >> ><br>
>> >> >> >> >> > Matt<br>
>> >> >> >> >> ><br>
>> >> >> >> >> >><br>
>> >> >> >> >> >> Thanks!<br>
>> >> >> >> >> >><br>
>> >> >> >> >> >> Eda<br>
>> >> >> >> >> ><br>
>> >> >> >> >> ><br>
>> >> >> >> >> ><br>
>> >> >> >> >> > --<br>
>> >> >> >> >> > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>
>> >> >> >> >> > -- Norbert Wiener<br>
>> >> >> >> >> ><br>
>> >> >> >> >> > <a href="https://www.cse.buffalo.edu/~knepley/" rel="noreferrer noreferrer" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br>
>> >> >> >> ><br>
>> >> >> >> ><br>
>> >> >> >> ><br>
>> >> >> >> > --<br>
>> >> >> >> > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>
>> >> >> >> > -- Norbert Wiener<br>
>> >> >> >> ><br>
>> >> >> >> > <a href="https://www.cse.buffalo.edu/~knepley/" rel="noreferrer noreferrer" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br>
>> >> >> ><br>
>> >> >> ><br>
>> >> >> ><br>
>> >> >> > --<br>
>> >> >> > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>
>> >> >> > -- Norbert Wiener<br>
>> >> >> ><br>
>> >> >> > <a href="https://www.cse.buffalo.edu/~knepley/" rel="noreferrer noreferrer" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br>
>> >> ><br>
>> >> ><br>
>> >> ><br>
>> >> > --<br>
>> >> > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>
>> >> > -- Norbert Wiener<br>
>> >> ><br>
>> >> > <a href="https://www.cse.buffalo.edu/~knepley/" rel="noreferrer noreferrer" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br>
>> ><br>
>> ><br>
>> ><br>
>> > --<br>
>> > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>
>> > -- Norbert Wiener<br>
>> ><br>
>> > <a href="https://www.cse.buffalo.edu/~knepley/" rel="noreferrer noreferrer" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br>
>> ><br>
>> ><br>
><br>
><br>
><br>
> --<br>
> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>
> -- Norbert Wiener<br>
><br>
> <a href="https://www.cse.buffalo.edu/~knepley/" rel="noreferrer noreferrer" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br>
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>-- Norbert Wiener</div><div><br></div><div><a href="http://www.cse.buffalo.edu/~knepley/" target="_blank" rel="noreferrer">https://www.cse.buffalo.edu/~knepley/</a><br></div></div></div></div></div></div></div></div>
</blockquote></div>