From mukkundsunjii at gmail.com Wed Jul 1 02:20:48 2020 From: mukkundsunjii at gmail.com (MUKKUND SUNJII) Date: Wed, 1 Jul 2020 09:20:48 +0200 Subject: [petsc-users] Regarding Adding to ex11.c In-Reply-To: References: <3F4CEA70-F9B3-4497-B931-BBB534F3A27B@gmail.com> <9537D2A9-3DE8-4BF5-BCCA-45185554B971@gmail.com> Message-ID: <299D2F8A-1D8D-406D-88D5-6B032CBF82A4@gmail.com> Excellent! That sounds great. Thank you for your response. Regards, Mukkund > On 30 Jun 2020, at 23:55, Mark Adams wrote: > > You want to make an MR (check to delete branch after merge and squash commits), you can mark as WIP, run the pipeline and after you get it to run clean and merge any comments, umark WIP. If it does not get merged in a few days you could ask Satish. > Mark > > On Tue, Jun 30, 2020 at 5:12 PM MUKKUND SUNJII > wrote: > Greetings, > > Thank you for your message. > > Excellent! Does it suffice to ask for the merge request once I made the necessary additions to ex11.c? > > Regards, > > Mukkund > >> On 30 Jun 2020, at 19:16, Matthew Knepley > wrote: >> >> On Tue, Jun 30, 2020 at 12:56 PM MUKKUND SUNJII > wrote: >> Greetings, >> >> I am a master?s student in TU Delft, The Netherlands, and I am carrying out my thesis using the adaptive mesh refinement solver provided in ./ts/tutorials/ex11.c. >> >> In the process of my work, I have added a HLL Riemann Solver for the shallow water equation. I had already tested and validated the Riemann Solver using several cases (I am happy to provide a copy of my thesis report). It would indeed be nice to see my work reflected in the current version of ex11.c as an addition to its capability. >> >> Furthermore, I had also added a Riemann Solver that contains topography terms and is well-balanced*. The caveat is that I see that there some variations in the well-balancedness when choosing between p4est and Plex DM?s. Perhaps, this could be looked at and fixed in the future. >> >> The purpose of this email is to propose the addition of the HLL Riemann Solver for the shallow water equations to ex11.c and perhaps later explore the possibility of adding the Riemann Solver with the bathymetry terms. >> >> I have already gone through the CI process instituted by PETSc (previously integrated ./ts/tutorials/ex54.c into the repository with the valuable assistance of Mr. Satish Balay). Hence, I would be more or less familiar with the process. >> >> I kindly ask you provide any feedback on this proposition. >> >> Yes, this sounds great to me. >> >> Thanks, >> >> Matt >> >> Regards, >> >> Mukkund Sunjii >> >> >> -- >> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. >> -- Norbert Wiener >> >> https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From eda.oktay at metu.edu.tr Wed Jul 1 05:34:22 2020 From: eda.oktay at metu.edu.tr (Eda Oktay) Date: Wed, 1 Jul 2020 13:34:22 +0300 Subject: [petsc-users] MatGetRow for global rows of a parallel matrix In-Reply-To: References: Message-ID: Der Barry, ? am trying to use your way but I couldn't understand how I create sub matrices by using MatCreateSubMatrices() since as input, the function needs const IS instead of IS and I couldn't understand how to get a const IS. I tried to use ISCreateStride since this IS should be 0:71 because as you mentioned, the sub matrix should consist of the entire matrix. However, since ISCreateStride produces IS, not const IS, I couldn't use it in MatCreateSubMatrices(). Thanks Eda Barry Smith , 10 Haz 2020 ?ar, 19:59 tarihinde ?unu yazd?: > > > You can use MatCreateSubMatrices() with each process getting a single sequential dense sub matrix that consists of the entire matrix. > > Use VecDuplicateVecs() to create an array of 72 vectors (create a single seq vector of size 4 as the input to this routine) > > Then use MatDenseGetArrayRead() to access the upper left corner of the new sequential dense matrix > > Loop over the vectors calling VecGetArray() > Then loop over the row of the dense array filling up the vector > > Because dense matrices are stored by column, you have to do this looping to fill up the vectors, they can't share the space with the matrix. > > Barry > > > > On Jun 10, 2020, at 11:36 AM, Matthew Knepley wrote: > > On Wed, Jun 10, 2020 at 12:26 PM Eda Oktay wrote: >> >> Matthew Knepley , 10 Haz 2020 ?ar, 19:13 tarihinde >> ?unu yazd?: >> > >> > On Wed, Jun 10, 2020 at 12:07 PM Eda Oktay wrote: >> >> >> >> Der Matt, >> >> >> >> When I looked at the results, I found that there are some problems I >> >> couldn't understand. >> >> >> >> First of all, I am working on a 72*4 matrix and as I said before, I >> >> want to have 72 different vectors having size 4 each, whose elements >> >> consist of the elements in the same row. And of course, all vectors >> >> should be in all processors (currently I am using 4 processors). >> >> >> >> When I use your scatter code, the output vector is divided into 4 >> >> parts for 4 processors and each vector consists of 18 row vectors >> >> whose elements are arranged in a way that if I want to find zeroth row >> >> vector, its elements are located in 0th,18th,36th,54th elements. >> > >> > >> > Was the global size of the vector you wrapped around the dense matrix 72*4? >> >> Yes it is. I set up its global size to 72*4. >> >> > >> > If you use CreateToAll(), it will make a vector on each process which has the global size of the original vector. >> >> Although I set 72*4, the size of the vectors in each process is 72. > > > You can understand how it is hard to accept, as this code is tested every night. Can you VecView() the input vector > to CreateToAll and the output vector, and send that output? > > Thanks, > > Matt > >> >> Thanks, >> >> Eda >> >> > >> > Thanks, >> > >> > Matt >> > >> >> >> >> So, isn't scatter's goal is to scatter all values to all processors? >> >> >> >> Furthermore, I am trying to use my vectors in that way but isn't there >> >> any possible way that I can reach my goal entirely? >> >> >> >> Thanks so much for your help, >> >> >> >> Eda >> >> >> >> Matthew Knepley , 10 Haz 2020 ?ar, 18:11 tarihinde >> >> ?unu yazd?: >> >> > >> >> > On Wed, Jun 10, 2020 at 10:09 AM Eda Oktay wrote: >> >> >> >> >> >> Dear Matt, >> >> >> >> >> >> I have one last question I believe. Up to creating a dense matrix I >> >> >> did what you've suggested. Thank you so much for that. >> >> >> >> >> >> I created a new dense matrix. Now, how should I wrap each vector in a >> >> >> MatDense again? I mean, what is wrapping vectors in a matrix? To put >> >> >> each of them again as rows? >> >> > >> >> > >> >> > I thought you need a dense matrix for something, since you started with one. If you >> >> > do not, just do VecGetArray() on the vector from CreateToAll and use the values. >> >> > >> >> > Thanks, >> >> > >> >> > Matt >> >> > >> >> >> >> >> >> Thanks! >> >> >> >> >> >> Eda >> >> >> >> >> >> Matthew Knepley , 10 Haz 2020 ?ar, 16:16 tarihinde >> >> >> ?unu yazd?: >> >> >> > >> >> >> > On Wed, Jun 10, 2020 at 9:08 AM Eda Oktay wrote: >> >> >> >> >> >> >> >> Dear Matt, >> >> >> >> >> >> >> >> Matthew Knepley , 10 Haz 2020 ?ar, 16:03 tarihinde >> >> >> >> ?unu yazd?: >> >> >> >> > >> >> >> >> > On Wed, Jun 10, 2020 at 8:56 AM Eda Oktay wrote: >> >> >> >> >> >> >> >> >> >> Hi all, >> >> >> >> >> >> >> >> >> >> I am trying to get all the rows of a parallel matrix as individual >> >> >> >> >> vectors. For instance, if I have 72*4 matrix, I want to get 72 >> >> >> >> >> different vectors having size 4. >> >> >> >> >> >> >> >> >> >> As far as I understood, MatGetRow is only for local rows, so >> >> >> >> >> MatGetOwnershipRange is used, however, when I tried this one, I >> >> >> >> >> couldn't get the whole and desired row vectors. >> >> >> >> >> >> >> >> >> >> In MatGetRow explanation, it is written that I should use >> >> >> >> >> MatCreateSubMatrices first, then use MatGetRow. But I couldn't >> >> >> >> >> understand to which extent I should create submatrices. I just need to >> >> >> >> >> have all 72 rows as 72 different vectors each having 4 elements. >> >> >> >> > >> >> >> >> > >> >> >> >> > 1) For sparse matrices, the storage is always divided by row, so that values can only be retrieved for local rows with MatGetRow() >> >> >> >> > >> >> >> >> > 2) Is this matrix sparse? It sounds like it is dense. >> >> >> >> >> >> >> >> Matrix is dense. >> >> >> >> >> >> >> >> > >> >> >> >> > 3) Are you asking to get all matrix values on all processes? If so, I think the easiest thing to do is first wrap a Vec around the >> >> >> >> > values, then use VecScatterToAll(), then wrap each one in a MatDense again. >> >> >> >> >> >> >> >> Yes, I want all row vectors on all processes. In a dense matrix, >> >> >> >> should I still wrap a Vec around the values? I know I should use >> >> >> >> scatter but I couldn't even wrap a Vec around them. >> >> >> > >> >> >> > >> >> >> > I would do >> >> >> > >> >> >> > MatGetSize(&N); >> >> >> > MatGetLocalSize(&m); >> >> >> > https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MatDenseGetArray.html >> >> >> > >> >> >> > https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Vec/VecPlaceArray.html >> >> >> > https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Vec/VecScatterCreateToAll.html >> >> >> > >> >> >> > https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Vec/VecResetArray.html#VecResetArray >> >> >> > https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MatCreateDense.html >> >> >> > >> >> >> > >> >> >> > >> >> >> > >> >> >> > Thanks, >> >> >> > >> >> >> > Matt >> >> >> > >> >> >> >> >> >> >> >> Thanks so much! >> >> >> >> >> >> >> >> Eda >> >> >> >> >> >> >> >> > >> >> >> >> > Thanks, >> >> >> >> > >> >> >> >> > Matt >> >> >> >> > >> >> >> >> >> >> >> >> >> >> Thanks! >> >> >> >> >> >> >> >> >> >> Eda >> >> >> >> > >> >> >> >> > >> >> >> >> > >> >> >> >> > -- >> >> >> >> > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. >> >> >> >> > -- Norbert Wiener >> >> >> >> > >> >> >> >> > https://www.cse.buffalo.edu/~knepley/ >> >> >> > >> >> >> > >> >> >> > >> >> >> > -- >> >> >> > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. >> >> >> > -- Norbert Wiener >> >> >> > >> >> >> > https://www.cse.buffalo.edu/~knepley/ >> >> > >> >> > >> >> > >> >> > -- >> >> > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. >> >> > -- Norbert Wiener >> >> > >> >> > https://www.cse.buffalo.edu/~knepley/ >> > >> > >> > >> > -- >> > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. >> > -- Norbert Wiener >> > >> > https://www.cse.buffalo.edu/~knepley/ > > > > -- > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ > > From knepley at gmail.com Wed Jul 1 05:43:37 2020 From: knepley at gmail.com (Matthew Knepley) Date: Wed, 1 Jul 2020 06:43:37 -0400 Subject: [petsc-users] MatGetRow for global rows of a parallel matrix In-Reply-To: References: Message-ID: On Wed, Jul 1, 2020 at 6:34 AM Eda Oktay wrote: > Der Barry, > > ? am trying to use your way but I couldn't understand how I create sub > matrices by using MatCreateSubMatrices() since as input, the function > needs const IS instead of IS and I couldn't understand how to get a > const IS. I tried to use ISCreateStride since this IS should be 0:71 > because as you mentioned, the sub matrix should consist of the entire > matrix. However, since ISCreateStride produces IS, not const IS, I > couldn't use it in MatCreateSubMatrices(). > The 'const' refers to the array, not the IS: https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MatCreateSubMatrices.html Thanks, Matt > Thanks > > Eda > > Barry Smith , 10 Haz 2020 ?ar, 19:59 tarihinde ?unu > yazd?: > > > > > > You can use MatCreateSubMatrices() with each process getting a single > sequential dense sub matrix that consists of the entire matrix. > > > > Use VecDuplicateVecs() to create an array of 72 vectors (create a > single seq vector of size 4 as the input to this routine) > > > > Then use MatDenseGetArrayRead() to access the upper left corner of the > new sequential dense matrix > > > > Loop over the vectors calling VecGetArray() > > Then loop over the row of the dense array filling up the vector > > > > Because dense matrices are stored by column, you have to do this > looping to fill up the vectors, they can't share the space with the matrix. > > > > Barry > > > > > > > > On Jun 10, 2020, at 11:36 AM, Matthew Knepley wrote: > > > > On Wed, Jun 10, 2020 at 12:26 PM Eda Oktay > wrote: > >> > >> Matthew Knepley , 10 Haz 2020 ?ar, 19:13 tarihinde > >> ?unu yazd?: > >> > > >> > On Wed, Jun 10, 2020 at 12:07 PM Eda Oktay > wrote: > >> >> > >> >> Der Matt, > >> >> > >> >> When I looked at the results, I found that there are some problems I > >> >> couldn't understand. > >> >> > >> >> First of all, I am working on a 72*4 matrix and as I said before, I > >> >> want to have 72 different vectors having size 4 each, whose elements > >> >> consist of the elements in the same row. And of course, all vectors > >> >> should be in all processors (currently I am using 4 processors). > >> >> > >> >> When I use your scatter code, the output vector is divided into 4 > >> >> parts for 4 processors and each vector consists of 18 row vectors > >> >> whose elements are arranged in a way that if I want to find zeroth > row > >> >> vector, its elements are located in 0th,18th,36th,54th elements. > >> > > >> > > >> > Was the global size of the vector you wrapped around the dense matrix > 72*4? > >> > >> Yes it is. I set up its global size to 72*4. > >> > >> > > >> > If you use CreateToAll(), it will make a vector on each process which > has the global size of the original vector. > >> > >> Although I set 72*4, the size of the vectors in each process is 72. > > > > > > You can understand how it is hard to accept, as this code is tested > every night. Can you VecView() the input vector > > to CreateToAll and the output vector, and send that output? > > > > Thanks, > > > > Matt > > > >> > >> Thanks, > >> > >> Eda > >> > >> > > >> > Thanks, > >> > > >> > Matt > >> > > >> >> > >> >> So, isn't scatter's goal is to scatter all values to all processors? > >> >> > >> >> Furthermore, I am trying to use my vectors in that way but isn't > there > >> >> any possible way that I can reach my goal entirely? > >> >> > >> >> Thanks so much for your help, > >> >> > >> >> Eda > >> >> > >> >> Matthew Knepley , 10 Haz 2020 ?ar, 18:11 > tarihinde > >> >> ?unu yazd?: > >> >> > > >> >> > On Wed, Jun 10, 2020 at 10:09 AM Eda Oktay > wrote: > >> >> >> > >> >> >> Dear Matt, > >> >> >> > >> >> >> I have one last question I believe. Up to creating a dense matrix > I > >> >> >> did what you've suggested. Thank you so much for that. > >> >> >> > >> >> >> I created a new dense matrix. Now, how should I wrap each vector > in a > >> >> >> MatDense again? I mean, what is wrapping vectors in a matrix? To > put > >> >> >> each of them again as rows? > >> >> > > >> >> > > >> >> > I thought you need a dense matrix for something, since you started > with one. If you > >> >> > do not, just do VecGetArray() on the vector from CreateToAll and > use the values. > >> >> > > >> >> > Thanks, > >> >> > > >> >> > Matt > >> >> > > >> >> >> > >> >> >> Thanks! > >> >> >> > >> >> >> Eda > >> >> >> > >> >> >> Matthew Knepley , 10 Haz 2020 ?ar, 16:16 > tarihinde > >> >> >> ?unu yazd?: > >> >> >> > > >> >> >> > On Wed, Jun 10, 2020 at 9:08 AM Eda Oktay < > eda.oktay at metu.edu.tr> wrote: > >> >> >> >> > >> >> >> >> Dear Matt, > >> >> >> >> > >> >> >> >> Matthew Knepley , 10 Haz 2020 ?ar, 16:03 > tarihinde > >> >> >> >> ?unu yazd?: > >> >> >> >> > > >> >> >> >> > On Wed, Jun 10, 2020 at 8:56 AM Eda Oktay < > eda.oktay at metu.edu.tr> wrote: > >> >> >> >> >> > >> >> >> >> >> Hi all, > >> >> >> >> >> > >> >> >> >> >> I am trying to get all the rows of a parallel matrix as > individual > >> >> >> >> >> vectors. For instance, if I have 72*4 matrix, I want to get > 72 > >> >> >> >> >> different vectors having size 4. > >> >> >> >> >> > >> >> >> >> >> As far as I understood, MatGetRow is only for local rows, so > >> >> >> >> >> MatGetOwnershipRange is used, however, when I tried this > one, I > >> >> >> >> >> couldn't get the whole and desired row vectors. > >> >> >> >> >> > >> >> >> >> >> In MatGetRow explanation, it is written that I should use > >> >> >> >> >> MatCreateSubMatrices first, then use MatGetRow. But I > couldn't > >> >> >> >> >> understand to which extent I should create submatrices. I > just need to > >> >> >> >> >> have all 72 rows as 72 different vectors each having 4 > elements. > >> >> >> >> > > >> >> >> >> > > >> >> >> >> > 1) For sparse matrices, the storage is always divided by > row, so that values can only be retrieved for local rows with MatGetRow() > >> >> >> >> > > >> >> >> >> > 2) Is this matrix sparse? It sounds like it is dense. > >> >> >> >> > >> >> >> >> Matrix is dense. > >> >> >> >> > >> >> >> >> > > >> >> >> >> > 3) Are you asking to get all matrix values on all processes? > If so, I think the easiest thing to do is first wrap a Vec around the > >> >> >> >> > values, then use VecScatterToAll(), then wrap each one > in a MatDense again. > >> >> >> >> > >> >> >> >> Yes, I want all row vectors on all processes. In a dense > matrix, > >> >> >> >> should I still wrap a Vec around the values? I know I should > use > >> >> >> >> scatter but I couldn't even wrap a Vec around them. > >> >> >> > > >> >> >> > > >> >> >> > I would do > >> >> >> > > >> >> >> > MatGetSize(&N); > >> >> >> > MatGetLocalSize(&m); > >> >> >> > > https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MatDenseGetArray.html > >> >> >> > > >> >> >> > > https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Vec/VecPlaceArray.html > >> >> >> > > https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Vec/VecScatterCreateToAll.html > >> >> >> > > >> >> >> > > https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Vec/VecResetArray.html#VecResetArray > >> >> >> > > https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MatCreateDense.html > >> >> >> > > >> >> >> > > >> >> >> > > >> >> >> > > >> >> >> > Thanks, > >> >> >> > > >> >> >> > Matt > >> >> >> > > >> >> >> >> > >> >> >> >> Thanks so much! > >> >> >> >> > >> >> >> >> Eda > >> >> >> >> > >> >> >> >> > > >> >> >> >> > Thanks, > >> >> >> >> > > >> >> >> >> > Matt > >> >> >> >> > > >> >> >> >> >> > >> >> >> >> >> Thanks! > >> >> >> >> >> > >> >> >> >> >> Eda > >> >> >> >> > > >> >> >> >> > > >> >> >> >> > > >> >> >> >> > -- > >> >> >> >> > What most experimenters take for granted before they begin > their experiments is infinitely more interesting than any results to which > their experiments lead. > >> >> >> >> > -- Norbert Wiener > >> >> >> >> > > >> >> >> >> > https://www.cse.buffalo.edu/~knepley/ > >> >> >> > > >> >> >> > > >> >> >> > > >> >> >> > -- > >> >> >> > What most experimenters take for granted before they begin > their experiments is infinitely more interesting than any results to which > their experiments lead. > >> >> >> > -- Norbert Wiener > >> >> >> > > >> >> >> > https://www.cse.buffalo.edu/~knepley/ > >> >> > > >> >> > > >> >> > > >> >> > -- > >> >> > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > >> >> > -- Norbert Wiener > >> >> > > >> >> > https://www.cse.buffalo.edu/~knepley/ > >> > > >> > > >> > > >> > -- > >> > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > >> > -- Norbert Wiener > >> > > >> > https://www.cse.buffalo.edu/~knepley/ > > > > > > > > -- > > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > > -- Norbert Wiener > > > > https://www.cse.buffalo.edu/~knepley/ > > > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From eda.oktay at metu.edu.tr Wed Jul 1 06:32:16 2020 From: eda.oktay at metu.edu.tr (Eda Oktay) Date: Wed, 1 Jul 2020 14:32:16 +0300 Subject: [petsc-users] MatGetRow for global rows of a parallel matrix In-Reply-To: References: Message-ID: I'm sorry but I still can't understand how to put an input as these parameters. It is the first time I saw 'const' for IS. If it doesn't refer IS, then as a parameter for row and column indices, how should I give them as a valid input? It still should be an IS, right? Lastly, I tried to use in the form: IS idUi; ISCreateStride(PETSC_COMM_WORLD,siz,0,1,&idUi); Mat *submat; MatCreateSubMatrices(U,nev,&idUi,&idUi,MAT_INITIAL_MATRIX,&submat); where nev = 4, siz = 72 and U is 72*4 matrix. But then, the error becomes about the matrix, U: [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [0]PETSC ERROR: [1]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [1]PETSC ERROR: Invalid argument [1]PETSC ERROR: Wrong type of object: Parameter # 1 Invalid argument [0]PETSC ERROR: Wrong type of object: Parameter # 1 [0]PETSC ERROR: See https://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [0]PETSC ERROR: Petsc Release Version 3.13.2, Jun 02, 2020 [0]PETSC ERROR: ./deneme_new_vecscatter_arastep on a arch-linux2-c-debug named b342.wls.metu.edu.tr by edaoktay Wed Jul 1 14:25:00 2020 [0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ --download-mpich --download-openblas --download-slepc --download-metis --download-parmetis --download-chaco --with-X=1 [1]PETSC ERROR: See https://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [1]PETSC ERROR: Petsc Release Version 3.13.2, Jun 02, 2020 [1]PETSC ERROR: [0]PETSC ERROR: #1 ISSorted() line 1777 in /home/edaoktay/petsc-3.13.2/src/vec/is/is/interface/index.c [0]PETSC ERROR: ./deneme_new_vecscatter_arastep on a arch-linux2-c-debug named b342.wls.metu.edu.tr by edaoktay Wed Jul 1 14:25:00 2020 [1]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ --download-mpich --download-openblas --download-slepc --download-metis --download-parmetis --download-chaco --with-X=1 [1]PETSC ERROR: #2 MatCreateSubMatrices_MPIDense_Local() line 104 in /home/edaoktay/petsc-3.13.2/src/mat/impls/dense/mpi/mmdense.c [0]PETSC ERROR: #3 MatCreateSubMatrices_MPIDense() line 66 in /home/edaoktay/petsc-3.13.2/src/mat/impls/dense/mpi/mmdense.c #1 ISSorted() line 1777 in /home/edaoktay/petsc-3.13.2/src/vec/is/is/interface/index.c [1]PETSC ERROR: #2 MatCreateSubMatrices_MPIDense_Local() line 104 in /home/edaoktay/petsc-3.13.2/src/mat/impls/dense/mpi/mmdense.c [0]PETSC ERROR: #4 MatCreateSubMatrices() line 6758 in /home/edaoktay/petsc-3.13.2/src/mat/interface/matrix.c [1]PETSC ERROR: #3 MatCreateSubMatrices_MPIDense() line 66 in /home/edaoktay/petsc-3.13.2/src/mat/impls/dense/mpi/mmdense.c [1]PETSC ERROR: #4 MatCreateSubMatrices() line 6758 in /home/edaoktay/petsc-3.13.2/src/mat/interface/matrix.c Before I used &idUi, I tried to use just idUi (without & sign) and it didn't work, so I decided to try using this way, since may be 'const' refers this one. But how can my matrix be of wrong type? It is MPI Dense. Thanks! Eda Matthew Knepley , 1 Tem 2020 ?ar, 13:43 tarihinde ?unu yazd?: > > On Wed, Jul 1, 2020 at 6:34 AM Eda Oktay wrote: >> >> Der Barry, >> >> ? am trying to use your way but I couldn't understand how I create sub >> matrices by using MatCreateSubMatrices() since as input, the function >> needs const IS instead of IS and I couldn't understand how to get a >> const IS. I tried to use ISCreateStride since this IS should be 0:71 >> because as you mentioned, the sub matrix should consist of the entire >> matrix. However, since ISCreateStride produces IS, not const IS, I >> couldn't use it in MatCreateSubMatrices(). > > > The 'const' refers to the array, not the IS: > > https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MatCreateSubMatrices.html > > Thanks, > > Matt > >> >> Thanks >> >> Eda >> >> Barry Smith , 10 Haz 2020 ?ar, 19:59 tarihinde ?unu yazd?: >> > >> > >> > You can use MatCreateSubMatrices() with each process getting a single sequential dense sub matrix that consists of the entire matrix. >> > >> > Use VecDuplicateVecs() to create an array of 72 vectors (create a single seq vector of size 4 as the input to this routine) >> > >> > Then use MatDenseGetArrayRead() to access the upper left corner of the new sequential dense matrix >> > >> > Loop over the vectors calling VecGetArray() >> > Then loop over the row of the dense array filling up the vector >> > >> > Because dense matrices are stored by column, you have to do this looping to fill up the vectors, they can't share the space with the matrix. >> > >> > Barry >> > >> > >> > >> > On Jun 10, 2020, at 11:36 AM, Matthew Knepley wrote: >> > >> > On Wed, Jun 10, 2020 at 12:26 PM Eda Oktay wrote: >> >> >> >> Matthew Knepley , 10 Haz 2020 ?ar, 19:13 tarihinde >> >> ?unu yazd?: >> >> > >> >> > On Wed, Jun 10, 2020 at 12:07 PM Eda Oktay wrote: >> >> >> >> >> >> Der Matt, >> >> >> >> >> >> When I looked at the results, I found that there are some problems I >> >> >> couldn't understand. >> >> >> >> >> >> First of all, I am working on a 72*4 matrix and as I said before, I >> >> >> want to have 72 different vectors having size 4 each, whose elements >> >> >> consist of the elements in the same row. And of course, all vectors >> >> >> should be in all processors (currently I am using 4 processors). >> >> >> >> >> >> When I use your scatter code, the output vector is divided into 4 >> >> >> parts for 4 processors and each vector consists of 18 row vectors >> >> >> whose elements are arranged in a way that if I want to find zeroth row >> >> >> vector, its elements are located in 0th,18th,36th,54th elements. >> >> > >> >> > >> >> > Was the global size of the vector you wrapped around the dense matrix 72*4? >> >> >> >> Yes it is. I set up its global size to 72*4. >> >> >> >> > >> >> > If you use CreateToAll(), it will make a vector on each process which has the global size of the original vector. >> >> >> >> Although I set 72*4, the size of the vectors in each process is 72. >> > >> > >> > You can understand how it is hard to accept, as this code is tested every night. Can you VecView() the input vector >> > to CreateToAll and the output vector, and send that output? >> > >> > Thanks, >> > >> > Matt >> > >> >> >> >> Thanks, >> >> >> >> Eda >> >> >> >> > >> >> > Thanks, >> >> > >> >> > Matt >> >> > >> >> >> >> >> >> So, isn't scatter's goal is to scatter all values to all processors? >> >> >> >> >> >> Furthermore, I am trying to use my vectors in that way but isn't there >> >> >> any possible way that I can reach my goal entirely? >> >> >> >> >> >> Thanks so much for your help, >> >> >> >> >> >> Eda >> >> >> >> >> >> Matthew Knepley , 10 Haz 2020 ?ar, 18:11 tarihinde >> >> >> ?unu yazd?: >> >> >> > >> >> >> > On Wed, Jun 10, 2020 at 10:09 AM Eda Oktay wrote: >> >> >> >> >> >> >> >> Dear Matt, >> >> >> >> >> >> >> >> I have one last question I believe. Up to creating a dense matrix I >> >> >> >> did what you've suggested. Thank you so much for that. >> >> >> >> >> >> >> >> I created a new dense matrix. Now, how should I wrap each vector in a >> >> >> >> MatDense again? I mean, what is wrapping vectors in a matrix? To put >> >> >> >> each of them again as rows? >> >> >> > >> >> >> > >> >> >> > I thought you need a dense matrix for something, since you started with one. If you >> >> >> > do not, just do VecGetArray() on the vector from CreateToAll and use the values. >> >> >> > >> >> >> > Thanks, >> >> >> > >> >> >> > Matt >> >> >> > >> >> >> >> >> >> >> >> Thanks! >> >> >> >> >> >> >> >> Eda >> >> >> >> >> >> >> >> Matthew Knepley , 10 Haz 2020 ?ar, 16:16 tarihinde >> >> >> >> ?unu yazd?: >> >> >> >> > >> >> >> >> > On Wed, Jun 10, 2020 at 9:08 AM Eda Oktay wrote: >> >> >> >> >> >> >> >> >> >> Dear Matt, >> >> >> >> >> >> >> >> >> >> Matthew Knepley , 10 Haz 2020 ?ar, 16:03 tarihinde >> >> >> >> >> ?unu yazd?: >> >> >> >> >> > >> >> >> >> >> > On Wed, Jun 10, 2020 at 8:56 AM Eda Oktay wrote: >> >> >> >> >> >> >> >> >> >> >> >> Hi all, >> >> >> >> >> >> >> >> >> >> >> >> I am trying to get all the rows of a parallel matrix as individual >> >> >> >> >> >> vectors. For instance, if I have 72*4 matrix, I want to get 72 >> >> >> >> >> >> different vectors having size 4. >> >> >> >> >> >> >> >> >> >> >> >> As far as I understood, MatGetRow is only for local rows, so >> >> >> >> >> >> MatGetOwnershipRange is used, however, when I tried this one, I >> >> >> >> >> >> couldn't get the whole and desired row vectors. >> >> >> >> >> >> >> >> >> >> >> >> In MatGetRow explanation, it is written that I should use >> >> >> >> >> >> MatCreateSubMatrices first, then use MatGetRow. But I couldn't >> >> >> >> >> >> understand to which extent I should create submatrices. I just need to >> >> >> >> >> >> have all 72 rows as 72 different vectors each having 4 elements. >> >> >> >> >> > >> >> >> >> >> > >> >> >> >> >> > 1) For sparse matrices, the storage is always divided by row, so that values can only be retrieved for local rows with MatGetRow() >> >> >> >> >> > >> >> >> >> >> > 2) Is this matrix sparse? It sounds like it is dense. >> >> >> >> >> >> >> >> >> >> Matrix is dense. >> >> >> >> >> >> >> >> >> >> > >> >> >> >> >> > 3) Are you asking to get all matrix values on all processes? If so, I think the easiest thing to do is first wrap a Vec around the >> >> >> >> >> > values, then use VecScatterToAll(), then wrap each one in a MatDense again. >> >> >> >> >> >> >> >> >> >> Yes, I want all row vectors on all processes. In a dense matrix, >> >> >> >> >> should I still wrap a Vec around the values? I know I should use >> >> >> >> >> scatter but I couldn't even wrap a Vec around them. >> >> >> >> > >> >> >> >> > >> >> >> >> > I would do >> >> >> >> > >> >> >> >> > MatGetSize(&N); >> >> >> >> > MatGetLocalSize(&m); >> >> >> >> > https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MatDenseGetArray.html >> >> >> >> > >> >> >> >> > https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Vec/VecPlaceArray.html >> >> >> >> > https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Vec/VecScatterCreateToAll.html >> >> >> >> > >> >> >> >> > https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Vec/VecResetArray.html#VecResetArray >> >> >> >> > https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MatCreateDense.html >> >> >> >> > >> >> >> >> > >> >> >> >> > >> >> >> >> > >> >> >> >> > Thanks, >> >> >> >> > >> >> >> >> > Matt >> >> >> >> > >> >> >> >> >> >> >> >> >> >> Thanks so much! >> >> >> >> >> >> >> >> >> >> Eda >> >> >> >> >> >> >> >> >> >> > >> >> >> >> >> > Thanks, >> >> >> >> >> > >> >> >> >> >> > Matt >> >> >> >> >> > >> >> >> >> >> >> >> >> >> >> >> >> Thanks! >> >> >> >> >> >> >> >> >> >> >> >> Eda >> >> >> >> >> > >> >> >> >> >> > >> >> >> >> >> > >> >> >> >> >> > -- >> >> >> >> >> > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. >> >> >> >> >> > -- Norbert Wiener >> >> >> >> >> > >> >> >> >> >> > https://www.cse.buffalo.edu/~knepley/ >> >> >> >> > >> >> >> >> > >> >> >> >> > >> >> >> >> > -- >> >> >> >> > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. >> >> >> >> > -- Norbert Wiener >> >> >> >> > >> >> >> >> > https://www.cse.buffalo.edu/~knepley/ >> >> >> > >> >> >> > >> >> >> > >> >> >> > -- >> >> >> > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. >> >> >> > -- Norbert Wiener >> >> >> > >> >> >> > https://www.cse.buffalo.edu/~knepley/ >> >> > >> >> > >> >> > >> >> > -- >> >> > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. >> >> > -- Norbert Wiener >> >> > >> >> > https://www.cse.buffalo.edu/~knepley/ >> > >> > >> > >> > -- >> > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. >> > -- Norbert Wiener >> > >> > https://www.cse.buffalo.edu/~knepley/ >> > >> > > > > > -- > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ From knepley at gmail.com Wed Jul 1 06:41:47 2020 From: knepley at gmail.com (Matthew Knepley) Date: Wed, 1 Jul 2020 07:41:47 -0400 Subject: [petsc-users] MatGetRow for global rows of a parallel matrix In-Reply-To: References: Message-ID: On Wed, Jul 1, 2020 at 7:32 AM Eda Oktay wrote: > I'm sorry but I still can't understand how to put an input as these > parameters. It is the first time I saw 'const' for IS. If it doesn't > refer IS, then as a parameter for row and column indices, how should I > give them as a valid input? It still should be an IS, right? > > Lastly, I tried to use in the form: > > IS idUi; > ISCreateStride(PETSC_COMM_WORLD,siz,0,1,&idUi); > Mat *submat; > MatCreateSubMatrices(U,nev,&idUi,&idUi,MAT_INITIAL_MATRIX,&submat); > You gave n = 4 (you call it nev), but you only have 1 rowIS. You are supposed to give an array of 4. Thanks, Matt > where nev = 4, siz = 72 and U is 72*4 matrix. > > But then, the error becomes about the matrix, U: > > [0]PETSC ERROR: --------------------- Error Message > -------------------------------------------------------------- > [0]PETSC ERROR: [1]PETSC ERROR: --------------------- Error Message > -------------------------------------------------------------- > [1]PETSC ERROR: Invalid argument > [1]PETSC ERROR: Wrong type of object: Parameter # 1 > Invalid argument > [0]PETSC ERROR: Wrong type of object: Parameter # 1 > [0]PETSC ERROR: See > https://www.mcs.anl.gov/petsc/documentation/faq.html for trouble > shooting. > [0]PETSC ERROR: Petsc Release Version 3.13.2, Jun 02, 2020 > [0]PETSC ERROR: ./deneme_new_vecscatter_arastep on a > arch-linux2-c-debug named b342.wls.metu.edu.tr by edaoktay Wed Jul 1 > 14:25:00 2020 > [0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ > --download-mpich --download-openblas --download-slepc --download-metis > --download-parmetis --download-chaco --with-X=1 > [1]PETSC ERROR: See > https://www.mcs.anl.gov/petsc/documentation/faq.html for trouble > shooting. > [1]PETSC ERROR: Petsc Release Version 3.13.2, Jun 02, 2020 > [1]PETSC ERROR: [0]PETSC ERROR: #1 ISSorted() line 1777 in > /home/edaoktay/petsc-3.13.2/src/vec/is/is/interface/index.c > [0]PETSC ERROR: ./deneme_new_vecscatter_arastep on a > arch-linux2-c-debug named b342.wls.metu.edu.tr by edaoktay Wed Jul 1 > 14:25:00 2020 > [1]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ > --download-mpich --download-openblas --download-slepc --download-metis > --download-parmetis --download-chaco --with-X=1 > [1]PETSC ERROR: #2 MatCreateSubMatrices_MPIDense_Local() line 104 in > /home/edaoktay/petsc-3.13.2/src/mat/impls/dense/mpi/mmdense.c > [0]PETSC ERROR: #3 MatCreateSubMatrices_MPIDense() line 66 in > /home/edaoktay/petsc-3.13.2/src/mat/impls/dense/mpi/mmdense.c > #1 ISSorted() line 1777 in > /home/edaoktay/petsc-3.13.2/src/vec/is/is/interface/index.c > [1]PETSC ERROR: #2 MatCreateSubMatrices_MPIDense_Local() line 104 in > /home/edaoktay/petsc-3.13.2/src/mat/impls/dense/mpi/mmdense.c > [0]PETSC ERROR: #4 MatCreateSubMatrices() line 6758 in > /home/edaoktay/petsc-3.13.2/src/mat/interface/matrix.c > [1]PETSC ERROR: #3 MatCreateSubMatrices_MPIDense() line 66 in > /home/edaoktay/petsc-3.13.2/src/mat/impls/dense/mpi/mmdense.c > [1]PETSC ERROR: #4 MatCreateSubMatrices() line 6758 in > /home/edaoktay/petsc-3.13.2/src/mat/interface/matrix.c > > Before I used &idUi, I tried to use just idUi (without & sign) and it > didn't work, so I decided to try using this way, since may be 'const' > refers this one. But how can my matrix be of wrong type? It is MPI > Dense. > > Thanks! > > Eda > > Matthew Knepley , 1 Tem 2020 ?ar, 13:43 tarihinde ?unu > yazd?: > > > > On Wed, Jul 1, 2020 at 6:34 AM Eda Oktay wrote: > >> > >> Der Barry, > >> > >> ? am trying to use your way but I couldn't understand how I create sub > >> matrices by using MatCreateSubMatrices() since as input, the function > >> needs const IS instead of IS and I couldn't understand how to get a > >> const IS. I tried to use ISCreateStride since this IS should be 0:71 > >> because as you mentioned, the sub matrix should consist of the entire > >> matrix. However, since ISCreateStride produces IS, not const IS, I > >> couldn't use it in MatCreateSubMatrices(). > > > > > > The 'const' refers to the array, not the IS: > > > > > https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MatCreateSubMatrices.html > > > > Thanks, > > > > Matt > > > >> > >> Thanks > >> > >> Eda > >> > >> Barry Smith , 10 Haz 2020 ?ar, 19:59 tarihinde ?unu > yazd?: > >> > > >> > > >> > You can use MatCreateSubMatrices() with each process getting a > single sequential dense sub matrix that consists of the entire matrix. > >> > > >> > Use VecDuplicateVecs() to create an array of 72 vectors (create a > single seq vector of size 4 as the input to this routine) > >> > > >> > Then use MatDenseGetArrayRead() to access the upper left corner of > the new sequential dense matrix > >> > > >> > Loop over the vectors calling VecGetArray() > >> > Then loop over the row of the dense array filling up the vector > >> > > >> > Because dense matrices are stored by column, you have to do this > looping to fill up the vectors, they can't share the space with the matrix. > >> > > >> > Barry > >> > > >> > > >> > > >> > On Jun 10, 2020, at 11:36 AM, Matthew Knepley > wrote: > >> > > >> > On Wed, Jun 10, 2020 at 12:26 PM Eda Oktay > wrote: > >> >> > >> >> Matthew Knepley , 10 Haz 2020 ?ar, 19:13 > tarihinde > >> >> ?unu yazd?: > >> >> > > >> >> > On Wed, Jun 10, 2020 at 12:07 PM Eda Oktay > wrote: > >> >> >> > >> >> >> Der Matt, > >> >> >> > >> >> >> When I looked at the results, I found that there are some > problems I > >> >> >> couldn't understand. > >> >> >> > >> >> >> First of all, I am working on a 72*4 matrix and as I said before, > I > >> >> >> want to have 72 different vectors having size 4 each, whose > elements > >> >> >> consist of the elements in the same row. And of course, all > vectors > >> >> >> should be in all processors (currently I am using 4 processors). > >> >> >> > >> >> >> When I use your scatter code, the output vector is divided into 4 > >> >> >> parts for 4 processors and each vector consists of 18 row vectors > >> >> >> whose elements are arranged in a way that if I want to find > zeroth row > >> >> >> vector, its elements are located in 0th,18th,36th,54th elements. > >> >> > > >> >> > > >> >> > Was the global size of the vector you wrapped around the dense > matrix 72*4? > >> >> > >> >> Yes it is. I set up its global size to 72*4. > >> >> > >> >> > > >> >> > If you use CreateToAll(), it will make a vector on each process > which has the global size of the original vector. > >> >> > >> >> Although I set 72*4, the size of the vectors in each process is 72. > >> > > >> > > >> > You can understand how it is hard to accept, as this code is tested > every night. Can you VecView() the input vector > >> > to CreateToAll and the output vector, and send that output? > >> > > >> > Thanks, > >> > > >> > Matt > >> > > >> >> > >> >> Thanks, > >> >> > >> >> Eda > >> >> > >> >> > > >> >> > Thanks, > >> >> > > >> >> > Matt > >> >> > > >> >> >> > >> >> >> So, isn't scatter's goal is to scatter all values to all > processors? > >> >> >> > >> >> >> Furthermore, I am trying to use my vectors in that way but isn't > there > >> >> >> any possible way that I can reach my goal entirely? > >> >> >> > >> >> >> Thanks so much for your help, > >> >> >> > >> >> >> Eda > >> >> >> > >> >> >> Matthew Knepley , 10 Haz 2020 ?ar, 18:11 > tarihinde > >> >> >> ?unu yazd?: > >> >> >> > > >> >> >> > On Wed, Jun 10, 2020 at 10:09 AM Eda Oktay < > eda.oktay at metu.edu.tr> wrote: > >> >> >> >> > >> >> >> >> Dear Matt, > >> >> >> >> > >> >> >> >> I have one last question I believe. Up to creating a dense > matrix I > >> >> >> >> did what you've suggested. Thank you so much for that. > >> >> >> >> > >> >> >> >> I created a new dense matrix. Now, how should I wrap each > vector in a > >> >> >> >> MatDense again? I mean, what is wrapping vectors in a matrix? > To put > >> >> >> >> each of them again as rows? > >> >> >> > > >> >> >> > > >> >> >> > I thought you need a dense matrix for something, since you > started with one. If you > >> >> >> > do not, just do VecGetArray() on the vector from CreateToAll > and use the values. > >> >> >> > > >> >> >> > Thanks, > >> >> >> > > >> >> >> > Matt > >> >> >> > > >> >> >> >> > >> >> >> >> Thanks! > >> >> >> >> > >> >> >> >> Eda > >> >> >> >> > >> >> >> >> Matthew Knepley , 10 Haz 2020 ?ar, 16:16 > tarihinde > >> >> >> >> ?unu yazd?: > >> >> >> >> > > >> >> >> >> > On Wed, Jun 10, 2020 at 9:08 AM Eda Oktay < > eda.oktay at metu.edu.tr> wrote: > >> >> >> >> >> > >> >> >> >> >> Dear Matt, > >> >> >> >> >> > >> >> >> >> >> Matthew Knepley , 10 Haz 2020 ?ar, > 16:03 tarihinde > >> >> >> >> >> ?unu yazd?: > >> >> >> >> >> > > >> >> >> >> >> > On Wed, Jun 10, 2020 at 8:56 AM Eda Oktay < > eda.oktay at metu.edu.tr> wrote: > >> >> >> >> >> >> > >> >> >> >> >> >> Hi all, > >> >> >> >> >> >> > >> >> >> >> >> >> I am trying to get all the rows of a parallel matrix as > individual > >> >> >> >> >> >> vectors. For instance, if I have 72*4 matrix, I want to > get 72 > >> >> >> >> >> >> different vectors having size 4. > >> >> >> >> >> >> > >> >> >> >> >> >> As far as I understood, MatGetRow is only for local > rows, so > >> >> >> >> >> >> MatGetOwnershipRange is used, however, when I tried this > one, I > >> >> >> >> >> >> couldn't get the whole and desired row vectors. > >> >> >> >> >> >> > >> >> >> >> >> >> In MatGetRow explanation, it is written that I should use > >> >> >> >> >> >> MatCreateSubMatrices first, then use MatGetRow. But I > couldn't > >> >> >> >> >> >> understand to which extent I should create submatrices. > I just need to > >> >> >> >> >> >> have all 72 rows as 72 different vectors each having 4 > elements. > >> >> >> >> >> > > >> >> >> >> >> > > >> >> >> >> >> > 1) For sparse matrices, the storage is always divided by > row, so that values can only be retrieved for local rows with MatGetRow() > >> >> >> >> >> > > >> >> >> >> >> > 2) Is this matrix sparse? It sounds like it is dense. > >> >> >> >> >> > >> >> >> >> >> Matrix is dense. > >> >> >> >> >> > >> >> >> >> >> > > >> >> >> >> >> > 3) Are you asking to get all matrix values on all > processes? If so, I think the easiest thing to do is first wrap a Vec > around the > >> >> >> >> >> > values, then use VecScatterToAll(), then wrap each > one in a MatDense again. > >> >> >> >> >> > >> >> >> >> >> Yes, I want all row vectors on all processes. In a dense > matrix, > >> >> >> >> >> should I still wrap a Vec around the values? I know I > should use > >> >> >> >> >> scatter but I couldn't even wrap a Vec around them. > >> >> >> >> > > >> >> >> >> > > >> >> >> >> > I would do > >> >> >> >> > > >> >> >> >> > MatGetSize(&N); > >> >> >> >> > MatGetLocalSize(&m); > >> >> >> >> > > https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MatDenseGetArray.html > >> >> >> >> > > >> >> >> >> > > https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Vec/VecPlaceArray.html > >> >> >> >> > > https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Vec/VecScatterCreateToAll.html > >> >> >> >> > > >> >> >> >> > > https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Vec/VecResetArray.html#VecResetArray > >> >> >> >> > > https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MatCreateDense.html > >> >> >> >> > > >> >> >> >> > > >> >> >> >> > > >> >> >> >> > > >> >> >> >> > Thanks, > >> >> >> >> > > >> >> >> >> > Matt > >> >> >> >> > > >> >> >> >> >> > >> >> >> >> >> Thanks so much! > >> >> >> >> >> > >> >> >> >> >> Eda > >> >> >> >> >> > >> >> >> >> >> > > >> >> >> >> >> > Thanks, > >> >> >> >> >> > > >> >> >> >> >> > Matt > >> >> >> >> >> > > >> >> >> >> >> >> > >> >> >> >> >> >> Thanks! > >> >> >> >> >> >> > >> >> >> >> >> >> Eda > >> >> >> >> >> > > >> >> >> >> >> > > >> >> >> >> >> > > >> >> >> >> >> > -- > >> >> >> >> >> > What most experimenters take for granted before they > begin their experiments is infinitely more interesting than any results to > which their experiments lead. > >> >> >> >> >> > -- Norbert Wiener > >> >> >> >> >> > > >> >> >> >> >> > https://www.cse.buffalo.edu/~knepley/ > >> >> >> >> > > >> >> >> >> > > >> >> >> >> > > >> >> >> >> > -- > >> >> >> >> > What most experimenters take for granted before they begin > their experiments is infinitely more interesting than any results to which > their experiments lead. > >> >> >> >> > -- Norbert Wiener > >> >> >> >> > > >> >> >> >> > https://www.cse.buffalo.edu/~knepley/ > >> >> >> > > >> >> >> > > >> >> >> > > >> >> >> > -- > >> >> >> > What most experimenters take for granted before they begin > their experiments is infinitely more interesting than any results to which > their experiments lead. > >> >> >> > -- Norbert Wiener > >> >> >> > > >> >> >> > https://www.cse.buffalo.edu/~knepley/ > >> >> > > >> >> > > >> >> > > >> >> > -- > >> >> > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > >> >> > -- Norbert Wiener > >> >> > > >> >> > https://www.cse.buffalo.edu/~knepley/ > >> > > >> > > >> > > >> > -- > >> > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > >> > -- Norbert Wiener > >> > > >> > https://www.cse.buffalo.edu/~knepley/ > >> > > >> > > > > > > > > > -- > > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > > -- Norbert Wiener > > > > https://www.cse.buffalo.edu/~knepley/ > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From mfadams at lbl.gov Wed Jul 1 07:15:13 2020 From: mfadams at lbl.gov (Mark Adams) Date: Wed, 1 Jul 2020 08:15:13 -0400 Subject: [petsc-users] DIVERGED_NANORINF when using HYPRE/BoomerAMG In-Reply-To: References: Message-ID: As Matt said AMG does not work for everything out-of-the-box. Hypre is pretty robust and I am surprised that every option gives INFs/NANs. But if it solves your stretched Lapacian then it's probably hooked up correctly. You might try a non-adjoint Navier-Stokes problem, if you can. You could try 'gamg' with the fgmres/ilu solver that works, as your smoother (use -ksp_view to check that you get everything set correctly). Use fgmres as the outer Krylov method and use -pc_gamg_agg_nsmooths 0. Mark On Tue, Jun 30, 2020 at 9:16 AM Matthew Knepley wrote: > On Tue, Jun 30, 2020 at 7:59 AM Andrea Iob wrote: > >> Hi, >> >> I'm trying to solve a linear system using HYPRE/BoomerAMG as >> preconditioner. The system comes from a two-dimensional adjoint >> Navier-Stokes problem. >> > > Do you expect the system you are solving to be elliptic? BoomerAMG is > designed for elliptic systems, and can easily fail if applied > to a more general system, say one with zeros on the diagonal. > > Thanks, > > Matt > > >> The mesh is structured (6400 cells) and there are 24 DoFs for each cell >> (the matrix has a total of 153600 rows). The condition number of the matrix >> should be in the order of 1e9. Using ILU preconditioner and FGMRES, the >> system is correctly solved (it takes 800 iterations to reach a residual of >> 1e-12). However, replacing ILU with HYPRE/BoomerAMG, the solver stops right >> after the first iteration (DIVERGED_NANORINF). I've tried different >> BoomerAMG options and different solvers (including Richardson to use >> BoomerAMG without a Krylov method), but I always get the same >> DIVERGED_NANORINF error. >> >> Using the same HYPRE/BoomerAMG + FGMERS setup on another problem >> (Laplacian on a highly stretched grid), the solver reaches convergence very >> quickly without any problems. >> >> At the bottom of this mail, you'll find the log of a run that stops with >> DIVERGED_NANORINF. Do you spot something in my setup that may explain why >> the solver is not converging? >> >> Thanks. Best regards, >> Andrea >> >> ----------------------------------------- >> Assembly started... >> Reading matrix file... >> - Number of rows (N) = 153600 >> - Number of non-zero entries (NNZ) = 18247680 >> Assembly completed. >> Time elapsed 83.3231s >> Set up started... >> Set up completed. >> Time elapsed 6.62441s >> Solution started... >> 0 KSP unpreconditioned resid norm 7.736650641501e-01 true resid norm >> 7.736650641501e-01 ||r(i)||/||b|| 1.000000000000e+00 >> 0 KSP Residual norm 7.736650641501e-01 % max 1.000000000000e+00 min >> 1.000000000000e+00 max/min 1.000000000000e+00 >> [0]PETSC ERROR: --------------------- Error Message >> -------------------------------------------------------------- >> [0]PETSC ERROR: >> [0]PETSC ERROR: KSPSolve has not converged due to Nan or Inf inner product >> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html >> for trouble shooting. >> [0]PETSC ERROR: Petsc Release Version 3.10.3, unknown >> [0]PETSC ERROR: bitpit_system_solver on a named xxx by xxx Tue Jun 30 >> 09:42:09 2020 >> [0]PETSC ERROR: Configure options PETSC_ARCH=arch-linux2-c-opt-gcc7-hypre >> --with-blaslapack-dir=/opt/lapack/3.8.0-gcc.7.4.0/lib64 --with-debugging=0 >> COPTFLAGS=-O3 CXXOPTFLAGS=-O3 FOPTFLAGS=-O3 --with-valgrind=1 >> --with-valgrind-dir=/opt/valgrind/3.14.0/ >> --prefix=/opt/petsc/3.10.3_gcc-7.4.0 --with-mpi=1 --download-hypre >> [0]PETSC ERROR: #1 KSPGMRESClassicalGramSchmidtOrthogonalization() line >> 67 in /root/InstallSources/petsc/src/ksp/ksp/impls/gmres/borthog2.c >> [0]PETSC ERROR: #2 KSPFGMRESCycle() line 175 in >> /root/InstallSources/petsc/src/ksp/ksp/impls/gmres/fgmres/fgmres.c >> [0]PETSC ERROR: #3 KSPSolve_FGMRES() line 291 in >> /root/InstallSources/petsc/src/ksp/ksp/impls/gmres/fgmres/fgmres.c >> [0]PETSC ERROR: #4 KSPSolve() line 780 in >> /root/InstallSources/petsc/src/ksp/ksp/interface/itfunc.c >> KSP Object: 1 MPI processes >> type: fgmres >> restart=45, using Classical (unmodified) Gram-Schmidt >> Orthogonalization with no iterative refinement >> happy breakdown tolerance 1e-30 >> maximum iterations=10000, nonzero initial guess >> tolerances: relative=1e-12, absolute=1e-50, divergence=10000. >> right preconditioning >> using UNPRECONDITIONED norm type for convergence test >> PC Object: 1 MPI processes >> type: hypre >> HYPRE BoomerAMG preconditioning >> Cycle type V >> Maximum number of levels 25 >> Maximum number of iterations PER hypre call 1 >> Convergence tolerance PER hypre call 0. >> Threshold for strong coupling 0.7 >> Interpolation truncation factor 0.3 >> Interpolation: max elements per row 2 >> Number of levels of aggressive coarsening 4 >> Number of paths for aggressive coarsening 5 >> Maximum row sums 0.9 >> Sweeps down 1 >> Sweeps up 1 >> Sweeps on coarse 1 >> Relax down sequential-Gauss-Seidel >> Relax up sequential-Gauss-Seidel >> Relax on coarse Gaussian-elimination >> Relax weight (all) 1. >> Outer relax weight (all) 1. >> Using CF-relaxation >> Smooth type Euclid >> Smooth num levels 25 >> Euclid ILU(k) levels 0 >> Euclid ILU(k) drop tolerance 0. >> Euclid ILU use Block-Jacobi? 1 >> Measure type local >> Coarsen type HMIS >> Interpolation type ext+i >> linear system matrix = precond matrix: >> Mat Object: Mat_0x1e88040_0 1 MPI processes >> type: seqaij >> rows=153600, cols=153600, bs=24 >> total: nonzeros=18247680, allocated nonzeros=18247680 >> total number of mallocs used during MatSetValues calls =0 >> using I-node routines: found 32000 nodes, limit used is 5 >> Solution completed. >> Time elapsed 6.93336s >> Solution information... >> Error (L2) : 32.8359 >> Error (Linf) : 1.93278 >> >> ************************************************************************************************************************ >> *** WIDEN YOUR WINDOW TO 120 CHARACTERS. Use 'enscript -r >> -fCourier9' to print this document *** >> >> ************************************************************************************************************************ >> >> ---------------------------------------------- PETSc Performance Summary: >> ---------------------------------------------- >> >> bitpit_system_solver on a named xxx with 1 processor, by xxx Tue Jun 30 >> 09:43:46 2020 >> Using Petsc Release Version 3.10.3, unknown >> >> Max Max/Min Avg Total >> Time (sec): 9.718e+01 1.000 9.718e+01 >> Objects: 5.300e+01 1.000 5.300e+01 >> Flop: 1.107e+08 1.000 1.107e+08 1.107e+08 >> Flop/sec: 1.139e+06 1.000 1.139e+06 1.139e+06 >> MPI Messages: 0.000e+00 0.000 0.000e+00 0.000e+00 >> MPI Message Lengths: 0.000e+00 0.000 0.000e+00 0.000e+00 >> MPI Reductions: 0.000e+00 0.000 >> >> Flop counting convention: 1 flop = 1 real number operation of type >> (multiply/divide/add/subtract) >> e.g., VecAXPY() for real vectors of length N >> --> 2N flop >> and VecAXPY() for complex vectors of length N >> --> 8N flop >> >> Summary of Stages: ----- Time ------ ----- Flop ------ --- Messages >> --- -- Message Lengths -- -- Reductions -- >> Avg %Total Avg %Total Count >> %Total Avg %Total Count %Total >> 0: Main Stage: 9.7178e+01 100.0% 1.1071e+08 100.0% 0.000e+00 >> 0.0% 0.000e+00 0.0% 0.000e+00 0.0% >> >> >> ------------------------------------------------------------------------------------------------------------------------ >> See the 'Profiling' chapter of the users' manual for details on >> interpreting output. >> Phase summary info: >> Count: number of times phase was executed >> Time and Flop: Max - maximum over all processors >> Ratio - ratio of maximum to minimum over all processors >> Mess: number of messages sent >> AvgLen: average message length (bytes) >> Reduct: number of global reductions >> Global: entire computation >> Stage: stages of a computation. Set stages with PetscLogStagePush() >> and PetscLogStagePop(). >> %T - percent time in this phase %F - percent flop in this >> phase >> %M - percent messages in this phase %L - percent message >> lengths in this phase >> %R - percent reductions in this phase >> Total Mflop/s: 10e-6 * (sum of flop over all processors)/(max time >> over all processors) >> >> ------------------------------------------------------------------------------------------------------------------------ >> Event Count Time (sec) >> Flop --- Global --- --- Stage ---- Total >> Max Ratio Max Ratio Max Ratio Mess AvgLen >> Reduct %T %F %M %L %R %T %F %M %L %R Mflop/s >> >> ------------------------------------------------------------------------------------------------------------------------ >> >> --- Event Stage 0: Main Stage >> >> BuildTwoSidedF 2 1.0 1.8471e-04 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 >> 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 >> MatMult 3 1.0 6.8688e-02 1.0 1.09e+08 1.0 0.0e+00 0.0e+00 >> 0.0e+00 0 98 0 0 0 0 98 0 0 0 1587 >> MatConvert 2 1.0 3.8534e-01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 >> 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 >> MatAssemblyBegin 3 1.0 2.8112e-04 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 >> 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 >> MatAssemblyEnd 3 1.0 7.8563e-02 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 >> 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 >> MatGetRowIJ 2 1.0 3.6100e-06 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 >> 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 >> MatView 2 1.0 5.4618e+01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 >> 0.0e+00 56 0 0 0 0 56 0 0 0 0 0 >> VecView 2 1.0 7.6981e-01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 >> 0.0e+00 1 0 0 0 0 1 0 0 0 0 0 >> VecMDot 1 1.0 1.7635e-04 1.0 3.07e+05 1.0 0.0e+00 0.0e+00 >> 0.0e+00 0 0 0 0 0 0 0 0 0 0 1742 >> VecNorm 3 1.0 5.4832e-04 1.0 9.22e+05 1.0 0.0e+00 0.0e+00 >> 0.0e+00 0 1 0 0 0 0 1 0 0 0 1681 >> VecScale 1 1.0 1.5216e-04 1.0 1.54e+05 1.0 0.0e+00 0.0e+00 >> 0.0e+00 0 0 0 0 0 0 0 0 0 0 1009 >> VecCopy 1 1.0 1.5406e-04 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 >> 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 >> VecSet 21 1.0 7.1106e-03 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 >> 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 >> VecAYPX 1 1.0 1.9940e-04 1.0 1.54e+05 1.0 0.0e+00 0.0e+00 >> 0.0e+00 0 0 0 0 0 0 0 0 0 0 770 >> VecWAXPY 1 1.0 7.8802e-04 1.0 1.54e+05 1.0 0.0e+00 0.0e+00 >> 0.0e+00 0 0 0 0 0 0 0 0 0 0 195 >> KSPSetUp 2 1.0 6.8995e-03 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 >> 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 >> PCSetUp 2 1.0 1.3190e+01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 >> 0.0e+00 14 0 0 0 0 14 0 0 0 0 0 >> PCApply 1 1.0 2.2728e-01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 >> 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 >> >> ------------------------------------------------------------------------------------------------------------------------ >> >> Memory usage is given in bytes: >> >> Object Type Creations Destructions Memory Descendants' >> Mem. >> Reports information only for process 0. >> >> --- Event Stage 0: Main Stage >> >> Matrix 3 1 2872 0. >> Vector 32 20 17234800 0. >> Index Set 4 4 3200 0. >> IS L to G Mapping 2 0 0 0. >> Vec Scatter 2 0 0 0. >> Viewer 6 4 3392 0. >> Krylov Solver 2 1 61676 0. >> Preconditioner 2 1 1432 0. >> >> ======================================================================================================================== >> Average time to get PetscTime(): 3.36e-08 >> #PETSc Option Table entries: >> -ksp_converged_reason >> -ksp_error_if_not_converged >> -ksp_monitor_singular_value >> -ksp_monitor_true_residual >> -log_view >> -pc_hypre_boomeramg_agg_nl 4 >> -pc_hypre_boomeramg_agg_num_paths 5 >> -pc_hypre_boomeramg_coarsen_type HMIS >> -pc_hypre_boomeramg_eu_bj true >> -pc_hypre_boomeramg_interp_type ext+i >> -pc_hypre_boomeramg_P_max 2 >> -pc_hypre_boomeramg_relax_type_all sequential-Gauss-Seidel >> -pc_hypre_boomeramg_smooth_type Euclid >> -pc_hypre_boomeramg_strong_threshold 0.7 >> -pc_hypre_boomeramg_truncfactor 0.3 >> #End of PETSc Option Table entries >> Compiled without FORTRAN kernels >> Compiled with full precision matrices (default) >> sizeof(short) 2 sizeof(int) 4 sizeof(long) 8 sizeof(void*) 8 >> sizeof(PetscScalar) 8 sizeof(PetscInt) 4 >> Configure options: PETSC_ARCH=arch-linux2-c-opt-gcc7-hypre >> --with-blaslapack-dir=/opt/lapack/3.8.0-gcc.7.4.0/lib64 --with-debugging=0 >> COPTFLAGS=-O3 CXXOPTFLAGS=-O3 FOPTFLAGS=-O3 --with-valgrind=1 >> --with-valgrind-dir=/opt/valgrind/3.14.0/ >> --prefix=/opt/petsc/3.10.3_gcc-7.4.0 --with-mpi=1 --download-hypre >> ----------------------------------------- >> Libraries compiled on 2020-06-26 12:35:46 on xxx >> Machine characteristics: >> Linux-3.10.0-1127.8.2.el7.x86_64-x86_64-with-centos-7.6.1810-Core >> Using PETSc directory: /opt/petsc/3.10.3_gcc-7.4.0 >> Using PETSc arch: >> ----------------------------------------- >> >> Using C compiler: mpicc -fPIC -Wall -Wwrite-strings >> -Wno-strict-aliasing -Wno-unknown-pragmas -fstack-protector >> -fvisibility=hidden -O3 >> Using Fortran compiler: mpif90 -fPIC -Wall -ffree-line-length-0 >> -Wno-unused-dummy-argument -O3 >> ----------------------------------------- >> >> Using include paths: -I/opt/petsc/3.10.3_gcc-7.4.0/include >> -I/opt/valgrind/3.14.0/include >> ----------------------------------------- >> >> Using C linker: mpicc >> Using Fortran linker: mpif90 >> Using libraries: -Wl,-rpath,/opt/petsc/3.10.3_gcc-7.4.0/lib >> -L/opt/petsc/3.10.3_gcc-7.4.0/lib -lpetsc >> -Wl,-rpath,/opt/petsc/3.10.3_gcc-7.4.0/lib >> -L/opt/petsc/3.10.3_gcc-7.4.0/lib >> -Wl,-rpath,/opt/lapack/3.8.0-gcc.7.4.0/lib64 >> -L/opt/lapack/3.8.0-gcc.7.4.0/lib64 >> -Wl,-rpath,/opt/openmpi4.0.0-gcc7.4.0_slurm/lib >> -L/opt/openmpi4.0.0-gcc7.4.0_slurm/lib >> -Wl,-rpath,/opt/gnu/7.4/lib/gcc/x86_64-pc-linux-gnu/7.4.0 >> -L/opt/gnu/7.4/lib/gcc/x86_64-pc-linux-gnu/7.4.0 >> -Wl,-rpath,/opt/gnu/7.4/lib/gcc -L/opt/gnu/7.4/lib/gcc >> -Wl,-rpath,/opt/gnu/7.4/lib64 -L/opt/gnu/7.4/lib64 >> -Wl,-rpath,/opt/gnu/7.4/lib -L/opt/gnu/7.4/lib -lHYPRE -llapack -lblas -lm >> -lX11 -lstdc++ -ldl -lmpi_usempif08 -lmpi_usempi_ignore_tkr -lmpi_mpifh >> -lmpi -lgfortran -lm -lgfortran -lm -lgcc_s -lquadmath -lpthread -lstdc++ >> -ldl >> ----------------------------------------- >> > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From eda.oktay at metu.edu.tr Wed Jul 1 07:27:52 2020 From: eda.oktay at metu.edu.tr (Eda Oktay) Date: Wed, 1 Jul 2020 15:27:52 +0300 Subject: [petsc-users] MatGetRow for global rows of a parallel matrix In-Reply-To: References: Message-ID: I got it, thanks so much! Eda On Wed, Jul 1, 2020, 2:42 PM Matthew Knepley wrote: > On Wed, Jul 1, 2020 at 7:32 AM Eda Oktay wrote: > >> I'm sorry but I still can't understand how to put an input as these >> parameters. It is the first time I saw 'const' for IS. If it doesn't >> refer IS, then as a parameter for row and column indices, how should I >> give them as a valid input? It still should be an IS, right? >> >> Lastly, I tried to use in the form: >> >> IS idUi; >> ISCreateStride(PETSC_COMM_WORLD,siz,0,1,&idUi); >> Mat *submat; >> MatCreateSubMatrices(U,nev,&idUi,&idUi,MAT_INITIAL_MATRIX,&submat); >> > > You gave n = 4 (you call it nev), but you only have 1 rowIS. You are > supposed to give an array of 4. > > Thanks, > > Matt > > >> where nev = 4, siz = 72 and U is 72*4 matrix. >> >> But then, the error becomes about the matrix, U: >> >> [0]PETSC ERROR: --------------------- Error Message >> -------------------------------------------------------------- >> [0]PETSC ERROR: [1]PETSC ERROR: --------------------- Error Message >> -------------------------------------------------------------- >> [1]PETSC ERROR: Invalid argument >> [1]PETSC ERROR: Wrong type of object: Parameter # 1 >> Invalid argument >> [0]PETSC ERROR: Wrong type of object: Parameter # 1 >> [0]PETSC ERROR: See >> https://www.mcs.anl.gov/petsc/documentation/faq.html for trouble >> shooting. >> [0]PETSC ERROR: Petsc Release Version 3.13.2, Jun 02, 2020 >> [0]PETSC ERROR: ./deneme_new_vecscatter_arastep on a >> arch-linux2-c-debug named b342.wls.metu.edu.tr by edaoktay Wed Jul 1 >> 14:25:00 2020 >> [0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ >> --download-mpich --download-openblas --download-slepc --download-metis >> --download-parmetis --download-chaco --with-X=1 >> [1]PETSC ERROR: See >> https://www.mcs.anl.gov/petsc/documentation/faq.html for trouble >> shooting. >> [1]PETSC ERROR: Petsc Release Version 3.13.2, Jun 02, 2020 >> [1]PETSC ERROR: [0]PETSC ERROR: #1 ISSorted() line 1777 in >> /home/edaoktay/petsc-3.13.2/src/vec/is/is/interface/index.c >> [0]PETSC ERROR: ./deneme_new_vecscatter_arastep on a >> arch-linux2-c-debug named b342.wls.metu.edu.tr by edaoktay Wed Jul 1 >> 14:25:00 2020 >> [1]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ >> --download-mpich --download-openblas --download-slepc --download-metis >> --download-parmetis --download-chaco --with-X=1 >> [1]PETSC ERROR: #2 MatCreateSubMatrices_MPIDense_Local() line 104 in >> /home/edaoktay/petsc-3.13.2/src/mat/impls/dense/mpi/mmdense.c >> [0]PETSC ERROR: #3 MatCreateSubMatrices_MPIDense() line 66 in >> /home/edaoktay/petsc-3.13.2/src/mat/impls/dense/mpi/mmdense.c >> #1 ISSorted() line 1777 in >> /home/edaoktay/petsc-3.13.2/src/vec/is/is/interface/index.c >> [1]PETSC ERROR: #2 MatCreateSubMatrices_MPIDense_Local() line 104 in >> /home/edaoktay/petsc-3.13.2/src/mat/impls/dense/mpi/mmdense.c >> [0]PETSC ERROR: #4 MatCreateSubMatrices() line 6758 in >> /home/edaoktay/petsc-3.13.2/src/mat/interface/matrix.c >> [1]PETSC ERROR: #3 MatCreateSubMatrices_MPIDense() line 66 in >> /home/edaoktay/petsc-3.13.2/src/mat/impls/dense/mpi/mmdense.c >> [1]PETSC ERROR: #4 MatCreateSubMatrices() line 6758 in >> /home/edaoktay/petsc-3.13.2/src/mat/interface/matrix.c >> >> Before I used &idUi, I tried to use just idUi (without & sign) and it >> didn't work, so I decided to try using this way, since may be 'const' >> refers this one. But how can my matrix be of wrong type? It is MPI >> Dense. >> >> Thanks! >> >> Eda >> >> Matthew Knepley , 1 Tem 2020 ?ar, 13:43 tarihinde >> ?unu yazd?: >> > >> > On Wed, Jul 1, 2020 at 6:34 AM Eda Oktay wrote: >> >> >> >> Der Barry, >> >> >> >> ? am trying to use your way but I couldn't understand how I create sub >> >> matrices by using MatCreateSubMatrices() since as input, the function >> >> needs const IS instead of IS and I couldn't understand how to get a >> >> const IS. I tried to use ISCreateStride since this IS should be 0:71 >> >> because as you mentioned, the sub matrix should consist of the entire >> >> matrix. However, since ISCreateStride produces IS, not const IS, I >> >> couldn't use it in MatCreateSubMatrices(). >> > >> > >> > The 'const' refers to the array, not the IS: >> > >> > >> https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MatCreateSubMatrices.html >> > >> > Thanks, >> > >> > Matt >> > >> >> >> >> Thanks >> >> >> >> Eda >> >> >> >> Barry Smith , 10 Haz 2020 ?ar, 19:59 tarihinde ?unu >> yazd?: >> >> > >> >> > >> >> > You can use MatCreateSubMatrices() with each process getting a >> single sequential dense sub matrix that consists of the entire matrix. >> >> > >> >> > Use VecDuplicateVecs() to create an array of 72 vectors (create a >> single seq vector of size 4 as the input to this routine) >> >> > >> >> > Then use MatDenseGetArrayRead() to access the upper left corner of >> the new sequential dense matrix >> >> > >> >> > Loop over the vectors calling VecGetArray() >> >> > Then loop over the row of the dense array filling up the vector >> >> > >> >> > Because dense matrices are stored by column, you have to do this >> looping to fill up the vectors, they can't share the space with the matrix. >> >> > >> >> > Barry >> >> > >> >> > >> >> > >> >> > On Jun 10, 2020, at 11:36 AM, Matthew Knepley >> wrote: >> >> > >> >> > On Wed, Jun 10, 2020 at 12:26 PM Eda Oktay >> wrote: >> >> >> >> >> >> Matthew Knepley , 10 Haz 2020 ?ar, 19:13 >> tarihinde >> >> >> ?unu yazd?: >> >> >> > >> >> >> > On Wed, Jun 10, 2020 at 12:07 PM Eda Oktay >> wrote: >> >> >> >> >> >> >> >> Der Matt, >> >> >> >> >> >> >> >> When I looked at the results, I found that there are some >> problems I >> >> >> >> couldn't understand. >> >> >> >> >> >> >> >> First of all, I am working on a 72*4 matrix and as I said >> before, I >> >> >> >> want to have 72 different vectors having size 4 each, whose >> elements >> >> >> >> consist of the elements in the same row. And of course, all >> vectors >> >> >> >> should be in all processors (currently I am using 4 processors). >> >> >> >> >> >> >> >> When I use your scatter code, the output vector is divided into 4 >> >> >> >> parts for 4 processors and each vector consists of 18 row vectors >> >> >> >> whose elements are arranged in a way that if I want to find >> zeroth row >> >> >> >> vector, its elements are located in 0th,18th,36th,54th elements. >> >> >> > >> >> >> > >> >> >> > Was the global size of the vector you wrapped around the dense >> matrix 72*4? >> >> >> >> >> >> Yes it is. I set up its global size to 72*4. >> >> >> >> >> >> > >> >> >> > If you use CreateToAll(), it will make a vector on each process >> which has the global size of the original vector. >> >> >> >> >> >> Although I set 72*4, the size of the vectors in each process is 72. >> >> > >> >> > >> >> > You can understand how it is hard to accept, as this code is tested >> every night. Can you VecView() the input vector >> >> > to CreateToAll and the output vector, and send that output? >> >> > >> >> > Thanks, >> >> > >> >> > Matt >> >> > >> >> >> >> >> >> Thanks, >> >> >> >> >> >> Eda >> >> >> >> >> >> > >> >> >> > Thanks, >> >> >> > >> >> >> > Matt >> >> >> > >> >> >> >> >> >> >> >> So, isn't scatter's goal is to scatter all values to all >> processors? >> >> >> >> >> >> >> >> Furthermore, I am trying to use my vectors in that way but isn't >> there >> >> >> >> any possible way that I can reach my goal entirely? >> >> >> >> >> >> >> >> Thanks so much for your help, >> >> >> >> >> >> >> >> Eda >> >> >> >> >> >> >> >> Matthew Knepley , 10 Haz 2020 ?ar, 18:11 >> tarihinde >> >> >> >> ?unu yazd?: >> >> >> >> > >> >> >> >> > On Wed, Jun 10, 2020 at 10:09 AM Eda Oktay < >> eda.oktay at metu.edu.tr> wrote: >> >> >> >> >> >> >> >> >> >> Dear Matt, >> >> >> >> >> >> >> >> >> >> I have one last question I believe. Up to creating a dense >> matrix I >> >> >> >> >> did what you've suggested. Thank you so much for that. >> >> >> >> >> >> >> >> >> >> I created a new dense matrix. Now, how should I wrap each >> vector in a >> >> >> >> >> MatDense again? I mean, what is wrapping vectors in a matrix? >> To put >> >> >> >> >> each of them again as rows? >> >> >> >> > >> >> >> >> > >> >> >> >> > I thought you need a dense matrix for something, since you >> started with one. If you >> >> >> >> > do not, just do VecGetArray() on the vector from CreateToAll >> and use the values. >> >> >> >> > >> >> >> >> > Thanks, >> >> >> >> > >> >> >> >> > Matt >> >> >> >> > >> >> >> >> >> >> >> >> >> >> Thanks! >> >> >> >> >> >> >> >> >> >> Eda >> >> >> >> >> >> >> >> >> >> Matthew Knepley , 10 Haz 2020 ?ar, 16:16 >> tarihinde >> >> >> >> >> ?unu yazd?: >> >> >> >> >> > >> >> >> >> >> > On Wed, Jun 10, 2020 at 9:08 AM Eda Oktay < >> eda.oktay at metu.edu.tr> wrote: >> >> >> >> >> >> >> >> >> >> >> >> Dear Matt, >> >> >> >> >> >> >> >> >> >> >> >> Matthew Knepley , 10 Haz 2020 ?ar, >> 16:03 tarihinde >> >> >> >> >> >> ?unu yazd?: >> >> >> >> >> >> > >> >> >> >> >> >> > On Wed, Jun 10, 2020 at 8:56 AM Eda Oktay < >> eda.oktay at metu.edu.tr> wrote: >> >> >> >> >> >> >> >> >> >> >> >> >> >> Hi all, >> >> >> >> >> >> >> >> >> >> >> >> >> >> I am trying to get all the rows of a parallel matrix as >> individual >> >> >> >> >> >> >> vectors. For instance, if I have 72*4 matrix, I want to >> get 72 >> >> >> >> >> >> >> different vectors having size 4. >> >> >> >> >> >> >> >> >> >> >> >> >> >> As far as I understood, MatGetRow is only for local >> rows, so >> >> >> >> >> >> >> MatGetOwnershipRange is used, however, when I tried >> this one, I >> >> >> >> >> >> >> couldn't get the whole and desired row vectors. >> >> >> >> >> >> >> >> >> >> >> >> >> >> In MatGetRow explanation, it is written that I should >> use >> >> >> >> >> >> >> MatCreateSubMatrices first, then use MatGetRow. But I >> couldn't >> >> >> >> >> >> >> understand to which extent I should create submatrices. >> I just need to >> >> >> >> >> >> >> have all 72 rows as 72 different vectors each having 4 >> elements. >> >> >> >> >> >> > >> >> >> >> >> >> > >> >> >> >> >> >> > 1) For sparse matrices, the storage is always divided by >> row, so that values can only be retrieved for local rows with MatGetRow() >> >> >> >> >> >> > >> >> >> >> >> >> > 2) Is this matrix sparse? It sounds like it is dense. >> >> >> >> >> >> >> >> >> >> >> >> Matrix is dense. >> >> >> >> >> >> >> >> >> >> >> >> > >> >> >> >> >> >> > 3) Are you asking to get all matrix values on all >> processes? If so, I think the easiest thing to do is first wrap a Vec >> around the >> >> >> >> >> >> > values, then use VecScatterToAll(), then wrap each >> one in a MatDense again. >> >> >> >> >> >> >> >> >> >> >> >> Yes, I want all row vectors on all processes. In a dense >> matrix, >> >> >> >> >> >> should I still wrap a Vec around the values? I know I >> should use >> >> >> >> >> >> scatter but I couldn't even wrap a Vec around them. >> >> >> >> >> > >> >> >> >> >> > >> >> >> >> >> > I would do >> >> >> >> >> > >> >> >> >> >> > MatGetSize(&N); >> >> >> >> >> > MatGetLocalSize(&m); >> >> >> >> >> > >> https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MatDenseGetArray.html >> >> >> >> >> > >> >> >> >> >> > >> https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Vec/VecPlaceArray.html >> >> >> >> >> > >> https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Vec/VecScatterCreateToAll.html >> >> >> >> >> > >> >> >> >> >> > >> https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Vec/VecResetArray.html#VecResetArray >> >> >> >> >> > >> https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MatCreateDense.html >> >> >> >> >> > >> >> >> >> >> > >> >> >> >> >> > >> >> >> >> >> > >> >> >> >> >> > Thanks, >> >> >> >> >> > >> >> >> >> >> > Matt >> >> >> >> >> > >> >> >> >> >> >> >> >> >> >> >> >> Thanks so much! >> >> >> >> >> >> >> >> >> >> >> >> Eda >> >> >> >> >> >> >> >> >> >> >> >> > >> >> >> >> >> >> > Thanks, >> >> >> >> >> >> > >> >> >> >> >> >> > Matt >> >> >> >> >> >> > >> >> >> >> >> >> >> >> >> >> >> >> >> >> Thanks! >> >> >> >> >> >> >> >> >> >> >> >> >> >> Eda >> >> >> >> >> >> > >> >> >> >> >> >> > >> >> >> >> >> >> > >> >> >> >> >> >> > -- >> >> >> >> >> >> > What most experimenters take for granted before they >> begin their experiments is infinitely more interesting than any results to >> which their experiments lead. >> >> >> >> >> >> > -- Norbert Wiener >> >> >> >> >> >> > >> >> >> >> >> >> > https://www.cse.buffalo.edu/~knepley/ >> >> >> >> >> > >> >> >> >> >> > >> >> >> >> >> > >> >> >> >> >> > -- >> >> >> >> >> > What most experimenters take for granted before they begin >> their experiments is infinitely more interesting than any results to which >> their experiments lead. >> >> >> >> >> > -- Norbert Wiener >> >> >> >> >> > >> >> >> >> >> > https://www.cse.buffalo.edu/~knepley/ >> >> >> >> > >> >> >> >> > >> >> >> >> > >> >> >> >> > -- >> >> >> >> > What most experimenters take for granted before they begin >> their experiments is infinitely more interesting than any results to which >> their experiments lead. >> >> >> >> > -- Norbert Wiener >> >> >> >> > >> >> >> >> > https://www.cse.buffalo.edu/~knepley/ >> >> >> > >> >> >> > >> >> >> > >> >> >> > -- >> >> >> > What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> >> >> > -- Norbert Wiener >> >> >> > >> >> >> > https://www.cse.buffalo.edu/~knepley/ >> >> > >> >> > >> >> > >> >> > -- >> >> > What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> >> > -- Norbert Wiener >> >> > >> >> > https://www.cse.buffalo.edu/~knepley/ >> >> > >> >> > >> > >> > >> > >> > -- >> > What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> > -- Norbert Wiener >> > >> > https://www.cse.buffalo.edu/~knepley/ >> > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From andrea_iob at hotmail.com Wed Jul 1 08:22:34 2020 From: andrea_iob at hotmail.com (Andrea Iob) Date: Wed, 1 Jul 2020 13:22:34 +0000 Subject: [petsc-users] DIVERGED_NANORINF when using HYPRE/BoomerAMG In-Reply-To: References: , Message-ID: Thanks for the answers. The matrix contains both the advection and the diffusion terms of the Navier-Stokes equations. The case is subsonic (M=0.5), Reynolds number is in the order of 5000 so I expect the hyperbolic features to be the dominant ones. I understand this is not the type of problems AMG is designed for, but I was hoping to still be able to solve the system. This is the first time I use BoomerAMG and I'm not familar with all the options, however all the tests I've done gave me INFs/NANs. You suggestion to use GAMG with custom smoothers seems to do the trick: after setting the GAMG smoother of each level to FGMRES/ILU, the solver converges in 93 iterations (overall it seems faster than ILU). Great! I've set the smoothers like this: for (int i = 0; i < nLevels; ++i) { KSP levelSmoother; PCMGGetSmoother(preconditioner, i, &levelSmoother); PC levelSmootherPreconditioner; KSPGetPC(levelSmoother, &levelSmootherPreconditioner); PCSetType(levelSmootherPreconditioner, PCILU); KSPSetType(levelSmoother, KSPFGMRES); } Thank you very much! Best regards, Andrea ________________________________ From: Mark Adams Sent: Wednesday, July 1, 2020 2:15 PM To: Matthew Knepley Cc: Andrea Iob ; petsc-users at mcs.anl.gov Subject: Re: [petsc-users] DIVERGED_NANORINF when using HYPRE/BoomerAMG As Matt said AMG does not work for everything out-of-the-box. Hypre is pretty robust and I am surprised that every option gives INFs/NANs. But if it solves your stretched Lapacian then it's probably hooked up correctly. You might try a non-adjoint Navier-Stokes problem, if you can. You could try 'gamg' with the fgmres/ilu solver that works, as your smoother (use -ksp_view to check that you get everything set correctly). Use fgmres as the outer Krylov method and use -pc_gamg_agg_nsmooths 0. Mark On Tue, Jun 30, 2020 at 9:16 AM Matthew Knepley > wrote: On Tue, Jun 30, 2020 at 7:59 AM Andrea Iob > wrote: Hi, I'm trying to solve a linear system using HYPRE/BoomerAMG as preconditioner. The system comes from a two-dimensional adjoint Navier-Stokes problem. Do you expect the system you are solving to be elliptic? BoomerAMG is designed for elliptic systems, and can easily fail if applied to a more general system, say one with zeros on the diagonal. Thanks, Matt The mesh is structured (6400 cells) and there are 24 DoFs for each cell (the matrix has a total of 153600 rows). The condition number of the matrix should be in the order of 1e9. Using ILU preconditioner and FGMRES, the system is correctly solved (it takes 800 iterations to reach a residual of 1e-12). However, replacing ILU with HYPRE/BoomerAMG, the solver stops right after the first iteration (DIVERGED_NANORINF). I've tried different BoomerAMG options and different solvers (including Richardson to use BoomerAMG without a Krylov method), but I always get the same DIVERGED_NANORINF error. Using the same HYPRE/BoomerAMG + FGMERS setup on another problem (Laplacian on a highly stretched grid), the solver reaches convergence very quickly without any problems. At the bottom of this mail, you'll find the log of a run that stops with DIVERGED_NANORINF. Do you spot something in my setup that may explain why the solver is not converging? Thanks. Best regards, Andrea ----------------------------------------- Assembly started... Reading matrix file... - Number of rows (N) = 153600 - Number of non-zero entries (NNZ) = 18247680 Assembly completed. Time elapsed 83.3231s Set up started... Set up completed. Time elapsed 6.62441s Solution started... 0 KSP unpreconditioned resid norm 7.736650641501e-01 true resid norm 7.736650641501e-01 ||r(i)||/||b|| 1.000000000000e+00 0 KSP Residual norm 7.736650641501e-01 % max 1.000000000000e+00 min 1.000000000000e+00 max/min 1.000000000000e+00 [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [0]PETSC ERROR: [0]PETSC ERROR: KSPSolve has not converged due to Nan or Inf inner product [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [0]PETSC ERROR: Petsc Release Version 3.10.3, unknown [0]PETSC ERROR: bitpit_system_solver on a named xxx by xxx Tue Jun 30 09:42:09 2020 [0]PETSC ERROR: Configure options PETSC_ARCH=arch-linux2-c-opt-gcc7-hypre --with-blaslapack-dir=/opt/lapack/3.8.0-gcc.7.4.0/lib64 --with-debugging=0 COPTFLAGS=-O3 CXXOPTFLAGS=-O3 FOPTFLAGS=-O3 --with-valgrind=1 --with-valgrind-dir=/opt/valgrind/3.14.0/ --prefix=/opt/petsc/3.10.3_gcc-7.4.0 --with-mpi=1 --download-hypre [0]PETSC ERROR: #1 KSPGMRESClassicalGramSchmidtOrthogonalization() line 67 in /root/InstallSources/petsc/src/ksp/ksp/impls/gmres/borthog2.c [0]PETSC ERROR: #2 KSPFGMRESCycle() line 175 in /root/InstallSources/petsc/src/ksp/ksp/impls/gmres/fgmres/fgmres.c [0]PETSC ERROR: #3 KSPSolve_FGMRES() line 291 in /root/InstallSources/petsc/src/ksp/ksp/impls/gmres/fgmres/fgmres.c [0]PETSC ERROR: #4 KSPSolve() line 780 in /root/InstallSources/petsc/src/ksp/ksp/interface/itfunc.c KSP Object: 1 MPI processes type: fgmres restart=45, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement happy breakdown tolerance 1e-30 maximum iterations=10000, nonzero initial guess tolerances: relative=1e-12, absolute=1e-50, divergence=10000. right preconditioning using UNPRECONDITIONED norm type for convergence test PC Object: 1 MPI processes type: hypre HYPRE BoomerAMG preconditioning Cycle type V Maximum number of levels 25 Maximum number of iterations PER hypre call 1 Convergence tolerance PER hypre call 0. Threshold for strong coupling 0.7 Interpolation truncation factor 0.3 Interpolation: max elements per row 2 Number of levels of aggressive coarsening 4 Number of paths for aggressive coarsening 5 Maximum row sums 0.9 Sweeps down 1 Sweeps up 1 Sweeps on coarse 1 Relax down sequential-Gauss-Seidel Relax up sequential-Gauss-Seidel Relax on coarse Gaussian-elimination Relax weight (all) 1. Outer relax weight (all) 1. Using CF-relaxation Smooth type Euclid Smooth num levels 25 Euclid ILU(k) levels 0 Euclid ILU(k) drop tolerance 0. Euclid ILU use Block-Jacobi? 1 Measure type local Coarsen type HMIS Interpolation type ext+i linear system matrix = precond matrix: Mat Object: Mat_0x1e88040_0 1 MPI processes type: seqaij rows=153600, cols=153600, bs=24 total: nonzeros=18247680, allocated nonzeros=18247680 total number of mallocs used during MatSetValues calls =0 using I-node routines: found 32000 nodes, limit used is 5 Solution completed. Time elapsed 6.93336s Solution information... Error (L2) : 32.8359 Error (Linf) : 1.93278 ************************************************************************************************************************ *** WIDEN YOUR WINDOW TO 120 CHARACTERS. Use 'enscript -r -fCourier9' to print this document *** ************************************************************************************************************************ ---------------------------------------------- PETSc Performance Summary: ---------------------------------------------- bitpit_system_solver on a named xxx with 1 processor, by xxx Tue Jun 30 09:43:46 2020 Using Petsc Release Version 3.10.3, unknown Max Max/Min Avg Total Time (sec): 9.718e+01 1.000 9.718e+01 Objects: 5.300e+01 1.000 5.300e+01 Flop: 1.107e+08 1.000 1.107e+08 1.107e+08 Flop/sec: 1.139e+06 1.000 1.139e+06 1.139e+06 MPI Messages: 0.000e+00 0.000 0.000e+00 0.000e+00 MPI Message Lengths: 0.000e+00 0.000 0.000e+00 0.000e+00 MPI Reductions: 0.000e+00 0.000 Flop counting convention: 1 flop = 1 real number operation of type (multiply/divide/add/subtract) e.g., VecAXPY() for real vectors of length N --> 2N flop and VecAXPY() for complex vectors of length N --> 8N flop Summary of Stages: ----- Time ------ ----- Flop ------ --- Messages --- -- Message Lengths -- -- Reductions -- Avg %Total Avg %Total Count %Total Avg %Total Count %Total 0: Main Stage: 9.7178e+01 100.0% 1.1071e+08 100.0% 0.000e+00 0.0% 0.000e+00 0.0% 0.000e+00 0.0% ------------------------------------------------------------------------------------------------------------------------ See the 'Profiling' chapter of the users' manual for details on interpreting output. Phase summary info: Count: number of times phase was executed Time and Flop: Max - maximum over all processors Ratio - ratio of maximum to minimum over all processors Mess: number of messages sent AvgLen: average message length (bytes) Reduct: number of global reductions Global: entire computation Stage: stages of a computation. Set stages with PetscLogStagePush() and PetscLogStagePop(). %T - percent time in this phase %F - percent flop in this phase %M - percent messages in this phase %L - percent message lengths in this phase %R - percent reductions in this phase Total Mflop/s: 10e-6 * (sum of flop over all processors)/(max time over all processors) ------------------------------------------------------------------------------------------------------------------------ Event Count Time (sec) Flop --- Global --- --- Stage ---- Total Max Ratio Max Ratio Max Ratio Mess AvgLen Reduct %T %F %M %L %R %T %F %M %L %R Mflop/s ------------------------------------------------------------------------------------------------------------------------ --- Event Stage 0: Main Stage BuildTwoSidedF 2 1.0 1.8471e-04 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatMult 3 1.0 6.8688e-02 1.0 1.09e+08 1.0 0.0e+00 0.0e+00 0.0e+00 0 98 0 0 0 0 98 0 0 0 1587 MatConvert 2 1.0 3.8534e-01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatAssemblyBegin 3 1.0 2.8112e-04 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatAssemblyEnd 3 1.0 7.8563e-02 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatGetRowIJ 2 1.0 3.6100e-06 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatView 2 1.0 5.4618e+01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 56 0 0 0 0 56 0 0 0 0 0 VecView 2 1.0 7.6981e-01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 1 0 0 0 0 1 0 0 0 0 0 VecMDot 1 1.0 1.7635e-04 1.0 3.07e+05 1.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 1742 VecNorm 3 1.0 5.4832e-04 1.0 9.22e+05 1.0 0.0e+00 0.0e+00 0.0e+00 0 1 0 0 0 0 1 0 0 0 1681 VecScale 1 1.0 1.5216e-04 1.0 1.54e+05 1.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 1009 VecCopy 1 1.0 1.5406e-04 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecSet 21 1.0 7.1106e-03 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecAYPX 1 1.0 1.9940e-04 1.0 1.54e+05 1.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 770 VecWAXPY 1 1.0 7.8802e-04 1.0 1.54e+05 1.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 195 KSPSetUp 2 1.0 6.8995e-03 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 PCSetUp 2 1.0 1.3190e+01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 14 0 0 0 0 14 0 0 0 0 0 PCApply 1 1.0 2.2728e-01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 ------------------------------------------------------------------------------------------------------------------------ Memory usage is given in bytes: Object Type Creations Destructions Memory Descendants' Mem. Reports information only for process 0. --- Event Stage 0: Main Stage Matrix 3 1 2872 0. Vector 32 20 17234800 0. Index Set 4 4 3200 0. IS L to G Mapping 2 0 0 0. Vec Scatter 2 0 0 0. Viewer 6 4 3392 0. Krylov Solver 2 1 61676 0. Preconditioner 2 1 1432 0. ======================================================================================================================== Average time to get PetscTime(): 3.36e-08 #PETSc Option Table entries: -ksp_converged_reason -ksp_error_if_not_converged -ksp_monitor_singular_value -ksp_monitor_true_residual -log_view -pc_hypre_boomeramg_agg_nl 4 -pc_hypre_boomeramg_agg_num_paths 5 -pc_hypre_boomeramg_coarsen_type HMIS -pc_hypre_boomeramg_eu_bj true -pc_hypre_boomeramg_interp_type ext+i -pc_hypre_boomeramg_P_max 2 -pc_hypre_boomeramg_relax_type_all sequential-Gauss-Seidel -pc_hypre_boomeramg_smooth_type Euclid -pc_hypre_boomeramg_strong_threshold 0.7 -pc_hypre_boomeramg_truncfactor 0.3 #End of PETSc Option Table entries Compiled without FORTRAN kernels Compiled with full precision matrices (default) sizeof(short) 2 sizeof(int) 4 sizeof(long) 8 sizeof(void*) 8 sizeof(PetscScalar) 8 sizeof(PetscInt) 4 Configure options: PETSC_ARCH=arch-linux2-c-opt-gcc7-hypre --with-blaslapack-dir=/opt/lapack/3.8.0-gcc.7.4.0/lib64 --with-debugging=0 COPTFLAGS=-O3 CXXOPTFLAGS=-O3 FOPTFLAGS=-O3 --with-valgrind=1 --with-valgrind-dir=/opt/valgrind/3.14.0/ --prefix=/opt/petsc/3.10.3_gcc-7.4.0 --with-mpi=1 --download-hypre ----------------------------------------- Libraries compiled on 2020-06-26 12:35:46 on xxx Machine characteristics: Linux-3.10.0-1127.8.2.el7.x86_64-x86_64-with-centos-7.6.1810-Core Using PETSc directory: /opt/petsc/3.10.3_gcc-7.4.0 Using PETSc arch: ----------------------------------------- Using C compiler: mpicc -fPIC -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas -fstack-protector -fvisibility=hidden -O3 Using Fortran compiler: mpif90 -fPIC -Wall -ffree-line-length-0 -Wno-unused-dummy-argument -O3 ----------------------------------------- Using include paths: -I/opt/petsc/3.10.3_gcc-7.4.0/include -I/opt/valgrind/3.14.0/include ----------------------------------------- Using C linker: mpicc Using Fortran linker: mpif90 Using libraries: -Wl,-rpath,/opt/petsc/3.10.3_gcc-7.4.0/lib -L/opt/petsc/3.10.3_gcc-7.4.0/lib -lpetsc -Wl,-rpath,/opt/petsc/3.10.3_gcc-7.4.0/lib -L/opt/petsc/3.10.3_gcc-7.4.0/lib -Wl,-rpath,/opt/lapack/3.8.0-gcc.7.4.0/lib64 -L/opt/lapack/3.8.0-gcc.7.4.0/lib64 -Wl,-rpath,/opt/openmpi4.0.0-gcc7.4.0_slurm/lib -L/opt/openmpi4.0.0-gcc7.4.0_slurm/lib -Wl,-rpath,/opt/gnu/7.4/lib/gcc/x86_64-pc-linux-gnu/7.4.0 -L/opt/gnu/7.4/lib/gcc/x86_64-pc-linux-gnu/7.4.0 -Wl,-rpath,/opt/gnu/7.4/lib/gcc -L/opt/gnu/7.4/lib/gcc -Wl,-rpath,/opt/gnu/7.4/lib64 -L/opt/gnu/7.4/lib64 -Wl,-rpath,/opt/gnu/7.4/lib -L/opt/gnu/7.4/lib -lHYPRE -llapack -lblas -lm -lX11 -lstdc++ -ldl -lmpi_usempif08 -lmpi_usempi_ignore_tkr -lmpi_mpifh -lmpi -lgfortran -lm -lgfortran -lm -lgcc_s -lquadmath -lpthread -lstdc++ -ldl ----------------------------------------- -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From mfadams at lbl.gov Wed Jul 1 10:04:43 2020 From: mfadams at lbl.gov (Mark Adams) Date: Wed, 1 Jul 2020 11:04:43 -0400 Subject: [petsc-users] DIVERGED_NANORINF when using HYPRE/BoomerAMG In-Reply-To: References: Message-ID: did you use: -pc_gamg_agg_nsmooths 0 As your problems become more advection dominated you will probably want to use this. The default is 1. and you can do this from the command line: -mg_levels_ksp_type fgmres -mg_levels_pc_type ilu On Wed, Jul 1, 2020 at 9:22 AM Andrea Iob wrote: > Thanks for the answers. > > The matrix contains both the advection and the diffusion terms of the > Navier-Stokes equations. The case is subsonic (M=0.5), Reynolds number is > in the order of 5000 so I expect the hyperbolic features to be the dominant > ones. I understand this is not the type of problems AMG is designed for, > but I was hoping to still be able to solve the system. This is the first > time I use BoomerAMG and I'm not familar with all the options, however all > the tests I've done gave me INFs/NANs. > > You suggestion to use GAMG with custom smoothers seems to do the trick: > after setting the GAMG smoother of each level to FGMRES/ILU, the solver > converges in 93 iterations (overall it seems faster than ILU). Great! I've > set the smoothers like this: > > for (int i = 0; i < nLevels; ++i) { > KSP levelSmoother; > PCMGGetSmoother(preconditioner, i, &levelSmoother); > > PC levelSmootherPreconditioner; > KSPGetPC(levelSmoother, &levelSmootherPreconditioner); > PCSetType(levelSmootherPreconditioner, PCILU); > > KSPSetType(levelSmoother, KSPFGMRES); > } > > Thank you very much! > Best regards, > Andrea > > ------------------------------ > *From:* Mark Adams > *Sent:* Wednesday, July 1, 2020 2:15 PM > *To:* Matthew Knepley > *Cc:* Andrea Iob ; petsc-users at mcs.anl.gov < > petsc-users at mcs.anl.gov> > *Subject:* Re: [petsc-users] DIVERGED_NANORINF when using HYPRE/BoomerAMG > > As Matt said AMG does not work for everything out-of-the-box. > > Hypre is pretty robust and I am surprised that every option gives > INFs/NANs. But if it solves your stretched Lapacian then it's probably > hooked up correctly. > > You might try a non-adjoint Navier-Stokes problem, if you can. > > You could try 'gamg' with the fgmres/ilu solver that works, as your > smoother (use -ksp_view to check that you get everything set correctly). > Use fgmres as the outer Krylov method and use -pc_gamg_agg_nsmooths 0. > > Mark > > > On Tue, Jun 30, 2020 at 9:16 AM Matthew Knepley wrote: > > On Tue, Jun 30, 2020 at 7:59 AM Andrea Iob wrote: > > Hi, > > I'm trying to solve a linear system using HYPRE/BoomerAMG as > preconditioner. The system comes from a two-dimensional adjoint > Navier-Stokes problem. > > > Do you expect the system you are solving to be elliptic? BoomerAMG is > designed for elliptic systems, and can easily fail if applied > to a more general system, say one with zeros on the diagonal. > > Thanks, > > Matt > > > The mesh is structured (6400 cells) and there are 24 DoFs for each cell > (the matrix has a total of 153600 rows). The condition number of the matrix > should be in the order of 1e9. Using ILU preconditioner and FGMRES, the > system is correctly solved (it takes 800 iterations to reach a residual of > 1e-12). However, replacing ILU with HYPRE/BoomerAMG, the solver stops right > after the first iteration (DIVERGED_NANORINF). I've tried different > BoomerAMG options and different solvers (including Richardson to use > BoomerAMG without a Krylov method), but I always get the same > DIVERGED_NANORINF error. > > Using the same HYPRE/BoomerAMG + FGMERS setup on another problem > (Laplacian on a highly stretched grid), the solver reaches convergence very > quickly without any problems. > > At the bottom of this mail, you'll find the log of a run that stops with > DIVERGED_NANORINF. Do you spot something in my setup that may explain why > the solver is not converging? > > Thanks. Best regards, > Andrea > > ----------------------------------------- > Assembly started... > Reading matrix file... > - Number of rows (N) = 153600 > - Number of non-zero entries (NNZ) = 18247680 > Assembly completed. > Time elapsed 83.3231s > Set up started... > Set up completed. > Time elapsed 6.62441s > Solution started... > 0 KSP unpreconditioned resid norm 7.736650641501e-01 true resid norm > 7.736650641501e-01 ||r(i)||/||b|| 1.000000000000e+00 > 0 KSP Residual norm 7.736650641501e-01 % max 1.000000000000e+00 min > 1.000000000000e+00 max/min 1.000000000000e+00 > [0]PETSC ERROR: --------------------- Error Message > -------------------------------------------------------------- > [0]PETSC ERROR: > [0]PETSC ERROR: KSPSolve has not converged due to Nan or Inf inner product > [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html > for trouble shooting. > [0]PETSC ERROR: Petsc Release Version 3.10.3, unknown > [0]PETSC ERROR: bitpit_system_solver on a named xxx by xxx Tue Jun 30 > 09:42:09 2020 > [0]PETSC ERROR: Configure options PETSC_ARCH=arch-linux2-c-opt-gcc7-hypre > --with-blaslapack-dir=/opt/lapack/3.8.0-gcc.7.4.0/lib64 --with-debugging=0 > COPTFLAGS=-O3 CXXOPTFLAGS=-O3 FOPTFLAGS=-O3 --with-valgrind=1 > --with-valgrind-dir=/opt/valgrind/3.14.0/ > --prefix=/opt/petsc/3.10.3_gcc-7.4.0 --with-mpi=1 --download-hypre > [0]PETSC ERROR: #1 KSPGMRESClassicalGramSchmidtOrthogonalization() line 67 > in /root/InstallSources/petsc/src/ksp/ksp/impls/gmres/borthog2.c > [0]PETSC ERROR: #2 KSPFGMRESCycle() line 175 in > /root/InstallSources/petsc/src/ksp/ksp/impls/gmres/fgmres/fgmres.c > [0]PETSC ERROR: #3 KSPSolve_FGMRES() line 291 in > /root/InstallSources/petsc/src/ksp/ksp/impls/gmres/fgmres/fgmres.c > [0]PETSC ERROR: #4 KSPSolve() line 780 in > /root/InstallSources/petsc/src/ksp/ksp/interface/itfunc.c > KSP Object: 1 MPI processes > type: fgmres > restart=45, using Classical (unmodified) Gram-Schmidt > Orthogonalization with no iterative refinement > happy breakdown tolerance 1e-30 > maximum iterations=10000, nonzero initial guess > tolerances: relative=1e-12, absolute=1e-50, divergence=10000. > right preconditioning > using UNPRECONDITIONED norm type for convergence test > PC Object: 1 MPI processes > type: hypre > HYPRE BoomerAMG preconditioning > Cycle type V > Maximum number of levels 25 > Maximum number of iterations PER hypre call 1 > Convergence tolerance PER hypre call 0. > Threshold for strong coupling 0.7 > Interpolation truncation factor 0.3 > Interpolation: max elements per row 2 > Number of levels of aggressive coarsening 4 > Number of paths for aggressive coarsening 5 > Maximum row sums 0.9 > Sweeps down 1 > Sweeps up 1 > Sweeps on coarse 1 > Relax down sequential-Gauss-Seidel > Relax up sequential-Gauss-Seidel > Relax on coarse Gaussian-elimination > Relax weight (all) 1. > Outer relax weight (all) 1. > Using CF-relaxation > Smooth type Euclid > Smooth num levels 25 > Euclid ILU(k) levels 0 > Euclid ILU(k) drop tolerance 0. > Euclid ILU use Block-Jacobi? 1 > Measure type local > Coarsen type HMIS > Interpolation type ext+i > linear system matrix = precond matrix: > Mat Object: Mat_0x1e88040_0 1 MPI processes > type: seqaij > rows=153600, cols=153600, bs=24 > total: nonzeros=18247680, allocated nonzeros=18247680 > total number of mallocs used during MatSetValues calls =0 > using I-node routines: found 32000 nodes, limit used is 5 > Solution completed. > Time elapsed 6.93336s > Solution information... > Error (L2) : 32.8359 > Error (Linf) : 1.93278 > > ************************************************************************************************************************ > *** WIDEN YOUR WINDOW TO 120 CHARACTERS. Use 'enscript -r > -fCourier9' to print this document *** > > ************************************************************************************************************************ > > ---------------------------------------------- PETSc Performance Summary: > ---------------------------------------------- > > bitpit_system_solver on a named xxx with 1 processor, by xxx Tue Jun 30 > 09:43:46 2020 > Using Petsc Release Version 3.10.3, unknown > > Max Max/Min Avg Total > Time (sec): 9.718e+01 1.000 9.718e+01 > Objects: 5.300e+01 1.000 5.300e+01 > Flop: 1.107e+08 1.000 1.107e+08 1.107e+08 > Flop/sec: 1.139e+06 1.000 1.139e+06 1.139e+06 > MPI Messages: 0.000e+00 0.000 0.000e+00 0.000e+00 > MPI Message Lengths: 0.000e+00 0.000 0.000e+00 0.000e+00 > MPI Reductions: 0.000e+00 0.000 > > Flop counting convention: 1 flop = 1 real number operation of type > (multiply/divide/add/subtract) > e.g., VecAXPY() for real vectors of length N > --> 2N flop > and VecAXPY() for complex vectors of length N > --> 8N flop > > Summary of Stages: ----- Time ------ ----- Flop ------ --- Messages > --- -- Message Lengths -- -- Reductions -- > Avg %Total Avg %Total Count > %Total Avg %Total Count %Total > 0: Main Stage: 9.7178e+01 100.0% 1.1071e+08 100.0% 0.000e+00 > 0.0% 0.000e+00 0.0% 0.000e+00 0.0% > > > ------------------------------------------------------------------------------------------------------------------------ > See the 'Profiling' chapter of the users' manual for details on > interpreting output. > Phase summary info: > Count: number of times phase was executed > Time and Flop: Max - maximum over all processors > Ratio - ratio of maximum to minimum over all processors > Mess: number of messages sent > AvgLen: average message length (bytes) > Reduct: number of global reductions > Global: entire computation > Stage: stages of a computation. Set stages with PetscLogStagePush() and > PetscLogStagePop(). > %T - percent time in this phase %F - percent flop in this > phase > %M - percent messages in this phase %L - percent message lengths > in this phase > %R - percent reductions in this phase > Total Mflop/s: 10e-6 * (sum of flop over all processors)/(max time over > all processors) > > ------------------------------------------------------------------------------------------------------------------------ > Event Count Time (sec) > Flop --- Global --- --- Stage ---- Total > Max Ratio Max Ratio Max Ratio Mess AvgLen > Reduct %T %F %M %L %R %T %F %M %L %R Mflop/s > > ------------------------------------------------------------------------------------------------------------------------ > > --- Event Stage 0: Main Stage > > BuildTwoSidedF 2 1.0 1.8471e-04 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 > 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > MatMult 3 1.0 6.8688e-02 1.0 1.09e+08 1.0 0.0e+00 0.0e+00 > 0.0e+00 0 98 0 0 0 0 98 0 0 0 1587 > MatConvert 2 1.0 3.8534e-01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 > 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > MatAssemblyBegin 3 1.0 2.8112e-04 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 > 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > MatAssemblyEnd 3 1.0 7.8563e-02 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 > 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > MatGetRowIJ 2 1.0 3.6100e-06 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 > 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > MatView 2 1.0 5.4618e+01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 > 0.0e+00 56 0 0 0 0 56 0 0 0 0 0 > VecView 2 1.0 7.6981e-01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 > 0.0e+00 1 0 0 0 0 1 0 0 0 0 0 > VecMDot 1 1.0 1.7635e-04 1.0 3.07e+05 1.0 0.0e+00 0.0e+00 > 0.0e+00 0 0 0 0 0 0 0 0 0 0 1742 > VecNorm 3 1.0 5.4832e-04 1.0 9.22e+05 1.0 0.0e+00 0.0e+00 > 0.0e+00 0 1 0 0 0 0 1 0 0 0 1681 > VecScale 1 1.0 1.5216e-04 1.0 1.54e+05 1.0 0.0e+00 0.0e+00 > 0.0e+00 0 0 0 0 0 0 0 0 0 0 1009 > VecCopy 1 1.0 1.5406e-04 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 > 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > VecSet 21 1.0 7.1106e-03 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 > 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > VecAYPX 1 1.0 1.9940e-04 1.0 1.54e+05 1.0 0.0e+00 0.0e+00 > 0.0e+00 0 0 0 0 0 0 0 0 0 0 770 > VecWAXPY 1 1.0 7.8802e-04 1.0 1.54e+05 1.0 0.0e+00 0.0e+00 > 0.0e+00 0 0 0 0 0 0 0 0 0 0 195 > KSPSetUp 2 1.0 6.8995e-03 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 > 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > PCSetUp 2 1.0 1.3190e+01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 > 0.0e+00 14 0 0 0 0 14 0 0 0 0 0 > PCApply 1 1.0 2.2728e-01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 > 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > > ------------------------------------------------------------------------------------------------------------------------ > > Memory usage is given in bytes: > > Object Type Creations Destructions Memory Descendants' Mem. > Reports information only for process 0. > > --- Event Stage 0: Main Stage > > Matrix 3 1 2872 0. > Vector 32 20 17234800 0. > Index Set 4 4 3200 0. > IS L to G Mapping 2 0 0 0. > Vec Scatter 2 0 0 0. > Viewer 6 4 3392 0. > Krylov Solver 2 1 61676 0. > Preconditioner 2 1 1432 0. > > ======================================================================================================================== > Average time to get PetscTime(): 3.36e-08 > #PETSc Option Table entries: > -ksp_converged_reason > -ksp_error_if_not_converged > -ksp_monitor_singular_value > -ksp_monitor_true_residual > -log_view > -pc_hypre_boomeramg_agg_nl 4 > -pc_hypre_boomeramg_agg_num_paths 5 > -pc_hypre_boomeramg_coarsen_type HMIS > -pc_hypre_boomeramg_eu_bj true > -pc_hypre_boomeramg_interp_type ext+i > -pc_hypre_boomeramg_P_max 2 > -pc_hypre_boomeramg_relax_type_all sequential-Gauss-Seidel > -pc_hypre_boomeramg_smooth_type Euclid > -pc_hypre_boomeramg_strong_threshold 0.7 > -pc_hypre_boomeramg_truncfactor 0.3 > #End of PETSc Option Table entries > Compiled without FORTRAN kernels > Compiled with full precision matrices (default) > sizeof(short) 2 sizeof(int) 4 sizeof(long) 8 sizeof(void*) 8 > sizeof(PetscScalar) 8 sizeof(PetscInt) 4 > Configure options: PETSC_ARCH=arch-linux2-c-opt-gcc7-hypre > --with-blaslapack-dir=/opt/lapack/3.8.0-gcc.7.4.0/lib64 --with-debugging=0 > COPTFLAGS=-O3 CXXOPTFLAGS=-O3 FOPTFLAGS=-O3 --with-valgrind=1 > --with-valgrind-dir=/opt/valgrind/3.14.0/ > --prefix=/opt/petsc/3.10.3_gcc-7.4.0 --with-mpi=1 --download-hypre > ----------------------------------------- > Libraries compiled on 2020-06-26 12:35:46 on xxx > Machine characteristics: > Linux-3.10.0-1127.8.2.el7.x86_64-x86_64-with-centos-7.6.1810-Core > Using PETSc directory: /opt/petsc/3.10.3_gcc-7.4.0 > Using PETSc arch: > ----------------------------------------- > > Using C compiler: mpicc -fPIC -Wall -Wwrite-strings -Wno-strict-aliasing > -Wno-unknown-pragmas -fstack-protector -fvisibility=hidden -O3 > Using Fortran compiler: mpif90 -fPIC -Wall -ffree-line-length-0 > -Wno-unused-dummy-argument -O3 > ----------------------------------------- > > Using include paths: -I/opt/petsc/3.10.3_gcc-7.4.0/include > -I/opt/valgrind/3.14.0/include > ----------------------------------------- > > Using C linker: mpicc > Using Fortran linker: mpif90 > Using libraries: -Wl,-rpath,/opt/petsc/3.10.3_gcc-7.4.0/lib > -L/opt/petsc/3.10.3_gcc-7.4.0/lib -lpetsc > -Wl,-rpath,/opt/petsc/3.10.3_gcc-7.4.0/lib > -L/opt/petsc/3.10.3_gcc-7.4.0/lib > -Wl,-rpath,/opt/lapack/3.8.0-gcc.7.4.0/lib64 > -L/opt/lapack/3.8.0-gcc.7.4.0/lib64 > -Wl,-rpath,/opt/openmpi4.0.0-gcc7.4.0_slurm/lib > -L/opt/openmpi4.0.0-gcc7.4.0_slurm/lib > -Wl,-rpath,/opt/gnu/7.4/lib/gcc/x86_64-pc-linux-gnu/7.4.0 > -L/opt/gnu/7.4/lib/gcc/x86_64-pc-linux-gnu/7.4.0 > -Wl,-rpath,/opt/gnu/7.4/lib/gcc -L/opt/gnu/7.4/lib/gcc > -Wl,-rpath,/opt/gnu/7.4/lib64 -L/opt/gnu/7.4/lib64 > -Wl,-rpath,/opt/gnu/7.4/lib -L/opt/gnu/7.4/lib -lHYPRE -llapack -lblas -lm > -lX11 -lstdc++ -ldl -lmpi_usempif08 -lmpi_usempi_ignore_tkr -lmpi_mpifh > -lmpi -lgfortran -lm -lgfortran -lm -lgcc_s -lquadmath -lpthread -lstdc++ > -ldl > ----------------------------------------- > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From andrea_iob at hotmail.com Wed Jul 1 10:14:02 2020 From: andrea_iob at hotmail.com (Andrea Iob) Date: Wed, 1 Jul 2020 15:14:02 +0000 Subject: [petsc-users] DIVERGED_NANORINF when using HYPRE/BoomerAMG In-Reply-To: References: , Message-ID: Yes, I did use "-pc_gamg_agg_nsmooths 0". Andrea ________________________________ From: Mark Adams Sent: Wednesday, July 1, 2020 5:04 PM To: Andrea Iob Cc: Matthew Knepley ; petsc-users at mcs.anl.gov Subject: Re: [petsc-users] DIVERGED_NANORINF when using HYPRE/BoomerAMG did you use: -pc_gamg_agg_nsmooths 0 As your problems become more advection dominated you will probably want to use this. The default is 1. and you can do this from the command line: -mg_levels_ksp_type fgmres -mg_levels_pc_type ilu On Wed, Jul 1, 2020 at 9:22 AM Andrea Iob > wrote: Thanks for the answers. The matrix contains both the advection and the diffusion terms of the Navier-Stokes equations. The case is subsonic (M=0.5), Reynolds number is in the order of 5000 so I expect the hyperbolic features to be the dominant ones. I understand this is not the type of problems AMG is designed for, but I was hoping to still be able to solve the system. This is the first time I use BoomerAMG and I'm not familar with all the options, however all the tests I've done gave me INFs/NANs. You suggestion to use GAMG with custom smoothers seems to do the trick: after setting the GAMG smoother of each level to FGMRES/ILU, the solver converges in 93 iterations (overall it seems faster than ILU). Great! I've set the smoothers like this: for (int i = 0; i < nLevels; ++i) { KSP levelSmoother; PCMGGetSmoother(preconditioner, i, &levelSmoother); PC levelSmootherPreconditioner; KSPGetPC(levelSmoother, &levelSmootherPreconditioner); PCSetType(levelSmootherPreconditioner, PCILU); KSPSetType(levelSmoother, KSPFGMRES); } Thank you very much! Best regards, Andrea ________________________________ From: Mark Adams > Sent: Wednesday, July 1, 2020 2:15 PM To: Matthew Knepley > Cc: Andrea Iob >; petsc-users at mcs.anl.gov > Subject: Re: [petsc-users] DIVERGED_NANORINF when using HYPRE/BoomerAMG As Matt said AMG does not work for everything out-of-the-box. Hypre is pretty robust and I am surprised that every option gives INFs/NANs. But if it solves your stretched Lapacian then it's probably hooked up correctly. You might try a non-adjoint Navier-Stokes problem, if you can. You could try 'gamg' with the fgmres/ilu solver that works, as your smoother (use -ksp_view to check that you get everything set correctly). Use fgmres as the outer Krylov method and use -pc_gamg_agg_nsmooths 0. Mark On Tue, Jun 30, 2020 at 9:16 AM Matthew Knepley > wrote: On Tue, Jun 30, 2020 at 7:59 AM Andrea Iob > wrote: Hi, I'm trying to solve a linear system using HYPRE/BoomerAMG as preconditioner. The system comes from a two-dimensional adjoint Navier-Stokes problem. Do you expect the system you are solving to be elliptic? BoomerAMG is designed for elliptic systems, and can easily fail if applied to a more general system, say one with zeros on the diagonal. Thanks, Matt The mesh is structured (6400 cells) and there are 24 DoFs for each cell (the matrix has a total of 153600 rows). The condition number of the matrix should be in the order of 1e9. Using ILU preconditioner and FGMRES, the system is correctly solved (it takes 800 iterations to reach a residual of 1e-12). However, replacing ILU with HYPRE/BoomerAMG, the solver stops right after the first iteration (DIVERGED_NANORINF). I've tried different BoomerAMG options and different solvers (including Richardson to use BoomerAMG without a Krylov method), but I always get the same DIVERGED_NANORINF error. Using the same HYPRE/BoomerAMG + FGMERS setup on another problem (Laplacian on a highly stretched grid), the solver reaches convergence very quickly without any problems. At the bottom of this mail, you'll find the log of a run that stops with DIVERGED_NANORINF. Do you spot something in my setup that may explain why the solver is not converging? Thanks. Best regards, Andrea ----------------------------------------- Assembly started... Reading matrix file... - Number of rows (N) = 153600 - Number of non-zero entries (NNZ) = 18247680 Assembly completed. Time elapsed 83.3231s Set up started... Set up completed. Time elapsed 6.62441s Solution started... 0 KSP unpreconditioned resid norm 7.736650641501e-01 true resid norm 7.736650641501e-01 ||r(i)||/||b|| 1.000000000000e+00 0 KSP Residual norm 7.736650641501e-01 % max 1.000000000000e+00 min 1.000000000000e+00 max/min 1.000000000000e+00 [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [0]PETSC ERROR: [0]PETSC ERROR: KSPSolve has not converged due to Nan or Inf inner product [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [0]PETSC ERROR: Petsc Release Version 3.10.3, unknown [0]PETSC ERROR: bitpit_system_solver on a named xxx by xxx Tue Jun 30 09:42:09 2020 [0]PETSC ERROR: Configure options PETSC_ARCH=arch-linux2-c-opt-gcc7-hypre --with-blaslapack-dir=/opt/lapack/3.8.0-gcc.7.4.0/lib64 --with-debugging=0 COPTFLAGS=-O3 CXXOPTFLAGS=-O3 FOPTFLAGS=-O3 --with-valgrind=1 --with-valgrind-dir=/opt/valgrind/3.14.0/ --prefix=/opt/petsc/3.10.3_gcc-7.4.0 --with-mpi=1 --download-hypre [0]PETSC ERROR: #1 KSPGMRESClassicalGramSchmidtOrthogonalization() line 67 in /root/InstallSources/petsc/src/ksp/ksp/impls/gmres/borthog2.c [0]PETSC ERROR: #2 KSPFGMRESCycle() line 175 in /root/InstallSources/petsc/src/ksp/ksp/impls/gmres/fgmres/fgmres.c [0]PETSC ERROR: #3 KSPSolve_FGMRES() line 291 in /root/InstallSources/petsc/src/ksp/ksp/impls/gmres/fgmres/fgmres.c [0]PETSC ERROR: #4 KSPSolve() line 780 in /root/InstallSources/petsc/src/ksp/ksp/interface/itfunc.c KSP Object: 1 MPI processes type: fgmres restart=45, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement happy breakdown tolerance 1e-30 maximum iterations=10000, nonzero initial guess tolerances: relative=1e-12, absolute=1e-50, divergence=10000. right preconditioning using UNPRECONDITIONED norm type for convergence test PC Object: 1 MPI processes type: hypre HYPRE BoomerAMG preconditioning Cycle type V Maximum number of levels 25 Maximum number of iterations PER hypre call 1 Convergence tolerance PER hypre call 0. Threshold for strong coupling 0.7 Interpolation truncation factor 0.3 Interpolation: max elements per row 2 Number of levels of aggressive coarsening 4 Number of paths for aggressive coarsening 5 Maximum row sums 0.9 Sweeps down 1 Sweeps up 1 Sweeps on coarse 1 Relax down sequential-Gauss-Seidel Relax up sequential-Gauss-Seidel Relax on coarse Gaussian-elimination Relax weight (all) 1. Outer relax weight (all) 1. Using CF-relaxation Smooth type Euclid Smooth num levels 25 Euclid ILU(k) levels 0 Euclid ILU(k) drop tolerance 0. Euclid ILU use Block-Jacobi? 1 Measure type local Coarsen type HMIS Interpolation type ext+i linear system matrix = precond matrix: Mat Object: Mat_0x1e88040_0 1 MPI processes type: seqaij rows=153600, cols=153600, bs=24 total: nonzeros=18247680, allocated nonzeros=18247680 total number of mallocs used during MatSetValues calls =0 using I-node routines: found 32000 nodes, limit used is 5 Solution completed. Time elapsed 6.93336s Solution information... Error (L2) : 32.8359 Error (Linf) : 1.93278 ************************************************************************************************************************ *** WIDEN YOUR WINDOW TO 120 CHARACTERS. Use 'enscript -r -fCourier9' to print this document *** ************************************************************************************************************************ ---------------------------------------------- PETSc Performance Summary: ---------------------------------------------- bitpit_system_solver on a named xxx with 1 processor, by xxx Tue Jun 30 09:43:46 2020 Using Petsc Release Version 3.10.3, unknown Max Max/Min Avg Total Time (sec): 9.718e+01 1.000 9.718e+01 Objects: 5.300e+01 1.000 5.300e+01 Flop: 1.107e+08 1.000 1.107e+08 1.107e+08 Flop/sec: 1.139e+06 1.000 1.139e+06 1.139e+06 MPI Messages: 0.000e+00 0.000 0.000e+00 0.000e+00 MPI Message Lengths: 0.000e+00 0.000 0.000e+00 0.000e+00 MPI Reductions: 0.000e+00 0.000 Flop counting convention: 1 flop = 1 real number operation of type (multiply/divide/add/subtract) e.g., VecAXPY() for real vectors of length N --> 2N flop and VecAXPY() for complex vectors of length N --> 8N flop Summary of Stages: ----- Time ------ ----- Flop ------ --- Messages --- -- Message Lengths -- -- Reductions -- Avg %Total Avg %Total Count %Total Avg %Total Count %Total 0: Main Stage: 9.7178e+01 100.0% 1.1071e+08 100.0% 0.000e+00 0.0% 0.000e+00 0.0% 0.000e+00 0.0% ------------------------------------------------------------------------------------------------------------------------ See the 'Profiling' chapter of the users' manual for details on interpreting output. Phase summary info: Count: number of times phase was executed Time and Flop: Max - maximum over all processors Ratio - ratio of maximum to minimum over all processors Mess: number of messages sent AvgLen: average message length (bytes) Reduct: number of global reductions Global: entire computation Stage: stages of a computation. Set stages with PetscLogStagePush() and PetscLogStagePop(). %T - percent time in this phase %F - percent flop in this phase %M - percent messages in this phase %L - percent message lengths in this phase %R - percent reductions in this phase Total Mflop/s: 10e-6 * (sum of flop over all processors)/(max time over all processors) ------------------------------------------------------------------------------------------------------------------------ Event Count Time (sec) Flop --- Global --- --- Stage ---- Total Max Ratio Max Ratio Max Ratio Mess AvgLen Reduct %T %F %M %L %R %T %F %M %L %R Mflop/s ------------------------------------------------------------------------------------------------------------------------ --- Event Stage 0: Main Stage BuildTwoSidedF 2 1.0 1.8471e-04 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatMult 3 1.0 6.8688e-02 1.0 1.09e+08 1.0 0.0e+00 0.0e+00 0.0e+00 0 98 0 0 0 0 98 0 0 0 1587 MatConvert 2 1.0 3.8534e-01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatAssemblyBegin 3 1.0 2.8112e-04 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatAssemblyEnd 3 1.0 7.8563e-02 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatGetRowIJ 2 1.0 3.6100e-06 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatView 2 1.0 5.4618e+01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 56 0 0 0 0 56 0 0 0 0 0 VecView 2 1.0 7.6981e-01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 1 0 0 0 0 1 0 0 0 0 0 VecMDot 1 1.0 1.7635e-04 1.0 3.07e+05 1.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 1742 VecNorm 3 1.0 5.4832e-04 1.0 9.22e+05 1.0 0.0e+00 0.0e+00 0.0e+00 0 1 0 0 0 0 1 0 0 0 1681 VecScale 1 1.0 1.5216e-04 1.0 1.54e+05 1.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 1009 VecCopy 1 1.0 1.5406e-04 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecSet 21 1.0 7.1106e-03 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecAYPX 1 1.0 1.9940e-04 1.0 1.54e+05 1.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 770 VecWAXPY 1 1.0 7.8802e-04 1.0 1.54e+05 1.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 195 KSPSetUp 2 1.0 6.8995e-03 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 PCSetUp 2 1.0 1.3190e+01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 14 0 0 0 0 14 0 0 0 0 0 PCApply 1 1.0 2.2728e-01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 ------------------------------------------------------------------------------------------------------------------------ Memory usage is given in bytes: Object Type Creations Destructions Memory Descendants' Mem. Reports information only for process 0. --- Event Stage 0: Main Stage Matrix 3 1 2872 0. Vector 32 20 17234800 0. Index Set 4 4 3200 0. IS L to G Mapping 2 0 0 0. Vec Scatter 2 0 0 0. Viewer 6 4 3392 0. Krylov Solver 2 1 61676 0. Preconditioner 2 1 1432 0. ======================================================================================================================== Average time to get PetscTime(): 3.36e-08 #PETSc Option Table entries: -ksp_converged_reason -ksp_error_if_not_converged -ksp_monitor_singular_value -ksp_monitor_true_residual -log_view -pc_hypre_boomeramg_agg_nl 4 -pc_hypre_boomeramg_agg_num_paths 5 -pc_hypre_boomeramg_coarsen_type HMIS -pc_hypre_boomeramg_eu_bj true -pc_hypre_boomeramg_interp_type ext+i -pc_hypre_boomeramg_P_max 2 -pc_hypre_boomeramg_relax_type_all sequential-Gauss-Seidel -pc_hypre_boomeramg_smooth_type Euclid -pc_hypre_boomeramg_strong_threshold 0.7 -pc_hypre_boomeramg_truncfactor 0.3 #End of PETSc Option Table entries Compiled without FORTRAN kernels Compiled with full precision matrices (default) sizeof(short) 2 sizeof(int) 4 sizeof(long) 8 sizeof(void*) 8 sizeof(PetscScalar) 8 sizeof(PetscInt) 4 Configure options: PETSC_ARCH=arch-linux2-c-opt-gcc7-hypre --with-blaslapack-dir=/opt/lapack/3.8.0-gcc.7.4.0/lib64 --with-debugging=0 COPTFLAGS=-O3 CXXOPTFLAGS=-O3 FOPTFLAGS=-O3 --with-valgrind=1 --with-valgrind-dir=/opt/valgrind/3.14.0/ --prefix=/opt/petsc/3.10.3_gcc-7.4.0 --with-mpi=1 --download-hypre ----------------------------------------- Libraries compiled on 2020-06-26 12:35:46 on xxx Machine characteristics: Linux-3.10.0-1127.8.2.el7.x86_64-x86_64-with-centos-7.6.1810-Core Using PETSc directory: /opt/petsc/3.10.3_gcc-7.4.0 Using PETSc arch: ----------------------------------------- Using C compiler: mpicc -fPIC -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas -fstack-protector -fvisibility=hidden -O3 Using Fortran compiler: mpif90 -fPIC -Wall -ffree-line-length-0 -Wno-unused-dummy-argument -O3 ----------------------------------------- Using include paths: -I/opt/petsc/3.10.3_gcc-7.4.0/include -I/opt/valgrind/3.14.0/include ----------------------------------------- Using C linker: mpicc Using Fortran linker: mpif90 Using libraries: -Wl,-rpath,/opt/petsc/3.10.3_gcc-7.4.0/lib -L/opt/petsc/3.10.3_gcc-7.4.0/lib -lpetsc -Wl,-rpath,/opt/petsc/3.10.3_gcc-7.4.0/lib -L/opt/petsc/3.10.3_gcc-7.4.0/lib -Wl,-rpath,/opt/lapack/3.8.0-gcc.7.4.0/lib64 -L/opt/lapack/3.8.0-gcc.7.4.0/lib64 -Wl,-rpath,/opt/openmpi4.0.0-gcc7.4.0_slurm/lib -L/opt/openmpi4.0.0-gcc7.4.0_slurm/lib -Wl,-rpath,/opt/gnu/7.4/lib/gcc/x86_64-pc-linux-gnu/7.4.0 -L/opt/gnu/7.4/lib/gcc/x86_64-pc-linux-gnu/7.4.0 -Wl,-rpath,/opt/gnu/7.4/lib/gcc -L/opt/gnu/7.4/lib/gcc -Wl,-rpath,/opt/gnu/7.4/lib64 -L/opt/gnu/7.4/lib64 -Wl,-rpath,/opt/gnu/7.4/lib -L/opt/gnu/7.4/lib -lHYPRE -llapack -lblas -lm -lX11 -lstdc++ -ldl -lmpi_usempif08 -lmpi_usempi_ignore_tkr -lmpi_mpifh -lmpi -lgfortran -lm -lgfortran -lm -lgcc_s -lquadmath -lpthread -lstdc++ -ldl ----------------------------------------- -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at petsc.dev Wed Jul 1 18:36:38 2020 From: bsmith at petsc.dev (Barry Smith) Date: Wed, 1 Jul 2020 18:36:38 -0500 Subject: [petsc-users] DIVERGED_NANORINF when using HYPRE/BoomerAMG In-Reply-To: References: Message-ID: You can run the hyper case with -fp_trap in a debugger and it may stop exactly in hypre where the inf/nan appears. Or if you know how to catch floating point exceptions in your debugger run in the debugger with that option to find where in hypre the inf/nan appears. Barry > On Jul 1, 2020, at 8:22 AM, Andrea Iob wrote: > > Thanks for the answers. > > The matrix contains both the advection and the diffusion terms of the Navier-Stokes equations. The case is subsonic (M=0.5), Reynolds number is in the order of 5000 so I expect the hyperbolic features to be the dominant ones. I understand this is not the type of problems AMG is designed for, but I was hoping to still be able to solve the system. This is the first time I use BoomerAMG and I'm not familar with all the options, however all the tests I've done gave me INFs/NANs. > > You suggestion to use GAMG with custom smoothers seems to do the trick: after setting the GAMG smoother of each level to FGMRES/ILU, the solver converges in 93 iterations (overall it seems faster than ILU). Great! I've set the smoothers like this: > > for (int i = 0; i < nLevels; ++i) { > KSP levelSmoother; > PCMGGetSmoother(preconditioner, i, &levelSmoother); > > PC levelSmootherPreconditioner; > KSPGetPC(levelSmoother, &levelSmootherPreconditioner); > PCSetType(levelSmootherPreconditioner, PCILU); > > KSPSetType(levelSmoother, KSPFGMRES); > } > > Thank you very much! > Best regards, > Andrea > > From: Mark Adams > Sent: Wednesday, July 1, 2020 2:15 PM > To: Matthew Knepley > Cc: Andrea Iob ; petsc-users at mcs.anl.gov > Subject: Re: [petsc-users] DIVERGED_NANORINF when using HYPRE/BoomerAMG > > As Matt said AMG does not work for everything out-of-the-box. > > Hypre is pretty robust and I am surprised that every option gives INFs/NANs. But if it solves your stretched Lapacian then it's probably hooked up correctly. > > You might try a non-adjoint Navier-Stokes problem, if you can. > > You could try 'gamg' with the fgmres/ilu solver that works, as your smoother (use -ksp_view to check that you get everything set correctly). Use fgmres as the outer Krylov method and use -pc_gamg_agg_nsmooths 0. > > Mark > > > On Tue, Jun 30, 2020 at 9:16 AM Matthew Knepley > wrote: > On Tue, Jun 30, 2020 at 7:59 AM Andrea Iob > wrote: > Hi, > > I'm trying to solve a linear system using HYPRE/BoomerAMG as preconditioner. The system comes from a two-dimensional adjoint Navier-Stokes problem. > > Do you expect the system you are solving to be elliptic? BoomerAMG is designed for elliptic systems, and can easily fail if applied > to a more general system, say one with zeros on the diagonal. > > Thanks, > > Matt > > The mesh is structured (6400 cells) and there are 24 DoFs for each cell (the matrix has a total of 153600 rows). The condition number of the matrix should be in the order of 1e9. Using ILU preconditioner and FGMRES, the system is correctly solved (it takes 800 iterations to reach a residual of 1e-12). However, replacing ILU with HYPRE/BoomerAMG, the solver stops right after the first iteration (DIVERGED_NANORINF). I've tried different BoomerAMG options and different solvers (including Richardson to use BoomerAMG without a Krylov method), but I always get the same DIVERGED_NANORINF error. > > Using the same HYPRE/BoomerAMG + FGMERS setup on another problem (Laplacian on a highly stretched grid), the solver reaches convergence very quickly without any problems. > > At the bottom of this mail, you'll find the log of a run that stops with DIVERGED_NANORINF. Do you spot something in my setup that may explain why the solver is not converging? > > Thanks. Best regards, > Andrea > > ----------------------------------------- > Assembly started... > Reading matrix file... > - Number of rows (N) = 153600 > - Number of non-zero entries (NNZ) = 18247680 > Assembly completed. > Time elapsed 83.3231s > Set up started... > Set up completed. > Time elapsed 6.62441s > Solution started... > 0 KSP unpreconditioned resid norm 7.736650641501e-01 true resid norm 7.736650641501e-01 ||r(i)||/||b|| 1.000000000000e+00 > 0 KSP Residual norm 7.736650641501e-01 % max 1.000000000000e+00 min 1.000000000000e+00 max/min 1.000000000000e+00 > [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- > [0]PETSC ERROR: > [0]PETSC ERROR: KSPSolve has not converged due to Nan or Inf inner product > [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. > [0]PETSC ERROR: Petsc Release Version 3.10.3, unknown > [0]PETSC ERROR: bitpit_system_solver on a named xxx by xxx Tue Jun 30 09:42:09 2020 > [0]PETSC ERROR: Configure options PETSC_ARCH=arch-linux2-c-opt-gcc7-hypre --with-blaslapack-dir=/opt/lapack/3.8.0-gcc.7.4.0/lib64 --with-debugging=0 COPTFLAGS=-O3 CXXOPTFLAGS=-O3 FOPTFLAGS=-O3 --with-valgrind=1 --with-valgrind-dir=/opt/valgrind/3.14.0/ --prefix=/opt/petsc/3.10.3_gcc-7.4.0 --with-mpi=1 --download-hypre > [0]PETSC ERROR: #1 KSPGMRESClassicalGramSchmidtOrthogonalization() line 67 in /root/InstallSources/petsc/src/ksp/ksp/impls/gmres/borthog2.c > [0]PETSC ERROR: #2 KSPFGMRESCycle() line 175 in /root/InstallSources/petsc/src/ksp/ksp/impls/gmres/fgmres/fgmres.c > [0]PETSC ERROR: #3 KSPSolve_FGMRES() line 291 in /root/InstallSources/petsc/src/ksp/ksp/impls/gmres/fgmres/fgmres.c > [0]PETSC ERROR: #4 KSPSolve() line 780 in /root/InstallSources/petsc/src/ksp/ksp/interface/itfunc.c > KSP Object: 1 MPI processes > type: fgmres > restart=45, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement > happy breakdown tolerance 1e-30 > maximum iterations=10000, nonzero initial guess > tolerances: relative=1e-12, absolute=1e-50, divergence=10000. > right preconditioning > using UNPRECONDITIONED norm type for convergence test > PC Object: 1 MPI processes > type: hypre > HYPRE BoomerAMG preconditioning > Cycle type V > Maximum number of levels 25 > Maximum number of iterations PER hypre call 1 > Convergence tolerance PER hypre call 0. > Threshold for strong coupling 0.7 > Interpolation truncation factor 0.3 > Interpolation: max elements per row 2 > Number of levels of aggressive coarsening 4 > Number of paths for aggressive coarsening 5 > Maximum row sums 0.9 > Sweeps down 1 > Sweeps up 1 > Sweeps on coarse 1 > Relax down sequential-Gauss-Seidel > Relax up sequential-Gauss-Seidel > Relax on coarse Gaussian-elimination > Relax weight (all) 1. > Outer relax weight (all) 1. > Using CF-relaxation > Smooth type Euclid > Smooth num levels 25 > Euclid ILU(k) levels 0 > Euclid ILU(k) drop tolerance 0. > Euclid ILU use Block-Jacobi? 1 > Measure type local > Coarsen type HMIS > Interpolation type ext+i > linear system matrix = precond matrix: > Mat Object: Mat_0x1e88040_0 1 MPI processes > type: seqaij > rows=153600, cols=153600, bs=24 > total: nonzeros=18247680, allocated nonzeros=18247680 > total number of mallocs used during MatSetValues calls =0 > using I-node routines: found 32000 nodes, limit used is 5 > Solution completed. > Time elapsed 6.93336s > Solution information... > Error (L2) : 32.8359 > Error (Linf) : 1.93278 > ************************************************************************************************************************ > *** WIDEN YOUR WINDOW TO 120 CHARACTERS. Use 'enscript -r -fCourier9' to print this document *** > ************************************************************************************************************************ > > ---------------------------------------------- PETSc Performance Summary: ---------------------------------------------- > > bitpit_system_solver on a named xxx with 1 processor, by xxx Tue Jun 30 09:43:46 2020 > Using Petsc Release Version 3.10.3, unknown > > Max Max/Min Avg Total > Time (sec): 9.718e+01 1.000 9.718e+01 > Objects: 5.300e+01 1.000 5.300e+01 > Flop: 1.107e+08 1.000 1.107e+08 1.107e+08 > Flop/sec: 1.139e+06 1.000 1.139e+06 1.139e+06 > MPI Messages: 0.000e+00 0.000 0.000e+00 0.000e+00 > MPI Message Lengths: 0.000e+00 0.000 0.000e+00 0.000e+00 > MPI Reductions: 0.000e+00 0.000 > > Flop counting convention: 1 flop = 1 real number operation of type (multiply/divide/add/subtract) > e.g., VecAXPY() for real vectors of length N --> 2N flop > and VecAXPY() for complex vectors of length N --> 8N flop > > Summary of Stages: ----- Time ------ ----- Flop ------ --- Messages --- -- Message Lengths -- -- Reductions -- > Avg %Total Avg %Total Count %Total Avg %Total Count %Total > 0: Main Stage: 9.7178e+01 100.0% 1.1071e+08 100.0% 0.000e+00 0.0% 0.000e+00 0.0% 0.000e+00 0.0% > > ------------------------------------------------------------------------------------------------------------------------ > See the 'Profiling' chapter of the users' manual for details on interpreting output. > Phase summary info: > Count: number of times phase was executed > Time and Flop: Max - maximum over all processors > Ratio - ratio of maximum to minimum over all processors > Mess: number of messages sent > AvgLen: average message length (bytes) > Reduct: number of global reductions > Global: entire computation > Stage: stages of a computation. Set stages with PetscLogStagePush() and PetscLogStagePop(). > %T - percent time in this phase %F - percent flop in this phase > %M - percent messages in this phase %L - percent message lengths in this phase > %R - percent reductions in this phase > Total Mflop/s: 10e-6 * (sum of flop over all processors)/(max time over all processors) > ------------------------------------------------------------------------------------------------------------------------ > Event Count Time (sec) Flop --- Global --- --- Stage ---- Total > Max Ratio Max Ratio Max Ratio Mess AvgLen Reduct %T %F %M %L %R %T %F %M %L %R Mflop/s > ------------------------------------------------------------------------------------------------------------------------ > > --- Event Stage 0: Main Stage > > BuildTwoSidedF 2 1.0 1.8471e-04 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > MatMult 3 1.0 6.8688e-02 1.0 1.09e+08 1.0 0.0e+00 0.0e+00 0.0e+00 0 98 0 0 0 0 98 0 0 0 1587 > MatConvert 2 1.0 3.8534e-01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > MatAssemblyBegin 3 1.0 2.8112e-04 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > MatAssemblyEnd 3 1.0 7.8563e-02 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > MatGetRowIJ 2 1.0 3.6100e-06 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > MatView 2 1.0 5.4618e+01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 56 0 0 0 0 56 0 0 0 0 0 > VecView 2 1.0 7.6981e-01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 1 0 0 0 0 1 0 0 0 0 0 > VecMDot 1 1.0 1.7635e-04 1.0 3.07e+05 1.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 1742 > VecNorm 3 1.0 5.4832e-04 1.0 9.22e+05 1.0 0.0e+00 0.0e+00 0.0e+00 0 1 0 0 0 0 1 0 0 0 1681 > VecScale 1 1.0 1.5216e-04 1.0 1.54e+05 1.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 1009 > VecCopy 1 1.0 1.5406e-04 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > VecSet 21 1.0 7.1106e-03 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > VecAYPX 1 1.0 1.9940e-04 1.0 1.54e+05 1.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 770 > VecWAXPY 1 1.0 7.8802e-04 1.0 1.54e+05 1.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 195 > KSPSetUp 2 1.0 6.8995e-03 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > PCSetUp 2 1.0 1.3190e+01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 14 0 0 0 0 14 0 0 0 0 0 > PCApply 1 1.0 2.2728e-01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > ------------------------------------------------------------------------------------------------------------------------ > > Memory usage is given in bytes: > > Object Type Creations Destructions Memory Descendants' Mem. > Reports information only for process 0. > > --- Event Stage 0: Main Stage > > Matrix 3 1 2872 0. > Vector 32 20 17234800 0. > Index Set 4 4 3200 0. > IS L to G Mapping 2 0 0 0. > Vec Scatter 2 0 0 0. > Viewer 6 4 3392 0. > Krylov Solver 2 1 61676 0. > Preconditioner 2 1 1432 0. > ======================================================================================================================== > Average time to get PetscTime(): 3.36e-08 > #PETSc Option Table entries: > -ksp_converged_reason > -ksp_error_if_not_converged > -ksp_monitor_singular_value > -ksp_monitor_true_residual > -log_view > -pc_hypre_boomeramg_agg_nl 4 > -pc_hypre_boomeramg_agg_num_paths 5 > -pc_hypre_boomeramg_coarsen_type HMIS > -pc_hypre_boomeramg_eu_bj true > -pc_hypre_boomeramg_interp_type ext+i > -pc_hypre_boomeramg_P_max 2 > -pc_hypre_boomeramg_relax_type_all sequential-Gauss-Seidel > -pc_hypre_boomeramg_smooth_type Euclid > -pc_hypre_boomeramg_strong_threshold 0.7 > -pc_hypre_boomeramg_truncfactor 0.3 > #End of PETSc Option Table entries > Compiled without FORTRAN kernels > Compiled with full precision matrices (default) > sizeof(short) 2 sizeof(int) 4 sizeof(long) 8 sizeof(void*) 8 sizeof(PetscScalar) 8 sizeof(PetscInt) 4 > Configure options: PETSC_ARCH=arch-linux2-c-opt-gcc7-hypre --with-blaslapack-dir=/opt/lapack/3.8.0-gcc.7.4.0/lib64 --with-debugging=0 COPTFLAGS=-O3 CXXOPTFLAGS=-O3 FOPTFLAGS=-O3 --with-valgrind=1 --with-valgrind-dir=/opt/valgrind/3.14.0/ --prefix=/opt/petsc/3.10.3_gcc-7.4.0 --with-mpi=1 --download-hypre > ----------------------------------------- > Libraries compiled on 2020-06-26 12:35:46 on xxx > Machine characteristics: Linux-3.10.0-1127.8.2.el7.x86_64-x86_64-with-centos-7.6.1810-Core > Using PETSc directory: /opt/petsc/3.10.3_gcc-7.4.0 > Using PETSc arch: > ----------------------------------------- > > Using C compiler: mpicc -fPIC -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas -fstack-protector -fvisibility=hidden -O3 > Using Fortran compiler: mpif90 -fPIC -Wall -ffree-line-length-0 -Wno-unused-dummy-argument -O3 > ----------------------------------------- > > Using include paths: -I/opt/petsc/3.10.3_gcc-7.4.0/include -I/opt/valgrind/3.14.0/include > ----------------------------------------- > > Using C linker: mpicc > Using Fortran linker: mpif90 > Using libraries: -Wl,-rpath,/opt/petsc/3.10.3_gcc-7.4.0/lib -L/opt/petsc/3.10.3_gcc-7.4.0/lib -lpetsc -Wl,-rpath,/opt/petsc/3.10.3_gcc-7.4.0/lib -L/opt/petsc/3.10.3_gcc-7.4.0/lib -Wl,-rpath,/opt/lapack/3.8.0-gcc.7.4.0/lib64 -L/opt/lapack/3.8.0-gcc.7.4.0/lib64 -Wl,-rpath,/opt/openmpi4.0.0-gcc7.4.0_slurm/lib -L/opt/openmpi4.0.0-gcc7.4.0_slurm/lib -Wl,-rpath,/opt/gnu/7.4/lib/gcc/x86_64-pc-linux-gnu/7.4.0 -L/opt/gnu/7.4/lib/gcc/x86_64-pc-linux-gnu/7.4.0 -Wl,-rpath,/opt/gnu/7.4/lib/gcc -L/opt/gnu/7.4/lib/gcc -Wl,-rpath,/opt/gnu/7.4/lib64 -L/opt/gnu/7.4/lib64 -Wl,-rpath,/opt/gnu/7.4/lib -L/opt/gnu/7.4/lib -lHYPRE -llapack -lblas -lm -lX11 -lstdc++ -ldl -lmpi_usempif08 -lmpi_usempi_ignore_tkr -lmpi_mpifh -lmpi -lgfortran -lm -lgfortran -lm -lgcc_s -lquadmath -lpthread -lstdc++ -ldl > ----------------------------------------- > > > -- > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From balay at mcs.anl.gov Wed Jul 1 21:37:52 2020 From: balay at mcs.anl.gov (Satish Balay) Date: Wed, 1 Jul 2020 21:37:52 -0500 (CDT) Subject: [petsc-users] petsc-3.13.3.tar.gz now available Message-ID: Dear PETSc users, The patch release petsc-3.13.3 is now available for download, with change list at 'PETSc-3.13 Changelog' http://www.mcs.anl.gov/petsc/download/index.html Satish From hongzhang at anl.gov Wed Jul 1 23:29:02 2020 From: hongzhang at anl.gov (Zhang, Hong) Date: Thu, 2 Jul 2020 04:29:02 +0000 Subject: [petsc-users] MatGetRow for global rows of a parallel matrix In-Reply-To: References: Message-ID: <81DE5F1A-EC69-49B2-A970-DD88945DB031@anl.gov> Dense matrix is stored in column-major order, so you might want to use MatGetColumnVector() to access columns of the matrix instead of the rows. For instance, you can get the n-th column out of the matrix and place it into a vector that shares the same parallel layout as the matrix with the following: MatDenseGetColumn(J,n,&xarr); VecPlaceArray(vec,xarr); ... VecResetArray(vec); MatDenseRestoreColumn(J,&xarr); Hong (Mr.) On Jun 10, 2020, at 8:08 AM, Eda Oktay > wrote: Dear Matt, Matthew Knepley >, 10 Haz 2020 ?ar, 16:03 tarihinde ?unu yazd?: On Wed, Jun 10, 2020 at 8:56 AM Eda Oktay wrote: Hi all, I am trying to get all the rows of a parallel matrix as individual vectors. For instance, if I have 72*4 matrix, I want to get 72 different vectors having size 4. As far as I understood, MatGetRow is only for local rows, so MatGetOwnershipRange is used, however, when I tried this one, I couldn't get the whole and desired row vectors. In MatGetRow explanation, it is written that I should use MatCreateSubMatrices first, then use MatGetRow. But I couldn't understand to which extent I should create submatrices. I just need to have all 72 rows as 72 different vectors each having 4 elements. 1) For sparse matrices, the storage is always divided by row, so that values can only be retrieved for local rows with MatGetRow() 2) Is this matrix sparse? It sounds like it is dense. Matrix is dense. 3) Are you asking to get all matrix values on all processes? If so, I think the easiest thing to do is first wrap a Vec around the values, then use VecScatterToAll(), then wrap each one in a MatDense again. Yes, I want all row vectors on all processes. In a dense matrix, should I still wrap a Vec around the values? I know I should use scatter but I couldn't even wrap a Vec around them. Thanks so much! Eda Thanks, Matt Thanks! Eda -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From eda.oktay at metu.edu.tr Thu Jul 2 04:30:51 2020 From: eda.oktay at metu.edu.tr (Eda Oktay) Date: Thu, 2 Jul 2020 12:30:51 +0300 Subject: [petsc-users] MatGetRow for global rows of a parallel matrix In-Reply-To: <81DE5F1A-EC69-49B2-A970-DD88945DB031@anl.gov> References: <81DE5F1A-EC69-49B2-A970-DD88945DB031@anl.gov> Message-ID: Thank you so much! Eda Zhang, Hong , 2 Tem 2020 Per, 07:29 tarihinde ?unu yazd?: > > Dense matrix is stored in column-major order, so you might want to use MatGetColumnVector() to access columns of the matrix instead of the rows. For instance, you can get the n-th column out of the matrix and place it into a vector that shares the same parallel layout as the matrix with the following: > > MatDenseGetColumn(J,n,&xarr); > VecPlaceArray(vec,xarr); > ... > VecResetArray(vec); > MatDenseRestoreColumn(J,&xarr); > > Hong (Mr.) > > On Jun 10, 2020, at 8:08 AM, Eda Oktay wrote: > > Dear Matt, > > Matthew Knepley , 10 Haz 2020 ?ar, 16:03 tarihinde > ?unu yazd?: > > > On Wed, Jun 10, 2020 at 8:56 AM Eda Oktay wrote: > > > Hi all, > > I am trying to get all the rows of a parallel matrix as individual > vectors. For instance, if I have 72*4 matrix, I want to get 72 > different vectors having size 4. > > As far as I understood, MatGetRow is only for local rows, so > MatGetOwnershipRange is used, however, when I tried this one, I > couldn't get the whole and desired row vectors. > > In MatGetRow explanation, it is written that I should use > MatCreateSubMatrices first, then use MatGetRow. But I couldn't > understand to which extent I should create submatrices. I just need to > have all 72 rows as 72 different vectors each having 4 elements. > > > > 1) For sparse matrices, the storage is always divided by row, so that values can only be retrieved for local rows with MatGetRow() > > 2) Is this matrix sparse? It sounds like it is dense. > > > Matrix is dense. > > > 3) Are you asking to get all matrix values on all processes? If so, I think the easiest thing to do is first wrap a Vec around the > values, then use VecScatterToAll(), then wrap each one in a MatDense again. > > > Yes, I want all row vectors on all processes. In a dense matrix, > should I still wrap a Vec around the values? I know I should use > scatter but I couldn't even wrap a Vec around them. > > Thanks so much! > > Eda > > > Thanks, > > Matt > > > Thanks! > > Eda > > > > > -- > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ > > From andrea_iob at hotmail.com Thu Jul 2 07:48:45 2020 From: andrea_iob at hotmail.com (Andrea Iob) Date: Thu, 2 Jul 2020 12:48:45 +0000 Subject: [petsc-users] DIVERGED_NANORINF when using HYPRE/BoomerAMG In-Reply-To: References: , Message-ID: I did some tests: depending on the smoother (set with the option "-pc_hypre_boomeramg_smooth_type"), the arithmetic exception is throw by different functions. Backtraces are at the bottom. Sorry for the lack of line numbers, but, at the moment, I don't have a version of the PETSc library compiled with debug symbols. >From my side, I'm quite happy with the performances of PCGAMG. The time it takes to solve the system went down from 60s (ILU) to 29s (PCGAMG with ILU+FGMRES as smoothers and "-pc_gamg_agg_nsmooths 0"). Since the problem I have with BoomerAMG seems related to the HYPRE package, maybe I can open an issue on their github repository. However, if you need additional information (or more tests) I will be glad to provide them. Andrea 1) Option not specified #0 0x00007ffff507ce5c in ddot_ () from /opt/lapack/3.8.0-gcc.7.4.0/lib64/libblas.so.3 #1 0x00007ffff65c278f in VecNorm_Seq () from /opt/petsc/3.10.3_gcc-7.4.0/lib/libpetsc.so.3.10 #2 0x00007ffff65eaf7e in VecNorm () from /opt/petsc/3.10.3_gcc-7.4.0/lib/libpetsc.so.3.10 #3 0x00007ffff6e3b241 in KSPFGMRESCycle () from /opt/petsc/3.10.3_gcc-7.4.0/lib/libpetsc.so.3.10 #4 0x00007ffff6e3c39d in KSPSolve_FGMRES () from /opt/petsc/3.10.3_gcc-7.4.0/lib/libpetsc.so.3.10 #5 0x00007ffff6e5832e in KSPSolve () from /opt/petsc/3.10.3_gcc-7.4.0/lib/libpetsc.so.3.10 #6 0x00007ffff7a59888 in bitpit::SystemSolver::solve (this=0x75dd30) at /xxx/bitpit/code/src/LA/system_solvers_large.cpp:678 2) Euclid #0 0x00007ffff720c40a in Factor_dhSolveSeq () from /opt/petsc/3.10.3_gcc-7.4.0/lib/libpetsc.so.3.10 #1 0x00007ffff720376a in Euclid_dhApply () from /opt/petsc/3.10.3_gcc-7.4.0/lib/libpetsc.so.3.10 #2 0x00007ffff71b6867 in HYPRE_EuclidSolve () from /opt/petsc/3.10.3_gcc-7.4.0/lib/libpetsc.so.3.10 #3 0x00007ffff7083953 in hypre_BoomerAMGCycle () from /opt/petsc/3.10.3_gcc-7.4.0/lib/libpetsc.so.3.10 #4 0x00007ffff706f4ce in hypre_BoomerAMGSolve () from /opt/petsc/3.10.3_gcc-7.4.0/lib/libpetsc.so.3.10 #5 0x00007ffff6d85cbb in PCApply_HYPRE () from /opt/petsc/3.10.3_gcc-7.4.0/lib/libpetsc.so.3.10 #6 0x00007ffff6db324b in PCApply () from /opt/petsc/3.10.3_gcc-7.4.0/lib/libpetsc.so.3.10 #7 0x00007ffff6e3b198 in KSPFGMRESCycle () from /opt/petsc/3.10.3_gcc-7.4.0/lib/libpetsc.so.3.10 #8 0x00007ffff6e3c39d in KSPSolve_FGMRES () from /opt/petsc/3.10.3_gcc-7.4.0/lib/libpetsc.so.3.10 #9 0x00007ffff6e5832e in KSPSolve () from /opt/petsc/3.10.3_gcc-7.4.0/lib/libpetsc.so.3.10 #10 0x00007ffff7a59888 in bitpit::SystemSolver::solve (this=0x75dd70) at /xxx/bitpit/code/src/LA/system_solvers_large.cpp:678 3) Schwarz-smoothers #0 0x00007ffff50897f4 in dtrsm_ () from /opt/lapack/3.8.0-gcc.7.4.0/lib64/libblas.so.3 #1 0x00007ffff550924b in dpotrs_ () from /opt/lapack/3.8.0-gcc.7.4.0/lib64/liblapack.so.3 #2 0x00007ffff70ed4bf in hypre_MPSchwarzSolve () from /opt/petsc/3.10.3_gcc-7.4.0/lib/libpetsc.so.3.10 #3 0x00007ffff70c9478 in hypre_SchwarzSolve () from /opt/petsc/3.10.3_gcc-7.4.0/lib/libpetsc.so.3.10 #4 0x00007ffff70836d1 in hypre_BoomerAMGCycle () from /opt/petsc/3.10.3_gcc-7.4.0/lib/libpetsc.so.3.10 #5 0x00007ffff706f4ce in hypre_BoomerAMGSolve () from /opt/petsc/3.10.3_gcc-7.4.0/lib/libpetsc.so.3.10 #6 0x00007ffff6d85cbb in PCApply_HYPRE () from /opt/petsc/3.10.3_gcc-7.4.0/lib/libpetsc.so.3.10 #7 0x00007ffff6db324b in PCApply () from /opt/petsc/3.10.3_gcc-7.4.0/lib/libpetsc.so.3.10 #8 0x00007ffff6e3b198 in KSPFGMRESCycle () from /opt/petsc/3.10.3_gcc-7.4.0/lib/libpetsc.so.3.10 #9 0x00007ffff6e3c39d in KSPSolve_FGMRES () from /opt/petsc/3.10.3_gcc-7.4.0/lib/libpetsc.so.3.10 #10 0x00007ffff6e5832e in KSPSolve () from /opt/petsc/3.10.3_gcc-7.4.0/lib/libpetsc.so.3.10 #11 0x00007ffff7a59888 in bitpit::SystemSolver::solve (this=0x75dd30) at /xxx/bitpit/code/src/LA/system_solvers_large.cpp:678 4) Pilut No errors but the solver doesn't converge: 0 KSP unpreconditioned resid norm 7.736650641501e-01 true resid norm 7.736650641501e-01 ||r(i)||/||b|| 1.000000000000e+00 0 KSP Residual norm 7.736650641501e-01 % max 1.000000000000e+00 min 1.000000000000e+00 max/min 1.000000000000e+00 1 KSP unpreconditioned resid norm 7.736650641501e-01 true resid norm 7.736650641501e-01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP Residual norm 7.736650641501e-01 % max 8.836643387885e+91 min 8.836643387885e+91 max/min 1.000000000000e+00 2 KSP unpreconditioned resid norm 7.736650641501e-01 true resid norm 7.736650641501e-01 ||r(i)||/||b|| 1.000000000000e+00 2 KSP Residual norm 7.736650641501e-01 % max 2.525644801510e+141 min 2.809949455965e+85 max/min 8.988221464797e+55 3 KSP unpreconditioned resid norm 7.736650641501e-01 true resid norm 7.736650641501e-01 ||r(i)||/||b|| 1.000000000000e+00 3 KSP Residual norm 7.736650641501e-01 % max 5.240099813864e+141 min 4.394901421200e+84 max/min 1.192313390372e+57 4 KSP unpreconditioned resid norm 7.736650641501e-01 true resid norm 7.736650641501e-01 ||r(i)||/||b|| 1.000000000000e+00 4 KSP Residual norm 7.736650641501e-01 % max 5.249800875338e+141 min 1.922154232776e+84 max/min 2.731206885389e+57 5 KSP unpreconditioned resid norm 7.736650641501e-01 true resid norm 7.736650641501e-01 ||r(i)||/||b|| 1.000000000000e+00 5 KSP Residual norm 7.736650641501e-01 % max 5.250261290126e+141 min 1.874815205276e+84 max/min 2.800415355792e+57 6 KSP unpreconditioned resid norm 7.736650641501e-01 true resid norm 7.736650641501e-01 ||r(i)||/||b|| 1.000000000000e+00 ________________________________ From: Barry Smith Sent: Thursday, July 2, 2020 1:36 AM To: Andrea Iob Cc: Mark Adams ; Matthew Knepley ; petsc-users at mcs.anl.gov Subject: Re: [petsc-users] DIVERGED_NANORINF when using HYPRE/BoomerAMG You can run the hyper case with -fp_trap in a debugger and it may stop exactly in hypre where the inf/nan appears. Or if you know how to catch floating point exceptions in your debugger run in the debugger with that option to find where in hypre the inf/nan appears. Barry On Jul 1, 2020, at 8:22 AM, Andrea Iob > wrote: Thanks for the answers. The matrix contains both the advection and the diffusion terms of the Navier-Stokes equations. The case is subsonic (M=0.5), Reynolds number is in the order of 5000 so I expect the hyperbolic features to be the dominant ones. I understand this is not the type of problems AMG is designed for, but I was hoping to still be able to solve the system. This is the first time I use BoomerAMG and I'm not familar with all the options, however all the tests I've done gave me INFs/NANs. You suggestion to use GAMG with custom smoothers seems to do the trick: after setting the GAMG smoother of each level to FGMRES/ILU, the solver converges in 93 iterations (overall it seems faster than ILU). Great! I've set the smoothers like this: for (int i = 0; i < nLevels; ++i) { KSP levelSmoother; PCMGGetSmoother(preconditioner, i, &levelSmoother); PC levelSmootherPreconditioner; KSPGetPC(levelSmoother, &levelSmootherPreconditioner); PCSetType(levelSmootherPreconditioner, PCILU); KSPSetType(levelSmoother, KSPFGMRES); } Thank you very much! Best regards, Andrea ________________________________ From: Mark Adams > Sent: Wednesday, July 1, 2020 2:15 PM To: Matthew Knepley > Cc: Andrea Iob >; petsc-users at mcs.anl.gov > Subject: Re: [petsc-users] DIVERGED_NANORINF when using HYPRE/BoomerAMG As Matt said AMG does not work for everything out-of-the-box. Hypre is pretty robust and I am surprised that every option gives INFs/NANs. But if it solves your stretched Lapacian then it's probably hooked up correctly. You might try a non-adjoint Navier-Stokes problem, if you can. You could try 'gamg' with the fgmres/ilu solver that works, as your smoother (use -ksp_view to check that you get everything set correctly). Use fgmres as the outer Krylov method and use -pc_gamg_agg_nsmooths 0. Mark On Tue, Jun 30, 2020 at 9:16 AM Matthew Knepley > wrote: On Tue, Jun 30, 2020 at 7:59 AM Andrea Iob > wrote: Hi, I'm trying to solve a linear system using HYPRE/BoomerAMG as preconditioner. The system comes from a two-dimensional adjoint Navier-Stokes problem. Do you expect the system you are solving to be elliptic? BoomerAMG is designed for elliptic systems, and can easily fail if applied to a more general system, say one with zeros on the diagonal. Thanks, Matt The mesh is structured (6400 cells) and there are 24 DoFs for each cell (the matrix has a total of 153600 rows). The condition number of the matrix should be in the order of 1e9. Using ILU preconditioner and FGMRES, the system is correctly solved (it takes 800 iterations to reach a residual of 1e-12). However, replacing ILU with HYPRE/BoomerAMG, the solver stops right after the first iteration (DIVERGED_NANORINF). I've tried different BoomerAMG options and different solvers (including Richardson to use BoomerAMG without a Krylov method), but I always get the same DIVERGED_NANORINF error. Using the same HYPRE/BoomerAMG + FGMERS setup on another problem (Laplacian on a highly stretched grid), the solver reaches convergence very quickly without any problems. At the bottom of this mail, you'll find the log of a run that stops with DIVERGED_NANORINF. Do you spot something in my setup that may explain why the solver is not converging? Thanks. Best regards, Andrea ----------------------------------------- Assembly started... Reading matrix file... - Number of rows (N) = 153600 - Number of non-zero entries (NNZ) = 18247680 Assembly completed. Time elapsed 83.3231s Set up started... Set up completed. Time elapsed 6.62441s Solution started... 0 KSP unpreconditioned resid norm 7.736650641501e-01 true resid norm 7.736650641501e-01 ||r(i)||/||b|| 1.000000000000e+00 0 KSP Residual norm 7.736650641501e-01 % max 1.000000000000e+00 min 1.000000000000e+00 max/min 1.000000000000e+00 [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [0]PETSC ERROR: [0]PETSC ERROR: KSPSolve has not converged due to Nan or Inf inner product [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [0]PETSC ERROR: Petsc Release Version 3.10.3, unknown [0]PETSC ERROR: bitpit_system_solver on a named xxx by xxx Tue Jun 30 09:42:09 2020 [0]PETSC ERROR: Configure options PETSC_ARCH=arch-linux2-c-opt-gcc7-hypre --with-blaslapack-dir=/opt/lapack/3.8.0-gcc.7.4.0/lib64 --with-debugging=0 COPTFLAGS=-O3 CXXOPTFLAGS=-O3 FOPTFLAGS=-O3 --with-valgrind=1 --with-valgrind-dir=/opt/valgrind/3.14.0/ --prefix=/opt/petsc/3.10.3_gcc-7.4.0 --with-mpi=1 --download-hypre [0]PETSC ERROR: #1 KSPGMRESClassicalGramSchmidtOrthogonalization() line 67 in /root/InstallSources/petsc/src/ksp/ksp/impls/gmres/borthog2.c [0]PETSC ERROR: #2 KSPFGMRESCycle() line 175 in /root/InstallSources/petsc/src/ksp/ksp/impls/gmres/fgmres/fgmres.c [0]PETSC ERROR: #3 KSPSolve_FGMRES() line 291 in /root/InstallSources/petsc/src/ksp/ksp/impls/gmres/fgmres/fgmres.c [0]PETSC ERROR: #4 KSPSolve() line 780 in /root/InstallSources/petsc/src/ksp/ksp/interface/itfunc.c KSP Object: 1 MPI processes type: fgmres restart=45, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement happy breakdown tolerance 1e-30 maximum iterations=10000, nonzero initial guess tolerances: relative=1e-12, absolute=1e-50, divergence=10000. right preconditioning using UNPRECONDITIONED norm type for convergence test PC Object: 1 MPI processes type: hypre HYPRE BoomerAMG preconditioning Cycle type V Maximum number of levels 25 Maximum number of iterations PER hypre call 1 Convergence tolerance PER hypre call 0. Threshold for strong coupling 0.7 Interpolation truncation factor 0.3 Interpolation: max elements per row 2 Number of levels of aggressive coarsening 4 Number of paths for aggressive coarsening 5 Maximum row sums 0.9 Sweeps down 1 Sweeps up 1 Sweeps on coarse 1 Relax down sequential-Gauss-Seidel Relax up sequential-Gauss-Seidel Relax on coarse Gaussian-elimination Relax weight (all) 1. Outer relax weight (all) 1. Using CF-relaxation Smooth type Euclid Smooth num levels 25 Euclid ILU(k) levels 0 Euclid ILU(k) drop tolerance 0. Euclid ILU use Block-Jacobi? 1 Measure type local Coarsen type HMIS Interpolation type ext+i linear system matrix = precond matrix: Mat Object: Mat_0x1e88040_0 1 MPI processes type: seqaij rows=153600, cols=153600, bs=24 total: nonzeros=18247680, allocated nonzeros=18247680 total number of mallocs used during MatSetValues calls =0 using I-node routines: found 32000 nodes, limit used is 5 Solution completed. Time elapsed 6.93336s Solution information... Error (L2) : 32.8359 Error (Linf) : 1.93278 ************************************************************************************************************************ *** WIDEN YOUR WINDOW TO 120 CHARACTERS. Use 'enscript -r -fCourier9' to print this document *** ************************************************************************************************************************ ---------------------------------------------- PETSc Performance Summary: ---------------------------------------------- bitpit_system_solver on a named xxx with 1 processor, by xxx Tue Jun 30 09:43:46 2020 Using Petsc Release Version 3.10.3, unknown Max Max/Min Avg Total Time (sec): 9.718e+01 1.000 9.718e+01 Objects: 5.300e+01 1.000 5.300e+01 Flop: 1.107e+08 1.000 1.107e+08 1.107e+08 Flop/sec: 1.139e+06 1.000 1.139e+06 1.139e+06 MPI Messages: 0.000e+00 0.000 0.000e+00 0.000e+00 MPI Message Lengths: 0.000e+00 0.000 0.000e+00 0.000e+00 MPI Reductions: 0.000e+00 0.000 Flop counting convention: 1 flop = 1 real number operation of type (multiply/divide/add/subtract) e.g., VecAXPY() for real vectors of length N --> 2N flop and VecAXPY() for complex vectors of length N --> 8N flop Summary of Stages: ----- Time ------ ----- Flop ------ --- Messages --- -- Message Lengths -- -- Reductions -- Avg %Total Avg %Total Count %Total Avg %Total Count %Total 0: Main Stage: 9.7178e+01 100.0% 1.1071e+08 100.0% 0.000e+00 0.0% 0.000e+00 0.0% 0.000e+00 0.0% ------------------------------------------------------------------------------------------------------------------------ See the 'Profiling' chapter of the users' manual for details on interpreting output. Phase summary info: Count: number of times phase was executed Time and Flop: Max - maximum over all processors Ratio - ratio of maximum to minimum over all processors Mess: number of messages sent AvgLen: average message length (bytes) Reduct: number of global reductions Global: entire computation Stage: stages of a computation. Set stages with PetscLogStagePush() and PetscLogStagePop(). %T - percent time in this phase %F - percent flop in this phase %M - percent messages in this phase %L - percent message lengths in this phase %R - percent reductions in this phase Total Mflop/s: 10e-6 * (sum of flop over all processors)/(max time over all processors) ------------------------------------------------------------------------------------------------------------------------ Event Count Time (sec) Flop --- Global --- --- Stage ---- Total Max Ratio Max Ratio Max Ratio Mess AvgLen Reduct %T %F %M %L %R %T %F %M %L %R Mflop/s ------------------------------------------------------------------------------------------------------------------------ --- Event Stage 0: Main Stage BuildTwoSidedF 2 1.0 1.8471e-04 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatMult 3 1.0 6.8688e-02 1.0 1.09e+08 1.0 0.0e+00 0.0e+00 0.0e+00 0 98 0 0 0 0 98 0 0 0 1587 MatConvert 2 1.0 3.8534e-01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatAssemblyBegin 3 1.0 2.8112e-04 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatAssemblyEnd 3 1.0 7.8563e-02 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatGetRowIJ 2 1.0 3.6100e-06 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatView 2 1.0 5.4618e+01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 56 0 0 0 0 56 0 0 0 0 0 VecView 2 1.0 7.6981e-01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 1 0 0 0 0 1 0 0 0 0 0 VecMDot 1 1.0 1.7635e-04 1.0 3.07e+05 1.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 1742 VecNorm 3 1.0 5.4832e-04 1.0 9.22e+05 1.0 0.0e+00 0.0e+00 0.0e+00 0 1 0 0 0 0 1 0 0 0 1681 VecScale 1 1.0 1.5216e-04 1.0 1.54e+05 1.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 1009 VecCopy 1 1.0 1.5406e-04 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecSet 21 1.0 7.1106e-03 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecAYPX 1 1.0 1.9940e-04 1.0 1.54e+05 1.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 770 VecWAXPY 1 1.0 7.8802e-04 1.0 1.54e+05 1.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 195 KSPSetUp 2 1.0 6.8995e-03 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 PCSetUp 2 1.0 1.3190e+01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 14 0 0 0 0 14 0 0 0 0 0 PCApply 1 1.0 2.2728e-01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 ------------------------------------------------------------------------------------------------------------------------ Memory usage is given in bytes: Object Type Creations Destructions Memory Descendants' Mem. Reports information only for process 0. --- Event Stage 0: Main Stage Matrix 3 1 2872 0. Vector 32 20 17234800 0. Index Set 4 4 3200 0. IS L to G Mapping 2 0 0 0. Vec Scatter 2 0 0 0. Viewer 6 4 3392 0. Krylov Solver 2 1 61676 0. Preconditioner 2 1 1432 0. ======================================================================================================================== Average time to get PetscTime(): 3.36e-08 #PETSc Option Table entries: -ksp_converged_reason -ksp_error_if_not_converged -ksp_monitor_singular_value -ksp_monitor_true_residual -log_view -pc_hypre_boomeramg_agg_nl 4 -pc_hypre_boomeramg_agg_num_paths 5 -pc_hypre_boomeramg_coarsen_type HMIS -pc_hypre_boomeramg_eu_bj true -pc_hypre_boomeramg_interp_type ext+i -pc_hypre_boomeramg_P_max 2 -pc_hypre_boomeramg_relax_type_all sequential-Gauss-Seidel -pc_hypre_boomeramg_smooth_type Euclid -pc_hypre_boomeramg_strong_threshold 0.7 -pc_hypre_boomeramg_truncfactor 0.3 #End of PETSc Option Table entries Compiled without FORTRAN kernels Compiled with full precision matrices (default) sizeof(short) 2 sizeof(int) 4 sizeof(long) 8 sizeof(void*) 8 sizeof(PetscScalar) 8 sizeof(PetscInt) 4 Configure options: PETSC_ARCH=arch-linux2-c-opt-gcc7-hypre --with-blaslapack-dir=/opt/lapack/3.8.0-gcc.7.4.0/lib64 --with-debugging=0 COPTFLAGS=-O3 CXXOPTFLAGS=-O3 FOPTFLAGS=-O3 --with-valgrind=1 --with-valgrind-dir=/opt/valgrind/3.14.0/ --prefix=/opt/petsc/3.10.3_gcc-7.4.0 --with-mpi=1 --download-hypre ----------------------------------------- Libraries compiled on 2020-06-26 12:35:46 on xxx Machine characteristics: Linux-3.10.0-1127.8.2.el7.x86_64-x86_64-with-centos-7.6.1810-Core Using PETSc directory: /opt/petsc/3.10.3_gcc-7.4.0 Using PETSc arch: ----------------------------------------- Using C compiler: mpicc -fPIC -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas -fstack-protector -fvisibility=hidden -O3 Using Fortran compiler: mpif90 -fPIC -Wall -ffree-line-length-0 -Wno-unused-dummy-argument -O3 ----------------------------------------- Using include paths: -I/opt/petsc/3.10.3_gcc-7.4.0/include -I/opt/valgrind/3.14.0/include ----------------------------------------- Using C linker: mpicc Using Fortran linker: mpif90 Using libraries: -Wl,-rpath,/opt/petsc/3.10.3_gcc-7.4.0/lib -L/opt/petsc/3.10.3_gcc-7.4.0/lib -lpetsc -Wl,-rpath,/opt/petsc/3.10.3_gcc-7.4.0/lib -L/opt/petsc/3.10.3_gcc-7.4.0/lib -Wl,-rpath,/opt/lapack/3.8.0-gcc.7.4.0/lib64 -L/opt/lapack/3.8.0-gcc.7.4.0/lib64 -Wl,-rpath,/opt/openmpi4.0.0-gcc7.4.0_slurm/lib -L/opt/openmpi4.0.0-gcc7.4.0_slurm/lib -Wl,-rpath,/opt/gnu/7.4/lib/gcc/x86_64-pc-linux-gnu/7.4.0 -L/opt/gnu/7.4/lib/gcc/x86_64-pc-linux-gnu/7.4.0 -Wl,-rpath,/opt/gnu/7.4/lib/gcc -L/opt/gnu/7.4/lib/gcc -Wl,-rpath,/opt/gnu/7.4/lib64 -L/opt/gnu/7.4/lib64 -Wl,-rpath,/opt/gnu/7.4/lib -L/opt/gnu/7.4/lib -lHYPRE -llapack -lblas -lm -lX11 -lstdc++ -ldl -lmpi_usempif08 -lmpi_usempi_ignore_tkr -lmpi_mpifh -lmpi -lgfortran -lm -lgfortran -lm -lgcc_s -lquadmath -lpthread -lstdc++ -ldl ----------------------------------------- -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at petsc.dev Thu Jul 2 09:30:54 2020 From: bsmith at petsc.dev (Barry Smith) Date: Thu, 2 Jul 2020 09:30:54 -0500 Subject: [petsc-users] DIVERGED_NANORINF when using HYPRE/BoomerAMG In-Reply-To: References: Message-ID: <23E7C803-04C4-4D3A-B22A-9F20F5003590@petsc.dev> > On Jul 2, 2020, at 7:48 AM, Andrea Iob wrote: > > I did some tests: depending on the smoother (set with the option "-pc_hypre_boomeramg_smooth_type"), the arithmetic exception is throw by different functions. Backtraces are at the bottom. Sorry for > the lack of line numbers, but, at the moment, I don't have a version of the PETSc library compiled with debug symbols. > > From my side, I'm quite happy with the performances of PCGAMG. The time it takes to solve the system went down from 60s (ILU) to 29s (PCGAMG with ILU+FGMRES as smoothers and "-pc_gamg_agg_nsmooths 0"). Since the problem I have with BoomerAMG seems related to the HYPRE package, maybe I can open an issue on their github repository. However, if you need additional information (or more tests) I will be glad to provide them. > > Andrea > > > 1) Option not specified A Nan or Inf detected when FMGRES does a norm on a vector returned from hypre. If hypre produced the Nan or Inf then one would expect that floating point exception would be caught there. So one possibility is that hypre returns an absurdly large (yet finite) value and when it is squared in computing the norm it becomes infinity. This would mean that hypre is returning essentially garbage but not easy to find out why. > > #0 0x00007ffff507ce5c in ddot_ () from /opt/lapack/3.8.0-gcc.7.4.0/lib64/libblas.so.3 > #1 0x00007ffff65c278f in VecNorm_Seq () from /opt/petsc/3.10.3_gcc-7.4.0/lib/libpetsc.so.3.10 > #2 0x00007ffff65eaf7e in VecNorm () from /opt/petsc/3.10.3_gcc-7.4.0/lib/libpetsc.so.3.10 > #3 0x00007ffff6e3b241 in KSPFGMRESCycle () from /opt/petsc/3.10.3_gcc-7.4.0/lib/libpetsc.so.3.10 > #4 0x00007ffff6e3c39d in KSPSolve_FGMRES () from /opt/petsc/3.10.3_gcc-7.4.0/lib/libpetsc.so.3.10 > #5 0x00007ffff6e5832e in KSPSolve () from /opt/petsc/3.10.3_gcc-7.4.0/lib/libpetsc.so.3.10 > #6 0x00007ffff7a59888 in bitpit::SystemSolver::solve (this=0x75dd30) at /xxx/bitpit/code/src/LA/system_solvers_large.cpp:678 > > > 2) Euclid > > #0 0x00007ffff720c40a in Factor_dhSolveSeq () from /opt/petsc/3.10.3_gcc-7.4.0/lib/libpetsc.so.3.10 > #1 0x00007ffff720376a in Euclid_dhApply () from /opt/petsc/3.10.3_gcc-7.4.0/lib/libpetsc.so.3.10 > #2 0x00007ffff71b6867 in HYPRE_EuclidSolve () from /opt/petsc/3.10.3_gcc-7.4.0/lib/libpetsc.so.3.10 > #3 0x00007ffff7083953 in hypre_BoomerAMGCycle () from /opt/petsc/3.10.3_gcc-7.4.0/lib/libpetsc.so.3.10 > #4 0x00007ffff706f4ce in hypre_BoomerAMGSolve () from /opt/petsc/3.10.3_gcc-7.4.0/lib/libpetsc.so.3.10 > #5 0x00007ffff6d85cbb in PCApply_HYPRE () from /opt/petsc/3.10.3_gcc-7.4.0/lib/libpetsc.so.3.10 > #6 0x00007ffff6db324b in PCApply () from /opt/petsc/3.10.3_gcc-7.4.0/lib/libpetsc.so.3.10 > #7 0x00007ffff6e3b198 in KSPFGMRESCycle () from /opt/petsc/3.10.3_gcc-7.4.0/lib/libpetsc.so.3.10 > #8 0x00007ffff6e3c39d in KSPSolve_FGMRES () from /opt/petsc/3.10.3_gcc-7.4.0/lib/libpetsc.so.3.10 > #9 0x00007ffff6e5832e in KSPSolve () from /opt/petsc/3.10.3_gcc-7.4.0/lib/libpetsc.so.3.10 > #10 0x00007ffff7a59888 in bitpit::SystemSolver::solve (this=0x75dd70) at /xxx/bitpit/code/src/LA/system_solvers_large.cpp:678 > > > 3) Schwarz-smoothers > Zero pivot in a triangular solve, likely the matrix for the subdomain is singular which could be true if the operator is uniformly elliptic > #0 0x00007ffff50897f4 in dtrsm_ () from /opt/lapack/3.8.0-gcc.7.4.0/lib64/libblas.so.3 > #1 0x00007ffff550924b in dpotrs_ () from /opt/lapack/3.8.0-gcc.7.4.0/lib64/liblapack.so.3 > #2 0x00007ffff70ed4bf in hypre_MPSchwarzSolve () from /opt/petsc/3.10.3_gcc-7.4.0/lib/libpetsc.so.3.10 > #3 0x00007ffff70c9478 in hypre_SchwarzSolve () from /opt/petsc/3.10.3_gcc-7.4.0/lib/libpetsc.so.3.10 > #4 0x00007ffff70836d1 in hypre_BoomerAMGCycle () from /opt/petsc/3.10.3_gcc-7.4.0/lib/libpetsc.so.3.10 > #5 0x00007ffff706f4ce in hypre_BoomerAMGSolve () from /opt/petsc/3.10.3_gcc-7.4.0/lib/libpetsc.so.3.10 > #6 0x00007ffff6d85cbb in PCApply_HYPRE () from /opt/petsc/3.10.3_gcc-7.4.0/lib/libpetsc.so.3.10 > #7 0x00007ffff6db324b in PCApply () from /opt/petsc/3.10.3_gcc-7.4.0/lib/libpetsc.so.3.10 > #8 0x00007ffff6e3b198 in KSPFGMRESCycle () from /opt/petsc/3.10.3_gcc-7.4.0/lib/libpetsc.so.3.10 > #9 0x00007ffff6e3c39d in KSPSolve_FGMRES () from /opt/petsc/3.10.3_gcc-7.4.0/lib/libpetsc.so.3.10 > #10 0x00007ffff6e5832e in KSPSolve () from /opt/petsc/3.10.3_gcc-7.4.0/lib/libpetsc.so.3.10 > #11 0x00007ffff7a59888 in bitpit::SystemSolver::solve (this=0x75dd30) at /xxx/bitpit/code/src/LA/system_solvers_large.cpp:678 > > > 4) Pilut > > No errors but the solver doesn't converge: > > 0 KSP unpreconditioned resid norm 7.736650641501e-01 true resid norm 7.736650641501e-01 ||r(i)||/||b|| 1.000000000000e+00 > 0 KSP Residual norm 7.736650641501e-01 % max 1.000000000000e+00 min 1.000000000000e+00 max/min 1.000000000000e+00 > 1 KSP unpreconditioned resid norm 7.736650641501e-01 true resid norm 7.736650641501e-01 ||r(i)||/||b|| 1.000000000000e+00 > 1 KSP Residual norm 7.736650641501e-01 % max 8.836643387885e+91 min 8.836643387885e+91 max/min 1.000000000000e+00 > 2 KSP unpreconditioned resid norm 7.736650641501e-01 true resid norm 7.736650641501e-01 ||r(i)||/||b|| 1.000000000000e+00 > 2 KSP Residual norm 7.736650641501e-01 % max 2.525644801510e+141 min 2.809949455965e+85 max/min 8.988221464797e+55 > 3 KSP unpreconditioned resid norm 7.736650641501e-01 true resid norm 7.736650641501e-01 ||r(i)||/||b|| 1.000000000000e+00 > 3 KSP Residual norm 7.736650641501e-01 % max 5.240099813864e+141 min 4.394901421200e+84 max/min 1.192313390372e+57 > 4 KSP unpreconditioned resid norm 7.736650641501e-01 true resid norm 7.736650641501e-01 ||r(i)||/||b|| 1.000000000000e+00 > 4 KSP Residual norm 7.736650641501e-01 % max 5.249800875338e+141 min 1.922154232776e+84 max/min 2.731206885389e+57 > 5 KSP unpreconditioned resid norm 7.736650641501e-01 true resid norm 7.736650641501e-01 ||r(i)||/||b|| 1.000000000000e+00 > 5 KSP Residual norm 7.736650641501e-01 % max 5.250261290126e+141 min 1.874815205276e+84 max/min 2.800415355792e+57 > 6 KSP unpreconditioned resid norm 7.736650641501e-01 true resid norm 7.736650641501e-01 ||r(i)||/||b|| 1.000000000000e+00 > > From: Barry Smith > > Sent: Thursday, July 2, 2020 1:36 AM > To: Andrea Iob > > Cc: Mark Adams >; Matthew Knepley >; petsc-users at mcs.anl.gov > > Subject: Re: [petsc-users] DIVERGED_NANORINF when using HYPRE/BoomerAMG > > > You can run the hyper case with -fp_trap in a debugger and it may stop exactly in hypre where the inf/nan appears. Or if you know how to catch floating point exceptions in your debugger run in the debugger with that option to find where in hypre the inf/nan appears. > > Barry > > >> On Jul 1, 2020, at 8:22 AM, Andrea Iob > wrote: >> >> Thanks for the answers. >> >> The matrix contains both the advection and the diffusion terms of the Navier-Stokes equations. The case is subsonic (M=0.5), Reynolds number is in the order of 5000 so I expect the hyperbolic features to be the dominant ones. I understand this is not the type of problems AMG is designed for, but I was hoping to still be able to solve the system. This is the first time I use BoomerAMG and I'm not familar with all the options, however all the tests I've done gave me INFs/NANs. >> >> You suggestion to use GAMG with custom smoothers seems to do the trick: after setting the GAMG smoother of each level to FGMRES/ILU, the solver converges in 93 iterations (overall it seems faster than ILU). Great! I've set the smoothers like this: >> >> for (int i = 0; i < nLevels; ++i) { >> KSP levelSmoother; >> PCMGGetSmoother(preconditioner, i, &levelSmoother); >> >> PC levelSmootherPreconditioner; >> KSPGetPC(levelSmoother, &levelSmootherPreconditioner); >> PCSetType(levelSmootherPreconditioner, PCILU); >> >> KSPSetType(levelSmoother, KSPFGMRES); >> } >> >> Thank you very much! >> Best regards, >> Andrea >> >> From: Mark Adams > >> Sent: Wednesday, July 1, 2020 2:15 PM >> To: Matthew Knepley > >> Cc: Andrea Iob >; petsc-users at mcs.anl.gov > >> Subject: Re: [petsc-users] DIVERGED_NANORINF when using HYPRE/BoomerAMG >> >> As Matt said AMG does not work for everything out-of-the-box. >> >> Hypre is pretty robust and I am surprised that every option gives INFs/NANs. But if it solves your stretched Lapacian then it's probably hooked up correctly. >> >> You might try a non-adjoint Navier-Stokes problem, if you can. >> >> You could try 'gamg' with the fgmres/ilu solver that works, as your smoother (use -ksp_view to check that you get everything set correctly). Use fgmres as the outer Krylov method and use -pc_gamg_agg_nsmooths 0. >> >> Mark >> >> >> On Tue, Jun 30, 2020 at 9:16 AM Matthew Knepley > wrote: >> On Tue, Jun 30, 2020 at 7:59 AM Andrea Iob > wrote: >> Hi, >> >> I'm trying to solve a linear system using HYPRE/BoomerAMG as preconditioner. The system comes from a two-dimensional adjoint Navier-Stokes problem. >> >> Do you expect the system you are solving to be elliptic? BoomerAMG is designed for elliptic systems, and can easily fail if applied >> to a more general system, say one with zeros on the diagonal. >> >> Thanks, >> >> Matt >> >> The mesh is structured (6400 cells) and there are 24 DoFs for each cell (the matrix has a total of 153600 rows). The condition number of the matrix should be in the order of 1e9. Using ILU preconditioner and FGMRES, the system is correctly solved (it takes 800 iterations to reach a residual of 1e-12). However, replacing ILU with HYPRE/BoomerAMG, the solver stops right after the first iteration (DIVERGED_NANORINF). I've tried different BoomerAMG options and different solvers (including Richardson to use BoomerAMG without a Krylov method), but I always get the same DIVERGED_NANORINF error. >> >> Using the same HYPRE/BoomerAMG + FGMERS setup on another problem (Laplacian on a highly stretched grid), the solver reaches convergence very quickly without any problems. >> >> At the bottom of this mail, you'll find the log of a run that stops with DIVERGED_NANORINF. Do you spot something in my setup that may explain why the solver is not converging? >> >> Thanks. Best regards, >> Andrea >> >> ----------------------------------------- >> Assembly started... >> Reading matrix file... >> - Number of rows (N) = 153600 >> - Number of non-zero entries (NNZ) = 18247680 >> Assembly completed. >> Time elapsed 83.3231s >> Set up started... >> Set up completed. >> Time elapsed 6.62441s >> Solution started... >> 0 KSP unpreconditioned resid norm 7.736650641501e-01 true resid norm 7.736650641501e-01 ||r(i)||/||b|| 1.000000000000e+00 >> 0 KSP Residual norm 7.736650641501e-01 % max 1.000000000000e+00 min 1.000000000000e+00 max/min 1.000000000000e+00 >> [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- >> [0]PETSC ERROR: >> [0]PETSC ERROR: KSPSolve has not converged due to Nan or Inf inner product >> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. >> [0]PETSC ERROR: Petsc Release Version 3.10.3, unknown >> [0]PETSC ERROR: bitpit_system_solver on a named xxx by xxx Tue Jun 30 09:42:09 2020 >> [0]PETSC ERROR: Configure options PETSC_ARCH=arch-linux2-c-opt-gcc7-hypre --with-blaslapack-dir=/opt/lapack/3.8.0-gcc.7.4.0/lib64 --with-debugging=0 COPTFLAGS=-O3 CXXOPTFLAGS=-O3 FOPTFLAGS=-O3 --with-valgrind=1 --with-valgrind-dir=/opt/valgrind/3.14.0/ --prefix=/opt/petsc/3.10.3_gcc-7.4.0 --with-mpi=1 --download-hypre >> [0]PETSC ERROR: #1 KSPGMRESClassicalGramSchmidtOrthogonalization() line 67 in /root/InstallSources/petsc/src/ksp/ksp/impls/gmres/borthog2.c >> [0]PETSC ERROR: #2 KSPFGMRESCycle() line 175 in /root/InstallSources/petsc/src/ksp/ksp/impls/gmres/fgmres/fgmres.c >> [0]PETSC ERROR: #3 KSPSolve_FGMRES() line 291 in /root/InstallSources/petsc/src/ksp/ksp/impls/gmres/fgmres/fgmres.c >> [0]PETSC ERROR: #4 KSPSolve() line 780 in /root/InstallSources/petsc/src/ksp/ksp/interface/itfunc.c >> KSP Object: 1 MPI processes >> type: fgmres >> restart=45, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement >> happy breakdown tolerance 1e-30 >> maximum iterations=10000, nonzero initial guess >> tolerances: relative=1e-12, absolute=1e-50, divergence=10000. >> right preconditioning >> using UNPRECONDITIONED norm type for convergence test >> PC Object: 1 MPI processes >> type: hypre >> HYPRE BoomerAMG preconditioning >> Cycle type V >> Maximum number of levels 25 >> Maximum number of iterations PER hypre call 1 >> Convergence tolerance PER hypre call 0. >> Threshold for strong coupling 0.7 >> Interpolation truncation factor 0.3 >> Interpolation: max elements per row 2 >> Number of levels of aggressive coarsening 4 >> Number of paths for aggressive coarsening 5 >> Maximum row sums 0.9 >> Sweeps down 1 >> Sweeps up 1 >> Sweeps on coarse 1 >> Relax down sequential-Gauss-Seidel >> Relax up sequential-Gauss-Seidel >> Relax on coarse Gaussian-elimination >> Relax weight (all) 1. >> Outer relax weight (all) 1. >> Using CF-relaxation >> Smooth type Euclid >> Smooth num levels 25 >> Euclid ILU(k) levels 0 >> Euclid ILU(k) drop tolerance 0. >> Euclid ILU use Block-Jacobi? 1 >> Measure type local >> Coarsen type HMIS >> Interpolation type ext+i >> linear system matrix = precond matrix: >> Mat Object: Mat_0x1e88040_0 1 MPI processes >> type: seqaij >> rows=153600, cols=153600, bs=24 >> total: nonzeros=18247680, allocated nonzeros=18247680 >> total number of mallocs used during MatSetValues calls =0 >> using I-node routines: found 32000 nodes, limit used is 5 >> Solution completed. >> Time elapsed 6.93336s >> Solution information... >> Error (L2) : 32.8359 >> Error (Linf) : 1.93278 >> ************************************************************************************************************************ >> *** WIDEN YOUR WINDOW TO 120 CHARACTERS. Use 'enscript -r -fCourier9' to print this document *** >> ************************************************************************************************************************ >> >> ---------------------------------------------- PETSc Performance Summary: ---------------------------------------------- >> >> bitpit_system_solver on a named xxx with 1 processor, by xxx Tue Jun 30 09:43:46 2020 >> Using Petsc Release Version 3.10.3, unknown >> >> Max Max/Min Avg Total >> Time (sec): 9.718e+01 1.000 9.718e+01 >> Objects: 5.300e+01 1.000 5.300e+01 >> Flop: 1.107e+08 1.000 1.107e+08 1.107e+08 >> Flop/sec: 1.139e+06 1.000 1.139e+06 1.139e+06 >> MPI Messages: 0.000e+00 0.000 0.000e+00 0.000e+00 >> MPI Message Lengths: 0.000e+00 0.000 0.000e+00 0.000e+00 >> MPI Reductions: 0.000e+00 0.000 >> >> Flop counting convention: 1 flop = 1 real number operation of type (multiply/divide/add/subtract) >> e.g., VecAXPY() for real vectors of length N --> 2N flop >> and VecAXPY() for complex vectors of length N --> 8N flop >> >> Summary of Stages: ----- Time ------ ----- Flop ------ --- Messages --- -- Message Lengths -- -- Reductions -- >> Avg %Total Avg %Total Count %Total Avg %Total Count %Total >> 0: Main Stage: 9.7178e+01 100.0% 1.1071e+08 100.0% 0.000e+00 0.0% 0.000e+00 0.0% 0.000e+00 0.0% >> >> ------------------------------------------------------------------------------------------------------------------------ >> See the 'Profiling' chapter of the users' manual for details on interpreting output. >> Phase summary info: >> Count: number of times phase was executed >> Time and Flop: Max - maximum over all processors >> Ratio - ratio of maximum to minimum over all processors >> Mess: number of messages sent >> AvgLen: average message length (bytes) >> Reduct: number of global reductions >> Global: entire computation >> Stage: stages of a computation. Set stages with PetscLogStagePush() and PetscLogStagePop(). >> %T - percent time in this phase %F - percent flop in this phase >> %M - percent messages in this phase %L - percent message lengths in this phase >> %R - percent reductions in this phase >> Total Mflop/s: 10e-6 * (sum of flop over all processors)/(max time over all processors) >> ------------------------------------------------------------------------------------------------------------------------ >> Event Count Time (sec) Flop --- Global --- --- Stage ---- Total >> Max Ratio Max Ratio Max Ratio Mess AvgLen Reduct %T %F %M %L %R %T %F %M %L %R Mflop/s >> ------------------------------------------------------------------------------------------------------------------------ >> >> --- Event Stage 0: Main Stage >> >> BuildTwoSidedF 2 1.0 1.8471e-04 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 >> MatMult 3 1.0 6.8688e-02 1.0 1.09e+08 1.0 0.0e+00 0.0e+00 0.0e+00 0 98 0 0 0 0 98 0 0 0 1587 >> MatConvert 2 1.0 3.8534e-01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 >> MatAssemblyBegin 3 1.0 2.8112e-04 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 >> MatAssemblyEnd 3 1.0 7.8563e-02 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 >> MatGetRowIJ 2 1.0 3.6100e-06 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 >> MatView 2 1.0 5.4618e+01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 56 0 0 0 0 56 0 0 0 0 0 >> VecView 2 1.0 7.6981e-01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 1 0 0 0 0 1 0 0 0 0 0 >> VecMDot 1 1.0 1.7635e-04 1.0 3.07e+05 1.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 1742 >> VecNorm 3 1.0 5.4832e-04 1.0 9.22e+05 1.0 0.0e+00 0.0e+00 0.0e+00 0 1 0 0 0 0 1 0 0 0 1681 >> VecScale 1 1.0 1.5216e-04 1.0 1.54e+05 1.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 1009 >> VecCopy 1 1.0 1.5406e-04 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 >> VecSet 21 1.0 7.1106e-03 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 >> VecAYPX 1 1.0 1.9940e-04 1.0 1.54e+05 1.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 770 >> VecWAXPY 1 1.0 7.8802e-04 1.0 1.54e+05 1.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 195 >> KSPSetUp 2 1.0 6.8995e-03 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 >> PCSetUp 2 1.0 1.3190e+01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 14 0 0 0 0 14 0 0 0 0 0 >> PCApply 1 1.0 2.2728e-01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 >> ------------------------------------------------------------------------------------------------------------------------ >> >> Memory usage is given in bytes: >> >> Object Type Creations Destructions Memory Descendants' Mem. >> Reports information only for process 0. >> >> --- Event Stage 0: Main Stage >> >> Matrix 3 1 2872 0. >> Vector 32 20 17234800 0. >> Index Set 4 4 3200 0. >> IS L to G Mapping 2 0 0 0. >> Vec Scatter 2 0 0 0. >> Viewer 6 4 3392 0. >> Krylov Solver 2 1 61676 0. >> Preconditioner 2 1 1432 0. >> ======================================================================================================================== >> Average time to get PetscTime(): 3.36e-08 >> #PETSc Option Table entries: >> -ksp_converged_reason >> -ksp_error_if_not_converged >> -ksp_monitor_singular_value >> -ksp_monitor_true_residual >> -log_view >> -pc_hypre_boomeramg_agg_nl 4 >> -pc_hypre_boomeramg_agg_num_paths 5 >> -pc_hypre_boomeramg_coarsen_type HMIS >> -pc_hypre_boomeramg_eu_bj true >> -pc_hypre_boomeramg_interp_type ext+i >> -pc_hypre_boomeramg_P_max 2 >> -pc_hypre_boomeramg_relax_type_all sequential-Gauss-Seidel >> -pc_hypre_boomeramg_smooth_type Euclid >> -pc_hypre_boomeramg_strong_threshold 0.7 >> -pc_hypre_boomeramg_truncfactor 0.3 >> #End of PETSc Option Table entries >> Compiled without FORTRAN kernels >> Compiled with full precision matrices (default) >> sizeof(short) 2 sizeof(int) 4 sizeof(long) 8 sizeof(void*) 8 sizeof(PetscScalar) 8 sizeof(PetscInt) 4 >> Configure options: PETSC_ARCH=arch-linux2-c-opt-gcc7-hypre --with-blaslapack-dir=/opt/lapack/3.8.0-gcc.7.4.0/lib64 --with-debugging=0 COPTFLAGS=-O3 CXXOPTFLAGS=-O3 FOPTFLAGS=-O3 --with-valgrind=1 --with-valgrind-dir=/opt/valgrind/3.14.0/ --prefix=/opt/petsc/3.10.3_gcc-7.4.0 --with-mpi=1 --download-hypre >> ----------------------------------------- >> Libraries compiled on 2020-06-26 12:35:46 on xxx >> Machine characteristics: Linux-3.10.0-1127.8.2.el7.x86_64-x86_64-with-centos-7.6.1810-Core >> Using PETSc directory: /opt/petsc/3.10.3_gcc-7.4.0 >> Using PETSc arch: >> ----------------------------------------- >> >> Using C compiler: mpicc -fPIC -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas -fstack-protector -fvisibility=hidden -O3 >> Using Fortran compiler: mpif90 -fPIC -Wall -ffree-line-length-0 -Wno-unused-dummy-argument -O3 >> ----------------------------------------- >> >> Using include paths: -I/opt/petsc/3.10.3_gcc-7.4.0/include -I/opt/valgrind/3.14.0/include >> ----------------------------------------- >> >> Using C linker: mpicc >> Using Fortran linker: mpif90 >> Using libraries: -Wl,-rpath,/opt/petsc/3.10.3_gcc-7.4.0/lib -L/opt/petsc/3.10.3_gcc-7.4.0/lib -lpetsc -Wl,-rpath,/opt/petsc/3.10.3_gcc-7.4.0/lib -L/opt/petsc/3.10.3_gcc-7.4.0/lib -Wl,-rpath,/opt/lapack/3.8.0-gcc.7.4.0/lib64 -L/opt/lapack/3.8.0-gcc.7.4.0/lib64 -Wl,-rpath,/opt/openmpi4.0.0-gcc7.4.0_slurm/lib -L/opt/openmpi4.0.0-gcc7.4.0_slurm/lib -Wl,-rpath,/opt/gnu/7.4/lib/gcc/x86_64-pc-linux-gnu/7.4.0 -L/opt/gnu/7.4/lib/gcc/x86_64-pc-linux-gnu/7.4.0 -Wl,-rpath,/opt/gnu/7.4/lib/gcc -L/opt/gnu/7.4/lib/gcc -Wl,-rpath,/opt/gnu/7.4/lib64 -L/opt/gnu/7.4/lib64 -Wl,-rpath,/opt/gnu/7.4/lib -L/opt/gnu/7.4/lib -lHYPRE -llapack -lblas -lm -lX11 -lstdc++ -ldl -lmpi_usempif08 -lmpi_usempi_ignore_tkr -lmpi_mpifh -lmpi -lgfortran -lm -lgfortran -lm -lgcc_s -lquadmath -lpthread -lstdc++ -ldl >> ----------------------------------------- >> >> >> -- >> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. >> -- Norbert Wiener >> >> https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From karl.linkui at gmail.com Thu Jul 2 18:30:17 2020 From: karl.linkui at gmail.com (Karl Lin) Date: Thu, 2 Jul 2020 18:30:17 -0500 Subject: [petsc-users] matcreate and assembly issue In-Reply-To: References: Message-ID: Hi, Matt Thanks for the tip last time. We just encountered another issue with large data sets. This time the behavior is the opposite from last time. The data is 13.5TB, the total number of matrix columns is 2.4 billion. Our program crashed during matrix loading due to memory overflow in one node. As said before, we have a little memory check during loading the matrix to keep track of rss. The printout of rss in the log shows normal increase in many nodes, i.e., if we load in a portion of the matrix that is 1GB, after MatSetValues for that portion, rss will increase roughly about 1GB. On the node that has memory overflow, the rss increased by 2GB after only 1GB of matrix is loaded through MatSetValues. We are very puzzled by this. What could make the memory footprint twice as much as needed? Thanks in advance for any insight. Regards, Karl On Thu, Jun 11, 2020 at 12:00 PM Matthew Knepley wrote: > On Thu, Jun 11, 2020 at 12:52 PM Karl Lin wrote: > >> Hi, Matthew >> >> Thanks for the suggestion, just did another run and here are some >> detailed stack traces, maybe will provide some more insight: >> *** Process received signal *** >> Signal: Aborted (6) >> Signal code: (-6) >> /lib64/libpthread.so.0(+0xf5f0)[0x2b56c46dc5f0] >> [ 1] /lib64/libc.so.6(gsignal+0x37)[0x2b56c5486337] >> [ 2] /lib64/libc.so.6(abort+0x148)[0x2b56c5487a28] >> [ 3] /libpetsc.so.3.10(PetscTraceBackErrorHandler+0xc4)[0x2b56c1e6a2d4] >> [ 4] /libpetsc.so.3.10(PetscError+0x1b5)[0x2b56c1e69f65] >> [ 5] /libpetsc.so.3.10(PetscCommBuildTwoSidedFReq+0x19f0)[0x2b56c1e03cf0] >> [ 6] /libpetsc.so.3.10(+0x77db17)[0x2b56c2425b17] >> [ 7] /libpetsc.so.3.10(+0x77a164)[0x2b56c2422164] >> [ 8] /libpetsc.so.3.10(MatAssemblyBegin_MPIAIJ+0x36)[0x2b56c23912b6] >> [ 9] /libpetsc.so.3.10(MatAssemblyBegin+0xca)[0x2b56c1feccda] >> >> By reconfiguring, you mean recompiling petsc with that option, correct? >> > > Reconfiguring. > > Thanks, > > Matt > > >> Thank you. >> >> Karl >> >> On Thu, Jun 11, 2020 at 10:56 AM Matthew Knepley >> wrote: >> >>> On Thu, Jun 11, 2020 at 11:51 AM Karl Lin wrote: >>> >>>> Hi, there >>>> >>>> We have written a program using Petsc to solve large sparse matrix >>>> system. It has been working fine for a while. Recently we encountered a >>>> problem when the size of the sparse matrix is larger than 10TB. We used >>>> several hundred nodes and 2200 processes. The program always crashes during >>>> MatAssemblyBegin.Upon a closer look, there seems to be something unusual. >>>> We have a little memory check during loading the matrix to keep track of >>>> rss. The printout of rss in the log shows normal increase up to rank 2160, >>>> i.e., if we load in a portion of matrix that is 1GB, after MatSetValues for >>>> that portion, rss will increase roughly about that number. From rank 2161 >>>> onwards, the rss in every rank doesn't increase after matrix loaded. Then >>>> comes MatAssemblyBegin, the program crashed on rank 2160. >>>> >>>> Is there a upper limit on the number of processes Petsc can handle? >>>> or is there a upper limit in terms of the size of the matrix petsc can >>>> handle? Thank you very much for any info. >>>> >>> >>> It sounds like you overflowed int somewhere. We try and check for this, >>> but catching every place is hard. Try reconfiguring with >>> >>> --with-64-bit-indices >>> >>> Thanks, >>> >>> Matt >>> >>> >>>> Regards, >>>> >>>> Karl >>>> >>> >>> >>> -- >>> What most experimenters take for granted before they begin their >>> experiments is infinitely more interesting than any results to which their >>> experiments lead. >>> -- Norbert Wiener >>> >>> https://www.cse.buffalo.edu/~knepley/ >>> >>> >> > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Thu Jul 2 20:10:45 2020 From: knepley at gmail.com (Matthew Knepley) Date: Thu, 2 Jul 2020 21:10:45 -0400 Subject: [petsc-users] matcreate and assembly issue In-Reply-To: References: Message-ID: On Thu, Jul 2, 2020 at 7:30 PM Karl Lin wrote: > Hi, Matt > > Thanks for the tip last time. We just encountered another issue with large > data sets. This time the behavior is the opposite from last time. The data > is 13.5TB, the total number of matrix columns is 2.4 billion. Our program > crashed during matrix loading due to memory overflow in one node. As said > before, we have a little memory check during loading the matrix to keep > track of rss. The printout of rss in the log shows normal increase in many > nodes, i.e., if we load in a portion of the matrix that is 1GB, after > MatSetValues for that portion, rss will increase roughly about 1GB. On the > node that has memory overflow, the rss increased by 2GB after only 1GB of > matrix is loaded through MatSetValues. We are very puzzled by this. What > could make the memory footprint twice as much as needed? Thanks in advance > for any insight. > The only way I can imagine this happening is that you have not preallocated correctly, so that some values are causing additional mallocs. Thanks, Matt > Regards, > > Karl > > On Thu, Jun 11, 2020 at 12:00 PM Matthew Knepley > wrote: > >> On Thu, Jun 11, 2020 at 12:52 PM Karl Lin wrote: >> >>> Hi, Matthew >>> >>> Thanks for the suggestion, just did another run and here are some >>> detailed stack traces, maybe will provide some more insight: >>> *** Process received signal *** >>> Signal: Aborted (6) >>> Signal code: (-6) >>> /lib64/libpthread.so.0(+0xf5f0)[0x2b56c46dc5f0] >>> [ 1] /lib64/libc.so.6(gsignal+0x37)[0x2b56c5486337] >>> [ 2] /lib64/libc.so.6(abort+0x148)[0x2b56c5487a28] >>> [ 3] /libpetsc.so.3.10(PetscTraceBackErrorHandler+0xc4)[0x2b56c1e6a2d4] >>> [ 4] /libpetsc.so.3.10(PetscError+0x1b5)[0x2b56c1e69f65] >>> [ 5] >>> /libpetsc.so.3.10(PetscCommBuildTwoSidedFReq+0x19f0)[0x2b56c1e03cf0] >>> [ 6] /libpetsc.so.3.10(+0x77db17)[0x2b56c2425b17] >>> [ 7] /libpetsc.so.3.10(+0x77a164)[0x2b56c2422164] >>> [ 8] /libpetsc.so.3.10(MatAssemblyBegin_MPIAIJ+0x36)[0x2b56c23912b6] >>> [ 9] /libpetsc.so.3.10(MatAssemblyBegin+0xca)[0x2b56c1feccda] >>> >>> By reconfiguring, you mean recompiling petsc with that option, correct? >>> >> >> Reconfiguring. >> >> Thanks, >> >> Matt >> >> >>> Thank you. >>> >>> Karl >>> >>> On Thu, Jun 11, 2020 at 10:56 AM Matthew Knepley >>> wrote: >>> >>>> On Thu, Jun 11, 2020 at 11:51 AM Karl Lin >>>> wrote: >>>> >>>>> Hi, there >>>>> >>>>> We have written a program using Petsc to solve large sparse matrix >>>>> system. It has been working fine for a while. Recently we encountered a >>>>> problem when the size of the sparse matrix is larger than 10TB. We used >>>>> several hundred nodes and 2200 processes. The program always crashes during >>>>> MatAssemblyBegin.Upon a closer look, there seems to be something unusual. >>>>> We have a little memory check during loading the matrix to keep track of >>>>> rss. The printout of rss in the log shows normal increase up to rank 2160, >>>>> i.e., if we load in a portion of matrix that is 1GB, after MatSetValues for >>>>> that portion, rss will increase roughly about that number. From rank 2161 >>>>> onwards, the rss in every rank doesn't increase after matrix loaded. Then >>>>> comes MatAssemblyBegin, the program crashed on rank 2160. >>>>> >>>>> Is there a upper limit on the number of processes Petsc can handle? >>>>> or is there a upper limit in terms of the size of the matrix petsc can >>>>> handle? Thank you very much for any info. >>>>> >>>> >>>> It sounds like you overflowed int somewhere. We try and check for this, >>>> but catching every place is hard. Try reconfiguring with >>>> >>>> --with-64-bit-indices >>>> >>>> Thanks, >>>> >>>> Matt >>>> >>>> >>>>> Regards, >>>>> >>>>> Karl >>>>> >>>> >>>> >>>> -- >>>> What most experimenters take for granted before they begin their >>>> experiments is infinitely more interesting than any results to which their >>>> experiments lead. >>>> -- Norbert Wiener >>>> >>>> https://www.cse.buffalo.edu/~knepley/ >>>> >>>> >>> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> -- Norbert Wiener >> >> https://www.cse.buffalo.edu/~knepley/ >> >> > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at petsc.dev Thu Jul 2 20:14:46 2020 From: bsmith at petsc.dev (Barry Smith) Date: Thu, 2 Jul 2020 20:14:46 -0500 Subject: [petsc-users] matcreate and assembly issue In-Reply-To: References: Message-ID: <07E5F2DC-8670-431B-B00C-476E249998B7@petsc.dev> https://www.mcs.anl.gov/petsc/documentation/faq.html#efficient-assembly Perhaps we should provide more information at this FAQ to help track down such issues. > On Jul 2, 2020, at 8:10 PM, Matthew Knepley wrote: > > On Thu, Jul 2, 2020 at 7:30 PM Karl Lin > wrote: > Hi, Matt > > Thanks for the tip last time. We just encountered another issue with large data sets. This time the behavior is the opposite from last time. The data is 13.5TB, the total number of matrix columns is 2.4 billion. Our program crashed during matrix loading due to memory overflow in one node. As said before, we have a little memory check during loading the matrix to keep track of rss. The printout of rss in the log shows normal increase in many nodes, i.e., if we load in a portion of the matrix that is 1GB, after MatSetValues for that portion, rss will increase roughly about 1GB. On the node that has memory overflow, the rss increased by 2GB after only 1GB of matrix is loaded through MatSetValues. We are very puzzled by this. What could make the memory footprint twice as much as needed? Thanks in advance for any insight. > > The only way I can imagine this happening is that you have not preallocated correctly, so that some values are causing additional mallocs. > > Thanks, > > Matt > > Regards, > > Karl > > On Thu, Jun 11, 2020 at 12:00 PM Matthew Knepley > wrote: > On Thu, Jun 11, 2020 at 12:52 PM Karl Lin > wrote: > Hi, Matthew > > Thanks for the suggestion, just did another run and here are some detailed stack traces, maybe will provide some more insight: > *** Process received signal *** > Signal: Aborted (6) > Signal code: (-6) > /lib64/libpthread.so.0(+0xf5f0)[0x2b56c46dc5f0] > [ 1] /lib64/libc.so.6(gsignal+0x37)[0x2b56c5486337] > [ 2] /lib64/libc.so.6(abort+0x148)[0x2b56c5487a28] > [ 3] /libpetsc.so.3.10(PetscTraceBackErrorHandler+0xc4)[0x2b56c1e6a2d4] > [ 4] /libpetsc.so.3.10(PetscError+0x1b5)[0x2b56c1e69f65] > [ 5] /libpetsc.so.3.10(PetscCommBuildTwoSidedFReq+0x19f0)[0x2b56c1e03cf0] > [ 6] /libpetsc.so.3.10(+0x77db17)[0x2b56c2425b17] > [ 7] /libpetsc.so.3.10(+0x77a164)[0x2b56c2422164] > [ 8] /libpetsc.so.3.10(MatAssemblyBegin_MPIAIJ+0x36)[0x2b56c23912b6] > [ 9] /libpetsc.so.3.10(MatAssemblyBegin+0xca)[0x2b56c1feccda] > > By reconfiguring, you mean recompiling petsc with that option, correct? > > Reconfiguring. > > Thanks, > > Matt > > Thank you. > > Karl > > On Thu, Jun 11, 2020 at 10:56 AM Matthew Knepley > wrote: > On Thu, Jun 11, 2020 at 11:51 AM Karl Lin > wrote: > Hi, there > > We have written a program using Petsc to solve large sparse matrix system. It has been working fine for a while. Recently we encountered a problem when the size of the sparse matrix is larger than 10TB. We used several hundred nodes and 2200 processes. The program always crashes during MatAssemblyBegin.Upon a closer look, there seems to be something unusual. We have a little memory check during loading the matrix to keep track of rss. The printout of rss in the log shows normal increase up to rank 2160, i.e., if we load in a portion of matrix that is 1GB, after MatSetValues for that portion, rss will increase roughly about that number. From rank 2161 onwards, the rss in every rank doesn't increase after matrix loaded. Then comes MatAssemblyBegin, the program crashed on rank 2160. > > Is there a upper limit on the number of processes Petsc can handle? or is there a upper limit in terms of the size of the matrix petsc can handle? Thank you very much for any info. > > It sounds like you overflowed int somewhere. We try and check for this, but catching every place is hard. Try reconfiguring with > > --with-64-bit-indices > > Thanks, > > Matt > > Regards, > > Karl > > > -- > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ > > > -- > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ > > > -- > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From karl.linkui at gmail.com Thu Jul 2 22:03:26 2020 From: karl.linkui at gmail.com (Karl Lin) Date: Thu, 2 Jul 2020 22:03:26 -0500 Subject: [petsc-users] matcreate and assembly issue In-Reply-To: References: Message-ID: Hi, Matthew Thanks for the reply. However, I don't really get why additional malloc would double the memory footprint. If I know there is only 1GB matrix being loaded, there shouldn't be 2GB memory occupied even if Petsc needs to allocate more space. regards, Karl On Thu, Jul 2, 2020 at 8:10 PM Matthew Knepley wrote: > On Thu, Jul 2, 2020 at 7:30 PM Karl Lin wrote: > >> Hi, Matt >> >> Thanks for the tip last time. We just encountered another issue with >> large data sets. This time the behavior is the opposite from last time. The >> data is 13.5TB, the total number of matrix columns is 2.4 billion. Our >> program crashed during matrix loading due to memory overflow in one node. >> As said before, we have a little memory check during loading the matrix to >> keep track of rss. The printout of rss in the log shows normal increase in >> many nodes, i.e., if we load in a portion of the matrix that is 1GB, after >> MatSetValues for that portion, rss will increase roughly about 1GB. On the >> node that has memory overflow, the rss increased by 2GB after only 1GB of >> matrix is loaded through MatSetValues. We are very puzzled by this. What >> could make the memory footprint twice as much as needed? Thanks in advance >> for any insight. >> > > The only way I can imagine this happening is that you have not > preallocated correctly, so that some values are causing additional mallocs. > > Thanks, > > Matt > > >> Regards, >> >> Karl >> >> On Thu, Jun 11, 2020 at 12:00 PM Matthew Knepley >> wrote: >> >>> On Thu, Jun 11, 2020 at 12:52 PM Karl Lin wrote: >>> >>>> Hi, Matthew >>>> >>>> Thanks for the suggestion, just did another run and here are some >>>> detailed stack traces, maybe will provide some more insight: >>>> *** Process received signal *** >>>> Signal: Aborted (6) >>>> Signal code: (-6) >>>> /lib64/libpthread.so.0(+0xf5f0)[0x2b56c46dc5f0] >>>> [ 1] /lib64/libc.so.6(gsignal+0x37)[0x2b56c5486337] >>>> [ 2] /lib64/libc.so.6(abort+0x148)[0x2b56c5487a28] >>>> [ 3] /libpetsc.so.3.10(PetscTraceBackErrorHandler+0xc4)[0x2b56c1e6a2d4] >>>> [ 4] /libpetsc.so.3.10(PetscError+0x1b5)[0x2b56c1e69f65] >>>> [ 5] >>>> /libpetsc.so.3.10(PetscCommBuildTwoSidedFReq+0x19f0)[0x2b56c1e03cf0] >>>> [ 6] /libpetsc.so.3.10(+0x77db17)[0x2b56c2425b17] >>>> [ 7] /libpetsc.so.3.10(+0x77a164)[0x2b56c2422164] >>>> [ 8] /libpetsc.so.3.10(MatAssemblyBegin_MPIAIJ+0x36)[0x2b56c23912b6] >>>> [ 9] /libpetsc.so.3.10(MatAssemblyBegin+0xca)[0x2b56c1feccda] >>>> >>>> By reconfiguring, you mean recompiling petsc with that option, correct? >>>> >>> >>> Reconfiguring. >>> >>> Thanks, >>> >>> Matt >>> >>> >>>> Thank you. >>>> >>>> Karl >>>> >>>> On Thu, Jun 11, 2020 at 10:56 AM Matthew Knepley >>>> wrote: >>>> >>>>> On Thu, Jun 11, 2020 at 11:51 AM Karl Lin >>>>> wrote: >>>>> >>>>>> Hi, there >>>>>> >>>>>> We have written a program using Petsc to solve large sparse matrix >>>>>> system. It has been working fine for a while. Recently we encountered a >>>>>> problem when the size of the sparse matrix is larger than 10TB. We used >>>>>> several hundred nodes and 2200 processes. The program always crashes during >>>>>> MatAssemblyBegin.Upon a closer look, there seems to be something unusual. >>>>>> We have a little memory check during loading the matrix to keep track of >>>>>> rss. The printout of rss in the log shows normal increase up to rank 2160, >>>>>> i.e., if we load in a portion of matrix that is 1GB, after MatSetValues for >>>>>> that portion, rss will increase roughly about that number. From rank 2161 >>>>>> onwards, the rss in every rank doesn't increase after matrix loaded. Then >>>>>> comes MatAssemblyBegin, the program crashed on rank 2160. >>>>>> >>>>>> Is there a upper limit on the number of processes Petsc can handle? >>>>>> or is there a upper limit in terms of the size of the matrix petsc can >>>>>> handle? Thank you very much for any info. >>>>>> >>>>> >>>>> It sounds like you overflowed int somewhere. We try and check for >>>>> this, but catching every place is hard. Try reconfiguring with >>>>> >>>>> --with-64-bit-indices >>>>> >>>>> Thanks, >>>>> >>>>> Matt >>>>> >>>>> >>>>>> Regards, >>>>>> >>>>>> Karl >>>>>> >>>>> >>>>> >>>>> -- >>>>> What most experimenters take for granted before they begin their >>>>> experiments is infinitely more interesting than any results to which their >>>>> experiments lead. >>>>> -- Norbert Wiener >>>>> >>>>> https://www.cse.buffalo.edu/~knepley/ >>>>> >>>>> >>>> >>> >>> -- >>> What most experimenters take for granted before they begin their >>> experiments is infinitely more interesting than any results to which their >>> experiments lead. >>> -- Norbert Wiener >>> >>> https://www.cse.buffalo.edu/~knepley/ >>> >>> >> > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From junchao.zhang at gmail.com Thu Jul 2 23:21:06 2020 From: junchao.zhang at gmail.com (Junchao Zhang) Date: Thu, 2 Jul 2020 23:21:06 -0500 Subject: [petsc-users] matcreate and assembly issue In-Reply-To: References: Message-ID: Is it because indices for the nonzeros also need memory? --Junchao Zhang On Thu, Jul 2, 2020 at 10:04 PM Karl Lin wrote: > Hi, Matthew > > Thanks for the reply. However, I don't really get why additional malloc > would double the memory footprint. If I know there is only 1GB matrix being > loaded, there shouldn't be 2GB memory occupied even if Petsc needs to > allocate more space. > > regards, > > Karl > > On Thu, Jul 2, 2020 at 8:10 PM Matthew Knepley wrote: > >> On Thu, Jul 2, 2020 at 7:30 PM Karl Lin wrote: >> >>> Hi, Matt >>> >>> Thanks for the tip last time. We just encountered another issue with >>> large data sets. This time the behavior is the opposite from last time. The >>> data is 13.5TB, the total number of matrix columns is 2.4 billion. Our >>> program crashed during matrix loading due to memory overflow in one node. >>> As said before, we have a little memory check during loading the matrix to >>> keep track of rss. The printout of rss in the log shows normal increase in >>> many nodes, i.e., if we load in a portion of the matrix that is 1GB, after >>> MatSetValues for that portion, rss will increase roughly about 1GB. On the >>> node that has memory overflow, the rss increased by 2GB after only 1GB of >>> matrix is loaded through MatSetValues. We are very puzzled by this. What >>> could make the memory footprint twice as much as needed? Thanks in advance >>> for any insight. >>> >> >> The only way I can imagine this happening is that you have not >> preallocated correctly, so that some values are causing additional mallocs. >> >> Thanks, >> >> Matt >> >> >>> Regards, >>> >>> Karl >>> >>> On Thu, Jun 11, 2020 at 12:00 PM Matthew Knepley >>> wrote: >>> >>>> On Thu, Jun 11, 2020 at 12:52 PM Karl Lin >>>> wrote: >>>> >>>>> Hi, Matthew >>>>> >>>>> Thanks for the suggestion, just did another run and here are some >>>>> detailed stack traces, maybe will provide some more insight: >>>>> *** Process received signal *** >>>>> Signal: Aborted (6) >>>>> Signal code: (-6) >>>>> /lib64/libpthread.so.0(+0xf5f0)[0x2b56c46dc5f0] >>>>> [ 1] /lib64/libc.so.6(gsignal+0x37)[0x2b56c5486337] >>>>> [ 2] /lib64/libc.so.6(abort+0x148)[0x2b56c5487a28] >>>>> [ 3] >>>>> /libpetsc.so.3.10(PetscTraceBackErrorHandler+0xc4)[0x2b56c1e6a2d4] >>>>> [ 4] /libpetsc.so.3.10(PetscError+0x1b5)[0x2b56c1e69f65] >>>>> [ 5] >>>>> /libpetsc.so.3.10(PetscCommBuildTwoSidedFReq+0x19f0)[0x2b56c1e03cf0] >>>>> [ 6] /libpetsc.so.3.10(+0x77db17)[0x2b56c2425b17] >>>>> [ 7] /libpetsc.so.3.10(+0x77a164)[0x2b56c2422164] >>>>> [ 8] /libpetsc.so.3.10(MatAssemblyBegin_MPIAIJ+0x36)[0x2b56c23912b6] >>>>> [ 9] /libpetsc.so.3.10(MatAssemblyBegin+0xca)[0x2b56c1feccda] >>>>> >>>>> By reconfiguring, you mean recompiling petsc with that option, correct? >>>>> >>>> >>>> Reconfiguring. >>>> >>>> Thanks, >>>> >>>> Matt >>>> >>>> >>>>> Thank you. >>>>> >>>>> Karl >>>>> >>>>> On Thu, Jun 11, 2020 at 10:56 AM Matthew Knepley >>>>> wrote: >>>>> >>>>>> On Thu, Jun 11, 2020 at 11:51 AM Karl Lin >>>>>> wrote: >>>>>> >>>>>>> Hi, there >>>>>>> >>>>>>> We have written a program using Petsc to solve large sparse matrix >>>>>>> system. It has been working fine for a while. Recently we encountered a >>>>>>> problem when the size of the sparse matrix is larger than 10TB. We used >>>>>>> several hundred nodes and 2200 processes. The program always crashes during >>>>>>> MatAssemblyBegin.Upon a closer look, there seems to be something unusual. >>>>>>> We have a little memory check during loading the matrix to keep track of >>>>>>> rss. The printout of rss in the log shows normal increase up to rank 2160, >>>>>>> i.e., if we load in a portion of matrix that is 1GB, after MatSetValues for >>>>>>> that portion, rss will increase roughly about that number. From rank 2161 >>>>>>> onwards, the rss in every rank doesn't increase after matrix loaded. Then >>>>>>> comes MatAssemblyBegin, the program crashed on rank 2160. >>>>>>> >>>>>>> Is there a upper limit on the number of processes Petsc can handle? >>>>>>> or is there a upper limit in terms of the size of the matrix petsc can >>>>>>> handle? Thank you very much for any info. >>>>>>> >>>>>> >>>>>> It sounds like you overflowed int somewhere. We try and check for >>>>>> this, but catching every place is hard. Try reconfiguring with >>>>>> >>>>>> --with-64-bit-indices >>>>>> >>>>>> Thanks, >>>>>> >>>>>> Matt >>>>>> >>>>>> >>>>>>> Regards, >>>>>>> >>>>>>> Karl >>>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> What most experimenters take for granted before they begin their >>>>>> experiments is infinitely more interesting than any results to which their >>>>>> experiments lead. >>>>>> -- Norbert Wiener >>>>>> >>>>>> https://www.cse.buffalo.edu/~knepley/ >>>>>> >>>>>> >>>>> >>>> >>>> -- >>>> What most experimenters take for granted before they begin their >>>> experiments is infinitely more interesting than any results to which their >>>> experiments lead. >>>> -- Norbert Wiener >>>> >>>> https://www.cse.buffalo.edu/~knepley/ >>>> >>>> >>> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> -- Norbert Wiener >> >> https://www.cse.buffalo.edu/~knepley/ >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From EPrudencio at slb.com Thu Jul 2 23:40:28 2020 From: EPrudencio at slb.com (Ernesto Prudencio) Date: Fri, 3 Jul 2020 04:40:28 +0000 Subject: [petsc-users] [Ext] Re: matcreate and assembly issue In-Reply-To: References: Message-ID: Karl, Are you taking into account that every "integer" index might be 64 bits instead of 32 bits, depending on the PETSc configuration / compilation choices for PetscInt? Ernesto. From: petsc-users [mailto:petsc-users-bounces at mcs.anl.gov] On Behalf Of Junchao Zhang Sent: Thursday, July 2, 2020 11:21 PM To: Karl Lin Cc: PETSc users list Subject: [Ext] Re: [petsc-users] matcreate and assembly issue Is it because indices for the nonzeros also need memory? --Junchao Zhang On Thu, Jul 2, 2020 at 10:04 PM Karl Lin > wrote: Hi, Matthew Thanks for the reply. However, I don't really get why additional malloc would double the memory footprint. If I know there is only 1GB matrix being loaded, there shouldn't be 2GB memory occupied even if Petsc needs to allocate more space. regards, Karl On Thu, Jul 2, 2020 at 8:10 PM Matthew Knepley > wrote: On Thu, Jul 2, 2020 at 7:30 PM Karl Lin > wrote: Hi, Matt Thanks for the tip last time. We just encountered another issue with large data sets. This time the behavior is the opposite from last time. The data is 13.5TB, the total number of matrix columns is 2.4 billion. Our program crashed during matrix loading due to memory overflow in one node. As said before, we have a little memory check during loading the matrix to keep track of rss. The printout of rss in the log shows normal increase in many nodes, i.e., if we load in a portion of the matrix that is 1GB, after MatSetValues for that portion, rss will increase roughly about 1GB. On the node that has memory overflow, the rss increased by 2GB after only 1GB of matrix is loaded through MatSetValues. We are very puzzled by this. What could make the memory footprint twice as much as needed? Thanks in advance for any insight. The only way I can imagine this happening is that you have not preallocated correctly, so that some values are causing additional mallocs. Thanks, Matt Regards, Karl On Thu, Jun 11, 2020 at 12:00 PM Matthew Knepley > wrote: On Thu, Jun 11, 2020 at 12:52 PM Karl Lin > wrote: Hi, Matthew Thanks for the suggestion, just did another run and here are some detailed stack traces, maybe will provide some more insight: *** Process received signal *** Signal: Aborted (6) Signal code: (-6) /lib64/libpthread.so.0(+0xf5f0)[0x2b56c46dc5f0] [ 1] /lib64/libc.so.6(gsignal+0x37)[0x2b56c5486337] [ 2] /lib64/libc.so.6(abort+0x148)[0x2b56c5487a28] [ 3] /libpetsc.so.3.10(PetscTraceBackErrorHandler+0xc4)[0x2b56c1e6a2d4] [ 4] /libpetsc.so.3.10(PetscError+0x1b5)[0x2b56c1e69f65] [ 5] /libpetsc.so.3.10(PetscCommBuildTwoSidedFReq+0x19f0)[0x2b56c1e03cf0] [ 6] /libpetsc.so.3.10(+0x77db17)[0x2b56c2425b17] [ 7] /libpetsc.so.3.10(+0x77a164)[0x2b56c2422164] [ 8] /libpetsc.so.3.10(MatAssemblyBegin_MPIAIJ+0x36)[0x2b56c23912b6] [ 9] /libpetsc.so.3.10(MatAssemblyBegin+0xca)[0x2b56c1feccda] By reconfiguring, you mean recompiling petsc with that option, correct? Reconfiguring. Thanks, Matt Thank you. Karl On Thu, Jun 11, 2020 at 10:56 AM Matthew Knepley > wrote: On Thu, Jun 11, 2020 at 11:51 AM Karl Lin > wrote: Hi, there We have written a program using Petsc to solve large sparse matrix system. It has been working fine for a while. Recently we encountered a problem when the size of the sparse matrix is larger than 10TB. We used several hundred nodes and 2200 processes. The program always crashes during MatAssemblyBegin.Upon a closer look, there seems to be something unusual. We have a little memory check during loading the matrix to keep track of rss. The printout of rss in the log shows normal increase up to rank 2160, i.e., if we load in a portion of matrix that is 1GB, after MatSetValues for that portion, rss will increase roughly about that number. From rank 2161 onwards, the rss in every rank doesn't increase after matrix loaded. Then comes MatAssemblyBegin, the program crashed on rank 2160. Is there a upper limit on the number of processes Petsc can handle? or is there a upper limit in terms of the size of the matrix petsc can handle? Thank you very much for any info. It sounds like you overflowed int somewhere. We try and check for this, but catching every place is hard. Try reconfiguring with --with-64-bit-indices Thanks, Matt Regards, Karl -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ Schlumberger-Private -------------- next part -------------- An HTML attachment was scrubbed... URL: From mfadams at lbl.gov Fri Jul 3 07:58:16 2020 From: mfadams at lbl.gov (Mark Adams) Date: Fri, 3 Jul 2020 08:58:16 -0400 Subject: [petsc-users] DIVERGED_NANORINF when using HYPRE/BoomerAMG In-Reply-To: References: Message-ID: > > > > 3) Schwarz-smoothers > > #0 0x00007ffff50897f4 in dtrsm_ () from > /opt/lapack/3.8.0-gcc.7.4.0/lib64/libblas.so.3 > #1 0x00007ffff550924b in dpotrs_ () from > /opt/lapack/3.8.0-gcc.7.4.0/lib64/liblapack.so.3 > This seems to be Cholesky ... which would not work for you if you are not symmetric (Navier-Stokes). If GAMG works then and pilut going crazy but quite there yet, I would think there is a hypre interface problem of some sort. Hypre has been used a lot so this is a surprise. Do you use AIJ matrices? Not BAIJ or anything else. -------------- next part -------------- An HTML attachment was scrubbed... URL: From karl.linkui at gmail.com Fri Jul 3 09:44:05 2020 From: karl.linkui at gmail.com (Karl Lin) Date: Fri, 3 Jul 2020 09:44:05 -0500 Subject: [petsc-users] [Ext] Re: matcreate and assembly issue In-Reply-To: References: Message-ID: Yes, I did. The memory check for rss computes the memory footprint of column index using size of unsigned long long instead of int. For Junchao, I wonder if keeping track of which loaded columns are owned by the current process and which loaded columns are not owned also needs some memory storage. Just a wild thought. On Thu, Jul 2, 2020 at 11:40 PM Ernesto Prudencio wrote: > Karl, > > > > > > > > > > > > > > > > > > > > > > > > * Are you taking into account that every ?integer? index might be 64 > bits instead of 32 bits, depending on the PETSc configuration / compilation > choices for PetscInt? Ernesto. From: petsc-users > [mailto:petsc-users-bounces at mcs.anl.gov ] > On Behalf Of Junchao Zhang Sent: Thursday, July 2, 2020 11:21 PM To: Karl > Lin > Cc: PETSc users list > > Subject: [Ext] Re: > [petsc-users] matcreate and assembly issue Is it because indices for the > nonzeros also need memory? --Junchao Zhang On Thu, Jul 2, 2020 at 10:04 > PM Karl Lin > wrote: Hi, > Matthew Thanks for the reply. However, I don't really get why additional > malloc would double the memory footprint. If I know there is only 1GB > matrix being loaded, there shouldn't be 2GB memory occupied even if Petsc > needs to allocate more space. regards, Karl On Thu, Jul 2, 2020 at > 8:10 PM Matthew Knepley > wrote: On > Thu, Jul 2, 2020 at 7:30 PM Karl Lin > wrote: Hi, Matt Thanks for the tip last time. We > just encountered another issue with large data sets. This time the behavior > is the opposite from last time. The data is 13.5TB, the total number of > matrix columns is 2.4 billion. Our program crashed during matrix loading > due to memory overflow in one node. As said before, we have a little memory > check during loading the matrix to keep track of rss. The printout of rss > in the log shows normal increase in many nodes, i.e., if we load in a > portion of the matrix that is 1GB, after MatSetValues for that portion, rss > will increase roughly about 1GB. On the node that has memory overflow, the > rss increased by 2GB after only 1GB of matrix is loaded through > MatSetValues. We are very puzzled by this. What could make the memory > footprint twice as much as needed? Thanks in advance for any insight. The > only way I can imagine this happening is that you have not preallocated > correctly, so that some values are causing additional mallocs. Thanks, > Matt Regards, Karl On Thu, Jun 11, 2020 at 12:00 PM Matthew > Knepley > wrote: On Thu, Jun 11, 2020 > at 12:52 PM Karl Lin > wrote: > Hi, Matthew Thanks for the suggestion, just did another run and here are > some detailed stack traces, maybe will provide some more insight: *** > Process received signal *** Signal: Aborted (6) Signal code: (-6) > /lib64/libpthread.so.0(+0xf5f0)[0x2b56c46dc5f0] [ 1] > /lib64/libc.so.6(gsignal+0x37)[0x2b56c5486337] [ 2] > /lib64/libc.so.6(abort+0x148)[0x2b56c5487a28] [ 3] > /libpetsc.so.3.10(PetscTraceBackErrorHandler+0xc4)[0x2b56c1e6a2d4] [ 4] > /libpetsc.so.3.10(PetscError+0x1b5)[0x2b56c1e69f65] [ 5] > /libpetsc.so.3.10(PetscCommBuildTwoSidedFReq+0x19f0)[0x2b56c1e03cf0] [ 6] > /libpetsc.so.3.10(+0x77db17)[0x2b56c2425b17] [ 7] > /libpetsc.so.3.10(+0x77a164)[0x2b56c2422164] [ 8] > /libpetsc.so.3.10(MatAssemblyBegin_MPIAIJ+0x36)[0x2b56c23912b6] [ 9] > /libpetsc.so.3.10(MatAssemblyBegin+0xca)[0x2b56c1feccda] By > reconfiguring, you mean recompiling petsc with that option, correct? > Reconfiguring. Thanks, Matt Thank you. Karl On Thu, Jun 11, > 2020 at 10:56 AM Matthew Knepley > > wrote: On Thu, Jun 11, 2020 at 11:51 AM Karl Lin > wrote: Hi, there We have written a program using > Petsc to solve large sparse matrix system. It has been working fine for a > while. Recently we encountered a problem when the size of the sparse matrix > is larger than 10TB. We used several hundred nodes and 2200 processes. The > program always crashes during MatAssemblyBegin.Upon a closer look, there > seems to be something unusual. We have a little memory check during loading > the matrix to keep track of rss. The printout of rss in the log shows > normal increase up to rank 2160, i.e., if we load in a portion of matrix > that is 1GB, after MatSetValues for that portion, rss will increase roughly > about that number. From rank 2161 onwards, the rss in every rank doesn't > increase after matrix loaded. Then comes MatAssemblyBegin, the program > crashed on rank 2160. Is there a upper limit on the number of processes > Petsc can handle? or is there a upper limit in terms of the size of the > matrix petsc can handle? Thank you very much for any info. It sounds like > you overflowed int somewhere. We try and check for this, but catching every > place is hard. Try reconfiguring with --with-64-bit-indices Thanks, > Matt Regards, Karl -- What most experimenters take for > granted before they begin their experiments is infinitely more interesting > than any results to which their experiments lead. -- Norbert Wiener > https://www.cse.buffalo.edu/~knepley/ > > -- What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ > > -- What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ > > Schlumberger-Private * > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at petsc.dev Fri Jul 3 09:50:13 2020 From: bsmith at petsc.dev (Barry Smith) Date: Fri, 3 Jul 2020 09:50:13 -0500 Subject: [petsc-users] [Ext] Re: matcreate and assembly issue In-Reply-To: References: Message-ID: Karl, If a particular process is receiving values with MatSetValues() that belong to a different process it needs to allocate temporary space for those values. If there are many values destined for a different process this space can be arbitrarily large. The values are not pass to the final owning process until the MatAssemblyBegin/End calls. If you have not preallocated enough room the matrix actually makes a complete copy of itself with extra space for additional values, copies the values over and then deletes the old matrix this the memory use can double when the preallocation is not correct. Barry > On Jul 3, 2020, at 9:44 AM, Karl Lin wrote: > > Yes, I did. The memory check for rss computes the memory footprint of column index using size of unsigned long long instead of int. > > For Junchao, I wonder if keeping track of which loaded columns are owned by the current process and which loaded columns are not owned also needs some memory storage. Just a wild thought. > > On Thu, Jul 2, 2020 at 11:40 PM Ernesto Prudencio > wrote: > Karl, > > > > Are you taking into account that every ?integer? index might be 64 bits instead of 32 bits, depending on the PETSc configuration / compilation choices for PetscInt? > > > > Ernesto. > > > > From: petsc-users [mailto:petsc-users-bounces at mcs.anl.gov ] On Behalf Of Junchao Zhang > Sent: Thursday, July 2, 2020 11:21 PM > To: Karl Lin > > Cc: PETSc users list > > Subject: [Ext] Re: [petsc-users] matcreate and assembly issue > > > > Is it because indices for the nonzeros also need memory? > > --Junchao Zhang > > > > > > On Thu, Jul 2, 2020 at 10:04 PM Karl Lin > wrote: > > Hi, Matthew > > > > Thanks for the reply. However, I don't really get why additional malloc would double the memory footprint. If I know there is only 1GB matrix being loaded, there shouldn't be 2GB memory occupied even if Petsc needs to allocate more space. > > > > regards, > > > > Karl > > > > On Thu, Jul 2, 2020 at 8:10 PM Matthew Knepley > wrote: > > On Thu, Jul 2, 2020 at 7:30 PM Karl Lin > wrote: > > Hi, Matt > > > > Thanks for the tip last time. We just encountered another issue with large data sets. This time the behavior is the opposite from last time. The data is 13.5TB, the total number of matrix columns is 2.4 billion. Our program crashed during matrix loading due to memory overflow in one node. As said before, we have a little memory check during loading the matrix to keep track of rss. The printout of rss in the log shows normal increase in many nodes, i.e., if we load in a portion of the matrix that is 1GB, after MatSetValues for that portion, rss will increase roughly about 1GB. On the node that has memory overflow, the rss increased by 2GB after only 1GB of matrix is loaded through MatSetValues. We are very puzzled by this. What could make the memory footprint twice as much as needed? Thanks in advance for any insight. > > > > The only way I can imagine this happening is that you have not preallocated correctly, so that some values are causing additional mallocs. > > > > Thanks, > > > > Matt > > > > Regards, > > > > Karl > > > > On Thu, Jun 11, 2020 at 12:00 PM Matthew Knepley > wrote: > > On Thu, Jun 11, 2020 at 12:52 PM Karl Lin > wrote: > > Hi, Matthew > > > > Thanks for the suggestion, just did another run and here are some detailed stack traces, maybe will provide some more insight: > > *** Process received signal *** > Signal: Aborted (6) > Signal code: (-6) > /lib64/libpthread.so.0(+0xf5f0)[0x2b56c46dc5f0] > > [ 1] /lib64/libc.so.6(gsignal+0x37)[0x2b56c5486337] > [ 2] /lib64/libc.so.6(abort+0x148)[0x2b56c5487a28] > [ 3] /libpetsc.so.3.10(PetscTraceBackErrorHandler+0xc4)[0x2b56c1e6a2d4] > [ 4] /libpetsc.so.3.10(PetscError+0x1b5)[0x2b56c1e69f65] > [ 5] /libpetsc.so.3.10(PetscCommBuildTwoSidedFReq+0x19f0)[0x2b56c1e03cf0] > [ 6] /libpetsc.so.3.10(+0x77db17)[0x2b56c2425b17] > [ 7] /libpetsc.so.3.10(+0x77a164)[0x2b56c2422164] > [ 8] /libpetsc.so.3.10(MatAssemblyBegin_MPIAIJ+0x36)[0x2b56c23912b6] > [ 9] /libpetsc.so.3.10(MatAssemblyBegin+0xca)[0x2b56c1feccda] > > > > By reconfiguring, you mean recompiling petsc with that option, correct? > > > > Reconfiguring. > > > > Thanks, > > > > Matt > > > > Thank you. > > > > Karl > > > > On Thu, Jun 11, 2020 at 10:56 AM Matthew Knepley > wrote: > > On Thu, Jun 11, 2020 at 11:51 AM Karl Lin > wrote: > > Hi, there > > > > We have written a program using Petsc to solve large sparse matrix system. It has been working fine for a while. Recently we encountered a problem when the size of the sparse matrix is larger than 10TB. We used several hundred nodes and 2200 processes. The program always crashes during MatAssemblyBegin.Upon a closer look, there seems to be something unusual. We have a little memory check during loading the matrix to keep track of rss. The printout of rss in the log shows normal increase up to rank 2160, i.e., if we load in a portion of matrix that is 1GB, after MatSetValues for that portion, rss will increase roughly about that number. From rank 2161 onwards, the rss in every rank doesn't increase after matrix loaded. Then comes MatAssemblyBegin, the program crashed on rank 2160. > > > > Is there a upper limit on the number of processes Petsc can handle? or is there a upper limit in terms of the size of the matrix petsc can handle? Thank you very much for any info. > > > > It sounds like you overflowed int somewhere. We try and check for this, but catching every place is hard. Try reconfiguring with > > > > --with-64-bit-indices > > > > Thanks, > > > > Matt > > > > Regards, > > > > Karl > > > > > > -- > > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. > -- Norbert Wiener > > > > https://www.cse.buffalo.edu/~knepley/ > > > > > -- > > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. > -- Norbert Wiener > > > > https://www.cse.buffalo.edu/~knepley/ > > > > > -- > > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. > -- Norbert Wiener > > > > https://www.cse.buffalo.edu/~knepley/ > > > Schlumberger-Private -------------- next part -------------- An HTML attachment was scrubbed... URL: From karl.linkui at gmail.com Fri Jul 3 10:37:28 2020 From: karl.linkui at gmail.com (Karl Lin) Date: Fri, 3 Jul 2020 10:37:28 -0500 Subject: [petsc-users] [Ext] Re: matcreate and assembly issue In-Reply-To: References: Message-ID: Hi, Barry Thanks for the explanation. Following your tip, I have a guess. We use MatCreateAIJ to create the matrix, I believe this call will preallocate as well. Before this call we figure out the number of nonzeros per row for all rows and put those number in an array, say numNonZero. We pass numNonZero as d_nnz and o_nnz to MatCreateAIJ call, so essentially we preallocate twice as much as needed. For the process that double the memory footprint and crashed, there are a lot of values in both the diagonal and off-diagonal part for the process, so the temporary space gets filled up for both diagonal and off-diagonal parts of the matrix, also there are unused temporary space until MatAssembly, so gradually fill up the preallocated space which doubles the memory footprint. Once MatAssembly is done, the unused temporary space gets squeezed out, we return the correct memory footprint of the matrix. But before MatAssembly, large amount of unused temporary space needs to be kept because of the diagonal and off-diagonal pattern of the input. Would you say this is a plausible explanation? thank you. Regards, Karl On Fri, Jul 3, 2020 at 9:50 AM Barry Smith wrote: > > Karl, > > If a particular process is receiving values with MatSetValues() that > belong to a different process it needs to allocate temporary space for > those values. If there are many values destined for a different process > this space can be arbitrarily large. The values are not pass to the final > owning process until the MatAssemblyBegin/End calls. > > If you have not preallocated enough room the matrix actually makes a > complete copy of itself with extra space for additional values, copies the > values over and then deletes the old matrix this the memory use can double > when the preallocation is not correct. > > > Barry > > > On Jul 3, 2020, at 9:44 AM, Karl Lin wrote: > > Yes, I did. The memory check for rss computes the memory footprint of > column index using size of unsigned long long instead of int. > > For Junchao, I wonder if keeping track of which loaded columns are owned > by the current process and which loaded columns are not owned also needs > some memory storage. Just a wild thought. > > On Thu, Jul 2, 2020 at 11:40 PM Ernesto Prudencio > wrote: > >> Karl, >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> * Are you taking into account that every ?integer? index might be 64 bits >> instead of 32 bits, depending on the PETSc configuration / compilation >> choices for PetscInt? Ernesto. From: petsc-users >> [mailto:petsc-users-bounces at mcs.anl.gov ] >> On Behalf Of Junchao Zhang Sent: Thursday, July 2, 2020 11:21 PM To: Karl >> Lin > Cc: PETSc users list >> > Subject: [Ext] Re: >> [petsc-users] matcreate and assembly issue Is it because indices for the >> nonzeros also need memory? --Junchao Zhang On Thu, Jul 2, 2020 at 10:04 >> PM Karl Lin > wrote: Hi, >> Matthew Thanks for the reply. However, I don't really get why additional >> malloc would double the memory footprint. If I know there is only 1GB >> matrix being loaded, there shouldn't be 2GB memory occupied even if Petsc >> needs to allocate more space. regards, Karl On Thu, Jul 2, 2020 at >> 8:10 PM Matthew Knepley > wrote: On >> Thu, Jul 2, 2020 at 7:30 PM Karl Lin > > wrote: Hi, Matt Thanks for the tip last time. We >> just encountered another issue with large data sets. This time the behavior >> is the opposite from last time. The data is 13.5TB, the total number of >> matrix columns is 2.4 billion. Our program crashed during matrix loading >> due to memory overflow in one node. As said before, we have a little memory >> check during loading the matrix to keep track of rss. The printout of rss >> in the log shows normal increase in many nodes, i.e., if we load in a >> portion of the matrix that is 1GB, after MatSetValues for that portion, rss >> will increase roughly about 1GB. On the node that has memory overflow, the >> rss increased by 2GB after only 1GB of matrix is loaded through >> MatSetValues. We are very puzzled by this. What could make the memory >> footprint twice as much as needed? Thanks in advance for any insight. The >> only way I can imagine this happening is that you have not preallocated >> correctly, so that some values are causing additional mallocs. Thanks, >> Matt Regards, Karl On Thu, Jun 11, 2020 at 12:00 PM Matthew >> Knepley > wrote: On Thu, Jun 11, 2020 >> at 12:52 PM Karl Lin > wrote: >> Hi, Matthew Thanks for the suggestion, just did another run and here are >> some detailed stack traces, maybe will provide some more insight: *** >> Process received signal *** Signal: Aborted (6) Signal code: (-6) >> /lib64/libpthread.so.0(+0xf5f0)[0x2b56c46dc5f0] [ 1] >> /lib64/libc.so.6(gsignal+0x37)[0x2b56c5486337] [ 2] >> /lib64/libc.so.6(abort+0x148)[0x2b56c5487a28] [ 3] >> /libpetsc.so.3.10(PetscTraceBackErrorHandler+0xc4)[0x2b56c1e6a2d4] [ 4] >> /libpetsc.so.3.10(PetscError+0x1b5)[0x2b56c1e69f65] [ 5] >> /libpetsc.so.3.10(PetscCommBuildTwoSidedFReq+0x19f0)[0x2b56c1e03cf0] [ 6] >> /libpetsc.so.3.10(+0x77db17)[0x2b56c2425b17] [ 7] >> /libpetsc.so.3.10(+0x77a164)[0x2b56c2422164] [ 8] >> /libpetsc.so.3.10(MatAssemblyBegin_MPIAIJ+0x36)[0x2b56c23912b6] [ 9] >> /libpetsc.so.3.10(MatAssemblyBegin+0xca)[0x2b56c1feccda] By >> reconfiguring, you mean recompiling petsc with that option, correct? >> Reconfiguring. Thanks, Matt Thank you. Karl On Thu, Jun 11, >> 2020 at 10:56 AM Matthew Knepley > >> wrote: On Thu, Jun 11, 2020 at 11:51 AM Karl Lin > > wrote: Hi, there We have written a program using >> Petsc to solve large sparse matrix system. It has been working fine for a >> while. Recently we encountered a problem when the size of the sparse matrix >> is larger than 10TB. We used several hundred nodes and 2200 processes. The >> program always crashes during MatAssemblyBegin.Upon a closer look, there >> seems to be something unusual. We have a little memory check during loading >> the matrix to keep track of rss. The printout of rss in the log shows >> normal increase up to rank 2160, i.e., if we load in a portion of matrix >> that is 1GB, after MatSetValues for that portion, rss will increase roughly >> about that number. From rank 2161 onwards, the rss in every rank doesn't >> increase after matrix loaded. Then comes MatAssemblyBegin, the program >> crashed on rank 2160. Is there a upper limit on the number of processes >> Petsc can handle? or is there a upper limit in terms of the size of the >> matrix petsc can handle? Thank you very much for any info. It sounds like >> you overflowed int somewhere. We try and check for this, but catching every >> place is hard. Try reconfiguring with --with-64-bit-indices Thanks, >> Matt Regards, Karl -- What most experimenters take for >> granted before they begin their experiments is infinitely more interesting >> than any results to which their experiments lead. -- Norbert Wiener >> https://www.cse.buffalo.edu/~knepley/ >> >> -- What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ >> >> -- What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ >> >> Schlumberger-Private * >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Fri Jul 3 10:57:21 2020 From: knepley at gmail.com (Matthew Knepley) Date: Fri, 3 Jul 2020 11:57:21 -0400 Subject: [petsc-users] [Ext] Re: matcreate and assembly issue In-Reply-To: References: Message-ID: On Fri, Jul 3, 2020 at 11:38 AM Karl Lin wrote: > Hi, Barry > > Thanks for the explanation. Following your tip, I have a guess. We use > MatCreateAIJ to create the matrix, I believe this call will preallocate as > well. Before this call we figure out the number of nonzeros per row for all > rows and put those number in an array, say numNonZero. We pass numNonZero > as d_nnz and o_nnz to MatCreateAIJ call, so essentially we preallocate > twice as much as needed. For the process that double the memory footprint > and crashed, there are a lot of values in both the diagonal and > off-diagonal part for the process, so the temporary space gets filled up > for both diagonal and off-diagonal parts of the matrix, also there are > unused temporary space until MatAssembly, so gradually fill up the > preallocated space which doubles the memory footprint. Once MatAssembly is > done, the unused temporary space gets squeezed out, we return the correct > memory footprint of the matrix. But before MatAssembly, large amount of > unused temporary space needs to be kept because of the diagonal and > off-diagonal pattern of the input. Would you say this is a plausible > explanation? thank you. > Yes. We find that it takes a very small amount of time to just loop over the insertion twice, the first time counting the nonzeros. We built something to do this for you: https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MatPreallocatorPreallocate.html Thanks, Matt > Regards, > > Karl > > On Fri, Jul 3, 2020 at 9:50 AM Barry Smith wrote: > >> >> Karl, >> >> If a particular process is receiving values with MatSetValues() that >> belong to a different process it needs to allocate temporary space for >> those values. If there are many values destined for a different process >> this space can be arbitrarily large. The values are not pass to the final >> owning process until the MatAssemblyBegin/End calls. >> >> If you have not preallocated enough room the matrix actually makes a >> complete copy of itself with extra space for additional values, copies the >> values over and then deletes the old matrix this the memory use can double >> when the preallocation is not correct. >> >> >> Barry >> >> >> On Jul 3, 2020, at 9:44 AM, Karl Lin wrote: >> >> Yes, I did. The memory check for rss computes the memory footprint of >> column index using size of unsigned long long instead of int. >> >> For Junchao, I wonder if keeping track of which loaded columns are owned >> by the current process and which loaded columns are not owned also needs >> some memory storage. Just a wild thought. >> >> On Thu, Jul 2, 2020 at 11:40 PM Ernesto Prudencio >> wrote: >> >>> Karl, >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> * Are you taking into account that every ?integer? index might be 64 >>> bits instead of 32 bits, depending on the PETSc configuration / compilation >>> choices for PetscInt? Ernesto. From: petsc-users >>> [mailto:petsc-users-bounces at mcs.anl.gov ] >>> On Behalf Of Junchao Zhang Sent: Thursday, July 2, 2020 11:21 PM To: Karl >>> Lin > Cc: PETSc users list >>> > Subject: [Ext] Re: >>> [petsc-users] matcreate and assembly issue Is it because indices for the >>> nonzeros also need memory? --Junchao Zhang On Thu, Jul 2, 2020 at 10:04 >>> PM Karl Lin > wrote: Hi, >>> Matthew Thanks for the reply. However, I don't really get why additional >>> malloc would double the memory footprint. If I know there is only 1GB >>> matrix being loaded, there shouldn't be 2GB memory occupied even if Petsc >>> needs to allocate more space. regards, Karl On Thu, Jul 2, 2020 at >>> 8:10 PM Matthew Knepley > wrote: On >>> Thu, Jul 2, 2020 at 7:30 PM Karl Lin >> > wrote: Hi, Matt Thanks for the tip last time. We >>> just encountered another issue with large data sets. This time the behavior >>> is the opposite from last time. The data is 13.5TB, the total number of >>> matrix columns is 2.4 billion. Our program crashed during matrix loading >>> due to memory overflow in one node. As said before, we have a little memory >>> check during loading the matrix to keep track of rss. The printout of rss >>> in the log shows normal increase in many nodes, i.e., if we load in a >>> portion of the matrix that is 1GB, after MatSetValues for that portion, rss >>> will increase roughly about 1GB. On the node that has memory overflow, the >>> rss increased by 2GB after only 1GB of matrix is loaded through >>> MatSetValues. We are very puzzled by this. What could make the memory >>> footprint twice as much as needed? Thanks in advance for any insight. The >>> only way I can imagine this happening is that you have not preallocated >>> correctly, so that some values are causing additional mallocs. Thanks, >>> Matt Regards, Karl On Thu, Jun 11, 2020 at 12:00 PM Matthew >>> Knepley > wrote: On Thu, Jun 11, 2020 >>> at 12:52 PM Karl Lin > wrote: >>> Hi, Matthew Thanks for the suggestion, just did another run and here are >>> some detailed stack traces, maybe will provide some more insight: *** >>> Process received signal *** Signal: Aborted (6) Signal code: (-6) >>> /lib64/libpthread.so.0(+0xf5f0)[0x2b56c46dc5f0] [ 1] >>> /lib64/libc.so.6(gsignal+0x37)[0x2b56c5486337] [ 2] >>> /lib64/libc.so.6(abort+0x148)[0x2b56c5487a28] [ 3] >>> /libpetsc.so.3.10(PetscTraceBackErrorHandler+0xc4)[0x2b56c1e6a2d4] [ 4] >>> /libpetsc.so.3.10(PetscError+0x1b5)[0x2b56c1e69f65] [ 5] >>> /libpetsc.so.3.10(PetscCommBuildTwoSidedFReq+0x19f0)[0x2b56c1e03cf0] [ 6] >>> /libpetsc.so.3.10(+0x77db17)[0x2b56c2425b17] [ 7] >>> /libpetsc.so.3.10(+0x77a164)[0x2b56c2422164] [ 8] >>> /libpetsc.so.3.10(MatAssemblyBegin_MPIAIJ+0x36)[0x2b56c23912b6] [ 9] >>> /libpetsc.so.3.10(MatAssemblyBegin+0xca)[0x2b56c1feccda] By >>> reconfiguring, you mean recompiling petsc with that option, correct? >>> Reconfiguring. Thanks, Matt Thank you. Karl On Thu, Jun 11, >>> 2020 at 10:56 AM Matthew Knepley > >>> wrote: On Thu, Jun 11, 2020 at 11:51 AM Karl Lin >> > wrote: Hi, there We have written a program using >>> Petsc to solve large sparse matrix system. It has been working fine for a >>> while. Recently we encountered a problem when the size of the sparse matrix >>> is larger than 10TB. We used several hundred nodes and 2200 processes. The >>> program always crashes during MatAssemblyBegin.Upon a closer look, there >>> seems to be something unusual. We have a little memory check during loading >>> the matrix to keep track of rss. The printout of rss in the log shows >>> normal increase up to rank 2160, i.e., if we load in a portion of matrix >>> that is 1GB, after MatSetValues for that portion, rss will increase roughly >>> about that number. From rank 2161 onwards, the rss in every rank doesn't >>> increase after matrix loaded. Then comes MatAssemblyBegin, the program >>> crashed on rank 2160. Is there a upper limit on the number of processes >>> Petsc can handle? or is there a upper limit in terms of the size of the >>> matrix petsc can handle? Thank you very much for any info. It sounds like >>> you overflowed int somewhere. We try and check for this, but catching every >>> place is hard. Try reconfiguring with --with-64-bit-indices Thanks, >>> Matt Regards, Karl -- What most experimenters take for >>> granted before they begin their experiments is infinitely more interesting >>> than any results to which their experiments lead. -- Norbert Wiener >>> https://www.cse.buffalo.edu/~knepley/ >>> >>> -- What most experimenters take for granted before they begin their >>> experiments is infinitely more interesting than any results to which their >>> experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ >>> >>> -- What most experimenters take for granted before they begin their >>> experiments is infinitely more interesting than any results to which their >>> experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ >>> >>> Schlumberger-Private * >>> >> >> -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From karl.linkui at gmail.com Fri Jul 3 11:52:02 2020 From: karl.linkui at gmail.com (Karl Lin) Date: Fri, 3 Jul 2020 11:52:02 -0500 Subject: [petsc-users] [Ext] Re: matcreate and assembly issue In-Reply-To: References: Message-ID: Hi, Matthew Thanks for the reply. However, if the matrix is huge, like 13.5TB in our case, it will take significant amount of time to loop over insertion twice. Any other time and resource saving options? Thank you very much. Regards, Karl On Fri, Jul 3, 2020 at 10:57 AM Matthew Knepley wrote: > On Fri, Jul 3, 2020 at 11:38 AM Karl Lin wrote: > >> Hi, Barry >> >> Thanks for the explanation. Following your tip, I have a guess. We use >> MatCreateAIJ to create the matrix, I believe this call will preallocate as >> well. Before this call we figure out the number of nonzeros per row for all >> rows and put those number in an array, say numNonZero. We pass numNonZero >> as d_nnz and o_nnz to MatCreateAIJ call, so essentially we preallocate >> twice as much as needed. For the process that double the memory footprint >> and crashed, there are a lot of values in both the diagonal and >> off-diagonal part for the process, so the temporary space gets filled up >> for both diagonal and off-diagonal parts of the matrix, also there are >> unused temporary space until MatAssembly, so gradually fill up the >> preallocated space which doubles the memory footprint. Once MatAssembly is >> done, the unused temporary space gets squeezed out, we return the correct >> memory footprint of the matrix. But before MatAssembly, large amount of >> unused temporary space needs to be kept because of the diagonal and >> off-diagonal pattern of the input. Would you say this is a plausible >> explanation? thank you. >> > > Yes. We find that it takes a very small amount of time to just loop over > the insertion twice, the first time counting the nonzeros. We built > something to do this for you: > > > https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MatPreallocatorPreallocate.html > > Thanks, > > Matt > > >> Regards, >> >> Karl >> >> On Fri, Jul 3, 2020 at 9:50 AM Barry Smith wrote: >> >>> >>> Karl, >>> >>> If a particular process is receiving values with MatSetValues() that >>> belong to a different process it needs to allocate temporary space for >>> those values. If there are many values destined for a different process >>> this space can be arbitrarily large. The values are not pass to the final >>> owning process until the MatAssemblyBegin/End calls. >>> >>> If you have not preallocated enough room the matrix actually makes a >>> complete copy of itself with extra space for additional values, copies the >>> values over and then deletes the old matrix this the memory use can double >>> when the preallocation is not correct. >>> >>> >>> Barry >>> >>> >>> On Jul 3, 2020, at 9:44 AM, Karl Lin wrote: >>> >>> Yes, I did. The memory check for rss computes the memory footprint of >>> column index using size of unsigned long long instead of int. >>> >>> For Junchao, I wonder if keeping track of which loaded columns are owned >>> by the current process and which loaded columns are not owned also needs >>> some memory storage. Just a wild thought. >>> >>> On Thu, Jul 2, 2020 at 11:40 PM Ernesto Prudencio >>> wrote: >>> >>>> Karl, >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> * Are you taking into account that every ?integer? index might be 64 >>>> bits instead of 32 bits, depending on the PETSc configuration / compilation >>>> choices for PetscInt? Ernesto. From: petsc-users >>>> [mailto:petsc-users-bounces at mcs.anl.gov ] >>>> On Behalf Of Junchao Zhang Sent: Thursday, July 2, 2020 11:21 PM To: Karl >>>> Lin > Cc: PETSc users list >>>> > Subject: [Ext] Re: >>>> [petsc-users] matcreate and assembly issue Is it because indices for the >>>> nonzeros also need memory? --Junchao Zhang On Thu, Jul 2, 2020 at 10:04 >>>> PM Karl Lin > wrote: Hi, >>>> Matthew Thanks for the reply. However, I don't really get why additional >>>> malloc would double the memory footprint. If I know there is only 1GB >>>> matrix being loaded, there shouldn't be 2GB memory occupied even if Petsc >>>> needs to allocate more space. regards, Karl On Thu, Jul 2, 2020 at >>>> 8:10 PM Matthew Knepley > wrote: On >>>> Thu, Jul 2, 2020 at 7:30 PM Karl Lin >>> > wrote: Hi, Matt Thanks for the tip last time. We >>>> just encountered another issue with large data sets. This time the behavior >>>> is the opposite from last time. The data is 13.5TB, the total number of >>>> matrix columns is 2.4 billion. Our program crashed during matrix loading >>>> due to memory overflow in one node. As said before, we have a little memory >>>> check during loading the matrix to keep track of rss. The printout of rss >>>> in the log shows normal increase in many nodes, i.e., if we load in a >>>> portion of the matrix that is 1GB, after MatSetValues for that portion, rss >>>> will increase roughly about 1GB. On the node that has memory overflow, the >>>> rss increased by 2GB after only 1GB of matrix is loaded through >>>> MatSetValues. We are very puzzled by this. What could make the memory >>>> footprint twice as much as needed? Thanks in advance for any insight. The >>>> only way I can imagine this happening is that you have not preallocated >>>> correctly, so that some values are causing additional mallocs. Thanks, >>>> Matt Regards, Karl On Thu, Jun 11, 2020 at 12:00 PM Matthew >>>> Knepley > wrote: On Thu, Jun 11, 2020 >>>> at 12:52 PM Karl Lin > wrote: >>>> Hi, Matthew Thanks for the suggestion, just did another run and here are >>>> some detailed stack traces, maybe will provide some more insight: *** >>>> Process received signal *** Signal: Aborted (6) Signal code: (-6) >>>> /lib64/libpthread.so.0(+0xf5f0)[0x2b56c46dc5f0] [ 1] >>>> /lib64/libc.so.6(gsignal+0x37)[0x2b56c5486337] [ 2] >>>> /lib64/libc.so.6(abort+0x148)[0x2b56c5487a28] [ 3] >>>> /libpetsc.so.3.10(PetscTraceBackErrorHandler+0xc4)[0x2b56c1e6a2d4] [ 4] >>>> /libpetsc.so.3.10(PetscError+0x1b5)[0x2b56c1e69f65] [ 5] >>>> /libpetsc.so.3.10(PetscCommBuildTwoSidedFReq+0x19f0)[0x2b56c1e03cf0] [ 6] >>>> /libpetsc.so.3.10(+0x77db17)[0x2b56c2425b17] [ 7] >>>> /libpetsc.so.3.10(+0x77a164)[0x2b56c2422164] [ 8] >>>> /libpetsc.so.3.10(MatAssemblyBegin_MPIAIJ+0x36)[0x2b56c23912b6] [ 9] >>>> /libpetsc.so.3.10(MatAssemblyBegin+0xca)[0x2b56c1feccda] By >>>> reconfiguring, you mean recompiling petsc with that option, correct? >>>> Reconfiguring. Thanks, Matt Thank you. Karl On Thu, Jun 11, >>>> 2020 at 10:56 AM Matthew Knepley > >>>> wrote: On Thu, Jun 11, 2020 at 11:51 AM Karl Lin >>> > wrote: Hi, there We have written a program using >>>> Petsc to solve large sparse matrix system. It has been working fine for a >>>> while. Recently we encountered a problem when the size of the sparse matrix >>>> is larger than 10TB. We used several hundred nodes and 2200 processes. The >>>> program always crashes during MatAssemblyBegin.Upon a closer look, there >>>> seems to be something unusual. We have a little memory check during loading >>>> the matrix to keep track of rss. The printout of rss in the log shows >>>> normal increase up to rank 2160, i.e., if we load in a portion of matrix >>>> that is 1GB, after MatSetValues for that portion, rss will increase roughly >>>> about that number. From rank 2161 onwards, the rss in every rank doesn't >>>> increase after matrix loaded. Then comes MatAssemblyBegin, the program >>>> crashed on rank 2160. Is there a upper limit on the number of processes >>>> Petsc can handle? or is there a upper limit in terms of the size of the >>>> matrix petsc can handle? Thank you very much for any info. It sounds like >>>> you overflowed int somewhere. We try and check for this, but catching every >>>> place is hard. Try reconfiguring with --with-64-bit-indices Thanks, >>>> Matt Regards, Karl -- What most experimenters take for >>>> granted before they begin their experiments is infinitely more interesting >>>> than any results to which their experiments lead. -- Norbert Wiener >>>> https://www.cse.buffalo.edu/~knepley/ >>>> >>>> -- What most experimenters take for granted before they begin their >>>> experiments is infinitely more interesting than any results to which their >>>> experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ >>>> >>>> -- What most experimenters take for granted before they begin their >>>> experiments is infinitely more interesting than any results to which their >>>> experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ >>>> >>>> Schlumberger-Private * >>>> >>> >>> > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Fri Jul 3 12:17:40 2020 From: knepley at gmail.com (Matthew Knepley) Date: Fri, 3 Jul 2020 13:17:40 -0400 Subject: [petsc-users] [Ext] Re: matcreate and assembly issue In-Reply-To: References: Message-ID: On Fri, Jul 3, 2020 at 12:52 PM Karl Lin wrote: > Hi, Matthew > > Thanks for the reply. However, if the matrix is huge, like 13.5TB in our > case, it will take significant amount of time to loop over insertion twice. > Any other time and resource saving options? Thank you very much. > Do you think you could do it once and time it? I would be surprised if it takes even 1% of your total runtime, and I would also like to see the timing in that we might be able to optimize something for you. Thanks, Matt > Regards, > > Karl > > On Fri, Jul 3, 2020 at 10:57 AM Matthew Knepley wrote: > >> On Fri, Jul 3, 2020 at 11:38 AM Karl Lin wrote: >> >>> Hi, Barry >>> >>> Thanks for the explanation. Following your tip, I have a guess. We use >>> MatCreateAIJ to create the matrix, I believe this call will preallocate as >>> well. Before this call we figure out the number of nonzeros per row for all >>> rows and put those number in an array, say numNonZero. We pass numNonZero >>> as d_nnz and o_nnz to MatCreateAIJ call, so essentially we preallocate >>> twice as much as needed. For the process that double the memory footprint >>> and crashed, there are a lot of values in both the diagonal and >>> off-diagonal part for the process, so the temporary space gets filled up >>> for both diagonal and off-diagonal parts of the matrix, also there are >>> unused temporary space until MatAssembly, so gradually fill up the >>> preallocated space which doubles the memory footprint. Once MatAssembly is >>> done, the unused temporary space gets squeezed out, we return the correct >>> memory footprint of the matrix. But before MatAssembly, large amount of >>> unused temporary space needs to be kept because of the diagonal and >>> off-diagonal pattern of the input. Would you say this is a plausible >>> explanation? thank you. >>> >> >> Yes. We find that it takes a very small amount of time to just loop over >> the insertion twice, the first time counting the nonzeros. We built >> something to do this for you: >> >> >> https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MatPreallocatorPreallocate.html >> >> Thanks, >> >> Matt >> >> >>> Regards, >>> >>> Karl >>> >>> On Fri, Jul 3, 2020 at 9:50 AM Barry Smith wrote: >>> >>>> >>>> Karl, >>>> >>>> If a particular process is receiving values with MatSetValues() >>>> that belong to a different process it needs to allocate temporary space for >>>> those values. If there are many values destined for a different process >>>> this space can be arbitrarily large. The values are not pass to the final >>>> owning process until the MatAssemblyBegin/End calls. >>>> >>>> If you have not preallocated enough room the matrix actually makes >>>> a complete copy of itself with extra space for additional values, copies >>>> the values over and then deletes the old matrix this the memory use can >>>> double when the preallocation is not correct. >>>> >>>> >>>> Barry >>>> >>>> >>>> On Jul 3, 2020, at 9:44 AM, Karl Lin wrote: >>>> >>>> Yes, I did. The memory check for rss computes the memory footprint of >>>> column index using size of unsigned long long instead of int. >>>> >>>> For Junchao, I wonder if keeping track of which loaded columns are >>>> owned by the current process and which loaded columns are not owned also >>>> needs some memory storage. Just a wild thought. >>>> >>>> On Thu, Jul 2, 2020 at 11:40 PM Ernesto Prudencio >>>> wrote: >>>> >>>>> Karl, >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> * Are you taking into account that every ?integer? index might be 64 >>>>> bits instead of 32 bits, depending on the PETSc configuration / compilation >>>>> choices for PetscInt? Ernesto. From: petsc-users >>>>> [mailto:petsc-users-bounces at mcs.anl.gov ] >>>>> On Behalf Of Junchao Zhang Sent: Thursday, July 2, 2020 11:21 PM To: Karl >>>>> Lin > Cc: PETSc users list >>>>> > Subject: [Ext] Re: >>>>> [petsc-users] matcreate and assembly issue Is it because indices for the >>>>> nonzeros also need memory? --Junchao Zhang On Thu, Jul 2, 2020 at 10:04 >>>>> PM Karl Lin > wrote: Hi, >>>>> Matthew Thanks for the reply. However, I don't really get why additional >>>>> malloc would double the memory footprint. If I know there is only 1GB >>>>> matrix being loaded, there shouldn't be 2GB memory occupied even if Petsc >>>>> needs to allocate more space. regards, Karl On Thu, Jul 2, 2020 at >>>>> 8:10 PM Matthew Knepley > wrote: On >>>>> Thu, Jul 2, 2020 at 7:30 PM Karl Lin >>>> > wrote: Hi, Matt Thanks for the tip last time. We >>>>> just encountered another issue with large data sets. This time the behavior >>>>> is the opposite from last time. The data is 13.5TB, the total number of >>>>> matrix columns is 2.4 billion. Our program crashed during matrix loading >>>>> due to memory overflow in one node. As said before, we have a little memory >>>>> check during loading the matrix to keep track of rss. The printout of rss >>>>> in the log shows normal increase in many nodes, i.e., if we load in a >>>>> portion of the matrix that is 1GB, after MatSetValues for that portion, rss >>>>> will increase roughly about 1GB. On the node that has memory overflow, the >>>>> rss increased by 2GB after only 1GB of matrix is loaded through >>>>> MatSetValues. We are very puzzled by this. What could make the memory >>>>> footprint twice as much as needed? Thanks in advance for any insight. The >>>>> only way I can imagine this happening is that you have not preallocated >>>>> correctly, so that some values are causing additional mallocs. Thanks, >>>>> Matt Regards, Karl On Thu, Jun 11, 2020 at 12:00 PM Matthew >>>>> Knepley > wrote: On Thu, Jun 11, 2020 >>>>> at 12:52 PM Karl Lin > wrote: >>>>> Hi, Matthew Thanks for the suggestion, just did another run and here are >>>>> some detailed stack traces, maybe will provide some more insight: *** >>>>> Process received signal *** Signal: Aborted (6) Signal code: (-6) >>>>> /lib64/libpthread.so.0(+0xf5f0)[0x2b56c46dc5f0] [ 1] >>>>> /lib64/libc.so.6(gsignal+0x37)[0x2b56c5486337] [ 2] >>>>> /lib64/libc.so.6(abort+0x148)[0x2b56c5487a28] [ 3] >>>>> /libpetsc.so.3.10(PetscTraceBackErrorHandler+0xc4)[0x2b56c1e6a2d4] [ 4] >>>>> /libpetsc.so.3.10(PetscError+0x1b5)[0x2b56c1e69f65] [ 5] >>>>> /libpetsc.so.3.10(PetscCommBuildTwoSidedFReq+0x19f0)[0x2b56c1e03cf0] [ 6] >>>>> /libpetsc.so.3.10(+0x77db17)[0x2b56c2425b17] [ 7] >>>>> /libpetsc.so.3.10(+0x77a164)[0x2b56c2422164] [ 8] >>>>> /libpetsc.so.3.10(MatAssemblyBegin_MPIAIJ+0x36)[0x2b56c23912b6] [ 9] >>>>> /libpetsc.so.3.10(MatAssemblyBegin+0xca)[0x2b56c1feccda] By >>>>> reconfiguring, you mean recompiling petsc with that option, correct? >>>>> Reconfiguring. Thanks, Matt Thank you. Karl On Thu, Jun 11, >>>>> 2020 at 10:56 AM Matthew Knepley > >>>>> wrote: On Thu, Jun 11, 2020 at 11:51 AM Karl Lin >>>> > wrote: Hi, there We have written a program using >>>>> Petsc to solve large sparse matrix system. It has been working fine for a >>>>> while. Recently we encountered a problem when the size of the sparse matrix >>>>> is larger than 10TB. We used several hundred nodes and 2200 processes. The >>>>> program always crashes during MatAssemblyBegin.Upon a closer look, there >>>>> seems to be something unusual. We have a little memory check during loading >>>>> the matrix to keep track of rss. The printout of rss in the log shows >>>>> normal increase up to rank 2160, i.e., if we load in a portion of matrix >>>>> that is 1GB, after MatSetValues for that portion, rss will increase roughly >>>>> about that number. From rank 2161 onwards, the rss in every rank doesn't >>>>> increase after matrix loaded. Then comes MatAssemblyBegin, the program >>>>> crashed on rank 2160. Is there a upper limit on the number of processes >>>>> Petsc can handle? or is there a upper limit in terms of the size of the >>>>> matrix petsc can handle? Thank you very much for any info. It sounds like >>>>> you overflowed int somewhere. We try and check for this, but catching every >>>>> place is hard. Try reconfiguring with --with-64-bit-indices Thanks, >>>>> Matt Regards, Karl -- What most experimenters take for >>>>> granted before they begin their experiments is infinitely more interesting >>>>> than any results to which their experiments lead. -- Norbert Wiener >>>>> https://www.cse.buffalo.edu/~knepley/ >>>>> >>>>> -- What most experimenters take for granted before they begin their >>>>> experiments is infinitely more interesting than any results to which their >>>>> experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ >>>>> >>>>> -- What most experimenters take for granted before they begin their >>>>> experiments is infinitely more interesting than any results to which their >>>>> experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ >>>>> >>>>> Schlumberger-Private * >>>>> >>>> >>>> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> -- Norbert Wiener >> >> https://www.cse.buffalo.edu/~knepley/ >> >> > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From jed at jedbrown.org Fri Jul 3 12:54:18 2020 From: jed at jedbrown.org (Jed Brown) Date: Fri, 03 Jul 2020 11:54:18 -0600 Subject: [petsc-users] [Ext] Re: matcreate and assembly issue In-Reply-To: References: Message-ID: <878sg0iiwl.fsf@jedbrown.org> Karl Lin writes: > Thanks for the reply. However, if the matrix is huge, like 13.5TB in our > case, it will take significant amount of time to loop over insertion twice. > Any other time and resource saving options? Thank you very much. Where do the matrix entries come from? Counting nonzeros should run at near STREAM bandwidth, which is a 200-300 GB/s for modern 2-socket compute nodes. How many nodes do you need to have the memory capacity? On 100 nodes, that preallocation counting pass should take less than a second. From karl.linkui at gmail.com Fri Jul 3 13:59:44 2020 From: karl.linkui at gmail.com (Karl Lin) Date: Fri, 3 Jul 2020 13:59:44 -0500 Subject: [petsc-users] [Ext] Re: matcreate and assembly issue In-Reply-To: References: Message-ID: I can certainly do that. Now for Jed, also to clarify it more, our matrix is not stored in csr format. It is actually stored row by row in a multi-level data structure. There are other levels of complexity on top of that data structure. You literally have to go through all column indices row by row, undergo indices transformation from global to local, before the accurate count of diagonal and off-diagonal structures can be determined for every row. So that is why it takes a long time to read. But now I understand the nature of the issue, I will try to explore some options and report back to you guys. Thanks a lot. On Fri, Jul 3, 2020 at 12:17 PM Matthew Knepley wrote: > On Fri, Jul 3, 2020 at 12:52 PM Karl Lin wrote: > >> Hi, Matthew >> >> Thanks for the reply. However, if the matrix is huge, like 13.5TB in our >> case, it will take significant amount of time to loop over insertion twice. >> Any other time and resource saving options? Thank you very much. >> > > Do you think you could do it once and time it? I would be surprised if it > takes even 1% of your total runtime, and I would also > like to see the timing in that we might be able to optimize something for > you. > > Thanks, > > Matt > > >> Regards, >> >> Karl >> >> On Fri, Jul 3, 2020 at 10:57 AM Matthew Knepley >> wrote: >> >>> On Fri, Jul 3, 2020 at 11:38 AM Karl Lin wrote: >>> >>>> Hi, Barry >>>> >>>> Thanks for the explanation. Following your tip, I have a guess. We use >>>> MatCreateAIJ to create the matrix, I believe this call will preallocate as >>>> well. Before this call we figure out the number of nonzeros per row for all >>>> rows and put those number in an array, say numNonZero. We pass numNonZero >>>> as d_nnz and o_nnz to MatCreateAIJ call, so essentially we preallocate >>>> twice as much as needed. For the process that double the memory footprint >>>> and crashed, there are a lot of values in both the diagonal and >>>> off-diagonal part for the process, so the temporary space gets filled up >>>> for both diagonal and off-diagonal parts of the matrix, also there are >>>> unused temporary space until MatAssembly, so gradually fill up the >>>> preallocated space which doubles the memory footprint. Once MatAssembly is >>>> done, the unused temporary space gets squeezed out, we return the correct >>>> memory footprint of the matrix. But before MatAssembly, large amount of >>>> unused temporary space needs to be kept because of the diagonal and >>>> off-diagonal pattern of the input. Would you say this is a plausible >>>> explanation? thank you. >>>> >>> >>> Yes. We find that it takes a very small amount of time to just loop over >>> the insertion twice, the first time counting the nonzeros. We built >>> something to do this for you: >>> >>> >>> https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MatPreallocatorPreallocate.html >>> >>> Thanks, >>> >>> Matt >>> >>> >>>> Regards, >>>> >>>> Karl >>>> >>>> On Fri, Jul 3, 2020 at 9:50 AM Barry Smith wrote: >>>> >>>>> >>>>> Karl, >>>>> >>>>> If a particular process is receiving values with MatSetValues() >>>>> that belong to a different process it needs to allocate temporary space for >>>>> those values. If there are many values destined for a different process >>>>> this space can be arbitrarily large. The values are not pass to the final >>>>> owning process until the MatAssemblyBegin/End calls. >>>>> >>>>> If you have not preallocated enough room the matrix actually makes >>>>> a complete copy of itself with extra space for additional values, copies >>>>> the values over and then deletes the old matrix this the memory use can >>>>> double when the preallocation is not correct. >>>>> >>>>> >>>>> Barry >>>>> >>>>> >>>>> On Jul 3, 2020, at 9:44 AM, Karl Lin wrote: >>>>> >>>>> Yes, I did. The memory check for rss computes the memory footprint of >>>>> column index using size of unsigned long long instead of int. >>>>> >>>>> For Junchao, I wonder if keeping track of which loaded columns are >>>>> owned by the current process and which loaded columns are not owned also >>>>> needs some memory storage. Just a wild thought. >>>>> >>>>> On Thu, Jul 2, 2020 at 11:40 PM Ernesto Prudencio >>>>> wrote: >>>>> >>>>>> Karl, >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> * Are you taking into account that every ?integer? index might be 64 >>>>>> bits instead of 32 bits, depending on the PETSc configuration / compilation >>>>>> choices for PetscInt? Ernesto. From: petsc-users >>>>>> [mailto:petsc-users-bounces at mcs.anl.gov ] >>>>>> On Behalf Of Junchao Zhang Sent: Thursday, July 2, 2020 11:21 PM To: Karl >>>>>> Lin > Cc: PETSc users list >>>>>> > Subject: [Ext] Re: >>>>>> [petsc-users] matcreate and assembly issue Is it because indices for the >>>>>> nonzeros also need memory? --Junchao Zhang On Thu, Jul 2, 2020 at 10:04 >>>>>> PM Karl Lin > wrote: Hi, >>>>>> Matthew Thanks for the reply. However, I don't really get why additional >>>>>> malloc would double the memory footprint. If I know there is only 1GB >>>>>> matrix being loaded, there shouldn't be 2GB memory occupied even if Petsc >>>>>> needs to allocate more space. regards, Karl On Thu, Jul 2, 2020 at >>>>>> 8:10 PM Matthew Knepley > wrote: On >>>>>> Thu, Jul 2, 2020 at 7:30 PM Karl Lin >>>>> > wrote: Hi, Matt Thanks for the tip last time. We >>>>>> just encountered another issue with large data sets. This time the behavior >>>>>> is the opposite from last time. The data is 13.5TB, the total number of >>>>>> matrix columns is 2.4 billion. Our program crashed during matrix loading >>>>>> due to memory overflow in one node. As said before, we have a little memory >>>>>> check during loading the matrix to keep track of rss. The printout of rss >>>>>> in the log shows normal increase in many nodes, i.e., if we load in a >>>>>> portion of the matrix that is 1GB, after MatSetValues for that portion, rss >>>>>> will increase roughly about 1GB. On the node that has memory overflow, the >>>>>> rss increased by 2GB after only 1GB of matrix is loaded through >>>>>> MatSetValues. We are very puzzled by this. What could make the memory >>>>>> footprint twice as much as needed? Thanks in advance for any insight. The >>>>>> only way I can imagine this happening is that you have not preallocated >>>>>> correctly, so that some values are causing additional mallocs. Thanks, >>>>>> Matt Regards, Karl On Thu, Jun 11, 2020 at 12:00 PM Matthew >>>>>> Knepley > wrote: On Thu, Jun 11, 2020 >>>>>> at 12:52 PM Karl Lin > wrote: >>>>>> Hi, Matthew Thanks for the suggestion, just did another run and here are >>>>>> some detailed stack traces, maybe will provide some more insight: *** >>>>>> Process received signal *** Signal: Aborted (6) Signal code: (-6) >>>>>> /lib64/libpthread.so.0(+0xf5f0)[0x2b56c46dc5f0] [ 1] >>>>>> /lib64/libc.so.6(gsignal+0x37)[0x2b56c5486337] [ 2] >>>>>> /lib64/libc.so.6(abort+0x148)[0x2b56c5487a28] [ 3] >>>>>> /libpetsc.so.3.10(PetscTraceBackErrorHandler+0xc4)[0x2b56c1e6a2d4] [ 4] >>>>>> /libpetsc.so.3.10(PetscError+0x1b5)[0x2b56c1e69f65] [ 5] >>>>>> /libpetsc.so.3.10(PetscCommBuildTwoSidedFReq+0x19f0)[0x2b56c1e03cf0] [ 6] >>>>>> /libpetsc.so.3.10(+0x77db17)[0x2b56c2425b17] [ 7] >>>>>> /libpetsc.so.3.10(+0x77a164)[0x2b56c2422164] [ 8] >>>>>> /libpetsc.so.3.10(MatAssemblyBegin_MPIAIJ+0x36)[0x2b56c23912b6] [ 9] >>>>>> /libpetsc.so.3.10(MatAssemblyBegin+0xca)[0x2b56c1feccda] By >>>>>> reconfiguring, you mean recompiling petsc with that option, correct? >>>>>> Reconfiguring. Thanks, Matt Thank you. Karl On Thu, Jun 11, >>>>>> 2020 at 10:56 AM Matthew Knepley > >>>>>> wrote: On Thu, Jun 11, 2020 at 11:51 AM Karl Lin >>>>> > wrote: Hi, there We have written a program using >>>>>> Petsc to solve large sparse matrix system. It has been working fine for a >>>>>> while. Recently we encountered a problem when the size of the sparse matrix >>>>>> is larger than 10TB. We used several hundred nodes and 2200 processes. The >>>>>> program always crashes during MatAssemblyBegin.Upon a closer look, there >>>>>> seems to be something unusual. We have a little memory check during loading >>>>>> the matrix to keep track of rss. The printout of rss in the log shows >>>>>> normal increase up to rank 2160, i.e., if we load in a portion of matrix >>>>>> that is 1GB, after MatSetValues for that portion, rss will increase roughly >>>>>> about that number. From rank 2161 onwards, the rss in every rank doesn't >>>>>> increase after matrix loaded. Then comes MatAssemblyBegin, the program >>>>>> crashed on rank 2160. Is there a upper limit on the number of processes >>>>>> Petsc can handle? or is there a upper limit in terms of the size of the >>>>>> matrix petsc can handle? Thank you very much for any info. It sounds like >>>>>> you overflowed int somewhere. We try and check for this, but catching every >>>>>> place is hard. Try reconfiguring with --with-64-bit-indices Thanks, >>>>>> Matt Regards, Karl -- What most experimenters take for >>>>>> granted before they begin their experiments is infinitely more interesting >>>>>> than any results to which their experiments lead. -- Norbert Wiener >>>>>> https://www.cse.buffalo.edu/~knepley/ >>>>>> >>>>>> -- What most experimenters take for granted before they begin their >>>>>> experiments is infinitely more interesting than any results to which their >>>>>> experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ >>>>>> >>>>>> -- What most experimenters take for granted before they begin their >>>>>> experiments is infinitely more interesting than any results to which their >>>>>> experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ >>>>>> >>>>>> Schlumberger-Private * >>>>>> >>>>> >>>>> >>> >>> -- >>> What most experimenters take for granted before they begin their >>> experiments is infinitely more interesting than any results to which their >>> experiments lead. >>> -- Norbert Wiener >>> >>> https://www.cse.buffalo.edu/~knepley/ >>> >>> >> > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From paololampitella at hotmail.com Sun Jul 5 06:15:51 2020 From: paololampitella at hotmail.com (Paolo Lampitella) Date: Sun, 5 Jul 2020 11:15:51 +0000 Subject: [petsc-users] R: PETSc and Windows 10 In-Reply-To: References: <32B37EFD-DDFA-41AE-ACC7-CB5E6AE57996@enseeiht.fr> <0558CCF7-8203-4520-8673-3B0945C43233@enseeiht.fr> <20AB9F7A-4471-498B-9A78-AA48163900CB@enseeiht.fr> <94A8E366-2361-409E-84CE-B90EA54D9AF9@enseeiht.fr> <849E7E3B-8B54-4B40-A2B7-8DD5B55428B3@enseeiht.fr> , Message-ID: Dear all, I just want to update you on my journey to PETSc compilation in Windows under MSYS2+MINGW64 Unfortunately, I haven?t been able to compile petsc-slepc trough Freefem but, as my final goal required also Fortran bindings (but I only needed blas, lapack, metis and hypre), I decided to follow my own route using the useful information from Pierre. * I started by installing MPI from https://www.microsoft.com/en-us/download/details.aspx?id=100593. I don?t think the SDK is actually needed in my specific workflow, but I installed it as well together with mpisetup. * Then I installed MSYS2 just following the wizard. Opened the MSYS2 terminal and updated with pacman -Syuu, closed if asked, reopened it and used again pacman -Syuu several times until no more updates were available. Closed it and opened it back. * Under the MSYS2 terminal installed just the following packages: * pacman -S base-devel git gcc gcc-fortran * pacman -S mingw-w64-x86_64-toolchain * pacman -S mingw-w64-x86_64-cmake * pacman -S mingw-w64-x86_64-msmpi * Closed the MSYS2 terminal and opened the MINGW64 one, went to /mingw64/include and compiled my mpi module following https://www.scivision.dev/windows-mpi-msys2/: * gfortran -c mpi.f90 -fno-range-check -fallow-invalid-boz However, I will keep an eye on the MS-MPI GitHub repository because the fortran side seems to be far from perfect. * Then I downloaded the 3.13.3 version of petsc and configured it, still under the MINGW64 terminal, with the following command: /usr/bin/python ./configure --prefix=/home/paolo/petsc --with-ar=/usr/bin/ar --with-shared-libraries=0 --with-debugging=0 --with-windows-graphics=0 --with-x=0 COPTFLAGS="-O3 -mtune=native" CXXOPTFLAGS="-O3 -mtune=native" FOPTFLAGS="-O3 -mtune=native" FFLAGS=-fallow-invalid-boz --with-mpiexec="/C/Program\ Files/Microsoft\ MPI/Bin/mpiexec" --download-fblaslapack --download-metis --download-hypre --download-metis-cmake-arguments='-G "MSYS Makefiles"' --download-hypre-configure-arguments="--build=x86_64-linux-gnu --host=x86_64-linux-gnu" Note that I just bypassed uninstalling python in mingw64 (which doesn?t work) by using /usr/bin/python and that, as opposed to Pierre, I needed to also use the MSYS2 archiver (/usr/bin/ar) as opposed to the mingw64 one (/mingw64/bin/ar that shows up in the Pierre configure) as also mentioned here http://hillyuan.blogspot.com/2017/11/build-petsc-in-windows-under-mingw64.html, probably because of this issue https://stackoverflow.com/questions/37504625/ar-on-msys2-shell-receives-truncated-paths-when-called-from-makefile. Then make all, make install and make check all went smooth. Also, I don?t know exactly what with-x=0 and with-windows-graphics=0 do, but I think it is stuff that I don?t need (yet configure worked with windows-graphics as well). * Finally I launched make test. As some tests failed, I replicated the same install procedure on all the systems I have available on this same Windows machine (Ubuntu 20.04 and Centos 8 under a VirtualBox 6.0.22 VM, Ubuntu 20.04 under WSL1 and the MSYS2-MINGW64 toolchain). I am attaching a file with the results printed to screen (not sure about which file should be used for a comparison/check). Note, however, that the tests in MSYS2 started with some cyclic reference issues for some .mod files, but this doesn?t show up in any file I could check. I am still left with some doubts about the archiver, the cyclic reference errors and the differences in the test results, but I am able to link my code with petsc. Unfortunately, as this Windows porting is part of a large code restructuring, I can?t do much more with it, now, from my code. But if you can suggest some specific tutorial to use as test also for the parallel, I would be glad to dig deeper into the matter. Best regards Paolo Inviato da Posta per Windows 10 Da: Pierre Jolivet Inviato: marted? 30 giugno 2020 15:22 A: Paolo Lampitella Cc: Matthew Knepley; petsc-users Oggetto: Re: [petsc-users] PETSc and Windows 10 Please use the 3.13.2 tarball, this was fixed by Satish in the previous commit I already linked (https://gitlab.com/petsc/petsc/-/commit/2cd8068296b34e127f055bb32f556e3599f17523). (If you want FreeFEM to do the dirty work for you, just switch to the develop branch, and redo ?make petsc-slepc?) But I think you?ve got everything you need now for a smooth compilation :) Thanks, Pierre On 30 Jun 2020, at 3:09 PM, Paolo Lampitella > wrote: Dear Pierre, thanks for the fast response. Unfortunately it still fails, but now in the configure of ScaLAPACK (which means that it went ok for slepc, tetgen, metis, parmetis, ptscotch, superlu and suitesparse). The way I applied the modification is by manually editing the Makefile in the 3rdparty/ff-petsc folder, adding -fallow-invalid-boz to both CFLAGS and FFLAGS (this entry added by me). Then executed make petsc-slepc. As my project is much less ambitious, I have a good feeling that I will be able to use your Makefile successfully, but as I am kind of slow I tought that it would have been useful for you to know. The configure.log is attached. This time the error is: Rank mismatch between actual argument at (1) and actual argument at (2) (scalar and rank-1) in subroutine pclarf.f of ScaLAPACK. However, before attampting with my project, I have few questions about your Makefile, in particular this piece: --with-mpi-lib=/c/Windows/System32/msmpi.dll --with-mpi-include=/home/paolo/FreeFem-sources/3rdparty/include/msmpi --with-mpiexec="/C/Program\ Files/Microsoft\ MPI/Bin/mpiexec" I see from MPI.py that I should not use ?--with-mpi-lib/include? if I want to use my now working mpi wrappers. Is this correct? Paolo Inviato da Posta per Windows 10 Da: Pierre Jolivet Inviato: luned? 29 giugno 2020 21:37 A: Paolo Lampitella Cc: Matthew Knepley; petsc-users Oggetto: Re: [petsc-users] PETSc and Windows 10 I do not give up easily on Windows problems: 1) that?s around 50% of our (FreeFEM) user-base (and I want them to use PETSc and SLEPc, ofc?) 2) most people I work with from corporations just have Windows laptops/desktops and I always recommend MSYS because it?s very lightweight and you can pass .exe around 3) I?ve bothered enough Satish, Jed, and Matt on GitLab to take (at least partially) the blame now when it doesn?t work on MSYS That being said, the magic keyword is the added flag FFLAGS="-fallow-invalid-boz" (see, I told you ./configure issues were easier to deal with than the others). Here you?ll see that everything goes through just fine (sorry, it took me a long time to post this because everything is slow on my VM): 1) http://jolivet.perso.enseeiht.fr/win10/configure.log 2) http://jolivet.perso.enseeiht.fr/win10/make.log (both steps #1 and #2 in MSYS terminal, gcc/gfortran 10, MS-MPI see screenshot) 3) http://jolivet.perso.enseeiht.fr/win10/ex2.txt (Command Prompt, 4 processes + MUMPS, I can send you the .exe if you want to try on your machine) I just realize that I didn?t generate the Fortran bindings, but you can see I compiled MUMPS and ScaLAPACK, so that shouldn?t be a problem. Or if there is a problem, we will need to fix this in PETSc. I?ll push this added flag to the FreeFEM repo, thanks for reminding me of the brokenness of gcc/gfortran 10 + MS-MPI. Here is to hoping this won?t affect PETSc ./configure with previous gcc/gfortran version (unlikely, this option is apparently 13-year old https://gcc.gnu.org/bugzilla/show_bug.cgi?id=29471) Let me know of the next hiccup, if any. Thanks, Pierre On 29 Jun 2020, at 8:09 PM, Paolo Lampitella > wrote: Dear Pierre, thanks again for your time I guess there is no way for me to use the toolchain you are using (I don?t remember having any choice on which version of MSYS or GCC I could install) Paolo Inviato da Posta per Windows 10 Da: Pierre Jolivet Inviato: luned? 29 giugno 2020 20:01 A: Matthew Knepley Cc: Paolo Lampitella; petsc-users Oggetto: Re: [petsc-users] PETSc and Windows 10 On 29 Jun 2020, at 7:47 PM, Matthew Knepley > wrote: On Mon, Jun 29, 2020 at 1:35 PM Paolo Lampitella > wrote: Dear Pierre, sorry to bother you, but I already have some issues. What I did: * pacman -R mingw-w64-x86_64-python mingw-w64-x86_64-gdb (is gdb also troublesome?) * Followed points 6 and 7 at https://doc.freefem.org/introduction/installation.html#compilation-on-windows I first got a warning on the configure at point 6, as ?disable-hips is not recognized. Then, on make ?petsc-slepc? of point 7 (no SUDO=sudo flag was necessary) I got to this point: tar xzf ../pkg/petsc-lite-3.13.0.tar.gz patch -p1 < petsc-suitesparse.patch patching file petsc-3.13.0/config/BuildSystem/config/packages/SuiteSparse.py touch petsc-3.13.0/tag-tar cd petsc-3.13.0 && ./configure MAKEFLAGS='' \ --prefix=/home/paolo/freefem/ff-petsc//r \ --with-debugging=0 COPTFLAGS='-O3 -mtune=generic' CXXOPTFLAGS='-O3 -mtune=generic' FOPTFLAGS='-O3 -mtune=generic' --with-cxx-dialect=C++11 --with-ssl=0 --with-x=0 --with-fortran-bindings=0 --with-shared-libraries=0 --with-cc='gcc' --with-cxx='g++' --with-fc='gfortran' CXXFLAGS='-fno-stack-protector' CFLAGS='-fno-stack-protector' --with-scalar-type=real --with-mpi-lib='/c/Windows/System32/msmpi.dll' --with-mpi-include='/home/paolo/FreeFem-sources/3rdparty/include/msmpi' --with-mpiexec='/C/Program\ Files/Microsoft\ MPI/Bin/mpiexec' --with-blaslapack-include='' --with-blaslapack-lib='/mingw64/bin/libopenblas.dll' --download-scalapack --download-metis --download-ptscotch --download-mumps --download-hypre --download-parmetis --download-superlu --download-suitesparse --download-tetgen --download-slepc '--download-metis-cmake-arguments=-G "MSYS Makefiles"' '--download-parmetis-cmake-arguments=-G "MSYS Makefiles"' '--download-superlu-cmake-arguments=-G "MSYS Makefiles"' '--download-hypre-configure-arguments=--build=x86_64-linux-gnu --host=x86_64-linux-gnu' PETSC_ARCH=fr =============================================================================== Configuring PETSc to compile on your system =============================================================================== TESTING: FortranMPICheck from config.packages.MPI(config/BuildSystem/config/pack******************************************************************************* UNABLE to CONFIGURE with GIVEN OPTIONS (see configure.log for details): ------------------------------------------------------------------------------- Fortran error! mpi_init() could not be located! ******************************************************************************* make: *** [Makefile:210: petsc-3.13.0/tag-conf-real] Errore 1 Note that I didn?t add anything to any PATH variable, because this is not mentioned in your documentation. On a side note, this is the same error I got when trying to build PETSc in Cygwin with the default OpenMPI available in Cygwin. I am attaching the configure.log? it seems to me that the error comes from the configure trying to include the mpif.h in your folder and not using the -fallow-invalid-boz flag that I had to use, for example, to compile mpi.f90 into mpi.mod But I?m not sure why this is happening Pierre, Could this be due to gcc 10? Sorry, I?m slow. You are right. Our workers use gcc 9, everything is fine, but I see on my VM which I updated that I use gcc 10 and had to disable Fortran, I guess the MUMPS run I showcased was with a prior PETSc build. I?ll try to resolve this and will keep you posted. They really caught a lot of people off guard with gfortran 10? Thanks, Pierre Executing: gfortran -c -o /tmp/petsc-ur0cff6a/config.libraries/conftest.o -I/tmp/petsc-ur0cff6a/config.compilers -I/tmp/petsc-ur0cff6a/config.setCompilers -I/tmp/petsc-ur0cff6a/config.compilersFortran -I/tmp/petsc-ur0cff6a/config.libraries -Wall -ffree-line-length-0 -Wno-unused-dummy-argument -O3 -mtune=generic -I/home/paolo/FreeFem-sources/3rdparty/include/msmpi /tmp/petsc-ur0cff6a/config.libraries/conftest.F90 Possible ERROR while running compiler: exit code 1 stderr: C:/msys64/home/paolo/FreeFem-sources/3rdparty/include/msmpi/mpif.h:227:36: 227 | PARAMETER (MPI_DATATYPE_NULL=z'0c000000') | 1 Error: BOZ literal constant at (1) is neither a data-stmt-constant nor an actual argument to INT, REAL, DBLE, or CMPLX intrinsic function [see '-fno-allow-invalid-boz'] C:/msys64/home/paolo/FreeFem-sources/3rdparty/include/msmpi/mpif.h:303:27: 303 | PARAMETER (MPI_CHAR=z'4c000101') | 1 Error: BOZ literal constant at (1) is neither a data-stmt-constant nor an actual argument to INT, REAL, DBLE, or CMPLX intrinsic function [see '-fno-allow-invalid-boz'] C:/msys64/home/paolo/FreeFem-sources/3rdparty/include/msmpi/mpif.h:305:36: 305 | PARAMETER (MPI_UNSIGNED_CHAR=z'4c000102') | 1 Thanks, Matt Thanks Paolo Inviato da Posta per Windows 10 Da: Pierre Jolivet Inviato: luned? 29 giugno 2020 18:34 A: Paolo Lampitella Cc: Satish Balay; petsc-users Oggetto: Re: [petsc-users] PETSc and Windows 10 On 29 Jun 2020, at 6:27 PM, Paolo Lampitella > wrote: I think I made the first step of having mingw64 from msys2 working with ms-mpi. I found that the issue I was having was related to: https://gcc.gnu.org/bugzilla/show_bug.cgi?id=91556 and, probably (but impossible to check now), I was using an msys2 and/or mingw mpi package before this fix: https://github.com/msys2/MINGW-packages/commit/11b4cff3d2ec7411037b692b0ad5a9f3e9b9978d#diff-eac59989e3096be97d940c8f47b50fba Admittedly, I never used gcc 10 before on any machine. Still, I feel that reporting that sort of error in that way is, at least, misleading (I would have preferred the initial implementation as mentioned in the gcc bug track). A second thing that I was not used to, and made me more uncertain of the procedure I was following, is having to compile myself the mpi module. There are several version of this out there, but I decided to stick with this one: https://www.scivision.dev/windows-mpi-msys2/ even if there seems to be no need to include -fno-range-check and the current mpi.f90 version is different from the mpif.h as reported here: https://github.com/microsoft/Microsoft-MPI/issues/33 which, to me, are both signs of lack of attention on the fortran side by those that maintain this thing. In summary, this is the procedure I followed so far (on a 64 bit machine with Windows 10): * Install MSYS2 from https://www.msys2.org/ and just follow the install wizard * Open the MSYS2 terminal and execute: pacman -Syuu * Close the terminal when asked and reopen it * Keep executing ?pacman -Syuu? until nothing else needs to be updated * Close the MSYS2 terminal and reopen it (I guess because was in paranoid mode), then install packages with: pacman -S base-devel git gcc gcc-fortran bsdcpio lndir pax-git unzip pacman -S mingw-w64-x86_64-toolchain pacman -S mingw-w64-x86_64-msmpi pacman -S mingw-w64-x86_64-cmake pacman -S mingw-w64-x86_64-freeglut pacman -S mingw-w64-x86_64-gsl pacman -S mingw-w64-x86_64-libmicroutils pacman -S mingw-w64-x86_64-hdf5 pacman -S mingw-w64-x86_64-openblas pacman -S mingw-w64-x86_64-arpack pacman -S mingw-w64-x86_64-jq This set should include all the libraries mentioned by Pierre and/or used by his Jenkins, as the final scope here is to have PETSc and dependencies working. But I think that for pure MPI one could stop to msmpi (even, maybe, just install msmpi and have the dependencies figured out by pacman). Honestly, I don?t remember the exact order I used to install the packages, but this should not affect things. Also, as I was still in paranoid mode, I kept executing ?pacman -Syuu? after each package was installed. After this, close the MSYS2 terminal. * Open the MINGW64 terminal and create the .mod file out of the mpi.f90 file, as mentioned here https://www.scivision.dev/windows-mpi-msys2/, with: cd /mingw64/include gfortran mpif90 -c -fno-range-check -fallow-invalid-boz Ah, yes, that?s new to gfortran 10 (we use gfortran 9 on our workers), which is now what?s ship with MSYS2 (we haven?t updated yet). Sorry that I forgot about that. This is needed to ?USE mpi? (as opposed to INCLUDE ?mpif.h?) * Install the latest MS-MPI (both sdk and setup) from https://www.microsoft.com/en-us/download/details.aspx?id=100593 At this point I?ve been able to compile (using the MINGW64 terminal) different mpi test programs and they run as expected in the classical Windows prompt. I added this function to my .bashrc in MSYS2 in order to easily copy the required dependencies out of MSYS: function copydep() { ldd $1 | grep "=> /$2" | awk '{print $3}' | xargs -I '{}' cp -v '{}' .; } which can be used, with the MINGW64 terminal, by navigating to the folder where the final executable, say, my.exe, resides (even if under a Windows path) and executing: copydep my.exe mingw64 This, of course, must be done before actually trying to execute the .exe in the windows cmd prompt. Hopefully, I should now be able to follow Pierre?s instructions for PETSc (but first I wanna give a try to the system python before removing it) Looks like the hard part is over. It?s usually easier to deal with ./configure issues. If you have weird errors like ?incomplete Cygwin install? or whatever, this is the kind of issues I was referring to earlier. In that case, what I?d suggest is just, as before: pacman -R mingw-w64-x86_64-python mingw-w64-x86_64-gdb pacman -S python Thanks, Pierre Thanks Paolo -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: petsc_test.txt URL: From pierre.jolivet at enseeiht.fr Sun Jul 5 06:45:33 2020 From: pierre.jolivet at enseeiht.fr (Pierre Jolivet) Date: Sun, 5 Jul 2020 13:45:33 +0200 Subject: [petsc-users] PETSc and Windows 10 In-Reply-To: References: <32B37EFD-DDFA-41AE-ACC7-CB5E6AE57996@enseeiht.fr> <0558CCF7-8203-4520-8673-3B0945C43233@enseeiht.fr> <20AB9F7A-4471-498B-9A78-AA48163900CB@enseeiht.fr> <94A8E366-2361-409E-84CE-B90EA54D9AF9@enseeiht.fr> <849E7E3B-8B54-4B40-A2B7-8DD5B55428B3@enseeiht.fr> Message-ID: <246CA2C6-E66F-4EF0-8600-81AB741D17D8@enseeiht.fr> Hello Paolo, > On 5 Jul 2020, at 1:15 PM, Paolo Lampitella wrote: > > Dear all, > > I just want to update you on my journey to PETSc compilation in Windows under MSYS2+MINGW64 > > Unfortunately, I haven?t been able to compile petsc-slepc trough Freefem but, as my final goal required also Fortran bindings (but I only needed blas, lapack, metis and hypre), I decided to follow my own route using the useful information from Pierre. > > I started by installing MPI from https://www.microsoft.com/en-us/download/details.aspx?id=100593 . I don?t think the SDK is actually needed in my specific workflow, but I installed it as well together with mpisetup. > Then I installed MSYS2 just following the wizard. Opened the MSYS2 terminal and updated with pacman -Syuu, closed if asked, reopened it and used again pacman -Syuu several times until no more updates were available. Closed it and opened it back. > Under the MSYS2 terminal installed just the following packages: > > pacman -S base-devel git gcc gcc-fortran > pacman -S mingw-w64-x86_64-toolchain > pacman -S mingw-w64-x86_64-cmake > pacman -S mingw-w64-x86_64-msmpi > > Closed the MSYS2 terminal and opened the MINGW64 one, went to /mingw64/include and compiled my mpi module following https://www.scivision.dev/windows-mpi-msys2/ : > > gfortran -c mpi.f90 -fno-range-check -fallow-invalid-boz > > However, I will keep an eye on the MS-MPI GitHub repository because the fortran side seems to be far from perfect. > > Then I downloaded the 3.13.3 version of petsc and configured it, still under the MINGW64 terminal, with the following command: > > /usr/bin/python ./configure --prefix=/home/paolo/petsc --with-ar=/usr/bin/ar > --with-shared-libraries=0 --with-debugging=0 --with-windows-graphics=0 --with-x=0 > COPTFLAGS="-O3 -mtune=native" > CXXOPTFLAGS="-O3 -mtune=native" > FOPTFLAGS="-O3 -mtune=native" > FFLAGS=-fallow-invalid-boz > --with-mpiexec="/C/Program\ Files/Microsoft\ MPI/Bin/mpiexec" > --download-fblaslapack --download-metis --download-hypre > --download-metis-cmake-arguments='-G "MSYS Makefiles"' > --download-hypre-configure-arguments="--build=x86_64-linux-gnu --host=x86_64-linux-gnu" > > Note that I just bypassed uninstalling python in mingw64 (which doesn?t work) by using /usr/bin/python and that, as opposed to Pierre, I needed to also use the MSYS2 archiver (/usr/bin/ar) as opposed to the mingw64 one (/mingw64/bin/ar that shows up in the Pierre configure) as also mentioned here http://hillyuan.blogspot.com/2017/11/build-petsc-in-windows-under-mingw64.html , probably because of this issue https://stackoverflow.com/questions/37504625/ar-on-msys2-shell-receives-truncated-paths-when-called-from-makefile . You are right that you can avoid deinstalling mingw-w64-x86_64-python if you can supply the proper Python yourself (we don?t have that luxury in our Makefile). If you want to avoid using that AR, and stick to /mingw64/bin/ar (not sure what the pros and cons are), you can either: - use another PETSC_ARCH (very short, like pw, for petsc-windows); - use --with-single-library=0. See this post on GitLab https://gitlab.com/petsc/petsc/-/issues/647#note_373507681 The OS I?m referring to is indeed my Windows + MSYS2 box. Thanks, Pierre > Then make all, make install and make check all went smooth. Also, I don?t know exactly what with-x=0 and with-windows-graphics=0 do, but I think it is stuff that I don?t need (yet configure worked with windows-graphics as well). > > Finally I launched make test. As some tests failed, I replicated the same install procedure on all the systems I have available on this same Windows machine (Ubuntu 20.04 and Centos 8 under a VirtualBox 6.0.22 VM, Ubuntu 20.04 under WSL1 and the MSYS2-MINGW64 toolchain). I am attaching a file with the results printed to screen (not sure about which file should be used for a comparison/check). Note, however, that the tests in MSYS2 started with some cyclic reference issues for some .mod files, but this doesn?t show up in any file I could check. > > I am still left with some doubts about the archiver, the cyclic reference errors and the differences in the test results, but I am able to link my code with petsc. Unfortunately, as this Windows porting is part of a large code restructuring, I can?t do much more with it, now, from my code. But if you can suggest some specific tutorial to use as test also for the parallel, I would be glad to dig deeper into the matter. > > Best regards > > Paolo > > Inviato da Posta per Windows 10 > > Da: Pierre Jolivet > Inviato: marted? 30 giugno 2020 15:22 > A: Paolo Lampitella > Cc: Matthew Knepley ; petsc-users > Oggetto: Re: [petsc-users] PETSc and Windows 10 > > Please use the 3.13.2 tarball, this was fixed by Satish in the previous commit I already linked (https://gitlab.com/petsc/petsc/-/commit/2cd8068296b34e127f055bb32f556e3599f17523 ). > (If you want FreeFEM to do the dirty work for you, just switch to the develop branch, and redo ?make petsc-slepc?) > But I think you?ve got everything you need now for a smooth compilation :) > > Thanks, > Pierre > > > On 30 Jun 2020, at 3:09 PM, Paolo Lampitella > wrote: > > Dear Pierre, > > thanks for the fast response. Unfortunately it still fails, but now in the configure of ScaLAPACK > (which means that it went ok for slepc, tetgen, metis, parmetis, ptscotch, superlu and suitesparse). > > The way I applied the modification is by manually editing the Makefile in the 3rdparty/ff-petsc folder, adding -fallow-invalid-boz to both CFLAGS and FFLAGS (this entry added by me). Then executed make petsc-slepc. > > As my project is much less ambitious, I have a good feeling that I will be able to use your Makefile successfully, but as I am kind of slow I tought that it would have been useful for you to know. The configure.log is attached. This time the error is: > > Rank mismatch between actual argument at (1) and actual argument at (2) (scalar and rank-1) > > in subroutine pclarf.f of ScaLAPACK. > > However, before attampting with my project, I have few questions about your Makefile, in particular this piece: > > --with-mpi-lib=/c/Windows/System32/msmpi.dll --with-mpi-include=/home/paolo/FreeFem-sources/3rdparty/include/msmpi --with-mpiexec="/C/Program\ Files/Microsoft\ MPI/Bin/mpiexec" > > I see from MPI.py that I should not use ?--with-mpi-lib/include? if I want to use my now working mpi wrappers. Is this correct? > > Paolo > > Inviato da Posta per Windows 10 > > Da: Pierre Jolivet > Inviato: luned? 29 giugno 2020 21:37 > A: Paolo Lampitella > Cc: Matthew Knepley ; petsc-users > Oggetto: Re: [petsc-users] PETSc and Windows 10 > > I do not give up easily on Windows problems: > 1) that?s around 50% of our (FreeFEM) user-base (and I want them to use PETSc and SLEPc, ofc?) > 2) most people I work with from corporations just have Windows laptops/desktops and I always recommend MSYS because it?s very lightweight and you can pass .exe around > 3) I?ve bothered enough Satish, Jed, and Matt on GitLab to take (at least partially) the blame now when it doesn?t work on MSYS > > That being said, the magic keyword is the added flag FFLAGS="-fallow-invalid-boz" (see, I told you ./configure issues were easier to deal with than the others). > Here you?ll see that everything goes through just fine (sorry, it took me a long time to post this because everything is slow on my VM): > 1) http://jolivet.perso.enseeiht.fr/win10/configure.log > 2) http://jolivet.perso.enseeiht.fr/win10/make.log (both steps #1 and #2 in MSYS terminal, gcc/gfortran 10, MS-MPI see screenshot) > 3) http://jolivet.perso.enseeiht.fr/win10/ex2.txt (Command Prompt, 4 processes + MUMPS, I can send you the .exe if you want to try on your machine) > I just realize that I didn?t generate the Fortran bindings, but you can see I compiled MUMPS and ScaLAPACK, so that shouldn?t be a problem. > Or if there is a problem, we will need to fix this in PETSc. > > I?ll push this added flag to the FreeFEM repo, thanks for reminding me of the brokenness of gcc/gfortran 10 + MS-MPI. > Here is to hoping this won?t affect PETSc ./configure with previous gcc/gfortran version (unlikely, this option is apparently 13-year old https://gcc.gnu.org/bugzilla/show_bug.cgi?id=29471 ) > > Let me know of the next hiccup, if any. > Thanks, > Pierre > > > > On 29 Jun 2020, at 8:09 PM, Paolo Lampitella > wrote: > > Dear Pierre, > > thanks again for your time > > I guess there is no way for me to use the toolchain you are using (I don?t remember having any choice on which version of MSYS or GCC I could install) > > Paolo > > Inviato da Posta per Windows 10 > > Da: Pierre Jolivet > Inviato: luned? 29 giugno 2020 20:01 > A: Matthew Knepley > Cc: Paolo Lampitella ; petsc-users > Oggetto: Re: [petsc-users] PETSc and Windows 10 > > > > > > > On 29 Jun 2020, at 7:47 PM, Matthew Knepley > wrote: > > On Mon, Jun 29, 2020 at 1:35 PM Paolo Lampitella > wrote: > Dear Pierre, sorry to bother you, but I already have some issues. What I did: > > pacman -R mingw-w64-x86_64-python mingw-w64-x86_64-gdb (is gdb also troublesome?) > Followed points 6 and 7 at https://doc.freefem.org/introduction/installation.html#compilation-on-windows > I first got a warning on the configure at point 6, as ?disable-hips is not recognized. Then, on make ?petsc-slepc? of point 7 (no SUDO=sudo flag was necessary) I got to this point: > > tar xzf ../pkg/petsc-lite-3.13.0.tar.gz > patch -p1 < petsc-suitesparse.patch > patching file petsc-3.13.0/config/BuildSystem/config/packages/SuiteSparse.py > touch petsc-3.13.0/tag-tar > cd petsc-3.13.0 && ./configure MAKEFLAGS='' \ > --prefix=/home/paolo/freefem/ff-petsc//r \ > --with-debugging=0 COPTFLAGS='-O3 -mtune=generic' CXXOPTFLAGS='-O3 -mtune=generic' FOPTFLAGS='-O3 -mtune=generic' --with-cxx-dialect=C++11 --with-ssl=0 --with-x=0 --with-fortran-bindings=0 --with-shared-libraries=0 --with-cc='gcc' --with-cxx='g++' --with-fc='gfortran' CXXFLAGS='-fno-stack-protector' CFLAGS='-fno-stack-protector' --with-scalar-type=real --with-mpi-lib='/c/Windows/System32/msmpi.dll' --with-mpi-include='/home/paolo/FreeFem-sources/3rdparty/include/msmpi' --with-mpiexec='/C/Program\ Files/Microsoft\ MPI/Bin/mpiexec' --with-blaslapack-include='' --with-blaslapack-lib='/mingw64/bin/libopenblas.dll' --download-scalapack --download-metis --download-ptscotch --download-mumps --download-hypre --download-parmetis --download-superlu --download-suitesparse --download-tetgen --download-slepc '--download-metis-cmake-arguments=-G "MSYS Makefiles"' '--download-parmetis-cmake-arguments=-G "MSYS Makefiles"' '--download-superlu-cmake-arguments=-G "MSYS Makefiles"' '--download-hypre-configure-arguments=--build=x86_64-linux-gnu --host=x86_64-linux-gnu' PETSC_ARCH=fr > =============================================================================== > Configuring PETSc to compile on your system > =============================================================================== > TESTING: FortranMPICheck from config.packages.MPI(config/BuildSystem/config/pack******************************************************************************* > UNABLE to CONFIGURE with GIVEN OPTIONS (see configure.log for details): > ------------------------------------------------------------------------------- > Fortran error! mpi_init() could not be located! > ******************************************************************************* > > make: *** [Makefile:210: petsc-3.13.0/tag-conf-real] Errore 1 > > Note that I didn?t add anything to any PATH variable, because this is not mentioned in your documentation. > > On a side note, this is the same error I got when trying to build PETSc in Cygwin with the default OpenMPI available in Cygwin. > > I am attaching the configure.log? it seems to me that the error comes from the configure trying to include the mpif.h in your folder and not using the -fallow-invalid-boz flag that I had to use, for example, to compile mpi.f90 into mpi.mod > > But I?m not sure why this is happening > > Pierre, > > Could this be due to gcc 10? > > Sorry, I?m slow. You are right. Our workers use gcc 9, everything is fine, but I see on my VM which I updated that I use gcc 10 and had to disable Fortran, I guess the MUMPS run I showcased was with a prior PETSc build. > I?ll try to resolve this and will keep you posted. > They really caught a lot of people off guard with gfortran 10? > > Thanks, > Pierre > > > > > Executing: gfortran -c -o /tmp/petsc-ur0cff6a/config.libraries/conftest.o -I/tmp/petsc-ur0cff6a/config.compilers -I/tmp/petsc-ur0cff6a/config.setCompilers -I/tmp/petsc-ur0cff6a/config.compilersFortran -I/tmp/petsc-ur0cff6a/config.libraries -Wall -ffree-line-length-0 -Wno-unused-dummy-argument -O3 -mtune=generic -I/home/paolo/FreeFem-sources/3rdparty/include/msmpi /tmp/petsc-ur0cff6a/config.libraries/conftest.F90 > Possible ERROR while running compiler: exit code 1 > stderr: > C:/msys64/home/paolo/FreeFem-sources/3rdparty/include/msmpi/mpif.h:227:36: > > 227 | PARAMETER (MPI_DATATYPE_NULL=z'0c000000') > | 1 > Error: BOZ literal constant at (1) is neither a data-stmt-constant nor an actual argument to INT, REAL, DBLE, or CMPLX intrinsic function [see '-fno-allow-invalid-boz'] > C:/msys64/home/paolo/FreeFem-sources/3rdparty/include/msmpi/mpif.h:303:27: > > 303 | PARAMETER (MPI_CHAR=z'4c000101') > | 1 > Error: BOZ literal constant at (1) is neither a data-stmt-constant nor an actual argument to INT, REAL, DBLE, or CMPLX intrinsic function [see '-fno-allow-invalid-boz'] > C:/msys64/home/paolo/FreeFem-sources/3rdparty/include/msmpi/mpif.h:305:36: > > 305 | PARAMETER (MPI_UNSIGNED_CHAR=z'4c000102') > | 1 > > Thanks, > > Matt > > Thanks > > Paolo > > Inviato da Posta per Windows 10 > > Da: Pierre Jolivet > Inviato: luned? 29 giugno 2020 18:34 > A: Paolo Lampitella > Cc: Satish Balay ; petsc-users > Oggetto: Re: [petsc-users] PETSc and Windows 10 > > > > On 29 Jun 2020, at 6:27 PM, Paolo Lampitella > wrote: > > I think I made the first step of having mingw64 from msys2 working with ms-mpi. > > I found that the issue I was having was related to: > > https://gcc.gnu.org/bugzilla/show_bug.cgi?id=91556 > > and, probably (but impossible to check now), I was using an msys2 and/or mingw mpi package before this fix: > > https://github.com/msys2/MINGW-packages/commit/11b4cff3d2ec7411037b692b0ad5a9f3e9b9978d#diff-eac59989e3096be97d940c8f47b50fba > > Admittedly, I never used gcc 10 before on any machine. Still, I feel that reporting that sort of error in that way is, > at least, misleading (I would have preferred the initial implementation as mentioned in the gcc bug track). > > A second thing that I was not used to, and made me more uncertain of the procedure I was following, is having to compile myself the mpi module. There are several version of this out there, but I decided to stick with this one: > > https://www.scivision.dev/windows-mpi-msys2/ > > even if there seems to be no need to include -fno-range-check and the current mpi.f90 version is different from the mpif.h as reported here: > > https://github.com/microsoft/Microsoft-MPI/issues/33 > > which, to me, are both signs of lack of attention on the fortran side by those that maintain this thing. > > In summary, this is the procedure I followed so far (on a 64 bit machine with Windows 10): > > Install MSYS2 from https://www.msys2.org/ and just follow the install wizard > Open the MSYS2 terminal and execute: pacman -Syuu > Close the terminal when asked and reopen it > Keep executing ?pacman -Syuu? until nothing else needs to be updated > Close the MSYS2 terminal and reopen it (I guess because was in paranoid mode), then install packages with: > > pacman -S base-devel git gcc gcc-fortran bsdcpio lndir pax-git unzip > pacman -S mingw-w64-x86_64-toolchain > pacman -S mingw-w64-x86_64-msmpi > pacman -S mingw-w64-x86_64-cmake > pacman -S mingw-w64-x86_64-freeglut > pacman -S mingw-w64-x86_64-gsl > pacman -S mingw-w64-x86_64-libmicroutils > pacman -S mingw-w64-x86_64-hdf5 > pacman -S mingw-w64-x86_64-openblas > pacman -S mingw-w64-x86_64-arpack > pacman -S mingw-w64-x86_64-jq > > This set should include all the libraries mentioned by Pierre and/or used by his Jenkins, as the final scope here is to have PETSc and dependencies working. But I think that for pure MPI one could stop to msmpi (even, maybe, just install msmpi and have the dependencies figured out by pacman). Honestly, I don?t remember the exact order I used to install the packages, but this should not affect things. Also, as I was still in paranoid mode, I kept executing ?pacman -Syuu? after each package was installed. After this, close the MSYS2 terminal. > > Open the MINGW64 terminal and create the .mod file out of the mpi.f90 file, as mentioned here https://www.scivision.dev/windows-mpi-msys2/ , with: > > cd /mingw64/include > gfortran mpif90 -c -fno-range-check -fallow-invalid-boz > > Ah, yes, that?s new to gfortran 10 (we use gfortran 9 on our workers), which is now what?s ship with MSYS2 (we haven?t updated yet). Sorry that I forgot about that. > > This is needed to ?USE mpi? (as opposed to INCLUDE ?mpif.h?) > > Install the latest MS-MPI (both sdk and setup) from https://www.microsoft.com/en-us/download/details.aspx?id=100593 > > At this point I?ve been able to compile (using the MINGW64 terminal) different mpi test programs and they run as expected in the classical Windows prompt. I added this function to my .bashrc in MSYS2 in order to easily copy the required dependencies out of MSYS: > > function copydep() { ldd $1 | grep "=> /$2" | awk '{print $3}' | xargs -I '{}' cp -v '{}' .; } > > which can be used, with the MINGW64 terminal, by navigating to the folder where the final executable, say, my.exe, resides (even if under a Windows path) and executing: > > copydep my.exe mingw64 > > This, of course, must be done before actually trying to execute the .exe in the windows cmd prompt. > > Hopefully, I should now be able to follow Pierre?s instructions for PETSc (but first I wanna give a try to the system python before removing it) > > Looks like the hard part is over. It?s usually easier to deal with ./configure issues. > If you have weird errors like ?incomplete Cygwin install? or whatever, this is the kind of issues I was referring to earlier. > In that case, what I?d suggest is just, as before: > pacman -R mingw-w64-x86_64-python mingw-w64-x86_64-gdb > pacman -S python > > Thanks, > Pierre > > Thanks > > Paolo > > > > > -- > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ > > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From paololampitella at hotmail.com Sun Jul 5 07:00:45 2020 From: paololampitella at hotmail.com (Paolo Lampitella) Date: Sun, 5 Jul 2020 12:00:45 +0000 Subject: [petsc-users] PETSc and Windows 10 In-Reply-To: <246CA2C6-E66F-4EF0-8600-81AB741D17D8@enseeiht.fr> References: <32B37EFD-DDFA-41AE-ACC7-CB5E6AE57996@enseeiht.fr> <0558CCF7-8203-4520-8673-3B0945C43233@enseeiht.fr> <20AB9F7A-4471-498B-9A78-AA48163900CB@enseeiht.fr> <94A8E366-2361-409E-84CE-B90EA54D9AF9@enseeiht.fr> <849E7E3B-8B54-4B40-A2B7-8DD5B55428B3@enseeiht.fr> , <246CA2C6-E66F-4EF0-8600-81AB741D17D8@enseeiht.fr> Message-ID: Thank you very much Pierre. I'll keep you informed in case I see any relevant change from the tests when using your suggestion. Paolo Inviato da smartphone Samsung Galaxy. -------- Messaggio originale -------- Da: Pierre Jolivet Data: 05/07/20 13:45 (GMT+01:00) A: Paolo Lampitella Cc: Matthew Knepley , petsc-users Oggetto: Re: [petsc-users] PETSc and Windows 10 Hello Paolo, On 5 Jul 2020, at 1:15 PM, Paolo Lampitella > wrote: Dear all, I just want to update you on my journey to PETSc compilation in Windows under MSYS2+MINGW64 Unfortunately, I haven?t been able to compile petsc-slepc trough Freefem but, as my final goal required also Fortran bindings (but I only needed blas, lapack, metis and hypre), I decided to follow my own route using the useful information from Pierre. * I started by installing MPI from https://www.microsoft.com/en-us/download/details.aspx?id=100593. I don?t think the SDK is actually needed in my specific workflow, but I installed it as well together with mpisetup. * Then I installed MSYS2 just following the wizard. Opened the MSYS2 terminal and updated with pacman -Syuu, closed if asked, reopened it and used again pacman -Syuu several times until no more updates were available. Closed it and opened it back. * Under the MSYS2 terminal installed just the following packages: * pacman -S base-devel git gcc gcc-fortran * pacman -S mingw-w64-x86_64-toolchain * pacman -S mingw-w64-x86_64-cmake * pacman -S mingw-w64-x86_64-msmpi * Closed the MSYS2 terminal and opened the MINGW64 one, went to /mingw64/include and compiled my mpi module following https://www.scivision.dev/windows-mpi-msys2/: * gfortran -c mpi.f90 -fno-range-check -fallow-invalid-boz However, I will keep an eye on the MS-MPI GitHub repository because the fortran side seems to be far from perfect. * Then I downloaded the 3.13.3 version of petsc and configured it, still under the MINGW64 terminal, with the following command: /usr/bin/python ./configure --prefix=/home/paolo/petsc --with-ar=/usr/bin/ar --with-shared-libraries=0 --with-debugging=0 --with-windows-graphics=0 --with-x=0 COPTFLAGS="-O3 -mtune=native" CXXOPTFLAGS="-O3 -mtune=native" FOPTFLAGS="-O3 -mtune=native" FFLAGS=-fallow-invalid-boz --with-mpiexec="/C/Program\ Files/Microsoft\ MPI/Bin/mpiexec" --download-fblaslapack --download-metis --download-hypre --download-metis-cmake-arguments='-G "MSYS Makefiles"' --download-hypre-configure-arguments="--build=x86_64-linux-gnu --host=x86_64-linux-gnu" Note that I just bypassed uninstalling python in mingw64 (which doesn?t work) by using /usr/bin/python and that, as opposed to Pierre, I needed to also use the MSYS2 archiver (/usr/bin/ar) as opposed to the mingw64 one (/mingw64/bin/ar that shows up in the Pierre configure) as also mentioned here http://hillyuan.blogspot.com/2017/11/build-petsc-in-windows-under-mingw64.html, probably because of this issue https://stackoverflow.com/questions/37504625/ar-on-msys2-shell-receives-truncated-paths-when-called-from-makefile. You are right that you can avoid deinstalling mingw-w64-x86_64-python if you can supply the proper Python yourself (we don?t have that luxury in our Makefile). If you want to avoid using that AR, and stick to /mingw64/bin/ar (not sure what the pros and cons are), you can either: - use another PETSC_ARCH (very short, like pw, for petsc-windows); - use --with-single-library=0. See this post on GitLab https://gitlab.com/petsc/petsc/-/issues/647#note_373507681 The OS I?m referring to is indeed my Windows + MSYS2 box. Thanks, Pierre Then make all, make install and make check all went smooth. Also, I don?t know exactly what with-x=0 and with-windows-graphics=0 do, but I think it is stuff that I don?t need (yet configure worked with windows-graphics as well). * Finally I launched make test. As some tests failed, I replicated the same install procedure on all the systems I have available on this same Windows machine (Ubuntu 20.04 and Centos 8 under a VirtualBox 6.0.22 VM, Ubuntu 20.04 under WSL1 and the MSYS2-MINGW64 toolchain). I am attaching a file with the results printed to screen (not sure about which file should be used for a comparison/check). Note, however, that the tests in MSYS2 started with some cyclic reference issues for some .mod files, but this doesn?t show up in any file I could check. I am still left with some doubts about the archiver, the cyclic reference errors and the differences in the test results, but I am able to link my code with petsc. Unfortunately, as this Windows porting is part of a large code restructuring, I can?t do much more with it, now, from my code. But if you can suggest some specific tutorial to use as test also for the parallel, I would be glad to dig deeper into the matter. Best regards Paolo Inviato da Posta per Windows 10 Da: Pierre Jolivet Inviato: marted? 30 giugno 2020 15:22 A: Paolo Lampitella Cc: Matthew Knepley; petsc-users Oggetto: Re: [petsc-users] PETSc and Windows 10 Please use the 3.13.2 tarball, this was fixed by Satish in the previous commit I already linked (https://gitlab.com/petsc/petsc/-/commit/2cd8068296b34e127f055bb32f556e3599f17523). (If you want FreeFEM to do the dirty work for you, just switch to the develop branch, and redo ?make petsc-slepc?) But I think you?ve got everything you need now for a smooth compilation :) Thanks, Pierre On 30 Jun 2020, at 3:09 PM, Paolo Lampitella > wrote: Dear Pierre, thanks for the fast response. Unfortunately it still fails, but now in the configure of ScaLAPACK (which means that it went ok for slepc, tetgen, metis, parmetis, ptscotch, superlu and suitesparse). The way I applied the modification is by manually editing the Makefile in the 3rdparty/ff-petsc folder, adding -fallow-invalid-boz to both CFLAGS and FFLAGS (this entry added by me). Then executed make petsc-slepc. As my project is much less ambitious, I have a good feeling that I will be able to use your Makefile successfully, but as I am kind of slow I tought that it would have been useful for you to know. The configure.log is attached. This time the error is: Rank mismatch between actual argument at (1) and actual argument at (2) (scalar and rank-1) in subroutine pclarf.f of ScaLAPACK. However, before attampting with my project, I have few questions about your Makefile, in particular this piece: --with-mpi-lib=/c/Windows/System32/msmpi.dll --with-mpi-include=/home/paolo/FreeFem-sources/3rdparty/include/msmpi --with-mpiexec="/C/Program\ Files/Microsoft\ MPI/Bin/mpiexec" I see from MPI.py that I should not use ?--with-mpi-lib/include? if I want to use my now working mpi wrappers. Is this correct? Paolo Inviato da Posta per Windows 10 Da: Pierre Jolivet Inviato: luned? 29 giugno 2020 21:37 A: Paolo Lampitella Cc: Matthew Knepley; petsc-users Oggetto: Re: [petsc-users] PETSc and Windows 10 I do not give up easily on Windows problems: 1) that?s around 50% of our (FreeFEM) user-base (and I want them to use PETSc and SLEPc, ofc?) 2) most people I work with from corporations just have Windows laptops/desktops and I always recommend MSYS because it?s very lightweight and you can pass .exe around 3) I?ve bothered enough Satish, Jed, and Matt on GitLab to take (at least partially) the blame now when it doesn?t work on MSYS That being said, the magic keyword is the added flag FFLAGS="-fallow-invalid-boz" (see, I told you ./configure issues were easier to deal with than the others). Here you?ll see that everything goes through just fine (sorry, it took me a long time to post this because everything is slow on my VM): 1) http://jolivet.perso.enseeiht.fr/win10/configure.log 2) http://jolivet.perso.enseeiht.fr/win10/make.log (both steps #1 and #2 in MSYS terminal, gcc/gfortran 10, MS-MPI see screenshot) 3) http://jolivet.perso.enseeiht.fr/win10/ex2.txt (Command Prompt, 4 processes + MUMPS, I can send you the .exe if you want to try on your machine) I just realize that I didn?t generate the Fortran bindings, but you can see I compiled MUMPS and ScaLAPACK, so that shouldn?t be a problem. Or if there is a problem, we will need to fix this in PETSc. I?ll push this added flag to the FreeFEM repo, thanks for reminding me of the brokenness of gcc/gfortran 10 + MS-MPI. Here is to hoping this won?t affect PETSc ./configure with previous gcc/gfortran version (unlikely, this option is apparently 13-year old https://gcc.gnu.org/bugzilla/show_bug.cgi?id=29471) Let me know of the next hiccup, if any. Thanks, Pierre On 29 Jun 2020, at 8:09 PM, Paolo Lampitella > wrote: Dear Pierre, thanks again for your time I guess there is no way for me to use the toolchain you are using (I don?t remember having any choice on which version of MSYS or GCC I could install) Paolo Inviato da Posta per Windows 10 Da: Pierre Jolivet Inviato: luned? 29 giugno 2020 20:01 A: Matthew Knepley Cc: Paolo Lampitella; petsc-users Oggetto: Re: [petsc-users] PETSc and Windows 10 On 29 Jun 2020, at 7:47 PM, Matthew Knepley > wrote: On Mon, Jun 29, 2020 at 1:35 PM Paolo Lampitella > wrote: Dear Pierre, sorry to bother you, but I already have some issues. What I did: * pacman -R mingw-w64-x86_64-python mingw-w64-x86_64-gdb (is gdb also troublesome?) * Followed points 6 and 7 at https://doc.freefem.org/introduction/installation.html#compilation-on-windows I first got a warning on the configure at point 6, as ?disable-hips is not recognized. Then, on make ?petsc-slepc? of point 7 (no SUDO=sudo flag was necessary) I got to this point: tar xzf ../pkg/petsc-lite-3.13.0.tar.gz patch -p1 < petsc-suitesparse.patch patching file petsc-3.13.0/config/BuildSystem/config/packages/SuiteSparse.py touch petsc-3.13.0/tag-tar cd petsc-3.13.0 && ./configure MAKEFLAGS='' \ --prefix=/home/paolo/freefem/ff-petsc//r \ --with-debugging=0 COPTFLAGS='-O3 -mtune=generic' CXXOPTFLAGS='-O3 -mtune=generic' FOPTFLAGS='-O3 -mtune=generic' --with-cxx-dialect=C++11 --with-ssl=0 --with-x=0 --with-fortran-bindings=0 --with-shared-libraries=0 --with-cc='gcc' --with-cxx='g++' --with-fc='gfortran' CXXFLAGS='-fno-stack-protector' CFLAGS='-fno-stack-protector' --with-scalar-type=real --with-mpi-lib='/c/Windows/System32/msmpi.dll' --with-mpi-include='/home/paolo/FreeFem-sources/3rdparty/include/msmpi' --with-mpiexec='/C/Program\ Files/Microsoft\ MPI/Bin/mpiexec' --with-blaslapack-include='' --with-blaslapack-lib='/mingw64/bin/libopenblas.dll' --download-scalapack --download-metis --download-ptscotch --download-mumps --download-hypre --download-parmetis --download-superlu --download-suitesparse --download-tetgen --download-slepc '--download-metis-cmake-arguments=-G "MSYS Makefiles"' '--download-parmetis-cmake-arguments=-G "MSYS Makefiles"' '--download-superlu-cmake-arguments=-G "MSYS Makefiles"' '--download-hypre-configure-arguments=--build=x86_64-linux-gnu --host=x86_64-linux-gnu' PETSC_ARCH=fr =============================================================================== Configuring PETSc to compile on your system =============================================================================== TESTING: FortranMPICheck from config.packages.MPI(config/BuildSystem/config/pack******************************************************************************* UNABLE to CONFIGURE with GIVEN OPTIONS (see configure.log for details): ------------------------------------------------------------------------------- Fortran error! mpi_init() could not be located! ******************************************************************************* make: *** [Makefile:210: petsc-3.13.0/tag-conf-real] Errore 1 Note that I didn?t add anything to any PATH variable, because this is not mentioned in your documentation. On a side note, this is the same error I got when trying to build PETSc in Cygwin with the default OpenMPI available in Cygwin. I am attaching the configure.log? it seems to me that the error comes from the configure trying to include the mpif.h in your folder and not using the -fallow-invalid-boz flag that I had to use, for example, to compile mpi.f90 into mpi.mod But I?m not sure why this is happening Pierre, Could this be due to gcc 10? Sorry, I?m slow. You are right. Our workers use gcc 9, everything is fine, but I see on my VM which I updated that I use gcc 10 and had to disable Fortran, I guess the MUMPS run I showcased was with a prior PETSc build. I?ll try to resolve this and will keep you posted. They really caught a lot of people off guard with gfortran 10? Thanks, Pierre Executing: gfortran -c -o /tmp/petsc-ur0cff6a/config.libraries/conftest.o -I/tmp/petsc-ur0cff6a/config.compilers -I/tmp/petsc-ur0cff6a/config.setCompilers -I/tmp/petsc-ur0cff6a/config.compilersFortran -I/tmp/petsc-ur0cff6a/config.libraries -Wall -ffree-line-length-0 -Wno-unused-dummy-argument -O3 -mtune=generic -I/home/paolo/FreeFem-sources/3rdparty/include/msmpi /tmp/petsc-ur0cff6a/config.libraries/conftest.F90 Possible ERROR while running compiler: exit code 1 stderr: C:/msys64/home/paolo/FreeFem-sources/3rdparty/include/msmpi/mpif.h:227:36: 227 | PARAMETER (MPI_DATATYPE_NULL=z'0c000000') | 1 Error: BOZ literal constant at (1) is neither a data-stmt-constant nor an actual argument to INT, REAL, DBLE, or CMPLX intrinsic function [see '-fno-allow-invalid-boz'] C:/msys64/home/paolo/FreeFem-sources/3rdparty/include/msmpi/mpif.h:303:27: 303 | PARAMETER (MPI_CHAR=z'4c000101') | 1 Error: BOZ literal constant at (1) is neither a data-stmt-constant nor an actual argument to INT, REAL, DBLE, or CMPLX intrinsic function [see '-fno-allow-invalid-boz'] C:/msys64/home/paolo/FreeFem-sources/3rdparty/include/msmpi/mpif.h:305:36: 305 | PARAMETER (MPI_UNSIGNED_CHAR=z'4c000102') | 1 Thanks, Matt Thanks Paolo Inviato da Posta per Windows 10 Da: Pierre Jolivet Inviato: luned? 29 giugno 2020 18:34 A: Paolo Lampitella Cc: Satish Balay; petsc-users Oggetto: Re: [petsc-users] PETSc and Windows 10 On 29 Jun 2020, at 6:27 PM, Paolo Lampitella > wrote: I think I made the first step of having mingw64 from msys2 working with ms-mpi. I found that the issue I was having was related to: https://gcc.gnu.org/bugzilla/show_bug.cgi?id=91556 and, probably (but impossible to check now), I was using an msys2 and/or mingw mpi package before this fix: https://github.com/msys2/MINGW-packages/commit/11b4cff3d2ec7411037b692b0ad5a9f3e9b9978d#diff-eac59989e3096be97d940c8f47b50fba Admittedly, I never used gcc 10 before on any machine. Still, I feel that reporting that sort of error in that way is, at least, misleading (I would have preferred the initial implementation as mentioned in the gcc bug track). A second thing that I was not used to, and made me more uncertain of the procedure I was following, is having to compile myself the mpi module. There are several version of this out there, but I decided to stick with this one: https://www.scivision.dev/windows-mpi-msys2/ even if there seems to be no need to include -fno-range-check and the current mpi.f90 version is different from the mpif.h as reported here: https://github.com/microsoft/Microsoft-MPI/issues/33 which, to me, are both signs of lack of attention on the fortran side by those that maintain this thing. In summary, this is the procedure I followed so far (on a 64 bit machine with Windows 10): * Install MSYS2 from https://www.msys2.org/ and just follow the install wizard * Open the MSYS2 terminal and execute: pacman -Syuu * Close the terminal when asked and reopen it * Keep executing ?pacman -Syuu? until nothing else needs to be updated * Close the MSYS2 terminal and reopen it (I guess because was in paranoid mode), then install packages with: pacman -S base-devel git gcc gcc-fortran bsdcpio lndir pax-git unzip pacman -S mingw-w64-x86_64-toolchain pacman -S mingw-w64-x86_64-msmpi pacman -S mingw-w64-x86_64-cmake pacman -S mingw-w64-x86_64-freeglut pacman -S mingw-w64-x86_64-gsl pacman -S mingw-w64-x86_64-libmicroutils pacman -S mingw-w64-x86_64-hdf5 pacman -S mingw-w64-x86_64-openblas pacman -S mingw-w64-x86_64-arpack pacman -S mingw-w64-x86_64-jq This set should include all the libraries mentioned by Pierre and/or used by his Jenkins, as the final scope here is to have PETSc and dependencies working. But I think that for pure MPI one could stop to msmpi (even, maybe, just install msmpi and have the dependencies figured out by pacman). Honestly, I don?t remember the exact order I used to install the packages, but this should not affect things. Also, as I was still in paranoid mode, I kept executing ?pacman -Syuu? after each package was installed. After this, close the MSYS2 terminal. * Open the MINGW64 terminal and create the .mod file out of the mpi.f90 file, as mentioned here https://www.scivision.dev/windows-mpi-msys2/, with: cd /mingw64/include gfortran mpif90 -c -fno-range-check -fallow-invalid-boz Ah, yes, that?s new to gfortran 10 (we use gfortran 9 on our workers), which is now what?s ship with MSYS2 (we haven?t updated yet). Sorry that I forgot about that. This is needed to ?USE mpi? (as opposed to INCLUDE ?mpif.h?) * Install the latest MS-MPI (both sdk and setup) from https://www.microsoft.com/en-us/download/details.aspx?id=100593 At this point I?ve been able to compile (using the MINGW64 terminal) different mpi test programs and they run as expected in the classical Windows prompt. I added this function to my .bashrc in MSYS2 in order to easily copy the required dependencies out of MSYS: function copydep() { ldd $1 | grep "=> /$2" | awk '{print $3}' | xargs -I '{}' cp -v '{}' .; } which can be used, with the MINGW64 terminal, by navigating to the folder where the final executable, say, my.exe, resides (even if under a Windows path) and executing: copydep my.exe mingw64 This, of course, must be done before actually trying to execute the .exe in the windows cmd prompt. Hopefully, I should now be able to follow Pierre?s instructions for PETSc (but first I wanna give a try to the system python before removing it) Looks like the hard part is over. It?s usually easier to deal with ./configure issues. If you have weird errors like ?incomplete Cygwin install? or whatever, this is the kind of issues I was referring to earlier. In that case, what I?d suggest is just, as before: pacman -R mingw-w64-x86_64-python mingw-w64-x86_64-gdb pacman -S python Thanks, Pierre Thanks Paolo -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From ys453 at cam.ac.uk Sun Jul 5 12:50:54 2020 From: ys453 at cam.ac.uk (Y. Shidi) Date: Sun, 05 Jul 2020 18:50:54 +0100 Subject: [petsc-users] Performance of ISCreateGeneral() Message-ID: <74e13a698286435235fc4a95d85fa853@cam.ac.uk> Dear developers, I am currently doing a weak scaling test, and find that the weak scaling results for ISCreateGeneral() is very bad. For 2, 4, 8 processors, the running times for completing ISCreateGeneral() are: 0.687494, 3.00597 and 10.0613. I am not sure if this is normal. Thank you for your time. Kind Regards, Shidi From mfadams at lbl.gov Sun Jul 5 14:15:53 2020 From: mfadams at lbl.gov (Mark Adams) Date: Sun, 5 Jul 2020 15:15:53 -0400 Subject: [petsc-users] Performance of ISCreateGeneral() In-Reply-To: <74e13a698286435235fc4a95d85fa853@cam.ac.uk> References: <74e13a698286435235fc4a95d85fa853@cam.ac.uk> Message-ID: ISCreateGeneral just takes your indices and caches them. But it is a synch point. What ratio (max/min) is PETSc reporting in this data? On Sun, Jul 5, 2020 at 1:51 PM Y. Shidi wrote: > Dear developers, > > I am currently doing a weak scaling test, and find that > the weak scaling results for ISCreateGeneral() is very bad. > For 2, 4, 8 processors, the running times for completing > ISCreateGeneral() are: > 0.687494, 3.00597 and 10.0613. > I am not sure if this is normal. > > Thank you for your time. > > Kind Regards, > Shidi > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mfadams at lbl.gov Sun Jul 5 15:27:09 2020 From: mfadams at lbl.gov (Mark Adams) Date: Sun, 5 Jul 2020 16:27:09 -0400 Subject: [petsc-users] Performance of ISCreateGeneral() In-Reply-To: <0dbf7bfc2048833073995d291429f316@cam.ac.uk> References: <74e13a698286435235fc4a95d85fa853@cam.ac.uk> <0dbf7bfc2048833073995d291429f316@cam.ac.uk> Message-ID: The column after the time in the PETSC output, on the ISCreateGeneral row, with -log_view, shows the ratio of the max time (shown) to min time. If you are using your own timers just report the min and max time that you measure. You can also put an MPI_Barrier before the call to ISCreateGeneral to get clean data. And I trust that the problem is well load balance, that is, all processors have the same amount of data. On Sun, Jul 5, 2020 at 4:00 PM Y. Shidi wrote: > Thank you for your reply Mark. > What do you mean by ratio (max/min)? > Do I need to turn on some options for this? > > Kind regards, > Shidi > > On 2020-07-05 20:15, Mark Adams wrote: > > ISCreateGeneral just takes your indices and caches them. But it is a > > synch point. What ratio (max/min) is PETSc reporting in this data? > > > > On Sun, Jul 5, 2020 at 1:51 PM Y. Shidi wrote: > > > >> Dear developers, > >> > >> I am currently doing a weak scaling test, and find that > >> the weak scaling results for ISCreateGeneral() is very bad. > >> For 2, 4, 8 processors, the running times for completing > >> ISCreateGeneral() are: > >> 0.687494, 3.00597 and 10.0613. > >> I am not sure if this is normal. > >> > >> Thank you for your time. > >> > >> Kind Regards, > >> Shidi > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at petsc.dev Sun Jul 5 15:55:48 2020 From: bsmith at petsc.dev (Barry Smith) Date: Sun, 5 Jul 2020 15:55:48 -0500 Subject: [petsc-users] Performance of ISCreateGeneral() In-Reply-To: References: <74e13a698286435235fc4a95d85fa853@cam.ac.uk> <0dbf7bfc2048833073995d291429f316@cam.ac.uk> Message-ID: <5A0E4E0D-F873-41F0-8F9F-A402265C80E1@petsc.dev> Mark is correct. Since ISCreateGeneral is labeled in the manual page as Collective this means that all processes synchronize during the call. Hence if any process gets to the call long before the other processes it will look like a great deal of time is being spent in the call, when in fact it is just waiting for the other processes. You should check the code before the call the ISCreateGeneral() for load balancing issues. Barry > On Jul 5, 2020, at 3:27 PM, Mark Adams wrote: > > The column after the time in the PETSC output, on the ISCreateGeneral row, with -log_view, shows the ratio of the max time (shown) to min time. > If you are using your own timers just report the min and max time that you measure. > You can also put an MPI_Barrier before the call to ISCreateGeneral to get clean data. > And I trust that the problem is well load balance, that is, all processors have the same amount of data. > > On Sun, Jul 5, 2020 at 4:00 PM Y. Shidi > wrote: > Thank you for your reply Mark. > What do you mean by ratio (max/min)? > Do I need to turn on some options for this? > > Kind regards, > Shidi > > On 2020-07-05 20:15, Mark Adams wrote: > > ISCreateGeneral just takes your indices and caches them. But it is a > > synch point. What ratio (max/min) is PETSc reporting in this data? > > > > On Sun, Jul 5, 2020 at 1:51 PM Y. Shidi > wrote: > > > >> Dear developers, > >> > >> I am currently doing a weak scaling test, and find that > >> the weak scaling results for ISCreateGeneral() is very bad. > >> For 2, 4, 8 processors, the running times for completing > >> ISCreateGeneral() are: > >> 0.687494, 3.00597 and 10.0613. > >> I am not sure if this is normal. > >> > >> Thank you for your time. > >> > >> Kind Regards, > >> Shidi -------------- next part -------------- An HTML attachment was scrubbed... URL: From paololampitella at hotmail.com Sun Jul 5 16:16:25 2020 From: paololampitella at hotmail.com (Paolo Lampitella) Date: Sun, 5 Jul 2020 21:16:25 +0000 Subject: [petsc-users] PETSc and Windows 10 In-Reply-To: References: <32B37EFD-DDFA-41AE-ACC7-CB5E6AE57996@enseeiht.fr> <0558CCF7-8203-4520-8673-3B0945C43233@enseeiht.fr> <20AB9F7A-4471-498B-9A78-AA48163900CB@enseeiht.fr> <94A8E366-2361-409E-84CE-B90EA54D9AF9@enseeiht.fr> <849E7E3B-8B54-4B40-A2B7-8DD5B55428B3@enseeiht.fr> , <246CA2C6-E66F-4EF0-8600-81AB741D17D8@enseeiht.fr>, Message-ID: Unfortunately, even PETSC_ARCH=i didn't work out. And while with-single-library=0 wasn't really appealing to me, it worked but only to later fail on make test. I guess all these differences are due to the fortran bindings and/or gcc 10. However, until I discover how they are different, I guess I'll be fine with /usr/bin/ar Paolo Inviato da smartphone Samsung Galaxy. -------- Messaggio originale -------- Da: Paolo Lampitella Data: 05/07/20 14:00 (GMT+01:00) A: Pierre Jolivet Cc: Matthew Knepley , petsc-users Oggetto: RE: [petsc-users] PETSc and Windows 10 Thank you very much Pierre. I'll keep you informed in case I see any relevant change from the tests when using your suggestion. Paolo Inviato da smartphone Samsung Galaxy. -------- Messaggio originale -------- Da: Pierre Jolivet Data: 05/07/20 13:45 (GMT+01:00) A: Paolo Lampitella Cc: Matthew Knepley , petsc-users Oggetto: Re: [petsc-users] PETSc and Windows 10 Hello Paolo, On 5 Jul 2020, at 1:15 PM, Paolo Lampitella > wrote: Dear all, I just want to update you on my journey to PETSc compilation in Windows under MSYS2+MINGW64 Unfortunately, I haven?t been able to compile petsc-slepc trough Freefem but, as my final goal required also Fortran bindings (but I only needed blas, lapack, metis and hypre), I decided to follow my own route using the useful information from Pierre. * I started by installing MPI from https://www.microsoft.com/en-us/download/details.aspx?id=100593. I don?t think the SDK is actually needed in my specific workflow, but I installed it as well together with mpisetup. * Then I installed MSYS2 just following the wizard. Opened the MSYS2 terminal and updated with pacman -Syuu, closed if asked, reopened it and used again pacman -Syuu several times until no more updates were available. Closed it and opened it back. * Under the MSYS2 terminal installed just the following packages: * pacman -S base-devel git gcc gcc-fortran * pacman -S mingw-w64-x86_64-toolchain * pacman -S mingw-w64-x86_64-cmake * pacman -S mingw-w64-x86_64-msmpi * Closed the MSYS2 terminal and opened the MINGW64 one, went to /mingw64/include and compiled my mpi module following https://www.scivision.dev/windows-mpi-msys2/: * gfortran -c mpi.f90 -fno-range-check -fallow-invalid-boz However, I will keep an eye on the MS-MPI GitHub repository because the fortran side seems to be far from perfect. * Then I downloaded the 3.13.3 version of petsc and configured it, still under the MINGW64 terminal, with the following command: /usr/bin/python ./configure --prefix=/home/paolo/petsc --with-ar=/usr/bin/ar --with-shared-libraries=0 --with-debugging=0 --with-windows-graphics=0 --with-x=0 COPTFLAGS="-O3 -mtune=native" CXXOPTFLAGS="-O3 -mtune=native" FOPTFLAGS="-O3 -mtune=native" FFLAGS=-fallow-invalid-boz --with-mpiexec="/C/Program\ Files/Microsoft\ MPI/Bin/mpiexec" --download-fblaslapack --download-metis --download-hypre --download-metis-cmake-arguments='-G "MSYS Makefiles"' --download-hypre-configure-arguments="--build=x86_64-linux-gnu --host=x86_64-linux-gnu" Note that I just bypassed uninstalling python in mingw64 (which doesn?t work) by using /usr/bin/python and that, as opposed to Pierre, I needed to also use the MSYS2 archiver (/usr/bin/ar) as opposed to the mingw64 one (/mingw64/bin/ar that shows up in the Pierre configure) as also mentioned here http://hillyuan.blogspot.com/2017/11/build-petsc-in-windows-under-mingw64.html, probably because of this issue https://stackoverflow.com/questions/37504625/ar-on-msys2-shell-receives-truncated-paths-when-called-from-makefile. You are right that you can avoid deinstalling mingw-w64-x86_64-python if you can supply the proper Python yourself (we don?t have that luxury in our Makefile). If you want to avoid using that AR, and stick to /mingw64/bin/ar (not sure what the pros and cons are), you can either: - use another PETSC_ARCH (very short, like pw, for petsc-windows); - use --with-single-library=0. See this post on GitLab https://gitlab.com/petsc/petsc/-/issues/647#note_373507681 The OS I?m referring to is indeed my Windows + MSYS2 box. Thanks, Pierre Then make all, make install and make check all went smooth. Also, I don?t know exactly what with-x=0 and with-windows-graphics=0 do, but I think it is stuff that I don?t need (yet configure worked with windows-graphics as well). * Finally I launched make test. As some tests failed, I replicated the same install procedure on all the systems I have available on this same Windows machine (Ubuntu 20.04 and Centos 8 under a VirtualBox 6.0.22 VM, Ubuntu 20.04 under WSL1 and the MSYS2-MINGW64 toolchain). I am attaching a file with the results printed to screen (not sure about which file should be used for a comparison/check). Note, however, that the tests in MSYS2 started with some cyclic reference issues for some .mod files, but this doesn?t show up in any file I could check. I am still left with some doubts about the archiver, the cyclic reference errors and the differences in the test results, but I am able to link my code with petsc. Unfortunately, as this Windows porting is part of a large code restructuring, I can?t do much more with it, now, from my code. But if you can suggest some specific tutorial to use as test also for the parallel, I would be glad to dig deeper into the matter. Best regards Paolo Inviato da Posta per Windows 10 Da: Pierre Jolivet Inviato: marted? 30 giugno 2020 15:22 A: Paolo Lampitella Cc: Matthew Knepley; petsc-users Oggetto: Re: [petsc-users] PETSc and Windows 10 Please use the 3.13.2 tarball, this was fixed by Satish in the previous commit I already linked (https://gitlab.com/petsc/petsc/-/commit/2cd8068296b34e127f055bb32f556e3599f17523). (If you want FreeFEM to do the dirty work for you, just switch to the develop branch, and redo ?make petsc-slepc?) But I think you?ve got everything you need now for a smooth compilation :) Thanks, Pierre On 30 Jun 2020, at 3:09 PM, Paolo Lampitella > wrote: Dear Pierre, thanks for the fast response. Unfortunately it still fails, but now in the configure of ScaLAPACK (which means that it went ok for slepc, tetgen, metis, parmetis, ptscotch, superlu and suitesparse). The way I applied the modification is by manually editing the Makefile in the 3rdparty/ff-petsc folder, adding -fallow-invalid-boz to both CFLAGS and FFLAGS (this entry added by me). Then executed make petsc-slepc. As my project is much less ambitious, I have a good feeling that I will be able to use your Makefile successfully, but as I am kind of slow I tought that it would have been useful for you to know. The configure.log is attached. This time the error is: Rank mismatch between actual argument at (1) and actual argument at (2) (scalar and rank-1) in subroutine pclarf.f of ScaLAPACK. However, before attampting with my project, I have few questions about your Makefile, in particular this piece: --with-mpi-lib=/c/Windows/System32/msmpi.dll --with-mpi-include=/home/paolo/FreeFem-sources/3rdparty/include/msmpi --with-mpiexec="/C/Program\ Files/Microsoft\ MPI/Bin/mpiexec" I see from MPI.py that I should not use ?--with-mpi-lib/include? if I want to use my now working mpi wrappers. Is this correct? Paolo Inviato da Posta per Windows 10 Da: Pierre Jolivet Inviato: luned? 29 giugno 2020 21:37 A: Paolo Lampitella Cc: Matthew Knepley; petsc-users Oggetto: Re: [petsc-users] PETSc and Windows 10 I do not give up easily on Windows problems: 1) that?s around 50% of our (FreeFEM) user-base (and I want them to use PETSc and SLEPc, ofc?) 2) most people I work with from corporations just have Windows laptops/desktops and I always recommend MSYS because it?s very lightweight and you can pass .exe around 3) I?ve bothered enough Satish, Jed, and Matt on GitLab to take (at least partially) the blame now when it doesn?t work on MSYS That being said, the magic keyword is the added flag FFLAGS="-fallow-invalid-boz" (see, I told you ./configure issues were easier to deal with than the others). Here you?ll see that everything goes through just fine (sorry, it took me a long time to post this because everything is slow on my VM): 1) http://jolivet.perso.enseeiht.fr/win10/configure.log 2) http://jolivet.perso.enseeiht.fr/win10/make.log (both steps #1 and #2 in MSYS terminal, gcc/gfortran 10, MS-MPI see screenshot) 3) http://jolivet.perso.enseeiht.fr/win10/ex2.txt (Command Prompt, 4 processes + MUMPS, I can send you the .exe if you want to try on your machine) I just realize that I didn?t generate the Fortran bindings, but you can see I compiled MUMPS and ScaLAPACK, so that shouldn?t be a problem. Or if there is a problem, we will need to fix this in PETSc. I?ll push this added flag to the FreeFEM repo, thanks for reminding me of the brokenness of gcc/gfortran 10 + MS-MPI. Here is to hoping this won?t affect PETSc ./configure with previous gcc/gfortran version (unlikely, this option is apparently 13-year old https://gcc.gnu.org/bugzilla/show_bug.cgi?id=29471) Let me know of the next hiccup, if any. Thanks, Pierre On 29 Jun 2020, at 8:09 PM, Paolo Lampitella > wrote: Dear Pierre, thanks again for your time I guess there is no way for me to use the toolchain you are using (I don?t remember having any choice on which version of MSYS or GCC I could install) Paolo Inviato da Posta per Windows 10 Da: Pierre Jolivet Inviato: luned? 29 giugno 2020 20:01 A: Matthew Knepley Cc: Paolo Lampitella; petsc-users Oggetto: Re: [petsc-users] PETSc and Windows 10 On 29 Jun 2020, at 7:47 PM, Matthew Knepley > wrote: On Mon, Jun 29, 2020 at 1:35 PM Paolo Lampitella > wrote: Dear Pierre, sorry to bother you, but I already have some issues. What I did: * pacman -R mingw-w64-x86_64-python mingw-w64-x86_64-gdb (is gdb also troublesome?) * Followed points 6 and 7 at https://doc.freefem.org/introduction/installation.html#compilation-on-windows I first got a warning on the configure at point 6, as ?disable-hips is not recognized. Then, on make ?petsc-slepc? of point 7 (no SUDO=sudo flag was necessary) I got to this point: tar xzf ../pkg/petsc-lite-3.13.0.tar.gz patch -p1 < petsc-suitesparse.patch patching file petsc-3.13.0/config/BuildSystem/config/packages/SuiteSparse.py touch petsc-3.13.0/tag-tar cd petsc-3.13.0 && ./configure MAKEFLAGS='' \ --prefix=/home/paolo/freefem/ff-petsc//r \ --with-debugging=0 COPTFLAGS='-O3 -mtune=generic' CXXOPTFLAGS='-O3 -mtune=generic' FOPTFLAGS='-O3 -mtune=generic' --with-cxx-dialect=C++11 --with-ssl=0 --with-x=0 --with-fortran-bindings=0 --with-shared-libraries=0 --with-cc='gcc' --with-cxx='g++' --with-fc='gfortran' CXXFLAGS='-fno-stack-protector' CFLAGS='-fno-stack-protector' --with-scalar-type=real --with-mpi-lib='/c/Windows/System32/msmpi.dll' --with-mpi-include='/home/paolo/FreeFem-sources/3rdparty/include/msmpi' --with-mpiexec='/C/Program\ Files/Microsoft\ MPI/Bin/mpiexec' --with-blaslapack-include='' --with-blaslapack-lib='/mingw64/bin/libopenblas.dll' --download-scalapack --download-metis --download-ptscotch --download-mumps --download-hypre --download-parmetis --download-superlu --download-suitesparse --download-tetgen --download-slepc '--download-metis-cmake-arguments=-G "MSYS Makefiles"' '--download-parmetis-cmake-arguments=-G "MSYS Makefiles"' '--download-superlu-cmake-arguments=-G "MSYS Makefiles"' '--download-hypre-configure-arguments=--build=x86_64-linux-gnu --host=x86_64-linux-gnu' PETSC_ARCH=fr =============================================================================== Configuring PETSc to compile on your system =============================================================================== TESTING: FortranMPICheck from config.packages.MPI(config/BuildSystem/config/pack******************************************************************************* UNABLE to CONFIGURE with GIVEN OPTIONS (see configure.log for details): ------------------------------------------------------------------------------- Fortran error! mpi_init() could not be located! ******************************************************************************* make: *** [Makefile:210: petsc-3.13.0/tag-conf-real] Errore 1 Note that I didn?t add anything to any PATH variable, because this is not mentioned in your documentation. On a side note, this is the same error I got when trying to build PETSc in Cygwin with the default OpenMPI available in Cygwin. I am attaching the configure.log? it seems to me that the error comes from the configure trying to include the mpif.h in your folder and not using the -fallow-invalid-boz flag that I had to use, for example, to compile mpi.f90 into mpi.mod But I?m not sure why this is happening Pierre, Could this be due to gcc 10? Sorry, I?m slow. You are right. Our workers use gcc 9, everything is fine, but I see on my VM which I updated that I use gcc 10 and had to disable Fortran, I guess the MUMPS run I showcased was with a prior PETSc build. I?ll try to resolve this and will keep you posted. They really caught a lot of people off guard with gfortran 10? Thanks, Pierre Executing: gfortran -c -o /tmp/petsc-ur0cff6a/config.libraries/conftest.o -I/tmp/petsc-ur0cff6a/config.compilers -I/tmp/petsc-ur0cff6a/config.setCompilers -I/tmp/petsc-ur0cff6a/config.compilersFortran -I/tmp/petsc-ur0cff6a/config.libraries -Wall -ffree-line-length-0 -Wno-unused-dummy-argument -O3 -mtune=generic -I/home/paolo/FreeFem-sources/3rdparty/include/msmpi /tmp/petsc-ur0cff6a/config.libraries/conftest.F90 Possible ERROR while running compiler: exit code 1 stderr: C:/msys64/home/paolo/FreeFem-sources/3rdparty/include/msmpi/mpif.h:227:36: 227 | PARAMETER (MPI_DATATYPE_NULL=z'0c000000') | 1 Error: BOZ literal constant at (1) is neither a data-stmt-constant nor an actual argument to INT, REAL, DBLE, or CMPLX intrinsic function [see '-fno-allow-invalid-boz'] C:/msys64/home/paolo/FreeFem-sources/3rdparty/include/msmpi/mpif.h:303:27: 303 | PARAMETER (MPI_CHAR=z'4c000101') | 1 Error: BOZ literal constant at (1) is neither a data-stmt-constant nor an actual argument to INT, REAL, DBLE, or CMPLX intrinsic function [see '-fno-allow-invalid-boz'] C:/msys64/home/paolo/FreeFem-sources/3rdparty/include/msmpi/mpif.h:305:36: 305 | PARAMETER (MPI_UNSIGNED_CHAR=z'4c000102') | 1 Thanks, Matt Thanks Paolo Inviato da Posta per Windows 10 Da: Pierre Jolivet Inviato: luned? 29 giugno 2020 18:34 A: Paolo Lampitella Cc: Satish Balay; petsc-users Oggetto: Re: [petsc-users] PETSc and Windows 10 On 29 Jun 2020, at 6:27 PM, Paolo Lampitella > wrote: I think I made the first step of having mingw64 from msys2 working with ms-mpi. I found that the issue I was having was related to: https://gcc.gnu.org/bugzilla/show_bug.cgi?id=91556 and, probably (but impossible to check now), I was using an msys2 and/or mingw mpi package before this fix: https://github.com/msys2/MINGW-packages/commit/11b4cff3d2ec7411037b692b0ad5a9f3e9b9978d#diff-eac59989e3096be97d940c8f47b50fba Admittedly, I never used gcc 10 before on any machine. Still, I feel that reporting that sort of error in that way is, at least, misleading (I would have preferred the initial implementation as mentioned in the gcc bug track). A second thing that I was not used to, and made me more uncertain of the procedure I was following, is having to compile myself the mpi module. There are several version of this out there, but I decided to stick with this one: https://www.scivision.dev/windows-mpi-msys2/ even if there seems to be no need to include -fno-range-check and the current mpi.f90 version is different from the mpif.h as reported here: https://github.com/microsoft/Microsoft-MPI/issues/33 which, to me, are both signs of lack of attention on the fortran side by those that maintain this thing. In summary, this is the procedure I followed so far (on a 64 bit machine with Windows 10): * Install MSYS2 from https://www.msys2.org/ and just follow the install wizard * Open the MSYS2 terminal and execute: pacman -Syuu * Close the terminal when asked and reopen it * Keep executing ?pacman -Syuu? until nothing else needs to be updated * Close the MSYS2 terminal and reopen it (I guess because was in paranoid mode), then install packages with: pacman -S base-devel git gcc gcc-fortran bsdcpio lndir pax-git unzip pacman -S mingw-w64-x86_64-toolchain pacman -S mingw-w64-x86_64-msmpi pacman -S mingw-w64-x86_64-cmake pacman -S mingw-w64-x86_64-freeglut pacman -S mingw-w64-x86_64-gsl pacman -S mingw-w64-x86_64-libmicroutils pacman -S mingw-w64-x86_64-hdf5 pacman -S mingw-w64-x86_64-openblas pacman -S mingw-w64-x86_64-arpack pacman -S mingw-w64-x86_64-jq This set should include all the libraries mentioned by Pierre and/or used by his Jenkins, as the final scope here is to have PETSc and dependencies working. But I think that for pure MPI one could stop to msmpi (even, maybe, just install msmpi and have the dependencies figured out by pacman). Honestly, I don?t remember the exact order I used to install the packages, but this should not affect things. Also, as I was still in paranoid mode, I kept executing ?pacman -Syuu? after each package was installed. After this, close the MSYS2 terminal. * Open the MINGW64 terminal and create the .mod file out of the mpi.f90 file, as mentioned here https://www.scivision.dev/windows-mpi-msys2/, with: cd /mingw64/include gfortran mpif90 -c -fno-range-check -fallow-invalid-boz Ah, yes, that?s new to gfortran 10 (we use gfortran 9 on our workers), which is now what?s ship with MSYS2 (we haven?t updated yet). Sorry that I forgot about that. This is needed to ?USE mpi? (as opposed to INCLUDE ?mpif.h?) * Install the latest MS-MPI (both sdk and setup) from https://www.microsoft.com/en-us/download/details.aspx?id=100593 At this point I?ve been able to compile (using the MINGW64 terminal) different mpi test programs and they run as expected in the classical Windows prompt. I added this function to my .bashrc in MSYS2 in order to easily copy the required dependencies out of MSYS: function copydep() { ldd $1 | grep "=> /$2" | awk '{print $3}' | xargs -I '{}' cp -v '{}' .; } which can be used, with the MINGW64 terminal, by navigating to the folder where the final executable, say, my.exe, resides (even if under a Windows path) and executing: copydep my.exe mingw64 This, of course, must be done before actually trying to execute the .exe in the windows cmd prompt. Hopefully, I should now be able to follow Pierre?s instructions for PETSc (but first I wanna give a try to the system python before removing it) Looks like the hard part is over. It?s usually easier to deal with ./configure issues. If you have weird errors like ?incomplete Cygwin install? or whatever, this is the kind of issues I was referring to earlier. In that case, what I?d suggest is just, as before: pacman -R mingw-w64-x86_64-python mingw-w64-x86_64-gdb pacman -S python Thanks, Pierre Thanks Paolo -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From balay at mcs.anl.gov Sun Jul 5 16:50:17 2020 From: balay at mcs.anl.gov (Satish Balay) Date: Sun, 5 Jul 2020 16:50:17 -0500 (CDT) Subject: [petsc-users] PETSc and Windows 10 In-Reply-To: References: <32B37EFD-DDFA-41AE-ACC7-CB5E6AE57996@enseeiht.fr> <0558CCF7-8203-4520-8673-3B0945C43233@enseeiht.fr> <20AB9F7A-4471-498B-9A78-AA48163900CB@enseeiht.fr> <94A8E366-2361-409E-84CE-B90EA54D9AF9@enseeiht.fr> <849E7E3B-8B54-4B40-A2B7-8DD5B55428B3@enseeiht.fr> , <246CA2C6-E66F-4EF0-8600-81AB741D17D8@enseeiht.fr>, Message-ID: Sounds like there are different mingw tools and msys2 tools. So I guess one could use mingw compilers even from cygwin [using cygwin tools] - i.e mingw compilers don't really need msys2 tools to work. Satish On Sun, 5 Jul 2020, Paolo Lampitella wrote: > Unfortunately, even PETSC_ARCH=i didn't work out. And while with-single-library=0 wasn't really appealing to me, it worked but only to later fail on make test. > > I guess all these differences are due to the fortran bindings and/or gcc 10. > > However, until I discover how they are different, I guess I'll be fine with /usr/bin/ar > > Paolo > > > > Inviato da smartphone Samsung Galaxy. > > > > -------- Messaggio originale -------- > Da: Paolo Lampitella > Data: 05/07/20 14:00 (GMT+01:00) > A: Pierre Jolivet > Cc: Matthew Knepley , petsc-users > Oggetto: RE: [petsc-users] PETSc and Windows 10 > > Thank you very much Pierre. > > I'll keep you informed in case I see any relevant change from the tests when using your suggestion. > > Paolo > > > > Inviato da smartphone Samsung Galaxy. > > > > -------- Messaggio originale -------- > Da: Pierre Jolivet > Data: 05/07/20 13:45 (GMT+01:00) > A: Paolo Lampitella > Cc: Matthew Knepley , petsc-users > Oggetto: Re: [petsc-users] PETSc and Windows 10 > > Hello Paolo, > > On 5 Jul 2020, at 1:15 PM, Paolo Lampitella > wrote: > > Dear all, > > I just want to update you on my journey to PETSc compilation in Windows under MSYS2+MINGW64 > > Unfortunately, I haven?t been able to compile petsc-slepc trough Freefem but, as my final goal required also Fortran bindings (but I only needed blas, lapack, metis and hypre), I decided to follow my own route using the useful information from Pierre. > > > * I started by installing MPI from https://www.microsoft.com/en-us/download/details.aspx?id=100593. I don?t think the SDK is actually needed in my specific workflow, but I installed it as well together with mpisetup. > * Then I installed MSYS2 just following the wizard. Opened the MSYS2 terminal and updated with pacman -Syuu, closed if asked, reopened it and used again pacman -Syuu several times until no more updates were available. Closed it and opened it back. > * Under the MSYS2 terminal installed just the following packages: > > > > * pacman -S base-devel git gcc gcc-fortran > * pacman -S mingw-w64-x86_64-toolchain > * pacman -S mingw-w64-x86_64-cmake > * pacman -S mingw-w64-x86_64-msmpi > > > > * Closed the MSYS2 terminal and opened the MINGW64 one, went to /mingw64/include and compiled my mpi module following https://www.scivision.dev/windows-mpi-msys2/: > > > > * gfortran -c mpi.f90 -fno-range-check -fallow-invalid-boz > > > However, I will keep an eye on the MS-MPI GitHub repository because the fortran side seems to be far from perfect. > > > * Then I downloaded the 3.13.3 version of petsc and configured it, still under the MINGW64 terminal, with the following command: > > > /usr/bin/python ./configure --prefix=/home/paolo/petsc --with-ar=/usr/bin/ar > --with-shared-libraries=0 --with-debugging=0 --with-windows-graphics=0 --with-x=0 > COPTFLAGS="-O3 -mtune=native" > CXXOPTFLAGS="-O3 -mtune=native" > FOPTFLAGS="-O3 -mtune=native" > FFLAGS=-fallow-invalid-boz > --with-mpiexec="/C/Program\ Files/Microsoft\ MPI/Bin/mpiexec" > --download-fblaslapack --download-metis --download-hypre > --download-metis-cmake-arguments='-G "MSYS Makefiles"' > --download-hypre-configure-arguments="--build=x86_64-linux-gnu --host=x86_64-linux-gnu" > > Note that I just bypassed uninstalling python in mingw64 (which doesn?t work) by using /usr/bin/python and that, as opposed to Pierre, I needed to also use the MSYS2 archiver (/usr/bin/ar) as opposed to the mingw64 one (/mingw64/bin/ar that shows up in the Pierre configure) as also mentioned here http://hillyuan.blogspot.com/2017/11/build-petsc-in-windows-under-mingw64.html, probably because of this issue https://stackoverflow.com/questions/37504625/ar-on-msys2-shell-receives-truncated-paths-when-called-from-makefile. > > You are right that you can avoid deinstalling mingw-w64-x86_64-python if you can supply the proper Python yourself (we don?t have that luxury in our Makefile). > If you want to avoid using that AR, and stick to /mingw64/bin/ar (not sure what the pros and cons are), you can either: > - use another PETSC_ARCH (very short, like pw, for petsc-windows); > - use --with-single-library=0. > See this post on GitLab https://gitlab.com/petsc/petsc/-/issues/647#note_373507681 > The OS I?m referring to is indeed my Windows + MSYS2 box. > > Thanks, > Pierre > > Then make all, make install and make check all went smooth. Also, I don?t know exactly what with-x=0 and with-windows-graphics=0 do, but I think it is stuff that I don?t need (yet configure worked with windows-graphics as well). > > > * Finally I launched make test. As some tests failed, I replicated the same install procedure on all the systems I have available on this same Windows machine (Ubuntu 20.04 and Centos 8 under a VirtualBox 6.0.22 VM, Ubuntu 20.04 under WSL1 and the MSYS2-MINGW64 toolchain). I am attaching a file with the results printed to screen (not sure about which file should be used for a comparison/check). Note, however, that the tests in MSYS2 started with some cyclic reference issues for some .mod files, but this doesn?t show up in any file I could check. > > > I am still left with some doubts about the archiver, the cyclic reference errors and the differences in the test results, but I am able to link my code with petsc. Unfortunately, as this Windows porting is part of a large code restructuring, I can?t do much more with it, now, from my code. But if you can suggest some specific tutorial to use as test also for the parallel, I would be glad to dig deeper into the matter. > > Best regards > > Paolo > > Inviato da Posta per Windows 10 > > Da: Pierre Jolivet > Inviato: marted? 30 giugno 2020 15:22 > A: Paolo Lampitella > Cc: Matthew Knepley; petsc-users > Oggetto: Re: [petsc-users] PETSc and Windows 10 > > Please use the 3.13.2 tarball, this was fixed by Satish in the previous commit I already linked (https://gitlab.com/petsc/petsc/-/commit/2cd8068296b34e127f055bb32f556e3599f17523). > (If you want FreeFEM to do the dirty work for you, just switch to the develop branch, and redo ?make petsc-slepc?) > But I think you?ve got everything you need now for a smooth compilation :) > > Thanks, > Pierre > > > On 30 Jun 2020, at 3:09 PM, Paolo Lampitella > wrote: > > Dear Pierre, > > thanks for the fast response. Unfortunately it still fails, but now in the configure of ScaLAPACK > (which means that it went ok for slepc, tetgen, metis, parmetis, ptscotch, superlu and suitesparse). > > The way I applied the modification is by manually editing the Makefile in the 3rdparty/ff-petsc folder, adding -fallow-invalid-boz to both CFLAGS and FFLAGS (this entry added by me). Then executed make petsc-slepc. > > As my project is much less ambitious, I have a good feeling that I will be able to use your Makefile successfully, but as I am kind of slow I tought that it would have been useful for you to know. The configure.log is attached. This time the error is: > > Rank mismatch between actual argument at (1) and actual argument at (2) (scalar and rank-1) > > in subroutine pclarf.f of ScaLAPACK. > > However, before attampting with my project, I have few questions about your Makefile, in particular this piece: > > --with-mpi-lib=/c/Windows/System32/msmpi.dll --with-mpi-include=/home/paolo/FreeFem-sources/3rdparty/include/msmpi --with-mpiexec="/C/Program\ Files/Microsoft\ MPI/Bin/mpiexec" > > I see from MPI.py that I should not use ?--with-mpi-lib/include? if I want to use my now working mpi wrappers. Is this correct? > > Paolo > > Inviato da Posta per Windows 10 > > Da: Pierre Jolivet > Inviato: luned? 29 giugno 2020 21:37 > A: Paolo Lampitella > Cc: Matthew Knepley; petsc-users > Oggetto: Re: [petsc-users] PETSc and Windows 10 > > I do not give up easily on Windows problems: > 1) that?s around 50% of our (FreeFEM) user-base (and I want them to use PETSc and SLEPc, ofc?) > 2) most people I work with from corporations just have Windows laptops/desktops and I always recommend MSYS because it?s very lightweight and you can pass .exe around > 3) I?ve bothered enough Satish, Jed, and Matt on GitLab to take (at least partially) the blame now when it doesn?t work on MSYS > > That being said, the magic keyword is the added flag FFLAGS="-fallow-invalid-boz" (see, I told you ./configure issues were easier to deal with than the others). > Here you?ll see that everything goes through just fine (sorry, it took me a long time to post this because everything is slow on my VM): > 1) http://jolivet.perso.enseeiht.fr/win10/configure.log > 2) http://jolivet.perso.enseeiht.fr/win10/make.log (both steps #1 and #2 in MSYS terminal, gcc/gfortran 10, MS-MPI see screenshot) > 3) http://jolivet.perso.enseeiht.fr/win10/ex2.txt (Command Prompt, 4 processes + MUMPS, I can send you the .exe if you want to try on your machine) > I just realize that I didn?t generate the Fortran bindings, but you can see I compiled MUMPS and ScaLAPACK, so that shouldn?t be a problem. > Or if there is a problem, we will need to fix this in PETSc. > > I?ll push this added flag to the FreeFEM repo, thanks for reminding me of the brokenness of gcc/gfortran 10 + MS-MPI. > Here is to hoping this won?t affect PETSc ./configure with previous gcc/gfortran version (unlikely, this option is apparently 13-year old https://gcc.gnu.org/bugzilla/show_bug.cgi?id=29471) > > Let me know of the next hiccup, if any. > Thanks, > Pierre > > > > On 29 Jun 2020, at 8:09 PM, Paolo Lampitella > wrote: > > Dear Pierre, > > thanks again for your time > > I guess there is no way for me to use the toolchain you are using (I don?t remember having any choice on which version of MSYS or GCC I could install) > > Paolo > > Inviato da Posta per Windows 10 > > Da: Pierre Jolivet > Inviato: luned? 29 giugno 2020 20:01 > A: Matthew Knepley > Cc: Paolo Lampitella; petsc-users > Oggetto: Re: [petsc-users] PETSc and Windows 10 > > > > > > > On 29 Jun 2020, at 7:47 PM, Matthew Knepley > wrote: > > On Mon, Jun 29, 2020 at 1:35 PM Paolo Lampitella > wrote: > Dear Pierre, sorry to bother you, but I already have some issues. What I did: > > > * pacman -R mingw-w64-x86_64-python mingw-w64-x86_64-gdb (is gdb also troublesome?) > * Followed points 6 and 7 at https://doc.freefem.org/introduction/installation.html#compilation-on-windows > > I first got a warning on the configure at point 6, as ?disable-hips is not recognized. Then, on make ?petsc-slepc? of point 7 (no SUDO=sudo flag was necessary) I got to this point: > > tar xzf ../pkg/petsc-lite-3.13.0.tar.gz > patch -p1 < petsc-suitesparse.patch > patching file petsc-3.13.0/config/BuildSystem/config/packages/SuiteSparse.py > touch petsc-3.13.0/tag-tar > cd petsc-3.13.0 && ./configure MAKEFLAGS='' \ > --prefix=/home/paolo/freefem/ff-petsc//r \ > --with-debugging=0 COPTFLAGS='-O3 -mtune=generic' CXXOPTFLAGS='-O3 -mtune=generic' FOPTFLAGS='-O3 -mtune=generic' --with-cxx-dialect=C++11 --with-ssl=0 --with-x=0 --with-fortran-bindings=0 --with-shared-libraries=0 --with-cc='gcc' --with-cxx='g++' --with-fc='gfortran' CXXFLAGS='-fno-stack-protector' CFLAGS='-fno-stack-protector' --with-scalar-type=real --with-mpi-lib='/c/Windows/System32/msmpi.dll' --with-mpi-include='/home/paolo/FreeFem-sources/3rdparty/include/msmpi' --with-mpiexec='/C/Program\ Files/Microsoft\ MPI/Bin/mpiexec' --with-blaslapack-include='' --with-blaslapack-lib='/mingw64/bin/libopenblas.dll' --download-scalapack --download-metis --download-ptscotch --download-mumps --download-hypre --download-parmetis --download-superlu --download-suitesparse --download-tetgen --download-slepc '--download-metis-cmake-arguments=-G "MSYS Makefiles"' '--download-parmetis-cmake-arguments=-G "MSYS Makefiles"' '--download-superlu-cmake-arguments=-G "MSYS Makefiles"' '-- download-hypre-configure-arguments=--build=x86_64-linux-gnu --host=x86_64-linux-gnu' PETSC_ARCH=fr > =============================================================================== > Configuring PETSc to compile on your system > =============================================================================== > TESTING: FortranMPICheck from config.packages.MPI(config/BuildSystem/config/pack******************************************************************************* > UNABLE to CONFIGURE with GIVEN OPTIONS (see configure.log for details): > ------------------------------------------------------------------------------- > Fortran error! mpi_init() could not be located! > ******************************************************************************* > > make: *** [Makefile:210: petsc-3.13.0/tag-conf-real] Errore 1 > > Note that I didn?t add anything to any PATH variable, because this is not mentioned in your documentation. > > On a side note, this is the same error I got when trying to build PETSc in Cygwin with the default OpenMPI available in Cygwin. > > I am attaching the configure.log? it seems to me that the error comes from the configure trying to include the mpif.h in your folder and not using the -fallow-invalid-boz flag that I had to use, for example, to compile mpi.f90 into mpi.mod > > But I?m not sure why this is happening > > Pierre, > > Could this be due to gcc 10? > > Sorry, I?m slow. You are right. Our workers use gcc 9, everything is fine, but I see on my VM which I updated that I use gcc 10 and had to disable Fortran, I guess the MUMPS run I showcased was with a prior PETSc build. > I?ll try to resolve this and will keep you posted. > They really caught a lot of people off guard with gfortran 10? > > Thanks, > Pierre > > > > > Executing: gfortran -c -o /tmp/petsc-ur0cff6a/config.libraries/conftest.o -I/tmp/petsc-ur0cff6a/config.compilers -I/tmp/petsc-ur0cff6a/config.setCompilers -I/tmp/petsc-ur0cff6a/config.compilersFortran -I/tmp/petsc-ur0cff6a/config.libraries -Wall -ffree-line-length-0 -Wno-unused-dummy-argument -O3 -mtune=generic -I/home/paolo/FreeFem-sources/3rdparty/include/msmpi /tmp/petsc-ur0cff6a/config.libraries/conftest.F90 > Possible ERROR while running compiler: exit code 1 > stderr: > C:/msys64/home/paolo/FreeFem-sources/3rdparty/include/msmpi/mpif.h:227:36: > > 227 | PARAMETER (MPI_DATATYPE_NULL=z'0c000000') > | 1 > Error: BOZ literal constant at (1) is neither a data-stmt-constant nor an actual argument to INT, REAL, DBLE, or CMPLX intrinsic function [see '-fno-allow-invalid-boz'] > C:/msys64/home/paolo/FreeFem-sources/3rdparty/include/msmpi/mpif.h:303:27: > > 303 | PARAMETER (MPI_CHAR=z'4c000101') > | 1 > Error: BOZ literal constant at (1) is neither a data-stmt-constant nor an actual argument to INT, REAL, DBLE, or CMPLX intrinsic function [see '-fno-allow-invalid-boz'] > C:/msys64/home/paolo/FreeFem-sources/3rdparty/include/msmpi/mpif.h:305:36: > > 305 | PARAMETER (MPI_UNSIGNED_CHAR=z'4c000102') > | 1 > > Thanks, > > Matt > > Thanks > > Paolo > > Inviato da Posta per Windows 10 > > Da: Pierre Jolivet > Inviato: luned? 29 giugno 2020 18:34 > A: Paolo Lampitella > Cc: Satish Balay; petsc-users > Oggetto: Re: [petsc-users] PETSc and Windows 10 > > > > On 29 Jun 2020, at 6:27 PM, Paolo Lampitella > wrote: > > I think I made the first step of having mingw64 from msys2 working with ms-mpi. > > I found that the issue I was having was related to: > > https://gcc.gnu.org/bugzilla/show_bug.cgi?id=91556 > > and, probably (but impossible to check now), I was using an msys2 and/or mingw mpi package before this fix: > > https://github.com/msys2/MINGW-packages/commit/11b4cff3d2ec7411037b692b0ad5a9f3e9b9978d#diff-eac59989e3096be97d940c8f47b50fba > > Admittedly, I never used gcc 10 before on any machine. Still, I feel that reporting that sort of error in that way is, > at least, misleading (I would have preferred the initial implementation as mentioned in the gcc bug track). > > A second thing that I was not used to, and made me more uncertain of the procedure I was following, is having to compile myself the mpi module. There are several version of this out there, but I decided to stick with this one: > > https://www.scivision.dev/windows-mpi-msys2/ > > even if there seems to be no need to include -fno-range-check and the current mpi.f90 version is different from the mpif.h as reported here: > > https://github.com/microsoft/Microsoft-MPI/issues/33 > > which, to me, are both signs of lack of attention on the fortran side by those that maintain this thing. > > In summary, this is the procedure I followed so far (on a 64 bit machine with Windows 10): > > > * Install MSYS2 from https://www.msys2.org/ and just follow the install wizard > * Open the MSYS2 terminal and execute: pacman -Syuu > * Close the terminal when asked and reopen it > * Keep executing ?pacman -Syuu? until nothing else needs to be updated > * Close the MSYS2 terminal and reopen it (I guess because was in paranoid mode), then install packages with: > > > pacman -S base-devel git gcc gcc-fortran bsdcpio lndir pax-git unzip > pacman -S mingw-w64-x86_64-toolchain > pacman -S mingw-w64-x86_64-msmpi > pacman -S mingw-w64-x86_64-cmake > pacman -S mingw-w64-x86_64-freeglut > pacman -S mingw-w64-x86_64-gsl > pacman -S mingw-w64-x86_64-libmicroutils > pacman -S mingw-w64-x86_64-hdf5 > pacman -S mingw-w64-x86_64-openblas > pacman -S mingw-w64-x86_64-arpack > pacman -S mingw-w64-x86_64-jq > > This set should include all the libraries mentioned by Pierre and/or used by his Jenkins, as the final scope here is to have PETSc and dependencies working. But I think that for pure MPI one could stop to msmpi (even, maybe, just install msmpi and have the dependencies figured out by pacman). Honestly, I don?t remember the exact order I used to install the packages, but this should not affect things. Also, as I was still in paranoid mode, I kept executing ?pacman -Syuu? after each package was installed. After this, close the MSYS2 terminal. > > > * Open the MINGW64 terminal and create the .mod file out of the mpi.f90 file, as mentioned here https://www.scivision.dev/windows-mpi-msys2/, with: > > > cd /mingw64/include > gfortran mpif90 -c -fno-range-check -fallow-invalid-boz > > Ah, yes, that?s new to gfortran 10 (we use gfortran 9 on our workers), which is now what?s ship with MSYS2 (we haven?t updated yet). Sorry that I forgot about that. > > This is needed to ?USE mpi? (as opposed to INCLUDE ?mpif.h?) > > > * Install the latest MS-MPI (both sdk and setup) from https://www.microsoft.com/en-us/download/details.aspx?id=100593 > > > At this point I?ve been able to compile (using the MINGW64 terminal) different mpi test programs and they run as expected in the classical Windows prompt. I added this function to my .bashrc in MSYS2 in order to easily copy the required dependencies out of MSYS: > > function copydep() { ldd $1 | grep "=> /$2" | awk '{print $3}' | xargs -I '{}' cp -v '{}' .; } > > which can be used, with the MINGW64 terminal, by navigating to the folder where the final executable, say, my.exe, resides (even if under a Windows path) and executing: > > copydep my.exe mingw64 > > This, of course, must be done before actually trying to execute the .exe in the windows cmd prompt. > > Hopefully, I should now be able to follow Pierre?s instructions for PETSc (but first I wanna give a try to the system python before removing it) > > Looks like the hard part is over. It?s usually easier to deal with ./configure issues. > If you have weird errors like ?incomplete Cygwin install? or whatever, this is the kind of issues I was referring to earlier. > In that case, what I?d suggest is just, as before: > pacman -R mingw-w64-x86_64-python mingw-w64-x86_64-gdb > pacman -S python > > Thanks, > Pierre > > Thanks > > Paolo > > > > > -- > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ > > > > > > > > From eda.oktay at metu.edu.tr Mon Jul 6 04:06:55 2020 From: eda.oktay at metu.edu.tr (Eda Oktay) Date: Mon, 6 Jul 2020 12:06:55 +0300 Subject: [petsc-users] EPSGetEigenpair Error Message-ID: Hello, I am using EPSGetEigenpair to find some matrices' eigenpairs: ierr = EPSGetType(eps,&type);CHKERRQ(ierr) ierr = EPSGetDimensions(eps,&nev,NULL,NULL);CHKERRQ(ierr); ierr = MatCreateVecs(L,&vr,NULL);CHKERRQ(ierr); Vec *V; VecDuplicateVecs(vr,nev,&V); for (i=0; i References: Message-ID: The index i should be a value between 0 and nconv-1 https://slepc.upv.es/documentation/current/docs/manualpages/EPS/EPSGetEigenpair.html Jose > El 6 jul 2020, a las 11:06, Eda Oktay escribi?: > > Hello, > > I am using EPSGetEigenpair to find some matrices' eigenpairs: > > ierr = EPSGetType(eps,&type);CHKERRQ(ierr) > ierr = EPSGetDimensions(eps,&nev,NULL,NULL);CHKERRQ(ierr); > ierr = MatCreateVecs(L,&vr,NULL);CHKERRQ(ierr); > Vec *V; > VecDuplicateVecs(vr,nev,&V); > for (i=0; i ierr = EPSGetEigenpair(eps,i,&kr,NULL,V[i],NULL); > } > > However, for some matrices, even though nev = 2, I am getting the > following error: > > [0]PETSC ERROR: --------------------- Error Message > -------------------------------------------------------------- > [0]PETSC ERROR: Argument out of range > [0]PETSC ERROR: Argument 2 out of range > [0]PETSC ERROR: See > https://www.mcs.anl.gov/petsc/documentation/faq.html for trouble > shooting. > [0]PETSC ERROR: Petsc Release Version 3.13.2, Jun 02, 2020 > [0]PETSC ERROR: ./deneme_new_vecscatter_arastep on a > arch-linux2-c-debug named b342.wls.metu.edu.tr by edaoktay Mon Jul 6 > 12:01:14 2020 > [0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ > --download-mpich --download-openblas --download-slepc --download-metis > --download-parmetis --download-chaco --with-X=1 > [0]PETSC ERROR: #1 EPSGetEigenpair() line 411 in > /home/edaoktay/petsc-3.13.2/arch-linux2-c-debug/externalpackages/git.slepc/src/eps/interface/epssolve.c > > =================================================================================== > = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES > = PID 8635 RUNNING AT b342.wls.metu.edu.tr > = EXIT CODE: 134 > = CLEANING UP REMAINING PROCESSES > = YOU CAN IGNORE THE BELOW CLEANUP MESSAGES > =================================================================================== > YOUR APPLICATION TERMINATED WITH THE EXIT STRING: Aborted (signal 6) > This typically refers to a problem with your application. > Please see the FAQ page for debugging suggestions > > What can be the reason? I am getting this message when i = 0, so how > can 0 be out of range for nev = 2? > > And why am I getting this message for 2 matrices out of 16? > > Thanks! From paololampitella at hotmail.com Mon Jul 6 04:18:54 2020 From: paololampitella at hotmail.com (Paolo Lampitella) Date: Mon, 6 Jul 2020 09:18:54 +0000 Subject: [petsc-users] R: PETSc and Windows 10 In-Reply-To: References: <32B37EFD-DDFA-41AE-ACC7-CB5E6AE57996@enseeiht.fr> <0558CCF7-8203-4520-8673-3B0945C43233@enseeiht.fr> <20AB9F7A-4471-498B-9A78-AA48163900CB@enseeiht.fr> <94A8E366-2361-409E-84CE-B90EA54D9AF9@enseeiht.fr> <849E7E3B-8B54-4B40-A2B7-8DD5B55428B3@enseeiht.fr> , <246CA2C6-E66F-4EF0-8600-81AB741D17D8@enseeiht.fr>, , Message-ID: Dear Satish, Yes indeed, or at least that is my understanding. Still, my experience so far with Cygwin has been, let?s say, controversial. I wasn?t able to compile myself MPICH, with both gcc and mingw. When having PETSc compile also MPICH, I was successful only with gcc but not mingw. I didn?t even try compiling OpenMPI with mingw, as PETSc compilation already failed using the OpenMPI available trough cygwin libraries (which is based on gcc and not mingw). Not sure if this is my fault, but in the end it didn?t go well Paolo Inviato da Posta per Windows 10 Da: Satish Balay Inviato: domenica 5 luglio 2020 23:50 A: Paolo Lampitella Cc: Pierre Jolivet; petsc-users Oggetto: Re: [petsc-users] PETSc and Windows 10 Sounds like there are different mingw tools and msys2 tools. So I guess one could use mingw compilers even from cygwin [using cygwin tools] - i.e mingw compilers don't really need msys2 tools to work. Satish On Sun, 5 Jul 2020, Paolo Lampitella wrote: > Unfortunately, even PETSC_ARCH=i didn't work out. And while with-single-library=0 wasn't really appealing to me, it worked but only to later fail on make test. > > I guess all these differences are due to the fortran bindings and/or gcc 10. > > However, until I discover how they are different, I guess I'll be fine with /usr/bin/ar > > Paolo > > > > Inviato da smartphone Samsung Galaxy. > > > > -------- Messaggio originale -------- > Da: Paolo Lampitella > Data: 05/07/20 14:00 (GMT+01:00) > A: Pierre Jolivet > Cc: Matthew Knepley , petsc-users > Oggetto: RE: [petsc-users] PETSc and Windows 10 > > Thank you very much Pierre. > > I'll keep you informed in case I see any relevant change from the tests when using your suggestion. > > Paolo > > > > Inviato da smartphone Samsung Galaxy. > > > > -------- Messaggio originale -------- > Da: Pierre Jolivet > Data: 05/07/20 13:45 (GMT+01:00) > A: Paolo Lampitella > Cc: Matthew Knepley , petsc-users > Oggetto: Re: [petsc-users] PETSc and Windows 10 > > Hello Paolo, > > On 5 Jul 2020, at 1:15 PM, Paolo Lampitella > wrote: > > Dear all, > > I just want to update you on my journey to PETSc compilation in Windows under MSYS2+MINGW64 > > Unfortunately, I haven?t been able to compile petsc-slepc trough Freefem but, as my final goal required also Fortran bindings (but I only needed blas, lapack, metis and hypre), I decided to follow my own route using the useful information from Pierre. > > > * I started by installing MPI from https://www.microsoft.com/en-us/download/details.aspx?id=100593. I don?t think the SDK is actually needed in my specific workflow, but I installed it as well together with mpisetup. > * Then I installed MSYS2 just following the wizard. Opened the MSYS2 terminal and updated with pacman -Syuu, closed if asked, reopened it and used again pacman -Syuu several times until no more updates were available. Closed it and opened it back. > * Under the MSYS2 terminal installed just the following packages: > > > > * pacman -S base-devel git gcc gcc-fortran > * pacman -S mingw-w64-x86_64-toolchain > * pacman -S mingw-w64-x86_64-cmake > * pacman -S mingw-w64-x86_64-msmpi > > > > * Closed the MSYS2 terminal and opened the MINGW64 one, went to /mingw64/include and compiled my mpi module following https://www.scivision.dev/windows-mpi-msys2/: > > > > * gfortran -c mpi.f90 -fno-range-check -fallow-invalid-boz > > > However, I will keep an eye on the MS-MPI GitHub repository because the fortran side seems to be far from perfect. > > > * Then I downloaded the 3.13.3 version of petsc and configured it, still under the MINGW64 terminal, with the following command: > > > /usr/bin/python ./configure --prefix=/home/paolo/petsc --with-ar=/usr/bin/ar > --with-shared-libraries=0 --with-debugging=0 --with-windows-graphics=0 --with-x=0 > COPTFLAGS="-O3 -mtune=native" > CXXOPTFLAGS="-O3 -mtune=native" > FOPTFLAGS="-O3 -mtune=native" > FFLAGS=-fallow-invalid-boz > --with-mpiexec="/C/Program\ Files/Microsoft\ MPI/Bin/mpiexec" > --download-fblaslapack --download-metis --download-hypre > --download-metis-cmake-arguments='-G "MSYS Makefiles"' > --download-hypre-configure-arguments="--build=x86_64-linux-gnu --host=x86_64-linux-gnu" > > Note that I just bypassed uninstalling python in mingw64 (which doesn?t work) by using /usr/bin/python and that, as opposed to Pierre, I needed to also use the MSYS2 archiver (/usr/bin/ar) as opposed to the mingw64 one (/mingw64/bin/ar that shows up in the Pierre configure) as also mentioned here http://hillyuan.blogspot.com/2017/11/build-petsc-in-windows-under-mingw64.html, probably because of this issue https://stackoverflow.com/questions/37504625/ar-on-msys2-shell-receives-truncated-paths-when-called-from-makefile. > > You are right that you can avoid deinstalling mingw-w64-x86_64-python if you can supply the proper Python yourself (we don?t have that luxury in our Makefile). > If you want to avoid using that AR, and stick to /mingw64/bin/ar (not sure what the pros and cons are), you can either: > - use another PETSC_ARCH (very short, like pw, for petsc-windows); > - use --with-single-library=0. > See this post on GitLab https://gitlab.com/petsc/petsc/-/issues/647#note_373507681 > The OS I?m referring to is indeed my Windows + MSYS2 box. > > Thanks, > Pierre > > Then make all, make install and make check all went smooth. Also, I don?t know exactly what with-x=0 and with-windows-graphics=0 do, but I think it is stuff that I don?t need (yet configure worked with windows-graphics as well). > > > * Finally I launched make test. As some tests failed, I replicated the same install procedure on all the systems I have available on this same Windows machine (Ubuntu 20.04 and Centos 8 under a VirtualBox 6.0.22 VM, Ubuntu 20.04 under WSL1 and the MSYS2-MINGW64 toolchain). I am attaching a file with the results printed to screen (not sure about which file should be used for a comparison/check). Note, however, that the tests in MSYS2 started with some cyclic reference issues for some .mod files, but this doesn?t show up in any file I could check. > > > I am still left with some doubts about the archiver, the cyclic reference errors and the differences in the test results, but I am able to link my code with petsc. Unfortunately, as this Windows porting is part of a large code restructuring, I can?t do much more with it, now, from my code. But if you can suggest some specific tutorial to use as test also for the parallel, I would be glad to dig deeper into the matter. > > Best regards > > Paolo > > Inviato da Posta per Windows 10 > > Da: Pierre Jolivet > Inviato: marted? 30 giugno 2020 15:22 > A: Paolo Lampitella > Cc: Matthew Knepley; petsc-users > Oggetto: Re: [petsc-users] PETSc and Windows 10 > > Please use the 3.13.2 tarball, this was fixed by Satish in the previous commit I already linked (https://gitlab.com/petsc/petsc/-/commit/2cd8068296b34e127f055bb32f556e3599f17523). > (If you want FreeFEM to do the dirty work for you, just switch to the develop branch, and redo ?make petsc-slepc?) > But I think you?ve got everything you need now for a smooth compilation :) > > Thanks, > Pierre > > > On 30 Jun 2020, at 3:09 PM, Paolo Lampitella > wrote: > > Dear Pierre, > > thanks for the fast response. Unfortunately it still fails, but now in the configure of ScaLAPACK > (which means that it went ok for slepc, tetgen, metis, parmetis, ptscotch, superlu and suitesparse). > > The way I applied the modification is by manually editing the Makefile in the 3rdparty/ff-petsc folder, adding -fallow-invalid-boz to both CFLAGS and FFLAGS (this entry added by me). Then executed make petsc-slepc. > > As my project is much less ambitious, I have a good feeling that I will be able to use your Makefile successfully, but as I am kind of slow I tought that it would have been useful for you to know. The configure.log is attached. This time the error is: > > Rank mismatch between actual argument at (1) and actual argument at (2) (scalar and rank-1) > > in subroutine pclarf.f of ScaLAPACK. > > However, before attampting with my project, I have few questions about your Makefile, in particular this piece: > > --with-mpi-lib=/c/Windows/System32/msmpi.dll --with-mpi-include=/home/paolo/FreeFem-sources/3rdparty/include/msmpi --with-mpiexec="/C/Program\ Files/Microsoft\ MPI/Bin/mpiexec" > > I see from MPI.py that I should not use ?--with-mpi-lib/include? if I want to use my now working mpi wrappers. Is this correct? > > Paolo > > Inviato da Posta per Windows 10 > > Da: Pierre Jolivet > Inviato: luned? 29 giugno 2020 21:37 > A: Paolo Lampitella > Cc: Matthew Knepley; petsc-users > Oggetto: Re: [petsc-users] PETSc and Windows 10 > > I do not give up easily on Windows problems: > 1) that?s around 50% of our (FreeFEM) user-base (and I want them to use PETSc and SLEPc, ofc?) > 2) most people I work with from corporations just have Windows laptops/desktops and I always recommend MSYS because it?s very lightweight and you can pass .exe around > 3) I?ve bothered enough Satish, Jed, and Matt on GitLab to take (at least partially) the blame now when it doesn?t work on MSYS > > That being said, the magic keyword is the added flag FFLAGS="-fallow-invalid-boz" (see, I told you ./configure issues were easier to deal with than the others). > Here you?ll see that everything goes through just fine (sorry, it took me a long time to post this because everything is slow on my VM): > 1) http://jolivet.perso.enseeiht.fr/win10/configure.log > 2) http://jolivet.perso.enseeiht.fr/win10/make.log (both steps #1 and #2 in MSYS terminal, gcc/gfortran 10, MS-MPI see screenshot) > 3) http://jolivet.perso.enseeiht.fr/win10/ex2.txt (Command Prompt, 4 processes + MUMPS, I can send you the .exe if you want to try on your machine) > I just realize that I didn?t generate the Fortran bindings, but you can see I compiled MUMPS and ScaLAPACK, so that shouldn?t be a problem. > Or if there is a problem, we will need to fix this in PETSc. > > I?ll push this added flag to the FreeFEM repo, thanks for reminding me of the brokenness of gcc/gfortran 10 + MS-MPI. > Here is to hoping this won?t affect PETSc ./configure with previous gcc/gfortran version (unlikely, this option is apparently 13-year old https://gcc.gnu.org/bugzilla/show_bug.cgi?id=29471) > > Let me know of the next hiccup, if any. > Thanks, > Pierre > > > > On 29 Jun 2020, at 8:09 PM, Paolo Lampitella > wrote: > > Dear Pierre, > > thanks again for your time > > I guess there is no way for me to use the toolchain you are using (I don?t remember having any choice on which version of MSYS or GCC I could install) > > Paolo > > Inviato da Posta per Windows 10 > > Da: Pierre Jolivet > Inviato: luned? 29 giugno 2020 20:01 > A: Matthew Knepley > Cc: Paolo Lampitella; petsc-users > Oggetto: Re: [petsc-users] PETSc and Windows 10 > > > > > > > On 29 Jun 2020, at 7:47 PM, Matthew Knepley > wrote: > > On Mon, Jun 29, 2020 at 1:35 PM Paolo Lampitella > wrote: > Dear Pierre, sorry to bother you, but I already have some issues. What I did: > > > * pacman -R mingw-w64-x86_64-python mingw-w64-x86_64-gdb (is gdb also troublesome?) > * Followed points 6 and 7 at https://doc.freefem.org/introduction/installation.html#compilation-on-windows > > I first got a warning on the configure at point 6, as ?disable-hips is not recognized. Then, on make ?petsc-slepc? of point 7 (no SUDO=sudo flag was necessary) I got to this point: > > tar xzf ../pkg/petsc-lite-3.13.0.tar.gz > patch -p1 < petsc-suitesparse.patch > patching file petsc-3.13.0/config/BuildSystem/config/packages/SuiteSparse.py > touch petsc-3.13.0/tag-tar > cd petsc-3.13.0 && ./configure MAKEFLAGS='' \ > --prefix=/home/paolo/freefem/ff-petsc//r \ > --with-debugging=0 COPTFLAGS='-O3 -mtune=generic' CXXOPTFLAGS='-O3 -mtune=generic' FOPTFLAGS='-O3 -mtune=generic' --with-cxx-dialect=C++11 --with-ssl=0 --with-x=0 --with-fortran-bindings=0 --with-shared-libraries=0 --with-cc='gcc' --with-cxx='g++' --with-fc='gfortran' CXXFLAGS='-fno-stack-protector' CFLAGS='-fno-stack-protector' --with-scalar-type=real --with-mpi-lib='/c/Windows/System32/msmpi.dll' --with-mpi-include='/home/paolo/FreeFem-sources/3rdparty/include/msmpi' --with-mpiexec='/C/Program\ Files/Microsoft\ MPI/Bin/mpiexec' --with-blaslapack-include='' --with-blaslapack-lib='/mingw64/bin/libopenblas.dll' --download-scalapack --download-metis --download-ptscotch --download-mumps --download-hypre --download-parmetis --download-superlu --download-suitesparse --download-tetgen --download-slepc '--download-metis-cmake-arguments=-G "MSYS Makefiles"' '--download-parmetis-cmake-arguments=-G "MSYS Makefiles"' '--download-superlu-cmake-arguments=-G "MSYS Makefiles"' '-- download-hypre-configure-arguments=--build=x86_64-linux-gnu --host=x86_64-linux-gnu' PETSC_ARCH=fr > =============================================================================== > Configuring PETSc to compile on your system > =============================================================================== > TESTING: FortranMPICheck from config.packages.MPI(config/BuildSystem/config/pack******************************************************************************* > UNABLE to CONFIGURE with GIVEN OPTIONS (see configure.log for details): > ------------------------------------------------------------------------------- > Fortran error! mpi_init() could not be located! > ******************************************************************************* > > make: *** [Makefile:210: petsc-3.13.0/tag-conf-real] Errore 1 > > Note that I didn?t add anything to any PATH variable, because this is not mentioned in your documentation. > > On a side note, this is the same error I got when trying to build PETSc in Cygwin with the default OpenMPI available in Cygwin. > > I am attaching the configure.log? it seems to me that the error comes from the configure trying to include the mpif.h in your folder and not using the -fallow-invalid-boz flag that I had to use, for example, to compile mpi.f90 into mpi.mod > > But I?m not sure why this is happening > > Pierre, > > Could this be due to gcc 10? > > Sorry, I?m slow. You are right. Our workers use gcc 9, everything is fine, but I see on my VM which I updated that I use gcc 10 and had to disable Fortran, I guess the MUMPS run I showcased was with a prior PETSc build. > I?ll try to resolve this and will keep you posted. > They really caught a lot of people off guard with gfortran 10? > > Thanks, > Pierre > > > > > Executing: gfortran -c -o /tmp/petsc-ur0cff6a/config.libraries/conftest.o -I/tmp/petsc-ur0cff6a/config.compilers -I/tmp/petsc-ur0cff6a/config.setCompilers -I/tmp/petsc-ur0cff6a/config.compilersFortran -I/tmp/petsc-ur0cff6a/config.libraries -Wall -ffree-line-length-0 -Wno-unused-dummy-argument -O3 -mtune=generic -I/home/paolo/FreeFem-sources/3rdparty/include/msmpi /tmp/petsc-ur0cff6a/config.libraries/conftest.F90 > Possible ERROR while running compiler: exit code 1 > stderr: > C:/msys64/home/paolo/FreeFem-sources/3rdparty/include/msmpi/mpif.h:227:36: > > 227 | PARAMETER (MPI_DATATYPE_NULL=z'0c000000') > | 1 > Error: BOZ literal constant at (1) is neither a data-stmt-constant nor an actual argument to INT, REAL, DBLE, or CMPLX intrinsic function [see '-fno-allow-invalid-boz'] > C:/msys64/home/paolo/FreeFem-sources/3rdparty/include/msmpi/mpif.h:303:27: > > 303 | PARAMETER (MPI_CHAR=z'4c000101') > | 1 > Error: BOZ literal constant at (1) is neither a data-stmt-constant nor an actual argument to INT, REAL, DBLE, or CMPLX intrinsic function [see '-fno-allow-invalid-boz'] > C:/msys64/home/paolo/FreeFem-sources/3rdparty/include/msmpi/mpif.h:305:36: > > 305 | PARAMETER (MPI_UNSIGNED_CHAR=z'4c000102') > | 1 > > Thanks, > > Matt > > Thanks > > Paolo > > Inviato da Posta per Windows 10 > > Da: Pierre Jolivet > Inviato: luned? 29 giugno 2020 18:34 > A: Paolo Lampitella > Cc: Satish Balay; petsc-users > Oggetto: Re: [petsc-users] PETSc and Windows 10 > > > > On 29 Jun 2020, at 6:27 PM, Paolo Lampitella > wrote: > > I think I made the first step of having mingw64 from msys2 working with ms-mpi. > > I found that the issue I was having was related to: > > https://gcc.gnu.org/bugzilla/show_bug.cgi?id=91556 > > and, probably (but impossible to check now), I was using an msys2 and/or mingw mpi package before this fix: > > https://github.com/msys2/MINGW-packages/commit/11b4cff3d2ec7411037b692b0ad5a9f3e9b9978d#diff-eac59989e3096be97d940c8f47b50fba > > Admittedly, I never used gcc 10 before on any machine. Still, I feel that reporting that sort of error in that way is, > at least, misleading (I would have preferred the initial implementation as mentioned in the gcc bug track). > > A second thing that I was not used to, and made me more uncertain of the procedure I was following, is having to compile myself the mpi module. There are several version of this out there, but I decided to stick with this one: > > https://www.scivision.dev/windows-mpi-msys2/ > > even if there seems to be no need to include -fno-range-check and the current mpi.f90 version is different from the mpif.h as reported here: > > https://github.com/microsoft/Microsoft-MPI/issues/33 > > which, to me, are both signs of lack of attention on the fortran side by those that maintain this thing. > > In summary, this is the procedure I followed so far (on a 64 bit machine with Windows 10): > > > * Install MSYS2 from https://www.msys2.org/ and just follow the install wizard > * Open the MSYS2 terminal and execute: pacman -Syuu > * Close the terminal when asked and reopen it > * Keep executing ?pacman -Syuu? until nothing else needs to be updated > * Close the MSYS2 terminal and reopen it (I guess because was in paranoid mode), then install packages with: > > > pacman -S base-devel git gcc gcc-fortran bsdcpio lndir pax-git unzip > pacman -S mingw-w64-x86_64-toolchain > pacman -S mingw-w64-x86_64-msmpi > pacman -S mingw-w64-x86_64-cmake > pacman -S mingw-w64-x86_64-freeglut > pacman -S mingw-w64-x86_64-gsl > pacman -S mingw-w64-x86_64-libmicroutils > pacman -S mingw-w64-x86_64-hdf5 > pacman -S mingw-w64-x86_64-openblas > pacman -S mingw-w64-x86_64-arpack > pacman -S mingw-w64-x86_64-jq > > This set should include all the libraries mentioned by Pierre and/or used by his Jenkins, as the final scope here is to have PETSc and dependencies working. But I think that for pure MPI one could stop to msmpi (even, maybe, just install msmpi and have the dependencies figured out by pacman). Honestly, I don?t remember the exact order I used to install the packages, but this should not affect things. Also, as I was still in paranoid mode, I kept executing ?pacman -Syuu? after each package was installed. After this, close the MSYS2 terminal. > > > * Open the MINGW64 terminal and create the .mod file out of the mpi.f90 file, as mentioned here https://www.scivision.dev/windows-mpi-msys2/, with: > > > cd /mingw64/include > gfortran mpif90 -c -fno-range-check -fallow-invalid-boz > > Ah, yes, that?s new to gfortran 10 (we use gfortran 9 on our workers), which is now what?s ship with MSYS2 (we haven?t updated yet). Sorry that I forgot about that. > > This is needed to ?USE mpi? (as opposed to INCLUDE ?mpif.h?) > > > * Install the latest MS-MPI (both sdk and setup) from https://www.microsoft.com/en-us/download/details.aspx?id=100593 > > > At this point I?ve been able to compile (using the MINGW64 terminal) different mpi test programs and they run as expected in the classical Windows prompt. I added this function to my .bashrc in MSYS2 in order to easily copy the required dependencies out of MSYS: > > function copydep() { ldd $1 | grep "=> /$2" | awk '{print $3}' | xargs -I '{}' cp -v '{}' .; } > > which can be used, with the MINGW64 terminal, by navigating to the folder where the final executable, say, my.exe, resides (even if under a Windows path) and executing: > > copydep my.exe mingw64 > > This, of course, must be done before actually trying to execute the .exe in the windows cmd prompt. > > Hopefully, I should now be able to follow Pierre?s instructions for PETSc (but first I wanna give a try to the system python before removing it) > > Looks like the hard part is over. It?s usually easier to deal with ./configure issues. > If you have weird errors like ?incomplete Cygwin install? or whatever, this is the kind of issues I was referring to earlier. > In that case, what I?d suggest is just, as before: > pacman -R mingw-w64-x86_64-python mingw-w64-x86_64-gdb > pacman -S python > > Thanks, > Pierre > > Thanks > > Paolo > > > > > -- > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/> > > > > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From eda.oktay at metu.edu.tr Mon Jul 6 04:21:32 2020 From: eda.oktay at metu.edu.tr (Eda Oktay) Date: Mon, 6 Jul 2020 12:21:32 +0300 Subject: [petsc-users] EPSGetEigenpair Error In-Reply-To: References: Message-ID: Thanks! Jose E. Roman , 6 Tem 2020 Pzt, 12:10 tarihinde ?unu yazd?: > > The index i should be a value between 0 and nconv-1 > https://slepc.upv.es/documentation/current/docs/manualpages/EPS/EPSGetEigenpair.html > Jose > > > El 6 jul 2020, a las 11:06, Eda Oktay escribi?: > > > > Hello, > > > > I am using EPSGetEigenpair to find some matrices' eigenpairs: > > > > ierr = EPSGetType(eps,&type);CHKERRQ(ierr) > > ierr = EPSGetDimensions(eps,&nev,NULL,NULL);CHKERRQ(ierr); > > ierr = MatCreateVecs(L,&vr,NULL);CHKERRQ(ierr); > > Vec *V; > > VecDuplicateVecs(vr,nev,&V); > > for (i=0; i > ierr = EPSGetEigenpair(eps,i,&kr,NULL,V[i],NULL); > > } > > > > However, for some matrices, even though nev = 2, I am getting the > > following error: > > > > [0]PETSC ERROR: --------------------- Error Message > > -------------------------------------------------------------- > > [0]PETSC ERROR: Argument out of range > > [0]PETSC ERROR: Argument 2 out of range > > [0]PETSC ERROR: See > > https://www.mcs.anl.gov/petsc/documentation/faq.html for trouble > > shooting. > > [0]PETSC ERROR: Petsc Release Version 3.13.2, Jun 02, 2020 > > [0]PETSC ERROR: ./deneme_new_vecscatter_arastep on a > > arch-linux2-c-debug named b342.wls.metu.edu.tr by edaoktay Mon Jul 6 > > 12:01:14 2020 > > [0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ > > --download-mpich --download-openblas --download-slepc --download-metis > > --download-parmetis --download-chaco --with-X=1 > > [0]PETSC ERROR: #1 EPSGetEigenpair() line 411 in > > /home/edaoktay/petsc-3.13.2/arch-linux2-c-debug/externalpackages/git.slepc/src/eps/interface/epssolve.c > > > > =================================================================================== > > = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES > > = PID 8635 RUNNING AT b342.wls.metu.edu.tr > > = EXIT CODE: 134 > > = CLEANING UP REMAINING PROCESSES > > = YOU CAN IGNORE THE BELOW CLEANUP MESSAGES > > =================================================================================== > > YOUR APPLICATION TERMINATED WITH THE EXIT STRING: Aborted (signal 6) > > This typically refers to a problem with your application. > > Please see the FAQ page for debugging suggestions > > > > What can be the reason? I am getting this message when i = 0, so how > > can 0 be out of range for nev = 2? > > > > And why am I getting this message for 2 matrices out of 16? > > > > Thanks! > From junchao.zhang at gmail.com Mon Jul 6 11:07:01 2020 From: junchao.zhang at gmail.com (Junchao Zhang) Date: Mon, 6 Jul 2020 11:07:01 -0500 Subject: [petsc-users] Performance of ISCreateGeneral() In-Reply-To: References: <74e13a698286435235fc4a95d85fa853@cam.ac.uk> <0dbf7bfc2048833073995d291429f316@cam.ac.uk> Message-ID: Shidi, Rethink the comm passed to your ISCreateGeneral(). Could you do it with PETSC_COMM_SELF? --Junchao Zhang On Sun, Jul 5, 2020 at 3:27 PM Mark Adams wrote: > The column after the time in the PETSC output, on the ISCreateGeneral > row, with -log_view, shows the ratio of the max time (shown) to min time. > If you are using your own timers just report the min and max time that you > measure. > You can also put an MPI_Barrier before the call to ISCreateGeneral to get > clean data. > And I trust that the problem is well load balance, that is, all processors > have the same amount of data. > > On Sun, Jul 5, 2020 at 4:00 PM Y. Shidi wrote: > >> Thank you for your reply Mark. >> What do you mean by ratio (max/min)? >> Do I need to turn on some options for this? >> >> Kind regards, >> Shidi >> >> On 2020-07-05 20:15, Mark Adams wrote: >> > ISCreateGeneral just takes your indices and caches them. But it is a >> > synch point. What ratio (max/min) is PETSc reporting in this data? >> > >> > On Sun, Jul 5, 2020 at 1:51 PM Y. Shidi wrote: >> > >> >> Dear developers, >> >> >> >> I am currently doing a weak scaling test, and find that >> >> the weak scaling results for ISCreateGeneral() is very bad. >> >> For 2, 4, 8 processors, the running times for completing >> >> ISCreateGeneral() are: >> >> 0.687494, 3.00597 and 10.0613. >> >> I am not sure if this is normal. >> >> >> >> Thank you for your time. >> >> >> >> Kind Regards, >> >> Shidi >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From eda.oktay at metu.edu.tr Mon Jul 6 13:14:15 2020 From: eda.oktay at metu.edu.tr (Eda Oktay) Date: Mon, 6 Jul 2020 21:14:15 +0300 Subject: [petsc-users] /lib64/libc.so.6(cfree+0x1f8)[0x7f810f237438] Signal received Message-ID: Hello, I am trying to run my codes 100 times and then take the average elapsed time. Although my program runs correctly for single time, when I put them into loop, I am getting the following error: *** Error in `./avg_coloredge_deneme_new_vecscatter_arastep': munmap_chunk(): invalid pointer: 0x0000000000b0cbc0 *** [1]PETSC ERROR: ------------------------------------------------------------------------ [1]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [1]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [1]PETSC ERROR: or see https://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [1]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [1]PETSC ERROR: likely location of problem given in stack below [1]PETSC ERROR: --------------------- Stack Frames ------------------------------------ [1]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, [1]PETSC ERROR: INSTEAD the line number of the start of the function [1]PETSC ERROR: is given. ======= Backtrace: ========= [1]PETSC ERROR: /lib64/libc.so.6(+0x791eb)[0x7f810f22a1eb] --------------------- Error Message -------------------------------------------------------------- [1]PETSC ERROR: /lib64/libc.so.6(cfree+0x1f8)[0x7f810f237438] Signal received [1]PETSC ERROR: See https://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [1]PETSC ERROR: Petsc Release Version 3.13.2, Jun 02, 2020 [1]PETSC ERROR: ./avg_coloredge_deneme_new_vecscatter_arastep on a arch-linux2-c-debug named localhost.localdomain by edaoktay Mon Jul 6 21:09:07 2020 [1]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ --download-mpich --download-openblas --download-slepc --download-metis --download-parmetis --download-chaco --with-X=1 [1]PETSC ERROR: #1 User provided function() line 0 in unknown file ./avg_coloredge_deneme_new_vecscatter_arastep[0x402dca] ./avg_coloredge_deneme_new_vecscatter_arastep[0x402f1e] ./avg_coloredge_deneme_new_vecscatter_arastep[0x402953] ./avg_coloredge_deneme_new_vecscatter_arastep[0x406c1c] application called MPI_Abort(MPI_COMM_WORLD, 50152059) - process 1 /lib64/libc.so.6(__libc_start_main+0xf1)[0x7f810f1d1431] ./avg_coloredge_deneme_new_vecscatter_arastep[0x40260a] ======= Memory map: ======== 00400000-00409000 r-xp 00000000 fd:02 29230434 /home/edaoktay/petsc-3.13.2/arch-linux2-c-debug/share/slepc/examples/src/eda/avg_coloredge_deneme_new_vecscatter_arastep 00608000-00609000 r--p 00008000 fd:02 29230434 /home/edaoktay/petsc-3.13.2/arch-linux2-c-debug/share/slepc/examples/src/eda/avg_coloredge_deneme_new_vecscatter_arastep 00609000-0060a000 rw-p 00009000 fd:02 29230434 /home/edaoktay/petsc-3.13.2/arch-linux2-c-debug/share/slepc/examples/src/eda/avg_coloredge_deneme_new_vecscatter_arastep 00956000-01100000 rw-p 00000000 00:00 0 [heap] 7f810c621000-7f810c6ca000 rw-p 00000000 00:00 0 7f810c6ca000-7f810e6ca000 rw-p 00000000 00:00 0 7f810e6ca000-7f810e6d4000 r-xp 00000000 fd:00 1974339 /usr/lib64/libnss_files-2.24.so 7f810e6d4000-7f810e8d4000 ---p 0000a000 fd:00 1974339 /usr/lib64/libnss_files-2.24.so 7f810e8d4000-7f810e8d5000 r--p 0000a000 fd:00 1974339 /usr/lib64/libnss_files-2.24.so 7f810e8d5000-7f810e8d6000 rw-p 0000b000 fd:00 1974339 /usr/lib64/libnss_files-2.24.so 7f810e8d6000-7f810eb7d000 rw-p 00000000 00:00 0 7f810eb7d000-7f810eb7f000 r-xp 00000000 fd:00 1974984 /usr/lib64/libXau.so.6.0.0 7f810eb7f000-7f810ed7f000 ---p 00002000 fd:00 1974984 /usr/lib64/libXau.so.6.0.0 7f810ed7f000-7f810ed80000 r--p 00002000 fd:00 1974984 /usr/lib64/libXau.so.6.0.0 7f810ed80000-7f810ed81000 rw-p 00000000 00:00 0 7f810ed81000-7f810ed88000 r-xp 00000000 fd:00 1974440 /usr/lib64/librt-2.24.so 7f810ed88000-7f810ef87000 ---p 00007000 fd:00 1974440 /usr/lib64/librt-2.24.so 7f810ef87000-7f810ef88000 r--p 00006000 fd:00 1974440 /usr/lib64/librt-2.24.so 7f810ef88000-7f810ef89000 rw-p 00007000 fd:00 1974440 /usr/lib64/librt-2.24.so 7f810ef89000-7f810efb0000 r-xp 00000000 fd:00 1975953 /usr/lib64/libxcb.so.1.1.0 7f810efb0000-7f810f1af000 ---p 00027000 fd:00 1975953 /usr/lib64/libxcb.so.1.1.0 7f810f1af000-7f810f1b0000 r--p 00026000 fd:00 1975953 /usr/lib64/libxcb.so.1.1.0 7f810f1b0000-7f810f1b1000 rw-p 00027000 fd:00 1975953 /usr/lib64/libxcb.so.1.1.0 7f810f1b1000-7f810f36e000 r-xp 00000000 fd:00 1973798 /usr/lib64/libc-2.24.so 7f810f36e000-7f810f56d000 ---p 001bd000 fd:00 1973798 /usr/lib64/libc-2.24.so 7f810f56d000-7f810f571000 r--p 001bc000 fd:00 1973798 /usr/lib64/libc-2.24.so 7f810f571000-7f810f573000 rw-p 001c0000 fd:00 1973798 /usr/lib64/libc-2.24.so 7f810f573000-7f810f577000 rw-p 00000000 00:00 0 7f810f577000-7f810f5b6000 r-xp 00000000 fd:00 1975094 /usr/lib64/libquadmath.so.0.0.0 7f810f5b6000-7f810f7b5000 ---p 0003f000 fd:00 1975094 /usr/lib64/libquadmath.so.0.0.0 7f810f7b5000-7f810f7b6000 r--p 0003e000 fd:00 1975094 /usr/lib64/libquadmath.so.0.0.0 7f810f7b6000-7f810f7b7000 rw-p 0003f000 fd:00 1975094 /usr/lib64/libquadmath.so.0.0.0 7f810f7b7000-7f810f7cd000 r-xp 00000000 fd:00 1973274 /usr/lib64/libgcc_s-6.4.1-20170727.so.1 7f810f7cd000-7f810f9cc000 ---p 00016000 fd:00 1973274 /usr/lib64/libgcc_s-6.4.1-20170727.so.1 7f810f9cc000-7f810f9cd000 r--p 00015000 fd:00 1973274 /usr/lib64/libgcc_s-6.4.1-20170727.so.1 7f810f9cd000-7f810f9ce000 rw-p 00016000 fd:00 1973274 /usr/lib64/libgcc_s-6.4.1-20170727.so.1 7f810f9ce000-7f810fafc000 r-xp 00000000 fd:00 1975116 /usr/lib64/libgfortran.so.3.0.0 7f810fafc000-7f810fcfc000 ---p 0012e000 fd:00 1975116 /usr/lib64/libgfortran.so.3.0.0 7f810fcfc000-7f810fcfd000 r--p 0012e000 fd:00 1975116 /usr/lib64/libgfortran.so.3.0.0 7f810fcfd000-7f810fcfe000 rw-p 0012f000 fd:00 1975116 /usr/lib64/libgfortran.so.3.0.0 7f810fcfe000-7f810fcff000 rw-p 00000000 00:00 0 7f810fcff000-7f8110168000 r-xp 00000000 fd:02 28457339 /home/edaoktay/petsc-3.13.2/arch-linux2-c-debug/lib/libmpi.so.12.1.8 7f8110168000-7f8110368000 ---p 00469000 fd:02 28457339 /home/edaoktay/petsc-3.13.2/arch-linux2-c-debug/lib/libmpi.so.12.1.8 7f8110368000-7f811037a000 r--p 00469000 fd:02 28457339 /home/edaoktay/petsc-3.13.2/arch-linux2-c-debug/lib/libmpi.so.12.1.8 7f811037a000-7f8110385000 rw-p 0047b000 fd:02 28457339 /home/edaoktay/petsc-3.13.2/arch-linux2-c-debug/lib/libmpi.so.12.1.8 7f8110385000-7f81103c4000 rw-p 00000000 00:00 0 7f81103c4000-7f8110435000 r-xp 00000000 fd:02 28457343 /home/edaoktay/petsc-3.13.2/arch-linux2-c-debug/lib/libmpifort.so.12.1.8 7f8110435000-7f8110635000 ---p 00071000 fd:02 28457343 /home/edaoktay/petsc-3.13.2/arch-linux2-c-debug/lib/libmpifort.so.12.1.8 7f8110635000-7f8110639000 r--p 00071000 fd:02 28457343 /home/edaoktay/petsc-3.13.2/arch-linux2-c-debug/lib/libmpifort.so.12.1.8 7f8110639000-7f811063b000 rw-p 00075000 fd:02 28457343 /home/edaoktay/petsc-3.13.2/arch-linux2-c-debug/lib/libmpifort.so.12.1.8 7f811063b000-7f811063e000 r-xp 00000000 fd:00 1973895 /usr/lib64/libdl-2.24.so 7f811063e000-7f811083d000 ---p 00003000 fd:00 1973895 /usr/lib64/libdl-2.24.so 7f811083d000-7f811083e000 r--p 00002000 fd:00 1973895 /usr/lib64/libdl-2.24.so 7f811083e000-7f811083f000 rw-p 00003000 fd:00 1973895 /usr/lib64/libdl-2.24.so 7f811083f000-7f81109b7000 r-xp 00000000 fd:00 1974512 /usr/lib64/libstdc++.so.6.0.22 7f81109b7000-7f8110bb7000 ---p 00178000 fd:00 1974512 /usr/lib64/libstdc++.so.6.0.22 7f8110bb7000-7f8110bc1000 r--p 00178000 fd:00 1974512 /usr/lib64/libstdc++.so.6.0.22 7f8110bc1000-7f8110bc3000 rw-p 00182000 fd:00 1974512 /usr/lib64/libstdc++.so.6.0.22 7f8110bc3000-7f8110bc7000 rw-p 00000000 00:00 0 7f8110bc7000-7f8110ccf000 r-xp 00000000 fd:00 1973951 /usr/lib64/libm-2.24.so 7f8110ccf000-7f8110ece000 ---p 00108000 fd:00 1973951 /usr/lib64/libm-2.24.so 7f8110ece000-7f8110ecf000 r--p 00107000 fd:00 1973951 /usr/lib64/libm-2.24.so 7f8110ecf000-7f8110ed0000 rw-p 00108000 fd:00 1973951 /usr/lib64/libm-2.24.so 7f8110ed0000-7f8110f60000 r-xp 00000000 fd:02 28457370 /home/edaoktay/petsc-3.13.2/arch-linux2-c-debug/lib/libmetis.so 7f8110f60000-7f811115f000 ---p 00090000 fd:02 28457370 /home/edaoktay/petsc-3.13.2/arch-linux2-c-debug/lib/libmetis.so 7f811115f000-7f8111160000 r--p 0008f000 fd:02 28457370 /home/edaoktay/petsc-3.13.2/arch-linux2-c-debug/lib/libmetis.so 7f8111160000-7f8111161000 rw-p 00090000 fd:02 28457370 /home/edaoktay/petsc-3.13.2/arch-linux2-c-debug/lib/libmetis.so 7f8111161000-7f8111169000 rw-p 00000000 00:00 0 7f8111169000-7f81111bf000 r-xp 00000000 fd:02 28457373 /home/edaoktay/petsc-3.13.2/arch-linux2-c-debug/lib/libparmetis.so 7f81111bf000-7f81113bf000 ---p 00056000 fd:02 28457373 /home/edaoktay/petsc-3.13.2/arch-linux2-c-debug/lib/libparmetis.so 7f81113bf000-7f81113c0000 r--p 00056000 fd:02 28457373 /home/edaoktay/petsc-3.13.2/arch-linux2-c-debug/lib/libparmetis.so 7f81113c0000-7f81113c1000 rw-p 00057000 fd:02 28457373 /home/edaoktay/petsc-3.13.2/arch-linux2-c-debug/lib/libparmetis.so 7f81113c1000-7f81114f9000 r-xp 00000000 fd:00 1973763 /usr/lib64/libX11.so.6.3.0 7f81114f9000-7f81116f9000 ---p 00138000 fd:00 1973763 /usr/lib64/libX11.so.6.3.0 7f81116f9000-7f81116fc000 r--p 00138000 fd:00 1973763 /usr/lib64/libX11.so.6.3.0 7f81116fc000-7f8111700000 rw-p 0013b000 fd:00 1973763 /usr/lib64/libX11.so.6.3.0 7f8111700000-7f8111718000 r-xp 00000000 fd:00 1974340 /usr/lib64/libpthread-2.24.so 7f8111718000-7f8111918000 ---p 00018000 fd:00 1974340 /usr/lib64/libpthread-2.24.so 7f8111918000-7f8111919000 r--p 00018000 fd:00 1974340 /usr/lib64/libpthread-2.24.so 7f8111919000-7f811191a000 rw-p 00019000 fd:00 1974340 /usr/lib64/libpthread-2.24.so 7f811191a000-7f811191e000 rw-p 00000000 00:00 0 7f811191e000-7f81124ff000 r-xp 00000000 fd:02 28457384 /home/edaoktay/petsc-3.13.2/arch-linux2-c-debug/lib/libopenblas_haswell-r0.3.8.dev.so 7f81124ff000-7f81126ff000 ---p 00be1000 fd:02 28457384 /home/edaoktay/petsc-3.13.2/arch-linux2-c-debug/lib/libopenblas_haswell-r0.3.8.dev.so 7f81126ff000-7f8112703000 r--p 00be1000 fd:02 28457384 /home/edaoktay/petsc-3.13.2/arch-linux2-c-debug/lib/libopenblas_haswell-r0.3.8.dev.so 7f8112703000-7f811270d000 rw-p 00be5000 fd:02 28457384 /home/edaoktay/petsc-3.13.2/arch-linux2-c-debug/lib/libopenblas_haswell-r0.3.8.dev.so 7f811270d000-7f811270e000 rw-p 00000000 00:00 0 7f811270e000-7f81147c2000 r-xp 00000000 fd:02 28457400 /home/edaoktay/petsc-3.13.2/arch-linux2-c-debug/lib/libpetsc.so.3.13.2 7f81147c2000-7f81149c1000 ---p 020b4000 fd:02 28457400 /home/edaoktay/petsc-3.13.2/arch-linux2-c-debug/lib/libpetsc.so.3.13.2 7f81149c1000-7f81149c7000 r--p 020b3000 fd:02 28457400 /home/edaoktay/petsc-3.13.2/arch-linux2-c-debug/lib/libpetsc.so.3.13.2 7f81149c7000-7f81149df000 rw-p 020b9000 fd:02 28457400 /home/edaoktay/petsc-3.13.2/arch-linux2-c-debug/lib/libpetsc.so.3.13.2 7f81149df000-7f8114bdf000 rw-p 00000000 00:00 0 7f8114bdf000-7f8115220000 r-xp 00000000 fd:02 28457416 /home/edaoktay/petsc-3.13.2/arch-linux2-c-debug/lib/libslepc.so.3.13.2 7f8115220000-7f8115420000 ---p 00641000 fd:02 28457416 /home/edaoktay/petsc-3.13.2/arch-linux2-c-debug/lib/libslepc.so.3.13.2 7f8115420000-7f8115421000 r--p 00641000 fd:02 28457416 /home/edaoktay/petsc-3.13.2/arch-linux2-c-debug/lib/libslepc.so.3.13.2 7f8115421000-7f8115425000 rw-p 00642000 fd:02 28457416 /home/edaoktay/petsc-3.13.2/arch-linux2-c-debug/lib/libslepc.so.3.13.2 7f8115425000-7f811544b000 r-xp 00000000 fd:00 1976838 /usr/lib64/ld-2.24.so 7f81154d3000-7f8115621000 rw-p 00000000 00:00 0 7f8115647000-7f811564a000 rw-p 00000000 00:00 0 7f811564a000-7f811564b000 r--p 00025000 fd:00 1976838 /usr/lib64/ld-2.24.so 7f811564b000-7f811564c000 rw-p 00026000 fd:00 1976838 /usr/lib64/ld-2.24.so 7f811564c000-7f811564d000 rw-p 00000000 00:00 0 7ffc79b33000-7ffc79b55000 rw-p 00000000 00:00 0 [stack] 7ffc79b67000-7ffc79b6a000 r--p 00000000 00:00 0 [vvar] 7ffc79b6a000-7ffc79b6c000 r-xp 00000000 00:00 0 [vdso] ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0 [vsyscall] I created and destroyed every vector, matrix and EPS in the loop to make sure that there won't be any segmentation fault but still I am getting some. Why am I getting this type of error? Thanks so much! Eda From balay at mcs.anl.gov Mon Jul 6 13:31:23 2020 From: balay at mcs.anl.gov (Satish Balay) Date: Mon, 6 Jul 2020 13:31:23 -0500 (CDT) Subject: [petsc-users] R: PETSc and Windows 10 In-Reply-To: References: <32B37EFD-DDFA-41AE-ACC7-CB5E6AE57996@enseeiht.fr> <0558CCF7-8203-4520-8673-3B0945C43233@enseeiht.fr> <20AB9F7A-4471-498B-9A78-AA48163900CB@enseeiht.fr> <94A8E366-2361-409E-84CE-B90EA54D9AF9@enseeiht.fr> <849E7E3B-8B54-4B40-A2B7-8DD5B55428B3@enseeiht.fr> , <246CA2C6-E66F-4EF0-8600-81AB741D17D8@enseeiht.fr>, , Message-ID: I was thinking in terms of: If using mingw-gcc from cygwin - then it could be used in the same way as mingw-gcc in msys2 is used - i.e with MS-MPI etc.. [one can install mingw-gcc in cygwin - which is different than cygwin native gcc - perhaps this is similar to mingw-gcc install in msys2] I haven't tried this though.. Likely cygwin doesn't have the equivalent of mingw-w64-x86_64-msmpi - for easy use of MS-MPI from mingw-gfortran Satish On Mon, 6 Jul 2020, Paolo Lampitella wrote: > Dear Satish, > > Yes indeed, or at least that is my understanding. Still, my experience so far with Cygwin has been, let?s say, controversial. > > I wasn?t able to compile myself MPICH, with both gcc and mingw. > > When having PETSc compile also MPICH, I was successful only with gcc but not mingw. > > I didn?t even try compiling OpenMPI with mingw, as PETSc compilation already failed using the OpenMPI available trough cygwin libraries (which is based on gcc and not mingw). > > Not sure if this is my fault, but in the end it didn?t go well > > Paolo > > Inviato da Posta per Windows 10 > > Da: Satish Balay > Inviato: domenica 5 luglio 2020 23:50 > A: Paolo Lampitella > Cc: Pierre Jolivet; petsc-users > Oggetto: Re: [petsc-users] PETSc and Windows 10 > > Sounds like there are different mingw tools and msys2 tools. > > So I guess one could use mingw compilers even from cygwin [using cygwin tools] - i.e mingw compilers don't really need msys2 tools to work. > > Satish > > On Sun, 5 Jul 2020, Paolo Lampitella wrote: > > > Unfortunately, even PETSC_ARCH=i didn't work out. And while with-single-library=0 wasn't really appealing to me, it worked but only to later fail on make test. > > > > I guess all these differences are due to the fortran bindings and/or gcc 10. > > > > However, until I discover how they are different, I guess I'll be fine with /usr/bin/ar > > > > Paolo > > > > > > > > Inviato da smartphone Samsung Galaxy. > > > > > > > > -------- Messaggio originale -------- > > Da: Paolo Lampitella > > Data: 05/07/20 14:00 (GMT+01:00) > > A: Pierre Jolivet > > Cc: Matthew Knepley , petsc-users > > Oggetto: RE: [petsc-users] PETSc and Windows 10 > > > > Thank you very much Pierre. > > > > I'll keep you informed in case I see any relevant change from the tests when using your suggestion. > > > > Paolo > > > > > > > > Inviato da smartphone Samsung Galaxy. > > > > > > > > -------- Messaggio originale -------- > > Da: Pierre Jolivet > > Data: 05/07/20 13:45 (GMT+01:00) > > A: Paolo Lampitella > > Cc: Matthew Knepley , petsc-users > > Oggetto: Re: [petsc-users] PETSc and Windows 10 > > > > Hello Paolo, > > > > On 5 Jul 2020, at 1:15 PM, Paolo Lampitella > wrote: > > > > Dear all, > > > > I just want to update you on my journey to PETSc compilation in Windows under MSYS2+MINGW64 > > > > Unfortunately, I haven?t been able to compile petsc-slepc trough Freefem but, as my final goal required also Fortran bindings (but I only needed blas, lapack, metis and hypre), I decided to follow my own route using the useful information from Pierre. > > > > > > * I started by installing MPI from https://www.microsoft.com/en-us/download/details.aspx?id=100593. I don?t think the SDK is actually needed in my specific workflow, but I installed it as well together with mpisetup. > > * Then I installed MSYS2 just following the wizard. Opened the MSYS2 terminal and updated with pacman -Syuu, closed if asked, reopened it and used again pacman -Syuu several times until no more updates were available. Closed it and opened it back. > > * Under the MSYS2 terminal installed just the following packages: > > > > > > > > * pacman -S base-devel git gcc gcc-fortran > > * pacman -S mingw-w64-x86_64-toolchain > > * pacman -S mingw-w64-x86_64-cmake > > * pacman -S mingw-w64-x86_64-msmpi > > > > > > > > * Closed the MSYS2 terminal and opened the MINGW64 one, went to /mingw64/include and compiled my mpi module following https://www.scivision.dev/windows-mpi-msys2/: > > > > > > > > * gfortran -c mpi.f90 -fno-range-check -fallow-invalid-boz > > > > > > However, I will keep an eye on the MS-MPI GitHub repository because the fortran side seems to be far from perfect. > > > > > > * Then I downloaded the 3.13.3 version of petsc and configured it, still under the MINGW64 terminal, with the following command: > > > > > > /usr/bin/python ./configure --prefix=/home/paolo/petsc --with-ar=/usr/bin/ar > > --with-shared-libraries=0 --with-debugging=0 --with-windows-graphics=0 --with-x=0 > > COPTFLAGS="-O3 -mtune=native" > > CXXOPTFLAGS="-O3 -mtune=native" > > FOPTFLAGS="-O3 -mtune=native" > > FFLAGS=-fallow-invalid-boz > > --with-mpiexec="/C/Program\ Files/Microsoft\ MPI/Bin/mpiexec" > > --download-fblaslapack --download-metis --download-hypre > > --download-metis-cmake-arguments='-G "MSYS Makefiles"' > > --download-hypre-configure-arguments="--build=x86_64-linux-gnu --host=x86_64-linux-gnu" > > > > Note that I just bypassed uninstalling python in mingw64 (which doesn?t work) by using /usr/bin/python and that, as opposed to Pierre, I needed to also use the MSYS2 archiver (/usr/bin/ar) as opposed to the mingw64 one (/mingw64/bin/ar that shows up in the Pierre configure) as also mentioned here http://hillyuan.blogspot.com/2017/11/build-petsc-in-windows-under-mingw64.html, probably because of this issue https://stackoverflow.com/questions/37504625/ar-on-msys2-shell-receives-truncated-paths-when-called-from-makefile. > > > > You are right that you can avoid deinstalling mingw-w64-x86_64-python if you can supply the proper Python yourself (we don?t have that luxury in our Makefile). > > If you want to avoid using that AR, and stick to /mingw64/bin/ar (not sure what the pros and cons are), you can either: > > - use another PETSC_ARCH (very short, like pw, for petsc-windows); > > - use --with-single-library=0. > > See this post on GitLab https://gitlab.com/petsc/petsc/-/issues/647#note_373507681 > > The OS I?m referring to is indeed my Windows + MSYS2 box. > > > > Thanks, > > Pierre > > > > Then make all, make install and make check all went smooth. Also, I don?t know exactly what with-x=0 and with-windows-graphics=0 do, but I think it is stuff that I don?t need (yet configure worked with windows-graphics as well). > > > > > > * Finally I launched make test. As some tests failed, I replicated the same install procedure on all the systems I have available on this same Windows machine (Ubuntu 20.04 and Centos 8 under a VirtualBox 6.0.22 VM, Ubuntu 20.04 under WSL1 and the MSYS2-MINGW64 toolchain). I am attaching a file with the results printed to screen (not sure about which file should be used for a comparison/check). Note, however, that the tests in MSYS2 started with some cyclic reference issues for some .mod files, but this doesn?t show up in any file I could check. > > > > > > I am still left with some doubts about the archiver, the cyclic reference errors and the differences in the test results, but I am able to link my code with petsc. Unfortunately, as this Windows porting is part of a large code restructuring, I can?t do much more with it, now, from my code. But if you can suggest some specific tutorial to use as test also for the parallel, I would be glad to dig deeper into the matter. > > > > Best regards > > > > Paolo > > > > Inviato da Posta per Windows 10 > > > > Da: Pierre Jolivet > > Inviato: marted? 30 giugno 2020 15:22 > > A: Paolo Lampitella > > Cc: Matthew Knepley; petsc-users > > Oggetto: Re: [petsc-users] PETSc and Windows 10 > > > > Please use the 3.13.2 tarball, this was fixed by Satish in the previous commit I already linked (https://gitlab.com/petsc/petsc/-/commit/2cd8068296b34e127f055bb32f556e3599f17523). > > (If you want FreeFEM to do the dirty work for you, just switch to the develop branch, and redo ?make petsc-slepc?) > > But I think you?ve got everything you need now for a smooth compilation :) > > > > Thanks, > > Pierre > > > > > > On 30 Jun 2020, at 3:09 PM, Paolo Lampitella > wrote: > > > > Dear Pierre, > > > > thanks for the fast response. Unfortunately it still fails, but now in the configure of ScaLAPACK > > (which means that it went ok for slepc, tetgen, metis, parmetis, ptscotch, superlu and suitesparse). > > > > The way I applied the modification is by manually editing the Makefile in the 3rdparty/ff-petsc folder, adding -fallow-invalid-boz to both CFLAGS and FFLAGS (this entry added by me). Then executed make petsc-slepc. > > > > As my project is much less ambitious, I have a good feeling that I will be able to use your Makefile successfully, but as I am kind of slow I tought that it would have been useful for you to know. The configure.log is attached. This time the error is: > > > > Rank mismatch between actual argument at (1) and actual argument at (2) (scalar and rank-1) > > > > in subroutine pclarf.f of ScaLAPACK. > > > > However, before attampting with my project, I have few questions about your Makefile, in particular this piece: > > > > --with-mpi-lib=/c/Windows/System32/msmpi.dll --with-mpi-include=/home/paolo/FreeFem-sources/3rdparty/include/msmpi --with-mpiexec="/C/Program\ Files/Microsoft\ MPI/Bin/mpiexec" > > > > I see from MPI.py that I should not use ?--with-mpi-lib/include? if I want to use my now working mpi wrappers. Is this correct? > > > > Paolo > > > > Inviato da Posta per Windows 10 > > > > Da: Pierre Jolivet > > Inviato: luned? 29 giugno 2020 21:37 > > A: Paolo Lampitella > > Cc: Matthew Knepley; petsc-users > > Oggetto: Re: [petsc-users] PETSc and Windows 10 > > > > I do not give up easily on Windows problems: > > 1) that?s around 50% of our (FreeFEM) user-base (and I want them to use PETSc and SLEPc, ofc?) > > 2) most people I work with from corporations just have Windows laptops/desktops and I always recommend MSYS because it?s very lightweight and you can pass .exe around > > 3) I?ve bothered enough Satish, Jed, and Matt on GitLab to take (at least partially) the blame now when it doesn?t work on MSYS > > > > That being said, the magic keyword is the added flag FFLAGS="-fallow-invalid-boz" (see, I told you ./configure issues were easier to deal with than the others). > > Here you?ll see that everything goes through just fine (sorry, it took me a long time to post this because everything is slow on my VM): > > 1) http://jolivet.perso.enseeiht.fr/win10/configure.log > > 2) http://jolivet.perso.enseeiht.fr/win10/make.log (both steps #1 and #2 in MSYS terminal, gcc/gfortran 10, MS-MPI see screenshot) > > 3) http://jolivet.perso.enseeiht.fr/win10/ex2.txt (Command Prompt, 4 processes + MUMPS, I can send you the .exe if you want to try on your machine) > > I just realize that I didn?t generate the Fortran bindings, but you can see I compiled MUMPS and ScaLAPACK, so that shouldn?t be a problem. > > Or if there is a problem, we will need to fix this in PETSc. > > > > I?ll push this added flag to the FreeFEM repo, thanks for reminding me of the brokenness of gcc/gfortran 10 + MS-MPI. > > Here is to hoping this won?t affect PETSc ./configure with previous gcc/gfortran version (unlikely, this option is apparently 13-year old https://gcc.gnu.org/bugzilla/show_bug.cgi?id=29471) > > > > Let me know of the next hiccup, if any. > > Thanks, > > Pierre > > > > > > > > On 29 Jun 2020, at 8:09 PM, Paolo Lampitella > wrote: > > > > Dear Pierre, > > > > thanks again for your time > > > > I guess there is no way for me to use the toolchain you are using (I don?t remember having any choice on which version of MSYS or GCC I could install) > > > > Paolo > > > > Inviato da Posta per Windows 10 > > > > Da: Pierre Jolivet > > Inviato: luned? 29 giugno 2020 20:01 > > A: Matthew Knepley > > Cc: Paolo Lampitella; petsc-users > > Oggetto: Re: [petsc-users] PETSc and Windows 10 > > > > > > > > > > > > > > On 29 Jun 2020, at 7:47 PM, Matthew Knepley > wrote: > > > > On Mon, Jun 29, 2020 at 1:35 PM Paolo Lampitella > wrote: > > Dear Pierre, sorry to bother you, but I already have some issues. What I did: > > > > > > * pacman -R mingw-w64-x86_64-python mingw-w64-x86_64-gdb (is gdb also troublesome?) > > * Followed points 6 and 7 at https://doc.freefem.org/introduction/installation.html#compilation-on-windows > > > > I first got a warning on the configure at point 6, as ?disable-hips is not recognized. Then, on make ?petsc-slepc? of point 7 (no SUDO=sudo flag was necessary) I got to this point: > > > > tar xzf ../pkg/petsc-lite-3.13.0.tar.gz > > patch -p1 < petsc-suitesparse.patch > > patching file petsc-3.13.0/config/BuildSystem/config/packages/SuiteSparse.py > > touch petsc-3.13.0/tag-tar > > cd petsc-3.13.0 && ./configure MAKEFLAGS='' \ > > --prefix=/home/paolo/freefem/ff-petsc//r \ > > --with-debugging=0 COPTFLAGS='-O3 -mtune=generic' CXXOPTFLAGS='-O3 -mtune=generic' FOPTFLAGS='-O3 -mtune=generic' --with-cxx-dialect=C++11 --with-ssl=0 --with-x=0 --with-fortran-bindings=0 --with-shared-libraries=0 --with-cc='gcc' --with-cxx='g++' --with-fc='gfortran' CXXFLAGS='-fno-stack-protector' CFLAGS='-fno-stack-protector' --with-scalar-type=real --with-mpi-lib='/c/Windows/System32/msmpi.dll' --with-mpi-include='/home/paolo/FreeFem-sources/3rdparty/include/msmpi' --with-mpiexec='/C/Program\ Files/Microsoft\ MPI/Bin/mpiexec' --with-blaslapack-include='' --with-blaslapack-lib='/mingw64/bin/libopenblas.dll' --download-scalapack --download-metis --download-ptscotch --download-mumps --download-hypre --download-parmetis --download-superlu --download-suitesparse --download-tetgen --download-slepc '--download-metis-cmake-arguments=-G "MSYS Makefiles"' '--download-parmetis-cmake-arguments=-G "MSYS Makefiles"' '--download-superlu-cmake-arguments=-G "MSYS Makefiles"' ' -- > download-hypre-configure-arguments=--build=x86_64-linux-gnu --host=x86_64-linux-gnu' PETSC_ARCH=fr > > =============================================================================== > > Configuring PETSc to compile on your system > > =============================================================================== > > TESTING: FortranMPICheck from config.packages.MPI(config/BuildSystem/config/pack******************************************************************************* > > UNABLE to CONFIGURE with GIVEN OPTIONS (see configure.log for details): > > ------------------------------------------------------------------------------- > > Fortran error! mpi_init() could not be located! > > ******************************************************************************* > > > > make: *** [Makefile:210: petsc-3.13.0/tag-conf-real] Errore 1 > > > > Note that I didn?t add anything to any PATH variable, because this is not mentioned in your documentation. > > > > On a side note, this is the same error I got when trying to build PETSc in Cygwin with the default OpenMPI available in Cygwin. > > > > I am attaching the configure.log? it seems to me that the error comes from the configure trying to include the mpif.h in your folder and not using the -fallow-invalid-boz flag that I had to use, for example, to compile mpi.f90 into mpi.mod > > > > But I?m not sure why this is happening > > > > Pierre, > > > > Could this be due to gcc 10? > > > > Sorry, I?m slow. You are right. Our workers use gcc 9, everything is fine, but I see on my VM which I updated that I use gcc 10 and had to disable Fortran, I guess the MUMPS run I showcased was with a prior PETSc build. > > I?ll try to resolve this and will keep you posted. > > They really caught a lot of people off guard with gfortran 10? > > > > Thanks, > > Pierre > > > > > > > > > > Executing: gfortran -c -o /tmp/petsc-ur0cff6a/config.libraries/conftest.o -I/tmp/petsc-ur0cff6a/config.compilers -I/tmp/petsc-ur0cff6a/config.setCompilers -I/tmp/petsc-ur0cff6a/config.compilersFortran -I/tmp/petsc-ur0cff6a/config.libraries -Wall -ffree-line-length-0 -Wno-unused-dummy-argument -O3 -mtune=generic -I/home/paolo/FreeFem-sources/3rdparty/include/msmpi /tmp/petsc-ur0cff6a/config.libraries/conftest.F90 > > Possible ERROR while running compiler: exit code 1 > > stderr: > > C:/msys64/home/paolo/FreeFem-sources/3rdparty/include/msmpi/mpif.h:227:36: > > > > 227 | PARAMETER (MPI_DATATYPE_NULL=z'0c000000') > > | 1 > > Error: BOZ literal constant at (1) is neither a data-stmt-constant nor an actual argument to INT, REAL, DBLE, or CMPLX intrinsic function [see '-fno-allow-invalid-boz'] > > C:/msys64/home/paolo/FreeFem-sources/3rdparty/include/msmpi/mpif.h:303:27: > > > > 303 | PARAMETER (MPI_CHAR=z'4c000101') > > | 1 > > Error: BOZ literal constant at (1) is neither a data-stmt-constant nor an actual argument to INT, REAL, DBLE, or CMPLX intrinsic function [see '-fno-allow-invalid-boz'] > > C:/msys64/home/paolo/FreeFem-sources/3rdparty/include/msmpi/mpif.h:305:36: > > > > 305 | PARAMETER (MPI_UNSIGNED_CHAR=z'4c000102') > > | 1 > > > > Thanks, > > > > Matt > > > > Thanks > > > > Paolo > > > > Inviato da Posta per Windows 10 > > > > Da: Pierre Jolivet > > Inviato: luned? 29 giugno 2020 18:34 > > A: Paolo Lampitella > > Cc: Satish Balay; petsc-users > > Oggetto: Re: [petsc-users] PETSc and Windows 10 > > > > > > > > On 29 Jun 2020, at 6:27 PM, Paolo Lampitella > wrote: > > > > I think I made the first step of having mingw64 from msys2 working with ms-mpi. > > > > I found that the issue I was having was related to: > > > > https://gcc.gnu.org/bugzilla/show_bug.cgi?id=91556 > > > > and, probably (but impossible to check now), I was using an msys2 and/or mingw mpi package before this fix: > > > > https://github.com/msys2/MINGW-packages/commit/11b4cff3d2ec7411037b692b0ad5a9f3e9b9978d#diff-eac59989e3096be97d940c8f47b50fba > > > > Admittedly, I never used gcc 10 before on any machine. Still, I feel that reporting that sort of error in that way is, > > at least, misleading (I would have preferred the initial implementation as mentioned in the gcc bug track). > > > > A second thing that I was not used to, and made me more uncertain of the procedure I was following, is having to compile myself the mpi module. There are several version of this out there, but I decided to stick with this one: > > > > https://www.scivision.dev/windows-mpi-msys2/ > > > > even if there seems to be no need to include -fno-range-check and the current mpi.f90 version is different from the mpif.h as reported here: > > > > https://github.com/microsoft/Microsoft-MPI/issues/33 > > > > which, to me, are both signs of lack of attention on the fortran side by those that maintain this thing. > > > > In summary, this is the procedure I followed so far (on a 64 bit machine with Windows 10): > > > > > > * Install MSYS2 from https://www.msys2.org/ and just follow the install wizard > > * Open the MSYS2 terminal and execute: pacman -Syuu > > * Close the terminal when asked and reopen it > > * Keep executing ?pacman -Syuu? until nothing else needs to be updated > > * Close the MSYS2 terminal and reopen it (I guess because was in paranoid mode), then install packages with: > > > > > > pacman -S base-devel git gcc gcc-fortran bsdcpio lndir pax-git unzip > > pacman -S mingw-w64-x86_64-toolchain > > pacman -S mingw-w64-x86_64-msmpi > > pacman -S mingw-w64-x86_64-cmake > > pacman -S mingw-w64-x86_64-freeglut > > pacman -S mingw-w64-x86_64-gsl > > pacman -S mingw-w64-x86_64-libmicroutils > > pacman -S mingw-w64-x86_64-hdf5 > > pacman -S mingw-w64-x86_64-openblas > > pacman -S mingw-w64-x86_64-arpack > > pacman -S mingw-w64-x86_64-jq > > > > This set should include all the libraries mentioned by Pierre and/or used by his Jenkins, as the final scope here is to have PETSc and dependencies working. But I think that for pure MPI one could stop to msmpi (even, maybe, just install msmpi and have the dependencies figured out by pacman). Honestly, I don?t remember the exact order I used to install the packages, but this should not affect things. Also, as I was still in paranoid mode, I kept executing ?pacman -Syuu? after each package was installed. After this, close the MSYS2 terminal. > > > > > > * Open the MINGW64 terminal and create the .mod file out of the mpi.f90 file, as mentioned here https://www.scivision.dev/windows-mpi-msys2/, with: > > > > > > cd /mingw64/include > > gfortran mpif90 -c -fno-range-check -fallow-invalid-boz > > > > Ah, yes, that?s new to gfortran 10 (we use gfortran 9 on our workers), which is now what?s ship with MSYS2 (we haven?t updated yet). Sorry that I forgot about that. > > > > This is needed to ?USE mpi? (as opposed to INCLUDE ?mpif.h?) > > > > > > * Install the latest MS-MPI (both sdk and setup) from https://www.microsoft.com/en-us/download/details.aspx?id=100593 > > > > > > At this point I?ve been able to compile (using the MINGW64 terminal) different mpi test programs and they run as expected in the classical Windows prompt. I added this function to my .bashrc in MSYS2 in order to easily copy the required dependencies out of MSYS: > > > > function copydep() { ldd $1 | grep "=> /$2" | awk '{print $3}' | xargs -I '{}' cp -v '{}' .; } > > > > which can be used, with the MINGW64 terminal, by navigating to the folder where the final executable, say, my.exe, resides (even if under a Windows path) and executing: > > > > copydep my.exe mingw64 > > > > This, of course, must be done before actually trying to execute the .exe in the windows cmd prompt. > > > > Hopefully, I should now be able to follow Pierre?s instructions for PETSc (but first I wanna give a try to the system python before removing it) > > > > Looks like the hard part is over. It?s usually easier to deal with ./configure issues. > > If you have weird errors like ?incomplete Cygwin install? or whatever, this is the kind of issues I was referring to earlier. > > In that case, what I?d suggest is just, as before: > > pacman -R mingw-w64-x86_64-python mingw-w64-x86_64-gdb > > pacman -S python > > > > Thanks, > > Pierre > > > > Thanks > > > > Paolo > > > > > > > > > > -- > > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. > > -- Norbert Wiener > > > > https://www.cse.buffalo.edu/~knepley/> > > > > > > > > > > > > > > > > > > From knepley at gmail.com Mon Jul 6 14:35:57 2020 From: knepley at gmail.com (Matthew Knepley) Date: Mon, 6 Jul 2020 15:35:57 -0400 Subject: [petsc-users] /lib64/libc.so.6(cfree+0x1f8)[0x7f810f237438] Signal received In-Reply-To: References: Message-ID: The first thing I would do is reduce to serial until it works there. It makes debugging 100x easier. THanks, Matt On Mon, Jul 6, 2020 at 2:14 PM Eda Oktay wrote: > Hello, > > I am trying to run my codes 100 times and then take the average > elapsed time. Although my program runs correctly for single time, when > I put them into loop, I am getting the following error: > > *** Error in `./avg_coloredge_deneme_new_vecscatter_arastep': > munmap_chunk(): invalid pointer: 0x0000000000b0cbc0 *** > [1]PETSC ERROR: > ------------------------------------------------------------------------ > [1]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, > probably memory access out of range > [1]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger > [1]PETSC ERROR: or see > https://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind > [1]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac > OS X to find memory corruption errors > [1]PETSC ERROR: likely location of problem given in stack below > [1]PETSC ERROR: --------------------- Stack Frames > ------------------------------------ > [1]PETSC ERROR: Note: The EXACT line numbers in the stack are not > available, > [1]PETSC ERROR: INSTEAD the line number of the start of the function > [1]PETSC ERROR: is given. > ======= Backtrace: ========= > [1]PETSC ERROR: /lib64/libc.so.6(+0x791eb)[0x7f810f22a1eb] > --------------------- Error Message > -------------------------------------------------------------- > [1]PETSC ERROR: /lib64/libc.so.6(cfree+0x1f8)[0x7f810f237438] > Signal received > [1]PETSC ERROR: See > https://www.mcs.anl.gov/petsc/documentation/faq.html for trouble > shooting. > [1]PETSC ERROR: Petsc Release Version 3.13.2, Jun 02, 2020 > [1]PETSC ERROR: ./avg_coloredge_deneme_new_vecscatter_arastep on a > arch-linux2-c-debug named localhost.localdomain by edaoktay Mon Jul 6 > 21:09:07 2020 > [1]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ > --download-mpich --download-openblas --download-slepc --download-metis > --download-parmetis --download-chaco --with-X=1 > [1]PETSC ERROR: #1 User provided function() line 0 in unknown file > ./avg_coloredge_deneme_new_vecscatter_arastep[0x402dca] > ./avg_coloredge_deneme_new_vecscatter_arastep[0x402f1e] > ./avg_coloredge_deneme_new_vecscatter_arastep[0x402953] > ./avg_coloredge_deneme_new_vecscatter_arastep[0x406c1c] > application called MPI_Abort(MPI_COMM_WORLD, 50152059) - process 1 > /lib64/libc.so.6(__libc_start_main+0xf1)[0x7f810f1d1431] > ./avg_coloredge_deneme_new_vecscatter_arastep[0x40260a] > ======= Memory map: ======== > 00400000-00409000 r-xp 00000000 fd:02 29230434 > > /home/edaoktay/petsc-3.13.2/arch-linux2-c-debug/share/slepc/examples/src/eda/avg_coloredge_deneme_new_vecscatter_arastep > 00608000-00609000 r--p 00008000 fd:02 29230434 > > /home/edaoktay/petsc-3.13.2/arch-linux2-c-debug/share/slepc/examples/src/eda/avg_coloredge_deneme_new_vecscatter_arastep > 00609000-0060a000 rw-p 00009000 fd:02 29230434 > > /home/edaoktay/petsc-3.13.2/arch-linux2-c-debug/share/slepc/examples/src/eda/avg_coloredge_deneme_new_vecscatter_arastep > 00956000-01100000 rw-p 00000000 00:00 0 > [heap] > 7f810c621000-7f810c6ca000 rw-p 00000000 00:00 0 > 7f810c6ca000-7f810e6ca000 rw-p 00000000 00:00 0 > 7f810e6ca000-7f810e6d4000 r-xp 00000000 fd:00 1974339 > /usr/lib64/libnss_files-2.24.so > 7f810e6d4000-7f810e8d4000 ---p 0000a000 fd:00 1974339 > /usr/lib64/libnss_files-2.24.so > 7f810e8d4000-7f810e8d5000 r--p 0000a000 fd:00 1974339 > /usr/lib64/libnss_files-2.24.so > 7f810e8d5000-7f810e8d6000 rw-p 0000b000 fd:00 1974339 > /usr/lib64/libnss_files-2.24.so > 7f810e8d6000-7f810eb7d000 rw-p 00000000 00:00 0 > 7f810eb7d000-7f810eb7f000 r-xp 00000000 fd:00 1974984 > /usr/lib64/libXau.so.6.0.0 > 7f810eb7f000-7f810ed7f000 ---p 00002000 fd:00 1974984 > /usr/lib64/libXau.so.6.0.0 > 7f810ed7f000-7f810ed80000 r--p 00002000 fd:00 1974984 > /usr/lib64/libXau.so.6.0.0 > 7f810ed80000-7f810ed81000 rw-p 00000000 00:00 0 > 7f810ed81000-7f810ed88000 r-xp 00000000 fd:00 1974440 > /usr/lib64/librt-2.24.so > 7f810ed88000-7f810ef87000 ---p 00007000 fd:00 1974440 > /usr/lib64/librt-2.24.so > 7f810ef87000-7f810ef88000 r--p 00006000 fd:00 1974440 > /usr/lib64/librt-2.24.so > 7f810ef88000-7f810ef89000 rw-p 00007000 fd:00 1974440 > /usr/lib64/librt-2.24.so > 7f810ef89000-7f810efb0000 r-xp 00000000 fd:00 1975953 > /usr/lib64/libxcb.so.1.1.0 > 7f810efb0000-7f810f1af000 ---p 00027000 fd:00 1975953 > /usr/lib64/libxcb.so.1.1.0 > 7f810f1af000-7f810f1b0000 r--p 00026000 fd:00 1975953 > /usr/lib64/libxcb.so.1.1.0 > 7f810f1b0000-7f810f1b1000 rw-p 00027000 fd:00 1975953 > /usr/lib64/libxcb.so.1.1.0 > 7f810f1b1000-7f810f36e000 r-xp 00000000 fd:00 1973798 > /usr/lib64/libc-2.24.so > 7f810f36e000-7f810f56d000 ---p 001bd000 fd:00 1973798 > /usr/lib64/libc-2.24.so > 7f810f56d000-7f810f571000 r--p 001bc000 fd:00 1973798 > /usr/lib64/libc-2.24.so > 7f810f571000-7f810f573000 rw-p 001c0000 fd:00 1973798 > /usr/lib64/libc-2.24.so > 7f810f573000-7f810f577000 rw-p 00000000 00:00 0 > 7f810f577000-7f810f5b6000 r-xp 00000000 fd:00 1975094 > /usr/lib64/libquadmath.so.0.0.0 > 7f810f5b6000-7f810f7b5000 ---p 0003f000 fd:00 1975094 > /usr/lib64/libquadmath.so.0.0.0 > 7f810f7b5000-7f810f7b6000 r--p 0003e000 fd:00 1975094 > /usr/lib64/libquadmath.so.0.0.0 > 7f810f7b6000-7f810f7b7000 rw-p 0003f000 fd:00 1975094 > /usr/lib64/libquadmath.so.0.0.0 > 7f810f7b7000-7f810f7cd000 r-xp 00000000 fd:00 1973274 > /usr/lib64/libgcc_s-6.4.1-20170727.so.1 > 7f810f7cd000-7f810f9cc000 ---p 00016000 fd:00 1973274 > /usr/lib64/libgcc_s-6.4.1-20170727.so.1 > 7f810f9cc000-7f810f9cd000 r--p 00015000 fd:00 1973274 > /usr/lib64/libgcc_s-6.4.1-20170727.so.1 > 7f810f9cd000-7f810f9ce000 rw-p 00016000 fd:00 1973274 > /usr/lib64/libgcc_s-6.4.1-20170727.so.1 > 7f810f9ce000-7f810fafc000 r-xp 00000000 fd:00 1975116 > /usr/lib64/libgfortran.so.3.0.0 > 7f810fafc000-7f810fcfc000 ---p 0012e000 fd:00 1975116 > /usr/lib64/libgfortran.so.3.0.0 > 7f810fcfc000-7f810fcfd000 r--p 0012e000 fd:00 1975116 > /usr/lib64/libgfortran.so.3.0.0 > 7f810fcfd000-7f810fcfe000 rw-p 0012f000 fd:00 1975116 > /usr/lib64/libgfortran.so.3.0.0 > 7f810fcfe000-7f810fcff000 rw-p 00000000 00:00 0 > 7f810fcff000-7f8110168000 r-xp 00000000 fd:02 28457339 > /home/edaoktay/petsc-3.13.2/arch-linux2-c-debug/lib/libmpi.so.12.1.8 > 7f8110168000-7f8110368000 ---p 00469000 fd:02 28457339 > /home/edaoktay/petsc-3.13.2/arch-linux2-c-debug/lib/libmpi.so.12.1.8 > 7f8110368000-7f811037a000 r--p 00469000 fd:02 28457339 > /home/edaoktay/petsc-3.13.2/arch-linux2-c-debug/lib/libmpi.so.12.1.8 > 7f811037a000-7f8110385000 rw-p 0047b000 fd:02 28457339 > /home/edaoktay/petsc-3.13.2/arch-linux2-c-debug/lib/libmpi.so.12.1.8 > 7f8110385000-7f81103c4000 rw-p 00000000 00:00 0 > 7f81103c4000-7f8110435000 r-xp 00000000 fd:02 28457343 > /home/edaoktay/petsc-3.13.2/arch-linux2-c-debug/lib/libmpifort.so.12.1.8 > 7f8110435000-7f8110635000 ---p 00071000 fd:02 28457343 > /home/edaoktay/petsc-3.13.2/arch-linux2-c-debug/lib/libmpifort.so.12.1.8 > 7f8110635000-7f8110639000 r--p 00071000 fd:02 28457343 > /home/edaoktay/petsc-3.13.2/arch-linux2-c-debug/lib/libmpifort.so.12.1.8 > 7f8110639000-7f811063b000 rw-p 00075000 fd:02 28457343 > /home/edaoktay/petsc-3.13.2/arch-linux2-c-debug/lib/libmpifort.so.12.1.8 > 7f811063b000-7f811063e000 r-xp 00000000 fd:00 1973895 > /usr/lib64/libdl-2.24.so > 7f811063e000-7f811083d000 ---p 00003000 fd:00 1973895 > /usr/lib64/libdl-2.24.so > 7f811083d000-7f811083e000 r--p 00002000 fd:00 1973895 > /usr/lib64/libdl-2.24.so > 7f811083e000-7f811083f000 rw-p 00003000 fd:00 1973895 > /usr/lib64/libdl-2.24.so > 7f811083f000-7f81109b7000 r-xp 00000000 fd:00 1974512 > /usr/lib64/libstdc++.so.6.0.22 > 7f81109b7000-7f8110bb7000 ---p 00178000 fd:00 1974512 > /usr/lib64/libstdc++.so.6.0.22 > 7f8110bb7000-7f8110bc1000 r--p 00178000 fd:00 1974512 > /usr/lib64/libstdc++.so.6.0.22 > 7f8110bc1000-7f8110bc3000 rw-p 00182000 fd:00 1974512 > /usr/lib64/libstdc++.so.6.0.22 > 7f8110bc3000-7f8110bc7000 rw-p 00000000 00:00 0 > 7f8110bc7000-7f8110ccf000 r-xp 00000000 fd:00 1973951 > /usr/lib64/libm-2.24.so > 7f8110ccf000-7f8110ece000 ---p 00108000 fd:00 1973951 > /usr/lib64/libm-2.24.so > 7f8110ece000-7f8110ecf000 r--p 00107000 fd:00 1973951 > /usr/lib64/libm-2.24.so > 7f8110ecf000-7f8110ed0000 rw-p 00108000 fd:00 1973951 > /usr/lib64/libm-2.24.so > 7f8110ed0000-7f8110f60000 r-xp 00000000 fd:02 28457370 > /home/edaoktay/petsc-3.13.2/arch-linux2-c-debug/lib/libmetis.so > 7f8110f60000-7f811115f000 ---p 00090000 fd:02 28457370 > /home/edaoktay/petsc-3.13.2/arch-linux2-c-debug/lib/libmetis.so > 7f811115f000-7f8111160000 r--p 0008f000 fd:02 28457370 > /home/edaoktay/petsc-3.13.2/arch-linux2-c-debug/lib/libmetis.so > 7f8111160000-7f8111161000 rw-p 00090000 fd:02 28457370 > /home/edaoktay/petsc-3.13.2/arch-linux2-c-debug/lib/libmetis.so > 7f8111161000-7f8111169000 rw-p 00000000 00:00 0 > 7f8111169000-7f81111bf000 r-xp 00000000 fd:02 28457373 > /home/edaoktay/petsc-3.13.2/arch-linux2-c-debug/lib/libparmetis.so > 7f81111bf000-7f81113bf000 ---p 00056000 fd:02 28457373 > /home/edaoktay/petsc-3.13.2/arch-linux2-c-debug/lib/libparmetis.so > 7f81113bf000-7f81113c0000 r--p 00056000 fd:02 28457373 > /home/edaoktay/petsc-3.13.2/arch-linux2-c-debug/lib/libparmetis.so > 7f81113c0000-7f81113c1000 rw-p 00057000 fd:02 28457373 > /home/edaoktay/petsc-3.13.2/arch-linux2-c-debug/lib/libparmetis.so > 7f81113c1000-7f81114f9000 r-xp 00000000 fd:00 1973763 > /usr/lib64/libX11.so.6.3.0 > 7f81114f9000-7f81116f9000 ---p 00138000 fd:00 1973763 > /usr/lib64/libX11.so.6.3.0 > 7f81116f9000-7f81116fc000 r--p 00138000 fd:00 1973763 > /usr/lib64/libX11.so.6.3.0 > 7f81116fc000-7f8111700000 rw-p 0013b000 fd:00 1973763 > /usr/lib64/libX11.so.6.3.0 > 7f8111700000-7f8111718000 r-xp 00000000 fd:00 1974340 > /usr/lib64/libpthread-2.24.so > 7f8111718000-7f8111918000 ---p 00018000 fd:00 1974340 > /usr/lib64/libpthread-2.24.so > 7f8111918000-7f8111919000 r--p 00018000 fd:00 1974340 > /usr/lib64/libpthread-2.24.so > 7f8111919000-7f811191a000 rw-p 00019000 fd:00 1974340 > /usr/lib64/libpthread-2.24.so > 7f811191a000-7f811191e000 rw-p 00000000 00:00 0 > 7f811191e000-7f81124ff000 r-xp 00000000 fd:02 28457384 > /home/edaoktay/petsc-3.13.2/arch-linux2-c-debug/lib/ > libopenblas_haswell-r0.3.8.dev.so > 7f81124ff000-7f81126ff000 ---p 00be1000 fd:02 28457384 > /home/edaoktay/petsc-3.13.2/arch-linux2-c-debug/lib/ > libopenblas_haswell-r0.3.8.dev.so > 7f81126ff000-7f8112703000 r--p 00be1000 fd:02 28457384 > /home/edaoktay/petsc-3.13.2/arch-linux2-c-debug/lib/ > libopenblas_haswell-r0.3.8.dev.so > 7f8112703000-7f811270d000 rw-p 00be5000 fd:02 28457384 > /home/edaoktay/petsc-3.13.2/arch-linux2-c-debug/lib/ > libopenblas_haswell-r0.3.8.dev.so > 7f811270d000-7f811270e000 rw-p 00000000 00:00 0 > 7f811270e000-7f81147c2000 r-xp 00000000 fd:02 28457400 > /home/edaoktay/petsc-3.13.2/arch-linux2-c-debug/lib/libpetsc.so.3.13.2 > 7f81147c2000-7f81149c1000 ---p 020b4000 fd:02 28457400 > /home/edaoktay/petsc-3.13.2/arch-linux2-c-debug/lib/libpetsc.so.3.13.2 > 7f81149c1000-7f81149c7000 r--p 020b3000 fd:02 28457400 > /home/edaoktay/petsc-3.13.2/arch-linux2-c-debug/lib/libpetsc.so.3.13.2 > 7f81149c7000-7f81149df000 rw-p 020b9000 fd:02 28457400 > /home/edaoktay/petsc-3.13.2/arch-linux2-c-debug/lib/libpetsc.so.3.13.2 > 7f81149df000-7f8114bdf000 rw-p 00000000 00:00 0 > 7f8114bdf000-7f8115220000 r-xp 00000000 fd:02 28457416 > /home/edaoktay/petsc-3.13.2/arch-linux2-c-debug/lib/libslepc.so.3.13.2 > 7f8115220000-7f8115420000 ---p 00641000 fd:02 28457416 > /home/edaoktay/petsc-3.13.2/arch-linux2-c-debug/lib/libslepc.so.3.13.2 > 7f8115420000-7f8115421000 r--p 00641000 fd:02 28457416 > /home/edaoktay/petsc-3.13.2/arch-linux2-c-debug/lib/libslepc.so.3.13.2 > 7f8115421000-7f8115425000 rw-p 00642000 fd:02 28457416 > /home/edaoktay/petsc-3.13.2/arch-linux2-c-debug/lib/libslepc.so.3.13.2 > 7f8115425000-7f811544b000 r-xp 00000000 fd:00 1976838 > /usr/lib64/ld-2.24.so > 7f81154d3000-7f8115621000 rw-p 00000000 00:00 0 > 7f8115647000-7f811564a000 rw-p 00000000 00:00 0 > 7f811564a000-7f811564b000 r--p 00025000 fd:00 1976838 > /usr/lib64/ld-2.24.so > 7f811564b000-7f811564c000 rw-p 00026000 fd:00 1976838 > /usr/lib64/ld-2.24.so > 7f811564c000-7f811564d000 rw-p 00000000 00:00 0 > 7ffc79b33000-7ffc79b55000 rw-p 00000000 00:00 0 > [stack] > 7ffc79b67000-7ffc79b6a000 r--p 00000000 00:00 0 > [vvar] > 7ffc79b6a000-7ffc79b6c000 r-xp 00000000 00:00 0 > [vdso] > ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0 > [vsyscall] > > > I created and destroyed every vector, matrix and EPS in the loop to > make sure that there won't be any segmentation fault but still I am > getting some. Why am I getting this type of error? > > Thanks so much! > > Eda > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at petsc.dev Mon Jul 6 14:51:29 2020 From: bsmith at petsc.dev (Barry Smith) Date: Mon, 6 Jul 2020 14:51:29 -0500 Subject: [petsc-users] /lib64/libc.so.6(cfree+0x1f8)[0x7f810f237438] Signal received In-Reply-To: References: Message-ID: Where is your loop? In a bash script or inside your compiled program? If inside the program I assume it is outside of PetscInitialize()? Is it outside of MPI_Init()? It can't be because MPI_Init() can only be called once. If inside the problem you can use -start_in_debugger to get the true location of the crash, if in a bash script you can use -on_error_attach_debugger Barry > On Jul 6, 2020, at 1:14 PM, Eda Oktay wrote: > > Hello, > > I am trying to run my codes 100 times and then take the average > elapsed time. Although my program runs correctly for single time, when > I put them into loop, I am getting the following error: > > *** Error in `./avg_coloredge_deneme_new_vecscatter_arastep': > munmap_chunk(): invalid pointer: 0x0000000000b0cbc0 *** > [1]PETSC ERROR: > ------------------------------------------------------------------------ > [1]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, > probably memory access out of range > [1]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger > [1]PETSC ERROR: or see > https://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind > [1]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac > OS X to find memory corruption errors > [1]PETSC ERROR: likely location of problem given in stack below > [1]PETSC ERROR: --------------------- Stack Frames > ------------------------------------ > [1]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, > [1]PETSC ERROR: INSTEAD the line number of the start of the function > [1]PETSC ERROR: is given. > ======= Backtrace: ========= > [1]PETSC ERROR: /lib64/libc.so.6(+0x791eb)[0x7f810f22a1eb] > --------------------- Error Message > -------------------------------------------------------------- > [1]PETSC ERROR: /lib64/libc.so.6(cfree+0x1f8)[0x7f810f237438] > Signal received > [1]PETSC ERROR: See > https://www.mcs.anl.gov/petsc/documentation/faq.html for trouble > shooting. > [1]PETSC ERROR: Petsc Release Version 3.13.2, Jun 02, 2020 > [1]PETSC ERROR: ./avg_coloredge_deneme_new_vecscatter_arastep on a > arch-linux2-c-debug named localhost.localdomain by edaoktay Mon Jul 6 > 21:09:07 2020 > [1]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ > --download-mpich --download-openblas --download-slepc --download-metis > --download-parmetis --download-chaco --with-X=1 > [1]PETSC ERROR: #1 User provided function() line 0 in unknown file > ./avg_coloredge_deneme_new_vecscatter_arastep[0x402dca] > ./avg_coloredge_deneme_new_vecscatter_arastep[0x402f1e] > ./avg_coloredge_deneme_new_vecscatter_arastep[0x402953] > ./avg_coloredge_deneme_new_vecscatter_arastep[0x406c1c] > application called MPI_Abort(MPI_COMM_WORLD, 50152059) - process 1 > /lib64/libc.so.6(__libc_start_main+0xf1)[0x7f810f1d1431] > ./avg_coloredge_deneme_new_vecscatter_arastep[0x40260a] > ======= Memory map: ======== > 00400000-00409000 r-xp 00000000 fd:02 29230434 > /home/edaoktay/petsc-3.13.2/arch-linux2-c-debug/share/slepc/examples/src/eda/avg_coloredge_deneme_new_vecscatter_arastep > 00608000-00609000 r--p 00008000 fd:02 29230434 > /home/edaoktay/petsc-3.13.2/arch-linux2-c-debug/share/slepc/examples/src/eda/avg_coloredge_deneme_new_vecscatter_arastep > 00609000-0060a000 rw-p 00009000 fd:02 29230434 > /home/edaoktay/petsc-3.13.2/arch-linux2-c-debug/share/slepc/examples/src/eda/avg_coloredge_deneme_new_vecscatter_arastep > 00956000-01100000 rw-p 00000000 00:00 0 [heap] > 7f810c621000-7f810c6ca000 rw-p 00000000 00:00 0 > 7f810c6ca000-7f810e6ca000 rw-p 00000000 00:00 0 > 7f810e6ca000-7f810e6d4000 r-xp 00000000 fd:00 1974339 > /usr/lib64/libnss_files-2.24.so > 7f810e6d4000-7f810e8d4000 ---p 0000a000 fd:00 1974339 > /usr/lib64/libnss_files-2.24.so > 7f810e8d4000-7f810e8d5000 r--p 0000a000 fd:00 1974339 > /usr/lib64/libnss_files-2.24.so > 7f810e8d5000-7f810e8d6000 rw-p 0000b000 fd:00 1974339 > /usr/lib64/libnss_files-2.24.so > 7f810e8d6000-7f810eb7d000 rw-p 00000000 00:00 0 > 7f810eb7d000-7f810eb7f000 r-xp 00000000 fd:00 1974984 > /usr/lib64/libXau.so.6.0.0 > 7f810eb7f000-7f810ed7f000 ---p 00002000 fd:00 1974984 > /usr/lib64/libXau.so.6.0.0 > 7f810ed7f000-7f810ed80000 r--p 00002000 fd:00 1974984 > /usr/lib64/libXau.so.6.0.0 > 7f810ed80000-7f810ed81000 rw-p 00000000 00:00 0 > 7f810ed81000-7f810ed88000 r-xp 00000000 fd:00 1974440 > /usr/lib64/librt-2.24.so > 7f810ed88000-7f810ef87000 ---p 00007000 fd:00 1974440 > /usr/lib64/librt-2.24.so > 7f810ef87000-7f810ef88000 r--p 00006000 fd:00 1974440 > /usr/lib64/librt-2.24.so > 7f810ef88000-7f810ef89000 rw-p 00007000 fd:00 1974440 > /usr/lib64/librt-2.24.so > 7f810ef89000-7f810efb0000 r-xp 00000000 fd:00 1975953 > /usr/lib64/libxcb.so.1.1.0 > 7f810efb0000-7f810f1af000 ---p 00027000 fd:00 1975953 > /usr/lib64/libxcb.so.1.1.0 > 7f810f1af000-7f810f1b0000 r--p 00026000 fd:00 1975953 > /usr/lib64/libxcb.so.1.1.0 > 7f810f1b0000-7f810f1b1000 rw-p 00027000 fd:00 1975953 > /usr/lib64/libxcb.so.1.1.0 > 7f810f1b1000-7f810f36e000 r-xp 00000000 fd:00 1973798 > /usr/lib64/libc-2.24.so > 7f810f36e000-7f810f56d000 ---p 001bd000 fd:00 1973798 > /usr/lib64/libc-2.24.so > 7f810f56d000-7f810f571000 r--p 001bc000 fd:00 1973798 > /usr/lib64/libc-2.24.so > 7f810f571000-7f810f573000 rw-p 001c0000 fd:00 1973798 > /usr/lib64/libc-2.24.so > 7f810f573000-7f810f577000 rw-p 00000000 00:00 0 > 7f810f577000-7f810f5b6000 r-xp 00000000 fd:00 1975094 > /usr/lib64/libquadmath.so.0.0.0 > 7f810f5b6000-7f810f7b5000 ---p 0003f000 fd:00 1975094 > /usr/lib64/libquadmath.so.0.0.0 > 7f810f7b5000-7f810f7b6000 r--p 0003e000 fd:00 1975094 > /usr/lib64/libquadmath.so.0.0.0 > 7f810f7b6000-7f810f7b7000 rw-p 0003f000 fd:00 1975094 > /usr/lib64/libquadmath.so.0.0.0 > 7f810f7b7000-7f810f7cd000 r-xp 00000000 fd:00 1973274 > /usr/lib64/libgcc_s-6.4.1-20170727.so.1 > 7f810f7cd000-7f810f9cc000 ---p 00016000 fd:00 1973274 > /usr/lib64/libgcc_s-6.4.1-20170727.so.1 > 7f810f9cc000-7f810f9cd000 r--p 00015000 fd:00 1973274 > /usr/lib64/libgcc_s-6.4.1-20170727.so.1 > 7f810f9cd000-7f810f9ce000 rw-p 00016000 fd:00 1973274 > /usr/lib64/libgcc_s-6.4.1-20170727.so.1 > 7f810f9ce000-7f810fafc000 r-xp 00000000 fd:00 1975116 > /usr/lib64/libgfortran.so.3.0.0 > 7f810fafc000-7f810fcfc000 ---p 0012e000 fd:00 1975116 > /usr/lib64/libgfortran.so.3.0.0 > 7f810fcfc000-7f810fcfd000 r--p 0012e000 fd:00 1975116 > /usr/lib64/libgfortran.so.3.0.0 > 7f810fcfd000-7f810fcfe000 rw-p 0012f000 fd:00 1975116 > /usr/lib64/libgfortran.so.3.0.0 > 7f810fcfe000-7f810fcff000 rw-p 00000000 00:00 0 > 7f810fcff000-7f8110168000 r-xp 00000000 fd:02 28457339 > /home/edaoktay/petsc-3.13.2/arch-linux2-c-debug/lib/libmpi.so.12.1.8 > 7f8110168000-7f8110368000 ---p 00469000 fd:02 28457339 > /home/edaoktay/petsc-3.13.2/arch-linux2-c-debug/lib/libmpi.so.12.1.8 > 7f8110368000-7f811037a000 r--p 00469000 fd:02 28457339 > /home/edaoktay/petsc-3.13.2/arch-linux2-c-debug/lib/libmpi.so.12.1.8 > 7f811037a000-7f8110385000 rw-p 0047b000 fd:02 28457339 > /home/edaoktay/petsc-3.13.2/arch-linux2-c-debug/lib/libmpi.so.12.1.8 > 7f8110385000-7f81103c4000 rw-p 00000000 00:00 0 > 7f81103c4000-7f8110435000 r-xp 00000000 fd:02 28457343 > /home/edaoktay/petsc-3.13.2/arch-linux2-c-debug/lib/libmpifort.so.12.1.8 > 7f8110435000-7f8110635000 ---p 00071000 fd:02 28457343 > /home/edaoktay/petsc-3.13.2/arch-linux2-c-debug/lib/libmpifort.so.12.1.8 > 7f8110635000-7f8110639000 r--p 00071000 fd:02 28457343 > /home/edaoktay/petsc-3.13.2/arch-linux2-c-debug/lib/libmpifort.so.12.1.8 > 7f8110639000-7f811063b000 rw-p 00075000 fd:02 28457343 > /home/edaoktay/petsc-3.13.2/arch-linux2-c-debug/lib/libmpifort.so.12.1.8 > 7f811063b000-7f811063e000 r-xp 00000000 fd:00 1973895 > /usr/lib64/libdl-2.24.so > 7f811063e000-7f811083d000 ---p 00003000 fd:00 1973895 > /usr/lib64/libdl-2.24.so > 7f811083d000-7f811083e000 r--p 00002000 fd:00 1973895 > /usr/lib64/libdl-2.24.so > 7f811083e000-7f811083f000 rw-p 00003000 fd:00 1973895 > /usr/lib64/libdl-2.24.so > 7f811083f000-7f81109b7000 r-xp 00000000 fd:00 1974512 > /usr/lib64/libstdc++.so.6.0.22 > 7f81109b7000-7f8110bb7000 ---p 00178000 fd:00 1974512 > /usr/lib64/libstdc++.so.6.0.22 > 7f8110bb7000-7f8110bc1000 r--p 00178000 fd:00 1974512 > /usr/lib64/libstdc++.so.6.0.22 > 7f8110bc1000-7f8110bc3000 rw-p 00182000 fd:00 1974512 > /usr/lib64/libstdc++.so.6.0.22 > 7f8110bc3000-7f8110bc7000 rw-p 00000000 00:00 0 > 7f8110bc7000-7f8110ccf000 r-xp 00000000 fd:00 1973951 > /usr/lib64/libm-2.24.so > 7f8110ccf000-7f8110ece000 ---p 00108000 fd:00 1973951 > /usr/lib64/libm-2.24.so > 7f8110ece000-7f8110ecf000 r--p 00107000 fd:00 1973951 > /usr/lib64/libm-2.24.so > 7f8110ecf000-7f8110ed0000 rw-p 00108000 fd:00 1973951 > /usr/lib64/libm-2.24.so > 7f8110ed0000-7f8110f60000 r-xp 00000000 fd:02 28457370 > /home/edaoktay/petsc-3.13.2/arch-linux2-c-debug/lib/libmetis.so > 7f8110f60000-7f811115f000 ---p 00090000 fd:02 28457370 > /home/edaoktay/petsc-3.13.2/arch-linux2-c-debug/lib/libmetis.so > 7f811115f000-7f8111160000 r--p 0008f000 fd:02 28457370 > /home/edaoktay/petsc-3.13.2/arch-linux2-c-debug/lib/libmetis.so > 7f8111160000-7f8111161000 rw-p 00090000 fd:02 28457370 > /home/edaoktay/petsc-3.13.2/arch-linux2-c-debug/lib/libmetis.so > 7f8111161000-7f8111169000 rw-p 00000000 00:00 0 > 7f8111169000-7f81111bf000 r-xp 00000000 fd:02 28457373 > /home/edaoktay/petsc-3.13.2/arch-linux2-c-debug/lib/libparmetis.so > 7f81111bf000-7f81113bf000 ---p 00056000 fd:02 28457373 > /home/edaoktay/petsc-3.13.2/arch-linux2-c-debug/lib/libparmetis.so > 7f81113bf000-7f81113c0000 r--p 00056000 fd:02 28457373 > /home/edaoktay/petsc-3.13.2/arch-linux2-c-debug/lib/libparmetis.so > 7f81113c0000-7f81113c1000 rw-p 00057000 fd:02 28457373 > /home/edaoktay/petsc-3.13.2/arch-linux2-c-debug/lib/libparmetis.so > 7f81113c1000-7f81114f9000 r-xp 00000000 fd:00 1973763 > /usr/lib64/libX11.so.6.3.0 > 7f81114f9000-7f81116f9000 ---p 00138000 fd:00 1973763 > /usr/lib64/libX11.so.6.3.0 > 7f81116f9000-7f81116fc000 r--p 00138000 fd:00 1973763 > /usr/lib64/libX11.so.6.3.0 > 7f81116fc000-7f8111700000 rw-p 0013b000 fd:00 1973763 > /usr/lib64/libX11.so.6.3.0 > 7f8111700000-7f8111718000 r-xp 00000000 fd:00 1974340 > /usr/lib64/libpthread-2.24.so > 7f8111718000-7f8111918000 ---p 00018000 fd:00 1974340 > /usr/lib64/libpthread-2.24.so > 7f8111918000-7f8111919000 r--p 00018000 fd:00 1974340 > /usr/lib64/libpthread-2.24.so > 7f8111919000-7f811191a000 rw-p 00019000 fd:00 1974340 > /usr/lib64/libpthread-2.24.so > 7f811191a000-7f811191e000 rw-p 00000000 00:00 0 > 7f811191e000-7f81124ff000 r-xp 00000000 fd:02 28457384 > /home/edaoktay/petsc-3.13.2/arch-linux2-c-debug/lib/libopenblas_haswell-r0.3.8.dev.so > 7f81124ff000-7f81126ff000 ---p 00be1000 fd:02 28457384 > /home/edaoktay/petsc-3.13.2/arch-linux2-c-debug/lib/libopenblas_haswell-r0.3.8.dev.so > 7f81126ff000-7f8112703000 r--p 00be1000 fd:02 28457384 > /home/edaoktay/petsc-3.13.2/arch-linux2-c-debug/lib/libopenblas_haswell-r0.3.8.dev.so > 7f8112703000-7f811270d000 rw-p 00be5000 fd:02 28457384 > /home/edaoktay/petsc-3.13.2/arch-linux2-c-debug/lib/libopenblas_haswell-r0.3.8.dev.so > 7f811270d000-7f811270e000 rw-p 00000000 00:00 0 > 7f811270e000-7f81147c2000 r-xp 00000000 fd:02 28457400 > /home/edaoktay/petsc-3.13.2/arch-linux2-c-debug/lib/libpetsc.so.3.13.2 > 7f81147c2000-7f81149c1000 ---p 020b4000 fd:02 28457400 > /home/edaoktay/petsc-3.13.2/arch-linux2-c-debug/lib/libpetsc.so.3.13.2 > 7f81149c1000-7f81149c7000 r--p 020b3000 fd:02 28457400 > /home/edaoktay/petsc-3.13.2/arch-linux2-c-debug/lib/libpetsc.so.3.13.2 > 7f81149c7000-7f81149df000 rw-p 020b9000 fd:02 28457400 > /home/edaoktay/petsc-3.13.2/arch-linux2-c-debug/lib/libpetsc.so.3.13.2 > 7f81149df000-7f8114bdf000 rw-p 00000000 00:00 0 > 7f8114bdf000-7f8115220000 r-xp 00000000 fd:02 28457416 > /home/edaoktay/petsc-3.13.2/arch-linux2-c-debug/lib/libslepc.so.3.13.2 > 7f8115220000-7f8115420000 ---p 00641000 fd:02 28457416 > /home/edaoktay/petsc-3.13.2/arch-linux2-c-debug/lib/libslepc.so.3.13.2 > 7f8115420000-7f8115421000 r--p 00641000 fd:02 28457416 > /home/edaoktay/petsc-3.13.2/arch-linux2-c-debug/lib/libslepc.so.3.13.2 > 7f8115421000-7f8115425000 rw-p 00642000 fd:02 28457416 > /home/edaoktay/petsc-3.13.2/arch-linux2-c-debug/lib/libslepc.so.3.13.2 > 7f8115425000-7f811544b000 r-xp 00000000 fd:00 1976838 > /usr/lib64/ld-2.24.so > 7f81154d3000-7f8115621000 rw-p 00000000 00:00 0 > 7f8115647000-7f811564a000 rw-p 00000000 00:00 0 > 7f811564a000-7f811564b000 r--p 00025000 fd:00 1976838 > /usr/lib64/ld-2.24.so > 7f811564b000-7f811564c000 rw-p 00026000 fd:00 1976838 > /usr/lib64/ld-2.24.so > 7f811564c000-7f811564d000 rw-p 00000000 00:00 0 > 7ffc79b33000-7ffc79b55000 rw-p 00000000 00:00 0 [stack] > 7ffc79b67000-7ffc79b6a000 r--p 00000000 00:00 0 [vvar] > 7ffc79b6a000-7ffc79b6c000 r-xp 00000000 00:00 0 [vdso] > ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0 > [vsyscall] > > > I created and destroyed every vector, matrix and EPS in the loop to > make sure that there won't be any segmentation fault but still I am > getting some. Why am I getting this type of error? > > Thanks so much! > > Eda From eda.oktay at metu.edu.tr Mon Jul 6 15:05:03 2020 From: eda.oktay at metu.edu.tr (Eda Oktay) Date: Mon, 6 Jul 2020 23:05:03 +0300 Subject: [petsc-users] /lib64/libc.so.6(cfree+0x1f8)[0x7f810f237438] Signal received In-Reply-To: References: Message-ID: I tried for (count=1;count<2;++count), it is working. When I raised it to 3, it started giving the same error. My loop is inside my compiled program, not bash. I started the loop after PetscInitialize and finished it before PetscFinalize. I am not using MPI_Init. Thanks! On Mon, Jul 6, 2020, 10:51 PM Barry Smith wrote: > > Where is your loop? In a bash script or inside your compiled program? > > If inside the program I assume it is outside of PetscInitialize()? > > Is it outside of MPI_Init()? It can't be because MPI_Init() can only be > called once. > > If inside the problem you can use -start_in_debugger to get the true > location of the crash, if in a bash script you can use > -on_error_attach_debugger > > Barry > > > > On Jul 6, 2020, at 1:14 PM, Eda Oktay wrote: > > > > Hello, > > > > I am trying to run my codes 100 times and then take the average > > elapsed time. Although my program runs correctly for single time, when > > I put them into loop, I am getting the following error: > > > > *** Error in `./avg_coloredge_deneme_new_vecscatter_arastep': > > munmap_chunk(): invalid pointer: 0x0000000000b0cbc0 *** > > [1]PETSC ERROR: > > ------------------------------------------------------------------------ > > [1]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, > > probably memory access out of range > > [1]PETSC ERROR: Try option -start_in_debugger or > -on_error_attach_debugger > > [1]PETSC ERROR: or see > > https://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind > > [1]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac > > OS X to find memory corruption errors > > [1]PETSC ERROR: likely location of problem given in stack below > > [1]PETSC ERROR: --------------------- Stack Frames > > ------------------------------------ > > [1]PETSC ERROR: Note: The EXACT line numbers in the stack are not > available, > > [1]PETSC ERROR: INSTEAD the line number of the start of the > function > > [1]PETSC ERROR: is given. > > ======= Backtrace: ========= > > [1]PETSC ERROR: /lib64/libc.so.6(+0x791eb)[0x7f810f22a1eb] > > --------------------- Error Message > > -------------------------------------------------------------- > > [1]PETSC ERROR: /lib64/libc.so.6(cfree+0x1f8)[0x7f810f237438] > > Signal received > > [1]PETSC ERROR: See > > https://www.mcs.anl.gov/petsc/documentation/faq.html for trouble > > shooting. > > [1]PETSC ERROR: Petsc Release Version 3.13.2, Jun 02, 2020 > > [1]PETSC ERROR: ./avg_coloredge_deneme_new_vecscatter_arastep on a > > arch-linux2-c-debug named localhost.localdomain by edaoktay Mon Jul 6 > > 21:09:07 2020 > > [1]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ > > --download-mpich --download-openblas --download-slepc --download-metis > > --download-parmetis --download-chaco --with-X=1 > > [1]PETSC ERROR: #1 User provided function() line 0 in unknown file > > ./avg_coloredge_deneme_new_vecscatter_arastep[0x402dca] > > ./avg_coloredge_deneme_new_vecscatter_arastep[0x402f1e] > > ./avg_coloredge_deneme_new_vecscatter_arastep[0x402953] > > ./avg_coloredge_deneme_new_vecscatter_arastep[0x406c1c] > > application called MPI_Abort(MPI_COMM_WORLD, 50152059) - process 1 > > /lib64/libc.so.6(__libc_start_main+0xf1)[0x7f810f1d1431] > > ./avg_coloredge_deneme_new_vecscatter_arastep[0x40260a] > > ======= Memory map: ======== > > 00400000-00409000 r-xp 00000000 fd:02 29230434 > > > /home/edaoktay/petsc-3.13.2/arch-linux2-c-debug/share/slepc/examples/src/eda/avg_coloredge_deneme_new_vecscatter_arastep > > 00608000-00609000 r--p 00008000 fd:02 29230434 > > > /home/edaoktay/petsc-3.13.2/arch-linux2-c-debug/share/slepc/examples/src/eda/avg_coloredge_deneme_new_vecscatter_arastep > > 00609000-0060a000 rw-p 00009000 fd:02 29230434 > > > /home/edaoktay/petsc-3.13.2/arch-linux2-c-debug/share/slepc/examples/src/eda/avg_coloredge_deneme_new_vecscatter_arastep > > 00956000-01100000 rw-p 00000000 00:00 0 > [heap] > > 7f810c621000-7f810c6ca000 rw-p 00000000 00:00 0 > > 7f810c6ca000-7f810e6ca000 rw-p 00000000 00:00 0 > > 7f810e6ca000-7f810e6d4000 r-xp 00000000 fd:00 1974339 > > /usr/lib64/libnss_files-2.24.so > > 7f810e6d4000-7f810e8d4000 ---p 0000a000 fd:00 1974339 > > /usr/lib64/libnss_files-2.24.so > > 7f810e8d4000-7f810e8d5000 r--p 0000a000 fd:00 1974339 > > /usr/lib64/libnss_files-2.24.so > > 7f810e8d5000-7f810e8d6000 rw-p 0000b000 fd:00 1974339 > > /usr/lib64/libnss_files-2.24.so > > 7f810e8d6000-7f810eb7d000 rw-p 00000000 00:00 0 > > 7f810eb7d000-7f810eb7f000 r-xp 00000000 fd:00 1974984 > > /usr/lib64/libXau.so.6.0.0 > > 7f810eb7f000-7f810ed7f000 ---p 00002000 fd:00 1974984 > > /usr/lib64/libXau.so.6.0.0 > > 7f810ed7f000-7f810ed80000 r--p 00002000 fd:00 1974984 > > /usr/lib64/libXau.so.6.0.0 > > 7f810ed80000-7f810ed81000 rw-p 00000000 00:00 0 > > 7f810ed81000-7f810ed88000 r-xp 00000000 fd:00 1974440 > > /usr/lib64/librt-2.24.so > > 7f810ed88000-7f810ef87000 ---p 00007000 fd:00 1974440 > > /usr/lib64/librt-2.24.so > > 7f810ef87000-7f810ef88000 r--p 00006000 fd:00 1974440 > > /usr/lib64/librt-2.24.so > > 7f810ef88000-7f810ef89000 rw-p 00007000 fd:00 1974440 > > /usr/lib64/librt-2.24.so > > 7f810ef89000-7f810efb0000 r-xp 00000000 fd:00 1975953 > > /usr/lib64/libxcb.so.1.1.0 > > 7f810efb0000-7f810f1af000 ---p 00027000 fd:00 1975953 > > /usr/lib64/libxcb.so.1.1.0 > > 7f810f1af000-7f810f1b0000 r--p 00026000 fd:00 1975953 > > /usr/lib64/libxcb.so.1.1.0 > > 7f810f1b0000-7f810f1b1000 rw-p 00027000 fd:00 1975953 > > /usr/lib64/libxcb.so.1.1.0 > > 7f810f1b1000-7f810f36e000 r-xp 00000000 fd:00 1973798 > > /usr/lib64/libc-2.24.so > > 7f810f36e000-7f810f56d000 ---p 001bd000 fd:00 1973798 > > /usr/lib64/libc-2.24.so > > 7f810f56d000-7f810f571000 r--p 001bc000 fd:00 1973798 > > /usr/lib64/libc-2.24.so > > 7f810f571000-7f810f573000 rw-p 001c0000 fd:00 1973798 > > /usr/lib64/libc-2.24.so > > 7f810f573000-7f810f577000 rw-p 00000000 00:00 0 > > 7f810f577000-7f810f5b6000 r-xp 00000000 fd:00 1975094 > > /usr/lib64/libquadmath.so.0.0.0 > > 7f810f5b6000-7f810f7b5000 ---p 0003f000 fd:00 1975094 > > /usr/lib64/libquadmath.so.0.0.0 > > 7f810f7b5000-7f810f7b6000 r--p 0003e000 fd:00 1975094 > > /usr/lib64/libquadmath.so.0.0.0 > > 7f810f7b6000-7f810f7b7000 rw-p 0003f000 fd:00 1975094 > > /usr/lib64/libquadmath.so.0.0.0 > > 7f810f7b7000-7f810f7cd000 r-xp 00000000 fd:00 1973274 > > /usr/lib64/libgcc_s-6.4.1-20170727.so.1 > > 7f810f7cd000-7f810f9cc000 ---p 00016000 fd:00 1973274 > > /usr/lib64/libgcc_s-6.4.1-20170727.so.1 > > 7f810f9cc000-7f810f9cd000 r--p 00015000 fd:00 1973274 > > /usr/lib64/libgcc_s-6.4.1-20170727.so.1 > > 7f810f9cd000-7f810f9ce000 rw-p 00016000 fd:00 1973274 > > /usr/lib64/libgcc_s-6.4.1-20170727.so.1 > > 7f810f9ce000-7f810fafc000 r-xp 00000000 fd:00 1975116 > > /usr/lib64/libgfortran.so.3.0.0 > > 7f810fafc000-7f810fcfc000 ---p 0012e000 fd:00 1975116 > > /usr/lib64/libgfortran.so.3.0.0 > > 7f810fcfc000-7f810fcfd000 r--p 0012e000 fd:00 1975116 > > /usr/lib64/libgfortran.so.3.0.0 > > 7f810fcfd000-7f810fcfe000 rw-p 0012f000 fd:00 1975116 > > /usr/lib64/libgfortran.so.3.0.0 > > 7f810fcfe000-7f810fcff000 rw-p 00000000 00:00 0 > > 7f810fcff000-7f8110168000 r-xp 00000000 fd:02 28457339 > > /home/edaoktay/petsc-3.13.2/arch-linux2-c-debug/lib/libmpi.so.12.1.8 > > 7f8110168000-7f8110368000 ---p 00469000 fd:02 28457339 > > /home/edaoktay/petsc-3.13.2/arch-linux2-c-debug/lib/libmpi.so.12.1.8 > > 7f8110368000-7f811037a000 r--p 00469000 fd:02 28457339 > > /home/edaoktay/petsc-3.13.2/arch-linux2-c-debug/lib/libmpi.so.12.1.8 > > 7f811037a000-7f8110385000 rw-p 0047b000 fd:02 28457339 > > /home/edaoktay/petsc-3.13.2/arch-linux2-c-debug/lib/libmpi.so.12.1.8 > > 7f8110385000-7f81103c4000 rw-p 00000000 00:00 0 > > 7f81103c4000-7f8110435000 r-xp 00000000 fd:02 28457343 > > /home/edaoktay/petsc-3.13.2/arch-linux2-c-debug/lib/libmpifort.so.12.1.8 > > 7f8110435000-7f8110635000 ---p 00071000 fd:02 28457343 > > /home/edaoktay/petsc-3.13.2/arch-linux2-c-debug/lib/libmpifort.so.12.1.8 > > 7f8110635000-7f8110639000 r--p 00071000 fd:02 28457343 > > /home/edaoktay/petsc-3.13.2/arch-linux2-c-debug/lib/libmpifort.so.12.1.8 > > 7f8110639000-7f811063b000 rw-p 00075000 fd:02 28457343 > > /home/edaoktay/petsc-3.13.2/arch-linux2-c-debug/lib/libmpifort.so.12.1.8 > > 7f811063b000-7f811063e000 r-xp 00000000 fd:00 1973895 > > /usr/lib64/libdl-2.24.so > > 7f811063e000-7f811083d000 ---p 00003000 fd:00 1973895 > > /usr/lib64/libdl-2.24.so > > 7f811083d000-7f811083e000 r--p 00002000 fd:00 1973895 > > /usr/lib64/libdl-2.24.so > > 7f811083e000-7f811083f000 rw-p 00003000 fd:00 1973895 > > /usr/lib64/libdl-2.24.so > > 7f811083f000-7f81109b7000 r-xp 00000000 fd:00 1974512 > > /usr/lib64/libstdc++.so.6.0.22 > > 7f81109b7000-7f8110bb7000 ---p 00178000 fd:00 1974512 > > /usr/lib64/libstdc++.so.6.0.22 > > 7f8110bb7000-7f8110bc1000 r--p 00178000 fd:00 1974512 > > /usr/lib64/libstdc++.so.6.0.22 > > 7f8110bc1000-7f8110bc3000 rw-p 00182000 fd:00 1974512 > > /usr/lib64/libstdc++.so.6.0.22 > > 7f8110bc3000-7f8110bc7000 rw-p 00000000 00:00 0 > > 7f8110bc7000-7f8110ccf000 r-xp 00000000 fd:00 1973951 > > /usr/lib64/libm-2.24.so > > 7f8110ccf000-7f8110ece000 ---p 00108000 fd:00 1973951 > > /usr/lib64/libm-2.24.so > > 7f8110ece000-7f8110ecf000 r--p 00107000 fd:00 1973951 > > /usr/lib64/libm-2.24.so > > 7f8110ecf000-7f8110ed0000 rw-p 00108000 fd:00 1973951 > > /usr/lib64/libm-2.24.so > > 7f8110ed0000-7f8110f60000 r-xp 00000000 fd:02 28457370 > > /home/edaoktay/petsc-3.13.2/arch-linux2-c-debug/lib/libmetis.so > > 7f8110f60000-7f811115f000 ---p 00090000 fd:02 28457370 > > /home/edaoktay/petsc-3.13.2/arch-linux2-c-debug/lib/libmetis.so > > 7f811115f000-7f8111160000 r--p 0008f000 fd:02 28457370 > > /home/edaoktay/petsc-3.13.2/arch-linux2-c-debug/lib/libmetis.so > > 7f8111160000-7f8111161000 rw-p 00090000 fd:02 28457370 > > /home/edaoktay/petsc-3.13.2/arch-linux2-c-debug/lib/libmetis.so > > 7f8111161000-7f8111169000 rw-p 00000000 00:00 0 > > 7f8111169000-7f81111bf000 r-xp 00000000 fd:02 28457373 > > /home/edaoktay/petsc-3.13.2/arch-linux2-c-debug/lib/libparmetis.so > > 7f81111bf000-7f81113bf000 ---p 00056000 fd:02 28457373 > > /home/edaoktay/petsc-3.13.2/arch-linux2-c-debug/lib/libparmetis.so > > 7f81113bf000-7f81113c0000 r--p 00056000 fd:02 28457373 > > /home/edaoktay/petsc-3.13.2/arch-linux2-c-debug/lib/libparmetis.so > > 7f81113c0000-7f81113c1000 rw-p 00057000 fd:02 28457373 > > /home/edaoktay/petsc-3.13.2/arch-linux2-c-debug/lib/libparmetis.so > > 7f81113c1000-7f81114f9000 r-xp 00000000 fd:00 1973763 > > /usr/lib64/libX11.so.6.3.0 > > 7f81114f9000-7f81116f9000 ---p 00138000 fd:00 1973763 > > /usr/lib64/libX11.so.6.3.0 > > 7f81116f9000-7f81116fc000 r--p 00138000 fd:00 1973763 > > /usr/lib64/libX11.so.6.3.0 > > 7f81116fc000-7f8111700000 rw-p 0013b000 fd:00 1973763 > > /usr/lib64/libX11.so.6.3.0 > > 7f8111700000-7f8111718000 r-xp 00000000 fd:00 1974340 > > /usr/lib64/libpthread-2.24.so > > 7f8111718000-7f8111918000 ---p 00018000 fd:00 1974340 > > /usr/lib64/libpthread-2.24.so > > 7f8111918000-7f8111919000 r--p 00018000 fd:00 1974340 > > /usr/lib64/libpthread-2.24.so > > 7f8111919000-7f811191a000 rw-p 00019000 fd:00 1974340 > > /usr/lib64/libpthread-2.24.so > > 7f811191a000-7f811191e000 rw-p 00000000 00:00 0 > > 7f811191e000-7f81124ff000 r-xp 00000000 fd:02 28457384 > > /home/edaoktay/petsc-3.13.2/arch-linux2-c-debug/lib/ > libopenblas_haswell-r0.3.8.dev.so > > 7f81124ff000-7f81126ff000 ---p 00be1000 fd:02 28457384 > > /home/edaoktay/petsc-3.13.2/arch-linux2-c-debug/lib/ > libopenblas_haswell-r0.3.8.dev.so > > 7f81126ff000-7f8112703000 r--p 00be1000 fd:02 28457384 > > /home/edaoktay/petsc-3.13.2/arch-linux2-c-debug/lib/ > libopenblas_haswell-r0.3.8.dev.so > > 7f8112703000-7f811270d000 rw-p 00be5000 fd:02 28457384 > > /home/edaoktay/petsc-3.13.2/arch-linux2-c-debug/lib/ > libopenblas_haswell-r0.3.8.dev.so > > 7f811270d000-7f811270e000 rw-p 00000000 00:00 0 > > 7f811270e000-7f81147c2000 r-xp 00000000 fd:02 28457400 > > /home/edaoktay/petsc-3.13.2/arch-linux2-c-debug/lib/libpetsc.so.3.13.2 > > 7f81147c2000-7f81149c1000 ---p 020b4000 fd:02 28457400 > > /home/edaoktay/petsc-3.13.2/arch-linux2-c-debug/lib/libpetsc.so.3.13.2 > > 7f81149c1000-7f81149c7000 r--p 020b3000 fd:02 28457400 > > /home/edaoktay/petsc-3.13.2/arch-linux2-c-debug/lib/libpetsc.so.3.13.2 > > 7f81149c7000-7f81149df000 rw-p 020b9000 fd:02 28457400 > > /home/edaoktay/petsc-3.13.2/arch-linux2-c-debug/lib/libpetsc.so.3.13.2 > > 7f81149df000-7f8114bdf000 rw-p 00000000 00:00 0 > > 7f8114bdf000-7f8115220000 r-xp 00000000 fd:02 28457416 > > /home/edaoktay/petsc-3.13.2/arch-linux2-c-debug/lib/libslepc.so.3.13.2 > > 7f8115220000-7f8115420000 ---p 00641000 fd:02 28457416 > > /home/edaoktay/petsc-3.13.2/arch-linux2-c-debug/lib/libslepc.so.3.13.2 > > 7f8115420000-7f8115421000 r--p 00641000 fd:02 28457416 > > /home/edaoktay/petsc-3.13.2/arch-linux2-c-debug/lib/libslepc.so.3.13.2 > > 7f8115421000-7f8115425000 rw-p 00642000 fd:02 28457416 > > /home/edaoktay/petsc-3.13.2/arch-linux2-c-debug/lib/libslepc.so.3.13.2 > > 7f8115425000-7f811544b000 r-xp 00000000 fd:00 1976838 > > /usr/lib64/ld-2.24.so > > 7f81154d3000-7f8115621000 rw-p 00000000 00:00 0 > > 7f8115647000-7f811564a000 rw-p 00000000 00:00 0 > > 7f811564a000-7f811564b000 r--p 00025000 fd:00 1976838 > > /usr/lib64/ld-2.24.so > > 7f811564b000-7f811564c000 rw-p 00026000 fd:00 1976838 > > /usr/lib64/ld-2.24.so > > 7f811564c000-7f811564d000 rw-p 00000000 00:00 0 > > 7ffc79b33000-7ffc79b55000 rw-p 00000000 00:00 0 > [stack] > > 7ffc79b67000-7ffc79b6a000 r--p 00000000 00:00 0 > [vvar] > > 7ffc79b6a000-7ffc79b6c000 r-xp 00000000 00:00 0 > [vdso] > > ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0 > > [vsyscall] > > > > > > I created and destroyed every vector, matrix and EPS in the loop to > > make sure that there won't be any segmentation fault but still I am > > getting some. Why am I getting this type of error? > > > > Thanks so much! > > > > Eda > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Mon Jul 6 15:08:35 2020 From: knepley at gmail.com (Matthew Knepley) Date: Mon, 6 Jul 2020 16:08:35 -0400 Subject: [petsc-users] /lib64/libc.so.6(cfree+0x1f8)[0x7f810f237438] Signal received In-Reply-To: References: Message-ID: On Mon, Jul 6, 2020 at 4:05 PM Eda Oktay wrote: > I tried for (count=1;count<2;++count), it is working. When I raised it to > 3, it started giving the same error. > > My loop is inside my compiled program, not bash. > Okay, give us the full stack trace from the serial run. Thanks, Matt > I started the loop after PetscInitialize and finished it before > PetscFinalize. > > I am not using MPI_Init. > > Thanks! > > On Mon, Jul 6, 2020, 10:51 PM Barry Smith wrote: > >> >> Where is your loop? In a bash script or inside your compiled program? >> >> If inside the program I assume it is outside of PetscInitialize()? >> >> Is it outside of MPI_Init()? It can't be because MPI_Init() can only be >> called once. >> >> If inside the problem you can use -start_in_debugger to get the true >> location of the crash, if in a bash script you can use >> -on_error_attach_debugger >> >> Barry >> >> >> > On Jul 6, 2020, at 1:14 PM, Eda Oktay wrote: >> > >> > Hello, >> > >> > I am trying to run my codes 100 times and then take the average >> > elapsed time. Although my program runs correctly for single time, when >> > I put them into loop, I am getting the following error: >> > >> > *** Error in `./avg_coloredge_deneme_new_vecscatter_arastep': >> > munmap_chunk(): invalid pointer: 0x0000000000b0cbc0 *** >> > [1]PETSC ERROR: >> > ------------------------------------------------------------------------ >> > [1]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, >> > probably memory access out of range >> > [1]PETSC ERROR: Try option -start_in_debugger or >> -on_error_attach_debugger >> > [1]PETSC ERROR: or see >> > https://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind >> > [1]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac >> > OS X to find memory corruption errors >> > [1]PETSC ERROR: likely location of problem given in stack below >> > [1]PETSC ERROR: --------------------- Stack Frames >> > ------------------------------------ >> > [1]PETSC ERROR: Note: The EXACT line numbers in the stack are not >> available, >> > [1]PETSC ERROR: INSTEAD the line number of the start of the >> function >> > [1]PETSC ERROR: is given. >> > ======= Backtrace: ========= >> > [1]PETSC ERROR: /lib64/libc.so.6(+0x791eb)[0x7f810f22a1eb] >> > --------------------- Error Message >> > -------------------------------------------------------------- >> > [1]PETSC ERROR: /lib64/libc.so.6(cfree+0x1f8)[0x7f810f237438] >> > Signal received >> > [1]PETSC ERROR: See >> > https://www.mcs.anl.gov/petsc/documentation/faq.html for trouble >> > shooting. >> > [1]PETSC ERROR: Petsc Release Version 3.13.2, Jun 02, 2020 >> > [1]PETSC ERROR: ./avg_coloredge_deneme_new_vecscatter_arastep on a >> > arch-linux2-c-debug named localhost.localdomain by edaoktay Mon Jul 6 >> > 21:09:07 2020 >> > [1]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ >> > --download-mpich --download-openblas --download-slepc --download-metis >> > --download-parmetis --download-chaco --with-X=1 >> > [1]PETSC ERROR: #1 User provided function() line 0 in unknown file >> > ./avg_coloredge_deneme_new_vecscatter_arastep[0x402dca] >> > ./avg_coloredge_deneme_new_vecscatter_arastep[0x402f1e] >> > ./avg_coloredge_deneme_new_vecscatter_arastep[0x402953] >> > ./avg_coloredge_deneme_new_vecscatter_arastep[0x406c1c] >> > application called MPI_Abort(MPI_COMM_WORLD, 50152059) - process 1 >> > /lib64/libc.so.6(__libc_start_main+0xf1)[0x7f810f1d1431] >> > ./avg_coloredge_deneme_new_vecscatter_arastep[0x40260a] >> > ======= Memory map: ======== >> > 00400000-00409000 r-xp 00000000 fd:02 29230434 >> > >> /home/edaoktay/petsc-3.13.2/arch-linux2-c-debug/share/slepc/examples/src/eda/avg_coloredge_deneme_new_vecscatter_arastep >> > 00608000-00609000 r--p 00008000 fd:02 29230434 >> > >> /home/edaoktay/petsc-3.13.2/arch-linux2-c-debug/share/slepc/examples/src/eda/avg_coloredge_deneme_new_vecscatter_arastep >> > 00609000-0060a000 rw-p 00009000 fd:02 29230434 >> > >> /home/edaoktay/petsc-3.13.2/arch-linux2-c-debug/share/slepc/examples/src/eda/avg_coloredge_deneme_new_vecscatter_arastep >> > 00956000-01100000 rw-p 00000000 00:00 0 >> [heap] >> > 7f810c621000-7f810c6ca000 rw-p 00000000 00:00 0 >> > 7f810c6ca000-7f810e6ca000 rw-p 00000000 00:00 0 >> > 7f810e6ca000-7f810e6d4000 r-xp 00000000 fd:00 1974339 >> > /usr/lib64/libnss_files-2.24.so >> > 7f810e6d4000-7f810e8d4000 ---p 0000a000 fd:00 1974339 >> > /usr/lib64/libnss_files-2.24.so >> > 7f810e8d4000-7f810e8d5000 r--p 0000a000 fd:00 1974339 >> > /usr/lib64/libnss_files-2.24.so >> > 7f810e8d5000-7f810e8d6000 rw-p 0000b000 fd:00 1974339 >> > /usr/lib64/libnss_files-2.24.so >> > 7f810e8d6000-7f810eb7d000 rw-p 00000000 00:00 0 >> > 7f810eb7d000-7f810eb7f000 r-xp 00000000 fd:00 1974984 >> > /usr/lib64/libXau.so.6.0.0 >> > 7f810eb7f000-7f810ed7f000 ---p 00002000 fd:00 1974984 >> > /usr/lib64/libXau.so.6.0.0 >> > 7f810ed7f000-7f810ed80000 r--p 00002000 fd:00 1974984 >> > /usr/lib64/libXau.so.6.0.0 >> > 7f810ed80000-7f810ed81000 rw-p 00000000 00:00 0 >> > 7f810ed81000-7f810ed88000 r-xp 00000000 fd:00 1974440 >> > /usr/lib64/librt-2.24.so >> > 7f810ed88000-7f810ef87000 ---p 00007000 fd:00 1974440 >> > /usr/lib64/librt-2.24.so >> > 7f810ef87000-7f810ef88000 r--p 00006000 fd:00 1974440 >> > /usr/lib64/librt-2.24.so >> > 7f810ef88000-7f810ef89000 rw-p 00007000 fd:00 1974440 >> > /usr/lib64/librt-2.24.so >> > 7f810ef89000-7f810efb0000 r-xp 00000000 fd:00 1975953 >> > /usr/lib64/libxcb.so.1.1.0 >> > 7f810efb0000-7f810f1af000 ---p 00027000 fd:00 1975953 >> > /usr/lib64/libxcb.so.1.1.0 >> > 7f810f1af000-7f810f1b0000 r--p 00026000 fd:00 1975953 >> > /usr/lib64/libxcb.so.1.1.0 >> > 7f810f1b0000-7f810f1b1000 rw-p 00027000 fd:00 1975953 >> > /usr/lib64/libxcb.so.1.1.0 >> > 7f810f1b1000-7f810f36e000 r-xp 00000000 fd:00 1973798 >> > /usr/lib64/libc-2.24.so >> > 7f810f36e000-7f810f56d000 ---p 001bd000 fd:00 1973798 >> > /usr/lib64/libc-2.24.so >> > 7f810f56d000-7f810f571000 r--p 001bc000 fd:00 1973798 >> > /usr/lib64/libc-2.24.so >> > 7f810f571000-7f810f573000 rw-p 001c0000 fd:00 1973798 >> > /usr/lib64/libc-2.24.so >> > 7f810f573000-7f810f577000 rw-p 00000000 00:00 0 >> > 7f810f577000-7f810f5b6000 r-xp 00000000 fd:00 1975094 >> > /usr/lib64/libquadmath.so.0.0.0 >> > 7f810f5b6000-7f810f7b5000 ---p 0003f000 fd:00 1975094 >> > /usr/lib64/libquadmath.so.0.0.0 >> > 7f810f7b5000-7f810f7b6000 r--p 0003e000 fd:00 1975094 >> > /usr/lib64/libquadmath.so.0.0.0 >> > 7f810f7b6000-7f810f7b7000 rw-p 0003f000 fd:00 1975094 >> > /usr/lib64/libquadmath.so.0.0.0 >> > 7f810f7b7000-7f810f7cd000 r-xp 00000000 fd:00 1973274 >> > /usr/lib64/libgcc_s-6.4.1-20170727.so.1 >> > 7f810f7cd000-7f810f9cc000 ---p 00016000 fd:00 1973274 >> > /usr/lib64/libgcc_s-6.4.1-20170727.so.1 >> > 7f810f9cc000-7f810f9cd000 r--p 00015000 fd:00 1973274 >> > /usr/lib64/libgcc_s-6.4.1-20170727.so.1 >> > 7f810f9cd000-7f810f9ce000 rw-p 00016000 fd:00 1973274 >> > /usr/lib64/libgcc_s-6.4.1-20170727.so.1 >> > 7f810f9ce000-7f810fafc000 r-xp 00000000 fd:00 1975116 >> > /usr/lib64/libgfortran.so.3.0.0 >> > 7f810fafc000-7f810fcfc000 ---p 0012e000 fd:00 1975116 >> > /usr/lib64/libgfortran.so.3.0.0 >> > 7f810fcfc000-7f810fcfd000 r--p 0012e000 fd:00 1975116 >> > /usr/lib64/libgfortran.so.3.0.0 >> > 7f810fcfd000-7f810fcfe000 rw-p 0012f000 fd:00 1975116 >> > /usr/lib64/libgfortran.so.3.0.0 >> > 7f810fcfe000-7f810fcff000 rw-p 00000000 00:00 0 >> > 7f810fcff000-7f8110168000 r-xp 00000000 fd:02 28457339 >> > /home/edaoktay/petsc-3.13.2/arch-linux2-c-debug/lib/libmpi.so.12.1.8 >> > 7f8110168000-7f8110368000 ---p 00469000 fd:02 28457339 >> > /home/edaoktay/petsc-3.13.2/arch-linux2-c-debug/lib/libmpi.so.12.1.8 >> > 7f8110368000-7f811037a000 r--p 00469000 fd:02 28457339 >> > /home/edaoktay/petsc-3.13.2/arch-linux2-c-debug/lib/libmpi.so.12.1.8 >> > 7f811037a000-7f8110385000 rw-p 0047b000 fd:02 28457339 >> > /home/edaoktay/petsc-3.13.2/arch-linux2-c-debug/lib/libmpi.so.12.1.8 >> > 7f8110385000-7f81103c4000 rw-p 00000000 00:00 0 >> > 7f81103c4000-7f8110435000 r-xp 00000000 fd:02 28457343 >> > >> /home/edaoktay/petsc-3.13.2/arch-linux2-c-debug/lib/libmpifort.so.12.1.8 >> > 7f8110435000-7f8110635000 ---p 00071000 fd:02 28457343 >> > >> /home/edaoktay/petsc-3.13.2/arch-linux2-c-debug/lib/libmpifort.so.12.1.8 >> > 7f8110635000-7f8110639000 r--p 00071000 fd:02 28457343 >> > >> /home/edaoktay/petsc-3.13.2/arch-linux2-c-debug/lib/libmpifort.so.12.1.8 >> > 7f8110639000-7f811063b000 rw-p 00075000 fd:02 28457343 >> > >> /home/edaoktay/petsc-3.13.2/arch-linux2-c-debug/lib/libmpifort.so.12.1.8 >> > 7f811063b000-7f811063e000 r-xp 00000000 fd:00 1973895 >> > /usr/lib64/libdl-2.24.so >> > 7f811063e000-7f811083d000 ---p 00003000 fd:00 1973895 >> > /usr/lib64/libdl-2.24.so >> > 7f811083d000-7f811083e000 r--p 00002000 fd:00 1973895 >> > /usr/lib64/libdl-2.24.so >> > 7f811083e000-7f811083f000 rw-p 00003000 fd:00 1973895 >> > /usr/lib64/libdl-2.24.so >> > 7f811083f000-7f81109b7000 r-xp 00000000 fd:00 1974512 >> > /usr/lib64/libstdc++.so.6.0.22 >> > 7f81109b7000-7f8110bb7000 ---p 00178000 fd:00 1974512 >> > /usr/lib64/libstdc++.so.6.0.22 >> > 7f8110bb7000-7f8110bc1000 r--p 00178000 fd:00 1974512 >> > /usr/lib64/libstdc++.so.6.0.22 >> > 7f8110bc1000-7f8110bc3000 rw-p 00182000 fd:00 1974512 >> > /usr/lib64/libstdc++.so.6.0.22 >> > 7f8110bc3000-7f8110bc7000 rw-p 00000000 00:00 0 >> > 7f8110bc7000-7f8110ccf000 r-xp 00000000 fd:00 1973951 >> > /usr/lib64/libm-2.24.so >> > 7f8110ccf000-7f8110ece000 ---p 00108000 fd:00 1973951 >> > /usr/lib64/libm-2.24.so >> > 7f8110ece000-7f8110ecf000 r--p 00107000 fd:00 1973951 >> > /usr/lib64/libm-2.24.so >> > 7f8110ecf000-7f8110ed0000 rw-p 00108000 fd:00 1973951 >> > /usr/lib64/libm-2.24.so >> > 7f8110ed0000-7f8110f60000 r-xp 00000000 fd:02 28457370 >> > /home/edaoktay/petsc-3.13.2/arch-linux2-c-debug/lib/libmetis.so >> > 7f8110f60000-7f811115f000 ---p 00090000 fd:02 28457370 >> > /home/edaoktay/petsc-3.13.2/arch-linux2-c-debug/lib/libmetis.so >> > 7f811115f000-7f8111160000 r--p 0008f000 fd:02 28457370 >> > /home/edaoktay/petsc-3.13.2/arch-linux2-c-debug/lib/libmetis.so >> > 7f8111160000-7f8111161000 rw-p 00090000 fd:02 28457370 >> > /home/edaoktay/petsc-3.13.2/arch-linux2-c-debug/lib/libmetis.so >> > 7f8111161000-7f8111169000 rw-p 00000000 00:00 0 >> > 7f8111169000-7f81111bf000 r-xp 00000000 fd:02 28457373 >> > /home/edaoktay/petsc-3.13.2/arch-linux2-c-debug/lib/libparmetis.so >> > 7f81111bf000-7f81113bf000 ---p 00056000 fd:02 28457373 >> > /home/edaoktay/petsc-3.13.2/arch-linux2-c-debug/lib/libparmetis.so >> > 7f81113bf000-7f81113c0000 r--p 00056000 fd:02 28457373 >> > /home/edaoktay/petsc-3.13.2/arch-linux2-c-debug/lib/libparmetis.so >> > 7f81113c0000-7f81113c1000 rw-p 00057000 fd:02 28457373 >> > /home/edaoktay/petsc-3.13.2/arch-linux2-c-debug/lib/libparmetis.so >> > 7f81113c1000-7f81114f9000 r-xp 00000000 fd:00 1973763 >> > /usr/lib64/libX11.so.6.3.0 >> > 7f81114f9000-7f81116f9000 ---p 00138000 fd:00 1973763 >> > /usr/lib64/libX11.so.6.3.0 >> > 7f81116f9000-7f81116fc000 r--p 00138000 fd:00 1973763 >> > /usr/lib64/libX11.so.6.3.0 >> > 7f81116fc000-7f8111700000 rw-p 0013b000 fd:00 1973763 >> > /usr/lib64/libX11.so.6.3.0 >> > 7f8111700000-7f8111718000 r-xp 00000000 fd:00 1974340 >> > /usr/lib64/libpthread-2.24.so >> > 7f8111718000-7f8111918000 ---p 00018000 fd:00 1974340 >> > /usr/lib64/libpthread-2.24.so >> > 7f8111918000-7f8111919000 r--p 00018000 fd:00 1974340 >> > /usr/lib64/libpthread-2.24.so >> > 7f8111919000-7f811191a000 rw-p 00019000 fd:00 1974340 >> > /usr/lib64/libpthread-2.24.so >> > 7f811191a000-7f811191e000 rw-p 00000000 00:00 0 >> > 7f811191e000-7f81124ff000 r-xp 00000000 fd:02 28457384 >> > /home/edaoktay/petsc-3.13.2/arch-linux2-c-debug/lib/ >> libopenblas_haswell-r0.3.8.dev.so >> > 7f81124ff000-7f81126ff000 ---p 00be1000 fd:02 28457384 >> > /home/edaoktay/petsc-3.13.2/arch-linux2-c-debug/lib/ >> libopenblas_haswell-r0.3.8.dev.so >> > 7f81126ff000-7f8112703000 r--p 00be1000 fd:02 28457384 >> > /home/edaoktay/petsc-3.13.2/arch-linux2-c-debug/lib/ >> libopenblas_haswell-r0.3.8.dev.so >> > 7f8112703000-7f811270d000 rw-p 00be5000 fd:02 28457384 >> > /home/edaoktay/petsc-3.13.2/arch-linux2-c-debug/lib/ >> libopenblas_haswell-r0.3.8.dev.so >> > 7f811270d000-7f811270e000 rw-p 00000000 00:00 0 >> > 7f811270e000-7f81147c2000 r-xp 00000000 fd:02 28457400 >> > /home/edaoktay/petsc-3.13.2/arch-linux2-c-debug/lib/libpetsc.so.3.13.2 >> > 7f81147c2000-7f81149c1000 ---p 020b4000 fd:02 28457400 >> > /home/edaoktay/petsc-3.13.2/arch-linux2-c-debug/lib/libpetsc.so.3.13.2 >> > 7f81149c1000-7f81149c7000 r--p 020b3000 fd:02 28457400 >> > /home/edaoktay/petsc-3.13.2/arch-linux2-c-debug/lib/libpetsc.so.3.13.2 >> > 7f81149c7000-7f81149df000 rw-p 020b9000 fd:02 28457400 >> > /home/edaoktay/petsc-3.13.2/arch-linux2-c-debug/lib/libpetsc.so.3.13.2 >> > 7f81149df000-7f8114bdf000 rw-p 00000000 00:00 0 >> > 7f8114bdf000-7f8115220000 r-xp 00000000 fd:02 28457416 >> > /home/edaoktay/petsc-3.13.2/arch-linux2-c-debug/lib/libslepc.so.3.13.2 >> > 7f8115220000-7f8115420000 ---p 00641000 fd:02 28457416 >> > /home/edaoktay/petsc-3.13.2/arch-linux2-c-debug/lib/libslepc.so.3.13.2 >> > 7f8115420000-7f8115421000 r--p 00641000 fd:02 28457416 >> > /home/edaoktay/petsc-3.13.2/arch-linux2-c-debug/lib/libslepc.so.3.13.2 >> > 7f8115421000-7f8115425000 rw-p 00642000 fd:02 28457416 >> > /home/edaoktay/petsc-3.13.2/arch-linux2-c-debug/lib/libslepc.so.3.13.2 >> > 7f8115425000-7f811544b000 r-xp 00000000 fd:00 1976838 >> > /usr/lib64/ld-2.24.so >> > 7f81154d3000-7f8115621000 rw-p 00000000 00:00 0 >> > 7f8115647000-7f811564a000 rw-p 00000000 00:00 0 >> > 7f811564a000-7f811564b000 r--p 00025000 fd:00 1976838 >> > /usr/lib64/ld-2.24.so >> > 7f811564b000-7f811564c000 rw-p 00026000 fd:00 1976838 >> > /usr/lib64/ld-2.24.so >> > 7f811564c000-7f811564d000 rw-p 00000000 00:00 0 >> > 7ffc79b33000-7ffc79b55000 rw-p 00000000 00:00 0 >> [stack] >> > 7ffc79b67000-7ffc79b6a000 r--p 00000000 00:00 0 >> [vvar] >> > 7ffc79b6a000-7ffc79b6c000 r-xp 00000000 00:00 0 >> [vdso] >> > ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0 >> > [vsyscall] >> > >> > >> > I created and destroyed every vector, matrix and EPS in the loop to >> > make sure that there won't be any segmentation fault but still I am >> > getting some. Why am I getting this type of error? >> > >> > Thanks so much! >> > >> > Eda >> >> -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From junchao.zhang at gmail.com Mon Jul 6 16:19:50 2020 From: junchao.zhang at gmail.com (Junchao Zhang) Date: Mon, 6 Jul 2020 16:19:50 -0500 Subject: [petsc-users] /lib64/libc.so.6(cfree+0x1f8)[0x7f810f237438] Signal received In-Reply-To: References: Message-ID: Try to run with valgrind. --Junchao Zhang On Mon, Jul 6, 2020 at 3:05 PM Eda Oktay wrote: > I tried for (count=1;count<2;++count), it is working. When I raised it to > 3, it started giving the same error. > > My loop is inside my compiled program, not bash. > > I started the loop after PetscInitialize and finished it before > PetscFinalize. > > I am not using MPI_Init. > > Thanks! > > On Mon, Jul 6, 2020, 10:51 PM Barry Smith wrote: > >> >> Where is your loop? In a bash script or inside your compiled program? >> >> If inside the program I assume it is outside of PetscInitialize()? >> >> Is it outside of MPI_Init()? It can't be because MPI_Init() can only be >> called once. >> >> If inside the problem you can use -start_in_debugger to get the true >> location of the crash, if in a bash script you can use >> -on_error_attach_debugger >> >> Barry >> >> >> > On Jul 6, 2020, at 1:14 PM, Eda Oktay wrote: >> > >> > Hello, >> > >> > I am trying to run my codes 100 times and then take the average >> > elapsed time. Although my program runs correctly for single time, when >> > I put them into loop, I am getting the following error: >> > >> > *** Error in `./avg_coloredge_deneme_new_vecscatter_arastep': >> > munmap_chunk(): invalid pointer: 0x0000000000b0cbc0 *** >> > [1]PETSC ERROR: >> > ------------------------------------------------------------------------ >> > [1]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, >> > probably memory access out of range >> > [1]PETSC ERROR: Try option -start_in_debugger or >> -on_error_attach_debugger >> > [1]PETSC ERROR: or see >> > https://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind >> > [1]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac >> > OS X to find memory corruption errors >> > [1]PETSC ERROR: likely location of problem given in stack below >> > [1]PETSC ERROR: --------------------- Stack Frames >> > ------------------------------------ >> > [1]PETSC ERROR: Note: The EXACT line numbers in the stack are not >> available, >> > [1]PETSC ERROR: INSTEAD the line number of the start of the >> function >> > [1]PETSC ERROR: is given. >> > ======= Backtrace: ========= >> > [1]PETSC ERROR: /lib64/libc.so.6(+0x791eb)[0x7f810f22a1eb] >> > --------------------- Error Message >> > -------------------------------------------------------------- >> > [1]PETSC ERROR: /lib64/libc.so.6(cfree+0x1f8)[0x7f810f237438] >> > Signal received >> > [1]PETSC ERROR: See >> > https://www.mcs.anl.gov/petsc/documentation/faq.html for trouble >> > shooting. >> > [1]PETSC ERROR: Petsc Release Version 3.13.2, Jun 02, 2020 >> > [1]PETSC ERROR: ./avg_coloredge_deneme_new_vecscatter_arastep on a >> > arch-linux2-c-debug named localhost.localdomain by edaoktay Mon Jul 6 >> > 21:09:07 2020 >> > [1]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ >> > --download-mpich --download-openblas --download-slepc --download-metis >> > --download-parmetis --download-chaco --with-X=1 >> > [1]PETSC ERROR: #1 User provided function() line 0 in unknown file >> > ./avg_coloredge_deneme_new_vecscatter_arastep[0x402dca] >> > ./avg_coloredge_deneme_new_vecscatter_arastep[0x402f1e] >> > ./avg_coloredge_deneme_new_vecscatter_arastep[0x402953] >> > ./avg_coloredge_deneme_new_vecscatter_arastep[0x406c1c] >> > application called MPI_Abort(MPI_COMM_WORLD, 50152059) - process 1 >> > /lib64/libc.so.6(__libc_start_main+0xf1)[0x7f810f1d1431] >> > ./avg_coloredge_deneme_new_vecscatter_arastep[0x40260a] >> > ======= Memory map: ======== >> > 00400000-00409000 r-xp 00000000 fd:02 29230434 >> > >> /home/edaoktay/petsc-3.13.2/arch-linux2-c-debug/share/slepc/examples/src/eda/avg_coloredge_deneme_new_vecscatter_arastep >> > 00608000-00609000 r--p 00008000 fd:02 29230434 >> > >> /home/edaoktay/petsc-3.13.2/arch-linux2-c-debug/share/slepc/examples/src/eda/avg_coloredge_deneme_new_vecscatter_arastep >> > 00609000-0060a000 rw-p 00009000 fd:02 29230434 >> > >> /home/edaoktay/petsc-3.13.2/arch-linux2-c-debug/share/slepc/examples/src/eda/avg_coloredge_deneme_new_vecscatter_arastep >> > 00956000-01100000 rw-p 00000000 00:00 0 >> [heap] >> > 7f810c621000-7f810c6ca000 rw-p 00000000 00:00 0 >> > 7f810c6ca000-7f810e6ca000 rw-p 00000000 00:00 0 >> > 7f810e6ca000-7f810e6d4000 r-xp 00000000 fd:00 1974339 >> > /usr/lib64/libnss_files-2.24.so >> > 7f810e6d4000-7f810e8d4000 ---p 0000a000 fd:00 1974339 >> > /usr/lib64/libnss_files-2.24.so >> > 7f810e8d4000-7f810e8d5000 r--p 0000a000 fd:00 1974339 >> > /usr/lib64/libnss_files-2.24.so >> > 7f810e8d5000-7f810e8d6000 rw-p 0000b000 fd:00 1974339 >> > /usr/lib64/libnss_files-2.24.so >> > 7f810e8d6000-7f810eb7d000 rw-p 00000000 00:00 0 >> > 7f810eb7d000-7f810eb7f000 r-xp 00000000 fd:00 1974984 >> > /usr/lib64/libXau.so.6.0.0 >> > 7f810eb7f000-7f810ed7f000 ---p 00002000 fd:00 1974984 >> > /usr/lib64/libXau.so.6.0.0 >> > 7f810ed7f000-7f810ed80000 r--p 00002000 fd:00 1974984 >> > /usr/lib64/libXau.so.6.0.0 >> > 7f810ed80000-7f810ed81000 rw-p 00000000 00:00 0 >> > 7f810ed81000-7f810ed88000 r-xp 00000000 fd:00 1974440 >> > /usr/lib64/librt-2.24.so >> > 7f810ed88000-7f810ef87000 ---p 00007000 fd:00 1974440 >> > /usr/lib64/librt-2.24.so >> > 7f810ef87000-7f810ef88000 r--p 00006000 fd:00 1974440 >> > /usr/lib64/librt-2.24.so >> > 7f810ef88000-7f810ef89000 rw-p 00007000 fd:00 1974440 >> > /usr/lib64/librt-2.24.so >> > 7f810ef89000-7f810efb0000 r-xp 00000000 fd:00 1975953 >> > /usr/lib64/libxcb.so.1.1.0 >> > 7f810efb0000-7f810f1af000 ---p 00027000 fd:00 1975953 >> > /usr/lib64/libxcb.so.1.1.0 >> > 7f810f1af000-7f810f1b0000 r--p 00026000 fd:00 1975953 >> > /usr/lib64/libxcb.so.1.1.0 >> > 7f810f1b0000-7f810f1b1000 rw-p 00027000 fd:00 1975953 >> > /usr/lib64/libxcb.so.1.1.0 >> > 7f810f1b1000-7f810f36e000 r-xp 00000000 fd:00 1973798 >> > /usr/lib64/libc-2.24.so >> > 7f810f36e000-7f810f56d000 ---p 001bd000 fd:00 1973798 >> > /usr/lib64/libc-2.24.so >> > 7f810f56d000-7f810f571000 r--p 001bc000 fd:00 1973798 >> > /usr/lib64/libc-2.24.so >> > 7f810f571000-7f810f573000 rw-p 001c0000 fd:00 1973798 >> > /usr/lib64/libc-2.24.so >> > 7f810f573000-7f810f577000 rw-p 00000000 00:00 0 >> > 7f810f577000-7f810f5b6000 r-xp 00000000 fd:00 1975094 >> > /usr/lib64/libquadmath.so.0.0.0 >> > 7f810f5b6000-7f810f7b5000 ---p 0003f000 fd:00 1975094 >> > /usr/lib64/libquadmath.so.0.0.0 >> > 7f810f7b5000-7f810f7b6000 r--p 0003e000 fd:00 1975094 >> > /usr/lib64/libquadmath.so.0.0.0 >> > 7f810f7b6000-7f810f7b7000 rw-p 0003f000 fd:00 1975094 >> > /usr/lib64/libquadmath.so.0.0.0 >> > 7f810f7b7000-7f810f7cd000 r-xp 00000000 fd:00 1973274 >> > /usr/lib64/libgcc_s-6.4.1-20170727.so.1 >> > 7f810f7cd000-7f810f9cc000 ---p 00016000 fd:00 1973274 >> > /usr/lib64/libgcc_s-6.4.1-20170727.so.1 >> > 7f810f9cc000-7f810f9cd000 r--p 00015000 fd:00 1973274 >> > /usr/lib64/libgcc_s-6.4.1-20170727.so.1 >> > 7f810f9cd000-7f810f9ce000 rw-p 00016000 fd:00 1973274 >> > /usr/lib64/libgcc_s-6.4.1-20170727.so.1 >> > 7f810f9ce000-7f810fafc000 r-xp 00000000 fd:00 1975116 >> > /usr/lib64/libgfortran.so.3.0.0 >> > 7f810fafc000-7f810fcfc000 ---p 0012e000 fd:00 1975116 >> > /usr/lib64/libgfortran.so.3.0.0 >> > 7f810fcfc000-7f810fcfd000 r--p 0012e000 fd:00 1975116 >> > /usr/lib64/libgfortran.so.3.0.0 >> > 7f810fcfd000-7f810fcfe000 rw-p 0012f000 fd:00 1975116 >> > /usr/lib64/libgfortran.so.3.0.0 >> > 7f810fcfe000-7f810fcff000 rw-p 00000000 00:00 0 >> > 7f810fcff000-7f8110168000 r-xp 00000000 fd:02 28457339 >> > /home/edaoktay/petsc-3.13.2/arch-linux2-c-debug/lib/libmpi.so.12.1.8 >> > 7f8110168000-7f8110368000 ---p 00469000 fd:02 28457339 >> > /home/edaoktay/petsc-3.13.2/arch-linux2-c-debug/lib/libmpi.so.12.1.8 >> > 7f8110368000-7f811037a000 r--p 00469000 fd:02 28457339 >> > /home/edaoktay/petsc-3.13.2/arch-linux2-c-debug/lib/libmpi.so.12.1.8 >> > 7f811037a000-7f8110385000 rw-p 0047b000 fd:02 28457339 >> > /home/edaoktay/petsc-3.13.2/arch-linux2-c-debug/lib/libmpi.so.12.1.8 >> > 7f8110385000-7f81103c4000 rw-p 00000000 00:00 0 >> > 7f81103c4000-7f8110435000 r-xp 00000000 fd:02 28457343 >> > >> /home/edaoktay/petsc-3.13.2/arch-linux2-c-debug/lib/libmpifort.so.12.1.8 >> > 7f8110435000-7f8110635000 ---p 00071000 fd:02 28457343 >> > >> /home/edaoktay/petsc-3.13.2/arch-linux2-c-debug/lib/libmpifort.so.12.1.8 >> > 7f8110635000-7f8110639000 r--p 00071000 fd:02 28457343 >> > >> /home/edaoktay/petsc-3.13.2/arch-linux2-c-debug/lib/libmpifort.so.12.1.8 >> > 7f8110639000-7f811063b000 rw-p 00075000 fd:02 28457343 >> > >> /home/edaoktay/petsc-3.13.2/arch-linux2-c-debug/lib/libmpifort.so.12.1.8 >> > 7f811063b000-7f811063e000 r-xp 00000000 fd:00 1973895 >> > /usr/lib64/libdl-2.24.so >> > 7f811063e000-7f811083d000 ---p 00003000 fd:00 1973895 >> > /usr/lib64/libdl-2.24.so >> > 7f811083d000-7f811083e000 r--p 00002000 fd:00 1973895 >> > /usr/lib64/libdl-2.24.so >> > 7f811083e000-7f811083f000 rw-p 00003000 fd:00 1973895 >> > /usr/lib64/libdl-2.24.so >> > 7f811083f000-7f81109b7000 r-xp 00000000 fd:00 1974512 >> > /usr/lib64/libstdc++.so.6.0.22 >> > 7f81109b7000-7f8110bb7000 ---p 00178000 fd:00 1974512 >> > /usr/lib64/libstdc++.so.6.0.22 >> > 7f8110bb7000-7f8110bc1000 r--p 00178000 fd:00 1974512 >> > /usr/lib64/libstdc++.so.6.0.22 >> > 7f8110bc1000-7f8110bc3000 rw-p 00182000 fd:00 1974512 >> > /usr/lib64/libstdc++.so.6.0.22 >> > 7f8110bc3000-7f8110bc7000 rw-p 00000000 00:00 0 >> > 7f8110bc7000-7f8110ccf000 r-xp 00000000 fd:00 1973951 >> > /usr/lib64/libm-2.24.so >> > 7f8110ccf000-7f8110ece000 ---p 00108000 fd:00 1973951 >> > /usr/lib64/libm-2.24.so >> > 7f8110ece000-7f8110ecf000 r--p 00107000 fd:00 1973951 >> > /usr/lib64/libm-2.24.so >> > 7f8110ecf000-7f8110ed0000 rw-p 00108000 fd:00 1973951 >> > /usr/lib64/libm-2.24.so >> > 7f8110ed0000-7f8110f60000 r-xp 00000000 fd:02 28457370 >> > /home/edaoktay/petsc-3.13.2/arch-linux2-c-debug/lib/libmetis.so >> > 7f8110f60000-7f811115f000 ---p 00090000 fd:02 28457370 >> > /home/edaoktay/petsc-3.13.2/arch-linux2-c-debug/lib/libmetis.so >> > 7f811115f000-7f8111160000 r--p 0008f000 fd:02 28457370 >> > /home/edaoktay/petsc-3.13.2/arch-linux2-c-debug/lib/libmetis.so >> > 7f8111160000-7f8111161000 rw-p 00090000 fd:02 28457370 >> > /home/edaoktay/petsc-3.13.2/arch-linux2-c-debug/lib/libmetis.so >> > 7f8111161000-7f8111169000 rw-p 00000000 00:00 0 >> > 7f8111169000-7f81111bf000 r-xp 00000000 fd:02 28457373 >> > /home/edaoktay/petsc-3.13.2/arch-linux2-c-debug/lib/libparmetis.so >> > 7f81111bf000-7f81113bf000 ---p 00056000 fd:02 28457373 >> > /home/edaoktay/petsc-3.13.2/arch-linux2-c-debug/lib/libparmetis.so >> > 7f81113bf000-7f81113c0000 r--p 00056000 fd:02 28457373 >> > /home/edaoktay/petsc-3.13.2/arch-linux2-c-debug/lib/libparmetis.so >> > 7f81113c0000-7f81113c1000 rw-p 00057000 fd:02 28457373 >> > /home/edaoktay/petsc-3.13.2/arch-linux2-c-debug/lib/libparmetis.so >> > 7f81113c1000-7f81114f9000 r-xp 00000000 fd:00 1973763 >> > /usr/lib64/libX11.so.6.3.0 >> > 7f81114f9000-7f81116f9000 ---p 00138000 fd:00 1973763 >> > /usr/lib64/libX11.so.6.3.0 >> > 7f81116f9000-7f81116fc000 r--p 00138000 fd:00 1973763 >> > /usr/lib64/libX11.so.6.3.0 >> > 7f81116fc000-7f8111700000 rw-p 0013b000 fd:00 1973763 >> > /usr/lib64/libX11.so.6.3.0 >> > 7f8111700000-7f8111718000 r-xp 00000000 fd:00 1974340 >> > /usr/lib64/libpthread-2.24.so >> > 7f8111718000-7f8111918000 ---p 00018000 fd:00 1974340 >> > /usr/lib64/libpthread-2.24.so >> > 7f8111918000-7f8111919000 r--p 00018000 fd:00 1974340 >> > /usr/lib64/libpthread-2.24.so >> > 7f8111919000-7f811191a000 rw-p 00019000 fd:00 1974340 >> > /usr/lib64/libpthread-2.24.so >> > 7f811191a000-7f811191e000 rw-p 00000000 00:00 0 >> > 7f811191e000-7f81124ff000 r-xp 00000000 fd:02 28457384 >> > /home/edaoktay/petsc-3.13.2/arch-linux2-c-debug/lib/ >> libopenblas_haswell-r0.3.8.dev.so >> > 7f81124ff000-7f81126ff000 ---p 00be1000 fd:02 28457384 >> > /home/edaoktay/petsc-3.13.2/arch-linux2-c-debug/lib/ >> libopenblas_haswell-r0.3.8.dev.so >> > 7f81126ff000-7f8112703000 r--p 00be1000 fd:02 28457384 >> > /home/edaoktay/petsc-3.13.2/arch-linux2-c-debug/lib/ >> libopenblas_haswell-r0.3.8.dev.so >> > 7f8112703000-7f811270d000 rw-p 00be5000 fd:02 28457384 >> > /home/edaoktay/petsc-3.13.2/arch-linux2-c-debug/lib/ >> libopenblas_haswell-r0.3.8.dev.so >> > 7f811270d000-7f811270e000 rw-p 00000000 00:00 0 >> > 7f811270e000-7f81147c2000 r-xp 00000000 fd:02 28457400 >> > /home/edaoktay/petsc-3.13.2/arch-linux2-c-debug/lib/libpetsc.so.3.13.2 >> > 7f81147c2000-7f81149c1000 ---p 020b4000 fd:02 28457400 >> > /home/edaoktay/petsc-3.13.2/arch-linux2-c-debug/lib/libpetsc.so.3.13.2 >> > 7f81149c1000-7f81149c7000 r--p 020b3000 fd:02 28457400 >> > /home/edaoktay/petsc-3.13.2/arch-linux2-c-debug/lib/libpetsc.so.3.13.2 >> > 7f81149c7000-7f81149df000 rw-p 020b9000 fd:02 28457400 >> > /home/edaoktay/petsc-3.13.2/arch-linux2-c-debug/lib/libpetsc.so.3.13.2 >> > 7f81149df000-7f8114bdf000 rw-p 00000000 00:00 0 >> > 7f8114bdf000-7f8115220000 r-xp 00000000 fd:02 28457416 >> > /home/edaoktay/petsc-3.13.2/arch-linux2-c-debug/lib/libslepc.so.3.13.2 >> > 7f8115220000-7f8115420000 ---p 00641000 fd:02 28457416 >> > /home/edaoktay/petsc-3.13.2/arch-linux2-c-debug/lib/libslepc.so.3.13.2 >> > 7f8115420000-7f8115421000 r--p 00641000 fd:02 28457416 >> > /home/edaoktay/petsc-3.13.2/arch-linux2-c-debug/lib/libslepc.so.3.13.2 >> > 7f8115421000-7f8115425000 rw-p 00642000 fd:02 28457416 >> > /home/edaoktay/petsc-3.13.2/arch-linux2-c-debug/lib/libslepc.so.3.13.2 >> > 7f8115425000-7f811544b000 r-xp 00000000 fd:00 1976838 >> > /usr/lib64/ld-2.24.so >> > 7f81154d3000-7f8115621000 rw-p 00000000 00:00 0 >> > 7f8115647000-7f811564a000 rw-p 00000000 00:00 0 >> > 7f811564a000-7f811564b000 r--p 00025000 fd:00 1976838 >> > /usr/lib64/ld-2.24.so >> > 7f811564b000-7f811564c000 rw-p 00026000 fd:00 1976838 >> > /usr/lib64/ld-2.24.so >> > 7f811564c000-7f811564d000 rw-p 00000000 00:00 0 >> > 7ffc79b33000-7ffc79b55000 rw-p 00000000 00:00 0 >> [stack] >> > 7ffc79b67000-7ffc79b6a000 r--p 00000000 00:00 0 >> [vvar] >> > 7ffc79b6a000-7ffc79b6c000 r-xp 00000000 00:00 0 >> [vdso] >> > ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0 >> > [vsyscall] >> > >> > >> > I created and destroyed every vector, matrix and EPS in the loop to >> > make sure that there won't be any segmentation fault but still I am >> > getting some. Why am I getting this type of error? >> > >> > Thanks so much! >> > >> > Eda >> >> -------------- next part -------------- An HTML attachment was scrubbed... URL: From EPrudencio at slb.com Mon Jul 6 17:09:20 2020 From: EPrudencio at slb.com (Ernesto Prudencio) Date: Mon, 6 Jul 2020 22:09:20 +0000 Subject: [petsc-users] Basic error " gcc: error: unrecognized command line option 'wd1572' " during the configuration step of SLEPc installation Message-ID: Hello. Until some days ago I was using packages MPICH and PETSc which were both downloaded and installed by me, locally in my Linux environment. I could then download and install SLEPc without issues. Recently I had to change to another environment where MPI and PETSc were already pre-installed. I have access to them through a mounted disk. I then tried to install a SLEPc in my local disk using such mounted packages, and I am getting the following error right away: > cd $SLEPC_DIR > ./configure --prefix=/home/EPrudencio/projs/changed/extraInstallations/slepc_3_10_2 Checking environment... done Checking PETSc installation... ERROR: Unable to link with PETSc ERROR: See "installed-arch-linux2-c-opt/lib/slepc/conf/configure.log" file for details > When I open the configure.log file, the contents are as follows: +---BEGIN---------------------------------------------------------------------- ================================================================================ Starting Configure Run at Mon Jul 6 21:54:40 2020 Configure Options: --prefix=/home/... ... ... ================================================================================ Checking PETSc installation... #include "petscsnes.h" int main() { Vec v; Mat m; KSP k; PetscInitializeNoArguments(); VecCreate(PETSC_COMM_WORLD,&v); MatCreate(PETSC_COMM_WORLD,&m); KSPCreate(PETSC_COMM_WORLD,&k); return 0; } mpicc -o checklink.o -c -fPIC -wd1572 -Ofast -xHost -I/mnt/disks/intel-2018-3-222-blade-runtime-env-2018-1-07-08-2018-132838/petsc_3.10.4_openmpi.4.0.3_intel2019.3_64index/include `pwd`/checklink.c gcc: error: unrecognized command line option 'wd1572' gmake: *** [checklink.o] Error 1 ERROR: Unable to link with PETSc +---END---------------------------------------------------------------------- When I run mpicc -version, I get "gcc (GCC) 4.9.2" The PETSc version is 3.10.4, and It seems it was installed with openmpi and intel compiler (I might be wrong here). The SLEPc version is 3.10.2. Any suggestions? Should I set variables CC and CXX in the configuration command line for SLEPc? Thank you in advance, Ernesto. Schlumberger-Private -------------- next part -------------- An HTML attachment was scrubbed... URL: From balay at mcs.anl.gov Mon Jul 6 17:18:13 2020 From: balay at mcs.anl.gov (Satish Balay) Date: Mon, 6 Jul 2020 17:18:13 -0500 (CDT) Subject: [petsc-users] Basic error " gcc: error: unrecognized command line option 'wd1572' " during the configuration step of SLEPc installation In-Reply-To: References: Message-ID: Likely the [MPI] compiler env is different between PETSc and SLEPc builds. PETSc was build with mpicc using icc. Now during SLEPc build - mpicc is attempting to use gcc. petsc configure.log would have more info on which mpicc was used to install PETSc. Satish On Mon, 6 Jul 2020, Ernesto Prudencio via petsc-users wrote: > Hello. > > Until some days ago I was using packages MPICH and PETSc which were both downloaded and installed by me, locally in my Linux environment. I could then download and install SLEPc without issues. > > Recently I had to change to another environment where MPI and PETSc were already pre-installed. I have access to them through a mounted disk. I then tried to install a SLEPc in my local disk using such mounted packages, and I am getting the following error right away: > > > cd $SLEPC_DIR > > ./configure --prefix=/home/EPrudencio/projs/changed/extraInstallations/slepc_3_10_2 > Checking environment... done > Checking PETSc installation... > ERROR: Unable to link with PETSc > ERROR: See "installed-arch-linux2-c-opt/lib/slepc/conf/configure.log" file for details > > > > When I open the configure.log file, the contents are as follows: > > +---BEGIN---------------------------------------------------------------------- > > ================================================================================ > Starting Configure Run at Mon Jul 6 21:54:40 2020 > Configure Options: --prefix=/home/... > ... > ... > ================================================================================ > Checking PETSc installation... > #include "petscsnes.h" > int main() { > Vec v; Mat m; KSP k; > PetscInitializeNoArguments(); > VecCreate(PETSC_COMM_WORLD,&v); > MatCreate(PETSC_COMM_WORLD,&m); > KSPCreate(PETSC_COMM_WORLD,&k); > return 0; > } > mpicc -o checklink.o -c -fPIC -wd1572 -Ofast -xHost -I/mnt/disks/intel-2018-3-222-blade-runtime-env-2018-1-07-08-2018-132838/petsc_3.10.4_openmpi.4.0.3_intel2019.3_64index/include `pwd`/checklink.c > gcc: error: unrecognized command line option 'wd1572' > gmake: *** [checklink.o] Error 1 > > ERROR: Unable to link with PETSc > > +---END---------------------------------------------------------------------- > > When I run mpicc -version, I get "gcc (GCC) 4.9.2" > > The PETSc version is 3.10.4, and It seems it was installed with openmpi and intel compiler (I might be wrong here). > > The SLEPc version is 3.10.2. > > Any suggestions? Should I set variables CC and CXX in the configuration command line for SLEPc? > > Thank you in advance, > > Ernesto. > > > Schlumberger-Private > From EPrudencio at slb.com Mon Jul 6 19:02:28 2020 From: EPrudencio at slb.com (Ernesto Prudencio) Date: Tue, 7 Jul 2020 00:02:28 +0000 Subject: [petsc-users] [Ext] Re: Basic error " gcc: error: unrecognized command line option 'wd1572' " during the configuration step of SLEPc installation In-Reply-To: References: Message-ID: Thank you, Satish. Ernesto. Schlumberger-Private -----Original Message----- From: Satish Balay [mailto:balay at mcs.anl.gov] Sent: Monday, July 6, 2020 5:18 PM To: Ernesto Prudencio Cc: PETSc users list Subject: [Ext] Re: [petsc-users] Basic error " gcc: error: unrecognized command line option 'wd1572' " during the configuration step of SLEPc installation Likely the [MPI] compiler env is different between PETSc and SLEPc builds. PETSc was build with mpicc using icc. Now during SLEPc build - mpicc is attempting to use gcc. petsc configure.log would have more info on which mpicc was used to install PETSc. Satish On Mon, 6 Jul 2020, Ernesto Prudencio via petsc-users wrote: > Hello. > > Until some days ago I was using packages MPICH and PETSc which were both downloaded and installed by me, locally in my Linux environment. I could then download and install SLEPc without issues. > > Recently I had to change to another environment where MPI and PETSc were already pre-installed. I have access to them through a mounted disk. I then tried to install a SLEPc in my local disk using such mounted packages, and I am getting the following error right away: > > > cd $SLEPC_DIR > > ./configure > > --prefix=/home/EPrudencio/projs/changed/extraInstallations/slepc_3_1 > > 0_2 > Checking environment... done > Checking PETSc installation... > ERROR: Unable to link with PETSc > ERROR: See "installed-arch-linux2-c-opt/lib/slepc/conf/configure.log" > file for details > > > > When I open the configure.log file, the contents are as follows: > > +---BEGIN------------------------------------------------------------- > +--------- > > ====================================================================== > ========== Starting Configure Run at Mon Jul 6 21:54:40 2020 > Configure Options: --prefix=/home/... > ... > ... > ====================================================================== > ========== > Checking PETSc installation... > #include "petscsnes.h" > int main() { > Vec v; Mat m; KSP k; > PetscInitializeNoArguments(); > VecCreate(PETSC_COMM_WORLD,&v); > MatCreate(PETSC_COMM_WORLD,&m); > KSPCreate(PETSC_COMM_WORLD,&k); > return 0; > } > mpicc -o checklink.o -c -fPIC -wd1572 -Ofast -xHost -I/mnt/disks/intel-2018-3-222-blade-runtime-env-2018-1-07-08-2018-132838/petsc_3.10.4_openmpi.4.0.3_intel2019.3_64index/include `pwd`/checklink.c > gcc: error: unrecognized command line option 'wd1572' > gmake: *** [checklink.o] Error 1 > > ERROR: Unable to link with PETSc > > +---END--------------------------------------------------------------- > +------- > > When I run mpicc -version, I get "gcc (GCC) 4.9.2" > > The PETSc version is 3.10.4, and It seems it was installed with openmpi and intel compiler (I might be wrong here). > > The SLEPc version is 3.10.2. > > Any suggestions? Should I set variables CC and CXX in the configuration command line for SLEPc? > > Thank you in advance, > > Ernesto. > > > Schlumberger-Private > From Antoine.Cote3 at USherbrooke.ca Tue Jul 7 08:14:21 2020 From: Antoine.Cote3 at USherbrooke.ca (=?iso-8859-1?Q?Antoine_C=F4t=E9?=) Date: Tue, 7 Jul 2020 13:14:21 +0000 Subject: [petsc-users] Parallel sort in Petsc Message-ID: Hi, I need to compute the permutations to sort scalar values of a Vec object in decreasing order. I've been developing on a single process for now, using VecGetArrayRead() to extract the values and PetscSortRealWithPermutation() to get the permutations. I would like to run the code on multiple processes. From these links, I can see it's an issue : https://lists.mcs.anl.gov/pipermail/petsc-users/2013-November/019628.html https://lists.mcs.anl.gov/pipermail/petsc-users/2009-June/004621.html One work around I've been considering : get the max scalar value using VecMax() ; compute a scaling factor that makes this max value equals the largest integer value tolerated by PetscInt ; scale all values of Vec and convert them to integers ; use PetscParallelSortInt() ; inverse the scaling on (now sorted) integers values and convert them back to scalars. This is only a patch really, the main issues being : (1) I would get a sorted Vec, not the permutations and (2) there would be a lost of information when converting scalars to int values. Isn't there a way to parallel sort real values using Petsc? Thank you very much! Antoine C?t? -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Tue Jul 7 09:09:30 2020 From: knepley at gmail.com (Matthew Knepley) Date: Tue, 7 Jul 2020 10:09:30 -0400 Subject: [petsc-users] Parallel sort in Petsc In-Reply-To: References: Message-ID: On Tue, Jul 7, 2020 at 9:14 AM Antoine C?t? wrote: > Hi, > > I need to compute the permutations to sort scalar values of a Vec object > in decreasing order. I've been developing on a single process for now, > using VecGetArrayRead() to extract the values > and PetscSortRealWithPermutation() to get the permutations. > > I would like to run the code on multiple processes. From these links, I > can see it's an issue : > https://lists.mcs.anl.gov/pipermail/petsc-users/2013-November/019628.html > https://lists.mcs.anl.gov/pipermail/petsc-users/2009-June/004621.html > > One work around I've been considering : get the max scalar value using > VecMax() ; compute a scaling factor that makes this max value equals the > largest integer value tolerated by PetscInt ; scale all values of Vec and > convert them to integers ; use PetscParallelSortInt() ; inverse the scaling > on (now sorted) integers values and convert them back to scalars. > > This is only a patch really, the main issues being : (1) I would get a > sorted Vec, not the permutations and (2) there would be a lost of > information when converting scalars to int values. > > Isn't there a way to parallel sort real values using Petsc? > I looked at the code. It does not depend on the value type do you could just copy it and replace all the Int types with Real. It would be nice to abstract the type and comparator away, but that is more work in C. Thanks, Matt > Thank you very much! > > Antoine C?t? > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From jed at jedbrown.org Tue Jul 7 09:49:36 2020 From: jed at jedbrown.org (Jed Brown) Date: Tue, 07 Jul 2020 08:49:36 -0600 Subject: [petsc-users] Parallel sort in Petsc In-Reply-To: References: Message-ID: <87k0zfqt1b.fsf@jedbrown.org> Matthew Knepley writes: > On Tue, Jul 7, 2020 at 9:14 AM Antoine C?t? > wrote: > >> Hi, >> >> I need to compute the permutations to sort scalar values of a Vec object >> in decreasing order. I've been developing on a single process for now, >> using VecGetArrayRead() to extract the values >> and PetscSortRealWithPermutation() to get the permutations. >> >> I would like to run the code on multiple processes. From these links, I >> can see it's an issue : >> https://lists.mcs.anl.gov/pipermail/petsc-users/2013-November/019628.html >> https://lists.mcs.anl.gov/pipermail/petsc-users/2009-June/004621.html >> >> One work around I've been considering : get the max scalar value using >> VecMax() ; compute a scaling factor that makes this max value equals the >> largest integer value tolerated by PetscInt ; scale all values of Vec and >> convert them to integers ; use PetscParallelSortInt() ; inverse the scaling >> on (now sorted) integers values and convert them back to scalars. >> >> This is only a patch really, the main issues being : (1) I would get a >> sorted Vec, not the permutations and (2) there would be a lost of >> information when converting scalars to int values. >> >> Isn't there a way to parallel sort real values using Petsc? >> > > I looked at the code. It does not depend on the value type do you could > just copy it and replace all the Int types with Real. It would be nice to > abstract the type and comparator away, but that is more work in C. It can be done with a comparison function like with qsort/qsort_r from libc. It'll get inlined at reasonable optimization and would still be useful for sorting user objects. From junchao.zhang at gmail.com Tue Jul 7 09:52:35 2020 From: junchao.zhang at gmail.com (Junchao Zhang) Date: Tue, 7 Jul 2020 09:52:35 -0500 Subject: [petsc-users] Parallel sort in Petsc In-Reply-To: References: Message-ID: Gather all values to rank 0 and do a sequential sort there, then profile to see whether it is a performance bottleneck in your code. --Junchao Zhang On Tue, Jul 7, 2020 at 8:14 AM Antoine C?t? wrote: > Hi, > > I need to compute the permutations to sort scalar values of a Vec object > in decreasing order. I've been developing on a single process for now, > using VecGetArrayRead() to extract the values > and PetscSortRealWithPermutation() to get the permutations. > > I would like to run the code on multiple processes. From these links, I > can see it's an issue : > https://lists.mcs.anl.gov/pipermail/petsc-users/2013-November/019628.html > https://lists.mcs.anl.gov/pipermail/petsc-users/2009-June/004621.html > > One work around I've been considering : get the max scalar value using > VecMax() ; compute a scaling factor that makes this max value equals the > largest integer value tolerated by PetscInt ; scale all values of Vec and > convert them to integers ; use PetscParallelSortInt() ; inverse the scaling > on (now sorted) integers values and convert them back to scalars. > > This is only a patch really, the main issues being : (1) I would get a > sorted Vec, not the permutations and (2) there would be a lost of > information when converting scalars to int values. > > Isn't there a way to parallel sort real values using Petsc? > > Thank you very much! > > Antoine C?t? > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From Eugenio.Aulisa at ttu.edu Tue Jul 7 10:40:05 2020 From: Eugenio.Aulisa at ttu.edu (Aulisa, Eugenio) Date: Tue, 7 Jul 2020 15:40:05 +0000 Subject: [petsc-users] Help with generalized eigenvalue problem Message-ID: Can somebody help me figure it out the solver options for the attached generalized eigenvalue problem? A x = lambda B x where A and B have the same null space, with 5 zero eigenvalues. The only non indefinite eigenvalue should be 41.1892 The problem is Hermite semi-positive so automatically slepc should purify it, but to be sure I added EPSSetPurify(eps, PETSC_TRUE); I guess with my non-solver-options slepc is still trying to do the inverse of the full B matrix with an LU decomposition and gets zero pivot. This is what I get as a PETSC error Message [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [0]PETSC ERROR: Zero pivot in LU factorization: https://www.mcs.anl.gov/petsc/documentation/faq.html#zeropivot [0]PETSC ERROR: Bad LU factorization [0]PETSC ERROR: See https://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [0]PETSC ERROR: Petsc Development GIT revision: v3.11.3-2263-gce77f2ed1a GIT Date: 2019-09-26 13:31:14 -0500 [0]PETSC ERROR: Nitsche_ex4a on a arch-linux2-c-opt named linux-8biu by eaulisa Tue Jul 7 10:17:55 2020 [0]PETSC ERROR: Configure options --with-debugging=0 --with-x=1 COPTFLAGS="-O3 -march=native -mtune=native" CXXOPTFLAGS="-O3 -march=native -mtune=native" FOPTFLAGS="-O3 -march=native -mtune=native" --download-openmpi=1 --download-fblaslapack=1 --download-hdf5=1 --download-metis=1 --download-parmetis=1 --with-shared-libraries=1 --download-blacs=1 --download-scalapack=1 --download-mumps=1 --download-suitesparse [0]PETSC ERROR: #1 MatLUFactor_SeqDense() line 633 in /home/eaulisa/software/petsc/src/mat/impls/dense/seq/dense.c [0]PETSC ERROR: #2 MatLUFactorNumeric_SeqDense() line 432 in /home/eaulisa/software/petsc/src/mat/impls/dense/seq/dense.c [0]PETSC ERROR: #3 MatLUFactorNumeric() line 3056 in /home/eaulisa/software/petsc/src/mat/interface/matrix.c [0]PETSC ERROR: #4 PCSetUp_LU() line 126 in /home/eaulisa/software/petsc/src/ksp/pc/impls/factor/lu/lu.c [0]PETSC ERROR: #5 PCSetUp() line 894 in /home/eaulisa/software/petsc/src/ksp/pc/interface/precon.c [0]PETSC ERROR: #6 KSPSetUp() line 377 in /home/eaulisa/software/petsc/src/ksp/ksp/interface/itfunc.c [0]PETSC ERROR: #7 STSetUp_Shift() line 119 in /home/eaulisa/software/slepc/src/sys/classes/st/impls/shift/shift.c [0]PETSC ERROR: #8 STSetUp() line 271 in /home/eaulisa/software/slepc/src/sys/classes/st/interface/stsolve.c [0]PETSC ERROR: #9 EPSSetUp() line 273 in /home/eaulisa/software/slepc/src/eps/interface/epssetup.c [0]PETSC ERROR: #10 EPSSolve() line 136 in /home/eaulisa/software/slepc/src/eps/interface/epssolve.c Thanks Eugenio Eugenio Aulisa Department of Mathematics and Statistics, Texas Tech University Lubbock TX, 79409-1042 room: 226 http://www.math.ttu.edu/~eaulisa/ phone: (806) 834-6684 fax: (806) 742-1112 -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: genEigen.cpp Type: text/x-c++src Size: 1914 bytes Desc: genEigen.cpp URL: From jroman at dsic.upv.es Tue Jul 7 11:01:49 2020 From: jroman at dsic.upv.es (Jose E. Roman) Date: Tue, 7 Jul 2020 18:01:49 +0200 Subject: [petsc-users] Help with generalized eigenvalue problem In-Reply-To: References: Message-ID: I don't know what options you are using, shift-and-invert or not, which, target. Anyway, if A and B have a common null space, then A-sigma*B will always be singular, so the KSP solver will have to deal with a singular system. If you pass the nullspace vectors to the EPS solver via https://slepc.upv.es/documentation/current/docs/manualpages/EPS/EPSSetDeflationSpace.html then these vectors will also be attached to the KSP solver. Also, since you have MUMPS, try adding -st_pc_factor_mat_solver_type mumps (see section 3.4.1 of the SLEPc users manual). MUMPS should be quite robust with respect to singular linear systems. Jose > El 7 jul 2020, a las 17:40, Aulisa, Eugenio escribi?: > > Can somebody help me figure it out the solver options for the attached generalized eigenvalue problem? > > A x = lambda B x > > where A and B have the same null space, with 5 zero eigenvalues. > > The only non indefinite eigenvalue should be 41.1892 > > The problem is Hermite semi-positive so automatically slepc should purify it, but to be sure I added > > EPSSetPurify(eps, PETSC_TRUE); > > I guess with my non-solver-options slepc is still trying to do the inverse of the full > B matrix with an LU decomposition and gets zero pivot. > > This is what I get as a PETSC error Message > > [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- > [0]PETSC ERROR: Zero pivot in LU factorization: https://www.mcs.anl.gov/petsc/documentation/faq.html#zeropivot > [0]PETSC ERROR: Bad LU factorization > [0]PETSC ERROR: See https://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. > [0]PETSC ERROR: Petsc Development GIT revision: v3.11.3-2263-gce77f2ed1a GIT Date: 2019-09-26 13:31:14 -0500 > [0]PETSC ERROR: Nitsche_ex4a on a arch-linux2-c-opt named linux-8biu by eaulisa Tue Jul 7 10:17:55 2020 > [0]PETSC ERROR: Configure options --with-debugging=0 --with-x=1 COPTFLAGS="-O3 -march=native -mtune=native" CXXOPTFLAGS="-O3 -march=native -mtune=native" FOPTFLAGS="-O3 -march=native -mtune=native" --download-openmpi=1 --download-fblaslapack=1 --download-hdf5=1 --download-metis=1 --download-parmetis=1 --with-shared-libraries=1 --download-blacs=1 --download-scalapack=1 --download-mumps=1 --download-suitesparse > [0]PETSC ERROR: #1 MatLUFactor_SeqDense() line 633 in /home/eaulisa/software/petsc/src/mat/impls/dense/seq/dense.c > [0]PETSC ERROR: #2 MatLUFactorNumeric_SeqDense() line 432 in /home/eaulisa/software/petsc/src/mat/impls/dense/seq/dense.c > [0]PETSC ERROR: #3 MatLUFactorNumeric() line 3056 in /home/eaulisa/software/petsc/src/mat/interface/matrix.c > [0]PETSC ERROR: #4 PCSetUp_LU() line 126 in /home/eaulisa/software/petsc/src/ksp/pc/impls/factor/lu/lu.c > [0]PETSC ERROR: #5 PCSetUp() line 894 in /home/eaulisa/software/petsc/src/ksp/pc/interface/precon.c > [0]PETSC ERROR: #6 KSPSetUp() line 377 in /home/eaulisa/software/petsc/src/ksp/ksp/interface/itfunc.c > [0]PETSC ERROR: #7 STSetUp_Shift() line 119 in /home/eaulisa/software/slepc/src/sys/classes/st/impls/shift/shift.c > [0]PETSC ERROR: #8 STSetUp() line 271 in /home/eaulisa/software/slepc/src/sys/classes/st/interface/stsolve.c > [0]PETSC ERROR: #9 EPSSetUp() line 273 in /home/eaulisa/software/slepc/src/eps/interface/epssetup.c > [0]PETSC ERROR: #10 EPSSolve() line 136 in /home/eaulisa/software/slepc/src/eps/interface/epssolve.c > > > > Thanks > Eugenio > > > > > Eugenio Aulisa > > Department of Mathematics and Statistics, > Texas Tech University > Lubbock TX, 79409-1042 > room: 226 > http://www.math.ttu.edu/~eaulisa/ > phone: (806) 834-6684 > fax: (806) 742-1112 > > > > From paololampitella at hotmail.com Wed Jul 8 04:12:38 2020 From: paololampitella at hotmail.com (Paolo Lampitella) Date: Wed, 8 Jul 2020 09:12:38 +0000 Subject: [petsc-users] R: R: PETSc and Windows 10 In-Reply-To: References: <32B37EFD-DDFA-41AE-ACC7-CB5E6AE57996@enseeiht.fr> <0558CCF7-8203-4520-8673-3B0945C43233@enseeiht.fr> <20AB9F7A-4471-498B-9A78-AA48163900CB@enseeiht.fr> <94A8E366-2361-409E-84CE-B90EA54D9AF9@enseeiht.fr> <849E7E3B-8B54-4B40-A2B7-8DD5B55428B3@enseeiht.fr> , <246CA2C6-E66F-4EF0-8600-81AB741D17D8@enseeiht.fr>, , , Message-ID: Ok, I see, but this seems to translate to compiling MS-MPI with mingw in cygwin by myself. Should be doable, maybe following how they managed to do that in MSYS in the first place (the instructions are available for each package). Actually, now I see I might have messed up when working in Cygwin because the mingw tool there might still be a cross compiler (while it isn?t in MSYS2), so it might have required something like ?--host=x86_64-w64-mingw32? Will update if I make any progress on this Paolo Inviato da Posta per Windows 10 Da: Satish Balay Inviato: luned? 6 luglio 2020 20:31 A: Paolo Lampitella Cc: petsc-users; Pierre Jolivet Oggetto: Re: [petsc-users] R: PETSc and Windows 10 I was thinking in terms of: If using mingw-gcc from cygwin - then it could be used in the same way as mingw-gcc in msys2 is used - i.e with MS-MPI etc.. [one can install mingw-gcc in cygwin - which is different than cygwin native gcc - perhaps this is similar to mingw-gcc install in msys2] I haven't tried this though.. Likely cygwin doesn't have the equivalent of mingw-w64-x86_64-msmpi - for easy use of MS-MPI from mingw-gfortran Satish On Mon, 6 Jul 2020, Paolo Lampitella wrote: > Dear Satish, > > Yes indeed, or at least that is my understanding. Still, my experience so far with Cygwin has been, let?s say, controversial. > > I wasn?t able to compile myself MPICH, with both gcc and mingw. > > When having PETSc compile also MPICH, I was successful only with gcc but not mingw. > > I didn?t even try compiling OpenMPI with mingw, as PETSc compilation already failed using the OpenMPI available trough cygwin libraries (which is based on gcc and not mingw). > > Not sure if this is my fault, but in the end it didn?t go well > > Paolo > > Inviato da Posta per Windows 10 > > Da: Satish Balay > Inviato: domenica 5 luglio 2020 23:50 > A: Paolo Lampitella > Cc: Pierre Jolivet; petsc-users > Oggetto: Re: [petsc-users] PETSc and Windows 10 > > Sounds like there are different mingw tools and msys2 tools. > > So I guess one could use mingw compilers even from cygwin [using cygwin tools] - i.e mingw compilers don't really need msys2 tools to work. > > Satish > > On Sun, 5 Jul 2020, Paolo Lampitella wrote: > > > Unfortunately, even PETSC_ARCH=i didn't work out. And while with-single-library=0 wasn't really appealing to me, it worked but only to later fail on make test. > > > > I guess all these differences are due to the fortran bindings and/or gcc 10. > > > > However, until I discover how they are different, I guess I'll be fine with /usr/bin/ar > > > > Paolo > > > > > > > > Inviato da smartphone Samsung Galaxy. > > > > > > > > -------- Messaggio originale -------- > > Da: Paolo Lampitella > > Data: 05/07/20 14:00 (GMT+01:00) > > A: Pierre Jolivet > > Cc: Matthew Knepley , petsc-users > > Oggetto: RE: [petsc-users] PETSc and Windows 10 > > > > Thank you very much Pierre. > > > > I'll keep you informed in case I see any relevant change from the tests when using your suggestion. > > > > Paolo > > > > > > > > Inviato da smartphone Samsung Galaxy. > > > > > > > > -------- Messaggio originale -------- > > Da: Pierre Jolivet > > Data: 05/07/20 13:45 (GMT+01:00) > > A: Paolo Lampitella > > Cc: Matthew Knepley , petsc-users > > Oggetto: Re: [petsc-users] PETSc and Windows 10 > > > > Hello Paolo, > > > > On 5 Jul 2020, at 1:15 PM, Paolo Lampitella > wrote: > > > > Dear all, > > > > I just want to update you on my journey to PETSc compilation in Windows under MSYS2+MINGW64 > > > > Unfortunately, I haven?t been able to compile petsc-slepc trough Freefem but, as my final goal required also Fortran bindings (but I only needed blas, lapack, metis and hypre), I decided to follow my own route using the useful information from Pierre. > > > > > > * I started by installing MPI from https://www.microsoft.com/en-us/download/details.aspx?id=100593. I don?t think the SDK is actually needed in my specific workflow, but I installed it as well together with mpisetup. > > * Then I installed MSYS2 just following the wizard. Opened the MSYS2 terminal and updated with pacman -Syuu, closed if asked, reopened it and used again pacman -Syuu several times until no more updates were available. Closed it and opened it back. > > * Under the MSYS2 terminal installed just the following packages: > > > > > > > > * pacman -S base-devel git gcc gcc-fortran > > * pacman -S mingw-w64-x86_64-toolchain > > * pacman -S mingw-w64-x86_64-cmake > > * pacman -S mingw-w64-x86_64-msmpi > > > > > > > > * Closed the MSYS2 terminal and opened the MINGW64 one, went to /mingw64/include and compiled my mpi module following https://www.scivision.dev/windows-mpi-msys2/: > > > > > > > > * gfortran -c mpi.f90 -fno-range-check -fallow-invalid-boz > > > > > > However, I will keep an eye on the MS-MPI GitHub repository because the fortran side seems to be far from perfect. > > > > > > * Then I downloaded the 3.13.3 version of petsc and configured it, still under the MINGW64 terminal, with the following command: > > > > > > /usr/bin/python ./configure --prefix=/home/paolo/petsc --with-ar=/usr/bin/ar > > --with-shared-libraries=0 --with-debugging=0 --with-windows-graphics=0 --with-x=0 > > COPTFLAGS="-O3 -mtune=native" > > CXXOPTFLAGS="-O3 -mtune=native" > > FOPTFLAGS="-O3 -mtune=native" > > FFLAGS=-fallow-invalid-boz > > --with-mpiexec="/C/Program\ Files/Microsoft\ MPI/Bin/mpiexec" > > --download-fblaslapack --download-metis --download-hypre > > --download-metis-cmake-arguments='-G "MSYS Makefiles"' > > --download-hypre-configure-arguments="--build=x86_64-linux-gnu --host=x86_64-linux-gnu" > > > > Note that I just bypassed uninstalling python in mingw64 (which doesn?t work) by using /usr/bin/python and that, as opposed to Pierre, I needed to also use the MSYS2 archiver (/usr/bin/ar) as opposed to the mingw64 one (/mingw64/bin/ar that shows up in the Pierre configure) as also mentioned here http://hillyuan.blogspot.com/2017/11/build-petsc-in-windows-under-mingw64.html, probably because of this issue https://stackoverflow.com/questions/37504625/ar-on-msys2-shell-receives-truncated-paths-when-called-from-makefile. > > > > You are right that you can avoid deinstalling mingw-w64-x86_64-python if you can supply the proper Python yourself (we don?t have that luxury in our Makefile). > > If you want to avoid using that AR, and stick to /mingw64/bin/ar (not sure what the pros and cons are), you can either: > > - use another PETSC_ARCH (very short, like pw, for petsc-windows); > > - use --with-single-library=0. > > See this post on GitLab https://gitlab.com/petsc/petsc/-/issues/647#note_373507681 > > The OS I?m referring to is indeed my Windows + MSYS2 box. > > > > Thanks, > > Pierre > > > > Then make all, make install and make check all went smooth. Also, I don?t know exactly what with-x=0 and with-windows-graphics=0 do, but I think it is stuff that I don?t need (yet configure worked with windows-graphics as well). > > > > > > * Finally I launched make test. As some tests failed, I replicated the same install procedure on all the systems I have available on this same Windows machine (Ubuntu 20.04 and Centos 8 under a VirtualBox 6.0.22 VM, Ubuntu 20.04 under WSL1 and the MSYS2-MINGW64 toolchain). I am attaching a file with the results printed to screen (not sure about which file should be used for a comparison/check). Note, however, that the tests in MSYS2 started with some cyclic reference issues for some .mod files, but this doesn?t show up in any file I could check. > > > > > > I am still left with some doubts about the archiver, the cyclic reference errors and the differences in the test results, but I am able to link my code with petsc. Unfortunately, as this Windows porting is part of a large code restructuring, I can?t do much more with it, now, from my code. But if you can suggest some specific tutorial to use as test also for the parallel, I would be glad to dig deeper into the matter. > > > > Best regards > > > > Paolo > > > > Inviato da Posta per Windows 10 > > > > Da: Pierre Jolivet > > Inviato: marted? 30 giugno 2020 15:22 > > A: Paolo Lampitella > > Cc: Matthew Knepley; petsc-users > > Oggetto: Re: [petsc-users] PETSc and Windows 10 > > > > Please use the 3.13.2 tarball, this was fixed by Satish in the previous commit I already linked (https://gitlab.com/petsc/petsc/-/commit/2cd8068296b34e127f055bb32f556e3599f17523). > > (If you want FreeFEM to do the dirty work for you, just switch to the develop branch, and redo ?make petsc-slepc?) > > But I think you?ve got everything you need now for a smooth compilation :) > > > > Thanks, > > Pierre > > > > > > On 30 Jun 2020, at 3:09 PM, Paolo Lampitella > wrote: > > > > Dear Pierre, > > > > thanks for the fast response. Unfortunately it still fails, but now in the configure of ScaLAPACK > > (which means that it went ok for slepc, tetgen, metis, parmetis, ptscotch, superlu and suitesparse). > > > > The way I applied the modification is by manually editing the Makefile in the 3rdparty/ff-petsc folder, adding -fallow-invalid-boz to both CFLAGS and FFLAGS (this entry added by me). Then executed make petsc-slepc. > > > > As my project is much less ambitious, I have a good feeling that I will be able to use your Makefile successfully, but as I am kind of slow I tought that it would have been useful for you to know. The configure.log is attached. This time the error is: > > > > Rank mismatch between actual argument at (1) and actual argument at (2) (scalar and rank-1) > > > > in subroutine pclarf.f of ScaLAPACK. > > > > However, before attampting with my project, I have few questions about your Makefile, in particular this piece: > > > > --with-mpi-lib=/c/Windows/System32/msmpi.dll --with-mpi-include=/home/paolo/FreeFem-sources/3rdparty/include/msmpi --with-mpiexec="/C/Program\ Files/Microsoft\ MPI/Bin/mpiexec" > > > > I see from MPI.py that I should not use ?--with-mpi-lib/include? if I want to use my now working mpi wrappers. Is this correct? > > > > Paolo > > > > Inviato da Posta per Windows 10 > > > > Da: Pierre Jolivet > > Inviato: luned? 29 giugno 2020 21:37 > > A: Paolo Lampitella > > Cc: Matthew Knepley; petsc-users > > Oggetto: Re: [petsc-users] PETSc and Windows 10 > > > > I do not give up easily on Windows problems: > > 1) that?s around 50% of our (FreeFEM) user-base (and I want them to use PETSc and SLEPc, ofc?) > > 2) most people I work with from corporations just have Windows laptops/desktops and I always recommend MSYS because it?s very lightweight and you can pass .exe around > > 3) I?ve bothered enough Satish, Jed, and Matt on GitLab to take (at least partially) the blame now when it doesn?t work on MSYS > > > > That being said, the magic keyword is the added flag FFLAGS="-fallow-invalid-boz" (see, I told you ./configure issues were easier to deal with than the others). > > Here you?ll see that everything goes through just fine (sorry, it took me a long time to post this because everything is slow on my VM): > > 1) http://jolivet.perso.enseeiht.fr/win10/configure.log > > 2) http://jolivet.perso.enseeiht.fr/win10/make.log (both steps #1 and #2 in MSYS terminal, gcc/gfortran 10, MS-MPI see screenshot) > > 3) http://jolivet.perso.enseeiht.fr/win10/ex2.txt (Command Prompt, 4 processes + MUMPS, I can send you the .exe if you want to try on your machine) > > I just realize that I didn?t generate the Fortran bindings, but you can see I compiled MUMPS and ScaLAPACK, so that shouldn?t be a problem. > > Or if there is a problem, we will need to fix this in PETSc. > > > > I?ll push this added flag to the FreeFEM repo, thanks for reminding me of the brokenness of gcc/gfortran 10 + MS-MPI. > > Here is to hoping this won?t affect PETSc ./configure with previous gcc/gfortran version (unlikely, this option is apparently 13-year old https://gcc.gnu.org/bugzilla/show_bug.cgi?id=29471) > > > > Let me know of the next hiccup, if any. > > Thanks, > > Pierre > > > > > > > > On 29 Jun 2020, at 8:09 PM, Paolo Lampitella > wrote: > > > > Dear Pierre, > > > > thanks again for your time > > > > I guess there is no way for me to use the toolchain you are using (I don?t remember having any choice on which version of MSYS or GCC I could install) > > > > Paolo > > > > Inviato da Posta per Windows 10 > > > > Da: Pierre Jolivet > > Inviato: luned? 29 giugno 2020 20:01 > > A: Matthew Knepley > > Cc: Paolo Lampitella; petsc-users > > Oggetto: Re: [petsc-users] PETSc and Windows 10 > > > > > > > > > > > > > > On 29 Jun 2020, at 7:47 PM, Matthew Knepley > wrote: > > > > On Mon, Jun 29, 2020 at 1:35 PM Paolo Lampitella > wrote: > > Dear Pierre, sorry to bother you, but I already have some issues. What I did: > > > > > > * pacman -R mingw-w64-x86_64-python mingw-w64-x86_64-gdb (is gdb also troublesome?) > > * Followed points 6 and 7 at https://doc.freefem.org/introduction/installation.html#compilation-on-windows > > > > I first got a warning on the configure at point 6, as ?disable-hips is not recognized. Then, on make ?petsc-slepc? of point 7 (no SUDO=sudo flag was necessary) I got to this point: > > > > tar xzf ../pkg/petsc-lite-3.13.0.tar.gz > > patch -p1 < petsc-suitesparse.patch > > patching file petsc-3.13.0/config/BuildSystem/config/packages/SuiteSparse.py > > touch petsc-3.13.0/tag-tar > > cd petsc-3.13.0 && ./configure MAKEFLAGS='' \ > > --prefix=/home/paolo/freefem/ff-petsc//r \ > > --with-debugging=0 COPTFLAGS='-O3 -mtune=generic' CXXOPTFLAGS='-O3 -mtune=generic' FOPTFLAGS='-O3 -mtune=generic' --with-cxx-dialect=C++11 --with-ssl=0 --with-x=0 --with-fortran-bindings=0 --with-shared-libraries=0 --with-cc='gcc' --with-cxx='g++' --with-fc='gfortran' CXXFLAGS='-fno-stack-protector' CFLAGS='-fno-stack-protector' --with-scalar-type=real --with-mpi-lib='/c/Windows/System32/msmpi.dll' --with-mpi-include='/home/paolo/FreeFem-sources/3rdparty/include/msmpi' --with-mpiexec='/C/Program\ Files/Microsoft\ MPI/Bin/mpiexec' --with-blaslapack-include='' --with-blaslapack-lib='/mingw64/bin/libopenblas.dll' --download-scalapack --download-metis --download-ptscotch --download-mumps --download-hypre --download-parmetis --download-superlu --download-suitesparse --download-tetgen --download-slepc '--download-metis-cmake-arguments=-G "MSYS Makefiles"' '--download-parmetis-cmake-arguments=-G "MSYS Makefiles"' '--download-superlu-cmake-arguments=-G "MSYS Makefiles"' ' -- > download-hypre-configure-arguments=--build=x86_64-linux-gnu --host=x86_64-linux-gnu' PETSC_ARCH=fr > > =============================================================================== > > Configuring PETSc to compile on your system > > =============================================================================== > > TESTING: FortranMPICheck from config.packages.MPI(config/BuildSystem/config/pack******************************************************************************* > > UNABLE to CONFIGURE with GIVEN OPTIONS (see configure.log for details): > > ------------------------------------------------------------------------------- > > Fortran error! mpi_init() could not be located! > > ******************************************************************************* > > > > make: *** [Makefile:210: petsc-3.13.0/tag-conf-real] Errore 1 > > > > Note that I didn?t add anything to any PATH variable, because this is not mentioned in your documentation. > > > > On a side note, this is the same error I got when trying to build PETSc in Cygwin with the default OpenMPI available in Cygwin. > > > > I am attaching the configure.log? it seems to me that the error comes from the configure trying to include the mpif.h in your folder and not using the -fallow-invalid-boz flag that I had to use, for example, to compile mpi.f90 into mpi.mod > > > > But I?m not sure why this is happening > > > > Pierre, > > > > Could this be due to gcc 10? > > > > Sorry, I?m slow. You are right. Our workers use gcc 9, everything is fine, but I see on my VM which I updated that I use gcc 10 and had to disable Fortran, I guess the MUMPS run I showcased was with a prior PETSc build. > > I?ll try to resolve this and will keep you posted. > > They really caught a lot of people off guard with gfortran 10? > > > > Thanks, > > Pierre > > > > > > > > > > Executing: gfortran -c -o /tmp/petsc-ur0cff6a/config.libraries/conftest.o -I/tmp/petsc-ur0cff6a/config.compilers -I/tmp/petsc-ur0cff6a/config.setCompilers -I/tmp/petsc-ur0cff6a/config.compilersFortran -I/tmp/petsc-ur0cff6a/config.libraries -Wall -ffree-line-length-0 -Wno-unused-dummy-argument -O3 -mtune=generic -I/home/paolo/FreeFem-sources/3rdparty/include/msmpi /tmp/petsc-ur0cff6a/config.libraries/conftest.F90 > > Possible ERROR while running compiler: exit code 1 > > stderr: > > C:/msys64/home/paolo/FreeFem-sources/3rdparty/include/msmpi/mpif.h:227:36: > > > > 227 | PARAMETER (MPI_DATATYPE_NULL=z'0c000000') > > | 1 > > Error: BOZ literal constant at (1) is neither a data-stmt-constant nor an actual argument to INT, REAL, DBLE, or CMPLX intrinsic function [see '-fno-allow-invalid-boz'] > > C:/msys64/home/paolo/FreeFem-sources/3rdparty/include/msmpi/mpif.h:303:27: > > > > 303 | PARAMETER (MPI_CHAR=z'4c000101') > > | 1 > > Error: BOZ literal constant at (1) is neither a data-stmt-constant nor an actual argument to INT, REAL, DBLE, or CMPLX intrinsic function [see '-fno-allow-invalid-boz'] > > C:/msys64/home/paolo/FreeFem-sources/3rdparty/include/msmpi/mpif.h:305:36: > > > > 305 | PARAMETER (MPI_UNSIGNED_CHAR=z'4c000102') > > | 1 > > > > Thanks, > > > > Matt > > > > Thanks > > > > Paolo > > > > Inviato da Posta per Windows 10 > > > > Da: Pierre Jolivet > > Inviato: luned? 29 giugno 2020 18:34 > > A: Paolo Lampitella > > Cc: Satish Balay; petsc-users > > Oggetto: Re: [petsc-users] PETSc and Windows 10 > > > > > > > > On 29 Jun 2020, at 6:27 PM, Paolo Lampitella > wrote: > > > > I think I made the first step of having mingw64 from msys2 working with ms-mpi. > > > > I found that the issue I was having was related to: > > > > https://gcc.gnu.org/bugzilla/show_bug.cgi?id=91556 > > > > and, probably (but impossible to check now), I was using an msys2 and/or mingw mpi package before this fix: > > > > https://github.com/msys2/MINGW-packages/commit/11b4cff3d2ec7411037b692b0ad5a9f3e9b9978d#diff-eac59989e3096be97d940c8f47b50fba > > > > Admittedly, I never used gcc 10 before on any machine. Still, I feel that reporting that sort of error in that way is, > > at least, misleading (I would have preferred the initial implementation as mentioned in the gcc bug track). > > > > A second thing that I was not used to, and made me more uncertain of the procedure I was following, is having to compile myself the mpi module. There are several version of this out there, but I decided to stick with this one: > > > > https://www.scivision.dev/windows-mpi-msys2/ > > > > even if there seems to be no need to include -fno-range-check and the current mpi.f90 version is different from the mpif.h as reported here: > > > > https://github.com/microsoft/Microsoft-MPI/issues/33 > > > > which, to me, are both signs of lack of attention on the fortran side by those that maintain this thing. > > > > In summary, this is the procedure I followed so far (on a 64 bit machine with Windows 10): > > > > > > * Install MSYS2 from https://www.msys2.org/ and just follow the install wizard > > * Open the MSYS2 terminal and execute: pacman -Syuu > > * Close the terminal when asked and reopen it > > * Keep executing ?pacman -Syuu? until nothing else needs to be updated > > * Close the MSYS2 terminal and reopen it (I guess because was in paranoid mode), then install packages with: > > > > > > pacman -S base-devel git gcc gcc-fortran bsdcpio lndir pax-git unzip > > pacman -S mingw-w64-x86_64-toolchain > > pacman -S mingw-w64-x86_64-msmpi > > pacman -S mingw-w64-x86_64-cmake > > pacman -S mingw-w64-x86_64-freeglut > > pacman -S mingw-w64-x86_64-gsl > > pacman -S mingw-w64-x86_64-libmicroutils > > pacman -S mingw-w64-x86_64-hdf5 > > pacman -S mingw-w64-x86_64-openblas > > pacman -S mingw-w64-x86_64-arpack > > pacman -S mingw-w64-x86_64-jq > > > > This set should include all the libraries mentioned by Pierre and/or used by his Jenkins, as the final scope here is to have PETSc and dependencies working. But I think that for pure MPI one could stop to msmpi (even, maybe, just install msmpi and have the dependencies figured out by pacman). Honestly, I don?t remember the exact order I used to install the packages, but this should not affect things. Also, as I was still in paranoid mode, I kept executing ?pacman -Syuu? after each package was installed. After this, close the MSYS2 terminal. > > > > > > * Open the MINGW64 terminal and create the .mod file out of the mpi.f90 file, as mentioned here https://www.scivision.dev/windows-mpi-msys2/, with: > > > > > > cd /mingw64/include > > gfortran mpif90 -c -fno-range-check -fallow-invalid-boz > > > > Ah, yes, that?s new to gfortran 10 (we use gfortran 9 on our workers), which is now what?s ship with MSYS2 (we haven?t updated yet). Sorry that I forgot about that. > > > > This is needed to ?USE mpi? (as opposed to INCLUDE ?mpif.h?) > > > > > > * Install the latest MS-MPI (both sdk and setup) from https://www.microsoft.com/en-us/download/details.aspx?id=100593 > > > > > > At this point I?ve been able to compile (using the MINGW64 terminal) different mpi test programs and they run as expected in the classical Windows prompt. I added this function to my .bashrc in MSYS2 in order to easily copy the required dependencies out of MSYS: > > > > function copydep() { ldd $1 | grep "=> /$2" | awk '{print $3}' | xargs -I '{}' cp -v '{}' .; } > > > > which can be used, with the MINGW64 terminal, by navigating to the folder where the final executable, say, my.exe, resides (even if under a Windows path) and executing: > > > > copydep my.exe mingw64 > > > > This, of course, must be done before actually trying to execute the .exe in the windows cmd prompt. > > > > Hopefully, I should now be able to follow Pierre?s instructions for PETSc (but first I wanna give a try to the system python before removing it) > > > > Looks like the hard part is over. It?s usually easier to deal with ./configure issues. > > If you have weird errors like ?incomplete Cygwin install? or whatever, this is the kind of issues I was referring to earlier. > > In that case, what I?d suggest is just, as before: > > pacman -R mingw-w64-x86_64-python mingw-w64-x86_64-gdb > > pacman -S python > > > > Thanks, > > Pierre > > > > Thanks > > > > Paolo > > > > > > > > > > -- > > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. > > -- Norbert Wiener > > > > https://www.cse.buffalo.edu/~knepley/>> > > > > > > > > > > > > > > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From Eugenio.Aulisa at ttu.edu Wed Jul 8 08:55:41 2020 From: Eugenio.Aulisa at ttu.edu (Aulisa, Eugenio) Date: Wed, 8 Jul 2020 13:55:41 +0000 Subject: [petsc-users] Help with generalized eigenvalue problem In-Reply-To: References: , Message-ID: Dear Jose, Thank you for your answer. I tried MLU but it did not help. I also tried to eliminate the null space of B, but I do not know it apriori, and at least to my knowledge of PETSC-SLEPC there is not an explicit way to find it but to find all the 0 eigenvectors of B, which again it requires the inverse of B. I am solving similar small problems millions of times, and this works fine most of the time, but once in a while it fails with the same LU zero pivot exception. One thing I tried yesterday, and it seams to work all the times, is perturbing the diagonal of B of a small factor, += 1.0e-10 ||B||, but I do not know how theoretically it sounds with all these indeterminate eigenvalues, maybe you have a clue on this. Thanks, Eugenio Eugenio Aulisa Department of Mathematics and Statistics, Texas Tech University Lubbock TX, 79409-1042 room: 226 http://www.math.ttu.edu/~eaulisa/ phone: (806) 834-6684 fax: (806) 742-1112 ________________________________ From: Jose E. Roman Sent: Tuesday, July 7, 2020 11:01 AM To: Aulisa, Eugenio Cc: petsc-users at mcs.anl.gov ; Kara, Erdi Subject: Re: [petsc-users] Help with generalized eigenvalue problem I don't know what options you are using, shift-and-invert or not, which, target. Anyway, if A and B have a common null space, then A-sigma*B will always be singular, so the KSP solver will have to deal with a singular system. If you pass the nullspace vectors to the EPS solver via https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fslepc.upv.es%2Fdocumentation%2Fcurrent%2Fdocs%2Fmanualpages%2FEPS%2FEPSSetDeflationSpace.html&data=02%7C01%7CEugenio.Aulisa%40ttu.edu%7C5b98b0c29ec246ed6c7908d8228f12bb%7C178a51bf8b2049ffb65556245d5c173c%7C0%7C0%7C637297345179456168&sdata=dP0Hi2ZPRo6rYubPYRK%2FimS0fZKBV9%2BJy7N%2BtXS6b30%3D&reserved=0 then these vectors will also be attached to the KSP solver. Also, since you have MUMPS, try adding -st_pc_factor_mat_solver_type mumps (see section 3.4.1 of the SLEPc users manual). MUMPS should be quite robust with respect to singular linear systems. Jose > El 7 jul 2020, a las 17:40, Aulisa, Eugenio escribi?: > > Can somebody help me figure it out the solver options for the attached generalized eigenvalue problem? > > A x = lambda B x > > where A and B have the same null space, with 5 zero eigenvalues. > > The only non indefinite eigenvalue should be 41.1892 > > The problem is Hermite semi-positive so automatically slepc should purify it, but to be sure I added > > EPSSetPurify(eps, PETSC_TRUE); > > I guess with my non-solver-options slepc is still trying to do the inverse of the full > B matrix with an LU decomposition and gets zero pivot. > > This is what I get as a PETSC error Message > > [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- > [0]PETSC ERROR: Zero pivot in LU factorization: https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.mcs.anl.gov%2Fpetsc%2Fdocumentation%2Ffaq.html%23zeropivot&data=02%7C01%7CEugenio.Aulisa%40ttu.edu%7C5b98b0c29ec246ed6c7908d8228f12bb%7C178a51bf8b2049ffb65556245d5c173c%7C0%7C0%7C637297345179456168&sdata=5JnAFaw%2B4SsXiMYji4FQFLoTZOpE8r7ggYWSH1Jqti0%3D&reserved=0 > [0]PETSC ERROR: Bad LU factorization > [0]PETSC ERROR: See https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.mcs.anl.gov%2Fpetsc%2Fdocumentation%2Ffaq.html&data=02%7C01%7CEugenio.Aulisa%40ttu.edu%7C5b98b0c29ec246ed6c7908d8228f12bb%7C178a51bf8b2049ffb65556245d5c173c%7C0%7C0%7C637297345179456168&sdata=HUax0aUTExvaWCeZFBty%2F7SZf3bRxPq0h3hwnhbDXwE%3D&reserved=0 for trouble shooting. > [0]PETSC ERROR: Petsc Development GIT revision: v3.11.3-2263-gce77f2ed1a GIT Date: 2019-09-26 13:31:14 -0500 > [0]PETSC ERROR: Nitsche_ex4a on a arch-linux2-c-opt named linux-8biu by eaulisa Tue Jul 7 10:17:55 2020 > [0]PETSC ERROR: Configure options --with-debugging=0 --with-x=1 COPTFLAGS="-O3 -march=native -mtune=native" CXXOPTFLAGS="-O3 -march=native -mtune=native" FOPTFLAGS="-O3 -march=native -mtune=native" --download-openmpi=1 --download-fblaslapack=1 --download-hdf5=1 --download-metis=1 --download-parmetis=1 --with-shared-libraries=1 --download-blacs=1 --download-scalapack=1 --download-mumps=1 --download-suitesparse > [0]PETSC ERROR: #1 MatLUFactor_SeqDense() line 633 in /home/eaulisa/software/petsc/src/mat/impls/dense/seq/dense.c > [0]PETSC ERROR: #2 MatLUFactorNumeric_SeqDense() line 432 in /home/eaulisa/software/petsc/src/mat/impls/dense/seq/dense.c > [0]PETSC ERROR: #3 MatLUFactorNumeric() line 3056 in /home/eaulisa/software/petsc/src/mat/interface/matrix.c > [0]PETSC ERROR: #4 PCSetUp_LU() line 126 in /home/eaulisa/software/petsc/src/ksp/pc/impls/factor/lu/lu.c > [0]PETSC ERROR: #5 PCSetUp() line 894 in /home/eaulisa/software/petsc/src/ksp/pc/interface/precon.c > [0]PETSC ERROR: #6 KSPSetUp() line 377 in /home/eaulisa/software/petsc/src/ksp/ksp/interface/itfunc.c > [0]PETSC ERROR: #7 STSetUp_Shift() line 119 in /home/eaulisa/software/slepc/src/sys/classes/st/impls/shift/shift.c > [0]PETSC ERROR: #8 STSetUp() line 271 in /home/eaulisa/software/slepc/src/sys/classes/st/interface/stsolve.c > [0]PETSC ERROR: #9 EPSSetUp() line 273 in /home/eaulisa/software/slepc/src/eps/interface/epssetup.c > [0]PETSC ERROR: #10 EPSSolve() line 136 in /home/eaulisa/software/slepc/src/eps/interface/epssolve.c > > > > Thanks > Eugenio > > > > > Eugenio Aulisa > > Department of Mathematics and Statistics, > Texas Tech University > Lubbock TX, 79409-1042 > room: 226 > http://www.math.ttu.edu/~eaulisa/ > phone: (806) 834-6684 > fax: (806) 742-1112 > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jroman at dsic.upv.es Wed Jul 8 10:01:26 2020 From: jroman at dsic.upv.es (Jose E. Roman) Date: Wed, 8 Jul 2020 17:01:26 +0200 Subject: [petsc-users] Help with generalized eigenvalue problem In-Reply-To: References: Message-ID: <66509B35-BCC5-4413-8807-5C7077E49B2E@dsic.upv.es> In a standard symmetric eigenproblem, a small perturbation of the matrix results in a small perturbation of the eigenvalues. In generalized eigenproblems, especially with singular matrices, I don't think perturbation theory guarantees this in all cases. What do you mean by small problems? Do you have millions of problems of size 6? In that case you should use EPSLAPACK. It will not factor the matrix (if solved as a non-symmetric problem). Jose > El 8 jul 2020, a las 15:55, Aulisa, Eugenio escribi?: > > Dear Jose, > > Thank you for your answer. > > I tried MLU but it did not help. > > I also tried to eliminate the null space of B, but I do not know it apriori, > and at least to my knowledge of PETSC-SLEPC there is not an explicit way to find it > but to find all the 0 eigenvectors of B, which again it requires the inverse of B. > > I am solving similar small problems millions of times, and this works fine most of the time, > but once in a while it fails with the same LU zero pivot exception. > > One thing I tried yesterday, and it seams to work all the times, > is perturbing the diagonal of B of a small factor, += 1.0e-10 ||B||, > but I do not know how theoretically it sounds with all these indeterminate eigenvalues, > maybe you have a clue on this. > > Thanks, > Eugenio > > > > Eugenio Aulisa > > Department of Mathematics and Statistics, > Texas Tech University > Lubbock TX, 79409-1042 > room: 226 > http://www.math.ttu.edu/~eaulisa/ > phone: (806) 834-6684 > fax: (806) 742-1112 > > > > > From: Jose E. Roman > Sent: Tuesday, July 7, 2020 11:01 AM > To: Aulisa, Eugenio > Cc: petsc-users at mcs.anl.gov ; Kara, Erdi > Subject: Re: [petsc-users] Help with generalized eigenvalue problem > > I don't know what options you are using, shift-and-invert or not, which, target. Anyway, if A and B have a common null space, then A-sigma*B will always be singular, so the KSP solver will have to deal with a singular system. > > If you pass the nullspace vectors to the EPS solver via https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fslepc.upv.es%2Fdocumentation%2Fcurrent%2Fdocs%2Fmanualpages%2FEPS%2FEPSSetDeflationSpace.html&data=02%7C01%7CEugenio.Aulisa%40ttu.edu%7C5b98b0c29ec246ed6c7908d8228f12bb%7C178a51bf8b2049ffb65556245d5c173c%7C0%7C0%7C637297345179456168&sdata=dP0Hi2ZPRo6rYubPYRK%2FimS0fZKBV9%2BJy7N%2BtXS6b30%3D&reserved=0 then these vectors will also be attached to the KSP solver. > > Also, since you have MUMPS, try adding > -st_pc_factor_mat_solver_type mumps > (see section 3.4.1 of the SLEPc users manual). MUMPS should be quite robust with respect to singular linear systems. > > Jose > > > > El 7 jul 2020, a las 17:40, Aulisa, Eugenio escribi?: > > > > Can somebody help me figure it out the solver options for the attached generalized eigenvalue problem? > > > > A x = lambda B x > > > > where A and B have the same null space, with 5 zero eigenvalues. > > > > The only non indefinite eigenvalue should be 41.1892 > > > > The problem is Hermite semi-positive so automatically slepc should purify it, but to be sure I added > > > > EPSSetPurify(eps, PETSC_TRUE); > > > > I guess with my non-solver-options slepc is still trying to do the inverse of the full > > B matrix with an LU decomposition and gets zero pivot. > > > > This is what I get as a PETSC error Message > > > > [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- > > [0]PETSC ERROR: Zero pivot in LU factorization: https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.mcs.anl.gov%2Fpetsc%2Fdocumentation%2Ffaq.html%23zeropivot&data=02%7C01%7CEugenio.Aulisa%40ttu.edu%7C5b98b0c29ec246ed6c7908d8228f12bb%7C178a51bf8b2049ffb65556245d5c173c%7C0%7C0%7C637297345179456168&sdata=5JnAFaw%2B4SsXiMYji4FQFLoTZOpE8r7ggYWSH1Jqti0%3D&reserved=0 > > [0]PETSC ERROR: Bad LU factorization > > [0]PETSC ERROR: See https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.mcs.anl.gov%2Fpetsc%2Fdocumentation%2Ffaq.html&data=02%7C01%7CEugenio.Aulisa%40ttu.edu%7C5b98b0c29ec246ed6c7908d8228f12bb%7C178a51bf8b2049ffb65556245d5c173c%7C0%7C0%7C637297345179456168&sdata=HUax0aUTExvaWCeZFBty%2F7SZf3bRxPq0h3hwnhbDXwE%3D&reserved=0 for trouble shooting. > > [0]PETSC ERROR: Petsc Development GIT revision: v3.11.3-2263-gce77f2ed1a GIT Date: 2019-09-26 13:31:14 -0500 > > [0]PETSC ERROR: Nitsche_ex4a on a arch-linux2-c-opt named linux-8biu by eaulisa Tue Jul 7 10:17:55 2020 > > [0]PETSC ERROR: Configure options --with-debugging=0 --with-x=1 COPTFLAGS="-O3 -march=native -mtune=native" CXXOPTFLAGS="-O3 -march=native -mtune=native" FOPTFLAGS="-O3 -march=native -mtune=native" --download-openmpi=1 --download-fblaslapack=1 --download-hdf5=1 --download-metis=1 --download-parmetis=1 --with-shared-libraries=1 --download-blacs=1 --download-scalapack=1 --download-mumps=1 --download-suitesparse > > [0]PETSC ERROR: #1 MatLUFactor_SeqDense() line 633 in /home/eaulisa/software/petsc/src/mat/impls/dense/seq/dense.c > > [0]PETSC ERROR: #2 MatLUFactorNumeric_SeqDense() line 432 in /home/eaulisa/software/petsc/src/mat/impls/dense/seq/dense.c > > [0]PETSC ERROR: #3 MatLUFactorNumeric() line 3056 in /home/eaulisa/software/petsc/src/mat/interface/matrix.c > > [0]PETSC ERROR: #4 PCSetUp_LU() line 126 in /home/eaulisa/software/petsc/src/ksp/pc/impls/factor/lu/lu.c > > [0]PETSC ERROR: #5 PCSetUp() line 894 in /home/eaulisa/software/petsc/src/ksp/pc/interface/precon.c > > [0]PETSC ERROR: #6 KSPSetUp() line 377 in /home/eaulisa/software/petsc/src/ksp/ksp/interface/itfunc.c > > [0]PETSC ERROR: #7 STSetUp_Shift() line 119 in /home/eaulisa/software/slepc/src/sys/classes/st/impls/shift/shift.c > > [0]PETSC ERROR: #8 STSetUp() line 271 in /home/eaulisa/software/slepc/src/sys/classes/st/interface/stsolve.c > > [0]PETSC ERROR: #9 EPSSetUp() line 273 in /home/eaulisa/software/slepc/src/eps/interface/epssetup.c > > [0]PETSC ERROR: #10 EPSSolve() line 136 in /home/eaulisa/software/slepc/src/eps/interface/epssolve.c > > > > > > > > Thanks > > Eugenio > > > > > > > > > > Eugenio Aulisa > > > > Department of Mathematics and Statistics, > > Texas Tech University > > Lubbock TX, 79409-1042 > > room: 226 > > http://www.math.ttu.edu/~eaulisa/ > > phone: (806) 834-6684 > > fax: (806) 742-1112 > > > > > > > > > From Eugenio.Aulisa at ttu.edu Wed Jul 8 10:06:23 2020 From: Eugenio.Aulisa at ttu.edu (Aulisa, Eugenio) Date: Wed, 8 Jul 2020 15:06:23 +0000 Subject: [petsc-users] Help with generalized eigenvalue problem In-Reply-To: <66509B35-BCC5-4413-8807-5C7077E49B2E@dsic.upv.es> References: , <66509B35-BCC5-4413-8807-5C7077E49B2E@dsic.upv.es> Message-ID: yes the size varies from 6 to 100, I will try EPSLAPACK or EIGEN to see how it works Thanks again Eugenio Eugenio Aulisa Department of Mathematics and Statistics, Texas Tech University Lubbock TX, 79409-1042 room: 226 http://www.math.ttu.edu/~eaulisa/ phone: (806) 834-6684 fax: (806) 742-1112 ________________________________ From: Jose E. Roman Sent: Wednesday, July 8, 2020 10:01 AM To: Aulisa, Eugenio Cc: petsc-users at mcs.anl.gov ; Kara, Erdi Subject: Re: [petsc-users] Help with generalized eigenvalue problem In a standard symmetric eigenproblem, a small perturbation of the matrix results in a small perturbation of the eigenvalues. In generalized eigenproblems, especially with singular matrices, I don't think perturbation theory guarantees this in all cases. What do you mean by small problems? Do you have millions of problems of size 6? In that case you should use EPSLAPACK. It will not factor the matrix (if solved as a non-symmetric problem). Jose > El 8 jul 2020, a las 15:55, Aulisa, Eugenio escribi?: > > Dear Jose, > > Thank you for your answer. > > I tried MLU but it did not help. > > I also tried to eliminate the null space of B, but I do not know it apriori, > and at least to my knowledge of PETSC-SLEPC there is not an explicit way to find it > but to find all the 0 eigenvectors of B, which again it requires the inverse of B. > > I am solving similar small problems millions of times, and this works fine most of the time, > but once in a while it fails with the same LU zero pivot exception. > > One thing I tried yesterday, and it seams to work all the times, > is perturbing the diagonal of B of a small factor, += 1.0e-10 ||B||, > but I do not know how theoretically it sounds with all these indeterminate eigenvalues, > maybe you have a clue on this. > > Thanks, > Eugenio > > > > Eugenio Aulisa > > Department of Mathematics and Statistics, > Texas Tech University > Lubbock TX, 79409-1042 > room: 226 > http://www.math.ttu.edu/~eaulisa/ > phone: (806) 834-6684 > fax: (806) 742-1112 > > > > > From: Jose E. Roman > Sent: Tuesday, July 7, 2020 11:01 AM > To: Aulisa, Eugenio > Cc: petsc-users at mcs.anl.gov ; Kara, Erdi > Subject: Re: [petsc-users] Help with generalized eigenvalue problem > > I don't know what options you are using, shift-and-invert or not, which, target. Anyway, if A and B have a common null space, then A-sigma*B will always be singular, so the KSP solver will have to deal with a singular system. > > If you pass the nullspace vectors to the EPS solver via https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fslepc.upv.es%2Fdocumentation%2Fcurrent%2Fdocs%2Fmanualpages%2FEPS%2FEPSSetDeflationSpace.html&data=02%7C01%7CEugenio.Aulisa%40ttu.edu%7C018ff09384674480f2e808d8234fce62%7C178a51bf8b2049ffb65556245d5c173c%7C0%7C0%7C637298172963555933&sdata=iPBn%2FYR7mw2Q8EEjTQ%2BhDphUftUctK2lLj1KjnPgUZw%3D&reserved=0 then these vectors will also be attached to the KSP solver. > > Also, since you have MUMPS, try adding > -st_pc_factor_mat_solver_type mumps > (see section 3.4.1 of the SLEPc users manual). MUMPS should be quite robust with respect to singular linear systems. > > Jose > > > > El 7 jul 2020, a las 17:40, Aulisa, Eugenio escribi?: > > > > Can somebody help me figure it out the solver options for the attached generalized eigenvalue problem? > > > > A x = lambda B x > > > > where A and B have the same null space, with 5 zero eigenvalues. > > > > The only non indefinite eigenvalue should be 41.1892 > > > > The problem is Hermite semi-positive so automatically slepc should purify it, but to be sure I added > > > > EPSSetPurify(eps, PETSC_TRUE); > > > > I guess with my non-solver-options slepc is still trying to do the inverse of the full > > B matrix with an LU decomposition and gets zero pivot. > > > > This is what I get as a PETSC error Message > > > > [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- > > [0]PETSC ERROR: Zero pivot in LU factorization: https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.mcs.anl.gov%2Fpetsc%2Fdocumentation%2Ffaq.html%23zeropivot&data=02%7C01%7CEugenio.Aulisa%40ttu.edu%7C018ff09384674480f2e808d8234fce62%7C178a51bf8b2049ffb65556245d5c173c%7C0%7C0%7C637298172963555933&sdata=4ZYkCiAdAjVgNZeQBJXyFmL%2BaICquqkJvc7gQ%2BNs0QQ%3D&reserved=0 > > [0]PETSC ERROR: Bad LU factorization > > [0]PETSC ERROR: See https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.mcs.anl.gov%2Fpetsc%2Fdocumentation%2Ffaq.html&data=02%7C01%7CEugenio.Aulisa%40ttu.edu%7C018ff09384674480f2e808d8234fce62%7C178a51bf8b2049ffb65556245d5c173c%7C0%7C0%7C637298172963555933&sdata=SV%2F7S0Se8T4ftzroH%2FAYJKB6GDmnrZVlwfCsKI4M1bo%3D&reserved=0 for trouble shooting. > > [0]PETSC ERROR: Petsc Development GIT revision: v3.11.3-2263-gce77f2ed1a GIT Date: 2019-09-26 13:31:14 -0500 > > [0]PETSC ERROR: Nitsche_ex4a on a arch-linux2-c-opt named linux-8biu by eaulisa Tue Jul 7 10:17:55 2020 > > [0]PETSC ERROR: Configure options --with-debugging=0 --with-x=1 COPTFLAGS="-O3 -march=native -mtune=native" CXXOPTFLAGS="-O3 -march=native -mtune=native" FOPTFLAGS="-O3 -march=native -mtune=native" --download-openmpi=1 --download-fblaslapack=1 --download-hdf5=1 --download-metis=1 --download-parmetis=1 --with-shared-libraries=1 --download-blacs=1 --download-scalapack=1 --download-mumps=1 --download-suitesparse > > [0]PETSC ERROR: #1 MatLUFactor_SeqDense() line 633 in /home/eaulisa/software/petsc/src/mat/impls/dense/seq/dense.c > > [0]PETSC ERROR: #2 MatLUFactorNumeric_SeqDense() line 432 in /home/eaulisa/software/petsc/src/mat/impls/dense/seq/dense.c > > [0]PETSC ERROR: #3 MatLUFactorNumeric() line 3056 in /home/eaulisa/software/petsc/src/mat/interface/matrix.c > > [0]PETSC ERROR: #4 PCSetUp_LU() line 126 in /home/eaulisa/software/petsc/src/ksp/pc/impls/factor/lu/lu.c > > [0]PETSC ERROR: #5 PCSetUp() line 894 in /home/eaulisa/software/petsc/src/ksp/pc/interface/precon.c > > [0]PETSC ERROR: #6 KSPSetUp() line 377 in /home/eaulisa/software/petsc/src/ksp/ksp/interface/itfunc.c > > [0]PETSC ERROR: #7 STSetUp_Shift() line 119 in /home/eaulisa/software/slepc/src/sys/classes/st/impls/shift/shift.c > > [0]PETSC ERROR: #8 STSetUp() line 271 in /home/eaulisa/software/slepc/src/sys/classes/st/interface/stsolve.c > > [0]PETSC ERROR: #9 EPSSetUp() line 273 in /home/eaulisa/software/slepc/src/eps/interface/epssetup.c > > [0]PETSC ERROR: #10 EPSSolve() line 136 in /home/eaulisa/software/slepc/src/eps/interface/epssolve.c > > > > > > > > Thanks > > Eugenio > > > > > > > > > > Eugenio Aulisa > > > > Department of Mathematics and Statistics, > > Texas Tech University > > Lubbock TX, 79409-1042 > > room: 226 > > http://www.math.ttu.edu/~eaulisa/ > > phone: (806) 834-6684 > > fax: (806) 742-1112 > > > > > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jroman at dsic.upv.es Wed Jul 8 10:07:57 2020 From: jroman at dsic.upv.es (Jose E. Roman) Date: Wed, 8 Jul 2020 17:07:57 +0200 Subject: [petsc-users] Help with generalized eigenvalue problem In-Reply-To: References: <66509B35-BCC5-4413-8807-5C7077E49B2E@dsic.upv.es> Message-ID: <009BBC4D-9E7C-481E-81FC-5A7B98920777@dsic.upv.es> Run with -eps_type lapack Eigen is not interfaced from SLEPc. Jose > El 8 jul 2020, a las 17:06, Aulisa, Eugenio escribi?: > > yes the size varies from 6 to 100, > > I will try EPSLAPACK or EIGEN to see how it works > > Thanks again > Eugenio > > > > > > Eugenio Aulisa > > Department of Mathematics and Statistics, > Texas Tech University > Lubbock TX, 79409-1042 > room: 226 > http://www.math.ttu.edu/~eaulisa/ > phone: (806) 834-6684 > fax: (806) 742-1112 > > > > > From: Jose E. Roman > Sent: Wednesday, July 8, 2020 10:01 AM > To: Aulisa, Eugenio > Cc: petsc-users at mcs.anl.gov ; Kara, Erdi > Subject: Re: [petsc-users] Help with generalized eigenvalue problem > > In a standard symmetric eigenproblem, a small perturbation of the matrix results in a small perturbation of the eigenvalues. In generalized eigenproblems, especially with singular matrices, I don't think perturbation theory guarantees this in all cases. > > What do you mean by small problems? Do you have millions of problems of size 6? In that case you should use EPSLAPACK. It will not factor the matrix (if solved as a non-symmetric problem). > > Jose > > > > El 8 jul 2020, a las 15:55, Aulisa, Eugenio escribi?: > > > > Dear Jose, > > > > Thank you for your answer. > > > > I tried MLU but it did not help. > > > > I also tried to eliminate the null space of B, but I do not know it apriori, > > and at least to my knowledge of PETSC-SLEPC there is not an explicit way to find it > > but to find all the 0 eigenvectors of B, which again it requires the inverse of B. > > > > I am solving similar small problems millions of times, and this works fine most of the time, > > but once in a while it fails with the same LU zero pivot exception. > > > > One thing I tried yesterday, and it seams to work all the times, > > is perturbing the diagonal of B of a small factor, += 1.0e-10 ||B||, > > but I do not know how theoretically it sounds with all these indeterminate eigenvalues, > > maybe you have a clue on this. > > > > Thanks, > > Eugenio > > > > > > > > Eugenio Aulisa > > > > Department of Mathematics and Statistics, > > Texas Tech University > > Lubbock TX, 79409-1042 > > room: 226 > > http://www.math.ttu.edu/~eaulisa/ > > phone: (806) 834-6684 > > fax: (806) 742-1112 > > > > > > > > > > From: Jose E. Roman > > Sent: Tuesday, July 7, 2020 11:01 AM > > To: Aulisa, Eugenio > > Cc: petsc-users at mcs.anl.gov ; Kara, Erdi > > Subject: Re: [petsc-users] Help with generalized eigenvalue problem > > > > I don't know what options you are using, shift-and-invert or not, which, target. Anyway, if A and B have a common null space, then A-sigma*B will always be singular, so the KSP solver will have to deal with a singular system. > > > > If you pass the nullspace vectors to the EPS solver via https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fslepc.upv.es%2Fdocumentation%2Fcurrent%2Fdocs%2Fmanualpages%2FEPS%2FEPSSetDeflationSpace.html&data=02%7C01%7CEugenio.Aulisa%40ttu.edu%7C018ff09384674480f2e808d8234fce62%7C178a51bf8b2049ffb65556245d5c173c%7C0%7C0%7C637298172963555933&sdata=iPBn%2FYR7mw2Q8EEjTQ%2BhDphUftUctK2lLj1KjnPgUZw%3D&reserved=0 then these vectors will also be attached to the KSP solver. > > > > Also, since you have MUMPS, try adding > > -st_pc_factor_mat_solver_type mumps > > (see section 3.4.1 of the SLEPc users manual). MUMPS should be quite robust with respect to singular linear systems. > > > > Jose > > > > > > > El 7 jul 2020, a las 17:40, Aulisa, Eugenio escribi?: > > > > > > Can somebody help me figure it out the solver options for the attached generalized eigenvalue problem? > > > > > > A x = lambda B x > > > > > > where A and B have the same null space, with 5 zero eigenvalues. > > > > > > The only non indefinite eigenvalue should be 41.1892 > > > > > > The problem is Hermite semi-positive so automatically slepc should purify it, but to be sure I added > > > > > > EPSSetPurify(eps, PETSC_TRUE); > > > > > > I guess with my non-solver-options slepc is still trying to do the inverse of the full > > > B matrix with an LU decomposition and gets zero pivot. > > > > > > This is what I get as a PETSC error Message > > > > > > [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- > > > [0]PETSC ERROR: Zero pivot in LU factorization: https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.mcs.anl.gov%2Fpetsc%2Fdocumentation%2Ffaq.html%23zeropivot&data=02%7C01%7CEugenio.Aulisa%40ttu.edu%7C018ff09384674480f2e808d8234fce62%7C178a51bf8b2049ffb65556245d5c173c%7C0%7C0%7C637298172963555933&sdata=4ZYkCiAdAjVgNZeQBJXyFmL%2BaICquqkJvc7gQ%2BNs0QQ%3D&reserved=0 > > > [0]PETSC ERROR: Bad LU factorization > > > [0]PETSC ERROR: See https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.mcs.anl.gov%2Fpetsc%2Fdocumentation%2Ffaq.html&data=02%7C01%7CEugenio.Aulisa%40ttu.edu%7C018ff09384674480f2e808d8234fce62%7C178a51bf8b2049ffb65556245d5c173c%7C0%7C0%7C637298172963555933&sdata=SV%2F7S0Se8T4ftzroH%2FAYJKB6GDmnrZVlwfCsKI4M1bo%3D&reserved=0 for trouble shooting. > > > [0]PETSC ERROR: Petsc Development GIT revision: v3.11.3-2263-gce77f2ed1a GIT Date: 2019-09-26 13:31:14 -0500 > > > [0]PETSC ERROR: Nitsche_ex4a on a arch-linux2-c-opt named linux-8biu by eaulisa Tue Jul 7 10:17:55 2020 > > > [0]PETSC ERROR: Configure options --with-debugging=0 --with-x=1 COPTFLAGS="-O3 -march=native -mtune=native" CXXOPTFLAGS="-O3 -march=native -mtune=native" FOPTFLAGS="-O3 -march=native -mtune=native" --download-openmpi=1 --download-fblaslapack=1 --download-hdf5=1 --download-metis=1 --download-parmetis=1 --with-shared-libraries=1 --download-blacs=1 --download-scalapack=1 --download-mumps=1 --download-suitesparse > > > [0]PETSC ERROR: #1 MatLUFactor_SeqDense() line 633 in /home/eaulisa/software/petsc/src/mat/impls/dense/seq/dense.c > > > [0]PETSC ERROR: #2 MatLUFactorNumeric_SeqDense() line 432 in /home/eaulisa/software/petsc/src/mat/impls/dense/seq/dense.c > > > [0]PETSC ERROR: #3 MatLUFactorNumeric() line 3056 in /home/eaulisa/software/petsc/src/mat/interface/matrix.c > > > [0]PETSC ERROR: #4 PCSetUp_LU() line 126 in /home/eaulisa/software/petsc/src/ksp/pc/impls/factor/lu/lu.c > > > [0]PETSC ERROR: #5 PCSetUp() line 894 in /home/eaulisa/software/petsc/src/ksp/pc/interface/precon.c > > > [0]PETSC ERROR: #6 KSPSetUp() line 377 in /home/eaulisa/software/petsc/src/ksp/ksp/interface/itfunc.c > > > [0]PETSC ERROR: #7 STSetUp_Shift() line 119 in /home/eaulisa/software/slepc/src/sys/classes/st/impls/shift/shift.c > > > [0]PETSC ERROR: #8 STSetUp() line 271 in /home/eaulisa/software/slepc/src/sys/classes/st/interface/stsolve.c > > > [0]PETSC ERROR: #9 EPSSetUp() line 273 in /home/eaulisa/software/slepc/src/eps/interface/epssetup.c > > > [0]PETSC ERROR: #10 EPSSolve() line 136 in /home/eaulisa/software/slepc/src/eps/interface/epssolve.c > > > > > > > > > > > > Thanks > > > Eugenio > > > > > > > > > > > > > > > Eugenio Aulisa > > > > > > Department of Mathematics and Statistics, > > > Texas Tech University > > > Lubbock TX, 79409-1042 > > > room: 226 > > > http://www.math.ttu.edu/~eaulisa/ > > > phone: (806) 834-6684 > > > fax: (806) 742-1112 > > > > > > > > > > > > > > > From Eugenio.Aulisa at ttu.edu Wed Jul 8 10:29:37 2020 From: Eugenio.Aulisa at ttu.edu (Aulisa, Eugenio) Date: Wed, 8 Jul 2020 15:29:37 +0000 Subject: [petsc-users] Help with generalized eigenvalue problem In-Reply-To: <009BBC4D-9E7C-481E-81FC-5A7B98920777@dsic.upv.es> References: <66509B35-BCC5-4413-8807-5C7077E49B2E@dsic.upv.es> , <009BBC4D-9E7C-481E-81FC-5A7B98920777@dsic.upv.es> Message-ID: I tried with -esp_type lapack, but it still tries to do LU I attached the code without perturbation of B, probably something is missing in my way of setting up the problem. I also attached the eps_view with perturbation Thanks Eugenio Nitsche_genEigen -esp_type lapack [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [0]PETSC ERROR: Zero pivot in LU factorization: https://www.mcs.anl.gov/petsc/documentation/faq.html#zeropivot [0]PETSC ERROR: Zero pivot row 1 value 0. tolerance 2.22045e-14 [0]PETSC ERROR: See https://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [0]PETSC ERROR: Petsc Development GIT revision: v3.11.3-2263-gce77f2ed1a GIT Date: 2019-09-26 13:31:14 -0500 [0]PETSC ERROR: Nitsche_genEigen on a arch-linux2-c-opt named linux-8biu by eaulisa Wed Jul 8 10:09:31 2020 [0]PETSC ERROR: Configure options --with-debugging=0 --with-x=1 COPTFLAGS="-O3 -march=native -mtune=native" CXXOPTFLAGS="-O3 -march=native -mtune=native" FOPTFLAGS="-O3 -march=native -mtune=native" --download-openmpi=1 --download-fblaslapack=1 --download-hdf5=1 --download-metis=1 --download-parmetis=1 --with-shared-libraries=1 --download-blacs=1 --download-scalapack=1 --download-mumps=1 --download-suitesparse [0]PETSC ERROR: #1 MatPivotCheck_none() line 731 in /home/eaulisa/software/petsc/include/petsc/private/matimpl.h [0]PETSC ERROR: #2 MatPivotCheck() line 748 in /home/eaulisa/software/petsc/include/petsc/private/matimpl.h [0]PETSC ERROR: #3 MatLUFactorNumeric_SeqAIJ_Inode() line 1676 in /home/eaulisa/software/petsc/src/mat/impls/aij/seq/inode.c [0]PETSC ERROR: #4 MatLUFactorNumeric() line 3056 in /home/eaulisa/software/petsc/src/mat/interface/matrix.c [0]PETSC ERROR: #5 PCSetUp_LU() line 126 in /home/eaulisa/software/petsc/src/ksp/pc/impls/factor/lu/lu.c [0]PETSC ERROR: #6 PCSetUp() line 894 in /home/eaulisa/software/petsc/src/ksp/pc/interface/precon.c [0]PETSC ERROR: #7 KSPSetUp() line 377 in /home/eaulisa/software/petsc/src/ksp/ksp/interface/itfunc.c [0]PETSC ERROR: #8 STSetUp_Shift() line 119 in /home/eaulisa/software/slepc/src/sys/classes/st/impls/shift/shift.c [0]PETSC ERROR: #9 STSetUp() line 271 in /home/eaulisa/software/slepc/src/sys/classes/st/interface/stsolve.c [0]PETSC ERROR: #10 EPSSetUp() line 273 in /home/eaulisa/software/slepc/src/eps/interface/epssetup.c [0]PETSC ERROR: #11 EPSSolve() line 136 in /home/eaulisa/software/slepc/src/eps/interface/epssolve.c [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [0]PETSC ERROR: Argument out of range [0]PETSC ERROR: Argument 2 out of range [0]PETSC ERROR: See https://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [0]PETSC ERROR: Petsc Development GIT revision: v3.11.3-2263-gce77f2ed1a GIT Date: 2019-09-26 13:31:14 -0500 [0]PETSC ERROR: Nitsche_genEigen on a arch-linux2-c-opt named linux-8biu by eaulisa Wed Jul 8 10:09:31 2020 [0]PETSC ERROR: Configure options --with-debugging=0 --with-x=1 COPTFLAGS="-O3 -march=native -mtune=native" CXXOPTFLAGS="-O3 -march=native -mtune=native" FOPTFLAGS="-O3 -march=native -mtune=native" --download-openmpi=1 --download-fblaslapack=1 --download-hdf5=1 --download-metis=1 --download-parmetis=1 --with-shared-libraries=1 --download-blacs=1 --download-scalapack=1 --download-mumps=1 --download-suitesparse [0]PETSC ERROR: #12 EPSGetEigenpair() line 399 in /home/eaulisa/software/slepc/src/eps/interface/epssolve.c 2.96439e-323 Eugenio Aulisa Department of Mathematics and Statistics, Texas Tech University Lubbock TX, 79409-1042 room: 226 http://www.math.ttu.edu/~eaulisa/ phone: (806) 834-6684 fax: (806) 742-1112 ________________________________ From: Jose E. Roman Sent: Wednesday, July 8, 2020 10:07 AM To: Aulisa, Eugenio Cc: petsc-users at mcs.anl.gov ; Kara, Erdi Subject: Re: [petsc-users] Help with generalized eigenvalue problem Run with -eps_type lapack Eigen is not interfaced from SLEPc. Jose > El 8 jul 2020, a las 17:06, Aulisa, Eugenio escribi?: > > yes the size varies from 6 to 100, > > I will try EPSLAPACK or EIGEN to see how it works > > Thanks again > Eugenio > > > > > > Eugenio Aulisa > > Department of Mathematics and Statistics, > Texas Tech University > Lubbock TX, 79409-1042 > room: 226 > http://www.math.ttu.edu/~eaulisa/ > phone: (806) 834-6684 > fax: (806) 742-1112 > > > > > From: Jose E. Roman > Sent: Wednesday, July 8, 2020 10:01 AM > To: Aulisa, Eugenio > Cc: petsc-users at mcs.anl.gov ; Kara, Erdi > Subject: Re: [petsc-users] Help with generalized eigenvalue problem > > In a standard symmetric eigenproblem, a small perturbation of the matrix results in a small perturbation of the eigenvalues. In generalized eigenproblems, especially with singular matrices, I don't think perturbation theory guarantees this in all cases. > > What do you mean by small problems? Do you have millions of problems of size 6? In that case you should use EPSLAPACK. It will not factor the matrix (if solved as a non-symmetric problem). > > Jose > > > > El 8 jul 2020, a las 15:55, Aulisa, Eugenio escribi?: > > > > Dear Jose, > > > > Thank you for your answer. > > > > I tried MLU but it did not help. > > > > I also tried to eliminate the null space of B, but I do not know it apriori, > > and at least to my knowledge of PETSC-SLEPC there is not an explicit way to find it > > but to find all the 0 eigenvectors of B, which again it requires the inverse of B. > > > > I am solving similar small problems millions of times, and this works fine most of the time, > > but once in a while it fails with the same LU zero pivot exception. > > > > One thing I tried yesterday, and it seams to work all the times, > > is perturbing the diagonal of B of a small factor, += 1.0e-10 ||B||, > > but I do not know how theoretically it sounds with all these indeterminate eigenvalues, > > maybe you have a clue on this. > > > > Thanks, > > Eugenio > > > > > > > > Eugenio Aulisa > > > > Department of Mathematics and Statistics, > > Texas Tech University > > Lubbock TX, 79409-1042 > > room: 226 > > http://www.math.ttu.edu/~eaulisa/ > > phone: (806) 834-6684 > > fax: (806) 742-1112 > > > > > > > > > > From: Jose E. Roman > > Sent: Tuesday, July 7, 2020 11:01 AM > > To: Aulisa, Eugenio > > Cc: petsc-users at mcs.anl.gov ; Kara, Erdi > > Subject: Re: [petsc-users] Help with generalized eigenvalue problem > > > > I don't know what options you are using, shift-and-invert or not, which, target. Anyway, if A and B have a common null space, then A-sigma*B will always be singular, so the KSP solver will have to deal with a singular system. > > > > If you pass the nullspace vectors to the EPS solver via https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fslepc.upv.es%2Fdocumentation%2Fcurrent%2Fdocs%2Fmanualpages%2FEPS%2FEPSSetDeflationSpace.html&data=02%7C01%7CEugenio.Aulisa%40ttu.edu%7C48eb040474e44a2af2f208d82350b5a7%7C178a51bf8b2049ffb65556245d5c173c%7C0%7C0%7C637298176878634547&sdata=lL%2B5GTRzFgeviU%2BedhKieyXDO%2Fgyko2CLYGU%2BxtilB0%3D&reserved=0 then these vectors will also be attached to the KSP solver. > > > > Also, since you have MUMPS, try adding > > -st_pc_factor_mat_solver_type mumps > > (see section 3.4.1 of the SLEPc users manual). MUMPS should be quite robust with respect to singular linear systems. > > > > Jose > > > > > > > El 7 jul 2020, a las 17:40, Aulisa, Eugenio escribi?: > > > > > > Can somebody help me figure it out the solver options for the attached generalized eigenvalue problem? > > > > > > A x = lambda B x > > > > > > where A and B have the same null space, with 5 zero eigenvalues. > > > > > > The only non indefinite eigenvalue should be 41.1892 > > > > > > The problem is Hermite semi-positive so automatically slepc should purify it, but to be sure I added > > > > > > EPSSetPurify(eps, PETSC_TRUE); > > > > > > I guess with my non-solver-options slepc is still trying to do the inverse of the full > > > B matrix with an LU decomposition and gets zero pivot. > > > > > > This is what I get as a PETSC error Message > > > > > > [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- > > > [0]PETSC ERROR: Zero pivot in LU factorization: https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.mcs.anl.gov%2Fpetsc%2Fdocumentation%2Ffaq.html%23zeropivot&data=02%7C01%7CEugenio.Aulisa%40ttu.edu%7C48eb040474e44a2af2f208d82350b5a7%7C178a51bf8b2049ffb65556245d5c173c%7C0%7C0%7C637298176878634547&sdata=MfAJt1HSEK5wvhRzx0BM9OEEGxWjnlI%2FwTlzhN4BDeE%3D&reserved=0 > > > [0]PETSC ERROR: Bad LU factorization > > > [0]PETSC ERROR: See https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.mcs.anl.gov%2Fpetsc%2Fdocumentation%2Ffaq.html&data=02%7C01%7CEugenio.Aulisa%40ttu.edu%7C48eb040474e44a2af2f208d82350b5a7%7C178a51bf8b2049ffb65556245d5c173c%7C0%7C0%7C637298176878634547&sdata=NTc7lCzIo1Vj2sYmAehiE5b8ToXDTvN0wzDyh7JC30w%3D&reserved=0 for trouble shooting. > > > [0]PETSC ERROR: Petsc Development GIT revision: v3.11.3-2263-gce77f2ed1a GIT Date: 2019-09-26 13:31:14 -0500 > > > [0]PETSC ERROR: Nitsche_ex4a on a arch-linux2-c-opt named linux-8biu by eaulisa Tue Jul 7 10:17:55 2020 > > > [0]PETSC ERROR: Configure options --with-debugging=0 --with-x=1 COPTFLAGS="-O3 -march=native -mtune=native" CXXOPTFLAGS="-O3 -march=native -mtune=native" FOPTFLAGS="-O3 -march=native -mtune=native" --download-openmpi=1 --download-fblaslapack=1 --download-hdf5=1 --download-metis=1 --download-parmetis=1 --with-shared-libraries=1 --download-blacs=1 --download-scalapack=1 --download-mumps=1 --download-suitesparse > > > [0]PETSC ERROR: #1 MatLUFactor_SeqDense() line 633 in /home/eaulisa/software/petsc/src/mat/impls/dense/seq/dense.c > > > [0]PETSC ERROR: #2 MatLUFactorNumeric_SeqDense() line 432 in /home/eaulisa/software/petsc/src/mat/impls/dense/seq/dense.c > > > [0]PETSC ERROR: #3 MatLUFactorNumeric() line 3056 in /home/eaulisa/software/petsc/src/mat/interface/matrix.c > > > [0]PETSC ERROR: #4 PCSetUp_LU() line 126 in /home/eaulisa/software/petsc/src/ksp/pc/impls/factor/lu/lu.c > > > [0]PETSC ERROR: #5 PCSetUp() line 894 in /home/eaulisa/software/petsc/src/ksp/pc/interface/precon.c > > > [0]PETSC ERROR: #6 KSPSetUp() line 377 in /home/eaulisa/software/petsc/src/ksp/ksp/interface/itfunc.c > > > [0]PETSC ERROR: #7 STSetUp_Shift() line 119 in /home/eaulisa/software/slepc/src/sys/classes/st/impls/shift/shift.c > > > [0]PETSC ERROR: #8 STSetUp() line 271 in /home/eaulisa/software/slepc/src/sys/classes/st/interface/stsolve.c > > > [0]PETSC ERROR: #9 EPSSetUp() line 273 in /home/eaulisa/software/slepc/src/eps/interface/epssetup.c > > > [0]PETSC ERROR: #10 EPSSolve() line 136 in /home/eaulisa/software/slepc/src/eps/interface/epssolve.c > > > > > > > > > > > > Thanks > > > Eugenio > > > > > > > > > > > > > > > Eugenio Aulisa > > > > > > Department of Mathematics and Statistics, > > > Texas Tech University > > > Lubbock TX, 79409-1042 > > > room: 226 > > > http://www.math.ttu.edu/~eaulisa/ > > > phone: (806) 834-6684 > > > fax: (806) 742-1112 > > > > > > > > > > > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: genEigen.cpp Type: text/x-c++src Size: 2151 bytes Desc: genEigen.cpp URL: -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: eps_view.txt URL: From jroman at dsic.upv.es Wed Jul 8 10:34:46 2020 From: jroman at dsic.upv.es (Jose E. Roman) Date: Wed, 8 Jul 2020 17:34:46 +0200 Subject: [petsc-users] Help with generalized eigenvalue problem In-Reply-To: References: <66509B35-BCC5-4413-8807-5C7077E49B2E@dsic.upv.es> <009BBC4D-9E7C-481E-81FC-5A7B98920777@dsic.upv.es> Message-ID: <01563E8A-80EB-4E3A-ACAB-17E81CE40E41@dsic.upv.es> eps_view shows it is using Krylov-Schur, it seems you mistyped the option: -eps_type lapack instead of -esp_type lapack > El 8 jul 2020, a las 17:29, Aulisa, Eugenio escribi?: > > I tried with -esp_type lapack, but it still tries to do LU > > I attached the code without perturbation of B, > probably something is missing in my way of setting up the problem. > > I also attached the eps_view with perturbation > > Thanks > Eugenio > > Nitsche_genEigen -esp_type lapack > [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- > [0]PETSC ERROR: Zero pivot in LU factorization: https://www.mcs.anl.gov/petsc/documentation/faq.html#zeropivot > [0]PETSC ERROR: Zero pivot row 1 value 0. tolerance 2.22045e-14 > > [0]PETSC ERROR: See https://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. > [0]PETSC ERROR: Petsc Development GIT revision: v3.11.3-2263-gce77f2ed1a GIT Date: 2019-09-26 13:31:14 -0500 > [0]PETSC ERROR: Nitsche_genEigen on a arch-linux2-c-opt named linux-8biu by eaulisa Wed Jul 8 10:09:31 2020 > [0]PETSC ERROR: Configure options --with-debugging=0 --with-x=1 COPTFLAGS="-O3 -march=native -mtune=native" CXXOPTFLAGS="-O3 -march=native -mtune=native" FOPTFLAGS="-O3 -march=native -mtune=native" --download-openmpi=1 --download-fblaslapack=1 --download-hdf5=1 --download-metis=1 --download-parmetis=1 --with-shared-libraries=1 --download-blacs=1 --download-scalapack=1 --download-mumps=1 --download-suitesparse > [0]PETSC ERROR: #1 MatPivotCheck_none() line 731 in /home/eaulisa/software/petsc/include/petsc/private/matimpl.h > [0]PETSC ERROR: #2 MatPivotCheck() line 748 in /home/eaulisa/software/petsc/include/petsc/private/matimpl.h > [0]PETSC ERROR: #3 MatLUFactorNumeric_SeqAIJ_Inode() line 1676 in /home/eaulisa/software/petsc/src/mat/impls/aij/seq/inode.c > [0]PETSC ERROR: #4 MatLUFactorNumeric() line 3056 in /home/eaulisa/software/petsc/src/mat/interface/matrix.c > [0]PETSC ERROR: #5 PCSetUp_LU() line 126 in /home/eaulisa/software/petsc/src/ksp/pc/impls/factor/lu/lu.c > [0]PETSC ERROR: #6 PCSetUp() line 894 in /home/eaulisa/software/petsc/src/ksp/pc/interface/precon.c > [0]PETSC ERROR: #7 KSPSetUp() line 377 in /home/eaulisa/software/petsc/src/ksp/ksp/interface/itfunc.c > [0]PETSC ERROR: #8 STSetUp_Shift() line 119 in /home/eaulisa/software/slepc/src/sys/classes/st/impls/shift/shift.c > [0]PETSC ERROR: #9 STSetUp() line 271 in /home/eaulisa/software/slepc/src/sys/classes/st/interface/stsolve.c > [0]PETSC ERROR: #10 EPSSetUp() line 273 in /home/eaulisa/software/slepc/src/eps/interface/epssetup.c > [0]PETSC ERROR: #11 EPSSolve() line 136 in /home/eaulisa/software/slepc/src/eps/interface/epssolve.c > [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- > [0]PETSC ERROR: Argument out of range > [0]PETSC ERROR: Argument 2 out of range > [0]PETSC ERROR: See https://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. > [0]PETSC ERROR: Petsc Development GIT revision: v3.11.3-2263-gce77f2ed1a GIT Date: 2019-09-26 13:31:14 -0500 > [0]PETSC ERROR: Nitsche_genEigen on a arch-linux2-c-opt named linux-8biu by eaulisa Wed Jul 8 10:09:31 2020 > [0]PETSC ERROR: Configure options --with-debugging=0 --with-x=1 COPTFLAGS="-O3 -march=native -mtune=native" CXXOPTFLAGS="-O3 -march=native -mtune=native" FOPTFLAGS="-O3 -march=native -mtune=native" --download-openmpi=1 --download-fblaslapack=1 --download-hdf5=1 --download-metis=1 --download-parmetis=1 --with-shared-libraries=1 --download-blacs=1 --download-scalapack=1 --download-mumps=1 --download-suitesparse > [0]PETSC ERROR: #12 EPSGetEigenpair() line 399 in /home/eaulisa/software/slepc/src/eps/interface/epssolve.c > 2.96439e-323 > > Eugenio Aulisa > > Department of Mathematics and Statistics, > Texas Tech University > Lubbock TX, 79409-1042 > room: 226 > http://www.math.ttu.edu/~eaulisa/ > phone: (806) 834-6684 > fax: (806) 742-1112 > > > > > From: Jose E. Roman > Sent: Wednesday, July 8, 2020 10:07 AM > To: Aulisa, Eugenio > Cc: petsc-users at mcs.anl.gov ; Kara, Erdi > Subject: Re: [petsc-users] Help with generalized eigenvalue problem > > Run with -eps_type lapack > > Eigen is not interfaced from SLEPc. > > Jose > > > > El 8 jul 2020, a las 17:06, Aulisa, Eugenio escribi?: > > > > yes the size varies from 6 to 100, > > > > I will try EPSLAPACK or EIGEN to see how it works > > > > Thanks again > > Eugenio > > > > > > > > > > > > Eugenio Aulisa > > > > Department of Mathematics and Statistics, > > Texas Tech University > > Lubbock TX, 79409-1042 > > room: 226 > > http://www.math.ttu.edu/~eaulisa/ > > phone: (806) 834-6684 > > fax: (806) 742-1112 > > > > > > > > > > From: Jose E. Roman > > Sent: Wednesday, July 8, 2020 10:01 AM > > To: Aulisa, Eugenio > > Cc: petsc-users at mcs.anl.gov ; Kara, Erdi > > Subject: Re: [petsc-users] Help with generalized eigenvalue problem > > > > In a standard symmetric eigenproblem, a small perturbation of the matrix results in a small perturbation of the eigenvalues. In generalized eigenproblems, especially with singular matrices, I don't think perturbation theory guarantees this in all cases. > > > > What do you mean by small problems? Do you have millions of problems of size 6? In that case you should use EPSLAPACK. It will not factor the matrix (if solved as a non-symmetric problem). > > > > Jose > > > > > > > El 8 jul 2020, a las 15:55, Aulisa, Eugenio escribi?: > > > > > > Dear Jose, > > > > > > Thank you for your answer. > > > > > > I tried MLU but it did not help. > > > > > > I also tried to eliminate the null space of B, but I do not know it apriori, > > > and at least to my knowledge of PETSC-SLEPC there is not an explicit way to find it > > > but to find all the 0 eigenvectors of B, which again it requires the inverse of B. > > > > > > I am solving similar small problems millions of times, and this works fine most of the time, > > > but once in a while it fails with the same LU zero pivot exception. > > > > > > One thing I tried yesterday, and it seams to work all the times, > > > is perturbing the diagonal of B of a small factor, += 1.0e-10 ||B||, > > > but I do not know how theoretically it sounds with all these indeterminate eigenvalues, > > > maybe you have a clue on this. > > > > > > Thanks, > > > Eugenio > > > > > > > > > > > > Eugenio Aulisa > > > > > > Department of Mathematics and Statistics, > > > Texas Tech University > > > Lubbock TX, 79409-1042 > > > room: 226 > > > http://www.math.ttu.edu/~eaulisa/ > > > phone: (806) 834-6684 > > > fax: (806) 742-1112 > > > > > > > > > > > > > > > From: Jose E. Roman > > > Sent: Tuesday, July 7, 2020 11:01 AM > > > To: Aulisa, Eugenio > > > Cc: petsc-users at mcs.anl.gov ; Kara, Erdi > > > Subject: Re: [petsc-users] Help with generalized eigenvalue problem > > > > > > I don't know what options you are using, shift-and-invert or not, which, target. Anyway, if A and B have a common null space, then A-sigma*B will always be singular, so the KSP solver will have to deal with a singular system. > > > > > > If you pass the nullspace vectors to the EPS solver via https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fslepc.upv.es%2Fdocumentation%2Fcurrent%2Fdocs%2Fmanualpages%2FEPS%2FEPSSetDeflationSpace.html&data=02%7C01%7CEugenio.Aulisa%40ttu.edu%7C48eb040474e44a2af2f208d82350b5a7%7C178a51bf8b2049ffb65556245d5c173c%7C0%7C0%7C637298176878634547&sdata=lL%2B5GTRzFgeviU%2BedhKieyXDO%2Fgyko2CLYGU%2BxtilB0%3D&reserved=0 then these vectors will also be attached to the KSP solver. > > > > > > Also, since you have MUMPS, try adding > > > -st_pc_factor_mat_solver_type mumps > > > (see section 3.4.1 of the SLEPc users manual). MUMPS should be quite robust with respect to singular linear systems. > > > > > > Jose > > > > > > > > > > El 7 jul 2020, a las 17:40, Aulisa, Eugenio escribi?: > > > > > > > > Can somebody help me figure it out the solver options for the attached generalized eigenvalue problem? > > > > > > > > A x = lambda B x > > > > > > > > where A and B have the same null space, with 5 zero eigenvalues. > > > > > > > > The only non indefinite eigenvalue should be 41.1892 > > > > > > > > The problem is Hermite semi-positive so automatically slepc should purify it, but to be sure I added > > > > > > > > EPSSetPurify(eps, PETSC_TRUE); > > > > > > > > I guess with my non-solver-options slepc is still trying to do the inverse of the full > > > > B matrix with an LU decomposition and gets zero pivot. > > > > > > > > This is what I get as a PETSC error Message > > > > > > > > [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- > > > > [0]PETSC ERROR: Zero pivot in LU factorization: https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.mcs.anl.gov%2Fpetsc%2Fdocumentation%2Ffaq.html%23zeropivot&data=02%7C01%7CEugenio.Aulisa%40ttu.edu%7C48eb040474e44a2af2f208d82350b5a7%7C178a51bf8b2049ffb65556245d5c173c%7C0%7C0%7C637298176878634547&sdata=MfAJt1HSEK5wvhRzx0BM9OEEGxWjnlI%2FwTlzhN4BDeE%3D&reserved=0 > > > > [0]PETSC ERROR: Bad LU factorization > > > > [0]PETSC ERROR: See https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.mcs.anl.gov%2Fpetsc%2Fdocumentation%2Ffaq.html&data=02%7C01%7CEugenio.Aulisa%40ttu.edu%7C48eb040474e44a2af2f208d82350b5a7%7C178a51bf8b2049ffb65556245d5c173c%7C0%7C0%7C637298176878634547&sdata=NTc7lCzIo1Vj2sYmAehiE5b8ToXDTvN0wzDyh7JC30w%3D&reserved=0 for trouble shooting. > > > > [0]PETSC ERROR: Petsc Development GIT revision: v3.11.3-2263-gce77f2ed1a GIT Date: 2019-09-26 13:31:14 -0500 > > > > [0]PETSC ERROR: Nitsche_ex4a on a arch-linux2-c-opt named linux-8biu by eaulisa Tue Jul 7 10:17:55 2020 > > > > [0]PETSC ERROR: Configure options --with-debugging=0 --with-x=1 COPTFLAGS="-O3 -march=native -mtune=native" CXXOPTFLAGS="-O3 -march=native -mtune=native" FOPTFLAGS="-O3 -march=native -mtune=native" --download-openmpi=1 --download-fblaslapack=1 --download-hdf5=1 --download-metis=1 --download-parmetis=1 --with-shared-libraries=1 --download-blacs=1 --download-scalapack=1 --download-mumps=1 --download-suitesparse > > > > [0]PETSC ERROR: #1 MatLUFactor_SeqDense() line 633 in /home/eaulisa/software/petsc/src/mat/impls/dense/seq/dense.c > > > > [0]PETSC ERROR: #2 MatLUFactorNumeric_SeqDense() line 432 in /home/eaulisa/software/petsc/src/mat/impls/dense/seq/dense.c > > > > [0]PETSC ERROR: #3 MatLUFactorNumeric() line 3056 in /home/eaulisa/software/petsc/src/mat/interface/matrix.c > > > > [0]PETSC ERROR: #4 PCSetUp_LU() line 126 in /home/eaulisa/software/petsc/src/ksp/pc/impls/factor/lu/lu.c > > > > [0]PETSC ERROR: #5 PCSetUp() line 894 in /home/eaulisa/software/petsc/src/ksp/pc/interface/precon.c > > > > [0]PETSC ERROR: #6 KSPSetUp() line 377 in /home/eaulisa/software/petsc/src/ksp/ksp/interface/itfunc.c > > > > [0]PETSC ERROR: #7 STSetUp_Shift() line 119 in /home/eaulisa/software/slepc/src/sys/classes/st/impls/shift/shift.c > > > > [0]PETSC ERROR: #8 STSetUp() line 271 in /home/eaulisa/software/slepc/src/sys/classes/st/interface/stsolve.c > > > > [0]PETSC ERROR: #9 EPSSetUp() line 273 in /home/eaulisa/software/slepc/src/eps/interface/epssetup.c > > > > [0]PETSC ERROR: #10 EPSSolve() line 136 in /home/eaulisa/software/slepc/src/eps/interface/epssolve.c > > > > > > > > > > > > > > > > Thanks > > > > Eugenio > > > > > > > > > > > > > > > > > > > > Eugenio Aulisa > > > > > > > > Department of Mathematics and Statistics, > > > > Texas Tech University > > > > Lubbock TX, 79409-1042 > > > > room: 226 > > > > http://www.math.ttu.edu/~eaulisa/ > > > > phone: (806) 834-6684 > > > > fax: (806) 742-1112 > > > > > > > > > > > > > > > > > > > > > > > From Eugenio.Aulisa at ttu.edu Wed Jul 8 10:52:44 2020 From: Eugenio.Aulisa at ttu.edu (Aulisa, Eugenio) Date: Wed, 8 Jul 2020 15:52:44 +0000 Subject: [petsc-users] Help with generalized eigenvalue problem In-Reply-To: <01563E8A-80EB-4E3A-ACAB-17E81CE40E41@dsic.upv.es> References: <66509B35-BCC5-4413-8807-5C7077E49B2E@dsic.upv.es> <009BBC4D-9E7C-481E-81FC-5A7B98920777@dsic.upv.es> , <01563E8A-80EB-4E3A-ACAB-17E81CE40E41@dsic.upv.es> Message-ID: Sorry you are right. Now it does not crash, but without perturbation it gives max eigenvalue -1.79769e+308 i.e. -infinity while with perturbation I get max eigenvalue 41.1892 which is the same I get with Mathematica (without perturbation) With this last statement I mean that -1.79769e+308 is not the real one. Also matrices A and B are semipositive definite. Eugenio Eugenio Aulisa Department of Mathematics and Statistics, Texas Tech University Lubbock TX, 79409-1042 room: 226 http://www.math.ttu.edu/~eaulisa/ phone: (806) 834-6684 fax: (806) 742-1112 ________________________________ From: Jose E. Roman Sent: Wednesday, July 8, 2020 10:34 AM To: Aulisa, Eugenio Cc: petsc-users at mcs.anl.gov ; Kara, Erdi Subject: Re: [petsc-users] Help with generalized eigenvalue problem eps_view shows it is using Krylov-Schur, it seems you mistyped the option: -eps_type lapack instead of -esp_type lapack > El 8 jul 2020, a las 17:29, Aulisa, Eugenio escribi?: > > I tried with -esp_type lapack, but it still tries to do LU > > I attached the code without perturbation of B, > probably something is missing in my way of setting up the problem. > > I also attached the eps_view with perturbation > > Thanks > Eugenio > > Nitsche_genEigen -esp_type lapack > [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- > [0]PETSC ERROR: Zero pivot in LU factorization: https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.mcs.anl.gov%2Fpetsc%2Fdocumentation%2Ffaq.html%23zeropivot&data=02%7C01%7CEugenio.Aulisa%40ttu.edu%7Cffdfdcc73b2d45bd055508d823547501%7C178a51bf8b2049ffb65556245d5c173c%7C0%7C0%7C637298192933095536&sdata=MB9cEbQo8PHWvVmap6Mk%2FRVKO7cID9qypNShiySTEK8%3D&reserved=0 > [0]PETSC ERROR: Zero pivot row 1 value 0. tolerance 2.22045e-14 > > [0]PETSC ERROR: See https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.mcs.anl.gov%2Fpetsc%2Fdocumentation%2Ffaq.html&data=02%7C01%7CEugenio.Aulisa%40ttu.edu%7Cffdfdcc73b2d45bd055508d823547501%7C178a51bf8b2049ffb65556245d5c173c%7C0%7C0%7C637298192933095536&sdata=q2HkYMvL%2FjZ5c03sTALoJ8pky%2BbSPJCHTsK6N70%2Fqbs%3D&reserved=0 for trouble shooting. > [0]PETSC ERROR: Petsc Development GIT revision: v3.11.3-2263-gce77f2ed1a GIT Date: 2019-09-26 13:31:14 -0500 > [0]PETSC ERROR: Nitsche_genEigen on a arch-linux2-c-opt named linux-8biu by eaulisa Wed Jul 8 10:09:31 2020 > [0]PETSC ERROR: Configure options --with-debugging=0 --with-x=1 COPTFLAGS="-O3 -march=native -mtune=native" CXXOPTFLAGS="-O3 -march=native -mtune=native" FOPTFLAGS="-O3 -march=native -mtune=native" --download-openmpi=1 --download-fblaslapack=1 --download-hdf5=1 --download-metis=1 --download-parmetis=1 --with-shared-libraries=1 --download-blacs=1 --download-scalapack=1 --download-mumps=1 --download-suitesparse > [0]PETSC ERROR: #1 MatPivotCheck_none() line 731 in /home/eaulisa/software/petsc/include/petsc/private/matimpl.h > [0]PETSC ERROR: #2 MatPivotCheck() line 748 in /home/eaulisa/software/petsc/include/petsc/private/matimpl.h > [0]PETSC ERROR: #3 MatLUFactorNumeric_SeqAIJ_Inode() line 1676 in /home/eaulisa/software/petsc/src/mat/impls/aij/seq/inode.c > [0]PETSC ERROR: #4 MatLUFactorNumeric() line 3056 in /home/eaulisa/software/petsc/src/mat/interface/matrix.c > [0]PETSC ERROR: #5 PCSetUp_LU() line 126 in /home/eaulisa/software/petsc/src/ksp/pc/impls/factor/lu/lu.c > [0]PETSC ERROR: #6 PCSetUp() line 894 in /home/eaulisa/software/petsc/src/ksp/pc/interface/precon.c > [0]PETSC ERROR: #7 KSPSetUp() line 377 in /home/eaulisa/software/petsc/src/ksp/ksp/interface/itfunc.c > [0]PETSC ERROR: #8 STSetUp_Shift() line 119 in /home/eaulisa/software/slepc/src/sys/classes/st/impls/shift/shift.c > [0]PETSC ERROR: #9 STSetUp() line 271 in /home/eaulisa/software/slepc/src/sys/classes/st/interface/stsolve.c > [0]PETSC ERROR: #10 EPSSetUp() line 273 in /home/eaulisa/software/slepc/src/eps/interface/epssetup.c > [0]PETSC ERROR: #11 EPSSolve() line 136 in /home/eaulisa/software/slepc/src/eps/interface/epssolve.c > [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- > [0]PETSC ERROR: Argument out of range > [0]PETSC ERROR: Argument 2 out of range > [0]PETSC ERROR: See https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.mcs.anl.gov%2Fpetsc%2Fdocumentation%2Ffaq.html&data=02%7C01%7CEugenio.Aulisa%40ttu.edu%7Cffdfdcc73b2d45bd055508d823547501%7C178a51bf8b2049ffb65556245d5c173c%7C0%7C0%7C637298192933095536&sdata=q2HkYMvL%2FjZ5c03sTALoJ8pky%2BbSPJCHTsK6N70%2Fqbs%3D&reserved=0 for trouble shooting. > [0]PETSC ERROR: Petsc Development GIT revision: v3.11.3-2263-gce77f2ed1a GIT Date: 2019-09-26 13:31:14 -0500 > [0]PETSC ERROR: Nitsche_genEigen on a arch-linux2-c-opt named linux-8biu by eaulisa Wed Jul 8 10:09:31 2020 > [0]PETSC ERROR: Configure options --with-debugging=0 --with-x=1 COPTFLAGS="-O3 -march=native -mtune=native" CXXOPTFLAGS="-O3 -march=native -mtune=native" FOPTFLAGS="-O3 -march=native -mtune=native" --download-openmpi=1 --download-fblaslapack=1 --download-hdf5=1 --download-metis=1 --download-parmetis=1 --with-shared-libraries=1 --download-blacs=1 --download-scalapack=1 --download-mumps=1 --download-suitesparse > [0]PETSC ERROR: #12 EPSGetEigenpair() line 399 in /home/eaulisa/software/slepc/src/eps/interface/epssolve.c > 2.96439e-323 > > Eugenio Aulisa > > Department of Mathematics and Statistics, > Texas Tech University > Lubbock TX, 79409-1042 > room: 226 > http://www.math.ttu.edu/~eaulisa/ > phone: (806) 834-6684 > fax: (806) 742-1112 > > > > > From: Jose E. Roman > Sent: Wednesday, July 8, 2020 10:07 AM > To: Aulisa, Eugenio > Cc: petsc-users at mcs.anl.gov ; Kara, Erdi > Subject: Re: [petsc-users] Help with generalized eigenvalue problem > > Run with -eps_type lapack > > Eigen is not interfaced from SLEPc. > > Jose > > > > El 8 jul 2020, a las 17:06, Aulisa, Eugenio escribi?: > > > > yes the size varies from 6 to 100, > > > > I will try EPSLAPACK or EIGEN to see how it works > > > > Thanks again > > Eugenio > > > > > > > > > > > > Eugenio Aulisa > > > > Department of Mathematics and Statistics, > > Texas Tech University > > Lubbock TX, 79409-1042 > > room: 226 > > http://www.math.ttu.edu/~eaulisa/ > > phone: (806) 834-6684 > > fax: (806) 742-1112 > > > > > > > > > > From: Jose E. Roman > > Sent: Wednesday, July 8, 2020 10:01 AM > > To: Aulisa, Eugenio > > Cc: petsc-users at mcs.anl.gov ; Kara, Erdi > > Subject: Re: [petsc-users] Help with generalized eigenvalue problem > > > > In a standard symmetric eigenproblem, a small perturbation of the matrix results in a small perturbation of the eigenvalues. In generalized eigenproblems, especially with singular matrices, I don't think perturbation theory guarantees this in all cases. > > > > What do you mean by small problems? Do you have millions of problems of size 6? In that case you should use EPSLAPACK. It will not factor the matrix (if solved as a non-symmetric problem). > > > > Jose > > > > > > > El 8 jul 2020, a las 15:55, Aulisa, Eugenio escribi?: > > > > > > Dear Jose, > > > > > > Thank you for your answer. > > > > > > I tried MLU but it did not help. > > > > > > I also tried to eliminate the null space of B, but I do not know it apriori, > > > and at least to my knowledge of PETSC-SLEPC there is not an explicit way to find it > > > but to find all the 0 eigenvectors of B, which again it requires the inverse of B. > > > > > > I am solving similar small problems millions of times, and this works fine most of the time, > > > but once in a while it fails with the same LU zero pivot exception. > > > > > > One thing I tried yesterday, and it seams to work all the times, > > > is perturbing the diagonal of B of a small factor, += 1.0e-10 ||B||, > > > but I do not know how theoretically it sounds with all these indeterminate eigenvalues, > > > maybe you have a clue on this. > > > > > > Thanks, > > > Eugenio > > > > > > > > > > > > Eugenio Aulisa > > > > > > Department of Mathematics and Statistics, > > > Texas Tech University > > > Lubbock TX, 79409-1042 > > > room: 226 > > > http://www.math.ttu.edu/~eaulisa/ > > > phone: (806) 834-6684 > > > fax: (806) 742-1112 > > > > > > > > > > > > > > > From: Jose E. Roman > > > Sent: Tuesday, July 7, 2020 11:01 AM > > > To: Aulisa, Eugenio > > > Cc: petsc-users at mcs.anl.gov ; Kara, Erdi > > > Subject: Re: [petsc-users] Help with generalized eigenvalue problem > > > > > > I don't know what options you are using, shift-and-invert or not, which, target. Anyway, if A and B have a common null space, then A-sigma*B will always be singular, so the KSP solver will have to deal with a singular system. > > > > > > If you pass the nullspace vectors to the EPS solver via https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fslepc.upv.es%2Fdocumentation%2Fcurrent%2Fdocs%2Fmanualpages%2FEPS%2FEPSSetDeflationSpace.html&data=02%7C01%7CEugenio.Aulisa%40ttu.edu%7Cffdfdcc73b2d45bd055508d823547501%7C178a51bf8b2049ffb65556245d5c173c%7C0%7C0%7C637298192933095536&sdata=UMcNl4UZYOg7sytYEAsyQRDmptm6XG%2B6C0Ylh%2F1dj8Q%3D&reserved=0 then these vectors will also be attached to the KSP solver. > > > > > > Also, since you have MUMPS, try adding > > > -st_pc_factor_mat_solver_type mumps > > > (see section 3.4.1 of the SLEPc users manual). MUMPS should be quite robust with respect to singular linear systems. > > > > > > Jose > > > > > > > > > > El 7 jul 2020, a las 17:40, Aulisa, Eugenio escribi?: > > > > > > > > Can somebody help me figure it out the solver options for the attached generalized eigenvalue problem? > > > > > > > > A x = lambda B x > > > > > > > > where A and B have the same null space, with 5 zero eigenvalues. > > > > > > > > The only non indefinite eigenvalue should be 41.1892 > > > > > > > > The problem is Hermite semi-positive so automatically slepc should purify it, but to be sure I added > > > > > > > > EPSSetPurify(eps, PETSC_TRUE); > > > > > > > > I guess with my non-solver-options slepc is still trying to do the inverse of the full > > > > B matrix with an LU decomposition and gets zero pivot. > > > > > > > > This is what I get as a PETSC error Message > > > > > > > > [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- > > > > [0]PETSC ERROR: Zero pivot in LU factorization: https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.mcs.anl.gov%2Fpetsc%2Fdocumentation%2Ffaq.html%23zeropivot&data=02%7C01%7CEugenio.Aulisa%40ttu.edu%7Cffdfdcc73b2d45bd055508d823547501%7C178a51bf8b2049ffb65556245d5c173c%7C0%7C0%7C637298192933095536&sdata=MB9cEbQo8PHWvVmap6Mk%2FRVKO7cID9qypNShiySTEK8%3D&reserved=0 > > > > [0]PETSC ERROR: Bad LU factorization > > > > [0]PETSC ERROR: See https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.mcs.anl.gov%2Fpetsc%2Fdocumentation%2Ffaq.html&data=02%7C01%7CEugenio.Aulisa%40ttu.edu%7Cffdfdcc73b2d45bd055508d823547501%7C178a51bf8b2049ffb65556245d5c173c%7C0%7C0%7C637298192933095536&sdata=q2HkYMvL%2FjZ5c03sTALoJ8pky%2BbSPJCHTsK6N70%2Fqbs%3D&reserved=0 for trouble shooting. > > > > [0]PETSC ERROR: Petsc Development GIT revision: v3.11.3-2263-gce77f2ed1a GIT Date: 2019-09-26 13:31:14 -0500 > > > > [0]PETSC ERROR: Nitsche_ex4a on a arch-linux2-c-opt named linux-8biu by eaulisa Tue Jul 7 10:17:55 2020 > > > > [0]PETSC ERROR: Configure options --with-debugging=0 --with-x=1 COPTFLAGS="-O3 -march=native -mtune=native" CXXOPTFLAGS="-O3 -march=native -mtune=native" FOPTFLAGS="-O3 -march=native -mtune=native" --download-openmpi=1 --download-fblaslapack=1 --download-hdf5=1 --download-metis=1 --download-parmetis=1 --with-shared-libraries=1 --download-blacs=1 --download-scalapack=1 --download-mumps=1 --download-suitesparse > > > > [0]PETSC ERROR: #1 MatLUFactor_SeqDense() line 633 in /home/eaulisa/software/petsc/src/mat/impls/dense/seq/dense.c > > > > [0]PETSC ERROR: #2 MatLUFactorNumeric_SeqDense() line 432 in /home/eaulisa/software/petsc/src/mat/impls/dense/seq/dense.c > > > > [0]PETSC ERROR: #3 MatLUFactorNumeric() line 3056 in /home/eaulisa/software/petsc/src/mat/interface/matrix.c > > > > [0]PETSC ERROR: #4 PCSetUp_LU() line 126 in /home/eaulisa/software/petsc/src/ksp/pc/impls/factor/lu/lu.c > > > > [0]PETSC ERROR: #5 PCSetUp() line 894 in /home/eaulisa/software/petsc/src/ksp/pc/interface/precon.c > > > > [0]PETSC ERROR: #6 KSPSetUp() line 377 in /home/eaulisa/software/petsc/src/ksp/ksp/interface/itfunc.c > > > > [0]PETSC ERROR: #7 STSetUp_Shift() line 119 in /home/eaulisa/software/slepc/src/sys/classes/st/impls/shift/shift.c > > > > [0]PETSC ERROR: #8 STSetUp() line 271 in /home/eaulisa/software/slepc/src/sys/classes/st/interface/stsolve.c > > > > [0]PETSC ERROR: #9 EPSSetUp() line 273 in /home/eaulisa/software/slepc/src/eps/interface/epssetup.c > > > > [0]PETSC ERROR: #10 EPSSolve() line 136 in /home/eaulisa/software/slepc/src/eps/interface/epssolve.c > > > > > > > > > > > > > > > > Thanks > > > > Eugenio > > > > > > > > > > > > > > > > > > > > Eugenio Aulisa > > > > > > > > Department of Mathematics and Statistics, > > > > Texas Tech University > > > > Lubbock TX, 79409-1042 > > > > room: 226 > > > > http://www.math.ttu.edu/~eaulisa/ > > > > phone: (806) 834-6684 > > > > fax: (806) 742-1112 > > > > > > > > > > > > > > > > > > > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: genEigen.nb Type: application/mathematica Size: 3791 bytes Desc: genEigen.nb URL: -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: eps_view.txt URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: genEigen.nb.pdf Type: application/pdf Size: 21111 bytes Desc: genEigen.nb.pdf URL: From jroman at dsic.upv.es Wed Jul 8 11:05:42 2020 From: jroman at dsic.upv.es (Jose E. Roman) Date: Wed, 8 Jul 2020 18:05:42 +0200 Subject: [petsc-users] Help with generalized eigenvalue problem In-Reply-To: References: <66509B35-BCC5-4413-8807-5C7077E49B2E@dsic.upv.es> <009BBC4D-9E7C-481E-81FC-5A7B98920777@dsic.upv.es> <01563E8A-80EB-4E3A-ACAB-17E81CE40E41@dsic.upv.es> Message-ID: The solver computes all the eigenvalues, including 41.1892, you just have to select the wanted one. If you add -eps_largest_real then you should get 41.1892 as the first eigenvalue. But this will not work if LAPACK returns +inf as one of the eigenvalues. > El 8 jul 2020, a las 17:52, Aulisa, Eugenio escribi?: > > Sorry you are right. > > Now it does not crash, but without perturbation it gives max eigenvalue > > -1.79769e+308 i.e. -infinity > > while with perturbation I get max eigenvalue > > 41.1892 > > which is the same I get with Mathematica (without perturbation) > > With this last statement I mean that -1.79769e+308 is not the real one. Also matrices A and B are semipositive definite. > > Eugenio > > > > > > > > Eugenio Aulisa > > Department of Mathematics and Statistics, > Texas Tech University > Lubbock TX, 79409-1042 > room: 226 > http://www.math.ttu.edu/~eaulisa/ > phone: (806) 834-6684 > fax: (806) 742-1112 > > > > > From: Jose E. Roman > Sent: Wednesday, July 8, 2020 10:34 AM > To: Aulisa, Eugenio > Cc: petsc-users at mcs.anl.gov ; Kara, Erdi > Subject: Re: [petsc-users] Help with generalized eigenvalue problem > > eps_view shows it is using Krylov-Schur, it seems you mistyped the option: -eps_type lapack instead of -esp_type lapack > > > > El 8 jul 2020, a las 17:29, Aulisa, Eugenio escribi?: > > > > I tried with -esp_type lapack, but it still tries to do LU > > > > I attached the code without perturbation of B, > > probably something is missing in my way of setting up the problem. > > > > I also attached the eps_view with perturbation > > > > Thanks > > Eugenio > > > > Nitsche_genEigen -esp_type lapack > > [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- > > [0]PETSC ERROR: Zero pivot in LU factorization: https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.mcs.anl.gov%2Fpetsc%2Fdocumentation%2Ffaq.html%23zeropivot&data=02%7C01%7CEugenio.Aulisa%40ttu.edu%7Cffdfdcc73b2d45bd055508d823547501%7C178a51bf8b2049ffb65556245d5c173c%7C0%7C0%7C637298192933095536&sdata=MB9cEbQo8PHWvVmap6Mk%2FRVKO7cID9qypNShiySTEK8%3D&reserved=0 > > [0]PETSC ERROR: Zero pivot row 1 value 0. tolerance 2.22045e-14 > > > > [0]PETSC ERROR: See https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.mcs.anl.gov%2Fpetsc%2Fdocumentation%2Ffaq.html&data=02%7C01%7CEugenio.Aulisa%40ttu.edu%7Cffdfdcc73b2d45bd055508d823547501%7C178a51bf8b2049ffb65556245d5c173c%7C0%7C0%7C637298192933095536&sdata=q2HkYMvL%2FjZ5c03sTALoJ8pky%2BbSPJCHTsK6N70%2Fqbs%3D&reserved=0 for trouble shooting. > > [0]PETSC ERROR: Petsc Development GIT revision: v3.11.3-2263-gce77f2ed1a GIT Date: 2019-09-26 13:31:14 -0500 > > [0]PETSC ERROR: Nitsche_genEigen on a arch-linux2-c-opt named linux-8biu by eaulisa Wed Jul 8 10:09:31 2020 > > [0]PETSC ERROR: Configure options --with-debugging=0 --with-x=1 COPTFLAGS="-O3 -march=native -mtune=native" CXXOPTFLAGS="-O3 -march=native -mtune=native" FOPTFLAGS="-O3 -march=native -mtune=native" --download-openmpi=1 --download-fblaslapack=1 --download-hdf5=1 --download-metis=1 --download-parmetis=1 --with-shared-libraries=1 --download-blacs=1 --download-scalapack=1 --download-mumps=1 --download-suitesparse > > [0]PETSC ERROR: #1 MatPivotCheck_none() line 731 in /home/eaulisa/software/petsc/include/petsc/private/matimpl.h > > [0]PETSC ERROR: #2 MatPivotCheck() line 748 in /home/eaulisa/software/petsc/include/petsc/private/matimpl.h > > [0]PETSC ERROR: #3 MatLUFactorNumeric_SeqAIJ_Inode() line 1676 in /home/eaulisa/software/petsc/src/mat/impls/aij/seq/inode.c > > [0]PETSC ERROR: #4 MatLUFactorNumeric() line 3056 in /home/eaulisa/software/petsc/src/mat/interface/matrix.c > > [0]PETSC ERROR: #5 PCSetUp_LU() line 126 in /home/eaulisa/software/petsc/src/ksp/pc/impls/factor/lu/lu.c > > [0]PETSC ERROR: #6 PCSetUp() line 894 in /home/eaulisa/software/petsc/src/ksp/pc/interface/precon.c > > [0]PETSC ERROR: #7 KSPSetUp() line 377 in /home/eaulisa/software/petsc/src/ksp/ksp/interface/itfunc.c > > [0]PETSC ERROR: #8 STSetUp_Shift() line 119 in /home/eaulisa/software/slepc/src/sys/classes/st/impls/shift/shift.c > > [0]PETSC ERROR: #9 STSetUp() line 271 in /home/eaulisa/software/slepc/src/sys/classes/st/interface/stsolve.c > > [0]PETSC ERROR: #10 EPSSetUp() line 273 in /home/eaulisa/software/slepc/src/eps/interface/epssetup.c > > [0]PETSC ERROR: #11 EPSSolve() line 136 in /home/eaulisa/software/slepc/src/eps/interface/epssolve.c > > [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- > > [0]PETSC ERROR: Argument out of range > > [0]PETSC ERROR: Argument 2 out of range > > [0]PETSC ERROR: See https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.mcs.anl.gov%2Fpetsc%2Fdocumentation%2Ffaq.html&data=02%7C01%7CEugenio.Aulisa%40ttu.edu%7Cffdfdcc73b2d45bd055508d823547501%7C178a51bf8b2049ffb65556245d5c173c%7C0%7C0%7C637298192933095536&sdata=q2HkYMvL%2FjZ5c03sTALoJ8pky%2BbSPJCHTsK6N70%2Fqbs%3D&reserved=0 for trouble shooting. > > [0]PETSC ERROR: Petsc Development GIT revision: v3.11.3-2263-gce77f2ed1a GIT Date: 2019-09-26 13:31:14 -0500 > > [0]PETSC ERROR: Nitsche_genEigen on a arch-linux2-c-opt named linux-8biu by eaulisa Wed Jul 8 10:09:31 2020 > > [0]PETSC ERROR: Configure options --with-debugging=0 --with-x=1 COPTFLAGS="-O3 -march=native -mtune=native" CXXOPTFLAGS="-O3 -march=native -mtune=native" FOPTFLAGS="-O3 -march=native -mtune=native" --download-openmpi=1 --download-fblaslapack=1 --download-hdf5=1 --download-metis=1 --download-parmetis=1 --with-shared-libraries=1 --download-blacs=1 --download-scalapack=1 --download-mumps=1 --download-suitesparse > > [0]PETSC ERROR: #12 EPSGetEigenpair() line 399 in /home/eaulisa/software/slepc/src/eps/interface/epssolve.c > > 2.96439e-323 > > > > Eugenio Aulisa > > > > Department of Mathematics and Statistics, > > Texas Tech University > > Lubbock TX, 79409-1042 > > room: 226 > > http://www.math.ttu.edu/~eaulisa/ > > phone: (806) 834-6684 > > fax: (806) 742-1112 > > > > > > > > > > From: Jose E. Roman > > Sent: Wednesday, July 8, 2020 10:07 AM > > To: Aulisa, Eugenio > > Cc: petsc-users at mcs.anl.gov ; Kara, Erdi > > Subject: Re: [petsc-users] Help with generalized eigenvalue problem > > > > Run with -eps_type lapack > > > > Eigen is not interfaced from SLEPc. > > > > Jose > > > > > > > El 8 jul 2020, a las 17:06, Aulisa, Eugenio escribi?: > > > > > > yes the size varies from 6 to 100, > > > > > > I will try EPSLAPACK or EIGEN to see how it works > > > > > > Thanks again > > > Eugenio > > > > > > > > > > > > > > > > > > Eugenio Aulisa > > > > > > Department of Mathematics and Statistics, > > > Texas Tech University > > > Lubbock TX, 79409-1042 > > > room: 226 > > > http://www.math.ttu.edu/~eaulisa/ > > > phone: (806) 834-6684 > > > fax: (806) 742-1112 > > > > > > > > > > > > > > > From: Jose E. Roman > > > Sent: Wednesday, July 8, 2020 10:01 AM > > > To: Aulisa, Eugenio > > > Cc: petsc-users at mcs.anl.gov ; Kara, Erdi > > > Subject: Re: [petsc-users] Help with generalized eigenvalue problem > > > > > > In a standard symmetric eigenproblem, a small perturbation of the matrix results in a small perturbation of the eigenvalues. In generalized eigenproblems, especially with singular matrices, I don't think perturbation theory guarantees this in all cases. > > > > > > What do you mean by small problems? Do you have millions of problems of size 6? In that case you should use EPSLAPACK. It will not factor the matrix (if solved as a non-symmetric problem). > > > > > > Jose > > > > > > > > > > El 8 jul 2020, a las 15:55, Aulisa, Eugenio escribi?: > > > > > > > > Dear Jose, > > > > > > > > Thank you for your answer. > > > > > > > > I tried MLU but it did not help. > > > > > > > > I also tried to eliminate the null space of B, but I do not know it apriori, > > > > and at least to my knowledge of PETSC-SLEPC there is not an explicit way to find it > > > > but to find all the 0 eigenvectors of B, which again it requires the inverse of B. > > > > > > > > I am solving similar small problems millions of times, and this works fine most of the time, > > > > but once in a while it fails with the same LU zero pivot exception. > > > > > > > > One thing I tried yesterday, and it seams to work all the times, > > > > is perturbing the diagonal of B of a small factor, += 1.0e-10 ||B||, > > > > but I do not know how theoretically it sounds with all these indeterminate eigenvalues, > > > > maybe you have a clue on this. > > > > > > > > Thanks, > > > > Eugenio > > > > > > > > > > > > > > > > Eugenio Aulisa > > > > > > > > Department of Mathematics and Statistics, > > > > Texas Tech University > > > > Lubbock TX, 79409-1042 > > > > room: 226 > > > > http://www.math.ttu.edu/~eaulisa/ > > > > phone: (806) 834-6684 > > > > fax: (806) 742-1112 > > > > > > > > > > > > > > > > > > > > From: Jose E. Roman > > > > Sent: Tuesday, July 7, 2020 11:01 AM > > > > To: Aulisa, Eugenio > > > > Cc: petsc-users at mcs.anl.gov ; Kara, Erdi > > > > Subject: Re: [petsc-users] Help with generalized eigenvalue problem > > > > > > > > I don't know what options you are using, shift-and-invert or not, which, target. Anyway, if A and B have a common null space, then A-sigma*B will always be singular, so the KSP solver will have to deal with a singular system. > > > > > > > > If you pass the nullspace vectors to the EPS solver via https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fslepc.upv.es%2Fdocumentation%2Fcurrent%2Fdocs%2Fmanualpages%2FEPS%2FEPSSetDeflationSpace.html&data=02%7C01%7CEugenio.Aulisa%40ttu.edu%7Cffdfdcc73b2d45bd055508d823547501%7C178a51bf8b2049ffb65556245d5c173c%7C0%7C0%7C637298192933095536&sdata=UMcNl4UZYOg7sytYEAsyQRDmptm6XG%2B6C0Ylh%2F1dj8Q%3D&reserved=0 then these vectors will also be attached to the KSP solver. > > > > > > > > Also, since you have MUMPS, try adding > > > > -st_pc_factor_mat_solver_type mumps > > > > (see section 3.4.1 of the SLEPc users manual). MUMPS should be quite robust with respect to singular linear systems. > > > > > > > > Jose > > > > > > > > > > > > > El 7 jul 2020, a las 17:40, Aulisa, Eugenio escribi?: > > > > > > > > > > Can somebody help me figure it out the solver options for the attached generalized eigenvalue problem? > > > > > > > > > > A x = lambda B x > > > > > > > > > > where A and B have the same null space, with 5 zero eigenvalues. > > > > > > > > > > The only non indefinite eigenvalue should be 41.1892 > > > > > > > > > > The problem is Hermite semi-positive so automatically slepc should purify it, but to be sure I added > > > > > > > > > > EPSSetPurify(eps, PETSC_TRUE); > > > > > > > > > > I guess with my non-solver-options slepc is still trying to do the inverse of the full > > > > > B matrix with an LU decomposition and gets zero pivot. > > > > > > > > > > This is what I get as a PETSC error Message > > > > > > > > > > [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- > > > > > [0]PETSC ERROR: Zero pivot in LU factorization: https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.mcs.anl.gov%2Fpetsc%2Fdocumentation%2Ffaq.html%23zeropivot&data=02%7C01%7CEugenio.Aulisa%40ttu.edu%7Cffdfdcc73b2d45bd055508d823547501%7C178a51bf8b2049ffb65556245d5c173c%7C0%7C0%7C637298192933095536&sdata=MB9cEbQo8PHWvVmap6Mk%2FRVKO7cID9qypNShiySTEK8%3D&reserved=0 > > > > > [0]PETSC ERROR: Bad LU factorization > > > > > [0]PETSC ERROR: See https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.mcs.anl.gov%2Fpetsc%2Fdocumentation%2Ffaq.html&data=02%7C01%7CEugenio.Aulisa%40ttu.edu%7Cffdfdcc73b2d45bd055508d823547501%7C178a51bf8b2049ffb65556245d5c173c%7C0%7C0%7C637298192933095536&sdata=q2HkYMvL%2FjZ5c03sTALoJ8pky%2BbSPJCHTsK6N70%2Fqbs%3D&reserved=0 for trouble shooting. > > > > > [0]PETSC ERROR: Petsc Development GIT revision: v3.11.3-2263-gce77f2ed1a GIT Date: 2019-09-26 13:31:14 -0500 > > > > > [0]PETSC ERROR: Nitsche_ex4a on a arch-linux2-c-opt named linux-8biu by eaulisa Tue Jul 7 10:17:55 2020 > > > > > [0]PETSC ERROR: Configure options --with-debugging=0 --with-x=1 COPTFLAGS="-O3 -march=native -mtune=native" CXXOPTFLAGS="-O3 -march=native -mtune=native" FOPTFLAGS="-O3 -march=native -mtune=native" --download-openmpi=1 --download-fblaslapack=1 --download-hdf5=1 --download-metis=1 --download-parmetis=1 --with-shared-libraries=1 --download-blacs=1 --download-scalapack=1 --download-mumps=1 --download-suitesparse > > > > > [0]PETSC ERROR: #1 MatLUFactor_SeqDense() line 633 in /home/eaulisa/software/petsc/src/mat/impls/dense/seq/dense.c > > > > > [0]PETSC ERROR: #2 MatLUFactorNumeric_SeqDense() line 432 in /home/eaulisa/software/petsc/src/mat/impls/dense/seq/dense.c > > > > > [0]PETSC ERROR: #3 MatLUFactorNumeric() line 3056 in /home/eaulisa/software/petsc/src/mat/interface/matrix.c > > > > > [0]PETSC ERROR: #4 PCSetUp_LU() line 126 in /home/eaulisa/software/petsc/src/ksp/pc/impls/factor/lu/lu.c > > > > > [0]PETSC ERROR: #5 PCSetUp() line 894 in /home/eaulisa/software/petsc/src/ksp/pc/interface/precon.c > > > > > [0]PETSC ERROR: #6 KSPSetUp() line 377 in /home/eaulisa/software/petsc/src/ksp/ksp/interface/itfunc.c > > > > > [0]PETSC ERROR: #7 STSetUp_Shift() line 119 in /home/eaulisa/software/slepc/src/sys/classes/st/impls/shift/shift.c > > > > > [0]PETSC ERROR: #8 STSetUp() line 271 in /home/eaulisa/software/slepc/src/sys/classes/st/interface/stsolve.c > > > > > [0]PETSC ERROR: #9 EPSSetUp() line 273 in /home/eaulisa/software/slepc/src/eps/interface/epssetup.c > > > > > [0]PETSC ERROR: #10 EPSSolve() line 136 in /home/eaulisa/software/slepc/src/eps/interface/epssolve.c > > > > > > > > > > > > > > > > > > > > Thanks > > > > > Eugenio > > > > > > > > > > > > > > > > > > > > > > > > > Eugenio Aulisa > > > > > > > > > > Department of Mathematics and Statistics, > > > > > Texas Tech University > > > > > Lubbock TX, 79409-1042 > > > > > room: 226 > > > > > http://www.math.ttu.edu/~eaulisa/ > > > > > phone: (806) 834-6684 > > > > > fax: (806) 742-1112 > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > From Eugenio.Aulisa at ttu.edu Wed Jul 8 11:51:23 2020 From: Eugenio.Aulisa at ttu.edu (Aulisa, Eugenio) Date: Wed, 8 Jul 2020 16:51:23 +0000 Subject: [petsc-users] Help with generalized eigenvalue problem In-Reply-To: References: <66509B35-BCC5-4413-8807-5C7077E49B2E@dsic.upv.es> <009BBC4D-9E7C-481E-81FC-5A7B98920777@dsic.upv.es> <01563E8A-80EB-4E3A-ACAB-17E81CE40E41@dsic.upv.es> , Message-ID: the option -eps_largest_real worked also without perturbation thanks a lot Eugenio Eugenio Aulisa Department of Mathematics and Statistics, Texas Tech University Lubbock TX, 79409-1042 room: 226 http://www.math.ttu.edu/~eaulisa/ phone: (806) 834-6684 fax: (806) 742-1112 ________________________________ From: Jose E. Roman Sent: Wednesday, July 8, 2020 11:05 AM To: Aulisa, Eugenio Cc: petsc-users at mcs.anl.gov ; Kara, Erdi Subject: Re: [petsc-users] Help with generalized eigenvalue problem The solver computes all the eigenvalues, including 41.1892, you just have to select the wanted one. If you add -eps_largest_real then you should get 41.1892 as the first eigenvalue. But this will not work if LAPACK returns +inf as one of the eigenvalues. > El 8 jul 2020, a las 17:52, Aulisa, Eugenio escribi?: > > Sorry you are right. > > Now it does not crash, but without perturbation it gives max eigenvalue > > -1.79769e+308 i.e. -infinity > > while with perturbation I get max eigenvalue > > 41.1892 > > which is the same I get with Mathematica (without perturbation) > > With this last statement I mean that -1.79769e+308 is not the real one. Also matrices A and B are semipositive definite. > > Eugenio > > > > > > > > Eugenio Aulisa > > Department of Mathematics and Statistics, > Texas Tech University > Lubbock TX, 79409-1042 > room: 226 > http://www.math.ttu.edu/~eaulisa/ > phone: (806) 834-6684 > fax: (806) 742-1112 > > > > > From: Jose E. Roman > Sent: Wednesday, July 8, 2020 10:34 AM > To: Aulisa, Eugenio > Cc: petsc-users at mcs.anl.gov ; Kara, Erdi > Subject: Re: [petsc-users] Help with generalized eigenvalue problem > > eps_view shows it is using Krylov-Schur, it seems you mistyped the option: -eps_type lapack instead of -esp_type lapack > > > > El 8 jul 2020, a las 17:29, Aulisa, Eugenio escribi?: > > > > I tried with -esp_type lapack, but it still tries to do LU > > > > I attached the code without perturbation of B, > > probably something is missing in my way of setting up the problem. > > > > I also attached the eps_view with perturbation > > > > Thanks > > Eugenio > > > > Nitsche_genEigen -esp_type lapack > > [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- > > [0]PETSC ERROR: Zero pivot in LU factorization: https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.mcs.anl.gov%2Fpetsc%2Fdocumentation%2Ffaq.html%23zeropivot&data=02%7C01%7CEugenio.Aulisa%40ttu.edu%7C951cc55964c449f2f8a508d82358c899%7C178a51bf8b2049ffb65556245d5c173c%7C0%7C0%7C637298211528226297&sdata=LOEPAnUL8Lfp9qxUT1btUNqi8uYEaPhpaAyCF3jeIdg%3D&reserved=0 > > [0]PETSC ERROR: Zero pivot row 1 value 0. tolerance 2.22045e-14 > > > > [0]PETSC ERROR: See https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.mcs.anl.gov%2Fpetsc%2Fdocumentation%2Ffaq.html&data=02%7C01%7CEugenio.Aulisa%40ttu.edu%7C951cc55964c449f2f8a508d82358c899%7C178a51bf8b2049ffb65556245d5c173c%7C0%7C0%7C637298211528226297&sdata=sRuvLhErTg8zgS2R%2Fvq9qJj8wcIiKoanc6vii8WQBak%3D&reserved=0 for trouble shooting. > > [0]PETSC ERROR: Petsc Development GIT revision: v3.11.3-2263-gce77f2ed1a GIT Date: 2019-09-26 13:31:14 -0500 > > [0]PETSC ERROR: Nitsche_genEigen on a arch-linux2-c-opt named linux-8biu by eaulisa Wed Jul 8 10:09:31 2020 > > [0]PETSC ERROR: Configure options --with-debugging=0 --with-x=1 COPTFLAGS="-O3 -march=native -mtune=native" CXXOPTFLAGS="-O3 -march=native -mtune=native" FOPTFLAGS="-O3 -march=native -mtune=native" --download-openmpi=1 --download-fblaslapack=1 --download-hdf5=1 --download-metis=1 --download-parmetis=1 --with-shared-libraries=1 --download-blacs=1 --download-scalapack=1 --download-mumps=1 --download-suitesparse > > [0]PETSC ERROR: #1 MatPivotCheck_none() line 731 in /home/eaulisa/software/petsc/include/petsc/private/matimpl.h > > [0]PETSC ERROR: #2 MatPivotCheck() line 748 in /home/eaulisa/software/petsc/include/petsc/private/matimpl.h > > [0]PETSC ERROR: #3 MatLUFactorNumeric_SeqAIJ_Inode() line 1676 in /home/eaulisa/software/petsc/src/mat/impls/aij/seq/inode.c > > [0]PETSC ERROR: #4 MatLUFactorNumeric() line 3056 in /home/eaulisa/software/petsc/src/mat/interface/matrix.c > > [0]PETSC ERROR: #5 PCSetUp_LU() line 126 in /home/eaulisa/software/petsc/src/ksp/pc/impls/factor/lu/lu.c > > [0]PETSC ERROR: #6 PCSetUp() line 894 in /home/eaulisa/software/petsc/src/ksp/pc/interface/precon.c > > [0]PETSC ERROR: #7 KSPSetUp() line 377 in /home/eaulisa/software/petsc/src/ksp/ksp/interface/itfunc.c > > [0]PETSC ERROR: #8 STSetUp_Shift() line 119 in /home/eaulisa/software/slepc/src/sys/classes/st/impls/shift/shift.c > > [0]PETSC ERROR: #9 STSetUp() line 271 in /home/eaulisa/software/slepc/src/sys/classes/st/interface/stsolve.c > > [0]PETSC ERROR: #10 EPSSetUp() line 273 in /home/eaulisa/software/slepc/src/eps/interface/epssetup.c > > [0]PETSC ERROR: #11 EPSSolve() line 136 in /home/eaulisa/software/slepc/src/eps/interface/epssolve.c > > [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- > > [0]PETSC ERROR: Argument out of range > > [0]PETSC ERROR: Argument 2 out of range > > [0]PETSC ERROR: See https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.mcs.anl.gov%2Fpetsc%2Fdocumentation%2Ffaq.html&data=02%7C01%7CEugenio.Aulisa%40ttu.edu%7C951cc55964c449f2f8a508d82358c899%7C178a51bf8b2049ffb65556245d5c173c%7C0%7C0%7C637298211528226297&sdata=sRuvLhErTg8zgS2R%2Fvq9qJj8wcIiKoanc6vii8WQBak%3D&reserved=0 for trouble shooting. > > [0]PETSC ERROR: Petsc Development GIT revision: v3.11.3-2263-gce77f2ed1a GIT Date: 2019-09-26 13:31:14 -0500 > > [0]PETSC ERROR: Nitsche_genEigen on a arch-linux2-c-opt named linux-8biu by eaulisa Wed Jul 8 10:09:31 2020 > > [0]PETSC ERROR: Configure options --with-debugging=0 --with-x=1 COPTFLAGS="-O3 -march=native -mtune=native" CXXOPTFLAGS="-O3 -march=native -mtune=native" FOPTFLAGS="-O3 -march=native -mtune=native" --download-openmpi=1 --download-fblaslapack=1 --download-hdf5=1 --download-metis=1 --download-parmetis=1 --with-shared-libraries=1 --download-blacs=1 --download-scalapack=1 --download-mumps=1 --download-suitesparse > > [0]PETSC ERROR: #12 EPSGetEigenpair() line 399 in /home/eaulisa/software/slepc/src/eps/interface/epssolve.c > > 2.96439e-323 > > > > Eugenio Aulisa > > > > Department of Mathematics and Statistics, > > Texas Tech University > > Lubbock TX, 79409-1042 > > room: 226 > > http://www.math.ttu.edu/~eaulisa/ > > phone: (806) 834-6684 > > fax: (806) 742-1112 > > > > > > > > > > From: Jose E. Roman > > Sent: Wednesday, July 8, 2020 10:07 AM > > To: Aulisa, Eugenio > > Cc: petsc-users at mcs.anl.gov ; Kara, Erdi > > Subject: Re: [petsc-users] Help with generalized eigenvalue problem > > > > Run with -eps_type lapack > > > > Eigen is not interfaced from SLEPc. > > > > Jose > > > > > > > El 8 jul 2020, a las 17:06, Aulisa, Eugenio escribi?: > > > > > > yes the size varies from 6 to 100, > > > > > > I will try EPSLAPACK or EIGEN to see how it works > > > > > > Thanks again > > > Eugenio > > > > > > > > > > > > > > > > > > Eugenio Aulisa > > > > > > Department of Mathematics and Statistics, > > > Texas Tech University > > > Lubbock TX, 79409-1042 > > > room: 226 > > > http://www.math.ttu.edu/~eaulisa/ > > > phone: (806) 834-6684 > > > fax: (806) 742-1112 > > > > > > > > > > > > > > > From: Jose E. Roman > > > Sent: Wednesday, July 8, 2020 10:01 AM > > > To: Aulisa, Eugenio > > > Cc: petsc-users at mcs.anl.gov ; Kara, Erdi > > > Subject: Re: [petsc-users] Help with generalized eigenvalue problem > > > > > > In a standard symmetric eigenproblem, a small perturbation of the matrix results in a small perturbation of the eigenvalues. In generalized eigenproblems, especially with singular matrices, I don't think perturbation theory guarantees this in all cases. > > > > > > What do you mean by small problems? Do you have millions of problems of size 6? In that case you should use EPSLAPACK. It will not factor the matrix (if solved as a non-symmetric problem). > > > > > > Jose > > > > > > > > > > El 8 jul 2020, a las 15:55, Aulisa, Eugenio escribi?: > > > > > > > > Dear Jose, > > > > > > > > Thank you for your answer. > > > > > > > > I tried MLU but it did not help. > > > > > > > > I also tried to eliminate the null space of B, but I do not know it apriori, > > > > and at least to my knowledge of PETSC-SLEPC there is not an explicit way to find it > > > > but to find all the 0 eigenvectors of B, which again it requires the inverse of B. > > > > > > > > I am solving similar small problems millions of times, and this works fine most of the time, > > > > but once in a while it fails with the same LU zero pivot exception. > > > > > > > > One thing I tried yesterday, and it seams to work all the times, > > > > is perturbing the diagonal of B of a small factor, += 1.0e-10 ||B||, > > > > but I do not know how theoretically it sounds with all these indeterminate eigenvalues, > > > > maybe you have a clue on this. > > > > > > > > Thanks, > > > > Eugenio > > > > > > > > > > > > > > > > Eugenio Aulisa > > > > > > > > Department of Mathematics and Statistics, > > > > Texas Tech University > > > > Lubbock TX, 79409-1042 > > > > room: 226 > > > > http://www.math.ttu.edu/~eaulisa/ > > > > phone: (806) 834-6684 > > > > fax: (806) 742-1112 > > > > > > > > > > > > > > > > > > > > From: Jose E. Roman > > > > Sent: Tuesday, July 7, 2020 11:01 AM > > > > To: Aulisa, Eugenio > > > > Cc: petsc-users at mcs.anl.gov ; Kara, Erdi > > > > Subject: Re: [petsc-users] Help with generalized eigenvalue problem > > > > > > > > I don't know what options you are using, shift-and-invert or not, which, target. Anyway, if A and B have a common null space, then A-sigma*B will always be singular, so the KSP solver will have to deal with a singular system. > > > > > > > > If you pass the nullspace vectors to the EPS solver via https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fslepc.upv.es%2Fdocumentation%2Fcurrent%2Fdocs%2Fmanualpages%2FEPS%2FEPSSetDeflationSpace.html&data=02%7C01%7CEugenio.Aulisa%40ttu.edu%7C951cc55964c449f2f8a508d82358c899%7C178a51bf8b2049ffb65556245d5c173c%7C0%7C0%7C637298211528226297&sdata=caA%2F9H6prcHP0z3lBK8cr65RLWGSJ2YOZDUDPhG0TRc%3D&reserved=0 then these vectors will also be attached to the KSP solver. > > > > > > > > Also, since you have MUMPS, try adding > > > > -st_pc_factor_mat_solver_type mumps > > > > (see section 3.4.1 of the SLEPc users manual). MUMPS should be quite robust with respect to singular linear systems. > > > > > > > > Jose > > > > > > > > > > > > > El 7 jul 2020, a las 17:40, Aulisa, Eugenio escribi?: > > > > > > > > > > Can somebody help me figure it out the solver options for the attached generalized eigenvalue problem? > > > > > > > > > > A x = lambda B x > > > > > > > > > > where A and B have the same null space, with 5 zero eigenvalues. > > > > > > > > > > The only non indefinite eigenvalue should be 41.1892 > > > > > > > > > > The problem is Hermite semi-positive so automatically slepc should purify it, but to be sure I added > > > > > > > > > > EPSSetPurify(eps, PETSC_TRUE); > > > > > > > > > > I guess with my non-solver-options slepc is still trying to do the inverse of the full > > > > > B matrix with an LU decomposition and gets zero pivot. > > > > > > > > > > This is what I get as a PETSC error Message > > > > > > > > > > [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- > > > > > [0]PETSC ERROR: Zero pivot in LU factorization: https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.mcs.anl.gov%2Fpetsc%2Fdocumentation%2Ffaq.html%23zeropivot&data=02%7C01%7CEugenio.Aulisa%40ttu.edu%7C951cc55964c449f2f8a508d82358c899%7C178a51bf8b2049ffb65556245d5c173c%7C0%7C0%7C637298211528236290&sdata=%2BAJlj0zMuPh2n5ngCwl6ekNUYx6IKIARihb5YYtFj9k%3D&reserved=0 > > > > > [0]PETSC ERROR: Bad LU factorization > > > > > [0]PETSC ERROR: See https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.mcs.anl.gov%2Fpetsc%2Fdocumentation%2Ffaq.html&data=02%7C01%7CEugenio.Aulisa%40ttu.edu%7C951cc55964c449f2f8a508d82358c899%7C178a51bf8b2049ffb65556245d5c173c%7C0%7C0%7C637298211528236290&sdata=AuuSqmYoElk9tNZUxPvWovk6RkUMQ47CIhQG7Dp8T9Q%3D&reserved=0 for trouble shooting. > > > > > [0]PETSC ERROR: Petsc Development GIT revision: v3.11.3-2263-gce77f2ed1a GIT Date: 2019-09-26 13:31:14 -0500 > > > > > [0]PETSC ERROR: Nitsche_ex4a on a arch-linux2-c-opt named linux-8biu by eaulisa Tue Jul 7 10:17:55 2020 > > > > > [0]PETSC ERROR: Configure options --with-debugging=0 --with-x=1 COPTFLAGS="-O3 -march=native -mtune=native" CXXOPTFLAGS="-O3 -march=native -mtune=native" FOPTFLAGS="-O3 -march=native -mtune=native" --download-openmpi=1 --download-fblaslapack=1 --download-hdf5=1 --download-metis=1 --download-parmetis=1 --with-shared-libraries=1 --download-blacs=1 --download-scalapack=1 --download-mumps=1 --download-suitesparse > > > > > [0]PETSC ERROR: #1 MatLUFactor_SeqDense() line 633 in /home/eaulisa/software/petsc/src/mat/impls/dense/seq/dense.c > > > > > [0]PETSC ERROR: #2 MatLUFactorNumeric_SeqDense() line 432 in /home/eaulisa/software/petsc/src/mat/impls/dense/seq/dense.c > > > > > [0]PETSC ERROR: #3 MatLUFactorNumeric() line 3056 in /home/eaulisa/software/petsc/src/mat/interface/matrix.c > > > > > [0]PETSC ERROR: #4 PCSetUp_LU() line 126 in /home/eaulisa/software/petsc/src/ksp/pc/impls/factor/lu/lu.c > > > > > [0]PETSC ERROR: #5 PCSetUp() line 894 in /home/eaulisa/software/petsc/src/ksp/pc/interface/precon.c > > > > > [0]PETSC ERROR: #6 KSPSetUp() line 377 in /home/eaulisa/software/petsc/src/ksp/ksp/interface/itfunc.c > > > > > [0]PETSC ERROR: #7 STSetUp_Shift() line 119 in /home/eaulisa/software/slepc/src/sys/classes/st/impls/shift/shift.c > > > > > [0]PETSC ERROR: #8 STSetUp() line 271 in /home/eaulisa/software/slepc/src/sys/classes/st/interface/stsolve.c > > > > > [0]PETSC ERROR: #9 EPSSetUp() line 273 in /home/eaulisa/software/slepc/src/eps/interface/epssetup.c > > > > > [0]PETSC ERROR: #10 EPSSolve() line 136 in /home/eaulisa/software/slepc/src/eps/interface/epssolve.c > > > > > > > > > > > > > > > > > > > > Thanks > > > > > Eugenio > > > > > > > > > > > > > > > > > > > > > > > > > Eugenio Aulisa > > > > > > > > > > Department of Mathematics and Statistics, > > > > > Texas Tech University > > > > > Lubbock TX, 79409-1042 > > > > > room: 226 > > > > > http://www.math.ttu.edu/~eaulisa/ > > > > > phone: (806) 834-6684 > > > > > fax: (806) 742-1112 > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From lu_qin_2000 at yahoo.com Wed Jul 8 13:42:55 2020 From: lu_qin_2000 at yahoo.com (Qin Lu) Date: Wed, 8 Jul 2020 18:42:55 +0000 (UTC) Subject: [petsc-users] Zero diagonal term for direct solver References: <709542050.2254524.1594233775302.ref@mail.yahoo.com> Message-ID: <709542050.2254524.1594233775302@mail.yahoo.com> Hello, I am using the Petsc native direct solver (with KSPPREONLY and PCLU) to solver a small Linear equation system. The matrix has a couple of diagonal terms with value 0, which makes the solver fail. Then I set a very small value (e.g., 1e-10) at these terms and the solver works. Can this cause?noticeable errors in the solution? Are there any options in Petsc to make the direct solver do row/column pivoting so that all the?diagonal terms are non-zero? Thanks for any suggestions. Regards,Qin -------------- next part -------------- An HTML attachment was scrubbed... URL: From mfadams at lbl.gov Wed Jul 8 14:11:35 2020 From: mfadams at lbl.gov (Mark Adams) Date: Wed, 8 Jul 2020 15:11:35 -0400 Subject: [petsc-users] Zero diagonal term for direct solver In-Reply-To: <709542050.2254524.1594233775302@mail.yahoo.com> References: <709542050.2254524.1594233775302.ref@mail.yahoo.com> <709542050.2254524.1594233775302@mail.yahoo.com> Message-ID: On Wed, Jul 8, 2020 at 2:43 PM Qin Lu via petsc-users < petsc-users at mcs.anl.gov> wrote: > Hello, > > I am using the Petsc native direct solver (with KSPPREONLY and PCLU) to > solver a small Linear equation system. The matrix has a couple of diagonal > terms with value 0, which makes the solver fail. Then I set a very small > value (e.g., 1e-10) at these terms and the solver works. Can this > cause noticeable errors in the solution? > Can't say. > Are there any options in Petsc to make the direct solver do row/column > pivoting so that all the diagonal terms are non-zero? > I see: -pc_factor_pivot_in_blocks - allow pivoting within the small blocks during factorization (may increase stability of factorization. and -pc_factor_shift_type - Sets shift type or PETSC_DECIDE for the default; use '-help' for a list of available types -pc_factor_shift_amount - Sets shift amount or PETSC_DECIDE for the default > > > Thanks for any suggestions. > > Regards, > Qin > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at petsc.dev Wed Jul 8 14:15:09 2020 From: bsmith at petsc.dev (Barry Smith) Date: Wed, 8 Jul 2020 14:15:09 -0500 Subject: [petsc-users] Zero diagonal term for direct solver In-Reply-To: References: <709542050.2254524.1594233775302.ref@mail.yahoo.com> <709542050.2254524.1594233775302@mail.yahoo.com> Message-ID: There is also -pc_factor_nonzeros_along_diagonal which reorders to move the zeros off the diagonal before the numerical factorization But if having 0 leads to a factorization without error then it will produce the correct answer (this means that zero got filled in during the factorization) > On Jul 8, 2020, at 2:11 PM, Mark Adams wrote: > > > > On Wed, Jul 8, 2020 at 2:43 PM Qin Lu via petsc-users > wrote: > Hello, > > I am using the Petsc native direct solver (with KSPPREONLY and PCLU) to solver a small Linear equation system. The matrix has a couple of diagonal terms with value 0, which makes the solver fail. Then I set a very small value (e.g., 1e-10) at these terms and the solver works. Can this cause noticeable errors in the solution? > > Can't say. > > Are there any options in Petsc to make the direct solver do row/column pivoting so that all the diagonal terms are non-zero? > > I see: > > > -pc_factor_pivot_in_blocks - allow pivoting within the small blocks during factorization (may increase stability of factorization. > > and > > > -pc_factor_shift_type - Sets shift type or PETSC_DECIDE for the default; use '-help' for a list of available types > -pc_factor_shift_amount - Sets shift amount or PETSC_DECIDE for the default > > > > Thanks for any suggestions. > > Regards, > Qin -------------- next part -------------- An HTML attachment was scrubbed... URL: From nathaniel.collier at gmail.com Wed Jul 8 14:16:36 2020 From: nathaniel.collier at gmail.com (Nathan Collier) Date: Wed, 8 Jul 2020 15:16:36 -0400 Subject: [petsc-users] Zero diagonal term for direct solver In-Reply-To: References: <709542050.2254524.1594233775302.ref@mail.yahoo.com> <709542050.2254524.1594233775302@mail.yahoo.com> Message-ID: I was given advice to use Suitesparse in these circumstances: * configure with --download-suitesparse * run with -pc_type lu -pc_factor_mat_solver_type umfpack Nate On Wed, Jul 8, 2020 at 3:12 PM Mark Adams wrote: > > > On Wed, Jul 8, 2020 at 2:43 PM Qin Lu via petsc-users < > petsc-users at mcs.anl.gov> wrote: > >> Hello, >> >> I am using the Petsc native direct solver (with KSPPREONLY and PCLU) to >> solver a small Linear equation system. The matrix has a couple of diagonal >> terms with value 0, which makes the solver fail. Then I set a very small >> value (e.g., 1e-10) at these terms and the solver works. Can this >> cause noticeable errors in the solution? >> > > Can't say. > > >> Are there any options in Petsc to make the direct solver do row/column >> pivoting so that all the diagonal terms are non-zero? >> > > I see: > > > -pc_factor_pivot_in_blocks - allow pivoting within the small > blocks during factorization (may increase stability of factorization. > > and > > > -pc_factor_shift_type - Sets shift type or PETSC_DECIDE for the > default; use '-help' for a list of available types > -pc_factor_shift_amount - Sets shift amount or PETSC_DECIDE > for the default > > >> >> >> Thanks for any suggestions. >> >> Regards, >> Qin >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From Serge.Van.Criekingen at idris.fr Thu Jul 9 07:48:29 2020 From: Serge.Van.Criekingen at idris.fr (Serge Van Criekingen) Date: Thu, 9 Jul 2020 14:48:29 +0200 Subject: [petsc-users] MatCreateSubMatrices in PETSc 3.12.5 vs. 3.13.3 Message-ID: Dear PETSc team, I have a parallel matrix that, in some cases, I want? to turn into a sequential matrix on rank 0. I have been succesfully using "MatCreateSubMatrices" for this purpose, along the following lines (a small but complete reproducer is attached): -- if (MPIrank==0){ ????? nSubMatToExtract=1; ????? ierr = ISCreateStride(PETSC_COMM_WORLD,globalSizeOfTheMatrix,0,1,&IS_MPI); CHKERRQ(ierr); } else { ????? nSubMatToExtract=0; ????? ierr = ISCreateStride(PETSC_COMM_WORLD,0,0,1,&IS_MPI); CHKERRQ(ierr); } ierr = MatCreateSubMatrices(A,nSubMatToExtract,&IS_MPI,&IS_MPI,MAT_INITIAL_MATRIX,Aseq); -- Unfortunately this does not seem to work anymore with petsc 3.13.3 (while it worked at least up to 3.12.5, the latest version I tried before 3.13.3). The error message mainly says: -- PETSC ERROR: Petsc has generated inconsistent data PETSC ERROR: MPI_Allreduce() called in different locations (code lines) on different processors -- So was I just lucky that it worked before? And/or is there another (better) way to reach this goal? For your information, I had this implemented to obtain and then solve a coarse-level domain decomposition matrix before discovering the possibility of using the "Telescope" option to agglomerate the unknowns spread on the subdomains. I managed to use Telescope, but I wish to know if I could go on using my previous way of doing in case I want to gather all the coarse-level unknowns on only one rank. Thanks a lot, Serge -------------- next part -------------- A non-text attachment was scrubbed... Name: mat.c Type: text/x-csrc Size: 1886 bytes Desc: not available URL: From bsmith at petsc.dev Thu Jul 9 07:53:15 2020 From: bsmith at petsc.dev (Barry Smith) Date: Thu, 9 Jul 2020 07:53:15 -0500 Subject: [petsc-users] MatCreateSubMatrices in PETSc 3.12.5 vs. 3.13.3 In-Reply-To: References: Message-ID: <51C18639-F550-46E8-88FB-4CF27566C242@petsc.dev> Change PETSC_COMM_WORLD to PETSC_COMM_SELF Barry > On Jul 9, 2020, at 7:48 AM, Serge Van Criekingen wrote: > > Dear PETSc team, > > I have a parallel matrix that, in some cases, I want to turn into a sequential matrix on rank 0. I have been succesfully using "MatCreateSubMatrices" for this purpose, along the following lines (a small but complete reproducer is attached): > -- > if (MPIrank==0){ > nSubMatToExtract=1; > ierr = ISCreateStride(PETSC_COMM_WORLD,globalSizeOfTheMatrix,0,1,&IS_MPI); CHKERRQ(ierr); > } else { > nSubMatToExtract=0; > ierr = ISCreateStride(PETSC_COMM_WORLD,0,0,1,&IS_MPI); CHKERRQ(ierr); > } > ierr = MatCreateSubMatrices(A,nSubMatToExtract,&IS_MPI,&IS_MPI,MAT_INITIAL_MATRIX,Aseq); > -- > Unfortunately this does not seem to work anymore with petsc 3.13.3 (while it worked at least up to 3.12.5, the latest version I tried before 3.13.3). > The error message mainly says: > -- > PETSC ERROR: Petsc has generated inconsistent data > PETSC ERROR: MPI_Allreduce() called in different locations (code lines) on different processors > -- > So was I just lucky that it worked before? And/or is there another (better) way to reach this goal? > > For your information, I had this implemented to obtain and then solve a coarse-level domain decomposition matrix before discovering the possibility of using the "Telescope" option to agglomerate the unknowns spread on the subdomains. I managed to use Telescope, but I wish to know if I could go on using my previous way of doing in case I want to gather all the coarse-level unknowns on only one rank. > > Thanks a lot, > > Serge > > > From andrea_iob at hotmail.com Thu Jul 9 07:55:23 2020 From: andrea_iob at hotmail.com (Andrea Iob) Date: Thu, 9 Jul 2020 12:55:23 +0000 Subject: [petsc-users] DIVERGED_NANORINF when using HYPRE/BoomerAMG In-Reply-To: References: , Message-ID: Yes, I'm using AIJ matrices. These are the options I'm using to set up BoomerAMG: PCHYPRESetType(preconditioner, "boomeramg"); PetscOptionsSetValue(nullptr, "-pc_hypre_boomeramg_smooth_type", "Pilut"); PetscOptionsSetValue(nullptr, "-pc_hypre_boomeramg_relax_type_all", "sequential-Gauss-Seidel"); PetscOptionsSetValue(nullptr, "-pc_hypre_boomeramg_strong_threshold", "0.7"); PetscOptionsSetValue(nullptr, "-pc_hypre_boomeramg_coarsen_type", "HMIS"); PetscOptionsSetValue(nullptr, "-pc_hypre_boomeramg_interp_type", "classical"); PetscOptionsSetValue(nullptr, "-pc_hypre_boomeramg_interp_type", "ext+i"); PetscOptionsSetValue(nullptr, "-pc_hypre_boomeramg_P_max", "2"); PetscOptionsSetValue(nullptr, "-pc_hypre_boomeramg_truncfactor", "0.3"); PetscOptionsSetValue(nullptr, "-pc_hypre_boomeramg_agg_nl", "4"); PetscOptionsSetValue(nullptr, "-pc_hypre_boomeramg_agg_num_paths", "5"); PetscOptionsSetValue(nullptr, "-ksp_error_if_not_converged", ""); This setup come from the following https://mooseframework.inl.gov/application_development/hypre.html ________________________________ From: Mark Adams Sent: Friday, July 3, 2020 2:58 PM To: Andrea Iob Cc: Barry Smith ; Matthew Knepley ; petsc-users at mcs.anl.gov Subject: Re: [petsc-users] DIVERGED_NANORINF when using HYPRE/BoomerAMG 3) Schwarz-smoothers #0 0x00007ffff50897f4 in dtrsm_ () from /opt/lapack/3.8.0-gcc.7.4.0/lib64/libblas.so.3 #1 0x00007ffff550924b in dpotrs_ () from /opt/lapack/3.8.0-gcc.7.4.0/lib64/liblapack.so.3 This seems to be Cholesky ... which would not work for you if you are not symmetric (Navier-Stokes). If GAMG works then and pilut going crazy but quite there yet, I would think there is a hypre interface problem of some sort. Hypre has been used a lot so this is a surprise. Do you use AIJ matrices? Not BAIJ or anything else. -------------- next part -------------- An HTML attachment was scrubbed... URL: From andrea_iob at hotmail.com Thu Jul 9 07:59:42 2020 From: andrea_iob at hotmail.com (Andrea Iob) Date: Thu, 9 Jul 2020 12:59:42 +0000 Subject: [petsc-users] DIVERGED_NANORINF when using HYPRE/BoomerAMG In-Reply-To: References: , , Message-ID: Sorry, I've clicked the send button by mistake. The setup I'm using comes from the following guide: https://mooseframework.inl.gov/application_development/hypre.html Since I'm new to BoomerAMG I've been using that guide as a reference, but I'm not sure if those setting are appropriate from my system. Best regards, Andrea ________________________________ From: Andrea Iob Sent: Thursday, July 9, 2020 2:55 PM To: Mark Adams Cc: Barry Smith ; Matthew Knepley ; petsc-users at mcs.anl.gov Subject: Re: [petsc-users] DIVERGED_NANORINF when using HYPRE/BoomerAMG Yes, I'm using AIJ matrices. These are the options I'm using to set up BoomerAMG: PCHYPRESetType(preconditioner, "boomeramg"); PetscOptionsSetValue(nullptr, "-pc_hypre_boomeramg_smooth_type", "Pilut"); PetscOptionsSetValue(nullptr, "-pc_hypre_boomeramg_relax_type_all", "sequential-Gauss-Seidel"); PetscOptionsSetValue(nullptr, "-pc_hypre_boomeramg_strong_threshold", "0.7"); PetscOptionsSetValue(nullptr, "-pc_hypre_boomeramg_coarsen_type", "HMIS"); PetscOptionsSetValue(nullptr, "-pc_hypre_boomeramg_interp_type", "classical"); PetscOptionsSetValue(nullptr, "-pc_hypre_boomeramg_interp_type", "ext+i"); PetscOptionsSetValue(nullptr, "-pc_hypre_boomeramg_P_max", "2"); PetscOptionsSetValue(nullptr, "-pc_hypre_boomeramg_truncfactor", "0.3"); PetscOptionsSetValue(nullptr, "-pc_hypre_boomeramg_agg_nl", "4"); PetscOptionsSetValue(nullptr, "-pc_hypre_boomeramg_agg_num_paths", "5"); PetscOptionsSetValue(nullptr, "-ksp_error_if_not_converged", ""); This setup come from the following https://mooseframework.inl.gov/application_development/hypre.html ________________________________ From: Mark Adams Sent: Friday, July 3, 2020 2:58 PM To: Andrea Iob Cc: Barry Smith ; Matthew Knepley ; petsc-users at mcs.anl.gov Subject: Re: [petsc-users] DIVERGED_NANORINF when using HYPRE/BoomerAMG 3) Schwarz-smoothers #0 0x00007ffff50897f4 in dtrsm_ () from /opt/lapack/3.8.0-gcc.7.4.0/lib64/libblas.so.3 #1 0x00007ffff550924b in dpotrs_ () from /opt/lapack/3.8.0-gcc.7.4.0/lib64/liblapack.so.3 This seems to be Cholesky ... which would not work for you if you are not symmetric (Navier-Stokes). If GAMG works then and pilut going crazy but quite there yet, I would think there is a hypre interface problem of some sort. Hypre has been used a lot so this is a surprise. Do you use AIJ matrices? Not BAIJ or anything else. -------------- next part -------------- An HTML attachment was scrubbed... URL: From mfadams at lbl.gov Thu Jul 9 09:06:20 2020 From: mfadams at lbl.gov (Mark Adams) Date: Thu, 9 Jul 2020 10:06:20 -0400 Subject: [petsc-users] DIVERGED_NANORINF when using HYPRE/BoomerAMG In-Reply-To: References: Message-ID: We don't know the details of hypre, we just provide an interface. For algorithmic questions you would need to ask the hypre tea. Tell them your equations and discretizations and what you have observed. I imagine they can help. Mark On Thu, Jul 9, 2020 at 8:59 AM Andrea Iob wrote: > Sorry, I've clicked the send button by mistake. > > The setup I'm using comes from the following guide: > > https://mooseframework.inl.gov/application_development/hypre.html > > Since I'm new to BoomerAMG I've been using that guide as a reference, but > I'm not sure if those setting are appropriate from my system. > > Best regards, > Andrea > > > ------------------------------ > *From:* Andrea Iob > *Sent:* Thursday, July 9, 2020 2:55 PM > *To:* Mark Adams > *Cc:* Barry Smith ; Matthew Knepley ; > petsc-users at mcs.anl.gov > *Subject:* Re: [petsc-users] DIVERGED_NANORINF when using HYPRE/BoomerAMG > > Yes, I'm using AIJ matrices. > > These are the options I'm using to set up BoomerAMG: > > PCHYPRESetType(preconditioner, "boomeramg"); > PetscOptionsSetValue(nullptr, "-pc_hypre_boomeramg_smooth_type", > "Pilut"); > PetscOptionsSetValue(nullptr, > "-pc_hypre_boomeramg_relax_type_all", "sequential-Gauss-Seidel"); > PetscOptionsSetValue(nullptr, > "-pc_hypre_boomeramg_strong_threshold", "0.7"); > PetscOptionsSetValue(nullptr, "-pc_hypre_boomeramg_coarsen_type", > "HMIS"); > PetscOptionsSetValue(nullptr, "-pc_hypre_boomeramg_interp_type", > "classical"); > PetscOptionsSetValue(nullptr, "-pc_hypre_boomeramg_interp_type", > "ext+i"); > PetscOptionsSetValue(nullptr, "-pc_hypre_boomeramg_P_max", "2"); > PetscOptionsSetValue(nullptr, "-pc_hypre_boomeramg_truncfactor", > "0.3"); > PetscOptionsSetValue(nullptr, "-pc_hypre_boomeramg_agg_nl", "4"); > PetscOptionsSetValue(nullptr, "-pc_hypre_boomeramg_agg_num_paths", > "5"); > PetscOptionsSetValue(nullptr, "-ksp_error_if_not_converged", ""); > > This setup come from the following > > https://mooseframework.inl.gov/application_development/hypre.html > > ------------------------------ > *From:* Mark Adams > *Sent:* Friday, July 3, 2020 2:58 PM > *To:* Andrea Iob > *Cc:* Barry Smith ; Matthew Knepley ; > petsc-users at mcs.anl.gov > *Subject:* Re: [petsc-users] DIVERGED_NANORINF when using HYPRE/BoomerAMG > > > > > 3) Schwarz-smoothers > > #0 0x00007ffff50897f4 in dtrsm_ () from > /opt/lapack/3.8.0-gcc.7.4.0/lib64/libblas.so.3 > #1 0x00007ffff550924b in dpotrs_ () from > /opt/lapack/3.8.0-gcc.7.4.0/lib64/liblapack.so.3 > > > This seems to be Cholesky ... which would not work for you if you are not > symmetric (Navier-Stokes). > > If GAMG works then and pilut going crazy but quite there yet, I would > think there is a hypre interface problem of some sort. > > Hypre has been used a lot so this is a surprise. Do you use AIJ matrices? > Not BAIJ or anything else. > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From lu_qin_2000 at yahoo.com Thu Jul 9 10:05:51 2020 From: lu_qin_2000 at yahoo.com (Qin Lu) Date: Thu, 9 Jul 2020 15:05:51 +0000 (UTC) Subject: [petsc-users] Zero diagonal term for direct solver In-Reply-To: References: <709542050.2254524.1594233775302.ref@mail.yahoo.com> <709542050.2254524.1594233775302@mail.yahoo.com> Message-ID: <1600308613.2572947.1594307151761@mail.yahoo.com> Thanks for all your input! How can I set these options in the program instead of as runtime options (in the .petsc file)? For example, which subroutine shall I call to set '-pc_factor_nonzeros_along_diagonal", and what is corresponding argument value? Thanks,Qin On Wednesday, July 8, 2020, 02:15:11 PM CDT, Barry Smith wrote: ? There is also?-pc_factor_nonzeros_along_diagonal which reorders to move the zeros off the diagonal before the numerical factorization ? But if having 0 leads to a factorization without error then it will produce the correct answer (this means that zero got filled in during the factorization) On Jul 8, 2020, at 2:11 PM, Mark Adams wrote: On Wed, Jul 8, 2020 at 2:43 PM Qin Lu via petsc-users wrote: Hello, I am using the Petsc native direct solver (with KSPPREONLY and PCLU) to solver a small Linear equation system. The matrix has a couple of diagonal terms with value 0, which makes the solver fail. Then I set a very small value (e.g., 1e-10) at these terms and the solver works. Can this cause?noticeable errors in the solution? Can't say.? Are there any options in Petsc to make the direct solver do row/column pivoting so that all the?diagonal terms are non-zero? I see: -pc_factor_pivot_in_blocks - allow pivoting within the small blocks during factorization (may increase stability of factorization. and -pc_factor_shift_type - Sets shift type or PETSC_DECIDE for the default; use '-help' for a list of available types -pc_factor_shift_amount - Sets shift amount or PETSC_DECIDE for the default ? Thanks for any suggestions. Regards,Qin -------------- next part -------------- An HTML attachment was scrubbed... URL: From hzhang at mcs.anl.gov Thu Jul 9 10:39:38 2020 From: hzhang at mcs.anl.gov (Zhang, Hong) Date: Thu, 9 Jul 2020 15:39:38 +0000 Subject: [petsc-users] Zero diagonal term for direct solver In-Reply-To: <1600308613.2572947.1594307151761@mail.yahoo.com> References: <709542050.2254524.1594233775302.ref@mail.yahoo.com> <709542050.2254524.1594233775302@mail.yahoo.com> , <1600308613.2572947.1594307151761@mail.yahoo.com> Message-ID: Qin: PCFactorReorderForNonzeroDiagonal() Hong ________________________________ From: petsc-users on behalf of Qin Lu via petsc-users Sent: Thursday, July 9, 2020 10:05 AM To: Mark Adams ; Barry Smith ; nathaniel.collier at gmail.com Cc: PETSc Users List Subject: Re: [petsc-users] Zero diagonal term for direct solver Thanks for all your input! How can I set these options in the program instead of as runtime options (in the .petsc file)? For example, which subroutine shall I call to set '-pc_factor_nonzeros_along_diagonal", and what is corresponding argument value? Thanks, Qin On Wednesday, July 8, 2020, 02:15:11 PM CDT, Barry Smith wrote: There is also -pc_factor_nonzeros_along_diagonal which reorders to move the zeros off the diagonal before the numerical factorization But if having 0 leads to a factorization without error then it will produce the correct answer (this means that zero got filled in during the factorization) On Jul 8, 2020, at 2:11 PM, Mark Adams > wrote: On Wed, Jul 8, 2020 at 2:43 PM Qin Lu via petsc-users > wrote: Hello, I am using the Petsc native direct solver (with KSPPREONLY and PCLU) to solver a small Linear equation system. The matrix has a couple of diagonal terms with value 0, which makes the solver fail. Then I set a very small value (e.g., 1e-10) at these terms and the solver works. Can this cause noticeable errors in the solution? Can't say. Are there any options in Petsc to make the direct solver do row/column pivoting so that all the diagonal terms are non-zero? I see: -pc_factor_pivot_in_blocks - allow pivoting within the small blocks during factorization (may increase stability of factorization. and -pc_factor_shift_type - Sets shift type or PETSC_DECIDE for the default; use '-help' for a list of available types -pc_factor_shift_amount - Sets shift amount or PETSC_DECIDE for the default Thanks for any suggestions. Regards, Qin -------------- next part -------------- An HTML attachment was scrubbed... URL: From d.scott at epcc.ed.ac.uk Thu Jul 9 10:53:12 2020 From: d.scott at epcc.ed.ac.uk (David Scott) Date: Thu, 9 Jul 2020 16:53:12 +0100 Subject: [petsc-users] DM_BOUNDARY_GHOSTED In-Reply-To: References: <0fb1334e-13f7-23f3-1c42-79286ed164c9@epcc.ed.ac.uk> Message-ID: <0f7c9c20-5594-78c5-2381-e3a202a3cffe@epcc.ed.ac.uk> Hello Matt, I am sorry that I have not replied before now. I think that the reason you did not understand the question was that the question was silly but I'll try to explain my thought process anyway. If you have a periodic boundary condition then when you construct a matrix to be used by a KSP it can address the ghost points corresponding to the periodic boundaries. So, I wondered if when you have ghosted boundary conditions you could refer to those ghosted points too when constructing the matrix. This led on to the question of how those values would be obtained when the linear solver came to be executed. At this point I expect you are asking why on earth I would want to do that and I do not have an adequate response. I am sorry I wasted your time and thanks for your help, David On 18/02/2020 20:23, Matthew Knepley wrote: > On Tue, Feb 18, 2020 at 10:23 AM SCOTT David > wrote: > > Hello Matt, > > Thanks for the quick response. > > Sorry to be dense but I want the ghost(ed) values to be used by > KSPSolve where the KSP has an associated DM. I can, of course > create a local vector using DMGetLocalVector and then populate the > ghosted locations but I do not know how to get the KSPSolve to use > these values in the way that automatically generated periodic, > ghost values would be. > > > Let me try and understand. We normally distinguish between > global?vectors and local vectors. Global vectors, used in the solver, > describe the entire space. Local vectors, used in assembly, describe > overlapping patches of the space. The ghost values would only be used > in assembly, not in the solver, since they help make the correct > value, but have no equation associated with themselves. Am I > misunderstanding something? > > ? Thanks, > > ? ? Matt > > Thanks, > > David > > On 18/02/2020 12:42, Matthew Knepley wrote: >> On Tue, Feb 18, 2020 at 6:03 AM David Scott >> > wrote: >> >> Hello, >> >> I wish to solve a channel flow problem with different boundary >> conditions. In the streamwise direction I may have periodic or >> inlet/outlet BCs. I would like to make my code for the two >> cases as >> similar as possible. If I use DM_BOUNDARY_PERIODIC then when >> performing >> a linear solve the ghost values will be set automatically. >> For the >> inlet/outlet case can I use DM_BOUNDARY_GHOSTED instead and >> somehow >> arrange for values that I specify to be placed in the ghost >> locations? >> >> >> Yes, that is the intent. >> >> ? Thanks, >> >> ? ? ?Matt >> >> Thanks, >> >> David >> >> The University of Edinburgh is a charitable body, registered >> in Scotland, with registration number SC005336. >> >> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to >> which their experiments lead. >> -- Norbert Wiener >> >> https://www.cse.buffalo.edu/~knepley/ >> > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which > their experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at petsc.dev Thu Jul 9 11:19:50 2020 From: bsmith at petsc.dev (Barry Smith) Date: Thu, 9 Jul 2020 11:19:50 -0500 Subject: [petsc-users] DM_BOUNDARY_GHOSTED In-Reply-To: <0f7c9c20-5594-78c5-2381-e3a202a3cffe@epcc.ed.ac.uk> References: <0fb1334e-13f7-23f3-1c42-79286ed164c9@epcc.ed.ac.uk> <0f7c9c20-5594-78c5-2381-e3a202a3cffe@epcc.ed.ac.uk> Message-ID: David, I don't think you can do it directly, you need to know how to adjust the matrix entry and make the entry. The reason is that when you put a value in the ghost location nothing knows that you have done this and thus nothing has the information to help with the Jacobian. Here is how you can do it with a contrived example Ghost location First location second location NULL x_0 x_1 .2*x_0 x_0 x_1 f(ghost,x_0) f(x_0,x_1) Now the derivative of f() with respect to x_0 is partial f/partial ghost * partial ghost /partial x_0 + partial f /partial x_0 partial f/partial ghost * .2 + partial f /partial x_0 Barry > On Jul 9, 2020, at 10:53 AM, David Scott wrote: > > Hello Matt, > > I am sorry that I have not replied before now. > > I think that the reason you did not understand the question was that the question was silly but I'll try to explain my thought process anyway. > > If you have a periodic boundary condition then when you construct a matrix to be used by a KSP it can address the ghost points corresponding to the periodic boundaries. So, I wondered if when you have ghosted boundary conditions you could refer to those ghosted points too when constructing the matrix. This led on to the question of how those values would be obtained when the linear solver came to be executed. > > At this point I expect you are asking why on earth I would want to do that and I do not have an adequate response. > > I am sorry I wasted your time and thanks for your help, > > David > > On 18/02/2020 20:23, Matthew Knepley wrote: >> On Tue, Feb 18, 2020 at 10:23 AM SCOTT David > wrote: >> Hello Matt, >> >> Thanks for the quick response. >> >> Sorry to be dense but I want the ghost(ed) values to be used by KSPSolve where the KSP has an associated DM. I can, of course create a local vector using DMGetLocalVector and then populate the ghosted locations but I do not know how to get the KSPSolve to use these values in the way that automatically generated periodic, ghost values would be. >> >> Let me try and understand. We normally distinguish between global vectors and local vectors. Global vectors, used in the solver, describe the entire space. Local vectors, used in assembly, describe overlapping patches of the space. The ghost values would only be used in assembly, not in the solver, since they help make the correct value, but have no equation associated with themselves. Am I misunderstanding something? >> >> Thanks, >> >> Matt >> >> Thanks, >> >> David >> >> On 18/02/2020 12:42, Matthew Knepley wrote: >>> On Tue, Feb 18, 2020 at 6:03 AM David Scott > wrote: >>> Hello, >>> >>> I wish to solve a channel flow problem with different boundary >>> conditions. In the streamwise direction I may have periodic or >>> inlet/outlet BCs. I would like to make my code for the two cases as >>> similar as possible. If I use DM_BOUNDARY_PERIODIC then when performing >>> a linear solve the ghost values will be set automatically. For the >>> inlet/outlet case can I use DM_BOUNDARY_GHOSTED instead and somehow >>> arrange for values that I specify to be placed in the ghost locations? >>> >>> Yes, that is the intent. >>> >>> Thanks, >>> >>> Matt >>> >>> Thanks, >>> >>> David >>> >>> The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. >>> >>> >>> -- >>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. >>> -- Norbert Wiener >>> >>> https://www.cse.buffalo.edu/~knepley/ >> >> >> >> -- >> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. >> -- Norbert Wiener >> >> https://www.cse.buffalo.edu/~knepley/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From d.scott at epcc.ed.ac.uk Thu Jul 9 13:50:57 2020 From: d.scott at epcc.ed.ac.uk (David Scott) Date: Thu, 9 Jul 2020 19:50:57 +0100 Subject: [petsc-users] HDF5 and ParaView Message-ID: Hello, I have written out a Vec using the (Fortran) HDF5 routines provided by PETSc. I now want to import the data into ParaView 5.8.0 and display a contour. I can read the data in using VisItPixieReader but the contour tool is not available. Can I use the HDF5 files produced by PETSc 13.3.3 with ParaView and, if so, how? David The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. From d.scott at epcc.ed.ac.uk Thu Jul 9 13:58:55 2020 From: d.scott at epcc.ed.ac.uk (David Scott) Date: Thu, 9 Jul 2020 19:58:55 +0100 Subject: [petsc-users] HDF5 and ParaView In-Reply-To: References: Message-ID: I should have said that I have created files with the suffix '.h5'. Was that the right thing to do? David On 09/07/2020 19:50, David Scott wrote: > Hello, > > I have written out a Vec using the (Fortran) HDF5 routines provided by > PETSc. > I now want to import the data into ParaView 5.8.0 and display a contour. > I can read the data in using VisItPixieReader but the contour tool is > not available. Can I use the HDF5 files produced by PETSc 13.3.3 with > ParaView and, if so, how? > > David > > > > The University of Edinburgh is a charitable body, registered in > Scotland, with registration number SC005336. From knepley at gmail.com Thu Jul 9 14:17:52 2020 From: knepley at gmail.com (Matthew Knepley) Date: Thu, 9 Jul 2020 15:17:52 -0400 Subject: [petsc-users] HDF5 and ParaView In-Reply-To: References: Message-ID: On Thu, Jul 9, 2020 at 2:51 PM David Scott wrote: > Hello, > > I have written out a Vec using the (Fortran) HDF5 routines provided by > PETSc. > I now want to import the data into ParaView 5.8.0 and display a contour. > I can read the data in using VisItPixieReader but the contour tool is > not available. Can I use the HDF5 files produced by PETSc 13.3.3 with > ParaView and, if so, how? > When you call VecView(), it just writes an array of real numbers to the HDF5 file. If Paraview can use an array of real numbers, then you are all set. I don't know how to make Paraview do that. What I do is to write an Xdmf file that points to the HDF5 file for the data and specifies the mesh using XML. Note that you can have a DMDA or DMPlex write VTK files, which Paraview can read directly. Thanks, Matt > David > > > > The University of Edinburgh is a charitable body, registered in Scotland, > with registration number SC005336. > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From d.scott at epcc.ed.ac.uk Thu Jul 9 14:28:46 2020 From: d.scott at epcc.ed.ac.uk (David Scott) Date: Thu, 9 Jul 2020 20:28:46 +0100 Subject: [petsc-users] HDF5 and ParaView In-Reply-To: References: Message-ID: <3d04f25d-7de7-4d44-79c1-ed736daf649d@epcc.ed.ac.uk> Matt, I'll have a go at writing Xdmf files. Am I right in thinking that VTK files are written sequentially? Thanks, David On 09/07/2020 20:17, Matthew Knepley wrote: > On Thu, Jul 9, 2020 at 2:51 PM David Scott > wrote: > > Hello, > > I have written out a Vec using the (Fortran) HDF5 routines provided by > PETSc. > I now want to import the data into ParaView 5.8.0 and display a > contour. > I can read the data in using VisItPixieReader but the contour tool is > not available. Can I use the HDF5 files produced by PETSc 13.3.3 with > ParaView and, if so, how? > > > When you call VecView(), it just writes an array of real numbers to > the HDF5 file. If Paraview can use an array > of real numbers, then you are all set. I don't know how to make > Paraview do that. What I do is to write an > Xdmf file that points to the HDF5 file for the data and specifies the > mesh using XML. Note? that you can have > a DMDA or DMPlex write VTK files, which Paraview can read directly. > > ? Thanks, > > ? ? ?Matt > > David > > > > The University of Edinburgh is a charitable body, registered in > Scotland, with registration number SC005336. > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which > their experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Thu Jul 9 15:10:34 2020 From: knepley at gmail.com (Matthew Knepley) Date: Thu, 9 Jul 2020 16:10:34 -0400 Subject: [petsc-users] HDF5 and ParaView In-Reply-To: <3d04f25d-7de7-4d44-79c1-ed736daf649d@epcc.ed.ac.uk> References: <3d04f25d-7de7-4d44-79c1-ed736daf649d@epcc.ed.ac.uk> Message-ID: On Thu, Jul 9, 2020 at 3:28 PM David Scott wrote: > Matt, > > I'll have a go at writing Xdmf files. > > Am I right in thinking that VTK files are written sequentially? > Yes, it is a rightly despised format, suitable only for beggars and serial jobs :) Thanks, Matt > Thanks, > > David > > On 09/07/2020 20:17, Matthew Knepley wrote: > > On Thu, Jul 9, 2020 at 2:51 PM David Scott wrote: > >> Hello, >> >> I have written out a Vec using the (Fortran) HDF5 routines provided by >> PETSc. >> I now want to import the data into ParaView 5.8.0 and display a contour. >> I can read the data in using VisItPixieReader but the contour tool is >> not available. Can I use the HDF5 files produced by PETSc 13.3.3 with >> ParaView and, if so, how? >> > > When you call VecView(), it just writes an array of real numbers to the > HDF5 file. If Paraview can use an array > of real numbers, then you are all set. I don't know how to make Paraview > do that. What I do is to write an > Xdmf file that points to the HDF5 file for the data and specifies the mesh > using XML. Note that you can have > a DMDA or DMPlex write VTK files, which Paraview can read directly. > > Thanks, > > Matt > > >> David >> >> >> >> The University of Edinburgh is a charitable body, registered in Scotland, >> with registration number SC005336. >> > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ > > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From junchao.zhang at gmail.com Thu Jul 9 16:45:27 2020 From: junchao.zhang at gmail.com (Junchao Zhang) Date: Thu, 9 Jul 2020 16:45:27 -0500 Subject: [petsc-users] Parallel sort in Petsc In-Reply-To: References: Message-ID: Antoine, See the attached example. It does sort on rank 0 but puts the permutation results in parallel. Run it with mpirun -n 3 ./ex2 for a demo. Thanks. --Junchao Zhang On Thu, Jul 9, 2020 at 1:24 PM Antoine C?t? wrote: > The Vec containing the scalars represents stresses at nodes points of a > DMDA. My program sorts stresses, and makes calculations according to their > rank. This needs to be done while keeping track of the original positions > of stresses in Vec. Using permutations, I can access the node of my choice > and get its associated rank, without changing the original Vec. > > Regards, > > Antoine > > > ------------------------------ > *De :* Junchao Zhang > *Envoy? :* 9 juillet 2020 12:42 > *? :* Antoine C?t? > *Objet :* Re: [petsc-users] Parallel sort in Petsc > > It is easier without permutations. So let me ask first do you need the > permutations or just a (parallel) sorted vector? > > --Junchao Zhang > > > On Thu, Jul 9, 2020 at 11:21 AM Antoine C?t? > wrote: > > That is worth trying! > > Say I gather to process 0 as proposed here : > https://www.mcs.anl.gov/petsc/documentation/faq.html#mpi-vec-to-mpi-vec > > Let Vec A be the vector containing the scalars. If I compute the > permutations I'm looking for, how do I : > - tell the code to run PetscSortRealWithPermutation() *only *on process > 0 ? > - distribute back the permutations on all process (that is, send only the > permutations concerning the local portion of Vec A stored by a given > process) ? > - make sure other processes wait until they receive the permutations > prior proceeding with the rest of the code ? > > Thank you very much! > > Antoine > > ------------------------------ > *De :* Junchao Zhang > *Envoy? :* 7 juillet 2020 10:52 > *? :* Antoine C?t? > *Cc :* petsc-users at mcs.anl.gov > *Objet :* Re: [petsc-users] Parallel sort in Petsc > > Gather all values to rank 0 and do a sequential sort there, then profile > to see whether it is a performance bottleneck in your code. > > --Junchao Zhang > > > On Tue, Jul 7, 2020 at 8:14 AM Antoine C?t? > wrote: > > Hi, > > I need to compute the permutations to sort scalar values of a Vec object > in decreasing order. I've been developing on a single process for now, > using VecGetArrayRead() to extract the values > and PetscSortRealWithPermutation() to get the permutations. > > I would like to run the code on multiple processes. From these links, I > can see it's an issue : > https://lists.mcs.anl.gov/pipermail/petsc-users/2013-November/019628.html > > https://lists.mcs.anl.gov/pipermail/petsc-users/2009-June/004621.html > > > One work around I've been considering : get the max scalar value using > VecMax() ; compute a scaling factor that makes this max value equals the > largest integer value tolerated by PetscInt ; scale all values of Vec and > convert them to integers ; use PetscParallelSortInt() ; inverse the scaling > on (now sorted) integers values and convert them back to scalars. > > This is only a patch really, the main issues being : (1) I would get a > sorted Vec, not the permutations and (2) there would be a lost of > information when converting scalars to int values. > > Isn't there a way to parallel sort real values using Petsc? > > Thank you very much! > > Antoine C?t? > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: ex2.c Type: application/octet-stream Size: 3365 bytes Desc: not available URL: From eda.oktay at metu.edu.tr Fri Jul 10 04:56:29 2020 From: eda.oktay at metu.edu.tr (Eda Oktay) Date: Fri, 10 Jul 2020 12:56:29 +0300 Subject: [petsc-users] SLEPc EPS Tolerance Message-ID: Hello, I am currently using EPS to find 2 smallest eigenvalues and corresponding eigenvectors of a matrix in order to be used in spectral graph partitioning algorithms. I know that default tolerance is E-8 and I also tried the solver with tolerances E-2, E-4 and E-6 expecting that in every matrix I used, I get the most accurate solution (After partitioning the matrix, I look at edge cut for accuracy) when tolerance is E-8. I even tried E-10 and the result was still the same, sometimes E-4 or E-2 gives better results. I was wondering if this is normal, if normal then how can we explain this? Because in theory, one can expect that if tolerance decreases, accuracy increases. Thanks! Eda From jroman at dsic.upv.es Fri Jul 10 05:01:11 2020 From: jroman at dsic.upv.es (Jose E. Roman) Date: Fri, 10 Jul 2020 12:01:11 +0200 Subject: [petsc-users] SLEPc EPS Tolerance In-Reply-To: References: Message-ID: <5FADECB0-7D5F-4249-B686-E67C3DBD9F16@dsic.upv.es> What do you mean "the result was still the same"? How do you measure accuracy? Which is the value of the computed residuals? What is the eigenvalue you are computing? Which convergence criterion are you using? > El 10 jul 2020, a las 11:56, Eda Oktay escribi?: > > Hello, > > I am currently using EPS to find 2 smallest eigenvalues and > corresponding eigenvectors of a matrix in order to be used in spectral > graph partitioning algorithms. I know that default tolerance is E-8 > and I also tried the solver with tolerances E-2, E-4 and E-6 expecting > that in every matrix I used, I get the most accurate solution (After > partitioning the matrix, I look at edge cut for accuracy) when > tolerance is E-8. > > I even tried E-10 and the result was still the same, sometimes E-4 or > E-2 gives better results. > > I was wondering if this is normal, if normal then how can we explain > this? Because in theory, one can expect that if tolerance decreases, > accuracy increases. > > Thanks! > > Eda From eda.oktay at metu.edu.tr Fri Jul 10 05:54:55 2020 From: eda.oktay at metu.edu.tr (Eda Oktay) Date: Fri, 10 Jul 2020 13:54:55 +0300 Subject: [petsc-users] SLEPc EPS Tolerance In-Reply-To: <5FADECB0-7D5F-4249-B686-E67C3DBD9F16@dsic.upv.es> References: <5FADECB0-7D5F-4249-B686-E67C3DBD9F16@dsic.upv.es> Message-ID: > How do you measure accuracy? Using the word accuracy may be not true actually, I am sorry. I am using eigenvectors corresponding to these eigenvalues in k-means algorithm, then do spectral graph partitioning. I must look at the partition quality. By quality, I mean, the resulting edge cut of my partitioned graph. I thought that the less tolerance results in more accuracy, hence more qualified partition. > What do you mean "the result was still the same"? I mean I am still not getting the most qualified solution in E-10, still E-2 or E-6 gives more qualified partitions, i.e. they give less edge cut. >What is the eigenvalue you are computing? I am computing the smallest eigenvalue of a Laplacian matrix. From jroman at dsic.upv.es Fri Jul 10 06:09:21 2020 From: jroman at dsic.upv.es (Jose E. Roman) Date: Fri, 10 Jul 2020 13:09:21 +0200 Subject: [petsc-users] SLEPc EPS Tolerance In-Reply-To: References: <5FADECB0-7D5F-4249-B686-E67C3DBD9F16@dsic.upv.es> Message-ID: <4ED7557C-6548-4E35-880A-4B481A70CE29@dsic.upv.es> > El 10 jul 2020, a las 12:54, Eda Oktay escribi?: > >> How do you measure accuracy? > Using the word accuracy may be not true actually, I am sorry. I am > using eigenvectors corresponding to these eigenvalues in k-means > algorithm, then do spectral graph partitioning. I must look at the > partition quality. By quality, I mean, the resulting edge cut of my > partitioned graph. I thought that the less tolerance results in more > accuracy, hence more qualified partition. > >> What do you mean "the result was still the same"? > I mean I am still not getting the most qualified solution in E-10, > still E-2 or E-6 gives more qualified partitions, i.e. they give less > edge cut. > >> What is the eigenvalue you are computing? > I am computing the smallest eigenvalue of a Laplacian matrix. You should compute the residual norm, for instance with -eps_error_relative ::ascii_info_detail (see section 2.5.4 of the manual). The relative residual error should be in the order of the tolerance (or smaller) if using the default convergence test, but if you are computing a zero eigenvalue then you may want to use an absolute convergence criterion (see table 2.6 or the manual). A graph Laplacian has at least one zero eigenvalue, unless you deflate it as explained in section 2.6.2 of the manual, see also ex11.c https://slepc.upv.es/documentation/current/src/eps/tutorials/ex11.c.html Jose From dalcinl at gmail.com Fri Jul 10 07:33:04 2020 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Fri, 10 Jul 2020 15:33:04 +0300 Subject: [petsc-users] HDF5 and ParaView In-Reply-To: References: <3d04f25d-7de7-4d44-79c1-ed736daf649d@epcc.ed.ac.uk> Message-ID: On Thu, 9 Jul 2020 at 23:11, Matthew Knepley wrote: > On Thu, Jul 9, 2020 at 3:28 PM David Scott wrote: > >> Matt, >> >> I'll have a go at writing Xdmf files. >> >> Am I right in thinking that VTK files are written sequentially? >> > > Yes, it is a rightly despised format, suitable only for beggars and serial > jobs :) > > By "rightly despised format", I hope you are talking about the legacy VTK files (the traditional .vtk extension). The new XML formats with are quite easy to write in parallel with MPI/IO (at least *.vtu files, which is what I'm using). -- Lisandro Dalcin ============ Research Scientist Extreme Computing Research Center (ECRC) King Abdullah University of Science and Technology (KAUST) http://ecrc.kaust.edu.sa/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Fri Jul 10 07:37:26 2020 From: knepley at gmail.com (Matthew Knepley) Date: Fri, 10 Jul 2020 08:37:26 -0400 Subject: [petsc-users] HDF5 and ParaView In-Reply-To: References: <3d04f25d-7de7-4d44-79c1-ed736daf649d@epcc.ed.ac.uk> Message-ID: On Fri, Jul 10, 2020 at 8:33 AM Lisandro Dalcin wrote: > On Thu, 9 Jul 2020 at 23:11, Matthew Knepley wrote: > >> On Thu, Jul 9, 2020 at 3:28 PM David Scott wrote: >> >>> Matt, >>> >>> I'll have a go at writing Xdmf files. >>> >>> Am I right in thinking that VTK files are written sequentially? >>> >> >> Yes, it is a rightly despised format, suitable only for beggars and >> serial jobs :) >> >> > By "rightly despised format", I hope you are talking about the legacy VTK > files (the traditional .vtk extension). > The new XML formats with are quite easy to > write in parallel with MPI/IO (at least *.vtu files, which is what I'm > using). > That is true. Do they also get rid of the single mesh and single timstep requirements? HDF5+XDMF makes it much easier since we can put multiple meshes and timesteps in one file. Thanks, Matt > -- > Lisandro Dalcin > ============ > Research Scientist > Extreme Computing Research Center (ECRC) > King Abdullah University of Science and Technology (KAUST) > http://ecrc.kaust.edu.sa/ > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From jroman at dsic.upv.es Fri Jul 10 11:31:16 2020 From: jroman at dsic.upv.es (Jose E. Roman) Date: Fri, 10 Jul 2020 18:31:16 +0200 Subject: [petsc-users] SLEPc EPS Tolerance In-Reply-To: References: <5FADECB0-7D5F-4249-B686-E67C3DBD9F16@dsic.upv.es> <4ED7557C-6548-4E35-880A-4B481A70CE29@dsic.upv.es> Message-ID: [Please respond to the list.] Is your matrix of size 8? This would explain the residuals. Jose > El 10 jul 2020, a las 17:10, Eda Oktay escribi?: > > I computed residual norm via -eps_error_relative::ascii_info_detail > for different tolerance numbers (e-4, e-6, e-8, e-10). In each > tolerance, I got the same table below: > > ---------------------- -------------------- > k ||Ax-kx||/||kx|| > ---------------------- -------------------- > 3.000000 6.25528e-16 > 3.000000 7.13774e-16 > 3.438447 2.64362e-16 > 5.000000 4.39333e-16 > 6.000000 1.63943e-16 > 6.000000 2.93737e-16 > 6.000000 3.95997e-16 > 7.561553 3.48664e-16 > ---------------------- -------------------- > > I understood that since relative error is E-16 and this table shows > eigenvalues whose relative error are below the tolerance, I am getting > the same table but I still couldn't understand although relative > errors are so small, how am I getting the most qualified partition in > e-4 tolerance, not e-8. I am not computing zero eigenvalue I believe, > since I am using EPSSetWhichEigenpairs(eps,EPS_SMALLEST_MAGNITUDE); > and I am not getting zero eigenvalue. > > Thanks so much for answering! > > Jose E. Roman , 10 Tem 2020 Cum, 14:09 tarihinde ?unu yazd?: >> >> >> >>> El 10 jul 2020, a las 12:54, Eda Oktay escribi?: >>> >>>> How do you measure accuracy? >>> Using the word accuracy may be not true actually, I am sorry. I am >>> using eigenvectors corresponding to these eigenvalues in k-means >>> algorithm, then do spectral graph partitioning. I must look at the >>> partition quality. By quality, I mean, the resulting edge cut of my >>> partitioned graph. I thought that the less tolerance results in more >>> accuracy, hence more qualified partition. >>> >>>> What do you mean "the result was still the same"? >>> I mean I am still not getting the most qualified solution in E-10, >>> still E-2 or E-6 gives more qualified partitions, i.e. they give less >>> edge cut. >>> >>>> What is the eigenvalue you are computing? >>> I am computing the smallest eigenvalue of a Laplacian matrix. >> >> You should compute the residual norm, for instance with -eps_error_relative ::ascii_info_detail (see section 2.5.4 of the manual). >> The relative residual error should be in the order of the tolerance (or smaller) if using the default convergence test, but if you are computing a zero eigenvalue then you may want to use an absolute convergence criterion (see table 2.6 or the manual). A graph Laplacian has at least one zero eigenvalue, unless you deflate it as explained in section 2.6.2 of the manual, see also ex11.c https://slepc.upv.es/documentation/current/src/eps/tutorials/ex11.c.html >> >> Jose >> From eda.oktay at metu.edu.tr Fri Jul 10 11:32:10 2020 From: eda.oktay at metu.edu.tr (Eda Oktay) Date: Fri, 10 Jul 2020 19:32:10 +0300 Subject: [petsc-users] SLEPc EPS Tolerance In-Reply-To: References: <5FADECB0-7D5F-4249-B686-E67C3DBD9F16@dsic.upv.es> <4ED7557C-6548-4E35-880A-4B481A70CE29@dsic.upv.es> Message-ID: No, it is of size 9 Jose E. Roman , 10 Tem 2020 Cum, 19:31 tarihinde ?unu yazd?: > > [Please respond to the list.] > > Is your matrix of size 8? This would explain the residuals. > > Jose > > > El 10 jul 2020, a las 17:10, Eda Oktay escribi?: > > > > I computed residual norm via -eps_error_relative::ascii_info_detail > > for different tolerance numbers (e-4, e-6, e-8, e-10). In each > > tolerance, I got the same table below: > > > > ---------------------- -------------------- > > k ||Ax-kx||/||kx|| > > ---------------------- -------------------- > > 3.000000 6.25528e-16 > > 3.000000 7.13774e-16 > > 3.438447 2.64362e-16 > > 5.000000 4.39333e-16 > > 6.000000 1.63943e-16 > > 6.000000 2.93737e-16 > > 6.000000 3.95997e-16 > > 7.561553 3.48664e-16 > > ---------------------- -------------------- > > > > I understood that since relative error is E-16 and this table shows > > eigenvalues whose relative error are below the tolerance, I am getting > > the same table but I still couldn't understand although relative > > errors are so small, how am I getting the most qualified partition in > > e-4 tolerance, not e-8. I am not computing zero eigenvalue I believe, > > since I am using EPSSetWhichEigenpairs(eps,EPS_SMALLEST_MAGNITUDE); > > and I am not getting zero eigenvalue. > > > > Thanks so much for answering! > > > > Jose E. Roman , 10 Tem 2020 Cum, 14:09 tarihinde ?unu yazd?: > >> > >> > >> > >>> El 10 jul 2020, a las 12:54, Eda Oktay escribi?: > >>> > >>>> How do you measure accuracy? > >>> Using the word accuracy may be not true actually, I am sorry. I am > >>> using eigenvectors corresponding to these eigenvalues in k-means > >>> algorithm, then do spectral graph partitioning. I must look at the > >>> partition quality. By quality, I mean, the resulting edge cut of my > >>> partitioned graph. I thought that the less tolerance results in more > >>> accuracy, hence more qualified partition. > >>> > >>>> What do you mean "the result was still the same"? > >>> I mean I am still not getting the most qualified solution in E-10, > >>> still E-2 or E-6 gives more qualified partitions, i.e. they give less > >>> edge cut. > >>> > >>>> What is the eigenvalue you are computing? > >>> I am computing the smallest eigenvalue of a Laplacian matrix. > >> > >> You should compute the residual norm, for instance with -eps_error_relative ::ascii_info_detail (see section 2.5.4 of the manual). > >> The relative residual error should be in the order of the tolerance (or smaller) if using the default convergence test, but if you are computing a zero eigenvalue then you may want to use an absolute convergence criterion (see table 2.6 or the manual). A graph Laplacian has at least one zero eigenvalue, unless you deflate it as explained in section 2.6.2 of the manual, see also ex11.c https://slepc.upv.es/documentation/current/src/eps/tutorials/ex11.c.html > >> > >> Jose > >> > From jroman at dsic.upv.es Fri Jul 10 11:36:07 2020 From: jroman at dsic.upv.es (Jose E. Roman) Date: Fri, 10 Jul 2020 18:36:07 +0200 Subject: [petsc-users] SLEPc EPS Tolerance In-Reply-To: References: <5FADECB0-7D5F-4249-B686-E67C3DBD9F16@dsic.upv.es> <4ED7557C-6548-4E35-880A-4B481A70CE29@dsic.upv.es> Message-ID: <111E5C52-89F2-4480-AF7D-5628F51BBA05@dsic.upv.es> For such small matrix, the solver is essentially solving with a direct method via LAPACK. If you understand how projection methods work, iteration gets in action when the matrix size is larger than the subspace size (ncv). The tolerance is relevant when the solver iterates, not when it computes the solution with LAPACK. Jose > El 10 jul 2020, a las 18:32, Eda Oktay escribi?: > > No, it is of size 9 > > Jose E. Roman , 10 Tem 2020 Cum, 19:31 tarihinde ?unu yazd?: >> >> [Please respond to the list.] >> >> Is your matrix of size 8? This would explain the residuals. >> >> Jose >> >>> El 10 jul 2020, a las 17:10, Eda Oktay escribi?: >>> >>> I computed residual norm via -eps_error_relative::ascii_info_detail >>> for different tolerance numbers (e-4, e-6, e-8, e-10). In each >>> tolerance, I got the same table below: >>> >>> ---------------------- -------------------- >>> k ||Ax-kx||/||kx|| >>> ---------------------- -------------------- >>> 3.000000 6.25528e-16 >>> 3.000000 7.13774e-16 >>> 3.438447 2.64362e-16 >>> 5.000000 4.39333e-16 >>> 6.000000 1.63943e-16 >>> 6.000000 2.93737e-16 >>> 6.000000 3.95997e-16 >>> 7.561553 3.48664e-16 >>> ---------------------- -------------------- >>> >>> I understood that since relative error is E-16 and this table shows >>> eigenvalues whose relative error are below the tolerance, I am getting >>> the same table but I still couldn't understand although relative >>> errors are so small, how am I getting the most qualified partition in >>> e-4 tolerance, not e-8. I am not computing zero eigenvalue I believe, >>> since I am using EPSSetWhichEigenpairs(eps,EPS_SMALLEST_MAGNITUDE); >>> and I am not getting zero eigenvalue. >>> >>> Thanks so much for answering! >>> >>> Jose E. Roman , 10 Tem 2020 Cum, 14:09 tarihinde ?unu yazd?: >>>> >>>> >>>> >>>>> El 10 jul 2020, a las 12:54, Eda Oktay escribi?: >>>>> >>>>>> How do you measure accuracy? >>>>> Using the word accuracy may be not true actually, I am sorry. I am >>>>> using eigenvectors corresponding to these eigenvalues in k-means >>>>> algorithm, then do spectral graph partitioning. I must look at the >>>>> partition quality. By quality, I mean, the resulting edge cut of my >>>>> partitioned graph. I thought that the less tolerance results in more >>>>> accuracy, hence more qualified partition. >>>>> >>>>>> What do you mean "the result was still the same"? >>>>> I mean I am still not getting the most qualified solution in E-10, >>>>> still E-2 or E-6 gives more qualified partitions, i.e. they give less >>>>> edge cut. >>>>> >>>>>> What is the eigenvalue you are computing? >>>>> I am computing the smallest eigenvalue of a Laplacian matrix. >>>> >>>> You should compute the residual norm, for instance with -eps_error_relative ::ascii_info_detail (see section 2.5.4 of the manual). >>>> The relative residual error should be in the order of the tolerance (or smaller) if using the default convergence test, but if you are computing a zero eigenvalue then you may want to use an absolute convergence criterion (see table 2.6 or the manual). A graph Laplacian has at least one zero eigenvalue, unless you deflate it as explained in section 2.6.2 of the manual, see also ex11.c https://slepc.upv.es/documentation/current/src/eps/tutorials/ex11.c.html >>>> >>>> Jose >>>> >> From eda.oktay at metu.edu.tr Fri Jul 10 11:45:56 2020 From: eda.oktay at metu.edu.tr (Eda Oktay) Date: Fri, 10 Jul 2020 19:45:56 +0300 Subject: [petsc-users] SLEPc EPS Tolerance In-Reply-To: <111E5C52-89F2-4480-AF7D-5628F51BBA05@dsic.upv.es> References: <5FADECB0-7D5F-4249-B686-E67C3DBD9F16@dsic.upv.es> <4ED7557C-6548-4E35-880A-4B481A70CE29@dsic.upv.es> <111E5C52-89F2-4480-AF7D-5628F51BBA05@dsic.upv.es> Message-ID: I looked at a2965*2965 sized matrix. I want to ask one last question. For e-8: Linear eigensolve converged (2 eigenpairs) due to CONVERGED_TOL; iterations 64 ---------------------- -------------------- k ||Ax-kx||/||kx|| ---------------------- -------------------- 0.002486 8.60766e-09 0.004466 1.68813e-09 ---------------------- -------------------- For e-10: Linear eigensolve converged (2 eigenpairs) due to CONVERGED_TOL; iterations 74 ---------------------- -------------------- k ||Ax-kx||/||kx|| ---------------------- -------------------- 0.002486 9.5257e-11 0.004466 2.12622e-11 ---------------------- -------------------- But my partition is more qualified when tol=e-8, not e-10. My last question is: Is this because of the iteration number that eigensolver tries to find suitable eigenvalues up to this tolerance? Again, thank you so much! Jose E. Roman , 10 Tem 2020 Cum, 19:36 tarihinde ?unu yazd?: > > For such small matrix, the solver is essentially solving with a direct method via LAPACK. If you understand how projection methods work, iteration gets in action when the matrix size is larger than the subspace size (ncv). The tolerance is relevant when the solver iterates, not when it computes the solution with LAPACK. > > Jose > > > > El 10 jul 2020, a las 18:32, Eda Oktay escribi?: > > > > No, it is of size 9 > > > > Jose E. Roman , 10 Tem 2020 Cum, 19:31 tarihinde ?unu yazd?: > >> > >> [Please respond to the list.] > >> > >> Is your matrix of size 8? This would explain the residuals. > >> > >> Jose > >> > >>> El 10 jul 2020, a las 17:10, Eda Oktay escribi?: > >>> > >>> I computed residual norm via -eps_error_relative::ascii_info_detail > >>> for different tolerance numbers (e-4, e-6, e-8, e-10). In each > >>> tolerance, I got the same table below: > >>> > >>> ---------------------- -------------------- > >>> k ||Ax-kx||/||kx|| > >>> ---------------------- -------------------- > >>> 3.000000 6.25528e-16 > >>> 3.000000 7.13774e-16 > >>> 3.438447 2.64362e-16 > >>> 5.000000 4.39333e-16 > >>> 6.000000 1.63943e-16 > >>> 6.000000 2.93737e-16 > >>> 6.000000 3.95997e-16 > >>> 7.561553 3.48664e-16 > >>> ---------------------- -------------------- > >>> > >>> I understood that since relative error is E-16 and this table shows > >>> eigenvalues whose relative error are below the tolerance, I am getting > >>> the same table but I still couldn't understand although relative > >>> errors are so small, how am I getting the most qualified partition in > >>> e-4 tolerance, not e-8. I am not computing zero eigenvalue I believe, > >>> since I am using EPSSetWhichEigenpairs(eps,EPS_SMALLEST_MAGNITUDE); > >>> and I am not getting zero eigenvalue. > >>> > >>> Thanks so much for answering! > >>> > >>> Jose E. Roman , 10 Tem 2020 Cum, 14:09 tarihinde ?unu yazd?: > >>>> > >>>> > >>>> > >>>>> El 10 jul 2020, a las 12:54, Eda Oktay escribi?: > >>>>> > >>>>>> How do you measure accuracy? > >>>>> Using the word accuracy may be not true actually, I am sorry. I am > >>>>> using eigenvectors corresponding to these eigenvalues in k-means > >>>>> algorithm, then do spectral graph partitioning. I must look at the > >>>>> partition quality. By quality, I mean, the resulting edge cut of my > >>>>> partitioned graph. I thought that the less tolerance results in more > >>>>> accuracy, hence more qualified partition. > >>>>> > >>>>>> What do you mean "the result was still the same"? > >>>>> I mean I am still not getting the most qualified solution in E-10, > >>>>> still E-2 or E-6 gives more qualified partitions, i.e. they give less > >>>>> edge cut. > >>>>> > >>>>>> What is the eigenvalue you are computing? > >>>>> I am computing the smallest eigenvalue of a Laplacian matrix. > >>>> > >>>> You should compute the residual norm, for instance with -eps_error_relative ::ascii_info_detail (see section 2.5.4 of the manual). > >>>> The relative residual error should be in the order of the tolerance (or smaller) if using the default convergence test, but if you are computing a zero eigenvalue then you may want to use an absolute convergence criterion (see table 2.6 or the manual). A graph Laplacian has at least one zero eigenvalue, unless you deflate it as explained in section 2.6.2 of the manual, see also ex11.c https://slepc.upv.es/documentation/current/src/eps/tutorials/ex11.c.html > >>>> > >>>> Jose > >>>> > >> > From jroman at dsic.upv.es Fri Jul 10 11:52:49 2020 From: jroman at dsic.upv.es (Jose E. Roman) Date: Fri, 10 Jul 2020 18:52:49 +0200 Subject: [petsc-users] SLEPc EPS Tolerance In-Reply-To: References: <5FADECB0-7D5F-4249-B686-E67C3DBD9F16@dsic.upv.es> <4ED7557C-6548-4E35-880A-4B481A70CE29@dsic.upv.es> <111E5C52-89F2-4480-AF7D-5628F51BBA05@dsic.upv.es> Message-ID: <9CE01E91-1190-4339-A554-7B4DCDD00A60@dsic.upv.es> I don't know. This question belongs to the application part, not the solver part. > El 10 jul 2020, a las 18:45, Eda Oktay escribi?: > > I looked at a2965*2965 sized matrix. I want to ask one last question. > > For e-8: > > Linear eigensolve converged (2 eigenpairs) due to CONVERGED_TOL; iterations 64 > ---------------------- -------------------- > k ||Ax-kx||/||kx|| > ---------------------- -------------------- > 0.002486 8.60766e-09 > 0.004466 1.68813e-09 > ---------------------- -------------------- > > For e-10: > Linear eigensolve converged (2 eigenpairs) due to CONVERGED_TOL; iterations 74 > ---------------------- -------------------- > k ||Ax-kx||/||kx|| > ---------------------- -------------------- > 0.002486 9.5257e-11 > 0.004466 2.12622e-11 > ---------------------- -------------------- > > But my partition is more qualified when tol=e-8, not e-10. > > My last question is: Is this because of the iteration number that > eigensolver tries to find suitable eigenvalues up to this tolerance? > > Again, thank you so much! > > Jose E. Roman , 10 Tem 2020 Cum, 19:36 tarihinde ?unu yazd?: >> >> For such small matrix, the solver is essentially solving with a direct method via LAPACK. If you understand how projection methods work, iteration gets in action when the matrix size is larger than the subspace size (ncv). The tolerance is relevant when the solver iterates, not when it computes the solution with LAPACK. >> >> Jose >> >> >>> El 10 jul 2020, a las 18:32, Eda Oktay escribi?: >>> >>> No, it is of size 9 >>> >>> Jose E. Roman , 10 Tem 2020 Cum, 19:31 tarihinde ?unu yazd?: >>>> >>>> [Please respond to the list.] >>>> >>>> Is your matrix of size 8? This would explain the residuals. >>>> >>>> Jose >>>> >>>>> El 10 jul 2020, a las 17:10, Eda Oktay escribi?: >>>>> >>>>> I computed residual norm via -eps_error_relative::ascii_info_detail >>>>> for different tolerance numbers (e-4, e-6, e-8, e-10). In each >>>>> tolerance, I got the same table below: >>>>> >>>>> ---------------------- -------------------- >>>>> k ||Ax-kx||/||kx|| >>>>> ---------------------- -------------------- >>>>> 3.000000 6.25528e-16 >>>>> 3.000000 7.13774e-16 >>>>> 3.438447 2.64362e-16 >>>>> 5.000000 4.39333e-16 >>>>> 6.000000 1.63943e-16 >>>>> 6.000000 2.93737e-16 >>>>> 6.000000 3.95997e-16 >>>>> 7.561553 3.48664e-16 >>>>> ---------------------- -------------------- >>>>> >>>>> I understood that since relative error is E-16 and this table shows >>>>> eigenvalues whose relative error are below the tolerance, I am getting >>>>> the same table but I still couldn't understand although relative >>>>> errors are so small, how am I getting the most qualified partition in >>>>> e-4 tolerance, not e-8. I am not computing zero eigenvalue I believe, >>>>> since I am using EPSSetWhichEigenpairs(eps,EPS_SMALLEST_MAGNITUDE); >>>>> and I am not getting zero eigenvalue. >>>>> >>>>> Thanks so much for answering! >>>>> >>>>> Jose E. Roman , 10 Tem 2020 Cum, 14:09 tarihinde ?unu yazd?: >>>>>> >>>>>> >>>>>> >>>>>>> El 10 jul 2020, a las 12:54, Eda Oktay escribi?: >>>>>>> >>>>>>>> How do you measure accuracy? >>>>>>> Using the word accuracy may be not true actually, I am sorry. I am >>>>>>> using eigenvectors corresponding to these eigenvalues in k-means >>>>>>> algorithm, then do spectral graph partitioning. I must look at the >>>>>>> partition quality. By quality, I mean, the resulting edge cut of my >>>>>>> partitioned graph. I thought that the less tolerance results in more >>>>>>> accuracy, hence more qualified partition. >>>>>>> >>>>>>>> What do you mean "the result was still the same"? >>>>>>> I mean I am still not getting the most qualified solution in E-10, >>>>>>> still E-2 or E-6 gives more qualified partitions, i.e. they give less >>>>>>> edge cut. >>>>>>> >>>>>>>> What is the eigenvalue you are computing? >>>>>>> I am computing the smallest eigenvalue of a Laplacian matrix. >>>>>> >>>>>> You should compute the residual norm, for instance with -eps_error_relative ::ascii_info_detail (see section 2.5.4 of the manual). >>>>>> The relative residual error should be in the order of the tolerance (or smaller) if using the default convergence test, but if you are computing a zero eigenvalue then you may want to use an absolute convergence criterion (see table 2.6 or the manual). A graph Laplacian has at least one zero eigenvalue, unless you deflate it as explained in section 2.6.2 of the manual, see also ex11.c https://slepc.upv.es/documentation/current/src/eps/tutorials/ex11.c.html >>>>>> >>>>>> Jose >>>>>> >>>> >> From eda.oktay at metu.edu.tr Fri Jul 10 12:01:24 2020 From: eda.oktay at metu.edu.tr (Eda Oktay) Date: Fri, 10 Jul 2020 20:01:24 +0300 Subject: [petsc-users] SLEPc EPS Tolerance In-Reply-To: <9CE01E91-1190-4339-A554-7B4DCDD00A60@dsic.upv.es> References: <5FADECB0-7D5F-4249-B686-E67C3DBD9F16@dsic.upv.es> <4ED7557C-6548-4E35-880A-4B481A70CE29@dsic.upv.es> <111E5C52-89F2-4480-AF7D-5628F51BBA05@dsic.upv.es> <9CE01E91-1190-4339-A554-7B4DCDD00A60@dsic.upv.es> Message-ID: Okay thanks again! On Fri, Jul 10, 2020, 7:52 PM Jose E. Roman wrote: > I don't know. This question belongs to the application part, not the > solver part. > > > El 10 jul 2020, a las 18:45, Eda Oktay escribi?: > > > > I looked at a2965*2965 sized matrix. I want to ask one last question. > > > > For e-8: > > > > Linear eigensolve converged (2 eigenpairs) due to CONVERGED_TOL; > iterations 64 > > ---------------------- -------------------- > > k ||Ax-kx||/||kx|| > > ---------------------- -------------------- > > 0.002486 8.60766e-09 > > 0.004466 1.68813e-09 > > ---------------------- -------------------- > > > > For e-10: > > Linear eigensolve converged (2 eigenpairs) due to CONVERGED_TOL; > iterations 74 > > ---------------------- -------------------- > > k ||Ax-kx||/||kx|| > > ---------------------- -------------------- > > 0.002486 9.5257e-11 > > 0.004466 2.12622e-11 > > ---------------------- -------------------- > > > > But my partition is more qualified when tol=e-8, not e-10. > > > > My last question is: Is this because of the iteration number that > > eigensolver tries to find suitable eigenvalues up to this tolerance? > > > > Again, thank you so much! > > > > Jose E. Roman , 10 Tem 2020 Cum, 19:36 tarihinde > ?unu yazd?: > >> > >> For such small matrix, the solver is essentially solving with a direct > method via LAPACK. If you understand how projection methods work, iteration > gets in action when the matrix size is larger than the subspace size (ncv). > The tolerance is relevant when the solver iterates, not when it computes > the solution with LAPACK. > >> > >> Jose > >> > >> > >>> El 10 jul 2020, a las 18:32, Eda Oktay > escribi?: > >>> > >>> No, it is of size 9 > >>> > >>> Jose E. Roman , 10 Tem 2020 Cum, 19:31 tarihinde > ?unu yazd?: > >>>> > >>>> [Please respond to the list.] > >>>> > >>>> Is your matrix of size 8? This would explain the residuals. > >>>> > >>>> Jose > >>>> > >>>>> El 10 jul 2020, a las 17:10, Eda Oktay > escribi?: > >>>>> > >>>>> I computed residual norm via -eps_error_relative::ascii_info_detail > >>>>> for different tolerance numbers (e-4, e-6, e-8, e-10). In each > >>>>> tolerance, I got the same table below: > >>>>> > >>>>> ---------------------- -------------------- > >>>>> k ||Ax-kx||/||kx|| > >>>>> ---------------------- -------------------- > >>>>> 3.000000 6.25528e-16 > >>>>> 3.000000 7.13774e-16 > >>>>> 3.438447 2.64362e-16 > >>>>> 5.000000 4.39333e-16 > >>>>> 6.000000 1.63943e-16 > >>>>> 6.000000 2.93737e-16 > >>>>> 6.000000 3.95997e-16 > >>>>> 7.561553 3.48664e-16 > >>>>> ---------------------- -------------------- > >>>>> > >>>>> I understood that since relative error is E-16 and this table shows > >>>>> eigenvalues whose relative error are below the tolerance, I am > getting > >>>>> the same table but I still couldn't understand although relative > >>>>> errors are so small, how am I getting the most qualified partition in > >>>>> e-4 tolerance, not e-8. I am not computing zero eigenvalue I believe, > >>>>> since I am using EPSSetWhichEigenpairs(eps,EPS_SMALLEST_MAGNITUDE); > >>>>> and I am not getting zero eigenvalue. > >>>>> > >>>>> Thanks so much for answering! > >>>>> > >>>>> Jose E. Roman , 10 Tem 2020 Cum, 14:09 > tarihinde ?unu yazd?: > >>>>>> > >>>>>> > >>>>>> > >>>>>>> El 10 jul 2020, a las 12:54, Eda Oktay > escribi?: > >>>>>>> > >>>>>>>> How do you measure accuracy? > >>>>>>> Using the word accuracy may be not true actually, I am sorry. I am > >>>>>>> using eigenvectors corresponding to these eigenvalues in k-means > >>>>>>> algorithm, then do spectral graph partitioning. I must look at the > >>>>>>> partition quality. By quality, I mean, the resulting edge cut of my > >>>>>>> partitioned graph. I thought that the less tolerance results in > more > >>>>>>> accuracy, hence more qualified partition. > >>>>>>> > >>>>>>>> What do you mean "the result was still the same"? > >>>>>>> I mean I am still not getting the most qualified solution in E-10, > >>>>>>> still E-2 or E-6 gives more qualified partitions, i.e. they give > less > >>>>>>> edge cut. > >>>>>>> > >>>>>>>> What is the eigenvalue you are computing? > >>>>>>> I am computing the smallest eigenvalue of a Laplacian matrix. > >>>>>> > >>>>>> You should compute the residual norm, for instance with > -eps_error_relative ::ascii_info_detail (see section 2.5.4 of the manual). > >>>>>> The relative residual error should be in the order of the tolerance > (or smaller) if using the default convergence test, but if you are > computing a zero eigenvalue then you may want to use an absolute > convergence criterion (see table 2.6 or the manual). A graph Laplacian has > at least one zero eigenvalue, unless you deflate it as explained in section > 2.6.2 of the manual, see also ex11.c > https://slepc.upv.es/documentation/current/src/eps/tutorials/ex11.c.html > >>>>>> > >>>>>> Jose > >>>>>> > >>>> > >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mukkundsunjii at gmail.com Fri Jul 10 12:57:13 2020 From: mukkundsunjii at gmail.com (MUKKUND SUNJII) Date: Fri, 10 Jul 2020 19:57:13 +0200 Subject: [petsc-users] Regarding P4est-Petsc Mapping References: Message-ID: Greetings, This is with regards to the exchange (see below) I had pertaining to P4est and DMPlex. The problem only persists with DM of type P4est. As suggested by Mark Adams, the possible source of the problem (in my case) could be in the mapping from P4est to PETSc. In an effort to understand more, I went digging and I found this file: https://gitlab.com/petsc/petsc/-/blob/master/src/dm/impls/forest/p4est/dmp4est.c#L4 In line 4, the mapping for the faces from p4est to PETSc are defined. Any explanation of the contents of the array P4estFaceToPetscFace[] can help me out a lot. Thank you in advance. Regards, Mukkund For Context: I am trying to expand the shallow water equation Riemann Solver provided in ./ts/tutorials/ex11.c to support bathymetry. I have successfully implemented a well-balanced Riemann Solver that works fine with PLEX DM but the behaviour changes when I use P4est DM (for adaptive mesh refinement). > Begin forwarded message: > > From: MUKKUND SUNJII > Subject: Re: [petsc-users] Regarding P4est > Date: 17 June 2020 at 21:20:29 CEST > To: Mark Adams > Cc: petsc-users , Domenico Lahaye > > Yes, precisely! I am not sure how I can replicate using the original version of ex11.c because it does not support bathymetry. > > Regardless, to demonstrate the discrepancy, I have uploaded three plots. The scenario is a lake at rest. Essentially, you have a varying bathymetry but a level water surface. If the model is well balanced, then the water surface height must not change. The description of the files are below > > 1) Bathymetry.png : It shows you the bathymetry profile (z(x)) and the water surface height (H = h+z(x)) at t = 0. > > > 2) Plex.png : This is the water surface height after 1 time step (0.007055 sec) and the dm type is Plex. As you can see, the water surface height is undisturbed as expected. > > > 3) P4est.png : This is the result after 1 time step (same final time) if I set the dm type as p4est. The noise is in the order of 1e-3 to be a little more specific. Since its not specifically at the boundaries and more or less spread throughout, it could indeed be noise introduced. But of course I could be wrong. > > > Maybe this paints a better picture. > > Regards, > > Mukkund > > For your reference, the Riemann Solver is a modified version of the HLL solver: A simple well-balanced and positive numerical scheme for the shallow-water system by Emmanuel Audusse, Christophe Chalons, Philippe Ung. > (https://www.intlpress.com/site/pub/files/_fulltext/journals/cms/2015/0013/0005/CMS-2015-0013-0005-a011.pdf ) > >> On 17 Jun 2020, at 20:47, Mark Adams > wrote: >> >> So you get this noise with a regular grid in p4est. So the same grid as will Plex, and you are not getting the same results. >> >> I don't know of any difference from p4est on a non-adapted grid. Can you reproduce this with ex11? >> >> Matt and Toby could answer this better. >> >> >> On Wed, Jun 17, 2020 at 1:33 PM MUKKUND SUNJII > wrote: >> Greetings, >> >> I am a master?s student working on the shallow water model of the TS example 'ex11.c' as part of my thesis. Therefore, I am working with DMForest for the implementation of adaptive grids. I have a question and an observation. >> >> I am trying to find relevant information about interpolation that takes place through the routine DMForestTransferVec. Perhaps it could be my inability to find it, but I am unable to locate the implementation of the routine >> >> (forest->transfervec)(dmIn,vecIn,dmOut,vecOut,useBCs,time). >> >> Any information on this particular routine is highly appreciated. >> >> Furthermore, I have developed a well balanced Riemann Solver that includes topography in the model. In the process of testing both the non-adaptive and adaptive version, I found that my results differed when I changed the type of DM. For instance, when I run a scenario in a fixed, non-adaptive grid with a DM of type 'P4est', I find that the well balanced nature is lost due to small perturbations all across the domain. However, this does not occur when I use a DM of type ?plex?. Is there a radical change in the routines between the two DM?s? This is not as much of a question as it is an observation. >> >> Thank you for all of your suggestions! >> >> Regards, >> >> Mukkund >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Bathymetry.png Type: image/png Size: 79963 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Plex.png Type: image/png Size: 68539 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: p4est.png Type: image/png Size: 150800 bytes Desc: not available URL: From knepley at gmail.com Fri Jul 10 13:11:51 2020 From: knepley at gmail.com (Matthew Knepley) Date: Fri, 10 Jul 2020 14:11:51 -0400 Subject: [petsc-users] Regarding P4est-Petsc Mapping In-Reply-To: References: Message-ID: On Fri, Jul 10, 2020 at 1:58 PM MUKKUND SUNJII wrote: > Greetings, > > This is with regards to the exchange (see below) I had pertaining to P4est > and DMPlex. The problem only persists with DM of type P4est. > > As suggested by Mark Adams, the possible source of the problem (in my > case) could be in the mapping from P4est to PETSc. In an effort to > understand more, I went digging and I found this file: > > https://gitlab.com/petsc/petsc/-/blob/master/src/dm/impls/forest/p4est/dmp4est.c#L4 > > In line 4, the mapping for the faces from p4est to PETSc are defined. Any > explanation of the contents of the array P4estFaceToPetscFace[] can help > me out a lot. > I believe this is about canonical numbering. DMPlex always wants outward normals, and so orients its quadrilateral in a loop: ---2--- | | 3 1 | | ---0--- and we have PetscFaceToP4estFace[4] = {2, 1, 3, 0} which would renumber it ---3--- | | 0 1 | | ---2--- The other one should be the inverse permutation, P4estFaceToPetscFace[4] = {3, 1, 0, 2} which it is. Thanks, Matt Thank you in advance. > > Regards, > > Mukkund > > For Context: I am trying to expand the shallow water equation Riemann > Solver provided in ./ts/tutorials/ex11.c to support bathymetry. I have > successfully implemented a well-balanced Riemann Solver that works fine > with PLEX DM but the behaviour changes when I use P4est DM (for adaptive > mesh refinement). > > > Begin forwarded message: > > *From: *MUKKUND SUNJII > *Subject: **Re: [petsc-users] Regarding P4est* > *Date: *17 June 2020 at 21:20:29 CEST > *To: *Mark Adams > *Cc: *petsc-users , Domenico Lahaye < > D.J.P.Lahaye at tudelft.nl> > > Yes, precisely! I am not sure how I can replicate using the original > version of ex11.c because it does not support bathymetry. > > Regardless, to demonstrate the discrepancy, I have uploaded three plots. > The scenario is a lake at rest. Essentially, you have a varying bathymetry > but a level water surface. If the model is well balanced, then the water > surface height must not change. The description of the files are below > > 1) Bathymetry.png : It shows you the bathymetry profile (z(x)) and the > water surface height (H = h+z(x)) at t = 0. > > 2) Plex.png : This is the water surface height after 1 time step (0.007055 > sec) and the dm type is Plex. As you can see, the water surface height > is undisturbed as expected. > > 3) P4est.png : This is the result after 1 time step (same final time) if I > set the dm type as p4est. The noise is in the order of 1e-3 to be a little > more specific. Since its not specifically at the boundaries and more or > less spread throughout, it could indeed be noise introduced. But of course > I could be wrong. > > Maybe this paints a better picture. > > Regards, > > Mukkund > > For your reference, the Riemann Solver is a modified version of the HLL > solver: *A simple well-balanced and positive numerical scheme for the > shallow-water system by **Emmanuel Audusse, Christophe Chalons, Philippe > Ung. * > ( > https://www.intlpress.com/site/pub/files/_fulltext/journals/cms/2015/0013/0005/CMS-2015-0013-0005-a011.pdf > ) > > On 17 Jun 2020, at 20:47, Mark Adams wrote: > > So you get this noise with a regular grid in p4est. So the same grid as > will Plex, and you are not getting the same results. > > I don't know of any difference from p4est on a non-adapted grid. Can you > reproduce this with ex11? > > Matt and Toby could answer this better. > > > On Wed, Jun 17, 2020 at 1:33 PM MUKKUND SUNJII > wrote: > Greetings, > > I am a master?s student working on the shallow water model of the TS > example 'ex11.c' as part of my thesis. Therefore, I am working with > DMForest for the implementation of adaptive grids. I have a question and an > observation. > > I am trying to find relevant information about interpolation that takes > place through the routine DMForestTransferVec. Perhaps it could be my > inability to find it, but I am unable to locate the implementation of the > routine > > (forest->transfervec)(dmIn,vecIn,dmOut,vecOut,useBCs,time). > > Any information on this particular routine is highly appreciated. > > Furthermore, I have developed a well balanced Riemann Solver that includes > topography in the model. In the process of testing both the non-adaptive > and adaptive version, I found that my results differed when I changed the > type of DM. For instance, when I run a scenario in a fixed, non-adaptive > grid with a DM of type 'P4est', I find that the well balanced nature is > lost due to small perturbations all across the domain. However, this does > not occur when I use a DM of type ?plex?. Is there a radical change in the > routines between the two DM?s? This is not as much of a question as it is > an observation. > > Thank you for all of your suggestions! > > Regards, > > Mukkund > > > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Bathymetry.png Type: image/png Size: 79963 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Plex.png Type: image/png Size: 68539 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: p4est.png Type: image/png Size: 150800 bytes Desc: not available URL: From dalcinl at gmail.com Mon Jul 13 01:50:44 2020 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Mon, 13 Jul 2020 09:50:44 +0300 Subject: [petsc-users] HDF5 and ParaView In-Reply-To: References: <3d04f25d-7de7-4d44-79c1-ed736daf649d@epcc.ed.ac.uk> Message-ID: On Fri, 10 Jul 2020 at 15:37, Matthew Knepley wrote: > > That is true. Do they also get rid of the single mesh and single timstep > requirements? HDF5+XDMF makes it much easier since > we can put multiple meshes and timesteps in one file. > Well, I use ParaView's *.pvd files for that, and I dump each timestep to its own *.vtu file in a folder to pack the files. But you still have a point, the format is indeed restricting. Why do you consider it so important to put multiple meshes and timesteps in one file? It is just that you hate to have so many files scattered around? Or something deeper? I do hate the fact that the VTK formats (either legacy or XML) do not allow you to dump a single mesh to be reused for multiple timestep. I would probably move out of VTK files in favor of something else if I had a way to encode VTK's (the library, not the file format) high-order Lagrange elements. Actually, I'm toying with dumping files with PETSc's raw binary I/O with MPI, and writing a proper ParaView plugin in Python to read the data. -- Lisandro Dalcin ============ Research Scientist Extreme Computing Research Center (ECRC) King Abdullah University of Science and Technology (KAUST) http://ecrc.kaust.edu.sa/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Mon Jul 13 04:46:54 2020 From: knepley at gmail.com (Matthew Knepley) Date: Mon, 13 Jul 2020 05:46:54 -0400 Subject: [petsc-users] HDF5 and ParaView In-Reply-To: References: <3d04f25d-7de7-4d44-79c1-ed736daf649d@epcc.ed.ac.uk> Message-ID: On Mon, Jul 13, 2020 at 2:51 AM Lisandro Dalcin wrote: > On Fri, 10 Jul 2020 at 15:37, Matthew Knepley wrote: > >> >> That is true. Do they also get rid of the single mesh and single timstep >> requirements? HDF5+XDMF makes it much easier since >> we can put multiple meshes and timesteps in one file. >> > > Well, I use ParaView's *.pvd files for that, and I dump each timestep to > its own *.vtu file in a folder to pack the files. But you still have a > point, the format is indeed restricting. > Why do you consider it so important to put multiple meshes and timesteps > in one file? It is just that you hate to have so many files scattered > around? Or something deeper? > I do hate the fact that the VTK formats (either legacy or XML) do not > allow you to dump a single mesh to be reused for multiple timestep. > For now its complexity of moving simulation data around and scripting for it. However, now I have at least two meshes in my problem, and I anticipate having several more. I believe this will be the long term trend. > I would probably move out of VTK files in favor of something else if I had > a way to encode VTK's (the library, not the file format) high-order > Lagrange elements. > Actually, I'm toying with dumping files with PETSc's raw binary I/O with > MPI, and writing a proper ParaView plugin in Python to read the data. > I have again discussed higher order with the Firedrake people. They are using the Paraview mechanism, but it is so fragile and baroque that I refuse. Currently, the cleanest way is still to refine the mesh and project to P_1. There have been so many attempts at high order in VTK, I can only conclude that Kitware does not give a crap about it. Thanks, Matt > -- > Lisandro Dalcin > ============ > Research Scientist > Extreme Computing Research Center (ECRC) > King Abdullah University of Science and Technology (KAUST) > http://ecrc.kaust.edu.sa/ > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From wence at gmx.li Mon Jul 13 05:22:25 2020 From: wence at gmx.li (Lawrence Mitchell) Date: Mon, 13 Jul 2020 11:22:25 +0100 Subject: [petsc-users] HDF5 and ParaView In-Reply-To: References: <3d04f25d-7de7-4d44-79c1-ed736daf649d@epcc.ed.ac.uk> Message-ID: <36ABC255-A3C4-4A02-B442-48F4A79556F3@gmx.li> > On 13 Jul 2020, at 10:46, Matthew Knepley wrote: > > For now its complexity of moving simulation data around and scripting for it. However, now I have at least two meshes in my problem, and > I anticipate having several more. I believe this will be the long term trend. > > I would probably move out of VTK files in favor of something else if I had a way to encode VTK's (the library, not the file format) high-order Lagrange elements. > Actually, I'm toying with dumping files with PETSc's raw binary I/O with MPI, and writing a proper ParaView plugin in Python to read the data. > > I have again discussed higher order with the Firedrake people. They are using the Paraview mechanism, but it is so fragile and baroque that I > refuse. Currently, the cleanest way is still to refine the mesh and project to P_1. There have been so many attempts at high order in VTK, I can > only conclude that Kitware does not give a crap about it. FWIW, they went with an approach to defining a basis, it just disagrees with what you want. Lisandro, in terms of using XDMF rather than VTK files for high order Lagrange. There appears to be work going on at the moment to add support in the VTK XDMF-reader for these elements: https://gitlab.kitware.com/vtk/vtk/-/merge_requests/7068 You still have to map your basis to theirs, of course. Lawrence From mfadams at lbl.gov Mon Jul 13 06:14:43 2020 From: mfadams at lbl.gov (Mark Adams) Date: Mon, 13 Jul 2020 07:14:43 -0400 Subject: [petsc-users] HDF5 and ParaView In-Reply-To: References: <3d04f25d-7de7-4d44-79c1-ed736daf649d@epcc.ed.ac.uk> Message-ID: > > > > I would probably move out of VTK files in favor of something else if I had > a way to encode VTK's (the library, not the file format) high-order > Lagrange elements. > Actually, I'm toying with dumping files with PETSc's raw binary I/O with > MPI, and writing a proper ParaView plugin in Python to read the data. > I'd love a high order viewer too. -------------- next part -------------- An HTML attachment was scrubbed... URL: From dalcinl at gmail.com Mon Jul 13 07:56:56 2020 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Mon, 13 Jul 2020 15:56:56 +0300 Subject: [petsc-users] HDF5 and ParaView In-Reply-To: References: <3d04f25d-7de7-4d44-79c1-ed736daf649d@epcc.ed.ac.uk> Message-ID: On Mon, 13 Jul 2020 at 14:14, Mark Adams wrote: > >> >> I would probably move out of VTK files in favor of something else if I >> had a way to encode VTK's (the library, not the file format) high-order >> Lagrange elements. >> Actually, I'm toying with dumping files with PETSc's raw binary I/O with >> MPI, and writing a proper ParaView plugin in Python to read the data. >> > > I'd love a high order viewer too. > Well, ParaView can actually do it. You need to open a vtu file with high-order Lagrange elements, and then look for "Nonlinear subdivision level" in the GUI and bump it up. The main issue is that the algorithm is not "adaptive" and you may need to bump too much, then the thing becomes quite slow. Another one that does things very well is GLVis, although not with all the features of the ParaView/VisIt+VTK combo. Stefano put a lot of effort integrating GLVIs into PETSc, but maybe the integration with PetscFE and DMPlex may require some extra work. -- Lisandro Dalcin ============ Research Scientist Extreme Computing Research Center (ECRC) King Abdullah University of Science and Technology (KAUST) http://ecrc.kaust.edu.sa/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From mukkundsunjii at gmail.com Tue Jul 14 04:46:21 2020 From: mukkundsunjii at gmail.com (MUKKUND SUNJII) Date: Tue, 14 Jul 2020 11:46:21 +0200 Subject: [petsc-users] Regarding Mapping Message-ID: <3AE47462-E7DB-4F49-95F4-AF5CFE96D3C0@gmail.com> Greetings, This message is again in relation to the previous series of exchanges I had with the PETSc development team and professors. To not be repetitive I will provide a concise description for those unaware. I have built a well-balanced shallow water solver based on ts/tutorials/ex11.c. However, the Riemann Solver that I have written includes the topography terms. To be well-balanced, I have made a change in the Riemann Solver interface in fv.c: Before the change it computes only one flux vector (*flux[]). But now with my modification, it computes 2 flux terms (*fluxL[] and *fluxR[]) and it is assigned to the left and right side of the interface respectively. The exact description of the problem itself is mentioned in the thread below. I might have accidentally found the solution while trying to figure out the origin of the problem. I noticed that the solver is no longer well-balanced (when running the adaptive mesh refinement case) when I use -dm_refine to prescribe the refinement level of the initial grid. I understand now, that the ?noise' all across the domain is indeed caused by the wrong mapping of the left and right fluxes at the interface by my modification in fv.c. However, interestingly, the solver becomes well-balanced (i.e., the mapping is correct again) when I use -dm_forest_initial_refinement instead. I have tested various cases with water completely at rest to verify the well-balancedness of the solver while using DM of type p4est. I thought this observation might be interesting to you as I see some of the AMR test cases in ex11.c use -dm_refine and some others use -dm_forest_initial_refinement. My knowledge on P4est and DMPlex is lacking to explain the discrepancy between the two. Perhaps, someone else can shine light on this matter. Thank you very much in advance. Regards, Mukkund > Begin forwarded message: > > From: MUKKUND SUNJII > > Subject: Re: [petsc-users] Regarding P4est > Date: 17 June 2020 at 21:20:29 CEST > To: Mark Adams > > Cc: petsc-users >, Domenico Lahaye > > > Yes, precisely! I am not sure how I can replicate using the original version of ex11.c because it does not support bathymetry. > > Regardless, to demonstrate the discrepancy, I have uploaded three plots. The scenario is a lake at rest. Essentially, you have a varying bathymetry but a level water surface. If the model is well balanced, then the water surface height must not change. The description of the files are below > > 1) Bathymetry.png : It shows you the bathymetry profile (z(x)) and the water surface height (H = h+z(x)) at t = 0. > > > 2) Plex.png : This is the water surface height after 1 time step (0.007055 sec) and the dm type is Plex. As you can see, the water surface height is undisturbed as expected. > > > 3) P4est.png : This is the result after 1 time step (same final time) if I set the dm type as p4est. The noise is in the order of 1e-3 to be a little more specific. Since its not specifically at the boundaries and more or less spread throughout, it could indeed be noise introduced. But of course I could be wrong. > > > Maybe this paints a better picture. > > Regards, > > Mukkund > > For your reference, the Riemann Solver is a modified version of the HLL solver: A simple well-balanced and positive numerical scheme for the shallow-water system by Emmanuel Audusse, Christophe Chalons, Philippe Ung. > (https://www.intlpress.com/site/pub/files/_fulltext/journals/cms/2015/0013/0005/CMS-2015-0013-0005-a011.pdf ) > >> On 17 Jun 2020, at 20:47, Mark Adams > wrote: >> >> So you get this noise with a regular grid in p4est. So the same grid as will Plex, and you are not getting the same results. >> >> I don't know of any difference from p4est on a non-adapted grid. Can you reproduce this with ex11? >> >> Matt and Toby could answer this better. >> >> >> On Wed, Jun 17, 2020 at 1:33 PM MUKKUND SUNJII > wrote: >> Greetings, >> >> I am a master?s student working on the shallow water model of the TS example 'ex11.c' as part of my thesis. Therefore, I am working with DMForest for the implementation of adaptive grids. I have a question and an observation. >> >> I am trying to find relevant information about interpolation that takes place through the routine DMForestTransferVec. Perhaps it could be my inability to find it, but I am unable to locate the implementation of the routine >> >> (forest->transfervec)(dmIn,vecIn,dmOut,vecOut,useBCs,time). >> >> Any information on this particular routine is highly appreciated. >> >> Furthermore, I have developed a well balanced Riemann Solver that includes topography in the model. In the process of testing both the non-adaptive and adaptive version, I found that my results differed when I changed the type of DM. For instance, when I run a scenario in a fixed, non-adaptive grid with a DM of type 'P4est', I find that the well balanced nature is lost due to small perturbations all across the domain. However, this does not occur when I use a DM of type ?plex?. Is there a radical change in the routines between the two DM?s? This is not as much of a question as it is an observation. >> >> Thank you for all of your suggestions! >> >> Regards, >> >> Mukkund -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Bathymetry.png Type: image/png Size: 79963 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Plex.png Type: image/png Size: 68539 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: p4est.png Type: image/png Size: 150800 bytes Desc: not available URL: From knepley at gmail.com Tue Jul 14 05:19:02 2020 From: knepley at gmail.com (Matthew Knepley) Date: Tue, 14 Jul 2020 06:19:02 -0400 Subject: [petsc-users] Regarding Mapping In-Reply-To: <3AE47462-E7DB-4F49-95F4-AF5CFE96D3C0@gmail.com> References: <3AE47462-E7DB-4F49-95F4-AF5CFE96D3C0@gmail.com> Message-ID: On Tue, Jul 14, 2020 at 5:47 AM MUKKUND SUNJII wrote: > Greetings, > > This message is again in relation to the previous series of exchanges I > had with the PETSc development team and professors. > > To not be repetitive I will provide a concise description for those > unaware. I have built a well-balanced shallow water solver based on > ts/tutorials/ex11.c. However, the Riemann Solver that I have written > includes the topography terms. To be well-balanced, I have made a change in > the Riemann Solver interface in fv.c: Before the change it computes only > one flux vector (*flux[]). But now with my modification, it computes 2 flux > terms (*fluxL[] and *fluxR[]) and it is assigned to the left and right > side of the interface respectively. The exact description of the problem > itself is mentioned in the thread below. > > I might have accidentally found the solution while trying to figure out > the origin of the problem. > > I noticed that the solver is no longer well-balanced (when running the > adaptive mesh refinement case) when I use -dm_refine to prescribe the > refinement level of the initial grid. I understand now, that the ?noise' > all across the domain is indeed caused by the wrong mapping of the left and > right fluxes at the interface by my modification in fv.c. > > However, interestingly, the solver becomes well-balanced (i.e., the > mapping is correct again) when I use -dm_forest_initial_refinement instead. > I have tested various cases with water completely at rest to verify the > well-balancedness of the solver while using DM of type p4est. > > I thought this observation might be interesting to you as I see some of > the AMR test cases in ex11.c use -dm_refine and some others use > -dm_forest_initial_refinement. My knowledge on P4est and DMPlex is lacking > to explain the discrepancy between the two. Perhaps, someone else can shine > light on this matter. > > Thank you very much in advance. > That is a very clear description. That will really help us figure out what is happening. For people like me who do not really understand FV methods, can you briefly explain why we would have 2 fluxes? The simple way I think about things, in FV you consider the state in 2 adjacent cells and then determine a flux between them, which can be positive or negative, but it is the same flux when viewed from either side. Thanks, Matt > Regards, > > Mukkund > > > Begin forwarded message: > > *From: *MUKKUND SUNJII > *Subject: **Re: [petsc-users] Regarding P4est* > *Date: *17 June 2020 at 21:20:29 CEST > *To: *Mark Adams > *Cc: *petsc-users , Domenico Lahaye < > D.J.P.Lahaye at tudelft.nl> > > Yes, precisely! I am not sure how I can replicate using the original > version of ex11.c because it does not support bathymetry. > > Regardless, to demonstrate the discrepancy, I have uploaded three plots. > The scenario is a lake at rest. Essentially, you have a varying bathymetry > but a level water surface. If the model is well balanced, then the water > surface height must not change. The description of the files are below > > 1) Bathymetry.png : It shows you the bathymetry profile (z(x)) and the > water surface height (H = h+z(x)) at t = 0. > > 2) Plex.png : This is the water surface height after 1 time step (0.007055 > sec) and the dm type is Plex. As you can see, the water surface height > is undisturbed as expected. > > 3) P4est.png : This is the result after 1 time step (same final time) if I > set the dm type as p4est. The noise is in the order of 1e-3 to be a little > more specific. Since its not specifically at the boundaries and more or > less spread throughout, it could indeed be noise introduced. But of course > I could be wrong. > > Maybe this paints a better picture. > > Regards, > > Mukkund > > For your reference, the Riemann Solver is a modified version of the HLL > solver: *A simple well-balanced and positive numerical scheme for the > shallow-water system by **Emmanuel Audusse, Christophe Chalons, Philippe > Ung. * > ( > https://www.intlpress.com/site/pub/files/_fulltext/journals/cms/2015/0013/0005/CMS-2015-0013-0005-a011.pdf > ) > > On 17 Jun 2020, at 20:47, Mark Adams wrote: > > So you get this noise with a regular grid in p4est. So the same grid as > will Plex, and you are not getting the same results. > > I don't know of any difference from p4est on a non-adapted grid. Can you > reproduce this with ex11? > > Matt and Toby could answer this better. > > > On Wed, Jun 17, 2020 at 1:33 PM MUKKUND SUNJII > wrote: > Greetings, > > I am a master?s student working on the shallow water model of the TS > example 'ex11.c' as part of my thesis. Therefore, I am working with > DMForest for the implementation of adaptive grids. I have a question and an > observation. > > I am trying to find relevant information about interpolation that takes > place through the routine DMForestTransferVec. Perhaps it could be my > inability to find it, but I am unable to locate the implementation of the > routine > > (forest->transfervec)(dmIn,vecIn,dmOut,vecOut,useBCs,time). > > Any information on this particular routine is highly appreciated. > > Furthermore, I have developed a well balanced Riemann Solver that includes > topography in the model. In the process of testing both the non-adaptive > and adaptive version, I found that my results differed when I changed the > type of DM. For instance, when I run a scenario in a fixed, non-adaptive > grid with a DM of type 'P4est', I find that the well balanced nature is > lost due to small perturbations all across the domain. However, this does > not occur when I use a DM of type ?plex?. Is there a radical change in the > routines between the two DM?s? This is not as much of a question as it is > an observation. > > Thank you for all of your suggestions! > > Regards, > > Mukkund > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Bathymetry.png Type: image/png Size: 79963 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Plex.png Type: image/png Size: 68539 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: p4est.png Type: image/png Size: 150800 bytes Desc: not available URL: From fdkong.jd at gmail.com Tue Jul 14 10:16:11 2020 From: fdkong.jd at gmail.com (Fande Kong) Date: Tue, 14 Jul 2020 09:16:11 -0600 Subject: [petsc-users] VecAssemblyEnd_MPI_BTS Message-ID: Hi All, I was doing a large-scale simulation using 12288 cores and had the following error. The code ran fine using less than 12288 cores. Any quick suggestions to track down this issue? Thanks, Fande, [3342]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [3342]PETSC ERROR: Petsc has generated inconsistent data [3342]PETSC ERROR: Received vector entry 0 out of local range [344829312,344964096)] [3342]PETSC ERROR: See https://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [3342]PETSC ERROR: Petsc Release Version 3.13.3, unknown [3342]PETSC ERROR: /home/kongf/workhome/sawtooth/griffin/griffin-opt on a arch-moose named r1i4n34 by kongf Tue Jul 14 08:44:02 2020 [3342]PETSC ERROR: Configure options --download-hypre=1 --with-debugging=no --with-shared-libraries=1 --download-fblaslapack=1 --download-metis=1 --download-ptscotch=1 --download-parmetis=1 --download-superlu_dist=1 --download-mumps=1 --download-scalapack=1 --download-slepc=1 --with-mpi=1 --with-cxx-dialect=C++11 --with-fortran-bindings=0 --with-sowing=0 --with-64-bit-indices --download-mumps=0 [3342]PETSC ERROR: #1 VecAssemblyEnd_MPI_BTS() line 324 in /home/kongf/workhome/sawtooth/moosers/petsc/src/vec/vec/impls/mpi/pbvec.c [3342]PETSC ERROR: #2 VecAssemblyEnd() line 171 in /home/kongf/workhome/sawtooth/moosers/petsc/src/vec/vec/interface/vector.c [cli_3342]: aborting job: application called MPI_Abort(MPI_COMM_WORLD, 1) - process 3342 -------------- next part -------------- An HTML attachment was scrubbed... URL: From fdkong.jd at gmail.com Tue Jul 14 11:29:15 2020 From: fdkong.jd at gmail.com (Fande Kong) Date: Tue, 14 Jul 2020 10:29:15 -0600 Subject: [petsc-users] VecAssemblyEnd_MPI_BTS In-Reply-To: References: Message-ID: The petsc configuration log was attached. Thanks, Fande, On Tue, Jul 14, 2020 at 9:16 AM Fande Kong wrote: > Hi All, > > > I was doing a large-scale simulation using 12288 cores and had the > following error. The code ran fine using less than 12288 cores. > > Any quick suggestions to track down this issue? > > Thanks, > > Fande, > > > [3342]PETSC ERROR: --------------------- Error Message > -------------------------------------------------------------- > [3342]PETSC ERROR: Petsc has generated inconsistent data > [3342]PETSC ERROR: Received vector entry 0 out of local range > [344829312,344964096)] > [3342]PETSC ERROR: See > https://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. > [3342]PETSC ERROR: Petsc Release Version 3.13.3, unknown > [3342]PETSC ERROR: /home/kongf/workhome/sawtooth/griffin/griffin-opt on a > arch-moose named r1i4n34 by kongf Tue Jul 14 08:44:02 2020 > [3342]PETSC ERROR: Configure options --download-hypre=1 > --with-debugging=no --with-shared-libraries=1 --download-fblaslapack=1 > --download-metis=1 --download-ptscotch=1 --download-parmetis=1 > --download-superlu_dist=1 --download-mumps=1 --download-scalapack=1 > --download-slepc=1 --with-mpi=1 --with-cxx-dialect=C++11 > --with-fortran-bindings=0 --with-sowing=0 --with-64-bit-indices > --download-mumps=0 > [3342]PETSC ERROR: #1 VecAssemblyEnd_MPI_BTS() line 324 in > /home/kongf/workhome/sawtooth/moosers/petsc/src/vec/vec/impls/mpi/pbvec.c > [3342]PETSC ERROR: #2 VecAssemblyEnd() line 171 in > /home/kongf/workhome/sawtooth/moosers/petsc/src/vec/vec/interface/vector.c > [cli_3342]: aborting job: > application called MPI_Abort(MPI_COMM_WORLD, 1) - process 3342 > -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: configure.log.zip Type: application/zip Size: 317017 bytes Desc: not available URL: From mukkundsunjii at gmail.com Tue Jul 14 14:30:24 2020 From: mukkundsunjii at gmail.com (MUKKUND SUNJII) Date: Tue, 14 Jul 2020 21:30:24 +0200 Subject: [petsc-users] Regarding Mapping In-Reply-To: References: <3AE47462-E7DB-4F49-95F4-AF5CFE96D3C0@gmail.com> Message-ID: Greetings, Thank you for your reply. You are right in your intuition about the Finite Volume Method. In the case of no bathymetry, this is not required as the fluxes stay the same on both the sides of the interface. However, with the bathymetry terms, the bed slope terms are also accounted into the Riemann Solver. However, in order to keep the model well-balanced, the left and right flux differ by the wave speeds as indicated by the equations in the diagram (see attachment). Perhaps, this is more of a localised requirement for my problem but other models might feature the same formulation. Regards, Mukkund > On 14 Jul 2020, at 12:19, Matthew Knepley wrote: > > On Tue, Jul 14, 2020 at 5:47 AM MUKKUND SUNJII > wrote: > Greetings, > > This message is again in relation to the previous series of exchanges I had with the PETSc development team and professors. > > To not be repetitive I will provide a concise description for those unaware. I have built a well-balanced shallow water solver based on ts/tutorials/ex11.c. However, the Riemann Solver that I have written includes the topography terms. To be well-balanced, I have made a change in the Riemann Solver interface in fv.c: Before the change it computes only one flux vector (*flux[]). But now with my modification, it computes 2 flux terms (*fluxL[] and *fluxR[]) and it is assigned to the left and right side of the interface respectively. The exact description of the problem itself is mentioned in the thread below. > > I might have accidentally found the solution while trying to figure out the origin of the problem. > > I noticed that the solver is no longer well-balanced (when running the adaptive mesh refinement case) when I use -dm_refine to prescribe the refinement level of the initial grid. I understand now, that the ?noise' all across the domain is indeed caused by the wrong mapping of the left and right fluxes at the interface by my modification in fv.c. > > However, interestingly, the solver becomes well-balanced (i.e., the mapping is correct again) when I use -dm_forest_initial_refinement instead. I have tested various cases with water completely at rest to verify the well-balancedness of the solver while using DM of type p4est. > > I thought this observation might be interesting to you as I see some of the AMR test cases in ex11.c use -dm_refine and some others use -dm_forest_initial_refinement. My knowledge on P4est and DMPlex is lacking to explain the discrepancy between the two. Perhaps, someone else can shine light on this matter. > > Thank you very much in advance. > > That is a very clear description. That will really help us figure out what is happening. > > For people like me who do not really understand FV methods, can you briefly explain why we would have 2 fluxes? The simple way I think > about things, in FV you consider the state in 2 adjacent cells and then determine a flux between them, which can be positive or negative, but > it is the same flux when viewed from either side. > > Thanks, > > Matt > > Regards, > > Mukkund > > >> Begin forwarded message: >> >> From: MUKKUND SUNJII > >> Subject: Re: [petsc-users] Regarding P4est >> Date: 17 June 2020 at 21:20:29 CEST >> To: Mark Adams > >> Cc: petsc-users >, Domenico Lahaye > >> >> Yes, precisely! I am not sure how I can replicate using the original version of ex11.c because it does not support bathymetry. >> >> Regardless, to demonstrate the discrepancy, I have uploaded three plots. The scenario is a lake at rest. Essentially, you have a varying bathymetry but a level water surface. If the model is well balanced, then the water surface height must not change. The description of the files are below >> >> 1) Bathymetry.png : It shows you the bathymetry profile (z(x)) and the water surface height (H = h+z(x)) at t = 0. >> >> >> 2) Plex.png : This is the water surface height after 1 time step (0.007055 sec) and the dm type is Plex. As you can see, the water surface height is undisturbed as expected. >> >> >> 3) P4est.png : This is the result after 1 time step (same final time) if I set the dm type as p4est. The noise is in the order of 1e-3 to be a little more specific. Since its not specifically at the boundaries and more or less spread throughout, it could indeed be noise introduced. But of course I could be wrong. >> >> >> Maybe this paints a better picture. >> >> Regards, >> >> Mukkund >> >> For your reference, the Riemann Solver is a modified version of the HLL solver: A simple well-balanced and positive numerical scheme for the shallow-water system by Emmanuel Audusse, Christophe Chalons, Philippe Ung. >> (https://www.intlpress.com/site/pub/files/_fulltext/journals/cms/2015/0013/0005/CMS-2015-0013-0005-a011.pdf ) >> >>> On 17 Jun 2020, at 20:47, Mark Adams > wrote: >>> >>> So you get this noise with a regular grid in p4est. So the same grid as will Plex, and you are not getting the same results. >>> >>> I don't know of any difference from p4est on a non-adapted grid. Can you reproduce this with ex11? >>> >>> Matt and Toby could answer this better. >>> >>> >>> On Wed, Jun 17, 2020 at 1:33 PM MUKKUND SUNJII > wrote: >>> Greetings, >>> >>> I am a master?s student working on the shallow water model of the TS example 'ex11.c' as part of my thesis. Therefore, I am working with DMForest for the implementation of adaptive grids. I have a question and an observation. >>> >>> I am trying to find relevant information about interpolation that takes place through the routine DMForestTransferVec. Perhaps it could be my inability to find it, but I am unable to locate the implementation of the routine >>> >>> (forest->transfervec)(dmIn,vecIn,dmOut,vecOut,useBCs,time). >>> >>> Any information on this particular routine is highly appreciated. >>> >>> Furthermore, I have developed a well balanced Riemann Solver that includes topography in the model. In the process of testing both the non-adaptive and adaptive version, I found that my results differed when I changed the type of DM. For instance, when I run a scenario in a fixed, non-adaptive grid with a DM of type 'P4est', I find that the well balanced nature is lost due to small perturbations all across the domain. However, this does not occur when I use a DM of type ?plex?. Is there a radical change in the routines between the two DM?s? This is not as much of a question as it is an observation. >>> >>> Thank you for all of your suggestions! >>> >>> Regards, >>> >>> Mukkund > > > -- > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Screenshot 2020-07-14 at 20.55.11.png Type: image/png Size: 128664 bytes Desc: not available URL: From jed at jedbrown.org Tue Jul 14 15:41:46 2020 From: jed at jedbrown.org (Jed Brown) Date: Tue, 14 Jul 2020 14:41:46 -0600 Subject: [petsc-users] VecAssemblyEnd_MPI_BTS In-Reply-To: References: Message-ID: <87h7u9u8vp.fsf@jedbrown.org> Is it possible to run this with a different MPI? Zero is suspicious in that perhaps the message wasn't delivered, rather than that it was delivered with incorrect data. This sounds like a tough one if it isn't reproducible (preferably at smaller scale). Fande Kong writes: > The petsc configuration log was attached. > > Thanks, > > Fande, > > > > On Tue, Jul 14, 2020 at 9:16 AM Fande Kong wrote: > >> Hi All, >> >> >> I was doing a large-scale simulation using 12288 cores and had the >> following error. The code ran fine using less than 12288 cores. >> >> Any quick suggestions to track down this issue? >> >> Thanks, >> >> Fande, >> >> >> [3342]PETSC ERROR: --------------------- Error Message >> -------------------------------------------------------------- >> [3342]PETSC ERROR: Petsc has generated inconsistent data >> [3342]PETSC ERROR: Received vector entry 0 out of local range >> [344829312,344964096)] >> [3342]PETSC ERROR: See >> https://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. >> [3342]PETSC ERROR: Petsc Release Version 3.13.3, unknown >> [3342]PETSC ERROR: /home/kongf/workhome/sawtooth/griffin/griffin-opt on a >> arch-moose named r1i4n34 by kongf Tue Jul 14 08:44:02 2020 >> [3342]PETSC ERROR: Configure options --download-hypre=1 >> --with-debugging=no --with-shared-libraries=1 --download-fblaslapack=1 >> --download-metis=1 --download-ptscotch=1 --download-parmetis=1 >> --download-superlu_dist=1 --download-mumps=1 --download-scalapack=1 >> --download-slepc=1 --with-mpi=1 --with-cxx-dialect=C++11 >> --with-fortran-bindings=0 --with-sowing=0 --with-64-bit-indices >> --download-mumps=0 >> [3342]PETSC ERROR: #1 VecAssemblyEnd_MPI_BTS() line 324 in >> /home/kongf/workhome/sawtooth/moosers/petsc/src/vec/vec/impls/mpi/pbvec.c >> [3342]PETSC ERROR: #2 VecAssemblyEnd() line 171 in >> /home/kongf/workhome/sawtooth/moosers/petsc/src/vec/vec/interface/vector.c >> [cli_3342]: aborting job: >> application called MPI_Abort(MPI_COMM_WORLD, 1) - process 3342 From bsmith at petsc.dev Tue Jul 14 16:09:48 2020 From: bsmith at petsc.dev (Barry Smith) Date: Tue, 14 Jul 2020 16:09:48 -0500 Subject: [petsc-users] VecAssemblyEnd_MPI_BTS In-Reply-To: References: Message-ID: <07873D46-8B49-42BD-8906-73AA8E8CB2F2@petsc.dev> valgrind? > On Jul 14, 2020, at 11:29 AM, Fande Kong wrote: > > The petsc configuration log was attached. > > Thanks, > > Fande, > > > > On Tue, Jul 14, 2020 at 9:16 AM Fande Kong > wrote: > Hi All, > > > I was doing a large-scale simulation using 12288 cores and had the following error. The code ran fine using less than 12288 cores. > > Any quick suggestions to track down this issue? > > Thanks, > > Fande, > > > [3342]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- > [3342]PETSC ERROR: Petsc has generated inconsistent data > [3342]PETSC ERROR: Received vector entry 0 out of local range [344829312,344964096)] > [3342]PETSC ERROR: See https://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. > [3342]PETSC ERROR: Petsc Release Version 3.13.3, unknown > [3342]PETSC ERROR: /home/kongf/workhome/sawtooth/griffin/griffin-opt on a arch-moose named r1i4n34 by kongf Tue Jul 14 08:44:02 2020 > [3342]PETSC ERROR: Configure options --download-hypre=1 --with-debugging=no --with-shared-libraries=1 --download-fblaslapack=1 --download-metis=1 --download-ptscotch=1 --download-parmetis=1 --download-superlu_dist=1 --download-mumps=1 --download-scalapack=1 --download-slepc=1 --with-mpi=1 --with-cxx-dialect=C++11 --with-fortran-bindings=0 --with-sowing=0 --with-64-bit-indices --download-mumps=0 > [3342]PETSC ERROR: #1 VecAssemblyEnd_MPI_BTS() line 324 in /home/kongf/workhome/sawtooth/moosers/petsc/src/vec/vec/impls/mpi/pbvec.c > [3342]PETSC ERROR: #2 VecAssemblyEnd() line 171 in /home/kongf/workhome/sawtooth/moosers/petsc/src/vec/vec/interface/vector.c > [cli_3342]: aborting job: > application called MPI_Abort(MPI_COMM_WORLD, 1) - process 3342 > -------------- next part -------------- An HTML attachment was scrubbed... URL: From junchao.zhang at gmail.com Tue Jul 14 14:14:48 2020 From: junchao.zhang at gmail.com (Junchao Zhang) Date: Tue, 14 Jul 2020 14:14:48 -0500 Subject: [petsc-users] VecAssemblyEnd_MPI_BTS In-Reply-To: References: Message-ID: I have no idea. You can try -build_twosided allreduce. If not working, try to write an example. --Junchao Zhang On Tue, Jul 14, 2020 at 11:30 AM Fande Kong wrote: > The petsc configuration log was attached. > > Thanks, > > Fande, > > > > On Tue, Jul 14, 2020 at 9:16 AM Fande Kong wrote: > >> Hi All, >> >> >> I was doing a large-scale simulation using 12288 cores and had the >> following error. The code ran fine using less than 12288 cores. >> >> Any quick suggestions to track down this issue? >> >> Thanks, >> >> Fande, >> >> >> [3342]PETSC ERROR: --------------------- Error Message >> -------------------------------------------------------------- >> [3342]PETSC ERROR: Petsc has generated inconsistent data >> [3342]PETSC ERROR: Received vector entry 0 out of local range >> [344829312,344964096)] >> [3342]PETSC ERROR: See >> https://www.mcs.anl.gov/petsc/documentation/faq.html for trouble >> shooting. >> [3342]PETSC ERROR: Petsc Release Version 3.13.3, unknown >> [3342]PETSC ERROR: /home/kongf/workhome/sawtooth/griffin/griffin-opt on a >> arch-moose named r1i4n34 by kongf Tue Jul 14 08:44:02 2020 >> [3342]PETSC ERROR: Configure options --download-hypre=1 >> --with-debugging=no --with-shared-libraries=1 --download-fblaslapack=1 >> --download-metis=1 --download-ptscotch=1 --download-parmetis=1 >> --download-superlu_dist=1 --download-mumps=1 --download-scalapack=1 >> --download-slepc=1 --with-mpi=1 --with-cxx-dialect=C++11 >> --with-fortran-bindings=0 --with-sowing=0 --with-64-bit-indices >> --download-mumps=0 >> [3342]PETSC ERROR: #1 VecAssemblyEnd_MPI_BTS() line 324 in >> /home/kongf/workhome/sawtooth/moosers/petsc/src/vec/vec/impls/mpi/pbvec.c >> [3342]PETSC ERROR: #2 VecAssemblyEnd() line 171 in >> /home/kongf/workhome/sawtooth/moosers/petsc/src/vec/vec/interface/vector.c >> [cli_3342]: aborting job: >> application called MPI_Abort(MPI_COMM_WORLD, 1) - process 3342 >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mfadams at lbl.gov Tue Jul 14 23:01:33 2020 From: mfadams at lbl.gov (Mark Adams) Date: Wed, 15 Jul 2020 00:01:33 -0400 Subject: [petsc-users] configure error Message-ID: This workflow worked, but I now get this error. KNL + Intel at NERSC. It's a METIS problem. cmake has been an issue in the past. nid02470 maint= ~/petsc_install/petsc$ cmake --version cmake version 3.14.4 -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: configure.log Type: application/octet-stream Size: 1117534 bytes Desc: not available URL: From balay at mcs.anl.gov Tue Jul 14 23:28:30 2020 From: balay at mcs.anl.gov (Satish Balay) Date: Tue, 14 Jul 2020 23:28:30 -0500 (CDT) Subject: [petsc-users] configure error In-Reply-To: References: Message-ID: Configure Options: --configModules=PETSc.Configure --optionsModule=config.compilerOptions --with-cc=cc --with-cxx=CC --with-fc=ftn COPTFLAGS=" -g -O2" CXXOPTFLAGS="-g -O2" FOPTFLAGS=" -g -O2" --download-fblaslapack=1 --download-hypre=0 --download-metis=1 --download-parmetis=1 --with-debugging=0 --with-mpiexec=srun --with-batch=1 --known-mpi-shared-libraries=1 --with-x=0 --with-hwloc=0 --with-64-bit-indices=1 PETSC_ARCH=arch-cori-knl-opt-intel --with-openmp=0 --download-p4est=0 --prefix=/global/common/software/m1041/petsc_install/petsc_knl_intel PETSC_DIR=/global/homes/m/madams/petsc_install/petsc =============================================================================== You do not have write permissions to the --prefix directory /global/common/software/m1041/petsc_install/petsc_knl_intel You will be prompted for the sudo password for any external package installs =============================================================================== Satish On Wed, 15 Jul 2020, Mark Adams wrote: > This workflow worked, but I now get this error. KNL + Intel at NERSC. > > It's a METIS problem. cmake has been an issue in the past. > > nid02470 maint= ~/petsc_install/petsc$ cmake --version > cmake version 3.14.4 > From mfadams at lbl.gov Wed Jul 15 11:14:21 2020 From: mfadams at lbl.gov (Mark Adams) Date: Wed, 15 Jul 2020 12:14:21 -0400 Subject: [petsc-users] configure error In-Reply-To: References: Message-ID: Thanks, Inexplicably, it says the project software directories are read-only on KNL (compute) nodes on Cori, which does not make sense to me, but maybe it's just my permissions. I can build on the login nodes with-batch=0 now. I guess that got fixed (executing a program) or maybe Hypre was installed already. On Wed, Jul 15, 2020 at 12:28 AM Satish Balay wrote: > Configure Options: --configModules=PETSc.Configure > --optionsModule=config.compilerOptions --with-cc=cc --with-cxx=CC > --with-fc=ftn COPTFLAGS=" -g -O2" CXXOPTFLAGS="-g -O2" FOPTFLAGS=" -g > -O2" --download-fblaslapack=1 --download-hypre=0 --download-metis=1 > --download-parmetis=1 --with-debugging=0 --with-mpiexec=srun --with-batch=1 > --known-mpi-shared-libraries=1 --with-x=0 --with-hwloc=0 > --with-64-bit-indices=1 PETSC_ARCH=arch-cori-knl-opt-intel --with-openmp=0 > --download-p4est=0 > --prefix=/global/common/software/m1041/petsc_install/petsc_knl_intel > PETSC_DIR=/global/homes/m/madams/petsc_install/petsc > > > > =============================================================================== > You do not have write permissions to the > --prefix directory > /global/common/software/m1041/petsc_install/petsc_knl_intel > You will be prompted for the sudo password > for any external package installs > > =============================================================================== > > > Satish > > > On Wed, 15 Jul 2020, Mark Adams wrote: > > > This workflow worked, but I now get this error. KNL + Intel at NERSC. > > > > It's a METIS problem. cmake has been an issue in the past. > > > > nid02470 maint= ~/petsc_install/petsc$ cmake --version > > cmake version 3.14.4 > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From y.juntao at hotmail.com Wed Jul 15 13:05:39 2020 From: y.juntao at hotmail.com (Karl Yang) Date: Thu, 16 Jul 2020 02:05:39 +0800 Subject: [petsc-users] Errors encountered with KSPsolver with MATNEST and VECNEST Message-ID: Hello, I'm trying to solve a 2*2 block matrix system. I managed to form a 2*2 MatNest and a 21 VecNest. However when I try to make use of ksp solve, I encounter the following error. There is only one or two examples on MatNest and VecNest. I could not figure out what could be the trouble. [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [0]PETSC ERROR: Invalid argument [0]PETSC ERROR: Nest vector argument 2 not setup. [0]PETSC ERROR: See https://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [0]PETSC ERROR: Petsc Release Version 3.12.5, Mar, 29, 2020 [0]PETSC ERROR: ./testSolve on a arch-linux2-c-debug named f601294ac804 by Unknown Thu Jul 16 01:56:11 2020 [0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --download-mpich --download-fblaslapack --with-cuda [0]PETSC ERROR: #1 VecCopy_Nest() line 68 in /usr/local/petsc/petsc-3.12.5/src/vec/vec/impls/nest/vecnest.c [0]PETSC ERROR: #2 VecCopy() line 1577 in /usr/local/petsc/petsc-3.12.5/src/vec/vec/interface/vector.c [0]PETSC ERROR: #3 KSPInitialResidual() line 61 in /usr/local/petsc/petsc-3.12.5/src/ksp/ksp/interface/itres.c [0]PETSC ERROR: #4 KSPSolve_GMRES() line 236 in /usr/local/petsc/petsc-3.12.5/src/ksp/ksp/impls/gmres/gmres.c [0]PETSC ERROR: #5 KSPSolve() line 760 in /usr/local/petsc/petsc-3.12.5/src/ksp/ksp/interface/itfunc.c Regards JT -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at petsc.dev Wed Jul 15 13:13:29 2020 From: bsmith at petsc.dev (Barry Smith) Date: Wed, 15 Jul 2020 13:13:29 -0500 Subject: [petsc-users] Errors encountered with KSPsolver with MATNEST and VECNEST In-Reply-To: References: Message-ID: <03ABA31B-426E-4610-B028-A79D743734B2@petsc.dev> From the error message the second vector you are using to form the nest vector has not been completely created/set up. Make sure you called either VecCreateSeq(), VecCreateMPI() or did VecCreate(), VecSetType(), VecSetUp() and if it is the right hand side make sure you put numerical values in it. Barry > On Jul 15, 2020, at 1:05 PM, Karl Yang wrote: > > Hello, > > I'm trying to solve a 2*2 block matrix system. > I managed to form a 2*2 MatNest and a 21 VecNest. However when I try to make use of ksp solve, I encounter the following error. There is only one or two examples on MatNest and VecNest. I could not figure out what could be the trouble. > > > [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- > [0]PETSC ERROR: Invalid argument > [0]PETSC ERROR: Nest vector argument 2 not setup. > [0]PETSC ERROR: See https://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. > [0]PETSC ERROR: Petsc Release Version 3.12.5, Mar, 29, 2020 > [0]PETSC ERROR: ./testSolve on a arch-linux2-c-debug named f601294ac804 by Unknown Thu Jul 16 01:56:11 2020 > [0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --download-mpich --download-fblaslapack --with-cuda > [0]PETSC ERROR: #1 VecCopy_Nest() line 68 in /usr/local/petsc/petsc-3.12.5/src/vec/vec/impls/nest/vecnest.c > [0]PETSC ERROR: #2 VecCopy() line 1577 in /usr/local/petsc/petsc-3.12.5/src/vec/vec/interface/vector.c > [0]PETSC ERROR: #3 KSPInitialResidual() line 61 in /usr/local/petsc/petsc-3.12.5/src/ksp/ksp/interface/itres.c > [0]PETSC ERROR: #4 KSPSolve_GMRES() line 236 in /usr/local/petsc/petsc-3.12.5/src/ksp/ksp/impls/gmres/gmres.c > [0]PETSC ERROR: #5 KSPSolve() line 760 in /usr/local/petsc/petsc-3.12.5/src/ksp/ksp/interface/itfunc.c > > > Regards > JT > -------------- next part -------------- An HTML attachment was scrubbed... URL: From y.juntao at hotmail.com Wed Jul 15 13:28:35 2020 From: y.juntao at hotmail.com (Karl Yang) Date: Thu, 16 Jul 2020 02:28:35 +0800 Subject: [petsc-users] Errors encountered with KSPsolver with MATNEST and VECNEST In-Reply-To: <03ABA31B-426E-4610-B028-A79D743734B2@petsc.dev> References: <03ABA31B-426E-4610-B028-A79D743734B2@petsc.dev> Message-ID: Hi, Barry, Thanks for your reply. I output the matrix and vec with matview and vecview. And I think I do have the values set as it has been shown. Mat Object: 1 MPI processes type: nest Matrix object: type=nest, rows=3, cols=3 MatNest structure: (0,0) : type=seqaij, rows=25, cols=25 (0,1) : NULL (0,2) : type=seqaij, rows=25, cols=25 (1,0) : NULL (1,1) : type=seqaij, rows=25, cols=25 (1,2) : type=seqaij, rows=25, cols=25 (2,0) : type=seqaij, rows=25, cols=25 (2,1) : type=seqaij, rows=25, cols=25 (2,2) : NULL Vec Object: 1 MPI processes type: nest VecNest, rows=3, structure: (0) : type=seq, rows=25 Vec Object: 1 MPI processes type: seq 0. 0. 0. 0. 0. 0. 75.8279 -51.0776 -75.8279 0. 0. -90.8404 58.7011 90.8404 0. 0. -75.8279 51.0776 75.8279 0. 0. 0. 0. 0. 0. (1) : name="Vec_0x84000000_0", type=seq, rows=25 Vec Object: Vec_0x84000000_0 1 MPI processes type: seq 0. 0. 0. 0. 0. 0. 75.8279 -51.0776 -75.8279 0. 0. -90.8404 58.7011 90.8404 0. 0. -75.8279 51.0776 75.8279 0. 0. 0. 0. 0. 0. (2) : type=seq, rows=25 Vec Object: 1 MPI processes type: seq 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [0]PETSC ERROR: Invalid argument [0]PETSC ERROR: Nest vector argument 4 not setup. [0]PETSC ERROR: See https://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [0]PETSC ERROR: Petsc Release Version 3.12.5, Mar, 29, 2020 [0]PETSC ERROR: ./testSolve on a arch-linux2-c-debug named f601294ac804 by Unknown Thu Jul 16 02:15:26 2020 [0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --download-mpich --download-fblaslapack --with-cuda [0]PETSC ERROR: #1 VecPointwiseMult_Nest() line 245 in /usr/local/petsc/petsc-3.12.5/src/vec/vec/impls/nest/vecnest.c [0]PETSC ERROR: #2 VecPointwiseMult() line 1107 in /usr/local/petsc/petsc-3.12.5/src/vec/vec/interface/vector.c [0]PETSC ERROR: #3 PCApply_Jacobi() line 272 in /usr/local/petsc/petsc-3.12.5/src/ksp/pc/impls/jacobi/jacobi.c [0]PETSC ERROR: #4 PCApply() line 444 in /usr/local/petsc/petsc-3.12.5/src/ksp/pc/interface/precon.c [0]PETSC ERROR: #5 KSP_PCApply() line 281 in /usr/local/petsc/petsc-3.12.5/include/petsc/private/kspimpl.h [0]PETSC ERROR: #6 KSPInitialResidual() line 65 in /usr/local/petsc/petsc-3.12.5/src/ksp/ksp/interface/itres.c [0]PETSC ERROR: #7 KSPSolve_GMRES() line 236 in /usr/local/petsc/petsc-3.12.5/src/ksp/ksp/impls/gmres/gmres.c [0]PETSC ERROR: #8 KSPSolve() line 760 in /usr/local/petsc/petsc-3.12.5/src/ksp/ksp/interface/itfunc.c On Jul 16 2020, at 2:13 am, Barry Smith wrote: > > From the error message the second vector you are using to form the nest vector has not been completely created/set up. Make sure you called either VecCreateSeq(), VecCreateMPI() or did VecCreate(), VecSetType(), VecSetUp() and if it is the right hand side make sure you put numerical values in it. > > Barry > > > > On Jul 15, 2020, at 1:05 PM, Karl Yang wrote: > > Hello, > > I'm trying to solve a 2*2 block matrix system. > > I managed to form a 2*2 MatNest and a 21 VecNest. However when I try to make use of ksp solve, I encounter the following error. There is only one or two examples on MatNest and VecNest. I could not figure out what could be the trouble. > > > > > > [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- > > [0]PETSC ERROR: Invalid argument > > [0]PETSC ERROR: Nest vector argument 2 not setup. > > [0]PETSC ERROR: See https://www.mcs.anl.gov/petsc/documentation/faq.html (https://link.getmailspring.com/link/540F752A-DB59-4878-9C98-B8F4B4D9FD0D at getmailspring.com/1?redirect=https%3A%2F%2Fwww.mcs.anl.gov%2Fpetsc%2Fdocumentation%2Ffaq.html&recipient=cGV0c2MtdXNlcnNAbWNzLmFubC5nb3Y%3D) for trouble shooting. > > [0]PETSC ERROR: Petsc Release Version 3.12.5, Mar, 29, 2020 > > [0]PETSC ERROR: ./testSolve on a arch-linux2-c-debug named f601294ac804 by Unknown Thu Jul 16 01:56:11 2020 > > [0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --download-mpich --download-fblaslapack --with-cuda > > [0]PETSC ERROR: #1 VecCopy_Nest() line 68 in /usr/local/petsc/petsc-3.12.5/src/vec/vec/impls/nest/vecnest.c > > [0]PETSC ERROR: #2 VecCopy() line 1577 in /usr/local/petsc/petsc-3.12.5/src/vec/vec/interface/vector.c > > [0]PETSC ERROR: #3 KSPInitialResidual() line 61 in /usr/local/petsc/petsc-3.12.5/src/ksp/ksp/interface/itres.c > > [0]PETSC ERROR: #4 KSPSolve_GMRES() line 236 in /usr/local/petsc/petsc-3.12.5/src/ksp/ksp/impls/gmres/gmres.c > > [0]PETSC ERROR: #5 KSPSolve() line 760 in /usr/local/petsc/petsc-3.12.5/src/ksp/ksp/interface/itfunc.c > > > > > > Regards > > JT > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at petsc.dev Wed Jul 15 13:39:30 2020 From: bsmith at petsc.dev (Barry Smith) Date: Wed, 15 Jul 2020 13:39:30 -0500 Subject: [petsc-users] Errors encountered with KSPsolver with MATNEST and VECNEST In-Reply-To: References: <03ABA31B-426E-4610-B028-A79D743734B2@petsc.dev> Message-ID: <5CC396D0-D707-4A29-917E-82DA69F49891@petsc.dev> Could be a bug in PETSc, please send a sample program that produces the problem. Barry > On Jul 15, 2020, at 1:28 PM, Karl Yang wrote: > > Hi, Barry, > > Thanks for your reply. I output the matrix and vec with matview and vecview. And I think I do have the values set as it has been shown. > > Mat Object: 1 MPI processes > type: nest > Matrix object: > type=nest, rows=3, cols=3 > MatNest structure: > (0,0) : type=seqaij, rows=25, cols=25 > (0,1) : NULL > (0,2) : type=seqaij, rows=25, cols=25 > (1,0) : NULL > (1,1) : type=seqaij, rows=25, cols=25 > (1,2) : type=seqaij, rows=25, cols=25 > (2,0) : type=seqaij, rows=25, cols=25 > (2,1) : type=seqaij, rows=25, cols=25 > (2,2) : NULL > Vec Object: 1 MPI processes > type: nest > VecNest, rows=3, structure: > (0) : type=seq, rows=25 > Vec Object: 1 MPI processes > type: seq > 0. > 0. > 0. > 0. > 0. > 0. > 75.8279 > -51.0776 > -75.8279 > 0. > 0. > -90.8404 > 58.7011 > 90.8404 > 0. > 0. > -75.8279 > 51.0776 > 75.8279 > 0. > 0. > 0. > 0. > 0. > 0. > (1) : name="Vec_0x84000000_0", type=seq, rows=25 > Vec Object: Vec_0x84000000_0 1 MPI processes > type: seq > 0. > 0. > 0. > 0. > 0. > 0. > 75.8279 > -51.0776 > -75.8279 > 0. > 0. > -90.8404 > 58.7011 > 90.8404 > 0. > 0. > -75.8279 > 51.0776 > 75.8279 > 0. > 0. > 0. > 0. > 0. > 0. > (2) : type=seq, rows=25 > Vec Object: 1 MPI processes > type: seq > 0. > 0. > 0. > 0. > 0. > 0. > 0. > 0. > 0. > 0. > 0. > 0. > 0. > 0. > 0. > 0. > 0. > 0. > 0. > 0. > 0. > 0. > 0. > 0. > 0. > [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- > [0]PETSC ERROR: Invalid argument > [0]PETSC ERROR: Nest vector argument 4 not setup. > [0]PETSC ERROR: See https://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. > [0]PETSC ERROR: Petsc Release Version 3.12.5, Mar, 29, 2020 > [0]PETSC ERROR: ./testSolve on a arch-linux2-c-debug named f601294ac804 by Unknown Thu Jul 16 02:15:26 2020 > [0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --download-mpich --download-fblaslapack --with-cuda > [0]PETSC ERROR: #1 VecPointwiseMult_Nest() line 245 in /usr/local/petsc/petsc-3.12.5/src/vec/vec/impls/nest/vecnest.c > [0]PETSC ERROR: #2 VecPointwiseMult() line 1107 in /usr/local/petsc/petsc-3.12.5/src/vec/vec/interface/vector.c > [0]PETSC ERROR: #3 PCApply_Jacobi() line 272 in /usr/local/petsc/petsc-3.12.5/src/ksp/pc/impls/jacobi/jacobi.c > [0]PETSC ERROR: #4 PCApply() line 444 in /usr/local/petsc/petsc-3.12.5/src/ksp/pc/interface/precon.c > [0]PETSC ERROR: #5 KSP_PCApply() line 281 in /usr/local/petsc/petsc-3.12.5/include/petsc/private/kspimpl.h > [0]PETSC ERROR: #6 KSPInitialResidual() line 65 in /usr/local/petsc/petsc-3.12.5/src/ksp/ksp/interface/itres.c > [0]PETSC ERROR: #7 KSPSolve_GMRES() line 236 in /usr/local/petsc/petsc-3.12.5/src/ksp/ksp/impls/gmres/gmres.c > [0]PETSC ERROR: #8 KSPSolve() line 760 in /usr/local/petsc/petsc-3.12.5/src/ksp/ksp/interface/itfunc.c > > On Jul 16 2020, at 2:13 am, Barry Smith wrote: > > From the error message the second vector you are using to form the nest vector has not been completely created/set up. Make sure you called either VecCreateSeq(), VecCreateMPI() or did VecCreate(), VecSetType(), VecSetUp() and if it is the right hand side make sure you put numerical values in it. > > Barry > > > On Jul 15, 2020, at 1:05 PM, Karl Yang > wrote: > > Hello, > > I'm trying to solve a 2*2 block matrix system. > I managed to form a 2*2 MatNest and a 21 VecNest. However when I try to make use of ksp solve, I encounter the following error. There is only one or two examples on MatNest and VecNest. I could not figure out what could be the trouble. > > > [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- > [0]PETSC ERROR: Invalid argument > [0]PETSC ERROR: Nest vector argument 2 not setup. > [0]PETSC ERROR: See https://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. > [0]PETSC ERROR: Petsc Release Version 3.12.5, Mar, 29, 2020 > [0]PETSC ERROR: ./testSolve on a arch-linux2-c-debug named f601294ac804 by Unknown Thu Jul 16 01:56:11 2020 > [0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --download-mpich --download-fblaslapack --with-cuda > [0]PETSC ERROR: #1 VecCopy_Nest() line 68 in /usr/local/petsc/petsc-3.12.5/src/vec/vec/impls/nest/vecnest.c > [0]PETSC ERROR: #2 VecCopy() line 1577 in /usr/local/petsc/petsc-3.12.5/src/vec/vec/interface/vector.c > [0]PETSC ERROR: #3 KSPInitialResidual() line 61 in /usr/local/petsc/petsc-3.12.5/src/ksp/ksp/interface/itres.c > [0]PETSC ERROR: #4 KSPSolve_GMRES() line 236 in /usr/local/petsc/petsc-3.12.5/src/ksp/ksp/impls/gmres/gmres.c > [0]PETSC ERROR: #5 KSPSolve() line 760 in /usr/local/petsc/petsc-3.12.5/src/ksp/ksp/interface/itfunc.c > > > Regards > JT > -------------- next part -------------- An HTML attachment was scrubbed... URL: From y.juntao at hotmail.com Wed Jul 15 14:16:00 2020 From: y.juntao at hotmail.com (Karl Yang) Date: Thu, 16 Jul 2020 03:16:00 +0800 Subject: [petsc-users] Errors encountered with KSPsolver with MATNEST and VECNEST In-Reply-To: <5CC396D0-D707-4A29-917E-82DA69F49891@petsc.dev> References: <5CC396D0-D707-4A29-917E-82DA69F49891@petsc.dev> Message-ID: Hi, Barry, I created a minimum example to reproduce the error. The example code is attached. Mat Object: 1 MPI processes type: nest Matrix object: type=nest, rows=3, cols=3 MatNest structure: (0,0) : type=seqaij, rows=5, cols=5 (0,1) : NULL (0,2) : type=seqaij, rows=5, cols=5 (1,0) : NULL (1,1) : type=seqaij, rows=5, cols=5 (1,2) : type=seqaij, rows=5, cols=5 (2,0) : type=seqaij, rows=5, cols=5 (2,1) : type=seqaij, rows=5, cols=5 (2,2) : NULL Vec Object: 1 MPI processes type: nest VecNest, rows=3, structure: (0) : type=seq, rows=5 Vec Object: 1 MPI processes type: seq 1. 1. 1. 1. 1. (1) : type=seq, rows=5 Vec Object: 1 MPI processes type: seq 1. 1. 1. 1. 1. (2) : type=seq, rows=5 Vec Object: 1 MPI processes type: seq 1. 1. 1. 1. 1. [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [0]PETSC ERROR: Invalid argument [0]PETSC ERROR: Nest vector argument 2 not setup. [0]PETSC ERROR: See https://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [0]PETSC ERROR: Petsc Release Version 3.12.5, Mar, 29, 2020 [0]PETSC ERROR: ./testSolve on a arch-linux2-c-debug named f601294ac804 by Unknown Thu Jul 16 03:11:02 2020 [0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --download-mpich --download-fblaslapack --with-cuda [0]PETSC ERROR: #1 VecCopy_Nest() line 68 in /usr/local/petsc/petsc-3.12.5/src/vec/vec/impls/nest/vecnest.c [0]PETSC ERROR: #2 VecCopy() line 1577 in /usr/local/petsc/petsc-3.12.5/src/vec/vec/interface/vector.c [0]PETSC ERROR: #3 KSPInitialResidual() line 61 in /usr/local/petsc/petsc-3.12.5/src/ksp/ksp/interface/itres.c [0]PETSC ERROR: #4 KSPSolve_GMRES() line 236 in /usr/local/petsc/petsc-3.12.5/src/ksp/ksp/impls/gmres/gmres.c [0]PETSC ERROR: #5 KSPSolve() line 760 in /usr/local/petsc/petsc-3.12.5/src/ksp/ksp/interface/itfunc.c On Jul 16 2020, at 2:39 am, Barry Smith wrote: > > Could be a bug in PETSc, please send a sample program that produces the problem. > > Barry > > > > On Jul 15, 2020, at 1:28 PM, Karl Yang wrote: > > Hi, Barry, > > Thanks for your reply. I output the matrix and vec with matview and vecview. And I think I do have the values set as it has been shown. > > Mat Object: 1 MPI processes > > type: nest > > Matrix object: > > type=nest, rows=3, cols=3 > > MatNest structure: > > (0,0) : type=seqaij, rows=25, cols=25 > > (0,1) : NULL > > (0,2) : type=seqaij, rows=25, cols=25 > > (1,0) : NULL > > (1,1) : type=seqaij, rows=25, cols=25 > > (1,2) : type=seqaij, rows=25, cols=25 > > (2,0) : type=seqaij, rows=25, cols=25 > > (2,1) : type=seqaij, rows=25, cols=25 > > (2,2) : NULL > > Vec Object: 1 MPI processes > > type: nest > > VecNest, rows=3, structure: > > (0) : type=seq, rows=25 > > Vec Object: 1 MPI processes > > type: seq > > 0. > > 0. > > 0. > > 0. > > 0. > > 0. > > 75.8279 > > -51.0776 > > -75.8279 > > 0. > > 0. > > -90.8404 > > 58.7011 > > 90.8404 > > 0. > > 0. > > -75.8279 > > 51.0776 > > 75.8279 > > 0. > > 0. > > 0. > > 0. > > 0. > > 0. > > (1) : name="Vec_0x84000000_0", type=seq, rows=25 > > Vec Object: Vec_0x84000000_0 1 MPI processes > > type: seq > > 0. > > 0. > > 0. > > 0. > > 0. > > 0. > > 75.8279 > > -51.0776 > > -75.8279 > > 0. > > 0. > > -90.8404 > > 58.7011 > > 90.8404 > > 0. > > 0. > > -75.8279 > > 51.0776 > > 75.8279 > > 0. > > 0. > > 0. > > 0. > > 0. > > 0. > > (2) : type=seq, rows=25 > > Vec Object: 1 MPI processes > > type: seq > > 0. > > 0. > > 0. > > 0. > > 0. > > 0. > > 0. > > 0. > > 0. > > 0. > > 0. > > 0. > > 0. > > 0. > > 0. > > 0. > > 0. > > 0. > > 0. > > 0. > > 0. > > 0. > > 0. > > 0. > > 0. > > [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- > > [0]PETSC ERROR: Invalid argument > > [0]PETSC ERROR: Nest vector argument 4 not setup. > > [0]PETSC ERROR: See https://www.mcs.anl.gov/petsc/documentation/faq.html (https://link.getmailspring.com/link/060AB2C3-F3D7-408C-99CF-C4D1B27DD59F at getmailspring.com/1?redirect=https%3A%2F%2Fwww.mcs.anl.gov%2Fpetsc%2Fdocumentation%2Ffaq.html&recipient=cGV0c2MtdXNlcnNAbWNzLmFubC5nb3Y%3D) for trouble shooting. > > [0]PETSC ERROR: Petsc Release Version 3.12.5, Mar, 29, 2020 > > [0]PETSC ERROR: ./testSolve on a arch-linux2-c-debug named f601294ac804 by Unknown Thu Jul 16 02:15:26 2020 > > [0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --download-mpich --download-fblaslapack --with-cuda > > [0]PETSC ERROR: #1 VecPointwiseMult_Nest() line 245 in /usr/local/petsc/petsc-3.12.5/src/vec/vec/impls/nest/vecnest.c > > [0]PETSC ERROR: #2 VecPointwiseMult() line 1107 in /usr/local/petsc/petsc-3.12.5/src/vec/vec/interface/vector.c > > [0]PETSC ERROR: #3 PCApply_Jacobi() line 272 in /usr/local/petsc/petsc-3.12.5/src/ksp/pc/impls/jacobi/jacobi.c > > [0]PETSC ERROR: #4 PCApply() line 444 in /usr/local/petsc/petsc-3.12.5/src/ksp/pc/interface/precon.c > > [0]PETSC ERROR: #5 KSP_PCApply() line 281 in /usr/local/petsc/petsc-3.12.5/include/petsc/private/kspimpl.h > > [0]PETSC ERROR: #6 KSPInitialResidual() line 65 in /usr/local/petsc/petsc-3.12.5/src/ksp/ksp/interface/itres.c > > [0]PETSC ERROR: #7 KSPSolve_GMRES() line 236 in /usr/local/petsc/petsc-3.12.5/src/ksp/ksp/impls/gmres/gmres.c > > [0]PETSC ERROR: #8 KSPSolve() line 760 in /usr/local/petsc/petsc-3.12.5/src/ksp/ksp/interface/itfunc.c > > > > On Jul 16 2020, at 2:13 am, Barry Smith wrote: > > > > > > From the error message the second vector you are using to form the nest vector has not been completely created/set up. Make sure you called either VecCreateSeq(), VecCreateMPI() or did VecCreate(), VecSetType(), VecSetUp() and if it is the right hand side make sure you put numerical values in it. > > > > > > Barry > > > > > > > > > > On Jul 15, 2020, at 1:05 PM, Karl Yang wrote: > > > > Hello, > > > > I'm trying to solve a 2*2 block matrix system. > > > > I managed to form a 2*2 MatNest and a 21 VecNest. However when I try to make use of ksp solve, I encounter the following error. There is only one or two examples on MatNest and VecNest. I could not figure out what could be the trouble. > > > > > > > > > > > > [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- > > > > [0]PETSC ERROR: Invalid argument > > > > [0]PETSC ERROR: Nest vector argument 2 not setup. > > > > [0]PETSC ERROR: See https://www.mcs.anl.gov/petsc/documentation/faq.html (https://link.getmailspring.com/link/060AB2C3-F3D7-408C-99CF-C4D1B27DD59F at getmailspring.com/4?redirect=https%3A%2F%2Flink.getmailspring.com%2Flink%2F540F752A-DB59-4878-9C98-B8F4B4D9FD0D%40getmailspring.com%2F1%3Fredirect%3Dhttps%253A%252F%252Fwww.mcs.anl.gov%252Fpetsc%252Fdocumentation%252Ffaq.html%26recipient%3DYnNtaXRoQHBldHNjLmRldg%253D%253D&recipient=cGV0c2MtdXNlcnNAbWNzLmFubC5nb3Y%3D) for trouble shooting. > > > > [0]PETSC ERROR: Petsc Release Version 3.12.5, Mar, 29, 2020 > > > > [0]PETSC ERROR: ./testSolve on a arch-linux2-c-debug named f601294ac804 by Unknown Thu Jul 16 01:56:11 2020 > > > > [0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --download-mpich --download-fblaslapack --with-cuda > > > > [0]PETSC ERROR: #1 VecCopy_Nest() line 68 in /usr/local/petsc/petsc-3.12.5/src/vec/vec/impls/nest/vecnest.c > > > > [0]PETSC ERROR: #2 VecCopy() line 1577 in /usr/local/petsc/petsc-3.12.5/src/vec/vec/interface/vector.c > > > > [0]PETSC ERROR: #3 KSPInitialResidual() line 61 in /usr/local/petsc/petsc-3.12.5/src/ksp/ksp/interface/itres.c > > > > [0]PETSC ERROR: #4 KSPSolve_GMRES() line 236 in /usr/local/petsc/petsc-3.12.5/src/ksp/ksp/impls/gmres/gmres.c > > > > [0]PETSC ERROR: #5 KSPSolve() line 760 in /usr/local/petsc/petsc-3.12.5/src/ksp/ksp/interface/itfunc.c > > > > > > > > > > > > Regards > > > > JT > > > > > > > > > > > > > > > > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: test.cpp Type: application/octet-stream Size: 3266 bytes Desc: not available URL: From junchao.zhang at gmail.com Wed Jul 15 17:57:50 2020 From: junchao.zhang at gmail.com (Junchao Zhang) Date: Wed, 15 Jul 2020 17:57:50 -0500 Subject: [petsc-users] Errors encountered with KSPsolver with MATNEST and VECNEST In-Reply-To: References: <5CC396D0-D707-4A29-917E-82DA69F49891@petsc.dev> Message-ID: Adding the MatNestSetVecType() line fixed the problem. But I don't know why petsc had this weird design originally. MatCreateNest(PETSC_COMM_WORLD, 3, NULL, 3, NULL, mblock, &testMesh->system); + MatNestSetVecType(testMesh->system,VECNEST); --Junchao Zhang On Wed, Jul 15, 2020 at 2:16 PM Karl Yang wrote: > Hi, Barry, > > I created a minimum example to reproduce the error. The example code is > attached. > > Mat Object: 1 MPI processes > type: nest > Matrix object: > type=nest, rows=3, cols=3 > MatNest structure: > (0,0) : type=seqaij, rows=5, cols=5 > (0,1) : NULL > (0,2) : type=seqaij, rows=5, cols=5 > (1,0) : NULL > (1,1) : type=seqaij, rows=5, cols=5 > (1,2) : type=seqaij, rows=5, cols=5 > (2,0) : type=seqaij, rows=5, cols=5 > (2,1) : type=seqaij, rows=5, cols=5 > (2,2) : NULL > Vec Object: 1 MPI processes > type: nest > VecNest, rows=3, structure: > (0) : type=seq, rows=5 > Vec Object: 1 MPI processes > type: seq > 1. > 1. > 1. > 1. > 1. > (1) : type=seq, rows=5 > Vec Object: 1 MPI processes > type: seq > 1. > 1. > 1. > 1. > 1. > (2) : type=seq, rows=5 > Vec Object: 1 MPI processes > type: seq > 1. > 1. > 1. > 1. > 1. > [0]PETSC ERROR: --------------------- Error Message > -------------------------------------------------------------- > [0]PETSC ERROR: Invalid argument > [0]PETSC ERROR: Nest vector argument 2 not setup. > [0]PETSC ERROR: See https://www.mcs.anl.gov/petsc/documentation/faq.html > for trouble shooting. > [0]PETSC ERROR: Petsc Release Version 3.12.5, Mar, 29, 2020 > [0]PETSC ERROR: ./testSolve on a arch-linux2-c-debug named f601294ac804 by > Unknown Thu Jul 16 03:11:02 2020 > [0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ > --with-fc=gfortran --download-mpich --download-fblaslapack --with-cuda > [0]PETSC ERROR: #1 VecCopy_Nest() line 68 in > /usr/local/petsc/petsc-3.12.5/src/vec/vec/impls/nest/vecnest.c > [0]PETSC ERROR: #2 VecCopy() line 1577 in > /usr/local/petsc/petsc-3.12.5/src/vec/vec/interface/vector.c > [0]PETSC ERROR: #3 KSPInitialResidual() line 61 in > /usr/local/petsc/petsc-3.12.5/src/ksp/ksp/interface/itres.c > [0]PETSC ERROR: #4 KSPSolve_GMRES() line 236 in > /usr/local/petsc/petsc-3.12.5/src/ksp/ksp/impls/gmres/gmres.c > [0]PETSC ERROR: #5 KSPSolve() line 760 in > /usr/local/petsc/petsc-3.12.5/src/ksp/ksp/interface/itfunc.c > On Jul 16 2020, at 2:39 am, Barry Smith wrote: > > > Could be a bug in PETSc, please send a sample program that produces the > problem. > > Barry > > > On Jul 15, 2020, at 1:28 PM, Karl Yang > > wrote: > > Hi, Barry, > > Thanks for your reply. I output the matrix and vec with matview and > vecview. And I think I do have the values set as it has been shown. > > Mat Object: 1 MPI processes > type: nest > Matrix object: > type=nest, rows=3, cols=3 > MatNest structure: > (0,0) : type=seqaij, rows=25, cols=25 > (0,1) : NULL > (0,2) : type=seqaij, rows=25, cols=25 > (1,0) : NULL > (1,1) : type=seqaij, rows=25, cols=25 > (1,2) : type=seqaij, rows=25, cols=25 > (2,0) : type=seqaij, rows=25, cols=25 > (2,1) : type=seqaij, rows=25, cols=25 > (2,2) : NULL > Vec Object: 1 MPI processes > type: nest > VecNest, rows=3, structure: > (0) : type=seq, rows=25 > Vec Object: 1 MPI processes > type: seq > 0. > 0. > 0. > 0. > 0. > 0. > 75.8279 > -51.0776 > -75.8279 > 0. > 0. > -90.8404 > 58.7011 > 90.8404 > 0. > 0. > -75.8279 > 51.0776 > 75.8279 > 0. > 0. > 0. > 0. > 0. > 0. > (1) : name="Vec_0x84000000_0", type=seq, rows=25 > Vec Object: Vec_0x84000000_0 1 MPI processes > type: seq > 0. > 0. > 0. > 0. > 0. > 0. > 75.8279 > -51.0776 > -75.8279 > 0. > 0. > -90.8404 > 58.7011 > 90.8404 > 0. > 0. > -75.8279 > 51.0776 > 75.8279 > 0. > 0. > 0. > 0. > 0. > 0. > (2) : type=seq, rows=25 > Vec Object: 1 MPI processes > type: seq > 0. > 0. > 0. > 0. > 0. > 0. > 0. > 0. > 0. > 0. > 0. > 0. > 0. > 0. > 0. > 0. > 0. > 0. > 0. > 0. > 0. > 0. > 0. > 0. > 0. > [0]PETSC ERROR: --------------------- Error Message > -------------------------------------------------------------- > [0]PETSC ERROR: Invalid argument > [0]PETSC ERROR: Nest vector argument 4 not setup. > [0]PETSC ERROR: See https://www.mcs.anl.gov/petsc/documentation/faq.html > for > trouble shooting. > [0]PETSC ERROR: Petsc Release Version 3.12.5, Mar, 29, 2020 > [0]PETSC ERROR: ./testSolve on a arch-linux2-c-debug named f601294ac804 by > Unknown Thu Jul 16 02:15:26 2020 > [0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ > --with-fc=gfortran --download-mpich --download-fblaslapack --with-cuda > [0]PETSC ERROR: #1 VecPointwiseMult_Nest() line 245 in > /usr/local/petsc/petsc-3.12.5/src/vec/vec/impls/nest/vecnest.c > [0]PETSC ERROR: #2 VecPointwiseMult() line 1107 in > /usr/local/petsc/petsc-3.12.5/src/vec/vec/interface/vector.c > [0]PETSC ERROR: #3 PCApply_Jacobi() line 272 in > /usr/local/petsc/petsc-3.12.5/src/ksp/pc/impls/jacobi/jacobi.c > [0]PETSC ERROR: #4 PCApply() line 444 in > /usr/local/petsc/petsc-3.12.5/src/ksp/pc/interface/precon.c > [0]PETSC ERROR: #5 KSP_PCApply() line 281 in > /usr/local/petsc/petsc-3.12.5/include/petsc/private/kspimpl.h > [0]PETSC ERROR: #6 KSPInitialResidual() line 65 in > /usr/local/petsc/petsc-3.12.5/src/ksp/ksp/interface/itres.c > [0]PETSC ERROR: #7 KSPSolve_GMRES() line 236 in > /usr/local/petsc/petsc-3.12.5/src/ksp/ksp/impls/gmres/gmres.c > [0]PETSC ERROR: #8 KSPSolve() line 760 in > /usr/local/petsc/petsc-3.12.5/src/ksp/ksp/interface/itfunc.c > > On Jul 16 2020, at 2:13 am, Barry Smith > > wrote: > > > From the error message the second vector you are using to form the nest > vector has not been completely created/set up. Make sure you called either > VecCreateSeq(), VecCreateMPI() or did VecCreate(), VecSetType(), VecSetUp() > and if it is the right hand side make sure you put numerical values in it. > > Barry > > > On Jul 15, 2020, at 1:05 PM, Karl Yang > > wrote: > > Hello, > > I'm trying to solve a 2*2 block matrix system. > I managed to form a 2*2 MatNest and a 21 VecNest. However when I try to > make use of ksp solve, I encounter the following error. There is only one > or two examples on MatNest and VecNest. I could not figure out what could > be the trouble. > > > [0]PETSC ERROR: --------------------- Error Message > -------------------------------------------------------------- > [0]PETSC ERROR: Invalid argument > [0]PETSC ERROR: Nest vector argument 2 not setup. > [0]PETSC ERROR: See https://www.mcs.anl.gov/petsc/documentation/faq.html > for > trouble shooting. > [0]PETSC ERROR: Petsc Release Version 3.12.5, Mar, 29, 2020 > [0]PETSC ERROR: ./testSolve on a arch-linux2-c-debug named f601294ac804 by > Unknown Thu Jul 16 01:56:11 2020 > [0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ > --with-fc=gfortran --download-mpich --download-fblaslapack --with-cuda > [0]PETSC ERROR: #1 VecCopy_Nest() line 68 in > /usr/local/petsc/petsc-3.12.5/src/vec/vec/impls/nest/vecnest.c > [0]PETSC ERROR: #2 VecCopy() line 1577 in > /usr/local/petsc/petsc-3.12.5/src/vec/vec/interface/vector.c > [0]PETSC ERROR: #3 KSPInitialResidual() line 61 in > /usr/local/petsc/petsc-3.12.5/src/ksp/ksp/interface/itres.c > [0]PETSC ERROR: #4 KSPSolve_GMRES() line 236 in > /usr/local/petsc/petsc-3.12.5/src/ksp/ksp/impls/gmres/gmres.c > [0]PETSC ERROR: #5 KSPSolve() line 760 in > /usr/local/petsc/petsc-3.12.5/src/ksp/ksp/interface/itfunc.c > > > Regards > JT > > [image: Sent from Mailspring] -------------- next part -------------- An HTML attachment was scrubbed... URL: From jed at jedbrown.org Wed Jul 15 23:03:57 2020 From: jed at jedbrown.org (Jed Brown) Date: Wed, 15 Jul 2020 22:03:57 -0600 Subject: [petsc-users] Errors encountered with KSPsolver with MATNEST and VECNEST In-Reply-To: References: <5CC396D0-D707-4A29-917E-82DA69F49891@petsc.dev> Message-ID: <87blkgrtqq.fsf@jedbrown.org> It's a bit tedious to determine whether all vectors have the same type. The problem here is that this function assumes that y is of the same type as x. static PetscErrorCode VecCopy_Nest(Vec x,Vec y) { Vec_Nest *bx = (Vec_Nest*)x->data; Vec_Nest *by = (Vec_Nest*)y->data; PetscInt i; PetscErrorCode ierr; PetscFunctionBegin; VecNestCheckCompatible2(x,1,y,2); for (i=0; inb; i++) { ierr = VecCopy(bx->v[i],by->v[i]);CHKERRQ(ierr); } PetscFunctionReturn(0); } IMO, a strong majority of MatNest users are better off using standard Vecs (not VecNest). Junchao Zhang writes: > Adding the MatNestSetVecType() line fixed the problem. But I don't know > why petsc had this weird design originally. > > MatCreateNest(PETSC_COMM_WORLD, 3, NULL, 3, NULL, mblock, > &testMesh->system); > + MatNestSetVecType(testMesh->system,VECNEST); > > --Junchao Zhang > > > On Wed, Jul 15, 2020 at 2:16 PM Karl Yang wrote: > >> Hi, Barry, >> >> I created a minimum example to reproduce the error. The example code is >> attached. >> >> Mat Object: 1 MPI processes >> type: nest >> Matrix object: >> type=nest, rows=3, cols=3 >> MatNest structure: >> (0,0) : type=seqaij, rows=5, cols=5 >> (0,1) : NULL >> (0,2) : type=seqaij, rows=5, cols=5 >> (1,0) : NULL >> (1,1) : type=seqaij, rows=5, cols=5 >> (1,2) : type=seqaij, rows=5, cols=5 >> (2,0) : type=seqaij, rows=5, cols=5 >> (2,1) : type=seqaij, rows=5, cols=5 >> (2,2) : NULL >> Vec Object: 1 MPI processes >> type: nest >> VecNest, rows=3, structure: >> (0) : type=seq, rows=5 >> Vec Object: 1 MPI processes >> type: seq >> 1. >> 1. >> 1. >> 1. >> 1. >> (1) : type=seq, rows=5 >> Vec Object: 1 MPI processes >> type: seq >> 1. >> 1. >> 1. >> 1. >> 1. >> (2) : type=seq, rows=5 >> Vec Object: 1 MPI processes >> type: seq >> 1. >> 1. >> 1. >> 1. >> 1. >> [0]PETSC ERROR: --------------------- Error Message >> -------------------------------------------------------------- >> [0]PETSC ERROR: Invalid argument >> [0]PETSC ERROR: Nest vector argument 2 not setup. >> [0]PETSC ERROR: See https://www.mcs.anl.gov/petsc/documentation/faq.html >> for trouble shooting. >> [0]PETSC ERROR: Petsc Release Version 3.12.5, Mar, 29, 2020 >> [0]PETSC ERROR: ./testSolve on a arch-linux2-c-debug named f601294ac804 by >> Unknown Thu Jul 16 03:11:02 2020 >> [0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ >> --with-fc=gfortran --download-mpich --download-fblaslapack --with-cuda >> [0]PETSC ERROR: #1 VecCopy_Nest() line 68 in >> /usr/local/petsc/petsc-3.12.5/src/vec/vec/impls/nest/vecnest.c >> [0]PETSC ERROR: #2 VecCopy() line 1577 in >> /usr/local/petsc/petsc-3.12.5/src/vec/vec/interface/vector.c >> [0]PETSC ERROR: #3 KSPInitialResidual() line 61 in >> /usr/local/petsc/petsc-3.12.5/src/ksp/ksp/interface/itres.c >> [0]PETSC ERROR: #4 KSPSolve_GMRES() line 236 in >> /usr/local/petsc/petsc-3.12.5/src/ksp/ksp/impls/gmres/gmres.c >> [0]PETSC ERROR: #5 KSPSolve() line 760 in >> /usr/local/petsc/petsc-3.12.5/src/ksp/ksp/interface/itfunc.c >> On Jul 16 2020, at 2:39 am, Barry Smith wrote: >> >> >> Could be a bug in PETSc, please send a sample program that produces the >> problem. >> >> Barry >> >> >> On Jul 15, 2020, at 1:28 PM, Karl Yang > > >> wrote: >> >> Hi, Barry, >> >> Thanks for your reply. I output the matrix and vec with matview and >> vecview. And I think I do have the values set as it has been shown. >> >> Mat Object: 1 MPI processes >> type: nest >> Matrix object: >> type=nest, rows=3, cols=3 >> MatNest structure: >> (0,0) : type=seqaij, rows=25, cols=25 >> (0,1) : NULL >> (0,2) : type=seqaij, rows=25, cols=25 >> (1,0) : NULL >> (1,1) : type=seqaij, rows=25, cols=25 >> (1,2) : type=seqaij, rows=25, cols=25 >> (2,0) : type=seqaij, rows=25, cols=25 >> (2,1) : type=seqaij, rows=25, cols=25 >> (2,2) : NULL >> Vec Object: 1 MPI processes >> type: nest >> VecNest, rows=3, structure: >> (0) : type=seq, rows=25 >> Vec Object: 1 MPI processes >> type: seq >> 0. >> 0. >> 0. >> 0. >> 0. >> 0. >> 75.8279 >> -51.0776 >> -75.8279 >> 0. >> 0. >> -90.8404 >> 58.7011 >> 90.8404 >> 0. >> 0. >> -75.8279 >> 51.0776 >> 75.8279 >> 0. >> 0. >> 0. >> 0. >> 0. >> 0. >> (1) : name="Vec_0x84000000_0", type=seq, rows=25 >> Vec Object: Vec_0x84000000_0 1 MPI processes >> type: seq >> 0. >> 0. >> 0. >> 0. >> 0. >> 0. >> 75.8279 >> -51.0776 >> -75.8279 >> 0. >> 0. >> -90.8404 >> 58.7011 >> 90.8404 >> 0. >> 0. >> -75.8279 >> 51.0776 >> 75.8279 >> 0. >> 0. >> 0. >> 0. >> 0. >> 0. >> (2) : type=seq, rows=25 >> Vec Object: 1 MPI processes >> type: seq >> 0. >> 0. >> 0. >> 0. >> 0. >> 0. >> 0. >> 0. >> 0. >> 0. >> 0. >> 0. >> 0. >> 0. >> 0. >> 0. >> 0. >> 0. >> 0. >> 0. >> 0. >> 0. >> 0. >> 0. >> 0. >> [0]PETSC ERROR: --------------------- Error Message >> -------------------------------------------------------------- >> [0]PETSC ERROR: Invalid argument >> [0]PETSC ERROR: Nest vector argument 4 not setup. >> [0]PETSC ERROR: See https://www.mcs.anl.gov/petsc/documentation/faq.html >> for >> trouble shooting. >> [0]PETSC ERROR: Petsc Release Version 3.12.5, Mar, 29, 2020 >> [0]PETSC ERROR: ./testSolve on a arch-linux2-c-debug named f601294ac804 by >> Unknown Thu Jul 16 02:15:26 2020 >> [0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ >> --with-fc=gfortran --download-mpich --download-fblaslapack --with-cuda >> [0]PETSC ERROR: #1 VecPointwiseMult_Nest() line 245 in >> /usr/local/petsc/petsc-3.12.5/src/vec/vec/impls/nest/vecnest.c >> [0]PETSC ERROR: #2 VecPointwiseMult() line 1107 in >> /usr/local/petsc/petsc-3.12.5/src/vec/vec/interface/vector.c >> [0]PETSC ERROR: #3 PCApply_Jacobi() line 272 in >> /usr/local/petsc/petsc-3.12.5/src/ksp/pc/impls/jacobi/jacobi.c >> [0]PETSC ERROR: #4 PCApply() line 444 in >> /usr/local/petsc/petsc-3.12.5/src/ksp/pc/interface/precon.c >> [0]PETSC ERROR: #5 KSP_PCApply() line 281 in >> /usr/local/petsc/petsc-3.12.5/include/petsc/private/kspimpl.h >> [0]PETSC ERROR: #6 KSPInitialResidual() line 65 in >> /usr/local/petsc/petsc-3.12.5/src/ksp/ksp/interface/itres.c >> [0]PETSC ERROR: #7 KSPSolve_GMRES() line 236 in >> /usr/local/petsc/petsc-3.12.5/src/ksp/ksp/impls/gmres/gmres.c >> [0]PETSC ERROR: #8 KSPSolve() line 760 in >> /usr/local/petsc/petsc-3.12.5/src/ksp/ksp/interface/itfunc.c >> >> On Jul 16 2020, at 2:13 am, Barry Smith > > >> wrote: >> >> >> From the error message the second vector you are using to form the nest >> vector has not been completely created/set up. Make sure you called either >> VecCreateSeq(), VecCreateMPI() or did VecCreate(), VecSetType(), VecSetUp() >> and if it is the right hand side make sure you put numerical values in it. >> >> Barry >> >> >> On Jul 15, 2020, at 1:05 PM, Karl Yang > > >> wrote: >> >> Hello, >> >> I'm trying to solve a 2*2 block matrix system. >> I managed to form a 2*2 MatNest and a 21 VecNest. However when I try to >> make use of ksp solve, I encounter the following error. There is only one >> or two examples on MatNest and VecNest. I could not figure out what could >> be the trouble. >> >> >> [0]PETSC ERROR: --------------------- Error Message >> -------------------------------------------------------------- >> [0]PETSC ERROR: Invalid argument >> [0]PETSC ERROR: Nest vector argument 2 not setup. >> [0]PETSC ERROR: See https://www.mcs.anl.gov/petsc/documentation/faq.html >> for >> trouble shooting. >> [0]PETSC ERROR: Petsc Release Version 3.12.5, Mar, 29, 2020 >> [0]PETSC ERROR: ./testSolve on a arch-linux2-c-debug named f601294ac804 by >> Unknown Thu Jul 16 01:56:11 2020 >> [0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ >> --with-fc=gfortran --download-mpich --download-fblaslapack --with-cuda >> [0]PETSC ERROR: #1 VecCopy_Nest() line 68 in >> /usr/local/petsc/petsc-3.12.5/src/vec/vec/impls/nest/vecnest.c >> [0]PETSC ERROR: #2 VecCopy() line 1577 in >> /usr/local/petsc/petsc-3.12.5/src/vec/vec/interface/vector.c >> [0]PETSC ERROR: #3 KSPInitialResidual() line 61 in >> /usr/local/petsc/petsc-3.12.5/src/ksp/ksp/interface/itres.c >> [0]PETSC ERROR: #4 KSPSolve_GMRES() line 236 in >> /usr/local/petsc/petsc-3.12.5/src/ksp/ksp/impls/gmres/gmres.c >> [0]PETSC ERROR: #5 KSPSolve() line 760 in >> /usr/local/petsc/petsc-3.12.5/src/ksp/ksp/interface/itfunc.c >> >> >> Regards >> JT >> >> [image: Sent from Mailspring] From y.juntao at hotmail.com Thu Jul 16 00:23:23 2020 From: y.juntao at hotmail.com (Karl Yang) Date: Thu, 16 Jul 2020 13:23:23 +0800 Subject: [petsc-users] Errors encountered with KSPsolver with MATNEST and VECNEST In-Reply-To: References: Message-ID: Hi, Junchao, Thank you. It works. :) May I ask some follow up question regarding the same example about fieldsplit. I am trying to learn about matnest with fieldsplit preconditioner on manual page 94. But I get a bit lost trying to understand fields, ISs once it is not eactly the same on the manual. In this same example, I tried to use PCFIELDSPLIT and PCFieldSplitSetDetectSaddlePoint as follows. IS rows[3]; IS cols[3]; MatNestGetISs(testMesh->system, rows, cols); KSP ksp; PC pc; KSPCreate(PETSC_COMM_WORLD, &ksp); KSPSetType(ksp, KSPFGMRES); KSPSetOperators(ksp, testMesh->system, testMesh->system); KSPGetPC(ksp, &pc); PCFieldSplitSetIS(pc, "u", rows[0]); PCFieldSplitSetIS(pc, "v", rows[1]); PCFieldSplitSetIS(pc, "p", rows[2]); PCSetType(pc, PCFIELDSPLIT); PCFieldSplitSetDetectSaddlePoint(pc, PETSC_TRUE); KSPSetUp(ksp); KSPSolve(ksp, testMesh->rhs, testMesh->solution); But I will get the following error, [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [0]PETSC ERROR: Arguments are incompatible [0]PETSC ERROR: Could not find index set [0]PETSC ERROR: See https://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [0]PETSC ERROR: Petsc Release Version 3.12.5, Mar, 29, 2020 [0]PETSC ERROR: ./testSolve on a arch-linux2-c-debug named f601294ac804 by Unknown Thu Jul 16 13:08:39 2020 [0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --download-mpich --download-fblaslapack --with-cuda [0]PETSC ERROR: #1 MatNestFindIS() line 365 in /usr/local/petsc/petsc-3.12.5/src/mat/impls/nest/matnest.c [0]PETSC ERROR: #2 MatNestFindSubMat() line 425 in /usr/local/petsc/petsc-3.12.5/src/mat/impls/nest/matnest.c [0]PETSC ERROR: #3 MatCreateSubMatrix_Nest() line 456 in /usr/local/petsc/petsc-3.12.5/src/mat/impls/nest/matnest.c [0]PETSC ERROR: #4 MatCreateSubMatrix() line 7859 in /usr/local/petsc/petsc-3.12.5/src/mat/interface/matrix.c [0]PETSC ERROR: #5 PCSetUp_FieldSplit() line 676 in /usr/local/petsc/petsc-3.12.5/src/ksp/pc/impls/fieldsplit/fieldsplit.c [0]PETSC ERROR: #6 PCSetUp() line 894 in /usr/local/petsc/petsc-3.12.5/src/ksp/pc/interface/precon.c [0]PETSC ERROR: #7 KSPSetUp() line 376 in /usr/local/petsc/petsc-3.12.5/src/ksp/ksp/interface/itfunc.c [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [0]PETSC ERROR: Corrupt argument: https://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [0]PETSC ERROR: Invalid Pointer to Object: Parameter # 1 [0]PETSC ERROR: See https://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [0]PETSC ERROR: Petsc Release Version 3.12.5, Mar, 29, 2020 [0]PETSC ERROR: ./testSolve on a arch-linux2-c-debug named f601294ac804 by Unknown Thu Jul 16 13:08:39 2020 [0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --download-mpich --download-fblaslapack --with-cuda [0]PETSC ERROR: #8 MatDestroy() line 1266 in /usr/local/petsc/petsc-3.12.5/src/mat/interface/matrix.c [0]PETSC ERROR: #9 PCSetUp_FieldSplit() line 697 in /usr/local/petsc/petsc-3.12.5/src/ksp/pc/impls/fieldsplit/fieldsplit.c [0]PETSC ERROR: #10 PCSetUp() line 894 in /usr/local/petsc/petsc-3.12.5/src/ksp/pc/interface/precon.c [0]PETSC ERROR: #11 KSPSetUp() line 376 in /usr/local/petsc/petsc-3.12.5/src/ksp/ksp/interface/itfunc.c [0]PETSC ERROR: #12 KSPSolve() line 703 in /usr/local/petsc/petsc-3.12.5/src/ksp/ksp/interface/itfunc.c I read about the examples on the manual and mostly are based on 22 blocks but still cannot figure out how pcfieldsplit should be set in this 3 *3 blocks, and how IS should be set accordingly. I read about -fieldsplit_1_pc_type none option suggested on the documentation, but I am not sure is it still valid in a 33 blocks. Regards JT On Jul 16 2020, at 6:57 am, Junchao Zhang wrote: > Adding the MatNestSetVecType() line fixed the problem. But I don't know why petsc had this weird design originally. > > MatCreateNest(PETSC_COMM_WORLD, 3, NULL, 3, NULL, mblock, &testMesh->system); > + MatNestSetVecType(testMesh->system,VECNEST); > > --Junchao Zhang > > On Wed, Jul 15, 2020 at 2:16 PM Karl Yang wrote: > > Hi, Barry, > > > > I created a minimum example to reproduce the error. The example code is attached. > > Mat Object: 1 MPI processes > > type: nest > > Matrix object: > > type=nest, rows=3, cols=3 > > MatNest structure: > > (0,0) : type=seqaij, rows=5, cols=5 > > (0,1) : NULL > > (0,2) : type=seqaij, rows=5, cols=5 > > (1,0) : NULL > > (1,1) : type=seqaij, rows=5, cols=5 > > (1,2) : type=seqaij, rows=5, cols=5 > > (2,0) : type=seqaij, rows=5, cols=5 > > (2,1) : type=seqaij, rows=5, cols=5 > > (2,2) : NULL > > Vec Object: 1 MPI processes > > type: nest > > VecNest, rows=3, structure: > > (0) : type=seq, rows=5 > > Vec Object: 1 MPI processes > > type: seq > > 1. > > 1. > > 1. > > 1. > > 1. > > (1) : type=seq, rows=5 > > Vec Object: 1 MPI processes > > type: seq > > 1. > > 1. > > 1. > > 1. > > 1. > > (2) : type=seq, rows=5 > > Vec Object: 1 MPI processes > > type: seq > > 1. > > 1. > > 1. > > 1. > > 1. > > [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- > > [0]PETSC ERROR: Invalid argument > > [0]PETSC ERROR: Nest vector argument 2 not setup. > > [0]PETSC ERROR: See https://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. > > [0]PETSC ERROR: Petsc Release Version 3.12.5, Mar, 29, 2020 > > [0]PETSC ERROR: ./testSolve on a arch-linux2-c-debug named f601294ac804 by Unknown Thu Jul 16 03:11:02 2020 > > [0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --download-mpich --download-fblaslapack --with-cuda > > [0]PETSC ERROR: #1 VecCopy_Nest() line 68 in /usr/local/petsc/petsc-3.12.5/src/vec/vec/impls/nest/vecnest.c > > [0]PETSC ERROR: #2 VecCopy() line 1577 in /usr/local/petsc/petsc-3.12.5/src/vec/vec/interface/vector.c > > [0]PETSC ERROR: #3 KSPInitialResidual() line 61 in /usr/local/petsc/petsc-3.12.5/src/ksp/ksp/interface/itres.c > > [0]PETSC ERROR: #4 KSPSolve_GMRES() line 236 in /usr/local/petsc/petsc-3.12.5/src/ksp/ksp/impls/gmres/gmres.c > > [0]PETSC ERROR: #5 KSPSolve() line 760 in /usr/local/petsc/petsc-3.12.5/src/ksp/ksp/interface/itfunc.c > > On Jul 16 2020, at 2:39 am, Barry Smith wrote: > > > > > > Could be a bug in PETSc, please send a sample program that produces the problem. > > > > > > Barry > > > > > > > > > > On Jul 15, 2020, at 1:28 PM, Karl Yang wrote: > > > > Hi, Barry, > > > > Thanks for your reply. I output the matrix and vec with matview and vecview. And I think I do have the values set as it has been shown. > > > > Mat Object: 1 MPI processes > > > > type: nest > > > > Matrix object: > > > > type=nest, rows=3, cols=3 > > > > MatNest structure: > > > > (0,0) : type=seqaij, rows=25, cols=25 > > > > (0,1) : NULL > > > > (0,2) : type=seqaij, rows=25, cols=25 > > > > (1,0) : NULL > > > > (1,1) : type=seqaij, rows=25, cols=25 > > > > (1,2) : type=seqaij, rows=25, cols=25 > > > > (2,0) : type=seqaij, rows=25, cols=25 > > > > (2,1) : type=seqaij, rows=25, cols=25 > > > > (2,2) : NULL > > > > Vec Object: 1 MPI processes > > > > type: nest > > > > VecNest, rows=3, structure: > > > > (0) : type=seq, rows=25 > > > > Vec Object: 1 MPI processes > > > > type: seq > > > > 0. > > > > 0. > > > > 0. > > > > 0. > > > > 0. > > > > 0. > > > > 75.8279 > > > > -51.0776 > > > > -75.8279 > > > > 0. > > > > 0. > > > > -90.8404 > > > > 58.7011 > > > > 90.8404 > > > > 0. > > > > 0. > > > > -75.8279 > > > > 51.0776 > > > > 75.8279 > > > > 0. > > > > 0. > > > > 0. > > > > 0. > > > > 0. > > > > 0. > > > > (1) : name="Vec_0x84000000_0", type=seq, rows=25 > > > > Vec Object: Vec_0x84000000_0 1 MPI processes > > > > type: seq > > > > 0. > > > > 0. > > > > 0. > > > > 0. > > > > 0. > > > > 0. > > > > 75.8279 > > > > -51.0776 > > > > -75.8279 > > > > 0. > > > > 0. > > > > -90.8404 > > > > 58.7011 > > > > 90.8404 > > > > 0. > > > > 0. > > > > -75.8279 > > > > 51.0776 > > > > 75.8279 > > > > 0. > > > > 0. > > > > 0. > > > > 0. > > > > 0. > > > > 0. > > > > (2) : type=seq, rows=25 > > > > Vec Object: 1 MPI processes > > > > type: seq > > > > 0. > > > > 0. > > > > 0. > > > > 0. > > > > 0. > > > > 0. > > > > 0. > > > > 0. > > > > 0. > > > > 0. > > > > 0. > > > > 0. > > > > 0. > > > > 0. > > > > 0. > > > > 0. > > > > 0. > > > > 0. > > > > 0. > > > > 0. > > > > 0. > > > > 0. > > > > 0. > > > > 0. > > > > 0. > > > > [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- > > > > [0]PETSC ERROR: Invalid argument > > > > [0]PETSC ERROR: Nest vector argument 4 not setup. > > > > [0]PETSC ERROR: See https://www.mcs.anl.gov/petsc/documentation/faq.html (https://link.getmailspring.com/link/060AB2C3-F3D7-408C-99CF-C4D1B27DD59F at getmailspring.com/1?redirect=https%3A%2F%2Fwww.mcs.anl.gov%2Fpetsc%2Fdocumentation%2Ffaq.html&recipient=cGV0c2MtdXNlcnNAbWNzLmFubC5nb3Y%3D) for trouble shooting. > > > > [0]PETSC ERROR: Petsc Release Version 3.12.5, Mar, 29, 2020 > > > > [0]PETSC ERROR: ./testSolve on a arch-linux2-c-debug named f601294ac804 by Unknown Thu Jul 16 02:15:26 2020 > > > > [0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --download-mpich --download-fblaslapack --with-cuda > > > > [0]PETSC ERROR: #1 VecPointwiseMult_Nest() line 245 in /usr/local/petsc/petsc-3.12.5/src/vec/vec/impls/nest/vecnest.c > > > > [0]PETSC ERROR: #2 VecPointwiseMult() line 1107 in /usr/local/petsc/petsc-3.12.5/src/vec/vec/interface/vector.c > > > > [0]PETSC ERROR: #3 PCApply_Jacobi() line 272 in /usr/local/petsc/petsc-3.12.5/src/ksp/pc/impls/jacobi/jacobi.c > > > > [0]PETSC ERROR: #4 PCApply() line 444 in /usr/local/petsc/petsc-3.12.5/src/ksp/pc/interface/precon.c > > > > [0]PETSC ERROR: #5 KSP_PCApply() line 281 in /usr/local/petsc/petsc-3.12.5/include/petsc/private/kspimpl.h > > > > [0]PETSC ERROR: #6 KSPInitialResidual() line 65 in /usr/local/petsc/petsc-3.12.5/src/ksp/ksp/interface/itres.c > > > > [0]PETSC ERROR: #7 KSPSolve_GMRES() line 236 in /usr/local/petsc/petsc-3.12.5/src/ksp/ksp/impls/gmres/gmres.c > > > > [0]PETSC ERROR: #8 KSPSolve() line 760 in /usr/local/petsc/petsc-3.12.5/src/ksp/ksp/interface/itfunc.c > > > > > > > > On Jul 16 2020, at 2:13 am, Barry Smith wrote: > > > > > > > > > > From the error message the second vector you are using to form the nest vector has not been completely created/set up. Make sure you called either VecCreateSeq(), VecCreateMPI() or did VecCreate(), VecSetType(), VecSetUp() and if it is the right hand side make sure you put numerical values in it. > > > > > > > > > > Barry > > > > > > > > > > > > > > > > On Jul 15, 2020, at 1:05 PM, Karl Yang wrote: > > > > > > Hello, > > > > > > I'm trying to solve a 2*2 block matrix system. > > > > > > I managed to form a 2*2 MatNest and a 21 VecNest. However when I try to make use of ksp solve, I encounter the following error. There is only one or two examples on MatNest and VecNest. I could not figure out what could be the trouble. > > > > > > > > > > > > > > > > > > [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- > > > > > > [0]PETSC ERROR: Invalid argument > > > > > > [0]PETSC ERROR: Nest vector argument 2 not setup. > > > > > > [0]PETSC ERROR: See https://www.mcs.anl.gov/petsc/documentation/faq.html (https://link.getmailspring.com/link/060AB2C3-F3D7-408C-99CF-C4D1B27DD59F at getmailspring.com/4?redirect=https%3A%2F%2Flink.getmailspring.com%2Flink%2F540F752A-DB59-4878-9C98-B8F4B4D9FD0D%40getmailspring.com%2F1%3Fredirect%3Dhttps%253A%252F%252Fwww.mcs.anl.gov%252Fpetsc%252Fdocumentation%252Ffaq.html%26recipient%3DYnNtaXRoQHBldHNjLmRldg%253D%253D&recipient=cGV0c2MtdXNlcnNAbWNzLmFubC5nb3Y%3D) for trouble shooting. > > > > > > [0]PETSC ERROR: Petsc Release Version 3.12.5, Mar, 29, 2020 > > > > > > [0]PETSC ERROR: ./testSolve on a arch-linux2-c-debug named f601294ac804 by Unknown Thu Jul 16 01:56:11 2020 > > > > > > [0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --download-mpich --download-fblaslapack --with-cuda > > > > > > [0]PETSC ERROR: #1 VecCopy_Nest() line 68 in /usr/local/petsc/petsc-3.12.5/src/vec/vec/impls/nest/vecnest.c > > > > > > [0]PETSC ERROR: #2 VecCopy() line 1577 in /usr/local/petsc/petsc-3.12.5/src/vec/vec/interface/vector.c > > > > > > [0]PETSC ERROR: #3 KSPInitialResidual() line 61 in /usr/local/petsc/petsc-3.12.5/src/ksp/ksp/interface/itres.c > > > > > > [0]PETSC ERROR: #4 KSPSolve_GMRES() line 236 in /usr/local/petsc/petsc-3.12.5/src/ksp/ksp/impls/gmres/gmres.c > > > > > > [0]PETSC ERROR: #5 KSPSolve() line 760 in /usr/local/petsc/petsc-3.12.5/src/ksp/ksp/interface/itfunc.c > > > > > > > > > > > > > > > > > > Regards > > > > > > JT > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Thu Jul 16 04:30:56 2020 From: knepley at gmail.com (Matthew Knepley) Date: Thu, 16 Jul 2020 05:30:56 -0400 Subject: [petsc-users] Errors encountered with KSPsolver with MATNEST and VECNEST In-Reply-To: References: Message-ID: On Thu, Jul 16, 2020 at 1:23 AM Karl Yang wrote: > Hi, Junchao, > > Thank you. It works. :) > > May I ask some follow up question regarding the same example about > fieldsplit. I am trying to learn about matnest with fieldsplit > preconditioner on manual page 94. But I get a bit lost trying to understand > fields, ISs once it is not eactly the same on the manual. > > In this same example, I tried to use PCFIELDSPLIT and > PCFieldSplitSetDetectSaddlePoint as follows. > You cannot mix DetectSaddlePoint() with SetIS() since the detection tries to make the proper ISes. Maybe we should check that. Thanks, Matt > IS rows[3]; > IS cols[3]; > MatNestGetISs(testMesh->system, rows, cols); > > KSP ksp; > PC pc; > KSPCreate(PETSC_COMM_WORLD, &ksp); > KSPSetType(ksp, KSPFGMRES); > KSPSetOperators(ksp, testMesh->system, testMesh->system); > KSPGetPC(ksp, &pc); > PCFieldSplitSetIS(pc, "u", rows[0]); > PCFieldSplitSetIS(pc, "v", rows[1]); > PCFieldSplitSetIS(pc, "p", rows[2]); > PCSetType(pc, PCFIELDSPLIT); > PCFieldSplitSetDetectSaddlePoint(pc, PETSC_TRUE); > KSPSetUp(ksp); > KSPSolve(ksp, testMesh->rhs, testMesh->solution); > > But I will get the following error, > > [0]PETSC ERROR: --------------------- Error Message > -------------------------------------------------------------- > [0]PETSC ERROR: Arguments are incompatible > [0]PETSC ERROR: Could not find index set > [0]PETSC ERROR: See https://www.mcs.anl.gov/petsc/documentation/faq.html > for trouble shooting. > [0]PETSC ERROR: Petsc Release Version 3.12.5, Mar, 29, 2020 > [0]PETSC ERROR: ./testSolve on a arch-linux2-c-debug named f601294ac804 by > Unknown Thu Jul 16 13:08:39 2020 > [0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ > --with-fc=gfortran --download-mpich --download-fblaslapack --with-cuda > [0]PETSC ERROR: #1 MatNestFindIS() line 365 in > /usr/local/petsc/petsc-3.12.5/src/mat/impls/nest/matnest.c > [0]PETSC ERROR: #2 MatNestFindSubMat() line 425 in > /usr/local/petsc/petsc-3.12.5/src/mat/impls/nest/matnest.c > [0]PETSC ERROR: #3 MatCreateSubMatrix_Nest() line 456 in > /usr/local/petsc/petsc-3.12.5/src/mat/impls/nest/matnest.c > [0]PETSC ERROR: #4 MatCreateSubMatrix() line 7859 in > /usr/local/petsc/petsc-3.12.5/src/mat/interface/matrix.c > [0]PETSC ERROR: #5 PCSetUp_FieldSplit() line 676 in > /usr/local/petsc/petsc-3.12.5/src/ksp/pc/impls/fieldsplit/fieldsplit.c > [0]PETSC ERROR: #6 PCSetUp() line 894 in > /usr/local/petsc/petsc-3.12.5/src/ksp/pc/interface/precon.c > [0]PETSC ERROR: #7 KSPSetUp() line 376 in > /usr/local/petsc/petsc-3.12.5/src/ksp/ksp/interface/itfunc.c > [0]PETSC ERROR: --------------------- Error Message > -------------------------------------------------------------- > [0]PETSC ERROR: Corrupt argument: > https://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind > [0]PETSC ERROR: Invalid Pointer to Object: Parameter # 1 > [0]PETSC ERROR: See https://www.mcs.anl.gov/petsc/documentation/faq.html > for trouble shooting. > [0]PETSC ERROR: Petsc Release Version 3.12.5, Mar, 29, 2020 > [0]PETSC ERROR: ./testSolve on a arch-linux2-c-debug named f601294ac804 by > Unknown Thu Jul 16 13:08:39 2020 > [0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ > --with-fc=gfortran --download-mpich --download-fblaslapack --with-cuda > [0]PETSC ERROR: #8 MatDestroy() line 1266 in > /usr/local/petsc/petsc-3.12.5/src/mat/interface/matrix.c > [0]PETSC ERROR: #9 PCSetUp_FieldSplit() line 697 in > /usr/local/petsc/petsc-3.12.5/src/ksp/pc/impls/fieldsplit/fieldsplit.c > [0]PETSC ERROR: #10 PCSetUp() line 894 in > /usr/local/petsc/petsc-3.12.5/src/ksp/pc/interface/precon.c > [0]PETSC ERROR: #11 KSPSetUp() line 376 in > /usr/local/petsc/petsc-3.12.5/src/ksp/ksp/interface/itfunc.c > [0]PETSC ERROR: #12 KSPSolve() line 703 in > /usr/local/petsc/petsc-3.12.5/src/ksp/ksp/interface/itfunc.c > > I read about the examples on the manual and mostly are based on 2*2 > blocks but still cannot figure out how pcfieldsplit should be set in this 3* > **3 blocks, and how IS should be set accordingly. I read about > -fieldsplit_1_pc_type none option suggested on the documentation, but I am > not sure is it still valid in a 33 blocks. * > > *Regards* > *JT* > On Jul 16 2020, at 6:57 am, Junchao Zhang wrote: > > Adding the MatNestSetVecType() line fixed the problem. But I don't know > why petsc had this weird design originally. > > MatCreateNest(PETSC_COMM_WORLD, 3, NULL, 3, NULL, mblock, > &testMesh->system); > + MatNestSetVecType(testMesh->system,VECNEST); > > --Junchao Zhang > > > [image: Sent from Mailspring] > On Wed, Jul 15, 2020 at 2:16 PM Karl Yang wrote: > > Hi, Barry, > > I created a minimum example to reproduce the error. The example code is > attached. > > Mat Object: 1 MPI processes > type: nest > Matrix object: > type=nest, rows=3, cols=3 > MatNest structure: > (0,0) : type=seqaij, rows=5, cols=5 > (0,1) : NULL > (0,2) : type=seqaij, rows=5, cols=5 > (1,0) : NULL > (1,1) : type=seqaij, rows=5, cols=5 > (1,2) : type=seqaij, rows=5, cols=5 > (2,0) : type=seqaij, rows=5, cols=5 > (2,1) : type=seqaij, rows=5, cols=5 > (2,2) : NULL > Vec Object: 1 MPI processes > type: nest > VecNest, rows=3, structure: > (0) : type=seq, rows=5 > Vec Object: 1 MPI processes > type: seq > 1. > 1. > 1. > 1. > 1. > (1) : type=seq, rows=5 > Vec Object: 1 MPI processes > type: seq > 1. > 1. > 1. > 1. > 1. > (2) : type=seq, rows=5 > Vec Object: 1 MPI processes > type: seq > 1. > 1. > 1. > 1. > 1. > [0]PETSC ERROR: --------------------- Error Message > -------------------------------------------------------------- > [0]PETSC ERROR: Invalid argument > [0]PETSC ERROR: Nest vector argument 2 not setup. > [0]PETSC ERROR: See https://www.mcs.anl.gov/petsc/documentation/faq.html for > trouble shooting. > [0]PETSC ERROR: Petsc Release Version 3.12.5, Mar, 29, 2020 > [0]PETSC ERROR: ./testSolve on a arch-linux2-c-debug named f601294ac804 by > Unknown Thu Jul 16 03:11:02 2020 > [0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ > --with-fc=gfortran --download-mpich --download-fblaslapack --with-cuda > [0]PETSC ERROR: #1 VecCopy_Nest() line 68 in > /usr/local/petsc/petsc-3.12.5/src/vec/vec/impls/nest/vecnest.c > [0]PETSC ERROR: #2 VecCopy() line 1577 in > /usr/local/petsc/petsc-3.12.5/src/vec/vec/interface/vector.c > [0]PETSC ERROR: #3 KSPInitialResidual() line 61 in > /usr/local/petsc/petsc-3.12.5/src/ksp/ksp/interface/itres.c > [0]PETSC ERROR: #4 KSPSolve_GMRES() line 236 in > /usr/local/petsc/petsc-3.12.5/src/ksp/ksp/impls/gmres/gmres.c > [0]PETSC ERROR: #5 KSPSolve() line 760 in > /usr/local/petsc/petsc-3.12.5/src/ksp/ksp/interface/itfunc.c > On Jul 16 2020, at 2:39 am, Barry Smith wrote: > > > Could be a bug in PETSc, please send a sample program that produces the > problem. > > Barry > > > On Jul 15, 2020, at 1:28 PM, Karl Yang > > wrote: > > Hi, Barry, > > Thanks for your reply. I output the matrix and vec with matview and > vecview. And I think I do have the values set as it has been shown. > > Mat Object: 1 MPI processes > type: nest > Matrix object: > type=nest, rows=3, cols=3 > MatNest structure: > (0,0) : type=seqaij, rows=25, cols=25 > (0,1) : NULL > (0,2) : type=seqaij, rows=25, cols=25 > (1,0) : NULL > (1,1) : type=seqaij, rows=25, cols=25 > (1,2) : type=seqaij, rows=25, cols=25 > (2,0) : type=seqaij, rows=25, cols=25 > (2,1) : type=seqaij, rows=25, cols=25 > (2,2) : NULL > Vec Object: 1 MPI processes > type: nest > VecNest, rows=3, structure: > (0) : type=seq, rows=25 > Vec Object: 1 MPI processes > type: seq > 0. > 0. > 0. > 0. > 0. > 0. > 75.8279 > -51.0776 > -75.8279 > 0. > 0. > -90.8404 > 58.7011 > 90.8404 > 0. > 0. > -75.8279 > 51.0776 > 75.8279 > 0. > 0. > 0. > 0. > 0. > 0. > (1) : name="Vec_0x84000000_0", type=seq, rows=25 > Vec Object: Vec_0x84000000_0 1 MPI processes > type: seq > 0. > 0. > 0. > 0. > 0. > 0. > 75.8279 > -51.0776 > -75.8279 > 0. > 0. > -90.8404 > 58.7011 > 90.8404 > 0. > 0. > -75.8279 > 51.0776 > 75.8279 > 0. > 0. > 0. > 0. > 0. > 0. > (2) : type=seq, rows=25 > Vec Object: 1 MPI processes > type: seq > 0. > 0. > 0. > 0. > 0. > 0. > 0. > 0. > 0. > 0. > 0. > 0. > 0. > 0. > 0. > 0. > 0. > 0. > 0. > 0. > 0. > 0. > 0. > 0. > 0. > [0]PETSC ERROR: --------------------- Error Message > -------------------------------------------------------------- > [0]PETSC ERROR: Invalid argument > [0]PETSC ERROR: Nest vector argument 4 not setup. > [0]PETSC ERROR: See https://www.mcs.anl.gov/petsc/documentation/faq.html > for > trouble shooting. > [0]PETSC ERROR: Petsc Release Version 3.12.5, Mar, 29, 2020 > [0]PETSC ERROR: ./testSolve on a arch-linux2-c-debug named f601294ac804 by > Unknown Thu Jul 16 02:15:26 2020 > [0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ > --with-fc=gfortran --download-mpich --download-fblaslapack --with-cuda > [0]PETSC ERROR: #1 VecPointwiseMult_Nest() line 245 in > /usr/local/petsc/petsc-3.12.5/src/vec/vec/impls/nest/vecnest.c > [0]PETSC ERROR: #2 VecPointwiseMult() line 1107 in > /usr/local/petsc/petsc-3.12.5/src/vec/vec/interface/vector.c > [0]PETSC ERROR: #3 PCApply_Jacobi() line 272 in > /usr/local/petsc/petsc-3.12.5/src/ksp/pc/impls/jacobi/jacobi.c > [0]PETSC ERROR: #4 PCApply() line 444 in > /usr/local/petsc/petsc-3.12.5/src/ksp/pc/interface/precon.c > [0]PETSC ERROR: #5 KSP_PCApply() line 281 in > /usr/local/petsc/petsc-3.12.5/include/petsc/private/kspimpl.h > [0]PETSC ERROR: #6 KSPInitialResidual() line 65 in > /usr/local/petsc/petsc-3.12.5/src/ksp/ksp/interface/itres.c > [0]PETSC ERROR: #7 KSPSolve_GMRES() line 236 in > /usr/local/petsc/petsc-3.12.5/src/ksp/ksp/impls/gmres/gmres.c > [0]PETSC ERROR: #8 KSPSolve() line 760 in > /usr/local/petsc/petsc-3.12.5/src/ksp/ksp/interface/itfunc.c > > On Jul 16 2020, at 2:13 am, Barry Smith > > wrote: > > > From the error message the second vector you are using to form the nest > vector has not been completely created/set up. Make sure you called either > VecCreateSeq(), VecCreateMPI() or did VecCreate(), VecSetType(), VecSetUp() > and if it is the right hand side make sure you put numerical values in it. > > Barry > > > On Jul 15, 2020, at 1:05 PM, Karl Yang > > wrote: > > Hello, > > I'm trying to solve a 2*2 block matrix system. > I managed to form a 2*2 MatNest and a 21 VecNest. However when I try to > make use of ksp solve, I encounter the following error. There is only one > or two examples on MatNest and VecNest. I could not figure out what could > be the trouble. > > > [0]PETSC ERROR: --------------------- Error Message > -------------------------------------------------------------- > [0]PETSC ERROR: Invalid argument > [0]PETSC ERROR: Nest vector argument 2 not setup. > [0]PETSC ERROR: See https://www.mcs.anl.gov/petsc/documentation/faq.html > for > trouble shooting. > [0]PETSC ERROR: Petsc Release Version 3.12.5, Mar, 29, 2020 > [0]PETSC ERROR: ./testSolve on a arch-linux2-c-debug named f601294ac804 by > Unknown Thu Jul 16 01:56:11 2020 > [0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ > --with-fc=gfortran --download-mpich --download-fblaslapack --with-cuda > [0]PETSC ERROR: #1 VecCopy_Nest() line 68 in > /usr/local/petsc/petsc-3.12.5/src/vec/vec/impls/nest/vecnest.c > [0]PETSC ERROR: #2 VecCopy() line 1577 in > /usr/local/petsc/petsc-3.12.5/src/vec/vec/interface/vector.c > [0]PETSC ERROR: #3 KSPInitialResidual() line 61 in > /usr/local/petsc/petsc-3.12.5/src/ksp/ksp/interface/itres.c > [0]PETSC ERROR: #4 KSPSolve_GMRES() line 236 in > /usr/local/petsc/petsc-3.12.5/src/ksp/ksp/impls/gmres/gmres.c > [0]PETSC ERROR: #5 KSPSolve() line 760 in > /usr/local/petsc/petsc-3.12.5/src/ksp/ksp/interface/itfunc.c > > > Regards > JT > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From y.juntao at hotmail.com Thu Jul 16 04:43:01 2020 From: y.juntao at hotmail.com (Karl Yang) Date: Thu, 16 Jul 2020 17:43:01 +0800 Subject: [petsc-users] Errors encountered with KSPsolver with MATNEST and VECNEST In-Reply-To: References: Message-ID: Hi, Matt, Sorry for the confusion. The error shows as below without SetIS(). [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [0]PETSC ERROR: Arguments are incompatible [0]PETSC ERROR: Could not find index set [0]PETSC ERROR: See https://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [0]PETSC ERROR: Petsc Release Version 3.12.5, Mar, 29, 2020 [0]PETSC ERROR: ./testSolve on a arch-linux2-c-debug named ae3f0ab5a8c4 by Unknown Thu Jul 16 17:36:52 2020 [0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --download-mpich --download-fblaslapack --with-cuda [0]PETSC ERROR: #1 MatNestFindIS() line 365 in /usr/local/petsc/petsc-3.12.5/src/mat/impls/nest/matnest.c [0]PETSC ERROR: #2 MatNestFindSubMat() line 425 in /usr/local/petsc/petsc-3.12.5/src/mat/impls/nest/matnest.c [0]PETSC ERROR: #3 MatCreateSubMatrix_Nest() line 456 in /usr/local/petsc/petsc-3.12.5/src/mat/impls/nest/matnest.c [0]PETSC ERROR: #4 MatCreateSubMatrix() line 7859 in /usr/local/petsc/petsc-3.12.5/src/mat/interface/matrix.c [0]PETSC ERROR: #5 PCSetUp_FieldSplit() line 676 in /usr/local/petsc/petsc-3.12.5/src/ksp/pc/impls/fieldsplit/fieldsplit.c [0]PETSC ERROR: #6 PCSetUp() line 894 in /usr/local/petsc/petsc-3.12.5/src/ksp/pc/interface/precon.c [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [0]PETSC ERROR: Corrupt argument: https://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [0]PETSC ERROR: Invalid Pointer to Object: Parameter # 1 [0]PETSC ERROR: See https://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [0]PETSC ERROR: Petsc Release Version 3.12.5, Mar, 29, 2020 [0]PETSC ERROR: ./testSolve on a arch-linux2-c-debug named ae3f0ab5a8c4 by Unknown Thu Jul 16 17:36:52 2020 [0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --download-mpich --download-fblaslapack --with-cuda [0]PETSC ERROR: #7 MatDestroy() line 1266 in /usr/local/petsc/petsc-3.12.5/src/mat/interface/matrix.c [0]PETSC ERROR: #8 PCSetUp_FieldSplit() line 697 in /usr/local/petsc/petsc-3.12.5/src/ksp/pc/impls/fieldsplit/fieldsplit.c [0]PETSC ERROR: #9 PCSetUp() line 894 in /usr/local/petsc/petsc-3.12.5/src/ksp/pc/interface/precon.c [0]PETSC ERROR: #10 KSPSetUp() line 376 in /usr/local/petsc/petsc-3.12.5/src/ksp/ksp/interface/itfunc.c KSP Object: 1 MPI processes type: fgmres restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement happy breakdown tolerance 1e-30 maximum iterations=10000, initial guess is zero tolerances: relative=1e-05, absolute=1e-50, divergence=10000. right preconditioning using UNPRECONDITIONED norm type for convergence test PC Object: 1 MPI processes type: fieldsplit PC has not been set up so information may be incomplete FieldSplit with Schur preconditioner, blocksize = 1, factorization FULL Preconditioner for the Schur complement formed from S itself Split info: Split number 0 Defined by IS Split number 1 Defined by IS KSP solver for A00 block KSP Object: (fieldsplit_0_) 1 MPI processes type: preonly maximum iterations=10000, initial guess is zero tolerances: relative=1e-05, absolute=1e-50, divergence=10000. left preconditioning using DEFAULT norm type for convergence test PC Object: (fieldsplit_0_) 1 MPI processes type not yet set PC has not been set up so information may be incomplete KSP solver for upper A00 in upper triangular factor not yet available KSP solver for S = A11 - A10 inv(A00) A01 not yet available linear system matrix = precond matrix: Mat Object: 1 MPI processes type: nest rows=15, cols=15 Matrix object: type=nest, rows=3, cols=3 MatNest structure: (0,0) : type=seqaij, rows=5, cols=5 (0,1) : NULL (0,2) : type=seqaij, rows=5, cols=5 (1,0) : NULL (1,1) : type=seqaij, rows=5, cols=5 (1,2) : type=seqaij, rows=5, cols=5 (2,0) : type=seqaij, rows=5, cols=5 (2,1) : type=seqaij, rows=5, cols=5 (2,2) : NULL Segmentation fault (core dumped) Attached is the test code. Regards JT On Jul 16 2020, at 5:30 pm, Matthew Knepley wrote: > On Thu, Jul 16, 2020 at 1:23 AM Karl Yang wrote: > > > Hi, Junchao, > > > > Thank you. It works. :) > > May I ask some follow up question regarding the same example about fieldsplit. I am trying to learn about matnest with fieldsplit preconditioner on manual page 94. But I get a bit lost trying to understand fields, ISs once it is not eactly the same on the manual. > > In this same example, I tried to use PCFIELDSPLIT and PCFieldSplitSetDetectSaddlePoint as follows. > > You cannot mix DetectSaddlePoint() with SetIS() since the detection tries to make the proper ISes. Maybe we should check that. > > Thanks, > > Matt > > > IS rows[3]; > > IS cols[3]; > > MatNestGetISs(testMesh->system, rows, cols); > > > > KSP ksp; > > PC pc; > > KSPCreate(PETSC_COMM_WORLD, &ksp); > > KSPSetType(ksp, KSPFGMRES); > > KSPSetOperators(ksp, testMesh->system, testMesh->system); > > KSPGetPC(ksp, &pc); > > PCFieldSplitSetIS(pc, "u", rows[0]); > > PCFieldSplitSetIS(pc, "v", rows[1]); > > PCFieldSplitSetIS(pc, "p", rows[2]); > > PCSetType(pc, PCFIELDSPLIT); > > PCFieldSplitSetDetectSaddlePoint(pc, PETSC_TRUE); > > KSPSetUp(ksp); > > KSPSolve(ksp, testMesh->rhs, testMesh->solution); > > > > But I will get the following error, > > > > [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- > > [0]PETSC ERROR: Arguments are incompatible > > [0]PETSC ERROR: Could not find index set > > [0]PETSC ERROR: See https://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. > > [0]PETSC ERROR: Petsc Release Version 3.12.5, Mar, 29, 2020 > > [0]PETSC ERROR: ./testSolve on a arch-linux2-c-debug named f601294ac804 by Unknown Thu Jul 16 13:08:39 2020 > > [0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --download-mpich --download-fblaslapack --with-cuda > > [0]PETSC ERROR: #1 MatNestFindIS() line 365 in /usr/local/petsc/petsc-3.12.5/src/mat/impls/nest/matnest.c > > [0]PETSC ERROR: #2 MatNestFindSubMat() line 425 in /usr/local/petsc/petsc-3.12.5/src/mat/impls/nest/matnest.c > > [0]PETSC ERROR: #3 MatCreateSubMatrix_Nest() line 456 in /usr/local/petsc/petsc-3.12.5/src/mat/impls/nest/matnest.c > > [0]PETSC ERROR: #4 MatCreateSubMatrix() line 7859 in /usr/local/petsc/petsc-3.12.5/src/mat/interface/matrix.c > > [0]PETSC ERROR: #5 PCSetUp_FieldSplit() line 676 in /usr/local/petsc/petsc-3.12.5/src/ksp/pc/impls/fieldsplit/fieldsplit.c > > [0]PETSC ERROR: #6 PCSetUp() line 894 in /usr/local/petsc/petsc-3.12.5/src/ksp/pc/interface/precon.c > > [0]PETSC ERROR: #7 KSPSetUp() line 376 in /usr/local/petsc/petsc-3.12.5/src/ksp/ksp/interface/itfunc.c > > [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- > > [0]PETSC ERROR: Corrupt argument: https://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind > > [0]PETSC ERROR: Invalid Pointer to Object: Parameter # 1 > > [0]PETSC ERROR: See https://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. > > [0]PETSC ERROR: Petsc Release Version 3.12.5, Mar, 29, 2020 > > [0]PETSC ERROR: ./testSolve on a arch-linux2-c-debug named f601294ac804 by Unknown Thu Jul 16 13:08:39 2020 > > [0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --download-mpich --download-fblaslapack --with-cuda > > [0]PETSC ERROR: #8 MatDestroy() line 1266 in /usr/local/petsc/petsc-3.12.5/src/mat/interface/matrix.c > > [0]PETSC ERROR: #9 PCSetUp_FieldSplit() line 697 in /usr/local/petsc/petsc-3.12.5/src/ksp/pc/impls/fieldsplit/fieldsplit.c > > [0]PETSC ERROR: #10 PCSetUp() line 894 in /usr/local/petsc/petsc-3.12.5/src/ksp/pc/interface/precon.c > > [0]PETSC ERROR: #11 KSPSetUp() line 376 in /usr/local/petsc/petsc-3.12.5/src/ksp/ksp/interface/itfunc.c > > [0]PETSC ERROR: #12 KSPSolve() line 703 in /usr/local/petsc/petsc-3.12.5/src/ksp/ksp/interface/itfunc.c > > > > I read about the examples on the manual and mostly are based on 22 blocks but still cannot figure out how pcfieldsplit should be set in this 3 *3 blocks, and how IS should be set accordingly. I read about -fieldsplit_1_pc_type none option suggested on the documentation, but I am not sure is it still valid in a 33 blocks. > > Regards > > JT > > On Jul 16 2020, at 6:57 am, Junchao Zhang wrote: > > > Adding the MatNestSetVecType() line fixed the problem. But I don't know why petsc had this weird design originally. > > > > > > MatCreateNest(PETSC_COMM_WORLD, 3, NULL, 3, NULL, mblock, &testMesh->system); > > > + MatNestSetVecType(testMesh->system,VECNEST); > > > > > > --Junchao Zhang > > > > > > On Wed, Jul 15, 2020 at 2:16 PM Karl Yang wrote: > > > > Hi, Barry, > > > > > > > > I created a minimum example to reproduce the error. The example code is attached. > > > > Mat Object: 1 MPI processes > > > > type: nest > > > > Matrix object: > > > > type=nest, rows=3, cols=3 > > > > MatNest structure: > > > > (0,0) : type=seqaij, rows=5, cols=5 > > > > (0,1) : NULL > > > > (0,2) : type=seqaij, rows=5, cols=5 > > > > (1,0) : NULL > > > > (1,1) : type=seqaij, rows=5, cols=5 > > > > (1,2) : type=seqaij, rows=5, cols=5 > > > > (2,0) : type=seqaij, rows=5, cols=5 > > > > (2,1) : type=seqaij, rows=5, cols=5 > > > > (2,2) : NULL > > > > Vec Object: 1 MPI processes > > > > type: nest > > > > VecNest, rows=3, structure: > > > > (0) : type=seq, rows=5 > > > > Vec Object: 1 MPI processes > > > > type: seq > > > > 1. > > > > 1. > > > > 1. > > > > 1. > > > > 1. > > > > (1) : type=seq, rows=5 > > > > Vec Object: 1 MPI processes > > > > type: seq > > > > 1. > > > > 1. > > > > 1. > > > > 1. > > > > 1. > > > > (2) : type=seq, rows=5 > > > > Vec Object: 1 MPI processes > > > > type: seq > > > > 1. > > > > 1. > > > > 1. > > > > 1. > > > > 1. > > > > [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- > > > > [0]PETSC ERROR: Invalid argument > > > > [0]PETSC ERROR: Nest vector argument 2 not setup. > > > > [0]PETSC ERROR: See https://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. > > > > [0]PETSC ERROR: Petsc Release Version 3.12.5, Mar, 29, 2020 > > > > [0]PETSC ERROR: ./testSolve on a arch-linux2-c-debug named f601294ac804 by Unknown Thu Jul 16 03:11:02 2020 > > > > [0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --download-mpich --download-fblaslapack --with-cuda > > > > [0]PETSC ERROR: #1 VecCopy_Nest() line 68 in /usr/local/petsc/petsc-3.12.5/src/vec/vec/impls/nest/vecnest.c > > > > [0]PETSC ERROR: #2 VecCopy() line 1577 in /usr/local/petsc/petsc-3.12.5/src/vec/vec/interface/vector.c > > > > [0]PETSC ERROR: #3 KSPInitialResidual() line 61 in /usr/local/petsc/petsc-3.12.5/src/ksp/ksp/interface/itres.c > > > > [0]PETSC ERROR: #4 KSPSolve_GMRES() line 236 in /usr/local/petsc/petsc-3.12.5/src/ksp/ksp/impls/gmres/gmres.c > > > > [0]PETSC ERROR: #5 KSPSolve() line 760 in /usr/local/petsc/petsc-3.12.5/src/ksp/ksp/interface/itfunc.c > > > > On Jul 16 2020, at 2:39 am, Barry Smith wrote: > > > > > > > > > > Could be a bug in PETSc, please send a sample program that produces the problem. > > > > > > > > > > Barry > > > > > > > > > > > > > > > > On Jul 15, 2020, at 1:28 PM, Karl Yang wrote: > > > > > > Hi, Barry, > > > > > > Thanks for your reply. I output the matrix and vec with matview and vecview. And I think I do have the values set as it has been shown. > > > > > > Mat Object: 1 MPI processes > > > > > > type: nest > > > > > > Matrix object: > > > > > > type=nest, rows=3, cols=3 > > > > > > MatNest structure: > > > > > > (0,0) : type=seqaij, rows=25, cols=25 > > > > > > (0,1) : NULL > > > > > > (0,2) : type=seqaij, rows=25, cols=25 > > > > > > (1,0) : NULL > > > > > > (1,1) : type=seqaij, rows=25, cols=25 > > > > > > (1,2) : type=seqaij, rows=25, cols=25 > > > > > > (2,0) : type=seqaij, rows=25, cols=25 > > > > > > (2,1) : type=seqaij, rows=25, cols=25 > > > > > > (2,2) : NULL > > > > > > Vec Object: 1 MPI processes > > > > > > type: nest > > > > > > VecNest, rows=3, structure: > > > > > > (0) : type=seq, rows=25 > > > > > > Vec Object: 1 MPI processes > > > > > > type: seq > > > > > > 0. > > > > > > 0. > > > > > > 0. > > > > > > 0. > > > > > > 0. > > > > > > 0. > > > > > > 75.8279 > > > > > > -51.0776 > > > > > > -75.8279 > > > > > > 0. > > > > > > 0. > > > > > > -90.8404 > > > > > > 58.7011 > > > > > > 90.8404 > > > > > > 0. > > > > > > 0. > > > > > > -75.8279 > > > > > > 51.0776 > > > > > > 75.8279 > > > > > > 0. > > > > > > 0. > > > > > > 0. > > > > > > 0. > > > > > > 0. > > > > > > 0. > > > > > > (1) : name="Vec_0x84000000_0", type=seq, rows=25 > > > > > > Vec Object: Vec_0x84000000_0 1 MPI processes > > > > > > type: seq > > > > > > 0. > > > > > > 0. > > > > > > 0. > > > > > > 0. > > > > > > 0. > > > > > > 0. > > > > > > 75.8279 > > > > > > -51.0776 > > > > > > -75.8279 > > > > > > 0. > > > > > > 0. > > > > > > -90.8404 > > > > > > 58.7011 > > > > > > 90.8404 > > > > > > 0. > > > > > > 0. > > > > > > -75.8279 > > > > > > 51.0776 > > > > > > 75.8279 > > > > > > 0. > > > > > > 0. > > > > > > 0. > > > > > > 0. > > > > > > 0. > > > > > > 0. > > > > > > (2) : type=seq, rows=25 > > > > > > Vec Object: 1 MPI processes > > > > > > type: seq > > > > > > 0. > > > > > > 0. > > > > > > 0. > > > > > > 0. > > > > > > 0. > > > > > > 0. > > > > > > 0. > > > > > > 0. > > > > > > 0. > > > > > > 0. > > > > > > 0. > > > > > > 0. > > > > > > 0. > > > > > > 0. > > > > > > 0. > > > > > > 0. > > > > > > 0. > > > > > > 0. > > > > > > 0. > > > > > > 0. > > > > > > 0. > > > > > > 0. > > > > > > 0. > > > > > > 0. > > > > > > 0. > > > > > > [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- > > > > > > [0]PETSC ERROR: Invalid argument > > > > > > [0]PETSC ERROR: Nest vector argument 4 not setup. > > > > > > [0]PETSC ERROR: See https://www.mcs.anl.gov/petsc/documentation/faq.html (https://link.getmailspring.com/link/060AB2C3-F3D7-408C-99CF-C4D1B27DD59F at getmailspring.com/1?redirect=https%3A%2F%2Fwww.mcs.anl.gov%2Fpetsc%2Fdocumentation%2Ffaq.html&recipient=cGV0c2MtdXNlcnNAbWNzLmFubC5nb3Y%3D) for trouble shooting. > > > > > > [0]PETSC ERROR: Petsc Release Version 3.12.5, Mar, 29, 2020 > > > > > > [0]PETSC ERROR: ./testSolve on a arch-linux2-c-debug named f601294ac804 by Unknown Thu Jul 16 02:15:26 2020 > > > > > > [0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --download-mpich --download-fblaslapack --with-cuda > > > > > > [0]PETSC ERROR: #1 VecPointwiseMult_Nest() line 245 in /usr/local/petsc/petsc-3.12.5/src/vec/vec/impls/nest/vecnest.c > > > > > > [0]PETSC ERROR: #2 VecPointwiseMult() line 1107 in /usr/local/petsc/petsc-3.12.5/src/vec/vec/interface/vector.c > > > > > > [0]PETSC ERROR: #3 PCApply_Jacobi() line 272 in /usr/local/petsc/petsc-3.12.5/src/ksp/pc/impls/jacobi/jacobi.c > > > > > > [0]PETSC ERROR: #4 PCApply() line 444 in /usr/local/petsc/petsc-3.12.5/src/ksp/pc/interface/precon.c > > > > > > [0]PETSC ERROR: #5 KSP_PCApply() line 281 in /usr/local/petsc/petsc-3.12.5/include/petsc/private/kspimpl.h > > > > > > [0]PETSC ERROR: #6 KSPInitialResidual() line 65 in /usr/local/petsc/petsc-3.12.5/src/ksp/ksp/interface/itres.c > > > > > > [0]PETSC ERROR: #7 KSPSolve_GMRES() line 236 in /usr/local/petsc/petsc-3.12.5/src/ksp/ksp/impls/gmres/gmres.c > > > > > > [0]PETSC ERROR: #8 KSPSolve() line 760 in /usr/local/petsc/petsc-3.12.5/src/ksp/ksp/interface/itfunc.c > > > > > > > > > > > > On Jul 16 2020, at 2:13 am, Barry Smith wrote: > > > > > > > > > > > > > > From the error message the second vector you are using to form the nest vector has not been completely created/set up. Make sure you called either VecCreateSeq(), VecCreateMPI() or did VecCreate(), VecSetType(), VecSetUp() and if it is the right hand side make sure you put numerical values in it. > > > > > > > > > > > > > > Barry > > > > > > > > > > > > > > > > > > > > > > On Jul 15, 2020, at 1:05 PM, Karl Yang wrote: > > > > > > > > Hello, > > > > > > > > I'm trying to solve a 2*2 block matrix system. > > > > > > > > I managed to form a 2*2 MatNest and a 21 VecNest. However when I try to make use of ksp solve, I encounter the following error. There is only one or two examples on MatNest and VecNest. I could not figure out what could be the trouble. > > > > > > > > > > > > > > > > > > > > > > > > [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- > > > > > > > > [0]PETSC ERROR: Invalid argument > > > > > > > > [0]PETSC ERROR: Nest vector argument 2 not setup. > > > > > > > > [0]PETSC ERROR: See https://www.mcs.anl.gov/petsc/documentation/faq.html (https://link.getmailspring.com/link/060AB2C3-F3D7-408C-99CF-C4D1B27DD59F at getmailspring.com/4?redirect=https%3A%2F%2Flink.getmailspring.com%2Flink%2F540F752A-DB59-4878-9C98-B8F4B4D9FD0D%40getmailspring.com%2F1%3Fredirect%3Dhttps%253A%252F%252Fwww.mcs.anl.gov%252Fpetsc%252Fdocumentation%252Ffaq.html%26recipient%3DYnNtaXRoQHBldHNjLmRldg%253D%253D&recipient=cGV0c2MtdXNlcnNAbWNzLmFubC5nb3Y%3D) for trouble shooting. > > > > > > > > [0]PETSC ERROR: Petsc Release Version 3.12.5, Mar, 29, 2020 > > > > > > > > [0]PETSC ERROR: ./testSolve on a arch-linux2-c-debug named f601294ac804 by Unknown Thu Jul 16 01:56:11 2020 > > > > > > > > [0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --download-mpich --download-fblaslapack --with-cuda > > > > > > > > [0]PETSC ERROR: #1 VecCopy_Nest() line 68 in /usr/local/petsc/petsc-3.12.5/src/vec/vec/impls/nest/vecnest.c > > > > > > > > [0]PETSC ERROR: #2 VecCopy() line 1577 in /usr/local/petsc/petsc-3.12.5/src/vec/vec/interface/vector.c > > > > > > > > [0]PETSC ERROR: #3 KSPInitialResidual() line 61 in /usr/local/petsc/petsc-3.12.5/src/ksp/ksp/interface/itres.c > > > > > > > > [0]PETSC ERROR: #4 KSPSolve_GMRES() line 236 in /usr/local/petsc/petsc-3.12.5/src/ksp/ksp/impls/gmres/gmres.c > > > > > > > > [0]PETSC ERROR: #5 KSPSolve() line 760 in /usr/local/petsc/petsc-3.12.5/src/ksp/ksp/interface/itfunc.c > > > > > > > > > > > > > > > > > > > > > > > > Regards > > > > > > > > JT > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. > -- Norbert Wiener > > > https://www.cse.buffalo.edu/~knepley/ (https://link.getmailspring.com/link/1D2D2990-B215-4510-B196-F29BD06D4684 at getmailspring.com/1?redirect=http%3A%2F%2Fwww.cse.buffalo.edu%2F~knepley%2F&recipient=cGV0c2MtdXNlcnNAbWNzLmFubC5nb3Y%3D) -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: test.cpp Type: application/octet-stream Size: 4178 bytes Desc: not available URL: From knepley at gmail.com Thu Jul 16 04:55:37 2020 From: knepley at gmail.com (Matthew Knepley) Date: Thu, 16 Jul 2020 05:55:37 -0400 Subject: [petsc-users] Errors encountered with KSPsolver with MATNEST and VECNEST In-Reply-To: References: Message-ID: On Thu, Jul 16, 2020 at 5:43 AM Karl Yang wrote: > Hi, Matt, > > Sorry for the confusion. The error shows as below without SetIS(). > I see the problem now. It is about the interface for MatNest. MatNest is intended _only_ as an optimization. I think you should never have MatNest in user code. Here the problem is that MatNest only supports extraction using the ISes it was created with, whereas detection is finding a different IS. I recommend getting everything working with AIJ, and then use MatSetType() or -mat_type at the end to try out Nest. The interfaces are the same. Thanks, Matt > [0]PETSC ERROR: --------------------- Error Message > -------------------------------------------------------------- > [0]PETSC ERROR: Arguments are incompatible > [0]PETSC ERROR: Could not find index set > [0]PETSC ERROR: See https://www.mcs.anl.gov/petsc/documentation/faq.html > for trouble shooting. > [0]PETSC ERROR: Petsc Release Version 3.12.5, Mar, 29, 2020 > [0]PETSC ERROR: ./testSolve on a arch-linux2-c-debug named ae3f0ab5a8c4 by > Unknown Thu Jul 16 17:36:52 2020 > [0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ > --with-fc=gfortran --download-mpich --download-fblaslapack --with-cuda > [0]PETSC ERROR: #1 MatNestFindIS() line 365 in > /usr/local/petsc/petsc-3.12.5/src/mat/impls/nest/matnest.c > [0]PETSC ERROR: #2 MatNestFindSubMat() line 425 in > /usr/local/petsc/petsc-3.12.5/src/mat/impls/nest/matnest.c > [0]PETSC ERROR: #3 MatCreateSubMatrix_Nest() line 456 in > /usr/local/petsc/petsc-3.12.5/src/mat/impls/nest/matnest.c > [0]PETSC ERROR: #4 MatCreateSubMatrix() line 7859 in > /usr/local/petsc/petsc-3.12.5/src/mat/interface/matrix.c > [0]PETSC ERROR: #5 PCSetUp_FieldSplit() line 676 in > /usr/local/petsc/petsc-3.12.5/src/ksp/pc/impls/fieldsplit/fieldsplit.c > [0]PETSC ERROR: #6 PCSetUp() line 894 in > /usr/local/petsc/petsc-3.12.5/src/ksp/pc/interface/precon.c > [0]PETSC ERROR: --------------------- Error Message > -------------------------------------------------------------- > [0]PETSC ERROR: Corrupt argument: > https://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind > [0]PETSC ERROR: Invalid Pointer to Object: Parameter # 1 > [0]PETSC ERROR: See https://www.mcs.anl.gov/petsc/documentation/faq.html > for trouble shooting. > [0]PETSC ERROR: Petsc Release Version 3.12.5, Mar, 29, 2020 > [0]PETSC ERROR: ./testSolve on a arch-linux2-c-debug named ae3f0ab5a8c4 by > Unknown Thu Jul 16 17:36:52 2020 > [0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ > --with-fc=gfortran --download-mpich --download-fblaslapack --with-cuda > [0]PETSC ERROR: #7 MatDestroy() line 1266 in > /usr/local/petsc/petsc-3.12.5/src/mat/interface/matrix.c > [0]PETSC ERROR: #8 PCSetUp_FieldSplit() line 697 in > /usr/local/petsc/petsc-3.12.5/src/ksp/pc/impls/fieldsplit/fieldsplit.c > [0]PETSC ERROR: #9 PCSetUp() line 894 in > /usr/local/petsc/petsc-3.12.5/src/ksp/pc/interface/precon.c > [0]PETSC ERROR: #10 KSPSetUp() line 376 in > /usr/local/petsc/petsc-3.12.5/src/ksp/ksp/interface/itfunc.c > KSP Object: 1 MPI processes > type: fgmres > restart=30, using Classical (unmodified) Gram-Schmidt > Orthogonalization with no iterative refinement > happy breakdown tolerance 1e-30 > maximum iterations=10000, initial guess is zero > tolerances: relative=1e-05, absolute=1e-50, divergence=10000. > right preconditioning > using UNPRECONDITIONED norm type for convergence test > PC Object: 1 MPI processes > type: fieldsplit > PC has not been set up so information may be incomplete > FieldSplit with Schur preconditioner, blocksize = 1, factorization FULL > Preconditioner for the Schur complement formed from S itself > Split info: > Split number 0 Defined by IS > Split number 1 Defined by IS > KSP solver for A00 block > KSP Object: (fieldsplit_0_) 1 MPI processes > type: preonly > maximum iterations=10000, initial guess is zero > tolerances: relative=1e-05, absolute=1e-50, divergence=10000. > left preconditioning > using DEFAULT norm type for convergence test > PC Object: (fieldsplit_0_) 1 MPI processes > type not yet set > PC has not been set up so information may be incomplete > KSP solver for upper A00 in upper triangular factor > not yet available > KSP solver for S = A11 - A10 inv(A00) A01 > not yet available > linear system matrix = precond matrix: > Mat Object: 1 MPI processes > type: nest > rows=15, cols=15 > Matrix object: > type=nest, rows=3, cols=3 > MatNest structure: > (0,0) : type=seqaij, rows=5, cols=5 > (0,1) : NULL > (0,2) : type=seqaij, rows=5, cols=5 > (1,0) : NULL > (1,1) : type=seqaij, rows=5, cols=5 > (1,2) : type=seqaij, rows=5, cols=5 > (2,0) : type=seqaij, rows=5, cols=5 > (2,1) : type=seqaij, rows=5, cols=5 > (2,2) : NULL > Segmentation fault (core dumped) > > > Attached is the test code. > > Regards > JT > > > > > On Jul 16 2020, at 5:30 pm, Matthew Knepley wrote: > > On Thu, Jul 16, 2020 at 1:23 AM Karl Yang > > wrote: > [image: Sent from Mailspring] > > Hi, Junchao, > > Thank you. It works. :) > > May I ask some follow up question regarding the same example about > fieldsplit. I am trying to learn about matnest with fieldsplit > preconditioner on manual page 94. But I get a bit lost trying to understand > fields, ISs once it is not eactly the same on the manual. > > In this same example, I tried to use PCFIELDSPLIT and > PCFieldSplitSetDetectSaddlePoint as follows. > > > You cannot mix DetectSaddlePoint() with SetIS() since the detection tries > to make the proper ISes. Maybe we should check that. > > Thanks, > > Matt > > > IS rows[3]; > IS cols[3]; > MatNestGetISs(testMesh->system, rows, cols); > > KSP ksp; > PC pc; > KSPCreate(PETSC_COMM_WORLD, &ksp); > KSPSetType(ksp, KSPFGMRES); > KSPSetOperators(ksp, testMesh->system, testMesh->system); > KSPGetPC(ksp, &pc); > PCFieldSplitSetIS(pc, "u", rows[0]); > PCFieldSplitSetIS(pc, "v", rows[1]); > PCFieldSplitSetIS(pc, "p", rows[2]); > PCSetType(pc, PCFIELDSPLIT); > PCFieldSplitSetDetectSaddlePoint(pc, PETSC_TRUE); > KSPSetUp(ksp); > KSPSolve(ksp, testMesh->rhs, testMesh->solution); > > But I will get the following error, > > [0]PETSC ERROR: --------------------- Error Message > -------------------------------------------------------------- > [0]PETSC ERROR: Arguments are incompatible > [0]PETSC ERROR: Could not find index set > [0]PETSC ERROR: See https://www.mcs.anl.gov/petsc/documentation/faq.html for > trouble shooting. > [0]PETSC ERROR: Petsc Release Version 3.12.5, Mar, 29, 2020 > [0]PETSC ERROR: ./testSolve on a arch-linux2-c-debug named f601294ac804 by > Unknown Thu Jul 16 13:08:39 2020 > [0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ > --with-fc=gfortran --download-mpich --download-fblaslapack --with-cuda > [0]PETSC ERROR: #1 MatNestFindIS() line 365 in > /usr/local/petsc/petsc-3.12.5/src/mat/impls/nest/matnest.c > [0]PETSC ERROR: #2 MatNestFindSubMat() line 425 in > /usr/local/petsc/petsc-3.12.5/src/mat/impls/nest/matnest.c > [0]PETSC ERROR: #3 MatCreateSubMatrix_Nest() line 456 in > /usr/local/petsc/petsc-3.12.5/src/mat/impls/nest/matnest.c > [0]PETSC ERROR: #4 MatCreateSubMatrix() line 7859 in > /usr/local/petsc/petsc-3.12.5/src/mat/interface/matrix.c > [0]PETSC ERROR: #5 PCSetUp_FieldSplit() line 676 in > /usr/local/petsc/petsc-3.12.5/src/ksp/pc/impls/fieldsplit/fieldsplit.c > [0]PETSC ERROR: #6 PCSetUp() line 894 in > /usr/local/petsc/petsc-3.12.5/src/ksp/pc/interface/precon.c > [0]PETSC ERROR: #7 KSPSetUp() line 376 in > /usr/local/petsc/petsc-3.12.5/src/ksp/ksp/interface/itfunc.c > [0]PETSC ERROR: --------------------- Error Message > -------------------------------------------------------------- > [0]PETSC ERROR: Corrupt argument: > https://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind > [0]PETSC ERROR: Invalid Pointer to Object: Parameter # 1 > [0]PETSC ERROR: See https://www.mcs.anl.gov/petsc/documentation/faq.html for > trouble shooting. > [0]PETSC ERROR: Petsc Release Version 3.12.5, Mar, 29, 2020 > [0]PETSC ERROR: ./testSolve on a arch-linux2-c-debug named f601294ac804 by > Unknown Thu Jul 16 13:08:39 2020 > [0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ > --with-fc=gfortran --download-mpich --download-fblaslapack --with-cuda > [0]PETSC ERROR: #8 MatDestroy() line 1266 in > /usr/local/petsc/petsc-3.12.5/src/mat/interface/matrix.c > [0]PETSC ERROR: #9 PCSetUp_FieldSplit() line 697 in > /usr/local/petsc/petsc-3.12.5/src/ksp/pc/impls/fieldsplit/fieldsplit.c > [0]PETSC ERROR: #10 PCSetUp() line 894 in > /usr/local/petsc/petsc-3.12.5/src/ksp/pc/interface/precon.c > [0]PETSC ERROR: #11 KSPSetUp() line 376 in > /usr/local/petsc/petsc-3.12.5/src/ksp/ksp/interface/itfunc.c > [0]PETSC ERROR: #12 KSPSolve() line 703 in > /usr/local/petsc/petsc-3.12.5/src/ksp/ksp/interface/itfunc.c > > I read about the examples on the manual and mostly are based on 2*2 > blocks but still cannot figure out how pcfieldsplit should be set in this 3* > **3 blocks, and how IS should be set accordingly. I read about > -fieldsplit_1_pc_type none option suggested on the documentation, but I am > not sure is it still valid in a 33 blocks.* > > *Regards* > *JT* > On Jul 16 2020, at 6:57 am, Junchao Zhang wrote: > > Adding the MatNestSetVecType() line fixed the problem. But I don't know > why petsc had this weird design originally. > > MatCreateNest(PETSC_COMM_WORLD, 3, NULL, 3, NULL, mblock, > &testMesh->system); > + MatNestSetVecType(testMesh->system,VECNEST); > > --Junchao Zhang > > > On Wed, Jul 15, 2020 at 2:16 PM Karl Yang wrote: > > Hi, Barry, > > I created a minimum example to reproduce the error. The example code is > attached. > > Mat Object: 1 MPI processes > type: nest > Matrix object: > type=nest, rows=3, cols=3 > MatNest structure: > (0,0) : type=seqaij, rows=5, cols=5 > (0,1) : NULL > (0,2) : type=seqaij, rows=5, cols=5 > (1,0) : NULL > (1,1) : type=seqaij, rows=5, cols=5 > (1,2) : type=seqaij, rows=5, cols=5 > (2,0) : type=seqaij, rows=5, cols=5 > (2,1) : type=seqaij, rows=5, cols=5 > (2,2) : NULL > Vec Object: 1 MPI processes > type: nest > VecNest, rows=3, structure: > (0) : type=seq, rows=5 > Vec Object: 1 MPI processes > type: seq > 1. > 1. > 1. > 1. > 1. > (1) : type=seq, rows=5 > Vec Object: 1 MPI processes > type: seq > 1. > 1. > 1. > 1. > 1. > (2) : type=seq, rows=5 > Vec Object: 1 MPI processes > type: seq > 1. > 1. > 1. > 1. > 1. > [0]PETSC ERROR: --------------------- Error Message > -------------------------------------------------------------- > [0]PETSC ERROR: Invalid argument > [0]PETSC ERROR: Nest vector argument 2 not setup. > [0]PETSC ERROR: See https://www.mcs.anl.gov/petsc/documentation/faq.html for > trouble shooting. > [0]PETSC ERROR: Petsc Release Version 3.12.5, Mar, 29, 2020 > [0]PETSC ERROR: ./testSolve on a arch-linux2-c-debug named f601294ac804 by > Unknown Thu Jul 16 03:11:02 2020 > [0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ > --with-fc=gfortran --download-mpich --download-fblaslapack --with-cuda > [0]PETSC ERROR: #1 VecCopy_Nest() line 68 in > /usr/local/petsc/petsc-3.12.5/src/vec/vec/impls/nest/vecnest.c > [0]PETSC ERROR: #2 VecCopy() line 1577 in > /usr/local/petsc/petsc-3.12.5/src/vec/vec/interface/vector.c > [0]PETSC ERROR: #3 KSPInitialResidual() line 61 in > /usr/local/petsc/petsc-3.12.5/src/ksp/ksp/interface/itres.c > [0]PETSC ERROR: #4 KSPSolve_GMRES() line 236 in > /usr/local/petsc/petsc-3.12.5/src/ksp/ksp/impls/gmres/gmres.c > [0]PETSC ERROR: #5 KSPSolve() line 760 in > /usr/local/petsc/petsc-3.12.5/src/ksp/ksp/interface/itfunc.c > On Jul 16 2020, at 2:39 am, Barry Smith wrote: > > > Could be a bug in PETSc, please send a sample program that produces the > problem. > > Barry > > > On Jul 15, 2020, at 1:28 PM, Karl Yang > > wrote: > > Hi, Barry, > > Thanks for your reply. I output the matrix and vec with matview and > vecview. And I think I do have the values set as it has been shown. > > Mat Object: 1 MPI processes > type: nest > Matrix object: > type=nest, rows=3, cols=3 > MatNest structure: > (0,0) : type=seqaij, rows=25, cols=25 > (0,1) : NULL > (0,2) : type=seqaij, rows=25, cols=25 > (1,0) : NULL > (1,1) : type=seqaij, rows=25, cols=25 > (1,2) : type=seqaij, rows=25, cols=25 > (2,0) : type=seqaij, rows=25, cols=25 > (2,1) : type=seqaij, rows=25, cols=25 > (2,2) : NULL > Vec Object: 1 MPI processes > type: nest > VecNest, rows=3, structure: > (0) : type=seq, rows=25 > Vec Object: 1 MPI processes > type: seq > 0. > 0. > 0. > 0. > 0. > 0. > 75.8279 > -51.0776 > -75.8279 > 0. > 0. > -90.8404 > 58.7011 > 90.8404 > 0. > 0. > -75.8279 > 51.0776 > 75.8279 > 0. > 0. > 0. > 0. > 0. > 0. > (1) : name="Vec_0x84000000_0", type=seq, rows=25 > Vec Object: Vec_0x84000000_0 1 MPI processes > type: seq > 0. > 0. > 0. > 0. > 0. > 0. > 75.8279 > -51.0776 > -75.8279 > 0. > 0. > -90.8404 > 58.7011 > 90.8404 > 0. > 0. > -75.8279 > 51.0776 > 75.8279 > 0. > 0. > 0. > 0. > 0. > 0. > (2) : type=seq, rows=25 > Vec Object: 1 MPI processes > type: seq > 0. > 0. > 0. > 0. > 0. > 0. > 0. > 0. > 0. > 0. > 0. > 0. > 0. > 0. > 0. > 0. > 0. > 0. > 0. > 0. > 0. > 0. > 0. > 0. > 0. > [0]PETSC ERROR: --------------------- Error Message > -------------------------------------------------------------- > [0]PETSC ERROR: Invalid argument > [0]PETSC ERROR: Nest vector argument 4 not setup. > [0]PETSC ERROR: See https://www.mcs.anl.gov/petsc/documentation/faq.html > for > trouble shooting. > [0]PETSC ERROR: Petsc Release Version 3.12.5, Mar, 29, 2020 > [0]PETSC ERROR: ./testSolve on a arch-linux2-c-debug named f601294ac804 by > Unknown Thu Jul 16 02:15:26 2020 > [0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ > --with-fc=gfortran --download-mpich --download-fblaslapack --with-cuda > [0]PETSC ERROR: #1 VecPointwiseMult_Nest() line 245 in > /usr/local/petsc/petsc-3.12.5/src/vec/vec/impls/nest/vecnest.c > [0]PETSC ERROR: #2 VecPointwiseMult() line 1107 in > /usr/local/petsc/petsc-3.12.5/src/vec/vec/interface/vector.c > [0]PETSC ERROR: #3 PCApply_Jacobi() line 272 in > /usr/local/petsc/petsc-3.12.5/src/ksp/pc/impls/jacobi/jacobi.c > [0]PETSC ERROR: #4 PCApply() line 444 in > /usr/local/petsc/petsc-3.12.5/src/ksp/pc/interface/precon.c > [0]PETSC ERROR: #5 KSP_PCApply() line 281 in > /usr/local/petsc/petsc-3.12.5/include/petsc/private/kspimpl.h > [0]PETSC ERROR: #6 KSPInitialResidual() line 65 in > /usr/local/petsc/petsc-3.12.5/src/ksp/ksp/interface/itres.c > [0]PETSC ERROR: #7 KSPSolve_GMRES() line 236 in > /usr/local/petsc/petsc-3.12.5/src/ksp/ksp/impls/gmres/gmres.c > [0]PETSC ERROR: #8 KSPSolve() line 760 in > /usr/local/petsc/petsc-3.12.5/src/ksp/ksp/interface/itfunc.c > > On Jul 16 2020, at 2:13 am, Barry Smith > > wrote: > > > From the error message the second vector you are using to form the nest > vector has not been completely created/set up. Make sure you called either > VecCreateSeq(), VecCreateMPI() or did VecCreate(), VecSetType(), VecSetUp() > and if it is the right hand side make sure you put numerical values in it. > > Barry > > > On Jul 15, 2020, at 1:05 PM, Karl Yang > > wrote: > > Hello, > > I'm trying to solve a 2*2 block matrix system. > I managed to form a 2*2 MatNest and a 21 VecNest. However when I try to > make use of ksp solve, I encounter the following error. There is only one > or two examples on MatNest and VecNest. I could not figure out what could > be the trouble. > > > [0]PETSC ERROR: --------------------- Error Message > -------------------------------------------------------------- > [0]PETSC ERROR: Invalid argument > [0]PETSC ERROR: Nest vector argument 2 not setup. > [0]PETSC ERROR: See https://www.mcs.anl.gov/petsc/documentation/faq.html > for > trouble shooting. > [0]PETSC ERROR: Petsc Release Version 3.12.5, Mar, 29, 2020 > [0]PETSC ERROR: ./testSolve on a arch-linux2-c-debug named f601294ac804 by > Unknown Thu Jul 16 01:56:11 2020 > [0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ > --with-fc=gfortran --download-mpich --download-fblaslapack --with-cuda > [0]PETSC ERROR: #1 VecCopy_Nest() line 68 in > /usr/local/petsc/petsc-3.12.5/src/vec/vec/impls/nest/vecnest.c > [0]PETSC ERROR: #2 VecCopy() line 1577 in > /usr/local/petsc/petsc-3.12.5/src/vec/vec/interface/vector.c > [0]PETSC ERROR: #3 KSPInitialResidual() line 61 in > /usr/local/petsc/petsc-3.12.5/src/ksp/ksp/interface/itres.c > [0]PETSC ERROR: #4 KSPSolve_GMRES() line 236 in > /usr/local/petsc/petsc-3.12.5/src/ksp/ksp/impls/gmres/gmres.c > [0]PETSC ERROR: #5 KSPSolve() line 760 in > /usr/local/petsc/petsc-3.12.5/src/ksp/ksp/interface/itfunc.c > > > Regards > JT > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From Y.Juntao at hotmail.com Thu Jul 16 05:53:57 2020 From: Y.Juntao at hotmail.com (Yang Juntao) Date: Thu, 16 Jul 2020 10:53:57 +0000 Subject: [petsc-users] Errors encountered with KSPsolver with MATNEST and VECNEST In-Reply-To: References: Message-ID: Hi, Matt, In case, I want to put blocks of matrices(assembled independently) together as the system matrix just as in the example. What would be your recommended approach? I first looked into MatCreateBlockMat, but it only supports sequential and then I was directed to MATNEST. Do I have to assemble directly on ?system matrix?. ? Matrices of this type are nominally-sparse matrices in which each "entry" is a Mat object. Each Mat must have the same size and be sequential. The local and global sizes must be compatible with this decomposition. For matrices containing parallel submatrices and variable block sizes, see MATNEST. ? Regards Juntao From: Matthew Knepley Sent: Thursday, July 16, 2020 5:56 PM To: Karl Yang Cc: Junchao Zhang ; petsc-users at mcs.anl.gov Subject: Re: [petsc-users] Errors encountered with KSPsolver with MATNEST and VECNEST On Thu, Jul 16, 2020 at 5:43 AM Karl Yang > wrote: Hi, Matt, Sorry for the confusion. The error shows as below without SetIS(). I see the problem now. It is about the interface for MatNest. MatNest is intended _only_ as an optimization. I think you should never have MatNest in user code. Here the problem is that MatNest only supports extraction using the ISes it was created with, whereas detection is finding a different IS. I recommend getting everything working with AIJ, and then use MatSetType() or -mat_type at the end to try out Nest. The interfaces are the same. Thanks, Matt [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [0]PETSC ERROR: Arguments are incompatible [0]PETSC ERROR: Could not find index set [0]PETSC ERROR: See https://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [0]PETSC ERROR: Petsc Release Version 3.12.5, Mar, 29, 2020 [0]PETSC ERROR: ./testSolve on a arch-linux2-c-debug named ae3f0ab5a8c4 by Unknown Thu Jul 16 17:36:52 2020 [0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --download-mpich --download-fblaslapack --with-cuda [0]PETSC ERROR: #1 MatNestFindIS() line 365 in /usr/local/petsc/petsc-3.12.5/src/mat/impls/nest/matnest.c [0]PETSC ERROR: #2 MatNestFindSubMat() line 425 in /usr/local/petsc/petsc-3.12.5/src/mat/impls/nest/matnest.c [0]PETSC ERROR: #3 MatCreateSubMatrix_Nest() line 456 in /usr/local/petsc/petsc-3.12.5/src/mat/impls/nest/matnest.c [0]PETSC ERROR: #4 MatCreateSubMatrix() line 7859 in /usr/local/petsc/petsc-3.12.5/src/mat/interface/matrix.c [0]PETSC ERROR: #5 PCSetUp_FieldSplit() line 676 in /usr/local/petsc/petsc-3.12.5/src/ksp/pc/impls/fieldsplit/fieldsplit.c [0]PETSC ERROR: #6 PCSetUp() line 894 in /usr/local/petsc/petsc-3.12.5/src/ksp/pc/interface/precon.c [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [0]PETSC ERROR: Corrupt argument: https://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [0]PETSC ERROR: Invalid Pointer to Object: Parameter # 1 [0]PETSC ERROR: See https://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [0]PETSC ERROR: Petsc Release Version 3.12.5, Mar, 29, 2020 [0]PETSC ERROR: ./testSolve on a arch-linux2-c-debug named ae3f0ab5a8c4 by Unknown Thu Jul 16 17:36:52 2020 [0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --download-mpich --download-fblaslapack --with-cuda [0]PETSC ERROR: #7 MatDestroy() line 1266 in /usr/local/petsc/petsc-3.12.5/src/mat/interface/matrix.c [0]PETSC ERROR: #8 PCSetUp_FieldSplit() line 697 in /usr/local/petsc/petsc-3.12.5/src/ksp/pc/impls/fieldsplit/fieldsplit.c [0]PETSC ERROR: #9 PCSetUp() line 894 in /usr/local/petsc/petsc-3.12.5/src/ksp/pc/interface/precon.c [0]PETSC ERROR: #10 KSPSetUp() line 376 in /usr/local/petsc/petsc-3.12.5/src/ksp/ksp/interface/itfunc.c KSP Object: 1 MPI processes type: fgmres restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement happy breakdown tolerance 1e-30 maximum iterations=10000, initial guess is zero tolerances: relative=1e-05, absolute=1e-50, divergence=10000. right preconditioning using UNPRECONDITIONED norm type for convergence test PC Object: 1 MPI processes type: fieldsplit PC has not been set up so information may be incomplete FieldSplit with Schur preconditioner, blocksize = 1, factorization FULL Preconditioner for the Schur complement formed from S itself Split info: Split number 0 Defined by IS Split number 1 Defined by IS KSP solver for A00 block KSP Object: (fieldsplit_0_) 1 MPI processes type: preonly maximum iterations=10000, initial guess is zero tolerances: relative=1e-05, absolute=1e-50, divergence=10000. left preconditioning using DEFAULT norm type for convergence test PC Object: (fieldsplit_0_) 1 MPI processes type not yet set PC has not been set up so information may be incomplete KSP solver for upper A00 in upper triangular factor not yet available KSP solver for S = A11 - A10 inv(A00) A01 not yet available linear system matrix = precond matrix: Mat Object: 1 MPI processes type: nest rows=15, cols=15 Matrix object: type=nest, rows=3, cols=3 MatNest structure: (0,0) : type=seqaij, rows=5, cols=5 (0,1) : NULL (0,2) : type=seqaij, rows=5, cols=5 (1,0) : NULL (1,1) : type=seqaij, rows=5, cols=5 (1,2) : type=seqaij, rows=5, cols=5 (2,0) : type=seqaij, rows=5, cols=5 (2,1) : type=seqaij, rows=5, cols=5 (2,2) : NULL Segmentation fault (core dumped) Attached is the test code. Regards JT On Jul 16 2020, at 5:30 pm, Matthew Knepley > wrote: On Thu, Jul 16, 2020 at 1:23 AM Karl Yang > wrote: Hi, Junchao, Thank you. It works. :) May I ask some follow up question regarding the same example about fieldsplit. I am trying to learn about matnest with fieldsplit preconditioner on manual page 94. But I get a bit lost trying to understand fields, ISs once it is not eactly the same on the manual. In this same example, I tried to use PCFIELDSPLIT and PCFieldSplitSetDetectSaddlePoint as follows. You cannot mix DetectSaddlePoint() with SetIS() since the detection tries to make the proper ISes. Maybe we should check that. Thanks, Matt IS rows[3]; IS cols[3]; MatNestGetISs(testMesh->system, rows, cols); KSP ksp; PC pc; KSPCreate(PETSC_COMM_WORLD, &ksp); KSPSetType(ksp, KSPFGMRES); KSPSetOperators(ksp, testMesh->system, testMesh->system); KSPGetPC(ksp, &pc); PCFieldSplitSetIS(pc, "u", rows[0]); PCFieldSplitSetIS(pc, "v", rows[1]); PCFieldSplitSetIS(pc, "p", rows[2]); PCSetType(pc, PCFIELDSPLIT); PCFieldSplitSetDetectSaddlePoint(pc, PETSC_TRUE); KSPSetUp(ksp); KSPSolve(ksp, testMesh->rhs, testMesh->solution); But I will get the following error, [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [0]PETSC ERROR: Arguments are incompatible [0]PETSC ERROR: Could not find index set [0]PETSC ERROR: See https://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [0]PETSC ERROR: Petsc Release Version 3.12.5, Mar, 29, 2020 [0]PETSC ERROR: ./testSolve on a arch-linux2-c-debug named f601294ac804 by Unknown Thu Jul 16 13:08:39 2020 [0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --download-mpich --download-fblaslapack --with-cuda [0]PETSC ERROR: #1 MatNestFindIS() line 365 in /usr/local/petsc/petsc-3.12.5/src/mat/impls/nest/matnest.c [0]PETSC ERROR: #2 MatNestFindSubMat() line 425 in /usr/local/petsc/petsc-3.12.5/src/mat/impls/nest/matnest.c [0]PETSC ERROR: #3 MatCreateSubMatrix_Nest() line 456 in /usr/local/petsc/petsc-3.12.5/src/mat/impls/nest/matnest.c [0]PETSC ERROR: #4 MatCreateSubMatrix() line 7859 in /usr/local/petsc/petsc-3.12.5/src/mat/interface/matrix.c [0]PETSC ERROR: #5 PCSetUp_FieldSplit() line 676 in /usr/local/petsc/petsc-3.12.5/src/ksp/pc/impls/fieldsplit/fieldsplit.c [0]PETSC ERROR: #6 PCSetUp() line 894 in /usr/local/petsc/petsc-3.12.5/src/ksp/pc/interface/precon.c [0]PETSC ERROR: #7 KSPSetUp() line 376 in /usr/local/petsc/petsc-3.12.5/src/ksp/ksp/interface/itfunc.c [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [0]PETSC ERROR: Corrupt argument: https://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [0]PETSC ERROR: Invalid Pointer to Object: Parameter # 1 [0]PETSC ERROR: See https://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [0]PETSC ERROR: Petsc Release Version 3.12.5, Mar, 29, 2020 [0]PETSC ERROR: ./testSolve on a arch-linux2-c-debug named f601294ac804 by Unknown Thu Jul 16 13:08:39 2020 [0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --download-mpich --download-fblaslapack --with-cuda [0]PETSC ERROR: #8 MatDestroy() line 1266 in /usr/local/petsc/petsc-3.12.5/src/mat/interface/matrix.c [0]PETSC ERROR: #9 PCSetUp_FieldSplit() line 697 in /usr/local/petsc/petsc-3.12.5/src/ksp/pc/impls/fieldsplit/fieldsplit.c [0]PETSC ERROR: #10 PCSetUp() line 894 in /usr/local/petsc/petsc-3.12.5/src/ksp/pc/interface/precon.c [0]PETSC ERROR: #11 KSPSetUp() line 376 in /usr/local/petsc/petsc-3.12.5/src/ksp/ksp/interface/itfunc.c [0]PETSC ERROR: #12 KSPSolve() line 703 in /usr/local/petsc/petsc-3.12.5/src/ksp/ksp/interface/itfunc.c I read about the examples on the manual and mostly are based on 22 blocks but still cannot figure out how pcfieldsplit should be set in this 3 *3 blocks, and how IS should be set accordingly. I read about -fieldsplit_1_pc_type none option suggested on the documentation, but I am not sure is it still valid in a 33 blocks. Regards JT On Jul 16 2020, at 6:57 am, Junchao Zhang > wrote: Adding the MatNestSetVecType() line fixed the problem. But I don't know why petsc had this weird design originally. MatCreateNest(PETSC_COMM_WORLD, 3, NULL, 3, NULL, mblock, &testMesh->system); + MatNestSetVecType(testMesh->system,VECNEST); --Junchao Zhang On Wed, Jul 15, 2020 at 2:16 PM Karl Yang > wrote: Hi, Barry, I created a minimum example to reproduce the error. The example code is attached. Mat Object: 1 MPI processes type: nest Matrix object: type=nest, rows=3, cols=3 MatNest structure: (0,0) : type=seqaij, rows=5, cols=5 (0,1) : NULL (0,2) : type=seqaij, rows=5, cols=5 (1,0) : NULL (1,1) : type=seqaij, rows=5, cols=5 (1,2) : type=seqaij, rows=5, cols=5 (2,0) : type=seqaij, rows=5, cols=5 (2,1) : type=seqaij, rows=5, cols=5 (2,2) : NULL Vec Object: 1 MPI processes type: nest VecNest, rows=3, structure: (0) : type=seq, rows=5 Vec Object: 1 MPI processes type: seq 1. 1. 1. 1. 1. (1) : type=seq, rows=5 Vec Object: 1 MPI processes type: seq 1. 1. 1. 1. 1. (2) : type=seq, rows=5 Vec Object: 1 MPI processes type: seq 1. 1. 1. 1. 1. [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [0]PETSC ERROR: Invalid argument [0]PETSC ERROR: Nest vector argument 2 not setup. [0]PETSC ERROR: See https://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [0]PETSC ERROR: Petsc Release Version 3.12.5, Mar, 29, 2020 [0]PETSC ERROR: ./testSolve on a arch-linux2-c-debug named f601294ac804 by Unknown Thu Jul 16 03:11:02 2020 [0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --download-mpich --download-fblaslapack --with-cuda [0]PETSC ERROR: #1 VecCopy_Nest() line 68 in /usr/local/petsc/petsc-3.12.5/src/vec/vec/impls/nest/vecnest.c [0]PETSC ERROR: #2 VecCopy() line 1577 in /usr/local/petsc/petsc-3.12.5/src/vec/vec/interface/vector.c [0]PETSC ERROR: #3 KSPInitialResidual() line 61 in /usr/local/petsc/petsc-3.12.5/src/ksp/ksp/interface/itres.c [0]PETSC ERROR: #4 KSPSolve_GMRES() line 236 in /usr/local/petsc/petsc-3.12.5/src/ksp/ksp/impls/gmres/gmres.c [0]PETSC ERROR: #5 KSPSolve() line 760 in /usr/local/petsc/petsc-3.12.5/src/ksp/ksp/interface/itfunc.c On Jul 16 2020, at 2:39 am, Barry Smith > wrote: Could be a bug in PETSc, please send a sample program that produces the problem. Barry On Jul 15, 2020, at 1:28 PM, Karl Yang > wrote: Hi, Barry, Thanks for your reply. I output the matrix and vec with matview and vecview. And I think I do have the values set as it has been shown. Mat Object: 1 MPI processes type: nest Matrix object: type=nest, rows=3, cols=3 MatNest structure: (0,0) : type=seqaij, rows=25, cols=25 (0,1) : NULL (0,2) : type=seqaij, rows=25, cols=25 (1,0) : NULL (1,1) : type=seqaij, rows=25, cols=25 (1,2) : type=seqaij, rows=25, cols=25 (2,0) : type=seqaij, rows=25, cols=25 (2,1) : type=seqaij, rows=25, cols=25 (2,2) : NULL Vec Object: 1 MPI processes type: nest VecNest, rows=3, structure: (0) : type=seq, rows=25 Vec Object: 1 MPI processes type: seq 0. 0. 0. 0. 0. 0. 75.8279 -51.0776 -75.8279 0. 0. -90.8404 58.7011 90.8404 0. 0. -75.8279 51.0776 75.8279 0. 0. 0. 0. 0. 0. (1) : name="Vec_0x84000000_0", type=seq, rows=25 Vec Object: Vec_0x84000000_0 1 MPI processes type: seq 0. 0. 0. 0. 0. 0. 75.8279 -51.0776 -75.8279 0. 0. -90.8404 58.7011 90.8404 0. 0. -75.8279 51.0776 75.8279 0. 0. 0. 0. 0. 0. (2) : type=seq, rows=25 Vec Object: 1 MPI processes type: seq 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [0]PETSC ERROR: Invalid argument [0]PETSC ERROR: Nest vector argument 4 not setup. [0]PETSC ERROR: See https://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [0]PETSC ERROR: Petsc Release Version 3.12.5, Mar, 29, 2020 [0]PETSC ERROR: ./testSolve on a arch-linux2-c-debug named f601294ac804 by Unknown Thu Jul 16 02:15:26 2020 [0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --download-mpich --download-fblaslapack --with-cuda [0]PETSC ERROR: #1 VecPointwiseMult_Nest() line 245 in /usr/local/petsc/petsc-3.12.5/src/vec/vec/impls/nest/vecnest.c [0]PETSC ERROR: #2 VecPointwiseMult() line 1107 in /usr/local/petsc/petsc-3.12.5/src/vec/vec/interface/vector.c [0]PETSC ERROR: #3 PCApply_Jacobi() line 272 in /usr/local/petsc/petsc-3.12.5/src/ksp/pc/impls/jacobi/jacobi.c [0]PETSC ERROR: #4 PCApply() line 444 in /usr/local/petsc/petsc-3.12.5/src/ksp/pc/interface/precon.c [0]PETSC ERROR: #5 KSP_PCApply() line 281 in /usr/local/petsc/petsc-3.12.5/include/petsc/private/kspimpl.h [0]PETSC ERROR: #6 KSPInitialResidual() line 65 in /usr/local/petsc/petsc-3.12.5/src/ksp/ksp/interface/itres.c [0]PETSC ERROR: #7 KSPSolve_GMRES() line 236 in /usr/local/petsc/petsc-3.12.5/src/ksp/ksp/impls/gmres/gmres.c [0]PETSC ERROR: #8 KSPSolve() line 760 in /usr/local/petsc/petsc-3.12.5/src/ksp/ksp/interface/itfunc.c On Jul 16 2020, at 2:13 am, Barry Smith > wrote: From the error message the second vector you are using to form the nest vector has not been completely created/set up. Make sure you called either VecCreateSeq(), VecCreateMPI() or did VecCreate(), VecSetType(), VecSetUp() and if it is the right hand side make sure you put numerical values in it. Barry On Jul 15, 2020, at 1:05 PM, Karl Yang > wrote: Hello, I'm trying to solve a 2*2 block matrix system. I managed to form a 2*2 MatNest and a 21 VecNest. However when I try to make use of ksp solve, I encounter the following error. There is only one or two examples on MatNest and VecNest. I could not figure out what could be the trouble. [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [0]PETSC ERROR: Invalid argument [0]PETSC ERROR: Nest vector argument 2 not setup. [0]PETSC ERROR: See https://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [0]PETSC ERROR: Petsc Release Version 3.12.5, Mar, 29, 2020 [0]PETSC ERROR: ./testSolve on a arch-linux2-c-debug named f601294ac804 by Unknown Thu Jul 16 01:56:11 2020 [0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --download-mpich --download-fblaslapack --with-cuda [0]PETSC ERROR: #1 VecCopy_Nest() line 68 in /usr/local/petsc/petsc-3.12.5/src/vec/vec/impls/nest/vecnest.c [0]PETSC ERROR: #2 VecCopy() line 1577 in /usr/local/petsc/petsc-3.12.5/src/vec/vec/interface/vector.c [0]PETSC ERROR: #3 KSPInitialResidual() line 61 in /usr/local/petsc/petsc-3.12.5/src/ksp/ksp/interface/itres.c [0]PETSC ERROR: #4 KSPSolve_GMRES() line 236 in /usr/local/petsc/petsc-3.12.5/src/ksp/ksp/impls/gmres/gmres.c [0]PETSC ERROR: #5 KSPSolve() line 760 in /usr/local/petsc/petsc-3.12.5/src/ksp/ksp/interface/itfunc.c Regards JT -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Thu Jul 16 06:43:31 2020 From: knepley at gmail.com (Matthew Knepley) Date: Thu, 16 Jul 2020 07:43:31 -0400 Subject: [petsc-users] Errors encountered with KSPsolver with MATNEST and VECNEST In-Reply-To: References: Message-ID: On Thu, Jul 16, 2020 at 6:54 AM Yang Juntao wrote: > Hi, Matt, > > > > In case, I want to put blocks of matrices(assembled independently) > together as the system matrix just as in the example. What would be your > recommended approach? > > I first looked into MatCreateBlockMat, but it only supports sequential and > then I was directed to MATNEST. Do I have to assemble directly on ?system > matrix?. > The idea is that you use https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MatGetLocalSubMatrix.html This does require the extra step of calling https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MatSetLocalToGlobalMapping.html#MatSetLocalToGlobalMapping but you must already have this in order to get the global indices. If you use that, then you can assemble locally into AIJ or Nest with the same code. Thanks, Matt > ? > > Matrices of this type are nominally-sparse matrices in which each "entry" > is a Mat > > object. Each Mat > > must have the same size and be sequential. The local and global sizes must > be compatible with this decomposition. For matrices containing parallel > submatrices and variable block sizes, see MATNEST > . > > > ? > > > > Regards > > Juntao > > > > *From:* Matthew Knepley > *Sent:* Thursday, July 16, 2020 5:56 PM > *To:* Karl Yang > *Cc:* Junchao Zhang ; petsc-users at mcs.anl.gov > *Subject:* Re: [petsc-users] Errors encountered with KSPsolver with > MATNEST and VECNEST > > > > On Thu, Jul 16, 2020 at 5:43 AM Karl Yang wrote: > > Hi, Matt, > > > > Sorry for the confusion. The error shows as below without SetIS(). > > > > I see the problem now. It is about the interface for MatNest. > > > > MatNest is intended _only_ as an optimization. I think you should never > have MatNest in user code. Here the > > problem is that MatNest only supports extraction using the ISes it was > created with, whereas detection is > > finding a different IS. I recommend getting everything working with AIJ, > and then use MatSetType() or > > -mat_type at the end to try out Nest. The interfaces are the same. > > > > Thanks, > > > > Matt > > > > [0]PETSC ERROR: --------------------- Error Message > -------------------------------------------------------------- > > [0]PETSC ERROR: Arguments are incompatible > > [0]PETSC ERROR: Could not find index set > > [0]PETSC ERROR: See https://www.mcs.anl.gov/petsc/documentation/faq.html > for trouble shooting. > > [0]PETSC ERROR: Petsc Release Version 3.12.5, Mar, 29, 2020 > > [0]PETSC ERROR: ./testSolve on a arch-linux2-c-debug named ae3f0ab5a8c4 by > Unknown Thu Jul 16 17:36:52 2020 > > [0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ > --with-fc=gfortran --download-mpich --download-fblaslapack --with-cuda > > [0]PETSC ERROR: #1 MatNestFindIS() line 365 in > /usr/local/petsc/petsc-3.12.5/src/mat/impls/nest/matnest.c > > [0]PETSC ERROR: #2 MatNestFindSubMat() line 425 in > /usr/local/petsc/petsc-3.12.5/src/mat/impls/nest/matnest.c > > [0]PETSC ERROR: #3 MatCreateSubMatrix_Nest() line 456 in > /usr/local/petsc/petsc-3.12.5/src/mat/impls/nest/matnest.c > > [0]PETSC ERROR: #4 MatCreateSubMatrix() line 7859 in > /usr/local/petsc/petsc-3.12.5/src/mat/interface/matrix.c > > [0]PETSC ERROR: #5 PCSetUp_FieldSplit() line 676 in > /usr/local/petsc/petsc-3.12.5/src/ksp/pc/impls/fieldsplit/fieldsplit.c > > [0]PETSC ERROR: #6 PCSetUp() line 894 in > /usr/local/petsc/petsc-3.12.5/src/ksp/pc/interface/precon.c > > [0]PETSC ERROR: --------------------- Error Message > -------------------------------------------------------------- > > [0]PETSC ERROR: Corrupt argument: > https://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind > > [0]PETSC ERROR: Invalid Pointer to Object: Parameter # 1 > > [0]PETSC ERROR: See https://www.mcs.anl.gov/petsc/documentation/faq.html > for trouble shooting. > > [0]PETSC ERROR: Petsc Release Version 3.12.5, Mar, 29, 2020 > > [0]PETSC ERROR: ./testSolve on a arch-linux2-c-debug named ae3f0ab5a8c4 by > Unknown Thu Jul 16 17:36:52 2020 > > [0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ > --with-fc=gfortran --download-mpich --download-fblaslapack --with-cuda > > [0]PETSC ERROR: #7 MatDestroy() line 1266 in > /usr/local/petsc/petsc-3.12.5/src/mat/interface/matrix.c > > [0]PETSC ERROR: #8 PCSetUp_FieldSplit() line 697 in > /usr/local/petsc/petsc-3.12.5/src/ksp/pc/impls/fieldsplit/fieldsplit.c > > [0]PETSC ERROR: #9 PCSetUp() line 894 in > /usr/local/petsc/petsc-3.12.5/src/ksp/pc/interface/precon.c > > [0]PETSC ERROR: #10 KSPSetUp() line 376 in > /usr/local/petsc/petsc-3.12.5/src/ksp/ksp/interface/itfunc.c > > KSP Object: 1 MPI processes > > type: fgmres > > restart=30, using Classical (unmodified) Gram-Schmidt > Orthogonalization with no iterative refinement > > happy breakdown tolerance 1e-30 > > maximum iterations=10000, initial guess is zero > > tolerances: relative=1e-05, absolute=1e-50, divergence=10000. > > right preconditioning > > using UNPRECONDITIONED norm type for convergence test > > PC Object: 1 MPI processes > > type: fieldsplit > > PC has not been set up so information may be incomplete > > FieldSplit with Schur preconditioner, blocksize = 1, factorization FULL > > Preconditioner for the Schur complement formed from S itself > > Split info: > > Split number 0 Defined by IS > > Split number 1 Defined by IS > > KSP solver for A00 block > > KSP Object: (fieldsplit_0_) 1 MPI processes > > type: preonly > > maximum iterations=10000, initial guess is zero > > tolerances: relative=1e-05, absolute=1e-50, divergence=10000. > > left preconditioning > > using DEFAULT norm type for convergence test > > PC Object: (fieldsplit_0_) 1 MPI processes > > type not yet set > > PC has not been set up so information may be incomplete > > KSP solver for upper A00 in upper triangular factor > > not yet available > > KSP solver for S = A11 - A10 inv(A00) A01 > > not yet available > > linear system matrix = precond matrix: > > Mat Object: 1 MPI processes > > type: nest > > rows=15, cols=15 > > Matrix object: > > type=nest, rows=3, cols=3 > > MatNest structure: > > (0,0) : type=seqaij, rows=5, cols=5 > > (0,1) : NULL > > (0,2) : type=seqaij, rows=5, cols=5 > > (1,0) : NULL > > (1,1) : type=seqaij, rows=5, cols=5 > > (1,2) : type=seqaij, rows=5, cols=5 > > (2,0) : type=seqaij, rows=5, cols=5 > > (2,1) : type=seqaij, rows=5, cols=5 > > (2,2) : NULL > > Segmentation fault (core dumped) > > > > Attached is the test code. > > > > Regards > > JT > > > > > On Jul 16 2020, at 5:30 pm, Matthew Knepley wrote: > > On Thu, Jul 16, 2020 at 1:23 AM Karl Yang > > wrote: > > [image: Sent from Mailspring] > > Hi, Junchao, > > > > Thank you. It works. :) > > > > May I ask some follow up question regarding the same example about > fieldsplit. I am trying to learn about matnest with fieldsplit > preconditioner on manual page 94. But I get a bit lost trying to understand > fields, ISs once it is not eactly the same on the manual. > > > > In this same example, I tried to use PCFIELDSPLIT and > PCFieldSplitSetDetectSaddlePoint as follows. > > > > You cannot mix DetectSaddlePoint() with SetIS() since the detection tries > to make the proper ISes. Maybe we should check that. > > > > Thanks, > > > > Matt > > > > IS rows[3]; > > IS cols[3]; > > MatNestGetISs(testMesh->system, rows, cols); > > > > KSP ksp; > > PC pc; > > KSPCreate(PETSC_COMM_WORLD, &ksp); > > KSPSetType(ksp, KSPFGMRES); > > KSPSetOperators(ksp, testMesh->system, testMesh->system); > > KSPGetPC(ksp, &pc); > > PCFieldSplitSetIS(pc, "u", rows[0]); > > PCFieldSplitSetIS(pc, "v", rows[1]); > > PCFieldSplitSetIS(pc, "p", rows[2]); > > PCSetType(pc, PCFIELDSPLIT); > > PCFieldSplitSetDetectSaddlePoint(pc, PETSC_TRUE); > > KSPSetUp(ksp); > > KSPSolve(ksp, testMesh->rhs, testMesh->solution); > > > > But I will get the following error, > > > > [0]PETSC ERROR: --------------------- Error Message > -------------------------------------------------------------- > > [0]PETSC ERROR: Arguments are incompatible > > [0]PETSC ERROR: Could not find index set > > [0]PETSC ERROR: See https://www.mcs.anl.gov/petsc/documentation/faq.html for > trouble shooting. > > [0]PETSC ERROR: Petsc Release Version 3.12.5, Mar, 29, 2020 > > [0]PETSC ERROR: ./testSolve on a arch-linux2-c-debug named f601294ac804 by > Unknown Thu Jul 16 13:08:39 2020 > > [0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ > --with-fc=gfortran --download-mpich --download-fblaslapack --with-cuda > > [0]PETSC ERROR: #1 MatNestFindIS() line 365 in > /usr/local/petsc/petsc-3.12.5/src/mat/impls/nest/matnest.c > > [0]PETSC ERROR: #2 MatNestFindSubMat() line 425 in > /usr/local/petsc/petsc-3.12.5/src/mat/impls/nest/matnest.c > > [0]PETSC ERROR: #3 MatCreateSubMatrix_Nest() line 456 in > /usr/local/petsc/petsc-3.12.5/src/mat/impls/nest/matnest.c > > [0]PETSC ERROR: #4 MatCreateSubMatrix() line 7859 in > /usr/local/petsc/petsc-3.12.5/src/mat/interface/matrix.c > > [0]PETSC ERROR: #5 PCSetUp_FieldSplit() line 676 in > /usr/local/petsc/petsc-3.12.5/src/ksp/pc/impls/fieldsplit/fieldsplit.c > > [0]PETSC ERROR: #6 PCSetUp() line 894 in > /usr/local/petsc/petsc-3.12.5/src/ksp/pc/interface/precon.c > > [0]PETSC ERROR: #7 KSPSetUp() line 376 in > /usr/local/petsc/petsc-3.12.5/src/ksp/ksp/interface/itfunc.c > > [0]PETSC ERROR: --------------------- Error Message > -------------------------------------------------------------- > > [0]PETSC ERROR: Corrupt argument: > https://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind > > [0]PETSC ERROR: Invalid Pointer to Object: Parameter # 1 > > [0]PETSC ERROR: See https://www.mcs.anl.gov/petsc/documentation/faq.html for > trouble shooting. > > [0]PETSC ERROR: Petsc Release Version 3.12.5, Mar, 29, 2020 > > [0]PETSC ERROR: ./testSolve on a arch-linux2-c-debug named f601294ac804 by > Unknown Thu Jul 16 13:08:39 2020 > > [0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ > --with-fc=gfortran --download-mpich --download-fblaslapack --with-cuda > > [0]PETSC ERROR: #8 MatDestroy() line 1266 in > /usr/local/petsc/petsc-3.12.5/src/mat/interface/matrix.c > > [0]PETSC ERROR: #9 PCSetUp_FieldSplit() line 697 in > /usr/local/petsc/petsc-3.12.5/src/ksp/pc/impls/fieldsplit/fieldsplit.c > > [0]PETSC ERROR: #10 PCSetUp() line 894 in > /usr/local/petsc/petsc-3.12.5/src/ksp/pc/interface/precon.c > > [0]PETSC ERROR: #11 KSPSetUp() line 376 in > /usr/local/petsc/petsc-3.12.5/src/ksp/ksp/interface/itfunc.c > > [0]PETSC ERROR: #12 KSPSolve() line 703 in > /usr/local/petsc/petsc-3.12.5/src/ksp/ksp/interface/itfunc.c > > > > I read about the examples on the manual and mostly are based on 2*2 > blocks but still cannot figure out how pcfieldsplit should be set in this 3* > **3 blocks, and how IS should be set accordingly. I read about > -fieldsplit_1_pc_type none option suggested on the documentation, but I am > not sure is it still valid in a 33 blocks.* > > > > *Regards* > > *JT* > > On Jul 16 2020, at 6:57 am, Junchao Zhang wrote: > > Adding the MatNestSetVecType() line fixed the problem. But I don't know > why petsc had this weird design originally. > > > > MatCreateNest(PETSC_COMM_WORLD, 3, NULL, 3, NULL, mblock, > &testMesh->system); > > + MatNestSetVecType(testMesh->system,VECNEST); > > > > --Junchao Zhang > > > > > > On Wed, Jul 15, 2020 at 2:16 PM Karl Yang wrote: > > Hi, Barry, > > > > I created a minimum example to reproduce the error. The example code is > attached. > > > > Mat Object: 1 MPI processes > > type: nest > > Matrix object: > > type=nest, rows=3, cols=3 > > MatNest structure: > > (0,0) : type=seqaij, rows=5, cols=5 > > (0,1) : NULL > > (0,2) : type=seqaij, rows=5, cols=5 > > (1,0) : NULL > > (1,1) : type=seqaij, rows=5, cols=5 > > (1,2) : type=seqaij, rows=5, cols=5 > > (2,0) : type=seqaij, rows=5, cols=5 > > (2,1) : type=seqaij, rows=5, cols=5 > > (2,2) : NULL > > Vec Object: 1 MPI processes > > type: nest > > VecNest, rows=3, structure: > > (0) : type=seq, rows=5 > > Vec Object: 1 MPI processes > > type: seq > > 1. > > 1. > > 1. > > 1. > > 1. > > (1) : type=seq, rows=5 > > Vec Object: 1 MPI processes > > type: seq > > 1. > > 1. > > 1. > > 1. > > 1. > > (2) : type=seq, rows=5 > > Vec Object: 1 MPI processes > > type: seq > > 1. > > 1. > > 1. > > 1. > > 1. > > [0]PETSC ERROR: --------------------- Error Message > -------------------------------------------------------------- > > [0]PETSC ERROR: Invalid argument > > [0]PETSC ERROR: Nest vector argument 2 not setup. > > [0]PETSC ERROR: See https://www.mcs.anl.gov/petsc/documentation/faq.html for > trouble shooting. > > [0]PETSC ERROR: Petsc Release Version 3.12.5, Mar, 29, 2020 > > [0]PETSC ERROR: ./testSolve on a arch-linux2-c-debug named f601294ac804 by > Unknown Thu Jul 16 03:11:02 2020 > > [0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ > --with-fc=gfortran --download-mpich --download-fblaslapack --with-cuda > > [0]PETSC ERROR: #1 VecCopy_Nest() line 68 in > /usr/local/petsc/petsc-3.12.5/src/vec/vec/impls/nest/vecnest.c > > [0]PETSC ERROR: #2 VecCopy() line 1577 in > /usr/local/petsc/petsc-3.12.5/src/vec/vec/interface/vector.c > > [0]PETSC ERROR: #3 KSPInitialResidual() line 61 in > /usr/local/petsc/petsc-3.12.5/src/ksp/ksp/interface/itres.c > > [0]PETSC ERROR: #4 KSPSolve_GMRES() line 236 in > /usr/local/petsc/petsc-3.12.5/src/ksp/ksp/impls/gmres/gmres.c > > [0]PETSC ERROR: #5 KSPSolve() line 760 in > /usr/local/petsc/petsc-3.12.5/src/ksp/ksp/interface/itfunc.c > > On Jul 16 2020, at 2:39 am, Barry Smith wrote: > > > > Could be a bug in PETSc, please send a sample program that produces the > problem. > > > > Barry > > > > > > On Jul 15, 2020, at 1:28 PM, Karl Yang > > wrote: > > > > Hi, Barry, > > > > Thanks for your reply. I output the matrix and vec with matview and > vecview. And I think I do have the values set as it has been shown. > > > > Mat Object: 1 MPI processes > > type: nest > > Matrix object: > > type=nest, rows=3, cols=3 > > MatNest structure: > > (0,0) : type=seqaij, rows=25, cols=25 > > (0,1) : NULL > > (0,2) : type=seqaij, rows=25, cols=25 > > (1,0) : NULL > > (1,1) : type=seqaij, rows=25, cols=25 > > (1,2) : type=seqaij, rows=25, cols=25 > > (2,0) : type=seqaij, rows=25, cols=25 > > (2,1) : type=seqaij, rows=25, cols=25 > > (2,2) : NULL > > Vec Object: 1 MPI processes > > type: nest > > VecNest, rows=3, structure: > > (0) : type=seq, rows=25 > > Vec Object: 1 MPI processes > > type: seq > > 0. > > 0. > > 0. > > 0. > > 0. > > 0. > > 75.8279 > > -51.0776 > > -75.8279 > > 0. > > 0. > > -90.8404 > > 58.7011 > > 90.8404 > > 0. > > 0. > > -75.8279 > > 51.0776 > > 75.8279 > > 0. > > 0. > > 0. > > 0. > > 0. > > 0. > > (1) : name="Vec_0x84000000_0", type=seq, rows=25 > > Vec Object: Vec_0x84000000_0 1 MPI processes > > type: seq > > 0. > > 0. > > 0. > > 0. > > 0. > > 0. > > 75.8279 > > -51.0776 > > -75.8279 > > 0. > > 0. > > -90.8404 > > 58.7011 > > 90.8404 > > 0. > > 0. > > -75.8279 > > 51.0776 > > 75.8279 > > 0. > > 0. > > 0. > > 0. > > 0. > > 0. > > (2) : type=seq, rows=25 > > Vec Object: 1 MPI processes > > type: seq > > 0. > > 0. > > 0. > > 0. > > 0. > > 0. > > 0. > > 0. > > 0. > > 0. > > 0. > > 0. > > 0. > > 0. > > 0. > > 0. > > 0. > > 0. > > 0. > > 0. > > 0. > > 0. > > 0. > > 0. > > 0. > > [0]PETSC ERROR: --------------------- Error Message > -------------------------------------------------------------- > > [0]PETSC ERROR: Invalid argument > > [0]PETSC ERROR: Nest vector argument 4 not setup. > > [0]PETSC ERROR: See https://www.mcs.anl.gov/petsc/documentation/faq.html > for > trouble shooting. > > [0]PETSC ERROR: Petsc Release Version 3.12.5, Mar, 29, 2020 > > [0]PETSC ERROR: ./testSolve on a arch-linux2-c-debug named f601294ac804 by > Unknown Thu Jul 16 02:15:26 2020 > > [0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ > --with-fc=gfortran --download-mpich --download-fblaslapack --with-cuda > > [0]PETSC ERROR: #1 VecPointwiseMult_Nest() line 245 in > /usr/local/petsc/petsc-3.12.5/src/vec/vec/impls/nest/vecnest.c > > [0]PETSC ERROR: #2 VecPointwiseMult() line 1107 in > /usr/local/petsc/petsc-3.12.5/src/vec/vec/interface/vector.c > > [0]PETSC ERROR: #3 PCApply_Jacobi() line 272 in > /usr/local/petsc/petsc-3.12.5/src/ksp/pc/impls/jacobi/jacobi.c > > [0]PETSC ERROR: #4 PCApply() line 444 in > /usr/local/petsc/petsc-3.12.5/src/ksp/pc/interface/precon.c > > [0]PETSC ERROR: #5 KSP_PCApply() line 281 in > /usr/local/petsc/petsc-3.12.5/include/petsc/private/kspimpl.h > > [0]PETSC ERROR: #6 KSPInitialResidual() line 65 in > /usr/local/petsc/petsc-3.12.5/src/ksp/ksp/interface/itres.c > > [0]PETSC ERROR: #7 KSPSolve_GMRES() line 236 in > /usr/local/petsc/petsc-3.12.5/src/ksp/ksp/impls/gmres/gmres.c > > [0]PETSC ERROR: #8 KSPSolve() line 760 in > /usr/local/petsc/petsc-3.12.5/src/ksp/ksp/interface/itfunc.c > > > > On Jul 16 2020, at 2:13 am, Barry Smith > > wrote: > > > > From the error message the second vector you are using to form the nest > vector has not been completely created/set up. Make sure you called either > VecCreateSeq(), VecCreateMPI() or did VecCreate(), VecSetType(), VecSetUp() > and if it is the right hand side make sure you put numerical values in it. > > > > Barry > > > > > > On Jul 15, 2020, at 1:05 PM, Karl Yang > > wrote: > > > > Hello, > > > > I'm trying to solve a 2*2 block matrix system. > > I managed to form a 2*2 MatNest and a 21 VecNest. However when I try to > make use of ksp solve, I encounter the following error. There is only one > or two examples on MatNest and VecNest. I could not figure out what could > be the trouble. > > > > [0]PETSC ERROR: --------------------- Error Message > -------------------------------------------------------------- > > [0]PETSC ERROR: Invalid argument > > [0]PETSC ERROR: Nest vector argument 2 not setup. > > [0]PETSC ERROR: See https://www.mcs.anl.gov/petsc/documentation/faq.html > for > trouble shooting. > > [0]PETSC ERROR: Petsc Release Version 3.12.5, Mar, 29, 2020 > > [0]PETSC ERROR: ./testSolve on a arch-linux2-c-debug named f601294ac804 by > Unknown Thu Jul 16 01:56:11 2020 > > [0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ > --with-fc=gfortran --download-mpich --download-fblaslapack --with-cuda > > [0]PETSC ERROR: #1 VecCopy_Nest() line 68 in > /usr/local/petsc/petsc-3.12.5/src/vec/vec/impls/nest/vecnest.c > > [0]PETSC ERROR: #2 VecCopy() line 1577 in > /usr/local/petsc/petsc-3.12.5/src/vec/vec/interface/vector.c > > [0]PETSC ERROR: #3 KSPInitialResidual() line 61 in > /usr/local/petsc/petsc-3.12.5/src/ksp/ksp/interface/itres.c > > [0]PETSC ERROR: #4 KSPSolve_GMRES() line 236 in > /usr/local/petsc/petsc-3.12.5/src/ksp/ksp/impls/gmres/gmres.c > > [0]PETSC ERROR: #5 KSPSolve() line 760 in > /usr/local/petsc/petsc-3.12.5/src/ksp/ksp/interface/itfunc.c > > > > Regards > > JT > > > > > > -- > > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > > -- Norbert Wiener > > > > https://www.cse.buffalo.edu/~knepley/ > > > > > > -- > > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > > > https://www.cse.buffalo.edu/~knepley/ > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From mfadams at lbl.gov Thu Jul 16 12:02:57 2020 From: mfadams at lbl.gov (Mark Adams) Date: Thu, 16 Jul 2020 13:02:57 -0400 Subject: [petsc-users] configure and C++ and MPICH Message-ID: -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: configure.log Type: application/octet-stream Size: 249821 bytes Desc: not available URL: From balay at mcs.anl.gov Thu Jul 16 12:31:09 2020 From: balay at mcs.anl.gov (Satish Balay) Date: Thu, 16 Jul 2020 12:31:09 -0500 (CDT) Subject: [petsc-users] configure and C++ and MPICH In-Reply-To: References: Message-ID: >>>> Configure Options: --configModules=PETSc.Configure --optionsModule=config.compilerOptions --with-mpi-dir=/usr/local/Cellar/mpich/3.3.2_1 COPTFLAGS="-g -O0 -fopenmp" CXXOPTdFLAGS="-g -O0 -fopenmp" FOPTFLAGS="-g -O0" --with-fc=0 --with-clanguage=cxx --download-metis=1 --download-parmetis=1 --download-ml=1 --download-p4est=1 --with-zlib=1 --with-cxx-dialect=C++11 --download-triangle=1 --download-hdf5=1 -with-cuda=0 --with-x=0 --with-debugging=1 PETSC_ARCH=arch-macosx-gnu-g-omp --with-64-bit-indices=0 --with-openmp=1 Executing: /usr/local/Cellar/mpich/3.3.2_1/bin/mpicc --version stdout: clang version 10.0.0 Target: x86_64-apple-darwin19.5.0 ld: library not found for -lomp <<<<<< Xcode clang does not support openmp Satish From luis.saturday at gmail.com Sat Jul 18 05:11:34 2020 From: luis.saturday at gmail.com (Alex Fleeter) Date: Sat, 18 Jul 2020 03:11:34 -0700 Subject: [petsc-users] Errors encountered with KSPsolver with MATNEST and VECNEST In-Reply-To: References: Message-ID: I guess using MatNest is still the most memory efficient one, am I correct? Does MatGetLocalSubMatrix demand extra memory for generating the sub matrices? On Thu, Jul 16, 2020 at 4:44 AM Matthew Knepley wrote: > On Thu, Jul 16, 2020 at 6:54 AM Yang Juntao wrote: > >> Hi, Matt, >> >> >> >> In case, I want to put blocks of matrices(assembled independently) >> together as the system matrix just as in the example. What would be your >> recommended approach? >> >> I first looked into MatCreateBlockMat, but it only supports sequential >> and then I was directed to MATNEST. Do I have to assemble directly on >> ?system matrix?. >> > > The idea is that you use > > > https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MatGetLocalSubMatrix.html > > This does require the extra step of calling > > > https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MatSetLocalToGlobalMapping.html#MatSetLocalToGlobalMapping > > but you must already have this in order to get the global indices. > > If you use that, then you can assemble locally into AIJ or Nest with the > same code. > > Thanks, > > Matt > > >> ? >> >> Matrices of this type are nominally-sparse matrices in which each "entry" >> is a Mat >> >> object. Each Mat >> >> must have the same size and be sequential. The local and global sizes must >> be compatible with this decomposition. For matrices containing parallel >> submatrices and variable block sizes, see MATNEST >> . >> >> >> ? >> >> >> >> Regards >> >> Juntao >> >> >> >> *From:* Matthew Knepley >> *Sent:* Thursday, July 16, 2020 5:56 PM >> *To:* Karl Yang >> *Cc:* Junchao Zhang ; petsc-users at mcs.anl.gov >> *Subject:* Re: [petsc-users] Errors encountered with KSPsolver with >> MATNEST and VECNEST >> >> >> >> On Thu, Jul 16, 2020 at 5:43 AM Karl Yang wrote: >> >> Hi, Matt, >> >> >> >> Sorry for the confusion. The error shows as below without SetIS(). >> >> >> >> I see the problem now. It is about the interface for MatNest. >> >> >> >> MatNest is intended _only_ as an optimization. I think you should never >> have MatNest in user code. Here the >> >> problem is that MatNest only supports extraction using the ISes it was >> created with, whereas detection is >> >> finding a different IS. I recommend getting everything working with AIJ, >> and then use MatSetType() or >> >> -mat_type at the end to try out Nest. The interfaces are the same. >> >> >> >> Thanks, >> >> >> >> Matt >> >> >> >> [0]PETSC ERROR: --------------------- Error Message >> -------------------------------------------------------------- >> >> [0]PETSC ERROR: Arguments are incompatible >> >> [0]PETSC ERROR: Could not find index set >> >> [0]PETSC ERROR: See https://www.mcs.anl.gov/petsc/documentation/faq.html >> for trouble shooting. >> >> [0]PETSC ERROR: Petsc Release Version 3.12.5, Mar, 29, 2020 >> >> [0]PETSC ERROR: ./testSolve on a arch-linux2-c-debug named ae3f0ab5a8c4 >> by Unknown Thu Jul 16 17:36:52 2020 >> >> [0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ >> --with-fc=gfortran --download-mpich --download-fblaslapack --with-cuda >> >> [0]PETSC ERROR: #1 MatNestFindIS() line 365 in >> /usr/local/petsc/petsc-3.12.5/src/mat/impls/nest/matnest.c >> >> [0]PETSC ERROR: #2 MatNestFindSubMat() line 425 in >> /usr/local/petsc/petsc-3.12.5/src/mat/impls/nest/matnest.c >> >> [0]PETSC ERROR: #3 MatCreateSubMatrix_Nest() line 456 in >> /usr/local/petsc/petsc-3.12.5/src/mat/impls/nest/matnest.c >> >> [0]PETSC ERROR: #4 MatCreateSubMatrix() line 7859 in >> /usr/local/petsc/petsc-3.12.5/src/mat/interface/matrix.c >> >> [0]PETSC ERROR: #5 PCSetUp_FieldSplit() line 676 in >> /usr/local/petsc/petsc-3.12.5/src/ksp/pc/impls/fieldsplit/fieldsplit.c >> >> [0]PETSC ERROR: #6 PCSetUp() line 894 in >> /usr/local/petsc/petsc-3.12.5/src/ksp/pc/interface/precon.c >> >> [0]PETSC ERROR: --------------------- Error Message >> -------------------------------------------------------------- >> >> [0]PETSC ERROR: Corrupt argument: >> https://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind >> >> [0]PETSC ERROR: Invalid Pointer to Object: Parameter # 1 >> >> [0]PETSC ERROR: See https://www.mcs.anl.gov/petsc/documentation/faq.html >> for trouble shooting. >> >> [0]PETSC ERROR: Petsc Release Version 3.12.5, Mar, 29, 2020 >> >> [0]PETSC ERROR: ./testSolve on a arch-linux2-c-debug named ae3f0ab5a8c4 >> by Unknown Thu Jul 16 17:36:52 2020 >> >> [0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ >> --with-fc=gfortran --download-mpich --download-fblaslapack --with-cuda >> >> [0]PETSC ERROR: #7 MatDestroy() line 1266 in >> /usr/local/petsc/petsc-3.12.5/src/mat/interface/matrix.c >> >> [0]PETSC ERROR: #8 PCSetUp_FieldSplit() line 697 in >> /usr/local/petsc/petsc-3.12.5/src/ksp/pc/impls/fieldsplit/fieldsplit.c >> >> [0]PETSC ERROR: #9 PCSetUp() line 894 in >> /usr/local/petsc/petsc-3.12.5/src/ksp/pc/interface/precon.c >> >> [0]PETSC ERROR: #10 KSPSetUp() line 376 in >> /usr/local/petsc/petsc-3.12.5/src/ksp/ksp/interface/itfunc.c >> >> KSP Object: 1 MPI processes >> >> type: fgmres >> >> restart=30, using Classical (unmodified) Gram-Schmidt >> Orthogonalization with no iterative refinement >> >> happy breakdown tolerance 1e-30 >> >> maximum iterations=10000, initial guess is zero >> >> tolerances: relative=1e-05, absolute=1e-50, divergence=10000. >> >> right preconditioning >> >> using UNPRECONDITIONED norm type for convergence test >> >> PC Object: 1 MPI processes >> >> type: fieldsplit >> >> PC has not been set up so information may be incomplete >> >> FieldSplit with Schur preconditioner, blocksize = 1, factorization >> FULL >> >> Preconditioner for the Schur complement formed from S itself >> >> Split info: >> >> Split number 0 Defined by IS >> >> Split number 1 Defined by IS >> >> KSP solver for A00 block >> >> KSP Object: (fieldsplit_0_) 1 MPI processes >> >> type: preonly >> >> maximum iterations=10000, initial guess is zero >> >> tolerances: relative=1e-05, absolute=1e-50, divergence=10000. >> >> left preconditioning >> >> using DEFAULT norm type for convergence test >> >> PC Object: (fieldsplit_0_) 1 MPI processes >> >> type not yet set >> >> PC has not been set up so information may be incomplete >> >> KSP solver for upper A00 in upper triangular factor >> >> not yet available >> >> KSP solver for S = A11 - A10 inv(A00) A01 >> >> not yet available >> >> linear system matrix = precond matrix: >> >> Mat Object: 1 MPI processes >> >> type: nest >> >> rows=15, cols=15 >> >> Matrix object: >> >> type=nest, rows=3, cols=3 >> >> MatNest structure: >> >> (0,0) : type=seqaij, rows=5, cols=5 >> >> (0,1) : NULL >> >> (0,2) : type=seqaij, rows=5, cols=5 >> >> (1,0) : NULL >> >> (1,1) : type=seqaij, rows=5, cols=5 >> >> (1,2) : type=seqaij, rows=5, cols=5 >> >> (2,0) : type=seqaij, rows=5, cols=5 >> >> (2,1) : type=seqaij, rows=5, cols=5 >> >> (2,2) : NULL >> >> Segmentation fault (core dumped) >> >> >> >> Attached is the test code. >> >> >> >> Regards >> >> JT >> >> >> >> >> On Jul 16 2020, at 5:30 pm, Matthew Knepley wrote: >> >> On Thu, Jul 16, 2020 at 1:23 AM Karl Yang > > >> wrote: >> >> [image: Sent from Mailspring] >> >> Hi, Junchao, >> >> >> >> Thank you. It works. :) >> >> >> >> May I ask some follow up question regarding the same example about >> fieldsplit. I am trying to learn about matnest with fieldsplit >> preconditioner on manual page 94. But I get a bit lost trying to understand >> fields, ISs once it is not eactly the same on the manual. >> >> >> >> In this same example, I tried to use PCFIELDSPLIT and >> PCFieldSplitSetDetectSaddlePoint as follows. >> >> >> >> You cannot mix DetectSaddlePoint() with SetIS() since the detection tries >> to make the proper ISes. Maybe we should check that. >> >> >> >> Thanks, >> >> >> >> Matt >> >> >> >> IS rows[3]; >> >> IS cols[3]; >> >> MatNestGetISs(testMesh->system, rows, cols); >> >> >> >> KSP ksp; >> >> PC pc; >> >> KSPCreate(PETSC_COMM_WORLD, &ksp); >> >> KSPSetType(ksp, KSPFGMRES); >> >> KSPSetOperators(ksp, testMesh->system, testMesh->system); >> >> KSPGetPC(ksp, &pc); >> >> PCFieldSplitSetIS(pc, "u", rows[0]); >> >> PCFieldSplitSetIS(pc, "v", rows[1]); >> >> PCFieldSplitSetIS(pc, "p", rows[2]); >> >> PCSetType(pc, PCFIELDSPLIT); >> >> PCFieldSplitSetDetectSaddlePoint(pc, PETSC_TRUE); >> >> KSPSetUp(ksp); >> >> KSPSolve(ksp, testMesh->rhs, testMesh->solution); >> >> >> >> But I will get the following error, >> >> >> >> [0]PETSC ERROR: --------------------- Error Message >> -------------------------------------------------------------- >> >> [0]PETSC ERROR: Arguments are incompatible >> >> [0]PETSC ERROR: Could not find index set >> >> [0]PETSC ERROR: See https://www.mcs.anl.gov/petsc/documentation/faq.html for >> trouble shooting. >> >> [0]PETSC ERROR: Petsc Release Version 3.12.5, Mar, 29, 2020 >> >> [0]PETSC ERROR: ./testSolve on a arch-linux2-c-debug named f601294ac804 >> by Unknown Thu Jul 16 13:08:39 2020 >> >> [0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ >> --with-fc=gfortran --download-mpich --download-fblaslapack --with-cuda >> >> [0]PETSC ERROR: #1 MatNestFindIS() line 365 in >> /usr/local/petsc/petsc-3.12.5/src/mat/impls/nest/matnest.c >> >> [0]PETSC ERROR: #2 MatNestFindSubMat() line 425 in >> /usr/local/petsc/petsc-3.12.5/src/mat/impls/nest/matnest.c >> >> [0]PETSC ERROR: #3 MatCreateSubMatrix_Nest() line 456 in >> /usr/local/petsc/petsc-3.12.5/src/mat/impls/nest/matnest.c >> >> [0]PETSC ERROR: #4 MatCreateSubMatrix() line 7859 in >> /usr/local/petsc/petsc-3.12.5/src/mat/interface/matrix.c >> >> [0]PETSC ERROR: #5 PCSetUp_FieldSplit() line 676 in >> /usr/local/petsc/petsc-3.12.5/src/ksp/pc/impls/fieldsplit/fieldsplit.c >> >> [0]PETSC ERROR: #6 PCSetUp() line 894 in >> /usr/local/petsc/petsc-3.12.5/src/ksp/pc/interface/precon.c >> >> [0]PETSC ERROR: #7 KSPSetUp() line 376 in >> /usr/local/petsc/petsc-3.12.5/src/ksp/ksp/interface/itfunc.c >> >> [0]PETSC ERROR: --------------------- Error Message >> -------------------------------------------------------------- >> >> [0]PETSC ERROR: Corrupt argument: >> https://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind >> >> [0]PETSC ERROR: Invalid Pointer to Object: Parameter # 1 >> >> [0]PETSC ERROR: See https://www.mcs.anl.gov/petsc/documentation/faq.html for >> trouble shooting. >> >> [0]PETSC ERROR: Petsc Release Version 3.12.5, Mar, 29, 2020 >> >> [0]PETSC ERROR: ./testSolve on a arch-linux2-c-debug named f601294ac804 >> by Unknown Thu Jul 16 13:08:39 2020 >> >> [0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ >> --with-fc=gfortran --download-mpich --download-fblaslapack --with-cuda >> >> [0]PETSC ERROR: #8 MatDestroy() line 1266 in >> /usr/local/petsc/petsc-3.12.5/src/mat/interface/matrix.c >> >> [0]PETSC ERROR: #9 PCSetUp_FieldSplit() line 697 in >> /usr/local/petsc/petsc-3.12.5/src/ksp/pc/impls/fieldsplit/fieldsplit.c >> >> [0]PETSC ERROR: #10 PCSetUp() line 894 in >> /usr/local/petsc/petsc-3.12.5/src/ksp/pc/interface/precon.c >> >> [0]PETSC ERROR: #11 KSPSetUp() line 376 in >> /usr/local/petsc/petsc-3.12.5/src/ksp/ksp/interface/itfunc.c >> >> [0]PETSC ERROR: #12 KSPSolve() line 703 in >> /usr/local/petsc/petsc-3.12.5/src/ksp/ksp/interface/itfunc.c >> >> >> >> I read about the examples on the manual and mostly are based on 2*2 >> blocks but still cannot figure out how pcfieldsplit should be set in this 3* >> **3 blocks, and how IS should be set accordingly. I read about >> -fieldsplit_1_pc_type none option suggested on the documentation, but I am >> not sure is it still valid in a 33 blocks.* >> >> >> >> *Regards* >> >> *JT* >> >> On Jul 16 2020, at 6:57 am, Junchao Zhang >> wrote: >> >> Adding the MatNestSetVecType() line fixed the problem. But I don't know >> why petsc had this weird design originally. >> >> >> >> MatCreateNest(PETSC_COMM_WORLD, 3, NULL, 3, NULL, mblock, >> &testMesh->system); >> >> + MatNestSetVecType(testMesh->system,VECNEST); >> >> >> >> --Junchao Zhang >> >> >> >> >> >> On Wed, Jul 15, 2020 at 2:16 PM Karl Yang wrote: >> >> Hi, Barry, >> >> >> >> I created a minimum example to reproduce the error. The example code is >> attached. >> >> >> >> Mat Object: 1 MPI processes >> >> type: nest >> >> Matrix object: >> >> type=nest, rows=3, cols=3 >> >> MatNest structure: >> >> (0,0) : type=seqaij, rows=5, cols=5 >> >> (0,1) : NULL >> >> (0,2) : type=seqaij, rows=5, cols=5 >> >> (1,0) : NULL >> >> (1,1) : type=seqaij, rows=5, cols=5 >> >> (1,2) : type=seqaij, rows=5, cols=5 >> >> (2,0) : type=seqaij, rows=5, cols=5 >> >> (2,1) : type=seqaij, rows=5, cols=5 >> >> (2,2) : NULL >> >> Vec Object: 1 MPI processes >> >> type: nest >> >> VecNest, rows=3, structure: >> >> (0) : type=seq, rows=5 >> >> Vec Object: 1 MPI processes >> >> type: seq >> >> 1. >> >> 1. >> >> 1. >> >> 1. >> >> 1. >> >> (1) : type=seq, rows=5 >> >> Vec Object: 1 MPI processes >> >> type: seq >> >> 1. >> >> 1. >> >> 1. >> >> 1. >> >> 1. >> >> (2) : type=seq, rows=5 >> >> Vec Object: 1 MPI processes >> >> type: seq >> >> 1. >> >> 1. >> >> 1. >> >> 1. >> >> 1. >> >> [0]PETSC ERROR: --------------------- Error Message >> -------------------------------------------------------------- >> >> [0]PETSC ERROR: Invalid argument >> >> [0]PETSC ERROR: Nest vector argument 2 not setup. >> >> [0]PETSC ERROR: See https://www.mcs.anl.gov/petsc/documentation/faq.html for >> trouble shooting. >> >> [0]PETSC ERROR: Petsc Release Version 3.12.5, Mar, 29, 2020 >> >> [0]PETSC ERROR: ./testSolve on a arch-linux2-c-debug named f601294ac804 >> by Unknown Thu Jul 16 03:11:02 2020 >> >> [0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ >> --with-fc=gfortran --download-mpich --download-fblaslapack --with-cuda >> >> [0]PETSC ERROR: #1 VecCopy_Nest() line 68 in >> /usr/local/petsc/petsc-3.12.5/src/vec/vec/impls/nest/vecnest.c >> >> [0]PETSC ERROR: #2 VecCopy() line 1577 in >> /usr/local/petsc/petsc-3.12.5/src/vec/vec/interface/vector.c >> >> [0]PETSC ERROR: #3 KSPInitialResidual() line 61 in >> /usr/local/petsc/petsc-3.12.5/src/ksp/ksp/interface/itres.c >> >> [0]PETSC ERROR: #4 KSPSolve_GMRES() line 236 in >> /usr/local/petsc/petsc-3.12.5/src/ksp/ksp/impls/gmres/gmres.c >> >> [0]PETSC ERROR: #5 KSPSolve() line 760 in >> /usr/local/petsc/petsc-3.12.5/src/ksp/ksp/interface/itfunc.c >> >> On Jul 16 2020, at 2:39 am, Barry Smith wrote: >> >> >> >> Could be a bug in PETSc, please send a sample program that produces the >> problem. >> >> >> >> Barry >> >> >> >> >> >> On Jul 15, 2020, at 1:28 PM, Karl Yang > > >> wrote: >> >> >> >> Hi, Barry, >> >> >> >> Thanks for your reply. I output the matrix and vec with matview and >> vecview. And I think I do have the values set as it has been shown. >> >> >> >> Mat Object: 1 MPI processes >> >> type: nest >> >> Matrix object: >> >> type=nest, rows=3, cols=3 >> >> MatNest structure: >> >> (0,0) : type=seqaij, rows=25, cols=25 >> >> (0,1) : NULL >> >> (0,2) : type=seqaij, rows=25, cols=25 >> >> (1,0) : NULL >> >> (1,1) : type=seqaij, rows=25, cols=25 >> >> (1,2) : type=seqaij, rows=25, cols=25 >> >> (2,0) : type=seqaij, rows=25, cols=25 >> >> (2,1) : type=seqaij, rows=25, cols=25 >> >> (2,2) : NULL >> >> Vec Object: 1 MPI processes >> >> type: nest >> >> VecNest, rows=3, structure: >> >> (0) : type=seq, rows=25 >> >> Vec Object: 1 MPI processes >> >> type: seq >> >> 0. >> >> 0. >> >> 0. >> >> 0. >> >> 0. >> >> 0. >> >> 75.8279 >> >> -51.0776 >> >> -75.8279 >> >> 0. >> >> 0. >> >> -90.8404 >> >> 58.7011 >> >> 90.8404 >> >> 0. >> >> 0. >> >> -75.8279 >> >> 51.0776 >> >> 75.8279 >> >> 0. >> >> 0. >> >> 0. >> >> 0. >> >> 0. >> >> 0. >> >> (1) : name="Vec_0x84000000_0", type=seq, rows=25 >> >> Vec Object: Vec_0x84000000_0 1 MPI processes >> >> type: seq >> >> 0. >> >> 0. >> >> 0. >> >> 0. >> >> 0. >> >> 0. >> >> 75.8279 >> >> -51.0776 >> >> -75.8279 >> >> 0. >> >> 0. >> >> -90.8404 >> >> 58.7011 >> >> 90.8404 >> >> 0. >> >> 0. >> >> -75.8279 >> >> 51.0776 >> >> 75.8279 >> >> 0. >> >> 0. >> >> 0. >> >> 0. >> >> 0. >> >> 0. >> >> (2) : type=seq, rows=25 >> >> Vec Object: 1 MPI processes >> >> type: seq >> >> 0. >> >> 0. >> >> 0. >> >> 0. >> >> 0. >> >> 0. >> >> 0. >> >> 0. >> >> 0. >> >> 0. >> >> 0. >> >> 0. >> >> 0. >> >> 0. >> >> 0. >> >> 0. >> >> 0. >> >> 0. >> >> 0. >> >> 0. >> >> 0. >> >> 0. >> >> 0. >> >> 0. >> >> 0. >> >> [0]PETSC ERROR: --------------------- Error Message >> -------------------------------------------------------------- >> >> [0]PETSC ERROR: Invalid argument >> >> [0]PETSC ERROR: Nest vector argument 4 not setup. >> >> [0]PETSC ERROR: See https://www.mcs.anl.gov/petsc/documentation/faq.html >> for >> trouble shooting. >> >> [0]PETSC ERROR: Petsc Release Version 3.12.5, Mar, 29, 2020 >> >> [0]PETSC ERROR: ./testSolve on a arch-linux2-c-debug named f601294ac804 >> by Unknown Thu Jul 16 02:15:26 2020 >> >> [0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ >> --with-fc=gfortran --download-mpich --download-fblaslapack --with-cuda >> >> [0]PETSC ERROR: #1 VecPointwiseMult_Nest() line 245 in >> /usr/local/petsc/petsc-3.12.5/src/vec/vec/impls/nest/vecnest.c >> >> [0]PETSC ERROR: #2 VecPointwiseMult() line 1107 in >> /usr/local/petsc/petsc-3.12.5/src/vec/vec/interface/vector.c >> >> [0]PETSC ERROR: #3 PCApply_Jacobi() line 272 in >> /usr/local/petsc/petsc-3.12.5/src/ksp/pc/impls/jacobi/jacobi.c >> >> [0]PETSC ERROR: #4 PCApply() line 444 in >> /usr/local/petsc/petsc-3.12.5/src/ksp/pc/interface/precon.c >> >> [0]PETSC ERROR: #5 KSP_PCApply() line 281 in >> /usr/local/petsc/petsc-3.12.5/include/petsc/private/kspimpl.h >> >> [0]PETSC ERROR: #6 KSPInitialResidual() line 65 in >> /usr/local/petsc/petsc-3.12.5/src/ksp/ksp/interface/itres.c >> >> [0]PETSC ERROR: #7 KSPSolve_GMRES() line 236 in >> /usr/local/petsc/petsc-3.12.5/src/ksp/ksp/impls/gmres/gmres.c >> >> [0]PETSC ERROR: #8 KSPSolve() line 760 in >> /usr/local/petsc/petsc-3.12.5/src/ksp/ksp/interface/itfunc.c >> >> >> >> On Jul 16 2020, at 2:13 am, Barry Smith > > >> wrote: >> >> >> >> From the error message the second vector you are using to form the nest >> vector has not been completely created/set up. Make sure you called either >> VecCreateSeq(), VecCreateMPI() or did VecCreate(), VecSetType(), VecSetUp() >> and if it is the right hand side make sure you put numerical values in it. >> >> >> >> Barry >> >> >> >> >> >> On Jul 15, 2020, at 1:05 PM, Karl Yang > > >> wrote: >> >> >> >> Hello, >> >> >> >> I'm trying to solve a 2*2 block matrix system. >> >> I managed to form a 2*2 MatNest and a 21 VecNest. However when I try to >> make use of ksp solve, I encounter the following error. There is only one >> or two examples on MatNest and VecNest. I could not figure out what could >> be the trouble. >> >> >> >> [0]PETSC ERROR: --------------------- Error Message >> -------------------------------------------------------------- >> >> [0]PETSC ERROR: Invalid argument >> >> [0]PETSC ERROR: Nest vector argument 2 not setup. >> >> [0]PETSC ERROR: See https://www.mcs.anl.gov/petsc/documentation/faq.html >> for >> trouble shooting. >> >> [0]PETSC ERROR: Petsc Release Version 3.12.5, Mar, 29, 2020 >> >> [0]PETSC ERROR: ./testSolve on a arch-linux2-c-debug named f601294ac804 >> by Unknown Thu Jul 16 01:56:11 2020 >> >> [0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ >> --with-fc=gfortran --download-mpich --download-fblaslapack --with-cuda >> >> [0]PETSC ERROR: #1 VecCopy_Nest() line 68 in >> /usr/local/petsc/petsc-3.12.5/src/vec/vec/impls/nest/vecnest.c >> >> [0]PETSC ERROR: #2 VecCopy() line 1577 in >> /usr/local/petsc/petsc-3.12.5/src/vec/vec/interface/vector.c >> >> [0]PETSC ERROR: #3 KSPInitialResidual() line 61 in >> /usr/local/petsc/petsc-3.12.5/src/ksp/ksp/interface/itres.c >> >> [0]PETSC ERROR: #4 KSPSolve_GMRES() line 236 in >> /usr/local/petsc/petsc-3.12.5/src/ksp/ksp/impls/gmres/gmres.c >> >> [0]PETSC ERROR: #5 KSPSolve() line 760 in >> /usr/local/petsc/petsc-3.12.5/src/ksp/ksp/interface/itfunc.c >> >> >> >> Regards >> >> JT >> >> >> >> >> >> -- >> >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> >> -- Norbert Wiener >> >> >> >> https://www.cse.buffalo.edu/~knepley/ >> >> >> >> >> >> -- >> >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> -- Norbert Wiener >> >> >> >> https://www.cse.buffalo.edu/~knepley/ >> >> > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Sat Jul 18 06:51:50 2020 From: knepley at gmail.com (Matthew Knepley) Date: Sat, 18 Jul 2020 07:51:50 -0400 Subject: [petsc-users] Errors encountered with KSPsolver with MATNEST and VECNEST In-Reply-To: References: Message-ID: On Sat, Jul 18, 2020 at 6:11 AM Alex Fleeter wrote: > I guess using MatNest is still the most memory efficient one, am I > correct? Does MatGetLocalSubMatrix demand extra memory for generating the > sub matrices? > No. LocalSubMatrix() is just a view into the backing matrix, whether it is AIJ or Nest, and it forwards the MatSetValues() calls. Matt > On Thu, Jul 16, 2020 at 4:44 AM Matthew Knepley wrote: > >> On Thu, Jul 16, 2020 at 6:54 AM Yang Juntao wrote: >> >>> Hi, Matt, >>> >>> >>> >>> In case, I want to put blocks of matrices(assembled independently) >>> together as the system matrix just as in the example. What would be your >>> recommended approach? >>> >>> I first looked into MatCreateBlockMat, but it only supports sequential >>> and then I was directed to MATNEST. Do I have to assemble directly on >>> ?system matrix?. >>> >> >> The idea is that you use >> >> >> https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MatGetLocalSubMatrix.html >> >> This does require the extra step of calling >> >> >> https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MatSetLocalToGlobalMapping.html#MatSetLocalToGlobalMapping >> >> but you must already have this in order to get the global indices. >> >> If you use that, then you can assemble locally into AIJ or Nest with the >> same code. >> >> Thanks, >> >> Matt >> >> >>> ? >>> >>> Matrices of this type are nominally-sparse matrices in which each >>> "entry" is a Mat >>> >>> object. Each Mat >>> >>> must have the same size and be sequential. The local and global sizes must >>> be compatible with this decomposition. For matrices containing parallel >>> submatrices and variable block sizes, see MATNEST >>> . >>> >>> >>> ? >>> >>> >>> >>> Regards >>> >>> Juntao >>> >>> >>> >>> *From:* Matthew Knepley >>> *Sent:* Thursday, July 16, 2020 5:56 PM >>> *To:* Karl Yang >>> *Cc:* Junchao Zhang ; petsc-users at mcs.anl.gov >>> *Subject:* Re: [petsc-users] Errors encountered with KSPsolver with >>> MATNEST and VECNEST >>> >>> >>> >>> On Thu, Jul 16, 2020 at 5:43 AM Karl Yang wrote: >>> >>> Hi, Matt, >>> >>> >>> >>> Sorry for the confusion. The error shows as below without SetIS(). >>> >>> >>> >>> I see the problem now. It is about the interface for MatNest. >>> >>> >>> >>> MatNest is intended _only_ as an optimization. I think you should never >>> have MatNest in user code. Here the >>> >>> problem is that MatNest only supports extraction using the ISes it was >>> created with, whereas detection is >>> >>> finding a different IS. I recommend getting everything working with AIJ, >>> and then use MatSetType() or >>> >>> -mat_type at the end to try out Nest. The interfaces are the same. >>> >>> >>> >>> Thanks, >>> >>> >>> >>> Matt >>> >>> >>> >>> [0]PETSC ERROR: --------------------- Error Message >>> -------------------------------------------------------------- >>> >>> [0]PETSC ERROR: Arguments are incompatible >>> >>> [0]PETSC ERROR: Could not find index set >>> >>> [0]PETSC ERROR: See https://www.mcs.anl.gov/petsc/documentation/faq.html >>> for trouble shooting. >>> >>> [0]PETSC ERROR: Petsc Release Version 3.12.5, Mar, 29, 2020 >>> >>> [0]PETSC ERROR: ./testSolve on a arch-linux2-c-debug named ae3f0ab5a8c4 >>> by Unknown Thu Jul 16 17:36:52 2020 >>> >>> [0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ >>> --with-fc=gfortran --download-mpich --download-fblaslapack --with-cuda >>> >>> [0]PETSC ERROR: #1 MatNestFindIS() line 365 in >>> /usr/local/petsc/petsc-3.12.5/src/mat/impls/nest/matnest.c >>> >>> [0]PETSC ERROR: #2 MatNestFindSubMat() line 425 in >>> /usr/local/petsc/petsc-3.12.5/src/mat/impls/nest/matnest.c >>> >>> [0]PETSC ERROR: #3 MatCreateSubMatrix_Nest() line 456 in >>> /usr/local/petsc/petsc-3.12.5/src/mat/impls/nest/matnest.c >>> >>> [0]PETSC ERROR: #4 MatCreateSubMatrix() line 7859 in >>> /usr/local/petsc/petsc-3.12.5/src/mat/interface/matrix.c >>> >>> [0]PETSC ERROR: #5 PCSetUp_FieldSplit() line 676 in >>> /usr/local/petsc/petsc-3.12.5/src/ksp/pc/impls/fieldsplit/fieldsplit.c >>> >>> [0]PETSC ERROR: #6 PCSetUp() line 894 in >>> /usr/local/petsc/petsc-3.12.5/src/ksp/pc/interface/precon.c >>> >>> [0]PETSC ERROR: --------------------- Error Message >>> -------------------------------------------------------------- >>> >>> [0]PETSC ERROR: Corrupt argument: >>> https://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind >>> >>> [0]PETSC ERROR: Invalid Pointer to Object: Parameter # 1 >>> >>> [0]PETSC ERROR: See https://www.mcs.anl.gov/petsc/documentation/faq.html >>> for trouble shooting. >>> >>> [0]PETSC ERROR: Petsc Release Version 3.12.5, Mar, 29, 2020 >>> >>> [0]PETSC ERROR: ./testSolve on a arch-linux2-c-debug named ae3f0ab5a8c4 >>> by Unknown Thu Jul 16 17:36:52 2020 >>> >>> [0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ >>> --with-fc=gfortran --download-mpich --download-fblaslapack --with-cuda >>> >>> [0]PETSC ERROR: #7 MatDestroy() line 1266 in >>> /usr/local/petsc/petsc-3.12.5/src/mat/interface/matrix.c >>> >>> [0]PETSC ERROR: #8 PCSetUp_FieldSplit() line 697 in >>> /usr/local/petsc/petsc-3.12.5/src/ksp/pc/impls/fieldsplit/fieldsplit.c >>> >>> [0]PETSC ERROR: #9 PCSetUp() line 894 in >>> /usr/local/petsc/petsc-3.12.5/src/ksp/pc/interface/precon.c >>> >>> [0]PETSC ERROR: #10 KSPSetUp() line 376 in >>> /usr/local/petsc/petsc-3.12.5/src/ksp/ksp/interface/itfunc.c >>> >>> KSP Object: 1 MPI processes >>> >>> type: fgmres >>> >>> restart=30, using Classical (unmodified) Gram-Schmidt >>> Orthogonalization with no iterative refinement >>> >>> happy breakdown tolerance 1e-30 >>> >>> maximum iterations=10000, initial guess is zero >>> >>> tolerances: relative=1e-05, absolute=1e-50, divergence=10000. >>> >>> right preconditioning >>> >>> using UNPRECONDITIONED norm type for convergence test >>> >>> PC Object: 1 MPI processes >>> >>> type: fieldsplit >>> >>> PC has not been set up so information may be incomplete >>> >>> FieldSplit with Schur preconditioner, blocksize = 1, factorization >>> FULL >>> >>> Preconditioner for the Schur complement formed from S itself >>> >>> Split info: >>> >>> Split number 0 Defined by IS >>> >>> Split number 1 Defined by IS >>> >>> KSP solver for A00 block >>> >>> KSP Object: (fieldsplit_0_) 1 MPI processes >>> >>> type: preonly >>> >>> maximum iterations=10000, initial guess is zero >>> >>> tolerances: relative=1e-05, absolute=1e-50, divergence=10000. >>> >>> left preconditioning >>> >>> using DEFAULT norm type for convergence test >>> >>> PC Object: (fieldsplit_0_) 1 MPI processes >>> >>> type not yet set >>> >>> PC has not been set up so information may be incomplete >>> >>> KSP solver for upper A00 in upper triangular factor >>> >>> not yet available >>> >>> KSP solver for S = A11 - A10 inv(A00) A01 >>> >>> not yet available >>> >>> linear system matrix = precond matrix: >>> >>> Mat Object: 1 MPI processes >>> >>> type: nest >>> >>> rows=15, cols=15 >>> >>> Matrix object: >>> >>> type=nest, rows=3, cols=3 >>> >>> MatNest structure: >>> >>> (0,0) : type=seqaij, rows=5, cols=5 >>> >>> (0,1) : NULL >>> >>> (0,2) : type=seqaij, rows=5, cols=5 >>> >>> (1,0) : NULL >>> >>> (1,1) : type=seqaij, rows=5, cols=5 >>> >>> (1,2) : type=seqaij, rows=5, cols=5 >>> >>> (2,0) : type=seqaij, rows=5, cols=5 >>> >>> (2,1) : type=seqaij, rows=5, cols=5 >>> >>> (2,2) : NULL >>> >>> Segmentation fault (core dumped) >>> >>> >>> >>> Attached is the test code. >>> >>> >>> >>> Regards >>> >>> JT >>> >>> >>> >>> >>> On Jul 16 2020, at 5:30 pm, Matthew Knepley wrote: >>> >>> On Thu, Jul 16, 2020 at 1:23 AM Karl Yang >> > >>> wrote: >>> >>> [image: Sent from Mailspring] >>> >>> Hi, Junchao, >>> >>> >>> >>> Thank you. It works. :) >>> >>> >>> >>> May I ask some follow up question regarding the same example about >>> fieldsplit. I am trying to learn about matnest with fieldsplit >>> preconditioner on manual page 94. But I get a bit lost trying to understand >>> fields, ISs once it is not eactly the same on the manual. >>> >>> >>> >>> In this same example, I tried to use PCFIELDSPLIT and >>> PCFieldSplitSetDetectSaddlePoint as follows. >>> >>> >>> >>> You cannot mix DetectSaddlePoint() with SetIS() since the detection >>> tries to make the proper ISes. Maybe we should check that. >>> >>> >>> >>> Thanks, >>> >>> >>> >>> Matt >>> >>> >>> >>> IS rows[3]; >>> >>> IS cols[3]; >>> >>> MatNestGetISs(testMesh->system, rows, cols); >>> >>> >>> >>> KSP ksp; >>> >>> PC pc; >>> >>> KSPCreate(PETSC_COMM_WORLD, &ksp); >>> >>> KSPSetType(ksp, KSPFGMRES); >>> >>> KSPSetOperators(ksp, testMesh->system, testMesh->system); >>> >>> KSPGetPC(ksp, &pc); >>> >>> PCFieldSplitSetIS(pc, "u", rows[0]); >>> >>> PCFieldSplitSetIS(pc, "v", rows[1]); >>> >>> PCFieldSplitSetIS(pc, "p", rows[2]); >>> >>> PCSetType(pc, PCFIELDSPLIT); >>> >>> PCFieldSplitSetDetectSaddlePoint(pc, PETSC_TRUE); >>> >>> KSPSetUp(ksp); >>> >>> KSPSolve(ksp, testMesh->rhs, testMesh->solution); >>> >>> >>> >>> But I will get the following error, >>> >>> >>> >>> [0]PETSC ERROR: --------------------- Error Message >>> -------------------------------------------------------------- >>> >>> [0]PETSC ERROR: Arguments are incompatible >>> >>> [0]PETSC ERROR: Could not find index set >>> >>> [0]PETSC ERROR: See https://www.mcs.anl.gov/petsc/documentation/faq.html for >>> trouble shooting. >>> >>> [0]PETSC ERROR: Petsc Release Version 3.12.5, Mar, 29, 2020 >>> >>> [0]PETSC ERROR: ./testSolve on a arch-linux2-c-debug named f601294ac804 >>> by Unknown Thu Jul 16 13:08:39 2020 >>> >>> [0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ >>> --with-fc=gfortran --download-mpich --download-fblaslapack --with-cuda >>> >>> [0]PETSC ERROR: #1 MatNestFindIS() line 365 in >>> /usr/local/petsc/petsc-3.12.5/src/mat/impls/nest/matnest.c >>> >>> [0]PETSC ERROR: #2 MatNestFindSubMat() line 425 in >>> /usr/local/petsc/petsc-3.12.5/src/mat/impls/nest/matnest.c >>> >>> [0]PETSC ERROR: #3 MatCreateSubMatrix_Nest() line 456 in >>> /usr/local/petsc/petsc-3.12.5/src/mat/impls/nest/matnest.c >>> >>> [0]PETSC ERROR: #4 MatCreateSubMatrix() line 7859 in >>> /usr/local/petsc/petsc-3.12.5/src/mat/interface/matrix.c >>> >>> [0]PETSC ERROR: #5 PCSetUp_FieldSplit() line 676 in >>> /usr/local/petsc/petsc-3.12.5/src/ksp/pc/impls/fieldsplit/fieldsplit.c >>> >>> [0]PETSC ERROR: #6 PCSetUp() line 894 in >>> /usr/local/petsc/petsc-3.12.5/src/ksp/pc/interface/precon.c >>> >>> [0]PETSC ERROR: #7 KSPSetUp() line 376 in >>> /usr/local/petsc/petsc-3.12.5/src/ksp/ksp/interface/itfunc.c >>> >>> [0]PETSC ERROR: --------------------- Error Message >>> -------------------------------------------------------------- >>> >>> [0]PETSC ERROR: Corrupt argument: >>> https://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind >>> >>> [0]PETSC ERROR: Invalid Pointer to Object: Parameter # 1 >>> >>> [0]PETSC ERROR: See https://www.mcs.anl.gov/petsc/documentation/faq.html for >>> trouble shooting. >>> >>> [0]PETSC ERROR: Petsc Release Version 3.12.5, Mar, 29, 2020 >>> >>> [0]PETSC ERROR: ./testSolve on a arch-linux2-c-debug named f601294ac804 >>> by Unknown Thu Jul 16 13:08:39 2020 >>> >>> [0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ >>> --with-fc=gfortran --download-mpich --download-fblaslapack --with-cuda >>> >>> [0]PETSC ERROR: #8 MatDestroy() line 1266 in >>> /usr/local/petsc/petsc-3.12.5/src/mat/interface/matrix.c >>> >>> [0]PETSC ERROR: #9 PCSetUp_FieldSplit() line 697 in >>> /usr/local/petsc/petsc-3.12.5/src/ksp/pc/impls/fieldsplit/fieldsplit.c >>> >>> [0]PETSC ERROR: #10 PCSetUp() line 894 in >>> /usr/local/petsc/petsc-3.12.5/src/ksp/pc/interface/precon.c >>> >>> [0]PETSC ERROR: #11 KSPSetUp() line 376 in >>> /usr/local/petsc/petsc-3.12.5/src/ksp/ksp/interface/itfunc.c >>> >>> [0]PETSC ERROR: #12 KSPSolve() line 703 in >>> /usr/local/petsc/petsc-3.12.5/src/ksp/ksp/interface/itfunc.c >>> >>> >>> >>> I read about the examples on the manual and mostly are based on 2*2 >>> blocks but still cannot figure out how pcfieldsplit should be set in this 3* >>> **3 blocks, and how IS should be set accordingly. I read about >>> -fieldsplit_1_pc_type none option suggested on the documentation, but I am >>> not sure is it still valid in a 33 blocks.* >>> >>> >>> >>> *Regards* >>> >>> *JT* >>> >>> On Jul 16 2020, at 6:57 am, Junchao Zhang >>> wrote: >>> >>> Adding the MatNestSetVecType() line fixed the problem. But I don't know >>> why petsc had this weird design originally. >>> >>> >>> >>> MatCreateNest(PETSC_COMM_WORLD, 3, NULL, 3, NULL, mblock, >>> &testMesh->system); >>> >>> + MatNestSetVecType(testMesh->system,VECNEST); >>> >>> >>> >>> --Junchao Zhang >>> >>> >>> >>> >>> >>> On Wed, Jul 15, 2020 at 2:16 PM Karl Yang wrote: >>> >>> Hi, Barry, >>> >>> >>> >>> I created a minimum example to reproduce the error. The example code is >>> attached. >>> >>> >>> >>> Mat Object: 1 MPI processes >>> >>> type: nest >>> >>> Matrix object: >>> >>> type=nest, rows=3, cols=3 >>> >>> MatNest structure: >>> >>> (0,0) : type=seqaij, rows=5, cols=5 >>> >>> (0,1) : NULL >>> >>> (0,2) : type=seqaij, rows=5, cols=5 >>> >>> (1,0) : NULL >>> >>> (1,1) : type=seqaij, rows=5, cols=5 >>> >>> (1,2) : type=seqaij, rows=5, cols=5 >>> >>> (2,0) : type=seqaij, rows=5, cols=5 >>> >>> (2,1) : type=seqaij, rows=5, cols=5 >>> >>> (2,2) : NULL >>> >>> Vec Object: 1 MPI processes >>> >>> type: nest >>> >>> VecNest, rows=3, structure: >>> >>> (0) : type=seq, rows=5 >>> >>> Vec Object: 1 MPI processes >>> >>> type: seq >>> >>> 1. >>> >>> 1. >>> >>> 1. >>> >>> 1. >>> >>> 1. >>> >>> (1) : type=seq, rows=5 >>> >>> Vec Object: 1 MPI processes >>> >>> type: seq >>> >>> 1. >>> >>> 1. >>> >>> 1. >>> >>> 1. >>> >>> 1. >>> >>> (2) : type=seq, rows=5 >>> >>> Vec Object: 1 MPI processes >>> >>> type: seq >>> >>> 1. >>> >>> 1. >>> >>> 1. >>> >>> 1. >>> >>> 1. >>> >>> [0]PETSC ERROR: --------------------- Error Message >>> -------------------------------------------------------------- >>> >>> [0]PETSC ERROR: Invalid argument >>> >>> [0]PETSC ERROR: Nest vector argument 2 not setup. >>> >>> [0]PETSC ERROR: See https://www.mcs.anl.gov/petsc/documentation/faq.html for >>> trouble shooting. >>> >>> [0]PETSC ERROR: Petsc Release Version 3.12.5, Mar, 29, 2020 >>> >>> [0]PETSC ERROR: ./testSolve on a arch-linux2-c-debug named f601294ac804 >>> by Unknown Thu Jul 16 03:11:02 2020 >>> >>> [0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ >>> --with-fc=gfortran --download-mpich --download-fblaslapack --with-cuda >>> >>> [0]PETSC ERROR: #1 VecCopy_Nest() line 68 in >>> /usr/local/petsc/petsc-3.12.5/src/vec/vec/impls/nest/vecnest.c >>> >>> [0]PETSC ERROR: #2 VecCopy() line 1577 in >>> /usr/local/petsc/petsc-3.12.5/src/vec/vec/interface/vector.c >>> >>> [0]PETSC ERROR: #3 KSPInitialResidual() line 61 in >>> /usr/local/petsc/petsc-3.12.5/src/ksp/ksp/interface/itres.c >>> >>> [0]PETSC ERROR: #4 KSPSolve_GMRES() line 236 in >>> /usr/local/petsc/petsc-3.12.5/src/ksp/ksp/impls/gmres/gmres.c >>> >>> [0]PETSC ERROR: #5 KSPSolve() line 760 in >>> /usr/local/petsc/petsc-3.12.5/src/ksp/ksp/interface/itfunc.c >>> >>> On Jul 16 2020, at 2:39 am, Barry Smith wrote: >>> >>> >>> >>> Could be a bug in PETSc, please send a sample program that produces >>> the problem. >>> >>> >>> >>> Barry >>> >>> >>> >>> >>> >>> On Jul 15, 2020, at 1:28 PM, Karl Yang >> > >>> wrote: >>> >>> >>> >>> Hi, Barry, >>> >>> >>> >>> Thanks for your reply. I output the matrix and vec with matview and >>> vecview. And I think I do have the values set as it has been shown. >>> >>> >>> >>> Mat Object: 1 MPI processes >>> >>> type: nest >>> >>> Matrix object: >>> >>> type=nest, rows=3, cols=3 >>> >>> MatNest structure: >>> >>> (0,0) : type=seqaij, rows=25, cols=25 >>> >>> (0,1) : NULL >>> >>> (0,2) : type=seqaij, rows=25, cols=25 >>> >>> (1,0) : NULL >>> >>> (1,1) : type=seqaij, rows=25, cols=25 >>> >>> (1,2) : type=seqaij, rows=25, cols=25 >>> >>> (2,0) : type=seqaij, rows=25, cols=25 >>> >>> (2,1) : type=seqaij, rows=25, cols=25 >>> >>> (2,2) : NULL >>> >>> Vec Object: 1 MPI processes >>> >>> type: nest >>> >>> VecNest, rows=3, structure: >>> >>> (0) : type=seq, rows=25 >>> >>> Vec Object: 1 MPI processes >>> >>> type: seq >>> >>> 0. >>> >>> 0. >>> >>> 0. >>> >>> 0. >>> >>> 0. >>> >>> 0. >>> >>> 75.8279 >>> >>> -51.0776 >>> >>> -75.8279 >>> >>> 0. >>> >>> 0. >>> >>> -90.8404 >>> >>> 58.7011 >>> >>> 90.8404 >>> >>> 0. >>> >>> 0. >>> >>> -75.8279 >>> >>> 51.0776 >>> >>> 75.8279 >>> >>> 0. >>> >>> 0. >>> >>> 0. >>> >>> 0. >>> >>> 0. >>> >>> 0. >>> >>> (1) : name="Vec_0x84000000_0", type=seq, rows=25 >>> >>> Vec Object: Vec_0x84000000_0 1 MPI processes >>> >>> type: seq >>> >>> 0. >>> >>> 0. >>> >>> 0. >>> >>> 0. >>> >>> 0. >>> >>> 0. >>> >>> 75.8279 >>> >>> -51.0776 >>> >>> -75.8279 >>> >>> 0. >>> >>> 0. >>> >>> -90.8404 >>> >>> 58.7011 >>> >>> 90.8404 >>> >>> 0. >>> >>> 0. >>> >>> -75.8279 >>> >>> 51.0776 >>> >>> 75.8279 >>> >>> 0. >>> >>> 0. >>> >>> 0. >>> >>> 0. >>> >>> 0. >>> >>> 0. >>> >>> (2) : type=seq, rows=25 >>> >>> Vec Object: 1 MPI processes >>> >>> type: seq >>> >>> 0. >>> >>> 0. >>> >>> 0. >>> >>> 0. >>> >>> 0. >>> >>> 0. >>> >>> 0. >>> >>> 0. >>> >>> 0. >>> >>> 0. >>> >>> 0. >>> >>> 0. >>> >>> 0. >>> >>> 0. >>> >>> 0. >>> >>> 0. >>> >>> 0. >>> >>> 0. >>> >>> 0. >>> >>> 0. >>> >>> 0. >>> >>> 0. >>> >>> 0. >>> >>> 0. >>> >>> 0. >>> >>> [0]PETSC ERROR: --------------------- Error Message >>> -------------------------------------------------------------- >>> >>> [0]PETSC ERROR: Invalid argument >>> >>> [0]PETSC ERROR: Nest vector argument 4 not setup. >>> >>> [0]PETSC ERROR: See https://www.mcs.anl.gov/petsc/documentation/faq.html >>> for >>> trouble shooting. >>> >>> [0]PETSC ERROR: Petsc Release Version 3.12.5, Mar, 29, 2020 >>> >>> [0]PETSC ERROR: ./testSolve on a arch-linux2-c-debug named f601294ac804 >>> by Unknown Thu Jul 16 02:15:26 2020 >>> >>> [0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ >>> --with-fc=gfortran --download-mpich --download-fblaslapack --with-cuda >>> >>> [0]PETSC ERROR: #1 VecPointwiseMult_Nest() line 245 in >>> /usr/local/petsc/petsc-3.12.5/src/vec/vec/impls/nest/vecnest.c >>> >>> [0]PETSC ERROR: #2 VecPointwiseMult() line 1107 in >>> /usr/local/petsc/petsc-3.12.5/src/vec/vec/interface/vector.c >>> >>> [0]PETSC ERROR: #3 PCApply_Jacobi() line 272 in >>> /usr/local/petsc/petsc-3.12.5/src/ksp/pc/impls/jacobi/jacobi.c >>> >>> [0]PETSC ERROR: #4 PCApply() line 444 in >>> /usr/local/petsc/petsc-3.12.5/src/ksp/pc/interface/precon.c >>> >>> [0]PETSC ERROR: #5 KSP_PCApply() line 281 in >>> /usr/local/petsc/petsc-3.12.5/include/petsc/private/kspimpl.h >>> >>> [0]PETSC ERROR: #6 KSPInitialResidual() line 65 in >>> /usr/local/petsc/petsc-3.12.5/src/ksp/ksp/interface/itres.c >>> >>> [0]PETSC ERROR: #7 KSPSolve_GMRES() line 236 in >>> /usr/local/petsc/petsc-3.12.5/src/ksp/ksp/impls/gmres/gmres.c >>> >>> [0]PETSC ERROR: #8 KSPSolve() line 760 in >>> /usr/local/petsc/petsc-3.12.5/src/ksp/ksp/interface/itfunc.c >>> >>> >>> >>> On Jul 16 2020, at 2:13 am, Barry Smith >> > >>> wrote: >>> >>> >>> >>> From the error message the second vector you are using to form the >>> nest vector has not been completely created/set up. Make sure you called >>> either VecCreateSeq(), VecCreateMPI() or did VecCreate(), VecSetType(), >>> VecSetUp() and if it is the right hand side make sure you put numerical >>> values in it. >>> >>> >>> >>> Barry >>> >>> >>> >>> >>> >>> On Jul 15, 2020, at 1:05 PM, Karl Yang >> > >>> wrote: >>> >>> >>> >>> Hello, >>> >>> >>> >>> I'm trying to solve a 2*2 block matrix system. >>> >>> I managed to form a 2*2 MatNest and a 21 VecNest. However when I try to >>> make use of ksp solve, I encounter the following error. There is only one >>> or two examples on MatNest and VecNest. I could not figure out what could >>> be the trouble. >>> >>> >>> >>> [0]PETSC ERROR: --------------------- Error Message >>> -------------------------------------------------------------- >>> >>> [0]PETSC ERROR: Invalid argument >>> >>> [0]PETSC ERROR: Nest vector argument 2 not setup. >>> >>> [0]PETSC ERROR: See https://www.mcs.anl.gov/petsc/documentation/faq.html >>> for >>> trouble shooting. >>> >>> [0]PETSC ERROR: Petsc Release Version 3.12.5, Mar, 29, 2020 >>> >>> [0]PETSC ERROR: ./testSolve on a arch-linux2-c-debug named f601294ac804 >>> by Unknown Thu Jul 16 01:56:11 2020 >>> >>> [0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ >>> --with-fc=gfortran --download-mpich --download-fblaslapack --with-cuda >>> >>> [0]PETSC ERROR: #1 VecCopy_Nest() line 68 in >>> /usr/local/petsc/petsc-3.12.5/src/vec/vec/impls/nest/vecnest.c >>> >>> [0]PETSC ERROR: #2 VecCopy() line 1577 in >>> /usr/local/petsc/petsc-3.12.5/src/vec/vec/interface/vector.c >>> >>> [0]PETSC ERROR: #3 KSPInitialResidual() line 61 in >>> /usr/local/petsc/petsc-3.12.5/src/ksp/ksp/interface/itres.c >>> >>> [0]PETSC ERROR: #4 KSPSolve_GMRES() line 236 in >>> /usr/local/petsc/petsc-3.12.5/src/ksp/ksp/impls/gmres/gmres.c >>> >>> [0]PETSC ERROR: #5 KSPSolve() line 760 in >>> /usr/local/petsc/petsc-3.12.5/src/ksp/ksp/interface/itfunc.c >>> >>> >>> >>> Regards >>> >>> JT >>> >>> >>> >>> >>> >>> -- >>> >>> What most experimenters take for granted before they begin their >>> experiments is infinitely more interesting than any results to which their >>> experiments lead. >>> >>> -- Norbert Wiener >>> >>> >>> >>> https://www.cse.buffalo.edu/~knepley/ >>> >>> >>> >>> >>> >>> -- >>> >>> What most experimenters take for granted before they begin their >>> experiments is infinitely more interesting than any results to which their >>> experiments lead. >>> -- Norbert Wiener >>> >>> >>> >>> https://www.cse.buffalo.edu/~knepley/ >>> >>> >> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> -- Norbert Wiener >> >> https://www.cse.buffalo.edu/~knepley/ >> >> > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From y.juntao at hotmail.com Sat Jul 18 23:27:30 2020 From: y.juntao at hotmail.com (Karl Yang) Date: Sun, 19 Jul 2020 12:27:30 +0800 Subject: [petsc-users] PetscLogFlops Error Message-ID: Hello, I was using FGMRES solver with fieldsplit preconditioner for solving a fluid equation. I have the code ran for coarser meshes and it works fine. But when I got it to run for very fine meshes I encountered the PetscLogFlops error. The following is the output of error and the output from kspview. [7]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [7]PETSC ERROR: Argument out of range [7]PETSC ERROR: Cannot log negative flops [7]PETSC ERROR: See https://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [7]PETSC ERROR: Petsc Release Version 3.12.5, Mar, 29, 2020 [7]PETSC ERROR: ./testSolve on a arch-linux2-c-debug named hsw222 by yjuntao Sat Jul 18 20:30:05 2020 [7]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --download-mpich --download-fblaslapack --with-cuda [7]PETSC ERROR: #1 PetscLogFlops() line 232 in /usr/local/petsc/petsc-3.12.5/include/petsclog.h [7]PETSC ERROR: #2 MatSolve_SeqAIJ() line 3200 in /usr/local/petsc/petsc-3.12.5/src/mat/impls/aij/seq/aijfact.c [7]PETSC ERROR: #3 MatSolve() line 3290 in /usr/local/petsc/petsc-3.12.5/src/mat/interface/matrix.c [7]PETSC ERROR: #4 PCApply_LU() line 177 in /usr/local/petsc/petsc-3.12.5/src/ksp/pc/impls/factor/lu/lu.c [7]PETSC ERROR: #5 PCApply() line 444 in /usr/local/petsc/petsc-3.12.5/src/ksp/pc/interface/precon.c [7]PETSC ERROR: #6 KSP_PCApply() line 281 in /usr/local/petsc/petsc-3.12.5/include/petsc/private/kspimpl.h [7]PETSC ERROR: #7 KSPInitialResidual() line 65 in /usr/local/petsc/petsc-3.12.5/src/ksp/ksp/interface/itres.c [7]PETSC ERROR: #8 KSPSolve_GMRES() line 236 in /usr/local/petsc/petsc-3.12.5/src/ksp/ksp/impls/gmres/gmres.c [7]PETSC ERROR: #9 KSPSolve() line 760 in /usr/local/petsc/petsc-3.12.5/src/ksp/ksp/interface/itfunc.c [7]PETSC ERROR: #10 PCApply_FieldSplit_Schur() line 1180 in /usr/local/petsc/petsc-3.12.5/src/ksp/pc/impls/fieldsplit/fieldsplit.c [7]PETSC ERROR: #11 PCApply() line 444 in /usr/local/petsc/petsc-3.12.5/src/ksp/pc/interface/precon.c [7]PETSC ERROR: #12 KSP_PCApply() line 281 in /usr/local/petsc/petsc-3.12.5/include/petsc/private/kspimpl.h [7]PETSC ERROR: #13 KSPFGMRESCycle() line 166 in /usr/local/petsc/petsc-3.12.5/src/ksp/ksp/impls/gmres/fgmres/fgmres.c [7]PETSC ERROR: #14 KSPSolve_FGMRES() line 291 in /usr/local/petsc/petsc-3.12.5/src/ksp/ksp/impls/gmres/fgmres/fgmres.c [7]PETSC ERROR: #15 KSPSolve() line 760 in /usr/local/petsc/petsc-3.12.5/src/ksp/ksp/interface/itfunc.c There is some issues during kspview if PCLSC is used as shown below. But the code could be ran if I do not show ksp details. In order to show the whole ksp setup, I also attached another kspview output without PCLSC. ======================kspview output ============================== KSP Object: 1 MPI processes type: fgmres restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement happy breakdown tolerance 1e-30 maximum iterations=10000, initial guess is zero tolerances: relative=1e-08, absolute=1e-50, divergence=10000. right preconditioning using UNPRECONDITIONED norm type for convergence test PC Object: 1 MPI processes type: fieldsplit FieldSplit with Schur preconditioner, blocksize = 1, factorization FULL Preconditioner for the Schur complement formed from S itself Split info: Split number 0 Defined by IS Split number 1 Defined by IS KSP solver for A00 block KSP Object: (fieldsplit_0_) 1 MPI processes type: gmres restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement happy breakdown tolerance 1e-30 maximum iterations=10000, initial guess is zero tolerances: relative=1e-05, absolute=1e-50, divergence=10000. left preconditioning using DEFAULT norm type for convergence test PC Object: (fieldsplit_0_) 1 MPI processes type: lu PC has not been set up so information may be incomplete out-of-place factorization tolerance for zero pivot 2.22045e-14 matrix ordering: nd linear system matrix = precond matrix: Mat Object: (fieldsplit_0_) 1 MPI processes type: seqaijcusparse rows=8396802, cols=8396802 total: nonzeros=75440146, allocated nonzeros=75440146 total number of mallocs used during MatSetValues calls=0 not using I-node routines KSP solver for S = A11 - A10 inv(A00) A01 KSP Object: (fieldsplit_1_) 1 MPI processes type: gmres restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement happy breakdown tolerance 1e-30 maximum iterations=10000, initial guess is zero tolerances: relative=1e-05, absolute=1e-50, divergence=10000. left preconditioning using DEFAULT norm type for convergence test PC Object: (fieldsplit_1_) 1 MPI processes type: lsc PC has not been set up so information may be incomplete [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [0]PETSC ERROR: Null argument, when expecting valid pointer [0]PETSC ERROR: Null Object: Parameter # 1 [0]PETSC ERROR: See https://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [0]PETSC ERROR: Petsc Release Version 3.12.5, Mar, 29, 2020 [0]PETSC ERROR: ./testSolve on a arch-linux2-c-debug named hsw221 by yjuntao Sat Jul 18 21:08:42 2020 [0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --download-mpich --download-fblaslapack --with-cuda [0]PETSC ERROR: #1 KSPView() line 111 in /usr/local/petsc/petsc-3.12.5/src/ksp/ksp/interface/itcreate.c [0]PETSC ERROR: #2 PCView_LSC() line 142 in /usr/local/petsc/petsc-3.12.5/src/ksp/pc/impls/lsc/lsc.c [0]PETSC ERROR: #3 PCView() line 1588 in /usr/local/petsc/petsc-3.12.5/src/ksp/pc/interface/precon.c [0]PETSC ERROR: #4 KSPView() line 217 in /usr/local/petsc/petsc-3.12.5/src/ksp/ksp/interface/itcreate.c [0]PETSC ERROR: #5 PCView_FieldSplit_Schur() line 238 in /usr/local/petsc/petsc-3.12.5/src/ksp/pc/impls/fieldsplit/fieldsplit.c [0]PETSC ERROR: #6 PCView() line 1588 in /usr/local/petsc/petsc-3.12.5/src/ksp/pc/interface/precon.c [0]PETSC ERROR: #7 KSPView() line 217 in /usr/local/petsc/petsc-3.12.5/src/ksp/ksp/interface/itcreate.c =====================KSP without PCLSC====================================== KSP Object: 1 MPI processes type: fgmres restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement happy breakdown tolerance 1e-30 maximum iterations=10000, initial guess is zero tolerances: relative=1e-08, absolute=1e-50, divergence=10000. right preconditioning using UNPRECONDITIONED norm type for convergence test PC Object: 1 MPI processes type: fieldsplit FieldSplit with Schur preconditioner, blocksize = 1, factorization FULL Preconditioner for the Schur complement formed from S itself Split info: Split number 0 Defined by IS Split number 1 Defined by IS KSP solver for A00 block KSP Object: (fieldsplit_0_) 1 MPI processes type: gmres restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement happy breakdown tolerance 1e-30 maximum iterations=10000, initial guess is zero tolerances: relative=1e-05, absolute=1e-50, divergence=10000. left preconditioning using DEFAULT norm type for convergence test PC Object: (fieldsplit_0_) 1 MPI processes type: lu PC has not been set up so information may be incomplete out-of-place factorization tolerance for zero pivot 2.22045e-14 matrix ordering: nd linear system matrix = precond matrix: Mat Object: (fieldsplit_0_) 1 MPI processes type: seqaijcusparse rows=8396802, cols=8396802 total: nonzeros=75440146, allocated nonzeros=75440146 total number of mallocs used during MatSetValues calls=0 not using I-node routines KSP solver for S = A11 - A10 inv(A00) A01 KSP Object: (fieldsplit_1_) 1 MPI processes type: gmres restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement happy breakdown tolerance 1e-30 maximum iterations=10000, initial guess is zero tolerances: relative=1e-05, absolute=1e-50, divergence=10000. left preconditioning using DEFAULT norm type for convergence test PC Object: (fieldsplit_1_) 1 MPI processes type: none PC has not been set up so information may be incomplete linear system matrix = precond matrix: Mat Object: (fieldsplit_1_) 1 MPI processes type: schurcomplement rows=1050625, cols=1050625 Schur complement A11 - A10 inv(A00) A01 A11 Mat Object: (fieldsplit_1_) 1 MPI processes type: seqaijcusparse rows=1050625, cols=1050625 total: nonzeros=0, allocated nonzeros=0 total number of mallocs used during MatSetValues calls=0 not using I-node routines A10 Mat Object: 1 MPI processes type: seqaijcusparse rows=1050625, cols=8396802 total: nonzeros=44060674, allocated nonzeros=44060674 total number of mallocs used during MatSetValues calls=0 not using I-node routines KSP of A00 KSP Object: (fieldsplit_0_) 1 MPI processes type: gmres restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement happy breakdown tolerance 1e-30 maximum iterations=10000, initial guess is zero tolerances: relative=1e-05, absolute=1e-50, divergence=10000. left preconditioning using DEFAULT norm type for convergence test PC Object: (fieldsplit_0_) 1 MPI processes type: lu PC has not been set up so information may be incomplete out-of-place factorization tolerance for zero pivot 2.22045e-14 matrix ordering: nd linear system matrix = precond matrix: Mat Object: (fieldsplit_0_) 1 MPI processes type: seqaijcusparse rows=8396802, cols=8396802 total: nonzeros=75440146, allocated nonzeros=75440146 total number of mallocs used during MatSetValues calls=0 not using I-node routines A01 Mat Object: 1 MPI processes type: seqaijcusparse rows=8396802, cols=1050625 total: nonzeros=43995146, allocated nonzeros=43995146 total number of mallocs used during MatSetValues calls=0 not using I-node routines linear system matrix = precond matrix: Mat Object: 1 MPI processes type: seqaijcusparse rows=9447427, cols=9447427 total: nonzeros=163495966, allocated nonzeros=163643398 total number of mallocs used during MatSetValues calls=0 not using I-node routines Regards JT -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at petsc.dev Sun Jul 19 00:26:22 2020 From: bsmith at petsc.dev (Barry Smith) Date: Sun, 19 Jul 2020 00:26:22 -0500 Subject: [petsc-users] PetscLogFlops Error In-Reply-To: References: Message-ID: <43499BB0-1B0D-455D-A164-34AABBE27591@petsc.dev> > On Jul 18, 2020, at 11:27 PM, Karl Yang wrote: > > > Hello, > > I was using FGMRES solver with fieldsplit preconditioner for solving a fluid equation. > I have the code ran for coarser meshes and it works fine. But when I got it to run for very fine meshes I encountered the PetscLogFlops error. > > The following is the output of error and the output from kspview. > > [7]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- > [7]PETSC ERROR: Argument out of range > [7]PETSC ERROR: Cannot log negative flops > [7]PETSC ERROR: See https://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. > [7]PETSC ERROR: Petsc Release Version 3.12.5, Mar, 29, 2020 > [7]PETSC ERROR: ./testSolve on a arch-linux2-c-debug named hsw222 by yjuntao Sat Jul 18 20:30:05 2020 > [7]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --download-mpich --download-fblaslapack --with-cuda > [7]PETSC ERROR: #1 PetscLogFlops() line 232 in /usr/local/petsc/petsc-3.12.5/include/petsclog.h > [7]PETSC ERROR: #2 MatSolve_SeqAIJ() line 3200 in /usr/local/petsc/petsc-3.12.5/src/mat/impls/aij/seq/aijfact.c > [7]PETSC ERROR: #3 MatSolve() line 3290 in /usr/local/petsc/petsc-3.12.5/src/mat/interface/matrix.c > [7]PETSC ERROR: #4 PCApply_LU() line 177 in /usr/local/petsc/petsc-3.12.5/src/ksp/pc/impls/factor/lu/lu.c > [7]PETSC ERROR: #5 PCApply() line 444 in /usr/local/petsc/petsc-3.12.5/src/ksp/pc/interface/precon.c > [7]PETSC ERROR: #6 KSP_PCApply() line 281 in /usr/local/petsc/petsc-3.12.5/include/petsc/private/kspimpl.h > [7]PETSC ERROR: #7 KSPInitialResidual() line 65 in /usr/local/petsc/petsc-3.12.5/src/ksp/ksp/interface/itres.c > [7]PETSC ERROR: #8 KSPSolve_GMRES() line 236 in /usr/local/petsc/petsc-3.12.5/src/ksp/ksp/impls/gmres/gmres.c > [7]PETSC ERROR: #9 KSPSolve() line 760 in /usr/local/petsc/petsc-3.12.5/src/ksp/ksp/interface/itfunc.c > [7]PETSC ERROR: #10 PCApply_FieldSplit_Schur() line 1180 in /usr/local/petsc/petsc-3.12.5/src/ksp/pc/impls/fieldsplit/fieldsplit.c > [7]PETSC ERROR: #11 PCApply() line 444 in /usr/local/petsc/petsc-3.12.5/src/ksp/pc/interface/precon.c > [7]PETSC ERROR: #12 KSP_PCApply() line 281 in /usr/local/petsc/petsc-3.12.5/include/petsc/private/kspimpl.h > [7]PETSC ERROR: #13 KSPFGMRESCycle() line 166 in /usr/local/petsc/petsc-3.12.5/src/ksp/ksp/impls/gmres/fgmres/fgmres.c > [7]PETSC ERROR: #14 KSPSolve_FGMRES() line 291 in /usr/local/petsc/petsc-3.12.5/src/ksp/ksp/impls/gmres/fgmres/fgmres.c > [7]PETSC ERROR: #15 KSPSolve() line 760 in /usr/local/petsc/petsc-3.12.5/src/ksp/ksp/interface/itfunc.c > PetscErrorCode MatSolve_SeqAIJ(Mat A,Vec bb,Vec xx) { Mat_SeqAIJ *a = (Mat_SeqAIJ*)A->data; IS iscol = a->col,isrow = a->row; PetscErrorCode ierr; PetscInt i,n=A->rmap->n,*vi,*ai=a->i,*aj=a->j,*adiag = a->diag,nz; const PetscInt *rout,*cout,*r,*c; PetscScalar *x,*tmp,sum; const PetscScalar *b; const MatScalar *aa = a->a,*v; PetscFunctionBegin; if (!n) PetscFunctionReturn(0); ...... ierr = PetscLogFlops(2*a->nz - A->cmap->n);CHKERRQ(ierr); PetscFunctionReturn(0); Could maybe be integer overflow where a->nz fits but 2*a->nz doesn't fit so it results in a negative integer that is then converted to a negative floating point number. It should have a 2.0*a->nz but we must have missed this place in the code. We can fix this. > There is some issues during kspview if PCLSC is used as shown below. But the code could be ran if I do not show ksp details. > In order to show the whole ksp setup, I also attached another kspview output without PCLSC. > ======================kspview output ============================== > KSP Object: 1 MPI processes > type: fgmres > restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement > happy breakdown tolerance 1e-30 > maximum iterations=10000, initial guess is zero > tolerances: relative=1e-08, absolute=1e-50, divergence=10000. > right preconditioning > using UNPRECONDITIONED norm type for convergence test > PC Object: 1 MPI processes > type: fieldsplit > FieldSplit with Schur preconditioner, blocksize = 1, factorization FULL > Preconditioner for the Schur complement formed from S itself > Split info: > Split number 0 Defined by IS > Split number 1 Defined by IS > KSP solver for A00 block > KSP Object: (fieldsplit_0_) 1 MPI processes > type: gmres > restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement > happy breakdown tolerance 1e-30 > maximum iterations=10000, initial guess is zero > tolerances: relative=1e-05, absolute=1e-50, divergence=10000. > left preconditioning > using DEFAULT norm type for convergence test > PC Object: (fieldsplit_0_) 1 MPI processes > type: lu > PC has not been set up so information may be incomplete > out-of-place factorization > tolerance for zero pivot 2.22045e-14 > matrix ordering: nd > linear system matrix = precond matrix: > Mat Object: (fieldsplit_0_) 1 MPI processes > type: seqaijcusparse > rows=8396802, cols=8396802 > total: nonzeros=75440146, allocated nonzeros=75440146 > total number of mallocs used during MatSetValues calls=0 > not using I-node routines > KSP solver for S = A11 - A10 inv(A00) A01 > KSP Object: (fieldsplit_1_) 1 MPI processes > type: gmres > restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement > happy breakdown tolerance 1e-30 > maximum iterations=10000, initial guess is zero > tolerances: relative=1e-05, absolute=1e-50, divergence=10000. > left preconditioning > using DEFAULT norm type for convergence test > PC Object: (fieldsplit_1_) 1 MPI processes > type: lsc > PC has not been set up so information may be incomplete > [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- > [0]PETSC ERROR: Null argument, when expecting valid pointer > [0]PETSC ERROR: Null Object: Parameter # 1 > [0]PETSC ERROR: See https://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. > [0]PETSC ERROR: Petsc Release Version 3.12.5, Mar, 29, 2020 > [0]PETSC ERROR: ./testSolve on a arch-linux2-c-debug named hsw221 by yjuntao Sat Jul 18 21:08:42 2020 > [0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --download-mpich --download-fblaslapack --with-cuda > [0]PETSC ERROR: #1 KSPView() line 111 in /usr/local/petsc/petsc-3.12.5/src/ksp/ksp/interface/itcreate.c > [0]PETSC ERROR: #2 PCView_LSC() line 142 in /usr/local/petsc/petsc-3.12.5/src/ksp/pc/impls/lsc/lsc.c > [0]PETSC ERROR: #3 PCView() line 1588 in /usr/local/petsc/petsc-3.12.5/src/ksp/pc/interface/precon.c > [0]PETSC ERROR: #4 KSPView() line 217 in /usr/local/petsc/petsc-3.12.5/src/ksp/ksp/interface/itcreate.c > [0]PETSC ERROR: #5 PCView_FieldSplit_Schur() line 238 in /usr/local/petsc/petsc-3.12.5/src/ksp/pc/impls/fieldsplit/fieldsplit.c > [0]PETSC ERROR: #6 PCView() line 1588 in /usr/local/petsc/petsc-3.12.5/src/ksp/pc/interface/precon.c > [0]PETSC ERROR: #7 KSPView() line 217 in /usr/local/petsc/petsc-3.12.5/src/ksp/ksp/interface/itcreate.c This is something we can fix. Since the LSC has not been setup yet it should skip the KSPView on the KSP that has not yet been created. static PetscErrorCode PCView_LSC(PC pc,PetscViewer viewer) { PC_LSC *jac = (PC_LSC*)pc->data; PetscErrorCode ierr; PetscBool iascii; PetscFunctionBegin; ierr = PetscObjectTypeCompare((PetscObject)viewer,PETSCVIEWERASCII,&iascii);CHKERRQ(ierr); if (iascii) { ierr = PetscViewerASCIIPushTab(viewer);CHKERRQ(ierr); ierr = KSPView(jac->kspL,viewer);CHKERRQ(ierr); ierr = PetscViewerASCIIPopTab(viewer);CHKERRQ(ierr); } PetscFunctionReturn(0); } > > =====================KSP without PCLSC====================================== > KSP Object: 1 MPI processes > type: fgmres > restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement > happy breakdown tolerance 1e-30 > maximum iterations=10000, initial guess is zero > tolerances: relative=1e-08, absolute=1e-50, divergence=10000. > right preconditioning > using UNPRECONDITIONED norm type for convergence test > PC Object: 1 MPI processes > type: fieldsplit > FieldSplit with Schur preconditioner, blocksize = 1, factorization FULL > Preconditioner for the Schur complement formed from S itself > Split info: > Split number 0 Defined by IS > Split number 1 Defined by IS > KSP solver for A00 block > KSP Object: (fieldsplit_0_) 1 MPI processes > type: gmres > restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement > happy breakdown tolerance 1e-30 > maximum iterations=10000, initial guess is zero > tolerances: relative=1e-05, absolute=1e-50, divergence=10000. > left preconditioning > using DEFAULT norm type for convergence test > PC Object: (fieldsplit_0_) 1 MPI processes > type: lu > PC has not been set up so information may be incomplete > out-of-place factorization > tolerance for zero pivot 2.22045e-14 > matrix ordering: nd > linear system matrix = precond matrix: > Mat Object: (fieldsplit_0_) 1 MPI processes > type: seqaijcusparse > rows=8396802, cols=8396802 > total: nonzeros=75440146, allocated nonzeros=75440146 > total number of mallocs used during MatSetValues calls=0 > not using I-node routines > KSP solver for S = A11 - A10 inv(A00) A01 > KSP Object: (fieldsplit_1_) 1 MPI processes > type: gmres > restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement > happy breakdown tolerance 1e-30 > maximum iterations=10000, initial guess is zero > tolerances: relative=1e-05, absolute=1e-50, divergence=10000. > left preconditioning > using DEFAULT norm type for convergence test > PC Object: (fieldsplit_1_) 1 MPI processes > type: none > PC has not been set up so information may be incomplete > linear system matrix = precond matrix: > Mat Object: (fieldsplit_1_) 1 MPI processes > type: schurcomplement > rows=1050625, cols=1050625 > Schur complement A11 - A10 inv(A00) A01 > A11 > Mat Object: (fieldsplit_1_) 1 MPI processes > type: seqaijcusparse > rows=1050625, cols=1050625 > total: nonzeros=0, allocated nonzeros=0 > total number of mallocs used during MatSetValues calls=0 > not using I-node routines > A10 > Mat Object: 1 MPI processes > type: seqaijcusparse > rows=1050625, cols=8396802 > total: nonzeros=44060674, allocated nonzeros=44060674 > total number of mallocs used during MatSetValues calls=0 > not using I-node routines > KSP of A00 > KSP Object: (fieldsplit_0_) 1 MPI processes > type: gmres > restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement > happy breakdown tolerance 1e-30 > maximum iterations=10000, initial guess is zero > tolerances: relative=1e-05, absolute=1e-50, divergence=10000. > left preconditioning > using DEFAULT norm type for convergence test > PC Object: (fieldsplit_0_) 1 MPI processes > type: lu > PC has not been set up so information may be incomplete > out-of-place factorization > tolerance for zero pivot 2.22045e-14 > matrix ordering: nd > linear system matrix = precond matrix: > Mat Object: (fieldsplit_0_) 1 MPI processes > type: seqaijcusparse > rows=8396802, cols=8396802 > total: nonzeros=75440146, allocated nonzeros=75440146 > total number of mallocs used during MatSetValues calls=0 > not using I-node routines > A01 > Mat Object: 1 MPI processes > type: seqaijcusparse > rows=8396802, cols=1050625 > total: nonzeros=43995146, allocated nonzeros=43995146 > total number of mallocs used during MatSetValues calls=0 > not using I-node routines > linear system matrix = precond matrix: > Mat Object: 1 MPI processes > type: seqaijcusparse > rows=9447427, cols=9447427 > total: nonzeros=163495966, allocated nonzeros=163643398 > total number of mallocs used during MatSetValues calls=0 > not using I-node routines > > Regards > JT From y.juntao at hotmail.com Sun Jul 19 01:52:29 2020 From: y.juntao at hotmail.com (Karl Yang) Date: Sun, 19 Jul 2020 14:52:29 +0800 Subject: [petsc-users] PetscLogFlops Error In-Reply-To: <43499BB0-1B0D-455D-A164-34AABBE27591@petsc.dev> References: <43499BB0-1B0D-455D-A164-34AABBE27591@petsc.dev> Message-ID: Hi, Barry, Thanks for your reply. I see the issue and I think I can now continue my work with a quick fix as you suggested. Regards JT On Jul 19 2020, at 1:26 pm, Barry Smith wrote: > > > > On Jul 18, 2020, at 11:27 PM, Karl Yang wrote: > > > > > > Hello, > > > > I was using FGMRES solver with fieldsplit preconditioner for solving a fluid equation. > > I have the code ran for coarser meshes and it works fine. But when I got it to run for very fine meshes I encountered the PetscLogFlops error. > > > > The following is the output of error and the output from kspview. > > > > [7]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- > > [7]PETSC ERROR: Argument out of range > > [7]PETSC ERROR: Cannot log negative flops > > [7]PETSC ERROR: See https://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. > > [7]PETSC ERROR: Petsc Release Version 3.12.5, Mar, 29, 2020 > > [7]PETSC ERROR: ./testSolve on a arch-linux2-c-debug named hsw222 by yjuntao Sat Jul 18 20:30:05 2020 > > [7]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --download-mpich --download-fblaslapack --with-cuda > > [7]PETSC ERROR: #1 PetscLogFlops() line 232 in /usr/local/petsc/petsc-3.12.5/include/petsclog.h > > [7]PETSC ERROR: #2 MatSolve_SeqAIJ() line 3200 in /usr/local/petsc/petsc-3.12.5/src/mat/impls/aij/seq/aijfact.c > > [7]PETSC ERROR: #3 MatSolve() line 3290 in /usr/local/petsc/petsc-3.12.5/src/mat/interface/matrix.c > > [7]PETSC ERROR: #4 PCApply_LU() line 177 in /usr/local/petsc/petsc-3.12.5/src/ksp/pc/impls/factor/lu/lu.c > > [7]PETSC ERROR: #5 PCApply() line 444 in /usr/local/petsc/petsc-3.12.5/src/ksp/pc/interface/precon.c > > [7]PETSC ERROR: #6 KSP_PCApply() line 281 in /usr/local/petsc/petsc-3.12.5/include/petsc/private/kspimpl.h > > [7]PETSC ERROR: #7 KSPInitialResidual() line 65 in /usr/local/petsc/petsc-3.12.5/src/ksp/ksp/interface/itres.c > > [7]PETSC ERROR: #8 KSPSolve_GMRES() line 236 in /usr/local/petsc/petsc-3.12.5/src/ksp/ksp/impls/gmres/gmres.c > > [7]PETSC ERROR: #9 KSPSolve() line 760 in /usr/local/petsc/petsc-3.12.5/src/ksp/ksp/interface/itfunc.c > > [7]PETSC ERROR: #10 PCApply_FieldSplit_Schur() line 1180 in /usr/local/petsc/petsc-3.12.5/src/ksp/pc/impls/fieldsplit/fieldsplit.c > > [7]PETSC ERROR: #11 PCApply() line 444 in /usr/local/petsc/petsc-3.12.5/src/ksp/pc/interface/precon.c > > [7]PETSC ERROR: #12 KSP_PCApply() line 281 in /usr/local/petsc/petsc-3.12.5/include/petsc/private/kspimpl.h > > [7]PETSC ERROR: #13 KSPFGMRESCycle() line 166 in /usr/local/petsc/petsc-3.12.5/src/ksp/ksp/impls/gmres/fgmres/fgmres.c > > [7]PETSC ERROR: #14 KSPSolve_FGMRES() line 291 in /usr/local/petsc/petsc-3.12.5/src/ksp/ksp/impls/gmres/fgmres/fgmres.c > > [7]PETSC ERROR: #15 KSPSolve() line 760 in /usr/local/petsc/petsc-3.12.5/src/ksp/ksp/interface/itfunc.c > > > > PetscErrorCode MatSolve_SeqAIJ(Mat A,Vec bb,Vec xx) > { > Mat_SeqAIJ *a = (Mat_SeqAIJ*)A->data; > IS iscol = a->col,isrow = a->row; > PetscErrorCode ierr; > PetscInt i,n=A->rmap->n,*vi,*ai=a->i,*aj=a->j,*adiag = a->diag,nz; > const PetscInt *rout,*cout,*r,*c; > PetscScalar *x,*tmp,sum; > const PetscScalar *b; > const MatScalar *aa = a->a,*v; > > PetscFunctionBegin; > if (!n) PetscFunctionReturn(0); > > ...... > ierr = PetscLogFlops(2*a->nz - A->cmap->n);CHKERRQ(ierr); > PetscFunctionReturn(0); > > Could maybe be integer overflow where a->nz fits but 2*a->nz doesn't fit so it results in a negative integer that is then converted to a negative floating point number. > It should have a 2.0*a->nz but we must have missed this place in the code. We can fix this. > > There is some issues during kspview if PCLSC is used as shown below. But the code could be ran if I do not show ksp details. > > In order to show the whole ksp setup, I also attached another kspview output without PCLSC. > > ======================kspview output ============================== > > KSP Object: 1 MPI processes > > type: fgmres > > restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement > > happy breakdown tolerance 1e-30 > > maximum iterations=10000, initial guess is zero > > tolerances: relative=1e-08, absolute=1e-50, divergence=10000. > > right preconditioning > > using UNPRECONDITIONED norm type for convergence test > > PC Object: 1 MPI processes > > type: fieldsplit > > FieldSplit with Schur preconditioner, blocksize = 1, factorization FULL > > Preconditioner for the Schur complement formed from S itself > > Split info: > > Split number 0 Defined by IS > > Split number 1 Defined by IS > > KSP solver for A00 block > > KSP Object: (fieldsplit_0_) 1 MPI processes > > type: gmres > > restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement > > happy breakdown tolerance 1e-30 > > maximum iterations=10000, initial guess is zero > > tolerances: relative=1e-05, absolute=1e-50, divergence=10000. > > left preconditioning > > using DEFAULT norm type for convergence test > > PC Object: (fieldsplit_0_) 1 MPI processes > > type: lu > > PC has not been set up so information may be incomplete > > out-of-place factorization > > tolerance for zero pivot 2.22045e-14 > > matrix ordering: nd > > linear system matrix = precond matrix: > > Mat Object: (fieldsplit_0_) 1 MPI processes > > type: seqaijcusparse > > rows=8396802, cols=8396802 > > total: nonzeros=75440146, allocated nonzeros=75440146 > > total number of mallocs used during MatSetValues calls=0 > > not using I-node routines > > KSP solver for S = A11 - A10 inv(A00) A01 > > KSP Object: (fieldsplit_1_) 1 MPI processes > > type: gmres > > restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement > > happy breakdown tolerance 1e-30 > > maximum iterations=10000, initial guess is zero > > tolerances: relative=1e-05, absolute=1e-50, divergence=10000. > > left preconditioning > > using DEFAULT norm type for convergence test > > PC Object: (fieldsplit_1_) 1 MPI processes > > type: lsc > > PC has not been set up so information may be incomplete > > [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- > > [0]PETSC ERROR: Null argument, when expecting valid pointer > > [0]PETSC ERROR: Null Object: Parameter # 1 > > [0]PETSC ERROR: See https://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. > > [0]PETSC ERROR: Petsc Release Version 3.12.5, Mar, 29, 2020 > > [0]PETSC ERROR: ./testSolve on a arch-linux2-c-debug named hsw221 by yjuntao Sat Jul 18 21:08:42 2020 > > [0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --download-mpich --download-fblaslapack --with-cuda > > [0]PETSC ERROR: #1 KSPView() line 111 in /usr/local/petsc/petsc-3.12.5/src/ksp/ksp/interface/itcreate.c > > [0]PETSC ERROR: #2 PCView_LSC() line 142 in /usr/local/petsc/petsc-3.12.5/src/ksp/pc/impls/lsc/lsc.c > > [0]PETSC ERROR: #3 PCView() line 1588 in /usr/local/petsc/petsc-3.12.5/src/ksp/pc/interface/precon.c > > [0]PETSC ERROR: #4 KSPView() line 217 in /usr/local/petsc/petsc-3.12.5/src/ksp/ksp/interface/itcreate.c > > [0]PETSC ERROR: #5 PCView_FieldSplit_Schur() line 238 in /usr/local/petsc/petsc-3.12.5/src/ksp/pc/impls/fieldsplit/fieldsplit.c > > [0]PETSC ERROR: #6 PCView() line 1588 in /usr/local/petsc/petsc-3.12.5/src/ksp/pc/interface/precon.c > > [0]PETSC ERROR: #7 KSPView() line 217 in /usr/local/petsc/petsc-3.12.5/src/ksp/ksp/interface/itcreate.c > > This is something we can fix. Since the LSC has not been setup yet it should skip the KSPView on the KSP that has not yet been created. > static PetscErrorCode PCView_LSC(PC pc,PetscViewer viewer) > { > PC_LSC *jac = (PC_LSC*)pc->data; > PetscErrorCode ierr; > PetscBool iascii; > > PetscFunctionBegin; > ierr = PetscObjectTypeCompare((PetscObject)viewer,PETSCVIEWERASCII,&iascii);CHKERRQ(ierr); > if (iascii) { > ierr = PetscViewerASCIIPushTab(viewer);CHKERRQ(ierr); > ierr = KSPView(jac->kspL,viewer);CHKERRQ(ierr); > ierr = PetscViewerASCIIPopTab(viewer);CHKERRQ(ierr); > } > PetscFunctionReturn(0); > } > > > > > =====================KSP without PCLSC====================================== > > KSP Object: 1 MPI processes > > type: fgmres > > restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement > > happy breakdown tolerance 1e-30 > > maximum iterations=10000, initial guess is zero > > tolerances: relative=1e-08, absolute=1e-50, divergence=10000. > > right preconditioning > > using UNPRECONDITIONED norm type for convergence test > > PC Object: 1 MPI processes > > type: fieldsplit > > FieldSplit with Schur preconditioner, blocksize = 1, factorization FULL > > Preconditioner for the Schur complement formed from S itself > > Split info: > > Split number 0 Defined by IS > > Split number 1 Defined by IS > > KSP solver for A00 block > > KSP Object: (fieldsplit_0_) 1 MPI processes > > type: gmres > > restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement > > happy breakdown tolerance 1e-30 > > maximum iterations=10000, initial guess is zero > > tolerances: relative=1e-05, absolute=1e-50, divergence=10000. > > left preconditioning > > using DEFAULT norm type for convergence test > > PC Object: (fieldsplit_0_) 1 MPI processes > > type: lu > > PC has not been set up so information may be incomplete > > out-of-place factorization > > tolerance for zero pivot 2.22045e-14 > > matrix ordering: nd > > linear system matrix = precond matrix: > > Mat Object: (fieldsplit_0_) 1 MPI processes > > type: seqaijcusparse > > rows=8396802, cols=8396802 > > total: nonzeros=75440146, allocated nonzeros=75440146 > > total number of mallocs used during MatSetValues calls=0 > > not using I-node routines > > KSP solver for S = A11 - A10 inv(A00) A01 > > KSP Object: (fieldsplit_1_) 1 MPI processes > > type: gmres > > restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement > > happy breakdown tolerance 1e-30 > > maximum iterations=10000, initial guess is zero > > tolerances: relative=1e-05, absolute=1e-50, divergence=10000. > > left preconditioning > > using DEFAULT norm type for convergence test > > PC Object: (fieldsplit_1_) 1 MPI processes > > type: none > > PC has not been set up so information may be incomplete > > linear system matrix = precond matrix: > > Mat Object: (fieldsplit_1_) 1 MPI processes > > type: schurcomplement > > rows=1050625, cols=1050625 > > Schur complement A11 - A10 inv(A00) A01 > > A11 > > Mat Object: (fieldsplit_1_) 1 MPI processes > > type: seqaijcusparse > > rows=1050625, cols=1050625 > > total: nonzeros=0, allocated nonzeros=0 > > total number of mallocs used during MatSetValues calls=0 > > not using I-node routines > > A10 > > Mat Object: 1 MPI processes > > type: seqaijcusparse > > rows=1050625, cols=8396802 > > total: nonzeros=44060674, allocated nonzeros=44060674 > > total number of mallocs used during MatSetValues calls=0 > > not using I-node routines > > KSP of A00 > > KSP Object: (fieldsplit_0_) 1 MPI processes > > type: gmres > > restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement > > happy breakdown tolerance 1e-30 > > maximum iterations=10000, initial guess is zero > > tolerances: relative=1e-05, absolute=1e-50, divergence=10000. > > left preconditioning > > using DEFAULT norm type for convergence test > > PC Object: (fieldsplit_0_) 1 MPI processes > > type: lu > > PC has not been set up so information may be incomplete > > out-of-place factorization > > tolerance for zero pivot 2.22045e-14 > > matrix ordering: nd > > linear system matrix = precond matrix: > > Mat Object: (fieldsplit_0_) 1 MPI processes > > type: seqaijcusparse > > rows=8396802, cols=8396802 > > total: nonzeros=75440146, allocated nonzeros=75440146 > > total number of mallocs used during MatSetValues calls=0 > > not using I-node routines > > A01 > > Mat Object: 1 MPI processes > > type: seqaijcusparse > > rows=8396802, cols=1050625 > > total: nonzeros=43995146, allocated nonzeros=43995146 > > total number of mallocs used during MatSetValues calls=0 > > not using I-node routines > > linear system matrix = precond matrix: > > Mat Object: 1 MPI processes > > type: seqaijcusparse > > rows=9447427, cols=9447427 > > total: nonzeros=163495966, allocated nonzeros=163643398 > > total number of mallocs used during MatSetValues calls=0 > > not using I-node routines > > > > Regards > > JT > -------------- next part -------------- An HTML attachment was scrubbed... URL: From eda.oktay at metu.edu.tr Sun Jul 19 09:58:46 2020 From: eda.oktay at metu.edu.tr (Eda Oktay) Date: Sun, 19 Jul 2020 17:58:46 +0300 Subject: [petsc-users] SLEPc Build Error Message-ID: Hi all, I am trying to install petsc 3.13.2 to my computer (fedora 25) with slepc and some other libraries by using the following code: ./configure --with-cc=gcc --with-cxx=g++ --download-mpich --download-openblas --download-slepc --download-metis --download-parmetis --download-chaco --with-X=1 Although I installed exact same things to an other computer with fedora 25, this time, after configuration, when I am asked to use "make PETSC_DIR=/home/eda/petsc-3.13.2 PETSC_ARCH=arch-linux2-c-debug all", I got an Slepc error: *** Building slepc *** **************************ERROR************************************* Error building slepc. Check arch-linux2-c-debug/lib/petsc/conf/slepc.log ******************************************************************** /home/eda/petsc-3.13.2/arch-linux2-c-debug/lib/petsc/conf/petscrules:32: recipe for target 'slepcbuild' failed gmake[2]: *** [slepcbuild] Error 1 **************************ERROR************************************* Error during compile, check arch-linux2-c-debug/lib/petsc/conf/make.log Send it and arch-linux2-c-debug/lib/petsc/conf/configure.log to petsc-maint at mcs.anl.gov ******************************************************************** makefile:33: recipe for target 'all' failed make[1]: *** [all] Error 1 GNUmakefile:9: recipe for target 'all' failed make: *** [all] Error 2 I attached slepc.log, make.log and configure.log. What is the problem? Can you help me please? Thanks a lot! Eda -------------- next part -------------- A non-text attachment was scrubbed... Name: make.log Type: text/x-log Size: 12558 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: configure.log Type: text/x-log Size: 7201505 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: slepc.log Type: text/x-log Size: 462 bytes Desc: not available URL: From bsmith at petsc.dev Sun Jul 19 11:25:53 2020 From: bsmith at petsc.dev (Barry Smith) Date: Sun, 19 Jul 2020 11:25:53 -0500 Subject: [petsc-users] PetscLogFlops Error In-Reply-To: References: <43499BB0-1B0D-455D-A164-34AABBE27591@petsc.dev> Message-ID: JT, These fixes will be merged into PETSc maint soon. https://gitlab.com/petsc/petsc/-/merge_requests/2986 Barry > On Jul 19, 2020, at 1:52 AM, Karl Yang wrote: > > Hi, Barry, > > Thanks for your reply. I see the issue and I think I can now continue my work with a quick fix as you suggested. > > Regards > JT > > On Jul 19 2020, at 1:26 pm, Barry Smith wrote: > > > > On Jul 18, 2020, at 11:27 PM, Karl Yang wrote: > > > > > > Hello, > > > > I was using FGMRES solver with fieldsplit preconditioner for solving a fluid equation. > > I have the code ran for coarser meshes and it works fine. But when I got it to run for very fine meshes I encountered the PetscLogFlops error. > > > > The following is the output of error and the output from kspview. > > > > [7]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- > > [7]PETSC ERROR: Argument out of range > > [7]PETSC ERROR: Cannot log negative flops > > [7]PETSC ERROR: See https://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. > > [7]PETSC ERROR: Petsc Release Version 3.12.5, Mar, 29, 2020 > > [7]PETSC ERROR: ./testSolve on a arch-linux2-c-debug named hsw222 by yjuntao Sat Jul 18 20:30:05 2020 > > [7]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --download-mpich --download-fblaslapack --with-cuda > > [7]PETSC ERROR: #1 PetscLogFlops() line 232 in /usr/local/petsc/petsc-3.12.5/include/petsclog.h > > [7]PETSC ERROR: #2 MatSolve_SeqAIJ() line 3200 in /usr/local/petsc/petsc-3.12.5/src/mat/impls/aij/seq/aijfact.c > > [7]PETSC ERROR: #3 MatSolve() line 3290 in /usr/local/petsc/petsc-3.12.5/src/mat/interface/matrix.c > > [7]PETSC ERROR: #4 PCApply_LU() line 177 in /usr/local/petsc/petsc-3.12.5/src/ksp/pc/impls/factor/lu/lu.c > > [7]PETSC ERROR: #5 PCApply() line 444 in /usr/local/petsc/petsc-3.12.5/src/ksp/pc/interface/precon.c > > [7]PETSC ERROR: #6 KSP_PCApply() line 281 in /usr/local/petsc/petsc-3.12.5/include/petsc/private/kspimpl.h > > [7]PETSC ERROR: #7 KSPInitialResidual() line 65 in /usr/local/petsc/petsc-3.12.5/src/ksp/ksp/interface/itres.c > > [7]PETSC ERROR: #8 KSPSolve_GMRES() line 236 in /usr/local/petsc/petsc-3.12.5/src/ksp/ksp/impls/gmres/gmres.c > > [7]PETSC ERROR: #9 KSPSolve() line 760 in /usr/local/petsc/petsc-3.12.5/src/ksp/ksp/interface/itfunc.c > > [7]PETSC ERROR: #10 PCApply_FieldSplit_Schur() line 1180 in /usr/local/petsc/petsc-3.12.5/src/ksp/pc/impls/fieldsplit/fieldsplit.c > > [7]PETSC ERROR: #11 PCApply() line 444 in /usr/local/petsc/petsc-3.12.5/src/ksp/pc/interface/precon.c > > [7]PETSC ERROR: #12 KSP_PCApply() line 281 in /usr/local/petsc/petsc-3.12.5/include/petsc/private/kspimpl.h > > [7]PETSC ERROR: #13 KSPFGMRESCycle() line 166 in /usr/local/petsc/petsc-3.12.5/src/ksp/ksp/impls/gmres/fgmres/fgmres.c > > [7]PETSC ERROR: #14 KSPSolve_FGMRES() line 291 in /usr/local/petsc/petsc-3.12.5/src/ksp/ksp/impls/gmres/fgmres/fgmres.c > > [7]PETSC ERROR: #15 KSPSolve() line 760 in /usr/local/petsc/petsc-3.12.5/src/ksp/ksp/interface/itfunc.c > > > > PetscErrorCode MatSolve_SeqAIJ(Mat A,Vec bb,Vec xx) > { > Mat_SeqAIJ *a = (Mat_SeqAIJ*)A->data; > IS iscol = a->col,isrow = a->row; > PetscErrorCode ierr; > PetscInt i,n=A->rmap->n,*vi,*ai=a->i,*aj=a->j,*adiag = a->diag,nz; > const PetscInt *rout,*cout,*r,*c; > PetscScalar *x,*tmp,sum; > const PetscScalar *b; > const MatScalar *aa = a->a,*v; > > PetscFunctionBegin; > if (!n) PetscFunctionReturn(0); > > ...... > > ierr = PetscLogFlops(2*a->nz - A->cmap->n);CHKERRQ(ierr); > PetscFunctionReturn(0); > > Could maybe be integer overflow where a->nz fits but 2*a->nz doesn't fit so it results in a negative integer that is then converted to a negative floating point number. > > It should have a 2.0*a->nz but we must have missed this place in the code. We can fix this. > > > There is some issues during kspview if PCLSC is used as shown below. But the code could be ran if I do not show ksp details. > > In order to show the whole ksp setup, I also attached another kspview output without PCLSC. > > ======================kspview output ============================== > > KSP Object: 1 MPI processes > > type: fgmres > > restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement > > happy breakdown tolerance 1e-30 > > maximum iterations=10000, initial guess is zero > > tolerances: relative=1e-08, absolute=1e-50, divergence=10000. > > right preconditioning > > using UNPRECONDITIONED norm type for convergence test > > PC Object: 1 MPI processes > > type: fieldsplit > > FieldSplit with Schur preconditioner, blocksize = 1, factorization FULL > > Preconditioner for the Schur complement formed from S itself > > Split info: > > Split number 0 Defined by IS > > Split number 1 Defined by IS > > KSP solver for A00 block > > KSP Object: (fieldsplit_0_) 1 MPI processes > > type: gmres > > restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement > > happy breakdown tolerance 1e-30 > > maximum iterations=10000, initial guess is zero > > tolerances: relative=1e-05, absolute=1e-50, divergence=10000. > > left preconditioning > > using DEFAULT norm type for convergence test > > PC Object: (fieldsplit_0_) 1 MPI processes > > type: lu > > PC has not been set up so information may be incomplete > > out-of-place factorization > > tolerance for zero pivot 2.22045e-14 > > matrix ordering: nd > > linear system matrix = precond matrix: > > Mat Object: (fieldsplit_0_) 1 MPI processes > > type: seqaijcusparse > > rows=8396802, cols=8396802 > > total: nonzeros=75440146, allocated nonzeros=75440146 > > total number of mallocs used during MatSetValues calls=0 > > not using I-node routines > > KSP solver for S = A11 - A10 inv(A00) A01 > > KSP Object: (fieldsplit_1_) 1 MPI processes > > type: gmres > > restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement > > happy breakdown tolerance 1e-30 > > maximum iterations=10000, initial guess is zero > > tolerances: relative=1e-05, absolute=1e-50, divergence=10000. > > left preconditioning > > using DEFAULT norm type for convergence test > > PC Object: (fieldsplit_1_) 1 MPI processes > > type: lsc > > PC has not been set up so information may be incomplete > > [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- > > [0]PETSC ERROR: Null argument, when expecting valid pointer > > [0]PETSC ERROR: Null Object: Parameter # 1 > > [0]PETSC ERROR: See https://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. > > [0]PETSC ERROR: Petsc Release Version 3.12.5, Mar, 29, 2020 > > [0]PETSC ERROR: ./testSolve on a arch-linux2-c-debug named hsw221 by yjuntao Sat Jul 18 21:08:42 2020 > > [0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --download-mpich --download-fblaslapack --with-cuda > > [0]PETSC ERROR: #1 KSPView() line 111 in /usr/local/petsc/petsc-3.12.5/src/ksp/ksp/interface/itcreate.c > > [0]PETSC ERROR: #2 PCView_LSC() line 142 in /usr/local/petsc/petsc-3.12.5/src/ksp/pc/impls/lsc/lsc.c > > [0]PETSC ERROR: #3 PCView() line 1588 in /usr/local/petsc/petsc-3.12.5/src/ksp/pc/interface/precon.c > > [0]PETSC ERROR: #4 KSPView() line 217 in /usr/local/petsc/petsc-3.12.5/src/ksp/ksp/interface/itcreate.c > > [0]PETSC ERROR: #5 PCView_FieldSplit_Schur() line 238 in /usr/local/petsc/petsc-3.12.5/src/ksp/pc/impls/fieldsplit/fieldsplit.c > > [0]PETSC ERROR: #6 PCView() line 1588 in /usr/local/petsc/petsc-3.12.5/src/ksp/pc/interface/precon.c > > [0]PETSC ERROR: #7 KSPView() line 217 in /usr/local/petsc/petsc-3.12.5/src/ksp/ksp/interface/itcreate.c > > This is something we can fix. Since the LSC has not been setup yet it should skip the KSPView on the KSP that has not yet been created. > > static PetscErrorCode PCView_LSC(PC pc,PetscViewer viewer) > { > PC_LSC *jac = (PC_LSC*)pc->data; > PetscErrorCode ierr; > PetscBool iascii; > > PetscFunctionBegin; > ierr = PetscObjectTypeCompare((PetscObject)viewer,PETSCVIEWERASCII,&iascii);CHKERRQ(ierr); > if (iascii) { > ierr = PetscViewerASCIIPushTab(viewer);CHKERRQ(ierr); > ierr = KSPView(jac->kspL,viewer);CHKERRQ(ierr); > ierr = PetscViewerASCIIPopTab(viewer);CHKERRQ(ierr); > } > PetscFunctionReturn(0); > } > > > > > =====================KSP without PCLSC====================================== > > KSP Object: 1 MPI processes > > type: fgmres > > restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement > > happy breakdown tolerance 1e-30 > > maximum iterations=10000, initial guess is zero > > tolerances: relative=1e-08, absolute=1e-50, divergence=10000. > > right preconditioning > > using UNPRECONDITIONED norm type for convergence test > > PC Object: 1 MPI processes > > type: fieldsplit > > FieldSplit with Schur preconditioner, blocksize = 1, factorization FULL > > Preconditioner for the Schur complement formed from S itself > > Split info: > > Split number 0 Defined by IS > > Split number 1 Defined by IS > > KSP solver for A00 block > > KSP Object: (fieldsplit_0_) 1 MPI processes > > type: gmres > > restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement > > happy breakdown tolerance 1e-30 > > maximum iterations=10000, initial guess is zero > > tolerances: relative=1e-05, absolute=1e-50, divergence=10000. > > left preconditioning > > using DEFAULT norm type for convergence test > > PC Object: (fieldsplit_0_) 1 MPI processes > > type: lu > > PC has not been set up so information may be incomplete > > out-of-place factorization > > tolerance for zero pivot 2.22045e-14 > > matrix ordering: nd > > linear system matrix = precond matrix: > > Mat Object: (fieldsplit_0_) 1 MPI processes > > type: seqaijcusparse > > rows=8396802, cols=8396802 > > total: nonzeros=75440146, allocated nonzeros=75440146 > > total number of mallocs used during MatSetValues calls=0 > > not using I-node routines > > KSP solver for S = A11 - A10 inv(A00) A01 > > KSP Object: (fieldsplit_1_) 1 MPI processes > > type: gmres > > restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement > > happy breakdown tolerance 1e-30 > > maximum iterations=10000, initial guess is zero > > tolerances: relative=1e-05, absolute=1e-50, divergence=10000. > > left preconditioning > > using DEFAULT norm type for convergence test > > PC Object: (fieldsplit_1_) 1 MPI processes > > type: none > > PC has not been set up so information may be incomplete > > linear system matrix = precond matrix: > > Mat Object: (fieldsplit_1_) 1 MPI processes > > type: schurcomplement > > rows=1050625, cols=1050625 > > Schur complement A11 - A10 inv(A00) A01 > > A11 > > Mat Object: (fieldsplit_1_) 1 MPI processes > > type: seqaijcusparse > > rows=1050625, cols=1050625 > > total: nonzeros=0, allocated nonzeros=0 > > total number of mallocs used during MatSetValues calls=0 > > not using I-node routines > > A10 > > Mat Object: 1 MPI processes > > type: seqaijcusparse > > rows=1050625, cols=8396802 > > total: nonzeros=44060674, allocated nonzeros=44060674 > > total number of mallocs used during MatSetValues calls=0 > > not using I-node routines > > KSP of A00 > > KSP Object: (fieldsplit_0_) 1 MPI processes > > type: gmres > > restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement > > happy breakdown tolerance 1e-30 > > maximum iterations=10000, initial guess is zero > > tolerances: relative=1e-05, absolute=1e-50, divergence=10000. > > left preconditioning > > using DEFAULT norm type for convergence test > > PC Object: (fieldsplit_0_) 1 MPI processes > > type: lu > > PC has not been set up so information may be incomplete > > out-of-place factorization > > tolerance for zero pivot 2.22045e-14 > > matrix ordering: nd > > linear system matrix = precond matrix: > > Mat Object: (fieldsplit_0_) 1 MPI processes > > type: seqaijcusparse > > rows=8396802, cols=8396802 > > total: nonzeros=75440146, allocated nonzeros=75440146 > > total number of mallocs used during MatSetValues calls=0 > > not using I-node routines > > A01 > > Mat Object: 1 MPI processes > > type: seqaijcusparse > > rows=8396802, cols=1050625 > > total: nonzeros=43995146, allocated nonzeros=43995146 > > total number of mallocs used during MatSetValues calls=0 > > not using I-node routines > > linear system matrix = precond matrix: > > Mat Object: 1 MPI processes > > type: seqaijcusparse > > rows=9447427, cols=9447427 > > total: nonzeros=163495966, allocated nonzeros=163643398 > > total number of mallocs used during MatSetValues calls=0 > > not using I-node routines > > > > Regards > > JT -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Sun Jul 19 12:49:16 2020 From: knepley at gmail.com (Matthew Knepley) Date: Sun, 19 Jul 2020 13:49:16 -0400 Subject: [petsc-users] SLEPc Build Error In-Reply-To: References: Message-ID: On Sun, Jul 19, 2020 at 11:01 AM Eda Oktay wrote: > Hi all, > > I am trying to install petsc 3.13.2 to my computer (fedora 25) with > slepc and some other libraries by using the following code: > > ./configure --with-cc=gcc --with-cxx=g++ --download-mpich > --download-openblas --download-slepc --download-metis > --download-parmetis --download-chaco --with-X=1 > > Although I installed exact same things to an other computer with > fedora 25, this time, after configuration, when I am asked to use > "make PETSC_DIR=/home/eda/petsc-3.13.2 PETSC_ARCH=arch-linux2-c-debug > all", I got an Slepc error: > PETSc thinks you are installing from a tarball, rather than from a Git clone, because it cannot find the $PETSC_DIR/lib/petsc/bin/maint directory, which we remove from the tarball. The tarball also contains the Fortran stubs, so we do not install Sowing in this case. When SLEPc is being installed, it needs to generate the Fortran stubs, since it was cloned. At bottom, you cannot install SLEPc automatically unless you install PETSc from a Git clone. Thanks, Matt > *** Building slepc *** > **************************ERROR************************************* > Error building slepc. Check arch-linux2-c-debug/lib/petsc/conf/slepc.log > ******************************************************************** > /home/eda/petsc-3.13.2/arch-linux2-c-debug/lib/petsc/conf/petscrules:32: > recipe for target 'slepcbuild' failed > gmake[2]: *** [slepcbuild] Error 1 > **************************ERROR************************************* > Error during compile, check arch-linux2-c-debug/lib/petsc/conf/make.log > Send it and arch-linux2-c-debug/lib/petsc/conf/configure.log to > petsc-maint at mcs.anl.gov > ******************************************************************** > makefile:33: recipe for target 'all' failed > make[1]: *** [all] Error 1 > GNUmakefile:9: recipe for target 'all' failed > make: *** [all] Error 2 > > I attached slepc.log, make.log and configure.log. What is the problem? > Can you help me please? > > Thanks a lot! > > Eda > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at petsc.dev Sun Jul 19 14:42:12 2020 From: bsmith at petsc.dev (Barry Smith) Date: Sun, 19 Jul 2020 14:42:12 -0500 Subject: [petsc-users] position posted at Argonne National Laboratory Message-ID: <74191C67-D197-4986-AA32-E1F031AD0906@petsc.dev> Some PETSc users or developers may be interested in this position posting at Argonne. From: Stefan Wild wild at anl.gov Date: July 19, 2020 Subject: Staff Position, Numerical PDEs and Scientific Computing, Argonne The Laboratory for Applied Mathematics, Numerical Software, and Statistics (LANS) in the Mathematics and Computer Science Division at Argonne National Laboratory is seeking a research staff member in the area of partial differential equations (PDEs), numerical software, and scientific computing. We value and strive for diversity in backgrounds, experiences, and perspectives. The staff member will lead a basic research program and build computational tools and methods for the solution of computational science problems on distributed/scalable parallel computers. The staff member will work with science and engineering application codes as well as math libraries such as PETSc. The appointment level will be commensurate with experience; US citizenship is not required. Senior applicants: https://bit.ly/2Otr7De Earlier-career applicants: https://bit.ly/2B1Rxch As an equal employment opportunity and affirmative action employer, Argonne National Laboratory is committed to a diverse and inclusive workplace that fosters collaborative scientific discovery and innovation. In support of this commitment, Argonne encourages minorities, women, veterans and individuals with disabilities to apply for employment. Argonne considers all qualified applicants for employment without regard to age, ancestry, citizenship status, color, disability, gender, gender identity, genetic information, marital status, national origin, pregnancy, race, religion, sexual orientation, veteran status or any other characteristic protected by law. -------------- next part -------------- An HTML attachment was scrubbed... URL: From chris at resfrac.com Sun Jul 19 16:27:10 2020 From: chris at resfrac.com (Chris Hewson) Date: Sun, 19 Jul 2020 15:27:10 -0600 Subject: [petsc-users] Tough to reproduce petsctablefind error Message-ID: Hi, I am having a bug that is occurring in PETSC with the return string: [7]PETSC ERROR: PetscTableFind() line 132 in /home/chewson/petsc-3.13.2/include/petscctable.h key 7556 is greater than largest key allowed 5693 This is using petsc-3.13.2, compiled and running using mpich with -O3 and debugging turned off tuned to the haswell architecture and occurring either before or during a KSPBCGS solve/setup or during a MUMPS factorization solve (I haven't been able to replicate this issue with the same set of instructions etc.). This is a terrible way to ask a question, I know, and not very helpful from your side, but this is what I have from a user's run and can't reproduce on my end (either with the optimization compilation or with debugging turned on). This happens when the code has run for quite some time and is happening somewhat rarely. More than likely I am using a static variable (code is written in c++) that I'm not updating when the matrix size is changing or something silly like that, but any help or guidance on this would be appreciated. *Chris Hewson* Senior Reservoir Simulation Engineer ResFrac +1.587.575.9792 -------------- next part -------------- An HTML attachment was scrubbed... URL: From jed at jedbrown.org Sun Jul 19 16:40:45 2020 From: jed at jedbrown.org (Jed Brown) Date: Sun, 19 Jul 2020 15:40:45 -0600 Subject: [petsc-users] Tough to reproduce petsctablefind error In-Reply-To: References: Message-ID: <87365np4iq.fsf@jedbrown.org> It'll be hard to narrow down without a stack trace or reproducer. I'd recommend running a small test to ensure that your code is Valgrind clean before going any further. Then try to get a stack trace by running inside a debugger or setting your environment to dump core on errors. Chris Hewson writes: > Hi, > > I am having a bug that is occurring in PETSC with the return string: > > [7]PETSC ERROR: PetscTableFind() line 132 in > /home/chewson/petsc-3.13.2/include/petscctable.h key 7556 is greater than > largest key allowed 5693 > > This is using petsc-3.13.2, compiled and running using mpich with -O3 and > debugging turned off tuned to the haswell architecture and occurring either > before or during a KSPBCGS solve/setup or during a MUMPS factorization > solve (I haven't been able to replicate this issue with the same set of > instructions etc.). > > This is a terrible way to ask a question, I know, and not very helpful from > your side, but this is what I have from a user's run and can't reproduce on > my end (either with the optimization compilation or with debugging turned > on). This happens when the code has run for quite some time and is > happening somewhat rarely. > > More than likely I am using a static variable (code is written in c++) that > I'm not updating when the matrix size is changing or something silly like > that, but any help or guidance on this would be appreciated. > > *Chris Hewson* > Senior Reservoir Simulation Engineer > ResFrac > +1.587.575.9792 From fdkong.jd at gmail.com Mon Jul 20 00:13:26 2020 From: fdkong.jd at gmail.com (Fande Kong) Date: Sun, 19 Jul 2020 23:13:26 -0600 Subject: [petsc-users] Tough to reproduce petsctablefind error In-Reply-To: References: Message-ID: I am not entirely sure what is happening, but we encountered similar issues recently. It was not reproducible. It might occur at different stages, and errors could be weird other than "ctable stuff." Our code was Valgrind clean since every PR in moose needs to go through rigorous Valgrind checks before it reaches the devel branch. The errors happened when we used mvapich. We changed to use HPE-MPT (a vendor stalled MPI), then everything was smooth. May you try a different MPI? It is better to try a system carried one. We did not get the bottom of this problem yet, but we at least know this is kind of MPI-related. Thanks, Fande, On Sun, Jul 19, 2020 at 3:28 PM Chris Hewson wrote: > Hi, > > I am having a bug that is occurring in PETSC with the return string: > > [7]PETSC ERROR: PetscTableFind() line 132 in > /home/chewson/petsc-3.13.2/include/petscctable.h key 7556 is greater than > largest key allowed 5693 > > This is using petsc-3.13.2, compiled and running using mpich with -O3 and > debugging turned off tuned to the haswell architecture and occurring either > before or during a KSPBCGS solve/setup or during a MUMPS factorization > solve (I haven't been able to replicate this issue with the same set of > instructions etc.). > > This is a terrible way to ask a question, I know, and not very helpful > from your side, but this is what I have from a user's run and can't > reproduce on my end (either with the optimization compilation or with > debugging turned on). This happens when the code has run for quite some > time and is happening somewhat rarely. > > More than likely I am using a static variable (code is written in c++) > that I'm not updating when the matrix size is changing or something silly > like that, but any help or guidance on this would be appreciated. > > *Chris Hewson* > Senior Reservoir Simulation Engineer > ResFrac > +1.587.575.9792 > -------------- next part -------------- An HTML attachment was scrubbed... URL: From fdkong.jd at gmail.com Mon Jul 20 00:16:00 2020 From: fdkong.jd at gmail.com (Fande Kong) Date: Sun, 19 Jul 2020 23:16:00 -0600 Subject: [petsc-users] Tough to reproduce petsctablefind error In-Reply-To: References: Message-ID: Trace could look like this: [640]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [640]PETSC ERROR: Argument out of range [640]PETSC ERROR: key 45226154 is greater than largest key allowed 740521 [640]PETSC ERROR: See https://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [640]PETSC ERROR: Petsc Release Version 3.13.3, unknown [640]PETSC ERROR: ../../griffin-opt on a arch-moose named r6i5n18 by wangy2 Sun Jul 19 17:14:28 2020 [640]PETSC ERROR: Configure options --download-hypre=1 --with-debugging=no --with-shared-libraries=1 --download-fblaslapack=1 --download-metis=1 --download-ptscotch=1 --download-parmetis=1 --download-superlu_dist=1 --download-mumps=1 --download-scalapack=1 --download-slepc=1 --with-mpi=1 --with-cxx-dialect=C++11 --with-fortran-bindings=0 --with-sowing=0 --with-64-bit-indices --download-mumps=0 [640]PETSC ERROR: #1 PetscTableFind() line 132 in /home/wangy2/trunk/sawtooth/griffin/moose/petsc/include/petscctable.h [640]PETSC ERROR: #2 MatSetUpMultiply_MPIAIJ() line 33 in /home/wangy2/trunk/sawtooth/griffin/moose/petsc/src/mat/impls/aij/mpi/mmaij.c [640]PETSC ERROR: #3 MatAssemblyEnd_MPIAIJ() line 876 in /home/wangy2/trunk/sawtooth/griffin/moose/petsc/src/mat/impls/aij/mpi/mpiaij.c [640]PETSC ERROR: #4 MatAssemblyEnd() line 5347 in /home/wangy2/trunk/sawtooth/griffin/moose/petsc/src/mat/interface/matrix.c [640]PETSC ERROR: #5 MatPtAPNumeric_MPIAIJ_MPIXAIJ_allatonce() line 901 in /home/wangy2/trunk/sawtooth/griffin/moose/petsc/src/mat/impls/aij/mpi/mpiptap.c [640]PETSC ERROR: #6 MatPtAPNumeric_MPIAIJ_MPIMAIJ_allatonce() line 3180 in /home/wangy2/trunk/sawtooth/griffin/moose/petsc/src/mat/impls/maij/maij.c [640]PETSC ERROR: #7 MatProductNumeric_PtAP() line 704 in /home/wangy2/trunk/sawtooth/griffin/moose/petsc/src/mat/interface/matproduct.c [640]PETSC ERROR: #8 MatProductNumeric() line 759 in /home/wangy2/trunk/sawtooth/griffin/moose/petsc/src/mat/interface/matproduct.c [640]PETSC ERROR: #9 MatPtAP() line 9199 in /home/wangy2/trunk/sawtooth/griffin/moose/petsc/src/mat/interface/matrix.c [640]PETSC ERROR: #10 MatGalerkin() line 10236 in /home/wangy2/trunk/sawtooth/griffin/moose/petsc/src/mat/interface/matrix.c [640]PETSC ERROR: #11 PCSetUp_MG() line 745 in /home/wangy2/trunk/sawtooth/griffin/moose/petsc/src/ksp/pc/impls/mg/mg.c [640]PETSC ERROR: #12 PCSetUp_HMG() line 220 in /home/wangy2/trunk/sawtooth/griffin/moose/petsc/src/ksp/pc/impls/hmg/hmg.c [640]PETSC ERROR: #13 PCSetUp() line 898 in /home/wangy2/trunk/sawtooth/griffin/moose/petsc/src/ksp/pc/interface/precon.c [640]PETSC ERROR: #14 KSPSetUp() line 376 in /home/wangy2/trunk/sawtooth/griffin/moose/petsc/src/ksp/ksp/interface/itfunc.c [640]PETSC ERROR: #15 KSPSolve_Private() line 633 in /home/wangy2/trunk/sawtooth/griffin/moose/petsc/src/ksp/ksp/interface/itfunc.c [640]PETSC ERROR: #16 KSPSolve() line 853 in /home/wangy2/trunk/sawtooth/griffin/moose/petsc/src/ksp/ksp/interface/itfunc.c [640]PETSC ERROR: #17 SNESSolve_NEWTONLS() line 225 in /home/wangy2/trunk/sawtooth/griffin/moose/petsc/src/snes/impls/ls/ls.c [640]PETSC ERROR: #18 SNESSolve() line 4519 in /home/wangy2/trunk/sawtooth/griffin/moose/petsc/src/snes/interface/snes.c On Sun, Jul 19, 2020 at 11:13 PM Fande Kong wrote: > I am not entirely sure what is happening, but we encountered similar > issues recently. It was not reproducible. It might occur at different > stages, and errors could be weird other than "ctable stuff." Our code was > Valgrind clean since every PR in moose needs to go through rigorous > Valgrind checks before it reaches the devel branch. The errors happened > when we used mvapich. > > We changed to use HPE-MPT (a vendor stalled MPI), then everything was > smooth. May you try a different MPI? It is better to try a system carried > one. > > We did not get the bottom of this problem yet, but we at least know this > is kind of MPI-related. > > Thanks, > > Fande, > > > On Sun, Jul 19, 2020 at 3:28 PM Chris Hewson wrote: > >> Hi, >> >> I am having a bug that is occurring in PETSC with the return string: >> >> [7]PETSC ERROR: PetscTableFind() line 132 in >> /home/chewson/petsc-3.13.2/include/petscctable.h key 7556 is greater than >> largest key allowed 5693 >> >> This is using petsc-3.13.2, compiled and running using mpich with -O3 and >> debugging turned off tuned to the haswell architecture and occurring either >> before or during a KSPBCGS solve/setup or during a MUMPS factorization >> solve (I haven't been able to replicate this issue with the same set of >> instructions etc.). >> >> This is a terrible way to ask a question, I know, and not very helpful >> from your side, but this is what I have from a user's run and can't >> reproduce on my end (either with the optimization compilation or with >> debugging turned on). This happens when the code has run for quite some >> time and is happening somewhat rarely. >> >> More than likely I am using a static variable (code is written in c++) >> that I'm not updating when the matrix size is changing or something silly >> like that, but any help or guidance on this would be appreciated. >> >> *Chris Hewson* >> Senior Reservoir Simulation Engineer >> ResFrac >> +1.587.575.9792 >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jroman at dsic.upv.es Mon Jul 20 02:59:28 2020 From: jroman at dsic.upv.es (Jose E. Roman) Date: Mon, 20 Jul 2020 09:59:28 +0200 Subject: [petsc-users] SLEPc Build Error In-Reply-To: References: Message-ID: <524AF3B0-4484-445C-A1AA-EEE6407B31F6@dsic.upv.es> I tried to reproduce this on Fedora 25 (with GCC 6.4.1), but the compilation went through without errors. My recommendation is that if you are using a PETSc tarball, use also a SLEPc tarball, and install it with the usual procedure described in SLEPc's documentation, not with --download-slepc However, the case of a tarball PETSc with --download-slepc should also work. From the messages in slepc.log, SLEPc has installed Sowing and it fails when generating the Fortran stubs. Maybe installation of Sowing was not successful. To find out, you will have to look at SLEPc logs, and also at Sowing logs under arch-linux2-c-debug/externalpackages/git.slepc/arch-linux2-c-debug/externalpackages/pkg-sowing/ Jose > El 19 jul 2020, a las 19:49, Matthew Knepley escribi?: > > On Sun, Jul 19, 2020 at 11:01 AM Eda Oktay wrote: > Hi all, > > I am trying to install petsc 3.13.2 to my computer (fedora 25) with > slepc and some other libraries by using the following code: > > ./configure --with-cc=gcc --with-cxx=g++ --download-mpich > --download-openblas --download-slepc --download-metis > --download-parmetis --download-chaco --with-X=1 > > Although I installed exact same things to an other computer with > fedora 25, this time, after configuration, when I am asked to use > "make PETSC_DIR=/home/eda/petsc-3.13.2 PETSC_ARCH=arch-linux2-c-debug > all", I got an Slepc error: > > PETSc thinks you are installing from a tarball, rather than from a Git clone, because it cannot > find the $PETSC_DIR/lib/petsc/bin/maint directory, which we remove from the tarball. The tarball > also contains the Fortran stubs, so we do not install Sowing in this case. When SLEPc is being > installed, it needs to generate the Fortran stubs, since it was cloned. > > At bottom, you cannot install SLEPc automatically unless you install PETSc from a Git clone. > > Thanks, > > Matt > > *** Building slepc *** > **************************ERROR************************************* > Error building slepc. Check arch-linux2-c-debug/lib/petsc/conf/slepc.log > ******************************************************************** > /home/eda/petsc-3.13.2/arch-linux2-c-debug/lib/petsc/conf/petscrules:32: > recipe for target 'slepcbuild' failed > gmake[2]: *** [slepcbuild] Error 1 > **************************ERROR************************************* > Error during compile, check arch-linux2-c-debug/lib/petsc/conf/make.log > Send it and arch-linux2-c-debug/lib/petsc/conf/configure.log to > petsc-maint at mcs.anl.gov > ******************************************************************** > makefile:33: recipe for target 'all' failed > make[1]: *** [all] Error 1 > GNUmakefile:9: recipe for target 'all' failed > make: *** [all] Error 2 > > I attached slepc.log, make.log and configure.log. What is the problem? > Can you help me please? > > Thanks a lot! > > Eda > > > -- > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ From thibault.bridelbertomeu at gmail.com Mon Jul 20 04:26:58 2020 From: thibault.bridelbertomeu at gmail.com (Thibault Bridel-Bertomeu) Date: Mon, 20 Jul 2020 11:26:58 +0200 Subject: [petsc-users] How to add a source term for PETSCFV ? Message-ID: Dear all, I have been studying ex11.c from ts/tutorials to understand how to solve an hyperbolic system of equations using PETSCFV. I first worked on the Euler equations for inviscid fluids and based on what ex11.c presents, I was able to add the right PETSc instructions in an already existing in-house code with different gas models to solve the problems in parallel (MPI) and with the AMR capabilities offered by P4EST. Now my goal is to move to Navier-Stokes equations. Theoretically the system is not completely hyperbolic and can be seen as one with an hyperbolic part (identical to the Euler equations) and a parabolic part coming from the RHS diffusion terms. I have been looking into the manual and also the sources of PETSc around the DM, DMPlex, DS and FV classes but I could not find anything that speaks to me as "adding a RHS to an hyperbolic system of equations" or "adding a source term to an hyperbolic system of equations". What's more, that source term depends on the derivatives of the context variables ... I wanted to know if anyone maybe had a suggestion regarding this issue ? Thank you very much in advance, Thibault Bridel-Bertomeu ? Eng, MSc, PhD Research Engineer CEA/CESTA 33114 LE BARP Tel.: (+33)557046924 Mob.: (+33)611025322 Mail: thibault.bridelbertomeu at gmail.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From eda.oktay at metu.edu.tr Mon Jul 20 05:00:17 2020 From: eda.oktay at metu.edu.tr (Eda Oktay) Date: Mon, 20 Jul 2020 13:00:17 +0300 Subject: [petsc-users] SLEPc Build Error In-Reply-To: <524AF3B0-4484-445C-A1AA-EEE6407B31F6@dsic.upv.es> References: <524AF3B0-4484-445C-A1AA-EEE6407B31F6@dsic.upv.es> Message-ID: Thank you so much, I will try them! Eda Jose E. Roman , 20 Tem 2020 Pzt, 10:59 tarihinde ?unu yazd?: > > I tried to reproduce this on Fedora 25 (with GCC 6.4.1), but the compilation went through without errors. > > My recommendation is that if you are using a PETSc tarball, use also a SLEPc tarball, and install it with the usual procedure described in SLEPc's documentation, not with --download-slepc > > However, the case of a tarball PETSc with --download-slepc should also work. From the messages in slepc.log, SLEPc has installed Sowing and it fails when generating the Fortran stubs. Maybe installation of Sowing was not successful. To find out, you will have to look at SLEPc logs, and also at Sowing logs under arch-linux2-c-debug/externalpackages/git.slepc/arch-linux2-c-debug/externalpackages/pkg-sowing/ > > Jose > > > > El 19 jul 2020, a las 19:49, Matthew Knepley escribi?: > > > > On Sun, Jul 19, 2020 at 11:01 AM Eda Oktay wrote: > > Hi all, > > > > I am trying to install petsc 3.13.2 to my computer (fedora 25) with > > slepc and some other libraries by using the following code: > > > > ./configure --with-cc=gcc --with-cxx=g++ --download-mpich > > --download-openblas --download-slepc --download-metis > > --download-parmetis --download-chaco --with-X=1 > > > > Although I installed exact same things to an other computer with > > fedora 25, this time, after configuration, when I am asked to use > > "make PETSC_DIR=/home/eda/petsc-3.13.2 PETSC_ARCH=arch-linux2-c-debug > > all", I got an Slepc error: > > > > PETSc thinks you are installing from a tarball, rather than from a Git clone, because it cannot > > find the $PETSC_DIR/lib/petsc/bin/maint directory, which we remove from the tarball. The tarball > > also contains the Fortran stubs, so we do not install Sowing in this case. When SLEPc is being > > installed, it needs to generate the Fortran stubs, since it was cloned. > > > > At bottom, you cannot install SLEPc automatically unless you install PETSc from a Git clone. > > > > Thanks, > > > > Matt > > > > *** Building slepc *** > > **************************ERROR************************************* > > Error building slepc. Check arch-linux2-c-debug/lib/petsc/conf/slepc.log > > ******************************************************************** > > /home/eda/petsc-3.13.2/arch-linux2-c-debug/lib/petsc/conf/petscrules:32: > > recipe for target 'slepcbuild' failed > > gmake[2]: *** [slepcbuild] Error 1 > > **************************ERROR************************************* > > Error during compile, check arch-linux2-c-debug/lib/petsc/conf/make.log > > Send it and arch-linux2-c-debug/lib/petsc/conf/configure.log to > > petsc-maint at mcs.anl.gov > > ******************************************************************** > > makefile:33: recipe for target 'all' failed > > make[1]: *** [all] Error 1 > > GNUmakefile:9: recipe for target 'all' failed > > make: *** [all] Error 2 > > > > I attached slepc.log, make.log and configure.log. What is the problem? > > Can you help me please? > > > > Thanks a lot! > > > > Eda > > > > > > -- > > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. > > -- Norbert Wiener > > > > https://www.cse.buffalo.edu/~knepley/ > From bsmith at petsc.dev Mon Jul 20 07:03:54 2020 From: bsmith at petsc.dev (Barry Smith) Date: Mon, 20 Jul 2020 07:03:54 -0500 Subject: [petsc-users] Tough to reproduce petsctablefind error In-Reply-To: References: Message-ID: <0AC37384-BC37-4A6C-857D-41CD507F84C2@petsc.dev> Is there a comprehensive MPI test suite (perhaps from MPICH)? Is there any way to run this full test suite under the problematic MPI and see if it detects any problems? Is so, could someone add it to the FAQ in the debugging section? Thanks Barry > On Jul 20, 2020, at 12:16 AM, Fande Kong wrote: > > Trace could look like this: > > [640]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- > [640]PETSC ERROR: Argument out of range > [640]PETSC ERROR: key 45226154 is greater than largest key allowed 740521 > [640]PETSC ERROR: See https://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. > [640]PETSC ERROR: Petsc Release Version 3.13.3, unknown > [640]PETSC ERROR: ../../griffin-opt on a arch-moose named r6i5n18 by wangy2 Sun Jul 19 17:14:28 2020 > [640]PETSC ERROR: Configure options --download-hypre=1 --with-debugging=no --with-shared-libraries=1 --download-fblaslapack=1 --download-metis=1 --download-ptscotch=1 --download-parmetis=1 --download-superlu_dist=1 --download-mumps=1 --download-scalapack=1 --download-slepc=1 --with-mpi=1 --with-cxx-dialect=C++11 --with-fortran-bindings=0 --with-sowing=0 --with-64-bit-indices --download-mumps=0 > [640]PETSC ERROR: #1 PetscTableFind() line 132 in /home/wangy2/trunk/sawtooth/griffin/moose/petsc/include/petscctable.h > [640]PETSC ERROR: #2 MatSetUpMultiply_MPIAIJ() line 33 in /home/wangy2/trunk/sawtooth/griffin/moose/petsc/src/mat/impls/aij/mpi/mmaij.c > [640]PETSC ERROR: #3 MatAssemblyEnd_MPIAIJ() line 876 in /home/wangy2/trunk/sawtooth/griffin/moose/petsc/src/mat/impls/aij/mpi/mpiaij.c > [640]PETSC ERROR: #4 MatAssemblyEnd() line 5347 in /home/wangy2/trunk/sawtooth/griffin/moose/petsc/src/mat/interface/matrix.c > [640]PETSC ERROR: #5 MatPtAPNumeric_MPIAIJ_MPIXAIJ_allatonce() line 901 in /home/wangy2/trunk/sawtooth/griffin/moose/petsc/src/mat/impls/aij/mpi/mpiptap.c > [640]PETSC ERROR: #6 MatPtAPNumeric_MPIAIJ_MPIMAIJ_allatonce() line 3180 in /home/wangy2/trunk/sawtooth/griffin/moose/petsc/src/mat/impls/maij/maij.c > [640]PETSC ERROR: #7 MatProductNumeric_PtAP() line 704 in /home/wangy2/trunk/sawtooth/griffin/moose/petsc/src/mat/interface/matproduct.c > [640]PETSC ERROR: #8 MatProductNumeric() line 759 in /home/wangy2/trunk/sawtooth/griffin/moose/petsc/src/mat/interface/matproduct.c > [640]PETSC ERROR: #9 MatPtAP() line 9199 in /home/wangy2/trunk/sawtooth/griffin/moose/petsc/src/mat/interface/matrix.c > [640]PETSC ERROR: #10 MatGalerkin() line 10236 in /home/wangy2/trunk/sawtooth/griffin/moose/petsc/src/mat/interface/matrix.c > [640]PETSC ERROR: #11 PCSetUp_MG() line 745 in /home/wangy2/trunk/sawtooth/griffin/moose/petsc/src/ksp/pc/impls/mg/mg.c > [640]PETSC ERROR: #12 PCSetUp_HMG() line 220 in /home/wangy2/trunk/sawtooth/griffin/moose/petsc/src/ksp/pc/impls/hmg/hmg.c > [640]PETSC ERROR: #13 PCSetUp() line 898 in /home/wangy2/trunk/sawtooth/griffin/moose/petsc/src/ksp/pc/interface/precon.c > [640]PETSC ERROR: #14 KSPSetUp() line 376 in /home/wangy2/trunk/sawtooth/griffin/moose/petsc/src/ksp/ksp/interface/itfunc.c > [640]PETSC ERROR: #15 KSPSolve_Private() line 633 in /home/wangy2/trunk/sawtooth/griffin/moose/petsc/src/ksp/ksp/interface/itfunc.c > [640]PETSC ERROR: #16 KSPSolve() line 853 in /home/wangy2/trunk/sawtooth/griffin/moose/petsc/src/ksp/ksp/interface/itfunc.c > [640]PETSC ERROR: #17 SNESSolve_NEWTONLS() line 225 in /home/wangy2/trunk/sawtooth/griffin/moose/petsc/src/snes/impls/ls/ls.c > [640]PETSC ERROR: #18 SNESSolve() line 4519 in /home/wangy2/trunk/sawtooth/griffin/moose/petsc/src/snes/interface/snes.c > > On Sun, Jul 19, 2020 at 11:13 PM Fande Kong > wrote: > I am not entirely sure what is happening, but we encountered similar issues recently. It was not reproducible. It might occur at different stages, and errors could be weird other than "ctable stuff." Our code was Valgrind clean since every PR in moose needs to go through rigorous Valgrind checks before it reaches the devel branch. The errors happened when we used mvapich. > > We changed to use HPE-MPT (a vendor stalled MPI), then everything was smooth. May you try a different MPI? It is better to try a system carried one. > > We did not get the bottom of this problem yet, but we at least know this is kind of MPI-related. > > Thanks, > > Fande, > > > On Sun, Jul 19, 2020 at 3:28 PM Chris Hewson > wrote: > Hi, > > I am having a bug that is occurring in PETSC with the return string: > > [7]PETSC ERROR: PetscTableFind() line 132 in /home/chewson/petsc-3.13.2/include/petscctable.h key 7556 is greater than largest key allowed 5693 > > This is using petsc-3.13.2, compiled and running using mpich with -O3 and debugging turned off tuned to the haswell architecture and occurring either before or during a KSPBCGS solve/setup or during a MUMPS factorization solve (I haven't been able to replicate this issue with the same set of instructions etc.). > > This is a terrible way to ask a question, I know, and not very helpful from your side, but this is what I have from a user's run and can't reproduce on my end (either with the optimization compilation or with debugging turned on). This happens when the code has run for quite some time and is happening somewhat rarely. > > More than likely I am using a static variable (code is written in c++) that I'm not updating when the matrix size is changing or something silly like that, but any help or guidance on this would be appreciated. > > Chris Hewson > Senior Reservoir Simulation Engineer > ResFrac > +1.587.575.9792 -------------- next part -------------- An HTML attachment was scrubbed... URL: From adantra at gmail.com Mon Jul 20 07:38:02 2020 From: adantra at gmail.com (Adolfo Rodriguez) Date: Mon, 20 Jul 2020 07:38:02 -0500 Subject: [petsc-users] snes setjacobian with csr sparse matrix Message-ID: I am trying to use PETSc to solve a non-linear problem. The problem I am facing is that I already have the Jacobian matrix formed in CSR format. Now I am trying to define a function FormJacobian which takes this preexisting matrix and writes it in PETSc format. For a small problem, I can use MatSetValue, looping over all non-zero elements one at a time. However, this turns out too slow for larger problems. In my current implementation, I am using MatCreateSeqAIJ which works great. However, this approach does not seem to work to set the Jacobian. Any suggestions? Regards, Adolfo -------------- next part -------------- An HTML attachment was scrubbed... URL: From mfadams at lbl.gov Mon Jul 20 08:13:39 2020 From: mfadams at lbl.gov (Mark Adams) Date: Mon, 20 Jul 2020 09:13:39 -0400 Subject: [petsc-users] Tough to reproduce petsctablefind error In-Reply-To: References: Message-ID: Fande, do you know if your 45226154 was out of range in the real matrix? What size integers do you use? Thanks, Mark On Mon, Jul 20, 2020 at 1:17 AM Fande Kong wrote: > Trace could look like this: > > [640]PETSC ERROR: --------------------- Error Message > -------------------------------------------------------------- > > [640]PETSC ERROR: Argument out of range > > [640]PETSC ERROR: key 45226154 is greater than largest key allowed 740521 > > [640]PETSC ERROR: See https://www.mcs.anl.gov/petsc/documentation/faq.html for > trouble shooting. > > [640]PETSC ERROR: Petsc Release Version 3.13.3, unknown > > [640]PETSC ERROR: ../../griffin-opt on a arch-moose named r6i5n18 by > wangy2 Sun Jul 19 17:14:28 2020 > > [640]PETSC ERROR: Configure options --download-hypre=1 --with-debugging=no > --with-shared-libraries=1 --download-fblaslapack=1 --download-metis=1 > --download-ptscotch=1 --download-parmetis=1 --download-superlu_dist=1 > --download-mumps=1 --download-scalapack=1 --download-slepc=1 --with-mpi=1 > --with-cxx-dialect=C++11 --with-fortran-bindings=0 --with-sowing=0 > --with-64-bit-indices --download-mumps=0 > > [640]PETSC ERROR: #1 PetscTableFind() line 132 in > /home/wangy2/trunk/sawtooth/griffin/moose/petsc/include/petscctable.h > > [640]PETSC ERROR: #2 MatSetUpMultiply_MPIAIJ() line 33 in > /home/wangy2/trunk/sawtooth/griffin/moose/petsc/src/mat/impls/aij/mpi/mmaij.c > > [640]PETSC ERROR: #3 MatAssemblyEnd_MPIAIJ() line 876 in > /home/wangy2/trunk/sawtooth/griffin/moose/petsc/src/mat/impls/aij/mpi/mpiaij.c > > [640]PETSC ERROR: #4 MatAssemblyEnd() line 5347 in > /home/wangy2/trunk/sawtooth/griffin/moose/petsc/src/mat/interface/matrix.c > > [640]PETSC ERROR: #5 MatPtAPNumeric_MPIAIJ_MPIXAIJ_allatonce() line 901 in > /home/wangy2/trunk/sawtooth/griffin/moose/petsc/src/mat/impls/aij/mpi/mpiptap.c > > [640]PETSC ERROR: #6 MatPtAPNumeric_MPIAIJ_MPIMAIJ_allatonce() line 3180 > in /home/wangy2/trunk/sawtooth/griffin/moose/petsc/src/mat/impls/maij/maij.c > > [640]PETSC ERROR: #7 MatProductNumeric_PtAP() line 704 in > /home/wangy2/trunk/sawtooth/griffin/moose/petsc/src/mat/interface/matproduct.c > > [640]PETSC ERROR: #8 MatProductNumeric() line 759 in > /home/wangy2/trunk/sawtooth/griffin/moose/petsc/src/mat/interface/matproduct.c > > [640]PETSC ERROR: #9 MatPtAP() line 9199 in > /home/wangy2/trunk/sawtooth/griffin/moose/petsc/src/mat/interface/matrix.c > > [640]PETSC ERROR: #10 MatGalerkin() line 10236 in > /home/wangy2/trunk/sawtooth/griffin/moose/petsc/src/mat/interface/matrix.c > > [640]PETSC ERROR: #11 PCSetUp_MG() line 745 in > /home/wangy2/trunk/sawtooth/griffin/moose/petsc/src/ksp/pc/impls/mg/mg.c > > [640]PETSC ERROR: #12 PCSetUp_HMG() line 220 in > /home/wangy2/trunk/sawtooth/griffin/moose/petsc/src/ksp/pc/impls/hmg/hmg.c > > [640]PETSC ERROR: #13 PCSetUp() line 898 in > /home/wangy2/trunk/sawtooth/griffin/moose/petsc/src/ksp/pc/interface/precon.c > > [640]PETSC ERROR: #14 KSPSetUp() line 376 in > /home/wangy2/trunk/sawtooth/griffin/moose/petsc/src/ksp/ksp/interface/itfunc.c > > [640]PETSC ERROR: #15 KSPSolve_Private() line 633 in > /home/wangy2/trunk/sawtooth/griffin/moose/petsc/src/ksp/ksp/interface/itfunc.c > > [640]PETSC ERROR: #16 KSPSolve() line 853 in > /home/wangy2/trunk/sawtooth/griffin/moose/petsc/src/ksp/ksp/interface/itfunc.c > > [640]PETSC ERROR: #17 SNESSolve_NEWTONLS() line 225 in > /home/wangy2/trunk/sawtooth/griffin/moose/petsc/src/snes/impls/ls/ls.c > > [640]PETSC ERROR: #18 SNESSolve() line 4519 in > /home/wangy2/trunk/sawtooth/griffin/moose/petsc/src/snes/interface/snes.c > > On Sun, Jul 19, 2020 at 11:13 PM Fande Kong wrote: > >> I am not entirely sure what is happening, but we encountered similar >> issues recently. It was not reproducible. It might occur at different >> stages, and errors could be weird other than "ctable stuff." Our code was >> Valgrind clean since every PR in moose needs to go through rigorous >> Valgrind checks before it reaches the devel branch. The errors happened >> when we used mvapich. >> >> We changed to use HPE-MPT (a vendor stalled MPI), then everything was >> smooth. May you try a different MPI? It is better to try a system carried >> one. >> >> We did not get the bottom of this problem yet, but we at least know this >> is kind of MPI-related. >> >> Thanks, >> >> Fande, >> >> >> On Sun, Jul 19, 2020 at 3:28 PM Chris Hewson wrote: >> >>> Hi, >>> >>> I am having a bug that is occurring in PETSC with the return string: >>> >>> [7]PETSC ERROR: PetscTableFind() line 132 in >>> /home/chewson/petsc-3.13.2/include/petscctable.h key 7556 is greater than >>> largest key allowed 5693 >>> >>> This is using petsc-3.13.2, compiled and running using mpich with -O3 >>> and debugging turned off tuned to the haswell architecture and >>> occurring either before or during a KSPBCGS solve/setup or during a MUMPS >>> factorization solve (I haven't been able to replicate this issue with the >>> same set of instructions etc.). >>> >>> This is a terrible way to ask a question, I know, and not very helpful >>> from your side, but this is what I have from a user's run and can't >>> reproduce on my end (either with the optimization compilation or with >>> debugging turned on). This happens when the code has run for quite some >>> time and is happening somewhat rarely. >>> >>> More than likely I am using a static variable (code is written in c++) >>> that I'm not updating when the matrix size is changing or something silly >>> like that, but any help or guidance on this would be appreciated. >>> >>> *Chris Hewson* >>> Senior Reservoir Simulation Engineer >>> ResFrac >>> +1.587.575.9792 >>> >> -------------- next part -------------- An HTML attachment was scrubbed... URL: From jed at jedbrown.org Mon Jul 20 08:28:06 2020 From: jed at jedbrown.org (Jed Brown) Date: Mon, 20 Jul 2020 07:28:06 -0600 Subject: [petsc-users] snes setjacobian with csr sparse matrix In-Reply-To: References: Message-ID: <87o8oaqpsp.fsf@jedbrown.org> Adolfo Rodriguez writes: > I am trying to use PETSc to solve a non-linear problem. The problem I am > facing is that I already have the Jacobian matrix formed in CSR format. Now > I am trying to define a function FormJacobian which takes this preexisting > matrix and writes it in PETSc format. For a small problem, I can use > MatSetValue, looping over all non-zero elements one at a time. However, > this turns out too slow for larger problems. Best to call MatSetValues once per row. If the matrix is preallocated, it should not be slow. Check https://www.mcs.anl.gov/petsc/documentation/faq.html#efficient-assembly > In my current implementation, I am using MatCreateSeqAIJ which works great. > However, this approach does not seem to work to set the Jacobian. > > Any suggestions? > > Regards, > > Adolfo From mfadams at lbl.gov Mon Jul 20 08:33:21 2020 From: mfadams at lbl.gov (Mark Adams) Date: Mon, 20 Jul 2020 09:33:21 -0400 Subject: [petsc-users] How to add a source term for PETSCFV ? In-Reply-To: References: Message-ID: If you search for (-i) "boundary" in ts/ex11.c you will see several examples of setting BCs with ghost cells in FV. Don't worry about hyperbolic per se. The PETSc interface is very abstract: G(x,xdot,t) = F(x,t), or something like that. You can decide which side of the equation to put each of your terms. I'm not sure what you mean by derivatives of the context variables, but I see that FV does not seem to have the auxiliary variable support that FE has, where we take spatial derivatives of anything that you want. If you want to have context variables in your cells so that you can take derivatives in your Riemann solver then you could add dummy variables to your system like Identity*q = Q. Then q would be in the L/R state variables. Hope this helps, Mark On Mon, Jul 20, 2020 at 5:28 AM Thibault Bridel-Bertomeu < thibault.bridelbertomeu at gmail.com> wrote: > Dear all, > > I have been studying ex11.c from ts/tutorials to understand how to solve > an hyperbolic system of equations using PETSCFV. I first worked on the > Euler equations for inviscid fluids and based on what ex11.c presents, I > was able to add the right PETSc instructions in an already existing > in-house code with different gas models to solve the problems in > parallel (MPI) and with the AMR capabilities offered by P4EST. > > Now my goal is to move to Navier-Stokes equations. Theoretically the > system is not completely hyperbolic and can be seen as one with an > hyperbolic part (identical to the Euler equations) and a parabolic part > coming from the RHS diffusion terms. > I have been looking into the manual and also the sources of PETSc around > the DM, DMPlex, DS and FV classes but I could not find anything that speaks > to me as "adding a RHS to an hyperbolic system of equations" or "adding a > source term to an hyperbolic system of equations". What's more, that source > term depends on the derivatives of the context variables ... > > I wanted to know if anyone maybe had a suggestion regarding this issue ? > > Thank you very much in advance, > > Thibault Bridel-Bertomeu > ? > Eng, MSc, PhD > Research Engineer > CEA/CESTA > 33114 LE BARP > Tel.: (+33)557046924 > Mob.: (+33)611025322 > Mail: thibault.bridelbertomeu at gmail.com > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jed at jedbrown.org Mon Jul 20 08:36:06 2020 From: jed at jedbrown.org (Jed Brown) Date: Mon, 20 Jul 2020 07:36:06 -0600 Subject: [petsc-users] How to add a source term for PETSCFV ? In-Reply-To: References: Message-ID: <87lfjeqpfd.fsf@jedbrown.org> How would you like to discretize the diffusive terms? The example has a type of gradient reconstruction so you can have cellwise gradients, but there are many techniques for discretizing diffusive terms in FV. It's simpler if you use an orthogonal grid, but I doubt that you are. As for terminology, the diffusive part is usually stiff and thus must be treated implicitly. In TS terminology, this would be part of the IFunction, not the RHSFunction. Thibault Bridel-Bertomeu writes: > Dear all, > > I have been studying ex11.c from ts/tutorials to understand how to solve an > hyperbolic system of equations using PETSCFV. I first worked on the Euler > equations for inviscid fluids and based on what ex11.c presents, I was able > to add the right PETSc instructions in an already existing in-house code > with different gas models to solve the problems in parallel (MPI) and with > the AMR capabilities offered by P4EST. > > Now my goal is to move to Navier-Stokes equations. Theoretically the system > is not completely hyperbolic and can be seen as one with an hyperbolic part > (identical to the Euler equations) and a parabolic part coming from the RHS > diffusion terms. > I have been looking into the manual and also the sources of PETSc around > the DM, DMPlex, DS and FV classes but I could not find anything that speaks > to me as "adding a RHS to an hyperbolic system of equations" or "adding a > source term to an hyperbolic system of equations". What's more, that source > term depends on the derivatives of the context variables ... > > I wanted to know if anyone maybe had a suggestion regarding this issue ? > > Thank you very much in advance, > > Thibault Bridel-Bertomeu > ? > Eng, MSc, PhD > Research Engineer > CEA/CESTA > 33114 LE BARP > Tel.: (+33)557046924 > Mob.: (+33)611025322 > Mail: thibault.bridelbertomeu at gmail.com From knepley at gmail.com Mon Jul 20 09:10:06 2020 From: knepley at gmail.com (Matthew Knepley) Date: Mon, 20 Jul 2020 10:10:06 -0400 Subject: [petsc-users] How to add a source term for PETSCFV ? In-Reply-To: <87lfjeqpfd.fsf@jedbrown.org> References: <87lfjeqpfd.fsf@jedbrown.org> Message-ID: On Mon, Jul 20, 2020 at 9:36 AM Jed Brown wrote: > How would you like to discretize the diffusive terms? The example has a > type of gradient reconstruction so you can have cellwise gradients, but > there are many techniques for discretizing diffusive terms in FV. It's > simpler if you use an orthogonal grid, but I doubt that you are. > > As for terminology, the diffusive part is usually stiff and thus must be > treated implicitly. In TS terminology, this would be part of the > IFunction, not the RHSFunction. > At a high level, I would say that this is doable, but complicated. You can see me trying to do something much easier (advection + visco-elasticity) in TS ex18, where I want to discretize the elliptic part with FEM and the advective part with FVM. I assume that is why Jed wants to know how you want to handle the elliptic terms, since this has a large impact on how you would implement. Thanks, Matt > Thibault Bridel-Bertomeu writes: > > > Dear all, > > > > I have been studying ex11.c from ts/tutorials to understand how to solve > an > > hyperbolic system of equations using PETSCFV. I first worked on the Euler > > equations for inviscid fluids and based on what ex11.c presents, I was > able > > to add the right PETSc instructions in an already existing in-house code > > with different gas models to solve the problems in parallel (MPI) and > with > > the AMR capabilities offered by P4EST. > > > > Now my goal is to move to Navier-Stokes equations. Theoretically the > system > > is not completely hyperbolic and can be seen as one with an hyperbolic > part > > (identical to the Euler equations) and a parabolic part coming from the > RHS > > diffusion terms. > > I have been looking into the manual and also the sources of PETSc around > > the DM, DMPlex, DS and FV classes but I could not find anything that > speaks > > to me as "adding a RHS to an hyperbolic system of equations" or "adding a > > source term to an hyperbolic system of equations". What's more, that > source > > term depends on the derivatives of the context variables ... > > > > I wanted to know if anyone maybe had a suggestion regarding this issue ? > > > > Thank you very much in advance, > > > > Thibault Bridel-Bertomeu > > ? > > Eng, MSc, PhD > > Research Engineer > > CEA/CESTA > > 33114 LE BARP > > Tel.: (+33)557046924 > > Mob.: (+33)611025322 > > Mail: thibault.bridelbertomeu at gmail.com > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From thibault.bridelbertomeu at gmail.com Mon Jul 20 10:03:55 2020 From: thibault.bridelbertomeu at gmail.com (Thibault Bridel-Bertomeu) Date: Mon, 20 Jul 2020 17:03:55 +0200 Subject: [petsc-users] How to add a source term for PETSCFV ? In-Reply-To: References: <87lfjeqpfd.fsf@jedbrown.org> Message-ID: Thank you Mark, Jed and Matthew for your quick answers ! I see now where I should be more accurate in my question. Mark, I mentioned the hyperbolicity because I would like to keep using the PetscDSSetRiemannSolver and the DMTSSetBoundaryLocal and DMTSSetRHSFunctionLocal with DMPlexTSComputeRHSFunctionFVM that are quite automatic and nice and efficient wrappers. Now aside from those which deal specifically with the hyperbolic part of the PDE, i would like to add the diffusive terms. I would rather stay in the FVM world, but if it is easier in the FEM world then I am open to it. Jed, as for the discretization let us say indeed that the mesh can be either cartesian or not, and the discretization should therefore be independent of the nature of the mesh - any unstructured mesh (i handle it with DMPlex in my case). I saw indeed that FV has gradient reconstruction, with or without a limiter, which is great. However I have not quite understood what function to use to get the gradient of any variable, be it in the context (e.g. for N-S, ro, rou, rov, etc...) or an auxiliary variable (e.g. the components of the strain tensor). I also agree that the diffusive part is usually the one that strongly limits the time step in explicit computations, but for now I would like to set up a fully explicit system. Matthew, I'll take a look at ex 18, thanks, I missed that one. So basically if I wanted to summarize, I want to keep the Riemann Solver capability from the DS, and use the "automatic" DMPlexTSComputeRHSFunctionFVM for the hyperbolic part and add on top of it a discretization of the diffusive terms. I was thinking maybe one way to go would be to hack the DMTSSetForcingFunction but 1/ I still am not sure what this function should return exactly, is it a Vec for the flux on all faces ? 2/ I still do not know how to compute all the derivatives involved in the diffusive terms of the N-S using the gradient reconstruction from PetscFV Thank you for your help, I hope I am clear enough in where I want to go ! Thibault Le lun. 20 juil. 2020 ? 16:10, Matthew Knepley a ?crit : > On Mon, Jul 20, 2020 at 9:36 AM Jed Brown wrote: > >> How would you like to discretize the diffusive terms? The example has a >> type of gradient reconstruction so you can have cellwise gradients, but >> there are many techniques for discretizing diffusive terms in FV. It's >> simpler if you use an orthogonal grid, but I doubt that you are. >> >> As for terminology, the diffusive part is usually stiff and thus must be >> treated implicitly. In TS terminology, this would be part of the >> IFunction, not the RHSFunction. >> > > At a high level, I would say that this is doable, but complicated. You can > see me trying to do something much easier (advection + visco-elasticity) in > TS ex18, > where I want to discretize the elliptic part with FEM and the advective > part with FVM. I assume that is why Jed wants to know how you want to > handle the > elliptic terms, since this has a large impact on how you would implement. > > Thanks, > > Matt > > >> Thibault Bridel-Bertomeu writes: >> >> > Dear all, >> > >> > I have been studying ex11.c from ts/tutorials to understand how to >> solve an >> > hyperbolic system of equations using PETSCFV. I first worked on the >> Euler >> > equations for inviscid fluids and based on what ex11.c presents, I was >> able >> > to add the right PETSc instructions in an already existing in-house code >> > with different gas models to solve the problems in parallel (MPI) and >> with >> > the AMR capabilities offered by P4EST. >> > >> > Now my goal is to move to Navier-Stokes equations. Theoretically the >> system >> > is not completely hyperbolic and can be seen as one with an hyperbolic >> part >> > (identical to the Euler equations) and a parabolic part coming from the >> RHS >> > diffusion terms. >> > I have been looking into the manual and also the sources of PETSc around >> > the DM, DMPlex, DS and FV classes but I could not find anything that >> speaks >> > to me as "adding a RHS to an hyperbolic system of equations" or "adding >> a >> > source term to an hyperbolic system of equations". What's more, that >> source >> > term depends on the derivatives of the context variables ... >> > >> > I wanted to know if anyone maybe had a suggestion regarding this issue ? >> > >> > Thank you very much in advance, >> > >> > Thibault Bridel-Bertomeu >> > ? >> > Eng, MSc, PhD >> > Research Engineer >> > CEA/CESTA >> > 33114 LE BARP >> > Tel.: (+33)557046924 >> > Mob.: (+33)611025322 >> > Mail: thibault.bridelbertomeu at gmail.com >> > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at petsc.dev Mon Jul 20 11:10:24 2020 From: bsmith at petsc.dev (Barry Smith) Date: Mon, 20 Jul 2020 11:10:24 -0500 Subject: [petsc-users] snes setjacobian with csr sparse matrix In-Reply-To: References: Message-ID: <9FC7A7FF-697F-408F-A13D-AE10D900C61C@petsc.dev> If you are running sequentially you likely want the first routine, in parallel the third routine. MatCreateSeqAIJWithArrays() PETSC_EXTERN PetscErrorCode MatSeqAIJSetPreallocationCSR(Mat,const PetscInt [],const PetscInt [],const PetscScalar []); PETSC_EXTERN PetscErrorCode MatMPIAIJSetPreallocationCSR(Mat,const PetscInt[],const PetscInt[],const PetscScalar[]); PETSC_EXTERN PetscErrorCode MatSeqBAIJSetPreallocationCSR(Mat,PetscInt,const PetscInt[],const PetscInt[],const PetscScalar[]); PETSC_EXTERN PetscErrorCode MatMPIBAIJSetPreallocationCSR(Mat,PetscInt,const PetscInt[],const PetscInt[],const PetscScalar[]); PETSC_EXTERN PetscErrorCode MatSeqSBAIJSetPreallocationCSR(Mat,PetscInt,const PetscInt[],const PetscInt[],const PetscScalar[]); PETSC_EXTERN PetscErrorCode MatMPISBAIJSetPreallocationCSR(Mat,PetscInt,const PetscInt[],const PetscInt[],const PetscScalar[]); > On Jul 20, 2020, at 7:38 AM, Adolfo Rodriguez wrote: > > I am trying to use PETSc to solve a non-linear problem. The problem I am facing is that I already have the Jacobian matrix formed in CSR format. Now I am trying to define a function FormJacobian which takes this preexisting matrix and writes it in PETSc format. For a small problem, I can use MatSetValue, looping over all non-zero elements one at a time. However, this turns out too slow for larger problems. > > In my current implementation, I am using MatCreateSeqAIJ which works great. However, this approach does not seem to work to set the Jacobian. > > Any suggestions? > > Regards, > > Adolfo From fdkong.jd at gmail.com Mon Jul 20 11:34:56 2020 From: fdkong.jd at gmail.com (Fande Kong) Date: Mon, 20 Jul 2020 10:34:56 -0600 Subject: [petsc-users] Tough to reproduce petsctablefind error In-Reply-To: References: Message-ID: Hi Mark, Thanks for your reply. On Mon, Jul 20, 2020 at 7:13 AM Mark Adams wrote: > Fande, > do you know if your 45226154 was out of range in the real matrix? > I do not know since it was in building the AMG hierarchy. The size of the original system is 1,428,284,880 > What size integers do you use? > We are using 64-bit via "--with-64-bit-indices" I am trying to catch the cause of this issue by running more simulations with different configurations. Thanks, Fande, Thanks, > Mark > > On Mon, Jul 20, 2020 at 1:17 AM Fande Kong wrote: > >> Trace could look like this: >> >> [640]PETSC ERROR: --------------------- Error Message >> -------------------------------------------------------------- >> >> [640]PETSC ERROR: Argument out of range >> >> [640]PETSC ERROR: key 45226154 is greater than largest key allowed 740521 >> >> [640]PETSC ERROR: See >> https://www.mcs.anl.gov/petsc/documentation/faq.html for trouble >> shooting. >> >> [640]PETSC ERROR: Petsc Release Version 3.13.3, unknown >> >> [640]PETSC ERROR: ../../griffin-opt on a arch-moose named r6i5n18 by >> wangy2 Sun Jul 19 17:14:28 2020 >> >> [640]PETSC ERROR: Configure options --download-hypre=1 >> --with-debugging=no --with-shared-libraries=1 --download-fblaslapack=1 >> --download-metis=1 --download-ptscotch=1 --download-parmetis=1 >> --download-superlu_dist=1 --download-mumps=1 --download-scalapack=1 >> --download-slepc=1 --with-mpi=1 --with-cxx-dialect=C++11 >> --with-fortran-bindings=0 --with-sowing=0 --with-64-bit-indices >> --download-mumps=0 >> >> [640]PETSC ERROR: #1 PetscTableFind() line 132 in >> /home/wangy2/trunk/sawtooth/griffin/moose/petsc/include/petscctable.h >> >> [640]PETSC ERROR: #2 MatSetUpMultiply_MPIAIJ() line 33 in >> /home/wangy2/trunk/sawtooth/griffin/moose/petsc/src/mat/impls/aij/mpi/mmaij.c >> >> [640]PETSC ERROR: #3 MatAssemblyEnd_MPIAIJ() line 876 in >> /home/wangy2/trunk/sawtooth/griffin/moose/petsc/src/mat/impls/aij/mpi/mpiaij.c >> >> [640]PETSC ERROR: #4 MatAssemblyEnd() line 5347 in >> /home/wangy2/trunk/sawtooth/griffin/moose/petsc/src/mat/interface/matrix.c >> >> [640]PETSC ERROR: #5 MatPtAPNumeric_MPIAIJ_MPIXAIJ_allatonce() line 901 >> in >> /home/wangy2/trunk/sawtooth/griffin/moose/petsc/src/mat/impls/aij/mpi/mpiptap.c >> >> [640]PETSC ERROR: #6 MatPtAPNumeric_MPIAIJ_MPIMAIJ_allatonce() line 3180 >> in /home/wangy2/trunk/sawtooth/griffin/moose/petsc/src/mat/impls/maij/maij.c >> >> [640]PETSC ERROR: #7 MatProductNumeric_PtAP() line 704 in >> /home/wangy2/trunk/sawtooth/griffin/moose/petsc/src/mat/interface/matproduct.c >> >> [640]PETSC ERROR: #8 MatProductNumeric() line 759 in >> /home/wangy2/trunk/sawtooth/griffin/moose/petsc/src/mat/interface/matproduct.c >> >> [640]PETSC ERROR: #9 MatPtAP() line 9199 in >> /home/wangy2/trunk/sawtooth/griffin/moose/petsc/src/mat/interface/matrix.c >> >> [640]PETSC ERROR: #10 MatGalerkin() line 10236 in >> /home/wangy2/trunk/sawtooth/griffin/moose/petsc/src/mat/interface/matrix.c >> >> [640]PETSC ERROR: #11 PCSetUp_MG() line 745 in >> /home/wangy2/trunk/sawtooth/griffin/moose/petsc/src/ksp/pc/impls/mg/mg.c >> >> [640]PETSC ERROR: #12 PCSetUp_HMG() line 220 in >> /home/wangy2/trunk/sawtooth/griffin/moose/petsc/src/ksp/pc/impls/hmg/hmg.c >> >> [640]PETSC ERROR: #13 PCSetUp() line 898 in >> /home/wangy2/trunk/sawtooth/griffin/moose/petsc/src/ksp/pc/interface/precon.c >> >> [640]PETSC ERROR: #14 KSPSetUp() line 376 in >> /home/wangy2/trunk/sawtooth/griffin/moose/petsc/src/ksp/ksp/interface/itfunc.c >> >> [640]PETSC ERROR: #15 KSPSolve_Private() line 633 in >> /home/wangy2/trunk/sawtooth/griffin/moose/petsc/src/ksp/ksp/interface/itfunc.c >> >> [640]PETSC ERROR: #16 KSPSolve() line 853 in >> /home/wangy2/trunk/sawtooth/griffin/moose/petsc/src/ksp/ksp/interface/itfunc.c >> >> [640]PETSC ERROR: #17 SNESSolve_NEWTONLS() line 225 in >> /home/wangy2/trunk/sawtooth/griffin/moose/petsc/src/snes/impls/ls/ls.c >> >> [640]PETSC ERROR: #18 SNESSolve() line 4519 in >> /home/wangy2/trunk/sawtooth/griffin/moose/petsc/src/snes/interface/snes.c >> >> On Sun, Jul 19, 2020 at 11:13 PM Fande Kong wrote: >> >>> I am not entirely sure what is happening, but we encountered similar >>> issues recently. It was not reproducible. It might occur at different >>> stages, and errors could be weird other than "ctable stuff." Our code was >>> Valgrind clean since every PR in moose needs to go through rigorous >>> Valgrind checks before it reaches the devel branch. The errors happened >>> when we used mvapich. >>> >>> We changed to use HPE-MPT (a vendor stalled MPI), then everything was >>> smooth. May you try a different MPI? It is better to try a system carried >>> one. >>> >>> We did not get the bottom of this problem yet, but we at least know this >>> is kind of MPI-related. >>> >>> Thanks, >>> >>> Fande, >>> >>> >>> On Sun, Jul 19, 2020 at 3:28 PM Chris Hewson wrote: >>> >>>> Hi, >>>> >>>> I am having a bug that is occurring in PETSC with the return string: >>>> >>>> [7]PETSC ERROR: PetscTableFind() line 132 in >>>> /home/chewson/petsc-3.13.2/include/petscctable.h key 7556 is greater than >>>> largest key allowed 5693 >>>> >>>> This is using petsc-3.13.2, compiled and running using mpich with -O3 >>>> and debugging turned off tuned to the haswell architecture and >>>> occurring either before or during a KSPBCGS solve/setup or during a MUMPS >>>> factorization solve (I haven't been able to replicate this issue with the >>>> same set of instructions etc.). >>>> >>>> This is a terrible way to ask a question, I know, and not very helpful >>>> from your side, but this is what I have from a user's run and can't >>>> reproduce on my end (either with the optimization compilation or with >>>> debugging turned on). This happens when the code has run for quite some >>>> time and is happening somewhat rarely. >>>> >>>> More than likely I am using a static variable (code is written in c++) >>>> that I'm not updating when the matrix size is changing or something silly >>>> like that, but any help or guidance on this would be appreciated. >>>> >>>> *Chris Hewson* >>>> Senior Reservoir Simulation Engineer >>>> ResFrac >>>> +1.587.575.9792 >>>> >>> -------------- next part -------------- An HTML attachment was scrubbed... URL: From mfadams at lbl.gov Mon Jul 20 13:24:05 2020 From: mfadams at lbl.gov (Mark Adams) Date: Mon, 20 Jul 2020 14:24:05 -0400 Subject: [petsc-users] Tough to reproduce petsctablefind error In-Reply-To: References: Message-ID: OK, so this is happening in MatProductNumeric_PtAP. This must be in constructing the coarse grid. GAMG sort of wants to coarse at a rate of 30:1 but that needs to be verified. With that your index is at about the size of the first coarse grid. I'm trying to figure out if the index is valid. But the size of the max-index is 740521. This is about what I would guess is the size of the second coarse grid. So it kinda looks like it has a "fine" grid index in the "coarse" grid (2nd - 3rd coarse grids). But Chris is not using GAMG. Chris: It sounds like you just have one matrix that you give to MUMPS. You seem to be creating a matrix in the middle of your run. Are you doing dynamic adaptivity? I think we generate unique tags for each operation but it sounds like maybe a message is getting mixed up in some way. On Mon, Jul 20, 2020 at 12:35 PM Fande Kong wrote: > Hi Mark, > > Thanks for your reply. > > On Mon, Jul 20, 2020 at 7:13 AM Mark Adams wrote: > >> Fande, >> do you know if your 45226154 was out of range in the real matrix? >> > > I do not know since it was in building the AMG hierarchy. The size of the > original system is 1,428,284,880 > > >> What size integers do you use? >> > > We are using 64-bit via "--with-64-bit-indices" > > > I am trying to catch the cause of this issue by running more simulations > with different configurations. > > Thanks, > > Fande, > > > Thanks, >> Mark >> >> On Mon, Jul 20, 2020 at 1:17 AM Fande Kong wrote: >> >>> Trace could look like this: >>> >>> [640]PETSC ERROR: --------------------- Error Message >>> -------------------------------------------------------------- >>> >>> [640]PETSC ERROR: Argument out of range >>> >>> [640]PETSC ERROR: key 45226154 is greater than largest key allowed 740521 >>> >>> [640]PETSC ERROR: See >>> https://www.mcs.anl.gov/petsc/documentation/faq.html for trouble >>> shooting. >>> >>> [640]PETSC ERROR: Petsc Release Version 3.13.3, unknown >>> >>> [640]PETSC ERROR: ../../griffin-opt on a arch-moose named r6i5n18 by >>> wangy2 Sun Jul 19 17:14:28 2020 >>> >>> [640]PETSC ERROR: Configure options --download-hypre=1 >>> --with-debugging=no --with-shared-libraries=1 --download-fblaslapack=1 >>> --download-metis=1 --download-ptscotch=1 --download-parmetis=1 >>> --download-superlu_dist=1 --download-mumps=1 --download-scalapack=1 >>> --download-slepc=1 --with-mpi=1 --with-cxx-dialect=C++11 >>> --with-fortran-bindings=0 --with-sowing=0 --with-64-bit-indices >>> --download-mumps=0 >>> >>> [640]PETSC ERROR: #1 PetscTableFind() line 132 in >>> /home/wangy2/trunk/sawtooth/griffin/moose/petsc/include/petscctable.h >>> >>> [640]PETSC ERROR: #2 MatSetUpMultiply_MPIAIJ() line 33 in >>> /home/wangy2/trunk/sawtooth/griffin/moose/petsc/src/mat/impls/aij/mpi/mmaij.c >>> >>> [640]PETSC ERROR: #3 MatAssemblyEnd_MPIAIJ() line 876 in >>> /home/wangy2/trunk/sawtooth/griffin/moose/petsc/src/mat/impls/aij/mpi/mpiaij.c >>> >>> [640]PETSC ERROR: #4 MatAssemblyEnd() line 5347 in >>> /home/wangy2/trunk/sawtooth/griffin/moose/petsc/src/mat/interface/matrix.c >>> >>> [640]PETSC ERROR: #5 MatPtAPNumeric_MPIAIJ_MPIXAIJ_allatonce() line 901 >>> in >>> /home/wangy2/trunk/sawtooth/griffin/moose/petsc/src/mat/impls/aij/mpi/mpiptap.c >>> >>> [640]PETSC ERROR: #6 MatPtAPNumeric_MPIAIJ_MPIMAIJ_allatonce() line 3180 >>> in /home/wangy2/trunk/sawtooth/griffin/moose/petsc/src/mat/impls/maij/maij.c >>> >>> [640]PETSC ERROR: #7 MatProductNumeric_PtAP() line 704 in >>> /home/wangy2/trunk/sawtooth/griffin/moose/petsc/src/mat/interface/matproduct.c >>> >>> [640]PETSC ERROR: #8 MatProductNumeric() line 759 in >>> /home/wangy2/trunk/sawtooth/griffin/moose/petsc/src/mat/interface/matproduct.c >>> >>> [640]PETSC ERROR: #9 MatPtAP() line 9199 in >>> /home/wangy2/trunk/sawtooth/griffin/moose/petsc/src/mat/interface/matrix.c >>> >>> [640]PETSC ERROR: #10 MatGalerkin() line 10236 in >>> /home/wangy2/trunk/sawtooth/griffin/moose/petsc/src/mat/interface/matrix.c >>> >>> [640]PETSC ERROR: #11 PCSetUp_MG() line 745 in >>> /home/wangy2/trunk/sawtooth/griffin/moose/petsc/src/ksp/pc/impls/mg/mg.c >>> >>> [640]PETSC ERROR: #12 PCSetUp_HMG() line 220 in >>> /home/wangy2/trunk/sawtooth/griffin/moose/petsc/src/ksp/pc/impls/hmg/hmg.c >>> >>> [640]PETSC ERROR: #13 PCSetUp() line 898 in >>> /home/wangy2/trunk/sawtooth/griffin/moose/petsc/src/ksp/pc/interface/precon.c >>> >>> [640]PETSC ERROR: #14 KSPSetUp() line 376 in >>> /home/wangy2/trunk/sawtooth/griffin/moose/petsc/src/ksp/ksp/interface/itfunc.c >>> >>> [640]PETSC ERROR: #15 KSPSolve_Private() line 633 in >>> /home/wangy2/trunk/sawtooth/griffin/moose/petsc/src/ksp/ksp/interface/itfunc.c >>> >>> [640]PETSC ERROR: #16 KSPSolve() line 853 in >>> /home/wangy2/trunk/sawtooth/griffin/moose/petsc/src/ksp/ksp/interface/itfunc.c >>> >>> [640]PETSC ERROR: #17 SNESSolve_NEWTONLS() line 225 in >>> /home/wangy2/trunk/sawtooth/griffin/moose/petsc/src/snes/impls/ls/ls.c >>> >>> [640]PETSC ERROR: #18 SNESSolve() line 4519 in >>> /home/wangy2/trunk/sawtooth/griffin/moose/petsc/src/snes/interface/snes.c >>> >>> On Sun, Jul 19, 2020 at 11:13 PM Fande Kong wrote: >>> >>>> I am not entirely sure what is happening, but we encountered similar >>>> issues recently. It was not reproducible. It might occur at different >>>> stages, and errors could be weird other than "ctable stuff." Our code was >>>> Valgrind clean since every PR in moose needs to go through rigorous >>>> Valgrind checks before it reaches the devel branch. The errors happened >>>> when we used mvapich. >>>> >>>> We changed to use HPE-MPT (a vendor stalled MPI), then everything was >>>> smooth. May you try a different MPI? It is better to try a system carried >>>> one. >>>> >>>> We did not get the bottom of this problem yet, but we at least know >>>> this is kind of MPI-related. >>>> >>>> Thanks, >>>> >>>> Fande, >>>> >>>> >>>> On Sun, Jul 19, 2020 at 3:28 PM Chris Hewson wrote: >>>> >>>>> Hi, >>>>> >>>>> I am having a bug that is occurring in PETSC with the return string: >>>>> >>>>> [7]PETSC ERROR: PetscTableFind() line 132 in >>>>> /home/chewson/petsc-3.13.2/include/petscctable.h key 7556 is greater than >>>>> largest key allowed 5693 >>>>> >>>>> This is using petsc-3.13.2, compiled and running using mpich with -O3 >>>>> and debugging turned off tuned to the haswell architecture and >>>>> occurring either before or during a KSPBCGS solve/setup or during a MUMPS >>>>> factorization solve (I haven't been able to replicate this issue with the >>>>> same set of instructions etc.). >>>>> >>>>> This is a terrible way to ask a question, I know, and not very helpful >>>>> from your side, but this is what I have from a user's run and can't >>>>> reproduce on my end (either with the optimization compilation or with >>>>> debugging turned on). This happens when the code has run for quite some >>>>> time and is happening somewhat rarely. >>>>> >>>>> More than likely I am using a static variable (code is written in c++) >>>>> that I'm not updating when the matrix size is changing or something silly >>>>> like that, but any help or guidance on this would be appreciated. >>>>> >>>>> *Chris Hewson* >>>>> Senior Reservoir Simulation Engineer >>>>> ResFrac >>>>> +1.587.575.9792 >>>>> >>>> -------------- next part -------------- An HTML attachment was scrubbed... URL: From fdkong.jd at gmail.com Mon Jul 20 13:36:19 2020 From: fdkong.jd at gmail.com (Fande Kong) Date: Mon, 20 Jul 2020 12:36:19 -0600 Subject: [petsc-users] Tough to reproduce petsctablefind error In-Reply-To: References: Message-ID: Hi Mark, Just to be clear, I do not think it is related to GAMG or PtAP. It is a communication issue: Reran the same code, and I just got : [252]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [252]PETSC ERROR: Petsc has generated inconsistent data [252]PETSC ERROR: Received vector entry 4469094877509280860 out of local range [255426072,256718616)] [252]PETSC ERROR: See https://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [252]PETSC ERROR: Petsc Release Version 3.13.3, unknown [252]PETSC ERROR: ../../griffin-opt on a arch-moose named r5i4n13 by kongf Mon Jul 20 12:16:47 2020 [252]PETSC ERROR: Configure options --download-hypre=1 --with-debugging=no --with-shared-libraries=1 --download-fblaslapack=1 --download-metis=1 --download-ptscotch=1 --download-parmetis=1 --download-superlu_dist=1 --download-mumps=1 --download-scalapack=1 --download-slepc=1 --with-mpi=1 --with-cxx-dialect=C++11 --with-fortran-bindings=0 --with-sowing=0 --with-64-bit-indices --download-mumps=0 [252]PETSC ERROR: #1 VecAssemblyEnd_MPI_BTS() line 324 in /home/kongf/workhome/sawtooth/moosers/petsc/src/vec/vec/impls/mpi/pbvec.c [252]PETSC ERROR: #2 VecAssemblyEnd() line 171 in /home/kongf/workhome/sawtooth/moosers/petsc/src/vec/vec/interface/vector.c [cli_252]: aborting job: application called MPI_Abort(MPI_COMM_WORLD, 1) - process 252 Thanks, Fande, On Mon, Jul 20, 2020 at 12:24 PM Mark Adams wrote: > OK, so this is happening in MatProductNumeric_PtAP. This must be in > constructing the coarse grid. > > GAMG sort of wants to coarse at a rate of 30:1 but that needs to be > verified. With that your index is at about the size of the first coarse > grid. I'm trying to figure out if the index is valid. But the size of the > max-index is 740521. This is about what I would guess is the size of the > second coarse grid. > > So it kinda looks like it has a "fine" grid index in the "coarse" grid > (2nd - 3rd coarse grids). > > But Chris is not using GAMG. > > Chris: It sounds like you just have one matrix that you give to MUMPS. You > seem to be creating a matrix in the middle of your run. Are you doing > dynamic adaptivity? > > I think we generate unique tags for each operation but it sounds like > maybe a message is getting mixed up in some way. > > > > On Mon, Jul 20, 2020 at 12:35 PM Fande Kong wrote: > >> Hi Mark, >> >> Thanks for your reply. >> >> On Mon, Jul 20, 2020 at 7:13 AM Mark Adams wrote: >> >>> Fande, >>> do you know if your 45226154 was out of range in the real matrix? >>> >> >> I do not know since it was in building the AMG hierarchy. The size of >> the original system is 1,428,284,880 >> >> >>> What size integers do you use? >>> >> >> We are using 64-bit via "--with-64-bit-indices" >> >> >> I am trying to catch the cause of this issue by running more simulations >> with different configurations. >> >> Thanks, >> >> Fande, >> >> >> Thanks, >>> Mark >>> >>> On Mon, Jul 20, 2020 at 1:17 AM Fande Kong wrote: >>> >>>> Trace could look like this: >>>> >>>> [640]PETSC ERROR: --------------------- Error Message >>>> -------------------------------------------------------------- >>>> >>>> [640]PETSC ERROR: Argument out of range >>>> >>>> [640]PETSC ERROR: key 45226154 is greater than largest key allowed >>>> 740521 >>>> >>>> [640]PETSC ERROR: See >>>> https://www.mcs.anl.gov/petsc/documentation/faq.html for trouble >>>> shooting. >>>> >>>> [640]PETSC ERROR: Petsc Release Version 3.13.3, unknown >>>> >>>> [640]PETSC ERROR: ../../griffin-opt on a arch-moose named r6i5n18 by >>>> wangy2 Sun Jul 19 17:14:28 2020 >>>> >>>> [640]PETSC ERROR: Configure options --download-hypre=1 >>>> --with-debugging=no --with-shared-libraries=1 --download-fblaslapack=1 >>>> --download-metis=1 --download-ptscotch=1 --download-parmetis=1 >>>> --download-superlu_dist=1 --download-mumps=1 --download-scalapack=1 >>>> --download-slepc=1 --with-mpi=1 --with-cxx-dialect=C++11 >>>> --with-fortran-bindings=0 --with-sowing=0 --with-64-bit-indices >>>> --download-mumps=0 >>>> >>>> [640]PETSC ERROR: #1 PetscTableFind() line 132 in >>>> /home/wangy2/trunk/sawtooth/griffin/moose/petsc/include/petscctable.h >>>> >>>> [640]PETSC ERROR: #2 MatSetUpMultiply_MPIAIJ() line 33 in >>>> /home/wangy2/trunk/sawtooth/griffin/moose/petsc/src/mat/impls/aij/mpi/mmaij.c >>>> >>>> [640]PETSC ERROR: #3 MatAssemblyEnd_MPIAIJ() line 876 in >>>> /home/wangy2/trunk/sawtooth/griffin/moose/petsc/src/mat/impls/aij/mpi/mpiaij.c >>>> >>>> [640]PETSC ERROR: #4 MatAssemblyEnd() line 5347 in >>>> /home/wangy2/trunk/sawtooth/griffin/moose/petsc/src/mat/interface/matrix.c >>>> >>>> [640]PETSC ERROR: #5 MatPtAPNumeric_MPIAIJ_MPIXAIJ_allatonce() line 901 >>>> in >>>> /home/wangy2/trunk/sawtooth/griffin/moose/petsc/src/mat/impls/aij/mpi/mpiptap.c >>>> >>>> [640]PETSC ERROR: #6 MatPtAPNumeric_MPIAIJ_MPIMAIJ_allatonce() line >>>> 3180 in >>>> /home/wangy2/trunk/sawtooth/griffin/moose/petsc/src/mat/impls/maij/maij.c >>>> >>>> [640]PETSC ERROR: #7 MatProductNumeric_PtAP() line 704 in >>>> /home/wangy2/trunk/sawtooth/griffin/moose/petsc/src/mat/interface/matproduct.c >>>> >>>> [640]PETSC ERROR: #8 MatProductNumeric() line 759 in >>>> /home/wangy2/trunk/sawtooth/griffin/moose/petsc/src/mat/interface/matproduct.c >>>> >>>> [640]PETSC ERROR: #9 MatPtAP() line 9199 in >>>> /home/wangy2/trunk/sawtooth/griffin/moose/petsc/src/mat/interface/matrix.c >>>> >>>> [640]PETSC ERROR: #10 MatGalerkin() line 10236 in >>>> /home/wangy2/trunk/sawtooth/griffin/moose/petsc/src/mat/interface/matrix.c >>>> >>>> [640]PETSC ERROR: #11 PCSetUp_MG() line 745 in >>>> /home/wangy2/trunk/sawtooth/griffin/moose/petsc/src/ksp/pc/impls/mg/mg.c >>>> >>>> [640]PETSC ERROR: #12 PCSetUp_HMG() line 220 in >>>> /home/wangy2/trunk/sawtooth/griffin/moose/petsc/src/ksp/pc/impls/hmg/hmg.c >>>> >>>> [640]PETSC ERROR: #13 PCSetUp() line 898 in >>>> /home/wangy2/trunk/sawtooth/griffin/moose/petsc/src/ksp/pc/interface/precon.c >>>> >>>> [640]PETSC ERROR: #14 KSPSetUp() line 376 in >>>> /home/wangy2/trunk/sawtooth/griffin/moose/petsc/src/ksp/ksp/interface/itfunc.c >>>> >>>> [640]PETSC ERROR: #15 KSPSolve_Private() line 633 in >>>> /home/wangy2/trunk/sawtooth/griffin/moose/petsc/src/ksp/ksp/interface/itfunc.c >>>> >>>> [640]PETSC ERROR: #16 KSPSolve() line 853 in >>>> /home/wangy2/trunk/sawtooth/griffin/moose/petsc/src/ksp/ksp/interface/itfunc.c >>>> >>>> [640]PETSC ERROR: #17 SNESSolve_NEWTONLS() line 225 in >>>> /home/wangy2/trunk/sawtooth/griffin/moose/petsc/src/snes/impls/ls/ls.c >>>> >>>> [640]PETSC ERROR: #18 SNESSolve() line 4519 in >>>> /home/wangy2/trunk/sawtooth/griffin/moose/petsc/src/snes/interface/snes.c >>>> >>>> On Sun, Jul 19, 2020 at 11:13 PM Fande Kong >>>> wrote: >>>> >>>>> I am not entirely sure what is happening, but we encountered similar >>>>> issues recently. It was not reproducible. It might occur at different >>>>> stages, and errors could be weird other than "ctable stuff." Our code was >>>>> Valgrind clean since every PR in moose needs to go through rigorous >>>>> Valgrind checks before it reaches the devel branch. The errors happened >>>>> when we used mvapich. >>>>> >>>>> We changed to use HPE-MPT (a vendor stalled MPI), then everything was >>>>> smooth. May you try a different MPI? It is better to try a system carried >>>>> one. >>>>> >>>>> We did not get the bottom of this problem yet, but we at least know >>>>> this is kind of MPI-related. >>>>> >>>>> Thanks, >>>>> >>>>> Fande, >>>>> >>>>> >>>>> On Sun, Jul 19, 2020 at 3:28 PM Chris Hewson >>>>> wrote: >>>>> >>>>>> Hi, >>>>>> >>>>>> I am having a bug that is occurring in PETSC with the return string: >>>>>> >>>>>> [7]PETSC ERROR: PetscTableFind() line 132 in >>>>>> /home/chewson/petsc-3.13.2/include/petscctable.h key 7556 is greater than >>>>>> largest key allowed 5693 >>>>>> >>>>>> This is using petsc-3.13.2, compiled and running using mpich with -O3 >>>>>> and debugging turned off tuned to the haswell architecture and >>>>>> occurring either before or during a KSPBCGS solve/setup or during a MUMPS >>>>>> factorization solve (I haven't been able to replicate this issue with the >>>>>> same set of instructions etc.). >>>>>> >>>>>> This is a terrible way to ask a question, I know, and not very >>>>>> helpful from your side, but this is what I have from a user's run and can't >>>>>> reproduce on my end (either with the optimization compilation or with >>>>>> debugging turned on). This happens when the code has run for quite some >>>>>> time and is happening somewhat rarely. >>>>>> >>>>>> More than likely I am using a static variable (code is written in >>>>>> c++) that I'm not updating when the matrix size is changing or something >>>>>> silly like that, but any help or guidance on this would be appreciated. >>>>>> >>>>>> *Chris Hewson* >>>>>> Senior Reservoir Simulation Engineer >>>>>> ResFrac >>>>>> +1.587.575.9792 >>>>>> >>>>> -------------- next part -------------- An HTML attachment was scrubbed... URL: From fdkong.jd at gmail.com Mon Jul 20 13:38:31 2020 From: fdkong.jd at gmail.com (Fande Kong) Date: Mon, 20 Jul 2020 12:38:31 -0600 Subject: [petsc-users] Tough to reproduce petsctablefind error In-Reply-To: References: Message-ID: The most frustrating part is that the issue is not reproducible. Fande, On Mon, Jul 20, 2020 at 12:36 PM Fande Kong wrote: > Hi Mark, > > Just to be clear, I do not think it is related to GAMG or PtAP. It is a > communication issue: > > Reran the same code, and I just got : > > [252]PETSC ERROR: --------------------- Error Message > -------------------------------------------------------------- > [252]PETSC ERROR: Petsc has generated inconsistent data > [252]PETSC ERROR: Received vector entry 4469094877509280860 out of local > range [255426072,256718616)] > [252]PETSC ERROR: See https://www.mcs.anl.gov/petsc/documentation/faq.html > for trouble shooting. > [252]PETSC ERROR: Petsc Release Version 3.13.3, unknown > [252]PETSC ERROR: ../../griffin-opt on a arch-moose named r5i4n13 by kongf > Mon Jul 20 12:16:47 2020 > [252]PETSC ERROR: Configure options --download-hypre=1 --with-debugging=no > --with-shared-libraries=1 --download-fblaslapack=1 --download-metis=1 > --download-ptscotch=1 --download-parmetis=1 --download-superlu_dist=1 > --download-mumps=1 --download-scalapack=1 --download-slepc=1 --with-mpi=1 > --with-cxx-dialect=C++11 --with-fortran-bindings=0 --with-sowing=0 > --with-64-bit-indices --download-mumps=0 > [252]PETSC ERROR: #1 VecAssemblyEnd_MPI_BTS() line 324 in > /home/kongf/workhome/sawtooth/moosers/petsc/src/vec/vec/impls/mpi/pbvec.c > [252]PETSC ERROR: #2 VecAssemblyEnd() line 171 in > /home/kongf/workhome/sawtooth/moosers/petsc/src/vec/vec/interface/vector.c > [cli_252]: aborting job: > application called MPI_Abort(MPI_COMM_WORLD, 1) - process 252 > > > Thanks, > > Fande, > > On Mon, Jul 20, 2020 at 12:24 PM Mark Adams wrote: > >> OK, so this is happening in MatProductNumeric_PtAP. This must be in >> constructing the coarse grid. >> >> GAMG sort of wants to coarse at a rate of 30:1 but that needs to be >> verified. With that your index is at about the size of the first coarse >> grid. I'm trying to figure out if the index is valid. But the size of the >> max-index is 740521. This is about what I would guess is the size of the >> second coarse grid. >> >> So it kinda looks like it has a "fine" grid index in the "coarse" grid >> (2nd - 3rd coarse grids). >> >> But Chris is not using GAMG. >> >> Chris: It sounds like you just have one matrix that you give to MUMPS. >> You seem to be creating a matrix in the middle of your run. Are you doing >> dynamic adaptivity? >> >> I think we generate unique tags for each operation but it sounds like >> maybe a message is getting mixed up in some way. >> >> >> >> On Mon, Jul 20, 2020 at 12:35 PM Fande Kong wrote: >> >>> Hi Mark, >>> >>> Thanks for your reply. >>> >>> On Mon, Jul 20, 2020 at 7:13 AM Mark Adams wrote: >>> >>>> Fande, >>>> do you know if your 45226154 was out of range in the real matrix? >>>> >>> >>> I do not know since it was in building the AMG hierarchy. The size of >>> the original system is 1,428,284,880 >>> >>> >>>> What size integers do you use? >>>> >>> >>> We are using 64-bit via "--with-64-bit-indices" >>> >>> >>> I am trying to catch the cause of this issue by running more simulations >>> with different configurations. >>> >>> Thanks, >>> >>> Fande, >>> >>> >>> Thanks, >>>> Mark >>>> >>>> On Mon, Jul 20, 2020 at 1:17 AM Fande Kong wrote: >>>> >>>>> Trace could look like this: >>>>> >>>>> [640]PETSC ERROR: --------------------- Error Message >>>>> -------------------------------------------------------------- >>>>> >>>>> [640]PETSC ERROR: Argument out of range >>>>> >>>>> [640]PETSC ERROR: key 45226154 is greater than largest key allowed >>>>> 740521 >>>>> >>>>> [640]PETSC ERROR: See >>>>> https://www.mcs.anl.gov/petsc/documentation/faq.html for trouble >>>>> shooting. >>>>> >>>>> [640]PETSC ERROR: Petsc Release Version 3.13.3, unknown >>>>> >>>>> [640]PETSC ERROR: ../../griffin-opt on a arch-moose named r6i5n18 by >>>>> wangy2 Sun Jul 19 17:14:28 2020 >>>>> >>>>> [640]PETSC ERROR: Configure options --download-hypre=1 >>>>> --with-debugging=no --with-shared-libraries=1 --download-fblaslapack=1 >>>>> --download-metis=1 --download-ptscotch=1 --download-parmetis=1 >>>>> --download-superlu_dist=1 --download-mumps=1 --download-scalapack=1 >>>>> --download-slepc=1 --with-mpi=1 --with-cxx-dialect=C++11 >>>>> --with-fortran-bindings=0 --with-sowing=0 --with-64-bit-indices >>>>> --download-mumps=0 >>>>> >>>>> [640]PETSC ERROR: #1 PetscTableFind() line 132 in >>>>> /home/wangy2/trunk/sawtooth/griffin/moose/petsc/include/petscctable.h >>>>> >>>>> [640]PETSC ERROR: #2 MatSetUpMultiply_MPIAIJ() line 33 in >>>>> /home/wangy2/trunk/sawtooth/griffin/moose/petsc/src/mat/impls/aij/mpi/mmaij.c >>>>> >>>>> [640]PETSC ERROR: #3 MatAssemblyEnd_MPIAIJ() line 876 in >>>>> /home/wangy2/trunk/sawtooth/griffin/moose/petsc/src/mat/impls/aij/mpi/mpiaij.c >>>>> >>>>> [640]PETSC ERROR: #4 MatAssemblyEnd() line 5347 in >>>>> /home/wangy2/trunk/sawtooth/griffin/moose/petsc/src/mat/interface/matrix.c >>>>> >>>>> [640]PETSC ERROR: #5 MatPtAPNumeric_MPIAIJ_MPIXAIJ_allatonce() line >>>>> 901 in >>>>> /home/wangy2/trunk/sawtooth/griffin/moose/petsc/src/mat/impls/aij/mpi/mpiptap.c >>>>> >>>>> [640]PETSC ERROR: #6 MatPtAPNumeric_MPIAIJ_MPIMAIJ_allatonce() line >>>>> 3180 in >>>>> /home/wangy2/trunk/sawtooth/griffin/moose/petsc/src/mat/impls/maij/maij.c >>>>> >>>>> [640]PETSC ERROR: #7 MatProductNumeric_PtAP() line 704 in >>>>> /home/wangy2/trunk/sawtooth/griffin/moose/petsc/src/mat/interface/matproduct.c >>>>> >>>>> [640]PETSC ERROR: #8 MatProductNumeric() line 759 in >>>>> /home/wangy2/trunk/sawtooth/griffin/moose/petsc/src/mat/interface/matproduct.c >>>>> >>>>> [640]PETSC ERROR: #9 MatPtAP() line 9199 in >>>>> /home/wangy2/trunk/sawtooth/griffin/moose/petsc/src/mat/interface/matrix.c >>>>> >>>>> [640]PETSC ERROR: #10 MatGalerkin() line 10236 in >>>>> /home/wangy2/trunk/sawtooth/griffin/moose/petsc/src/mat/interface/matrix.c >>>>> >>>>> [640]PETSC ERROR: #11 PCSetUp_MG() line 745 in >>>>> /home/wangy2/trunk/sawtooth/griffin/moose/petsc/src/ksp/pc/impls/mg/mg.c >>>>> >>>>> [640]PETSC ERROR: #12 PCSetUp_HMG() line 220 in >>>>> /home/wangy2/trunk/sawtooth/griffin/moose/petsc/src/ksp/pc/impls/hmg/hmg.c >>>>> >>>>> [640]PETSC ERROR: #13 PCSetUp() line 898 in >>>>> /home/wangy2/trunk/sawtooth/griffin/moose/petsc/src/ksp/pc/interface/precon.c >>>>> >>>>> [640]PETSC ERROR: #14 KSPSetUp() line 376 in >>>>> /home/wangy2/trunk/sawtooth/griffin/moose/petsc/src/ksp/ksp/interface/itfunc.c >>>>> >>>>> [640]PETSC ERROR: #15 KSPSolve_Private() line 633 in >>>>> /home/wangy2/trunk/sawtooth/griffin/moose/petsc/src/ksp/ksp/interface/itfunc.c >>>>> >>>>> [640]PETSC ERROR: #16 KSPSolve() line 853 in >>>>> /home/wangy2/trunk/sawtooth/griffin/moose/petsc/src/ksp/ksp/interface/itfunc.c >>>>> >>>>> [640]PETSC ERROR: #17 SNESSolve_NEWTONLS() line 225 in >>>>> /home/wangy2/trunk/sawtooth/griffin/moose/petsc/src/snes/impls/ls/ls.c >>>>> >>>>> [640]PETSC ERROR: #18 SNESSolve() line 4519 in >>>>> /home/wangy2/trunk/sawtooth/griffin/moose/petsc/src/snes/interface/snes.c >>>>> >>>>> On Sun, Jul 19, 2020 at 11:13 PM Fande Kong >>>>> wrote: >>>>> >>>>>> I am not entirely sure what is happening, but we encountered similar >>>>>> issues recently. It was not reproducible. It might occur at different >>>>>> stages, and errors could be weird other than "ctable stuff." Our code was >>>>>> Valgrind clean since every PR in moose needs to go through rigorous >>>>>> Valgrind checks before it reaches the devel branch. The errors happened >>>>>> when we used mvapich. >>>>>> >>>>>> We changed to use HPE-MPT (a vendor stalled MPI), then everything was >>>>>> smooth. May you try a different MPI? It is better to try a system carried >>>>>> one. >>>>>> >>>>>> We did not get the bottom of this problem yet, but we at least know >>>>>> this is kind of MPI-related. >>>>>> >>>>>> Thanks, >>>>>> >>>>>> Fande, >>>>>> >>>>>> >>>>>> On Sun, Jul 19, 2020 at 3:28 PM Chris Hewson >>>>>> wrote: >>>>>> >>>>>>> Hi, >>>>>>> >>>>>>> I am having a bug that is occurring in PETSC with the return string: >>>>>>> >>>>>>> [7]PETSC ERROR: PetscTableFind() line 132 in >>>>>>> /home/chewson/petsc-3.13.2/include/petscctable.h key 7556 is greater than >>>>>>> largest key allowed 5693 >>>>>>> >>>>>>> This is using petsc-3.13.2, compiled and running using mpich with >>>>>>> -O3 and debugging turned off tuned to the haswell architecture and >>>>>>> occurring either before or during a KSPBCGS solve/setup or during a MUMPS >>>>>>> factorization solve (I haven't been able to replicate this issue with the >>>>>>> same set of instructions etc.). >>>>>>> >>>>>>> This is a terrible way to ask a question, I know, and not very >>>>>>> helpful from your side, but this is what I have from a user's run and can't >>>>>>> reproduce on my end (either with the optimization compilation or with >>>>>>> debugging turned on). This happens when the code has run for quite some >>>>>>> time and is happening somewhat rarely. >>>>>>> >>>>>>> More than likely I am using a static variable (code is written in >>>>>>> c++) that I'm not updating when the matrix size is changing or something >>>>>>> silly like that, but any help or guidance on this would be appreciated. >>>>>>> >>>>>>> *Chris Hewson* >>>>>>> Senior Reservoir Simulation Engineer >>>>>>> ResFrac >>>>>>> +1.587.575.9792 >>>>>>> >>>>>> -------------- next part -------------- An HTML attachment was scrubbed... URL: From mfadams at lbl.gov Mon Jul 20 13:41:19 2020 From: mfadams at lbl.gov (Mark Adams) Date: Mon, 20 Jul 2020 14:41:19 -0400 Subject: [petsc-users] Tough to reproduce petsctablefind error In-Reply-To: References: Message-ID: On Mon, Jul 20, 2020 at 2:36 PM Fande Kong wrote: > Hi Mark, > > Just to be clear, I do not think it is related to GAMG or PtAP. It is a > communication issue: > Youe stack trace was from PtAP, but Chris's problem is not. > > Reran the same code, and I just got : > > [252]PETSC ERROR: --------------------- Error Message > -------------------------------------------------------------- > [252]PETSC ERROR: Petsc has generated inconsistent data > [252]PETSC ERROR: Received vector entry 4469094877509280860 out of local > range [255426072,256718616)] > OK, now this (4469094877509280860) is clearly garbage. THat is the important thing. I have to think your MPI is buggy. -------------- next part -------------- An HTML attachment was scrubbed... URL: From chris at resfrac.com Mon Jul 20 14:04:07 2020 From: chris at resfrac.com (Chris Hewson) Date: Mon, 20 Jul 2020 13:04:07 -0600 Subject: [petsc-users] Tough to reproduce petsctablefind error In-Reply-To: References: Message-ID: Hi Mark, Chris: It sounds like you just have one matrix that you give to MUMPS. You seem to be creating a matrix in the middle of your run. Are you doing dynamic adaptivity? - I have 2 separate matrices I give to mumps, but as this is happening in the production build of my code, I can't determine with certainty what call to MUMPS it's happening or what call to KSPBCGS or UMFPACK it's happening in. I do destroy and recreate matrices in the middle of my runs, but this happens multiple times before the fault happens and in (presumably) the same way. I also do checks on matrix sizes and what I am sending to PETSc and those all pass, just at some point there are size mismatches somewhere, understandably this is not a lot to go on. I am not doing dynamic adaptivity, the mesh is instead changing its size. And I agree with Fande, the most frustrating part is that it's not reproducible, but yah not 100% sure that the problem lies within the PETSc code base either. Current working theories are: 1. Some sort of MPI problem with the sending of one the matrix elements (using mpich version 3.3a2) 2. Some of the memory of static pointers gets corrupted, although I would expect a garbage number and not something that could possibly make sense. *Chris Hewson* Senior Reservoir Simulation Engineer ResFrac +1.587.575.9792 On Mon, Jul 20, 2020 at 12:41 PM Mark Adams wrote: > > > On Mon, Jul 20, 2020 at 2:36 PM Fande Kong wrote: > >> Hi Mark, >> >> Just to be clear, I do not think it is related to GAMG or PtAP. It is a >> communication issue: >> > > Youe stack trace was from PtAP, but Chris's problem is not. > > >> >> Reran the same code, and I just got : >> >> [252]PETSC ERROR: --------------------- Error Message >> -------------------------------------------------------------- >> [252]PETSC ERROR: Petsc has generated inconsistent data >> [252]PETSC ERROR: Received vector entry 4469094877509280860 out of local >> range [255426072,256718616)] >> > > OK, now this (4469094877509280860) is clearly garbage. THat is the > important thing. I have to think your MPI is buggy. > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mfadams at lbl.gov Mon Jul 20 14:14:26 2020 From: mfadams at lbl.gov (Mark Adams) Date: Mon, 20 Jul 2020 15:14:26 -0400 Subject: [petsc-users] Tough to reproduce petsctablefind error In-Reply-To: References: Message-ID: This is indeed a nasty bug, but having two separate should be useful. Chris is using Haswell, what MPI are you using? I trust you are not using Moose. Fande what machine/MPI are you using? On Mon, Jul 20, 2020 at 3:04 PM Chris Hewson wrote: > Hi Mark, > > Chris: It sounds like you just have one matrix that you give to MUMPS. You > seem to be creating a matrix in the middle of your run. Are you doing > dynamic adaptivity? > - I have 2 separate matrices I give to mumps, but as this is happening in > the production build of my code, I can't determine with certainty what call > to MUMPS it's happening or what call to KSPBCGS or UMFPACK it's happening > in. > > I do destroy and recreate matrices in the middle of my runs, but this > happens multiple times before the fault happens and in (presumably) the > same way. I also do checks on matrix sizes and what I am sending to PETSc > and those all pass, just at some point there are size mismatches > somewhere, understandably this is not a lot to go on. I am not doing > dynamic adaptivity, the mesh is instead changing its size. > > And I agree with Fande, the most frustrating part is that it's not > reproducible, but yah not 100% sure that the problem lies within the PETSc > code base either. > > Current working theories are: > 1. Some sort of MPI problem with the sending of one the matrix elements > (using mpich version 3.3a2) > 2. Some of the memory of static pointers gets corrupted, although I would > expect a garbage number and not something that could possibly make sense. > > *Chris Hewson* > Senior Reservoir Simulation Engineer > ResFrac > +1.587.575.9792 > > > On Mon, Jul 20, 2020 at 12:41 PM Mark Adams wrote: > >> >> >> On Mon, Jul 20, 2020 at 2:36 PM Fande Kong wrote: >> >>> Hi Mark, >>> >>> Just to be clear, I do not think it is related to GAMG or PtAP. It is a >>> communication issue: >>> >> >> Youe stack trace was from PtAP, but Chris's problem is not. >> >> >>> >>> Reran the same code, and I just got : >>> >>> [252]PETSC ERROR: --------------------- Error Message >>> -------------------------------------------------------------- >>> [252]PETSC ERROR: Petsc has generated inconsistent data >>> [252]PETSC ERROR: Received vector entry 4469094877509280860 out of local >>> range [255426072,256718616)] >>> >> >> OK, now this (4469094877509280860) is clearly garbage. THat is the >> important thing. I have to think your MPI is buggy. >> >> >> -------------- next part -------------- An HTML attachment was scrubbed... URL: From chris at resfrac.com Mon Jul 20 14:25:26 2020 From: chris at resfrac.com (Chris Hewson) Date: Mon, 20 Jul 2020 13:25:26 -0600 Subject: [petsc-users] Tough to reproduce petsctablefind error In-Reply-To: References: Message-ID: Chris is using Haswell, what MPI are you using? I trust you are not using Moose. - yes, using haswell, mpi is mpich v3.3a2 on ubuntu 18.04. I am not using MOOSE. *Chris Hewson* Senior Reservoir Simulation Engineer ResFrac +1.587.575.9792 On Mon, Jul 20, 2020 at 1:14 PM Mark Adams wrote: > This is indeed a nasty bug, but having two separate should be useful. > > Chris is using Haswell, what MPI are you using? I trust you are not using > Moose. > > Fande what machine/MPI are you using? > > On Mon, Jul 20, 2020 at 3:04 PM Chris Hewson wrote: > >> Hi Mark, >> >> Chris: It sounds like you just have one matrix that you give to MUMPS. >> You seem to be creating a matrix in the middle of your run. Are you doing >> dynamic adaptivity? >> - I have 2 separate matrices I give to mumps, but as this is happening in >> the production build of my code, I can't determine with certainty what call >> to MUMPS it's happening or what call to KSPBCGS or UMFPACK it's happening >> in. >> >> I do destroy and recreate matrices in the middle of my runs, but this >> happens multiple times before the fault happens and in (presumably) the >> same way. I also do checks on matrix sizes and what I am sending to PETSc >> and those all pass, just at some point there are size mismatches >> somewhere, understandably this is not a lot to go on. I am not doing >> dynamic adaptivity, the mesh is instead changing its size. >> >> And I agree with Fande, the most frustrating part is that it's not >> reproducible, but yah not 100% sure that the problem lies within the PETSc >> code base either. >> >> Current working theories are: >> 1. Some sort of MPI problem with the sending of one the matrix elements >> (using mpich version 3.3a2) >> 2. Some of the memory of static pointers gets corrupted, although I would >> expect a garbage number and not something that could possibly make sense. >> >> *Chris Hewson* >> Senior Reservoir Simulation Engineer >> ResFrac >> +1.587.575.9792 >> >> >> On Mon, Jul 20, 2020 at 12:41 PM Mark Adams wrote: >> >>> >>> >>> On Mon, Jul 20, 2020 at 2:36 PM Fande Kong wrote: >>> >>>> Hi Mark, >>>> >>>> Just to be clear, I do not think it is related to GAMG or PtAP. It is a >>>> communication issue: >>>> >>> >>> Youe stack trace was from PtAP, but Chris's problem is not. >>> >>> >>>> >>>> Reran the same code, and I just got : >>>> >>>> [252]PETSC ERROR: --------------------- Error Message >>>> -------------------------------------------------------------- >>>> [252]PETSC ERROR: Petsc has generated inconsistent data >>>> [252]PETSC ERROR: Received vector entry 4469094877509280860 out of >>>> local range [255426072,256718616)] >>>> >>> >>> OK, now this (4469094877509280860) is clearly garbage. THat is the >>> important thing. I have to think your MPI is buggy. >>> >>> >>> -------------- next part -------------- An HTML attachment was scrubbed... URL: From junchao.zhang at gmail.com Mon Jul 20 15:30:11 2020 From: junchao.zhang at gmail.com (Junchao Zhang) Date: Mon, 20 Jul 2020 15:30:11 -0500 Subject: [petsc-users] Tough to reproduce petsctablefind error In-Reply-To: References: Message-ID: On Mon, Jul 20, 2020 at 2:26 PM Chris Hewson wrote: > Chris is using Haswell, what MPI are you using? I trust you are not using > Moose. > - yes, using haswell, mpi is mpich v3.3a2 on ubuntu 18.04. I am not using > MOOSE. > Do not use mpich v3.3a2, which is an alpha version released in 2016. Use current stable release mpich-3.3.2 > > > > *Chris Hewson* > Senior Reservoir Simulation Engineer > ResFrac > +1.587.575.9792 > > > On Mon, Jul 20, 2020 at 1:14 PM Mark Adams wrote: > >> This is indeed a nasty bug, but having two separate should be useful. >> >> Chris is using Haswell, what MPI are you using? I trust you are not using >> Moose. >> >> Fande what machine/MPI are you using? >> >> On Mon, Jul 20, 2020 at 3:04 PM Chris Hewson wrote: >> >>> Hi Mark, >>> >>> Chris: It sounds like you just have one matrix that you give to MUMPS. >>> You seem to be creating a matrix in the middle of your run. Are you doing >>> dynamic adaptivity? >>> - I have 2 separate matrices I give to mumps, but as this is happening >>> in the production build of my code, I can't determine with certainty what >>> call to MUMPS it's happening or what call to KSPBCGS or UMFPACK it's >>> happening in. >>> >>> I do destroy and recreate matrices in the middle of my runs, but this >>> happens multiple times before the fault happens and in (presumably) the >>> same way. I also do checks on matrix sizes and what I am sending to PETSc >>> and those all pass, just at some point there are size mismatches >>> somewhere, understandably this is not a lot to go on. I am not doing >>> dynamic adaptivity, the mesh is instead changing its size. >>> >>> And I agree with Fande, the most frustrating part is that it's not >>> reproducible, but yah not 100% sure that the problem lies within the PETSc >>> code base either. >>> >>> Current working theories are: >>> 1. Some sort of MPI problem with the sending of one the matrix elements >>> (using mpich version 3.3a2) >>> 2. Some of the memory of static pointers gets corrupted, although I >>> would expect a garbage number and not something that could possibly make >>> sense. >>> >>> *Chris Hewson* >>> Senior Reservoir Simulation Engineer >>> ResFrac >>> +1.587.575.9792 >>> >>> >>> On Mon, Jul 20, 2020 at 12:41 PM Mark Adams wrote: >>> >>>> >>>> >>>> On Mon, Jul 20, 2020 at 2:36 PM Fande Kong wrote: >>>> >>>>> Hi Mark, >>>>> >>>>> Just to be clear, I do not think it is related to GAMG or PtAP. It is >>>>> a communication issue: >>>>> >>>> >>>> Youe stack trace was from PtAP, but Chris's problem is not. >>>> >>>> >>>>> >>>>> Reran the same code, and I just got : >>>>> >>>>> [252]PETSC ERROR: --------------------- Error Message >>>>> -------------------------------------------------------------- >>>>> [252]PETSC ERROR: Petsc has generated inconsistent data >>>>> [252]PETSC ERROR: Received vector entry 4469094877509280860 out of >>>>> local range [255426072,256718616)] >>>>> >>>> >>>> OK, now this (4469094877509280860) is clearly garbage. THat is the >>>> important thing. I have to think your MPI is buggy. >>>> >>>> >>>> -------------- next part -------------- An HTML attachment was scrubbed... URL: From fdkong.jd at gmail.com Mon Jul 20 17:19:55 2020 From: fdkong.jd at gmail.com (Fande Kong) Date: Mon, 20 Jul 2020 16:19:55 -0600 Subject: [petsc-users] Tough to reproduce petsctablefind error In-Reply-To: References: Message-ID: On Mon, Jul 20, 2020 at 1:14 PM Mark Adams wrote: > This is indeed a nasty bug, but having two separate should be useful. > > Chris is using Haswell, what MPI are you using? I trust you are not using > Moose. > > Fande what machine/MPI are you using? > #define PETSC_MPICC_SHOW "/apps/local/spack/software/gcc-4.8.5/gcc-9.2.0-bxc7mvbmrfcrusa6ij7ux3exfqabmq5y/bin/gcc -I/apps/local/mvapich2/2.3.3-gcc-9.2.0/include -L/apps/local/mvapich2/2.3.3-gcc-9.2.0/lib -Wl,-rpath -Wl,/apps/local/mvapich2/2.3.3-gcc-9.2.0/lib -Wl,--enable-new-dtags -lmpi" I guess it is mvapich2-2.3.3. Here is the machine configuration https://www.top500.org/system/179708/ BTW (if you missed my earlier posts), if I switch to MPT-MPI (a vendor installed MPI), everything runs well so far. I will stick with MPT from now. Thanks, Fande, > > On Mon, Jul 20, 2020 at 3:04 PM Chris Hewson wrote: > >> Hi Mark, >> >> Chris: It sounds like you just have one matrix that you give to MUMPS. >> You seem to be creating a matrix in the middle of your run. Are you doing >> dynamic adaptivity? >> - I have 2 separate matrices I give to mumps, but as this is happening in >> the production build of my code, I can't determine with certainty what call >> to MUMPS it's happening or what call to KSPBCGS or UMFPACK it's happening >> in. >> >> I do destroy and recreate matrices in the middle of my runs, but this >> happens multiple times before the fault happens and in (presumably) the >> same way. I also do checks on matrix sizes and what I am sending to PETSc >> and those all pass, just at some point there are size mismatches >> somewhere, understandably this is not a lot to go on. I am not doing >> dynamic adaptivity, the mesh is instead changing its size. >> >> And I agree with Fande, the most frustrating part is that it's not >> reproducible, but yah not 100% sure that the problem lies within the PETSc >> code base either. >> >> Current working theories are: >> 1. Some sort of MPI problem with the sending of one the matrix elements >> (using mpich version 3.3a2) >> 2. Some of the memory of static pointers gets corrupted, although I would >> expect a garbage number and not something that could possibly make sense. >> >> *Chris Hewson* >> Senior Reservoir Simulation Engineer >> ResFrac >> +1.587.575.9792 >> >> >> On Mon, Jul 20, 2020 at 12:41 PM Mark Adams wrote: >> >>> >>> >>> On Mon, Jul 20, 2020 at 2:36 PM Fande Kong wrote: >>> >>>> Hi Mark, >>>> >>>> Just to be clear, I do not think it is related to GAMG or PtAP. It is a >>>> communication issue: >>>> >>> >>> Youe stack trace was from PtAP, but Chris's problem is not. >>> >>> >>>> >>>> Reran the same code, and I just got : >>>> >>>> [252]PETSC ERROR: --------------------- Error Message >>>> -------------------------------------------------------------- >>>> [252]PETSC ERROR: Petsc has generated inconsistent data >>>> [252]PETSC ERROR: Received vector entry 4469094877509280860 out of >>>> local range [255426072,256718616)] >>>> >>> >>> OK, now this (4469094877509280860) is clearly garbage. THat is the >>> important thing. I have to think your MPI is buggy. >>> >>> >>> -------------- next part -------------- An HTML attachment was scrubbed... URL: From chris at resfrac.com Mon Jul 20 18:00:00 2020 From: chris at resfrac.com (Chris Hewson) Date: Mon, 20 Jul 2020 17:00:00 -0600 Subject: [petsc-users] Tough to reproduce petsctablefind error In-Reply-To: References: Message-ID: Do not use mpich v3.3a2, which is an alpha version released in 2016. Use current stable release mpich-3.3.2 - Thanks Junchao, that makes sense also with Fande's observations. I will give this a try and see *Chris Hewson* Senior Reservoir Simulation Engineer ResFrac +1.587.575.9792 On Mon, Jul 20, 2020 at 4:20 PM Fande Kong wrote: > > > On Mon, Jul 20, 2020 at 1:14 PM Mark Adams wrote: > >> This is indeed a nasty bug, but having two separate should be useful. >> >> Chris is using Haswell, what MPI are you using? I trust you are not using >> Moose. >> >> Fande what machine/MPI are you using? >> > > #define PETSC_MPICC_SHOW > "/apps/local/spack/software/gcc-4.8.5/gcc-9.2.0-bxc7mvbmrfcrusa6ij7ux3exfqabmq5y/bin/gcc > -I/apps/local/mvapich2/2.3.3-gcc-9.2.0/include > -L/apps/local/mvapich2/2.3.3-gcc-9.2.0/lib -Wl,-rpath > -Wl,/apps/local/mvapich2/2.3.3-gcc-9.2.0/lib -Wl,--enable-new-dtags -lmpi" > > I guess it is mvapich2-2.3.3. > > Here is the machine configuration https://www.top500.org/system/179708/ > > > BTW (if you missed my earlier posts), if I switch to MPT-MPI (a vendor > installed MPI), everything runs well so far. > > I will stick with MPT from now. > > Thanks, > > Fande, > > > >> >> On Mon, Jul 20, 2020 at 3:04 PM Chris Hewson wrote: >> >>> Hi Mark, >>> >>> Chris: It sounds like you just have one matrix that you give to MUMPS. >>> You seem to be creating a matrix in the middle of your run. Are you doing >>> dynamic adaptivity? >>> - I have 2 separate matrices I give to mumps, but as this is happening >>> in the production build of my code, I can't determine with certainty what >>> call to MUMPS it's happening or what call to KSPBCGS or UMFPACK it's >>> happening in. >>> >>> I do destroy and recreate matrices in the middle of my runs, but this >>> happens multiple times before the fault happens and in (presumably) the >>> same way. I also do checks on matrix sizes and what I am sending to PETSc >>> and those all pass, just at some point there are size mismatches >>> somewhere, understandably this is not a lot to go on. I am not doing >>> dynamic adaptivity, the mesh is instead changing its size. >>> >>> And I agree with Fande, the most frustrating part is that it's not >>> reproducible, but yah not 100% sure that the problem lies within the PETSc >>> code base either. >>> >>> Current working theories are: >>> 1. Some sort of MPI problem with the sending of one the matrix elements >>> (using mpich version 3.3a2) >>> 2. Some of the memory of static pointers gets corrupted, although I >>> would expect a garbage number and not something that could possibly make >>> sense. >>> >>> *Chris Hewson* >>> Senior Reservoir Simulation Engineer >>> ResFrac >>> +1.587.575.9792 >>> >>> >>> On Mon, Jul 20, 2020 at 12:41 PM Mark Adams wrote: >>> >>>> >>>> >>>> On Mon, Jul 20, 2020 at 2:36 PM Fande Kong wrote: >>>> >>>>> Hi Mark, >>>>> >>>>> Just to be clear, I do not think it is related to GAMG or PtAP. It is >>>>> a communication issue: >>>>> >>>> >>>> Youe stack trace was from PtAP, but Chris's problem is not. >>>> >>>> >>>>> >>>>> Reran the same code, and I just got : >>>>> >>>>> [252]PETSC ERROR: --------------------- Error Message >>>>> -------------------------------------------------------------- >>>>> [252]PETSC ERROR: Petsc has generated inconsistent data >>>>> [252]PETSC ERROR: Received vector entry 4469094877509280860 out of >>>>> local range [255426072,256718616)] >>>>> >>>> >>>> OK, now this (4469094877509280860) is clearly garbage. THat is the >>>> important thing. I have to think your MPI is buggy. >>>> >>>> >>>> -------------- next part -------------- An HTML attachment was scrubbed... URL: From eda.oktay at metu.edu.tr Tue Jul 21 04:49:08 2020 From: eda.oktay at metu.edu.tr (Eda Oktay) Date: Tue, 21 Jul 2020 12:49:08 +0300 Subject: [petsc-users] Is PETSc using internet? Message-ID: Hi all, I am using the following libraries and for some reason, I figured out that if ? am disconnected to internet, my program is not working: #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include I thought that there is no reason for PETSc to go online for libraries but I couldn't find any other reason. The first thing I am doing in my program after initializing SLEPc and PETSc is reading the binary matrix from my file and when I am offline, the program can't even get the matrix. Thanks, Eda From wencel at gmail.com Tue Jul 21 04:58:10 2020 From: wencel at gmail.com (Lawrence Mitchell) Date: Tue, 21 Jul 2020 10:58:10 +0100 Subject: [petsc-users] Is PETSc using internet? In-Reply-To: References: Message-ID: <091AE85F-3B71-4D03-A63B-9FEE610668DF@gmail.com> > On 21 Jul 2020, at 10:49, Eda Oktay wrote: > > Hi all, > > I am using the following libraries and for some reason, I figured out > that if ? am disconnected to internet, my program is not working: [...] > I thought that there is no reason for PETSc to go online for libraries > but I couldn't find any other reason. > > The first thing I am doing in my program after initializing SLEPc and > PETSc is reading the binary matrix from my file and when I am offline, > the program can't even get the matrix. My crystal ball (which sometimes is wrong) guess that you're using a Mac, and MPICH as your MPI version? If so, it may be a problem in MPI_Init failing to resolve localhost to an IP address. If these guesses are correct, does it help to add 127.0.0.1 LOCALHOSTNAME.local 127.0.0.1 LOCALHOSTNAME to /etc/hosts where LOCALHOSTNAME is the name of the machine (as reported by hostname) Lawrence From eda.oktay at metu.edu.tr Tue Jul 21 05:06:36 2020 From: eda.oktay at metu.edu.tr (Eda Oktay) Date: Tue, 21 Jul 2020 13:06:36 +0300 Subject: [petsc-users] Is PETSc using internet? In-Reply-To: <091AE85F-3B71-4D03-A63B-9FEE610668DF@gmail.com> References: <091AE85F-3B71-4D03-A63B-9FEE610668DF@gmail.com> Message-ID: Dear Lawrence, I am using MPICC but not Mac, Fedora 25. If it will still work, I will try that. Thanks! Eda Lawrence Mitchell , 21 Tem 2020 Sal, 12:58 tarihinde ?unu yazd?: > > > > > On 21 Jul 2020, at 10:49, Eda Oktay wrote: > > > > Hi all, > > > > I am using the following libraries and for some reason, I figured out > > that if ? am disconnected to internet, my program is not working: > > [...] > > > I thought that there is no reason for PETSc to go online for libraries > > but I couldn't find any other reason. > > > > The first thing I am doing in my program after initializing SLEPc and > > PETSc is reading the binary matrix from my file and when I am offline, > > the program can't even get the matrix. > > My crystal ball (which sometimes is wrong) guess that you're using a Mac, and MPICH as your MPI version? If so, it may be a problem in MPI_Init failing to resolve localhost to an IP address. > > If these guesses are correct, does it help to add > > 127.0.0.1 LOCALHOSTNAME.local > 127.0.0.1 LOCALHOSTNAME > > to /etc/hosts > > where LOCALHOSTNAME is the name of the machine (as reported by hostname) > > Lawrence From thibault.bridelbertomeu at gmail.com Tue Jul 21 05:10:54 2020 From: thibault.bridelbertomeu at gmail.com (Thibault Bridel-Bertomeu) Date: Tue, 21 Jul 2020 12:10:54 +0200 Subject: [petsc-users] PETSC FVM view gradient Message-ID: Dear all, This request stems from the "How to add a source term for PETSCFV ?" thread. Basically, I would like to output in a VTK file all the components of the gradient computed using the following piece of code (given sol is a global representation of the solution over the whole mesh described by the manager dm with a unique field fvm). ierr = PetscFVSetComputeGradients(fvm,PETSC_TRUE);CHKERRQ(ierr); ierr = DMConvert(dm, DMPLEX, &plex);CHKERRQ(ierr); ierr = DMPlexGetDataFVM(plex, fvm, &cellGeom, &faceGeom, &gradDM);CHKERRQ(ierr); ierr = DMCreateLocalVector(plex,&locX);CHKERRQ(ierr); ierr = DMPlexInsertBoundaryValues(plex, PETSC_TRUE, locX, 0.0, faceGeom, cellGeom, NULL);CHKERRQ(ierr); ierr = DMGlobalToLocalBegin(plex, sol, INSERT_VALUES, locX);CHKERRQ(ierr); ierr = DMGlobalToLocalEnd (plex, sol, INSERT_VALUES, locX);CHKERRQ(ierr); ierr = DMCreateGlobalVector(gradDM, &grad);CHKERRQ(ierr); ierr = DMPlexReconstructGradientsFVM(plex, locX, grad);CHKERRQ(ierr); ierr = DMCreateLocalVector(gradDM, &locGrad);CHKERRQ(ierr); ierr = DMGlobalToLocalBegin(gradDM, grad, INSERT_VALUES, locGrad);CHKERRQ(ierr); ierr = DMGlobalToLocalEnd(gradDM, grad, INSERT_VALUES, locGrad);CHKERRQ(ierr); ierr = VecDestroy(&grad);CHKERRQ(ierr); ierr = PetscViewerCreate(PetscObjectComm((PetscObject)gradDM), &viewer);CHKERRQ(ierr); ierr = PetscSNPrintf(filename,sizeof filename,"%s-%03D-gradient.vtu",user->outputBasename,stepnum);CHKERRQ(ierr); ierr = PetscViewerSetType(viewer, PETSCVIEWERVTK);CHKERRQ(ierr); ierr = PetscViewerFileSetName(viewer, filename);CHKERRQ(ierr); ierr = VecView(locGrad,viewer);CHKERRQ(ierr); I am having troubles with a few things I think but I cannot figure out how to put it together in the right order ... (i) I am not sure whether VecView should be called with a local or a global vector (ii) I can see from the error log that gradDM does not have fields and thus when it goes through dmplexvtu.c it crashes and cannot write the file. I tried to add the fvm as a field, but it does not work either, it crashes on a memory corruption. Could somebody please advise ? Thank you very much in advance, Thibault Bridel-Bertomeu ? Eng, MSc, PhD Research Engineer CEA/CESTA 33114 LE BARP Tel.: (+33)557046924 Mob.: (+33)611025322 Mail: thibault.bridelbertomeu at gmail.com >> >> -------------- next part -------------- An HTML attachment was scrubbed... URL: From wencel at gmail.com Tue Jul 21 05:16:11 2020 From: wencel at gmail.com (Lawrence Mitchell) Date: Tue, 21 Jul 2020 11:16:11 +0100 Subject: [petsc-users] Is PETSc using internet? In-Reply-To: References: <091AE85F-3B71-4D03-A63B-9FEE610668DF@gmail.com> Message-ID: <9442785D-77DA-4AC7-B6BA-DEDE52D984F8@gmail.com> > On 21 Jul 2020, at 11:06, Eda Oktay wrote: > > Dear Lawrence, > > I am using MPICC but not Mac, Fedora 25. If it will still work, I will try that. > > Thanks! It might be the case. When you observe the error, does "nslookup localhost" take a long time? Lawrence From knepley at gmail.com Tue Jul 21 05:53:18 2020 From: knepley at gmail.com (Matthew Knepley) Date: Tue, 21 Jul 2020 06:53:18 -0400 Subject: [petsc-users] PETSC FVM view gradient In-Reply-To: References: Message-ID: On Tue, Jul 21, 2020 at 6:12 AM Thibault Bridel-Bertomeu < thibault.bridelbertomeu at gmail.com> wrote: > Dear all, > > This request stems from the "How to add a source term for PETSCFV ?" > thread. > Basically, I would like to output in a VTK file all the components of the > gradient computed using the following piece of code (given sol is a global > representation of the solution over the whole mesh described by the manager > dm with a unique field fvm). > > ierr = PetscFVSetComputeGradients(fvm,PETSC_TRUE);CHKERRQ(ierr); ierr = DMConvert(dm, DMPLEX, &plex);CHKERRQ(ierr); ierr = DMPlexGetDataFVM(plex, fvm, &cellGeom, &faceGeom, &gradDM);CHKERRQ(ierr); ierr = DMCreateLocalVector(plex,&locX);CHKERRQ(ierr); ierr = DMPlexInsertBoundaryValues(plex, PETSC_TRUE, locX, 0.0, faceGeom, cellGeom, NULL);CHKERRQ(ierr); ierr = DMGlobalToLocalBegin(plex, sol, INSERT_VALUES, locX);CHKERRQ(ierr); ierr = DMGlobalToLocalEnd (plex, sol, INSERT_VALUES, locX);CHKERRQ(ierr); ierr = DMCreateGlobalVector(gradDM, &grad);CHKERRQ(ierr); ierr = DMPlexReconstructGradientsFVM(plex, locX, grad);CHKERRQ(ierr); ierr = DMCreateLocalVector(gradDM, &locGrad);CHKERRQ(ierr); ierr = DMGlobalToLocalBegin(gradDM, grad, INSERT_VALUES, locGrad);CHKERRQ(ierr); ierr = DMGlobalToLocalEnd(gradDM, grad, INSERT_VALUES, locGrad);CHKERRQ(ierr); ierr = VecDestroy(&grad);CHKERRQ(ierr); > > ierr = PetscViewerCreate(PetscObjectComm((PetscObject)gradDM), &viewer);CHKERRQ(ierr); > > ierr = PetscSNPrintf(filename,sizeof filename,"%s-%03D-gradient.vtu",user->outputBasename,stepnum);CHKERRQ(ierr); > > ierr = PetscViewerSetType(viewer, PETSCVIEWERVTK);CHKERRQ(ierr); ierr = PetscViewerFileSetName(viewer, filename);CHKERRQ(ierr); > > ierr = VecView(locGrad,viewer);CHKERRQ(ierr); > > > I am having troubles with a few things I think but I cannot figure out how > to put it together in the right order ... > (i) I am not sure whether VecView should be called with a local or a > global vector > Does not matter, so you can just use the global vector. > (ii) I can see from the error log that gradDM does not have fields and > thus when it goes through dmplexvtu.c it crashes and cannot write the file. > I tried to add the fvm as a field, but it does not work either, it crashes > on a memory corruption. > Can you show the error? The above looks correct to me. Thanks, Matt > Could somebody please advise ? > > Thank you very much in advance, > > Thibault Bridel-Bertomeu > ? > Eng, MSc, PhD > Research Engineer > CEA/CESTA > 33114 LE BARP > Tel.: (+33)557046924 > Mob.: (+33)611025322 > Mail: thibault.bridelbertomeu at gmail.com > > >>> >>> -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From pierpaolo.minelli at cnr.it Tue Jul 21 06:32:13 2020 From: pierpaolo.minelli at cnr.it (Pierpaolo Minelli) Date: Tue, 21 Jul 2020 13:32:13 +0200 Subject: [petsc-users] Error on INTEGER SIZE using DMDACreate3d In-Reply-To: References: <88A21A58-CDCD-42C0-9A20-1A6C5CBCDF8B@cnr.it> Message-ID: Hi, I have asked to compile a Petsc Version updated and with 64bit indices. Now I have Version 3.13.3 and these are the configure options used: #!/bin/python if __name__ == '__main__': import sys import os sys.path.insert(0, os.path.abspath('config')) import configure configure_options = [ '--CC=mpiicc', '--CXX=mpiicpc', '--download-hypre', '--download-metis', '--download-mumps=yes', '--download-parmetis', '--download-scalapack', '--download-superlu_dist', '--known-64-bit-blas-indices', '--prefix=/cineca/prod/opt/libraries/petsc/3.13.3_int64/intelmpi--2018--binary', '--with-64-bit-indices=1', '--with-blaslapack-dir=/cineca/prod/opt/compilers/intel/pe-xe-2018/binary/mkl', '--with-cmake-dir=/cineca/prod/opt/tools/cmake/3.12.0/none', '--with-debugging=0', '--with-fortran-interfaces=1', '--with-fortran=1', 'FC=mpiifort', 'PETSC_ARCH=arch-linux2-c-opt', ] configure.petsc_configure(configure_options) Now, I receive an error on hypre: forrtl: error (78): process killed (SIGTERM) Image PC Routine Line Source libHYPRE-2.18.2.s 00002B33CF465D3F for__signal_handl Unknown Unknown libpthread-2.17.s 00002B33D5BFD370 Unknown Unknown Unknown libpthread-2.17.s 00002B33D5BF96D3 pthread_cond_wait Unknown Unknown libiomp5.so 00002B33DBA14E07 Unknown Unknown Unknown libiomp5.so 00002B33DB98810C Unknown Unknown Unknown libiomp5.so 00002B33DB990578 Unknown Unknown Unknown libiomp5.so 00002B33DB9D9659 Unknown Unknown Unknown libiomp5.so 00002B33DB9D8C39 Unknown Unknown Unknown libiomp5.so 00002B33DB993BCE __kmpc_fork_call Unknown Unknown PIC_3D 00000000004071C0 Unknown Unknown Unknown PIC_3D 0000000000490299 Unknown Unknown Unknown PIC_3D 0000000000492C17 Unknown Unknown Unknown PIC_3D 000000000040562E Unknown Unknown Unknown libc-2.17.so 00002B33DC5BEB35 __libc_start_main Unknown Unknown PIC_3D 0000000000405539 Unknown Unknown Unknown Is it possible that I need to ask also to compile hypre with an option for 64bit indices? Is it possible to instruct this inside Petsc configure? Alternatively, is it possible to use a different multigrid PC inside PETSc that accept 64bit indices? Thanks in advance Pierpaolo > Il giorno 27 mag 2020, alle ore 11:26, Stefano Zampini ha scritto: > > You need a version of PETSc compiled with 64bit indices, since the message indicates the number of dofs in this case is larger the INT_MAX > 2501?3401?1601 = 13617947501 > > I also suggest you upgrade to a newer version, 3.8.3 is quite old as the error message reports > > Il giorno mer 27 mag 2020 alle ore 11:50 Pierpaolo Minelli > ha scritto: > Hi, > > I am trying to solve a Poisson equation on this grid: > > Nx = 2501 > Ny = 3401 > Nz = 1601 > > I received this error: > > [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- > [0]PETSC ERROR: Overflow in integer operation: http://www.mcs.anl.gov/petsc/documentation/faq.html#64-bit-indices > [0]PETSC ERROR: Mesh of 2501 by 3401 by 1 (dof) is too large for 32 bit indices > [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. > [0]PETSC ERROR: Petsc Release Version 3.8.3, Dec, 09, 2017 > [0]PETSC ERROR: /marconi_scratch/userexternal/pminelli/PIC3D/2500_3400_1600/./PIC_3D on a arch-linux2-c-opt named r129c09s02 by pminelli Tu > e May 26 20:16:34 2020 > [0]PETSC ERROR: Configure options --prefix=/cineca/prod/opt/libraries/petsc/3.8.3/intelmpi--2018--binary CC=mpiicc FC=mpiifort CXX=mpiicpc > F77=mpiifort F90=mpiifort --with-debugging=0 --with-blaslapack-dir=/cineca/prod/opt/compilers/intel/pe-xe-2018/binary/mkl --with-fortran=1 > --with-fortran-interfaces=1 --with-cmake-dir=/cineca/prod/opt/tools/cmake/3.5.2/none --with-mpi-dir=/cineca/prod/opt/compilers/intel/pe-xe- > 2018/binary/impi/2018.4.274 --download-scalapack --download-mumps=yes --download-hypre --download-superlu_dist --download-parmetis --downlo > ad-metis > [0]PETSC ERROR: #1 DMSetUp_DA_3D() line 218 in /marconi/prod/build/libraries/petsc/3.8.3/intelmpi--2018--binary/BA_WORK/petsc-3.8.3/src/dm/ > impls/da/da3.c > [0]PETSC ERROR: #2 DMSetUp_DA() line 25 in /marconi/prod/build/libraries/petsc/3.8.3/intelmpi--2018--binary/BA_WORK/petsc-3.8.3/src/dm/impl > s/da/dareg.c > [0]PETSC ERROR: #3 DMSetUp() line 720 in /marconi/prod/build/libraries/petsc/3.8.3/intelmpi--2018--binary/BA_WORK/petsc-3.8.3/src/dm/interf > ace/dm.c > forrtl: error (76): Abort trap signal > > > I am on an HPC facility and after I loaded PETSC module, I have seen that it is configured with INTEGER size = 32 > > I solve my problem with these options and it works perfectly with smaller grids: > > -dm_mat_type hypre -pc_type hypre -pc_hypre_type boomeramg -pc_hypre_boomeramg_relax_type_all SOR/Jacobi -pc_hypre_boomeramg_coarsen_type PMIS -pc_hypre_boomeramg_interp_type FF1 -ksp_type richardson > > Is it possible to overcome this if I ask them to install a version with INTEGER SIZE = 64? > Alternatively, is it possible to overcome this using intel compiler options? > > Thanks in advance > > Pierpaolo Minelli > > > -- > Stefano -------------- next part -------------- An HTML attachment was scrubbed... URL: From thibault.bridelbertomeu at gmail.com Tue Jul 21 06:35:36 2020 From: thibault.bridelbertomeu at gmail.com (Thibault Bridel-Bertomeu) Date: Tue, 21 Jul 2020 13:35:36 +0200 Subject: [petsc-users] PETSC FVM view gradient In-Reply-To: References: Message-ID: Thank you Matthew for your answer. I cannot provide an exact listing of the error as the computation is ran on a closed network but in substance it goes like this : Argument out of range Field number 0 must be in [0, 0) . . . #1 DMGetField() line 4657 in dm.c #2 DMPlexVTKWriteAll_VTU() line 273 in plexvtu.c #3 DMPlexVTKWriteAll() . . . . . . The HEAD of my repo is at commit # g2115eb0. I checked the sources in PETSc and the case where a DM has no field seems to be handled so I do not understand here what it does not like ..? Thanks !!! Thibault Le mar. 21 juil. 2020 ? 12:53, Matthew Knepley a ?crit : > On Tue, Jul 21, 2020 at 6:12 AM Thibault Bridel-Bertomeu < > thibault.bridelbertomeu at gmail.com> wrote: > >> Dear all, >> >> This request stems from the "How to add a source term for PETSCFV ?" >> thread. >> Basically, I would like to output in a VTK file all the components of the >> gradient computed using the following piece of code (given sol is a global >> representation of the solution over the whole mesh described by the manager >> dm with a unique field fvm). >> >> ierr = PetscFVSetComputeGradients(fvm,PETSC_TRUE);CHKERRQ(ierr); ierr = DMConvert(dm, DMPLEX, &plex);CHKERRQ(ierr); ierr = DMPlexGetDataFVM(plex, fvm, &cellGeom, &faceGeom, &gradDM);CHKERRQ(ierr); ierr = DMCreateLocalVector(plex,&locX);CHKERRQ(ierr); ierr = DMPlexInsertBoundaryValues(plex, PETSC_TRUE, locX, 0.0, faceGeom, cellGeom, NULL);CHKERRQ(ierr); ierr = DMGlobalToLocalBegin(plex, sol, INSERT_VALUES, locX);CHKERRQ(ierr); ierr = DMGlobalToLocalEnd (plex, sol, INSERT_VALUES, locX);CHKERRQ(ierr); ierr = DMCreateGlobalVector(gradDM, &grad);CHKERRQ(ierr); ierr = DMPlexReconstructGradientsFVM(plex, locX, grad);CHKERRQ(ierr); ierr = DMCreateLocalVector(gradDM, &locGrad);CHKERRQ(ierr); ierr = DMGlobalToLocalBegin(gradDM, grad, INSERT_VALUES, locGrad);CHKERRQ(ierr); ierr = DMGlobalToLocalEnd(gradDM, grad, INSERT_VALUES, locGrad);CHKERRQ(ierr); ierr = VecDestroy(&grad);CHKERRQ(ierr); >> >> ierr = PetscViewerCreate(PetscObjectComm((PetscObject)gradDM), &viewer);CHKERRQ(ierr); >> >> ierr = PetscSNPrintf(filename,sizeof filename,"%s-%03D-gradient.vtu",user->outputBasename,stepnum);CHKERRQ(ierr); >> >> ierr = PetscViewerSetType(viewer, PETSCVIEWERVTK);CHKERRQ(ierr); ierr = PetscViewerFileSetName(viewer, filename);CHKERRQ(ierr); >> >> ierr = VecView(locGrad,viewer);CHKERRQ(ierr); >> >> >> I am having troubles with a few things I think but I cannot figure out >> how to put it together in the right order ... >> (i) I am not sure whether VecView should be called with a local or a >> global vector >> > > Does not matter, so you can just use the global vector. > > >> (ii) I can see from the error log that gradDM does not have fields and >> thus when it goes through dmplexvtu.c it crashes and cannot write the file. >> I tried to add the fvm as a field, but it does not work either, it crashes >> on a memory corruption. >> > > Can you show the error? The above looks correct to me. > > Thanks, > > Matt > > >> Could somebody please advise ? >> >> Thank you very much in advance, >> >> Thibault Bridel-Bertomeu >> ? >> Eng, MSc, PhD >> Research Engineer >> CEA/CESTA >> 33114 LE BARP >> Tel.: (+33)557046924 >> Mob.: (+33)611025322 >> Mail: thibault.bridelbertomeu at gmail.com >> >> >>>> >>>> > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From dave.mayhem23 at gmail.com Tue Jul 21 06:36:58 2020 From: dave.mayhem23 at gmail.com (Dave May) Date: Tue, 21 Jul 2020 12:36:58 +0100 Subject: [petsc-users] Error on INTEGER SIZE using DMDACreate3d In-Reply-To: References: <88A21A58-CDCD-42C0-9A20-1A6C5CBCDF8B@cnr.it> Message-ID: On Tue, 21 Jul 2020 at 12:32, Pierpaolo Minelli wrote: > Hi, > > I have asked to compile a Petsc Version updated and with 64bit indices. > Now I have Version 3.13.3 and these are the configure options used: > > #!/bin/python > if __name__ == '__main__': > import sys > import os > sys.path.insert(0, os.path.abspath('config')) > import configure > configure_options = [ > '--CC=mpiicc', > '--CXX=mpiicpc', > '--download-hypre', > '--download-metis', > '--download-mumps=yes', > '--download-parmetis', > '--download-scalapack', > '--download-superlu_dist', > '--known-64-bit-blas-indices', > > '--prefix=/cineca/prod/opt/libraries/petsc/3.13.3_int64/intelmpi--2018--binary', > '--with-64-bit-indices=1', > > '--with-blaslapack-dir=/cineca/prod/opt/compilers/intel/pe-xe-2018/binary/mkl', > '--with-cmake-dir=/cineca/prod/opt/tools/cmake/3.12.0/none', > '--with-debugging=0', > '--with-fortran-interfaces=1', > '--with-fortran=1', > 'FC=mpiifort', > 'PETSC_ARCH=arch-linux2-c-opt', > ] > configure.petsc_configure(configure_options) > > Now, I receive an error on hypre: > > forrtl: error (78): process killed (SIGTERM) > Image PC Routine Line > Source > libHYPRE-2.18.2.s 00002B33CF465D3F for__signal_handl Unknown Unknown > libpthread-2.17.s 00002B33D5BFD370 Unknown Unknown Unknown > libpthread-2.17.s 00002B33D5BF96D3 pthread_cond_wait Unknown Unknown > libiomp5.so 00002B33DBA14E07 Unknown Unknown Unknown > libiomp5.so 00002B33DB98810C Unknown Unknown Unknown > libiomp5.so 00002B33DB990578 Unknown Unknown Unknown > libiomp5.so 00002B33DB9D9659 Unknown Unknown Unknown > libiomp5.so 00002B33DB9D8C39 Unknown Unknown Unknown > libiomp5.so 00002B33DB993BCE __kmpc_fork_call Unknown Unknown > PIC_3D 00000000004071C0 Unknown Unknown Unknown > PIC_3D 0000000000490299 Unknown Unknown Unknown > PIC_3D 0000000000492C17 Unknown Unknown Unknown > PIC_3D 000000000040562E Unknown Unknown Unknown > libc-2.17.so 00002B33DC5BEB35 __libc_start_main > Unknown Unknown > PIC_3D 0000000000405539 Unknown Unknown Unknown > > Is it possible that I need to ask also to compile hypre with an option for > 64bit indices? > Is it possible to instruct this inside Petsc configure? > Alternatively, is it possible to use a different multigrid PC inside PETSc > that accept 64bit indices? > You can use -pc_type gamg All native PETSc implementations support 64bit indices. > > Thanks in advance > > Pierpaolo > > > Il giorno 27 mag 2020, alle ore 11:26, Stefano Zampini < > stefano.zampini at gmail.com> ha scritto: > > You need a version of PETSc compiled with 64bit indices, since the message > indicates the number of dofs in this case is larger the INT_MAX > 2501?3401?1601 = 13617947501 > > I also suggest you upgrade to a newer version, 3.8.3 is quite old as the > error message reports > > Il giorno mer 27 mag 2020 alle ore 11:50 Pierpaolo Minelli < > pierpaolo.minelli at cnr.it> ha scritto: > >> Hi, >> >> I am trying to solve a Poisson equation on this grid: >> >> Nx = 2501 >> Ny = 3401 >> Nz = 1601 >> >> I received this error: >> >> [0]PETSC ERROR: --------------------- Error Message >> -------------------------------------------------------------- >> [0]PETSC ERROR: Overflow in integer operation: >> http://www.mcs.anl.gov/petsc/documentation/faq.html#64-bit-indices >> [0]PETSC ERROR: Mesh of 2501 by 3401 by 1 (dof) is too large for 32 bit >> indices >> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html >> for trouble shooting. >> [0]PETSC ERROR: Petsc Release Version 3.8.3, Dec, 09, 2017 >> [0]PETSC ERROR: >> /marconi_scratch/userexternal/pminelli/PIC3D/2500_3400_1600/./PIC_3D on a >> arch-linux2-c-opt named r129c09s02 by pminelli Tu >> e May 26 20:16:34 2020 >> [0]PETSC ERROR: Configure options >> --prefix=/cineca/prod/opt/libraries/petsc/3.8.3/intelmpi--2018--binary >> CC=mpiicc FC=mpiifort CXX=mpiicpc >> F77=mpiifort F90=mpiifort --with-debugging=0 >> --with-blaslapack-dir=/cineca/prod/opt/compilers/intel/pe-xe-2018/binary/mkl >> --with-fortran=1 >> --with-fortran-interfaces=1 >> --with-cmake-dir=/cineca/prod/opt/tools/cmake/3.5.2/none >> --with-mpi-dir=/cineca/prod/opt/compilers/intel/pe-xe- >> 2018/binary/impi/2018.4.274 --download-scalapack --download-mumps=yes >> --download-hypre --download-superlu_dist --download-parmetis --downlo >> ad-metis >> [0]PETSC ERROR: #1 DMSetUp_DA_3D() line 218 in >> /marconi/prod/build/libraries/petsc/3.8.3/intelmpi--2018--binary/BA_WORK/petsc-3.8.3/src/dm/ >> impls/da/da3.c >> [0]PETSC ERROR: #2 DMSetUp_DA() line 25 in >> /marconi/prod/build/libraries/petsc/3.8.3/intelmpi--2018--binary/BA_WORK/petsc-3.8.3/src/dm/impl >> s/da/dareg.c >> [0]PETSC ERROR: #3 DMSetUp() line 720 in >> /marconi/prod/build/libraries/petsc/3.8.3/intelmpi--2018--binary/BA_WORK/petsc-3.8.3/src/dm/interf >> ace/dm.c >> forrtl: error (76): Abort trap signal >> >> >> I am on an HPC facility and after I loaded PETSC module, I have seen that >> it is configured with INTEGER size = 32 >> >> I solve my problem with these options and it works perfectly with smaller >> grids: >> >> -dm_mat_type hypre -pc_type hypre -pc_hypre_type boomeramg >> -pc_hypre_boomeramg_relax_type_all SOR/Jacobi >> -pc_hypre_boomeramg_coarsen_type PMIS -pc_hypre_boomeramg_interp_type FF1 >> -ksp_type richardson >> >> Is it possible to overcome this if I ask them to install a version with >> INTEGER SIZE = 64? >> Alternatively, is it possible to overcome this using intel compiler >> options? >> >> Thanks in advance >> >> Pierpaolo Minelli > > > > -- > Stefano > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From pierpaolo.minelli at cnr.it Tue Jul 21 08:35:31 2020 From: pierpaolo.minelli at cnr.it (Pierpaolo Minelli) Date: Tue, 21 Jul 2020 15:35:31 +0200 Subject: [petsc-users] Error on INTEGER SIZE using DMDACreate3d In-Reply-To: References: <88A21A58-CDCD-42C0-9A20-1A6C5CBCDF8B@cnr.it> Message-ID: Thanks for your reply. As I wrote before, I use these settings: -dm_mat_type hypre -pc_type hypre -pc_hypre_type boomeramg -pc_hypre_boomeramg_relax_type_all SOR/Jacobi -pc_hypre_boomeramg_coarsen_type PMIS -pc_hypre_boomeramg_interp_type FF1 -ksp_type richardson Is there a way to emulate this features also with GAMG? It would be better to use only native Petsc implementations, but these settings, up to single precision indexing for integers, gave me optimal performances. For this reason I asked also, if it was possible to configure hypre (inside Petsc) with 64bit integers. Pierpaolo > Il giorno 21 lug 2020, alle ore 13:36, Dave May ha scritto: > > > > On Tue, 21 Jul 2020 at 12:32, Pierpaolo Minelli > wrote: > Hi, > > I have asked to compile a Petsc Version updated and with 64bit indices. > Now I have Version 3.13.3 and these are the configure options used: > > #!/bin/python > if __name__ == '__main__': > import sys > import os > sys.path.insert(0, os.path.abspath('config')) > import configure > configure_options = [ > '--CC=mpiicc', > '--CXX=mpiicpc', > '--download-hypre', > '--download-metis', > '--download-mumps=yes', > '--download-parmetis', > '--download-scalapack', > '--download-superlu_dist', > '--known-64-bit-blas-indices', > '--prefix=/cineca/prod/opt/libraries/petsc/3.13.3_int64/intelmpi--2018--binary', > '--with-64-bit-indices=1', > '--with-blaslapack-dir=/cineca/prod/opt/compilers/intel/pe-xe-2018/binary/mkl', > '--with-cmake-dir=/cineca/prod/opt/tools/cmake/3.12.0/none', > '--with-debugging=0', > '--with-fortran-interfaces=1', > '--with-fortran=1', > 'FC=mpiifort', > 'PETSC_ARCH=arch-linux2-c-opt', > ] > configure.petsc_configure(configure_options) > > Now, I receive an error on hypre: > > forrtl: error (78): process killed (SIGTERM) > Image PC Routine Line Source > libHYPRE-2.18.2.s 00002B33CF465D3F for__signal_handl Unknown Unknown > libpthread-2.17.s 00002B33D5BFD370 Unknown Unknown Unknown > libpthread-2.17.s 00002B33D5BF96D3 pthread_cond_wait Unknown Unknown > libiomp5.so 00002B33DBA14E07 Unknown Unknown Unknown > libiomp5.so 00002B33DB98810C Unknown Unknown Unknown > libiomp5.so 00002B33DB990578 Unknown Unknown Unknown > libiomp5.so 00002B33DB9D9659 Unknown Unknown Unknown > libiomp5.so 00002B33DB9D8C39 Unknown Unknown Unknown > libiomp5.so 00002B33DB993BCE __kmpc_fork_call Unknown Unknown > PIC_3D 00000000004071C0 Unknown Unknown Unknown > PIC_3D 0000000000490299 Unknown Unknown Unknown > PIC_3D 0000000000492C17 Unknown Unknown Unknown > PIC_3D 000000000040562E Unknown Unknown Unknown > libc-2.17.so 00002B33DC5BEB35 __libc_start_main Unknown Unknown > PIC_3D 0000000000405539 Unknown Unknown Unknown > > Is it possible that I need to ask also to compile hypre with an option for 64bit indices? > Is it possible to instruct this inside Petsc configure? > Alternatively, is it possible to use a different multigrid PC inside PETSc that accept 64bit indices? > > You can use > -pc_type gamg > All native PETSc implementations support 64bit indices. > > > Thanks in advance > > Pierpaolo > > >> Il giorno 27 mag 2020, alle ore 11:26, Stefano Zampini > ha scritto: >> >> You need a version of PETSc compiled with 64bit indices, since the message indicates the number of dofs in this case is larger the INT_MAX >> 2501?3401?1601 = 13617947501 >> >> I also suggest you upgrade to a newer version, 3.8.3 is quite old as the error message reports >> >> Il giorno mer 27 mag 2020 alle ore 11:50 Pierpaolo Minelli > ha scritto: >> Hi, >> >> I am trying to solve a Poisson equation on this grid: >> >> Nx = 2501 >> Ny = 3401 >> Nz = 1601 >> >> I received this error: >> >> [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- >> [0]PETSC ERROR: Overflow in integer operation: http://www.mcs.anl.gov/petsc/documentation/faq.html#64-bit-indices >> [0]PETSC ERROR: Mesh of 2501 by 3401 by 1 (dof) is too large for 32 bit indices >> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. >> [0]PETSC ERROR: Petsc Release Version 3.8.3, Dec, 09, 2017 >> [0]PETSC ERROR: /marconi_scratch/userexternal/pminelli/PIC3D/2500_3400_1600/./PIC_3D on a arch-linux2-c-opt named r129c09s02 by pminelli Tu >> e May 26 20:16:34 2020 >> [0]PETSC ERROR: Configure options --prefix=/cineca/prod/opt/libraries/petsc/3.8.3/intelmpi--2018--binary CC=mpiicc FC=mpiifort CXX=mpiicpc >> F77=mpiifort F90=mpiifort --with-debugging=0 --with-blaslapack-dir=/cineca/prod/opt/compilers/intel/pe-xe-2018/binary/mkl --with-fortran=1 >> --with-fortran-interfaces=1 --with-cmake-dir=/cineca/prod/opt/tools/cmake/3.5.2/none --with-mpi-dir=/cineca/prod/opt/compilers/intel/pe-xe- >> 2018/binary/impi/2018.4.274 --download-scalapack --download-mumps=yes --download-hypre --download-superlu_dist --download-parmetis --downlo >> ad-metis >> [0]PETSC ERROR: #1 DMSetUp_DA_3D() line 218 in /marconi/prod/build/libraries/petsc/3.8.3/intelmpi--2018--binary/BA_WORK/petsc-3.8.3/src/dm/ >> impls/da/da3.c >> [0]PETSC ERROR: #2 DMSetUp_DA() line 25 in /marconi/prod/build/libraries/petsc/3.8.3/intelmpi--2018--binary/BA_WORK/petsc-3.8.3/src/dm/impl >> s/da/dareg.c >> [0]PETSC ERROR: #3 DMSetUp() line 720 in /marconi/prod/build/libraries/petsc/3.8.3/intelmpi--2018--binary/BA_WORK/petsc-3.8.3/src/dm/interf >> ace/dm.c >> forrtl: error (76): Abort trap signal >> >> >> I am on an HPC facility and after I loaded PETSC module, I have seen that it is configured with INTEGER size = 32 >> >> I solve my problem with these options and it works perfectly with smaller grids: >> >> -dm_mat_type hypre -pc_type hypre -pc_hypre_type boomeramg -pc_hypre_boomeramg_relax_type_all SOR/Jacobi -pc_hypre_boomeramg_coarsen_type PMIS -pc_hypre_boomeramg_interp_type FF1 -ksp_type richardson >> >> Is it possible to overcome this if I ask them to install a version with INTEGER SIZE = 64? >> Alternatively, is it possible to overcome this using intel compiler options? >> >> Thanks in advance >> >> Pierpaolo Minelli >> >> >> -- >> Stefano -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Tue Jul 21 08:45:29 2020 From: knepley at gmail.com (Matthew Knepley) Date: Tue, 21 Jul 2020 09:45:29 -0400 Subject: [petsc-users] Error on INTEGER SIZE using DMDACreate3d In-Reply-To: References: <88A21A58-CDCD-42C0-9A20-1A6C5CBCDF8B@cnr.it> Message-ID: On Tue, Jul 21, 2020 at 9:35 AM Pierpaolo Minelli wrote: > Thanks for your reply. > As I wrote before, I use these settings: > > -dm_mat_type hypre -pc_type hypre -pc_hypre_type boomeramg > -pc_hypre_boomeramg_relax_type_all SOR/Jacobi > -pc_hypre_boomeramg_coarsen_type PMIS -pc_hypre_boomeramg_interp_type FF1 > -ksp_type richardson > > Is there a way to emulate this features also with GAMG? > Smoothers: You have complete control here -mg_levels_pc_type sor (the default is Chebyshev which you could also try) Coarsening: This is much different in agglomeration AMG. There is a discussion here: https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/PC/PCGAMGSetThreshold.html https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/PC/PCGAMGSetSquareGraph.html Interpolation: This is built-in for agglomeration AMG. > It would be better to use only native Petsc implementations, but these > settings, up to single precision indexing for integers, gave me optimal > performances. > For this reason I asked also, if it was possible to configure hypre > (inside Petsc) with 64bit integers. > Yes. That happened when you reconfigured for 64 bits. You may have encountered a Hypre bug. Thanks, Matt > Pierpaolo > > > Il giorno 21 lug 2020, alle ore 13:36, Dave May > ha scritto: > > > > On Tue, 21 Jul 2020 at 12:32, Pierpaolo Minelli > wrote: > >> Hi, >> >> I have asked to compile a Petsc Version updated and with 64bit indices. >> Now I have Version 3.13.3 and these are the configure options used: >> >> #!/bin/python >> if __name__ == '__main__': >> import sys >> import os >> sys.path.insert(0, os.path.abspath('config')) >> import configure >> configure_options = [ >> '--CC=mpiicc', >> '--CXX=mpiicpc', >> '--download-hypre', >> '--download-metis', >> '--download-mumps=yes', >> '--download-parmetis', >> '--download-scalapack', >> '--download-superlu_dist', >> '--known-64-bit-blas-indices', >> >> '--prefix=/cineca/prod/opt/libraries/petsc/3.13.3_int64/intelmpi--2018--binary', >> '--with-64-bit-indices=1', >> >> '--with-blaslapack-dir=/cineca/prod/opt/compilers/intel/pe-xe-2018/binary/mkl', >> '--with-cmake-dir=/cineca/prod/opt/tools/cmake/3.12.0/none', >> '--with-debugging=0', >> '--with-fortran-interfaces=1', >> '--with-fortran=1', >> 'FC=mpiifort', >> 'PETSC_ARCH=arch-linux2-c-opt', >> ] >> configure.petsc_configure(configure_options) >> >> Now, I receive an error on hypre: >> >> forrtl: error (78): process killed (SIGTERM) >> Image PC Routine Line >> Source >> libHYPRE-2.18.2.s 00002B33CF465D3F for__signal_handl >> Unknown Unknown >> libpthread-2.17.s 00002B33D5BFD370 Unknown >> Unknown Unknown >> libpthread-2.17.s 00002B33D5BF96D3 pthread_cond_wait >> Unknown Unknown >> libiomp5.so 00002B33DBA14E07 Unknown >> Unknown Unknown >> libiomp5.so 00002B33DB98810C Unknown >> Unknown Unknown >> libiomp5.so 00002B33DB990578 Unknown >> Unknown Unknown >> libiomp5.so 00002B33DB9D9659 Unknown >> Unknown Unknown >> libiomp5.so 00002B33DB9D8C39 Unknown >> Unknown Unknown >> libiomp5.so 00002B33DB993BCE __kmpc_fork_call >> Unknown Unknown >> PIC_3D 00000000004071C0 Unknown >> Unknown Unknown >> PIC_3D 0000000000490299 Unknown >> Unknown Unknown >> PIC_3D 0000000000492C17 Unknown >> Unknown Unknown >> PIC_3D 000000000040562E Unknown >> Unknown Unknown >> libc-2.17.so 00002B33DC5BEB35 __libc_start_main >> Unknown Unknown >> PIC_3D 0000000000405539 Unknown >> Unknown Unknown >> >> Is it possible that I need to ask also to compile hypre with an option >> for 64bit indices? >> Is it possible to instruct this inside Petsc configure? >> Alternatively, is it possible to use a different multigrid PC inside >> PETSc that accept 64bit indices? >> > > You can use > -pc_type gamg > All native PETSc implementations support 64bit indices. > > >> >> Thanks in advance >> >> Pierpaolo >> >> >> Il giorno 27 mag 2020, alle ore 11:26, Stefano Zampini < >> stefano.zampini at gmail.com> ha scritto: >> >> You need a version of PETSc compiled with 64bit indices, since the >> message indicates the number of dofs in this case is larger the INT_MAX >> 2501?3401?1601 = 13617947501 >> >> I also suggest you upgrade to a newer version, 3.8.3 is quite old as the >> error message reports >> >> Il giorno mer 27 mag 2020 alle ore 11:50 Pierpaolo Minelli < >> pierpaolo.minelli at cnr.it> ha scritto: >> >>> Hi, >>> >>> I am trying to solve a Poisson equation on this grid: >>> >>> Nx = 2501 >>> Ny = 3401 >>> Nz = 1601 >>> >>> I received this error: >>> >>> [0]PETSC ERROR: --------------------- Error Message >>> -------------------------------------------------------------- >>> [0]PETSC ERROR: Overflow in integer operation: >>> http://www.mcs.anl.gov/petsc/documentation/faq.html#64-bit-indices >>> [0]PETSC ERROR: Mesh of 2501 by 3401 by 1 (dof) is too large for 32 bit >>> indices >>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for >>> trouble shooting. >>> [0]PETSC ERROR: Petsc Release Version 3.8.3, Dec, 09, 2017 >>> [0]PETSC ERROR: >>> /marconi_scratch/userexternal/pminelli/PIC3D/2500_3400_1600/./PIC_3D on a >>> arch-linux2-c-opt named r129c09s02 by pminelli Tu >>> e May 26 20:16:34 2020 >>> [0]PETSC ERROR: Configure options >>> --prefix=/cineca/prod/opt/libraries/petsc/3.8.3/intelmpi--2018--binary >>> CC=mpiicc FC=mpiifort CXX=mpiicpc >>> F77=mpiifort F90=mpiifort --with-debugging=0 >>> --with-blaslapack-dir=/cineca/prod/opt/compilers/intel/pe-xe-2018/binary/mkl >>> --with-fortran=1 >>> --with-fortran-interfaces=1 >>> --with-cmake-dir=/cineca/prod/opt/tools/cmake/3.5.2/none >>> --with-mpi-dir=/cineca/prod/opt/compilers/intel/pe-xe- >>> 2018/binary/impi/2018.4.274 --download-scalapack --download-mumps=yes >>> --download-hypre --download-superlu_dist --download-parmetis --downlo >>> ad-metis >>> [0]PETSC ERROR: #1 DMSetUp_DA_3D() line 218 in >>> /marconi/prod/build/libraries/petsc/3.8.3/intelmpi--2018--binary/BA_WORK/petsc-3.8.3/src/dm/ >>> impls/da/da3.c >>> [0]PETSC ERROR: #2 DMSetUp_DA() line 25 in >>> /marconi/prod/build/libraries/petsc/3.8.3/intelmpi--2018--binary/BA_WORK/petsc-3.8.3/src/dm/impl >>> s/da/dareg.c >>> [0]PETSC ERROR: #3 DMSetUp() line 720 in >>> /marconi/prod/build/libraries/petsc/3.8.3/intelmpi--2018--binary/BA_WORK/petsc-3.8.3/src/dm/interf >>> ace/dm.c >>> forrtl: error (76): Abort trap signal >>> >>> >>> I am on an HPC facility and after I loaded PETSC module, I have seen >>> that it is configured with INTEGER size = 32 >>> >>> I solve my problem with these options and it works perfectly with >>> smaller grids: >>> >>> -dm_mat_type hypre -pc_type hypre -pc_hypre_type boomeramg >>> -pc_hypre_boomeramg_relax_type_all SOR/Jacobi >>> -pc_hypre_boomeramg_coarsen_type PMIS -pc_hypre_boomeramg_interp_type FF1 >>> -ksp_type richardson >>> >>> Is it possible to overcome this if I ask them to install a version with >>> INTEGER SIZE = 64? >>> Alternatively, is it possible to overcome this using intel compiler >>> options? >>> >>> Thanks in advance >>> >>> Pierpaolo Minelli >> >> >> >> -- >> Stefano >> >> > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From flw at rzg.mpg.de Tue Jul 21 08:58:07 2020 From: flw at rzg.mpg.de (flw at rzg.mpg.de) Date: Tue, 21 Jul 2020 15:58:07 +0200 Subject: [petsc-users] SUPERLU_DIST in single precision Message-ID: <20200721155807.Horde.5ZYOzrQ7dmwNTQquK_6ebeS@webmail.mpcdf.mpg.de> Dear PETSc support team, some time ago you told me that you are planning on releasing a version that supports SUPERLU_DIST in single-precision soon. Can you tell me roughly what time frame you had in mind? Best regards, Felix From pierpaolo.minelli at cnr.it Tue Jul 21 09:26:41 2020 From: pierpaolo.minelli at cnr.it (Pierpaolo Minelli) Date: Tue, 21 Jul 2020 16:26:41 +0200 Subject: [petsc-users] Error on INTEGER SIZE using DMDACreate3d In-Reply-To: References: <88A21A58-CDCD-42C0-9A20-1A6C5CBCDF8B@cnr.it> Message-ID: <58356AA6-A6B6-4110-A36C-D3B143300175@cnr.it> Thanks for your useful suggestions. Pierpaolo > Il giorno 21 lug 2020, alle ore 15:45, Matthew Knepley ha scritto: > > On Tue, Jul 21, 2020 at 9:35 AM Pierpaolo Minelli > wrote: > Thanks for your reply. > As I wrote before, I use these settings: > > -dm_mat_type hypre -pc_type hypre -pc_hypre_type boomeramg -pc_hypre_boomeramg_relax_type_all SOR/Jacobi -pc_hypre_boomeramg_coarsen_type PMIS -pc_hypre_boomeramg_interp_type FF1 -ksp_type richardson > > Is there a way to emulate this features also with GAMG? > > Smoothers: You have complete control here > > -mg_levels_pc_type sor (the default is Chebyshev which you could also try) > > Coarsening: This is much different in agglomeration AMG. There is a discussion here: > > https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/PC/PCGAMGSetThreshold.html > https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/PC/PCGAMGSetSquareGraph.html > > Interpolation: This is built-in for agglomeration AMG. > > It would be better to use only native Petsc implementations, but these settings, up to single precision indexing for integers, gave me optimal performances. > For this reason I asked also, if it was possible to configure hypre (inside Petsc) with 64bit integers. > > Yes. That happened when you reconfigured for 64 bits. You may have encountered a Hypre bug. > > Thanks, > > Matt > > Pierpaolo > > >> Il giorno 21 lug 2020, alle ore 13:36, Dave May > ha scritto: >> >> >> >> On Tue, 21 Jul 2020 at 12:32, Pierpaolo Minelli > wrote: >> Hi, >> >> I have asked to compile a Petsc Version updated and with 64bit indices. >> Now I have Version 3.13.3 and these are the configure options used: >> >> #!/bin/python >> if __name__ == '__main__': >> import sys >> import os >> sys.path.insert(0, os.path.abspath('config')) >> import configure >> configure_options = [ >> '--CC=mpiicc', >> '--CXX=mpiicpc', >> '--download-hypre', >> '--download-metis', >> '--download-mumps=yes', >> '--download-parmetis', >> '--download-scalapack', >> '--download-superlu_dist', >> '--known-64-bit-blas-indices', >> '--prefix=/cineca/prod/opt/libraries/petsc/3.13.3_int64/intelmpi--2018--binary', >> '--with-64-bit-indices=1', >> '--with-blaslapack-dir=/cineca/prod/opt/compilers/intel/pe-xe-2018/binary/mkl', >> '--with-cmake-dir=/cineca/prod/opt/tools/cmake/3.12.0/none', >> '--with-debugging=0', >> '--with-fortran-interfaces=1', >> '--with-fortran=1', >> 'FC=mpiifort', >> 'PETSC_ARCH=arch-linux2-c-opt', >> ] >> configure.petsc_configure(configure_options) >> >> Now, I receive an error on hypre: >> >> forrtl: error (78): process killed (SIGTERM) >> Image PC Routine Line Source >> libHYPRE-2.18.2.s 00002B33CF465D3F for__signal_handl Unknown Unknown >> libpthread-2.17.s 00002B33D5BFD370 Unknown Unknown Unknown >> libpthread-2.17.s 00002B33D5BF96D3 pthread_cond_wait Unknown Unknown >> libiomp5.so 00002B33DBA14E07 Unknown Unknown Unknown >> libiomp5.so 00002B33DB98810C Unknown Unknown Unknown >> libiomp5.so 00002B33DB990578 Unknown Unknown Unknown >> libiomp5.so 00002B33DB9D9659 Unknown Unknown Unknown >> libiomp5.so 00002B33DB9D8C39 Unknown Unknown Unknown >> libiomp5.so 00002B33DB993BCE __kmpc_fork_call Unknown Unknown >> PIC_3D 00000000004071C0 Unknown Unknown Unknown >> PIC_3D 0000000000490299 Unknown Unknown Unknown >> PIC_3D 0000000000492C17 Unknown Unknown Unknown >> PIC_3D 000000000040562E Unknown Unknown Unknown >> libc-2.17.so 00002B33DC5BEB35 __libc_start_main Unknown Unknown >> PIC_3D 0000000000405539 Unknown Unknown Unknown >> >> Is it possible that I need to ask also to compile hypre with an option for 64bit indices? >> Is it possible to instruct this inside Petsc configure? >> Alternatively, is it possible to use a different multigrid PC inside PETSc that accept 64bit indices? >> >> You can use >> -pc_type gamg >> All native PETSc implementations support 64bit indices. >> >> >> Thanks in advance >> >> Pierpaolo >> >> >>> Il giorno 27 mag 2020, alle ore 11:26, Stefano Zampini > ha scritto: >>> >>> You need a version of PETSc compiled with 64bit indices, since the message indicates the number of dofs in this case is larger the INT_MAX >>> 2501?3401?1601 = 13617947501 >>> >>> I also suggest you upgrade to a newer version, 3.8.3 is quite old as the error message reports >>> >>> Il giorno mer 27 mag 2020 alle ore 11:50 Pierpaolo Minelli > ha scritto: >>> Hi, >>> >>> I am trying to solve a Poisson equation on this grid: >>> >>> Nx = 2501 >>> Ny = 3401 >>> Nz = 1601 >>> >>> I received this error: >>> >>> [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- >>> [0]PETSC ERROR: Overflow in integer operation: http://www.mcs.anl.gov/petsc/documentation/faq.html#64-bit-indices >>> [0]PETSC ERROR: Mesh of 2501 by 3401 by 1 (dof) is too large for 32 bit indices >>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. >>> [0]PETSC ERROR: Petsc Release Version 3.8.3, Dec, 09, 2017 >>> [0]PETSC ERROR: /marconi_scratch/userexternal/pminelli/PIC3D/2500_3400_1600/./PIC_3D on a arch-linux2-c-opt named r129c09s02 by pminelli Tu >>> e May 26 20:16:34 2020 >>> [0]PETSC ERROR: Configure options --prefix=/cineca/prod/opt/libraries/petsc/3.8.3/intelmpi--2018--binary CC=mpiicc FC=mpiifort CXX=mpiicpc >>> F77=mpiifort F90=mpiifort --with-debugging=0 --with-blaslapack-dir=/cineca/prod/opt/compilers/intel/pe-xe-2018/binary/mkl --with-fortran=1 >>> --with-fortran-interfaces=1 --with-cmake-dir=/cineca/prod/opt/tools/cmake/3.5.2/none --with-mpi-dir=/cineca/prod/opt/compilers/intel/pe-xe- >>> 2018/binary/impi/2018.4.274 --download-scalapack --download-mumps=yes --download-hypre --download-superlu_dist --download-parmetis --downlo >>> ad-metis >>> [0]PETSC ERROR: #1 DMSetUp_DA_3D() line 218 in /marconi/prod/build/libraries/petsc/3.8.3/intelmpi--2018--binary/BA_WORK/petsc-3.8.3/src/dm/ >>> impls/da/da3.c >>> [0]PETSC ERROR: #2 DMSetUp_DA() line 25 in /marconi/prod/build/libraries/petsc/3.8.3/intelmpi--2018--binary/BA_WORK/petsc-3.8.3/src/dm/impl >>> s/da/dareg.c >>> [0]PETSC ERROR: #3 DMSetUp() line 720 in /marconi/prod/build/libraries/petsc/3.8.3/intelmpi--2018--binary/BA_WORK/petsc-3.8.3/src/dm/interf >>> ace/dm.c >>> forrtl: error (76): Abort trap signal >>> >>> >>> I am on an HPC facility and after I loaded PETSC module, I have seen that it is configured with INTEGER size = 32 >>> >>> I solve my problem with these options and it works perfectly with smaller grids: >>> >>> -dm_mat_type hypre -pc_type hypre -pc_hypre_type boomeramg -pc_hypre_boomeramg_relax_type_all SOR/Jacobi -pc_hypre_boomeramg_coarsen_type PMIS -pc_hypre_boomeramg_interp_type FF1 -ksp_type richardson >>> >>> Is it possible to overcome this if I ask them to install a version with INTEGER SIZE = 64? >>> Alternatively, is it possible to overcome this using intel compiler options? >>> >>> Thanks in advance >>> >>> Pierpaolo Minelli >>> >>> >>> -- >>> Stefano > > > > -- > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From stefano.zampini at gmail.com Tue Jul 21 09:44:44 2020 From: stefano.zampini at gmail.com (Stefano Zampini) Date: Tue, 21 Jul 2020 16:44:44 +0200 Subject: [petsc-users] Error on INTEGER SIZE using DMDACreate3d In-Reply-To: References: <88A21A58-CDCD-42C0-9A20-1A6C5CBCDF8B@cnr.it> Message-ID: > On Jul 21, 2020, at 1:32 PM, Pierpaolo Minelli wrote: > > Hi, > > I have asked to compile a Petsc Version updated and with 64bit indices. > Now I have Version 3.13.3 and these are the configure options used: > > #!/bin/python > if __name__ == '__main__': > import sys > import os > sys.path.insert(0, os.path.abspath('config')) > import configure > configure_options = [ > '--CC=mpiicc', > '--CXX=mpiicpc', > '--download-hypre', > '--download-metis', > '--download-mumps=yes', > '--download-parmetis', > '--download-scalapack', > '--download-superlu_dist', > '--known-64-bit-blas-indices', > '--prefix=/cineca/prod/opt/libraries/petsc/3.13.3_int64/intelmpi--2018--binary', > '--with-64-bit-indices=1', > '--with-blaslapack-dir=/cineca/prod/opt/compilers/intel/pe-xe-2018/binary/mkl', > '--with-cmake-dir=/cineca/prod/opt/tools/cmake/3.12.0/none', > '--with-debugging=0', > '--with-fortran-interfaces=1', > '--with-fortran=1', > 'FC=mpiifort', > 'PETSC_ARCH=arch-linux2-c-opt', > ] > configure.petsc_configure(configure_options) > > Now, I receive an error on hypre: > > forrtl: error (78): process killed (SIGTERM) > Image PC Routine Line Source > libHYPRE-2.18.2.s 00002B33CF465D3F for__signal_handl Unknown Unknown > libpthread-2.17.s 00002B33D5BFD370 Unknown Unknown Unknown > libpthread-2.17.s 00002B33D5BF96D3 pthread_cond_wait Unknown Unknown > libiomp5.so 00002B33DBA14E07 Unknown Unknown Unknown > libiomp5.so 00002B33DB98810C Unknown Unknown Unknown > libiomp5.so 00002B33DB990578 Unknown Unknown Unknown > libiomp5.so 00002B33DB9D9659 Unknown Unknown Unknown > libiomp5.so 00002B33DB9D8C39 Unknown Unknown Unknown > libiomp5.so 00002B33DB993BCE __kmpc_fork_call Unknown Unknown > PIC_3D 00000000004071C0 Unknown Unknown Unknown > PIC_3D 0000000000490299 Unknown Unknown Unknown > PIC_3D 0000000000492C17 Unknown Unknown Unknown > PIC_3D 000000000040562E Unknown Unknown Unknown > libc-2.17.so 00002B33DC5BEB35 __libc_start_main Unknown Unknown > PIC_3D 0000000000405539 Unknown Unknown Unknown > > Is it possible > that I need to ask also to compile hypre with an option for 64bit indices? These configure options compile hypre with 64bit indices support. It should work just fine. Can you run a very small case of your code to confirm? > Is it possible to instruct this inside Petsc configure? > Alternatively, is it possible to use a different multigrid PC inside PETSc that accept 64bit indices? > > Thanks in advance > > Pierpaolo > > >> Il giorno 27 mag 2020, alle ore 11:26, Stefano Zampini > ha scritto: >> >> You need a version of PETSc compiled with 64bit indices, since the message indicates the number of dofs in this case is larger the INT_MAX >> 2501?3401?1601 = 13617947501 >> >> I also suggest you upgrade to a newer version, 3.8.3 is quite old as the error message reports >> >> Il giorno mer 27 mag 2020 alle ore 11:50 Pierpaolo Minelli > ha scritto: >> Hi, >> >> I am trying to solve a Poisson equation on this grid: >> >> Nx = 2501 >> Ny = 3401 >> Nz = 1601 >> >> I received this error: >> >> [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- >> [0]PETSC ERROR: Overflow in integer operation: http://www.mcs.anl.gov/petsc/documentation/faq.html#64-bit-indices >> [0]PETSC ERROR: Mesh of 2501 by 3401 by 1 (dof) is too large for 32 bit indices >> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. >> [0]PETSC ERROR: Petsc Release Version 3.8.3, Dec, 09, 2017 >> [0]PETSC ERROR: /marconi_scratch/userexternal/pminelli/PIC3D/2500_3400_1600/./PIC_3D on a arch-linux2-c-opt named r129c09s02 by pminelli Tu >> e May 26 20:16:34 2020 >> [0]PETSC ERROR: Configure options --prefix=/cineca/prod/opt/libraries/petsc/3.8.3/intelmpi--2018--binary CC=mpiicc FC=mpiifort CXX=mpiicpc >> F77=mpiifort F90=mpiifort --with-debugging=0 --with-blaslapack-dir=/cineca/prod/opt/compilers/intel/pe-xe-2018/binary/mkl --with-fortran=1 >> --with-fortran-interfaces=1 --with-cmake-dir=/cineca/prod/opt/tools/cmake/3.5.2/none --with-mpi-dir=/cineca/prod/opt/compilers/intel/pe-xe- >> 2018/binary/impi/2018.4.274 --download-scalapack --download-mumps=yes --download-hypre --download-superlu_dist --download-parmetis --downlo >> ad-metis >> [0]PETSC ERROR: #1 DMSetUp_DA_3D() line 218 in /marconi/prod/build/libraries/petsc/3.8.3/intelmpi--2018--binary/BA_WORK/petsc-3.8.3/src/dm/ >> impls/da/da3.c >> [0]PETSC ERROR: #2 DMSetUp_DA() line 25 in /marconi/prod/build/libraries/petsc/3.8.3/intelmpi--2018--binary/BA_WORK/petsc-3.8.3/src/dm/impl >> s/da/dareg.c >> [0]PETSC ERROR: #3 DMSetUp() line 720 in /marconi/prod/build/libraries/petsc/3.8.3/intelmpi--2018--binary/BA_WORK/petsc-3.8.3/src/dm/interf >> ace/dm.c >> forrtl: error (76): Abort trap signal >> >> >> I am on an HPC facility and after I loaded PETSC module, I have seen that it is configured with INTEGER size = 32 >> >> I solve my problem with these options and it works perfectly with smaller grids: >> >> -dm_mat_type hypre -pc_type hypre -pc_hypre_type boomeramg -pc_hypre_boomeramg_relax_type_all SOR/Jacobi -pc_hypre_boomeramg_coarsen_type PMIS -pc_hypre_boomeramg_interp_type FF1 -ksp_type richardson >> >> Is it possible to overcome this if I ask them to install a version with INTEGER SIZE = 64? >> Alternatively, is it possible to overcome this using intel compiler options? >> >> Thanks in advance >> >> Pierpaolo Minelli >> >> >> -- >> Stefano > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mfadams at lbl.gov Tue Jul 21 09:56:48 2020 From: mfadams at lbl.gov (Mark Adams) Date: Tue, 21 Jul 2020 10:56:48 -0400 Subject: [petsc-users] Error on INTEGER SIZE using DMDACreate3d In-Reply-To: References: <88A21A58-CDCD-42C0-9A20-1A6C5CBCDF8B@cnr.it> Message-ID: On Tue, Jul 21, 2020 at 9:46 AM Matthew Knepley wrote: > On Tue, Jul 21, 2020 at 9:35 AM Pierpaolo Minelli < > pierpaolo.minelli at cnr.it> wrote: > >> Thanks for your reply. >> As I wrote before, I use these settings: >> >> -dm_mat_type hypre -pc_type hypre -pc_hypre_type boomeramg >> -pc_hypre_boomeramg_relax_type_all SOR/Jacobi >> -pc_hypre_boomeramg_coarsen_type PMIS -pc_hypre_boomeramg_interp_type FF1 >> -ksp_type richardson >> >> Is there a way to emulate this features also with GAMG? >> > > Smoothers: You have complete control here > > -mg_levels_pc_type sor (the default is Chebyshev which you could also > try) > And you set the KSP type. You have -ksp_type richardson above but that is not used for Hypre. It is for GAMG. Chebyshev is a ksp type (-ksp_type chebyshev). Hypre is very good on Poisson. THe grid complexity (cost per iteration) can be high but the convergence rate will be better than GAMG. But, you should be able to get hypre to work. > > Coarsening: This is much different in agglomeration AMG. There is a > discussion here: > > > https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/PC/PCGAMGSetThreshold.html > > https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/PC/PCGAMGSetSquareGraph.html > > Interpolation: This is built-in for agglomeration AMG. > > >> It would be better to use only native Petsc implementations, but these >> settings, up to single precision indexing for integers, gave me optimal >> performances. >> For this reason I asked also, if it was possible to configure hypre >> (inside Petsc) with 64bit integers. >> > > Yes. That happened when you reconfigured for 64 bits. You may have > encountered a Hypre bug. > > Thanks, > > Matt > > >> Pierpaolo >> >> >> Il giorno 21 lug 2020, alle ore 13:36, Dave May >> ha scritto: >> >> >> >> On Tue, 21 Jul 2020 at 12:32, Pierpaolo Minelli >> wrote: >> >>> Hi, >>> >>> I have asked to compile a Petsc Version updated and with 64bit indices. >>> Now I have Version 3.13.3 and these are the configure options used: >>> >>> #!/bin/python >>> if __name__ == '__main__': >>> import sys >>> import os >>> sys.path.insert(0, os.path.abspath('config')) >>> import configure >>> configure_options = [ >>> '--CC=mpiicc', >>> '--CXX=mpiicpc', >>> '--download-hypre', >>> '--download-metis', >>> '--download-mumps=yes', >>> '--download-parmetis', >>> '--download-scalapack', >>> '--download-superlu_dist', >>> '--known-64-bit-blas-indices', >>> >>> '--prefix=/cineca/prod/opt/libraries/petsc/3.13.3_int64/intelmpi--2018--binary', >>> '--with-64-bit-indices=1', >>> >>> '--with-blaslapack-dir=/cineca/prod/opt/compilers/intel/pe-xe-2018/binary/mkl', >>> '--with-cmake-dir=/cineca/prod/opt/tools/cmake/3.12.0/none', >>> '--with-debugging=0', >>> '--with-fortran-interfaces=1', >>> '--with-fortran=1', >>> 'FC=mpiifort', >>> 'PETSC_ARCH=arch-linux2-c-opt', >>> ] >>> configure.petsc_configure(configure_options) >>> >>> Now, I receive an error on hypre: >>> >>> forrtl: error (78): process killed (SIGTERM) >>> Image PC Routine Line >>> Source >>> libHYPRE-2.18.2.s 00002B33CF465D3F for__signal_handl >>> Unknown Unknown >>> libpthread-2.17.s 00002B33D5BFD370 Unknown >>> Unknown Unknown >>> libpthread-2.17.s 00002B33D5BF96D3 pthread_cond_wait >>> Unknown Unknown >>> libiomp5.so 00002B33DBA14E07 Unknown >>> Unknown Unknown >>> libiomp5.so 00002B33DB98810C Unknown >>> Unknown Unknown >>> libiomp5.so 00002B33DB990578 Unknown >>> Unknown Unknown >>> libiomp5.so 00002B33DB9D9659 Unknown >>> Unknown Unknown >>> libiomp5.so 00002B33DB9D8C39 Unknown >>> Unknown Unknown >>> libiomp5.so 00002B33DB993BCE __kmpc_fork_call >>> Unknown Unknown >>> PIC_3D 00000000004071C0 Unknown >>> Unknown Unknown >>> PIC_3D 0000000000490299 Unknown >>> Unknown Unknown >>> PIC_3D 0000000000492C17 Unknown >>> Unknown Unknown >>> PIC_3D 000000000040562E Unknown >>> Unknown Unknown >>> libc-2.17.so 00002B33DC5BEB35 __libc_start_main >>> Unknown Unknown >>> PIC_3D 0000000000405539 Unknown >>> Unknown Unknown >>> >>> Is it possible that I need to ask also to compile hypre with an option >>> for 64bit indices? >>> Is it possible to instruct this inside Petsc configure? >>> Alternatively, is it possible to use a different multigrid PC inside >>> PETSc that accept 64bit indices? >>> >> >> You can use >> -pc_type gamg >> All native PETSc implementations support 64bit indices. >> >> >>> >>> Thanks in advance >>> >>> Pierpaolo >>> >>> >>> Il giorno 27 mag 2020, alle ore 11:26, Stefano Zampini < >>> stefano.zampini at gmail.com> ha scritto: >>> >>> You need a version of PETSc compiled with 64bit indices, since the >>> message indicates the number of dofs in this case is larger the INT_MAX >>> 2501?3401?1601 = 13617947501 >>> >>> I also suggest you upgrade to a newer version, 3.8.3 is quite old as the >>> error message reports >>> >>> Il giorno mer 27 mag 2020 alle ore 11:50 Pierpaolo Minelli < >>> pierpaolo.minelli at cnr.it> ha scritto: >>> >>>> Hi, >>>> >>>> I am trying to solve a Poisson equation on this grid: >>>> >>>> Nx = 2501 >>>> Ny = 3401 >>>> Nz = 1601 >>>> >>>> I received this error: >>>> >>>> [0]PETSC ERROR: --------------------- Error Message >>>> -------------------------------------------------------------- >>>> [0]PETSC ERROR: Overflow in integer operation: >>>> http://www.mcs.anl.gov/petsc/documentation/faq.html#64-bit-indices >>>> [0]PETSC ERROR: Mesh of 2501 by 3401 by 1 (dof) is too large for 32 bit >>>> indices >>>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html >>>> for trouble shooting. >>>> [0]PETSC ERROR: Petsc Release Version 3.8.3, Dec, 09, 2017 >>>> [0]PETSC ERROR: >>>> /marconi_scratch/userexternal/pminelli/PIC3D/2500_3400_1600/./PIC_3D on a >>>> arch-linux2-c-opt named r129c09s02 by pminelli Tu >>>> e May 26 20:16:34 2020 >>>> [0]PETSC ERROR: Configure options >>>> --prefix=/cineca/prod/opt/libraries/petsc/3.8.3/intelmpi--2018--binary >>>> CC=mpiicc FC=mpiifort CXX=mpiicpc >>>> F77=mpiifort F90=mpiifort --with-debugging=0 >>>> --with-blaslapack-dir=/cineca/prod/opt/compilers/intel/pe-xe-2018/binary/mkl >>>> --with-fortran=1 >>>> --with-fortran-interfaces=1 >>>> --with-cmake-dir=/cineca/prod/opt/tools/cmake/3.5.2/none >>>> --with-mpi-dir=/cineca/prod/opt/compilers/intel/pe-xe- >>>> 2018/binary/impi/2018.4.274 --download-scalapack --download-mumps=yes >>>> --download-hypre --download-superlu_dist --download-parmetis --downlo >>>> ad-metis >>>> [0]PETSC ERROR: #1 DMSetUp_DA_3D() line 218 in >>>> /marconi/prod/build/libraries/petsc/3.8.3/intelmpi--2018--binary/BA_WORK/petsc-3.8.3/src/dm/ >>>> impls/da/da3.c >>>> [0]PETSC ERROR: #2 DMSetUp_DA() line 25 in >>>> /marconi/prod/build/libraries/petsc/3.8.3/intelmpi--2018--binary/BA_WORK/petsc-3.8.3/src/dm/impl >>>> s/da/dareg.c >>>> [0]PETSC ERROR: #3 DMSetUp() line 720 in >>>> /marconi/prod/build/libraries/petsc/3.8.3/intelmpi--2018--binary/BA_WORK/petsc-3.8.3/src/dm/interf >>>> ace/dm.c >>>> forrtl: error (76): Abort trap signal >>>> >>>> >>>> I am on an HPC facility and after I loaded PETSC module, I have seen >>>> that it is configured with INTEGER size = 32 >>>> >>>> I solve my problem with these options and it works perfectly with >>>> smaller grids: >>>> >>>> -dm_mat_type hypre -pc_type hypre -pc_hypre_type boomeramg >>>> -pc_hypre_boomeramg_relax_type_all SOR/Jacobi >>>> -pc_hypre_boomeramg_coarsen_type PMIS -pc_hypre_boomeramg_interp_type FF1 >>>> -ksp_type richardson >>>> >>>> Is it possible to overcome this if I ask them to install a version with >>>> INTEGER SIZE = 64? >>>> Alternatively, is it possible to overcome this using intel compiler >>>> options? >>>> >>>> Thanks in advance >>>> >>>> Pierpaolo Minelli >>> >>> >>> >>> -- >>> Stefano >>> >>> >> > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mfadams at lbl.gov Tue Jul 21 09:58:32 2020 From: mfadams at lbl.gov (Mark Adams) Date: Tue, 21 Jul 2020 10:58:32 -0400 Subject: [petsc-users] Error on INTEGER SIZE using DMDACreate3d In-Reply-To: References: <88A21A58-CDCD-42C0-9A20-1A6C5CBCDF8B@cnr.it> Message-ID: This also looks like it could be some sort of library mismatch. You might try deleting your architecture directory and start over. This PETSc's "make realclean" On Tue, Jul 21, 2020 at 10:45 AM Stefano Zampini wrote: > > > On Jul 21, 2020, at 1:32 PM, Pierpaolo Minelli > wrote: > > Hi, > > I have asked to compile a Petsc Version updated and with 64bit indices. > Now I have Version 3.13.3 and these are the configure options used: > > #!/bin/python > if __name__ == '__main__': > import sys > import os > sys.path.insert(0, os.path.abspath('config')) > import configure > configure_options = [ > '--CC=mpiicc', > '--CXX=mpiicpc', > '--download-hypre', > '--download-metis', > '--download-mumps=yes', > '--download-parmetis', > '--download-scalapack', > '--download-superlu_dist', > '--known-64-bit-blas-indices', > > '--prefix=/cineca/prod/opt/libraries/petsc/3.13.3_int64/intelmpi--2018--binary', > '--with-64-bit-indices=1', > > '--with-blaslapack-dir=/cineca/prod/opt/compilers/intel/pe-xe-2018/binary/mkl', > '--with-cmake-dir=/cineca/prod/opt/tools/cmake/3.12.0/none', > '--with-debugging=0', > '--with-fortran-interfaces=1', > '--with-fortran=1', > 'FC=mpiifort', > 'PETSC_ARCH=arch-linux2-c-opt', > ] > configure.petsc_configure(configure_options) > > Now, I receive an error on hypre: > > forrtl: error (78): process killed (SIGTERM) > Image PC Routine Line > Source > libHYPRE-2.18.2.s 00002B33CF465D3F for__signal_handl Unknown Unknown > libpthread-2.17.s 00002B33D5BFD370 Unknown Unknown Unknown > libpthread-2.17.s 00002B33D5BF96D3 pthread_cond_wait Unknown Unknown > libiomp5.so 00002B33DBA14E07 Unknown Unknown Unknown > libiomp5.so 00002B33DB98810C Unknown Unknown Unknown > libiomp5.so 00002B33DB990578 Unknown Unknown Unknown > libiomp5.so 00002B33DB9D9659 Unknown Unknown Unknown > libiomp5.so 00002B33DB9D8C39 Unknown Unknown Unknown > libiomp5.so 00002B33DB993BCE __kmpc_fork_call Unknown Unknown > PIC_3D 00000000004071C0 Unknown Unknown Unknown > PIC_3D 0000000000490299 Unknown Unknown Unknown > PIC_3D 0000000000492C17 Unknown Unknown Unknown > PIC_3D 000000000040562E Unknown Unknown Unknown > libc-2.17.so 00002B33DC5BEB35 __libc_start_main > Unknown Unknown > PIC_3D 0000000000405539 Unknown Unknown Unknown > > Is it possible > > that I need to ask also to compile hypre with an option for 64bit indices? > > > These configure options compile hypre with 64bit indices support. > It should work just fine. Can you run a very small case of your code to > confirm? > > > Is it possible to instruct this inside Petsc configure? > Alternatively, is it possible to use a different multigrid PC inside PETSc > that accept 64bit indices? > > Thanks in advance > > Pierpaolo > > > Il giorno 27 mag 2020, alle ore 11:26, Stefano Zampini < > stefano.zampini at gmail.com> ha scritto: > > You need a version of PETSc compiled with 64bit indices, since the message > indicates the number of dofs in this case is larger the INT_MAX > 2501?3401?1601 = 13617947501 > > I also suggest you upgrade to a newer version, 3.8.3 is quite old as the > error message reports > > Il giorno mer 27 mag 2020 alle ore 11:50 Pierpaolo Minelli < > pierpaolo.minelli at cnr.it> ha scritto: > >> Hi, >> >> I am trying to solve a Poisson equation on this grid: >> >> Nx = 2501 >> Ny = 3401 >> Nz = 1601 >> >> I received this error: >> >> [0]PETSC ERROR: --------------------- Error Message >> -------------------------------------------------------------- >> [0]PETSC ERROR: Overflow in integer operation: >> http://www.mcs.anl.gov/petsc/documentation/faq.html#64-bit-indices >> [0]PETSC ERROR: Mesh of 2501 by 3401 by 1 (dof) is too large for 32 bit >> indices >> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html >> for trouble shooting. >> [0]PETSC ERROR: Petsc Release Version 3.8.3, Dec, 09, 2017 >> [0]PETSC ERROR: >> /marconi_scratch/userexternal/pminelli/PIC3D/2500_3400_1600/./PIC_3D on a >> arch-linux2-c-opt named r129c09s02 by pminelli Tu >> e May 26 20:16:34 2020 >> [0]PETSC ERROR: Configure options >> --prefix=/cineca/prod/opt/libraries/petsc/3.8.3/intelmpi--2018--binary >> CC=mpiicc FC=mpiifort CXX=mpiicpc >> F77=mpiifort F90=mpiifort --with-debugging=0 >> --with-blaslapack-dir=/cineca/prod/opt/compilers/intel/pe-xe-2018/binary/mkl >> --with-fortran=1 >> --with-fortran-interfaces=1 >> --with-cmake-dir=/cineca/prod/opt/tools/cmake/3.5.2/none >> --with-mpi-dir=/cineca/prod/opt/compilers/intel/pe-xe- >> 2018/binary/impi/2018.4.274 --download-scalapack --download-mumps=yes >> --download-hypre --download-superlu_dist --download-parmetis --downlo >> ad-metis >> [0]PETSC ERROR: #1 DMSetUp_DA_3D() line 218 in >> /marconi/prod/build/libraries/petsc/3.8.3/intelmpi--2018--binary/BA_WORK/petsc-3.8.3/src/dm/ >> impls/da/da3.c >> [0]PETSC ERROR: #2 DMSetUp_DA() line 25 in >> /marconi/prod/build/libraries/petsc/3.8.3/intelmpi--2018--binary/BA_WORK/petsc-3.8.3/src/dm/impl >> s/da/dareg.c >> [0]PETSC ERROR: #3 DMSetUp() line 720 in >> /marconi/prod/build/libraries/petsc/3.8.3/intelmpi--2018--binary/BA_WORK/petsc-3.8.3/src/dm/interf >> ace/dm.c >> forrtl: error (76): Abort trap signal >> >> >> I am on an HPC facility and after I loaded PETSC module, I have seen that >> it is configured with INTEGER size = 32 >> >> I solve my problem with these options and it works perfectly with smaller >> grids: >> >> -dm_mat_type hypre -pc_type hypre -pc_hypre_type boomeramg >> -pc_hypre_boomeramg_relax_type_all SOR/Jacobi >> -pc_hypre_boomeramg_coarsen_type PMIS -pc_hypre_boomeramg_interp_type FF1 >> -ksp_type richardson >> >> Is it possible to overcome this if I ask them to install a version with >> INTEGER SIZE = 64? >> Alternatively, is it possible to overcome this using intel compiler >> options? >> >> Thanks in advance >> >> Pierpaolo Minelli > > > > -- > Stefano > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From pierpaolo.minelli at cnr.it Tue Jul 21 11:06:52 2020 From: pierpaolo.minelli at cnr.it (Pierpaolo Minelli) Date: Tue, 21 Jul 2020 18:06:52 +0200 Subject: [petsc-users] Error on INTEGER SIZE using DMDACreate3d In-Reply-To: References: <88A21A58-CDCD-42C0-9A20-1A6C5CBCDF8B@cnr.it> Message-ID: <5DF5D7B1-0937-4147-BF8E-069C9E493614@cnr.it> > Il giorno 21 lug 2020, alle ore 16:56, Mark Adams ha scritto: > > > > On Tue, Jul 21, 2020 at 9:46 AM Matthew Knepley > wrote: > On Tue, Jul 21, 2020 at 9:35 AM Pierpaolo Minelli > wrote: > Thanks for your reply. > As I wrote before, I use these settings: > > -dm_mat_type hypre -pc_type hypre -pc_hypre_type boomeramg -pc_hypre_boomeramg_relax_type_all SOR/Jacobi -pc_hypre_boomeramg_coarsen_type PMIS -pc_hypre_boomeramg_interp_type FF1 -ksp_type richardson > > Is there a way to emulate this features also with GAMG? > > Smoothers: You have complete control here > > -mg_levels_pc_type sor (the default is Chebyshev which you could also try) > > And you set the KSP type. You have -ksp_type richardson above but that is not used for Hypre. It is for GAMG. Chebyshev is a ksp type (-ksp_type chebyshev). > > Hypre is very good on Poisson. THe grid complexity (cost per iteration) can be high but the convergence rate will be better than GAMG. > > But, you should be able to get hypre to work. Yes it is very good for Poisson, and on a smaller case, at the beginning of my code development, I have tried Hypre, ML, and GAMG (without adding more options I have to admit) and hypre was faster without losing in precision and accurateness of results (I have checked them with -ksp_monitor_true_residual). I left -kps_type Richardson instead of default gmres only because from residuals it seems more accurate. So first, i will try again to see if hypre (with integer 64bit) is able to work on a smaller case as suggested by Stefano. Then I will investigate GAMG options and I will give you a feedback. The problem is that I need 64bit integers because of my problem size so I have to follow both path, but I hope that I will be able to continue to use hypre. Thanks Pierpaolo > > > Coarsening: This is much different in agglomeration AMG. There is a discussion here: > > https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/PC/PCGAMGSetThreshold.html > https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/PC/PCGAMGSetSquareGraph.html > > Interpolation: This is built-in for agglomeration AMG. > > It would be better to use only native Petsc implementations, but these settings, up to single precision indexing for integers, gave me optimal performances. > For this reason I asked also, if it was possible to configure hypre (inside Petsc) with 64bit integers. > > Yes. That happened when you reconfigured for 64 bits. You may have encountered a Hypre bug. > > Thanks, > > Matt > > Pierpaolo > > >> Il giorno 21 lug 2020, alle ore 13:36, Dave May > ha scritto: >> >> >> >> On Tue, 21 Jul 2020 at 12:32, Pierpaolo Minelli > wrote: >> Hi, >> >> I have asked to compile a Petsc Version updated and with 64bit indices. >> Now I have Version 3.13.3 and these are the configure options used: >> >> #!/bin/python >> if __name__ == '__main__': >> import sys >> import os >> sys.path.insert(0, os.path.abspath('config')) >> import configure >> configure_options = [ >> '--CC=mpiicc', >> '--CXX=mpiicpc', >> '--download-hypre', >> '--download-metis', >> '--download-mumps=yes', >> '--download-parmetis', >> '--download-scalapack', >> '--download-superlu_dist', >> '--known-64-bit-blas-indices', >> '--prefix=/cineca/prod/opt/libraries/petsc/3.13.3_int64/intelmpi--2018--binary', >> '--with-64-bit-indices=1', >> '--with-blaslapack-dir=/cineca/prod/opt/compilers/intel/pe-xe-2018/binary/mkl', >> '--with-cmake-dir=/cineca/prod/opt/tools/cmake/3.12.0/none', >> '--with-debugging=0', >> '--with-fortran-interfaces=1', >> '--with-fortran=1', >> 'FC=mpiifort', >> 'PETSC_ARCH=arch-linux2-c-opt', >> ] >> configure.petsc_configure(configure_options) >> >> Now, I receive an error on hypre: >> >> forrtl: error (78): process killed (SIGTERM) >> Image PC Routine Line Source >> libHYPRE-2.18.2.s 00002B33CF465D3F for__signal_handl Unknown Unknown >> libpthread-2.17.s 00002B33D5BFD370 Unknown Unknown Unknown >> libpthread-2.17.s 00002B33D5BF96D3 pthread_cond_wait Unknown Unknown >> libiomp5.so 00002B33DBA14E07 Unknown Unknown Unknown >> libiomp5.so 00002B33DB98810C Unknown Unknown Unknown >> libiomp5.so 00002B33DB990578 Unknown Unknown Unknown >> libiomp5.so 00002B33DB9D9659 Unknown Unknown Unknown >> libiomp5.so 00002B33DB9D8C39 Unknown Unknown Unknown >> libiomp5.so 00002B33DB993BCE __kmpc_fork_call Unknown Unknown >> PIC_3D 00000000004071C0 Unknown Unknown Unknown >> PIC_3D 0000000000490299 Unknown Unknown Unknown >> PIC_3D 0000000000492C17 Unknown Unknown Unknown >> PIC_3D 000000000040562E Unknown Unknown Unknown >> libc-2.17.so 00002B33DC5BEB35 __libc_start_main Unknown Unknown >> PIC_3D 0000000000405539 Unknown Unknown Unknown >> >> Is it possible that I need to ask also to compile hypre with an option for 64bit indices? >> Is it possible to instruct this inside Petsc configure? >> Alternatively, is it possible to use a different multigrid PC inside PETSc that accept 64bit indices? >> >> You can use >> -pc_type gamg >> All native PETSc implementations support 64bit indices. >> >> >> Thanks in advance >> >> Pierpaolo >> >> >>> Il giorno 27 mag 2020, alle ore 11:26, Stefano Zampini > ha scritto: >>> >>> You need a version of PETSc compiled with 64bit indices, since the message indicates the number of dofs in this case is larger the INT_MAX >>> 2501?3401?1601 = 13617947501 >>> >>> I also suggest you upgrade to a newer version, 3.8.3 is quite old as the error message reports >>> >>> Il giorno mer 27 mag 2020 alle ore 11:50 Pierpaolo Minelli > ha scritto: >>> Hi, >>> >>> I am trying to solve a Poisson equation on this grid: >>> >>> Nx = 2501 >>> Ny = 3401 >>> Nz = 1601 >>> >>> I received this error: >>> >>> [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- >>> [0]PETSC ERROR: Overflow in integer operation: http://www.mcs.anl.gov/petsc/documentation/faq.html#64-bit-indices >>> [0]PETSC ERROR: Mesh of 2501 by 3401 by 1 (dof) is too large for 32 bit indices >>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. >>> [0]PETSC ERROR: Petsc Release Version 3.8.3, Dec, 09, 2017 >>> [0]PETSC ERROR: /marconi_scratch/userexternal/pminelli/PIC3D/2500_3400_1600/./PIC_3D on a arch-linux2-c-opt named r129c09s02 by pminelli Tu >>> e May 26 20:16:34 2020 >>> [0]PETSC ERROR: Configure options --prefix=/cineca/prod/opt/libraries/petsc/3.8.3/intelmpi--2018--binary CC=mpiicc FC=mpiifort CXX=mpiicpc >>> F77=mpiifort F90=mpiifort --with-debugging=0 --with-blaslapack-dir=/cineca/prod/opt/compilers/intel/pe-xe-2018/binary/mkl --with-fortran=1 >>> --with-fortran-interfaces=1 --with-cmake-dir=/cineca/prod/opt/tools/cmake/3.5.2/none --with-mpi-dir=/cineca/prod/opt/compilers/intel/pe-xe- >>> 2018/binary/impi/2018.4.274 --download-scalapack --download-mumps=yes --download-hypre --download-superlu_dist --download-parmetis --downlo >>> ad-metis >>> [0]PETSC ERROR: #1 DMSetUp_DA_3D() line 218 in /marconi/prod/build/libraries/petsc/3.8.3/intelmpi--2018--binary/BA_WORK/petsc-3.8.3/src/dm/ >>> impls/da/da3.c >>> [0]PETSC ERROR: #2 DMSetUp_DA() line 25 in /marconi/prod/build/libraries/petsc/3.8.3/intelmpi--2018--binary/BA_WORK/petsc-3.8.3/src/dm/impl >>> s/da/dareg.c >>> [0]PETSC ERROR: #3 DMSetUp() line 720 in /marconi/prod/build/libraries/petsc/3.8.3/intelmpi--2018--binary/BA_WORK/petsc-3.8.3/src/dm/interf >>> ace/dm.c >>> forrtl: error (76): Abort trap signal >>> >>> >>> I am on an HPC facility and after I loaded PETSC module, I have seen that it is configured with INTEGER size = 32 >>> >>> I solve my problem with these options and it works perfectly with smaller grids: >>> >>> -dm_mat_type hypre -pc_type hypre -pc_hypre_type boomeramg -pc_hypre_boomeramg_relax_type_all SOR/Jacobi -pc_hypre_boomeramg_coarsen_type PMIS -pc_hypre_boomeramg_interp_type FF1 -ksp_type richardson >>> >>> Is it possible to overcome this if I ask them to install a version with INTEGER SIZE = 64? >>> Alternatively, is it possible to overcome this using intel compiler options? >>> >>> Thanks in advance >>> >>> Pierpaolo Minelli >>> >>> >>> -- >>> Stefano > > > > -- > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From pierpaolo.minelli at cnr.it Tue Jul 21 11:11:34 2020 From: pierpaolo.minelli at cnr.it (Pierpaolo Minelli) Date: Tue, 21 Jul 2020 18:11:34 +0200 Subject: [petsc-users] Error on INTEGER SIZE using DMDACreate3d In-Reply-To: References: <88A21A58-CDCD-42C0-9A20-1A6C5CBCDF8B@cnr.it> Message-ID: > Il giorno 21 lug 2020, alle ore 16:58, Mark Adams ha scritto: > > This also looks like it could be some sort of library mismatch. You might try deleting your architecture directory and start over. This PETSc's "make realclean" I hope this is not the case, because I am working on CINECA HPC Facility (Italy) and in this center I need to load modules for each software I need. I asked Cineca support to compile a version of Petsc with 64 bit integers and all that external packages and after they have done it, I loaded directly this new module, so the older version (3.8.x with integer in single precision) is not involved at all. At least I hope? Thanks Pierpaolo > > On Tue, Jul 21, 2020 at 10:45 AM Stefano Zampini > wrote: > > >> On Jul 21, 2020, at 1:32 PM, Pierpaolo Minelli > wrote: >> >> Hi, >> >> I have asked to compile a Petsc Version updated and with 64bit indices. >> Now I have Version 3.13.3 and these are the configure options used: >> >> #!/bin/python >> if __name__ == '__main__': >> import sys >> import os >> sys.path.insert(0, os.path.abspath('config')) >> import configure >> configure_options = [ >> '--CC=mpiicc', >> '--CXX=mpiicpc', >> '--download-hypre', >> '--download-metis', >> '--download-mumps=yes', >> '--download-parmetis', >> '--download-scalapack', >> '--download-superlu_dist', >> '--known-64-bit-blas-indices', >> '--prefix=/cineca/prod/opt/libraries/petsc/3.13.3_int64/intelmpi--2018--binary', >> '--with-64-bit-indices=1', >> '--with-blaslapack-dir=/cineca/prod/opt/compilers/intel/pe-xe-2018/binary/mkl', >> '--with-cmake-dir=/cineca/prod/opt/tools/cmake/3.12.0/none', >> '--with-debugging=0', >> '--with-fortran-interfaces=1', >> '--with-fortran=1', >> 'FC=mpiifort', >> 'PETSC_ARCH=arch-linux2-c-opt', >> ] >> configure.petsc_configure(configure_options) >> >> Now, I receive an error on hypre: >> >> forrtl: error (78): process killed (SIGTERM) >> Image PC Routine Line Source >> libHYPRE-2.18.2.s 00002B33CF465D3F for__signal_handl Unknown Unknown >> libpthread-2.17.s 00002B33D5BFD370 Unknown Unknown Unknown >> libpthread-2.17.s 00002B33D5BF96D3 pthread_cond_wait Unknown Unknown >> libiomp5.so 00002B33DBA14E07 Unknown Unknown Unknown >> libiomp5.so 00002B33DB98810C Unknown Unknown Unknown >> libiomp5.so 00002B33DB990578 Unknown Unknown Unknown >> libiomp5.so 00002B33DB9D9659 Unknown Unknown Unknown >> libiomp5.so 00002B33DB9D8C39 Unknown Unknown Unknown >> libiomp5.so 00002B33DB993BCE __kmpc_fork_call Unknown Unknown >> PIC_3D 00000000004071C0 Unknown Unknown Unknown >> PIC_3D 0000000000490299 Unknown Unknown Unknown >> PIC_3D 0000000000492C17 Unknown Unknown Unknown >> PIC_3D 000000000040562E Unknown Unknown Unknown >> libc-2.17.so 00002B33DC5BEB35 __libc_start_main Unknown Unknown >> PIC_3D 0000000000405539 Unknown Unknown Unknown >> >> Is it possible >> that I need to ask also to compile hypre with an option for 64bit indices? > > These configure options compile hypre with 64bit indices support. > It should work just fine. Can you run a very small case of your code to confirm? > > >> Is it possible to instruct this inside Petsc configure? >> Alternatively, is it possible to use a different multigrid PC inside PETSc that accept 64bit indices? >> >> Thanks in advance >> >> Pierpaolo >> >> >>> Il giorno 27 mag 2020, alle ore 11:26, Stefano Zampini > ha scritto: >>> >>> You need a version of PETSc compiled with 64bit indices, since the message indicates the number of dofs in this case is larger the INT_MAX >>> 2501?3401?1601 = 13617947501 >>> >>> I also suggest you upgrade to a newer version, 3.8.3 is quite old as the error message reports >>> >>> Il giorno mer 27 mag 2020 alle ore 11:50 Pierpaolo Minelli > ha scritto: >>> Hi, >>> >>> I am trying to solve a Poisson equation on this grid: >>> >>> Nx = 2501 >>> Ny = 3401 >>> Nz = 1601 >>> >>> I received this error: >>> >>> [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- >>> [0]PETSC ERROR: Overflow in integer operation: http://www.mcs.anl.gov/petsc/documentation/faq.html#64-bit-indices >>> [0]PETSC ERROR: Mesh of 2501 by 3401 by 1 (dof) is too large for 32 bit indices >>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. >>> [0]PETSC ERROR: Petsc Release Version 3.8.3, Dec, 09, 2017 >>> [0]PETSC ERROR: /marconi_scratch/userexternal/pminelli/PIC3D/2500_3400_1600/./PIC_3D on a arch-linux2-c-opt named r129c09s02 by pminelli Tu >>> e May 26 20:16:34 2020 >>> [0]PETSC ERROR: Configure options --prefix=/cineca/prod/opt/libraries/petsc/3.8.3/intelmpi--2018--binary CC=mpiicc FC=mpiifort CXX=mpiicpc >>> F77=mpiifort F90=mpiifort --with-debugging=0 --with-blaslapack-dir=/cineca/prod/opt/compilers/intel/pe-xe-2018/binary/mkl --with-fortran=1 >>> --with-fortran-interfaces=1 --with-cmake-dir=/cineca/prod/opt/tools/cmake/3.5.2/none --with-mpi-dir=/cineca/prod/opt/compilers/intel/pe-xe- >>> 2018/binary/impi/2018.4.274 --download-scalapack --download-mumps=yes --download-hypre --download-superlu_dist --download-parmetis --downlo >>> ad-metis >>> [0]PETSC ERROR: #1 DMSetUp_DA_3D() line 218 in /marconi/prod/build/libraries/petsc/3.8.3/intelmpi--2018--binary/BA_WORK/petsc-3.8.3/src/dm/ >>> impls/da/da3.c >>> [0]PETSC ERROR: #2 DMSetUp_DA() line 25 in /marconi/prod/build/libraries/petsc/3.8.3/intelmpi--2018--binary/BA_WORK/petsc-3.8.3/src/dm/impl >>> s/da/dareg.c >>> [0]PETSC ERROR: #3 DMSetUp() line 720 in /marconi/prod/build/libraries/petsc/3.8.3/intelmpi--2018--binary/BA_WORK/petsc-3.8.3/src/dm/interf >>> ace/dm.c >>> forrtl: error (76): Abort trap signal >>> >>> >>> I am on an HPC facility and after I loaded PETSC module, I have seen that it is configured with INTEGER size = 32 >>> >>> I solve my problem with these options and it works perfectly with smaller grids: >>> >>> -dm_mat_type hypre -pc_type hypre -pc_hypre_type boomeramg -pc_hypre_boomeramg_relax_type_all SOR/Jacobi -pc_hypre_boomeramg_coarsen_type PMIS -pc_hypre_boomeramg_interp_type FF1 -ksp_type richardson >>> >>> Is it possible to overcome this if I ask them to install a version with INTEGER SIZE = 64? >>> Alternatively, is it possible to overcome this using intel compiler options? >>> >>> Thanks in advance >>> >>> Pierpaolo Minelli >>> >>> >>> -- >>> Stefano >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From adantra at gmail.com Tue Jul 21 11:37:44 2020 From: adantra at gmail.com (Adolfo Rodriguez) Date: Tue, 21 Jul 2020 11:37:44 -0500 Subject: [petsc-users] snes setjacobian with csr sparse matrix In-Reply-To: <9FC7A7FF-697F-408F-A13D-AE10D900C61C@petsc.dev> References: <9FC7A7FF-697F-408F-A13D-AE10D900C61C@petsc.dev> Message-ID: Thanks! That worked. On Mon, Jul 20, 2020 at 11:10 AM Barry Smith wrote: > > If you are running sequentially you likely want the first routine, in > parallel the third routine. > > MatCreateSeqAIJWithArrays() > PETSC_EXTERN PetscErrorCode MatSeqAIJSetPreallocationCSR(Mat,const > PetscInt [],const PetscInt [],const PetscScalar []); > PETSC_EXTERN PetscErrorCode MatMPIAIJSetPreallocationCSR(Mat,const > PetscInt[],const PetscInt[],const PetscScalar[]); > > > PETSC_EXTERN PetscErrorCode > MatSeqBAIJSetPreallocationCSR(Mat,PetscInt,const PetscInt[],const > PetscInt[],const PetscScalar[]); > PETSC_EXTERN PetscErrorCode > MatMPIBAIJSetPreallocationCSR(Mat,PetscInt,const PetscInt[],const > PetscInt[],const PetscScalar[]); > PETSC_EXTERN PetscErrorCode > MatSeqSBAIJSetPreallocationCSR(Mat,PetscInt,const PetscInt[],const > PetscInt[],const PetscScalar[]); > PETSC_EXTERN PetscErrorCode > MatMPISBAIJSetPreallocationCSR(Mat,PetscInt,const PetscInt[],const > PetscInt[],const PetscScalar[]); > > > > > On Jul 20, 2020, at 7:38 AM, Adolfo Rodriguez wrote: > > > > I am trying to use PETSc to solve a non-linear problem. The problem I am > facing is that I already have the Jacobian matrix formed in CSR format. Now > I am trying to define a function FormJacobian which takes this preexisting > matrix and writes it in PETSc format. For a small problem, I can use > MatSetValue, looping over all non-zero elements one at a time. However, > this turns out too slow for larger problems. > > > > In my current implementation, I am using MatCreateSeqAIJ which works > great. However, this approach does not seem to work to set the Jacobian. > > > > Any suggestions? > > > > Regards, > > > > Adolfo > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From eda.oktay at metu.edu.tr Tue Jul 21 11:54:04 2020 From: eda.oktay at metu.edu.tr (Eda Oktay) Date: Tue, 21 Jul 2020 19:54:04 +0300 Subject: [petsc-users] Is PETSc using internet? In-Reply-To: <9442785D-77DA-4AC7-B6BA-DEDE52D984F8@gmail.com> References: <091AE85F-3B71-4D03-A63B-9FEE610668DF@gmail.com> <9442785D-77DA-4AC7-B6BA-DEDE52D984F8@gmail.com> Message-ID: Dear Lawrence, The problem is not the error by the way, my program is waiting something without stopping and it is not giving error. It just does nothing. Does the problem is still because of hostname? Thanks! Eda On Tue, Jul 21, 2020, 1:16 PM Lawrence Mitchell wrote: > > > > On 21 Jul 2020, at 11:06, Eda Oktay wrote: > > > > Dear Lawrence, > > > > I am using MPICC but not Mac, Fedora 25. If it will still work, I will > try that. > > > > Thanks! > > It might be the case. When you observe the error, does "nslookup > localhost" take a long time? > > Lawrence -------------- next part -------------- An HTML attachment was scrubbed... URL: From balay at mcs.anl.gov Tue Jul 21 11:57:28 2020 From: balay at mcs.anl.gov (Satish Balay) Date: Tue, 21 Jul 2020 11:57:28 -0500 (CDT) Subject: [petsc-users] Is PETSc using internet? In-Reply-To: References: <091AE85F-3B71-4D03-A63B-9FEE610668DF@gmail.com> <9442785D-77DA-4AC7-B6BA-DEDE52D984F8@gmail.com> Message-ID: you can run in the gdb to see if its hanging in a call to gethostbyname() or somewhere else. Satish On Tue, 21 Jul 2020, Eda Oktay wrote: > Dear Lawrence, > > The problem is not the error by the way, my program is waiting something > without stopping and it is not giving error. It just does nothing. > > Does the problem is still because of hostname? > > Thanks! > > Eda > > On Tue, Jul 21, 2020, 1:16 PM Lawrence Mitchell wrote: > > > > > > > > On 21 Jul 2020, at 11:06, Eda Oktay wrote: > > > > > > Dear Lawrence, > > > > > > I am using MPICC but not Mac, Fedora 25. If it will still work, I will > > try that. > > > > > > Thanks! > > > > It might be the case. When you observe the error, does "nslookup > > localhost" take a long time? > > > > Lawrence > From balay at mcs.anl.gov Tue Jul 21 12:00:34 2020 From: balay at mcs.anl.gov (Satish Balay) Date: Tue, 21 Jul 2020 12:00:34 -0500 (CDT) Subject: [petsc-users] Is PETSc using internet? In-Reply-To: <091AE85F-3B71-4D03-A63B-9FEE610668DF@gmail.com> References: <091AE85F-3B71-4D03-A63B-9FEE610668DF@gmail.com> Message-ID: On Tue, 21 Jul 2020, Lawrence Mitchell wrote: > > > > On 21 Jul 2020, at 10:49, Eda Oktay wrote: > > > > Hi all, > > > > I am using the following libraries and for some reason, I figured out > > that if ? am disconnected to internet, my program is not working: > > [...] > > > I thought that there is no reason for PETSc to go online for libraries > > but I couldn't find any other reason. > > > > The first thing I am doing in my program after initializing SLEPc and > > PETSc is reading the binary matrix from my file and when I am offline, > > the program can't even get the matrix. > > My crystal ball (which sometimes is wrong) guess that you're using a Mac, and MPICH as your MPI version? If so, it may be a problem in MPI_Init failing to resolve localhost to an IP address. > > If these guesses are correct, does it help to add > > 127.0.0.1 LOCALHOSTNAME.local > 127.0.0.1 LOCALHOSTNAME > > to /etc/hosts > > where LOCALHOSTNAME is the name of the machine (as reported by hostname) The equivalent command is: echo 127.0.0.1 `hostname` | sudo tee -a /etc/hosts Satish From mfadams at lbl.gov Tue Jul 21 12:10:57 2020 From: mfadams at lbl.gov (Mark Adams) Date: Tue, 21 Jul 2020 13:10:57 -0400 Subject: [petsc-users] Error on INTEGER SIZE using DMDACreate3d In-Reply-To: <5DF5D7B1-0937-4147-BF8E-069C9E493614@cnr.it> References: <88A21A58-CDCD-42C0-9A20-1A6C5CBCDF8B@cnr.it> <5DF5D7B1-0937-4147-BF8E-069C9E493614@cnr.it> Message-ID: On Tue, Jul 21, 2020 at 12:06 PM Pierpaolo Minelli wrote: > > > Il giorno 21 lug 2020, alle ore 16:56, Mark Adams ha > scritto: > > > > On Tue, Jul 21, 2020 at 9:46 AM Matthew Knepley wrote: > >> On Tue, Jul 21, 2020 at 9:35 AM Pierpaolo Minelli < >> pierpaolo.minelli at cnr.it> wrote: >> >>> Thanks for your reply. >>> As I wrote before, I use these settings: >>> >>> -dm_mat_type hypre -pc_type hypre -pc_hypre_type boomeramg >>> -pc_hypre_boomeramg_relax_type_all SOR/Jacobi >>> -pc_hypre_boomeramg_coarsen_type PMIS -pc_hypre_boomeramg_interp_type FF1 >>> -ksp_type richardson >>> >>> Is there a way to emulate this features also with GAMG? >>> >> >> Smoothers: You have complete control here >> >> -mg_levels_pc_type sor (the default is Chebyshev which you could also >> try) >> > > And you set the KSP type. You have -ksp_type richardson above but that is > not used for Hypre. It is for GAMG. Chebyshev is a ksp type (-ksp_type > chebyshev). > > > > Hypre is very good on Poisson. THe grid complexity (cost per iteration) > can be high but the convergence rate will be better than GAMG. > > But, you should be able to get hypre to work. > > > Yes it is very good for Poisson, and on a smaller case, at the beginning > of my code development, I have tried Hypre, ML, and GAMG (without adding > more options I have to admit) and hypre was faster without losing in > precision and accurateness of results (I have checked them with > -ksp_monitor_true_residual). > I left -kps_type Richardson instead of default gmres only because from > residuals it seems more accurate. > Whoops, I made a mistake. I was talking about the smoother. So -mg_levels_pc_type sor -mg_levels_ksp_type chebyshev > > So first, i will try again to see if hypre (with integer 64bit) is able > to work on a smaller case as suggested by Stefano. > Then I will investigate GAMG options and I will give you a feedback. > The problem is that I need 64bit integers because of my problem size so I > have to follow both path, but I hope that I will be able to continue to use > hypre. > > Thanks > > Pierpaolo > > > > >> >> Coarsening: This is much different in agglomeration AMG. There is a >> discussion here: >> >> >> https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/PC/PCGAMGSetThreshold.html >> >> https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/PC/PCGAMGSetSquareGraph.html >> >> Interpolation: This is built-in for agglomeration AMG. >> >> >>> It would be better to use only native Petsc implementations, but these >>> settings, up to single precision indexing for integers, gave me optimal >>> performances. >>> For this reason I asked also, if it was possible to configure hypre >>> (inside Petsc) with 64bit integers. >>> >> >> Yes. That happened when you reconfigured for 64 bits. You may have >> encountered a Hypre bug. >> >> Thanks, >> >> Matt >> >> >>> Pierpaolo >>> >>> >>> Il giorno 21 lug 2020, alle ore 13:36, Dave May >>> ha scritto: >>> >>> >>> >>> On Tue, 21 Jul 2020 at 12:32, Pierpaolo Minelli < >>> pierpaolo.minelli at cnr.it> wrote: >>> >>>> Hi, >>>> >>>> I have asked to compile a Petsc Version updated and with 64bit indices. >>>> Now I have Version 3.13.3 and these are the configure options used: >>>> >>>> #!/bin/python >>>> if __name__ == '__main__': >>>> import sys >>>> import os >>>> sys.path.insert(0, os.path.abspath('config')) >>>> import configure >>>> configure_options = [ >>>> '--CC=mpiicc', >>>> '--CXX=mpiicpc', >>>> '--download-hypre', >>>> '--download-metis', >>>> '--download-mumps=yes', >>>> '--download-parmetis', >>>> '--download-scalapack', >>>> '--download-superlu_dist', >>>> '--known-64-bit-blas-indices', >>>> >>>> '--prefix=/cineca/prod/opt/libraries/petsc/3.13.3_int64/intelmpi--2018--binary', >>>> '--with-64-bit-indices=1', >>>> >>>> '--with-blaslapack-dir=/cineca/prod/opt/compilers/intel/pe-xe-2018/binary/mkl', >>>> '--with-cmake-dir=/cineca/prod/opt/tools/cmake/3.12.0/none', >>>> '--with-debugging=0', >>>> '--with-fortran-interfaces=1', >>>> '--with-fortran=1', >>>> 'FC=mpiifort', >>>> 'PETSC_ARCH=arch-linux2-c-opt', >>>> ] >>>> configure.petsc_configure(configure_options) >>>> >>>> Now, I receive an error on hypre: >>>> >>>> forrtl: error (78): process killed (SIGTERM) >>>> Image PC Routine Line >>>> Source >>>> libHYPRE-2.18.2.s 00002B33CF465D3F for__signal_handl >>>> Unknown Unknown >>>> libpthread-2.17.s 00002B33D5BFD370 Unknown >>>> Unknown Unknown >>>> libpthread-2.17.s 00002B33D5BF96D3 pthread_cond_wait >>>> Unknown Unknown >>>> libiomp5.so 00002B33DBA14E07 Unknown >>>> Unknown Unknown >>>> libiomp5.so 00002B33DB98810C Unknown >>>> Unknown Unknown >>>> libiomp5.so 00002B33DB990578 Unknown >>>> Unknown Unknown >>>> libiomp5.so 00002B33DB9D9659 Unknown >>>> Unknown Unknown >>>> libiomp5.so 00002B33DB9D8C39 Unknown >>>> Unknown Unknown >>>> libiomp5.so 00002B33DB993BCE __kmpc_fork_call >>>> Unknown Unknown >>>> PIC_3D 00000000004071C0 Unknown >>>> Unknown Unknown >>>> PIC_3D 0000000000490299 Unknown >>>> Unknown Unknown >>>> PIC_3D 0000000000492C17 Unknown >>>> Unknown Unknown >>>> PIC_3D 000000000040562E Unknown >>>> Unknown Unknown >>>> libc-2.17.so 00002B33DC5BEB35 __libc_start_main >>>> Unknown Unknown >>>> PIC_3D 0000000000405539 Unknown >>>> Unknown Unknown >>>> >>>> Is it possible that I need to ask also to compile hypre with an option >>>> for 64bit indices? >>>> Is it possible to instruct this inside Petsc configure? >>>> Alternatively, is it possible to use a different multigrid PC inside >>>> PETSc that accept 64bit indices? >>>> >>> >>> You can use >>> -pc_type gamg >>> All native PETSc implementations support 64bit indices. >>> >>> >>>> >>>> Thanks in advance >>>> >>>> Pierpaolo >>>> >>>> >>>> Il giorno 27 mag 2020, alle ore 11:26, Stefano Zampini < >>>> stefano.zampini at gmail.com> ha scritto: >>>> >>>> You need a version of PETSc compiled with 64bit indices, since the >>>> message indicates the number of dofs in this case is larger the INT_MAX >>>> 2501?3401?1601 = 13617947501 >>>> >>>> I also suggest you upgrade to a newer version, 3.8.3 is quite old as >>>> the error message reports >>>> >>>> Il giorno mer 27 mag 2020 alle ore 11:50 Pierpaolo Minelli < >>>> pierpaolo.minelli at cnr.it> ha scritto: >>>> >>>>> Hi, >>>>> >>>>> I am trying to solve a Poisson equation on this grid: >>>>> >>>>> Nx = 2501 >>>>> Ny = 3401 >>>>> Nz = 1601 >>>>> >>>>> I received this error: >>>>> >>>>> [0]PETSC ERROR: --------------------- Error Message >>>>> -------------------------------------------------------------- >>>>> [0]PETSC ERROR: Overflow in integer operation: >>>>> http://www.mcs.anl.gov/petsc/documentation/faq.html#64-bit-indices >>>>> [0]PETSC ERROR: Mesh of 2501 by 3401 by 1 (dof) is too large for 32 >>>>> bit indices >>>>> [0]PETSC ERROR: See >>>>> http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble >>>>> shooting. >>>>> [0]PETSC ERROR: Petsc Release Version 3.8.3, Dec, 09, 2017 >>>>> [0]PETSC ERROR: >>>>> /marconi_scratch/userexternal/pminelli/PIC3D/2500_3400_1600/./PIC_3D on a >>>>> arch-linux2-c-opt named r129c09s02 by pminelli Tu >>>>> e May 26 20:16:34 2020 >>>>> [0]PETSC ERROR: Configure options >>>>> --prefix=/cineca/prod/opt/libraries/petsc/3.8.3/intelmpi--2018--binary >>>>> CC=mpiicc FC=mpiifort CXX=mpiicpc >>>>> F77=mpiifort F90=mpiifort --with-debugging=0 >>>>> --with-blaslapack-dir=/cineca/prod/opt/compilers/intel/pe-xe-2018/binary/mkl >>>>> --with-fortran=1 >>>>> --with-fortran-interfaces=1 >>>>> --with-cmake-dir=/cineca/prod/opt/tools/cmake/3.5.2/none >>>>> --with-mpi-dir=/cineca/prod/opt/compilers/intel/pe-xe- >>>>> 2018/binary/impi/2018.4.274 --download-scalapack --download-mumps=yes >>>>> --download-hypre --download-superlu_dist --download-parmetis --downlo >>>>> ad-metis >>>>> [0]PETSC ERROR: #1 DMSetUp_DA_3D() line 218 in >>>>> /marconi/prod/build/libraries/petsc/3.8.3/intelmpi--2018--binary/BA_WORK/petsc-3.8.3/src/dm/ >>>>> impls/da/da3.c >>>>> [0]PETSC ERROR: #2 DMSetUp_DA() line 25 in >>>>> /marconi/prod/build/libraries/petsc/3.8.3/intelmpi--2018--binary/BA_WORK/petsc-3.8.3/src/dm/impl >>>>> s/da/dareg.c >>>>> [0]PETSC ERROR: #3 DMSetUp() line 720 in >>>>> /marconi/prod/build/libraries/petsc/3.8.3/intelmpi--2018--binary/BA_WORK/petsc-3.8.3/src/dm/interf >>>>> ace/dm.c >>>>> forrtl: error (76): Abort trap signal >>>>> >>>>> >>>>> I am on an HPC facility and after I loaded PETSC module, I have seen >>>>> that it is configured with INTEGER size = 32 >>>>> >>>>> I solve my problem with these options and it works perfectly with >>>>> smaller grids: >>>>> >>>>> -dm_mat_type hypre -pc_type hypre -pc_hypre_type boomeramg >>>>> -pc_hypre_boomeramg_relax_type_all SOR/Jacobi >>>>> -pc_hypre_boomeramg_coarsen_type PMIS -pc_hypre_boomeramg_interp_type FF1 >>>>> -ksp_type richardson >>>>> >>>>> Is it possible to overcome this if I ask them to install a version >>>>> with INTEGER SIZE = 64? >>>>> Alternatively, is it possible to overcome this using intel compiler >>>>> options? >>>>> >>>>> Thanks in advance >>>>> >>>>> Pierpaolo Minelli >>>> >>>> >>>> >>>> -- >>>> Stefano >>>> >>>> >>> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> -- Norbert Wiener >> >> https://www.cse.buffalo.edu/~knepley/ >> >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mfadams at lbl.gov Tue Jul 21 12:45:12 2020 From: mfadams at lbl.gov (Mark Adams) Date: Tue, 21 Jul 2020 13:45:12 -0400 Subject: [petsc-users] Error on INTEGER SIZE using DMDACreate3d In-Reply-To: References: <88A21A58-CDCD-42C0-9A20-1A6C5CBCDF8B@cnr.it> Message-ID: On Tue, Jul 21, 2020 at 12:11 PM Pierpaolo Minelli wrote: > > > Il giorno 21 lug 2020, alle ore 16:58, Mark Adams ha > scritto: > > This also looks like it could be some sort of library mismatch. You might > try deleting your architecture directory and start over. This PETSc's "make > realclean" > > > I hope this is not the case, because I am working on CINECA HPC Facility > (Italy) and in this center I need to load modules for each software I need. > I asked Cineca support to compile a version of Petsc with 64 bit integers > and all that external packages and after they have done it, I loaded > directly this new module, so the older version (3.8.x with integer in > single precision) is not involved at all. > At least I hope? > You can run a PETSc test. Download PETSc and go into the PETSc directory. On an execution node or batch script run 'make check' with PETSC_ARCH and PETSC_DIR set as usual (ie, not $PWD). THis should run a few tests including a hypre test. If it works, then there may be a problem with how you are linking your program. If it fails then then ask the Facility to help you. > > Thanks > > Pierpaolo > > > On Tue, Jul 21, 2020 at 10:45 AM Stefano Zampini < > stefano.zampini at gmail.com> wrote: > >> >> >> On Jul 21, 2020, at 1:32 PM, Pierpaolo Minelli >> wrote: >> >> Hi, >> >> I have asked to compile a Petsc Version updated and with 64bit indices. >> Now I have Version 3.13.3 and these are the configure options used: >> >> #!/bin/python >> if __name__ == '__main__': >> import sys >> import os >> sys.path.insert(0, os.path.abspath('config')) >> import configure >> configure_options = [ >> '--CC=mpiicc', >> '--CXX=mpiicpc', >> '--download-hypre', >> '--download-metis', >> '--download-mumps=yes', >> '--download-parmetis', >> '--download-scalapack', >> '--download-superlu_dist', >> '--known-64-bit-blas-indices', >> >> '--prefix=/cineca/prod/opt/libraries/petsc/3.13.3_int64/intelmpi--2018--binary', >> '--with-64-bit-indices=1', >> >> '--with-blaslapack-dir=/cineca/prod/opt/compilers/intel/pe-xe-2018/binary/mkl', >> '--with-cmake-dir=/cineca/prod/opt/tools/cmake/3.12.0/none', >> '--with-debugging=0', >> '--with-fortran-interfaces=1', >> '--with-fortran=1', >> 'FC=mpiifort', >> 'PETSC_ARCH=arch-linux2-c-opt', >> ] >> configure.petsc_configure(configure_options) >> >> Now, I receive an error on hypre: >> >> forrtl: error (78): process killed (SIGTERM) >> Image PC Routine Line >> Source >> libHYPRE-2.18.2.s 00002B33CF465D3F for__signal_handl >> Unknown Unknown >> libpthread-2.17.s 00002B33D5BFD370 Unknown >> Unknown Unknown >> libpthread-2.17.s 00002B33D5BF96D3 pthread_cond_wait >> Unknown Unknown >> libiomp5.so 00002B33DBA14E07 Unknown >> Unknown Unknown >> libiomp5.so 00002B33DB98810C Unknown >> Unknown Unknown >> libiomp5.so 00002B33DB990578 Unknown >> Unknown Unknown >> libiomp5.so 00002B33DB9D9659 Unknown >> Unknown Unknown >> libiomp5.so 00002B33DB9D8C39 Unknown >> Unknown Unknown >> libiomp5.so 00002B33DB993BCE __kmpc_fork_call >> Unknown Unknown >> PIC_3D 00000000004071C0 Unknown >> Unknown Unknown >> PIC_3D 0000000000490299 Unknown >> Unknown Unknown >> PIC_3D 0000000000492C17 Unknown >> Unknown Unknown >> PIC_3D 000000000040562E Unknown >> Unknown Unknown >> libc-2.17.so 00002B33DC5BEB35 __libc_start_main >> Unknown Unknown >> PIC_3D 0000000000405539 Unknown >> Unknown Unknown >> >> Is it possible >> >> that I need to ask also to compile hypre with an option for 64bit indices? >> >> >> These configure options compile hypre with 64bit indices support. >> It should work just fine. Can you run a very small case of your code to >> confirm? >> >> >> Is it possible to instruct this inside Petsc configure? >> Alternatively, is it possible to use a different multigrid PC inside >> PETSc that accept 64bit indices? >> >> Thanks in advance >> >> Pierpaolo >> >> >> Il giorno 27 mag 2020, alle ore 11:26, Stefano Zampini < >> stefano.zampini at gmail.com> ha scritto: >> >> You need a version of PETSc compiled with 64bit indices, since the >> message indicates the number of dofs in this case is larger the INT_MAX >> 2501?3401?1601 = 13617947501 >> >> I also suggest you upgrade to a newer version, 3.8.3 is quite old as the >> error message reports >> >> Il giorno mer 27 mag 2020 alle ore 11:50 Pierpaolo Minelli < >> pierpaolo.minelli at cnr.it> ha scritto: >> >>> Hi, >>> >>> I am trying to solve a Poisson equation on this grid: >>> >>> Nx = 2501 >>> Ny = 3401 >>> Nz = 1601 >>> >>> I received this error: >>> >>> [0]PETSC ERROR: --------------------- Error Message >>> -------------------------------------------------------------- >>> [0]PETSC ERROR: Overflow in integer operation: >>> http://www.mcs.anl.gov/petsc/documentation/faq.html#64-bit-indices >>> [0]PETSC ERROR: Mesh of 2501 by 3401 by 1 (dof) is too large for 32 bit >>> indices >>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html >>> for trouble shooting. >>> [0]PETSC ERROR: Petsc Release Version 3.8.3, Dec, 09, 2017 >>> [0]PETSC ERROR: >>> /marconi_scratch/userexternal/pminelli/PIC3D/2500_3400_1600/./PIC_3D on a >>> arch-linux2-c-opt named r129c09s02 by pminelli Tu >>> e May 26 20:16:34 2020 >>> [0]PETSC ERROR: Configure options >>> --prefix=/cineca/prod/opt/libraries/petsc/3.8.3/intelmpi--2018--binary >>> CC=mpiicc FC=mpiifort CXX=mpiicpc >>> F77=mpiifort F90=mpiifort --with-debugging=0 >>> --with-blaslapack-dir=/cineca/prod/opt/compilers/intel/pe-xe-2018/binary/mkl >>> --with-fortran=1 >>> --with-fortran-interfaces=1 >>> --with-cmake-dir=/cineca/prod/opt/tools/cmake/3.5.2/none >>> --with-mpi-dir=/cineca/prod/opt/compilers/intel/pe-xe- >>> 2018/binary/impi/2018.4.274 --download-scalapack --download-mumps=yes >>> --download-hypre --download-superlu_dist --download-parmetis --downlo >>> ad-metis >>> [0]PETSC ERROR: #1 DMSetUp_DA_3D() line 218 in >>> /marconi/prod/build/libraries/petsc/3.8.3/intelmpi--2018--binary/BA_WORK/petsc-3.8.3/src/dm/ >>> impls/da/da3.c >>> [0]PETSC ERROR: #2 DMSetUp_DA() line 25 in >>> /marconi/prod/build/libraries/petsc/3.8.3/intelmpi--2018--binary/BA_WORK/petsc-3.8.3/src/dm/impl >>> s/da/dareg.c >>> [0]PETSC ERROR: #3 DMSetUp() line 720 in >>> /marconi/prod/build/libraries/petsc/3.8.3/intelmpi--2018--binary/BA_WORK/petsc-3.8.3/src/dm/interf >>> ace/dm.c >>> forrtl: error (76): Abort trap signal >>> >>> >>> I am on an HPC facility and after I loaded PETSC module, I have seen >>> that it is configured with INTEGER size = 32 >>> >>> I solve my problem with these options and it works perfectly with >>> smaller grids: >>> >>> -dm_mat_type hypre -pc_type hypre -pc_hypre_type boomeramg >>> -pc_hypre_boomeramg_relax_type_all SOR/Jacobi >>> -pc_hypre_boomeramg_coarsen_type PMIS -pc_hypre_boomeramg_interp_type FF1 >>> -ksp_type richardson >>> >>> Is it possible to overcome this if I ask them to install a version with >>> INTEGER SIZE = 64? >>> Alternatively, is it possible to overcome this using intel compiler >>> options? >>> >>> Thanks in advance >>> >>> Pierpaolo Minelli >> >> >> >> -- >> Stefano >> >> >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From luis.saturday at gmail.com Tue Jul 21 14:44:23 2020 From: luis.saturday at gmail.com (Alex Fleeter) Date: Tue, 21 Jul 2020 12:44:23 -0700 Subject: [petsc-users] pcsetup Message-ID: Hi: I want to ask under what circumstance will trigger a call for pc setup. I call KSPSolve to solve with the same Mat object with different entry values each time. I can see that the pc setup is only called at the beginning of the first solve. I tried to read the implementation, but quickly get lost... -------------- next part -------------- An HTML attachment was scrubbed... URL: From luis.saturday at gmail.com Tue Jul 21 17:11:22 2020 From: luis.saturday at gmail.com (Alex Fleeter) Date: Tue, 21 Jul 2020 15:11:22 -0700 Subject: [petsc-users] pcsetup In-Reply-To: References: Message-ID: Hi: I guess I could rephrase my question like this. I am using MATNEST. When I updated the matrix, I did assembly only for the sub-matrices. Then when I call the KSP solver, I can see that the pcsetup was called only in the first time the solver gets called. I tried to add MatAssembly for the big nest matrix as a second test. Then, when I called my solver, I could see that the pcsetup gets called. I assume that there is a state flag in the MatNest object that tells the solver if it needs to update the preconditioner. After an assembly routine gets called, this flag will be activated. Could you confirm that? The problem is that I am trying to implement a shell preconditioner, which gets the sub-matrices in the setup routine like the following MatNestGetSubMat(A, 0, 0, &newctx->submat[0]); MatNestGetSubMat(A, 0, 1, &newctx->submat[1]); MatNestGetSubMat(A, 1, 0, &newctx->submat[2]); MatNestGetSubMat(A, 1, 1, &newctx->submat[3]); KSPSetOperators(newctx->upper, newctx->submat[0], newctx->submat[0]); KSPSetOperators(newctx->lower, newctx->submat[3], newctx->submat[3]); In the Apply stage, I do PetscErrorCode blockpc_apply(PC pc, Vec x, Vec y) { std::cout<<"APPLY PC\n"; blockpc * newctx; PCShellGetContext(pc,(void**)&newctx); Vec x_v, x_p, y_v, y_p; VecGetSubVector(x, newctx->is[0], &x_v); VecGetSubVector(x, newctx->is[1], &x_p); VecGetSubVector(y, newctx->is[0], &y_v); VecGetSubVector(y, newctx->is[1], &y_p); KSPSolve( newctx -> upper, x_v, y_v ); KSPSolve( newctx -> lower, x_p, y_p ); VecRestoreSubVector(x, newctx->is[0], &x_v); VecRestoreSubVector(x, newctx->is[1], &x_p); VecRestoreSubVector(y, newctx->is[0], &y_v); VecRestoreSubVector(y, newctx->is[1], &y_p); return 0; } I want to know if I shall call an assembly for the big nest matrix so that the pcsetup can be called everytime the solver is dealing with a new matrix. Thanks, On Tue, Jul 21, 2020 at 12:44 PM Alex Fleeter wrote: > Hi: > > I want to ask under what circumstance will trigger a call for pc setup. > > I call KSPSolve to solve with the same Mat object with different entry > values each time. I can see that the pc setup is only called at the > beginning of the first solve. > > I tried to read the implementation, but quickly get lost... > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mfadams at lbl.gov Tue Jul 21 18:35:24 2020 From: mfadams at lbl.gov (Mark Adams) Date: Tue, 21 Jul 2020 19:35:24 -0400 Subject: [petsc-users] pcsetup In-Reply-To: References: Message-ID: KSPSetOperator tells the KSP that the PC should be resetup. On Tue, Jul 21, 2020 at 3:45 PM Alex Fleeter wrote: > Hi: > > I want to ask under what circumstance will trigger a call for pc setup. > > I call KSPSolve to solve with the same Mat object with different entry > values each time. I can see that the pc setup is only called at the > beginning of the first solve. > > I tried to read the implementation, but quickly get lost... > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at petsc.dev Tue Jul 21 19:21:40 2020 From: bsmith at petsc.dev (Barry Smith) Date: Tue, 21 Jul 2020 19:21:40 -0500 Subject: [petsc-users] SUPERLU_DIST in single precision In-Reply-To: <20200721155807.Horde.5ZYOzrQ7dmwNTQquK_6ebeS@webmail.mpcdf.mpg.de> References: <20200721155807.Horde.5ZYOzrQ7dmwNTQquK_6ebeS@webmail.mpcdf.mpg.de> Message-ID: Felix, What are your needs, do you want this for CPUs or for GPUs? Do you wish to run all your code in single precision or just the SuperLU_Dist solver while the rest of your code double? If you want to run everything on CPUs using single precision then adding the support is very easy, we can provide that for you any time. The other cases will require more thought. Barry > On Jul 21, 2020, at 8:58 AM, flw at rzg.mpg.de wrote: > > Dear PETSc support team, > some time ago you told me that you are planning on releasing a version that supports SUPERLU_DIST in single-precision soon. Can you tell me roughly what time frame you had in mind? > > Best regards, > Felix > From xsli at lbl.gov Tue Jul 21 19:33:03 2020 From: xsli at lbl.gov (Xiaoye S. Li) Date: Tue, 21 Jul 2020 17:33:03 -0700 Subject: [petsc-users] SUPERLU_DIST in single precision In-Reply-To: References: <20200721155807.Horde.5ZYOzrQ7dmwNTQquK_6ebeS@webmail.mpcdf.mpg.de> Message-ID: Barry, We have a macro-ized basefile to produced all 4 precisions for SuperLU_DIST. In the past, we didn't generate single precision code, since there was no request. Since now we started working on mixed precision algorithms. I generated single precision code a few weeks ago, and fixed some bugs on the way. A few days ago, I nailed down a final bug (hopefully) related to single precision SGEMM or STRSM BLAS problems on GPU (i.e., single precision cuBLAS error). All I know is that CPU code is working on Summit, but it has computation errors after I enable GPU. I plan to do some more investigation on this before asking Nvidia folks. I am busy this week, but should get to it next week. I suggest you wait for me to clear this, instead of duplicating the work. Sherry On Tue, Jul 21, 2020 at 5:22 PM Barry Smith wrote: > > Felix, > > What are your needs, do you want this for CPUs or for GPUs? Do you > wish to run all your code in single precision or just the SuperLU_Dist > solver while the rest of your code double? > > If you want to run everything on CPUs using single precision then > adding the support is very easy, we can provide that for you any time. The > other cases will require more thought. > > Barry > > > > On Jul 21, 2020, at 8:58 AM, flw at rzg.mpg.de wrote: > > > > Dear PETSc support team, > > some time ago you told me that you are planning on releasing a version > that supports SUPERLU_DIST in single-precision soon. Can you tell me > roughly what time frame you had in mind? > > > > Best regards, > > Felix > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at petsc.dev Tue Jul 21 19:38:09 2020 From: bsmith at petsc.dev (Barry Smith) Date: Tue, 21 Jul 2020 19:38:09 -0500 Subject: [petsc-users] Is PETSc using internet? In-Reply-To: References: <091AE85F-3B71-4D03-A63B-9FEE610668DF@gmail.com> <9442785D-77DA-4AC7-B6BA-DEDE52D984F8@gmail.com> Message-ID: <69D4EF5F-4C74-4497-A10D-10838A0E2AD1@petsc.dev> Here is one type of hang. $ petscmpiexec -n 2 ./ex1 then in another window $ ps | grep ex1 12015 ttys000 0:00.01 /bin/csh -f /Users/barrysmith/Src/petsc/lib/petsc/bin/petscmpiexec -n 2 ./ex1 12038 ttys000 0:00.01 mpiexec -n 2 ./ex1 12193 ttys001 0:00.00 grep ex1 ~/Src/petsc/src/snes/tests (barry/2020-07-12/factor-view-no-malloc *=) $ lldb -p 12038 (lldb) process attach --pid 12038 Process 12038 stopped * thread #1, queue = 'com.apple.main-thread', stop reason = signal SIGSTOP frame #0: 0x00007fff6dbe73d6 libsystem_kernel.dylib`poll + 10 libsystem_kernel.dylib`poll: -> 0x7fff6dbe73d6 <+10>: jae 0x7fff6dbe73e0 ; <+20> 0x7fff6dbe73d8 <+12>: movq %rax, %rdi 0x7fff6dbe73db <+15>: jmp 0x7fff6dbe222d ; cerror 0x7fff6dbe73e0 <+20>: retq Target 0: (mpiexec) stopped. Executable module set to "/Users/barrysmith/soft/clang-ifort/bin/mpiexec". Architecture set to: x86_64h-apple-macosx-. (lldb) bt * thread #1, queue = 'com.apple.main-thread', stop reason = signal SIGSTOP * frame #0: 0x00007fff6dbe73d6 libsystem_kernel.dylib`poll + 10 frame #1: 0x0000000106f35ff1 mpiexec`HYDT_dmxu_poll_wait_for_event + 737 frame #2: 0x0000000106f35897 mpiexec`HYDT_dmx_wait_for_event + 23 frame #3: 0x0000000106ef7208 mpiexec`HYD_pmci_wait_for_completion + 984 frame #4: 0x0000000106ecbe67 mpiexec`main + 8391 frame #5: 0x00007fff6da9fcc9 libdyld.dylib`start + 1 It is indicative of some "network" problem even though I am planning to run both processes on my Mac. It doesn't have anything to do with PETSc, but the network state of your machine (even when disconnected from the network) and MPICH Where do you get the hang if you run like above? Barry > On Jul 21, 2020, at 11:57 AM, Satish Balay via petsc-users wrote: > > you can run in the gdb to see if its hanging in a call to gethostbyname() or somewhere else. > > Satish > > On Tue, 21 Jul 2020, Eda Oktay wrote: > >> Dear Lawrence, >> >> The problem is not the error by the way, my program is waiting something >> without stopping and it is not giving error. It just does nothing. >> >> Does the problem is still because of hostname? >> >> Thanks! >> >> Eda >> >> On Tue, Jul 21, 2020, 1:16 PM Lawrence Mitchell wrote: >> >>> >>> >>>> On 21 Jul 2020, at 11:06, Eda Oktay wrote: >>>> >>>> Dear Lawrence, >>>> >>>> I am using MPICC but not Mac, Fedora 25. If it will still work, I will >>> try that. >>>> >>>> Thanks! >>> >>> It might be the case. When you observe the error, does "nslookup >>> localhost" take a long time? >>> >>> Lawrence >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at petsc.dev Tue Jul 21 19:51:36 2020 From: bsmith at petsc.dev (Barry Smith) Date: Tue, 21 Jul 2020 19:51:36 -0500 Subject: [petsc-users] Error on INTEGER SIZE using DMDACreate3d In-Reply-To: References: <88A21A58-CDCD-42C0-9A20-1A6C5CBCDF8B@cnr.it> Message-ID: <18FA0288-9131-4069-9080-60D6CC034DCD@petsc.dev> I assume PIC_3D is your code and you are using OpenMP? Are you calling hypre from inside your OpenMP parallelism? From inside PIC_3D? The SIGTERM is confusing to me. Are you using signals in any way? Usually a sigterm comes outside a process not process or thread crash. I assume for__signal_handl... is a Fortran signal handler forrtl: error (78): process killed (SIGTERM) Image PC Routine Line Source libHYPRE-2.18.2.s 00002B33CF465D3F for__signal_handl Unknown Unknown libpthread-2.17.s 00002B33D5BFD370 Unknown Unknown Unknown libpthread-2.17.s 00002B33D5BF96D3 pthread_cond_wait Unknown Unknown libiomp5.so 00002B33DBA14E07 Unknown Unknown Unknown libiomp5.so 00002B33DB98810C Unknown Unknown Unknown libiomp5.so 00002B33DB990578 Unknown Unknown Unknown libiomp5.so 00002B33DB9D9659 Unknown Unknown Unknown libiomp5.so 00002B33DB9D8C39 Unknown Unknown Unknown libiomp5.so 00002B33DB993BCE __kmpc_fork_call Unknown Unknown PIC_3D 00000000004071C0 Unknown Unknown Unknown PIC_3D 0000000000490299 Unknown Unknown Unknown PIC_3D 0000000000492C17 Unknown Unknown Unknown PIC_3D 000000000040562E Unknown Unknown Unknown libc-2.17.so 00002B33DC5BEB35 __libc_start_main Unknown Unknown PIC_3D 0000000000405539 Unknown Unknown Unknown > On Jul 21, 2020, at 6:32 AM, Pierpaolo Minelli wrote: > > Hi, > > I have asked to compile a Petsc Version updated and with 64bit indices. > Now I have Version 3.13.3 and these are the configure options used: > > #!/bin/python > if __name__ == '__main__': > import sys > import os > sys.path.insert(0, os.path.abspath('config')) > import configure > configure_options = [ > '--CC=mpiicc', > '--CXX=mpiicpc', > '--download-hypre', > '--download-metis', > '--download-mumps=yes', > '--download-parmetis', > '--download-scalapack', > '--download-superlu_dist', > '--known-64-bit-blas-indices', > '--prefix=/cineca/prod/opt/libraries/petsc/3.13.3_int64/intelmpi--2018--binary', > '--with-64-bit-indices=1', > '--with-blaslapack-dir=/cineca/prod/opt/compilers/intel/pe-xe-2018/binary/mkl', > '--with-cmake-dir=/cineca/prod/opt/tools/cmake/3.12.0/none', > '--with-debugging=0', > '--with-fortran-interfaces=1', > '--with-fortran=1', > 'FC=mpiifort', > 'PETSC_ARCH=arch-linux2-c-opt', > ] > configure.petsc_configure(configure_options) > > Now, I receive an error on hypre: > > forrtl: error (78): process killed (SIGTERM) > Image PC Routine Line Source > libHYPRE-2.18.2.s 00002B33CF465D3F for__signal_handl Unknown Unknown > libpthread-2.17.s 00002B33D5BFD370 Unknown Unknown Unknown > libpthread-2.17.s 00002B33D5BF96D3 pthread_cond_wait Unknown Unknown > libiomp5.so 00002B33DBA14E07 Unknown Unknown Unknown > libiomp5.so 00002B33DB98810C Unknown Unknown Unknown > libiomp5.so 00002B33DB990578 Unknown Unknown Unknown > libiomp5.so 00002B33DB9D9659 Unknown Unknown Unknown > libiomp5.so 00002B33DB9D8C39 Unknown Unknown Unknown > libiomp5.so 00002B33DB993BCE __kmpc_fork_call Unknown Unknown > PIC_3D 00000000004071C0 Unknown Unknown Unknown > PIC_3D 0000000000490299 Unknown Unknown Unknown > PIC_3D 0000000000492C17 Unknown Unknown Unknown > PIC_3D 000000000040562E Unknown Unknown Unknown > libc-2.17.so 00002B33DC5BEB35 __libc_start_main Unknown Unknown > PIC_3D 0000000000405539 Unknown Unknown Unknown > > Is it possible that I need to ask also to compile hypre with an option for 64bit indices? > Is it possible to instruct this inside Petsc configure? > Alternatively, is it possible to use a different multigrid PC inside PETSc that accept 64bit indices? > > Thanks in advance > > Pierpaolo > > >> Il giorno 27 mag 2020, alle ore 11:26, Stefano Zampini > ha scritto: >> >> You need a version of PETSc compiled with 64bit indices, since the message indicates the number of dofs in this case is larger the INT_MAX >> 2501?3401?1601 = 13617947501 >> >> I also suggest you upgrade to a newer version, 3.8.3 is quite old as the error message reports >> >> Il giorno mer 27 mag 2020 alle ore 11:50 Pierpaolo Minelli > ha scritto: >> Hi, >> >> I am trying to solve a Poisson equation on this grid: >> >> Nx = 2501 >> Ny = 3401 >> Nz = 1601 >> >> I received this error: >> >> [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- >> [0]PETSC ERROR: Overflow in integer operation: http://www.mcs.anl.gov/petsc/documentation/faq.html#64-bit-indices >> [0]PETSC ERROR: Mesh of 2501 by 3401 by 1 (dof) is too large for 32 bit indices >> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. >> [0]PETSC ERROR: Petsc Release Version 3.8.3, Dec, 09, 2017 >> [0]PETSC ERROR: /marconi_scratch/userexternal/pminelli/PIC3D/2500_3400_1600/./PIC_3D on a arch-linux2-c-opt named r129c09s02 by pminelli Tu >> e May 26 20:16:34 2020 >> [0]PETSC ERROR: Configure options --prefix=/cineca/prod/opt/libraries/petsc/3.8.3/intelmpi--2018--binary CC=mpiicc FC=mpiifort CXX=mpiicpc >> F77=mpiifort F90=mpiifort --with-debugging=0 --with-blaslapack-dir=/cineca/prod/opt/compilers/intel/pe-xe-2018/binary/mkl --with-fortran=1 >> --with-fortran-interfaces=1 --with-cmake-dir=/cineca/prod/opt/tools/cmake/3.5.2/none --with-mpi-dir=/cineca/prod/opt/compilers/intel/pe-xe- >> 2018/binary/impi/2018.4.274 --download-scalapack --download-mumps=yes --download-hypre --download-superlu_dist --download-parmetis --downlo >> ad-metis >> [0]PETSC ERROR: #1 DMSetUp_DA_3D() line 218 in /marconi/prod/build/libraries/petsc/3.8.3/intelmpi--2018--binary/BA_WORK/petsc-3.8.3/src/dm/ >> impls/da/da3.c >> [0]PETSC ERROR: #2 DMSetUp_DA() line 25 in /marconi/prod/build/libraries/petsc/3.8.3/intelmpi--2018--binary/BA_WORK/petsc-3.8.3/src/dm/impl >> s/da/dareg.c >> [0]PETSC ERROR: #3 DMSetUp() line 720 in /marconi/prod/build/libraries/petsc/3.8.3/intelmpi--2018--binary/BA_WORK/petsc-3.8.3/src/dm/interf >> ace/dm.c >> forrtl: error (76): Abort trap signal >> >> >> I am on an HPC facility and after I loaded PETSC module, I have seen that it is configured with INTEGER size = 32 >> >> I solve my problem with these options and it works perfectly with smaller grids: >> >> -dm_mat_type hypre -pc_type hypre -pc_hypre_type boomeramg -pc_hypre_boomeramg_relax_type_all SOR/Jacobi -pc_hypre_boomeramg_coarsen_type PMIS -pc_hypre_boomeramg_interp_type FF1 -ksp_type richardson >> >> Is it possible to overcome this if I ask them to install a version with INTEGER SIZE = 64? >> Alternatively, is it possible to overcome this using intel compiler options? >> >> Thanks in advance >> >> Pierpaolo Minelli >> >> >> -- >> Stefano > -------------- next part -------------- An HTML attachment was scrubbed... URL: From luis.saturday at gmail.com Tue Jul 21 20:16:07 2020 From: luis.saturday at gmail.com (Alex Fleeter) Date: Tue, 21 Jul 2020 18:16:07 -0700 Subject: [petsc-users] pcsetup In-Reply-To: References: Message-ID: If I call MatAssembly for individual sub-matrices, MatAssemblyBegin(subA[ii], MAT_FINAL_ASSEMBLY); MatAssemblyEnd(subA[ii], MAT_FINAL_ASSEMBLY); then call KSPSetOperators, the PC setup is not called. I have to call the assembly for the whole nest matrix MatAssemblyBegin(A, MAT_FINAL_ASSEMBLY); MatAssemblyEnd(A, MAT_FINAL_ASSEMBLY); to trigger the setup of PC. I found some comments in matnest.c , line 474-478: "Note: split assembly will fail if the same block appears more than once (even indirectly through a nested 475: * sub-block). This could be fixed by adding a flag to Mat so that there was a way to check if a Mat was476: * already performing an assembly, but the result would by more complicated and appears to offer less477: * potential for diagnostics and correctness checking. Split assembly should be fixed once there is an478: * interface for libraries to make asynchronous progress in "user-defined non-blocking collectives"." I guess the comment suggests calling matassembly for the big nest matrix after having setted values for individual sub-matrices, instead of calling matassembly for individual submatrices. Can you confirm that? On Tue, Jul 21, 2020 at 4:35 PM Mark Adams wrote: > KSPSetOperator tells the KSP that the PC should be resetup. > > On Tue, Jul 21, 2020 at 3:45 PM Alex Fleeter > wrote: > >> Hi: >> >> I want to ask under what circumstance will trigger a call for pc setup. >> >> I call KSPSolve to solve with the same Mat object with different entry >> values each time. I can see that the pc setup is only called at the >> beginning of the first solve. >> >> I tried to read the implementation, but quickly get lost... >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at petsc.dev Tue Jul 21 20:21:18 2020 From: bsmith at petsc.dev (Barry Smith) Date: Tue, 21 Jul 2020 20:21:18 -0500 Subject: [petsc-users] pcsetup In-Reply-To: References: Message-ID: <1C928033-973D-4166-8938-037910B4B641@petsc.dev> > On Jul 21, 2020, at 5:11 PM, Alex Fleeter wrote: > > Hi: > > I guess I could rephrase my question like this. I am using MATNEST. > > When I updated the matrix, I did assembly only for the sub-matrices. Then when I call the KSP solver, I can see that the pcsetup was called only in the first time the solver gets called. > > I tried to add MatAssembly for the big nest matrix as a second test. Then, when I called my solver, I could see that the pcsetup gets called. > > I assume that there is a state flag in the MatNest object that tells the solver if it needs to update the preconditioner. After an assembly routine gets called, this flag will be activated. Could you confirm that? PCSetUp() has ierr = PetscObjectStateGet((PetscObject)pc->pmat,&matstate);CHKERRQ(ierr); ierr = MatGetNonzeroState(pc->pmat,&matnonzerostate);CHKERRQ(ierr); if (!pc->setupcalled) { ierr = PetscInfo(pc,"Setting up PC for first time\n");CHKERRQ(ierr); pc->flag = DIFFERENT_NONZERO_PATTERN; } else if (matstate == pc->matstate) { ierr = PetscInfo(pc,"Leaving PC with identical preconditioner since operator is unchanged\n");CHKERRQ(ierr); PetscFunctionReturn(0); so it rebuilds the preconditioner if the PetscObjectState of the matrix increased since the last application of the preconditioner. Calling MatAssemblyBegin/End on a matrix always increases the matrix state (even if you have not changed the matrix entries). This is why "MatAssembly for the big nest matrix as a second test. Then, when I called my solver, I could see that the pcsetup gets called." MatNest however is fundamentally broken because if you change entries in the sub-matrices you used to build the MatNest, the MatNest state is not increased hence any call to PCSetUp() with the MatNest as an argument does not trigger a new PCSetUp(). The author of MatNest knows about this flaw but has not bothered to fix it. Thus every time you change values in the submatrices you must also call MatAssemblyBegin/End on the nest matrix (as you discovered) so the preconditioner THAT DIRECTLY USES the MatNest knows it needs to rebuild. Now I don't understand completely the code you have below and why or if there is a problem. The two KSP newctx->upper and newctx->lower do not know about the MatNest or use the MatNest state, they only know about the sub matrices and the sub matrix states. So if you change entries in the sub matrices then KSPSolve(newctx->upper) will trigger rebuilding the preconditioner inside it. That is you get the behavior you expect. In other words blockpc_apply() will rebuild the two interior PCs each time you change the sub matrices. In other words for your custom PCShell you do not need to call MatAssemblyBegin/End on the nest matrix since the two interior PCs know directly about changes to the sub matrices, since their input is the sub matrices not the nest matrix. Do you not get them rebuilding the preconditioner? If you run in the debugger and put a break point in PCSetUp() you should see those two interior PCs getting rebuilt exactly when they should be. I hope this is not confusing: a KSP/PC that takes a MatNest argument will not rebuild the preconditioner that depends on the MatNest argument if you don't call MatAssemblyBegin/End on the MatNest. But any KSP/PC that depend on individual submatrices will get rebuilt correctly since they use the state of the individual sub matrices. Barry > > The problem is that I am trying to implement a shell preconditioner, which gets the sub-matrices in the setup routine like the following > MatNestGetSubMat(A, 0, 0, &newctx->submat[0]); > MatNestGetSubMat(A, 0, 1, &newctx->submat[1]); > MatNestGetSubMat(A, 1, 0, &newctx->submat[2]); > MatNestGetSubMat(A, 1, 1, &newctx->submat[3]); > > KSPSetOperators(newctx->upper, newctx->submat[0], newctx->submat[0]); > KSPSetOperators(newctx->lower, newctx->submat[3], newctx->submat[3]); > > In the Apply stage, I do > > PetscErrorCode blockpc_apply(PC pc, Vec x, Vec y) > { > std::cout<<"APPLY PC\n"; > blockpc * newctx; > PCShellGetContext(pc,(void**)&newctx); > > Vec x_v, x_p, y_v, y_p; > VecGetSubVector(x, newctx->is[0], &x_v); > VecGetSubVector(x, newctx->is[1], &x_p); > VecGetSubVector(y, newctx->is[0], &y_v); > VecGetSubVector(y, newctx->is[1], &y_p); > > KSPSolve( newctx -> upper, x_v, y_v ); > KSPSolve( newctx -> lower, x_p, y_p ); > > VecRestoreSubVector(x, newctx->is[0], &x_v); > VecRestoreSubVector(x, newctx->is[1], &x_p); > VecRestoreSubVector(y, newctx->is[0], &y_v); > VecRestoreSubVector(y, newctx->is[1], &y_p); > > return 0; > } > > I want to know if I shall call an assembly for the big nest matrix so that the pcsetup can be called everytime the solver is dealing with a new matrix. > > Thanks, > > > > > > > On Tue, Jul 21, 2020 at 12:44 PM Alex Fleeter > wrote: > Hi: > > I want to ask under what circumstance will trigger a call for pc setup. > > I call KSPSolve to solve with the same Mat object with different entry values each time. I can see that the pc setup is only called at the beginning of the first solve. > > I tried to read the implementation, but quickly get lost... -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at petsc.dev Tue Jul 21 20:28:34 2020 From: bsmith at petsc.dev (Barry Smith) Date: Tue, 21 Jul 2020 20:28:34 -0500 Subject: [petsc-users] pcsetup In-Reply-To: References: Message-ID: Alex, See my long email. The thing with nested preconditioners (like yours) is they have multiple PCSetUps(), possibly outer ones and inner ones. Each of these PC independently triggers a rebuild based on its own state, hence the inner PCSetUps will be triggered when you change the inner matrices (even when you don't call MatAssemblyBegin/End() on the MatNest. In your case since the preconditioner seems to depend only on the two sub matrices the fact that PCSetUp() is not called on the MatNest is harmless, since that setup wouldn't do anything anyways. For Schur complement based PCFieldsplits however the result maybe be wrong unless you call MatAssemblyBegin/End on the nest matrix since the outer setup actually does something Barry > On Jul 21, 2020, at 8:16 PM, Alex Fleeter wrote: > > If I call MatAssembly for individual sub-matrices, > MatAssemblyBegin(subA[ii], MAT_FINAL_ASSEMBLY); > MatAssemblyEnd(subA[ii], MAT_FINAL_ASSEMBLY); > > then call KSPSetOperators, the PC setup is not called. > > I have to call the assembly for the whole nest matrix > MatAssemblyBegin(A, MAT_FINAL_ASSEMBLY); > MatAssemblyEnd(A, MAT_FINAL_ASSEMBLY); > to trigger the setup of PC. > > I found some comments in matnest.c , line 474-478: "Note: split assembly will fail if the same block appears more than once (even indirectly through a nested > 475: <> * sub-block). This could be fixed by adding a flag to Mat so that there was a way to check if a Mat was > 476: <> * already performing an assembly, but the result would by more complicated and appears to offer less > 477: <> * potential for diagnostics and correctness checking. Split assembly should be fixed once there is an > 478: <> * interface for libraries to make asynchronous progress in "user-defined non-blocking collectives"." > I guess the comment suggests calling matassembly for the big nest matrix after having setted values for individual sub-matrices, instead of calling matassembly for individual submatrices. > > Can you confirm that? > > > On Tue, Jul 21, 2020 at 4:35 PM Mark Adams > wrote: > KSPSetOperator tells the KSP that the PC should be resetup. > > On Tue, Jul 21, 2020 at 3:45 PM Alex Fleeter > wrote: > Hi: > > I want to ask under what circumstance will trigger a call for pc setup. > > I call KSPSolve to solve with the same Mat object with different entry values each time. I can see that the pc setup is only called at the beginning of the first solve. > > I tried to read the implementation, but quickly get lost... -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at petsc.dev Tue Jul 21 20:30:43 2020 From: bsmith at petsc.dev (Barry Smith) Date: Tue, 21 Jul 2020 20:30:43 -0500 Subject: [petsc-users] SUPERLU_DIST in single precision In-Reply-To: References: <20200721155807.Horde.5ZYOzrQ7dmwNTQquK_6ebeS@webmail.mpcdf.mpg.de> Message-ID: <92B4C046-9B8D-4287-BD31-C0BCEF379548@petsc.dev> Thanks Sherry. But the CPU version runs correctly right so if Felix is only interested in CPUs then I could give it to him now? Barry > On Jul 21, 2020, at 7:33 PM, Xiaoye S. Li wrote: > > Barry, > > We have a macro-ized basefile to produced all 4 precisions for SuperLU_DIST. In the past, we didn't generate single precision code, since there was no request. > > Since now we started working on mixed precision algorithms. I generated single precision code a few weeks ago, and fixed some bugs on the way. A few days ago, I nailed down a final bug (hopefully) related to single precision SGEMM or STRSM BLAS problems on GPU (i.e., single precision cuBLAS error). All I know is that CPU code is working on Summit, but it has computation errors after I enable GPU. I plan to do some more investigation on this before asking Nvidia folks. > > I am busy this week, but should get to it next week. I suggest you wait for me to clear this, instead of duplicating the work. > > Sherry > > > On Tue, Jul 21, 2020 at 5:22 PM Barry Smith > wrote: > > Felix, > > What are your needs, do you want this for CPUs or for GPUs? Do you wish to run all your code in single precision or just the SuperLU_Dist solver while the rest of your code double? > > If you want to run everything on CPUs using single precision then adding the support is very easy, we can provide that for you any time. The other cases will require more thought. > > Barry > > > > On Jul 21, 2020, at 8:58 AM, flw at rzg.mpg.de wrote: > > > > Dear PETSc support team, > > some time ago you told me that you are planning on releasing a version that supports SUPERLU_DIST in single-precision soon. Can you tell me roughly what time frame you had in mind? > > > > Best regards, > > Felix > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From xsli at lbl.gov Tue Jul 21 20:36:16 2020 From: xsli at lbl.gov (Xiaoye S. Li) Date: Tue, 21 Jul 2020 18:36:16 -0700 Subject: [petsc-users] SUPERLU_DIST in single precision In-Reply-To: <92B4C046-9B8D-4287-BD31-C0BCEF379548@petsc.dev> References: <20200721155807.Horde.5ZYOzrQ7dmwNTQquK_6ebeS@webmail.mpcdf.mpg.de> <92B4C046-9B8D-4287-BD31-C0BCEF379548@petsc.dev> Message-ID: Yes. We will wait for his reply. -Sherry On Tue, Jul 21, 2020 at 6:30 PM Barry Smith wrote: > > Thanks Sherry. But the CPU version runs correctly right so if Felix is > only interested in CPUs then I could give it to him now? > > Barry > > > On Jul 21, 2020, at 7:33 PM, Xiaoye S. Li wrote: > > Barry, > > We have a macro-ized basefile to produced all 4 precisions for > SuperLU_DIST. In the past, we didn't generate single precision code, since > there was no request. > > Since now we started working on mixed precision algorithms. I generated > single precision code a few weeks ago, and fixed some bugs on the way. A > few days ago, I nailed down a final bug (hopefully) related to single > precision SGEMM or STRSM BLAS problems on GPU (i.e., single precision > cuBLAS error). All I know is that CPU code is working on Summit, but it > has computation errors after I enable GPU. I plan to do some more > investigation on this before asking Nvidia folks. > > I am busy this week, but should get to it next week. I suggest you wait > for me to clear this, instead of duplicating the work. > > Sherry > > > On Tue, Jul 21, 2020 at 5:22 PM Barry Smith wrote: > >> >> Felix, >> >> What are your needs, do you want this for CPUs or for GPUs? Do you >> wish to run all your code in single precision or just the SuperLU_Dist >> solver while the rest of your code double? >> >> If you want to run everything on CPUs using single precision then >> adding the support is very easy, we can provide that for you any time. The >> other cases will require more thought. >> >> Barry >> >> >> > On Jul 21, 2020, at 8:58 AM, flw at rzg.mpg.de wrote: >> > >> > Dear PETSc support team, >> > some time ago you told me that you are planning on releasing a version >> that supports SUPERLU_DIST in single-precision soon. Can you tell me >> roughly what time frame you had in mind? >> > >> > Best regards, >> > Felix >> > >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From adantra at gmail.com Wed Jul 22 07:03:44 2020 From: adantra at gmail.com (Adolfo Rodriguez) Date: Wed, 22 Jul 2020 07:03:44 -0500 Subject: [petsc-users] Matrix assembly in SNES problem Message-ID: I am trying to replace the non-linear solver in a flow simulation problem where the matrix sparsity can change during the iterations. I tried successfully to create the matrix within the FormJacobian function but I have a memory leak which I don't know how to fix. My FormJacobian function is something like this: .. FormJacobian( ..){ matrixconstruction(); // returns csr matrix in c++ regular vectors MatSeqAIJSetPreallocationCSR ...; //loop over the entries and MatSetValues; MatAssemblyBegin(); MatAssemblyEnd(); return 0; } This approach works for small problems but, when dealing with large ones I observe a memory leak. I am pretty sure that It is related to the fact that I am creating new matrices every time FormJacobian is called but I don't know how to destroy it between iterations. Any suggestions? Regards, Adolfo -------------- next part -------------- An HTML attachment was scrubbed... URL: From mfadams at lbl.gov Wed Jul 22 07:45:31 2020 From: mfadams at lbl.gov (Mark Adams) Date: Wed, 22 Jul 2020 08:45:31 -0400 Subject: [petsc-users] Matrix assembly in SNES problem In-Reply-To: References: Message-ID: On Wed, Jul 22, 2020 at 8:05 AM Adolfo Rodriguez wrote: > I am trying to replace the non-linear solver in a flow simulation problem > where the matrix sparsity can change during the iterations. I tried > successfully to create the matrix within the FormJacobian function but I > have a memory leak which I don't know how to fix. > > My FormJacobian function is something like this: > > .. FormJacobian( ..){ > matrixconstruction(); // returns csr matrix in c++ regular vectors > MatSeqAIJSetPreallocationCSR ...; > //loop over the entries and > MatSetValues; > MatAssemblyBegin(); > MatAssemblyEnd(); > return 0; > } > > This approach works for small problems but, when dealing with large ones I > observe a memory leak. > How are you detecting a memory leak? It's odd that you would not see it on a small problem. You can run with -malloc_debug to get a listing of PETSc memory that was not freed. > I am pretty sure that It is related to the fact that I am creating new > matrices every time FormJacobian is called but I don't know how to destroy > it between iterations. > You don't create the matrix in FormJacobian, do you? Do your changes (eg, AMR) before or after the time step (you can do this with a TSSetPostStep and TSSetPreStep), then destroy the old matrix and vectors. > > Any suggestions? > > Regards, > > Adolfo > -------------- next part -------------- An HTML attachment was scrubbed... URL: From flw at rzg.mpg.de Wed Jul 22 08:04:03 2020 From: flw at rzg.mpg.de (flw at rzg.mpg.de) Date: Wed, 22 Jul 2020 15:04:03 +0200 Subject: [petsc-users] SUPERLU_DIST in single precision In-Reply-To: References: <20200721155807.Horde.5ZYOzrQ7dmwNTQquK_6ebeS@webmail.mpcdf.mpg.de> Message-ID: <20200722150403.Horde.NO015I2GI2E-524J-SXqwui@webmail.mpcdf.mpg.de> Hi Barry, for now I just want to run everything in single on CPUs only with SUPERLU_DIST. Maybe we will also incorporate GPUs in the future, but there are no immediate plans yet. So if you could provide the support, that would be awesome. Best regards, Felix Zitat von Barry Smith : > Felix, > > What are your needs, do you want this for CPUs or for GPUs? Do > you wish to run all your code in single precision or just the > SuperLU_Dist solver while the rest of your code double? > > If you want to run everything on CPUs using single precision > then adding the support is very easy, we can provide that for you > any time. The other cases will require more thought. > > Barry > > >> On Jul 21, 2020, at 8:58 AM, flw at rzg.mpg.de wrote: >> >> Dear PETSc support team, >> some time ago you told me that you are planning on releasing a >> version that supports SUPERLU_DIST in single-precision soon. Can >> you tell me roughly what time frame you had in mind? >> >> Best regards, >> Felix >> From adantra at gmail.com Wed Jul 22 08:08:57 2020 From: adantra at gmail.com (Adolfo Rodriguez) Date: Wed, 22 Jul 2020 08:08:57 -0500 Subject: [petsc-users] Matrix assembly in SNES problem In-Reply-To: References: Message-ID: I am on windows. I am detecting the memory leak through the windows resource monitor, I can see the memory utilization going up during the iteration. Memory utilization goes down after the destruction of SNES, the solution vector and residual, but ends higher than at the beginning. Within FormJacobian I call a function that creates the Jacobian in c++ objects which I use to popular the PETSc Jacobian matrix as indicated above. I don't create the Jacobian beforehand. I am not using PETSc for the time-stepping part of the problem so I am not sure if I can use TSSetPostStep and TSSetPreStep. Hopefully, this clarifies my question a little bit. Adolfo On Wed, Jul 22, 2020 at 7:45 AM Mark Adams wrote: > > > On Wed, Jul 22, 2020 at 8:05 AM Adolfo Rodriguez > wrote: > >> I am trying to replace the non-linear solver in a flow simulation problem >> where the matrix sparsity can change during the iterations. I tried >> successfully to create the matrix within the FormJacobian function but I >> have a memory leak which I don't know how to fix. >> >> My FormJacobian function is something like this: >> >> .. FormJacobian( ..){ >> matrixconstruction(); // returns csr matrix in c++ regular vectors >> MatSeqAIJSetPreallocationCSR ...; >> //loop over the entries and >> MatSetValues; >> MatAssemblyBegin(); >> MatAssemblyEnd(); >> return 0; >> } >> >> This approach works for small problems but, when dealing with large ones >> I observe a memory leak. >> > > How are you detecting a memory leak? It's odd that you would not see it on > a small problem. You can run with -malloc_debug to get a listing of PETSc > memory that was not freed. > > >> I am pretty sure that It is related to the fact that I am creating new >> matrices every time FormJacobian is called but I don't know how to destroy >> it between iterations. >> > > You don't create the matrix in FormJacobian, do you? > Do your changes (eg, AMR) before or after the time step (you can do this > with a TSSetPostStep and TSSetPreStep), then destroy the old matrix and > vectors. > > >> >> Any suggestions? >> >> Regards, >> >> Adolfo >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From hongzhang at anl.gov Wed Jul 22 08:32:44 2020 From: hongzhang at anl.gov (Zhang, Hong) Date: Wed, 22 Jul 2020 13:32:44 +0000 Subject: [petsc-users] Matrix assembly in SNES problem In-Reply-To: References: Message-ID: <7F78E62D-9A20-41FF-976B-2A467D316FFB@anl.gov> > On Jul 22, 2020, at 7:03 AM, Adolfo Rodriguez wrote: > > I am trying to replace the non-linear solver in a flow simulation problem where the matrix sparsity can change during the iterations. I tried successfully to create the matrix within the FormJacobian function but I have a memory leak which I don't know how to fix. > > My FormJacobian function is something like this: > > .. FormJacobian( ..){ > matrixconstruction(); // returns csr matrix in c++ regular vectors > MatSeqAIJSetPreallocationCSR ...; > //loop over the entries and > MatSetValues; > MatAssemblyBegin(); > MatAssemblyEnd(); > return 0; > } Yes, the leak is caused by the new matrix created every time FormJacobian is called. The working matrix for SNES needs to be allocated only once and passed to SNES by using SNESSetJacobian(SNES snes,Mat Amat,Mat Pmat,PetscErrorCode (*J)(SNES,Vec,Mat,Mat,void*),void *ctx) In your FormJacobian(SNES snes,Vec x,Mat A,Mat B,void *ctx), you just need to copy the csr matrix to the matrix A, which has already been allocated. The working matrix should be destroyed after SNESSolve(). Hong (Mr.) > This approach works for small problems but, when dealing with large ones I observe a memory leak. I am pretty sure that It is related to the fact that I am creating new matrices every time FormJacobian is called but I don't know how to destroy it between iterations. > > Any suggestions? > > Regards, > > Adolfo From knepley at gmail.com Wed Jul 22 09:30:56 2020 From: knepley at gmail.com (Matthew Knepley) Date: Wed, 22 Jul 2020 10:30:56 -0400 Subject: [petsc-users] Matrix assembly in SNES problem In-Reply-To: <7F78E62D-9A20-41FF-976B-2A467D316FFB@anl.gov> References: <7F78E62D-9A20-41FF-976B-2A467D316FFB@anl.gov> Message-ID: On Wed, Jul 22, 2020 at 9:32 AM Zhang, Hong via petsc-users < petsc-users at mcs.anl.gov> wrote: > > > > On Jul 22, 2020, at 7:03 AM, Adolfo Rodriguez wrote: > > > > I am trying to replace the non-linear solver in a flow simulation > problem where the matrix sparsity can change during the iterations. I tried > successfully to create the matrix within the FormJacobian function but I > have a memory leak which I don't know how to fix. > > > > My FormJacobian function is something like this: > > > > .. FormJacobian( ..){ > > matrixconstruction(); // returns csr matrix in c++ regular vectors > > MatSeqAIJSetPreallocationCSR ...; > > //loop over the entries and > > MatSetValues; > > MatAssemblyBegin(); > > MatAssemblyEnd(); > > return 0; > > } > > Yes, the leak is caused by the new matrix created every time FormJacobian > is called. The working matrix for SNES needs to be allocated only once and > passed to SNES by using > > SNESSetJacobian(SNES snes,Mat Amat,Mat Pmat,PetscErrorCode > (*J)(SNES,Vec,Mat,Mat,void*),void *ctx) > > In your FormJacobian(SNES snes,Vec x,Mat A,Mat B,void *ctx), you just need > to copy the csr matrix to the matrix A, which has already been allocated. > > The working matrix should be destroyed after SNESSolve(). > Hong, what is the best way to wipe out the existing structure, preallocate, and set new values? Thanks, Matt > Hong (Mr.) > > > > This approach works for small problems but, when dealing with large ones > I observe a memory leak. I am pretty sure that It is related to the fact > that I am creating new matrices every time FormJacobian is called but I > don't know how to destroy it between iterations. > > > > Any suggestions? > > > > Regards, > > > > Adolfo > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From adantra at gmail.com Wed Jul 22 09:35:56 2020 From: adantra at gmail.com (Adolfo Rodriguez) Date: Wed, 22 Jul 2020 09:35:56 -0500 Subject: [petsc-users] Matrix assembly in SNES problem In-Reply-To: <7F78E62D-9A20-41FF-976B-2A467D316FFB@anl.gov> References: <7F78E62D-9A20-41FF-976B-2A467D316FFB@anl.gov> Message-ID: Hong, Thanks for your email. My question now is how to handle situations where the matrix structure changes during the iteration process? Adolfo On Wed, Jul 22, 2020 at 8:32 AM Zhang, Hong wrote: > > > > On Jul 22, 2020, at 7:03 AM, Adolfo Rodriguez wrote: > > > > I am trying to replace the non-linear solver in a flow simulation > problem where the matrix sparsity can change during the iterations. I tried > successfully to create the matrix within the FormJacobian function but I > have a memory leak which I don't know how to fix. > > > > My FormJacobian function is something like this: > > > > .. FormJacobian( ..){ > > matrixconstruction(); // returns csr matrix in c++ regular vectors > > MatSeqAIJSetPreallocationCSR ...; > > //loop over the entries and > > MatSetValues; > > MatAssemblyBegin(); > > MatAssemblyEnd(); > > return 0; > > } > > Yes, the leak is caused by the new matrix created every time FormJacobian > is called. The working matrix for SNES needs to be allocated only once and > passed to SNES by using > > SNESSetJacobian(SNES snes,Mat Amat,Mat Pmat,PetscErrorCode > (*J)(SNES,Vec,Mat,Mat,void*),void *ctx) > > In your FormJacobian(SNES snes,Vec x,Mat A,Mat B,void *ctx), you just need > to copy the csr matrix to the matrix A, which has already been allocated. > > The working matrix should be destroyed after SNESSolve(). > > Hong (Mr.) > > > > This approach works for small problems but, when dealing with large ones > I observe a memory leak. I am pretty sure that It is related to the fact > that I am creating new matrices every time FormJacobian is called but I > don't know how to destroy it between iterations. > > > > Any suggestions? > > > > Regards, > > > > Adolfo > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at petsc.dev Wed Jul 22 09:38:18 2020 From: bsmith at petsc.dev (Barry Smith) Date: Wed, 22 Jul 2020 09:38:18 -0500 Subject: [petsc-users] Matrix assembly in SNES problem In-Reply-To: References: Message-ID: > On Jul 22, 2020, at 7:03 AM, Adolfo Rodriguez wrote: > > I am trying to replace the non-linear solver in a flow simulation problem where the matrix sparsity can change during the iterations. I tried successfully to create the matrix within the FormJacobian function but I have a memory leak which I don't know how to fix. > > My FormJacobian function is something like this: > > .. FormJacobian( ..){ > matrixconstruction(); // returns csr matrix in c++ regular vectors What do you mean by c++ regular vectors? Do you mean basic arrays you obtain with New or malloc() or do you mean some std: vector objects? I am guessing just arrays obtained with new. > MatSeqAIJSetPreallocationCSR ...; > //loop over the entries and > MatSetValues; > MatAssemblyBegin(); > MatAssemblyEnd(); > return 0; > } 1) side note The routine MatSeqAIJSetPreallocationCSR() which takes ii, jj,vv arguments not only preallocates the memory but also fills up the matrix so you don't need the part of the code > //loop over the entries and > MatSetValues; > MatAssemblyBegin(); > MatAssemblyEnd(); (Yes the function name is confusing) You should remove it and still get the same results but faster. 2) the memory leak could be in your code or in PETSc. Are you freeing the arrays you get from matrixconstruction? You can run a smallish problem with -malloc_debug and at PetscFinalize() it will print all the memory PETSc has allocated and not freed, and where in the code it was allocated. If the leak is in PETSc or from not freeing PETSc objects then this should find the problem. If the leak is in your C++ allocations you will need to use the Microsoft tools. Good luck, Barry > > This approach works for small problems but, when dealing with large ones I observe a memory leak. I am pretty sure that It is related to the fact that I am creating new matrices every time FormJacobian is called but I don't know how to destroy it between iterations. > > Any suggestions? > > Regards, > > Adolfo From bsmith at petsc.dev Wed Jul 22 09:43:14 2020 From: bsmith at petsc.dev (Barry Smith) Date: Wed, 22 Jul 2020 09:43:14 -0500 Subject: [petsc-users] Matrix assembly in SNES problem In-Reply-To: References: <7F78E62D-9A20-41FF-976B-2A467D316FFB@anl.gov> Message-ID: <17A9A3D3-2CE2-4D98-9619-31D64831AA2F@petsc.dev> > On Jul 22, 2020, at 9:35 AM, Adolfo Rodriguez wrote: > > Hong, > > Thanks for your email. My question now is how to handle situations where the matrix structure changes during the iteration process? The routine you are calling MatSeqAIJSetPreallocationCSR() is fine. It allows you to put in new CSR data structures with different nonzero patters. See my previous email. Barry > Adolfo > > On Wed, Jul 22, 2020 at 8:32 AM Zhang, Hong > wrote: > > > > On Jul 22, 2020, at 7:03 AM, Adolfo Rodriguez > wrote: > > > > I am trying to replace the non-linear solver in a flow simulation problem where the matrix sparsity can change during the iterations. I tried successfully to create the matrix within the FormJacobian function but I have a memory leak which I don't know how to fix. > > > > My FormJacobian function is something like this: > > > > .. FormJacobian( ..){ > > matrixconstruction(); // returns csr matrix in c++ regular vectors > > MatSeqAIJSetPreallocationCSR ...; > > //loop over the entries and > > MatSetValues; > > MatAssemblyBegin(); > > MatAssemblyEnd(); > > return 0; > > } > > Yes, the leak is caused by the new matrix created every time FormJacobian is called. The working matrix for SNES needs to be allocated only once and passed to SNES by using > > SNESSetJacobian(SNES snes,Mat Amat,Mat Pmat,PetscErrorCode (*J)(SNES,Vec,Mat,Mat,void*),void *ctx) > > In your FormJacobian(SNES snes,Vec x,Mat A,Mat B,void *ctx), you just need to copy the csr matrix to the matrix A, which has already been allocated. > > The working matrix should be destroyed after SNESSolve(). > > Hong (Mr.) > > > > This approach works for small problems but, when dealing with large ones I observe a memory leak. I am pretty sure that It is related to the fact that I am creating new matrices every time FormJacobian is called but I don't know how to destroy it between iterations. > > > > Any suggestions? > > > > Regards, > > > > Adolfo > -------------- next part -------------- An HTML attachment was scrubbed... URL: From rmondaini at csrc.ac.cn Wed Jul 22 03:25:40 2020 From: rmondaini at csrc.ac.cn (rmondaini at csrc.ac.cn) Date: Wed, 22 Jul 2020 16:25:40 +0800 Subject: [petsc-users] Passing array of PETSc Vec's to a function and returning it Message-ID: <59a3d989-6780-af7c-daae-a5696dd5da61@csrc.ac.cn> I am trying to pass an array of Vec's in PETSc to a function, modify it internally and retrieve the results. The idea is to copy a handful of eigenvectors from the EPS solver to the main routine. A pseudocode is as follows: #include ??? PetscErrorCode foo(Vec **y, int n) { ??????? EPS??????????? eps;???????? // eigenproblem solver context ? ?? ? // ... ??????? ierr = MatCreateVecs(H, &x, NULL); CHKERRQ(ierr); ??????? ierr = EPSSolve(eps); CHKERRQ(ierr); ??? // ... ????? ierr = VecDuplicateVecs(x, n, y); CHKERRQ(ierr); ??? ?? for (int i = 0; i < n ; i++) {??? // I can guarantee that n < nconv ?????????? ierr = EPSGetEigenvector(eps, i, *y[i], NULL); CHKERRQ(ierr);?? // this breaks for i = 1 ?????? ??? ierr = VecNorm(*y[i],NORM_2,&norm); CHKERRQ(ierr);?? // this prints out fine for i = 0 (norm = 1) ?????????? printf("norm = %f\n", norm); ??? ??? } ??????? ierr = EPSDestroy(&eps); CHKERRQ(ierr); ? ?? ?? ierr = VecDestroy(&x); CHKERRQ(ierr); ??????? return ierr; ??? } ??? int main(int argc,char **argv) ??? { ??????? PetscErrorCode ierr; ??????? PetscScalar norm; ??????? Vec *y; ??????? foo(&y, 3); ??????? ierr = VecDestroyVecs(3, &y); CHKERRQ(ierr); ??????? return 0; ??? } Am I making a naive mistake here? From jroman at dsic.upv.es Wed Jul 22 10:10:49 2020 From: jroman at dsic.upv.es (Jose E. Roman) Date: Wed, 22 Jul 2020 17:10:49 +0200 Subject: [petsc-users] Passing array of PETSc Vec's to a function and returning it In-Reply-To: <59a3d989-6780-af7c-daae-a5696dd5da61@csrc.ac.cn> References: <59a3d989-6780-af7c-daae-a5696dd5da61@csrc.ac.cn> Message-ID: <38DC79AE-0052-4125-A61E-BF3C1296EC73@dsic.upv.es> Probably you are requesting more eigenvectors than actually computed. Argument i should be smaller than nconv, see https://slepc.upv.es/documentation/current/docs/manualpages/EPS/EPSGetEigenvector.html Jose > El 22 jul 2020, a las 10:25, rmondaini at csrc.ac.cn escribi?: > > I am trying to pass an array of Vec's in PETSc to a function, modify it internally and retrieve the results. The idea is to copy a handful of eigenvectors from the EPS solver to the main routine. A pseudocode is as follows: > > #include > > PetscErrorCode foo(Vec **y, int n) { > > EPS eps; // eigenproblem solver context > > // ... > > ierr = MatCreateVecs(H, &x, NULL); CHKERRQ(ierr); > > ierr = EPSSolve(eps); CHKERRQ(ierr); > > // ... > > ierr = VecDuplicateVecs(x, n, y); CHKERRQ(ierr); > > for (int i = 0; i < n ; i++) { // I can guarantee that n < nconv > > ierr = EPSGetEigenvector(eps, i, *y[i], NULL); CHKERRQ(ierr); // this breaks for i = 1 > > ierr = VecNorm(*y[i],NORM_2,&norm); CHKERRQ(ierr); // this prints out fine for i = 0 (norm = 1) > > printf("norm = %f\n", norm); > > } > > ierr = EPSDestroy(&eps); CHKERRQ(ierr); > ierr = VecDestroy(&x); CHKERRQ(ierr); > > return ierr; > > } > > int main(int argc,char **argv) > { > PetscErrorCode ierr; > PetscScalar norm; > Vec *y; > > foo(&y, 3); > > ierr = VecDestroyVecs(3, &y); CHKERRQ(ierr); > > return 0; > } > > Am I making a naive mistake here? > From luis.saturday at gmail.com Wed Jul 22 14:50:50 2020 From: luis.saturday at gmail.com (Alex Fleeter) Date: Wed, 22 Jul 2020 12:50:50 -0700 Subject: [petsc-users] pcsetup In-Reply-To: References: Message-ID: Hi Barry: That explains and makes sense. Suppose I have associated a Mat object with KSP, by KSPSetOperators in the beginning of a Newton-Raphson iteration, say. Then in the following iteration steps, the entries in the Mat are updated. I only need to directly call KSPSolve without doing KSPSetOperators, since in KSPSolve the state of the Mat object is automatically detected and the PC will be reconstructed if necessary, correct? On Tue, Jul 21, 2020 at 6:28 PM Barry Smith wrote: > > Alex, > > See my long email. The thing with nested preconditioners (like yours) > is they have multiple PCSetUps(), possibly outer ones and inner ones. Each > of these PC independently triggers a rebuild based on its own state, hence > the inner PCSetUps will be triggered when you change the inner matrices > (even when you don't call MatAssemblyBegin/End() on the MatNest. > > In your case since the preconditioner seems to depend only on the two > sub matrices the fact that PCSetUp() is not called on the MatNest is > harmless, since that setup wouldn't do anything anyways. > > For Schur complement based PCFieldsplits however the result maybe be > wrong unless you call MatAssemblyBegin/End on the nest matrix since the > outer setup actually does something > > Barry > > > On Jul 21, 2020, at 8:16 PM, Alex Fleeter wrote: > > If I call MatAssembly for individual sub-matrices, > MatAssemblyBegin(subA[ii], MAT_FINAL_ASSEMBLY); > MatAssemblyEnd(subA[ii], MAT_FINAL_ASSEMBLY); > > then call KSPSetOperators, the PC setup is not called. > > I have to call the assembly for the whole nest matrix > MatAssemblyBegin(A, MAT_FINAL_ASSEMBLY); > MatAssemblyEnd(A, MAT_FINAL_ASSEMBLY); > to trigger the setup of PC. > > I found some comments in matnest.c > , > line 474-478: "Note: split assembly will fail if the same block appears > more than once (even indirectly through a nested > > 475: * sub-block). This could be fixed by adding a flag to Mat so that there was a way to check if a Mat was476: * already performing an assembly, but the result would by more complicated and appears to offer less477: * potential for diagnostics and correctness checking. Split assembly should be fixed once there is an478: * interface for libraries to make asynchronous progress in "user-defined non-blocking collectives"." > > I guess the comment suggests calling matassembly for the big nest matrix after having setted values for individual sub-matrices, instead of calling matassembly for individual submatrices. > > > Can you confirm that? > > > > On Tue, Jul 21, 2020 at 4:35 PM Mark Adams wrote: > >> KSPSetOperator tells the KSP that the PC should be resetup. >> >> On Tue, Jul 21, 2020 at 3:45 PM Alex Fleeter >> wrote: >> >>> Hi: >>> >>> I want to ask under what circumstance will trigger a call for pc setup. >>> >>> I call KSPSolve to solve with the same Mat object with different entry >>> values each time. I can see that the pc setup is only called at the >>> beginning of the first solve. >>> >>> I tried to read the implementation, but quickly get lost... >>> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Wed Jul 22 15:25:52 2020 From: knepley at gmail.com (Matthew Knepley) Date: Wed, 22 Jul 2020 16:25:52 -0400 Subject: [petsc-users] pcsetup In-Reply-To: References: Message-ID: On Wed, Jul 22, 2020 at 3:52 PM Alex Fleeter wrote: > Hi Barry: > > That explains and makes sense. > > Suppose I have associated a Mat object with KSP, by KSPSetOperators in the > beginning of a Newton-Raphson iteration, say. > > Then in the following iteration steps, the entries in the Mat are updated. > I only need to directly call KSPSolve without doing KSPSetOperators, since > in KSPSolve the state of the Mat object is automatically detected and > the PC will be reconstructed if necessary, correct? > Yes. Thanks, Matt > On Tue, Jul 21, 2020 at 6:28 PM Barry Smith wrote: > >> >> Alex, >> >> See my long email. The thing with nested preconditioners (like >> yours) is they have multiple PCSetUps(), possibly outer ones and inner >> ones. Each of these PC independently triggers a rebuild based on its own >> state, hence the inner PCSetUps will be triggered when you change the inner >> matrices (even when you don't call MatAssemblyBegin/End() on the MatNest. >> >> In your case since the preconditioner seems to depend only on the two >> sub matrices the fact that PCSetUp() is not called on the MatNest is >> harmless, since that setup wouldn't do anything anyways. >> >> For Schur complement based PCFieldsplits however the result maybe be >> wrong unless you call MatAssemblyBegin/End on the nest matrix since the >> outer setup actually does something >> >> Barry >> >> >> On Jul 21, 2020, at 8:16 PM, Alex Fleeter >> wrote: >> >> If I call MatAssembly for individual sub-matrices, >> MatAssemblyBegin(subA[ii], MAT_FINAL_ASSEMBLY); >> MatAssemblyEnd(subA[ii], MAT_FINAL_ASSEMBLY); >> >> then call KSPSetOperators, the PC setup is not called. >> >> I have to call the assembly for the whole nest matrix >> MatAssemblyBegin(A, MAT_FINAL_ASSEMBLY); >> MatAssemblyEnd(A, MAT_FINAL_ASSEMBLY); >> to trigger the setup of PC. >> >> I found some comments in matnest.c >> , >> line 474-478: "Note: split assembly will fail if the same block appears >> more than once (even indirectly through a nested >> >> 475: * sub-block). This could be fixed by adding a flag to Mat so that there was a way to check if a Mat was476: * already performing an assembly, but the result would by more complicated and appears to offer less477: * potential for diagnostics and correctness checking. Split assembly should be fixed once there is an478: * interface for libraries to make asynchronous progress in "user-defined non-blocking collectives"." >> >> I guess the comment suggests calling matassembly for the big nest matrix after having setted values for individual sub-matrices, instead of calling matassembly for individual submatrices. >> >> >> Can you confirm that? >> >> >> >> On Tue, Jul 21, 2020 at 4:35 PM Mark Adams wrote: >> >>> KSPSetOperator tells the KSP that the PC should be resetup. >>> >>> On Tue, Jul 21, 2020 at 3:45 PM Alex Fleeter >>> wrote: >>> >>>> Hi: >>>> >>>> I want to ask under what circumstance will trigger a call for pc setup. >>>> >>>> I call KSPSolve to solve with the same Mat object with different entry >>>> values each time. I can see that the pc setup is only called at the >>>> beginning of the first solve. >>>> >>>> I tried to read the implementation, but quickly get lost... >>>> >>> >> -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From xsli at lbl.gov Wed Jul 22 17:40:15 2020 From: xsli at lbl.gov (Xiaoye S. Li) Date: Wed, 22 Jul 2020 15:40:15 -0700 Subject: [petsc-users] SUPERLU_DIST in single precision In-Reply-To: <20200722150403.Horde.NO015I2GI2E-524J-SXqwui@webmail.mpcdf.mpg.de> References: <20200721155807.Horde.5ZYOzrQ7dmwNTQquK_6ebeS@webmail.mpcdf.mpg.de> <20200722150403.Horde.NO015I2GI2E-524J-SXqwui@webmail.mpcdf.mpg.de> Message-ID: I can package a CPU version over the weekend. Sherry On Wed, Jul 22, 2020 at 6:04 AM wrote: > Hi Barry, > for now I just want to run everything in single on CPUs only with > SUPERLU_DIST. Maybe we will also incorporate GPUs in the future, but > there are no immediate plans yet. So if you could provide the support, > that would be awesome. > > Best regards, > Felix > > Zitat von Barry Smith : > > > Felix, > > > > What are your needs, do you want this for CPUs or for GPUs? Do > > you wish to run all your code in single precision or just the > > SuperLU_Dist solver while the rest of your code double? > > > > If you want to run everything on CPUs using single precision > > then adding the support is very easy, we can provide that for you > > any time. The other cases will require more thought. > > > > Barry > > > > > >> On Jul 21, 2020, at 8:58 AM, flw at rzg.mpg.de wrote: > >> > >> Dear PETSc support team, > >> some time ago you told me that you are planning on releasing a > >> version that supports SUPERLU_DIST in single-precision soon. Can > >> you tell me roughly what time frame you had in mind? > >> > >> Best regards, > >> Felix > >> > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From rmondaini at csrc.ac.cn Wed Jul 22 21:48:54 2020 From: rmondaini at csrc.ac.cn (Rubem Mondaini) Date: Thu, 23 Jul 2020 10:48:54 +0800 Subject: [petsc-users] Passing array of PETSc Vec's to a function and returning it In-Reply-To: <38DC79AE-0052-4125-A61E-BF3C1296EC73@dsic.upv.es> References: <59a3d989-6780-af7c-daae-a5696dd5da61@csrc.ac.cn> <38DC79AE-0052-4125-A61E-BF3C1296EC73@dsic.upv.es> Message-ID: Dear Jose, thank you very much for your prompt reply. I did check before I was not requesting more eigenvectors than the ones converged. In fact, to make things more practical, I made a short version of my application, where I reproduce the problem. test.c: ################################ #include #include #include #include #include // NOTE: Petsc was compiled with "--with-scalar-type=complex" static char help[] = "Hermitian Eigenproblem\n\n"; PetscErrorCode foo(Vec **y, PetscInt n, MPI_Comm mpi_comm) { ??? PetscErrorCode ierr; ??? EPS??????????? eps;???????? // eigenproblem solver context ??? Mat A; ??? PetscInt nnz, nrows, nconv; ??? PetscScalar *val_sp_A; ??? PetscInt *i_sp_A, *j_sp_A; ??? PetscReal norm; ??? Vec x; ??? nnz = 6; nrows = 4; ??? // The matrix looks like: //?????????? [1?? 0? 0? 1+i] //?????????? [0?? 1? 0? 0? ] //???? [A] = [0?? 0? 1? 0? ] //?????????? [1-i 0? 0? 1? ] ??? // I am doing everything with *one* MPI task for the sake of simplicity ??? val_sp_A = malloc(nnz*sizeof(PetscScalar)); ??? i_sp_A = malloc( (nrows+1)*sizeof(PetscInt)); ??? j_sp_A = malloc(nnz*sizeof(PetscInt)); ??? // Building the matrix (CSR first) ??? val_sp_A[0] = 1.0; val_sp_A[1] = 1.0 + 1.0*PETSC_i; val_sp_A[2] = 1.0; val_sp_A[3] = 1.0; val_sp_A[4] = 1.0 -1.0*PETSC_i; val_sp_A[5] = 1.0; ??? j_sp_A[0] = 0;???? j_sp_A[1] = 3; j_sp_A[2] = 1;???? j_sp_A[3] = 2;???? j_sp_A[4] = 0;????????????????? j_sp_A[5] = 3; ??? // zero-based index ??? i_sp_A[0] = 0; i_sp_A[1] = 2; i_sp_A[2] = 3; i_sp_A[3] = 4; i_sp_A[4] = 6; ??? // Building the matrix and vecs ??? ierr = MatCreate(mpi_comm, &A);???????????????????????????????? CHKERRQ(ierr); ??? ierr = MatCreateMPIAIJWithArrays(mpi_comm, nrows, nrows, nrows, nrows, i_sp_A, j_sp_A, val_sp_A, &A); ??? ierr = MatSetUp(A); CHKERRQ(ierr); ??? ierr = MatAssemblyBegin(A, MAT_FINAL_ASSEMBLY);???????????????? CHKERRQ(ierr); ??? ierr = MatAssemblyEnd(A, MAT_FINAL_ASSEMBLY);?????????????????? CHKERRQ(ierr); ??? ierr = MatCreateVecs(A, &x, NULL);????????????????????????????? CHKERRQ(ierr); ??? // Solve eigenvec problem ??? ierr = EPSCreate(mpi_comm, &eps);?????????????????????????????? CHKERRQ(ierr); ??? ierr = EPSSetOperators( eps, A, NULL);????????????????????????? CHKERRQ(ierr); ??? ierr = EPSSetProblemType( eps, EPS_HEP );?????????????????????? CHKERRQ(ierr); ??? ierr = EPSSetWhichEigenpairs(eps, EPS_SMALLEST_REAL);?????????? CHKERRQ(ierr); ??? ierr = EPSSetDimensions(eps, n, 2*n, 2*n);????????????????????? CHKERRQ(ierr); ??? ierr = EPSSolve(eps); CHKERRQ(ierr); ??? ierr = EPSGetConverged(eps, &nconv);??????????????????????????? CHKERRQ(ierr); ??? printf("\nnconv = %lu\n", nconv);???? // I am getting all four eigenvalues converged, you can test it. ??? assert(n < nconv);??????????????????? // This guarantees the number of backed up eigenvectors *will be smaller than nconv* ??? // Now doing the backup of the eigenvectors ??? ierr = VecDuplicateVecs(x, n, y); CHKERRQ(ierr); ??? for (PetscInt i = 0; i < n ; i++) {??? // I can guarantee that n < nconv ??????? ierr = EPSGetEigenvector(eps, i, *y[i], NULL); CHKERRQ(ierr); ??????? ierr = VecNorm(*y[i],NORM_2,&norm); CHKERRQ(ierr);?????? // this prints out fine for i = 0 (norm = 1) ??????? printf("i = %lu\tnorm = %f\n", i, norm); ??? } ??? // Deallocating ??? ierr = EPSDestroy(&eps); CHKERRQ(ierr); ??? ierr = VecDestroy(&x); CHKERRQ(ierr); ??? ierr = MatDestroy(&A); CHKERRQ(ierr); ??? free(val_sp_A); ??? free(i_sp_A); ??? free(j_sp_A); ??? return ierr; } int main(int argc,char **argv) { ??? PetscErrorCode ierr; ??? PetscScalar norm; ??? Vec *y; ??? PetscInt n = 2;? // Backing up *two* eigenvectors ??? ierr = SlepcInitialize(&argc,&argv,(char*)0,help); CHKERRQ(ierr); ??? foo(&y, n, PETSC_COMM_WORLD); ??? ierr = VecDestroyVecs(n, &y); CHKERRQ(ierr); ??? return 0; } ################################################# Makefile: ################################################# #Source File name SRC = test CC = mpiicc default: $(SRC) include ${SLEPC_DIR}/lib/slepc/conf/slepc_common # include? $(SLEPC_DIR)/lib/slepc/conf/slepc_variables CFLAGS_FOR_ICC = -qopt-report=4 -qopt-report-phase ipo -O3 -g -w2 -std=c99? -qopenmp -DMKL_ILP64 -I$(MKLROOT)/include INTEL_LIB =? -Wl,--start-group ${MKLROOT}/lib/intel64/libmkl_intel_ilp64.a ${MKLROOT}/lib/intel64/libmkl_intel_thread.a ${MKLROOT}/lib/intel64/libmkl_core.a -Wl,--end-group -liomp5 -lpthread -lm -ldl CFLAGS=$(CFLAGS_FOR_ICC) $(SRC): $(SRC).o ??? -${CLINKER} -o $(SRC) $(SRC).o ${SLEPC_EPS_LIB} ${INTEL_LIB} # # ??? ${RM} *.o $(SRC).o: $(SRC).c ??? -${CLINKER} -I${PETSC_DIR}/include -I${PETSC_DIR}/linux-intel/include -I${SLEPC_DIR}/include -I${SLEPC_DIR}/linux-intel/include -c $(SRC).c ################################################# Execution ################################################# [rmondaini at manager slepc_test]$ ./test nconv = 4 i = 0??? norm = 1.000000 [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [0]PETSC ERROR: Invalid argument [0]PETSC ERROR: Wrong type of object: Parameter # 3 [0]PETSC ERROR: See https://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [0]PETSC ERROR: Petsc Release Version 3.13.3, Jul 01, 2020 [0]PETSC ERROR: ./test on a linux-intel named manager by rmondaini Thu Jul 23 10:43:15 2020 [0]PETSC ERROR: Configure options PETSC_ARCH=linux-intel --with-cc=mpiicc --with-cxx=mpiicpc --with-fc=mpiifort --with-blaslapack-dir=/opt/intel/compilers_and_libraries_2016/linux/mkl/lib/intel64 --with-mpi-include=/opt/intel/compilers_and_libraries_2016/linux/mpi/intel64/include --with-mpi-lib=/opt/intel/compilers_and_libraries_2016/linux/mpi/intel64/lib/libmpicxx.a --with-mpiexec=/opt/intel/compilers_and_libraries_2016/linux/mpi/intel64/bin/mpiexec --with-scalar-type=complex --with-64-bit-blas-indices --with-64-bit-indices --download-make --force [0]PETSC ERROR: #1 EPSGetEigenvector() line 504 in /home/rmondaini/libraries/slepc-3.13.3/src/eps/interface/epssolve.c [0]PETSC ERROR: #2 foo() line 66 in test.c ################################################# So the offending statement is precisely the one to retrieve eigenvectors, and occurs when i = 1 (as in my previous much larger application). Do you have suggestions in how to proceed? Thank you *very* much! Best, Rubem On 7/22/20 11:10 PM, Jose E. Roman wrote: > Probably you are requesting more eigenvectors than actually computed. Argument i should be smaller than nconv, see https://slepc.upv.es/documentation/current/docs/manualpages/EPS/EPSGetEigenvector.html > Jose > > >> El 22 jul 2020, a las 10:25, rmondaini at csrc.ac.cn escribi?: >> >> I am trying to pass an array of Vec's in PETSc to a function, modify it internally and retrieve the results. The idea is to copy a handful of eigenvectors from the EPS solver to the main routine. A pseudocode is as follows: >> >> #include >> >> PetscErrorCode foo(Vec **y, int n) { >> >> EPS eps; // eigenproblem solver context >> >> // ... >> >> ierr = MatCreateVecs(H, &x, NULL); CHKERRQ(ierr); >> >> ierr = EPSSolve(eps); CHKERRQ(ierr); >> >> // ... >> >> ierr = VecDuplicateVecs(x, n, y); CHKERRQ(ierr); >> >> for (int i = 0; i < n ; i++) { // I can guarantee that n < nconv >> >> ierr = EPSGetEigenvector(eps, i, *y[i], NULL); CHKERRQ(ierr); // this breaks for i = 1 >> >> ierr = VecNorm(*y[i],NORM_2,&norm); CHKERRQ(ierr); // this prints out fine for i = 0 (norm = 1) >> >> printf("norm = %f\n", norm); >> >> } >> >> ierr = EPSDestroy(&eps); CHKERRQ(ierr); >> ierr = VecDestroy(&x); CHKERRQ(ierr); >> >> return ierr; >> >> } >> >> int main(int argc,char **argv) >> { >> PetscErrorCode ierr; >> PetscScalar norm; >> Vec *y; >> >> foo(&y, 3); >> >> ierr = VecDestroyVecs(3, &y); CHKERRQ(ierr); >> >> return 0; >> } >> >> Am I making a naive mistake here? >> From jroman at dsic.upv.es Thu Jul 23 02:08:50 2020 From: jroman at dsic.upv.es (Jose E. Roman) Date: Thu, 23 Jul 2020 09:08:50 +0200 Subject: [petsc-users] Passing array of PETSc Vec's to a function and returning it In-Reply-To: References: <59a3d989-6780-af7c-daae-a5696dd5da61@csrc.ac.cn> <38DC79AE-0052-4125-A61E-BF3C1296EC73@dsic.upv.es> Message-ID: It's a problem with double pointers. You should write (*y)[i] instead of *y[i]. Jose > El 23 jul 2020, a las 4:48, Rubem Mondaini escribi?: > > Dear Jose, > > thank you very much for your prompt reply. I did check before I was not requesting more eigenvectors than the ones converged. > > In fact, to make things more practical, I made a short version of my application, where I reproduce the problem. > > test.c: > > ################################ > > #include > #include > #include > #include > #include > > // NOTE: Petsc was compiled with "--with-scalar-type=complex" > static char help[] = "Hermitian Eigenproblem\n\n"; > > PetscErrorCode foo(Vec **y, PetscInt n, MPI_Comm mpi_comm) > { > > PetscErrorCode ierr; > EPS eps; // eigenproblem solver context > Mat A; > PetscInt nnz, nrows, nconv; > PetscScalar *val_sp_A; > PetscInt *i_sp_A, *j_sp_A; > PetscReal norm; > Vec x; > > nnz = 6; nrows = 4; > > // The matrix looks like: > // [1 0 0 1+i] > // [0 1 0 0 ] > // [A] = [0 0 1 0 ] > // [1-i 0 0 1 ] > > // I am doing everything with *one* MPI task for the sake of simplicity > val_sp_A = malloc(nnz*sizeof(PetscScalar)); > i_sp_A = malloc( (nrows+1)*sizeof(PetscInt)); > j_sp_A = malloc(nnz*sizeof(PetscInt)); > > // Building the matrix (CSR first) > val_sp_A[0] = 1.0; val_sp_A[1] = 1.0 + 1.0*PETSC_i; val_sp_A[2] = 1.0; val_sp_A[3] = 1.0; val_sp_A[4] = 1.0 -1.0*PETSC_i; val_sp_A[5] = 1.0; > j_sp_A[0] = 0; j_sp_A[1] = 3; j_sp_A[2] = 1; j_sp_A[3] = 2; j_sp_A[4] = 0; j_sp_A[5] = 3; > > // zero-based index > i_sp_A[0] = 0; i_sp_A[1] = 2; i_sp_A[2] = 3; i_sp_A[3] = 4; i_sp_A[4] = 6; > > // Building the matrix and vecs > ierr = MatCreate(mpi_comm, &A); CHKERRQ(ierr); > ierr = MatCreateMPIAIJWithArrays(mpi_comm, nrows, nrows, nrows, nrows, i_sp_A, j_sp_A, val_sp_A, &A); > ierr = MatSetUp(A); CHKERRQ(ierr); > ierr = MatAssemblyBegin(A, MAT_FINAL_ASSEMBLY); CHKERRQ(ierr); > ierr = MatAssemblyEnd(A, MAT_FINAL_ASSEMBLY); CHKERRQ(ierr); > ierr = MatCreateVecs(A, &x, NULL); CHKERRQ(ierr); > > // Solve eigenvec problem > ierr = EPSCreate(mpi_comm, &eps); CHKERRQ(ierr); > ierr = EPSSetOperators( eps, A, NULL); CHKERRQ(ierr); > ierr = EPSSetProblemType( eps, EPS_HEP ); CHKERRQ(ierr); > ierr = EPSSetWhichEigenpairs(eps, EPS_SMALLEST_REAL); CHKERRQ(ierr); > ierr = EPSSetDimensions(eps, n, 2*n, 2*n); CHKERRQ(ierr); > ierr = EPSSolve(eps); CHKERRQ(ierr); > ierr = EPSGetConverged(eps, &nconv); CHKERRQ(ierr); > > printf("\nnconv = %lu\n", nconv); // I am getting all four eigenvalues converged, you can test it. > > assert(n < nconv); // This guarantees the number of backed up eigenvectors *will be smaller than nconv* > > // Now doing the backup of the eigenvectors > ierr = VecDuplicateVecs(x, n, y); CHKERRQ(ierr); > for (PetscInt i = 0; i < n ; i++) { // I can guarantee that n < nconv > ierr = EPSGetEigenvector(eps, i, *y[i], NULL); CHKERRQ(ierr); > ierr = VecNorm(*y[i],NORM_2,&norm); CHKERRQ(ierr); // this prints out fine for i = 0 (norm = 1) > printf("i = %lu\tnorm = %f\n", i, norm); > } > > // Deallocating > ierr = EPSDestroy(&eps); CHKERRQ(ierr); > ierr = VecDestroy(&x); CHKERRQ(ierr); > ierr = MatDestroy(&A); CHKERRQ(ierr); > > free(val_sp_A); > free(i_sp_A); > free(j_sp_A); > > return ierr; > > } > > int main(int argc,char **argv) > { > PetscErrorCode ierr; > PetscScalar norm; > Vec *y; > PetscInt n = 2; // Backing up *two* eigenvectors > > ierr = SlepcInitialize(&argc,&argv,(char*)0,help); CHKERRQ(ierr); > > foo(&y, n, PETSC_COMM_WORLD); > > ierr = VecDestroyVecs(n, &y); CHKERRQ(ierr); > > return 0; > } > > ################################################# > > Makefile: > > ################################################# > > #Source File name > SRC = test > CC = mpiicc > > default: $(SRC) > > include ${SLEPC_DIR}/lib/slepc/conf/slepc_common > # include $(SLEPC_DIR)/lib/slepc/conf/slepc_variables > > CFLAGS_FOR_ICC = -qopt-report=4 -qopt-report-phase ipo -O3 -g -w2 -std=c99 -qopenmp -DMKL_ILP64 -I$(MKLROOT)/include > > INTEL_LIB = -Wl,--start-group ${MKLROOT}/lib/intel64/libmkl_intel_ilp64.a ${MKLROOT}/lib/intel64/libmkl_intel_thread.a ${MKLROOT}/lib/intel64/libmkl_core.a -Wl,--end-group -liomp5 -lpthread -lm -ldl > > CFLAGS=$(CFLAGS_FOR_ICC) > > $(SRC): $(SRC).o > -${CLINKER} -o $(SRC) $(SRC).o ${SLEPC_EPS_LIB} ${INTEL_LIB} > # # ${RM} *.o > > $(SRC).o: $(SRC).c > -${CLINKER} -I${PETSC_DIR}/include -I${PETSC_DIR}/linux-intel/include -I${SLEPC_DIR}/include -I${SLEPC_DIR}/linux-intel/include -c $(SRC).c > > ################################################# > > Execution > > ################################################# > > [rmondaini at manager slepc_test]$ ./test > > nconv = 4 > i = 0 norm = 1.000000 > [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- > [0]PETSC ERROR: Invalid argument > [0]PETSC ERROR: Wrong type of object: Parameter # 3 > [0]PETSC ERROR: See https://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. > [0]PETSC ERROR: Petsc Release Version 3.13.3, Jul 01, 2020 > [0]PETSC ERROR: ./test on a linux-intel named manager by rmondaini Thu Jul 23 10:43:15 2020 > [0]PETSC ERROR: Configure options PETSC_ARCH=linux-intel --with-cc=mpiicc --with-cxx=mpiicpc --with-fc=mpiifort --with-blaslapack-dir=/opt/intel/compilers_and_libraries_2016/linux/mkl/lib/intel64 --with-mpi-include=/opt/intel/compilers_and_libraries_2016/linux/mpi/intel64/include --with-mpi-lib=/opt/intel/compilers_and_libraries_2016/linux/mpi/intel64/lib/libmpicxx.a --with-mpiexec=/opt/intel/compilers_and_libraries_2016/linux/mpi/intel64/bin/mpiexec --with-scalar-type=complex --with-64-bit-blas-indices --with-64-bit-indices --download-make --force > [0]PETSC ERROR: #1 EPSGetEigenvector() line 504 in /home/rmondaini/libraries/slepc-3.13.3/src/eps/interface/epssolve.c > [0]PETSC ERROR: #2 foo() line 66 in test.c > > ################################################# > > So the offending statement is precisely the one to retrieve eigenvectors, and occurs when i = 1 (as in my previous much larger application). > > Do you have suggestions in how to proceed? > > Thank you *very* much! > > Best, > > Rubem > > > On 7/22/20 11:10 PM, Jose E. Roman wrote: >> Probably you are requesting more eigenvectors than actually computed. Argument i should be smaller than nconv, see https://slepc.upv.es/documentation/current/docs/manualpages/EPS/EPSGetEigenvector.html >> Jose >> >> >>> El 22 jul 2020, a las 10:25, rmondaini at csrc.ac.cn escribi?: >>> >>> I am trying to pass an array of Vec's in PETSc to a function, modify it internally and retrieve the results. The idea is to copy a handful of eigenvectors from the EPS solver to the main routine. A pseudocode is as follows: >>> >>> #include >>> >>> PetscErrorCode foo(Vec **y, int n) { >>> >>> EPS eps; // eigenproblem solver context >>> >>> // ... >>> >>> ierr = MatCreateVecs(H, &x, NULL); CHKERRQ(ierr); >>> >>> ierr = EPSSolve(eps); CHKERRQ(ierr); >>> >>> // ... >>> >>> ierr = VecDuplicateVecs(x, n, y); CHKERRQ(ierr); >>> >>> for (int i = 0; i < n ; i++) { // I can guarantee that n < nconv >>> >>> ierr = EPSGetEigenvector(eps, i, *y[i], NULL); CHKERRQ(ierr); // this breaks for i = 1 >>> >>> ierr = VecNorm(*y[i],NORM_2,&norm); CHKERRQ(ierr); // this prints out fine for i = 0 (norm = 1) >>> >>> printf("norm = %f\n", norm); >>> >>> } >>> >>> ierr = EPSDestroy(&eps); CHKERRQ(ierr); >>> ierr = VecDestroy(&x); CHKERRQ(ierr); >>> >>> return ierr; >>> >>> } >>> >>> int main(int argc,char **argv) >>> { >>> PetscErrorCode ierr; >>> PetscScalar norm; >>> Vec *y; >>> >>> foo(&y, 3); >>> >>> ierr = VecDestroyVecs(3, &y); CHKERRQ(ierr); >>> >>> return 0; >>> } >>> >>> Am I making a naive mistake here? >>> > From rmondaini at csrc.ac.cn Thu Jul 23 02:31:34 2020 From: rmondaini at csrc.ac.cn (Rubem Mondaini) Date: Thu, 23 Jul 2020 15:31:34 +0800 Subject: [petsc-users] Passing array of PETSc Vec's to a function and returning it In-Reply-To: References: <59a3d989-6780-af7c-daae-a5696dd5da61@csrc.ac.cn> <38DC79AE-0052-4125-A61E-BF3C1296EC73@dsic.upv.es> Message-ID: That's is precisely the issue. Thank you! On 7/23/20 3:08 PM, Jose E. Roman wrote: > It's a problem with double pointers. You should write (*y)[i] instead of *y[i]. > Jose > > >> El 23 jul 2020, a las 4:48, Rubem Mondaini escribi?: >> >> Dear Jose, >> >> thank you very much for your prompt reply. I did check before I was not requesting more eigenvectors than the ones converged. >> >> In fact, to make things more practical, I made a short version of my application, where I reproduce the problem. >> >> test.c: >> >> ################################ >> >> #include >> #include >> #include >> #include >> #include >> >> // NOTE: Petsc was compiled with "--with-scalar-type=complex" >> static char help[] = "Hermitian Eigenproblem\n\n"; >> >> PetscErrorCode foo(Vec **y, PetscInt n, MPI_Comm mpi_comm) >> { >> >> PetscErrorCode ierr; >> EPS eps; // eigenproblem solver context >> Mat A; >> PetscInt nnz, nrows, nconv; >> PetscScalar *val_sp_A; >> PetscInt *i_sp_A, *j_sp_A; >> PetscReal norm; >> Vec x; >> >> nnz = 6; nrows = 4; >> >> // The matrix looks like: >> // [1 0 0 1+i] >> // [0 1 0 0 ] >> // [A] = [0 0 1 0 ] >> // [1-i 0 0 1 ] >> >> // I am doing everything with *one* MPI task for the sake of simplicity >> val_sp_A = malloc(nnz*sizeof(PetscScalar)); >> i_sp_A = malloc( (nrows+1)*sizeof(PetscInt)); >> j_sp_A = malloc(nnz*sizeof(PetscInt)); >> >> // Building the matrix (CSR first) >> val_sp_A[0] = 1.0; val_sp_A[1] = 1.0 + 1.0*PETSC_i; val_sp_A[2] = 1.0; val_sp_A[3] = 1.0; val_sp_A[4] = 1.0 -1.0*PETSC_i; val_sp_A[5] = 1.0; >> j_sp_A[0] = 0; j_sp_A[1] = 3; j_sp_A[2] = 1; j_sp_A[3] = 2; j_sp_A[4] = 0; j_sp_A[5] = 3; >> >> // zero-based index >> i_sp_A[0] = 0; i_sp_A[1] = 2; i_sp_A[2] = 3; i_sp_A[3] = 4; i_sp_A[4] = 6; >> >> // Building the matrix and vecs >> ierr = MatCreate(mpi_comm, &A); CHKERRQ(ierr); >> ierr = MatCreateMPIAIJWithArrays(mpi_comm, nrows, nrows, nrows, nrows, i_sp_A, j_sp_A, val_sp_A, &A); >> ierr = MatSetUp(A); CHKERRQ(ierr); >> ierr = MatAssemblyBegin(A, MAT_FINAL_ASSEMBLY); CHKERRQ(ierr); >> ierr = MatAssemblyEnd(A, MAT_FINAL_ASSEMBLY); CHKERRQ(ierr); >> ierr = MatCreateVecs(A, &x, NULL); CHKERRQ(ierr); >> >> // Solve eigenvec problem >> ierr = EPSCreate(mpi_comm, &eps); CHKERRQ(ierr); >> ierr = EPSSetOperators( eps, A, NULL); CHKERRQ(ierr); >> ierr = EPSSetProblemType( eps, EPS_HEP ); CHKERRQ(ierr); >> ierr = EPSSetWhichEigenpairs(eps, EPS_SMALLEST_REAL); CHKERRQ(ierr); >> ierr = EPSSetDimensions(eps, n, 2*n, 2*n); CHKERRQ(ierr); >> ierr = EPSSolve(eps); CHKERRQ(ierr); >> ierr = EPSGetConverged(eps, &nconv); CHKERRQ(ierr); >> >> printf("\nnconv = %lu\n", nconv); // I am getting all four eigenvalues converged, you can test it. >> >> assert(n < nconv); // This guarantees the number of backed up eigenvectors *will be smaller than nconv* >> >> // Now doing the backup of the eigenvectors >> ierr = VecDuplicateVecs(x, n, y); CHKERRQ(ierr); >> for (PetscInt i = 0; i < n ; i++) { // I can guarantee that n < nconv >> ierr = EPSGetEigenvector(eps, i, *y[i], NULL); CHKERRQ(ierr); >> ierr = VecNorm(*y[i],NORM_2,&norm); CHKERRQ(ierr); // this prints out fine for i = 0 (norm = 1) >> printf("i = %lu\tnorm = %f\n", i, norm); >> } >> >> // Deallocating >> ierr = EPSDestroy(&eps); CHKERRQ(ierr); >> ierr = VecDestroy(&x); CHKERRQ(ierr); >> ierr = MatDestroy(&A); CHKERRQ(ierr); >> >> free(val_sp_A); >> free(i_sp_A); >> free(j_sp_A); >> >> return ierr; >> >> } >> >> int main(int argc,char **argv) >> { >> PetscErrorCode ierr; >> PetscScalar norm; >> Vec *y; >> PetscInt n = 2; // Backing up *two* eigenvectors >> >> ierr = SlepcInitialize(&argc,&argv,(char*)0,help); CHKERRQ(ierr); >> >> foo(&y, n, PETSC_COMM_WORLD); >> >> ierr = VecDestroyVecs(n, &y); CHKERRQ(ierr); >> >> return 0; >> } >> >> ################################################# >> >> Makefile: >> >> ################################################# >> >> #Source File name >> SRC = test >> CC = mpiicc >> >> default: $(SRC) >> >> include ${SLEPC_DIR}/lib/slepc/conf/slepc_common >> # include $(SLEPC_DIR)/lib/slepc/conf/slepc_variables >> >> CFLAGS_FOR_ICC = -qopt-report=4 -qopt-report-phase ipo -O3 -g -w2 -std=c99 -qopenmp -DMKL_ILP64 -I$(MKLROOT)/include >> >> INTEL_LIB = -Wl,--start-group ${MKLROOT}/lib/intel64/libmkl_intel_ilp64.a ${MKLROOT}/lib/intel64/libmkl_intel_thread.a ${MKLROOT}/lib/intel64/libmkl_core.a -Wl,--end-group -liomp5 -lpthread -lm -ldl >> >> CFLAGS=$(CFLAGS_FOR_ICC) >> >> $(SRC): $(SRC).o >> -${CLINKER} -o $(SRC) $(SRC).o ${SLEPC_EPS_LIB} ${INTEL_LIB} >> # # ${RM} *.o >> >> $(SRC).o: $(SRC).c >> -${CLINKER} -I${PETSC_DIR}/include -I${PETSC_DIR}/linux-intel/include -I${SLEPC_DIR}/include -I${SLEPC_DIR}/linux-intel/include -c $(SRC).c >> >> ################################################# >> >> Execution >> >> ################################################# >> >> [rmondaini at manager slepc_test]$ ./test >> >> nconv = 4 >> i = 0 norm = 1.000000 >> [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- >> [0]PETSC ERROR: Invalid argument >> [0]PETSC ERROR: Wrong type of object: Parameter # 3 >> [0]PETSC ERROR: See https://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. >> [0]PETSC ERROR: Petsc Release Version 3.13.3, Jul 01, 2020 >> [0]PETSC ERROR: ./test on a linux-intel named manager by rmondaini Thu Jul 23 10:43:15 2020 >> [0]PETSC ERROR: Configure options PETSC_ARCH=linux-intel --with-cc=mpiicc --with-cxx=mpiicpc --with-fc=mpiifort --with-blaslapack-dir=/opt/intel/compilers_and_libraries_2016/linux/mkl/lib/intel64 --with-mpi-include=/opt/intel/compilers_and_libraries_2016/linux/mpi/intel64/include --with-mpi-lib=/opt/intel/compilers_and_libraries_2016/linux/mpi/intel64/lib/libmpicxx.a --with-mpiexec=/opt/intel/compilers_and_libraries_2016/linux/mpi/intel64/bin/mpiexec --with-scalar-type=complex --with-64-bit-blas-indices --with-64-bit-indices --download-make --force >> [0]PETSC ERROR: #1 EPSGetEigenvector() line 504 in /home/rmondaini/libraries/slepc-3.13.3/src/eps/interface/epssolve.c >> [0]PETSC ERROR: #2 foo() line 66 in test.c >> >> ################################################# >> >> So the offending statement is precisely the one to retrieve eigenvectors, and occurs when i = 1 (as in my previous much larger application). >> >> Do you have suggestions in how to proceed? >> >> Thank you *very* much! >> >> Best, >> >> Rubem >> >> >> On 7/22/20 11:10 PM, Jose E. Roman wrote: >>> Probably you are requesting more eigenvectors than actually computed. Argument i should be smaller than nconv, see https://slepc.upv.es/documentation/current/docs/manualpages/EPS/EPSGetEigenvector.html >>> Jose >>> >>> >>>> El 22 jul 2020, a las 10:25, rmondaini at csrc.ac.cn escribi?: >>>> >>>> I am trying to pass an array of Vec's in PETSc to a function, modify it internally and retrieve the results. The idea is to copy a handful of eigenvectors from the EPS solver to the main routine. A pseudocode is as follows: >>>> >>>> #include >>>> >>>> PetscErrorCode foo(Vec **y, int n) { >>>> >>>> EPS eps; // eigenproblem solver context >>>> >>>> // ... >>>> >>>> ierr = MatCreateVecs(H, &x, NULL); CHKERRQ(ierr); >>>> >>>> ierr = EPSSolve(eps); CHKERRQ(ierr); >>>> >>>> // ... >>>> >>>> ierr = VecDuplicateVecs(x, n, y); CHKERRQ(ierr); >>>> >>>> for (int i = 0; i < n ; i++) { // I can guarantee that n < nconv >>>> >>>> ierr = EPSGetEigenvector(eps, i, *y[i], NULL); CHKERRQ(ierr); // this breaks for i = 1 >>>> >>>> ierr = VecNorm(*y[i],NORM_2,&norm); CHKERRQ(ierr); // this prints out fine for i = 0 (norm = 1) >>>> >>>> printf("norm = %f\n", norm); >>>> >>>> } >>>> >>>> ierr = EPSDestroy(&eps); CHKERRQ(ierr); >>>> ierr = VecDestroy(&x); CHKERRQ(ierr); >>>> >>>> return ierr; >>>> >>>> } >>>> >>>> int main(int argc,char **argv) >>>> { >>>> PetscErrorCode ierr; >>>> PetscScalar norm; >>>> Vec *y; >>>> >>>> foo(&y, 3); >>>> >>>> ierr = VecDestroyVecs(3, &y); CHKERRQ(ierr); >>>> >>>> return 0; >>>> } >>>> >>>> Am I making a naive mistake here? >>>> From flw at rzg.mpg.de Thu Jul 23 03:02:05 2020 From: flw at rzg.mpg.de (flw at rzg.mpg.de) Date: Thu, 23 Jul 2020 10:02:05 +0200 Subject: [petsc-users] SUPERLU_DIST in single precision In-Reply-To: References: <20200721155807.Horde.5ZYOzrQ7dmwNTQquK_6ebeS@webmail.mpcdf.mpg.de> <20200722150403.Horde.NO015I2GI2E-524J-SXqwui@webmail.mpcdf.mpg.de> Message-ID: <20200723100205.Horde.dGl64UB0Pnqh3IBVbmi-rs-@webmail.mpcdf.mpg.de> Hi Sherry, thanks, that would be awesome! Best regards, Felix Quoting "Xiaoye S. Li" : > I can package a CPU version over the weekend. > > Sherry > > On Wed, Jul 22, 2020 at 6:04 AM wrote: > >> Hi Barry, >> for now I just want to run everything in single on CPUs only with >> SUPERLU_DIST. Maybe we will also incorporate GPUs in the future, but >> there are no immediate plans yet. So if you could provide the support, >> that would be awesome. >> >> Best regards, >> Felix >> >> Zitat von Barry Smith : >> >> > Felix, >> > >> > What are your needs, do you want this for CPUs or for GPUs? Do >> > you wish to run all your code in single precision or just the >> > SuperLU_Dist solver while the rest of your code double? >> > >> > If you want to run everything on CPUs using single precision >> > then adding the support is very easy, we can provide that for you >> > any time. The other cases will require more thought. >> > >> > Barry >> > >> > >> >> On Jul 21, 2020, at 8:58 AM, flw at rzg.mpg.de wrote: >> >> >> >> Dear PETSc support team, >> >> some time ago you told me that you are planning on releasing a >> >> version that supports SUPERLU_DIST in single-precision soon. Can >> >> you tell me roughly what time frame you had in mind? >> >> >> >> Best regards, >> >> Felix >> >> >> >> >> >> From junchao.zhang at gmail.com Thu Jul 23 18:57:53 2020 From: junchao.zhang at gmail.com (Junchao Zhang) Date: Thu, 23 Jul 2020 18:57:53 -0500 Subject: [petsc-users] Tough to reproduce petsctablefind error In-Reply-To: <0AC37384-BC37-4A6C-857D-41CD507F84C2@petsc.dev> References: <0AC37384-BC37-4A6C-857D-41CD507F84C2@petsc.dev> Message-ID: On Mon, Jul 20, 2020 at 7:05 AM Barry Smith wrote: > > Is there a comprehensive MPI test suite (perhaps from MPICH)? Is > there any way to run this full test suite under the problematic MPI and see > if it detects any problems? > > Is so, could someone add it to the FAQ in the debugging section? > MPICH does have a test suite. It is at the subdir test/mpi of downloaded mpich . It annoyed me since it is not user-friendly. It might be helpful in catching bugs at very small scale. But say if I want to test allreduce on 1024 ranks on 100 doubles, I have to hack the test suite. Anyway, the instructions are here. For the purpose of petsc, under test/mpi one can configure it with $./configure CC=mpicc CXX=mpicxx FC=mpifort --enable-strictmpi --enable-threads=funneled --enable-fortran=f77,f90 --enable-fast --disable-spawn --disable-cxx --disable-ft-tests // It is weird I disabled cxx but I had to set CXX! $make -k -j8 // -k is to keep going and ignore compilation errors, e.g., when building tests for MPICH extensions not in MPI standard, but your MPI is OpenMPI. $ // edit testlist, remove lines mpi_t, rma, f77, impls. Those are sub-dirs containing tests for MPI routines Petsc does not rely on. $ make testings or directly './runtests -tests=testlist' On a batch system, $export MPITEST_BATCHDIR=`pwd`/btest // specify a batch dir, say btest, $./runtests -batch -mpiexec=mpirun -np=1024 -tests=testlist // Use 1024 ranks if a test does no specify the number of processes. $ // It copies test binaries to the batch dir and generates a script runtests.batch there. Edit the script to fit your batch system and then submit a job and wait for its finish. $ cd btest && ../checktests --ignorebogus PS: Fande, changing an MPI fixed your problem does not necessarily mean the old MPI has bugs. It is complicated. It could be a petsc bug. You need to provide us a code to reproduce your error. It does not matter if the code is big. > Thanks > > Barry > > > On Jul 20, 2020, at 12:16 AM, Fande Kong wrote: > > Trace could look like this: > > [640]PETSC ERROR: --------------------- Error Message > -------------------------------------------------------------- > [640]PETSC ERROR: Argument out of range > [640]PETSC ERROR: key 45226154 is greater than largest key allowed 740521 > [640]PETSC ERROR: See https://www.mcs.anl.gov/petsc/documentation/faq.html for > trouble shooting. > [640]PETSC ERROR: Petsc Release Version 3.13.3, unknown > [640]PETSC ERROR: ../../griffin-opt on a arch-moose named r6i5n18 by > wangy2 Sun Jul 19 17:14:28 2020 > [640]PETSC ERROR: Configure options --download-hypre=1 --with-debugging=no > --with-shared-libraries=1 --download-fblaslapack=1 --download-metis=1 > --download-ptscotch=1 --download-parmetis=1 --download-superlu_dist=1 > --download-mumps=1 --download-scalapack=1 --download-slepc=1 --with-mpi=1 > --with-cxx-dialect=C++11 --with-fortran-bindings=0 --with-sowing=0 > --with-64-bit-indices --download-mumps=0 > [640]PETSC ERROR: #1 PetscTableFind() line 132 in > /home/wangy2/trunk/sawtooth/griffin/moose/petsc/include/petscctable.h > [640]PETSC ERROR: #2 MatSetUpMultiply_MPIAIJ() line 33 in > /home/wangy2/trunk/sawtooth/griffin/moose/petsc/src/mat/impls/aij/mpi/mmaij.c > [640]PETSC ERROR: #3 MatAssemblyEnd_MPIAIJ() line 876 in > /home/wangy2/trunk/sawtooth/griffin/moose/petsc/src/mat/impls/aij/mpi/mpiaij.c > [640]PETSC ERROR: #4 MatAssemblyEnd() line 5347 in > /home/wangy2/trunk/sawtooth/griffin/moose/petsc/src/mat/interface/matrix.c > [640]PETSC ERROR: #5 MatPtAPNumeric_MPIAIJ_MPIXAIJ_allatonce() line 901 in > /home/wangy2/trunk/sawtooth/griffin/moose/petsc/src/mat/impls/aij/mpi/mpiptap.c > [640]PETSC ERROR: #6 MatPtAPNumeric_MPIAIJ_MPIMAIJ_allatonce() line 3180 > in /home/wangy2/trunk/sawtooth/griffin/moose/petsc/src/mat/impls/maij/maij.c > [640]PETSC ERROR: #7 MatProductNumeric_PtAP() line 704 in > /home/wangy2/trunk/sawtooth/griffin/moose/petsc/src/mat/interface/matproduct.c > [640]PETSC ERROR: #8 MatProductNumeric() line 759 in > /home/wangy2/trunk/sawtooth/griffin/moose/petsc/src/mat/interface/matproduct.c > [640]PETSC ERROR: #9 MatPtAP() line 9199 in > /home/wangy2/trunk/sawtooth/griffin/moose/petsc/src/mat/interface/matrix.c > [640]PETSC ERROR: #10 MatGalerkin() line 10236 in > /home/wangy2/trunk/sawtooth/griffin/moose/petsc/src/mat/interface/matrix.c > [640]PETSC ERROR: #11 PCSetUp_MG() line 745 in > /home/wangy2/trunk/sawtooth/griffin/moose/petsc/src/ksp/pc/impls/mg/mg.c > [640]PETSC ERROR: #12 PCSetUp_HMG() line 220 in > /home/wangy2/trunk/sawtooth/griffin/moose/petsc/src/ksp/pc/impls/hmg/hmg.c > [640]PETSC ERROR: #13 PCSetUp() line 898 in > /home/wangy2/trunk/sawtooth/griffin/moose/petsc/src/ksp/pc/interface/precon.c > [640]PETSC ERROR: #14 KSPSetUp() line 376 in > /home/wangy2/trunk/sawtooth/griffin/moose/petsc/src/ksp/ksp/interface/itfunc.c > [640]PETSC ERROR: #15 KSPSolve_Private() line 633 in > /home/wangy2/trunk/sawtooth/griffin/moose/petsc/src/ksp/ksp/interface/itfunc.c > [640]PETSC ERROR: #16 KSPSolve() line 853 in > /home/wangy2/trunk/sawtooth/griffin/moose/petsc/src/ksp/ksp/interface/itfunc.c > [640]PETSC ERROR: #17 SNESSolve_NEWTONLS() line 225 in > /home/wangy2/trunk/sawtooth/griffin/moose/petsc/src/snes/impls/ls/ls.c > [640]PETSC ERROR: #18 SNESSolve() line 4519 in > /home/wangy2/trunk/sawtooth/griffin/moose/petsc/src/snes/interface/snes.c > > On Sun, Jul 19, 2020 at 11:13 PM Fande Kong wrote: > >> I am not entirely sure what is happening, but we encountered similar >> issues recently. It was not reproducible. It might occur at different >> stages, and errors could be weird other than "ctable stuff." Our code was >> Valgrind clean since every PR in moose needs to go through rigorous >> Valgrind checks before it reaches the devel branch. The errors happened >> when we used mvapich. >> >> We changed to use HPE-MPT (a vendor stalled MPI), then everything was >> smooth. May you try a different MPI? It is better to try a system carried >> one. >> >> We did not get the bottom of this problem yet, but we at least know this >> is kind of MPI-related. >> >> Thanks, >> >> Fande, >> >> >> On Sun, Jul 19, 2020 at 3:28 PM Chris Hewson wrote: >> >>> Hi, >>> >>> I am having a bug that is occurring in PETSC with the return string: >>> >>> [7]PETSC ERROR: PetscTableFind() line 132 in >>> /home/chewson/petsc-3.13.2/include/petscctable.h key 7556 is greater than >>> largest key allowed 5693 >>> >>> This is using petsc-3.13.2, compiled and running using mpich with -O3 >>> and debugging turned off tuned to the haswell architecture and >>> occurring either before or during a KSPBCGS solve/setup or during a MUMPS >>> factorization solve (I haven't been able to replicate this issue with the >>> same set of instructions etc.). >>> >>> This is a terrible way to ask a question, I know, and not very helpful >>> from your side, but this is what I have from a user's run and can't >>> reproduce on my end (either with the optimization compilation or with >>> debugging turned on). This happens when the code has run for quite some >>> time and is happening somewhat rarely. >>> >>> More than likely I am using a static variable (code is written in c++) >>> that I'm not updating when the matrix size is changing or something silly >>> like that, but any help or guidance on this would be appreciated. >>> >>> *Chris Hewson* >>> Senior Reservoir Simulation Engineer >>> ResFrac >>> +1.587.575.9792 >>> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From swarnava89 at gmail.com Sat Jul 25 03:08:53 2020 From: swarnava89 at gmail.com (Swarnava Ghosh) Date: Sat, 25 Jul 2020 01:08:53 -0700 Subject: [petsc-users] coordinates of vertices of a cell in 3D DMPlex Message-ID: Dear Petsc users, I had a trivial question about DMPlex. Suppose I have a 3D mesh of tetrahedrons. I want to find out the 3D coordinates of the vertices of a particular cell. What would be the function to do this? Thank you, SG -------------- next part -------------- An HTML attachment was scrubbed... URL: From mfadams at lbl.gov Sat Jul 25 06:13:02 2020 From: mfadams at lbl.gov (Mark Adams) Date: Sat, 25 Jul 2020 07:13:02 -0400 Subject: [petsc-users] coordinates of vertices of a cell in 3D DMPlex In-Reply-To: References: Message-ID: I get all the coordinates with this method: static PetscErrorCode crd_func(PetscInt dim, PetscReal time, const PetscReal x[], PetscInt Nf_dummy, PetscScalar *u, void *actx) { int i; PetscFunctionBeginUser; for (i = 0; i < dim; ++i) u[i] = x[i]; PetscFunctionReturn(0); } PetscErrorCode (*initu[1])(PetscInt, PetscReal, const PetscReal [], PetscInt, PetscScalar [], void *); /* project coordinates to vertices */ ierr = DMCreateGlobalVector(crddm, &crd_vec);CHKERRV(ierr); initu[0] = crd_func; ierr = DMProjectFunction(crddm, 0.0, initu, NULL, INSERT_ALL_VALUES, crd_vec);CHKERRV(ierr); ierr = VecViewFromOptions(crd_vec, NULL, "-coord_view");CHKERRV(ierr); /* iterate over mesh data and get indices */ ierr = VecGetArrayRead(crd_vec,&xx);CHKERRV(ierr); ierr = VecGetLocalSize(rho,&N);CHKERRV(ierr); /* access grid data here */ for (p=0;p wrote: > Dear Petsc users, > > I had a trivial question about DMPlex. Suppose I have a 3D mesh of > tetrahedrons. I want to find out the 3D coordinates of the vertices of a > particular cell. What would be the function to do this? > > Thank you, > SG > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mfadams at lbl.gov Sat Jul 25 06:39:06 2020 From: mfadams at lbl.gov (Mark Adams) Date: Sat, 25 Jul 2020 07:39:06 -0400 Subject: [petsc-users] coordinates of vertices of a cell in 3D DMPlex In-Reply-To: References: Message-ID: Oh, here is a way to get an elements coordinates from the coordinate vector: DM plex; ierr = DMConvert(dm, DMPLEX, &plex);CHKERRQ(ierr); PetscSection section,globsection; ierr = DMGetLocalSection(plex, §ion);CHKERRQ(ierr); ierr = DMPlexVecGetClosure(plex, section, crd_vec, cell, NULL, &coef);CHKERRQ(ierr); .... ierr = DMPlexVecRestoreClosure(plex, section, crd_vec, cell, NULL, &coef);CHKERRQ(ierr); On Sat, Jul 25, 2020 at 7:13 AM Mark Adams wrote: > I get all the coordinates with this method: > > static PetscErrorCode crd_func(PetscInt dim, PetscReal time, const > PetscReal x[], PetscInt Nf_dummy, PetscScalar *u, void *actx) > { > int i; > PetscFunctionBeginUser; > for (i = 0; i < dim; ++i) u[i] = x[i]; > PetscFunctionReturn(0); > } > > PetscErrorCode (*initu[1])(PetscInt, PetscReal, const PetscReal [], > PetscInt, PetscScalar [], void *); > /* project coordinates to vertices */ > ierr = DMCreateGlobalVector(crddm, &crd_vec);CHKERRV(ierr); > initu[0] = crd_func; > ierr = DMProjectFunction(crddm, 0.0, initu, NULL, INSERT_ALL_VALUES, > crd_vec);CHKERRV(ierr); > ierr = VecViewFromOptions(crd_vec, NULL, "-coord_view");CHKERRV(ierr); > /* iterate over mesh data and get indices */ > ierr = VecGetArrayRead(crd_vec,&xx);CHKERRV(ierr); > ierr = VecGetLocalSize(rho,&N);CHKERRV(ierr); > /* access grid data here */ > for (p=0;p for (i=0;i PetscPrintf(PETSC_COMM_SELF,"xx = (%g, %g)\n", xx[p*dim+0], > xx[p*dim+1]); > } > ierr = VecRestoreArrayRead(crd_vec,&xx);CHKERRV(ierr); > ierr = VecDestroy(&crd_vec);CHKERRV(ierr); > > On Sat, Jul 25, 2020 at 4:10 AM Swarnava Ghosh > wrote: > >> Dear Petsc users, >> >> I had a trivial question about DMPlex. Suppose I have a 3D mesh of >> tetrahedrons. I want to find out the 3D coordinates of the vertices of a >> particular cell. What would be the function to do this? >> >> Thank you, >> SG >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From stefano.zampini at gmail.com Sat Jul 25 06:40:57 2020 From: stefano.zampini at gmail.com (Stefano Zampini) Date: Sat, 25 Jul 2020 13:40:57 +0200 Subject: [petsc-users] coordinates of vertices of a cell in 3D DMPlex In-Reply-To: References: Message-ID: <15B2A4A3-085C-4F14-8935-934FFC58D919@gmail.com> Mark This will only work if you have a vector space for the function > On Jul 25, 2020, at 1:13 PM, Mark Adams wrote: > > I get all the coordinates with this method: > > static PetscErrorCode crd_func(PetscInt dim, PetscReal time, const PetscReal x[], PetscInt Nf_dummy, PetscScalar *u, void *actx) > { > int i; > PetscFunctionBeginUser; > for (i = 0; i < dim; ++i) u[i] = x[i]; > PetscFunctionReturn(0); > } > > PetscErrorCode (*initu[1])(PetscInt, PetscReal, const PetscReal [], PetscInt, PetscScalar [], void *); > /* project coordinates to vertices */ > ierr = DMCreateGlobalVector(crddm, &crd_vec);CHKERRV(ierr); > initu[0] = crd_func; > ierr = DMProjectFunction(crddm, 0.0, initu, NULL, INSERT_ALL_VALUES, crd_vec);CHKERRV(ierr); > ierr = VecViewFromOptions(crd_vec, NULL, "-coord_view");CHKERRV(ierr); > /* iterate over mesh data and get indices */ > ierr = VecGetArrayRead(crd_vec,&xx);CHKERRV(ierr); > ierr = VecGetLocalSize(rho,&N);CHKERRV(ierr); > /* access grid data here */ > for (p=0;p for (i=0;i PetscPrintf(PETSC_COMM_SELF,"xx = (%g, %g)\n", xx[p*dim+0], xx[p*dim+1]); > } > ierr = VecRestoreArrayRead(crd_vec,&xx);CHKERRV(ierr); > ierr = VecDestroy(&crd_vec);CHKERRV(ierr); > > On Sat, Jul 25, 2020 at 4:10 AM Swarnava Ghosh > wrote: > Dear Petsc users, > > I had a trivial question about DMPlex. Suppose I have a 3D mesh of tetrahedrons. I want to find out the 3D coordinates of the vertices of a particular cell. What would be the function to do this? > > Thank you, > SG -------------- next part -------------- An HTML attachment was scrubbed... URL: From mfadams at lbl.gov Sat Jul 25 08:08:33 2020 From: mfadams at lbl.gov (Mark Adams) Date: Sat, 25 Jul 2020 09:08:33 -0400 Subject: [petsc-users] coordinates of vertices of a cell in 3D DMPlex In-Reply-To: <15B2A4A3-085C-4F14-8935-934FFC58D919@gmail.com> References: <15B2A4A3-085C-4F14-8935-934FFC58D919@gmail.com> Message-ID: Yea, I did not get all the code you need. Here is an example of making crddm. I'm not sure if this is all best practices (Matt?) /* create coordinate DM */ ierr = DMClone(dm, &crddm);CHKERRV(ierr); ierr = PetscFECreateDefault(PETSC_COMM_SELF, dim, dim, PETSC_FALSE, "", PETSC_DECIDE, &fe);CHKERRV(ierr); // ierr = PetscFECreateLagrange(PETSC_COMM_SELF, dim, dim, PETSC_FALSE, "", PETSC_DECIDE, &fe);CHKERRQ(ierr); ierr = PetscFESetFromOptions(fe);CHKERRV(ierr); ierr = DMSetField(crddm, field, NULL, (PetscObject)fe);CHKERRV(ierr); ierr = DMCreateDS(crddm);CHKERRV(ierr); ierr = PetscFEDestroy(&fe);CHKERRV(ierr); On Sat, Jul 25, 2020 at 7:40 AM Stefano Zampini wrote: > Mark > > This will only work if you have a vector space for the function > > > On Jul 25, 2020, at 1:13 PM, Mark Adams wrote: > > I get all the coordinates with this method: > > static PetscErrorCode crd_func(PetscInt dim, PetscReal time, const > PetscReal x[], PetscInt Nf_dummy, PetscScalar *u, void *actx) > { > int i; > PetscFunctionBeginUser; > for (i = 0; i < dim; ++i) u[i] = x[i]; > PetscFunctionReturn(0); > } > > PetscErrorCode (*initu[1])(PetscInt, PetscReal, const PetscReal [], > PetscInt, PetscScalar [], void *); > /* project coordinates to vertices */ > ierr = DMCreateGlobalVector(crddm, &crd_vec);CHKERRV(ierr); > initu[0] = crd_func; > ierr = DMProjectFunction(crddm, 0.0, initu, NULL, INSERT_ALL_VALUES, > crd_vec);CHKERRV(ierr); > ierr = VecViewFromOptions(crd_vec, NULL, "-coord_view");CHKERRV(ierr); > /* iterate over mesh data and get indices */ > ierr = VecGetArrayRead(crd_vec,&xx);CHKERRV(ierr); > ierr = VecGetLocalSize(rho,&N);CHKERRV(ierr); > /* access grid data here */ > for (p=0;p for (i=0;i PetscPrintf(PETSC_COMM_SELF,"xx = (%g, %g)\n", xx[p*dim+0], > xx[p*dim+1]); > } > ierr = VecRestoreArrayRead(crd_vec,&xx);CHKERRV(ierr); > ierr = VecDestroy(&crd_vec);CHKERRV(ierr); > > On Sat, Jul 25, 2020 at 4:10 AM Swarnava Ghosh > wrote: > >> Dear Petsc users, >> >> I had a trivial question about DMPlex. Suppose I have a 3D mesh of >> tetrahedrons. I want to find out the 3D coordinates of the vertices of a >> particular cell. What would be the function to do this? >> >> Thank you, >> SG >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mfadams at lbl.gov Sat Jul 25 08:13:04 2020 From: mfadams at lbl.gov (Mark Adams) Date: Sat, 25 Jul 2020 09:13:04 -0400 Subject: [petsc-users] coordinates of vertices of a cell in 3D DMPlex In-Reply-To: References: <15B2A4A3-085C-4F14-8935-934FFC58D919@gmail.com> Message-ID: On Sat, Jul 25, 2020 at 9:08 AM Mark Adams wrote: > Yea, I did not get all the code you need. Here is an example of making > crddm. I'm not sure if this is all best practices (Matt?) > > /* create coordinate DM */ > ierr = DMClone(dm, &crddm);CHKERRV(ierr); > PetscFE fe; > ierr = PetscFECreateDefault(PETSC_COMM_SELF, dim, dim, PETSC_FALSE, "", > PETSC_DECIDE, &fe);CHKERRV(ierr); > // ierr = PetscFECreateLagrange(PETSC_COMM_SELF, dim, dim, PETSC_FALSE, > "", PETSC_DECIDE, &fe);CHKERRQ(ierr); > ierr = PetscFESetFromOptions(fe);CHKERRV(ierr); > And you need to take care to construct 'fe' in the same way that you construct the PetscFE in 'dm'. (is there a better way to clone a PetscFE?) -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Sat Jul 25 10:15:42 2020 From: knepley at gmail.com (Matthew Knepley) Date: Sat, 25 Jul 2020 11:15:42 -0400 Subject: [petsc-users] coordinates of vertices of a cell in 3D DMPlex In-Reply-To: References: Message-ID: On Sat, Jul 25, 2020 at 4:10 AM Swarnava Ghosh wrote: > Dear Petsc users, > > I had a trivial question about DMPlex. Suppose I have a 3D mesh of > tetrahedrons. I want to find out the 3D coordinates of the vertices of a > particular cell. What would be the function to do this? > Lots of good responses. You can see that it is taking some time for canonical patterns to emerge as the "right way" to do something. The reason we provide multiple layers of interface is that user codes rely on different abstractions and would like to interact with PETSc using different assumptions. If you just want the coordinates of each vertex in some cell with the vertices in a canonical ordering, you can do as Mark suggested, with a slight modification: DM plex, cdm; Vec coordinates; ierr = DMGetCoordinateDM(dm, &cdm);CHKERRQ(ierr); ierr = DMGetCoordinatesLocal(dm, &coordinates);CHKERRQ(ierr); ierr = DMConvert(cdm, DMPLEX, &plex);CHKERRQ(ierr); ierr = DMPlexVecGetClosure(plex, NULL, coordinates, cell, NULL, &coef);CHKERRQ(ierr); .... ierr = DMPlexVecRestoreClosure(plex, NULL, coordinates, cell, NULL, &coef);CHKERRQ(ierr); We get a local coordinate vector, because local vectors are guaranteed to store everything in the closure of anything in the Plex. Global vectors are non-overlapping partitions, suitable for solvers, and might not have some of the values. Thanks, Matt > Thank you, > SG > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From swarnava89 at gmail.com Sun Jul 26 14:41:51 2020 From: swarnava89 at gmail.com (Swarnava Ghosh) Date: Sun, 26 Jul 2020 12:41:51 -0700 Subject: [petsc-users] coordinates of vertices of a cell in 3D DMPlex In-Reply-To: References: Message-ID: Dear Mark and Matt, Thank you for your suggestions. I tried what you mentioned. It works for one call of DMPlexVecGetClosure+DMPlexVecRestoreClosure. However, in my code, I would need to loop over cell values, and this is when it crashes. A vanilla code snippet is ierr=DMGetCoordinateDM(pCgdft->dmplexloc,&pCgdft->cdm);CHKERRQ(ierr); ierr=DMGetCoordinatesLocal(pCgdft->dmplexloc,&pCgdft->VCloc);CHKERRQ(ierr); ierr=DMConvert(pCgdft->cdm,DMPLEX,&pCgdft->cdmplexloc);CHKERRQ(ierr); ierr=DMGetCoordinatesLocal(pCgdft->dmplexloc,&pCgdft->VCloc);CHKERRQ(ierr); ierr=DMPlexVecGetClosure(pCgdft->cdmplexloc,NULL,pCgdft->VCloc,cell,&closureSize,&coef);CHKERRQ(ierr); // first call if(rank==0) { for(i=0;icdmplexloc,NULL,pCgdft->VCloc,cell,&closureSize,&coef);CHKERRQ(ierr); // second call ierr=DMPlexVecGetClosure(pCgdft->cdmplexloc,NULL,pCgdft->VCloc,cell,&closureSize,&coef);CHKERRQ(ierr); if(rank==0) { for(i=0;icdmplexloc,NULL,pCgdft->VCloc,cell,&closureSize,&coef);CHKERRQ(ierr); the first printf statement gets printed, and then it crashes with the following error message [44]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [44]PETSC ERROR: Object is in wrong state [44]PETSC ERROR: Array was not checked out [44]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [44]PETSC ERROR: Petsc Release Version 3.8.4, Mar, 24, 2018 [44]PETSC ERROR: ../lib/cgdft on a arch-linux2-c-opt named hpc-82-22.cm.cluster by swarnava Sun Jul 26 12:26:40 2020 [44]PETSC ERROR: Configure options --prefix=/software/PETSc/3.8.4-intel --with-cc=mpicc --with-cxx=mpicxx --with-fc=mpif90 --with-blaslapack-dir=/software/Intel/2018.1/compilers_and_libraries_2018.1.163/linux/mkl\ /lib/intel64_lin --with-debugging=no --with-shared-libraries=0 --download-metis --download-parmetis --download-superlu_dist [44]PETSC ERROR: #1 DMRestoreWorkArray() line 1281 in /groups/hpc-support/install/PETSc/petsc-3.8.4_intel_no-debug/src/dm/interface/dm.c [44]PETSC ERROR: #2 DMPlexVecRestoreClosure() line 4099 in /groups/hpc-support/install/PETSc/petsc-3.8.4_intel_no-debug/src/dm/impls/plex/plex.c [44]PETSC ERROR: #3 CreateLocalDMPLEX() line 749 in ./src/cgdft_localdmplex.cc Also attached is the error file. Sincerely, SG On Sat, Jul 25, 2020 at 8:15 AM Matthew Knepley wrote: > On Sat, Jul 25, 2020 at 4:10 AM Swarnava Ghosh > wrote: > >> Dear Petsc users, >> >> I had a trivial question about DMPlex. Suppose I have a 3D mesh of >> tetrahedrons. I want to find out the 3D coordinates of the vertices of a >> particular cell. What would be the function to do this? >> > > Lots of good responses. You can see that it is taking some time for > canonical patterns to emerge as the "right way" to do something. The reason > we provide > multiple layers of interface is that user codes rely on different > abstractions and would like to interact with PETSc using different > assumptions. > > If you just want the coordinates of each vertex in some cell with the > vertices in a canonical ordering, you can do as Mark suggested, with a > slight modification: > > DM plex, cdm; > Vec coordinates; > > ierr = DMGetCoordinateDM(dm, &cdm);CHKERRQ(ierr); > ierr = DMGetCoordinatesLocal(dm, &coordinates);CHKERRQ(ierr); > ierr = DMConvert(cdm, DMPLEX, &plex);CHKERRQ(ierr); > ierr = DMPlexVecGetClosure(plex, NULL, coordinates, cell, NULL, > &coef);CHKERRQ(ierr); > .... > ierr = DMPlexVecRestoreClosure(plex, NULL, coordinates, cell, NULL, > &coef);CHKERRQ(ierr); > > We get a local coordinate vector, because local vectors are guaranteed to > store everything in the closure of anything in the Plex. Global vectors > are non-overlapping partitions, suitable for solvers, and might not have > some of the values. > > Thanks, > > Matt > > >> Thank you, >> SG >> > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ > > -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: MgVNx3Ny3Nz3eta1.6297.Err_out Type: application/octet-stream Size: 140619 bytes Desc: not available URL: From xsli at lbl.gov Sun Jul 26 14:45:09 2020 From: xsli at lbl.gov (Xiaoye S. Li) Date: Sun, 26 Jul 2020 12:45:09 -0700 Subject: [petsc-users] SUPERLU_DIST in single precision In-Reply-To: <20200722150403.Horde.NO015I2GI2E-524J-SXqwui@webmail.mpcdf.mpg.de> References: <20200721155807.Horde.5ZYOzrQ7dmwNTQquK_6ebeS@webmail.mpcdf.mpg.de> <20200722150403.Horde.NO015I2GI2E-524J-SXqwui@webmail.mpcdf.mpg.de> Message-ID: Barry, I have a branch 'Mixed-precision' working with single precision FP32. I assume Felix wants to use superlu_dist from petsc. How do you want to incorporate it in petsc? https://github.com/xiaoyeli/superlu_dist PS1: in this version, FP32 only works on CPU. FP64 and complex-FP64 all work on GPU. PS2: currently there is no mixed-precision yet, but it is the branch we are adding mix-prec support. Will take a while before merging to master. Sherry On Wed, Jul 22, 2020 at 6:04 AM wrote: > Hi Barry, > for now I just want to run everything in single on CPUs only with > SUPERLU_DIST. Maybe we will also incorporate GPUs in the future, but > there are no immediate plans yet. So if you could provide the support, > that would be awesome. > > Best regards, > Felix > > Zitat von Barry Smith : > > > Felix, > > > > What are your needs, do you want this for CPUs or for GPUs? Do > > you wish to run all your code in single precision or just the > > SuperLU_Dist solver while the rest of your code double? > > > > If you want to run everything on CPUs using single precision > > then adding the support is very easy, we can provide that for you > > any time. The other cases will require more thought. > > > > Barry > > > > > >> On Jul 21, 2020, at 8:58 AM, flw at rzg.mpg.de wrote: > >> > >> Dear PETSc support team, > >> some time ago you told me that you are planning on releasing a > >> version that supports SUPERLU_DIST in single-precision soon. Can > >> you tell me roughly what time frame you had in mind? > >> > >> Best regards, > >> Felix > >> > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Sun Jul 26 15:23:23 2020 From: knepley at gmail.com (Matthew Knepley) Date: Sun, 26 Jul 2020 16:23:23 -0400 Subject: [petsc-users] coordinates of vertices of a cell in 3D DMPlex In-Reply-To: References: Message-ID: On Sun, Jul 26, 2020 at 3:42 PM Swarnava Ghosh wrote: > Dear Mark and Matt, > > Thank you for your suggestions. I tried what you mentioned. It works for > one call of DMPlexVecGetClosure+DMPlexVecRestoreClosure. > However, in my code, I would need to loop over cell values, and this is > when it crashes. A vanilla code snippet is > Reset the input array to NULL for each invocation of GetClosure(). Thanks, Matt > ierr=DMGetCoordinateDM(pCgdft->dmplexloc,&pCgdft->cdm);CHKERRQ(ierr); > > ierr=DMGetCoordinatesLocal(pCgdft->dmplexloc,&pCgdft->VCloc);CHKERRQ(ierr); > ierr=DMConvert(pCgdft->cdm,DMPLEX,&pCgdft->cdmplexloc);CHKERRQ(ierr); > > > ierr=DMGetCoordinatesLocal(pCgdft->dmplexloc,&pCgdft->VCloc);CHKERRQ(ierr); > > ierr=DMPlexVecGetClosure(pCgdft->cdmplexloc,NULL,pCgdft->VCloc,cell,&closureSize,&coef);CHKERRQ(ierr); > > // first call > if(rank==0) > { > for(i=0;i { > printf("&& cell=0, first call, coef[%d]=%lf \n",i,coef[i]); > } > } > > ierr=DMPlexVecRestoreClosure(pCgdft->cdmplexloc,NULL,pCgdft->VCloc,cell,&closureSize,&coef);CHKERRQ(ierr); > > // second call > > ierr=DMPlexVecGetClosure(pCgdft->cdmplexloc,NULL,pCgdft->VCloc,cell,&closureSize,&coef);CHKERRQ(ierr); > > if(rank==0) > { > for(i=0;i { > printf("&& cell=0, second call, coef[%d]=%lf \n",i,coef[i]); > } > } > > ierr=DMPlexVecRestoreClosure(pCgdft->cdmplexloc,NULL,pCgdft->VCloc,cell,&closureSize,&coef);CHKERRQ(ierr); > > > the first printf statement gets printed, and then it crashes with the > following error message > > [44]PETSC ERROR: --------------------- Error Message > -------------------------------------------------------------- > [44]PETSC ERROR: Object is in wrong state > [44]PETSC ERROR: Array was not checked out > [44]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html > for trouble shooting. > [44]PETSC ERROR: Petsc Release Version 3.8.4, Mar, 24, 2018 > [44]PETSC ERROR: ../lib/cgdft on a arch-linux2-c-opt named > hpc-82-22.cm.cluster by swarnava Sun Jul 26 12:26:40 2020 > [44]PETSC ERROR: Configure options --prefix=/software/PETSc/3.8.4-intel > --with-cc=mpicc --with-cxx=mpicxx --with-fc=mpif90 > --with-blaslapack-dir=/software/Intel/2018.1/compilers_and_libraries_2018.1.163/linux/mkl\ > /lib/intel64_lin --with-debugging=no --with-shared-libraries=0 > --download-metis --download-parmetis --download-superlu_dist > [44]PETSC ERROR: #1 DMRestoreWorkArray() line 1281 in > /groups/hpc-support/install/PETSc/petsc-3.8.4_intel_no-debug/src/dm/interface/dm.c > [44]PETSC ERROR: #2 DMPlexVecRestoreClosure() line 4099 in > /groups/hpc-support/install/PETSc/petsc-3.8.4_intel_no-debug/src/dm/impls/plex/plex.c > [44]PETSC ERROR: #3 CreateLocalDMPLEX() line 749 in > ./src/cgdft_localdmplex.cc > > Also attached is the error file. > > Sincerely, > SG > > > > On Sat, Jul 25, 2020 at 8:15 AM Matthew Knepley wrote: > >> On Sat, Jul 25, 2020 at 4:10 AM Swarnava Ghosh >> wrote: >> >>> Dear Petsc users, >>> >>> I had a trivial question about DMPlex. Suppose I have a 3D mesh of >>> tetrahedrons. I want to find out the 3D coordinates of the vertices of a >>> particular cell. What would be the function to do this? >>> >> >> Lots of good responses. You can see that it is taking some time for >> canonical patterns to emerge as the "right way" to do something. The reason >> we provide >> multiple layers of interface is that user codes rely on different >> abstractions and would like to interact with PETSc using different >> assumptions. >> >> If you just want the coordinates of each vertex in some cell with the >> vertices in a canonical ordering, you can do as Mark suggested, with a >> slight modification: >> >> DM plex, cdm; >> Vec coordinates; >> >> ierr = DMGetCoordinateDM(dm, &cdm);CHKERRQ(ierr); >> ierr = DMGetCoordinatesLocal(dm, &coordinates);CHKERRQ(ierr); >> ierr = DMConvert(cdm, DMPLEX, &plex);CHKERRQ(ierr); >> ierr = DMPlexVecGetClosure(plex, NULL, coordinates, cell, NULL, >> &coef);CHKERRQ(ierr); >> .... >> ierr = DMPlexVecRestoreClosure(plex, NULL, coordinates, cell, NULL, >> &coef);CHKERRQ(ierr); >> >> We get a local coordinate vector, because local vectors are guaranteed to >> store everything in the closure of anything in the Plex. Global vectors >> are non-overlapping partitions, suitable for solvers, and might not have >> some of the values. >> >> Thanks, >> >> Matt >> >> >>> Thank you, >>> SG >>> >> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> -- Norbert Wiener >> >> https://www.cse.buffalo.edu/~knepley/ >> >> > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From swarnava89 at gmail.com Sun Jul 26 16:27:59 2020 From: swarnava89 at gmail.com (Swarnava Ghosh) Date: Sun, 26 Jul 2020 14:27:59 -0700 Subject: [petsc-users] coordinates of vertices of a cell in 3D DMPlex In-Reply-To: References: Message-ID: Got it! Thanks a lot. Sincerely, Swarnava On Sun, Jul 26, 2020 at 1:23 PM Matthew Knepley wrote: > On Sun, Jul 26, 2020 at 3:42 PM Swarnava Ghosh > wrote: > >> Dear Mark and Matt, >> >> Thank you for your suggestions. I tried what you mentioned. It works for >> one call of DMPlexVecGetClosure+DMPlexVecRestoreClosure. >> However, in my code, I would need to loop over cell values, and this is >> when it crashes. A vanilla code snippet is >> > > Reset the input array to NULL for each invocation of GetClosure(). > > Thanks, > > Matt > > >> ierr=DMGetCoordinateDM(pCgdft->dmplexloc,&pCgdft->cdm);CHKERRQ(ierr); >> >> ierr=DMGetCoordinatesLocal(pCgdft->dmplexloc,&pCgdft->VCloc);CHKERRQ(ierr); >> ierr=DMConvert(pCgdft->cdm,DMPLEX,&pCgdft->cdmplexloc);CHKERRQ(ierr); >> >> >> ierr=DMGetCoordinatesLocal(pCgdft->dmplexloc,&pCgdft->VCloc);CHKERRQ(ierr); >> >> ierr=DMPlexVecGetClosure(pCgdft->cdmplexloc,NULL,pCgdft->VCloc,cell,&closureSize,&coef);CHKERRQ(ierr); >> >> // first call >> if(rank==0) >> { >> for(i=0;i> { >> printf("&& cell=0, first call, coef[%d]=%lf \n",i,coef[i]); >> } >> } >> >> ierr=DMPlexVecRestoreClosure(pCgdft->cdmplexloc,NULL,pCgdft->VCloc,cell,&closureSize,&coef);CHKERRQ(ierr); >> >> // second call >> >> ierr=DMPlexVecGetClosure(pCgdft->cdmplexloc,NULL,pCgdft->VCloc,cell,&closureSize,&coef);CHKERRQ(ierr); >> >> if(rank==0) >> { >> for(i=0;i> { >> printf("&& cell=0, second call, coef[%d]=%lf \n",i,coef[i]); >> } >> } >> >> ierr=DMPlexVecRestoreClosure(pCgdft->cdmplexloc,NULL,pCgdft->VCloc,cell,&closureSize,&coef);CHKERRQ(ierr); >> >> >> the first printf statement gets printed, and then it crashes with the >> following error message >> >> [44]PETSC ERROR: --------------------- Error Message >> -------------------------------------------------------------- >> [44]PETSC ERROR: Object is in wrong state >> [44]PETSC ERROR: Array was not checked out >> [44]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html >> for trouble shooting. >> [44]PETSC ERROR: Petsc Release Version 3.8.4, Mar, 24, 2018 >> [44]PETSC ERROR: ../lib/cgdft on a arch-linux2-c-opt named >> hpc-82-22.cm.cluster by swarnava Sun Jul 26 12:26:40 2020 >> [44]PETSC ERROR: Configure options --prefix=/software/PETSc/3.8.4-intel >> --with-cc=mpicc --with-cxx=mpicxx --with-fc=mpif90 >> --with-blaslapack-dir=/software/Intel/2018.1/compilers_and_libraries_2018.1.163/linux/mkl\ >> /lib/intel64_lin --with-debugging=no --with-shared-libraries=0 >> --download-metis --download-parmetis --download-superlu_dist >> [44]PETSC ERROR: #1 DMRestoreWorkArray() line 1281 in >> /groups/hpc-support/install/PETSc/petsc-3.8.4_intel_no-debug/src/dm/interface/dm.c >> [44]PETSC ERROR: #2 DMPlexVecRestoreClosure() line 4099 in >> /groups/hpc-support/install/PETSc/petsc-3.8.4_intel_no-debug/src/dm/impls/plex/plex.c >> [44]PETSC ERROR: #3 CreateLocalDMPLEX() line 749 in >> ./src/cgdft_localdmplex.cc >> >> Also attached is the error file. >> >> Sincerely, >> SG >> >> >> >> On Sat, Jul 25, 2020 at 8:15 AM Matthew Knepley >> wrote: >> >>> On Sat, Jul 25, 2020 at 4:10 AM Swarnava Ghosh >>> wrote: >>> >>>> Dear Petsc users, >>>> >>>> I had a trivial question about DMPlex. Suppose I have a 3D mesh of >>>> tetrahedrons. I want to find out the 3D coordinates of the vertices of a >>>> particular cell. What would be the function to do this? >>>> >>> >>> Lots of good responses. You can see that it is taking some time for >>> canonical patterns to emerge as the "right way" to do something. The reason >>> we provide >>> multiple layers of interface is that user codes rely on different >>> abstractions and would like to interact with PETSc using different >>> assumptions. >>> >>> If you just want the coordinates of each vertex in some cell with the >>> vertices in a canonical ordering, you can do as Mark suggested, with a >>> slight modification: >>> >>> DM plex, cdm; >>> Vec coordinates; >>> >>> ierr = DMGetCoordinateDM(dm, &cdm);CHKERRQ(ierr); >>> ierr = DMGetCoordinatesLocal(dm, &coordinates);CHKERRQ(ierr); >>> ierr = DMConvert(cdm, DMPLEX, &plex);CHKERRQ(ierr); >>> ierr = DMPlexVecGetClosure(plex, NULL, coordinates, cell, NULL, >>> &coef);CHKERRQ(ierr); >>> .... >>> ierr = DMPlexVecRestoreClosure(plex, NULL, coordinates, cell, NULL, >>> &coef);CHKERRQ(ierr); >>> >>> We get a local coordinate vector, because local vectors are guaranteed >>> to store everything in the closure of anything in the Plex. Global vectors >>> are non-overlapping partitions, suitable for solvers, and might not have >>> some of the values. >>> >>> Thanks, >>> >>> Matt >>> >>> >>>> Thank you, >>>> SG >>>> >>> >>> >>> -- >>> What most experimenters take for granted before they begin their >>> experiments is infinitely more interesting than any results to which their >>> experiments lead. >>> -- Norbert Wiener >>> >>> https://www.cse.buffalo.edu/~knepley/ >>> >>> >> > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From rmondaini at csrc.ac.cn Mon Jul 27 02:06:24 2020 From: rmondaini at csrc.ac.cn (Rubem Mondaini) Date: Mon, 27 Jul 2020 15:06:24 +0800 Subject: [petsc-users] Installation failure Message-ID: <1e44c082-ceef-db8c-65b2-0a92375f3d4b@csrc.ac.cn> Dears, I have tried to install the PETSc library on my machine and I am failing. I have managed to install it on another one, which possess similar configurations for the compilers, and there things do seem to be very much fine. I am attaching the corresponding configure.log. Is there something simple that I am missing? Best, -------------- next part -------------- A non-text attachment was scrubbed... Name: configure.log Type: text/x-log Size: 303575 bytes Desc: not available URL: From knepley at gmail.com Mon Jul 27 05:34:00 2020 From: knepley at gmail.com (Matthew Knepley) Date: Mon, 27 Jul 2020 06:34:00 -0400 Subject: [petsc-users] Installation failure In-Reply-To: <1e44c082-ceef-db8c-65b2-0a92375f3d4b@csrc.ac.cn> References: <1e44c082-ceef-db8c-65b2-0a92375f3d4b@csrc.ac.cn> Message-ID: On Mon, Jul 27, 2020 at 3:08 AM Rubem Mondaini wrote: > Dears, > > I have tried to install the PETSc library on my machine and I am > failing. I have managed to install it on another one, which possess > similar configurations for the compilers, and there things do seem to be > very much fine. > > I am attaching the corresponding configure.log. Is there something > simple that I am missing? > Your configuration does not need C++, so you can use --with-cxx=0 If you want to fix it, the first thing to try is turning off autodetection --with-cxxlib-autodetect=0 which usually means you need something like --LIBS=-lstdc++ Thanks, Matt > Best, > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From flw at rzg.mpg.de Mon Jul 27 14:10:54 2020 From: flw at rzg.mpg.de (flw at rzg.mpg.de) Date: Mon, 27 Jul 2020 21:10:54 +0200 Subject: [petsc-users] SUPERLU_DIST in single precision In-Reply-To: References: <20200721155807.Horde.5ZYOzrQ7dmwNTQquK_6ebeS@webmail.mpcdf.mpg.de> <20200722150403.Horde.NO015I2GI2E-524J-SXqwui@webmail.mpcdf.mpg.de> Message-ID: <20200727211054.Horde.ZgzkW110frSEh8mw1TA2Zdn@webmail.mpcdf.mpg.de> Hi Shery, Yes, ideally we would like to compile PETSc in single precision and simply run a single precision version of SUPERLU_DIST just like e.g. MUMPS. Best regards and thanks, Felix Zitat von "Xiaoye S. Li" : > Barry, > > I have a branch 'Mixed-precision' working with single precision FP32. I > assume Felix wants to use superlu_dist from petsc. How do you want to > incorporate it in petsc? > > https://github.com/xiaoyeli/superlu_dist > > PS1: in this version, FP32 only works on CPU. FP64 and complex-FP64 all > work on GPU. > > PS2: currently there is no mixed-precision yet, but it is the branch we are > adding mix-prec support. Will take a while before merging to master. > > Sherry > > > On Wed, Jul 22, 2020 at 6:04 AM wrote: > >> Hi Barry, >> for now I just want to run everything in single on CPUs only with >> SUPERLU_DIST. Maybe we will also incorporate GPUs in the future, but >> there are no immediate plans yet. So if you could provide the support, >> that would be awesome. >> >> Best regards, >> Felix >> >> Zitat von Barry Smith : >> >> > Felix, >> > >> > What are your needs, do you want this for CPUs or for GPUs? Do >> > you wish to run all your code in single precision or just the >> > SuperLU_Dist solver while the rest of your code double? >> > >> > If you want to run everything on CPUs using single precision >> > then adding the support is very easy, we can provide that for you >> > any time. The other cases will require more thought. >> > >> > Barry >> > >> > >> >> On Jul 21, 2020, at 8:58 AM, flw at rzg.mpg.de wrote: >> >> >> >> Dear PETSc support team, >> >> some time ago you told me that you are planning on releasing a >> >> version that supports SUPERLU_DIST in single-precision soon. Can >> >> you tell me roughly what time frame you had in mind? >> >> >> >> Best regards, >> >> Felix >> >> >> >> >> >> From sajidsyed2021 at u.northwestern.edu Mon Jul 27 17:07:17 2020 From: sajidsyed2021 at u.northwestern.edu (Sajid Ali) Date: Mon, 27 Jul 2020 17:07:17 -0500 Subject: [petsc-users] VecLoad into a SubVector ? Message-ID: Hi PETSc-developers, When I load data (using VecLoad) into a subvector, the parent vector does not seem to get the data after the subvector is restored. I tried doing a VecSet to verify that the index set (used to select the subvector) is valid and the values set by VecSet are transferred back to the parent vector. Could anyone point out if I'm missing something when I try transferring the data via a VecLoad from a subvector to the parent vector ? I'm attaching the code for selecting the subvector and loading data along with a test hdf5 file (filled with random values). I expect the output `testload.h5` to be a vector of size 100 with the first 50 being 1 and the rest being the input values. Thank You, Sajid Ali | PhD Candidate Applied Physics Northwestern University s-sajid-ali.github.io -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: testinput.h5 Type: application/octet-stream Size: 2448 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: ex2.c Type: application/octet-stream Size: 2377 bytes Desc: not available URL: From eda.oktay at metu.edu.tr Mon Jul 27 17:49:15 2020 From: eda.oktay at metu.edu.tr (Eda Oktay) Date: Tue, 28 Jul 2020 01:49:15 +0300 Subject: [petsc-users] SLEPc download error Message-ID: Hi all, I am trying to download Petsc with Slepc a linux computer (probably ubuntu, I don't know because I am using it via ssh) but I am getting the following error: ******************************************************************************* UNABLE to CONFIGURE with GIVEN OPTIONS (see configure.log for details): ------------------------------------------------------------------------------- Error during download/extract/detection of SLEPC: Unable to download package SLEPC from: https://gitlab.com/slepc/slepc/-/archive/v3.13.2/slepc-v3.13.2.tar.gz * If URL specified manually - perhaps there is a typo? * If your network is disconnected - please reconnect and rerun ./configure * Or perhaps you have a firewall blocking the download * You can run with --with-packages-download-dir=/adirectory and ./configure will instruct you what packages to download manually * or you can download the above URL manually, to /yourselectedlocation/slepc-v3.13.2.tar.gz and use the configure option: --download-slepc=/yourselectedlocation/slepc-v3.13.2.tar.gz ******************************************************************************* I attached configure.log. Then, I downloaded slepc and used --download-slepc=/yourselectedlocation/slepc-v3.13.2.tar.gz. Since I used WinSCP, tar.gz file was uploaded to my user as "eda" and I used --download-slepc=/home/eda/eda and ran configure again. However, I got the following error: ******************************************************************************* UNABLE to CONFIGURE with GIVEN OPTIONS (see configure.log for details): ------------------------------------------------------------------------------- Error during download/extract/detection of SLEPC: Unknown compression type in URL: file:///home/eda/eda ******************************************************************************* I also attached this confiigure.log as 2configure.log. When I tried to untar eda, I could and I got slepc. So I don't understand why I am getting this error. This is the third computer I am building Petsc with Slepc but this is the first time I am getting such an error. Can you help me with this problem? Thanks! Eda -------------- next part -------------- A non-text attachment was scrubbed... Name: configure.log Type: application/octet-stream Size: 884240 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: 2configure.log Type: application/octet-stream Size: 882738 bytes Desc: not available URL: From knepley at gmail.com Mon Jul 27 19:06:19 2020 From: knepley at gmail.com (Matthew Knepley) Date: Mon, 27 Jul 2020 20:06:19 -0400 Subject: [petsc-users] VecLoad into a SubVector ? In-Reply-To: References: Message-ID: On Mon, Jul 27, 2020 at 6:09 PM Sajid Ali wrote: > Hi PETSc-developers, > > When I load data (using VecLoad) into a subvector, the parent vector does > not seem to get the data after the subvector is restored. I tried doing a > VecSet to verify that the index set (used to select the subvector) is valid > and the values set by VecSet are transferred back to the parent vector. > Could anyone point out if I'm missing something when I try transferring the > data via a VecLoad from a subvector to the parent vector ? > > I'm attaching the code for selecting the subvector and loading data along > with a test hdf5 file (filled with random values). I expect the output > `testload.h5` to be a vector of size 100 with the first 50 being 1 and the > rest being the input values. > You should be able to see with the debugger why the update does not happen. My guess is that the state is not updated. It should be updated by VecReplaceArray(), so I do not understand, but you should be able to see it clearly in gdb. Thanks, Matt > Thank You, > Sajid Ali | PhD Candidate > Applied Physics > Northwestern University > s-sajid-ali.github.io > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Mon Jul 27 19:08:14 2020 From: knepley at gmail.com (Matthew Knepley) Date: Mon, 27 Jul 2020 20:08:14 -0400 Subject: [petsc-users] SLEPc download error In-Reply-To: References: Message-ID: On Mon, Jul 27, 2020 at 6:50 PM Eda Oktay wrote: > Hi all, > > I am trying to download Petsc with Slepc a linux computer (probably > ubuntu, I don't know because I am using it via ssh) but I am getting > the following error: > > > ******************************************************************************* > UNABLE to CONFIGURE with GIVEN OPTIONS (see configure.log > for details): > > ------------------------------------------------------------------------------- > Error during download/extract/detection of SLEPC: > Unable to download package SLEPC from: > https://gitlab.com/slepc/slepc/-/archive/v3.13.2/slepc-v3.13.2.tar.gz > * If URL specified manually - perhaps there is a typo? > * If your network is disconnected - please reconnect and rerun ./configure > * Or perhaps you have a firewall blocking the download > * You can run with --with-packages-download-dir=/adirectory and > ./configure will instruct you what packages to download manually > * or you can download the above URL manually, to > /yourselectedlocation/slepc-v3.13.2.tar.gz > and use the configure option: > --download-slepc=/yourselectedlocation/slepc-v3.13.2.tar.gz > > ******************************************************************************* > > I attached configure.log. > > Then, I downloaded slepc and used > --download-slepc=/yourselectedlocation/slepc-v3.13.2.tar.gz. > > Since I used WinSCP, tar.gz file was uploaded to my user as "eda" and > I used --download-slepc=/home/eda/eda and ran configure again. > > However, I got the following error: > > ******************************************************************************* > UNABLE to CONFIGURE with GIVEN OPTIONS (see configure.log > for details): > > ------------------------------------------------------------------------------- > Error during download/extract/detection of SLEPC: > Unknown compression type in URL: file:///home/eda/eda > > ******************************************************************************* > I also attached this confiigure.log as 2configure.log. > > When I tried to untar eda, I could and I got slepc. So I don't > understand why I am getting this error. This is the third computer I > am building Petsc with Slepc but this is the first time I am getting > such an error. > > Can you help me with this problem? > Rename your downloaded file to have the .tar.gz suffix. Thanks, Matt > Thanks! > Eda > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From balay at mcs.anl.gov Mon Jul 27 20:38:06 2020 From: balay at mcs.anl.gov (Satish Balay) Date: Mon, 27 Jul 2020 20:38:06 -0500 (CDT) Subject: [petsc-users] SLEPc download error In-Reply-To: References: Message-ID: Looks like python urlretrive() is failing on glitab tarball urls [but its working with wget etc. I don't know why. Perhaps its best to have git installed - so that git-urls work.. Satish On Tue, 28 Jul 2020, Eda Oktay wrote: > Hi all, > > I am trying to download Petsc with Slepc a linux computer (probably > ubuntu, I don't know because I am using it via ssh) but I am getting > the following error: > > ******************************************************************************* > UNABLE to CONFIGURE with GIVEN OPTIONS (see configure.log > for details): > ------------------------------------------------------------------------------- > Error during download/extract/detection of SLEPC: > Unable to download package SLEPC from: > https://gitlab.com/slepc/slepc/-/archive/v3.13.2/slepc-v3.13.2.tar.gz > * If URL specified manually - perhaps there is a typo? > * If your network is disconnected - please reconnect and rerun ./configure > * Or perhaps you have a firewall blocking the download > * You can run with --with-packages-download-dir=/adirectory and > ./configure will instruct you what packages to download manually > * or you can download the above URL manually, to > /yourselectedlocation/slepc-v3.13.2.tar.gz > and use the configure option: > --download-slepc=/yourselectedlocation/slepc-v3.13.2.tar.gz > ******************************************************************************* > > I attached configure.log. > > Then, I downloaded slepc and used > --download-slepc=/yourselectedlocation/slepc-v3.13.2.tar.gz. > > Since I used WinSCP, tar.gz file was uploaded to my user as "eda" and > I used --download-slepc=/home/eda/eda and ran configure again. > > However, I got the following error: > ******************************************************************************* > UNABLE to CONFIGURE with GIVEN OPTIONS (see configure.log > for details): > ------------------------------------------------------------------------------- > Error during download/extract/detection of SLEPC: > Unknown compression type in URL: file:///home/eda/eda > ******************************************************************************* > I also attached this confiigure.log as 2configure.log. > > When I tried to untar eda, I could and I got slepc. So I don't > understand why I am getting this error. This is the third computer I > am building Petsc with Slepc but this is the first time I am getting > such an error. > > Can you help me with this problem? > > Thanks! > Eda > From balay at mcs.anl.gov Mon Jul 27 20:46:23 2020 From: balay at mcs.anl.gov (Satish Balay) Date: Mon, 27 Jul 2020 20:46:23 -0500 (CDT) Subject: [petsc-users] SLEPc download error In-Reply-To: References: Message-ID: [balay at pj01 petsc]$ curl -O https://gitlab.com/slepc/slepc/-/archive/v3.13.3/slepc-v3.13.3.tar.gz % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 1253k 0 1253k 0 0 16.1M 0 --:--:-- --:--:-- --:--:-- 16.1M [balay at pj01 petsc]$ file slepc-v3.13.3.tar.gz slepc-v3.13.3.tar.gz: gzip compressed data, from Unix, original size modulo 2^32 7690240 [balay at pj01 petsc]$ python Python 3.7.7 (default, Jun 4 2020, 15:43:14) [GCC 9.3.1 20200408 (Red Hat 9.3.1-2)] on linux Type "help", "copyright", "credits" or "license" for more information. >>> from urllib.request import urlretrieve >>> urlretrieve('https://gitlab.com/slepc/slepc/-/archive/v3.13.3/slepc-v3.13.3.tar.gz','x.tar.gz') Traceback (most recent call last): File "", line 1, in File "/usr/lib64/python3.7/urllib/request.py", line 247, in urlretrieve with contextlib.closing(urlopen(url, data)) as fp: File "/usr/lib64/python3.7/urllib/request.py", line 222, in urlopen return opener.open(url, data, timeout) File "/usr/lib64/python3.7/urllib/request.py", line 531, in open response = meth(req, response) File "/usr/lib64/python3.7/urllib/request.py", line 641, in http_response 'http', request, response, code, msg, hdrs) File "/usr/lib64/python3.7/urllib/request.py", line 569, in error return self._call_chain(*args) File "/usr/lib64/python3.7/urllib/request.py", line 503, in _call_chain result = func(*args) File "/usr/lib64/python3.7/urllib/request.py", line 649, in http_error_default raise HTTPError(req.full_url, code, msg, hdrs, fp) urllib.error.HTTPError: HTTP Error 403: Forbidden >>> [balay at pj01 petsc]$ python2 Python 2.7.18 (default, Apr 20 2020, 00:00:00) [GCC 9.3.1 20200408 (Red Hat 9.3.1-2)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> from urllib import urlretrieve >>> urlretrieve('https://gitlab.com/slepc/slepc/-/archive/v3.13.3/slepc-v3.13.3.tar.gz','x.tar.gz') ('x.tar.gz', ) >>> [1]+ Stopped python2 [balay at pj01 petsc]$ cat x.tar.gz error code: 1010[balay at pj01 petsc]$ file x.tar.gz x.tar.gz: ASCII text, with no line terminators [balay at pj01 petsc]$ On Mon, 27 Jul 2020, Satish Balay via petsc-users wrote: > Looks like python urlretrive() is failing on glitab tarball urls [but its working with wget etc. I don't know why. > > Perhaps its best to have git installed - so that git-urls work.. > > Satish > > On Tue, 28 Jul 2020, Eda Oktay wrote: > > > Hi all, > > > > I am trying to download Petsc with Slepc a linux computer (probably > > ubuntu, I don't know because I am using it via ssh) but I am getting > > the following error: > > > > ******************************************************************************* > > UNABLE to CONFIGURE with GIVEN OPTIONS (see configure.log > > for details): > > ------------------------------------------------------------------------------- > > Error during download/extract/detection of SLEPC: > > Unable to download package SLEPC from: > > https://gitlab.com/slepc/slepc/-/archive/v3.13.2/slepc-v3.13.2.tar.gz > > * If URL specified manually - perhaps there is a typo? > > * If your network is disconnected - please reconnect and rerun ./configure > > * Or perhaps you have a firewall blocking the download > > * You can run with --with-packages-download-dir=/adirectory and > > ./configure will instruct you what packages to download manually > > * or you can download the above URL manually, to > > /yourselectedlocation/slepc-v3.13.2.tar.gz > > and use the configure option: > > --download-slepc=/yourselectedlocation/slepc-v3.13.2.tar.gz > > ******************************************************************************* > > > > I attached configure.log. > > > > Then, I downloaded slepc and used > > --download-slepc=/yourselectedlocation/slepc-v3.13.2.tar.gz. > > > > Since I used WinSCP, tar.gz file was uploaded to my user as "eda" and > > I used --download-slepc=/home/eda/eda and ran configure again. > > > > However, I got the following error: > > ******************************************************************************* > > UNABLE to CONFIGURE with GIVEN OPTIONS (see configure.log > > for details): > > ------------------------------------------------------------------------------- > > Error during download/extract/detection of SLEPC: > > Unknown compression type in URL: file:///home/eda/eda > > ******************************************************************************* > > I also attached this confiigure.log as 2configure.log. > > > > When I tried to untar eda, I could and I got slepc. So I don't > > understand why I am getting this error. This is the third computer I > > am building Petsc with Slepc but this is the first time I am getting > > such an error. > > > > Can you help me with this problem? > > > > Thanks! > > Eda > > > From bsmith at petsc.dev Mon Jul 27 21:32:28 2020 From: bsmith at petsc.dev (Barry Smith) Date: Mon, 27 Jul 2020 21:32:28 -0500 Subject: [petsc-users] Installation failure In-Reply-To: <1e44c082-ceef-db8c-65b2-0a92375f3d4b@csrc.ac.cn> References: <1e44c082-ceef-db8c-65b2-0a92375f3d4b@csrc.ac.cn> Message-ID: Hmm, configure is a mess. Since you explicitly use MPI compilers --with-cc=mpiicc --with-cxx=mpiicpc --with-fc=mpiifort the --with-mpi-include=/opt/intel/compilers_and_libraries_2016/linux/mpi/intel64/include --with-mpi-lib=/opt/intel/compilers_and_libraries_2016.3.210/linux/mpi/intel64/lib/libmpicxx.a are not needed and should not be provided (Satish I thought this would generate an error? Why doesn't it?) Please try without those arguments and send the configure.log if it fails. The error message isn't completely right. Cannot run executables created with C. This is because the call the call to self.setCompilers.checkCompiler('C') doesn't have enough context to print a decent error message. Barry > On Jul 27, 2020, at 2:06 AM, Rubem Mondaini wrote: > > Dears, > > I have tried to install the PETSc library on my machine and I am failing. I have managed to install it on another one, which possess similar configurations for the compilers, and there things do seem to be very much fine. > > I am attaching the corresponding configure.log. Is there something simple that I am missing? > > Best, > > > From bsmith at petsc.dev Mon Jul 27 21:47:21 2020 From: bsmith at petsc.dev (Barry Smith) Date: Mon, 27 Jul 2020 21:47:21 -0500 Subject: [petsc-users] VecLoad into a SubVector ? In-Reply-To: References: Message-ID: <7F9E26AC-276F-40B5-B2F3-05098C379685@petsc.dev> I don't understand the logic of VecRestoreSubVector() at all. I don't see how it could work. } else { PETSC_UNUSED PetscObjectState dummystate = 0; PetscBool valid; ierr = PetscObjectComposedDataGetInt((PetscObject)*Y,VecGetSubVectorSavedStateId,dummystate,valid);CHKERRQ(ierr); if (!valid) { VecScatter scatter; ierr = PetscObjectQuery((PetscObject)*Y,"VecGetSubVector_Scatter",(PetscObject*)&scatter);CHKERRQ(ierr); if (scatter) { ierr = VecScatterBegin(scatter,*Y,X,INSERT_VALUES,SCATTER_REVERSE);CHKERRQ(ierr); ierr = VecScatterEnd(scatter,*Y,X,INSERT_VALUES,SCATTER_REVERSE);CHKERRQ(ierr); } } Seems completely wrong to me. It never compares the saved state of the Y vector with the current vector. Matt is right stepping through with the debugger could should why it is not doing the scatter. Barry > On Jul 27, 2020, at 7:06 PM, Matthew Knepley wrote: > > On Mon, Jul 27, 2020 at 6:09 PM Sajid Ali > wrote: > Hi PETSc-developers, > > When I load data (using VecLoad) into a subvector, the parent vector does not seem to get the data after the subvector is restored. I tried doing a VecSet to verify that the index set (used to select the subvector) is valid and the values set by VecSet are transferred back to the parent vector. Could anyone point out if I'm missing something when I try transferring the data via a VecLoad from a subvector to the parent vector ? > > I'm attaching the code for selecting the subvector and loading data along with a test hdf5 file (filled with random values). I expect the output `testload.h5` to be a vector of size 100 with the first 50 being 1 and the rest being the input values. > > You should be able to see with the debugger why the update does not happen. My guess is that the state is not updated. It should be updated > by VecReplaceArray(), so I do not understand, but you should be able to see it clearly in gdb. > > Thanks, > > Matt > > Thank You, > Sajid Ali | PhD Candidate > Applied Physics > Northwestern University > s-sajid-ali.github.io > > -- > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From sajidsyed2021 at u.northwestern.edu Mon Jul 27 22:52:37 2020 From: sajidsyed2021 at u.northwestern.edu (Sajid Ali) Date: Mon, 27 Jul 2020 22:52:37 -0500 Subject: [petsc-users] VecLoad into a SubVector ? In-Reply-To: <7F9E26AC-276F-40B5-B2F3-05098C379685@petsc.dev> References: <7F9E26AC-276F-40B5-B2F3-05098C379685@petsc.dev> Message-ID: Hi Barry/Matt, I now have a simpler test (attached with this email) for this bug which does the following : - Create a vector of size 50, set it to 2 and save to disk. - Create a vector of size 100, set it to 1 and extract the last 50 elements as a subvector. - Load the saved vector from disk into the subvector and restore the subvector. - Test for VecSum, it should be 150. With one mpi rank (implying no scatters were used in the creation of the subvector), the above works as expected when using binary IO but if one uses HDF5 for IO, VecSum outputs 100 showing that the subvector didn?t restore correctly. Running the executable in gdb I see that for both cases the VecRestoreSubVector reads the variable VecGetSubVectorSavedStateId as 4 with the boolean variable valid being false. My guess regarding the origin of the error with HDF5-IO is the fact that VecLoad_HDF5 uses a VecReplaceArray to load the data and this somehow messes up the assumptions regarding SubVector data pointers upon creation by using VecPlaceArray. As Barry mentioned the check for validity of subvector data is faulty and would need to be fixed and that should be able to transfer the subvector data back to the parent vector regardless of how the subvector is modified. Thank You, Sajid Ali | PhD Candidate Applied Physics Northwestern University s-sajid-ali.github.io -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: ex_subvecio.c Type: application/octet-stream Size: 3392 bytes Desc: not available URL: From sajidsyed2021 at u.northwestern.edu Mon Jul 27 23:34:33 2020 From: sajidsyed2021 at u.northwestern.edu (Sajid Ali) Date: Mon, 27 Jul 2020 23:34:33 -0500 Subject: [petsc-users] VecLoad into a SubVector ? In-Reply-To: References: <7F9E26AC-276F-40B5-B2F3-05098C379685@petsc.dev> Message-ID: Hi Barry/Matt, The fix to this bug would be to disable replacearray op on a subvector. I modified the source code for vecio.c forcing VecLoad_HDF5 to always perform an array copy and the above test passes for both binary and hdf5 viewers in serial and parallel. I can open a PR that adds a line Z->ops->replacearray = NULL; at line 1286 in the rvector.c file if one of you can confirm that the above logic is correct. The example attached in the last email could be used as a test for the same if necessary. Thank You, Sajid Ali | PhD Candidate Applied Physics Northwestern University s-sajid-ali.github.io -------------- next part -------------- An HTML attachment was scrubbed... URL: From rmondaini at csrc.ac.cn Tue Jul 28 00:25:08 2020 From: rmondaini at csrc.ac.cn (Rubem Mondaini) Date: Tue, 28 Jul 2020 13:25:08 +0800 Subject: [petsc-users] [SPAM] Re: Installation failure In-Reply-To: References: <1e44c082-ceef-db8c-65b2-0a92375f3d4b@csrc.ac.cn> Message-ID: Dear Barry, thank you very much for the reply. I removed the options "--with-mpi-include" and "--with-mpi-lib", but I still get the same error. I attach the configure.log file. I can confirm that I can compile and run other codes with my compilers, however. Many thanks! Rubem On 7/28/20 10:32 AM, Barry Smith wrote: > Hmm, configure is a mess. > > Since you explicitly use MPI compilers > > --with-cc=mpiicc --with-cxx=mpiicpc --with-fc=mpiifort > > the > > --with-mpi-include=/opt/intel/compilers_and_libraries_2016/linux/mpi/intel64/include --with-mpi-lib=/opt/intel/compilers_and_libraries_2016.3.210/linux/mpi/intel64/lib/libmpicxx.a > > are not needed and should not be provided (Satish I thought this would generate an error? Why doesn't it?) > > Please try without those arguments and send the configure.log if it fails. > > The error message isn't completely right. > > Cannot run executables created with C. > > This is because the call the call to self.setCompilers.checkCompiler('C') doesn't have enough context to print a decent error message. > > Barry > > >> On Jul 27, 2020, at 2:06 AM, Rubem Mondaini wrote: >> >> Dears, >> >> I have tried to install the PETSc library on my machine and I am failing. I have managed to install it on another one, which possess similar configurations for the compilers, and there things do seem to be very much fine. >> >> I am attaching the corresponding configure.log. Is there something simple that I am missing? >> >> Best, >> >> >> -------------- next part -------------- A non-text attachment was scrubbed... Name: configure.log Type: text/x-log Size: 308361 bytes Desc: not available URL: From balay at mcs.anl.gov Tue Jul 28 00:38:09 2020 From: balay at mcs.anl.gov (Satish Balay) Date: Tue, 28 Jul 2020 00:38:09 -0500 (CDT) Subject: [petsc-users] [SPAM] Re: Installation failure In-Reply-To: References: <1e44c082-ceef-db8c-65b2-0a92375f3d4b@csrc.ac.cn> Message-ID: something went wrong - but petsc configure didn't capture this error. You can retry with: ./configure PETSC_ARCH=linux-intel --with-cc=mpiicc --with-cxx=0 --with-fc=mpiifort --with-blaslapack-dir=/opt/intel/compilers_and_libraries_2016/linux/mkl --with-scalar-type=complex --with-64-bit-indices And perhaps add one more option: LIBS="-Bstatic -lifcore -Bdynamic" Note: --with-64-bit-blas-indices is not appropriate here. Satish On Tue, 28 Jul 2020, Rubem Mondaini wrote: > Dear Barry, > > thank you very much for the reply. I removed the options "--with-mpi-include" > and "--with-mpi-lib", but I still get the same error. I attach the > configure.log file. > > I can confirm that I can compile and run other codes with my compilers, > however. > > Many thanks! > > Rubem > > On 7/28/20 10:32 AM, Barry Smith wrote: > > Hmm, configure is a mess. > > > > Since you explicitly use MPI compilers > > > > --with-cc=mpiicc --with-cxx=mpiicpc --with-fc=mpiifort > > > > the > > > > --with-mpi-include=/opt/intel/compilers_and_libraries_2016/linux/mpi/intel64/include > > --with-mpi-lib=/opt/intel/compilers_and_libraries_2016.3.210/linux/mpi/intel64/lib/libmpicxx.a > > > > are not needed and should not be provided (Satish I thought this would > > generate an error? Why doesn't it?) > > > > Please try without those arguments and send the configure.log if it > > fails. > > > > The error message isn't completely right. > > > > Cannot run executables created with C. > > > > This is because the call the call to > > self.setCompilers.checkCompiler('C') doesn't have enough context to > > print a decent error message. > > > > Barry > > > > > >> On Jul 27, 2020, at 2:06 AM, Rubem Mondaini wrote: > >> > >> Dears, > >> > >> I have tried to install the PETSc library on my machine and I am failing. I > >> have managed to install it on another one, which possess similar > >> configurations for the compilers, and there things do seem to be very much > >> fine. > >> > >> I am attaching the corresponding configure.log. Is there something simple > >> that I am missing? > >> > >> Best, > >> > >> > >> > From rmondaini at csrc.ac.cn Tue Jul 28 01:38:38 2020 From: rmondaini at csrc.ac.cn (Rubem Mondaini) Date: Tue, 28 Jul 2020 14:38:38 +0800 (GMT+08:00) Subject: [petsc-users] Installation failure In-Reply-To: References: <1e44c082-ceef-db8c-65b2-0a92375f3d4b@csrc.ac.cn> Message-ID: <40fc10a7.5ce4.1739424fc28.Coremail.rmondaini@csrc.ac.cn> Thanks a bunch, Satish! I can configure PETSc now (see configure.log attached), but the compilation breaks down (see make.log). I am wondering if my problem is related to this: https://community.intel.com/t5/Intel-C-Compiler/LIB-VERSION-TYPE-is-undefined-icc-and-icps-errors-under-Debian/td-p/1160017 > -----Original Messages----- > From: "Satish Balay" > Sent Time: 2020-07-28 13:38:09 (Tuesday) > To: "Rubem Mondaini" > Cc: "Barry Smith" , petsc-users at mcs.anl.gov > Subject: Re: [petsc-users] [SPAM] Re: Installation failure > > something went wrong - but petsc configure didn't capture this error. > > You can retry with: > > ./configure PETSC_ARCH=linux-intel --with-cc=mpiicc --with-cxx=0 --with-fc=mpiifort --with-blaslapack-dir=/opt/intel/compilers_and_libraries_2016/linux/mkl --with-scalar-type=complex --with-64-bit-indices > > And perhaps add one more option: LIBS="-Bstatic -lifcore -Bdynamic" > > Note: --with-64-bit-blas-indices is not appropriate here. > > Satish > > On Tue, 28 Jul 2020, Rubem Mondaini wrote: > > > Dear Barry, > > > > thank you very much for the reply. I removed the options "--with-mpi-include" > > and "--with-mpi-lib", but I still get the same error. I attach the > > configure.log file. > > > > I can confirm that I can compile and run other codes with my compilers, > > however. > > > > Many thanks! > > > > Rubem > > > > On 7/28/20 10:32 AM, Barry Smith wrote: > > > Hmm, configure is a mess. > > > > > > Since you explicitly use MPI compilers > > > > > > --with-cc=mpiicc --with-cxx=mpiicpc --with-fc=mpiifort > > > > > > the > > > > > > --with-mpi-include=/opt/intel/compilers_and_libraries_2016/linux/mpi/intel64/include > > > --with-mpi-lib=/opt/intel/compilers_and_libraries_2016.3.210/linux/mpi/intel64/lib/libmpicxx.a > > > > > > are not needed and should not be provided (Satish I thought this would > > > generate an error? Why doesn't it?) > > > > > > Please try without those arguments and send the configure.log if it > > > fails. > > > > > > The error message isn't completely right. > > > > > > Cannot run executables created with C. > > > > > > This is because the call the call to > > > self.setCompilers.checkCompiler('C') doesn't have enough context to > > > print a decent error message. > > > > > > Barry > > > > > > > > >> On Jul 27, 2020, at 2:06 AM, Rubem Mondaini wrote: > > >> > > >> Dears, > > >> > > >> I have tried to install the PETSc library on my machine and I am failing. I > > >> have managed to install it on another one, which possess similar > > >> configurations for the compilers, and there things do seem to be very much > > >> fine. > > >> > > >> I am attaching the corresponding configure.log. Is there something simple > > >> that I am missing? > > >> > > >> Best, > > >> > > >> > > >> > > -------------- next part -------------- A non-text attachment was scrubbed... Name: configure.log Type: text/x-log Size: 1022601 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: make.log Type: text/x-log Size: 134448 bytes Desc: not available URL: From eda.oktay at metu.edu.tr Tue Jul 28 04:59:52 2020 From: eda.oktay at metu.edu.tr (Eda Oktay) Date: Tue, 28 Jul 2020 12:59:52 +0300 Subject: [petsc-users] SLEPc download error In-Reply-To: References: Message-ID: Thanks so much! Eda Satish Balay , 28 Tem 2020 Sal, 04:46 tarihinde ?unu yazd?: > > [balay at pj01 petsc]$ curl -O https://gitlab.com/slepc/slepc/-/archive/v3.13.3/slepc-v3.13.3.tar.gz > % Total % Received % Xferd Average Speed Time Time Time Current > Dload Upload Total Spent Left Speed > 100 1253k 0 1253k 0 0 16.1M 0 --:--:-- --:--:-- --:--:-- 16.1M > [balay at pj01 petsc]$ file slepc-v3.13.3.tar.gz > slepc-v3.13.3.tar.gz: gzip compressed data, from Unix, original size modulo 2^32 7690240 > [balay at pj01 petsc]$ python > Python 3.7.7 (default, Jun 4 2020, 15:43:14) > [GCC 9.3.1 20200408 (Red Hat 9.3.1-2)] on linux > Type "help", "copyright", "credits" or "license" for more information. > >>> from urllib.request import urlretrieve > >>> urlretrieve('https://gitlab.com/slepc/slepc/-/archive/v3.13.3/slepc-v3.13.3.tar.gz','x.tar.gz') > Traceback (most recent call last): > File "", line 1, in > File "/usr/lib64/python3.7/urllib/request.py", line 247, in urlretrieve > with contextlib.closing(urlopen(url, data)) as fp: > File "/usr/lib64/python3.7/urllib/request.py", line 222, in urlopen > return opener.open(url, data, timeout) > File "/usr/lib64/python3.7/urllib/request.py", line 531, in open > response = meth(req, response) > File "/usr/lib64/python3.7/urllib/request.py", line 641, in http_response > 'http', request, response, code, msg, hdrs) > File "/usr/lib64/python3.7/urllib/request.py", line 569, in error > return self._call_chain(*args) > File "/usr/lib64/python3.7/urllib/request.py", line 503, in _call_chain > result = func(*args) > File "/usr/lib64/python3.7/urllib/request.py", line 649, in http_error_default > raise HTTPError(req.full_url, code, msg, hdrs, fp) > urllib.error.HTTPError: HTTP Error 403: Forbidden > >>> > [balay at pj01 petsc]$ python2 > Python 2.7.18 (default, Apr 20 2020, 00:00:00) > [GCC 9.3.1 20200408 (Red Hat 9.3.1-2)] on linux2 > Type "help", "copyright", "credits" or "license" for more information. > >>> from urllib import urlretrieve > >>> urlretrieve('https://gitlab.com/slepc/slepc/-/archive/v3.13.3/slepc-v3.13.3.tar.gz','x.tar.gz') > ('x.tar.gz', ) > >>> > [1]+ Stopped python2 > [balay at pj01 petsc]$ cat x.tar.gz > error code: 1010[balay at pj01 petsc]$ file x.tar.gz > x.tar.gz: ASCII text, with no line terminators > [balay at pj01 petsc]$ > > On Mon, 27 Jul 2020, Satish Balay via petsc-users wrote: > > > Looks like python urlretrive() is failing on glitab tarball urls [but its working with wget etc. I don't know why. > > > > Perhaps its best to have git installed - so that git-urls work.. > > > > Satish > > > > On Tue, 28 Jul 2020, Eda Oktay wrote: > > > > > Hi all, > > > > > > I am trying to download Petsc with Slepc a linux computer (probably > > > ubuntu, I don't know because I am using it via ssh) but I am getting > > > the following error: > > > > > > ******************************************************************************* > > > UNABLE to CONFIGURE with GIVEN OPTIONS (see configure.log > > > for details): > > > ------------------------------------------------------------------------------- > > > Error during download/extract/detection of SLEPC: > > > Unable to download package SLEPC from: > > > https://gitlab.com/slepc/slepc/-/archive/v3.13.2/slepc-v3.13.2.tar.gz > > > * If URL specified manually - perhaps there is a typo? > > > * If your network is disconnected - please reconnect and rerun ./configure > > > * Or perhaps you have a firewall blocking the download > > > * You can run with --with-packages-download-dir=/adirectory and > > > ./configure will instruct you what packages to download manually > > > * or you can download the above URL manually, to > > > /yourselectedlocation/slepc-v3.13.2.tar.gz > > > and use the configure option: > > > --download-slepc=/yourselectedlocation/slepc-v3.13.2.tar.gz > > > ******************************************************************************* > > > > > > I attached configure.log. > > > > > > Then, I downloaded slepc and used > > > --download-slepc=/yourselectedlocation/slepc-v3.13.2.tar.gz. > > > > > > Since I used WinSCP, tar.gz file was uploaded to my user as "eda" and > > > I used --download-slepc=/home/eda/eda and ran configure again. > > > > > > However, I got the following error: > > > ******************************************************************************* > > > UNABLE to CONFIGURE with GIVEN OPTIONS (see configure.log > > > for details): > > > ------------------------------------------------------------------------------- > > > Error during download/extract/detection of SLEPC: > > > Unknown compression type in URL: file:///home/eda/eda > > > ******************************************************************************* > > > I also attached this confiigure.log as 2configure.log. > > > > > > When I tried to untar eda, I could and I got slepc. So I don't > > > understand why I am getting this error. This is the third computer I > > > am building Petsc with Slepc but this is the first time I am getting > > > such an error. > > > > > > Can you help me with this problem? > > > > > > Thanks! > > > Eda > > > > > > From lbllm2018 at hotmail.com Tue Jul 28 05:37:03 2020 From: lbllm2018 at hotmail.com (Bin Liu) Date: Tue, 28 Jul 2020 10:37:03 +0000 Subject: [petsc-users] solving saddle point problem in distributed memory Message-ID: I would like to solve a saddle point problem arising from the stokes equation. I got successful to use the direct solvers in sequential runs. However, I would like to extend it for distributed memory computation. I tried to use superlu_dist, but the program returns errors. Is it possible to solve a saddle point problem in distributed memory using superlu_dist? Could anyone give a simple sample code to set up the parameters of the solver? Thanks -------------- next part -------------- An HTML attachment was scrubbed... URL: From stefano.zampini at gmail.com Tue Jul 28 05:54:50 2020 From: stefano.zampini at gmail.com (Stefano Zampini) Date: Tue, 28 Jul 2020 12:54:50 +0200 Subject: [petsc-users] solving saddle point problem in distributed memory In-Reply-To: References: Message-ID: <74D5C00B-69F3-4F9B-BD2F-CAA94EEAAD83@gmail.com> If you want advice you should post the error trace PETSc reports. Anyway, solving Stokes is not so trivial (without direct solvers, you may need mesh dependent information), but we have examples for it https://gitlab.com/petsc/petsc/-/blob/master/src/ksp/ksp/tutorials/ex42.c https://gitlab.com/petsc/petsc/-/blob/master/src/ksp/ksp/tutorials/ex43.c https://gitlab.com/petsc/petsc/-/blob/master/src/snes/tutorials/ex69.c If you scroll to the end of those files, you see a bunch of possible options either using PCFIELDSPLIT, PCBDDC or KSPFETIDP. > On Jul 28, 2020, at 12:37 PM, Bin Liu wrote: > > I would like to solve a saddle point problem arising from the stokes equation. I got successful to use the direct solvers in sequential runs. However, I would like to extend it for distributed memory computation. I tried to use superlu_dist, but the program returns errors. Is it possible to solve a saddle point problem in distributed memory using superlu_dist? Could anyone give a simple sample code to set up the parameters of the solver? > > Thanks -------------- next part -------------- An HTML attachment was scrubbed... URL: From eda.oktay at metu.edu.tr Tue Jul 28 09:44:42 2020 From: eda.oktay at metu.edu.tr (Eda Oktay) Date: Tue, 28 Jul 2020 17:44:42 +0300 Subject: [petsc-users] Petsc configure error (cmake) Message-ID: I am trying to download Petsc with slepc,openblas,mpich,meti,parmetis and chaco to a linux computer (probably ubuntu, I don't know because I am using it via ssh). Due to an error, for metis, I needed to download cmake, so I added this option, too. However, I got an error telling me that "Error running configure on cmake". I attached configure.log. What should I do? Where is the problem and how can ? fix it? Thank you so much for answering, Eda -------------- next part -------------- A non-text attachment was scrubbed... Name: configure.log Type: text/x-log Size: 931043 bytes Desc: not available URL: From knepley at gmail.com Tue Jul 28 09:48:56 2020 From: knepley at gmail.com (Matthew Knepley) Date: Tue, 28 Jul 2020 10:48:56 -0400 Subject: [petsc-users] Petsc configure error (cmake) In-Reply-To: References: Message-ID: On Tue, Jul 28, 2020 at 10:45 AM Eda Oktay wrote: > I am trying to download Petsc with slepc,openblas,mpich,meti,parmetis > and chaco to a linux computer (probably > ubuntu, I don't know because I am using it via ssh). > > Due to an error, for metis, I needed to download cmake, so I added > this option, too. However, I got an error telling me that "Error > running configure on cmake". I attached configure.log. > > What should I do? Where is the problem and how can ? fix it? > > Thank you so much for answering, > It looks like you do not have 'make' in your path, and CMake is feaking out: CMAKE was just downloaded, forcing a rebuild because cannot determine if package has changed =============================================================================== Running configure on CMAKE; this may take several minutes =============================================================================== Running Executable WITHOUT threads to time it out Executing: ./configure --prefix=/home/eda/petsc-3.13.2/arch-linux-c-debug --parallel=40 stdout: --------------------------------------------- CMake 3.15.6, Copyright 2000-2019 Kitware, Inc. and Contributors Found GNU toolchain C compiler on this system is: gcc C++ compiler on this system is: g++ --------------------------------------------- Error when bootstrapping CMake: Cannot find appropriate Makefile processor on this system. Please specify one using environment variable MAKE. --------------------------------------------- Log of errors: /home/eda/petsc-3.13.2/arch-linux-c-debug/externalpackages/cmake-3.15.6/Bootstrap.cmk/cmake_bootstrap.log --------------------------------------------- Thanks, Matt > Eda > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From balay at mcs.anl.gov Tue Jul 28 10:23:58 2020 From: balay at mcs.anl.gov (Satish Balay) Date: Tue, 28 Jul 2020 10:23:58 -0500 (CDT) Subject: [petsc-users] Installation failure In-Reply-To: <40fc10a7.5ce4.1739424fc28.Coremail.rmondaini@csrc.ac.cn> References: <1e44c082-ceef-db8c-65b2-0a92375f3d4b@csrc.ac.cn> <40fc10a7.5ce4.1739424fc28.Coremail.rmondaini@csrc.ac.cn> Message-ID: >>>>>>> CC linux-intel/obj/sys/error/fp.o In file included from /usr/include/complex.h(152), from /opt/intel/compilers_and_libraries_2016.3.210/linux/compiler/include/complex.h(22), from /home/rmondaini/libraries/petsc-3.13.3/include/petscsystypes.h(234), from /home/rmondaini/libraries/petsc-3.13.3/include/petscsys.h(36), from /home/rmondaini/libraries/petsc-3.13.3/src/sys/error/fp.c(19): /usr/include/x86_64-linux-gnu/bits/cmathcalls.h(55): error: identifier "_Float32" is undefined __MATHCALL (cacos, (_Mdouble_complex_ __z)); ^ CC linux-intel/obj/sys/error/errstop.o In file included from /home/rmondaini/libraries/petsc-3.13.3/include/petscmath.h(13), from /home/rmondaini/libraries/petsc-3.13.3/include/petscsys.h(322), from /home/rmondaini/libraries/petsc-3.13.3/src/sys/error/errstop.c(2): /opt/intel/compilers_and_libraries_2016.3.210/linux/compiler/include/math.h(1214): error: identifier "_LIB_VERSION_TYPE" is undefined _LIBIMF_EXT _LIB_VERSIONIMF_TYPE _LIBIMF_PUBVAR _LIB_VERSIONIMF; ^ <<<<< Looks like this version of intel compiler has issues with system (gcc) include files. Can you use gcc/g++ instead for this build? Satish On Tue, 28 Jul 2020, Rubem Mondaini wrote: > Thanks a bunch, Satish! > > I can configure PETSc now (see configure.log attached), but the compilation breaks down (see make.log). I am wondering if my problem is related to this: https://community.intel.com/t5/Intel-C-Compiler/LIB-VERSION-TYPE-is-undefined-icc-and-icps-errors-under-Debian/td-p/1160017 > > > > -----Original Messages----- > > From: "Satish Balay" > > Sent Time: 2020-07-28 13:38:09 (Tuesday) > > To: "Rubem Mondaini" > > Cc: "Barry Smith" , petsc-users at mcs.anl.gov > > Subject: Re: [petsc-users] [SPAM] Re: Installation failure > > > > something went wrong - but petsc configure didn't capture this error. > > > > You can retry with: > > > > ./configure PETSC_ARCH=linux-intel --with-cc=mpiicc --with-cxx=0 --with-fc=mpiifort --with-blaslapack-dir=/opt/intel/compilers_and_libraries_2016/linux/mkl --with-scalar-type=complex --with-64-bit-indices > > > > And perhaps add one more option: LIBS="-Bstatic -lifcore -Bdynamic" > > > > Note: --with-64-bit-blas-indices is not appropriate here. > > > > Satish > > > > On Tue, 28 Jul 2020, Rubem Mondaini wrote: > > > > > Dear Barry, > > > > > > thank you very much for the reply. I removed the options "--with-mpi-include" > > > and "--with-mpi-lib", but I still get the same error. I attach the > > > configure.log file. > > > > > > I can confirm that I can compile and run other codes with my compilers, > > > however. > > > > > > Many thanks! > > > > > > Rubem > > > > > > On 7/28/20 10:32 AM, Barry Smith wrote: > > > > Hmm, configure is a mess. > > > > > > > > Since you explicitly use MPI compilers > > > > > > > > --with-cc=mpiicc --with-cxx=mpiicpc --with-fc=mpiifort > > > > > > > > the > > > > > > > > --with-mpi-include=/opt/intel/compilers_and_libraries_2016/linux/mpi/intel64/include > > > > --with-mpi-lib=/opt/intel/compilers_and_libraries_2016.3.210/linux/mpi/intel64/lib/libmpicxx.a > > > > > > > > are not needed and should not be provided (Satish I thought this would > > > > generate an error? Why doesn't it?) > > > > > > > > Please try without those arguments and send the configure.log if it > > > > fails. > > > > > > > > The error message isn't completely right. > > > > > > > > Cannot run executables created with C. > > > > > > > > This is because the call the call to > > > > self.setCompilers.checkCompiler('C') doesn't have enough context to > > > > print a decent error message. > > > > > > > > Barry > > > > > > > > > > > >> On Jul 27, 2020, at 2:06 AM, Rubem Mondaini wrote: > > > >> > > > >> Dears, > > > >> > > > >> I have tried to install the PETSc library on my machine and I am failing. I > > > >> have managed to install it on another one, which possess similar > > > >> configurations for the compilers, and there things do seem to be very much > > > >> fine. > > > >> > > > >> I am attaching the corresponding configure.log. Is there something simple > > > >> that I am missing? > > > >> > > > >> Best, > > > >> > > > >> > > > >> > > > > From alexlindsay239 at gmail.com Tue Jul 28 10:43:07 2020 From: alexlindsay239 at gmail.com (Alexander Lindsay) Date: Tue, 28 Jul 2020 08:43:07 -0700 Subject: [petsc-users] Only print converged reason when not converged Message-ID: To help debug the many emails we get about solves that fail to converge, in MOOSE we recently appended `-snes_converged_reason -ksp_converged_reason` for every call to `SNESSolve`. Of course, now we have users complaining about the new text printed to their screens that they didn't have before. Some of them have made a reasonable request to only print the convergence reason when the solve has actually failed to converge. Is there some way we can only print the reason if we've diverged, e.g. if reason < 0 ? Alex -------------- next part -------------- An HTML attachment was scrubbed... URL: From mfadams at lbl.gov Tue Jul 28 11:09:54 2020 From: mfadams at lbl.gov (Mark Adams) Date: Tue, 28 Jul 2020 12:09:54 -0400 Subject: [petsc-users] Only print converged reason when not converged In-Reply-To: References: Message-ID: You can add your own convergence test and do anything that you want: https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/SNES/SNESSetConvergenceTest.html On Tue, Jul 28, 2020 at 11:44 AM Alexander Lindsay wrote: > To help debug the many emails we get about solves that fail to converge, > in MOOSE we recently appended `-snes_converged_reason > -ksp_converged_reason` for every call to `SNESSolve`. Of course, now we > have users complaining about the new text printed to their screens that > they didn't have before. Some of them have made a reasonable request to > only print the convergence reason when the solve has actually failed to > converge. Is there some way we can only print the reason if we've diverged, > e.g. if reason < 0 ? > > Alex > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mfadams at lbl.gov Tue Jul 28 11:14:47 2020 From: mfadams at lbl.gov (Mark Adams) Date: Tue, 28 Jul 2020 12:14:47 -0400 Subject: [petsc-users] Only print converged reason when not converged In-Reply-To: References: Message-ID: You can also do this with a monitor and get the converged reason to do what you want: https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/SNES/SNESMonitorSet.html#SNESMonitorSet On Tue, Jul 28, 2020 at 12:09 PM Mark Adams wrote: > You can add your own convergence test and do anything that you want: > > > https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/SNES/SNESSetConvergenceTest.html > > On Tue, Jul 28, 2020 at 11:44 AM Alexander Lindsay < > alexlindsay239 at gmail.com> wrote: > >> To help debug the many emails we get about solves that fail to converge, >> in MOOSE we recently appended `-snes_converged_reason >> -ksp_converged_reason` for every call to `SNESSolve`. Of course, now we >> have users complaining about the new text printed to their screens that >> they didn't have before. Some of them have made a reasonable request to >> only print the convergence reason when the solve has actually failed to >> converge. Is there some way we can only print the reason if we've diverged, >> e.g. if reason < 0 ? >> >> Alex >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at petsc.dev Tue Jul 28 11:56:20 2020 From: bsmith at petsc.dev (Barry Smith) Date: Tue, 28 Jul 2020 11:56:20 -0500 Subject: [petsc-users] VecLoad into a SubVector ? In-Reply-To: References: <7F9E26AC-276F-40B5-B2F3-05098C379685@petsc.dev> Message-ID: <437EB33F-4363-4B86-AEA4-8D68F26C1693@petsc.dev> Sajid, This seems like a good fix. Barry > On Jul 27, 2020, at 11:34 PM, Sajid Ali wrote: > > Hi Barry/Matt, > > The fix to this bug would be to disable replacearray op on a subvector. I modified the source code for vecio.c forcing VecLoad_HDF5 to always perform an array copy and the above test passes for both binary and hdf5 viewers in serial and parallel. > > I can open a PR that adds a line Z->ops->replacearray = NULL; at line 1286 in the rvector.c file if one of you can confirm that the above logic is correct. The example attached in the last email could be used as a test for the same if necessary. > > Thank You, > Sajid Ali | PhD Candidate > Applied Physics > Northwestern University > s-sajid-ali.github.io -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at petsc.dev Tue Jul 28 12:55:50 2020 From: bsmith at petsc.dev (Barry Smith) Date: Tue, 28 Jul 2020 12:55:50 -0500 Subject: [petsc-users] Petsc configure error (cmake) In-Reply-To: References: Message-ID: Eda, The branch barry/2020-07-28/cmake-use-make-if-given/maint should resolve the problem. It will force the cmake build to use the make obtained with --download-make Barry > On Jul 28, 2020, at 9:48 AM, Matthew Knepley wrote: > > On Tue, Jul 28, 2020 at 10:45 AM Eda Oktay > wrote: > I am trying to download Petsc with slepc,openblas,mpich,meti,parmetis > and chaco to a linux computer (probably > ubuntu, I don't know because I am using it via ssh). > > Due to an error, for metis, I needed to download cmake, so I added > this option, too. However, I got an error telling me that "Error > running configure on cmake". I attached configure.log. > > What should I do? Where is the problem and how can ? fix it? > > Thank you so much for answering, > > It looks like you do not have 'make' in your path, and CMake is feaking out: > > CMAKE was just downloaded, forcing a rebuild because cannot determine if package has changed > =============================================================================== > Running configure on CMAKE; this may take several minutes > =============================================================================== > > Running Executable WITHOUT threads to time it out > Executing: ./configure --prefix=/home/eda/petsc-3.13.2/arch-linux-c-debug --parallel=40 > stdout: > --------------------------------------------- > CMake 3.15.6, Copyright 2000-2019 Kitware, Inc. and Contributors > Found GNU toolchain > C compiler on this system is: gcc > C++ compiler on this system is: g++ > --------------------------------------------- > Error when bootstrapping CMake: > Cannot find appropriate Makefile processor on this system. > Please specify one using environment variable MAKE. > --------------------------------------------- > Log of errors: /home/eda/petsc-3.13.2/arch-linux-c-debug/externalpackages/cmake-3.15.6/Bootstrap.cmk/cmake_bootstrap.log > --------------------------------------------- > > Thanks, > > Matt > > Eda > > > -- > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at petsc.dev Tue Jul 28 13:01:24 2020 From: bsmith at petsc.dev (Barry Smith) Date: Tue, 28 Jul 2020 13:01:24 -0500 Subject: [petsc-users] Only print converged reason when not converged In-Reply-To: References: Message-ID: Alex, The actual printing is done with SNESReasonView() and KSPReasonView() I would suggest copying those files to Moose with a name change and removing all the code you don't want. Then you can call your versions immediately after SNESSolve() and KSPSolve(). Barry > On Jul 28, 2020, at 10:43 AM, Alexander Lindsay wrote: > > To help debug the many emails we get about solves that fail to converge, in MOOSE we recently appended `-snes_converged_reason -ksp_converged_reason` for every call to `SNESSolve`. Of course, now we have users complaining about the new text printed to their screens that they didn't have before. Some of them have made a reasonable request to only print the convergence reason when the solve has actually failed to converge. Is there some way we can only print the reason if we've diverged, e.g. if reason < 0 ? > > Alex From fdkong.jd at gmail.com Tue Jul 28 13:12:44 2020 From: fdkong.jd at gmail.com (Fande Kong) Date: Tue, 28 Jul 2020 12:12:44 -0600 Subject: [petsc-users] Only print converged reason when not converged In-Reply-To: References: Message-ID: One alternative is to support a plugable KSP/SNESReasonView system. We then could hook up KSP/SNESReasonView_MOOSE. We could call our views from SNES/KSP"done"Solve as well if such a system is not affordable. What are the final functions we should call, where we guarantee SNES/KSP is already done? Thanks, Fande, On Tue, Jul 28, 2020 at 12:02 PM Barry Smith wrote: > > Alex, > > The actual printing is done with SNESReasonView() and KSPReasonView() > I would suggest copying those files to Moose with a name change and > removing all the code you don't want. Then you can call your versions > immediately after SNESSolve() and KSPSolve(). > > Barry > > > > On Jul 28, 2020, at 10:43 AM, Alexander Lindsay < > alexlindsay239 at gmail.com> wrote: > > > > To help debug the many emails we get about solves that fail to converge, > in MOOSE we recently appended `-snes_converged_reason > -ksp_converged_reason` for every call to `SNESSolve`. Of course, now we > have users complaining about the new text printed to their screens that > they didn't have before. Some of them have made a reasonable request to > only print the convergence reason when the solve has actually failed to > converge. Is there some way we can only print the reason if we've diverged, > e.g. if reason < 0 ? > > > > Alex > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at petsc.dev Tue Jul 28 14:20:33 2020 From: bsmith at petsc.dev (Barry Smith) Date: Tue, 28 Jul 2020 14:20:33 -0500 Subject: [petsc-users] Only print converged reason when not converged In-Reply-To: References: Message-ID: KPSSetPostSolve(), There is no SNESSetPostSolve() but there could/should be. Barry Note inside the SNESSolve there are a bunch of pre and post hooks, don't get the confused with an outside SNESSetPostSolve() > On Jul 28, 2020, at 1:12 PM, Fande Kong wrote: > > One alternative is to support a plugable KSP/SNESReasonView system. We then could hook up KSP/SNESReasonView_MOOSE. > > We could call our views from SNES/KSP"done"Solve as well if such a system is not affordable. What are the final functions we should call, where we guarantee SNES/KSP is already done? > > Thanks, > > Fande, > > On Tue, Jul 28, 2020 at 12:02 PM Barry Smith > wrote: > > Alex, > > The actual printing is done with SNESReasonView() and KSPReasonView() I would suggest copying those files to Moose with a name change and removing all the code you don't want. Then you can call your versions immediately after SNESSolve() and KSPSolve(). > > Barry > > > > On Jul 28, 2020, at 10:43 AM, Alexander Lindsay > wrote: > > > > To help debug the many emails we get about solves that fail to converge, in MOOSE we recently appended `-snes_converged_reason -ksp_converged_reason` for every call to `SNESSolve`. Of course, now we have users complaining about the new text printed to their screens that they didn't have before. Some of them have made a reasonable request to only print the convergence reason when the solve has actually failed to converge. Is there some way we can only print the reason if we've diverged, e.g. if reason < 0 ? > > > > Alex > -------------- next part -------------- An HTML attachment was scrubbed... URL: From eda.oktay at metu.edu.tr Tue Jul 28 15:28:17 2020 From: eda.oktay at metu.edu.tr (Eda Oktay) Date: Tue, 28 Jul 2020 23:28:17 +0300 Subject: [petsc-users] Petsc configure error (cmake) In-Reply-To: References: Message-ID: Dear Barry, Thank you so much for answering. So, I downloaded the tar.gz file and now, should I configure this file with my previous options? I am using the following line: ./configure --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --download-make --download-mpich --download-openblas --download-metis --download-parmetis --download-chaco --download-slepc=/home/eda/slepc-v3.13.2.tar.gz --download-cmake --with-X=1 Thanks so much again! Eda Barry Smith , 28 Tem 2020 Sal, 20:55 tarihinde ?unu yazd?: > > > Eda, > > The branch barry/2020-07-28/cmake-use-make-if-given/maint should resolve the problem. It will force the cmake build to use the make obtained with --download-make > > Barry > > > On Jul 28, 2020, at 9:48 AM, Matthew Knepley wrote: > > On Tue, Jul 28, 2020 at 10:45 AM Eda Oktay wrote: >> >> I am trying to download Petsc with slepc,openblas,mpich,meti,parmetis >> and chaco to a linux computer (probably >> ubuntu, I don't know because I am using it via ssh). >> >> Due to an error, for metis, I needed to download cmake, so I added >> this option, too. However, I got an error telling me that "Error >> running configure on cmake". I attached configure.log. >> >> What should I do? Where is the problem and how can ? fix it? >> >> Thank you so much for answering, > > > It looks like you do not have 'make' in your path, and CMake is feaking out: > > CMAKE was just downloaded, forcing a rebuild because cannot determine if package has changed > =============================================================================== > Running configure on CMAKE; this may take several minutes > =============================================================================== > > Running Executable WITHOUT threads to time it out > Executing: ./configure --prefix=/home/eda/petsc-3.13.2/arch-linux-c-debug --parallel=40 > stdout: > --------------------------------------------- > CMake 3.15.6, Copyright 2000-2019 Kitware, Inc. and Contributors > Found GNU toolchain > C compiler on this system is: gcc > C++ compiler on this system is: g++ > --------------------------------------------- > Error when bootstrapping CMake: > Cannot find appropriate Makefile processor on this system. > Please specify one using environment variable MAKE. > --------------------------------------------- > Log of errors: /home/eda/petsc-3.13.2/arch-linux-c-debug/externalpackages/cmake-3.15.6/Bootstrap.cmk/cmake_bootstrap.log > --------------------------------------------- > > Thanks, > > Matt > >> >> Eda > > > > -- > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ > > From bsmith at petsc.dev Tue Jul 28 16:10:00 2020 From: bsmith at petsc.dev (Barry Smith) Date: Tue, 28 Jul 2020 16:10:00 -0500 Subject: [petsc-users] Petsc configure error (cmake) In-Reply-To: References: Message-ID: It's better to use git than download tarballs. With tarballs it is not possible to access recent fixes or try branches. Here is what you can do, save the attachment as cmakemake.patch and do patch -p1 < cmakemake.patch in the PETSc directory Then run configure as before Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: cmakemake.patch Type: application/octet-stream Size: 1316 bytes Desc: not available URL: -------------- next part -------------- > On Jul 28, 2020, at 3:28 PM, Eda Oktay wrote: > > Dear Barry, > > Thank you so much for answering. So, I downloaded the tar.gz file and > now, should I configure this file with my previous options? I am > using the following line: > > ./configure --with-cc=gcc --with-cxx=g++ --with-fc=gfortran > --download-make --download-mpich --download-openblas --download-metis > --download-parmetis --download-chaco > --download-slepc=/home/eda/slepc-v3.13.2.tar.gz --download-cmake > --with-X=1 > > Thanks so much again! > > Eda > > Barry Smith , 28 Tem 2020 Sal, 20:55 tarihinde ?unu yazd?: >> >> >> Eda, >> >> The branch barry/2020-07-28/cmake-use-make-if-given/maint should resolve the problem. It will force the cmake build to use the make obtained with --download-make >> >> Barry >> >> >> On Jul 28, 2020, at 9:48 AM, Matthew Knepley wrote: >> >> On Tue, Jul 28, 2020 at 10:45 AM Eda Oktay wrote: >>> >>> I am trying to download Petsc with slepc,openblas,mpich,meti,parmetis >>> and chaco to a linux computer (probably >>> ubuntu, I don't know because I am using it via ssh). >>> >>> Due to an error, for metis, I needed to download cmake, so I added >>> this option, too. However, I got an error telling me that "Error >>> running configure on cmake". I attached configure.log. >>> >>> What should I do? Where is the problem and how can ? fix it? >>> >>> Thank you so much for answering, >> >> >> It looks like you do not have 'make' in your path, and CMake is feaking out: >> >> CMAKE was just downloaded, forcing a rebuild because cannot determine if package has changed >> =============================================================================== >> Running configure on CMAKE; this may take several minutes >> =============================================================================== >> >> Running Executable WITHOUT threads to time it out >> Executing: ./configure --prefix=/home/eda/petsc-3.13.2/arch-linux-c-debug --parallel=40 >> stdout: >> --------------------------------------------- >> CMake 3.15.6, Copyright 2000-2019 Kitware, Inc. and Contributors >> Found GNU toolchain >> C compiler on this system is: gcc >> C++ compiler on this system is: g++ >> --------------------------------------------- >> Error when bootstrapping CMake: >> Cannot find appropriate Makefile processor on this system. >> Please specify one using environment variable MAKE. >> --------------------------------------------- >> Log of errors: /home/eda/petsc-3.13.2/arch-linux-c-debug/externalpackages/cmake-3.15.6/Bootstrap.cmk/cmake_bootstrap.log >> --------------------------------------------- >> >> Thanks, >> >> Matt >> >>> >>> Eda >> >> >> >> -- >> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. >> -- Norbert Wiener >> >> https://www.cse.buffalo.edu/~knepley/ >> >> From alexlindsay239 at gmail.com Tue Jul 28 19:07:52 2020 From: alexlindsay239 at gmail.com (Alexander Lindsay) Date: Tue, 28 Jul 2020 17:07:52 -0700 Subject: [petsc-users] Only print converged reason when not converged In-Reply-To: References: Message-ID: The only slight annoyance with doing this through a PostSolve hook as opposed to a pluggable ReasonView system is that if a user passed `-snes_converged_reason` on the command line, we would get redundant printout from SNESSolve and our PostSolve. Although I suppose we could solve this by intercepting their option, not passing it to PETSc, and then just using that info to control the output from our PostSolve (e.g. if a user passes -snes_converged_reason from the command line, then we should always print the reason, instead of just printing when we don't converge). So maybe the PostSolve will work fine. On Tue, Jul 28, 2020 at 12:20 PM Barry Smith wrote: > > KPSSetPostSolve(), There is no SNESSetPostSolve() but there > could/should be. > > Barry > > Note inside the SNESSolve there are a bunch of pre and post hooks, don't > get the confused with an outside SNESSetPostSolve() > > On Jul 28, 2020, at 1:12 PM, Fande Kong wrote: > > One alternative is to support a plugable KSP/SNESReasonView system. We > then could hook up KSP/SNESReasonView_MOOSE. > > We could call our views from SNES/KSP"done"Solve as well if such a > system is not affordable. What are the final functions we should call, > where we guarantee SNES/KSP is already done? > > Thanks, > > Fande, > > On Tue, Jul 28, 2020 at 12:02 PM Barry Smith wrote: > >> >> Alex, >> >> The actual printing is done with SNESReasonView() and KSPReasonView() >> I would suggest copying those files to Moose with a name change and >> removing all the code you don't want. Then you can call your versions >> immediately after SNESSolve() and KSPSolve(). >> >> Barry >> >> >> > On Jul 28, 2020, at 10:43 AM, Alexander Lindsay < >> alexlindsay239 at gmail.com> wrote: >> > >> > To help debug the many emails we get about solves that fail to >> converge, in MOOSE we recently appended `-snes_converged_reason >> -ksp_converged_reason` for every call to `SNESSolve`. Of course, now we >> have users complaining about the new text printed to their screens that >> they didn't have before. Some of them have made a reasonable request to >> only print the convergence reason when the solve has actually failed to >> converge. Is there some way we can only print the reason if we've diverged, >> e.g. if reason < 0 ? >> > >> > Alex >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Tue Jul 28 19:50:49 2020 From: knepley at gmail.com (Matthew Knepley) Date: Tue, 28 Jul 2020 20:50:49 -0400 Subject: [petsc-users] Only print converged reason when not converged In-Reply-To: References: Message-ID: On Tue, Jul 28, 2020 at 8:09 PM Alexander Lindsay wrote: > The only slight annoyance with doing this through a PostSolve hook as > opposed to a pluggable ReasonView system is that if a user passed > `-snes_converged_reason` on the command line, we would get redundant > printout from SNESSolve and our PostSolve. Although I suppose we could > solve this by intercepting their option, not passing it to PETSc, and then > just using that info to control the output from our PostSolve (e.g. if a > user passes -snes_converged_reason from the command line, then we should > always print the reason, instead of just printing when we don't converge). > So maybe the PostSolve will work fine. > Can't you just define another ASCII format that prints what you want for ReasonView? Matt > On Tue, Jul 28, 2020 at 12:20 PM Barry Smith wrote: > >> >> KPSSetPostSolve(), There is no SNESSetPostSolve() but there >> could/should be. >> >> Barry >> >> Note inside the SNESSolve there are a bunch of pre and post hooks, >> don't get the confused with an outside SNESSetPostSolve() >> >> On Jul 28, 2020, at 1:12 PM, Fande Kong wrote: >> >> One alternative is to support a plugable KSP/SNESReasonView system. We >> then could hook up KSP/SNESReasonView_MOOSE. >> >> We could call our views from SNES/KSP"done"Solve as well if such a >> system is not affordable. What are the final functions we should call, >> where we guarantee SNES/KSP is already done? >> >> Thanks, >> >> Fande, >> >> On Tue, Jul 28, 2020 at 12:02 PM Barry Smith wrote: >> >>> >>> Alex, >>> >>> The actual printing is done with SNESReasonView() and >>> KSPReasonView() I would suggest copying those files to Moose with a name >>> change and removing all the code you don't want. Then you can call your >>> versions immediately after SNESSolve() and KSPSolve(). >>> >>> Barry >>> >>> >>> > On Jul 28, 2020, at 10:43 AM, Alexander Lindsay < >>> alexlindsay239 at gmail.com> wrote: >>> > >>> > To help debug the many emails we get about solves that fail to >>> converge, in MOOSE we recently appended `-snes_converged_reason >>> -ksp_converged_reason` for every call to `SNESSolve`. Of course, now we >>> have users complaining about the new text printed to their screens that >>> they didn't have before. Some of them have made a reasonable request to >>> only print the convergence reason when the solve has actually failed to >>> converge. Is there some way we can only print the reason if we've diverged, >>> e.g. if reason < 0 ? >>> > >>> > Alex >>> >>> >> -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From lbllm2018 at hotmail.com Tue Jul 28 19:59:50 2020 From: lbllm2018 at hotmail.com (Bin Liu) Date: Wed, 29 Jul 2020 00:59:50 +0000 Subject: [petsc-users] solving saddle point problem in distributed memory In-Reply-To: <74D5C00B-69F3-4F9B-BD2F-CAA94EEAAD83@gmail.com> References: <74D5C00B-69F3-4F9B-BD2F-CAA94EEAAD83@gmail.com> Message-ID: Thanks for your tutorials. Yes. I tried PCFIELDSPLIT. However, it only works for sequential runs. When I run the code in distributed memory, it reports errors. In fact, the essence of my wonder is (a) how to set up superlu_dist in petsc for solving saddle point problem in distributed memory? (b) does the direct solvers in petsc can run in distributed memory for solving saddle point problem? From: Stefano Zampini [mailto:stefano.zampini at gmail.com] Sent: Tuesday, 28 July 2020 6:55 PM To: Bin Liu Cc: petsc-users at mcs.anl.gov Subject: Re: [petsc-users] solving saddle point problem in distributed memory If you want advice you should post the error trace PETSc reports. Anyway, solving Stokes is not so trivial (without direct solvers, you may need mesh dependent information), but we have examples for it https://gitlab.com/petsc/petsc/-/blob/master/src/ksp/ksp/tutorials/ex42.c https://gitlab.com/petsc/petsc/-/blob/master/src/ksp/ksp/tutorials/ex43.c https://gitlab.com/petsc/petsc/-/blob/master/src/snes/tutorials/ex69.c If you scroll to the end of those files, you see a bunch of possible options either using PCFIELDSPLIT, PCBDDC or KSPFETIDP. On Jul 28, 2020, at 12:37 PM, Bin Liu > wrote: I would like to solve a saddle point problem arising from the stokes equation. I got successful to use the direct solvers in sequential runs. However, I would like to extend it for distributed memory computation. I tried to use superlu_dist, but the program returns errors. Is it possible to solve a saddle point problem in distributed memory using superlu_dist? Could anyone give a simple sample code to set up the parameters of the solver? Thanks -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Tue Jul 28 20:35:32 2020 From: knepley at gmail.com (Matthew Knepley) Date: Tue, 28 Jul 2020 21:35:32 -0400 Subject: [petsc-users] solving saddle point problem in distributed memory In-Reply-To: References: <74D5C00B-69F3-4F9B-BD2F-CAA94EEAAD83@gmail.com> Message-ID: On Tue, Jul 28, 2020 at 9:00 PM Bin Liu wrote: > Thanks for your tutorials. Yes. I tried PCFIELDSPLIT. However, it only > works for sequential runs. When I run the code in distributed memory, it > reports errors. > This would mean that your setup is incorrect. We run FIELDSPLIT all the time in parallel, and it has been used on some of the largest run ever made on supercomputers. Second, the way to directly solve a saddle-point problem is to directly solve the (0,0) block and the Schur complement. This is not hard: -pc_type fieldsplit -pc_fieldsplit_detect_saddlepoint -pc_fieldsplit_type schur -pc_fieldsplit_schur_fact_type full -pc_fieldsplit_schur_precondition full -fieldsplit_0_pc_type lu -fieldsplit_0_pc_factor_mat_solver_type superlu_dist -fieldsplit_1_pc_type lu -fieldsplit_1_pc_factor_mat_solver_type superlu_dist Thanks, Matt > In fact, the essence of my wonder is (a) how to set up superlu_dist in > petsc for solving saddle point problem in distributed memory? (b) does the > direct solvers in petsc can run in distributed memory for solving saddle > point problem? > > > > *From:* Stefano Zampini [mailto:stefano.zampini at gmail.com] > *Sent:* Tuesday, 28 July 2020 6:55 PM > *To:* Bin Liu > *Cc:* petsc-users at mcs.anl.gov > *Subject:* Re: [petsc-users] solving saddle point problem in distributed > memory > > > > If you want advice you should post the error trace PETSc reports. > > > > Anyway, solving Stokes is not so trivial (without direct solvers, you may > need mesh dependent information), but we have examples for it > > > > https://gitlab.com/petsc/petsc/-/blob/master/src/ksp/ksp/tutorials/ex42.c > > https://gitlab.com/petsc/petsc/-/blob/master/src/ksp/ksp/tutorials/ex43.c > > https://gitlab.com/petsc/petsc/-/blob/master/src/snes/tutorials/ex69.c > > > > If you scroll to the end of those files, you see a bunch of possible > options either using PCFIELDSPLIT, PCBDDC or KSPFETIDP. > > > > > > On Jul 28, 2020, at 12:37 PM, Bin Liu wrote: > > > > I would like to solve a saddle point problem arising from the stokes > equation. I got successful to use the direct solvers in sequential runs. > However, I would like to extend it for distributed memory computation. I > tried to use superlu_dist, but the program returns errors. Is it possible > to solve a saddle point problem in distributed memory using superlu_dist? > Could anyone give a simple sample code to set up the parameters of the > solver? > > > > Thanks > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at petsc.dev Tue Jul 28 20:42:42 2020 From: bsmith at petsc.dev (Barry Smith) Date: Tue, 28 Jul 2020 20:42:42 -0500 Subject: [petsc-users] Only print converged reason when not converged In-Reply-To: References: Message-ID: Yes, this is a good idea. Simple extension of current code. The command line would then be -snes_converged_reason ::failed add in petscviewer.h a new PetscViewerFormat PETSC_VIEWER_FAILED, add in viewera.c PetscViewerFormats FAILED. add in SNESReasonView if (snes->reason > 0 && format != PETSC_VIEWER_FAILED) { Document it in SNESReasonView manual page > On Jul 28, 2020, at 7:50 PM, Matthew Knepley wrote: > > On Tue, Jul 28, 2020 at 8:09 PM Alexander Lindsay > wrote: > The only slight annoyance with doing this through a PostSolve hook as opposed to a pluggable ReasonView system is that if a user passed `-snes_converged_reason` on the command line, we would get redundant printout from SNESSolve and our PostSolve. Although I suppose we could solve this by intercepting their option, not passing it to PETSc, and then just using that info to control the output from our PostSolve (e.g. if a user passes -snes_converged_reason from the command line, then we should always print the reason, instead of just printing when we don't converge). So maybe the PostSolve will work fine. > > Can't you just define another ASCII format that prints what you want for ReasonView? > > Matt > > On Tue, Jul 28, 2020 at 12:20 PM Barry Smith > wrote: > > KPSSetPostSolve(), There is no SNESSetPostSolve() but there could/should be. > > Barry > > Note inside the SNESSolve there are a bunch of pre and post hooks, don't get the confused with an outside SNESSetPostSolve() > >> On Jul 28, 2020, at 1:12 PM, Fande Kong > wrote: >> >> One alternative is to support a plugable KSP/SNESReasonView system. We then could hook up KSP/SNESReasonView_MOOSE. >> >> We could call our views from SNES/KSP"done"Solve as well if such a system is not affordable. What are the final functions we should call, where we guarantee SNES/KSP is already done? >> >> Thanks, >> >> Fande, >> >> On Tue, Jul 28, 2020 at 12:02 PM Barry Smith > wrote: >> >> Alex, >> >> The actual printing is done with SNESReasonView() and KSPReasonView() I would suggest copying those files to Moose with a name change and removing all the code you don't want. Then you can call your versions immediately after SNESSolve() and KSPSolve(). >> >> Barry >> >> >> > On Jul 28, 2020, at 10:43 AM, Alexander Lindsay > wrote: >> > >> > To help debug the many emails we get about solves that fail to converge, in MOOSE we recently appended `-snes_converged_reason -ksp_converged_reason` for every call to `SNESSolve`. Of course, now we have users complaining about the new text printed to their screens that they didn't have before. Some of them have made a reasonable request to only print the convergence reason when the solve has actually failed to converge. Is there some way we can only print the reason if we've diverged, e.g. if reason < 0 ? >> > >> > Alex >> > > > > -- > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at petsc.dev Tue Jul 28 20:47:32 2020 From: bsmith at petsc.dev (Barry Smith) Date: Tue, 28 Jul 2020 20:47:32 -0500 Subject: [petsc-users] solving saddle point problem in distributed memory In-Reply-To: References: <74D5C00B-69F3-4F9B-BD2F-CAA94EEAAD83@gmail.com> Message-ID: <13D4A6F0-137D-435F-882B-042F85990C8C@petsc.dev> SuperLU_DIST won't "magically" run on saddle point problems. It only does limited pivoting, realistically a parallel LU cannot always do complete pivoting or it becomes a sequential algorithm. For parallel you need to use PCFIELDSPLIT, for sequential you can use SuperLU (not SuperLU_DIST) since it can do more pivoting, being a sequential algorithm. Barry > On Jul 28, 2020, at 7:59 PM, Bin Liu wrote: > > Thanks for your tutorials. Yes. I tried PCFIELDSPLIT. However, it only works for sequential runs. When I run the code in distributed memory, it reports errors. In fact, the essence of my wonder is (a) how to set up superlu_dist in petsc for solving saddle point problem in distributed memory? (b) does the direct solvers in petsc can run in distributed memory for solving saddle point problem? > ? <> > From: Stefano Zampini [mailto:stefano.zampini at gmail.com ] > Sent: Tuesday, 28 July 2020 6:55 PM > To: Bin Liu > > Cc: petsc-users at mcs.anl.gov > Subject: Re: [petsc-users] solving saddle point problem in distributed memory > > If you want advice you should post the error trace PETSc reports. > > Anyway, solving Stokes is not so trivial (without direct solvers, you may need mesh dependent information), but we have examples for it > > https://gitlab.com/petsc/petsc/-/blob/master/src/ksp/ksp/tutorials/ex42.c > https://gitlab.com/petsc/petsc/-/blob/master/src/ksp/ksp/tutorials/ex43.c > https://gitlab.com/petsc/petsc/-/blob/master/src/snes/tutorials/ex69.c > > If you scroll to the end of those files, you see a bunch of possible options either using PCFIELDSPLIT, PCBDDC or KSPFETIDP. > > > > On Jul 28, 2020, at 12:37 PM, Bin Liu > wrote: > > I would like to solve a saddle point problem arising from the stokes equation. I got successful to use the direct solvers in sequential runs. However, I would like to extend it for distributed memory computation. I tried to use superlu_dist, but the program returns errors. Is it possible to solve a saddle point problem in distributed memory using superlu_dist? Could anyone give a simple sample code to set up the parameters of the solver? > > Thanks -------------- next part -------------- An HTML attachment was scrubbed... URL: From lbllm2018 at hotmail.com Tue Jul 28 20:56:42 2020 From: lbllm2018 at hotmail.com (Bin Liu) Date: Wed, 29 Jul 2020 01:56:42 +0000 Subject: [petsc-users] solving saddle point problem in distributed memory In-Reply-To: References: <74D5C00B-69F3-4F9B-BD2F-CAA94EEAAD83@gmail.com> Message-ID: Sure. Thank you very much for your suggestions on setup. I also believe there are some problems in my setup. I will definitely give a try From: Matthew Knepley [mailto:knepley at gmail.com] Sent: Wednesday, 29 July 2020 9:36 AM To: Bin Liu Cc: Stefano Zampini ; petsc-users at mcs.anl.gov Subject: Re: [petsc-users] solving saddle point problem in distributed memory On Tue, Jul 28, 2020 at 9:00 PM Bin Liu > wrote: Thanks for your tutorials. Yes. I tried PCFIELDSPLIT. However, it only works for sequential runs. When I run the code in distributed memory, it reports errors. This would mean that your setup is incorrect. We run FIELDSPLIT all the time in parallel, and it has been used on some of the largest run ever made on supercomputers. Second, the way to directly solve a saddle-point problem is to directly solve the (0,0) block and the Schur complement. This is not hard: -pc_type fieldsplit -pc_fieldsplit_detect_saddlepoint -pc_fieldsplit_type schur -pc_fieldsplit_schur_fact_type full -pc_fieldsplit_schur_precondition full -fieldsplit_0_pc_type lu -fieldsplit_0_pc_factor_mat_solver_type superlu_dist -fieldsplit_1_pc_type lu -fieldsplit_1_pc_factor_mat_solver_type superlu_dist Thanks, Matt In fact, the essence of my wonder is (a) how to set up superlu_dist in petsc for solving saddle point problem in distributed memory? (b) does the direct solvers in petsc can run in distributed memory for solving saddle point problem? From: Stefano Zampini [mailto:stefano.zampini at gmail.com] Sent: Tuesday, 28 July 2020 6:55 PM To: Bin Liu > Cc: petsc-users at mcs.anl.gov Subject: Re: [petsc-users] solving saddle point problem in distributed memory If you want advice you should post the error trace PETSc reports. Anyway, solving Stokes is not so trivial (without direct solvers, you may need mesh dependent information), but we have examples for it https://gitlab.com/petsc/petsc/-/blob/master/src/ksp/ksp/tutorials/ex42.c https://gitlab.com/petsc/petsc/-/blob/master/src/ksp/ksp/tutorials/ex43.c https://gitlab.com/petsc/petsc/-/blob/master/src/snes/tutorials/ex69.c If you scroll to the end of those files, you see a bunch of possible options either using PCFIELDSPLIT, PCBDDC or KSPFETIDP. On Jul 28, 2020, at 12:37 PM, Bin Liu > wrote: I would like to solve a saddle point problem arising from the stokes equation. I got successful to use the direct solvers in sequential runs. However, I would like to extend it for distributed memory computation. I tried to use superlu_dist, but the program returns errors. Is it possible to solve a saddle point problem in distributed memory using superlu_dist? Could anyone give a simple sample code to set up the parameters of the solver? Thanks -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From lbllm2018 at hotmail.com Tue Jul 28 21:01:40 2020 From: lbllm2018 at hotmail.com (Bin Liu) Date: Wed, 29 Jul 2020 02:01:40 +0000 Subject: [petsc-users] solving saddle point problem in distributed memory In-Reply-To: <13D4A6F0-137D-435F-882B-042F85990C8C@petsc.dev> References: <74D5C00B-69F3-4F9B-BD2F-CAA94EEAAD83@gmail.com> <13D4A6F0-137D-435F-882B-042F85990C8C@petsc.dev> Message-ID: Dear Barry, Thanks for your explanation. Does it mean the default director solver LU in PETSc can also run in the distributed memory mode? I only used iterative solvers with preconditioners in distributed memory before. My experience in using director solver in parallel is indeed limited, especially the saddle point problem in parallel. I have done some search online, but I did not find a working set up at this moment. Could you give a sample setup of the direct solver for parallel run? It is really appreciated. From: Barry Smith [mailto:bsmith at petsc.dev] Sent: Wednesday, 29 July 2020 9:48 AM To: Bin Liu Cc: petsc-users at mcs.anl.gov Subject: Re: [petsc-users] solving saddle point problem in distributed memory SuperLU_DIST won't "magically" run on saddle point problems. It only does limited pivoting, realistically a parallel LU cannot always do complete pivoting or it becomes a sequential algorithm. For parallel you need to use PCFIELDSPLIT, for sequential you can use SuperLU (not SuperLU_DIST) since it can do more pivoting, being a sequential algorithm. Barry On Jul 28, 2020, at 7:59 PM, Bin Liu > wrote: Thanks for your tutorials. Yes. I tried PCFIELDSPLIT. However, it only works for sequential runs. When I run the code in distributed memory, it reports errors. In fact, the essence of my wonder is (a) how to set up superlu_dist in petsc for solving saddle point problem in distributed memory? (b) does the direct solvers in petsc can run in distributed memory for solving saddle point problem? From: Stefano Zampini [mailto:stefano.zampini at gmail.com] Sent: Tuesday, 28 July 2020 6:55 PM To: Bin Liu > Cc: petsc-users at mcs.anl.gov Subject: Re: [petsc-users] solving saddle point problem in distributed memory If you want advice you should post the error trace PETSc reports. Anyway, solving Stokes is not so trivial (without direct solvers, you may need mesh dependent information), but we have examples for it https://gitlab.com/petsc/petsc/-/blob/master/src/ksp/ksp/tutorials/ex42.c https://gitlab.com/petsc/petsc/-/blob/master/src/ksp/ksp/tutorials/ex43.c https://gitlab.com/petsc/petsc/-/blob/master/src/snes/tutorials/ex69.c If you scroll to the end of those files, you see a bunch of possible options either using PCFIELDSPLIT, PCBDDC or KSPFETIDP. On Jul 28, 2020, at 12:37 PM, Bin Liu > wrote: I would like to solve a saddle point problem arising from the stokes equation. I got successful to use the direct solvers in sequential runs. However, I would like to extend it for distributed memory computation. I tried to use superlu_dist, but the program returns errors. Is it possible to solve a saddle point problem in distributed memory using superlu_dist? Could anyone give a simple sample code to set up the parameters of the solver? Thanks -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at petsc.dev Tue Jul 28 21:49:47 2020 From: bsmith at petsc.dev (Barry Smith) Date: Tue, 28 Jul 2020 21:49:47 -0500 Subject: [petsc-users] solving saddle point problem in distributed memory In-Reply-To: References: <74D5C00B-69F3-4F9B-BD2F-CAA94EEAAD83@gmail.com> <13D4A6F0-137D-435F-882B-042F85990C8C@petsc.dev> Message-ID: <5654AFFA-500D-44F7-B793-F560C6311DEF@petsc.dev> No, PETSc direct solvers do not work in parallel. Only SuperLU_DIST, MUMPS, and PasTix. -pc_type lu (or cholesky depending on the package) -pc_factor_mat_solver_type superlu_dist or mumps or pastix to run the direct solver in parallel. > On Jul 28, 2020, at 9:01 PM, Bin Liu wrote: > > Dear Barry, > > Thanks for your explanation. Does it mean the default director solver LU in PETSc can also run in the distributed memory mode? I only used iterative solvers with preconditioners in distributed memory before. My experience in using director solver in parallel is indeed limited, especially the saddle point problem in parallel. I have done some search online, but I did not find a working set up at this moment. Could you give a sample setup of the direct solver for parallel run? It is really appreciated. > ? <> > From: Barry Smith [mailto:bsmith at petsc.dev] > Sent: Wednesday, 29 July 2020 9:48 AM > To: Bin Liu > Cc: petsc-users at mcs.anl.gov > Subject: Re: [petsc-users] solving saddle point problem in distributed memory > > > SuperLU_DIST won't "magically" run on saddle point problems. It only does limited pivoting, realistically a parallel LU cannot always do complete pivoting or it becomes a sequential algorithm. For parallel you need to use PCFIELDSPLIT, for sequential you can use SuperLU (not SuperLU_DIST) since it can do more pivoting, being a sequential algorithm. > > Barry > > > > On Jul 28, 2020, at 7:59 PM, Bin Liu > wrote: > > Thanks for your tutorials. Yes. I tried PCFIELDSPLIT. However, it only works for sequential runs. When I run the code in distributed memory, it reports errors. In fact, the essence of my wonder is (a) how to set up superlu_dist in petsc for solving saddle point problem in distributed memory? (b) does the direct solvers in petsc can run in distributed memory for solving saddle point problem? > > From: Stefano Zampini [mailto:stefano.zampini at gmail.com ] > Sent: Tuesday, 28 July 2020 6:55 PM > To: Bin Liu > > Cc: petsc-users at mcs.anl.gov > Subject: Re: [petsc-users] solving saddle point problem in distributed memory > > If you want advice you should post the error trace PETSc reports. > > Anyway, solving Stokes is not so trivial (without direct solvers, you may need mesh dependent information), but we have examples for it > > https://gitlab.com/petsc/petsc/-/blob/master/src/ksp/ksp/tutorials/ex42.c > https://gitlab.com/petsc/petsc/-/blob/master/src/ksp/ksp/tutorials/ex43.c > https://gitlab.com/petsc/petsc/-/blob/master/src/snes/tutorials/ex69.c > > If you scroll to the end of those files, you see a bunch of possible options either using PCFIELDSPLIT, PCBDDC or KSPFETIDP. > > > > > On Jul 28, 2020, at 12:37 PM, Bin Liu > wrote: > > I would like to solve a saddle point problem arising from the stokes equation. I got successful to use the direct solvers in sequential runs. However, I would like to extend it for distributed memory computation. I tried to use superlu_dist, but the program returns errors. Is it possible to solve a saddle point problem in distributed memory using superlu_dist? Could anyone give a simple sample code to set up the parameters of the solver? > > Thanks > -------------- next part -------------- An HTML attachment was scrubbed... URL: From lbllm2018 at hotmail.com Tue Jul 28 22:33:29 2020 From: lbllm2018 at hotmail.com (Bin Liu) Date: Wed, 29 Jul 2020 03:33:29 +0000 Subject: [petsc-users] solving saddle point problem in distributed memory In-Reply-To: <5654AFFA-500D-44F7-B793-F560C6311DEF@petsc.dev> References: <74D5C00B-69F3-4F9B-BD2F-CAA94EEAAD83@gmail.com> <13D4A6F0-137D-435F-882B-042F85990C8C@petsc.dev> , <5654AFFA-500D-44F7-B793-F560C6311DEF@petsc.dev> Message-ID: Dear Barry, Thanks for your suggestions. I just tested it. I still get the same error. I attached it below: [1]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [1]PETSC ERROR: Object is in wrong state [1]PETSC ERROR: Matrix is missing diagonal entry in the zeroed row 24214 [1]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [1]PETSC ERROR: Petsc Release Version 3.9.1, Apr, 29, 2018 [1]PETSC ERROR: ./re100 on a named kdp-MS-7850 by kdp Wed Jul 29 11:26:57 2020 [1]PETSC ERROR: Configure options --prefix=/home/kdp/petsc --debugging=0 --with-mpi-dir=/home/kdp/mpich --download-fblaslapack --download-metis --download-parmetis --download-hypre=/home/kdp/hypre-2.14.0.tar.gz --download-superlu_dist [1]PETSC ERROR: #1 MatZeroRows_SeqAIJ() line 1819 in /home/kdp/petsc-3.9.1/src/mat/impls/aij/seq/aij.c [1]PETSC ERROR: #2 MatZeroRows() line 5710 in /home/kdp/petsc-3.9.1/src/mat/interface/matrix.c [1]PETSC ERROR: #3 MatZeroRows_MPIAIJ() line 797 in /home/kdp/petsc-3.9.1/src/mat/impls/aij/mpi/mpiaij.c [1]PETSC ERROR: #4 MatZeroRows() line 5710 in /home/kdp/petsc-3.9.1/src/mat/interface/matrix.c It still report missing diagonal entry. However I have epxlicitly assign 0.0 to the diagonal element of the system matrix and solve it in sequential mode without error. Where could possibly my wrong set up be? Best ________________________________ From: Barry Smith Sent: Wednesday, July 29, 2020 10:49 AM To: Bin Liu Cc: petsc-users at mcs.anl.gov Subject: Re: [petsc-users] solving saddle point problem in distributed memory No, PETSc direct solvers do not work in parallel. Only SuperLU_DIST, MUMPS, and PasTix. -pc_type lu (or cholesky depending on the package) -pc_factor_mat_solver_type superlu_dist or mumps or pastix to run the direct solver in parallel. On Jul 28, 2020, at 9:01 PM, Bin Liu > wrote: Dear Barry, Thanks for your explanation. Does it mean the default director solver LU in PETSc can also run in the distributed memory mode? I only used iterative solvers with preconditioners in distributed memory before. My experience in using director solver in parallel is indeed limited, especially the saddle point problem in parallel. I have done some search online, but I did not find a working set up at this moment. Could you give a sample setup of the direct solver for parallel run? It is really appreciated. From: Barry Smith [mailto:bsmith at petsc.dev] Sent: Wednesday, 29 July 2020 9:48 AM To: Bin Liu > Cc: petsc-users at mcs.anl.gov Subject: Re: [petsc-users] solving saddle point problem in distributed memory SuperLU_DIST won't "magically" run on saddle point problems. It only does limited pivoting, realistically a parallel LU cannot always do complete pivoting or it becomes a sequential algorithm. For parallel you need to use PCFIELDSPLIT, for sequential you can use SuperLU (not SuperLU_DIST) since it can do more pivoting, being a sequential algorithm. Barry On Jul 28, 2020, at 7:59 PM, Bin Liu > wrote: Thanks for your tutorials. Yes. I tried PCFIELDSPLIT. However, it only works for sequential runs. When I run the code in distributed memory, it reports errors. In fact, the essence of my wonder is (a) how to set up superlu_dist in petsc for solving saddle point problem in distributed memory? (b) does the direct solvers in petsc can run in distributed memory for solving saddle point problem? From: Stefano Zampini [mailto:stefano.zampini at gmail.com] Sent: Tuesday, 28 July 2020 6:55 PM To: Bin Liu > Cc: petsc-users at mcs.anl.gov Subject: Re: [petsc-users] solving saddle point problem in distributed memory If you want advice you should post the error trace PETSc reports. Anyway, solving Stokes is not so trivial (without direct solvers, you may need mesh dependent information), but we have examples for it https://gitlab.com/petsc/petsc/-/blob/master/src/ksp/ksp/tutorials/ex42.c https://gitlab.com/petsc/petsc/-/blob/master/src/ksp/ksp/tutorials/ex43.c https://gitlab.com/petsc/petsc/-/blob/master/src/snes/tutorials/ex69.c If you scroll to the end of those files, you see a bunch of possible options either using PCFIELDSPLIT, PCBDDC or KSPFETIDP. On Jul 28, 2020, at 12:37 PM, Bin Liu > wrote: I would like to solve a saddle point problem arising from the stokes equation. I got successful to use the direct solvers in sequential runs. However, I would like to extend it for distributed memory computation. I tried to use superlu_dist, but the program returns errors. Is it possible to solve a saddle point problem in distributed memory using superlu_dist? Could anyone give a simple sample code to set up the parameters of the solver? Thanks -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Wed Jul 29 04:20:44 2020 From: knepley at gmail.com (Matthew Knepley) Date: Wed, 29 Jul 2020 05:20:44 -0400 Subject: [petsc-users] solving saddle point problem in distributed memory In-Reply-To: References: <74D5C00B-69F3-4F9B-BD2F-CAA94EEAAD83@gmail.com> <13D4A6F0-137D-435F-882B-042F85990C8C@petsc.dev> <5654AFFA-500D-44F7-B793-F560C6311DEF@petsc.dev> Message-ID: On Tue, Jul 28, 2020 at 11:33 PM Bin Liu wrote: > Dear Barry, > > Thanks for your suggestions. I just tested it. I still get the same error. > I attached it below: > > [1]PETSC ERROR: --------------------- Error Message > -------------------------------------------------------------- > [1]PETSC ERROR: Object is in wrong state > [1]PETSC ERROR: Matrix is missing diagonal entry in the zeroed row 24214 > [1]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html > for trouble shooting. > [1]PETSC ERROR: Petsc Release Version 3.9.1, Apr, 29, 2018 > [1]PETSC ERROR: ./re100 on a named kdp-MS-7850 by kdp Wed Jul 29 11:26:57 > 2020 > [1]PETSC ERROR: Configure options --prefix=/home/kdp/petsc --debugging=0 > --with-mpi-dir=/home/kdp/mpich --download-fblaslapack --download-metis > --download-parmetis --download-hypre=/home/kdp/hypre-2.14.0.tar.gz > --download-superlu_dist > [1]PETSC ERROR: #1 MatZeroRows_SeqAIJ() line 1819 in > /home/kdp/petsc-3.9.1/src/mat/impls/aij/seq/aij.c > [1]PETSC ERROR: #2 MatZeroRows() line 5710 in > /home/kdp/petsc-3.9.1/src/mat/interface/matrix.c > [1]PETSC ERROR: #3 MatZeroRows_MPIAIJ() line 797 in > /home/kdp/petsc-3.9.1/src/mat/impls/aij/mpi/mpiaij.c > [1]PETSC ERROR: #4 MatZeroRows() line 5710 in > /home/kdp/petsc-3.9.1/src/mat/interface/matrix.c > > It still report missing diagonal entry. However I have epxlicitly assign > 0.0 to the diagonal element of the system matrix and solve it in sequential > mode without error. Where could possibly my wrong set up be? > 1) Please send the complete error message. The stack trace here is cut off. 2) It does not look like it is failing in the solver, but in the call to MatZeroRows(), which as it says, requires diagonal elements 3) It looks like you are not assembling the matrix correctly in parallel I recommend making as small a problem as possible, running on 1 and 2 processes, and printing out the matrix using -mat_view to compare them entry-by-entry. Thanks, Matt Best > > ------------------------------ > *From:* Barry Smith > *Sent:* Wednesday, July 29, 2020 10:49 AM > *To:* Bin Liu > *Cc:* petsc-users at mcs.anl.gov > *Subject:* Re: [petsc-users] solving saddle point problem in distributed > memory > > > No, PETSc direct solvers do not work in parallel. Only SuperLU_DIST, > MUMPS, and PasTix. > > -pc_type lu (or cholesky depending on the package) > -pc_factor_mat_solver_type superlu_dist or mumps or pastix > > to run the direct solver in parallel. > > > > On Jul 28, 2020, at 9:01 PM, Bin Liu wrote: > > Dear Barry, > > > > Thanks for your explanation. Does it mean the default director solver LU > in PETSc can also run in the distributed memory mode? I only used iterative > solvers with preconditioners in distributed memory before. My experience > in using director solver in parallel is indeed limited, especially the > saddle point problem in parallel. I have done some search online, but I did > not find a working set up at this moment. Could you give a sample setup of > the direct solver for parallel run? It is really appreciated. > > > > *From:* Barry Smith [mailto:bsmith at petsc.dev ] > *Sent:* Wednesday, 29 July 2020 9:48 AM > *To:* Bin Liu > *Cc:* petsc-users at mcs.anl.gov > *Subject:* Re: [petsc-users] solving saddle point problem in distributed > memory > > > > > > SuperLU_DIST won't "magically" run on saddle point problems. It only > does limited pivoting, realistically a parallel LU cannot always do > complete pivoting or it becomes a sequential algorithm. For parallel you > need to use PCFIELDSPLIT, for sequential you can use SuperLU (not > SuperLU_DIST) since it can do more pivoting, being a sequential algorithm. > > > > Barry > > > > > > On Jul 28, 2020, at 7:59 PM, Bin Liu wrote: > > > > Thanks for your tutorials. Yes. I tried PCFIELDSPLIT. However, it only > works for sequential runs. When I run the code in distributed memory, it > reports errors. In fact, the essence of my wonder is (a) how to set up > superlu_dist in petsc for solving saddle point problem in distributed > memory? (b) does the direct solvers in petsc can run in distributed memory > for solving saddle point problem? > > > > *From:* Stefano Zampini [mailto:stefano.zampini at gmail.com > ] > *Sent:* Tuesday, 28 July 2020 6:55 PM > *To:* Bin Liu > *Cc:* petsc-users at mcs.anl.gov > *Subject:* Re: [petsc-users] solving saddle point problem in distributed > memory > > > > If you want advice you should post the error trace PETSc reports. > > > > Anyway, solving Stokes is not so trivial (without direct solvers, you may > need mesh dependent information), but we have examples for it > > > > https://gitlab.com/petsc/petsc/-/blob/master/src/ksp/ksp/tutorials/ex42.c > > https://gitlab.com/petsc/petsc/-/blob/master/src/ksp/ksp/tutorials/ex43.c > > https://gitlab.com/petsc/petsc/-/blob/master/src/snes/tutorials/ex69.c > > > > If you scroll to the end of those files, you see a bunch of possible > options either using PCFIELDSPLIT, PCBDDC or KSPFETIDP. > > > > > > > On Jul 28, 2020, at 12:37 PM, Bin Liu wrote: > > > > I would like to solve a saddle point problem arising from the stokes > equation. I got successful to use the direct solvers in sequential runs. > However, I would like to extend it for distributed memory computation. I > tried to use superlu_dist, but the program returns errors. Is it possible > to solve a saddle point problem in distributed memory using superlu_dist? > Could anyone give a simple sample code to set up the parameters of the > solver? > > > > Thanks > > > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From lbllm2018 at hotmail.com Wed Jul 29 05:30:34 2020 From: lbllm2018 at hotmail.com (Bin Liu) Date: Wed, 29 Jul 2020 10:30:34 +0000 Subject: [petsc-users] solving saddle point problem in distributed memory In-Reply-To: References: <74D5C00B-69F3-4F9B-BD2F-CAA94EEAAD83@gmail.com> <13D4A6F0-137D-435F-882B-042F85990C8C@petsc.dev> <5654AFFA-500D-44F7-B793-F560C6311DEF@petsc.dev> Message-ID: Thanks for your suggestions. Sure. I will check my codes of MatZeroRows from a small problem. The program does not terminate during solving in distributed memory, but repeating this error in run time. In fact, this is the entire error message repeating itself during runtime. From: Matthew Knepley [mailto:knepley at gmail.com] Sent: Wednesday, 29 July 2020 5:21 PM To: Bin Liu Cc: Barry Smith ; petsc-users at mcs.anl.gov Subject: Re: [petsc-users] solving saddle point problem in distributed memory On Tue, Jul 28, 2020 at 11:33 PM Bin Liu > wrote: Dear Barry, Thanks for your suggestions. I just tested it. I still get the same error. I attached it below: [1]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [1]PETSC ERROR: Object is in wrong state [1]PETSC ERROR: Matrix is missing diagonal entry in the zeroed row 24214 [1]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [1]PETSC ERROR: Petsc Release Version 3.9.1, Apr, 29, 2018 [1]PETSC ERROR: ./re100 on a named kdp-MS-7850 by kdp Wed Jul 29 11:26:57 2020 [1]PETSC ERROR: Configure options --prefix=/home/kdp/petsc --debugging=0 --with-mpi-dir=/home/kdp/mpich --download-fblaslapack --download-metis --download-parmetis --download-hypre=/home/kdp/hypre-2.14.0.tar.gz --download-superlu_dist [1]PETSC ERROR: #1 MatZeroRows_SeqAIJ() line 1819 in /home/kdp/petsc-3.9.1/src/mat/impls/aij/seq/aij.c [1]PETSC ERROR: #2 MatZeroRows() line 5710 in /home/kdp/petsc-3.9.1/src/mat/interface/matrix.c [1]PETSC ERROR: #3 MatZeroRows_MPIAIJ() line 797 in /home/kdp/petsc-3.9.1/src/mat/impls/aij/mpi/mpiaij.c [1]PETSC ERROR: #4 MatZeroRows() line 5710 in /home/kdp/petsc-3.9.1/src/mat/interface/matrix.c It still report missing diagonal entry. However I have epxlicitly assign 0.0 to the diagonal element of the system matrix and solve it in sequential mode without error. Where could possibly my wrong set up be? 1) Please send the complete error message. The stack trace here is cut off. 2) It does not look like it is failing in the solver, but in the call to MatZeroRows(), which as it says, requires diagonal elements 3) It looks like you are not assembling the matrix correctly in parallel I recommend making as small a problem as possible, running on 1 and 2 processes, and printing out the matrix using -mat_view to compare them entry-by-entry. Thanks, Matt Best ________________________________ From: Barry Smith > Sent: Wednesday, July 29, 2020 10:49 AM To: Bin Liu > Cc: petsc-users at mcs.anl.gov > Subject: Re: [petsc-users] solving saddle point problem in distributed memory No, PETSc direct solvers do not work in parallel. Only SuperLU_DIST, MUMPS, and PasTix. -pc_type lu (or cholesky depending on the package) -pc_factor_mat_solver_type superlu_dist or mumps or pastix to run the direct solver in parallel. On Jul 28, 2020, at 9:01 PM, Bin Liu > wrote: Dear Barry, Thanks for your explanation. Does it mean the default director solver LU in PETSc can also run in the distributed memory mode? I only used iterative solvers with preconditioners in distributed memory before. My experience in using director solver in parallel is indeed limited, especially the saddle point problem in parallel. I have done some search online, but I did not find a working set up at this moment. Could you give a sample setup of the direct solver for parallel run? It is really appreciated. From: Barry Smith [mailto:bsmith at petsc.dev] Sent: Wednesday, 29 July 2020 9:48 AM To: Bin Liu > Cc: petsc-users at mcs.anl.gov Subject: Re: [petsc-users] solving saddle point problem in distributed memory SuperLU_DIST won't "magically" run on saddle point problems. It only does limited pivoting, realistically a parallel LU cannot always do complete pivoting or it becomes a sequential algorithm. For parallel you need to use PCFIELDSPLIT, for sequential you can use SuperLU (not SuperLU_DIST) since it can do more pivoting, being a sequential algorithm. Barry On Jul 28, 2020, at 7:59 PM, Bin Liu > wrote: Thanks for your tutorials. Yes. I tried PCFIELDSPLIT. However, it only works for sequential runs. When I run the code in distributed memory, it reports errors. In fact, the essence of my wonder is (a) how to set up superlu_dist in petsc for solving saddle point problem in distributed memory? (b) does the direct solvers in petsc can run in distributed memory for solving saddle point problem? From: Stefano Zampini [mailto:stefano.zampini at gmail.com] Sent: Tuesday, 28 July 2020 6:55 PM To: Bin Liu > Cc: petsc-users at mcs.anl.gov Subject: Re: [petsc-users] solving saddle point problem in distributed memory If you want advice you should post the error trace PETSc reports. Anyway, solving Stokes is not so trivial (without direct solvers, you may need mesh dependent information), but we have examples for it https://gitlab.com/petsc/petsc/-/blob/master/src/ksp/ksp/tutorials/ex42.c https://gitlab.com/petsc/petsc/-/blob/master/src/ksp/ksp/tutorials/ex43.c https://gitlab.com/petsc/petsc/-/blob/master/src/snes/tutorials/ex69.c If you scroll to the end of those files, you see a bunch of possible options either using PCFIELDSPLIT, PCBDDC or KSPFETIDP. On Jul 28, 2020, at 12:37 PM, Bin Liu > wrote: I would like to solve a saddle point problem arising from the stokes equation. I got successful to use the direct solvers in sequential runs. However, I would like to extend it for distributed memory computation. I tried to use superlu_dist, but the program returns errors. Is it possible to solve a saddle point problem in distributed memory using superlu_dist? Could anyone give a simple sample code to set up the parameters of the solver? Thanks -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From eda.oktay at metu.edu.tr Wed Jul 29 05:51:50 2020 From: eda.oktay at metu.edu.tr (Eda Oktay) Date: Wed, 29 Jul 2020 13:51:50 +0300 Subject: [petsc-users] Petsc configure error (cmake) In-Reply-To: References: Message-ID: Dear Barry, I wrote the followings: $ patch -p1 < cmakemake.patch And it gave the following line: (Stripping trailing CRs from patch; use --binary to disable.) patching file config/BuildSystem/config/packages/cmake.py Then I configured as follows: ./configure --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --download-make --download-mpich --download-openblas --download-metis --download-parmetis --download-chaco --download-slepc=/home/eda/slepc-v3.13.2.tar.gz --download-cmake --with-X=1 But I am still getting the following error: ******************************************************************************* UNABLE to CONFIGURE with GIVEN OPTIONS (see configure.log for details): ------------------------------------------------------------------------------- Error running configure on CMAKE ******************************************************************************* Am I doing something wrong? Thanks! Eda Barry Smith , 29 Tem 2020 ?ar, 00:10 tarihinde ?unu yazd?: > > > It's better to use git than download tarballs. With tarballs it is not possible to access recent fixes or try branches. > > Here is what you can do, save the attachment as cmakemake.patch and do > > patch -p1 < cmakemake.patch > > in the PETSc directory > > Then run configure as before > > Barry > > > > > On Jul 28, 2020, at 3:28 PM, Eda Oktay wrote: > > > > Dear Barry, > > > > Thank you so much for answering. So, I downloaded the tar.gz file and > > now, should I configure this file with my previous options? I am > > using the following line: > > > > ./configure --with-cc=gcc --with-cxx=g++ --with-fc=gfortran > > --download-make --download-mpich --download-openblas --download-metis > > --download-parmetis --download-chaco > > --download-slepc=/home/eda/slepc-v3.13.2.tar.gz --download-cmake > > --with-X=1 > > > > Thanks so much again! > > > > Eda > > > > Barry Smith , 28 Tem 2020 Sal, 20:55 tarihinde ?unu yazd?: > >> > >> > >> Eda, > >> > >> The branch barry/2020-07-28/cmake-use-make-if-given/maint should resolve the problem. It will force the cmake build to use the make obtained with --download-make > >> > >> Barry > >> > >> > >> On Jul 28, 2020, at 9:48 AM, Matthew Knepley wrote: > >> > >> On Tue, Jul 28, 2020 at 10:45 AM Eda Oktay wrote: > >>> > >>> I am trying to download Petsc with slepc,openblas,mpich,meti,parmetis > >>> and chaco to a linux computer (probably > >>> ubuntu, I don't know because I am using it via ssh). > >>> > >>> Due to an error, for metis, I needed to download cmake, so I added > >>> this option, too. However, I got an error telling me that "Error > >>> running configure on cmake". I attached configure.log. > >>> > >>> What should I do? Where is the problem and how can ? fix it? > >>> > >>> Thank you so much for answering, > >> > >> > >> It looks like you do not have 'make' in your path, and CMake is feaking out: > >> > >> CMAKE was just downloaded, forcing a rebuild because cannot determine if package has changed > >> =============================================================================== > >> Running configure on CMAKE; this may take several minutes > >> =============================================================================== > >> > >> Running Executable WITHOUT threads to time it out > >> Executing: ./configure --prefix=/home/eda/petsc-3.13.2/arch-linux-c-debug --parallel=40 > >> stdout: > >> --------------------------------------------- > >> CMake 3.15.6, Copyright 2000-2019 Kitware, Inc. and Contributors > >> Found GNU toolchain > >> C compiler on this system is: gcc > >> C++ compiler on this system is: g++ > >> --------------------------------------------- > >> Error when bootstrapping CMake: > >> Cannot find appropriate Makefile processor on this system. > >> Please specify one using environment variable MAKE. > >> --------------------------------------------- > >> Log of errors: /home/eda/petsc-3.13.2/arch-linux-c-debug/externalpackages/cmake-3.15.6/Bootstrap.cmk/cmake_bootstrap.log > >> --------------------------------------------- > >> > >> Thanks, > >> > >> Matt > >> > >>> > >>> Eda > >> > >> > >> > >> -- > >> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. > >> -- Norbert Wiener > >> > >> https://www.cse.buffalo.edu/~knepley/ > >> > >> > -------------- next part -------------- A non-text attachment was scrubbed... Name: configure.log Type: text/x-log Size: 930903 bytes Desc: not available URL: From knepley at gmail.com Wed Jul 29 06:53:49 2020 From: knepley at gmail.com (Matthew Knepley) Date: Wed, 29 Jul 2020 07:53:49 -0400 Subject: [petsc-users] solving saddle point problem in distributed memory In-Reply-To: References: <74D5C00B-69F3-4F9B-BD2F-CAA94EEAAD83@gmail.com> <13D4A6F0-137D-435F-882B-042F85990C8C@petsc.dev> <5654AFFA-500D-44F7-B793-F560C6311DEF@petsc.dev> Message-ID: On Wed, Jul 29, 2020 at 6:30 AM Bin Liu wrote: > Thanks for your suggestions. Sure. I will check my codes of MatZeroRows > from a small problem. The program does not terminate during solving in > distributed memory, but repeating this error in run time. In fact, this is > the entire error message repeating itself during runtime. > Are you not checking the return codes for PETSc calls in your code? Matt > > > > > *From:* Matthew Knepley [mailto:knepley at gmail.com] > *Sent:* Wednesday, 29 July 2020 5:21 PM > *To:* Bin Liu > *Cc:* Barry Smith ; petsc-users at mcs.anl.gov > *Subject:* Re: [petsc-users] solving saddle point problem in distributed > memory > > > > On Tue, Jul 28, 2020 at 11:33 PM Bin Liu wrote: > > Dear Barry, > > > > Thanks for your suggestions. I just tested it. I still get the same error. > I attached it below: > > > > [1]PETSC ERROR: --------------------- Error Message > -------------------------------------------------------------- > > [1]PETSC ERROR: Object is in wrong state > > [1]PETSC ERROR: Matrix is missing diagonal entry in the zeroed row 24214 > > [1]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html > for trouble shooting. > > [1]PETSC ERROR: Petsc Release Version 3.9.1, Apr, 29, 2018 > > [1]PETSC ERROR: ./re100 on a named kdp-MS-7850 by kdp Wed Jul 29 11:26:57 > 2020 > > [1]PETSC ERROR: Configure options --prefix=/home/kdp/petsc --debugging=0 > --with-mpi-dir=/home/kdp/mpich --download-fblaslapack --download-metis > --download-parmetis --download-hypre=/home/kdp/hypre-2.14.0.tar.gz > --download-superlu_dist > > [1]PETSC ERROR: #1 MatZeroRows_SeqAIJ() line 1819 in > /home/kdp/petsc-3.9.1/src/mat/impls/aij/seq/aij.c > > [1]PETSC ERROR: #2 MatZeroRows() line 5710 in > /home/kdp/petsc-3.9.1/src/mat/interface/matrix.c > > [1]PETSC ERROR: #3 MatZeroRows_MPIAIJ() line 797 in > /home/kdp/petsc-3.9.1/src/mat/impls/aij/mpi/mpiaij.c > > [1]PETSC ERROR: #4 MatZeroRows() line 5710 in > /home/kdp/petsc-3.9.1/src/mat/interface/matrix.c > > > It still report missing diagonal entry. However I have epxlicitly assign > 0.0 to the diagonal element of the system matrix and solve it in sequential > mode without error. Where could possibly my wrong set up be? > > > > 1) Please send the complete error message. The stack trace here is cut off. > > > > 2) It does not look like it is failing in the solver, but in the call to > MatZeroRows(), which as it says, requires diagonal elements > > > > 3) It looks like you are not assembling the matrix correctly in parallel > > > > I recommend making as small a problem as possible, running on 1 and 2 > processes, and printing out the matrix using -mat_view to compare them > entry-by-entry. > > > > Thanks, > > > > Matt > > > > Best > > > ------------------------------ > > *From:* Barry Smith > *Sent:* Wednesday, July 29, 2020 10:49 AM > *To:* Bin Liu > *Cc:* petsc-users at mcs.anl.gov > *Subject:* Re: [petsc-users] solving saddle point problem in distributed > memory > > > > > > No, PETSc direct solvers do not work in parallel. Only SuperLU_DIST, > MUMPS, and PasTix. > > > > -pc_type lu (or cholesky depending on the package) > -pc_factor_mat_solver_type superlu_dist or mumps or pastix > > > > to run the direct solver in parallel. > > > > > > > > On Jul 28, 2020, at 9:01 PM, Bin Liu wrote: > > > > Dear Barry, > > > > Thanks for your explanation. Does it mean the default director solver LU > in PETSc can also run in the distributed memory mode? I only used iterative > solvers with preconditioners in distributed memory before. My experience > in using director solver in parallel is indeed limited, especially the > saddle point problem in parallel. I have done some search online, but I did > not find a working set up at this moment. Could you give a sample setup of > the direct solver for parallel run? It is really appreciated. > > > > *From:* Barry Smith [mailto:bsmith at petsc.dev ] > *Sent:* Wednesday, 29 July 2020 9:48 AM > *To:* Bin Liu > *Cc:* petsc-users at mcs.anl.gov > *Subject:* Re: [petsc-users] solving saddle point problem in distributed > memory > > > > > > SuperLU_DIST won't "magically" run on saddle point problems. It only > does limited pivoting, realistically a parallel LU cannot always do > complete pivoting or it becomes a sequential algorithm. For parallel you > need to use PCFIELDSPLIT, for sequential you can use SuperLU (not > SuperLU_DIST) since it can do more pivoting, being a sequential algorithm. > > > > Barry > > > > > > On Jul 28, 2020, at 7:59 PM, Bin Liu wrote: > > > > Thanks for your tutorials. Yes. I tried PCFIELDSPLIT. However, it only > works for sequential runs. When I run the code in distributed memory, it > reports errors. In fact, the essence of my wonder is (a) how to set up > superlu_dist in petsc for solving saddle point problem in distributed > memory? (b) does the direct solvers in petsc can run in distributed memory > for solving saddle point problem? > > > > *From:* Stefano Zampini [mailto:stefano.zampini at gmail.com > ] > *Sent:* Tuesday, 28 July 2020 6:55 PM > *To:* Bin Liu > *Cc:* petsc-users at mcs.anl.gov > *Subject:* Re: [petsc-users] solving saddle point problem in distributed > memory > > > > If you want advice you should post the error trace PETSc reports. > > > > Anyway, solving Stokes is not so trivial (without direct solvers, you may > need mesh dependent information), but we have examples for it > > > > https://gitlab.com/petsc/petsc/-/blob/master/src/ksp/ksp/tutorials/ex42.c > > https://gitlab.com/petsc/petsc/-/blob/master/src/ksp/ksp/tutorials/ex43.c > > https://gitlab.com/petsc/petsc/-/blob/master/src/snes/tutorials/ex69.c > > > > If you scroll to the end of those files, you see a bunch of possible > options either using PCFIELDSPLIT, PCBDDC or KSPFETIDP. > > > > > > On Jul 28, 2020, at 12:37 PM, Bin Liu wrote: > > > > I would like to solve a saddle point problem arising from the stokes > equation. I got successful to use the direct solvers in sequential runs. > However, I would like to extend it for distributed memory computation. I > tried to use superlu_dist, but the program returns errors. Is it possible > to solve a saddle point problem in distributed memory using superlu_dist? > Could anyone give a simple sample code to set up the parameters of the > solver? > > > > Thanks > > > > > > > > > -- > > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > > > https://www.cse.buffalo.edu/~knepley/ > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at petsc.dev Wed Jul 29 10:51:06 2020 From: bsmith at petsc.dev (Barry Smith) Date: Wed, 29 Jul 2020 10:51:06 -0500 Subject: [petsc-users] Petsc configure error (cmake) In-Reply-To: References: Message-ID: <50206B2B-19EC-479F-A4C3-0959B4B180E2@petsc.dev> Please try rm -rf /home/eda/petsc-3.13.2/arch-linux-c-debug and then run configure again. It lists the correct make to use Makefile processor on this system is: /home/eda/petsc-3.13.2/arch-linux-c-debug/bin/make So I am hoping the problem is from the crazy cmake caching nonsense and cleaning out the old files will resolve the problem. Barry > On Jul 29, 2020, at 5:51 AM, Eda Oktay wrote: > > Dear Barry, > > I wrote the followings: > > $ patch -p1 < cmakemake.patch > > And it gave the following line: > > (Stripping trailing CRs from patch; use --binary to disable.) > patching file config/BuildSystem/config/packages/cmake.py > > Then I configured as follows: > > ./configure --with-cc=gcc --with-cxx=g++ --with-fc=gfortran > --download-make --download-mpich --download-openblas --download-metis > --download-parmetis --download-chaco > --download-slepc=/home/eda/slepc-v3.13.2.tar.gz --download-cmake > --with-X=1 > > But I am still getting the following error: > > ******************************************************************************* > UNABLE to CONFIGURE with GIVEN OPTIONS (see configure.log > for details): > ------------------------------------------------------------------------------- > Error running configure on CMAKE > ******************************************************************************* > > Am I doing something wrong? > > Thanks! > > Eda > > Barry Smith , 29 Tem 2020 ?ar, 00:10 tarihinde ?unu yazd?: >> >> >> It's better to use git than download tarballs. With tarballs it is not possible to access recent fixes or try branches. >> >> Here is what you can do, save the attachment as cmakemake.patch and do >> >> patch -p1 < cmakemake.patch >> >> in the PETSc directory >> >> Then run configure as before >> >> Barry >> >> >> >>> On Jul 28, 2020, at 3:28 PM, Eda Oktay wrote: >>> >>> Dear Barry, >>> >>> Thank you so much for answering. So, I downloaded the tar.gz file and >>> now, should I configure this file with my previous options? I am >>> using the following line: >>> >>> ./configure --with-cc=gcc --with-cxx=g++ --with-fc=gfortran >>> --download-make --download-mpich --download-openblas --download-metis >>> --download-parmetis --download-chaco >>> --download-slepc=/home/eda/slepc-v3.13.2.tar.gz --download-cmake >>> --with-X=1 >>> >>> Thanks so much again! >>> >>> Eda >>> >>> Barry Smith , 28 Tem 2020 Sal, 20:55 tarihinde ?unu yazd?: >>>> >>>> >>>> Eda, >>>> >>>> The branch barry/2020-07-28/cmake-use-make-if-given/maint should resolve the problem. It will force the cmake build to use the make obtained with --download-make >>>> >>>> Barry >>>> >>>> >>>> On Jul 28, 2020, at 9:48 AM, Matthew Knepley wrote: >>>> >>>> On Tue, Jul 28, 2020 at 10:45 AM Eda Oktay wrote: >>>>> >>>>> I am trying to download Petsc with slepc,openblas,mpich,meti,parmetis >>>>> and chaco to a linux computer (probably >>>>> ubuntu, I don't know because I am using it via ssh). >>>>> >>>>> Due to an error, for metis, I needed to download cmake, so I added >>>>> this option, too. However, I got an error telling me that "Error >>>>> running configure on cmake". I attached configure.log. >>>>> >>>>> What should I do? Where is the problem and how can ? fix it? >>>>> >>>>> Thank you so much for answering, >>>> >>>> >>>> It looks like you do not have 'make' in your path, and CMake is feaking out: >>>> >>>> CMAKE was just downloaded, forcing a rebuild because cannot determine if package has changed >>>> =============================================================================== >>>> Running configure on CMAKE; this may take several minutes >>>> =============================================================================== >>>> >>>> Running Executable WITHOUT threads to time it out >>>> Executing: ./configure --prefix=/home/eda/petsc-3.13.2/arch-linux-c-debug --parallel=40 >>>> stdout: >>>> --------------------------------------------- >>>> CMake 3.15.6, Copyright 2000-2019 Kitware, Inc. and Contributors >>>> Found GNU toolchain >>>> C compiler on this system is: gcc >>>> C++ compiler on this system is: g++ >>>> --------------------------------------------- >>>> Error when bootstrapping CMake: >>>> Cannot find appropriate Makefile processor on this system. >>>> Please specify one using environment variable MAKE. >>>> --------------------------------------------- >>>> Log of errors: /home/eda/petsc-3.13.2/arch-linux-c-debug/externalpackages/cmake-3.15.6/Bootstrap.cmk/cmake_bootstrap.log >>>> --------------------------------------------- >>>> >>>> Thanks, >>>> >>>> Matt >>>> >>>>> >>>>> Eda >>>> >>>> >>>> >>>> -- >>>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. >>>> -- Norbert Wiener >>>> >>>> https://www.cse.buffalo.edu/~knepley/ >>>> >>>> >> > From eda.oktay at metu.edu.tr Wed Jul 29 11:07:23 2020 From: eda.oktay at metu.edu.tr (Eda Oktay) Date: Wed, 29 Jul 2020 19:07:23 +0300 Subject: [petsc-users] Petsc configure error (cmake) In-Reply-To: <50206B2B-19EC-479F-A4C3-0959B4B180E2@petsc.dev> References: <50206B2B-19EC-479F-A4C3-0959B4B180E2@petsc.dev> Message-ID: Dear Barry, I am afraid I am still getting the same error. Thanks, Eda Barry Smith , 29 Tem 2020 ?ar, 18:51 tarihinde ?unu yazd?: > > > Please try > > rm -rf /home/eda/petsc-3.13.2/arch-linux-c-debug > > and then run configure again. > > It lists the correct make to use > > Makefile processor on this system is: /home/eda/petsc-3.13.2/arch-linux-c-debug/bin/make > > So I am hoping the problem is from the crazy cmake caching nonsense and cleaning out the old files will resolve the problem. > > Barry > > > > On Jul 29, 2020, at 5:51 AM, Eda Oktay wrote: > > > > Dear Barry, > > > > I wrote the followings: > > > > $ patch -p1 < cmakemake.patch > > > > And it gave the following line: > > > > (Stripping trailing CRs from patch; use --binary to disable.) > > patching file config/BuildSystem/config/packages/cmake.py > > > > Then I configured as follows: > > > > ./configure --with-cc=gcc --with-cxx=g++ --with-fc=gfortran > > --download-make --download-mpich --download-openblas --download-metis > > --download-parmetis --download-chaco > > --download-slepc=/home/eda/slepc-v3.13.2.tar.gz --download-cmake > > --with-X=1 > > > > But I am still getting the following error: > > > > ******************************************************************************* > > UNABLE to CONFIGURE with GIVEN OPTIONS (see configure.log > > for details): > > ------------------------------------------------------------------------------- > > Error running configure on CMAKE > > ******************************************************************************* > > > > Am I doing something wrong? > > > > Thanks! > > > > Eda > > > > Barry Smith , 29 Tem 2020 ?ar, 00:10 tarihinde ?unu yazd?: > >> > >> > >> It's better to use git than download tarballs. With tarballs it is not possible to access recent fixes or try branches. > >> > >> Here is what you can do, save the attachment as cmakemake.patch and do > >> > >> patch -p1 < cmakemake.patch > >> > >> in the PETSc directory > >> > >> Then run configure as before > >> > >> Barry > >> > >> > >> > >>> On Jul 28, 2020, at 3:28 PM, Eda Oktay wrote: > >>> > >>> Dear Barry, > >>> > >>> Thank you so much for answering. So, I downloaded the tar.gz file and > >>> now, should I configure this file with my previous options? I am > >>> using the following line: > >>> > >>> ./configure --with-cc=gcc --with-cxx=g++ --with-fc=gfortran > >>> --download-make --download-mpich --download-openblas --download-metis > >>> --download-parmetis --download-chaco > >>> --download-slepc=/home/eda/slepc-v3.13.2.tar.gz --download-cmake > >>> --with-X=1 > >>> > >>> Thanks so much again! > >>> > >>> Eda > >>> > >>> Barry Smith , 28 Tem 2020 Sal, 20:55 tarihinde ?unu yazd?: > >>>> > >>>> > >>>> Eda, > >>>> > >>>> The branch barry/2020-07-28/cmake-use-make-if-given/maint should resolve the problem. It will force the cmake build to use the make obtained with --download-make > >>>> > >>>> Barry > >>>> > >>>> > >>>> On Jul 28, 2020, at 9:48 AM, Matthew Knepley wrote: > >>>> > >>>> On Tue, Jul 28, 2020 at 10:45 AM Eda Oktay wrote: > >>>>> > >>>>> I am trying to download Petsc with slepc,openblas,mpich,meti,parmetis > >>>>> and chaco to a linux computer (probably > >>>>> ubuntu, I don't know because I am using it via ssh). > >>>>> > >>>>> Due to an error, for metis, I needed to download cmake, so I added > >>>>> this option, too. However, I got an error telling me that "Error > >>>>> running configure on cmake". I attached configure.log. > >>>>> > >>>>> What should I do? Where is the problem and how can ? fix it? > >>>>> > >>>>> Thank you so much for answering, > >>>> > >>>> > >>>> It looks like you do not have 'make' in your path, and CMake is feaking out: > >>>> > >>>> CMAKE was just downloaded, forcing a rebuild because cannot determine if package has changed > >>>> =============================================================================== > >>>> Running configure on CMAKE; this may take several minutes > >>>> =============================================================================== > >>>> > >>>> Running Executable WITHOUT threads to time it out > >>>> Executing: ./configure --prefix=/home/eda/petsc-3.13.2/arch-linux-c-debug --parallel=40 > >>>> stdout: > >>>> --------------------------------------------- > >>>> CMake 3.15.6, Copyright 2000-2019 Kitware, Inc. and Contributors > >>>> Found GNU toolchain > >>>> C compiler on this system is: gcc > >>>> C++ compiler on this system is: g++ > >>>> --------------------------------------------- > >>>> Error when bootstrapping CMake: > >>>> Cannot find appropriate Makefile processor on this system. > >>>> Please specify one using environment variable MAKE. > >>>> --------------------------------------------- > >>>> Log of errors: /home/eda/petsc-3.13.2/arch-linux-c-debug/externalpackages/cmake-3.15.6/Bootstrap.cmk/cmake_bootstrap.log > >>>> --------------------------------------------- > >>>> > >>>> Thanks, > >>>> > >>>> Matt > >>>> > >>>>> > >>>>> Eda > >>>> > >>>> > >>>> > >>>> -- > >>>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. > >>>> -- Norbert Wiener > >>>> > >>>> https://www.cse.buffalo.edu/~knepley/ > >>>> > >>>> > >> > > > -------------- next part -------------- A non-text attachment was scrubbed... Name: configure.log Type: text/x-log Size: 2215992 bytes Desc: not available URL: From balay at mcs.anl.gov Wed Jul 29 11:25:18 2020 From: balay at mcs.anl.gov (Satish Balay) Date: Wed, 29 Jul 2020 11:25:18 -0500 (CDT) Subject: [petsc-users] Petsc configure error (cmake) In-Reply-To: References: <50206B2B-19EC-479F-A4C3-0959B4B180E2@petsc.dev> Message-ID: I can reproduce this - by uninstalling system make. Will check. Satish On Wed, 29 Jul 2020, Eda Oktay wrote: > Dear Barry, > > I am afraid I am still getting the same error. > > Thanks, > > Eda > > Barry Smith , 29 Tem 2020 ?ar, 18:51 tarihinde ?unu yazd?: > > > > > > Please try > > > > rm -rf /home/eda/petsc-3.13.2/arch-linux-c-debug > > > > and then run configure again. > > > > It lists the correct make to use > > > > Makefile processor on this system is: /home/eda/petsc-3.13.2/arch-linux-c-debug/bin/make > > > > So I am hoping the problem is from the crazy cmake caching nonsense and cleaning out the old files will resolve the problem. > > > > Barry > > > > > > > On Jul 29, 2020, at 5:51 AM, Eda Oktay wrote: > > > > > > Dear Barry, > > > > > > I wrote the followings: > > > > > > $ patch -p1 < cmakemake.patch > > > > > > And it gave the following line: > > > > > > (Stripping trailing CRs from patch; use --binary to disable.) > > > patching file config/BuildSystem/config/packages/cmake.py > > > > > > Then I configured as follows: > > > > > > ./configure --with-cc=gcc --with-cxx=g++ --with-fc=gfortran > > > --download-make --download-mpich --download-openblas --download-metis > > > --download-parmetis --download-chaco > > > --download-slepc=/home/eda/slepc-v3.13.2.tar.gz --download-cmake > > > --with-X=1 > > > > > > But I am still getting the following error: > > > > > > ******************************************************************************* > > > UNABLE to CONFIGURE with GIVEN OPTIONS (see configure.log > > > for details): > > > ------------------------------------------------------------------------------- > > > Error running configure on CMAKE > > > ******************************************************************************* > > > > > > Am I doing something wrong? > > > > > > Thanks! > > > > > > Eda > > > > > > Barry Smith , 29 Tem 2020 ?ar, 00:10 tarihinde ?unu yazd?: > > >> > > >> > > >> It's better to use git than download tarballs. With tarballs it is not possible to access recent fixes or try branches. > > >> > > >> Here is what you can do, save the attachment as cmakemake.patch and do > > >> > > >> patch -p1 < cmakemake.patch > > >> > > >> in the PETSc directory > > >> > > >> Then run configure as before > > >> > > >> Barry > > >> > > >> > > >> > > >>> On Jul 28, 2020, at 3:28 PM, Eda Oktay wrote: > > >>> > > >>> Dear Barry, > > >>> > > >>> Thank you so much for answering. So, I downloaded the tar.gz file and > > >>> now, should I configure this file with my previous options? I am > > >>> using the following line: > > >>> > > >>> ./configure --with-cc=gcc --with-cxx=g++ --with-fc=gfortran > > >>> --download-make --download-mpich --download-openblas --download-metis > > >>> --download-parmetis --download-chaco > > >>> --download-slepc=/home/eda/slepc-v3.13.2.tar.gz --download-cmake > > >>> --with-X=1 > > >>> > > >>> Thanks so much again! > > >>> > > >>> Eda > > >>> > > >>> Barry Smith , 28 Tem 2020 Sal, 20:55 tarihinde ?unu yazd?: > > >>>> > > >>>> > > >>>> Eda, > > >>>> > > >>>> The branch barry/2020-07-28/cmake-use-make-if-given/maint should resolve the problem. It will force the cmake build to use the make obtained with --download-make > > >>>> > > >>>> Barry > > >>>> > > >>>> > > >>>> On Jul 28, 2020, at 9:48 AM, Matthew Knepley wrote: > > >>>> > > >>>> On Tue, Jul 28, 2020 at 10:45 AM Eda Oktay wrote: > > >>>>> > > >>>>> I am trying to download Petsc with slepc,openblas,mpich,meti,parmetis > > >>>>> and chaco to a linux computer (probably > > >>>>> ubuntu, I don't know because I am using it via ssh). > > >>>>> > > >>>>> Due to an error, for metis, I needed to download cmake, so I added > > >>>>> this option, too. However, I got an error telling me that "Error > > >>>>> running configure on cmake". I attached configure.log. > > >>>>> > > >>>>> What should I do? Where is the problem and how can ? fix it? > > >>>>> > > >>>>> Thank you so much for answering, > > >>>> > > >>>> > > >>>> It looks like you do not have 'make' in your path, and CMake is feaking out: > > >>>> > > >>>> CMAKE was just downloaded, forcing a rebuild because cannot determine if package has changed > > >>>> =============================================================================== > > >>>> Running configure on CMAKE; this may take several minutes > > >>>> =============================================================================== > > >>>> > > >>>> Running Executable WITHOUT threads to time it out > > >>>> Executing: ./configure --prefix=/home/eda/petsc-3.13.2/arch-linux-c-debug --parallel=40 > > >>>> stdout: > > >>>> --------------------------------------------- > > >>>> CMake 3.15.6, Copyright 2000-2019 Kitware, Inc. and Contributors > > >>>> Found GNU toolchain > > >>>> C compiler on this system is: gcc > > >>>> C++ compiler on this system is: g++ > > >>>> --------------------------------------------- > > >>>> Error when bootstrapping CMake: > > >>>> Cannot find appropriate Makefile processor on this system. > > >>>> Please specify one using environment variable MAKE. > > >>>> --------------------------------------------- > > >>>> Log of errors: /home/eda/petsc-3.13.2/arch-linux-c-debug/externalpackages/cmake-3.15.6/Bootstrap.cmk/cmake_bootstrap.log > > >>>> --------------------------------------------- > > >>>> > > >>>> Thanks, > > >>>> > > >>>> Matt > > >>>> > > >>>>> > > >>>>> Eda > > >>>> > > >>>> > > >>>> > > >>>> -- > > >>>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. > > >>>> -- Norbert Wiener > > >>>> > > >>>> https://www.cse.buffalo.edu/~knepley/ > > >>>> > > >>>> > > >> > > > > > > From balay at mcs.anl.gov Wed Jul 29 12:20:22 2020 From: balay at mcs.anl.gov (Satish Balay) Date: Wed, 29 Jul 2020 12:20:22 -0500 (CDT) Subject: [petsc-users] Petsc configure error (cmake) In-Reply-To: References: <50206B2B-19EC-479F-A4C3-0959B4B180E2@petsc.dev> Message-ID: This is likely a cmake install bug. I submitted a report. https://gitlab.kitware.com/cmake/cmake/-/issues/21035 The workaround is to have make in PATH. Attaching this patch. And updating barry/2020-07-28/cmake-use-make-if-given/maint and https://gitlab.com/petsc/petsc/-/merge_requests/3004 with this change. Satish On Wed, 29 Jul 2020, Satish Balay via petsc-users wrote: > I can reproduce this - by uninstalling system make. > > Will check. > > Satish > > On Wed, 29 Jul 2020, Eda Oktay wrote: > > > Dear Barry, > > > > I am afraid I am still getting the same error. > > > > Thanks, > > > > Eda > > > > Barry Smith , 29 Tem 2020 ?ar, 18:51 tarihinde ?unu yazd?: > > > > > > > > > Please try > > > > > > rm -rf /home/eda/petsc-3.13.2/arch-linux-c-debug > > > > > > and then run configure again. > > > > > > It lists the correct make to use > > > > > > Makefile processor on this system is: /home/eda/petsc-3.13.2/arch-linux-c-debug/bin/make > > > > > > So I am hoping the problem is from the crazy cmake caching nonsense and cleaning out the old files will resolve the problem. > > > > > > Barry > > > > > > > > > > On Jul 29, 2020, at 5:51 AM, Eda Oktay wrote: > > > > > > > > Dear Barry, > > > > > > > > I wrote the followings: > > > > > > > > $ patch -p1 < cmakemake.patch > > > > > > > > And it gave the following line: > > > > > > > > (Stripping trailing CRs from patch; use --binary to disable.) > > > > patching file config/BuildSystem/config/packages/cmake.py > > > > > > > > Then I configured as follows: > > > > > > > > ./configure --with-cc=gcc --with-cxx=g++ --with-fc=gfortran > > > > --download-make --download-mpich --download-openblas --download-metis > > > > --download-parmetis --download-chaco > > > > --download-slepc=/home/eda/slepc-v3.13.2.tar.gz --download-cmake > > > > --with-X=1 > > > > > > > > But I am still getting the following error: > > > > > > > > ******************************************************************************* > > > > UNABLE to CONFIGURE with GIVEN OPTIONS (see configure.log > > > > for details): > > > > ------------------------------------------------------------------------------- > > > > Error running configure on CMAKE > > > > ******************************************************************************* > > > > > > > > Am I doing something wrong? > > > > > > > > Thanks! > > > > > > > > Eda > > > > > > > > Barry Smith , 29 Tem 2020 ?ar, 00:10 tarihinde ?unu yazd?: > > > >> > > > >> > > > >> It's better to use git than download tarballs. With tarballs it is not possible to access recent fixes or try branches. > > > >> > > > >> Here is what you can do, save the attachment as cmakemake.patch and do > > > >> > > > >> patch -p1 < cmakemake.patch > > > >> > > > >> in the PETSc directory > > > >> > > > >> Then run configure as before > > > >> > > > >> Barry > > > >> > > > >> > > > >> > > > >>> On Jul 28, 2020, at 3:28 PM, Eda Oktay wrote: > > > >>> > > > >>> Dear Barry, > > > >>> > > > >>> Thank you so much for answering. So, I downloaded the tar.gz file and > > > >>> now, should I configure this file with my previous options? I am > > > >>> using the following line: > > > >>> > > > >>> ./configure --with-cc=gcc --with-cxx=g++ --with-fc=gfortran > > > >>> --download-make --download-mpich --download-openblas --download-metis > > > >>> --download-parmetis --download-chaco > > > >>> --download-slepc=/home/eda/slepc-v3.13.2.tar.gz --download-cmake > > > >>> --with-X=1 > > > >>> > > > >>> Thanks so much again! > > > >>> > > > >>> Eda > > > >>> > > > >>> Barry Smith , 28 Tem 2020 Sal, 20:55 tarihinde ?unu yazd?: > > > >>>> > > > >>>> > > > >>>> Eda, > > > >>>> > > > >>>> The branch barry/2020-07-28/cmake-use-make-if-given/maint should resolve the problem. It will force the cmake build to use the make obtained with --download-make > > > >>>> > > > >>>> Barry > > > >>>> > > > >>>> > > > >>>> On Jul 28, 2020, at 9:48 AM, Matthew Knepley wrote: > > > >>>> > > > >>>> On Tue, Jul 28, 2020 at 10:45 AM Eda Oktay wrote: > > > >>>>> > > > >>>>> I am trying to download Petsc with slepc,openblas,mpich,meti,parmetis > > > >>>>> and chaco to a linux computer (probably > > > >>>>> ubuntu, I don't know because I am using it via ssh). > > > >>>>> > > > >>>>> Due to an error, for metis, I needed to download cmake, so I added > > > >>>>> this option, too. However, I got an error telling me that "Error > > > >>>>> running configure on cmake". I attached configure.log. > > > >>>>> > > > >>>>> What should I do? Where is the problem and how can ? fix it? > > > >>>>> > > > >>>>> Thank you so much for answering, > > > >>>> > > > >>>> > > > >>>> It looks like you do not have 'make' in your path, and CMake is feaking out: > > > >>>> > > > >>>> CMAKE was just downloaded, forcing a rebuild because cannot determine if package has changed > > > >>>> =============================================================================== > > > >>>> Running configure on CMAKE; this may take several minutes > > > >>>> =============================================================================== > > > >>>> > > > >>>> Running Executable WITHOUT threads to time it out > > > >>>> Executing: ./configure --prefix=/home/eda/petsc-3.13.2/arch-linux-c-debug --parallel=40 > > > >>>> stdout: > > > >>>> --------------------------------------------- > > > >>>> CMake 3.15.6, Copyright 2000-2019 Kitware, Inc. and Contributors > > > >>>> Found GNU toolchain > > > >>>> C compiler on this system is: gcc > > > >>>> C++ compiler on this system is: g++ > > > >>>> --------------------------------------------- > > > >>>> Error when bootstrapping CMake: > > > >>>> Cannot find appropriate Makefile processor on this system. > > > >>>> Please specify one using environment variable MAKE. > > > >>>> --------------------------------------------- > > > >>>> Log of errors: /home/eda/petsc-3.13.2/arch-linux-c-debug/externalpackages/cmake-3.15.6/Bootstrap.cmk/cmake_bootstrap.log > > > >>>> --------------------------------------------- > > > >>>> > > > >>>> Thanks, > > > >>>> > > > >>>> Matt > > > >>>> > > > >>>>> > > > >>>>> Eda > > > >>>> > > > >>>> > > > >>>> > > > >>>> -- > > > >>>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. > > > >>>> -- Norbert Wiener > > > >>>> > > > >>>> https://www.cse.buffalo.edu/~knepley/ > > > >>>> > > > >>>> > > > >> > > > > > > > > > > -------------- next part -------------- commit 36cb5b562a18a3d476298b4d9097b3109fa4691c Author: Barry Smith Date: Tue Jul 28 12:51:38 2020 -0500 cmake: provide make via PATH to cmake build. [so that make from --download-make is used] Note: providing MAKE env variable gives build error CMake Error: CMake was unable to find a build program corresponding to "Unix Makefiles". CMAKE_MAKE_PROGRAM is not set. Commit-type: error-checking, bug-fix, portability-fix, configure /spend 10m Reported-by: Eda Oktay diff --git a/config/BuildSystem/config/packages/cmake.py b/config/BuildSystem/config/packages/cmake.py index 75d6078bb3..bd5bc9ed32 100644 --- a/config/BuildSystem/config/packages/cmake.py +++ b/config/BuildSystem/config/packages/cmake.py @@ -39,6 +39,16 @@ class Configure(config.package.GNUPackage): args.append(self.argDB['download-cmake-configure-options']) return args + def Install(self): + import sys + save_path = os.environ['PATH'] + make_loc = os.path.dirname(self.make.make) + self.log.write('CMAKE build - adding MAKE location to PATH: '+make_loc+'\n') + os.environ['PATH'] += ':'+make_loc + retdir = config.package.GNUPackage.Install(self) + os.environ['PATH'] = save_path + return retdir + def locateCMake(self): if 'with-cmake-exec' in self.argDB: self.log.write('Looking for specified CMake executable '+self.argDB['with-cmake-exec']+'\n') From rtmills at anl.gov Wed Jul 29 17:51:29 2020 From: rtmills at anl.gov (Mills, Richard Tran) Date: Wed, 29 Jul 2020 22:51:29 +0000 Subject: [petsc-users] Argonne National Laboratory hiring for staff position in Numerical PDEs and Scientific Computing Message-ID: <21956ae3-9e70-a8ea-db90-7d82f00da149@anl.gov> Dear PETSc Users and Developers, The Laboratory for Applied Mathematics, Numerical Software, and Statistics (LANS, https://www.anl.gov/mcs/lans) in the Mathematics and Computer Science Division at Argonne National Laboratory -- which has served as the "home" for PETSc development for over two decades -- is seeking a research staff member in the area of partial differential equations (PDEs), numerical software, and scientific computing. Because this position could provide an excellent opportunity for someone interested in pursuing a research program furthering the development and applications of PETSc, I want to make members of the PETSc user and developer community aware of it. Applicants at junior or senior career levels will be considered. For further details and to apply, please visit the job listings posted at Senior applicants: https://bit.ly/2Otr7De Earlier-career applicants: https://bit.ly/2B1Rxch Best regards, Richard -------------- next part -------------- An HTML attachment was scrubbed... URL: From lbllm2018 at hotmail.com Wed Jul 29 20:35:39 2020 From: lbllm2018 at hotmail.com (Bin Liu) Date: Thu, 30 Jul 2020 01:35:39 +0000 Subject: [petsc-users] solving saddle point problem in distributed memory In-Reply-To: References: <74D5C00B-69F3-4F9B-BD2F-CAA94EEAAD83@gmail.com> <13D4A6F0-137D-435F-882B-042F85990C8C@petsc.dev> <5654AFFA-500D-44F7-B793-F560C6311DEF@petsc.dev> Message-ID: Yes, I did not wait until it aborts. I will try a small problem. From: Matthew Knepley [mailto:knepley at gmail.com] Sent: Wednesday, 29 July 2020 7:54 PM To: Bin Liu Cc: Barry Smith ; petsc-users at mcs.anl.gov Subject: Re: [petsc-users] solving saddle point problem in distributed memory On Wed, Jul 29, 2020 at 6:30 AM Bin Liu > wrote: Thanks for your suggestions. Sure. I will check my codes of MatZeroRows from a small problem. The program does not terminate during solving in distributed memory, but repeating this error in run time. In fact, this is the entire error message repeating itself during runtime. Are you not checking the return codes for PETSc calls in your code? Matt From: Matthew Knepley [mailto:knepley at gmail.com] Sent: Wednesday, 29 July 2020 5:21 PM To: Bin Liu > Cc: Barry Smith >; petsc-users at mcs.anl.gov Subject: Re: [petsc-users] solving saddle point problem in distributed memory On Tue, Jul 28, 2020 at 11:33 PM Bin Liu > wrote: Dear Barry, Thanks for your suggestions. I just tested it. I still get the same error. I attached it below: [1]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [1]PETSC ERROR: Object is in wrong state [1]PETSC ERROR: Matrix is missing diagonal entry in the zeroed row 24214 [1]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [1]PETSC ERROR: Petsc Release Version 3.9.1, Apr, 29, 2018 [1]PETSC ERROR: ./re100 on a named kdp-MS-7850 by kdp Wed Jul 29 11:26:57 2020 [1]PETSC ERROR: Configure options --prefix=/home/kdp/petsc --debugging=0 --with-mpi-dir=/home/kdp/mpich --download-fblaslapack --download-metis --download-parmetis --download-hypre=/home/kdp/hypre-2.14.0.tar.gz --download-superlu_dist [1]PETSC ERROR: #1 MatZeroRows_SeqAIJ() line 1819 in /home/kdp/petsc-3.9.1/src/mat/impls/aij/seq/aij.c [1]PETSC ERROR: #2 MatZeroRows() line 5710 in /home/kdp/petsc-3.9.1/src/mat/interface/matrix.c [1]PETSC ERROR: #3 MatZeroRows_MPIAIJ() line 797 in /home/kdp/petsc-3.9.1/src/mat/impls/aij/mpi/mpiaij.c [1]PETSC ERROR: #4 MatZeroRows() line 5710 in /home/kdp/petsc-3.9.1/src/mat/interface/matrix.c It still report missing diagonal entry. However I have epxlicitly assign 0.0 to the diagonal element of the system matrix and solve it in sequential mode without error. Where could possibly my wrong set up be? 1) Please send the complete error message. The stack trace here is cut off. 2) It does not look like it is failing in the solver, but in the call to MatZeroRows(), which as it says, requires diagonal elements 3) It looks like you are not assembling the matrix correctly in parallel I recommend making as small a problem as possible, running on 1 and 2 processes, and printing out the matrix using -mat_view to compare them entry-by-entry. Thanks, Matt Best ________________________________ From: Barry Smith > Sent: Wednesday, July 29, 2020 10:49 AM To: Bin Liu > Cc: petsc-users at mcs.anl.gov > Subject: Re: [petsc-users] solving saddle point problem in distributed memory No, PETSc direct solvers do not work in parallel. Only SuperLU_DIST, MUMPS, and PasTix. -pc_type lu (or cholesky depending on the package) -pc_factor_mat_solver_type superlu_dist or mumps or pastix to run the direct solver in parallel. On Jul 28, 2020, at 9:01 PM, Bin Liu > wrote: Dear Barry, Thanks for your explanation. Does it mean the default director solver LU in PETSc can also run in the distributed memory mode? I only used iterative solvers with preconditioners in distributed memory before. My experience in using director solver in parallel is indeed limited, especially the saddle point problem in parallel. I have done some search online, but I did not find a working set up at this moment. Could you give a sample setup of the direct solver for parallel run? It is really appreciated. From: Barry Smith [mailto:bsmith at petsc.dev] Sent: Wednesday, 29 July 2020 9:48 AM To: Bin Liu > Cc: petsc-users at mcs.anl.gov Subject: Re: [petsc-users] solving saddle point problem in distributed memory SuperLU_DIST won't "magically" run on saddle point problems. It only does limited pivoting, realistically a parallel LU cannot always do complete pivoting or it becomes a sequential algorithm. For parallel you need to use PCFIELDSPLIT, for sequential you can use SuperLU (not SuperLU_DIST) since it can do more pivoting, being a sequential algorithm. Barry On Jul 28, 2020, at 7:59 PM, Bin Liu > wrote: Thanks for your tutorials. Yes. I tried PCFIELDSPLIT. However, it only works for sequential runs. When I run the code in distributed memory, it reports errors. In fact, the essence of my wonder is (a) how to set up superlu_dist in petsc for solving saddle point problem in distributed memory? (b) does the direct solvers in petsc can run in distributed memory for solving saddle point problem? From: Stefano Zampini [mailto:stefano.zampini at gmail.com] Sent: Tuesday, 28 July 2020 6:55 PM To: Bin Liu > Cc: petsc-users at mcs.anl.gov Subject: Re: [petsc-users] solving saddle point problem in distributed memory If you want advice you should post the error trace PETSc reports. Anyway, solving Stokes is not so trivial (without direct solvers, you may need mesh dependent information), but we have examples for it https://gitlab.com/petsc/petsc/-/blob/master/src/ksp/ksp/tutorials/ex42.c https://gitlab.com/petsc/petsc/-/blob/master/src/ksp/ksp/tutorials/ex43.c https://gitlab.com/petsc/petsc/-/blob/master/src/snes/tutorials/ex69.c If you scroll to the end of those files, you see a bunch of possible options either using PCFIELDSPLIT, PCBDDC or KSPFETIDP. On Jul 28, 2020, at 12:37 PM, Bin Liu > wrote: I would like to solve a saddle point problem arising from the stokes equation. I got successful to use the direct solvers in sequential runs. However, I would like to extend it for distributed memory computation. I tried to use superlu_dist, but the program returns errors. Is it possible to solve a saddle point problem in distributed memory using superlu_dist? Could anyone give a simple sample code to set up the parameters of the solver? Thanks -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From eda.oktay at metu.edu.tr Thu Jul 30 04:05:39 2020 From: eda.oktay at metu.edu.tr (Eda Oktay) Date: Thu, 30 Jul 2020 12:05:39 +0300 Subject: [petsc-users] Configuration error during OpenBLAS Message-ID: Hi all, I am trying to configure petsc-3.13.2 to ubuntu with the following line: ./configure --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --download-mpich --download-openblas --download-metis --download-parmetis --download-chaco --download-slepc=/home/eda/slepc-v3.13.2.tar.gz --with-X=1 However, I am getting the following error message: ******************************************************************************* UNABLE to CONFIGURE with GIVEN OPTIONS (see configure.log for details): ------------------------------------------------------------------------------- Error during download/extract/detection of OPENBLAS: Could not locate downloaded package OPENBLAS in /home/eda/petsc-3.13.2/arch-linux-c-debug/externalpackages ****************************************************************************** I attached the configure.log, too. Should I download OpenBLAS separately and then use --download-opeblas=/home/.....? If so, which version should I use for petsc-3.13.2? Thanks! Eda -------------- next part -------------- A non-text attachment was scrubbed... Name: configure.log Type: text/x-log Size: 2322288 bytes Desc: not available URL: From knepley at gmail.com Thu Jul 30 07:17:17 2020 From: knepley at gmail.com (Matthew Knepley) Date: Thu, 30 Jul 2020 08:17:17 -0400 Subject: [petsc-users] Configuration error during OpenBLAS In-Reply-To: References: Message-ID: Satish, Shouldn't we be checking in a case insensitive way, or do we want case sensitivity here? Looking for OPENBLAS at git.openblas, hg.openblas or a directory starting with ['openblas'] Could not locate an existing copy of OPENBLAS: ['Chaco-2.2-p2', 'slepc-v3.13.2', 'petsc-pkg-metis-ca7a59e6283f', 'OpenBLAS-e7c4d6705a41910240dd19b9e7082a422563bf15', 'petsc-pkg-parmetis-475d8facbb32', 'mpich-3.3.2'] Thanks, Matt On Thu, Jul 30, 2020 at 5:07 AM Eda Oktay wrote: > Hi all, > > I am trying to configure petsc-3.13.2 to ubuntu with the following line: > > ./configure --with-cc=gcc --with-cxx=g++ --with-fc=gfortran > --download-mpich --download-openblas --download-metis > --download-parmetis --download-chaco > --download-slepc=/home/eda/slepc-v3.13.2.tar.gz --with-X=1 > > However, I am getting the following error message: > > > > ******************************************************************************* > UNABLE to CONFIGURE with GIVEN OPTIONS (see configure.log > for details): > > ------------------------------------------------------------------------------- > Error during download/extract/detection of OPENBLAS: > Could not locate downloaded package OPENBLAS in > /home/eda/petsc-3.13.2/arch-linux-c-debug/externalpackages > > ****************************************************************************** > > I attached the configure.log, too. > > Should I download OpenBLAS separately and then use > --download-opeblas=/home/.....? > > If so, which version should I use for petsc-3.13.2? > > Thanks! > > Eda > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From eijkhout at tacc.utexas.edu Thu Jul 30 09:15:39 2020 From: eijkhout at tacc.utexas.edu (Victor Eijkhout) Date: Thu, 30 Jul 2020 14:15:39 +0000 Subject: [petsc-users] Configuration error during OpenBLAS In-Reply-To: References: Message-ID: <16D509FA-1FA7-4644-AD5E-9AA55631F862@tacc.utexas.edu> On , 2020Jul30, at 07:17, Matthew Knepley > wrote: Shouldn't we be Be recommending Blis instead of OpenBlas? Victor. -------------- next part -------------- An HTML attachment was scrubbed... URL: From balay at mcs.anl.gov Thu Jul 30 10:13:27 2020 From: balay at mcs.anl.gov (Satish Balay) Date: Thu, 30 Jul 2020 10:13:27 -0500 (CDT) Subject: [petsc-users] Configuration error during OpenBLAS In-Reply-To: References: Message-ID: Normally - when the name is picked up from PACKAGENAME.py - and it can be overwritten by - for ex: diff --git a/config/BuildSystem/config/packages/openblas.py b/config/BuildSystem/config/packages/openblas.py index 97fd5da67e..2f65b94fde 100644 --- a/config/BuildSystem/config/packages/openblas.py +++ b/config/BuildSystem/config/packages/openblas.py @@ -15,6 +15,7 @@ class Configure(config.package.Package): self.gitcommit = 'e7c4d6705a41910240dd19b9e7082a422563bf15' self.versionname = 'OPENBLAS_VERSION' self.download = ['git://https://github.com/xianyi/OpenBLAS.git','https://github.com/xianyi/OpenBLAS/archive/'+self.gitcommit+'.tar.gz'] + self.downloaddirnames = ['OpenBLAS','openblas'] self.optionalincludes = ['openblas_config.h'] self.functions = ['openblas_get_config'] self.liblist = [['libopenblas.a']] And fixing this to do the check in a 'case insensitive way' should be fine. diff --git a/config/BuildSystem/config/package.py b/config/BuildSystem/config/package.py index 130ef1db3e..5066f53e42 100644 --- a/config/BuildSystem/config/package.py +++ b/config/BuildSystem/config/package.py @@ -638,7 +638,7 @@ class Package(config.base.Configure): def matchExcludeDir(self,dir): '''Check is the dir matches something in the excluded directory list''' for exdir in self.excludedDirs: - if dir.startswith(exdir): + if dir.lower().startswith(exdir.lower()): return 1 return 0 @@ -718,7 +718,7 @@ If its a remote branch, use: origin/'+self.gitcommit+' for commit.') Dir.append(hgpkg) for d in pkgdirs: for j in self.downloaddirnames: - if d.startswith(j) and os.path.isdir(os.path.join(packages, d)) and not self.matchExcludeDir(d): + if d.lower().startswith(j.lower()) and os.path.isdir(os.path.join(packages, d)) and not self.matchExcludeDir(d): Dir.append(d) if len(Dir) > 1: Ok - the MR is at https://gitlab.com/petsc/petsc/-/merge_requests/3011 Satish On Thu, 30 Jul 2020, Matthew Knepley wrote: > Satish, > > Shouldn't we be checking in a case insensitive way, or do we want case > sensitivity here? > > Looking for OPENBLAS at git.openblas, hg.openblas or > a directory starting with ['openblas'] > Could not locate an existing copy of OPENBLAS: > ['Chaco-2.2-p2', 'slepc-v3.13.2', > 'petsc-pkg-metis-ca7a59e6283f', > 'OpenBLAS-e7c4d6705a41910240dd19b9e7082a422563bf15', > 'petsc-pkg-parmetis-475d8facbb32', 'mpich-3.3.2'] > > Thanks, > > Matt > > On Thu, Jul 30, 2020 at 5:07 AM Eda Oktay wrote: > > > Hi all, > > > > I am trying to configure petsc-3.13.2 to ubuntu with the following line: > > > > ./configure --with-cc=gcc --with-cxx=g++ --with-fc=gfortran > > --download-mpich --download-openblas --download-metis > > --download-parmetis --download-chaco > > --download-slepc=/home/eda/slepc-v3.13.2.tar.gz --with-X=1 > > > > However, I am getting the following error message: > > > > > > > > ******************************************************************************* > > UNABLE to CONFIGURE with GIVEN OPTIONS (see configure.log > > for details): > > > > ------------------------------------------------------------------------------- > > Error during download/extract/detection of OPENBLAS: > > Could not locate downloaded package OPENBLAS in > > /home/eda/petsc-3.13.2/arch-linux-c-debug/externalpackages > > > > ****************************************************************************** > > > > I attached the configure.log, too. > > > > Should I download OpenBLAS separately and then use > > --download-opeblas=/home/.....? > > > > If so, which version should I use for petsc-3.13.2? > > > > Thanks! > > > > Eda > > > > > From balay at mcs.anl.gov Thu Jul 30 10:21:41 2020 From: balay at mcs.anl.gov (Satish Balay) Date: Thu, 30 Jul 2020 10:21:41 -0500 (CDT) Subject: [petsc-users] Configuration error during OpenBLAS In-Reply-To: <16D509FA-1FA7-4644-AD5E-9AA55631F862@tacc.utexas.edu> References: <16D509FA-1FA7-4644-AD5E-9AA55631F862@tacc.utexas.edu> Message-ID: Generally we recommend using system blas/lapack Wrt: Blis - the usage is slightly complicated. It needs additional lapack So I think - if using blis - the following 2 modes might work --download-blis --download-f2cblaslapack --download-blis [system lapack] Satish On Thu, 30 Jul 2020, Victor Eijkhout wrote: > > > On , 2020Jul30, at 07:17, Matthew Knepley > wrote: > > Shouldn't we be > > Be recommending Blis instead of OpenBlas? > > Victor. > > From eda.oktay at metu.edu.tr Thu Jul 30 10:24:47 2020 From: eda.oktay at metu.edu.tr (Eda Oktay) Date: Thu, 30 Jul 2020 18:24:47 +0300 Subject: [petsc-users] Configuration error during OpenBLAS In-Reply-To: References: <16D509FA-1FA7-4644-AD5E-9AA55631F862@tacc.utexas.edu> Message-ID: Dear Satish, I am sorry but I couldn't understand what should I do. Am I getting this error because of case insensitivity? If I am, why now? I mean I configured Petsc with exact same options before in 2 machines. What can be the reason? Thanks! Eda On Thu, Jul 30, 2020, 6:21 PM Satish Balay via petsc-users < petsc-users at mcs.anl.gov> wrote: > Generally we recommend using system blas/lapack > > Wrt: Blis - the usage is slightly complicated. It needs additional lapack > > So I think - if using blis - the following 2 modes might work > > --download-blis --download-f2cblaslapack > --download-blis [system lapack] > > Satish > > On Thu, 30 Jul 2020, Victor Eijkhout wrote: > > > > > > > On , 2020Jul30, at 07:17, Matthew Knepley knepley at gmail.com>> wrote: > > > > Shouldn't we be > > > > Be recommending Blis instead of OpenBlas? > > > > Victor. > > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From balay at mcs.anl.gov Thu Jul 30 10:32:37 2020 From: balay at mcs.anl.gov (Satish Balay) Date: Thu, 30 Jul 2020 10:32:37 -0500 (CDT) Subject: [petsc-users] Configuration error during OpenBLAS In-Reply-To: References: <16D509FA-1FA7-4644-AD5E-9AA55631F862@tacc.utexas.edu> Message-ID: On Thu, 30 Jul 2020, Eda Oktay wrote: > Dear Satish, > > I am sorry but I couldn't understand what should I do. Am I getting this > error because of case insensitivity? Its a bug in configure. I sent the fix [diff - 2 different fixes, you can try any one of them] in a different reply. > > If I am, why now? I mean I configured Petsc with exact same options before > in 2 machines. What can be the reason? Likely you have git installed on the other 2 machines - so this issue didn't come up. Without git - a tarball gets used, this issue is with tarball usage. [or if don't have git on these machines as well - they likely have case insensitive file system - aka MacOS] Satish From alexlindsay239 at gmail.com Thu Jul 30 11:14:58 2020 From: alexlindsay239 at gmail.com (Alexander Lindsay) Date: Thu, 30 Jul 2020 09:14:58 -0700 Subject: [petsc-users] Only print converged reason when not converged In-Reply-To: References: Message-ID: I like that a lot. I'll try to spin-up a PR (MR on gitlab?) today. On Tue, Jul 28, 2020 at 6:42 PM Barry Smith wrote: > > Yes, this is a good idea. Simple extension of current code. > > The command line would then be > > -snes_converged_reason ::failed > > add in petscviewer.h > > a new PetscViewerFormat PETSC_VIEWER_FAILED, > > add in viewera.c PetscViewerFormats FAILED. > > add in SNESReasonView > > if (snes->reason > 0 && format != PETSC_VIEWER_FAILED) { > > > Document it in SNESReasonView manual page > > > On Jul 28, 2020, at 7:50 PM, Matthew Knepley wrote: > > On Tue, Jul 28, 2020 at 8:09 PM Alexander Lindsay < > alexlindsay239 at gmail.com> wrote: > >> The only slight annoyance with doing this through a PostSolve hook as >> opposed to a pluggable ReasonView system is that if a user passed >> `-snes_converged_reason` on the command line, we would get redundant >> printout from SNESSolve and our PostSolve. Although I suppose we could >> solve this by intercepting their option, not passing it to PETSc, and then >> just using that info to control the output from our PostSolve (e.g. if a >> user passes -snes_converged_reason from the command line, then we should >> always print the reason, instead of just printing when we don't converge). >> So maybe the PostSolve will work fine. >> > > Can't you just define another ASCII format that prints what you want for > ReasonView? > > Matt > > >> On Tue, Jul 28, 2020 at 12:20 PM Barry Smith wrote: >> >>> >>> KPSSetPostSolve(), There is no SNESSetPostSolve() but there >>> could/should be. >>> >>> Barry >>> >>> Note inside the SNESSolve there are a bunch of pre and post hooks, >>> don't get the confused with an outside SNESSetPostSolve() >>> >>> On Jul 28, 2020, at 1:12 PM, Fande Kong wrote: >>> >>> One alternative is to support a plugable KSP/SNESReasonView system. We >>> then could hook up KSP/SNESReasonView_MOOSE. >>> >>> We could call our views from SNES/KSP"done"Solve as well if such a >>> system is not affordable. What are the final functions we should call, >>> where we guarantee SNES/KSP is already done? >>> >>> Thanks, >>> >>> Fande, >>> >>> On Tue, Jul 28, 2020 at 12:02 PM Barry Smith wrote: >>> >>>> >>>> Alex, >>>> >>>> The actual printing is done with SNESReasonView() and >>>> KSPReasonView() I would suggest copying those files to Moose with a name >>>> change and removing all the code you don't want. Then you can call your >>>> versions immediately after SNESSolve() and KSPSolve(). >>>> >>>> Barry >>>> >>>> >>>> > On Jul 28, 2020, at 10:43 AM, Alexander Lindsay < >>>> alexlindsay239 at gmail.com> wrote: >>>> > >>>> > To help debug the many emails we get about solves that fail to >>>> converge, in MOOSE we recently appended `-snes_converged_reason >>>> -ksp_converged_reason` for every call to `SNESSolve`. Of course, now we >>>> have users complaining about the new text printed to their screens that >>>> they didn't have before. Some of them have made a reasonable request to >>>> only print the convergence reason when the solve has actually failed to >>>> converge. Is there some way we can only print the reason if we've diverged, >>>> e.g. if reason < 0 ? >>>> > >>>> > Alex >>>> >>>> >>> > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From salazardetro1 at llnl.gov Thu Jul 30 18:52:45 2020 From: salazardetro1 at llnl.gov (Salazar De Troya, Miguel) Date: Thu, 30 Jul 2020 23:52:45 +0000 Subject: [petsc-users] Accessing the adjoint in TSSetRHSJacobianP Message-ID: Hi, The jacobian of the RHS w.r.t. the parameter P is a matrix that will be multiplied by the adjoint solution within TSAdjoint(). I am interested in having access to that adjoint so I can calculate shape derivatives with firedrake_adjoint. I will have to use that adjoint function, apply it to the residual and calculate the shape derivative from the resulting 0-form to obtain my RHSJacobianP*adjoint. This is the workaround done in pyadjoint: https://github.com/dolfin-adjoint/pyadjoint/blob/e6fbd13f8ac63290804852fa2f27307c44bdc9e3/dolfin_adjoint_common/blocks/solving.py#L204 Is there another alternative within PETSc? Maybe using a matrix free approach? Thanks Miguel Miguel A. Salazar de Troya Postdoctoral Researcher, Lawrence Livermore National Laboratory B141 Rm: 1085-5 Ph: 1(925) 422-6411 -------------- next part -------------- An HTML attachment was scrubbed... URL: From eda.oktay at metu.edu.tr Fri Jul 31 02:23:01 2020 From: eda.oktay at metu.edu.tr (Eda Oktay) Date: Fri, 31 Jul 2020 10:23:01 +0300 Subject: [petsc-users] PETSc Make error Message-ID: Hi all, I configured petsc-3.13.2 with following options: ./configure --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --download-mpich --download-openblas --download-metis --download-parmetis --download-chaco --download-slepc --with-X=1 However, I got a make error. I attached make.log and configure.log, how can I fix that problem? Thanks! Eda -------------- next part -------------- A non-text attachment was scrubbed... Name: make.log Type: application/octet-stream Size: 13380 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: configure.log Type: application/octet-stream Size: 1365485 bytes Desc: not available URL: From eda.oktay at metu.edu.tr Fri Jul 31 02:23:49 2020 From: eda.oktay at metu.edu.tr (Eda Oktay) Date: Fri, 31 Jul 2020 10:23:49 +0300 Subject: [petsc-users] Configuration error during OpenBLAS In-Reply-To: References: <16D509FA-1FA7-4644-AD5E-9AA55631F862@tacc.utexas.edu> Message-ID: Dear Satish, Thank you so much for your replies. I installed git and I was able to configure PETSc. Thanks again! Eda Satish Balay , 30 Tem 2020 Per, 18:32 tarihinde ?unu yazd?: > > On Thu, 30 Jul 2020, Eda Oktay wrote: > > > Dear Satish, > > > > I am sorry but I couldn't understand what should I do. Am I getting this > > error because of case insensitivity? > > Its a bug in configure. I sent the fix [diff - 2 different fixes, you can try any one of them] in a different reply. > > > > > If I am, why now? I mean I configured Petsc with exact same options before > > in 2 machines. What can be the reason? > > Likely you have git installed on the other 2 machines - so this issue didn't come up. Without git - a tarball gets used, this issue is with tarball usage. > > [or if don't have git on these machines as well - they likely have case insensitive file system - aka MacOS] > > Satish From eda.oktay at metu.edu.tr Fri Jul 31 02:27:28 2020 From: eda.oktay at metu.edu.tr (Eda Oktay) Date: Fri, 31 Jul 2020 10:27:28 +0300 Subject: [petsc-users] Petsc configure error (cmake) In-Reply-To: References: <50206B2B-19EC-479F-A4C3-0959B4B180E2@petsc.dev> Message-ID: Dear all, I made administrator of the server download cmake and configured it again and it was successful. Thank you so much for your replies! Eda Satish Balay , 29 Tem 2020 ?ar, 20:20 tarihinde ?unu yazd?: > > This is likely a cmake install bug. I submitted a report. > > https://gitlab.kitware.com/cmake/cmake/-/issues/21035 > > The workaround is to have make in PATH. Attaching this patch. > > And updating barry/2020-07-28/cmake-use-make-if-given/maint and https://gitlab.com/petsc/petsc/-/merge_requests/3004 with this change. > > Satish > > On Wed, 29 Jul 2020, Satish Balay via petsc-users wrote: > > > I can reproduce this - by uninstalling system make. > > > > Will check. > > > > Satish > > > > On Wed, 29 Jul 2020, Eda Oktay wrote: > > > > > Dear Barry, > > > > > > I am afraid I am still getting the same error. > > > > > > Thanks, > > > > > > Eda > > > > > > Barry Smith , 29 Tem 2020 ?ar, 18:51 tarihinde ?unu yazd?: > > > > > > > > > > > > Please try > > > > > > > > rm -rf /home/eda/petsc-3.13.2/arch-linux-c-debug > > > > > > > > and then run configure again. > > > > > > > > It lists the correct make to use > > > > > > > > Makefile processor on this system is: /home/eda/petsc-3.13.2/arch-linux-c-debug/bin/make > > > > > > > > So I am hoping the problem is from the crazy cmake caching nonsense and cleaning out the old files will resolve the problem. > > > > > > > > Barry > > > > > > > > > > > > > On Jul 29, 2020, at 5:51 AM, Eda Oktay wrote: > > > > > > > > > > Dear Barry, > > > > > > > > > > I wrote the followings: > > > > > > > > > > $ patch -p1 < cmakemake.patch > > > > > > > > > > And it gave the following line: > > > > > > > > > > (Stripping trailing CRs from patch; use --binary to disable.) > > > > > patching file config/BuildSystem/config/packages/cmake.py > > > > > > > > > > Then I configured as follows: > > > > > > > > > > ./configure --with-cc=gcc --with-cxx=g++ --with-fc=gfortran > > > > > --download-make --download-mpich --download-openblas --download-metis > > > > > --download-parmetis --download-chaco > > > > > --download-slepc=/home/eda/slepc-v3.13.2.tar.gz --download-cmake > > > > > --with-X=1 > > > > > > > > > > But I am still getting the following error: > > > > > > > > > > ******************************************************************************* > > > > > UNABLE to CONFIGURE with GIVEN OPTIONS (see configure.log > > > > > for details): > > > > > ------------------------------------------------------------------------------- > > > > > Error running configure on CMAKE > > > > > ******************************************************************************* > > > > > > > > > > Am I doing something wrong? > > > > > > > > > > Thanks! > > > > > > > > > > Eda > > > > > > > > > > Barry Smith , 29 Tem 2020 ?ar, 00:10 tarihinde ?unu yazd?: > > > > >> > > > > >> > > > > >> It's better to use git than download tarballs. With tarballs it is not possible to access recent fixes or try branches. > > > > >> > > > > >> Here is what you can do, save the attachment as cmakemake.patch and do > > > > >> > > > > >> patch -p1 < cmakemake.patch > > > > >> > > > > >> in the PETSc directory > > > > >> > > > > >> Then run configure as before > > > > >> > > > > >> Barry > > > > >> > > > > >> > > > > >> > > > > >>> On Jul 28, 2020, at 3:28 PM, Eda Oktay wrote: > > > > >>> > > > > >>> Dear Barry, > > > > >>> > > > > >>> Thank you so much for answering. So, I downloaded the tar.gz file and > > > > >>> now, should I configure this file with my previous options? I am > > > > >>> using the following line: > > > > >>> > > > > >>> ./configure --with-cc=gcc --with-cxx=g++ --with-fc=gfortran > > > > >>> --download-make --download-mpich --download-openblas --download-metis > > > > >>> --download-parmetis --download-chaco > > > > >>> --download-slepc=/home/eda/slepc-v3.13.2.tar.gz --download-cmake > > > > >>> --with-X=1 > > > > >>> > > > > >>> Thanks so much again! > > > > >>> > > > > >>> Eda > > > > >>> > > > > >>> Barry Smith , 28 Tem 2020 Sal, 20:55 tarihinde ?unu yazd?: > > > > >>>> > > > > >>>> > > > > >>>> Eda, > > > > >>>> > > > > >>>> The branch barry/2020-07-28/cmake-use-make-if-given/maint should resolve the problem. It will force the cmake build to use the make obtained with --download-make > > > > >>>> > > > > >>>> Barry > > > > >>>> > > > > >>>> > > > > >>>> On Jul 28, 2020, at 9:48 AM, Matthew Knepley wrote: > > > > >>>> > > > > >>>> On Tue, Jul 28, 2020 at 10:45 AM Eda Oktay wrote: > > > > >>>>> > > > > >>>>> I am trying to download Petsc with slepc,openblas,mpich,meti,parmetis > > > > >>>>> and chaco to a linux computer (probably > > > > >>>>> ubuntu, I don't know because I am using it via ssh). > > > > >>>>> > > > > >>>>> Due to an error, for metis, I needed to download cmake, so I added > > > > >>>>> this option, too. However, I got an error telling me that "Error > > > > >>>>> running configure on cmake". I attached configure.log. > > > > >>>>> > > > > >>>>> What should I do? Where is the problem and how can ? fix it? > > > > >>>>> > > > > >>>>> Thank you so much for answering, > > > > >>>> > > > > >>>> > > > > >>>> It looks like you do not have 'make' in your path, and CMake is feaking out: > > > > >>>> > > > > >>>> CMAKE was just downloaded, forcing a rebuild because cannot determine if package has changed > > > > >>>> =============================================================================== > > > > >>>> Running configure on CMAKE; this may take several minutes > > > > >>>> =============================================================================== > > > > >>>> > > > > >>>> Running Executable WITHOUT threads to time it out > > > > >>>> Executing: ./configure --prefix=/home/eda/petsc-3.13.2/arch-linux-c-debug --parallel=40 > > > > >>>> stdout: > > > > >>>> --------------------------------------------- > > > > >>>> CMake 3.15.6, Copyright 2000-2019 Kitware, Inc. and Contributors > > > > >>>> Found GNU toolchain > > > > >>>> C compiler on this system is: gcc > > > > >>>> C++ compiler on this system is: g++ > > > > >>>> --------------------------------------------- > > > > >>>> Error when bootstrapping CMake: > > > > >>>> Cannot find appropriate Makefile processor on this system. > > > > >>>> Please specify one using environment variable MAKE. > > > > >>>> --------------------------------------------- > > > > >>>> Log of errors: /home/eda/petsc-3.13.2/arch-linux-c-debug/externalpackages/cmake-3.15.6/Bootstrap.cmk/cmake_bootstrap.log > > > > >>>> --------------------------------------------- > > > > >>>> > > > > >>>> Thanks, > > > > >>>> > > > > >>>> Matt > > > > >>>> > > > > >>>>> > > > > >>>>> Eda > > > > >>>> > > > > >>>> > > > > >>>> > > > > >>>> -- > > > > >>>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. > > > > >>>> -- Norbert Wiener > > > > >>>> > > > > >>>> https://www.cse.buffalo.edu/~knepley/ > > > > >>>> > > > > >>>> > > > > >> > > > > > > > > > > > > > > From balay at mcs.anl.gov Fri Jul 31 02:50:03 2020 From: balay at mcs.anl.gov (Satish Balay) Date: Fri, 31 Jul 2020 02:50:03 -0500 (CDT) Subject: [petsc-users] PETSc Make error In-Reply-To: References: Message-ID: >>>>>>>> File "./config/gmakegen.py", line 4, in from distutils.sysconfig import parse_makefile ModuleNotFoundError: No module named 'distutils.sysconfig' <<<<< For one - if you used petsc-3.13.3 - it would have also tried python2 - or given a reasonable error message. If you have python2 - you can try: python2 ./configure ... Or you can ask sysadmin on this machine to install python3-distutils or equivalent package. Satish On Fri, 31 Jul 2020, Eda Oktay wrote: > Hi all, > > I configured petsc-3.13.2 with following options: > > ./configure --with-cc=gcc --with-cxx=g++ --with-fc=gfortran > --download-mpich --download-openblas --download-metis > --download-parmetis --download-chaco --download-slepc --with-X=1 > > However, I got a make error. > > I attached make.log and configure.log, how can I fix that problem? > > Thanks! > > Eda > From eda.oktay at metu.edu.tr Fri Jul 31 03:15:04 2020 From: eda.oktay at metu.edu.tr (Eda Oktay) Date: Fri, 31 Jul 2020 11:15:04 +0300 Subject: [petsc-users] PETSc Make error In-Reply-To: References: Message-ID: Dear Satish, I configured petsc-3.13.2. Is this still the same problem? Thanks! Eda On Fri, Jul 31, 2020, 10:50 AM Satish Balay wrote: > > >>>>>>>> > File "./config/gmakegen.py", line 4, in > from distutils.sysconfig import parse_makefile > ModuleNotFoundError: No module named 'distutils.sysconfig' > <<<<< > > For one - if you used petsc-3.13.3 - it would have also tried python2 - or > given a reasonable error message. > > If you have python2 - you can try: > > python2 ./configure ... > > Or you can ask sysadmin on this machine to install python3-distutils or > equivalent package. > > Satish > > On Fri, 31 Jul 2020, Eda Oktay wrote: > > > Hi all, > > > > I configured petsc-3.13.2 with following options: > > > > ./configure --with-cc=gcc --with-cxx=g++ --with-fc=gfortran > > --download-mpich --download-openblas --download-metis > > --download-parmetis --download-chaco --download-slepc --with-X=1 > > > > However, I got a make error. > > > > I attached make.log and configure.log, how can I fix that problem? > > > > Thanks! > > > > Eda > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Fri Jul 31 05:20:45 2020 From: knepley at gmail.com (Matthew Knepley) Date: Fri, 31 Jul 2020 06:20:45 -0400 Subject: [petsc-users] PETSc Make error In-Reply-To: References: Message-ID: On Fri, Jul 31, 2020 at 4:15 AM Eda Oktay wrote: > Dear Satish, > > I configured petsc-3.13.2. Is this still the same problem? > Yes, you need the distutils package installed. Thanks, Matt > Thanks! > > Eda > > On Fri, Jul 31, 2020, 10:50 AM Satish Balay wrote: > >> >> >>>>>>>> >> File "./config/gmakegen.py", line 4, in >> from distutils.sysconfig import parse_makefile >> ModuleNotFoundError: No module named 'distutils.sysconfig' >> <<<<< >> >> For one - if you used petsc-3.13.3 - it would have also tried python2 - >> or given a reasonable error message. >> >> If you have python2 - you can try: >> >> python2 ./configure ... >> >> Or you can ask sysadmin on this machine to install python3-distutils or >> equivalent package. >> >> Satish >> >> On Fri, 31 Jul 2020, Eda Oktay wrote: >> >> > Hi all, >> > >> > I configured petsc-3.13.2 with following options: >> > >> > ./configure --with-cc=gcc --with-cxx=g++ --with-fc=gfortran >> > --download-mpich --download-openblas --download-metis >> > --download-parmetis --download-chaco --download-slepc --with-X=1 >> > >> > However, I got a make error. >> > >> > I attached make.log and configure.log, how can I fix that problem? >> > >> > Thanks! >> > >> > Eda >> > >> >> -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From pierpaolo.minelli at cnr.it Fri Jul 31 05:46:33 2020 From: pierpaolo.minelli at cnr.it (Pierpaolo Minelli) Date: Fri, 31 Jul 2020 12:46:33 +0200 Subject: [petsc-users] Error on INTEGER SIZE using DMDACreate3d In-Reply-To: <18FA0288-9131-4069-9080-60D6CC034DCD@petsc.dev> References: <88A21A58-CDCD-42C0-9A20-1A6C5CBCDF8B@cnr.it> <18FA0288-9131-4069-9080-60D6CC034DCD@petsc.dev> Message-ID: <28C01AB0-6304-46BD-8DF1-77A18CF05A71@cnr.it> Hi, I want to thank you for all your useful suggestions, but luckily fo me it was non a problem related to Petsc as I wrongly supposed reading this error. There was a problem on GPFS filesystem that CINECA support solved and also a problem related to my code that was writing too much files when using 512 nodes. Once these two problem were solved I was able to continue to use Hypre without any problem. Sorry for the trouble and thanks again for your suggestions. Pierpaolo > Il giorno 22 lug 2020, alle ore 02:51, Barry Smith ha scritto: > > > I assume PIC_3D is your code and you are using OpenMP? > > Are you calling hypre from inside your OpenMP parallelism? From inside PIC_3D? > > The SIGTERM is confusing to me. Are you using signals in any way? Usually a sigterm comes outside a process not process or thread crash. > > I assume for__signal_handl... is a Fortran signal handler > > forrtl: error (78): process killed (SIGTERM) > Image PC Routine Line Source > libHYPRE-2.18.2.s 00002B33CF465D3F for__signal_handl Unknown Unknown > libpthread-2.17.s 00002B33D5BFD370 Unknown Unknown Unknown > libpthread-2.17.s 00002B33D5BF96D3 pthread_cond_wait Unknown Unknown > libiomp5.so 00002B33DBA14E07 Unknown Unknown Unknown > libiomp5.so 00002B33DB98810C Unknown Unknown Unknown > libiomp5.so 00002B33DB990578 Unknown Unknown Unknown > libiomp5.so 00002B33DB9D9659 Unknown Unknown Unknown > libiomp5.so 00002B33DB9D8C39 Unknown Unknown Unknown > libiomp5.so 00002B33DB993BCE __kmpc_fork_call Unknown Unknown > PIC_3D 00000000004071C0 Unknown Unknown Unknown > PIC_3D 0000000000490299 Unknown Unknown Unknown > PIC_3D 0000000000492C17 Unknown Unknown Unknown > PIC_3D 000000000040562E Unknown Unknown Unknown > libc-2.17.so 00002B33DC5BEB35 __libc_start_main Unknown Unknown > PIC_3D 0000000000405539 Unknown Unknown Unknown > > > >> On Jul 21, 2020, at 6:32 AM, Pierpaolo Minelli > wrote: >> >> Hi, >> >> I have asked to compile a Petsc Version updated and with 64bit indices. >> Now I have Version 3.13.3 and these are the configure options used: >> >> #!/bin/python >> if __name__ == '__main__': >> import sys >> import os >> sys.path.insert(0, os.path.abspath('config')) >> import configure >> configure_options = [ >> '--CC=mpiicc', >> '--CXX=mpiicpc', >> '--download-hypre', >> '--download-metis', >> '--download-mumps=yes', >> '--download-parmetis', >> '--download-scalapack', >> '--download-superlu_dist', >> '--known-64-bit-blas-indices', >> '--prefix=/cineca/prod/opt/libraries/petsc/3.13.3_int64/intelmpi--2018--binary', >> '--with-64-bit-indices=1', >> '--with-blaslapack-dir=/cineca/prod/opt/compilers/intel/pe-xe-2018/binary/mkl', >> '--with-cmake-dir=/cineca/prod/opt/tools/cmake/3.12.0/none', >> '--with-debugging=0', >> '--with-fortran-interfaces=1', >> '--with-fortran=1', >> 'FC=mpiifort', >> 'PETSC_ARCH=arch-linux2-c-opt', >> ] >> configure.petsc_configure(configure_options) >> >> Now, I receive an error on hypre: >> >> forrtl: error (78): process killed (SIGTERM) >> Image PC Routine Line Source >> libHYPRE-2.18.2.s 00002B33CF465D3F for__signal_handl Unknown Unknown >> libpthread-2.17.s 00002B33D5BFD370 Unknown Unknown Unknown >> libpthread-2.17.s 00002B33D5BF96D3 pthread_cond_wait Unknown Unknown >> libiomp5.so 00002B33DBA14E07 Unknown Unknown Unknown >> libiomp5.so 00002B33DB98810C Unknown Unknown Unknown >> libiomp5.so 00002B33DB990578 Unknown Unknown Unknown >> libiomp5.so 00002B33DB9D9659 Unknown Unknown Unknown >> libiomp5.so 00002B33DB9D8C39 Unknown Unknown Unknown >> libiomp5.so 00002B33DB993BCE __kmpc_fork_call Unknown Unknown >> PIC_3D 00000000004071C0 Unknown Unknown Unknown >> PIC_3D 0000000000490299 Unknown Unknown Unknown >> PIC_3D 0000000000492C17 Unknown Unknown Unknown >> PIC_3D 000000000040562E Unknown Unknown Unknown >> libc-2.17.so 00002B33DC5BEB35 __libc_start_main Unknown Unknown >> PIC_3D 0000000000405539 Unknown Unknown Unknown >> >> Is it possible that I need to ask also to compile hypre with an option for 64bit indices? >> Is it possible to instruct this inside Petsc configure? >> Alternatively, is it possible to use a different multigrid PC inside PETSc that accept 64bit indices? >> >> Thanks in advance >> >> Pierpaolo >> >> >>> Il giorno 27 mag 2020, alle ore 11:26, Stefano Zampini > ha scritto: >>> >>> You need a version of PETSc compiled with 64bit indices, since the message indicates the number of dofs in this case is larger the INT_MAX >>> 2501?3401?1601 = 13617947501 >>> >>> I also suggest you upgrade to a newer version, 3.8.3 is quite old as the error message reports >>> >>> Il giorno mer 27 mag 2020 alle ore 11:50 Pierpaolo Minelli > ha scritto: >>> Hi, >>> >>> I am trying to solve a Poisson equation on this grid: >>> >>> Nx = 2501 >>> Ny = 3401 >>> Nz = 1601 >>> >>> I received this error: >>> >>> [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- >>> [0]PETSC ERROR: Overflow in integer operation: http://www.mcs.anl.gov/petsc/documentation/faq.html#64-bit-indices >>> [0]PETSC ERROR: Mesh of 2501 by 3401 by 1 (dof) is too large for 32 bit indices >>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. >>> [0]PETSC ERROR: Petsc Release Version 3.8.3, Dec, 09, 2017 >>> [0]PETSC ERROR: /marconi_scratch/userexternal/pminelli/PIC3D/2500_3400_1600/./PIC_3D on a arch-linux2-c-opt named r129c09s02 by pminelli Tu >>> e May 26 20:16:34 2020 >>> [0]PETSC ERROR: Configure options --prefix=/cineca/prod/opt/libraries/petsc/3.8.3/intelmpi--2018--binary CC=mpiicc FC=mpiifort CXX=mpiicpc >>> F77=mpiifort F90=mpiifort --with-debugging=0 --with-blaslapack-dir=/cineca/prod/opt/compilers/intel/pe-xe-2018/binary/mkl --with-fortran=1 >>> --with-fortran-interfaces=1 --with-cmake-dir=/cineca/prod/opt/tools/cmake/3.5.2/none --with-mpi-dir=/cineca/prod/opt/compilers/intel/pe-xe- >>> 2018/binary/impi/2018.4.274 --download-scalapack --download-mumps=yes --download-hypre --download-superlu_dist --download-parmetis --downlo >>> ad-metis >>> [0]PETSC ERROR: #1 DMSetUp_DA_3D() line 218 in /marconi/prod/build/libraries/petsc/3.8.3/intelmpi--2018--binary/BA_WORK/petsc-3.8.3/src/dm/ >>> impls/da/da3.c >>> [0]PETSC ERROR: #2 DMSetUp_DA() line 25 in /marconi/prod/build/libraries/petsc/3.8.3/intelmpi--2018--binary/BA_WORK/petsc-3.8.3/src/dm/impl >>> s/da/dareg.c >>> [0]PETSC ERROR: #3 DMSetUp() line 720 in /marconi/prod/build/libraries/petsc/3.8.3/intelmpi--2018--binary/BA_WORK/petsc-3.8.3/src/dm/interf >>> ace/dm.c >>> forrtl: error (76): Abort trap signal >>> >>> >>> I am on an HPC facility and after I loaded PETSC module, I have seen that it is configured with INTEGER size = 32 >>> >>> I solve my problem with these options and it works perfectly with smaller grids: >>> >>> -dm_mat_type hypre -pc_type hypre -pc_hypre_type boomeramg -pc_hypre_boomeramg_relax_type_all SOR/Jacobi -pc_hypre_boomeramg_coarsen_type PMIS -pc_hypre_boomeramg_interp_type FF1 -ksp_type richardson >>> >>> Is it possible to overcome this if I ask them to install a version with INTEGER SIZE = 64? >>> Alternatively, is it possible to overcome this using intel compiler options? >>> >>> Thanks in advance >>> >>> Pierpaolo Minelli >>> >>> >>> -- >>> Stefano >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From hongzhang at anl.gov Fri Jul 31 09:23:12 2020 From: hongzhang at anl.gov (Zhang, Hong) Date: Fri, 31 Jul 2020 14:23:12 +0000 Subject: [petsc-users] Accessing the adjoint in TSSetRHSJacobianP In-Reply-To: References: Message-ID: <10EA2E87-9CCF-40D7-BC96-61644EDD5D3E@anl.gov> Yes, you can use a matrix-free approach on this matrix just like what you would do on the IJacobian matrix. Only the multTranspose operation needs to be overloaded for the adjoint. Hong On Jul 30, 2020, at 6:52 PM, Salazar De Troya, Miguel via petsc-users > wrote: Hi, The jacobian of the RHS w.r.t. the parameter P is a matrix that will be multiplied by the adjoint solution within TSAdjoint(). I am interested in having access to that adjoint so I can calculate shape derivatives with firedrake_adjoint. I will have to use that adjoint function, apply it to the residual and calculate the shape derivative from the resulting 0-form to obtain my RHSJacobianP*adjoint. This is the workaround done in pyadjoint: https://github.com/dolfin-adjoint/pyadjoint/blob/e6fbd13f8ac63290804852fa2f27307c44bdc9e3/dolfin_adjoint_common/blocks/solving.py#L204 Is there another alternative within PETSc? Maybe using a matrix free approach? Thanks Miguel Miguel A. Salazar de Troya Postdoctoral Researcher, Lawrence Livermore National Laboratory B141 Rm: 1085-5 Ph: 1(925) 422-6411 -------------- next part -------------- An HTML attachment was scrubbed... URL: From eda.oktay at metu.edu.tr Fri Jul 31 10:49:52 2020 From: eda.oktay at metu.edu.tr (Eda Oktay) Date: Fri, 31 Jul 2020 18:49:52 +0300 Subject: [petsc-users] PETSc Make error In-Reply-To: References: Message-ID: Dear Matt, Administrator installed python3-dib-utils but still I am getting the same error message. I attached configure.log and make.log again. Thanks! Eda Matthew Knepley , 31 Tem 2020 Cum, 13:20 tarihinde ?unu yazd?: > > On Fri, Jul 31, 2020 at 4:15 AM Eda Oktay wrote: >> >> Dear Satish, >> >> I configured petsc-3.13.2. Is this still the same problem? > > > Yes, you need the distutils package installed. > > Thanks, > > Matt > >> >> Thanks! >> >> Eda >> >> On Fri, Jul 31, 2020, 10:50 AM Satish Balay wrote: >>> >>> >>> >>>>>>>> >>> File "./config/gmakegen.py", line 4, in >>> from distutils.sysconfig import parse_makefile >>> ModuleNotFoundError: No module named 'distutils.sysconfig' >>> <<<<< >>> >>> For one - if you used petsc-3.13.3 - it would have also tried python2 - or given a reasonable error message. >>> >>> If you have python2 - you can try: >>> >>> python2 ./configure ... >>> >>> Or you can ask sysadmin on this machine to install python3-distutils or equivalent package. >>> >>> Satish >>> >>> On Fri, 31 Jul 2020, Eda Oktay wrote: >>> >>> > Hi all, >>> > >>> > I configured petsc-3.13.2 with following options: >>> > >>> > ./configure --with-cc=gcc --with-cxx=g++ --with-fc=gfortran >>> > --download-mpich --download-openblas --download-metis >>> > --download-parmetis --download-chaco --download-slepc --with-X=1 >>> > >>> > However, I got a make error. >>> > >>> > I attached make.log and configure.log, how can I fix that problem? >>> > >>> > Thanks! >>> > >>> > Eda >>> > >>> > > > -- > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- A non-text attachment was scrubbed... Name: make.log Type: text/x-log Size: 13372 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: configure.log Type: text/x-log Size: 7052622 bytes Desc: not available URL: From balay at mcs.anl.gov Fri Jul 31 11:00:44 2020 From: balay at mcs.anl.gov (Satish Balay) Date: Fri, 31 Jul 2020 11:00:44 -0500 (CDT) Subject: [petsc-users] PETSc Make error In-Reply-To: References: Message-ID: You still get this error message - so its likely not installed. To confirm you [and the admin] can run: python3 -c "import distutils.sysconfig" [and see if you get an error or not] You can also try python2 as I suggested in my earlier e-mail [if distutils is installed for python2] python2 -c "import distutils.sysconfig" python2 ./configure ... Satish On Fri, 31 Jul 2020, Eda Oktay wrote: > Dear Matt, > > Administrator installed python3-dib-utils but still I am getting the > same error message. I attached configure.log and make.log again. > > Thanks! > > Eda > > Matthew Knepley , 31 Tem 2020 Cum, 13:20 tarihinde > ?unu yazd?: > > > > On Fri, Jul 31, 2020 at 4:15 AM Eda Oktay wrote: > >> > >> Dear Satish, > >> > >> I configured petsc-3.13.2. Is this still the same problem? > > > > > > Yes, you need the distutils package installed. > > > > Thanks, > > > > Matt > > > >> > >> Thanks! > >> > >> Eda > >> > >> On Fri, Jul 31, 2020, 10:50 AM Satish Balay wrote: > >>> > >>> > >>> >>>>>>>> > >>> File "./config/gmakegen.py", line 4, in > >>> from distutils.sysconfig import parse_makefile > >>> ModuleNotFoundError: No module named 'distutils.sysconfig' > >>> <<<<< > >>> > >>> For one - if you used petsc-3.13.3 - it would have also tried python2 - or given a reasonable error message. > >>> > >>> If you have python2 - you can try: > >>> > >>> python2 ./configure ... > >>> > >>> Or you can ask sysadmin on this machine to install python3-distutils or equivalent package. > >>> > >>> Satish > >>> > >>> On Fri, 31 Jul 2020, Eda Oktay wrote: > >>> > >>> > Hi all, > >>> > > >>> > I configured petsc-3.13.2 with following options: > >>> > > >>> > ./configure --with-cc=gcc --with-cxx=g++ --with-fc=gfortran > >>> > --download-mpich --download-openblas --download-metis > >>> > --download-parmetis --download-chaco --download-slepc --with-X=1 > >>> > > >>> > However, I got a make error. > >>> > > >>> > I attached make.log and configure.log, how can I fix that problem? > >>> > > >>> > Thanks! > >>> > > >>> > Eda > >>> > > >>> > > > > > > -- > > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. > > -- Norbert Wiener > > > > https://www.cse.buffalo.edu/~knepley/ > From bsmith at petsc.dev Fri Jul 31 14:11:09 2020 From: bsmith at petsc.dev (Barry Smith) Date: Fri, 31 Jul 2020 14:11:09 -0500 Subject: [petsc-users] SUPERLU_DIST in single precision In-Reply-To: <20200727211054.Horde.ZgzkW110frSEh8mw1TA2Zdn@webmail.mpcdf.mpg.de> References: <20200721155807.Horde.5ZYOzrQ7dmwNTQquK_6ebeS@webmail.mpcdf.mpg.de> <20200722150403.Horde.NO015I2GI2E-524J-SXqwui@webmail.mpcdf.mpg.de> <20200727211054.Horde.ZgzkW110frSEh8mw1TA2Zdn@webmail.mpcdf.mpg.de> Message-ID: <85D88902-7FC6-4714-A102-B863F29F2A42@petsc.dev> Felix, Sorry for the delay. Another issue came up with building SuperLU_DIST for this formulation that Sherry has to look at. We haven't forgotten you, Barry duplicate symbol '_FormFullA' in: CMakeFiles/superlu_dist.dir/sreadhb.c.o CMakeFiles/superlu_dist.dir/dreadhb.c.o ld: 1 duplicate symbol for architecture x86_64 clang: error: linker command failed with exit code 1 (use -v to see invocation) gmake[2]: *** [SRC/CMakeFiles/superlu_dist.dir/build.make:2072: SRC/libsuperlu_dist.6.2.0.dylib] Error 1 gmake[1]: *** [CMakeFiles/Makefile2:930: SRC/CMakeFiles/superlu_dist.dir/all] Error 2 gmake: *** [Makefile:164: all] Error 2 > On Jul 27, 2020, at 2:10 PM, flw at rzg.mpg.de wrote: > > Hi Shery, > Yes, ideally we would like to compile PETSc in single precision and simply run a single precision version of SUPERLU_DIST just like e.g. MUMPS. > > Best regards and thanks, > Felix > Zitat von "Xiaoye S. Li" : > >> Barry, >> >> I have a branch 'Mixed-precision' working with single precision FP32. I >> assume Felix wants to use superlu_dist from petsc. How do you want to >> incorporate it in petsc? >> >> https://github.com/xiaoyeli/superlu_dist >> >> PS1: in this version, FP32 only works on CPU. FP64 and complex-FP64 all >> work on GPU. >> >> PS2: currently there is no mixed-precision yet, but it is the branch we are >> adding mix-prec support. Will take a while before merging to master. >> >> Sherry >> >> >> On Wed, Jul 22, 2020 at 6:04 AM wrote: >> >>> Hi Barry, >>> for now I just want to run everything in single on CPUs only with >>> SUPERLU_DIST. Maybe we will also incorporate GPUs in the future, but >>> there are no immediate plans yet. So if you could provide the support, >>> that would be awesome. >>> >>> Best regards, >>> Felix >>> >>> Zitat von Barry Smith : >>> >>> > Felix, >>> > >>> > What are your needs, do you want this for CPUs or for GPUs? Do >>> > you wish to run all your code in single precision or just the >>> > SuperLU_Dist solver while the rest of your code double? >>> > >>> > If you want to run everything on CPUs using single precision >>> > then adding the support is very easy, we can provide that for you >>> > any time. The other cases will require more thought. >>> > >>> > Barry >>> > >>> > >>> >> On Jul 21, 2020, at 8:58 AM, flw at rzg.mpg.de wrote: >>> >> >>> >> Dear PETSc support team, >>> >> some time ago you told me that you are planning on releasing a >>> >> version that supports SUPERLU_DIST in single-precision soon. Can >>> >> you tell me roughly what time frame you had in mind? >>> >> >>> >> Best regards, >>> >> Felix >>> >> >>> >>> >>> >>> > > > From mandhapati.raju at convergecfd.com Fri Jul 31 19:05:39 2020 From: mandhapati.raju at convergecfd.com (Raju Mandhapati) Date: Fri, 31 Jul 2020 19:05:39 -0500 Subject: [petsc-users] using fieldsplit for matrix assembled using aij Message-ID: Hello, I have a matrix assembled in interlaced format. I am not able to follow how to use fieldsplit. Is there a good example to show how to do this. My variable are in this format [ {u1,v1,p1}, {u2,v2,p2}, ....,{un,vn,pn}], where n is the number of grid points and u,v,p are the velocity and pressure components. The matrix is assembled as [ uu11 uv11 up11 uu12 uv12 up12 ...... vu11 vv11 vp11 vu12 vv12 vp12 ....... pu11 pv11 pp11 pu12 pv12 pp12 ...... uu21 uv21 up21 uu22 uv22 up22 ...... vu21 vv21 vp21 vu22 vv22 vp22 ....... pu21 pv21 pp21 pu22 pv22 pp22 ...... . . ] I want to split u and v fields as one block and p field as another block. Any help would be appreciated. Raju. -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at petsc.dev Fri Jul 31 19:21:15 2020 From: bsmith at petsc.dev (Barry Smith) Date: Fri, 31 Jul 2020 19:21:15 -0500 Subject: [petsc-users] using fieldsplit for matrix assembled using aij In-Reply-To: References: Message-ID: <31179BF0-9426-4203-A81D-098F1BE7C5A6@petsc.dev> -pc_fieldsplit_block_size 3 (you may not need this if the code sets the block size to 3). -pc_fieldsplit_0_fields 0,1 -pc_fieldsplit_1_fields 2 > On Jul 31, 2020, at 7:05 PM, Raju Mandhapati via petsc-users wrote: > > Hello, > > I have a matrix assembled in interlaced format. I am not able to follow how to use fieldsplit. Is there a good example to show how to do this. My variable are in this format [ {u1,v1,p1}, {u2,v2,p2}, ....,{un,vn,pn}], where n is the number of grid points and u,v,p are the velocity and pressure components. The matrix is assembled as > > [ uu11 uv11 up11 uu12 uv12 up12 ...... > vu11 vv11 vp11 vu12 vv12 vp12 ....... > pu11 pv11 pp11 pu12 pv12 pp12 ...... > > uu21 uv21 up21 uu22 uv22 up22 ...... > vu21 vv21 vp21 vu22 vv22 vp22 ....... > pu21 pv21 pp21 pu22 pv22 pp22 ...... > > . > . > > ] > > I want to split u and v fields as one block and p field as another block. > > Any help would be appreciated. > > Raju. > > > From mandhapati.raju at convergecfd.com Fri Jul 31 19:31:54 2020 From: mandhapati.raju at convergecfd.com (Raju Mandhapati) Date: Fri, 31 Jul 2020 19:31:54 -0500 Subject: [petsc-users] using fieldsplit for matrix assembled using aij In-Reply-To: <31179BF0-9426-4203-A81D-098F1BE7C5A6@petsc.dev> References: <31179BF0-9426-4203-A81D-098F1BE7C5A6@petsc.dev> Message-ID: Barry, Thank you for the reply. I am a beginner. I was confused with the following statement in the manual " If the matrices are stored as interlaced then PCFieldSplitSetFields() can be called repeatedly to indicate which fields belong to each block. " Does it mean that we need to set each row number which field does it belong to? If I set the block size to 3 and set -pc_fieldsplit_0_fields 0,1 -pc_fieldsplit_1_fields 2 In what format is Petsc expecting for the matrix to be assembled? In other words, how does Petsc know where the entries of A00, A01, A10 and A11 are located? thanks, Raju. On Fri, Jul 31, 2020 at 7:21 PM Barry Smith wrote: > > -pc_fieldsplit_block_size 3 (you may not need this if the code sets the > block size to 3). > > -pc_fieldsplit_0_fields 0,1 > -pc_fieldsplit_1_fields 2 > > > > > > > On Jul 31, 2020, at 7:05 PM, Raju Mandhapati via petsc-users < > petsc-users at mcs.anl.gov> wrote: > > > > Hello, > > > > I have a matrix assembled in interlaced format. I am not able to follow > how to use fieldsplit. Is there a good example to show how to do this. My > variable are in this format [ {u1,v1,p1}, {u2,v2,p2}, ....,{un,vn,pn}], > where n is the number of grid points and u,v,p are the velocity and > pressure components. The matrix is assembled as > > > > [ uu11 uv11 up11 uu12 uv12 up12 ...... > > vu11 vv11 vp11 vu12 vv12 vp12 ....... > > pu11 pv11 pp11 pu12 pv12 pp12 ...... > > > > uu21 uv21 up21 uu22 uv22 up22 ...... > > vu21 vv21 vp21 vu22 vv22 vp22 ....... > > pu21 pv21 pp21 pu22 pv22 pp22 ...... > > > > . > > . > > > > ] > > > > I want to split u and v fields as one block and p field as another block. > > > > Any help would be appreciated. > > > > Raju. > > > > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at petsc.dev Fri Jul 31 20:43:38 2020 From: bsmith at petsc.dev (Barry Smith) Date: Fri, 31 Jul 2020 20:43:38 -0500 Subject: [petsc-users] using fieldsplit for matrix assembled using aij In-Reply-To: References: <31179BF0-9426-4203-A81D-098F1BE7C5A6@petsc.dev> Message-ID: <1C8BC3C4-246A-46E9-B940-70715D18C14E@petsc.dev> > On Jul 31, 2020, at 7:31 PM, Raju Mandhapati wrote: > > Barry, > > Thank you for the reply. I am a beginner. I was confused with the following statement in the manual > > " If the matrices are stored as interlaced then PCFieldSplitSetFields() can be called repeatedly to indicate which fields belong to each block. " > > Does it mean that we need to set each row number which field does it belong to? > > If I set the block size to 3 and set > -pc_fieldsplit_0_fields 0,1 > -pc_fieldsplit_1_fields 2 > > In what format is Petsc expecting for the matrix to be assembled? In other words, how does Petsc know where the entries of A00, A01, A10 and A11 are located? The use of PCFieldSplitSetFields() or the command line options -pc_fieldsplit_0_fields assumes and requires that the variables are stored interlaced. Generally for each interlaced variable you want to make sure that it is included in at least one fieldplit (yes, the same interlaced variable can go into multiple fields but this is uncommon). All you need to define the split in your case is -pc_fieldsplit_0_fields 0,1 -pc_fieldsplit_1_fields 2 Barry > > thanks, > Raju. > > > > > On Fri, Jul 31, 2020 at 7:21 PM Barry Smith > wrote: > > -pc_fieldsplit_block_size 3 (you may not need this if the code sets the block size to 3). > > -pc_fieldsplit_0_fields 0,1 > -pc_fieldsplit_1_fields 2 > > > > > > > On Jul 31, 2020, at 7:05 PM, Raju Mandhapati via petsc-users > wrote: > > > > Hello, > > > > I have a matrix assembled in interlaced format. I am not able to follow how to use fieldsplit. Is there a good example to show how to do this. My variable are in this format [ {u1,v1,p1}, {u2,v2,p2}, ....,{un,vn,pn}], where n is the number of grid points and u,v,p are the velocity and pressure components. The matrix is assembled as > > > > [ uu11 uv11 up11 uu12 uv12 up12 ...... > > vu11 vv11 vp11 vu12 vv12 vp12 ....... > > pu11 pv11 pp11 pu12 pv12 pp12 ...... > > > > uu21 uv21 up21 uu22 uv22 up22 ...... > > vu21 vv21 vp21 vu22 vv22 vp22 ....... > > pu21 pv21 pp21 pu22 pv22 pp22 ...... > > > > . > > . > > > > ] > > > > I want to split u and v fields as one block and p field as another block. > > > > Any help would be appreciated. > > > > Raju. > > > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From nicola.varini at gmail.com Fri Jul 31 09:23:03 2020 From: nicola.varini at gmail.com (nicola varini) Date: Fri, 31 Jul 2020 16:23:03 +0200 Subject: [petsc-users] overlap cpu and gpu? Message-ID: Hello, I would like to know if it is possible to overlap CPU and GPU with DMDA. I've a machine where each node has 1P100+1Haswell. I've to resolve Poisson and Ampere equation for each time step. I'm using 2D DMDA for each of them. Would be possible to compute poisson and ampere equation at the same time? One on CPU and the other on GPU? Thanks -------------- next part -------------- An HTML attachment was scrubbed... URL: